From patchwork Thu Oct 6 11:01:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dariusz Sosnowski X-Patchwork-Id: 117451 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 718DCA00C2; Thu, 6 Oct 2022 13:02:24 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7396F42C11; Thu, 6 Oct 2022 13:02:11 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2061.outbound.protection.outlook.com [40.107.223.61]) by mails.dpdk.org (Postfix) with ESMTP id 4682E42C11 for ; Thu, 6 Oct 2022 13:02:10 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PQ1D6rtcuqeOMmJ0jjXD8ZTDnpExBrR6jbD9UEDyxsD4t8ni3MSCL8hhfbx+QrML1UiIXO+270KAI6lh848z7dglItMUlumli0JQX9IMmXh8WvmljXopGEW0a/YqWby3/WjlAMQQMp79NHKmoGAbJCATevazmvuLJQr0EtnYlYTs2zdbDUC7rmYecIKlkszq8das/sX5xIgGLNWf09OZkyoxXJe2T8H6rTv2Hi0zcTRdeOLkh1mT9oOK+FCrowftbfhY8Xv8mzcb/iHLvDmkfsJENPrbgiv6N9o1f9Hn9fZwuCsF3e14Wpcypin1IT2RfAE0LAmvYP6RV0cg0VT5Cg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OZ+DO+gmAr5KDH02vPQCSuYdZBiu00MfWbjeD33mTBM=; b=TiXoYPvSgsaNVIHqyW7UuQcdBS0S8TKhpGan+sATz9knZUrp3uLaeQQNjgjFfTu0DYe9gwzm2swxatEPqHJDZP4KjG416q9xgMcvqIGVMQJrc5HsNvMAuY/I/78JqX2j7KIhJJAHBGkPkweNnFLWgnapOYFdosHCCXisPz1COF8eQTyu/YYcDXmYGsfUmyvqCyuBwBej+9YgW+YEInx3N8tk8G4VSZYHXFC+fq37aLpoOMEtIY6AwVkd8U1hKLTyVLLRG+qgM1Amuc6oES4s4SJLkc+d1SQitcn6aQRMqYOnMjv8AbSGPuvpF7bYZQDfiLq4tmPv8aLLPTL1eSj0wg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OZ+DO+gmAr5KDH02vPQCSuYdZBiu00MfWbjeD33mTBM=; b=Nd+2xkwRh48PUJLwqo6oSFS7OqumXm1a/jjlj0YyujmnXsib0VOVy5VvaJN6qM+e6qnO6vM42XaUh0vFbsdQYOn+LCEeKsPRGLTQgih6nnCaUs8+VVczCflAHyQFsODVKI0vY2KO61PHOnrsa8o76ZVs9XL75fx9MceFdvrRGwDHqkQ0ido1BxH53Iyak4FnJEHTP47GAwiSb0o07FDNvv/5WyH8fReuQ9WBpJHl87xtV6X08x1bCeXBnRfX0PBQcDgK7DISH/JjbQM4xktF8/fQXH7Bv/99wPDq32WzuW6uUR7Lt50AWt1Goj8e3U+6MSd2ZMXdsw66+vhVJMiXjw== Received: from DS7PR03CA0017.namprd03.prod.outlook.com (2603:10b6:5:3b8::22) by BL3PR12MB6473.namprd12.prod.outlook.com (2603:10b6:208:3b9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5654.20; Thu, 6 Oct 2022 11:02:08 +0000 Received: from DM6NAM11FT024.eop-nam11.prod.protection.outlook.com (2603:10b6:5:3b8:cafe::3a) by DS7PR03CA0017.outlook.office365.com (2603:10b6:5:3b8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.26 via Frontend Transport; Thu, 6 Oct 2022 11:02:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT024.mail.protection.outlook.com (10.13.172.159) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5709.10 via Frontend Transport; Thu, 6 Oct 2022 11:02:07 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Thu, 6 Oct 2022 04:01:55 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Thu, 6 Oct 2022 04:01:53 -0700 From: Dariusz Sosnowski To: Matan Azrad , Viacheslav Ovsiienko CC: Subject: [PATCH v2 3/8] common/mlx5: add hairpin RQ buffer type capabilities Date: Thu, 6 Oct 2022 11:01:00 +0000 Message-ID: <20221006110105.2986966-4-dsosnowski@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221006110105.2986966-1-dsosnowski@nvidia.com> References: <20221006110105.2986966-1-dsosnowski@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT024:EE_|BL3PR12MB6473:EE_ X-MS-Office365-Filtering-Correlation-Id: e1270f39-6215-4938-9160-08daa78a3619 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: PTwRUHVQMz+qv7ILmNuYbkrb3hOvCFNdl3zu9UquqsKRwCmyRFgfswCQh7iGbaBnQnAJJqDgNmQLAkJVPljVosI6r4TZd2G+c5INnfcRMFT8HTn+xaAE2c76qzm1f7tj9wMKEnCQ4Hu1sADA6ORQZERQOHSNNsv+SsGSuiT0ER2jI8udccQ6X0p11p+HK4TzBTH3NknqFfA6UEeLfpmmjBGIu3n28+KQ6MDvPKs6FBOxgp1HkZHP0bLJ0UuexcNdi1nj+TD+nX/ZS5c/STBukJXUsx86h/anqctfRT2lmJcYpU5EemKbxYCM37h69DiZJ3ro2XWy5wQQAqaWaa1UgnWW7ft5qYVifhB2xKiFeLQJ58qcx7PHW6v1gC8M3iJDWCGzJEviCi8uII37/aiNbv1rF/vFR3zj+j1kGFtKnKEOkExvmjJzUOl23iRVb6XcFnhQYmxDrqFlS4B+K9rU7ZJ95QEAJjyB/1kC7tHcDXPVXfDKbjs2a3Fe5m9r1EhvnwUGgkgsoz/GynIXN9RXU7davjjA/nZRgAZ9w0o3U5ouaTLmWHufW6w5+qytADf/K/b6cZlcfypJcL0miaYY+/Xm3fs+Ra64LfUWiTg6KC/9z1+8OCuAEQu3C3QioCWV4fzMK0k/p0iMqf7ZhLMeybYCj/PTnRx+TDQ4wzutrE+zfTrJSknQwnfRoDgqcPZ67lwdNHXU1BZN3NRAWQzE45HQck+HZfit4s/52Q9mu1B9xH5ZfwmOkmS+NRVGv//WszU8TVfcEOUS+YI5vQDSkQ== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230022)(4636009)(346002)(376002)(39860400002)(396003)(136003)(451199015)(40470700004)(36840700001)(46966006)(1076003)(36756003)(55016003)(82310400005)(40480700001)(40460700003)(110136005)(83380400001)(2616005)(426003)(70206006)(47076005)(16526019)(41300700001)(6666004)(8676002)(2906002)(478600001)(186003)(7696005)(8936002)(6286002)(86362001)(5660300002)(336012)(356005)(36860700001)(82740400003)(26005)(4326008)(70586007)(6636002)(316002)(7636003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2022 11:02:07.9598 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e1270f39-6215-4938-9160-08daa78a3619 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT024.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR12MB6473 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch adds new HCA capability related to hairpin RQs. This new capability, hairpin_data_buffer_locked, indicates whether HCA supports locking data buffer of hairpin RQ in ICMC (Interconnect Context Memory Cache). Struct used to define RQ configuration (RQ context) is extended with hairpin_data_buffer_type field, which configures data buffer for hairpin RQ. It can take the following values: - MLX5_RQC_HAIRPIN_DATA_BUFFER_TYPE_UNLOCKED_INTERNAL_BUFFER - hairpin RQ's data buffer is stored in unlocked memory in ICMC. - MLX5_RQC_HAIRPIN_DATA_BUFFER_TYPE_LOCKED_INTERNAL_BUFFER - hairpin RQ's data buffer is stored in locked memory in ICMC. Signed-off-by: Dariusz Sosnowski Acked-by: Viacheslav Ovsiienko --- drivers/common/mlx5/mlx5_devx_cmds.c | 3 +++ drivers/common/mlx5/mlx5_devx_cmds.h | 2 ++ drivers/common/mlx5/mlx5_prm.h | 12 ++++++++++-- 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c index a1e8179568..76f0b6724f 100644 --- a/drivers/common/mlx5/mlx5_devx_cmds.c +++ b/drivers/common/mlx5/mlx5_devx_cmds.c @@ -993,6 +993,8 @@ mlx5_devx_cmd_query_hca_attr(void *ctx, hairpin_sq_wqe_bb_size); attr->hairpin_sq_wq_in_host_mem = MLX5_GET(cmd_hca_cap_2, hcattr, hairpin_sq_wq_in_host_mem); + attr->hairpin_data_buffer_locked = MLX5_GET(cmd_hca_cap_2, hcattr, + hairpin_data_buffer_locked); } if (attr->log_min_stride_wqe_sz == 0) attr->log_min_stride_wqe_sz = MLX5_MPRQ_LOG_MIN_STRIDE_WQE_SIZE; @@ -1293,6 +1295,7 @@ mlx5_devx_cmd_create_rq(void *ctx, MLX5_SET(rqc, rq_ctx, state, rq_attr->state); MLX5_SET(rqc, rq_ctx, flush_in_error_en, rq_attr->flush_in_error_en); MLX5_SET(rqc, rq_ctx, hairpin, rq_attr->hairpin); + MLX5_SET(rqc, rq_ctx, hairpin_data_buffer_type, rq_attr->hairpin_data_buffer_type); MLX5_SET(rqc, rq_ctx, user_index, rq_attr->user_index); MLX5_SET(rqc, rq_ctx, cqn, rq_attr->cqn); MLX5_SET(rqc, rq_ctx, counter_set_id, rq_attr->counter_set_id); diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h index 9ac2d75df4..cceaf3411d 100644 --- a/drivers/common/mlx5/mlx5_devx_cmds.h +++ b/drivers/common/mlx5/mlx5_devx_cmds.h @@ -193,6 +193,7 @@ struct mlx5_hca_attr { uint32_t log_max_hairpin_num_packets:5; uint32_t hairpin_sq_wqe_bb_size:4; uint32_t hairpin_sq_wq_in_host_mem:1; + uint32_t hairpin_data_buffer_locked:1; uint32_t vhca_id:16; uint32_t relaxed_ordering_write:1; uint32_t relaxed_ordering_read:1; @@ -313,6 +314,7 @@ struct mlx5_devx_create_rq_attr { uint32_t state:4; uint32_t flush_in_error_en:1; uint32_t hairpin:1; + uint32_t hairpin_data_buffer_type:3; uint32_t ts_format:2; uint32_t user_index:24; uint32_t cqn:24; diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 04d35ca845..9c1c93f916 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -2024,7 +2024,8 @@ struct mlx5_ifc_cmd_hca_cap_2_bits { u8 reserved_at_160[0x3]; u8 hairpin_sq_wqe_bb_size[0x5]; u8 hairpin_sq_wq_in_host_mem[0x1]; - u8 reserved_at_169[0x697]; + u8 hairpin_data_buffer_locked[0x1]; + u8 reserved_at_16a[0x696]; }; struct mlx5_ifc_esw_cap_bits { @@ -2304,7 +2305,9 @@ struct mlx5_ifc_rqc_bits { u8 reserved_at_c[0x1]; u8 flush_in_error_en[0x1]; u8 hairpin[0x1]; - u8 reserved_at_f[0xB]; + u8 reserved_at_f[0x6]; + u8 hairpin_data_buffer_type[0x3]; + u8 reserved_at_a8[0x2]; u8 ts_format[0x02]; u8 reserved_at_1c[0x4]; u8 reserved_at_20[0x8]; @@ -2813,6 +2816,11 @@ enum { MLX5_CQE_SIZE_128B = 0x1, }; +enum { + MLX5_RQC_HAIRPIN_DATA_BUFFER_TYPE_UNLOCKED_INTERNAL_BUFFER = 0x0, + MLX5_RQC_HAIRPIN_DATA_BUFFER_TYPE_LOCKED_INTERNAL_BUFFER = 0x1, +}; + struct mlx5_ifc_cqc_bits { u8 status[0x4]; u8 as_notify[0x1];