From patchwork Fri Mar 11 23:08:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Kozyrev X-Patchwork-Id: 108696 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4AA45A00C2; Sat, 12 Mar 2022 00:08:47 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E2E1F4067B; Sat, 12 Mar 2022 00:08:46 +0100 (CET) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2067.outbound.protection.outlook.com [40.107.220.67]) by mails.dpdk.org (Postfix) with ESMTP id 024C640151 for ; Sat, 12 Mar 2022 00:08:45 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dxwN5mmL2itVY5MfG+nXca/XRFkX9S82yAQDvWwUd/aVLOxRG90eva65qnq0W+ie+00a3qLjzrxyy2yuyZegOq7Ny5Cqf6ajr1NcTliRIJCKXjfx2uDKrxXw79cOOx0m7sZUXo+oCDo1EgHwTR54Awj9Colk4DKGnToswgE4eqF4QftEnNaUCjVV0++SnWgX6kB5kem0l6Y6t4HPvi6EfV1Yf9CWKs3gRRtSXbNKBkfajhiH/PQgSwWuPeW1HBkluRMyu8TYUXpYI+rKrrU+B5zWsGj5ruSJ0iF3ok2szaKjtJOD+DNvBGNkzO9LXDIDVGmR9DKziyI56JrrGEIILQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qEcY/+C15WL4aLaR9pOnvC/IZbC9K7YwWEEVzm7tCJg=; b=D5rlYLAkfb9TLTnhAS/YOzReCg0Ls+YpXY1xhn3cYrBztWH3EoOdG1hd3DQAyjXPSx7vNJuQflI1KsMBoN2svgd1fxCaBp+jXtGuZzOT5r1qREgkUhdWjrhOHI3KZaaMOiPVZBcRhmOYz0cAn/euJmFngp2E+bL7mvCKfgp/E6fA7h62U7SCupoShP3UHWYj116LIqBQCdQHNVKv1JiAomJ3kptzuiUXFlXAlQKWYuaHmZAcn/sLXlI9fbMDNM4Vn3L6ACi1eNo1YnkvwQpoy4GjcbT2UGzx6cQH4CIOdhaeou/EJcVaPxfeDjBMCijmnblIlFbca5zY+6mNn+SdEQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qEcY/+C15WL4aLaR9pOnvC/IZbC9K7YwWEEVzm7tCJg=; b=mb7i4KRD6lFLZ2hfSZQDnToMv6JLQ+Nu5J2w3SFIFRtu2RDq4sKDa1rgxTt6eV+42Wvjxod/YJAuHZWF90ztL5hwsIVNtWYzaVC9DZsG7s4x9VwFj7CobDRmC2Bxbu1LCjrZj0Vpp1RXmzolIIvFamVRMpjHXwnfyXPoUFhBCnGVHy3nWlQ+s4kZfjojLujRZ7JU5m1QE2MGrZorBzqt9SMasjduI6GB+QOCB6xf3ytTPDfCFG90zHB6GTVtNGlVAo5V5TmQz3WxotUvbyPprexrbkdARcGVKYxa1K3kYNJ4gVC5u132U6TRhR0K33hxXMPuKySLPLs7AVVgFJC3Qw== Received: from MW4PR03CA0042.namprd03.prod.outlook.com (2603:10b6:303:8e::17) by CY4PR12MB1493.namprd12.prod.outlook.com (2603:10b6:910:11::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.27; Fri, 11 Mar 2022 23:08:43 +0000 Received: from CO1NAM11FT012.eop-nam11.prod.protection.outlook.com (2603:10b6:303:8e:cafe::a5) by MW4PR03CA0042.outlook.office365.com (2603:10b6:303:8e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5061.24 via Frontend Transport; Fri, 11 Mar 2022 23:08:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.235) by CO1NAM11FT012.mail.protection.outlook.com (10.13.175.192) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5061.22 via Frontend Transport; Fri, 11 Mar 2022 23:08:43 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Fri, 11 Mar 2022 23:08:42 +0000 Received: from pegasus01.mtr.labs.mlnx (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Fri, 11 Mar 2022 15:08:40 -0800 From: Alexander Kozyrev To: CC: , , Subject: [PATCH v3] net/mlx5: handle MPRQ incompatibility with external buffers Date: Sat, 12 Mar 2022 01:08:25 +0200 Message-ID: <20220311230825.3904165-1-akozyrev@nvidia.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20220310234518.3823100-1-akozyrev@nvidia.com> References: <20220310234518.3823100-1-akozyrev@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c44510e9-5e16-41e4-cf26-08da03b41681 X-MS-TrafficTypeDiagnostic: CY4PR12MB1493:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iqwgciN6qctiirA6e99VJ9Hf0uITU1E+NZC+0/xle0mkIble/YauTHPph/DgRXDemCIXGMgzsP7RWiBdqzsxpQ15BZcZrQX3VcMSl5M9bqg2h95ouI1fXINMViO0KRrr5Nw14FcdTCs7dw2igW0VcOkyYiHAsIB84kSNVail6grzRkHHvhEpAjjrBqkv3soun7+a6nM18EvMg70E44ZwZ2kDGvnR14hesZ8XAeR8Jcgrd6D4n3dkKdaq4F5y6xSPmBCNCf1iccYUbaJCZ7BGvlj/iS/gtvYGkD1KtqfRpSVpuq0iOn6sdNnGW4XiFIAfQN1VhXCcCkzbVCbBsOBUXQwabqsXU0pP402ZhlGizEwtbMCKf/vmlIbqUqFKvjStdqCwOW/ngO41cEv6JVctLY0CXPr6ve715QvyRhI2+R7COTJ8nOhbLXFONUGpayCRw/yIT+AfuAHedfa0nLCpesrMzlvaVbMFW5M4GOUXOSPSbTB5+8DkhhfeGjDRjjSK1GnTcAbj1jSihhLMU/MInXAbbsCoMzXXPxRGIlgRH8dIJbDYt8fFAuMw1yRsmTdefZrze1AHTqRRPwQDePcp6zXWRqiB8nqsNfTPyeBoizYOSAthqxlb8n+/F8wdRf4iFN+wutsBMiyk3ohcoa6wJHSKD/XnQikL4Zy55BEOGajAlaGZkvwbZP5XRVjJZa5azn2yi1ghhyZ6brOAUeCYVg== X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(8936002)(6916009)(316002)(107886003)(426003)(336012)(47076005)(26005)(186003)(1076003)(16526019)(86362001)(5660300002)(2616005)(82310400004)(83380400001)(508600001)(36756003)(36860700001)(54906003)(356005)(70206006)(81166007)(70586007)(4326008)(8676002)(40460700003)(2906002)(6666004)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Mar 2022 23:08:43.1855 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c44510e9-5e16-41e4-cf26-08da03b41681 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT012.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1493 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Multi-Packet Rx queue uses PMD-managed buffers to store packets. These buffers are externally attached to user mbufs. This conflicts with the feature that allows using user-managed externally attached buffers in an application. Fall back to SPRQ in case external buffers mempool is configured. Add the corresponding limitation to MLX5 documentation that MPRQ and external data buffers cannot be used together. Signed-off-by: Alexander Kozyrev Acked-by: Viacheslav Ovsiienko --- v3: fixed coverity issue with NULL pointer derefence v2: fixed typo in warning message doc/guides/nics/mlx5.rst | 4 +++- drivers/net/mlx5/mlx5_rx.h | 2 +- drivers/net/mlx5/mlx5_rxq.c | 23 +++++++++++++++-------- 3 files changed, 19 insertions(+), 10 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 679481bed5..4799875263 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -260,7 +260,9 @@ Limitations ol_flags. As the mempool for the external buffer is managed by PMD, all the Rx mbufs must be freed before the device is closed. Otherwise, the mempool of the external buffers will be freed by PMD and the application which still - holds the external buffers may be corrupted. + holds the external buffers may be corrupted. User-managed mempools with + external pinned data buffers cannot be used in conjunction with MPRQ + since packets may be already attached to PMD-managed external buffers. - If Multi-Packet Rx queue is configured (``mprq_en``) and Rx CQE compression is enabled (``rxq_cqe_comp_en``) at the same time, RSS hash result is not fully diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h index acebe3348c..5bf88b6181 100644 --- a/drivers/net/mlx5/mlx5_rx.h +++ b/drivers/net/mlx5/mlx5_rx.h @@ -209,7 +209,7 @@ struct mlx5_rxq_ctrl *mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_rxconf *conf, const struct rte_eth_rxseg_split *rx_seg, - uint16_t n_seg); + uint16_t n_seg, bool is_extmem); struct mlx5_rxq_ctrl *mlx5_rxq_hairpin_new (struct rte_eth_dev *dev, struct mlx5_rxq_priv *rxq, uint16_t desc, const struct rte_eth_hairpin_conf *hairpin_conf); diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index f16795bac3..925544ae3d 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -840,6 +840,7 @@ mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, int res; uint64_t offloads = conf->offloads | dev->data->dev_conf.rxmode.offloads; + bool is_extmem = false; if (mp) { /* @@ -849,6 +850,8 @@ mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, */ rx_seg = &rx_single; n_seg = 1; + is_extmem = rte_pktmbuf_priv_flags(mp) & + RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF; } if (n_seg > 1) { /* The offloads should be checked on rte_eth_dev layer. */ @@ -912,7 +915,7 @@ mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, } if (rxq_ctrl == NULL) { rxq_ctrl = mlx5_rxq_new(dev, idx, desc, socket, conf, rx_seg, - n_seg); + n_seg, is_extmem); if (rxq_ctrl == NULL) { DRV_LOG(ERR, "port %u unable to allocate rx queue index %u", dev->data->port_id, idx); @@ -1548,7 +1551,8 @@ mlx5_max_lro_msg_size_adjust(struct rte_eth_dev *dev, uint16_t idx, * Log number of strides to configure for this queue. * @param actual_log_stride_size * Log stride size to configure for this queue. - * + * @param is_extmem + * Is external pinned memory pool used. * @return * 0 if Multi-Packet RQ is supported, otherwise -1. */ @@ -1556,7 +1560,8 @@ static int mlx5_mprq_prepare(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, bool rx_seg_en, uint32_t min_mbuf_size, uint32_t *actual_log_stride_num, - uint32_t *actual_log_stride_size) + uint32_t *actual_log_stride_size, + bool is_extmem) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_port_config *config = &priv->config; @@ -1575,7 +1580,7 @@ mlx5_mprq_prepare(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, log_max_stride_size); uint32_t log_stride_wqe_size; - if (mlx5_check_mprq_support(dev) != 1 || rx_seg_en) + if (mlx5_check_mprq_support(dev) != 1 || rx_seg_en || is_extmem) goto unsupport; /* Checks if chosen number of strides is in supported range. */ if (config->mprq.log_stride_num > log_max_stride_num || @@ -1641,7 +1646,7 @@ mlx5_mprq_prepare(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, " rxq_num = %u, stride_sz = %u, stride_num = %u\n" " supported: min_rxqs_num = %u, min_buf_wqe_sz = %u" " min_stride_sz = %u, max_stride_sz = %u).\n" - "Rx segment is %senable.", + "Rx segment is %senabled. External mempool is %sused.", dev->data->port_id, min_mbuf_size, desc, priv->rxqs_n, RTE_BIT32(config->mprq.log_stride_size), RTE_BIT32(config->mprq.log_stride_num), @@ -1649,7 +1654,7 @@ mlx5_mprq_prepare(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, RTE_BIT32(dev_cap->mprq.log_min_stride_wqe_size), RTE_BIT32(dev_cap->mprq.log_min_stride_size), RTE_BIT32(dev_cap->mprq.log_max_stride_size), - rx_seg_en ? "" : "not "); + rx_seg_en ? "" : "not ", is_extmem ? "" : "not "); return -1; } @@ -1671,7 +1676,8 @@ mlx5_mprq_prepare(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, struct mlx5_rxq_ctrl * mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, const struct rte_eth_rxconf *conf, - const struct rte_eth_rxseg_split *rx_seg, uint16_t n_seg) + const struct rte_eth_rxseg_split *rx_seg, uint16_t n_seg, + bool is_extmem) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_rxq_ctrl *tmpl; @@ -1694,7 +1700,8 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, const int mprq_en = !mlx5_mprq_prepare(dev, idx, desc, rx_seg_en, non_scatter_min_mbuf_size, &mprq_log_actual_stride_num, - &mprq_log_actual_stride_size); + &mprq_log_actual_stride_size, + is_extmem); /* * Always allocate extra slots, even if eventually * the vector Rx will not be used.