From patchwork Thu Oct 1 14:09:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79470 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id A66E8A04BA; Thu, 1 Oct 2020 16:10:35 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0FB1A1DB82; Thu, 1 Oct 2020 16:10:25 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 3A3C81DB7C for ; Thu, 1 Oct 2020 16:10:23 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:10:17 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAEH012743; Thu, 1 Oct 2020 17:10:17 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko , stable@dpdk.org Date: Thu, 1 Oct 2020 14:09:12 +0000 Message-Id: <1601561366-1821-2-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 01/15] net/mlx5: fix send queue doorbell typo X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" As part of SQ creation for Tx queue objects, a HW doorbell memory should be allocated and mapped to the HW. The SQ doorbell handler was wrongly saved on the CQ fields what caused wrong doorbell release in the Tx queue object destroy flow. Save the SQ doorbell handler in the SQ fields. Fixes: 3a87b964edd3 ("net/mlx5: create Tx queues with DevX") Cc: stable@dpdk.org Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5_txq.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 1bb667d..fc730fa 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -1050,8 +1050,8 @@ dev->data->port_id, txq_data->idx); goto error; } - /* Allocate doorbell record for completion queue. */ - txq_obj->cq_dbrec_offset = mlx5_get_dbr(sh->ctx, + /* Allocate doorbell record for send queue. */ + txq_obj->sq_dbrec_offset = mlx5_get_dbr(sh->ctx, &priv->dbrpgs, &txq_obj->sq_dbrec_page); if (txq_obj->sq_dbrec_offset < 0) @@ -1076,9 +1076,9 @@ sq_attr.wq_attr.log_wq_stride = rte_log2_u32(MLX5_WQE_SIZE); sq_attr.wq_attr.log_wq_sz = txq_data->wqe_n; sq_attr.wq_attr.dbr_umem_valid = 1; - sq_attr.wq_attr.dbr_addr = txq_obj->cq_dbrec_offset; + sq_attr.wq_attr.dbr_addr = txq_obj->sq_dbrec_offset; sq_attr.wq_attr.dbr_umem_id = - mlx5_os_get_umem_id(txq_obj->cq_dbrec_page->umem); + mlx5_os_get_umem_id(txq_obj->sq_dbrec_page->umem); sq_attr.wq_attr.wq_umem_valid = 1; sq_attr.wq_attr.wq_umem_id = mlx5_os_get_umem_id(txq_obj->sq_umem); sq_attr.wq_attr.wq_umem_offset = (uintptr_t)txq_obj->sq_buf % page_size; From patchwork Thu Oct 1 14:09:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79471 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id A7D48A04BA; Thu, 1 Oct 2020 16:10:57 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B5ADD1DB8F; Thu, 1 Oct 2020 16:10:29 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 4A06E1DB8E for ; Thu, 1 Oct 2020 16:10:28 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:10:24 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAEI012743; Thu, 1 Oct 2020 17:10:24 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko , stable@dpdk.org Date: Thu, 1 Oct 2020 14:09:13 +0000 Message-Id: <1601561366-1821-3-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 02/15] net/mlx5: fix unused variable in Txq creation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When a CQ is not created by DevX, it be allocated by either DV function or by regular Verbs function. The CQ DV attributes variable was wrongly defined and initialized in Tx queue creation while the CQ is created by the regular Verbs function what remained the attributes variable unused. Remove the unused variable. Fixes: faf2667fe8d5 ("net/mlx5: separate DPDK from verbs Tx queue objects") Cc: stable@dpdk.org Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5_txq.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index fc730fa..ef3137b 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -1168,7 +1168,6 @@ struct mlx5_txq_obj * struct mlx5_txq_obj *txq_obj = NULL; union { struct ibv_qp_init_attr_ex init; - struct ibv_cq_init_attr_ex cq; struct ibv_qp_attr mod; } attr; unsigned int cqe_n; @@ -1198,9 +1197,6 @@ struct mlx5_txq_obj * return NULL; } memset(&tmpl, 0, sizeof(struct mlx5_txq_obj)); - attr.cq = (struct ibv_cq_init_attr_ex){ - .comp_mask = 0, - }; cqe_n = desc / MLX5_TX_COMP_THRESH + 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; tmpl.cq = mlx5_glue->create_cq(priv->sh->ctx, cqe_n, NULL, NULL, 0); From patchwork Thu Oct 1 14:09:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79472 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3292BA04BA; Thu, 1 Oct 2020 16:11:21 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 78E821DB7F; Thu, 1 Oct 2020 16:10:39 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 718891BF7B for ; Thu, 1 Oct 2020 16:10:37 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:10:31 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAEJ012743; Thu, 1 Oct 2020 17:10:31 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:14 +0000 Message-Id: <1601561366-1821-4-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 03/15] net/mlx5: mitigate Tx queue reference counters X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The Tx queue structures manage 2 different reference counter per queue: txq_ctrl reference counter and txq_obj reference counter. There is no real need to use two different counters, it just complicates the release functions. Remove the txq_obj counter and use only the txq_ctrl counter. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5_rxtx.h | 4 +- drivers/net/mlx5/mlx5_txq.c | 98 ++++++++++++++------------------------------ 2 files changed, 32 insertions(+), 70 deletions(-) diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 9ffa028..d947e0e 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -276,7 +276,6 @@ enum mlx5_txq_type { /* Verbs/DevX Tx queue elements. */ struct mlx5_txq_obj { LIST_ENTRY(mlx5_txq_obj) next; /* Pointer to the next element. */ - rte_atomic32_t refcnt; /* Reference counter. */ struct mlx5_txq_ctrl *txq_ctrl; /* Pointer to the control queue. */ enum mlx5_txq_obj_type type; /* The txq object type. */ RTE_STD_C11 @@ -405,8 +404,7 @@ int mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, void mlx5_tx_uar_uninit_secondary(struct rte_eth_dev *dev); struct mlx5_txq_obj *mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx, enum mlx5_txq_obj_type type); -struct mlx5_txq_obj *mlx5_txq_obj_get(struct rte_eth_dev *dev, uint16_t idx); -int mlx5_txq_obj_release(struct mlx5_txq_obj *txq_ibv); +void mlx5_txq_obj_release(struct mlx5_txq_obj *txq_obj); int mlx5_txq_obj_verify(struct rte_eth_dev *dev); struct mlx5_txq_ctrl *mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index ef3137b..e8bf7d7 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -437,6 +437,7 @@ mlx5_txq_release(dev, idx); return 0; } + /** * DPDK callback to configure a TX queue. * @@ -833,7 +834,6 @@ } DRV_LOG(DEBUG, "port %u sxq %u updated with %p", dev->data->port_id, idx, (void *)&tmpl); - rte_atomic32_inc(&tmpl->refcnt); LIST_INSERT_HEAD(&priv->txqsobj, tmpl, next); return tmpl; } @@ -1126,7 +1126,6 @@ txq_ctrl->bf_reg = reg_addr; txq_ctrl->uar_mmap_offset = mlx5_os_get_devx_uar_mmap_offset(sh->tx_uar); - rte_atomic32_set(&txq_obj->refcnt, 1); txq_uar_init(txq_ctrl); LIST_INSERT_HEAD(&priv->txqsobj, txq_obj, next); return txq_obj; @@ -1360,7 +1359,6 @@ struct mlx5_txq_obj * #endif txq_obj->qp = tmpl.qp; txq_obj->cq = tmpl.cq; - rte_atomic32_inc(&txq_obj->refcnt); txq_ctrl->bf_reg = qp.bf.reg; if (qp.comp_mask & MLX5DV_QP_MASK_UAR_MMAP_OFFSET) { txq_ctrl->uar_mmap_offset = qp.uar_mmap_offset; @@ -1397,64 +1395,30 @@ struct mlx5_txq_obj * } /** - * Get an Tx queue Verbs object. - * - * @param dev - * Pointer to Ethernet device. - * @param idx - * Queue index in DPDK Tx queue array. - * - * @return - * The Verbs object if it exists. - */ -struct mlx5_txq_obj * -mlx5_txq_obj_get(struct rte_eth_dev *dev, uint16_t idx) -{ - struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_txq_ctrl *txq_ctrl; - - if (idx >= priv->txqs_n) - return NULL; - if (!(*priv->txqs)[idx]) - return NULL; - txq_ctrl = container_of((*priv->txqs)[idx], struct mlx5_txq_ctrl, txq); - if (txq_ctrl->obj) - rte_atomic32_inc(&txq_ctrl->obj->refcnt); - return txq_ctrl->obj; -} - -/** * Release an Tx verbs queue object. * * @param txq_obj - * Verbs Tx queue object. - * - * @return - * 1 while a reference on it exists, 0 when freed. + * Verbs Tx queue object.. */ -int +void mlx5_txq_obj_release(struct mlx5_txq_obj *txq_obj) { MLX5_ASSERT(txq_obj); - if (rte_atomic32_dec_and_test(&txq_obj->refcnt)) { - if (txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN) { - if (txq_obj->tis) - claim_zero(mlx5_devx_cmd_destroy(txq_obj->tis)); - } else if (txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ) { - txq_release_sq_resources(txq_obj); - } else { - claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); - claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); - } - if (txq_obj->txq_ctrl->txq.fcqs) { - mlx5_free(txq_obj->txq_ctrl->txq.fcqs); - txq_obj->txq_ctrl->txq.fcqs = NULL; - } - LIST_REMOVE(txq_obj, next); - mlx5_free(txq_obj); - return 0; + if (txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN) { + if (txq_obj->tis) + claim_zero(mlx5_devx_cmd_destroy(txq_obj->tis)); + } else if (txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ) { + txq_release_sq_resources(txq_obj); + } else { + claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); + claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); } - return 1; + if (txq_obj->txq_ctrl->txq.fcqs) { + mlx5_free(txq_obj->txq_ctrl->txq.fcqs); + txq_obj->txq_ctrl->txq.fcqs = NULL; + } + LIST_REMOVE(txq_obj, next); + mlx5_free(txq_obj); } /** @@ -1967,12 +1931,11 @@ struct mlx5_txq_ctrl * mlx5_txq_get(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; struct mlx5_txq_ctrl *ctrl = NULL; - if ((*priv->txqs)[idx]) { - ctrl = container_of((*priv->txqs)[idx], struct mlx5_txq_ctrl, - txq); - mlx5_txq_obj_get(dev, idx); + if (txq_data) { + ctrl = container_of(txq_data, struct mlx5_txq_ctrl, txq); rte_atomic32_inc(&ctrl->refcnt); } return ctrl; @@ -1998,18 +1961,19 @@ struct mlx5_txq_ctrl * if (!(*priv->txqs)[idx]) return 0; txq = container_of((*priv->txqs)[idx], struct mlx5_txq_ctrl, txq); - if (txq->obj && !mlx5_txq_obj_release(txq->obj)) + if (!rte_atomic32_dec_and_test(&txq->refcnt)) + return 1; + if (txq->obj) { + mlx5_txq_obj_release(txq->obj); txq->obj = NULL; - if (rte_atomic32_dec_and_test(&txq->refcnt)) { - txq_free_elts(txq); - mlx5_mr_btree_free(&txq->txq.mr_ctrl.cache_bh); - LIST_REMOVE(txq, next); - mlx5_free(txq); - (*priv->txqs)[idx] = NULL; - dev->data->tx_queue_state[idx] = RTE_ETH_QUEUE_STATE_STOPPED; - return 0; } - return 1; + txq_free_elts(txq); + mlx5_mr_btree_free(&txq->txq.mr_ctrl.cache_bh); + LIST_REMOVE(txq, next); + mlx5_free(txq); + (*priv->txqs)[idx] = NULL; + dev->data->tx_queue_state[idx] = RTE_ETH_QUEUE_STATE_STOPPED; + return 0; } /** From patchwork Thu Oct 1 14:09:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79473 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id DE6AFA04BA; Thu, 1 Oct 2020 16:11:49 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6D6701DBAB; Thu, 1 Oct 2020 16:10:43 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 8D18E1DBAB for ; Thu, 1 Oct 2020 16:10:41 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:10:39 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAEK012743; Thu, 1 Oct 2020 17:10:39 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:15 +0000 Message-Id: <1601561366-1821-5-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 04/15] net/mlx5: reorder Tx queue DevX object creation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Move the creation of the send queue and the completion queue resources from the mlx5_txq_obj_devx_new function into auxiliary functions. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5_txq.c | 361 +++++++++++++++++++++++++++++--------------- 1 file changed, 239 insertions(+), 122 deletions(-) diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index e8bf7d7..686b452 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -839,140 +839,133 @@ } /** - * Destroy the Tx queue DevX object. + * Release DevX SQ resources. * - * @param txq_obj - * Txq object to destroy + * @param txq_ctrl + * DevX Tx queue object. */ static void -txq_release_sq_resources(struct mlx5_txq_obj *txq_obj) +txq_release_devx_sq_resources(struct mlx5_txq_obj *txq_obj) { - MLX5_ASSERT(txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ); - if (txq_obj->sq_devx) claim_zero(mlx5_devx_cmd_destroy(txq_obj->sq_devx)); - if (txq_obj->sq_dbrec_page) - claim_zero(mlx5_release_dbr - (&txq_obj->txq_ctrl->priv->dbrpgs, - mlx5_os_get_umem_id - (txq_obj->sq_dbrec_page->umem), - txq_obj->sq_dbrec_offset)); if (txq_obj->sq_umem) claim_zero(mlx5_glue->devx_umem_dereg(txq_obj->sq_umem)); if (txq_obj->sq_buf) mlx5_free(txq_obj->sq_buf); + if (txq_obj->sq_dbrec_page) + claim_zero(mlx5_release_dbr(&txq_obj->txq_ctrl->priv->dbrpgs, + mlx5_os_get_umem_id + (txq_obj->sq_dbrec_page->umem), + txq_obj->sq_dbrec_offset)); +} + +/** + * Release DevX Tx CQ resources. + * + * @param txq_ctrl + * DevX Tx queue object. + */ +static void +txq_release_devx_cq_resources(struct mlx5_txq_obj *txq_obj) +{ if (txq_obj->cq_devx) claim_zero(mlx5_devx_cmd_destroy(txq_obj->cq_devx)); - if (txq_obj->cq_dbrec_page) - claim_zero(mlx5_release_dbr - (&txq_obj->txq_ctrl->priv->dbrpgs, - mlx5_os_get_umem_id - (txq_obj->cq_dbrec_page->umem), - txq_obj->cq_dbrec_offset)); if (txq_obj->cq_umem) claim_zero(mlx5_glue->devx_umem_dereg(txq_obj->cq_umem)); if (txq_obj->cq_buf) mlx5_free(txq_obj->cq_buf); + if (txq_obj->cq_dbrec_page) + claim_zero(mlx5_release_dbr(&txq_obj->txq_ctrl->priv->dbrpgs, + mlx5_os_get_umem_id + (txq_obj->cq_dbrec_page->umem), + txq_obj->cq_dbrec_offset)); } /** - * Create the Tx queue DevX object. + * Destroy the Tx queue DevX object. + * + * @param txq_obj + * Txq object to destroy + */ +static void +txq_release_devx_resources(struct mlx5_txq_obj *txq_obj) +{ + MLX5_ASSERT(txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ); + + txq_release_devx_sq_resources(txq_obj); + txq_release_devx_cq_resources(txq_obj); +} + +#ifdef HAVE_MLX5DV_DEVX_UAR_OFFSET +/** + * Create a DevX CQ object for a Tx queue. * * @param dev * Pointer to Ethernet device. + * @param cqe_n + * Number of entries in the CQ. * @param idx - * Queue index in DPDK Tx queue array + * Queue index in DPDK Tx queue array. + * @param type + * Type of the Tx queue object to create. * * @return - * The DevX object initialised, NULL otherwise and rte_errno is set. + * The DevX CQ object initialized, NULL otherwise and rte_errno is set. */ -static struct mlx5_txq_obj * -mlx5_txq_obj_devx_new(struct rte_eth_dev *dev, uint16_t idx) +static struct mlx5_devx_obj * +mlx5_devx_cq_new(struct rte_eth_dev *dev, uint32_t cqe_n, uint16_t idx, + struct mlx5_txq_obj *txq_obj) { -#ifndef HAVE_MLX5DV_DEVX_UAR_OFFSET - DRV_LOG(ERR, "port %u Tx queue %u cannot create with DevX, no UAR", - dev->data->port_id, idx); - rte_errno = ENOMEM; - return NULL; -#else struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_dev_ctx_shared *sh = priv->sh; struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; - struct mlx5_txq_ctrl *txq_ctrl = - container_of(txq_data, struct mlx5_txq_ctrl, txq); - struct mlx5_devx_create_sq_attr sq_attr = { 0 }; - struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; + struct mlx5_devx_obj *cq_obj = NULL; struct mlx5_devx_cq_attr cq_attr = { 0 }; - struct mlx5_txq_obj *txq_obj = NULL; - size_t page_size; struct mlx5_cqe *cqe; - uint32_t i, nqe; - void *reg_addr; - size_t alignment = (size_t)-1; - int ret = 0; + size_t page_size; + size_t alignment; + uint32_t i; + int ret; MLX5_ASSERT(txq_data); - MLX5_ASSERT(!txq_ctrl->obj); + MLX5_ASSERT(txq_obj); page_size = rte_mem_page_size(); if (page_size == (size_t)-1) { DRV_LOG(ERR, "Failed to get mem page size"); rte_errno = ENOMEM; return NULL; } - txq_obj = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - sizeof(struct mlx5_txq_obj), 0, - txq_ctrl->socket); - if (!txq_obj) { - DRV_LOG(ERR, - "port %u Tx queue %u cannot allocate memory resources", - dev->data->port_id, txq_data->idx); + /* Allocate memory buffer for CQEs. */ + alignment = MLX5_CQE_BUF_ALIGNMENT; + if (alignment == (size_t)-1) { + DRV_LOG(ERR, "Failed to get CQE buf alignment"); rte_errno = ENOMEM; return NULL; } - txq_obj->type = MLX5_TXQ_OBJ_TYPE_DEVX_SQ; - txq_obj->txq_ctrl = txq_ctrl; - txq_obj->dev = dev; - /* Create the Completion Queue. */ - nqe = (1UL << txq_data->elts_n) / MLX5_TX_COMP_THRESH + - 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; - nqe = 1UL << log2above(nqe); - if (nqe > UINT16_MAX) { + cqe_n = 1UL << log2above(cqe_n); + if (cqe_n > UINT16_MAX) { DRV_LOG(ERR, "port %u Tx queue %u requests to many CQEs %u", - dev->data->port_id, txq_data->idx, nqe); + dev->data->port_id, txq_data->idx, cqe_n); rte_errno = EINVAL; - goto error; - } - /* Allocate memory buffer for CQEs. */ - alignment = MLX5_CQE_BUF_ALIGNMENT; - if (alignment == (size_t)-1) { - DRV_LOG(ERR, "Failed to get mem page size"); - rte_errno = ENOMEM; - goto error; + return NULL; } txq_obj->cq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - nqe * sizeof(struct mlx5_cqe), + cqe_n * sizeof(struct mlx5_cqe), alignment, - sh->numa_node); + priv->sh->numa_node); if (!txq_obj->cq_buf) { DRV_LOG(ERR, "port %u Tx queue %u cannot allocate memory (CQ)", dev->data->port_id, txq_data->idx); rte_errno = ENOMEM; - goto error; + return NULL; } - txq_data->cqe_n = log2above(nqe); - txq_data->cqe_s = 1 << txq_data->cqe_n; - txq_data->cqe_m = txq_data->cqe_s - 1; - txq_data->cqes = (volatile struct mlx5_cqe *)txq_obj->cq_buf; - txq_data->cq_ci = 0; - txq_data->cq_pi = 0; /* Register allocated buffer in user space with DevX. */ - txq_obj->cq_umem = mlx5_glue->devx_umem_reg - (sh->ctx, - (void *)txq_obj->cq_buf, - nqe * sizeof(struct mlx5_cqe), - IBV_ACCESS_LOCAL_WRITE); + txq_obj->cq_umem = mlx5_glue->devx_umem_reg(priv->sh->ctx, + (void *)txq_obj->cq_buf, + cqe_n * sizeof(struct mlx5_cqe), + IBV_ACCESS_LOCAL_WRITE); if (!txq_obj->cq_umem) { rte_errno = errno; DRV_LOG(ERR, @@ -981,46 +974,88 @@ goto error; } /* Allocate doorbell record for completion queue. */ - txq_obj->cq_dbrec_offset = mlx5_get_dbr(sh->ctx, + txq_obj->cq_dbrec_offset = mlx5_get_dbr(priv->sh->ctx, &priv->dbrpgs, &txq_obj->cq_dbrec_page); - if (txq_obj->cq_dbrec_offset < 0) + if (txq_obj->cq_dbrec_offset < 0) { + rte_errno = errno; + DRV_LOG(ERR, "Failed to allocate CQ door-bell."); goto error; - txq_data->cq_db = (volatile uint32_t *)(txq_obj->cq_dbrec_page->dbrs + - txq_obj->cq_dbrec_offset); - *txq_data->cq_db = 0; - /* Create completion queue object with DevX. */ + } cq_attr.cqe_size = (sizeof(struct mlx5_cqe) == 128) ? MLX5_CQE_SIZE_128B : MLX5_CQE_SIZE_64B; - cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(sh->tx_uar); - cq_attr.eqn = sh->txpp.eqn; + cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar); + cq_attr.eqn = priv->sh->txpp.eqn; cq_attr.q_umem_valid = 1; cq_attr.q_umem_offset = (uintptr_t)txq_obj->cq_buf % page_size; cq_attr.q_umem_id = mlx5_os_get_umem_id(txq_obj->cq_umem); cq_attr.db_umem_valid = 1; cq_attr.db_umem_offset = txq_obj->cq_dbrec_offset; cq_attr.db_umem_id = mlx5_os_get_umem_id(txq_obj->cq_dbrec_page->umem); - cq_attr.log_cq_size = rte_log2_u32(nqe); + cq_attr.log_cq_size = rte_log2_u32(cqe_n); cq_attr.log_page_size = rte_log2_u32(page_size); - txq_obj->cq_devx = mlx5_devx_cmd_create_cq(sh->ctx, &cq_attr); - if (!txq_obj->cq_devx) { + /* Create completion queue object with DevX. */ + cq_obj = mlx5_devx_cmd_create_cq(priv->sh->ctx, &cq_attr); + if (!cq_obj) { rte_errno = errno; DRV_LOG(ERR, "port %u Tx queue %u CQ creation failure", dev->data->port_id, idx); goto error; } + txq_data->cqe_n = log2above(cqe_n); + txq_data->cqe_s = 1 << txq_data->cqe_n; /* Initial fill CQ buffer with invalid CQE opcode. */ cqe = (struct mlx5_cqe *)txq_obj->cq_buf; for (i = 0; i < txq_data->cqe_s; i++) { cqe->op_own = (MLX5_CQE_INVALID << 4) | MLX5_CQE_OWNER_MASK; ++cqe; } - /* Create the Work Queue. */ - nqe = RTE_MIN(1UL << txq_data->elts_n, - (uint32_t)sh->device_attr.max_qp_wr); + return cq_obj; +error: + ret = rte_errno; + txq_release_devx_cq_resources(txq_obj); + rte_errno = ret; + return NULL; +} + +/** + * Create a SQ object using DevX. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Tx queue array. + * @param type + * Type of the Tx queue object to create. + * + * @return + * The DevX object initialized, NULL otherwise and rte_errno is set. + */ +static struct mlx5_devx_obj * +mlx5_devx_sq_new(struct rte_eth_dev *dev, uint16_t idx, + struct mlx5_txq_obj *txq_obj) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_devx_create_sq_attr sq_attr = { 0 }; + struct mlx5_devx_obj *sq_obj = NULL; + size_t page_size; + uint32_t wqe_n; + int ret; + + MLX5_ASSERT(txq_data); + MLX5_ASSERT(txq_obj); + page_size = rte_mem_page_size(); + if (page_size == (size_t)-1) { + DRV_LOG(ERR, "Failed to get mem page size"); + rte_errno = ENOMEM; + return NULL; + } + wqe_n = RTE_MIN(1UL << txq_data->elts_n, + (uint32_t)priv->sh->device_attr.max_qp_wr); txq_obj->sq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - nqe * sizeof(struct mlx5_wqe), - page_size, sh->numa_node); + wqe_n * sizeof(struct mlx5_wqe), + page_size, priv->sh->numa_node); if (!txq_obj->sq_buf) { DRV_LOG(ERR, "port %u Tx queue %u cannot allocate memory (SQ)", @@ -1028,20 +1063,11 @@ rte_errno = ENOMEM; goto error; } - txq_data->wqe_n = log2above(nqe); - txq_data->wqe_s = 1 << txq_data->wqe_n; - txq_data->wqe_m = txq_data->wqe_s - 1; - txq_data->wqes = (struct mlx5_wqe *)txq_obj->sq_buf; - txq_data->wqes_end = txq_data->wqes + txq_data->wqe_s; - txq_data->wqe_ci = 0; - txq_data->wqe_pi = 0; - txq_data->wqe_comp = 0; - txq_data->wqe_thres = txq_data->wqe_s / MLX5_TX_COMP_THRESH_INLINE_DIV; /* Register allocated buffer in user space with DevX. */ txq_obj->sq_umem = mlx5_glue->devx_umem_reg - (sh->ctx, + (priv->sh->ctx, (void *)txq_obj->sq_buf, - nqe * sizeof(struct mlx5_wqe), + wqe_n * sizeof(struct mlx5_wqe), IBV_ACCESS_LOCAL_WRITE); if (!txq_obj->sq_umem) { rte_errno = errno; @@ -1051,30 +1077,28 @@ goto error; } /* Allocate doorbell record for send queue. */ - txq_obj->sq_dbrec_offset = mlx5_get_dbr(sh->ctx, + txq_obj->sq_dbrec_offset = mlx5_get_dbr(priv->sh->ctx, &priv->dbrpgs, &txq_obj->sq_dbrec_page); - if (txq_obj->sq_dbrec_offset < 0) + if (txq_obj->sq_dbrec_offset < 0) { + rte_errno = errno; + DRV_LOG(ERR, "Failed to allocate SQ door-bell."); goto error; - txq_data->qp_db = (volatile uint32_t *) - (txq_obj->sq_dbrec_page->dbrs + - txq_obj->sq_dbrec_offset + - MLX5_SND_DBR * sizeof(uint32_t)); - *txq_data->qp_db = 0; - /* Create Send Queue object with DevX. */ + } sq_attr.tis_lst_sz = 1; - sq_attr.tis_num = sh->tis->id; + sq_attr.tis_num = priv->sh->tis->id; sq_attr.state = MLX5_SQC_STATE_RST; sq_attr.cqn = txq_obj->cq_devx->id; sq_attr.flush_in_error_en = 1; sq_attr.allow_multi_pkt_send_wqe = !!priv->config.mps; sq_attr.allow_swp = !!priv->config.swp; sq_attr.min_wqe_inline_mode = priv->config.hca_attr.vport_inline_mode; - sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar); + sq_attr.wq_attr.uar_page = + mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar); sq_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC; - sq_attr.wq_attr.pd = sh->pdn; + sq_attr.wq_attr.pd = priv->sh->pdn; sq_attr.wq_attr.log_wq_stride = rte_log2_u32(MLX5_WQE_SIZE); - sq_attr.wq_attr.log_wq_sz = txq_data->wqe_n; + sq_attr.wq_attr.log_wq_sz = log2above(wqe_n); sq_attr.wq_attr.dbr_umem_valid = 1; sq_attr.wq_attr.dbr_addr = txq_obj->sq_dbrec_offset; sq_attr.wq_attr.dbr_umem_id = @@ -1082,13 +1106,106 @@ sq_attr.wq_attr.wq_umem_valid = 1; sq_attr.wq_attr.wq_umem_id = mlx5_os_get_umem_id(txq_obj->sq_umem); sq_attr.wq_attr.wq_umem_offset = (uintptr_t)txq_obj->sq_buf % page_size; - txq_obj->sq_devx = mlx5_devx_cmd_create_sq(sh->ctx, &sq_attr); - if (!txq_obj->sq_devx) { + /* Create Send Queue object with DevX. */ + sq_obj = mlx5_devx_cmd_create_sq(priv->sh->ctx, &sq_attr); + if (!sq_obj) { rte_errno = errno; DRV_LOG(ERR, "port %u Tx queue %u SQ creation failure", dev->data->port_id, idx); goto error; } + txq_data->wqe_n = log2above(wqe_n); + return sq_obj; +error: + ret = rte_errno; + txq_release_devx_sq_resources(txq_obj); + rte_errno = ret; + return NULL; +} +#endif + +/** + * Create the Tx queue DevX object. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Tx queue array. + * + * @return + * The DevX object initialised, NULL otherwise and rte_errno is set. + */ +static struct mlx5_txq_obj * +mlx5_txq_obj_devx_new(struct rte_eth_dev *dev, uint16_t idx) +{ +#ifndef HAVE_MLX5DV_DEVX_UAR_OFFSET + DRV_LOG(ERR, "port %u Tx queue %u cannot create with DevX, no UAR", + dev->data->port_id, idx); + rte_errno = ENOMEM; + return NULL; +#else + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_dev_ctx_shared *sh = priv->sh; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq_data, struct mlx5_txq_ctrl, txq); + struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; + struct mlx5_txq_obj *txq_obj = NULL; + void *reg_addr; + uint32_t cqe_n; + int ret = 0; + + MLX5_ASSERT(txq_data); + MLX5_ASSERT(!txq_ctrl->obj); + txq_obj = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, + sizeof(struct mlx5_txq_obj), 0, + txq_ctrl->socket); + if (!txq_obj) { + DRV_LOG(ERR, + "port %u Tx queue %u cannot allocate memory resources", + dev->data->port_id, txq_data->idx); + rte_errno = ENOMEM; + return NULL; + } + txq_obj->type = MLX5_TXQ_OBJ_TYPE_DEVX_SQ; + txq_obj->txq_ctrl = txq_ctrl; + txq_obj->dev = dev; + /* Create the Completion Queue. */ + cqe_n = (1UL << txq_data->elts_n) / MLX5_TX_COMP_THRESH + + 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; + /* Create completion queue object with DevX. */ + txq_obj->cq_devx = mlx5_devx_cq_new(dev, cqe_n, idx, txq_obj); + if (!txq_obj->cq_devx) { + rte_errno = errno; + goto error; + } + txq_data->cqe_m = txq_data->cqe_s - 1; + txq_data->cqes = (volatile struct mlx5_cqe *)txq_obj->cq_buf; + txq_data->cq_ci = 0; + txq_data->cq_pi = 0; + txq_data->cq_db = (volatile uint32_t *)(txq_obj->cq_dbrec_page->dbrs + + txq_obj->cq_dbrec_offset); + *txq_data->cq_db = 0; + /* Create Send Queue object with DevX. */ + txq_obj->sq_devx = mlx5_devx_sq_new(dev, idx, txq_obj); + if (!txq_obj->sq_devx) { + rte_errno = errno; + goto error; + } + /* Create the Work Queue. */ + txq_data->wqe_s = 1 << txq_data->wqe_n; + txq_data->wqe_m = txq_data->wqe_s - 1; + txq_data->wqes = (struct mlx5_wqe *)txq_obj->sq_buf; + txq_data->wqes_end = txq_data->wqes + txq_data->wqe_s; + txq_data->wqe_ci = 0; + txq_data->wqe_pi = 0; + txq_data->wqe_comp = 0; + txq_data->wqe_thres = txq_data->wqe_s / MLX5_TX_COMP_THRESH_INLINE_DIV; + txq_data->qp_db = (volatile uint32_t *) + (txq_obj->sq_dbrec_page->dbrs + + txq_obj->sq_dbrec_offset + + MLX5_SND_DBR * sizeof(uint32_t)); + *txq_data->qp_db = 0; txq_data->qp_num_8s = txq_obj->sq_devx->id << 8; /* Change Send Queue state to Ready-to-Send. */ msq_attr.sq_state = MLX5_SQC_STATE_RST; @@ -1131,7 +1248,7 @@ return txq_obj; error: ret = rte_errno; /* Save rte_errno before cleanup. */ - txq_release_sq_resources(txq_obj); + txq_release_devx_resources(txq_obj); if (txq_data->fcqs) { mlx5_free(txq_data->fcqs); txq_data->fcqs = NULL; @@ -1408,7 +1525,7 @@ struct mlx5_txq_obj * if (txq_obj->tis) claim_zero(mlx5_devx_cmd_destroy(txq_obj->tis)); } else if (txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ) { - txq_release_sq_resources(txq_obj); + txq_release_devx_resources(txq_obj); } else { claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); From patchwork Thu Oct 1 14:09:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79474 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id EAF4DA04BA; Thu, 1 Oct 2020 16:12:14 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E70591DB8E; Thu, 1 Oct 2020 16:10:53 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id AB4A81DB69 for ; Thu, 1 Oct 2020 16:10:52 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:10:47 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAEL012743; Thu, 1 Oct 2020 17:10:47 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:16 +0000 Message-Id: <1601561366-1821-6-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 05/15] net/mlx5: reorder Tx queue Verbs object creation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Move the creation of the completion queue from the mlx5_txq_obj_new function into an auxiliary function. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5_txq.c | 117 +++++++++++++++++++++++++------------------- 1 file changed, 68 insertions(+), 49 deletions(-) diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 686b452..f2ecfc4 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -1260,6 +1260,66 @@ } /** + * Create a QP Verbs object. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Tx queue array. + * @param rxq_obj + * Pointer to Tx queue object data. + * + * @return + * The QP Verbs object initialized, NULL otherwise and rte_errno is set. + */ +static struct ibv_qp * +mlx5_ibv_qp_new(struct rte_eth_dev *dev, uint16_t idx, + struct mlx5_txq_obj *txq_obj) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq_data, struct mlx5_txq_ctrl, txq); + struct ibv_qp *qp_obj = NULL; + struct ibv_qp_init_attr_ex qp_attr = { 0 }; + const int desc = 1 << txq_data->elts_n; + + MLX5_ASSERT(!txq_ctrl->obj); + /* CQ to be associated with the send queue. */ + qp_attr.send_cq = txq_obj->cq; + /* CQ to be associated with the receive queue. */ + qp_attr.recv_cq = txq_obj->cq; + /* Max number of outstanding WRs. */ + qp_attr.cap.max_send_wr = ((priv->sh->device_attr.max_qp_wr < desc) ? + priv->sh->device_attr.max_qp_wr : desc); + /* + * Max number of scatter/gather elements in a WR, must be 1 to prevent + * libmlx5 from trying to affect must be 1 to prevent libmlx5 from + * trying to affect too much memory. TX gather is not impacted by the + * device_attr.max_sge limit and will still work properly. + */ + qp_attr.cap.max_send_sge = 1; + qp_attr.qp_type = IBV_QPT_RAW_PACKET, + /* Do *NOT* enable this, completions events are managed per Tx burst. */ + qp_attr.sq_sig_all = 0; + qp_attr.pd = priv->sh->pd; + qp_attr.comp_mask = IBV_QP_INIT_ATTR_PD; + if (txq_data->inlen_send) + qp_attr.cap.max_inline_data = txq_ctrl->max_inline_data; + if (txq_data->tso_en) { + qp_attr.max_tso_header = txq_ctrl->max_tso_header; + qp_attr.comp_mask |= IBV_QP_INIT_ATTR_MAX_TSO_HEADER; + } + qp_obj = mlx5_glue->create_qp_ex(priv->sh->ctx, &qp_attr); + if (qp_obj == NULL) { + DRV_LOG(ERR, "port %u Tx queue %u QP creation failure", + dev->data->port_id, idx); + rte_errno = errno; + } + return qp_obj; +} + +/** * Create the Tx queue Verbs object. * * @param dev @@ -1282,10 +1342,7 @@ struct mlx5_txq_obj * container_of(txq_data, struct mlx5_txq_ctrl, txq); struct mlx5_txq_obj tmpl; struct mlx5_txq_obj *txq_obj = NULL; - union { - struct ibv_qp_init_attr_ex init; - struct ibv_qp_attr mod; - } attr; + struct ibv_qp_attr mod; unsigned int cqe_n; struct mlx5dv_qp qp = { .comp_mask = MLX5DV_QP_MASK_UAR_MMAP_OFFSET }; struct mlx5dv_cq cq_info; @@ -1322,56 +1379,18 @@ struct mlx5_txq_obj * rte_errno = errno; goto error; } - attr.init = (struct ibv_qp_init_attr_ex){ - /* CQ to be associated with the send queue. */ - .send_cq = tmpl.cq, - /* CQ to be associated with the receive queue. */ - .recv_cq = tmpl.cq, - .cap = { - /* Max number of outstanding WRs. */ - .max_send_wr = - ((priv->sh->device_attr.max_qp_wr < - desc) ? - priv->sh->device_attr.max_qp_wr : - desc), - /* - * Max number of scatter/gather elements in a WR, - * must be 1 to prevent libmlx5 from trying to affect - * too much memory. TX gather is not impacted by the - * device_attr.max_sge limit and will still work - * properly. - */ - .max_send_sge = 1, - }, - .qp_type = IBV_QPT_RAW_PACKET, - /* - * Do *NOT* enable this, completions events are managed per - * Tx burst. - */ - .sq_sig_all = 0, - .pd = priv->sh->pd, - .comp_mask = IBV_QP_INIT_ATTR_PD, - }; - if (txq_data->inlen_send) - attr.init.cap.max_inline_data = txq_ctrl->max_inline_data; - if (txq_data->tso_en) { - attr.init.max_tso_header = txq_ctrl->max_tso_header; - attr.init.comp_mask |= IBV_QP_INIT_ATTR_MAX_TSO_HEADER; - } - tmpl.qp = mlx5_glue->create_qp_ex(priv->sh->ctx, &attr.init); + tmpl.qp = mlx5_ibv_qp_new(dev, idx, &tmpl); if (tmpl.qp == NULL) { - DRV_LOG(ERR, "port %u Tx queue %u QP creation failure", - dev->data->port_id, idx); rte_errno = errno; goto error; } - attr.mod = (struct ibv_qp_attr){ + mod = (struct ibv_qp_attr){ /* Move the QP to this state. */ .qp_state = IBV_QPS_INIT, /* IB device port number. */ .port_num = (uint8_t)priv->dev_port, }; - ret = mlx5_glue->modify_qp(tmpl.qp, &attr.mod, + ret = mlx5_glue->modify_qp(tmpl.qp, &mod, (IBV_QP_STATE | IBV_QP_PORT)); if (ret) { DRV_LOG(ERR, @@ -1380,10 +1399,10 @@ struct mlx5_txq_obj * rte_errno = errno; goto error; } - attr.mod = (struct ibv_qp_attr){ + mod = (struct ibv_qp_attr){ .qp_state = IBV_QPS_RTR }; - ret = mlx5_glue->modify_qp(tmpl.qp, &attr.mod, IBV_QP_STATE); + ret = mlx5_glue->modify_qp(tmpl.qp, &mod, IBV_QP_STATE); if (ret) { DRV_LOG(ERR, "port %u Tx queue %u QP state to IBV_QPS_RTR failed", @@ -1391,8 +1410,8 @@ struct mlx5_txq_obj * rte_errno = errno; goto error; } - attr.mod.qp_state = IBV_QPS_RTS; - ret = mlx5_glue->modify_qp(tmpl.qp, &attr.mod, IBV_QP_STATE); + mod.qp_state = IBV_QPS_RTS; + ret = mlx5_glue->modify_qp(tmpl.qp, &mod, IBV_QP_STATE); if (ret) { DRV_LOG(ERR, "port %u Tx queue %u QP state to IBV_QPS_RTS failed", From patchwork Thu Oct 1 14:09:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79475 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id E9EE1A04BA; Thu, 1 Oct 2020 16:12:36 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 060B91DBA0; Thu, 1 Oct 2020 16:10:59 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id B4C6F1DBA0 for ; Thu, 1 Oct 2020 16:10:56 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:10:53 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAEM012743; Thu, 1 Oct 2020 17:10:53 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:17 +0000 Message-Id: <1601561366-1821-7-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 06/15] net/mlx5: reposition the event queue number field X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The eqn field has become a field of sh directly since it is also relevant for Tx and Rx. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5.c | 10 ++++++++++ drivers/net/mlx5/mlx5.h | 2 +- drivers/net/mlx5/mlx5_devx.c | 9 +-------- drivers/net/mlx5/mlx5_txpp.c | 28 ++++++++-------------------- drivers/net/mlx5/mlx5_txq.c | 2 +- 5 files changed, 21 insertions(+), 30 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 01ead6e..e5ca392 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -925,6 +925,16 @@ struct mlx5_dev_ctx_shared * goto error; } if (sh->devx) { + uint32_t lcore = (uint32_t)rte_lcore_to_cpu_id(-1); + + /* Query the EQN for this core. */ + err = mlx5_glue->devx_query_eqn(sh->ctx, lcore, &sh->eqn); + if (err) { + rte_errno = errno; + DRV_LOG(ERR, "Failed to query event queue number %d.", + rte_errno); + goto error; + } err = mlx5_os_get_pdn(sh->pd, &sh->pdn); if (err) { DRV_LOG(ERR, "Fail to extract pdn from PD"); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index bd91e16..050d3a9 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -561,7 +561,6 @@ struct mlx5_dev_txpp { uint32_t tick; /* Completion tick duration in nanoseconds. */ uint32_t test; /* Packet pacing test mode. */ int32_t skew; /* Scheduling skew. */ - uint32_t eqn; /* Event Queue number. */ struct rte_intr_handle intr_handle; /* Periodic interrupt. */ void *echan; /* Event Channel. */ struct mlx5_txpp_wq clock_queue; /* Clock Queue. */ @@ -603,6 +602,7 @@ struct mlx5_dev_ctx_shared { LIST_ENTRY(mlx5_dev_ctx_shared) next; uint32_t refcnt; uint32_t devx:1; /* Opened with DV. */ + uint32_t eqn; /* Event Queue number. */ uint32_t max_port; /* Maximal IB device port index. */ void *ctx; /* Verbs/DV/DevX context. */ void *pd; /* Protection Domain. */ diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index cb4a522..cddfe43 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -350,11 +350,9 @@ struct mlx5_rxq_ctrl *rxq_ctrl = container_of(rxq_data, struct mlx5_rxq_ctrl, rxq); size_t page_size = rte_mem_page_size(); - uint32_t lcore = (uint32_t)rte_lcore_to_cpu_id(-1); unsigned int cqe_n = mlx5_rxq_cqe_num(rxq_data); struct mlx5_devx_dbr_page *dbr_page; int64_t dbr_offset; - uint32_t eqn = 0; void *buf = NULL; uint16_t event_nums[1] = {0}; uint32_t log_cqe_n; @@ -392,12 +390,6 @@ cq_attr.cqe_size = MLX5_CQE_SIZE_128B; log_cqe_n = log2above(cqe_n); cq_size = sizeof(struct mlx5_cqe) * (1 << log_cqe_n); - /* Query the EQN for this core. */ - if (mlx5_glue->devx_query_eqn(priv->sh->ctx, lcore, &eqn)) { - DRV_LOG(ERR, "Failed to query EQN for CQ."); - goto error; - } - cq_attr.eqn = eqn; buf = rte_calloc_socket(__func__, 1, cq_size, page_size, rxq_ctrl->socket); if (!buf) { @@ -425,6 +417,7 @@ rxq_data->cq_uar = mlx5_os_get_devx_uar_base_addr(priv->sh->devx_rx_uar); /* Create CQ using DevX API. */ + cq_attr.eqn = priv->sh->eqn; cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->sh->devx_rx_uar); cq_attr.q_umem_id = mlx5_os_get_umem_id(rxq_ctrl->cq_umem); diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c index 011e479..37355fa 100644 --- a/drivers/net/mlx5/mlx5_txpp.c +++ b/drivers/net/mlx5/mlx5_txpp.c @@ -31,36 +31,24 @@ /* Destroy Event Queue Notification Channel. */ static void -mlx5_txpp_destroy_eqn(struct mlx5_dev_ctx_shared *sh) +mlx5_txpp_destroy_event_channel(struct mlx5_dev_ctx_shared *sh) { if (sh->txpp.echan) { mlx5_glue->devx_destroy_event_channel(sh->txpp.echan); sh->txpp.echan = NULL; } - sh->txpp.eqn = 0; } /* Create Event Queue Notification Channel. */ static int -mlx5_txpp_create_eqn(struct mlx5_dev_ctx_shared *sh) +mlx5_txpp_create_event_channel(struct mlx5_dev_ctx_shared *sh) { - uint32_t lcore; - MLX5_ASSERT(!sh->txpp.echan); - lcore = (uint32_t)rte_lcore_to_cpu_id(-1); - if (mlx5_glue->devx_query_eqn(sh->ctx, lcore, &sh->txpp.eqn)) { - rte_errno = errno; - DRV_LOG(ERR, "Failed to query EQ number %d.", rte_errno); - sh->txpp.eqn = 0; - return -rte_errno; - } sh->txpp.echan = mlx5_glue->devx_create_event_channel(sh->ctx, MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA); if (!sh->txpp.echan) { - sh->txpp.eqn = 0; rte_errno = errno; - DRV_LOG(ERR, "Failed to create event channel %d.", - rte_errno); + DRV_LOG(ERR, "Failed to create event channel %d.", rte_errno); return -rte_errno; } return 0; @@ -285,7 +273,7 @@ cq_attr.cqe_size = (sizeof(struct mlx5_cqe) == 128) ? MLX5_CQE_SIZE_128B : MLX5_CQE_SIZE_64B; cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(sh->tx_uar); - cq_attr.eqn = sh->txpp.eqn; + cq_attr.eqn = sh->eqn; cq_attr.q_umem_valid = 1; cq_attr.q_umem_offset = 0; cq_attr.q_umem_id = mlx5_os_get_umem_id(wq->cq_umem); @@ -525,7 +513,7 @@ cq_attr.use_first_only = 1; cq_attr.overrun_ignore = 1; cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(sh->tx_uar); - cq_attr.eqn = sh->txpp.eqn; + cq_attr.eqn = sh->eqn; cq_attr.q_umem_valid = 1; cq_attr.q_umem_offset = 0; cq_attr.q_umem_id = mlx5_os_get_umem_id(wq->cq_umem); @@ -951,7 +939,7 @@ sh->txpp.test = !!(tx_pp < 0); sh->txpp.skew = priv->config.tx_skew; sh->txpp.freq = priv->config.hca_attr.dev_freq_khz; - ret = mlx5_txpp_create_eqn(sh); + ret = mlx5_txpp_create_event_channel(sh); if (ret) goto exit; ret = mlx5_txpp_alloc_pp_index(sh); @@ -972,7 +960,7 @@ mlx5_txpp_destroy_rearm_queue(sh); mlx5_txpp_destroy_clock_queue(sh); mlx5_txpp_free_pp_index(sh); - mlx5_txpp_destroy_eqn(sh); + mlx5_txpp_destroy_event_channel(sh); sh->txpp.tick = 0; sh->txpp.test = 0; sh->txpp.skew = 0; @@ -994,7 +982,7 @@ mlx5_txpp_destroy_rearm_queue(sh); mlx5_txpp_destroy_clock_queue(sh); mlx5_txpp_free_pp_index(sh); - mlx5_txpp_destroy_eqn(sh); + mlx5_txpp_destroy_event_channel(sh); sh->txpp.tick = 0; sh->txpp.test = 0; sh->txpp.skew = 0; diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index f2ecfc4..c678971 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -985,7 +985,7 @@ cq_attr.cqe_size = (sizeof(struct mlx5_cqe) == 128) ? MLX5_CQE_SIZE_128B : MLX5_CQE_SIZE_64B; cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar); - cq_attr.eqn = priv->sh->txpp.eqn; + cq_attr.eqn = priv->sh->eqn; cq_attr.q_umem_valid = 1; cq_attr.q_umem_offset = (uintptr_t)txq_obj->cq_buf % page_size; cq_attr.q_umem_id = mlx5_os_get_umem_id(txq_obj->cq_umem); From patchwork Thu Oct 1 14:09:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79476 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id EB1DAA04BA; Thu, 1 Oct 2020 16:13:03 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 143251DB71; Thu, 1 Oct 2020 16:11:10 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id C979E1DB70 for ; Thu, 1 Oct 2020 16:11:07 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:11:01 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAEN012743; Thu, 1 Oct 2020 17:11:01 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:18 +0000 Message-Id: <1601561366-1821-8-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 07/15] net/mlx5: separate Tx queue object creations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" As an arrangement to Windows OS support, the Verbs operations should be separated to another file. By this way, the build can easily cut the unsupported Verbs APIs from the compilation process. Define operation structure and DevX module in addition to the existing linux Verbs module. Separate Tx object creation into the Verbs/DevX modules and update the operation structure according to the OS support and the user configuration. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/linux/mlx5_os.c | 73 ++++ drivers/net/mlx5/linux/mlx5_verbs.c | 285 +++++++++++++ drivers/net/mlx5/linux/mlx5_verbs.h | 4 + drivers/net/mlx5/mlx5.h | 42 ++ drivers/net/mlx5/mlx5_devx.c | 531 +++++++++++++++++++++++- drivers/net/mlx5/mlx5_devx.h | 4 + drivers/net/mlx5/mlx5_rxtx.h | 43 +- drivers/net/mlx5/mlx5_trigger.c | 11 +- drivers/net/mlx5/mlx5_txq.c | 798 +----------------------------------- 9 files changed, 942 insertions(+), 849 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 188a6d4..c5332a0 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -512,6 +512,70 @@ } /** + * Create the Tx queue DevX/Verbs object. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Tx queue array. + * + * @return + * The DevX/Verbs object initialized, NULL otherwise and rte_errno is set. + */ +static struct mlx5_txq_obj * +mlx5_os_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_dev_config *config = &priv->config; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq_data, struct mlx5_txq_ctrl, txq); + + /* + * When DevX is supported and DV flow is enable, and dest tir is enable, + * hairpin functions use DevX API. + * When, in addition, DV E-Switch is enable and DevX uar offset is + * supported, all Tx functions also use DevX API. + * Otherwise, all Tx functions use Verbs API. + */ + if (config->devx && config->dv_flow_en && config->dest_tir) { + if (txq_ctrl->type == MLX5_TXQ_TYPE_HAIRPIN) + return mlx5_txq_devx_obj_new(dev, idx); +#ifdef HAVE_MLX5DV_DEVX_UAR_OFFSET + if (config->dv_esw_en) + return mlx5_txq_devx_obj_new(dev, idx); +#endif + } + return mlx5_txq_ibv_obj_new(dev, idx); +} + +/** + * Release an Tx DevX/verbs queue object. + * + * @param txq_obj + * DevX/Verbs Tx queue object. + */ +static void +mlx5_os_txq_obj_release(struct mlx5_txq_obj *txq_obj) +{ + struct mlx5_dev_config *config = &txq_obj->txq_ctrl->priv->config; + + if (config->devx && config->dv_flow_en && config->dest_tir) { +#ifdef HAVE_MLX5DV_DEVX_UAR_OFFSET + if (config->dv_esw_en) { + mlx5_txq_devx_obj_release(txq_obj); + return; + } +#endif + if (txq_obj->txq_ctrl->type == MLX5_TXQ_TYPE_HAIRPIN) { + mlx5_txq_devx_obj_release(txq_obj); + return; + } + } + mlx5_txq_ibv_obj_release(txq_obj); +} + +/** * Spawn an Ethernet device from Verbs information. * * @param dpdk_dev @@ -1299,6 +1363,12 @@ goto error; } } + /* + * Initialize the dev_ops structure with DevX/Verbs function pointers. + * When DevX is supported and both DV flow and dest tir are enabled, all + * Rx functions use DevX API (except for drop that has not yet been + * implemented in DevX). + */ if (config->devx && config->dv_flow_en && config->dest_tir) { priv->obj_ops = devx_obj_ops; priv->obj_ops.drop_action_create = @@ -1308,6 +1378,9 @@ } else { priv->obj_ops = ibv_obj_ops; } + /* The Tx objects are managed by a specific linux wrapper functions. */ + priv->obj_ops.txq_obj_new = mlx5_os_txq_obj_new; + priv->obj_ops.txq_obj_release = mlx5_os_txq_obj_release; /* Supported Verbs flow priority number detection. */ err = mlx5_flow_discover_priorities(eth_dev); if (err < 0) { diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index 20f659e..c79c4a2 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -782,6 +782,289 @@ mlx5_rxq_ibv_obj_drop_release(dev); } +/** + * Create a QP Verbs object. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Tx queue array. + * @param rxq_obj + * Pointer to Tx queue object data. + * + * @return + * The QP Verbs object initialized, NULL otherwise and rte_errno is set. + */ +static struct ibv_qp * +mlx5_ibv_qp_new(struct rte_eth_dev *dev, uint16_t idx, + struct mlx5_txq_obj *txq_obj) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq_data, struct mlx5_txq_ctrl, txq); + struct ibv_qp *qp_obj = NULL; + struct ibv_qp_init_attr_ex qp_attr = { 0 }; + const int desc = 1 << txq_data->elts_n; + + MLX5_ASSERT(!txq_ctrl->obj); + /* CQ to be associated with the send queue. */ + qp_attr.send_cq = txq_obj->cq; + /* CQ to be associated with the receive queue. */ + qp_attr.recv_cq = txq_obj->cq; + /* Max number of outstanding WRs. */ + qp_attr.cap.max_send_wr = ((priv->sh->device_attr.max_qp_wr < desc) ? + priv->sh->device_attr.max_qp_wr : desc); + /* + * Max number of scatter/gather elements in a WR, must be 1 to prevent + * libmlx5 from trying to affect must be 1 to prevent libmlx5 from + * trying to affect too much memory. TX gather is not impacted by the + * device_attr.max_sge limit and will still work properly. + */ + qp_attr.cap.max_send_sge = 1; + qp_attr.qp_type = IBV_QPT_RAW_PACKET, + /* Do *NOT* enable this, completions events are managed per Tx burst. */ + qp_attr.sq_sig_all = 0; + qp_attr.pd = priv->sh->pd; + qp_attr.comp_mask = IBV_QP_INIT_ATTR_PD; + if (txq_data->inlen_send) + qp_attr.cap.max_inline_data = txq_ctrl->max_inline_data; + if (txq_data->tso_en) { + qp_attr.max_tso_header = txq_ctrl->max_tso_header; + qp_attr.comp_mask |= IBV_QP_INIT_ATTR_MAX_TSO_HEADER; + } + qp_obj = mlx5_glue->create_qp_ex(priv->sh->ctx, &qp_attr); + if (qp_obj == NULL) { + DRV_LOG(ERR, "Port %u Tx queue %u QP creation failure.", + dev->data->port_id, idx); + rte_errno = errno; + } + return qp_obj; +} + +/** + * Create the Tx queue Verbs object. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Tx queue array. + * + * @return + * The Verbs object initialized, NULL otherwise and rte_errno is set. + */ +struct mlx5_txq_obj * +mlx5_txq_ibv_obj_new(struct rte_eth_dev *dev, uint16_t idx) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq_data, struct mlx5_txq_ctrl, txq); + struct mlx5_txq_obj tmpl; + struct mlx5_txq_obj *txq_obj = NULL; + struct ibv_qp_attr mod; + unsigned int cqe_n; + struct mlx5dv_qp qp; + struct mlx5dv_cq cq_info; + struct mlx5dv_obj obj; + const int desc = 1 << txq_data->elts_n; + int ret = 0; + + MLX5_ASSERT(txq_data); + priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_TX_QUEUE; + priv->verbs_alloc_ctx.obj = txq_ctrl; + if (mlx5_getenv_int("MLX5_ENABLE_CQE_COMPRESSION")) { + DRV_LOG(ERR, "Port %u MLX5_ENABLE_CQE_COMPRESSION " + "must never be set.", dev->data->port_id); + rte_errno = EINVAL; + return NULL; + } + memset(&tmpl, 0, sizeof(struct mlx5_txq_obj)); + cqe_n = desc / MLX5_TX_COMP_THRESH + + 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; + tmpl.cq = mlx5_glue->create_cq(priv->sh->ctx, cqe_n, NULL, NULL, 0); + if (tmpl.cq == NULL) { + DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.", + dev->data->port_id, idx); + rte_errno = errno; + goto error; + } + tmpl.qp = mlx5_ibv_qp_new(dev, idx, &tmpl); + if (tmpl.qp == NULL) { + rte_errno = errno; + goto error; + } + mod = (struct ibv_qp_attr){ + /* Move the QP to this state. */ + .qp_state = IBV_QPS_INIT, + /* IB device port number. */ + .port_num = (uint8_t)priv->dev_port, + }; + ret = mlx5_glue->modify_qp(tmpl.qp, &mod, (IBV_QP_STATE | IBV_QP_PORT)); + if (ret) { + DRV_LOG(ERR, + "Port %u Tx queue %u QP state to IBV_QPS_INIT failed.", + dev->data->port_id, idx); + rte_errno = errno; + goto error; + } + mod = (struct ibv_qp_attr){ + .qp_state = IBV_QPS_RTR + }; + ret = mlx5_glue->modify_qp(tmpl.qp, &mod, IBV_QP_STATE); + if (ret) { + DRV_LOG(ERR, + "Port %u Tx queue %u QP state to IBV_QPS_RTR failed.", + dev->data->port_id, idx); + rte_errno = errno; + goto error; + } + mod.qp_state = IBV_QPS_RTS; + ret = mlx5_glue->modify_qp(tmpl.qp, &mod, IBV_QP_STATE); + if (ret) { + DRV_LOG(ERR, + "Port %u Tx queue %u QP state to IBV_QPS_RTS failed.", + dev->data->port_id, idx); + rte_errno = errno; + goto error; + } + txq_obj = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, + sizeof(struct mlx5_txq_obj), 0, + txq_ctrl->socket); + if (!txq_obj) { + DRV_LOG(ERR, "Port %u Tx queue %u cannot allocate memory.", + dev->data->port_id, idx); + rte_errno = ENOMEM; + goto error; + } + qp.comp_mask = MLX5DV_QP_MASK_UAR_MMAP_OFFSET; +#ifdef HAVE_IBV_FLOW_DV_SUPPORT + /* If using DevX, need additional mask to read tisn value. */ + if (priv->sh->devx && !priv->sh->tdn) + qp.comp_mask |= MLX5DV_QP_MASK_RAW_QP_HANDLES; +#endif + obj.cq.in = tmpl.cq; + obj.cq.out = &cq_info; + obj.qp.in = tmpl.qp; + obj.qp.out = &qp; + ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_CQ | MLX5DV_OBJ_QP); + if (ret != 0) { + rte_errno = errno; + goto error; + } + if (cq_info.cqe_size != RTE_CACHE_LINE_SIZE) { + DRV_LOG(ERR, + "Port %u wrong MLX5_CQE_SIZE environment variable" + " value: it should be set to %u.", + dev->data->port_id, RTE_CACHE_LINE_SIZE); + rte_errno = EINVAL; + goto error; + } + txq_data->cqe_n = log2above(cq_info.cqe_cnt); + txq_data->cqe_s = 1 << txq_data->cqe_n; + txq_data->cqe_m = txq_data->cqe_s - 1; + txq_data->qp_num_8s = ((struct ibv_qp *)tmpl.qp)->qp_num << 8; + txq_data->wqes = qp.sq.buf; + txq_data->wqe_n = log2above(qp.sq.wqe_cnt); + txq_data->wqe_s = 1 << txq_data->wqe_n; + txq_data->wqe_m = txq_data->wqe_s - 1; + txq_data->wqes_end = txq_data->wqes + txq_data->wqe_s; + txq_data->qp_db = &qp.dbrec[MLX5_SND_DBR]; + txq_data->cq_db = cq_info.dbrec; + txq_data->cqes = (volatile struct mlx5_cqe *)cq_info.buf; + txq_data->cq_ci = 0; + txq_data->cq_pi = 0; + txq_data->wqe_ci = 0; + txq_data->wqe_pi = 0; + txq_data->wqe_comp = 0; + txq_data->wqe_thres = txq_data->wqe_s / MLX5_TX_COMP_THRESH_INLINE_DIV; + txq_data->fcqs = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, + txq_data->cqe_s * sizeof(*txq_data->fcqs), + RTE_CACHE_LINE_SIZE, txq_ctrl->socket); + if (!txq_data->fcqs) { + DRV_LOG(ERR, "Port %u Tx queue %u can't allocate memory (FCQ).", + dev->data->port_id, idx); + rte_errno = ENOMEM; + goto error; + } +#ifdef HAVE_IBV_FLOW_DV_SUPPORT + /* + * If using DevX need to query and store TIS transport domain value. + * This is done once per port. + * Will use this value on Rx, when creating matching TIR. + */ + if (priv->sh->devx && !priv->sh->tdn) { + ret = mlx5_devx_cmd_qp_query_tis_td(tmpl.qp, qp.tisn, + &priv->sh->tdn); + if (ret) { + DRV_LOG(ERR, "Fail to query port %u Tx queue %u QP TIS " + "transport domain.", dev->data->port_id, idx); + rte_errno = EINVAL; + goto error; + } else { + DRV_LOG(DEBUG, "Port %u Tx queue %u TIS number %d " + "transport domain %d.", dev->data->port_id, + idx, qp.tisn, priv->sh->tdn); + } + } +#endif + txq_obj->qp = tmpl.qp; + txq_obj->cq = tmpl.cq; + txq_ctrl->bf_reg = qp.bf.reg; + if (qp.comp_mask & MLX5DV_QP_MASK_UAR_MMAP_OFFSET) { + txq_ctrl->uar_mmap_offset = qp.uar_mmap_offset; + DRV_LOG(DEBUG, "Port %u: uar_mmap_offset 0x%" PRIx64 ".", + dev->data->port_id, txq_ctrl->uar_mmap_offset); + } else { + DRV_LOG(ERR, + "Port %u failed to retrieve UAR info, invalid" + " libmlx5.so", + dev->data->port_id); + rte_errno = EINVAL; + goto error; + } + txq_uar_init(txq_ctrl); + txq_obj->txq_ctrl = txq_ctrl; + LIST_INSERT_HEAD(&priv->txqsobj, txq_obj, next); + priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; + return txq_obj; +error: + ret = rte_errno; /* Save rte_errno before cleanup. */ + if (tmpl.cq) + claim_zero(mlx5_glue->destroy_cq(tmpl.cq)); + if (tmpl.qp) + claim_zero(mlx5_glue->destroy_qp(tmpl.qp)); + if (txq_data->fcqs) { + mlx5_free(txq_data->fcqs); + txq_data->fcqs = NULL; + } + if (txq_obj) + mlx5_free(txq_obj); + priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; + rte_errno = ret; /* Restore rte_errno. */ + return NULL; +} + +/** + * Release an Tx verbs queue object. + * + * @param txq_obj + * Verbs Tx queue object.. + */ +void +mlx5_txq_ibv_obj_release(struct mlx5_txq_obj *txq_obj) +{ + MLX5_ASSERT(txq_obj); + claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); + claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); + if (txq_obj->txq_ctrl->txq.fcqs) { + mlx5_free(txq_obj->txq_ctrl->txq.fcqs); + txq_obj->txq_ctrl->txq.fcqs = NULL; + } + LIST_REMOVE(txq_obj, next); + mlx5_free(txq_obj); +} + struct mlx5_obj_ops ibv_obj_ops = { .rxq_obj_modify_vlan_strip = mlx5_rxq_obj_modify_wq_vlan_strip, .rxq_obj_new = mlx5_rxq_ibv_obj_new, @@ -794,4 +1077,6 @@ struct mlx5_obj_ops ibv_obj_ops = { .hrxq_destroy = mlx5_ibv_qp_destroy, .drop_action_create = mlx5_ibv_drop_action_create, .drop_action_destroy = mlx5_ibv_drop_action_destroy, + .txq_obj_new = mlx5_txq_ibv_obj_new, + .txq_obj_release = mlx5_txq_ibv_obj_release, }; diff --git a/drivers/net/mlx5/linux/mlx5_verbs.h b/drivers/net/mlx5/linux/mlx5_verbs.h index 2e69c0f..7f6bb99 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.h +++ b/drivers/net/mlx5/linux/mlx5_verbs.h @@ -12,6 +12,10 @@ struct mlx5_verbs_ops { mlx5_dereg_mr_t dereg_mr; }; +struct mlx5_txq_obj *mlx5_txq_ibv_obj_new(struct rte_eth_dev *dev, + uint16_t idx); +void mlx5_txq_ibv_obj_release(struct mlx5_txq_obj *txq_obj); + /* Verbs ops struct */ extern const struct mlx5_verbs_ops mlx5_verbs_ops; extern struct mlx5_obj_ops ibv_obj_ops; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 050d3a9..8679750 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -735,6 +735,45 @@ struct mlx5_hrxq { uint8_t rss_key[]; /* Hash key. */ }; +enum mlx5_txq_obj_type { + MLX5_TXQ_OBJ_TYPE_IBV, /* mlx5_txq_obj with ibv_wq. */ + MLX5_TXQ_OBJ_TYPE_DEVX_SQ, /* mlx5_txq_obj with mlx5_devx_sq. */ + MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN, + /* mlx5_txq_obj with mlx5_devx_tq and hairpin support. */ +}; + +/* Verbs/DevX Tx queue elements. */ +struct mlx5_txq_obj { + LIST_ENTRY(mlx5_txq_obj) next; /* Pointer to the next element. */ + struct mlx5_txq_ctrl *txq_ctrl; /* Pointer to the control queue. */ + enum mlx5_txq_obj_type type; /* The txq object type. */ + RTE_STD_C11 + union { + struct { + void *cq; /* Completion Queue. */ + void *qp; /* Queue Pair. */ + }; + struct { + struct mlx5_devx_obj *sq; + /* DevX object for Sx queue. */ + struct mlx5_devx_obj *tis; /* The TIS object. */ + }; + struct { + struct rte_eth_dev *dev; + struct mlx5_devx_obj *cq_devx; + void *cq_umem; + void *cq_buf; + int64_t cq_dbrec_offset; + struct mlx5_devx_dbr_page *cq_dbrec_page; + struct mlx5_devx_obj *sq_devx; + void *sq_umem; + void *sq_buf; + int64_t sq_dbrec_offset; + struct mlx5_devx_dbr_page *sq_dbrec_page; + }; + }; +}; + /* HW objects operations structure. */ struct mlx5_obj_ops { int (*rxq_obj_modify_vlan_strip)(struct mlx5_rxq_obj *rxq_obj, int on); @@ -750,6 +789,9 @@ struct mlx5_obj_ops { void (*hrxq_destroy)(struct mlx5_hrxq *hrxq); int (*drop_action_create)(struct rte_eth_dev *dev); void (*drop_action_destroy)(struct rte_eth_dev *dev); + struct mlx5_txq_obj *(*txq_obj_new)(struct rte_eth_dev *dev, + uint16_t idx); + void (*txq_obj_release)(struct mlx5_txq_obj *txq_obj); }; struct mlx5_priv { diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index cddfe43..0b6e116 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -791,7 +791,7 @@ mlx5_devx_drop_action_create(struct rte_eth_dev *dev) { (void)dev; - DRV_LOG(ERR, "DevX drop action is not supported yet"); + DRV_LOG(ERR, "DevX drop action is not supported yet."); rte_errno = ENOTSUP; return -rte_errno; } @@ -806,10 +806,535 @@ mlx5_devx_drop_action_destroy(struct rte_eth_dev *dev) { (void)dev; - DRV_LOG(ERR, "DevX drop action is not supported yet"); + DRV_LOG(ERR, "DevX drop action is not supported yet."); rte_errno = ENOTSUP; } +/** + * Create the Tx hairpin queue object. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Tx queue array. + * + * @return + * The hairpin DevX object initialized, NULL otherwise and rte_errno is set. + */ +static struct mlx5_txq_obj * +mlx5_txq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq_data, struct mlx5_txq_ctrl, txq); + struct mlx5_devx_create_sq_attr attr = { 0 }; + struct mlx5_txq_obj *tmpl = NULL; + uint32_t max_wq_data; + + MLX5_ASSERT(txq_data); + MLX5_ASSERT(!txq_ctrl->obj); + tmpl = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, sizeof(*tmpl), 0, + txq_ctrl->socket); + if (!tmpl) { + DRV_LOG(ERR, + "Port %u Tx queue %u cannot allocate memory resources.", + dev->data->port_id, txq_data->idx); + rte_errno = ENOMEM; + return NULL; + } + tmpl->type = MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN; + tmpl->txq_ctrl = txq_ctrl; + attr.hairpin = 1; + attr.tis_lst_sz = 1; + max_wq_data = priv->config.hca_attr.log_max_hairpin_wq_data_sz; + /* Jumbo frames > 9KB should be supported, and more packets. */ + if (priv->config.log_hp_size != (uint32_t)MLX5_ARG_UNSET) { + if (priv->config.log_hp_size > max_wq_data) { + DRV_LOG(ERR, "Total data size %u power of 2 is " + "too large for hairpin.", + priv->config.log_hp_size); + mlx5_free(tmpl); + rte_errno = ERANGE; + return NULL; + } + attr.wq_attr.log_hairpin_data_sz = priv->config.log_hp_size; + } else { + attr.wq_attr.log_hairpin_data_sz = + (max_wq_data < MLX5_HAIRPIN_JUMBO_LOG_SIZE) ? + max_wq_data : MLX5_HAIRPIN_JUMBO_LOG_SIZE; + } + /* Set the packets number to the maximum value for performance. */ + attr.wq_attr.log_hairpin_num_packets = + attr.wq_attr.log_hairpin_data_sz - + MLX5_HAIRPIN_QUEUE_STRIDE; + attr.tis_num = priv->sh->tis->id; + tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->ctx, &attr); + if (!tmpl->sq) { + DRV_LOG(ERR, + "Port %u tx hairpin queue %u can't create SQ object.", + dev->data->port_id, idx); + mlx5_free(tmpl); + rte_errno = errno; + return NULL; + } + DRV_LOG(DEBUG, "Port %u sxq %u updated with %p.", dev->data->port_id, + idx, (void *)&tmpl); + LIST_INSERT_HEAD(&priv->txqsobj, tmpl, next); + return tmpl; +} + +#ifdef HAVE_MLX5DV_DEVX_UAR_OFFSET +/** + * Release DevX SQ resources. + * + * @param txq_obj + * DevX Tx queue object. + */ +static void +txq_release_devx_sq_resources(struct mlx5_txq_obj *txq_obj) +{ + if (txq_obj->sq_devx) + claim_zero(mlx5_devx_cmd_destroy(txq_obj->sq_devx)); + if (txq_obj->sq_umem) + claim_zero(mlx5_glue->devx_umem_dereg(txq_obj->sq_umem)); + if (txq_obj->sq_buf) + mlx5_free(txq_obj->sq_buf); + if (txq_obj->sq_dbrec_page) + claim_zero(mlx5_release_dbr(&txq_obj->txq_ctrl->priv->dbrpgs, + mlx5_os_get_umem_id + (txq_obj->sq_dbrec_page->umem), + txq_obj->sq_dbrec_offset)); +} + +/** + * Release DevX Tx CQ resources. + * + * @param txq_obj + * DevX Tx queue object. + */ +static void +txq_release_devx_cq_resources(struct mlx5_txq_obj *txq_obj) +{ + if (txq_obj->cq_devx) + claim_zero(mlx5_devx_cmd_destroy(txq_obj->cq_devx)); + if (txq_obj->cq_umem) + claim_zero(mlx5_glue->devx_umem_dereg(txq_obj->cq_umem)); + if (txq_obj->cq_buf) + mlx5_free(txq_obj->cq_buf); + if (txq_obj->cq_dbrec_page) + claim_zero(mlx5_release_dbr(&txq_obj->txq_ctrl->priv->dbrpgs, + mlx5_os_get_umem_id + (txq_obj->cq_dbrec_page->umem), + txq_obj->cq_dbrec_offset)); +} + +/** + * Destroy the Tx queue DevX object. + * + * @param txq_obj + * Txq object to destroy. + */ +static void +txq_release_devx_resources(struct mlx5_txq_obj *txq_obj) +{ + MLX5_ASSERT(txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ); + + txq_release_devx_cq_resources(txq_obj); + txq_release_devx_sq_resources(txq_obj); +} + +/** + * Create a DevX CQ object for an Tx queue. + * + * @param dev + * Pointer to Ethernet device. + * @param cqe_n + * Number of entries in the CQ. + * @param idx + * Queue index in DPDK Tx queue array. + * @param rxq_obj + * Pointer to Tx queue object data. + * + * @return + * The DevX CQ object initialized, NULL otherwise and rte_errno is set. + */ +static struct mlx5_devx_obj * +mlx5_tx_devx_cq_new(struct rte_eth_dev *dev, uint32_t cqe_n, uint16_t idx, + struct mlx5_txq_obj *txq_obj) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_devx_obj *cq_obj = NULL; + struct mlx5_devx_cq_attr cq_attr = { 0 }; + struct mlx5_cqe *cqe; + size_t page_size; + size_t alignment; + uint32_t i; + int ret; + + MLX5_ASSERT(txq_data); + MLX5_ASSERT(txq_obj); + page_size = rte_mem_page_size(); + if (page_size == (size_t)-1) { + DRV_LOG(ERR, "Failed to get mem page size."); + rte_errno = ENOMEM; + return NULL; + } + /* Allocate memory buffer for CQEs. */ + alignment = MLX5_CQE_BUF_ALIGNMENT; + if (alignment == (size_t)-1) { + DRV_LOG(ERR, "Failed to get CQE buf alignment."); + rte_errno = ENOMEM; + return NULL; + } + cqe_n = 1UL << log2above(cqe_n); + if (cqe_n > UINT16_MAX) { + DRV_LOG(ERR, + "Port %u Tx queue %u requests to many CQEs %u.", + dev->data->port_id, txq_data->idx, cqe_n); + rte_errno = EINVAL; + return NULL; + } + txq_obj->cq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, + cqe_n * sizeof(struct mlx5_cqe), + alignment, + priv->sh->numa_node); + if (!txq_obj->cq_buf) { + DRV_LOG(ERR, + "Port %u Tx queue %u cannot allocate memory (CQ).", + dev->data->port_id, txq_data->idx); + rte_errno = ENOMEM; + return NULL; + } + /* Register allocated buffer in user space with DevX. */ + txq_obj->cq_umem = mlx5_glue->devx_umem_reg(priv->sh->ctx, + (void *)txq_obj->cq_buf, + cqe_n * sizeof(struct mlx5_cqe), + IBV_ACCESS_LOCAL_WRITE); + if (!txq_obj->cq_umem) { + rte_errno = errno; + DRV_LOG(ERR, + "Port %u Tx queue %u cannot register memory (CQ).", + dev->data->port_id, txq_data->idx); + goto error; + } + /* Allocate doorbell record for completion queue. */ + txq_obj->cq_dbrec_offset = mlx5_get_dbr(priv->sh->ctx, + &priv->dbrpgs, + &txq_obj->cq_dbrec_page); + if (txq_obj->cq_dbrec_offset < 0) { + rte_errno = errno; + DRV_LOG(ERR, "Failed to allocate CQ door-bell."); + goto error; + } + cq_attr.cqe_size = (sizeof(struct mlx5_cqe) == 128) ? + MLX5_CQE_SIZE_128B : MLX5_CQE_SIZE_64B; + cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar); + cq_attr.eqn = priv->sh->eqn; + cq_attr.q_umem_valid = 1; + cq_attr.q_umem_offset = (uintptr_t)txq_obj->cq_buf % page_size; + cq_attr.q_umem_id = mlx5_os_get_umem_id(txq_obj->cq_umem); + cq_attr.db_umem_valid = 1; + cq_attr.db_umem_offset = txq_obj->cq_dbrec_offset; + cq_attr.db_umem_id = mlx5_os_get_umem_id(txq_obj->cq_dbrec_page->umem); + cq_attr.log_cq_size = rte_log2_u32(cqe_n); + cq_attr.log_page_size = rte_log2_u32(page_size); + /* Create completion queue object with DevX. */ + cq_obj = mlx5_devx_cmd_create_cq(priv->sh->ctx, &cq_attr); + if (!cq_obj) { + rte_errno = errno; + DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.", + dev->data->port_id, idx); + goto error; + } + txq_data->cqe_n = log2above(cqe_n); + txq_data->cqe_s = 1 << txq_data->cqe_n; + /* Initial fill CQ buffer with invalid CQE opcode. */ + cqe = (struct mlx5_cqe *)txq_obj->cq_buf; + for (i = 0; i < txq_data->cqe_s; i++) { + cqe->op_own = (MLX5_CQE_INVALID << 4) | MLX5_CQE_OWNER_MASK; + ++cqe; + } + return cq_obj; +error: + ret = rte_errno; + txq_release_devx_cq_resources(txq_obj); + rte_errno = ret; + return NULL; +} + +/** + * Create a SQ object using DevX. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Tx queue array. + * @param rxq_obj + * Pointer to Tx queue object data. + * + * @return + * The DevX SQ object initialized, NULL otherwise and rte_errno is set. + */ +static struct mlx5_devx_obj * +mlx5_devx_sq_new(struct rte_eth_dev *dev, uint16_t idx, + struct mlx5_txq_obj *txq_obj) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_devx_create_sq_attr sq_attr = { 0 }; + struct mlx5_devx_obj *sq_obj = NULL; + size_t page_size; + uint32_t wqe_n; + int ret; + + MLX5_ASSERT(txq_data); + MLX5_ASSERT(txq_obj); + page_size = rte_mem_page_size(); + if (page_size == (size_t)-1) { + DRV_LOG(ERR, "Failed to get mem page size."); + rte_errno = ENOMEM; + return NULL; + } + wqe_n = RTE_MIN(1UL << txq_data->elts_n, + (uint32_t)priv->sh->device_attr.max_qp_wr); + txq_obj->sq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, + wqe_n * sizeof(struct mlx5_wqe), + page_size, priv->sh->numa_node); + if (!txq_obj->sq_buf) { + DRV_LOG(ERR, + "Port %u Tx queue %u cannot allocate memory (SQ).", + dev->data->port_id, txq_data->idx); + rte_errno = ENOMEM; + goto error; + } + /* Register allocated buffer in user space with DevX. */ + txq_obj->sq_umem = mlx5_glue->devx_umem_reg + (priv->sh->ctx, + (void *)txq_obj->sq_buf, + wqe_n * sizeof(struct mlx5_wqe), + IBV_ACCESS_LOCAL_WRITE); + if (!txq_obj->sq_umem) { + rte_errno = errno; + DRV_LOG(ERR, + "Port %u Tx queue %u cannot register memory (SQ).", + dev->data->port_id, txq_data->idx); + goto error; + } + /* Allocate doorbell record for send queue. */ + txq_obj->sq_dbrec_offset = mlx5_get_dbr(priv->sh->ctx, + &priv->dbrpgs, + &txq_obj->sq_dbrec_page); + if (txq_obj->sq_dbrec_offset < 0) { + rte_errno = errno; + DRV_LOG(ERR, "Failed to allocate SQ door-bell."); + goto error; + } + txq_data->wqe_n = log2above(wqe_n); + sq_attr.tis_lst_sz = 1; + sq_attr.tis_num = priv->sh->tis->id; + sq_attr.state = MLX5_SQC_STATE_RST; + sq_attr.cqn = txq_obj->cq_devx->id; + sq_attr.flush_in_error_en = 1; + sq_attr.allow_multi_pkt_send_wqe = !!priv->config.mps; + sq_attr.allow_swp = !!priv->config.swp; + sq_attr.min_wqe_inline_mode = priv->config.hca_attr.vport_inline_mode; + sq_attr.wq_attr.uar_page = + mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar); + sq_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC; + sq_attr.wq_attr.pd = priv->sh->pdn; + sq_attr.wq_attr.log_wq_stride = rte_log2_u32(MLX5_WQE_SIZE); + sq_attr.wq_attr.log_wq_sz = txq_data->wqe_n; + sq_attr.wq_attr.dbr_umem_valid = 1; + sq_attr.wq_attr.dbr_addr = txq_obj->sq_dbrec_offset; + sq_attr.wq_attr.dbr_umem_id = + mlx5_os_get_umem_id(txq_obj->sq_dbrec_page->umem); + sq_attr.wq_attr.wq_umem_valid = 1; + sq_attr.wq_attr.wq_umem_id = mlx5_os_get_umem_id(txq_obj->sq_umem); + sq_attr.wq_attr.wq_umem_offset = (uintptr_t)txq_obj->sq_buf % page_size; + /* Create Send Queue object with DevX. */ + sq_obj = mlx5_devx_cmd_create_sq(priv->sh->ctx, &sq_attr); + if (!sq_obj) { + rte_errno = errno; + DRV_LOG(ERR, "Port %u Tx queue %u SQ creation failure.", + dev->data->port_id, idx); + goto error; + } + return sq_obj; +error: + ret = rte_errno; + txq_release_devx_sq_resources(txq_obj); + rte_errno = ret; + return NULL; +} +#endif + +/** + * Create the Tx queue DevX object. + * + * @param dev + * Pointer to Ethernet device. + * @param idx + * Queue index in DPDK Tx queue array. + * + * @return + * The DevX object initialized, NULL otherwise and rte_errno is set. + */ +struct mlx5_txq_obj * +mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq_data, struct mlx5_txq_ctrl, txq); + + if (txq_ctrl->type == MLX5_TXQ_TYPE_HAIRPIN) + return mlx5_txq_obj_hairpin_new(dev, idx); +#ifndef HAVE_MLX5DV_DEVX_UAR_OFFSET + DRV_LOG(ERR, "Port %u Tx queue %u cannot create with DevX, no UAR.", + dev->data->port_id, idx); + rte_errno = ENOMEM; + return NULL; +#else + struct mlx5_dev_ctx_shared *sh = priv->sh; + struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; + struct mlx5_txq_obj *txq_obj = NULL; + void *reg_addr; + uint32_t cqe_n; + int ret = 0; + + MLX5_ASSERT(txq_data); + MLX5_ASSERT(!txq_ctrl->obj); + txq_obj = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, + sizeof(struct mlx5_txq_obj), 0, + txq_ctrl->socket); + if (!txq_obj) { + DRV_LOG(ERR, + "Port %u Tx queue %u cannot allocate memory resources.", + dev->data->port_id, txq_data->idx); + rte_errno = ENOMEM; + return NULL; + } + txq_obj->type = MLX5_TXQ_OBJ_TYPE_DEVX_SQ; + txq_obj->txq_ctrl = txq_ctrl; + txq_obj->dev = dev; + /* Create the Completion Queue. */ + cqe_n = (1UL << txq_data->elts_n) / MLX5_TX_COMP_THRESH + + 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; + /* Create completion queue object with DevX. */ + txq_obj->cq_devx = mlx5_tx_devx_cq_new(dev, cqe_n, idx, txq_obj); + if (!txq_obj->cq_devx) { + rte_errno = errno; + goto error; + } + txq_data->cqe_m = txq_data->cqe_s - 1; + txq_data->cqes = (volatile struct mlx5_cqe *)txq_obj->cq_buf; + txq_data->cq_ci = 0; + txq_data->cq_pi = 0; + txq_data->cq_db = (volatile uint32_t *)(txq_obj->cq_dbrec_page->dbrs + + txq_obj->cq_dbrec_offset); + *txq_data->cq_db = 0; + /* Create Send Queue object with DevX. */ + txq_obj->sq_devx = mlx5_devx_sq_new(dev, idx, txq_obj); + if (!txq_obj->sq_devx) { + rte_errno = errno; + goto error; + } + /* Create the Work Queue. */ + txq_data->wqe_s = 1 << txq_data->wqe_n; + txq_data->wqe_m = txq_data->wqe_s - 1; + txq_data->wqes = (struct mlx5_wqe *)txq_obj->sq_buf; + txq_data->wqes_end = txq_data->wqes + txq_data->wqe_s; + txq_data->wqe_ci = 0; + txq_data->wqe_pi = 0; + txq_data->wqe_comp = 0; + txq_data->wqe_thres = txq_data->wqe_s / MLX5_TX_COMP_THRESH_INLINE_DIV; + txq_data->qp_db = (volatile uint32_t *) + (txq_obj->sq_dbrec_page->dbrs + + txq_obj->sq_dbrec_offset + + MLX5_SND_DBR * sizeof(uint32_t)); + *txq_data->qp_db = 0; + txq_data->qp_num_8s = txq_obj->sq_devx->id << 8; + /* Change Send Queue state to Ready-to-Send. */ + msq_attr.sq_state = MLX5_SQC_STATE_RST; + msq_attr.state = MLX5_SQC_STATE_RDY; + ret = mlx5_devx_cmd_modify_sq(txq_obj->sq_devx, &msq_attr); + if (ret) { + rte_errno = errno; + DRV_LOG(ERR, + "Port %u Tx queue %u SP state to SQC_STATE_RDY failed.", + dev->data->port_id, idx); + goto error; + } + txq_data->fcqs = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, + txq_data->cqe_s * sizeof(*txq_data->fcqs), + RTE_CACHE_LINE_SIZE, + txq_ctrl->socket); + if (!txq_data->fcqs) { + DRV_LOG(ERR, + "Port %u Tx queue %u cannot allocate memory (FCQ).", + dev->data->port_id, idx); + rte_errno = ENOMEM; + goto error; + } +#ifdef HAVE_IBV_FLOW_DV_SUPPORT + /* + * If using DevX need to query and store TIS transport domain value. + * This is done once per port. + * Will use this value on Rx, when creating matching TIR. + */ + if (!priv->sh->tdn) + priv->sh->tdn = priv->sh->td->id; +#endif + MLX5_ASSERT(sh->tx_uar); + reg_addr = mlx5_os_get_devx_uar_reg_addr(sh->tx_uar); + MLX5_ASSERT(reg_addr); + txq_ctrl->bf_reg = reg_addr; + txq_ctrl->uar_mmap_offset = + mlx5_os_get_devx_uar_mmap_offset(sh->tx_uar); + txq_uar_init(txq_ctrl); + LIST_INSERT_HEAD(&priv->txqsobj, txq_obj, next); + return txq_obj; +error: + ret = rte_errno; /* Save rte_errno before cleanup. */ + txq_release_devx_resources(txq_obj); + if (txq_data->fcqs) { + mlx5_free(txq_data->fcqs); + txq_data->fcqs = NULL; + } + mlx5_free(txq_obj); + rte_errno = ret; /* Restore rte_errno. */ + return NULL; +#endif +} + +/** + * Release an Tx DevX queue object. + * + * @param txq_obj + * DevX Tx queue object. + */ +void +mlx5_txq_devx_obj_release(struct mlx5_txq_obj *txq_obj) +{ + MLX5_ASSERT(txq_obj); + if (txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN) { + if (txq_obj->tis) + claim_zero(mlx5_devx_cmd_destroy(txq_obj->tis)); +#ifdef HAVE_MLX5DV_DEVX_UAR_OFFSET + } else { + txq_release_devx_resources(txq_obj); +#endif + } + if (txq_obj->txq_ctrl->txq.fcqs) { + mlx5_free(txq_obj->txq_ctrl->txq.fcqs); + txq_obj->txq_ctrl->txq.fcqs = NULL; + } + LIST_REMOVE(txq_obj, next); + mlx5_free(txq_obj); +} + struct mlx5_obj_ops devx_obj_ops = { .rxq_obj_modify_vlan_strip = mlx5_rxq_obj_modify_rq_vlan_strip, .rxq_obj_new = mlx5_rxq_devx_obj_new, @@ -822,4 +1347,6 @@ struct mlx5_obj_ops devx_obj_ops = { .hrxq_destroy = mlx5_devx_tir_destroy, .drop_action_create = mlx5_devx_drop_action_create, .drop_action_destroy = mlx5_devx_drop_action_destroy, + .txq_obj_new = mlx5_txq_devx_obj_new, + .txq_obj_release = mlx5_txq_devx_obj_release, }; diff --git a/drivers/net/mlx5/mlx5_devx.h b/drivers/net/mlx5/mlx5_devx.h index 844985c..0bbbbc0 100644 --- a/drivers/net/mlx5/mlx5_devx.h +++ b/drivers/net/mlx5/mlx5_devx.h @@ -7,6 +7,10 @@ #include "mlx5.h" +struct mlx5_txq_obj *mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, + uint16_t idx); +void mlx5_txq_devx_obj_release(struct mlx5_txq_obj *txq_obj); + extern struct mlx5_obj_ops devx_obj_ops; #endif /* RTE_PMD_MLX5_DEVX_H_ */ diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index d947e0e..674296e 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -261,50 +261,11 @@ struct mlx5_txq_data { /* Storage for queued packets, must be the last field. */ } __rte_cache_aligned; -enum mlx5_txq_obj_type { - MLX5_TXQ_OBJ_TYPE_IBV, /* mlx5_txq_obj with ibv_wq. */ - MLX5_TXQ_OBJ_TYPE_DEVX_SQ, /* mlx5_txq_obj with mlx5_devx_sq. */ - MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN, - /* mlx5_txq_obj with mlx5_devx_tq and hairpin support. */ -}; - enum mlx5_txq_type { MLX5_TXQ_TYPE_STANDARD, /* Standard Tx queue. */ MLX5_TXQ_TYPE_HAIRPIN, /* Hairpin Rx queue. */ }; -/* Verbs/DevX Tx queue elements. */ -struct mlx5_txq_obj { - LIST_ENTRY(mlx5_txq_obj) next; /* Pointer to the next element. */ - struct mlx5_txq_ctrl *txq_ctrl; /* Pointer to the control queue. */ - enum mlx5_txq_obj_type type; /* The txq object type. */ - RTE_STD_C11 - union { - struct { - void *cq; /* Completion Queue. */ - void *qp; /* Queue Pair. */ - }; - struct { - struct mlx5_devx_obj *sq; - /* DevX object for Sx queue. */ - struct mlx5_devx_obj *tis; /* The TIS object. */ - }; - struct { - struct rte_eth_dev *dev; - struct mlx5_devx_obj *cq_devx; - void *cq_umem; - void *cq_buf; - int64_t cq_dbrec_offset; - struct mlx5_devx_dbr_page *cq_dbrec_page; - struct mlx5_devx_obj *sq_devx; - void *sq_umem; - void *sq_buf; - int64_t sq_dbrec_offset; - struct mlx5_devx_dbr_page *sq_dbrec_page; - }; - }; -}; - /* TX queue control descriptor. */ struct mlx5_txq_ctrl { LIST_ENTRY(mlx5_txq_ctrl) next; /* Pointer to the next element. */ @@ -400,11 +361,9 @@ int mlx5_tx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, const struct rte_eth_hairpin_conf *hairpin_conf); void mlx5_tx_queue_release(void *dpdk_txq); +void txq_uar_init(struct mlx5_txq_ctrl *txq_ctrl); int mlx5_tx_uar_init_secondary(struct rte_eth_dev *dev, int fd); void mlx5_tx_uar_uninit_secondary(struct rte_eth_dev *dev); -struct mlx5_txq_obj *mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx, - enum mlx5_txq_obj_type type); -void mlx5_txq_obj_release(struct mlx5_txq_obj *txq_obj); int mlx5_txq_obj_verify(struct rte_eth_dev *dev); struct mlx5_txq_ctrl *mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, unsigned int socket, diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index 0f4d031..6763042 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -55,16 +55,9 @@ if (!txq_ctrl) continue; - if (txq_ctrl->type == MLX5_TXQ_TYPE_HAIRPIN) { - txq_ctrl->obj = mlx5_txq_obj_new - (dev, i, MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN); - } else { + if (txq_ctrl->type == MLX5_TXQ_TYPE_STANDARD) txq_alloc_elts(txq_ctrl); - txq_ctrl->obj = mlx5_txq_obj_new - (dev, i, priv->txpp_en ? - MLX5_TXQ_OBJ_TYPE_DEVX_SQ : - MLX5_TXQ_OBJ_TYPE_IBV); - } + txq_ctrl->obj = priv->obj_ops.txq_obj_new(dev, i); if (!txq_ctrl->obj) { rte_errno = ENOMEM; goto error; diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index c678971..c1d36c3 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -20,7 +20,6 @@ #include #include #include -#include #include #include "mlx5_defs.h" @@ -589,7 +588,7 @@ * @param txq_ctrl * Pointer to Tx queue control structure. */ -static void +void txq_uar_init(struct mlx5_txq_ctrl *txq_ctrl) { struct mlx5_priv *priv = txq_ctrl->priv; @@ -765,799 +764,6 @@ } /** - * Create the Tx hairpin queue object. - * - * @param dev - * Pointer to Ethernet device. - * @param idx - * Queue index in DPDK Tx queue array - * - * @return - * The hairpin DevX object initialised, NULL otherwise and rte_errno is set. - */ -static struct mlx5_txq_obj * -mlx5_txq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx) -{ - struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; - struct mlx5_txq_ctrl *txq_ctrl = - container_of(txq_data, struct mlx5_txq_ctrl, txq); - struct mlx5_devx_create_sq_attr attr = { 0 }; - struct mlx5_txq_obj *tmpl = NULL; - uint32_t max_wq_data; - - MLX5_ASSERT(txq_data); - MLX5_ASSERT(!txq_ctrl->obj); - tmpl = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, sizeof(*tmpl), 0, - txq_ctrl->socket); - if (!tmpl) { - DRV_LOG(ERR, - "port %u Tx queue %u cannot allocate memory resources", - dev->data->port_id, txq_data->idx); - rte_errno = ENOMEM; - return NULL; - } - tmpl->type = MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN; - tmpl->txq_ctrl = txq_ctrl; - attr.hairpin = 1; - attr.tis_lst_sz = 1; - max_wq_data = priv->config.hca_attr.log_max_hairpin_wq_data_sz; - /* Jumbo frames > 9KB should be supported, and more packets. */ - if (priv->config.log_hp_size != (uint32_t)MLX5_ARG_UNSET) { - if (priv->config.log_hp_size > max_wq_data) { - DRV_LOG(ERR, "total data size %u power of 2 is " - "too large for hairpin", - priv->config.log_hp_size); - mlx5_free(tmpl); - rte_errno = ERANGE; - return NULL; - } - attr.wq_attr.log_hairpin_data_sz = priv->config.log_hp_size; - } else { - attr.wq_attr.log_hairpin_data_sz = - (max_wq_data < MLX5_HAIRPIN_JUMBO_LOG_SIZE) ? - max_wq_data : MLX5_HAIRPIN_JUMBO_LOG_SIZE; - } - /* Set the packets number to the maximum value for performance. */ - attr.wq_attr.log_hairpin_num_packets = - attr.wq_attr.log_hairpin_data_sz - - MLX5_HAIRPIN_QUEUE_STRIDE; - attr.tis_num = priv->sh->tis->id; - tmpl->sq = mlx5_devx_cmd_create_sq(priv->sh->ctx, &attr); - if (!tmpl->sq) { - DRV_LOG(ERR, - "port %u tx hairpin queue %u can't create sq object", - dev->data->port_id, idx); - mlx5_free(tmpl); - rte_errno = errno; - return NULL; - } - DRV_LOG(DEBUG, "port %u sxq %u updated with %p", dev->data->port_id, - idx, (void *)&tmpl); - LIST_INSERT_HEAD(&priv->txqsobj, tmpl, next); - return tmpl; -} - -/** - * Release DevX SQ resources. - * - * @param txq_ctrl - * DevX Tx queue object. - */ -static void -txq_release_devx_sq_resources(struct mlx5_txq_obj *txq_obj) -{ - if (txq_obj->sq_devx) - claim_zero(mlx5_devx_cmd_destroy(txq_obj->sq_devx)); - if (txq_obj->sq_umem) - claim_zero(mlx5_glue->devx_umem_dereg(txq_obj->sq_umem)); - if (txq_obj->sq_buf) - mlx5_free(txq_obj->sq_buf); - if (txq_obj->sq_dbrec_page) - claim_zero(mlx5_release_dbr(&txq_obj->txq_ctrl->priv->dbrpgs, - mlx5_os_get_umem_id - (txq_obj->sq_dbrec_page->umem), - txq_obj->sq_dbrec_offset)); -} - -/** - * Release DevX Tx CQ resources. - * - * @param txq_ctrl - * DevX Tx queue object. - */ -static void -txq_release_devx_cq_resources(struct mlx5_txq_obj *txq_obj) -{ - if (txq_obj->cq_devx) - claim_zero(mlx5_devx_cmd_destroy(txq_obj->cq_devx)); - if (txq_obj->cq_umem) - claim_zero(mlx5_glue->devx_umem_dereg(txq_obj->cq_umem)); - if (txq_obj->cq_buf) - mlx5_free(txq_obj->cq_buf); - if (txq_obj->cq_dbrec_page) - claim_zero(mlx5_release_dbr(&txq_obj->txq_ctrl->priv->dbrpgs, - mlx5_os_get_umem_id - (txq_obj->cq_dbrec_page->umem), - txq_obj->cq_dbrec_offset)); -} - -/** - * Destroy the Tx queue DevX object. - * - * @param txq_obj - * Txq object to destroy - */ -static void -txq_release_devx_resources(struct mlx5_txq_obj *txq_obj) -{ - MLX5_ASSERT(txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ); - - txq_release_devx_sq_resources(txq_obj); - txq_release_devx_cq_resources(txq_obj); -} - -#ifdef HAVE_MLX5DV_DEVX_UAR_OFFSET -/** - * Create a DevX CQ object for a Tx queue. - * - * @param dev - * Pointer to Ethernet device. - * @param cqe_n - * Number of entries in the CQ. - * @param idx - * Queue index in DPDK Tx queue array. - * @param type - * Type of the Tx queue object to create. - * - * @return - * The DevX CQ object initialized, NULL otherwise and rte_errno is set. - */ -static struct mlx5_devx_obj * -mlx5_devx_cq_new(struct rte_eth_dev *dev, uint32_t cqe_n, uint16_t idx, - struct mlx5_txq_obj *txq_obj) -{ - struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; - struct mlx5_devx_obj *cq_obj = NULL; - struct mlx5_devx_cq_attr cq_attr = { 0 }; - struct mlx5_cqe *cqe; - size_t page_size; - size_t alignment; - uint32_t i; - int ret; - - MLX5_ASSERT(txq_data); - MLX5_ASSERT(txq_obj); - page_size = rte_mem_page_size(); - if (page_size == (size_t)-1) { - DRV_LOG(ERR, "Failed to get mem page size"); - rte_errno = ENOMEM; - return NULL; - } - /* Allocate memory buffer for CQEs. */ - alignment = MLX5_CQE_BUF_ALIGNMENT; - if (alignment == (size_t)-1) { - DRV_LOG(ERR, "Failed to get CQE buf alignment"); - rte_errno = ENOMEM; - return NULL; - } - cqe_n = 1UL << log2above(cqe_n); - if (cqe_n > UINT16_MAX) { - DRV_LOG(ERR, - "port %u Tx queue %u requests to many CQEs %u", - dev->data->port_id, txq_data->idx, cqe_n); - rte_errno = EINVAL; - return NULL; - } - txq_obj->cq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - cqe_n * sizeof(struct mlx5_cqe), - alignment, - priv->sh->numa_node); - if (!txq_obj->cq_buf) { - DRV_LOG(ERR, - "port %u Tx queue %u cannot allocate memory (CQ)", - dev->data->port_id, txq_data->idx); - rte_errno = ENOMEM; - return NULL; - } - /* Register allocated buffer in user space with DevX. */ - txq_obj->cq_umem = mlx5_glue->devx_umem_reg(priv->sh->ctx, - (void *)txq_obj->cq_buf, - cqe_n * sizeof(struct mlx5_cqe), - IBV_ACCESS_LOCAL_WRITE); - if (!txq_obj->cq_umem) { - rte_errno = errno; - DRV_LOG(ERR, - "port %u Tx queue %u cannot register memory (CQ)", - dev->data->port_id, txq_data->idx); - goto error; - } - /* Allocate doorbell record for completion queue. */ - txq_obj->cq_dbrec_offset = mlx5_get_dbr(priv->sh->ctx, - &priv->dbrpgs, - &txq_obj->cq_dbrec_page); - if (txq_obj->cq_dbrec_offset < 0) { - rte_errno = errno; - DRV_LOG(ERR, "Failed to allocate CQ door-bell."); - goto error; - } - cq_attr.cqe_size = (sizeof(struct mlx5_cqe) == 128) ? - MLX5_CQE_SIZE_128B : MLX5_CQE_SIZE_64B; - cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar); - cq_attr.eqn = priv->sh->eqn; - cq_attr.q_umem_valid = 1; - cq_attr.q_umem_offset = (uintptr_t)txq_obj->cq_buf % page_size; - cq_attr.q_umem_id = mlx5_os_get_umem_id(txq_obj->cq_umem); - cq_attr.db_umem_valid = 1; - cq_attr.db_umem_offset = txq_obj->cq_dbrec_offset; - cq_attr.db_umem_id = mlx5_os_get_umem_id(txq_obj->cq_dbrec_page->umem); - cq_attr.log_cq_size = rte_log2_u32(cqe_n); - cq_attr.log_page_size = rte_log2_u32(page_size); - /* Create completion queue object with DevX. */ - cq_obj = mlx5_devx_cmd_create_cq(priv->sh->ctx, &cq_attr); - if (!cq_obj) { - rte_errno = errno; - DRV_LOG(ERR, "port %u Tx queue %u CQ creation failure", - dev->data->port_id, idx); - goto error; - } - txq_data->cqe_n = log2above(cqe_n); - txq_data->cqe_s = 1 << txq_data->cqe_n; - /* Initial fill CQ buffer with invalid CQE opcode. */ - cqe = (struct mlx5_cqe *)txq_obj->cq_buf; - for (i = 0; i < txq_data->cqe_s; i++) { - cqe->op_own = (MLX5_CQE_INVALID << 4) | MLX5_CQE_OWNER_MASK; - ++cqe; - } - return cq_obj; -error: - ret = rte_errno; - txq_release_devx_cq_resources(txq_obj); - rte_errno = ret; - return NULL; -} - -/** - * Create a SQ object using DevX. - * - * @param dev - * Pointer to Ethernet device. - * @param idx - * Queue index in DPDK Tx queue array. - * @param type - * Type of the Tx queue object to create. - * - * @return - * The DevX object initialized, NULL otherwise and rte_errno is set. - */ -static struct mlx5_devx_obj * -mlx5_devx_sq_new(struct rte_eth_dev *dev, uint16_t idx, - struct mlx5_txq_obj *txq_obj) -{ - struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; - struct mlx5_devx_create_sq_attr sq_attr = { 0 }; - struct mlx5_devx_obj *sq_obj = NULL; - size_t page_size; - uint32_t wqe_n; - int ret; - - MLX5_ASSERT(txq_data); - MLX5_ASSERT(txq_obj); - page_size = rte_mem_page_size(); - if (page_size == (size_t)-1) { - DRV_LOG(ERR, "Failed to get mem page size"); - rte_errno = ENOMEM; - return NULL; - } - wqe_n = RTE_MIN(1UL << txq_data->elts_n, - (uint32_t)priv->sh->device_attr.max_qp_wr); - txq_obj->sq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - wqe_n * sizeof(struct mlx5_wqe), - page_size, priv->sh->numa_node); - if (!txq_obj->sq_buf) { - DRV_LOG(ERR, - "port %u Tx queue %u cannot allocate memory (SQ)", - dev->data->port_id, txq_data->idx); - rte_errno = ENOMEM; - goto error; - } - /* Register allocated buffer in user space with DevX. */ - txq_obj->sq_umem = mlx5_glue->devx_umem_reg - (priv->sh->ctx, - (void *)txq_obj->sq_buf, - wqe_n * sizeof(struct mlx5_wqe), - IBV_ACCESS_LOCAL_WRITE); - if (!txq_obj->sq_umem) { - rte_errno = errno; - DRV_LOG(ERR, - "port %u Tx queue %u cannot register memory (SQ)", - dev->data->port_id, txq_data->idx); - goto error; - } - /* Allocate doorbell record for send queue. */ - txq_obj->sq_dbrec_offset = mlx5_get_dbr(priv->sh->ctx, - &priv->dbrpgs, - &txq_obj->sq_dbrec_page); - if (txq_obj->sq_dbrec_offset < 0) { - rte_errno = errno; - DRV_LOG(ERR, "Failed to allocate SQ door-bell."); - goto error; - } - sq_attr.tis_lst_sz = 1; - sq_attr.tis_num = priv->sh->tis->id; - sq_attr.state = MLX5_SQC_STATE_RST; - sq_attr.cqn = txq_obj->cq_devx->id; - sq_attr.flush_in_error_en = 1; - sq_attr.allow_multi_pkt_send_wqe = !!priv->config.mps; - sq_attr.allow_swp = !!priv->config.swp; - sq_attr.min_wqe_inline_mode = priv->config.hca_attr.vport_inline_mode; - sq_attr.wq_attr.uar_page = - mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar); - sq_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC; - sq_attr.wq_attr.pd = priv->sh->pdn; - sq_attr.wq_attr.log_wq_stride = rte_log2_u32(MLX5_WQE_SIZE); - sq_attr.wq_attr.log_wq_sz = log2above(wqe_n); - sq_attr.wq_attr.dbr_umem_valid = 1; - sq_attr.wq_attr.dbr_addr = txq_obj->sq_dbrec_offset; - sq_attr.wq_attr.dbr_umem_id = - mlx5_os_get_umem_id(txq_obj->sq_dbrec_page->umem); - sq_attr.wq_attr.wq_umem_valid = 1; - sq_attr.wq_attr.wq_umem_id = mlx5_os_get_umem_id(txq_obj->sq_umem); - sq_attr.wq_attr.wq_umem_offset = (uintptr_t)txq_obj->sq_buf % page_size; - /* Create Send Queue object with DevX. */ - sq_obj = mlx5_devx_cmd_create_sq(priv->sh->ctx, &sq_attr); - if (!sq_obj) { - rte_errno = errno; - DRV_LOG(ERR, "port %u Tx queue %u SQ creation failure", - dev->data->port_id, idx); - goto error; - } - txq_data->wqe_n = log2above(wqe_n); - return sq_obj; -error: - ret = rte_errno; - txq_release_devx_sq_resources(txq_obj); - rte_errno = ret; - return NULL; -} -#endif - -/** - * Create the Tx queue DevX object. - * - * @param dev - * Pointer to Ethernet device. - * @param idx - * Queue index in DPDK Tx queue array. - * - * @return - * The DevX object initialised, NULL otherwise and rte_errno is set. - */ -static struct mlx5_txq_obj * -mlx5_txq_obj_devx_new(struct rte_eth_dev *dev, uint16_t idx) -{ -#ifndef HAVE_MLX5DV_DEVX_UAR_OFFSET - DRV_LOG(ERR, "port %u Tx queue %u cannot create with DevX, no UAR", - dev->data->port_id, idx); - rte_errno = ENOMEM; - return NULL; -#else - struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_dev_ctx_shared *sh = priv->sh; - struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; - struct mlx5_txq_ctrl *txq_ctrl = - container_of(txq_data, struct mlx5_txq_ctrl, txq); - struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; - struct mlx5_txq_obj *txq_obj = NULL; - void *reg_addr; - uint32_t cqe_n; - int ret = 0; - - MLX5_ASSERT(txq_data); - MLX5_ASSERT(!txq_ctrl->obj); - txq_obj = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - sizeof(struct mlx5_txq_obj), 0, - txq_ctrl->socket); - if (!txq_obj) { - DRV_LOG(ERR, - "port %u Tx queue %u cannot allocate memory resources", - dev->data->port_id, txq_data->idx); - rte_errno = ENOMEM; - return NULL; - } - txq_obj->type = MLX5_TXQ_OBJ_TYPE_DEVX_SQ; - txq_obj->txq_ctrl = txq_ctrl; - txq_obj->dev = dev; - /* Create the Completion Queue. */ - cqe_n = (1UL << txq_data->elts_n) / MLX5_TX_COMP_THRESH + - 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; - /* Create completion queue object with DevX. */ - txq_obj->cq_devx = mlx5_devx_cq_new(dev, cqe_n, idx, txq_obj); - if (!txq_obj->cq_devx) { - rte_errno = errno; - goto error; - } - txq_data->cqe_m = txq_data->cqe_s - 1; - txq_data->cqes = (volatile struct mlx5_cqe *)txq_obj->cq_buf; - txq_data->cq_ci = 0; - txq_data->cq_pi = 0; - txq_data->cq_db = (volatile uint32_t *)(txq_obj->cq_dbrec_page->dbrs + - txq_obj->cq_dbrec_offset); - *txq_data->cq_db = 0; - /* Create Send Queue object with DevX. */ - txq_obj->sq_devx = mlx5_devx_sq_new(dev, idx, txq_obj); - if (!txq_obj->sq_devx) { - rte_errno = errno; - goto error; - } - /* Create the Work Queue. */ - txq_data->wqe_s = 1 << txq_data->wqe_n; - txq_data->wqe_m = txq_data->wqe_s - 1; - txq_data->wqes = (struct mlx5_wqe *)txq_obj->sq_buf; - txq_data->wqes_end = txq_data->wqes + txq_data->wqe_s; - txq_data->wqe_ci = 0; - txq_data->wqe_pi = 0; - txq_data->wqe_comp = 0; - txq_data->wqe_thres = txq_data->wqe_s / MLX5_TX_COMP_THRESH_INLINE_DIV; - txq_data->qp_db = (volatile uint32_t *) - (txq_obj->sq_dbrec_page->dbrs + - txq_obj->sq_dbrec_offset + - MLX5_SND_DBR * sizeof(uint32_t)); - *txq_data->qp_db = 0; - txq_data->qp_num_8s = txq_obj->sq_devx->id << 8; - /* Change Send Queue state to Ready-to-Send. */ - msq_attr.sq_state = MLX5_SQC_STATE_RST; - msq_attr.state = MLX5_SQC_STATE_RDY; - ret = mlx5_devx_cmd_modify_sq(txq_obj->sq_devx, &msq_attr); - if (ret) { - rte_errno = errno; - DRV_LOG(ERR, - "port %u Tx queue %u SP state to SQC_STATE_RDY failed", - dev->data->port_id, idx); - goto error; - } - txq_data->fcqs = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - txq_data->cqe_s * sizeof(*txq_data->fcqs), - RTE_CACHE_LINE_SIZE, - txq_ctrl->socket); - if (!txq_data->fcqs) { - DRV_LOG(ERR, "port %u Tx queue %u cannot allocate memory (FCQ)", - dev->data->port_id, idx); - rte_errno = ENOMEM; - goto error; - } -#ifdef HAVE_IBV_FLOW_DV_SUPPORT - /* - * If using DevX need to query and store TIS transport domain value. - * This is done once per port. - * Will use this value on Rx, when creating matching TIR. - */ - if (priv->config.devx && !priv->sh->tdn) - priv->sh->tdn = priv->sh->td->id; -#endif - MLX5_ASSERT(sh->tx_uar); - reg_addr = mlx5_os_get_devx_uar_reg_addr(sh->tx_uar); - MLX5_ASSERT(reg_addr); - txq_ctrl->bf_reg = reg_addr; - txq_ctrl->uar_mmap_offset = - mlx5_os_get_devx_uar_mmap_offset(sh->tx_uar); - txq_uar_init(txq_ctrl); - LIST_INSERT_HEAD(&priv->txqsobj, txq_obj, next); - return txq_obj; -error: - ret = rte_errno; /* Save rte_errno before cleanup. */ - txq_release_devx_resources(txq_obj); - if (txq_data->fcqs) { - mlx5_free(txq_data->fcqs); - txq_data->fcqs = NULL; - } - mlx5_free(txq_obj); - rte_errno = ret; /* Restore rte_errno. */ - return NULL; -#endif -} - -/** - * Create a QP Verbs object. - * - * @param dev - * Pointer to Ethernet device. - * @param idx - * Queue index in DPDK Tx queue array. - * @param rxq_obj - * Pointer to Tx queue object data. - * - * @return - * The QP Verbs object initialized, NULL otherwise and rte_errno is set. - */ -static struct ibv_qp * -mlx5_ibv_qp_new(struct rte_eth_dev *dev, uint16_t idx, - struct mlx5_txq_obj *txq_obj) -{ - struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; - struct mlx5_txq_ctrl *txq_ctrl = - container_of(txq_data, struct mlx5_txq_ctrl, txq); - struct ibv_qp *qp_obj = NULL; - struct ibv_qp_init_attr_ex qp_attr = { 0 }; - const int desc = 1 << txq_data->elts_n; - - MLX5_ASSERT(!txq_ctrl->obj); - /* CQ to be associated with the send queue. */ - qp_attr.send_cq = txq_obj->cq; - /* CQ to be associated with the receive queue. */ - qp_attr.recv_cq = txq_obj->cq; - /* Max number of outstanding WRs. */ - qp_attr.cap.max_send_wr = ((priv->sh->device_attr.max_qp_wr < desc) ? - priv->sh->device_attr.max_qp_wr : desc); - /* - * Max number of scatter/gather elements in a WR, must be 1 to prevent - * libmlx5 from trying to affect must be 1 to prevent libmlx5 from - * trying to affect too much memory. TX gather is not impacted by the - * device_attr.max_sge limit and will still work properly. - */ - qp_attr.cap.max_send_sge = 1; - qp_attr.qp_type = IBV_QPT_RAW_PACKET, - /* Do *NOT* enable this, completions events are managed per Tx burst. */ - qp_attr.sq_sig_all = 0; - qp_attr.pd = priv->sh->pd; - qp_attr.comp_mask = IBV_QP_INIT_ATTR_PD; - if (txq_data->inlen_send) - qp_attr.cap.max_inline_data = txq_ctrl->max_inline_data; - if (txq_data->tso_en) { - qp_attr.max_tso_header = txq_ctrl->max_tso_header; - qp_attr.comp_mask |= IBV_QP_INIT_ATTR_MAX_TSO_HEADER; - } - qp_obj = mlx5_glue->create_qp_ex(priv->sh->ctx, &qp_attr); - if (qp_obj == NULL) { - DRV_LOG(ERR, "port %u Tx queue %u QP creation failure", - dev->data->port_id, idx); - rte_errno = errno; - } - return qp_obj; -} - -/** - * Create the Tx queue Verbs object. - * - * @param dev - * Pointer to Ethernet device. - * @param idx - * Queue index in DPDK Tx queue array. - * @param type - * Type of the Tx queue object to create. - * - * @return - * The Verbs object initialised, NULL otherwise and rte_errno is set. - */ -struct mlx5_txq_obj * -mlx5_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx, - enum mlx5_txq_obj_type type) -{ - struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; - struct mlx5_txq_ctrl *txq_ctrl = - container_of(txq_data, struct mlx5_txq_ctrl, txq); - struct mlx5_txq_obj tmpl; - struct mlx5_txq_obj *txq_obj = NULL; - struct ibv_qp_attr mod; - unsigned int cqe_n; - struct mlx5dv_qp qp = { .comp_mask = MLX5DV_QP_MASK_UAR_MMAP_OFFSET }; - struct mlx5dv_cq cq_info; - struct mlx5dv_obj obj; - const int desc = 1 << txq_data->elts_n; - int ret = 0; - - if (type == MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN) - return mlx5_txq_obj_hairpin_new(dev, idx); - if (type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ) - return mlx5_txq_obj_devx_new(dev, idx); -#ifdef HAVE_IBV_FLOW_DV_SUPPORT - /* If using DevX, need additional mask to read tisn value. */ - if (priv->config.devx && !priv->sh->tdn) - qp.comp_mask |= MLX5DV_QP_MASK_RAW_QP_HANDLES; -#endif - MLX5_ASSERT(txq_data); - priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_TX_QUEUE; - priv->verbs_alloc_ctx.obj = txq_ctrl; - if (mlx5_getenv_int("MLX5_ENABLE_CQE_COMPRESSION")) { - DRV_LOG(ERR, - "port %u MLX5_ENABLE_CQE_COMPRESSION must never be set", - dev->data->port_id); - rte_errno = EINVAL; - return NULL; - } - memset(&tmpl, 0, sizeof(struct mlx5_txq_obj)); - cqe_n = desc / MLX5_TX_COMP_THRESH + - 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; - tmpl.cq = mlx5_glue->create_cq(priv->sh->ctx, cqe_n, NULL, NULL, 0); - if (tmpl.cq == NULL) { - DRV_LOG(ERR, "port %u Tx queue %u CQ creation failure", - dev->data->port_id, idx); - rte_errno = errno; - goto error; - } - tmpl.qp = mlx5_ibv_qp_new(dev, idx, &tmpl); - if (tmpl.qp == NULL) { - rte_errno = errno; - goto error; - } - mod = (struct ibv_qp_attr){ - /* Move the QP to this state. */ - .qp_state = IBV_QPS_INIT, - /* IB device port number. */ - .port_num = (uint8_t)priv->dev_port, - }; - ret = mlx5_glue->modify_qp(tmpl.qp, &mod, - (IBV_QP_STATE | IBV_QP_PORT)); - if (ret) { - DRV_LOG(ERR, - "port %u Tx queue %u QP state to IBV_QPS_INIT failed", - dev->data->port_id, idx); - rte_errno = errno; - goto error; - } - mod = (struct ibv_qp_attr){ - .qp_state = IBV_QPS_RTR - }; - ret = mlx5_glue->modify_qp(tmpl.qp, &mod, IBV_QP_STATE); - if (ret) { - DRV_LOG(ERR, - "port %u Tx queue %u QP state to IBV_QPS_RTR failed", - dev->data->port_id, idx); - rte_errno = errno; - goto error; - } - mod.qp_state = IBV_QPS_RTS; - ret = mlx5_glue->modify_qp(tmpl.qp, &mod, IBV_QP_STATE); - if (ret) { - DRV_LOG(ERR, - "port %u Tx queue %u QP state to IBV_QPS_RTS failed", - dev->data->port_id, idx); - rte_errno = errno; - goto error; - } - txq_obj = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - sizeof(struct mlx5_txq_obj), 0, - txq_ctrl->socket); - if (!txq_obj) { - DRV_LOG(ERR, "port %u Tx queue %u cannot allocate memory", - dev->data->port_id, idx); - rte_errno = ENOMEM; - goto error; - } - obj.cq.in = tmpl.cq; - obj.cq.out = &cq_info; - obj.qp.in = tmpl.qp; - obj.qp.out = &qp; - ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_CQ | MLX5DV_OBJ_QP); - if (ret != 0) { - rte_errno = errno; - goto error; - } - if (cq_info.cqe_size != RTE_CACHE_LINE_SIZE) { - DRV_LOG(ERR, - "port %u wrong MLX5_CQE_SIZE environment variable" - " value: it should be set to %u", - dev->data->port_id, RTE_CACHE_LINE_SIZE); - rte_errno = EINVAL; - goto error; - } - txq_data->cqe_n = log2above(cq_info.cqe_cnt); - txq_data->cqe_s = 1 << txq_data->cqe_n; - txq_data->cqe_m = txq_data->cqe_s - 1; - txq_data->qp_num_8s = ((struct ibv_qp *)tmpl.qp)->qp_num << 8; - txq_data->wqes = qp.sq.buf; - txq_data->wqe_n = log2above(qp.sq.wqe_cnt); - txq_data->wqe_s = 1 << txq_data->wqe_n; - txq_data->wqe_m = txq_data->wqe_s - 1; - txq_data->wqes_end = txq_data->wqes + txq_data->wqe_s; - txq_data->qp_db = &qp.dbrec[MLX5_SND_DBR]; - txq_data->cq_db = cq_info.dbrec; - txq_data->cqes = (volatile struct mlx5_cqe *)cq_info.buf; - txq_data->cq_ci = 0; - txq_data->cq_pi = 0; - txq_data->wqe_ci = 0; - txq_data->wqe_pi = 0; - txq_data->wqe_comp = 0; - txq_data->wqe_thres = txq_data->wqe_s / MLX5_TX_COMP_THRESH_INLINE_DIV; - txq_data->fcqs = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - txq_data->cqe_s * sizeof(*txq_data->fcqs), - RTE_CACHE_LINE_SIZE, txq_ctrl->socket); - if (!txq_data->fcqs) { - DRV_LOG(ERR, "port %u Tx queue %u cannot allocate memory (FCQ)", - dev->data->port_id, idx); - rte_errno = ENOMEM; - goto error; - } -#ifdef HAVE_IBV_FLOW_DV_SUPPORT - /* - * If using DevX need to query and store TIS transport domain value. - * This is done once per port. - * Will use this value on Rx, when creating matching TIR. - */ - if (priv->config.devx && !priv->sh->tdn) { - ret = mlx5_devx_cmd_qp_query_tis_td(tmpl.qp, qp.tisn, - &priv->sh->tdn); - if (ret) { - DRV_LOG(ERR, "Fail to query port %u Tx queue %u QP TIS " - "transport domain", dev->data->port_id, idx); - rte_errno = EINVAL; - goto error; - } else { - DRV_LOG(DEBUG, "port %u Tx queue %u TIS number %d " - "transport domain %d", dev->data->port_id, - idx, qp.tisn, priv->sh->tdn); - } - } -#endif - txq_obj->qp = tmpl.qp; - txq_obj->cq = tmpl.cq; - txq_ctrl->bf_reg = qp.bf.reg; - if (qp.comp_mask & MLX5DV_QP_MASK_UAR_MMAP_OFFSET) { - txq_ctrl->uar_mmap_offset = qp.uar_mmap_offset; - DRV_LOG(DEBUG, "port %u: uar_mmap_offset 0x%"PRIx64, - dev->data->port_id, txq_ctrl->uar_mmap_offset); - } else { - DRV_LOG(ERR, - "port %u failed to retrieve UAR info, invalid" - " libmlx5.so", - dev->data->port_id); - rte_errno = EINVAL; - goto error; - } - txq_uar_init(txq_ctrl); - LIST_INSERT_HEAD(&priv->txqsobj, txq_obj, next); - txq_obj->txq_ctrl = txq_ctrl; - priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; - return txq_obj; -error: - ret = rte_errno; /* Save rte_errno before cleanup. */ - if (tmpl.cq) - claim_zero(mlx5_glue->destroy_cq(tmpl.cq)); - if (tmpl.qp) - claim_zero(mlx5_glue->destroy_qp(tmpl.qp)); - if (txq_data && txq_data->fcqs) { - mlx5_free(txq_data->fcqs); - txq_data->fcqs = NULL; - } - if (txq_obj) - mlx5_free(txq_obj); - priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; - rte_errno = ret; /* Restore rte_errno. */ - return NULL; -} - -/** - * Release an Tx verbs queue object. - * - * @param txq_obj - * Verbs Tx queue object.. - */ -void -mlx5_txq_obj_release(struct mlx5_txq_obj *txq_obj) -{ - MLX5_ASSERT(txq_obj); - if (txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN) { - if (txq_obj->tis) - claim_zero(mlx5_devx_cmd_destroy(txq_obj->tis)); - } else if (txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ) { - txq_release_devx_resources(txq_obj); - } else { - claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); - claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); - } - if (txq_obj->txq_ctrl->txq.fcqs) { - mlx5_free(txq_obj->txq_ctrl->txq.fcqs); - txq_obj->txq_ctrl->txq.fcqs = NULL; - } - LIST_REMOVE(txq_obj, next); - mlx5_free(txq_obj); -} - -/** * Verify the Verbs Tx queue list is empty * * @param dev @@ -2100,7 +1306,7 @@ struct mlx5_txq_ctrl * if (!rte_atomic32_dec_and_test(&txq->refcnt)) return 1; if (txq->obj) { - mlx5_txq_obj_release(txq->obj); + priv->obj_ops.txq_obj_release(txq->obj); txq->obj = NULL; } txq_free_elts(txq); From patchwork Thu Oct 1 14:09:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79477 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id F1458A04BA; Thu, 1 Oct 2020 16:13:34 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2E8931DBAA; Thu, 1 Oct 2020 16:11:14 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id D6A771DBD1 for ; Thu, 1 Oct 2020 16:11:12 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:11:08 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAEO012743; Thu, 1 Oct 2020 17:11:08 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:19 +0000 Message-Id: <1601561366-1821-9-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 08/15] net/mlx5: share Tx control code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Move Tx object similar resources allocations and debug logs from DevX and Verbs modules to a shared location. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/linux/mlx5_os.c | 4 +- drivers/net/mlx5/linux/mlx5_verbs.c | 84 ++++++++++++------------------------- drivers/net/mlx5/linux/mlx5_verbs.h | 3 +- drivers/net/mlx5/mlx5.h | 3 +- drivers/net/mlx5/mlx5_devx.c | 75 +++++++-------------------------- drivers/net/mlx5/mlx5_devx.h | 3 +- drivers/net/mlx5/mlx5_trigger.c | 31 +++++++++++++- drivers/net/mlx5/mlx5_txq.c | 28 ++++++++----- 8 files changed, 93 insertions(+), 138 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index c5332a0..0db2b5a 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -520,9 +520,9 @@ * Queue index in DPDK Tx queue array. * * @return - * The DevX/Verbs object initialized, NULL otherwise and rte_errno is set. + * 0 on success, a negative errno value otherwise and rte_errno is set. */ -static struct mlx5_txq_obj * +static int mlx5_os_txq_obj_new(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index c79c4a2..5568c75 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -807,7 +807,7 @@ struct ibv_qp_init_attr_ex qp_attr = { 0 }; const int desc = 1 << txq_data->elts_n; - MLX5_ASSERT(!txq_ctrl->obj); + MLX5_ASSERT(txq_ctrl->obj); /* CQ to be associated with the send queue. */ qp_attr.send_cq = txq_obj->cq; /* CQ to be associated with the receive queue. */ @@ -851,17 +851,16 @@ * Queue index in DPDK Tx queue array. * * @return - * The Verbs object initialized, NULL otherwise and rte_errno is set. + * 0 on success, a negative errno value otherwise and rte_errno is set. */ -struct mlx5_txq_obj * +int mlx5_txq_ibv_obj_new(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; struct mlx5_txq_ctrl *txq_ctrl = container_of(txq_data, struct mlx5_txq_ctrl, txq); - struct mlx5_txq_obj tmpl; - struct mlx5_txq_obj *txq_obj = NULL; + struct mlx5_txq_obj *txq_obj = txq_ctrl->obj; struct ibv_qp_attr mod; unsigned int cqe_n; struct mlx5dv_qp qp; @@ -871,26 +870,28 @@ struct mlx5_txq_obj * int ret = 0; MLX5_ASSERT(txq_data); + MLX5_ASSERT(txq_obj); + txq_obj->type = MLX5_TXQ_OBJ_TYPE_IBV; + txq_obj->txq_ctrl = txq_ctrl; priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_TX_QUEUE; priv->verbs_alloc_ctx.obj = txq_ctrl; if (mlx5_getenv_int("MLX5_ENABLE_CQE_COMPRESSION")) { DRV_LOG(ERR, "Port %u MLX5_ENABLE_CQE_COMPRESSION " "must never be set.", dev->data->port_id); rte_errno = EINVAL; - return NULL; + return -rte_errno; } - memset(&tmpl, 0, sizeof(struct mlx5_txq_obj)); cqe_n = desc / MLX5_TX_COMP_THRESH + 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; - tmpl.cq = mlx5_glue->create_cq(priv->sh->ctx, cqe_n, NULL, NULL, 0); - if (tmpl.cq == NULL) { + txq_obj->cq = mlx5_glue->create_cq(priv->sh->ctx, cqe_n, NULL, NULL, 0); + if (txq_obj->cq == NULL) { DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.", dev->data->port_id, idx); rte_errno = errno; goto error; } - tmpl.qp = mlx5_ibv_qp_new(dev, idx, &tmpl); - if (tmpl.qp == NULL) { + txq_obj->qp = mlx5_ibv_qp_new(dev, idx, txq_obj); + if (txq_obj->qp == NULL) { rte_errno = errno; goto error; } @@ -900,7 +901,8 @@ struct mlx5_txq_obj * /* IB device port number. */ .port_num = (uint8_t)priv->dev_port, }; - ret = mlx5_glue->modify_qp(tmpl.qp, &mod, (IBV_QP_STATE | IBV_QP_PORT)); + ret = mlx5_glue->modify_qp(txq_obj->qp, &mod, + (IBV_QP_STATE | IBV_QP_PORT)); if (ret) { DRV_LOG(ERR, "Port %u Tx queue %u QP state to IBV_QPS_INIT failed.", @@ -911,7 +913,7 @@ struct mlx5_txq_obj * mod = (struct ibv_qp_attr){ .qp_state = IBV_QPS_RTR }; - ret = mlx5_glue->modify_qp(tmpl.qp, &mod, IBV_QP_STATE); + ret = mlx5_glue->modify_qp(txq_obj->qp, &mod, IBV_QP_STATE); if (ret) { DRV_LOG(ERR, "Port %u Tx queue %u QP state to IBV_QPS_RTR failed.", @@ -920,7 +922,7 @@ struct mlx5_txq_obj * goto error; } mod.qp_state = IBV_QPS_RTS; - ret = mlx5_glue->modify_qp(tmpl.qp, &mod, IBV_QP_STATE); + ret = mlx5_glue->modify_qp(txq_obj->qp, &mod, IBV_QP_STATE); if (ret) { DRV_LOG(ERR, "Port %u Tx queue %u QP state to IBV_QPS_RTS failed.", @@ -928,24 +930,15 @@ struct mlx5_txq_obj * rte_errno = errno; goto error; } - txq_obj = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - sizeof(struct mlx5_txq_obj), 0, - txq_ctrl->socket); - if (!txq_obj) { - DRV_LOG(ERR, "Port %u Tx queue %u cannot allocate memory.", - dev->data->port_id, idx); - rte_errno = ENOMEM; - goto error; - } qp.comp_mask = MLX5DV_QP_MASK_UAR_MMAP_OFFSET; #ifdef HAVE_IBV_FLOW_DV_SUPPORT /* If using DevX, need additional mask to read tisn value. */ if (priv->sh->devx && !priv->sh->tdn) qp.comp_mask |= MLX5DV_QP_MASK_RAW_QP_HANDLES; #endif - obj.cq.in = tmpl.cq; + obj.cq.in = txq_obj->cq; obj.cq.out = &cq_info; - obj.qp.in = tmpl.qp; + obj.qp.in = txq_obj->qp; obj.qp.out = &qp; ret = mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_CQ | MLX5DV_OBJ_QP); if (ret != 0) { @@ -963,7 +956,7 @@ struct mlx5_txq_obj * txq_data->cqe_n = log2above(cq_info.cqe_cnt); txq_data->cqe_s = 1 << txq_data->cqe_n; txq_data->cqe_m = txq_data->cqe_s - 1; - txq_data->qp_num_8s = ((struct ibv_qp *)tmpl.qp)->qp_num << 8; + txq_data->qp_num_8s = ((struct ibv_qp *)txq_obj->qp)->qp_num << 8; txq_data->wqes = qp.sq.buf; txq_data->wqe_n = log2above(qp.sq.wqe_cnt); txq_data->wqe_s = 1 << txq_data->wqe_n; @@ -978,15 +971,6 @@ struct mlx5_txq_obj * txq_data->wqe_pi = 0; txq_data->wqe_comp = 0; txq_data->wqe_thres = txq_data->wqe_s / MLX5_TX_COMP_THRESH_INLINE_DIV; - txq_data->fcqs = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - txq_data->cqe_s * sizeof(*txq_data->fcqs), - RTE_CACHE_LINE_SIZE, txq_ctrl->socket); - if (!txq_data->fcqs) { - DRV_LOG(ERR, "Port %u Tx queue %u can't allocate memory (FCQ).", - dev->data->port_id, idx); - rte_errno = ENOMEM; - goto error; - } #ifdef HAVE_IBV_FLOW_DV_SUPPORT /* * If using DevX need to query and store TIS transport domain value. @@ -994,7 +978,7 @@ struct mlx5_txq_obj * * Will use this value on Rx, when creating matching TIR. */ if (priv->sh->devx && !priv->sh->tdn) { - ret = mlx5_devx_cmd_qp_query_tis_td(tmpl.qp, qp.tisn, + ret = mlx5_devx_cmd_qp_query_tis_td(txq_obj->qp, qp.tisn, &priv->sh->tdn); if (ret) { DRV_LOG(ERR, "Fail to query port %u Tx queue %u QP TIS " @@ -1008,8 +992,6 @@ struct mlx5_txq_obj * } } #endif - txq_obj->qp = tmpl.qp; - txq_obj->cq = tmpl.cq; txq_ctrl->bf_reg = qp.bf.reg; if (qp.comp_mask & MLX5DV_QP_MASK_UAR_MMAP_OFFSET) { txq_ctrl->uar_mmap_offset = qp.uar_mmap_offset; @@ -1024,25 +1006,17 @@ struct mlx5_txq_obj * goto error; } txq_uar_init(txq_ctrl); - txq_obj->txq_ctrl = txq_ctrl; - LIST_INSERT_HEAD(&priv->txqsobj, txq_obj, next); priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; - return txq_obj; + return 0; error: ret = rte_errno; /* Save rte_errno before cleanup. */ - if (tmpl.cq) - claim_zero(mlx5_glue->destroy_cq(tmpl.cq)); - if (tmpl.qp) - claim_zero(mlx5_glue->destroy_qp(tmpl.qp)); - if (txq_data->fcqs) { - mlx5_free(txq_data->fcqs); - txq_data->fcqs = NULL; - } - if (txq_obj) - mlx5_free(txq_obj); + if (txq_obj->cq) + claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); + if (txq_obj->qp) + claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; rte_errno = ret; /* Restore rte_errno. */ - return NULL; + return -rte_errno; } /** @@ -1057,12 +1031,6 @@ struct mlx5_txq_obj * MLX5_ASSERT(txq_obj); claim_zero(mlx5_glue->destroy_qp(txq_obj->qp)); claim_zero(mlx5_glue->destroy_cq(txq_obj->cq)); - if (txq_obj->txq_ctrl->txq.fcqs) { - mlx5_free(txq_obj->txq_ctrl->txq.fcqs); - txq_obj->txq_ctrl->txq.fcqs = NULL; - } - LIST_REMOVE(txq_obj, next); - mlx5_free(txq_obj); } struct mlx5_obj_ops ibv_obj_ops = { diff --git a/drivers/net/mlx5/linux/mlx5_verbs.h b/drivers/net/mlx5/linux/mlx5_verbs.h index 7f6bb99..0670f6c 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.h +++ b/drivers/net/mlx5/linux/mlx5_verbs.h @@ -12,8 +12,7 @@ struct mlx5_verbs_ops { mlx5_dereg_mr_t dereg_mr; }; -struct mlx5_txq_obj *mlx5_txq_ibv_obj_new(struct rte_eth_dev *dev, - uint16_t idx); +int mlx5_txq_ibv_obj_new(struct rte_eth_dev *dev, uint16_t idx); void mlx5_txq_ibv_obj_release(struct mlx5_txq_obj *txq_obj); /* Verbs ops struct */ diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 8679750..3093f6e 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -789,8 +789,7 @@ struct mlx5_obj_ops { void (*hrxq_destroy)(struct mlx5_hrxq *hrxq); int (*drop_action_create)(struct rte_eth_dev *dev); void (*drop_action_destroy)(struct rte_eth_dev *dev); - struct mlx5_txq_obj *(*txq_obj_new)(struct rte_eth_dev *dev, - uint16_t idx); + int (*txq_obj_new)(struct rte_eth_dev *dev, uint16_t idx); void (*txq_obj_release)(struct mlx5_txq_obj *txq_obj); }; diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 0b6e116..f3437a6 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -819,9 +819,9 @@ * Queue index in DPDK Tx queue array. * * @return - * The hairpin DevX object initialized, NULL otherwise and rte_errno is set. + * 0 on success, a negative errno value otherwise and rte_errno is set. */ -static struct mlx5_txq_obj * +static int mlx5_txq_obj_hairpin_new(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; @@ -829,20 +829,11 @@ struct mlx5_txq_ctrl *txq_ctrl = container_of(txq_data, struct mlx5_txq_ctrl, txq); struct mlx5_devx_create_sq_attr attr = { 0 }; - struct mlx5_txq_obj *tmpl = NULL; + struct mlx5_txq_obj *tmpl = txq_ctrl->obj; uint32_t max_wq_data; MLX5_ASSERT(txq_data); - MLX5_ASSERT(!txq_ctrl->obj); - tmpl = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, sizeof(*tmpl), 0, - txq_ctrl->socket); - if (!tmpl) { - DRV_LOG(ERR, - "Port %u Tx queue %u cannot allocate memory resources.", - dev->data->port_id, txq_data->idx); - rte_errno = ENOMEM; - return NULL; - } + MLX5_ASSERT(tmpl); tmpl->type = MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN; tmpl->txq_ctrl = txq_ctrl; attr.hairpin = 1; @@ -854,9 +845,8 @@ DRV_LOG(ERR, "Total data size %u power of 2 is " "too large for hairpin.", priv->config.log_hp_size); - mlx5_free(tmpl); rte_errno = ERANGE; - return NULL; + return -rte_errno; } attr.wq_attr.log_hairpin_data_sz = priv->config.log_hp_size; } else { @@ -874,14 +864,10 @@ DRV_LOG(ERR, "Port %u tx hairpin queue %u can't create SQ object.", dev->data->port_id, idx); - mlx5_free(tmpl); rte_errno = errno; - return NULL; + return -rte_errno; } - DRV_LOG(DEBUG, "Port %u sxq %u updated with %p.", dev->data->port_id, - idx, (void *)&tmpl); - LIST_INSERT_HEAD(&priv->txqsobj, tmpl, next); - return tmpl; + return 0; } #ifdef HAVE_MLX5DV_DEVX_UAR_OFFSET @@ -1179,9 +1165,9 @@ * Queue index in DPDK Tx queue array. * * @return - * The DevX object initialized, NULL otherwise and rte_errno is set. + * 0 on success, a negative errno value otherwise and rte_errno is set. */ -struct mlx5_txq_obj * +int mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; @@ -1195,27 +1181,17 @@ struct mlx5_txq_obj * DRV_LOG(ERR, "Port %u Tx queue %u cannot create with DevX, no UAR.", dev->data->port_id, idx); rte_errno = ENOMEM; - return NULL; + return -rte_errno; #else struct mlx5_dev_ctx_shared *sh = priv->sh; struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; - struct mlx5_txq_obj *txq_obj = NULL; + struct mlx5_txq_obj *txq_obj = txq_ctrl->obj; void *reg_addr; uint32_t cqe_n; int ret = 0; MLX5_ASSERT(txq_data); - MLX5_ASSERT(!txq_ctrl->obj); - txq_obj = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - sizeof(struct mlx5_txq_obj), 0, - txq_ctrl->socket); - if (!txq_obj) { - DRV_LOG(ERR, - "Port %u Tx queue %u cannot allocate memory resources.", - dev->data->port_id, txq_data->idx); - rte_errno = ENOMEM; - return NULL; - } + MLX5_ASSERT(txq_obj); txq_obj->type = MLX5_TXQ_OBJ_TYPE_DEVX_SQ; txq_obj->txq_ctrl = txq_ctrl; txq_obj->dev = dev; @@ -1267,17 +1243,6 @@ struct mlx5_txq_obj * dev->data->port_id, idx); goto error; } - txq_data->fcqs = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - txq_data->cqe_s * sizeof(*txq_data->fcqs), - RTE_CACHE_LINE_SIZE, - txq_ctrl->socket); - if (!txq_data->fcqs) { - DRV_LOG(ERR, - "Port %u Tx queue %u cannot allocate memory (FCQ).", - dev->data->port_id, idx); - rte_errno = ENOMEM; - goto error; - } #ifdef HAVE_IBV_FLOW_DV_SUPPORT /* * If using DevX need to query and store TIS transport domain value. @@ -1294,18 +1259,12 @@ struct mlx5_txq_obj * txq_ctrl->uar_mmap_offset = mlx5_os_get_devx_uar_mmap_offset(sh->tx_uar); txq_uar_init(txq_ctrl); - LIST_INSERT_HEAD(&priv->txqsobj, txq_obj, next); - return txq_obj; + return 0; error: ret = rte_errno; /* Save rte_errno before cleanup. */ txq_release_devx_resources(txq_obj); - if (txq_data->fcqs) { - mlx5_free(txq_data->fcqs); - txq_data->fcqs = NULL; - } - mlx5_free(txq_obj); rte_errno = ret; /* Restore rte_errno. */ - return NULL; + return -rte_errno; #endif } @@ -1327,12 +1286,6 @@ struct mlx5_txq_obj * txq_release_devx_resources(txq_obj); #endif } - if (txq_obj->txq_ctrl->txq.fcqs) { - mlx5_free(txq_obj->txq_ctrl->txq.fcqs); - txq_obj->txq_ctrl->txq.fcqs = NULL; - } - LIST_REMOVE(txq_obj, next); - mlx5_free(txq_obj); } struct mlx5_obj_ops devx_obj_ops = { diff --git a/drivers/net/mlx5/mlx5_devx.h b/drivers/net/mlx5/mlx5_devx.h index 0bbbbc0..bc8a8d6 100644 --- a/drivers/net/mlx5/mlx5_devx.h +++ b/drivers/net/mlx5/mlx5_devx.h @@ -7,8 +7,7 @@ #include "mlx5.h" -struct mlx5_txq_obj *mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, - uint16_t idx); +int mlx5_txq_devx_obj_new(struct rte_eth_dev *dev, uint16_t idx); void mlx5_txq_devx_obj_release(struct mlx5_txq_obj *txq_obj); extern struct mlx5_obj_ops devx_obj_ops; diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index 6763042..e72e5fb 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -52,16 +52,45 @@ for (i = 0; i != priv->txqs_n; ++i) { struct mlx5_txq_ctrl *txq_ctrl = mlx5_txq_get(dev, i); + struct mlx5_txq_data *txq_data = &txq_ctrl->txq; + uint32_t flags = MLX5_MEM_RTE | MLX5_MEM_ZERO; if (!txq_ctrl) continue; if (txq_ctrl->type == MLX5_TXQ_TYPE_STANDARD) txq_alloc_elts(txq_ctrl); - txq_ctrl->obj = priv->obj_ops.txq_obj_new(dev, i); + MLX5_ASSERT(!txq_ctrl->obj); + txq_ctrl->obj = mlx5_malloc(flags, sizeof(struct mlx5_txq_obj), + 0, txq_ctrl->socket); if (!txq_ctrl->obj) { + DRV_LOG(ERR, "Port %u Tx queue %u cannot allocate " + "memory resources.", dev->data->port_id, + txq_data->idx); rte_errno = ENOMEM; goto error; } + ret = priv->obj_ops.txq_obj_new(dev, i); + if (ret < 0) { + mlx5_free(txq_ctrl->obj); + txq_ctrl->obj = NULL; + goto error; + } + if (txq_ctrl->type == MLX5_TXQ_TYPE_STANDARD) { + size_t size = txq_data->cqe_s * sizeof(*txq_data->fcqs); + txq_data->fcqs = mlx5_malloc(flags, size, + RTE_CACHE_LINE_SIZE, + txq_ctrl->socket); + if (!txq_data->fcqs) { + DRV_LOG(ERR, "Port %u Tx queue %u cannot " + "allocate memory (FCQ).", + dev->data->port_id, i); + rte_errno = ENOMEM; + goto error; + } + } + DRV_LOG(DEBUG, "Port %u txq %u updated with %p.", + dev->data->port_id, i, (void *)&txq_ctrl->obj); + LIST_INSERT_HEAD(&priv->txqsobj, txq_ctrl->obj, next); } return 0; error: diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index c1d36c3..23213d9 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -1298,21 +1298,29 @@ struct mlx5_txq_ctrl * mlx5_txq_release(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_txq_ctrl *txq; + struct mlx5_txq_ctrl *txq_ctrl; if (!(*priv->txqs)[idx]) return 0; - txq = container_of((*priv->txqs)[idx], struct mlx5_txq_ctrl, txq); - if (!rte_atomic32_dec_and_test(&txq->refcnt)) + txq_ctrl = container_of((*priv->txqs)[idx], struct mlx5_txq_ctrl, txq); + if (!rte_atomic32_dec_and_test(&txq_ctrl->refcnt)) return 1; - if (txq->obj) { - priv->obj_ops.txq_obj_release(txq->obj); - txq->obj = NULL; + if (txq_ctrl->obj) { + priv->obj_ops.txq_obj_release(txq_ctrl->obj); + LIST_REMOVE(txq_ctrl->obj, next); + mlx5_free(txq_ctrl->obj); + txq_ctrl->obj = NULL; + } + if (txq_ctrl->type == MLX5_TXQ_TYPE_STANDARD) { + if (txq_ctrl->txq.fcqs) { + mlx5_free(txq_ctrl->txq.fcqs); + txq_ctrl->txq.fcqs = NULL; + } + txq_free_elts(txq_ctrl); + mlx5_mr_btree_free(&txq_ctrl->txq.mr_ctrl.cache_bh); } - txq_free_elts(txq); - mlx5_mr_btree_free(&txq->txq.mr_ctrl.cache_bh); - LIST_REMOVE(txq, next); - mlx5_free(txq); + LIST_REMOVE(txq_ctrl, next); + mlx5_free(txq_ctrl); (*priv->txqs)[idx] = NULL; dev->data->tx_queue_state[idx] = RTE_ETH_QUEUE_STATE_STOPPED; return 0; From patchwork Thu Oct 1 14:09:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79478 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2EB38A04BA; Thu, 1 Oct 2020 16:13:59 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8D0521DBC9; Thu, 1 Oct 2020 16:11:19 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id F07AB1DBA9 for ; Thu, 1 Oct 2020 16:11:17 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:11:14 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAEP012743; Thu, 1 Oct 2020 17:11:14 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:20 +0000 Message-Id: <1601561366-1821-10-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 09/15] net/mlx5: rearrange SQ and CQ creation in DevX module X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" 1. Rename functions to mention the internal resources. 2. Reduce the number of function arguments. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5_devx.c | 127 +++++++++++++++++++++---------------------- 1 file changed, 62 insertions(+), 65 deletions(-) diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index f3437a6..55fe946 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -79,7 +79,7 @@ * DevX Rx queue object. */ static void -rxq_release_devx_rq_resources(struct mlx5_rxq_ctrl *rxq_ctrl) +mlx5_rxq_release_devx_rq_resources(struct mlx5_rxq_ctrl *rxq_ctrl) { struct mlx5_devx_dbr_page *dbr_page = rxq_ctrl->rq_dbrec_page; @@ -106,7 +106,7 @@ * DevX Rx queue object. */ static void -rxq_release_devx_cq_resources(struct mlx5_rxq_ctrl *rxq_ctrl) +mlx5_rxq_release_devx_cq_resources(struct mlx5_rxq_ctrl *rxq_ctrl) { struct mlx5_devx_dbr_page *dbr_page = rxq_ctrl->cq_dbrec_page; @@ -147,8 +147,8 @@ if (rxq_obj->devx_channel) mlx5_glue->devx_destroy_event_channel (rxq_obj->devx_channel); - rxq_release_devx_rq_resources(rxq_obj->rxq_ctrl); - rxq_release_devx_cq_resources(rxq_obj->rxq_ctrl); + mlx5_rxq_release_devx_rq_resources(rxq_obj->rxq_ctrl); + mlx5_rxq_release_devx_cq_resources(rxq_obj->rxq_ctrl); } } @@ -247,7 +247,7 @@ * The DevX RQ object initialized, NULL otherwise and rte_errno is set. */ static struct mlx5_devx_obj * -rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx) +mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_rxq_data *rxq_data = (*priv->rxqs)[idx]; @@ -325,7 +325,7 @@ goto error; return rq; error: - rxq_release_devx_rq_resources(rxq_ctrl); + mlx5_rxq_release_devx_rq_resources(rxq_ctrl); return NULL; } @@ -341,7 +341,7 @@ * The DevX CQ object initialized, NULL otherwise and rte_errno is set. */ static struct mlx5_devx_obj * -rxq_create_devx_cq_resources(struct rte_eth_dev *dev, uint16_t idx) +mlx5_rxq_create_devx_cq_resources(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_devx_obj *cq_obj = 0; struct mlx5_devx_cq_attr cq_attr = { 0 }; @@ -451,7 +451,7 @@ error: if (cq_obj) mlx5_devx_cmd_destroy(cq_obj); - rxq_release_devx_cq_resources(rxq_ctrl); + mlx5_rxq_release_devx_cq_resources(rxq_ctrl); return NULL; } @@ -558,13 +558,13 @@ tmpl->fd = mlx5_os_get_devx_channel_fd(tmpl->devx_channel); } /* Create CQ using DevX API. */ - tmpl->devx_cq = rxq_create_devx_cq_resources(dev, idx); + tmpl->devx_cq = mlx5_rxq_create_devx_cq_resources(dev, idx); if (!tmpl->devx_cq) { DRV_LOG(ERR, "Failed to create CQ."); goto error; } /* Create RQ using DevX API. */ - tmpl->rq = rxq_create_devx_rq_resources(dev, idx); + tmpl->rq = mlx5_rxq_create_devx_rq_resources(dev, idx); if (!tmpl->rq) { DRV_LOG(ERR, "Port %u Rx queue %u RQ creation failure.", dev->data->port_id, idx); @@ -589,8 +589,8 @@ claim_zero(mlx5_devx_cmd_destroy(tmpl->devx_cq)); if (tmpl->devx_channel) mlx5_glue->devx_destroy_event_channel(tmpl->devx_channel); - rxq_release_devx_rq_resources(rxq_ctrl); - rxq_release_devx_cq_resources(rxq_ctrl); + mlx5_rxq_release_devx_rq_resources(rxq_ctrl); + mlx5_rxq_release_devx_cq_resources(rxq_ctrl); rte_errno = ret; /* Restore rte_errno. */ return -rte_errno; } @@ -878,7 +878,7 @@ * DevX Tx queue object. */ static void -txq_release_devx_sq_resources(struct mlx5_txq_obj *txq_obj) +mlx5_txq_release_devx_sq_resources(struct mlx5_txq_obj *txq_obj) { if (txq_obj->sq_devx) claim_zero(mlx5_devx_cmd_destroy(txq_obj->sq_devx)); @@ -900,7 +900,7 @@ * DevX Tx queue object. */ static void -txq_release_devx_cq_resources(struct mlx5_txq_obj *txq_obj) +mlx5_txq_release_devx_cq_resources(struct mlx5_txq_obj *txq_obj) { if (txq_obj->cq_devx) claim_zero(mlx5_devx_cmd_destroy(txq_obj->cq_devx)); @@ -922,40 +922,38 @@ * Txq object to destroy. */ static void -txq_release_devx_resources(struct mlx5_txq_obj *txq_obj) +mlx5_txq_release_devx_resources(struct mlx5_txq_obj *txq_obj) { MLX5_ASSERT(txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ); - txq_release_devx_cq_resources(txq_obj); - txq_release_devx_sq_resources(txq_obj); + mlx5_txq_release_devx_cq_resources(txq_obj); + mlx5_txq_release_devx_sq_resources(txq_obj); } /** - * Create a DevX CQ object for an Tx queue. + * Create a DevX CQ object and its resources for an Tx queue. * * @param dev * Pointer to Ethernet device. - * @param cqe_n - * Number of entries in the CQ. * @param idx * Queue index in DPDK Tx queue array. - * @param rxq_obj - * Pointer to Tx queue object data. * * @return - * The DevX CQ object initialized, NULL otherwise and rte_errno is set. + * Number of CQEs in CQ, 0 otherwise and rte_errno is set. */ -static struct mlx5_devx_obj * -mlx5_tx_devx_cq_new(struct rte_eth_dev *dev, uint32_t cqe_n, uint16_t idx, - struct mlx5_txq_obj *txq_obj) +static uint32_t +mlx5_txq_create_devx_cq_resources(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; - struct mlx5_devx_obj *cq_obj = NULL; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq_data, struct mlx5_txq_ctrl, txq); + struct mlx5_txq_obj *txq_obj = txq_ctrl->obj; struct mlx5_devx_cq_attr cq_attr = { 0 }; struct mlx5_cqe *cqe; size_t page_size; size_t alignment; + uint32_t cqe_n; uint32_t i; int ret; @@ -965,22 +963,25 @@ if (page_size == (size_t)-1) { DRV_LOG(ERR, "Failed to get mem page size."); rte_errno = ENOMEM; - return NULL; + return 0; } /* Allocate memory buffer for CQEs. */ alignment = MLX5_CQE_BUF_ALIGNMENT; if (alignment == (size_t)-1) { DRV_LOG(ERR, "Failed to get CQE buf alignment."); rte_errno = ENOMEM; - return NULL; + return 0; } + /* Create the Completion Queue. */ + cqe_n = (1UL << txq_data->elts_n) / MLX5_TX_COMP_THRESH + + 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; cqe_n = 1UL << log2above(cqe_n); if (cqe_n > UINT16_MAX) { DRV_LOG(ERR, "Port %u Tx queue %u requests to many CQEs %u.", dev->data->port_id, txq_data->idx, cqe_n); rte_errno = EINVAL; - return NULL; + return 0; } txq_obj->cq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, cqe_n * sizeof(struct mlx5_cqe), @@ -991,7 +992,7 @@ "Port %u Tx queue %u cannot allocate memory (CQ).", dev->data->port_id, txq_data->idx); rte_errno = ENOMEM; - return NULL; + return 0; } /* Register allocated buffer in user space with DevX. */ txq_obj->cq_umem = mlx5_glue->devx_umem_reg(priv->sh->ctx, @@ -1027,50 +1028,47 @@ cq_attr.log_cq_size = rte_log2_u32(cqe_n); cq_attr.log_page_size = rte_log2_u32(page_size); /* Create completion queue object with DevX. */ - cq_obj = mlx5_devx_cmd_create_cq(priv->sh->ctx, &cq_attr); - if (!cq_obj) { + txq_obj->cq_devx = mlx5_devx_cmd_create_cq(priv->sh->ctx, &cq_attr); + if (!txq_obj->cq_devx) { rte_errno = errno; DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.", dev->data->port_id, idx); goto error; } - txq_data->cqe_n = log2above(cqe_n); - txq_data->cqe_s = 1 << txq_data->cqe_n; /* Initial fill CQ buffer with invalid CQE opcode. */ cqe = (struct mlx5_cqe *)txq_obj->cq_buf; - for (i = 0; i < txq_data->cqe_s; i++) { + for (i = 0; i < cqe_n; i++) { cqe->op_own = (MLX5_CQE_INVALID << 4) | MLX5_CQE_OWNER_MASK; ++cqe; } - return cq_obj; + return cqe_n; error: ret = rte_errno; - txq_release_devx_cq_resources(txq_obj); + mlx5_txq_release_devx_cq_resources(txq_obj); rte_errno = ret; - return NULL; + return 0; } /** - * Create a SQ object using DevX. + * Create a SQ object and its resources using DevX. * * @param dev * Pointer to Ethernet device. * @param idx * Queue index in DPDK Tx queue array. - * @param rxq_obj - * Pointer to Tx queue object data. * * @return - * The DevX SQ object initialized, NULL otherwise and rte_errno is set. + * Number of WQEs in SQ, 0 otherwise and rte_errno is set. */ -static struct mlx5_devx_obj * -mlx5_devx_sq_new(struct rte_eth_dev *dev, uint16_t idx, - struct mlx5_txq_obj *txq_obj) +static uint32_t +mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq_data, struct mlx5_txq_ctrl, txq); + struct mlx5_txq_obj *txq_obj = txq_ctrl->obj; struct mlx5_devx_create_sq_attr sq_attr = { 0 }; - struct mlx5_devx_obj *sq_obj = NULL; size_t page_size; uint32_t wqe_n; int ret; @@ -1081,7 +1079,7 @@ if (page_size == (size_t)-1) { DRV_LOG(ERR, "Failed to get mem page size."); rte_errno = ENOMEM; - return NULL; + return 0; } wqe_n = RTE_MIN(1UL << txq_data->elts_n, (uint32_t)priv->sh->device_attr.max_qp_wr); @@ -1117,7 +1115,6 @@ DRV_LOG(ERR, "Failed to allocate SQ door-bell."); goto error; } - txq_data->wqe_n = log2above(wqe_n); sq_attr.tis_lst_sz = 1; sq_attr.tis_num = priv->sh->tis->id; sq_attr.state = MLX5_SQC_STATE_RST; @@ -1131,7 +1128,7 @@ sq_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC; sq_attr.wq_attr.pd = priv->sh->pdn; sq_attr.wq_attr.log_wq_stride = rte_log2_u32(MLX5_WQE_SIZE); - sq_attr.wq_attr.log_wq_sz = txq_data->wqe_n; + sq_attr.wq_attr.log_wq_sz = log2above(wqe_n); sq_attr.wq_attr.dbr_umem_valid = 1; sq_attr.wq_attr.dbr_addr = txq_obj->sq_dbrec_offset; sq_attr.wq_attr.dbr_umem_id = @@ -1140,19 +1137,19 @@ sq_attr.wq_attr.wq_umem_id = mlx5_os_get_umem_id(txq_obj->sq_umem); sq_attr.wq_attr.wq_umem_offset = (uintptr_t)txq_obj->sq_buf % page_size; /* Create Send Queue object with DevX. */ - sq_obj = mlx5_devx_cmd_create_sq(priv->sh->ctx, &sq_attr); - if (!sq_obj) { + txq_obj->sq_devx = mlx5_devx_cmd_create_sq(priv->sh->ctx, &sq_attr); + if (!txq_obj->sq_devx) { rte_errno = errno; DRV_LOG(ERR, "Port %u Tx queue %u SQ creation failure.", dev->data->port_id, idx); goto error; } - return sq_obj; + return wqe_n; error: ret = rte_errno; - txq_release_devx_sq_resources(txq_obj); + mlx5_txq_release_devx_sq_resources(txq_obj); rte_errno = ret; - return NULL; + return 0; } #endif @@ -1188,6 +1185,7 @@ struct mlx5_txq_obj *txq_obj = txq_ctrl->obj; void *reg_addr; uint32_t cqe_n; + uint32_t wqe_n; int ret = 0; MLX5_ASSERT(txq_data); @@ -1195,15 +1193,13 @@ txq_obj->type = MLX5_TXQ_OBJ_TYPE_DEVX_SQ; txq_obj->txq_ctrl = txq_ctrl; txq_obj->dev = dev; - /* Create the Completion Queue. */ - cqe_n = (1UL << txq_data->elts_n) / MLX5_TX_COMP_THRESH + - 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; - /* Create completion queue object with DevX. */ - txq_obj->cq_devx = mlx5_tx_devx_cq_new(dev, cqe_n, idx, txq_obj); - if (!txq_obj->cq_devx) { + cqe_n = mlx5_txq_create_devx_cq_resources(dev, idx); + if (!cqe_n) { rte_errno = errno; goto error; } + txq_data->cqe_n = log2above(cqe_n); + txq_data->cqe_s = 1 << txq_data->cqe_n; txq_data->cqe_m = txq_data->cqe_s - 1; txq_data->cqes = (volatile struct mlx5_cqe *)txq_obj->cq_buf; txq_data->cq_ci = 0; @@ -1212,12 +1208,13 @@ txq_obj->cq_dbrec_offset); *txq_data->cq_db = 0; /* Create Send Queue object with DevX. */ - txq_obj->sq_devx = mlx5_devx_sq_new(dev, idx, txq_obj); - if (!txq_obj->sq_devx) { + wqe_n = mlx5_txq_create_devx_sq_resources(dev, idx); + if (!wqe_n) { rte_errno = errno; goto error; } /* Create the Work Queue. */ + txq_data->wqe_n = log2above(wqe_n); txq_data->wqe_s = 1 << txq_data->wqe_n; txq_data->wqe_m = txq_data->wqe_s - 1; txq_data->wqes = (struct mlx5_wqe *)txq_obj->sq_buf; @@ -1262,7 +1259,7 @@ return 0; error: ret = rte_errno; /* Save rte_errno before cleanup. */ - txq_release_devx_resources(txq_obj); + mlx5_txq_release_devx_resources(txq_obj); rte_errno = ret; /* Restore rte_errno. */ return -rte_errno; #endif @@ -1283,7 +1280,7 @@ claim_zero(mlx5_devx_cmd_destroy(txq_obj->tis)); #ifdef HAVE_MLX5DV_DEVX_UAR_OFFSET } else { - txq_release_devx_resources(txq_obj); + mlx5_txq_release_devx_resources(txq_obj); #endif } } From patchwork Thu Oct 1 14:09:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79479 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8D21BA04BA; Thu, 1 Oct 2020 16:14:26 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7E83D1DB85; Thu, 1 Oct 2020 16:11:24 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 0AF0F1DBE7 for ; Thu, 1 Oct 2020 16:11:21 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:11:21 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAEQ012743; Thu, 1 Oct 2020 17:11:21 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:21 +0000 Message-Id: <1601561366-1821-11-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 10/15] net/mlx5: rearrange QP creation in Verbs module X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" 1. Rename function to mention the internal resources. 2. Reduce the number of function arguments. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/linux/mlx5_verbs.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index 5568c75..0476d94 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -789,15 +789,12 @@ * Pointer to Ethernet device. * @param idx * Queue index in DPDK Tx queue array. - * @param rxq_obj - * Pointer to Tx queue object data. * * @return - * The QP Verbs object initialized, NULL otherwise and rte_errno is set. + * The QP Verbs object, NULL otherwise and rte_errno is set. */ static struct ibv_qp * -mlx5_ibv_qp_new(struct rte_eth_dev *dev, uint16_t idx, - struct mlx5_txq_obj *txq_obj) +mlx5_txq_ibv_qp_create(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; @@ -807,11 +804,11 @@ struct ibv_qp_init_attr_ex qp_attr = { 0 }; const int desc = 1 << txq_data->elts_n; - MLX5_ASSERT(txq_ctrl->obj); + MLX5_ASSERT(txq_ctrl->obj->cq); /* CQ to be associated with the send queue. */ - qp_attr.send_cq = txq_obj->cq; + qp_attr.send_cq = txq_ctrl->obj->cq; /* CQ to be associated with the receive queue. */ - qp_attr.recv_cq = txq_obj->cq; + qp_attr.recv_cq = txq_ctrl->obj->cq; /* Max number of outstanding WRs. */ qp_attr.cap.max_send_wr = ((priv->sh->device_attr.max_qp_wr < desc) ? priv->sh->device_attr.max_qp_wr : desc); @@ -890,7 +887,7 @@ rte_errno = errno; goto error; } - txq_obj->qp = mlx5_ibv_qp_new(dev, idx, txq_obj); + txq_obj->qp = mlx5_txq_ibv_qp_create(dev, idx); if (txq_obj->qp == NULL) { rte_errno = errno; goto error; From patchwork Thu Oct 1 14:09:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79480 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 286FBA04BA; Thu, 1 Oct 2020 16:14:48 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id F322A1DBFC; Thu, 1 Oct 2020 16:11:33 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 1C90B1DB96 for ; Thu, 1 Oct 2020 16:11:33 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:11:27 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAER012743; Thu, 1 Oct 2020 17:11:27 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:22 +0000 Message-Id: <1601561366-1821-12-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 11/15] net/mlx5: separate Tx queue object modification X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Separate Tx object modification to the Verbs and DevX modules. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/linux/mlx5_os.c | 7 +++ drivers/net/mlx5/linux/mlx5_verbs.c | 65 ++++++++++++++++++++++ drivers/net/mlx5/mlx5.h | 9 ++++ drivers/net/mlx5/mlx5_devx.c | 57 ++++++++++++++++++++ drivers/net/mlx5/mlx5_rxtx.c | 78 ++------------------------- drivers/net/mlx5/mlx5_txq.c | 104 ++++-------------------------------- 6 files changed, 152 insertions(+), 168 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 0db2b5a..487714f 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1375,6 +1375,13 @@ ibv_obj_ops.drop_action_create; priv->obj_ops.drop_action_destroy = ibv_obj_ops.drop_action_destroy; +#ifndef HAVE_MLX5DV_DEVX_UAR_OFFSET + priv->obj_ops.txq_obj_modify = ibv_obj_ops.txq_obj_modify; +#else + if (!config->dv_esw_en) + priv->obj_ops.txq_obj_modify = + ibv_obj_ops.txq_obj_modify; +#endif } else { priv->obj_ops = ibv_obj_ops; } diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index 0476d94..7d5ea37 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -113,6 +113,70 @@ } /** + * Modify QP using Verbs API. + * + * @param txq_obj + * Verbs Tx queue object. + * @param type + * Type of change queue state. + * @param dev_port + * IB device port number. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_ibv_modify_qp(struct mlx5_txq_obj *obj, enum mlx5_txq_modify_type type, + uint8_t dev_port) +{ + struct ibv_qp_attr mod = { + .qp_state = IBV_QPS_RESET, + .port_num = dev_port, + }; + int attr_mask = (IBV_QP_STATE | IBV_QP_PORT); + int ret; + + if (type != MLX5_TXQ_MOD_RST2RDY) { + ret = mlx5_glue->modify_qp(obj->qp, &mod, IBV_QP_STATE); + if (ret) { + DRV_LOG(ERR, "Cannot change Tx QP state to RESET %s", + strerror(errno)); + rte_errno = errno; + return ret; + } + if (type == MLX5_TXQ_MOD_RDY2RST) + return 0; + } + if (type == MLX5_TXQ_MOD_ERR2RDY) + attr_mask = IBV_QP_STATE; + mod.qp_state = IBV_QPS_INIT; + ret = mlx5_glue->modify_qp(obj->qp, &mod, attr_mask); + if (ret) { + DRV_LOG(ERR, "Cannot change Tx QP state to INIT %s", + strerror(errno)); + rte_errno = errno; + return ret; + } + mod.qp_state = IBV_QPS_RTR; + ret = mlx5_glue->modify_qp(obj->qp, &mod, IBV_QP_STATE); + if (ret) { + DRV_LOG(ERR, "Cannot change Tx QP state to RTR %s", + strerror(errno)); + rte_errno = errno; + return ret; + } + mod.qp_state = IBV_QPS_RTS; + ret = mlx5_glue->modify_qp(obj->qp, &mod, IBV_QP_STATE); + if (ret) { + DRV_LOG(ERR, "Cannot change Tx QP state to RTS %s", + strerror(errno)); + rte_errno = errno; + return ret; + } + return 0; +} + +/** * Create a CQ Verbs object. * * @param dev @@ -1043,5 +1107,6 @@ struct mlx5_obj_ops ibv_obj_ops = { .drop_action_create = mlx5_ibv_drop_action_create, .drop_action_destroy = mlx5_ibv_drop_action_destroy, .txq_obj_new = mlx5_txq_ibv_obj_new, + .txq_obj_modify = mlx5_ibv_modify_qp, .txq_obj_release = mlx5_txq_ibv_obj_release, }; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 3093f6e..7cbb09b 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -774,6 +774,13 @@ struct mlx5_txq_obj { }; }; +enum mlx5_txq_modify_type { + MLX5_TXQ_MOD_RDY2RDY, /* modify state from ready to ready. */ + MLX5_TXQ_MOD_RST2RDY, /* modify state from reset to ready. */ + MLX5_TXQ_MOD_RDY2RST, /* modify state from ready to reset. */ + MLX5_TXQ_MOD_ERR2RDY, /* modify state from error to ready. */ +}; + /* HW objects operations structure. */ struct mlx5_obj_ops { int (*rxq_obj_modify_vlan_strip)(struct mlx5_rxq_obj *rxq_obj, int on); @@ -790,6 +797,8 @@ struct mlx5_obj_ops { int (*drop_action_create)(struct rte_eth_dev *dev); void (*drop_action_destroy)(struct rte_eth_dev *dev); int (*txq_obj_new)(struct rte_eth_dev *dev, uint16_t idx); + int (*txq_obj_modify)(struct mlx5_txq_obj *obj, + enum mlx5_txq_modify_type type, uint8_t dev_port); void (*txq_obj_release)(struct mlx5_txq_obj *txq_obj); }; diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 55fe946..7404a15 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -73,6 +73,62 @@ } /** + * Modify SQ using DevX API. + * + * @param txq_obj + * DevX Tx queue object. + * @param type + * Type of change queue state. + * @param dev_port + * Unnecessary. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_devx_modify_sq(struct mlx5_txq_obj *obj, enum mlx5_txq_modify_type type, + uint8_t dev_port) +{ + struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; + int ret; + + if (type != MLX5_TXQ_MOD_RST2RDY) { + /* Change queue state to reset. */ + if (type == MLX5_TXQ_MOD_ERR2RDY) + msq_attr.sq_state = MLX5_SQC_STATE_ERR; + else + msq_attr.sq_state = MLX5_SQC_STATE_RDY; + msq_attr.state = MLX5_SQC_STATE_RST; + ret = mlx5_devx_cmd_modify_sq(obj->sq_devx, &msq_attr); + if (ret) { + DRV_LOG(ERR, "Cannot change the Tx SQ state to RESET" + " %s", strerror(errno)); + rte_errno = errno; + return ret; + } + } + if (type != MLX5_TXQ_MOD_RDY2RST) { + /* Change queue state to ready. */ + msq_attr.sq_state = MLX5_SQC_STATE_RST; + msq_attr.state = MLX5_SQC_STATE_RDY; + ret = mlx5_devx_cmd_modify_sq(obj->sq_devx, &msq_attr); + if (ret) { + DRV_LOG(ERR, "Cannot change the Tx SQ state to READY" + " %s", strerror(errno)); + rte_errno = errno; + return ret; + } + } + /* + * The dev_port variable is relevant only in Verbs API, and there is a + * pointer that points to this function and a parallel function in verbs + * intermittently, so they should have the same parameters. + */ + (void)dev_port; + return 0; +} + +/** * Release the resources allocated for an RQ DevX object. * * @param rxq_ctrl @@ -1298,5 +1354,6 @@ struct mlx5_obj_ops devx_obj_ops = { .drop_action_create = mlx5_devx_drop_action_create, .drop_action_destroy = mlx5_devx_drop_action_destroy, .txq_obj_new = mlx5_txq_devx_obj_new, + .txq_obj_modify = mlx5_devx_modify_sq, .txq_obj_release = mlx5_txq_devx_obj_release, }; diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index 0b87be1..af4b4ba 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -936,79 +936,11 @@ enum mlx5_txcmp_code { struct mlx5_txq_ctrl *txq_ctrl = container_of(txq, struct mlx5_txq_ctrl, txq); - if (txq_ctrl->obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ) { - struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; - - /* Change queue state to reset. */ - msq_attr.sq_state = MLX5_SQC_STATE_ERR; - msq_attr.state = MLX5_SQC_STATE_RST; - ret = mlx5_devx_cmd_modify_sq(txq_ctrl->obj->sq_devx, - &msq_attr); - if (ret) { - DRV_LOG(ERR, "Cannot change the " - "Tx QP state to RESET %s", - strerror(errno)); - rte_errno = errno; - return ret; - } - /* Change queue state to ready. */ - msq_attr.sq_state = MLX5_SQC_STATE_RST; - msq_attr.state = MLX5_SQC_STATE_RDY; - ret = mlx5_devx_cmd_modify_sq(txq_ctrl->obj->sq_devx, - &msq_attr); - if (ret) { - DRV_LOG(ERR, "Cannot change the " - "Tx QP state to READY %s", - strerror(errno)); - rte_errno = errno; - return ret; - } - } else { - struct ibv_qp_attr mod = { - .qp_state = IBV_QPS_RESET, - .port_num = (uint8_t)priv->dev_port, - }; - struct ibv_qp *qp = txq_ctrl->obj->qp; - - MLX5_ASSERT - (txq_ctrl->obj->type == MLX5_TXQ_OBJ_TYPE_IBV); - - ret = mlx5_glue->modify_qp(qp, &mod, IBV_QP_STATE); - if (ret) { - DRV_LOG(ERR, "Cannot change the " - "Tx QP state to RESET %s", - strerror(errno)); - rte_errno = errno; - return ret; - } - mod.qp_state = IBV_QPS_INIT; - ret = mlx5_glue->modify_qp(qp, &mod, IBV_QP_STATE); - if (ret) { - DRV_LOG(ERR, "Cannot change the " - "Tx QP state to INIT %s", - strerror(errno)); - rte_errno = errno; - return ret; - } - mod.qp_state = IBV_QPS_RTR; - ret = mlx5_glue->modify_qp(qp, &mod, IBV_QP_STATE); - if (ret) { - DRV_LOG(ERR, "Cannot change the " - "Tx QP state to RTR %s", - strerror(errno)); - rte_errno = errno; - return ret; - } - mod.qp_state = IBV_QPS_RTS; - ret = mlx5_glue->modify_qp(qp, &mod, IBV_QP_STATE); - if (ret) { - DRV_LOG(ERR, "Cannot change the " - "Tx QP state to RTS %s", - strerror(errno)); - rte_errno = errno; - return ret; - } - } + ret = priv->obj_ops.txq_obj_modify(txq_ctrl->obj, + MLX5_TXQ_MOD_ERR2RDY, + (uint8_t)priv->dev_port); + if (ret) + return ret; } return 0; } diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 23213d9..c31e446 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -182,37 +182,10 @@ MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY); /* Move QP to RESET state. */ - if (txq_ctrl->obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ) { - struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; - - /* Change queue state to reset with DevX. */ - msq_attr.sq_state = MLX5_SQC_STATE_RDY; - msq_attr.state = MLX5_SQC_STATE_RST; - ret = mlx5_devx_cmd_modify_sq(txq_ctrl->obj->sq_devx, - &msq_attr); - if (ret) { - DRV_LOG(ERR, "Cannot change the " - "Tx QP state to RESET %s", - strerror(errno)); - rte_errno = errno; - return ret; - } - } else { - struct ibv_qp_attr mod = { - .qp_state = IBV_QPS_RESET, - .port_num = (uint8_t)priv->dev_port, - }; - struct ibv_qp *qp = txq_ctrl->obj->qp; - - /* Change queue state to reset with Verbs. */ - ret = mlx5_glue->modify_qp(qp, &mod, IBV_QP_STATE); - if (ret) { - DRV_LOG(ERR, "Cannot change the Tx QP state to RESET " - "%s", strerror(errno)); - rte_errno = errno; - return ret; - } - } + ret = priv->obj_ops.txq_obj_modify(txq_ctrl->obj, MLX5_TXQ_MOD_RDY2RST, + (uint8_t)priv->dev_port); + if (ret) + return ret; /* Handle all send completions. */ txq_sync_cq(txq); /* Free elts stored in the SQ. */ @@ -281,70 +254,11 @@ int ret; MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY); - if (txq_ctrl->obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ) { - struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; - struct mlx5_txq_obj *obj = txq_ctrl->obj; - - msq_attr.sq_state = MLX5_SQC_STATE_RDY; - msq_attr.state = MLX5_SQC_STATE_RST; - ret = mlx5_devx_cmd_modify_sq(obj->sq_devx, &msq_attr); - if (ret) { - rte_errno = errno; - DRV_LOG(ERR, - "Cannot change the Tx QP state to RESET " - "%s", strerror(errno)); - return ret; - } - msq_attr.sq_state = MLX5_SQC_STATE_RST; - msq_attr.state = MLX5_SQC_STATE_RDY; - ret = mlx5_devx_cmd_modify_sq(obj->sq_devx, &msq_attr); - if (ret) { - rte_errno = errno; - DRV_LOG(ERR, - "Cannot change the Tx QP state to READY " - "%s", strerror(errno)); - return ret; - } - } else { - struct ibv_qp_attr mod = { - .qp_state = IBV_QPS_RESET, - .port_num = (uint8_t)priv->dev_port, - }; - struct ibv_qp *qp = txq_ctrl->obj->qp; - - ret = mlx5_glue->modify_qp(qp, &mod, IBV_QP_STATE); - if (ret) { - DRV_LOG(ERR, "Cannot change the Tx QP state to RESET " - "%s", strerror(errno)); - rte_errno = errno; - return ret; - } - mod.qp_state = IBV_QPS_INIT; - ret = mlx5_glue->modify_qp(qp, &mod, - (IBV_QP_STATE | IBV_QP_PORT)); - if (ret) { - DRV_LOG(ERR, "Cannot change Tx QP state to INIT %s", - strerror(errno)); - rte_errno = errno; - return ret; - } - mod.qp_state = IBV_QPS_RTR; - ret = mlx5_glue->modify_qp(qp, &mod, IBV_QP_STATE); - if (ret) { - DRV_LOG(ERR, "Cannot change Tx QP state to RTR %s", - strerror(errno)); - rte_errno = errno; - return ret; - } - mod.qp_state = IBV_QPS_RTS; - ret = mlx5_glue->modify_qp(qp, &mod, IBV_QP_STATE); - if (ret) { - DRV_LOG(ERR, "Cannot change Tx QP state to RTS %s", - strerror(errno)); - rte_errno = errno; - return ret; - } - } + ret = priv->obj_ops.txq_obj_modify(txq_ctrl->obj, + MLX5_TXQ_MOD_RDY2RDY, + (uint8_t)priv->dev_port); + if (ret) + return ret; txq_ctrl->txq.wqe_ci = 0; txq_ctrl->txq.wqe_pi = 0; txq_ctrl->txq.elts_comp = 0; From patchwork Thu Oct 1 14:09:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79481 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 69047A04BA; Thu, 1 Oct 2020 16:15:34 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0DBD91DBF6; Thu, 1 Oct 2020 16:11:39 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 2C3361DB73 for ; Thu, 1 Oct 2020 16:11:38 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:11:35 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAES012743; Thu, 1 Oct 2020 17:11:35 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:23 +0000 Message-Id: <1601561366-1821-13-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 12/15] net/mlx5: share Tx queue object modification X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Use new modify_qp functions for Tx object creation in DevX and Verbs modules. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/linux/mlx5_verbs.c | 34 +++------------------------------- drivers/net/mlx5/mlx5_devx.c | 7 ++----- drivers/net/mlx5/mlx5_txq.c | 2 -- 3 files changed, 5 insertions(+), 38 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index 7d5ea37..ad6e3d7 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -922,7 +922,6 @@ struct mlx5_txq_ctrl *txq_ctrl = container_of(txq_data, struct mlx5_txq_ctrl, txq); struct mlx5_txq_obj *txq_obj = txq_ctrl->obj; - struct ibv_qp_attr mod; unsigned int cqe_n; struct mlx5dv_qp qp; struct mlx5dv_cq cq_info; @@ -956,37 +955,10 @@ rte_errno = errno; goto error; } - mod = (struct ibv_qp_attr){ - /* Move the QP to this state. */ - .qp_state = IBV_QPS_INIT, - /* IB device port number. */ - .port_num = (uint8_t)priv->dev_port, - }; - ret = mlx5_glue->modify_qp(txq_obj->qp, &mod, - (IBV_QP_STATE | IBV_QP_PORT)); - if (ret) { - DRV_LOG(ERR, - "Port %u Tx queue %u QP state to IBV_QPS_INIT failed.", - dev->data->port_id, idx); - rte_errno = errno; - goto error; - } - mod = (struct ibv_qp_attr){ - .qp_state = IBV_QPS_RTR - }; - ret = mlx5_glue->modify_qp(txq_obj->qp, &mod, IBV_QP_STATE); + ret = mlx5_ibv_modify_qp(txq_obj, MLX5_TXQ_MOD_RST2RDY, + (uint8_t)priv->dev_port); if (ret) { - DRV_LOG(ERR, - "Port %u Tx queue %u QP state to IBV_QPS_RTR failed.", - dev->data->port_id, idx); - rte_errno = errno; - goto error; - } - mod.qp_state = IBV_QPS_RTS; - ret = mlx5_glue->modify_qp(txq_obj->qp, &mod, IBV_QP_STATE); - if (ret) { - DRV_LOG(ERR, - "Port %u Tx queue %u QP state to IBV_QPS_RTS failed.", + DRV_LOG(ERR, "Port %u Tx queue %u QP state modifying failed.", dev->data->port_id, idx); rte_errno = errno; goto error; diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 7404a15..c876ae9 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -1237,7 +1237,6 @@ return -rte_errno; #else struct mlx5_dev_ctx_shared *sh = priv->sh; - struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; struct mlx5_txq_obj *txq_obj = txq_ctrl->obj; void *reg_addr; uint32_t cqe_n; @@ -1286,13 +1285,11 @@ *txq_data->qp_db = 0; txq_data->qp_num_8s = txq_obj->sq_devx->id << 8; /* Change Send Queue state to Ready-to-Send. */ - msq_attr.sq_state = MLX5_SQC_STATE_RST; - msq_attr.state = MLX5_SQC_STATE_RDY; - ret = mlx5_devx_cmd_modify_sq(txq_obj->sq_devx, &msq_attr); + ret = mlx5_devx_modify_sq(txq_obj, MLX5_TXQ_MOD_RST2RDY, 0); if (ret) { rte_errno = errno; DRV_LOG(ERR, - "Port %u Tx queue %u SP state to SQC_STATE_RDY failed.", + "Port %u Tx queue %u SQ state to SQC_STATE_RDY failed.", dev->data->port_id, idx); goto error; } diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index c31e446..af84f5f 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -16,8 +16,6 @@ #include #include -#include -#include #include #include #include From patchwork Thu Oct 1 14:09:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79482 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 300AFA04BA; Thu, 1 Oct 2020 16:15:54 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4654B1C1AC; Thu, 1 Oct 2020 16:11:49 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 3BCD61C1CA for ; Thu, 1 Oct 2020 16:11:48 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:11:42 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAET012743; Thu, 1 Oct 2020 17:11:42 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:24 +0000 Message-Id: <1601561366-1821-14-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 13/15] net/mlx5: remove Tx queue object type field X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Once the separation between Verbs and DevX is done using function pointers, the type field of the Tx queue object structure becomes redundant and no more code is used. Remove the unnecessary field from the structure. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/linux/mlx5_verbs.c | 1 - drivers/net/mlx5/mlx5.h | 8 -------- drivers/net/mlx5/mlx5_devx.c | 6 +----- 3 files changed, 1 insertion(+), 14 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index ad6e3d7..6260b4e 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -931,7 +931,6 @@ MLX5_ASSERT(txq_data); MLX5_ASSERT(txq_obj); - txq_obj->type = MLX5_TXQ_OBJ_TYPE_IBV; txq_obj->txq_ctrl = txq_ctrl; priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_TX_QUEUE; priv->verbs_alloc_ctx.obj = txq_ctrl; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 7cbb09b..fd5dd87 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -735,18 +735,10 @@ struct mlx5_hrxq { uint8_t rss_key[]; /* Hash key. */ }; -enum mlx5_txq_obj_type { - MLX5_TXQ_OBJ_TYPE_IBV, /* mlx5_txq_obj with ibv_wq. */ - MLX5_TXQ_OBJ_TYPE_DEVX_SQ, /* mlx5_txq_obj with mlx5_devx_sq. */ - MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN, - /* mlx5_txq_obj with mlx5_devx_tq and hairpin support. */ -}; - /* Verbs/DevX Tx queue elements. */ struct mlx5_txq_obj { LIST_ENTRY(mlx5_txq_obj) next; /* Pointer to the next element. */ struct mlx5_txq_ctrl *txq_ctrl; /* Pointer to the control queue. */ - enum mlx5_txq_obj_type type; /* The txq object type. */ RTE_STD_C11 union { struct { diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index c876ae9..430ad08 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -890,7 +890,6 @@ MLX5_ASSERT(txq_data); MLX5_ASSERT(tmpl); - tmpl->type = MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN; tmpl->txq_ctrl = txq_ctrl; attr.hairpin = 1; attr.tis_lst_sz = 1; @@ -980,8 +979,6 @@ static void mlx5_txq_release_devx_resources(struct mlx5_txq_obj *txq_obj) { - MLX5_ASSERT(txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_SQ); - mlx5_txq_release_devx_cq_resources(txq_obj); mlx5_txq_release_devx_sq_resources(txq_obj); } @@ -1245,7 +1242,6 @@ MLX5_ASSERT(txq_data); MLX5_ASSERT(txq_obj); - txq_obj->type = MLX5_TXQ_OBJ_TYPE_DEVX_SQ; txq_obj->txq_ctrl = txq_ctrl; txq_obj->dev = dev; cqe_n = mlx5_txq_create_devx_cq_resources(dev, idx); @@ -1328,7 +1324,7 @@ mlx5_txq_devx_obj_release(struct mlx5_txq_obj *txq_obj) { MLX5_ASSERT(txq_obj); - if (txq_obj->type == MLX5_TXQ_OBJ_TYPE_DEVX_HAIRPIN) { + if (txq_obj->txq_ctrl->type == MLX5_TXQ_TYPE_HAIRPIN) { if (txq_obj->tis) claim_zero(mlx5_devx_cmd_destroy(txq_obj->tis)); #ifdef HAVE_MLX5DV_DEVX_UAR_OFFSET From patchwork Thu Oct 1 14:09:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79483 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id F2A3CA04BA; Thu, 1 Oct 2020 16:16:18 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E87AC1DBDB; Thu, 1 Oct 2020 16:11:54 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 3C5541DB8F for ; Thu, 1 Oct 2020 16:11:53 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:11:48 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAEU012743; Thu, 1 Oct 2020 17:11:48 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:25 +0000 Message-Id: <1601561366-1821-15-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 14/15] net/mlx5: separate Rx queue state modification X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Separate Rx state modification to the Verbs and DevX modules. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/linux/mlx5_verbs.c | 9 +++++---- drivers/net/mlx5/mlx5.h | 9 ++++++++- drivers/net/mlx5/mlx5_devx.c | 25 ++++++++++++++++++++----- drivers/net/mlx5/mlx5_rxq.c | 4 ++-- drivers/net/mlx5/mlx5_rxtx.c | 27 +-------------------------- 5 files changed, 36 insertions(+), 38 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index 6260b4e..b4a6b5e 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -4,7 +4,6 @@ #include #include -#include #include #include #include @@ -97,16 +96,18 @@ * * @param rxq_obj * Verbs Rx queue object. + * @param type + * Type of change queue state. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ static int -mlx5_ibv_modify_wq(struct mlx5_rxq_obj *rxq_obj, bool is_start) +mlx5_ibv_modify_wq(struct mlx5_rxq_obj *rxq_obj, uint8_t type) { struct ibv_wq_attr mod = { .attr_mask = IBV_WQ_ATTR_STATE, - .wq_state = is_start ? IBV_WQS_RDY : IBV_WQS_RESET, + .wq_state = (enum ibv_wq_state)type, }; return mlx5_glue->modify_wq(rxq_obj->wq, &mod); @@ -418,7 +419,7 @@ goto error; } /* Change queue state to ready. */ - ret = mlx5_ibv_modify_wq(tmpl, true); + ret = mlx5_ibv_modify_wq(tmpl, IBV_WQS_RDY); if (ret) { DRV_LOG(ERR, "Port %u Rx queue %u WQ state to IBV_WQS_RDY failed.", diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index fd5dd87..f385b48 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -766,6 +766,13 @@ struct mlx5_txq_obj { }; }; +enum mlx5_rxq_modify_type { + MLX5_RXQ_MOD_ERR2RST, /* modify state from error to reset. */ + MLX5_RXQ_MOD_RST2RDY, /* modify state from reset to ready. */ + MLX5_RXQ_MOD_RDY2ERR, /* modify state from ready to error. */ + MLX5_RXQ_MOD_RDY2RST, /* modify state from ready to reset. */ +}; + enum mlx5_txq_modify_type { MLX5_TXQ_MOD_RDY2RDY, /* modify state from ready to ready. */ MLX5_TXQ_MOD_RST2RDY, /* modify state from reset to ready. */ @@ -778,7 +785,7 @@ struct mlx5_obj_ops { int (*rxq_obj_modify_vlan_strip)(struct mlx5_rxq_obj *rxq_obj, int on); int (*rxq_obj_new)(struct rte_eth_dev *dev, uint16_t idx); int (*rxq_event_get)(struct mlx5_rxq_obj *rxq_obj); - int (*rxq_obj_modify)(struct mlx5_rxq_obj *rxq_obj, bool is_start); + int (*rxq_obj_modify)(struct mlx5_rxq_obj *rxq_obj, uint8_t type); void (*rxq_obj_release)(struct mlx5_rxq_obj *rxq_obj); int (*ind_table_new)(struct rte_eth_dev *dev, const unsigned int log_n, struct mlx5_ind_table_obj *ind_tbl); diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 430ad08..a7c941c 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -52,22 +52,37 @@ * * @param rxq_obj * DevX Rx queue object. + * @param type + * Type of change queue state. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ static int -mlx5_devx_modify_rq(struct mlx5_rxq_obj *rxq_obj, bool is_start) +mlx5_devx_modify_rq(struct mlx5_rxq_obj *rxq_obj, uint8_t type) { struct mlx5_devx_modify_rq_attr rq_attr; memset(&rq_attr, 0, sizeof(rq_attr)); - if (is_start) { + switch (type) { + case MLX5_RXQ_MOD_ERR2RST: + rq_attr.rq_state = MLX5_RQC_STATE_ERR; + rq_attr.state = MLX5_RQC_STATE_RST; + break; + case MLX5_RXQ_MOD_RST2RDY: rq_attr.rq_state = MLX5_RQC_STATE_RST; rq_attr.state = MLX5_RQC_STATE_RDY; - } else { + break; + case MLX5_RXQ_MOD_RDY2ERR: + rq_attr.rq_state = MLX5_RQC_STATE_RDY; + rq_attr.state = MLX5_RQC_STATE_ERR; + break; + case MLX5_RXQ_MOD_RDY2RST: rq_attr.rq_state = MLX5_RQC_STATE_RDY; rq_attr.state = MLX5_RQC_STATE_RST; + break; + default: + break; } return mlx5_devx_cmd_modify_rq(rxq_obj->rq, &rq_attr); } @@ -194,7 +209,7 @@ MLX5_ASSERT(rxq_obj); MLX5_ASSERT(rxq_obj->rq); if (rxq_obj->type == MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN) { - mlx5_devx_modify_rq(rxq_obj, false); + mlx5_devx_modify_rq(rxq_obj, MLX5_RXQ_MOD_RDY2RST); claim_zero(mlx5_devx_cmd_destroy(rxq_obj->rq)); } else { MLX5_ASSERT(rxq_obj->devx_cq); @@ -628,7 +643,7 @@ goto error; } /* Change queue state to ready. */ - ret = mlx5_devx_modify_rq(tmpl, true); + ret = mlx5_devx_modify_rq(tmpl, MLX5_RXQ_MOD_RST2RDY); if (ret) goto error; rxq_data->cq_arm_sn = 0; diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index c059e21..f1d8373 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -513,7 +513,7 @@ int ret; MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY); - ret = priv->obj_ops.rxq_obj_modify(rxq_ctrl->obj, false); + ret = priv->obj_ops.rxq_obj_modify(rxq_ctrl->obj, MLX5_RXQ_MOD_RDY2RST); if (ret) { DRV_LOG(ERR, "Cannot change Rx WQ state to RESET: %s", strerror(errno)); @@ -612,7 +612,7 @@ /* Reset RQ consumer before moving queue ro READY state. */ *rxq->rq_db = rte_cpu_to_be_32(0); rte_io_wmb(); - ret = priv->obj_ops.rxq_obj_modify(rxq_ctrl->obj, true); + ret = priv->obj_ops.rxq_obj_modify(rxq_ctrl->obj, MLX5_RXQ_MOD_RST2RDY); if (ret) { DRV_LOG(ERR, "Cannot change Rx WQ state to READY: %s", strerror(errno)); diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index af4b4ba..dc22479 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -16,8 +16,6 @@ #include #include -#include -#include #include #include @@ -901,30 +899,7 @@ enum mlx5_txcmp_code { struct mlx5_rxq_ctrl *rxq_ctrl = container_of(rxq, struct mlx5_rxq_ctrl, rxq); - if (rxq_ctrl->obj->type == MLX5_RXQ_OBJ_TYPE_IBV) { - struct ibv_wq_attr mod = { - .attr_mask = IBV_WQ_ATTR_STATE, - .wq_state = sm->state, - }; - - ret = mlx5_glue->modify_wq(rxq_ctrl->obj->wq, &mod); - } else { /* rxq_ctrl->obj->type == MLX5_RXQ_OBJ_TYPE_DEVX_RQ. */ - struct mlx5_devx_modify_rq_attr rq_attr; - - memset(&rq_attr, 0, sizeof(rq_attr)); - if (sm->state == IBV_WQS_RESET) { - rq_attr.rq_state = MLX5_RQC_STATE_ERR; - rq_attr.state = MLX5_RQC_STATE_RST; - } else if (sm->state == IBV_WQS_RDY) { - rq_attr.rq_state = MLX5_RQC_STATE_RST; - rq_attr.state = MLX5_RQC_STATE_RDY; - } else if (sm->state == IBV_WQS_ERR) { - rq_attr.rq_state = MLX5_RQC_STATE_RDY; - rq_attr.state = MLX5_RQC_STATE_ERR; - } - ret = mlx5_devx_cmd_modify_rq(rxq_ctrl->obj->rq, - &rq_attr); - } + ret = priv->obj_ops.rxq_obj_modify(rxq_ctrl->obj, sm->state); if (ret) { DRV_LOG(ERR, "Cannot change Rx WQ state to %u - %s", sm->state, strerror(errno)); From patchwork Thu Oct 1 14:09:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 79484 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0237EA04BA; Thu, 1 Oct 2020 16:16:40 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 924711DC21; Thu, 1 Oct 2020 16:11:59 +0200 (CEST) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 4FF4F1DC21 for ; Thu, 1 Oct 2020 16:11:58 +0200 (CEST) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 1 Oct 2020 17:11:54 +0300 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 091EAAEV012743; Thu, 1 Oct 2020 17:11:54 +0300 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 1 Oct 2020 14:09:26 +0000 Message-Id: <1601561366-1821-16-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> References: <1601561366-1821-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH v1 15/15] net/mlx5: remove Rx queue object type field X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Once the separation between Verbs and DevX is done using function pointers, the type field of the Rx queue object structure becomes redundant and no more code is used. Remove the unnecessary field from the structure. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/linux/mlx5_verbs.c | 1 - drivers/net/mlx5/mlx5.h | 8 -------- drivers/net/mlx5/mlx5_devx.c | 4 +--- drivers/net/mlx5/mlx5_vlan.c | 5 ++--- 4 files changed, 3 insertions(+), 15 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c b/drivers/net/mlx5/linux/mlx5_verbs.c index b4a6b5e..494ddba 100644 --- a/drivers/net/mlx5/linux/mlx5_verbs.c +++ b/drivers/net/mlx5/linux/mlx5_verbs.c @@ -368,7 +368,6 @@ MLX5_ASSERT(tmpl); priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_RX_QUEUE; priv->verbs_alloc_ctx.obj = rxq_ctrl; - tmpl->type = MLX5_RXQ_OBJ_TYPE_IBV; tmpl->rxq_ctrl = rxq_ctrl; if (rxq_ctrl->irq) { tmpl->ibv_channel = diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index f385b48..87d3c15 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -676,18 +676,10 @@ struct mlx5_proc_priv { #define MLX5_PROC_PRIV(port_id) \ ((struct mlx5_proc_priv *)rte_eth_devices[port_id].process_private) -enum mlx5_rxq_obj_type { - MLX5_RXQ_OBJ_TYPE_IBV, /* mlx5_rxq_obj with ibv_wq. */ - MLX5_RXQ_OBJ_TYPE_DEVX_RQ, /* mlx5_rxq_obj with mlx5_devx_rq. */ - MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN, - /* mlx5_rxq_obj with mlx5_devx_rq and hairpin support. */ -}; - /* Verbs/DevX Rx queue elements. */ struct mlx5_rxq_obj { LIST_ENTRY(mlx5_rxq_obj) next; /* Pointer to the next element. */ struct mlx5_rxq_ctrl *rxq_ctrl; /* Back pointer to parent. */ - enum mlx5_rxq_obj_type type; int fd; /* File descriptor for event channel */ RTE_STD_C11 union { diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index a7c941c..11bda32 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -208,7 +208,7 @@ { MLX5_ASSERT(rxq_obj); MLX5_ASSERT(rxq_obj->rq); - if (rxq_obj->type == MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN) { + if (rxq_obj->rxq_ctrl->type == MLX5_RXQ_TYPE_HAIRPIN) { mlx5_devx_modify_rq(rxq_obj, MLX5_RXQ_MOD_RDY2RST); claim_zero(mlx5_devx_cmd_destroy(rxq_obj->rq)); } else { @@ -550,7 +550,6 @@ MLX5_ASSERT(rxq_data); MLX5_ASSERT(tmpl); - tmpl->type = MLX5_RXQ_OBJ_TYPE_DEVX_HAIRPIN; tmpl->rxq_ctrl = rxq_ctrl; attr.hairpin = 1; max_wq_data = priv->config.hca_attr.log_max_hairpin_wq_data_sz; @@ -611,7 +610,6 @@ MLX5_ASSERT(tmpl); if (rxq_ctrl->type == MLX5_RXQ_TYPE_HAIRPIN) return mlx5_rxq_obj_hairpin_new(dev, idx); - tmpl->type = MLX5_RXQ_OBJ_TYPE_DEVX_RQ; tmpl->rxq_ctrl = rxq_ctrl; if (rxq_ctrl->irq) { int devx_ev_flag = diff --git a/drivers/net/mlx5/mlx5_vlan.c b/drivers/net/mlx5/mlx5_vlan.c index 290503a..dbb9d36 100644 --- a/drivers/net/mlx5/mlx5_vlan.c +++ b/drivers/net/mlx5/mlx5_vlan.c @@ -116,9 +116,8 @@ } ret = priv->obj_ops.rxq_obj_modify_vlan_strip(rxq_ctrl->obj, on); if (ret) { - DRV_LOG(ERR, "port %u failed to modify object %d stripping " - "mode: %s", dev->data->port_id, - rxq_ctrl->obj->type, strerror(rte_errno)); + DRV_LOG(ERR, "Port %u failed to modify object stripping mode:" + " %s", dev->data->port_id, strerror(rte_errno)); return; } /* Update related bits in RX queue. */