From patchwork Thu Dec 17 11:44:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85311 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id BAF42A09F6; Thu, 17 Dec 2020 12:46:11 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 5EC35CA44; Thu, 17 Dec 2020 12:45:05 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 6E71BCA1A for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:50 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2Q004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko , stable@dpdk.org Date: Thu, 17 Dec 2020 11:44:19 +0000 Message-Id: <1608205475-20067-2-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 01/17] net/mlx5: fix ASO SQ creation error flow X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" In ASO SQ creation, the PMD allocates umem buffer for SQ. When umem buffer allocation is fails, the MR and CQ memory are not freed what caused a memory leak. Free it. Fixes: f935ed4b645a ("net/mlx5: support flow hit action for aging") Cc: stable@dpdk.org Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5_flow_age.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/net/mlx5/mlx5_flow_age.c b/drivers/net/mlx5/mlx5_flow_age.c index cea2cf7..0ea61be 100644 --- a/drivers/net/mlx5/mlx5_flow_age.c +++ b/drivers/net/mlx5/mlx5_flow_age.c @@ -278,7 +278,8 @@ sizeof(*sq->db_rec) * 2, 4096, socket); if (!sq->umem_buf) { DRV_LOG(ERR, "Can't allocate wqe buffer."); - return -ENOMEM; + rte_errno = ENOMEM; + goto error; } sq->wqe_umem = mlx5_glue->devx_umem_reg(ctx, (void *)(uintptr_t)sq->umem_buf, From patchwork Thu Dec 17 11:44:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85309 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id CF0F4A09F6; Thu, 17 Dec 2020 12:45:36 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1059FCA35; Thu, 17 Dec 2020 12:45:02 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 463F0CA0C for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:50 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2R004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:20 +0000 Message-Id: <1608205475-20067-3-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 02/17] common/mlx5: share DevX CQ creation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The CQ object in DevX is created in several places and in several different drivers. In all places almost all the details are the same, and in particular the allocations of the required resources. Add a structure that contains all the resources, and provide creation and release functions for it. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/common/mlx5/meson.build | 1 + drivers/common/mlx5/mlx5_common_devx.c | 157 +++++++++++++++++++++++++++++++++ drivers/common/mlx5/mlx5_common_devx.h | 28 ++++++ 3 files changed, 186 insertions(+) create mode 100644 drivers/common/mlx5/mlx5_common_devx.c create mode 100644 drivers/common/mlx5/mlx5_common_devx.h diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build index 3dacc6f..26cee06 100644 --- a/drivers/common/mlx5/meson.build +++ b/drivers/common/mlx5/meson.build @@ -16,6 +16,7 @@ sources += files( 'mlx5_common_mr.c', 'mlx5_malloc.c', 'mlx5_common_pci.c', + 'mlx5_common_devx.c', ) cflags_options = [ diff --git a/drivers/common/mlx5/mlx5_common_devx.c b/drivers/common/mlx5/mlx5_common_devx.c new file mode 100644 index 0000000..324c6ea --- /dev/null +++ b/drivers/common/mlx5/mlx5_common_devx.c @@ -0,0 +1,157 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ +#include + +#include +#include +#include + +#include +#include + +#include "mlx5_prm.h" +#include "mlx5_devx_cmds.h" +#include "mlx5_common_utils.h" +#include "mlx5_malloc.h" +#include "mlx5_common.h" +#include "mlx5_common_devx.h" + +/** + * Destroy DevX Completion Queue. + * + * @param[in] cq + * DevX CQ to destroy. + */ +void +mlx5_devx_cq_destroy(struct mlx5_devx_cq *cq) +{ + if (cq->cq) + claim_zero(mlx5_devx_cmd_destroy(cq->cq)); + if (cq->umem_obj) + claim_zero(mlx5_glue->devx_umem_dereg(cq->umem_obj)); + if (cq->umem_buf) + mlx5_free((void *)(uintptr_t)cq->umem_buf); +} + +/* Mark all CQEs initially as invalid. */ +static void +mlx5_cq_init(struct mlx5_devx_cq *cq_obj, uint16_t cq_size) +{ + volatile struct mlx5_cqe *cqe = cq_obj->cqes; + uint16_t i; + + for (i = 0; i < cq_size; i++, cqe++) + cqe->op_own = (MLX5_CQE_INVALID << 4) | MLX5_CQE_OWNER_MASK; +} + +/** + * Create Completion Queue using DevX API. + * + * Get a pointer to partially initialized attributes structure, and updates the + * following fields: + * q_umem_valid + * q_umem_id + * q_umem_offset + * db_umem_valid + * db_umem_id + * db_umem_offset + * eqn + * log_cq_size + * log_page_size + * All other fields are updated by caller. + * + * @param[in] ctx + * Context returned from mlx5 open_device() glue function. + * @param[in/out] cq_obj + * Pointer to CQ to create. + * @param[in] log_desc_n + * Log of number of descriptors in queue. + * @param[in] attr + * Pointer to CQ attributes structure. + * @param[in] socket + * Socket to use for allocation. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_devx_cq_create(void *ctx, struct mlx5_devx_cq *cq_obj, uint16_t log_desc_n, + struct mlx5_devx_cq_attr *attr, int socket) +{ + struct mlx5_devx_obj *cq = NULL; + struct mlx5dv_devx_umem *umem_obj = NULL; + void *umem_buf = NULL; + size_t page_size = rte_mem_page_size(); + size_t alignment = MLX5_CQE_BUF_ALIGNMENT; + uint32_t umem_size, umem_dbrec; + uint32_t eqn; + uint16_t cq_size = 1 << log_desc_n; + int ret; + + if (page_size == (size_t)-1 || alignment == (size_t)-1) { + DRV_LOG(ERR, "Failed to get page_size."); + rte_errno = ENOMEM; + return -rte_errno; + } + /* Query first EQN. */ + ret = mlx5_glue->devx_query_eqn(ctx, 0, &eqn); + if (ret) { + rte_errno = errno; + DRV_LOG(ERR, "Failed to query event queue number."); + goto error; + } + /* Allocate memory buffer for CQEs and doorbell record. */ + umem_size = sizeof(struct mlx5_cqe) * cq_size; + umem_dbrec = RTE_ALIGN(umem_size, MLX5_DBR_SIZE); + umem_size += MLX5_DBR_SIZE; + umem_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, umem_size, + alignment, socket); + if (!umem_buf) { + DRV_LOG(ERR, "Failed to allocate memory for CQ."); + rte_errno = ENOMEM; + return -rte_errno; + } + /* Register allocated buffer in user space with DevX. */ + umem_obj = mlx5_glue->devx_umem_reg(ctx, (void *)(uintptr_t)umem_buf, + umem_size, IBV_ACCESS_LOCAL_WRITE); + if (!umem_obj) { + DRV_LOG(ERR, "Failed to register umem for CQ."); + rte_errno = errno; + goto error; + } + /* Fill attributes for CQ object creation. */ + attr->q_umem_valid = 1; + attr->q_umem_id = mlx5_os_get_umem_id(umem_obj); + attr->q_umem_offset = 0; + attr->db_umem_valid = 1; + attr->db_umem_id = attr->q_umem_id; + attr->db_umem_offset = umem_dbrec; + attr->eqn = eqn; + attr->log_cq_size = log_desc_n; + attr->log_page_size = rte_log2_u32(page_size); + /* Create completion queue object with DevX. */ + cq = mlx5_devx_cmd_create_cq(ctx, attr); + if (!cq) { + DRV_LOG(ERR, "Can't create DevX CQ object."); + rte_errno = ENOMEM; + goto error; + } + cq_obj->umem_buf = umem_buf; + cq_obj->umem_obj = umem_obj; + cq_obj->cq = cq; + cq_obj->db_rec = RTE_PTR_ADD(cq_obj->umem_buf, umem_dbrec); + /* Mark all CQEs initially as invalid. */ + mlx5_cq_init(cq_obj, cq_size); + return 0; +error: + ret = rte_errno; + if (cq) + claim_zero(mlx5_devx_cmd_destroy(cq)); + if (umem_obj) + claim_zero(mlx5_glue->devx_umem_dereg(umem_obj)); + if (umem_buf) + mlx5_free((void *)(uintptr_t)umem_buf); + rte_errno = ret; + return -rte_errno; +} diff --git a/drivers/common/mlx5/mlx5_common_devx.h b/drivers/common/mlx5/mlx5_common_devx.h new file mode 100644 index 0000000..31cb804 --- /dev/null +++ b/drivers/common/mlx5/mlx5_common_devx.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#ifndef RTE_PMD_MLX5_COMMON_DEVX_H_ +#define RTE_PMD_MLX5_COMMON_DEVX_H_ + +#include "mlx5_devx_cmds.h" + +/* DevX Completion Queue structure. */ +struct mlx5_devx_cq { + struct mlx5_devx_obj *cq; /* The CQ DevX object. */ + struct mlx5dv_devx_umem *umem_obj; /* The CQ umem object. */ + union { + volatile void *umem_buf; + volatile struct mlx5_cqe *cqes; /* The CQ ring buffer. */ + }; + volatile uint32_t *db_rec; /* The CQ doorbell record. */ +}; + +/* mlx5_common_devx.c */ + +void mlx5_devx_cq_destroy(struct mlx5_devx_cq *cq); +int mlx5_devx_cq_create(void *ctx, struct mlx5_devx_cq *cq_obj, + uint16_t log_desc_n, struct mlx5_devx_cq_attr *attr, + int socket); + +#endif /* RTE_PMD_MLX5_COMMON_DEVX_H_ */ From patchwork Thu Dec 17 11:44:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85308 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id CF5F4A09F6; Thu, 17 Dec 2020 12:45:13 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0E620CA24; Thu, 17 Dec 2020 12:45:00 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 4FC50CA16 for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:50 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2S004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:21 +0000 Message-Id: <1608205475-20067-4-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 03/17] regex/mlx5: move DevX CQ creation to common X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Using common function for DevX CQ creation. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/regex/mlx5/mlx5_regex.c | 6 --- drivers/regex/mlx5/mlx5_regex.h | 9 +--- drivers/regex/mlx5/mlx5_regex_control.c | 91 ++++++-------------------------- drivers/regex/mlx5/mlx5_regex_fastpath.c | 4 +- 4 files changed, 20 insertions(+), 90 deletions(-) diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c index c91c444..c0d6331 100644 --- a/drivers/regex/mlx5/mlx5_regex.c +++ b/drivers/regex/mlx5/mlx5_regex.c @@ -170,12 +170,6 @@ rte_errno = rte_errno ? rte_errno : EINVAL; goto error; } - ret = mlx5_glue->devx_query_eqn(ctx, 0, &priv->eqn); - if (ret) { - DRV_LOG(ERR, "can't query event queue number."); - rte_errno = ENOMEM; - goto error; - } /* * This PMD always claims the write memory barrier on UAR * registers writings, it is safe to allocate UAR with any diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h index 2c4877c..9f7a388 100644 --- a/drivers/regex/mlx5/mlx5_regex.h +++ b/drivers/regex/mlx5/mlx5_regex.h @@ -12,6 +12,7 @@ #include #include +#include #include "mlx5_rxp.h" @@ -30,13 +31,8 @@ struct mlx5_regex_sq { struct mlx5_regex_cq { uint32_t log_nb_desc; /* Log 2 number of desc for this object. */ - struct mlx5_devx_obj *obj; /* The CQ DevX object. */ - int64_t dbr_offset; /* Door bell record offset. */ - uint32_t dbr_umem; /* Door bell record umem id. */ - volatile struct mlx5_cqe *cqe; /* The CQ ring buffer. */ - struct mlx5dv_devx_umem *cqe_umem; /* CQ buffer umem. */ + struct mlx5_devx_cq cq_obj; /* The CQ DevX object. */ size_t ci; - uint32_t *dbr; }; struct mlx5_regex_qp { @@ -75,7 +71,6 @@ struct mlx5_regex_priv { struct mlx5_regex_db db[MLX5_RXP_MAX_ENGINES + MLX5_RXP_EM_COUNT]; uint32_t nb_engines; /* Number of RegEx engines. */ - uint32_t eqn; /* EQ number. */ struct mlx5dv_devx_uar *uar; /* UAR object. */ struct ibv_pd *pd; struct mlx5_dbr_page_list dbrpgs; /* Door-bell pages. */ diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c index d6f452b..ca6c0f5 100644 --- a/drivers/regex/mlx5/mlx5_regex_control.c +++ b/drivers/regex/mlx5/mlx5_regex_control.c @@ -6,6 +6,7 @@ #include #include +#include #include #include #include @@ -17,6 +18,7 @@ #include #include #include +#include #include "mlx5_regex.h" #include "mlx5_regex_utils.h" @@ -44,8 +46,6 @@ /** * destroy CQ. * - * @param priv - * Pointer to the priv object. * @param cp * Pointer to the CQ to be destroyed. * @@ -53,24 +53,10 @@ * 0 on success, a negative errno value otherwise and rte_errno is set. */ static int -regex_ctrl_destroy_cq(struct mlx5_regex_priv *priv, struct mlx5_regex_cq *cq) +regex_ctrl_destroy_cq(struct mlx5_regex_cq *cq) { - if (cq->cqe_umem) { - mlx5_glue->devx_umem_dereg(cq->cqe_umem); - cq->cqe_umem = NULL; - } - if (cq->cqe) { - rte_free((void *)(uintptr_t)cq->cqe); - cq->cqe = NULL; - } - if (cq->dbr_offset) { - mlx5_release_dbr(&priv->dbrpgs, cq->dbr_umem, cq->dbr_offset); - cq->dbr_offset = -1; - } - if (cq->obj) { - mlx5_devx_cmd_destroy(cq->obj); - cq->obj = NULL; - } + mlx5_devx_cq_destroy(&cq->cq_obj); + memset(cq, 0, sizeof(*cq)); return 0; } @@ -89,65 +75,20 @@ regex_ctrl_create_cq(struct mlx5_regex_priv *priv, struct mlx5_regex_cq *cq) { struct mlx5_devx_cq_attr attr = { - .q_umem_valid = 1, - .db_umem_valid = 1, - .eqn = priv->eqn, + .uar_page_id = priv->uar->page_id, }; - struct mlx5_devx_dbr_page *dbr_page = NULL; - void *buf = NULL; - size_t pgsize = sysconf(_SC_PAGESIZE); - uint32_t cq_size = 1 << cq->log_nb_desc; - uint32_t i; - - cq->dbr_offset = mlx5_get_dbr(priv->ctx, &priv->dbrpgs, &dbr_page); - if (cq->dbr_offset < 0) { - DRV_LOG(ERR, "Can't allocate cq door bell record."); - rte_errno = ENOMEM; - goto error; - } - cq->dbr_umem = mlx5_os_get_umem_id(dbr_page->umem); - cq->dbr = (uint32_t *)((uintptr_t)dbr_page->dbrs + - (uintptr_t)cq->dbr_offset); + int ret; - buf = rte_calloc(NULL, 1, sizeof(struct mlx5_cqe) * cq_size, 4096); - if (!buf) { - DRV_LOG(ERR, "Can't allocate cqe buffer."); - rte_errno = ENOMEM; - goto error; - } - cq->cqe = buf; - for (i = 0; i < cq_size; i++) - cq->cqe[i].op_own = 0xff; - cq->cqe_umem = mlx5_glue->devx_umem_reg(priv->ctx, buf, - sizeof(struct mlx5_cqe) * - cq_size, 7); cq->ci = 0; - if (!cq->cqe_umem) { - DRV_LOG(ERR, "Can't register cqe mem."); - rte_errno = ENOMEM; - goto error; - } - attr.db_umem_offset = cq->dbr_offset; - attr.db_umem_id = cq->dbr_umem; - attr.q_umem_id = mlx5_os_get_umem_id(cq->cqe_umem); - attr.log_cq_size = cq->log_nb_desc; - attr.uar_page_id = priv->uar->page_id; - attr.log_page_size = rte_log2_u32(pgsize); - cq->obj = mlx5_devx_cmd_create_cq(priv->ctx, &attr); - if (!cq->obj) { - DRV_LOG(ERR, "Can't create cq object."); - rte_errno = ENOMEM; - goto error; + ret = mlx5_devx_cq_create(priv->ctx, &cq->cq_obj, cq->log_nb_desc, + &attr, SOCKET_ID_ANY); + if (ret) { + DRV_LOG(ERR, "Can't create CQ object."); + memset(cq, 0, sizeof(*cq)); + rte_errno = ENOMEM; + return -rte_errno; } return 0; -error: - if (cq->cqe_umem) - mlx5_glue->devx_umem_dereg(cq->cqe_umem); - if (buf) - rte_free(buf); - if (cq->dbr_offset) - mlx5_release_dbr(&priv->dbrpgs, cq->dbr_umem, cq->dbr_offset); - return -rte_errno; } #ifdef HAVE_IBV_FLOW_DV_SUPPORT @@ -232,7 +173,7 @@ attr.tis_lst_sz = 0; attr.tis_num = 0; attr.user_index = q_ind; - attr.cqn = qp->cq.obj->id; + attr.cqn = qp->cq.cq_obj.cq->id; wq_attr->uar_page = priv->uar->page_id; regex_get_pdn(priv->pd, &pd_num); wq_attr->pd = pd_num; @@ -389,7 +330,7 @@ err_btree: for (i = 0; i < nb_sq_config; i++) regex_ctrl_destroy_sq(priv, qp, i); - regex_ctrl_destroy_cq(priv, &qp->cq); + regex_ctrl_destroy_cq(&qp->cq); err_cq: rte_free(qp->sqs); return ret; diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c index 5857617..255fd40 100644 --- a/drivers/regex/mlx5/mlx5_regex_fastpath.c +++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c @@ -224,7 +224,7 @@ struct mlx5_regex_job { size_t next_cqe_offset; next_cqe_offset = (cq->ci & (cq_size_get(cq) - 1)); - cqe = (volatile struct mlx5_cqe *)(cq->cqe + next_cqe_offset); + cqe = (volatile struct mlx5_cqe *)(cq->cq_obj.cqes + next_cqe_offset); rte_io_wmb(); int ret = check_cqe(cqe, cq_size_get(cq), cq->ci); @@ -285,7 +285,7 @@ struct mlx5_regex_job { } cq->ci = (cq->ci + 1) & 0xffffff; rte_wmb(); - cq->dbr[0] = rte_cpu_to_be_32(cq->ci); + cq->cq_obj.db_rec[0] = rte_cpu_to_be_32(cq->ci); queue->free_sqs |= (1 << sqid); } From patchwork Thu Dec 17 11:44:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85310 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id BB723A09F6; Thu, 17 Dec 2020 12:45:57 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id CAC29CA3A; Thu, 17 Dec 2020 12:45:03 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 4AE44CA12 for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:50 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2T004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:22 +0000 Message-Id: <1608205475-20067-5-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 04/17] vdpa/mlx5: move DevX CQ creation to common X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Using common function for DevX CQ creation. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/vdpa/mlx5/mlx5_vdpa.h | 10 +---- drivers/vdpa/mlx5/mlx5_vdpa_event.c | 81 +++++++++++-------------------------- drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 2 +- 3 files changed, 26 insertions(+), 67 deletions(-) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index d039ada..ddee9dc 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -22,6 +22,7 @@ #include #include +#include #include @@ -46,13 +47,7 @@ struct mlx5_vdpa_cq { uint32_t armed:1; int callfd; rte_spinlock_t sl; - struct mlx5_devx_obj *cq; - struct mlx5dv_devx_umem *umem_obj; - union { - volatile void *umem_buf; - volatile struct mlx5_cqe *cqes; - }; - volatile uint32_t *db_rec; + struct mlx5_devx_cq cq_obj; uint64_t errors; }; @@ -144,7 +139,6 @@ struct mlx5_vdpa_priv { uint32_t gpa_mkey_index; struct ibv_mr *null_mr; struct rte_vhost_memory *vmem; - uint32_t eqn; struct mlx5dv_devx_event_channel *eventc; struct mlx5dv_devx_event_channel *err_chnl; struct mlx5dv_devx_uar *uar; diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c index 3aeaeb8..ef92338 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -15,6 +16,7 @@ #include #include +#include #include #include "mlx5_vdpa_utils.h" @@ -47,7 +49,6 @@ priv->eventc = NULL; } #endif - priv->eqn = 0; } /* Prepare all the global resources for all the event objects.*/ @@ -58,11 +59,6 @@ if (priv->eventc) return 0; - if (mlx5_glue->devx_query_eqn(priv->ctx, 0, &priv->eqn)) { - rte_errno = errno; - DRV_LOG(ERR, "Failed to query EQ number %d.", rte_errno); - return -1; - } priv->eventc = mlx5_glue->devx_create_event_channel(priv->ctx, MLX5DV_DEVX_CREATE_EVENT_CHANNEL_FLAGS_OMIT_EV_DATA); if (!priv->eventc) { @@ -97,12 +93,7 @@ static void mlx5_vdpa_cq_destroy(struct mlx5_vdpa_cq *cq) { - if (cq->cq) - claim_zero(mlx5_devx_cmd_destroy(cq->cq)); - if (cq->umem_obj) - claim_zero(mlx5_glue->devx_umem_dereg(cq->umem_obj)); - if (cq->umem_buf) - rte_free((void *)(uintptr_t)cq->umem_buf); + mlx5_devx_cq_destroy(&cq->cq_obj); memset(cq, 0, sizeof(*cq)); } @@ -112,12 +103,12 @@ uint32_t arm_sn = cq->arm_sn << MLX5_CQ_SQN_OFFSET; uint32_t cq_ci = cq->cq_ci & MLX5_CI_MASK; uint32_t doorbell_hi = arm_sn | MLX5_CQ_DBR_CMD_ALL | cq_ci; - uint64_t doorbell = ((uint64_t)doorbell_hi << 32) | cq->cq->id; + uint64_t doorbell = ((uint64_t)doorbell_hi << 32) | cq->cq_obj.cq->id; uint64_t db_be = rte_cpu_to_be_64(doorbell); uint32_t *addr = RTE_PTR_ADD(priv->uar->base_addr, MLX5_CQ_DOORBELL); rte_io_wmb(); - cq->db_rec[MLX5_CQ_ARM_DB] = rte_cpu_to_be_32(doorbell_hi); + cq->cq_obj.db_rec[MLX5_CQ_ARM_DB] = rte_cpu_to_be_32(doorbell_hi); rte_wmb(); #ifdef RTE_ARCH_64 *(uint64_t *)addr = db_be; @@ -134,49 +125,23 @@ mlx5_vdpa_cq_create(struct mlx5_vdpa_priv *priv, uint16_t log_desc_n, int callfd, struct mlx5_vdpa_cq *cq) { - struct mlx5_devx_cq_attr attr = {0}; - size_t pgsize = sysconf(_SC_PAGESIZE); - uint32_t umem_size; + struct mlx5_devx_cq_attr attr = { + .use_first_only = 1, + .uar_page_id = priv->uar->page_id, + }; uint16_t event_nums[1] = {0}; - uint16_t cq_size = 1 << log_desc_n; int ret; - cq->log_desc_n = log_desc_n; - umem_size = sizeof(struct mlx5_cqe) * cq_size + sizeof(*cq->db_rec) * 2; - cq->umem_buf = rte_zmalloc(__func__, umem_size, 4096); - if (!cq->umem_buf) { - DRV_LOG(ERR, "Failed to allocate memory for CQ."); - rte_errno = ENOMEM; - return -ENOMEM; - } - cq->umem_obj = mlx5_glue->devx_umem_reg(priv->ctx, - (void *)(uintptr_t)cq->umem_buf, - umem_size, - IBV_ACCESS_LOCAL_WRITE); - if (!cq->umem_obj) { - DRV_LOG(ERR, "Failed to register umem for CQ."); - goto error; - } - attr.q_umem_valid = 1; - attr.db_umem_valid = 1; - attr.use_first_only = 1; - attr.overrun_ignore = 0; - attr.uar_page_id = priv->uar->page_id; - attr.q_umem_id = cq->umem_obj->umem_id; - attr.q_umem_offset = 0; - attr.db_umem_id = cq->umem_obj->umem_id; - attr.db_umem_offset = sizeof(struct mlx5_cqe) * cq_size; - attr.eqn = priv->eqn; - attr.log_cq_size = log_desc_n; - attr.log_page_size = rte_log2_u32(pgsize); - cq->cq = mlx5_devx_cmd_create_cq(priv->ctx, &attr); - if (!cq->cq) + ret = mlx5_devx_cq_create(priv->ctx, &cq->cq_obj, log_desc_n, &attr, + SOCKET_ID_ANY); + if (ret) goto error; - cq->db_rec = RTE_PTR_ADD(cq->umem_buf, (uintptr_t)attr.db_umem_offset); cq->cq_ci = 0; + cq->log_desc_n = log_desc_n; rte_spinlock_init(&cq->sl); /* Subscribe CQ event to the event channel controlled by the driver. */ - ret = mlx5_glue->devx_subscribe_devx_event(priv->eventc, cq->cq->obj, + ret = mlx5_glue->devx_subscribe_devx_event(priv->eventc, + cq->cq_obj.cq->obj, sizeof(event_nums), event_nums, (uint64_t)(uintptr_t)cq); @@ -187,8 +152,8 @@ } cq->callfd = callfd; /* Init CQ to ones to be in HW owner in the start. */ - cq->cqes[0].op_own = MLX5_CQE_OWNER_MASK; - cq->cqes[0].wqe_counter = rte_cpu_to_be_16(UINT16_MAX); + cq->cq_obj.cqes[0].op_own = MLX5_CQE_OWNER_MASK; + cq->cq_obj.cqes[0].wqe_counter = rte_cpu_to_be_16(UINT16_MAX); /* First arming. */ mlx5_vdpa_cq_arm(priv, cq); return 0; @@ -215,7 +180,7 @@ uint16_t cur_wqe_counter; uint16_t comp; - last_word.word = rte_read32(&cq->cqes[0].wqe_counter); + last_word.word = rte_read32(&cq->cq_obj.cqes[0].wqe_counter); cur_wqe_counter = rte_be_to_cpu_16(last_word.wqe_counter); comp = cur_wqe_counter + (uint16_t)1 - next_wqe_counter; if (comp) { @@ -229,7 +194,7 @@ cq->errors++; rte_io_wmb(); /* Ring CQ doorbell record. */ - cq->db_rec[0] = rte_cpu_to_be_32(cq->cq_ci); + cq->cq_obj.db_rec[0] = rte_cpu_to_be_32(cq->cq_ci); rte_io_wmb(); /* Ring SW QP doorbell record. */ eqp->db_rec[0] = rte_cpu_to_be_32(cq->cq_ci + cq_size); @@ -245,7 +210,7 @@ for (i = 0; i < priv->nr_virtqs; i++) { cq = &priv->virtqs[i].eqp.cq; - if (cq->cq && !cq->armed) + if (cq->cq_obj.cq && !cq->armed) mlx5_vdpa_cq_arm(priv, cq); } } @@ -290,7 +255,7 @@ pthread_mutex_lock(&priv->vq_config_lock); for (i = 0; i < priv->nr_virtqs; i++) { cq = &priv->virtqs[i].eqp.cq; - if (cq->cq && !cq->armed) { + if (cq->cq_obj.cq && !cq->armed) { uint32_t comp = mlx5_vdpa_cq_poll(cq); if (comp) { @@ -369,7 +334,7 @@ DRV_LOG(DEBUG, "Device %s virtq %d cq %d event was captured." " Timer is %s, cq ci is %u.\n", priv->vdev->device->name, - (int)virtq->index, cq->cq->id, + (int)virtq->index, cq->cq_obj.cq->id, priv->timer_on ? "on" : "off", cq->cq_ci); cq->armed = 0; } @@ -679,7 +644,7 @@ goto error; } attr.uar_index = priv->uar->page_id; - attr.cqn = eqp->cq.cq->id; + attr.cqn = eqp->cq.cq_obj.cq->id; attr.log_page_size = rte_log2_u32(sysconf(_SC_PAGESIZE)); attr.rq_size = 1 << log_desc_n; attr.log_rq_stride = rte_log2_u32(MLX5_WSEG_SIZE); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c index 3e882e4..cc77314 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa_virtq.c @@ -497,7 +497,7 @@ return -1; if (vq.size != virtq->vq_size || vq.kickfd != virtq->intr_handle.fd) return 1; - if (virtq->eqp.cq.cq) { + if (virtq->eqp.cq.cq_obj.cq) { if (vq.callfd != virtq->eqp.cq.callfd) return 1; } else if (vq.callfd != -1) { From patchwork Thu Dec 17 11:44:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85307 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 36F1BA09F6; Thu, 17 Dec 2020 12:45:00 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 25B50CA0C; Thu, 17 Dec 2020 12:44:58 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 3BF5CCA08 for ; Thu, 17 Dec 2020 12:44:55 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:50 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2U004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:23 +0000 Message-Id: <1608205475-20067-6-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 05/17] net/mlx5: move rearm and clock queue CQ creation to common X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Using common function for CQ creation at rearm queue and clock queue. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5.h | 9 +-- drivers/net/mlx5/mlx5_rxtx.c | 2 +- drivers/net/mlx5/mlx5_txpp.c | 147 +++++++++++-------------------------------- 3 files changed, 40 insertions(+), 118 deletions(-) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 121d726..00ccaee 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -26,6 +26,7 @@ #include #include #include +#include #include "mlx5_defs.h" #include "mlx5_utils.h" @@ -612,13 +613,7 @@ struct mlx5_flow_id_pool { /* Tx pacing queue structure - for Clock and Rearm queues. */ struct mlx5_txpp_wq { /* Completion Queue related data.*/ - struct mlx5_devx_obj *cq; - void *cq_umem; - union { - volatile void *cq_buf; - volatile struct mlx5_cqe *cqes; - }; - volatile uint32_t *cq_dbrec; + struct mlx5_devx_cq cq_obj; uint32_t cq_ci:24; uint32_t arm_sn:2; /* Send Queue related data.*/ diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index d12d746..dad24a3 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -2277,7 +2277,7 @@ enum mlx5_txcmp_code { qs = RTE_PTR_ADD(wqe, MLX5_WSEG_SIZE); qs->max_index = rte_cpu_to_be_32(wci); - qs->qpn_cqn = rte_cpu_to_be_32(txq->sh->txpp.clock_queue.cq->id); + qs->qpn_cqn = rte_cpu_to_be_32(txq->sh->txpp.clock_queue.cq_obj.cq->id); qs->reserved0 = RTE_BE32(0); qs->reserved1 = RTE_BE32(0); } diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c index 2438bf1..54ea572 100644 --- a/drivers/net/mlx5/mlx5_txpp.c +++ b/drivers/net/mlx5/mlx5_txpp.c @@ -13,6 +13,7 @@ #include #include +#include #include "mlx5.h" #include "mlx5_rxtx.h" @@ -126,12 +127,7 @@ claim_zero(mlx5_glue->devx_umem_dereg(wq->sq_umem)); if (wq->sq_buf) mlx5_free((void *)(uintptr_t)wq->sq_buf); - if (wq->cq) - claim_zero(mlx5_devx_cmd_destroy(wq->cq)); - if (wq->cq_umem) - claim_zero(mlx5_glue->devx_umem_dereg(wq->cq_umem)); - if (wq->cq_buf) - mlx5_free((void *)(uintptr_t)wq->cq_buf); + mlx5_devx_cq_destroy(&wq->cq_obj); memset(wq, 0, sizeof(*wq)); } @@ -181,19 +177,6 @@ } static void -mlx5_txpp_fill_cqe_rearm_queue(struct mlx5_dev_ctx_shared *sh) -{ - struct mlx5_txpp_wq *wq = &sh->txpp.rearm_queue; - struct mlx5_cqe *cqe = (struct mlx5_cqe *)(uintptr_t)wq->cqes; - uint32_t i; - - for (i = 0; i < MLX5_TXPP_REARM_CQ_SIZE; i++) { - cqe->op_own = (MLX5_CQE_INVALID << 4) | MLX5_CQE_OWNER_MASK; - ++cqe; - } -} - -static void mlx5_txpp_fill_wqe_rearm_queue(struct mlx5_dev_ctx_shared *sh) { struct mlx5_txpp_wq *wq = &sh->txpp.rearm_queue; @@ -228,7 +211,8 @@ index = (i * MLX5_TXPP_REARM / 2 + MLX5_TXPP_REARM / 2) & ((1 << MLX5_CQ_INDEX_WIDTH) - 1); qs->max_index = rte_cpu_to_be_32(index); - qs->qpn_cqn = rte_cpu_to_be_32(sh->txpp.clock_queue.cq->id); + qs->qpn_cqn = + rte_cpu_to_be_32(sh->txpp.clock_queue.cq_obj.cq->id); } } @@ -238,7 +222,11 @@ { struct mlx5_devx_create_sq_attr sq_attr = { 0 }; struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; - struct mlx5_devx_cq_attr cq_attr = { 0 }; + struct mlx5_devx_cq_attr cq_attr = { + .cqe_size = (sizeof(struct mlx5_cqe) == 128) ? + MLX5_CQE_SIZE_128B : MLX5_CQE_SIZE_64B, + .uar_page_id = mlx5_os_get_devx_uar_page_id(sh->tx_uar), + }; struct mlx5_txpp_wq *wq = &sh->txpp.rearm_queue; size_t page_size; uint32_t umem_size, umem_dbrec; @@ -249,50 +237,16 @@ DRV_LOG(ERR, "Failed to get mem page size"); return -ENOMEM; } - /* Allocate memory buffer for CQEs and doorbell record. */ - umem_size = sizeof(struct mlx5_cqe) * MLX5_TXPP_REARM_CQ_SIZE; - umem_dbrec = RTE_ALIGN(umem_size, MLX5_DBR_SIZE); - umem_size += MLX5_DBR_SIZE; - wq->cq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, umem_size, - page_size, sh->numa_node); - if (!wq->cq_buf) { - DRV_LOG(ERR, "Failed to allocate memory for Rearm Queue."); - return -ENOMEM; - } - /* Register allocated buffer in user space with DevX. */ - wq->cq_umem = mlx5_glue->devx_umem_reg(sh->ctx, - (void *)(uintptr_t)wq->cq_buf, - umem_size, - IBV_ACCESS_LOCAL_WRITE); - if (!wq->cq_umem) { - rte_errno = errno; - DRV_LOG(ERR, "Failed to register umem for Rearm Queue."); - goto error; - } /* Create completion queue object for Rearm Queue. */ - cq_attr.cqe_size = (sizeof(struct mlx5_cqe) == 128) ? - MLX5_CQE_SIZE_128B : MLX5_CQE_SIZE_64B; - cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(sh->tx_uar); - cq_attr.eqn = sh->eqn; - cq_attr.q_umem_valid = 1; - cq_attr.q_umem_offset = 0; - cq_attr.q_umem_id = mlx5_os_get_umem_id(wq->cq_umem); - cq_attr.db_umem_valid = 1; - cq_attr.db_umem_offset = umem_dbrec; - cq_attr.db_umem_id = mlx5_os_get_umem_id(wq->cq_umem); - cq_attr.log_cq_size = rte_log2_u32(MLX5_TXPP_REARM_CQ_SIZE); - cq_attr.log_page_size = rte_log2_u32(page_size); - wq->cq = mlx5_devx_cmd_create_cq(sh->ctx, &cq_attr); - if (!wq->cq) { - rte_errno = errno; + ret = mlx5_devx_cq_create(sh->ctx, &wq->cq_obj, + log2above(MLX5_TXPP_REARM_CQ_SIZE), &cq_attr, + sh->numa_node); + if (ret) { DRV_LOG(ERR, "Failed to create CQ for Rearm Queue."); - goto error; + return ret; } - wq->cq_dbrec = RTE_PTR_ADD(wq->cq_buf, umem_dbrec); wq->cq_ci = 0; wq->arm_sn = 0; - /* Mark all CQEs initially as invalid. */ - mlx5_txpp_fill_cqe_rearm_queue(sh); /* * Allocate memory buffer for Send Queue WQEs. * There should be no WQE leftovers in the cyclic queue. @@ -323,7 +277,7 @@ sq_attr.state = MLX5_SQC_STATE_RST; sq_attr.tis_lst_sz = 1; sq_attr.tis_num = sh->tis->id; - sq_attr.cqn = wq->cq->id; + sq_attr.cqn = wq->cq_obj.cq->id; sq_attr.cd_master = 1; sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar); sq_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC; @@ -466,7 +420,13 @@ { struct mlx5_devx_create_sq_attr sq_attr = { 0 }; struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; - struct mlx5_devx_cq_attr cq_attr = { 0 }; + struct mlx5_devx_cq_attr cq_attr = { + .cqe_size = (sizeof(struct mlx5_cqe) == 128) ? + MLX5_CQE_SIZE_128B : MLX5_CQE_SIZE_64B, + .use_first_only = 1, + .overrun_ignore = 1, + .uar_page_id = mlx5_os_get_devx_uar_page_id(sh->tx_uar), + }; struct mlx5_txpp_wq *wq = &sh->txpp.clock_queue; size_t page_size; uint32_t umem_size, umem_dbrec; @@ -487,48 +447,14 @@ } sh->txpp.ts_p = 0; sh->txpp.ts_n = 0; - /* Allocate memory buffer for CQEs and doorbell record. */ - umem_size = sizeof(struct mlx5_cqe) * MLX5_TXPP_CLKQ_SIZE; - umem_dbrec = RTE_ALIGN(umem_size, MLX5_DBR_SIZE); - umem_size += MLX5_DBR_SIZE; - wq->cq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, umem_size, - page_size, sh->numa_node); - if (!wq->cq_buf) { - DRV_LOG(ERR, "Failed to allocate memory for Clock Queue."); - return -ENOMEM; - } - /* Register allocated buffer in user space with DevX. */ - wq->cq_umem = mlx5_glue->devx_umem_reg(sh->ctx, - (void *)(uintptr_t)wq->cq_buf, - umem_size, - IBV_ACCESS_LOCAL_WRITE); - if (!wq->cq_umem) { - rte_errno = errno; - DRV_LOG(ERR, "Failed to register umem for Clock Queue."); - goto error; - } /* Create completion queue object for Clock Queue. */ - cq_attr.cqe_size = (sizeof(struct mlx5_cqe) == 128) ? - MLX5_CQE_SIZE_128B : MLX5_CQE_SIZE_64B; - cq_attr.use_first_only = 1; - cq_attr.overrun_ignore = 1; - cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(sh->tx_uar); - cq_attr.eqn = sh->eqn; - cq_attr.q_umem_valid = 1; - cq_attr.q_umem_offset = 0; - cq_attr.q_umem_id = mlx5_os_get_umem_id(wq->cq_umem); - cq_attr.db_umem_valid = 1; - cq_attr.db_umem_offset = umem_dbrec; - cq_attr.db_umem_id = mlx5_os_get_umem_id(wq->cq_umem); - cq_attr.log_cq_size = rte_log2_u32(MLX5_TXPP_CLKQ_SIZE); - cq_attr.log_page_size = rte_log2_u32(page_size); - wq->cq = mlx5_devx_cmd_create_cq(sh->ctx, &cq_attr); - if (!wq->cq) { - rte_errno = errno; + ret = mlx5_devx_cq_create(sh->ctx, &wq->cq_obj, + log2above(MLX5_TXPP_CLKQ_SIZE), &cq_attr, + sh->numa_node); + if (ret) { DRV_LOG(ERR, "Failed to create CQ for Clock Queue."); goto error; } - wq->cq_dbrec = RTE_PTR_ADD(wq->cq_buf, umem_dbrec); wq->cq_ci = 0; /* Allocate memory buffer for Send Queue WQEs. */ if (sh->txpp.test) { @@ -574,7 +500,7 @@ sq_attr.static_sq_wq = 1; } sq_attr.state = MLX5_SQC_STATE_RST; - sq_attr.cqn = wq->cq->id; + sq_attr.cqn = wq->cq_obj.cq->id; sq_attr.packet_pacing_rate_limit_index = sh->txpp.pp_id; sq_attr.wq_attr.cd_slave = 1; sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar); @@ -625,12 +551,13 @@ struct mlx5_txpp_wq *aq = &sh->txpp.rearm_queue; uint32_t arm_sn = aq->arm_sn << MLX5_CQ_SQN_OFFSET; uint32_t db_hi = arm_sn | MLX5_CQ_DBR_CMD_ALL | aq->cq_ci; - uint64_t db_be = rte_cpu_to_be_64(((uint64_t)db_hi << 32) | aq->cq->id); + uint64_t db_be = + rte_cpu_to_be_64(((uint64_t)db_hi << 32) | aq->cq_obj.cq->id); base_addr = mlx5_os_get_devx_uar_base_addr(sh->tx_uar); uint32_t *addr = RTE_PTR_ADD(base_addr, MLX5_CQ_DOORBELL); rte_compiler_barrier(); - aq->cq_dbrec[MLX5_CQ_ARM_DB] = rte_cpu_to_be_32(db_hi); + aq->cq_obj.db_rec[MLX5_CQ_ARM_DB] = rte_cpu_to_be_32(db_hi); rte_wmb(); #ifdef RTE_ARCH_64 *(uint64_t *)addr = db_be; @@ -728,7 +655,7 @@ mlx5_txpp_update_timestamp(struct mlx5_dev_ctx_shared *sh) { struct mlx5_txpp_wq *wq = &sh->txpp.clock_queue; - struct mlx5_cqe *cqe = (struct mlx5_cqe *)(uintptr_t)wq->cqes; + struct mlx5_cqe *cqe = (struct mlx5_cqe *)(uintptr_t)wq->cq_obj.cqes; union { rte_int128_t u128; struct mlx5_cqe_ts cts; @@ -809,7 +736,7 @@ do { volatile struct mlx5_cqe *cqe; - cqe = &wq->cqes[cq_ci & (MLX5_TXPP_REARM_CQ_SIZE - 1)]; + cqe = &wq->cq_obj.cqes[cq_ci & (MLX5_TXPP_REARM_CQ_SIZE - 1)]; ret = check_cqe(cqe, MLX5_TXPP_REARM_CQ_SIZE, cq_ci); switch (ret) { case MLX5_CQE_STATUS_ERR: @@ -841,7 +768,7 @@ } /* Update doorbell record to notify hardware. */ rte_compiler_barrier(); - *wq->cq_dbrec = rte_cpu_to_be_32(cq_ci); + *wq->cq_obj.db_rec = rte_cpu_to_be_32(cq_ci); rte_wmb(); wq->cq_ci = cq_ci; /* Fire new requests to Rearm Queue. */ @@ -936,9 +863,8 @@ } /* Subscribe CQ event to the event channel controlled by the driver. */ ret = mlx5_glue->devx_subscribe_devx_event(sh->txpp.echan, - sh->txpp.rearm_queue.cq->obj, - sizeof(event_nums), - event_nums, 0); + sh->txpp.rearm_queue.cq_obj.cq->obj, + sizeof(event_nums), event_nums, 0); if (ret) { DRV_LOG(ERR, "Failed to subscribe CQE event."); rte_errno = errno; @@ -1140,7 +1066,8 @@ if (sh->txpp.refcnt) { struct mlx5_txpp_wq *wq = &sh->txpp.clock_queue; - struct mlx5_cqe *cqe = (struct mlx5_cqe *)(uintptr_t)wq->cqes; + struct mlx5_cqe *cqe = + (struct mlx5_cqe *)(uintptr_t)wq->cq_obj.cqes; union { rte_int128_t u128; struct mlx5_cqe_ts cts; From patchwork Thu Dec 17 11:44:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85313 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 22F75A09F6; Thu, 17 Dec 2020 12:46:50 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 56244CA52; Thu, 17 Dec 2020 12:45:08 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id AB804CA24 for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:50 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2V004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:24 +0000 Message-Id: <1608205475-20067-7-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 06/17] net/mlx5: move ASO CQ creation to common X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Use common function for ASO CQ creation. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5.h | 8 +--- drivers/net/mlx5/mlx5_flow_age.c | 81 +++++++++------------------------------- 2 files changed, 19 insertions(+), 70 deletions(-) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 00ccaee..e02faed 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -463,13 +463,7 @@ struct mlx5_flow_counter_mng { struct mlx5_aso_cq { uint16_t log_desc_n; uint32_t cq_ci:24; - struct mlx5_devx_obj *cq; - struct mlx5dv_devx_umem *umem_obj; - union { - volatile void *umem_buf; - volatile struct mlx5_cqe *cqes; - }; - volatile uint32_t *db_rec; + struct mlx5_devx_cq cq_obj; uint64_t errors; }; diff --git a/drivers/net/mlx5/mlx5_flow_age.c b/drivers/net/mlx5/mlx5_flow_age.c index 0ea61be..60a8d2a 100644 --- a/drivers/net/mlx5/mlx5_flow_age.c +++ b/drivers/net/mlx5/mlx5_flow_age.c @@ -7,10 +7,12 @@ #include #include +#include #include "mlx5.h" #include "mlx5_flow.h" + /** * Destroy Completion Queue used for ASO access. * @@ -20,12 +22,8 @@ static void mlx5_aso_cq_destroy(struct mlx5_aso_cq *cq) { - if (cq->cq) - claim_zero(mlx5_devx_cmd_destroy(cq->cq)); - if (cq->umem_obj) - claim_zero(mlx5_glue->devx_umem_dereg(cq->umem_obj)); - if (cq->umem_buf) - mlx5_free((void *)(uintptr_t)cq->umem_buf); + if (cq->cq_obj.cq) + mlx5_devx_cq_destroy(&cq->cq_obj); memset(cq, 0, sizeof(*cq)); } @@ -42,60 +40,21 @@ * Socket to use for allocation. * @param[in] uar_page_id * UAR page ID to use. - * @param[in] eqn - * EQ number. * * @return * 0 on success, a negative errno value otherwise and rte_errno is set. */ static int mlx5_aso_cq_create(void *ctx, struct mlx5_aso_cq *cq, uint16_t log_desc_n, - int socket, int uar_page_id, uint32_t eqn) + int socket, int uar_page_id) { - struct mlx5_devx_cq_attr attr = { 0 }; - size_t pgsize = sysconf(_SC_PAGESIZE); - uint32_t umem_size; - uint16_t cq_size = 1 << log_desc_n; + struct mlx5_devx_cq_attr attr = { + .uar_page_id = uar_page_id, + }; cq->log_desc_n = log_desc_n; - umem_size = sizeof(struct mlx5_cqe) * cq_size + sizeof(*cq->db_rec) * 2; - cq->umem_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, umem_size, - 4096, socket); - if (!cq->umem_buf) { - DRV_LOG(ERR, "Failed to allocate memory for CQ."); - rte_errno = ENOMEM; - return -ENOMEM; - } - cq->umem_obj = mlx5_glue->devx_umem_reg(ctx, - (void *)(uintptr_t)cq->umem_buf, - umem_size, - IBV_ACCESS_LOCAL_WRITE); - if (!cq->umem_obj) { - DRV_LOG(ERR, "Failed to register umem for aso CQ."); - goto error; - } - attr.q_umem_valid = 1; - attr.db_umem_valid = 1; - attr.use_first_only = 0; - attr.overrun_ignore = 0; - attr.uar_page_id = uar_page_id; - attr.q_umem_id = mlx5_os_get_umem_id(cq->umem_obj); - attr.q_umem_offset = 0; - attr.db_umem_id = attr.q_umem_id; - attr.db_umem_offset = sizeof(struct mlx5_cqe) * cq_size; - attr.eqn = eqn; - attr.log_cq_size = log_desc_n; - attr.log_page_size = rte_log2_u32(pgsize); - cq->cq = mlx5_devx_cmd_create_cq(ctx, &attr); - if (!cq->cq) - goto error; - cq->db_rec = RTE_PTR_ADD(cq->umem_buf, (uintptr_t)attr.db_umem_offset); cq->cq_ci = 0; - memset((void *)(uintptr_t)cq->umem_buf, 0xFF, attr.db_umem_offset); - return 0; -error: - mlx5_aso_cq_destroy(cq); - return -1; + return mlx5_devx_cq_create(ctx, &cq->cq_obj, log_desc_n, &attr, socket); } /** @@ -194,8 +153,7 @@ mlx5_devx_cmd_destroy(sq->sq); sq->sq = NULL; } - if (sq->cq.cq) - mlx5_aso_cq_destroy(&sq->cq); + mlx5_aso_cq_destroy(&sq->cq); mlx5_aso_devx_dereg_mr(&sq->mr); memset(sq, 0, sizeof(*sq)); } @@ -246,8 +204,6 @@ * User Access Region object. * @param[in] pdn * Protection Domain number to use. - * @param[in] eqn - * EQ number. * @param[in] log_desc_n * Log of number of descriptors in queue. * @@ -257,7 +213,7 @@ static int mlx5_aso_sq_create(void *ctx, struct mlx5_aso_sq *sq, int socket, struct mlx5dv_devx_uar *uar, uint32_t pdn, - uint32_t eqn, uint16_t log_desc_n) + uint16_t log_desc_n) { struct mlx5_devx_create_sq_attr attr = { 0 }; struct mlx5_devx_modify_sq_attr modify_attr = { 0 }; @@ -271,7 +227,7 @@ sq_desc_n, &sq->mr, socket, pdn)) return -1; if (mlx5_aso_cq_create(ctx, &sq->cq, log_desc_n, socket, - mlx5_os_get_devx_uar_page_id(uar), eqn)) + mlx5_os_get_devx_uar_page_id(uar))) goto error; sq->log_desc_n = log_desc_n; sq->umem_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, wq_size + @@ -295,7 +251,7 @@ attr.tis_lst_sz = 0; attr.tis_num = 0; attr.user_index = 0xFFFF; - attr.cqn = sq->cq.cq->id; + attr.cqn = sq->cq.cq_obj.cq->id; wq_attr->uar_page = mlx5_os_get_devx_uar_page_id(uar); wq_attr->pd = pdn; wq_attr->wq_type = MLX5_WQ_TYPE_CYCLIC; @@ -347,8 +303,7 @@ mlx5_aso_queue_init(struct mlx5_dev_ctx_shared *sh) { return mlx5_aso_sq_create(sh->ctx, &sh->aso_age_mng->aso_sq, 0, - sh->tx_uar, sh->pdn, sh->eqn, - MLX5_ASO_QUEUE_LOG_DESC); + sh->tx_uar, sh->pdn, MLX5_ASO_QUEUE_LOG_DESC); } /** @@ -458,7 +413,7 @@ struct mlx5_aso_cq *cq = &sq->cq; uint32_t idx = cq->cq_ci & ((1 << cq->log_desc_n) - 1); volatile struct mlx5_err_cqe *cqe = - (volatile struct mlx5_err_cqe *)&cq->cqes[idx]; + (volatile struct mlx5_err_cqe *)&cq->cq_obj.cqes[idx]; cq->errors++; idx = rte_be_to_cpu_16(cqe->wqe_counter) & (1u << sq->log_desc_n); @@ -571,8 +526,8 @@ do { idx = next_idx; next_idx = (cq->cq_ci + 1) & mask; - rte_prefetch0(&cq->cqes[next_idx]); - cqe = &cq->cqes[idx]; + rte_prefetch0(&cq->cq_obj.cqes[next_idx]); + cqe = &cq->cq_obj.cqes[idx]; ret = check_cqe(cqe, cq_size, cq->cq_ci); /* * Be sure owner read is done before any other cookie field or @@ -592,7 +547,7 @@ mlx5_aso_age_action_update(sh, i); sq->tail += i; rte_io_wmb(); - cq->db_rec[0] = rte_cpu_to_be_32(cq->cq_ci); + cq->cq_obj.db_rec[0] = rte_cpu_to_be_32(cq->cq_ci); } return i; } From patchwork Thu Dec 17 11:44:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85316 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8C116A09F6; Thu, 17 Dec 2020 12:47:50 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 342D8CA6E; Thu, 17 Dec 2020 12:45:12 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id ABAF1CA28 for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:50 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2W004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:25 +0000 Message-Id: <1608205475-20067-8-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 07/17] net/mlx5: move Tx CQ creation to common X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Using common function for Tx CQ creation. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5.h | 6 +- drivers/net/mlx5/mlx5_devx.c | 182 +++++++------------------------------------ 2 files changed, 31 insertions(+), 157 deletions(-) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index e02faed..2e75498 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -843,11 +843,7 @@ struct mlx5_txq_obj { }; struct { struct rte_eth_dev *dev; - struct mlx5_devx_obj *cq_devx; - void *cq_umem; - void *cq_buf; - int64_t cq_dbrec_offset; - struct mlx5_devx_dbr_page *cq_dbrec_page; + struct mlx5_devx_cq cq_obj; struct mlx5_devx_obj *sq_devx; void *sq_umem; void *sq_buf; diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index de9b204..9560f2b 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -15,6 +15,7 @@ #include #include +#include #include #include "mlx5.h" @@ -1144,28 +1145,6 @@ } /** - * Release DevX Tx CQ resources. - * - * @param txq_obj - * DevX Tx queue object. - */ -static void -mlx5_txq_release_devx_cq_resources(struct mlx5_txq_obj *txq_obj) -{ - if (txq_obj->cq_devx) - claim_zero(mlx5_devx_cmd_destroy(txq_obj->cq_devx)); - if (txq_obj->cq_umem) - claim_zero(mlx5_glue->devx_umem_dereg(txq_obj->cq_umem)); - if (txq_obj->cq_buf) - mlx5_free(txq_obj->cq_buf); - if (txq_obj->cq_dbrec_page) - claim_zero(mlx5_release_dbr(&txq_obj->txq_ctrl->priv->dbrpgs, - mlx5_os_get_umem_id - (txq_obj->cq_dbrec_page->umem), - txq_obj->cq_dbrec_offset)); -} - -/** * Destroy the Tx queue DevX object. * * @param txq_obj @@ -1175,126 +1154,8 @@ mlx5_txq_release_devx_resources(struct mlx5_txq_obj *txq_obj) { mlx5_txq_release_devx_sq_resources(txq_obj); - mlx5_txq_release_devx_cq_resources(txq_obj); -} - -/** - * Create a DevX CQ object and its resources for an Tx queue. - * - * @param dev - * Pointer to Ethernet device. - * @param idx - * Queue index in DPDK Tx queue array. - * - * @return - * Number of CQEs in CQ, 0 otherwise and rte_errno is set. - */ -static uint32_t -mlx5_txq_create_devx_cq_resources(struct rte_eth_dev *dev, uint16_t idx) -{ - struct mlx5_priv *priv = dev->data->dev_private; - struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; - struct mlx5_txq_ctrl *txq_ctrl = - container_of(txq_data, struct mlx5_txq_ctrl, txq); - struct mlx5_txq_obj *txq_obj = txq_ctrl->obj; - struct mlx5_devx_cq_attr cq_attr = { 0 }; - struct mlx5_cqe *cqe; - size_t page_size; - size_t alignment; - uint32_t cqe_n; - uint32_t i; - int ret; - - MLX5_ASSERT(txq_data); - MLX5_ASSERT(txq_obj); - page_size = rte_mem_page_size(); - if (page_size == (size_t)-1) { - DRV_LOG(ERR, "Failed to get mem page size."); - rte_errno = ENOMEM; - return 0; - } - /* Allocate memory buffer for CQEs. */ - alignment = MLX5_CQE_BUF_ALIGNMENT; - if (alignment == (size_t)-1) { - DRV_LOG(ERR, "Failed to get CQE buf alignment."); - rte_errno = ENOMEM; - return 0; - } - /* Create the Completion Queue. */ - cqe_n = (1UL << txq_data->elts_n) / MLX5_TX_COMP_THRESH + - 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; - cqe_n = 1UL << log2above(cqe_n); - if (cqe_n > UINT16_MAX) { - DRV_LOG(ERR, - "Port %u Tx queue %u requests to many CQEs %u.", - dev->data->port_id, txq_data->idx, cqe_n); - rte_errno = EINVAL; - return 0; - } - txq_obj->cq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - cqe_n * sizeof(struct mlx5_cqe), - alignment, - priv->sh->numa_node); - if (!txq_obj->cq_buf) { - DRV_LOG(ERR, - "Port %u Tx queue %u cannot allocate memory (CQ).", - dev->data->port_id, txq_data->idx); - rte_errno = ENOMEM; - return 0; - } - /* Register allocated buffer in user space with DevX. */ - txq_obj->cq_umem = mlx5_glue->devx_umem_reg(priv->sh->ctx, - (void *)txq_obj->cq_buf, - cqe_n * sizeof(struct mlx5_cqe), - IBV_ACCESS_LOCAL_WRITE); - if (!txq_obj->cq_umem) { - rte_errno = errno; - DRV_LOG(ERR, - "Port %u Tx queue %u cannot register memory (CQ).", - dev->data->port_id, txq_data->idx); - goto error; - } - /* Allocate doorbell record for completion queue. */ - txq_obj->cq_dbrec_offset = mlx5_get_dbr(priv->sh->ctx, - &priv->dbrpgs, - &txq_obj->cq_dbrec_page); - if (txq_obj->cq_dbrec_offset < 0) { - rte_errno = errno; - DRV_LOG(ERR, "Failed to allocate CQ door-bell."); - goto error; - } - cq_attr.cqe_size = (sizeof(struct mlx5_cqe) == 128) ? - MLX5_CQE_SIZE_128B : MLX5_CQE_SIZE_64B; - cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar); - cq_attr.eqn = priv->sh->eqn; - cq_attr.q_umem_valid = 1; - cq_attr.q_umem_offset = (uintptr_t)txq_obj->cq_buf % page_size; - cq_attr.q_umem_id = mlx5_os_get_umem_id(txq_obj->cq_umem); - cq_attr.db_umem_valid = 1; - cq_attr.db_umem_offset = txq_obj->cq_dbrec_offset; - cq_attr.db_umem_id = mlx5_os_get_umem_id(txq_obj->cq_dbrec_page->umem); - cq_attr.log_cq_size = rte_log2_u32(cqe_n); - cq_attr.log_page_size = rte_log2_u32(page_size); - /* Create completion queue object with DevX. */ - txq_obj->cq_devx = mlx5_devx_cmd_create_cq(priv->sh->ctx, &cq_attr); - if (!txq_obj->cq_devx) { - rte_errno = errno; - DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.", - dev->data->port_id, idx); - goto error; - } - /* Initial fill CQ buffer with invalid CQE opcode. */ - cqe = (struct mlx5_cqe *)txq_obj->cq_buf; - for (i = 0; i < cqe_n; i++) { - cqe->op_own = (MLX5_CQE_INVALID << 4) | MLX5_CQE_OWNER_MASK; - ++cqe; - } - return cqe_n; -error: - ret = rte_errno; - mlx5_txq_release_devx_cq_resources(txq_obj); - rte_errno = ret; - return 0; + mlx5_devx_cq_destroy(&txq_obj->cq_obj); + memset(&txq_obj->cq_obj, 0, sizeof(txq_obj->cq_obj)); } /** @@ -1366,7 +1227,7 @@ sq_attr.tis_lst_sz = 1; sq_attr.tis_num = priv->sh->tis->id; sq_attr.state = MLX5_SQC_STATE_RST; - sq_attr.cqn = txq_obj->cq_devx->id; + sq_attr.cqn = txq_obj->cq_obj.cq->id; sq_attr.flush_in_error_en = 1; sq_attr.allow_multi_pkt_send_wqe = !!priv->config.mps; sq_attr.allow_swp = !!priv->config.swp; @@ -1430,8 +1291,13 @@ #else struct mlx5_dev_ctx_shared *sh = priv->sh; struct mlx5_txq_obj *txq_obj = txq_ctrl->obj; + struct mlx5_devx_cq_attr cq_attr = { + .uar_page_id = mlx5_os_get_devx_uar_page_id(sh->tx_uar), + .cqe_size = (sizeof(struct mlx5_cqe) == 128) ? + MLX5_CQE_SIZE_128B : MLX5_CQE_SIZE_64B, + }; void *reg_addr; - uint32_t cqe_n; + uint32_t cqe_n, log_desc_n; uint32_t wqe_n; int ret = 0; @@ -1439,19 +1305,31 @@ MLX5_ASSERT(txq_obj); txq_obj->txq_ctrl = txq_ctrl; txq_obj->dev = dev; - cqe_n = mlx5_txq_create_devx_cq_resources(dev, idx); - if (!cqe_n) { - rte_errno = errno; + cqe_n = (1UL << txq_data->elts_n) / MLX5_TX_COMP_THRESH + + 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; + log_desc_n = log2above(cqe_n); + cqe_n = 1UL << log_desc_n; + if (cqe_n > UINT16_MAX) { + DRV_LOG(ERR, "Port %u Tx queue %u requests to many CQEs %u.", + dev->data->port_id, txq_data->idx, cqe_n); + rte_errno = EINVAL; + return 0; + } + /* Create completion queue object with DevX. */ + ret = mlx5_devx_cq_create(sh->ctx, &txq_obj->cq_obj, log_desc_n, + &cq_attr, priv->sh->numa_node); + if (ret) { + DRV_LOG(ERR, "Port %u Tx queue %u CQ creation failure.", + dev->data->port_id, idx); goto error; } - txq_data->cqe_n = log2above(cqe_n); - txq_data->cqe_s = 1 << txq_data->cqe_n; + txq_data->cqe_n = log_desc_n; + txq_data->cqe_s = cqe_n; txq_data->cqe_m = txq_data->cqe_s - 1; - txq_data->cqes = (volatile struct mlx5_cqe *)txq_obj->cq_buf; + txq_data->cqes = txq_obj->cq_obj.cqes; txq_data->cq_ci = 0; txq_data->cq_pi = 0; - txq_data->cq_db = (volatile uint32_t *)(txq_obj->cq_dbrec_page->dbrs + - txq_obj->cq_dbrec_offset); + txq_data->cq_db = txq_obj->cq_obj.db_rec; *txq_data->cq_db = 0; /* Create Send Queue object with DevX. */ wqe_n = mlx5_txq_create_devx_sq_resources(dev, idx); From patchwork Thu Dec 17 11:44:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85312 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5BA16A09F6; Thu, 17 Dec 2020 12:46:35 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E87A5CA4B; Thu, 17 Dec 2020 12:45:06 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 6D082CA18 for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:50 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2X004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:26 +0000 Message-Id: <1608205475-20067-9-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 08/17] net/mlx5: move Rx CQ creation to common X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Using common function for Rx CQ creation. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5.c | 8 --- drivers/net/mlx5/mlx5.h | 3 +- drivers/net/mlx5/mlx5_devx.c | 142 +++++++++++++------------------------------ drivers/net/mlx5/mlx5_rxtx.h | 4 -- 4 files changed, 42 insertions(+), 115 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 52a8a25..3c7e5d2 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -938,14 +938,6 @@ struct mlx5_dev_ctx_shared * goto error; } if (sh->devx) { - /* Query the EQN for this core. */ - err = mlx5_glue->devx_query_eqn(sh->ctx, 0, &sh->eqn); - if (err) { - rte_errno = errno; - DRV_LOG(ERR, "Failed to query event queue number %d.", - rte_errno); - goto error; - } err = mlx5_os_get_pdn(sh->pd, &sh->pdn); if (err) { DRV_LOG(ERR, "Fail to extract pdn from PD"); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 2e75498..9a59c26 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -679,7 +679,6 @@ struct mlx5_dev_ctx_shared { uint16_t bond_dev; /* Bond primary device id. */ uint32_t devx:1; /* Opened with DV. */ uint32_t flow_hit_aso_en:1; /* Flow Hit ASO is supported. */ - uint32_t eqn; /* Event Queue number. */ uint32_t max_port; /* Maximal IB device port index. */ void *ctx; /* Verbs/DV/DevX context. */ void *pd; /* Protection Domain. */ @@ -787,7 +786,7 @@ struct mlx5_rxq_obj { }; struct { struct mlx5_devx_obj *rq; /* DevX Rx Queue object. */ - struct mlx5_devx_obj *devx_cq; /* DevX CQ object. */ + struct mlx5_devx_cq cq_obj; /* DevX CQ object. */ void *devx_channel; }; }; diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 9560f2b..6ad70f2 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -172,30 +172,17 @@ } /** - * Release the resources allocated for the Rx CQ DevX object. + * Destroy the Rx queue DevX object. * - * @param rxq_ctrl - * DevX Rx queue object. + * @param rxq_obj + * Rxq object to destroy. */ static void -mlx5_rxq_release_devx_cq_resources(struct mlx5_rxq_ctrl *rxq_ctrl) +mlx5_rxq_release_devx_resources(struct mlx5_rxq_ctrl *rxq_ctrl) { - struct mlx5_devx_dbr_page *dbr_page = rxq_ctrl->cq_dbrec_page; - - if (rxq_ctrl->cq_umem) { - mlx5_glue->devx_umem_dereg(rxq_ctrl->cq_umem); - rxq_ctrl->cq_umem = NULL; - } - if (rxq_ctrl->rxq.cqes) { - rte_free((void *)(uintptr_t)rxq_ctrl->rxq.cqes); - rxq_ctrl->rxq.cqes = NULL; - } - if (dbr_page) { - claim_zero(mlx5_release_dbr(&rxq_ctrl->priv->dbrpgs, - mlx5_os_get_umem_id(dbr_page->umem), - rxq_ctrl->cq_dbr_offset)); - rxq_ctrl->cq_dbrec_page = NULL; - } + mlx5_rxq_release_devx_rq_resources(rxq_ctrl); + mlx5_devx_cq_destroy(&rxq_ctrl->obj->cq_obj); + memset(&rxq_ctrl->obj->cq_obj, 0, sizeof(rxq_ctrl->obj->cq_obj)); } /** @@ -213,14 +200,12 @@ mlx5_devx_modify_rq(rxq_obj, MLX5_RXQ_MOD_RDY2RST); claim_zero(mlx5_devx_cmd_destroy(rxq_obj->rq)); } else { - MLX5_ASSERT(rxq_obj->devx_cq); + MLX5_ASSERT(rxq_obj->cq_obj); claim_zero(mlx5_devx_cmd_destroy(rxq_obj->rq)); - claim_zero(mlx5_devx_cmd_destroy(rxq_obj->devx_cq)); if (rxq_obj->devx_channel) mlx5_glue->devx_destroy_event_channel (rxq_obj->devx_channel); - mlx5_rxq_release_devx_rq_resources(rxq_obj->rxq_ctrl); - mlx5_rxq_release_devx_cq_resources(rxq_obj->rxq_ctrl); + mlx5_rxq_release_devx_resources(rxq_obj->rxq_ctrl); } } @@ -249,7 +234,7 @@ rte_errno = errno; return -rte_errno; } - if (out.event_resp.cookie != (uint64_t)(uintptr_t)rxq_obj->devx_cq) { + if (out.event_resp.cookie != (uint64_t)(uintptr_t)rxq_obj->cq_obj.cq) { rte_errno = EINVAL; return -rte_errno; } @@ -327,7 +312,7 @@ container_of(rxq_data, struct mlx5_rxq_ctrl, rxq); struct mlx5_devx_create_rq_attr rq_attr = { 0 }; uint32_t wqe_n = 1 << (rxq_data->elts_n - rxq_data->sges_n); - uint32_t cqn = rxq_ctrl->obj->devx_cq->id; + uint32_t cqn = rxq_ctrl->obj->cq_obj.cq->id; struct mlx5_devx_dbr_page *dbr_page; int64_t dbr_offset; uint32_t wq_size = 0; @@ -410,31 +395,23 @@ * Queue index in DPDK Rx queue array. * * @return - * The DevX CQ object initialized, NULL otherwise and rte_errno is set. + * 0 on success, a negative errno value otherwise and rte_errno is set. */ -static struct mlx5_devx_obj * +static int mlx5_rxq_create_devx_cq_resources(struct rte_eth_dev *dev, uint16_t idx) { - struct mlx5_devx_obj *cq_obj = 0; + struct mlx5_devx_cq *cq_obj = 0; struct mlx5_devx_cq_attr cq_attr = { 0 }; struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_dev_ctx_shared *sh = priv->sh; struct mlx5_rxq_data *rxq_data = (*priv->rxqs)[idx]; struct mlx5_rxq_ctrl *rxq_ctrl = container_of(rxq_data, struct mlx5_rxq_ctrl, rxq); - size_t page_size = rte_mem_page_size(); unsigned int cqe_n = mlx5_rxq_cqe_num(rxq_data); - struct mlx5_devx_dbr_page *dbr_page; - int64_t dbr_offset; - void *buf = NULL; - uint16_t event_nums[1] = {0}; uint32_t log_cqe_n; - uint32_t cq_size; + uint16_t event_nums[1] = { 0 }; int ret = 0; - if (page_size == (size_t)-1) { - DRV_LOG(ERR, "Failed to get page_size."); - goto error; - } if (priv->config.cqe_comp && !rxq_data->hw_timestamp && !rxq_data->lro) { cq_attr.cqe_comp_en = 1u; @@ -489,71 +466,37 @@ } if (priv->config.cqe_pad) cq_attr.cqe_size = MLX5_CQE_SIZE_128B; + cq_attr.uar_page_id = mlx5_os_get_devx_uar_page_id(sh->devx_rx_uar); log_cqe_n = log2above(cqe_n); - cq_size = sizeof(struct mlx5_cqe) * (1 << log_cqe_n); - buf = rte_calloc_socket(__func__, 1, cq_size, page_size, - rxq_ctrl->socket); - if (!buf) { - DRV_LOG(ERR, "Failed to allocate memory for CQ."); - goto error; - } - rxq_data->cqes = (volatile struct mlx5_cqe (*)[])(uintptr_t)buf; - rxq_ctrl->cq_umem = mlx5_glue->devx_umem_reg(priv->sh->ctx, buf, - cq_size, - IBV_ACCESS_LOCAL_WRITE); - if (!rxq_ctrl->cq_umem) { - DRV_LOG(ERR, "Failed to register umem for CQ."); - goto error; - } - /* Allocate CQ door-bell. */ - dbr_offset = mlx5_get_dbr(priv->sh->ctx, &priv->dbrpgs, &dbr_page); - if (dbr_offset < 0) { - DRV_LOG(ERR, "Failed to allocate CQ door-bell."); - goto error; - } - rxq_ctrl->cq_dbr_offset = dbr_offset; - rxq_ctrl->cq_dbrec_page = dbr_page; - rxq_data->cq_db = (uint32_t *)((uintptr_t)dbr_page->dbrs + - (uintptr_t)rxq_ctrl->cq_dbr_offset); - rxq_data->cq_uar = - mlx5_os_get_devx_uar_base_addr(priv->sh->devx_rx_uar); /* Create CQ using DevX API. */ - cq_attr.eqn = priv->sh->eqn; - cq_attr.uar_page_id = - mlx5_os_get_devx_uar_page_id(priv->sh->devx_rx_uar); - cq_attr.q_umem_id = mlx5_os_get_umem_id(rxq_ctrl->cq_umem); - cq_attr.q_umem_valid = 1; - cq_attr.log_cq_size = log_cqe_n; - cq_attr.log_page_size = rte_log2_u32(page_size); - cq_attr.db_umem_offset = rxq_ctrl->cq_dbr_offset; - cq_attr.db_umem_id = mlx5_os_get_umem_id(dbr_page->umem); - cq_attr.db_umem_valid = 1; - cq_obj = mlx5_devx_cmd_create_cq(priv->sh->ctx, &cq_attr); - if (!cq_obj) - goto error; + ret = mlx5_devx_cq_create(sh->ctx, &rxq_ctrl->obj->cq_obj, log_cqe_n, + &cq_attr, sh->numa_node); + if (ret) + return ret; + cq_obj = &rxq_ctrl->obj->cq_obj; + rxq_data->cqes = (volatile struct mlx5_cqe (*)[]) + (uintptr_t)cq_obj->cqes; + rxq_data->cq_db = cq_obj->db_rec; + rxq_data->cq_uar = mlx5_os_get_devx_uar_base_addr(sh->devx_rx_uar); rxq_data->cqe_n = log_cqe_n; - rxq_data->cqn = cq_obj->id; + rxq_data->cqn = cq_obj->cq->id; if (rxq_ctrl->obj->devx_channel) { ret = mlx5_glue->devx_subscribe_devx_event - (rxq_ctrl->obj->devx_channel, - cq_obj->obj, - sizeof(event_nums), - event_nums, - (uint64_t)(uintptr_t)cq_obj); + (rxq_ctrl->obj->devx_channel, + cq_obj->cq->obj, + sizeof(event_nums), + event_nums, + (uint64_t)(uintptr_t)cq_obj->cq); if (ret) { DRV_LOG(ERR, "Fail to subscribe CQ to event channel."); - rte_errno = errno; - goto error; + ret = errno; + mlx5_devx_cq_destroy(cq_obj); + memset(cq_obj, 0, sizeof(*cq_obj)); + rte_errno = ret; + return -ret; } } - /* Initialise CQ to 1's to mark HW ownership for all CQEs. */ - memset((void *)(uintptr_t)rxq_data->cqes, 0xFF, cq_size); - return cq_obj; -error: - if (cq_obj) - mlx5_devx_cmd_destroy(cq_obj); - mlx5_rxq_release_devx_cq_resources(rxq_ctrl); - return NULL; + return 0; } /** @@ -657,8 +600,8 @@ tmpl->fd = mlx5_os_get_devx_channel_fd(tmpl->devx_channel); } /* Create CQ using DevX API. */ - tmpl->devx_cq = mlx5_rxq_create_devx_cq_resources(dev, idx); - if (!tmpl->devx_cq) { + ret = mlx5_rxq_create_devx_cq_resources(dev, idx); + if (ret) { DRV_LOG(ERR, "Failed to create CQ."); goto error; } @@ -684,12 +627,9 @@ ret = rte_errno; /* Save rte_errno before cleanup. */ if (tmpl->rq) claim_zero(mlx5_devx_cmd_destroy(tmpl->rq)); - if (tmpl->devx_cq) - claim_zero(mlx5_devx_cmd_destroy(tmpl->devx_cq)); if (tmpl->devx_channel) mlx5_glue->devx_destroy_event_channel(tmpl->devx_channel); - mlx5_rxq_release_devx_rq_resources(rxq_ctrl); - mlx5_rxq_release_devx_cq_resources(rxq_ctrl); + mlx5_rxq_release_devx_resources(rxq_ctrl); rte_errno = ret; /* Restore rte_errno. */ return -rte_errno; } diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 7989a50..6a71791 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -196,11 +196,7 @@ struct mlx5_rxq_ctrl { struct mlx5_devx_dbr_page *rq_dbrec_page; uint64_t rq_dbr_offset; /* Storing RQ door-bell information, needed when freeing door-bell. */ - struct mlx5_devx_dbr_page *cq_dbrec_page; - uint64_t cq_dbr_offset; - /* Storing CQ door-bell information, needed when freeing door-bell. */ void *wq_umem; /* WQ buffer registration info. */ - void *cq_umem; /* CQ buffer registration info. */ struct rte_eth_hairpin_conf hairpin_conf; /* Hairpin configuration. */ uint32_t hairpin_status; /* Hairpin binding status. */ }; From patchwork Thu Dec 17 11:44:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85317 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id E26EFA09F6; Thu, 17 Dec 2020 12:48:08 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 63C34CA8F; Thu, 17 Dec 2020 12:45:13 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id C8511CA08 for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:50 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2Y004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:27 +0000 Message-Id: <1608205475-20067-10-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 09/17] common/mlx5: enhance page size configuration X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The PRM calculates page size in 4K, so need to reduce the log_wq_pg_sz attribute. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/common/mlx5/mlx5_devx_cmds.c | 53 ++++++++++++++++-------------------- drivers/net/mlx5/mlx5_devx.c | 13 +++++---- 2 files changed, 30 insertions(+), 36 deletions(-) diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c index 9c1d188..09e204b 100644 --- a/drivers/common/mlx5/mlx5_devx_cmds.c +++ b/drivers/common/mlx5/mlx5_devx_cmds.c @@ -268,9 +268,8 @@ struct mlx5_devx_obj * MLX5_SET(mkc, mkc, mkey_7_0, attr->umem_id & 0xFF); MLX5_SET(mkc, mkc, translations_octword_size, translation_size); MLX5_SET(mkc, mkc, relaxed_ordering_write, - attr->relaxed_ordering_write); - MLX5_SET(mkc, mkc, relaxed_ordering_read, - attr->relaxed_ordering_read); + attr->relaxed_ordering_write); + MLX5_SET(mkc, mkc, relaxed_ordering_read, attr->relaxed_ordering_read); MLX5_SET64(mkc, mkc, start_addr, attr->addr); MLX5_SET64(mkc, mkc, len, attr->size); mkey->obj = mlx5_glue->devx_obj_create(ctx, in, in_size_dw * 4, out, @@ -308,7 +307,7 @@ struct mlx5_devx_obj * if (status) { int syndrome = MLX5_GET(query_flow_counter_out, out, syndrome); - DRV_LOG(ERR, "Bad devX status %x, syndrome = %x", status, + DRV_LOG(ERR, "Bad DevX status %x, syndrome = %x", status, syndrome); } return status; @@ -374,8 +373,7 @@ struct mlx5_devx_obj * syndrome = MLX5_GET(query_nic_vport_context_out, out, syndrome); if (status) { DRV_LOG(DEBUG, "Failed to query NIC vport context, " - "status %x, syndrome = %x", - status, syndrome); + "status %x, syndrome = %x", status, syndrome); return -1; } vctx = MLX5_ADDR_OF(query_nic_vport_context_out, out, @@ -662,8 +660,7 @@ struct mlx5_devx_obj * syndrome = MLX5_GET(query_hca_cap_out, out, syndrome); if (status) { DRV_LOG(DEBUG, "Failed to query devx HCA capabilities, " - "status %x, syndrome = %x", - status, syndrome); + "status %x, syndrome = %x", status, syndrome); return -1; } hcattr = MLX5_ADDR_OF(query_hca_cap_out, out, capability); @@ -683,11 +680,11 @@ struct mlx5_devx_obj * (cmd_hca_cap, hcattr, log_min_hairpin_wq_data_sz); attr->vhca_id = MLX5_GET(cmd_hca_cap, hcattr, vhca_id); attr->relaxed_ordering_write = MLX5_GET(cmd_hca_cap, hcattr, - relaxed_ordering_write); + relaxed_ordering_write); attr->relaxed_ordering_read = MLX5_GET(cmd_hca_cap, hcattr, - relaxed_ordering_read); + relaxed_ordering_read); attr->access_register_user = MLX5_GET(cmd_hca_cap, hcattr, - access_register_user); + access_register_user); attr->eth_net_offloads = MLX5_GET(cmd_hca_cap, hcattr, eth_net_offloads); attr->eth_virt = MLX5_GET(cmd_hca_cap, hcattr, eth_virt); @@ -730,8 +727,7 @@ struct mlx5_devx_obj * goto error; if (status) { DRV_LOG(DEBUG, "Failed to query devx QOS capabilities," - " status %x, syndrome = %x", - status, syndrome); + " status %x, syndrome = %x", status, syndrome); return -1; } hcattr = MLX5_ADDR_OF(query_hca_cap_out, out, capability); @@ -761,17 +757,14 @@ struct mlx5_devx_obj * MLX5_GET_HCA_CAP_OP_MOD_NIC_FLOW_TABLE | MLX5_HCA_CAP_OPMOD_GET_CUR); - rc = mlx5_glue->devx_general_cmd(ctx, - in, sizeof(in), - out, sizeof(out)); + rc = mlx5_glue->devx_general_cmd(ctx, in, sizeof(in), out, sizeof(out)); if (rc) goto error; status = MLX5_GET(query_hca_cap_out, out, status); syndrome = MLX5_GET(query_hca_cap_out, out, syndrome); if (status) { DRV_LOG(DEBUG, "Failed to query devx HCA capabilities, " - "status %x, syndrome = %x", - status, syndrome); + "status %x, syndrome = %x", status, syndrome); attr->log_max_ft_sampler_num = 0; return -1; } @@ -788,9 +781,7 @@ struct mlx5_devx_obj * MLX5_GET_HCA_CAP_OP_MOD_ETHERNET_OFFLOAD_CAPS | MLX5_HCA_CAP_OPMOD_GET_CUR); - rc = mlx5_glue->devx_general_cmd(ctx, - in, sizeof(in), - out, sizeof(out)); + rc = mlx5_glue->devx_general_cmd(ctx, in, sizeof(in), out, sizeof(out)); if (rc) { attr->eth_net_offloads = 0; goto error; @@ -799,8 +790,7 @@ struct mlx5_devx_obj * syndrome = MLX5_GET(query_hca_cap_out, out, syndrome); if (status) { DRV_LOG(DEBUG, "Failed to query devx HCA capabilities, " - "status %x, syndrome = %x", - status, syndrome); + "status %x, syndrome = %x", status, syndrome); attr->eth_net_offloads = 0; return -1; } @@ -916,7 +906,9 @@ struct mlx5_devx_obj * MLX5_SET(wq, wq_ctx, hw_counter, wq_attr->hw_counter); MLX5_SET(wq, wq_ctx, sw_counter, wq_attr->sw_counter); MLX5_SET(wq, wq_ctx, log_wq_stride, wq_attr->log_wq_stride); - MLX5_SET(wq, wq_ctx, log_wq_pg_sz, wq_attr->log_wq_pg_sz); + if (wq_attr->log_wq_pg_sz > MLX5_ADAPTER_PAGE_SHIFT) + MLX5_SET(wq, wq_ctx, log_wq_pg_sz, + wq_attr->log_wq_pg_sz - MLX5_ADAPTER_PAGE_SHIFT); MLX5_SET(wq, wq_ctx, log_wq_sz, wq_attr->log_wq_sz); MLX5_SET(wq, wq_ctx, dbr_umem_valid, wq_attr->dbr_umem_valid); MLX5_SET(wq, wq_ctx, wq_umem_valid, wq_attr->wq_umem_valid); @@ -1562,13 +1554,13 @@ struct mlx5_devx_obj * MLX5_SET(cqc, cqctx, cc, attr->use_first_only); MLX5_SET(cqc, cqctx, oi, attr->overrun_ignore); MLX5_SET(cqc, cqctx, log_cq_size, attr->log_cq_size); - MLX5_SET(cqc, cqctx, log_page_size, attr->log_page_size - - MLX5_ADAPTER_PAGE_SHIFT); + if (attr->log_page_size > MLX5_ADAPTER_PAGE_SHIFT) + MLX5_SET(cqc, cqctx, log_page_size, + attr->log_page_size - MLX5_ADAPTER_PAGE_SHIFT); MLX5_SET(cqc, cqctx, c_eqn, attr->eqn); MLX5_SET(cqc, cqctx, uar_page, attr->uar_page_id); MLX5_SET(cqc, cqctx, cqe_comp_en, !!attr->cqe_comp_en); - MLX5_SET(cqc, cqctx, mini_cqe_res_format, - attr->mini_cqe_res_format); + MLX5_SET(cqc, cqctx, mini_cqe_res_format, attr->mini_cqe_res_format); MLX5_SET(cqc, cqctx, mini_cqe_res_format_ext, attr->mini_cqe_res_format_ext); MLX5_SET(cqc, cqctx, cqe_sz, attr->cqe_size); @@ -1798,8 +1790,9 @@ struct mlx5_devx_obj * if (attr->uar_index) { MLX5_SET(qpc, qpc, pm_state, MLX5_QP_PM_MIGRATED); MLX5_SET(qpc, qpc, uar_page, attr->uar_index); - MLX5_SET(qpc, qpc, log_page_size, attr->log_page_size - - MLX5_ADAPTER_PAGE_SHIFT); + if (attr->log_page_size > MLX5_ADAPTER_PAGE_SHIFT) + MLX5_SET(qpc, qpc, log_page_size, + attr->log_page_size - MLX5_ADAPTER_PAGE_SHIFT); if (attr->sq_size) { MLX5_ASSERT(RTE_IS_POWER_OF_2(attr->sq_size)); MLX5_SET(qpc, qpc, cqn_snd, attr->cqn); diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 6ad70f2..fe103a7 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -320,7 +320,13 @@ uint32_t log_wqe_size = 0; void *buf = NULL; struct mlx5_devx_obj *rq; + size_t alignment = MLX5_WQE_BUF_ALIGNMENT; + if (alignment == (size_t)-1) { + DRV_LOG(ERR, "Failed to get mem page size"); + rte_errno = ENOMEM; + return NULL; + } /* Fill RQ attributes. */ rq_attr.mem_rq_type = MLX5_RQC_MEM_RQ_TYPE_MEMORY_RQ_INLINE; rq_attr.flush_in_error_en = 1; @@ -347,15 +353,10 @@ log_wqe_size = log2above(wqe_size) + rxq_data->sges_n; rq_attr.wq_attr.log_wq_stride = log_wqe_size; rq_attr.wq_attr.log_wq_sz = rxq_data->elts_n - rxq_data->sges_n; + rq_attr.wq_attr.log_wq_pg_sz = log2above(alignment); /* Calculate and allocate WQ memory space. */ wqe_size = 1 << log_wqe_size; /* round up power of two.*/ wq_size = wqe_n * wqe_size; - size_t alignment = MLX5_WQE_BUF_ALIGNMENT; - if (alignment == (size_t)-1) { - DRV_LOG(ERR, "Failed to get mem page size"); - rte_errno = ENOMEM; - return NULL; - } buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, wq_size, alignment, rxq_ctrl->socket); if (!buf) From patchwork Thu Dec 17 11:44:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85315 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 545FAA09F6; Thu, 17 Dec 2020 12:47:28 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8DEDECA64; Thu, 17 Dec 2020 12:45:10 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id AB955CA26 for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:50 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2Z004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:28 +0000 Message-Id: <1608205475-20067-11-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 10/17] common/mlx5: share DevX SQ creation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The SQ object in DevX is created in several places and in several different drivers. In all places almost all the details are the same, and in particular the allocations of the required resources. Add a structure that contains all the resources, and provide creation and release functions for it. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/common/mlx5/mlx5_common_devx.c | 122 +++++++++++++++++++++++++++++++++ drivers/common/mlx5/mlx5_common_devx.h | 20 +++++- 2 files changed, 140 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common_devx.c b/drivers/common/mlx5/mlx5_common_devx.c index 324c6ea..46404d8 100644 --- a/drivers/common/mlx5/mlx5_common_devx.c +++ b/drivers/common/mlx5/mlx5_common_devx.c @@ -155,3 +155,125 @@ rte_errno = ret; return -rte_errno; } + +/** + * Destroy DevX Send Queue. + * + * @param[in] sq + * DevX SQ to destroy. + */ +void +mlx5_devx_sq_destroy(struct mlx5_devx_sq *sq) +{ + if (sq->sq) + claim_zero(mlx5_devx_cmd_destroy(sq->sq)); + if (sq->umem_obj) + claim_zero(mlx5_glue->devx_umem_dereg(sq->umem_obj)); + if (sq->umem_buf) + mlx5_free((void *)(uintptr_t)sq->umem_buf); +} + +/** + * Create Send Queue using DevX API. + * + * Get a pointer to partially initialized attributes structure, and updates the + * following fields: + * wq_type + * wq_umem_valid + * wq_umem_id + * wq_umem_offset + * dbr_umem_valid + * dbr_umem_id + * dbr_addr + * log_wq_stride + * log_wq_sz + * log_wq_pg_sz + * All other fields are updated by caller. + * + * @param[in] ctx + * Context returned from mlx5 open_device() glue function. + * @param[in/out] sq_obj + * Pointer to SQ to create. + * @param[in] log_wqbb_n + * Log of number of WQBBs in queue. + * @param[in] attr + * Pointer to SQ attributes structure. + * @param[in] socket + * Socket to use for allocation. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_devx_sq_create(void *ctx, struct mlx5_devx_sq *sq_obj, uint16_t log_wqbb_n, + struct mlx5_devx_create_sq_attr *attr, int socket) +{ + struct mlx5_devx_obj *sq = NULL; + struct mlx5dv_devx_umem *umem_obj = NULL; + void *umem_buf = NULL; + size_t page_size = rte_mem_page_size(); + size_t alignment = MLX5_WQE_BUF_ALIGNMENT; + uint32_t umem_size, umem_dbrec; + uint16_t sq_size = 1 << log_wqbb_n; + int ret; + + if (page_size == (size_t)-1 || alignment == (size_t)-1) { + DRV_LOG(ERR, "Failed to get page_size."); + rte_errno = ENOMEM; + return -rte_errno; + } + /* Allocate memory buffer for WQEs and doorbell record. */ + umem_size = MLX5_WQE_SIZE * sq_size; + umem_dbrec = RTE_ALIGN(umem_size, MLX5_DBR_SIZE); + umem_size += MLX5_DBR_SIZE; + umem_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, umem_size, + alignment, socket); + if (!umem_buf) { + DRV_LOG(ERR, "Failed to allocate memory for SQ."); + rte_errno = ENOMEM; + return -rte_errno; + } + /* Register allocated buffer in user space with DevX. */ + umem_obj = mlx5_glue->devx_umem_reg(ctx, (void *)(uintptr_t)umem_buf, + umem_size, IBV_ACCESS_LOCAL_WRITE); + if (!umem_obj) { + DRV_LOG(ERR, "Failed to register umem for SQ."); + rte_errno = errno; + goto error; + } + /* Fill attributes for SQ object creation. */ + attr->wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC; + attr->wq_attr.wq_umem_valid = 1; + attr->wq_attr.wq_umem_id = mlx5_os_get_umem_id(sq_obj->umem_obj); + attr->wq_attr.wq_umem_offset = 0; + attr->wq_attr.dbr_umem_valid = 1; + attr->wq_attr.dbr_umem_id = attr->wq_attr.wq_umem_id; + attr->wq_attr.dbr_addr = umem_dbrec; + attr->wq_attr.log_wq_stride = rte_log2_u32(MLX5_WQE_SIZE); + attr->wq_attr.log_wq_sz = log_wqbb_n; + attr->wq_attr.log_wq_pg_sz = rte_log2_u32(page_size); + /* Create send queue object with DevX. */ + sq = mlx5_devx_cmd_create_sq(ctx, attr); + if (!sq) { + DRV_LOG(ERR, "Can't create DevX SQ object."); + rte_errno = ENOMEM; + goto error; + } + sq_obj->umem_buf = umem_buf; + sq_obj->umem_obj = umem_obj; + sq_obj->sq = sq; + sq_obj->db_rec = RTE_PTR_ADD(sq_obj->umem_buf, umem_dbrec); + return 0; +error: + ret = rte_errno; + if (sq) + claim_zero(mlx5_devx_cmd_destroy(sq)); + if (umem_obj) + claim_zero(mlx5_glue->devx_umem_dereg(umem_obj)); + if (umem_buf) + mlx5_free((void *)(uintptr_t)umem_buf); + rte_errno = ret; + return -rte_errno; +} + + diff --git a/drivers/common/mlx5/mlx5_common_devx.h b/drivers/common/mlx5/mlx5_common_devx.h index 31cb804..88d520b 100644 --- a/drivers/common/mlx5/mlx5_common_devx.h +++ b/drivers/common/mlx5/mlx5_common_devx.h @@ -18,11 +18,27 @@ struct mlx5_devx_cq { volatile uint32_t *db_rec; /* The CQ doorbell record. */ }; +/* DevX Send Queue structure. */ +struct mlx5_devx_sq { + struct mlx5_devx_obj *sq; /* The SQ DevX object. */ + struct mlx5dv_devx_umem *umem_obj; /* The SQ umem object. */ + union { + volatile void *umem_buf; + volatile struct mlx5_wqe *wqes; /* The SQ ring buffer. */ + }; + volatile uint32_t *db_rec; /* The SQ doorbell record. */ +}; + + /* mlx5_common_devx.c */ void mlx5_devx_cq_destroy(struct mlx5_devx_cq *cq); int mlx5_devx_cq_create(void *ctx, struct mlx5_devx_cq *cq_obj, - uint16_t log_desc_n, struct mlx5_devx_cq_attr *attr, - int socket); + uint16_t log_desc_n, + struct mlx5_devx_cq_attr *attr, int socket); +void mlx5_devx_sq_destroy(struct mlx5_devx_sq *sq); +int mlx5_devx_sq_create(void *ctx, struct mlx5_devx_sq *sq_obj, + uint16_t log_wqbb_n, + struct mlx5_devx_create_sq_attr *attr, int socket); #endif /* RTE_PMD_MLX5_COMMON_DEVX_H_ */ From patchwork Thu Dec 17 11:44:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85322 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id A82E7A09F6; Thu, 17 Dec 2020 12:49:40 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 54488CAC3; Thu, 17 Dec 2020 12:45:19 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 0BAC8CA30 for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:51 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2a004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:29 +0000 Message-Id: <1608205475-20067-12-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 11/17] regex/mlx5: move DevX SQ creation to common X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Using common function for DevX SQ creation. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/regex/mlx5/mlx5_regex.h | 8 +- drivers/regex/mlx5/mlx5_regex_control.c | 153 ++++++++++--------------------- drivers/regex/mlx5/mlx5_regex_fastpath.c | 14 +-- 3 files changed, 55 insertions(+), 120 deletions(-) diff --git a/drivers/regex/mlx5/mlx5_regex.h b/drivers/regex/mlx5/mlx5_regex.h index 9f7a388..7e1b2a9 100644 --- a/drivers/regex/mlx5/mlx5_regex.h +++ b/drivers/regex/mlx5/mlx5_regex.h @@ -18,15 +18,10 @@ struct mlx5_regex_sq { uint16_t log_nb_desc; /* Log 2 number of desc for this object. */ - struct mlx5_devx_obj *obj; /* The SQ DevX object. */ - int64_t dbr_offset; /* Door bell record offset. */ - uint32_t dbr_umem; /* Door bell record umem id. */ - uint8_t *wqe; /* The SQ ring buffer. */ - struct mlx5dv_devx_umem *wqe_umem; /* SQ buffer umem. */ + struct mlx5_devx_sq sq_obj; /* The SQ DevX object. */ size_t pi, db_pi; size_t ci; uint32_t sqn; - uint32_t *dbr; }; struct mlx5_regex_cq { @@ -73,7 +68,6 @@ struct mlx5_regex_priv { uint32_t nb_engines; /* Number of RegEx engines. */ struct mlx5dv_devx_uar *uar; /* UAR object. */ struct ibv_pd *pd; - struct mlx5_dbr_page_list dbrpgs; /* Door-bell pages. */ struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. */ }; diff --git a/drivers/regex/mlx5/mlx5_regex_control.c b/drivers/regex/mlx5/mlx5_regex_control.c index ca6c0f5..df57fad 100644 --- a/drivers/regex/mlx5/mlx5_regex_control.c +++ b/drivers/regex/mlx5/mlx5_regex_control.c @@ -112,6 +112,27 @@ #endif /** + * Destroy the SQ object. + * + * @param qp + * Pointer to the QP element + * @param q_ind + * The index of the queue. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +regex_ctrl_destroy_sq(struct mlx5_regex_qp *qp, uint16_t q_ind) +{ + struct mlx5_regex_sq *sq = &qp->sqs[q_ind]; + + mlx5_devx_sq_destroy(&sq->sq_obj); + memset(sq, 0, sizeof(*sq)); + return 0; +} + +/** * create the SQ object. * * @param priv @@ -131,84 +152,42 @@ uint16_t q_ind, uint16_t log_nb_desc) { #ifdef HAVE_IBV_FLOW_DV_SUPPORT - struct mlx5_devx_create_sq_attr attr = { 0 }; - struct mlx5_devx_modify_sq_attr modify_attr = { 0 }; - struct mlx5_devx_wq_attr *wq_attr = &attr.wq_attr; - struct mlx5_devx_dbr_page *dbr_page = NULL; + struct mlx5_devx_create_sq_attr attr = { + .user_index = q_ind, + .cqn = qp->cq.cq_obj.cq->id, + .wq_attr = (struct mlx5_devx_wq_attr){ + .uar_page = priv->uar->page_id, + }, + }; + struct mlx5_devx_modify_sq_attr modify_attr = { + .state = MLX5_SQC_STATE_RDY, + }; struct mlx5_regex_sq *sq = &qp->sqs[q_ind]; - void *buf = NULL; - uint32_t sq_size; uint32_t pd_num = 0; int ret; sq->log_nb_desc = log_nb_desc; - sq_size = 1 << sq->log_nb_desc; - sq->dbr_offset = mlx5_get_dbr(priv->ctx, &priv->dbrpgs, &dbr_page); - if (sq->dbr_offset < 0) { - DRV_LOG(ERR, "Can't allocate sq door bell record."); - rte_errno = ENOMEM; - goto error; - } - sq->dbr_umem = mlx5_os_get_umem_id(dbr_page->umem); - sq->dbr = (uint32_t *)((uintptr_t)dbr_page->dbrs + - (uintptr_t)sq->dbr_offset); - - buf = rte_calloc(NULL, 1, 64 * sq_size, 4096); - if (!buf) { - DRV_LOG(ERR, "Can't allocate wqe buffer."); - rte_errno = ENOMEM; - goto error; - } - sq->wqe = buf; - sq->wqe_umem = mlx5_glue->devx_umem_reg(priv->ctx, buf, 64 * sq_size, - 7); sq->ci = 0; sq->pi = 0; - if (!sq->wqe_umem) { - DRV_LOG(ERR, "Can't register wqe mem."); - rte_errno = ENOMEM; - goto error; - } - attr.state = MLX5_SQC_STATE_RST; - attr.tis_lst_sz = 0; - attr.tis_num = 0; - attr.user_index = q_ind; - attr.cqn = qp->cq.cq_obj.cq->id; - wq_attr->uar_page = priv->uar->page_id; - regex_get_pdn(priv->pd, &pd_num); - wq_attr->pd = pd_num; - wq_attr->wq_type = MLX5_WQ_TYPE_CYCLIC; - wq_attr->dbr_umem_id = sq->dbr_umem; - wq_attr->dbr_addr = sq->dbr_offset; - wq_attr->dbr_umem_valid = 1; - wq_attr->wq_umem_id = mlx5_os_get_umem_id(sq->wqe_umem); - wq_attr->wq_umem_offset = 0; - wq_attr->wq_umem_valid = 1; - wq_attr->log_wq_stride = 6; - wq_attr->log_wq_sz = sq->log_nb_desc; - sq->obj = mlx5_devx_cmd_create_sq(priv->ctx, &attr); - if (!sq->obj) { - DRV_LOG(ERR, "Can't create sq object."); - rte_errno = ENOMEM; - goto error; + ret = regex_get_pdn(priv->pd, &pd_num); + if (ret) + return ret; + attr.wq_attr.pd = pd_num; + ret = mlx5_devx_sq_create(priv->ctx, &sq->sq_obj, log_nb_desc, &attr, + SOCKET_ID_ANY); + if (ret) { + DRV_LOG(ERR, "Can't create SQ object."); + rte_errno = ENOMEM; + return -rte_errno; } - modify_attr.state = MLX5_SQC_STATE_RDY; - ret = mlx5_devx_cmd_modify_sq(sq->obj, &modify_attr); + ret = mlx5_devx_cmd_modify_sq(sq->sq_obj.sq, &modify_attr); if (ret) { - DRV_LOG(ERR, "Can't change sq state to ready."); - rte_errno = ENOMEM; - goto error; + DRV_LOG(ERR, "Can't change SQ state to ready."); + regex_ctrl_destroy_sq(qp, q_ind); + rte_errno = ENOMEM; + return -rte_errno; } - return 0; -error: - if (sq->wqe_umem) - mlx5_glue->devx_umem_dereg(sq->wqe_umem); - if (buf) - rte_free(buf); - if (sq->dbr_offset) - mlx5_release_dbr(&priv->dbrpgs, sq->dbr_umem, sq->dbr_offset); - return -rte_errno; #else (void)priv; (void)qp; @@ -220,44 +199,6 @@ } /** - * Destroy the SQ object. - * - * @param priv - * Pointer to the priv object. - * @param qp - * Pointer to the QP element - * @param q_ind - * The index of the queue. - * - * @return - * 0 on success, a negative errno value otherwise and rte_errno is set. - */ -static int -regex_ctrl_destroy_sq(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp, - uint16_t q_ind) -{ - struct mlx5_regex_sq *sq = &qp->sqs[q_ind]; - - if (sq->wqe_umem) { - mlx5_glue->devx_umem_dereg(sq->wqe_umem); - sq->wqe_umem = NULL; - } - if (sq->wqe) { - rte_free((void *)(uintptr_t)sq->wqe); - sq->wqe = NULL; - } - if (sq->dbr_offset) { - mlx5_release_dbr(&priv->dbrpgs, sq->dbr_umem, sq->dbr_offset); - sq->dbr_offset = -1; - } - if (sq->obj) { - mlx5_devx_cmd_destroy(sq->obj); - sq->obj = NULL; - } - return 0; -} - -/** * Setup the qp. * * @param dev @@ -329,7 +270,7 @@ mlx5_mr_btree_free(&qp->mr_ctrl.cache_bh); err_btree: for (i = 0; i < nb_sq_config; i++) - regex_ctrl_destroy_sq(priv, qp, i); + regex_ctrl_destroy_sq(qp, i); regex_ctrl_destroy_cq(&qp->cq); err_cq: rte_free(qp->sqs); diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c b/drivers/regex/mlx5/mlx5_regex_fastpath.c index 255fd40..cd0f9bd 100644 --- a/drivers/regex/mlx5/mlx5_regex_fastpath.c +++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c @@ -110,12 +110,12 @@ struct mlx5_regex_job { &priv->mr_scache, &qp->mr_ctrl, rte_pktmbuf_mtod(op->mbuf, uintptr_t), !!(op->mbuf->ol_flags & EXT_ATTACHED_MBUF)); - uint8_t *wqe = (uint8_t *)sq->wqe + wqe_offset; + uint8_t *wqe = (uint8_t *)(uintptr_t)sq->sq_obj.wqes + wqe_offset; int ds = 4; /* ctrl + meta + input + output */ set_wqe_ctrl_seg((struct mlx5_wqe_ctrl_seg *)wqe, sq->pi, - MLX5_OPCODE_MMO, MLX5_OPC_MOD_MMO_REGEX, sq->obj->id, - 0, ds, 0, 0); + MLX5_OPCODE_MMO, MLX5_OPC_MOD_MMO_REGEX, + sq->sq_obj.sq->id, 0, ds, 0, 0); set_regex_ctrl_seg(wqe + 12, 0, op->group_id0, op->group_id1, op->group_id2, op->group_id3, 0); @@ -137,12 +137,12 @@ struct mlx5_regex_job { { size_t wqe_offset = (sq->db_pi & (sq_size_get(sq) - 1)) * MLX5_SEND_WQE_BB; - uint8_t *wqe = (uint8_t *)sq->wqe + wqe_offset; + uint8_t *wqe = (uint8_t *)(uintptr_t)sq->sq_obj.wqes + wqe_offset; ((struct mlx5_wqe_ctrl_seg *)wqe)->fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE; uint64_t *doorbell_addr = (uint64_t *)((uint8_t *)uar->base_addr + 0x800); rte_io_wmb(); - sq->dbr[MLX5_SND_DBR] = rte_cpu_to_be_32((sq->db_pi + 1) & + sq->sq_obj.db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32((sq->db_pi + 1) & MLX5_REGEX_MAX_WQE_INDEX); rte_wmb(); *doorbell_addr = *(volatile uint64_t *)wqe; @@ -301,7 +301,7 @@ struct mlx5_regex_job { uint32_t job_id; for (sqid = 0; sqid < queue->nb_obj; sqid++) { struct mlx5_regex_sq *sq = &queue->sqs[sqid]; - uint8_t *wqe = (uint8_t *)sq->wqe; + uint8_t *wqe = (uint8_t *)(uintptr_t)sq->sq_obj.wqes; for (entry = 0 ; entry < sq_size_get(sq); entry++) { job_id = sqid * sq_size_get(sq) + entry; struct mlx5_regex_job *job = &queue->jobs[job_id]; @@ -334,7 +334,7 @@ struct mlx5_regex_job { return -ENOMEM; qp->metadata = mlx5_glue->reg_mr(pd, ptr, - MLX5_REGEX_METADATA_SIZE*qp->nb_desc, + MLX5_REGEX_METADATA_SIZE * qp->nb_desc, IBV_ACCESS_LOCAL_WRITE); if (!qp->metadata) { DRV_LOG(ERR, "Failed to register metadata"); From patchwork Thu Dec 17 11:44:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85324 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 52658A09F6; Thu, 17 Dec 2020 12:50:13 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 76F6CCAD6; Thu, 17 Dec 2020 12:45:21 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 0C0BCCA32 for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:51 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2b004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:30 +0000 Message-Id: <1608205475-20067-13-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 12/17] net/mlx5: move rearm and clock queue SQ creation to common X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Using common function for DevX SQ creation for rearm and clock queue. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5.h | 8 +-- drivers/net/mlx5/mlx5_txpp.c | 147 +++++++++++-------------------------------- 2 files changed, 36 insertions(+), 119 deletions(-) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 9a59c26..192a5a7 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -611,15 +611,9 @@ struct mlx5_txpp_wq { uint32_t cq_ci:24; uint32_t arm_sn:2; /* Send Queue related data.*/ - struct mlx5_devx_obj *sq; - void *sq_umem; - union { - volatile void *sq_buf; - volatile struct mlx5_wqe *wqes; - }; + struct mlx5_devx_sq sq_obj; uint16_t sq_size; /* Number of WQEs in the queue. */ uint16_t sq_ci; /* Next WQE to execute. */ - volatile uint32_t *sq_dbrec; }; /* Tx packet pacing internal timestamp. */ diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c index 54ea572..b6ff7e0 100644 --- a/drivers/net/mlx5/mlx5_txpp.c +++ b/drivers/net/mlx5/mlx5_txpp.c @@ -121,12 +121,7 @@ static void mlx5_txpp_destroy_send_queue(struct mlx5_txpp_wq *wq) { - if (wq->sq) - claim_zero(mlx5_devx_cmd_destroy(wq->sq)); - if (wq->sq_umem) - claim_zero(mlx5_glue->devx_umem_dereg(wq->sq_umem)); - if (wq->sq_buf) - mlx5_free((void *)(uintptr_t)wq->sq_buf); + mlx5_devx_sq_destroy(&wq->sq_obj); mlx5_devx_cq_destroy(&wq->cq_obj); memset(wq, 0, sizeof(*wq)); } @@ -155,6 +150,7 @@ mlx5_txpp_doorbell_rearm_queue(struct mlx5_dev_ctx_shared *sh, uint16_t ci) { struct mlx5_txpp_wq *wq = &sh->txpp.rearm_queue; + struct mlx5_wqe *wqe = (struct mlx5_wqe *)(uintptr_t)wq->sq_obj.wqes; union { uint32_t w32[2]; uint64_t w64; @@ -163,11 +159,11 @@ wq->sq_ci = ci + 1; cs.w32[0] = rte_cpu_to_be_32(rte_be_to_cpu_32 - (wq->wqes[ci & (wq->sq_size - 1)].ctrl[0]) | (ci - 1) << 8); - cs.w32[1] = wq->wqes[ci & (wq->sq_size - 1)].ctrl[1]; + (wqe[ci & (wq->sq_size - 1)].ctrl[0]) | (ci - 1) << 8); + cs.w32[1] = wqe[ci & (wq->sq_size - 1)].ctrl[1]; /* Update SQ doorbell record with new SQ ci. */ rte_compiler_barrier(); - *wq->sq_dbrec = rte_cpu_to_be_32(wq->sq_ci); + *wq->sq_obj.db_rec = rte_cpu_to_be_32(wq->sq_ci); /* Make sure the doorbell record is updated. */ rte_wmb(); /* Write to doorbel register to start processing. */ @@ -180,7 +176,7 @@ mlx5_txpp_fill_wqe_rearm_queue(struct mlx5_dev_ctx_shared *sh) { struct mlx5_txpp_wq *wq = &sh->txpp.rearm_queue; - struct mlx5_wqe *wqe = (struct mlx5_wqe *)(uintptr_t)wq->wqes; + struct mlx5_wqe *wqe = (struct mlx5_wqe *)(uintptr_t)wq->sq_obj.wqes; uint32_t i; for (i = 0; i < wq->sq_size; i += 2) { @@ -191,7 +187,7 @@ /* Build SEND_EN request with slave WQE index. */ cs = &wqe[i + 0].cseg; cs->opcode = RTE_BE32(MLX5_OPCODE_SEND_EN | 0); - cs->sq_ds = rte_cpu_to_be_32((wq->sq->id << 8) | 2); + cs->sq_ds = rte_cpu_to_be_32((wq->sq_obj.sq->id << 8) | 2); cs->flags = RTE_BE32(MLX5_COMP_ALWAYS << MLX5_COMP_MODE_OFFSET); cs->misc = RTE_BE32(0); @@ -199,11 +195,12 @@ index = (i * MLX5_TXPP_REARM / 2 + MLX5_TXPP_REARM) & ((1 << MLX5_WQ_INDEX_WIDTH) - 1); qs->max_index = rte_cpu_to_be_32(index); - qs->qpn_cqn = rte_cpu_to_be_32(sh->txpp.clock_queue.sq->id); + qs->qpn_cqn = + rte_cpu_to_be_32(sh->txpp.clock_queue.sq_obj.sq->id); /* Build WAIT request with slave CQE index. */ cs = &wqe[i + 1].cseg; cs->opcode = RTE_BE32(MLX5_OPCODE_WAIT | 0); - cs->sq_ds = rte_cpu_to_be_32((wq->sq->id << 8) | 2); + cs->sq_ds = rte_cpu_to_be_32((wq->sq_obj.sq->id << 8) | 2); cs->flags = RTE_BE32(MLX5_COMP_ONLY_ERR << MLX5_COMP_MODE_OFFSET); cs->misc = RTE_BE32(0); @@ -220,7 +217,16 @@ static int mlx5_txpp_create_rearm_queue(struct mlx5_dev_ctx_shared *sh) { - struct mlx5_devx_create_sq_attr sq_attr = { 0 }; + struct mlx5_devx_create_sq_attr sq_attr = { + .cd_master = 1, + .state = MLX5_SQC_STATE_RST, + .tis_lst_sz = 1, + .tis_num = sh->tis->id, + .wq_attr = (struct mlx5_devx_wq_attr){ + .pd = sh->pdn, + .uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar), + }, + }; struct mlx5_devx_modify_sq_attr msq_attr = { 0 }; struct mlx5_devx_cq_attr cq_attr = { .cqe_size = (sizeof(struct mlx5_cqe) == 128) ? @@ -228,15 +234,8 @@ .uar_page_id = mlx5_os_get_devx_uar_page_id(sh->tx_uar), }; struct mlx5_txpp_wq *wq = &sh->txpp.rearm_queue; - size_t page_size; - uint32_t umem_size, umem_dbrec; int ret; - page_size = rte_mem_page_size(); - if (page_size == (size_t)-1) { - DRV_LOG(ERR, "Failed to get mem page size"); - return -ENOMEM; - } /* Create completion queue object for Rearm Queue. */ ret = mlx5_devx_cq_create(sh->ctx, &wq->cq_obj, log2above(MLX5_TXPP_REARM_CQ_SIZE), &cq_attr, @@ -247,63 +246,25 @@ } wq->cq_ci = 0; wq->arm_sn = 0; - /* - * Allocate memory buffer for Send Queue WQEs. - * There should be no WQE leftovers in the cyclic queue. - */ wq->sq_size = MLX5_TXPP_REARM_SQ_SIZE; MLX5_ASSERT(wq->sq_size == (1 << log2above(wq->sq_size))); - umem_size = MLX5_WQE_SIZE * wq->sq_size; - umem_dbrec = RTE_ALIGN(umem_size, MLX5_DBR_SIZE); - umem_size += MLX5_DBR_SIZE; - wq->sq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, umem_size, - page_size, sh->numa_node); - if (!wq->sq_buf) { - DRV_LOG(ERR, "Failed to allocate memory for Rearm Queue."); - rte_errno = ENOMEM; - goto error; - } - /* Register allocated buffer in user space with DevX. */ - wq->sq_umem = mlx5_glue->devx_umem_reg(sh->ctx, - (void *)(uintptr_t)wq->sq_buf, - umem_size, - IBV_ACCESS_LOCAL_WRITE); - if (!wq->sq_umem) { - rte_errno = errno; - DRV_LOG(ERR, "Failed to register umem for Rearm Queue."); - goto error; - } /* Create send queue object for Rearm Queue. */ - sq_attr.state = MLX5_SQC_STATE_RST; - sq_attr.tis_lst_sz = 1; - sq_attr.tis_num = sh->tis->id; sq_attr.cqn = wq->cq_obj.cq->id; - sq_attr.cd_master = 1; - sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar); - sq_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC; - sq_attr.wq_attr.pd = sh->pdn; - sq_attr.wq_attr.log_wq_stride = rte_log2_u32(MLX5_WQE_SIZE); - sq_attr.wq_attr.log_wq_sz = rte_log2_u32(wq->sq_size); - sq_attr.wq_attr.dbr_umem_valid = 1; - sq_attr.wq_attr.dbr_addr = umem_dbrec; - sq_attr.wq_attr.dbr_umem_id = mlx5_os_get_umem_id(wq->sq_umem); - sq_attr.wq_attr.wq_umem_valid = 1; - sq_attr.wq_attr.wq_umem_id = mlx5_os_get_umem_id(wq->sq_umem); - sq_attr.wq_attr.wq_umem_offset = 0; - wq->sq = mlx5_devx_cmd_create_sq(sh->ctx, &sq_attr); - if (!wq->sq) { + /* There should be no WQE leftovers in the cyclic queue. */ + ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj, + log2above(MLX5_TXPP_REARM_SQ_SIZE), &sq_attr, + sh->numa_node); + if (ret) { rte_errno = errno; DRV_LOG(ERR, "Failed to create SQ for Rearm Queue."); goto error; } - wq->sq_dbrec = RTE_PTR_ADD(wq->sq_buf, umem_dbrec + - MLX5_SND_DBR * sizeof(uint32_t)); /* Build the WQEs in the Send Queue before goto Ready state. */ mlx5_txpp_fill_wqe_rearm_queue(sh); /* Change queue state to ready. */ msq_attr.sq_state = MLX5_SQC_STATE_RST; msq_attr.state = MLX5_SQC_STATE_RDY; - ret = mlx5_devx_cmd_modify_sq(wq->sq, &msq_attr); + ret = mlx5_devx_cmd_modify_sq(wq->sq_obj.sq, &msq_attr); if (ret) { DRV_LOG(ERR, "Failed to set SQ ready state Rearm Queue."); goto error; @@ -320,7 +281,7 @@ mlx5_txpp_fill_wqe_clock_queue(struct mlx5_dev_ctx_shared *sh) { struct mlx5_txpp_wq *wq = &sh->txpp.clock_queue; - struct mlx5_wqe *wqe = (struct mlx5_wqe *)(uintptr_t)wq->wqes; + struct mlx5_wqe *wqe = (struct mlx5_wqe *)(uintptr_t)wq->sq_obj.wqes; struct mlx5_wqe_cseg *cs = &wqe->cseg; uint32_t wqe_size, opcode, i; uint8_t *dst; @@ -338,7 +299,7 @@ opcode = MLX5_OPCODE_NOP; } cs->opcode = rte_cpu_to_be_32(opcode | 0); /* Index is ignored. */ - cs->sq_ds = rte_cpu_to_be_32((wq->sq->id << 8) | + cs->sq_ds = rte_cpu_to_be_32((wq->sq_obj.sq->id << 8) | (wqe_size / MLX5_WSEG_SIZE)); cs->flags = RTE_BE32(MLX5_COMP_ALWAYS << MLX5_COMP_MODE_OFFSET); cs->misc = RTE_BE32(0); @@ -407,10 +368,11 @@ } wcopy: /* Duplicate the pattern to the next WQEs. */ - dst = (uint8_t *)(uintptr_t)wq->sq_buf; + dst = (uint8_t *)(uintptr_t)wq->sq_obj.umem_buf; for (i = 1; i < MLX5_TXPP_CLKQ_SIZE; i++) { dst += wqe_size; - rte_memcpy(dst, (void *)(uintptr_t)wq->sq_buf, wqe_size); + rte_memcpy(dst, (void *)(uintptr_t)wq->sq_obj.umem_buf, + wqe_size); } } @@ -428,15 +390,8 @@ .uar_page_id = mlx5_os_get_devx_uar_page_id(sh->tx_uar), }; struct mlx5_txpp_wq *wq = &sh->txpp.clock_queue; - size_t page_size; - uint32_t umem_size, umem_dbrec; int ret; - page_size = rte_mem_page_size(); - if (page_size == (size_t)-1) { - DRV_LOG(ERR, "Failed to get mem page size"); - return -ENOMEM; - } sh->txpp.tsa = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, MLX5_TXPP_REARM_SQ_SIZE * sizeof(struct mlx5_txpp_ts), @@ -469,26 +424,6 @@ } /* There should not be WQE leftovers in the cyclic queue. */ MLX5_ASSERT(wq->sq_size == (1 << log2above(wq->sq_size))); - umem_size = MLX5_WQE_SIZE * wq->sq_size; - umem_dbrec = RTE_ALIGN(umem_size, MLX5_DBR_SIZE); - umem_size += MLX5_DBR_SIZE; - wq->sq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, umem_size, - page_size, sh->numa_node); - if (!wq->sq_buf) { - DRV_LOG(ERR, "Failed to allocate memory for Clock Queue."); - rte_errno = ENOMEM; - goto error; - } - /* Register allocated buffer in user space with DevX. */ - wq->sq_umem = mlx5_glue->devx_umem_reg(sh->ctx, - (void *)(uintptr_t)wq->sq_buf, - umem_size, - IBV_ACCESS_LOCAL_WRITE); - if (!wq->sq_umem) { - rte_errno = errno; - DRV_LOG(ERR, "Failed to register umem for Clock Queue."); - goto error; - } /* Create send queue object for Clock Queue. */ if (sh->txpp.test) { sq_attr.tis_lst_sz = 1; @@ -499,37 +434,25 @@ sq_attr.non_wire = 1; sq_attr.static_sq_wq = 1; } - sq_attr.state = MLX5_SQC_STATE_RST; sq_attr.cqn = wq->cq_obj.cq->id; sq_attr.packet_pacing_rate_limit_index = sh->txpp.pp_id; sq_attr.wq_attr.cd_slave = 1; sq_attr.wq_attr.uar_page = mlx5_os_get_devx_uar_page_id(sh->tx_uar); - sq_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC; sq_attr.wq_attr.pd = sh->pdn; - sq_attr.wq_attr.log_wq_stride = rte_log2_u32(MLX5_WQE_SIZE); - sq_attr.wq_attr.log_wq_sz = rte_log2_u32(wq->sq_size); - sq_attr.wq_attr.dbr_umem_valid = 1; - sq_attr.wq_attr.dbr_addr = umem_dbrec; - sq_attr.wq_attr.dbr_umem_id = mlx5_os_get_umem_id(wq->sq_umem); - sq_attr.wq_attr.wq_umem_valid = 1; - sq_attr.wq_attr.wq_umem_id = mlx5_os_get_umem_id(wq->sq_umem); - /* umem_offset must be zero for static_sq_wq queue. */ - sq_attr.wq_attr.wq_umem_offset = 0; - wq->sq = mlx5_devx_cmd_create_sq(sh->ctx, &sq_attr); - if (!wq->sq) { + ret = mlx5_devx_sq_create(sh->ctx, &wq->sq_obj, log2above(wq->sq_size), + &sq_attr, sh->numa_node); + if (ret) { rte_errno = errno; DRV_LOG(ERR, "Failed to create SQ for Clock Queue."); goto error; } - wq->sq_dbrec = RTE_PTR_ADD(wq->sq_buf, umem_dbrec + - MLX5_SND_DBR * sizeof(uint32_t)); /* Build the WQEs in the Send Queue before goto Ready state. */ mlx5_txpp_fill_wqe_clock_queue(sh); /* Change queue state to ready. */ msq_attr.sq_state = MLX5_SQC_STATE_RST; msq_attr.state = MLX5_SQC_STATE_RDY; wq->sq_ci = 0; - ret = mlx5_devx_cmd_modify_sq(wq->sq, &msq_attr); + ret = mlx5_devx_cmd_modify_sq(wq->sq_obj.sq, &msq_attr); if (ret) { DRV_LOG(ERR, "Failed to set SQ ready state Clock Queue."); goto error; From patchwork Thu Dec 17 11:44:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85323 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 725EDA09F6; Thu, 17 Dec 2020 12:49:58 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8C305CACE; Thu, 17 Dec 2020 12:45:20 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id E584DCA2E for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:51 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2c004524; Thu, 17 Dec 2020 13:44:51 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:31 +0000 Message-Id: <1608205475-20067-14-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 13/17] net/mlx5: move Tx SQ creation to common X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Using common function for Tx SQ creation. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5.h | 8 +-- drivers/net/mlx5/mlx5_devx.c | 160 ++++++++++--------------------------------- 2 files changed, 40 insertions(+), 128 deletions(-) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 192a5a7..6977eac 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -837,11 +837,9 @@ struct mlx5_txq_obj { struct { struct rte_eth_dev *dev; struct mlx5_devx_cq cq_obj; - struct mlx5_devx_obj *sq_devx; - void *sq_umem; - void *sq_buf; - int64_t sq_dbrec_offset; - struct mlx5_devx_dbr_page *sq_dbrec_page; + /* DevX CQ object and its resources. */ + struct mlx5_devx_sq sq_obj; + /* DevX SQ object and its resources. */ }; }; }; diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index fe103a7..4154c52 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -115,7 +115,7 @@ else msq_attr.sq_state = MLX5_SQC_STATE_RDY; msq_attr.state = MLX5_SQC_STATE_RST; - ret = mlx5_devx_cmd_modify_sq(obj->sq_devx, &msq_attr); + ret = mlx5_devx_cmd_modify_sq(obj->sq_obj.sq, &msq_attr); if (ret) { DRV_LOG(ERR, "Cannot change the Tx SQ state to RESET" " %s", strerror(errno)); @@ -127,7 +127,7 @@ /* Change queue state to ready. */ msq_attr.sq_state = MLX5_SQC_STATE_RST; msq_attr.state = MLX5_SQC_STATE_RDY; - ret = mlx5_devx_cmd_modify_sq(obj->sq_devx, &msq_attr); + ret = mlx5_devx_cmd_modify_sq(obj->sq_obj.sq, &msq_attr); if (ret) { DRV_LOG(ERR, "Cannot change the Tx SQ state to READY" " %s", strerror(errno)); @@ -1056,36 +1056,6 @@ #ifdef HAVE_MLX5DV_DEVX_UAR_OFFSET /** - * Release DevX SQ resources. - * - * @param txq_obj - * DevX Tx queue object. - */ -static void -mlx5_txq_release_devx_sq_resources(struct mlx5_txq_obj *txq_obj) -{ - if (txq_obj->sq_devx) { - claim_zero(mlx5_devx_cmd_destroy(txq_obj->sq_devx)); - txq_obj->sq_devx = NULL; - } - if (txq_obj->sq_umem) { - claim_zero(mlx5_glue->devx_umem_dereg(txq_obj->sq_umem)); - txq_obj->sq_umem = NULL; - } - if (txq_obj->sq_buf) { - mlx5_free(txq_obj->sq_buf); - txq_obj->sq_buf = NULL; - } - if (txq_obj->sq_dbrec_page) { - claim_zero(mlx5_release_dbr(&txq_obj->txq_ctrl->priv->dbrpgs, - mlx5_os_get_umem_id - (txq_obj->sq_dbrec_page->umem), - txq_obj->sq_dbrec_offset)); - txq_obj->sq_dbrec_page = NULL; - } -} - -/** * Destroy the Tx queue DevX object. * * @param txq_obj @@ -1094,7 +1064,8 @@ static void mlx5_txq_release_devx_resources(struct mlx5_txq_obj *txq_obj) { - mlx5_txq_release_devx_sq_resources(txq_obj); + mlx5_devx_sq_destroy(&txq_obj->sq_obj); + memset(&txq_obj->sq_obj, 0, sizeof(txq_obj->sq_obj)); mlx5_devx_cq_destroy(&txq_obj->cq_obj); memset(&txq_obj->cq_obj, 0, sizeof(txq_obj->cq_obj)); } @@ -1106,100 +1077,41 @@ * Pointer to Ethernet device. * @param idx * Queue index in DPDK Tx queue array. + * @param[in] log_desc_n + * Log of number of descriptors in queue. * * @return - * Number of WQEs in SQ, 0 otherwise and rte_errno is set. + * 0 on success, a negative errno value otherwise and rte_errno is set. */ -static uint32_t -mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx) +static int +mlx5_txq_create_devx_sq_resources(struct rte_eth_dev *dev, uint16_t idx, + uint16_t log_desc_n) { struct mlx5_priv *priv = dev->data->dev_private; struct mlx5_txq_data *txq_data = (*priv->txqs)[idx]; struct mlx5_txq_ctrl *txq_ctrl = container_of(txq_data, struct mlx5_txq_ctrl, txq); struct mlx5_txq_obj *txq_obj = txq_ctrl->obj; - struct mlx5_devx_create_sq_attr sq_attr = { 0 }; - size_t page_size; - uint32_t wqe_n; - int ret; + struct mlx5_devx_create_sq_attr sq_attr = { + .flush_in_error_en = 1, + .allow_multi_pkt_send_wqe = !!priv->config.mps, + .min_wqe_inline_mode = priv->config.hca_attr.vport_inline_mode, + .allow_swp = !!priv->config.swp, + .cqn = txq_obj->cq_obj.cq->id, + .tis_lst_sz = 1, + .tis_num = priv->sh->tis->id, + .wq_attr = (struct mlx5_devx_wq_attr){ + .pd = priv->sh->pdn, + .uar_page = + mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar), + }, + }; MLX5_ASSERT(txq_data); MLX5_ASSERT(txq_obj); - page_size = rte_mem_page_size(); - if (page_size == (size_t)-1) { - DRV_LOG(ERR, "Failed to get mem page size."); - rte_errno = ENOMEM; - return 0; - } - wqe_n = RTE_MIN(1UL << txq_data->elts_n, - (uint32_t)priv->sh->device_attr.max_qp_wr); - txq_obj->sq_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - wqe_n * sizeof(struct mlx5_wqe), - page_size, priv->sh->numa_node); - if (!txq_obj->sq_buf) { - DRV_LOG(ERR, - "Port %u Tx queue %u cannot allocate memory (SQ).", - dev->data->port_id, txq_data->idx); - rte_errno = ENOMEM; - goto error; - } - /* Register allocated buffer in user space with DevX. */ - txq_obj->sq_umem = mlx5_glue->devx_umem_reg - (priv->sh->ctx, - (void *)txq_obj->sq_buf, - wqe_n * sizeof(struct mlx5_wqe), - IBV_ACCESS_LOCAL_WRITE); - if (!txq_obj->sq_umem) { - rte_errno = errno; - DRV_LOG(ERR, - "Port %u Tx queue %u cannot register memory (SQ).", - dev->data->port_id, txq_data->idx); - goto error; - } - /* Allocate doorbell record for send queue. */ - txq_obj->sq_dbrec_offset = mlx5_get_dbr(priv->sh->ctx, - &priv->dbrpgs, - &txq_obj->sq_dbrec_page); - if (txq_obj->sq_dbrec_offset < 0) { - rte_errno = errno; - DRV_LOG(ERR, "Failed to allocate SQ door-bell."); - goto error; - } - sq_attr.tis_lst_sz = 1; - sq_attr.tis_num = priv->sh->tis->id; - sq_attr.state = MLX5_SQC_STATE_RST; - sq_attr.cqn = txq_obj->cq_obj.cq->id; - sq_attr.flush_in_error_en = 1; - sq_attr.allow_multi_pkt_send_wqe = !!priv->config.mps; - sq_attr.allow_swp = !!priv->config.swp; - sq_attr.min_wqe_inline_mode = priv->config.hca_attr.vport_inline_mode; - sq_attr.wq_attr.uar_page = - mlx5_os_get_devx_uar_page_id(priv->sh->tx_uar); - sq_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC; - sq_attr.wq_attr.pd = priv->sh->pdn; - sq_attr.wq_attr.log_wq_stride = rte_log2_u32(MLX5_WQE_SIZE); - sq_attr.wq_attr.log_wq_sz = log2above(wqe_n); - sq_attr.wq_attr.dbr_umem_valid = 1; - sq_attr.wq_attr.dbr_addr = txq_obj->sq_dbrec_offset; - sq_attr.wq_attr.dbr_umem_id = - mlx5_os_get_umem_id(txq_obj->sq_dbrec_page->umem); - sq_attr.wq_attr.wq_umem_valid = 1; - sq_attr.wq_attr.wq_umem_id = mlx5_os_get_umem_id(txq_obj->sq_umem); - sq_attr.wq_attr.wq_umem_offset = (uintptr_t)txq_obj->sq_buf % page_size; /* Create Send Queue object with DevX. */ - txq_obj->sq_devx = mlx5_devx_cmd_create_sq(priv->sh->ctx, &sq_attr); - if (!txq_obj->sq_devx) { - rte_errno = errno; - DRV_LOG(ERR, "Port %u Tx queue %u SQ creation failure.", - dev->data->port_id, idx); - goto error; - } - return wqe_n; -error: - ret = rte_errno; - mlx5_txq_release_devx_sq_resources(txq_obj); - rte_errno = ret; - return 0; + return mlx5_devx_sq_create(priv->sh->ctx, &txq_obj->sq_obj, log_desc_n, + &sq_attr, priv->sh->numa_node); } #endif @@ -1273,27 +1185,29 @@ txq_data->cq_db = txq_obj->cq_obj.db_rec; *txq_data->cq_db = 0; /* Create Send Queue object with DevX. */ - wqe_n = mlx5_txq_create_devx_sq_resources(dev, idx); - if (!wqe_n) { + wqe_n = RTE_MIN(1UL << txq_data->elts_n, + (uint32_t)priv->sh->device_attr.max_qp_wr); + log_desc_n = log2above(wqe_n); + ret = mlx5_txq_create_devx_sq_resources(dev, idx, log_desc_n); + if (ret) { + DRV_LOG(ERR, "Port %u Tx queue %u SQ creation failure.", + dev->data->port_id, idx); rte_errno = errno; goto error; } /* Create the Work Queue. */ - txq_data->wqe_n = log2above(wqe_n); + txq_data->wqe_n = log_desc_n; txq_data->wqe_s = 1 << txq_data->wqe_n; txq_data->wqe_m = txq_data->wqe_s - 1; - txq_data->wqes = (struct mlx5_wqe *)txq_obj->sq_buf; + txq_data->wqes = (struct mlx5_wqe *)(uintptr_t)txq_obj->sq_obj.wqes; txq_data->wqes_end = txq_data->wqes + txq_data->wqe_s; txq_data->wqe_ci = 0; txq_data->wqe_pi = 0; txq_data->wqe_comp = 0; txq_data->wqe_thres = txq_data->wqe_s / MLX5_TX_COMP_THRESH_INLINE_DIV; - txq_data->qp_db = (volatile uint32_t *) - (txq_obj->sq_dbrec_page->dbrs + - txq_obj->sq_dbrec_offset + - MLX5_SND_DBR * sizeof(uint32_t)); + txq_data->qp_db = txq_obj->sq_obj.db_rec; *txq_data->qp_db = 0; - txq_data->qp_num_8s = txq_obj->sq_devx->id << 8; + txq_data->qp_num_8s = txq_obj->sq_obj.sq->id << 8; /* Change Send Queue state to Ready-to-Send. */ ret = mlx5_devx_modify_sq(txq_obj, MLX5_TXQ_MOD_RST2RDY, 0); if (ret) { From patchwork Thu Dec 17 11:44:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85320 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9422DA09F6; Thu, 17 Dec 2020 12:49:04 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DCFA9CAB2; Thu, 17 Dec 2020 12:45:16 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id E1AAACA2C for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:51 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2d004524; Thu, 17 Dec 2020 13:44:51 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:32 +0000 Message-Id: <1608205475-20067-15-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 14/17] net/mlx5: move ASO SQ creation to common X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Using common function for ASO SQ creation. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/common/mlx5/mlx5_common_devx.h | 1 + drivers/net/mlx5/mlx5.h | 8 +-- drivers/net/mlx5/mlx5_flow_age.c | 94 ++++++++++------------------------ 3 files changed, 30 insertions(+), 73 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common_devx.h b/drivers/common/mlx5/mlx5_common_devx.h index 88d520b..8377d34 100644 --- a/drivers/common/mlx5/mlx5_common_devx.h +++ b/drivers/common/mlx5/mlx5_common_devx.h @@ -25,6 +25,7 @@ struct mlx5_devx_sq { union { volatile void *umem_buf; volatile struct mlx5_wqe *wqes; /* The SQ ring buffer. */ + volatile struct mlx5_aso_wqe *aso_wqes; }; volatile uint32_t *db_rec; /* The SQ doorbell record. */ }; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 6977eac..86ada23 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -483,13 +483,7 @@ struct mlx5_aso_sq_elem { struct mlx5_aso_sq { uint16_t log_desc_n; struct mlx5_aso_cq cq; - struct mlx5_devx_obj *sq; - struct mlx5dv_devx_umem *wqe_umem; /* SQ buffer umem. */ - union { - volatile void *umem_buf; - volatile struct mlx5_aso_wqe *wqes; - }; - volatile uint32_t *db_rec; + struct mlx5_devx_sq sq_obj; volatile uint64_t *uar_addr; struct mlx5_aso_devx_mr mr; uint16_t pi; diff --git a/drivers/net/mlx5/mlx5_flow_age.c b/drivers/net/mlx5/mlx5_flow_age.c index 60a8d2a..9681cbf 100644 --- a/drivers/net/mlx5/mlx5_flow_age.c +++ b/drivers/net/mlx5/mlx5_flow_age.c @@ -141,18 +141,7 @@ static void mlx5_aso_destroy_sq(struct mlx5_aso_sq *sq) { - if (sq->wqe_umem) { - mlx5_glue->devx_umem_dereg(sq->wqe_umem); - sq->wqe_umem = NULL; - } - if (sq->umem_buf) { - mlx5_free((void *)(uintptr_t)sq->umem_buf); - sq->umem_buf = NULL; - } - if (sq->sq) { - mlx5_devx_cmd_destroy(sq->sq); - sq->sq = NULL; - } + mlx5_devx_sq_destroy(&sq->sq_obj); mlx5_aso_cq_destroy(&sq->cq); mlx5_aso_devx_dereg_mr(&sq->mr); memset(sq, 0, sizeof(*sq)); @@ -173,7 +162,7 @@ uint64_t addr; /* All the next fields state should stay constant. */ - for (i = 0, wqe = &sq->wqes[0]; i < size; ++i, ++wqe) { + for (i = 0, wqe = &sq->sq_obj.aso_wqes[0]; i < size; ++i, ++wqe) { wqe->general_cseg.sq_ds = rte_cpu_to_be_32((sq->sqn << 8) | (sizeof(*wqe) >> 4)); wqe->aso_cseg.lkey = rte_cpu_to_be_32(sq->mr.mkey->id); @@ -215,12 +204,18 @@ struct mlx5dv_devx_uar *uar, uint32_t pdn, uint16_t log_desc_n) { - struct mlx5_devx_create_sq_attr attr = { 0 }; - struct mlx5_devx_modify_sq_attr modify_attr = { 0 }; - size_t pgsize = sysconf(_SC_PAGESIZE); - struct mlx5_devx_wq_attr *wq_attr = &attr.wq_attr; + struct mlx5_devx_create_sq_attr attr = { + .user_index = 0xFFFF, + .wq_attr = (struct mlx5_devx_wq_attr){ + .pd = pdn, + .uar_page = mlx5_os_get_devx_uar_page_id(uar), + }, + }; + struct mlx5_devx_modify_sq_attr modify_attr = { + .state = MLX5_SQC_STATE_RDY, + }; uint32_t sq_desc_n = 1 << log_desc_n; - uint32_t wq_size = sizeof(struct mlx5_aso_wqe) * sq_desc_n; + uint16_t log_wqbb_n; int ret; if (mlx5_aso_devx_reg_mr(ctx, (MLX5_ASO_AGE_ACTIONS_PER_POOL / 8) * @@ -230,58 +225,25 @@ mlx5_os_get_devx_uar_page_id(uar))) goto error; sq->log_desc_n = log_desc_n; - sq->umem_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, wq_size + - sizeof(*sq->db_rec) * 2, 4096, socket); - if (!sq->umem_buf) { - DRV_LOG(ERR, "Can't allocate wqe buffer."); - rte_errno = ENOMEM; - goto error; - } - sq->wqe_umem = mlx5_glue->devx_umem_reg(ctx, - (void *)(uintptr_t)sq->umem_buf, - wq_size + - sizeof(*sq->db_rec) * 2, - IBV_ACCESS_LOCAL_WRITE); - if (!sq->wqe_umem) { - DRV_LOG(ERR, "Failed to register umem for SQ."); - rte_errno = ENOMEM; - goto error; - } - attr.state = MLX5_SQC_STATE_RST; - attr.tis_lst_sz = 0; - attr.tis_num = 0; - attr.user_index = 0xFFFF; attr.cqn = sq->cq.cq_obj.cq->id; - wq_attr->uar_page = mlx5_os_get_devx_uar_page_id(uar); - wq_attr->pd = pdn; - wq_attr->wq_type = MLX5_WQ_TYPE_CYCLIC; - wq_attr->log_wq_pg_sz = rte_log2_u32(pgsize); - wq_attr->wq_umem_id = mlx5_os_get_umem_id(sq->wqe_umem); - wq_attr->wq_umem_offset = 0; - wq_attr->wq_umem_valid = 1; - wq_attr->log_wq_stride = 6; - wq_attr->log_wq_sz = rte_log2_u32(wq_size) - 6; - wq_attr->dbr_umem_id = wq_attr->wq_umem_id; - wq_attr->dbr_addr = wq_size; - wq_attr->dbr_umem_valid = 1; - sq->sq = mlx5_devx_cmd_create_sq(ctx, &attr); - if (!sq->sq) { - DRV_LOG(ERR, "Can't create sq object."); - rte_errno = ENOMEM; + /* for mlx5_aso_wqe that is twice the size of mlx5_wqe */ + log_wqbb_n = log_desc_n + 1; + ret = mlx5_devx_sq_create(ctx, &sq->sq_obj, log_wqbb_n, &attr, socket); + if (ret) { + DRV_LOG(ERR, "Can't create SQ object."); + rte_errno = ENOMEM; goto error; } - modify_attr.state = MLX5_SQC_STATE_RDY; - ret = mlx5_devx_cmd_modify_sq(sq->sq, &modify_attr); + ret = mlx5_devx_cmd_modify_sq(sq->sq_obj.sq, &modify_attr); if (ret) { - DRV_LOG(ERR, "Can't change sq state to ready."); - rte_errno = ENOMEM; + DRV_LOG(ERR, "Can't change SQ state to ready."); + rte_errno = ENOMEM; goto error; } sq->pi = 0; sq->head = 0; sq->tail = 0; - sq->sqn = sq->sq->id; - sq->db_rec = RTE_PTR_ADD(sq->umem_buf, (uintptr_t)(wq_attr->dbr_addr)); + sq->sqn = sq->sq_obj.sq->id; sq->uar_addr = (volatile uint64_t *)((uint8_t *)uar->base_addr + 0x800); mlx5_aso_init_sq(sq); return 0; @@ -345,8 +307,8 @@ return 0; sq->elts[start_head & mask].burst_size = max; do { - wqe = &sq->wqes[sq->head & mask]; - rte_prefetch0(&sq->wqes[(sq->head + 1) & mask]); + wqe = &sq->sq_obj.aso_wqes[sq->head & mask]; + rte_prefetch0(&sq->sq_obj.aso_wqes[(sq->head + 1) & mask]); /* Fill next WQE. */ rte_spinlock_lock(&mng->resize_sl); pool = mng->pools[sq->next]; @@ -371,7 +333,7 @@ wqe->general_cseg.flags = RTE_BE32(MLX5_COMP_ALWAYS << MLX5_COMP_MODE_OFFSET); rte_io_wmb(); - sq->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(sq->pi); + sq->sq_obj.db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(sq->pi); rte_wmb(); *sq->uar_addr = *(volatile uint64_t *)wqe; /* Assume 64 bit ARCH.*/ rte_wmb(); @@ -418,7 +380,7 @@ cq->errors++; idx = rte_be_to_cpu_16(cqe->wqe_counter) & (1u << sq->log_desc_n); mlx5_aso_dump_err_objs((volatile uint32_t *)cqe, - (volatile uint32_t *)&sq->wqes[idx]); + (volatile uint32_t *)&sq->sq_obj.aso_wqes[idx]); } /** @@ -613,7 +575,7 @@ { int retries = 1024; - if (!sh->aso_age_mng->aso_sq.sq) + if (!sh->aso_age_mng->aso_sq.sq_obj.sq) return -EINVAL; rte_errno = 0; while (--retries) { From patchwork Thu Dec 17 11:44:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85321 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 896ABA09F6; Thu, 17 Dec 2020 12:49:21 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id EDD05CABB; Thu, 17 Dec 2020 12:45:17 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id E56C4CA12 for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:51 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2e004524; Thu, 17 Dec 2020 13:44:51 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:33 +0000 Message-Id: <1608205475-20067-16-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 15/17] common/mlx5: share DevX RQ creation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The RQ object in DevX is used currently only in net driver, but it share for future. Add a structure that contains all the resources, and provide creation and release functions for it. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/common/mlx5/mlx5_common_devx.c | 116 +++++++++++++++++++++++++++++++++ drivers/common/mlx5/mlx5_common_devx.h | 11 ++++ 2 files changed, 127 insertions(+) diff --git a/drivers/common/mlx5/mlx5_common_devx.c b/drivers/common/mlx5/mlx5_common_devx.c index 46404d8..0ac67bd 100644 --- a/drivers/common/mlx5/mlx5_common_devx.c +++ b/drivers/common/mlx5/mlx5_common_devx.c @@ -276,4 +276,120 @@ return -rte_errno; } +/** + * Destroy DevX Receive Queue. + * + * @param[in] rq + * DevX RQ to destroy. + */ +void +mlx5_devx_rq_destroy(struct mlx5_devx_rq *rq) +{ + if (rq->rq) + claim_zero(mlx5_devx_cmd_destroy(rq->rq)); + if (rq->umem_obj) + claim_zero(mlx5_glue->devx_umem_dereg(rq->umem_obj)); + if (rq->umem_buf) + mlx5_free((void *)(uintptr_t)rq->umem_buf); +} + +/** + * Create Receive Queue using DevX API. + * + * Get a pointer to partially initialized attributes structure, and updates the + * following fields: + * wq_umem_valid + * wq_umem_id + * wq_umem_offset + * dbr_umem_valid + * dbr_umem_id + * dbr_addr + * log_wq_pg_sz + * All other fields are updated by caller. + * + * @param[in] ctx + * Context returned from mlx5 open_device() glue function. + * @param[in/out] rq_obj + * Pointer to RQ to create. + * @param[in] wqe_size + * Size of WQE structure. + * @param[in] log_wqbb_n + * Log of number of WQBBs in queue. + * @param[in] attr + * Pointer to RQ attributes structure. + * @param[in] socket + * Socket to use for allocation. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_devx_rq_create(void *ctx, struct mlx5_devx_rq *rq_obj, uint32_t wqe_size, + uint16_t log_wqbb_n, + struct mlx5_devx_create_rq_attr *attr, int socket) +{ + struct mlx5_devx_obj *rq = NULL; + struct mlx5dv_devx_umem *umem_obj = NULL; + void *umem_buf = NULL; + size_t page_size = rte_mem_page_size(); + size_t alignment = MLX5_WQE_BUF_ALIGNMENT; + uint32_t umem_size, umem_dbrec; + uint16_t rq_size = 1 << log_wqbb_n; + int ret; + + if (page_size == (size_t)-1 || alignment == (size_t)-1) { + DRV_LOG(ERR, "Failed to get page_size."); + rte_errno = ENOMEM; + return -rte_errno; + } + /* Allocate memory buffer for WQEs and doorbell record. */ + umem_size = wqe_size * rq_size; + umem_dbrec = RTE_ALIGN(umem_size, MLX5_DBR_SIZE); + umem_size += MLX5_DBR_SIZE; + umem_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, umem_size, + alignment, socket); + if (!umem_buf) { + DRV_LOG(ERR, "Failed to allocate memory for RQ."); + rte_errno = ENOMEM; + return -rte_errno; + } + /* Register allocated buffer in user space with DevX. */ + umem_obj = mlx5_glue->devx_umem_reg(ctx, (void *)(uintptr_t)umem_buf, + umem_size, 0); + if (!umem_obj) { + DRV_LOG(ERR, "Failed to register umem for RQ."); + rte_errno = errno; + goto error; + } + /* Fill attributes for RQ object creation. */ + attr->wq_attr.wq_umem_valid = 1; + attr->wq_attr.wq_umem_id = mlx5_os_get_umem_id(rq_obj->umem_obj); + attr->wq_attr.wq_umem_offset = 0; + attr->wq_attr.dbr_umem_valid = 1; + attr->wq_attr.dbr_umem_id = attr->wq_attr.wq_umem_id; + attr->wq_attr.dbr_addr = umem_dbrec; + attr->wq_attr.log_wq_pg_sz = rte_log2_u32(page_size); + /* Create receive queue object with DevX. */ + rq = mlx5_devx_cmd_create_rq(ctx, attr, socket); + if (!rq) { + DRV_LOG(ERR, "Can't create DevX RQ object."); + rte_errno = ENOMEM; + goto error; + } + rq_obj->umem_buf = umem_buf; + rq_obj->umem_obj = umem_obj; + rq_obj->rq = rq; + rq_obj->db_rec = RTE_PTR_ADD(rq_obj->umem_buf, umem_dbrec); + return 0; +error: + ret = rte_errno; + if (rq) + claim_zero(mlx5_devx_cmd_destroy(rq)); + if (umem_obj) + claim_zero(mlx5_glue->devx_umem_dereg(umem_obj)); + if (umem_buf) + mlx5_free((void *)(uintptr_t)umem_buf); + rte_errno = ret; + return -rte_errno; +} diff --git a/drivers/common/mlx5/mlx5_common_devx.h b/drivers/common/mlx5/mlx5_common_devx.h index 8377d34..1dafbf5 100644 --- a/drivers/common/mlx5/mlx5_common_devx.h +++ b/drivers/common/mlx5/mlx5_common_devx.h @@ -30,6 +30,13 @@ struct mlx5_devx_sq { volatile uint32_t *db_rec; /* The SQ doorbell record. */ }; +/* DevX Receive Queue structure. */ +struct mlx5_devx_rq { + struct mlx5_devx_obj *rq; /* The RQ DevX object. */ + struct mlx5dv_devx_umem *umem_obj; /* The RQ umem object. */ + volatile void *umem_buf; + volatile uint32_t *db_rec; /* The RQ doorbell record. */ +}; /* mlx5_common_devx.c */ @@ -41,5 +48,9 @@ int mlx5_devx_cq_create(void *ctx, struct mlx5_devx_cq *cq_obj, int mlx5_devx_sq_create(void *ctx, struct mlx5_devx_sq *sq_obj, uint16_t log_wqbb_n, struct mlx5_devx_create_sq_attr *attr, int socket); +void mlx5_devx_rq_destroy(struct mlx5_devx_rq *rq); +int mlx5_devx_rq_create(void *ctx, struct mlx5_devx_rq *rq_obj, + uint32_t wqe_size, uint16_t log_wqbb_n, + struct mlx5_devx_create_rq_attr *attr, int socket); #endif /* RTE_PMD_MLX5_COMMON_DEVX_H_ */ From patchwork Thu Dec 17 11:44:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85319 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id D1D51A09F6; Thu, 17 Dec 2020 12:48:42 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C6F58CAA6; Thu, 17 Dec 2020 12:45:15 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id DFD0ECA0C for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:51 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2f004524; Thu, 17 Dec 2020 13:44:51 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:34 +0000 Message-Id: <1608205475-20067-17-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 16/17] net/mlx5: move Rx RQ creation to common X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Using common function for Rx RQ creation. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5.h | 4 +- drivers/net/mlx5/mlx5_devx.c | 178 +++++++++---------------------------------- drivers/net/mlx5/mlx5_rxtx.h | 4 - 3 files changed, 37 insertions(+), 149 deletions(-) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 86ada23..5bf6886 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -772,8 +772,9 @@ struct mlx5_rxq_obj { void *ibv_cq; /* Completion Queue. */ void *ibv_channel; }; + struct mlx5_devx_obj *rq; /* DevX RQ object for hairpin. */ struct { - struct mlx5_devx_obj *rq; /* DevX Rx Queue object. */ + struct mlx5_devx_rq rq_obj; /* DevX RQ object. */ struct mlx5_devx_cq cq_obj; /* DevX CQ object. */ void *devx_channel; }; @@ -944,7 +945,6 @@ struct mlx5_priv { /* Context for Verbs allocator. */ int nl_socket_rdma; /* Netlink socket (NETLINK_RDMA). */ int nl_socket_route; /* Netlink socket (NETLINK_ROUTE). */ - struct mlx5_dbr_page_list dbrpgs; /* Door-bell pages. */ struct mlx5_nl_vlan_vmwa_context *vmwa_context; /* VLAN WA context. */ struct mlx5_hlist *mreg_cp_tbl; /* Hash table of Rx metadata register copy table. */ diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c index 4154c52..9e825ce 100644 --- a/drivers/net/mlx5/mlx5_devx.c +++ b/drivers/net/mlx5/mlx5_devx.c @@ -45,7 +45,7 @@ rq_attr.state = MLX5_RQC_STATE_RDY; rq_attr.vsd = (on ? 0 : 1); rq_attr.modify_bitmask = MLX5_MODIFY_RQ_IN_MODIFY_BITMASK_VSD; - return mlx5_devx_cmd_modify_rq(rxq_obj->rq, &rq_attr); + return mlx5_devx_cmd_modify_rq(rxq_obj->rq_obj.rq, &rq_attr); } /** @@ -85,7 +85,7 @@ default: break; } - return mlx5_devx_cmd_modify_rq(rxq_obj->rq, &rq_attr); + return mlx5_devx_cmd_modify_rq(rxq_obj->rq_obj.rq, &rq_attr); } /** @@ -145,44 +145,18 @@ } /** - * Release the resources allocated for an RQ DevX object. - * - * @param rxq_ctrl - * DevX Rx queue object. - */ -static void -mlx5_rxq_release_devx_rq_resources(struct mlx5_rxq_ctrl *rxq_ctrl) -{ - struct mlx5_devx_dbr_page *dbr_page = rxq_ctrl->rq_dbrec_page; - - if (rxq_ctrl->wq_umem) { - mlx5_glue->devx_umem_dereg(rxq_ctrl->wq_umem); - rxq_ctrl->wq_umem = NULL; - } - if (rxq_ctrl->rxq.wqes) { - mlx5_free((void *)(uintptr_t)rxq_ctrl->rxq.wqes); - rxq_ctrl->rxq.wqes = NULL; - } - if (dbr_page) { - claim_zero(mlx5_release_dbr(&rxq_ctrl->priv->dbrpgs, - mlx5_os_get_umem_id(dbr_page->umem), - rxq_ctrl->rq_dbr_offset)); - rxq_ctrl->rq_dbrec_page = NULL; - } -} - -/** * Destroy the Rx queue DevX object. * * @param rxq_obj * Rxq object to destroy. */ static void -mlx5_rxq_release_devx_resources(struct mlx5_rxq_ctrl *rxq_ctrl) +mlx5_rxq_release_devx_resources(struct mlx5_rxq_obj *rxq_obj) { - mlx5_rxq_release_devx_rq_resources(rxq_ctrl); - mlx5_devx_cq_destroy(&rxq_ctrl->obj->cq_obj); - memset(&rxq_ctrl->obj->cq_obj, 0, sizeof(rxq_ctrl->obj->cq_obj)); + mlx5_devx_rq_destroy(&rxq_obj->rq_obj); + memset(&rxq_obj->rq_obj, 0, sizeof(rxq_obj->rq_obj)); + mlx5_devx_cq_destroy(&rxq_obj->cq_obj); + memset(&rxq_obj->cq_obj, 0, sizeof(rxq_obj->cq_obj)); } /** @@ -195,17 +169,17 @@ mlx5_rxq_devx_obj_release(struct mlx5_rxq_obj *rxq_obj) { MLX5_ASSERT(rxq_obj); - MLX5_ASSERT(rxq_obj->rq); if (rxq_obj->rxq_ctrl->type == MLX5_RXQ_TYPE_HAIRPIN) { + MLX5_ASSERT(rxq_obj->rq); mlx5_devx_modify_rq(rxq_obj, MLX5_RXQ_MOD_RDY2RST); claim_zero(mlx5_devx_cmd_destroy(rxq_obj->rq)); } else { - MLX5_ASSERT(rxq_obj->cq_obj); - claim_zero(mlx5_devx_cmd_destroy(rxq_obj->rq)); + MLX5_ASSERT(rxq_obj->cq_obj.cq); + MLX5_ASSERT(rxq_obj->rq_obj.rq); + mlx5_rxq_release_devx_resources(rxq_obj); if (rxq_obj->devx_channel) mlx5_glue->devx_destroy_event_channel (rxq_obj->devx_channel); - mlx5_rxq_release_devx_resources(rxq_obj->rxq_ctrl); } } @@ -247,52 +221,6 @@ } /** - * Fill common fields of create RQ attributes structure. - * - * @param rxq_data - * Pointer to Rx queue data. - * @param cqn - * CQ number to use with this RQ. - * @param rq_attr - * RQ attributes structure to fill.. - */ -static void -mlx5_devx_create_rq_attr_fill(struct mlx5_rxq_data *rxq_data, uint32_t cqn, - struct mlx5_devx_create_rq_attr *rq_attr) -{ - rq_attr->state = MLX5_RQC_STATE_RST; - rq_attr->vsd = (rxq_data->vlan_strip) ? 0 : 1; - rq_attr->cqn = cqn; - rq_attr->scatter_fcs = (rxq_data->crc_present) ? 1 : 0; -} - -/** - * Fill common fields of DevX WQ attributes structure. - * - * @param priv - * Pointer to device private data. - * @param rxq_ctrl - * Pointer to Rx queue control structure. - * @param wq_attr - * WQ attributes structure to fill.. - */ -static void -mlx5_devx_wq_attr_fill(struct mlx5_priv *priv, struct mlx5_rxq_ctrl *rxq_ctrl, - struct mlx5_devx_wq_attr *wq_attr) -{ - wq_attr->end_padding_mode = priv->config.hw_padding ? - MLX5_WQ_END_PAD_MODE_ALIGN : - MLX5_WQ_END_PAD_MODE_NONE; - wq_attr->pd = priv->sh->pdn; - wq_attr->dbr_addr = rxq_ctrl->rq_dbr_offset; - wq_attr->dbr_umem_id = - mlx5_os_get_umem_id(rxq_ctrl->rq_dbrec_page->umem); - wq_attr->dbr_umem_valid = 1; - wq_attr->wq_umem_id = mlx5_os_get_umem_id(rxq_ctrl->wq_umem); - wq_attr->wq_umem_valid = 1; -} - -/** * Create a RQ object using DevX. * * @param dev @@ -301,9 +229,9 @@ * Queue index in DPDK Rx queue array. * * @return - * The DevX RQ object initialized, NULL otherwise and rte_errno is set. + * 0 on success, a negative errno value otherwise and rte_errno is set. */ -static struct mlx5_devx_obj * +static int mlx5_rxq_create_devx_rq_resources(struct rte_eth_dev *dev, uint16_t idx) { struct mlx5_priv *priv = dev->data->dev_private; @@ -311,26 +239,15 @@ struct mlx5_rxq_ctrl *rxq_ctrl = container_of(rxq_data, struct mlx5_rxq_ctrl, rxq); struct mlx5_devx_create_rq_attr rq_attr = { 0 }; - uint32_t wqe_n = 1 << (rxq_data->elts_n - rxq_data->sges_n); - uint32_t cqn = rxq_ctrl->obj->cq_obj.cq->id; - struct mlx5_devx_dbr_page *dbr_page; - int64_t dbr_offset; - uint32_t wq_size = 0; - uint32_t wqe_size = 0; - uint32_t log_wqe_size = 0; - void *buf = NULL; - struct mlx5_devx_obj *rq; - size_t alignment = MLX5_WQE_BUF_ALIGNMENT; + uint16_t log_desc_n = rxq_data->elts_n - rxq_data->sges_n; + uint32_t wqe_size, log_wqe_size; - if (alignment == (size_t)-1) { - DRV_LOG(ERR, "Failed to get mem page size"); - rte_errno = ENOMEM; - return NULL; - } /* Fill RQ attributes. */ rq_attr.mem_rq_type = MLX5_RQC_MEM_RQ_TYPE_MEMORY_RQ_INLINE; rq_attr.flush_in_error_en = 1; - mlx5_devx_create_rq_attr_fill(rxq_data, cqn, &rq_attr); + rq_attr.vsd = (rxq_data->vlan_strip) ? 0 : 1; + rq_attr.cqn = rxq_ctrl->obj->cq_obj.cq->id; + rq_attr.scatter_fcs = (rxq_data->crc_present) ? 1 : 0; /* Fill WQ attributes for this RQ. */ if (mlx5_rxq_mprq_enabled(rxq_data)) { rq_attr.wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC_STRIDING_RQ; @@ -351,40 +268,17 @@ wqe_size = sizeof(struct mlx5_wqe_data_seg); } log_wqe_size = log2above(wqe_size) + rxq_data->sges_n; - rq_attr.wq_attr.log_wq_stride = log_wqe_size; - rq_attr.wq_attr.log_wq_sz = rxq_data->elts_n - rxq_data->sges_n; - rq_attr.wq_attr.log_wq_pg_sz = log2above(alignment); - /* Calculate and allocate WQ memory space. */ wqe_size = 1 << log_wqe_size; /* round up power of two.*/ - wq_size = wqe_n * wqe_size; - buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, wq_size, - alignment, rxq_ctrl->socket); - if (!buf) - return NULL; - rxq_data->wqes = buf; - rxq_ctrl->wq_umem = mlx5_glue->devx_umem_reg(priv->sh->ctx, - buf, wq_size, 0); - if (!rxq_ctrl->wq_umem) - goto error; - /* Allocate RQ door-bell. */ - dbr_offset = mlx5_get_dbr(priv->sh->ctx, &priv->dbrpgs, &dbr_page); - if (dbr_offset < 0) { - DRV_LOG(ERR, "Failed to allocate RQ door-bell."); - goto error; - } - rxq_ctrl->rq_dbr_offset = dbr_offset; - rxq_ctrl->rq_dbrec_page = dbr_page; - rxq_data->rq_db = (uint32_t *)((uintptr_t)dbr_page->dbrs + - (uintptr_t)rxq_ctrl->rq_dbr_offset); + rq_attr.wq_attr.log_wq_stride = log_wqe_size; + rq_attr.wq_attr.log_wq_sz = log_desc_n; + rq_attr.wq_attr.end_padding_mode = priv->config.hw_padding ? + MLX5_WQ_END_PAD_MODE_ALIGN : + MLX5_WQ_END_PAD_MODE_NONE; + rq_attr.wq_attr.pd = priv->sh->pdn; /* Create RQ using DevX API. */ - mlx5_devx_wq_attr_fill(priv, rxq_ctrl, &rq_attr.wq_attr); - rq = mlx5_devx_cmd_create_rq(priv->sh->ctx, &rq_attr, rxq_ctrl->socket); - if (!rq) - goto error; - return rq; -error: - mlx5_rxq_release_devx_rq_resources(rxq_ctrl); - return NULL; + return mlx5_devx_rq_create(priv->sh->ctx, &rxq_ctrl->obj->rq_obj, + wqe_size, log_desc_n, &rq_attr, + rxq_ctrl->socket); } /** @@ -607,8 +501,8 @@ goto error; } /* Create RQ using DevX API. */ - tmpl->rq = mlx5_rxq_create_devx_rq_resources(dev, idx); - if (!tmpl->rq) { + ret = mlx5_rxq_create_devx_rq_resources(dev, idx); + if (ret) { DRV_LOG(ERR, "Port %u Rx queue %u RQ creation failure.", dev->data->port_id, idx); rte_errno = ENOMEM; @@ -618,19 +512,17 @@ ret = mlx5_devx_modify_rq(tmpl, MLX5_RXQ_MOD_RST2RDY); if (ret) goto error; + rxq_data->wqes = (void *)(uintptr_t)tmpl->rq_obj.umem_buf; + rxq_data->rq_db = (uint32_t *)(uintptr_t)tmpl->rq_obj.db_rec; rxq_data->cq_arm_sn = 0; - mlx5_rxq_initialize(rxq_data); rxq_data->cq_ci = 0; + mlx5_rxq_initialize(rxq_data); dev->data->rx_queue_state[idx] = RTE_ETH_QUEUE_STATE_STARTED; - rxq_ctrl->wqn = tmpl->rq->id; + rxq_ctrl->wqn = tmpl->rq_obj.rq->id; return 0; error: ret = rte_errno; /* Save rte_errno before cleanup. */ - if (tmpl->rq) - claim_zero(mlx5_devx_cmd_destroy(tmpl->rq)); - if (tmpl->devx_channel) - mlx5_glue->devx_destroy_event_channel(tmpl->devx_channel); - mlx5_rxq_release_devx_resources(rxq_ctrl); + mlx5_rxq_devx_obj_release(tmpl); rte_errno = ret; /* Restore rte_errno. */ return -rte_errno; } @@ -674,7 +566,7 @@ struct mlx5_rxq_ctrl *rxq_ctrl = container_of(rxq, struct mlx5_rxq_ctrl, rxq); - rqt_attr->rq_list[i] = rxq_ctrl->obj->rq->id; + rqt_attr->rq_list[i] = rxq_ctrl->obj->rq_obj.rq->id; } MLX5_ASSERT(i > 0); for (j = 0; i != rqt_n; ++j, ++i) diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 6a71791..b0041c9 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -193,10 +193,6 @@ struct mlx5_rxq_ctrl { uint32_t flow_tunnels_n[MLX5_FLOW_TUNNEL]; /* Tunnels counters. */ uint32_t wqn; /* WQ number. */ uint16_t dump_file_n; /* Number of dump files. */ - struct mlx5_devx_dbr_page *rq_dbrec_page; - uint64_t rq_dbr_offset; - /* Storing RQ door-bell information, needed when freeing door-bell. */ - void *wq_umem; /* WQ buffer registration info. */ struct rte_eth_hairpin_conf hairpin_conf; /* Hairpin configuration. */ uint32_t hairpin_status; /* Hairpin binding status. */ }; From patchwork Thu Dec 17 11:44:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85314 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2A214A09F6; Thu, 17 Dec 2020 12:47:09 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6592ACA5A; Thu, 17 Dec 2020 12:45:09 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id A13E0CA20 for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:51 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2g004524; Thu, 17 Dec 2020 13:44:51 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:35 +0000 Message-Id: <1608205475-20067-18-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 17/17] common/mlx5: remove doorbell allocation API X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The mlx5_devx_dbr_page structure was used to allocate and release the umem of the doorbells. Since doorbell and buffer have used same umem, this structure is useless. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/common/mlx5/mlx5_common.c | 122 -------------------------------------- drivers/common/mlx5/mlx5_common.h | 23 ------- 2 files changed, 145 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 0445132..c26a2cf 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -8,12 +8,10 @@ #include #include -#include #include "mlx5_common.h" #include "mlx5_common_os.h" #include "mlx5_common_utils.h" -#include "mlx5_malloc.h" #include "mlx5_common_pci.h" int mlx5_common_logtype; @@ -126,126 +124,6 @@ static inline void mlx5_cpu_id(unsigned int level, } /** - * Allocate page of door-bells and register it using DevX API. - * - * @param [in] ctx - * Pointer to the device context. - * - * @return - * Pointer to new page on success, NULL otherwise. - */ -static struct mlx5_devx_dbr_page * -mlx5_alloc_dbr_page(void *ctx) -{ - struct mlx5_devx_dbr_page *page; - - /* Allocate space for door-bell page and management data. */ - page = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, - sizeof(struct mlx5_devx_dbr_page), - RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY); - if (!page) { - DRV_LOG(ERR, "cannot allocate dbr page"); - return NULL; - } - /* Register allocated memory. */ - page->umem = mlx5_glue->devx_umem_reg(ctx, page->dbrs, - MLX5_DBR_PAGE_SIZE, 0); - if (!page->umem) { - DRV_LOG(ERR, "cannot umem reg dbr page"); - mlx5_free(page); - return NULL; - } - return page; -} - -/** - * Find the next available door-bell, allocate new page if needed. - * - * @param [in] ctx - * Pointer to device context. - * @param [in] head - * Pointer to the head of dbr pages list. - * @param [out] dbr_page - * Door-bell page containing the page data. - * - * @return - * Door-bell address offset on success, a negative error value otherwise. - */ -int64_t -mlx5_get_dbr(void *ctx, struct mlx5_dbr_page_list *head, - struct mlx5_devx_dbr_page **dbr_page) -{ - struct mlx5_devx_dbr_page *page = NULL; - uint32_t i, j; - - LIST_FOREACH(page, head, next) - if (page->dbr_count < MLX5_DBR_PER_PAGE) - break; - if (!page) { /* No page with free door-bell exists. */ - page = mlx5_alloc_dbr_page(ctx); - if (!page) /* Failed to allocate new page. */ - return (-1); - LIST_INSERT_HEAD(head, page, next); - } - /* Loop to find bitmap part with clear bit. */ - for (i = 0; - i < MLX5_DBR_BITMAP_SIZE && page->dbr_bitmap[i] == UINT64_MAX; - i++) - ; /* Empty. */ - /* Find the first clear bit. */ - MLX5_ASSERT(i < MLX5_DBR_BITMAP_SIZE); - j = rte_bsf64(~page->dbr_bitmap[i]); - page->dbr_bitmap[i] |= (UINT64_C(1) << j); - page->dbr_count++; - *dbr_page = page; - return (i * CHAR_BIT * sizeof(uint64_t) + j) * MLX5_DBR_SIZE; -} - -/** - * Release a door-bell record. - * - * @param [in] head - * Pointer to the head of dbr pages list. - * @param [in] umem_id - * UMEM ID of page containing the door-bell record to release. - * @param [in] offset - * Offset of door-bell record in page. - * - * @return - * 0 on success, a negative error value otherwise. - */ -int32_t -mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id, - uint64_t offset) -{ - struct mlx5_devx_dbr_page *page = NULL; - int ret = 0; - - LIST_FOREACH(page, head, next) - /* Find the page this address belongs to. */ - if (mlx5_os_get_umem_id(page->umem) == umem_id) - break; - if (!page) - return -EINVAL; - page->dbr_count--; - if (!page->dbr_count) { - /* Page not used, free it and remove from list. */ - LIST_REMOVE(page, next); - if (page->umem) - ret = -mlx5_glue->devx_umem_dereg(page->umem); - mlx5_free(page); - } else { - /* Mark in bitmap that this door-bell is not in use. */ - offset /= MLX5_DBR_SIZE; - int i = offset / 64; - int j = offset % 64; - - page->dbr_bitmap[i] &= ~(UINT64_C(1) << j); - } - return ret; -} - -/** * Allocate the User Access Region with DevX on specified device. * * @param [in] ctx diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index a484b74..e35188d 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -220,21 +220,6 @@ enum mlx5_class { }; #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE -#define MLX5_DBR_PER_PAGE 64 -/* Must be >= CHAR_BIT * sizeof(uint64_t) */ -#define MLX5_DBR_PAGE_SIZE (MLX5_DBR_PER_PAGE * MLX5_DBR_SIZE) -/* Page size must be >= 512. */ -#define MLX5_DBR_BITMAP_SIZE (MLX5_DBR_PER_PAGE / (CHAR_BIT * sizeof(uint64_t))) - -struct mlx5_devx_dbr_page { - /* Door-bell records, must be first member in structure. */ - uint8_t dbrs[MLX5_DBR_PAGE_SIZE]; - LIST_ENTRY(mlx5_devx_dbr_page) next; /* Pointer to the next element. */ - void *umem; - uint32_t dbr_count; /* Number of door-bell records in use. */ - /* 1 bit marks matching door-bell is in use. */ - uint64_t dbr_bitmap[MLX5_DBR_BITMAP_SIZE]; -}; /* devX creation object */ struct mlx5_devx_obj { @@ -249,19 +234,11 @@ struct mlx5_klm { uint64_t address; }; -LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page); - __rte_internal void mlx5_translate_port_name(const char *port_name_in, struct mlx5_switch_info *port_info_out); void mlx5_glue_constructor(void); __rte_internal -int64_t mlx5_get_dbr(void *ctx, struct mlx5_dbr_page_list *head, - struct mlx5_devx_dbr_page **dbr_page); -__rte_internal -int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id, - uint64_t offset); -__rte_internal void *mlx5_devx_alloc_uar(void *ctx, int mapping); extern uint8_t haswell_broadwell_cpu;