From patchwork Thu Dec 17 11:44:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 85315 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 545FAA09F6; Thu, 17 Dec 2020 12:47:28 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8DEDECA64; Thu, 17 Dec 2020 12:45:10 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id AB955CA26 for ; Thu, 17 Dec 2020 12:44:56 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from michaelba@nvidia.com) with SMTP; 17 Dec 2020 13:44:50 +0200 Received: from nvidia.com (pegasus07.mtr.labs.mlnx [10.210.16.112]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0BHBio2Z004524; Thu, 17 Dec 2020 13:44:50 +0200 From: Michael Baum To: dev@dpdk.org Cc: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Thu, 17 Dec 2020 11:44:28 +0000 Message-Id: <1608205475-20067-11-git-send-email-michaelba@nvidia.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> References: <1608205475-20067-1-git-send-email-michaelba@nvidia.com> Subject: [dpdk-dev] [PATCH 10/17] common/mlx5: share DevX SQ creation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The SQ object in DevX is created in several places and in several different drivers. In all places almost all the details are the same, and in particular the allocations of the required resources. Add a structure that contains all the resources, and provide creation and release functions for it. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- drivers/common/mlx5/mlx5_common_devx.c | 122 +++++++++++++++++++++++++++++++++ drivers/common/mlx5/mlx5_common_devx.h | 20 +++++- 2 files changed, 140 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common_devx.c b/drivers/common/mlx5/mlx5_common_devx.c index 324c6ea..46404d8 100644 --- a/drivers/common/mlx5/mlx5_common_devx.c +++ b/drivers/common/mlx5/mlx5_common_devx.c @@ -155,3 +155,125 @@ rte_errno = ret; return -rte_errno; } + +/** + * Destroy DevX Send Queue. + * + * @param[in] sq + * DevX SQ to destroy. + */ +void +mlx5_devx_sq_destroy(struct mlx5_devx_sq *sq) +{ + if (sq->sq) + claim_zero(mlx5_devx_cmd_destroy(sq->sq)); + if (sq->umem_obj) + claim_zero(mlx5_glue->devx_umem_dereg(sq->umem_obj)); + if (sq->umem_buf) + mlx5_free((void *)(uintptr_t)sq->umem_buf); +} + +/** + * Create Send Queue using DevX API. + * + * Get a pointer to partially initialized attributes structure, and updates the + * following fields: + * wq_type + * wq_umem_valid + * wq_umem_id + * wq_umem_offset + * dbr_umem_valid + * dbr_umem_id + * dbr_addr + * log_wq_stride + * log_wq_sz + * log_wq_pg_sz + * All other fields are updated by caller. + * + * @param[in] ctx + * Context returned from mlx5 open_device() glue function. + * @param[in/out] sq_obj + * Pointer to SQ to create. + * @param[in] log_wqbb_n + * Log of number of WQBBs in queue. + * @param[in] attr + * Pointer to SQ attributes structure. + * @param[in] socket + * Socket to use for allocation. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +int +mlx5_devx_sq_create(void *ctx, struct mlx5_devx_sq *sq_obj, uint16_t log_wqbb_n, + struct mlx5_devx_create_sq_attr *attr, int socket) +{ + struct mlx5_devx_obj *sq = NULL; + struct mlx5dv_devx_umem *umem_obj = NULL; + void *umem_buf = NULL; + size_t page_size = rte_mem_page_size(); + size_t alignment = MLX5_WQE_BUF_ALIGNMENT; + uint32_t umem_size, umem_dbrec; + uint16_t sq_size = 1 << log_wqbb_n; + int ret; + + if (page_size == (size_t)-1 || alignment == (size_t)-1) { + DRV_LOG(ERR, "Failed to get page_size."); + rte_errno = ENOMEM; + return -rte_errno; + } + /* Allocate memory buffer for WQEs and doorbell record. */ + umem_size = MLX5_WQE_SIZE * sq_size; + umem_dbrec = RTE_ALIGN(umem_size, MLX5_DBR_SIZE); + umem_size += MLX5_DBR_SIZE; + umem_buf = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, umem_size, + alignment, socket); + if (!umem_buf) { + DRV_LOG(ERR, "Failed to allocate memory for SQ."); + rte_errno = ENOMEM; + return -rte_errno; + } + /* Register allocated buffer in user space with DevX. */ + umem_obj = mlx5_glue->devx_umem_reg(ctx, (void *)(uintptr_t)umem_buf, + umem_size, IBV_ACCESS_LOCAL_WRITE); + if (!umem_obj) { + DRV_LOG(ERR, "Failed to register umem for SQ."); + rte_errno = errno; + goto error; + } + /* Fill attributes for SQ object creation. */ + attr->wq_attr.wq_type = MLX5_WQ_TYPE_CYCLIC; + attr->wq_attr.wq_umem_valid = 1; + attr->wq_attr.wq_umem_id = mlx5_os_get_umem_id(sq_obj->umem_obj); + attr->wq_attr.wq_umem_offset = 0; + attr->wq_attr.dbr_umem_valid = 1; + attr->wq_attr.dbr_umem_id = attr->wq_attr.wq_umem_id; + attr->wq_attr.dbr_addr = umem_dbrec; + attr->wq_attr.log_wq_stride = rte_log2_u32(MLX5_WQE_SIZE); + attr->wq_attr.log_wq_sz = log_wqbb_n; + attr->wq_attr.log_wq_pg_sz = rte_log2_u32(page_size); + /* Create send queue object with DevX. */ + sq = mlx5_devx_cmd_create_sq(ctx, attr); + if (!sq) { + DRV_LOG(ERR, "Can't create DevX SQ object."); + rte_errno = ENOMEM; + goto error; + } + sq_obj->umem_buf = umem_buf; + sq_obj->umem_obj = umem_obj; + sq_obj->sq = sq; + sq_obj->db_rec = RTE_PTR_ADD(sq_obj->umem_buf, umem_dbrec); + return 0; +error: + ret = rte_errno; + if (sq) + claim_zero(mlx5_devx_cmd_destroy(sq)); + if (umem_obj) + claim_zero(mlx5_glue->devx_umem_dereg(umem_obj)); + if (umem_buf) + mlx5_free((void *)(uintptr_t)umem_buf); + rte_errno = ret; + return -rte_errno; +} + + diff --git a/drivers/common/mlx5/mlx5_common_devx.h b/drivers/common/mlx5/mlx5_common_devx.h index 31cb804..88d520b 100644 --- a/drivers/common/mlx5/mlx5_common_devx.h +++ b/drivers/common/mlx5/mlx5_common_devx.h @@ -18,11 +18,27 @@ struct mlx5_devx_cq { volatile uint32_t *db_rec; /* The CQ doorbell record. */ }; +/* DevX Send Queue structure. */ +struct mlx5_devx_sq { + struct mlx5_devx_obj *sq; /* The SQ DevX object. */ + struct mlx5dv_devx_umem *umem_obj; /* The SQ umem object. */ + union { + volatile void *umem_buf; + volatile struct mlx5_wqe *wqes; /* The SQ ring buffer. */ + }; + volatile uint32_t *db_rec; /* The SQ doorbell record. */ +}; + + /* mlx5_common_devx.c */ void mlx5_devx_cq_destroy(struct mlx5_devx_cq *cq); int mlx5_devx_cq_create(void *ctx, struct mlx5_devx_cq *cq_obj, - uint16_t log_desc_n, struct mlx5_devx_cq_attr *attr, - int socket); + uint16_t log_desc_n, + struct mlx5_devx_cq_attr *attr, int socket); +void mlx5_devx_sq_destroy(struct mlx5_devx_sq *sq); +int mlx5_devx_sq_create(void *ctx, struct mlx5_devx_sq *sq_obj, + uint16_t log_wqbb_n, + struct mlx5_devx_create_sq_attr *attr, int socket); #endif /* RTE_PMD_MLX5_COMMON_DEVX_H_ */