From patchwork Thu Jul 15 16:41:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shiri Kuzin X-Patchwork-Id: 95918 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8DFFCA0C41; Thu, 15 Jul 2021 18:43:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 469A24125B; Thu, 15 Jul 2021 18:42:34 +0200 (CEST) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2085.outbound.protection.outlook.com [40.107.93.85]) by mails.dpdk.org (Postfix) with ESMTP id 9E67B41275 for ; Thu, 15 Jul 2021 18:42:23 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KGIkpWVktvLEO21Jgryg/5ucHC3k3hSANYVDc5jQxihPSb92Bj5pkcIP3YIECch+d3fiY8H5l8o10Jz8nqeHv/UQ/w0xETjLZUwz9DXDoq2Xr8xcwhmQyZBvbL1qEaKDhBD0LLAbNpkwYUno52+Hq251mLLPu/ng1vrquDNpB37zFtoNdDLhvoacH+ehnWa3QECUlfv5nb/UW6TfZaM3Es2ykHThAniUg8yG2JSb4PagAL1HwQ5zpV3XeHel7VP7N7/uX0ipoU7ZMr6ukSGkX+QFZx9QFMRdlNpJCc+ccYTC1vrADIT1yoP4yjMy1ykNcivaAPRGwhmP70H+H+AWtg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=S6dGTk2rSEm4EVTpfADNvh0e8eNklBib34M21ZVJkCk=; b=WcoF96B5vpRQEZrxFqwacn3McNh6Hap4W1OYUc3OQgVpTJ+XCOxK4Br+fMA8mbldrIaP9TaJ4aLucI5vZFs9JlMXwj90P4KM7n60ITOQOuaHcOsCwo3ZeXeEvp8NONNUXrS9ecwagrmP+cNgZ/VLWO6ICrek/7wF+64IcIkQl4QSX+wxHVU2RQXeI2AGNd+5GoNlMBOxwwsdgiBih4SsFhLyXPmt3tJ2ByeNaghQP/i6lxCNt7t0a2zOOsFRhsqvBAanDKOzEhz9Dtu5739Tbb0w0b3N6G+KI5sKng6caL6yFHBsJ8ZEi19c32tSAVlwHOfAm/4xrOEgq0rU3AynzA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=S6dGTk2rSEm4EVTpfADNvh0e8eNklBib34M21ZVJkCk=; b=AL6ER+hR68jBxQ2lwhQ/1kdzGRKMjRHKYjjsKk3LaZ/CazCmrKbJDeRncL8vXMOmXM2V9ZGXnTFnmcObAIF2gmOHNcQg8jkpkswUHmVSD/PRmYUPKdIu0ytJj3G4hb9pnf8OhdDjx3d4yNalEEJNuxR9gHrUoS+V3Cn1/0yRDAwqXRfNuVpP1dMW1bQbglxYpmxlRZvDpJbWnpcdIEpCr8zMv9q9NsMdzVyYyFxnClbYLEFGaDjpjshMuYzcwJYPuCczUxc7Ws1b5Dky0y4fiR/BR4sex9wVbCsiBE2SajhdSFynm903TxcyH080p47Y8YLnTdMfOPXJkQoW35kMkw== Received: from MW4P221CA0027.NAMP221.PROD.OUTLOOK.COM (2603:10b6:303:8b::32) by BYAPR12MB2981.namprd12.prod.outlook.com (2603:10b6:a03:de::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21; Thu, 15 Jul 2021 16:42:22 +0000 Received: from CO1NAM11FT015.eop-nam11.prod.protection.outlook.com (2603:10b6:303:8b:cafe::61) by MW4P221CA0027.outlook.office365.com (2603:10b6:303:8b::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.23 via Frontend Transport; Thu, 15 Jul 2021 16:42:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by CO1NAM11FT015.mail.protection.outlook.com (10.13.175.130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4331.21 via Frontend Transport; Thu, 15 Jul 2021 16:42:21 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 15 Jul 2021 16:42:19 +0000 From: Shiri Kuzin To: CC: , , , , Michael Baum Date: Thu, 15 Jul 2021 19:41:22 +0300 Message-ID: <20210715164126.54073-13-shirik@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210715164126.54073-1-shirik@nvidia.com> References: <20210715150817.51485-1-shirik@nvidia.com> <20210715164126.54073-1-shirik@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0107e686-a4e0-4bb8-fd0f-08d947af8492 X-MS-TrafficTypeDiagnostic: BYAPR12MB2981: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:316; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: lDqWL3Y5GirvNK7yDqrNjXqSffsJ81wwlo5yjJNgGejg5QY7MZu58jeOJy9USWFaP2HFgUoVDC+00EX1v1RecoZ9IjEUuGOcL3nIj1AfuSnOK3LRq2s8+wqbECbUDCiyGdiOdq2vjkfCk5HnlwAnx5kYkO6U7KqNkKyLSR7EkOKQDBtDxhCAz5aSAzK3ajlr23E19oDEOwGsWVcZElWSnEbiGRAjBYo3g0AvPoCP53Ozo7VWAjF3Dunsd8NJTedhgmUF/QeXwMm5lIEylKVPpcqEGge22HEz4v7kRpqFk6nRrQ8p6COT3lUxhFtJsgaeNIJRNct2Cfr01sXw30hO7DyqO32b+V6cns26wLfpYPO4tKjRphvUH5qANqySLBwl075V0l47MCoAWfh1ft2ajJw5R3puOrCaXF1d2muPo9Be4kAgf44gb+XxIhC1IXEXpGlFSCj28fYcP7yer3Ex4Tmdw6ywdPWiIMN8XcOs+tox54dUU+RX/6vU3MukW/HaU8KyZSAMMmsrdgIJFhIskJ5u9txQ8RtcNueJZ80rtDH7DjvmACwk4bBGoEKnxEFNqv8++yh+Z+EvRtEBLV5IMItV3t6wcfAxXTHyILEpuPQYfCS+spv9Kr109HSZvFA6Jag8uq8JYtkFR+u9g3CM1nrQp9BjQf0wAxt0pi0Gec/n+aW9N4Xoyn1rA+eZiDLw2l9A+IZciBgUN/dwSggCcTCyieHon/Fk3+0XduGkq7Y= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(376002)(136003)(39860400002)(346002)(396003)(36840700001)(46966006)(5660300002)(82740400003)(6286002)(356005)(6666004)(70206006)(47076005)(83380400001)(316002)(34020700004)(8936002)(54906003)(7636003)(1076003)(2906002)(8676002)(70586007)(30864003)(36906005)(186003)(82310400003)(4326008)(2616005)(36860700001)(426003)(55016002)(336012)(478600001)(16526019)(7696005)(36756003)(86362001)(6916009)(26005)(107886003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2021 16:42:21.7860 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0107e686-a4e0-4bb8-fd0f-08d947af8492 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT015.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR12MB2981 Subject: [dpdk-dev] [PATCH v8 12/16] crypto/mlx5: add enqueue and dequeue operations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Suanming Mou The crypto operations are done with the WQE set which contains one UMR WQE and one rdma write WQE. Most segments of the WQE set are initialized properly during queue setup, only limited segments are initialized according to the crypto detail in the datapath process. This commit adds the enquue and dequeue operations and updates the WQE set segments accordingly. Signed-off-by: Suanming Mou Signed-off-by: Matan Azrad Signed-off-by: Michael Baum --- doc/guides/cryptodevs/features/mlx5.ini | 5 + drivers/crypto/mlx5/mlx5_crypto.c | 286 +++++++++++++++++++++++- drivers/crypto/mlx5/mlx5_crypto.h | 3 + 3 files changed, 290 insertions(+), 4 deletions(-) diff --git a/doc/guides/cryptodevs/features/mlx5.ini b/doc/guides/cryptodevs/features/mlx5.ini index bd757b5211..a89526add0 100644 --- a/doc/guides/cryptodevs/features/mlx5.ini +++ b/doc/guides/cryptodevs/features/mlx5.ini @@ -6,6 +6,11 @@ [Features] Symmetric crypto = Y HW Accelerated = Y +In Place SGL = Y +OOP SGL In SGL Out = Y +OOP SGL In LB Out = Y +OOP LB In SGL Out = Y +OOP LB In LB Out = Y Cipher multiple data units = Y Cipher wrapped key = Y diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c index e5f8d96ff7..b467739a3c 100644 --- a/drivers/crypto/mlx5/mlx5_crypto.c +++ b/drivers/crypto/mlx5/mlx5_crypto.c @@ -25,6 +25,10 @@ #define MLX5_CRYPTO_FEATURE_FLAGS \ (RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | \ + RTE_CRYPTODEV_FF_IN_PLACE_SGL | RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | \ + RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | \ + RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | \ + RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | \ RTE_CRYPTODEV_FF_CIPHER_WRAPPED_KEY | \ RTE_CRYPTODEV_FF_CIPHER_MULTIPLE_DATA_UNITS) @@ -297,6 +301,279 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp) return 0; } +static __rte_noinline uint32_t +mlx5_crypto_get_block_size(struct rte_crypto_op *op) +{ + uint32_t bl = op->sym->cipher.data.length; + + switch (bl) { + case (1 << 20): + return RTE_BE32(MLX5_BLOCK_SIZE_1MB << MLX5_BLOCK_SIZE_OFFSET); + case (1 << 12): + return RTE_BE32(MLX5_BLOCK_SIZE_4096B << + MLX5_BLOCK_SIZE_OFFSET); + case (1 << 9): + return RTE_BE32(MLX5_BLOCK_SIZE_512B << MLX5_BLOCK_SIZE_OFFSET); + default: + DRV_LOG(ERR, "Unknown block size: %u.", bl); + return UINT32_MAX; + } +} + +/** + * Query LKey from a packet buffer for QP. If not found, add the mempool. + * + * @param priv + * Pointer to the priv object. + * @param addr + * Search key. + * @param mr_ctrl + * Pointer to per-queue MR control structure. + * @param ol_flags + * Mbuf offload features. + * + * @return + * Searched LKey on success, UINT32_MAX on no match. + */ +static __rte_always_inline uint32_t +mlx5_crypto_addr2mr(struct mlx5_crypto_priv *priv, uintptr_t addr, + struct mlx5_mr_ctrl *mr_ctrl, uint64_t ol_flags) +{ + uint32_t lkey; + + /* Check generation bit to see if there's any change on existing MRs. */ + if (unlikely(*mr_ctrl->dev_gen_ptr != mr_ctrl->cur_gen)) + mlx5_mr_flush_local_cache(mr_ctrl); + /* Linear search on MR cache array. */ + lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru, + MLX5_MR_CACHE_N, addr); + if (likely(lkey != UINT32_MAX)) + return lkey; + /* Take slower bottom-half on miss. */ + return mlx5_mr_addr2mr_bh(priv->pd, 0, &priv->mr_scache, mr_ctrl, addr, + !!(ol_flags & EXT_ATTACHED_MBUF)); +} + +static __rte_always_inline uint32_t +mlx5_crypto_klm_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp, + struct rte_mbuf *mbuf, struct mlx5_wqe_dseg *klm, + uint32_t offset, uint32_t *remain) +{ + uint32_t data_len = (rte_pktmbuf_data_len(mbuf) - offset); + uintptr_t addr = rte_pktmbuf_mtod_offset(mbuf, uintptr_t, offset); + + if (data_len > *remain) + data_len = *remain; + *remain -= data_len; + klm->bcount = rte_cpu_to_be_32(data_len); + klm->pbuf = rte_cpu_to_be_64(addr); + klm->lkey = mlx5_crypto_addr2mr(priv, addr, &qp->mr_ctrl, + mbuf->ol_flags); + return klm->lkey; + +} + +static __rte_always_inline uint32_t +mlx5_crypto_klms_set(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp, + struct rte_crypto_op *op, struct rte_mbuf *mbuf, + struct mlx5_wqe_dseg *klm) +{ + uint32_t remain_len = op->sym->cipher.data.length; + uint32_t nb_segs = mbuf->nb_segs; + uint32_t klm_n = 1u; + + /* First mbuf needs to take the cipher offset. */ + if (unlikely(mlx5_crypto_klm_set(priv, qp, mbuf, klm, + op->sym->cipher.data.offset, &remain_len) == UINT32_MAX)) { + op->status = RTE_CRYPTO_OP_STATUS_ERROR; + return 0; + } + while (remain_len) { + nb_segs--; + mbuf = mbuf->next; + if (unlikely(mbuf == NULL || nb_segs == 0)) { + op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + return 0; + } + if (unlikely(mlx5_crypto_klm_set(priv, qp, mbuf, ++klm, 0, + &remain_len) == UINT32_MAX)) { + op->status = RTE_CRYPTO_OP_STATUS_ERROR; + return 0; + } + klm_n++; + } + return klm_n; +} + +static __rte_always_inline int +mlx5_crypto_wqe_set(struct mlx5_crypto_priv *priv, + struct mlx5_crypto_qp *qp, + struct rte_crypto_op *op, + struct mlx5_umr_wqe *umr) +{ + struct mlx5_crypto_session *sess = get_sym_session_private_data + (op->sym->session, mlx5_crypto_driver_id); + struct mlx5_wqe_cseg *cseg = &umr->ctr; + struct mlx5_wqe_mkey_cseg *mkc = &umr->mkc; + struct mlx5_wqe_dseg *klms = &umr->kseg[0]; + struct mlx5_wqe_umr_bsf_seg *bsf = ((struct mlx5_wqe_umr_bsf_seg *) + RTE_PTR_ADD(umr, priv->umr_wqe_size)) - 1; + uint32_t ds; + bool ipl = op->sym->m_dst == NULL || op->sym->m_dst == op->sym->m_src; + /* Set UMR WQE. */ + uint32_t klm_n = mlx5_crypto_klms_set(priv, qp, op, + ipl ? op->sym->m_src : op->sym->m_dst, klms); + + if (unlikely(klm_n == 0)) + return 0; + bsf->bs_bpt_eo_es = sess->bs_bpt_eo_es; + if (unlikely(!sess->bsp_res)) { + bsf->bsp_res = mlx5_crypto_get_block_size(op); + if (unlikely(bsf->bsp_res == UINT32_MAX)) { + op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; + return 0; + } + } else { + bsf->bsp_res = sess->bsp_res; + } + bsf->raw_data_size = rte_cpu_to_be_32(op->sym->cipher.data.length); + memcpy(bsf->xts_initial_tweak, + rte_crypto_op_ctod_offset(op, uint8_t *, sess->iv_offset), 16); + bsf->res_dp = sess->dek_id; + mkc->len = rte_cpu_to_be_64(op->sym->cipher.data.length); + cseg->opcode = rte_cpu_to_be_32((qp->db_pi << 8) | MLX5_OPCODE_UMR); + qp->db_pi += priv->umr_wqe_stride; + /* Set RDMA_WRITE WQE. */ + cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size); + klms = RTE_PTR_ADD(cseg, sizeof(struct mlx5_rdma_write_wqe)); + if (!ipl) { + klm_n = mlx5_crypto_klms_set(priv, qp, op, op->sym->m_src, + klms); + if (unlikely(klm_n == 0)) + return 0; + } else { + memcpy(klms, &umr->kseg[0], sizeof(*klms) * klm_n); + } + ds = 2 + klm_n; + cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) | ds); + cseg->opcode = rte_cpu_to_be_32((qp->db_pi << 8) | + MLX5_OPCODE_RDMA_WRITE); + ds = RTE_ALIGN(ds, 4); + qp->db_pi += ds >> 2; + /* Set NOP WQE if needed. */ + if (priv->max_rdmar_ds > ds) { + cseg += ds; + ds = priv->max_rdmar_ds - ds; + cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) | ds); + cseg->opcode = rte_cpu_to_be_32((qp->db_pi << 8) | + MLX5_OPCODE_NOP); + qp->db_pi += ds >> 2; /* Here, DS is 4 aligned for sure. */ + } + qp->wqe = (uint8_t *)cseg; + return 1; +} + +static __rte_always_inline void +mlx5_crypto_uar_write(uint64_t val, struct mlx5_crypto_priv *priv) +{ +#ifdef RTE_ARCH_64 + *priv->uar_addr = val; +#else /* !RTE_ARCH_64 */ + rte_spinlock_lock(&priv->uar32_sl); + *(volatile uint32_t *)priv->uar_addr = val; + rte_io_wmb(); + *((volatile uint32_t *)priv->uar_addr + 1) = val >> 32; + rte_spinlock_unlock(&priv->uar32_sl); +#endif +} + +static uint16_t +mlx5_crypto_enqueue_burst(void *queue_pair, struct rte_crypto_op **ops, + uint16_t nb_ops) +{ + struct mlx5_crypto_qp *qp = queue_pair; + struct mlx5_crypto_priv *priv = qp->priv; + struct mlx5_umr_wqe *umr; + struct rte_crypto_op *op; + uint16_t mask = qp->entries_n - 1; + uint16_t remain = qp->entries_n - (qp->pi - qp->ci); + + if (remain < nb_ops) + nb_ops = remain; + else + remain = nb_ops; + if (unlikely(remain == 0)) + return 0; + do { + op = *ops++; + umr = RTE_PTR_ADD(qp->umem_buf, priv->wqe_set_size * qp->pi); + if (unlikely(mlx5_crypto_wqe_set(priv, qp, op, umr) == 0)) { + if (remain != nb_ops) + break; + return 0; + } + qp->ops[qp->pi] = op; + qp->pi = (qp->pi + 1) & mask; + } while (--remain); + rte_io_wmb(); + qp->db_rec[MLX5_SND_DBR] = rte_cpu_to_be_32(qp->db_pi); + rte_wmb(); + mlx5_crypto_uar_write(*(volatile uint64_t *)qp->wqe, qp->priv); + rte_wmb(); + return nb_ops; +} + +static __rte_noinline void +mlx5_crypto_cqe_err_handle(struct mlx5_crypto_qp *qp, struct rte_crypto_op *op) +{ + const uint32_t idx = qp->ci & (qp->entries_n - 1); + volatile struct mlx5_err_cqe *cqe = (volatile struct mlx5_err_cqe *) + &qp->cq_obj.cqes[idx]; + + op->status = RTE_CRYPTO_OP_STATUS_ERROR; + DRV_LOG(ERR, "CQE ERR:%x.\n", rte_be_to_cpu_32(cqe->syndrome)); +} + +static uint16_t +mlx5_crypto_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops, + uint16_t nb_ops) +{ + struct mlx5_crypto_qp *qp = queue_pair; + volatile struct mlx5_cqe *restrict cqe; + struct rte_crypto_op *restrict op; + const unsigned int cq_size = qp->entries_n; + const unsigned int mask = cq_size - 1; + uint32_t idx; + uint32_t next_idx = qp->ci & mask; + const uint16_t max = RTE_MIN((uint16_t)(qp->pi - qp->ci), nb_ops); + uint16_t i = 0; + int ret; + + if (unlikely(max == 0)) + return 0; + do { + idx = next_idx; + next_idx = (qp->ci + 1) & mask; + op = qp->ops[idx]; + cqe = &qp->cq_obj.cqes[idx]; + ret = check_cqe(cqe, cq_size, qp->ci); + rte_io_rmb(); + if (unlikely(ret != MLX5_CQE_STATUS_SW_OWN)) { + if (unlikely(ret != MLX5_CQE_STATUS_HW_OWN)) + mlx5_crypto_cqe_err_handle(qp, op); + break; + } + op->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + ops[i++] = op; + qp->ci++; + } while (i < max); + if (likely(i != 0)) { + rte_io_wmb(); + qp->cq_obj.db_rec[0] = rte_cpu_to_be_32(qp->ci); + } + return i; +} + static void mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp) { @@ -521,8 +798,9 @@ mlx5_crypto_hw_global_prepare(struct mlx5_crypto_priv *priv) if (mlx5_crypto_pd_create(priv) != 0) return -1; priv->uar = mlx5_devx_alloc_uar(priv->ctx, -1); - if (priv->uar == NULL || mlx5_os_get_devx_uar_reg_addr(priv->uar) == - NULL) { + if (priv->uar) + priv->uar_addr = mlx5_os_get_devx_uar_reg_addr(priv->uar); + if (priv->uar == NULL || priv->uar_addr == NULL) { rte_errno = errno; claim_zero(mlx5_glue->dealloc_pd(priv->pd)); DRV_LOG(ERR, "Failed to allocate UAR."); @@ -752,8 +1030,8 @@ mlx5_crypto_pci_probe(struct rte_pci_driver *pci_drv, DRV_LOG(INFO, "Crypto device %s was created successfully.", ibv->name); crypto_dev->dev_ops = &mlx5_crypto_ops; - crypto_dev->dequeue_burst = NULL; - crypto_dev->enqueue_burst = NULL; + crypto_dev->dequeue_burst = mlx5_crypto_dequeue_burst; + crypto_dev->enqueue_burst = mlx5_crypto_enqueue_burst; crypto_dev->feature_flags = MLX5_CRYPTO_FEATURE_FLAGS; crypto_dev->driver_id = mlx5_crypto_driver_id; priv = crypto_dev->data->dev_private; diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h index d9b1ff8e99..1350513b9e 100644 --- a/drivers/crypto/mlx5/mlx5_crypto.h +++ b/drivers/crypto/mlx5/mlx5_crypto.h @@ -36,6 +36,9 @@ struct mlx5_crypto_priv { uint16_t umr_wqe_size; uint16_t umr_wqe_stride; uint16_t max_rdmar_ds; +#ifndef RTE_ARCH_64 + rte_spinlock_t uar32_sl; +#endif /* RTE_ARCH_64 */ }; struct mlx5_crypto_qp {