From patchwork Thu Jul 15 15:08:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shiri Kuzin X-Patchwork-Id: 95898 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C4FF8A0A0C; Thu, 15 Jul 2021 17:10:36 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 99DE941324; Thu, 15 Jul 2021 17:09:29 +0200 (CEST) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2062.outbound.protection.outlook.com [40.107.93.62]) by mails.dpdk.org (Postfix) with ESMTP id 69622412F3 for ; Thu, 15 Jul 2021 17:09:22 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=j0ey117DIEBd/tfha4AbS7qo+4E29dkZzHggCYjhOaPLJdRh0FWb7jsHs2TNHoK/05iVDAl4lWZo4h3ZcT0jAJ0OXK42Fyu6gVTOheTgmpf/RAGdB1dEhRcaUIEXyl/8atj1/D02sINjr/I/ccaG5s6C9/HXswfPUDmBYu0apcX8KNUGEGw060pS0iYS+W7BlCdPwzZLYgB0mtXkWxLihXlSSf9exMgDSgNqNPFShsqIvVHpqIU0Tfff4E9H4EUTTGEdrudx2gTnEH+rvkGsT27ADxblo/KbSlbEnJnGB3mx+UbROnDgtRVj3hf5TwTZBp6umu5DAhSxRO2mFdADbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wVwQ7Eqq76RE+rii2v4RFdk1WzYL7s/6A3OWdXzqUwg=; b=nBPVLLQpo1QAcRLB1ncTIlwpluEeKUereVy5sqsrPPpNm0aJ2F2g8X2YRfII9PJBaIds6kKe81HqPmqv//IRu49lsPhnq3p4mD9OYYQ7pLUKIhnfgpKeCvqr/g/EU56hofiPCocdZS1CDIUDuM+o92NE659JfgdXKLqOAhKlLZxaoJ1rCjkH48xS8oy1nNs1om7ZtmdbqnJFuq7oRPZ7nwH9izJ8lFnHoL1TmJThhYCj7KOeIqTQS0ZDdfxlGLWnYh44k0KXzlvdokuMTEqCQ7ZdReEecerHHM1D9zPQusSC5bXwA+RGgdkS/9uit4J4oawuDfvRV55YI1eqC4rIkQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wVwQ7Eqq76RE+rii2v4RFdk1WzYL7s/6A3OWdXzqUwg=; b=LnDrkqWZcqjIWrHRKRaMerKkjYQPXMUdEubTpWg01aDkH3GBIXjWtXZXwzdX21CwFIArk9EExigMdpSjgxUBIW2TEQUb4T4jcKQfU3QqDHglbhYct6DHKVTb/d8KOVsUQIAqgic6IxaGGEn4vR2L24FHm+71TBZxosp8b/Zp4NfGGjIaDHTpnWk6lrc/6BRhKz0YZgYi4zB9AhZckjWVBfTR3VarRF3bBgB9u8cm7ckIBZ3y6eYK7mu0Dj8Oi+zaiHGqgW28l+OK3NgT5wQ97ioO87U60elKLNdJBbnrFX7853ZmPcywm9fOf8g9AMboUI1QJht2tw6HkfAkkmihnQ== Received: from MW3PR06CA0016.namprd06.prod.outlook.com (2603:10b6:303:2a::21) by DM6PR12MB4140.namprd12.prod.outlook.com (2603:10b6:5:221::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21; Thu, 15 Jul 2021 15:09:21 +0000 Received: from CO1NAM11FT035.eop-nam11.prod.protection.outlook.com (2603:10b6:303:2a:cafe::ee) by MW3PR06CA0016.outlook.office365.com (2603:10b6:303:2a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Thu, 15 Jul 2021 15:09:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by CO1NAM11FT035.mail.protection.outlook.com (10.13.175.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4331.21 via Frontend Transport; Thu, 15 Jul 2021 15:09:20 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 15 Jul 2021 15:09:18 +0000 From: Shiri Kuzin To: CC: , , , Date: Thu, 15 Jul 2021 18:08:12 +0300 Message-ID: <20210715150817.51485-12-shirik@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210715150817.51485-1-shirik@nvidia.com> References: <20210708152530.25835-1-shirik@nvidia.com> <20210715150817.51485-1-shirik@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 032985e3-703f-4e5b-8d90-08d947a2861b X-MS-TrafficTypeDiagnostic: DM6PR12MB4140: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:295; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: o54EHBtszeAwmaEVdar6LZmjQ6bk6Rf6Y9B0gZP6m/wGKLLE35y8KoRtuYTq6QwmAHHnl9bOBkh7qloFxwow5XAEYFHBb1S3bD9BbNTK8SUZm7qcj+r80k1ylCYIwN6O32UxecbjUp6evIE6MggApohU0TWw/1AQuIdDuse2Q8nDVPokiPEPOVbXjAlzAnyulwE0EPtEQFarvVQD3lInonOCwGJQN10yU0cVmCy+3TO6FHrifKe7424t/W7XB9fhhNaJFxM96Gm9NY5TmdnewV2e/9HFbvaHz0JFANF0erpJaQL7zUzXiWBQu1uY/zvxbfEaywBJ7U+ChQPzELotVePSOE+A4Lr5zpmt4XMIURoEFhZjFXUQyYcZw+sXmBJtkgaU2yBHggoRuJZ+B9aID5dFyBBAGjqFCzF+lwFZgQiHco/H/z1j5Rv6XtLfzBsnw/+FmbYp9UuKWLKXnqQyyTZHgCtaNQ+tNj52OET2a8nNXV+lYOpZhJloLlwBWKWyKSzMaKeImCEonDqWAYvGRuNHv3HJ1U2l2X8dh70diQmD9aw4vqwIMK+U16niKzyf66joNLuocsYXH8khcfKNxlaXtRVHLDwnq6tD3fAd1yy9zDs3mrw9E1Oy0r+ZekY7yJ1wTeurhAoWT7COXd5s4F6iZYsyEyq4Q8ZgskfYvNKyZxosDIyrUHmKs7EzTBlERVKbMGnpMvk74hQwVOvMh9KOvvAovecL9k+0NjG1SyM= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(86362001)(36756003)(82310400003)(36906005)(34020700004)(316002)(5660300002)(7636003)(83380400001)(1076003)(54906003)(2906002)(186003)(2616005)(16526019)(426003)(6286002)(36860700001)(336012)(55016002)(6916009)(8936002)(356005)(47076005)(478600001)(8676002)(70206006)(7696005)(6666004)(4326008)(70586007)(26005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jul 2021 15:09:20.8233 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 032985e3-703f-4e5b-8d90-08d947a2861b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT035.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4140 Subject: [dpdk-dev] [PATCH v7 11/16] crypto/mlx5: add WQE set initialization X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Suanming Mou Currently, HW handles the WQEs much faster than the software, Using the constant WQE set layout can initialize most of the WQE segments in advanced, and software only needs to configure very limited segments in datapath. This accelerates the software WQE organize in datapath. This commit initializes the fixed WQE set segments. Signed-off-by: Suanming Mou Signed-off-by: Matan Azrad --- drivers/crypto/mlx5/mlx5_crypto.c | 83 +++++++++++++++++++++++++++++-- drivers/crypto/mlx5/mlx5_crypto.h | 10 +++- 2 files changed, 87 insertions(+), 6 deletions(-) diff --git a/drivers/crypto/mlx5/mlx5_crypto.c b/drivers/crypto/mlx5/mlx5_crypto.c index fc251f3f4c..e5f8d96ff7 100644 --- a/drivers/crypto/mlx5/mlx5_crypto.c +++ b/drivers/crypto/mlx5/mlx5_crypto.c @@ -297,6 +297,69 @@ mlx5_crypto_qp2rts(struct mlx5_crypto_qp *qp) return 0; } +static void +mlx5_crypto_qp_init(struct mlx5_crypto_priv *priv, struct mlx5_crypto_qp *qp) +{ + uint32_t i; + + for (i = 0 ; i < qp->entries_n; i++) { + struct mlx5_wqe_cseg *cseg = RTE_PTR_ADD(qp->umem_buf, i * + priv->wqe_set_size); + struct mlx5_wqe_umr_cseg *ucseg = (struct mlx5_wqe_umr_cseg *) + (cseg + 1); + struct mlx5_wqe_umr_bsf_seg *bsf = + (struct mlx5_wqe_umr_bsf_seg *)(RTE_PTR_ADD(cseg, + priv->umr_wqe_size)) - 1; + struct mlx5_wqe_rseg *rseg; + + /* Init UMR WQE. */ + cseg->sq_ds = rte_cpu_to_be_32((qp->qp_obj->id << 8) | + (priv->umr_wqe_size / MLX5_WSEG_SIZE)); + cseg->flags = RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR << + MLX5_COMP_MODE_OFFSET); + cseg->misc = rte_cpu_to_be_32(qp->mkey[i]->id); + ucseg->if_cf_toe_cq_res = RTE_BE32(1u << MLX5_UMRC_IF_OFFSET); + ucseg->mkey_mask = RTE_BE64(1u << 0); /* Mkey length bit. */ + ucseg->ko_to_bs = rte_cpu_to_be_32 + ((RTE_ALIGN(priv->max_segs_num, 4u) << + MLX5_UMRC_KO_OFFSET) | (4 << MLX5_UMRC_TO_BS_OFFSET)); + bsf->keytag = priv->keytag; + /* Init RDMA WRITE WQE. */ + cseg = RTE_PTR_ADD(cseg, priv->umr_wqe_size); + cseg->flags = RTE_BE32((MLX5_COMP_ALWAYS << + MLX5_COMP_MODE_OFFSET) | + MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE); + rseg = (struct mlx5_wqe_rseg *)(cseg + 1); + rseg->rkey = rte_cpu_to_be_32(qp->mkey[i]->id); + } +} + +static int +mlx5_crypto_indirect_mkeys_prepare(struct mlx5_crypto_priv *priv, + struct mlx5_crypto_qp *qp) +{ + struct mlx5_umr_wqe *umr; + uint32_t i; + struct mlx5_devx_mkey_attr attr = { + .pd = priv->pdn, + .umr_en = 1, + .crypto_en = 1, + .set_remote_rw = 1, + .klm_num = RTE_ALIGN(priv->max_segs_num, 4), + }; + + for (umr = (struct mlx5_umr_wqe *)qp->umem_buf, i = 0; + i < qp->entries_n; i++, umr = RTE_PTR_ADD(umr, priv->wqe_set_size)) { + attr.klm_array = (struct mlx5_klm *)&umr->kseg[0]; + qp->mkey[i] = mlx5_devx_cmd_mkey_create(priv->ctx, &attr); + if (!qp->mkey[i]) { + DRV_LOG(ERR, "Failed to allocate indirect mkey."); + return -1; + } + } + return 0; +} + static int mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id, const struct rte_cryptodev_qp_conf *qp_conf, @@ -307,7 +370,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id, struct mlx5_crypto_qp *qp; uint16_t log_nb_desc = rte_log2_u32(qp_conf->nb_descriptors); uint32_t umem_size = RTE_BIT32(log_nb_desc) * - MLX5_CRYPTO_WQE_SET_SIZE + + priv->wqe_set_size + sizeof(*qp->db_rec) * 2; uint32_t alloc_size = sizeof(*qp); struct mlx5_devx_cq_attr cq_attr = { @@ -317,7 +380,9 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id, if (dev->data->queue_pairs[qp_id] != NULL) mlx5_crypto_queue_pair_release(dev, qp_id); alloc_size = RTE_ALIGN(alloc_size, RTE_CACHE_LINE_SIZE); - alloc_size += sizeof(struct rte_crypto_op *) * RTE_BIT32(log_nb_desc); + alloc_size += (sizeof(struct rte_crypto_op *) + + sizeof(struct mlx5_devx_obj *)) * + RTE_BIT32(log_nb_desc); qp = rte_zmalloc_socket(__func__, alloc_size, RTE_CACHE_LINE_SIZE, socket_id); if (qp == NULL) { @@ -362,8 +427,7 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id, attr.wq_umem_id = qp->umem_obj->umem_id; attr.wq_umem_offset = 0; attr.dbr_umem_id = qp->umem_obj->umem_id; - attr.dbr_address = RTE_BIT64(log_nb_desc) * - MLX5_CRYPTO_WQE_SET_SIZE; + attr.dbr_address = RTE_BIT64(log_nb_desc) * priv->wqe_set_size; qp->qp_obj = mlx5_devx_cmd_create_qp(priv->ctx, &attr); if (qp->qp_obj == NULL) { DRV_LOG(ERR, "Failed to create QP(%u).", rte_errno); @@ -372,8 +436,17 @@ mlx5_crypto_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id, qp->db_rec = RTE_PTR_ADD(qp->umem_buf, (uintptr_t)attr.dbr_address); if (mlx5_crypto_qp2rts(qp)) goto error; - qp->ops = (struct rte_crypto_op **)RTE_ALIGN((uintptr_t)(qp + 1), + qp->mkey = (struct mlx5_devx_obj **)RTE_ALIGN((uintptr_t)(qp + 1), RTE_CACHE_LINE_SIZE); + qp->ops = (struct rte_crypto_op **)(qp->mkey + RTE_BIT32(log_nb_desc)); + qp->entries_n = 1 << log_nb_desc; + if (mlx5_crypto_indirect_mkeys_prepare(priv, qp)) { + DRV_LOG(ERR, "Cannot allocate indirect memory regions."); + rte_errno = ENOMEM; + goto error; + } + mlx5_crypto_qp_init(priv, qp); + qp->priv = priv; dev->data->queue_pairs[qp_id] = qp; return 0; error: diff --git a/drivers/crypto/mlx5/mlx5_crypto.h b/drivers/crypto/mlx5/mlx5_crypto.h index 7a386fe76a..d9b1ff8e99 100644 --- a/drivers/crypto/mlx5/mlx5_crypto.h +++ b/drivers/crypto/mlx5/mlx5_crypto.h @@ -16,7 +16,6 @@ #define MLX5_CRYPTO_DEK_HTABLE_SZ (1 << 11) #define MLX5_CRYPTO_KEY_LENGTH 80 -#define MLX5_CRYPTO_WQE_SET_SIZE 1024 struct mlx5_crypto_priv { TAILQ_ENTRY(mlx5_crypto_priv) next; @@ -24,6 +23,7 @@ struct mlx5_crypto_priv { struct rte_pci_device *pci_dev; struct rte_cryptodev *crypto_dev; void *uar; /* User Access Region. */ + volatile uint64_t *uar_addr; uint32_t pdn; /* Protection Domain number. */ uint32_t max_segs_num; /* Maximum supported data segs. */ struct ibv_pd *pd; @@ -39,13 +39,21 @@ struct mlx5_crypto_priv { }; struct mlx5_crypto_qp { + struct mlx5_crypto_priv *priv; struct mlx5_devx_cq cq_obj; struct mlx5_devx_obj *qp_obj; + struct rte_cryptodev_stats stats; struct mlx5dv_devx_umem *umem_obj; void *umem_buf; volatile uint32_t *db_rec; struct rte_crypto_op **ops; + struct mlx5_devx_obj **mkey; /* WQE's indirect mekys. */ struct mlx5_mr_ctrl mr_ctrl; + uint8_t *wqe; + uint16_t entries_n; + uint16_t pi; + uint16_t ci; + uint16_t db_pi; }; struct mlx5_crypto_dek {