From patchwork Mon Jan 22 08:46:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akhil Goyal X-Patchwork-Id: 34223 X-Patchwork-Delegate: pablo.de.lara.guarch@intel.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2A844199B0; Mon, 22 Jan 2018 12:24:55 +0100 (CET) Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0050.outbound.protection.outlook.com [104.47.34.50]) by dpdk.org (Postfix) with ESMTP id 76F89A49F for ; Mon, 22 Jan 2018 12:24:49 +0100 (CET) Received: from CY4PR03CA0084.namprd03.prod.outlook.com (2603:10b6:910:4d::25) by SN2PR03MB2368.namprd03.prod.outlook.com (2603:10b6:804:e::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.428.17; Mon, 22 Jan 2018 11:24:47 +0000 Received: from BY2FFO11FD023.protection.gbl (2a01:111:f400:7c0c::167) by CY4PR03CA0084.outlook.office365.com (2603:10b6:910:4d::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.428.17 via Frontend Transport; Mon, 22 Jan 2018 11:24:47 +0000 Authentication-Results: spf=fail (sender IP is 192.88.168.50) smtp.mailfrom=nxp.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=fail action=none header.from=nxp.com; Received-SPF: Fail (protection.outlook.com: domain of nxp.com does not designate 192.88.168.50 as permitted sender) receiver=protection.outlook.com; client-ip=192.88.168.50; helo=tx30smr01.am.freescale.net; Received: from tx30smr01.am.freescale.net (192.88.168.50) by BY2FFO11FD023.mail.protection.outlook.com (10.1.15.212) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.20.428.12 via Frontend Transport; Mon, 22 Jan 2018 11:24:46 +0000 Received: from netperf2.ap.freescale.net ([10.232.133.164]) by tx30smr01.am.freescale.net (8.14.3/8.14.0) with ESMTP id w0M8ogN3032715; Mon, 22 Jan 2018 01:50:51 -0700 From: Akhil Goyal To: CC: , , , , , , , Akhil Goyal Date: Mon, 22 Jan 2018 14:16:38 +0530 Message-ID: <20180122084638.32370-3-akhil.goyal@nxp.com> X-Mailer: git-send-email 2.9.3 In-Reply-To: <20180122084638.32370-1-akhil.goyal@nxp.com> References: <20180119115102.2247-1-akhil.goyal@nxp.com> <20180122084638.32370-1-akhil.goyal@nxp.com> X-EOPAttributedMessage: 0 X-Matching-Connectors: 131610938869283351; (91ab9b29-cfa4-454e-5278-08d120cd25b8); () X-Forefront-Antispam-Report: CIP:192.88.168.50; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(39860400002)(346002)(376002)(396003)(39380400002)(2980300002)(1110001)(1109001)(339900001)(189003)(199004)(104016004)(4326008)(316002)(305945005)(36756003)(336011)(50466002)(498600001)(51416003)(106466001)(48376002)(105606002)(47776003)(2906002)(356003)(26005)(1076002)(6666003)(2351001)(86362001)(6916009)(76176011)(5660300001)(97736004)(53936002)(2950100002)(81166006)(8676002)(81156014)(68736007)(50226002)(16586007)(54906003)(59450400001)(77096007)(85426001)(8936002); DIR:OUT; SFP:1101; SCL:1; SRVR:SN2PR03MB2368; H:tx30smr01.am.freescale.net; FPR:; SPF:Fail; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD023; 1:y2ul9ts+FGq+XTc0Hdr/mNh5WaOr7pieNBWLqy6mQ5ysb4LJ/e+clz3jHB+0qh9OZrgyghdc4YqJv3/MxpXXjRJCrDumDNqIl3HlJVIgW+/IGq6TYqS/9l2Kz6Rilcub MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4b9b7421-87a7-4994-ef0f-08d5618abe3f X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(5600026)(4604075)(2017052603307); SRVR:SN2PR03MB2368; X-Microsoft-Exchange-Diagnostics: 1; SN2PR03MB2368; 3:Jik8Hu6KQhHoYCi7dIyeI6W1Sv1hek43dFStNNMc3xg45iSLKoxhYm+t7yYXAL/hOk+zxJ7SmRVypRwEqqbRV+yQArdOBDnHhx23NtxJN7aOw/OAtzl1ghM5bqx8R7R2Iy+D2o++McRLHpc4VQ7jE8w0i6bhpYvh5j/S4J/fDTRD1mheOqPX2am9fUStYpw4ITU1MEvvpqk5TlUGqTQ68kWH4kEWB4Xwel7FLIUYiugWLhdARvrJdxy7AXJ4FbYilsCZWbExTtzCZw3QLCBc1I20bzEgFOoKcT+kI6WTwZTj76Z7TrngR85VZLRVGCfjPpzaQMraBUbVd6t4WDpiktjH03WpOjVcVn/WTE0wABU=; 25:WjCUgfhouVSe5wRiRCw7k9yynSU4PdHNiOCzTTK1YfXT/5YRpmiVmDEA378NrPr/b5WHFXaW2rdNB6Gw6daUiW131J7KwvE6kX6jg3NiOadvaIqpL8e4V+r8uS81wT4uZZWEK7k/M069Mod3E4mlHOrdcRYERuqGjj3Gumo1SHDxSgPzgpdEusIdrq9gBdf3mMGx79w3klTUo2gohMWnopus+5oId4f136nv0A4wS5fboY+xfDeH4GWMNcoJQ106r/AY68FP8JHnjU7UmDyrIXE+5Ebg6nRcRiCU+7KAmbRxCJcXzVmUXuFC12OiSLm+AJpwntCwIIgsJ2GK7n8dEw== X-MS-TrafficTypeDiagnostic: SN2PR03MB2368: X-Microsoft-Exchange-Diagnostics: 1; SN2PR03MB2368; 31:LYT0E+XIHx9DCXznNFngwyhiQ0dsSGqVz4or6CnzMgN/miS74R2h4gH4yT5O6aPpA0JOiNu3dUgqLgotgq5pXj06zr4DnE9RGrsI1DqnQ+x8f4AygeYpZ4Ko5NjJsRdhcwOH+WiWX3UaT5ZoZ06PHfisHFusM4z77LOeupCRYMxRuRRaBCyE8ZIXKiNse6hVrNV5fDb/8YpxoJc3C+jnwQfSnJyLkt6ToVU7jASKkRk=; 4:KS/5Jn4Zeo5Pa2iuS570VZ7a5l8RrH8xxKtyMk9TKKRRpDmiTHZVNON5go4ldEcYacyEP6suUpxBl+xMZzvmqxH6IXlSI9BDxLbLC4oY0AGx/xvddw2VFxUVSPgS0sS38Bw6korFUanRITMUCyXETjFIM4l8kfkBvNKrWgtlVGUVIfbC+IvulcOwCE4NZQh+fJJtapxt2/jPZj+fpeaBPqzjY/fK6Gny9jpl0xC3M1CTdNtvSwgCi4cwKDyGCoSWsOfjXsL8pM/u/V9d+980MZYz+KoUYMI5aHx9c3hHfbnPtK3lgyIjoxZRnSY4Susb X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(185117386973197); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6095135)(2401047)(8121501046)(5005006)(3231023)(2400081)(944501161)(93006095)(93001095)(10201501046)(3002001)(6055026)(6096035)(201703131430075)(201703131433075)(201703131448075)(201703151042153)(20161123561025)(20161123563025)(20161123559100)(20161123556025)(20161123565025)(201708071742011); SRVR:SN2PR03MB2368; BCL:0; PCL:0; RULEID:(100000803101)(100110400095)(400006); SRVR:SN2PR03MB2368; X-Forefront-PRVS: 0560A2214D X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; SN2PR03MB2368; 23:Aw71UtuL9UvWa0qQzzB8jtWLztYqaPmS9+5pA5VMR?= R87hUP/Me//ULtbwW5Mng3znJHFCR0ETbJ0qqh8iHFhggGDkYpgnwpjrXBXAYK53nGzeqi+/Wptn/4nqf20VpImOkb+FhWLhCdTrSSp4+mIPy+EwBW4B1+Ag1OwXgkalCxl5FJjPDTZVM//wjqRu1Y54cje6enu25q0+gNTQo4XRWsmV39658eOIU4usbZ9AK9WoMIle+UuKILIjkA7vI5m19XWYH6318EXYLA7gVCARpgQEDo+koeNc1zTGvtlJ/T1hXhuawFWrmr2KcSbM1jz3plvD00IffJ/08O1adQHAf0z/jRK3RiCGPHc5UUJg92omPGgPib5c3lwQE8yCPASpPz4z4xKUu6pycw8oFsT/YDTYZkw0IZtJ9jKheYtmbS7J92m554FlChrndZ9YOIiKHkPI62F379KYxvW4AWu8RvZk8O/NpZQeJur4x+O9+FWGIor+vFW5kbCl8pfiB4jmJhH/Ju4QBmvYj8IdbxJw7eoy0toxnWXaZba56PBmfzpHXT1cJ+i0C966WHbZmyULxIVgJ89PFJOrSVrle8ot/o7VJR8LxgrdWyuS6Cz+XYVOwEJUczwUO+hMRtOWB4YbdhfS4yErJaxnUzz0RT308xxCftrcWcpq69z4Kya1u/B283+0KL3YV5/T9bA66fPr+0HLabZBkU/MoW98jFyJO2ws8mZeGeWTZItknIBSzR61rRhZXBTzKSma3nxzGQKBxMOx5eyT9NURCSdM7Q6ER5WvBDNPWSLjrXneX973unWJJxtfjOx8Rbbguy8pd+kOPfIG0XU4yk3VLWBIH84vVlAABDJZskTdpyKaws8m5LxaRvw1tZyDx+2c7jDB/JVL1rcJSIPu0ZcRNKTwGAGU2t5dyJqk+Z2qi/4jS2n2ef6d0On0/K+pQF+Fxu8iva7ARWxQfGr3sAIb61XrL9CQdJpgJcpM4UGfCCtfNk/k+bVaMuhfkrefuzm9OA5dRugM6JqPIeYiirvSmRK2ISckOcLbUeb76lCBj1+OwG7gf7NtzBzCCnFIRWHnO+fLKguCn8C+BUb0q0NS7313Ys62UECrce4HV5+zP+HflfYs7s= X-Microsoft-Exchange-Diagnostics: 1; SN2PR03MB2368; 6:dxDy8VC1yD5k3te+BH7VbhntIoshBymJdR/ZeFDpeybCawEiSSm2KJHFliz2TvQDYTZE4F6bsowyZBaYzSirOVTipqHVUjxvXNLbbiNwbS1iKs1BmTGHH702l81AIub9i0I+hUnCzQu62uMo93E0tarnYLr85TrBSIVb53bsVohOVL6u3ncBrJlPedzdOv5lO5ZHE81WyFWIXMjSQcvPyeoaYPt2sg0s3Uhc1yaMb655OxAvbBrTcgtnb+VZQwswe2A/OjpbTVUmAu5jV3oGoy2/cGpmRrTY38aoC9kDovX9Hw7bEpauxCwDkyj+SSNDlxTPBcqp0UJJHtsiUbPgwTDGPo57V8tspp0GAGnMYyI=; 5:367c5TzxwhRGmccZndV75nGS6aG9TyyAIXoD3NvvtvUuIF0hJxrKi/0vhWdIV/3axEM2l/hJ3w39p4smQNgOgWl5AK/2vjGdvI+E1h+nCadIvfMDL5uMpotnLJl2GSDSxf1MIzpv+06IYubERmzwnzzQfqdhdDqCIe3+Mt5m9m0=; 24:dch8gcGu9o5YBtY+kRNihlv6YmP3QklPTDwb3y2VyFexKDlHNv6jutTmVjSQ3hmKf5Da2xsW9c3JlJux+za6NHhW+8dxp1hwBjcxVWS1+vc=; 7:I+oysncXpyx7HWqBa6D/VWEWnerCcEVDdTsWWkr+d91U1BlApywmrGPasUSfy++hwhpMEBWjOSfb755GET/UIWdmLbSv2E/zORFPzNew6LiUc405DzhSZ7mNiP6dlQOQJhPFH5m5rm0k7JNv0dpLxmO8lHOj21SqKPCRLhKxGMrF/aFZMrn1s2wzzOOIhhMfHSbJmYRhoBnF/VcwglEoCyzUAyE4JzWcNDpJnFX78jAXAa667J+/DeN4scmdzLU8 SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jan 2018 11:24:46.6163 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4b9b7421-87a7-4994-ef0f-08d5618abe3f X-MS-Exchange-CrossTenant-Id: 5afe0b00-7697-4969-b663-5eab37d5f47e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=5afe0b00-7697-4969-b663-5eab37d5f47e; Ip=[192.88.168.50]; Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN2PR03MB2368 Subject: [dpdk-dev] [PATCH v2 3/3] crypto/dpaa_sec: support for scatter gather X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Signed-off-by: Alok Makhariya Signed-off-by: Akhil Goyal Acked-by: Hemant Agrawal --- doc/guides/cryptodevs/features/dpaa_sec.ini | 1 + drivers/crypto/dpaa_sec/dpaa_sec.c | 501 +++++++++++++++++++++++++-- test/test/test_cryptodev.c | 10 + test/test/test_cryptodev_aes_test_vectors.h | 11 +- test/test/test_cryptodev_hash_test_vectors.h | 6 +- 5 files changed, 498 insertions(+), 31 deletions(-) diff --git a/doc/guides/cryptodevs/features/dpaa_sec.ini b/doc/guides/cryptodevs/features/dpaa_sec.ini index deab53a..260fae7 100644 --- a/doc/guides/cryptodevs/features/dpaa_sec.ini +++ b/doc/guides/cryptodevs/features/dpaa_sec.ini @@ -8,6 +8,7 @@ Symmetric crypto = Y Sym operation chaining = Y HW Accelerated = Y Protocol offload = Y +Mbuf scatter gather = Y ; ; Supported crypto algorithms of the 'dpaa_sec' crypto driver. diff --git a/drivers/crypto/dpaa_sec/dpaa_sec.c b/drivers/crypto/dpaa_sec/dpaa_sec.c index a402e61..18681cf 100644 --- a/drivers/crypto/dpaa_sec/dpaa_sec.c +++ b/drivers/crypto/dpaa_sec/dpaa_sec.c @@ -599,6 +599,86 @@ dpaa_sec_deq(struct dpaa_sec_qp *qp, struct rte_crypto_op **ops, int nb_ops) return pkts; } +static inline struct dpaa_sec_job * +build_auth_only_sg(struct rte_crypto_op *op, dpaa_sec_session *ses) +{ + struct rte_crypto_sym_op *sym = op->sym; + struct rte_mbuf *mbuf = sym->m_src; + struct dpaa_sec_job *cf; + struct dpaa_sec_op_ctx *ctx; + struct qm_sg_entry *sg, *out_sg, *in_sg; + phys_addr_t start_addr; + uint8_t *old_digest, extra_segs; + + if (is_decode(ses)) + extra_segs = 3; + else + extra_segs = 2; + + if ((mbuf->nb_segs + extra_segs) > MAX_SG_ENTRIES) { + PMD_TX_LOG(ERR, "Auth: Max sec segs supported is %d\n", + MAX_SG_ENTRIES); + return NULL; + } + ctx = dpaa_sec_alloc_ctx(ses); + if (!ctx) + return NULL; + + cf = &ctx->job; + ctx->op = op; + old_digest = ctx->digest; + + /* output */ + out_sg = &cf->sg[0]; + qm_sg_entry_set64(out_sg, sym->auth.digest.phys_addr); + out_sg->length = ses->digest_length; + cpu_to_hw_sg(out_sg); + + /* input */ + in_sg = &cf->sg[1]; + /* need to extend the input to a compound frame */ + in_sg->extension = 1; + in_sg->final = 1; + in_sg->length = sym->auth.data.length; + qm_sg_entry_set64(in_sg, dpaa_mem_vtop_ctx(ctx, &cf->sg[2])); + + /* 1st seg */ + sg = in_sg + 1; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->auth.data.offset; + sg->offset = sym->auth.data.offset; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + + if (is_decode(ses)) { + /* Digest verification case */ + cpu_to_hw_sg(sg); + sg++; + rte_memcpy(old_digest, sym->auth.digest.data, + ses->digest_length); + start_addr = dpaa_mem_vtop_ctx(ctx, old_digest); + qm_sg_entry_set64(sg, start_addr); + sg->length = ses->digest_length; + in_sg->length += ses->digest_length; + } else { + /* Digest calculation case */ + sg->length -= ses->digest_length; + } + sg->final = 1; + cpu_to_hw_sg(sg); + cpu_to_hw_sg(in_sg); + + return cf; +} + /** * packet looks like: * |<----data_len------->| @@ -669,6 +749,101 @@ build_auth_only(struct rte_crypto_op *op, dpaa_sec_session *ses) } static inline struct dpaa_sec_job * +build_cipher_only_sg(struct rte_crypto_op *op, dpaa_sec_session *ses) +{ + struct rte_crypto_sym_op *sym = op->sym; + struct dpaa_sec_job *cf; + struct dpaa_sec_op_ctx *ctx; + struct qm_sg_entry *sg, *out_sg, *in_sg; + struct rte_mbuf *mbuf; + uint8_t req_segs; + uint8_t *IV_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, + ses->iv.offset); + + if (sym->m_dst) { + mbuf = sym->m_dst; + req_segs = mbuf->nb_segs + sym->m_src->nb_segs + 3; + } else { + mbuf = sym->m_src; + req_segs = mbuf->nb_segs * 2 + 3; + } + + if (req_segs > MAX_SG_ENTRIES) { + PMD_TX_LOG(ERR, "Cipher: Max sec segs supported is %d\n", + MAX_SG_ENTRIES); + return NULL; + } + + ctx = dpaa_sec_alloc_ctx(ses); + if (!ctx) + return NULL; + + cf = &ctx->job; + ctx->op = op; + + /* output */ + out_sg = &cf->sg[0]; + out_sg->extension = 1; + out_sg->length = sym->cipher.data.length; + qm_sg_entry_set64(out_sg, dpaa_mem_vtop_ctx(ctx, &cf->sg[2])); + cpu_to_hw_sg(out_sg); + + /* 1st seg */ + sg = &cf->sg[2]; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->cipher.data.offset; + sg->offset = sym->cipher.data.offset; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + sg->final = 1; + cpu_to_hw_sg(sg); + + /* input */ + mbuf = sym->m_src; + in_sg = &cf->sg[1]; + in_sg->extension = 1; + in_sg->final = 1; + in_sg->length = sym->cipher.data.length + ses->iv.length; + + sg++; + qm_sg_entry_set64(in_sg, dpaa_mem_vtop_ctx(ctx, sg)); + cpu_to_hw_sg(in_sg); + + /* IV */ + qm_sg_entry_set64(sg, dpaa_mem_vtop(IV_ptr)); + sg->length = ses->iv.length; + cpu_to_hw_sg(sg); + + /* 1st seg */ + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->cipher.data.offset; + sg->offset = sym->cipher.data.offset; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + sg->final = 1; + cpu_to_hw_sg(sg); + + return cf; +} + +static inline struct dpaa_sec_job * build_cipher_only(struct rte_crypto_op *op, dpaa_sec_session *ses) { struct rte_crypto_sym_op *sym = op->sym; @@ -724,6 +899,145 @@ build_cipher_only(struct rte_crypto_op *op, dpaa_sec_session *ses) } static inline struct dpaa_sec_job * +build_cipher_auth_gcm_sg(struct rte_crypto_op *op, dpaa_sec_session *ses) +{ + struct rte_crypto_sym_op *sym = op->sym; + struct dpaa_sec_job *cf; + struct dpaa_sec_op_ctx *ctx; + struct qm_sg_entry *sg, *out_sg, *in_sg; + struct rte_mbuf *mbuf; + uint8_t req_segs; + uint8_t *IV_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, + ses->iv.offset); + + if (sym->m_dst) { + mbuf = sym->m_dst; + req_segs = mbuf->nb_segs + sym->m_src->nb_segs + 4; + } else { + mbuf = sym->m_src; + req_segs = mbuf->nb_segs * 2 + 4; + } + + if (ses->auth_only_len) + req_segs++; + + if (req_segs > MAX_SG_ENTRIES) { + PMD_TX_LOG(ERR, "AEAD: Max sec segs supported is %d\n", + MAX_SG_ENTRIES); + return NULL; + } + + ctx = dpaa_sec_alloc_ctx(ses); + if (!ctx) + return NULL; + + cf = &ctx->job; + ctx->op = op; + + rte_prefetch0(cf->sg); + + /* output */ + out_sg = &cf->sg[0]; + out_sg->extension = 1; + if (is_encode(ses)) + out_sg->length = sym->aead.data.length + ses->auth_only_len + + ses->digest_length; + else + out_sg->length = sym->aead.data.length + ses->auth_only_len; + + /* output sg entries */ + sg = &cf->sg[2]; + qm_sg_entry_set64(out_sg, dpaa_mem_vtop_ctx(ctx, sg)); + cpu_to_hw_sg(out_sg); + + /* 1st seg */ + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->aead.data.offset + + ses->auth_only_len; + sg->offset = sym->aead.data.offset - ses->auth_only_len; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + sg->length -= ses->digest_length; + + if (is_encode(ses)) { + cpu_to_hw_sg(sg); + /* set auth output */ + sg++; + qm_sg_entry_set64(sg, sym->aead.digest.phys_addr); + sg->length = ses->digest_length; + } + sg->final = 1; + cpu_to_hw_sg(sg); + + /* input */ + mbuf = sym->m_src; + in_sg = &cf->sg[1]; + in_sg->extension = 1; + in_sg->final = 1; + if (is_encode(ses)) + in_sg->length = ses->iv.length + sym->aead.data.length + + ses->auth_only_len; + else + in_sg->length = ses->iv.length + sym->aead.data.length + + ses->auth_only_len + ses->digest_length; + + /* input sg entries */ + sg++; + qm_sg_entry_set64(in_sg, dpaa_mem_vtop_ctx(ctx, sg)); + cpu_to_hw_sg(in_sg); + + /* 1st seg IV */ + qm_sg_entry_set64(sg, dpaa_mem_vtop(IV_ptr)); + sg->length = ses->iv.length; + cpu_to_hw_sg(sg); + + /* 2nd seg auth only */ + if (ses->auth_only_len) { + sg++; + qm_sg_entry_set64(sg, dpaa_mem_vtop(sym->aead.aad.data)); + sg->length = ses->auth_only_len; + cpu_to_hw_sg(sg); + } + + /* 3rd seg */ + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->aead.data.offset; + sg->offset = sym->aead.data.offset; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + + if (is_decode(ses)) { + cpu_to_hw_sg(sg); + sg++; + memcpy(ctx->digest, sym->aead.digest.data, + ses->digest_length); + qm_sg_entry_set64(sg, dpaa_mem_vtop_ctx(ctx, ctx->digest)); + sg->length = ses->digest_length; + } + sg->final = 1; + cpu_to_hw_sg(sg); + + return cf; +} + +static inline struct dpaa_sec_job * build_cipher_auth_gcm(struct rte_crypto_op *op, dpaa_sec_session *ses) { struct rte_crypto_sym_op *sym = op->sym; @@ -836,6 +1150,132 @@ build_cipher_auth_gcm(struct rte_crypto_op *op, dpaa_sec_session *ses) } static inline struct dpaa_sec_job * +build_cipher_auth_sg(struct rte_crypto_op *op, dpaa_sec_session *ses) +{ + struct rte_crypto_sym_op *sym = op->sym; + struct dpaa_sec_job *cf; + struct dpaa_sec_op_ctx *ctx; + struct qm_sg_entry *sg, *out_sg, *in_sg; + struct rte_mbuf *mbuf; + uint8_t req_segs; + uint8_t *IV_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, + ses->iv.offset); + + if (sym->m_dst) { + mbuf = sym->m_dst; + req_segs = mbuf->nb_segs + sym->m_src->nb_segs + 4; + } else { + mbuf = sym->m_src; + req_segs = mbuf->nb_segs * 2 + 4; + } + + if (req_segs > MAX_SG_ENTRIES) { + PMD_TX_LOG(ERR, "Cipher-Auth: Max sec segs supported is %d\n", + MAX_SG_ENTRIES); + return NULL; + } + + ctx = dpaa_sec_alloc_ctx(ses); + if (!ctx) + return NULL; + + cf = &ctx->job; + ctx->op = op; + + rte_prefetch0(cf->sg); + + /* output */ + out_sg = &cf->sg[0]; + out_sg->extension = 1; + if (is_encode(ses)) + out_sg->length = sym->auth.data.length + ses->digest_length; + else + out_sg->length = sym->auth.data.length; + + /* output sg entries */ + sg = &cf->sg[2]; + qm_sg_entry_set64(out_sg, dpaa_mem_vtop_ctx(ctx, sg)); + cpu_to_hw_sg(out_sg); + + /* 1st seg */ + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->auth.data.offset; + sg->offset = sym->auth.data.offset; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + sg->length -= ses->digest_length; + + if (is_encode(ses)) { + cpu_to_hw_sg(sg); + /* set auth output */ + sg++; + qm_sg_entry_set64(sg, sym->auth.digest.phys_addr); + sg->length = ses->digest_length; + } + sg->final = 1; + cpu_to_hw_sg(sg); + + /* input */ + mbuf = sym->m_src; + in_sg = &cf->sg[1]; + in_sg->extension = 1; + in_sg->final = 1; + if (is_encode(ses)) + in_sg->length = ses->iv.length + sym->auth.data.length; + else + in_sg->length = ses->iv.length + sym->auth.data.length + + ses->digest_length; + + /* input sg entries */ + sg++; + qm_sg_entry_set64(in_sg, dpaa_mem_vtop_ctx(ctx, sg)); + cpu_to_hw_sg(in_sg); + + /* 1st seg IV */ + qm_sg_entry_set64(sg, dpaa_mem_vtop(IV_ptr)); + sg->length = ses->iv.length; + cpu_to_hw_sg(sg); + + /* 2nd seg */ + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->auth.data.offset; + sg->offset = sym->auth.data.offset; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + + sg->length -= ses->digest_length; + if (is_decode(ses)) { + cpu_to_hw_sg(sg); + sg++; + memcpy(ctx->digest, sym->auth.digest.data, + ses->digest_length); + qm_sg_entry_set64(sg, dpaa_mem_vtop_ctx(ctx, ctx->digest)); + sg->length = ses->digest_length; + } + sg->final = 1; + cpu_to_hw_sg(sg); + + return cf; +} + +static inline struct dpaa_sec_job * build_cipher_auth(struct rte_crypto_op *op, dpaa_sec_session *ses) { struct rte_crypto_sym_op *sym = op->sym; @@ -1020,34 +1460,42 @@ dpaa_sec_enqueue_burst(void *qp, struct rte_crypto_op **ops, } } - /* - * Segmented buffer is not supported. - */ - if (!rte_pktmbuf_is_contiguous(op->sym->m_src)) { - op->status = RTE_CRYPTO_OP_STATUS_ERROR; - frames_to_send = loop; - nb_ops = loop; - goto send_pkts; - } auth_only_len = op->sym->auth.data.length - op->sym->cipher.data.length; - - if (is_auth_only(ses)) { - cf = build_auth_only(op, ses); - } else if (is_cipher_only(ses)) { - cf = build_cipher_only(op, ses); - } else if (is_aead(ses)) { - cf = build_cipher_auth_gcm(op, ses); - auth_only_len = ses->auth_only_len; - } else if (is_auth_cipher(ses)) { - cf = build_cipher_auth(op, ses); - } else if (is_proto_ipsec(ses)) { - cf = build_proto(op, ses); + if (rte_pktmbuf_is_contiguous(op->sym->m_src)) { + if (is_auth_only(ses)) { + cf = build_auth_only(op, ses); + } else if (is_cipher_only(ses)) { + cf = build_cipher_only(op, ses); + } else if (is_aead(ses)) { + cf = build_cipher_auth_gcm(op, ses); + auth_only_len = ses->auth_only_len; + } else if (is_auth_cipher(ses)) { + cf = build_cipher_auth(op, ses); + } else if (is_proto_ipsec(ses)) { + cf = build_proto(op, ses); + } else { + PMD_TX_LOG(ERR, "not supported sec op"); + frames_to_send = loop; + nb_ops = loop; + goto send_pkts; + } } else { - PMD_TX_LOG(ERR, "not supported sec op"); - frames_to_send = loop; - nb_ops = loop; - goto send_pkts; + if (is_auth_only(ses)) { + cf = build_auth_only_sg(op, ses); + } else if (is_cipher_only(ses)) { + cf = build_cipher_only_sg(op, ses); + } else if (is_aead(ses)) { + cf = build_cipher_auth_gcm_sg(op, ses); + auth_only_len = ses->auth_only_len; + } else if (is_auth_cipher(ses)) { + cf = build_cipher_auth_sg(op, ses); + } else { + PMD_TX_LOG(ERR, "not supported sec op"); + frames_to_send = loop; + nb_ops = loop; + goto send_pkts; + } } if (unlikely(!cf)) { frames_to_send = loop; @@ -1834,7 +2282,8 @@ dpaa_sec_dev_init(struct rte_cryptodev *cryptodev) cryptodev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - RTE_CRYPTODEV_FF_SECURITY; + RTE_CRYPTODEV_FF_SECURITY | + RTE_CRYPTODEV_FF_MBUF_SCATTER_GATHER; internals = cryptodev->data->dev_private; internals->max_nb_queue_pairs = RTE_DPAA_MAX_NB_SEC_QPS; diff --git a/test/test/test_cryptodev.c b/test/test/test_cryptodev.c index 6f22896..0f2a045 100644 --- a/test/test/test_cryptodev.c +++ b/test/test/test_cryptodev.c @@ -9365,6 +9365,16 @@ static struct unit_test_suite cryptodev_dpaa_sec_testsuite = { TEST_CASE_ST(ut_setup, ut_teardown, test_AES_GCM_authenticated_decryption_oop_test_case_1), + /** Scatter-Gather */ + TEST_CASE_ST(ut_setup, ut_teardown, + test_AES_GCM_auth_encrypt_SGL_in_place_1500B), + TEST_CASE_ST(ut_setup, ut_teardown, + test_AES_GCM_auth_encrypt_SGL_out_of_place_400B_400B), + TEST_CASE_ST(ut_setup, ut_teardown, + test_AES_GCM_auth_encrypt_SGL_out_of_place_400B_1seg), + TEST_CASE_ST(ut_setup, ut_teardown, + test_AES_GCM_auth_encrypt_SGL_out_of_place_1500B_2000B), + TEST_CASES_END() /**< NULL terminate unit test array */ } }; diff --git a/test/test/test_cryptodev_aes_test_vectors.h b/test/test/test_cryptodev_aes_test_vectors.h index 20f5285..3577ef4 100644 --- a/test/test/test_cryptodev_aes_test_vectors.h +++ b/test/test/test_cryptodev_aes_test_vectors.h @@ -1270,7 +1270,8 @@ static const struct blockcipher_test_case aes_chain_test_cases[] = { BLOCKCIPHER_TEST_TARGET_PMD_OPENSSL | BLOCKCIPHER_TEST_TARGET_PMD_QAT | BLOCKCIPHER_TEST_TARGET_PMD_SCHEDULER | - BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC + BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC }, { .test_descr = "AES-128-CBC HMAC-SHA1 Decryption Digest " @@ -1292,7 +1293,8 @@ static const struct blockcipher_test_case aes_chain_test_cases[] = { .test_data = &aes_test_data_4, .op_mask = BLOCKCIPHER_TEST_OP_AUTH_VERIFY_DEC, .feature_mask = BLOCKCIPHER_TEST_FEATURE_SG, - .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC + .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC }, { .test_descr = "AES-128-CBC HMAC-SHA1 Decryption Digest " @@ -1401,6 +1403,7 @@ static const struct blockcipher_test_case aes_chain_test_cases[] = { BLOCKCIPHER_TEST_TARGET_PMD_OPENSSL | BLOCKCIPHER_TEST_TARGET_PMD_QAT | BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC | BLOCKCIPHER_TEST_TARGET_PMD_SCHEDULER }, { @@ -1555,6 +1558,7 @@ static const struct blockcipher_test_case aes_cipheronly_test_cases[] = { .feature_mask = BLOCKCIPHER_TEST_FEATURE_SG | BLOCKCIPHER_TEST_FEATURE_OOP, .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_OPENSSL | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC | BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC }, { @@ -1573,7 +1577,8 @@ static const struct blockcipher_test_case aes_cipheronly_test_cases[] = { .test_data = &aes_test_data_10, .op_mask = BLOCKCIPHER_TEST_OP_DECRYPT, .feature_mask = BLOCKCIPHER_TEST_FEATURE_SG, - .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC + .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC }, { .test_descr = "AES-256-CBC Encryption", diff --git a/test/test/test_cryptodev_hash_test_vectors.h b/test/test/test_cryptodev_hash_test_vectors.h index bd793f4..93dacb7 100644 --- a/test/test/test_cryptodev_hash_test_vectors.h +++ b/test/test/test_cryptodev_hash_test_vectors.h @@ -382,7 +382,8 @@ static const struct blockcipher_test_case hash_test_cases[] = { .test_data = &hmac_sha1_test_vector, .op_mask = BLOCKCIPHER_TEST_OP_AUTH_GEN, .feature_mask = BLOCKCIPHER_TEST_FEATURE_SG, - .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC + .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC }, { .test_descr = "HMAC-SHA1 Digest Verify", @@ -400,7 +401,8 @@ static const struct blockcipher_test_case hash_test_cases[] = { .test_data = &hmac_sha1_test_vector, .op_mask = BLOCKCIPHER_TEST_OP_AUTH_VERIFY, .feature_mask = BLOCKCIPHER_TEST_FEATURE_SG, - .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC + .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC }, { .test_descr = "SHA224 Digest",