From patchwork Fri Jan 19 11:51:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akhil Goyal X-Patchwork-Id: 34093 X-Patchwork-Delegate: pablo.de.lara.guarch@intel.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 606E61B297; Fri, 19 Jan 2018 14:24:59 +0100 (CET) Received: from NAM03-DM3-obe.outbound.protection.outlook.com (mail-dm3nam03on0073.outbound.protection.outlook.com [104.47.41.73]) by dpdk.org (Postfix) with ESMTP id ACC351B294 for ; Fri, 19 Jan 2018 14:24:57 +0100 (CET) Received: from BN6PR03CA0059.namprd03.prod.outlook.com (2603:10b6:404:4c::21) by SN2PR03MB2365.namprd03.prod.outlook.com (2603:10b6:804:e::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.428.17; Fri, 19 Jan 2018 13:24:55 +0000 Received: from BY2FFO11OLC007.protection.gbl (2a01:111:f400:7c0c::160) by BN6PR03CA0059.outlook.office365.com (2603:10b6:404:4c::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.428.17 via Frontend Transport; Fri, 19 Jan 2018 13:24:55 +0000 Authentication-Results: spf=fail (sender IP is 192.88.168.50) smtp.mailfrom=nxp.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=fail action=none header.from=nxp.com; Received-SPF: Fail (protection.outlook.com: domain of nxp.com does not designate 192.88.168.50 as permitted sender) receiver=protection.outlook.com; client-ip=192.88.168.50; helo=tx30smr01.am.freescale.net; Received: from tx30smr01.am.freescale.net (192.88.168.50) by BY2FFO11OLC007.mail.protection.outlook.com (10.1.14.254) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.20.345.12 via Frontend Transport; Fri, 19 Jan 2018 13:24:54 +0000 Received: from netperf2.ap.freescale.net ([10.232.133.164]) by tx30smr01.am.freescale.net (8.14.3/8.14.0) with ESMTP id w0JBtBHO027589; Fri, 19 Jan 2018 04:55:18 -0700 From: Akhil Goyal To: CC: , , , Akhil Goyal Date: Fri, 19 Jan 2018 17:21:02 +0530 Message-ID: <20180119115102.2247-2-akhil.goyal@nxp.com> X-Mailer: git-send-email 2.9.3 In-Reply-To: <20180119115102.2247-1-akhil.goyal@nxp.com> References: <20180119115102.2247-1-akhil.goyal@nxp.com> X-EOPAttributedMessage: 0 X-Matching-Connectors: 131608418952702578; (91ab9b29-cfa4-454e-5278-08d120cd25b8); () X-Forefront-Antispam-Report: CIP:192.88.168.50; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(7966004)(396003)(346002)(376002)(39380400002)(39860400002)(2980300002)(1109001)(1110001)(339900001)(199004)(189003)(305945005)(2351001)(106466001)(1076002)(81166006)(105606002)(81156014)(50226002)(8676002)(8936002)(356003)(2906002)(68736007)(104016004)(4326008)(508600001)(36756003)(6916009)(2950100002)(6666003)(97736004)(47776003)(5660300001)(76176011)(85426001)(316002)(51416003)(53936002)(50466002)(59450400001)(48376002)(26005)(54906003)(86362001)(77096007)(16586007); DIR:OUT; SFP:1101; SCL:1; SRVR:SN2PR03MB2365; H:tx30smr01.am.freescale.net; FPR:; SPF:Fail; PTR:InfoDomainNonexistent; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11OLC007; 1:A8Tq57IuCmZl5piScQ9ZHsshE5yS6/yb8LHrePBr+6RPawIHoiElk4qVo5pXt1KL3kOql2hliUwO3CBtAkwvlMKHthbLQ14hdP6fi+xIV7zG9tTWdf6uu/YRfFfknVTj MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7cfbdae6-7332-42f4-6a72-08d55f400797 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(5600026)(4604075)(2017052603307); SRVR:SN2PR03MB2365; X-Microsoft-Exchange-Diagnostics: 1; SN2PR03MB2365; 3:nRXtwg9N46fHeuA2oUsI/9mNlODQYmypTGwhk6uOfoqQ4ZKHxCfF+j3hC1OCTzLksiOTUZIBFFz4BQq0TuvSMLv63laTYWM3FRUTf26TWYJhBs2qWVbqQELSEl2ZPrzdGNuN4GlIlr4rK3lBp37kcZ6MARY3zRy2e5z2wVujwYdXREIwIgLGivPloC0pFeYLGfB0R2tCqdrULrFSFZx/kihCex6mS0Ms+c+yvWsNc28YgPZxcEadSf8hts4F7gyrSyNZNFd228MqUUT7XlYEDiwExaEG/XU474m1OMptgHse5BdYCGA9UVYyE1RwUhdoGEAFNW6HKpps2zi8qo+Xh2OQG/FBZmFkJ4FsSXiTrOc=; 25:ISukKNyOlipfShhD14vqm8rY15KddxxKC3nICUCNbuZVXhhSQ7FZSJ8cJQqrpOboimL8AUBOECBxWiTXy+EMrAaGpwMgHrj96ww2jFawov4FL1yUXUJnvb8hxt7fQkyj7oJQwI4x5WzNhE702yDYBODyVnOWG1FuYR5yvz0uZPaUAe2J+3gYb++3R2gA7euztmNGH1LsNqSopKUcMipXOHc2K4AR5lw9ApzNZmxLlG9J4PVYKpDUduXNcDz96n2KBTeiJT1jjkmUHDhKUohnoREXT4Q1qRbjuiSC1H8oktXyi5s6M2HfIdcV46Cwg6YrCqn9TYYKMehpd/KPM63OzQ== X-MS-TrafficTypeDiagnostic: SN2PR03MB2365: X-Microsoft-Exchange-Diagnostics: 1; SN2PR03MB2365; 31:T6pjI+ACMV96OXzSi4Q6fPPixTTTxGz1kOkRkTbcN68+phcnX+HVeQ+0NDiVwBETnLu06Pugkan4MrfnZ8Lhmn0JLxymaV8gsHgTJwfT4ksT0RzI5PmtZ72XdAeGjBlh/WV2A0NJE8DASL6dNsICmkVSNqIljNk7aYiTmQKlLz2JTSbIL+2KQdFRrdpSnweAwCHLR3o5/7Ueb7G04lB4aZzFzEUOE17LMgTYhUEzEik=; 4:dUzWxQs7jl76srcxS0KqWoT0FepK3u8t/WAspNJG7A6nEJGo23jRpJME/ia6kzxjYwK6dO490dRiIZ95njbtYDrj2dOgg3yFipYmTAJsfuhUAsfHNRhuspKrLiEXLRHioutVRKKFGhioCEHUbaG1OS4eNL74dLfidvF73Ig86/1htXTdO7Mm6q/Dh3ZUpik7v2ijCGVdLbkJpjEIOuFPyF2uxNIS7IF1LYB7hLwxO+Z8ThiwR8iXxmMiezaVgzVM/+ZApkLlWGzd+Sk2FKJ2rkFPMU0qjTHw5pLwX1RgQrJgy6JvoOz95RDOwyGcjgZv X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(185117386973197); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6095135)(2401047)(8121501046)(5005006)(3002001)(10201501046)(3231023)(2400078)(944501161)(93006095)(93001095)(6055026)(6096035)(201703131430075)(201703131441075)(201703131448075)(201703131433075)(20161123563025)(20161123559100)(20161123561025)(20161123556025)(20161123565025)(201708071742011); SRVR:SN2PR03MB2365; BCL:0; PCL:0; RULEID:(100000803101)(100110400095)(400006); SRVR:SN2PR03MB2365; X-Forefront-PRVS: 0557CBAD84 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; SN2PR03MB2365; 23:DOF4dvdlYthkyA5VuGr2sPza77vmgoow9Xl3oIM+W?= A9+adY0NjQMwUq0794VCzNM7YV7GQeDmQ966/VcgVyJWxRi9ijK8YUGRyCf7AjDwV3Ynq4eTBcH2DlRSB2+Cb4TuDZORmkhXO06QXfyleIf3rbNHX5s1z7NXbnQ4m/MDo3vImpMoFvoG0o+9mBemPBAvx816Ucd8IAr2vRLdDl9+0SV5HNGBDwc4Vj9bg3v1nvEiTFORWZY/Ln8MrXWVBGWMNegeSln2iZkMseohcEt6QjKGhKi46Prlf9PP19NT7QyDvQ7g6V87KmsHWcvm++x/rxjvOghKd6L6RPo5OEiZTB80ZTqSmYptAJfZUPdETCkexhtBA7YM+xWWjQfoGxYReXWkbTK5Ndfl0GBhyf6QKCp6NSoQzqgr1P5Ls2GG3R8Ln38jMW8t9vy6T2cnwEERBTSN+IqDXoHUXdm9g8tJpFxhk5Ar4bw6jha3O22+33+d2Yq/ZhgmwB9gEZ+pWDQE/fg23vzkegWR4HlGblDgwzeljv6vlnNSLzndtiw/Svx3wVawhwJUd/Fma3JkPGBZUqClXuioN2JJXW2LaIwjMZklpOb4xzEDa6wrxivL+k4hDvUIbcIq/Uys4ggHk4ZuvXnoIMmGOQPm7h3aBqEviJkxYsZNGDEheK6qWoxx1OBfe8TpkJE1qARgVwyh8eR+NG2Hkc2oQhnFChVT82t8lbgf8aK/kjMfb9eVyOzGddcowixH1EcVpZQRaG1Pmm56Nq6oErHWiS/ehrrf281NiIM7Sc6zxHwlgoUApCcrpahEFiPXSF/XCgUMfRb3QWmQmoCBvSPwuaIxhzE4rj3IqbGTGnoBFYaHnp1abGU0+rWczvc1lc3GEHOuS1Puhnvus1C/7gnqGOmbVR2kdJ8z0uzpZIQ0p6JsE5R1bWbiyOFaHom/xhCAbGjp9yQclE5iRhrpwBVogjKbcEUNajIDynbvXz5YtAPuODVkkjntrTL2YuMtWYfiKXQbusmnbuLGSQl5fnAqxPJduBkOemJut/XOZtkM9/7E3LZZ5sSbkC7mOS5FVkNHD+TBMbpXb6rZjJhLM5glWUCtPMllM8ULugLtgiF/ShXhhgTjO+yDMA= X-Microsoft-Exchange-Diagnostics: 1; SN2PR03MB2365; 6:WF8nzG2T0s5uck3a9nzMz//ZelAGunLpjme8Dd8mrYNnyRik+GRjzXpyRcRNby5R98sWalHdJGwtfxo5MtjvDZ/3y6MlcbmYjWdy3Srvc80fs+5CxTXoquVIVoEAF7oOwEThTzyL1Kdk6+6UsLlkasAci01qg2t3rDvDsaKG6Ed8wv0n35EAQ6+WgtsnvlkNGWqOVQhKew6ga2Djju6k1PRDFqORFoOFhv7gAiyp2y+kqlgXc5nIghsqbyTJvGJwyFMWR6ronjrhYsKBFTVxuUGbiK2oijkP7w9TSu94tS1fq0AH0QkzYyrDbyXZoezyalgagacw6rhk4kC7aB6Pl3+ExQJmguMAMRWWpiA1PhA=; 5:07cCX4wVkUSTNDq8VzHabJp0a1VLIASpB6hpviZgra5crlD55wGtSBWvmBmwIG7XOlwgXNnpALJ9/xi8kln5oU34Urck9Z8IzWOAPVgSC8BURqNMaf0NRqMq1BeYkjt8jfXleTWW+Hel1jzv65vi3tJaD9oXyXtwMu2bI0woSNo=; 24:9QUNAleFmdgtoqBvitjN88hCBfMDDd1vvsFXbTQcqCSj4s9KHu4suMweTGGt78xuifTpCRLw2KCD/tY7qffFY2DB5JLATmmxf53aP1N991g=; 7:8FV0rO7qCWzPyXI6JVCbRa/7hPAUifWK4u1n6LSSmQLS9Ad4+1S+l0nb71xdVDq+iTctkhvytSonwwXMlq11m5nSAGA8XwtZE1TJ9IyQj1rhSYqlkQdguSow1uOP46zStJ40TQq0hSQ3L6IxhvGCQgP/9caeoV9Cc3MiCiLFU3fhfN+Hn/nK46qTo20X6RDrX8pFXf0tGQc7fRw9T0HJIIeLj1EQMZnDl7goqLuR9hAwLiFxBtZcxro67rfhAs1H SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jan 2018 13:24:54.4122 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7cfbdae6-7332-42f4-6a72-08d55f400797 X-MS-Exchange-CrossTenant-Id: 5afe0b00-7697-4969-b663-5eab37d5f47e X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=5afe0b00-7697-4969-b663-5eab37d5f47e; Ip=[192.88.168.50]; Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN2PR03MB2365 Subject: [dpdk-dev] [PATCH 2/2] crypto/dpaa_sec: support for scatter gather X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Signed-off-by: Alok Makhariya Signed-off-by: Akhil Goyal Acked-by: Hemant Agrawal --- doc/guides/cryptodevs/features/dpaa_sec.ini | 1 + drivers/crypto/dpaa_sec/dpaa_sec.c | 501 +++++++++++++++++++++++++-- test/test/test_cryptodev.c | 10 + test/test/test_cryptodev_aes_test_vectors.h | 11 +- test/test/test_cryptodev_hash_test_vectors.h | 6 +- 5 files changed, 498 insertions(+), 31 deletions(-) diff --git a/doc/guides/cryptodevs/features/dpaa_sec.ini b/doc/guides/cryptodevs/features/dpaa_sec.ini index deab53a..260fae7 100644 --- a/doc/guides/cryptodevs/features/dpaa_sec.ini +++ b/doc/guides/cryptodevs/features/dpaa_sec.ini @@ -8,6 +8,7 @@ Symmetric crypto = Y Sym operation chaining = Y HW Accelerated = Y Protocol offload = Y +Mbuf scatter gather = Y ; ; Supported crypto algorithms of the 'dpaa_sec' crypto driver. diff --git a/drivers/crypto/dpaa_sec/dpaa_sec.c b/drivers/crypto/dpaa_sec/dpaa_sec.c index a402e61..18681cf 100644 --- a/drivers/crypto/dpaa_sec/dpaa_sec.c +++ b/drivers/crypto/dpaa_sec/dpaa_sec.c @@ -599,6 +599,86 @@ dpaa_sec_deq(struct dpaa_sec_qp *qp, struct rte_crypto_op **ops, int nb_ops) return pkts; } +static inline struct dpaa_sec_job * +build_auth_only_sg(struct rte_crypto_op *op, dpaa_sec_session *ses) +{ + struct rte_crypto_sym_op *sym = op->sym; + struct rte_mbuf *mbuf = sym->m_src; + struct dpaa_sec_job *cf; + struct dpaa_sec_op_ctx *ctx; + struct qm_sg_entry *sg, *out_sg, *in_sg; + phys_addr_t start_addr; + uint8_t *old_digest, extra_segs; + + if (is_decode(ses)) + extra_segs = 3; + else + extra_segs = 2; + + if ((mbuf->nb_segs + extra_segs) > MAX_SG_ENTRIES) { + PMD_TX_LOG(ERR, "Auth: Max sec segs supported is %d\n", + MAX_SG_ENTRIES); + return NULL; + } + ctx = dpaa_sec_alloc_ctx(ses); + if (!ctx) + return NULL; + + cf = &ctx->job; + ctx->op = op; + old_digest = ctx->digest; + + /* output */ + out_sg = &cf->sg[0]; + qm_sg_entry_set64(out_sg, sym->auth.digest.phys_addr); + out_sg->length = ses->digest_length; + cpu_to_hw_sg(out_sg); + + /* input */ + in_sg = &cf->sg[1]; + /* need to extend the input to a compound frame */ + in_sg->extension = 1; + in_sg->final = 1; + in_sg->length = sym->auth.data.length; + qm_sg_entry_set64(in_sg, dpaa_mem_vtop_ctx(ctx, &cf->sg[2])); + + /* 1st seg */ + sg = in_sg + 1; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->auth.data.offset; + sg->offset = sym->auth.data.offset; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + + if (is_decode(ses)) { + /* Digest verification case */ + cpu_to_hw_sg(sg); + sg++; + rte_memcpy(old_digest, sym->auth.digest.data, + ses->digest_length); + start_addr = dpaa_mem_vtop_ctx(ctx, old_digest); + qm_sg_entry_set64(sg, start_addr); + sg->length = ses->digest_length; + in_sg->length += ses->digest_length; + } else { + /* Digest calculation case */ + sg->length -= ses->digest_length; + } + sg->final = 1; + cpu_to_hw_sg(sg); + cpu_to_hw_sg(in_sg); + + return cf; +} + /** * packet looks like: * |<----data_len------->| @@ -669,6 +749,101 @@ build_auth_only(struct rte_crypto_op *op, dpaa_sec_session *ses) } static inline struct dpaa_sec_job * +build_cipher_only_sg(struct rte_crypto_op *op, dpaa_sec_session *ses) +{ + struct rte_crypto_sym_op *sym = op->sym; + struct dpaa_sec_job *cf; + struct dpaa_sec_op_ctx *ctx; + struct qm_sg_entry *sg, *out_sg, *in_sg; + struct rte_mbuf *mbuf; + uint8_t req_segs; + uint8_t *IV_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, + ses->iv.offset); + + if (sym->m_dst) { + mbuf = sym->m_dst; + req_segs = mbuf->nb_segs + sym->m_src->nb_segs + 3; + } else { + mbuf = sym->m_src; + req_segs = mbuf->nb_segs * 2 + 3; + } + + if (req_segs > MAX_SG_ENTRIES) { + PMD_TX_LOG(ERR, "Cipher: Max sec segs supported is %d\n", + MAX_SG_ENTRIES); + return NULL; + } + + ctx = dpaa_sec_alloc_ctx(ses); + if (!ctx) + return NULL; + + cf = &ctx->job; + ctx->op = op; + + /* output */ + out_sg = &cf->sg[0]; + out_sg->extension = 1; + out_sg->length = sym->cipher.data.length; + qm_sg_entry_set64(out_sg, dpaa_mem_vtop_ctx(ctx, &cf->sg[2])); + cpu_to_hw_sg(out_sg); + + /* 1st seg */ + sg = &cf->sg[2]; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->cipher.data.offset; + sg->offset = sym->cipher.data.offset; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + sg->final = 1; + cpu_to_hw_sg(sg); + + /* input */ + mbuf = sym->m_src; + in_sg = &cf->sg[1]; + in_sg->extension = 1; + in_sg->final = 1; + in_sg->length = sym->cipher.data.length + ses->iv.length; + + sg++; + qm_sg_entry_set64(in_sg, dpaa_mem_vtop_ctx(ctx, sg)); + cpu_to_hw_sg(in_sg); + + /* IV */ + qm_sg_entry_set64(sg, dpaa_mem_vtop(IV_ptr)); + sg->length = ses->iv.length; + cpu_to_hw_sg(sg); + + /* 1st seg */ + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->cipher.data.offset; + sg->offset = sym->cipher.data.offset; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + sg->final = 1; + cpu_to_hw_sg(sg); + + return cf; +} + +static inline struct dpaa_sec_job * build_cipher_only(struct rte_crypto_op *op, dpaa_sec_session *ses) { struct rte_crypto_sym_op *sym = op->sym; @@ -724,6 +899,145 @@ build_cipher_only(struct rte_crypto_op *op, dpaa_sec_session *ses) } static inline struct dpaa_sec_job * +build_cipher_auth_gcm_sg(struct rte_crypto_op *op, dpaa_sec_session *ses) +{ + struct rte_crypto_sym_op *sym = op->sym; + struct dpaa_sec_job *cf; + struct dpaa_sec_op_ctx *ctx; + struct qm_sg_entry *sg, *out_sg, *in_sg; + struct rte_mbuf *mbuf; + uint8_t req_segs; + uint8_t *IV_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, + ses->iv.offset); + + if (sym->m_dst) { + mbuf = sym->m_dst; + req_segs = mbuf->nb_segs + sym->m_src->nb_segs + 4; + } else { + mbuf = sym->m_src; + req_segs = mbuf->nb_segs * 2 + 4; + } + + if (ses->auth_only_len) + req_segs++; + + if (req_segs > MAX_SG_ENTRIES) { + PMD_TX_LOG(ERR, "AEAD: Max sec segs supported is %d\n", + MAX_SG_ENTRIES); + return NULL; + } + + ctx = dpaa_sec_alloc_ctx(ses); + if (!ctx) + return NULL; + + cf = &ctx->job; + ctx->op = op; + + rte_prefetch0(cf->sg); + + /* output */ + out_sg = &cf->sg[0]; + out_sg->extension = 1; + if (is_encode(ses)) + out_sg->length = sym->aead.data.length + ses->auth_only_len + + ses->digest_length; + else + out_sg->length = sym->aead.data.length + ses->auth_only_len; + + /* output sg entries */ + sg = &cf->sg[2]; + qm_sg_entry_set64(out_sg, dpaa_mem_vtop_ctx(ctx, sg)); + cpu_to_hw_sg(out_sg); + + /* 1st seg */ + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->aead.data.offset + + ses->auth_only_len; + sg->offset = sym->aead.data.offset - ses->auth_only_len; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + sg->length -= ses->digest_length; + + if (is_encode(ses)) { + cpu_to_hw_sg(sg); + /* set auth output */ + sg++; + qm_sg_entry_set64(sg, sym->aead.digest.phys_addr); + sg->length = ses->digest_length; + } + sg->final = 1; + cpu_to_hw_sg(sg); + + /* input */ + mbuf = sym->m_src; + in_sg = &cf->sg[1]; + in_sg->extension = 1; + in_sg->final = 1; + if (is_encode(ses)) + in_sg->length = ses->iv.length + sym->aead.data.length + + ses->auth_only_len; + else + in_sg->length = ses->iv.length + sym->aead.data.length + + ses->auth_only_len + ses->digest_length; + + /* input sg entries */ + sg++; + qm_sg_entry_set64(in_sg, dpaa_mem_vtop_ctx(ctx, sg)); + cpu_to_hw_sg(in_sg); + + /* 1st seg IV */ + qm_sg_entry_set64(sg, dpaa_mem_vtop(IV_ptr)); + sg->length = ses->iv.length; + cpu_to_hw_sg(sg); + + /* 2nd seg auth only */ + if (ses->auth_only_len) { + sg++; + qm_sg_entry_set64(sg, dpaa_mem_vtop(sym->aead.aad.data)); + sg->length = ses->auth_only_len; + cpu_to_hw_sg(sg); + } + + /* 3rd seg */ + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->aead.data.offset; + sg->offset = sym->aead.data.offset; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + + if (is_decode(ses)) { + cpu_to_hw_sg(sg); + sg++; + memcpy(ctx->digest, sym->aead.digest.data, + ses->digest_length); + qm_sg_entry_set64(sg, dpaa_mem_vtop_ctx(ctx, ctx->digest)); + sg->length = ses->digest_length; + } + sg->final = 1; + cpu_to_hw_sg(sg); + + return cf; +} + +static inline struct dpaa_sec_job * build_cipher_auth_gcm(struct rte_crypto_op *op, dpaa_sec_session *ses) { struct rte_crypto_sym_op *sym = op->sym; @@ -836,6 +1150,132 @@ build_cipher_auth_gcm(struct rte_crypto_op *op, dpaa_sec_session *ses) } static inline struct dpaa_sec_job * +build_cipher_auth_sg(struct rte_crypto_op *op, dpaa_sec_session *ses) +{ + struct rte_crypto_sym_op *sym = op->sym; + struct dpaa_sec_job *cf; + struct dpaa_sec_op_ctx *ctx; + struct qm_sg_entry *sg, *out_sg, *in_sg; + struct rte_mbuf *mbuf; + uint8_t req_segs; + uint8_t *IV_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, + ses->iv.offset); + + if (sym->m_dst) { + mbuf = sym->m_dst; + req_segs = mbuf->nb_segs + sym->m_src->nb_segs + 4; + } else { + mbuf = sym->m_src; + req_segs = mbuf->nb_segs * 2 + 4; + } + + if (req_segs > MAX_SG_ENTRIES) { + PMD_TX_LOG(ERR, "Cipher-Auth: Max sec segs supported is %d\n", + MAX_SG_ENTRIES); + return NULL; + } + + ctx = dpaa_sec_alloc_ctx(ses); + if (!ctx) + return NULL; + + cf = &ctx->job; + ctx->op = op; + + rte_prefetch0(cf->sg); + + /* output */ + out_sg = &cf->sg[0]; + out_sg->extension = 1; + if (is_encode(ses)) + out_sg->length = sym->auth.data.length + ses->digest_length; + else + out_sg->length = sym->auth.data.length; + + /* output sg entries */ + sg = &cf->sg[2]; + qm_sg_entry_set64(out_sg, dpaa_mem_vtop_ctx(ctx, sg)); + cpu_to_hw_sg(out_sg); + + /* 1st seg */ + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->auth.data.offset; + sg->offset = sym->auth.data.offset; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + sg->length -= ses->digest_length; + + if (is_encode(ses)) { + cpu_to_hw_sg(sg); + /* set auth output */ + sg++; + qm_sg_entry_set64(sg, sym->auth.digest.phys_addr); + sg->length = ses->digest_length; + } + sg->final = 1; + cpu_to_hw_sg(sg); + + /* input */ + mbuf = sym->m_src; + in_sg = &cf->sg[1]; + in_sg->extension = 1; + in_sg->final = 1; + if (is_encode(ses)) + in_sg->length = ses->iv.length + sym->auth.data.length; + else + in_sg->length = ses->iv.length + sym->auth.data.length + + ses->digest_length; + + /* input sg entries */ + sg++; + qm_sg_entry_set64(in_sg, dpaa_mem_vtop_ctx(ctx, sg)); + cpu_to_hw_sg(in_sg); + + /* 1st seg IV */ + qm_sg_entry_set64(sg, dpaa_mem_vtop(IV_ptr)); + sg->length = ses->iv.length; + cpu_to_hw_sg(sg); + + /* 2nd seg */ + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len - sym->auth.data.offset; + sg->offset = sym->auth.data.offset; + + /* Successive segs */ + mbuf = mbuf->next; + while (mbuf) { + cpu_to_hw_sg(sg); + sg++; + qm_sg_entry_set64(sg, rte_pktmbuf_mtophys(mbuf)); + sg->length = mbuf->data_len; + mbuf = mbuf->next; + } + + sg->length -= ses->digest_length; + if (is_decode(ses)) { + cpu_to_hw_sg(sg); + sg++; + memcpy(ctx->digest, sym->auth.digest.data, + ses->digest_length); + qm_sg_entry_set64(sg, dpaa_mem_vtop_ctx(ctx, ctx->digest)); + sg->length = ses->digest_length; + } + sg->final = 1; + cpu_to_hw_sg(sg); + + return cf; +} + +static inline struct dpaa_sec_job * build_cipher_auth(struct rte_crypto_op *op, dpaa_sec_session *ses) { struct rte_crypto_sym_op *sym = op->sym; @@ -1020,34 +1460,42 @@ dpaa_sec_enqueue_burst(void *qp, struct rte_crypto_op **ops, } } - /* - * Segmented buffer is not supported. - */ - if (!rte_pktmbuf_is_contiguous(op->sym->m_src)) { - op->status = RTE_CRYPTO_OP_STATUS_ERROR; - frames_to_send = loop; - nb_ops = loop; - goto send_pkts; - } auth_only_len = op->sym->auth.data.length - op->sym->cipher.data.length; - - if (is_auth_only(ses)) { - cf = build_auth_only(op, ses); - } else if (is_cipher_only(ses)) { - cf = build_cipher_only(op, ses); - } else if (is_aead(ses)) { - cf = build_cipher_auth_gcm(op, ses); - auth_only_len = ses->auth_only_len; - } else if (is_auth_cipher(ses)) { - cf = build_cipher_auth(op, ses); - } else if (is_proto_ipsec(ses)) { - cf = build_proto(op, ses); + if (rte_pktmbuf_is_contiguous(op->sym->m_src)) { + if (is_auth_only(ses)) { + cf = build_auth_only(op, ses); + } else if (is_cipher_only(ses)) { + cf = build_cipher_only(op, ses); + } else if (is_aead(ses)) { + cf = build_cipher_auth_gcm(op, ses); + auth_only_len = ses->auth_only_len; + } else if (is_auth_cipher(ses)) { + cf = build_cipher_auth(op, ses); + } else if (is_proto_ipsec(ses)) { + cf = build_proto(op, ses); + } else { + PMD_TX_LOG(ERR, "not supported sec op"); + frames_to_send = loop; + nb_ops = loop; + goto send_pkts; + } } else { - PMD_TX_LOG(ERR, "not supported sec op"); - frames_to_send = loop; - nb_ops = loop; - goto send_pkts; + if (is_auth_only(ses)) { + cf = build_auth_only_sg(op, ses); + } else if (is_cipher_only(ses)) { + cf = build_cipher_only_sg(op, ses); + } else if (is_aead(ses)) { + cf = build_cipher_auth_gcm_sg(op, ses); + auth_only_len = ses->auth_only_len; + } else if (is_auth_cipher(ses)) { + cf = build_cipher_auth_sg(op, ses); + } else { + PMD_TX_LOG(ERR, "not supported sec op"); + frames_to_send = loop; + nb_ops = loop; + goto send_pkts; + } } if (unlikely(!cf)) { frames_to_send = loop; @@ -1834,7 +2282,8 @@ dpaa_sec_dev_init(struct rte_cryptodev *cryptodev) cryptodev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - RTE_CRYPTODEV_FF_SECURITY; + RTE_CRYPTODEV_FF_SECURITY | + RTE_CRYPTODEV_FF_MBUF_SCATTER_GATHER; internals = cryptodev->data->dev_private; internals->max_nb_queue_pairs = RTE_DPAA_MAX_NB_SEC_QPS; diff --git a/test/test/test_cryptodev.c b/test/test/test_cryptodev.c index 6f22896..0f2a045 100644 --- a/test/test/test_cryptodev.c +++ b/test/test/test_cryptodev.c @@ -9365,6 +9365,16 @@ static struct unit_test_suite cryptodev_dpaa_sec_testsuite = { TEST_CASE_ST(ut_setup, ut_teardown, test_AES_GCM_authenticated_decryption_oop_test_case_1), + /** Scatter-Gather */ + TEST_CASE_ST(ut_setup, ut_teardown, + test_AES_GCM_auth_encrypt_SGL_in_place_1500B), + TEST_CASE_ST(ut_setup, ut_teardown, + test_AES_GCM_auth_encrypt_SGL_out_of_place_400B_400B), + TEST_CASE_ST(ut_setup, ut_teardown, + test_AES_GCM_auth_encrypt_SGL_out_of_place_400B_1seg), + TEST_CASE_ST(ut_setup, ut_teardown, + test_AES_GCM_auth_encrypt_SGL_out_of_place_1500B_2000B), + TEST_CASES_END() /**< NULL terminate unit test array */ } }; diff --git a/test/test/test_cryptodev_aes_test_vectors.h b/test/test/test_cryptodev_aes_test_vectors.h index 20f5285..3577ef4 100644 --- a/test/test/test_cryptodev_aes_test_vectors.h +++ b/test/test/test_cryptodev_aes_test_vectors.h @@ -1270,7 +1270,8 @@ static const struct blockcipher_test_case aes_chain_test_cases[] = { BLOCKCIPHER_TEST_TARGET_PMD_OPENSSL | BLOCKCIPHER_TEST_TARGET_PMD_QAT | BLOCKCIPHER_TEST_TARGET_PMD_SCHEDULER | - BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC + BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC }, { .test_descr = "AES-128-CBC HMAC-SHA1 Decryption Digest " @@ -1292,7 +1293,8 @@ static const struct blockcipher_test_case aes_chain_test_cases[] = { .test_data = &aes_test_data_4, .op_mask = BLOCKCIPHER_TEST_OP_AUTH_VERIFY_DEC, .feature_mask = BLOCKCIPHER_TEST_FEATURE_SG, - .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC + .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC }, { .test_descr = "AES-128-CBC HMAC-SHA1 Decryption Digest " @@ -1401,6 +1403,7 @@ static const struct blockcipher_test_case aes_chain_test_cases[] = { BLOCKCIPHER_TEST_TARGET_PMD_OPENSSL | BLOCKCIPHER_TEST_TARGET_PMD_QAT | BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC | BLOCKCIPHER_TEST_TARGET_PMD_SCHEDULER }, { @@ -1555,6 +1558,7 @@ static const struct blockcipher_test_case aes_cipheronly_test_cases[] = { .feature_mask = BLOCKCIPHER_TEST_FEATURE_SG | BLOCKCIPHER_TEST_FEATURE_OOP, .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_OPENSSL | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC | BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC }, { @@ -1573,7 +1577,8 @@ static const struct blockcipher_test_case aes_cipheronly_test_cases[] = { .test_data = &aes_test_data_10, .op_mask = BLOCKCIPHER_TEST_OP_DECRYPT, .feature_mask = BLOCKCIPHER_TEST_FEATURE_SG, - .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC + .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC }, { .test_descr = "AES-256-CBC Encryption", diff --git a/test/test/test_cryptodev_hash_test_vectors.h b/test/test/test_cryptodev_hash_test_vectors.h index bd793f4..93dacb7 100644 --- a/test/test/test_cryptodev_hash_test_vectors.h +++ b/test/test/test_cryptodev_hash_test_vectors.h @@ -382,7 +382,8 @@ static const struct blockcipher_test_case hash_test_cases[] = { .test_data = &hmac_sha1_test_vector, .op_mask = BLOCKCIPHER_TEST_OP_AUTH_GEN, .feature_mask = BLOCKCIPHER_TEST_FEATURE_SG, - .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC + .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC }, { .test_descr = "HMAC-SHA1 Digest Verify", @@ -400,7 +401,8 @@ static const struct blockcipher_test_case hash_test_cases[] = { .test_data = &hmac_sha1_test_vector, .op_mask = BLOCKCIPHER_TEST_OP_AUTH_VERIFY, .feature_mask = BLOCKCIPHER_TEST_FEATURE_SG, - .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC + .pmd_mask = BLOCKCIPHER_TEST_TARGET_PMD_DPAA2_SEC | + BLOCKCIPHER_TEST_TARGET_PMD_DPAA_SEC }, { .test_descr = "SHA224 Digest",