From patchwork Fri Sep 27 06:26:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nagadheeraj Rottela X-Patchwork-Id: 59962 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BF5251B203; Fri, 27 Sep 2019 08:26:49 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id 2EABE5B32 for ; Fri, 27 Sep 2019 08:26:46 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x8R5nnFI002375; Thu, 26 Sep 2019 23:26:45 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pfpt0818; bh=qSRPtpQY7FVdcNYDLQ/4Bz2bAOU9c0YY0JjB1+CdGEc=; b=X7i2aRQlYcjAGfofcpVsbYftUxnxi7kOc2zSUrTdjS6NbNbJkHs6GM94wyopeOxv0M8k nSoXlZAyW1rQmEPfLZ8c5ceCDg2rrM7ziQLVsfTg9GzGF6AyDRMXGZIb9uUwdGNlpm/S 9jjx2XKd7mQftXikaIejfkFmB7zdFM/vLn86C9YN7l9BnWGP6adfBWYnIMGj4AsJNyxz wAHedgSNvU+gorjRXaLTThvXyibOUAT4sHtA1lB9t5AHulnpMSPhntrrqgHP4hJ69TOo 4+hDlOlCRzacrXXUWyYOTVnWChVFf61wSa7Un3TIDUJQH21tJt1PdA1V1A0i23ro6Lpp HA== Received: from sc-exch04.marvell.com ([199.233.58.184]) by mx0b-0016f401.pphosted.com with ESMTP id 2v8vf2469d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 26 Sep 2019 23:26:45 -0700 Received: from SC-EXCH02.marvell.com (10.93.176.82) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Thu, 26 Sep 2019 23:26:43 -0700 Received: from NAM05-DM3-obe.outbound.protection.outlook.com (104.47.49.53) by SC-EXCH02.marvell.com (10.93.176.82) with Microsoft SMTP Server (TLS) id 15.0.1367.3 via Frontend Transport; Thu, 26 Sep 2019 23:26:43 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UEA8hucxrCHDY1sWk9lS3W9xiIUpL5soxT8e+FgPcHTY+iU6UyGS8Ck0Js+Jw8Tjj4F9Hvvv/p+NKD1Px6Ae81NXeN6EeqNaY7Bzxm94f8hHjBgrqZq9Lb1OQTjhZs42+8ChePocYrtQD13fjzGSfqJQYc+T8ARjNl6KtwIKnG9xhjH+rqAt1y0ACVoVIHSf6DtyEVu81+wG8u/tIozEeQKcfekPWvy6cMOttjoFR68oMQbAOnoqdqcV4jqLUfbCRGQSj63XmONfnCSnl49A/xInhgZbaSjnWM4yG8ff2CSycKXkpJ8GxWWLuQyxIyPWB7qzwLxF9anOQ+l1tfGTJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qSRPtpQY7FVdcNYDLQ/4Bz2bAOU9c0YY0JjB1+CdGEc=; b=hyeNfxosRorVhKIP4gUJvALWEETnra+fFolohA2fkBEyfDGZGl3TTXm/noJjpe8h4gUQ8WWpD6SZeLbKKnZTCCVw+MMOIXsk4+bpgCHAFgT34543+tBR9M1bgrblJvUOD8FjMN/qqZVohSEHzq1xQ8r5K3Ed94ayek4m18Fddn6PiR6587Q1WhdxsDMNEwbcR2pWtYoozONl7Phq/vq5qsZCU7dWFf/s3+Bk3SQ8u7OUNIoKZJZ1IfwO3gO4qqaD3LOV2+dsOexn3fbz8L0+uspsAsIQdTinGL8JDru0Zq5Eynd6IBbrNXaIyjgX6h7ZMyCZgtRhxGO3FxoLz1lyFg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector2-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qSRPtpQY7FVdcNYDLQ/4Bz2bAOU9c0YY0JjB1+CdGEc=; b=udFk2R/oyutXPpoZ/0DEqOgLHs7rhzPwA46VvCYSFMvkGdZuncnHCW1TcudHiI+F1CCDTg/zlPBZeiMiIrV36Ux9AJRMZ8ODSsBqyhAHD1qcxvf+L2tYNvnHeyCKNjTKWScSg9gpu1A4OtFZD5nMe1X7rcJB1aoD+PxUiDIV1uo= Received: from BYAPR18MB2792.namprd18.prod.outlook.com (20.179.56.216) by BYAPR18MB2984.namprd18.prod.outlook.com (20.179.59.97) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2284.19; Fri, 27 Sep 2019 06:26:41 +0000 Received: from BYAPR18MB2792.namprd18.prod.outlook.com ([fe80::7112:68c6:eb44:e9aa]) by BYAPR18MB2792.namprd18.prod.outlook.com ([fe80::7112:68c6:eb44:e9aa%3]) with mapi id 15.20.2284.028; Fri, 27 Sep 2019 06:26:41 +0000 From: Nagadheeraj Rottela To: "akhil.goyal@nxp.com" , "pablo.de.lara.guarch@intel.com" CC: Srikanth Jampala , "dev@dpdk.org" , Nagadheeraj Rottela Thread-Topic: [PATCH v6 7/8] crypto/nitrox: add cipher auth crypto chain processing Thread-Index: AQHVdPyGCWsT0ZiGp0a0RKeRGTqA0A== Date: Fri, 27 Sep 2019 06:26:41 +0000 Message-ID: <20190927062533.19005-8-rnagadheeraj@marvell.com> References: <20190716091016.4788-1-rnagadheeraj@marvell.com> <20190927062533.19005-1-rnagadheeraj@marvell.com> In-Reply-To: <20190927062533.19005-1-rnagadheeraj@marvell.com> Accept-Language: en-IN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: PN1PR0101CA0025.INDPRD01.PROD.OUTLOOK.COM (2603:1096:c00:c::11) To BYAPR18MB2792.namprd18.prod.outlook.com (2603:10b6:a03:105::24) x-ms-exchange-messagesentrepresentingtype: 1 x-mailer: git-send-email 2.13.6 x-originating-ip: [115.113.156.2] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: ab58a94b-7d5b-4a2e-93c8-08d74313a8ce x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(5600167)(711020)(4605104)(1401327)(4534185)(7168020)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7193020); SRVR:BYAPR18MB2984; x-ms-traffictypediagnostic: BYAPR18MB2984: x-ms-exchange-purlcount: 1 x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6108; x-forefront-prvs: 0173C6D4D5 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(189003)(199004)(25214002)(6436002)(26005)(30864003)(2906002)(107886003)(14444005)(102836004)(3846002)(1076003)(66066001)(14454004)(6486002)(55236004)(36756003)(305945005)(256004)(478600001)(186003)(6506007)(66946007)(110136005)(76176011)(8936002)(8676002)(54906003)(6116002)(71190400001)(5660300002)(71200400001)(52116002)(386003)(66556008)(66446008)(446003)(11346002)(2501003)(86362001)(81166006)(64756008)(966005)(316002)(4326008)(66476007)(25786009)(6306002)(50226002)(99286004)(2616005)(476003)(7736002)(486006)(81156014)(6512007); DIR:OUT; SFP:1101; SCL:1; SRVR:BYAPR18MB2984; H:BYAPR18MB2792.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: marvell.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: 3g5irCmiLdfUvbfF3xEf3aRgZ4fRggJLqJN+blyw5pSJ+GJxQwCl7tooM7hWZHw3hYce7cH0bA7n0hnBs1UapseCHa606+WJ4dZiy7rcrUGsivNIbKjQ85jA4LWjcU63F0+iFqwLTSsBMDSLBxmtUTZ7Cs3Oo7gMOm5ir+LXgyth51RQZT//ZfPTOV7zOjp900m1dEK8JeuNQ9IvewLUx4rdW7g7aVYgDZCf+G6zChiKkZHOVqJkPJNVhWe0dOqSUDXWdDqg/BIsO1fmStOz0MWphEi2kBOcmMgkdaC8dTP0uNUnXbk4xUt2hUQ/91nuKz8mSOrlHWs76t0U0y5CUqURiO4/pYWxvUfUbKTUcbef9Qob5HQmjCcrwo+QEEdSQIUClM1DfKyLeEezGSFzcTnMYp9JpOzzNYxT/c/s3sPXDdHmo67kIFMWZjfQRL4AH8gt96ShSqko4xv3RS58sA== MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: ab58a94b-7d5b-4a2e-93c8-08d74313a8ce X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Sep 2019 06:26:41.4333 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: vKHMI/b1CqhXjlUesdqgVwW0UCOvlUdjf0nBVwgYA38IImYKH0vKbI2mPTDxvXl+CklqEpzhqVhmplvx0JqyWOVZ8Pp08ujHKwR0rn791aE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR18MB2984 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.95,1.0.8 definitions=2019-09-27_04:2019-09-25,2019-09-27 signatures=0 Subject: [dpdk-dev] [PATCH v6 7/8] crypto/nitrox: add cipher auth crypto chain processing X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add cipher auth crypto chain processing functionality in symmetric request manager. Updated the release notes. Signed-off-by: Nagadheeraj Rottela --- doc/guides/cryptodevs/features/nitrox.ini | 40 +++ doc/guides/cryptodevs/nitrox.rst | 21 ++ doc/guides/rel_notes/release_19_11.rst | 5 + drivers/crypto/nitrox/nitrox_sym.c | 7 +- drivers/crypto/nitrox/nitrox_sym_reqmgr.c | 412 +++++++++++++++++++++++++++++- 5 files changed, 482 insertions(+), 3 deletions(-) create mode 100644 doc/guides/cryptodevs/features/nitrox.ini diff --git a/doc/guides/cryptodevs/features/nitrox.ini b/doc/guides/cryptodevs/features/nitrox.ini new file mode 100644 index 000000000..ddc3c05f4 --- /dev/null +++ b/doc/guides/cryptodevs/features/nitrox.ini @@ -0,0 +1,40 @@ +; +; Supported features of the 'nitrox' crypto driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Symmetric crypto = Y +Sym operation chaining = Y +HW Accelerated = Y +In Place SGL = Y +OOP SGL In SGL Out = Y +OOP SGL In LB Out = Y +OOP LB In SGL Out = Y +OOP LB In LB Out = Y + +; +; Supported crypto algorithms of the 'nitrox' crypto driver. +; +[Cipher] +AES CBC (128) = Y +AES CBC (192) = Y +AES CBC (256) = Y + +; +; Supported authentication algorithms of the 'nitrox' crypto driver. +; +[Auth] +SHA1 HMAC = Y +SHA224 HMAC = Y +SHA256 HMAC = Y + +; +; Supported AEAD algorithms of the 'nitrox' crypto driver. +; +[AEAD] + +; +; Supported Asymmetric algorithms of the 'nitrox' crypto driver. +; +[Asymmetric] diff --git a/doc/guides/cryptodevs/nitrox.rst b/doc/guides/cryptodevs/nitrox.rst index cb7f92755..f8a527c05 100644 --- a/doc/guides/cryptodevs/nitrox.rst +++ b/doc/guides/cryptodevs/nitrox.rst @@ -10,6 +10,27 @@ information about the NITROX V security processor can be obtained here: * https://www.marvell.com/security-solutions/nitrox-security-processors/nitrox-v/ +Features +-------- + +Nitrox crypto PMD has support for: + +Cipher algorithms: + +* ``RTE_CRYPTO_CIPHER_AES_CBC`` + +Hash algorithms: + +* ``RTE_CRYPTO_AUTH_SHA1_HMAC`` +* ``RTE_CRYPTO_AUTH_SHA224_HMAC`` +* ``RTE_CRYPTO_AUTH_SHA256_HMAC`` + +Limitations +----------- + +* AES_CBC Cipher Only combination is not supported. +* Session-less APIs are not supported. + Installation ------------ diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst index 27cfbd9e3..875964f3b 100644 --- a/doc/guides/rel_notes/release_19_11.rst +++ b/doc/guides/rel_notes/release_19_11.rst @@ -56,6 +56,11 @@ New Features Also, make sure to start the actual text at the margin. ========================================================= +* **Added Marvell NITROX symmetric crypto PMD.** + + Added a symmetric crypto PMD for Marvell NITROX V security processor. + See the :doc:`../cryptodevs/nitrox` guide for more details on this new driver. + Removed Items ------------- diff --git a/drivers/crypto/nitrox/nitrox_sym.c b/drivers/crypto/nitrox/nitrox_sym.c index 0ca15f847..e25baec2d 100644 --- a/drivers/crypto/nitrox/nitrox_sym.c +++ b/drivers/crypto/nitrox/nitrox_sym.c @@ -706,7 +706,12 @@ nitrox_sym_pmd_create(struct nitrox_device *ndev) cdev->dequeue_burst = nitrox_sym_dev_deq_burst; cdev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_HW_ACCELERATED | - RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING; + RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | + RTE_CRYPTODEV_FF_IN_PLACE_SGL | + RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT | + RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | + RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | + RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; ndev->sym_dev = cdev->data->dev_private; ndev->sym_dev->cdev = cdev; diff --git a/drivers/crypto/nitrox/nitrox_sym_reqmgr.c b/drivers/crypto/nitrox/nitrox_sym_reqmgr.c index a37b754f2..d96e72718 100644 --- a/drivers/crypto/nitrox/nitrox_sym_reqmgr.c +++ b/drivers/crypto/nitrox/nitrox_sym_reqmgr.c @@ -10,9 +10,24 @@ #include "nitrox_sym_reqmgr.h" #include "nitrox_logs.h" +#define MAX_SGBUF_CNT 16 +#define MAX_SGCOMP_CNT 5 +/* SLC_STORE_INFO */ +#define MIN_UDD_LEN 16 +/* PKT_IN_HDR + SLC_STORE_INFO */ +#define FDATA_SIZE 32 +/* Base destination port for the solicited requests */ +#define SOLICIT_BASE_DPORT 256 #define PENDING_SIG 0xFFFFFFFFFFFFFFFFUL #define CMD_TIMEOUT 2 +struct gphdr { + uint16_t param0; + uint16_t param1; + uint16_t param2; + uint16_t param3; +}; + union pkt_instr_hdr { uint64_t value; struct { @@ -105,12 +120,46 @@ struct resp_hdr { uint64_t completion; }; +struct nitrox_sglist { + uint16_t len; + uint16_t raz0; + uint32_t raz1; + rte_iova_t iova; + void *virt; +}; + +struct nitrox_sgcomp { + uint16_t len[4]; + uint64_t iova[4]; +}; + +struct nitrox_sgtable { + uint8_t map_bufs_cnt; + uint8_t nr_sgcomp; + uint16_t total_bytes; + + struct nitrox_sglist sglist[MAX_SGBUF_CNT]; + struct nitrox_sgcomp sgcomp[MAX_SGCOMP_CNT]; +}; + +struct iv { + uint8_t *virt; + rte_iova_t iova; + uint16_t len; +}; + struct nitrox_softreq { struct nitrox_crypto_ctx *ctx; struct rte_crypto_op *op; + struct gphdr gph; struct nps_pkt_instr instr; struct resp_hdr resp; + struct nitrox_sgtable in; + struct nitrox_sgtable out; + struct iv iv; uint64_t timeout; + rte_iova_t dptr; + rte_iova_t rptr; rte_iova_t iova; }; @@ -121,10 +170,369 @@ softreq_init(struct nitrox_softreq *sr, rte_iova_t iova) sr->iova = iova; } +/* + * 64-Byte Instruction Format + * + * ---------------------- + * | DPTR0 | 8 bytes + * ---------------------- + * | PKT_IN_INSTR_HDR | 8 bytes + * ---------------------- + * | PKT_IN_HDR | 16 bytes + * ---------------------- + * | SLC_INFO | 16 bytes + * ---------------------- + * | Front data | 16 bytes + * ---------------------- + */ +static void +create_se_instr(struct nitrox_softreq *sr, uint8_t qno) +{ + struct nitrox_crypto_ctx *ctx = sr->ctx; + rte_iova_t ctx_handle; + + /* fill the packet instruction */ + /* word 0 */ + sr->instr.dptr0 = rte_cpu_to_be_64(sr->dptr); + + /* word 1 */ + sr->instr.ih.value = 0; + sr->instr.ih.s.g = 1; + sr->instr.ih.s.gsz = sr->in.map_bufs_cnt; + sr->instr.ih.s.ssz = sr->out.map_bufs_cnt; + sr->instr.ih.s.fsz = FDATA_SIZE + sizeof(struct gphdr); + sr->instr.ih.s.tlen = sr->instr.ih.s.fsz + sr->in.total_bytes; + sr->instr.ih.value = rte_cpu_to_be_64(sr->instr.ih.value); + + /* word 2 */ + sr->instr.irh.value[0] = 0; + sr->instr.irh.s.uddl = MIN_UDD_LEN; + /* context length in 64-bit words */ + sr->instr.irh.s.ctxl = RTE_ALIGN_MUL_CEIL(sizeof(ctx->fctx), 8) / 8; + /* offset from solicit base port 256 */ + sr->instr.irh.s.destport = SOLICIT_BASE_DPORT + qno; + /* Invalid context cache */ + sr->instr.irh.s.ctxc = 0x3; + sr->instr.irh.s.arg = ctx->req_op; + sr->instr.irh.s.opcode = ctx->opcode; + sr->instr.irh.value[0] = rte_cpu_to_be_64(sr->instr.irh.value[0]); + + /* word 3 */ + ctx_handle = ctx->iova + offsetof(struct nitrox_crypto_ctx, fctx); + sr->instr.irh.s.ctxp = rte_cpu_to_be_64(ctx_handle); + + /* word 4 */ + sr->instr.slc.value[0] = 0; + sr->instr.slc.s.ssz = sr->out.map_bufs_cnt; + sr->instr.slc.value[0] = rte_cpu_to_be_64(sr->instr.slc.value[0]); + + /* word 5 */ + sr->instr.slc.s.rptr = rte_cpu_to_be_64(sr->rptr); + /* + * No conversion for front data, + * It goes into payload + * put GP Header in front data + */ + memcpy(&sr->instr.fdata[0], &sr->gph, sizeof(sr->instr.fdata[0])); + sr->instr.fdata[1] = 0; + /* flush the soft_req changes before posting the cmd */ + rte_wmb(); +} + +static void +softreq_copy_iv(struct nitrox_softreq *sr) +{ + sr->iv.virt = rte_crypto_op_ctod_offset(sr->op, uint8_t *, + sr->ctx->iv.offset); + sr->iv.iova = rte_crypto_op_ctophys_offset(sr->op, sr->ctx->iv.offset); + sr->iv.len = sr->ctx->iv.length; +} + +static int +extract_cipher_auth_digest(struct nitrox_softreq *sr, + struct nitrox_sglist *digest) +{ + struct rte_crypto_op *op = sr->op; + struct rte_mbuf *mdst = op->sym->m_dst ? op->sym->m_dst : + op->sym->m_src; + + if (sr->ctx->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY && + unlikely(!op->sym->auth.digest.data)) + return -EINVAL; + + digest->len = sr->ctx->digest_length; + if (op->sym->auth.digest.data) { + digest->iova = op->sym->auth.digest.phys_addr; + digest->virt = op->sym->auth.digest.data; + return 0; + } + + if (unlikely(rte_pktmbuf_data_len(mdst) < op->sym->auth.data.offset + + op->sym->auth.data.length + digest->len)) + return -EINVAL; + + digest->iova = rte_pktmbuf_mtophys_offset(mdst, + op->sym->auth.data.offset + + op->sym->auth.data.length); + digest->virt = rte_pktmbuf_mtod_offset(mdst, uint8_t *, + op->sym->auth.data.offset + + op->sym->auth.data.length); + return 0; +} + +static void +fill_sglist(struct nitrox_sgtable *sgtbl, uint16_t len, rte_iova_t iova, + void *virt) +{ + struct nitrox_sglist *sglist = sgtbl->sglist; + uint8_t cnt = sgtbl->map_bufs_cnt; + + if (unlikely(!len)) + return; + + sglist[cnt].len = len; + sglist[cnt].iova = iova; + sglist[cnt].virt = virt; + sgtbl->total_bytes += len; + cnt++; + sgtbl->map_bufs_cnt = cnt; +} + +static int +create_sglist_from_mbuf(struct nitrox_sgtable *sgtbl, struct rte_mbuf *mbuf, + uint32_t off, int datalen) +{ + struct nitrox_sglist *sglist = sgtbl->sglist; + uint8_t cnt = sgtbl->map_bufs_cnt; + struct rte_mbuf *m; + int mlen; + + if (unlikely(datalen <= 0)) + return 0; + + for (m = mbuf; m && off > rte_pktmbuf_data_len(m); m = m->next) + off -= rte_pktmbuf_data_len(m); + + if (unlikely(!m)) + return -EIO; + + mlen = rte_pktmbuf_data_len(m) - off; + if (datalen <= mlen) + mlen = datalen; + sglist[cnt].len = mlen; + sglist[cnt].iova = rte_pktmbuf_mtophys_offset(m, off); + sglist[cnt].virt = rte_pktmbuf_mtod_offset(m, uint8_t *, off); + sgtbl->total_bytes += mlen; + cnt++; + datalen -= mlen; + for (m = m->next; m && datalen; m = m->next) { + mlen = rte_pktmbuf_data_len(m) < datalen ? + rte_pktmbuf_data_len(m) : datalen; + sglist[cnt].len = mlen; + sglist[cnt].iova = rte_pktmbuf_mtophys(m); + sglist[cnt].virt = rte_pktmbuf_mtod(m, uint8_t *); + sgtbl->total_bytes += mlen; + cnt++; + datalen -= mlen; + } + + RTE_VERIFY(cnt <= MAX_SGBUF_CNT); + sgtbl->map_bufs_cnt = cnt; + return 0; +} + +static int +create_cipher_auth_sglist(struct nitrox_softreq *sr, + struct nitrox_sgtable *sgtbl, struct rte_mbuf *mbuf) +{ + struct rte_crypto_op *op = sr->op; + int auth_only_len; + int err; + + fill_sglist(sgtbl, sr->iv.len, sr->iv.iova, sr->iv.virt); + auth_only_len = op->sym->auth.data.length - op->sym->cipher.data.length; + if (unlikely(auth_only_len < 0)) + return -EINVAL; + + err = create_sglist_from_mbuf(sgtbl, mbuf, op->sym->auth.data.offset, + auth_only_len); + if (unlikely(err)) + return err; + + err = create_sglist_from_mbuf(sgtbl, mbuf, op->sym->cipher.data.offset, + op->sym->cipher.data.length); + if (unlikely(err)) + return err; + + return 0; +} + +static void +create_sgcomp(struct nitrox_sgtable *sgtbl) +{ + int i, j, nr_sgcomp; + struct nitrox_sgcomp *sgcomp = sgtbl->sgcomp; + struct nitrox_sglist *sglist = sgtbl->sglist; + + nr_sgcomp = RTE_ALIGN_MUL_CEIL(sgtbl->map_bufs_cnt, 4) / 4; + sgtbl->nr_sgcomp = nr_sgcomp; + for (i = 0; i < nr_sgcomp; i++, sgcomp++) { + for (j = 0; j < 4; j++, sglist++) { + sgcomp->len[j] = rte_cpu_to_be_16(sglist->len); + sgcomp->iova[j] = rte_cpu_to_be_64(sglist->iova); + } + } +} + +static int +create_cipher_auth_inbuf(struct nitrox_softreq *sr, + struct nitrox_sglist *digest) +{ + int err; + struct nitrox_crypto_ctx *ctx = sr->ctx; + + err = create_cipher_auth_sglist(sr, &sr->in, sr->op->sym->m_src); + if (unlikely(err)) + return err; + + if (ctx->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) + fill_sglist(&sr->in, digest->len, digest->iova, digest->virt); + + create_sgcomp(&sr->in); + sr->dptr = sr->iova + offsetof(struct nitrox_softreq, in.sgcomp); + return 0; +} + +static int +create_cipher_auth_oop_outbuf(struct nitrox_softreq *sr, + struct nitrox_sglist *digest) +{ + int err; + struct nitrox_crypto_ctx *ctx = sr->ctx; + + err = create_cipher_auth_sglist(sr, &sr->out, sr->op->sym->m_dst); + if (unlikely(err)) + return err; + + if (ctx->auth_op == RTE_CRYPTO_AUTH_OP_GENERATE) + fill_sglist(&sr->out, digest->len, digest->iova, digest->virt); + + return 0; +} + +static void +create_cipher_auth_inplace_outbuf(struct nitrox_softreq *sr, + struct nitrox_sglist *digest) +{ + int i, cnt; + struct nitrox_crypto_ctx *ctx = sr->ctx; + + cnt = sr->out.map_bufs_cnt; + for (i = 0; i < sr->in.map_bufs_cnt; i++, cnt++) { + sr->out.sglist[cnt].len = sr->in.sglist[i].len; + sr->out.sglist[cnt].iova = sr->in.sglist[i].iova; + sr->out.sglist[cnt].virt = sr->in.sglist[i].virt; + } + + sr->out.map_bufs_cnt = cnt; + if (ctx->auth_op == RTE_CRYPTO_AUTH_OP_GENERATE) { + fill_sglist(&sr->out, digest->len, digest->iova, + digest->virt); + } else if (ctx->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) { + sr->out.map_bufs_cnt--; + } +} + +static int +create_cipher_auth_outbuf(struct nitrox_softreq *sr, + struct nitrox_sglist *digest) +{ + struct rte_crypto_op *op = sr->op; + int cnt = 0; + + sr->resp.orh = PENDING_SIG; + sr->out.sglist[cnt].len = sizeof(sr->resp.orh); + sr->out.sglist[cnt].iova = sr->iova + offsetof(struct nitrox_softreq, + resp.orh); + sr->out.sglist[cnt].virt = &sr->resp.orh; + cnt++; + sr->out.map_bufs_cnt = cnt; + if (op->sym->m_dst) { + int err; + + err = create_cipher_auth_oop_outbuf(sr, digest); + if (unlikely(err)) + return err; + } else { + create_cipher_auth_inplace_outbuf(sr, digest); + } + + cnt = sr->out.map_bufs_cnt; + sr->resp.completion = PENDING_SIG; + sr->out.sglist[cnt].len = sizeof(sr->resp.completion); + sr->out.sglist[cnt].iova = sr->iova + offsetof(struct nitrox_softreq, + resp.completion); + sr->out.sglist[cnt].virt = &sr->resp.completion; + cnt++; + RTE_VERIFY(cnt <= MAX_SGBUF_CNT); + sr->out.map_bufs_cnt = cnt; + + create_sgcomp(&sr->out); + sr->rptr = sr->iova + offsetof(struct nitrox_softreq, out.sgcomp); + return 0; +} + +static void +create_aead_gph(uint32_t cryptlen, uint16_t ivlen, uint32_t authlen, + struct gphdr *gph) +{ + int auth_only_len; + union { + struct { +#if RTE_BYTE_ORDER == RTE_BIG_ENDIAN + uint16_t iv_offset : 8; + uint16_t auth_offset : 8; +#else + uint16_t auth_offset : 8; + uint16_t iv_offset : 8; +#endif + }; + uint16_t value; + } param3; + + gph->param0 = rte_cpu_to_be_16(cryptlen); + gph->param1 = rte_cpu_to_be_16(authlen); + + auth_only_len = authlen - cryptlen; + gph->param2 = rte_cpu_to_be_16(ivlen + auth_only_len); + + param3.iv_offset = 0; + param3.auth_offset = ivlen; + gph->param3 = rte_cpu_to_be_16(param3.value); +} + static int process_cipher_auth_data(struct nitrox_softreq *sr) { - RTE_SET_USED(sr); + struct rte_crypto_op *op = sr->op; + int err; + struct nitrox_sglist digest; + + softreq_copy_iv(sr); + err = extract_cipher_auth_digest(sr, &digest); + if (unlikely(err)) + return err; + + err = create_cipher_auth_inbuf(sr, &digest); + if (unlikely(err)) + return err; + + err = create_cipher_auth_outbuf(sr, &digest); + if (unlikely(err)) + return err; + + create_aead_gph(op->sym->cipher.data.length, sr->iv.len, + op->sym->auth.data.length, &sr->gph); return 0; } @@ -152,11 +560,11 @@ nitrox_process_se_req(uint16_t qno, struct rte_crypto_op *op, struct nitrox_crypto_ctx *ctx, struct nitrox_softreq *sr) { - RTE_SET_USED(qno); softreq_init(sr, sr->iova); sr->ctx = ctx; sr->op = op; process_softreq(sr); + create_se_instr(sr, qno); sr->timeout = rte_get_timer_cycles() + CMD_TIMEOUT * rte_get_timer_hz(); return 0; }