From patchwork Fri Mar 1 18:42:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anoob Joseph X-Patchwork-Id: 50738 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6FDA94CE4; Fri, 1 Mar 2019 19:42:40 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by dpdk.org (Postfix) with ESMTP id 413D94C8D for ; Fri, 1 Mar 2019 19:42:36 +0100 (CET) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x21IfPHK005405; Fri, 1 Mar 2019 10:42:35 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pfpt0818; bh=+ha8X+g9l70rljYgsFRo15u2DRlfS8fx8+WJuh+FeT4=; b=hLpzFXOAColfv8VhWpKcb9AhLcWtccX4AttjBPJzHJJs8ME09lvTNS37W8k76JPfbd75 5oEK7XStFHIxYTOa1dfxB8YMTYKcba1Xu1EL2AydpZgTFNTZCQEENocyPgu+Zz37dMCP yKbj5gmBdH5jVZjoo7VETqQMeGQW8mzNziyzXvcMtmhvceBgwVqnFrElOTIgFT47JneB wiDs5KU75SDjMmcBJFvPaMmfLU1P35n65EJYnybihG75JFSP9RPF+gPKJWLs4XfJqRBq 9NV8s+V2Cu5u/DpPt3aczhkQ9FLAig4CiiJBjwLqSZE3TKJdz1h2b7nrUgpbGLxSXR6F AQ== Received: from sc-exch01.marvell.com ([199.233.58.181]) by mx0a-0016f401.pphosted.com with ESMTP id 2qxprq2vwb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Fri, 01 Mar 2019 10:42:35 -0800 Received: from SC-EXCH01.marvell.com (10.93.176.81) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Fri, 1 Mar 2019 10:42:34 -0800 Received: from NAM05-CO1-obe.outbound.protection.outlook.com (104.47.48.57) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server (TLS) id 15.0.1367.3 via Frontend Transport; Fri, 1 Mar 2019 10:42:33 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector1-marvell-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+ha8X+g9l70rljYgsFRo15u2DRlfS8fx8+WJuh+FeT4=; b=OQ+j8D5cpKpg+nKZQynfbZiJiKn9Gs6BKvH69S8Yb9CBIZTYI6ZsT3ptSv8APdCHWCFS8/IeOnsmFOQyX/EqgzKRBF/T7skMmzu/8i2zX6bSaFqbTW8svwR5yEUEGOzFG6yEu+IwskV5KVL+dYQmD/dt2eZRFsoeUhRzNAs6dEc= Received: from MN2PR18MB2877.namprd18.prod.outlook.com (20.179.20.218) by MN2PR18MB2605.namprd18.prod.outlook.com (20.179.83.161) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1665.18; Fri, 1 Mar 2019 18:42:32 +0000 Received: from MN2PR18MB2877.namprd18.prod.outlook.com ([fe80::1905:8cf6:f4a1:5d9a]) by MN2PR18MB2877.namprd18.prod.outlook.com ([fe80::1905:8cf6:f4a1:5d9a%4]) with mapi id 15.20.1665.017; Fri, 1 Mar 2019 18:42:32 +0000 From: Anoob Joseph To: Akhil Goyal , Pablo de Lara CC: Anoob Joseph , Jerin Jacob Kollanukkaran , Narayana Prasad Raju Athreya , Suheil Chandran , Ankur Dwivedi , "dev@dpdk.org" , Archana Muniganti Thread-Topic: [PATCH 3/3] crypto/octeontx: move device specific code to driver Thread-Index: AQHU0F6HVIGaq66djUmBE1WyX0TrHA== Date: Fri, 1 Mar 2019 18:42:32 +0000 Message-ID: <1551465683-26721-3-git-send-email-anoobj@marvell.com> References: <1551465683-26721-1-git-send-email-anoobj@marvell.com> In-Reply-To: <1551465683-26721-1-git-send-email-anoobj@marvell.com> Accept-Language: en-IN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BMXPR01CA0008.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:d::18) To MN2PR18MB2877.namprd18.prod.outlook.com (2603:10b6:208:3b::26) x-ms-exchange-messagesentrepresentingtype: 1 x-mailer: git-send-email 2.7.4 x-originating-ip: [115.113.156.2] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b3281acc-9786-4942-e141-08d69e75aa09 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(7168020)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(2017052603328)(7153060)(7193020); SRVR:MN2PR18MB2605; x-ms-traffictypediagnostic: MN2PR18MB2605: x-microsoft-exchange-diagnostics: =?iso-8859-1?Q?1; MN2PR18MB2605; 23:dLKtRwZy0K640gN+zYu0cC3Mt3HZ9sz9YuNDlo9?= =?iso-8859-1?q?KxQwZsOlL9+z6lCmX3D/sLTH5?= =?iso-8859-1?q?FGM8Rpkyg5fBLyTH9cbhz5uGcPlgPiMbkpNU7BnBTBpfH4DDsjP?= =?iso-8859-1?q?6LXbLo0nkOg/pE3c9OHeMk4+dn7e3uVeWrzVOyur7Kgcmu5v3eE?= =?iso-8859-1?q?O1/bQUe1rz1QS6Z7EHdZZs8T25cw6nF77n/fZRbPnX7WvonQbaz?= =?iso-8859-1?q?mFPsaqRWp2TkEt3FzjClF7HrRgl6ClQcxz5cgn95b8WCC38FE08?= =?iso-8859-1?q?x6pvpWoOB5pCBnvGwxUwE+iJRiEvGfK3Z1ipWCKuLsUJ6E/t6rR?= =?iso-8859-1?q?7EkVdI+i3cQGE08oLz0q/1MqmxCw8Ewjb5Exh6RrYEx9uOrWKiV?= =?iso-8859-1?q?QNJYHmjBmd8T0jfvvGsLJ6BkwW/LLB8+FsBAQI99kXDGyOAHtyV?= =?iso-8859-1?q?inmGyktf439bIfxMW9wBa6FOelOcwEQ6DI3ttcTq+7d8Wpcr1cw?= =?iso-8859-1?q?hYtv+91X7v8YvRRQjzMEpm32jxSf7HjrO8jW0OS5f7EHID9Ryn+?= =?iso-8859-1?q?bzv5dZ9sra/PhE0AAl9md7+F7MdqcsQ7d4UizuHt3dDvAH6tvzs?= =?iso-8859-1?q?a1PKYypEkYr65aojrjmJpLUJhuo74U9vbytMveAEX18t0mxgZ6L?= =?iso-8859-1?q?EePW9wmTEraUi9soFdz2Ys3xHrW7kekLY6+0okTOh8ADO4amj9w?= =?iso-8859-1?q?drOa3s2HDFo5uWuVv092ed+ivk3yVbKT4DROHgyskyIHVA9ExaS?= =?iso-8859-1?q?zAeG8AEIVts9OSYgdy1poejdp6WoUoQ/srzbazpvG/aqaa6876w?= =?iso-8859-1?q?oeQDabyLG5JE9Hg/AFMslPRUy/RnyxhYWBeuYfXV0w3b0NOhgpy?= =?iso-8859-1?q?QqWqTHvCsoFM5rFkEXZE0jllbLd3x9COjYFa+PlClbHE2SYshVn?= =?iso-8859-1?q?fTHjShrz2SqKQFgAUEE4OyFBALOUG9/KasmTcmDTEkwVi/G4wAm?= =?iso-8859-1?q?RKJfrV8lOE7pDA5fUaW7lGs2c69vrHZCKhesZdd2XJhX0e5ob9t?= =?iso-8859-1?q?AIHJPAn8wyIaBcj24Xgp60sEO/o8uPFeTUTstp3ctO8GG2auznT?= =?iso-8859-1?q?t6sTGPAsH1xAXHtrTtrEmhSG9TrqNBnnXX4ZhHB+E//u0F4ekmZ?= =?iso-8859-1?q?Iny0nsAudnZrV99X/ET3x7v8ekkIj9qU+ZKAY2ZiYgfRbV8Gudm?= =?iso-8859-1?q?cozhUtqREZQUYNWaAmF8sBcKLWA5I0TMIOE9ejQ9cDYrLc+8bq3?= =?iso-8859-1?q?Kcx7BIUY2/giGuBht+lMUci4qZ9OlrY7i+5eyCR9xPDreH7aZFw?= =?iso-8859-1?q?bykTyszFNk0ycaTddbCiBwg8cAVCMLqjd0yU5aEF/T0XDlzugoT?= =?iso-8859-1?q?6EtjRLDHFia1dA=3D=3D?= x-microsoft-antispam-prvs: x-forefront-prvs: 09634B1196 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(136003)(346002)(376002)(39850400004)(396003)(189003)(199004)(6436002)(66066001)(68736007)(71200400001)(25786009)(4326008)(107886003)(6486002)(50226002)(14454004)(53946003)(6512007)(5660300002)(36756003)(71190400001)(305945005)(7736002)(53936002)(478600001)(81166006)(8676002)(8936002)(316002)(81156014)(476003)(2616005)(11346002)(26005)(102836004)(55236004)(106356001)(3846002)(186003)(86362001)(6116002)(110136005)(54906003)(99286004)(256004)(14444005)(76176011)(52116002)(6506007)(105586002)(386003)(486006)(30864003)(446003)(97736004)(2906002); DIR:OUT; SFP:1101; SCL:1; SRVR:MN2PR18MB2605; H:MN2PR18MB2877.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: marvell.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: gyr161VKz/v7sodXhJobqepGmq90iFYiYBLruPw/1Oar3VFQpVulvSCPzvKn0cDZMGQa+o1IaBs4vYJ2fAtsR7LoDt/2KgNXjw9xc53VemjRKWKW8+BmdnbdSCWORVIDb8mFEXw+DrBmdymcl5aMxT3mMGJvrlWH1wRL4B8dkmNtMc0ePoDKRFtOKeOSABvYyVTHFzAmlwPJW9KVlXY0luVvwprOtNv49hF6O5Uor4QJGN0YTFDhpzNyXUsXpdMu8MaGucCyq/yWVCAush3yGMOORDotSUYkBi/TFAG0pXZBS69/9iBEg/Bf1tIaT/YpaMPaRDskT4U7pO9hTfLvm/6rw3OPpvSmNhcA2KvPXlfzsxZOrJ5lhmftU67LOMCtA3++n5lS37I+nS/a7A5VNocq40z4/4S7LPo7ha8htW4= MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: b3281acc-9786-4942-e141-08d69e75aa09 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Mar 2019 18:42:32.4623 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR18MB2605 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-03-01_13:, , signatures=0 X-Proofpoint-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1903010129 Subject: [dpdk-dev] [PATCH 3/3] crypto/octeontx: move device specific code to driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Moving upper level enqueue/dequeue routines to driver. The h/w interface used to submit request has enough differences to substantiate the need for separate routines. Signed-off-by: Ankur Dwivedi Signed-off-by: Anoob Joseph Signed-off-by: Archana Muniganti --- drivers/common/cpt/cpt_common.h | 6 - drivers/common/cpt/cpt_mcode_defines.h | 12 + drivers/common/cpt/cpt_request_mgr.h | 185 --------------- drivers/crypto/octeontx/Makefile | 3 +- drivers/crypto/octeontx/meson.build | 2 +- drivers/crypto/octeontx/otx_cryptodev_hw_access.h | 2 + drivers/crypto/octeontx/otx_cryptodev_ops.c | 262 ++++++++++++++++++++-- 7 files changed, 255 insertions(+), 217 deletions(-) delete mode 100644 drivers/common/cpt/cpt_request_mgr.h diff --git a/drivers/common/cpt/cpt_common.h b/drivers/common/cpt/cpt_common.h index 8568d5b..ceb32f2 100644 --- a/drivers/common/cpt/cpt_common.h +++ b/drivers/common/cpt/cpt_common.h @@ -9,12 +9,6 @@ * This file defines common macros and structs */ -/* - * Macros to determine CPT model. Driver makefile will define CPT_MODEL - * accordingly - */ -#define CRYPTO_OCTEONTX 0x1 - #define TIME_IN_RESET_COUNT 5 /* Default command timeout in seconds */ diff --git a/drivers/common/cpt/cpt_mcode_defines.h b/drivers/common/cpt/cpt_mcode_defines.h index becc14f..c0adbd5 100644 --- a/drivers/common/cpt/cpt_mcode_defines.h +++ b/drivers/common/cpt/cpt_mcode_defines.h @@ -383,4 +383,16 @@ typedef mc_hash_type_t auth_type_t; #define SESS_PRIV(__sess) \ (void *)((uint8_t *)__sess + sizeof(struct cpt_sess_misc)) +/* + * Get the session size + * + * @return + * - session size + */ +static __rte_always_inline unsigned int +cpt_get_session_size(void) +{ + unsigned int ctx_len = sizeof(struct cpt_ctx); + return (sizeof(struct cpt_sess_misc) + RTE_ALIGN_CEIL(ctx_len, 8)); +} #endif /* _CPT_MCODE_DEFINES_H_ */ diff --git a/drivers/common/cpt/cpt_request_mgr.h b/drivers/common/cpt/cpt_request_mgr.h deleted file mode 100644 index 81bddf4..0000000 --- a/drivers/common/cpt/cpt_request_mgr.h +++ /dev/null @@ -1,185 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2018 Cavium, Inc - */ - -#ifndef _CPT_REQUEST_MGR_H_ -#define _CPT_REQUEST_MGR_H_ - -#include -#include - -#include "cpt_common.h" -#include "cpt_mcode_defines.h" - -#if CPT_MODEL == CRYPTO_OCTEONTX -#include "../../crypto/octeontx/otx_cryptodev_hw_access.h" -#endif - -/* - * This file defines the agreement between the common layer and the individual - * crypto drivers for OCTEON TX series. Datapath in otx* directory include this - * file and all these functions are static inlined for better performance. - * - */ - -/* - * Get the session size - * - * This function is used in the data path. - * - * @return - * - session size - */ -static __rte_always_inline unsigned int -cpt_get_session_size(void) -{ - unsigned int ctx_len = sizeof(struct cpt_ctx); - return (sizeof(struct cpt_sess_misc) + RTE_ALIGN_CEIL(ctx_len, 8)); -} - -static __rte_always_inline int32_t __hot -cpt_enqueue_req(struct cpt_instance *instance, struct pending_queue *pqueue, - void *req) -{ - struct cpt_request_info *user_req = (struct cpt_request_info *)req; - int32_t ret = 0; - - if (unlikely(!req)) - return 0; - - if (unlikely(pqueue->pending_count >= DEFAULT_CMD_QLEN)) - return -EAGAIN; - - fill_cpt_inst(instance, req); - - CPT_LOG_DP_DEBUG("req: %p op: %p ", req, user_req->op); - - /* Fill time_out cycles */ - user_req->time_out = rte_get_timer_cycles() + - DEFAULT_COMMAND_TIMEOUT * rte_get_timer_hz(); - user_req->extra_time = 0; - - /* Default mode of software queue */ - mark_cpt_inst(instance); - - pqueue->rid_queue[pqueue->enq_tail].rid = - (uintptr_t)user_req; - /* We will use soft queue length here to limit - * requests - */ - MOD_INC(pqueue->enq_tail, DEFAULT_CMD_QLEN); - pqueue->pending_count += 1; - - CPT_LOG_DP_DEBUG("Submitted NB cmd with request: %p " - "op: %p", user_req, user_req->op); - - return ret; -} - -static __rte_always_inline int __hot -cpt_pmd_crypto_operation(struct cpt_instance *instance, - struct rte_crypto_op *op, struct pending_queue *pqueue, - uint8_t cpt_driver_id) -{ - struct cpt_sess_misc *sess = NULL; - struct rte_crypto_sym_op *sym_op = op->sym; - void *prep_req = NULL, *mdata = NULL; - int ret = 0; - uint64_t cpt_op; - struct cpt_vf *cptvf = (struct cpt_vf *)instance; - - if (unlikely(op->sess_type == RTE_CRYPTO_OP_SESSIONLESS)) { - int sess_len; - - sess_len = cpt_get_session_size(); - - sess = rte_calloc(__func__, 1, sess_len, 8); - if (!sess) - return -ENOMEM; - - sess->ctx_dma_addr = rte_malloc_virt2iova(sess) + - sizeof(struct cpt_sess_misc); - - ret = instance_session_cfg(sym_op->xform, (void *)sess); - if (unlikely(ret)) - return -EINVAL; - } else { - sess = (struct cpt_sess_misc *) - get_sym_session_private_data(sym_op->session, - cpt_driver_id); - } - - cpt_op = sess->cpt_op; - - if (likely(cpt_op & CPT_OP_CIPHER_MASK)) - ret = fill_fc_params(op, sess, &cptvf->meta_info, &mdata, - &prep_req); - else - ret = fill_digest_params(op, sess, &cptvf->meta_info, - &mdata, &prep_req); - - if (unlikely(ret)) { - CPT_LOG_DP_ERR("prep cryto req : op %p, cpt_op 0x%x " - "ret 0x%x", op, (unsigned int)cpt_op, ret); - return ret; - } - - /* Enqueue prepared instruction to HW */ - ret = cpt_enqueue_req(instance, pqueue, prep_req); - - if (unlikely(ret)) { - if (unlikely(ret == -EAGAIN)) - goto req_fail; - CPT_LOG_DP_ERR("Error enqueing crypto request : error " - "code %d", ret); - goto req_fail; - } - - return 0; - -req_fail: - if (mdata) - free_op_meta(mdata, cptvf->meta_info.cptvf_meta_pool); - return ret; -} - -static __rte_always_inline int32_t __hot -cpt_dequeue_burst(struct cpt_instance *instance, uint16_t cnt, - void *resp[], uint8_t cc[], struct pending_queue *pqueue) -{ - struct cpt_request_info *user_req; - struct rid *rid_e; - int i, count, pcount; - uint8_t ret; - - pcount = pqueue->pending_count; - count = (cnt > pcount) ? pcount : cnt; - - for (i = 0; i < count; i++) { - rid_e = &pqueue->rid_queue[pqueue->deq_head]; - user_req = (struct cpt_request_info *)(rid_e->rid); - - if (likely((i+1) < count)) - rte_prefetch_non_temporal((void *)rid_e[1].rid); - - ret = check_nb_command_id(user_req, instance); - - if (unlikely(ret == ERR_REQ_PENDING)) { - /* Stop checking for completions */ - break; - } - - /* Return completion code and op handle */ - cc[i] = (uint8_t)ret; - resp[i] = user_req->op; - CPT_LOG_DP_DEBUG("Request %p Op %p completed with code %d", - user_req, user_req->op, ret); - - MOD_INC(pqueue->deq_head, DEFAULT_CMD_QLEN); - pqueue->pending_count -= 1; - } - - return i; -} - -#endif /* _CPT_REQUEST_MGR_H_ */ diff --git a/drivers/crypto/octeontx/Makefile b/drivers/crypto/octeontx/Makefile index d6412d1..2752cbc 100644 --- a/drivers/crypto/octeontx/Makefile +++ b/drivers/crypto/octeontx/Makefile @@ -20,8 +20,9 @@ LDLIBS += -lrte_common_cpt VPATH += $(RTE_SDK)/drivers/crypto/octeontx -CFLAGS += -O3 -DCPT_MODEL=CRYPTO_OCTEONTX +CFLAGS += -O3 CFLAGS += -I$(RTE_SDK)/drivers/common/cpt +CFLAGS += -DALLOW_EXPERIMENTAL_API # PMD code SRCS-$(CONFIG_RTE_LIBRTE_PMD_OCTEONTX_CRYPTO) += otx_cryptodev.c diff --git a/drivers/crypto/octeontx/meson.build b/drivers/crypto/octeontx/meson.build index 6511b40..423737e 100644 --- a/drivers/crypto/octeontx/meson.build +++ b/drivers/crypto/octeontx/meson.build @@ -8,6 +8,7 @@ deps += ['bus_pci'] deps += ['common_cpt'] name = 'octeontx_crypto' +allow_experimental_apis = true sources = files('otx_cryptodev.c', 'otx_cryptodev_capabilities.c', 'otx_cryptodev_hw_access.c', @@ -15,4 +16,3 @@ sources = files('otx_cryptodev.c', 'otx_cryptodev_ops.c') includes += include_directories('../../common/cpt') -cflags += '-DCPT_MODEL=CRYPTO_OCTEONTX' diff --git a/drivers/crypto/octeontx/otx_cryptodev_hw_access.h b/drivers/crypto/octeontx/otx_cryptodev_hw_access.h index 82b15ee..dea4cba 100644 --- a/drivers/crypto/octeontx/otx_cryptodev_hw_access.h +++ b/drivers/crypto/octeontx/otx_cryptodev_hw_access.h @@ -39,6 +39,8 @@ struct cpt_instance { uint32_t queue_id; uintptr_t rsvd; + struct rte_mempool *sess_mp; + struct rte_mempool *sess_mp_priv; }; struct command_chunk { diff --git a/drivers/crypto/octeontx/otx_cryptodev_ops.c b/drivers/crypto/octeontx/otx_cryptodev_ops.c index 6a0cf83..0f9f2a2 100644 --- a/drivers/crypto/octeontx/otx_cryptodev_ops.c +++ b/drivers/crypto/octeontx/otx_cryptodev_ops.c @@ -11,7 +11,6 @@ #include "cpt_pmd_logs.h" #include "cpt_pmd_ops_helper.h" #include "cpt_ucode.h" -#include "cpt_request_mgr.h" #include "otx_cryptodev.h" #include "otx_cryptodev_capabilities.h" @@ -222,6 +221,8 @@ otx_cpt_que_pair_setup(struct rte_cryptodev *dev, } instance->queue_id = que_pair_id; + instance->sess_mp = qp_conf->mp_session; + instance->sess_mp_priv = qp_conf->mp_session_private; dev->data->queue_pairs[que_pair_id] = instance; return 0; @@ -340,11 +341,160 @@ otx_cpt_session_clear(struct rte_cryptodev *dev, } } +static __rte_always_inline int32_t __hot +otx_cpt_request_enqueue(struct cpt_instance *instance, + struct pending_queue *pqueue, + void *req) +{ + struct cpt_request_info *user_req = (struct cpt_request_info *)req; + + if (unlikely(pqueue->pending_count >= DEFAULT_CMD_QLEN)) + return -EAGAIN; + + fill_cpt_inst(instance, req); + + CPT_LOG_DP_DEBUG("req: %p op: %p ", req, user_req->op); + + /* Fill time_out cycles */ + user_req->time_out = rte_get_timer_cycles() + + DEFAULT_COMMAND_TIMEOUT * rte_get_timer_hz(); + user_req->extra_time = 0; + + /* Default mode of software queue */ + mark_cpt_inst(instance); + + pqueue->rid_queue[pqueue->enq_tail].rid = (uintptr_t)user_req; + + /* We will use soft queue length here to limit requests */ + MOD_INC(pqueue->enq_tail, DEFAULT_CMD_QLEN); + pqueue->pending_count += 1; + + CPT_LOG_DP_DEBUG("Submitted NB cmd with request: %p " + "op: %p", user_req, user_req->op); + return 0; +} + +static __rte_always_inline int __hot +otx_cpt_enq_single_sym(struct cpt_instance *instance, + struct rte_crypto_op *op, + struct pending_queue *pqueue) +{ + struct cpt_sess_misc *sess; + struct rte_crypto_sym_op *sym_op = op->sym; + void *prep_req, *mdata = NULL; + int ret = 0; + uint64_t cpt_op; + struct cpt_vf *cptvf = (struct cpt_vf *)instance; + + sess = (struct cpt_sess_misc *) + get_sym_session_private_data(sym_op->session, + otx_cryptodev_driver_id); + + cpt_op = sess->cpt_op; + + if (likely(cpt_op & CPT_OP_CIPHER_MASK)) + ret = fill_fc_params(op, sess, &cptvf->meta_info, &mdata, + &prep_req); + else + ret = fill_digest_params(op, sess, &cptvf->meta_info, + &mdata, &prep_req); + + if (unlikely(ret)) { + CPT_LOG_DP_ERR("prep cryto req : op %p, cpt_op 0x%x " + "ret 0x%x", op, (unsigned int)cpt_op, ret); + return ret; + } + + /* Enqueue prepared instruction to h/w */ + ret = otx_cpt_request_enqueue(instance, pqueue, prep_req); + + if (unlikely(ret)) { + /* Buffer allocated for request preparation need to be freed */ + free_op_meta(mdata, cptvf->meta_info.cptvf_meta_pool); + return ret; + } + + return 0; +} + +static __rte_always_inline int __hot +otx_cpt_enq_single_sym_sessless(struct cpt_instance *instance, + struct rte_crypto_op *op, + struct pending_queue *pqueue) +{ + struct cpt_sess_misc *sess; + struct rte_crypto_sym_op *sym_op = op->sym; + int ret; + void *sess_t = NULL; + void *sess_private_data_t = NULL; + + /* Create tmp session */ + + if (rte_mempool_get(instance->sess_mp, (void **)&sess_t)) { + ret = -ENOMEM; + goto exit; + } + + if (rte_mempool_get(instance->sess_mp_priv, + (void **)&sess_private_data_t)) { + ret = -ENOMEM; + goto free_sess; + } + + sess = (struct cpt_sess_misc *)sess_private_data_t; + + sess->ctx_dma_addr = rte_mempool_virt2iova(sess) + + sizeof(struct cpt_sess_misc); + + ret = instance_session_cfg(sym_op->xform, (void *)sess); + if (unlikely(ret)) { + ret = -EINVAL; + goto free_sess_priv; + } + + /* Save tmp session in op */ + + sym_op->session = (struct rte_cryptodev_sym_session *)sess_t; + set_sym_session_private_data(sym_op->session, otx_cryptodev_driver_id, + sess_private_data_t); + + /* Enqueue op with the tmp session set */ + ret = otx_cpt_enq_single_sym(instance, op, pqueue); + + if (unlikely(ret)) + goto free_sess_priv; + + return 0; + +free_sess_priv: + rte_mempool_put(instance->sess_mp_priv, sess_private_data_t); +free_sess: + rte_mempool_put(instance->sess_mp, sess_t); +exit: + return ret; +} + +static __rte_always_inline int __hot +otx_cpt_enq_single(struct cpt_instance *inst, + struct rte_crypto_op *op, + struct pending_queue *pqueue) +{ + /* Check for the type */ + + if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) + return otx_cpt_enq_single_sym(inst, op, pqueue); + else if (unlikely(op->sess_type == RTE_CRYPTO_OP_SESSIONLESS)) + return otx_cpt_enq_single_sym_sessless(inst, op, pqueue); + + /* Should not reach here */ + return -EINVAL; +} + static uint16_t otx_cpt_pkt_enqueue(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) { struct cpt_instance *instance = (struct cpt_instance *)qptr; - uint16_t count = 0; + uint16_t count; int ret; struct cpt_vf *cptvf = (struct cpt_vf *)instance; struct pending_queue *pqueue = &cptvf->pqueue; @@ -355,8 +505,10 @@ otx_cpt_pkt_enqueue(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) count = 0; while (likely(count < nb_ops)) { - ret = cpt_pmd_crypto_operation(instance, ops[count], pqueue, - otx_cryptodev_driver_id); + + /* Enqueue single op */ + ret = otx_cpt_enq_single(instance, ops[count], pqueue); + if (unlikely(ret)) break; count++; @@ -365,48 +517,110 @@ otx_cpt_pkt_enqueue(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) return count; } +static __rte_always_inline void +otx_cpt_dequeue_post_process(struct rte_crypto_op *cop, uintptr_t *rsp) +{ + /* H/w has returned success */ + cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + + /* Perform further post processing */ + + if (cop->type == RTE_CRYPTO_OP_TYPE_SYMMETRIC) { + /* Check if auth verify need to be completed */ + if (unlikely(rsp[2])) + compl_auth_verify(cop, (uint8_t *)rsp[2], rsp[3]); + return; + } +} + static uint16_t otx_cpt_pkt_dequeue(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) { struct cpt_instance *instance = (struct cpt_instance *)qptr; + struct cpt_request_info *user_req; struct cpt_vf *cptvf = (struct cpt_vf *)instance; + struct rid *rid_e; + uint8_t cc[nb_ops]; + int i, count, pcount; + uint8_t ret; + int nb_completed; struct pending_queue *pqueue = &cptvf->pqueue; - uint16_t nb_completed, i = 0; - uint8_t compcode[nb_ops]; + struct rte_crypto_op *cop; + void *metabuf; + uintptr_t *rsp; + + pcount = pqueue->pending_count; + count = (nb_ops > pcount) ? pcount : nb_ops; + + for (i = 0; i < count; i++) { + rid_e = &pqueue->rid_queue[pqueue->deq_head]; + user_req = (struct cpt_request_info *)(rid_e->rid); + + if (likely((i+1) < count)) + rte_prefetch_non_temporal((void *)rid_e[1].rid); + + ret = check_nb_command_id(user_req, instance); + + if (unlikely(ret == ERR_REQ_PENDING)) { + /* Stop checking for completions */ + break; + } + + /* Return completion code and op handle */ + cc[i] = ret; + ops[i] = user_req->op; + + CPT_LOG_DP_DEBUG("Request %p Op %p completed with code %d", + user_req, user_req->op, ret); - nb_completed = cpt_dequeue_burst(instance, nb_ops, - (void **)ops, compcode, pqueue); - while (likely(i < nb_completed)) { - struct rte_crypto_op *cop; - void *metabuf; - uintptr_t *rsp; - uint8_t status; + MOD_INC(pqueue->deq_head, DEFAULT_CMD_QLEN); + pqueue->pending_count -= 1; + } + + nb_completed = i; + + for (i = 0; i < nb_completed; i++) { rsp = (void *)ops[i]; - status = compcode[i]; + if (likely((i + 1) < nb_completed)) rte_prefetch0(ops[i+1]); + metabuf = (void *)rsp[0]; cop = (void *)rsp[1]; ops[i] = cop; - if (likely(status == 0)) { - if (likely(!rsp[2])) - cop->status = - RTE_CRYPTO_OP_STATUS_SUCCESS; - else - compl_auth_verify(cop, (uint8_t *)rsp[2], - rsp[3]); - } else if (status == ERR_GC_ICV_MISCOMPARE) { - /*auth data mismatch */ + /* Check completion code */ + + if (likely(cc[i] == 0)) { + /* H/w success pkt. Post process */ + otx_cpt_dequeue_post_process(cop, rsp); + } else if (cc[i] == ERR_GC_ICV_MISCOMPARE) { + /* auth data mismatch */ cop->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; } else { + /* Error */ cop->status = RTE_CRYPTO_OP_STATUS_ERROR; } + + if (unlikely(cop->sess_type == RTE_CRYPTO_OP_SESSIONLESS)) { + void *sess_private_data_t = + get_sym_session_private_data(cop->sym->session, + otx_cryptodev_driver_id); + memset(sess_private_data_t, 0, + cpt_get_session_size()); + memset(cop->sym->session, 0, + rte_cryptodev_sym_get_existing_header_session_size( + cop->sym->session)); + rte_mempool_put(instance->sess_mp_priv, + sess_private_data_t); + rte_mempool_put(instance->sess_mp, cop->sym->session); + cop->sym->session = NULL; + } free_op_meta(metabuf, cptvf->meta_info.cptvf_meta_pool); - i++; } + return nb_completed; }