From patchwork Thu Mar 5 15:34:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "De Lara Guarch, Pablo" X-Patchwork-Id: 66304 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id DE65EA0564; Thu, 5 Mar 2020 18:09:48 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 5EBC71BFDA; Thu, 5 Mar 2020 18:09:40 +0100 (CET) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 3282F2BA8 for ; Thu, 5 Mar 2020 18:09:37 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Mar 2020 09:09:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,518,1574150400"; d="scan'208";a="275173376" Received: from silpixa00400565.ir.intel.com (HELO silpixa00400565.ger.corp.intel.com) ([10.237.222.249]) by fmsmga002.fm.intel.com with ESMTP; 05 Mar 2020 09:09:35 -0800 From: Pablo de Lara To: dev@dpdk.org Cc: Pablo de Lara Date: Thu, 5 Mar 2020 15:34:50 +0000 Message-Id: <20200305153454.724874-2-pablo.de.lara.guarch@intel.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200305153454.724874-1-pablo.de.lara.guarch@intel.com> References: <20200305153454.724874-1-pablo.de.lara.guarch@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [RFC PATCH 1/5] crypto/zuc: use IPSec MB library v0.53 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Link against Intel IPSec Multi-buffer library, which added support for ZUC-EEA3 and ZUC-EIA3 from version v0.53, moving from libSSO ZUC library. Signed-off-by: Pablo de Lara --- devtools/test-build.sh | 6 +-- doc/guides/cryptodevs/zuc.rst | 52 ++++++++++++--------- drivers/crypto/zuc/Makefile | 28 +++++++----- drivers/crypto/zuc/meson.build | 13 +++++- drivers/crypto/zuc/rte_zuc_pmd.c | 58 +++++++++++++++++------- drivers/crypto/zuc/rte_zuc_pmd_ops.c | 2 + drivers/crypto/zuc/rte_zuc_pmd_private.h | 6 ++- mk/rte.app.mk | 2 +- 8 files changed, 110 insertions(+), 57 deletions(-) diff --git a/devtools/test-build.sh b/devtools/test-build.sh index 42f4ad003..911f77e01 100755 --- a/devtools/test-build.sh +++ b/devtools/test-build.sh @@ -25,7 +25,6 @@ default_path=$PATH # - LIBMUSDK_PATH # - LIBSSO_SNOW3G_PATH # - LIBSSO_KASUMI_PATH -# - LIBSSO_ZUC_PATH . $(dirname $(readlink -e $0))/load-devel-config print_usage () { @@ -111,7 +110,6 @@ reset_env () unset LIBMUSDK_PATH unset LIBSSO_SNOW3G_PATH unset LIBSSO_KASUMI_PATH - unset LIBSSO_ZUC_PATH unset PQOS_INSTALL_PATH } @@ -165,12 +163,12 @@ config () # sed -ri 's,(PMD_AESNI_MB=)n,\1y,' $1/.config test "$DPDK_DEP_IPSEC_MB" != y || \ sed -ri 's,(PMD_AESNI_GCM=)n,\1y,' $1/.config + test "$DPDK_DEP_IPSEC_MB" != y || \ + sed -ri 's,(PMD_ZUC=)n,\1y,' $1/.config test -z "$LIBSSO_SNOW3G_PATH" || \ sed -ri 's,(PMD_SNOW3G=)n,\1y,' $1/.config test -z "$LIBSSO_KASUMI_PATH" || \ sed -ri 's,(PMD_KASUMI=)n,\1y,' $1/.config - test -z "$LIBSSO_ZUC_PATH" || \ - sed -ri 's,(PMD_ZUC=)n,\1y,' $1/.config test "$DPDK_DEP_SSL" != y || \ sed -ri 's,(PMD_CCP=)n,\1y,' $1/.config test "$DPDK_DEP_SSL" != y || \ diff --git a/doc/guides/cryptodevs/zuc.rst b/doc/guides/cryptodevs/zuc.rst index e38989968..d4001b1f4 100644 --- a/doc/guides/cryptodevs/zuc.rst +++ b/doc/guides/cryptodevs/zuc.rst @@ -1,12 +1,12 @@ .. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2016 Intel Corporation. + Copyright(c) 2016-2019 Intel Corporation. ZUC Crypto Poll Mode Driver =========================== -The ZUC PMD (**librte_pmd_zuc**) provides poll mode crypto driver -support for utilizing Intel Libsso library, which implements F8 and F9 functions -for ZUC EEA3 cipher and EIA3 hash algorithms. +The ZUC PMD (**librte_pmd_zuc**) provides poll mode crypto driver support for +utilizing `Intel IPSec Multi-buffer library `_ +which implements F8 and F9 functions for ZUC EEA3 cipher and EIA3 hash algorithms. Features -------- @@ -27,36 +27,46 @@ Limitations * Chained mbufs are not supported. * ZUC (EIA3) supported only if hash offset field is byte-aligned. * ZUC (EEA3) supported only if cipher length, cipher offset fields are byte-aligned. -* ZUC PMD cannot be built as a shared library, due to limitations in - in the underlying library. Installation ------------ -To build DPDK with the ZUC_PMD the user is required to download -the export controlled ``libsso_zuc`` library, by registering in -`Intel Resource & Design Center `_. -Once approval has been granted, the user needs to search for -*ZUC 128-EAA3 and 128-EIA3 3GPP cryptographic algorithms Software Library* to download the -library or directly through this `link `_. +To build DPDK with the ZUC_PMD the user is required to download the multi-buffer +library from `here `_ +and compile it on their user system before building DPDK. +The latest version of the library supported by this PMD is v0.53, which +can be downloaded from ``_. + After downloading the library, the user needs to unpack and compile it -on their system before building DPDK:: +on their system before building DPDK: + +.. code-block:: console + + make + make install + +As a reference, the following table shows a mapping between the past DPDK versions +and the external crypto libraries supported by them: + +.. _table_zuc_versions: + +.. table:: DPDK and external crypto library version compatibility + + ============= ================================ + DPDK version Crypto library version + ============= ================================ + 16.11 - 19.11 LibSSO ZUC + 20.02+ Multi-buffer library 0.53 + ============= ================================ - make Initialization -------------- In order to enable this virtual crypto PMD, user must: -* Export the environmental variable LIBSSO_ZUC_PATH with the path where - the library was extracted (zuc folder). - -* Export the environmental variable LD_LIBRARY_PATH with the path - where the built libsso library is (LIBSSO_ZUC_PATH/build). - -* Build the LIBSSO_ZUC library (explained in Installation section). +* Build the multi buffer library (explained in Installation section). * Build DPDK as follows: diff --git a/drivers/crypto/zuc/Makefile b/drivers/crypto/zuc/Makefile index 68d84eebc..cc0d7943b 100644 --- a/drivers/crypto/zuc/Makefile +++ b/drivers/crypto/zuc/Makefile @@ -1,14 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2016 Intel Corporation +# Copyright(c) 2016-2019 Intel Corporation include $(RTE_SDK)/mk/rte.vars.mk -ifneq ($(MAKECMDGOALS),clean) -ifeq ($(LIBSSO_ZUC_PATH),) -$(error "Please define LIBSSO_ZUC_PATH environment variable") -endif -endif - # library name LIB = librte_pmd_zuc.a @@ -23,14 +17,26 @@ LIBABIVER := 1 EXPORT_MAP := rte_pmd_zuc_version.map # external library dependencies -CFLAGS += -I$(LIBSSO_ZUC_PATH) -CFLAGS += -I$(LIBSSO_ZUC_PATH)/include -CFLAGS += -I$(LIBSSO_ZUC_PATH)/build -LDLIBS += -L$(LIBSSO_ZUC_PATH)/build -lsso_zuc +LDLIBS += -lIPSec_MB LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_cryptodev LDLIBS += -lrte_bus_vdev +IMB_HDR = $(shell echo '\#include ' | \ + $(CC) -E $(EXTRA_CFLAGS) - | grep 'intel-ipsec-mb.h' | \ + head -n1 | cut -d'"' -f2) + +# Detect library version +IMB_VERSION = $(shell grep -e "IMB_VERSION_STR" $(IMB_HDR) | cut -d'"' -f2) +IMB_VERSION_NUM = $(shell grep -e "IMB_VERSION_NUM" $(IMB_HDR) | cut -d' ' -f3) + +ifeq ($(IMB_VERSION),) +$(error "IPSec_MB version >= 0.53 is required") +endif + +ifeq ($(shell expr $(IMB_VERSION_NUM) \< 0x3400), 1) +$(error "IPSec_MB version >= 0.53 is required") +endif # library source files SRCS-$(CONFIG_RTE_LIBRTE_PMD_ZUC) += rte_zuc_pmd.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_ZUC) += rte_zuc_pmd_ops.c diff --git a/drivers/crypto/zuc/meson.build b/drivers/crypto/zuc/meson.build index b8ca7107e..f0fcd8246 100644 --- a/drivers/crypto/zuc/meson.build +++ b/drivers/crypto/zuc/meson.build @@ -1,11 +1,20 @@ # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2018 Intel Corporation +# Copyright(c) 2018-2020 Intel Corporation -lib = cc.find_library('libsso_zuc', required: false) +IMB_required_ver = '0.53.0' +lib = cc.find_library('IPSec_MB', required: false) if not lib.found() build = false else ext_deps += lib + # version comes with quotes, so we split based on " and take the middle + imb_ver = cc.get_define('IMB_VERSION_STR', + prefix : '#include').split('"')[1] + + if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver)) + build = false + endif + endif sources = files('rte_zuc_pmd.c', 'rte_zuc_pmd_ops.c') diff --git a/drivers/crypto/zuc/rte_zuc_pmd.c b/drivers/crypto/zuc/rte_zuc_pmd.c index 313f4590b..c880eea7c 100644 --- a/drivers/crypto/zuc/rte_zuc_pmd.c +++ b/drivers/crypto/zuc/rte_zuc_pmd.c @@ -11,7 +11,7 @@ #include #include "rte_zuc_pmd_private.h" -#define ZUC_MAX_BURST 4 +#define ZUC_MAX_BURST 16 #define BYTE_LEN 8 static uint8_t cryptodev_driver_id; @@ -169,16 +169,17 @@ zuc_get_session(struct zuc_qp *qp, struct rte_crypto_op *op) /** Encrypt/decrypt mbufs. */ static uint8_t -process_zuc_cipher_op(struct rte_crypto_op **ops, +process_zuc_cipher_op(struct zuc_qp *qp, struct rte_crypto_op **ops, struct zuc_session **sessions, uint8_t num_ops) { unsigned i; uint8_t processed_ops = 0; - uint8_t *src[ZUC_MAX_BURST], *dst[ZUC_MAX_BURST]; - uint8_t *iv[ZUC_MAX_BURST]; + const void *src[ZUC_MAX_BURST]; + void *dst[ZUC_MAX_BURST]; + const void *iv[ZUC_MAX_BURST]; uint32_t num_bytes[ZUC_MAX_BURST]; - uint8_t *cipher_keys[ZUC_MAX_BURST]; + const void *cipher_keys[ZUC_MAX_BURST]; struct zuc_session *sess; for (i = 0; i < num_ops; i++) { @@ -221,7 +222,8 @@ process_zuc_cipher_op(struct rte_crypto_op **ops, processed_ops++; } - sso_zuc_eea3_n_buffer(cipher_keys, iv, src, dst, + IMB_ZUC_EEA3_N_BUFFER(qp->mb_mgr, (const void **)cipher_keys, + (const void **)iv, (const void **)src, (void **)dst, num_bytes, processed_ops); return processed_ops; @@ -261,7 +263,7 @@ process_zuc_hash_op(struct zuc_qp *qp, struct rte_crypto_op **ops, if (sess->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) { dst = (uint32_t *)qp->temp_digest; - sso_zuc_eia3_1_buffer(sess->pKey_hash, + IMB_ZUC_EIA3_1_BUFFER(qp->mb_mgr, sess->pKey_hash, iv, src, length_in_bits, dst); /* Verify digest. */ @@ -271,7 +273,7 @@ process_zuc_hash_op(struct zuc_qp *qp, struct rte_crypto_op **ops, } else { dst = (uint32_t *)ops[i]->sym->auth.digest.data; - sso_zuc_eia3_1_buffer(sess->pKey_hash, + IMB_ZUC_EIA3_1_BUFFER(qp->mb_mgr, sess->pKey_hash, iv, src, length_in_bits, dst); } @@ -293,7 +295,7 @@ process_ops(struct rte_crypto_op **ops, enum zuc_operation op_type, switch (op_type) { case ZUC_OP_ONLY_CIPHER: - processed_ops = process_zuc_cipher_op(ops, + processed_ops = process_zuc_cipher_op(qp, ops, sessions, num_ops); break; case ZUC_OP_ONLY_AUTH: @@ -301,14 +303,14 @@ process_ops(struct rte_crypto_op **ops, enum zuc_operation op_type, num_ops); break; case ZUC_OP_CIPHER_AUTH: - processed_ops = process_zuc_cipher_op(ops, sessions, + processed_ops = process_zuc_cipher_op(qp, ops, sessions, num_ops); process_zuc_hash_op(qp, ops, sessions, processed_ops); break; case ZUC_OP_AUTH_CIPHER: processed_ops = process_zuc_hash_op(qp, ops, sessions, num_ops); - process_zuc_cipher_op(ops, sessions, processed_ops); + process_zuc_cipher_op(qp, ops, sessions, processed_ops); break; default: /* Operation not supported. */ @@ -455,8 +457,7 @@ cryptodev_zuc_create(const char *name, { struct rte_cryptodev *dev; struct zuc_private *internals; - uint64_t cpu_flags = RTE_CRYPTODEV_FF_CPU_SSE; - + MB_MGR *mb_mgr; dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params); if (dev == NULL) { @@ -464,6 +465,27 @@ cryptodev_zuc_create(const char *name, goto init_error; } + dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | + RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING; + + mb_mgr = alloc_mb_mgr(0); + if (mb_mgr == NULL) + return -ENOMEM; + + if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F)) { + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512; + init_mb_mgr_avx512(mb_mgr); + } else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) { + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; + init_mb_mgr_avx2(mb_mgr); + } else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) { + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; + init_mb_mgr_avx(mb_mgr); + } else { + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; + init_mb_mgr_sse(mb_mgr); + } + dev->driver_id = cryptodev_driver_id; dev->dev_ops = rte_zuc_pmd_ops; @@ -471,11 +493,8 @@ cryptodev_zuc_create(const char *name, dev->dequeue_burst = zuc_pmd_dequeue_burst; dev->enqueue_burst = zuc_pmd_enqueue_burst; - dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | - RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - cpu_flags; - internals = dev->data->dev_private; + internals->mb_mgr = mb_mgr; internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs; @@ -516,6 +535,7 @@ cryptodev_zuc_remove(struct rte_vdev_device *vdev) struct rte_cryptodev *cryptodev; const char *name; + struct zuc_private *internals; name = rte_vdev_device_name(vdev); if (name == NULL) @@ -525,6 +545,10 @@ cryptodev_zuc_remove(struct rte_vdev_device *vdev) if (cryptodev == NULL) return -ENODEV; + internals = cryptodev->data->dev_private; + + free_mb_mgr(internals->mb_mgr); + return rte_cryptodev_pmd_destroy(cryptodev); } diff --git a/drivers/crypto/zuc/rte_zuc_pmd_ops.c b/drivers/crypto/zuc/rte_zuc_pmd_ops.c index 6da396542..14a831867 100644 --- a/drivers/crypto/zuc/rte_zuc_pmd_ops.c +++ b/drivers/crypto/zuc/rte_zuc_pmd_ops.c @@ -196,6 +196,7 @@ zuc_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, int socket_id, struct rte_mempool *session_pool) { struct zuc_qp *qp = NULL; + struct zuc_private *internals = dev->data->dev_private; /* Free memory prior to re-allocation if needed. */ if (dev->data->queue_pairs[qp_id] != NULL) @@ -218,6 +219,7 @@ zuc_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, if (qp->processed_ops == NULL) goto qp_setup_cleanup; + qp->mb_mgr = internals->mb_mgr; qp->sess_mp = session_pool; memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); diff --git a/drivers/crypto/zuc/rte_zuc_pmd_private.h b/drivers/crypto/zuc/rte_zuc_pmd_private.h index 5e5906ddb..e049df377 100644 --- a/drivers/crypto/zuc/rte_zuc_pmd_private.h +++ b/drivers/crypto/zuc/rte_zuc_pmd_private.h @@ -5,7 +5,7 @@ #ifndef _RTE_ZUC_PMD_PRIVATE_H_ #define _RTE_ZUC_PMD_PRIVATE_H_ -#include +#include #define CRYPTODEV_NAME_ZUC_PMD crypto_zuc /**< KASUMI PMD device name */ @@ -24,6 +24,8 @@ int zuc_logtype_driver; struct zuc_private { unsigned max_nb_queue_pairs; /**< Max number of queue pairs supported by device */ + MB_MGR *mb_mgr; + /**< Multi-buffer instance */ }; /** ZUC buffer queue pair */ @@ -43,6 +45,8 @@ struct zuc_qp { * by the driver when verifying a digest provided * by the user (using authentication verify operation) */ + MB_MGR *mb_mgr; + /**< Multi-buffer instance */ } __rte_cache_aligned; enum zuc_operation { diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 5699d979d..7c04ee490 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -233,7 +233,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G) += -L$(LIBSSO_SNOW3G_PATH)/build -l _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI) += -lrte_pmd_kasumi _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI) += -L$(LIBSSO_KASUMI_PATH)/build -lsso_kasumi _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ZUC) += -lrte_pmd_zuc -_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ZUC) += -L$(LIBSSO_ZUC_PATH)/build -lsso_zuc +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ZUC) += -lIPSec_MB _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ARMV8_CRYPTO) += -lrte_pmd_armv8 _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ARMV8_CRYPTO) += -L$(ARMV8_CRYPTO_LIB_PATH) -larmv8_crypto _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MVSAM_CRYPTO) += -L$(LIBMUSDK_PATH)/lib -lrte_pmd_mvsam_crypto -lmusdk From patchwork Thu Mar 5 15:34:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "De Lara Guarch, Pablo" X-Patchwork-Id: 66305 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 57602A0564; Thu, 5 Mar 2020 18:10:03 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 9CF5F1BFF2; Thu, 5 Mar 2020 18:09:42 +0100 (CET) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 257572BA8 for ; Thu, 5 Mar 2020 18:09:37 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Mar 2020 09:09:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,518,1574150400"; d="scan'208";a="275173381" Received: from silpixa00400565.ir.intel.com (HELO silpixa00400565.ger.corp.intel.com) ([10.237.222.249]) by fmsmga002.fm.intel.com with ESMTP; 05 Mar 2020 09:09:36 -0800 From: Pablo de Lara To: dev@dpdk.org Cc: Pablo de Lara Date: Thu, 5 Mar 2020 15:34:51 +0000 Message-Id: <20200305153454.724874-3-pablo.de.lara.guarch@intel.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200305153454.724874-1-pablo.de.lara.guarch@intel.com> References: <20200305153454.724874-1-pablo.de.lara.guarch@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [RFC PATCH 2/5] crypto/snow3g: use IPSec MB library v0.53 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Link against Intel IPSec Multi-buffer library, which added support for SNOW3G-UEA2 and SNOW3G-UIA2 from version v0.53, moving from libSSO SNOW3G library. Signed-off-by: Pablo de Lara --- devtools/test-build.sh | 4 +- doc/guides/cryptodevs/snow3g.rst | 58 ++++++++------ drivers/crypto/snow3g/Makefile | 29 ++++--- drivers/crypto/snow3g/meson.build | 21 +++++ drivers/crypto/snow3g/rte_snow3g_pmd.c | 79 ++++++++++++------- drivers/crypto/snow3g/rte_snow3g_pmd_ops.c | 8 +- .../crypto/snow3g/rte_snow3g_pmd_private.h | 12 ++- mk/rte.app.mk | 2 +- 8 files changed, 139 insertions(+), 74 deletions(-) create mode 100644 drivers/crypto/snow3g/meson.build diff --git a/devtools/test-build.sh b/devtools/test-build.sh index 911f77e01..0c62c3950 100755 --- a/devtools/test-build.sh +++ b/devtools/test-build.sh @@ -23,7 +23,6 @@ default_path=$PATH # - DPDK_NOTIFY (notify-send) # - FLEXRAN_SDK # - LIBMUSDK_PATH -# - LIBSSO_SNOW3G_PATH # - LIBSSO_KASUMI_PATH . $(dirname $(readlink -e $0))/load-devel-config @@ -108,7 +107,6 @@ reset_env () unset ARMV8_CRYPTO_LIB_PATH unset FLEXRAN_SDK unset LIBMUSDK_PATH - unset LIBSSO_SNOW3G_PATH unset LIBSSO_KASUMI_PATH unset PQOS_INSTALL_PATH } @@ -165,7 +163,7 @@ config () # sed -ri 's,(PMD_AESNI_GCM=)n,\1y,' $1/.config test "$DPDK_DEP_IPSEC_MB" != y || \ sed -ri 's,(PMD_ZUC=)n,\1y,' $1/.config - test -z "$LIBSSO_SNOW3G_PATH" || \ + test "$DPDK_DEP_IPSEC_MB" != y || \ sed -ri 's,(PMD_SNOW3G=)n,\1y,' $1/.config test -z "$LIBSSO_KASUMI_PATH" || \ sed -ri 's,(PMD_KASUMI=)n,\1y,' $1/.config diff --git a/doc/guides/cryptodevs/snow3g.rst b/doc/guides/cryptodevs/snow3g.rst index 7cba712c1..d45bcad67 100644 --- a/doc/guides/cryptodevs/snow3g.rst +++ b/doc/guides/cryptodevs/snow3g.rst @@ -1,12 +1,12 @@ .. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2016 Intel Corporation. + Copyright(c) 2016-2019 Intel Corporation. SNOW 3G Crypto Poll Mode Driver =============================== -The SNOW 3G PMD (**librte_pmd_snow3g**) provides poll mode crypto driver -support for utilizing Intel Libsso library, which implements F8 and F9 functions -for SNOW 3G UEA2 cipher and UIA2 hash algorithms. +The SNOW3G PMD (**librte_snow3g_zuc**) provides poll mode crypto driver support for +utilizing `Intel IPSec Multi-buffer library `_ +which implements F8 and F8 functions for SNOW 3G UEA2 cipher and UIA2 hash algorithms. Features -------- @@ -32,26 +32,33 @@ Limitations Installation ------------ -To build DPDK with the SNOW3G_PMD the user is required to download -the export controlled ``libsso_snow3g`` library, by registering in -`Intel Resource & Design Center `_. -Once approval has been granted, the user needs to search for -*Snow3G F8 F9 3GPP cryptographic algorithms Software Library* to download the -library or directly through this `link `_. +To build DPDK with the SNOW3G_PMD the user is required to download the multi-buffer +library from `here `_ +and compile it on their user system before building DPDK. +The latest version of the library supported by this PMD is v0.53, which +can be downloaded from ``_. + After downloading the library, the user needs to unpack and compile it -on their system before building DPDK:: +on their system before building DPDK: + +.. code-block:: console + + make + make install - make snow3G +As a reference, the following table shows a mapping between the past DPDK versions +and the external crypto libraries supported by them: -**Note**: When encrypting with SNOW3G UEA2, by default the library -encrypts blocks of 4 bytes, regardless the number of bytes to -be encrypted provided (which leads to a possible buffer overflow). -To avoid this situation, it is necessary not to pass -3GPP_SAFE_BUFFERS as a compilation flag. -For this, in the Makefile of the library, make sure that this flag -is commented out.:: +.. _table_zuc_versions: - #EXTRA_CFLAGS += -D_3GPP_SAFE_BUFFERS +.. table:: DPDK and external crypto library version compatibility + + ============= ================================ + DPDK version Crypto library version + ============= ================================ + 16.04 - 19.11 LibSSO SNOW3G + 20.02+ Multi-buffer library 0.53 + ============= ================================ Initialization @@ -59,12 +66,15 @@ Initialization In order to enable this virtual crypto PMD, user must: -* Export the environmental variable LIBSSO_SNOW3G_PATH with the path where - the library was extracted (snow3g folder). +* Build the multi buffer library (explained in Installation section). + +* Build DPDK as follows: -* Build the LIBSSO_SNOW3G library (explained in Installation section). +.. code-block:: console -* Set CONFIG_RTE_LIBRTE_PMD_SNOW3G=y in config/common_base. + make config T=x86_64-native-linux-gcc + sed -i 's,\(CONFIG_RTE_LIBRTE_PMD_SNOW3G\)=n,\1=y,' build/.config + make To use the PMD in an application, user must: diff --git a/drivers/crypto/snow3g/Makefile b/drivers/crypto/snow3g/Makefile index ee5027d0c..9c2f59d36 100644 --- a/drivers/crypto/snow3g/Makefile +++ b/drivers/crypto/snow3g/Makefile @@ -1,14 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2016 Intel Corporation +# Copyright(c) 2016-2019 Intel Corporation include $(RTE_SDK)/mk/rte.vars.mk -ifneq ($(MAKECMDGOALS),clean) -ifeq ($(LIBSSO_SNOW3G_PATH),) -$(error "Please define LIBSSO_SNOW3G_PATH environment variable") -endif -endif - # library name LIB = librte_pmd_snow3g.a @@ -23,14 +17,27 @@ LIBABIVER := 1 EXPORT_MAP := rte_pmd_snow3g_version.map # external library dependencies -CFLAGS += -I$(LIBSSO_SNOW3G_PATH) -CFLAGS += -I$(LIBSSO_SNOW3G_PATH)/include -CFLAGS += -I$(LIBSSO_SNOW3G_PATH)/build -LDLIBS += -L$(LIBSSO_SNOW3G_PATH)/build -lsso_snow3g +LDLIBS += -lIPSec_MB LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_cryptodev LDLIBS += -lrte_bus_vdev +IMB_HDR = $(shell echo '\#include ' | \ + $(CC) -E $(EXTRA_CFLAGS) - | grep 'intel-ipsec-mb.h' | \ + head -n1 | cut -d'"' -f2) + +# Detect library version +IMB_VERSION = $(shell grep -e "IMB_VERSION_STR" $(IMB_HDR) | cut -d'"' -f2) +IMB_VERSION_NUM = $(shell grep -e "IMB_VERSION_NUM" $(IMB_HDR) | cut -d' ' -f3) + +ifeq ($(IMB_VERSION),) +$(error "IPSec_MB version >= 0.53 is required") +endif + +ifeq ($(shell expr $(IMB_VERSION_NUM) \< 0x3400), 1) +$(error "IPSec_MB version >= 0.53 is required") +endif + # library source files SRCS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G) += rte_snow3g_pmd.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G) += rte_snow3g_pmd_ops.c diff --git a/drivers/crypto/snow3g/meson.build b/drivers/crypto/snow3g/meson.build new file mode 100644 index 000000000..e04138e92 --- /dev/null +++ b/drivers/crypto/snow3g/meson.build @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2018-2020 Intel Corporation + +IMB_required_ver = '0.53.0' +lib = cc.find_library('IPSec_MB', required: false) +if not lib.found() + build = false +else + ext_deps += lib + # version comes with quotes, so we split based on " and take the middle + imb_ver = cc.get_define('IMB_VERSION_STR', + prefix : '#include').split('"')[1] + + if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver)) + build = false + endif + +endif + +sources = files('rte_snow3g_pmd.c', 'rte_snow3g_pmd_ops.c') +deps += ['bus_vdev'] diff --git a/drivers/crypto/snow3g/rte_snow3g_pmd.c b/drivers/crypto/snow3g/rte_snow3g_pmd.c index a17536b77..5abf7e090 100644 --- a/drivers/crypto/snow3g/rte_snow3g_pmd.c +++ b/drivers/crypto/snow3g/rte_snow3g_pmd.c @@ -53,7 +53,7 @@ snow3g_get_mode(const struct rte_crypto_sym_xform *xform) /** Parse crypto xform chain and set private session parameters. */ int -snow3g_set_session_parameters(struct snow3g_session *sess, +snow3g_set_session_parameters(MB_MGR *mgr, struct snow3g_session *sess, const struct rte_crypto_sym_xform *xform) { const struct rte_crypto_sym_xform *auth_xform = NULL; @@ -95,8 +95,8 @@ snow3g_set_session_parameters(struct snow3g_session *sess, sess->cipher_iv_offset = cipher_xform->cipher.iv.offset; /* Initialize key */ - sso_snow3g_init_key_sched(cipher_xform->cipher.key.data, - &sess->pKeySched_cipher); + IMB_SNOW3G_INIT_KEY_SCHED(mgr, cipher_xform->cipher.key.data, + &sess->pKeySched_cipher); } if (auth_xform) { @@ -118,11 +118,10 @@ snow3g_set_session_parameters(struct snow3g_session *sess, sess->auth_iv_offset = auth_xform->auth.iv.offset; /* Initialize key */ - sso_snow3g_init_key_sched(auth_xform->auth.key.data, - &sess->pKeySched_hash); + IMB_SNOW3G_INIT_KEY_SCHED(mgr, auth_xform->auth.key.data, + &sess->pKeySched_hash); } - sess->op = mode; return 0; @@ -152,7 +151,7 @@ snow3g_get_session(struct snow3g_qp *qp, struct rte_crypto_op *op) sess = (struct snow3g_session *)_sess_private_data; - if (unlikely(snow3g_set_session_parameters(sess, + if (unlikely(snow3g_set_session_parameters(qp->mgr, sess, op->sym->xform) != 0)) { rte_mempool_put(qp->sess_mp, _sess); rte_mempool_put(qp->sess_mp, _sess_private_data); @@ -172,14 +171,15 @@ snow3g_get_session(struct snow3g_qp *qp, struct rte_crypto_op *op) /** Encrypt/decrypt mbufs with same cipher key. */ static uint8_t -process_snow3g_cipher_op(struct rte_crypto_op **ops, +process_snow3g_cipher_op(struct snow3g_qp *qp, struct rte_crypto_op **ops, struct snow3g_session *session, uint8_t num_ops) { unsigned i; uint8_t processed_ops = 0; - uint8_t *src[SNOW3G_MAX_BURST], *dst[SNOW3G_MAX_BURST]; - uint8_t *iv[SNOW3G_MAX_BURST]; + const void *src[SNOW3G_MAX_BURST]; + void *dst[SNOW3G_MAX_BURST]; + const void *iv[SNOW3G_MAX_BURST]; uint32_t num_bytes[SNOW3G_MAX_BURST]; for (i = 0; i < num_ops; i++) { @@ -197,15 +197,16 @@ process_snow3g_cipher_op(struct rte_crypto_op **ops, processed_ops++; } - sso_snow3g_f8_n_buffer(&session->pKeySched_cipher, iv, src, dst, - num_bytes, processed_ops); + IMB_SNOW3G_F8_N_BUFFER(qp->mgr, &session->pKeySched_cipher, iv, + src, dst, num_bytes, processed_ops); return processed_ops; } /** Encrypt/decrypt mbuf (bit level function). */ static uint8_t -process_snow3g_cipher_op_bit(struct rte_crypto_op *op, +process_snow3g_cipher_op_bit(struct snow3g_qp *qp, + struct rte_crypto_op *op, struct snow3g_session *session) { uint8_t *src, *dst; @@ -224,7 +225,7 @@ process_snow3g_cipher_op_bit(struct rte_crypto_op *op, session->cipher_iv_offset); length_in_bits = op->sym->cipher.data.length; - sso_snow3g_f8_1_buffer_bit(&session->pKeySched_cipher, iv, + IMB_SNOW3G_F8_1_BUFFER_BIT(qp->mgr, &session->pKeySched_cipher, iv, src, dst, length_in_bits, offset_in_bits); return 1; @@ -260,9 +261,9 @@ process_snow3g_hash_op(struct snow3g_qp *qp, struct rte_crypto_op **ops, if (session->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) { dst = qp->temp_digest; - sso_snow3g_f9_1_buffer(&session->pKeySched_hash, - iv, src, - length_in_bits, dst); + IMB_SNOW3G_F9_1_BUFFER(qp->mgr, + &session->pKeySched_hash, + iv, src, length_in_bits, dst); /* Verify digest. */ if (memcmp(dst, ops[i]->sym->auth.digest.data, SNOW3G_DIGEST_LENGTH) != 0) @@ -270,9 +271,9 @@ process_snow3g_hash_op(struct snow3g_qp *qp, struct rte_crypto_op **ops, } else { dst = ops[i]->sym->auth.digest.data; - sso_snow3g_f9_1_buffer(&session->pKeySched_hash, - iv, src, - length_in_bits, dst); + IMB_SNOW3G_F9_1_BUFFER(qp->mgr, + &session->pKeySched_hash, + iv, src, length_in_bits, dst); } processed_ops++; } @@ -306,7 +307,7 @@ process_ops(struct rte_crypto_op **ops, struct snow3g_session *session, switch (session->op) { case SNOW3G_OP_ONLY_CIPHER: - processed_ops = process_snow3g_cipher_op(ops, + processed_ops = process_snow3g_cipher_op(qp, ops, session, num_ops); break; case SNOW3G_OP_ONLY_AUTH: @@ -314,14 +315,14 @@ process_ops(struct rte_crypto_op **ops, struct snow3g_session *session, num_ops); break; case SNOW3G_OP_CIPHER_AUTH: - processed_ops = process_snow3g_cipher_op(ops, session, + processed_ops = process_snow3g_cipher_op(qp, ops, session, num_ops); process_snow3g_hash_op(qp, ops, session, processed_ops); break; case SNOW3G_OP_AUTH_CIPHER: processed_ops = process_snow3g_hash_op(qp, ops, session, num_ops); - process_snow3g_cipher_op(ops, session, processed_ops); + process_snow3g_cipher_op(qp, ops, session, processed_ops); break; default: /* Operation not supported. */ @@ -363,21 +364,21 @@ process_op_bit(struct rte_crypto_op *op, struct snow3g_session *session, switch (session->op) { case SNOW3G_OP_ONLY_CIPHER: - processed_op = process_snow3g_cipher_op_bit(op, + processed_op = process_snow3g_cipher_op_bit(qp, op, session); break; case SNOW3G_OP_ONLY_AUTH: processed_op = process_snow3g_hash_op(qp, &op, session, 1); break; case SNOW3G_OP_CIPHER_AUTH: - processed_op = process_snow3g_cipher_op_bit(op, session); + processed_op = process_snow3g_cipher_op_bit(qp, op, session); if (processed_op == 1) process_snow3g_hash_op(qp, &op, session, 1); break; case SNOW3G_OP_AUTH_CIPHER: processed_op = process_snow3g_hash_op(qp, &op, session, 1); if (processed_op == 1) - process_snow3g_cipher_op_bit(op, session); + process_snow3g_cipher_op_bit(qp, op, session); break; default: /* Operation not supported. */ @@ -533,7 +534,7 @@ cryptodev_snow3g_create(const char *name, { struct rte_cryptodev *dev; struct snow3g_private *internals; - uint64_t cpu_flags = RTE_CRYPTODEV_FF_CPU_SSE; + MB_MGR *mgr; dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params); if (dev == NULL) { @@ -549,10 +550,25 @@ cryptodev_snow3g_create(const char *name, dev->enqueue_burst = snow3g_pmd_enqueue_burst; dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | - RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - cpu_flags; + RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING; + + mgr = alloc_mb_mgr(0); + if (mgr == NULL) + return -ENOMEM; + + if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) { + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; + init_mb_mgr_avx2(mgr); + } else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) { + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; + init_mb_mgr_avx(mgr); + } else { + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; + init_mb_mgr_sse(mgr); + } internals = dev->data->dev_private; + internals->mgr = mgr; internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs; @@ -592,6 +608,7 @@ cryptodev_snow3g_remove(struct rte_vdev_device *vdev) { struct rte_cryptodev *cryptodev; const char *name; + struct snow3g_private *internals; name = rte_vdev_device_name(vdev); if (name == NULL) @@ -601,6 +618,10 @@ cryptodev_snow3g_remove(struct rte_vdev_device *vdev) if (cryptodev == NULL) return -ENODEV; + internals = cryptodev->data->dev_private; + + free_mb_mgr(internals->mgr); + return rte_cryptodev_pmd_destroy(cryptodev); } diff --git a/drivers/crypto/snow3g/rte_snow3g_pmd_ops.c b/drivers/crypto/snow3g/rte_snow3g_pmd_ops.c index cfbc9522a..414029b5a 100644 --- a/drivers/crypto/snow3g/rte_snow3g_pmd_ops.c +++ b/drivers/crypto/snow3g/rte_snow3g_pmd_ops.c @@ -196,6 +196,7 @@ snow3g_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, int socket_id, struct rte_mempool *session_pool) { struct snow3g_qp *qp = NULL; + struct snow3g_private *internals = dev->data->dev_private; /* Free memory prior to re-allocation if needed. */ if (dev->data->queue_pairs[qp_id] != NULL) @@ -218,6 +219,7 @@ snow3g_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, if (qp->processed_ops == NULL) goto qp_setup_cleanup; + qp->mgr = internals->mgr; qp->sess_mp = session_pool; memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); @@ -247,13 +249,14 @@ snow3g_pmd_sym_session_get_size(struct rte_cryptodev *dev __rte_unused) /** Configure a SNOW 3G session from a crypto xform chain */ static int -snow3g_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused, +snow3g_pmd_sym_session_configure(struct rte_cryptodev *dev, struct rte_crypto_sym_xform *xform, struct rte_cryptodev_sym_session *sess, struct rte_mempool *mempool) { void *sess_private_data; int ret; + struct snow3g_private *internals = dev->data->dev_private; if (unlikely(sess == NULL)) { SNOW3G_LOG(ERR, "invalid session struct"); @@ -266,7 +269,8 @@ snow3g_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused, return -ENOMEM; } - ret = snow3g_set_session_parameters(sess_private_data, xform); + ret = snow3g_set_session_parameters(internals->mgr, + sess_private_data, xform); if (ret != 0) { SNOW3G_LOG(ERR, "failed configure session parameters"); diff --git a/drivers/crypto/snow3g/rte_snow3g_pmd_private.h b/drivers/crypto/snow3g/rte_snow3g_pmd_private.h index b7807b621..670140144 100644 --- a/drivers/crypto/snow3g/rte_snow3g_pmd_private.h +++ b/drivers/crypto/snow3g/rte_snow3g_pmd_private.h @@ -5,7 +5,7 @@ #ifndef _RTE_SNOW3G_PMD_PRIVATE_H_ #define _RTE_SNOW3G_PMD_PRIVATE_H_ -#include +#include #define CRYPTODEV_NAME_SNOW3G_PMD crypto_snow3g /**< SNOW 3G PMD device name */ @@ -24,6 +24,8 @@ int snow3g_logtype_driver; struct snow3g_private { unsigned max_nb_queue_pairs; /**< Max number of queue pairs supported by device */ + MB_MGR *mgr; + /**< Multi-buffer instance */ }; /** SNOW 3G buffer queue pair */ @@ -43,6 +45,8 @@ struct snow3g_qp { * by the driver when verifying a digest provided * by the user (using authentication verify operation) */ + MB_MGR *mgr; + /**< Multi-buffer instance */ } __rte_cache_aligned; enum snow3g_operation { @@ -57,15 +61,15 @@ enum snow3g_operation { struct snow3g_session { enum snow3g_operation op; enum rte_crypto_auth_operation auth_op; - sso_snow3g_key_schedule_t pKeySched_cipher; - sso_snow3g_key_schedule_t pKeySched_hash; + snow3g_key_schedule_t pKeySched_cipher; + snow3g_key_schedule_t pKeySched_hash; uint16_t cipher_iv_offset; uint16_t auth_iv_offset; } __rte_cache_aligned; extern int -snow3g_set_session_parameters(struct snow3g_session *sess, +snow3g_set_session_parameters(MB_MGR *mgr, struct snow3g_session *sess, const struct rte_crypto_sym_xform *xform); diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 7c04ee490..b33603ef5 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -229,7 +229,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_PMD_QAT),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_QAT_SYM) += -lrte_pmd_qat -lcrypto endif # CONFIG_RTE_LIBRTE_PMD_QAT _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G) += -lrte_pmd_snow3g -_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G) += -L$(LIBSSO_SNOW3G_PATH)/build -lsso_snow3g +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G) += -lIPSec_MB _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI) += -lrte_pmd_kasumi _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI) += -L$(LIBSSO_KASUMI_PATH)/build -lsso_kasumi _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ZUC) += -lrte_pmd_zuc From patchwork Thu Mar 5 15:34:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "De Lara Guarch, Pablo" X-Patchwork-Id: 66306 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id CA32AA0564; Thu, 5 Mar 2020 18:10:14 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 08BFB1BFF7; Thu, 5 Mar 2020 18:09:44 +0100 (CET) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 88BFE1BFCE for ; Thu, 5 Mar 2020 18:09:39 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Mar 2020 09:09:39 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,518,1574150400"; d="scan'208";a="275173391" Received: from silpixa00400565.ir.intel.com (HELO silpixa00400565.ger.corp.intel.com) ([10.237.222.249]) by fmsmga002.fm.intel.com with ESMTP; 05 Mar 2020 09:09:37 -0800 From: Pablo de Lara To: dev@dpdk.org Cc: Pablo de Lara Date: Thu, 5 Mar 2020 15:34:52 +0000 Message-Id: <20200305153454.724874-4-pablo.de.lara.guarch@intel.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200305153454.724874-1-pablo.de.lara.guarch@intel.com> References: <20200305153454.724874-1-pablo.de.lara.guarch@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [RFC PATCH 3/5] crypto/kasumi: use IPSec MB library v0.53 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Link against Intel IPSec Multi-buffer library, which added support for KASUMI-F8 and KASUMI-F9 from version v0.53, moving from libSSO KASUMI library. Signed-off-by: Pablo de Lara --- devtools/test-build.sh | 4 +- doc/guides/cryptodevs/kasumi.rst | 62 ++++++++------- drivers/crypto/kasumi/Makefile | 26 +++--- drivers/crypto/kasumi/meson.build | 11 ++- drivers/crypto/kasumi/rte_kasumi_pmd.c | 79 +++++++++++-------- drivers/crypto/kasumi/rte_kasumi_pmd_ops.c | 8 +- .../crypto/kasumi/rte_kasumi_pmd_private.h | 12 ++- mk/rte.app.mk | 2 +- 8 files changed, 120 insertions(+), 84 deletions(-) diff --git a/devtools/test-build.sh b/devtools/test-build.sh index 0c62c3950..73b4bdd1a 100755 --- a/devtools/test-build.sh +++ b/devtools/test-build.sh @@ -23,7 +23,6 @@ default_path=$PATH # - DPDK_NOTIFY (notify-send) # - FLEXRAN_SDK # - LIBMUSDK_PATH -# - LIBSSO_KASUMI_PATH . $(dirname $(readlink -e $0))/load-devel-config print_usage () { @@ -107,7 +106,6 @@ reset_env () unset ARMV8_CRYPTO_LIB_PATH unset FLEXRAN_SDK unset LIBMUSDK_PATH - unset LIBSSO_KASUMI_PATH unset PQOS_INSTALL_PATH } @@ -165,7 +163,7 @@ config () # sed -ri 's,(PMD_ZUC=)n,\1y,' $1/.config test "$DPDK_DEP_IPSEC_MB" != y || \ sed -ri 's,(PMD_SNOW3G=)n,\1y,' $1/.config - test -z "$LIBSSO_KASUMI_PATH" || \ + test "$DPDK_DEP_IPSEC_MB" != y || \ sed -ri 's,(PMD_KASUMI=)n,\1y,' $1/.config test "$DPDK_DEP_SSL" != y || \ sed -ri 's,(PMD_CCP=)n,\1y,' $1/.config diff --git a/doc/guides/cryptodevs/kasumi.rst b/doc/guides/cryptodevs/kasumi.rst index 2265eee4e..6c86fe264 100644 --- a/doc/guides/cryptodevs/kasumi.rst +++ b/doc/guides/cryptodevs/kasumi.rst @@ -1,12 +1,12 @@ .. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2016 Intel Corporation. + Copyright(c) 2016-2019 Intel Corporation. KASUMI Crypto Poll Mode Driver =============================== -The KASUMI PMD (**librte_pmd_kasumi**) provides poll mode crypto driver -support for utilizing Intel Libsso library, which implements F8 and F9 functions -for KASUMI UEA1 cipher and UIA1 hash algorithms. +The KASUMI PMD (**librte_pmd_kasumi**) provides poll mode crypto driver support for +utilizing `Intel IPSec Multi-buffer library `_ +which implements F8 and F9 functions for KASUMI UEA1 cipher and UIA1 hash algorithms. Features -------- @@ -33,33 +33,33 @@ Limitations Installation ------------ -To build DPDK with the KASUMI_PMD the user is required to download -the export controlled ``libsso_kasumi`` library, by registering in -`Intel Resource & Design Center `_. -Once approval has been granted, the user needs to search for -*Kasumi F8 F9 3GPP cryptographic algorithms Software Library* to download the -library or directly through this `link `_. +To build DPDK with the KASUMI_PMD the user is required to download the multi-buffer +library from `here `_ +and compile it on their user system before building DPDK. +The latest version of the library supported by this PMD is v0.53, which +can be downloaded from ``_. + After downloading the library, the user needs to unpack and compile it -on their system before building DPDK:: +on their system before building DPDK: + +.. code-block:: console - make + make + make install -**Note**: When encrypting with KASUMI F8, by default the library -encrypts full blocks of 8 bytes, regardless the number of bytes to -be encrypted provided (which leads to a possible buffer overflow). -To avoid this situation, it is necessary not to pass -3GPP_SAFE_BUFFERS as a compilation flag. -Also, this is required when using chained operations -(cipher-then-auth/auth-then-cipher). -For this, in the Makefile of the library, make sure that this flag -is commented out:: +As a reference, the following table shows a mapping between the past DPDK versions +and the external crypto libraries supported by them: - #EXTRA_CFLAGS += -D_3GPP_SAFE_BUFFERS +.. _table_kasumi_versions: -**Note**: To build the PMD as a shared library, the libsso_kasumi -library must be built as follows:: +.. table:: DPDK and external crypto library version compatibility - make KASUMI_CFLAGS=-DKASUMI_C + ============= ================================ + DPDK version Crypto library version + ============= ================================ + 16.11 - 19.11 LibSSO KASUMI + 20.02+ Multi-buffer library 0.53 + ============= ================================ Initialization @@ -67,12 +67,16 @@ Initialization In order to enable this virtual crypto PMD, user must: -* Export the environmental variable LIBSSO_KASUMI_PATH with the path where - the library was extracted (kasumi folder). +* Build the multi buffer library (explained in Installation section). + +* Build DPDK as follows: + +.. code-block:: console -* Build the LIBSSO library (explained in Installation section). + make config T=x86_64-native-linux-gcc + sed -i 's,\(CONFIG_RTE_LIBRTE_PMD_KASUMI\)=n,\1=y,' build/.config + make -* Set CONFIG_RTE_LIBRTE_PMD_KASUMI=y in config/common_base. To use the PMD in an application, user must: diff --git a/drivers/crypto/kasumi/Makefile b/drivers/crypto/kasumi/Makefile index cafe94986..51f31d0aa 100644 --- a/drivers/crypto/kasumi/Makefile +++ b/drivers/crypto/kasumi/Makefile @@ -3,12 +3,6 @@ include $(RTE_SDK)/mk/rte.vars.mk -ifneq ($(MAKECMDGOALS),clean) -ifeq ($(LIBSSO_KASUMI_PATH),) -$(error "Please define LIBSSO_KASUMI_PATH environment variable") -endif -endif - # library name LIB = librte_pmd_kasumi.a @@ -23,14 +17,26 @@ LIBABIVER := 1 EXPORT_MAP := rte_pmd_kasumi_version.map # external library dependencies -CFLAGS += -I$(LIBSSO_KASUMI_PATH) -CFLAGS += -I$(LIBSSO_KASUMI_PATH)/include -CFLAGS += -I$(LIBSSO_KASUMI_PATH)/build -LDLIBS += -L$(LIBSSO_KASUMI_PATH)/build -lsso_kasumi +LDLIBS += -lIPSec_MB LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_cryptodev LDLIBS += -lrte_bus_vdev +IMB_HDR = $(shell echo '\#include ' | \ + $(CC) -E $(EXTRA_CFLAGS) - | grep 'intel-ipsec-mb.h' | \ + head -n1 | cut -d'"' -f2) + +# Detect library version +IMB_VERSION = $(shell grep -e "IMB_VERSION_STR" $(IMB_HDR) | cut -d'"' -f2) +IMB_VERSION_NUM = $(shell grep -e "IMB_VERSION_NUM" $(IMB_HDR) | cut -d' ' -f3) + +ifeq ($(IMB_VERSION),) +$(error "IPSec_MB version >= 0.53 is required") +endif + +ifeq ($(shell expr $(IMB_VERSION_NUM) \< 0x3400), 1) +$(error "IPSec_MB version >= 0.53 is required") +endif # library source files SRCS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI) += rte_kasumi_pmd.c SRCS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI) += rte_kasumi_pmd_ops.c diff --git a/drivers/crypto/kasumi/meson.build b/drivers/crypto/kasumi/meson.build index a09b0e251..bda9a4052 100644 --- a/drivers/crypto/kasumi/meson.build +++ b/drivers/crypto/kasumi/meson.build @@ -1,11 +1,20 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2018 Intel Corporation -lib = cc.find_library('libsso_kasumi', required: false) +IMB_required_ver = '0.53.0' +lib = cc.find_library('IPSec_MB', required: false) if not lib.found() build = false else ext_deps += lib + # version comes with quotes, so we split based on " and take the middle + imb_ver = cc.get_define('IMB_VERSION_STR', + prefix : '#include').split('"')[1] + + if (imb_ver == '') or (imb_ver.version_compare('<' + IMB_required_ver)) + build = false + endif + endif sources = files('rte_kasumi_pmd.c', 'rte_kasumi_pmd_ops.c') diff --git a/drivers/crypto/kasumi/rte_kasumi_pmd.c b/drivers/crypto/kasumi/rte_kasumi_pmd.c index 239a1cf44..037abf710 100644 --- a/drivers/crypto/kasumi/rte_kasumi_pmd.c +++ b/drivers/crypto/kasumi/rte_kasumi_pmd.c @@ -54,7 +54,7 @@ kasumi_get_mode(const struct rte_crypto_sym_xform *xform) /** Parse crypto xform chain and set private session parameters. */ int -kasumi_set_session_parameters(struct kasumi_session *sess, +kasumi_set_session_parameters(MB_MGR *mgr, struct kasumi_session *sess, const struct rte_crypto_sym_xform *xform) { const struct rte_crypto_sym_xform *auth_xform = NULL; @@ -97,7 +97,7 @@ kasumi_set_session_parameters(struct kasumi_session *sess, } /* Initialize key */ - sso_kasumi_init_f8_key_sched(cipher_xform->cipher.key.data, + IMB_KASUMI_INIT_F8_KEY_SCHED(mgr, cipher_xform->cipher.key.data, &sess->pKeySched_cipher); } @@ -116,7 +116,7 @@ kasumi_set_session_parameters(struct kasumi_session *sess, sess->auth_op = auth_xform->auth.op; /* Initialize key */ - sso_kasumi_init_f9_key_sched(auth_xform->auth.key.data, + IMB_KASUMI_INIT_F9_KEY_SCHED(mgr, auth_xform->auth.key.data, &sess->pKeySched_hash); } @@ -150,7 +150,7 @@ kasumi_get_session(struct kasumi_qp *qp, struct rte_crypto_op *op) sess = (struct kasumi_session *)_sess_private_data; - if (unlikely(kasumi_set_session_parameters(sess, + if (unlikely(kasumi_set_session_parameters(qp->mgr, sess, op->sym->xform) != 0)) { rte_mempool_put(qp->sess_mp, _sess); rte_mempool_put(qp->sess_mp, _sess_private_data); @@ -169,13 +169,13 @@ kasumi_get_session(struct kasumi_qp *qp, struct rte_crypto_op *op) /** Encrypt/decrypt mbufs with same cipher key. */ static uint8_t -process_kasumi_cipher_op(struct rte_crypto_op **ops, - struct kasumi_session *session, - uint8_t num_ops) +process_kasumi_cipher_op(struct kasumi_qp *qp, struct rte_crypto_op **ops, + struct kasumi_session *session, uint8_t num_ops) { unsigned i; uint8_t processed_ops = 0; - uint8_t *src[num_ops], *dst[num_ops]; + const void *src[num_ops]; + void *dst[num_ops]; uint8_t *iv_ptr; uint64_t iv[num_ops]; uint32_t num_bytes[num_ops]; @@ -197,7 +197,7 @@ process_kasumi_cipher_op(struct rte_crypto_op **ops, } if (processed_ops != 0) - sso_kasumi_f8_n_buffer(&session->pKeySched_cipher, iv, + IMB_KASUMI_F8_N_BUFFER(qp->mgr, &session->pKeySched_cipher, iv, src, dst, num_bytes, processed_ops); return processed_ops; @@ -205,7 +205,7 @@ process_kasumi_cipher_op(struct rte_crypto_op **ops, /** Encrypt/decrypt mbuf (bit level function). */ static uint8_t -process_kasumi_cipher_op_bit(struct rte_crypto_op *op, +process_kasumi_cipher_op_bit(struct kasumi_qp *qp, struct rte_crypto_op *op, struct kasumi_session *session) { uint8_t *src, *dst; @@ -215,18 +215,16 @@ process_kasumi_cipher_op_bit(struct rte_crypto_op *op, offset_in_bits = op->sym->cipher.data.offset; src = rte_pktmbuf_mtod(op->sym->m_src, uint8_t *); - if (op->sym->m_dst == NULL) { - op->status = RTE_CRYPTO_OP_STATUS_INVALID_ARGS; - KASUMI_LOG(ERR, "bit-level in-place not supported"); - return 0; - } - dst = rte_pktmbuf_mtod(op->sym->m_dst, uint8_t *); + if (op->sym->m_dst == NULL) + dst = src; + else + dst = rte_pktmbuf_mtod(op->sym->m_dst, uint8_t *); iv_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, session->cipher_iv_offset); iv = *((uint64_t *)(iv_ptr)); length_in_bits = op->sym->cipher.data.length; - sso_kasumi_f8_1_buffer_bit(&session->pKeySched_cipher, iv, + IMB_KASUMI_F8_1_BUFFER_BIT(qp->mgr, &session->pKeySched_cipher, iv, src, dst, length_in_bits, offset_in_bits); return 1; @@ -261,7 +259,8 @@ process_kasumi_hash_op(struct kasumi_qp *qp, struct rte_crypto_op **ops, if (session->auth_op == RTE_CRYPTO_AUTH_OP_VERIFY) { dst = qp->temp_digest; - sso_kasumi_f9_1_buffer(&session->pKeySched_hash, src, + IMB_KASUMI_F9_1_BUFFER(qp->mgr, + &session->pKeySched_hash, src, num_bytes, dst); /* Verify digest. */ @@ -271,7 +270,8 @@ process_kasumi_hash_op(struct kasumi_qp *qp, struct rte_crypto_op **ops, } else { dst = ops[i]->sym->auth.digest.data; - sso_kasumi_f9_1_buffer(&session->pKeySched_hash, src, + IMB_KASUMI_F9_1_BUFFER(qp->mgr, + &session->pKeySched_hash, src, num_bytes, dst); } processed_ops++; @@ -291,7 +291,7 @@ process_ops(struct rte_crypto_op **ops, struct kasumi_session *session, switch (session->op) { case KASUMI_OP_ONLY_CIPHER: - processed_ops = process_kasumi_cipher_op(ops, + processed_ops = process_kasumi_cipher_op(qp, ops, session, num_ops); break; case KASUMI_OP_ONLY_AUTH: @@ -299,14 +299,14 @@ process_ops(struct rte_crypto_op **ops, struct kasumi_session *session, num_ops); break; case KASUMI_OP_CIPHER_AUTH: - processed_ops = process_kasumi_cipher_op(ops, session, + processed_ops = process_kasumi_cipher_op(qp, ops, session, num_ops); process_kasumi_hash_op(qp, ops, session, processed_ops); break; case KASUMI_OP_AUTH_CIPHER: processed_ops = process_kasumi_hash_op(qp, ops, session, num_ops); - process_kasumi_cipher_op(ops, session, processed_ops); + process_kasumi_cipher_op(qp, ops, session, processed_ops); break; default: /* Operation not supported. */ @@ -348,21 +348,21 @@ process_op_bit(struct rte_crypto_op *op, struct kasumi_session *session, switch (session->op) { case KASUMI_OP_ONLY_CIPHER: - processed_op = process_kasumi_cipher_op_bit(op, + processed_op = process_kasumi_cipher_op_bit(qp, op, session); break; case KASUMI_OP_ONLY_AUTH: processed_op = process_kasumi_hash_op(qp, &op, session, 1); break; case KASUMI_OP_CIPHER_AUTH: - processed_op = process_kasumi_cipher_op_bit(op, session); + processed_op = process_kasumi_cipher_op_bit(qp, op, session); if (processed_op == 1) process_kasumi_hash_op(qp, &op, session, 1); break; case KASUMI_OP_AUTH_CIPHER: processed_op = process_kasumi_hash_op(qp, &op, session, 1); if (processed_op == 1) - process_kasumi_cipher_op_bit(op, session); + process_kasumi_cipher_op_bit(qp, op, session); break; default: /* Operation not supported. */ @@ -531,7 +531,7 @@ cryptodev_kasumi_create(const char *name, { struct rte_cryptodev *dev; struct kasumi_private *internals; - uint64_t cpu_flags = 0; + MB_MGR *mgr; dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params); if (dev == NULL) { @@ -539,12 +539,6 @@ cryptodev_kasumi_create(const char *name, goto init_error; } - /* Check CPU for supported vector instruction set */ - if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) - cpu_flags |= RTE_CRYPTODEV_FF_CPU_AVX; - else - cpu_flags |= RTE_CRYPTODEV_FF_CPU_SSE; - dev->driver_id = cryptodev_driver_id; dev->dev_ops = rte_kasumi_pmd_ops; @@ -553,12 +547,24 @@ cryptodev_kasumi_create(const char *name, dev->enqueue_burst = kasumi_pmd_enqueue_burst; dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | - RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - cpu_flags; + RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING; + + mgr = alloc_mb_mgr(0); + if (mgr == NULL) + return -ENOMEM; + + if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) { + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; + init_mb_mgr_avx(mgr); + } else { + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; + init_mb_mgr_sse(mgr); + } internals = dev->data->dev_private; internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs; + internals->mgr = mgr; return 0; init_error: @@ -596,6 +602,7 @@ cryptodev_kasumi_remove(struct rte_vdev_device *vdev) { struct rte_cryptodev *cryptodev; const char *name; + struct kasumi_private *internals; name = rte_vdev_device_name(vdev); if (name == NULL) @@ -605,6 +612,10 @@ cryptodev_kasumi_remove(struct rte_vdev_device *vdev) if (cryptodev == NULL) return -ENODEV; + internals = cryptodev->data->dev_private; + + free_mb_mgr(internals->mgr); + return rte_cryptodev_pmd_destroy(cryptodev); } diff --git a/drivers/crypto/kasumi/rte_kasumi_pmd_ops.c b/drivers/crypto/kasumi/rte_kasumi_pmd_ops.c index 9e4bf1b52..2f30115b9 100644 --- a/drivers/crypto/kasumi/rte_kasumi_pmd_ops.c +++ b/drivers/crypto/kasumi/rte_kasumi_pmd_ops.c @@ -195,6 +195,7 @@ kasumi_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, int socket_id, struct rte_mempool *session_pool) { struct kasumi_qp *qp = NULL; + struct kasumi_private *internals = dev->data->dev_private; /* Free memory prior to re-allocation if needed. */ if (dev->data->queue_pairs[qp_id] != NULL) @@ -217,6 +218,7 @@ kasumi_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, if (qp->processed_ops == NULL) goto qp_setup_cleanup; + qp->mgr = internals->mgr; qp->sess_mp = session_pool; memset(&qp->qp_stats, 0, sizeof(qp->qp_stats)); @@ -245,13 +247,14 @@ kasumi_pmd_sym_session_get_size(struct rte_cryptodev *dev __rte_unused) /** Configure a KASUMI session from a crypto xform chain */ static int -kasumi_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused, +kasumi_pmd_sym_session_configure(struct rte_cryptodev *dev, struct rte_crypto_sym_xform *xform, struct rte_cryptodev_sym_session *sess, struct rte_mempool *mempool) { void *sess_private_data; int ret; + struct kasumi_private *internals = dev->data->dev_private; if (unlikely(sess == NULL)) { KASUMI_LOG(ERR, "invalid session struct"); @@ -264,7 +267,8 @@ kasumi_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused, return -ENOMEM; } - ret = kasumi_set_session_parameters(sess_private_data, xform); + ret = kasumi_set_session_parameters(internals->mgr, + sess_private_data, xform); if (ret != 0) { KASUMI_LOG(ERR, "failed configure session parameters"); diff --git a/drivers/crypto/kasumi/rte_kasumi_pmd_private.h b/drivers/crypto/kasumi/rte_kasumi_pmd_private.h index 488777ca8..3db52c03a 100644 --- a/drivers/crypto/kasumi/rte_kasumi_pmd_private.h +++ b/drivers/crypto/kasumi/rte_kasumi_pmd_private.h @@ -5,7 +5,7 @@ #ifndef _RTE_KASUMI_PMD_PRIVATE_H_ #define _RTE_KASUMI_PMD_PRIVATE_H_ -#include +#include #define CRYPTODEV_NAME_KASUMI_PMD crypto_kasumi /**< KASUMI PMD device name */ @@ -24,6 +24,8 @@ int kasumi_logtype_driver; struct kasumi_private { unsigned max_nb_queue_pairs; /**< Max number of queue pairs supported by device */ + MB_MGR *mgr; + /**< Multi-buffer instance */ }; /** KASUMI buffer queue pair */ @@ -43,6 +45,8 @@ struct kasumi_qp { * by the driver when verifying a digest provided * by the user (using authentication verify operation) */ + MB_MGR *mgr; + /**< Multi-buffer instance */ } __rte_cache_aligned; enum kasumi_operation { @@ -56,8 +60,8 @@ enum kasumi_operation { /** KASUMI private session structure */ struct kasumi_session { /* Keys have to be 16-byte aligned */ - sso_kasumi_key_sched_t pKeySched_cipher; - sso_kasumi_key_sched_t pKeySched_hash; + kasumi_key_sched_t pKeySched_cipher; + kasumi_key_sched_t pKeySched_hash; enum kasumi_operation op; enum rte_crypto_auth_operation auth_op; uint16_t cipher_iv_offset; @@ -65,7 +69,7 @@ struct kasumi_session { int -kasumi_set_session_parameters(struct kasumi_session *sess, +kasumi_set_session_parameters(MB_MGR *mgr, struct kasumi_session *sess, const struct rte_crypto_sym_xform *xform); diff --git a/mk/rte.app.mk b/mk/rte.app.mk index b33603ef5..0ec48482c 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -231,7 +231,7 @@ endif # CONFIG_RTE_LIBRTE_PMD_QAT _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G) += -lrte_pmd_snow3g _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SNOW3G) += -lIPSec_MB _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI) += -lrte_pmd_kasumi -_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI) += -L$(LIBSSO_KASUMI_PATH)/build -lsso_kasumi +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_KASUMI) += -lIPSec_MB _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ZUC) += -lrte_pmd_zuc _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ZUC) += -lIPSec_MB _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_ARMV8_CRYPTO) += -lrte_pmd_armv8 From patchwork Thu Mar 5 15:34:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "De Lara Guarch, Pablo" X-Patchwork-Id: 66307 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 58AFDA0564; Thu, 5 Mar 2020 18:10:26 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8C9531BFFE; Thu, 5 Mar 2020 18:09:45 +0100 (CET) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 460491BFE2 for ; Thu, 5 Mar 2020 18:09:41 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Mar 2020 09:09:40 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,518,1574150400"; d="scan'208";a="275173399" Received: from silpixa00400565.ir.intel.com (HELO silpixa00400565.ger.corp.intel.com) ([10.237.222.249]) by fmsmga002.fm.intel.com with ESMTP; 05 Mar 2020 09:09:39 -0800 From: Pablo de Lara To: dev@dpdk.org Cc: Pablo de Lara Date: Thu, 5 Mar 2020 15:34:53 +0000 Message-Id: <20200305153454.724874-5-pablo.de.lara.guarch@intel.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200305153454.724874-1-pablo.de.lara.guarch@intel.com> References: <20200305153454.724874-1-pablo.de.lara.guarch@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [RFC PATCH 4/5] crypto/aesni_mb: support IPSec MB library v0.53 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add support for underlying IPSec Multi-buffer library v0.53. Signed-off-by: Pablo de Lara --- drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c | 476 ++++++++++++------ .../crypto/aesni_mb/rte_aesni_mb_pmd_ops.c | 205 +++++--- .../aesni_mb/rte_aesni_mb_pmd_private.h | 30 +- 3 files changed, 489 insertions(+), 222 deletions(-) diff --git a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c index 83250e32c..9dfa89f71 100644 --- a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c +++ b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd.c @@ -35,7 +35,7 @@ typedef void (*aes_keyexp_t)(const void *key, void *enc_exp_keys, void *dec_exp_ static void calculate_auth_precomputes(hash_one_block_t one_block_hash, uint8_t *ipad, uint8_t *opad, - uint8_t *hkey, uint16_t hkey_len, + const uint8_t *hkey, uint16_t hkey_len, uint16_t blocksize) { unsigned i, length; @@ -85,6 +85,25 @@ aesni_mb_get_chain_order(const struct rte_crypto_sym_xform *xform) return AESNI_MB_OP_HASH_CIPHER; } +#if IMB_VERSION_NUM > IMB_VERSION(0, 52, 0) + if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) { + if (xform->aead.op == RTE_CRYPTO_AEAD_OP_ENCRYPT) { + /* + * CCM requires to hash first and cipher later + * when encrypting + */ + if (xform->aead.algo == RTE_CRYPTO_AEAD_AES_CCM) + return AESNI_MB_OP_AEAD_HASH_CIPHER; + else + return AESNI_MB_OP_AEAD_CIPHER_HASH; + } else { + if (xform->aead.algo == RTE_CRYPTO_AEAD_AES_CCM) + return AESNI_MB_OP_AEAD_CIPHER_HASH; + else + return AESNI_MB_OP_AEAD_HASH_CIPHER; + } + } +#else if (xform->type == RTE_CRYPTO_SYM_XFORM_AEAD) { if (xform->aead.algo == RTE_CRYPTO_AEAD_AES_CCM || xform->aead.algo == RTE_CRYPTO_AEAD_AES_GCM) { @@ -94,19 +113,21 @@ aesni_mb_get_chain_order(const struct rte_crypto_sym_xform *xform) return AESNI_MB_OP_AEAD_HASH_CIPHER; } } +#endif return AESNI_MB_OP_NOT_SUPPORTED; } /** Set session authentication parameters */ static int -aesni_mb_set_session_auth_parameters(const struct aesni_mb_op_fns *mb_ops, +aesni_mb_set_session_auth_parameters(const MB_MGR *mb_mgr, struct aesni_mb_session *sess, const struct rte_crypto_sym_xform *xform) { - hash_one_block_t hash_oneblock_fn; + hash_one_block_t hash_oneblock_fn = NULL; unsigned int key_larger_block_size = 0; uint8_t hashed_key[HMAC_MAX_BLOCK_SIZE] = { 0 }; + uint32_t auth_precompute = 1; if (xform == NULL) { sess->auth.algo = NULL_HASH; @@ -135,13 +156,16 @@ aesni_mb_set_session_auth_parameters(const struct aesni_mb_op_fns *mb_ops, return -EINVAL; } sess->auth.gen_digest_len = sess->auth.req_digest_len; - (*mb_ops->aux.keyexp.aes_xcbc)(xform->auth.key.data, + + IMB_AES_XCBC_KEYEXP(mb_mgr, xform->auth.key.data, sess->auth.xcbc.k1_expanded, sess->auth.xcbc.k2, sess->auth.xcbc.k3); return 0; } if (xform->auth.algo == RTE_CRYPTO_AUTH_AES_CMAC) { + uint32_t dust[4*15]; + sess->auth.algo = AES_CMAC; uint16_t cmac_digest_len = get_digest_byte_length(AES_CMAC); @@ -150,102 +174,144 @@ aesni_mb_set_session_auth_parameters(const struct aesni_mb_op_fns *mb_ops, AESNI_MB_LOG(ERR, "Invalid digest size\n"); return -EINVAL; } + if (sess->auth.req_digest_len < 4) + sess->auth.gen_digest_len = cmac_digest_len; + else + sess->auth.gen_digest_len = sess->auth.req_digest_len; + IMB_AES_KEYEXP_128(mb_mgr, xform->auth.key.data, + sess->auth.cmac.expkey, dust); + IMB_AES_CMAC_SUBKEY_GEN_128(mb_mgr, sess->auth.cmac.expkey, + sess->auth.cmac.skey1, sess->auth.cmac.skey2); + return 0; + } + + if (xform->auth.algo == RTE_CRYPTO_AUTH_AES_GMAC) { + if (xform->auth.op == RTE_CRYPTO_AUTH_OP_GENERATE) { + sess->cipher.direction = ENCRYPT; + sess->chain_order = CIPHER_HASH; + } else + sess->cipher.direction = DECRYPT; + + sess->auth.algo = AES_GMAC; /* - * Multi-buffer lib supports digest sizes from 4 to 16 bytes - * in version 0.50 and sizes of 12 and 16 bytes, - * in version 0.49. + * Multi-buffer lib supports 8, 12 and 16 bytes of digest. * If size requested is different, generate the full digest * (16 bytes) in a temporary location and then memcpy * the requested number of bytes. */ -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) - if (sess->auth.req_digest_len < 4) -#else - uint16_t cmac_trunc_digest_len = - get_truncated_digest_byte_length(AES_CMAC); - if (sess->auth.req_digest_len != cmac_digest_len && - sess->auth.req_digest_len != cmac_trunc_digest_len) -#endif - sess->auth.gen_digest_len = cmac_digest_len; - else + if (sess->auth.req_digest_len != 16 && + sess->auth.req_digest_len != 12 && + sess->auth.req_digest_len != 8) { + sess->auth.gen_digest_len = 16; + } else { sess->auth.gen_digest_len = sess->auth.req_digest_len; - (*mb_ops->aux.keyexp.aes_cmac_expkey)(xform->auth.key.data, - sess->auth.cmac.expkey); + } + sess->iv.length = xform->auth.iv.length; + sess->iv.offset = xform->auth.iv.offset; + + switch (xform->auth.key.length) { + case AES_128_BYTES: + IMB_AES128_GCM_PRE(mb_mgr, xform->auth.key.data, + &sess->cipher.gcm_key); + sess->cipher.key_length_in_bytes = AES_128_BYTES; + break; + case AES_192_BYTES: + IMB_AES192_GCM_PRE(mb_mgr, xform->auth.key.data, + &sess->cipher.gcm_key); + sess->cipher.key_length_in_bytes = AES_192_BYTES; + break; + case AES_256_BYTES: + IMB_AES256_GCM_PRE(mb_mgr, xform->auth.key.data, + &sess->cipher.gcm_key); + sess->cipher.key_length_in_bytes = AES_256_BYTES; + break; + default: + RTE_LOG(ERR, PMD, "failed to parse test type\n"); + return -EINVAL; + } - (*mb_ops->aux.keyexp.aes_cmac_subkey)(sess->auth.cmac.expkey, - sess->auth.cmac.skey1, sess->auth.cmac.skey2); return 0; } switch (xform->auth.algo) { case RTE_CRYPTO_AUTH_MD5_HMAC: sess->auth.algo = MD5; - hash_oneblock_fn = mb_ops->aux.one_block.md5; + hash_oneblock_fn = mb_mgr->md5_one_block; break; case RTE_CRYPTO_AUTH_SHA1_HMAC: sess->auth.algo = SHA1; - hash_oneblock_fn = mb_ops->aux.one_block.sha1; -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) + hash_oneblock_fn = mb_mgr->sha1_one_block; if (xform->auth.key.length > get_auth_algo_blocksize(SHA1)) { - mb_ops->aux.multi_block.sha1( + IMB_SHA1(mb_mgr, xform->auth.key.data, xform->auth.key.length, hashed_key); key_larger_block_size = 1; } -#endif + break; + case RTE_CRYPTO_AUTH_SHA1: + sess->auth.algo = PLAIN_SHA1; + auth_precompute = 0; break; case RTE_CRYPTO_AUTH_SHA224_HMAC: sess->auth.algo = SHA_224; - hash_oneblock_fn = mb_ops->aux.one_block.sha224; -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) + hash_oneblock_fn = mb_mgr->sha224_one_block; if (xform->auth.key.length > get_auth_algo_blocksize(SHA_224)) { - mb_ops->aux.multi_block.sha224( + IMB_SHA224(mb_mgr, xform->auth.key.data, xform->auth.key.length, hashed_key); key_larger_block_size = 1; } -#endif + break; + case RTE_CRYPTO_AUTH_SHA224: + sess->auth.algo = PLAIN_SHA_224; + auth_precompute = 0; break; case RTE_CRYPTO_AUTH_SHA256_HMAC: sess->auth.algo = SHA_256; - hash_oneblock_fn = mb_ops->aux.one_block.sha256; -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) + hash_oneblock_fn = mb_mgr->sha256_one_block; if (xform->auth.key.length > get_auth_algo_blocksize(SHA_256)) { - mb_ops->aux.multi_block.sha256( + IMB_SHA256(mb_mgr, xform->auth.key.data, xform->auth.key.length, hashed_key); key_larger_block_size = 1; } -#endif + break; + case RTE_CRYPTO_AUTH_SHA256: + sess->auth.algo = PLAIN_SHA_256; + auth_precompute = 0; break; case RTE_CRYPTO_AUTH_SHA384_HMAC: sess->auth.algo = SHA_384; - hash_oneblock_fn = mb_ops->aux.one_block.sha384; -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) + hash_oneblock_fn = mb_mgr->sha384_one_block; if (xform->auth.key.length > get_auth_algo_blocksize(SHA_384)) { - mb_ops->aux.multi_block.sha384( + IMB_SHA384(mb_mgr, xform->auth.key.data, xform->auth.key.length, hashed_key); key_larger_block_size = 1; } -#endif + break; + case RTE_CRYPTO_AUTH_SHA384: + sess->auth.algo = PLAIN_SHA_384; + auth_precompute = 0; break; case RTE_CRYPTO_AUTH_SHA512_HMAC: sess->auth.algo = SHA_512; - hash_oneblock_fn = mb_ops->aux.one_block.sha512; -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) + hash_oneblock_fn = mb_mgr->sha512_one_block; if (xform->auth.key.length > get_auth_algo_blocksize(SHA_512)) { - mb_ops->aux.multi_block.sha512( + IMB_SHA512(mb_mgr, xform->auth.key.data, xform->auth.key.length, hashed_key); key_larger_block_size = 1; } -#endif + break; + case RTE_CRYPTO_AUTH_SHA512: + sess->auth.algo = PLAIN_SHA_512; + auth_precompute = 0; break; default: AESNI_MB_LOG(ERR, "Unsupported authentication algorithm selection"); @@ -256,12 +322,8 @@ aesni_mb_set_session_auth_parameters(const struct aesni_mb_op_fns *mb_ops, uint16_t full_digest_size = get_digest_byte_length(sess->auth.algo); -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) if (sess->auth.req_digest_len > full_digest_size || sess->auth.req_digest_len == 0) { -#else - if (sess->auth.req_digest_len != trunc_digest_size) { -#endif AESNI_MB_LOG(ERR, "Invalid digest size\n"); return -EINVAL; } @@ -272,6 +334,10 @@ aesni_mb_set_session_auth_parameters(const struct aesni_mb_op_fns *mb_ops, else sess->auth.gen_digest_len = sess->auth.req_digest_len; + /* Plain SHA does not require precompute key */ + if (auth_precompute == 0) + return 0; + /* Calculate Authentication precomputes */ if (key_larger_block_size) { calculate_auth_precomputes(hash_oneblock_fn, @@ -292,13 +358,12 @@ aesni_mb_set_session_auth_parameters(const struct aesni_mb_op_fns *mb_ops, /** Set session cipher parameters */ static int -aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops, +aesni_mb_set_session_cipher_parameters(const MB_MGR *mb_mgr, struct aesni_mb_session *sess, const struct rte_crypto_sym_xform *xform) { uint8_t is_aes = 0; uint8_t is_3DES = 0; - aes_keyexp_t aes_keyexp_fn; if (xform == NULL) { sess->cipher.mode = NULL_CIPHER; @@ -361,26 +426,26 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops, switch (xform->cipher.key.length) { case AES_128_BYTES: sess->cipher.key_length_in_bytes = AES_128_BYTES; - aes_keyexp_fn = mb_ops->aux.keyexp.aes128; + IMB_AES_KEYEXP_128(mb_mgr, xform->cipher.key.data, + sess->cipher.expanded_aes_keys.encode, + sess->cipher.expanded_aes_keys.decode); break; case AES_192_BYTES: sess->cipher.key_length_in_bytes = AES_192_BYTES; - aes_keyexp_fn = mb_ops->aux.keyexp.aes192; + IMB_AES_KEYEXP_192(mb_mgr, xform->cipher.key.data, + sess->cipher.expanded_aes_keys.encode, + sess->cipher.expanded_aes_keys.decode); break; case AES_256_BYTES: sess->cipher.key_length_in_bytes = AES_256_BYTES; - aes_keyexp_fn = mb_ops->aux.keyexp.aes256; + IMB_AES_KEYEXP_256(mb_mgr, xform->cipher.key.data, + sess->cipher.expanded_aes_keys.encode, + sess->cipher.expanded_aes_keys.decode); break; default: AESNI_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } - - /* Expanded cipher keys */ - (*aes_keyexp_fn)(xform->cipher.key.data, - sess->cipher.expanded_aes_keys.encode, - sess->cipher.expanded_aes_keys.decode); - } else if (is_3DES) { uint64_t *keys[3] = {sess->cipher.exp_3des_keys.key[0], sess->cipher.exp_3des_keys.key[1], @@ -388,9 +453,12 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops, switch (xform->cipher.key.length) { case 24: - des_key_schedule(keys[0], xform->cipher.key.data); - des_key_schedule(keys[1], xform->cipher.key.data+8); - des_key_schedule(keys[2], xform->cipher.key.data+16); + IMB_DES_KEYSCHED(mb_mgr, keys[0], + xform->cipher.key.data); + IMB_DES_KEYSCHED(mb_mgr, keys[1], + xform->cipher.key.data + 8); + IMB_DES_KEYSCHED(mb_mgr, keys[2], + xform->cipher.key.data + 16); /* Initialize keys - 24 bytes: [K1-K2-K3] */ sess->cipher.exp_3des_keys.ks_ptr[0] = keys[0]; @@ -398,8 +466,10 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops, sess->cipher.exp_3des_keys.ks_ptr[2] = keys[2]; break; case 16: - des_key_schedule(keys[0], xform->cipher.key.data); - des_key_schedule(keys[1], xform->cipher.key.data+8); + IMB_DES_KEYSCHED(mb_mgr, keys[0], + xform->cipher.key.data); + IMB_DES_KEYSCHED(mb_mgr, keys[1], + xform->cipher.key.data + 8); /* Initialize keys - 16 bytes: [K1=K1,K2=K2,K3=K1] */ sess->cipher.exp_3des_keys.ks_ptr[0] = keys[0]; @@ -407,7 +477,8 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops, sess->cipher.exp_3des_keys.ks_ptr[2] = keys[0]; break; case 8: - des_key_schedule(keys[0], xform->cipher.key.data); + IMB_DES_KEYSCHED(mb_mgr, keys[0], + xform->cipher.key.data); /* Initialize keys - 8 bytes: [K1 = K2 = K3] */ sess->cipher.exp_3des_keys.ks_ptr[0] = keys[0]; @@ -419,11 +490,7 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops, return -EINVAL; } -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) sess->cipher.key_length_in_bytes = 24; -#else - sess->cipher.key_length_in_bytes = 8; -#endif } else { if (xform->cipher.key.length != 8) { AESNI_MB_LOG(ERR, "Invalid cipher key length"); @@ -431,9 +498,11 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops, } sess->cipher.key_length_in_bytes = 8; - des_key_schedule((uint64_t *)sess->cipher.expanded_aes_keys.encode, + IMB_DES_KEYSCHED(mb_mgr, + (uint64_t *)sess->cipher.expanded_aes_keys.encode, xform->cipher.key.data); - des_key_schedule((uint64_t *)sess->cipher.expanded_aes_keys.decode, + IMB_DES_KEYSCHED(mb_mgr, + (uint64_t *)sess->cipher.expanded_aes_keys.decode, xform->cipher.key.data); } @@ -441,15 +510,10 @@ aesni_mb_set_session_cipher_parameters(const struct aesni_mb_op_fns *mb_ops, } static int -aesni_mb_set_session_aead_parameters(const struct aesni_mb_op_fns *mb_ops, +aesni_mb_set_session_aead_parameters(const MB_MGR *mb_mgr, struct aesni_mb_session *sess, const struct rte_crypto_sym_xform *xform) { - union { - aes_keyexp_t aes_keyexp_fn; - aes_gcm_keyexp_t aes_gcm_keyexp_fn; - } keyexp; - switch (xform->aead.op) { case RTE_CRYPTO_AEAD_OP_ENCRYPT: sess->cipher.direction = ENCRYPT; @@ -473,17 +537,15 @@ aesni_mb_set_session_aead_parameters(const struct aesni_mb_op_fns *mb_ops, switch (xform->aead.key.length) { case AES_128_BYTES: sess->cipher.key_length_in_bytes = AES_128_BYTES; - keyexp.aes_keyexp_fn = mb_ops->aux.keyexp.aes128; + IMB_AES_KEYEXP_128(mb_mgr, xform->aead.key.data, + sess->cipher.expanded_aes_keys.encode, + sess->cipher.expanded_aes_keys.decode); break; default: AESNI_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } - /* Expanded cipher keys */ - (*keyexp.aes_keyexp_fn)(xform->aead.key.data, - sess->cipher.expanded_aes_keys.encode, - sess->cipher.expanded_aes_keys.decode); break; case RTE_CRYPTO_AEAD_AES_GCM: @@ -493,26 +555,24 @@ aesni_mb_set_session_aead_parameters(const struct aesni_mb_op_fns *mb_ops, switch (xform->aead.key.length) { case AES_128_BYTES: sess->cipher.key_length_in_bytes = AES_128_BYTES; - keyexp.aes_gcm_keyexp_fn = - mb_ops->aux.keyexp.aes_gcm_128; + IMB_AES128_GCM_PRE(mb_mgr, xform->aead.key.data, + &sess->cipher.gcm_key); break; case AES_192_BYTES: sess->cipher.key_length_in_bytes = AES_192_BYTES; - keyexp.aes_gcm_keyexp_fn = - mb_ops->aux.keyexp.aes_gcm_192; + IMB_AES192_GCM_PRE(mb_mgr, xform->aead.key.data, + &sess->cipher.gcm_key); break; case AES_256_BYTES: sess->cipher.key_length_in_bytes = AES_256_BYTES; - keyexp.aes_gcm_keyexp_fn = - mb_ops->aux.keyexp.aes_gcm_256; + IMB_AES256_GCM_PRE(mb_mgr, xform->aead.key.data, + &sess->cipher.gcm_key); break; default: AESNI_MB_LOG(ERR, "Invalid cipher key length"); return -EINVAL; } - (keyexp.aes_gcm_keyexp_fn)(xform->aead.key.data, - &sess->cipher.gcm_key); break; default: @@ -539,7 +599,7 @@ aesni_mb_set_session_aead_parameters(const struct aesni_mb_op_fns *mb_ops, /** Parse crypto xform chain and set private session parameters */ int -aesni_mb_set_session_parameters(const struct aesni_mb_op_fns *mb_ops, +aesni_mb_set_session_parameters(const MB_MGR *mb_mgr, struct aesni_mb_session *sess, const struct rte_crypto_sym_xform *xform) { @@ -598,13 +658,13 @@ aesni_mb_set_session_parameters(const struct aesni_mb_op_fns *mb_ops, /* Default IV length = 0 */ sess->iv.length = 0; - ret = aesni_mb_set_session_auth_parameters(mb_ops, sess, auth_xform); + ret = aesni_mb_set_session_auth_parameters(mb_mgr, sess, auth_xform); if (ret != 0) { AESNI_MB_LOG(ERR, "Invalid/unsupported authentication parameters"); return ret; } - ret = aesni_mb_set_session_cipher_parameters(mb_ops, sess, + ret = aesni_mb_set_session_cipher_parameters(mb_mgr, sess, cipher_xform); if (ret != 0) { AESNI_MB_LOG(ERR, "Invalid/unsupported cipher parameters"); @@ -612,7 +672,7 @@ aesni_mb_set_session_parameters(const struct aesni_mb_op_fns *mb_ops, } if (aead_xform) { - ret = aesni_mb_set_session_aead_parameters(mb_ops, sess, + ret = aesni_mb_set_session_aead_parameters(mb_mgr, sess, aead_xform); if (ret != 0) { AESNI_MB_LOG(ERR, "Invalid/unsupported aead parameters"); @@ -673,7 +733,7 @@ get_session(struct aesni_mb_qp *qp, struct rte_crypto_op *op) sess = (struct aesni_mb_session *)_sess_private_data; - if (unlikely(aesni_mb_set_session_parameters(qp->op_fns, + if (unlikely(aesni_mb_set_session_parameters(qp->mb_mgr, sess, op->sym->xform) != 0)) { rte_mempool_put(qp->sess_mp, _sess); rte_mempool_put(qp->sess_mp, _sess_private_data); @@ -690,6 +750,56 @@ get_session(struct aesni_mb_qp *qp, struct rte_crypto_op *op) return sess; } +static inline uint64_t +auth_start_offset(struct rte_crypto_op *op, struct aesni_mb_session *session, + uint32_t oop) +{ + struct rte_mbuf *m_src, *m_dst; + uint8_t *p_src, *p_dst; + uintptr_t u_src, u_dst; + uint32_t cipher_end, auth_end; + + /* Only cipher then hash needs special calculation. */ + if (!oop || session->chain_order != CIPHER_HASH) + return op->sym->auth.data.offset; + + m_src = op->sym->m_src; + m_dst = op->sym->m_dst; + + p_src = rte_pktmbuf_mtod(m_src, uint8_t *); + p_dst = rte_pktmbuf_mtod(m_dst, uint8_t *); + u_src = (uintptr_t)p_src; + u_dst = (uintptr_t)p_dst + op->sym->auth.data.offset; + + /** + * Copy the content between cipher offset and auth offset for generating + * correct digest. + */ + if (op->sym->cipher.data.offset > op->sym->auth.data.offset) + memcpy(p_dst + op->sym->auth.data.offset, + p_src + op->sym->auth.data.offset, + op->sym->cipher.data.offset - + op->sym->auth.data.offset); + + /** + * Copy the content between (cipher offset + length) and (auth offset + + * length) for generating correct digest + */ + cipher_end = op->sym->cipher.data.offset + op->sym->cipher.data.length; + auth_end = op->sym->auth.data.offset + op->sym->auth.data.length; + if (cipher_end < auth_end) + memcpy(p_dst + cipher_end, p_src + cipher_end, + auth_end - cipher_end); + + /** + * Since intel-ipsec-mb only supports positive values, + * we need to deduct the correct offset between src and dst. + */ + + return u_src < u_dst ? (u_dst - u_src) : + (UINT64_MAX - u_src + u_dst + 1); +} + /** * Process a crypto operation and complete a JOB_AES_HMAC job structure for * submission to the multi buffer library for processing. @@ -708,7 +818,7 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, { struct rte_mbuf *m_src = op->sym->m_src, *m_dst; struct aesni_mb_session *session; - uint16_t m_offset = 0; + uint32_t m_offset, oop; session = get_session(qp, op); if (session == NULL) { @@ -760,8 +870,16 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, break; case AES_GMAC: - job->u.GCM.aad = op->sym->aead.aad.data; - job->u.GCM.aad_len_in_bytes = session->aead.aad_len; + if (session->cipher.mode == GCM) { + job->u.GCM.aad = op->sym->aead.aad.data; + job->u.GCM.aad_len_in_bytes = session->aead.aad_len; + } else { + /* For GMAC */ + job->u.GCM.aad = rte_pktmbuf_mtod_offset(m_src, + uint8_t *, op->sym->auth.data.offset); + job->u.GCM.aad_len_in_bytes = op->sym->auth.data.length; + job->cipher_mode = GCM; + } job->aes_enc_key_expanded = &session->cipher.gcm_key; job->aes_dec_key_expanded = &session->cipher.gcm_key; break; @@ -783,37 +901,34 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, } } - /* Mutable crypto operation parameters */ - if (op->sym->m_dst) { - m_src = m_dst = op->sym->m_dst; - - /* append space for output data to mbuf */ - char *odata = rte_pktmbuf_append(m_dst, - rte_pktmbuf_data_len(op->sym->m_src)); - if (odata == NULL) { - AESNI_MB_LOG(ERR, "failed to allocate space in destination " - "mbuf for source data"); - op->status = RTE_CRYPTO_OP_STATUS_ERROR; - return -1; - } - - memcpy(odata, rte_pktmbuf_mtod(op->sym->m_src, void*), - rte_pktmbuf_data_len(op->sym->m_src)); - } else { + if (!op->sym->m_dst) { + /* in-place operation */ m_dst = m_src; - if (job->hash_alg == AES_CCM || job->hash_alg == AES_GMAC) - m_offset = op->sym->aead.data.offset; - else - m_offset = op->sym->cipher.data.offset; + oop = 0; + } else if (op->sym->m_dst == op->sym->m_src) { + /* in-place operation */ + m_dst = m_src; + oop = 0; + } else { + /* out-of-place operation */ + m_dst = op->sym->m_dst; + oop = 1; } + if (job->hash_alg == AES_CCM || (job->hash_alg == AES_GMAC && + session->cipher.mode == GCM)) + m_offset = op->sym->aead.data.offset; + else + m_offset = op->sym->cipher.data.offset; + /* Set digest output location */ if (job->hash_alg != NULL_HASH && session->auth.operation == RTE_CRYPTO_AUTH_OP_VERIFY) { job->auth_tag_output = qp->temp_digests[*digest_idx]; *digest_idx = (*digest_idx + 1) % MAX_JOBS; } else { - if (job->hash_alg == AES_CCM || job->hash_alg == AES_GMAC) + if (job->hash_alg == AES_CCM || (job->hash_alg == AES_GMAC && + session->cipher.mode == GCM)) job->auth_tag_output = op->sym->aead.digest.data; else job->auth_tag_output = op->sym->auth.digest.data; @@ -834,7 +949,7 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, /* Set IV parameters */ job->iv_len_in_bytes = session->iv.length; - /* Data Parameter */ + /* Data Parameters */ job->src = rte_pktmbuf_mtod(m_src, uint8_t *); job->dst = rte_pktmbuf_mtod_offset(m_dst, uint8_t *, m_offset); @@ -851,11 +966,24 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, break; case AES_GMAC: - job->cipher_start_src_offset_in_bytes = - op->sym->aead.data.offset; - job->hash_start_src_offset_in_bytes = op->sym->aead.data.offset; - job->msg_len_to_cipher_in_bytes = op->sym->aead.data.length; - job->msg_len_to_hash_in_bytes = job->msg_len_to_cipher_in_bytes; + if (session->cipher.mode == GCM) { + job->cipher_start_src_offset_in_bytes = + op->sym->aead.data.offset; + job->hash_start_src_offset_in_bytes = + op->sym->aead.data.offset; + job->msg_len_to_cipher_in_bytes = + op->sym->aead.data.length; + job->msg_len_to_hash_in_bytes = + op->sym->aead.data.length; + } else { + job->cipher_start_src_offset_in_bytes = + op->sym->auth.data.offset; + job->hash_start_src_offset_in_bytes = + op->sym->auth.data.offset; + job->msg_len_to_cipher_in_bytes = 0; + job->msg_len_to_hash_in_bytes = 0; + } + job->iv = rte_crypto_op_ctod_offset(op, uint8_t *, session->iv.offset); break; @@ -865,7 +993,8 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, op->sym->cipher.data.offset; job->msg_len_to_cipher_in_bytes = op->sym->cipher.data.length; - job->hash_start_src_offset_in_bytes = op->sym->auth.data.offset; + job->hash_start_src_offset_in_bytes = auth_start_offset(op, + session, oop); job->msg_len_to_hash_in_bytes = op->sym->auth.data.length; job->iv = rte_crypto_op_ctod_offset(op, uint8_t *, @@ -879,26 +1008,18 @@ set_mb_job_params(JOB_AES_HMAC *job, struct aesni_mb_qp *qp, } static inline void -verify_digest(JOB_AES_HMAC *job, struct rte_crypto_op *op, - struct aesni_mb_session *sess) +verify_digest(JOB_AES_HMAC *job, void *digest, uint16_t len, uint8_t *status) { /* Verify digest if required */ - if (job->hash_alg == AES_CCM || job->hash_alg == AES_GMAC) { - if (memcmp(job->auth_tag_output, op->sym->aead.digest.data, - sess->auth.req_digest_len) != 0) - op->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; - } else { - if (memcmp(job->auth_tag_output, op->sym->auth.digest.data, - sess->auth.req_digest_len) != 0) - op->status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; - } + if (memcmp(job->auth_tag_output, digest, len) != 0) + *status = RTE_CRYPTO_OP_STATUS_AUTH_FAILED; } static inline void generate_digest(JOB_AES_HMAC *job, struct rte_crypto_op *op, struct aesni_mb_session *sess) { - /* No extra copy neeed */ + /* No extra copy needed */ if (likely(sess->auth.req_digest_len == sess->auth.gen_digest_len)) return; @@ -933,13 +1054,24 @@ post_process_mb_job(struct aesni_mb_qp *qp, JOB_AES_HMAC *job) case STS_COMPLETED: op->status = RTE_CRYPTO_OP_STATUS_SUCCESS; - if (job->hash_alg != NULL_HASH) { - if (sess->auth.operation == - RTE_CRYPTO_AUTH_OP_VERIFY) - verify_digest(job, op, sess); + if (job->hash_alg == NULL_HASH) + break; + + if (sess->auth.operation == RTE_CRYPTO_AUTH_OP_VERIFY) { + if (job->hash_alg == AES_CCM || + (job->hash_alg == AES_GMAC && + sess->cipher.mode == GCM)) + verify_digest(job, + op->sym->aead.digest.data, + sess->auth.req_digest_len, + &op->status); else - generate_digest(job, op, sess); - } + verify_digest(job, + op->sym->auth.digest.data, + sess->auth.req_digest_len, + &op->status); + } else + generate_digest(job, op, sess); break; default: op->status = RTE_CRYPTO_OP_STATUS_ERROR; @@ -989,7 +1121,7 @@ handle_completed_jobs(struct aesni_mb_qp *qp, JOB_AES_HMAC *job, if (processed_jobs == nb_ops) break; - job = (*qp->op_fns->job.get_completed_job)(qp->mb_mgr); + job = IMB_GET_COMPLETED_JOB(qp->mb_mgr); } return processed_jobs; @@ -1002,7 +1134,7 @@ flush_mb_mgr(struct aesni_mb_qp *qp, struct rte_crypto_op **ops, int processed_ops = 0; /* Flush the remaining jobs */ - JOB_AES_HMAC *job = (*qp->op_fns->job.flush_job)(qp->mb_mgr); + JOB_AES_HMAC *job = IMB_FLUSH_JOB(qp->mb_mgr); if (job) processed_ops += handle_completed_jobs(qp, job, @@ -1042,7 +1174,7 @@ aesni_mb_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops, uint8_t digest_idx = qp->digest_idx; do { /* Get next free mb job struct from mb manager */ - job = (*qp->op_fns->job.get_next)(qp->mb_mgr); + job = IMB_GET_NEXT_JOB(qp->mb_mgr); if (unlikely(job == NULL)) { /* if no free mb job structs we need to flush mb_mgr */ processed_jobs += flush_mb_mgr(qp, @@ -1052,7 +1184,7 @@ aesni_mb_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops, if (nb_ops == processed_jobs) break; - job = (*qp->op_fns->job.get_next)(qp->mb_mgr); + job = IMB_GET_NEXT_JOB(qp->mb_mgr); } /* @@ -1072,8 +1204,11 @@ aesni_mb_pmd_dequeue_burst(void *queue_pair, struct rte_crypto_op **ops, } /* Submit job to multi-buffer for processing */ - job = (*qp->op_fns->job.submit)(qp->mb_mgr); - +#ifdef RTE_LIBRTE_PMD_AESNI_MB_DEBUG + job = IMB_SUBMIT_JOB(qp->mb_mgr); +#else + job = IMB_SUBMIT_JOB_NOCHECK(qp->mb_mgr); +#endif /* * If submit returns a processed job then handle it, * before submitting subsequent jobs @@ -1105,12 +1240,7 @@ cryptodev_aesni_mb_create(const char *name, struct rte_cryptodev *dev; struct aesni_mb_private *internals; enum aesni_mb_vector_mode vector_mode; - - /* Check CPU for support for AES instruction set */ - if (!rte_cpu_get_flag_enabled(RTE_CPUFLAG_AES)) { - AESNI_MB_LOG(ERR, "AES instructions not supported by CPU"); - return -EFAULT; - } + MB_MGR *mb_mgr; dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params); if (dev == NULL) { @@ -1137,23 +1267,38 @@ cryptodev_aesni_mb_create(const char *name, dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - RTE_CRYPTODEV_FF_CPU_AESNI; + RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; + + /* Check CPU for support for AES instruction set */ + if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AES)) + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AESNI; + else + AESNI_MB_LOG(WARNING, "AES instructions not supported by CPU"); + + mb_mgr = alloc_mb_mgr(0); + if (mb_mgr == NULL) + return -ENOMEM; switch (vector_mode) { case RTE_AESNI_MB_SSE: dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; + init_mb_mgr_sse(mb_mgr); break; case RTE_AESNI_MB_AVX: dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; + init_mb_mgr_avx(mb_mgr); break; case RTE_AESNI_MB_AVX2: dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; + init_mb_mgr_avx2(mb_mgr); break; case RTE_AESNI_MB_AVX512: dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512; + init_mb_mgr_avx512(mb_mgr); break; default: - break; + AESNI_MB_LOG(ERR, "Unsupported vector mode %u\n", vector_mode); + goto error_exit; } /* Set vector instructions mode supported */ @@ -1161,15 +1306,19 @@ cryptodev_aesni_mb_create(const char *name, internals->vector_mode = vector_mode; internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs; + internals->mb_mgr = mb_mgr; -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) AESNI_MB_LOG(INFO, "IPSec Multi-buffer library version used: %s\n", imb_get_version_str()); -#else - AESNI_MB_LOG(INFO, "IPSec Multi-buffer library version used: 0.49.0\n"); -#endif return 0; +error_exit: + if (mb_mgr) + free_mb_mgr(mb_mgr); + + rte_cryptodev_pmd_destroy(dev); + + return -1; } static int @@ -1204,6 +1353,7 @@ static int cryptodev_aesni_mb_remove(struct rte_vdev_device *vdev) { struct rte_cryptodev *cryptodev; + struct aesni_mb_private *internals; const char *name; name = rte_vdev_device_name(vdev); @@ -1214,6 +1364,10 @@ cryptodev_aesni_mb_remove(struct rte_vdev_device *vdev) if (cryptodev == NULL) return -ENODEV; + internals = cryptodev->data->dev_private; + + free_mb_mgr(internals->mb_mgr); + return rte_cryptodev_pmd_destroy(cryptodev); } diff --git a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_ops.c b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_ops.c index f3eff2685..e58dfa33b 100644 --- a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_ops.c +++ b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_ops.c @@ -25,15 +25,9 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { .increment = 1 }, .digest_size = { -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) .min = 1, .max = 16, .increment = 1 -#else - .min = 12, - .max = 12, - .increment = 0 -#endif }, .iv_size = { 0 } }, } @@ -48,23 +42,34 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { .block_size = 64, .key_size = { .min = 1, -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) .max = 65535, -#else - .max = 64, -#endif .increment = 1 }, .digest_size = { -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) .min = 1, .max = 20, .increment = 1 -#else - .min = 12, - .max = 12, + }, + .iv_size = { 0 } + }, } + }, } + }, + { /* SHA1 */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_SHA1, + .block_size = 64, + .key_size = { + .min = 0, + .max = 0, .increment = 0 -#endif + }, + .digest_size = { + .min = 1, + .max = 20, + .increment = 1 }, .iv_size = { 0 } }, } @@ -79,23 +84,34 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { .block_size = 64, .key_size = { .min = 1, -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) .max = 65535, -#else - .max = 64, -#endif .increment = 1 }, .digest_size = { -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) .min = 1, .max = 28, .increment = 1 -#else - .min = 14, - .max = 14, + }, + .iv_size = { 0 } + }, } + }, } + }, + { /* SHA224 */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_SHA224, + .block_size = 64, + .key_size = { + .min = 0, + .max = 0, .increment = 0 -#endif + }, + .digest_size = { + .min = 1, + .max = 28, + .increment = 1 }, .iv_size = { 0 } }, } @@ -110,23 +126,34 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { .block_size = 64, .key_size = { .min = 1, -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) .max = 65535, -#else - .max = 64, -#endif .increment = 1 }, .digest_size = { -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) .min = 1, .max = 32, .increment = 1 -#else - .min = 16, - .max = 16, + }, + .iv_size = { 0 } + }, } + }, } + }, + { /* SHA256 */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_SHA256, + .block_size = 64, + .key_size = { + .min = 0, + .max = 0, .increment = 0 -#endif + }, + .digest_size = { + .min = 1, + .max = 32, + .increment = 1 }, .iv_size = { 0 } }, } @@ -141,23 +168,34 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { .block_size = 128, .key_size = { .min = 1, -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) .max = 65535, -#else - .max = 128, -#endif .increment = 1 }, .digest_size = { -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) .min = 1, .max = 48, .increment = 1 -#else - .min = 24, - .max = 24, + }, + .iv_size = { 0 } + }, } + }, } + }, + { /* SHA384 */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_SHA384, + .block_size = 128, + .key_size = { + .min = 0, + .max = 0, .increment = 0 -#endif + }, + .digest_size = { + .min = 1, + .max = 48, + .increment = 1 }, .iv_size = { 0 } }, } @@ -172,23 +210,34 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { .block_size = 128, .key_size = { .min = 1, -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) .max = 65535, -#else - .max = 128, -#endif .increment = 1 }, .digest_size = { -#if IMB_VERSION_NUM >= IMB_VERSION(0, 50, 0) .min = 1, .max = 64, .increment = 1 -#else - .min = 32, - .max = 32, + }, + .iv_size = { 0 } + }, } + }, } + }, + { /* SHA512 */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_SHA512, + .block_size = 128, + .key_size = { + .min = 0, + .max = 0, .increment = 0 -#endif + }, + .digest_size = { + .min = 1, + .max = 64, + .increment = 1 }, .iv_size = { 0 } }, } @@ -416,6 +465,31 @@ static const struct rte_cryptodev_capabilities aesni_mb_pmd_capabilities[] = { }, } }, } }, + { /* AES GMAC (AUTH) */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_AES_GMAC, + .block_size = 16, + .key_size = { + .min = 16, + .max = 32, + .increment = 8 + }, + .digest_size = { + .min = 8, + .max = 16, + .increment = 4 + }, + .iv_size = { + .min = 12, + .max = 12, + .increment = 0 + } + }, } + }, } + }, RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() }; @@ -595,7 +669,28 @@ aesni_mb_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, goto qp_setup_cleanup; } - qp->op_fns = &job_ops[internals->vector_mode]; + switch (internals->vector_mode) { + case RTE_AESNI_MB_SSE: + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; + init_mb_mgr_sse(qp->mb_mgr); + break; + case RTE_AESNI_MB_AVX: + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; + init_mb_mgr_avx(qp->mb_mgr); + break; + case RTE_AESNI_MB_AVX2: + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; + init_mb_mgr_avx2(qp->mb_mgr); + break; + case RTE_AESNI_MB_AVX512: + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX512; + init_mb_mgr_avx512(qp->mb_mgr); + break; + default: + AESNI_MB_LOG(ERR, "Unsupported vector mode %u\n", + internals->vector_mode); + goto qp_setup_cleanup; + } qp->ingress_queue = aesni_mb_pmd_qp_create_processed_ops_ring(qp, qp_conf->nb_descriptors, socket_id); @@ -613,13 +708,11 @@ aesni_mb_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, snprintf(mp_name, RTE_MEMPOOL_NAMESIZE, "digest_mp_%u_%u", dev->data->dev_id, qp_id); - /* Initialise multi-buffer manager */ - (*qp->op_fns->job.init_mgr)(qp->mb_mgr); return 0; qp_setup_cleanup: if (qp) { - if (qp->mb_mgr == NULL) + if (qp->mb_mgr) free_mb_mgr(qp->mb_mgr); rte_free(qp); } @@ -663,7 +756,7 @@ aesni_mb_pmd_sym_session_configure(struct rte_cryptodev *dev, return -ENOMEM; } - ret = aesni_mb_set_session_parameters(&job_ops[internals->vector_mode], + ret = aesni_mb_set_session_parameters(internals->mb_mgr, sess_private_data, xform); if (ret != 0) { AESNI_MB_LOG(ERR, "failed configure session parameters"); diff --git a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_private.h b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_private.h index d8021cdaa..6a5df942c 100644 --- a/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_private.h +++ b/drivers/crypto/aesni_mb/rte_aesni_mb_pmd_private.h @@ -25,6 +25,7 @@ int aesni_mb_logtype_driver; /* Maximum length for digest */ #define DIGEST_LENGTH_MAX 64 static const unsigned auth_blocksize[] = { + [NULL_HASH] = 0, [MD5] = 64, [SHA1] = 64, [SHA_224] = 64, @@ -33,6 +34,13 @@ static const unsigned auth_blocksize[] = { [SHA_512] = 128, [AES_XCBC] = 16, [AES_CCM] = 16, + [AES_CMAC] = 16, + [AES_GMAC] = 16, + [PLAIN_SHA1] = 64, + [PLAIN_SHA_224] = 64, + [PLAIN_SHA_256] = 64, + [PLAIN_SHA_384] = 128, + [PLAIN_SHA_512] = 128 }; /** @@ -57,7 +65,13 @@ static const unsigned auth_truncated_digest_byte_lengths[] = { [AES_XCBC] = 12, [AES_CMAC] = 12, [AES_CCM] = 8, - [NULL_HASH] = 0 + [NULL_HASH] = 0, + [AES_GMAC] = 16, + [PLAIN_SHA1] = 20, + [PLAIN_SHA_224] = 28, + [PLAIN_SHA_256] = 32, + [PLAIN_SHA_384] = 48, + [PLAIN_SHA_512] = 64 }; /** @@ -82,8 +96,14 @@ static const unsigned auth_digest_byte_lengths[] = { [SHA_512] = 64, [AES_XCBC] = 16, [AES_CMAC] = 16, + [AES_CCM] = 16, [AES_GMAC] = 12, - [NULL_HASH] = 0 + [NULL_HASH] = 0, + [PLAIN_SHA1] = 20, + [PLAIN_SHA_224] = 28, + [PLAIN_SHA_256] = 32, + [PLAIN_SHA_384] = 48, + [PLAIN_SHA_512] = 64 }; /** @@ -115,6 +135,8 @@ struct aesni_mb_private { /**< CPU vector instruction set mode */ unsigned max_nb_queue_pairs; /**< Max number of queue pairs supported by device */ + MB_MGR *mb_mgr; + /**< Multi-buffer instance */ }; /** AESNI Multi buffer queue pair */ @@ -123,8 +145,6 @@ struct aesni_mb_qp { /**< Queue Pair Identifier */ char name[RTE_CRYPTODEV_NAME_MAX_LEN]; /**< Unique Queue Pair Name */ - const struct aesni_mb_op_fns *op_fns; - /**< Vector mode dependent pointer table of the multi-buffer APIs */ MB_MGR *mb_mgr; /**< Multi-buffer instance */ struct rte_ring *ingress_queue; @@ -238,7 +258,7 @@ struct aesni_mb_session { * */ extern int -aesni_mb_set_session_parameters(const struct aesni_mb_op_fns *mb_ops, +aesni_mb_set_session_parameters(const MB_MGR *mb_mgr, struct aesni_mb_session *sess, const struct rte_crypto_sym_xform *xform); From patchwork Thu Mar 5 15:34:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "De Lara Guarch, Pablo" X-Patchwork-Id: 66308 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id E5EDEA0564; Thu, 5 Mar 2020 18:10:38 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D67931C00E; Thu, 5 Mar 2020 18:09:46 +0100 (CET) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 6250D1BFE2 for ; Thu, 5 Mar 2020 18:09:42 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Mar 2020 09:09:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,518,1574150400"; d="scan'208";a="275173411" Received: from silpixa00400565.ir.intel.com (HELO silpixa00400565.ger.corp.intel.com) ([10.237.222.249]) by fmsmga002.fm.intel.com with ESMTP; 05 Mar 2020 09:09:40 -0800 From: Pablo de Lara To: dev@dpdk.org Cc: Pablo de Lara Date: Thu, 5 Mar 2020 15:34:54 +0000 Message-Id: <20200305153454.724874-6-pablo.de.lara.guarch@intel.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200305153454.724874-1-pablo.de.lara.guarch@intel.com> References: <20200305153454.724874-1-pablo.de.lara.guarch@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [RFC PATCH 5/5] crypto/aesni_gcm: support IPSec MB library v0.53 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add support for underlying Intel IPSec Multi-buffer library v0.53. Signed-off-by: Pablo de Lara --- drivers/crypto/aesni_gcm/aesni_gcm_ops.h | 65 ++------- drivers/crypto/aesni_gcm/aesni_gcm_pmd.c | 130 +++++++++++++----- drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c | 4 +- .../crypto/aesni_gcm/aesni_gcm_pmd_private.h | 4 + 4 files changed, 114 insertions(+), 89 deletions(-) diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_ops.h b/drivers/crypto/aesni_gcm/aesni_gcm_ops.h index 450616698..b2cc4002e 100644 --- a/drivers/crypto/aesni_gcm/aesni_gcm_ops.h +++ b/drivers/crypto/aesni_gcm/aesni_gcm_ops.h @@ -17,14 +17,15 @@ enum aesni_gcm_vector_mode { RTE_AESNI_GCM_SSE, RTE_AESNI_GCM_AVX, RTE_AESNI_GCM_AVX2, + RTE_AESNI_GCM_AVX512, RTE_AESNI_GCM_VECTOR_NUM }; enum aesni_gcm_key { - AESNI_GCM_KEY_128, - AESNI_GCM_KEY_192, - AESNI_GCM_KEY_256, - AESNI_GCM_KEY_NUM + GCM_KEY_128 = 0, + GCM_KEY_192, + GCM_KEY_256, + GCM_KEY_NUM }; @@ -34,7 +35,7 @@ typedef void (*aesni_gcm_t)(const struct gcm_key_data *gcm_key_data, const uint8_t *aad, uint64_t aad_len, uint8_t *auth_tag, uint64_t auth_tag_len); -typedef void (*aesni_gcm_precomp_t)(const void *key, struct gcm_key_data *gcm_data); +typedef void (*aesni_gcm_pre_t)(const void *key, struct gcm_key_data *gcm_data); typedef void (*aesni_gcm_init_t)(const struct gcm_key_data *gcm_key_data, struct gcm_context_data *gcm_ctx_data, @@ -57,60 +58,12 @@ typedef void (*aesni_gcm_finalize_t)(const struct gcm_key_data *gcm_key_data, struct aesni_gcm_ops { aesni_gcm_t enc; /**< GCM encode function pointer */ aesni_gcm_t dec; /**< GCM decode function pointer */ - aesni_gcm_precomp_t precomp; /**< GCM pre-compute */ + aesni_gcm_pre_t pre; /**< GCM pre-compute */ aesni_gcm_init_t init; aesni_gcm_update_t update_enc; aesni_gcm_update_t update_dec; - aesni_gcm_finalize_t finalize; + aesni_gcm_finalize_t finalize_enc; + aesni_gcm_finalize_t finalize_dec; }; -#define AES_GCM_FN(keylen, arch) \ -aes_gcm_enc_##keylen##_##arch,\ -aes_gcm_dec_##keylen##_##arch,\ -aes_gcm_pre_##keylen##_##arch,\ -aes_gcm_init_##keylen##_##arch,\ -aes_gcm_enc_##keylen##_update_##arch,\ -aes_gcm_dec_##keylen##_update_##arch,\ -aes_gcm_enc_##keylen##_finalize_##arch, - -static const struct aesni_gcm_ops gcm_ops[RTE_AESNI_GCM_VECTOR_NUM][AESNI_GCM_KEY_NUM] = { - [RTE_AESNI_GCM_NOT_SUPPORTED] = { - [AESNI_GCM_KEY_128] = {NULL}, - [AESNI_GCM_KEY_192] = {NULL}, - [AESNI_GCM_KEY_256] = {NULL} - }, - [RTE_AESNI_GCM_SSE] = { - [AESNI_GCM_KEY_128] = { - AES_GCM_FN(128, sse) - }, - [AESNI_GCM_KEY_192] = { - AES_GCM_FN(192, sse) - }, - [AESNI_GCM_KEY_256] = { - AES_GCM_FN(256, sse) - } - }, - [RTE_AESNI_GCM_AVX] = { - [AESNI_GCM_KEY_128] = { - AES_GCM_FN(128, avx_gen2) - }, - [AESNI_GCM_KEY_192] = { - AES_GCM_FN(192, avx_gen2) - }, - [AESNI_GCM_KEY_256] = { - AES_GCM_FN(256, avx_gen2) - } - }, - [RTE_AESNI_GCM_AVX2] = { - [AESNI_GCM_KEY_128] = { - AES_GCM_FN(128, avx_gen4) - }, - [AESNI_GCM_KEY_192] = { - AES_GCM_FN(192, avx_gen4) - }, - [AESNI_GCM_KEY_256] = { - AES_GCM_FN(256, avx_gen4) - } - } -}; #endif /* _AESNI_GCM_OPS_H_ */ diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c b/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c index ebdf7c35a..2bda5a560 100644 --- a/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c +++ b/drivers/crypto/aesni_gcm/aesni_gcm_pmd.c @@ -24,7 +24,7 @@ aesni_gcm_set_session_parameters(const struct aesni_gcm_ops *gcm_ops, const struct rte_crypto_sym_xform *auth_xform; const struct rte_crypto_sym_xform *aead_xform; uint8_t key_length; - uint8_t *key; + const uint8_t *key; /* AES-GMAC */ if (xform->type == RTE_CRYPTO_SYM_XFORM_AUTH) { @@ -89,20 +89,20 @@ aesni_gcm_set_session_parameters(const struct aesni_gcm_ops *gcm_ops, /* Check key length and calculate GCM pre-compute. */ switch (key_length) { case 16: - sess->key = AESNI_GCM_KEY_128; + sess->key = GCM_KEY_128; break; case 24: - sess->key = AESNI_GCM_KEY_192; + sess->key = GCM_KEY_192; break; case 32: - sess->key = AESNI_GCM_KEY_256; + sess->key = GCM_KEY_256; break; default: AESNI_GCM_LOG(ERR, "Invalid key length"); return -EINVAL; } - gcm_ops[sess->key].precomp(key, &sess->gdata_key); + gcm_ops[sess->key].pre(key, &sess->gdata_key); /* Digest check */ if (sess->req_digest_length > 16) { @@ -195,6 +195,7 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op, uint32_t offset, data_offset, data_length; uint32_t part_len, total_len, data_len; uint8_t *tag; + unsigned int oop = 0; if (session->op == AESNI_GCM_OP_AUTHENTICATED_ENCRYPTION || session->op == AESNI_GCM_OP_AUTHENTICATED_DECRYPTION) { @@ -216,27 +217,30 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op, RTE_ASSERT(m_src != NULL); } + src = rte_pktmbuf_mtod_offset(m_src, uint8_t *, offset); + data_len = m_src->data_len - offset; part_len = (data_len < data_length) ? data_len : data_length; - /* Destination buffer is required when segmented source buffer */ - RTE_ASSERT((part_len == data_length) || - ((part_len != data_length) && - (sym_op->m_dst != NULL))); - /* Segmented destination buffer is not supported */ RTE_ASSERT((sym_op->m_dst == NULL) || ((sym_op->m_dst != NULL) && rte_pktmbuf_is_contiguous(sym_op->m_dst))); - - dst = sym_op->m_dst ? - rte_pktmbuf_mtod_offset(sym_op->m_dst, uint8_t *, - data_offset) : - rte_pktmbuf_mtod_offset(sym_op->m_src, uint8_t *, + /* In-place */ + if (sym_op->m_dst == NULL || (sym_op->m_dst == sym_op->m_src)) + dst = src; + /* Out-of-place */ + else { + oop = 1; + /* + * Segmented destination buffer is not supported if operation is + * Out-of-place + */ + RTE_ASSERT(rte_pktmbuf_is_contiguous(sym_op->m_dst)); + dst = rte_pktmbuf_mtod_offset(sym_op->m_dst, uint8_t *, data_offset); - - src = rte_pktmbuf_mtod_offset(m_src, uint8_t *, offset); + } iv_ptr = rte_crypto_op_ctod_offset(op, uint8_t *, session->iv.offset); @@ -254,12 +258,15 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op, total_len = data_length - part_len; while (total_len) { - dst += part_len; m_src = m_src->next; RTE_ASSERT(m_src != NULL); src = rte_pktmbuf_mtod(m_src, uint8_t *); + if (oop) + dst += part_len; + else + dst = src; part_len = (m_src->data_len < total_len) ? m_src->data_len : total_len; @@ -274,7 +281,7 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op, else tag = sym_op->aead.digest.data; - qp->ops[session->key].finalize(&session->gdata_key, + qp->ops[session->key].finalize_enc(&session->gdata_key, &qp->gdata_ctx, tag, session->gen_digest_length); @@ -291,12 +298,15 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op, total_len = data_length - part_len; while (total_len) { - dst += part_len; m_src = m_src->next; RTE_ASSERT(m_src != NULL); src = rte_pktmbuf_mtod(m_src, uint8_t *); + if (oop) + dst += part_len; + else + dst = src; part_len = (m_src->data_len < total_len) ? m_src->data_len : total_len; @@ -308,7 +318,7 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op, } tag = qp->temp_digest; - qp->ops[session->key].finalize(&session->gdata_key, + qp->ops[session->key].finalize_dec(&session->gdata_key, &qp->gdata_ctx, tag, session->gen_digest_length); @@ -322,7 +332,7 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op, tag = qp->temp_digest; else tag = sym_op->auth.digest.data; - qp->ops[session->key].finalize(&session->gdata_key, + qp->ops[session->key].finalize_enc(&session->gdata_key, &qp->gdata_ctx, tag, session->gen_digest_length); @@ -338,7 +348,7 @@ process_gcm_crypto_op(struct aesni_gcm_qp *qp, struct rte_crypto_op *op, * the bytes passed. */ tag = qp->temp_digest; - qp->ops[session->key].finalize(&session->gdata_key, + qp->ops[session->key].finalize_enc(&session->gdata_key, &qp->gdata_ctx, tag, session->gen_digest_length); @@ -487,12 +497,8 @@ aesni_gcm_create(const char *name, struct rte_cryptodev *dev; struct aesni_gcm_private *internals; enum aesni_gcm_vector_mode vector_mode; + MB_MGR *mb_mgr; - /* Check CPU for support for AES instruction set */ - if (!rte_cpu_get_flag_enabled(RTE_CPUFLAG_AES)) { - AESNI_GCM_LOG(ERR, "AES instructions not supported by CPU"); - return -EFAULT; - } dev = rte_cryptodev_pmd_create(name, &vdev->device, init_params); if (dev == NULL) { AESNI_GCM_LOG(ERR, "driver %s: create failed", @@ -501,7 +507,9 @@ aesni_gcm_create(const char *name, } /* Check CPU for supported vector instruction set */ - if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) + if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F)) + vector_mode = RTE_AESNI_GCM_AVX512; + else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) vector_mode = RTE_AESNI_GCM_AVX2; else if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX)) vector_mode = RTE_AESNI_GCM_AVX; @@ -517,27 +525,74 @@ aesni_gcm_create(const char *name, dev->feature_flags = RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO | RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING | - RTE_CRYPTODEV_FF_CPU_AESNI | + RTE_CRYPTODEV_FF_IN_PLACE_SGL | RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; + /* Check CPU for support for AES instruction set */ + if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AES)) + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AESNI; + else + AESNI_GCM_LOG(WARNING, "AES instructions not supported by CPU"); + + mb_mgr = alloc_mb_mgr(0); + if (mb_mgr == NULL) + return -ENOMEM; + switch (vector_mode) { case RTE_AESNI_GCM_SSE: dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_SSE; + init_mb_mgr_sse(mb_mgr); break; case RTE_AESNI_GCM_AVX: dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX; + init_mb_mgr_avx(mb_mgr); break; case RTE_AESNI_GCM_AVX2: dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; + init_mb_mgr_avx2(mb_mgr); break; - default: + case RTE_AESNI_GCM_AVX512: + dev->feature_flags |= RTE_CRYPTODEV_FF_CPU_AVX2; + init_mb_mgr_avx512(mb_mgr); break; + default: + AESNI_GCM_LOG(ERR, "Unsupported vector mode %u\n", vector_mode); + goto error_exit; } internals = dev->data->dev_private; internals->vector_mode = vector_mode; + internals->mb_mgr = mb_mgr; + + /* Set arch independent function pointers, based on key size */ + internals->ops[GCM_KEY_128].enc = mb_mgr->gcm128_enc; + internals->ops[GCM_KEY_128].dec = mb_mgr->gcm128_dec; + internals->ops[GCM_KEY_128].pre = mb_mgr->gcm128_pre; + internals->ops[GCM_KEY_128].init = mb_mgr->gcm128_init; + internals->ops[GCM_KEY_128].update_enc = mb_mgr->gcm128_enc_update; + internals->ops[GCM_KEY_128].update_dec = mb_mgr->gcm128_dec_update; + internals->ops[GCM_KEY_128].finalize_enc = mb_mgr->gcm128_enc_finalize; + internals->ops[GCM_KEY_128].finalize_dec = mb_mgr->gcm128_dec_finalize; + + internals->ops[GCM_KEY_192].enc = mb_mgr->gcm192_enc; + internals->ops[GCM_KEY_192].dec = mb_mgr->gcm192_dec; + internals->ops[GCM_KEY_192].pre = mb_mgr->gcm192_pre; + internals->ops[GCM_KEY_192].init = mb_mgr->gcm192_init; + internals->ops[GCM_KEY_192].update_enc = mb_mgr->gcm192_enc_update; + internals->ops[GCM_KEY_192].update_dec = mb_mgr->gcm192_dec_update; + internals->ops[GCM_KEY_192].finalize_enc = mb_mgr->gcm192_enc_finalize; + internals->ops[GCM_KEY_192].finalize_dec = mb_mgr->gcm192_dec_finalize; + + internals->ops[GCM_KEY_256].enc = mb_mgr->gcm256_enc; + internals->ops[GCM_KEY_256].dec = mb_mgr->gcm256_dec; + internals->ops[GCM_KEY_256].pre = mb_mgr->gcm256_pre; + internals->ops[GCM_KEY_256].init = mb_mgr->gcm256_init; + internals->ops[GCM_KEY_256].update_enc = mb_mgr->gcm256_enc_update; + internals->ops[GCM_KEY_256].update_dec = mb_mgr->gcm256_dec_update; + internals->ops[GCM_KEY_256].finalize_enc = mb_mgr->gcm256_enc_finalize; + internals->ops[GCM_KEY_256].finalize_dec = mb_mgr->gcm256_dec_finalize; internals->max_nb_queue_pairs = init_params->max_nb_queue_pairs; @@ -549,6 +604,14 @@ aesni_gcm_create(const char *name, #endif return 0; + +error_exit: + if (mb_mgr) + free_mb_mgr(mb_mgr); + + rte_cryptodev_pmd_destroy(dev); + + return -1; } static int @@ -576,6 +639,7 @@ static int aesni_gcm_remove(struct rte_vdev_device *vdev) { struct rte_cryptodev *cryptodev; + struct aesni_gcm_private *internals; const char *name; name = rte_vdev_device_name(vdev); @@ -586,6 +650,10 @@ aesni_gcm_remove(struct rte_vdev_device *vdev) if (cryptodev == NULL) return -ENODEV; + internals = cryptodev->data->dev_private; + + free_mb_mgr(internals->mb_mgr); + return rte_cryptodev_pmd_destroy(cryptodev); } diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c b/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c index c343a393f..f599fc3f7 100644 --- a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c +++ b/drivers/crypto/aesni_gcm/aesni_gcm_pmd_ops.c @@ -222,7 +222,7 @@ aesni_gcm_pmd_qp_setup(struct rte_cryptodev *dev, uint16_t qp_id, if (aesni_gcm_pmd_qp_set_unique_name(dev, qp)) goto qp_setup_cleanup; - qp->ops = (const struct aesni_gcm_ops *)gcm_ops[internals->vector_mode]; + qp->ops = (const struct aesni_gcm_ops *)internals->ops; qp->processed_pkts = aesni_gcm_pmd_qp_create_processed_pkts_ring(qp, qp_conf->nb_descriptors, socket_id); @@ -277,7 +277,7 @@ aesni_gcm_pmd_sym_session_configure(struct rte_cryptodev *dev __rte_unused, "Couldn't get object from session mempool"); return -ENOMEM; } - ret = aesni_gcm_set_session_parameters(gcm_ops[internals->vector_mode], + ret = aesni_gcm_set_session_parameters(internals->ops, sess_private_data, xform); if (ret != 0) { AESNI_GCM_LOG(ERR, "failed configure session parameters"); diff --git a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_private.h b/drivers/crypto/aesni_gcm/aesni_gcm_pmd_private.h index 92b041354..fd43df3cf 100644 --- a/drivers/crypto/aesni_gcm/aesni_gcm_pmd_private.h +++ b/drivers/crypto/aesni_gcm/aesni_gcm_pmd_private.h @@ -35,6 +35,10 @@ struct aesni_gcm_private { /**< Vector mode */ unsigned max_nb_queue_pairs; /**< Max number of queue pairs supported by device */ + MB_MGR *mb_mgr; + /**< Multi-buffer instance */ + struct aesni_gcm_ops ops[GCM_KEY_NUM]; + /**< Function pointer table of the gcm APIs */ }; struct aesni_gcm_qp {