From patchwork Fri Jul 3 11:09:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Zhang X-Patchwork-Id: 72984 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 90347A0519; Fri, 3 Jul 2020 13:09:37 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 053621D990; Fri, 3 Jul 2020 13:09:32 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 0020E1D739 for ; Fri, 3 Jul 2020 13:09:28 +0200 (CEST) IronPort-SDR: DFGhWh1icIlH9K8n42b3EIocFlVd3cy7eUux1mWDlqx2wbktcx1z+atAa4ct6n+GqHIdRTwmzJ i6OH6JEQKdtg== X-IronPort-AV: E=McAfee;i="6000,8403,9670"; a="126745842" X-IronPort-AV: E=Sophos;i="5.75,308,1589266800"; d="scan'208";a="126745842" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2020 04:09:28 -0700 IronPort-SDR: SzyIpwEob9W1VZ75Z+5C7qRxHPbssWiHK2c6LyFW3ZczRAElvdlhfE+VFTXdHZhYSYIywMbM3G 5Gvm/Ht190pw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,308,1589266800"; d="scan'208";a="304555978" Received: from silpixa00398673.ir.intel.com (HELO silpixa00398673.ger.corp.intel.com) ([10.237.223.136]) by fmsmga004.fm.intel.com with ESMTP; 03 Jul 2020 04:09:26 -0700 From: Fan Zhang To: dev@dpdk.org Cc: fiona.trahe@intel.com, akhil.goyal@nxp.com, thomas@monjalon.net, jerinjacobk@gmail.com, Fan Zhang , Piotr Bronowski Date: Fri, 3 Jul 2020 12:09:17 +0100 Message-Id: <20200703110923.26452-2-roy.fan.zhang@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200703110923.26452-1-roy.fan.zhang@intel.com> References: <20200625133138.42827-1-roy.fan.zhang@intel.com> <20200703110923.26452-1-roy.fan.zhang@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [dpdk-dev v3 1/4] cryptodev: add symmetric crypto data-path APIs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch adds data-path APIs to cryptodev. The APIs are organized as a data structure containing function pointers for different enqueue and dequeue operations. An added public API is added to obtain the function pointers and necessary queue pair data from the device queue pair. This patch depends on patch-72157 ("cryptodev: add function to check if qp was setup") Signed-off-by: Fan Zhang Signed-off-by: Piotr Bronowski --- lib/librte_cryptodev/rte_crypto_sym.h | 48 +++++ lib/librte_cryptodev/rte_cryptodev.c | 22 +++ lib/librte_cryptodev/rte_cryptodev.h | 173 +++++++++++++++++- lib/librte_cryptodev/rte_cryptodev_pmd.h | 12 +- .../rte_cryptodev_version.map | 4 + 5 files changed, 255 insertions(+), 4 deletions(-) diff --git a/lib/librte_cryptodev/rte_crypto_sym.h b/lib/librte_cryptodev/rte_crypto_sym.h index da961a19d..e237e3cfa 100644 --- a/lib/librte_cryptodev/rte_crypto_sym.h +++ b/lib/librte_cryptodev/rte_crypto_sym.h @@ -87,6 +87,54 @@ union rte_crypto_sym_ofs { } ofs; }; + +/** + * Asynchronous operation job descriptor. + * Used by HW crypto devices direct API call that supports such activity + **/ +struct rte_crypto_sym_job { + union { + /** + * When RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL bit is set in flags, sgl + * field is used as input data. Otherwise data_iova is + * used. + **/ + rte_iova_t data_iova; + struct rte_crypto_sgl *sgl; + }; + union { + /** + * Different than cryptodev ops, all ofs and len fields have + * the unit of bytes (including Snow3G/Kasumi/Zuc. + **/ + struct { + uint32_t cipher_ofs; + uint32_t cipher_len; + } cipher_only; + struct { + uint32_t auth_ofs; + uint32_t auth_len; + rte_iova_t digest_iova; + } auth_only; + struct { + uint32_t aead_ofs; + uint32_t aead_len; + rte_iova_t tag_iova; + uint8_t *aad; + rte_iova_t aad_iova; + } aead; + struct { + uint32_t cipher_ofs; + uint32_t cipher_len; + uint32_t auth_ofs; + uint32_t auth_len; + rte_iova_t digest_iova; + } chain; + }; + uint8_t *iv; + rte_iova_t iv_iova; +}; + /** Symmetric Cipher Algorithms */ enum rte_crypto_cipher_algorithm { RTE_CRYPTO_CIPHER_NULL = 1, diff --git a/lib/librte_cryptodev/rte_cryptodev.c b/lib/librte_cryptodev/rte_cryptodev.c index 705387b8b..5d5f84e27 100644 --- a/lib/librte_cryptodev/rte_cryptodev.c +++ b/lib/librte_cryptodev/rte_cryptodev.c @@ -1866,6 +1866,28 @@ rte_cryptodev_sym_cpu_crypto_process(uint8_t dev_id, return dev->dev_ops->sym_cpu_process(dev, sess, ofs, vec); } +int +rte_cryptodev_sym_get_hw_ops(uint8_t dev_id, uint16_t qp_id, + struct rte_crypto_hw_ops *hw_ops) +{ + struct rte_cryptodev *dev; + + if (!hw_ops) + return -EINVAL; + + memset(hw_ops, 0, sizeof(*hw_ops)); + + if (!rte_cryptodev_get_qp_status(dev_id, qp_id)) + return -EINVAL; + + dev = rte_cryptodev_pmd_get_dev(dev_id); + if (!(dev->feature_flags & RTE_CRYPTODEV_FF_SYM_HW_DIRECT_API) || + *dev->dev_ops->sym_get_hw_ops == NULL) + return -ENOTSUP; + + return dev->dev_ops->sym_get_hw_ops(dev, qp_id, hw_ops); +} + /** Initialise rte_crypto_op mempool element */ static void rte_crypto_op_init(struct rte_mempool *mempool, diff --git a/lib/librte_cryptodev/rte_cryptodev.h b/lib/librte_cryptodev/rte_cryptodev.h index d01a65825..7cd2095d7 100644 --- a/lib/librte_cryptodev/rte_cryptodev.h +++ b/lib/librte_cryptodev/rte_cryptodev.h @@ -466,7 +466,8 @@ rte_cryptodev_asym_get_xform_enum(enum rte_crypto_asym_xform_type *xform_enum, /**< Support symmetric session-less operations */ #define RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA (1ULL << 23) /**< Support operations on data which is not byte aligned */ - +#define RTE_CRYPTODEV_FF_SYM_HW_DIRECT_API (1ULL << 24) +/**< Support hardware accelerator specific raw data as input */ /** * Get the name of a crypto device feature flag @@ -737,7 +738,7 @@ rte_cryptodev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id, * - 1: qp was configured * - -ENODEV: device was not configured */ -int +__rte_experimental int rte_cryptodev_get_qp_status(uint8_t dev_id, uint16_t queue_pair_id); /** @@ -1348,6 +1349,174 @@ rte_cryptodev_sym_cpu_crypto_process(uint8_t dev_id, struct rte_cryptodev_sym_session *sess, union rte_crypto_sym_ofs ofs, struct rte_crypto_sym_vec *vec); + +/* HW direct symmetric crypto data-path APIs */ + +/* Bit-masks used for enqueuing job */ +#define RTE_CRYPTO_HW_ENQ_FLAG_START (1ULL << 0) +/**< Bit-mask to indicate the first job in a burst. With this bit set the + * driver may write but not read the drv_data buffer, otherwise the driver + * shall read and update the drv_data. + */ +#define RTE_CRYPTO_HW_ENQ_FLAG_SET_OPAQUE (1ULL << 1) +/**< Bit-mask to indicate write opaque pointer into HW crypto descriptor. */ +#define RTE_CRYPTO_HW_ENQ_FLAG_END (1ULL << 2) +/**< Bit-mask to indicate the last job in a burst. With this bit set the + * driver may read but not write the drv_data buffer, and kick the HW to + * start processing all jobs written. + */ +#define RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL (1ULL << 3) +/**< Bit-mask to indicate the input job is an SGL buffer */ + +/* Bit-masks used for dequeuing job */ +#define RTE_CRYPTO_HW_DEQ_FLAG_START (1ULL << 0) +/**< Bit-mask to indicate the first job to be dequeued. With this bit set the + * driver may write but not read the drv_data buffer, otherwise the driver + * shall read and update the drv_data. + */ +#define RTE_CRYPTO_HW_DEQ_FLAG_EXHAUST (1ULL << 1) +/**< Bit-mask to indicate dequeuing as many as n jobs in dequeue-many function. + * Without this bit once the driver found out the ready-to-dequeue jobs are + * not as many as n, it shall stop immediate, leave all processed jobs in the + * queue, and return the ready jobs in negative. With this bit set the + * function shall continue dequeue all done jobs and return the dequeued + * job count in positive. + */ + +/** + * Typedef for HW direct data-path enqueue callback function. + * + * @param qp Queue pair data. + * @param sess Cryptodev session. + * @param job Job data. + * @param opaque Opaque data to be written to queue descriptor + * when RTE_CRYPTO_HW_ENQ_SET_OPAQUE is + * set. + * @param drv_data User created temporary driver data for the + * driver to store and update data used between + * adjacent enqueues operations. + * @param flags Bitmask of RTE_CRYPTO_HW_ENQ_* flags + * @return + * - On success return 0 + * - On fail return -1 + **/ +typedef int (*rte_crypto_hw_enq_cb_fn)(void *qp, + struct rte_cryptodev_sym_session *sess, + struct rte_crypto_sym_job *job, void *opaque, uint64_t *drv_data, + uint64_t flags); + +/** + * Typedef for HW direct data-path dequeue one job callback function. + * + * @param qp Queue pair data. + * @param drv_data User created temporary driver data for the + * driver to store and update data used between + * adjacent enqueues operations. + * @param flags Bitmask of RTE_CRYPTO_HW_DEQ_* flags + * @param status The buffer for the driver to write operation + * status. + * @return + * - On success return the opaque data user write in enqueue (if any) and + * - status written as 1 when operation is successful. + * - status written as -1 when operation is failed (e.g. bad MAC) + * - On fail return NULL and status written as 0 when operation is still + * under processing. + **/ +typedef void * (*rte_crypto_hw_deq_one_cb_fn)(void *qp, uint64_t *drv_data, + uint64_t flags, int *status); + +/** + * Typedef that the user provided to deal with jobs' status when + * dequeue in a bulk. + * + * @param data User provided data. + * @param index Index number of the processed job. + * @param is_op_success Driver filled operation status. + **/ +typedef void (*rte_crpyto_hw_user_post_deq_cb_fn)(void *data, uint32_t index, + uint8_t is_op_success); + +/** + * Typedef for HW direct data-path dequeue bulk jobs callback function. + * + * @param qp Queue pair data. + * @param drv_data User created temporary driver data for the + * driver to store and update data used between + * adjacent enqueues operations. + * @param user_data User provided data to be passed into cb + * function. + * @param cb User provided callback functions to deal with + * driver returned job status. + * @param n The number of expected jobs to be dequeued. + * @param flags Bitmask of RTE_CRYPTO_HW_DEQ_* flags + * @param n_fail The buffer for driver to write the number of + * failed jobs. + * @return + * - Return the number of dequeued jobs. + **/ +typedef uint32_t (*rte_crypto_hw_deq_many_cb_fn)(void *qp, uint64_t *drv_data, + void *user_data, rte_crpyto_hw_user_post_deq_cb_fn cb, + uint32_t n, uint64_t flags, uint32_t *n_fail); +/** + * Typedef for querying HW the number of processed jobs. + * + * @param qp Queue pair data. + * @param nb_jobs The expected processed job number. + * @return + * - If the nb_jobs ready, return 1. + * - Otherwise return 0. + **/ +typedef int (*rte_crypto_hw_query_processed)(void *qp, uint32_t nb_jobs); + +/* Struct for user to perform HW specific enqueue/dequeue function calls */ +struct rte_crypto_hw_ops { + /* Driver written queue pair data pointer, should NOT be alterred by + * the user. + */ + void *qp; + /* Function handler to enqueue AEAD job */ + rte_crypto_hw_enq_cb_fn enqueue_aead; + /* Function handler to enqueue cipher only job */ + rte_crypto_hw_enq_cb_fn enqueue_cipher; + /* Function handler to enqueue auth only job */ + rte_crypto_hw_enq_cb_fn enqueue_auth; + /* Function handler to enqueue cipher + hash chaining job */ + rte_crypto_hw_enq_cb_fn enqueue_chain; + /* Function handler to query processed jobs */ + rte_crypto_hw_query_processed query_processed; + /* Function handler to dequeue one job and return opaque data stored */ + rte_crypto_hw_deq_one_cb_fn dequeue_one; + /* Function handler to dequeue many jobs */ + rte_crypto_hw_deq_many_cb_fn dequeue_many; + /* Reserved */ + void *reserved[8]; +}; + +/** + * Get the symmetric crypto hardware ops function pointers and queue pair data. + * + * @param dev_id The device identifier. + * @param qp_id The index of the queue pair from which to retrieve + * processed packets. The value must be in the range + * [0, nb_queue_pair - 1] previously supplied to + * rte_cryptodev_configure(). + * @param hw_ops User provided rte_crypto_hw_ops buffer. + * + * @return + * - On success hw_ops will be written the HW crypto device's queue pair data + * and function pointers for data enqueue/dequeue. + * - On fail hw_ops is cleared and negative integer is returned. + */ +__rte_experimental +int +rte_cryptodev_sym_get_hw_ops( + uint8_t dev_id, uint16_t qp_id, + struct rte_crypto_hw_ops *hw_ops); +int +rte_cryptodev_sym_get_hw_ops( + uint8_t dev_id, uint16_t qp_id, + struct rte_crypto_hw_ops *hw_ops); + #ifdef __cplusplus } #endif diff --git a/lib/librte_cryptodev/rte_cryptodev_pmd.h b/lib/librte_cryptodev/rte_cryptodev_pmd.h index 81975d72b..28f75d1da 100644 --- a/lib/librte_cryptodev/rte_cryptodev_pmd.h +++ b/lib/librte_cryptodev/rte_cryptodev_pmd.h @@ -316,6 +316,10 @@ typedef uint32_t (*cryptodev_sym_cpu_crypto_process_t) (struct rte_cryptodev *dev, struct rte_cryptodev_sym_session *sess, union rte_crypto_sym_ofs ofs, struct rte_crypto_sym_vec *vec); +struct rte_crypto_hw_ops; + +typedef int (*cryptodev_sym_hw_get_ops_t)(struct rte_cryptodev *dev, + uint16_t qp_id, struct rte_crypto_hw_ops *hw_ops); /** Crypto device operations function pointer table */ struct rte_cryptodev_ops { @@ -348,8 +352,12 @@ struct rte_cryptodev_ops { /**< Clear a Crypto sessions private data. */ cryptodev_asym_free_session_t asym_session_clear; /**< Clear a Crypto sessions private data. */ - cryptodev_sym_cpu_crypto_process_t sym_cpu_process; - /**< process input data synchronously (cpu-crypto). */ + union { + cryptodev_sym_cpu_crypto_process_t sym_cpu_process; + /**< process input data synchronously (cpu-crypto). */ + cryptodev_sym_hw_get_ops_t sym_get_hw_ops; + /**< Get HW crypto data-path call back functions and data */ + }; }; diff --git a/lib/librte_cryptodev/rte_cryptodev_version.map b/lib/librte_cryptodev/rte_cryptodev_version.map index 07a2d2f02..56f5684c8 100644 --- a/lib/librte_cryptodev/rte_cryptodev_version.map +++ b/lib/librte_cryptodev/rte_cryptodev_version.map @@ -85,6 +85,7 @@ EXPERIMENTAL { rte_cryptodev_sym_session_set_user_data; rte_crypto_asym_op_strings; rte_crypto_asym_xform_strings; + rte_cryptodev_get_qp_status; # added in 20.05 __rte_cryptodev_trace_configure; @@ -103,4 +104,7 @@ EXPERIMENTAL { __rte_cryptodev_trace_asym_session_clear; __rte_cryptodev_trace_dequeue_burst; __rte_cryptodev_trace_enqueue_burst; + + # added in 20.08 + rte_cryptodev_sym_get_hw_ops; }; From patchwork Fri Jul 3 11:09:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Zhang X-Patchwork-Id: 72986 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id EF31AA0519; Fri, 3 Jul 2020 13:09:58 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 270F81DAAE; Fri, 3 Jul 2020 13:09:35 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 051F91D9D2 for ; Fri, 3 Jul 2020 13:09:32 +0200 (CEST) IronPort-SDR: kkC7ydYcgz46N1sEe03AsLuH5Dg5OIPt7+7cn243q8cpYvejpp4skLRzA1CmvK/Veh1JTflgnv 2dW3Isfr9uTA== X-IronPort-AV: E=McAfee;i="6000,8403,9670"; a="126745847" X-IronPort-AV: E=Sophos;i="5.75,308,1589266800"; d="scan'208";a="126745847" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2020 04:09:32 -0700 IronPort-SDR: kHiCYkE0y9lt+oZIZVc2ClUKvwLemqjRr0uYzuptiorEQjbGKhmi6X0csPQ8kZb30Q3xXfHH6o IZ9PPiqrH7Xg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,308,1589266800"; d="scan'208";a="304555990" Received: from silpixa00398673.ir.intel.com (HELO silpixa00398673.ger.corp.intel.com) ([10.237.223.136]) by fmsmga004.fm.intel.com with ESMTP; 03 Jul 2020 04:09:30 -0700 From: Fan Zhang To: dev@dpdk.org Cc: fiona.trahe@intel.com, akhil.goyal@nxp.com, thomas@monjalon.net, jerinjacobk@gmail.com, Fan Zhang Date: Fri, 3 Jul 2020 12:09:19 +0100 Message-Id: <20200703110923.26452-4-roy.fan.zhang@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200703110923.26452-1-roy.fan.zhang@intel.com> References: <20200625133138.42827-1-roy.fan.zhang@intel.com> <20200703110923.26452-1-roy.fan.zhang@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [dpdk-dev v3 2/4] crypto/qat: add support to direct data-path APIs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch add symmetric crypto data-path APIs support to QAT-SYM PMD. Signed-off-by: Fan Zhang --- drivers/common/qat/Makefile | 2 + drivers/common/qat/qat_qp.c | 4 +- drivers/common/qat/qat_qp.h | 3 + drivers/crypto/qat/meson.build | 1 + drivers/crypto/qat/qat_sym.c | 1 - drivers/crypto/qat/qat_sym_job.c | 661 +++++++++++++++++++++++++++++++ drivers/crypto/qat/qat_sym_job.h | 12 + drivers/crypto/qat/qat_sym_pmd.c | 7 +- 8 files changed, 686 insertions(+), 5 deletions(-) create mode 100644 drivers/crypto/qat/qat_sym_job.c create mode 100644 drivers/crypto/qat/qat_sym_job.h diff --git a/drivers/common/qat/Makefile b/drivers/common/qat/Makefile index 28bd5668f..6655fd2bc 100644 --- a/drivers/common/qat/Makefile +++ b/drivers/common/qat/Makefile @@ -39,6 +39,8 @@ ifeq ($(CONFIG_RTE_LIBRTE_PMD_QAT_SYM),y) SRCS-y += qat_sym.c SRCS-y += qat_sym_session.c SRCS-y += qat_sym_pmd.c + SRCS-y += qat_sym_job.c + build_qat = yes endif endif diff --git a/drivers/common/qat/qat_qp.c b/drivers/common/qat/qat_qp.c index 8e6dd04eb..06e2d8c8a 100644 --- a/drivers/common/qat/qat_qp.c +++ b/drivers/common/qat/qat_qp.c @@ -547,8 +547,8 @@ txq_write_tail(struct qat_qp *qp, struct qat_queue *q) { q->csr_tail = q->tail; } -static inline -void rxq_free_desc(struct qat_qp *qp, struct qat_queue *q) +void +rxq_free_desc(struct qat_qp *qp, struct qat_queue *q) { uint32_t old_head, new_head; uint32_t max_head; diff --git a/drivers/common/qat/qat_qp.h b/drivers/common/qat/qat_qp.h index 575d69059..8add1b049 100644 --- a/drivers/common/qat/qat_qp.h +++ b/drivers/common/qat/qat_qp.h @@ -116,4 +116,7 @@ qat_comp_process_response(void **op __rte_unused, uint8_t *resp __rte_unused, void *op_cookie __rte_unused, uint64_t *dequeue_err_count __rte_unused); +void +rxq_free_desc(struct qat_qp *qp, struct qat_queue *q); + #endif /* _QAT_QP_H_ */ diff --git a/drivers/crypto/qat/meson.build b/drivers/crypto/qat/meson.build index fc65923a7..8a3921293 100644 --- a/drivers/crypto/qat/meson.build +++ b/drivers/crypto/qat/meson.build @@ -13,6 +13,7 @@ if dep.found() qat_sources += files('qat_sym_pmd.c', 'qat_sym.c', 'qat_sym_session.c', + 'qat_sym_job.c', 'qat_asym_pmd.c', 'qat_asym.c') qat_ext_deps += dep diff --git a/drivers/crypto/qat/qat_sym.c b/drivers/crypto/qat/qat_sym.c index 25b6dd5f4..609180d3f 100644 --- a/drivers/crypto/qat/qat_sym.c +++ b/drivers/crypto/qat/qat_sym.c @@ -336,7 +336,6 @@ qat_sym_build_request(void *in_op, uint8_t *out_msg, set_cipher_iv(ctx->cipher_iv.length, ctx->cipher_iv.offset, cipher_param, op, qat_req); - } else if (ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC) { diff --git a/drivers/crypto/qat/qat_sym_job.c b/drivers/crypto/qat/qat_sym_job.c new file mode 100644 index 000000000..7c0913459 --- /dev/null +++ b/drivers/crypto/qat/qat_sym_job.c @@ -0,0 +1,661 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2015-2019 Intel Corporation + */ + +#include + +#include "adf_transport_access_macros.h" +#include "icp_qat_fw.h" +#include "icp_qat_fw_la.h" + +#include "qat_sym.h" +#include "qat_sym_pmd.h" +#include "qat_sym_session.h" +#include "qat_qp.h" +#include "qat_sym_job.h" + +static __rte_always_inline int +qat_sym_frame_fill_sgl(struct qat_qp *qp, struct icp_qat_fw_la_bulk_req *req, + struct rte_crypto_sgl *sgl, uint32_t max_len) +{ + struct qat_queue *tx_queue = &qp->tx_q; + struct qat_sym_op_cookie *cookie; + struct qat_sgl *list; + int64_t len = max_len; + uint32_t i; + + if (!sgl) + return -EINVAL; + if (sgl->num < 2 || sgl->num > QAT_SYM_SGL_MAX_NUMBER || !sgl->vec) + return -EINVAL; + + ICP_QAT_FW_COMN_PTR_TYPE_SET(req->comn_hdr.comn_req_flags, + QAT_COMN_PTR_TYPE_SGL); + cookie = qp->op_cookies[tx_queue->tail >> tx_queue->trailz]; + list = (struct qat_sgl *)&cookie->qat_sgl_src; + + for (i = 0; i < sgl->num && len > 0; i++) { + list->buffers[i].len = RTE_MIN(sgl->vec[i].len, len); + list->buffers[i].resrvd = 0; + list->buffers[i].addr = sgl->vec[i].iova; + len -= list->buffers[i].len; + } + + if (unlikely(len > 0)) + return -1; + + list->num_bufs = i; + req->comn_mid.src_data_addr = req->comn_mid.dest_data_addr = + cookie->qat_sgl_src_phys_addr; + req->comn_mid.src_length = req->comn_mid.dst_length = 0; + return 0; +} + +static __rte_always_inline void +qat_sym_set_cipher_param(struct icp_qat_fw_la_cipher_req_params *cipher_param, + uint32_t cipher_ofs, uint32_t cipher_len) +{ + cipher_param->cipher_offset = cipher_ofs; + cipher_param->cipher_length = cipher_len; +} + +static __rte_always_inline void +qat_sym_set_auth_param(struct icp_qat_fw_la_auth_req_params *auth_param, + uint32_t auth_ofs, uint32_t auth_len, + rte_iova_t digest_iova, rte_iova_t aad_iova) +{ + auth_param->auth_off = auth_ofs; + auth_param->auth_len = auth_len; + auth_param->auth_res_addr = digest_iova; + auth_param->u1.aad_adr = aad_iova; +} + +static __rte_always_inline int +qat_sym_job_enqueue_aead(void *qat_sym_qp, + struct rte_cryptodev_sym_session *session, + struct rte_crypto_sym_job *job, void *opaque, uint64_t *drv_data, + uint64_t flags) +{ + struct qat_qp *qp = qat_sym_qp; + struct qat_queue *tx_queue = &qp->tx_q; + struct qat_sym_session *ctx; + register struct icp_qat_fw_la_bulk_req *req; + struct icp_qat_fw_la_cipher_req_params *cipher_param; + struct icp_qat_fw_la_auth_req_params *auth_param; + uint32_t t; + /* In case of AES-CCM this may point to user selected + * memory or iv offset in cypto_op + */ + uint8_t *aad_data; + /* This is true AAD length, it not includes 18 bytes of + * preceding data + */ + uint8_t aad_ccm_real_len; + uint8_t aad_len_field_sz; + uint32_t msg_len_be; + rte_iova_t aad_iova; + uint8_t q; + + ctx = (struct qat_sym_session *)get_sym_session_private_data( + session, cryptodev_qat_driver_id); + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_START) != 0)) { + t = tx_queue->tail; + req = (struct icp_qat_fw_la_bulk_req *)( + (uint8_t *)tx_queue->base_addr + t); + rte_mov128((uint8_t *)req, (const uint8_t *)&(ctx->fw_req)); + req->comn_mid.opaque_data = (uintptr_t)opaque; + } else { + t = (uint32_t)*drv_data; + req = (struct icp_qat_fw_la_bulk_req *)( + (uint8_t *)tx_queue->base_addr + t); + rte_mov128((uint8_t *)req, (const uint8_t *)&(ctx->fw_req)); + } + + cipher_param = (void *)&req->serv_specif_rqpars; + auth_param = (void *)((uint8_t *)cipher_param + + ICP_QAT_FW_HASH_REQUEST_PARAMETERS_OFFSET); + + req->comn_mid.src_data_addr = req->comn_mid.dest_data_addr = + job->data_iova; + req->comn_mid.src_length = req->comn_mid.dst_length = + job->aead.aead_ofs + job->aead.aead_len; + + switch (ctx->qat_hash_alg) { + case ICP_QAT_HW_AUTH_ALGO_GALOIS_128: + case ICP_QAT_HW_AUTH_ALGO_GALOIS_64: + ICP_QAT_FW_LA_GCM_IV_LEN_FLAG_SET( + req->comn_hdr.serv_specif_flags, + ICP_QAT_FW_LA_GCM_IV_LEN_12_OCTETS); + rte_memcpy_generic(cipher_param->u.cipher_IV_array, + job->iv, ctx->cipher_iv.length); + aad_iova = job->aead.aad_iova; + break; + case ICP_QAT_HW_AUTH_ALGO_AES_CBC_MAC: + aad_data = job->aead.aad; + aad_iova = job->aead.aad_iova; + aad_ccm_real_len = 0; + aad_len_field_sz = 0; + msg_len_be = rte_bswap32(job->aead.aead_len); + + if (ctx->aad_len > ICP_QAT_HW_CCM_AAD_DATA_OFFSET) { + aad_len_field_sz = ICP_QAT_HW_CCM_AAD_LEN_INFO; + aad_ccm_real_len = ctx->aad_len - + ICP_QAT_HW_CCM_AAD_B0_LEN - + ICP_QAT_HW_CCM_AAD_LEN_INFO; + } else { + aad_data = job->iv; + aad_iova = job->iv_iova; + } + + q = ICP_QAT_HW_CCM_NQ_CONST - ctx->cipher_iv.length; + aad_data[0] = ICP_QAT_HW_CCM_BUILD_B0_FLAGS(aad_len_field_sz, + ctx->digest_length, q); + if (q > ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE) { + memcpy(aad_data + ctx->cipher_iv.length + + ICP_QAT_HW_CCM_NONCE_OFFSET + + (q - ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE), + (uint8_t *)&msg_len_be, + ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE); + } else { + memcpy(aad_data + ctx->cipher_iv.length + + ICP_QAT_HW_CCM_NONCE_OFFSET, + (uint8_t *)&msg_len_be + + (ICP_QAT_HW_CCM_MSG_LEN_MAX_FIELD_SIZE + - q), q); + } + + if (aad_len_field_sz > 0) { + *(uint16_t *)&aad_data[ICP_QAT_HW_CCM_AAD_B0_LEN] = + rte_bswap16(aad_ccm_real_len); + + if ((aad_ccm_real_len + aad_len_field_sz) + % ICP_QAT_HW_CCM_AAD_B0_LEN) { + uint8_t pad_len = 0; + uint8_t pad_idx = 0; + + pad_len = ICP_QAT_HW_CCM_AAD_B0_LEN - + ((aad_ccm_real_len + aad_len_field_sz) % + ICP_QAT_HW_CCM_AAD_B0_LEN); + pad_idx = ICP_QAT_HW_CCM_AAD_B0_LEN + + aad_ccm_real_len + aad_len_field_sz; + memset(&aad_data[pad_idx], 0, pad_len); + } + + rte_memcpy(((uint8_t *)cipher_param->u.cipher_IV_array) + + ICP_QAT_HW_CCM_NONCE_OFFSET, + job->iv + ICP_QAT_HW_CCM_NONCE_OFFSET, + ctx->cipher_iv.length); + *(uint8_t *)&cipher_param->u.cipher_IV_array[0] = + q - ICP_QAT_HW_CCM_NONCE_OFFSET; + + if (aad_len_field_sz) + rte_memcpy(job->aead.aad + + ICP_QAT_HW_CCM_NONCE_OFFSET, + job->iv + ICP_QAT_HW_CCM_NONCE_OFFSET, + ctx->cipher_iv.length); + + } + break; + default: + return -1; + } + + qat_sym_set_cipher_param(cipher_param, job->aead.aead_ofs, + job->aead.aead_len); + qat_sym_set_auth_param(auth_param, job->aead.aead_ofs, + job->aead.aead_len, job->aead.tag_iova, aad_iova); + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL) != 0)) { + int ret = qat_sym_frame_fill_sgl(qp, req, job->sgl, + job->aead.aead_ofs + job->aead.aead_len); + if (unlikely(ret < 0)) + return -1; + } + + if (ctx->is_single_pass) { + cipher_param->spc_aad_addr = aad_iova; + cipher_param->spc_auth_res_addr = job->aead.tag_iova; + } + + qp->enqueued++; + qp->stats.enqueued_count++; + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_END) != 0)) { + tx_queue->tail = (t + tx_queue->msg_size) & + tx_queue->modulo_mask; + WRITE_CSR_RING_TAIL(qp->mmap_bar_addr, + tx_queue->hw_bundle_number, + tx_queue->hw_queue_number, + tx_queue->tail); + tx_queue->csr_tail = tx_queue->tail; + } else + *drv_data = (t + tx_queue->msg_size) & tx_queue->modulo_mask; + + return 0; +} + + +static __rte_always_inline int +qat_sym_job_enqueue_cipher(void *qat_sym_qp, + struct rte_cryptodev_sym_session *session, + struct rte_crypto_sym_job *job, void *opaque, uint64_t *drv_data, + uint64_t flags) +{ + struct qat_qp *qp = qat_sym_qp; + struct qat_queue *tx_queue = &qp->tx_q; + struct qat_sym_session *ctx; + struct icp_qat_fw_la_bulk_req *req; + struct icp_qat_fw_la_cipher_req_params *cipher_param; + uint32_t t; + + ctx = (struct qat_sym_session *)get_sym_session_private_data( + session, cryptodev_qat_driver_id); + if (unlikely(ctx->bpi_ctx)) { + QAT_DP_LOG(ERR, "DOCSIS is not supported"); + return -1; + } + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_START) != 0)) { + t = tx_queue->tail; + req = (struct icp_qat_fw_la_bulk_req *)( + (uint8_t *)tx_queue->base_addr + t); + rte_mov128((uint8_t *)req, (const uint8_t *)&(ctx->fw_req)); + req->comn_mid.opaque_data = (uintptr_t)opaque; + } else { + t = (uint32_t)*drv_data; + req = (struct icp_qat_fw_la_bulk_req *)( + (uint8_t *)tx_queue->base_addr + t); + rte_mov128((uint8_t *)req, (const uint8_t *)&(ctx->fw_req)); + } + + cipher_param = (void *)&req->serv_specif_rqpars; + + req->comn_mid.src_data_addr = req->comn_mid.dest_data_addr = + job->data_iova; + req->comn_mid.src_length = req->comn_mid.dst_length = + job->cipher_only.cipher_ofs + + job->cipher_only.cipher_len; + + /* cipher IV */ + rte_memcpy_generic(cipher_param->u.cipher_IV_array, + job->iv, ctx->cipher_iv.length); + qat_sym_set_cipher_param(cipher_param, job->cipher_only.cipher_ofs, + job->cipher_only.cipher_len); + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL) != 0)) { + int ret = qat_sym_frame_fill_sgl(qp, req, job->sgl, + job->cipher_only.cipher_ofs + + job->cipher_only.cipher_len); + if (unlikely(ret < 0)) + return -1; + } + + qp->enqueued++; + qp->stats.enqueued_count++; + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_END) != 0)) { + tx_queue->tail = (t + tx_queue->msg_size) & + tx_queue->modulo_mask; + WRITE_CSR_RING_TAIL(qp->mmap_bar_addr, + tx_queue->hw_bundle_number, + tx_queue->hw_queue_number, + tx_queue->tail); + tx_queue->csr_tail = tx_queue->tail; + } else + *drv_data = (t + tx_queue->msg_size) & tx_queue->modulo_mask; + + return 0; +} + +static __rte_always_inline int +qat_sym_job_enqueue_auth(void *qat_sym_qp, + struct rte_cryptodev_sym_session *session, + struct rte_crypto_sym_job *job, void *opaque, uint64_t *drv_data, + uint64_t flags) +{ + struct qat_qp *qp = qat_sym_qp; + struct qat_queue *tx_queue = &qp->tx_q; + struct qat_sym_session *ctx; + struct icp_qat_fw_la_bulk_req *req; + struct icp_qat_fw_la_auth_req_params *auth_param; + uint32_t t; + + ctx = (struct qat_sym_session *)get_sym_session_private_data( + session, cryptodev_qat_driver_id); + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_START) != 0)) { + t = tx_queue->tail; + req = (struct icp_qat_fw_la_bulk_req *)( + (uint8_t *)tx_queue->base_addr + t); + rte_mov128((uint8_t *)req, (const uint8_t *)&(ctx->fw_req)); + req->comn_mid.opaque_data = (uintptr_t)opaque; + } else { + t = (uint32_t)*drv_data; + req = (struct icp_qat_fw_la_bulk_req *)( + (uint8_t *)tx_queue->base_addr + t); + rte_mov128((uint8_t *)req, (const uint8_t *)&(ctx->fw_req)); + } + + auth_param = (void *)((uint8_t *)&req->serv_specif_rqpars + + ICP_QAT_FW_HASH_REQUEST_PARAMETERS_OFFSET); + + req->comn_mid.src_data_addr = req->comn_mid.dest_data_addr = + job->data_iova; + req->comn_mid.src_length = req->comn_mid.dst_length = + job->auth_only.auth_ofs + job->auth_only.auth_len; + + /* auth */ + qat_sym_set_auth_param(auth_param, job->auth_only.auth_ofs, + job->auth_only.auth_len, job->auth_only.digest_iova, 0); + + switch (ctx->qat_hash_alg) { + case ICP_QAT_HW_AUTH_ALGO_SNOW_3G_UIA2: + case ICP_QAT_HW_AUTH_ALGO_KASUMI_F9: + case ICP_QAT_HW_AUTH_ALGO_ZUC_3G_128_EIA3: + auth_param->u1.aad_adr = job->iv_iova; + break; + case ICP_QAT_HW_AUTH_ALGO_GALOIS_128: + case ICP_QAT_HW_AUTH_ALGO_GALOIS_64: + QAT_DP_LOG(ERR, "GMAC as chained auth algo is not supported"); + return -1; + default: + break; + } + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL) != 0)) { + int ret = qat_sym_frame_fill_sgl(qp, req, job->sgl, + job->auth_only.auth_ofs + + job->auth_only.auth_len); + if (unlikely(ret < 0)) + return -1; + } + + qp->enqueued++; + qp->stats.enqueued_count++; + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_END) != 0)) { + tx_queue->tail = (t + tx_queue->msg_size) & + tx_queue->modulo_mask; + WRITE_CSR_RING_TAIL(qp->mmap_bar_addr, + tx_queue->hw_bundle_number, + tx_queue->hw_queue_number, + tx_queue->tail); + tx_queue->csr_tail = tx_queue->tail; + } else + *drv_data = (t + tx_queue->msg_size) & tx_queue->modulo_mask; + + return 0; +} + +static __rte_always_inline int +qat_sym_job_enqueue_chain(void *qat_sym_qp, + struct rte_cryptodev_sym_session *session, + struct rte_crypto_sym_job *job, void *opaque, uint64_t *drv_data, + uint64_t flags) +{ + struct qat_qp *qp = qat_sym_qp; + struct qat_queue *tx_queue = &qp->tx_q; + struct qat_sym_session *ctx; + struct icp_qat_fw_la_bulk_req *req; + struct icp_qat_fw_la_cipher_req_params *cipher_param; + struct icp_qat_fw_la_auth_req_params *auth_param; + uint32_t min_ofs = RTE_MIN(job->chain.cipher_ofs, job->chain.auth_ofs); + uint32_t max_len = RTE_MAX(job->chain.cipher_len, job->chain.auth_len); + rte_iova_t auth_iova_end; + uint32_t t; + + ctx = (struct qat_sym_session *)get_sym_session_private_data( + session, cryptodev_qat_driver_id); + if (unlikely(ctx->bpi_ctx)) { + QAT_DP_LOG(ERR, "DOCSIS is not supported"); + return -1; + } + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_START) != 0)) { + t = tx_queue->tail; + req = (struct icp_qat_fw_la_bulk_req *)( + (uint8_t *)tx_queue->base_addr + t); + rte_mov128((uint8_t *)req, (const uint8_t *)&(ctx->fw_req)); + req->comn_mid.opaque_data = (uintptr_t)opaque; + } else { + t = (uint32_t)*drv_data; + req = (struct icp_qat_fw_la_bulk_req *)( + (uint8_t *)tx_queue->base_addr + t); + rte_mov128((uint8_t *)req, (const uint8_t *)&(ctx->fw_req)); + } + + cipher_param = (void *)&req->serv_specif_rqpars; + auth_param = (void *)((uint8_t *)cipher_param + + ICP_QAT_FW_HASH_REQUEST_PARAMETERS_OFFSET); + + req->comn_mid.src_data_addr = + req->comn_mid.dest_data_addr = job->data_iova; + req->comn_mid.src_length = req->comn_mid.dst_length = min_ofs + max_len; + + /* cipher IV */ + rte_memcpy_generic(cipher_param->u.cipher_IV_array, + job->iv, ctx->cipher_iv.length); + qat_sym_set_cipher_param(cipher_param, job->chain.cipher_ofs, + job->chain.cipher_len); + + /* auth */ + qat_sym_set_auth_param(auth_param, job->chain.auth_ofs, + job->chain.auth_len, job->chain.digest_iova, 0); + + switch (ctx->qat_hash_alg) { + case ICP_QAT_HW_AUTH_ALGO_SNOW_3G_UIA2: + case ICP_QAT_HW_AUTH_ALGO_KASUMI_F9: + case ICP_QAT_HW_AUTH_ALGO_ZUC_3G_128_EIA3: + auth_param->u1.aad_adr = job->iv_iova; + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL) != 0)) { + uint32_t len = job->chain.auth_ofs + + job->chain.auth_len; + struct rte_crypto_vec *vec = job->sgl->vec; + int auth_end_get = 0; + while (len) { + if (len <= vec->len) { + auth_iova_end = vec->iova + len; + auth_end_get = 1; + break; + } + len -= vec->len; + vec++; + } + if (!auth_end_get) { + QAT_DP_LOG(ERR, "Failed to get auth end"); + return -1; + } + } else + auth_iova_end = job->data_iova + job->chain.auth_ofs + + job->chain.auth_len; + + /* Then check if digest-encrypted conditions are met */ + if ((auth_param->auth_off + auth_param->auth_len < + cipher_param->cipher_offset + + cipher_param->cipher_length) && + (job->chain.digest_iova == auth_iova_end)) { + /* Handle partial digest encryption */ + if (cipher_param->cipher_offset + + cipher_param->cipher_length < + auth_param->auth_off + + auth_param->auth_len + + ctx->digest_length) + req->comn_mid.dst_length = + req->comn_mid.src_length = + auth_param->auth_off + + auth_param->auth_len + + ctx->digest_length; + struct icp_qat_fw_comn_req_hdr *header = + &req->comn_hdr; + ICP_QAT_FW_LA_DIGEST_IN_BUFFER_SET( + header->serv_specif_flags, + ICP_QAT_FW_LA_DIGEST_IN_BUFFER); + } + break; + case ICP_QAT_HW_AUTH_ALGO_GALOIS_128: + case ICP_QAT_HW_AUTH_ALGO_GALOIS_64: + QAT_DP_LOG(ERR, "GMAC as chained auth algo is not supported"); + return -1; + default: + break; + } + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL) != 0)) { + int ret = qat_sym_frame_fill_sgl(qp, req, job->sgl, + min_ofs + max_len); + if (unlikely(ret < 0)) + return -1; + } + + qp->enqueued++; + qp->stats.enqueued_count++; + + if (unlikely((flags & RTE_CRYPTO_HW_ENQ_FLAG_END) != 0)) { + tx_queue->tail = (t + tx_queue->msg_size) & + tx_queue->modulo_mask; + WRITE_CSR_RING_TAIL(qp->mmap_bar_addr, + tx_queue->hw_bundle_number, + tx_queue->hw_queue_number, + tx_queue->tail); + tx_queue->csr_tail = tx_queue->tail; + } else + *drv_data = (t + tx_queue->msg_size) & tx_queue->modulo_mask; + + return 0; +} + +#define get_rx_queue_message_at_index(q, h, i) \ + (void *)((uint8_t *)q->base_addr + ((h + q->msg_size * (i)) & \ + q->modulo_mask)) + +static __rte_always_inline int +qat_is_rx_msg_ok(struct icp_qat_fw_comn_resp *resp_msg) +{ + return ICP_QAT_FW_COMN_STATUS_FLAG_OK == + ICP_QAT_FW_COMN_RESP_CRYPTO_STAT_GET( + resp_msg->comn_hdr.comn_status); +} + +static __rte_always_inline int +qat_sym_query_processed_jobs(void *qat_sym_qp, uint32_t nb_jobs) +{ + struct qat_qp *qp = qat_sym_qp; + struct qat_queue *rx_queue = &qp->rx_q; + struct icp_qat_fw_comn_resp *resp; + uint32_t head = (rx_queue->head + (nb_jobs - 1) * rx_queue->msg_size) & + rx_queue->modulo_mask; + + resp = (struct icp_qat_fw_comn_resp *)( + (uint8_t *)rx_queue->base_addr + head); + if (*(uint32_t *)resp == ADF_RING_EMPTY_SIG) + return 0; + + return 1; +} + +static __rte_always_inline void * +qat_sym_job_dequeue_one(void *qat_sym_qp, uint64_t *drv_data, uint64_t flags, + int *is_op_success) +{ + struct qat_qp *qp = qat_sym_qp; + struct qat_queue *rx_queue = &qp->rx_q; + struct icp_qat_fw_comn_resp *resp; + uint32_t head; + void *opaque; + + if (flags & RTE_CRYPTO_HW_DEQ_FLAG_START) + head = rx_queue->head; + else + head = (uint32_t)*drv_data; + + resp = (struct icp_qat_fw_comn_resp *)( + (uint8_t *)rx_queue->base_addr + head); + + if (unlikely(*(uint32_t *)resp == ADF_RING_EMPTY_SIG)) { + *is_op_success = 0; + return NULL; + } + + if (unlikely(qat_is_rx_msg_ok(resp) == 0)) + *is_op_success = -1; + else + *is_op_success = 1; + + opaque = (void *)(uintptr_t)resp->opaque_data; + + rx_queue->head = (head + rx_queue->msg_size) & rx_queue->modulo_mask; + rx_queue->nb_processed_responses++; + qp->dequeued++; + qp->stats.dequeued_count++; + if (rx_queue->nb_processed_responses > QAT_CSR_HEAD_WRITE_THRESH) + rxq_free_desc(qp, rx_queue); + + return opaque; +} + +static __rte_always_inline uint32_t +qat_sym_job_dequeue_n(void *qat_sym_qp, uint64_t *drv_data, + void *user_data, rte_crpyto_hw_user_post_deq_cb_fn cb, + uint32_t n, uint64_t flags, uint32_t *n_failed_jobs) +{ + struct qat_qp *qp = qat_sym_qp; + struct qat_queue *rx_queue = &qp->rx_q; + struct icp_qat_fw_comn_resp *resp; + uint32_t head, i; + uint32_t status, total_fail = 0; + + if (flags & RTE_CRYPTO_HW_DEQ_FLAG_START) + head = rx_queue->head; + else + head = (uint32_t)*drv_data; + + for (i = 0; i < n; i++) { + resp = (struct icp_qat_fw_comn_resp *)( + (uint8_t *)rx_queue->base_addr + head); + + if (unlikely(*(uint32_t *)resp == ADF_RING_EMPTY_SIG)) { + if (flags & RTE_CRYPTO_HW_DEQ_FLAG_EXHAUST) + break; + return -i; + } + + status = qat_is_rx_msg_ok(resp); + total_fail += status; + cb(user_data, i, status); + + head = (head + rx_queue->msg_size) & rx_queue->modulo_mask; + } + + rx_queue->head = head; + rx_queue->nb_processed_responses += i; + qp->dequeued += i; + qp->stats.dequeued_count += i; + if (rx_queue->nb_processed_responses > QAT_CSR_HEAD_WRITE_THRESH) + rxq_free_desc(qp, rx_queue); + *n_failed_jobs = total_fail; + + return i; +} + +int +qat_sym_get_ops(struct rte_cryptodev *dev, + uint16_t qp_id, struct rte_crypto_hw_ops *hw_ops) +{ + struct qat_qp *qp = dev->data->queue_pairs[qp_id]; + + if (qp->service_type != QAT_SERVICE_SYMMETRIC) + return -EINVAL; + + hw_ops->qp = (void *)qp; + hw_ops->enqueue_aead = qat_sym_job_enqueue_aead; + hw_ops->enqueue_cipher = qat_sym_job_enqueue_cipher; + hw_ops->enqueue_auth = qat_sym_job_enqueue_auth; + hw_ops->enqueue_chain = qat_sym_job_enqueue_chain; + hw_ops->dequeue_one = qat_sym_job_dequeue_one; + hw_ops->dequeue_many = qat_sym_job_dequeue_n; + hw_ops->query_processed = qat_sym_query_processed_jobs; + + return 0; +} diff --git a/drivers/crypto/qat/qat_sym_job.h b/drivers/crypto/qat/qat_sym_job.h new file mode 100644 index 000000000..b11aeb841 --- /dev/null +++ b/drivers/crypto/qat/qat_sym_job.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2015-2018 Intel Corporation + */ + +#ifndef _QAT_SYM_FRAME_H_ +#define _QAT_SYM_FRAME_H_ + +int +qat_sym_get_ops(struct rte_cryptodev *dev, + uint16_t qp_id, struct rte_crypto_hw_ops *hw_ops); + +#endif /* _QAT_SYM_FRAME_H_ */ diff --git a/drivers/crypto/qat/qat_sym_pmd.c b/drivers/crypto/qat/qat_sym_pmd.c index e887c880f..be9d73c0a 100644 --- a/drivers/crypto/qat/qat_sym_pmd.c +++ b/drivers/crypto/qat/qat_sym_pmd.c @@ -13,6 +13,7 @@ #include "qat_sym.h" #include "qat_sym_session.h" #include "qat_sym_pmd.h" +#include "qat_sym_job.h" #define MIXED_CRYPTO_MIN_FW_VER 0x04090000 @@ -234,7 +235,8 @@ static struct rte_cryptodev_ops crypto_qat_ops = { /* Crypto related operations */ .sym_session_get_size = qat_sym_session_get_private_size, .sym_session_configure = qat_sym_session_configure, - .sym_session_clear = qat_sym_session_clear + .sym_session_clear = qat_sym_session_clear, + .sym_get_hw_ops = qat_sym_get_ops, }; static uint16_t @@ -308,7 +310,8 @@ qat_sym_dev_create(struct qat_pci_device *qat_pci_dev, RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT | RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT | RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT | - RTE_CRYPTODEV_FF_DIGEST_ENCRYPTED; + RTE_CRYPTODEV_FF_DIGEST_ENCRYPTED | + RTE_CRYPTODEV_FF_SYM_HW_DIRECT_API; internals = cryptodev->data->dev_private; internals->qat_dev = qat_pci_dev; From patchwork Fri Jul 3 11:09:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Zhang X-Patchwork-Id: 72988 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5316FA0519; Fri, 3 Jul 2020 13:10:21 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0E2D21DB5D; Fri, 3 Jul 2020 13:09:40 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id A49161DACC for ; Fri, 3 Jul 2020 13:09:36 +0200 (CEST) IronPort-SDR: qyw/yxEGBaTyfD4rps+Fj2uYC8N/UZqijDEFQ2egsH+xaMg88Xi+DMQrOxpaxj8jkZqv0c1Fe3 3kS8iwSG9s8A== X-IronPort-AV: E=McAfee;i="6000,8403,9670"; a="126745855" X-IronPort-AV: E=Sophos;i="5.75,308,1589266800"; d="scan'208";a="126745855" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2020 04:09:36 -0700 IronPort-SDR: dpK6qkGjTwJk85QEnMihTNEoF9bvddrBkVUDG81Sr3fvkXMecdxgrhvom02tbJ1IOJLZIIyGPq Zm6XcA+cGrOQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,308,1589266800"; d="scan'208";a="304556001" Received: from silpixa00398673.ir.intel.com (HELO silpixa00398673.ger.corp.intel.com) ([10.237.223.136]) by fmsmga004.fm.intel.com with ESMTP; 03 Jul 2020 04:09:34 -0700 From: Fan Zhang To: dev@dpdk.org Cc: fiona.trahe@intel.com, akhil.goyal@nxp.com, thomas@monjalon.net, jerinjacobk@gmail.com, Fan Zhang Date: Fri, 3 Jul 2020 12:09:21 +0100 Message-Id: <20200703110923.26452-6-roy.fan.zhang@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200703110923.26452-1-roy.fan.zhang@intel.com> References: <20200625133138.42827-1-roy.fan.zhang@intel.com> <20200703110923.26452-1-roy.fan.zhang@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [dpdk-dev v3 3/3] doc: add cryptodev direct APIs guide X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch updates programmer's guide to demonstrate the usage and limitations of cryptodev symmetric crypto data-path APIs. Signed-off-by: Fan Zhang --- doc/guides/prog_guide/cryptodev_lib.rst | 266 ++++++++++++++++++++++++ doc/guides/rel_notes/release_20_08.rst | 8 + 2 files changed, 274 insertions(+) diff --git a/doc/guides/prog_guide/cryptodev_lib.rst b/doc/guides/prog_guide/cryptodev_lib.rst index c14f750fa..9900a593a 100644 --- a/doc/guides/prog_guide/cryptodev_lib.rst +++ b/doc/guides/prog_guide/cryptodev_lib.rst @@ -861,6 +861,272 @@ using one of the crypto PMDs available in DPDK. num_dequeued_ops); } while (total_num_dequeued_ops < num_enqueued_ops); +Cryptodev Direct Symmetric Crypto Data-path APIs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Direct symmetric crypto data-path APIs are a set of APIs that especially +provided for Symmetric HW Crypto PMD that provides fast data-path +enqueue/dequeue operations. The direct data-path APIs take advantage of +existing Cryptodev APIs for device, queue pairs, and session management. In +addition the user are required to get the queue pair pointer data and function +pointers. The APIs are provided as an advanced feature as an alternative +to ``rte_cryptodev_enqueue_burst`` and ``rte_cryptodev_dequeue_burst``. The +APIs are designed for the user to develop close-to-native performance symmetric +crypto data-path implementation for their applications that do not necessarily +depend on cryptodev operations and cryptodev operation mempools, or mbufs. + +Cryptodev PMDs who supports this feature will have +``RTE_CRYPTODEV_FF_SYM_HW_DIRECT_API`` feature flag presented. The user uses +``rte_cryptodev_sym_get_hw_ops`` function call to get all the function pointers +for different enqueue and dequeue operations, plus the device specific +queue pair data. After the ``rte_crypto_hw_ops`` structure is properly set by +the driver, the user can use the function pointers and the queue data pointers +in the structure to enqueue and dequeue crypto jobs. + +To simply the enqueue APIs a symmetric job structure is defined: + +.. code-block:: c + + /** + * Asynchronous operation job descriptor. + * Used by HW crypto devices direct API call that supports such activity + **/ + struct rte_crypto_sym_job { + union { + /** + * When RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL bit is set in flags, sgl + * field is used as input data. Otherwise data_iova is + * used. + **/ + rte_iova_t data_iova; + struct rte_crypto_sgl *sgl; + }; + union { + /** + * Different than cryptodev ops, all ofs and len fields have + * the unit of bytes (including Snow3G/Kasumi/Zuc. + **/ + struct { + uint32_t cipher_ofs; + uint32_t cipher_len; + } cipher_only; + struct { + uint32_t auth_ofs; + uint32_t auth_len; + rte_iova_t digest_iova; + } auth_only; + struct { + uint32_t aead_ofs; + uint32_t aead_len; + rte_iova_t tag_iova; + uint8_t *aad; + rte_iova_t aad_iova; + } aead; + struct { + uint32_t cipher_ofs; + uint32_t cipher_len; + uint32_t auth_ofs; + uint32_t auth_len; + rte_iova_t digest_iova; + } chain; + }; + uint8_t *iv; + rte_iova_t iv_iova; + }; + +Different than Cryptodev operation, the ``rte_crypto_sym_job`` structure +focuses only on the data field required for crypto PMD to execute a single job, +and is not supposed stored as opaque data. The user can freely allocate the +structure buffer from stack and reuse it to fill all jobs. + +To use the direct symmetric crypto APIs safely, the user has to carefully +set the correct fields in rte_crypto_sym_job structure, otherwise the +application or the system may crash. Also there are a few limitations to the +direct symmetric crypto APIs: + +* Only support in-place operations. +* APIs are NOT thread-safe. +* CANNOT mix the direct API's enqueue with rte_cryptodev_enqueue_burst, or + vice versa. + +The following sample code shows how to use Cryptodev direct API to process a +user defined frame with maximum 32 buffers with AES-CBC and HMAC-SHA chained +algorithm of a frame defined by user. + +See *DPDK API Reference* for details on each API definitions. + +.. code-block:: c + + #include + + #define FRAME_ELT_OK 0 + #define FRAME_ELT_FAIL 1 + #define FRAME_OK 0 + #define FRAME_SOME_ELT_ERROR 1 + #define FRAME_SIZE 32 + + /* Sample frame element struct */ + struct sample_frame_elt { + /* The status field of frame element */ + uint8_t status; + /* Pre-created and initialized cryptodev session */ + struct rte_cryptodev_sym_session *session; + union { + __rte_iova_t data; + struct rte_crypto_sgl sgl; + }; + uint32_t data_len; + __rte_iova_t digest; + uint8_t *iv; + uint8_t is_sgl; + }; + + /* Sample frame struct to describe up to 32 crypto jobs */ + struct sample_frame { + struct sample_frame_elt elts[FRAME_SIZE]; /**< All frame elements */ + uint32_t n_elts; /**< Number of elements */ + }; + + /* Global Cryptodev Direct API structure */ + static struct rte_crypto_hw_ops hw_ops; + + /* Initialization */ + static int + frame_operation_init( + uint8_t cryptodev_id, /**< Initialized cryptodev ID */ + uint16_t qp_id /**< Initialized queue pair ID */) + { + /* Get APIs */ + ret = rte_cryptodev_sym_get_hw_ops(cryptodev_id, qp_id, &hw_ops); + /* If the device does not support this feature or queue pair is not + initialized, return -1 */ + if (!ret) + return -1; + return 0; + } + + /* Frame enqueue function use direct AES-CBC-* + HMAC-SHA* API */ + static int + enqueue_frame_to_direct_api( + struct sample_frame *frame /**< Initialized user frame struct */) + { + struct rte_crypto_hw_ops hw_ops; + struct rte_crypto_sym_job job; + uint64_t drv_data, flags = 0; + uint32_t i; + int ret; + + /* Fill all sample frame element data into HW queue pair */ + for (i = 0; i < frame->n_elts; i++) { + struct sample_frame_elt *fe = &frame->elts[i]; + int ret; + + /* if it is the first element in the frame, set FIRST flag to + let the driver to know it is first frame and fill drv_data. */ + if (i == 0) + flags |= RTE_CRYPTO_HW_ENQ_FLAG_START; + else + flags &= ~RTE_CRYPTO_HW_ENQ_FLAG_START; + + /* if it is the last element in the frame, write LAST flag to + kick HW queue */ + if (i == frame->n_elts - 1) + flags |= RTE_CRYPTO_HW_ENQ_FLAG_LAST; + else + flags &= ~RTE_CRYPTO_HW_ENQ_FLAG_LAST; + + /* Fill the job data with frame element data */ + if (fe->is_sgl != 0) { + /* The buffer is a SGL buffer */ + job.sgl = &frame->sgl; + /* Set SGL flag */ + flags |= RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL; + } else { + job.data_iova = fe->data; + /* Unset SGL flag in the job */ + flags &= ~RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL; + } + + job.chain.cipher_ofs = job.chain.auth_ofs = 0; + job.chain.cipher_len = job.chain.auth_len = fe->data_len; + job.chain.digest_iova = fe->digest; + + job.iv = fe->iv; + + /* Call direct data-path enqueue chaining op API */ + ret = hw_ops.enqueue_chain(hw_ops.qp, fe->session, &job, + (void *frame), &drv_data, flags); + /** + * In case one element is failed to be enqueued, simply abandon + * enqueuing the whole frame. + **/ + if (!ret) + return -1; + + /** + * To this point the frame is enqueued. The job buffer can be + * safely reused for enqueuing next frame element. + **/ + } + + return 0; + } + + /** + * Sample function to write frame element status field based on + * driver returned operation result. The function return and parameter + * should follow the prototype rte_crpyto_hw_user_post_deq_cb_fn() in + * rte_cryptodev.h + **/ + static __rte_always_inline void + write_frame_elt_status(void *data, uint32_t index, uint8_t is_op_success) + { + struct sample_frame *frame = data; + frame->elts[index + 1].status = is_op_success ? FRAME_ELT_OK : + FRAME_ELT_FAIL; + } + + /* Frame dequeue function use direct dequeue API */ + static struct sample_frame * + dequeue_frame_with_direct_api(void) + { + struct sample_frame *ret_frame; + uint64_t flags, drv_data; + uint32_t n, n_fail, n_fail_first = 0; + int ret; + + /* Dequeue first job, which should have frame data stored in opaque */ + flags = RTE_CRYPTO_HW_DEQ_FLAG_START; + ret_frame = hw_ops.dequeue_one(hw_ops.qp, &drv_data, flags, &ret); + if (ret == 0) { + /* ret == 0, means it is still under processing */ + return NULL; + } else if (ret == 1) { + /* ret_frame is successfully retrieved, the ret stores the + operation result */ + ret_frame->elts[0].status = FRAME_ELT_OK; + } else { + ret_frame->elts[0].status = FRAME_ELT_FAIL; + n_fail_first = 1; + } + + /* Query if n_elts has been processed, if not return NULL */ + if (!hw_ops.query_processed(hw_ops.qp, frame->n_elts)) + return NULL; + + /* We are sure all elements have been processed, dequeue them all */ + flag = 0; + ret = hw_ops.dequeue_many(hw_ops.qp, &drv_data, (void *)ret_frame, + write_frame_elt_status, ret_frame->n_elts - 1, flag, &n_fail); + + if (n_fail + n_fail_first > 0) + ret_frame->status = FRAME_SOME_ELT_ERROR; + else + ret_frame->status = FRAME_OK; + + return ret_frame; + } + Asymmetric Cryptography ----------------------- diff --git a/doc/guides/rel_notes/release_20_08.rst b/doc/guides/rel_notes/release_20_08.rst index 39064afbe..eb973693d 100644 --- a/doc/guides/rel_notes/release_20_08.rst +++ b/doc/guides/rel_notes/release_20_08.rst @@ -56,6 +56,14 @@ New Features Also, make sure to start the actual text at the margin. ========================================================= + * **Add Cryptodev data-path APIs for no mbuf-centric data-path.** + + Cryptodev is added a set of data-path APIs that are not based on + cryptodev operations. The APIs are designed for external applications + or libraries that want to use cryptodev but their data-path + implementations are not mbuf-centric. QAT Symmetric PMD is also updated + to add the support to this API. + Removed Items ------------- From patchwork Fri Jul 3 11:09:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Zhang X-Patchwork-Id: 72990 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id D5B8AA0519; Fri, 3 Jul 2020 13:10:40 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 968931DB7B; Fri, 3 Jul 2020 13:09:43 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 7038B1DB74 for ; Fri, 3 Jul 2020 13:09:40 +0200 (CEST) IronPort-SDR: zJ7A4D/ud61PlPFKERivIAP2YVUDkGrl8s1s/gcICHpTvDaYQUigR7KVs3dMEvH4FP5Rlj6uhD zezqB/yEd4yA== X-IronPort-AV: E=McAfee;i="6000,8403,9670"; a="126745857" X-IronPort-AV: E=Sophos;i="5.75,308,1589266800"; d="scan'208";a="126745857" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2020 04:09:40 -0700 IronPort-SDR: MxfarUMnQXehClbRh5Pr9tDp4vZLyDjz28Ahy0Il5vSVTlME+LQ4pa19WFHVrrXh44/J0uyWDq M3Ejz9584Phg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,308,1589266800"; d="scan'208";a="304556008" Received: from silpixa00398673.ir.intel.com (HELO silpixa00398673.ger.corp.intel.com) ([10.237.223.136]) by fmsmga004.fm.intel.com with ESMTP; 03 Jul 2020 04:09:38 -0700 From: Fan Zhang To: dev@dpdk.org Cc: fiona.trahe@intel.com, akhil.goyal@nxp.com, thomas@monjalon.net, jerinjacobk@gmail.com, Fan Zhang Date: Fri, 3 Jul 2020 12:09:23 +0100 Message-Id: <20200703110923.26452-8-roy.fan.zhang@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200703110923.26452-1-roy.fan.zhang@intel.com> References: <20200625133138.42827-1-roy.fan.zhang@intel.com> <20200703110923.26452-1-roy.fan.zhang@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [dpdk-dev v3 4/4] doc: add cryptodev direct APIs guide X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch updates programmer's guide to demonstrate the usage and limitations of cryptodev symmetric crypto data-path APIs. Signed-off-by: Fan Zhang --- doc/guides/prog_guide/cryptodev_lib.rst | 266 ++++++++++++++++++++++++ doc/guides/rel_notes/release_20_08.rst | 8 + 2 files changed, 274 insertions(+) diff --git a/doc/guides/prog_guide/cryptodev_lib.rst b/doc/guides/prog_guide/cryptodev_lib.rst index c14f750fa..9900a593a 100644 --- a/doc/guides/prog_guide/cryptodev_lib.rst +++ b/doc/guides/prog_guide/cryptodev_lib.rst @@ -861,6 +861,272 @@ using one of the crypto PMDs available in DPDK. num_dequeued_ops); } while (total_num_dequeued_ops < num_enqueued_ops); +Cryptodev Direct Symmetric Crypto Data-path APIs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Direct symmetric crypto data-path APIs are a set of APIs that especially +provided for Symmetric HW Crypto PMD that provides fast data-path +enqueue/dequeue operations. The direct data-path APIs take advantage of +existing Cryptodev APIs for device, queue pairs, and session management. In +addition the user are required to get the queue pair pointer data and function +pointers. The APIs are provided as an advanced feature as an alternative +to ``rte_cryptodev_enqueue_burst`` and ``rte_cryptodev_dequeue_burst``. The +APIs are designed for the user to develop close-to-native performance symmetric +crypto data-path implementation for their applications that do not necessarily +depend on cryptodev operations and cryptodev operation mempools, or mbufs. + +Cryptodev PMDs who supports this feature will have +``RTE_CRYPTODEV_FF_SYM_HW_DIRECT_API`` feature flag presented. The user uses +``rte_cryptodev_sym_get_hw_ops`` function call to get all the function pointers +for different enqueue and dequeue operations, plus the device specific +queue pair data. After the ``rte_crypto_hw_ops`` structure is properly set by +the driver, the user can use the function pointers and the queue data pointers +in the structure to enqueue and dequeue crypto jobs. + +To simply the enqueue APIs a symmetric job structure is defined: + +.. code-block:: c + + /** + * Asynchronous operation job descriptor. + * Used by HW crypto devices direct API call that supports such activity + **/ + struct rte_crypto_sym_job { + union { + /** + * When RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL bit is set in flags, sgl + * field is used as input data. Otherwise data_iova is + * used. + **/ + rte_iova_t data_iova; + struct rte_crypto_sgl *sgl; + }; + union { + /** + * Different than cryptodev ops, all ofs and len fields have + * the unit of bytes (including Snow3G/Kasumi/Zuc. + **/ + struct { + uint32_t cipher_ofs; + uint32_t cipher_len; + } cipher_only; + struct { + uint32_t auth_ofs; + uint32_t auth_len; + rte_iova_t digest_iova; + } auth_only; + struct { + uint32_t aead_ofs; + uint32_t aead_len; + rte_iova_t tag_iova; + uint8_t *aad; + rte_iova_t aad_iova; + } aead; + struct { + uint32_t cipher_ofs; + uint32_t cipher_len; + uint32_t auth_ofs; + uint32_t auth_len; + rte_iova_t digest_iova; + } chain; + }; + uint8_t *iv; + rte_iova_t iv_iova; + }; + +Different than Cryptodev operation, the ``rte_crypto_sym_job`` structure +focuses only on the data field required for crypto PMD to execute a single job, +and is not supposed stored as opaque data. The user can freely allocate the +structure buffer from stack and reuse it to fill all jobs. + +To use the direct symmetric crypto APIs safely, the user has to carefully +set the correct fields in rte_crypto_sym_job structure, otherwise the +application or the system may crash. Also there are a few limitations to the +direct symmetric crypto APIs: + +* Only support in-place operations. +* APIs are NOT thread-safe. +* CANNOT mix the direct API's enqueue with rte_cryptodev_enqueue_burst, or + vice versa. + +The following sample code shows how to use Cryptodev direct API to process a +user defined frame with maximum 32 buffers with AES-CBC and HMAC-SHA chained +algorithm of a frame defined by user. + +See *DPDK API Reference* for details on each API definitions. + +.. code-block:: c + + #include + + #define FRAME_ELT_OK 0 + #define FRAME_ELT_FAIL 1 + #define FRAME_OK 0 + #define FRAME_SOME_ELT_ERROR 1 + #define FRAME_SIZE 32 + + /* Sample frame element struct */ + struct sample_frame_elt { + /* The status field of frame element */ + uint8_t status; + /* Pre-created and initialized cryptodev session */ + struct rte_cryptodev_sym_session *session; + union { + __rte_iova_t data; + struct rte_crypto_sgl sgl; + }; + uint32_t data_len; + __rte_iova_t digest; + uint8_t *iv; + uint8_t is_sgl; + }; + + /* Sample frame struct to describe up to 32 crypto jobs */ + struct sample_frame { + struct sample_frame_elt elts[FRAME_SIZE]; /**< All frame elements */ + uint32_t n_elts; /**< Number of elements */ + }; + + /* Global Cryptodev Direct API structure */ + static struct rte_crypto_hw_ops hw_ops; + + /* Initialization */ + static int + frame_operation_init( + uint8_t cryptodev_id, /**< Initialized cryptodev ID */ + uint16_t qp_id /**< Initialized queue pair ID */) + { + /* Get APIs */ + ret = rte_cryptodev_sym_get_hw_ops(cryptodev_id, qp_id, &hw_ops); + /* If the device does not support this feature or queue pair is not + initialized, return -1 */ + if (!ret) + return -1; + return 0; + } + + /* Frame enqueue function use direct AES-CBC-* + HMAC-SHA* API */ + static int + enqueue_frame_to_direct_api( + struct sample_frame *frame /**< Initialized user frame struct */) + { + struct rte_crypto_hw_ops hw_ops; + struct rte_crypto_sym_job job; + uint64_t drv_data, flags = 0; + uint32_t i; + int ret; + + /* Fill all sample frame element data into HW queue pair */ + for (i = 0; i < frame->n_elts; i++) { + struct sample_frame_elt *fe = &frame->elts[i]; + int ret; + + /* if it is the first element in the frame, set FIRST flag to + let the driver to know it is first frame and fill drv_data. */ + if (i == 0) + flags |= RTE_CRYPTO_HW_ENQ_FLAG_START; + else + flags &= ~RTE_CRYPTO_HW_ENQ_FLAG_START; + + /* if it is the last element in the frame, write LAST flag to + kick HW queue */ + if (i == frame->n_elts - 1) + flags |= RTE_CRYPTO_HW_ENQ_FLAG_LAST; + else + flags &= ~RTE_CRYPTO_HW_ENQ_FLAG_LAST; + + /* Fill the job data with frame element data */ + if (fe->is_sgl != 0) { + /* The buffer is a SGL buffer */ + job.sgl = &frame->sgl; + /* Set SGL flag */ + flags |= RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL; + } else { + job.data_iova = fe->data; + /* Unset SGL flag in the job */ + flags &= ~RTE_CRYPTO_HW_ENQ_FLAG_IS_SGL; + } + + job.chain.cipher_ofs = job.chain.auth_ofs = 0; + job.chain.cipher_len = job.chain.auth_len = fe->data_len; + job.chain.digest_iova = fe->digest; + + job.iv = fe->iv; + + /* Call direct data-path enqueue chaining op API */ + ret = hw_ops.enqueue_chain(hw_ops.qp, fe->session, &job, + (void *frame), &drv_data, flags); + /** + * In case one element is failed to be enqueued, simply abandon + * enqueuing the whole frame. + **/ + if (!ret) + return -1; + + /** + * To this point the frame is enqueued. The job buffer can be + * safely reused for enqueuing next frame element. + **/ + } + + return 0; + } + + /** + * Sample function to write frame element status field based on + * driver returned operation result. The function return and parameter + * should follow the prototype rte_crpyto_hw_user_post_deq_cb_fn() in + * rte_cryptodev.h + **/ + static __rte_always_inline void + write_frame_elt_status(void *data, uint32_t index, uint8_t is_op_success) + { + struct sample_frame *frame = data; + frame->elts[index + 1].status = is_op_success ? FRAME_ELT_OK : + FRAME_ELT_FAIL; + } + + /* Frame dequeue function use direct dequeue API */ + static struct sample_frame * + dequeue_frame_with_direct_api(void) + { + struct sample_frame *ret_frame; + uint64_t flags, drv_data; + uint32_t n, n_fail, n_fail_first = 0; + int ret; + + /* Dequeue first job, which should have frame data stored in opaque */ + flags = RTE_CRYPTO_HW_DEQ_FLAG_START; + ret_frame = hw_ops.dequeue_one(hw_ops.qp, &drv_data, flags, &ret); + if (ret == 0) { + /* ret == 0, means it is still under processing */ + return NULL; + } else if (ret == 1) { + /* ret_frame is successfully retrieved, the ret stores the + operation result */ + ret_frame->elts[0].status = FRAME_ELT_OK; + } else { + ret_frame->elts[0].status = FRAME_ELT_FAIL; + n_fail_first = 1; + } + + /* Query if n_elts has been processed, if not return NULL */ + if (!hw_ops.query_processed(hw_ops.qp, frame->n_elts)) + return NULL; + + /* We are sure all elements have been processed, dequeue them all */ + flag = 0; + ret = hw_ops.dequeue_many(hw_ops.qp, &drv_data, (void *)ret_frame, + write_frame_elt_status, ret_frame->n_elts - 1, flag, &n_fail); + + if (n_fail + n_fail_first > 0) + ret_frame->status = FRAME_SOME_ELT_ERROR; + else + ret_frame->status = FRAME_OK; + + return ret_frame; + } + Asymmetric Cryptography ----------------------- diff --git a/doc/guides/rel_notes/release_20_08.rst b/doc/guides/rel_notes/release_20_08.rst index 39064afbe..eb973693d 100644 --- a/doc/guides/rel_notes/release_20_08.rst +++ b/doc/guides/rel_notes/release_20_08.rst @@ -56,6 +56,14 @@ New Features Also, make sure to start the actual text at the margin. ========================================================= + * **Add Cryptodev data-path APIs for no mbuf-centric data-path.** + + Cryptodev is added a set of data-path APIs that are not based on + cryptodev operations. The APIs are designed for external applications + or libraries that want to use cryptodev but their data-path + implementations are not mbuf-centric. QAT Symmetric PMD is also updated + to add the support to this API. + Removed Items -------------