From patchwork Sun Oct 11 00:32:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fan Zhang X-Patchwork-Id: 80297 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id B43CAA04B6; Sun, 11 Oct 2020 02:34:06 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 040021D50B; Sun, 11 Oct 2020 02:33:10 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by dpdk.org (Postfix) with ESMTP id 26C941D43F for ; Sun, 11 Oct 2020 02:33:03 +0200 (CEST) IronPort-SDR: SMRswLXjI42xwcPJoQHaogE5NwFaTgQxt0yrm5b/PbEd29x3Wev773cjeVYz9UsyKh1ob+pA52 b33UYhbiULoA== X-IronPort-AV: E=McAfee;i="6000,8403,9770"; a="153447306" X-IronPort-AV: E=Sophos;i="5.77,360,1596524400"; d="scan'208";a="153447306" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Oct 2020 17:32:59 -0700 IronPort-SDR: 8H4qJgpk0+u1Ue02mb5fxTLb4W8KgA9Uru1iSmvhOAkRshqGa9ds5vJIL+X1NzZ+jdPAbSc8M+ cuNo1o6dcARw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,360,1596524400"; d="scan'208";a="329294824" Received: from silpixa00398673.ir.intel.com (HELO silpixa00398673.ger.corp.intel.com) ([10.237.223.136]) by orsmga002.jf.intel.com with ESMTP; 10 Oct 2020 17:32:58 -0700 From: Fan Zhang To: dev@dpdk.org Cc: akhil.goyal@nxp.com, Fan Zhang , Piotr Bronowski , Adam Dybkowski Date: Sun, 11 Oct 2020 01:32:50 +0100 Message-Id: <20201011003252.54558-3-roy.fan.zhang@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20201011003252.54558-1-roy.fan.zhang@intel.com> References: <20201009211141.14435-1-roy.fan.zhang@intel.com> <20201011003252.54558-1-roy.fan.zhang@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [dpdk-dev v12 2/4] cryptodev: add raw crypto data-path APIs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch adds raw data-path APIs for enqueue and dequeue operations to cryptodev. The APIs support flexible user-define enqueue and dequeue behaviors. Signed-off-by: Fan Zhang Signed-off-by: Piotr Bronowski Acked-by: Adam Dybkowski --- doc/guides/cryptodevs/features/default.ini | 1 + doc/guides/prog_guide/cryptodev_lib.rst | 106 +++++ doc/guides/rel_notes/release_20_11.rst | 7 + lib/librte_cryptodev/rte_cryptodev.c | 80 ++++ lib/librte_cryptodev/rte_cryptodev.h | 413 +++++++++++++++++- lib/librte_cryptodev/rte_cryptodev_pmd.h | 51 ++- .../rte_cryptodev_version.map | 10 + 7 files changed, 665 insertions(+), 3 deletions(-) diff --git a/doc/guides/cryptodevs/features/default.ini b/doc/guides/cryptodevs/features/default.ini index 133a246ee..17b177fc4 100644 --- a/doc/guides/cryptodevs/features/default.ini +++ b/doc/guides/cryptodevs/features/default.ini @@ -30,6 +30,7 @@ Asymmetric sessionless = CPU crypto = Symmetric sessionless = Non-Byte aligned data = +Sym raw data path API = ; ; Supported crypto algorithms of a default crypto driver. diff --git a/doc/guides/prog_guide/cryptodev_lib.rst b/doc/guides/prog_guide/cryptodev_lib.rst index e7ba35c2d..0c018b982 100644 --- a/doc/guides/prog_guide/cryptodev_lib.rst +++ b/doc/guides/prog_guide/cryptodev_lib.rst @@ -632,6 +632,112 @@ a call argument. Status different than zero must be treated as error. For more details, e.g. how to convert an mbuf to an SGL, please refer to an example usage in the IPsec library implementation. +Cryptodev Raw Data-path APIs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Crypto Raw data-path APIs are a set of APIs designed to enable external +libraries/applications to leverage the cryptographic processing provided by +DPDK crypto PMDs through the cryptodev API but in a manner that is not +dependent on native DPDK data structures (eg. rte_mbuf, rte_crypto_op, ... etc) +in their data-path implementation. + +The raw data-path APIs have the following advantages: +- External data structure friendly design. The new APIs uses the operation + descriptor ``struct rte_crypto_sym_vec`` that supports raw data pointer and + IOVA addresses as input. Moreover, the APIs does not require the user to + allocate the descriptor from mempool, nor requiring mbufs to describe input + data's virtual and IOVA addresses. All these features made the translation + from user's own data structure into the descriptor easier and more efficient. +- Flexible enqueue and dequeue operation. The raw data-path APIs gives the + user more control to the enqueue and dequeue operations, including the + capability of precious enqueue/dequeue count, abandoning enqueue or dequeue + at any time, and operation status translation and set on the fly. + +Cryptodev PMDs which support the raw data-path APIs will have +``RTE_CRYPTODEV_FF_SYM_RAW_DP`` feature flag presented. To use this feature, +the user shall create a local ``struct rte_crypto_raw_dp_ctx`` buffer and +extend to at least the length returned by ``rte_cryptodev_get_raw_dp_ctx_size`` +function call. The created buffer is then initialized using +``rte_cryptodev_configure_raw_dp_ctx`` function with the ``is_update`` +parameter as 0. The library and the crypto device driver will then set the +buffer and attach either the cryptodev sym session, the rte_security session, +or the cryptodev xform for session-less operation into the ctx buffer, and +set the corresponding enqueue and dequeue function handlers based on the +algorithm information stored in the session or xform. When the ``is_update`` +parameter passed into ``rte_cryptodev_configure_raw_dp_ctx`` is 1, the driver +will not initialize the buffer but only update the session or xform and +the function handlers accordingly. + +After the ``struct rte_crypto_raw_dp_ctx`` buffer is initialized, it is now +ready for enqueue and dequeue operation. There are two different enqueue +functions: ``rte_cryptodev_raw_enqueue`` to enqueue single raw data +operation, and ``rte_cryptodev_raw_enqueue_burst`` to enqueue a descriptor +with multiple operations. In case of the application uses similar approach to +``struct rte_crypto_sym_vec`` to manage its data burst but with different +data structure, using the ``rte_cryptodev_raw_enqueue_burst`` function may be +less efficient as this is a situation where the application has to loop over +all crypto operations to assemble the ``struct rte_crypto_sym_vec`` descriptor +from its own data structure, and then the driver will loop over them again to +translate every operation in the descriptor to the driver's specific queue data. +The ``rte_cryptodev_raw_enqueue`` should be used to save one loop for each data +burst instead. + +The ``rte_cryptodev_raw_enqueue`` and ``rte_cryptodev_raw_enqueue_burst`` +functions will return or set the enqueue status. ``rte_cryptodev_raw_enqueue`` +will return the status directly, ``rte_cryptodev_raw_enqueue_burst`` will +return the number of operations enqueued or stored (explained as follows) and +set the ``enqueue_status`` buffer provided by the user. The possible +enqueue status values are: + +- ``1``: the operation(s) is/are enqueued successfully. +- ``0``: the operation(s) is/are cached successfully in the crypto device queue + but is not actually enqueued. The user shall call + ``rte_cryptodev_raw_enqueue_done`` function after the expected operations + are stored. The crypto device will then start enqueuing all of them at + once. +- The negative integer: error occured during enqueue. + +Calling ``rte_cryptodev_configure_raw_dp_ctx`` with the parameter ``is_update`` +set as 0 twice without the enqueue function returning or setting enqueue status +to 1 or ``rte_cryptodev_raw_enqueue_done`` function being called in between will +invalidate any operation stored in the device queue but not enqueued. This +feature is useful when the user wants to abandon partially enqueued operations +for a failed enqueue burst operation and try enqueuing in a whole later. + +Similar as enqueue, there are two dequeue functions: +``rte_cryptodev_raw_dequeue`` for dequeing single operation, and +``rte_cryptodev_raw_dequeue_burst`` for dequeuing a burst of operations (e.g. +all operations in a ``struct rte_crypto_sym_vec`` descriptor). The +``rte_cryptodev_raw_dequeue_burst`` function allows the user to provide callback +functions to retrieve dequeue count from the enqueued user data and write the +expected status value to the user data on the fly. The dequeue functions also +set the dequeue status: + +- ``1``: the operation(s) is/are dequeued successfully. +- ``0``: the operation(s) is/are completed but is not actually dequeued (hence + still kept in the device queue). The user shall call the + ``rte_cryptodev_raw_dequeue_done`` function after the expected number of + operations (e.g. all operations in a descriptor) are dequeued. The crypto + device driver will then free them from the queue at once. +- The negative integer: error occurred during dequeue. + +Calling ``rte_cryptodev_configure_raw_dp_ctx`` with the parameter ``is_update`` +set as 0 twice without the dequeue functions execution changed dequeue_status +to 1 or ``rte_cryptodev_raw_dequeue_done`` function being called in between will +revert the crypto device queue's dequeue effort to the moment when the +``struct rte_crypto_raw_dp_ctx`` buffer is initialized. This feature is useful +when the user wants to abandon partially dequeued data and try dequeuing again +later in a whole. + +There are a few limitations to the raw data path APIs: + +* Only support in-place operations. +* APIs are NOT thread-safe. +* CANNOT mix the raw data-path API's enqueue with rte_cryptodev_enqueue_burst, + or vice versa. + +See *DPDK API Reference* for details on each API definitions. + Sample code ----------- diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst index 2973b2a33..85a07d86e 100644 --- a/doc/guides/rel_notes/release_20_11.rst +++ b/doc/guides/rel_notes/release_20_11.rst @@ -74,6 +74,13 @@ New Features * Added SR-IOV PF support +* **Added raw data-path APIs for cryptodev library.** + + Cryptodev is added with raw data-path APIs to accelerate external + libraries or applications which need to avail fast cryptodev + enqueue/dequeue operations but does not necessarily depends on + mbufs and cryptodev operation mempools. + * **Updated the aesni_mb crypto PMD.** * Added support for AES-ECB 128, 192 and 256. diff --git a/lib/librte_cryptodev/rte_cryptodev.c b/lib/librte_cryptodev/rte_cryptodev.c index cda160f61..3d95ac6ea 100644 --- a/lib/librte_cryptodev/rte_cryptodev.c +++ b/lib/librte_cryptodev/rte_cryptodev.c @@ -1772,6 +1772,86 @@ rte_cryptodev_sym_cpu_crypto_process(uint8_t dev_id, return dev->dev_ops->sym_cpu_process(dev, sess, ofs, vec); } +int +rte_cryptodev_get_raw_dp_ctx_size(uint8_t dev_id) +{ + struct rte_cryptodev *dev; + int32_t size = sizeof(struct rte_crypto_raw_dp_ctx); + int32_t priv_size; + + if (!rte_cryptodev_pmd_is_valid_dev(dev_id)) + return -EINVAL; + + dev = rte_cryptodev_pmd_get_dev(dev_id); + + if (*dev->dev_ops->sym_get_raw_dp_ctx_size == NULL || + !(dev->feature_flags & RTE_CRYPTODEV_FF_SYM_RAW_DP)) { + return -ENOTSUP; + } + + priv_size = (*dev->dev_ops->sym_get_raw_dp_ctx_size)(dev); + if (priv_size < 0) + return -ENOTSUP; + + return RTE_ALIGN_CEIL((size + priv_size), 8); +} + +int +rte_cryptodev_configure_raw_dp_ctx(uint8_t dev_id, uint16_t qp_id, + struct rte_crypto_raw_dp_ctx *ctx, + enum rte_crypto_op_sess_type sess_type, + union rte_cryptodev_session_ctx session_ctx, + uint8_t is_update) +{ + struct rte_cryptodev *dev; + + if (!rte_cryptodev_get_qp_status(dev_id, qp_id)) + return -EINVAL; + + dev = rte_cryptodev_pmd_get_dev(dev_id); + if (!(dev->feature_flags & RTE_CRYPTODEV_FF_SYM_RAW_DP) + || dev->dev_ops->sym_configure_raw_dp_ctx == NULL) + return -ENOTSUP; + + return (*dev->dev_ops->sym_configure_raw_dp_ctx)(dev, qp_id, ctx, + sess_type, session_ctx, is_update); +} + +uint32_t +rte_cryptodev_raw_enqueue_burst(struct rte_crypto_raw_dp_ctx *ctx, + struct rte_crypto_sym_vec *vec, union rte_crypto_sym_ofs ofs, + void **user_data, int *enqueue_status) +{ + return (*ctx->enqueue_burst)(ctx->qp_data, ctx->drv_ctx_data, vec, + ofs, user_data, enqueue_status); +} + +int +rte_cryptodev_raw_enqueue_done(struct rte_crypto_raw_dp_ctx *ctx, + uint32_t n) +{ + return (*ctx->enqueue_done)(ctx->qp_data, ctx->drv_ctx_data, n); +} + +uint32_t +rte_cryptodev_raw_dequeue_burst(struct rte_crypto_raw_dp_ctx *ctx, + rte_cryptodev_raw_get_dequeue_count_t get_dequeue_count, + rte_cryptodev_raw_post_dequeue_t post_dequeue, + void **out_user_data, uint8_t is_user_data_array, + uint32_t *n_success_jobs, int *status) +{ + return (*ctx->dequeue_burst)(ctx->qp_data, ctx->drv_ctx_data, + get_dequeue_count, post_dequeue, out_user_data, + is_user_data_array, n_success_jobs, status); +} + +int +rte_cryptodev_raw_dequeue_done(struct rte_crypto_raw_dp_ctx *ctx, + uint32_t n) +{ + return (*ctx->dequeue_done)(ctx->qp_data, ctx->drv_ctx_data, n); +} + /** Initialise rte_crypto_op mempool element */ static void rte_crypto_op_init(struct rte_mempool *mempool, diff --git a/lib/librte_cryptodev/rte_cryptodev.h b/lib/librte_cryptodev/rte_cryptodev.h index 26abd0c52..b5db8fd81 100644 --- a/lib/librte_cryptodev/rte_cryptodev.h +++ b/lib/librte_cryptodev/rte_cryptodev.h @@ -458,7 +458,8 @@ rte_cryptodev_asym_get_xform_enum(enum rte_crypto_asym_xform_type *xform_enum, /**< Support symmetric session-less operations */ #define RTE_CRYPTODEV_FF_NON_BYTE_ALIGNED_DATA (1ULL << 23) /**< Support operations on data which is not byte aligned */ - +#define RTE_CRYPTODEV_FF_SYM_RAW_DP (1ULL << 24) +/**< Support accelerated specific symmetric raw data-path APIs */ /** * Get the name of a crypto device feature flag @@ -1319,6 +1320,416 @@ rte_cryptodev_sym_cpu_crypto_process(uint8_t dev_id, struct rte_cryptodev_sym_session *sess, union rte_crypto_sym_ofs ofs, struct rte_crypto_sym_vec *vec); +/** + * Get the size of the raw data-path context buffer. + * + * @param dev_id The device identifier. + * + * @return + * - If the device supports raw data-path APIs, return the context size. + * - If the device does not support the APIs, return -1. + */ +__rte_experimental +int +rte_cryptodev_get_raw_dp_ctx_size(uint8_t dev_id); + +/** + * Union of different crypto session types, including session-less xform + * pointer. + */ +union rte_cryptodev_session_ctx { + struct rte_cryptodev_sym_session *crypto_sess; + struct rte_crypto_sym_xform *xform; + struct rte_security_session *sec_sess; +}; + +/** + * Enqueue a vectorized operation descriptor into the device queue but the + * driver may or may not start processing until rte_cryptodev_raw_enqueue_done() + * is called. + * + * @param qp Driver specific queue pair data. + * @param drv_ctx Driver specific context data. + * @param vec Vectorized operation descriptor. + * @param ofs Start and stop offsets for auth and cipher + * operations. + * @param user_data The array of user data for dequeue later. + * @param enqueue_status Driver written value to specify the + * enqueue status. Possible values: + * - 1: The number of operations returned are + * enqueued successfully. + * - 0: The number of operations returned are + * cached into the queue but are not processed + * until rte_cryptodev_raw_enqueue_done() is + * called. + * - negative integer: Error occurred. + * @return + * - The number of operations in the descriptor successfully enqueued or + * cached into the queue but not enqueued yet, depends on the + * "enqueue_status" value. + */ +typedef uint32_t (*cryptodev_sym_raw_enqueue_burst_t)( + void *qp, uint8_t *drv_ctx, struct rte_crypto_sym_vec *vec, + union rte_crypto_sym_ofs ofs, void *user_data[], int *enqueue_status); + +/** + * Enqueue single raw data vector into the device queue but the driver may or + * may not start processing until rte_cryptodev_raw_enqueue_done() is called. + * + * @param qp Driver specific queue pair data. + * @param drv_ctx Driver specific context data. + * @param data_vec The buffer data vector. + * @param n_data_vecs Number of buffer data vectors. + * @param ofs Start and stop offsets for auth and cipher + * operations. + * @param iv IV virtual and IOVA addresses + * @param digest digest virtual and IOVA addresses + * @param aad_or_auth_iv AAD or auth IV virtual and IOVA addresses, + * depends on the algorithm used. + * @param user_data The user data. + * @return + * - 1: The data vector is enqueued successfully. + * - 0: The data vector is cached into the queue but is not processed + * until rte_cryptodev_raw_enqueue_done() is called. + * - negative integer: failure. + */ +typedef int (*cryptodev_sym_raw_enqueue_t)( + void *qp, uint8_t *drv_ctx, struct rte_crypto_vec *data_vec, + uint16_t n_data_vecs, union rte_crypto_sym_ofs ofs, + struct rte_crypto_va_iova_ptr *iv, + struct rte_crypto_va_iova_ptr *digest, + struct rte_crypto_va_iova_ptr *aad_or_auth_iv, + void *user_data); + +/** + * Inform the cryptodev queue pair to start processing or finish dequeuing all + * enqueued/dequeued operations. + * + * @param qp Driver specific queue pair data. + * @param drv_ctx Driver specific context data. + * @param n The total number of processed operations. + * @return + * - On success return 0. + * - On failure return negative integer. + */ +typedef int (*cryptodev_sym_raw_operation_done_t)(void *qp, uint8_t *drv_ctx, + uint32_t n); + +/** + * Typedef that the user provided for the driver to get the dequeue count. + * The function may return a fixed number or the number parsed from the user + * data stored in the first processed operation. + * + * @param user_data Dequeued user data. + * @return + * - The number of operations to be dequeued. + **/ +typedef uint32_t (*rte_cryptodev_raw_get_dequeue_count_t)(void *user_data); + +/** + * Typedef that the user provided to deal with post dequeue operation, such + * as filling status. + * + * @param user_data Dequeued user data. + * @param index Index number of the processed descriptor. + * @param is_op_success Operation status provided by the driver. + **/ +typedef void (*rte_cryptodev_raw_post_dequeue_t)(void *user_data, + uint32_t index, uint8_t is_op_success); + +/** + * Dequeue a burst of symmetric crypto processing. + * + * @param qp Driver specific queue pair data. + * @param drv_ctx Driver specific context data. + * @param get_dequeue_count User provided callback function to + * obtain dequeue operation count. + * @param post_dequeue User provided callback function to + * post-process a dequeued operation. + * @param out_user_data User data pointer array to be retrieve + * from device queue. In case of + * *is_user_data_array* is set there + * should be enough room to store all + * user data. + * @param is_user_data_array Set 1 if every dequeued user data will + * be written into out_user_data array. + * Set 0 if only the first user data will + * be written into out_user_data array. + * @param n_success Driver written value to specific the + * total successful operations count. + * @param dequeue_status Driver written value to specify the + * dequeue status. Possible values: + * - 1: Successfully dequeued the number + * of operations returned. The user + * data previously set during enqueue + * is stored in the "out_user_data". + * - 0: The number of operations returned + * are completed and the user data is + * stored in the "out_user_data", but + * they are not freed from the queue + * until + * rte_cryptodev_raw_dequeue_done() + * is called. + * - negative integer: Error occurred. + * @return + * - The number of operations dequeued or completed but not freed from the + * queue, depends on "dequeue_status" value. + */ +typedef uint32_t (*cryptodev_sym_raw_dequeue_burst_t)(void *qp, + uint8_t *drv_ctx, + rte_cryptodev_raw_get_dequeue_count_t get_dequeue_count, + rte_cryptodev_raw_post_dequeue_t post_dequeue, + void **out_user_data, uint8_t is_user_data_array, + uint32_t *n_success, int *dequeue_status); + +/** + * Dequeue a symmetric crypto processing. + * + * @param qp Driver specific queue pair data. + * @param drv_ctx Driver specific context data. + * @param dequeue_status Driver written value to specify the + * dequeue status. Possible values: + * - 1: Successfully dequeued a operation. + * The user data is returned. + * - 0: The first operation in the queue + * is completed and the user data + * previously set during enqueue is + * returned, but it is not freed from + * the queue until + * rte_cryptodev_raw_dequeue_done() is + * called. + * - negative integer: Error occurred. + * @param op_status Driver written value to specify + * operation status. + * @return + * - The user data pointer retrieved from device queue or NULL if no + * operation is ready for dequeue. + */ +typedef void * (*cryptodev_sym_raw_dequeue_t)( + void *qp, uint8_t *drv_ctx, int *dequeue_status, + enum rte_crypto_op_status *op_status); + +/** + * Context data for raw data-path API crypto process. The buffer of this + * structure is to be allocated by the user application with the size equal + * or bigger than rte_cryptodev_get_raw_dp_ctx_size() returned value. + */ +struct rte_crypto_raw_dp_ctx { + void *qp_data; + + cryptodev_sym_raw_enqueue_t enqueue; + cryptodev_sym_raw_enqueue_burst_t enqueue_burst; + cryptodev_sym_raw_operation_done_t enqueue_done; + cryptodev_sym_raw_dequeue_t dequeue; + cryptodev_sym_raw_dequeue_burst_t dequeue_burst; + cryptodev_sym_raw_operation_done_t dequeue_done; + + /* Driver specific context data */ + __extension__ uint8_t drv_ctx_data[]; +}; + +/** + * Configure raw data-path context data. + * + * NOTE: + * After the context data is configured, the user should call + * rte_cryptodev_raw_attach_session() before using it in + * rte_cryptodev_raw_enqueue/dequeue function call. + * + * @param dev_id The device identifier. + * @param qp_id The index of the queue pair from which to + * retrieve processed packets. The value must be + * in the range [0, nb_queue_pair - 1] previously + * supplied to rte_cryptodev_configure(). + * @param ctx The raw data-path context data. + * @param sess_type session type. + * @param session_ctx Session context data. + * @param is_update Set 0 if it is to initialize the ctx. + * Set 1 if ctx is initialized and only to update + * session context data. + * @return + * - On success return 0. + * - On failure return negative integer. + */ +__rte_experimental +int +rte_cryptodev_configure_raw_dp_ctx(uint8_t dev_id, uint16_t qp_id, + struct rte_crypto_raw_dp_ctx *ctx, + enum rte_crypto_op_sess_type sess_type, + union rte_cryptodev_session_ctx session_ctx, + uint8_t is_update); + +/** + * Enqueue a vectorized operation descriptor into the device queue but the + * driver may or may not start processing until rte_cryptodev_raw_enqueue_done() + * is called. + * + * @param ctx The initialized raw data-path context data. + * @param vec Vectorized operation descriptor. + * @param ofs Start and stop offsets for auth and cipher + * operations. + * @param user_data The array of user data for dequeue later. + * @param enqueue_status Driver written value to specify the + * enqueue status. Possible values: + * - 1: The number of operations returned are + * enqueued successfully. + * - 0: The number of operations returned are + * cached into the queue but are not processed + * until rte_cryptodev_raw_enqueue_done() is + * called. + * - negative integer: Error occurred. + * @return + * - The number of operations in the descriptor successfully enqueued or + * cached into the queue but not enqueued yet, depends on the + * "enqueue_status" value. + */ +__rte_experimental +uint32_t +rte_cryptodev_raw_enqueue_burst(struct rte_crypto_raw_dp_ctx *ctx, + struct rte_crypto_sym_vec *vec, union rte_crypto_sym_ofs ofs, + void **user_data, int *enqueue_status); + +/** + * Enqueue single raw data vector into the device queue but the driver may or + * may not start processing until rte_cryptodev_raw_enqueue_done() is called. + * + * @param ctx The initialized raw data-path context data. + * @param data_vec The buffer data vector. + * @param n_data_vecs Number of buffer data vectors. + * @param ofs Start and stop offsets for auth and cipher + * operations. + * @param iv IV virtual and IOVA addresses + * @param digest digest virtual and IOVA addresses + * @param aad_or_auth_iv AAD or auth IV virtual and IOVA addresses, + * depends on the algorithm used. + * @param user_data The user data. + * @return + * - 1: The data vector is enqueued successfully. + * - 0: The data vector is cached into the queue but is not processed + * until rte_cryptodev_raw_enqueue_done() is called. + * - negative integer: failure. + */ +__rte_experimental +static __rte_always_inline int +rte_cryptodev_raw_enqueue(struct rte_crypto_raw_dp_ctx *ctx, + struct rte_crypto_vec *data_vec, uint16_t n_data_vecs, + union rte_crypto_sym_ofs ofs, + struct rte_crypto_va_iova_ptr *iv, + struct rte_crypto_va_iova_ptr *digest, + struct rte_crypto_va_iova_ptr *aad_or_auth_iv, + void *user_data) +{ + return (*ctx->enqueue)(ctx->qp_data, ctx->drv_ctx_data, data_vec, + n_data_vecs, ofs, iv, digest, aad_or_auth_iv, user_data); +} + +/** + * Start processing all enqueued operations from last + * rte_cryptodev_configure_raw_dp_ctx() call. + * + * @param ctx The initialized raw data-path context data. + * @param n The number of operations cached. + * @return + * - On success return 0. + * - On failure return negative integer. + */ +__rte_experimental +int +rte_cryptodev_raw_enqueue_done(struct rte_crypto_raw_dp_ctx *ctx, + uint32_t n); + +/** + * Dequeue a burst of symmetric crypto processing. + * + * @param ctx The initialized raw data-path context + * data. + * @param get_dequeue_count User provided callback function to + * obtain dequeue operation count. + * @param post_dequeue User provided callback function to + * post-process a dequeued operation. + * @param out_user_data User data pointer array to be retrieve + * from device queue. In case of + * *is_user_data_array* is set there + * should be enough room to store all + * user data. + * @param is_user_data_array Set 1 if every dequeued user data will + * be written into out_user_data array. + * Set 0 if only the first user data will + * be written into out_user_data array. + * @param n_success Driver written value to specific the + * total successful operations count. + * @param dequeue_status Driver written value to specify the + * dequeue status. Possible values: + * - 1: Successfully dequeued the number + * of operations returned. The user + * data previously set during enqueue + * is stored in the "out_user_data". + * - 0: The number of operations returned + * are completed and the user data is + * stored in the "out_user_data", but + * they are not freed from the queue + * until + * rte_cryptodev_raw_dequeue_done() + * is called. + * - negative integer: Error occurred. + * @return + * - The number of operations dequeued or completed but not freed from the + * queue, depends on "dequeue_status" value. + */ +__rte_experimental +uint32_t +rte_cryptodev_raw_dequeue_burst(struct rte_crypto_raw_dp_ctx *ctx, + rte_cryptodev_raw_get_dequeue_count_t get_dequeue_count, + rte_cryptodev_raw_post_dequeue_t post_dequeue, + void **out_user_data, uint8_t is_user_data_array, + uint32_t *n_success, int *dequeue_status); + +/** + * Dequeue a symmetric crypto processing. + * + * @param ctx The initialized raw data-path context + * data. + * @param dequeue_status Driver written value to specify the + * dequeue status. Possible values: + * - 1: Successfully dequeued a operation. + * The user data is returned. + * - 0: The first operation in the queue + * is completed and the user data + * previously set during enqueue is + * returned, but it is not freed from + * the queue until + * rte_cryptodev_raw_dequeue_done() is + * called. + * - negative integer: Error occurred. + * @param op_status Driver written value to specify + * operation status. + * @return + * - The user data pointer retrieved from device queue or NULL if no + * operation is ready for dequeue. + */ +__rte_experimental +static __rte_always_inline void * +rte_cryptodev_raw_dequeue(struct rte_crypto_raw_dp_ctx *ctx, + int *dequeue_status, enum rte_crypto_op_status *op_status) +{ + return (*ctx->dequeue)(ctx->qp_data, ctx->drv_ctx_data, dequeue_status, + op_status); +} + +/** + * Inform the queue pair dequeue operations is finished. + * + * @param ctx The initialized raw data-path context data. + * @param n The number of operations. + * @return + * - On success return 0. + * - On failure return negative integer. + */ +__rte_experimental +int +rte_cryptodev_raw_dequeue_done(struct rte_crypto_raw_dp_ctx *ctx, + uint32_t n); + #ifdef __cplusplus } #endif diff --git a/lib/librte_cryptodev/rte_cryptodev_pmd.h b/lib/librte_cryptodev/rte_cryptodev_pmd.h index 1367222f7..9a8a7e632 100644 --- a/lib/librte_cryptodev/rte_cryptodev_pmd.h +++ b/lib/librte_cryptodev/rte_cryptodev_pmd.h @@ -317,6 +317,42 @@ typedef uint32_t (*cryptodev_sym_cpu_crypto_process_t) (struct rte_cryptodev *dev, struct rte_cryptodev_sym_session *sess, union rte_crypto_sym_ofs ofs, struct rte_crypto_sym_vec *vec); +/** + * Typedef that the driver provided to get service context private date size. + * + * @param dev Crypto device pointer. + * + * @return + * - On success return the size of the device's service context private data. + * - On failure return negative integer. + */ +typedef int (*cryptodev_sym_get_raw_dp_ctx_size_t)(struct rte_cryptodev *dev); + +/** + * Typedef that the driver provided to configure raw data-path context. + * + * @param dev Crypto device pointer. + * @param qp_id Crypto device queue pair index. + * @param service_type Type of the service requested. + * @param ctx The raw data-path context data. + * @param sess_type session type. + * @param session_ctx Session context data. If NULL the driver + * shall only configure the drv_ctx_data in + * ctx buffer. Otherwise the driver shall only + * parse the session_ctx to set appropriate + * function pointers in ctx. + * @param is_update Set 0 if it is to initialize the ctx. + * Set 1 if ctx is initialized and only to update + * session context data. + * @return + * - On success return 0. + * - On failure return negative integer. + */ +typedef int (*cryptodev_sym_configure_raw_dp_ctx_t)( + struct rte_cryptodev *dev, uint16_t qp_id, + struct rte_crypto_raw_dp_ctx *ctx, + enum rte_crypto_op_sess_type sess_type, + union rte_cryptodev_session_ctx session_ctx, uint8_t is_update); /** Crypto device operations function pointer table */ struct rte_cryptodev_ops { @@ -349,8 +385,19 @@ struct rte_cryptodev_ops { /**< Clear a Crypto sessions private data. */ cryptodev_asym_free_session_t asym_session_clear; /**< Clear a Crypto sessions private data. */ - cryptodev_sym_cpu_crypto_process_t sym_cpu_process; - /**< process input data synchronously (cpu-crypto). */ + union { + cryptodev_sym_cpu_crypto_process_t sym_cpu_process; + /**< process input data synchronously (cpu-crypto). */ + __extension__ + struct { + cryptodev_sym_get_raw_dp_ctx_size_t + sym_get_raw_dp_ctx_size; + /**< Get raw data path service context data size. */ + cryptodev_sym_configure_raw_dp_ctx_t + sym_configure_raw_dp_ctx; + /**< Initialize raw data path context data. */ + }; + }; }; diff --git a/lib/librte_cryptodev/rte_cryptodev_version.map b/lib/librte_cryptodev/rte_cryptodev_version.map index 7727286ac..7e4360ff0 100644 --- a/lib/librte_cryptodev/rte_cryptodev_version.map +++ b/lib/librte_cryptodev/rte_cryptodev_version.map @@ -99,4 +99,14 @@ EXPERIMENTAL { # added in 20.08 rte_cryptodev_get_qp_status; + + # added in 20.11 + rte_cryptodev_configure_raw_dp_ctx; + rte_cryptodev_get_raw_dp_ctx_size; + rte_cryptodev_raw_dequeue; + rte_cryptodev_raw_dequeue_burst; + rte_cryptodev_raw_dequeue_done; + rte_cryptodev_raw_enqueue; + rte_cryptodev_raw_enqueue_burst; + rte_cryptodev_raw_enqueue_done; };