Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/85658/?format=api
{ "id": 85658, "url": "https://patches.dpdk.org/api/patches/85658/?format=api", "web_url": "https://patches.dpdk.org/project/dpdk/patch/20201222144243.552262-2-abhinandan.gujjar@intel.com/", "project": { "id": 1, "url": "https://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20201222144243.552262-2-abhinandan.gujjar@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20201222144243.552262-2-abhinandan.gujjar@intel.com", "date": "2020-12-22T14:42:42", "name": "[v7,1/2] cryptodev: support enqueue and dequeue callback functions", "commit_ref": null, "pull_url": null, "state": "accepted", "archived": true, "hash": "2fba1cb0eb4846a879909183329bb964611f207e", "submitter": { "id": 883, "url": "https://patches.dpdk.org/api/people/883/?format=api", "name": "Gujjar, Abhinandan S", "email": "abhinandan.gujjar@intel.com" }, "delegate": { "id": 6690, "url": "https://patches.dpdk.org/api/users/6690/?format=api", "username": "akhil", "first_name": "akhil", "last_name": "goyal", "email": "gakhil@marvell.com" }, "mbox": "https://patches.dpdk.org/project/dpdk/patch/20201222144243.552262-2-abhinandan.gujjar@intel.com/mbox/", "series": [ { "id": 14425, "url": "https://patches.dpdk.org/api/series/14425/?format=api", "web_url": "https://patches.dpdk.org/project/dpdk/list/?series=14425", "date": "2020-12-22T14:42:41", "name": "support enqueue & dequeue callbacks on cryptodev", "version": 7, "mbox": "https://patches.dpdk.org/series/14425/mbox/" } ], "comments": "https://patches.dpdk.org/api/patches/85658/comments/", "check": "warning", "checks": "https://patches.dpdk.org/api/patches/85658/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from dpdk.org (dpdk.org [92.243.14.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 590D4A052A;\n\tTue, 22 Dec 2020 15:43:43 +0100 (CET)", "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 92B3ACA4B;\n\tTue, 22 Dec 2020 15:43:32 +0100 (CET)", "from mga04.intel.com (mga04.intel.com [192.55.52.120])\n by dpdk.org (Postfix) with ESMTP id 31B02CA4B\n for <dev@dpdk.org>; Tue, 22 Dec 2020 15:43:31 +0100 (CET)", "from fmsmga003.fm.intel.com ([10.253.24.29])\n by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;\n 22 Dec 2020 06:43:29 -0800", "from unknown (HELO broadwell-dev-2.iind.intel.com) ([10.190.210.98])\n by FMSMGA003.fm.intel.com with ESMTP; 22 Dec 2020 06:43:26 -0800" ], "IronPort-SDR": [ "\n r0NzZnIaO6MULzVf7f6y5QDqwooXrTYwjZJWq2ChxRxADjSkPU2g3hHhs2yieFTHZWQNTt+4+w\n GB6icY50vWFw==", "\n Mosbk4jgW+o3gguwmzbD3U+dtFNdepWrYGwo8aLoBi+f2CKq/CftlEJcQ6AYha0e/Zm3e3GzHH\n G2XTf6XIPHHw==" ], "X-IronPort-AV": [ "E=McAfee;i=\"6000,8403,9842\"; a=\"173307011\"", "E=Sophos;i=\"5.78,438,1599548400\"; d=\"scan'208\";a=\"173307011\"", "E=Sophos;i=\"5.78,438,1599548400\"; d=\"scan'208\";a=\"397228574\"" ], "X-ExtLoop1": "1", "From": "Abhinandan Gujjar <abhinandan.gujjar@intel.com>", "To": "dev@dpdk.org,\n\takhil.goyal@nxp.com,\n\tkonstantin.ananyev@intel.com", "Cc": "abhinandan.gujjar@intel.com", "Date": "Tue, 22 Dec 2020 20:12:42 +0530", "Message-Id": "<20201222144243.552262-2-abhinandan.gujjar@intel.com>", "X-Mailer": "git-send-email 2.25.1", "In-Reply-To": "<20201222144243.552262-1-abhinandan.gujjar@intel.com>", "References": "<20201222144243.552262-1-abhinandan.gujjar@intel.com>", "MIME-Version": "1.0", "Content-Transfer-Encoding": "8bit", "Subject": "[dpdk-dev] [PATCH v7 1/2] cryptodev: support enqueue and dequeue\n\tcallback functions", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "This patch adds APIs to add/remove callback functions on crypto\nenqueue/dequeue burst. The callback function will be called for\neach burst of crypto ops received/sent on a given crypto device\nqueue pair.\n\nSigned-off-by: Abhinandan Gujjar <abhinandan.gujjar@intel.com>\nAcked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>\n---\n config/rte_config.h | 1 +\n doc/guides/prog_guide/cryptodev_lib.rst | 44 +++\n doc/guides/rel_notes/release_21_02.rst | 9 +\n lib/librte_cryptodev/meson.build | 2 +-\n lib/librte_cryptodev/rte_cryptodev.c | 398 +++++++++++++++++++++++-\n lib/librte_cryptodev/rte_cryptodev.h | 246 ++++++++++++++-\n lib/librte_cryptodev/version.map | 7 +\n 7 files changed, 702 insertions(+), 5 deletions(-)", "diff": "diff --git a/config/rte_config.h b/config/rte_config.h\nindex a0b5160ff..87f9786d7 100644\n--- a/config/rte_config.h\n+++ b/config/rte_config.h\n@@ -62,6 +62,7 @@\n /* cryptodev defines */\n #define RTE_CRYPTO_MAX_DEVS 64\n #define RTE_CRYPTODEV_NAME_LEN 64\n+#define RTE_CRYPTO_CALLBACKS 1\n \n /* compressdev defines */\n #define RTE_COMPRESS_MAX_DEVS 64\ndiff --git a/doc/guides/prog_guide/cryptodev_lib.rst b/doc/guides/prog_guide/cryptodev_lib.rst\nindex 473b014a1..9b1cf8d49 100644\n--- a/doc/guides/prog_guide/cryptodev_lib.rst\n+++ b/doc/guides/prog_guide/cryptodev_lib.rst\n@@ -338,6 +338,50 @@ start of private data information. The offset is counted from the start of the\n rte_crypto_op including other crypto information such as the IVs (since there can\n be an IV also for authentication).\n \n+User callback APIs\n+~~~~~~~~~~~~~~~~~~\n+The add APIs configures a user callback function to be called for each burst of crypto\n+ops received/sent on a given crypto device queue pair. The return value is a pointer\n+that can be used later to remove the callback using remove API. Application is expected\n+to register a callback function of type ``rte_cryptodev_callback_fn``. Multiple callback\n+functions can be added for a given queue pair. API does not restrict on maximum number of\n+callbacks.\n+\n+Callbacks registered by application would not survive ``rte_cryptodev_configure`` as it\n+reinitializes the callback list. It is user responsibility to remove all installed\n+callbacks before calling ``rte_cryptodev_configure`` to avoid possible memory leakage.\n+\n+So, the application is expected to add user callback after ``rte_cryptodev_configure``.\n+The callbacks can also be added at the runtime. These callbacks get executed when\n+``rte_cryptodev_enqueue_burst``/``rte_cryptodev_dequeue_burst`` is called.\n+\n+.. code-block:: c\n+\n+\tstruct rte_cryptodev_cb *\n+\t\trte_cryptodev_add_enq_callback(uint8_t dev_id, uint16_t qp_id,\n+\t\t\t\t\t rte_cryptodev_callback_fn cb_fn,\n+\t\t\t\t\t void *cb_arg);\n+\n+\tstruct rte_cryptodev_cb *\n+\t\trte_cryptodev_add_deq_callback(uint8_t dev_id, uint16_t qp_id,\n+\t\t\t\t\t rte_cryptodev_callback_fn cb_fn,\n+\t\t\t\t\t void *cb_arg);\n+\n+\tuint16_t (* rte_cryptodev_callback_fn)(uint16_t dev_id, uint16_t qp_id,\n+\t\t\t\t\t struct rte_crypto_op **ops,\n+\t\t\t\t\t uint16_t nb_ops, void *user_param);\n+\n+The remove API removes a callback function added by\n+``rte_cryptodev_add_enq_callback``/``rte_cryptodev_add_deq_callback``.\n+\n+.. code-block:: c\n+\n+\tint rte_cryptodev_remove_enq_callback(uint8_t dev_id, uint16_t qp_id,\n+\t\t\t\t\t struct rte_cryptodev_cb *cb);\n+\n+\tint rte_cryptodev_remove_deq_callback(uint8_t dev_id, uint16_t qp_id,\n+\t\t\t\t\t struct rte_cryptodev_cb *cb);\n+\n \n Enqueue / Dequeue Burst APIs\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~\ndiff --git a/doc/guides/rel_notes/release_21_02.rst b/doc/guides/rel_notes/release_21_02.rst\nindex 638f98168..8c7866401 100644\n--- a/doc/guides/rel_notes/release_21_02.rst\n+++ b/doc/guides/rel_notes/release_21_02.rst\n@@ -55,6 +55,13 @@ New Features\n Also, make sure to start the actual text at the margin.\n =======================================================\n \n+* **Added enqueue & dequeue callback APIs for cryptodev library.**\n+\n+ Cryptodev library is added with enqueue & dequeue callback APIs to\n+ enable applications to add/remove user callbacks which gets called\n+ for every enqueue/dequeue operation.\n+\n+\n \n Removed Items\n -------------\n@@ -84,6 +91,8 @@ API Changes\n Also, make sure to start the actual text at the margin.\n =======================================================\n \n+* cryptodev: The structure ``rte_cryptodev`` has been updated with pointers\n+ for adding enqueue and dequeue callbacks.\n \n ABI Changes\n -----------\ndiff --git a/lib/librte_cryptodev/meson.build b/lib/librte_cryptodev/meson.build\nindex c4c6b3b6a..8c5493f4c 100644\n--- a/lib/librte_cryptodev/meson.build\n+++ b/lib/librte_cryptodev/meson.build\n@@ -9,4 +9,4 @@ headers = files('rte_cryptodev.h',\n \t'rte_crypto.h',\n \t'rte_crypto_sym.h',\n \t'rte_crypto_asym.h')\n-deps += ['kvargs', 'mbuf']\n+deps += ['kvargs', 'mbuf', 'rcu']\ndiff --git a/lib/librte_cryptodev/rte_cryptodev.c b/lib/librte_cryptodev/rte_cryptodev.c\nindex 3d95ac6ea..40f55a3cd 100644\n--- a/lib/librte_cryptodev/rte_cryptodev.c\n+++ b/lib/librte_cryptodev/rte_cryptodev.c\n@@ -448,6 +448,122 @@ rte_cryptodev_asym_xform_capability_check_modlen(\n \treturn 0;\n }\n \n+/* spinlock for crypto device enq callbacks */\n+static rte_spinlock_t rte_cryptodev_callback_lock = RTE_SPINLOCK_INITIALIZER;\n+\n+static void\n+cryptodev_cb_cleanup(struct rte_cryptodev *dev)\n+{\n+\tstruct rte_cryptodev_cb_rcu *list;\n+\tstruct rte_cryptodev_cb *cb, *next;\n+\tuint16_t qp_id;\n+\n+\tif (dev->enq_cbs == NULL && dev->deq_cbs == NULL)\n+\t\treturn;\n+\n+\tfor (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {\n+\t\tlist = &dev->enq_cbs[qp_id];\n+\t\tcb = list->next;\n+\t\twhile (cb != NULL) {\n+\t\t\tnext = cb->next;\n+\t\t\trte_free(cb);\n+\t\t\tcb = next;\n+\t\t}\n+\n+\t\trte_free(list->qsbr);\n+\t}\n+\n+\tfor (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {\n+\t\tlist = &dev->deq_cbs[qp_id];\n+\t\tcb = list->next;\n+\t\twhile (cb != NULL) {\n+\t\t\tnext = cb->next;\n+\t\t\trte_free(cb);\n+\t\t\tcb = next;\n+\t\t}\n+\n+\t\trte_free(list->qsbr);\n+\t}\n+\n+\trte_free(dev->enq_cbs);\n+\tdev->enq_cbs = NULL;\n+\trte_free(dev->deq_cbs);\n+\tdev->deq_cbs = NULL;\n+}\n+\n+static int\n+cryptodev_cb_init(struct rte_cryptodev *dev)\n+{\n+\tstruct rte_cryptodev_cb_rcu *list;\n+\tstruct rte_rcu_qsbr *qsbr;\n+\tuint16_t qp_id;\n+\tsize_t size;\n+\n+\t/* Max thread set to 1, as one DP thread accessing a queue-pair */\n+\tconst uint32_t max_threads = 1;\n+\n+\tdev->enq_cbs = rte_zmalloc(NULL,\n+\t\t\t\t sizeof(struct rte_cryptodev_cb_rcu) *\n+\t\t\t\t dev->data->nb_queue_pairs, 0);\n+\tif (dev->enq_cbs == NULL) {\n+\t\tCDEV_LOG_ERR(\"Failed to allocate memory for enq callbacks\");\n+\t\treturn -ENOMEM;\n+\t}\n+\n+\tdev->deq_cbs = rte_zmalloc(NULL,\n+\t\t\t\t sizeof(struct rte_cryptodev_cb_rcu) *\n+\t\t\t\t dev->data->nb_queue_pairs, 0);\n+\tif (dev->deq_cbs == NULL) {\n+\t\tCDEV_LOG_ERR(\"Failed to allocate memory for deq callbacks\");\n+\t\trte_free(dev->enq_cbs);\n+\t\treturn -ENOMEM;\n+\t}\n+\n+\t/* Create RCU QSBR variable */\n+\tsize = rte_rcu_qsbr_get_memsize(max_threads);\n+\n+\tfor (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {\n+\t\tlist = &dev->enq_cbs[qp_id];\n+\t\tqsbr = rte_zmalloc(NULL, size, RTE_CACHE_LINE_SIZE);\n+\t\tif (qsbr == NULL) {\n+\t\t\tCDEV_LOG_ERR(\"Failed to allocate memory for RCU on \"\n+\t\t\t\t\"queue_pair_id=%d\", qp_id);\n+\t\t\tgoto cb_init_err;\n+\t\t}\n+\n+\t\tif (rte_rcu_qsbr_init(qsbr, max_threads)) {\n+\t\t\tCDEV_LOG_ERR(\"Failed to initialize for RCU on \"\n+\t\t\t\t\"queue_pair_id=%d\", qp_id);\n+\t\t\tgoto cb_init_err;\n+\t\t}\n+\n+\t\tlist->qsbr = qsbr;\n+\t}\n+\n+\tfor (qp_id = 0; qp_id < dev->data->nb_queue_pairs; qp_id++) {\n+\t\tlist = &dev->deq_cbs[qp_id];\n+\t\tqsbr = rte_zmalloc(NULL, size, RTE_CACHE_LINE_SIZE);\n+\t\tif (qsbr == NULL) {\n+\t\t\tCDEV_LOG_ERR(\"Failed to allocate memory for RCU on \"\n+\t\t\t\t\"queue_pair_id=%d\", qp_id);\n+\t\t\tgoto cb_init_err;\n+\t\t}\n+\n+\t\tif (rte_rcu_qsbr_init(qsbr, max_threads)) {\n+\t\t\tCDEV_LOG_ERR(\"Failed to initialize for RCU on \"\n+\t\t\t\t\"queue_pair_id=%d\", qp_id);\n+\t\t\tgoto cb_init_err;\n+\t\t}\n+\n+\t\tlist->qsbr = qsbr;\n+\t}\n+\n+\treturn 0;\n+\n+cb_init_err:\n+\tcryptodev_cb_cleanup(dev);\n+\treturn -ENOMEM;\n+}\n \n const char *\n rte_cryptodev_get_feature_name(uint64_t flag)\n@@ -927,6 +1043,10 @@ rte_cryptodev_configure(uint8_t dev_id, struct rte_cryptodev_config *config)\n \n \tRTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_configure, -ENOTSUP);\n \n+\trte_spinlock_lock(&rte_cryptodev_callback_lock);\n+\tcryptodev_cb_cleanup(dev);\n+\trte_spinlock_unlock(&rte_cryptodev_callback_lock);\n+\n \t/* Setup new number of queue pairs and reconfigure device. */\n \tdiag = rte_cryptodev_queue_pairs_config(dev, config->nb_queue_pairs,\n \t\t\tconfig->socket_id);\n@@ -936,11 +1056,18 @@ rte_cryptodev_configure(uint8_t dev_id, struct rte_cryptodev_config *config)\n \t\treturn diag;\n \t}\n \n+\trte_spinlock_lock(&rte_cryptodev_callback_lock);\n+\tdiag = cryptodev_cb_init(dev);\n+\trte_spinlock_unlock(&rte_cryptodev_callback_lock);\n+\tif (diag) {\n+\t\tCDEV_LOG_ERR(\"Callback init failed for dev_id=%d\", dev_id);\n+\t\treturn diag;\n+\t}\n+\n \trte_cryptodev_trace_configure(dev_id, config);\n \treturn (*dev->dev_ops->dev_configure)(dev, config);\n }\n \n-\n int\n rte_cryptodev_start(uint8_t dev_id)\n {\n@@ -1136,6 +1263,275 @@ rte_cryptodev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,\n \t\t\tsocket_id);\n }\n \n+struct rte_cryptodev_cb *\n+rte_cryptodev_add_enq_callback(uint8_t dev_id,\n+\t\t\t uint16_t qp_id,\n+\t\t\t rte_cryptodev_callback_fn cb_fn,\n+\t\t\t void *cb_arg)\n+{\n+\tstruct rte_cryptodev *dev;\n+\tstruct rte_cryptodev_cb_rcu *list;\n+\tstruct rte_cryptodev_cb *cb, *tail;\n+\n+\tif (!cb_fn) {\n+\t\tCDEV_LOG_ERR(\"Callback is NULL on dev_id=%d\", dev_id);\n+\t\trte_errno = EINVAL;\n+\t\treturn NULL;\n+\t}\n+\n+\tif (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {\n+\t\tCDEV_LOG_ERR(\"Invalid dev_id=%d\", dev_id);\n+\t\trte_errno = ENODEV;\n+\t\treturn NULL;\n+\t}\n+\n+\tdev = &rte_crypto_devices[dev_id];\n+\tif (qp_id >= dev->data->nb_queue_pairs) {\n+\t\tCDEV_LOG_ERR(\"Invalid queue_pair_id=%d\", qp_id);\n+\t\trte_errno = ENODEV;\n+\t\treturn NULL;\n+\t}\n+\n+\tcb = rte_zmalloc(NULL, sizeof(*cb), 0);\n+\tif (cb == NULL) {\n+\t\tCDEV_LOG_ERR(\"Failed to allocate memory for callback on \"\n+\t\t\t \"dev=%d, queue_pair_id=%d\", dev_id, qp_id);\n+\t\trte_errno = ENOMEM;\n+\t\treturn NULL;\n+\t}\n+\n+\trte_spinlock_lock(&rte_cryptodev_callback_lock);\n+\n+\tcb->fn = cb_fn;\n+\tcb->arg = cb_arg;\n+\n+\t/* Add the callbacks in fifo order. */\n+\tlist = &dev->enq_cbs[qp_id];\n+\ttail = list->next;\n+\n+\tif (tail) {\n+\t\twhile (tail->next)\n+\t\t\ttail = tail->next;\n+\t\t/* Stores to cb->fn and cb->param should complete before\n+\t\t * cb is visible to data plane.\n+\t\t */\n+\t\t__atomic_store_n(&tail->next, cb, __ATOMIC_RELEASE);\n+\t} else {\n+\t\t/* Stores to cb->fn and cb->param should complete before\n+\t\t * cb is visible to data plane.\n+\t\t */\n+\t\t__atomic_store_n(&list->next, cb, __ATOMIC_RELEASE);\n+\t}\n+\n+\trte_spinlock_unlock(&rte_cryptodev_callback_lock);\n+\n+\treturn cb;\n+}\n+\n+int\n+rte_cryptodev_remove_enq_callback(uint8_t dev_id,\n+\t\t\t\t uint16_t qp_id,\n+\t\t\t\t struct rte_cryptodev_cb *cb)\n+{\n+\tstruct rte_cryptodev *dev;\n+\tstruct rte_cryptodev_cb **prev_cb, *curr_cb;\n+\tstruct rte_cryptodev_cb_rcu *list;\n+\tint ret;\n+\n+\tret = -EINVAL;\n+\n+\tif (!cb) {\n+\t\tCDEV_LOG_ERR(\"Callback is NULL\");\n+\t\treturn -EINVAL;\n+\t}\n+\n+\tif (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {\n+\t\tCDEV_LOG_ERR(\"Invalid dev_id=%d\", dev_id);\n+\t\treturn -ENODEV;\n+\t}\n+\n+\tdev = &rte_crypto_devices[dev_id];\n+\tif (qp_id >= dev->data->nb_queue_pairs) {\n+\t\tCDEV_LOG_ERR(\"Invalid queue_pair_id=%d\", qp_id);\n+\t\treturn -ENODEV;\n+\t}\n+\n+\trte_spinlock_lock(&rte_cryptodev_callback_lock);\n+\tif (dev->enq_cbs == NULL) {\n+\t\tCDEV_LOG_ERR(\"Callback not initialized\");\n+\t\tgoto cb_err;\n+\t}\n+\n+\tlist = &dev->enq_cbs[qp_id];\n+\tif (list == NULL) {\n+\t\tCDEV_LOG_ERR(\"Callback list is NULL\");\n+\t\tgoto cb_err;\n+\t}\n+\n+\tif (list->qsbr == NULL) {\n+\t\tCDEV_LOG_ERR(\"Rcu qsbr is NULL\");\n+\t\tgoto cb_err;\n+\t}\n+\n+\tprev_cb = &list->next;\n+\tfor (; *prev_cb != NULL; prev_cb = &curr_cb->next) {\n+\t\tcurr_cb = *prev_cb;\n+\t\tif (curr_cb == cb) {\n+\t\t\t/* Remove the user cb from the callback list. */\n+\t\t\t__atomic_store_n(prev_cb, curr_cb->next,\n+\t\t\t\t__ATOMIC_RELAXED);\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (!ret) {\n+\t\t/* Call sync with invalid thread id as this is part of\n+\t\t * control plane API\n+\t\t */\n+\t\trte_rcu_qsbr_synchronize(list->qsbr, RTE_QSBR_THRID_INVALID);\n+\t\trte_free(cb);\n+\t}\n+\n+cb_err:\n+\trte_spinlock_unlock(&rte_cryptodev_callback_lock);\n+\treturn ret;\n+}\n+\n+struct rte_cryptodev_cb *\n+rte_cryptodev_add_deq_callback(uint8_t dev_id,\n+\t\t\t uint16_t qp_id,\n+\t\t\t rte_cryptodev_callback_fn cb_fn,\n+\t\t\t void *cb_arg)\n+{\n+\tstruct rte_cryptodev *dev;\n+\tstruct rte_cryptodev_cb_rcu *list;\n+\tstruct rte_cryptodev_cb *cb, *tail;\n+\n+\tif (!cb_fn) {\n+\t\tCDEV_LOG_ERR(\"Callback is NULL on dev_id=%d\", dev_id);\n+\t\trte_errno = EINVAL;\n+\t\treturn NULL;\n+\t}\n+\n+\tif (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {\n+\t\tCDEV_LOG_ERR(\"Invalid dev_id=%d\", dev_id);\n+\t\trte_errno = ENODEV;\n+\t\treturn NULL;\n+\t}\n+\n+\tdev = &rte_crypto_devices[dev_id];\n+\tif (qp_id >= dev->data->nb_queue_pairs) {\n+\t\tCDEV_LOG_ERR(\"Invalid queue_pair_id=%d\", qp_id);\n+\t\trte_errno = ENODEV;\n+\t\treturn NULL;\n+\t}\n+\n+\tcb = rte_zmalloc(NULL, sizeof(*cb), 0);\n+\tif (cb == NULL) {\n+\t\tCDEV_LOG_ERR(\"Failed to allocate memory for callback on \"\n+\t\t\t \"dev=%d, queue_pair_id=%d\", dev_id, qp_id);\n+\t\trte_errno = ENOMEM;\n+\t\treturn NULL;\n+\t}\n+\n+\trte_spinlock_lock(&rte_cryptodev_callback_lock);\n+\n+\tcb->fn = cb_fn;\n+\tcb->arg = cb_arg;\n+\n+\t/* Add the callbacks in fifo order. */\n+\tlist = &dev->deq_cbs[qp_id];\n+\ttail = list->next;\n+\n+\tif (tail) {\n+\t\twhile (tail->next)\n+\t\t\ttail = tail->next;\n+\t\t/* Stores to cb->fn and cb->param should complete before\n+\t\t * cb is visible to data plane.\n+\t\t */\n+\t\t__atomic_store_n(&tail->next, cb, __ATOMIC_RELEASE);\n+\t} else {\n+\t\t/* Stores to cb->fn and cb->param should complete before\n+\t\t * cb is visible to data plane.\n+\t\t */\n+\t\t__atomic_store_n(&list->next, cb, __ATOMIC_RELEASE);\n+\t}\n+\n+\trte_spinlock_unlock(&rte_cryptodev_callback_lock);\n+\n+\treturn cb;\n+}\n+\n+int\n+rte_cryptodev_remove_deq_callback(uint8_t dev_id,\n+\t\t\t\t uint16_t qp_id,\n+\t\t\t\t struct rte_cryptodev_cb *cb)\n+{\n+\tstruct rte_cryptodev *dev;\n+\tstruct rte_cryptodev_cb **prev_cb, *curr_cb;\n+\tstruct rte_cryptodev_cb_rcu *list;\n+\tint ret;\n+\n+\tret = -EINVAL;\n+\n+\tif (!cb) {\n+\t\tCDEV_LOG_ERR(\"Callback is NULL\");\n+\t\treturn -EINVAL;\n+\t}\n+\n+\tif (!rte_cryptodev_pmd_is_valid_dev(dev_id)) {\n+\t\tCDEV_LOG_ERR(\"Invalid dev_id=%d\", dev_id);\n+\t\treturn -ENODEV;\n+\t}\n+\n+\tdev = &rte_crypto_devices[dev_id];\n+\tif (qp_id >= dev->data->nb_queue_pairs) {\n+\t\tCDEV_LOG_ERR(\"Invalid queue_pair_id=%d\", qp_id);\n+\t\treturn -ENODEV;\n+\t}\n+\n+\trte_spinlock_lock(&rte_cryptodev_callback_lock);\n+\tif (dev->enq_cbs == NULL) {\n+\t\tCDEV_LOG_ERR(\"Callback not initialized\");\n+\t\tgoto cb_err;\n+\t}\n+\n+\tlist = &dev->deq_cbs[qp_id];\n+\tif (list == NULL) {\n+\t\tCDEV_LOG_ERR(\"Callback list is NULL\");\n+\t\tgoto cb_err;\n+\t}\n+\n+\tif (list->qsbr == NULL) {\n+\t\tCDEV_LOG_ERR(\"Rcu qsbr is NULL\");\n+\t\tgoto cb_err;\n+\t}\n+\n+\tprev_cb = &list->next;\n+\tfor (; *prev_cb != NULL; prev_cb = &curr_cb->next) {\n+\t\tcurr_cb = *prev_cb;\n+\t\tif (curr_cb == cb) {\n+\t\t\t/* Remove the user cb from the callback list. */\n+\t\t\t__atomic_store_n(prev_cb, curr_cb->next,\n+\t\t\t\t__ATOMIC_RELAXED);\n+\t\t\tret = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (!ret) {\n+\t\t/* Call sync with invalid thread id as this is part of\n+\t\t * control plane API\n+\t\t */\n+\t\trte_rcu_qsbr_synchronize(list->qsbr, RTE_QSBR_THRID_INVALID);\n+\t\trte_free(cb);\n+\t}\n+\n+cb_err:\n+\trte_spinlock_unlock(&rte_cryptodev_callback_lock);\n+\treturn ret;\n+}\n \n int\n rte_cryptodev_stats_get(uint8_t dev_id, struct rte_cryptodev_stats *stats)\ndiff --git a/lib/librte_cryptodev/rte_cryptodev.h b/lib/librte_cryptodev/rte_cryptodev.h\nindex 0935fd587..ae34f33f6 100644\n--- a/lib/librte_cryptodev/rte_cryptodev.h\n+++ b/lib/librte_cryptodev/rte_cryptodev.h\n@@ -23,6 +23,7 @@ extern \"C\" {\n #include \"rte_dev.h\"\n #include <rte_common.h>\n #include <rte_config.h>\n+#include <rte_rcu_qsbr.h>\n \n #include \"rte_cryptodev_trace_fp.h\"\n \n@@ -522,6 +523,30 @@ struct rte_cryptodev_qp_conf {\n \t/**< The mempool for creating sess private data in sessionless mode */\n };\n \n+/**\n+ * Function type used for processing crypto ops when enqueue/dequeue burst is\n+ * called.\n+ *\n+ * The callback function is called on enqueue/dequeue burst immediately.\n+ *\n+ * @param\tdev_id\t\tThe identifier of the device.\n+ * @param\tqp_id\t\tThe index of the queue pair on which ops are\n+ *\t\t\t\tenqueued/dequeued. The value must be in the\n+ *\t\t\t\trange [0, nb_queue_pairs - 1] previously\n+ *\t\t\t\tsupplied to *rte_cryptodev_configure*.\n+ * @param\tops\t\tThe address of an array of *nb_ops* pointers\n+ *\t\t\t\tto *rte_crypto_op* structures which contain\n+ *\t\t\t\tthe crypto operations to be processed.\n+ * @param\tnb_ops\t\tThe number of operations to process.\n+ * @param\tuser_param\tThe arbitrary user parameter passed in by the\n+ *\t\t\t\tapplication when the callback was originally\n+ *\t\t\t\tregistered.\n+ * @return\t\t\tThe number of ops to be enqueued to the\n+ *\t\t\t\tcrypto device.\n+ */\n+typedef uint16_t (*rte_cryptodev_callback_fn)(uint16_t dev_id, uint16_t qp_id,\n+\t\tstruct rte_crypto_op **ops, uint16_t nb_ops, void *user_param);\n+\n /**\n * Typedef for application callback function to be registered by application\n * software for notification of device events\n@@ -822,7 +847,6 @@ rte_cryptodev_callback_unregister(uint8_t dev_id,\n \t\tenum rte_cryptodev_event_type event,\n \t\trte_cryptodev_cb_fn cb_fn, void *cb_arg);\n \n-\n typedef uint16_t (*dequeue_pkt_burst_t)(void *qp,\n \t\tstruct rte_crypto_op **ops,\tuint16_t nb_ops);\n /**< Dequeue processed packets from queue pair of a device. */\n@@ -839,6 +863,30 @@ struct rte_cryptodev_callback;\n /** Structure to keep track of registered callbacks */\n TAILQ_HEAD(rte_cryptodev_cb_list, rte_cryptodev_callback);\n \n+/**\n+ * Structure used to hold information about the callbacks to be called for a\n+ * queue pair on enqueue/dequeue.\n+ */\n+struct rte_cryptodev_cb {\n+\tstruct rte_cryptodev_cb *next;\n+\t/**< Pointer to next callback */\n+\trte_cryptodev_callback_fn fn;\n+\t/**< Pointer to callback function */\n+\tvoid *arg;\n+\t/**< Pointer to argument */\n+};\n+\n+/**\n+ * @internal\n+ * Structure used to hold information about the RCU for a queue pair.\n+ */\n+struct rte_cryptodev_cb_rcu {\n+\tstruct rte_cryptodev_cb *next;\n+\t/**< Pointer to next callback */\n+\tstruct rte_rcu_qsbr *qsbr;\n+\t/**< RCU QSBR variable per queue pair */\n+};\n+\n /** The data structure associated with each crypto device. */\n struct rte_cryptodev {\n \tdequeue_pkt_burst_t dequeue_burst;\n@@ -867,6 +915,12 @@ struct rte_cryptodev {\n \t__extension__\n \tuint8_t attached : 1;\n \t/**< Flag indicating the device is attached */\n+\n+\tstruct rte_cryptodev_cb_rcu *enq_cbs;\n+\t/**< User application callback for pre enqueue processing */\n+\n+\tstruct rte_cryptodev_cb_rcu *deq_cbs;\n+\t/**< User application callback for post dequeue processing */\n } __rte_cache_aligned;\n \n void *\n@@ -945,10 +999,33 @@ rte_cryptodev_dequeue_burst(uint8_t dev_id, uint16_t qp_id,\n {\n \tstruct rte_cryptodev *dev = &rte_cryptodevs[dev_id];\n \n+\trte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);\n \tnb_ops = (*dev->dequeue_burst)\n \t\t\t(dev->data->queue_pairs[qp_id], ops, nb_ops);\n-\n-\trte_cryptodev_trace_dequeue_burst(dev_id, qp_id, (void **)ops, nb_ops);\n+#ifdef RTE_CRYPTO_CALLBACKS\n+\tif (unlikely(dev->deq_cbs != NULL)) {\n+\t\tstruct rte_cryptodev_cb_rcu *list;\n+\t\tstruct rte_cryptodev_cb *cb;\n+\n+\t\t/* __ATOMIC_RELEASE memory order was used when the\n+\t\t * call back was inserted into the list.\n+\t\t * Since there is a clear dependency between loading\n+\t\t * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is\n+\t\t * not required.\n+\t\t */\n+\t\tlist = &dev->deq_cbs[qp_id];\n+\t\trte_rcu_qsbr_thread_online(list->qsbr, 0);\n+\t\tcb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);\n+\n+\t\twhile (cb != NULL) {\n+\t\t\tnb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,\n+\t\t\t\t\tcb->arg);\n+\t\t\tcb = cb->next;\n+\t\t};\n+\n+\t\trte_rcu_qsbr_thread_offline(list->qsbr, 0);\n+\t}\n+#endif\n \treturn nb_ops;\n }\n \n@@ -989,6 +1066,31 @@ rte_cryptodev_enqueue_burst(uint8_t dev_id, uint16_t qp_id,\n {\n \tstruct rte_cryptodev *dev = &rte_cryptodevs[dev_id];\n \n+#ifdef RTE_CRYPTO_CALLBACKS\n+\tif (unlikely(dev->enq_cbs != NULL)) {\n+\t\tstruct rte_cryptodev_cb_rcu *list;\n+\t\tstruct rte_cryptodev_cb *cb;\n+\n+\t\t/* __ATOMIC_RELEASE memory order was used when the\n+\t\t * call back was inserted into the list.\n+\t\t * Since there is a clear dependency between loading\n+\t\t * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is\n+\t\t * not required.\n+\t\t */\n+\t\tlist = &dev->enq_cbs[qp_id];\n+\t\trte_rcu_qsbr_thread_online(list->qsbr, 0);\n+\t\tcb = __atomic_load_n(&list->next, __ATOMIC_RELAXED);\n+\n+\t\twhile (cb != NULL) {\n+\t\t\tnb_ops = cb->fn(dev_id, qp_id, ops, nb_ops,\n+\t\t\t\t\tcb->arg);\n+\t\t\tcb = cb->next;\n+\t\t};\n+\n+\t\trte_rcu_qsbr_thread_offline(list->qsbr, 0);\n+\t}\n+#endif\n+\n \trte_cryptodev_trace_enqueue_burst(dev_id, qp_id, (void **)ops, nb_ops);\n \treturn (*dev->enqueue_burst)(\n \t\t\tdev->data->queue_pairs[qp_id], ops, nb_ops);\n@@ -1730,6 +1832,144 @@ int\n rte_cryptodev_raw_dequeue_done(struct rte_crypto_raw_dp_ctx *ctx,\n \t\tuint32_t n);\n \n+/**\n+ * Add a user callback for a given crypto device and queue pair which will be\n+ * called on crypto ops enqueue.\n+ *\n+ * This API configures a function to be called for each burst of crypto ops\n+ * received on a given crypto device queue pair. The return value is a pointer\n+ * that can be used later to remove the callback using\n+ * rte_cryptodev_remove_enq_callback().\n+ *\n+ * Callbacks registered by application would not survive\n+ * rte_cryptodev_configure() as it reinitializes the callback list.\n+ * It is user responsibility to remove all installed callbacks before\n+ * calling rte_cryptodev_configure() to avoid possible memory leakage.\n+ * Application is expected to call add API after rte_cryptodev_configure().\n+ *\n+ * Multiple functions can be registered per queue pair & they are called\n+ * in the order they were added. The API does not restrict on maximum number\n+ * of callbacks.\n+ *\n+ * @param\tdev_id\t\tThe identifier of the device.\n+ * @param\tqp_id\t\tThe index of the queue pair on which ops are\n+ *\t\t\t\tto be enqueued for processing. The value\n+ *\t\t\t\tmust be in the range [0, nb_queue_pairs - 1]\n+ *\t\t\t\tpreviously supplied to\n+ *\t\t\t\t*rte_cryptodev_configure*.\n+ * @param\tcb_fn\t\tThe callback function\n+ * @param\tcb_arg\t\tA generic pointer parameter which will be passed\n+ *\t\t\t\tto each invocation of the callback function on\n+ *\t\t\t\tthis crypto device and queue pair.\n+ *\n+ * @return\n+ * - NULL on error & rte_errno will contain the error code.\n+ * - On success, a pointer value which can later be used to remove the\n+ * callback.\n+ */\n+\n+__rte_experimental\n+struct rte_cryptodev_cb *\n+rte_cryptodev_add_enq_callback(uint8_t dev_id,\n+\t\t\t uint16_t qp_id,\n+\t\t\t rte_cryptodev_callback_fn cb_fn,\n+\t\t\t void *cb_arg);\n+\n+/**\n+ * Remove a user callback function for given crypto device and queue pair.\n+ *\n+ * This function is used to remove enqueue callbacks that were added to a\n+ * crypto device queue pair using rte_cryptodev_add_enq_callback().\n+ *\n+ *\n+ *\n+ * @param\tdev_id\t\tThe identifier of the device.\n+ * @param\tqp_id\t\tThe index of the queue pair on which ops are\n+ *\t\t\t\tto be enqueued. The value must be in the\n+ *\t\t\t\trange [0, nb_queue_pairs - 1] previously\n+ *\t\t\t\tsupplied to *rte_cryptodev_configure*.\n+ * @param\tcb\t\tPointer to user supplied callback created via\n+ *\t\t\t\trte_cryptodev_add_enq_callback().\n+ *\n+ * @return\n+ * - 0: Success. Callback was removed.\n+ * - <0: The dev_id or the qp_id is out of range, or the callback\n+ * is NULL or not found for the crypto device queue pair.\n+ */\n+\n+__rte_experimental\n+int rte_cryptodev_remove_enq_callback(uint8_t dev_id,\n+\t\t\t\t uint16_t qp_id,\n+\t\t\t\t struct rte_cryptodev_cb *cb);\n+\n+/**\n+ * Add a user callback for a given crypto device and queue pair which will be\n+ * called on crypto ops dequeue.\n+ *\n+ * This API configures a function to be called for each burst of crypto ops\n+ * received on a given crypto device queue pair. The return value is a pointer\n+ * that can be used later to remove the callback using\n+ * rte_cryptodev_remove_deq_callback().\n+ *\n+ * Callbacks registered by application would not survive\n+ * rte_cryptodev_configure() as it reinitializes the callback list.\n+ * It is user responsibility to remove all installed callbacks before\n+ * calling rte_cryptodev_configure() to avoid possible memory leakage.\n+ * Application is expected to call add API after rte_cryptodev_configure().\n+ *\n+ * Multiple functions can be registered per queue pair & they are called\n+ * in the order they were added. The API does not restrict on maximum number\n+ * of callbacks.\n+ *\n+ * @param\tdev_id\t\tThe identifier of the device.\n+ * @param\tqp_id\t\tThe index of the queue pair on which ops are\n+ *\t\t\t\tto be dequeued. The value must be in the\n+ *\t\t\t\trange [0, nb_queue_pairs - 1] previously\n+ *\t\t\t\tsupplied to *rte_cryptodev_configure*.\n+ * @param\tcb_fn\t\tThe callback function\n+ * @param\tcb_arg\t\tA generic pointer parameter which will be passed\n+ *\t\t\t\tto each invocation of the callback function on\n+ *\t\t\t\tthis crypto device and queue pair.\n+ *\n+ * @return\n+ * - NULL on error & rte_errno will contain the error code.\n+ * - On success, a pointer value which can later be used to remove the\n+ * callback.\n+ */\n+\n+__rte_experimental\n+struct rte_cryptodev_cb *\n+rte_cryptodev_add_deq_callback(uint8_t dev_id,\n+\t\t\t uint16_t qp_id,\n+\t\t\t rte_cryptodev_callback_fn cb_fn,\n+\t\t\t void *cb_arg);\n+\n+/**\n+ * Remove a user callback function for given crypto device and queue pair.\n+ *\n+ * This function is used to remove dequeue callbacks that were added to a\n+ * crypto device queue pair using rte_cryptodev_add_deq_callback().\n+ *\n+ *\n+ *\n+ * @param\tdev_id\t\tThe identifier of the device.\n+ * @param\tqp_id\t\tThe index of the queue pair on which ops are\n+ *\t\t\t\tto be dequeued. The value must be in the\n+ *\t\t\t\trange [0, nb_queue_pairs - 1] previously\n+ *\t\t\t\tsupplied to *rte_cryptodev_configure*.\n+ * @param\tcb\t\tPointer to user supplied callback created via\n+ *\t\t\t\trte_cryptodev_add_deq_callback().\n+ *\n+ * @return\n+ * - 0: Success. Callback was removed.\n+ * - <0: The dev_id or the qp_id is out of range, or the callback\n+ * is NULL or not found for the crypto device queue pair.\n+ */\n+__rte_experimental\n+int rte_cryptodev_remove_deq_callback(uint8_t dev_id,\n+\t\t\t\t uint16_t qp_id,\n+\t\t\t\t struct rte_cryptodev_cb *cb);\n+\n #ifdef __cplusplus\n }\n #endif\ndiff --git a/lib/librte_cryptodev/version.map b/lib/librte_cryptodev/version.map\nindex 7e4360ff0..9f04737ae 100644\n--- a/lib/librte_cryptodev/version.map\n+++ b/lib/librte_cryptodev/version.map\n@@ -109,4 +109,11 @@ EXPERIMENTAL {\n \trte_cryptodev_raw_enqueue;\n \trte_cryptodev_raw_enqueue_burst;\n \trte_cryptodev_raw_enqueue_done;\n+\n+\t# added in 21.02\n+\trte_cryptodev_add_deq_callback;\n+\trte_cryptodev_add_enq_callback;\n+\trte_cryptodev_remove_deq_callback;\n+\trte_cryptodev_remove_enq_callback;\n+\n };\n", "prefixes": [ "v7", "1/2" ] }