Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/130171/?format=api
{ "id": 130171, "url": "https://patches.dpdk.org/api/patches/130171/?format=api", "web_url": "https://patches.dpdk.org/project/dpdk/patch/f91003785ebd4c1075629b3b3e654db6f77f61e9.1691768110.git.anatoly.burakov@intel.com/", "project": { "id": 1, "url": "https://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<f91003785ebd4c1075629b3b3e654db6f77f61e9.1691768110.git.anatoly.burakov@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/f91003785ebd4c1075629b3b3e654db6f77f61e9.1691768110.git.anatoly.burakov@intel.com", "date": "2023-08-11T16:14:46", "name": "[v1,3/3] dma/idxd: add API to create and attach to window", "commit_ref": null, "pull_url": null, "state": "new", "archived": false, "hash": "73099f22c1578d87983828dd01a47ce917a1beed", "submitter": { "id": 4, "url": "https://patches.dpdk.org/api/people/4/?format=api", "name": "Anatoly Burakov", "email": "anatoly.burakov@intel.com" }, "delegate": { "id": 1, "url": "https://patches.dpdk.org/api/users/1/?format=api", "username": "tmonjalo", "first_name": "Thomas", "last_name": "Monjalon", "email": "thomas@monjalon.net" }, "mbox": "https://patches.dpdk.org/project/dpdk/patch/f91003785ebd4c1075629b3b3e654db6f77f61e9.1691768110.git.anatoly.burakov@intel.com/mbox/", "series": [ { "id": 29192, "url": "https://patches.dpdk.org/api/series/29192/?format=api", "web_url": "https://patches.dpdk.org/project/dpdk/list/?series=29192", "date": "2023-08-11T16:14:43", "name": "Add support for inter-domain DMA operations", "version": 1, "mbox": "https://patches.dpdk.org/series/29192/mbox/" } ], "comments": "https://patches.dpdk.org/api/patches/130171/comments/", "check": "fail", "checks": "https://patches.dpdk.org/api/patches/130171/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from mails.dpdk.org (mails.dpdk.org [217.70.189.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 3233843036;\n\tFri, 11 Aug 2023 18:15:11 +0200 (CEST)", "from mails.dpdk.org (localhost [127.0.0.1])\n\tby mails.dpdk.org (Postfix) with ESMTP id B23974325A;\n\tFri, 11 Aug 2023 18:14:58 +0200 (CEST)", "from mgamail.intel.com (mgamail.intel.com [192.55.52.136])\n by mails.dpdk.org (Postfix) with ESMTP id 883444325A\n for <dev@dpdk.org>; Fri, 11 Aug 2023 18:14:56 +0200 (CEST)", "from orsmga005.jf.intel.com ([10.7.209.41])\n by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;\n 11 Aug 2023 09:14:56 -0700", "from silpixa00401191.ir.intel.com ([10.55.128.139])\n by orsmga005.jf.intel.com with ESMTP; 11 Aug 2023 09:14:54 -0700" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/simple;\n d=intel.com; i=@intel.com; q=dns/txt; s=Intel;\n t=1691770496; x=1723306496;\n h=from:to:cc:subject:date:message-id:in-reply-to:\n references:mime-version:content-transfer-encoding;\n bh=vQTH2B6xZeu4yWp0eoIE5VpqT9F8OxJgmRz4cr7MQsY=;\n b=ln7ffY1DXktlTllFfyANCHtJaJXwvMHPM4j5zpmM8B3+kjDvmMWViFIw\n 2ArNajUlC9VOyK3amTATMBdDTm9NyRFd6MtltFuDOKZNavNroE9YHjvKw\n nTbXMRl5tAlMjFiUQszhGo26/Lm6pSbuvJQn7p6UWyCIU352AM/KalfVs\n L0Y0BAddFR771AtwTfecz9LCel43PgtvHMkOZkQk6GnpjIDNlR8B4O4o4\n +aQYrUps6zpMUt+p4r3YQ7tUEEZ9Opj2DChv5LTahethFdQGMkIdJq0sI\n 6WtKwjeFsNwHmkkYJft6UwFzRcK6bXY36VW3LecFGrg7u3wnCWrL7SNlI Q==;", "X-IronPort-AV": [ "E=McAfee;i=\"6600,9927,10799\"; a=\"351312964\"", "E=Sophos;i=\"6.01,166,1684825200\"; d=\"scan'208\";a=\"351312964\"", "E=McAfee;i=\"6600,9927,10799\"; a=\"906499515\"", "E=Sophos;i=\"6.01,166,1684825200\"; d=\"scan'208\";a=\"906499515\"" ], "X-ExtLoop1": "1", "From": "Anatoly Burakov <anatoly.burakov@intel.com>", "To": "dev@dpdk.org, Bruce Richardson <bruce.richardson@intel.com>,\n Kevin Laatz <kevin.laatz@intel.com>", "Cc": "Vladimir Medvedkin <vladimir.medvedkin@intel.com>", "Subject": "[PATCH v1 3/3] dma/idxd: add API to create and attach to window", "Date": "Fri, 11 Aug 2023 16:14:46 +0000", "Message-Id": "\n <f91003785ebd4c1075629b3b3e654db6f77f61e9.1691768110.git.anatoly.burakov@intel.com>", "X-Mailer": "git-send-email 2.37.2", "In-Reply-To": "<cover.1691768109.git.anatoly.burakov@intel.com>", "References": "<cover.1691768109.git.anatoly.burakov@intel.com>", "MIME-Version": "1.0", "Content-Transfer-Encoding": "8bit", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org" }, "content": "This commit implements functions necessary to use inter-domain\noperations with idxd driver.\n\nThe process is as follows:\n\n1. Process A that wishes to share its memory with others, shall call\n `rte_idxd_window_create()`, which will return a file descriptor\n2. Process A is to send above mentioned file descriptor to any\n recipient process (usually over kernel IPC) that wishes to attach to\n that window\n3. Process B, after receiving above mentioned file descriptor from\n process A over IPC, shall call `rte_idxd_window_attach()` and\n receive an inter-pasid handle\n4. Process B shall use this handle as an argument for inter-domain\n operations using DMA device API\n\nSigned-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>\nSigned-off-by: Anatoly Burakov <anatoly.burakov@intel.com>\n---\n doc/guides/dmadevs/idxd.rst | 52 ++++++++\n drivers/dma/idxd/idxd_inter_dom.c | 166 ++++++++++++++++++++++++++\n drivers/dma/idxd/meson.build | 7 +-\n drivers/dma/idxd/rte_idxd_inter_dom.h | 79 ++++++++++++\n drivers/dma/idxd/version.map | 11 ++\n 5 files changed, 314 insertions(+), 1 deletion(-)\n create mode 100644 drivers/dma/idxd/idxd_inter_dom.c\n create mode 100644 drivers/dma/idxd/rte_idxd_inter_dom.h\n create mode 100644 drivers/dma/idxd/version.map", "diff": "diff --git a/doc/guides/dmadevs/idxd.rst b/doc/guides/dmadevs/idxd.rst\nindex cb8f1fe729..b0439377f8 100644\n--- a/doc/guides/dmadevs/idxd.rst\n+++ b/doc/guides/dmadevs/idxd.rst\n@@ -225,3 +225,55 @@ which operation failed and kick off the device to continue processing operations\n if (error){\n status_count = rte_dma_completed_status(dev_id, vchan, COMP_BURST_SZ, &idx, status);\n }\n+\n+Performing Inter-Domain operations\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Refer to the :ref:`Enqueue / Dequeue APIs <dmadev_enqueue_dequeue>` section of the dmadev library\n+documentation for details on operation enqueue, submission and completion API usage.\n+\n+Refer to the :ref:`Inter-domain operations <dmadev_inter_dom>` section of the dmadev library\n+documentation for details on inter-domain operations.\n+\n+Intel(R) IDXD currently supports the following inter-domain operations:\n+\n+* Copy operation\n+* Fill operation\n+\n+To use these operations with the IDXD driver, the following program flow should\n+be adhered to:\n+\n+* Process A that wishes to share its memory with others, shall call\n+ ``rte_idxd_window_create()``, which will return a file descriptor\n+* Process A is to send above mentioned file descriptor to any recipient process\n+ (usually over IPC) that wishes to attach to that window\n+* Process B, after receiving above mentioned file descriptor from process A over\n+ IPC, shall call ``rte_idxd_window_attach()`` and receive an inter-pasid handle\n+* Process B shall use this handle as an argument for inter-domain operations\n+ using DMA device API\n+\n+The controller ID parameter for create/attach functions in this case would be\n+the controller ID of configured DSA2 devices (located under ``rte_dma_info``\n+structure), but which can also be read from ``accel-config`` tool, or from the\n+DSA2 work queue name (e.g. work queue ``wq0.3`` would have ``0`` as its\n+controller ID).\n+\n+The ``rte_idxd_window_create()`` call will accept a ``flags`` argument, which\n+can contain the following bits:\n+\n+* ``RTE_IDXD_WIN_FLAGS_PROT_READ`` - allow other process to read from memory\n+ region to be shared\n+ - In this case, the remote process will be using the resulting inter-pasid\n+ handle as source handle for inter-domain DMA operations (and set the\n+ ``RTE_DMA_OP_FLAG_SRC_HANDLE`` DMA operation flag)\n+* ``RTE_IDXD_WIN_FLAGS_PROT_WRITE`` - allow other process to write into memory\n+ region to be shared\n+ - In this case, the remote process will be using the resulting inter-pasid\n+ handle as destination handle for inter-domain DMA operations (and set the\n+ ``RTE_DMA_OP_FLAG_DST_HANDLE`` DMA operation flag)\n+* ``RTE_IDXD_WIN_FLAGS_WIN_CHECK`` - if this flag is not set, the remote process\n+ will be allowed unrestricted access to entire memory space of the owner\n+ process\n+* ``RTE_IDXD_WIN_FLAGS_OFFSET_MODE`` - addresses for DMA operations will have to\n+ be specified as offsets from base address of the memory region to be shared\n+* ``RTE_IDXD_WIN_FLAGS_TYPE_SAMS`` - enable multi-submitter mode.\ndiff --git a/drivers/dma/idxd/idxd_inter_dom.c b/drivers/dma/idxd/idxd_inter_dom.c\nnew file mode 100644\nindex 0000000000..21dcd6980d\n--- /dev/null\n+++ b/drivers/dma/idxd/idxd_inter_dom.c\n@@ -0,0 +1,166 @@\n+/* SPDX-License-Identifier: BSD-3-Clause\n+ * Copyright(c) 2023 Intel Corporation\n+ */\n+\n+#include <stdlib.h>\n+#include <stdint.h>\n+#include <sys/ioctl.h>\n+#include <fcntl.h>\n+#include <unistd.h>\n+#include <dirent.h>\n+\n+#include <rte_common.h>\n+#include <rte_errno.h>\n+#include <rte_idxd_inter_dom.h>\n+\n+#include \"idxd_internal.h\"\n+\n+#define IDXD_TYPE ('d')\n+#define IDXD_IOC_BASE 100\n+#define IDXD_WIN_BASE 200\n+\n+enum idxd_win_type {\n+\tIDXD_WIN_TYPE_SA_SS = 0,\n+\tIDXD_WIN_TYPE_SA_MS,\n+};\n+\n+#define IDXD_WIN_FLAGS_MASK (RTE_IDXD_WIN_FLAGS_PROT_READ | RTE_IDXD_WIN_FLAGS_PROT_WRITE |\\\n+\t\tRTE_IDXD_WIN_FLAGS_WIN_CHECK | RTE_IDXD_WIN_FLAGS_OFFSET_MODE|\\\n+\t\tRTE_IDXD_WIN_FLAGS_TYPE_SAMS)\n+\n+struct idxd_win_param {\n+\tuint64_t base; /* Window base */\n+\tuint64_t size; /* Window size */\n+\tuint32_t type; /* Window type, see enum idxd_win_type */\n+\tuint16_t flags; /* See IDXD windows flags */\n+\tuint16_t handle; /* Window handle returned by driver */\n+} __attribute__((packed));\n+\n+struct idxd_win_attach {\n+\tuint32_t fd; /* Window file descriptor returned by IDXD_WIN_CREATE */\n+\tuint16_t handle; /* Window handle returned by driver */\n+} __attribute__((packed));\n+\n+struct idxd_win_fault {\n+\tuint64_t offset; /* Window offset of faulting address */\n+\tuint64_t len; /* Faulting range */\n+\tuint32_t write_fault; /* Fault generated on write */\n+} __attribute__((packed));\n+\n+#define IDXD_WIN_CREATE _IOWR(IDXD_TYPE, IDXD_IOC_BASE + 1, struct idxd_win_param)\n+#define IDXD_WIN_ATTACH _IOR(IDXD_TYPE, IDXD_IOC_BASE + 2, struct idxd_win_attach)\n+#define IDXD_WIN_FAULT _IOR(IDXD_TYPE, IDXD_WIN_BASE + 1, struct idxd_win_fault)\n+#define DSA_DEV_PATH \"/dev/dsa\"\n+\n+static inline const char *\n+dsa_get_dev_path(void)\n+{\n+\tconst char *path = getenv(\"DSA_DEV_PATH\");\n+\treturn path ? path : DSA_DEV_PATH;\n+}\n+\n+static int\n+dsa_find_work_queue(int controller_id)\n+{\n+\tchar dev_templ[PATH_MAX], path_templ[PATH_MAX];\n+\tconst char *path = dsa_get_dev_path();\n+\tstruct dirent *wq;\n+\tDIR *dev_dir;\n+\tint fd = -1;\n+\n+\t/* construct work queue path template */\n+\tsnprintf(dev_templ, sizeof(dev_templ), \"wq%d.\", controller_id);\n+\n+\t/* open the DSA device directory */\n+\tdev_dir = opendir(path);\n+\tif (dev_dir == NULL)\n+\t\treturn -1;\n+\n+\t/* find any available work queue */\n+\twhile ((wq = readdir(dev_dir)) != NULL) {\n+\t\t/* skip things that aren't work queues */\n+\t\tif (strncmp(wq->d_name, dev_templ, strlen(dev_templ)) != 0)\n+\t\t\tcontinue;\n+\n+\t\t/* try this work queue */\n+\t\tsnprintf(path_templ, sizeof(path_templ), \"%s/%s\", path, wq->d_name);\n+\n+\t\tfd = open(path_templ, O_RDWR);\n+\t\tif (fd < 0)\n+\t\t\tcontinue;\n+\n+\t\tbreak;\n+\t}\n+\n+\treturn fd;\n+}\n+\n+int\n+rte_idxd_window_create(int controller_id, void *win_addr,\n+\tunsigned int win_len, int flags)\n+{\n+\tstruct idxd_win_param param = {0};\n+\tint idpte_fd, fd;\n+\n+\tfd = dsa_find_work_queue(controller_id);\n+\n+\t/* did we find anything? */\n+\tif (fd < 0) {\n+\t\tIDXD_PMD_ERR(\"%s(): creatomg idpt window failed\", __func__);\n+\t\treturn -1;\n+\t}\n+\n+\t/* create a wormhole into a parallel reality... */\n+\tparam.base = (uint64_t)win_addr;\n+\tparam.size = win_len;\n+\tparam.flags = flags & IDXD_WIN_FLAGS_MASK;\n+\tparam.type = (flags & RTE_IDXD_WIN_FLAGS_TYPE_SAMS) ?\n+\t\tIDXD_WIN_TYPE_SA_MS : IDXD_WIN_TYPE_SA_SS;\n+\n+\tidpte_fd = ioctl(fd, IDXD_WIN_CREATE, ¶m);\n+\n+\tclose(fd);\n+\n+\tif (idpte_fd < 0)\n+\t\trte_errno = idpte_fd;\n+\n+\treturn idpte_fd;\n+}\n+\n+int\n+rte_idxd_window_attach(int controller_id, int idpte_fd,\n+\tuint16_t *handle)\n+{\n+\n+\tstruct idxd_win_attach win_attach = {0};\n+\tint ret, fd;\n+\n+\tif (handle == NULL) {\n+\t\trte_errno = EINVAL;\n+\t\treturn -1;\n+\t}\n+\n+\tfd = dsa_find_work_queue(controller_id);\n+\n+\t/* did we find anything? */\n+\tif (fd < 0) {\n+\t\tIDXD_PMD_ERR(\"%s(): creatomg idpt window failed\", __func__);\n+\t\trte_errno = ENOENT;\n+\t\treturn -1;\n+\t}\n+\n+\t/* get access to someone else's wormhole */\n+\twin_attach.fd = idpte_fd;\n+\n+\tret = ioctl(fd, IDXD_WIN_ATTACH, &win_attach);\n+\tif (ret != 0) {\n+\t\tIDXD_PMD_ERR(\"%s(): attaching idpt window failed: %s\",\n+\t\t\t\t__func__, strerror(ret));\n+\t\trte_errno = ret;\n+\t\treturn -1;\n+\t}\n+\n+\t*handle = win_attach.handle;\n+\n+\treturn 0;\n+}\ndiff --git a/drivers/dma/idxd/meson.build b/drivers/dma/idxd/meson.build\nindex c5403b431c..da73ab340c 100644\n--- a/drivers/dma/idxd/meson.build\n+++ b/drivers/dma/idxd/meson.build\n@@ -22,5 +22,10 @@ sources = files(\n )\n \n if is_linux\n- sources += files('idxd_bus.c')\n+ sources += files(\n+ 'idxd_bus.c',\n+ 'idxd_inter_dom.c',\n+)\n endif\n+\n+headers = files('rte_idxd_inter_dom.h')\ndiff --git a/drivers/dma/idxd/rte_idxd_inter_dom.h b/drivers/dma/idxd/rte_idxd_inter_dom.h\nnew file mode 100644\nindex 0000000000..c31f3777c9\n--- /dev/null\n+++ b/drivers/dma/idxd/rte_idxd_inter_dom.h\n@@ -0,0 +1,79 @@\n+/* SPDX-License-Identifier: BSD-3-Clause\n+ * Copyright(c) 2023 Intel Corporation\n+ */\n+\n+#ifndef _RTE_IDXD_INTER_DOM_H_\n+#define _RTE_IDXD_INTER_DOM_H_\n+\n+#ifdef __cplusplus\n+extern \"C\" {\n+#endif\n+\n+#include <stdint.h>\n+\n+#include <rte_compat.h>\n+\n+/** Allow reading from address space. */\n+#define RTE_IDXD_WIN_FLAGS_PROT_READ 0x0001\n+/** Allow writing to address space. */\n+#define RTE_IDXD_WIN_FLAGS_PROT_WRITE 0x0002\n+/** If this flag not set, the entire address space will be accessible. */\n+#define RTE_IDXD_WIN_FLAGS_WIN_CHECK 0x0004\n+/** Destination addresses are offsets from window base address. */\n+#define RTE_IDXD_WIN_FLAGS_OFFSET_MODE 0x0008\n+/* multiple submitter flag. If not set - single submitter type will be used. */\n+#define RTE_IDXD_WIN_FLAGS_TYPE_SAMS 0x0010\n+\n+/**\n+ * @warning\n+ * @b EXPERIMENTAL: this API may change without prior notice.\n+ *\n+ * Create an inter-pasid window to allow another process to access this process'\n+ * memory. This function returns a file descriptor for the window, that can be\n+ * used by another process to access this window.\n+ *\n+ * @param controller_id\n+ * IDXD controller device ID.\n+ * @param win_addr\n+ * Base address of memory chunk being shared (ignored if\n+ * `RTE_IDXD_WIN_FLAGS_WIN_CHECK` is not set).\n+ * @param win_len\n+ * Length of memory chunk being shared (ignored if\n+ * `RTE_IDXD_WIN_FLAGS_WIN_CHECK` is not set).\n+ * @param flags\n+ * Flags to configure the window.\n+ * @return\n+ * Non-negative on success.\n+ * Negative on error.\n+ */\n+__rte_experimental\n+int rte_idxd_window_create(int controller_id, void *win_addr,\n+\tunsigned int win_len, int flags);\n+\n+/**\n+ * @warning\n+ * @b EXPERIMENTAL: this API may change without prior notice.\n+ *\n+ * Attach to an inter-pasid window of another process. This function expects a\n+ * file descriptor returned by `rte_idxd_window_create()`, and will set the\n+ * value pointed to by `handle`. This handle can then be used to perform\n+ * inter-domain DMA operations.\n+ *\n+ * @param controller_id\n+ * IDXD controller device ID.\n+ * @param idpte_fd\n+ * File descriptor for another process's window\n+ * @param handle\n+ * Pointer to a variable to receive the handle.\n+ * @return\n+ * 0 on success.\n+ * Negative on error.\n+ */\n+__rte_experimental\n+int rte_idxd_window_attach(int controller_id, int idpte_fd, uint16_t *handle);\n+\n+#ifdef __cplusplus\n+}\n+#endif\n+\n+#endif /* _RTE_IDXD_INTER_DOM_H_ */\ndiff --git a/drivers/dma/idxd/version.map b/drivers/dma/idxd/version.map\nnew file mode 100644\nindex 0000000000..e091bb7c09\n--- /dev/null\n+++ b/drivers/dma/idxd/version.map\n@@ -0,0 +1,11 @@\n+DPDK_23 {\n+\tlocal: *;\n+};\n+\n+\n+EXPERIMENTAL {\n+\tglobal:\n+\n+\trte_idxd_window_create;\n+\trte_idxd_window_attach;\n+};\n", "prefixes": [ "v1", "3/3" ] }