Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/50907/?format=api
{ "id": 50907, "url": "http://patches.dpdk.org/api/patches/50907/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/patch/20190307074151.18815-5-yskoh@mellanox.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20190307074151.18815-5-yskoh@mellanox.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20190307074151.18815-5-yskoh@mellanox.com", "date": "2019-03-07T07:41:49", "name": "[4/6] net/mlx5: enable secondary process to register DMA memory", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": true, "hash": "f8ecddccc9b45478e4b3092ce3bc37af4a8bfb44", "submitter": { "id": 636, "url": "http://patches.dpdk.org/api/people/636/?format=api", "name": "Yongseok Koh", "email": "yskoh@mellanox.com" }, "delegate": { "id": 6624, "url": "http://patches.dpdk.org/api/users/6624/?format=api", "username": "shahafs", "first_name": "Shahaf", "last_name": "Shuler", "email": "shahafs@mellanox.com" }, "mbox": "http://patches.dpdk.org/project/dpdk/patch/20190307074151.18815-5-yskoh@mellanox.com/mbox/", "series": [ { "id": 3654, "url": "http://patches.dpdk.org/api/series/3654/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=3654", "date": "2019-03-07T07:41:45", "name": "net/mlx: enable secondary process to register DMA memory", "version": 1, "mbox": "http://patches.dpdk.org/series/3654/mbox/" } ], "comments": "http://patches.dpdk.org/api/patches/50907/comments/", "check": "fail", "checks": "http://patches.dpdk.org/api/patches/50907/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id F0C185398;\n\tThu, 7 Mar 2019 08:42:18 +0100 (CET)", "from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129])\n\tby dpdk.org (Postfix) with ESMTP id AFCF64F98\n\tfor <dev@dpdk.org>; Thu, 7 Mar 2019 08:42:15 +0100 (CET)", "from Internal Mail-Server by MTLPINE1 (envelope-from\n\tyskoh@mellanox.com)\n\twith ESMTPS (AES256-SHA encrypted); 7 Mar 2019 09:42:11 +0200", "from scfae-sc-2.mti.labs.mlnx (scfae-sc-2.mti.labs.mlnx\n\t[10.101.0.96])\n\tby labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x277g24M014151;\n\tThu, 7 Mar 2019 09:42:09 +0200" ], "From": "Yongseok Koh <yskoh@mellanox.com>", "To": "shahafs@mellanox.com", "Cc": "dev@dpdk.org", "Date": "Wed, 6 Mar 2019 23:41:49 -0800", "Message-Id": "<20190307074151.18815-5-yskoh@mellanox.com>", "X-Mailer": "git-send-email 2.11.0", "In-Reply-To": "<20190307074151.18815-1-yskoh@mellanox.com>", "References": "<20190307074151.18815-1-yskoh@mellanox.com>", "Subject": "[dpdk-dev] [PATCH 4/6] net/mlx5: enable secondary process to\n\tregister DMA memory", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "The Memory Region (MR) for DMA memory can't be created from secondary\nprocess due to lib/driver limitation. Whenever it is needed, secondary\nprocess can make a request to primary process through the EAL IPC channel\n(rte_mp_msg) which is established on initialization. Once a MR is created\nby primary process, it is immediately visible to secondary process because\nthe MR list is global per a device. Thus, secondary process can look up the\nlist after the request is successfully returned.\n\nSigned-off-by: Yongseok Koh <yskoh@mellanox.com>\n---\n doc/guides/nics/mlx5.rst | 1 -\n drivers/net/mlx5/mlx5.h | 6 +++\n drivers/net/mlx5/mlx5_mp.c | 51 ++++++++++++++++++++++++\n drivers/net/mlx5/mlx5_mr.c | 96 ++++++++++++++++++++++++++++++++++++++++------\n drivers/net/mlx5/mlx5_mr.h | 2 +\n 5 files changed, 143 insertions(+), 13 deletions(-)", "diff": "diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst\nindex 945dbc25f4..fb62daf6cc 100644\n--- a/doc/guides/nics/mlx5.rst\n+++ b/doc/guides/nics/mlx5.rst\n@@ -85,7 +85,6 @@ Limitations\n - For secondary process:\n \n - Forked secondary process not supported.\n- - All mempools must be initialized before rte_eth_dev_start().\n - External memory unregistered in EAL memseg list cannot be used for DMA\n unless such memory has been registered by ``mlx5_mr_update_ext_mp()`` in\n primary process and remapped to the same virtual address in secondary\ndiff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h\nindex 2763fb8124..80784c22e0 100644\n--- a/drivers/net/mlx5/mlx5.h\n+++ b/drivers/net/mlx5/mlx5.h\n@@ -59,6 +59,7 @@ enum {\n /* Request types for IPC. */\n enum mlx5_mp_req_type {\n \tMLX5_MP_REQ_VERBS_CMD_FD = 1,\n+\tMLX5_MP_REQ_CREATE_MR,\n \tMLX5_MP_REQ_START_RXTX,\n \tMLX5_MP_REQ_STOP_RXTX,\n };\n@@ -68,6 +69,10 @@ struct mlx5_mp_param {\n \tenum mlx5_mp_req_type type;\n \tint port_id;\n \tint result;\n+\tRTE_STD_C11\n+\tunion {\n+\t\tuintptr_t addr; /* MLX5_MP_REQ_CREATE_MR */\n+\t} args;\n };\n \n /** Key string for IPC. */\n@@ -431,6 +436,7 @@ void mlx5_flow_delete_drop_queue(struct rte_eth_dev *dev);\n /* mlx5_mp.c */\n void mlx5_mp_req_start_rxtx(struct rte_eth_dev *dev);\n void mlx5_mp_req_stop_rxtx(struct rte_eth_dev *dev);\n+int mlx5_mp_req_mr_create(struct rte_eth_dev *dev, uintptr_t addr);\n int mlx5_mp_req_verbs_cmd_fd(struct rte_eth_dev *dev);\n void mlx5_mp_init_primary(void);\n void mlx5_mp_uninit_primary(void);\ndiff --git a/drivers/net/mlx5/mlx5_mp.c b/drivers/net/mlx5/mlx5_mp.c\nindex 623dfb8097..a04b912de8 100644\n--- a/drivers/net/mlx5/mlx5_mp.c\n+++ b/drivers/net/mlx5/mlx5_mp.c\n@@ -58,10 +58,19 @@ mp_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)\n \t\t(const struct mlx5_mp_param *)mp_msg->param;\n \tstruct rte_eth_dev *dev = &rte_eth_devices[param->port_id];\n \tstruct mlx5_priv *priv = dev->data->dev_private;\n+\tstruct mlx5_mr_cache entry;\n+\tuint32_t lkey;\n \tint ret = 0;\n \n \tassert(rte_eal_process_type() == RTE_PROC_PRIMARY);\n \tswitch (param->type) {\n+\tcase MLX5_MP_REQ_CREATE_MR:\n+\t\tmp_init_msg(dev, &mp_res, param->type);\n+\t\tlkey = mlx5_mr_create_primary(dev, &entry, param->args.addr);\n+\t\tif (lkey == UINT32_MAX)\n+\t\t\tres->result = -rte_errno;\n+\t\tret = rte_mp_reply(&mp_res, peer);\n+\t\tbreak;\n \tcase MLX5_MP_REQ_VERBS_CMD_FD:\n \t\tmp_init_msg(dev, &mp_res, param->type);\n \t\tmp_res.num_fds = 1;\n@@ -202,6 +211,48 @@ mlx5_mp_req_stop_rxtx(struct rte_eth_dev *dev)\n }\n \n /**\n+ * Request Memory Region creation to the primary process.\n+ *\n+ * @param[in] dev\n+ * Pointer to Ethernet structure.\n+ * @param addr\n+ * Target virtual address to register.\n+ *\n+ * @return\n+ * 0 on success, a negative errno value otherwise and rte_errno is set.\n+ */\n+int\n+mlx5_mp_req_mr_create(struct rte_eth_dev *dev, uintptr_t addr)\n+{\n+\tstruct rte_mp_msg mp_req;\n+\tstruct rte_mp_msg *mp_res;\n+\tstruct rte_mp_reply mp_rep;\n+\tstruct mlx5_mp_param *req = (struct mlx5_mp_param *)mp_req.param;\n+\tstruct mlx5_mp_param *res;\n+\tstruct timespec ts = {.tv_sec = 5, .tv_nsec = 0};\n+\tint ret;\n+\n+\tassert(rte_eal_process_type() == RTE_PROC_SECONDARY);\n+\tmp_init_msg(dev, &mp_req, MLX5_MP_REQ_CREATE_MR);\n+\treq->args.addr = addr;\n+\tret = rte_mp_request_sync(&mp_req, &mp_rep, &ts);\n+\tif (ret) {\n+\t\tDRV_LOG(ERR,\n+\t\t\t\"port %u failed to get command FD from primary process\",\n+\t\t\tdev->data->port_id);\n+\t\treturn -rte_errno;\n+\t}\n+\tassert(mp_rep.nb_received == 1);\n+\tmp_res = &mp_rep.msgs[0];\n+\tres = (struct mlx5_mp_param *)mp_res->param;\n+\tret = res->result;\n+\tif (ret)\n+\t\trte_errno = -ret;\n+\tfree(mp_rep.msgs);\n+\treturn ret;\n+}\n+\n+/**\n * Request Verbs command file descriptor for mmap to the primary process.\n *\n * @param[in] dev\ndiff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c\nindex e9eda975ff..576a3c298b 100644\n--- a/drivers/net/mlx5/mlx5_mr.c\n+++ b/drivers/net/mlx5/mlx5_mr.c\n@@ -516,7 +516,10 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl,\n \n /**\n * Create a new global Memroy Region (MR) for a missing virtual address.\n- * Register entire virtually contiguous memory chunk around the address.\n+ * This API should be called on a secondary process, then a request is sent to\n+ * the primary process in order to create a MR for the address. As the global MR\n+ * list is on the shared memory, following LKey lookup should succeed unless the\n+ * request fails.\n *\n * @param dev\n * Pointer to Ethernet device.\n@@ -530,8 +533,52 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl,\n * Searched LKey on success, UINT32_MAX on failure and rte_errno is set.\n */\n static uint32_t\n-mlx5_mr_create(struct rte_eth_dev *dev, struct mlx5_mr_cache *entry,\n-\t uintptr_t addr)\n+mlx5_mr_create_secondary(struct rte_eth_dev *dev, struct mlx5_mr_cache *entry,\n+\t\t\t uintptr_t addr)\n+{\n+\tstruct mlx5_priv *priv = dev->data->dev_private;\n+\tint ret;\n+\n+\tDEBUG(\"port %u requesting MR creation for address (%p)\",\n+\t dev->data->port_id, (void *)addr);\n+\tret = mlx5_mp_req_mr_create(dev, addr);\n+\tif (ret) {\n+\t\tDEBUG(\"port %u fail to request MR creation for address (%p)\",\n+\t\t dev->data->port_id, (void *)addr);\n+\t\treturn UINT32_MAX;\n+\t}\n+\trte_rwlock_read_lock(&priv->mr.rwlock);\n+\t/* Fill in output data. */\n+\tmr_lookup_dev(dev, entry, addr);\n+\t/* Lookup can't fail. */\n+\tassert(entry->lkey != UINT32_MAX);\n+\trte_rwlock_read_unlock(&priv->mr.rwlock);\n+\tDEBUG(\"port %u MR CREATED by primary process for %p:\\n\"\n+\t \" [0x%\" PRIxPTR \", 0x%\" PRIxPTR \"), lkey=0x%x\",\n+\t dev->data->port_id, (void *)addr,\n+\t entry->start, entry->end, entry->lkey);\n+\treturn entry->lkey;\n+}\n+\n+/**\n+ * Create a new global Memroy Region (MR) for a missing virtual address.\n+ * Register entire virtually contiguous memory chunk around the address.\n+ * This must be called from the primary process.\n+ *\n+ * @param dev\n+ * Pointer to Ethernet device.\n+ * @param[out] entry\n+ * Pointer to returning MR cache entry, found in the global cache or newly\n+ * created. If failed to create one, this will not be updated.\n+ * @param addr\n+ * Target virtual address to register.\n+ *\n+ * @return\n+ * Searched LKey on success, UINT32_MAX on failure and rte_errno is set.\n+ */\n+uint32_t\n+mlx5_mr_create_primary(struct rte_eth_dev *dev, struct mlx5_mr_cache *entry,\n+\t\t uintptr_t addr)\n {\n \tstruct mlx5_priv *priv = dev->data->dev_private;\n \tstruct mlx5_dev_config *config = &priv->config;\n@@ -552,15 +599,6 @@ mlx5_mr_create(struct rte_eth_dev *dev, struct mlx5_mr_cache *entry,\n \n \tDRV_LOG(DEBUG, \"port %u creating a MR using address (%p)\",\n \t\tdev->data->port_id, (void *)addr);\n-\tif (rte_eal_process_type() != RTE_PROC_PRIMARY) {\n-\t\tDRV_LOG(WARNING,\n-\t\t\t\"port %u using address (%p) of unregistered mempool\"\n-\t\t\t\" in secondary process, please create mempool\"\n-\t\t\t\" before rte_eth_dev_start()\",\n-\t\t\tdev->data->port_id, (void *)addr);\n-\t\trte_errno = EPERM;\n-\t\tgoto err_nolock;\n-\t}\n \t/*\n \t * Release detached MRs if any. This can't be called with holding either\n \t * memory_hotplug_lock or priv->mr.rwlock. MRs on the free list have\n@@ -772,6 +810,40 @@ mlx5_mr_create(struct rte_eth_dev *dev, struct mlx5_mr_cache *entry,\n }\n \n /**\n+ * Create a new global Memroy Region (MR) for a missing virtual address.\n+ * This can be called from primary and secondary process.\n+ *\n+ * @param dev\n+ * Pointer to Ethernet device.\n+ * @param[out] entry\n+ * Pointer to returning MR cache entry, found in the global cache or newly\n+ * created. If failed to create one, this will not be updated.\n+ * @param addr\n+ * Target virtual address to register.\n+ *\n+ * @return\n+ * Searched LKey on success, UINT32_MAX on failure and rte_errno is set.\n+ */\n+static uint32_t\n+mlx5_mr_create(struct rte_eth_dev *dev, struct mlx5_mr_cache *entry,\n+\t uintptr_t addr)\n+{\n+\tuint32_t ret = 0;\n+\n+\tswitch (rte_eal_process_type()) {\n+\tcase RTE_PROC_PRIMARY:\n+\t\tret = mlx5_mr_create_primary(dev, entry, addr);\n+\t\tbreak;\n+\tcase RTE_PROC_SECONDARY:\n+\t\tret = mlx5_mr_create_secondary(dev, entry, addr);\n+\t\tbreak;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\treturn ret;\n+}\n+\n+/**\n * Rebuild the global B-tree cache of device from the original MR list.\n *\n * @param dev\ndiff --git a/drivers/net/mlx5/mlx5_mr.h b/drivers/net/mlx5/mlx5_mr.h\nindex a57003fe92..786f6a3148 100644\n--- a/drivers/net/mlx5/mlx5_mr.h\n+++ b/drivers/net/mlx5/mlx5_mr.h\n@@ -70,6 +70,8 @@ extern rte_rwlock_t mlx5_mem_event_rwlock;\n \n int mlx5_mr_btree_init(struct mlx5_mr_btree *bt, int n, int socket);\n void mlx5_mr_btree_free(struct mlx5_mr_btree *bt);\n+uint32_t mlx5_mr_create_primary(struct rte_eth_dev *dev,\n+\t\t\t\tstruct mlx5_mr_cache *entry, uintptr_t addr);\n void mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,\n \t\t\t size_t len, void *arg);\n int mlx5_mr_update_mp(struct rte_eth_dev *dev, struct mlx5_mr_ctrl *mr_ctrl,\n", "prefixes": [ "4/6" ] }