Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/51668/?format=api
https://patches.dpdk.org/api/patches/51668/?format=api", "web_url": "https://patches.dpdk.org/project/dpdk/patch/20190325192238.20940-7-yskoh@mellanox.com/", "project": { "id": 1, "url": "https://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20190325192238.20940-7-yskoh@mellanox.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20190325192238.20940-7-yskoh@mellanox.com", "date": "2019-03-25T19:22:38", "name": "[v2,6/6] net/mlx4: enable secondary process to register DMA memory", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": true, "hash": "025080c19337e4e6478edc3812c5e7244226c368", "submitter": { "id": 636, "url": "https://patches.dpdk.org/api/people/636/?format=api", "name": "Yongseok Koh", "email": "yskoh@mellanox.com" }, "delegate": { "id": 6624, "url": "https://patches.dpdk.org/api/users/6624/?format=api", "username": "shahafs", "first_name": "Shahaf", "last_name": "Shuler", "email": "shahafs@mellanox.com" }, "mbox": "https://patches.dpdk.org/project/dpdk/patch/20190325192238.20940-7-yskoh@mellanox.com/mbox/", "series": [ { "id": 3907, "url": "https://patches.dpdk.org/api/series/3907/?format=api", "web_url": "https://patches.dpdk.org/project/dpdk/list/?series=3907", "date": "2019-03-25T19:22:33", "name": "net/mlx: enable secondary process to register DMA memory", "version": 2, "mbox": "https://patches.dpdk.org/series/3907/mbox/" } ], "comments": "https://patches.dpdk.org/api/patches/51668/comments/", "check": "fail", "checks": "https://patches.dpdk.org/api/patches/51668/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 8A94D5A44;\n\tMon, 25 Mar 2019 20:23:09 +0100 (CET)", "from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129])\n\tby dpdk.org (Postfix) with ESMTP id 1F6753772\n\tfor <dev@dpdk.org>; Mon, 25 Mar 2019 20:22:57 +0100 (CET)", "from Internal Mail-Server by MTLPINE1 (envelope-from\n\tyskoh@mellanox.com)\n\twith ESMTPS (AES256-SHA encrypted); 25 Mar 2019 21:22:52 +0200", "from scfae-sc-2.mti.labs.mlnx (scfae-sc-2.mti.labs.mlnx\n\t[10.101.0.96])\n\tby labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x2PJMfrJ027389;\n\tMon, 25 Mar 2019 21:22:51 +0200" ], "From": "Yongseok Koh <yskoh@mellanox.com>", "To": "shahafs@mellanox.com", "Cc": "dev@dpdk.org", "Date": "Mon, 25 Mar 2019 12:22:38 -0700", "Message-Id": "<20190325192238.20940-7-yskoh@mellanox.com>", "X-Mailer": "git-send-email 2.11.0", "In-Reply-To": "<20190325192238.20940-1-yskoh@mellanox.com>", "References": "<20190307074151.18815-1-yskoh@mellanox.com>\n\t<20190325192238.20940-1-yskoh@mellanox.com>", "Subject": "[dpdk-dev] [PATCH v2 6/6] net/mlx4: enable secondary process to\n\tregister DMA memory", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "The Memory Region (MR) for DMA memory can't be created from secondary\nprocess due to lib/driver limitation. Whenever it is needed, secondary\nprocess can make a request to primary process through the EAL IPC channel\n(rte_mp_msg) which is established on initialization. Once a MR is created\nby primary process, it is immediately visible to secondary process because\nthe MR list is global per a device. Thus, secondary process can look up the\nlist after the request is successfully returned.\n\nSigned-off-by: Yongseok Koh <yskoh@mellanox.com>\nAcked-by: Shahaf Shuler <shahafs@mellanox.com>\n---\n doc/guides/nics/mlx4.rst | 1 -\n drivers/net/mlx4/mlx4.h | 6 +++\n drivers/net/mlx4/mlx4_mp.c | 50 ++++++++++++++++++++++++\n drivers/net/mlx4/mlx4_mr.c | 95 ++++++++++++++++++++++++++++++++++++++++------\n drivers/net/mlx4/mlx4_mr.h | 2 +\n 5 files changed, 142 insertions(+), 12 deletions(-)", "diff": "diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst\nindex c8a02be4dd..aaf1907532 100644\n--- a/doc/guides/nics/mlx4.rst\n+++ b/doc/guides/nics/mlx4.rst\n@@ -159,7 +159,6 @@ Limitations\n - For secondary process:\n \n - Forked secondary process not supported.\n- - All mempools must be initialized before rte_eth_dev_start().\n - External memory unregistered in EAL memseg list cannot be used for DMA\n unless such memory has been registered by ``mlx4_mr_update_ext_mp()`` in\n primary process and remapped to the same virtual address in secondary\ndiff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h\nindex 5316c51247..3881943ef0 100644\n--- a/drivers/net/mlx4/mlx4.h\n+++ b/drivers/net/mlx4/mlx4.h\n@@ -79,6 +79,7 @@ enum {\n /* Request types for IPC. */\n enum mlx4_mp_req_type {\n \tMLX4_MP_REQ_VERBS_CMD_FD = 1,\n+\tMLX4_MP_REQ_CREATE_MR,\n \tMLX4_MP_REQ_START_RXTX,\n \tMLX4_MP_REQ_STOP_RXTX,\n };\n@@ -88,6 +89,10 @@ struct mlx4_mp_param {\n \tenum mlx4_mp_req_type type;\n \tint port_id;\n \tint result;\n+\tRTE_STD_C11\n+\tunion {\n+\t\tuintptr_t addr; /* MLX4_MP_REQ_CREATE_MR */\n+\t} args;\n };\n \n /** Request timeout for IPC. */\n@@ -234,6 +239,7 @@ int mlx4_rx_intr_enable(struct rte_eth_dev *dev, uint16_t idx);\n /* mlx4_mp.c */\n void mlx4_mp_req_start_rxtx(struct rte_eth_dev *dev);\n void mlx4_mp_req_stop_rxtx(struct rte_eth_dev *dev);\n+int mlx4_mp_req_mr_create(struct rte_eth_dev *dev, uintptr_t addr);\n int mlx4_mp_req_verbs_cmd_fd(struct rte_eth_dev *dev);\n void mlx4_mp_init_primary(void);\n void mlx4_mp_uninit_primary(void);\ndiff --git a/drivers/net/mlx4/mlx4_mp.c b/drivers/net/mlx4/mlx4_mp.c\nindex eaeb257348..183622453c 100644\n--- a/drivers/net/mlx4/mlx4_mp.c\n+++ b/drivers/net/mlx4/mlx4_mp.c\n@@ -58,6 +58,8 @@ mp_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)\n \t\t(const struct mlx4_mp_param *)mp_msg->param;\n \tstruct rte_eth_dev *dev;\n \tstruct mlx4_priv *priv;\n+\tstruct mlx4_mr_cache entry;\n+\tuint32_t lkey;\n \tint ret;\n \n \tassert(rte_eal_process_type() == RTE_PROC_PRIMARY);\n@@ -69,6 +71,13 @@ mp_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer)\n \tdev = &rte_eth_devices[param->port_id];\n \tpriv = dev->data->dev_private;\n \tswitch (param->type) {\n+\tcase MLX4_MP_REQ_CREATE_MR:\n+\t\tmp_init_msg(dev, &mp_res, param->type);\n+\t\tlkey = mlx4_mr_create_primary(dev, &entry, param->args.addr);\n+\t\tif (lkey == UINT32_MAX)\n+\t\t\tres->result = -rte_errno;\n+\t\tret = rte_mp_reply(&mp_res, peer);\n+\t\tbreak;\n \tcase MLX4_MP_REQ_VERBS_CMD_FD:\n \t\tmp_init_msg(dev, &mp_res, param->type);\n \t\tmp_res.num_fds = 1;\n@@ -218,6 +227,47 @@ mlx4_mp_req_stop_rxtx(struct rte_eth_dev *dev)\n }\n \n /**\n+ * Request Memory Region creation to the primary process.\n+ *\n+ * @param[in] dev\n+ * Pointer to Ethernet structure.\n+ * @param addr\n+ * Target virtual address to register.\n+ *\n+ * @return\n+ * 0 on success, a negative errno value otherwise and rte_errno is set.\n+ */\n+int\n+mlx4_mp_req_mr_create(struct rte_eth_dev *dev, uintptr_t addr)\n+{\n+\tstruct rte_mp_msg mp_req;\n+\tstruct rte_mp_msg *mp_res;\n+\tstruct rte_mp_reply mp_rep;\n+\tstruct mlx4_mp_param *req = (struct mlx4_mp_param *)mp_req.param;\n+\tstruct mlx4_mp_param *res;\n+\tstruct timespec ts = {.tv_sec = MLX4_MP_REQ_TIMEOUT_SEC, .tv_nsec = 0};\n+\tint ret;\n+\n+\tassert(rte_eal_process_type() == RTE_PROC_SECONDARY);\n+\tmp_init_msg(dev, &mp_req, MLX4_MP_REQ_CREATE_MR);\n+\treq->args.addr = addr;\n+\tret = rte_mp_request_sync(&mp_req, &mp_rep, &ts);\n+\tif (ret) {\n+\t\tERROR(\"port %u request to primary process failed\",\n+\t\t dev->data->port_id);\n+\t\treturn -rte_errno;\n+\t}\n+\tassert(mp_rep.nb_received == 1);\n+\tmp_res = &mp_rep.msgs[0];\n+\tres = (struct mlx4_mp_param *)mp_res->param;\n+\tret = res->result;\n+\tif (ret)\n+\t\trte_errno = -ret;\n+\tfree(mp_rep.msgs);\n+\treturn ret;\n+}\n+\n+/**\n * IPC message handler of primary process.\n *\n * @param[in] dev\ndiff --git a/drivers/net/mlx4/mlx4_mr.c b/drivers/net/mlx4/mlx4_mr.c\nindex 6db917a092..ad7d4832f2 100644\n--- a/drivers/net/mlx4/mlx4_mr.c\n+++ b/drivers/net/mlx4/mlx4_mr.c\n@@ -528,7 +528,10 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl,\n \n /**\n * Create a new global Memroy Region (MR) for a missing virtual address.\n- * Register entire virtually contiguous memory chunk around the address.\n+ * This API should be called on a secondary process, then a request is sent to\n+ * the primary process in order to create a MR for the address. As the global MR\n+ * list is on the shared memory, following LKey lookup should succeed unless the\n+ * request fails.\n *\n * @param dev\n * Pointer to Ethernet device.\n@@ -542,8 +545,52 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl,\n * Searched LKey on success, UINT32_MAX on failure and rte_errno is set.\n */\n static uint32_t\n-mlx4_mr_create(struct rte_eth_dev *dev, struct mlx4_mr_cache *entry,\n-\t uintptr_t addr)\n+mlx4_mr_create_secondary(struct rte_eth_dev *dev, struct mlx4_mr_cache *entry,\n+\t\t\t uintptr_t addr)\n+{\n+\tstruct mlx4_priv *priv = dev->data->dev_private;\n+\tint ret;\n+\n+\tDEBUG(\"port %u requesting MR creation for address (%p)\",\n+\t dev->data->port_id, (void *)addr);\n+\tret = mlx4_mp_req_mr_create(dev, addr);\n+\tif (ret) {\n+\t\tDEBUG(\"port %u fail to request MR creation for address (%p)\",\n+\t\t dev->data->port_id, (void *)addr);\n+\t\treturn UINT32_MAX;\n+\t}\n+\trte_rwlock_read_lock(&priv->mr.rwlock);\n+\t/* Fill in output data. */\n+\tmr_lookup_dev(dev, entry, addr);\n+\t/* Lookup can't fail. */\n+\tassert(entry->lkey != UINT32_MAX);\n+\trte_rwlock_read_unlock(&priv->mr.rwlock);\n+\tDEBUG(\"port %u MR CREATED by primary process for %p:\\n\"\n+\t \" [0x%\" PRIxPTR \", 0x%\" PRIxPTR \"), lkey=0x%x\",\n+\t dev->data->port_id, (void *)addr,\n+\t entry->start, entry->end, entry->lkey);\n+\treturn entry->lkey;\n+}\n+\n+/**\n+ * Create a new global Memroy Region (MR) for a missing virtual address.\n+ * Register entire virtually contiguous memory chunk around the address.\n+ * This must be called from the primary process.\n+ *\n+ * @param dev\n+ * Pointer to Ethernet device.\n+ * @param[out] entry\n+ * Pointer to returning MR cache entry, found in the global cache or newly\n+ * created. If failed to create one, this will not be updated.\n+ * @param addr\n+ * Target virtual address to register.\n+ *\n+ * @return\n+ * Searched LKey on success, UINT32_MAX on failure and rte_errno is set.\n+ */\n+uint32_t\n+mlx4_mr_create_primary(struct rte_eth_dev *dev, struct mlx4_mr_cache *entry,\n+\t\t uintptr_t addr)\n {\n \tstruct mlx4_priv *priv = dev->data->dev_private;\n \tstruct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;\n@@ -563,14 +610,6 @@ mlx4_mr_create(struct rte_eth_dev *dev, struct mlx4_mr_cache *entry,\n \n \tDEBUG(\"port %u creating a MR using address (%p)\",\n \t dev->data->port_id, (void *)addr);\n-\tif (rte_eal_process_type() != RTE_PROC_PRIMARY) {\n-\t\tWARN(\"port %u using address (%p) of unregistered mempool\"\n-\t\t \" in secondary process, please create mempool\"\n-\t\t \" before rte_eth_dev_start()\",\n-\t\t dev->data->port_id, (void *)addr);\n-\t\trte_errno = EPERM;\n-\t\tgoto err_nolock;\n-\t}\n \t/*\n \t * Release detached MRs if any. This can't be called with holding either\n \t * memory_hotplug_lock or priv->mr.rwlock. MRs on the free list have\n@@ -781,6 +820,40 @@ mlx4_mr_create(struct rte_eth_dev *dev, struct mlx4_mr_cache *entry,\n }\n \n /**\n+ * Create a new global Memroy Region (MR) for a missing virtual address.\n+ * This can be called from primary and secondary process.\n+ *\n+ * @param dev\n+ * Pointer to Ethernet device.\n+ * @param[out] entry\n+ * Pointer to returning MR cache entry, found in the global cache or newly\n+ * created. If failed to create one, this will not be updated.\n+ * @param addr\n+ * Target virtual address to register.\n+ *\n+ * @return\n+ * Searched LKey on success, UINT32_MAX on failure and rte_errno is set.\n+ */\n+static uint32_t\n+mlx4_mr_create(struct rte_eth_dev *dev, struct mlx4_mr_cache *entry,\n+\t uintptr_t addr)\n+{\n+\tuint32_t ret = 0;\n+\n+\tswitch (rte_eal_process_type()) {\n+\tcase RTE_PROC_PRIMARY:\n+\t\tret = mlx4_mr_create_primary(dev, entry, addr);\n+\t\tbreak;\n+\tcase RTE_PROC_SECONDARY:\n+\t\tret = mlx4_mr_create_secondary(dev, entry, addr);\n+\t\tbreak;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\treturn ret;\n+}\n+\n+/**\n * Rebuild the global B-tree cache of device from the original MR list.\n *\n * @param dev\ndiff --git a/drivers/net/mlx4/mlx4_mr.h b/drivers/net/mlx4/mlx4_mr.h\nindex 37a365a8b5..9d125e239d 100644\n--- a/drivers/net/mlx4/mlx4_mr.h\n+++ b/drivers/net/mlx4/mlx4_mr.h\n@@ -75,6 +75,8 @@ extern rte_rwlock_t mlx4_mem_event_rwlock;\n int mlx4_mr_btree_init(struct mlx4_mr_btree *bt, int n, int socket);\n void mlx4_mr_btree_free(struct mlx4_mr_btree *bt);\n void mlx4_mr_btree_dump(struct mlx4_mr_btree *bt);\n+uint32_t mlx4_mr_create_primary(struct rte_eth_dev *dev,\n+\t\t\t\tstruct mlx4_mr_cache *entry, uintptr_t addr);\n void mlx4_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr,\n \t\t\t size_t len, void *arg);\n int mlx4_mr_update_mp(struct rte_eth_dev *dev, struct mlx4_mr_ctrl *mr_ctrl,\n", "prefixes": [ "v2", "6/6" ] }{ "id": 51668, "url": "