Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/115675/?format=api
{ "id": 115675, "url": "https://patches.dpdk.org/api/patches/115675/?format=api", "web_url": "https://patches.dpdk.org/project/dpdk/patch/1661899911-13086-11-git-send-email-longli@linuxonhyperv.com/", "project": { "id": 1, "url": "https://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<1661899911-13086-11-git-send-email-longli@linuxonhyperv.com>", "list_archive_url": "https://inbox.dpdk.org/dev/1661899911-13086-11-git-send-email-longli@linuxonhyperv.com", "date": "2022-08-30T22:51:43", "name": "[v6,10/18] net/mana: implement memory registration", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": true, "hash": "d8963e79533cdc3c7158fc16a7ceb48ba47523cf", "submitter": { "id": 1784, "url": "https://patches.dpdk.org/api/people/1784/?format=api", "name": "Long Li", "email": "longli@linuxonhyperv.com" }, "delegate": { "id": 319, "url": "https://patches.dpdk.org/api/users/319/?format=api", "username": "fyigit", "first_name": "Ferruh", "last_name": "Yigit", "email": "ferruh.yigit@amd.com" }, "mbox": "https://patches.dpdk.org/project/dpdk/patch/1661899911-13086-11-git-send-email-longli@linuxonhyperv.com/mbox/", "series": [ { "id": 24471, "url": "https://patches.dpdk.org/api/series/24471/?format=api", "web_url": "https://patches.dpdk.org/project/dpdk/list/?series=24471", "date": "2022-08-30T22:51:33", "name": "Introduce Microsoft Azure Network Adatper (MANA) PMD", "version": 6, "mbox": "https://patches.dpdk.org/series/24471/mbox/" } ], "comments": "https://patches.dpdk.org/api/patches/115675/comments/", "check": "warning", "checks": "https://patches.dpdk.org/api/patches/115675/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from mails.dpdk.org (mails.dpdk.org [217.70.189.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id D0B51A00C5;\n\tWed, 31 Aug 2022 00:52:57 +0200 (CEST)", "from [217.70.189.124] (localhost [127.0.0.1])\n\tby mails.dpdk.org (Postfix) with ESMTP id 0260742B88;\n\tWed, 31 Aug 2022 00:52:07 +0200 (CEST)", "from linux.microsoft.com (linux.microsoft.com [13.77.154.182])\n by mails.dpdk.org (Postfix) with ESMTP id 2E4B942B76\n for <dev@dpdk.org>; Wed, 31 Aug 2022 00:52:03 +0200 (CEST)", "by linux.microsoft.com (Postfix, from userid 1004)\n id DEF8F2045E1E; Tue, 30 Aug 2022 15:52:02 -0700 (PDT)" ], "DKIM-Filter": "OpenDKIM Filter v2.11.0 linux.microsoft.com DEF8F2045E1E", "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxonhyperv.com;\n s=default; t=1661899922;\n bh=5CZgS94awAdS0/jB2Q+PA+nRK1DmxR/tuGWO/j8ZOr4=;\n h=From:To:Cc:Subject:Date:In-Reply-To:References:Reply-To:From;\n b=dbZiZXs7EpT+tWM0Gv6yZ4tRxx5OjTjvXbsmPkk/UB9HFTqKoCXXdVEHKKsuWMPZ2\n DgdZTgaCFHx0Y9a6g07W+YiIS7eTbkiZTlnOPRWaI6KTJWJQyIWIA8GgVtBqtPc6Ip\n CDCULBDZ1vaMBjGVkFb4qYXs9sEksDBwFcW7hF2Q=", "From": "longli@linuxonhyperv.com", "To": "Ferruh Yigit <ferruh.yigit@xilinx.com>", "Cc": "dev@dpdk.org, Ajay Sharma <sharmaajay@microsoft.com>,\n Stephen Hemminger <sthemmin@microsoft.com>, Long Li <longli@microsoft.com>", "Subject": "[Patch v6 10/18] net/mana: implement memory registration", "Date": "Tue, 30 Aug 2022 15:51:43 -0700", "Message-Id": "<1661899911-13086-11-git-send-email-longli@linuxonhyperv.com>", "X-Mailer": "git-send-email 1.8.3.1", "In-Reply-To": "<1661899911-13086-1-git-send-email-longli@linuxonhyperv.com>", "References": "<1661899911-13086-1-git-send-email-longli@linuxonhyperv.com>", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Reply-To": "longli@microsoft.com", "Errors-To": "dev-bounces@dpdk.org" }, "content": "From: Long Li <longli@microsoft.com>\n\nMANA hardware has iommu built-in, that provides hardware safe access to\nuser memory through memory registration. Since memory registration is an\nexpensive operation, this patch implements a two level memory registration\ncache mechanisum for each queue and for each port.\n\nSigned-off-by: Long Li <longli@microsoft.com>\n---\nChange log:\nv2:\nChange all header file functions to start with mana_.\nUse spinlock in place of rwlock to memory cache access.\nRemove unused header files.\nv4:\nRemove extra \"\\n\" in logging function.\n\n drivers/net/mana/mana.c | 20 +++\n drivers/net/mana/mana.h | 39 +++++\n drivers/net/mana/meson.build | 1 +\n drivers/net/mana/mp.c | 85 +++++++++\n drivers/net/mana/mr.c | 324 +++++++++++++++++++++++++++++++++++\n 5 files changed, 469 insertions(+)\n create mode 100644 drivers/net/mana/mr.c", "diff": "diff --git a/drivers/net/mana/mana.c b/drivers/net/mana/mana.c\nindex 490686f404..d18cc4ab0e 100644\n--- a/drivers/net/mana/mana.c\n+++ b/drivers/net/mana/mana.c\n@@ -103,6 +103,8 @@ mana_dev_close(struct rte_eth_dev *dev)\n \tstruct mana_priv *priv = dev->data->dev_private;\n \tint ret;\n \n+\tmana_remove_all_mr(priv);\n+\n \tret = mana_intr_uninstall(priv);\n \tif (ret)\n \t\treturn ret;\n@@ -317,6 +319,13 @@ static int mana_dev_tx_queue_setup(struct rte_eth_dev *dev,\n \t\tgoto fail;\n \t}\n \n+\tret = mana_mr_btree_init(&txq->mr_btree,\n+\t\t\t\t MANA_MR_BTREE_PER_QUEUE_N, socket_id);\n+\tif (ret) {\n+\t\tDRV_LOG(ERR, \"Failed to init TXQ MR btree\");\n+\t\tgoto fail;\n+\t}\n+\n \tDRV_LOG(DEBUG, \"idx %u nb_desc %u socket %u txq->desc_ring %p\",\n \t\tqueue_idx, nb_desc, socket_id, txq->desc_ring);\n \n@@ -338,6 +347,8 @@ static void mana_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid)\n {\n \tstruct mana_txq *txq = dev->data->tx_queues[qid];\n \n+\tmana_mr_btree_free(&txq->mr_btree);\n+\n \trte_free(txq->desc_ring);\n \trte_free(txq);\n }\n@@ -374,6 +385,13 @@ static int mana_dev_rx_queue_setup(struct rte_eth_dev *dev,\n \t\tgoto fail;\n \t}\n \n+\tret = mana_mr_btree_init(&rxq->mr_btree,\n+\t\t\t\t MANA_MR_BTREE_PER_QUEUE_N, socket_id);\n+\tif (ret) {\n+\t\tDRV_LOG(ERR, \"Failed to init RXQ MR btree\");\n+\t\tgoto fail;\n+\t}\n+\n \trxq->num_desc = nb_desc;\n \n \trxq->priv = priv;\n@@ -393,6 +411,8 @@ static void mana_dev_rx_queue_release(struct rte_eth_dev *dev, uint16_t qid)\n {\n \tstruct mana_rxq *rxq = dev->data->rx_queues[qid];\n \n+\tmana_mr_btree_free(&rxq->mr_btree);\n+\n \trte_free(rxq->desc_ring);\n \trte_free(rxq);\n }\ndiff --git a/drivers/net/mana/mana.h b/drivers/net/mana/mana.h\nindex 33f68b3d1b..9e15b43275 100644\n--- a/drivers/net/mana/mana.h\n+++ b/drivers/net/mana/mana.h\n@@ -50,6 +50,22 @@ struct mana_shared_data {\n #define MAX_RECEIVE_BUFFERS_PER_QUEUE\t256\n #define MAX_SEND_BUFFERS_PER_QUEUE\t256\n \n+struct mana_mr_cache {\n+\tuint32_t\tlkey;\n+\tuintptr_t\taddr;\n+\tsize_t\t\tlen;\n+\tvoid\t\t*verb_obj;\n+};\n+\n+#define MANA_MR_BTREE_CACHE_N\t512\n+struct mana_mr_btree {\n+\tuint16_t\tlen;\t/* Used entries */\n+\tuint16_t\tsize;\t/* Total entries */\n+\tint\t\toverflow;\n+\tint\t\tsocket;\n+\tstruct mana_mr_cache *table;\n+};\n+\n struct mana_process_priv {\n \tvoid *db_page;\n };\n@@ -82,6 +98,8 @@ struct mana_priv {\n \tint max_recv_sge;\n \tint max_mr;\n \tuint64_t max_mr_size;\n+\tstruct mana_mr_btree mr_btree;\n+\trte_spinlock_t\tmr_btree_lock;\n };\n \n struct mana_txq_desc {\n@@ -131,6 +149,7 @@ struct mana_txq {\n \tuint32_t desc_ring_head, desc_ring_tail;\n \n \tstruct mana_stats stats;\n+\tstruct mana_mr_btree mr_btree;\n \tunsigned int socket;\n };\n \n@@ -153,6 +172,7 @@ struct mana_rxq {\n \tstruct mana_gdma_queue gdma_cq;\n \n \tstruct mana_stats stats;\n+\tstruct mana_mr_btree mr_btree;\n \n \tunsigned int socket;\n };\n@@ -176,6 +196,24 @@ uint16_t mana_rx_burst_removed(void *dpdk_rxq, struct rte_mbuf **pkts,\n uint16_t mana_tx_burst_removed(void *dpdk_rxq, struct rte_mbuf **pkts,\n \t\t\t uint16_t pkts_n);\n \n+struct mana_mr_cache *mana_find_pmd_mr(struct mana_mr_btree *local_tree,\n+\t\t\t\t struct mana_priv *priv,\n+\t\t\t\t struct rte_mbuf *mbuf);\n+int mana_new_pmd_mr(struct mana_mr_btree *local_tree, struct mana_priv *priv,\n+\t\t struct rte_mempool *pool);\n+void mana_remove_all_mr(struct mana_priv *priv);\n+void mana_del_pmd_mr(struct mana_mr_cache *mr);\n+\n+void mana_mempool_chunk_cb(struct rte_mempool *mp, void *opaque,\n+\t\t\t struct rte_mempool_memhdr *memhdr, unsigned int idx);\n+\n+struct mana_mr_cache *mana_mr_btree_lookup(struct mana_mr_btree *bt,\n+\t\t\t\t\t uint16_t *idx,\n+\t\t\t\t\t uintptr_t addr, size_t len);\n+int mana_mr_btree_insert(struct mana_mr_btree *bt, struct mana_mr_cache *entry);\n+int mana_mr_btree_init(struct mana_mr_btree *bt, int n, int socket);\n+void mana_mr_btree_free(struct mana_mr_btree *bt);\n+\n /** Request timeout for IPC. */\n #define MANA_MP_REQ_TIMEOUT_SEC 5\n \n@@ -204,6 +242,7 @@ int mana_mp_init_secondary(void);\n void mana_mp_uninit_primary(void);\n void mana_mp_uninit_secondary(void);\n int mana_mp_req_verbs_cmd_fd(struct rte_eth_dev *dev);\n+int mana_mp_req_mr_create(struct mana_priv *priv, uintptr_t addr, uint32_t len);\n \n void mana_mp_req_on_rxtx(struct rte_eth_dev *dev, enum mana_mp_req_type type);\n \ndiff --git a/drivers/net/mana/meson.build b/drivers/net/mana/meson.build\nindex 81c4118f53..9771394370 100644\n--- a/drivers/net/mana/meson.build\n+++ b/drivers/net/mana/meson.build\n@@ -11,6 +11,7 @@ deps += ['pci', 'bus_pci', 'net', 'eal', 'kvargs']\n \n sources += files(\n \t'mana.c',\n+\t'mr.c',\n \t'mp.c',\n )\n \ndiff --git a/drivers/net/mana/mp.c b/drivers/net/mana/mp.c\nindex d7580e8a28..f4f78d2787 100644\n--- a/drivers/net/mana/mp.c\n+++ b/drivers/net/mana/mp.c\n@@ -12,6 +12,52 @@\n \n extern struct mana_shared_data *mana_shared_data;\n \n+static int mana_mp_mr_create(struct mana_priv *priv, uintptr_t addr,\n+\t\t\t uint32_t len)\n+{\n+\tstruct ibv_mr *ibv_mr;\n+\tint ret;\n+\tstruct mana_mr_cache *mr;\n+\n+\tibv_mr = ibv_reg_mr(priv->ib_pd, (void *)addr, len,\n+\t\t\t IBV_ACCESS_LOCAL_WRITE);\n+\n+\tif (!ibv_mr)\n+\t\treturn -errno;\n+\n+\tDRV_LOG(DEBUG, \"MR (2nd) lkey %u addr %p len %zu\",\n+\t\tibv_mr->lkey, ibv_mr->addr, ibv_mr->length);\n+\n+\tmr = rte_calloc(\"MANA MR\", 1, sizeof(*mr), 0);\n+\tif (!mr) {\n+\t\tDRV_LOG(ERR, \"(2nd) Failed to allocate MR\");\n+\t\tret = -ENOMEM;\n+\t\tgoto fail_alloc;\n+\t}\n+\tmr->lkey = ibv_mr->lkey;\n+\tmr->addr = (uintptr_t)ibv_mr->addr;\n+\tmr->len = ibv_mr->length;\n+\tmr->verb_obj = ibv_mr;\n+\n+\trte_spinlock_lock(&priv->mr_btree_lock);\n+\tret = mana_mr_btree_insert(&priv->mr_btree, mr);\n+\trte_spinlock_unlock(&priv->mr_btree_lock);\n+\tif (ret) {\n+\t\tDRV_LOG(ERR, \"(2nd) Failed to add to global MR btree\");\n+\t\tgoto fail_btree;\n+\t}\n+\n+\treturn 0;\n+\n+fail_btree:\n+\trte_free(mr);\n+\n+fail_alloc:\n+\tibv_dereg_mr(ibv_mr);\n+\n+\treturn ret;\n+}\n+\n static void mp_init_msg(struct rte_mp_msg *msg, enum mana_mp_req_type type,\n \t\t\tint port_id)\n {\n@@ -47,6 +93,12 @@ static int mana_mp_primary_handle(const struct rte_mp_msg *mp_msg,\n \tmp_init_msg(&mp_res, param->type, param->port_id);\n \n \tswitch (param->type) {\n+\tcase MANA_MP_REQ_CREATE_MR:\n+\t\tret = mana_mp_mr_create(priv, param->addr, param->len);\n+\t\tres->result = ret;\n+\t\tret = rte_mp_reply(&mp_res, peer);\n+\t\tbreak;\n+\n \tcase MANA_MP_REQ_VERBS_CMD_FD:\n \t\tmp_res.num_fds = 1;\n \t\tmp_res.fds[0] = priv->ib_ctx->cmd_fd;\n@@ -189,6 +241,39 @@ int mana_mp_req_verbs_cmd_fd(struct rte_eth_dev *dev)\n \treturn ret;\n }\n \n+int mana_mp_req_mr_create(struct mana_priv *priv, uintptr_t addr, uint32_t len)\n+{\n+\tstruct rte_mp_msg mp_req = { 0 };\n+\tstruct rte_mp_msg *mp_res;\n+\tstruct rte_mp_reply mp_rep;\n+\tstruct mana_mp_param *req = (struct mana_mp_param *)mp_req.param;\n+\tstruct mana_mp_param *res;\n+\tstruct timespec ts = {.tv_sec = MANA_MP_REQ_TIMEOUT_SEC, .tv_nsec = 0};\n+\tint ret;\n+\n+\tmp_init_msg(&mp_req, MANA_MP_REQ_CREATE_MR, priv->port_id);\n+\treq->addr = addr;\n+\treq->len = len;\n+\n+\tret = rte_mp_request_sync(&mp_req, &mp_rep, &ts);\n+\tif (ret) {\n+\t\tDRV_LOG(ERR, \"Port %u request to primary failed\",\n+\t\t\treq->port_id);\n+\t\treturn ret;\n+\t}\n+\n+\tif (mp_rep.nb_received != 1)\n+\t\treturn -EPROTO;\n+\n+\tmp_res = &mp_rep.msgs[0];\n+\tres = (struct mana_mp_param *)mp_res->param;\n+\tret = res->result;\n+\n+\tfree(mp_rep.msgs);\n+\n+\treturn ret;\n+}\n+\n void mana_mp_req_on_rxtx(struct rte_eth_dev *dev, enum mana_mp_req_type type)\n {\n \tstruct rte_mp_msg mp_req = { 0 };\ndiff --git a/drivers/net/mana/mr.c b/drivers/net/mana/mr.c\nnew file mode 100644\nindex 0000000000..81b64b840f\n--- /dev/null\n+++ b/drivers/net/mana/mr.c\n@@ -0,0 +1,324 @@\n+/* SPDX-License-Identifier: BSD-3-Clause\n+ * Copyright 2022 Microsoft Corporation\n+ */\n+\n+#include <rte_malloc.h>\n+#include <ethdev_driver.h>\n+#include <rte_eal_paging.h>\n+\n+#include <infiniband/verbs.h>\n+\n+#include \"mana.h\"\n+\n+struct mana_range {\n+\tuintptr_t\tstart;\n+\tuintptr_t\tend;\n+\tuint32_t\tlen;\n+};\n+\n+void mana_mempool_chunk_cb(struct rte_mempool *mp __rte_unused, void *opaque,\n+\t\t\t struct rte_mempool_memhdr *memhdr, unsigned int idx)\n+{\n+\tstruct mana_range *ranges = opaque;\n+\tstruct mana_range *range = &ranges[idx];\n+\tuint64_t page_size = rte_mem_page_size();\n+\n+\trange->start = RTE_ALIGN_FLOOR((uintptr_t)memhdr->addr, page_size);\n+\trange->end = RTE_ALIGN_CEIL((uintptr_t)memhdr->addr + memhdr->len,\n+\t\t\t\t page_size);\n+\trange->len = range->end - range->start;\n+}\n+\n+int mana_new_pmd_mr(struct mana_mr_btree *local_tree, struct mana_priv *priv,\n+\t\t struct rte_mempool *pool)\n+{\n+\tstruct ibv_mr *ibv_mr;\n+\tstruct mana_range ranges[pool->nb_mem_chunks];\n+\tuint32_t i;\n+\tstruct mana_mr_cache *mr;\n+\tint ret;\n+\n+\trte_mempool_mem_iter(pool, mana_mempool_chunk_cb, ranges);\n+\n+\tfor (i = 0; i < pool->nb_mem_chunks; i++) {\n+\t\tif (ranges[i].len > priv->max_mr_size) {\n+\t\t\tDRV_LOG(ERR, \"memory chunk size %u exceeding max MR\",\n+\t\t\t\tranges[i].len);\n+\t\t\treturn -ENOMEM;\n+\t\t}\n+\n+\t\tDRV_LOG(DEBUG,\n+\t\t\t\"registering memory chunk start 0x%\" PRIx64 \" len %u\",\n+\t\t\tranges[i].start, ranges[i].len);\n+\n+\t\tif (rte_eal_process_type() == RTE_PROC_SECONDARY) {\n+\t\t\t/* Send a message to the primary to do MR */\n+\t\t\tret = mana_mp_req_mr_create(priv, ranges[i].start,\n+\t\t\t\t\t\t ranges[i].len);\n+\t\t\tif (ret) {\n+\t\t\t\tDRV_LOG(ERR,\n+\t\t\t\t\t\"MR failed start 0x%\" PRIx64 \" len %u\",\n+\t\t\t\t\tranges[i].start, ranges[i].len);\n+\t\t\t\treturn ret;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tibv_mr = ibv_reg_mr(priv->ib_pd, (void *)ranges[i].start,\n+\t\t\t\t ranges[i].len, IBV_ACCESS_LOCAL_WRITE);\n+\t\tif (ibv_mr) {\n+\t\t\tDRV_LOG(DEBUG, \"MR lkey %u addr %p len %\" PRIu64,\n+\t\t\t\tibv_mr->lkey, ibv_mr->addr, ibv_mr->length);\n+\n+\t\t\tmr = rte_calloc(\"MANA MR\", 1, sizeof(*mr), 0);\n+\t\t\tmr->lkey = ibv_mr->lkey;\n+\t\t\tmr->addr = (uintptr_t)ibv_mr->addr;\n+\t\t\tmr->len = ibv_mr->length;\n+\t\t\tmr->verb_obj = ibv_mr;\n+\n+\t\t\trte_spinlock_lock(&priv->mr_btree_lock);\n+\t\t\tret = mana_mr_btree_insert(&priv->mr_btree, mr);\n+\t\t\trte_spinlock_unlock(&priv->mr_btree_lock);\n+\t\t\tif (ret) {\n+\t\t\t\tibv_dereg_mr(ibv_mr);\n+\t\t\t\tDRV_LOG(ERR, \"Failed to add to global MR btree\");\n+\t\t\t\treturn ret;\n+\t\t\t}\n+\n+\t\t\tret = mana_mr_btree_insert(local_tree, mr);\n+\t\t\tif (ret) {\n+\t\t\t\t/* Don't need to clean up MR as it's already\n+\t\t\t\t * in the global tree\n+\t\t\t\t */\n+\t\t\t\tDRV_LOG(ERR, \"Failed to add to local MR btree\");\n+\t\t\t\treturn ret;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tDRV_LOG(ERR, \"MR failed at 0x%\" PRIx64 \" len %u\",\n+\t\t\t\tranges[i].start, ranges[i].len);\n+\t\t\treturn -errno;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+void mana_del_pmd_mr(struct mana_mr_cache *mr)\n+{\n+\tint ret;\n+\tstruct ibv_mr *ibv_mr = (struct ibv_mr *)mr->verb_obj;\n+\n+\tret = ibv_dereg_mr(ibv_mr);\n+\tif (ret)\n+\t\tDRV_LOG(ERR, \"dereg MR failed ret %d\", ret);\n+}\n+\n+struct mana_mr_cache *mana_find_pmd_mr(struct mana_mr_btree *local_mr_btree,\n+\t\t\t\t struct mana_priv *priv,\n+\t\t\t\t struct rte_mbuf *mbuf)\n+{\n+\tstruct rte_mempool *pool = mbuf->pool;\n+\tint ret, second_try = 0;\n+\tstruct mana_mr_cache *mr;\n+\tuint16_t idx;\n+\n+\tDRV_LOG(DEBUG, \"finding mr for mbuf addr %p len %d\",\n+\t\tmbuf->buf_addr, mbuf->buf_len);\n+\n+try_again:\n+\t/* First try to find the MR in local queue tree */\n+\tmr = mana_mr_btree_lookup(local_mr_btree, &idx,\n+\t\t\t\t (uintptr_t)mbuf->buf_addr, mbuf->buf_len);\n+\tif (mr) {\n+\t\tDRV_LOG(DEBUG,\n+\t\t\t\"Local mr lkey %u addr 0x%\" PRIx64 \" len %\" PRIu64,\n+\t\t\tmr->lkey, mr->addr, mr->len);\n+\t\treturn mr;\n+\t}\n+\n+\t/* If not found, try to find the MR in global tree */\n+\trte_spinlock_lock(&priv->mr_btree_lock);\n+\tmr = mana_mr_btree_lookup(&priv->mr_btree, &idx,\n+\t\t\t\t (uintptr_t)mbuf->buf_addr,\n+\t\t\t\t mbuf->buf_len);\n+\trte_spinlock_unlock(&priv->mr_btree_lock);\n+\n+\t/* If found in the global tree, add it to the local tree */\n+\tif (mr) {\n+\t\tret = mana_mr_btree_insert(local_mr_btree, mr);\n+\t\tif (ret) {\n+\t\t\tDRV_LOG(DEBUG, \"Failed to add MR to local tree.\");\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\tDRV_LOG(DEBUG,\n+\t\t\t\"Added local MR key %u addr 0x%\" PRIx64 \" len %\" PRIu64,\n+\t\t\tmr->lkey, mr->addr, mr->len);\n+\t\treturn mr;\n+\t}\n+\n+\tif (second_try) {\n+\t\tDRV_LOG(ERR, \"Internal error second try failed\");\n+\t\treturn NULL;\n+\t}\n+\n+\tret = mana_new_pmd_mr(local_mr_btree, priv, pool);\n+\tif (ret) {\n+\t\tDRV_LOG(ERR, \"Failed to allocate MR ret %d addr %p len %d\",\n+\t\t\tret, mbuf->buf_addr, mbuf->buf_len);\n+\t\treturn NULL;\n+\t}\n+\n+\tsecond_try = 1;\n+\tgoto try_again;\n+}\n+\n+void mana_remove_all_mr(struct mana_priv *priv)\n+{\n+\tstruct mana_mr_btree *bt = &priv->mr_btree;\n+\tstruct mana_mr_cache *mr;\n+\tstruct ibv_mr *ibv_mr;\n+\tuint16_t i;\n+\n+\trte_spinlock_lock(&priv->mr_btree_lock);\n+\t/* Start with index 1 as the 1st entry is always NULL */\n+\tfor (i = 1; i < bt->len; i++) {\n+\t\tmr = &bt->table[i];\n+\t\tibv_mr = mr->verb_obj;\n+\t\tibv_dereg_mr(ibv_mr);\n+\t}\n+\tbt->len = 1;\n+\trte_spinlock_unlock(&priv->mr_btree_lock);\n+}\n+\n+static int mana_mr_btree_expand(struct mana_mr_btree *bt, int n)\n+{\n+\tvoid *mem;\n+\n+\tmem = rte_realloc_socket(bt->table, n * sizeof(struct mana_mr_cache),\n+\t\t\t\t 0, bt->socket);\n+\tif (!mem) {\n+\t\tDRV_LOG(ERR, \"Failed to expand btree size %d\", n);\n+\t\treturn -1;\n+\t}\n+\n+\tDRV_LOG(ERR, \"Expanded btree to size %d\", n);\n+\tbt->table = mem;\n+\tbt->size = n;\n+\n+\treturn 0;\n+}\n+\n+struct mana_mr_cache *mana_mr_btree_lookup(struct mana_mr_btree *bt,\n+\t\t\t\t\t uint16_t *idx,\n+\t\t\t\t\t uintptr_t addr, size_t len)\n+{\n+\tstruct mana_mr_cache *table;\n+\tuint16_t n;\n+\tuint16_t base = 0;\n+\tint ret;\n+\n+\tn = bt->len;\n+\n+\t/* Try to double the cache if it's full */\n+\tif (n == bt->size) {\n+\t\tret = mana_mr_btree_expand(bt, bt->size << 1);\n+\t\tif (ret)\n+\t\t\treturn NULL;\n+\t}\n+\n+\ttable = bt->table;\n+\n+\t/* Do binary search on addr */\n+\tdo {\n+\t\tuint16_t delta = n >> 1;\n+\n+\t\tif (addr < table[base + delta].addr) {\n+\t\t\tn = delta;\n+\t\t} else {\n+\t\t\tbase += delta;\n+\t\t\tn -= delta;\n+\t\t}\n+\t} while (n > 1);\n+\n+\t*idx = base;\n+\n+\tif (addr + len <= table[base].addr + table[base].len)\n+\t\treturn &table[base];\n+\n+\tDRV_LOG(DEBUG,\n+\t\t\"addr 0x%\" PRIx64 \" len %zu idx %u sum 0x%\" PRIx64 \" not found\",\n+\t\taddr, len, *idx, addr + len);\n+\n+\treturn NULL;\n+}\n+\n+int mana_mr_btree_init(struct mana_mr_btree *bt, int n, int socket)\n+{\n+\tmemset(bt, 0, sizeof(*bt));\n+\tbt->table = rte_calloc_socket(\"MANA B-tree table\",\n+\t\t\t\t n,\n+\t\t\t\t sizeof(struct mana_mr_cache),\n+\t\t\t\t 0, socket);\n+\tif (!bt->table) {\n+\t\tDRV_LOG(ERR, \"Failed to allocate B-tree n %d socket %d\",\n+\t\t\tn, socket);\n+\t\treturn -ENOMEM;\n+\t}\n+\n+\tbt->socket = socket;\n+\tbt->size = n;\n+\n+\t/* First entry must be NULL for binary search to work */\n+\tbt->table[0] = (struct mana_mr_cache) {\n+\t\t.lkey = UINT32_MAX,\n+\t};\n+\tbt->len = 1;\n+\n+\tDRV_LOG(ERR, \"B-tree initialized table %p size %d len %d\",\n+\t\tbt->table, n, bt->len);\n+\n+\treturn 0;\n+}\n+\n+void mana_mr_btree_free(struct mana_mr_btree *bt)\n+{\n+\trte_free(bt->table);\n+\tmemset(bt, 0, sizeof(*bt));\n+}\n+\n+int mana_mr_btree_insert(struct mana_mr_btree *bt, struct mana_mr_cache *entry)\n+{\n+\tstruct mana_mr_cache *table;\n+\tuint16_t idx = 0;\n+\tuint16_t shift;\n+\n+\tif (mana_mr_btree_lookup(bt, &idx, entry->addr, entry->len)) {\n+\t\tDRV_LOG(DEBUG, \"Addr 0x%\" PRIx64 \" len %zu exists in btree\",\n+\t\t\tentry->addr, entry->len);\n+\t\treturn 0;\n+\t}\n+\n+\tif (bt->len >= bt->size) {\n+\t\tbt->overflow = 1;\n+\t\treturn -1;\n+\t}\n+\n+\ttable = bt->table;\n+\n+\tidx++;\n+\tshift = (bt->len - idx) * sizeof(struct mana_mr_cache);\n+\tif (shift) {\n+\t\tDRV_LOG(DEBUG, \"Moving %u bytes from idx %u to %u\",\n+\t\t\tshift, idx, idx + 1);\n+\t\tmemmove(&table[idx + 1], &table[idx], shift);\n+\t}\n+\n+\ttable[idx] = *entry;\n+\tbt->len++;\n+\n+\tDRV_LOG(DEBUG,\n+\t\t\"Inserted MR b-tree table %p idx %d addr 0x%\" PRIx64 \" len %zu\",\n+\t\ttable, idx, entry->addr, entry->len);\n+\n+\treturn 0;\n+}\n", "prefixes": [ "v6", "10/18" ] }