Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/125683/?format=api
{ "id": 125683, "url": "http://patches.dpdk.org/api/patches/125683/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/patch/20230331154259.1447831-8-maxime.coquelin@redhat.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20230331154259.1447831-8-maxime.coquelin@redhat.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20230331154259.1447831-8-maxime.coquelin@redhat.com", "date": "2023-03-31T15:42:39", "name": "[RFC,07/27] vhost: change to single IOTLB cache per device", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": true, "hash": "ffc0666f42f64356df9765db969662c0336b558e", "submitter": { "id": 512, "url": "http://patches.dpdk.org/api/people/512/?format=api", "name": "Maxime Coquelin", "email": "maxime.coquelin@redhat.com" }, "delegate": { "id": 2642, "url": "http://patches.dpdk.org/api/users/2642/?format=api", "username": "mcoquelin", "first_name": "Maxime", "last_name": "Coquelin", "email": "maxime.coquelin@redhat.com" }, "mbox": "http://patches.dpdk.org/project/dpdk/patch/20230331154259.1447831-8-maxime.coquelin@redhat.com/mbox/", "series": [ { "id": 27594, "url": "http://patches.dpdk.org/api/series/27594/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=27594", "date": "2023-03-31T15:42:32", "name": "Add VDUSE support to Vhost library", "version": 1, "mbox": "http://patches.dpdk.org/series/27594/mbox/" } ], "comments": "http://patches.dpdk.org/api/patches/125683/comments/", "check": "pending", "checks": "http://patches.dpdk.org/api/patches/125683/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from mails.dpdk.org (mails.dpdk.org [217.70.189.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id BB48F42887;\n\tFri, 31 Mar 2023 17:44:03 +0200 (CEST)", "from mails.dpdk.org (localhost [127.0.0.1])\n\tby mails.dpdk.org (Postfix) with ESMTP id 9381142D86;\n\tFri, 31 Mar 2023 17:43:30 +0200 (CEST)", "from us-smtp-delivery-124.mimecast.com\n (us-smtp-delivery-124.mimecast.com [170.10.133.124])\n by mails.dpdk.org (Postfix) with ESMTP id 6C00E42D9B\n for <dev@dpdk.org>; Fri, 31 Mar 2023 17:43:28 +0200 (CEST)", "from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com\n [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n us-mta-382-sr2Bl0mQNy-IzP6ofedb6Q-1; Fri, 31 Mar 2023 11:43:24 -0400", "from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com\n [10.11.54.4])\n (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n (No client certificate requested)\n by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 485B02810C11;\n Fri, 31 Mar 2023 15:43:24 +0000 (UTC)", "from max-t490s.redhat.com (unknown [10.39.208.6])\n by smtp.corp.redhat.com (Postfix) with ESMTP id D22E32027040;\n Fri, 31 Mar 2023 15:43:21 +0000 (UTC)" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1680277408;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n content-transfer-encoding:content-transfer-encoding:\n in-reply-to:in-reply-to:references:references;\n bh=rrHtcleNyRPxarVUrmoArgi/ukFN/XB0c6AAtPTe1m4=;\n b=FpanshvakWILpNUIWoTUTqcEQ87rtSIF8Rwk9BALSbtPds38iRiCH5jrtHakhbjDzdpclx\n kbKLkQjnu9rLnRkCqF5fzqJpAsKyUvbbHMDpvwR6hvkESErrsMvlt8x5hWakIXxHA6AOWJ\n oEXzWIBgvLiVUWH1QhEchEFe0vOYBsY=", "X-MC-Unique": "sr2Bl0mQNy-IzP6ofedb6Q-1", "From": "Maxime Coquelin <maxime.coquelin@redhat.com>", "To": "dev@dpdk.org, david.marchand@redhat.com, chenbo.xia@intel.com,\n mkp@redhat.com, fbl@redhat.com, jasowang@redhat.com,\n cunming.liang@intel.com, xieyongji@bytedance.com, echaudro@redhat.com,\n eperezma@redhat.com, amorenoz@redhat.com", "Cc": "Maxime Coquelin <maxime.coquelin@redhat.com>", "Subject": "[RFC 07/27] vhost: change to single IOTLB cache per device", "Date": "Fri, 31 Mar 2023 17:42:39 +0200", "Message-Id": "<20230331154259.1447831-8-maxime.coquelin@redhat.com>", "In-Reply-To": "<20230331154259.1447831-1-maxime.coquelin@redhat.com>", "References": "<20230331154259.1447831-1-maxime.coquelin@redhat.com>", "MIME-Version": "1.0", "X-Scanned-By": "MIMEDefang 3.1 on 10.11.54.4", "X-Mimecast-Spam-Score": "0", "X-Mimecast-Originator": "redhat.com", "Content-Transfer-Encoding": "8bit", "Content-Type": "text/plain; charset=\"US-ASCII\"; x-default=true", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org" }, "content": "This patch simplifies IOTLB implementation and improves\nIOTLB memory consumption by having a single IOTLB cache\nper device, instead of having one per queue.\n\nIn order to not impact performance, it keeps an IOTLB lock\nper virtqueue, so that there is no contention between\nmultiple queue trying to acquire it.\n\nSigned-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>\n---\n lib/vhost/iotlb.c | 212 +++++++++++++++++++----------------------\n lib/vhost/iotlb.h | 43 ++++++---\n lib/vhost/vhost.c | 18 ++--\n lib/vhost/vhost.h | 16 ++--\n lib/vhost/vhost_user.c | 25 +++--\n 5 files changed, 160 insertions(+), 154 deletions(-)", "diff": "diff --git a/lib/vhost/iotlb.c b/lib/vhost/iotlb.c\nindex f598c0a8c4..a91115cf1c 100644\n--- a/lib/vhost/iotlb.c\n+++ b/lib/vhost/iotlb.c\n@@ -74,86 +74,81 @@ vhost_user_iotlb_clear_dump(struct virtio_net *dev, struct vhost_iotlb_entry *no\n }\n \n static struct vhost_iotlb_entry *\n-vhost_user_iotlb_pool_get(struct vhost_virtqueue *vq)\n+vhost_user_iotlb_pool_get(struct virtio_net *dev)\n {\n \tstruct vhost_iotlb_entry *node;\n \n-\trte_spinlock_lock(&vq->iotlb_free_lock);\n-\tnode = SLIST_FIRST(&vq->iotlb_free_list);\n+\trte_spinlock_lock(&dev->iotlb_free_lock);\n+\tnode = SLIST_FIRST(&dev->iotlb_free_list);\n \tif (node != NULL)\n-\t\tSLIST_REMOVE_HEAD(&vq->iotlb_free_list, next_free);\n-\trte_spinlock_unlock(&vq->iotlb_free_lock);\n+\t\tSLIST_REMOVE_HEAD(&dev->iotlb_free_list, next_free);\n+\trte_spinlock_unlock(&dev->iotlb_free_lock);\n \treturn node;\n }\n \n static void\n-vhost_user_iotlb_pool_put(struct vhost_virtqueue *vq,\n-\tstruct vhost_iotlb_entry *node)\n+vhost_user_iotlb_pool_put(struct virtio_net *dev, struct vhost_iotlb_entry *node)\n {\n-\trte_spinlock_lock(&vq->iotlb_free_lock);\n-\tSLIST_INSERT_HEAD(&vq->iotlb_free_list, node, next_free);\n-\trte_spinlock_unlock(&vq->iotlb_free_lock);\n+\trte_spinlock_lock(&dev->iotlb_free_lock);\n+\tSLIST_INSERT_HEAD(&dev->iotlb_free_list, node, next_free);\n+\trte_spinlock_unlock(&dev->iotlb_free_lock);\n }\n \n static void\n-vhost_user_iotlb_cache_random_evict(struct virtio_net *dev, struct vhost_virtqueue *vq);\n+vhost_user_iotlb_cache_random_evict(struct virtio_net *dev);\n \n static void\n-vhost_user_iotlb_pending_remove_all(struct vhost_virtqueue *vq)\n+vhost_user_iotlb_pending_remove_all(struct virtio_net *dev)\n {\n \tstruct vhost_iotlb_entry *node, *temp_node;\n \n-\trte_rwlock_write_lock(&vq->iotlb_pending_lock);\n+\trte_rwlock_write_lock(&dev->iotlb_pending_lock);\n \n-\tRTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_pending_list, next, temp_node) {\n-\t\tTAILQ_REMOVE(&vq->iotlb_pending_list, node, next);\n-\t\tvhost_user_iotlb_pool_put(vq, node);\n+\tRTE_TAILQ_FOREACH_SAFE(node, &dev->iotlb_pending_list, next, temp_node) {\n+\t\tTAILQ_REMOVE(&dev->iotlb_pending_list, node, next);\n+\t\tvhost_user_iotlb_pool_put(dev, node);\n \t}\n \n-\trte_rwlock_write_unlock(&vq->iotlb_pending_lock);\n+\trte_rwlock_write_unlock(&dev->iotlb_pending_lock);\n }\n \n bool\n-vhost_user_iotlb_pending_miss(struct vhost_virtqueue *vq, uint64_t iova,\n-\t\t\t\tuint8_t perm)\n+vhost_user_iotlb_pending_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm)\n {\n \tstruct vhost_iotlb_entry *node;\n \tbool found = false;\n \n-\trte_rwlock_read_lock(&vq->iotlb_pending_lock);\n+\trte_rwlock_read_lock(&dev->iotlb_pending_lock);\n \n-\tTAILQ_FOREACH(node, &vq->iotlb_pending_list, next) {\n+\tTAILQ_FOREACH(node, &dev->iotlb_pending_list, next) {\n \t\tif ((node->iova == iova) && (node->perm == perm)) {\n \t\t\tfound = true;\n \t\t\tbreak;\n \t\t}\n \t}\n \n-\trte_rwlock_read_unlock(&vq->iotlb_pending_lock);\n+\trte_rwlock_read_unlock(&dev->iotlb_pending_lock);\n \n \treturn found;\n }\n \n void\n-vhost_user_iotlb_pending_insert(struct virtio_net *dev, struct vhost_virtqueue *vq,\n-\t\t\t\tuint64_t iova, uint8_t perm)\n+vhost_user_iotlb_pending_insert(struct virtio_net *dev, uint64_t iova, uint8_t perm)\n {\n \tstruct vhost_iotlb_entry *node;\n \n-\tnode = vhost_user_iotlb_pool_get(vq);\n+\tnode = vhost_user_iotlb_pool_get(dev);\n \tif (node == NULL) {\n \t\tVHOST_LOG_CONFIG(dev->ifname, DEBUG,\n-\t\t\t\"IOTLB pool for vq %\"PRIu32\" empty, clear entries for pending insertion\\n\",\n-\t\t\tvq->index);\n-\t\tif (!TAILQ_EMPTY(&vq->iotlb_pending_list))\n-\t\t\tvhost_user_iotlb_pending_remove_all(vq);\n+\t\t\t\"IOTLB pool empty, clear entries for pending insertion\\n\");\n+\t\tif (!TAILQ_EMPTY(&dev->iotlb_pending_list))\n+\t\t\tvhost_user_iotlb_pending_remove_all(dev);\n \t\telse\n-\t\t\tvhost_user_iotlb_cache_random_evict(dev, vq);\n-\t\tnode = vhost_user_iotlb_pool_get(vq);\n+\t\t\tvhost_user_iotlb_cache_random_evict(dev);\n+\t\tnode = vhost_user_iotlb_pool_get(dev);\n \t\tif (node == NULL) {\n \t\t\tVHOST_LOG_CONFIG(dev->ifname, ERR,\n-\t\t\t\t\"IOTLB pool vq %\"PRIu32\" still empty, pending insertion failure\\n\",\n-\t\t\t\tvq->index);\n+\t\t\t\t\"IOTLB pool still empty, pending insertion failure\\n\");\n \t\t\treturn;\n \t\t}\n \t}\n@@ -161,22 +156,21 @@ vhost_user_iotlb_pending_insert(struct virtio_net *dev, struct vhost_virtqueue *\n \tnode->iova = iova;\n \tnode->perm = perm;\n \n-\trte_rwlock_write_lock(&vq->iotlb_pending_lock);\n+\trte_rwlock_write_lock(&dev->iotlb_pending_lock);\n \n-\tTAILQ_INSERT_TAIL(&vq->iotlb_pending_list, node, next);\n+\tTAILQ_INSERT_TAIL(&dev->iotlb_pending_list, node, next);\n \n-\trte_rwlock_write_unlock(&vq->iotlb_pending_lock);\n+\trte_rwlock_write_unlock(&dev->iotlb_pending_lock);\n }\n \n void\n-vhost_user_iotlb_pending_remove(struct vhost_virtqueue *vq,\n-\t\t\t\tuint64_t iova, uint64_t size, uint8_t perm)\n+vhost_user_iotlb_pending_remove(struct virtio_net *dev, uint64_t iova, uint64_t size, uint8_t perm)\n {\n \tstruct vhost_iotlb_entry *node, *temp_node;\n \n-\trte_rwlock_write_lock(&vq->iotlb_pending_lock);\n+\trte_rwlock_write_lock(&dev->iotlb_pending_lock);\n \n-\tRTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_pending_list, next,\n+\tRTE_TAILQ_FOREACH_SAFE(node, &dev->iotlb_pending_list, next,\n \t\t\t\ttemp_node) {\n \t\tif (node->iova < iova)\n \t\t\tcontinue;\n@@ -184,81 +178,78 @@ vhost_user_iotlb_pending_remove(struct vhost_virtqueue *vq,\n \t\t\tcontinue;\n \t\tif ((node->perm & perm) != node->perm)\n \t\t\tcontinue;\n-\t\tTAILQ_REMOVE(&vq->iotlb_pending_list, node, next);\n-\t\tvhost_user_iotlb_pool_put(vq, node);\n+\t\tTAILQ_REMOVE(&dev->iotlb_pending_list, node, next);\n+\t\tvhost_user_iotlb_pool_put(dev, node);\n \t}\n \n-\trte_rwlock_write_unlock(&vq->iotlb_pending_lock);\n+\trte_rwlock_write_unlock(&dev->iotlb_pending_lock);\n }\n \n static void\n-vhost_user_iotlb_cache_remove_all(struct virtio_net *dev, struct vhost_virtqueue *vq)\n+vhost_user_iotlb_cache_remove_all(struct virtio_net *dev)\n {\n \tstruct vhost_iotlb_entry *node, *temp_node;\n \n-\trte_rwlock_write_lock(&vq->iotlb_lock);\n+\tvhost_user_iotlb_wr_lock_all(dev);\n \n-\tRTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_list, next, temp_node) {\n+\tRTE_TAILQ_FOREACH_SAFE(node, &dev->iotlb_list, next, temp_node) {\n \t\tvhost_user_iotlb_set_dump(dev, node);\n \n-\t\tTAILQ_REMOVE(&vq->iotlb_list, node, next);\n-\t\tvhost_user_iotlb_pool_put(vq, node);\n+\t\tTAILQ_REMOVE(&dev->iotlb_list, node, next);\n+\t\tvhost_user_iotlb_pool_put(dev, node);\n \t}\n \n-\tvq->iotlb_cache_nr = 0;\n+\tdev->iotlb_cache_nr = 0;\n \n-\trte_rwlock_write_unlock(&vq->iotlb_lock);\n+\tvhost_user_iotlb_wr_unlock_all(dev);\n }\n \n static void\n-vhost_user_iotlb_cache_random_evict(struct virtio_net *dev, struct vhost_virtqueue *vq)\n+vhost_user_iotlb_cache_random_evict(struct virtio_net *dev)\n {\n \tstruct vhost_iotlb_entry *node, *temp_node, *prev_node = NULL;\n \tint entry_idx;\n \n-\trte_rwlock_write_lock(&vq->iotlb_lock);\n+\tvhost_user_iotlb_wr_lock_all(dev);\n \n-\tentry_idx = rte_rand() % vq->iotlb_cache_nr;\n+\tentry_idx = rte_rand() % dev->iotlb_cache_nr;\n \n-\tRTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_list, next, temp_node) {\n+\tRTE_TAILQ_FOREACH_SAFE(node, &dev->iotlb_list, next, temp_node) {\n \t\tif (!entry_idx) {\n \t\t\tstruct vhost_iotlb_entry *next_node = RTE_TAILQ_NEXT(node, next);\n \n \t\t\tvhost_user_iotlb_clear_dump(dev, node, prev_node, next_node);\n \n-\t\t\tTAILQ_REMOVE(&vq->iotlb_list, node, next);\n-\t\t\tvhost_user_iotlb_pool_put(vq, node);\n-\t\t\tvq->iotlb_cache_nr--;\n+\t\t\tTAILQ_REMOVE(&dev->iotlb_list, node, next);\n+\t\t\tvhost_user_iotlb_pool_put(dev, node);\n+\t\t\tdev->iotlb_cache_nr--;\n \t\t\tbreak;\n \t\t}\n \t\tprev_node = node;\n \t\tentry_idx--;\n \t}\n \n-\trte_rwlock_write_unlock(&vq->iotlb_lock);\n+\tvhost_user_iotlb_wr_unlock_all(dev);\n }\n \n void\n-vhost_user_iotlb_cache_insert(struct virtio_net *dev, struct vhost_virtqueue *vq,\n-\t\t\t\tuint64_t iova, uint64_t uaddr,\n+vhost_user_iotlb_cache_insert(struct virtio_net *dev, uint64_t iova, uint64_t uaddr,\n \t\t\t\tuint64_t size, uint8_t perm)\n {\n \tstruct vhost_iotlb_entry *node, *new_node;\n \n-\tnew_node = vhost_user_iotlb_pool_get(vq);\n+\tnew_node = vhost_user_iotlb_pool_get(dev);\n \tif (new_node == NULL) {\n \t\tVHOST_LOG_CONFIG(dev->ifname, DEBUG,\n-\t\t\t\"IOTLB pool vq %\"PRIu32\" empty, clear entries for cache insertion\\n\",\n-\t\t\tvq->index);\n-\t\tif (!TAILQ_EMPTY(&vq->iotlb_list))\n-\t\t\tvhost_user_iotlb_cache_random_evict(dev, vq);\n+\t\t\t\"IOTLB pool empty, clear entries for cache insertion\\n\");\n+\t\tif (!TAILQ_EMPTY(&dev->iotlb_list))\n+\t\t\tvhost_user_iotlb_cache_random_evict(dev);\n \t\telse\n-\t\t\tvhost_user_iotlb_pending_remove_all(vq);\n-\t\tnew_node = vhost_user_iotlb_pool_get(vq);\n+\t\t\tvhost_user_iotlb_pending_remove_all(dev);\n+\t\tnew_node = vhost_user_iotlb_pool_get(dev);\n \t\tif (new_node == NULL) {\n \t\t\tVHOST_LOG_CONFIG(dev->ifname, ERR,\n-\t\t\t\t\"IOTLB pool vq %\"PRIu32\" still empty, cache insertion failed\\n\",\n-\t\t\t\tvq->index);\n+\t\t\t\t\"IOTLB pool still empty, cache insertion failed\\n\");\n \t\t\treturn;\n \t\t}\n \t}\n@@ -268,49 +259,47 @@ vhost_user_iotlb_cache_insert(struct virtio_net *dev, struct vhost_virtqueue *vq\n \tnew_node->size = size;\n \tnew_node->perm = perm;\n \n-\trte_rwlock_write_lock(&vq->iotlb_lock);\n+\tvhost_user_iotlb_wr_lock_all(dev);\n \n-\tTAILQ_FOREACH(node, &vq->iotlb_list, next) {\n+\tTAILQ_FOREACH(node, &dev->iotlb_list, next) {\n \t\t/*\n \t\t * Entries must be invalidated before being updated.\n \t\t * So if iova already in list, assume identical.\n \t\t */\n \t\tif (node->iova == new_node->iova) {\n-\t\t\tvhost_user_iotlb_pool_put(vq, new_node);\n+\t\t\tvhost_user_iotlb_pool_put(dev, new_node);\n \t\t\tgoto unlock;\n \t\t} else if (node->iova > new_node->iova) {\n \t\t\tvhost_user_iotlb_set_dump(dev, new_node);\n \n \t\t\tTAILQ_INSERT_BEFORE(node, new_node, next);\n-\t\t\tvq->iotlb_cache_nr++;\n+\t\t\tdev->iotlb_cache_nr++;\n \t\t\tgoto unlock;\n \t\t}\n \t}\n \n \tvhost_user_iotlb_set_dump(dev, new_node);\n \n-\tTAILQ_INSERT_TAIL(&vq->iotlb_list, new_node, next);\n-\tvq->iotlb_cache_nr++;\n+\tTAILQ_INSERT_TAIL(&dev->iotlb_list, new_node, next);\n+\tdev->iotlb_cache_nr++;\n \n unlock:\n-\tvhost_user_iotlb_pending_remove(vq, iova, size, perm);\n-\n-\trte_rwlock_write_unlock(&vq->iotlb_lock);\n+\tvhost_user_iotlb_pending_remove(dev, iova, size, perm);\n \n+\tvhost_user_iotlb_wr_unlock_all(dev);\n }\n \n void\n-vhost_user_iotlb_cache_remove(struct virtio_net *dev, struct vhost_virtqueue *vq,\n-\t\t\t\t\tuint64_t iova, uint64_t size)\n+vhost_user_iotlb_cache_remove(struct virtio_net *dev, uint64_t iova, uint64_t size)\n {\n \tstruct vhost_iotlb_entry *node, *temp_node, *prev_node = NULL;\n \n \tif (unlikely(!size))\n \t\treturn;\n \n-\trte_rwlock_write_lock(&vq->iotlb_lock);\n+\tvhost_user_iotlb_wr_lock_all(dev);\n \n-\tRTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_list, next, temp_node) {\n+\tRTE_TAILQ_FOREACH_SAFE(node, &dev->iotlb_list, next, temp_node) {\n \t\t/* Sorted list */\n \t\tif (unlikely(iova + size < node->iova))\n \t\t\tbreak;\n@@ -320,19 +309,19 @@ vhost_user_iotlb_cache_remove(struct virtio_net *dev, struct vhost_virtqueue *vq\n \n \t\t\tvhost_user_iotlb_clear_dump(dev, node, prev_node, next_node);\n \n-\t\t\tTAILQ_REMOVE(&vq->iotlb_list, node, next);\n-\t\t\tvhost_user_iotlb_pool_put(vq, node);\n-\t\t\tvq->iotlb_cache_nr--;\n-\t\t} else\n+\t\t\tTAILQ_REMOVE(&dev->iotlb_list, node, next);\n+\t\t\tvhost_user_iotlb_pool_put(dev, node);\n+\t\t\tdev->iotlb_cache_nr--;\n+\t\t} else {\n \t\t\tprev_node = node;\n+\t\t}\n \t}\n \n-\trte_rwlock_write_unlock(&vq->iotlb_lock);\n+\tvhost_user_iotlb_wr_unlock_all(dev);\n }\n \n uint64_t\n-vhost_user_iotlb_cache_find(struct vhost_virtqueue *vq, uint64_t iova,\n-\t\t\t\t\t\tuint64_t *size, uint8_t perm)\n+vhost_user_iotlb_cache_find(struct virtio_net *dev, uint64_t iova, uint64_t *size, uint8_t perm)\n {\n \tstruct vhost_iotlb_entry *node;\n \tuint64_t offset, vva = 0, mapped = 0;\n@@ -340,7 +329,7 @@ vhost_user_iotlb_cache_find(struct vhost_virtqueue *vq, uint64_t iova,\n \tif (unlikely(!*size))\n \t\tgoto out;\n \n-\tTAILQ_FOREACH(node, &vq->iotlb_list, next) {\n+\tTAILQ_FOREACH(node, &dev->iotlb_list, next) {\n \t\t/* List sorted by iova */\n \t\tif (unlikely(iova < node->iova))\n \t\t\tbreak;\n@@ -373,60 +362,57 @@ vhost_user_iotlb_cache_find(struct vhost_virtqueue *vq, uint64_t iova,\n }\n \n void\n-vhost_user_iotlb_flush_all(struct virtio_net *dev, struct vhost_virtqueue *vq)\n+vhost_user_iotlb_flush_all(struct virtio_net *dev)\n {\n-\tvhost_user_iotlb_cache_remove_all(dev, vq);\n-\tvhost_user_iotlb_pending_remove_all(vq);\n+\tvhost_user_iotlb_cache_remove_all(dev);\n+\tvhost_user_iotlb_pending_remove_all(dev);\n }\n \n int\n-vhost_user_iotlb_init(struct virtio_net *dev, struct vhost_virtqueue *vq)\n+vhost_user_iotlb_init(struct virtio_net *dev)\n {\n \tunsigned int i;\n \tint socket = 0;\n \n-\tif (vq->iotlb_pool) {\n+\tif (dev->iotlb_pool) {\n \t\t/*\n \t\t * The cache has already been initialized,\n \t\t * just drop all cached and pending entries.\n \t\t */\n-\t\tvhost_user_iotlb_flush_all(dev, vq);\n-\t\trte_free(vq->iotlb_pool);\n+\t\tvhost_user_iotlb_flush_all(dev);\n+\t\trte_free(dev->iotlb_pool);\n \t}\n \n #ifdef RTE_LIBRTE_VHOST_NUMA\n-\tif (get_mempolicy(&socket, NULL, 0, vq, MPOL_F_NODE | MPOL_F_ADDR) != 0)\n+\tif (get_mempolicy(&socket, NULL, 0, dev, MPOL_F_NODE | MPOL_F_ADDR) != 0)\n \t\tsocket = 0;\n #endif\n \n-\trte_spinlock_init(&vq->iotlb_free_lock);\n-\trte_rwlock_init(&vq->iotlb_lock);\n-\trte_rwlock_init(&vq->iotlb_pending_lock);\n+\trte_spinlock_init(&dev->iotlb_free_lock);\n+\trte_rwlock_init(&dev->iotlb_pending_lock);\n \n-\tSLIST_INIT(&vq->iotlb_free_list);\n-\tTAILQ_INIT(&vq->iotlb_list);\n-\tTAILQ_INIT(&vq->iotlb_pending_list);\n+\tSLIST_INIT(&dev->iotlb_free_list);\n+\tTAILQ_INIT(&dev->iotlb_list);\n+\tTAILQ_INIT(&dev->iotlb_pending_list);\n \n \tif (dev->flags & VIRTIO_DEV_SUPPORT_IOMMU) {\n-\t\tvq->iotlb_pool = rte_calloc_socket(\"iotlb\", IOTLB_CACHE_SIZE,\n+\t\tdev->iotlb_pool = rte_calloc_socket(\"iotlb\", IOTLB_CACHE_SIZE,\n \t\t\tsizeof(struct vhost_iotlb_entry), 0, socket);\n-\t\tif (!vq->iotlb_pool) {\n-\t\t\tVHOST_LOG_CONFIG(dev->ifname, ERR,\n-\t\t\t\t\"Failed to create IOTLB cache pool for vq %\"PRIu32\"\\n\",\n-\t\t\t\tvq->index);\n+\t\tif (!dev->iotlb_pool) {\n+\t\t\tVHOST_LOG_CONFIG(dev->ifname, ERR, \"Failed to create IOTLB cache pool\\n\");\n \t\t\treturn -1;\n \t\t}\n \t\tfor (i = 0; i < IOTLB_CACHE_SIZE; i++)\n-\t\t\tvhost_user_iotlb_pool_put(vq, &vq->iotlb_pool[i]);\n+\t\t\tvhost_user_iotlb_pool_put(dev, &dev->iotlb_pool[i]);\n \t}\n \n-\tvq->iotlb_cache_nr = 0;\n+\tdev->iotlb_cache_nr = 0;\n \n \treturn 0;\n }\n \n void\n-vhost_user_iotlb_destroy(struct vhost_virtqueue *vq)\n+vhost_user_iotlb_destroy(struct virtio_net *dev)\n {\n-\trte_free(vq->iotlb_pool);\n+\trte_free(dev->iotlb_pool);\n }\ndiff --git a/lib/vhost/iotlb.h b/lib/vhost/iotlb.h\nindex 73b5465b41..3490b9e6be 100644\n--- a/lib/vhost/iotlb.h\n+++ b/lib/vhost/iotlb.h\n@@ -37,20 +37,37 @@ vhost_user_iotlb_wr_unlock(struct vhost_virtqueue *vq)\n \trte_rwlock_write_unlock(&vq->iotlb_lock);\n }\n \n-void vhost_user_iotlb_cache_insert(struct virtio_net *dev, struct vhost_virtqueue *vq,\n-\t\t\t\t\tuint64_t iova, uint64_t uaddr,\n+static __rte_always_inline void\n+vhost_user_iotlb_wr_lock_all(struct virtio_net *dev)\n+\t__rte_no_thread_safety_analysis\n+{\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < dev->nr_vring; i++)\n+\t\trte_rwlock_write_lock(&dev->virtqueue[i]->iotlb_lock);\n+}\n+\n+static __rte_always_inline void\n+vhost_user_iotlb_wr_unlock_all(struct virtio_net *dev)\n+\t__rte_no_thread_safety_analysis\n+{\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < dev->nr_vring; i++)\n+\t\trte_rwlock_write_unlock(&dev->virtqueue[i]->iotlb_lock);\n+}\n+\n+void vhost_user_iotlb_cache_insert(struct virtio_net *dev, uint64_t iova, uint64_t uaddr,\n \t\t\t\t\tuint64_t size, uint8_t perm);\n-void vhost_user_iotlb_cache_remove(struct virtio_net *dev, struct vhost_virtqueue *vq,\n-\t\t\t\t\tuint64_t iova, uint64_t size);\n-uint64_t vhost_user_iotlb_cache_find(struct vhost_virtqueue *vq, uint64_t iova,\n+void vhost_user_iotlb_cache_remove(struct virtio_net *dev, uint64_t iova, uint64_t size);\n+uint64_t vhost_user_iotlb_cache_find(struct virtio_net *dev, uint64_t iova,\n \t\t\t\t\tuint64_t *size, uint8_t perm);\n-bool vhost_user_iotlb_pending_miss(struct vhost_virtqueue *vq, uint64_t iova,\n-\t\t\t\t\t\tuint8_t perm);\n-void vhost_user_iotlb_pending_insert(struct virtio_net *dev, struct vhost_virtqueue *vq,\n-\t\t\t\t\t\tuint64_t iova, uint8_t perm);\n-void vhost_user_iotlb_pending_remove(struct vhost_virtqueue *vq, uint64_t iova,\n+bool vhost_user_iotlb_pending_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm);\n+void vhost_user_iotlb_pending_insert(struct virtio_net *dev, uint64_t iova, uint8_t perm);\n+void vhost_user_iotlb_pending_remove(struct virtio_net *dev, uint64_t iova,\n \t\t\t\t\t\tuint64_t size, uint8_t perm);\n-void vhost_user_iotlb_flush_all(struct virtio_net *dev, struct vhost_virtqueue *vq);\n-int vhost_user_iotlb_init(struct virtio_net *dev, struct vhost_virtqueue *vq);\n-void vhost_user_iotlb_destroy(struct vhost_virtqueue *vq);\n+void vhost_user_iotlb_flush_all(struct virtio_net *dev);\n+int vhost_user_iotlb_init(struct virtio_net *dev);\n+void vhost_user_iotlb_destroy(struct virtio_net *dev);\n+\n #endif /* _VHOST_IOTLB_H_ */\ndiff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c\nindex ef37943817..d35075b96c 100644\n--- a/lib/vhost/vhost.c\n+++ b/lib/vhost/vhost.c\n@@ -63,7 +63,7 @@ __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,\n \n \ttmp_size = *size;\n \n-\tvva = vhost_user_iotlb_cache_find(vq, iova, &tmp_size, perm);\n+\tvva = vhost_user_iotlb_cache_find(dev, iova, &tmp_size, perm);\n \tif (tmp_size == *size) {\n \t\tif (dev->flags & VIRTIO_DEV_STATS_ENABLED)\n \t\t\tvq->stats.iotlb_hits++;\n@@ -75,7 +75,7 @@ __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,\n \n \tiova += tmp_size;\n \n-\tif (!vhost_user_iotlb_pending_miss(vq, iova, perm)) {\n+\tif (!vhost_user_iotlb_pending_miss(dev, iova, perm)) {\n \t\t/*\n \t\t * iotlb_lock is read-locked for a full burst,\n \t\t * but it only protects the iotlb cache.\n@@ -85,12 +85,12 @@ __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,\n \t\t */\n \t\tvhost_user_iotlb_rd_unlock(vq);\n \n-\t\tvhost_user_iotlb_pending_insert(dev, vq, iova, perm);\n+\t\tvhost_user_iotlb_pending_insert(dev, iova, perm);\n \t\tif (vhost_user_iotlb_miss(dev, iova, perm)) {\n \t\t\tVHOST_LOG_DATA(dev->ifname, ERR,\n \t\t\t\t\"IOTLB miss req failed for IOVA 0x%\" PRIx64 \"\\n\",\n \t\t\t\tiova);\n-\t\t\tvhost_user_iotlb_pending_remove(vq, iova, 1, perm);\n+\t\t\tvhost_user_iotlb_pending_remove(dev, iova, 1, perm);\n \t\t}\n \n \t\tvhost_user_iotlb_rd_lock(vq);\n@@ -397,7 +397,6 @@ free_vq(struct virtio_net *dev, struct vhost_virtqueue *vq)\n \tvhost_free_async_mem(vq);\n \trte_spinlock_unlock(&vq->access_lock);\n \trte_free(vq->batch_copy_elems);\n-\tvhost_user_iotlb_destroy(vq);\n \trte_free(vq->log_cache);\n \trte_free(vq);\n }\n@@ -575,7 +574,7 @@ vring_invalidate(struct virtio_net *dev __rte_unused, struct vhost_virtqueue *vq\n }\n \n static void\n-init_vring_queue(struct virtio_net *dev, struct vhost_virtqueue *vq,\n+init_vring_queue(struct virtio_net *dev __rte_unused, struct vhost_virtqueue *vq,\n \tuint32_t vring_idx)\n {\n \tint numa_node = SOCKET_ID_ANY;\n@@ -595,8 +594,6 @@ init_vring_queue(struct virtio_net *dev, struct vhost_virtqueue *vq,\n \t}\n #endif\n \tvq->numa_node = numa_node;\n-\n-\tvhost_user_iotlb_init(dev, vq);\n }\n \n static void\n@@ -631,6 +628,7 @@ alloc_vring_queue(struct virtio_net *dev, uint32_t vring_idx)\n \t\tdev->virtqueue[i] = vq;\n \t\tinit_vring_queue(dev, vq, i);\n \t\trte_spinlock_init(&vq->access_lock);\n+\t\trte_rwlock_init(&vq->iotlb_lock);\n \t\tvq->avail_wrap_counter = 1;\n \t\tvq->used_wrap_counter = 1;\n \t\tvq->signalled_used_valid = false;\n@@ -795,6 +793,10 @@ vhost_setup_virtio_net(int vid, bool enable, bool compliant_ol_flags, bool stats\n \t\tdev->flags |= VIRTIO_DEV_SUPPORT_IOMMU;\n \telse\n \t\tdev->flags &= ~VIRTIO_DEV_SUPPORT_IOMMU;\n+\n+\tif (vhost_user_iotlb_init(dev) < 0)\n+\t\tVHOST_LOG_CONFIG(\"device\", ERR, \"failed to init IOTLB\\n\");\n+\n }\n \n void\ndiff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h\nindex 40863f7bfd..67cc4a2fdb 100644\n--- a/lib/vhost/vhost.h\n+++ b/lib/vhost/vhost.h\n@@ -302,13 +302,6 @@ struct vhost_virtqueue {\n \tstruct log_cache_entry\t*log_cache;\n \n \trte_rwlock_t\tiotlb_lock;\n-\trte_rwlock_t\tiotlb_pending_lock;\n-\tstruct vhost_iotlb_entry *iotlb_pool;\n-\tTAILQ_HEAD(, vhost_iotlb_entry) iotlb_list;\n-\tTAILQ_HEAD(, vhost_iotlb_entry) iotlb_pending_list;\n-\tint\t\t\t\tiotlb_cache_nr;\n-\trte_spinlock_t\tiotlb_free_lock;\n-\tSLIST_HEAD(, vhost_iotlb_entry) iotlb_free_list;\n \n \t/* Used to notify the guest (trigger interrupt) */\n \tint\t\t\tcallfd;\n@@ -483,6 +476,15 @@ struct virtio_net {\n \tint\t\t\textbuf;\n \tint\t\t\tlinearbuf;\n \tstruct vhost_virtqueue\t*virtqueue[VHOST_MAX_QUEUE_PAIRS * 2];\n+\n+\trte_rwlock_t\tiotlb_pending_lock;\n+\tstruct vhost_iotlb_entry *iotlb_pool;\n+\tTAILQ_HEAD(, vhost_iotlb_entry) iotlb_list;\n+\tTAILQ_HEAD(, vhost_iotlb_entry) iotlb_pending_list;\n+\tint\t\t\t\tiotlb_cache_nr;\n+\trte_spinlock_t\tiotlb_free_lock;\n+\tSLIST_HEAD(, vhost_iotlb_entry) iotlb_free_list;\n+\n \tstruct inflight_mem_info *inflight_info;\n #define IF_NAME_SZ (PATH_MAX > IFNAMSIZ ? PATH_MAX : IFNAMSIZ)\n \tchar\t\t\tifname[IF_NAME_SZ];\ndiff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c\nindex d60e39b6bc..81ebef0137 100644\n--- a/lib/vhost/vhost_user.c\n+++ b/lib/vhost/vhost_user.c\n@@ -7,7 +7,7 @@\n * The vhost-user protocol connection is an external interface, so it must be\n * robust against invalid inputs.\n *\n- * This is important because the vhost-user frontend is only one step removed\n+* This is important because the vhost-user frontend is only one step removed\n * from the guest. Malicious guests that have escaped will then launch further\n * attacks from the vhost-user frontend.\n *\n@@ -237,6 +237,8 @@ vhost_backend_cleanup(struct virtio_net *dev)\n \t}\n \n \tdev->postcopy_listening = 0;\n+\n+\tvhost_user_iotlb_destroy(dev);\n }\n \n static void\n@@ -539,7 +541,6 @@ numa_realloc(struct virtio_net **pdev, struct vhost_virtqueue **pvq)\n \tif (vq != dev->virtqueue[vq->index]) {\n \t\tVHOST_LOG_CONFIG(dev->ifname, INFO, \"reallocated virtqueue on node %d\\n\", node);\n \t\tdev->virtqueue[vq->index] = vq;\n-\t\tvhost_user_iotlb_init(dev, vq);\n \t}\n \n \tif (vq_is_packed(dev)) {\n@@ -664,6 +665,8 @@ numa_realloc(struct virtio_net **pdev, struct vhost_virtqueue **pvq)\n \t\treturn;\n \t}\n \tdev->guest_pages = gp;\n+\n+\tvhost_user_iotlb_init(dev);\n }\n #else\n static void\n@@ -1360,8 +1363,7 @@ vhost_user_set_mem_table(struct virtio_net **pdev,\n \n \t\t/* Flush IOTLB cache as previous HVAs are now invalid */\n \t\tif (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))\n-\t\t\tfor (i = 0; i < dev->nr_vring; i++)\n-\t\t\t\tvhost_user_iotlb_flush_all(dev, dev->virtqueue[i]);\n+\t\t\tvhost_user_iotlb_flush_all(dev);\n \n \t\tfree_mem_region(dev);\n \t\trte_free(dev->mem);\n@@ -2194,7 +2196,7 @@ vhost_user_get_vring_base(struct virtio_net **pdev,\n \tctx->msg.size = sizeof(ctx->msg.payload.state);\n \tctx->fd_num = 0;\n \n-\tvhost_user_iotlb_flush_all(dev, vq);\n+\tvhost_user_iotlb_flush_all(dev);\n \n \tvring_invalidate(dev, vq);\n \n@@ -2639,15 +2641,14 @@ vhost_user_iotlb_msg(struct virtio_net **pdev,\n \t\tif (!vva)\n \t\t\treturn RTE_VHOST_MSG_RESULT_ERR;\n \n+\t\tvhost_user_iotlb_cache_insert(dev, imsg->iova, vva, len, imsg->perm);\n+\n \t\tfor (i = 0; i < dev->nr_vring; i++) {\n \t\t\tstruct vhost_virtqueue *vq = dev->virtqueue[i];\n \n \t\t\tif (!vq)\n \t\t\t\tcontinue;\n \n-\t\t\tvhost_user_iotlb_cache_insert(dev, vq, imsg->iova, vva,\n-\t\t\t\t\tlen, imsg->perm);\n-\n \t\t\tif (is_vring_iotlb(dev, vq, imsg)) {\n \t\t\t\trte_spinlock_lock(&vq->access_lock);\n \t\t\t\ttranslate_ring_addresses(&dev, &vq);\n@@ -2657,15 +2658,14 @@ vhost_user_iotlb_msg(struct virtio_net **pdev,\n \t\t}\n \t\tbreak;\n \tcase VHOST_IOTLB_INVALIDATE:\n+\t\tvhost_user_iotlb_cache_remove(dev, imsg->iova, imsg->size);\n+\n \t\tfor (i = 0; i < dev->nr_vring; i++) {\n \t\t\tstruct vhost_virtqueue *vq = dev->virtqueue[i];\n \n \t\t\tif (!vq)\n \t\t\t\tcontinue;\n \n-\t\t\tvhost_user_iotlb_cache_remove(dev, vq, imsg->iova,\n-\t\t\t\t\timsg->size);\n-\n \t\t\tif (is_vring_iotlb(dev, vq, imsg)) {\n \t\t\t\trte_spinlock_lock(&vq->access_lock);\n \t\t\t\tvring_invalidate(dev, vq);\n@@ -2674,8 +2674,7 @@ vhost_user_iotlb_msg(struct virtio_net **pdev,\n \t\t}\n \t\tbreak;\n \tdefault:\n-\t\tVHOST_LOG_CONFIG(dev->ifname, ERR,\n-\t\t\t\"invalid IOTLB message type (%d)\\n\",\n+\t\tVHOST_LOG_CONFIG(dev->ifname, ERR, \"invalid IOTLB message type (%d)\\n\",\n \t\t\timsg->type);\n \t\treturn RTE_VHOST_MSG_RESULT_ERR;\n \t}\n", "prefixes": [ "RFC", "07/27" ] }