From patchwork Thu Jun 17 10:58:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 94347 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 00466A0C4D; Thu, 17 Jun 2021 13:00:45 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0392241158; Thu, 17 Jun 2021 12:59:02 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id B807941175 for ; Thu, 17 Jun 2021 12:58:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623927538; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oJhGR6NiPWyKfoOC8Q6Oj2J0o69GIZIwPS2nwBQYCOY=; b=hRBjsqK4gAw/UPJZsyPRZWGSoHuQCcvBF4bP+TzSa3NC9uSaRB4P5y8YxniSPMwDky0wIu kqRHaO3NlvsWXyLs3jm66g2gw1uVPX1Iac9wzEFxnGdfiJv4L7lzVfmY//NHFJ1cKzYIdG jC7NBBmC2ngoJfmAJBPviBimfcm51UE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-195-Gy7TO6E2ONanoQPf9cRkEQ-1; Thu, 17 Jun 2021 06:58:55 -0400 X-MC-Unique: Gy7TO6E2ONanoQPf9cRkEQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B34BC3EB9; Thu, 17 Jun 2021 10:58:52 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.36.110.45]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1914F60E3A; Thu, 17 Jun 2021 10:58:50 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, david.marchand@redhat.com, chenbo.xia@intel.com Cc: Maxime Coquelin , stable@dpdk.org Date: Thu, 17 Jun 2021 12:58:39 +0200 Message-Id: <20210617105845.125472-2-maxime.coquelin@redhat.com> In-Reply-To: <20210617105845.125472-1-maxime.coquelin@redhat.com> References: <20210617105845.125472-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH 1/7] vhost: fix missing memory table NUMA realloc X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When the guest allocates virtqueues on a different NUMA node than the one the Vhost metadata are allocated, both the Vhost device struct and the virtqueues struct are reallocated. However, reallocating the Vhost memory table was missing, which likely causes iat least one cross-NUMA accesses for every burst of packets. This patch reallocates this table on the same NUMA node as the other metadata. Fixes: 552e8fd3d2b4 ("vhost: simplify memory regions handling") Cc: stable@dpdk.org Reported-by: David Marchand Signed-off-by: Maxime Coquelin --- lib/vhost/vhost_user.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 8f0eba6412..031e3bfa2f 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -557,6 +557,9 @@ numa_realloc(struct virtio_net *dev, int index) goto out; } if (oldnode != newnode) { + struct rte_vhost_memory *old_mem; + ssize_t mem_size; + VHOST_LOG_CONFIG(INFO, "reallocate dev from %d to %d node\n", oldnode, newnode); @@ -568,6 +571,18 @@ numa_realloc(struct virtio_net *dev, int index) memcpy(dev, old_dev, sizeof(*dev)); rte_free(old_dev); + + mem_size = sizeof(struct rte_vhost_memory) + + sizeof(struct rte_vhost_mem_region) * dev->mem->nregions; + old_mem = dev->mem; + dev->mem = rte_malloc_socket(NULL, mem_size, 0, newnode); + if (!dev->mem) { + dev->mem = old_mem; + goto out; + } + + memcpy(dev->mem, old_mem, mem_size); + rte_free(old_mem); } out: From patchwork Thu Jun 17 10:58:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 94346 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 80B7CA0C4D; Thu, 17 Jun 2021 13:00:40 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AC34341181; Thu, 17 Jun 2021 12:59:00 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id 7A4344111B for ; Thu, 17 Jun 2021 12:58:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623927537; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eGzY5Z7Uhq4x+kamaF7STpZ+s1TxFHVRuU61XxkXeik=; b=flRBLLDSYzrqlkpC0t5jfv1n2Gy5a0Dk/OoyIop2MSsAl6dEZhwo6jVVzbeW9ds3W75Bo+ sOvW/Ge/Bu19YFd7j0xUTjz6FmGcmEC2gfPeTUO753pZGqaXjaroCcyc2eS1eNYkGIbSev woSTIARBtDp3cD9xHS/UUT952+r8IJY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-125-C3eNWzqdM7S4WJ5-s7jdFg-1; Thu, 17 Jun 2021 06:58:56 -0400 X-MC-Unique: C3eNWzqdM7S4WJ5-s7jdFg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 513F156AA7; Thu, 17 Jun 2021 10:58:54 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.36.110.45]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1816E519D0; Thu, 17 Jun 2021 10:58:52 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, david.marchand@redhat.com, chenbo.xia@intel.com Cc: Maxime Coquelin , stable@dpdk.org Date: Thu, 17 Jun 2021 12:58:40 +0200 Message-Id: <20210617105845.125472-3-maxime.coquelin@redhat.com> In-Reply-To: <20210617105845.125472-1-maxime.coquelin@redhat.com> References: <20210617105845.125472-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH 2/7] vhost: fix missing guest pages table NUMA realloc X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When the guest allocates virtqueues on a different NUMA node than the one the Vhost metadata are allocated, both the Vhost device struct and the virtqueues struct are reallocated. However, reallocating the guest pages table was missing, which likely causes at least one cross-NUMA accesses for every burst of packets. This patch reallocates this table on the same NUMA node as the other metadata. Fixes: e246896178e6 ("vhost: get guest/host physical address mappings") Cc: stable@dpdk.org Signed-off-by: Maxime Coquelin --- lib/vhost/vhost_user.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 031e3bfa2f..cbfdf1b4d8 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -558,7 +558,8 @@ numa_realloc(struct virtio_net *dev, int index) } if (oldnode != newnode) { struct rte_vhost_memory *old_mem; - ssize_t mem_size; + struct guest_page *old_gp; + ssize_t mem_size, gp_size; VHOST_LOG_CONFIG(INFO, "reallocate dev from %d to %d node\n", @@ -583,6 +584,17 @@ numa_realloc(struct virtio_net *dev, int index) memcpy(dev->mem, old_mem, mem_size); rte_free(old_mem); + + gp_size = dev->max_guest_pages * sizeof(*dev->guest_pages); + old_gp = dev->guest_pages; + dev->guest_pages = rte_malloc_socket(NULL, gp_size, RTE_CACHE_LINE_SIZE, newnode); + if (!dev->guest_pages) { + dev->guest_pages = old_gp; + goto out; + } + + memcpy(dev->guest_pages, old_gp, gp_size); + rte_free(old_gp); } out: From patchwork Thu Jun 17 10:58:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 94348 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 55415A0C4D; Thu, 17 Jun 2021 13:00:53 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6D19D4118F; Thu, 17 Jun 2021 12:59:03 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 72D5E41176 for ; Thu, 17 Jun 2021 12:59:01 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623927541; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ChwUlThbHfFzZ91px8E/6E27SMFCt+z7hNlONUcROCk=; b=HqZzbH8znLQ9Avm/0HqxMrpypZJBF69TdZXWO5TdYVs50zWVd1rFpy15tcaU8OOdnOryWT lo1MkhslBERaotS6b3auzNjo2Mh0i20hk0PcUL3wZhT8qzJ5n0Y2ONlR7XuC1kOBW2a6w9 lQkxYnPG6riG0KeB4vg3xsN1kTbRB6I= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-338-dSrGA0E5MMiLRsz1VtKreg-1; Thu, 17 Jun 2021 06:58:57 -0400 X-MC-Unique: dSrGA0E5MMiLRsz1VtKreg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B13F7107B0F6; Thu, 17 Jun 2021 10:58:55 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.36.110.45]) by smtp.corp.redhat.com (Postfix) with ESMTP id A3FB661008; Thu, 17 Jun 2021 10:58:54 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, david.marchand@redhat.com, chenbo.xia@intel.com Cc: Maxime Coquelin Date: Thu, 17 Jun 2021 12:58:41 +0200 Message-Id: <20210617105845.125472-4-maxime.coquelin@redhat.com> In-Reply-To: <20210617105845.125472-1-maxime.coquelin@redhat.com> References: <20210617105845.125472-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH 3/7] vhost: fix missing cache logging NUMA realloc X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When the guest allocates virtqueues on a different NUMA node than the one the Vhost metadata are allocated, both the Vhost device struct and the virtqueues struct are reallocated. However, reallocating the log cache on the new NUMA node was not done. This patch fixes this by reallocating it if it has been allocated already, which means a live-migration is on-going. Fixes: 1818a63147fb ("vhost: move dirty logging cache out of virtqueue") Signed-off-by: Maxime Coquelin --- lib/vhost/vhost_user.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index cbfdf1b4d8..0e9e26ebe0 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -545,6 +545,16 @@ numa_realloc(struct virtio_net *dev, int index) vq->batch_copy_elems = new_batch_copy_elems; } + if (vq->log_cache) { + struct log_cache_entry *log_cache; + + log_cache = rte_realloc_socket(vq->log_cache, + sizeof(struct log_cache_entry) * VHOST_LOG_CACHE_NR, + 0, newnode); + if (log_cache) + vq->log_cache = log_cache; + } + rte_free(old_vq); } From patchwork Thu Jun 17 10:58:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 94349 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 448F1A0C4D; Thu, 17 Jun 2021 13:00:59 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8FA8541187; Thu, 17 Jun 2021 12:59:05 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id 3227441185 for ; Thu, 17 Jun 2021 12:59:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623927541; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r1kzLqE55BKPl9kvlE/mAMlmEuwJ0D1b3PHuPSGwcBs=; b=QbGHb3Bq/jWKemTmqjQh3RzLLfe3v2EKOeTnIgwKelMvABwT3bv2uXrfPVNsBZbgGgO57r 6c8MjP0ZWFwo0i59Ul7jhD5rcdEsUmYYTpvgzp/TCMQh3N57cd40r94u5flP3sgPqbNsKY ifvT9PhSdXcifmskCMab9kBk24holQM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-531-gkqHEVTcPdSxDf5q36pnBA-1; Thu, 17 Jun 2021 06:59:00 -0400 X-MC-Unique: gkqHEVTcPdSxDf5q36pnBA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 696E8100C631; Thu, 17 Jun 2021 10:58:57 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.36.110.45]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0C9FA70598; Thu, 17 Jun 2021 10:58:55 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, david.marchand@redhat.com, chenbo.xia@intel.com Cc: Maxime Coquelin , stable@dpdk.org Date: Thu, 17 Jun 2021 12:58:42 +0200 Message-Id: <20210617105845.125472-5-maxime.coquelin@redhat.com> In-Reply-To: <20210617105845.125472-1-maxime.coquelin@redhat.com> References: <20210617105845.125472-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH 4/7] vhost: fix NUMA reallocation with multiqueue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Since the Vhost-user device initialization has been reworked, enabling the application to start using the device as soon as the first queue pair is ready, NUMA reallocation no more happened on queue pairs other than the first one since numa_realloc() was returning early if the device was running. This patch fixes this issue by only preventing the device metadata to be allocated if the device is running. For the virtqueues, a vring state change notification is sent to notify the application of its disablement. Since the callback is supposed to be blocking, it is safe to reallocate it afterwards. Fixes: d0fcc38f5fa4 ("vhost: improve device readiness notifications") Cc: stable@dpdk.org Signed-off-by: Maxime Coquelin --- lib/vhost/vhost_user.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 0e9e26ebe0..6e7b327ef8 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -488,9 +488,6 @@ numa_realloc(struct virtio_net *dev, int index) struct batch_copy_elem *new_batch_copy_elems; int ret; - if (dev->flags & VIRTIO_DEV_RUNNING) - return dev; - old_dev = dev; vq = old_vq = dev->virtqueue[index]; @@ -506,6 +503,11 @@ numa_realloc(struct virtio_net *dev, int index) return dev; } if (oldnode != newnode) { + if (vq->ready) { + vq->ready = false; + vhost_user_notify_queue_state(dev, index, 0); + } + VHOST_LOG_CONFIG(INFO, "reallocate vq from %d to %d node\n", oldnode, newnode); vq = rte_malloc_socket(NULL, sizeof(*vq), 0, newnode); @@ -558,6 +560,9 @@ numa_realloc(struct virtio_net *dev, int index) rte_free(old_vq); } + if (dev->flags & VIRTIO_DEV_RUNNING) + goto out; + /* check if we need to reallocate dev */ ret = get_mempolicy(&oldnode, NULL, 0, old_dev, MPOL_F_NODE | MPOL_F_ADDR); From patchwork Thu Jun 17 10:58:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 94350 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9FAB7A0C4D; Thu, 17 Jun 2021 13:01:05 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DF5D741199; Thu, 17 Jun 2021 12:59:06 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 2320241187 for ; Thu, 17 Jun 2021 12:59:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623927544; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E0+sO/kOVMFWMqSyGAIC6ojuPpfGQFUZYlxpy+BWn+I=; b=dAh8/PgXNlOU4rGZuR7e/LQO/IhSwntS0i7N7/dtuQKwmuFzjgkjjJ0TXuc4JEtMuo/Ahr rc+FwgnKPKFp3Hby8VxEd7w0V5Jah9bnpxMtbKg8IMcln1WJ7emGwrRjQ7a4J55Wn189fx x1zYCrIShtYz1KquZ1ldyrc5+R++9D0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-558-7Dt5qGEeN8ymvTQwArFlQg-1; Thu, 17 Jun 2021 06:59:01 -0400 X-MC-Unique: 7Dt5qGEeN8ymvTQwArFlQg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B5B0380EDB1; Thu, 17 Jun 2021 10:58:58 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.36.110.45]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9AD5D60E3A; Thu, 17 Jun 2021 10:58:57 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, david.marchand@redhat.com, chenbo.xia@intel.com Cc: Maxime Coquelin Date: Thu, 17 Jun 2021 12:58:43 +0200 Message-Id: <20210617105845.125472-6-maxime.coquelin@redhat.com> In-Reply-To: <20210617105845.125472-1-maxime.coquelin@redhat.com> References: <20210617105845.125472-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH 5/7] vhost: improve NUMA reallocation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch improves the numa_realloc() function by making use of rte_realloc_socket(), which takes care of the memory copy and freeing of the old data. Suggested-by: David Marchand Signed-off-by: Maxime Coquelin --- lib/vhost/vhost_user.c | 190 +++++++++++++++++------------------------ 1 file changed, 80 insertions(+), 110 deletions(-) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 6e7b327ef8..50830f0870 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -480,144 +480,114 @@ vhost_user_set_vring_num(struct virtio_net **pdev, static struct virtio_net* numa_realloc(struct virtio_net *dev, int index) { - int oldnode, newnode; + int node; struct virtio_net *old_dev; - struct vhost_virtqueue *old_vq, *vq; - struct vring_used_elem *new_shadow_used_split; - struct vring_used_elem_packed *new_shadow_used_packed; - struct batch_copy_elem *new_batch_copy_elems; + struct vhost_virtqueue *vq; + struct batch_copy_elem *bce; + struct guest_page *gp; + struct rte_vhost_memory *mem; + size_t mem_size; int ret; old_dev = dev; - vq = old_vq = dev->virtqueue[index]; - - ret = get_mempolicy(&newnode, NULL, 0, old_vq->desc, - MPOL_F_NODE | MPOL_F_ADDR); + vq = dev->virtqueue[index]; - /* check if we need to reallocate vq */ - ret |= get_mempolicy(&oldnode, NULL, 0, old_vq, - MPOL_F_NODE | MPOL_F_ADDR); + ret = get_mempolicy(&node, NULL, 0, vq->desc, MPOL_F_NODE | MPOL_F_ADDR); if (ret) { - VHOST_LOG_CONFIG(ERR, - "Unable to get vq numa information.\n"); + VHOST_LOG_CONFIG(ERR, "Unable to get virtqueue %d numa information.\n", index); return dev; } - if (oldnode != newnode) { - if (vq->ready) { - vq->ready = false; - vhost_user_notify_queue_state(dev, index, 0); - } - VHOST_LOG_CONFIG(INFO, - "reallocate vq from %d to %d node\n", oldnode, newnode); - vq = rte_malloc_socket(NULL, sizeof(*vq), 0, newnode); - if (!vq) - return dev; + if (vq->ready) { + vq->ready = false; + vhost_user_notify_queue_state(dev, index, 0); + } + + vq = rte_realloc_socket(vq, sizeof(*vq), 0, node); + if (!vq) { + VHOST_LOG_CONFIG(ERR, "Failed to realloc virtqueue %d on node %d\n", + index, node); + return dev; + } - memcpy(vq, old_vq, sizeof(*vq)); + if (vq != dev->virtqueue[index]) { + VHOST_LOG_CONFIG(INFO, "reallocated virtqueue on node %d\n", node); + dev->virtqueue[index] = vq; + vhost_user_iotlb_init(dev, index); + } - if (vq_is_packed(dev)) { - new_shadow_used_packed = rte_malloc_socket(NULL, - vq->size * - sizeof(struct vring_used_elem_packed), - RTE_CACHE_LINE_SIZE, - newnode); - if (new_shadow_used_packed) { - rte_free(vq->shadow_used_packed); - vq->shadow_used_packed = new_shadow_used_packed; - } - } else { - new_shadow_used_split = rte_malloc_socket(NULL, - vq->size * - sizeof(struct vring_used_elem), - RTE_CACHE_LINE_SIZE, - newnode); - if (new_shadow_used_split) { - rte_free(vq->shadow_used_split); - vq->shadow_used_split = new_shadow_used_split; - } - } + if (vq_is_packed(dev)) { + struct vring_used_elem_packed *sup; - new_batch_copy_elems = rte_malloc_socket(NULL, - vq->size * sizeof(struct batch_copy_elem), - RTE_CACHE_LINE_SIZE, - newnode); - if (new_batch_copy_elems) { - rte_free(vq->batch_copy_elems); - vq->batch_copy_elems = new_batch_copy_elems; + sup = rte_realloc_socket(vq->shadow_used_packed, vq->size * sizeof(*sup), + RTE_CACHE_LINE_SIZE, node); + if (!sup) { + VHOST_LOG_CONFIG(ERR, "Failed to realloc shadow packed on node %d\n", node); + return dev; } + vq->shadow_used_packed = sup; - if (vq->log_cache) { - struct log_cache_entry *log_cache; + } else { + struct vring_used_elem *sus; - log_cache = rte_realloc_socket(vq->log_cache, - sizeof(struct log_cache_entry) * VHOST_LOG_CACHE_NR, - 0, newnode); - if (log_cache) - vq->log_cache = log_cache; + sus = rte_realloc_socket(vq->shadow_used_split, vq->size * sizeof(*sus), + RTE_CACHE_LINE_SIZE, node); + if (!sus) { + VHOST_LOG_CONFIG(ERR, "Failed to realloc shadow split on node %d\n", node); + return dev; } - - rte_free(old_vq); + vq->shadow_used_split = sus; } - if (dev->flags & VIRTIO_DEV_RUNNING) - goto out; - - /* check if we need to reallocate dev */ - ret = get_mempolicy(&oldnode, NULL, 0, old_dev, - MPOL_F_NODE | MPOL_F_ADDR); - if (ret) { - VHOST_LOG_CONFIG(ERR, - "Unable to get dev numa information.\n"); - goto out; + bce = rte_realloc_socket(vq->batch_copy_elems, vq->size * sizeof(*bce), + RTE_CACHE_LINE_SIZE, node); + if (!bce) { + VHOST_LOG_CONFIG(ERR, "Failed to realloc batch copy elem on node %d\n", node); + return dev; } - if (oldnode != newnode) { - struct rte_vhost_memory *old_mem; - struct guest_page *old_gp; - ssize_t mem_size, gp_size; - - VHOST_LOG_CONFIG(INFO, - "reallocate dev from %d to %d node\n", - oldnode, newnode); - dev = rte_malloc_socket(NULL, sizeof(*dev), 0, newnode); - if (!dev) { - dev = old_dev; - goto out; - } + vq->batch_copy_elems = bce; - memcpy(dev, old_dev, sizeof(*dev)); - rte_free(old_dev); + if (vq->log_cache) { + struct log_cache_entry *lc; - mem_size = sizeof(struct rte_vhost_memory) + - sizeof(struct rte_vhost_mem_region) * dev->mem->nregions; - old_mem = dev->mem; - dev->mem = rte_malloc_socket(NULL, mem_size, 0, newnode); - if (!dev->mem) { - dev->mem = old_mem; - goto out; + lc = rte_realloc_socket(vq->log_cache, sizeof(*lc) * VHOST_LOG_CACHE_NR, 0, node); + if (!lc) { + VHOST_LOG_CONFIG(ERR, "Failed to realloc log cache on node %d\n", node); + return dev; } + vq->log_cache = lc; + } - memcpy(dev->mem, old_mem, mem_size); - rte_free(old_mem); + if (dev->flags & VIRTIO_DEV_RUNNING) + return dev; - gp_size = dev->max_guest_pages * sizeof(*dev->guest_pages); - old_gp = dev->guest_pages; - dev->guest_pages = rte_malloc_socket(NULL, gp_size, RTE_CACHE_LINE_SIZE, newnode); - if (!dev->guest_pages) { - dev->guest_pages = old_gp; - goto out; - } + dev = rte_realloc_socket(old_dev, sizeof(*dev), 0, node); + if (!dev) { + VHOST_LOG_CONFIG(ERR, "Failed to realloc dev on node %d\n", node); + return old_dev; + } - memcpy(dev->guest_pages, old_gp, gp_size); - rte_free(old_gp); + if (dev != old_dev) { + VHOST_LOG_CONFIG(INFO, "reallocated device on node %d\n", node); + vhost_devices[dev->vid] = dev; } -out: - dev->virtqueue[index] = vq; - vhost_devices[dev->vid] = dev; + mem_size = sizeof(struct rte_vhost_memory) + + sizeof(struct rte_vhost_mem_region) * dev->mem->nregions; + mem = rte_realloc_socket(dev->mem, mem_size, 0, node); + if (!mem) { + VHOST_LOG_CONFIG(ERR, "Failed to realloc mem table on node %d\n", node); + return dev; + } + dev->mem = mem; - if (old_vq != vq) - vhost_user_iotlb_init(dev, index); + gp = rte_realloc_socket(dev->guest_pages, dev->max_guest_pages * sizeof(*gp), + RTE_CACHE_LINE_SIZE, node); + if (!gp) { + VHOST_LOG_CONFIG(ERR, "Failed to realloc guest pages on node %d\n", node); + return dev; + } + dev->guest_pages = gp; return dev; } From patchwork Thu Jun 17 10:58:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 94351 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0D00BA0C4D; Thu, 17 Jun 2021 13:01:12 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1C1FE41197; Thu, 17 Jun 2021 12:59:08 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id 2237641188 for ; Thu, 17 Jun 2021 12:59:06 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623927546; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pK6n1T0jzCB89w6SPQYzPGr0rHb3MHADJAu3bqhTaqU=; b=ZZPEo5eTIjj9rtqDH0pcVpjOgyvW1zjAdbYbaF5ANYisI22PVC7YI8mVEyBGCs/4DPbLtk flQXZ6d0QT8+9pXiyGSZlP7MG+aUnrpzzNRx6DblkJ+w5nNy76oYnqlzasf5PhtIjR9SaG uAXR99xEK6TfYL6xI1zoZFzIlF/0KkM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-519-SjjNY1-dMcqLxwwTLv5euQ-1; Thu, 17 Jun 2021 06:59:03 -0400 X-MC-Unique: SjjNY1-dMcqLxwwTLv5euQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2686210125B1; Thu, 17 Jun 2021 10:59:00 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.36.110.45]) by smtp.corp.redhat.com (Postfix) with ESMTP id 16FF17012F; Thu, 17 Jun 2021 10:58:58 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, david.marchand@redhat.com, chenbo.xia@intel.com Cc: Maxime Coquelin Date: Thu, 17 Jun 2021 12:58:44 +0200 Message-Id: <20210617105845.125472-7-maxime.coquelin@redhat.com> In-Reply-To: <20210617105845.125472-1-maxime.coquelin@redhat.com> References: <20210617105845.125472-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH 6/7] vhost: allocate all data on same node as virtqueue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch saves the NUMA node the virtqueue is allocated on at init time, in order to allocate all other data on the same node. While most of the data are allocated before numa_realloc() is called and so the data will be reallocated properly, some data like the log cache are most likely allocated after. For the virtio device metadata, we decide to allocate them on the same node as the VQ 0. Signed-off-by: Maxime Coquelin --- lib/vhost/vhost.c | 34 ++++++++++++++++------------------ lib/vhost/vhost.h | 1 + lib/vhost/vhost_user.c | 41 ++++++++++++++++++++++++++++------------- 3 files changed, 45 insertions(+), 31 deletions(-) diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c index c96f6335c8..0000cd3297 100644 --- a/lib/vhost/vhost.c +++ b/lib/vhost/vhost.c @@ -261,7 +261,7 @@ vhost_alloc_copy_ind_table(struct virtio_net *dev, struct vhost_virtqueue *vq, uint64_t src, dst; uint64_t len, remain = desc_len; - idesc = rte_malloc(__func__, desc_len, 0); + idesc = rte_malloc_socket(__func__, desc_len, 0, vq->numa_node); if (unlikely(!idesc)) return NULL; @@ -549,6 +549,7 @@ static void init_vring_queue(struct virtio_net *dev, uint32_t vring_idx) { struct vhost_virtqueue *vq; + int numa_node = SOCKET_ID_ANY; if (vring_idx >= VHOST_MAX_VRING) { VHOST_LOG_CONFIG(ERR, @@ -570,6 +571,15 @@ init_vring_queue(struct virtio_net *dev, uint32_t vring_idx) vq->callfd = VIRTIO_UNINITIALIZED_EVENTFD; vq->notif_enable = VIRTIO_UNINITIALIZED_NOTIF; +#ifdef RTE_LIBRTE_VHOST_NUMA + if (get_mempolicy(&numa_node, NULL, 0, vq, MPOL_F_NODE | MPOL_F_ADDR)) { + VHOST_LOG_CONFIG(ERR, "(%d) failed to query numa node: %s\n", + dev->vid, rte_strerror(errno)); + numa_node = SOCKET_ID_ANY; + } +#endif + vq->numa_node = numa_node; + vhost_user_iotlb_init(dev, vring_idx); } @@ -1616,7 +1626,6 @@ int rte_vhost_async_channel_register(int vid, uint16_t queue_id, struct vhost_virtqueue *vq; struct virtio_net *dev = get_device(vid); struct rte_vhost_async_features f; - int node; if (dev == NULL || ops == NULL) return -1; @@ -1651,20 +1660,9 @@ int rte_vhost_async_channel_register(int vid, uint16_t queue_id, goto reg_out; } -#ifdef RTE_LIBRTE_VHOST_NUMA - if (get_mempolicy(&node, NULL, 0, vq, MPOL_F_NODE | MPOL_F_ADDR)) { - VHOST_LOG_CONFIG(ERR, - "unable to get numa information in async register. " - "allocating async buffer memory on the caller thread node\n"); - node = SOCKET_ID_ANY; - } -#else - node = SOCKET_ID_ANY; -#endif - vq->async_pkts_info = rte_malloc_socket(NULL, vq->size * sizeof(struct async_inflight_info), - RTE_CACHE_LINE_SIZE, node); + RTE_CACHE_LINE_SIZE, vq->numa_node); if (!vq->async_pkts_info) { vhost_free_async_mem(vq); VHOST_LOG_CONFIG(ERR, @@ -1675,7 +1673,7 @@ int rte_vhost_async_channel_register(int vid, uint16_t queue_id, vq->it_pool = rte_malloc_socket(NULL, VHOST_MAX_ASYNC_IT * sizeof(struct rte_vhost_iov_iter), - RTE_CACHE_LINE_SIZE, node); + RTE_CACHE_LINE_SIZE, vq->numa_node); if (!vq->it_pool) { vhost_free_async_mem(vq); VHOST_LOG_CONFIG(ERR, @@ -1686,7 +1684,7 @@ int rte_vhost_async_channel_register(int vid, uint16_t queue_id, vq->vec_pool = rte_malloc_socket(NULL, VHOST_MAX_ASYNC_VEC * sizeof(struct iovec), - RTE_CACHE_LINE_SIZE, node); + RTE_CACHE_LINE_SIZE, vq->numa_node); if (!vq->vec_pool) { vhost_free_async_mem(vq); VHOST_LOG_CONFIG(ERR, @@ -1698,7 +1696,7 @@ int rte_vhost_async_channel_register(int vid, uint16_t queue_id, if (vq_is_packed(dev)) { vq->async_buffers_packed = rte_malloc_socket(NULL, vq->size * sizeof(struct vring_used_elem_packed), - RTE_CACHE_LINE_SIZE, node); + RTE_CACHE_LINE_SIZE, vq->numa_node); if (!vq->async_buffers_packed) { vhost_free_async_mem(vq); VHOST_LOG_CONFIG(ERR, @@ -1709,7 +1707,7 @@ int rte_vhost_async_channel_register(int vid, uint16_t queue_id, } else { vq->async_descs_split = rte_malloc_socket(NULL, vq->size * sizeof(struct vring_used_elem), - RTE_CACHE_LINE_SIZE, node); + RTE_CACHE_LINE_SIZE, vq->numa_node); if (!vq->async_descs_split) { vhost_free_async_mem(vq); VHOST_LOG_CONFIG(ERR, diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index 8078ddff79..8ffe387556 100644 --- a/lib/vhost/vhost.h +++ b/lib/vhost/vhost.h @@ -164,6 +164,7 @@ struct vhost_virtqueue { uint16_t batch_copy_nb_elems; struct batch_copy_elem *batch_copy_elems; + int numa_node; bool used_wrap_counter; bool avail_wrap_counter; diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 50830f0870..81615cc145 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -433,10 +433,10 @@ vhost_user_set_vring_num(struct virtio_net **pdev, if (vq_is_packed(dev)) { if (vq->shadow_used_packed) rte_free(vq->shadow_used_packed); - vq->shadow_used_packed = rte_malloc(NULL, + vq->shadow_used_packed = rte_malloc_socket(NULL, vq->size * sizeof(struct vring_used_elem_packed), - RTE_CACHE_LINE_SIZE); + RTE_CACHE_LINE_SIZE, vq->numa_node); if (!vq->shadow_used_packed) { VHOST_LOG_CONFIG(ERR, "failed to allocate memory for shadow used ring.\n"); @@ -447,9 +447,9 @@ vhost_user_set_vring_num(struct virtio_net **pdev, if (vq->shadow_used_split) rte_free(vq->shadow_used_split); - vq->shadow_used_split = rte_malloc(NULL, + vq->shadow_used_split = rte_malloc_socket(NULL, vq->size * sizeof(struct vring_used_elem), - RTE_CACHE_LINE_SIZE); + RTE_CACHE_LINE_SIZE, vq->numa_node); if (!vq->shadow_used_split) { VHOST_LOG_CONFIG(ERR, @@ -460,9 +460,9 @@ vhost_user_set_vring_num(struct virtio_net **pdev, if (vq->batch_copy_elems) rte_free(vq->batch_copy_elems); - vq->batch_copy_elems = rte_malloc(NULL, + vq->batch_copy_elems = rte_malloc_socket(NULL, vq->size * sizeof(struct batch_copy_elem), - RTE_CACHE_LINE_SIZE); + RTE_CACHE_LINE_SIZE, vq->numa_node); if (!vq->batch_copy_elems) { VHOST_LOG_CONFIG(ERR, "failed to allocate memory for batching copy.\n"); @@ -498,6 +498,9 @@ numa_realloc(struct virtio_net *dev, int index) return dev; } + if (node == vq->numa_node) + goto out_dev_realloc; + if (vq->ready) { vq->ready = false; vhost_user_notify_queue_state(dev, index, 0); @@ -558,6 +561,10 @@ numa_realloc(struct virtio_net *dev, int index) vq->log_cache = lc; } + vq->numa_node = node; + +out_dev_realloc: + if (dev->flags & VIRTIO_DEV_RUNNING) return dev; @@ -1205,7 +1212,7 @@ vhost_user_set_mem_table(struct virtio_net **pdev, struct VhostUserMsg *msg, struct virtio_net *dev = *pdev; struct VhostUserMemory *memory = &msg->payload.memory; struct rte_vhost_mem_region *reg; - + int numa_node = SOCKET_ID_ANY; uint64_t mmap_offset; uint32_t i; @@ -1245,13 +1252,21 @@ vhost_user_set_mem_table(struct virtio_net **pdev, struct VhostUserMsg *msg, for (i = 0; i < dev->nr_vring; i++) vhost_user_iotlb_flush_all(dev->virtqueue[i]); + /* + * If VQ 0 has already been allocated, try to allocate on the same + * NUMA node. It can be reallocated later in numa_realloc(). + */ + if (dev->nr_vring > 0) + numa_node = dev->virtqueue[0]->numa_node; + dev->nr_guest_pages = 0; if (dev->guest_pages == NULL) { dev->max_guest_pages = 8; - dev->guest_pages = rte_zmalloc(NULL, + dev->guest_pages = rte_zmalloc_socket(NULL, dev->max_guest_pages * sizeof(struct guest_page), - RTE_CACHE_LINE_SIZE); + RTE_CACHE_LINE_SIZE, + numa_node); if (dev->guest_pages == NULL) { VHOST_LOG_CONFIG(ERR, "(%d) failed to allocate memory " @@ -1261,8 +1276,8 @@ vhost_user_set_mem_table(struct virtio_net **pdev, struct VhostUserMsg *msg, } } - dev->mem = rte_zmalloc("vhost-mem-table", sizeof(struct rte_vhost_memory) + - sizeof(struct rte_vhost_mem_region) * memory->nregions, 0); + dev->mem = rte_zmalloc_socket("vhost-mem-table", sizeof(struct rte_vhost_memory) + + sizeof(struct rte_vhost_mem_region) * memory->nregions, 0, numa_node); if (dev->mem == NULL) { VHOST_LOG_CONFIG(ERR, "(%d) failed to allocate memory for dev->mem\n", @@ -2185,9 +2200,9 @@ vhost_user_set_log_base(struct virtio_net **pdev, struct VhostUserMsg *msg, rte_free(vq->log_cache); vq->log_cache = NULL; vq->log_cache_nb_elem = 0; - vq->log_cache = rte_zmalloc("vq log cache", + vq->log_cache = rte_malloc_socket("vq log cache", sizeof(struct log_cache_entry) * VHOST_LOG_CACHE_NR, - 0); + 0, vq->numa_node); /* * If log cache alloc fail, don't fail migration, but no * caching will be done, which will impact performance From patchwork Thu Jun 17 10:58:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Coquelin X-Patchwork-Id: 94352 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6A11BA0C4D; Thu, 17 Jun 2021 13:01:19 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A7008411A6; Thu, 17 Jun 2021 12:59:09 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id 6695E41197 for ; Thu, 17 Jun 2021 12:59:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623927546; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3J6ECgBC0uP2hje4S1KT0m4NcuVO0AaxybjnqjvshOI=; b=MHxVB/9j0uZf5nTjKxPVou+EmyTGSBFSs4FVvSXSYXIiW+Izq7E+gWnbuFHMUWifxPdeMy yWyiiS0CPXOCL3cMKjvXzoGYvZETvsC9JureRfBVp3QAGhwuud5jDgB31m/fSzOcNtmDS8 rZF4SKVfHUzWY86nPYvpRcpdTESU1Is= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-367-oF4j5GBtPuWqPnKSL5l2mA-1; Thu, 17 Jun 2021 06:59:05 -0400 X-MC-Unique: oF4j5GBtPuWqPnKSL5l2mA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AFAFD1084F7B; Thu, 17 Jun 2021 10:59:01 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.36.110.45]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6EA3170592; Thu, 17 Jun 2021 10:59:00 +0000 (UTC) From: Maxime Coquelin To: dev@dpdk.org, david.marchand@redhat.com, chenbo.xia@intel.com Cc: Maxime Coquelin Date: Thu, 17 Jun 2021 12:58:45 +0200 Message-Id: <20210617105845.125472-8-maxime.coquelin@redhat.com> In-Reply-To: <20210617105845.125472-1-maxime.coquelin@redhat.com> References: <20210617105845.125472-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [dpdk-dev] [PATCH 7/7] vhost: convert inflight data to DPDK allocation API X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Inflight metadata are allocated using glibc's calloc. This patch converts them to rte_zmalloc_socket to take care of the NUMA affinity. Signed-off-by: Maxime Coquelin --- lib/vhost/vhost.c | 4 +-- lib/vhost/vhost_user.c | 67 +++++++++++++++++++++++++++++++++++------- 2 files changed, 58 insertions(+), 13 deletions(-) diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c index 0000cd3297..53a470f547 100644 --- a/lib/vhost/vhost.c +++ b/lib/vhost/vhost.c @@ -312,10 +312,10 @@ cleanup_vq_inflight(struct virtio_net *dev, struct vhost_virtqueue *vq) if (vq->resubmit_inflight) { if (vq->resubmit_inflight->resubmit_list) { - free(vq->resubmit_inflight->resubmit_list); + rte_free(vq->resubmit_inflight->resubmit_list); vq->resubmit_inflight->resubmit_list = NULL; } - free(vq->resubmit_inflight); + rte_free(vq->resubmit_inflight); vq->resubmit_inflight = NULL; } } diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 81615cc145..d972073690 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -188,7 +188,7 @@ vhost_backend_cleanup(struct virtio_net *dev) dev->inflight_info->fd = -1; } - free(dev->inflight_info); + rte_free(dev->inflight_info); dev->inflight_info = NULL; } @@ -561,6 +561,31 @@ numa_realloc(struct virtio_net *dev, int index) vq->log_cache = lc; } + if (vq->resubmit_inflight) { + struct rte_vhost_resubmit_info *ri; + + ri = rte_realloc_socket(vq->resubmit_inflight, sizeof(*ri), 0, node); + if (!ri) { + VHOST_LOG_CONFIG(ERR, "Failed to realloc resubmit inflight on node %d\n", + node); + return dev; + } + vq->resubmit_inflight = ri; + + if (ri->resubmit_list) { + struct rte_vhost_resubmit_desc *rd; + + rd = rte_realloc_socket(ri->resubmit_list, sizeof(*rd) * ri->resubmit_num, + 0, node); + if (!rd) { + VHOST_LOG_CONFIG(ERR, "Failed to realloc resubmit list on node %d\n", + node); + return dev; + } + ri->resubmit_list = rd; + } + } + vq->numa_node = node; out_dev_realloc: @@ -1483,6 +1508,7 @@ vhost_user_get_inflight_fd(struct virtio_net **pdev, uint16_t num_queues, queue_size; struct virtio_net *dev = *pdev; int fd, i, j; + int numa_node = SOCKET_ID_ANY; void *addr; if (msg->size != sizeof(msg->payload.inflight)) { @@ -1492,9 +1518,16 @@ vhost_user_get_inflight_fd(struct virtio_net **pdev, return RTE_VHOST_MSG_RESULT_ERR; } + /* + * If VQ 0 has already been allocated, try to allocate on the same + * NUMA node. It can be reallocated later in numa_realloc(). + */ + if (dev->nr_vring > 0) + numa_node = dev->virtqueue[0]->numa_node; + if (dev->inflight_info == NULL) { - dev->inflight_info = calloc(1, - sizeof(struct inflight_mem_info)); + dev->inflight_info = rte_zmalloc_socket("inflight_info", + sizeof(struct inflight_mem_info), 0, numa_node); if (!dev->inflight_info) { VHOST_LOG_CONFIG(ERR, "failed to alloc dev inflight area\n"); @@ -1577,6 +1610,7 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev, VhostUserMsg *msg, struct vhost_virtqueue *vq; void *addr; int fd, i; + int numa_node = SOCKET_ID_ANY; fd = msg->fds[0]; if (msg->size != sizeof(msg->payload.inflight) || fd < 0) { @@ -1610,9 +1644,16 @@ vhost_user_set_inflight_fd(struct virtio_net **pdev, VhostUserMsg *msg, "set_inflight_fd pervq_inflight_size: %d\n", pervq_inflight_size); + /* + * If VQ 0 has already been allocated, try to allocate on the same + * NUMA node. It can be reallocated later in numa_realloc(). + */ + if (dev->nr_vring > 0) + numa_node = dev->virtqueue[0]->numa_node; + if (!dev->inflight_info) { - dev->inflight_info = calloc(1, - sizeof(struct inflight_mem_info)); + dev->inflight_info = rte_zmalloc_socket("inflight_info", + sizeof(struct inflight_mem_info), 0, numa_node); if (dev->inflight_info == NULL) { VHOST_LOG_CONFIG(ERR, "failed to alloc dev inflight area\n"); @@ -1771,15 +1812,17 @@ vhost_check_queue_inflights_split(struct virtio_net *dev, vq->last_avail_idx += resubmit_num; if (resubmit_num) { - resubmit = calloc(1, sizeof(struct rte_vhost_resubmit_info)); + resubmit = rte_zmalloc_socket("resubmit", sizeof(struct rte_vhost_resubmit_info), + 0, vq->numa_node); if (!resubmit) { VHOST_LOG_CONFIG(ERR, "failed to allocate memory for resubmit info.\n"); return RTE_VHOST_MSG_RESULT_ERR; } - resubmit->resubmit_list = calloc(resubmit_num, - sizeof(struct rte_vhost_resubmit_desc)); + resubmit->resubmit_list = rte_zmalloc_socket("resubmit_list", + resubmit_num * sizeof(struct rte_vhost_resubmit_desc), + 0, vq->numa_node); if (!resubmit->resubmit_list) { VHOST_LOG_CONFIG(ERR, "failed to allocate memory for inflight desc.\n"); @@ -1865,15 +1908,17 @@ vhost_check_queue_inflights_packed(struct virtio_net *dev, } if (resubmit_num) { - resubmit = calloc(1, sizeof(struct rte_vhost_resubmit_info)); + resubmit = rte_zmalloc_socket("resubmit", sizeof(struct rte_vhost_resubmit_info), + 0, vq->numa_node); if (resubmit == NULL) { VHOST_LOG_CONFIG(ERR, "failed to allocate memory for resubmit info.\n"); return RTE_VHOST_MSG_RESULT_ERR; } - resubmit->resubmit_list = calloc(resubmit_num, - sizeof(struct rte_vhost_resubmit_desc)); + resubmit->resubmit_list = rte_zmalloc_socket("resubmit_list", + resubmit_num * sizeof(struct rte_vhost_resubmit_desc), + 0, vq->numa_node); if (resubmit->resubmit_list == NULL) { VHOST_LOG_CONFIG(ERR, "failed to allocate memory for resubmit desc.\n");