From patchwork Fri Sep 16 06:16:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116383 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3BC0DA0032; Fri, 16 Sep 2022 08:17:06 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DB95E427F1; Fri, 16 Sep 2022 08:17:03 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 9067F40DFD for ; Fri, 16 Sep 2022 08:17:01 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663309021; x=1694845021; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=QTKIFjFBCmAo3WXghn1z7hr6ODsRiM4i3BtiRZiewZU=; b=QcMmHat4UcvtAogHZqGwU8bWKVWEFMIFclVCLBCHcY6aRScHaIweUJvF MM3m8T+Mwyvf/yapgi8KIUQPoOC+AJxMDzMnA6ry2PSM6tYXokX0QED9e wxVuNCI8t9NFNkOiq85ntIacET95pPBNq5MUuQiIAJEdjBqQQQOyXA5id +CsxbP9Dtfc6S/jrD/ZmhHUXmHLbt6dL4mPefM6Fn0C1vz6DFz1sKF9/6 dhCrHqoeai1ZzGlTQ8KQ6qUcmrMzsKHO7s8mMPxzkFlmLyql0uyLSXlZG ecZQCP2qSboR6PKcDVNp9JwZhr5kFhqvjylxEdTKRuKA6/LCN86lp9uAA Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10471"; a="278650900" X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="278650900" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Sep 2022 23:16:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="568724513" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by orsmga003.jf.intel.com with ESMTP; 15 Sep 2022 23:16:27 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v3 1/8] vdpa/ifc: add new device ID for legacy network device Date: Fri, 16 Sep 2022 14:16:23 +0800 Message-Id: <1663308990-621-2-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1663308990-621-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1663308990-621-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Huang Wei Add new device id to support IFCVF_NET_TRANSITIONAL_DEVICE_ID (0x1000). Rename macro from "IFCVF_BLK_DEVICE_ID" to "IFCVF_SUBSYS_BLK_DEVICE_ID". Signed-off-by: Huang Wei Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.h | 6 ++++-- drivers/vdpa/ifc/ifcvf_vdpa.c | 13 ++++++++++--- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 9d95aac..ef7697a 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -12,12 +12,14 @@ #define IFCVF_BLK 1 #define IFCVF_VENDOR_ID 0x1AF4 -#define IFCVF_NET_DEVICE_ID 0x1041 +#define IFCVF_NET_MODERN_DEVICE_ID 0x1041 #define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_NET_TRANSITIONAL_DEVICE_ID 0x1000 #define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 #define IFCVF_SUBSYS_VENDOR_ID 0x8086 #define IFCVF_SUBSYS_DEVICE_ID 0x001A -#define IFCVF_BLK_DEVICE_ID 0x0002 +#define IFCVF_SUBSYS_NET_DEVICE_ID 0x0001 +#define IFCVF_SUBSYS_BLK_DEVICE_ID 0x0002 #define IFCVF_MAX_QUEUES 1 diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index ac42de9..07cc63d 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1684,23 +1684,30 @@ struct rte_vdpa_dev_info dev_info[] = { static const struct rte_pci_id pci_id_ifcvf_map[] = { { .class_id = RTE_CLASS_ANY_ID, .vendor_id = IFCVF_VENDOR_ID, - .device_id = IFCVF_NET_DEVICE_ID, + .device_id = IFCVF_NET_MODERN_DEVICE_ID, .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, { .class_id = RTE_CLASS_ANY_ID, .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_NET_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_SUBSYS_NET_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, - .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + .subsystem_device_id = IFCVF_SUBSYS_BLK_DEVICE_ID, }, { .class_id = RTE_CLASS_ANY_ID, .vendor_id = IFCVF_VENDOR_ID, .device_id = IFCVF_BLK_MODERN_DEVICE_ID, .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, - .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + .subsystem_device_id = IFCVF_SUBSYS_BLK_DEVICE_ID, }, { .vendor_id = 0, /* sentinel */ From patchwork Fri Sep 16 06:16:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116384 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9C8B5A0032; Fri, 16 Sep 2022 08:17:10 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E51AC4280D; Fri, 16 Sep 2022 08:17:06 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 48D83427EE for ; Fri, 16 Sep 2022 08:17:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663309023; x=1694845023; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=BV2y/XNmjxjn/1JnW3prM9s1CMZkSf8tdWtScQT4Ojs=; b=OiK811MdRUXOfiJylZtCIri5mAt9jq2D0k8SYRbLq6B6Zy1KlwIB7CD1 IgOPRi1nMp7agYgRdjn562HogKksvp++b5nVr6pzLtfB5FN0An8My11p5 ayBt4Oja+fVts18Xq7xXmoBjkT5lhVGbc8ybgFakAbbDSN+KO/G7d3CUB xrslNqudkg1hfw32XszGRi9t6050mkvP0VTmsgnDDTlVVJttDjlZDLVzu 7t/ZTCdSa5xr+YYvoE6WWrde4svags2kCn+N5o6Wu5r7ldiA9UH2ok+KG MX0BGY9Zy+wS22alKS23wpX0eq5dugatYU27CH1TVM+fgQ2+5W9Nt9rvp w==; X-IronPort-AV: E=McAfee;i="6500,9779,10471"; a="278650932" X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="278650932" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Sep 2022 23:16:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="568724548" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by orsmga003.jf.intel.com with ESMTP; 15 Sep 2022 23:16:30 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v3 2/8] vdpa/ifc: add multi-queue support Date: Fri, 16 Sep 2022 14:16:24 +0800 Message-Id: <1663308990-621-3-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1663308990-621-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1663308990-621-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enable VHOST_USER_PROTOCOL_F_MQ feature. Expose IFCVF_MQ_OFFSET register to enable multi queue. Signed-off-by: Andy Pei Signed-off-by: Huang Wei --- drivers/vdpa/ifc/base/ifcvf.c | 5 +++++ drivers/vdpa/ifc/base/ifcvf.h | 2 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 1 + 3 files changed, 8 insertions(+) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index f1e1474..34c8226 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -90,6 +90,11 @@ if (!hw->lm_cfg) WARNINGOUT("HW support live migration not support!\n"); + if (hw->mem_resource[4].addr) + hw->mq_cfg = hw->mem_resource[4].addr + IFCVF_MQ_OFFSET; + else + hw->mq_cfg = NULL; + if (hw->common_cfg == NULL || hw->notify_base == NULL || hw->isr == NULL || hw->dev_cfg == NULL) { DEBUGOUT("capability incomplete\n"); diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index ef7697a..d16d9ab 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -50,6 +50,7 @@ #define IFCVF_LM_CFG_SIZE 0x40 #define IFCVF_LM_RING_STATE_OFFSET 0x20 +#define IFCVF_MQ_OFFSET 0x28 #define IFCVF_LM_LOGGING_CTRL 0x0 @@ -149,6 +150,7 @@ struct ifcvf_hw { u16 *notify_base; u16 *notify_addr[IFCVF_MAX_QUEUES * 2]; u8 *lm_cfg; + u8 *mq_cfg; struct vring_info vring[IFCVF_MAX_QUEUES * 2]; u8 nr_vring; int device_type; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 07cc63d..3e5ffba 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1248,6 +1248,7 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD | \ 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ + 1ULL << VHOST_USER_PROTOCOL_F_MQ | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) #define VDPA_BLK_PROTOCOL_FEATURES \ From patchwork Fri Sep 16 06:16:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116385 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E51F1A0032; Fri, 16 Sep 2022 08:17:15 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 001DA42826; Fri, 16 Sep 2022 08:17:07 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 3304E427F7 for ; Fri, 16 Sep 2022 08:17:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663309024; x=1694845024; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=kMSHcYV7EuNl9w7jvbLWU83QFbAAWPdqpRkAqMVpxP0=; b=kP2N2CGo7NAauHUh2bi6HjViY8U362S2u2sfmPmqSjN5chnNj6P4kU21 tzwb3a5onnRUDF0BW/IpeD84mcauCUAafEAViSLP1qhswwBGMvlyeru7G pPhmTsNHcuQLhAnojURTmlSfISjaq5gOjNl/FVRUEDhN97P0McGnIgpQT KgJ61ranE1cV/fMKW3N2D0qrq/FX0A9OQ1doNldoHiYFM9GGbaXnHOe5Y QUimEVFYuHFuOwqOmFdssZ6qFebDHwe3nBM5BDN8c7BAF3i7iKeKwAQJF yD5FCC0P8ecraQ8XeZ7F4130XlZKSU++oqe3RQrv4KTRN01gSUy1axUqZ A==; X-IronPort-AV: E=McAfee;i="6500,9779,10471"; a="278650944" X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="278650944" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Sep 2022 23:16:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="568724569" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by orsmga003.jf.intel.com with ESMTP; 15 Sep 2022 23:16:33 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v3 3/8] vdpa/ifc: set max queues based on virtio spec Date: Fri, 16 Sep 2022 14:16:25 +0800 Message-Id: <1663308990-621-4-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1663308990-621-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1663308990-621-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Set max_queues according to virtio HW spec. For virtio BLK device, set max_queues to the value of "num_queues". "num_queues" is element of struct virtio_blk_config. For virtio NET device, read num_queues from truct ifcvf_pci_common_cfg, calculate "queue_pairs = (num_queues - 1) / 2" and get queue_pairs. Set max_queues to the value of "queue_pairs". Signed-off-by: Andy Pei Signed-off-by: Huang Wei --- drivers/vdpa/ifc/base/ifcvf.h | 2 +- drivers/vdpa/ifc/ifcvf_vdpa.c | 19 ++++++++++++++++++- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index d16d9ab..1e133c0 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -21,7 +21,7 @@ #define IFCVF_SUBSYS_NET_DEVICE_ID 0x0001 #define IFCVF_SUBSYS_BLK_DEVICE_ID 0x0002 -#define IFCVF_MAX_QUEUES 1 +#define IFCVF_MAX_QUEUES 32 #ifndef VIRTIO_F_IOMMU_PLATFORM #define VIRTIO_F_IOMMU_PLATFORM 33 diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 3e5ffba..376239a 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -26,6 +26,12 @@ #include "base/ifcvf.h" +/* + * RTE_MIN() cannot be used since braced-group within expression allowed + * only inside a function. + */ +#define MIN(v1, v2) ((v1) < (v2) ? (v1) : (v2)) + RTE_LOG_REGISTER(ifcvf_vdpa_logtype, pmd.vdpa.ifcvf, NOTICE); #define DRV_LOG(level, fmt, args...) \ rte_log(RTE_LOG_ ## level, ifcvf_vdpa_logtype, \ @@ -1512,6 +1518,7 @@ struct rte_vdpa_dev_info dev_info[] = { uint64_t capacity = 0; uint8_t *byte; uint32_t i; + uint16_t queue_pairs; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1559,7 +1566,6 @@ struct rte_vdpa_dev_info dev_info[] = { } internal->configured = 0; - internal->max_queues = IFCVF_MAX_QUEUES; features = ifcvf_get_features(&internal->hw); device_id = ifcvf_pci_get_device_type(pci_dev); @@ -1570,6 +1576,14 @@ struct rte_vdpa_dev_info dev_info[] = { if (device_id == VIRTIO_ID_NET) { internal->hw.device_type = IFCVF_NET; + /* + * ifc driver always has CTRL_VQ, + * and supports VIRTIO_NET_F_CTRL_VQ feature. + */ + queue_pairs = (internal->hw.common_cfg->num_queues - 1) / 2; + DRV_LOG(INFO, "%s support %u queue pairs", pci_dev->name, + queue_pairs); + internal->max_queues = MIN(IFCVF_MAX_QUEUES, queue_pairs); internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_NET].features; @@ -1609,6 +1623,9 @@ struct rte_vdpa_dev_info dev_info[] = { internal->hw.blk_cfg->geometry.sectors); DRV_LOG(DEBUG, "num_queues: 0x%08x", internal->hw.blk_cfg->num_queues); + + internal->max_queues = MIN(IFCVF_MAX_QUEUES, + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Fri Sep 16 06:16:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116386 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3D32AA0032; Fri, 16 Sep 2022 08:17:21 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 112394282E; Fri, 16 Sep 2022 08:17:09 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id D664F427F7 for ; Fri, 16 Sep 2022 08:17:04 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663309025; x=1694845025; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Cr+274HMhI8uCWY9FQjFva1ZlzsPTCgLAwj6QRFecMg=; b=V2/mgg3MJkfun4x1WBok68sQUR1Gd2mEpzxpuP6A93S5F6FGkuNpomjv 8zIImq5uiYKk2K+0u0AYxqn3Snz6EECyxSORMGP/1+CPdsd4tT/8zaHbY sSEfdmIpWnMxPUQ4bZhZWB5iG74xKGuaRZ8oZWVmbY7n5MYQJ5IzyQMS3 NzphYrjb0kWrJAricnC65STqTGEwAuSS7z4uphZA+DE8jPrQPCbgmTvFl L2AYt47iwOlEExUUMcan7snKAB+wIn51u5TehUTVre/uw89nUnDp7s/vq 1hnzXeU2q9DsjpkFGkRQNjPW/U48fuwOo7bJGGzNnuO9knTtpW/tB+mU8 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10471"; a="278650950" X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="278650950" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Sep 2022 23:16:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="568724596" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by orsmga003.jf.intel.com with ESMTP; 15 Sep 2022 23:16:36 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v3 4/8] vdpa/ifc: write queue count to MQ register Date: Fri, 16 Sep 2022 14:16:26 +0800 Message-Id: <1663308990-621-5-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1663308990-621-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1663308990-621-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Write queue count to IFCVF_MQ_OFFSET register to enable multi queue feature. Signed-off-by: Andy Pei Signed-off-by: Huang Wei --- drivers/vdpa/ifc/base/ifcvf.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 34c8226..7efb408 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -198,6 +198,35 @@ IFCVF_WRITE_REG32(val >> 32, hi); } +STATIC void +ifcvf_enable_multiqueue(struct ifcvf_hw *hw) +{ + u8 *mq_cfg; + int qid; + int nr_queue_pair = 0; + + for (qid = 0; qid < hw->nr_vring; qid++) { + if (!hw->vring[qid].enable) + continue; + nr_queue_pair++; + } + + if (nr_queue_pair == 0) { + WARNINGOUT("no enabled vring\n"); + return; + } + + if (hw->device_type == IFCVF_NET) + nr_queue_pair = (nr_queue_pair + 1) / 2; + + mq_cfg = hw->mq_cfg; + if (mq_cfg) { + *(u32 *)mq_cfg = nr_queue_pair; + RTE_LOG(INFO, PMD, "%d queue pairs are enabled\n", + nr_queue_pair); + } +} + STATIC int ifcvf_hw_enable(struct ifcvf_hw *hw) { @@ -215,6 +244,7 @@ return -1; } + ifcvf_enable_multiqueue(hw); for (i = 0; i < hw->nr_vring; i++) { IFCVF_WRITE_REG16(i, &cfg->queue_select); io_write64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo, From patchwork Fri Sep 16 06:16:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116387 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1097BA0032; Fri, 16 Sep 2022 08:17:26 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1B69942905; Fri, 16 Sep 2022 08:17:10 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 44F97427FF for ; Fri, 16 Sep 2022 08:17:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663309025; x=1694845025; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=84JcuC9nEOG9R2MUbPR9mej/TtQ8fCOuxSx6zuiqGqw=; b=S9AmPhBe0DThWGoYhAHVZjRJfbMLKn0VRWL0Ewk2suGAzSV4xHM2oCU5 Yd1KloWIuKVs4GpbXpohex0JcVxunefMljMwmlM01ZT/fzyBIjLrlZLjR irIepFt3a9da+5yHvPl45SdA4nmuD0jJOE6KfY7P2SirNv8cl7NjOdY3h 2tO+BmNgeBHzi0TnqzVxE9sAbD6UK498kBtee4l19yISCwaeiwmYUO0PE /XN5JjpQQ6h587mUlM9Jpk2yEcF7PFNqGAPoXVQjzHT3h3kt+d0X+PEY9 ZX9LThKvD6xLBYcbUmt2SyB4HwEl6q0OQUMKq/KvmvvNIyfzB45CprcSR g==; X-IronPort-AV: E=McAfee;i="6500,9779,10471"; a="278650960" X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="278650960" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Sep 2022 23:16:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="568724639" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by orsmga003.jf.intel.com with ESMTP; 15 Sep 2022 23:16:38 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v3 5/8] vdpa/ifc: only configure enabled queue Date: Fri, 16 Sep 2022 14:16:27 +0800 Message-Id: <1663308990-621-6-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1663308990-621-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1663308990-621-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org when configure the hardware queue, we only configure queues which have been enabled by vhost. Signed-off-by: Andy Pei Signed-off-by: Huang Wei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.c | 3 +++ drivers/vdpa/ifc/ifcvf_vdpa.c | 16 ++++++++++++++-- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 7efb408..619b034 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -246,6 +246,9 @@ ifcvf_enable_multiqueue(hw); for (i = 0; i < hw->nr_vring; i++) { + if (!hw->vring[i].enable) + continue; + IFCVF_WRITE_REG16(i, &cfg->queue_select); io_write64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo, &cfg->queue_desc_hi); diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 376239a..b00afdb 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -284,6 +284,8 @@ struct rte_vdpa_dev_info { rte_vhost_get_negotiated_features(vid, &hw->req_features); for (i = 0; i < nr_vring; i++) { + if (!hw->vring[i].enable) + continue; rte_vhost_get_vhost_vring(vid, i, &vq); gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.desc); if (gpa == 0) { @@ -499,6 +501,8 @@ struct rte_vdpa_dev_info { vring.kickfd = -1; for (qid = 0; qid < q_num; qid++) { + if (!hw->vring[qid].enable) + continue; ev.events = EPOLLIN | EPOLLPRI; rte_vhost_get_vhost_vring(internal->vid, qid, &vring); ev.data.u64 = qid | (uint64_t)vring.kickfd << 32; @@ -1058,6 +1062,8 @@ struct rte_vdpa_dev_info { struct rte_vdpa_device *vdev; struct internal_list *list; struct ifcvf_internal *internal; + struct ifcvf_hw *hw; + uint16_t i; vdev = rte_vhost_get_vdpa_device(vid); list = find_internal_resource_by_vdev(vdev); @@ -1071,11 +1077,17 @@ struct rte_vdpa_dev_info { rte_atomic32_set(&internal->dev_attached, 1); update_datapath(internal); - if (rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, true) != 0) - DRV_LOG(NOTICE, "vDPA (%s): software relay is used.", + hw = &internal->hw; + for (i = 0; i < hw->nr_vring; i++) { + if (!hw->vring[i].enable) + continue; + if (rte_vhost_host_notifier_ctrl(vid, i, true) != 0) + DRV_LOG(NOTICE, "vDPA (%s): software relay is used.", vdev->device->name); + } internal->configured = 1; + DRV_LOG(INFO, "vDPA device %s is configured", vdev->device->name); return 0; } From patchwork Fri Sep 16 06:16:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116388 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6A1D4A0032; Fri, 16 Sep 2022 08:17:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0CAC242B6F; Fri, 16 Sep 2022 08:17:11 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 9406442802 for ; Fri, 16 Sep 2022 08:17:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663309026; x=1694845026; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=QfrpEk+HOqjLVcx6UhPJT/1lH9iePijCsyIoXHVqQvg=; b=WHuTjyWD7ZKOKKxNS0rSXQbxRdQVjT38gTptjk+ufbgV+joGaiU5rAoh 6E/z4ulx9OG7C4dCip+tNNYQ4SDYWibtkx4sHm9bNWZcouoThSsgE04Gh 8+9HAdLMZqGM9kESMyhDa/ty3gHG3n4cOo7icFRUSF3r0J96pYpWIJ6GY q629vJRo5DaumjOchoaIT92tiEGxSm03t3isuLTEWi8QFccrDeG16h6bn 1hUwEttmB7wBMUsAOefgnbC1h3Bdo/0nh3RQclb2baK6XPnH2ZfrOCLBd CVJWE1Oh4Bae5Ie3PgkR8PDz9FhIeavjQHkHG9Qs+tX3YmUXTdsCoH51W w==; X-IronPort-AV: E=McAfee;i="6500,9779,10471"; a="278650971" X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="278650971" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Sep 2022 23:16:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="568724656" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by orsmga003.jf.intel.com with ESMTP; 15 Sep 2022 23:16:41 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v3 6/8] vdpa/ifc: support dynamic enable/disable queue Date: Fri, 16 Sep 2022 14:16:28 +0800 Message-Id: <1663308990-621-7-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1663308990-621-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1663308990-621-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Huang Wei Support dynamic enable or disable queue. For front end, like QEMU, user can use ethtool to configurate queue. For example, "ethtool -L eth0 combined 3" to enable 3 queues pairs. Signed-off-by: Huang Wei Signed-off-by: Andy Pei --- drivers/vdpa/ifc/base/ifcvf.c | 99 +++++++++++++++++++++++++++++++++++++++++++ drivers/vdpa/ifc/base/ifcvf.h | 6 +++ drivers/vdpa/ifc/ifcvf_vdpa.c | 93 +++++++++++++++++++++++++++++++++------- 3 files changed, 183 insertions(+), 15 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 619b034..792d258 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -227,6 +227,105 @@ } } +int +ifcvf_enable_vring_hw(struct ifcvf_hw *hw, int i) +{ + struct ifcvf_pci_common_cfg *cfg; + u8 *lm_cfg; + u16 notify_off; + int msix_vector; + + if (i >= (int)hw->nr_vring) + return -1; + + cfg = hw->common_cfg; + if (!cfg) { + WARNINGOUT("common_cfg in HW is NULL.\n"); + return -1; + } + + ifcvf_enable_multiqueue(hw); + + IFCVF_WRITE_REG16(i, &cfg->queue_select); + msix_vector = IFCVF_READ_REG16(&cfg->queue_msix_vector); + if (msix_vector != (i + 1)) { + IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector); + msix_vector = IFCVF_READ_REG16(&cfg->queue_msix_vector); + if (msix_vector == IFCVF_MSI_NO_VECTOR) { + WARNINGOUT("queue %u, msix vec alloc failed\n", i); + return -1; + } + } + + io_write64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo, + &cfg->queue_desc_hi); + io_write64_twopart(hw->vring[i].avail, &cfg->queue_avail_lo, + &cfg->queue_avail_hi); + io_write64_twopart(hw->vring[i].used, &cfg->queue_used_lo, + &cfg->queue_used_hi); + IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size); + + lm_cfg = hw->lm_cfg; + if (lm_cfg) { + if (hw->device_type == IFCVF_BLK) + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + else + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } + + notify_off = IFCVF_READ_REG16(&cfg->queue_notify_off); + hw->notify_addr[i] = (void *)((u8 *)hw->notify_base + + notify_off * hw->notify_off_multiplier); + IFCVF_WRITE_REG16(1, &cfg->queue_enable); + + return 0; +} + +void +ifcvf_disable_vring_hw(struct ifcvf_hw *hw, int i) +{ + struct ifcvf_pci_common_cfg *cfg; + u32 ring_state; + u8 *lm_cfg; + + if (i >= (int)hw->nr_vring) + return; + + cfg = hw->common_cfg; + if (!cfg) { + WARNINGOUT("common_cfg in HW is NULL.\n"); + return; + } + + IFCVF_WRITE_REG16(i, &cfg->queue_select); + IFCVF_WRITE_REG16(0, &cfg->queue_enable); + + lm_cfg = hw->lm_cfg; + if (lm_cfg) { + if (hw->device_type == IFCVF_BLK) { + ring_state = *(u32 *)(lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + } else { + ring_state = *(u32 *)(lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4); + hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); + } + hw->vring[i].last_used_idx = (u16)(ring_state >> 16); + } +} + STATIC int ifcvf_hw_enable(struct ifcvf_hw *hw) { diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 1e133c0..3726da7 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -164,6 +164,12 @@ struct ifcvf_hw { ifcvf_get_features(struct ifcvf_hw *hw); int +ifcvf_enable_vring_hw(struct ifcvf_hw *hw, int i); + +void +ifcvf_disable_vring_hw(struct ifcvf_hw *hw, int i); + +int ifcvf_start_hw(struct ifcvf_hw *hw); void diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index b00afdb..32bc1c9 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1282,13 +1282,59 @@ struct rte_vdpa_dev_info { } static int +ifcvf_config_vring(struct ifcvf_internal *internal, int vring) +{ + struct ifcvf_hw *hw = &internal->hw; + int vid = internal->vid; + struct rte_vhost_vring vq; + uint64_t gpa; + + if (hw->vring[vring].enable) { + rte_vhost_get_vhost_vring(vid, vring, &vq); + gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.desc); + if (gpa == 0) { + DRV_LOG(ERR, "Fail to get GPA for descriptor ring."); + return -1; + } + hw->vring[vring].desc = gpa; + + gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.avail); + if (gpa == 0) { + DRV_LOG(ERR, "Fail to get GPA for available ring."); + return -1; + } + hw->vring[vring].avail = gpa; + + gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); + if (gpa == 0) { + DRV_LOG(ERR, "Fail to get GPA for used ring."); + return -1; + } + hw->vring[vring].used = gpa; + + hw->vring[vring].size = vq.size; + rte_vhost_get_vring_base(vid, vring, + &hw->vring[vring].last_avail_idx, + &hw->vring[vring].last_used_idx); + ifcvf_enable_vring_hw(&internal->hw, vring); + } else { + ifcvf_disable_vring_hw(&internal->hw, vring); + rte_vhost_set_vring_base(vid, vring, + hw->vring[vring].last_avail_idx, + hw->vring[vring].last_used_idx); + } + + return 0; +} + +static int ifcvf_set_vring_state(int vid, int vring, int state) { struct rte_vdpa_device *vdev; struct internal_list *list; struct ifcvf_internal *internal; struct ifcvf_hw *hw; - struct ifcvf_pci_common_cfg *cfg; + bool enable = !!state; int ret = 0; vdev = rte_vhost_get_vdpa_device(vid); @@ -1298,6 +1344,9 @@ struct rte_vdpa_dev_info { return -1; } + DRV_LOG(INFO, "%s queue %d of vDPA device %s", + enable ? "enable" : "disable", vring, vdev->device->name); + internal = list->internal; if (vring < 0 || vring >= internal->max_queues * 2) { DRV_LOG(ERR, "Vring index %d not correct", vring); @@ -1305,27 +1354,41 @@ struct rte_vdpa_dev_info { } hw = &internal->hw; + hw->vring[vring].enable = enable; + if (!internal->configured) - goto exit; + return 0; - cfg = hw->common_cfg; - IFCVF_WRITE_REG16(vring, &cfg->queue_select); - IFCVF_WRITE_REG16(!!state, &cfg->queue_enable); + unset_notify_relay(internal); - if (!state && hw->vring[vring].enable) { - ret = vdpa_disable_vfio_intr(internal); - if (ret) - return ret; + ret = vdpa_enable_vfio_intr(internal, false); + if (ret) { + DRV_LOG(ERR, "failed to set vfio interrupt of vDPA device %s", + vdev->device->name); + return ret; } - if (state && !hw->vring[vring].enable) { - ret = vdpa_enable_vfio_intr(internal, false); - if (ret) - return ret; + ret = ifcvf_config_vring(internal, vring); + if (ret) { + DRV_LOG(ERR, "failed to configure queue %d of vDPA device %s", + vring, vdev->device->name); + return ret; + } + + ret = setup_notify_relay(internal); + if (ret) { + DRV_LOG(ERR, "failed to setup notify relay of vDPA device %s", + vdev->device->name); + return ret; + } + + ret = rte_vhost_host_notifier_ctrl(vid, vring, enable); + if (ret) { + DRV_LOG(ERR, "vDPA device %s queue %d host notifier ctrl fail", + vdev->device->name, vring); + return ret; } -exit: - hw->vring[vring].enable = !!state; return 0; } From patchwork Fri Sep 16 06:16:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116389 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2F48EA0032; Fri, 16 Sep 2022 08:17:36 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1096342B73; Fri, 16 Sep 2022 08:17:12 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 15A56427EE for ; Fri, 16 Sep 2022 08:17:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663309027; x=1694845027; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=5SkPNFrVx/hpgQWgkOwA+0hNZfJt821dx9xKENOEJNQ=; b=FpdEwp+SYDOvWIslP0xqUOMQtsJpZi/ajLa41SFXq6iYX35Wf+Pocd7O Opjy+5Y+ZzdH3DXFHwVf+PyLtbRBVvGYm0uqEbv4JpYLU75WTxJatbE7z 0otgOraO9Y+jJHfPbh7zpfwBH1PdZJzWj5ubYKudpUprA1BRDv14VJP93 8Nj7Dje65Ez0rwZyJDDP9eLLx5xMAaA4CValWwXV/9CCbUNJ+pRM0FtBN OVkHRjgEo+p0iF445BpI4OGTxmLZ1ANoZGKvpYSxTVHv2DqWlmVO4q7tR f+yt41V2PynkpBpW7KChaMhXwfHC4GQOwASR+GRcNPWVsGDPBGpVj6Sl7 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10471"; a="278650979" X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="278650979" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Sep 2022 23:16:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="568724681" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by orsmga003.jf.intel.com with ESMTP; 15 Sep 2022 23:16:44 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v3 7/8] vhost: vDPA blk device gets ready when any queue is ready Date: Fri, 16 Sep 2022 14:16:29 +0800 Message-Id: <1663308990-621-8-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1663308990-621-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1663308990-621-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org When boot from virtio blk device, seabios in QEMU only enables one queue. To work in this scenario, vDPA BLK device back-end conf_dev when any queue is ready. Signed-off-by: Andy Pei Signed-off-by: Huang Wei --- lib/vhost/vhost_user.c | 51 ++++++++++++++++++++++++++++++++------------------ 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 4ad28ba..9169cf5 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -1449,9 +1449,10 @@ } #define VIRTIO_BUILTIN_NUM_VQS_TO_BE_READY 2u +#define VIRTIO_BLK_NUM_VQS_TO_BE_READY 1u static int -virtio_is_ready(struct virtio_net *dev) +virtio_is_ready(struct virtio_net *dev, uint32_t vdpa_type) { struct vhost_virtqueue *vq; uint32_t i, nr_vring = dev->nr_vring; @@ -1462,13 +1463,20 @@ if (!dev->nr_vring) return 0; - if (dev->flags & VIRTIO_DEV_BUILTIN_VIRTIO_NET) { - nr_vring = VIRTIO_BUILTIN_NUM_VQS_TO_BE_READY; - - if (dev->nr_vring < nr_vring) - return 0; + if (vdpa_type == RTE_VHOST_VDPA_DEVICE_TYPE_NET) { + if (dev->flags & VIRTIO_DEV_BUILTIN_VIRTIO_NET) + nr_vring = VIRTIO_BUILTIN_NUM_VQS_TO_BE_READY; + } else { + /* + * vdpa_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK + * is the only case currently + */ + nr_vring = VIRTIO_BLK_NUM_VQS_TO_BE_READY; } + if (dev->nr_vring < nr_vring) + return 0; + for (i = 0; i < nr_vring; i++) { vq = dev->virtqueue[i]; @@ -3167,7 +3175,25 @@ static int is_vring_iotlb(struct virtio_net *dev, if (unlock_required) vhost_user_unlock_all_queue_pairs(dev); - if (ret != 0 || !virtio_is_ready(dev)) + if (ret != 0) + goto out; + + vdpa_dev = dev->vdpa_dev; + if (vdpa_dev) { + if (vdpa_dev->ops->get_dev_type) { + ret = vdpa_dev->ops->get_dev_type(vdpa_dev, &vdpa_type); + if (ret) { + VHOST_LOG_CONFIG(dev->ifname, ERR, + "failed to get vdpa dev type.\n"); + ret = -1; + goto out; + } + } else { + vdpa_type = RTE_VHOST_VDPA_DEVICE_TYPE_NET; + } + } + + if (!virtio_is_ready(dev, vdpa_type)) goto out; /* @@ -3181,20 +3207,9 @@ static int is_vring_iotlb(struct virtio_net *dev, dev->flags |= VIRTIO_DEV_RUNNING; } - vdpa_dev = dev->vdpa_dev; if (!vdpa_dev) goto out; - if (vdpa_dev->ops->get_dev_type) { - ret = vdpa_dev->ops->get_dev_type(vdpa_dev, &vdpa_type); - if (ret) { - VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to get vdpa dev type.\n"); - ret = -1; - goto out; - } - } else { - vdpa_type = RTE_VHOST_VDPA_DEVICE_TYPE_NET; - } if (vdpa_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK && request != VHOST_USER_SET_VRING_CALL) goto out; From patchwork Fri Sep 16 06:16:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 116390 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C3E52A0032; Fri, 16 Sep 2022 08:17:40 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 087C542B79; Fri, 16 Sep 2022 08:17:13 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 3742C427EE for ; Fri, 16 Sep 2022 08:17:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663309028; x=1694845028; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=KQRu9mTPOSuE8GLcxxp5/4t3hXE2GlFWx77/85djTyY=; b=Gx8mWjxCrA/Cyi/xv8OcZxldd0WH/vfpwUCiPYF6UsegjgxdKsHeXZd+ sW2Ptemq1weTMs7hyRJY/fpCCxVfOFjpNwuxwyODX6jFWgmWWHef4MvjR 7EMiWpncONami3sE6xaI3i31FFA1ux+CPklmuoXt+7zSIwRBUiao0o7ru Bi6R3SdAQ7VUWXqlNmKKKHrqr0wTI8Ft7aD99ztW7/EFnSeDRWWuj/ZZ0 BV3+FIxZ0NrEDSCulWQfnWeVQFap9kYrQ2qPdlU/vt+12OtJWh7ELoypY 3sVj39DzHTIK+fGB+ll/wvn4m351QJaFtJmt5906aRIVtXKt3F58LQZlj g==; X-IronPort-AV: E=McAfee;i="6500,9779,10471"; a="278650986" X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="278650986" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Sep 2022 23:16:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,320,1654585200"; d="scan'208";a="568724693" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by orsmga003.jf.intel.com with ESMTP; 15 Sep 2022 23:16:47 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v3 8/8] vhost: improve vDPA blk device readiness condition Date: Fri, 16 Sep 2022 14:16:30 +0800 Message-Id: <1663308990-621-9-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1663308990-621-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1663308990-621-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In the virtio blk vDPA live migration use case, for the target VM, before the live migration process, QEMU will set call fd to all queues of vDPA back-end. QEMU and vDPA back-end stand by until live migration starts. During live migration process, QEMU sets kick fd and new call fd. However, after the kick fd is set to the vDPA back-end, the vDPA back-end configures device and data path starts. The new call fd will cause some kind of "re-configuration", this kind of "re-configuration" cause IO drop. After this patch, vDPA back-end configures device after kick fd and call fd are well set and make sure no IO drops. This patch only impact virtio blk vDPA device and does not impact net device. Signed-off-by: Andy Pei Signed-off-by: Huang Wei --- lib/vhost/vhost_user.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index 9169cf5..14ff266 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -2983,6 +2983,7 @@ static int is_vring_iotlb(struct virtio_net *dev, uint32_t vdpa_type = 0; uint32_t request; uint32_t i; + uint16_t blk_call_fd; dev = get_device(vid); if (dev == NULL) @@ -3210,9 +3211,15 @@ static int is_vring_iotlb(struct virtio_net *dev, if (!vdpa_dev) goto out; - if (vdpa_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK - && request != VHOST_USER_SET_VRING_CALL) - goto out; + if (vdpa_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK) { + if (request == VHOST_USER_SET_VRING_CALL) { + blk_call_fd = ctx.msg.payload.u64 & VHOST_USER_VRING_IDX_MASK; + if (blk_call_fd != dev->nr_vring - 1) + goto out; + } else { + goto out; + } + } if (!(dev->flags & VIRTIO_DEV_VDPA_CONFIGURED)) { if (vdpa_dev->ops->dev_conf(dev->vid))