From patchwork Tue Oct 18 12:07:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 118403 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3BEFEA0560; Tue, 18 Oct 2022 14:58:28 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2F06E410EE; Tue, 18 Oct 2022 14:58:28 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 29A504021D for ; Tue, 18 Oct 2022 14:58:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666097907; x=1697633907; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=rakQV2IHcfDwfZqNZi7dr/4nebJyRsH3mVtU0TtwwFc=; b=iuIo3GPlWXENG6IC11B7cie2JsUrJkGwCgghqFyqWyYhg+kBCUE/3hE2 WLBskZuHwjriGJ/IaM94LvS1yVl0O28CGgmDtbd2bT16VHgNkGLrUuya7 9eUMyf8Q+A5OK+HhxBE5QuAciT6oUJiM5xIIAEP0OcPbnchhPyZdaogiX n9qnYAQKS4OTAHU8YZl92ZK55+jqo6qAUap3qz+vG2OS2SZcStbOeCeBg 7YG4QrIL765A/KtWpR29fEuMtAS3RebHQKYbKwDdinYeJs+Apfk5D+TGe ygryVovNFkkcc+Ux8VqY01JnEhq0rQhfCBpbwrSY8nGK0VQQd7NlOzkMC w==; X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="307179404" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="307179404" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 05:58:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="733614355" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="733614355" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga002.fm.intel.com with ESMTP; 18 Oct 2022 05:58:05 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v8 01/12] vdpa/ifc: add new device ID for legacy network device Date: Tue, 18 Oct 2022 20:07:40 +0800 Message-Id: <1666094871-25721-2-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1666094871-25721-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1666094871-25721-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Huang Wei Add new device id to support IFCVF_NET_TRANSITIONAL_DEVICE_ID (0x1000). Rename macro from "IFCVF_BLK_DEVICE_ID" to "IFCVF_SUBSYS_BLK_DEVICE_ID". Signed-off-by: Huang Wei Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia Reviewed-by: Maxime Coquelin --- drivers/vdpa/ifc/base/ifcvf.h | 6 ++++-- drivers/vdpa/ifc/ifcvf_vdpa.c | 13 ++++++++++--- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 9d95aac..ef7697a 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -12,12 +12,14 @@ #define IFCVF_BLK 1 #define IFCVF_VENDOR_ID 0x1AF4 -#define IFCVF_NET_DEVICE_ID 0x1041 +#define IFCVF_NET_MODERN_DEVICE_ID 0x1041 #define IFCVF_BLK_MODERN_DEVICE_ID 0x1042 +#define IFCVF_NET_TRANSITIONAL_DEVICE_ID 0x1000 #define IFCVF_BLK_TRANSITIONAL_DEVICE_ID 0x1001 #define IFCVF_SUBSYS_VENDOR_ID 0x8086 #define IFCVF_SUBSYS_DEVICE_ID 0x001A -#define IFCVF_BLK_DEVICE_ID 0x0002 +#define IFCVF_SUBSYS_NET_DEVICE_ID 0x0001 +#define IFCVF_SUBSYS_BLK_DEVICE_ID 0x0002 #define IFCVF_MAX_QUEUES 1 diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index d5ac583..b4389a0 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1684,23 +1684,30 @@ struct rte_vdpa_dev_info dev_info[] = { static const struct rte_pci_id pci_id_ifcvf_map[] = { { .class_id = RTE_CLASS_ANY_ID, .vendor_id = IFCVF_VENDOR_ID, - .device_id = IFCVF_NET_DEVICE_ID, + .device_id = IFCVF_NET_MODERN_DEVICE_ID, .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID, }, { .class_id = RTE_CLASS_ANY_ID, .vendor_id = IFCVF_VENDOR_ID, + .device_id = IFCVF_NET_TRANSITIONAL_DEVICE_ID, + .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, + .subsystem_device_id = IFCVF_SUBSYS_NET_DEVICE_ID, + }, + + { .class_id = RTE_CLASS_ANY_ID, + .vendor_id = IFCVF_VENDOR_ID, .device_id = IFCVF_BLK_TRANSITIONAL_DEVICE_ID, .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, - .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + .subsystem_device_id = IFCVF_SUBSYS_BLK_DEVICE_ID, }, { .class_id = RTE_CLASS_ANY_ID, .vendor_id = IFCVF_VENDOR_ID, .device_id = IFCVF_BLK_MODERN_DEVICE_ID, .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID, - .subsystem_device_id = IFCVF_BLK_DEVICE_ID, + .subsystem_device_id = IFCVF_SUBSYS_BLK_DEVICE_ID, }, { .vendor_id = 0, /* sentinel */ From patchwork Tue Oct 18 12:07:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 118404 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9613CA0560; Tue, 18 Oct 2022 14:58:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2849A41147; Tue, 18 Oct 2022 14:58:31 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id C99BF4021D for ; Tue, 18 Oct 2022 14:58:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666097908; x=1697633908; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=MZkE2BhYeMTxAZlKloqSsOQ5PvQYywfYmwmnp/qtYZA=; b=NlAxpux8IhPjm2DKK9/rbyz2OYezKkHG5kmD+anqaZjW1n2kTnvgdBFK oqHcvf3xxrdzP9G/R5eo3PQFzwieZdvPxL0AwUUTl622JI+DMZmeEg7zO Lq1AHWGd250NWUMfc/kYtwa+UDhH3K4WM5ioIzp8Zf9dVtCKQt+6p7r3l UmcZILN6RyKIh8onV+NuHvf2IF9WcRHj/3nAqZc6FDYw36/DB+0mcJe8T 6NEGrAnnhB9ivGfQ09xIwXeFoI/U13uW4X1BWNORBNqfT0RcIhLHPz5h+ VnDvlG8kJAMyA+GLPhIJE+xbmOJ8FUn9VoUqE+1cQmPdJmDNND8YtZli5 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="307179409" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="307179409" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 05:58:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="733614382" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="733614382" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga002.fm.intel.com with ESMTP; 18 Oct 2022 05:58:08 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v8 02/12] vdpa/ifc: add multi-queue support Date: Tue, 18 Oct 2022 20:07:41 +0800 Message-Id: <1666094871-25721-3-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1666094871-25721-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1666094871-25721-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Enable VHOST_USER_PROTOCOL_F_MQ feature. Expose IFCVF_MQ_OFFSET register to enable multi-queue. Signed-off-by: Andy Pei Signed-off-by: Huang Wei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.c | 9 +++++++++ drivers/vdpa/ifc/base/ifcvf.h | 2 ++ drivers/vdpa/ifc/ifcvf_vdpa.c | 1 + 3 files changed, 12 insertions(+) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index f1e1474..81c68c0 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -90,6 +90,15 @@ if (!hw->lm_cfg) WARNINGOUT("HW support live migration not support!\n"); + /* For some hardware implementation, for example: + * the BAR 4 of PF is NULL, while BAR 4 of VF is not. + * This code makes sure hw->mq_cfg is a valid address. + */ + if (hw->mem_resource[4].addr) + hw->mq_cfg = hw->mem_resource[4].addr + IFCVF_MQ_OFFSET; + else + hw->mq_cfg = NULL; + if (hw->common_cfg == NULL || hw->notify_base == NULL || hw->isr == NULL || hw->dev_cfg == NULL) { DEBUGOUT("capability incomplete\n"); diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index ef7697a..d16d9ab 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -50,6 +50,7 @@ #define IFCVF_LM_CFG_SIZE 0x40 #define IFCVF_LM_RING_STATE_OFFSET 0x20 +#define IFCVF_MQ_OFFSET 0x28 #define IFCVF_LM_LOGGING_CTRL 0x0 @@ -149,6 +150,7 @@ struct ifcvf_hw { u16 *notify_base; u16 *notify_addr[IFCVF_MAX_QUEUES * 2]; u8 *lm_cfg; + u8 *mq_cfg; struct vring_info vring[IFCVF_MAX_QUEUES * 2]; u8 nr_vring; int device_type; diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index b4389a0..008cf89 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1248,6 +1248,7 @@ struct rte_vdpa_dev_info { 1ULL << VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD | \ 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \ 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | \ + 1ULL << VHOST_USER_PROTOCOL_F_MQ | \ 1ULL << VHOST_USER_PROTOCOL_F_STATUS) #define VDPA_BLK_PROTOCOL_FEATURES \ From patchwork Tue Oct 18 12:07:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 118405 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6B610A0560; Tue, 18 Oct 2022 14:58:36 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F2BA041155; Tue, 18 Oct 2022 14:58:31 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 4E0CA41144 for ; Tue, 18 Oct 2022 14:58:28 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666097908; x=1697633908; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=B8ztTGzaGj+iPoPQvv5PN5iCSJ5ejZf1TdCqRnf0ebI=; b=IcUCTISlOWbANMgBv8fw/jjHRQRt9IfwjCD7JZkwZdY0OfsPC4iftotP LmndiwZhG+PEuiw5qRRUJZSEAbG52mLQ775UruUrKpTsLinx9JwHdqK5L 8HQbfA4EaVlQ+1pi/UT3OGVLiqyGTZffAk9QbdnPoRqePRPlh3j4kvs9E TdG12zOuJlHNGNXNI2OIzdfCuMKpk8CVwHGfPytUx5wSanbhVD3v9sSZ3 TYwhmpG1hBHxafvOogLBDaqJ2lXC8p/bBTsF3h++VYe4BeSmeI4kXuItL TP138eQK0TkqW8ndyx2bpv3SfuNe4PX3eE77cVcnMTkqvVqLAYIja7nsg g==; X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="307179420" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="307179420" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 05:58:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="733614419" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="733614419" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga002.fm.intel.com with ESMTP; 18 Oct 2022 05:58:11 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v8 03/12] vdpa/ifc: set max queues based on virtio spec Date: Tue, 18 Oct 2022 20:07:42 +0800 Message-Id: <1666094871-25721-4-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1666094871-25721-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1666094871-25721-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Set max_queues according to virtio spec. For virtio BLK device, set max_queues to the value of num_queues in struct virtio_blk_config. For virtio NET device, read num_queues from struct ifcvf_pci_common_cfg, get the queue pair number using num_queues and set max_queues to it. Signed-off-by: Andy Pei Signed-off-by: Huang Wei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.h | 2 +- drivers/vdpa/ifc/ifcvf_vdpa.c | 19 ++++++++++++++++++- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index d16d9ab..1e133c0 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -21,7 +21,7 @@ #define IFCVF_SUBSYS_NET_DEVICE_ID 0x0001 #define IFCVF_SUBSYS_BLK_DEVICE_ID 0x0002 -#define IFCVF_MAX_QUEUES 1 +#define IFCVF_MAX_QUEUES 32 #ifndef VIRTIO_F_IOMMU_PLATFORM #define VIRTIO_F_IOMMU_PLATFORM 33 diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 008cf89..5a24204 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -26,6 +26,12 @@ #include "base/ifcvf.h" +/* + * RTE_MIN() cannot be used since braced-group within expression allowed + * only inside a function. + */ +#define MIN(v1, v2) ((v1) < (v2) ? (v1) : (v2)) + RTE_LOG_REGISTER(ifcvf_vdpa_logtype, pmd.vdpa.ifcvf, NOTICE); #define DRV_LOG(level, fmt, args...) \ rte_log(RTE_LOG_ ## level, ifcvf_vdpa_logtype, \ @@ -1512,6 +1518,7 @@ struct rte_vdpa_dev_info dev_info[] = { uint64_t capacity = 0; uint8_t *byte; uint32_t i; + uint16_t queue_pairs; if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1559,7 +1566,6 @@ struct rte_vdpa_dev_info dev_info[] = { } internal->configured = 0; - internal->max_queues = IFCVF_MAX_QUEUES; features = ifcvf_get_features(&internal->hw); device_id = ifcvf_pci_get_device_type(pci_dev); @@ -1570,6 +1576,14 @@ struct rte_vdpa_dev_info dev_info[] = { if (device_id == VIRTIO_ID_NET) { internal->hw.device_type = IFCVF_NET; + /* + * ifc device always has CTRL_VQ, + * and supports VIRTIO_NET_F_CTRL_VQ feature. + */ + queue_pairs = (internal->hw.common_cfg->num_queues - 1) / 2; + DRV_LOG(INFO, "%s support %u queue pairs", pci_dev->name, + queue_pairs); + internal->max_queues = MIN(IFCVF_MAX_QUEUES, queue_pairs); internal->features = features & ~(1ULL << VIRTIO_F_IOMMU_PLATFORM); internal->features |= dev_info[IFCVF_NET].features; @@ -1609,6 +1623,9 @@ struct rte_vdpa_dev_info dev_info[] = { internal->hw.blk_cfg->geometry.sectors); DRV_LOG(DEBUG, "num_queues: 0x%08x", internal->hw.blk_cfg->num_queues); + + internal->max_queues = MIN(IFCVF_MAX_QUEUES, + internal->hw.blk_cfg->num_queues); } list->internal = internal; From patchwork Tue Oct 18 12:07:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 118406 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7CFF4A0560; Tue, 18 Oct 2022 14:58:41 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id ECB26415D7; Tue, 18 Oct 2022 14:58:32 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id DE0F341144 for ; Tue, 18 Oct 2022 14:58:28 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666097909; x=1697633909; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Vw+EQojkuGZtMt5TTcckCCxAzKV7MQu6TroOvfBfjhM=; b=IyiN/Yf90a1SFyXNm0cG/kEJSC3DInxNcX1AJvOgIOt8jDUr/IYVO6/i B2w6k5k+elSKH0KfvfKi7/2Rv9CGoahk22JyU+bKXymjg7/3JjoYKnHxI /FXXVR4qlFue8SjYV4mCLGUVsZczbpkI7gkS4AayNacHWd9WXKNcLznxM uFs3caLXib9ManIOvhlQCwtb3bVjMwZG3gWcLt5amOwxfZRvOaxUbQXGU ywndk6OO27Dppbjq73lLoabDnViys2MdjUzzsxW9QzMrkHcSD6+tpe4kE kEjpQvGN2VD52M2MLEQqaRYLiQGTsgYrPN74Lko3YecYsOOowoiwoQtp9 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="307179443" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="307179443" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 05:58:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="733614450" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="733614450" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga002.fm.intel.com with ESMTP; 18 Oct 2022 05:58:14 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v8 04/12] vdpa/ifc: write queue count to MQ register Date: Tue, 18 Oct 2022 20:07:43 +0800 Message-Id: <1666094871-25721-5-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1666094871-25721-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1666094871-25721-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Write queue count to IFCVF_MQ_OFFSET register to enable multi-queue feature. Signed-off-by: Andy Pei Signed-off-by: Huang Wei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 81c68c0..b377126 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -202,6 +202,37 @@ IFCVF_WRITE_REG32(val >> 32, hi); } +STATIC void +ifcvf_enable_mq(struct ifcvf_hw *hw) +{ + u8 *mq_cfg; + u8 qid; + int nr_queue = 0; + + for (qid = 0; qid < hw->nr_vring; qid++) { + if (!hw->vring[qid].enable) + continue; + nr_queue++; + } + + if (nr_queue == 0) { + WARNINGOUT("no enabled vring\n"); + return; + } + + mq_cfg = hw->mq_cfg; + if (mq_cfg) { + if (hw->device_type == IFCVF_BLK) { + *(u32 *)mq_cfg = nr_queue; + RTE_LOG(INFO, PMD, "%d queues are enabled\n", nr_queue); + } else { + *(u32 *)mq_cfg = nr_queue / 2; + RTE_LOG(INFO, PMD, "%d queue pairs are enabled\n", + nr_queue / 2); + } + } +} + STATIC int ifcvf_hw_enable(struct ifcvf_hw *hw) { @@ -219,6 +250,7 @@ return -1; } + ifcvf_enable_mq(hw); for (i = 0; i < hw->nr_vring; i++) { IFCVF_WRITE_REG16(i, &cfg->queue_select); io_write64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo, From patchwork Tue Oct 18 12:07:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 118407 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DC2B0A0566; Tue, 18 Oct 2022 14:58:48 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1DD99427F4; Tue, 18 Oct 2022 14:58:35 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 91ABC410E8 for ; Tue, 18 Oct 2022 14:58:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666097910; x=1697633910; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=u6gMBbApdXSB3Ld3Jjg3QEGS4spcCcxNroeYHYyn54Q=; b=FOI+Ptm1WNN2m3iGl3L6qzEDKJBAY7UkK1HtlZyYorbOXdQ9NMhZan/D Y+aTYK1mN8gYQLEmTlu+Rzr70MgNFtI5Vuza5IYcJ/e9cJv6I2hv3V8E6 1YOAfHVkg275aPmEZVPGHNYmb56NBifPE3/Lym82v4l3nn4oxgSm3LkfM G+ioV+1Je1rKHSY8WjsnhKDlK5aAgtjgZKJkYPzlGvt4nYT65z+yLzHHw vyD6sqL4qM3k5V2sqF61tuQnVttoDsRS6mru6xlH+DYZ5s+BS7S9jAgvX 0X8UBXUhtisRSeSB7yxmG/hR7DWCbUy4sP8Bv24imk4iNd1yr2l4oyKrc A==; X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="307179460" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="307179460" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 05:58:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="733614473" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="733614473" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga002.fm.intel.com with ESMTP; 18 Oct 2022 05:58:17 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v8 05/12] vdpa/ifc: only configure enabled queue Date: Tue, 18 Oct 2022 20:07:44 +0800 Message-Id: <1666094871-25721-6-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1666094871-25721-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1666094871-25721-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org When configuring the hardware queue, we only configure queues which have been enabled by vhost. Signed-off-by: Andy Pei Signed-off-by: Huang Wei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.c | 3 +++ drivers/vdpa/ifc/ifcvf_vdpa.c | 16 ++++++++++++++-- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index b377126..30bb8cb 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -252,6 +252,9 @@ ifcvf_enable_mq(hw); for (i = 0; i < hw->nr_vring; i++) { + if (!hw->vring[i].enable) + continue; + IFCVF_WRITE_REG16(i, &cfg->queue_select); io_write64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo, &cfg->queue_desc_hi); diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 5a24204..0c3407a 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -284,6 +284,8 @@ struct rte_vdpa_dev_info { rte_vhost_get_negotiated_features(vid, &hw->req_features); for (i = 0; i < nr_vring; i++) { + if (!hw->vring[i].enable) + continue; rte_vhost_get_vhost_vring(vid, i, &vq); gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.desc); if (gpa == 0) { @@ -499,6 +501,8 @@ struct rte_vdpa_dev_info { vring.kickfd = -1; for (qid = 0; qid < q_num; qid++) { + if (!hw->vring[qid].enable) + continue; ev.events = EPOLLIN | EPOLLPRI; rte_vhost_get_vhost_vring(internal->vid, qid, &vring); ev.data.u64 = qid | (uint64_t)vring.kickfd << 32; @@ -1058,6 +1062,8 @@ struct rte_vdpa_dev_info { struct rte_vdpa_device *vdev; struct internal_list *list; struct ifcvf_internal *internal; + struct ifcvf_hw *hw; + uint16_t i; vdev = rte_vhost_get_vdpa_device(vid); list = find_internal_resource_by_vdev(vdev); @@ -1071,11 +1077,17 @@ struct rte_vdpa_dev_info { rte_atomic32_set(&internal->dev_attached, 1); update_datapath(internal); - if (rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, true) != 0) - DRV_LOG(NOTICE, "vDPA (%s): software relay is used.", + hw = &internal->hw; + for (i = 0; i < hw->nr_vring; i++) { + if (!hw->vring[i].enable) + continue; + if (rte_vhost_host_notifier_ctrl(vid, i, true) != 0) + DRV_LOG(NOTICE, "vDPA (%s): software relay is used.", vdev->device->name); + } internal->configured = 1; + DRV_LOG(INFO, "vDPA device %s is configured", vdev->device->name); return 0; } From patchwork Tue Oct 18 12:07:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 118408 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 57B17A0560; Tue, 18 Oct 2022 14:58:54 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 20163427F9; Tue, 18 Oct 2022 14:58:36 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 80569410E8 for ; Tue, 18 Oct 2022 14:58:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666097911; x=1697633911; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=VMY/XQgY3LiXk3abQWsaQCKGKFqsmcMrD2+Gz98WQA0=; b=U6eEGT+PA75yy8OnSJ++HUrhjfA7nnif/lImQAzPtTwFHvO77yrI9NZD +62IIf3K2/Lvqa7s9pnlUOBCJ+4m0iEWyDWohVRw60frcbQDdIrks+5R0 yiSXMEr1iLMdXZ8zjfRT3fj2xXyiXJA300fw0ZHpzlNX9faO/XbQ3Ojd7 5jMRax+EaisDvnM5+D7vxhp9cjTvQSveKXdZ6Oe0HGilp2p6r1xAe8/MM D1qDh/gYBcA6HHUXlID+cjP4908EQNNEUgKFIpC75rNiKti535ChVlNUf cSElN0fsb0hW0VL7FsOoqEj8yWILjnQg58pyM6wAlMtMjXQYSBiS8pcpr g==; X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="307179471" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="307179471" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 05:58:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="733614496" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="733614496" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga002.fm.intel.com with ESMTP; 18 Oct 2022 05:58:19 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v8 06/12] vdpa/ifc: support dynamic enable/disable queue Date: Tue, 18 Oct 2022 20:07:45 +0800 Message-Id: <1666094871-25721-7-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1666094871-25721-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1666094871-25721-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Huang Wei Support dynamic enable or disable queue. For front end, like QEMU, user can use ethtool to configure queue. For example, "ethtool -L eth0 combined 3" to enable 3 queues pairs. Signed-off-by: Huang Wei Signed-off-by: Andy Pei Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/base/ifcvf.c | 100 ++++++++++++++++++++++++++++++++++++++++++ drivers/vdpa/ifc/base/ifcvf.h | 6 +++ drivers/vdpa/ifc/ifcvf_vdpa.c | 93 ++++++++++++++++++++++++++++++++------- 3 files changed, 184 insertions(+), 15 deletions(-) diff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c index 30bb8cb..869ddd6 100644 --- a/drivers/vdpa/ifc/base/ifcvf.c +++ b/drivers/vdpa/ifc/base/ifcvf.c @@ -233,6 +233,106 @@ } } +int +ifcvf_enable_vring_hw(struct ifcvf_hw *hw, int i) +{ + struct ifcvf_pci_common_cfg *cfg; + u8 *lm_cfg; + u16 notify_off; + int msix_vector; + + if (i >= (int)hw->nr_vring) + return -1; + + cfg = hw->common_cfg; + if (!cfg) { + RTE_LOG(ERR, PMD, "common_cfg in HW is NULL.\n"); + return -1; + } + + ifcvf_enable_mq(hw); + + IFCVF_WRITE_REG16(i, &cfg->queue_select); + msix_vector = IFCVF_READ_REG16(&cfg->queue_msix_vector); + if (msix_vector != (i + 1)) { + IFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector); + msix_vector = IFCVF_READ_REG16(&cfg->queue_msix_vector); + if (msix_vector == IFCVF_MSI_NO_VECTOR) { + RTE_LOG(ERR, PMD, "queue %d, msix vec alloc failed\n", + i); + return -1; + } + } + + io_write64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo, + &cfg->queue_desc_hi); + io_write64_twopart(hw->vring[i].avail, &cfg->queue_avail_lo, + &cfg->queue_avail_hi); + io_write64_twopart(hw->vring[i].used, &cfg->queue_used_lo, + &cfg->queue_used_hi); + IFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size); + + lm_cfg = hw->lm_cfg; + if (lm_cfg) { + if (hw->device_type == IFCVF_BLK) + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + else + *(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4) = + (u32)hw->vring[i].last_avail_idx | + ((u32)hw->vring[i].last_used_idx << 16); + } + + notify_off = IFCVF_READ_REG16(&cfg->queue_notify_off); + hw->notify_addr[i] = (void *)((u8 *)hw->notify_base + + notify_off * hw->notify_off_multiplier); + IFCVF_WRITE_REG16(1, &cfg->queue_enable); + + return 0; +} + +void +ifcvf_disable_vring_hw(struct ifcvf_hw *hw, int i) +{ + struct ifcvf_pci_common_cfg *cfg; + u32 ring_state; + u8 *lm_cfg; + + if (i >= (int)hw->nr_vring) + return; + + cfg = hw->common_cfg; + if (!cfg) { + RTE_LOG(ERR, PMD, "common_cfg in HW is NULL.\n"); + return; + } + + IFCVF_WRITE_REG16(i, &cfg->queue_select); + IFCVF_WRITE_REG16(0, &cfg->queue_enable); + + lm_cfg = hw->lm_cfg; + if (lm_cfg) { + if (hw->device_type == IFCVF_BLK) { + ring_state = *(u32 *)(lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + i * IFCVF_LM_CFG_SIZE); + hw->vring[i].last_avail_idx = + (u16)(ring_state & IFCVF_16_BIT_MASK); + } else { + ring_state = *(u32 *)(lm_cfg + + IFCVF_LM_RING_STATE_OFFSET + + (i / 2) * IFCVF_LM_CFG_SIZE + + (i % 2) * 4); + hw->vring[i].last_avail_idx = (u16)(ring_state >> 16); + } + hw->vring[i].last_used_idx = (u16)(ring_state >> 16); + } +} + STATIC int ifcvf_hw_enable(struct ifcvf_hw *hw) { diff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h index 1e133c0..3726da7 100644 --- a/drivers/vdpa/ifc/base/ifcvf.h +++ b/drivers/vdpa/ifc/base/ifcvf.h @@ -164,6 +164,12 @@ struct ifcvf_hw { ifcvf_get_features(struct ifcvf_hw *hw); int +ifcvf_enable_vring_hw(struct ifcvf_hw *hw, int i); + +void +ifcvf_disable_vring_hw(struct ifcvf_hw *hw, int i); + +int ifcvf_start_hw(struct ifcvf_hw *hw); void diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 0c3407a..9c49f9c 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1282,13 +1282,59 @@ struct rte_vdpa_dev_info { } static int +ifcvf_config_vring(struct ifcvf_internal *internal, int vring) +{ + struct ifcvf_hw *hw = &internal->hw; + int vid = internal->vid; + struct rte_vhost_vring vq; + uint64_t gpa; + + if (hw->vring[vring].enable) { + rte_vhost_get_vhost_vring(vid, vring, &vq); + gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.desc); + if (gpa == 0) { + DRV_LOG(ERR, "Fail to get GPA for descriptor ring."); + return -1; + } + hw->vring[vring].desc = gpa; + + gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.avail); + if (gpa == 0) { + DRV_LOG(ERR, "Fail to get GPA for available ring."); + return -1; + } + hw->vring[vring].avail = gpa; + + gpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used); + if (gpa == 0) { + DRV_LOG(ERR, "Fail to get GPA for used ring."); + return -1; + } + hw->vring[vring].used = gpa; + + hw->vring[vring].size = vq.size; + rte_vhost_get_vring_base(vid, vring, + &hw->vring[vring].last_avail_idx, + &hw->vring[vring].last_used_idx); + ifcvf_enable_vring_hw(&internal->hw, vring); + } else { + ifcvf_disable_vring_hw(&internal->hw, vring); + rte_vhost_set_vring_base(vid, vring, + hw->vring[vring].last_avail_idx, + hw->vring[vring].last_used_idx); + } + + return 0; +} + +static int ifcvf_set_vring_state(int vid, int vring, int state) { struct rte_vdpa_device *vdev; struct internal_list *list; struct ifcvf_internal *internal; struct ifcvf_hw *hw; - struct ifcvf_pci_common_cfg *cfg; + bool enable = !!state; int ret = 0; vdev = rte_vhost_get_vdpa_device(vid); @@ -1298,6 +1344,9 @@ struct rte_vdpa_dev_info { return -1; } + DRV_LOG(INFO, "%s queue %d of vDPA device %s", + enable ? "enable" : "disable", vring, vdev->device->name); + internal = list->internal; if (vring < 0 || vring >= internal->max_queues * 2) { DRV_LOG(ERR, "Vring index %d not correct", vring); @@ -1305,27 +1354,41 @@ struct rte_vdpa_dev_info { } hw = &internal->hw; + hw->vring[vring].enable = enable; + if (!internal->configured) - goto exit; + return 0; - cfg = hw->common_cfg; - IFCVF_WRITE_REG16(vring, &cfg->queue_select); - IFCVF_WRITE_REG16(!!state, &cfg->queue_enable); + unset_notify_relay(internal); - if (!state && hw->vring[vring].enable) { - ret = vdpa_disable_vfio_intr(internal); - if (ret) - return ret; + ret = vdpa_enable_vfio_intr(internal, false); + if (ret) { + DRV_LOG(ERR, "failed to set vfio interrupt of vDPA device %s", + vdev->device->name); + return ret; } - if (state && !hw->vring[vring].enable) { - ret = vdpa_enable_vfio_intr(internal, false); - if (ret) - return ret; + ret = ifcvf_config_vring(internal, vring); + if (ret) { + DRV_LOG(ERR, "failed to configure queue %d of vDPA device %s", + vring, vdev->device->name); + return ret; + } + + ret = setup_notify_relay(internal); + if (ret) { + DRV_LOG(ERR, "failed to setup notify relay of vDPA device %s", + vdev->device->name); + return ret; + } + + ret = rte_vhost_host_notifier_ctrl(vid, vring, enable); + if (ret) { + DRV_LOG(ERR, "vDPA device %s queue %d host notifier ctrl fail", + vdev->device->name, vring); + return ret; } -exit: - hw->vring[vring].enable = !!state; return 0; } From patchwork Tue Oct 18 12:07:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 118409 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3958CA0560; Tue, 18 Oct 2022 14:59:00 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 04ADF4281A; Tue, 18 Oct 2022 14:58:37 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 6A7CF415D7 for ; Tue, 18 Oct 2022 14:58:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666097912; x=1697633912; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=5Z/TE3nEwNM6fzA82zyDJZ3KPU8sJuMUSRQVSAqEOMs=; b=FJZOraSp9mRUXPaPJfmXbvYl2MJa2MxpY17atEagYPsPisgWVbrtowoq QBdndZMRC1RwUWMpdMICjTUPQwqQRFntOj5VAOBdb2UZdlyNx9MO4H35c xK6ehT18Z+d79+5VECinWuc6Ekabh0J/RO7mkzfbyl7t61+0SjE3JQbTa pGzI9/i6WlKWAWC7VoCvux55WicsJiRKbWnQEnuCY9IZudljhEYgp+5e7 BgwlNLKVFKiY+FRIlZW/ToZbAAb24cDzEViEGSrwoWV3NHdbkdMNKSMka jYXlQOaWX5bpitgWosi64bWCcSB+s2ZxR4ti9lS4VbP5GATtg03A7tsq/ A==; X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="307179476" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="307179476" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 05:58:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="733614515" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="733614515" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga002.fm.intel.com with ESMTP; 18 Oct 2022 05:58:22 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v8 07/12] vdpa/ifc: change internal function name Date: Tue, 18 Oct 2022 20:07:46 +0800 Message-Id: <1666094871-25721-8-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1666094871-25721-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1666094871-25721-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Change internal function name "find_internal_resource_by_dev" to "find_internal_resource_by_pci_dev". Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 9c49f9c..73d04ed 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -121,7 +121,7 @@ struct rte_vdpa_dev_info { } static struct internal_list * -find_internal_resource_by_dev(struct rte_pci_device *pdev) +find_internal_resource_by_pci_dev(struct rte_pci_device *pdev) { int found = 0; struct internal_list *list; @@ -1746,7 +1746,7 @@ struct rte_vdpa_dev_info dev_info[] = { if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; - list = find_internal_resource_by_dev(pci_dev); + list = find_internal_resource_by_pci_dev(pci_dev); if (list == NULL) { DRV_LOG(ERR, "Invalid device: %s", pci_dev->name); return -1; From patchwork Tue Oct 18 12:07:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 118410 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CE581A0560; Tue, 18 Oct 2022 14:59:05 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CB70A42825; Tue, 18 Oct 2022 14:58:37 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 2C13D41144 for ; Tue, 18 Oct 2022 14:58:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666097913; x=1697633913; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=l1Bu1k78MHDNX89TD8QTc9klvuQoflngCjx+ntbssTU=; b=hv03Ze3Qux9/rSUIcx1/3zxexu0XXbXjwmMHY+8tNTAyGa+i7lMMlVlV G8Vob7SRqvUr3SV/COPz0vbAjkq2wUY1DhpK7zNKwCGb7LkZA/T4tf6n8 txSRsNuiK3unFjQSWr82MG9TEnCwuTfKro8rMsprDYotEu4Y6OZ3ozYqp LOVlH7WA/JeqfIXREyoogqb+0U5bv4b3vM8k2S67vBeF+mqEOCSKtFU+P 7PuFIxGWShPiB1ptisKmHGMbAezF6t/5gINQBPpRQs1TYnt6h5Xoq314Z gqvPb6UF9B1DpYO+PWKW46ITDhNYyjurDsp31ewibSSk8qMPwVgOfU1Na A==; X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="307179481" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="307179481" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 05:58:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="733614522" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="733614522" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga002.fm.intel.com with ESMTP; 18 Oct 2022 05:58:24 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v8 08/12] vdpa/ifc: add internal API to get device Date: Tue, 18 Oct 2022 20:07:47 +0800 Message-Id: <1666094871-25721-9-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1666094871-25721-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1666094871-25721-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add new internal API "find_internal_resource_by_rte_dev" to get device. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- drivers/vdpa/ifc/ifcvf_vdpa.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index 73d04ed..c16e263 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -144,6 +144,29 @@ struct rte_vdpa_dev_info { return list; } +static struct internal_list * +find_internal_resource_by_rte_dev(struct rte_device *rte_dev) +{ + int found = 0; + struct internal_list *list; + + pthread_mutex_lock(&internal_list_lock); + + TAILQ_FOREACH(list, &internal_list, next) { + if (rte_dev == &list->internal->pdev->device) { + found = 1; + break; + } + } + + pthread_mutex_unlock(&internal_list_lock); + + if (!found) + return NULL; + + return list; +} + static int ifcvf_vfio_setup(struct ifcvf_internal *internal) { @@ -1398,10 +1421,11 @@ struct rte_vdpa_dev_info { { struct ifcvf_internal *internal; struct internal_list *list; + struct rte_device *rte_dev = vdev->device; - list = find_internal_resource_by_vdev(vdev); + list = find_internal_resource_by_rte_dev(rte_dev); if (list == NULL) { - DRV_LOG(ERR, "Invalid vDPA device: %p", vdev); + DRV_LOG(ERR, "Invalid rte device: %p", rte_dev); return -1; } From patchwork Tue Oct 18 12:07:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 118411 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8BD49A0560; Tue, 18 Oct 2022 14:59:11 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A850142836; Tue, 18 Oct 2022 14:58:38 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 0FD21427F4 for ; Tue, 18 Oct 2022 14:58:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666097914; x=1697633914; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=o8f1sWoMenWEr9E6VK1GnKrRShbXo+TYIdd8C7jFeK0=; b=H0ziioDEinnqNnL2fGHAnxjsiyEqlUpHdiCfRWvQAU6JjJRGXMdUjpIa eOtb2a6PiE0IOemJ6ol/xpLqrt7Sa27x3HxLUWY+ul1OChaQjbwLvZt2q /aXsHwTONcc0WYu0z4JMg0B8DRdt+y+pQ3ZefHr0bcVrBJzkpfcj22ATb GPgGiJmliwU55fOBwvSXM2BVMyWP2OoC5OAK1a8HFdpNc1TNKH123x9Iq SSnFRlphkKWmPSd6MCu/wCEc2bK4MjwenME7SCZnkfLSPQ5/JuDMBBJf+ ojWmldyjhBq5/HJS5USjVgdqPdaJTJ9d8DJxP2hW7vPWN9RJI5R1DZUwv w==; X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="307179494" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="307179494" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 05:58:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="733614536" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="733614536" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga002.fm.intel.com with ESMTP; 18 Oct 2022 05:58:27 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v8 09/12] vdpa/ifc: change some driver logic Date: Tue, 18 Oct 2022 20:07:48 +0800 Message-Id: <1666094871-25721-10-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1666094871-25721-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1666094871-25721-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Insert internal list element to internal list before register vdpa device, in order to call vdpa ops during vdpa device registration. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin --- drivers/vdpa/ifc/ifcvf_vdpa.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c index c16e263..8dfd493 100644 --- a/drivers/vdpa/ifc/ifcvf_vdpa.c +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c @@ -1737,17 +1737,20 @@ struct rte_vdpa_dev_info dev_info[] = { } internal->sw_lm = sw_fallback_lm; + pthread_mutex_lock(&internal_list_lock); + TAILQ_INSERT_TAIL(&internal_list, list, next); + pthread_mutex_unlock(&internal_list_lock); + internal->vdev = rte_vdpa_register_device(&pci_dev->device, dev_info[internal->hw.device_type].ops); if (internal->vdev == NULL) { DRV_LOG(ERR, "failed to register device %s", pci_dev->name); + pthread_mutex_lock(&internal_list_lock); + TAILQ_REMOVE(&internal_list, list, next); + pthread_mutex_unlock(&internal_list_lock); goto error; } - pthread_mutex_lock(&internal_list_lock); - TAILQ_INSERT_TAIL(&internal_list, list, next); - pthread_mutex_unlock(&internal_list_lock); - rte_atomic32_set(&internal->started, 1); update_datapath(internal); From patchwork Tue Oct 18 12:07:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 118412 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 66FE9A0560; Tue, 18 Oct 2022 14:59:17 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6B33342905; Tue, 18 Oct 2022 14:58:40 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id ADECB4114E for ; Tue, 18 Oct 2022 14:58:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666097914; x=1697633914; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=rmntRoYemBdBkVYAkztI3VB+WJhfNvZrUw0KxUA9b9Q=; b=g6J6w/FfCyN+xunNUsehizla2TroLW9EOGjbvoeg3vw6GMTd0Q47Q7nK VkuzGyH2eeN9R4i0aJvRie5AmLpqdqHfLO8IDl2iWnpcJlAmB2SLMPz8i EIPM7rTHPKBKTdO0G2DeWiZaf3GjfL74UikNbIt4zxOsRwhcux/dB8w+k esQr5dyrTPCSW9ztP98f1BX5FyZraq28s+whIXYO0BzrfDvAOkOzwg9GV QkH57A2KNRSTbp4x1t3X7S+9px3g4WBV7sUR46qYfHfrp1EwZqlkTGJCe 3x3ktYzX8dui+moRbPqI8uyayoqxz1iiQV4Nhaf6EpaDD8ByPuIhjpqL/ w==; X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="307179504" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="307179504" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 05:58:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="733614547" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="733614547" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga002.fm.intel.com with ESMTP; 18 Oct 2022 05:58:29 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v8 10/12] vhost: add type to rte vdpa device Date: Tue, 18 Oct 2022 20:07:49 +0800 Message-Id: <1666094871-25721-11-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1666094871-25721-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1666094871-25721-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add type to rte_vdpa_device to store device type. Call vdpa ops get_dev_type to fill type when register vdpa device. Signed-off-by: Andy Pei Reviewed-by: Maxime Coquelin --- lib/vhost/socket.c | 15 +-------------- lib/vhost/vdpa.c | 15 +++++++++++++++ lib/vhost/vdpa_driver.h | 2 ++ 3 files changed, 18 insertions(+), 14 deletions(-) diff --git a/lib/vhost/socket.c b/lib/vhost/socket.c index 608ae57..863a6f6 100644 --- a/lib/vhost/socket.c +++ b/lib/vhost/socket.c @@ -627,7 +627,6 @@ struct rte_vdpa_device * { struct vhost_user_socket *vsocket; struct rte_vdpa_device *vdpa_dev; - uint32_t vdpa_type = 0; int ret = 0; pthread_mutex_lock(&vhost_user.mutex); @@ -644,19 +643,7 @@ struct rte_vdpa_device * goto unlock_exit; } - if (vdpa_dev->ops->get_dev_type) { - ret = vdpa_dev->ops->get_dev_type(vdpa_dev, &vdpa_type); - if (ret) { - VHOST_LOG_CONFIG(path, ERR, - "failed to get vdpa dev type for socket file.\n"); - ret = -1; - goto unlock_exit; - } - } else { - vdpa_type = RTE_VHOST_VDPA_DEVICE_TYPE_NET; - } - - *type = vdpa_type; + *type = vdpa_dev->type; unlock_exit: pthread_mutex_unlock(&vhost_user.mutex); diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c index bb82857..577cb00 100644 --- a/lib/vhost/vdpa.c +++ b/lib/vhost/vdpa.c @@ -73,6 +73,7 @@ struct rte_vdpa_device * struct rte_vdpa_dev_ops *ops) { struct rte_vdpa_device *dev; + int ret = 0; if (ops == NULL) return NULL; @@ -101,6 +102,20 @@ struct rte_vdpa_device * dev->device = rte_dev; dev->ops = ops; + + if (ops->get_dev_type) { + ret = ops->get_dev_type(dev, &dev->type); + if (ret) { + VHOST_LOG_CONFIG(rte_dev->name, ERR, + "Failed to get vdpa dev type.\n"); + ret = -1; + goto out_unlock; + } + } else { + /** by default, we assume vdpa device is a net device */ + dev->type = RTE_VHOST_VDPA_DEVICE_TYPE_NET; + } + TAILQ_INSERT_TAIL(&vdpa_device_list, dev, next); out_unlock: rte_spinlock_unlock(&vdpa_device_list_lock); diff --git a/lib/vhost/vdpa_driver.h b/lib/vhost/vdpa_driver.h index 8b88a53..8db4ab9 100644 --- a/lib/vhost/vdpa_driver.h +++ b/lib/vhost/vdpa_driver.h @@ -92,6 +92,8 @@ struct rte_vdpa_device { struct rte_device *device; /** vdpa device operations */ struct rte_vdpa_dev_ops *ops; + /** vdpa device type: net, blk... */ + uint32_t type; }; /** From patchwork Tue Oct 18 12:07:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 118413 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EEC17A0560; Tue, 18 Oct 2022 14:59:22 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 369D442B70; Tue, 18 Oct 2022 14:58:41 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 5AACE4114E for ; Tue, 18 Oct 2022 14:58:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666097915; x=1697633915; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=+mntjR832AHqL/BPHzIKYTvjG68kpUjzvVIdw5lZBO4=; b=nZ/LBrMomYd2PZ7ytR/R1AXh74VX0HDpd9eGlG0RW6m3hkaPFvnWhTCS 8qhH2qx5OQVbkMx0z5MeVNX35HvTjAkCltGb4uNfgvoU0m+YjsXuBN3aD swWHs2Sx+s5ub815IJBrp4izgOMPnocOrJ3JULElyvSsFJwr+geCIYXaC 2LuJaHyEdMIGBR9AcPSZTrufl+Kcn86jVTlrJmTCYXxNjATWfQ1oyId0B b8P9zvhK0q0ImUYjn1LG87wDtIDZhVbydwADO2/xSk6jRvm466xdkZKF5 7+35kekahAwHisnDg7Geq3gR0zqfqLLOsCCPftCNV+rS36vJNT4AuufoW A==; X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="307179510" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="307179510" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 05:58:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="733614564" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="733614564" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga002.fm.intel.com with ESMTP; 18 Oct 2022 05:58:32 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v8 11/12] vhost: vDPA blk device gets ready when the first queue is ready Date: Tue, 18 Oct 2022 20:07:50 +0800 Message-Id: <1666094871-25721-12-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1666094871-25721-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1666094871-25721-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org When boot from virtio blk device, seabios in QEMU only enables one queue. To work in this scenario, vDPA BLK device back-end configure device when the first queue is ready. Signed-off-by: Andy Pei Signed-off-by: Huang Wei Reviewed-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vhost_user.c | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index cd65257..e0ff79d 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -1441,11 +1441,14 @@ } #define VIRTIO_BUILTIN_NUM_VQS_TO_BE_READY 2u +#define VIRTIO_BLK_NUM_VQS_TO_BE_READY 1u static int virtio_is_ready(struct virtio_net *dev) { + struct rte_vdpa_device *vdpa_dev; struct vhost_virtqueue *vq; + uint32_t vdpa_type; uint32_t i, nr_vring = dev->nr_vring; if (dev->flags & VIRTIO_DEV_READY) @@ -1454,13 +1457,22 @@ if (!dev->nr_vring) return 0; - if (dev->flags & VIRTIO_DEV_BUILTIN_VIRTIO_NET) { - nr_vring = VIRTIO_BUILTIN_NUM_VQS_TO_BE_READY; + vdpa_dev = dev->vdpa_dev; + if (vdpa_dev) + vdpa_type = vdpa_dev->type; + else + vdpa_type = -1; - if (dev->nr_vring < nr_vring) - return 0; + if (vdpa_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK) { + nr_vring = VIRTIO_BLK_NUM_VQS_TO_BE_READY; + } else { + if (dev->flags & VIRTIO_DEV_BUILTIN_VIRTIO_NET) + nr_vring = VIRTIO_BUILTIN_NUM_VQS_TO_BE_READY; } + if (dev->nr_vring < nr_vring) + return 0; + for (i = 0; i < nr_vring; i++) { vq = dev->virtqueue[i]; @@ -2958,7 +2970,6 @@ static int is_vring_iotlb(struct virtio_net *dev, int ret; int unlock_required = 0; bool handled; - uint32_t vdpa_type = 0; uint32_t request; uint32_t i; @@ -3170,17 +3181,7 @@ static int is_vring_iotlb(struct virtio_net *dev, if (!vdpa_dev) goto out; - if (vdpa_dev->ops->get_dev_type) { - ret = vdpa_dev->ops->get_dev_type(vdpa_dev, &vdpa_type); - if (ret) { - VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to get vdpa dev type.\n"); - ret = -1; - goto out; - } - } else { - vdpa_type = RTE_VHOST_VDPA_DEVICE_TYPE_NET; - } - if (vdpa_type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK + if (vdpa_dev->type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK && request != VHOST_USER_SET_VRING_CALL) goto out; From patchwork Tue Oct 18 12:07:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Pei, Andy" X-Patchwork-Id: 118414 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E3F30A0560; Tue, 18 Oct 2022 14:59:28 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 009FC42B74; Tue, 18 Oct 2022 14:58:42 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mails.dpdk.org (Postfix) with ESMTP id 4A29142802 for ; Tue, 18 Oct 2022 14:58:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666097916; x=1697633916; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=zpYT4ceyZdgCqIXy2mjswzZ6fTkZDyAKoc1OuH2VzE4=; b=laW1ASXkiKEiIOt24g9UzyUAYdANlOUPv0aB79q0AmIxfp503Wg62neY q5vuwVnk+HzD725WEr8BYgdelVjxU2PUwEiHCCRMAfLxoeEz1IPQduRrx pHqJCWxEpS/TRk6gZY6i81ko83+BAoOhkrlnfO8EzZsXv/clozt07+Jqv ZwhFEHo0K0DTWxAV36aoztthz+cxOC2cyO1ct1/oZcYDmHTL6i2GrXt9y yRrn1yznCU4rFXTI5dhJUbbS4Y0EWT54qLemtODAYYt9uk/mf5qvSJbdE W0X6cLZGbsV4t/lGxoDmfcU1ic0LOaR0m5vruJouPQ5abgRO9DdGPqrOD w==; X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="307179518" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="307179518" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2022 05:58:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10504"; a="733614577" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="733614577" Received: from dpdk-dipei.sh.intel.com ([10.67.110.251]) by fmsmga002.fm.intel.com with ESMTP; 18 Oct 2022 05:58:34 -0700 From: Andy Pei To: dev@dpdk.org Cc: chenbo.xia@intel.com, rosen.xu@intel.com, wei.huang@intel.com, gang.cao@intel.com, maxime.coquelin@redhat.com Subject: [PATCH v8 12/12] vhost: improve vDPA blk device configure condition Date: Tue, 18 Oct 2022 20:07:51 +0800 Message-Id: <1666094871-25721-13-git-send-email-andy.pei@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1666094871-25721-1-git-send-email-andy.pei@intel.com> References: <1661229305-240952-2-git-send-email-andy.pei@intel.com> <1666094871-25721-1-git-send-email-andy.pei@intel.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org To support multi-queue, configure device after call fd of all queues are set. Signed-off-by: Andy Pei Signed-off-by: Huang Wei Reviewed-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- lib/vhost/vhost_user.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index e0ff79d..9902ae9 100644 --- a/lib/vhost/vhost_user.c +++ b/lib/vhost/vhost_user.c @@ -2972,6 +2972,7 @@ static int is_vring_iotlb(struct virtio_net *dev, bool handled; uint32_t request; uint32_t i; + uint16_t blk_call_fd; dev = get_device(vid); if (dev == NULL) @@ -3181,9 +3182,15 @@ static int is_vring_iotlb(struct virtio_net *dev, if (!vdpa_dev) goto out; - if (vdpa_dev->type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK - && request != VHOST_USER_SET_VRING_CALL) - goto out; + if (vdpa_dev->type == RTE_VHOST_VDPA_DEVICE_TYPE_BLK) { + if (request == VHOST_USER_SET_VRING_CALL) { + blk_call_fd = ctx.msg.payload.u64 & VHOST_USER_VRING_IDX_MASK; + if (blk_call_fd != dev->nr_vring - 1) + goto out; + } else { + goto out; + } + } if (!(dev->flags & VIRTIO_DEV_VDPA_CONFIGURED)) { if (vdpa_dev->ops->dev_conf(dev->vid))