Patch Detail

HTTP 200 OK
Allow: GET, PUT, PATCH, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "id": 64339,
    "url": "http://patches.dpdk.org/api/patches/64339/?format=api",
    "web_url": "http://patches.dpdk.org/project/dpdk/patch/1578567617-3541-4-git-send-email-matan@mellanox.com/",
    "project": {
        "id": 1,
        "url": "http://patches.dpdk.org/api/projects/1/?format=api",
        "name": "DPDK",
        "link_name": "dpdk",
        "list_id": "dev.dpdk.org",
        "list_email": "dev@dpdk.org",
        "web_url": "http://core.dpdk.org",
        "scm_url": "git://dpdk.org/dpdk",
        "webscm_url": "http://git.dpdk.org/dpdk",
        "list_archive_url": "https://inbox.dpdk.org/dev",
        "list_archive_url_format": "https://inbox.dpdk.org/dev/{}",
        "commit_url_format": ""
    },
    "msgid": "<1578567617-3541-4-git-send-email-matan@mellanox.com>",
    "list_archive_url": "https://inbox.dpdk.org/dev/1578567617-3541-4-git-send-email-matan@mellanox.com",
    "date": "2020-01-09T11:00:17",
    "name": "[v2,3/3] drivers: move ifc driver to the vDPA class",
    "commit_ref": null,
    "pull_url": null,
    "state": "accepted",
    "archived": true,
    "hash": "352ff5637a6d9db5d034356f533f4608d5b0c1de",
    "submitter": {
        "id": 796,
        "url": "http://patches.dpdk.org/api/people/796/?format=api",
        "name": "Matan Azrad",
        "email": "matan@mellanox.com"
    },
    "delegate": {
        "id": 1,
        "url": "http://patches.dpdk.org/api/users/1/?format=api",
        "username": "tmonjalo",
        "first_name": "Thomas",
        "last_name": "Monjalon",
        "email": "thomas@monjalon.net"
    },
    "mbox": "http://patches.dpdk.org/project/dpdk/patch/1578567617-3541-4-git-send-email-matan@mellanox.com/mbox/",
    "series": [
        {
            "id": 8028,
            "url": "http://patches.dpdk.org/api/series/8028/?format=api",
            "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=8028",
            "date": "2020-01-09T11:00:14",
            "name": "Introduce new class for vDPA device drivers",
            "version": 2,
            "mbox": "http://patches.dpdk.org/series/8028/mbox/"
        }
    ],
    "comments": "http://patches.dpdk.org/api/patches/64339/comments/",
    "check": "success",
    "checks": "http://patches.dpdk.org/api/patches/64339/checks/",
    "tags": {},
    "related": [],
    "headers": {
        "Return-Path": "<dev-bounces@dpdk.org>",
        "X-Original-To": "patchwork@inbox.dpdk.org",
        "Delivered-To": "patchwork@inbox.dpdk.org",
        "Received": [
            "from dpdk.org (dpdk.org [92.243.14.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 23819A04F9;\n\tThu,  9 Jan 2020 12:01:07 +0100 (CET)",
            "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 84FE61DC74;\n\tThu,  9 Jan 2020 12:00:49 +0100 (CET)",
            "from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129])\n by dpdk.org (Postfix) with ESMTP id 03C201DC62\n for <dev@dpdk.org>; Thu,  9 Jan 2020 12:00:40 +0100 (CET)",
            "from Internal Mail-Server by MTLPINE1 (envelope-from\n asafp@mellanox.com)\n with ESMTPS (AES256-SHA encrypted); 9 Jan 2020 13:00:36 +0200",
            "from pegasus07.mtr.labs.mlnx (pegasus07.mtr.labs.mlnx\n [10.210.16.112])\n by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 009B0ZLl013007;\n Thu, 9 Jan 2020 13:00:36 +0200"
        ],
        "From": "Matan Azrad <matan@mellanox.com>",
        "To": "Maxime Coquelin <maxime.coquelin@redhat.com>,\n Tiwei Bie <tiwei.bie@intel.com>, Zhihong Wang <zhihong.wang@intel.com>,\n Xiao Wang <xiao.w.wang@intel.com>",
        "Cc": "Ferruh Yigit <ferruh.yigit@intel.com>, dev@dpdk.org,\n Thomas Monjalon <thomas@monjalon.net>,\n Andrew Rybchenko <arybchenko@solarflare.com>",
        "Date": "Thu,  9 Jan 2020 11:00:17 +0000",
        "Message-Id": "<1578567617-3541-4-git-send-email-matan@mellanox.com>",
        "X-Mailer": "git-send-email 1.8.3.1",
        "In-Reply-To": "<1578567617-3541-1-git-send-email-matan@mellanox.com>",
        "References": "<1577287161-10321-1-git-send-email-matan@mellanox.com>\n <1578567617-3541-1-git-send-email-matan@mellanox.com>",
        "Subject": "[dpdk-dev] [PATCH v2 3/3] drivers: move ifc driver to the vDPA class",
        "X-BeenThere": "dev@dpdk.org",
        "X-Mailman-Version": "2.1.15",
        "Precedence": "list",
        "List-Id": "DPDK patches and discussions <dev.dpdk.org>",
        "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>",
        "List-Archive": "<http://mails.dpdk.org/archives/dev/>",
        "List-Post": "<mailto:dev@dpdk.org>",
        "List-Help": "<mailto:dev-request@dpdk.org?subject=help>",
        "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>",
        "Errors-To": "dev-bounces@dpdk.org",
        "Sender": "\"dev\" <dev-bounces@dpdk.org>"
    },
    "content": "A new vDPA class was recently introduced.\n\nIFC driver implements the vDPA operations, hence it should be moved to\nthe vDPA class.\n\nMove it.\n\nSigned-off-by: Matan Azrad <matan@mellanox.com>\n---\n MAINTAINERS                              |   14 +-\n doc/guides/nics/features/ifcvf.ini       |    8 -\n doc/guides/nics/ifc.rst                  |  106 ---\n doc/guides/nics/index.rst                |    1 -\n doc/guides/vdpadevs/features/ifcvf.ini   |    8 +\n doc/guides/vdpadevs/ifc.rst              |  106 +++\n doc/guides/vdpadevs/index.rst            |    1 +\n drivers/net/Makefile                     |    3 -\n drivers/net/ifc/Makefile                 |   34 -\n drivers/net/ifc/base/ifcvf.c             |  329 --------\n drivers/net/ifc/base/ifcvf.h             |  162 ----\n drivers/net/ifc/base/ifcvf_osdep.h       |   52 --\n drivers/net/ifc/ifcvf_vdpa.c             | 1280 ------------------------------\n drivers/net/ifc/meson.build              |    9 -\n drivers/net/ifc/rte_pmd_ifc_version.map  |    3 -\n drivers/net/meson.build                  |    1 -\n drivers/vdpa/Makefile                    |    6 +\n drivers/vdpa/ifc/Makefile                |   34 +\n drivers/vdpa/ifc/base/ifcvf.c            |  329 ++++++++\n drivers/vdpa/ifc/base/ifcvf.h            |  162 ++++\n drivers/vdpa/ifc/base/ifcvf_osdep.h      |   52 ++\n drivers/vdpa/ifc/ifcvf_vdpa.c            | 1280 ++++++++++++++++++++++++++++++\n drivers/vdpa/ifc/meson.build             |    9 +\n drivers/vdpa/ifc/rte_pmd_ifc_version.map |    3 +\n drivers/vdpa/meson.build                 |    2 +-\n 25 files changed, 1997 insertions(+), 1997 deletions(-)\n delete mode 100644 doc/guides/nics/features/ifcvf.ini\n delete mode 100644 doc/guides/nics/ifc.rst\n create mode 100644 doc/guides/vdpadevs/features/ifcvf.ini\n create mode 100644 doc/guides/vdpadevs/ifc.rst\n delete mode 100644 drivers/net/ifc/Makefile\n delete mode 100644 drivers/net/ifc/base/ifcvf.c\n delete mode 100644 drivers/net/ifc/base/ifcvf.h\n delete mode 100644 drivers/net/ifc/base/ifcvf_osdep.h\n delete mode 100644 drivers/net/ifc/ifcvf_vdpa.c\n delete mode 100644 drivers/net/ifc/meson.build\n delete mode 100644 drivers/net/ifc/rte_pmd_ifc_version.map\n create mode 100644 drivers/vdpa/ifc/Makefile\n create mode 100644 drivers/vdpa/ifc/base/ifcvf.c\n create mode 100644 drivers/vdpa/ifc/base/ifcvf.h\n create mode 100644 drivers/vdpa/ifc/base/ifcvf_osdep.h\n create mode 100644 drivers/vdpa/ifc/ifcvf_vdpa.c\n create mode 100644 drivers/vdpa/ifc/meson.build\n create mode 100644 drivers/vdpa/ifc/rte_pmd_ifc_version.map",
    "diff": "diff --git a/MAINTAINERS b/MAINTAINERS\nindex 17c2df7..16facba 100644\n--- a/MAINTAINERS\n+++ b/MAINTAINERS\n@@ -679,14 +679,6 @@ T: git://dpdk.org/next/dpdk-next-net-intel\n F: drivers/net/iavf/\n F: doc/guides/nics/features/iavf*.ini\n \n-Intel ifc\n-M: Xiao Wang <xiao.w.wang@intel.com>\n-T: git://dpdk.org/next/dpdk-next-net-intel\n-F: drivers/net/ifc/\n-F: doc/guides/nics/ifc.rst\n-F: doc/guides/nics/features/ifc*.ini\n-\n-Intel ice\n M: Qiming Yang <qiming.yang@intel.com>\n M: Wenzhuo Lu <wenzhuo.lu@intel.com>\n T: git://dpdk.org/next/dpdk-next-net-intel\n@@ -1093,6 +1085,12 @@ vDPA Drivers\n ------------\n T: git://dpdk.org/next/dpdk-next-virtio\n \n+Intel ifc\n+M: Xiao Wang <xiao.w.wang@intel.com>\n+F: drivers/vdpa/ifc/\n+F: doc/guides/vdpadevs/ifc.rst\n+F: doc/guides/vdpadevs/features/ifcvf.ini\n+\n \n Eventdev Drivers\n ----------------\ndiff --git a/doc/guides/nics/features/ifcvf.ini b/doc/guides/nics/features/ifcvf.ini\ndeleted file mode 100644\nindex ef1fc47..0000000\n--- a/doc/guides/nics/features/ifcvf.ini\n+++ /dev/null\n@@ -1,8 +0,0 @@\n-;\n-; Supported features of the 'ifcvf' vDPA driver.\n-;\n-; Refer to default.ini for the full list of available PMD features.\n-;\n-[Features]\n-x86-32               = Y\n-x86-64               = Y\ndiff --git a/doc/guides/nics/ifc.rst b/doc/guides/nics/ifc.rst\ndeleted file mode 100644\nindex 12a2a34..0000000\n--- a/doc/guides/nics/ifc.rst\n+++ /dev/null\n@@ -1,106 +0,0 @@\n-..  SPDX-License-Identifier: BSD-3-Clause\n-    Copyright(c) 2018 Intel Corporation.\n-\n-IFCVF vDPA driver\n-=================\n-\n-The IFCVF vDPA (vhost data path acceleration) driver provides support for the\n-Intel FPGA 100G VF (IFCVF). IFCVF's datapath is virtio ring compatible, it\n-works as a HW vhost backend which can send/receive packets to/from virtio\n-directly by DMA. Besides, it supports dirty page logging and device state\n-report/restore, this driver enables its vDPA functionality.\n-\n-\n-Pre-Installation Configuration\n-------------------------------\n-\n-Config File Options\n-~~~~~~~~~~~~~~~~~~~\n-\n-The following option can be modified in the ``config`` file.\n-\n-- ``CONFIG_RTE_LIBRTE_IFC_PMD`` (default ``y`` for linux)\n-\n-  Toggle compilation of the ``librte_pmd_ifc`` driver.\n-\n-\n-IFCVF vDPA Implementation\n--------------------------\n-\n-IFCVF's vendor ID and device ID are same as that of virtio net pci device,\n-with its specific subsystem vendor ID and device ID. To let the device be\n-probed by IFCVF driver, adding \"vdpa=1\" parameter helps to specify that this\n-device is to be used in vDPA mode, rather than polling mode, virtio pmd will\n-skip when it detects this message. If no this parameter specified, device\n-will not be used as a vDPA device, and it will be driven by virtio pmd.\n-\n-Different VF devices serve different virtio frontends which are in different\n-VMs, so each VF needs to have its own DMA address translation service. During\n-the driver probe a new container is created for this device, with this\n-container vDPA driver can program DMA remapping table with the VM's memory\n-region information.\n-\n-The device argument \"sw-live-migration=1\" will configure the driver into SW\n-assisted live migration mode. In this mode, the driver will set up a SW relay\n-thread when LM happens, this thread will help device to log dirty pages. Thus\n-this mode does not require HW to implement a dirty page logging function block,\n-but will consume some percentage of CPU resource depending on the network\n-throughput. If no this parameter specified, driver will rely on device's logging\n-capability.\n-\n-Key IFCVF vDPA driver ops\n-~~~~~~~~~~~~~~~~~~~~~~~~~\n-\n-- ifcvf_dev_config:\n-  Enable VF data path with virtio information provided by vhost lib, including\n-  IOMMU programming to enable VF DMA to VM's memory, VFIO interrupt setup to\n-  route HW interrupt to virtio driver, create notify relay thread to translate\n-  virtio driver's kick to a MMIO write onto HW, HW queues configuration.\n-\n-  This function gets called to set up HW data path backend when virtio driver\n-  in VM gets ready.\n-\n-- ifcvf_dev_close:\n-  Revoke all the setup in ifcvf_dev_config.\n-\n-  This function gets called when virtio driver stops device in VM.\n-\n-To create a vhost port with IFC VF\n-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n-\n-- Create a vhost socket and assign a VF's device ID to this socket via\n-  vhost API. When QEMU vhost connection gets ready, the assigned VF will\n-  get configured automatically.\n-\n-\n-Features\n---------\n-\n-Features of the IFCVF driver are:\n-\n-- Compatibility with virtio 0.95 and 1.0.\n-- SW assisted vDPA live migration.\n-\n-\n-Prerequisites\n--------------\n-\n-- Platform with IOMMU feature. IFC VF needs address translation service to\n-  Rx/Tx directly with virtio driver in VM.\n-\n-\n-Limitations\n------------\n-\n-Dependency on vfio-pci\n-~~~~~~~~~~~~~~~~~~~~~~\n-\n-vDPA driver needs to setup VF MSIX interrupts, each queue's interrupt vector\n-is mapped to a callfd associated with a virtio ring. Currently only vfio-pci\n-allows multiple interrupts, so the IFCVF driver is dependent on vfio-pci.\n-\n-Live Migration with VIRTIO_NET_F_GUEST_ANNOUNCE\n-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n-\n-IFC VF doesn't support RARP packet generation, virtio frontend supporting\n-VIRTIO_NET_F_GUEST_ANNOUNCE feature can help to do that.\ndiff --git a/doc/guides/nics/index.rst b/doc/guides/nics/index.rst\nindex d61c27f..8c540c0 100644\n--- a/doc/guides/nics/index.rst\n+++ b/doc/guides/nics/index.rst\n@@ -31,7 +31,6 @@ Network Interface Controller Drivers\n     hns3\n     i40e\n     ice\n-    ifc\n     igb\n     ipn3ke\n     ixgbe\ndiff --git a/doc/guides/vdpadevs/features/ifcvf.ini b/doc/guides/vdpadevs/features/ifcvf.ini\nnew file mode 100644\nindex 0000000..ef1fc47\n--- /dev/null\n+++ b/doc/guides/vdpadevs/features/ifcvf.ini\n@@ -0,0 +1,8 @@\n+;\n+; Supported features of the 'ifcvf' vDPA driver.\n+;\n+; Refer to default.ini for the full list of available PMD features.\n+;\n+[Features]\n+x86-32               = Y\n+x86-64               = Y\ndiff --git a/doc/guides/vdpadevs/ifc.rst b/doc/guides/vdpadevs/ifc.rst\nnew file mode 100644\nindex 0000000..12a2a34\n--- /dev/null\n+++ b/doc/guides/vdpadevs/ifc.rst\n@@ -0,0 +1,106 @@\n+..  SPDX-License-Identifier: BSD-3-Clause\n+    Copyright(c) 2018 Intel Corporation.\n+\n+IFCVF vDPA driver\n+=================\n+\n+The IFCVF vDPA (vhost data path acceleration) driver provides support for the\n+Intel FPGA 100G VF (IFCVF). IFCVF's datapath is virtio ring compatible, it\n+works as a HW vhost backend which can send/receive packets to/from virtio\n+directly by DMA. Besides, it supports dirty page logging and device state\n+report/restore, this driver enables its vDPA functionality.\n+\n+\n+Pre-Installation Configuration\n+------------------------------\n+\n+Config File Options\n+~~~~~~~~~~~~~~~~~~~\n+\n+The following option can be modified in the ``config`` file.\n+\n+- ``CONFIG_RTE_LIBRTE_IFC_PMD`` (default ``y`` for linux)\n+\n+  Toggle compilation of the ``librte_pmd_ifc`` driver.\n+\n+\n+IFCVF vDPA Implementation\n+-------------------------\n+\n+IFCVF's vendor ID and device ID are same as that of virtio net pci device,\n+with its specific subsystem vendor ID and device ID. To let the device be\n+probed by IFCVF driver, adding \"vdpa=1\" parameter helps to specify that this\n+device is to be used in vDPA mode, rather than polling mode, virtio pmd will\n+skip when it detects this message. If no this parameter specified, device\n+will not be used as a vDPA device, and it will be driven by virtio pmd.\n+\n+Different VF devices serve different virtio frontends which are in different\n+VMs, so each VF needs to have its own DMA address translation service. During\n+the driver probe a new container is created for this device, with this\n+container vDPA driver can program DMA remapping table with the VM's memory\n+region information.\n+\n+The device argument \"sw-live-migration=1\" will configure the driver into SW\n+assisted live migration mode. In this mode, the driver will set up a SW relay\n+thread when LM happens, this thread will help device to log dirty pages. Thus\n+this mode does not require HW to implement a dirty page logging function block,\n+but will consume some percentage of CPU resource depending on the network\n+throughput. If no this parameter specified, driver will rely on device's logging\n+capability.\n+\n+Key IFCVF vDPA driver ops\n+~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+- ifcvf_dev_config:\n+  Enable VF data path with virtio information provided by vhost lib, including\n+  IOMMU programming to enable VF DMA to VM's memory, VFIO interrupt setup to\n+  route HW interrupt to virtio driver, create notify relay thread to translate\n+  virtio driver's kick to a MMIO write onto HW, HW queues configuration.\n+\n+  This function gets called to set up HW data path backend when virtio driver\n+  in VM gets ready.\n+\n+- ifcvf_dev_close:\n+  Revoke all the setup in ifcvf_dev_config.\n+\n+  This function gets called when virtio driver stops device in VM.\n+\n+To create a vhost port with IFC VF\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+- Create a vhost socket and assign a VF's device ID to this socket via\n+  vhost API. When QEMU vhost connection gets ready, the assigned VF will\n+  get configured automatically.\n+\n+\n+Features\n+--------\n+\n+Features of the IFCVF driver are:\n+\n+- Compatibility with virtio 0.95 and 1.0.\n+- SW assisted vDPA live migration.\n+\n+\n+Prerequisites\n+-------------\n+\n+- Platform with IOMMU feature. IFC VF needs address translation service to\n+  Rx/Tx directly with virtio driver in VM.\n+\n+\n+Limitations\n+-----------\n+\n+Dependency on vfio-pci\n+~~~~~~~~~~~~~~~~~~~~~~\n+\n+vDPA driver needs to setup VF MSIX interrupts, each queue's interrupt vector\n+is mapped to a callfd associated with a virtio ring. Currently only vfio-pci\n+allows multiple interrupts, so the IFCVF driver is dependent on vfio-pci.\n+\n+Live Migration with VIRTIO_NET_F_GUEST_ANNOUNCE\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+IFC VF doesn't support RARP packet generation, virtio frontend supporting\n+VIRTIO_NET_F_GUEST_ANNOUNCE feature can help to do that.\ndiff --git a/doc/guides/vdpadevs/index.rst b/doc/guides/vdpadevs/index.rst\nindex 89e2b03..6cf0827 100644\n--- a/doc/guides/vdpadevs/index.rst\n+++ b/doc/guides/vdpadevs/index.rst\n@@ -12,3 +12,4 @@ which can be used from an application through vhost API.\n     :numbered:\n \n     features_overview\n+    ifc\ndiff --git a/drivers/net/Makefile b/drivers/net/Makefile\nindex cee3036..cca3c44 100644\n--- a/drivers/net/Makefile\n+++ b/drivers/net/Makefile\n@@ -71,9 +71,6 @@ endif # $(CONFIG_RTE_LIBRTE_SCHED)\n \n ifeq ($(CONFIG_RTE_LIBRTE_VHOST),y)\n DIRS-$(CONFIG_RTE_LIBRTE_PMD_VHOST) += vhost\n-ifeq ($(CONFIG_RTE_EAL_VFIO),y)\n-DIRS-$(CONFIG_RTE_LIBRTE_IFC_PMD) += ifc\n-endif\n endif # $(CONFIG_RTE_LIBRTE_VHOST)\n \n ifeq ($(CONFIG_RTE_LIBRTE_MVPP2_PMD),y)\ndiff --git a/drivers/net/ifc/Makefile b/drivers/net/ifc/Makefile\ndeleted file mode 100644\nindex fe227b8..0000000\n--- a/drivers/net/ifc/Makefile\n+++ /dev/null\n@@ -1,34 +0,0 @@\n-# SPDX-License-Identifier: BSD-3-Clause\n-# Copyright(c) 2018 Intel Corporation\n-\n-include $(RTE_SDK)/mk/rte.vars.mk\n-\n-#\n-# library name\n-#\n-LIB = librte_pmd_ifc.a\n-\n-LDLIBS += -lpthread\n-LDLIBS += -lrte_eal -lrte_pci -lrte_vhost -lrte_bus_pci\n-LDLIBS += -lrte_kvargs\n-\n-CFLAGS += -O3\n-CFLAGS += $(WERROR_FLAGS)\n-CFLAGS += -DALLOW_EXPERIMENTAL_API\n-\n-#\n-# Add extra flags for base driver source files to disable warnings in them\n-#\n-BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))\n-\n-VPATH += $(SRCDIR)/base\n-\n-EXPORT_MAP := rte_pmd_ifc_version.map\n-\n-#\n-# all source are stored in SRCS-y\n-#\n-SRCS-$(CONFIG_RTE_LIBRTE_IFC_PMD) += ifcvf_vdpa.c\n-SRCS-$(CONFIG_RTE_LIBRTE_IFC_PMD) += ifcvf.c\n-\n-include $(RTE_SDK)/mk/rte.lib.mk\ndiff --git a/drivers/net/ifc/base/ifcvf.c b/drivers/net/ifc/base/ifcvf.c\ndeleted file mode 100644\nindex 3c0b2df..0000000\n--- a/drivers/net/ifc/base/ifcvf.c\n+++ /dev/null\n@@ -1,329 +0,0 @@\n-/* SPDX-License-Identifier: BSD-3-Clause\n- * Copyright(c) 2018 Intel Corporation\n- */\n-\n-#include \"ifcvf.h\"\n-#include \"ifcvf_osdep.h\"\n-\n-STATIC void *\n-get_cap_addr(struct ifcvf_hw *hw, struct ifcvf_pci_cap *cap)\n-{\n-\tu8 bar = cap->bar;\n-\tu32 length = cap->length;\n-\tu32 offset = cap->offset;\n-\n-\tif (bar > IFCVF_PCI_MAX_RESOURCE - 1) {\n-\t\tDEBUGOUT(\"invalid bar: %u\\n\", bar);\n-\t\treturn NULL;\n-\t}\n-\n-\tif (offset + length < offset) {\n-\t\tDEBUGOUT(\"offset(%u) + length(%u) overflows\\n\",\n-\t\t\toffset, length);\n-\t\treturn NULL;\n-\t}\n-\n-\tif (offset + length > hw->mem_resource[cap->bar].len) {\n-\t\tDEBUGOUT(\"offset(%u) + length(%u) overflows bar length(%u)\",\n-\t\t\toffset, length, (u32)hw->mem_resource[cap->bar].len);\n-\t\treturn NULL;\n-\t}\n-\n-\treturn hw->mem_resource[bar].addr + offset;\n-}\n-\n-int\n-ifcvf_init_hw(struct ifcvf_hw *hw, PCI_DEV *dev)\n-{\n-\tint ret;\n-\tu8 pos;\n-\tstruct ifcvf_pci_cap cap;\n-\n-\tret = PCI_READ_CONFIG_BYTE(dev, &pos, PCI_CAPABILITY_LIST);\n-\tif (ret < 0) {\n-\t\tDEBUGOUT(\"failed to read pci capability list\\n\");\n-\t\treturn -1;\n-\t}\n-\n-\twhile (pos) {\n-\t\tret = PCI_READ_CONFIG_RANGE(dev, (u32 *)&cap,\n-\t\t\t\tsizeof(cap), pos);\n-\t\tif (ret < 0) {\n-\t\t\tDEBUGOUT(\"failed to read cap at pos: %x\", pos);\n-\t\t\tbreak;\n-\t\t}\n-\n-\t\tif (cap.cap_vndr != PCI_CAP_ID_VNDR)\n-\t\t\tgoto next;\n-\n-\t\tDEBUGOUT(\"cfg type: %u, bar: %u, offset: %u, \"\n-\t\t\t\t\"len: %u\\n\", cap.cfg_type, cap.bar,\n-\t\t\t\tcap.offset, cap.length);\n-\n-\t\tswitch (cap.cfg_type) {\n-\t\tcase IFCVF_PCI_CAP_COMMON_CFG:\n-\t\t\thw->common_cfg = get_cap_addr(hw, &cap);\n-\t\t\tbreak;\n-\t\tcase IFCVF_PCI_CAP_NOTIFY_CFG:\n-\t\t\tPCI_READ_CONFIG_DWORD(dev, &hw->notify_off_multiplier,\n-\t\t\t\t\tpos + sizeof(cap));\n-\t\t\thw->notify_base = get_cap_addr(hw, &cap);\n-\t\t\thw->notify_region = cap.bar;\n-\t\t\tbreak;\n-\t\tcase IFCVF_PCI_CAP_ISR_CFG:\n-\t\t\thw->isr = get_cap_addr(hw, &cap);\n-\t\t\tbreak;\n-\t\tcase IFCVF_PCI_CAP_DEVICE_CFG:\n-\t\t\thw->dev_cfg = get_cap_addr(hw, &cap);\n-\t\t\tbreak;\n-\t\t}\n-next:\n-\t\tpos = cap.cap_next;\n-\t}\n-\n-\thw->lm_cfg = hw->mem_resource[4].addr;\n-\n-\tif (hw->common_cfg == NULL || hw->notify_base == NULL ||\n-\t\t\thw->isr == NULL || hw->dev_cfg == NULL) {\n-\t\tDEBUGOUT(\"capability incomplete\\n\");\n-\t\treturn -1;\n-\t}\n-\n-\tDEBUGOUT(\"capability mapping:\\ncommon cfg: %p\\n\"\n-\t\t\t\"notify base: %p\\nisr cfg: %p\\ndevice cfg: %p\\n\"\n-\t\t\t\"multiplier: %u\\n\",\n-\t\t\thw->common_cfg, hw->dev_cfg,\n-\t\t\thw->isr, hw->notify_base,\n-\t\t\thw->notify_off_multiplier);\n-\n-\treturn 0;\n-}\n-\n-STATIC u8\n-ifcvf_get_status(struct ifcvf_hw *hw)\n-{\n-\treturn IFCVF_READ_REG8(&hw->common_cfg->device_status);\n-}\n-\n-STATIC void\n-ifcvf_set_status(struct ifcvf_hw *hw, u8 status)\n-{\n-\tIFCVF_WRITE_REG8(status, &hw->common_cfg->device_status);\n-}\n-\n-STATIC void\n-ifcvf_reset(struct ifcvf_hw *hw)\n-{\n-\tifcvf_set_status(hw, 0);\n-\n-\t/* flush status write */\n-\twhile (ifcvf_get_status(hw))\n-\t\tmsec_delay(1);\n-}\n-\n-STATIC void\n-ifcvf_add_status(struct ifcvf_hw *hw, u8 status)\n-{\n-\tif (status != 0)\n-\t\tstatus |= ifcvf_get_status(hw);\n-\n-\tifcvf_set_status(hw, status);\n-\tifcvf_get_status(hw);\n-}\n-\n-u64\n-ifcvf_get_features(struct ifcvf_hw *hw)\n-{\n-\tu32 features_lo, features_hi;\n-\tstruct ifcvf_pci_common_cfg *cfg = hw->common_cfg;\n-\n-\tIFCVF_WRITE_REG32(0, &cfg->device_feature_select);\n-\tfeatures_lo = IFCVF_READ_REG32(&cfg->device_feature);\n-\n-\tIFCVF_WRITE_REG32(1, &cfg->device_feature_select);\n-\tfeatures_hi = IFCVF_READ_REG32(&cfg->device_feature);\n-\n-\treturn ((u64)features_hi << 32) | features_lo;\n-}\n-\n-STATIC void\n-ifcvf_set_features(struct ifcvf_hw *hw, u64 features)\n-{\n-\tstruct ifcvf_pci_common_cfg *cfg = hw->common_cfg;\n-\n-\tIFCVF_WRITE_REG32(0, &cfg->guest_feature_select);\n-\tIFCVF_WRITE_REG32(features & ((1ULL << 32) - 1), &cfg->guest_feature);\n-\n-\tIFCVF_WRITE_REG32(1, &cfg->guest_feature_select);\n-\tIFCVF_WRITE_REG32(features >> 32, &cfg->guest_feature);\n-}\n-\n-STATIC int\n-ifcvf_config_features(struct ifcvf_hw *hw)\n-{\n-\tu64 host_features;\n-\n-\thost_features = ifcvf_get_features(hw);\n-\thw->req_features &= host_features;\n-\n-\tifcvf_set_features(hw, hw->req_features);\n-\tifcvf_add_status(hw, IFCVF_CONFIG_STATUS_FEATURES_OK);\n-\n-\tif (!(ifcvf_get_status(hw) & IFCVF_CONFIG_STATUS_FEATURES_OK)) {\n-\t\tDEBUGOUT(\"failed to set FEATURES_OK status\\n\");\n-\t\treturn -1;\n-\t}\n-\n-\treturn 0;\n-}\n-\n-STATIC void\n-io_write64_twopart(u64 val, u32 *lo, u32 *hi)\n-{\n-\tIFCVF_WRITE_REG32(val & ((1ULL << 32) - 1), lo);\n-\tIFCVF_WRITE_REG32(val >> 32, hi);\n-}\n-\n-STATIC int\n-ifcvf_hw_enable(struct ifcvf_hw *hw)\n-{\n-\tstruct ifcvf_pci_common_cfg *cfg;\n-\tu8 *lm_cfg;\n-\tu32 i;\n-\tu16 notify_off;\n-\n-\tcfg = hw->common_cfg;\n-\tlm_cfg = hw->lm_cfg;\n-\n-\tIFCVF_WRITE_REG16(0, &cfg->msix_config);\n-\tif (IFCVF_READ_REG16(&cfg->msix_config) == IFCVF_MSI_NO_VECTOR) {\n-\t\tDEBUGOUT(\"msix vec alloc failed for device config\\n\");\n-\t\treturn -1;\n-\t}\n-\n-\tfor (i = 0; i < hw->nr_vring; i++) {\n-\t\tIFCVF_WRITE_REG16(i, &cfg->queue_select);\n-\t\tio_write64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo,\n-\t\t\t\t&cfg->queue_desc_hi);\n-\t\tio_write64_twopart(hw->vring[i].avail, &cfg->queue_avail_lo,\n-\t\t\t\t&cfg->queue_avail_hi);\n-\t\tio_write64_twopart(hw->vring[i].used, &cfg->queue_used_lo,\n-\t\t\t\t&cfg->queue_used_hi);\n-\t\tIFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size);\n-\n-\t\t*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +\n-\t\t\t\t(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) =\n-\t\t\t(u32)hw->vring[i].last_avail_idx |\n-\t\t\t((u32)hw->vring[i].last_used_idx << 16);\n-\n-\t\tIFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector);\n-\t\tif (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==\n-\t\t\t\tIFCVF_MSI_NO_VECTOR) {\n-\t\t\tDEBUGOUT(\"queue %u, msix vec alloc failed\\n\",\n-\t\t\t\t\ti);\n-\t\t\treturn -1;\n-\t\t}\n-\n-\t\tnotify_off = IFCVF_READ_REG16(&cfg->queue_notify_off);\n-\t\thw->notify_addr[i] = (void *)((u8 *)hw->notify_base +\n-\t\t\t\tnotify_off * hw->notify_off_multiplier);\n-\t\tIFCVF_WRITE_REG16(1, &cfg->queue_enable);\n-\t}\n-\n-\treturn 0;\n-}\n-\n-STATIC void\n-ifcvf_hw_disable(struct ifcvf_hw *hw)\n-{\n-\tu32 i;\n-\tstruct ifcvf_pci_common_cfg *cfg;\n-\tu32 ring_state;\n-\n-\tcfg = hw->common_cfg;\n-\n-\tIFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->msix_config);\n-\tfor (i = 0; i < hw->nr_vring; i++) {\n-\t\tIFCVF_WRITE_REG16(i, &cfg->queue_select);\n-\t\tIFCVF_WRITE_REG16(0, &cfg->queue_enable);\n-\t\tIFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector);\n-\t\tring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET +\n-\t\t\t\t(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4);\n-\t\thw->vring[i].last_avail_idx = (u16)(ring_state >> 16);\n-\t\thw->vring[i].last_used_idx = (u16)(ring_state >> 16);\n-\t}\n-}\n-\n-int\n-ifcvf_start_hw(struct ifcvf_hw *hw)\n-{\n-\tifcvf_reset(hw);\n-\tifcvf_add_status(hw, IFCVF_CONFIG_STATUS_ACK);\n-\tifcvf_add_status(hw, IFCVF_CONFIG_STATUS_DRIVER);\n-\n-\tif (ifcvf_config_features(hw) < 0)\n-\t\treturn -1;\n-\n-\tif (ifcvf_hw_enable(hw) < 0)\n-\t\treturn -1;\n-\n-\tifcvf_add_status(hw, IFCVF_CONFIG_STATUS_DRIVER_OK);\n-\treturn 0;\n-}\n-\n-void\n-ifcvf_stop_hw(struct ifcvf_hw *hw)\n-{\n-\tifcvf_hw_disable(hw);\n-\tifcvf_reset(hw);\n-}\n-\n-void\n-ifcvf_enable_logging(struct ifcvf_hw *hw, u64 log_base, u64 log_size)\n-{\n-\tu8 *lm_cfg;\n-\n-\tlm_cfg = hw->lm_cfg;\n-\n-\t*(u32 *)(lm_cfg + IFCVF_LM_BASE_ADDR_LOW) =\n-\t\tlog_base & IFCVF_32_BIT_MASK;\n-\n-\t*(u32 *)(lm_cfg + IFCVF_LM_BASE_ADDR_HIGH) =\n-\t\t(log_base >> 32) & IFCVF_32_BIT_MASK;\n-\n-\t*(u32 *)(lm_cfg + IFCVF_LM_END_ADDR_LOW) =\n-\t\t(log_base + log_size) & IFCVF_32_BIT_MASK;\n-\n-\t*(u32 *)(lm_cfg + IFCVF_LM_END_ADDR_HIGH) =\n-\t\t((log_base + log_size) >> 32) & IFCVF_32_BIT_MASK;\n-\n-\t*(u32 *)(lm_cfg + IFCVF_LM_LOGGING_CTRL) = IFCVF_LM_ENABLE_VF;\n-}\n-\n-void\n-ifcvf_disable_logging(struct ifcvf_hw *hw)\n-{\n-\tu8 *lm_cfg;\n-\n-\tlm_cfg = hw->lm_cfg;\n-\t*(u32 *)(lm_cfg + IFCVF_LM_LOGGING_CTRL) = IFCVF_LM_DISABLE;\n-}\n-\n-void\n-ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid)\n-{\n-\tIFCVF_WRITE_REG16(qid, hw->notify_addr[qid]);\n-}\n-\n-u8\n-ifcvf_get_notify_region(struct ifcvf_hw *hw)\n-{\n-\treturn hw->notify_region;\n-}\n-\n-u64\n-ifcvf_get_queue_notify_off(struct ifcvf_hw *hw, int qid)\n-{\n-\treturn (u8 *)hw->notify_addr[qid] -\n-\t\t(u8 *)hw->mem_resource[hw->notify_region].addr;\n-}\ndiff --git a/drivers/net/ifc/base/ifcvf.h b/drivers/net/ifc/base/ifcvf.h\ndeleted file mode 100644\nindex 9be2770..0000000\n--- a/drivers/net/ifc/base/ifcvf.h\n+++ /dev/null\n@@ -1,162 +0,0 @@\n-/* SPDX-License-Identifier: BSD-3-Clause\n- * Copyright(c) 2018 Intel Corporation\n- */\n-\n-#ifndef _IFCVF_H_\n-#define _IFCVF_H_\n-\n-#include \"ifcvf_osdep.h\"\n-\n-#define IFCVF_VENDOR_ID\t\t0x1AF4\n-#define IFCVF_DEVICE_ID\t\t0x1041\n-#define IFCVF_SUBSYS_VENDOR_ID\t0x8086\n-#define IFCVF_SUBSYS_DEVICE_ID\t0x001A\n-\n-#define IFCVF_MAX_QUEUES\t\t1\n-#define VIRTIO_F_IOMMU_PLATFORM\t\t33\n-\n-/* Common configuration */\n-#define IFCVF_PCI_CAP_COMMON_CFG\t1\n-/* Notifications */\n-#define IFCVF_PCI_CAP_NOTIFY_CFG\t2\n-/* ISR Status */\n-#define IFCVF_PCI_CAP_ISR_CFG\t\t3\n-/* Device specific configuration */\n-#define IFCVF_PCI_CAP_DEVICE_CFG\t4\n-/* PCI configuration access */\n-#define IFCVF_PCI_CAP_PCI_CFG\t\t5\n-\n-#define IFCVF_CONFIG_STATUS_RESET     0x00\n-#define IFCVF_CONFIG_STATUS_ACK       0x01\n-#define IFCVF_CONFIG_STATUS_DRIVER    0x02\n-#define IFCVF_CONFIG_STATUS_DRIVER_OK 0x04\n-#define IFCVF_CONFIG_STATUS_FEATURES_OK 0x08\n-#define IFCVF_CONFIG_STATUS_FAILED    0x80\n-\n-#define IFCVF_MSI_NO_VECTOR\t0xffff\n-#define IFCVF_PCI_MAX_RESOURCE\t6\n-\n-#define IFCVF_LM_CFG_SIZE\t\t0x40\n-#define IFCVF_LM_RING_STATE_OFFSET\t0x20\n-\n-#define IFCVF_LM_LOGGING_CTRL\t\t0x0\n-\n-#define IFCVF_LM_BASE_ADDR_LOW\t\t0x10\n-#define IFCVF_LM_BASE_ADDR_HIGH\t\t0x14\n-#define IFCVF_LM_END_ADDR_LOW\t\t0x18\n-#define IFCVF_LM_END_ADDR_HIGH\t\t0x1c\n-\n-#define IFCVF_LM_DISABLE\t\t0x0\n-#define IFCVF_LM_ENABLE_VF\t\t0x1\n-#define IFCVF_LM_ENABLE_PF\t\t0x3\n-#define IFCVF_LOG_BASE\t\t\t0x100000000000\n-#define IFCVF_MEDIATED_VRING\t\t0x200000000000\n-\n-#define IFCVF_32_BIT_MASK\t\t0xffffffff\n-\n-\n-struct ifcvf_pci_cap {\n-\tu8 cap_vndr;            /* Generic PCI field: PCI_CAP_ID_VNDR */\n-\tu8 cap_next;            /* Generic PCI field: next ptr. */\n-\tu8 cap_len;             /* Generic PCI field: capability length */\n-\tu8 cfg_type;            /* Identifies the structure. */\n-\tu8 bar;                 /* Where to find it. */\n-\tu8 padding[3];          /* Pad to full dword. */\n-\tu32 offset;             /* Offset within bar. */\n-\tu32 length;             /* Length of the structure, in bytes. */\n-};\n-\n-struct ifcvf_pci_notify_cap {\n-\tstruct ifcvf_pci_cap cap;\n-\tu32 notify_off_multiplier;  /* Multiplier for queue_notify_off. */\n-};\n-\n-struct ifcvf_pci_common_cfg {\n-\t/* About the whole device. */\n-\tu32 device_feature_select;\n-\tu32 device_feature;\n-\tu32 guest_feature_select;\n-\tu32 guest_feature;\n-\tu16 msix_config;\n-\tu16 num_queues;\n-\tu8 device_status;\n-\tu8 config_generation;\n-\n-\t/* About a specific virtqueue. */\n-\tu16 queue_select;\n-\tu16 queue_size;\n-\tu16 queue_msix_vector;\n-\tu16 queue_enable;\n-\tu16 queue_notify_off;\n-\tu32 queue_desc_lo;\n-\tu32 queue_desc_hi;\n-\tu32 queue_avail_lo;\n-\tu32 queue_avail_hi;\n-\tu32 queue_used_lo;\n-\tu32 queue_used_hi;\n-};\n-\n-struct ifcvf_net_config {\n-\tu8    mac[6];\n-\tu16   status;\n-\tu16   max_virtqueue_pairs;\n-} __attribute__((packed));\n-\n-struct ifcvf_pci_mem_resource {\n-\tu64      phys_addr; /**< Physical address, 0 if not resource. */\n-\tu64      len;       /**< Length of the resource. */\n-\tu8       *addr;     /**< Virtual address, NULL when not mapped. */\n-};\n-\n-struct vring_info {\n-\tu64 desc;\n-\tu64 avail;\n-\tu64 used;\n-\tu16 size;\n-\tu16 last_avail_idx;\n-\tu16 last_used_idx;\n-};\n-\n-struct ifcvf_hw {\n-\tu64    req_features;\n-\tu8     notify_region;\n-\tu32    notify_off_multiplier;\n-\tstruct ifcvf_pci_common_cfg *common_cfg;\n-\tstruct ifcvf_net_config *dev_cfg;\n-\tu8     *isr;\n-\tu16    *notify_base;\n-\tu16    *notify_addr[IFCVF_MAX_QUEUES * 2];\n-\tu8     *lm_cfg;\n-\tstruct vring_info vring[IFCVF_MAX_QUEUES * 2];\n-\tu8 nr_vring;\n-\tstruct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE];\n-};\n-\n-int\n-ifcvf_init_hw(struct ifcvf_hw *hw, PCI_DEV *dev);\n-\n-u64\n-ifcvf_get_features(struct ifcvf_hw *hw);\n-\n-int\n-ifcvf_start_hw(struct ifcvf_hw *hw);\n-\n-void\n-ifcvf_stop_hw(struct ifcvf_hw *hw);\n-\n-void\n-ifcvf_enable_logging(struct ifcvf_hw *hw, u64 log_base, u64 log_size);\n-\n-void\n-ifcvf_disable_logging(struct ifcvf_hw *hw);\n-\n-void\n-ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid);\n-\n-u8\n-ifcvf_get_notify_region(struct ifcvf_hw *hw);\n-\n-u64\n-ifcvf_get_queue_notify_off(struct ifcvf_hw *hw, int qid);\n-\n-#endif /* _IFCVF_H_ */\ndiff --git a/drivers/net/ifc/base/ifcvf_osdep.h b/drivers/net/ifc/base/ifcvf_osdep.h\ndeleted file mode 100644\nindex 6aef25e..0000000\n--- a/drivers/net/ifc/base/ifcvf_osdep.h\n+++ /dev/null\n@@ -1,52 +0,0 @@\n-/* SPDX-License-Identifier: BSD-3-Clause\n- * Copyright(c) 2018 Intel Corporation\n- */\n-\n-#ifndef _IFCVF_OSDEP_H_\n-#define _IFCVF_OSDEP_H_\n-\n-#include <stdint.h>\n-#include <linux/pci_regs.h>\n-\n-#include <rte_cycles.h>\n-#include <rte_pci.h>\n-#include <rte_bus_pci.h>\n-#include <rte_log.h>\n-#include <rte_io.h>\n-\n-#define DEBUGOUT(S, args...)    RTE_LOG(DEBUG, PMD, S, ##args)\n-#define STATIC                  static\n-\n-#define msec_delay(x)\trte_delay_us_sleep(1000 * (x))\n-\n-#define IFCVF_READ_REG8(reg)\t\trte_read8(reg)\n-#define IFCVF_WRITE_REG8(val, reg)\trte_write8((val), (reg))\n-#define IFCVF_READ_REG16(reg)\t\trte_read16(reg)\n-#define IFCVF_WRITE_REG16(val, reg)\trte_write16((val), (reg))\n-#define IFCVF_READ_REG32(reg)\t\trte_read32(reg)\n-#define IFCVF_WRITE_REG32(val, reg)\trte_write32((val), (reg))\n-\n-typedef struct rte_pci_device PCI_DEV;\n-\n-#define PCI_READ_CONFIG_BYTE(dev, val, where) \\\n-\trte_pci_read_config(dev, val, 1, where)\n-\n-#define PCI_READ_CONFIG_DWORD(dev, val, where) \\\n-\trte_pci_read_config(dev, val, 4, where)\n-\n-typedef uint8_t    u8;\n-typedef int8_t     s8;\n-typedef uint16_t   u16;\n-typedef int16_t    s16;\n-typedef uint32_t   u32;\n-typedef int32_t    s32;\n-typedef int64_t    s64;\n-typedef uint64_t   u64;\n-\n-static inline int\n-PCI_READ_CONFIG_RANGE(PCI_DEV *dev, uint32_t *val, int size, int where)\n-{\n-\treturn rte_pci_read_config(dev, val, size, where);\n-}\n-\n-#endif /* _IFCVF_OSDEP_H_ */\ndiff --git a/drivers/net/ifc/ifcvf_vdpa.c b/drivers/net/ifc/ifcvf_vdpa.c\ndeleted file mode 100644\nindex da4667b..0000000\n--- a/drivers/net/ifc/ifcvf_vdpa.c\n+++ /dev/null\n@@ -1,1280 +0,0 @@\n-/* SPDX-License-Identifier: BSD-3-Clause\n- * Copyright(c) 2018 Intel Corporation\n- */\n-\n-#include <unistd.h>\n-#include <pthread.h>\n-#include <fcntl.h>\n-#include <string.h>\n-#include <sys/ioctl.h>\n-#include <sys/epoll.h>\n-#include <linux/virtio_net.h>\n-#include <stdbool.h>\n-\n-#include <rte_malloc.h>\n-#include <rte_memory.h>\n-#include <rte_bus_pci.h>\n-#include <rte_vhost.h>\n-#include <rte_vdpa.h>\n-#include <rte_vfio.h>\n-#include <rte_spinlock.h>\n-#include <rte_log.h>\n-#include <rte_kvargs.h>\n-#include <rte_devargs.h>\n-\n-#include \"base/ifcvf.h\"\n-\n-#define DRV_LOG(level, fmt, args...) \\\n-\trte_log(RTE_LOG_ ## level, ifcvf_vdpa_logtype, \\\n-\t\t\"IFCVF %s(): \" fmt \"\\n\", __func__, ##args)\n-\n-#ifndef PAGE_SIZE\n-#define PAGE_SIZE 4096\n-#endif\n-\n-#define IFCVF_USED_RING_LEN(size) \\\n-\t((size) * sizeof(struct vring_used_elem) + sizeof(uint16_t) * 3)\n-\n-#define IFCVF_VDPA_MODE\t\t\"vdpa\"\n-#define IFCVF_SW_FALLBACK_LM\t\"sw-live-migration\"\n-\n-static const char * const ifcvf_valid_arguments[] = {\n-\tIFCVF_VDPA_MODE,\n-\tIFCVF_SW_FALLBACK_LM,\n-\tNULL\n-};\n-\n-static int ifcvf_vdpa_logtype;\n-\n-struct ifcvf_internal {\n-\tstruct rte_vdpa_dev_addr dev_addr;\n-\tstruct rte_pci_device *pdev;\n-\tstruct ifcvf_hw hw;\n-\tint vfio_container_fd;\n-\tint vfio_group_fd;\n-\tint vfio_dev_fd;\n-\tpthread_t tid;\t/* thread for notify relay */\n-\tint epfd;\n-\tint vid;\n-\tint did;\n-\tuint16_t max_queues;\n-\tuint64_t features;\n-\trte_atomic32_t started;\n-\trte_atomic32_t dev_attached;\n-\trte_atomic32_t running;\n-\trte_spinlock_t lock;\n-\tbool sw_lm;\n-\tbool sw_fallback_running;\n-\t/* mediated vring for sw fallback */\n-\tstruct vring m_vring[IFCVF_MAX_QUEUES * 2];\n-\t/* eventfd for used ring interrupt */\n-\tint intr_fd[IFCVF_MAX_QUEUES * 2];\n-};\n-\n-struct internal_list {\n-\tTAILQ_ENTRY(internal_list) next;\n-\tstruct ifcvf_internal *internal;\n-};\n-\n-TAILQ_HEAD(internal_list_head, internal_list);\n-static struct internal_list_head internal_list =\n-\tTAILQ_HEAD_INITIALIZER(internal_list);\n-\n-static pthread_mutex_t internal_list_lock = PTHREAD_MUTEX_INITIALIZER;\n-\n-static void update_used_ring(struct ifcvf_internal *internal, uint16_t qid);\n-\n-static struct internal_list *\n-find_internal_resource_by_did(int did)\n-{\n-\tint found = 0;\n-\tstruct internal_list *list;\n-\n-\tpthread_mutex_lock(&internal_list_lock);\n-\n-\tTAILQ_FOREACH(list, &internal_list, next) {\n-\t\tif (did == list->internal->did) {\n-\t\t\tfound = 1;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n-\n-\tpthread_mutex_unlock(&internal_list_lock);\n-\n-\tif (!found)\n-\t\treturn NULL;\n-\n-\treturn list;\n-}\n-\n-static struct internal_list *\n-find_internal_resource_by_dev(struct rte_pci_device *pdev)\n-{\n-\tint found = 0;\n-\tstruct internal_list *list;\n-\n-\tpthread_mutex_lock(&internal_list_lock);\n-\n-\tTAILQ_FOREACH(list, &internal_list, next) {\n-\t\tif (pdev == list->internal->pdev) {\n-\t\t\tfound = 1;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n-\n-\tpthread_mutex_unlock(&internal_list_lock);\n-\n-\tif (!found)\n-\t\treturn NULL;\n-\n-\treturn list;\n-}\n-\n-static int\n-ifcvf_vfio_setup(struct ifcvf_internal *internal)\n-{\n-\tstruct rte_pci_device *dev = internal->pdev;\n-\tchar devname[RTE_DEV_NAME_MAX_LEN] = {0};\n-\tint iommu_group_num;\n-\tint i, ret;\n-\n-\tinternal->vfio_dev_fd = -1;\n-\tinternal->vfio_group_fd = -1;\n-\tinternal->vfio_container_fd = -1;\n-\n-\trte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);\n-\tret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,\n-\t\t\t&iommu_group_num);\n-\tif (ret <= 0) {\n-\t\tDRV_LOG(ERR, \"%s failed to get IOMMU group\", devname);\n-\t\treturn -1;\n-\t}\n-\n-\tinternal->vfio_container_fd = rte_vfio_container_create();\n-\tif (internal->vfio_container_fd < 0)\n-\t\treturn -1;\n-\n-\tinternal->vfio_group_fd = rte_vfio_container_group_bind(\n-\t\t\tinternal->vfio_container_fd, iommu_group_num);\n-\tif (internal->vfio_group_fd < 0)\n-\t\tgoto err;\n-\n-\tif (rte_pci_map_device(dev))\n-\t\tgoto err;\n-\n-\tinternal->vfio_dev_fd = dev->intr_handle.vfio_dev_fd;\n-\n-\tfor (i = 0; i < RTE_MIN(PCI_MAX_RESOURCE, IFCVF_PCI_MAX_RESOURCE);\n-\t\t\ti++) {\n-\t\tinternal->hw.mem_resource[i].addr =\n-\t\t\tinternal->pdev->mem_resource[i].addr;\n-\t\tinternal->hw.mem_resource[i].phys_addr =\n-\t\t\tinternal->pdev->mem_resource[i].phys_addr;\n-\t\tinternal->hw.mem_resource[i].len =\n-\t\t\tinternal->pdev->mem_resource[i].len;\n-\t}\n-\n-\treturn 0;\n-\n-err:\n-\trte_vfio_container_destroy(internal->vfio_container_fd);\n-\treturn -1;\n-}\n-\n-static int\n-ifcvf_dma_map(struct ifcvf_internal *internal, int do_map)\n-{\n-\tuint32_t i;\n-\tint ret;\n-\tstruct rte_vhost_memory *mem = NULL;\n-\tint vfio_container_fd;\n-\n-\tret = rte_vhost_get_mem_table(internal->vid, &mem);\n-\tif (ret < 0) {\n-\t\tDRV_LOG(ERR, \"failed to get VM memory layout.\");\n-\t\tgoto exit;\n-\t}\n-\n-\tvfio_container_fd = internal->vfio_container_fd;\n-\n-\tfor (i = 0; i < mem->nregions; i++) {\n-\t\tstruct rte_vhost_mem_region *reg;\n-\n-\t\treg = &mem->regions[i];\n-\t\tDRV_LOG(INFO, \"%s, region %u: HVA 0x%\" PRIx64 \", \"\n-\t\t\t\"GPA 0x%\" PRIx64 \", size 0x%\" PRIx64 \".\",\n-\t\t\tdo_map ? \"DMA map\" : \"DMA unmap\", i,\n-\t\t\treg->host_user_addr, reg->guest_phys_addr, reg->size);\n-\n-\t\tif (do_map) {\n-\t\t\tret = rte_vfio_container_dma_map(vfio_container_fd,\n-\t\t\t\treg->host_user_addr, reg->guest_phys_addr,\n-\t\t\t\treg->size);\n-\t\t\tif (ret < 0) {\n-\t\t\t\tDRV_LOG(ERR, \"DMA map failed.\");\n-\t\t\t\tgoto exit;\n-\t\t\t}\n-\t\t} else {\n-\t\t\tret = rte_vfio_container_dma_unmap(vfio_container_fd,\n-\t\t\t\treg->host_user_addr, reg->guest_phys_addr,\n-\t\t\t\treg->size);\n-\t\t\tif (ret < 0) {\n-\t\t\t\tDRV_LOG(ERR, \"DMA unmap failed.\");\n-\t\t\t\tgoto exit;\n-\t\t\t}\n-\t\t}\n-\t}\n-\n-exit:\n-\tif (mem)\n-\t\tfree(mem);\n-\treturn ret;\n-}\n-\n-static uint64_t\n-hva_to_gpa(int vid, uint64_t hva)\n-{\n-\tstruct rte_vhost_memory *mem = NULL;\n-\tstruct rte_vhost_mem_region *reg;\n-\tuint32_t i;\n-\tuint64_t gpa = 0;\n-\n-\tif (rte_vhost_get_mem_table(vid, &mem) < 0)\n-\t\tgoto exit;\n-\n-\tfor (i = 0; i < mem->nregions; i++) {\n-\t\treg = &mem->regions[i];\n-\n-\t\tif (hva >= reg->host_user_addr &&\n-\t\t\t\thva < reg->host_user_addr + reg->size) {\n-\t\t\tgpa = hva - reg->host_user_addr + reg->guest_phys_addr;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n-\n-exit:\n-\tif (mem)\n-\t\tfree(mem);\n-\treturn gpa;\n-}\n-\n-static int\n-vdpa_ifcvf_start(struct ifcvf_internal *internal)\n-{\n-\tstruct ifcvf_hw *hw = &internal->hw;\n-\tint i, nr_vring;\n-\tint vid;\n-\tstruct rte_vhost_vring vq;\n-\tuint64_t gpa;\n-\n-\tvid = internal->vid;\n-\tnr_vring = rte_vhost_get_vring_num(vid);\n-\trte_vhost_get_negotiated_features(vid, &hw->req_features);\n-\n-\tfor (i = 0; i < nr_vring; i++) {\n-\t\trte_vhost_get_vhost_vring(vid, i, &vq);\n-\t\tgpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.desc);\n-\t\tif (gpa == 0) {\n-\t\t\tDRV_LOG(ERR, \"Fail to get GPA for descriptor ring.\");\n-\t\t\treturn -1;\n-\t\t}\n-\t\thw->vring[i].desc = gpa;\n-\n-\t\tgpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.avail);\n-\t\tif (gpa == 0) {\n-\t\t\tDRV_LOG(ERR, \"Fail to get GPA for available ring.\");\n-\t\t\treturn -1;\n-\t\t}\n-\t\thw->vring[i].avail = gpa;\n-\n-\t\tgpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used);\n-\t\tif (gpa == 0) {\n-\t\t\tDRV_LOG(ERR, \"Fail to get GPA for used ring.\");\n-\t\t\treturn -1;\n-\t\t}\n-\t\thw->vring[i].used = gpa;\n-\n-\t\thw->vring[i].size = vq.size;\n-\t\trte_vhost_get_vring_base(vid, i, &hw->vring[i].last_avail_idx,\n-\t\t\t\t&hw->vring[i].last_used_idx);\n-\t}\n-\thw->nr_vring = i;\n-\n-\treturn ifcvf_start_hw(&internal->hw);\n-}\n-\n-static void\n-vdpa_ifcvf_stop(struct ifcvf_internal *internal)\n-{\n-\tstruct ifcvf_hw *hw = &internal->hw;\n-\tuint32_t i;\n-\tint vid;\n-\tuint64_t features = 0;\n-\tuint64_t log_base = 0, log_size = 0;\n-\tuint64_t len;\n-\n-\tvid = internal->vid;\n-\tifcvf_stop_hw(hw);\n-\n-\tfor (i = 0; i < hw->nr_vring; i++)\n-\t\trte_vhost_set_vring_base(vid, i, hw->vring[i].last_avail_idx,\n-\t\t\t\thw->vring[i].last_used_idx);\n-\n-\tif (internal->sw_lm)\n-\t\treturn;\n-\n-\trte_vhost_get_negotiated_features(vid, &features);\n-\tif (RTE_VHOST_NEED_LOG(features)) {\n-\t\tifcvf_disable_logging(hw);\n-\t\trte_vhost_get_log_base(internal->vid, &log_base, &log_size);\n-\t\trte_vfio_container_dma_unmap(internal->vfio_container_fd,\n-\t\t\t\tlog_base, IFCVF_LOG_BASE, log_size);\n-\t\t/*\n-\t\t * IFCVF marks dirty memory pages for only packet buffer,\n-\t\t * SW helps to mark the used ring as dirty after device stops.\n-\t\t */\n-\t\tfor (i = 0; i < hw->nr_vring; i++) {\n-\t\t\tlen = IFCVF_USED_RING_LEN(hw->vring[i].size);\n-\t\t\trte_vhost_log_used_vring(vid, i, 0, len);\n-\t\t}\n-\t}\n-}\n-\n-#define MSIX_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \\\n-\t\tsizeof(int) * (IFCVF_MAX_QUEUES * 2 + 1))\n-static int\n-vdpa_enable_vfio_intr(struct ifcvf_internal *internal, bool m_rx)\n-{\n-\tint ret;\n-\tuint32_t i, nr_vring;\n-\tchar irq_set_buf[MSIX_IRQ_SET_BUF_LEN];\n-\tstruct vfio_irq_set *irq_set;\n-\tint *fd_ptr;\n-\tstruct rte_vhost_vring vring;\n-\tint fd;\n-\n-\tvring.callfd = -1;\n-\n-\tnr_vring = rte_vhost_get_vring_num(internal->vid);\n-\n-\tirq_set = (struct vfio_irq_set *)irq_set_buf;\n-\tirq_set->argsz = sizeof(irq_set_buf);\n-\tirq_set->count = nr_vring + 1;\n-\tirq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD |\n-\t\t\t VFIO_IRQ_SET_ACTION_TRIGGER;\n-\tirq_set->index = VFIO_PCI_MSIX_IRQ_INDEX;\n-\tirq_set->start = 0;\n-\tfd_ptr = (int *)&irq_set->data;\n-\tfd_ptr[RTE_INTR_VEC_ZERO_OFFSET] = internal->pdev->intr_handle.fd;\n-\n-\tfor (i = 0; i < nr_vring; i++)\n-\t\tinternal->intr_fd[i] = -1;\n-\n-\tfor (i = 0; i < nr_vring; i++) {\n-\t\trte_vhost_get_vhost_vring(internal->vid, i, &vring);\n-\t\tfd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;\n-\t\tif ((i & 1) == 0 && m_rx == true) {\n-\t\t\tfd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);\n-\t\t\tif (fd < 0) {\n-\t\t\t\tDRV_LOG(ERR, \"can't setup eventfd: %s\",\n-\t\t\t\t\tstrerror(errno));\n-\t\t\t\treturn -1;\n-\t\t\t}\n-\t\t\tinternal->intr_fd[i] = fd;\n-\t\t\tfd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd;\n-\t\t}\n-\t}\n-\n-\tret = ioctl(internal->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);\n-\tif (ret) {\n-\t\tDRV_LOG(ERR, \"Error enabling MSI-X interrupts: %s\",\n-\t\t\t\tstrerror(errno));\n-\t\treturn -1;\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int\n-vdpa_disable_vfio_intr(struct ifcvf_internal *internal)\n-{\n-\tint ret;\n-\tuint32_t i, nr_vring;\n-\tchar irq_set_buf[MSIX_IRQ_SET_BUF_LEN];\n-\tstruct vfio_irq_set *irq_set;\n-\n-\tirq_set = (struct vfio_irq_set *)irq_set_buf;\n-\tirq_set->argsz = sizeof(irq_set_buf);\n-\tirq_set->count = 0;\n-\tirq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;\n-\tirq_set->index = VFIO_PCI_MSIX_IRQ_INDEX;\n-\tirq_set->start = 0;\n-\n-\tnr_vring = rte_vhost_get_vring_num(internal->vid);\n-\tfor (i = 0; i < nr_vring; i++) {\n-\t\tif (internal->intr_fd[i] >= 0)\n-\t\t\tclose(internal->intr_fd[i]);\n-\t\tinternal->intr_fd[i] = -1;\n-\t}\n-\n-\tret = ioctl(internal->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);\n-\tif (ret) {\n-\t\tDRV_LOG(ERR, \"Error disabling MSI-X interrupts: %s\",\n-\t\t\t\tstrerror(errno));\n-\t\treturn -1;\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static void *\n-notify_relay(void *arg)\n-{\n-\tint i, kickfd, epfd, nfds = 0;\n-\tuint32_t qid, q_num;\n-\tstruct epoll_event events[IFCVF_MAX_QUEUES * 2];\n-\tstruct epoll_event ev;\n-\tuint64_t buf;\n-\tint nbytes;\n-\tstruct rte_vhost_vring vring;\n-\tstruct ifcvf_internal *internal = (struct ifcvf_internal *)arg;\n-\tstruct ifcvf_hw *hw = &internal->hw;\n-\n-\tq_num = rte_vhost_get_vring_num(internal->vid);\n-\n-\tepfd = epoll_create(IFCVF_MAX_QUEUES * 2);\n-\tif (epfd < 0) {\n-\t\tDRV_LOG(ERR, \"failed to create epoll instance.\");\n-\t\treturn NULL;\n-\t}\n-\tinternal->epfd = epfd;\n-\n-\tvring.kickfd = -1;\n-\tfor (qid = 0; qid < q_num; qid++) {\n-\t\tev.events = EPOLLIN | EPOLLPRI;\n-\t\trte_vhost_get_vhost_vring(internal->vid, qid, &vring);\n-\t\tev.data.u64 = qid | (uint64_t)vring.kickfd << 32;\n-\t\tif (epoll_ctl(epfd, EPOLL_CTL_ADD, vring.kickfd, &ev) < 0) {\n-\t\t\tDRV_LOG(ERR, \"epoll add error: %s\", strerror(errno));\n-\t\t\treturn NULL;\n-\t\t}\n-\t}\n-\n-\tfor (;;) {\n-\t\tnfds = epoll_wait(epfd, events, q_num, -1);\n-\t\tif (nfds < 0) {\n-\t\t\tif (errno == EINTR)\n-\t\t\t\tcontinue;\n-\t\t\tDRV_LOG(ERR, \"epoll_wait return fail\\n\");\n-\t\t\treturn NULL;\n-\t\t}\n-\n-\t\tfor (i = 0; i < nfds; i++) {\n-\t\t\tqid = events[i].data.u32;\n-\t\t\tkickfd = (uint32_t)(events[i].data.u64 >> 32);\n-\t\t\tdo {\n-\t\t\t\tnbytes = read(kickfd, &buf, 8);\n-\t\t\t\tif (nbytes < 0) {\n-\t\t\t\t\tif (errno == EINTR ||\n-\t\t\t\t\t    errno == EWOULDBLOCK ||\n-\t\t\t\t\t    errno == EAGAIN)\n-\t\t\t\t\t\tcontinue;\n-\t\t\t\t\tDRV_LOG(INFO, \"Error reading \"\n-\t\t\t\t\t\t\"kickfd: %s\",\n-\t\t\t\t\t\tstrerror(errno));\n-\t\t\t\t}\n-\t\t\t\tbreak;\n-\t\t\t} while (1);\n-\n-\t\t\tifcvf_notify_queue(hw, qid);\n-\t\t}\n-\t}\n-\n-\treturn NULL;\n-}\n-\n-static int\n-setup_notify_relay(struct ifcvf_internal *internal)\n-{\n-\tint ret;\n-\n-\tret = pthread_create(&internal->tid, NULL, notify_relay,\n-\t\t\t(void *)internal);\n-\tif (ret) {\n-\t\tDRV_LOG(ERR, \"failed to create notify relay pthread.\");\n-\t\treturn -1;\n-\t}\n-\treturn 0;\n-}\n-\n-static int\n-unset_notify_relay(struct ifcvf_internal *internal)\n-{\n-\tvoid *status;\n-\n-\tif (internal->tid) {\n-\t\tpthread_cancel(internal->tid);\n-\t\tpthread_join(internal->tid, &status);\n-\t}\n-\tinternal->tid = 0;\n-\n-\tif (internal->epfd >= 0)\n-\t\tclose(internal->epfd);\n-\tinternal->epfd = -1;\n-\n-\treturn 0;\n-}\n-\n-static int\n-update_datapath(struct ifcvf_internal *internal)\n-{\n-\tint ret;\n-\n-\trte_spinlock_lock(&internal->lock);\n-\n-\tif (!rte_atomic32_read(&internal->running) &&\n-\t    (rte_atomic32_read(&internal->started) &&\n-\t     rte_atomic32_read(&internal->dev_attached))) {\n-\t\tret = ifcvf_dma_map(internal, 1);\n-\t\tif (ret)\n-\t\t\tgoto err;\n-\n-\t\tret = vdpa_enable_vfio_intr(internal, 0);\n-\t\tif (ret)\n-\t\t\tgoto err;\n-\n-\t\tret = vdpa_ifcvf_start(internal);\n-\t\tif (ret)\n-\t\t\tgoto err;\n-\n-\t\tret = setup_notify_relay(internal);\n-\t\tif (ret)\n-\t\t\tgoto err;\n-\n-\t\trte_atomic32_set(&internal->running, 1);\n-\t} else if (rte_atomic32_read(&internal->running) &&\n-\t\t   (!rte_atomic32_read(&internal->started) ||\n-\t\t    !rte_atomic32_read(&internal->dev_attached))) {\n-\t\tret = unset_notify_relay(internal);\n-\t\tif (ret)\n-\t\t\tgoto err;\n-\n-\t\tvdpa_ifcvf_stop(internal);\n-\n-\t\tret = vdpa_disable_vfio_intr(internal);\n-\t\tif (ret)\n-\t\t\tgoto err;\n-\n-\t\tret = ifcvf_dma_map(internal, 0);\n-\t\tif (ret)\n-\t\t\tgoto err;\n-\n-\t\trte_atomic32_set(&internal->running, 0);\n-\t}\n-\n-\trte_spinlock_unlock(&internal->lock);\n-\treturn 0;\n-err:\n-\trte_spinlock_unlock(&internal->lock);\n-\treturn ret;\n-}\n-\n-static int\n-m_ifcvf_start(struct ifcvf_internal *internal)\n-{\n-\tstruct ifcvf_hw *hw = &internal->hw;\n-\tuint32_t i, nr_vring;\n-\tint vid, ret;\n-\tstruct rte_vhost_vring vq;\n-\tvoid *vring_buf;\n-\tuint64_t m_vring_iova = IFCVF_MEDIATED_VRING;\n-\tuint64_t size;\n-\tuint64_t gpa;\n-\n-\tmemset(&vq, 0, sizeof(vq));\n-\tvid = internal->vid;\n-\tnr_vring = rte_vhost_get_vring_num(vid);\n-\trte_vhost_get_negotiated_features(vid, &hw->req_features);\n-\n-\tfor (i = 0; i < nr_vring; i++) {\n-\t\trte_vhost_get_vhost_vring(vid, i, &vq);\n-\n-\t\tsize = RTE_ALIGN_CEIL(vring_size(vq.size, PAGE_SIZE),\n-\t\t\t\tPAGE_SIZE);\n-\t\tvring_buf = rte_zmalloc(\"ifcvf\", size, PAGE_SIZE);\n-\t\tvring_init(&internal->m_vring[i], vq.size, vring_buf,\n-\t\t\t\tPAGE_SIZE);\n-\n-\t\tret = rte_vfio_container_dma_map(internal->vfio_container_fd,\n-\t\t\t(uint64_t)(uintptr_t)vring_buf, m_vring_iova, size);\n-\t\tif (ret < 0) {\n-\t\t\tDRV_LOG(ERR, \"mediated vring DMA map failed.\");\n-\t\t\tgoto error;\n-\t\t}\n-\n-\t\tgpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.desc);\n-\t\tif (gpa == 0) {\n-\t\t\tDRV_LOG(ERR, \"Fail to get GPA for descriptor ring.\");\n-\t\t\treturn -1;\n-\t\t}\n-\t\thw->vring[i].desc = gpa;\n-\n-\t\tgpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.avail);\n-\t\tif (gpa == 0) {\n-\t\t\tDRV_LOG(ERR, \"Fail to get GPA for available ring.\");\n-\t\t\treturn -1;\n-\t\t}\n-\t\thw->vring[i].avail = gpa;\n-\n-\t\t/* Direct I/O for Tx queue, relay for Rx queue */\n-\t\tif (i & 1) {\n-\t\t\tgpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used);\n-\t\t\tif (gpa == 0) {\n-\t\t\t\tDRV_LOG(ERR, \"Fail to get GPA for used ring.\");\n-\t\t\t\treturn -1;\n-\t\t\t}\n-\t\t\thw->vring[i].used = gpa;\n-\t\t} else {\n-\t\t\thw->vring[i].used = m_vring_iova +\n-\t\t\t\t(char *)internal->m_vring[i].used -\n-\t\t\t\t(char *)internal->m_vring[i].desc;\n-\t\t}\n-\n-\t\thw->vring[i].size = vq.size;\n-\n-\t\trte_vhost_get_vring_base(vid, i,\n-\t\t\t\t&internal->m_vring[i].avail->idx,\n-\t\t\t\t&internal->m_vring[i].used->idx);\n-\n-\t\trte_vhost_get_vring_base(vid, i, &hw->vring[i].last_avail_idx,\n-\t\t\t\t&hw->vring[i].last_used_idx);\n-\n-\t\tm_vring_iova += size;\n-\t}\n-\thw->nr_vring = nr_vring;\n-\n-\treturn ifcvf_start_hw(&internal->hw);\n-\n-error:\n-\tfor (i = 0; i < nr_vring; i++)\n-\t\tif (internal->m_vring[i].desc)\n-\t\t\trte_free(internal->m_vring[i].desc);\n-\n-\treturn -1;\n-}\n-\n-static int\n-m_ifcvf_stop(struct ifcvf_internal *internal)\n-{\n-\tint vid;\n-\tuint32_t i;\n-\tstruct rte_vhost_vring vq;\n-\tstruct ifcvf_hw *hw = &internal->hw;\n-\tuint64_t m_vring_iova = IFCVF_MEDIATED_VRING;\n-\tuint64_t size, len;\n-\n-\tvid = internal->vid;\n-\tifcvf_stop_hw(hw);\n-\n-\tfor (i = 0; i < hw->nr_vring; i++) {\n-\t\t/* synchronize remaining new used entries if any */\n-\t\tif ((i & 1) == 0)\n-\t\t\tupdate_used_ring(internal, i);\n-\n-\t\trte_vhost_get_vhost_vring(vid, i, &vq);\n-\t\tlen = IFCVF_USED_RING_LEN(vq.size);\n-\t\trte_vhost_log_used_vring(vid, i, 0, len);\n-\n-\t\tsize = RTE_ALIGN_CEIL(vring_size(vq.size, PAGE_SIZE),\n-\t\t\t\tPAGE_SIZE);\n-\t\trte_vfio_container_dma_unmap(internal->vfio_container_fd,\n-\t\t\t(uint64_t)(uintptr_t)internal->m_vring[i].desc,\n-\t\t\tm_vring_iova, size);\n-\n-\t\trte_vhost_set_vring_base(vid, i, hw->vring[i].last_avail_idx,\n-\t\t\t\thw->vring[i].last_used_idx);\n-\t\trte_free(internal->m_vring[i].desc);\n-\t\tm_vring_iova += size;\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static void\n-update_used_ring(struct ifcvf_internal *internal, uint16_t qid)\n-{\n-\trte_vdpa_relay_vring_used(internal->vid, qid, &internal->m_vring[qid]);\n-\trte_vhost_vring_call(internal->vid, qid);\n-}\n-\n-static void *\n-vring_relay(void *arg)\n-{\n-\tint i, vid, epfd, fd, nfds;\n-\tstruct ifcvf_internal *internal = (struct ifcvf_internal *)arg;\n-\tstruct rte_vhost_vring vring;\n-\tuint16_t qid, q_num;\n-\tstruct epoll_event events[IFCVF_MAX_QUEUES * 4];\n-\tstruct epoll_event ev;\n-\tint nbytes;\n-\tuint64_t buf;\n-\n-\tvid = internal->vid;\n-\tq_num = rte_vhost_get_vring_num(vid);\n-\n-\t/* add notify fd and interrupt fd to epoll */\n-\tepfd = epoll_create(IFCVF_MAX_QUEUES * 2);\n-\tif (epfd < 0) {\n-\t\tDRV_LOG(ERR, \"failed to create epoll instance.\");\n-\t\treturn NULL;\n-\t}\n-\tinternal->epfd = epfd;\n-\n-\tvring.kickfd = -1;\n-\tfor (qid = 0; qid < q_num; qid++) {\n-\t\tev.events = EPOLLIN | EPOLLPRI;\n-\t\trte_vhost_get_vhost_vring(vid, qid, &vring);\n-\t\tev.data.u64 = qid << 1 | (uint64_t)vring.kickfd << 32;\n-\t\tif (epoll_ctl(epfd, EPOLL_CTL_ADD, vring.kickfd, &ev) < 0) {\n-\t\t\tDRV_LOG(ERR, \"epoll add error: %s\", strerror(errno));\n-\t\t\treturn NULL;\n-\t\t}\n-\t}\n-\n-\tfor (qid = 0; qid < q_num; qid += 2) {\n-\t\tev.events = EPOLLIN | EPOLLPRI;\n-\t\t/* leave a flag to mark it's for interrupt */\n-\t\tev.data.u64 = 1 | qid << 1 |\n-\t\t\t(uint64_t)internal->intr_fd[qid] << 32;\n-\t\tif (epoll_ctl(epfd, EPOLL_CTL_ADD, internal->intr_fd[qid], &ev)\n-\t\t\t\t< 0) {\n-\t\t\tDRV_LOG(ERR, \"epoll add error: %s\", strerror(errno));\n-\t\t\treturn NULL;\n-\t\t}\n-\t\tupdate_used_ring(internal, qid);\n-\t}\n-\n-\t/* start relay with a first kick */\n-\tfor (qid = 0; qid < q_num; qid++)\n-\t\tifcvf_notify_queue(&internal->hw, qid);\n-\n-\t/* listen to the events and react accordingly */\n-\tfor (;;) {\n-\t\tnfds = epoll_wait(epfd, events, q_num * 2, -1);\n-\t\tif (nfds < 0) {\n-\t\t\tif (errno == EINTR)\n-\t\t\t\tcontinue;\n-\t\t\tDRV_LOG(ERR, \"epoll_wait return fail\\n\");\n-\t\t\treturn NULL;\n-\t\t}\n-\n-\t\tfor (i = 0; i < nfds; i++) {\n-\t\t\tfd = (uint32_t)(events[i].data.u64 >> 32);\n-\t\t\tdo {\n-\t\t\t\tnbytes = read(fd, &buf, 8);\n-\t\t\t\tif (nbytes < 0) {\n-\t\t\t\t\tif (errno == EINTR ||\n-\t\t\t\t\t    errno == EWOULDBLOCK ||\n-\t\t\t\t\t    errno == EAGAIN)\n-\t\t\t\t\t\tcontinue;\n-\t\t\t\t\tDRV_LOG(INFO, \"Error reading \"\n-\t\t\t\t\t\t\"kickfd: %s\",\n-\t\t\t\t\t\tstrerror(errno));\n-\t\t\t\t}\n-\t\t\t\tbreak;\n-\t\t\t} while (1);\n-\n-\t\t\tqid = events[i].data.u32 >> 1;\n-\n-\t\t\tif (events[i].data.u32 & 1)\n-\t\t\t\tupdate_used_ring(internal, qid);\n-\t\t\telse\n-\t\t\t\tifcvf_notify_queue(&internal->hw, qid);\n-\t\t}\n-\t}\n-\n-\treturn NULL;\n-}\n-\n-static int\n-setup_vring_relay(struct ifcvf_internal *internal)\n-{\n-\tint ret;\n-\n-\tret = pthread_create(&internal->tid, NULL, vring_relay,\n-\t\t\t(void *)internal);\n-\tif (ret) {\n-\t\tDRV_LOG(ERR, \"failed to create ring relay pthread.\");\n-\t\treturn -1;\n-\t}\n-\treturn 0;\n-}\n-\n-static int\n-unset_vring_relay(struct ifcvf_internal *internal)\n-{\n-\tvoid *status;\n-\n-\tif (internal->tid) {\n-\t\tpthread_cancel(internal->tid);\n-\t\tpthread_join(internal->tid, &status);\n-\t}\n-\tinternal->tid = 0;\n-\n-\tif (internal->epfd >= 0)\n-\t\tclose(internal->epfd);\n-\tinternal->epfd = -1;\n-\n-\treturn 0;\n-}\n-\n-static int\n-ifcvf_sw_fallback_switchover(struct ifcvf_internal *internal)\n-{\n-\tint ret;\n-\tint vid = internal->vid;\n-\n-\t/* stop the direct IO data path */\n-\tunset_notify_relay(internal);\n-\tvdpa_ifcvf_stop(internal);\n-\tvdpa_disable_vfio_intr(internal);\n-\n-\tret = rte_vhost_host_notifier_ctrl(vid, false);\n-\tif (ret && ret != -ENOTSUP)\n-\t\tgoto error;\n-\n-\t/* set up interrupt for interrupt relay */\n-\tret = vdpa_enable_vfio_intr(internal, 1);\n-\tif (ret)\n-\t\tgoto unmap;\n-\n-\t/* config the VF */\n-\tret = m_ifcvf_start(internal);\n-\tif (ret)\n-\t\tgoto unset_intr;\n-\n-\t/* set up vring relay thread */\n-\tret = setup_vring_relay(internal);\n-\tif (ret)\n-\t\tgoto stop_vf;\n-\n-\trte_vhost_host_notifier_ctrl(vid, true);\n-\n-\tinternal->sw_fallback_running = true;\n-\n-\treturn 0;\n-\n-stop_vf:\n-\tm_ifcvf_stop(internal);\n-unset_intr:\n-\tvdpa_disable_vfio_intr(internal);\n-unmap:\n-\tifcvf_dma_map(internal, 0);\n-error:\n-\treturn -1;\n-}\n-\n-static int\n-ifcvf_dev_config(int vid)\n-{\n-\tint did;\n-\tstruct internal_list *list;\n-\tstruct ifcvf_internal *internal;\n-\n-\tdid = rte_vhost_get_vdpa_device_id(vid);\n-\tlist = find_internal_resource_by_did(did);\n-\tif (list == NULL) {\n-\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n-\t\treturn -1;\n-\t}\n-\n-\tinternal = list->internal;\n-\tinternal->vid = vid;\n-\trte_atomic32_set(&internal->dev_attached, 1);\n-\tupdate_datapath(internal);\n-\n-\tif (rte_vhost_host_notifier_ctrl(vid, true) != 0)\n-\t\tDRV_LOG(NOTICE, \"vDPA (%d): software relay is used.\", did);\n-\n-\treturn 0;\n-}\n-\n-static int\n-ifcvf_dev_close(int vid)\n-{\n-\tint did;\n-\tstruct internal_list *list;\n-\tstruct ifcvf_internal *internal;\n-\n-\tdid = rte_vhost_get_vdpa_device_id(vid);\n-\tlist = find_internal_resource_by_did(did);\n-\tif (list == NULL) {\n-\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n-\t\treturn -1;\n-\t}\n-\n-\tinternal = list->internal;\n-\n-\tif (internal->sw_fallback_running) {\n-\t\t/* unset ring relay */\n-\t\tunset_vring_relay(internal);\n-\n-\t\t/* reset VF */\n-\t\tm_ifcvf_stop(internal);\n-\n-\t\t/* remove interrupt setting */\n-\t\tvdpa_disable_vfio_intr(internal);\n-\n-\t\t/* unset DMA map for guest memory */\n-\t\tifcvf_dma_map(internal, 0);\n-\n-\t\tinternal->sw_fallback_running = false;\n-\t} else {\n-\t\trte_atomic32_set(&internal->dev_attached, 0);\n-\t\tupdate_datapath(internal);\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int\n-ifcvf_set_features(int vid)\n-{\n-\tuint64_t features = 0;\n-\tint did;\n-\tstruct internal_list *list;\n-\tstruct ifcvf_internal *internal;\n-\tuint64_t log_base = 0, log_size = 0;\n-\n-\tdid = rte_vhost_get_vdpa_device_id(vid);\n-\tlist = find_internal_resource_by_did(did);\n-\tif (list == NULL) {\n-\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n-\t\treturn -1;\n-\t}\n-\n-\tinternal = list->internal;\n-\trte_vhost_get_negotiated_features(vid, &features);\n-\n-\tif (!RTE_VHOST_NEED_LOG(features))\n-\t\treturn 0;\n-\n-\tif (internal->sw_lm) {\n-\t\tifcvf_sw_fallback_switchover(internal);\n-\t} else {\n-\t\trte_vhost_get_log_base(vid, &log_base, &log_size);\n-\t\trte_vfio_container_dma_map(internal->vfio_container_fd,\n-\t\t\t\tlog_base, IFCVF_LOG_BASE, log_size);\n-\t\tifcvf_enable_logging(&internal->hw, IFCVF_LOG_BASE, log_size);\n-\t}\n-\n-\treturn 0;\n-}\n-\n-static int\n-ifcvf_get_vfio_group_fd(int vid)\n-{\n-\tint did;\n-\tstruct internal_list *list;\n-\n-\tdid = rte_vhost_get_vdpa_device_id(vid);\n-\tlist = find_internal_resource_by_did(did);\n-\tif (list == NULL) {\n-\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n-\t\treturn -1;\n-\t}\n-\n-\treturn list->internal->vfio_group_fd;\n-}\n-\n-static int\n-ifcvf_get_vfio_device_fd(int vid)\n-{\n-\tint did;\n-\tstruct internal_list *list;\n-\n-\tdid = rte_vhost_get_vdpa_device_id(vid);\n-\tlist = find_internal_resource_by_did(did);\n-\tif (list == NULL) {\n-\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n-\t\treturn -1;\n-\t}\n-\n-\treturn list->internal->vfio_dev_fd;\n-}\n-\n-static int\n-ifcvf_get_notify_area(int vid, int qid, uint64_t *offset, uint64_t *size)\n-{\n-\tint did;\n-\tstruct internal_list *list;\n-\tstruct ifcvf_internal *internal;\n-\tstruct vfio_region_info reg = { .argsz = sizeof(reg) };\n-\tint ret;\n-\n-\tdid = rte_vhost_get_vdpa_device_id(vid);\n-\tlist = find_internal_resource_by_did(did);\n-\tif (list == NULL) {\n-\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n-\t\treturn -1;\n-\t}\n-\n-\tinternal = list->internal;\n-\n-\treg.index = ifcvf_get_notify_region(&internal->hw);\n-\tret = ioctl(internal->vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, &reg);\n-\tif (ret) {\n-\t\tDRV_LOG(ERR, \"Get not get device region info: %s\",\n-\t\t\t\tstrerror(errno));\n-\t\treturn -1;\n-\t}\n-\n-\t*offset = ifcvf_get_queue_notify_off(&internal->hw, qid) + reg.offset;\n-\t*size = 0x1000;\n-\n-\treturn 0;\n-}\n-\n-static int\n-ifcvf_get_queue_num(int did, uint32_t *queue_num)\n-{\n-\tstruct internal_list *list;\n-\n-\tlist = find_internal_resource_by_did(did);\n-\tif (list == NULL) {\n-\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n-\t\treturn -1;\n-\t}\n-\n-\t*queue_num = list->internal->max_queues;\n-\n-\treturn 0;\n-}\n-\n-static int\n-ifcvf_get_vdpa_features(int did, uint64_t *features)\n-{\n-\tstruct internal_list *list;\n-\n-\tlist = find_internal_resource_by_did(did);\n-\tif (list == NULL) {\n-\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n-\t\treturn -1;\n-\t}\n-\n-\t*features = list->internal->features;\n-\n-\treturn 0;\n-}\n-\n-#define VDPA_SUPPORTED_PROTOCOL_FEATURES \\\n-\t\t(1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK | \\\n-\t\t 1ULL << VHOST_USER_PROTOCOL_F_SLAVE_REQ | \\\n-\t\t 1ULL << VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD | \\\n-\t\t 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \\\n-\t\t 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD)\n-static int\n-ifcvf_get_protocol_features(int did __rte_unused, uint64_t *features)\n-{\n-\t*features = VDPA_SUPPORTED_PROTOCOL_FEATURES;\n-\treturn 0;\n-}\n-\n-static struct rte_vdpa_dev_ops ifcvf_ops = {\n-\t.get_queue_num = ifcvf_get_queue_num,\n-\t.get_features = ifcvf_get_vdpa_features,\n-\t.get_protocol_features = ifcvf_get_protocol_features,\n-\t.dev_conf = ifcvf_dev_config,\n-\t.dev_close = ifcvf_dev_close,\n-\t.set_vring_state = NULL,\n-\t.set_features = ifcvf_set_features,\n-\t.migration_done = NULL,\n-\t.get_vfio_group_fd = ifcvf_get_vfio_group_fd,\n-\t.get_vfio_device_fd = ifcvf_get_vfio_device_fd,\n-\t.get_notify_area = ifcvf_get_notify_area,\n-};\n-\n-static inline int\n-open_int(const char *key __rte_unused, const char *value, void *extra_args)\n-{\n-\tuint16_t *n = extra_args;\n-\n-\tif (value == NULL || extra_args == NULL)\n-\t\treturn -EINVAL;\n-\n-\t*n = (uint16_t)strtoul(value, NULL, 0);\n-\tif (*n == USHRT_MAX && errno == ERANGE)\n-\t\treturn -1;\n-\n-\treturn 0;\n-}\n-\n-static int\n-ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,\n-\t\tstruct rte_pci_device *pci_dev)\n-{\n-\tuint64_t features;\n-\tstruct ifcvf_internal *internal = NULL;\n-\tstruct internal_list *list = NULL;\n-\tint vdpa_mode = 0;\n-\tint sw_fallback_lm = 0;\n-\tstruct rte_kvargs *kvlist = NULL;\n-\tint ret = 0;\n-\n-\tif (rte_eal_process_type() != RTE_PROC_PRIMARY)\n-\t\treturn 0;\n-\n-\tif (!pci_dev->device.devargs)\n-\t\treturn 1;\n-\n-\tkvlist = rte_kvargs_parse(pci_dev->device.devargs->args,\n-\t\t\tifcvf_valid_arguments);\n-\tif (kvlist == NULL)\n-\t\treturn 1;\n-\n-\t/* probe only when vdpa mode is specified */\n-\tif (rte_kvargs_count(kvlist, IFCVF_VDPA_MODE) == 0) {\n-\t\trte_kvargs_free(kvlist);\n-\t\treturn 1;\n-\t}\n-\n-\tret = rte_kvargs_process(kvlist, IFCVF_VDPA_MODE, &open_int,\n-\t\t\t&vdpa_mode);\n-\tif (ret < 0 || vdpa_mode == 0) {\n-\t\trte_kvargs_free(kvlist);\n-\t\treturn 1;\n-\t}\n-\n-\tlist = rte_zmalloc(\"ifcvf\", sizeof(*list), 0);\n-\tif (list == NULL)\n-\t\tgoto error;\n-\n-\tinternal = rte_zmalloc(\"ifcvf\", sizeof(*internal), 0);\n-\tif (internal == NULL)\n-\t\tgoto error;\n-\n-\tinternal->pdev = pci_dev;\n-\trte_spinlock_init(&internal->lock);\n-\n-\tif (ifcvf_vfio_setup(internal) < 0) {\n-\t\tDRV_LOG(ERR, \"failed to setup device %s\", pci_dev->name);\n-\t\tgoto error;\n-\t}\n-\n-\tif (ifcvf_init_hw(&internal->hw, internal->pdev) < 0) {\n-\t\tDRV_LOG(ERR, \"failed to init device %s\", pci_dev->name);\n-\t\tgoto error;\n-\t}\n-\n-\tinternal->max_queues = IFCVF_MAX_QUEUES;\n-\tfeatures = ifcvf_get_features(&internal->hw);\n-\tinternal->features = (features &\n-\t\t~(1ULL << VIRTIO_F_IOMMU_PLATFORM)) |\n-\t\t(1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |\n-\t\t(1ULL << VIRTIO_NET_F_CTRL_VQ) |\n-\t\t(1ULL << VIRTIO_NET_F_STATUS) |\n-\t\t(1ULL << VHOST_USER_F_PROTOCOL_FEATURES) |\n-\t\t(1ULL << VHOST_F_LOG_ALL);\n-\n-\tinternal->dev_addr.pci_addr = pci_dev->addr;\n-\tinternal->dev_addr.type = PCI_ADDR;\n-\tlist->internal = internal;\n-\n-\tif (rte_kvargs_count(kvlist, IFCVF_SW_FALLBACK_LM)) {\n-\t\tret = rte_kvargs_process(kvlist, IFCVF_SW_FALLBACK_LM,\n-\t\t\t\t&open_int, &sw_fallback_lm);\n-\t\tif (ret < 0)\n-\t\t\tgoto error;\n-\t}\n-\tinternal->sw_lm = sw_fallback_lm;\n-\n-\tinternal->did = rte_vdpa_register_device(&internal->dev_addr,\n-\t\t\t\t&ifcvf_ops);\n-\tif (internal->did < 0) {\n-\t\tDRV_LOG(ERR, \"failed to register device %s\", pci_dev->name);\n-\t\tgoto error;\n-\t}\n-\n-\tpthread_mutex_lock(&internal_list_lock);\n-\tTAILQ_INSERT_TAIL(&internal_list, list, next);\n-\tpthread_mutex_unlock(&internal_list_lock);\n-\n-\trte_atomic32_set(&internal->started, 1);\n-\tupdate_datapath(internal);\n-\n-\trte_kvargs_free(kvlist);\n-\treturn 0;\n-\n-error:\n-\trte_kvargs_free(kvlist);\n-\trte_free(list);\n-\trte_free(internal);\n-\treturn -1;\n-}\n-\n-static int\n-ifcvf_pci_remove(struct rte_pci_device *pci_dev)\n-{\n-\tstruct ifcvf_internal *internal;\n-\tstruct internal_list *list;\n-\n-\tif (rte_eal_process_type() != RTE_PROC_PRIMARY)\n-\t\treturn 0;\n-\n-\tlist = find_internal_resource_by_dev(pci_dev);\n-\tif (list == NULL) {\n-\t\tDRV_LOG(ERR, \"Invalid device: %s\", pci_dev->name);\n-\t\treturn -1;\n-\t}\n-\n-\tinternal = list->internal;\n-\trte_atomic32_set(&internal->started, 0);\n-\tupdate_datapath(internal);\n-\n-\trte_pci_unmap_device(internal->pdev);\n-\trte_vfio_container_destroy(internal->vfio_container_fd);\n-\trte_vdpa_unregister_device(internal->did);\n-\n-\tpthread_mutex_lock(&internal_list_lock);\n-\tTAILQ_REMOVE(&internal_list, list, next);\n-\tpthread_mutex_unlock(&internal_list_lock);\n-\n-\trte_free(list);\n-\trte_free(internal);\n-\n-\treturn 0;\n-}\n-\n-/*\n- * IFCVF has the same vendor ID and device ID as virtio net PCI\n- * device, with its specific subsystem vendor ID and device ID.\n- */\n-static const struct rte_pci_id pci_id_ifcvf_map[] = {\n-\t{ .class_id = RTE_CLASS_ANY_ID,\n-\t  .vendor_id = IFCVF_VENDOR_ID,\n-\t  .device_id = IFCVF_DEVICE_ID,\n-\t  .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID,\n-\t  .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID,\n-\t},\n-\n-\t{ .vendor_id = 0, /* sentinel */\n-\t},\n-};\n-\n-static struct rte_pci_driver rte_ifcvf_vdpa = {\n-\t.id_table = pci_id_ifcvf_map,\n-\t.drv_flags = 0,\n-\t.probe = ifcvf_pci_probe,\n-\t.remove = ifcvf_pci_remove,\n-};\n-\n-RTE_PMD_REGISTER_PCI(net_ifcvf, rte_ifcvf_vdpa);\n-RTE_PMD_REGISTER_PCI_TABLE(net_ifcvf, pci_id_ifcvf_map);\n-RTE_PMD_REGISTER_KMOD_DEP(net_ifcvf, \"* vfio-pci\");\n-\n-RTE_INIT(ifcvf_vdpa_init_log)\n-{\n-\tifcvf_vdpa_logtype = rte_log_register(\"pmd.net.ifcvf_vdpa\");\n-\tif (ifcvf_vdpa_logtype >= 0)\n-\t\trte_log_set_level(ifcvf_vdpa_logtype, RTE_LOG_NOTICE);\n-}\ndiff --git a/drivers/net/ifc/meson.build b/drivers/net/ifc/meson.build\ndeleted file mode 100644\nindex adc9ed9..0000000\n--- a/drivers/net/ifc/meson.build\n+++ /dev/null\n@@ -1,9 +0,0 @@\n-# SPDX-License-Identifier: BSD-3-Clause\n-# Copyright(c) 2018 Intel Corporation\n-\n-build = dpdk_conf.has('RTE_LIBRTE_VHOST')\n-reason = 'missing dependency, DPDK vhost library'\n-allow_experimental_apis = true\n-sources = files('ifcvf_vdpa.c', 'base/ifcvf.c')\n-includes += include_directories('base')\n-deps += 'vhost'\ndiff --git a/drivers/net/ifc/rte_pmd_ifc_version.map b/drivers/net/ifc/rte_pmd_ifc_version.map\ndeleted file mode 100644\nindex f9f17e4..0000000\n--- a/drivers/net/ifc/rte_pmd_ifc_version.map\n+++ /dev/null\n@@ -1,3 +0,0 @@\n-DPDK_20.0 {\n-\tlocal: *;\n-};\ndiff --git a/drivers/net/meson.build b/drivers/net/meson.build\nindex c300afb..b0ea8fe 100644\n--- a/drivers/net/meson.build\n+++ b/drivers/net/meson.build\n@@ -21,7 +21,6 @@ drivers = ['af_packet',\n \t'hns3',\n \t'iavf',\n \t'ice',\n-\t'ifc',\n \t'ipn3ke',\n \t'ixgbe',\n \t'kni',\ndiff --git a/drivers/vdpa/Makefile b/drivers/vdpa/Makefile\nindex 82a2b70..27fec96 100644\n--- a/drivers/vdpa/Makefile\n+++ b/drivers/vdpa/Makefile\n@@ -5,4 +5,10 @@ include $(RTE_SDK)/mk/rte.vars.mk\n \n # DIRS-$(<configuration>) += <directory>\n \n+ifeq ($(CONFIG_RTE_LIBRTE_VHOST),y)\n+ifeq ($(CONFIG_RTE_EAL_VFIO),y)\n+DIRS-$(CONFIG_RTE_LIBRTE_IFC_PMD) += ifc\n+endif\n+endif # $(CONFIG_RTE_LIBRTE_VHOST)\n+\n include $(RTE_SDK)/mk/rte.subdir.mk\ndiff --git a/drivers/vdpa/ifc/Makefile b/drivers/vdpa/ifc/Makefile\nnew file mode 100644\nindex 0000000..fe227b8\n--- /dev/null\n+++ b/drivers/vdpa/ifc/Makefile\n@@ -0,0 +1,34 @@\n+# SPDX-License-Identifier: BSD-3-Clause\n+# Copyright(c) 2018 Intel Corporation\n+\n+include $(RTE_SDK)/mk/rte.vars.mk\n+\n+#\n+# library name\n+#\n+LIB = librte_pmd_ifc.a\n+\n+LDLIBS += -lpthread\n+LDLIBS += -lrte_eal -lrte_pci -lrte_vhost -lrte_bus_pci\n+LDLIBS += -lrte_kvargs\n+\n+CFLAGS += -O3\n+CFLAGS += $(WERROR_FLAGS)\n+CFLAGS += -DALLOW_EXPERIMENTAL_API\n+\n+#\n+# Add extra flags for base driver source files to disable warnings in them\n+#\n+BASE_DRIVER_OBJS=$(sort $(patsubst %.c,%.o,$(notdir $(wildcard $(SRCDIR)/base/*.c))))\n+\n+VPATH += $(SRCDIR)/base\n+\n+EXPORT_MAP := rte_pmd_ifc_version.map\n+\n+#\n+# all source are stored in SRCS-y\n+#\n+SRCS-$(CONFIG_RTE_LIBRTE_IFC_PMD) += ifcvf_vdpa.c\n+SRCS-$(CONFIG_RTE_LIBRTE_IFC_PMD) += ifcvf.c\n+\n+include $(RTE_SDK)/mk/rte.lib.mk\ndiff --git a/drivers/vdpa/ifc/base/ifcvf.c b/drivers/vdpa/ifc/base/ifcvf.c\nnew file mode 100644\nindex 0000000..3c0b2df\n--- /dev/null\n+++ b/drivers/vdpa/ifc/base/ifcvf.c\n@@ -0,0 +1,329 @@\n+/* SPDX-License-Identifier: BSD-3-Clause\n+ * Copyright(c) 2018 Intel Corporation\n+ */\n+\n+#include \"ifcvf.h\"\n+#include \"ifcvf_osdep.h\"\n+\n+STATIC void *\n+get_cap_addr(struct ifcvf_hw *hw, struct ifcvf_pci_cap *cap)\n+{\n+\tu8 bar = cap->bar;\n+\tu32 length = cap->length;\n+\tu32 offset = cap->offset;\n+\n+\tif (bar > IFCVF_PCI_MAX_RESOURCE - 1) {\n+\t\tDEBUGOUT(\"invalid bar: %u\\n\", bar);\n+\t\treturn NULL;\n+\t}\n+\n+\tif (offset + length < offset) {\n+\t\tDEBUGOUT(\"offset(%u) + length(%u) overflows\\n\",\n+\t\t\toffset, length);\n+\t\treturn NULL;\n+\t}\n+\n+\tif (offset + length > hw->mem_resource[cap->bar].len) {\n+\t\tDEBUGOUT(\"offset(%u) + length(%u) overflows bar length(%u)\",\n+\t\t\toffset, length, (u32)hw->mem_resource[cap->bar].len);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn hw->mem_resource[bar].addr + offset;\n+}\n+\n+int\n+ifcvf_init_hw(struct ifcvf_hw *hw, PCI_DEV *dev)\n+{\n+\tint ret;\n+\tu8 pos;\n+\tstruct ifcvf_pci_cap cap;\n+\n+\tret = PCI_READ_CONFIG_BYTE(dev, &pos, PCI_CAPABILITY_LIST);\n+\tif (ret < 0) {\n+\t\tDEBUGOUT(\"failed to read pci capability list\\n\");\n+\t\treturn -1;\n+\t}\n+\n+\twhile (pos) {\n+\t\tret = PCI_READ_CONFIG_RANGE(dev, (u32 *)&cap,\n+\t\t\t\tsizeof(cap), pos);\n+\t\tif (ret < 0) {\n+\t\t\tDEBUGOUT(\"failed to read cap at pos: %x\", pos);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (cap.cap_vndr != PCI_CAP_ID_VNDR)\n+\t\t\tgoto next;\n+\n+\t\tDEBUGOUT(\"cfg type: %u, bar: %u, offset: %u, \"\n+\t\t\t\t\"len: %u\\n\", cap.cfg_type, cap.bar,\n+\t\t\t\tcap.offset, cap.length);\n+\n+\t\tswitch (cap.cfg_type) {\n+\t\tcase IFCVF_PCI_CAP_COMMON_CFG:\n+\t\t\thw->common_cfg = get_cap_addr(hw, &cap);\n+\t\t\tbreak;\n+\t\tcase IFCVF_PCI_CAP_NOTIFY_CFG:\n+\t\t\tPCI_READ_CONFIG_DWORD(dev, &hw->notify_off_multiplier,\n+\t\t\t\t\tpos + sizeof(cap));\n+\t\t\thw->notify_base = get_cap_addr(hw, &cap);\n+\t\t\thw->notify_region = cap.bar;\n+\t\t\tbreak;\n+\t\tcase IFCVF_PCI_CAP_ISR_CFG:\n+\t\t\thw->isr = get_cap_addr(hw, &cap);\n+\t\t\tbreak;\n+\t\tcase IFCVF_PCI_CAP_DEVICE_CFG:\n+\t\t\thw->dev_cfg = get_cap_addr(hw, &cap);\n+\t\t\tbreak;\n+\t\t}\n+next:\n+\t\tpos = cap.cap_next;\n+\t}\n+\n+\thw->lm_cfg = hw->mem_resource[4].addr;\n+\n+\tif (hw->common_cfg == NULL || hw->notify_base == NULL ||\n+\t\t\thw->isr == NULL || hw->dev_cfg == NULL) {\n+\t\tDEBUGOUT(\"capability incomplete\\n\");\n+\t\treturn -1;\n+\t}\n+\n+\tDEBUGOUT(\"capability mapping:\\ncommon cfg: %p\\n\"\n+\t\t\t\"notify base: %p\\nisr cfg: %p\\ndevice cfg: %p\\n\"\n+\t\t\t\"multiplier: %u\\n\",\n+\t\t\thw->common_cfg, hw->dev_cfg,\n+\t\t\thw->isr, hw->notify_base,\n+\t\t\thw->notify_off_multiplier);\n+\n+\treturn 0;\n+}\n+\n+STATIC u8\n+ifcvf_get_status(struct ifcvf_hw *hw)\n+{\n+\treturn IFCVF_READ_REG8(&hw->common_cfg->device_status);\n+}\n+\n+STATIC void\n+ifcvf_set_status(struct ifcvf_hw *hw, u8 status)\n+{\n+\tIFCVF_WRITE_REG8(status, &hw->common_cfg->device_status);\n+}\n+\n+STATIC void\n+ifcvf_reset(struct ifcvf_hw *hw)\n+{\n+\tifcvf_set_status(hw, 0);\n+\n+\t/* flush status write */\n+\twhile (ifcvf_get_status(hw))\n+\t\tmsec_delay(1);\n+}\n+\n+STATIC void\n+ifcvf_add_status(struct ifcvf_hw *hw, u8 status)\n+{\n+\tif (status != 0)\n+\t\tstatus |= ifcvf_get_status(hw);\n+\n+\tifcvf_set_status(hw, status);\n+\tifcvf_get_status(hw);\n+}\n+\n+u64\n+ifcvf_get_features(struct ifcvf_hw *hw)\n+{\n+\tu32 features_lo, features_hi;\n+\tstruct ifcvf_pci_common_cfg *cfg = hw->common_cfg;\n+\n+\tIFCVF_WRITE_REG32(0, &cfg->device_feature_select);\n+\tfeatures_lo = IFCVF_READ_REG32(&cfg->device_feature);\n+\n+\tIFCVF_WRITE_REG32(1, &cfg->device_feature_select);\n+\tfeatures_hi = IFCVF_READ_REG32(&cfg->device_feature);\n+\n+\treturn ((u64)features_hi << 32) | features_lo;\n+}\n+\n+STATIC void\n+ifcvf_set_features(struct ifcvf_hw *hw, u64 features)\n+{\n+\tstruct ifcvf_pci_common_cfg *cfg = hw->common_cfg;\n+\n+\tIFCVF_WRITE_REG32(0, &cfg->guest_feature_select);\n+\tIFCVF_WRITE_REG32(features & ((1ULL << 32) - 1), &cfg->guest_feature);\n+\n+\tIFCVF_WRITE_REG32(1, &cfg->guest_feature_select);\n+\tIFCVF_WRITE_REG32(features >> 32, &cfg->guest_feature);\n+}\n+\n+STATIC int\n+ifcvf_config_features(struct ifcvf_hw *hw)\n+{\n+\tu64 host_features;\n+\n+\thost_features = ifcvf_get_features(hw);\n+\thw->req_features &= host_features;\n+\n+\tifcvf_set_features(hw, hw->req_features);\n+\tifcvf_add_status(hw, IFCVF_CONFIG_STATUS_FEATURES_OK);\n+\n+\tif (!(ifcvf_get_status(hw) & IFCVF_CONFIG_STATUS_FEATURES_OK)) {\n+\t\tDEBUGOUT(\"failed to set FEATURES_OK status\\n\");\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+STATIC void\n+io_write64_twopart(u64 val, u32 *lo, u32 *hi)\n+{\n+\tIFCVF_WRITE_REG32(val & ((1ULL << 32) - 1), lo);\n+\tIFCVF_WRITE_REG32(val >> 32, hi);\n+}\n+\n+STATIC int\n+ifcvf_hw_enable(struct ifcvf_hw *hw)\n+{\n+\tstruct ifcvf_pci_common_cfg *cfg;\n+\tu8 *lm_cfg;\n+\tu32 i;\n+\tu16 notify_off;\n+\n+\tcfg = hw->common_cfg;\n+\tlm_cfg = hw->lm_cfg;\n+\n+\tIFCVF_WRITE_REG16(0, &cfg->msix_config);\n+\tif (IFCVF_READ_REG16(&cfg->msix_config) == IFCVF_MSI_NO_VECTOR) {\n+\t\tDEBUGOUT(\"msix vec alloc failed for device config\\n\");\n+\t\treturn -1;\n+\t}\n+\n+\tfor (i = 0; i < hw->nr_vring; i++) {\n+\t\tIFCVF_WRITE_REG16(i, &cfg->queue_select);\n+\t\tio_write64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo,\n+\t\t\t\t&cfg->queue_desc_hi);\n+\t\tio_write64_twopart(hw->vring[i].avail, &cfg->queue_avail_lo,\n+\t\t\t\t&cfg->queue_avail_hi);\n+\t\tio_write64_twopart(hw->vring[i].used, &cfg->queue_used_lo,\n+\t\t\t\t&cfg->queue_used_hi);\n+\t\tIFCVF_WRITE_REG16(hw->vring[i].size, &cfg->queue_size);\n+\n+\t\t*(u32 *)(lm_cfg + IFCVF_LM_RING_STATE_OFFSET +\n+\t\t\t\t(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4) =\n+\t\t\t(u32)hw->vring[i].last_avail_idx |\n+\t\t\t((u32)hw->vring[i].last_used_idx << 16);\n+\n+\t\tIFCVF_WRITE_REG16(i + 1, &cfg->queue_msix_vector);\n+\t\tif (IFCVF_READ_REG16(&cfg->queue_msix_vector) ==\n+\t\t\t\tIFCVF_MSI_NO_VECTOR) {\n+\t\t\tDEBUGOUT(\"queue %u, msix vec alloc failed\\n\",\n+\t\t\t\t\ti);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tnotify_off = IFCVF_READ_REG16(&cfg->queue_notify_off);\n+\t\thw->notify_addr[i] = (void *)((u8 *)hw->notify_base +\n+\t\t\t\tnotify_off * hw->notify_off_multiplier);\n+\t\tIFCVF_WRITE_REG16(1, &cfg->queue_enable);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+STATIC void\n+ifcvf_hw_disable(struct ifcvf_hw *hw)\n+{\n+\tu32 i;\n+\tstruct ifcvf_pci_common_cfg *cfg;\n+\tu32 ring_state;\n+\n+\tcfg = hw->common_cfg;\n+\n+\tIFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->msix_config);\n+\tfor (i = 0; i < hw->nr_vring; i++) {\n+\t\tIFCVF_WRITE_REG16(i, &cfg->queue_select);\n+\t\tIFCVF_WRITE_REG16(0, &cfg->queue_enable);\n+\t\tIFCVF_WRITE_REG16(IFCVF_MSI_NO_VECTOR, &cfg->queue_msix_vector);\n+\t\tring_state = *(u32 *)(hw->lm_cfg + IFCVF_LM_RING_STATE_OFFSET +\n+\t\t\t\t(i / 2) * IFCVF_LM_CFG_SIZE + (i % 2) * 4);\n+\t\thw->vring[i].last_avail_idx = (u16)(ring_state >> 16);\n+\t\thw->vring[i].last_used_idx = (u16)(ring_state >> 16);\n+\t}\n+}\n+\n+int\n+ifcvf_start_hw(struct ifcvf_hw *hw)\n+{\n+\tifcvf_reset(hw);\n+\tifcvf_add_status(hw, IFCVF_CONFIG_STATUS_ACK);\n+\tifcvf_add_status(hw, IFCVF_CONFIG_STATUS_DRIVER);\n+\n+\tif (ifcvf_config_features(hw) < 0)\n+\t\treturn -1;\n+\n+\tif (ifcvf_hw_enable(hw) < 0)\n+\t\treturn -1;\n+\n+\tifcvf_add_status(hw, IFCVF_CONFIG_STATUS_DRIVER_OK);\n+\treturn 0;\n+}\n+\n+void\n+ifcvf_stop_hw(struct ifcvf_hw *hw)\n+{\n+\tifcvf_hw_disable(hw);\n+\tifcvf_reset(hw);\n+}\n+\n+void\n+ifcvf_enable_logging(struct ifcvf_hw *hw, u64 log_base, u64 log_size)\n+{\n+\tu8 *lm_cfg;\n+\n+\tlm_cfg = hw->lm_cfg;\n+\n+\t*(u32 *)(lm_cfg + IFCVF_LM_BASE_ADDR_LOW) =\n+\t\tlog_base & IFCVF_32_BIT_MASK;\n+\n+\t*(u32 *)(lm_cfg + IFCVF_LM_BASE_ADDR_HIGH) =\n+\t\t(log_base >> 32) & IFCVF_32_BIT_MASK;\n+\n+\t*(u32 *)(lm_cfg + IFCVF_LM_END_ADDR_LOW) =\n+\t\t(log_base + log_size) & IFCVF_32_BIT_MASK;\n+\n+\t*(u32 *)(lm_cfg + IFCVF_LM_END_ADDR_HIGH) =\n+\t\t((log_base + log_size) >> 32) & IFCVF_32_BIT_MASK;\n+\n+\t*(u32 *)(lm_cfg + IFCVF_LM_LOGGING_CTRL) = IFCVF_LM_ENABLE_VF;\n+}\n+\n+void\n+ifcvf_disable_logging(struct ifcvf_hw *hw)\n+{\n+\tu8 *lm_cfg;\n+\n+\tlm_cfg = hw->lm_cfg;\n+\t*(u32 *)(lm_cfg + IFCVF_LM_LOGGING_CTRL) = IFCVF_LM_DISABLE;\n+}\n+\n+void\n+ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid)\n+{\n+\tIFCVF_WRITE_REG16(qid, hw->notify_addr[qid]);\n+}\n+\n+u8\n+ifcvf_get_notify_region(struct ifcvf_hw *hw)\n+{\n+\treturn hw->notify_region;\n+}\n+\n+u64\n+ifcvf_get_queue_notify_off(struct ifcvf_hw *hw, int qid)\n+{\n+\treturn (u8 *)hw->notify_addr[qid] -\n+\t\t(u8 *)hw->mem_resource[hw->notify_region].addr;\n+}\ndiff --git a/drivers/vdpa/ifc/base/ifcvf.h b/drivers/vdpa/ifc/base/ifcvf.h\nnew file mode 100644\nindex 0000000..9be2770\n--- /dev/null\n+++ b/drivers/vdpa/ifc/base/ifcvf.h\n@@ -0,0 +1,162 @@\n+/* SPDX-License-Identifier: BSD-3-Clause\n+ * Copyright(c) 2018 Intel Corporation\n+ */\n+\n+#ifndef _IFCVF_H_\n+#define _IFCVF_H_\n+\n+#include \"ifcvf_osdep.h\"\n+\n+#define IFCVF_VENDOR_ID\t\t0x1AF4\n+#define IFCVF_DEVICE_ID\t\t0x1041\n+#define IFCVF_SUBSYS_VENDOR_ID\t0x8086\n+#define IFCVF_SUBSYS_DEVICE_ID\t0x001A\n+\n+#define IFCVF_MAX_QUEUES\t\t1\n+#define VIRTIO_F_IOMMU_PLATFORM\t\t33\n+\n+/* Common configuration */\n+#define IFCVF_PCI_CAP_COMMON_CFG\t1\n+/* Notifications */\n+#define IFCVF_PCI_CAP_NOTIFY_CFG\t2\n+/* ISR Status */\n+#define IFCVF_PCI_CAP_ISR_CFG\t\t3\n+/* Device specific configuration */\n+#define IFCVF_PCI_CAP_DEVICE_CFG\t4\n+/* PCI configuration access */\n+#define IFCVF_PCI_CAP_PCI_CFG\t\t5\n+\n+#define IFCVF_CONFIG_STATUS_RESET     0x00\n+#define IFCVF_CONFIG_STATUS_ACK       0x01\n+#define IFCVF_CONFIG_STATUS_DRIVER    0x02\n+#define IFCVF_CONFIG_STATUS_DRIVER_OK 0x04\n+#define IFCVF_CONFIG_STATUS_FEATURES_OK 0x08\n+#define IFCVF_CONFIG_STATUS_FAILED    0x80\n+\n+#define IFCVF_MSI_NO_VECTOR\t0xffff\n+#define IFCVF_PCI_MAX_RESOURCE\t6\n+\n+#define IFCVF_LM_CFG_SIZE\t\t0x40\n+#define IFCVF_LM_RING_STATE_OFFSET\t0x20\n+\n+#define IFCVF_LM_LOGGING_CTRL\t\t0x0\n+\n+#define IFCVF_LM_BASE_ADDR_LOW\t\t0x10\n+#define IFCVF_LM_BASE_ADDR_HIGH\t\t0x14\n+#define IFCVF_LM_END_ADDR_LOW\t\t0x18\n+#define IFCVF_LM_END_ADDR_HIGH\t\t0x1c\n+\n+#define IFCVF_LM_DISABLE\t\t0x0\n+#define IFCVF_LM_ENABLE_VF\t\t0x1\n+#define IFCVF_LM_ENABLE_PF\t\t0x3\n+#define IFCVF_LOG_BASE\t\t\t0x100000000000\n+#define IFCVF_MEDIATED_VRING\t\t0x200000000000\n+\n+#define IFCVF_32_BIT_MASK\t\t0xffffffff\n+\n+\n+struct ifcvf_pci_cap {\n+\tu8 cap_vndr;            /* Generic PCI field: PCI_CAP_ID_VNDR */\n+\tu8 cap_next;            /* Generic PCI field: next ptr. */\n+\tu8 cap_len;             /* Generic PCI field: capability length */\n+\tu8 cfg_type;            /* Identifies the structure. */\n+\tu8 bar;                 /* Where to find it. */\n+\tu8 padding[3];          /* Pad to full dword. */\n+\tu32 offset;             /* Offset within bar. */\n+\tu32 length;             /* Length of the structure, in bytes. */\n+};\n+\n+struct ifcvf_pci_notify_cap {\n+\tstruct ifcvf_pci_cap cap;\n+\tu32 notify_off_multiplier;  /* Multiplier for queue_notify_off. */\n+};\n+\n+struct ifcvf_pci_common_cfg {\n+\t/* About the whole device. */\n+\tu32 device_feature_select;\n+\tu32 device_feature;\n+\tu32 guest_feature_select;\n+\tu32 guest_feature;\n+\tu16 msix_config;\n+\tu16 num_queues;\n+\tu8 device_status;\n+\tu8 config_generation;\n+\n+\t/* About a specific virtqueue. */\n+\tu16 queue_select;\n+\tu16 queue_size;\n+\tu16 queue_msix_vector;\n+\tu16 queue_enable;\n+\tu16 queue_notify_off;\n+\tu32 queue_desc_lo;\n+\tu32 queue_desc_hi;\n+\tu32 queue_avail_lo;\n+\tu32 queue_avail_hi;\n+\tu32 queue_used_lo;\n+\tu32 queue_used_hi;\n+};\n+\n+struct ifcvf_net_config {\n+\tu8    mac[6];\n+\tu16   status;\n+\tu16   max_virtqueue_pairs;\n+} __attribute__((packed));\n+\n+struct ifcvf_pci_mem_resource {\n+\tu64      phys_addr; /**< Physical address, 0 if not resource. */\n+\tu64      len;       /**< Length of the resource. */\n+\tu8       *addr;     /**< Virtual address, NULL when not mapped. */\n+};\n+\n+struct vring_info {\n+\tu64 desc;\n+\tu64 avail;\n+\tu64 used;\n+\tu16 size;\n+\tu16 last_avail_idx;\n+\tu16 last_used_idx;\n+};\n+\n+struct ifcvf_hw {\n+\tu64    req_features;\n+\tu8     notify_region;\n+\tu32    notify_off_multiplier;\n+\tstruct ifcvf_pci_common_cfg *common_cfg;\n+\tstruct ifcvf_net_config *dev_cfg;\n+\tu8     *isr;\n+\tu16    *notify_base;\n+\tu16    *notify_addr[IFCVF_MAX_QUEUES * 2];\n+\tu8     *lm_cfg;\n+\tstruct vring_info vring[IFCVF_MAX_QUEUES * 2];\n+\tu8 nr_vring;\n+\tstruct ifcvf_pci_mem_resource mem_resource[IFCVF_PCI_MAX_RESOURCE];\n+};\n+\n+int\n+ifcvf_init_hw(struct ifcvf_hw *hw, PCI_DEV *dev);\n+\n+u64\n+ifcvf_get_features(struct ifcvf_hw *hw);\n+\n+int\n+ifcvf_start_hw(struct ifcvf_hw *hw);\n+\n+void\n+ifcvf_stop_hw(struct ifcvf_hw *hw);\n+\n+void\n+ifcvf_enable_logging(struct ifcvf_hw *hw, u64 log_base, u64 log_size);\n+\n+void\n+ifcvf_disable_logging(struct ifcvf_hw *hw);\n+\n+void\n+ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid);\n+\n+u8\n+ifcvf_get_notify_region(struct ifcvf_hw *hw);\n+\n+u64\n+ifcvf_get_queue_notify_off(struct ifcvf_hw *hw, int qid);\n+\n+#endif /* _IFCVF_H_ */\ndiff --git a/drivers/vdpa/ifc/base/ifcvf_osdep.h b/drivers/vdpa/ifc/base/ifcvf_osdep.h\nnew file mode 100644\nindex 0000000..6aef25e\n--- /dev/null\n+++ b/drivers/vdpa/ifc/base/ifcvf_osdep.h\n@@ -0,0 +1,52 @@\n+/* SPDX-License-Identifier: BSD-3-Clause\n+ * Copyright(c) 2018 Intel Corporation\n+ */\n+\n+#ifndef _IFCVF_OSDEP_H_\n+#define _IFCVF_OSDEP_H_\n+\n+#include <stdint.h>\n+#include <linux/pci_regs.h>\n+\n+#include <rte_cycles.h>\n+#include <rte_pci.h>\n+#include <rte_bus_pci.h>\n+#include <rte_log.h>\n+#include <rte_io.h>\n+\n+#define DEBUGOUT(S, args...)    RTE_LOG(DEBUG, PMD, S, ##args)\n+#define STATIC                  static\n+\n+#define msec_delay(x)\trte_delay_us_sleep(1000 * (x))\n+\n+#define IFCVF_READ_REG8(reg)\t\trte_read8(reg)\n+#define IFCVF_WRITE_REG8(val, reg)\trte_write8((val), (reg))\n+#define IFCVF_READ_REG16(reg)\t\trte_read16(reg)\n+#define IFCVF_WRITE_REG16(val, reg)\trte_write16((val), (reg))\n+#define IFCVF_READ_REG32(reg)\t\trte_read32(reg)\n+#define IFCVF_WRITE_REG32(val, reg)\trte_write32((val), (reg))\n+\n+typedef struct rte_pci_device PCI_DEV;\n+\n+#define PCI_READ_CONFIG_BYTE(dev, val, where) \\\n+\trte_pci_read_config(dev, val, 1, where)\n+\n+#define PCI_READ_CONFIG_DWORD(dev, val, where) \\\n+\trte_pci_read_config(dev, val, 4, where)\n+\n+typedef uint8_t    u8;\n+typedef int8_t     s8;\n+typedef uint16_t   u16;\n+typedef int16_t    s16;\n+typedef uint32_t   u32;\n+typedef int32_t    s32;\n+typedef int64_t    s64;\n+typedef uint64_t   u64;\n+\n+static inline int\n+PCI_READ_CONFIG_RANGE(PCI_DEV *dev, uint32_t *val, int size, int where)\n+{\n+\treturn rte_pci_read_config(dev, val, size, where);\n+}\n+\n+#endif /* _IFCVF_OSDEP_H_ */\ndiff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.c\nnew file mode 100644\nindex 0000000..da4667b\n--- /dev/null\n+++ b/drivers/vdpa/ifc/ifcvf_vdpa.c\n@@ -0,0 +1,1280 @@\n+/* SPDX-License-Identifier: BSD-3-Clause\n+ * Copyright(c) 2018 Intel Corporation\n+ */\n+\n+#include <unistd.h>\n+#include <pthread.h>\n+#include <fcntl.h>\n+#include <string.h>\n+#include <sys/ioctl.h>\n+#include <sys/epoll.h>\n+#include <linux/virtio_net.h>\n+#include <stdbool.h>\n+\n+#include <rte_malloc.h>\n+#include <rte_memory.h>\n+#include <rte_bus_pci.h>\n+#include <rte_vhost.h>\n+#include <rte_vdpa.h>\n+#include <rte_vfio.h>\n+#include <rte_spinlock.h>\n+#include <rte_log.h>\n+#include <rte_kvargs.h>\n+#include <rte_devargs.h>\n+\n+#include \"base/ifcvf.h\"\n+\n+#define DRV_LOG(level, fmt, args...) \\\n+\trte_log(RTE_LOG_ ## level, ifcvf_vdpa_logtype, \\\n+\t\t\"IFCVF %s(): \" fmt \"\\n\", __func__, ##args)\n+\n+#ifndef PAGE_SIZE\n+#define PAGE_SIZE 4096\n+#endif\n+\n+#define IFCVF_USED_RING_LEN(size) \\\n+\t((size) * sizeof(struct vring_used_elem) + sizeof(uint16_t) * 3)\n+\n+#define IFCVF_VDPA_MODE\t\t\"vdpa\"\n+#define IFCVF_SW_FALLBACK_LM\t\"sw-live-migration\"\n+\n+static const char * const ifcvf_valid_arguments[] = {\n+\tIFCVF_VDPA_MODE,\n+\tIFCVF_SW_FALLBACK_LM,\n+\tNULL\n+};\n+\n+static int ifcvf_vdpa_logtype;\n+\n+struct ifcvf_internal {\n+\tstruct rte_vdpa_dev_addr dev_addr;\n+\tstruct rte_pci_device *pdev;\n+\tstruct ifcvf_hw hw;\n+\tint vfio_container_fd;\n+\tint vfio_group_fd;\n+\tint vfio_dev_fd;\n+\tpthread_t tid;\t/* thread for notify relay */\n+\tint epfd;\n+\tint vid;\n+\tint did;\n+\tuint16_t max_queues;\n+\tuint64_t features;\n+\trte_atomic32_t started;\n+\trte_atomic32_t dev_attached;\n+\trte_atomic32_t running;\n+\trte_spinlock_t lock;\n+\tbool sw_lm;\n+\tbool sw_fallback_running;\n+\t/* mediated vring for sw fallback */\n+\tstruct vring m_vring[IFCVF_MAX_QUEUES * 2];\n+\t/* eventfd for used ring interrupt */\n+\tint intr_fd[IFCVF_MAX_QUEUES * 2];\n+};\n+\n+struct internal_list {\n+\tTAILQ_ENTRY(internal_list) next;\n+\tstruct ifcvf_internal *internal;\n+};\n+\n+TAILQ_HEAD(internal_list_head, internal_list);\n+static struct internal_list_head internal_list =\n+\tTAILQ_HEAD_INITIALIZER(internal_list);\n+\n+static pthread_mutex_t internal_list_lock = PTHREAD_MUTEX_INITIALIZER;\n+\n+static void update_used_ring(struct ifcvf_internal *internal, uint16_t qid);\n+\n+static struct internal_list *\n+find_internal_resource_by_did(int did)\n+{\n+\tint found = 0;\n+\tstruct internal_list *list;\n+\n+\tpthread_mutex_lock(&internal_list_lock);\n+\n+\tTAILQ_FOREACH(list, &internal_list, next) {\n+\t\tif (did == list->internal->did) {\n+\t\t\tfound = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&internal_list_lock);\n+\n+\tif (!found)\n+\t\treturn NULL;\n+\n+\treturn list;\n+}\n+\n+static struct internal_list *\n+find_internal_resource_by_dev(struct rte_pci_device *pdev)\n+{\n+\tint found = 0;\n+\tstruct internal_list *list;\n+\n+\tpthread_mutex_lock(&internal_list_lock);\n+\n+\tTAILQ_FOREACH(list, &internal_list, next) {\n+\t\tif (pdev == list->internal->pdev) {\n+\t\t\tfound = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tpthread_mutex_unlock(&internal_list_lock);\n+\n+\tif (!found)\n+\t\treturn NULL;\n+\n+\treturn list;\n+}\n+\n+static int\n+ifcvf_vfio_setup(struct ifcvf_internal *internal)\n+{\n+\tstruct rte_pci_device *dev = internal->pdev;\n+\tchar devname[RTE_DEV_NAME_MAX_LEN] = {0};\n+\tint iommu_group_num;\n+\tint i, ret;\n+\n+\tinternal->vfio_dev_fd = -1;\n+\tinternal->vfio_group_fd = -1;\n+\tinternal->vfio_container_fd = -1;\n+\n+\trte_pci_device_name(&dev->addr, devname, RTE_DEV_NAME_MAX_LEN);\n+\tret = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), devname,\n+\t\t\t&iommu_group_num);\n+\tif (ret <= 0) {\n+\t\tDRV_LOG(ERR, \"%s failed to get IOMMU group\", devname);\n+\t\treturn -1;\n+\t}\n+\n+\tinternal->vfio_container_fd = rte_vfio_container_create();\n+\tif (internal->vfio_container_fd < 0)\n+\t\treturn -1;\n+\n+\tinternal->vfio_group_fd = rte_vfio_container_group_bind(\n+\t\t\tinternal->vfio_container_fd, iommu_group_num);\n+\tif (internal->vfio_group_fd < 0)\n+\t\tgoto err;\n+\n+\tif (rte_pci_map_device(dev))\n+\t\tgoto err;\n+\n+\tinternal->vfio_dev_fd = dev->intr_handle.vfio_dev_fd;\n+\n+\tfor (i = 0; i < RTE_MIN(PCI_MAX_RESOURCE, IFCVF_PCI_MAX_RESOURCE);\n+\t\t\ti++) {\n+\t\tinternal->hw.mem_resource[i].addr =\n+\t\t\tinternal->pdev->mem_resource[i].addr;\n+\t\tinternal->hw.mem_resource[i].phys_addr =\n+\t\t\tinternal->pdev->mem_resource[i].phys_addr;\n+\t\tinternal->hw.mem_resource[i].len =\n+\t\t\tinternal->pdev->mem_resource[i].len;\n+\t}\n+\n+\treturn 0;\n+\n+err:\n+\trte_vfio_container_destroy(internal->vfio_container_fd);\n+\treturn -1;\n+}\n+\n+static int\n+ifcvf_dma_map(struct ifcvf_internal *internal, int do_map)\n+{\n+\tuint32_t i;\n+\tint ret;\n+\tstruct rte_vhost_memory *mem = NULL;\n+\tint vfio_container_fd;\n+\n+\tret = rte_vhost_get_mem_table(internal->vid, &mem);\n+\tif (ret < 0) {\n+\t\tDRV_LOG(ERR, \"failed to get VM memory layout.\");\n+\t\tgoto exit;\n+\t}\n+\n+\tvfio_container_fd = internal->vfio_container_fd;\n+\n+\tfor (i = 0; i < mem->nregions; i++) {\n+\t\tstruct rte_vhost_mem_region *reg;\n+\n+\t\treg = &mem->regions[i];\n+\t\tDRV_LOG(INFO, \"%s, region %u: HVA 0x%\" PRIx64 \", \"\n+\t\t\t\"GPA 0x%\" PRIx64 \", size 0x%\" PRIx64 \".\",\n+\t\t\tdo_map ? \"DMA map\" : \"DMA unmap\", i,\n+\t\t\treg->host_user_addr, reg->guest_phys_addr, reg->size);\n+\n+\t\tif (do_map) {\n+\t\t\tret = rte_vfio_container_dma_map(vfio_container_fd,\n+\t\t\t\treg->host_user_addr, reg->guest_phys_addr,\n+\t\t\t\treg->size);\n+\t\t\tif (ret < 0) {\n+\t\t\t\tDRV_LOG(ERR, \"DMA map failed.\");\n+\t\t\t\tgoto exit;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tret = rte_vfio_container_dma_unmap(vfio_container_fd,\n+\t\t\t\treg->host_user_addr, reg->guest_phys_addr,\n+\t\t\t\treg->size);\n+\t\t\tif (ret < 0) {\n+\t\t\t\tDRV_LOG(ERR, \"DMA unmap failed.\");\n+\t\t\t\tgoto exit;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+exit:\n+\tif (mem)\n+\t\tfree(mem);\n+\treturn ret;\n+}\n+\n+static uint64_t\n+hva_to_gpa(int vid, uint64_t hva)\n+{\n+\tstruct rte_vhost_memory *mem = NULL;\n+\tstruct rte_vhost_mem_region *reg;\n+\tuint32_t i;\n+\tuint64_t gpa = 0;\n+\n+\tif (rte_vhost_get_mem_table(vid, &mem) < 0)\n+\t\tgoto exit;\n+\n+\tfor (i = 0; i < mem->nregions; i++) {\n+\t\treg = &mem->regions[i];\n+\n+\t\tif (hva >= reg->host_user_addr &&\n+\t\t\t\thva < reg->host_user_addr + reg->size) {\n+\t\t\tgpa = hva - reg->host_user_addr + reg->guest_phys_addr;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+exit:\n+\tif (mem)\n+\t\tfree(mem);\n+\treturn gpa;\n+}\n+\n+static int\n+vdpa_ifcvf_start(struct ifcvf_internal *internal)\n+{\n+\tstruct ifcvf_hw *hw = &internal->hw;\n+\tint i, nr_vring;\n+\tint vid;\n+\tstruct rte_vhost_vring vq;\n+\tuint64_t gpa;\n+\n+\tvid = internal->vid;\n+\tnr_vring = rte_vhost_get_vring_num(vid);\n+\trte_vhost_get_negotiated_features(vid, &hw->req_features);\n+\n+\tfor (i = 0; i < nr_vring; i++) {\n+\t\trte_vhost_get_vhost_vring(vid, i, &vq);\n+\t\tgpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.desc);\n+\t\tif (gpa == 0) {\n+\t\t\tDRV_LOG(ERR, \"Fail to get GPA for descriptor ring.\");\n+\t\t\treturn -1;\n+\t\t}\n+\t\thw->vring[i].desc = gpa;\n+\n+\t\tgpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.avail);\n+\t\tif (gpa == 0) {\n+\t\t\tDRV_LOG(ERR, \"Fail to get GPA for available ring.\");\n+\t\t\treturn -1;\n+\t\t}\n+\t\thw->vring[i].avail = gpa;\n+\n+\t\tgpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used);\n+\t\tif (gpa == 0) {\n+\t\t\tDRV_LOG(ERR, \"Fail to get GPA for used ring.\");\n+\t\t\treturn -1;\n+\t\t}\n+\t\thw->vring[i].used = gpa;\n+\n+\t\thw->vring[i].size = vq.size;\n+\t\trte_vhost_get_vring_base(vid, i, &hw->vring[i].last_avail_idx,\n+\t\t\t\t&hw->vring[i].last_used_idx);\n+\t}\n+\thw->nr_vring = i;\n+\n+\treturn ifcvf_start_hw(&internal->hw);\n+}\n+\n+static void\n+vdpa_ifcvf_stop(struct ifcvf_internal *internal)\n+{\n+\tstruct ifcvf_hw *hw = &internal->hw;\n+\tuint32_t i;\n+\tint vid;\n+\tuint64_t features = 0;\n+\tuint64_t log_base = 0, log_size = 0;\n+\tuint64_t len;\n+\n+\tvid = internal->vid;\n+\tifcvf_stop_hw(hw);\n+\n+\tfor (i = 0; i < hw->nr_vring; i++)\n+\t\trte_vhost_set_vring_base(vid, i, hw->vring[i].last_avail_idx,\n+\t\t\t\thw->vring[i].last_used_idx);\n+\n+\tif (internal->sw_lm)\n+\t\treturn;\n+\n+\trte_vhost_get_negotiated_features(vid, &features);\n+\tif (RTE_VHOST_NEED_LOG(features)) {\n+\t\tifcvf_disable_logging(hw);\n+\t\trte_vhost_get_log_base(internal->vid, &log_base, &log_size);\n+\t\trte_vfio_container_dma_unmap(internal->vfio_container_fd,\n+\t\t\t\tlog_base, IFCVF_LOG_BASE, log_size);\n+\t\t/*\n+\t\t * IFCVF marks dirty memory pages for only packet buffer,\n+\t\t * SW helps to mark the used ring as dirty after device stops.\n+\t\t */\n+\t\tfor (i = 0; i < hw->nr_vring; i++) {\n+\t\t\tlen = IFCVF_USED_RING_LEN(hw->vring[i].size);\n+\t\t\trte_vhost_log_used_vring(vid, i, 0, len);\n+\t\t}\n+\t}\n+}\n+\n+#define MSIX_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \\\n+\t\tsizeof(int) * (IFCVF_MAX_QUEUES * 2 + 1))\n+static int\n+vdpa_enable_vfio_intr(struct ifcvf_internal *internal, bool m_rx)\n+{\n+\tint ret;\n+\tuint32_t i, nr_vring;\n+\tchar irq_set_buf[MSIX_IRQ_SET_BUF_LEN];\n+\tstruct vfio_irq_set *irq_set;\n+\tint *fd_ptr;\n+\tstruct rte_vhost_vring vring;\n+\tint fd;\n+\n+\tvring.callfd = -1;\n+\n+\tnr_vring = rte_vhost_get_vring_num(internal->vid);\n+\n+\tirq_set = (struct vfio_irq_set *)irq_set_buf;\n+\tirq_set->argsz = sizeof(irq_set_buf);\n+\tirq_set->count = nr_vring + 1;\n+\tirq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD |\n+\t\t\t VFIO_IRQ_SET_ACTION_TRIGGER;\n+\tirq_set->index = VFIO_PCI_MSIX_IRQ_INDEX;\n+\tirq_set->start = 0;\n+\tfd_ptr = (int *)&irq_set->data;\n+\tfd_ptr[RTE_INTR_VEC_ZERO_OFFSET] = internal->pdev->intr_handle.fd;\n+\n+\tfor (i = 0; i < nr_vring; i++)\n+\t\tinternal->intr_fd[i] = -1;\n+\n+\tfor (i = 0; i < nr_vring; i++) {\n+\t\trte_vhost_get_vhost_vring(internal->vid, i, &vring);\n+\t\tfd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;\n+\t\tif ((i & 1) == 0 && m_rx == true) {\n+\t\t\tfd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);\n+\t\t\tif (fd < 0) {\n+\t\t\t\tDRV_LOG(ERR, \"can't setup eventfd: %s\",\n+\t\t\t\t\tstrerror(errno));\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t\tinternal->intr_fd[i] = fd;\n+\t\t\tfd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd;\n+\t\t}\n+\t}\n+\n+\tret = ioctl(internal->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);\n+\tif (ret) {\n+\t\tDRV_LOG(ERR, \"Error enabling MSI-X interrupts: %s\",\n+\t\t\t\tstrerror(errno));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int\n+vdpa_disable_vfio_intr(struct ifcvf_internal *internal)\n+{\n+\tint ret;\n+\tuint32_t i, nr_vring;\n+\tchar irq_set_buf[MSIX_IRQ_SET_BUF_LEN];\n+\tstruct vfio_irq_set *irq_set;\n+\n+\tirq_set = (struct vfio_irq_set *)irq_set_buf;\n+\tirq_set->argsz = sizeof(irq_set_buf);\n+\tirq_set->count = 0;\n+\tirq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER;\n+\tirq_set->index = VFIO_PCI_MSIX_IRQ_INDEX;\n+\tirq_set->start = 0;\n+\n+\tnr_vring = rte_vhost_get_vring_num(internal->vid);\n+\tfor (i = 0; i < nr_vring; i++) {\n+\t\tif (internal->intr_fd[i] >= 0)\n+\t\t\tclose(internal->intr_fd[i]);\n+\t\tinternal->intr_fd[i] = -1;\n+\t}\n+\n+\tret = ioctl(internal->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);\n+\tif (ret) {\n+\t\tDRV_LOG(ERR, \"Error disabling MSI-X interrupts: %s\",\n+\t\t\t\tstrerror(errno));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void *\n+notify_relay(void *arg)\n+{\n+\tint i, kickfd, epfd, nfds = 0;\n+\tuint32_t qid, q_num;\n+\tstruct epoll_event events[IFCVF_MAX_QUEUES * 2];\n+\tstruct epoll_event ev;\n+\tuint64_t buf;\n+\tint nbytes;\n+\tstruct rte_vhost_vring vring;\n+\tstruct ifcvf_internal *internal = (struct ifcvf_internal *)arg;\n+\tstruct ifcvf_hw *hw = &internal->hw;\n+\n+\tq_num = rte_vhost_get_vring_num(internal->vid);\n+\n+\tepfd = epoll_create(IFCVF_MAX_QUEUES * 2);\n+\tif (epfd < 0) {\n+\t\tDRV_LOG(ERR, \"failed to create epoll instance.\");\n+\t\treturn NULL;\n+\t}\n+\tinternal->epfd = epfd;\n+\n+\tvring.kickfd = -1;\n+\tfor (qid = 0; qid < q_num; qid++) {\n+\t\tev.events = EPOLLIN | EPOLLPRI;\n+\t\trte_vhost_get_vhost_vring(internal->vid, qid, &vring);\n+\t\tev.data.u64 = qid | (uint64_t)vring.kickfd << 32;\n+\t\tif (epoll_ctl(epfd, EPOLL_CTL_ADD, vring.kickfd, &ev) < 0) {\n+\t\t\tDRV_LOG(ERR, \"epoll add error: %s\", strerror(errno));\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\n+\tfor (;;) {\n+\t\tnfds = epoll_wait(epfd, events, q_num, -1);\n+\t\tif (nfds < 0) {\n+\t\t\tif (errno == EINTR)\n+\t\t\t\tcontinue;\n+\t\t\tDRV_LOG(ERR, \"epoll_wait return fail\\n\");\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\tfor (i = 0; i < nfds; i++) {\n+\t\t\tqid = events[i].data.u32;\n+\t\t\tkickfd = (uint32_t)(events[i].data.u64 >> 32);\n+\t\t\tdo {\n+\t\t\t\tnbytes = read(kickfd, &buf, 8);\n+\t\t\t\tif (nbytes < 0) {\n+\t\t\t\t\tif (errno == EINTR ||\n+\t\t\t\t\t    errno == EWOULDBLOCK ||\n+\t\t\t\t\t    errno == EAGAIN)\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\tDRV_LOG(INFO, \"Error reading \"\n+\t\t\t\t\t\t\"kickfd: %s\",\n+\t\t\t\t\t\tstrerror(errno));\n+\t\t\t\t}\n+\t\t\t\tbreak;\n+\t\t\t} while (1);\n+\n+\t\t\tifcvf_notify_queue(hw, qid);\n+\t\t}\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static int\n+setup_notify_relay(struct ifcvf_internal *internal)\n+{\n+\tint ret;\n+\n+\tret = pthread_create(&internal->tid, NULL, notify_relay,\n+\t\t\t(void *)internal);\n+\tif (ret) {\n+\t\tDRV_LOG(ERR, \"failed to create notify relay pthread.\");\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int\n+unset_notify_relay(struct ifcvf_internal *internal)\n+{\n+\tvoid *status;\n+\n+\tif (internal->tid) {\n+\t\tpthread_cancel(internal->tid);\n+\t\tpthread_join(internal->tid, &status);\n+\t}\n+\tinternal->tid = 0;\n+\n+\tif (internal->epfd >= 0)\n+\t\tclose(internal->epfd);\n+\tinternal->epfd = -1;\n+\n+\treturn 0;\n+}\n+\n+static int\n+update_datapath(struct ifcvf_internal *internal)\n+{\n+\tint ret;\n+\n+\trte_spinlock_lock(&internal->lock);\n+\n+\tif (!rte_atomic32_read(&internal->running) &&\n+\t    (rte_atomic32_read(&internal->started) &&\n+\t     rte_atomic32_read(&internal->dev_attached))) {\n+\t\tret = ifcvf_dma_map(internal, 1);\n+\t\tif (ret)\n+\t\t\tgoto err;\n+\n+\t\tret = vdpa_enable_vfio_intr(internal, 0);\n+\t\tif (ret)\n+\t\t\tgoto err;\n+\n+\t\tret = vdpa_ifcvf_start(internal);\n+\t\tif (ret)\n+\t\t\tgoto err;\n+\n+\t\tret = setup_notify_relay(internal);\n+\t\tif (ret)\n+\t\t\tgoto err;\n+\n+\t\trte_atomic32_set(&internal->running, 1);\n+\t} else if (rte_atomic32_read(&internal->running) &&\n+\t\t   (!rte_atomic32_read(&internal->started) ||\n+\t\t    !rte_atomic32_read(&internal->dev_attached))) {\n+\t\tret = unset_notify_relay(internal);\n+\t\tif (ret)\n+\t\t\tgoto err;\n+\n+\t\tvdpa_ifcvf_stop(internal);\n+\n+\t\tret = vdpa_disable_vfio_intr(internal);\n+\t\tif (ret)\n+\t\t\tgoto err;\n+\n+\t\tret = ifcvf_dma_map(internal, 0);\n+\t\tif (ret)\n+\t\t\tgoto err;\n+\n+\t\trte_atomic32_set(&internal->running, 0);\n+\t}\n+\n+\trte_spinlock_unlock(&internal->lock);\n+\treturn 0;\n+err:\n+\trte_spinlock_unlock(&internal->lock);\n+\treturn ret;\n+}\n+\n+static int\n+m_ifcvf_start(struct ifcvf_internal *internal)\n+{\n+\tstruct ifcvf_hw *hw = &internal->hw;\n+\tuint32_t i, nr_vring;\n+\tint vid, ret;\n+\tstruct rte_vhost_vring vq;\n+\tvoid *vring_buf;\n+\tuint64_t m_vring_iova = IFCVF_MEDIATED_VRING;\n+\tuint64_t size;\n+\tuint64_t gpa;\n+\n+\tmemset(&vq, 0, sizeof(vq));\n+\tvid = internal->vid;\n+\tnr_vring = rte_vhost_get_vring_num(vid);\n+\trte_vhost_get_negotiated_features(vid, &hw->req_features);\n+\n+\tfor (i = 0; i < nr_vring; i++) {\n+\t\trte_vhost_get_vhost_vring(vid, i, &vq);\n+\n+\t\tsize = RTE_ALIGN_CEIL(vring_size(vq.size, PAGE_SIZE),\n+\t\t\t\tPAGE_SIZE);\n+\t\tvring_buf = rte_zmalloc(\"ifcvf\", size, PAGE_SIZE);\n+\t\tvring_init(&internal->m_vring[i], vq.size, vring_buf,\n+\t\t\t\tPAGE_SIZE);\n+\n+\t\tret = rte_vfio_container_dma_map(internal->vfio_container_fd,\n+\t\t\t(uint64_t)(uintptr_t)vring_buf, m_vring_iova, size);\n+\t\tif (ret < 0) {\n+\t\t\tDRV_LOG(ERR, \"mediated vring DMA map failed.\");\n+\t\t\tgoto error;\n+\t\t}\n+\n+\t\tgpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.desc);\n+\t\tif (gpa == 0) {\n+\t\t\tDRV_LOG(ERR, \"Fail to get GPA for descriptor ring.\");\n+\t\t\treturn -1;\n+\t\t}\n+\t\thw->vring[i].desc = gpa;\n+\n+\t\tgpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.avail);\n+\t\tif (gpa == 0) {\n+\t\t\tDRV_LOG(ERR, \"Fail to get GPA for available ring.\");\n+\t\t\treturn -1;\n+\t\t}\n+\t\thw->vring[i].avail = gpa;\n+\n+\t\t/* Direct I/O for Tx queue, relay for Rx queue */\n+\t\tif (i & 1) {\n+\t\t\tgpa = hva_to_gpa(vid, (uint64_t)(uintptr_t)vq.used);\n+\t\t\tif (gpa == 0) {\n+\t\t\t\tDRV_LOG(ERR, \"Fail to get GPA for used ring.\");\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t\thw->vring[i].used = gpa;\n+\t\t} else {\n+\t\t\thw->vring[i].used = m_vring_iova +\n+\t\t\t\t(char *)internal->m_vring[i].used -\n+\t\t\t\t(char *)internal->m_vring[i].desc;\n+\t\t}\n+\n+\t\thw->vring[i].size = vq.size;\n+\n+\t\trte_vhost_get_vring_base(vid, i,\n+\t\t\t\t&internal->m_vring[i].avail->idx,\n+\t\t\t\t&internal->m_vring[i].used->idx);\n+\n+\t\trte_vhost_get_vring_base(vid, i, &hw->vring[i].last_avail_idx,\n+\t\t\t\t&hw->vring[i].last_used_idx);\n+\n+\t\tm_vring_iova += size;\n+\t}\n+\thw->nr_vring = nr_vring;\n+\n+\treturn ifcvf_start_hw(&internal->hw);\n+\n+error:\n+\tfor (i = 0; i < nr_vring; i++)\n+\t\tif (internal->m_vring[i].desc)\n+\t\t\trte_free(internal->m_vring[i].desc);\n+\n+\treturn -1;\n+}\n+\n+static int\n+m_ifcvf_stop(struct ifcvf_internal *internal)\n+{\n+\tint vid;\n+\tuint32_t i;\n+\tstruct rte_vhost_vring vq;\n+\tstruct ifcvf_hw *hw = &internal->hw;\n+\tuint64_t m_vring_iova = IFCVF_MEDIATED_VRING;\n+\tuint64_t size, len;\n+\n+\tvid = internal->vid;\n+\tifcvf_stop_hw(hw);\n+\n+\tfor (i = 0; i < hw->nr_vring; i++) {\n+\t\t/* synchronize remaining new used entries if any */\n+\t\tif ((i & 1) == 0)\n+\t\t\tupdate_used_ring(internal, i);\n+\n+\t\trte_vhost_get_vhost_vring(vid, i, &vq);\n+\t\tlen = IFCVF_USED_RING_LEN(vq.size);\n+\t\trte_vhost_log_used_vring(vid, i, 0, len);\n+\n+\t\tsize = RTE_ALIGN_CEIL(vring_size(vq.size, PAGE_SIZE),\n+\t\t\t\tPAGE_SIZE);\n+\t\trte_vfio_container_dma_unmap(internal->vfio_container_fd,\n+\t\t\t(uint64_t)(uintptr_t)internal->m_vring[i].desc,\n+\t\t\tm_vring_iova, size);\n+\n+\t\trte_vhost_set_vring_base(vid, i, hw->vring[i].last_avail_idx,\n+\t\t\t\thw->vring[i].last_used_idx);\n+\t\trte_free(internal->m_vring[i].desc);\n+\t\tm_vring_iova += size;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void\n+update_used_ring(struct ifcvf_internal *internal, uint16_t qid)\n+{\n+\trte_vdpa_relay_vring_used(internal->vid, qid, &internal->m_vring[qid]);\n+\trte_vhost_vring_call(internal->vid, qid);\n+}\n+\n+static void *\n+vring_relay(void *arg)\n+{\n+\tint i, vid, epfd, fd, nfds;\n+\tstruct ifcvf_internal *internal = (struct ifcvf_internal *)arg;\n+\tstruct rte_vhost_vring vring;\n+\tuint16_t qid, q_num;\n+\tstruct epoll_event events[IFCVF_MAX_QUEUES * 4];\n+\tstruct epoll_event ev;\n+\tint nbytes;\n+\tuint64_t buf;\n+\n+\tvid = internal->vid;\n+\tq_num = rte_vhost_get_vring_num(vid);\n+\n+\t/* add notify fd and interrupt fd to epoll */\n+\tepfd = epoll_create(IFCVF_MAX_QUEUES * 2);\n+\tif (epfd < 0) {\n+\t\tDRV_LOG(ERR, \"failed to create epoll instance.\");\n+\t\treturn NULL;\n+\t}\n+\tinternal->epfd = epfd;\n+\n+\tvring.kickfd = -1;\n+\tfor (qid = 0; qid < q_num; qid++) {\n+\t\tev.events = EPOLLIN | EPOLLPRI;\n+\t\trte_vhost_get_vhost_vring(vid, qid, &vring);\n+\t\tev.data.u64 = qid << 1 | (uint64_t)vring.kickfd << 32;\n+\t\tif (epoll_ctl(epfd, EPOLL_CTL_ADD, vring.kickfd, &ev) < 0) {\n+\t\t\tDRV_LOG(ERR, \"epoll add error: %s\", strerror(errno));\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\n+\tfor (qid = 0; qid < q_num; qid += 2) {\n+\t\tev.events = EPOLLIN | EPOLLPRI;\n+\t\t/* leave a flag to mark it's for interrupt */\n+\t\tev.data.u64 = 1 | qid << 1 |\n+\t\t\t(uint64_t)internal->intr_fd[qid] << 32;\n+\t\tif (epoll_ctl(epfd, EPOLL_CTL_ADD, internal->intr_fd[qid], &ev)\n+\t\t\t\t< 0) {\n+\t\t\tDRV_LOG(ERR, \"epoll add error: %s\", strerror(errno));\n+\t\t\treturn NULL;\n+\t\t}\n+\t\tupdate_used_ring(internal, qid);\n+\t}\n+\n+\t/* start relay with a first kick */\n+\tfor (qid = 0; qid < q_num; qid++)\n+\t\tifcvf_notify_queue(&internal->hw, qid);\n+\n+\t/* listen to the events and react accordingly */\n+\tfor (;;) {\n+\t\tnfds = epoll_wait(epfd, events, q_num * 2, -1);\n+\t\tif (nfds < 0) {\n+\t\t\tif (errno == EINTR)\n+\t\t\t\tcontinue;\n+\t\t\tDRV_LOG(ERR, \"epoll_wait return fail\\n\");\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\tfor (i = 0; i < nfds; i++) {\n+\t\t\tfd = (uint32_t)(events[i].data.u64 >> 32);\n+\t\t\tdo {\n+\t\t\t\tnbytes = read(fd, &buf, 8);\n+\t\t\t\tif (nbytes < 0) {\n+\t\t\t\t\tif (errno == EINTR ||\n+\t\t\t\t\t    errno == EWOULDBLOCK ||\n+\t\t\t\t\t    errno == EAGAIN)\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\tDRV_LOG(INFO, \"Error reading \"\n+\t\t\t\t\t\t\"kickfd: %s\",\n+\t\t\t\t\t\tstrerror(errno));\n+\t\t\t\t}\n+\t\t\t\tbreak;\n+\t\t\t} while (1);\n+\n+\t\t\tqid = events[i].data.u32 >> 1;\n+\n+\t\t\tif (events[i].data.u32 & 1)\n+\t\t\t\tupdate_used_ring(internal, qid);\n+\t\t\telse\n+\t\t\t\tifcvf_notify_queue(&internal->hw, qid);\n+\t\t}\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static int\n+setup_vring_relay(struct ifcvf_internal *internal)\n+{\n+\tint ret;\n+\n+\tret = pthread_create(&internal->tid, NULL, vring_relay,\n+\t\t\t(void *)internal);\n+\tif (ret) {\n+\t\tDRV_LOG(ERR, \"failed to create ring relay pthread.\");\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int\n+unset_vring_relay(struct ifcvf_internal *internal)\n+{\n+\tvoid *status;\n+\n+\tif (internal->tid) {\n+\t\tpthread_cancel(internal->tid);\n+\t\tpthread_join(internal->tid, &status);\n+\t}\n+\tinternal->tid = 0;\n+\n+\tif (internal->epfd >= 0)\n+\t\tclose(internal->epfd);\n+\tinternal->epfd = -1;\n+\n+\treturn 0;\n+}\n+\n+static int\n+ifcvf_sw_fallback_switchover(struct ifcvf_internal *internal)\n+{\n+\tint ret;\n+\tint vid = internal->vid;\n+\n+\t/* stop the direct IO data path */\n+\tunset_notify_relay(internal);\n+\tvdpa_ifcvf_stop(internal);\n+\tvdpa_disable_vfio_intr(internal);\n+\n+\tret = rte_vhost_host_notifier_ctrl(vid, false);\n+\tif (ret && ret != -ENOTSUP)\n+\t\tgoto error;\n+\n+\t/* set up interrupt for interrupt relay */\n+\tret = vdpa_enable_vfio_intr(internal, 1);\n+\tif (ret)\n+\t\tgoto unmap;\n+\n+\t/* config the VF */\n+\tret = m_ifcvf_start(internal);\n+\tif (ret)\n+\t\tgoto unset_intr;\n+\n+\t/* set up vring relay thread */\n+\tret = setup_vring_relay(internal);\n+\tif (ret)\n+\t\tgoto stop_vf;\n+\n+\trte_vhost_host_notifier_ctrl(vid, true);\n+\n+\tinternal->sw_fallback_running = true;\n+\n+\treturn 0;\n+\n+stop_vf:\n+\tm_ifcvf_stop(internal);\n+unset_intr:\n+\tvdpa_disable_vfio_intr(internal);\n+unmap:\n+\tifcvf_dma_map(internal, 0);\n+error:\n+\treturn -1;\n+}\n+\n+static int\n+ifcvf_dev_config(int vid)\n+{\n+\tint did;\n+\tstruct internal_list *list;\n+\tstruct ifcvf_internal *internal;\n+\n+\tdid = rte_vhost_get_vdpa_device_id(vid);\n+\tlist = find_internal_resource_by_did(did);\n+\tif (list == NULL) {\n+\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n+\t\treturn -1;\n+\t}\n+\n+\tinternal = list->internal;\n+\tinternal->vid = vid;\n+\trte_atomic32_set(&internal->dev_attached, 1);\n+\tupdate_datapath(internal);\n+\n+\tif (rte_vhost_host_notifier_ctrl(vid, true) != 0)\n+\t\tDRV_LOG(NOTICE, \"vDPA (%d): software relay is used.\", did);\n+\n+\treturn 0;\n+}\n+\n+static int\n+ifcvf_dev_close(int vid)\n+{\n+\tint did;\n+\tstruct internal_list *list;\n+\tstruct ifcvf_internal *internal;\n+\n+\tdid = rte_vhost_get_vdpa_device_id(vid);\n+\tlist = find_internal_resource_by_did(did);\n+\tif (list == NULL) {\n+\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n+\t\treturn -1;\n+\t}\n+\n+\tinternal = list->internal;\n+\n+\tif (internal->sw_fallback_running) {\n+\t\t/* unset ring relay */\n+\t\tunset_vring_relay(internal);\n+\n+\t\t/* reset VF */\n+\t\tm_ifcvf_stop(internal);\n+\n+\t\t/* remove interrupt setting */\n+\t\tvdpa_disable_vfio_intr(internal);\n+\n+\t\t/* unset DMA map for guest memory */\n+\t\tifcvf_dma_map(internal, 0);\n+\n+\t\tinternal->sw_fallback_running = false;\n+\t} else {\n+\t\trte_atomic32_set(&internal->dev_attached, 0);\n+\t\tupdate_datapath(internal);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int\n+ifcvf_set_features(int vid)\n+{\n+\tuint64_t features = 0;\n+\tint did;\n+\tstruct internal_list *list;\n+\tstruct ifcvf_internal *internal;\n+\tuint64_t log_base = 0, log_size = 0;\n+\n+\tdid = rte_vhost_get_vdpa_device_id(vid);\n+\tlist = find_internal_resource_by_did(did);\n+\tif (list == NULL) {\n+\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n+\t\treturn -1;\n+\t}\n+\n+\tinternal = list->internal;\n+\trte_vhost_get_negotiated_features(vid, &features);\n+\n+\tif (!RTE_VHOST_NEED_LOG(features))\n+\t\treturn 0;\n+\n+\tif (internal->sw_lm) {\n+\t\tifcvf_sw_fallback_switchover(internal);\n+\t} else {\n+\t\trte_vhost_get_log_base(vid, &log_base, &log_size);\n+\t\trte_vfio_container_dma_map(internal->vfio_container_fd,\n+\t\t\t\tlog_base, IFCVF_LOG_BASE, log_size);\n+\t\tifcvf_enable_logging(&internal->hw, IFCVF_LOG_BASE, log_size);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int\n+ifcvf_get_vfio_group_fd(int vid)\n+{\n+\tint did;\n+\tstruct internal_list *list;\n+\n+\tdid = rte_vhost_get_vdpa_device_id(vid);\n+\tlist = find_internal_resource_by_did(did);\n+\tif (list == NULL) {\n+\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n+\t\treturn -1;\n+\t}\n+\n+\treturn list->internal->vfio_group_fd;\n+}\n+\n+static int\n+ifcvf_get_vfio_device_fd(int vid)\n+{\n+\tint did;\n+\tstruct internal_list *list;\n+\n+\tdid = rte_vhost_get_vdpa_device_id(vid);\n+\tlist = find_internal_resource_by_did(did);\n+\tif (list == NULL) {\n+\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n+\t\treturn -1;\n+\t}\n+\n+\treturn list->internal->vfio_dev_fd;\n+}\n+\n+static int\n+ifcvf_get_notify_area(int vid, int qid, uint64_t *offset, uint64_t *size)\n+{\n+\tint did;\n+\tstruct internal_list *list;\n+\tstruct ifcvf_internal *internal;\n+\tstruct vfio_region_info reg = { .argsz = sizeof(reg) };\n+\tint ret;\n+\n+\tdid = rte_vhost_get_vdpa_device_id(vid);\n+\tlist = find_internal_resource_by_did(did);\n+\tif (list == NULL) {\n+\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n+\t\treturn -1;\n+\t}\n+\n+\tinternal = list->internal;\n+\n+\treg.index = ifcvf_get_notify_region(&internal->hw);\n+\tret = ioctl(internal->vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, &reg);\n+\tif (ret) {\n+\t\tDRV_LOG(ERR, \"Get not get device region info: %s\",\n+\t\t\t\tstrerror(errno));\n+\t\treturn -1;\n+\t}\n+\n+\t*offset = ifcvf_get_queue_notify_off(&internal->hw, qid) + reg.offset;\n+\t*size = 0x1000;\n+\n+\treturn 0;\n+}\n+\n+static int\n+ifcvf_get_queue_num(int did, uint32_t *queue_num)\n+{\n+\tstruct internal_list *list;\n+\n+\tlist = find_internal_resource_by_did(did);\n+\tif (list == NULL) {\n+\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n+\t\treturn -1;\n+\t}\n+\n+\t*queue_num = list->internal->max_queues;\n+\n+\treturn 0;\n+}\n+\n+static int\n+ifcvf_get_vdpa_features(int did, uint64_t *features)\n+{\n+\tstruct internal_list *list;\n+\n+\tlist = find_internal_resource_by_did(did);\n+\tif (list == NULL) {\n+\t\tDRV_LOG(ERR, \"Invalid device id: %d\", did);\n+\t\treturn -1;\n+\t}\n+\n+\t*features = list->internal->features;\n+\n+\treturn 0;\n+}\n+\n+#define VDPA_SUPPORTED_PROTOCOL_FEATURES \\\n+\t\t(1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK | \\\n+\t\t 1ULL << VHOST_USER_PROTOCOL_F_SLAVE_REQ | \\\n+\t\t 1ULL << VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD | \\\n+\t\t 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER | \\\n+\t\t 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD)\n+static int\n+ifcvf_get_protocol_features(int did __rte_unused, uint64_t *features)\n+{\n+\t*features = VDPA_SUPPORTED_PROTOCOL_FEATURES;\n+\treturn 0;\n+}\n+\n+static struct rte_vdpa_dev_ops ifcvf_ops = {\n+\t.get_queue_num = ifcvf_get_queue_num,\n+\t.get_features = ifcvf_get_vdpa_features,\n+\t.get_protocol_features = ifcvf_get_protocol_features,\n+\t.dev_conf = ifcvf_dev_config,\n+\t.dev_close = ifcvf_dev_close,\n+\t.set_vring_state = NULL,\n+\t.set_features = ifcvf_set_features,\n+\t.migration_done = NULL,\n+\t.get_vfio_group_fd = ifcvf_get_vfio_group_fd,\n+\t.get_vfio_device_fd = ifcvf_get_vfio_device_fd,\n+\t.get_notify_area = ifcvf_get_notify_area,\n+};\n+\n+static inline int\n+open_int(const char *key __rte_unused, const char *value, void *extra_args)\n+{\n+\tuint16_t *n = extra_args;\n+\n+\tif (value == NULL || extra_args == NULL)\n+\t\treturn -EINVAL;\n+\n+\t*n = (uint16_t)strtoul(value, NULL, 0);\n+\tif (*n == USHRT_MAX && errno == ERANGE)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+static int\n+ifcvf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,\n+\t\tstruct rte_pci_device *pci_dev)\n+{\n+\tuint64_t features;\n+\tstruct ifcvf_internal *internal = NULL;\n+\tstruct internal_list *list = NULL;\n+\tint vdpa_mode = 0;\n+\tint sw_fallback_lm = 0;\n+\tstruct rte_kvargs *kvlist = NULL;\n+\tint ret = 0;\n+\n+\tif (rte_eal_process_type() != RTE_PROC_PRIMARY)\n+\t\treturn 0;\n+\n+\tif (!pci_dev->device.devargs)\n+\t\treturn 1;\n+\n+\tkvlist = rte_kvargs_parse(pci_dev->device.devargs->args,\n+\t\t\tifcvf_valid_arguments);\n+\tif (kvlist == NULL)\n+\t\treturn 1;\n+\n+\t/* probe only when vdpa mode is specified */\n+\tif (rte_kvargs_count(kvlist, IFCVF_VDPA_MODE) == 0) {\n+\t\trte_kvargs_free(kvlist);\n+\t\treturn 1;\n+\t}\n+\n+\tret = rte_kvargs_process(kvlist, IFCVF_VDPA_MODE, &open_int,\n+\t\t\t&vdpa_mode);\n+\tif (ret < 0 || vdpa_mode == 0) {\n+\t\trte_kvargs_free(kvlist);\n+\t\treturn 1;\n+\t}\n+\n+\tlist = rte_zmalloc(\"ifcvf\", sizeof(*list), 0);\n+\tif (list == NULL)\n+\t\tgoto error;\n+\n+\tinternal = rte_zmalloc(\"ifcvf\", sizeof(*internal), 0);\n+\tif (internal == NULL)\n+\t\tgoto error;\n+\n+\tinternal->pdev = pci_dev;\n+\trte_spinlock_init(&internal->lock);\n+\n+\tif (ifcvf_vfio_setup(internal) < 0) {\n+\t\tDRV_LOG(ERR, \"failed to setup device %s\", pci_dev->name);\n+\t\tgoto error;\n+\t}\n+\n+\tif (ifcvf_init_hw(&internal->hw, internal->pdev) < 0) {\n+\t\tDRV_LOG(ERR, \"failed to init device %s\", pci_dev->name);\n+\t\tgoto error;\n+\t}\n+\n+\tinternal->max_queues = IFCVF_MAX_QUEUES;\n+\tfeatures = ifcvf_get_features(&internal->hw);\n+\tinternal->features = (features &\n+\t\t~(1ULL << VIRTIO_F_IOMMU_PLATFORM)) |\n+\t\t(1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) |\n+\t\t(1ULL << VIRTIO_NET_F_CTRL_VQ) |\n+\t\t(1ULL << VIRTIO_NET_F_STATUS) |\n+\t\t(1ULL << VHOST_USER_F_PROTOCOL_FEATURES) |\n+\t\t(1ULL << VHOST_F_LOG_ALL);\n+\n+\tinternal->dev_addr.pci_addr = pci_dev->addr;\n+\tinternal->dev_addr.type = PCI_ADDR;\n+\tlist->internal = internal;\n+\n+\tif (rte_kvargs_count(kvlist, IFCVF_SW_FALLBACK_LM)) {\n+\t\tret = rte_kvargs_process(kvlist, IFCVF_SW_FALLBACK_LM,\n+\t\t\t\t&open_int, &sw_fallback_lm);\n+\t\tif (ret < 0)\n+\t\t\tgoto error;\n+\t}\n+\tinternal->sw_lm = sw_fallback_lm;\n+\n+\tinternal->did = rte_vdpa_register_device(&internal->dev_addr,\n+\t\t\t\t&ifcvf_ops);\n+\tif (internal->did < 0) {\n+\t\tDRV_LOG(ERR, \"failed to register device %s\", pci_dev->name);\n+\t\tgoto error;\n+\t}\n+\n+\tpthread_mutex_lock(&internal_list_lock);\n+\tTAILQ_INSERT_TAIL(&internal_list, list, next);\n+\tpthread_mutex_unlock(&internal_list_lock);\n+\n+\trte_atomic32_set(&internal->started, 1);\n+\tupdate_datapath(internal);\n+\n+\trte_kvargs_free(kvlist);\n+\treturn 0;\n+\n+error:\n+\trte_kvargs_free(kvlist);\n+\trte_free(list);\n+\trte_free(internal);\n+\treturn -1;\n+}\n+\n+static int\n+ifcvf_pci_remove(struct rte_pci_device *pci_dev)\n+{\n+\tstruct ifcvf_internal *internal;\n+\tstruct internal_list *list;\n+\n+\tif (rte_eal_process_type() != RTE_PROC_PRIMARY)\n+\t\treturn 0;\n+\n+\tlist = find_internal_resource_by_dev(pci_dev);\n+\tif (list == NULL) {\n+\t\tDRV_LOG(ERR, \"Invalid device: %s\", pci_dev->name);\n+\t\treturn -1;\n+\t}\n+\n+\tinternal = list->internal;\n+\trte_atomic32_set(&internal->started, 0);\n+\tupdate_datapath(internal);\n+\n+\trte_pci_unmap_device(internal->pdev);\n+\trte_vfio_container_destroy(internal->vfio_container_fd);\n+\trte_vdpa_unregister_device(internal->did);\n+\n+\tpthread_mutex_lock(&internal_list_lock);\n+\tTAILQ_REMOVE(&internal_list, list, next);\n+\tpthread_mutex_unlock(&internal_list_lock);\n+\n+\trte_free(list);\n+\trte_free(internal);\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * IFCVF has the same vendor ID and device ID as virtio net PCI\n+ * device, with its specific subsystem vendor ID and device ID.\n+ */\n+static const struct rte_pci_id pci_id_ifcvf_map[] = {\n+\t{ .class_id = RTE_CLASS_ANY_ID,\n+\t  .vendor_id = IFCVF_VENDOR_ID,\n+\t  .device_id = IFCVF_DEVICE_ID,\n+\t  .subsystem_vendor_id = IFCVF_SUBSYS_VENDOR_ID,\n+\t  .subsystem_device_id = IFCVF_SUBSYS_DEVICE_ID,\n+\t},\n+\n+\t{ .vendor_id = 0, /* sentinel */\n+\t},\n+};\n+\n+static struct rte_pci_driver rte_ifcvf_vdpa = {\n+\t.id_table = pci_id_ifcvf_map,\n+\t.drv_flags = 0,\n+\t.probe = ifcvf_pci_probe,\n+\t.remove = ifcvf_pci_remove,\n+};\n+\n+RTE_PMD_REGISTER_PCI(net_ifcvf, rte_ifcvf_vdpa);\n+RTE_PMD_REGISTER_PCI_TABLE(net_ifcvf, pci_id_ifcvf_map);\n+RTE_PMD_REGISTER_KMOD_DEP(net_ifcvf, \"* vfio-pci\");\n+\n+RTE_INIT(ifcvf_vdpa_init_log)\n+{\n+\tifcvf_vdpa_logtype = rte_log_register(\"pmd.net.ifcvf_vdpa\");\n+\tif (ifcvf_vdpa_logtype >= 0)\n+\t\trte_log_set_level(ifcvf_vdpa_logtype, RTE_LOG_NOTICE);\n+}\ndiff --git a/drivers/vdpa/ifc/meson.build b/drivers/vdpa/ifc/meson.build\nnew file mode 100644\nindex 0000000..adc9ed9\n--- /dev/null\n+++ b/drivers/vdpa/ifc/meson.build\n@@ -0,0 +1,9 @@\n+# SPDX-License-Identifier: BSD-3-Clause\n+# Copyright(c) 2018 Intel Corporation\n+\n+build = dpdk_conf.has('RTE_LIBRTE_VHOST')\n+reason = 'missing dependency, DPDK vhost library'\n+allow_experimental_apis = true\n+sources = files('ifcvf_vdpa.c', 'base/ifcvf.c')\n+includes += include_directories('base')\n+deps += 'vhost'\ndiff --git a/drivers/vdpa/ifc/rte_pmd_ifc_version.map b/drivers/vdpa/ifc/rte_pmd_ifc_version.map\nnew file mode 100644\nindex 0000000..f9f17e4\n--- /dev/null\n+++ b/drivers/vdpa/ifc/rte_pmd_ifc_version.map\n@@ -0,0 +1,3 @@\n+DPDK_20.0 {\n+\tlocal: *;\n+};\ndiff --git a/drivers/vdpa/meson.build b/drivers/vdpa/meson.build\nindex a839ff5..fd164d3 100644\n--- a/drivers/vdpa/meson.build\n+++ b/drivers/vdpa/meson.build\n@@ -1,7 +1,7 @@\n #   SPDX-License-Identifier: BSD-3-Clause\n #   Copyright 2019 Mellanox Technologies, Ltd\n \n-drivers = []\n+drivers = ['ifc']\n std_deps = ['bus_pci', 'kvargs']\n std_deps += ['vhost']\n config_flag_fmt = 'RTE_LIBRTE_@0@_PMD'\n",
    "prefixes": [
        "v2",
        "3/3"
    ]
}