Patch Detail

HTTP 200 OK
Allow: GET, PUT, PATCH, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "id": 85619,
    "url": "https://patches.dpdk.org/api/patches/85619/?format=api",
    "web_url": "https://patches.dpdk.org/project/dpdk/patch/1608586402-20145-1-git-send-email-longli@linuxonhyperv.com/",
    "project": {
        "id": 1,
        "url": "https://patches.dpdk.org/api/projects/1/?format=api",
        "name": "DPDK",
        "link_name": "dpdk",
        "list_id": "dev.dpdk.org",
        "list_email": "dev@dpdk.org",
        "web_url": "http://core.dpdk.org",
        "scm_url": "git://dpdk.org/dpdk",
        "webscm_url": "http://git.dpdk.org/dpdk",
        "list_archive_url": "https://inbox.dpdk.org/dev",
        "list_archive_url_format": "https://inbox.dpdk.org/dev/{}",
        "commit_url_format": ""
    },
    "msgid": "<1608586402-20145-1-git-send-email-longli@linuxonhyperv.com>",
    "list_archive_url": "https://inbox.dpdk.org/dev/1608586402-20145-1-git-send-email-longli@linuxonhyperv.com",
    "date": "2020-12-21T21:33:22",
    "name": "[v2,2/2] net/netvsc: support VF device hot add/remove",
    "commit_ref": null,
    "pull_url": null,
    "state": "accepted",
    "archived": true,
    "hash": "8446ebbfd1f7cc16edf291b2484df5e0340827cf",
    "submitter": {
        "id": 1784,
        "url": "https://patches.dpdk.org/api/people/1784/?format=api",
        "name": "Long Li",
        "email": "longli@linuxonhyperv.com"
    },
    "delegate": {
        "id": 1,
        "url": "https://patches.dpdk.org/api/users/1/?format=api",
        "username": "tmonjalo",
        "first_name": "Thomas",
        "last_name": "Monjalon",
        "email": "thomas@monjalon.net"
    },
    "mbox": "https://patches.dpdk.org/project/dpdk/patch/1608586402-20145-1-git-send-email-longli@linuxonhyperv.com/mbox/",
    "series": [
        {
            "id": 14410,
            "url": "https://patches.dpdk.org/api/series/14410/?format=api",
            "web_url": "https://patches.dpdk.org/project/dpdk/list/?series=14410",
            "date": "2020-12-21T21:32:36",
            "name": "[v2,1/2] eal/hotplug: allow monitor to be setup by multiple places",
            "version": 2,
            "mbox": "https://patches.dpdk.org/series/14410/mbox/"
        }
    ],
    "comments": "https://patches.dpdk.org/api/patches/85619/comments/",
    "check": "success",
    "checks": "https://patches.dpdk.org/api/patches/85619/checks/",
    "tags": {},
    "related": [],
    "headers": {
        "Return-Path": "<dev-bounces@dpdk.org>",
        "X-Original-To": "patchwork@inbox.dpdk.org",
        "Delivered-To": "patchwork@inbox.dpdk.org",
        "Received": [
            "from dpdk.org (dpdk.org [92.243.14.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 3FFE7A09EF;\n\tMon, 21 Dec 2020 22:33:29 +0100 (CET)",
            "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 20E0DCB79;\n\tMon, 21 Dec 2020 22:33:28 +0100 (CET)",
            "from linux.microsoft.com (linux.microsoft.com [13.77.154.182])\n by dpdk.org (Postfix) with ESMTP id 0D11DCB76\n for <dev@dpdk.org>; Mon, 21 Dec 2020 22:33:26 +0100 (CET)",
            "by linux.microsoft.com (Postfix, from userid 1004)\n id 603F620B83DE; Mon, 21 Dec 2020 13:33:25 -0800 (PST)"
        ],
        "DKIM-Filter": "OpenDKIM Filter v2.11.0 linux.microsoft.com 603F620B83DE",
        "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxonhyperv.com;\n s=default; t=1608586405;\n bh=Ltip68XPGUGl0mMwHFL3aRVJ3a4qozxiTxjXx4nqdIw=;\n h=From:To:Cc:Subject:Date:In-Reply-To:References:From;\n b=qQwKXf0YnDi1WMIa7G7uoI4KoJn3l4yKZlbL0VCSm4jc7wCuDqx3v9JDVdt4EHejw\n ziJ4tnHsQoBJhR+A1zzL25OhAGUg4wj/iC5CkDSahbOf79abC/WCrMMUh3V8tgmcHq\n vZDJ7UfKpa2oq6u51hQL3cCQYDAU++RgCLtP5cLc=",
        "From": "Long Li <longli@linuxonhyperv.com>",
        "To": "Stephen Hemminger <sthemmin@microsoft.com>",
        "Cc": "dev@dpdk.org,\n\tLong Li <longli@microsoft.com>",
        "Date": "Mon, 21 Dec 2020 13:33:22 -0800",
        "Message-Id": "<1608586402-20145-1-git-send-email-longli@linuxonhyperv.com>",
        "X-Mailer": "git-send-email 1.8.3.1",
        "In-Reply-To": "<1606809383-26660-2-git-send-email-longli@linuxonhyperv.com>",
        "References": "<1606809383-26660-2-git-send-email-longli@linuxonhyperv.com>",
        "Subject": "[dpdk-dev] [PATCH v2 2/2] net/netvsc: support VF device hot\n\tadd/remove",
        "X-BeenThere": "dev@dpdk.org",
        "X-Mailman-Version": "2.1.15",
        "Precedence": "list",
        "List-Id": "DPDK patches and discussions <dev.dpdk.org>",
        "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>",
        "List-Archive": "<http://mails.dpdk.org/archives/dev/>",
        "List-Post": "<mailto:dev@dpdk.org>",
        "List-Help": "<mailto:dev-request@dpdk.org?subject=help>",
        "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>",
        "Errors-To": "dev-bounces@dpdk.org",
        "Sender": "\"dev\" <dev-bounces@dpdk.org>"
    },
    "content": "From: Long Li <longli@microsoft.com>\n\nWhen a VF device is present, netvsc can send or receive packets over the\nVF device. The VF device driver communicates directly with the PCI device\nvia the PF from the host hypervisor. This is faster than exchanging data\nwith netvsp via vmbus, i.e. syntheic path.\n\nIn Azure and Hyper-v environments, VF device can be hot added or hot\nremoved at anytime while guest VM is running. This patch improves netvsc\nto support VF device hot add/remove.\n\n1. netvsc monitors all system hot add activities over the PCI bus. When it\ndetects a VF device is added to the system and is managed under this\nnetvsc device, it asks EAL to probe and start this VF device, then it\nattaches and switches data path to the VF device.\n\n2. After a VF device is attached to netvsc, netvsc monitors this device on\nhot remove. When this VF device is hot removed, netvsc switches data path\nto synthetic, stops this VF device and removes it from EAL.\n\n3. If any failure happens during a VF device hot remove or add, the netvsc\nfalls back to synthetic path for all data traffic.\n\nSigned-off-by: Long Li <longli@microsoft.com>\n---\n drivers/net/netvsc/hn_ethdev.c | 166 ++++++++++++++++-\n drivers/net/netvsc/hn_nvs.c    |   4 +-\n drivers/net/netvsc/hn_nvs.h    |   2 +-\n drivers/net/netvsc/hn_rxtx.c   |  36 ++--\n drivers/net/netvsc/hn_var.h    |  52 ++++--\n drivers/net/netvsc/hn_vf.c     | 318 +++++++++++++++++++++++++++------\n 6 files changed, 489 insertions(+), 89 deletions(-)",
    "diff": "diff --git a/drivers/net/netvsc/hn_ethdev.c b/drivers/net/netvsc/hn_ethdev.c\nindex 49f954305d..5a401b4b06 100644\n--- a/drivers/net/netvsc/hn_ethdev.c\n+++ b/drivers/net/netvsc/hn_ethdev.c\n@@ -9,6 +9,10 @@\n #include <stdio.h>\n #include <errno.h>\n #include <unistd.h>\n+#include <dirent.h>\n+#include <net/if.h>\n+#include <net/if_arp.h>\n+#include <sys/ioctl.h>\n \n #include <rte_ethdev.h>\n #include <rte_memcpy.h>\n@@ -27,6 +31,7 @@\n #include <rte_eal.h>\n #include <rte_dev.h>\n #include <rte_bus_vmbus.h>\n+#include <rte_alarm.h>\n \n #include \"hn_logs.h\"\n #include \"hn_var.h\"\n@@ -50,6 +55,9 @@\n #define NETVSC_ARG_TXBREAK \"tx_copybreak\"\n #define NETVSC_ARG_RX_EXTMBUF_ENABLE \"rx_extmbuf_enable\"\n \n+/* The max number of retry when hot adding a VF device */\n+#define NETVSC_MAX_HOTADD_RETRY 10\n+\n struct hn_xstats_name_off {\n \tchar name[RTE_ETH_XSTATS_NAME_SIZE];\n \tunsigned int offset;\n@@ -542,6 +550,128 @@ static int hn_subchan_configure(struct hn_data *hv,\n \treturn err;\n }\n \n+static void netvsc_hotplug_retry(void *args)\n+{\n+\tint ret;\n+\tstruct hn_data *hv = args;\n+\tstruct rte_eth_dev *dev = &rte_eth_devices[hv->port_id];\n+\tstruct rte_devargs *d = &hv->devargs;\n+\tchar buf[256];\n+\n+\tDIR *di;\n+\tstruct dirent *dir;\n+\tstruct ifreq req;\n+\tstruct rte_ether_addr eth_addr;\n+\tint s;\n+\n+\tPMD_DRV_LOG(DEBUG, \"%s: retry count %d\\n\",\n+\t\t    __func__, hv->eal_hot_plug_retry);\n+\n+\tif (hv->eal_hot_plug_retry++ > NETVSC_MAX_HOTADD_RETRY)\n+\t\treturn;\n+\n+\tsnprintf(buf, sizeof(buf), \"/sys/bus/pci/devices/%s/net\", d->name);\n+\tdi = opendir(buf);\n+\tif (!di) {\n+\t\tPMD_DRV_LOG(DEBUG, \"%s: can't open directory %s, \"\n+\t\t\t    \"retrying in 1 second\\n\", __func__, buf);\n+\t\tgoto retry;\n+\t}\n+\n+\twhile ((dir = readdir(di))) {\n+\t\t/* Skip . and .. directories */\n+\t\tif (!strcmp(dir->d_name, \".\") || !strcmp(dir->d_name, \"..\"))\n+\t\t\tcontinue;\n+\n+\t\t/* trying to get mac address if this is a network device*/\n+\t\ts = socket(PF_INET, SOCK_DGRAM, IPPROTO_IP);\n+\t\tif (s == -1) {\n+\t\t\tPMD_DRV_LOG(ERR, \"Failed to create socket errno %d\\n\",\n+\t\t\t\t    errno);\n+\t\t\tbreak;\n+\t\t}\n+\t\tstrlcpy(req.ifr_name, dir->d_name, sizeof(req.ifr_name));\n+\t\tret = ioctl(s, SIOCGIFHWADDR, &req);\n+\t\tclose(s);\n+\t\tif (ret == -1) {\n+\t\t\tPMD_DRV_LOG(ERR, \"Failed to send SIOCGIFHWADDR for \"\n+\t\t\t\t    \"device %s\\n\", dir->d_name);\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (req.ifr_hwaddr.sa_family != ARPHRD_ETHER) {\n+\t\t\tclosedir(di);\n+\t\t\treturn;\n+\t\t}\n+\t\tmemcpy(eth_addr.addr_bytes, req.ifr_hwaddr.sa_data,\n+\t\t       RTE_DIM(eth_addr.addr_bytes));\n+\n+\t\tif (rte_is_same_ether_addr(&eth_addr, dev->data->mac_addrs)) {\n+\t\t\tPMD_DRV_LOG(NOTICE, \"Found matching MAC address, \"\n+\t\t\t\t    \"adding device %s network name %s\\n\",\n+\t\t\t\t    d->name, dir->d_name);\n+\t\t\tret = rte_eal_hotplug_add(d->bus->name, d->name,\n+\t\t\t\t\t\t  d->args);\n+\t\t\tif (ret) {\n+\t\t\t\tPMD_DRV_LOG(ERR,\n+\t\t\t\t\t    \"Failed to add PCI device %s\\n\",\n+\t\t\t\t\t    d->name);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t\t/* When the code reaches here, we either have already added\n+\t\t * the device, or its MAC address did not match.\n+\t\t */\n+\t\tclosedir(di);\n+\t\treturn;\n+\t}\n+\tclosedir(di);\n+retry:\n+\t/* The device is still being initialized, retry after 1 second */\n+\trte_eal_alarm_set(1000000, netvsc_hotplug_retry, hv);\n+}\n+\n+static void\n+netvsc_hotadd_callback(const char *device_name, enum rte_dev_event_type type,\n+\t\t       void *arg)\n+{\n+\tstruct hn_data *hv = arg;\n+\tstruct rte_devargs *d = &hv->devargs;\n+\tint ret;\n+\n+\tPMD_DRV_LOG(INFO, \"Device notification type=%d device_name=%s\\n\",\n+\t\t    type, device_name);\n+\n+\tswitch (type) {\n+\tcase RTE_DEV_EVENT_ADD:\n+\t\t/* if we already has a VF, don't check on hot add */\n+\t\tif (hv->vf_ctx.vf_state > vf_removed)\n+\t\t\tbreak;\n+\n+\t\tret = rte_devargs_parse(d, device_name);\n+\t\tif (ret) {\n+\t\t\tPMD_DRV_LOG(ERR,\n+\t\t\t\t    \"devargs parsing failed ret=%d\\n\", ret);\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tif (!strcmp(d->bus->name, \"pci\")) {\n+\t\t\t/* Start the process of figuring out if this\n+\t\t\t * PCI device is a VF device\n+\t\t\t */\n+\t\t\thv->eal_hot_plug_retry = 0;\n+\t\t\trte_eal_alarm_set(1000000, netvsc_hotplug_retry, hv);\n+\t\t}\n+\n+\t\t/* We will switch to VF on RDNIS configure message\n+\t\t * sent from VSP\n+\t\t */\n+\n+\t\tbreak;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+}\n+\n static int hn_dev_configure(struct rte_eth_dev *dev)\n {\n \tstruct rte_eth_conf *dev_conf = &dev->data->dev_conf;\n@@ -617,7 +747,7 @@ static int hn_dev_configure(struct rte_eth_dev *dev)\n \t\t}\n \t}\n \n-\treturn hn_vf_configure(dev, dev_conf);\n+\treturn hn_vf_configure_locked(dev, dev_conf);\n }\n \n static int hn_dev_stats_get(struct rte_eth_dev *dev,\n@@ -827,6 +957,14 @@ hn_dev_start(struct rte_eth_dev *dev)\n \n \tPMD_INIT_FUNC_TRACE();\n \n+\t/* Register to monitor hot plug events */\n+\terror = rte_dev_event_callback_register(NULL, netvsc_hotadd_callback,\n+\t\t\t\t\t\thv);\n+\tif (error) {\n+\t\tPMD_DRV_LOG(ERR, \"failed to register device event callback\\n\");\n+\t\treturn error;\n+\t}\n+\n \terror = hn_rndis_set_rxfilter(hv,\n \t\t\t\t      NDIS_PACKET_TYPE_BROADCAST |\n \t\t\t\t      NDIS_PACKET_TYPE_ALL_MULTICAST |\n@@ -853,6 +991,7 @@ hn_dev_stop(struct rte_eth_dev *dev)\n \tPMD_INIT_FUNC_TRACE();\n \tdev->data->dev_started = 0;\n \n+\trte_dev_event_callback_unregister(NULL, netvsc_hotadd_callback, hv);\n \thn_rndis_set_rxfilter(hv, 0);\n \treturn hn_vf_stop(dev);\n }\n@@ -861,11 +1000,14 @@ static int\n hn_dev_close(struct rte_eth_dev *dev)\n {\n \tint ret;\n+\tstruct hn_data *hv = dev->data->dev_private;\n \n \tPMD_INIT_FUNC_TRACE();\n \tif (rte_eal_process_type() != RTE_PROC_PRIMARY)\n \t\treturn 0;\n \n+\trte_eal_alarm_cancel(netvsc_hotplug_retry, &hv->devargs);\n+\n \tret = hn_vf_close(dev);\n \thn_dev_free_queues(dev);\n \n@@ -990,7 +1132,10 @@ eth_hn_dev_init(struct rte_eth_dev *eth_dev)\n \thv->max_queues = 1;\n \n \trte_rwlock_init(&hv->vf_lock);\n-\thv->vf_port = HN_INVALID_PORT;\n+\thv->vf_ctx.vf_vsc_switched = false;\n+\thv->vf_ctx.vf_vsp_reported = false;\n+\thv->vf_ctx.vf_attached = false;\n+\thv->vf_ctx.vf_state = vf_unknown;\n \n \terr = hn_parse_args(eth_dev);\n \tif (err)\n@@ -1044,12 +1189,10 @@ eth_hn_dev_init(struct rte_eth_dev *eth_dev)\n \thv->max_queues = RTE_MIN(rxr_cnt, (unsigned int)max_chan);\n \n \t/* If VF was reported but not added, do it now */\n-\tif (hv->vf_present && !hn_vf_attached(hv)) {\n+\tif (hv->vf_ctx.vf_vsp_reported && !hv->vf_ctx.vf_vsc_switched) {\n \t\tPMD_INIT_LOG(DEBUG, \"Adding VF device\");\n \n \t\terr = hn_vf_add(eth_dev, hv);\n-\t\tif (err)\n-\t\t\thv->vf_present = 0;\n \t}\n \n \treturn 0;\n@@ -1095,15 +1238,23 @@ static int eth_hn_probe(struct rte_vmbus_driver *drv __rte_unused,\n \n \tPMD_INIT_FUNC_TRACE();\n \n+\tret = rte_dev_event_monitor_start();\n+\tif (ret) {\n+\t\tPMD_DRV_LOG(ERR, \"Failed to start device event monitoring\\n\");\n+\t\treturn ret;\n+\t}\n+\n \teth_dev = eth_dev_vmbus_allocate(dev, sizeof(struct hn_data));\n \tif (!eth_dev)\n \t\treturn -ENOMEM;\n \n \tret = eth_hn_dev_init(eth_dev);\n-\tif (ret)\n+\tif (ret) {\n \t\teth_dev_vmbus_release(eth_dev);\n-\telse\n+\t\trte_dev_event_monitor_stop();\n+\t} else {\n \t\trte_eth_dev_probing_finish(eth_dev);\n+\t}\n \n \treturn ret;\n }\n@@ -1124,6 +1275,7 @@ static int eth_hn_remove(struct rte_vmbus_device *dev)\n \t\treturn ret;\n \n \teth_dev_vmbus_release(eth_dev);\n+\trte_dev_event_monitor_stop();\n \treturn 0;\n }\n \ndiff --git a/drivers/net/netvsc/hn_nvs.c b/drivers/net/netvsc/hn_nvs.c\nindex eeb82ab9ee..a0ee7d8bfa 100644\n--- a/drivers/net/netvsc/hn_nvs.c\n+++ b/drivers/net/netvsc/hn_nvs.c\n@@ -569,7 +569,7 @@ hn_nvs_alloc_subchans(struct hn_data *hv, uint32_t *nsubch)\n \treturn 0;\n }\n \n-void\n+int\n hn_nvs_set_datapath(struct hn_data *hv, uint32_t path)\n {\n \tstruct hn_nvs_datapath dp;\n@@ -588,4 +588,6 @@ hn_nvs_set_datapath(struct hn_data *hv, uint32_t path)\n \t\t\t    \"send set datapath failed: %d\",\n \t\t\t    error);\n \t}\n+\n+\treturn error;\n }\ndiff --git a/drivers/net/netvsc/hn_nvs.h b/drivers/net/netvsc/hn_nvs.h\nindex 015839e364..3766d2ee34 100644\n--- a/drivers/net/netvsc/hn_nvs.h\n+++ b/drivers/net/netvsc/hn_nvs.h\n@@ -212,7 +212,7 @@ int\thn_nvs_attach(struct hn_data *hv, unsigned int mtu);\n void\thn_nvs_detach(struct hn_data *hv);\n void\thn_nvs_ack_rxbuf(struct vmbus_channel *chan, uint64_t tid);\n int\thn_nvs_alloc_subchans(struct hn_data *hv, uint32_t *nsubch);\n-void\thn_nvs_set_datapath(struct hn_data *hv, uint32_t path);\n+int\thn_nvs_set_datapath(struct hn_data *hv, uint32_t path);\n void\thn_nvs_handle_vfassoc(struct rte_eth_dev *dev,\n \t\t\t      const struct vmbus_chanpkt_hdr *hdr,\n \t\t\t      const void *data);\ndiff --git a/drivers/net/netvsc/hn_rxtx.c b/drivers/net/netvsc/hn_rxtx.c\nindex 015662fdb4..0f4ef0100b 100644\n--- a/drivers/net/netvsc/hn_rxtx.c\n+++ b/drivers/net/netvsc/hn_rxtx.c\n@@ -1492,16 +1492,20 @@ hn_xmit_pkts(void *ptxq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts)\n \t\thn_process_events(hv, txq->queue_id, 0);\n \n \t/* Transmit over VF if present and up */\n-\trte_rwlock_read_lock(&hv->vf_lock);\n-\tvf_dev = hn_get_vf_dev(hv);\n-\tif (vf_dev && vf_dev->data->dev_started) {\n-\t\tvoid *sub_q = vf_dev->data->tx_queues[queue_id];\n-\n-\t\tnb_tx = (*vf_dev->tx_pkt_burst)(sub_q, tx_pkts, nb_pkts);\n+\tif (hv->vf_ctx.vf_vsc_switched) {\n+\t\trte_rwlock_read_lock(&hv->vf_lock);\n+\t\tvf_dev = hn_get_vf_dev(hv);\n+\t\tif (hv->vf_ctx.vf_vsc_switched && vf_dev &&\n+\t\t    vf_dev->data->dev_started) {\n+\t\t\tvoid *sub_q = vf_dev->data->tx_queues[queue_id];\n+\n+\t\t\tnb_tx = (*vf_dev->tx_pkt_burst)\n+\t\t\t\t\t(sub_q, tx_pkts, nb_pkts);\n+\t\t\trte_rwlock_read_unlock(&hv->vf_lock);\n+\t\t\treturn nb_tx;\n+\t\t}\n \t\trte_rwlock_read_unlock(&hv->vf_lock);\n-\t\treturn nb_tx;\n \t}\n-\trte_rwlock_read_unlock(&hv->vf_lock);\n \n \tfor (nb_tx = 0; nb_tx < nb_pkts; nb_tx++) {\n \t\tstruct rte_mbuf *m = tx_pkts[nb_tx];\n@@ -1614,13 +1618,17 @@ hn_recv_pkts(void *prxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)\n \t\t\t\t\t   (void **)rx_pkts, nb_pkts, NULL);\n \n \t/* If VF is available, check that as well */\n-\trte_rwlock_read_lock(&hv->vf_lock);\n-\tvf_dev = hn_get_vf_dev(hv);\n-\tif (vf_dev && vf_dev->data->dev_started)\n-\t\tnb_rcv += hn_recv_vf(vf_dev->data->port_id, rxq,\n-\t\t\t\t     rx_pkts + nb_rcv, nb_pkts - nb_rcv);\n+\tif (hv->vf_ctx.vf_vsc_switched) {\n+\t\trte_rwlock_read_lock(&hv->vf_lock);\n+\t\tvf_dev = hn_get_vf_dev(hv);\n+\t\tif (hv->vf_ctx.vf_vsc_switched && vf_dev &&\n+\t\t    vf_dev->data->dev_started)\n+\t\t\tnb_rcv += hn_recv_vf(vf_dev->data->port_id, rxq,\n+\t\t\t\t\t     rx_pkts + nb_rcv,\n+\t\t\t\t\t     nb_pkts - nb_rcv);\n \n-\trte_rwlock_read_unlock(&hv->vf_lock);\n+\t\trte_rwlock_read_unlock(&hv->vf_lock);\n+\t}\n \treturn nb_rcv;\n }\n \ndiff --git a/drivers/net/netvsc/hn_var.h b/drivers/net/netvsc/hn_var.h\nindex bd874c6b4d..b7405ca726 100644\n--- a/drivers/net/netvsc/hn_var.h\n+++ b/drivers/net/netvsc/hn_var.h\n@@ -105,14 +105,37 @@ struct hn_rx_bufinfo {\n \n #define HN_INVALID_PORT\tUINT16_MAX\n \n+enum vf_device_state {\n+\tvf_unknown = 0,\n+\tvf_removed,\n+\tvf_configured,\n+\tvf_started,\n+\tvf_stopped,\n+};\n+\n+struct hn_vf_ctx {\n+\tuint16_t\tvf_port;\n+\n+\t/* We have taken ownership of this VF port from DPDK */\n+\tbool\t\tvf_attached;\n+\n+\t/* VSC has requested to switch data path to VF */\n+\tbool\t\tvf_vsc_switched;\n+\n+\t/* VSP has reported the VF is present for this NIC */\n+\tbool\t\tvf_vsp_reported;\n+\n+\tenum vf_device_state\tvf_state;\n+};\n+\n struct hn_data {\n \tstruct rte_vmbus_device *vmbus;\n \tstruct hn_rx_queue *primary;\n \trte_rwlock_t    vf_lock;\n \tuint16_t\tport_id;\n-\tuint16_t\tvf_port;\n \n-\tuint8_t\t\tvf_present;\n+\tstruct hn_vf_ctx\tvf_ctx;\n+\n \tuint8_t\t\tclosed;\n \tuint8_t\t\tvlan_strip;\n \n@@ -153,6 +176,9 @@ struct hn_data {\n \tstruct rte_eth_dev_owner owner;\n \n \tstruct vmbus_channel *channels[HN_MAX_CHANNELS];\n+\n+\tstruct rte_devargs devargs;\n+\tint\t\teal_hot_plug_retry;\n };\n \n static inline struct vmbus_channel *\n@@ -196,13 +222,6 @@ uint32_t hn_dev_rx_queue_count(struct rte_eth_dev *dev, uint16_t queue_id);\n int\thn_dev_rx_queue_status(void *rxq, uint16_t offset);\n void\thn_dev_free_queues(struct rte_eth_dev *dev);\n \n-/* Check if VF is attached */\n-static inline bool\n-hn_vf_attached(const struct hn_data *hv)\n-{\n-\treturn hv->vf_port != HN_INVALID_PORT;\n-}\n-\n /*\n  * Get VF device for existing netvsc device\n  * Assumes vf_lock is held.\n@@ -210,19 +229,17 @@ hn_vf_attached(const struct hn_data *hv)\n static inline struct rte_eth_dev *\n hn_get_vf_dev(const struct hn_data *hv)\n {\n-\tuint16_t vf_port = hv->vf_port;\n-\n-\tif (vf_port == HN_INVALID_PORT)\n-\t\treturn NULL;\n+\tif (hv->vf_ctx.vf_attached)\n+\t\treturn &rte_eth_devices[hv->vf_ctx.vf_port];\n \telse\n-\t\treturn &rte_eth_devices[vf_port];\n+\t\treturn NULL;\n }\n \n int\thn_vf_info_get(struct hn_data *hv,\n \t\t       struct rte_eth_dev_info *info);\n int\thn_vf_add(struct rte_eth_dev *dev, struct hn_data *hv);\n-int\thn_vf_configure(struct rte_eth_dev *dev,\n-\t\t\tconst struct rte_eth_conf *dev_conf);\n+int\thn_vf_configure_locked(struct rte_eth_dev *dev,\n+\t\t\t       const struct rte_eth_conf *dev_conf);\n const uint32_t *hn_vf_supported_ptypes(struct rte_eth_dev *dev);\n int\thn_vf_start(struct rte_eth_dev *dev);\n void\thn_vf_reset(struct rte_eth_dev *dev);\n@@ -265,3 +282,6 @@ int\thn_vf_rss_hash_update(struct rte_eth_dev *dev,\n int\thn_vf_reta_hash_update(struct rte_eth_dev *dev,\n \t\t\t       struct rte_eth_rss_reta_entry64 *reta_conf,\n \t\t\t       uint16_t reta_size);\n+int\thn_eth_rmv_event_callback(uint16_t port_id,\n+\t\t\t\t  enum rte_eth_event_type event __rte_unused,\n+\t\t\t\t  void *cb_arg, void *out __rte_unused);\ndiff --git a/drivers/net/netvsc/hn_vf.c b/drivers/net/netvsc/hn_vf.c\nindex d43ebaa69f..86392917c5 100644\n--- a/drivers/net/netvsc/hn_vf.c\n+++ b/drivers/net/netvsc/hn_vf.c\n@@ -24,6 +24,7 @@\n #include <rte_bus_pci.h>\n #include <rte_log.h>\n #include <rte_string_fns.h>\n+#include <rte_alarm.h>\n \n #include \"hn_logs.h\"\n #include \"hn_var.h\"\n@@ -52,73 +53,252 @@ static int hn_vf_match(const struct rte_eth_dev *dev)\n /*\n  * Attach new PCI VF device and return the port_id\n  */\n-static int hn_vf_attach(struct hn_data *hv, uint16_t port_id)\n+static int hn_vf_attach(struct rte_eth_dev *dev, struct hn_data *hv)\n {\n \tstruct rte_eth_dev_owner owner = { .id = RTE_ETH_DEV_NO_OWNER };\n-\tint ret;\n+\tint port, ret;\n \n-\tif (hn_vf_attached(hv)) {\n+\tif (hv->vf_ctx.vf_attached) {\n \t\tPMD_DRV_LOG(ERR, \"VF already attached\");\n-\t\treturn -EEXIST;\n+\t\treturn 0;\n \t}\n \n-\tret = rte_eth_dev_owner_get(port_id, &owner);\n+\tport = hn_vf_match(dev);\n+\tif (port < 0) {\n+\t\tPMD_DRV_LOG(NOTICE, \"Couldn't find port for VF\");\n+\t\treturn port;\n+\t}\n+\n+\tPMD_DRV_LOG(NOTICE, \"found matching VF port %d\\n\", port);\n+\tret = rte_eth_dev_owner_get(port, &owner);\n \tif (ret < 0) {\n-\t\tPMD_DRV_LOG(ERR, \"Can not find owner for port %d\", port_id);\n+\t\tPMD_DRV_LOG(ERR, \"Can not find owner for port %d\", port);\n \t\treturn ret;\n \t}\n \n \tif (owner.id != RTE_ETH_DEV_NO_OWNER) {\n \t\tPMD_DRV_LOG(ERR, \"Port %u already owned by other device %s\",\n-\t\t\t    port_id, owner.name);\n+\t\t\t    port, owner.name);\n \t\treturn -EBUSY;\n \t}\n \n-\tret = rte_eth_dev_owner_set(port_id, &hv->owner);\n+\tret = rte_eth_dev_owner_set(port, &hv->owner);\n \tif (ret < 0) {\n-\t\tPMD_DRV_LOG(ERR, \"Can set owner for port %d\", port_id);\n+\t\tPMD_DRV_LOG(ERR, \"Can set owner for port %d\", port);\n \t\treturn ret;\n \t}\n \n-\tPMD_DRV_LOG(DEBUG, \"Attach VF device %u\", port_id);\n-\thv->vf_port = port_id;\n+\tPMD_DRV_LOG(DEBUG, \"Attach VF device %u\", port);\n+\thv->vf_ctx.vf_attached = true;\n+\thv->vf_ctx.vf_port = port;\n+\treturn 0;\n+}\n+\n+static void hn_vf_remove(struct hn_data *hv);\n+\n+static void hn_remove_delayed(void *args)\n+{\n+\tstruct hn_data *hv = args;\n+\tuint16_t port_id = hv->vf_ctx.vf_port;\n+\tstruct rte_device *dev = rte_eth_devices[port_id].device;\n+\tint ret;\n+\n+\t/* Tell VSP to switch data path to synthentic */\n+\thn_vf_remove(hv);\n+\n+\tPMD_DRV_LOG(NOTICE, \"Start to remove port %d\\n\", port_id);\n+\trte_rwlock_write_lock(&hv->vf_lock);\n+\n+\t/* Give back ownership */\n+\tret = rte_eth_dev_owner_unset(port_id, hv->owner.id);\n+\tif (ret)\n+\t\tPMD_DRV_LOG(ERR, \"rte_eth_dev_owner_unset failed ret=%d\\n\",\n+\t\t\t    ret);\n+\thv->vf_ctx.vf_attached = false;\n+\n+\tret = rte_eth_dev_callback_unregister(port_id, RTE_ETH_EVENT_INTR_RMV,\n+\t\t\t\t\t      hn_eth_rmv_event_callback, hv);\n+\tif (ret)\n+\t\tPMD_DRV_LOG(ERR,\n+\t\t\t    \"rte_eth_dev_callback_unregister failed ret=%d\\n\",\n+\t\t\t    ret);\n+\n+\t/* Detach and release port_id from system */\n+\tret = rte_eth_dev_stop(port_id);\n+\tif (ret)\n+\t\tPMD_DRV_LOG(ERR, \"rte_eth_dev_stop failed port_id=%u ret=%d\\n\",\n+\t\t\t    port_id, ret);\n+\n+\tret = rte_eth_dev_close(port_id);\n+\tif (ret)\n+\t\tPMD_DRV_LOG(ERR, \"rte_eth_dev_close failed port_id=%u ret=%d\\n\",\n+\t\t\t    port_id, ret);\n+\n+\tret = rte_dev_remove(dev);\n+\thv->vf_ctx.vf_state = vf_removed;\n+\n+\trte_rwlock_write_unlock(&hv->vf_lock);\n+}\n+\n+int hn_eth_rmv_event_callback(uint16_t port_id,\n+\t\t\t      enum rte_eth_event_type event __rte_unused,\n+\t\t\t      void *cb_arg, void *out __rte_unused)\n+{\n+\tstruct hn_data *hv = cb_arg;\n+\n+\tPMD_DRV_LOG(NOTICE, \"Removing VF portid %d\\n\", port_id);\n+\trte_eal_alarm_set(1, hn_remove_delayed, hv);\n+\n \treturn 0;\n }\n \n+static int hn_setup_vf_queues(int port, struct rte_eth_dev *dev)\n+{\n+\tstruct hn_rx_queue *rx_queue;\n+\tstruct rte_eth_txq_info txinfo;\n+\tstruct rte_eth_rxq_info rxinfo;\n+\tint i, ret = 0;\n+\n+\tfor (i = 0; i < dev->data->nb_tx_queues; i++) {\n+\t\tret = rte_eth_tx_queue_info_get(dev->data->port_id, i, &txinfo);\n+\t\tif (ret) {\n+\t\t\tPMD_DRV_LOG(ERR,\n+\t\t\t\t    \"rte_eth_tx_queue_info_get failed ret=%d\\n\",\n+\t\t\t\t    ret);\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\tret = rte_eth_tx_queue_setup(port, i, txinfo.nb_desc, 0,\n+\t\t\t\t\t     &txinfo.conf);\n+\t\tif (ret) {\n+\t\t\tPMD_DRV_LOG(ERR,\n+\t\t\t\t    \"rte_eth_tx_queue_setup failed ret=%d\\n\",\n+\t\t\t\t    ret);\n+\t\t\treturn ret;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < dev->data->nb_rx_queues; i++) {\n+\t\tret = rte_eth_rx_queue_info_get(dev->data->port_id, i, &rxinfo);\n+\t\tif (ret) {\n+\t\t\tPMD_DRV_LOG(ERR,\n+\t\t\t\t    \"rte_eth_rx_queue_info_get failed ret=%d\\n\",\n+\t\t\t\t    ret);\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\trx_queue = dev->data->rx_queues[i];\n+\n+\t\tret = rte_eth_rx_queue_setup(port, i, rxinfo.nb_desc, 0,\n+\t\t\t\t\t     &rxinfo.conf, rx_queue->mb_pool);\n+\t\tif (ret) {\n+\t\t\tPMD_DRV_LOG(ERR,\n+\t\t\t\t    \"rte_eth_rx_queue_setup failed ret=%d\\n\",\n+\t\t\t\t    ret);\n+\t\t\treturn ret;\n+\t\t}\n+\t}\n+\n+\treturn ret;\n+}\n+\n+int hn_vf_add(struct rte_eth_dev *dev, struct hn_data *hv);\n+\n+static void hn_vf_add_retry(void *args)\n+{\n+\tstruct rte_eth_dev *dev = args;\n+\tstruct hn_data *hv = dev->data->dev_private;\n+\n+\thn_vf_add(dev, hv);\n+}\n+\n+int hn_vf_configure(struct rte_eth_dev *dev,\n+\t\t    const struct rte_eth_conf *dev_conf);\n+\n /* Add new VF device to synthetic device */\n int hn_vf_add(struct rte_eth_dev *dev, struct hn_data *hv)\n {\n-\tint port, err;\n+\tint ret, port;\n \n-\tport = hn_vf_match(dev);\n-\tif (port < 0) {\n-\t\tPMD_DRV_LOG(NOTICE, \"No matching MAC found\");\n-\t\treturn port;\n+\tif (!hv->vf_ctx.vf_vsp_reported || hv->vf_ctx.vf_vsc_switched)\n+\t\treturn 0;\n+\n+\trte_rwlock_write_lock(&hv->vf_lock);\n+\n+\tret = hn_vf_attach(dev, hv);\n+\tif (ret) {\n+\t\tPMD_DRV_LOG(NOTICE,\n+\t\t\t    \"RNDIS reports VF but device not found, retrying\");\n+\t\trte_eal_alarm_set(1000000, hn_vf_add_retry, dev);\n+\t\tgoto exit;\n \t}\n \n-\terr = hn_vf_attach(hv, port);\n-\tif (err == 0)\n-\t\thn_nvs_set_datapath(hv, NVS_DATAPATH_VF);\n+\tport = hv->vf_ctx.vf_port;\n \n-\treturn err;\n+\t/* If the primary device has started, this is a VF host add.\n+\t * Configure and start VF device.\n+\t */\n+\tif (dev->data->dev_started) {\n+\t\tif (rte_eth_devices[port].data->dev_started) {\n+\t\t\tPMD_DRV_LOG(ERR, \"VF already started on hot add\");\n+\t\t\tgoto exit;\n+\t\t}\n+\n+\t\tPMD_DRV_LOG(NOTICE, \"configuring VF port %d\\n\", port);\n+\t\tret = hn_vf_configure(dev, &dev->data->dev_conf);\n+\t\tif (ret) {\n+\t\t\tPMD_DRV_LOG(ERR, \"Failed to configure VF port %d\\n\",\n+\t\t\t\t    port);\n+\t\t\tgoto exit;\n+\t\t}\n+\n+\t\tret = hn_setup_vf_queues(port, dev);\n+\t\tif (ret) {\n+\t\t\tPMD_DRV_LOG(ERR,\n+\t\t\t\t    \"Failed to configure VF queues port %d\\n\",\n+\t\t\t\t    port);\n+\t\t\tgoto exit;\n+\t\t}\n+\n+\t\tPMD_DRV_LOG(NOTICE, \"Starting VF port %d\\n\", port);\n+\t\tret = rte_eth_dev_start(port);\n+\t\tif (ret) {\n+\t\t\tPMD_DRV_LOG(ERR, \"rte_eth_dev_start failed ret=%d\\n\",\n+\t\t\t\t    ret);\n+\t\t\tgoto exit;\n+\t\t}\n+\t\thv->vf_ctx.vf_state = vf_started;\n+\t}\n+\n+\tret = hn_nvs_set_datapath(hv, NVS_DATAPATH_VF);\n+\tif (ret == 0)\n+\t\thv->vf_ctx.vf_vsc_switched = true;\n+\n+exit:\n+\trte_rwlock_write_unlock(&hv->vf_lock);\n+\treturn ret;\n }\n \n-/* Remove new VF device */\n+/* Switch data path to VF device */\n static void hn_vf_remove(struct hn_data *hv)\n {\n+\tint ret;\n \n-\tif (!hn_vf_attached(hv)) {\n+\tif (!hv->vf_ctx.vf_vsc_switched) {\n+\t\tPMD_DRV_LOG(ERR, \"VF path not active\");\n+\t\treturn;\n+\t}\n+\n+\trte_rwlock_write_lock(&hv->vf_lock);\n+\tif (!hv->vf_ctx.vf_vsc_switched) {\n \t\tPMD_DRV_LOG(ERR, \"VF path not active\");\n \t} else {\n \t\t/* Stop incoming packets from arriving on VF */\n-\t\thn_nvs_set_datapath(hv, NVS_DATAPATH_SYNTHETIC);\n-\n-\t\t/* Give back ownership */\n-\t\trte_eth_dev_owner_unset(hv->vf_port, hv->owner.id);\n-\n-\t\t/* Stop transmission over VF */\n-\t\thv->vf_port = HN_INVALID_PORT;\n+\t\tret = hn_nvs_set_datapath(hv, NVS_DATAPATH_SYNTHETIC);\n+\t\tif (ret == 0)\n+\t\t\thv->vf_ctx.vf_vsc_switched = false;\n \t}\n+\trte_rwlock_write_unlock(&hv->vf_lock);\n }\n \n /* Handle VF association message from host */\n@@ -140,8 +320,7 @@ hn_nvs_handle_vfassoc(struct rte_eth_dev *dev,\n \t\t    vf_assoc->allocated ? \"add to\" : \"remove from\",\n \t\t    dev->data->port_id);\n \n-\trte_rwlock_write_lock(&hv->vf_lock);\n-\thv->vf_present = vf_assoc->allocated;\n+\thv->vf_ctx.vf_vsp_reported = vf_assoc->allocated;\n \n \tif (dev->state == RTE_ETH_DEV_ATTACHED) {\n \t\tif (vf_assoc->allocated)\n@@ -149,7 +328,6 @@ hn_nvs_handle_vfassoc(struct rte_eth_dev *dev,\n \t\telse\n \t\t\thn_vf_remove(hv);\n \t}\n-\trte_rwlock_write_unlock(&hv->vf_lock);\n }\n \n static void\n@@ -216,10 +394,6 @@ int hn_vf_info_get(struct hn_data *hv, struct rte_eth_dev_info *info)\n \treturn ret;\n }\n \n-/*\n- * Configure VF if present.\n- * Force VF to have same number of queues as synthetic device\n- */\n int hn_vf_configure(struct rte_eth_dev *dev,\n \t\t    const struct rte_eth_conf *dev_conf)\n {\n@@ -230,17 +404,56 @@ int hn_vf_configure(struct rte_eth_dev *dev,\n \t/* link state interrupt does not matter here. */\n \tvf_conf.intr_conf.lsc = 0;\n \n-\trte_rwlock_read_lock(&hv->vf_lock);\n-\tif (hv->vf_port != HN_INVALID_PORT) {\n-\t\tret = rte_eth_dev_configure(hv->vf_port,\n+\t/* need to monitor removal event */\n+\tvf_conf.intr_conf.rmv = 1;\n+\n+\tif (hv->vf_ctx.vf_attached) {\n+\t\tret = rte_eth_dev_callback_register(hv->vf_ctx.vf_port,\n+\t\t\t\t\t\t    RTE_ETH_EVENT_INTR_RMV,\n+\t\t\t\t\t\t    hn_eth_rmv_event_callback,\n+\t\t\t\t\t\t    hv);\n+\t\tif (ret) {\n+\t\t\tPMD_DRV_LOG(ERR,\n+\t\t\t\t    \"Registering callback failed for \"\n+\t\t\t\t    \"vf port %d ret %d\\n\",\n+\t\t\t\t    hv->vf_ctx.vf_port, ret);\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\tret = rte_eth_dev_configure(hv->vf_ctx.vf_port,\n \t\t\t\t\t    dev->data->nb_rx_queues,\n \t\t\t\t\t    dev->data->nb_tx_queues,\n \t\t\t\t\t    &vf_conf);\n-\t\tif (ret != 0)\n-\t\t\tPMD_DRV_LOG(ERR,\n-\t\t\t\t    \"VF configuration failed: %d\", ret);\n+\t\tif (ret) {\n+\t\t\tPMD_DRV_LOG(ERR, \"VF configuration failed: %d\", ret);\n+\n+\t\t\trte_eth_dev_callback_unregister(hv->vf_ctx.vf_port,\n+\t\t\t\t\t\t\tRTE_ETH_EVENT_INTR_RMV,\n+\t\t\t\t\t\t\thn_eth_rmv_event_callback,\n+\t\t\t\t\t\t\thv);\n+\n+\t\t\treturn ret;\n+\t\t}\n+\n+\t\thv->vf_ctx.vf_state = vf_configured;\n \t}\n-\trte_rwlock_read_unlock(&hv->vf_lock);\n+\n+\treturn ret;\n+}\n+\n+/* Configure VF if present.\n+ * VF device will have the same number of queues as the synthetic device\n+ */\n+int hn_vf_configure_locked(struct rte_eth_dev *dev,\n+\t\t\t   const struct rte_eth_conf *dev_conf)\n+{\n+\tstruct hn_data *hv = dev->data->dev_private;\n+\tint ret = 0;\n+\n+\trte_rwlock_write_lock(&hv->vf_lock);\n+\tret = hn_vf_configure(dev, dev_conf);\n+\trte_rwlock_write_unlock(&hv->vf_lock);\n+\n \treturn ret;\n }\n \n@@ -325,16 +538,21 @@ void hn_vf_reset(struct rte_eth_dev *dev)\n \n int hn_vf_close(struct rte_eth_dev *dev)\n {\n-\tstruct hn_data *hv = dev->data->dev_private;\n-\tuint16_t vf_port;\n \tint ret = 0;\n+\tstruct hn_data *hv = dev->data->dev_private;\n \n-\trte_rwlock_read_lock(&hv->vf_lock);\n-\tvf_port = hv->vf_port;\n-\tif (vf_port != HN_INVALID_PORT)\n-\t\tret = rte_eth_dev_close(vf_port);\n+\trte_eal_alarm_cancel(hn_vf_add_retry, dev);\n \n-\thv->vf_port = HN_INVALID_PORT;\n+\trte_rwlock_read_lock(&hv->vf_lock);\n+\tif (hv->vf_ctx.vf_attached) {\n+\t\trte_eth_dev_callback_unregister(hv->vf_ctx.vf_port,\n+\t\t\t\t\t\tRTE_ETH_EVENT_INTR_RMV,\n+\t\t\t\t\t\thn_eth_rmv_event_callback,\n+\t\t\t\t\t\thv);\n+\t\trte_eal_alarm_cancel(hn_remove_delayed, hv);\n+\t\tret = rte_eth_dev_close(hv->vf_ctx.vf_port);\n+\t\thv->vf_ctx.vf_attached = false;\n+\t}\n \trte_rwlock_read_unlock(&hv->vf_lock);\n \n \treturn ret;\n",
    "prefixes": [
        "v2",
        "2/2"
    ]
}