| Message ID | 1534503091-31910-1-git-send-email-jia.guo@intel.com (mailing list archive) |
|---|---|
| Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0C1F210A3; Fri, 17 Aug 2018 12:54:17 +0200 (CEST) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id EEC7598 for <dev@dpdk.org>; Fri, 17 Aug 2018 12:54:14 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Aug 2018 03:54:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,250,1531810800"; d="scan'208";a="73536775" Received: from jeffguo-z170x-ud5.sh.intel.com (HELO localhost.localdomain) ([10.67.104.10]) by FMSMGA003.fm.intel.com with ESMTP; 17 Aug 2018 03:54:11 -0700 From: Jeff Guo <jia.guo@intel.com> To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com, bernard.iremonger@intel.com, arybchenko@solarflare.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Fri, 17 Aug 2018 18:51:26 +0800 Message-Id: <1534503091-31910-1-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v1 0/5] Enable hotplug in vfio X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
| Series |
Enable hotplug in vfio
|
|
Message
Guo, Jia
Aug. 17, 2018, 10:51 a.m. UTC
As we may know that the process of hotplug is different between igb_uio and vfio. For igb_uio, it could use uevent notification and memory failure handle mechanism for hotplug. But for vfio, when device is be hotplug-out, the uevent can not be detected immediately, because of the vfio kernel module will use a special mechanism to guaranty the pci device would not be deleted until the user space release the resources, so it will use another event “req notifier” at first to notify user space to release resources for hotplug. This patch will add a new interrupt type of req notifier in eal interrupt, and add the new interrupt handler in pci device to handle the req device event. When the req notifier be detected, it can trigger the device event callback process to process for hotplug. With this mechanism, hotplug could be enable in vfio. Jeff Guo (5): eal: add a new req notifier to eal interrupt eal: add a new req event to device event eal: modify device event callback process func pci: add req handler field to generic pci device vfio: enable vfio hotplug by req notifier handler drivers/bus/pci/linux/pci_vfio.c | 104 +++++++++++++++++++++ drivers/bus/pci/pci_common.c | 10 ++ drivers/bus/pci/rte_bus_pci.h | 1 + lib/librte_eal/common/eal_common_dev.c | 5 +- lib/librte_eal/common/eal_private.h | 12 --- lib/librte_eal/common/include/rte_dev.h | 20 +++- lib/librte_eal/common/include/rte_eal_interrupts.h | 1 + lib/librte_eal/linuxapp/eal/eal_dev.c | 2 +- lib/librte_eal/linuxapp/eal/eal_interrupts.c | 71 ++++++++++++++ lib/librte_ethdev/rte_ethdev.c | 3 +- 10 files changed, 212 insertions(+), 17 deletions(-)
Comments
Hi Jeff, On Fri, Aug 17, 2018 at 06:51:26PM +0800, Jeff Guo wrote: > As we may know that the process of hotplug is different between igb_uio > and vfio. For igb_uio, it could use uevent notification and memory > failure handle mechanism for hotplug. But for vfio, when device is be > hotplug-out, the uevent can not be detected immediately, because of the > vfio kernel module will use a special mechanism to guaranty the pci > device would not be deleted until the user space release the resources, > so it will use another event “req notifier” at first to notify user space > to release resources for hotplug. > > This patch will add a new interrupt type of req notifier in eal interrupt, > and add the new interrupt handler in pci device to handle the req device > event. When the req notifier be detected, it can trigger the device event > callback process to process for hotplug. With this mechanism, hotplug > could be enable in vfio. > The REQ event seems VFIO specific. Why not leave it within the PCI bus, around the vfio glue, and leave the EAL unmodified? > Jeff Guo (5): > eal: add a new req notifier to eal interrupt > eal: add a new req event to device event > eal: modify device event callback process func > pci: add req handler field to generic pci device > vfio: enable vfio hotplug by req notifier handler > > drivers/bus/pci/linux/pci_vfio.c | 104 +++++++++++++++++++++ > drivers/bus/pci/pci_common.c | 10 ++ > drivers/bus/pci/rte_bus_pci.h | 1 + > lib/librte_eal/common/eal_common_dev.c | 5 +- > lib/librte_eal/common/eal_private.h | 12 --- > lib/librte_eal/common/include/rte_dev.h | 20 +++- > lib/librte_eal/common/include/rte_eal_interrupts.h | 1 + > lib/librte_eal/linuxapp/eal/eal_dev.c | 2 +- > lib/librte_eal/linuxapp/eal/eal_interrupts.c | 71 ++++++++++++++ > lib/librte_ethdev/rte_ethdev.c | 3 +- > 10 files changed, 212 insertions(+), 17 deletions(-) > > -- > 2.7.4 >
hi, gaetan On 8/20/2018 5:15 PM, Gaëtan Rivet wrote: > Hi Jeff, > > On Fri, Aug 17, 2018 at 06:51:26PM +0800, Jeff Guo wrote: >> As we may know that the process of hotplug is different between igb_uio >> and vfio. For igb_uio, it could use uevent notification and memory >> failure handle mechanism for hotplug. But for vfio, when device is be >> hotplug-out, the uevent can not be detected immediately, because of the >> vfio kernel module will use a special mechanism to guaranty the pci >> device would not be deleted until the user space release the resources, >> so it will use another event “req notifier” at first to notify user space >> to release resources for hotplug. >> >> This patch will add a new interrupt type of req notifier in eal interrupt, >> and add the new interrupt handler in pci device to handle the req device >> event. When the req notifier be detected, it can trigger the device event >> callback process to process for hotplug. With this mechanism, hotplug >> could be enable in vfio. >> > The REQ event seems VFIO specific. > Why not leave it within the PCI bus, around the vfio glue, and leave the > EAL unmodified? Sorry i don' t see if there are any problem. Firstly, as we can see the eal interrupt type, it cover the ext/uio/vfio/dev event types, which are not general for all platform/bus/driver type. so i think base on the current framework, the interrupt type structure should be considerate as combined set, it could extend for other adding interrupts case by case. And secondly, i don't know what is your way about leave it within the PCI bus, because i need to use the eal interrupt epoll to process this req interrupt which is familiar with other interrupts. What do you think about that, if you still have better suggestion, please detail it for clarify. >> Jeff Guo (5): >> eal: add a new req notifier to eal interrupt >> eal: add a new req event to device event >> eal: modify device event callback process func >> pci: add req handler field to generic pci device >> vfio: enable vfio hotplug by req notifier handler >> >> drivers/bus/pci/linux/pci_vfio.c | 104 +++++++++++++++++++++ >> drivers/bus/pci/pci_common.c | 10 ++ >> drivers/bus/pci/rte_bus_pci.h | 1 + >> lib/librte_eal/common/eal_common_dev.c | 5 +- >> lib/librte_eal/common/eal_private.h | 12 --- >> lib/librte_eal/common/include/rte_dev.h | 20 +++- >> lib/librte_eal/common/include/rte_eal_interrupts.h | 1 + >> lib/librte_eal/linuxapp/eal/eal_dev.c | 2 +- >> lib/librte_eal/linuxapp/eal/eal_interrupts.c | 71 ++++++++++++++ >> lib/librte_ethdev/rte_ethdev.c | 3 +- >> 10 files changed, 212 insertions(+), 17 deletions(-) >> >> -- >> 2.7.4 >>
On Tue, Aug 21, 2018 at 02:45:32PM +0800, Jeff Guo wrote: > hi, gaetan > > > On 8/20/2018 5:15 PM, Gaëtan Rivet wrote: > > Hi Jeff, > > > > On Fri, Aug 17, 2018 at 06:51:26PM +0800, Jeff Guo wrote: > > > As we may know that the process of hotplug is different between igb_uio > > > and vfio. For igb_uio, it could use uevent notification and memory > > > failure handle mechanism for hotplug. But for vfio, when device is be > > > hotplug-out, the uevent can not be detected immediately, because of the > > > vfio kernel module will use a special mechanism to guaranty the pci > > > device would not be deleted until the user space release the resources, > > > so it will use another event “req notifier” at first to notify user space > > > to release resources for hotplug. > > > > > > This patch will add a new interrupt type of req notifier in eal interrupt, > > > and add the new interrupt handler in pci device to handle the req device > > > event. When the req notifier be detected, it can trigger the device event > > > callback process to process for hotplug. With this mechanism, hotplug > > > could be enable in vfio. > > > > > The REQ event seems VFIO specific. > > Why not leave it within the PCI bus, around the vfio glue, and leave the > > EAL unmodified? > > Sorry i don' t see if there are any problem. Firstly, as we can see the eal > interrupt type, it cover the ext/uio/vfio/dev event types, which are not > general for all platform/bus/driver type. > so i think base on the current framework, the interrupt type structure > should be considerate as combined set, it could extend for other adding > interrupts case by case. And secondly, i don't know what > is your way about leave it within the PCI bus, because i need to use the eal > interrupt epoll to process this req interrupt which is familiar with other > interrupts. What do you think about that, if you still > have better suggestion, please detail it for clarify. > Ah, yes, unfortunately you're right. There is a genericization effort going on for the EAL, but these parts are still depending on the interrupts being reworked. So for now, not much can be done to avoid it. > > > Jeff Guo (5): > > > eal: add a new req notifier to eal interrupt > > > eal: add a new req event to device event > > > eal: modify device event callback process func > > > pci: add req handler field to generic pci device > > > vfio: enable vfio hotplug by req notifier handler > > > > > > drivers/bus/pci/linux/pci_vfio.c | 104 +++++++++++++++++++++ > > > drivers/bus/pci/pci_common.c | 10 ++ > > > drivers/bus/pci/rte_bus_pci.h | 1 + > > > lib/librte_eal/common/eal_common_dev.c | 5 +- > > > lib/librte_eal/common/eal_private.h | 12 --- > > > lib/librte_eal/common/include/rte_dev.h | 20 +++- > > > lib/librte_eal/common/include/rte_eal_interrupts.h | 1 + > > > lib/librte_eal/linuxapp/eal/eal_dev.c | 2 +- > > > lib/librte_eal/linuxapp/eal/eal_interrupts.c | 71 ++++++++++++++ > > > lib/librte_ethdev/rte_ethdev.c | 3 +- > > > 10 files changed, 212 insertions(+), 17 deletions(-) > > > > > > -- > > > 2.7.4 > > > >
As we may know that the process of hotplug is different between igb_uio and vfio. For igb_uio, it could use uevent notification and memory failure handle mechanism for hot-unplug. But for vfio, when device is be hot-unplugged, the uevent can not be detected immediately, because of the vfio kernel module will use a special mechanism to guaranty the pci device would not be deleted until the user space release the resources, so it will use another event “req notifier” at first to notify user space to release resources for hotplug. This patch will add a new interrupt type of req notifier in eal interrupt, and add the new interrupt handler in pci device to handle the req device event. When the req notifier be detected, it can trigger the device event callback process to process for hot-unplug. With this mechanism, hotplug could be enable in vfio. patchset history: v3->v2: change some commit log and coding style and typo. v2->v1: change the rte_dev_event_callback_prcess from internal to external api for bus or app usage. change some code logic. Jeff Guo (4): eal: add a new req notifier to eal interrupt eal: modify device event callback process func pci: add req handler field to generic pci device vfio: enable vfio hotplug by req notifier handler app/test-pmd/testpmd.c | 4 +- drivers/bus/pci/linux/pci_vfio.c | 111 +++++++++++++++++++++ drivers/bus/pci/pci_common.c | 10 ++ drivers/bus/pci/rte_bus_pci.h | 1 + lib/librte_eal/bsdapp/eal/eal_dev.c | 8 ++ lib/librte_eal/common/eal_common_dev.c | 5 +- lib/librte_eal/common/eal_private.h | 12 --- lib/librte_eal/common/include/rte_dev.h | 18 +++- lib/librte_eal/common/include/rte_eal_interrupts.h | 1 + lib/librte_eal/linuxapp/eal/eal_dev.c | 2 +- lib/librte_eal/linuxapp/eal/eal_interrupts.c | 71 +++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 12 files changed, 226 insertions(+), 18 deletions(-)
As we may know that the process of hotplug is different between igb_uio and vfio. For igb_uio, it could use uevent notification and memory failure handle mechanism for hot-unplug. But for vfio, when device is be hot-unplugged, the uevent can not be detected immediately, because of the vfio kernel module will use a special mechanism to guaranty the pci device would not be deleted until the user space release the resources, so it will use another event “req notifier” at first to notify user space to release resources for hotplug. This patch will add a new interrupt type of req notifier in eal interrupt, and add the new interrupt handler in pci device to handle the req device event. When the req notifier be detected, it can trigger the device event callback process to process for hot-unplug. With this mechanism, hotplug could be enable in vfio. patchset history: v3->v2: change some commit log and coding style and typo. v2->v1: change the rte_dev_event_callback_prcess from internal to external api for bus or app usage. change some code logic. Jeff Guo (4): eal: add a new req notifier to eal interrupt eal: modify device event callback process func pci: add req handler field to generic pci device vfio: enable vfio hotplug by req notifier handler app/test-pmd/testpmd.c | 4 +- drivers/bus/pci/linux/pci_vfio.c | 111 +++++++++++++++++++++ drivers/bus/pci/pci_common.c | 10 ++ drivers/bus/pci/rte_bus_pci.h | 1 + lib/librte_eal/bsdapp/eal/eal_dev.c | 8 ++ lib/librte_eal/common/eal_common_dev.c | 5 +- lib/librte_eal/common/eal_private.h | 12 --- lib/librte_eal/common/include/rte_dev.h | 18 +++- lib/librte_eal/common/include/rte_eal_interrupts.h | 1 + lib/librte_eal/linuxapp/eal/eal_dev.c | 2 +- lib/librte_eal/linuxapp/eal/eal_interrupts.c | 71 +++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 12 files changed, 226 insertions(+), 18 deletions(-)