[v1,0/5] Enable hotplug in vfio
mbox series

Message ID 1534503091-31910-1-git-send-email-jia.guo@intel.com
Headers show
Series
  • Enable hotplug in vfio
Related show

Message

Guo, Jia Aug. 17, 2018, 10:51 a.m. UTC
As we may know that the process of hotplug is different between igb_uio
and vfio. For igb_uio, it could use uevent notification and memory
failure handle mechanism for hotplug. But for vfio, when device is be
hotplug-out, the uevent can not be detected immediately, because of the
vfio kernel module will use a special mechanism to guaranty the pci
device would not be deleted until the user space release the resources,
so it will use another event “req notifier” at first to notify user space
to release resources for hotplug.

This patch will add a new interrupt type of req notifier in eal interrupt,
and add the new interrupt handler in pci device to handle the req device
event. When the req notifier be detected, it can trigger the device event
callback process to process for hotplug. With this mechanism, hotplug
could be enable in vfio.

Jeff Guo (5):
  eal: add a new req notifier to eal interrupt
  eal: add a new req event to device event
  eal: modify device event callback process func
  pci: add req handler field to generic pci device
  vfio: enable vfio hotplug by req notifier handler

 drivers/bus/pci/linux/pci_vfio.c                   | 104 +++++++++++++++++++++
 drivers/bus/pci/pci_common.c                       |  10 ++
 drivers/bus/pci/rte_bus_pci.h                      |   1 +
 lib/librte_eal/common/eal_common_dev.c             |   5 +-
 lib/librte_eal/common/eal_private.h                |  12 ---
 lib/librte_eal/common/include/rte_dev.h            |  20 +++-
 lib/librte_eal/common/include/rte_eal_interrupts.h |   1 +
 lib/librte_eal/linuxapp/eal/eal_dev.c              |   2 +-
 lib/librte_eal/linuxapp/eal/eal_interrupts.c       |  71 ++++++++++++++
 lib/librte_ethdev/rte_ethdev.c                     |   3 +-
 10 files changed, 212 insertions(+), 17 deletions(-)

Comments

Gaëtan Rivet Aug. 20, 2018, 9:15 a.m. UTC | #1
Hi Jeff,

On Fri, Aug 17, 2018 at 06:51:26PM +0800, Jeff Guo wrote:
> As we may know that the process of hotplug is different between igb_uio
> and vfio. For igb_uio, it could use uevent notification and memory
> failure handle mechanism for hotplug. But for vfio, when device is be
> hotplug-out, the uevent can not be detected immediately, because of the
> vfio kernel module will use a special mechanism to guaranty the pci
> device would not be deleted until the user space release the resources,
> so it will use another event “req notifier” at first to notify user space
> to release resources for hotplug.
> 
> This patch will add a new interrupt type of req notifier in eal interrupt,
> and add the new interrupt handler in pci device to handle the req device
> event. When the req notifier be detected, it can trigger the device event
> callback process to process for hotplug. With this mechanism, hotplug
> could be enable in vfio.
> 

The REQ event seems VFIO specific.
Why not leave it within the PCI bus, around the vfio glue, and leave the
EAL unmodified?

> Jeff Guo (5):
>   eal: add a new req notifier to eal interrupt
>   eal: add a new req event to device event
>   eal: modify device event callback process func
>   pci: add req handler field to generic pci device
>   vfio: enable vfio hotplug by req notifier handler
> 
>  drivers/bus/pci/linux/pci_vfio.c                   | 104 +++++++++++++++++++++
>  drivers/bus/pci/pci_common.c                       |  10 ++
>  drivers/bus/pci/rte_bus_pci.h                      |   1 +
>  lib/librte_eal/common/eal_common_dev.c             |   5 +-
>  lib/librte_eal/common/eal_private.h                |  12 ---
>  lib/librte_eal/common/include/rte_dev.h            |  20 +++-
>  lib/librte_eal/common/include/rte_eal_interrupts.h |   1 +
>  lib/librte_eal/linuxapp/eal/eal_dev.c              |   2 +-
>  lib/librte_eal/linuxapp/eal/eal_interrupts.c       |  71 ++++++++++++++
>  lib/librte_ethdev/rte_ethdev.c                     |   3 +-
>  10 files changed, 212 insertions(+), 17 deletions(-)
> 
> -- 
> 2.7.4
>
Guo, Jia Aug. 21, 2018, 6:45 a.m. UTC | #2
hi, gaetan


On 8/20/2018 5:15 PM, Gaëtan Rivet wrote:
> Hi Jeff,
>
> On Fri, Aug 17, 2018 at 06:51:26PM +0800, Jeff Guo wrote:
>> As we may know that the process of hotplug is different between igb_uio
>> and vfio. For igb_uio, it could use uevent notification and memory
>> failure handle mechanism for hotplug. But for vfio, when device is be
>> hotplug-out, the uevent can not be detected immediately, because of the
>> vfio kernel module will use a special mechanism to guaranty the pci
>> device would not be deleted until the user space release the resources,
>> so it will use another event “req notifier” at first to notify user space
>> to release resources for hotplug.
>>
>> This patch will add a new interrupt type of req notifier in eal interrupt,
>> and add the new interrupt handler in pci device to handle the req device
>> event. When the req notifier be detected, it can trigger the device event
>> callback process to process for hotplug. With this mechanism, hotplug
>> could be enable in vfio.
>>
> The REQ event seems VFIO specific.
> Why not leave it within the PCI bus, around the vfio glue, and leave the
> EAL unmodified?

Sorry i don' t see if there are any problem. Firstly, as we can see the 
eal interrupt type, it cover the ext/uio/vfio/dev event types, which are 
not general for all platform/bus/driver type.
so i think base on the current framework, the interrupt type structure 
should be considerate as combined set, it could extend for other adding 
interrupts case by case. And secondly, i don't know what
is your way about leave it within the PCI bus, because i need to use the 
eal interrupt epoll to process this req interrupt which is familiar with 
other interrupts. What do you think about that, if you still
have better suggestion, please detail it for clarify.

>> Jeff Guo (5):
>>    eal: add a new req notifier to eal interrupt
>>    eal: add a new req event to device event
>>    eal: modify device event callback process func
>>    pci: add req handler field to generic pci device
>>    vfio: enable vfio hotplug by req notifier handler
>>
>>   drivers/bus/pci/linux/pci_vfio.c                   | 104 +++++++++++++++++++++
>>   drivers/bus/pci/pci_common.c                       |  10 ++
>>   drivers/bus/pci/rte_bus_pci.h                      |   1 +
>>   lib/librte_eal/common/eal_common_dev.c             |   5 +-
>>   lib/librte_eal/common/eal_private.h                |  12 ---
>>   lib/librte_eal/common/include/rte_dev.h            |  20 +++-
>>   lib/librte_eal/common/include/rte_eal_interrupts.h |   1 +
>>   lib/librte_eal/linuxapp/eal/eal_dev.c              |   2 +-
>>   lib/librte_eal/linuxapp/eal/eal_interrupts.c       |  71 ++++++++++++++
>>   lib/librte_ethdev/rte_ethdev.c                     |   3 +-
>>   10 files changed, 212 insertions(+), 17 deletions(-)
>>
>> -- 
>> 2.7.4
>>
Gaëtan Rivet Aug. 21, 2018, 8:17 a.m. UTC | #3
On Tue, Aug 21, 2018 at 02:45:32PM +0800, Jeff Guo wrote:
> hi, gaetan
> 
> 
> On 8/20/2018 5:15 PM, Gaëtan Rivet wrote:
> > Hi Jeff,
> > 
> > On Fri, Aug 17, 2018 at 06:51:26PM +0800, Jeff Guo wrote:
> > > As we may know that the process of hotplug is different between igb_uio
> > > and vfio. For igb_uio, it could use uevent notification and memory
> > > failure handle mechanism for hotplug. But for vfio, when device is be
> > > hotplug-out, the uevent can not be detected immediately, because of the
> > > vfio kernel module will use a special mechanism to guaranty the pci
> > > device would not be deleted until the user space release the resources,
> > > so it will use another event “req notifier” at first to notify user space
> > > to release resources for hotplug.
> > > 
> > > This patch will add a new interrupt type of req notifier in eal interrupt,
> > > and add the new interrupt handler in pci device to handle the req device
> > > event. When the req notifier be detected, it can trigger the device event
> > > callback process to process for hotplug. With this mechanism, hotplug
> > > could be enable in vfio.
> > > 
> > The REQ event seems VFIO specific.
> > Why not leave it within the PCI bus, around the vfio glue, and leave the
> > EAL unmodified?
> 
> Sorry i don' t see if there are any problem. Firstly, as we can see the eal
> interrupt type, it cover the ext/uio/vfio/dev event types, which are not
> general for all platform/bus/driver type.
> so i think base on the current framework, the interrupt type structure
> should be considerate as combined set, it could extend for other adding
> interrupts case by case. And secondly, i don't know what
> is your way about leave it within the PCI bus, because i need to use the eal
> interrupt epoll to process this req interrupt which is familiar with other
> interrupts. What do you think about that, if you still
> have better suggestion, please detail it for clarify.
> 

Ah, yes, unfortunately you're right.
There is a genericization effort going on for the EAL, but these parts
are still depending on the interrupts being reworked. So for now, not
much can be done to avoid it.

> > > Jeff Guo (5):
> > >    eal: add a new req notifier to eal interrupt
> > >    eal: add a new req event to device event
> > >    eal: modify device event callback process func
> > >    pci: add req handler field to generic pci device
> > >    vfio: enable vfio hotplug by req notifier handler
> > > 
> > >   drivers/bus/pci/linux/pci_vfio.c                   | 104 +++++++++++++++++++++
> > >   drivers/bus/pci/pci_common.c                       |  10 ++
> > >   drivers/bus/pci/rte_bus_pci.h                      |   1 +
> > >   lib/librte_eal/common/eal_common_dev.c             |   5 +-
> > >   lib/librte_eal/common/eal_private.h                |  12 ---
> > >   lib/librte_eal/common/include/rte_dev.h            |  20 +++-
> > >   lib/librte_eal/common/include/rte_eal_interrupts.h |   1 +
> > >   lib/librte_eal/linuxapp/eal/eal_dev.c              |   2 +-
> > >   lib/librte_eal/linuxapp/eal/eal_interrupts.c       |  71 ++++++++++++++
> > >   lib/librte_ethdev/rte_ethdev.c                     |   3 +-
> > >   10 files changed, 212 insertions(+), 17 deletions(-)
> > > 
> > > -- 
> > > 2.7.4
> > > 
>
Guo, Jia Oct. 2, 2018, 12:42 p.m. UTC | #4
As we may know that the process of hotplug is different between igb_uio
and vfio. For igb_uio, it could use uevent notification and memory
failure handle mechanism for hot-unplug. But for vfio, when device is be
hot-unplugged, the uevent can not be detected immediately, because of the
vfio kernel module will use a special mechanism to guaranty the pci
device would not be deleted until the user space release the resources,
so it will use another event “req notifier” at first to notify user space
to release resources for hotplug.

This patch will add a new interrupt type of req notifier in eal interrupt,
and add the new interrupt handler in pci device to handle the req device
event. When the req notifier be detected, it can trigger the device event
callback process to process for hot-unplug. With this mechanism, hotplug
could be enable in vfio.

patchset history:
v3->v2:
change some commit log and coding style and typo.

v2->v1:
change the rte_dev_event_callback_prcess from internal to external api
for bus or app usage.
change some code logic.

Jeff Guo (4):
  eal: add a new req notifier to eal interrupt
  eal: modify device event callback process func
  pci: add req handler field to generic pci device
  vfio: enable vfio hotplug by req notifier handler

 app/test-pmd/testpmd.c                             |   4 +-
 drivers/bus/pci/linux/pci_vfio.c                   | 111 +++++++++++++++++++++
 drivers/bus/pci/pci_common.c                       |  10 ++
 drivers/bus/pci/rte_bus_pci.h                      |   1 +
 lib/librte_eal/bsdapp/eal/eal_dev.c                |   8 ++
 lib/librte_eal/common/eal_common_dev.c             |   5 +-
 lib/librte_eal/common/eal_private.h                |  12 ---
 lib/librte_eal/common/include/rte_dev.h            |  18 +++-
 lib/librte_eal/common/include/rte_eal_interrupts.h |   1 +
 lib/librte_eal/linuxapp/eal/eal_dev.c              |   2 +-
 lib/librte_eal/linuxapp/eal/eal_interrupts.c       |  71 +++++++++++++
 lib/librte_eal/rte_eal_version.map                 |   1 +
 12 files changed, 226 insertions(+), 18 deletions(-)
Guo, Jia Oct. 2, 2018, 12:58 p.m. UTC | #5
As we may know that the process of hotplug is different between igb_uio
and vfio. For igb_uio, it could use uevent notification and memory
failure handle mechanism for hot-unplug. But for vfio, when device is be
hot-unplugged, the uevent can not be detected immediately, because of the
vfio kernel module will use a special mechanism to guaranty the pci
device would not be deleted until the user space release the resources,
so it will use another event “req notifier” at first to notify user space
to release resources for hotplug.

This patch will add a new interrupt type of req notifier in eal interrupt,
and add the new interrupt handler in pci device to handle the req device
event. When the req notifier be detected, it can trigger the device event
callback process to process for hot-unplug. With this mechanism, hotplug
could be enable in vfio.

patchset history:
v3->v2:
change some commit log and coding style and typo.

v2->v1:
change the rte_dev_event_callback_prcess from internal to external api
for bus or app usage.
change some code logic.

Jeff Guo (4):
  eal: add a new req notifier to eal interrupt
  eal: modify device event callback process func
  pci: add req handler field to generic pci device
  vfio: enable vfio hotplug by req notifier handler

 app/test-pmd/testpmd.c                             |   4 +-
 drivers/bus/pci/linux/pci_vfio.c                   | 111 +++++++++++++++++++++
 drivers/bus/pci/pci_common.c                       |  10 ++
 drivers/bus/pci/rte_bus_pci.h                      |   1 +
 lib/librte_eal/bsdapp/eal/eal_dev.c                |   8 ++
 lib/librte_eal/common/eal_common_dev.c             |   5 +-
 lib/librte_eal/common/eal_private.h                |  12 ---
 lib/librte_eal/common/include/rte_dev.h            |  18 +++-
 lib/librte_eal/common/include/rte_eal_interrupts.h |   1 +
 lib/librte_eal/linuxapp/eal/eal_dev.c              |   2 +-
 lib/librte_eal/linuxapp/eal/eal_interrupts.c       |  71 +++++++++++++
 lib/librte_eal/rte_eal_version.map                 |   1 +
 12 files changed, 226 insertions(+), 18 deletions(-)