[v5,0/3] support both PIO and MMIO BAR for virtio PMD

  From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>

Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of
virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR.

Kernel supports both PIO  and MMIO BAR for legacy virtio-pci device. We handles
different type of BAR in the similar way.

In previous implementation, with igb_uio we get PIO address from igb_uio
sysfs entry; with uio_pci_generic, we get PIO address from
/proc/ioports.
For PIO/MMIO RW, there is different path for different drivers and arch.
For VFIO, PIO/MMIO RW is through syscall, which has big performance
issue.
On X86, it assumes only PIO is supported.

All of the above is too much twisted.
This patch unifies the way to get both PIO and MMIO address for different driver
and arch, all from standard resource attr under pci sysfs.

We distinguish PIO and MMIO by their address like how kernel does. It is ugly but works.

v2 changes:
	 - add more explanation in the commit message

v3 changes:
	 - fix patch format issues

v4 changes:
	 - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC

v5 changes:
	 - split into three seperate patches

huawei.xhw (3):
  PCI: use PCI standard sysfs entry to get PIO address
  PCI: support MMIO in rte_pci_ioport_map/unap/read/write
  PCI: don't use vfio ioctl call to access PIO resource

 drivers/bus/pci/linux/pci.c     |  89 +-------------------
 drivers/bus/pci/linux/pci_uio.c | 177 ++++++++++++++++++++++++++++------------
 2 files changed, 128 insertions(+), 138 deletions(-)
  

@Ferruh: this patch is tested with both PIO and MMIO bar using testpmd 
and start tx_first.

vfio/igb_uio tested with MMIO bar (uio_pci_generic doesn't work with 
msix, so it isn't tested)

uio_pci_generic tested with PIO bar (igb_uio has unknown symbols, not 
tested).

Weird igb_uio doens't have Makefile.


On 2020/10/22 23:51, 谢华伟(此时此刻） wrote:
> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>
> Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of
> virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR.
>
> Kernel supports both PIO  and MMIO BAR for legacy virtio-pci device. We handles
> different type of BAR in the similar way.
>
> In previous implementation, with igb_uio we get PIO address from igb_uio
> sysfs entry; with uio_pci_generic, we get PIO address from
> /proc/ioports.
> For PIO/MMIO RW, there is different path for different drivers and arch.
> For VFIO, PIO/MMIO RW is through syscall, which has big performance
> issue.
> On X86, it assumes only PIO is supported.
>
> All of the above is too much twisted.
> This patch unifies the way to get both PIO and MMIO address for different driver
> and arch, all from standard resource attr under pci sysfs.
>
> We distinguish PIO and MMIO by their address like how kernel does. It is ugly but works.
>
> v2 changes:
> 	 - add more explanation in the commit message
>
> v3 changes:
> 	 - fix patch format issues
>
> v4 changes:
> 	 - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC
>
> v5 changes:
> 	 - split into three seperate patches
>
> huawei.xhw (3):
>    PCI: use PCI standard sysfs entry to get PIO address
>    PCI: support MMIO in rte_pci_ioport_map/unap/read/write
>    PCI: don't use vfio ioctl call to access PIO resource
>
>   drivers/bus/pci/linux/pci.c     |  89 +-------------------
>   drivers/bus/pci/linux/pci_uio.c | 177 ++++++++++++++++++++++++++++------------
>   2 files changed, 128 insertions(+), 138 deletions(-)
>
  

On 2020/10/27 16:50, chris wrote:
> @Ferruh: this patch is tested with both PIO and MMIO bar using testpmd 
> and start tx_first.
>
> vfio/igb_uio tested with MMIO bar (uio_pci_generic doesn't work with 
> msix, so it isn't tested)
>
> uio_pci_generic tested with PIO bar (igb_uio has unknown symbols, not 
> tested).
     igb_uio with PIO bar is also tested.
>
> Weird igb_uio doens't have Makefile.
>
>
> On 2020/10/22 23:51, 谢华伟(此时此刻） wrote:
>> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>>
>> Legacy virtio-pci only supports PIO BAR resource. As we need to 
>> create lots of
>> virtio devices and PIO resource on x86 is very limited, we expose 
>> MMIO BAR.
>>
>> Kernel supports both PIO  and MMIO BAR for legacy virtio-pci device. 
>> We handles
>> different type of BAR in the similar way.
>>
>> In previous implementation, with igb_uio we get PIO address from igb_uio
>> sysfs entry; with uio_pci_generic, we get PIO address from
>> /proc/ioports.
>> For PIO/MMIO RW, there is different path for different drivers and arch.
>> For VFIO, PIO/MMIO RW is through syscall, which has big performance
>> issue.
>> On X86, it assumes only PIO is supported.
>>
>> All of the above is too much twisted.
>> This patch unifies the way to get both PIO and MMIO address for 
>> different driver
>> and arch, all from standard resource attr under pci sysfs.
>>
>> We distinguish PIO and MMIO by their address like how kernel does. It 
>> is ugly but works.
>>
>> v2 changes:
>>      - add more explanation in the commit message
>>
>> v3 changes:
>>      - fix patch format issues
>>
>> v4 changes:
>>      - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC
>>
>> v5 changes:
>>      - split into three seperate patches
>>
>> huawei.xhw (3):
>>    PCI: use PCI standard sysfs entry to get PIO address
>>    PCI: support MMIO in rte_pci_ioport_map/unap/read/write
>>    PCI: don't use vfio ioctl call to access PIO resource
>>
>>   drivers/bus/pci/linux/pci.c     |  89 +-------------------
>>   drivers/bus/pci/linux/pci_uio.c | 177 
>> ++++++++++++++++++++++++++++------------
>>   2 files changed, 128 insertions(+), 138 deletions(-)
>>
  

Hi Ferruh:

Comments to this v5 version?

On 2020/10/22 23:51, 谢华伟(此时此刻） wrote:
> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>
> Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of
> virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR.
>
> Kernel supports both PIO  and MMIO BAR for legacy virtio-pci device. We handles
> different type of BAR in the similar way.
>
> In previous implementation, with igb_uio we get PIO address from igb_uio
> sysfs entry; with uio_pci_generic, we get PIO address from
> /proc/ioports.
> For PIO/MMIO RW, there is different path for different drivers and arch.
> For VFIO, PIO/MMIO RW is through syscall, which has big performance
> issue.
> On X86, it assumes only PIO is supported.
>
> All of the above is too much twisted.
> This patch unifies the way to get both PIO and MMIO address for different driver
> and arch, all from standard resource attr under pci sysfs.
>
> We distinguish PIO and MMIO by their address like how kernel does. It is ugly but works.
>
> v2 changes:
> 	 - add more explanation in the commit message
>
> v3 changes:
> 	 - fix patch format issues
>
> v4 changes:
> 	 - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC
>
> v5 changes:
> 	 - split into three seperate patches
>
> huawei.xhw (3):
>    PCI: use PCI standard sysfs entry to get PIO address
>    PCI: support MMIO in rte_pci_ioport_map/unap/read/write
>    PCI: don't use vfio ioctl call to access PIO resource
>
>   drivers/bus/pci/linux/pci.c     |  89 +-------------------
>   drivers/bus/pci/linux/pci_uio.c | 177 ++++++++++++++++++++++++++++------------
>   2 files changed, 128 insertions(+), 138 deletions(-)
>
  

Hi David:

I see that you are assigned the reviewer of this patch, and Ferruh have 
helped reviewed it. I rebased this patch based on his comments.

Previously there are different ways to get port address based on 
different DPDK uio driver(IGB_UIO/UIO_PCI_GENERIC/VFIO), which is 
actually not necessary.

This patch makes IO/MMIO port map/RW API more generic, which also 
supports MMIO.  It also fixes performance issue with vfio.

Could you spare some time to have time to review this?


Thanks


On 2020/10/22 23:51, 谢华伟(此时此刻） wrote:
> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>
> Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of
> virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR.
>
> Kernel supports both PIO  and MMIO BAR for legacy virtio-pci device. We handles
> different type of BAR in the similar way.
>
> In previous implementation, with igb_uio we get PIO address from igb_uio
> sysfs entry; with uio_pci_generic, we get PIO address from
> /proc/ioports.
> For PIO/MMIO RW, there is different path for different drivers and arch.
> For VFIO, PIO/MMIO RW is through syscall, which has big performance
> issue.
> On X86, it assumes only PIO is supported.
>
> All of the above is too much twisted.
> This patch unifies the way to get both PIO and MMIO address for different driver
> and arch, all from standard resource attr under pci sysfs.
>
> We distinguish PIO and MMIO by their address like how kernel does. It is ugly but works.
>
> v2 changes:
> 	 - add more explanation in the commit message
>
> v3 changes:
> 	 - fix patch format issues
>
> v4 changes:
> 	 - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC
>
> v5 changes:
> 	 - split into three seperate patches
>
> huawei.xhw (3):
>    PCI: use PCI standard sysfs entry to get PIO address
>    PCI: support MMIO in rte_pci_ioport_map/unap/read/write
>    PCI: don't use vfio ioctl call to access PIO resource
>
>   drivers/bus/pci/linux/pci.c     |  89 +-------------------
>   drivers/bus/pci/linux/pci_uio.c | 177 ++++++++++++++++++++++++++++------------
>   2 files changed, 128 insertions(+), 138 deletions(-)
>
  

On Tue, Nov 10, 2020 at 1:35 PM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
> Previously there are different ways to get port address based on
> different DPDK uio driver(IGB_UIO/UIO_PCI_GENERIC/VFIO), which is
> actually not necessary.
>
> This patch makes IO/MMIO port map/RW API more generic, which also
> supports MMIO.  It also fixes performance issue with vfio.
>
> Could you spare some time to have time to review this?

This is too touchy and I don't want to mess virtio support this late
in the release.
I asked for Maxime to have a look, but he seems really busy.

This will have to wait next release.
  

On 2020/11/10 20:42, David Marchand wrote:
> On Tue, Nov 10, 2020 at 1:35 PM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
>> Previously there are different ways to get port address based on
>> different DPDK uio driver(IGB_UIO/UIO_PCI_GENERIC/VFIO), which is
>> actually not necessary.
>>
>> This patch makes IO/MMIO port map/RW API more generic, which also
>> supports MMIO.  It also fixes performance issue with vfio.
>>
>> Could you spare some time to have time to review this?
> This is too touchy and I don't want to mess virtio support this late
> in the release.
> I asked for Maxime to have a look, but he seems really busy.
>
> This will have to wait next release.

OK.  Actually it isn't that intrusive as it looks.

Then customers have to use git to clone latest DPDK when they run virtio 
with MMIO.  How about backporting this patch to this release after we 
merge it later?

Maxime:

Could we high prioritize this patch a bit? Customers are frequently 
pushing us.

Thanks.

/huawei

>
  

Hi Maxime and David:

Could we start to review this patch?

/Thanks, huawei


On 2020/11/10 20:42, David Marchand wrote:
> On Tue, Nov 10, 2020 at 1:35 PM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
>> Previously there are different ways to get port address based on
>> different DPDK uio driver(IGB_UIO/UIO_PCI_GENERIC/VFIO), which is
>> actually not necessary.
>>
>> This patch makes IO/MMIO port map/RW API more generic, which also
>> supports MMIO.  It also fixes performance issue with vfio.
>>
>> Could you spare some time to have time to review this?
> This is too touchy and I don't want to mess virtio support this late
> in the release.
> I asked for Maxime to have a look, but he seems really busy.
>
> This will have to wait next release.
>
  

Hi Huawei,

On 12/14/20 3:24 PM, 谢华伟(此时此刻） wrote:
> Hi Maxime and David:
> 
> Could we start to review this patch?

Yes, I plan to work on it after the holidays, let's target -rc1.

Thanks,
Maxime

> /Thanks, huawei
> 
> 
> On 2020/11/10 20:42, David Marchand wrote:
>> On Tue, Nov 10, 2020 at 1:35 PM 谢华伟(此时此刻）
>> <huawei.xhw@alibaba-inc.com> wrote:
>>> Previously there are different ways to get port address based on
>>> different DPDK uio driver(IGB_UIO/UIO_PCI_GENERIC/VFIO), which is
>>> actually not necessary.
>>>
>>> This patch makes IO/MMIO port map/RW API more generic, which also
>>> supports MMIO.  It also fixes performance issue with vfio.
>>>
>>> Could you spare some time to have time to review this?
>> This is too touchy and I don't want to mess virtio support this late
>> in the release.
>> I asked for Maxime to have a look, but he seems really busy.
>>
>> This will have to wait next release.
>>
>
  

On 10/22/20 5:51 PM, 谢华伟(此时此刻） wrote:
> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
> 
> Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of
> virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR.
> 
> Kernel supports both PIO  and MMIO BAR for legacy virtio-pci device. We handles
> different type of BAR in the similar way.
> 
> In previous implementation, with igb_uio we get PIO address from igb_uio
> sysfs entry; with uio_pci_generic, we get PIO address from
> /proc/ioports.
> For PIO/MMIO RW, there is different path for different drivers and arch.
> For VFIO, PIO/MMIO RW is through syscall, which has big performance
> issue.

Regarding the performance issue, do you have some numbers to share?
AFAICS, it can only have an impact on performance when interrupt mode is
used or queue notification is enabled.

Does your HW Virtio implementation requires notification?

Is performance the only issue to have your HW working with Virtio PMD,
or is this series also fixing some functionnal issues?

Best regards,
Maxime

> On X86, it assumes only PIO is supported.
> 
> All of the above is too much twisted.
> This patch unifies the way to get both PIO and MMIO address for different driver
> and arch, all from standard resource attr under pci sysfs.
> 
> We distinguish PIO and MMIO by their address like how kernel does. It is ugly but works.
> 
> v2 changes:
> 	 - add more explanation in the commit message
> 
> v3 changes:
> 	 - fix patch format issues
> 
> v4 changes:
> 	 - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC
> 
> v5 changes:
> 	 - split into three seperate patches
> 
> huawei.xhw (3):
>   PCI: use PCI standard sysfs entry to get PIO address
>   PCI: support MMIO in rte_pci_ioport_map/unap/read/write
>   PCI: don't use vfio ioctl call to access PIO resource
> 
>  drivers/bus/pci/linux/pci.c     |  89 +-------------------
>  drivers/bus/pci/linux/pci_uio.c | 177 ++++++++++++++++++++++++++++------------
>  2 files changed, 128 insertions(+), 138 deletions(-)
>
  

On 2021/1/13 1:37, Maxime Coquelin wrote:
>
> On 10/22/20 5:51 PM, 谢华伟(此时此刻） wrote:
>> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>>
>> Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of
>> virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR.
>>
>> Kernel supports both PIO  and MMIO BAR for legacy virtio-pci device. We handles
>> different type of BAR in the similar way.
>>
>> In previous implementation, with igb_uio we get PIO address from igb_uio
>> sysfs entry; with uio_pci_generic, we get PIO address from
>> /proc/ioports.
>> For PIO/MMIO RW, there is different path for different drivers and arch.
>> For VFIO, PIO/MMIO RW is through syscall, which has big performance
>> issue.
> Regarding the performance issue, do you have some numbers to share?
> AFAICS, it can only have an impact on performance when interrupt mode is
> used or queue notification is enabled.

I didn't have performance number, but would do it when have time.

Yes, it is not needed when virtio backend/device is working in polling mode.

But anyway, ioctl isn't needed at all.

/huawei

>
> Does your HW Virtio implementation requires notification?
>
> Is performance the only issue to have your HW working with Virtio PMD,
> or is this series also fixing some functionnal issues?

There is two purpose with this patch. One is to support MMIO, and the 
other is to

unify/simplify the way to get IO/MMIO resource and read/write IO/MMIO 
port for virtio PMD.

Current implementation is too complicated.

/huawei

>
> Best regards,
> Maxime



>> On X86, it assumes only PIO is supported.
>>
>> All of the above is too much twisted.
>> This patch unifies the way to get both PIO and MMIO address for different driver
>> and arch, all from standard resource attr under pci sysfs.
>>
>> We distinguish PIO and MMIO by their address like how kernel does. It is ugly but works.
>>
>> v2 changes:
>> 	 - add more explanation in the commit message
>>
>> v3 changes:
>> 	 - fix patch format issues
>>
>> v4 changes:
>> 	 - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC
>>
>> v5 changes:
>> 	 - split into three seperate patches
>>
>> huawei.xhw (3):
>>    PCI: use PCI standard sysfs entry to get PIO address
>>    PCI: support MMIO in rte_pci_ioport_map/unap/read/write
>>    PCI: don't use vfio ioctl call to access PIO resource
>>
>>   drivers/bus/pci/linux/pci.c     |  89 +-------------------
>>   drivers/bus/pci/linux/pci_uio.c | 177 ++++++++++++++++++++++++++++------------
>>   2 files changed, 128 insertions(+), 138 deletions(-)
>>
  

On 2021/1/13 1:37, Maxime Coquelin wrote:
>
> On 10/22/20 5:51 PM, 谢华伟(此时此刻） wrote:
>> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>>
>> Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of
>> virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR.
>>
>> Kernel supports both PIO  and MMIO BAR for legacy virtio-pci device. We handles
>> different type of BAR in the similar way.
>>
>> In previous implementation, with igb_uio we get PIO address from igb_uio
>> sysfs entry; with uio_pci_generic, we get PIO address from
>> /proc/ioports.
>> For PIO/MMIO RW, there is different path for different drivers and arch.
>> For VFIO, PIO/MMIO RW is through syscall, which has big performance
>> issue.
> Regarding the performance issue, do you have some numbers to share?
> AFAICS, it can only have an impact on performance when interrupt mode is
> used or queue notification is enabled.
>
> Does your HW Virtio implementation requires notification?

Yes, hardware needs notification to tell which queue has more buffer.

vhost backend also needs notification when it is not running in polling 
mode.

It is easy for software backend to sync with frontend whether it needs 
notification through memory but a big burden for hardware.

Anyway, using vfio ioctl isn't needed at all. virtio PMD is only the 
consumer of pci_vfio_ioport_read.

we could consider if we still need pci_vfio_ioport_read related API in 
future.

/huawei
>
> Is performance the only issue to have your HW working with Virtio PMD,
> or is this series also fixing some functionnal issues?
>
> Best regards,
> Maxime
>
  

On 1/21/21 5:12 AM, 谢华伟(此时此刻） wrote:
> 
> On 2021/1/13 1:37, Maxime Coquelin wrote:
>>
>> On 10/22/20 5:51 PM, 谢华伟(此时此刻） wrote:
>>> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>>>
>>> Legacy virtio-pci only supports PIO BAR resource. As we need to
>>> create lots of
>>> virtio devices and PIO resource on x86 is very limited, we expose
>>> MMIO BAR.
>>>
>>> Kernel supports both PIO  and MMIO BAR for legacy virtio-pci device.
>>> We handles
>>> different type of BAR in the similar way.
>>>
>>> In previous implementation, with igb_uio we get PIO address from igb_uio
>>> sysfs entry; with uio_pci_generic, we get PIO address from
>>> /proc/ioports.
>>> For PIO/MMIO RW, there is different path for different drivers and arch.
>>> For VFIO, PIO/MMIO RW is through syscall, which has big performance
>>> issue.
>> Regarding the performance issue, do you have some numbers to share?
>> AFAICS, it can only have an impact on performance when interrupt mode is
>> used or queue notification is enabled.
>>
>> Does your HW Virtio implementation requires notification?
> 
> Yes, hardware needs notification to tell which queue has more buffer.
> 
> vhost backend also needs notification when it is not running in polling
> mode.
> 
> It is easy for software backend to sync with frontend whether it needs
> notification through memory but a big burden for hardware.

Yes, I understand, thanks for the clarification.

> Anyway, using vfio ioctl isn't needed at all. virtio PMD is only the
> consumer of pci_vfio_ioport_read.

My understanding is that using VFIO read/write ops is required for IOMMU
enabled case without cap_sys_rawio. And anyway, using inb/outb is just
bypassing VFIO. As I suggest in my other reply, it is better to document
that in the case of devices having PIO BARs, the user should consider
using UIO driver if performance is a concern.

> we could consider if we still need pci_vfio_ioport_read related API in
> future.

I disagree. I think the pci_vfio_ioport_* API is required at least for
the IOMMU enabled case.

Documentation is the way to go in my opinion, we can also add a warning
that performance may be degraded compared to UIO in
pci_vfio_ioport_map() when IOMMU is disabled if you think it may help
the users.

Thanks,
Maxime

> /huawei
>>
>> Is performance the only issue to have your HW working with Virtio PMD,
>> or is this series also fixing some functionnal issues?
>>
>> Best regards,
>> Maxime
>>
>
  

On 2021/1/21 16:47, Maxime Coquelin wrote:
>
> On 1/21/21 5:12 AM, 谢华伟(此时此刻） wrote:
>> On 2021/1/13 1:37, Maxime Coquelin wrote:
>>> On 10/22/20 5:51 PM, 谢华伟(此时此刻） wrote:
>>>> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
>>>>
>>>> Legacy virtio-pci only supports PIO BAR resource. As we need to
>>>> create lots of
>>>> virtio devices and PIO resource on x86 is very limited, we expose
>>>> MMIO BAR.
>>>>
>>>> Kernel supports both PIO  and MMIO BAR for legacy virtio-pci device.
>>>> We handles
>>>> different type of BAR in the similar way.
>>>>
>>>> In previous implementation, with igb_uio we get PIO address from igb_uio
>>>> sysfs entry; with uio_pci_generic, we get PIO address from
>>>> /proc/ioports.
>>>> For PIO/MMIO RW, there is different path for different drivers and arch.
>>>> For VFIO, PIO/MMIO RW is through syscall, which has big performance
>>>> issue.
>>> Regarding the performance issue, do you have some numbers to share?
>>> AFAICS, it can only have an impact on performance when interrupt mode is
>>> used or queue notification is enabled.
>>>
>>> Does your HW Virtio implementation requires notification?
>> Yes, hardware needs notification to tell which queue has more buffer.
>>
>> vhost backend also needs notification when it is not running in polling
>> mode.
>>
>> It is easy for software backend to sync with frontend whether it needs
>> notification through memory but a big burden for hardware.
> Yes, I understand, thanks for the clarification.
>
>> Anyway, using vfio ioctl isn't needed at all. virtio PMD is only the
>> consumer of pci_vfio_ioport_read.
> My understanding is that using VFIO read/write ops is required for IOMMU
> enabled case without cap_sys_rawio. And anyway, using inb/outb is just
> bypassing VFIO. As I suggest in my other reply, it is better to document
> that in the case of devices having PIO BARs, the user should consider
> using UIO driver if performance is a concern.

Get it. so user could read/write PIO using VFIO without iopl permission, 
with some performance penalty.


>> we could consider if we still need pci_vfio_ioport_read related API in
>> future.
> I disagree. I think the pci_vfio_ioport_* API is required at least for
> the IOMMU enabled case.
>
> Documentation is the way to go in my opinion, we can also add a warning
> that performance may be degraded compared to UIO in
> pci_vfio_ioport_map() when IOMMU is disabled if you think it may help
> the users.
>
> Thanks,
> Maxime
>
>> /huawei
>>> Is performance the only issue to have your HW working with Virtio PMD,
>>> or is this series also fixing some functionnal issues?
>>>
>>> Best regards,
>>> Maxime
>>>