[v11,0/2] support both PIO and MMIO BAR for legacy virito device

  virtio PMD assumes legacy device only supports PIO(port-mapped) BAR
resource. This is wrong. As we need to create lots of devices, adn PIO
resource on x86 is very limited, we expose MMIO(memory-mapped I/O) BAR.

Kernel supports both PIO and MMIO BAR for legacy virtio-pci device, and
for all other pci devices. This patchset handles different type of BAR in
the similar way.

In previous implementation, under igb_uio driver we get PIO address from
igb_uio sysfs entry; with uio_pci_generic, we get PIO address from
/proc/ioports for x86, and for other ARCHs, we get PIO address from
standard PCI sysfs entry. For PIO/MMIO RW, there is different path for
different drivers and arch.

All of the above is too much twisted. This patchset unifies the way to get
both PIO and MMIO address for different driver and ARCHs, all from standard
resource attr under pci sysfs. This is most generic.

We distinguish PIO and MMIO by their address range like how kernel does.
It is ugly but works.

v2 changes:
    - add more explanation in the commit message

v3 changes:
    - fix patch format issues

v4 changes:
    - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC

v5 changes:
    - split into three seperate patches

v6 changes:
    - change to DEBUG level for IO bar detection in pci_uio_ioport_map
    - rework the code in iobar branch
    - fixes commit message format issue
    - temporarily remove the 3rd patch for vfio path, leave it for future discusssion
    - rework against virtio_pmd_rework_v2

v7 changes:
    - fix compilation issues of in/out instruction on non X86 archs

v8 changes:
    - change the word fix to refactor in patch 1's commit message

v9 changes:
    - keep pause version in in/out instructions

v10 changes:
    - trival fixes in commit message, like > 75 chars

v11 changes:
    - commit message fix and change

huawei xie (2):
  bus/pci: use PCI standard sysfs entry to get PIO address
  bus/pci: support MMIO in PCI ioport accessors

 drivers/bus/pci/linux/pci.c     |  81 ----------------
 drivers/bus/pci/linux/pci_uio.c | 202 +++++++++++++++++++++++++++++-----------
 2 files changed, 150 insertions(+), 133 deletions(-)
  

Tested-by: Wang, Yinan <yinan.wang@intel.com>

Tested PVP case with virtio PMD assumes legacy device, VM with below kernel LTS versions, all pass except vfio-pci test blocked with kernel v4.4.
5.10.0-051000-generic                     virtio-pmd test with  vfio-pci/ igb_uio/uio_pci_generic  all pass
5.4.0-050400-generic                       virtio-pmd test with  vfio-pci/ igb_uio/uio_pci_generic  all pass
4.19.179-0419179-generic             virtio-pmd test with  vfio-pci/ igb_uio/uio_pci_generic  all pass
4.9.260-0409260-generic               virtio-pmd test with  vfio-pci/ igb_uio/uio_pci_generic  all pass
4.4.260-0404260-generic               virtio-pmd test with  igb_uio/uio_pci_generic can pass ; vfio-pci blocked as fail to bind vfio-pci to virtio-pmd

Error info:		
root@vmubuntu2004:~/dpdk/usertools# ./dpdk-devbind.py -b vfio-pci 00:04.0
lspci: Unable to load libkmod resources: error -12

root@vmubuntu2004:~/dpdk/usertools# dmesg
[  161.553493] VFIO - User Level meta-driver version: 0.3
[  179.430529] vfio-pci: probe of 0000:00:04.0 failed with error -22

BR,
Yinan
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of ???(????)
> Sent: 2021年3月11日 1:36
> To: david.marchand@redhat.com; maxime.coquelin@redhat.com; Yigit,
> Ferruh <ferruh.yigit@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>;
> xuemingl@nvidia.com; grive@u256.net; 谢华伟(此时此刻）
> <huawei.xhw@alibaba-inc.com>
> Subject: [dpdk-dev] [PATCH v11 0/2] support both PIO and MMIO BAR for
> legacy virito device
> 
> virtio PMD assumes legacy device only supports PIO(port-mapped) BAR
> resource. This is wrong. As we need to create lots of devices, adn PIO
> resource on x86 is very limited, we expose MMIO(memory-mapped I/O) BAR.
> 
> Kernel supports both PIO and MMIO BAR for legacy virtio-pci device, and
> for all other pci devices. This patchset handles different type of BAR in
> the similar way.
> 
> In previous implementation, under igb_uio driver we get PIO address from
> igb_uio sysfs entry; with uio_pci_generic, we get PIO address from
> /proc/ioports for x86, and for other ARCHs, we get PIO address from
> standard PCI sysfs entry. For PIO/MMIO RW, there is different path for
> different drivers and arch.
> 
> All of the above is too much twisted. This patchset unifies the way to get
> both PIO and MMIO address for different driver and ARCHs, all from standard
> resource attr under pci sysfs. This is most generic.
> 
> We distinguish PIO and MMIO by their address range like how kernel does.
> It is ugly but works.
> 
> v2 changes:
>     - add more explanation in the commit message
> 
> v3 changes:
>     - fix patch format issues
> 
> v4 changes:
>     - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC
> 
> v5 changes:
>     - split into three seperate patches
> 
> v6 changes:
>     - change to DEBUG level for IO bar detection in pci_uio_ioport_map
>     - rework the code in iobar branch
>     - fixes commit message format issue
>     - temporarily remove the 3rd patch for vfio path, leave it for future
> discusssion
>     - rework against virtio_pmd_rework_v2
> 
> v7 changes:
>     - fix compilation issues of in/out instruction on non X86 archs
> 
> v8 changes:
>     - change the word fix to refactor in patch 1's commit message
> 
> v9 changes:
>     - keep pause version in in/out instructions
> 
> v10 changes:
>     - trival fixes in commit message, like > 75 chars
> 
> v11 changes:
>     - commit message fix and change
> 
> huawei xie (2):
>   bus/pci: use PCI standard sysfs entry to get PIO address
>   bus/pci: support MMIO in PCI ioport accessors
> 
>  drivers/bus/pci/linux/pci.c     |  81 ----------------
>  drivers/bus/pci/linux/pci_uio.c | 202 +++++++++++++++++++++++++++++--
> ---------
>  2 files changed, 150 insertions(+), 133 deletions(-)
> 
> --
> 1.8.3.1
  

Hello,

On Thu, Mar 11, 2021 at 12:55 PM Wang, Yinan <yinan.wang@intel.com> wrote:
>
> Tested-by: Wang, Yinan <yinan.wang@intel.com>
>
> Tested PVP case with virtio PMD assumes legacy device, VM with below kernel LTS versions, all pass except vfio-pci test blocked with kernel v4.4.
> 5.10.0-051000-generic                     virtio-pmd test with  vfio-pci/ igb_uio/uio_pci_generic  all pass
> 5.4.0-050400-generic                       virtio-pmd test with  vfio-pci/ igb_uio/uio_pci_generic  all pass
> 4.19.179-0419179-generic             virtio-pmd test with  vfio-pci/ igb_uio/uio_pci_generic  all pass
> 4.9.260-0409260-generic               virtio-pmd test with  vfio-pci/ igb_uio/uio_pci_generic  all pass
> 4.4.260-0404260-generic               virtio-pmd test with  igb_uio/uio_pci_generic can pass ; vfio-pci blocked as fail to bind vfio-pci to virtio-pmd
>
> Error info:
> root@vmubuntu2004:~/dpdk/usertools# ./dpdk-devbind.py -b vfio-pci 00:04.0
> lspci: Unable to load libkmod resources: error -12
>
> root@vmubuntu2004:~/dpdk/usertools# dmesg
> [  161.553493] VFIO - User Level meta-driver version: 0.3
> [  179.430529] vfio-pci: probe of 0000:00:04.0 failed with error -22

The vanilla stable 4.4 kernel does not support "No-IOMMU mode" for vfio.
ae5515d66362 - Revert: "vfio: Include No-IOMMU mode" (5 years ago)
<Alex Williamson>

So this behavior is expected if you were testing with no iommu in your
virtual machine.


Thanks for the tests!
  

On Wed, Mar 10, 2021 at 6:37 PM 谢华伟(此时此刻） <huawei.xhw@alibaba-inc.com> wrote:
>
> virtio PMD assumes legacy device only supports PIO(port-mapped) BAR
> resource. This is wrong. As we need to create lots of devices, adn PIO
> resource on x86 is very limited, we expose MMIO(memory-mapped I/O) BAR.
>
> Kernel supports both PIO and MMIO BAR for legacy virtio-pci device, and
> for all other pci devices. This patchset handles different type of BAR in
> the similar way.
>
> In previous implementation, under igb_uio driver we get PIO address from
> igb_uio sysfs entry; with uio_pci_generic, we get PIO address from
> /proc/ioports for x86, and for other ARCHs, we get PIO address from
> standard PCI sysfs entry. For PIO/MMIO RW, there is different path for
> different drivers and arch.
>
> All of the above is too much twisted. This patchset unifies the way to get
> both PIO and MMIO address for different driver and ARCHs, all from standard
> resource attr under pci sysfs. This is most generic.
>
> We distinguish PIO and MMIO by their address range like how kernel does.
> It is ugly but works.
>
> v2 changes:
>     - add more explanation in the commit message
>
> v3 changes:
>     - fix patch format issues
>
> v4 changes:
>     - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC
>
> v5 changes:
>     - split into three seperate patches
>
> v6 changes:
>     - change to DEBUG level for IO bar detection in pci_uio_ioport_map
>     - rework the code in iobar branch
>     - fixes commit message format issue
>     - temporarily remove the 3rd patch for vfio path, leave it for future discusssion
>     - rework against virtio_pmd_rework_v2
>
> v7 changes:
>     - fix compilation issues of in/out instruction on non X86 archs
>
> v8 changes:
>     - change the word fix to refactor in patch 1's commit message
>
> v9 changes:
>     - keep pause version in in/out instructions
>
> v10 changes:
>     - trival fixes in commit message, like > 75 chars
>
> v11 changes:
>     - commit message fix and change
>

Aligned Sob and Author to fix the last checkpatch warning.

Series applied to the main branch.
Thanks Huawei and thanks too to reviewers/testers.
  

On 2021/3/15 22:16, David Marchand wrote:
>> v10 changes:
>>      - trival fixes in commit message, like > 75 chars
>>
>> v11 changes:
>>      - commit message fix and change
>>
> Aligned Sob and Author to fix the last checkpatch warning.
>
> Series applied to the main branch.
> Thanks Huawei and thanks too to reviewers/testers.

Thanks!