Message ID | 1603381885-88819-1-git-send-email-huawei.xhw@alibaba-inc.com (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8793AA04DD; Thu, 22 Oct 2020 17:51:41 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DD519AC47; Thu, 22 Oct 2020 17:51:39 +0200 (CEST) Received: from out0-153.mail.aliyun.com (out0-153.mail.aliyun.com [140.205.0.153]) by dpdk.org (Postfix) with ESMTP id 60C7FAC47 for <dev@dpdk.org>; Thu, 22 Oct 2020 17:51:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alibaba-inc.com; s=default; t=1603381892; h=From:To:Subject:Date:Message-Id; bh=1JM/jk6RbtfOQuGqcSekZWgloDvAQa8k2ivNpPLz7Yk=; b=p8QzHX/5l4/z1OABl3xZvydANFYMwRmlkeGPHUhzIHGh0Td8I2DHnDPihYwqt/ETTV4DIyADS8IqS1NMNzvRrVPuPXvMm/gf/Q2c289lBF4tP9o4OWPy+vFdt0nspTLxMowulhJIXMu1yAGhrY+Y34VwDa2iLAt3eduaKEpTZMA= X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R171e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=ay29a033018047187; MF=huawei.xhw@alibaba-inc.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---.In.51nB_1603381889; Received: from rs3a10040.et2sqa.z1.et2sqa.tbsite.net(mailfrom:huawei.xhw@alibaba-inc.com fp:SMTPD_---.In.51nB_1603381889) by smtp.aliyun-inc.com(127.0.0.1); Thu, 22 Oct 2020 23:51:31 +0800 From: " =?utf-8?b?6LCi5Y2O5LyfKOatpOaXtuatpOWIu++8iQ==?= " <huawei.xhw@alibaba-inc.com> To: ferruh.yigit@intel.com Cc: <dev@dpdk.org>, <maxime.coquelin@redhat.com>, <anatoly.burakov@intel.com>, <david.marchand@redhat.com>, <zhihong.wang@intel.com>, <chenbo.xia@intel.com>, <grive@u256.net>, " =?utf-8?b?6LCi5Y2O5LyfKOatpA==?= =?utf-8?b?5pe25q2k5Yi777yJ?= " <huawei.xhw@alibaba-inc.com> Date: Thu, 22 Oct 2020 23:51:22 +0800 Message-Id: <1603381885-88819-1-git-send-email-huawei.xhw@alibaba-inc.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <68ecd941-9c56-4de7-fae2-2ad15bdfd81a@alibaba-inc.com> References: <68ecd941-9c56-4de7-fae2-2ad15bdfd81a@alibaba-inc.com> Subject: [dpdk-dev] [PATCH v5 0/3] support both PIO and MMIO BAR for virtio PMD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
Series |
support both PIO and MMIO BAR for virtio PMD
|
|
Message
谢华伟(此时此刻)
Oct. 22, 2020, 3:51 p.m. UTC
From: "huawei.xhw" <huawei.xhw@alibaba-inc.com>
Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of
virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR.
Kernel supports both PIO and MMIO BAR for legacy virtio-pci device. We handles
different type of BAR in the similar way.
In previous implementation, with igb_uio we get PIO address from igb_uio
sysfs entry; with uio_pci_generic, we get PIO address from
/proc/ioports.
For PIO/MMIO RW, there is different path for different drivers and arch.
For VFIO, PIO/MMIO RW is through syscall, which has big performance
issue.
On X86, it assumes only PIO is supported.
All of the above is too much twisted.
This patch unifies the way to get both PIO and MMIO address for different driver
and arch, all from standard resource attr under pci sysfs.
We distinguish PIO and MMIO by their address like how kernel does. It is ugly but works.
v2 changes:
- add more explanation in the commit message
v3 changes:
- fix patch format issues
v4 changes:
- fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC
v5 changes:
- split into three seperate patches
huawei.xhw (3):
PCI: use PCI standard sysfs entry to get PIO address
PCI: support MMIO in rte_pci_ioport_map/unap/read/write
PCI: don't use vfio ioctl call to access PIO resource
drivers/bus/pci/linux/pci.c | 89 +-------------------
drivers/bus/pci/linux/pci_uio.c | 177 ++++++++++++++++++++++++++++------------
2 files changed, 128 insertions(+), 138 deletions(-)
Comments
@Ferruh: this patch is tested with both PIO and MMIO bar using testpmd and start tx_first. vfio/igb_uio tested with MMIO bar (uio_pci_generic doesn't work with msix, so it isn't tested) uio_pci_generic tested with PIO bar (igb_uio has unknown symbols, not tested). Weird igb_uio doens't have Makefile. On 2020/10/22 23:51, 谢华伟(此时此刻) wrote: > From: "huawei.xhw" <huawei.xhw@alibaba-inc.com> > > Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of > virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR. > > Kernel supports both PIO and MMIO BAR for legacy virtio-pci device. We handles > different type of BAR in the similar way. > > In previous implementation, with igb_uio we get PIO address from igb_uio > sysfs entry; with uio_pci_generic, we get PIO address from > /proc/ioports. > For PIO/MMIO RW, there is different path for different drivers and arch. > For VFIO, PIO/MMIO RW is through syscall, which has big performance > issue. > On X86, it assumes only PIO is supported. > > All of the above is too much twisted. > This patch unifies the way to get both PIO and MMIO address for different driver > and arch, all from standard resource attr under pci sysfs. > > We distinguish PIO and MMIO by their address like how kernel does. It is ugly but works. > > v2 changes: > - add more explanation in the commit message > > v3 changes: > - fix patch format issues > > v4 changes: > - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC > > v5 changes: > - split into three seperate patches > > huawei.xhw (3): > PCI: use PCI standard sysfs entry to get PIO address > PCI: support MMIO in rte_pci_ioport_map/unap/read/write > PCI: don't use vfio ioctl call to access PIO resource > > drivers/bus/pci/linux/pci.c | 89 +------------------- > drivers/bus/pci/linux/pci_uio.c | 177 ++++++++++++++++++++++++++++------------ > 2 files changed, 128 insertions(+), 138 deletions(-) >
On 2020/10/27 16:50, chris wrote: > @Ferruh: this patch is tested with both PIO and MMIO bar using testpmd > and start tx_first. > > vfio/igb_uio tested with MMIO bar (uio_pci_generic doesn't work with > msix, so it isn't tested) > > uio_pci_generic tested with PIO bar (igb_uio has unknown symbols, not > tested). igb_uio with PIO bar is also tested. > > Weird igb_uio doens't have Makefile. > > > On 2020/10/22 23:51, 谢华伟(此时此刻) wrote: >> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com> >> >> Legacy virtio-pci only supports PIO BAR resource. As we need to >> create lots of >> virtio devices and PIO resource on x86 is very limited, we expose >> MMIO BAR. >> >> Kernel supports both PIO and MMIO BAR for legacy virtio-pci device. >> We handles >> different type of BAR in the similar way. >> >> In previous implementation, with igb_uio we get PIO address from igb_uio >> sysfs entry; with uio_pci_generic, we get PIO address from >> /proc/ioports. >> For PIO/MMIO RW, there is different path for different drivers and arch. >> For VFIO, PIO/MMIO RW is through syscall, which has big performance >> issue. >> On X86, it assumes only PIO is supported. >> >> All of the above is too much twisted. >> This patch unifies the way to get both PIO and MMIO address for >> different driver >> and arch, all from standard resource attr under pci sysfs. >> >> We distinguish PIO and MMIO by their address like how kernel does. It >> is ugly but works. >> >> v2 changes: >> - add more explanation in the commit message >> >> v3 changes: >> - fix patch format issues >> >> v4 changes: >> - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC >> >> v5 changes: >> - split into three seperate patches >> >> huawei.xhw (3): >> PCI: use PCI standard sysfs entry to get PIO address >> PCI: support MMIO in rte_pci_ioport_map/unap/read/write >> PCI: don't use vfio ioctl call to access PIO resource >> >> drivers/bus/pci/linux/pci.c | 89 +------------------- >> drivers/bus/pci/linux/pci_uio.c | 177 >> ++++++++++++++++++++++++++++------------ >> 2 files changed, 128 insertions(+), 138 deletions(-) >>
Hi Ferruh: Comments to this v5 version? On 2020/10/22 23:51, 谢华伟(此时此刻) wrote: > From: "huawei.xhw" <huawei.xhw@alibaba-inc.com> > > Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of > virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR. > > Kernel supports both PIO and MMIO BAR for legacy virtio-pci device. We handles > different type of BAR in the similar way. > > In previous implementation, with igb_uio we get PIO address from igb_uio > sysfs entry; with uio_pci_generic, we get PIO address from > /proc/ioports. > For PIO/MMIO RW, there is different path for different drivers and arch. > For VFIO, PIO/MMIO RW is through syscall, which has big performance > issue. > On X86, it assumes only PIO is supported. > > All of the above is too much twisted. > This patch unifies the way to get both PIO and MMIO address for different driver > and arch, all from standard resource attr under pci sysfs. > > We distinguish PIO and MMIO by their address like how kernel does. It is ugly but works. > > v2 changes: > - add more explanation in the commit message > > v3 changes: > - fix patch format issues > > v4 changes: > - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC > > v5 changes: > - split into three seperate patches > > huawei.xhw (3): > PCI: use PCI standard sysfs entry to get PIO address > PCI: support MMIO in rte_pci_ioport_map/unap/read/write > PCI: don't use vfio ioctl call to access PIO resource > > drivers/bus/pci/linux/pci.c | 89 +------------------- > drivers/bus/pci/linux/pci_uio.c | 177 ++++++++++++++++++++++++++++------------ > 2 files changed, 128 insertions(+), 138 deletions(-) >
Hi David: I see that you are assigned the reviewer of this patch, and Ferruh have helped reviewed it. I rebased this patch based on his comments. Previously there are different ways to get port address based on different DPDK uio driver(IGB_UIO/UIO_PCI_GENERIC/VFIO), which is actually not necessary. This patch makes IO/MMIO port map/RW API more generic, which also supports MMIO. It also fixes performance issue with vfio. Could you spare some time to have time to review this? Thanks On 2020/10/22 23:51, 谢华伟(此时此刻) wrote: > From: "huawei.xhw" <huawei.xhw@alibaba-inc.com> > > Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of > virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR. > > Kernel supports both PIO and MMIO BAR for legacy virtio-pci device. We handles > different type of BAR in the similar way. > > In previous implementation, with igb_uio we get PIO address from igb_uio > sysfs entry; with uio_pci_generic, we get PIO address from > /proc/ioports. > For PIO/MMIO RW, there is different path for different drivers and arch. > For VFIO, PIO/MMIO RW is through syscall, which has big performance > issue. > On X86, it assumes only PIO is supported. > > All of the above is too much twisted. > This patch unifies the way to get both PIO and MMIO address for different driver > and arch, all from standard resource attr under pci sysfs. > > We distinguish PIO and MMIO by their address like how kernel does. It is ugly but works. > > v2 changes: > - add more explanation in the commit message > > v3 changes: > - fix patch format issues > > v4 changes: > - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC > > v5 changes: > - split into three seperate patches > > huawei.xhw (3): > PCI: use PCI standard sysfs entry to get PIO address > PCI: support MMIO in rte_pci_ioport_map/unap/read/write > PCI: don't use vfio ioctl call to access PIO resource > > drivers/bus/pci/linux/pci.c | 89 +------------------- > drivers/bus/pci/linux/pci_uio.c | 177 ++++++++++++++++++++++++++++------------ > 2 files changed, 128 insertions(+), 138 deletions(-) >
On Tue, Nov 10, 2020 at 1:35 PM 谢华伟(此时此刻) <huawei.xhw@alibaba-inc.com> wrote: > Previously there are different ways to get port address based on > different DPDK uio driver(IGB_UIO/UIO_PCI_GENERIC/VFIO), which is > actually not necessary. > > This patch makes IO/MMIO port map/RW API more generic, which also > supports MMIO. It also fixes performance issue with vfio. > > Could you spare some time to have time to review this? This is too touchy and I don't want to mess virtio support this late in the release. I asked for Maxime to have a look, but he seems really busy. This will have to wait next release.
On 2020/11/10 20:42, David Marchand wrote: > On Tue, Nov 10, 2020 at 1:35 PM 谢华伟(此时此刻) <huawei.xhw@alibaba-inc.com> wrote: >> Previously there are different ways to get port address based on >> different DPDK uio driver(IGB_UIO/UIO_PCI_GENERIC/VFIO), which is >> actually not necessary. >> >> This patch makes IO/MMIO port map/RW API more generic, which also >> supports MMIO. It also fixes performance issue with vfio. >> >> Could you spare some time to have time to review this? > This is too touchy and I don't want to mess virtio support this late > in the release. > I asked for Maxime to have a look, but he seems really busy. > > This will have to wait next release. OK. Actually it isn't that intrusive as it looks. Then customers have to use git to clone latest DPDK when they run virtio with MMIO. How about backporting this patch to this release after we merge it later? Maxime: Could we high prioritize this patch a bit? Customers are frequently pushing us. Thanks. /huawei >
Hi Maxime and David: Could we start to review this patch? /Thanks, huawei On 2020/11/10 20:42, David Marchand wrote: > On Tue, Nov 10, 2020 at 1:35 PM 谢华伟(此时此刻) <huawei.xhw@alibaba-inc.com> wrote: >> Previously there are different ways to get port address based on >> different DPDK uio driver(IGB_UIO/UIO_PCI_GENERIC/VFIO), which is >> actually not necessary. >> >> This patch makes IO/MMIO port map/RW API more generic, which also >> supports MMIO. It also fixes performance issue with vfio. >> >> Could you spare some time to have time to review this? > This is too touchy and I don't want to mess virtio support this late > in the release. > I asked for Maxime to have a look, but he seems really busy. > > This will have to wait next release. >
Hi Huawei, On 12/14/20 3:24 PM, 谢华伟(此时此刻) wrote: > Hi Maxime and David: > > Could we start to review this patch? Yes, I plan to work on it after the holidays, let's target -rc1. Thanks, Maxime > /Thanks, huawei > > > On 2020/11/10 20:42, David Marchand wrote: >> On Tue, Nov 10, 2020 at 1:35 PM 谢华伟(此时此刻) >> <huawei.xhw@alibaba-inc.com> wrote: >>> Previously there are different ways to get port address based on >>> different DPDK uio driver(IGB_UIO/UIO_PCI_GENERIC/VFIO), which is >>> actually not necessary. >>> >>> This patch makes IO/MMIO port map/RW API more generic, which also >>> supports MMIO. It also fixes performance issue with vfio. >>> >>> Could you spare some time to have time to review this? >> This is too touchy and I don't want to mess virtio support this late >> in the release. >> I asked for Maxime to have a look, but he seems really busy. >> >> This will have to wait next release. >> >
On 10/22/20 5:51 PM, 谢华伟(此时此刻) wrote: > From: "huawei.xhw" <huawei.xhw@alibaba-inc.com> > > Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of > virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR. > > Kernel supports both PIO and MMIO BAR for legacy virtio-pci device. We handles > different type of BAR in the similar way. > > In previous implementation, with igb_uio we get PIO address from igb_uio > sysfs entry; with uio_pci_generic, we get PIO address from > /proc/ioports. > For PIO/MMIO RW, there is different path for different drivers and arch. > For VFIO, PIO/MMIO RW is through syscall, which has big performance > issue. Regarding the performance issue, do you have some numbers to share? AFAICS, it can only have an impact on performance when interrupt mode is used or queue notification is enabled. Does your HW Virtio implementation requires notification? Is performance the only issue to have your HW working with Virtio PMD, or is this series also fixing some functionnal issues? Best regards, Maxime > On X86, it assumes only PIO is supported. > > All of the above is too much twisted. > This patch unifies the way to get both PIO and MMIO address for different driver > and arch, all from standard resource attr under pci sysfs. > > We distinguish PIO and MMIO by their address like how kernel does. It is ugly but works. > > v2 changes: > - add more explanation in the commit message > > v3 changes: > - fix patch format issues > > v4 changes: > - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC > > v5 changes: > - split into three seperate patches > > huawei.xhw (3): > PCI: use PCI standard sysfs entry to get PIO address > PCI: support MMIO in rte_pci_ioport_map/unap/read/write > PCI: don't use vfio ioctl call to access PIO resource > > drivers/bus/pci/linux/pci.c | 89 +------------------- > drivers/bus/pci/linux/pci_uio.c | 177 ++++++++++++++++++++++++++++------------ > 2 files changed, 128 insertions(+), 138 deletions(-) >
On 2021/1/13 1:37, Maxime Coquelin wrote: > > On 10/22/20 5:51 PM, 谢华伟(此时此刻) wrote: >> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com> >> >> Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of >> virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR. >> >> Kernel supports both PIO and MMIO BAR for legacy virtio-pci device. We handles >> different type of BAR in the similar way. >> >> In previous implementation, with igb_uio we get PIO address from igb_uio >> sysfs entry; with uio_pci_generic, we get PIO address from >> /proc/ioports. >> For PIO/MMIO RW, there is different path for different drivers and arch. >> For VFIO, PIO/MMIO RW is through syscall, which has big performance >> issue. > Regarding the performance issue, do you have some numbers to share? > AFAICS, it can only have an impact on performance when interrupt mode is > used or queue notification is enabled. I didn't have performance number, but would do it when have time. Yes, it is not needed when virtio backend/device is working in polling mode. But anyway, ioctl isn't needed at all. /huawei > > Does your HW Virtio implementation requires notification? > > Is performance the only issue to have your HW working with Virtio PMD, > or is this series also fixing some functionnal issues? There is two purpose with this patch. One is to support MMIO, and the other is to unify/simplify the way to get IO/MMIO resource and read/write IO/MMIO port for virtio PMD. Current implementation is too complicated. /huawei > > Best regards, > Maxime >> On X86, it assumes only PIO is supported. >> >> All of the above is too much twisted. >> This patch unifies the way to get both PIO and MMIO address for different driver >> and arch, all from standard resource attr under pci sysfs. >> >> We distinguish PIO and MMIO by their address like how kernel does. It is ugly but works. >> >> v2 changes: >> - add more explanation in the commit message >> >> v3 changes: >> - fix patch format issues >> >> v4 changes: >> - fixes for RTE_KDRV_UIO_GENERIC -> RTE_PCI_KDRV_UIO_GENERIC >> >> v5 changes: >> - split into three seperate patches >> >> huawei.xhw (3): >> PCI: use PCI standard sysfs entry to get PIO address >> PCI: support MMIO in rte_pci_ioport_map/unap/read/write >> PCI: don't use vfio ioctl call to access PIO resource >> >> drivers/bus/pci/linux/pci.c | 89 +------------------- >> drivers/bus/pci/linux/pci_uio.c | 177 ++++++++++++++++++++++++++++------------ >> 2 files changed, 128 insertions(+), 138 deletions(-) >>
On 2021/1/13 1:37, Maxime Coquelin wrote: > > On 10/22/20 5:51 PM, 谢华伟(此时此刻) wrote: >> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com> >> >> Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of >> virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR. >> >> Kernel supports both PIO and MMIO BAR for legacy virtio-pci device. We handles >> different type of BAR in the similar way. >> >> In previous implementation, with igb_uio we get PIO address from igb_uio >> sysfs entry; with uio_pci_generic, we get PIO address from >> /proc/ioports. >> For PIO/MMIO RW, there is different path for different drivers and arch. >> For VFIO, PIO/MMIO RW is through syscall, which has big performance >> issue. > Regarding the performance issue, do you have some numbers to share? > AFAICS, it can only have an impact on performance when interrupt mode is > used or queue notification is enabled. > > Does your HW Virtio implementation requires notification? Yes, hardware needs notification to tell which queue has more buffer. vhost backend also needs notification when it is not running in polling mode. It is easy for software backend to sync with frontend whether it needs notification through memory but a big burden for hardware. Anyway, using vfio ioctl isn't needed at all. virtio PMD is only the consumer of pci_vfio_ioport_read. we could consider if we still need pci_vfio_ioport_read related API in future. /huawei > > Is performance the only issue to have your HW working with Virtio PMD, > or is this series also fixing some functionnal issues? > > Best regards, > Maxime >
On 1/21/21 5:12 AM, 谢华伟(此时此刻) wrote: > > On 2021/1/13 1:37, Maxime Coquelin wrote: >> >> On 10/22/20 5:51 PM, 谢华伟(此时此刻) wrote: >>> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com> >>> >>> Legacy virtio-pci only supports PIO BAR resource. As we need to >>> create lots of >>> virtio devices and PIO resource on x86 is very limited, we expose >>> MMIO BAR. >>> >>> Kernel supports both PIO and MMIO BAR for legacy virtio-pci device. >>> We handles >>> different type of BAR in the similar way. >>> >>> In previous implementation, with igb_uio we get PIO address from igb_uio >>> sysfs entry; with uio_pci_generic, we get PIO address from >>> /proc/ioports. >>> For PIO/MMIO RW, there is different path for different drivers and arch. >>> For VFIO, PIO/MMIO RW is through syscall, which has big performance >>> issue. >> Regarding the performance issue, do you have some numbers to share? >> AFAICS, it can only have an impact on performance when interrupt mode is >> used or queue notification is enabled. >> >> Does your HW Virtio implementation requires notification? > > Yes, hardware needs notification to tell which queue has more buffer. > > vhost backend also needs notification when it is not running in polling > mode. > > It is easy for software backend to sync with frontend whether it needs > notification through memory but a big burden for hardware. Yes, I understand, thanks for the clarification. > Anyway, using vfio ioctl isn't needed at all. virtio PMD is only the > consumer of pci_vfio_ioport_read. My understanding is that using VFIO read/write ops is required for IOMMU enabled case without cap_sys_rawio. And anyway, using inb/outb is just bypassing VFIO. As I suggest in my other reply, it is better to document that in the case of devices having PIO BARs, the user should consider using UIO driver if performance is a concern. > we could consider if we still need pci_vfio_ioport_read related API in > future. I disagree. I think the pci_vfio_ioport_* API is required at least for the IOMMU enabled case. Documentation is the way to go in my opinion, we can also add a warning that performance may be degraded compared to UIO in pci_vfio_ioport_map() when IOMMU is disabled if you think it may help the users. Thanks, Maxime > /huawei >> >> Is performance the only issue to have your HW working with Virtio PMD, >> or is this series also fixing some functionnal issues? >> >> Best regards, >> Maxime >> >
On 2021/1/21 16:47, Maxime Coquelin wrote: > > On 1/21/21 5:12 AM, 谢华伟(此时此刻) wrote: >> On 2021/1/13 1:37, Maxime Coquelin wrote: >>> On 10/22/20 5:51 PM, 谢华伟(此时此刻) wrote: >>>> From: "huawei.xhw" <huawei.xhw@alibaba-inc.com> >>>> >>>> Legacy virtio-pci only supports PIO BAR resource. As we need to >>>> create lots of >>>> virtio devices and PIO resource on x86 is very limited, we expose >>>> MMIO BAR. >>>> >>>> Kernel supports both PIO and MMIO BAR for legacy virtio-pci device. >>>> We handles >>>> different type of BAR in the similar way. >>>> >>>> In previous implementation, with igb_uio we get PIO address from igb_uio >>>> sysfs entry; with uio_pci_generic, we get PIO address from >>>> /proc/ioports. >>>> For PIO/MMIO RW, there is different path for different drivers and arch. >>>> For VFIO, PIO/MMIO RW is through syscall, which has big performance >>>> issue. >>> Regarding the performance issue, do you have some numbers to share? >>> AFAICS, it can only have an impact on performance when interrupt mode is >>> used or queue notification is enabled. >>> >>> Does your HW Virtio implementation requires notification? >> Yes, hardware needs notification to tell which queue has more buffer. >> >> vhost backend also needs notification when it is not running in polling >> mode. >> >> It is easy for software backend to sync with frontend whether it needs >> notification through memory but a big burden for hardware. > Yes, I understand, thanks for the clarification. > >> Anyway, using vfio ioctl isn't needed at all. virtio PMD is only the >> consumer of pci_vfio_ioport_read. > My understanding is that using VFIO read/write ops is required for IOMMU > enabled case without cap_sys_rawio. And anyway, using inb/outb is just > bypassing VFIO. As I suggest in my other reply, it is better to document > that in the case of devices having PIO BARs, the user should consider > using UIO driver if performance is a concern. Get it. so user could read/write PIO using VFIO without iopl permission, with some performance penalty. >> we could consider if we still need pci_vfio_ioport_read related API in >> future. > I disagree. I think the pci_vfio_ioport_* API is required at least for > the IOMMU enabled case. > > Documentation is the way to go in my opinion, we can also add a warning > that performance may be degraded compared to UIO in > pci_vfio_ioport_map() when IOMMU is disabled if you think it may help > the users. > > Thanks, > Maxime > >> /huawei >>> Is performance the only issue to have your HW working with Virtio PMD, >>> or is this series also fixing some functionnal issues? >>> >>> Best regards, >>> Maxime >>>