[dpdk-dev] pci/uio: enable prefetchable resources mapping
Checks
Commit Message
For PCI prefetchable resources, Linux will create a
write combined file as well, the library will try
to map resourceX_wc file first, if the file does
not exist, then it will map resourceX as usual.
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
---
drivers/bus/pci/linux/pci_uio.c | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
Comments
On Thu, Feb 01, 2018 at 09:18:22AM +0800, Changpeng Liu wrote:
> For PCI prefetchable resources, Linux will create a
> write combined file as well, the library will try
> to map resourceX_wc file first, if the file does
> not exist, then it will map resourceX as usual.
>
> Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
> ---
> drivers/bus/pci/linux/pci_uio.c | 19 ++++++++++++++-----
> 1 file changed, 14 insertions(+), 5 deletions(-)
>
Hi,
Given the lack of ordering guarantees with write-combined memory, I
would have thought that this is very risky to do without a complete set
of changes inside the PMDs to add in the necessary memory barriers to
ensure ordering of operations to the BARs. Therefore, instead of
mapping one file or another, I think the change should be made to map
*both* in DPDK if available. Then each driver can chose whether to write
a given device register using uncacheable memory type or write-combining
memory type + any appropriate barriers.
For example, with many NICs the initialization of the device involves
many register writes in a pretty defined order, so wc operations are
probably to suitable as performance is not a concern. However, for data
path operations, a driver may chose to use wc memory for the occasional
device writes there, for performance reasons.
Regards,
/Bruce
On 02/01/2018 12:59 PM, Bruce Richardson wrote:
> On Thu, Feb 01, 2018 at 09:18:22AM +0800, Changpeng Liu wrote:
>> For PCI prefetchable resources, Linux will create a
>> write combined file as well, the library will try
>> to map resourceX_wc file first, if the file does
>> not exist, then it will map resourceX as usual.
>>
>> Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
>> ---
>> drivers/bus/pci/linux/pci_uio.c | 19 ++++++++++++++-----
>> 1 file changed, 14 insertions(+), 5 deletions(-)
>>
> Hi,
>
> Given the lack of ordering guarantees with write-combined memory, I
> would have thought that this is very risky to do without a complete set
> of changes inside the PMDs to add in the necessary memory barriers to
> ensure ordering of operations to the BARs. Therefore, instead of
> mapping one file or another, I think the change should be made to map
> *both* in DPDK if available. Then each driver can chose whether to write
> a given device register using uncacheable memory type or write-combining
> memory type + any appropriate barriers.
>
> For example, with many NICs the initialization of the device involves
> many register writes in a pretty defined order, so wc operations are
> probably to suitable as performance is not a concern. However, for data
> path operations, a driver may chose to use wc memory for the occasional
> device writes there, for performance reasons.
+1
I think so too that it would be useful to have both mappings available and
allow driver to choose which one to use.
@@ -293,7 +293,7 @@
/* update devname for mmap */
snprintf(devname, sizeof(devname),
- "%s/" PCI_PRI_FMT "/resource%d",
+ "%s/" PCI_PRI_FMT "/resource%d_wc",
rte_pci_get_sysfs_path(),
loc->domain, loc->bus, loc->devid,
loc->function, res_idx);
@@ -307,13 +307,22 @@
}
/*
- * open resource file, to mmap it
+ * open prefetchable resource file first, try to mmap it
*/
fd = open(devname, O_RDWR);
if (fd < 0) {
- RTE_LOG(ERR, EAL, "Cannot open %s: %s\n",
- devname, strerror(errno));
- goto error;
+ snprintf(devname, sizeof(devname),
+ "%s/" PCI_PRI_FMT "/resource%d",
+ rte_pci_get_sysfs_path(),
+ loc->domain, loc->bus, loc->devid,
+ loc->function, res_idx);
+ /* then try to map resource file */
+ fd = open(devname, O_RDWR);
+ if (fd < 0) {
+ RTE_LOG(ERR, EAL, "Cannot open %s: %s\n",
+ devname, strerror(errno));
+ goto error;
+ }
}
/* try mapping somewhere close to the end of hugepages */