drivers/bus: set device NUMA node to unknown by default
Checks
Commit Message
The dev->device.numa_node field is set by each bus driver for
every device it manages to indicate on which NUMA node this device lies.
When this information is unknown, the assigned value is not consistent
across the bus drivers.
Set the default value to SOCKET_ID_ANY (-1) by all bus drivers
when the NUMA information is unavailable.
Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
---
doc/guides/rel_notes/deprecation.rst | 7 -------
doc/guides/rel_notes/release_22_11.rst | 6 ++++++
drivers/bus/auxiliary/auxiliary_common.c | 8 ++------
drivers/bus/auxiliary/linux/auxiliary.c | 13 +++++--------
drivers/bus/dpaa/dpaa_bus.c | 1 +
drivers/bus/fslmc/fslmc_bus.c | 1 +
drivers/bus/pci/bsd/pci.c | 2 +-
drivers/bus/pci/linux/pci.c | 16 ++++++----------
drivers/bus/pci/pci_common.c | 8 ++------
drivers/bus/pci/windows/pci.c | 1 -
drivers/bus/vmbus/linux/vmbus_bus.c | 1 -
drivers/bus/vmbus/vmbus_common.c | 8 ++------
12 files changed, 26 insertions(+), 46 deletions(-)
Comments
On Thu, Sep 29, 2022 at 2:05 PM Olivier Matz <olivier.matz@6wind.com> wrote:
>
> The dev->device.numa_node field is set by each bus driver for
> every device it manages to indicate on which NUMA node this device lies.
>
> When this information is unknown, the assigned value is not consistent
> across the bus drivers.
>
> Set the default value to SOCKET_ID_ANY (-1) by all bus drivers
> when the NUMA information is unavailable.
>
> Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
- The dma/idxd driver, which has its own bus, seems ok, though I
would align its code for readability.
@@ -322,7 +322,7 @@ dsa_scan(void)
while ((wq = readdir(dev_dir)) != NULL) {
struct rte_dsa_device *dev;
- int numa_node = -1;
+ int numa_node = SOCKET_ID_ANY;
if (strncmp(wq->d_name, "wq", 2) != 0)
continue;
- There is an impact on upper classes of devices.
For ethdev, a port inherits the numa_node value from the rte_device object.
Yet, rte_eth_dev_socket_id() is described as:
* @return
* The NUMA socket ID to which the Ethernet device is connected or
* a default of zero if the socket could not be determined.
* -1 is returned is the port_id value is out of range.
On Fri, Sep 30, 2022 at 09:10:55AM +0200, David Marchand wrote:
> On Thu, Sep 29, 2022 at 2:05 PM Olivier Matz <olivier.matz@6wind.com> wrote:
> >
> > The dev->device.numa_node field is set by each bus driver for
> > every device it manages to indicate on which NUMA node this device lies.
> >
> > When this information is unknown, the assigned value is not consistent
> > across the bus drivers.
> >
> > Set the default value to SOCKET_ID_ANY (-1) by all bus drivers
> > when the NUMA information is unavailable.
> >
> > Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
>
> - The dma/idxd driver, which has its own bus, seems ok, though I
> would align its code for readability.
>
> @@ -322,7 +322,7 @@ dsa_scan(void)
>
> while ((wq = readdir(dev_dir)) != NULL) {
> struct rte_dsa_device *dev;
> - int numa_node = -1;
> + int numa_node = SOCKET_ID_ANY;
>
> if (strncmp(wq->d_name, "wq", 2) != 0)
> continue;
>
>
> - There is an impact on upper classes of devices.
>
> For ethdev, a port inherits the numa_node value from the rte_device object.
> Yet, rte_eth_dev_socket_id() is described as:
> * @return
> * The NUMA socket ID to which the Ethernet device is connected or
> * a default of zero if the socket could not be determined.
> * -1 is returned is the port_id value is out of range.
Good catches, I'll fix them in v2.
Thanks,
Olivier
@@ -36,13 +36,6 @@ Deprecation Notices
``__atomic_thread_fence`` must be used for patches that need to be merged in
20.08 onwards. This change will not introduce any performance degradation.
-* bus: The ``dev->device.numa_node`` field is set by each bus driver for
- every device it manages to indicate on which NUMA node this device lies.
- When this information is unknown, the assigned value is not consistent
- across the bus drivers.
- In DPDK 22.11, the default value will be set to -1 by all bus drivers
- when the NUMA information is unavailable.
-
* kni: The KNI kernel module and library are not recommended for use by new
applications - other technologies such as virtio-user are recommended instead.
Following the DPDK technical board
@@ -266,6 +266,12 @@ ABI Changes
* eventdev: Added ``weight`` and ``affinity`` fields
to ``rte_event_queue_conf`` structure.
+* bus: Changed the device numa node to -1 when NUMA information is unavailable.
+ The ``dev->device.numa_node`` field is set by each bus driver for
+ every device it manages to indicate on which NUMA node this device lies.
+ When this information is unknown, the assigned value was not consistent
+ across the bus drivers.
+
Known Issues
------------
@@ -105,12 +105,8 @@ rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv,
return -1;
}
- if (dev->device.numa_node < 0) {
- if (rte_socket_count() > 1)
- AUXILIARY_LOG(INFO, "Device %s is not NUMA-aware, defaulting socket to 0",
- dev->name);
- dev->device.numa_node = 0;
- }
+ if (dev->device.numa_node < 0 && rte_socket_count() > 1)
+ RTE_LOG(INFO, EAL, "Device %s is not NUMA-aware\n", dev->name);
iova_mode = rte_eal_iova_mode();
if ((drv->drv_flags & RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA) > 0 &&
@@ -40,14 +40,11 @@ auxiliary_scan_one(const char *dirname, const char *name)
/* Get NUMA node, default to 0 if not present */
snprintf(filename, sizeof(filename), "%s/%s/numa_node",
dirname, name);
- if (access(filename, F_OK) != -1) {
- if (eal_parse_sysfs_value(filename, &tmp) == 0)
- dev->device.numa_node = tmp;
- else
- dev->device.numa_node = -1;
- } else {
- dev->device.numa_node = 0;
- }
+ if (access(filename, F_OK) == 0 &&
+ eal_parse_sysfs_value(filename, &tmp) == 0)
+ dev->device.numa_node = tmp;
+ else
+ dev->device.numa_node = SOCKET_ID_ANY;
auxiliary_on_scan(dev);
@@ -179,6 +179,7 @@ dpaa_create_device_list(void)
}
dev->device.bus = &rte_dpaa_bus.bus;
+ dev->device.numa_node = SOCKET_ID_ANY;
/* Allocate interrupt handle instance */
dev->intr_handle =
@@ -156,6 +156,7 @@ scan_one_fslmc_device(char *dev_name)
}
dev->device.bus = &rte_fslmc_bus.bus;
+ dev->device.numa_node = SOCKET_ID_ANY;
/* Allocate interrupt instance */
dev->intr_handle =
@@ -246,7 +246,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)
dev->max_vfs = 0;
/* FreeBSD has no NUMA support (yet) */
- dev->device.numa_node = 0;
+ dev->device.numa_node = SOCKET_ID_ANY;
pci_common_set(dev);
@@ -283,17 +283,13 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)
}
/* get numa node, default to 0 if not present */
- snprintf(filename, sizeof(filename), "%s/numa_node",
- dirname);
+ snprintf(filename, sizeof(filename), "%s/numa_node", dirname);
- if (access(filename, F_OK) != -1) {
- if (eal_parse_sysfs_value(filename, &tmp) == 0)
- dev->device.numa_node = tmp;
- else
- dev->device.numa_node = -1;
- } else {
- dev->device.numa_node = 0;
- }
+ if (access(filename, F_OK) == 0 &&
+ eal_parse_sysfs_value(filename, &tmp) == 0)
+ dev->device.numa_node = tmp;
+ else
+ dev->device.numa_node = SOCKET_ID_ANY;
pci_common_set(dev);
@@ -235,12 +235,8 @@ rte_pci_probe_one_driver(struct rte_pci_driver *dr,
return 1;
}
- if (dev->device.numa_node < 0) {
- if (rte_socket_count() > 1)
- RTE_LOG(INFO, EAL, "Device %s is not NUMA-aware, defaulting socket to 0\n",
- dev->name);
- dev->device.numa_node = 0;
- }
+ if (dev->device.numa_node < 0 && rte_socket_count() > 1)
+ RTE_LOG(INFO, EAL, "Device %s is not NUMA-aware\n", dev->name);
already_probed = rte_dev_is_probed(&dev->device);
if (already_probed && !(dr->drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
@@ -249,7 +249,6 @@ get_device_resource_info(HDEVINFO dev_info,
DWORD error = GetLastError();
if (error == ERROR_NOT_FOUND) {
/* On older CPUs, NUMA is not bound to PCIe locality. */
- dev->device.numa_node = 0;
return ERROR_SUCCESS;
}
RTE_LOG_WIN32_ERR("SetupDiGetDevicePropertyW"
@@ -293,7 +293,6 @@ vmbus_scan_one(const char *name)
goto error;
dev->device.numa_node = tmp;
} else {
- /* if no NUMA support, set default to 0 */
dev->device.numa_node = SOCKET_ID_ANY;
}
@@ -111,12 +111,8 @@ vmbus_probe_one_driver(struct rte_vmbus_driver *dr,
/* reference driver structure */
dev->driver = dr;
- if (dev->device.numa_node < 0) {
- if (rte_socket_count() > 1)
- VMBUS_LOG(INFO, "Device %s is not NUMA-aware, defaulting socket to 0",
- guid);
- dev->device.numa_node = 0;
- }
+ if (dev->device.numa_node < 0 && rte_socket_count() > 1)
+ VMBUS_LOG(INFO, "Device %s is not NUMA-aware", guid);
/* call the driver probe() function */
VMBUS_LOG(INFO, " probe driver: %s", dr->driver.name);