diff mbox series

[v8,2/2] bus/auxiliary: introduce auxiliary bus

Message ID 20210705064512.13116-2-xuemingl@nvidia.com (mailing list archive)
State Accepted, archived
Delegated to: Thomas Monjalon
Headers show
Series None | expand

Checks

Context Check Description
ci/Intel-compilation warning apply issues
ci/iol-testing warning apply patch failure
ci/checkpatch warning coding style issues

Commit Message

Xueming(Steven) Li July 5, 2021, 6:45 a.m. UTC
Auxiliary bus [1] provides a way to split function into child-devices
representing sub-domains of functionality. Each auxiliary device
represents a part of its parent functionality.

Auxiliary device is identified by unique device name, sysfs path:
  /sys/bus/auxiliary/devices/<name>

Devargs legacy syntax of auxiliary device:
  -a auxiliary:<name>[,args...]
Devargs generic syntax of auxiliary device:
  -a bus=auxiliary,name=<name>/class=<class>/driver=<driver>[,args...]

[1] kernel auxiliary bus document:
https://www.kernel.org/doc/html/latest/driver-api/auxiliary_bus.html

Signed-off-by: Xueming Li <xuemingl@nvidia.com>
Cc: Wang Haiyue <haiyue.wang@intel.com>
Cc: Thomas Monjalon <thomas@monjalon.net>
Cc: Kinsella Ray <mdr@ashroe.eu>
Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
---
 MAINTAINERS                               |   5 +
 doc/guides/rel_notes/release_21_08.rst    |   6 +
 drivers/bus/auxiliary/auxiliary_common.c  | 411 ++++++++++++++++++++++
 drivers/bus/auxiliary/auxiliary_params.c  |  59 ++++
 drivers/bus/auxiliary/linux/auxiliary.c   | 141 ++++++++
 drivers/bus/auxiliary/meson.build         |  16 +
 drivers/bus/auxiliary/private.h           |  91 +++++
 drivers/bus/auxiliary/rte_bus_auxiliary.h | 193 ++++++++++
 drivers/bus/auxiliary/version.map         |   7 +
 drivers/bus/meson.build                   |   1 +
 10 files changed, 930 insertions(+)
 create mode 100644 drivers/bus/auxiliary/auxiliary_common.c
 create mode 100644 drivers/bus/auxiliary/auxiliary_params.c
 create mode 100644 drivers/bus/auxiliary/linux/auxiliary.c
 create mode 100644 drivers/bus/auxiliary/meson.build
 create mode 100644 drivers/bus/auxiliary/private.h
 create mode 100644 drivers/bus/auxiliary/rte_bus_auxiliary.h
 create mode 100644 drivers/bus/auxiliary/version.map

Comments

Andrew Rybchenko July 5, 2021, 9:19 a.m. UTC | #1
On 7/5/21 9:45 AM, Xueming Li wrote:
> Auxiliary bus [1] provides a way to split function into child-devices
> representing sub-domains of functionality. Each auxiliary device
> represents a part of its parent functionality.
> 
> Auxiliary device is identified by unique device name, sysfs path:
>   /sys/bus/auxiliary/devices/<name>
> 
> Devargs legacy syntax of auxiliary device:
>   -a auxiliary:<name>[,args...]
> Devargs generic syntax of auxiliary device:
>   -a bus=auxiliary,name=<name>/class=<class>/driver=<driver>[,args...]
> 
> [1] kernel auxiliary bus document:
> https://www.kernel.org/doc/html/latest/driver-api/auxiliary_bus.html
> 
> Signed-off-by: Xueming Li <xuemingl@nvidia.com>
> Cc: Wang Haiyue <haiyue.wang@intel.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: Kinsella Ray <mdr@ashroe.eu>
> Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>

I still don't understand if we really need to make the API
a part of stable API/ABI in the future. Can it be internal?
Xueming(Steven) Li July 5, 2021, 9:30 a.m. UTC | #2
Hi Andrew,

> -----Original Message-----
> From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> Sent: Monday, July 5, 2021 5:19 PM
> To: Xueming(Steven) Li <xuemingl@nvidia.com>
> Cc: dev@dpdk.org; Wang Haiyue <haiyue.wang@intel.com>; NBU-Contact-Thomas Monjalon <thomas@monjalon.net>; Kinsella Ray
> <mdr@ashroe.eu>; Parav Pandit <parav@nvidia.com>; Neil Horman <nhorman@tuxdriver.com>
> Subject: Re: [PATCH v8 2/2] bus/auxiliary: introduce auxiliary bus
> 
> On 7/5/21 9:45 AM, Xueming Li wrote:
> > Auxiliary bus [1] provides a way to split function into child-devices
> > representing sub-domains of functionality. Each auxiliary device
> > represents a part of its parent functionality.
> >
> > Auxiliary device is identified by unique device name, sysfs path:
> >   /sys/bus/auxiliary/devices/<name>
> >
> > Devargs legacy syntax of auxiliary device:
> >   -a auxiliary:<name>[,args...]
> > Devargs generic syntax of auxiliary device:
> >   -a bus=auxiliary,name=<name>/class=<class>/driver=<driver>[,args...]
> >
> > [1] kernel auxiliary bus document:
> > https://www.kernel.org/doc/html/latest/driver-api/auxiliary_bus.html
> >
> > Signed-off-by: Xueming Li <xuemingl@nvidia.com>
> > Cc: Wang Haiyue <haiyue.wang@intel.com>
> > Cc: Thomas Monjalon <thomas@monjalon.net>
> > Cc: Kinsella Ray <mdr@ashroe.eu>
> > Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> 
> I still don't understand if we really need to make the API a part of stable API/ABI in the future. Can it be internal?

There was some discussion on this with Thomas in earlier version. Users might want to register/unregister their own PMD driver,
Is this a valid scenario?
Andrew Rybchenko July 5, 2021, 9:35 a.m. UTC | #3
On 7/5/21 12:30 PM, Xueming(Steven) Li wrote:
> Hi Andrew,
> 
>> -----Original Message-----
>> From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
>> Sent: Monday, July 5, 2021 5:19 PM
>> To: Xueming(Steven) Li <xuemingl@nvidia.com>
>> Cc: dev@dpdk.org; Wang Haiyue <haiyue.wang@intel.com>; NBU-Contact-Thomas Monjalon <thomas@monjalon.net>; Kinsella Ray
>> <mdr@ashroe.eu>; Parav Pandit <parav@nvidia.com>; Neil Horman <nhorman@tuxdriver.com>
>> Subject: Re: [PATCH v8 2/2] bus/auxiliary: introduce auxiliary bus
>>
>> On 7/5/21 9:45 AM, Xueming Li wrote:
>>> Auxiliary bus [1] provides a way to split function into child-devices
>>> representing sub-domains of functionality. Each auxiliary device
>>> represents a part of its parent functionality.
>>>
>>> Auxiliary device is identified by unique device name, sysfs path:
>>>   /sys/bus/auxiliary/devices/<name>
>>>
>>> Devargs legacy syntax of auxiliary device:
>>>   -a auxiliary:<name>[,args...]
>>> Devargs generic syntax of auxiliary device:
>>>   -a bus=auxiliary,name=<name>/class=<class>/driver=<driver>[,args...]
>>>
>>> [1] kernel auxiliary bus document:
>>> https://www.kernel.org/doc/html/latest/driver-api/auxiliary_bus.html
>>>
>>> Signed-off-by: Xueming Li <xuemingl@nvidia.com>
>>> Cc: Wang Haiyue <haiyue.wang@intel.com>
>>> Cc: Thomas Monjalon <thomas@monjalon.net>
>>> Cc: Kinsella Ray <mdr@ashroe.eu>
>>> Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
>>
>> I still don't understand if we really need to make the API a part of stable API/ABI in the future. Can it be internal?
> 
> There was some discussion on this with Thomas in earlier version.
> Users might want to register/unregister their own PMD driver,
> Is this a valid scenario?

Yes, it is true, but should DPDK care that much about
out-of-tree drivers. I'm just asking since don't know
techboard position on it.
Thomas Monjalon July 5, 2021, 2:57 p.m. UTC | #4
05/07/2021 11:35, Andrew Rybchenko:
> On 7/5/21 12:30 PM, Xueming(Steven) Li wrote:
> > From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> >> I still don't understand if we really need to make the API a part of stable API/ABI in the future. Can it be internal?
> > 
> > There was some discussion on this with Thomas in earlier version.
> > Users might want to register/unregister their own PMD driver,
> > Is this a valid scenario?
> 
> Yes, it is true, but should DPDK care that much about
> out-of-tree drivers. I'm just asking since don't know
> techboard position on it.

I think there is a consensus to allow out-of-tree drivers
without any compatibility commitment.

Some other bus drivers are exporting some API like in this patch.
We could discuss again in techboard what to make internal.
If it is decided to hide buses API, we could change all bus drivers
later in DPDK 21.11.
Andrew Rybchenko July 5, 2021, 3:06 p.m. UTC | #5
On 7/5/21 5:57 PM, Thomas Monjalon wrote:
> 05/07/2021 11:35, Andrew Rybchenko:
>> On 7/5/21 12:30 PM, Xueming(Steven) Li wrote:
>>> From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
>>>> I still don't understand if we really need to make the API a part of stable API/ABI in the future. Can it be internal?
>>>
>>> There was some discussion on this with Thomas in earlier version.
>>> Users might want to register/unregister their own PMD driver,
>>> Is this a valid scenario?
>>
>> Yes, it is true, but should DPDK care that much about
>> out-of-tree drivers. I'm just asking since don't know
>> techboard position on it.
> 
> I think there is a consensus to allow out-of-tree drivers
> without any compatibility commitment.
> 
> Some other bus drivers are exporting some API like in this patch.
> We could discuss again in techboard what to make internal.
> If it is decided to hide buses API, we could change all bus drivers
> later in DPDK 21.11.

OK, thanks.
Thomas Monjalon July 5, 2021, 4:47 p.m. UTC | #6
05/07/2021 08:45, Xueming Li:
> Auxiliary bus [1] provides a way to split function into child-devices
> representing sub-domains of functionality. Each auxiliary device
> represents a part of its parent functionality.
> 
> Auxiliary device is identified by unique device name, sysfs path:
>   /sys/bus/auxiliary/devices/<name>
> 
> Devargs legacy syntax of auxiliary device:
>   -a auxiliary:<name>[,args...]
> Devargs generic syntax of auxiliary device:
>   -a bus=auxiliary,name=<name>/class=<class>/driver=<driver>[,args...]
> 
> [1] kernel auxiliary bus document:
> https://www.kernel.org/doc/html/latest/driver-api/auxiliary_bus.html
> 
> Signed-off-by: Xueming Li <xuemingl@nvidia.com>
> Cc: Wang Haiyue <haiyue.wang@intel.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>
> Cc: Kinsella Ray <mdr@ashroe.eu>
> Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>

Acked-by: Thomas Monjalon <thomas@monjalon.net>

Applied, thanks
diff mbox series

Patch

diff --git a/MAINTAINERS b/MAINTAINERS
index 5877a16971..9b0d0d4348 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -525,6 +525,11 @@  F: doc/guides/mempool/octeontx2.rst
 Bus Drivers
 -----------
 
+Auxiliary bus driver - EXPERIMENTAL
+M: Parav Pandit <parav@nvidia.com>
+M: Xueming Li <xuemingl@nvidia.com>
+F: drivers/bus/auxiliary/
+
 Intel FPGA bus
 M: Rosen Xu <rosen.xu@intel.com>
 F: drivers/bus/ifpga/
diff --git a/doc/guides/rel_notes/release_21_08.rst b/doc/guides/rel_notes/release_21_08.rst
index a6ecfdf3ce..e7ef4c8a05 100644
--- a/doc/guides/rel_notes/release_21_08.rst
+++ b/doc/guides/rel_notes/release_21_08.rst
@@ -55,6 +55,12 @@  New Features
      Also, make sure to start the actual text at the margin.
      =======================================================
 
+* **Added auxiliary bus support.**
+
+  Auxiliary bus provides a way to split function into child-devices
+  representing sub-domains of functionality. Each auxiliary device
+  represents a part of its parent functionality.
+
 
 Removed Items
 -------------
diff --git a/drivers/bus/auxiliary/auxiliary_common.c b/drivers/bus/auxiliary/auxiliary_common.c
new file mode 100644
index 0000000000..abe6aef098
--- /dev/null
+++ b/drivers/bus/auxiliary/auxiliary_common.c
@@ -0,0 +1,411 @@ 
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#include <string.h>
+#include <inttypes.h>
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <sys/queue.h>
+#include <rte_errno.h>
+#include <rte_interrupts.h>
+#include <rte_log.h>
+#include <rte_bus.h>
+#include <rte_per_lcore.h>
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_eal_paging.h>
+#include <rte_string_fns.h>
+#include <rte_common.h>
+#include <rte_devargs.h>
+
+#include "private.h"
+#include "rte_bus_auxiliary.h"
+
+static struct rte_devargs *
+auxiliary_devargs_lookup(const char *name)
+{
+	struct rte_devargs *devargs;
+
+	RTE_EAL_DEVARGS_FOREACH(RTE_BUS_AUXILIARY_NAME, devargs) {
+		if (strcmp(devargs->name, name) == 0)
+			return devargs;
+	}
+	return NULL;
+}
+
+/*
+ * Test whether the auxiliary device exist.
+ *
+ * Stub for OS not supporting auxiliary bus.
+ */
+__rte_weak bool
+auxiliary_dev_exists(const char *name)
+{
+	RTE_SET_USED(name);
+	return false;
+}
+
+/*
+ * Scan the devices in the auxiliary bus.
+ *
+ * Stub for OS not supporting auxiliary bus.
+ */
+__rte_weak int
+auxiliary_scan(void)
+{
+	return 0;
+}
+
+/*
+ * Update a device's devargs being scanned.
+ *
+ * @param aux_dev
+ *	AUXILIARY device.
+ */
+void
+auxiliary_on_scan(struct rte_auxiliary_device *aux_dev)
+{
+	aux_dev->device.devargs = auxiliary_devargs_lookup(aux_dev->name);
+}
+
+/*
+ * Match the auxiliary driver and device using driver function.
+ */
+bool
+auxiliary_match(const struct rte_auxiliary_driver *aux_drv,
+		const struct rte_auxiliary_device *aux_dev)
+{
+	if (aux_drv->match == NULL)
+		return false;
+	return aux_drv->match(aux_dev->name);
+}
+
+/*
+ * Call the probe() function of the driver.
+ */
+static int
+rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv,
+			       struct rte_auxiliary_device *dev)
+{
+	enum rte_iova_mode iova_mode;
+	int ret;
+
+	if (drv == NULL || dev == NULL)
+		return -EINVAL;
+
+	/* Check if driver supports it. */
+	if (!auxiliary_match(drv, dev))
+		/* Match of device and driver failed */
+		return 1;
+
+	/* No initialization when marked as blocked, return without error. */
+	if (dev->device.devargs != NULL &&
+	    dev->device.devargs->policy == RTE_DEV_BLOCKED) {
+		AUXILIARY_LOG(INFO, "Device is blocked, not initializing");
+		return -1;
+	}
+
+	if (dev->device.numa_node < 0) {
+		AUXILIARY_LOG(INFO, "Device is not NUMA-aware, defaulting NUMA node to 0");
+		dev->device.numa_node = 0;
+	}
+
+	iova_mode = rte_eal_iova_mode();
+	if ((drv->drv_flags & RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA) > 0 &&
+	    iova_mode != RTE_IOVA_VA) {
+		AUXILIARY_LOG(ERR, "Driver %s expecting VA IOVA mode but current mode is PA, not initializing",
+			      drv->driver.name);
+		return -EINVAL;
+	}
+
+	dev->driver = drv;
+
+	AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (NUMA node %i)",
+		      drv->driver.name, dev->name, dev->device.numa_node);
+	ret = drv->probe(drv, dev);
+	if (ret != 0)
+		dev->driver = NULL;
+	else
+		dev->device.driver = &drv->driver;
+
+	return ret;
+}
+
+/*
+ * Call the remove() function of the driver.
+ */
+static int
+rte_auxiliary_driver_remove_dev(struct rte_auxiliary_device *dev)
+{
+	struct rte_auxiliary_driver *drv;
+	int ret = 0;
+
+	if (dev == NULL)
+		return -EINVAL;
+
+	drv = dev->driver;
+
+	AUXILIARY_LOG(DEBUG, "Driver %s remove auxiliary device %s on NUMA node %i",
+		      drv->driver.name, dev->name, dev->device.numa_node);
+
+	if (drv->remove != NULL) {
+		ret = drv->remove(dev);
+		if (ret < 0)
+			return ret;
+	}
+
+	/* clear driver structure */
+	dev->driver = NULL;
+	dev->device.driver = NULL;
+
+	return 0;
+}
+
+/*
+ * Call the probe() function of all registered driver for the given device.
+ * Return < 0 if initialization failed.
+ * Return 1 if no driver is found for this device.
+ */
+static int
+auxiliary_probe_all_drivers(struct rte_auxiliary_device *dev)
+{
+	struct rte_auxiliary_driver *drv;
+	int rc;
+
+	if (dev == NULL)
+		return -EINVAL;
+
+	FOREACH_DRIVER_ON_AUXILIARY_BUS(drv) {
+		if (!drv->match(dev->name))
+			continue;
+
+		rc = rte_auxiliary_probe_one_driver(drv, dev);
+		if (rc < 0)
+			/* negative value is an error */
+			return rc;
+		if (rc > 0)
+			/* positive value means driver doesn't support it */
+			continue;
+		return 0;
+	}
+	return 1;
+}
+
+/*
+ * Scan the content of the auxiliary bus, and call the probe function for
+ * all registered drivers to try to probe discovered devices.
+ */
+static int
+auxiliary_probe(void)
+{
+	struct rte_auxiliary_device *dev = NULL;
+	size_t probed = 0, failed = 0;
+	int ret = 0;
+
+	FOREACH_DEVICE_ON_AUXILIARY_BUS(dev) {
+		probed++;
+
+		ret = auxiliary_probe_all_drivers(dev);
+		if (ret < 0) {
+			if (ret != -EEXIST) {
+				AUXILIARY_LOG(ERR, "Requested device %s cannot be used",
+					      dev->name);
+				rte_errno = errno;
+				failed++;
+			}
+			ret = 0;
+		}
+	}
+
+	return (probed && probed == failed) ? -1 : 0;
+}
+
+static int
+auxiliary_parse(const char *name, void *addr)
+{
+	struct rte_auxiliary_driver *drv = NULL;
+	const char **out = addr;
+
+	/* Allow empty device name "auxiliary:" to bypass entire bus scan. */
+	if (strlen(name) == 0)
+		return 0;
+
+	FOREACH_DRIVER_ON_AUXILIARY_BUS(drv) {
+		if (drv->match(name))
+			break;
+	}
+	if (drv != NULL && addr != NULL)
+		*out = name;
+	return drv != NULL ? 0 : -1;
+}
+
+/* Register a driver */
+void
+rte_auxiliary_register(struct rte_auxiliary_driver *driver)
+{
+	TAILQ_INSERT_TAIL(&auxiliary_bus.driver_list, driver, next);
+	driver->bus = &auxiliary_bus;
+}
+
+/* Unregister a driver */
+void
+rte_auxiliary_unregister(struct rte_auxiliary_driver *driver)
+{
+	TAILQ_REMOVE(&auxiliary_bus.driver_list, driver, next);
+	driver->bus = NULL;
+}
+
+/* Add a device to auxiliary bus */
+void
+auxiliary_add_device(struct rte_auxiliary_device *aux_dev)
+{
+	TAILQ_INSERT_TAIL(&auxiliary_bus.device_list, aux_dev, next);
+}
+
+/* Insert a device into a predefined position in auxiliary bus */
+void
+auxiliary_insert_device(struct rte_auxiliary_device *exist_aux_dev,
+			struct rte_auxiliary_device *new_aux_dev)
+{
+	TAILQ_INSERT_BEFORE(exist_aux_dev, new_aux_dev, next);
+}
+
+/* Remove a device from auxiliary bus */
+static void
+rte_auxiliary_remove_device(struct rte_auxiliary_device *auxiliary_dev)
+{
+	TAILQ_REMOVE(&auxiliary_bus.device_list, auxiliary_dev, next);
+}
+
+static struct rte_device *
+auxiliary_find_device(const struct rte_device *start, rte_dev_cmp_t cmp,
+		      const void *data)
+{
+	const struct rte_auxiliary_device *pstart;
+	struct rte_auxiliary_device *adev;
+
+	if (start != NULL) {
+		pstart = RTE_DEV_TO_AUXILIARY_CONST(start);
+		adev = TAILQ_NEXT(pstart, next);
+	} else {
+		adev = TAILQ_FIRST(&auxiliary_bus.device_list);
+	}
+	while (adev != NULL) {
+		if (cmp(&adev->device, data) == 0)
+			return &adev->device;
+		adev = TAILQ_NEXT(adev, next);
+	}
+	return NULL;
+}
+
+static int
+auxiliary_plug(struct rte_device *dev)
+{
+	if (!auxiliary_dev_exists(dev->name))
+		return -ENOENT;
+	return auxiliary_probe_all_drivers(RTE_DEV_TO_AUXILIARY(dev));
+}
+
+static int
+auxiliary_unplug(struct rte_device *dev)
+{
+	struct rte_auxiliary_device *adev;
+	int ret;
+
+	adev = RTE_DEV_TO_AUXILIARY(dev);
+	ret = rte_auxiliary_driver_remove_dev(adev);
+	if (ret == 0) {
+		rte_auxiliary_remove_device(adev);
+		rte_devargs_remove(dev->devargs);
+		free(adev);
+	}
+	return ret;
+}
+
+static int
+auxiliary_dma_map(struct rte_device *dev, void *addr, uint64_t iova, size_t len)
+{
+	struct rte_auxiliary_device *aux_dev = RTE_DEV_TO_AUXILIARY(dev);
+
+	if (dev == NULL || aux_dev->driver == NULL) {
+		rte_errno = EINVAL;
+		return -1;
+	}
+	if (aux_dev->driver->dma_map == NULL) {
+		rte_errno = ENOTSUP;
+		return -1;
+	}
+	return aux_dev->driver->dma_map(aux_dev, addr, iova, len);
+}
+
+static int
+auxiliary_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova,
+		    size_t len)
+{
+	struct rte_auxiliary_device *aux_dev = RTE_DEV_TO_AUXILIARY(dev);
+
+	if (dev == NULL || aux_dev->driver == NULL) {
+		rte_errno = EINVAL;
+		return -1;
+	}
+	if (aux_dev->driver->dma_unmap == NULL) {
+		rte_errno = ENOTSUP;
+		return -1;
+	}
+	return aux_dev->driver->dma_unmap(aux_dev, addr, iova, len);
+}
+
+bool
+auxiliary_is_ignored_device(const char *name)
+{
+	struct rte_devargs *devargs = auxiliary_devargs_lookup(name);
+
+	switch (auxiliary_bus.bus.conf.scan_mode) {
+	case RTE_BUS_SCAN_ALLOWLIST:
+		if (devargs && devargs->policy == RTE_DEV_ALLOWED)
+			return false;
+		break;
+	case RTE_BUS_SCAN_UNDEFINED:
+	case RTE_BUS_SCAN_BLOCKLIST:
+		if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED)
+			return false;
+		break;
+	}
+	return true;
+}
+
+static enum rte_iova_mode
+auxiliary_get_iommu_class(void)
+{
+	const struct rte_auxiliary_driver *drv;
+
+	FOREACH_DRIVER_ON_AUXILIARY_BUS(drv) {
+		if ((drv->drv_flags & RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA) > 0)
+			return RTE_IOVA_VA;
+	}
+
+	return RTE_IOVA_DC;
+}
+
+struct rte_auxiliary_bus auxiliary_bus = {
+	.bus = {
+		.scan = auxiliary_scan,
+		.probe = auxiliary_probe,
+		.find_device = auxiliary_find_device,
+		.plug = auxiliary_plug,
+		.unplug = auxiliary_unplug,
+		.parse = auxiliary_parse,
+		.dma_map = auxiliary_dma_map,
+		.dma_unmap = auxiliary_dma_unmap,
+		.get_iommu_class = auxiliary_get_iommu_class,
+		.dev_iterate = auxiliary_dev_iterate,
+	},
+	.device_list = TAILQ_HEAD_INITIALIZER(auxiliary_bus.device_list),
+	.driver_list = TAILQ_HEAD_INITIALIZER(auxiliary_bus.driver_list),
+};
+
+RTE_REGISTER_BUS(auxiliary, auxiliary_bus.bus);
+RTE_LOG_REGISTER_DEFAULT(auxiliary_bus_logtype, NOTICE);
diff --git a/drivers/bus/auxiliary/auxiliary_params.c b/drivers/bus/auxiliary/auxiliary_params.c
new file mode 100644
index 0000000000..cd3fa56cb4
--- /dev/null
+++ b/drivers/bus/auxiliary/auxiliary_params.c
@@ -0,0 +1,59 @@ 
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#include <string.h>
+
+#include <rte_bus.h>
+#include <rte_dev.h>
+#include <rte_errno.h>
+#include <rte_kvargs.h>
+
+#include "private.h"
+#include "rte_bus_auxiliary.h"
+
+enum auxiliary_params {
+	RTE_AUXILIARY_PARAM_NAME,
+};
+
+static const char * const auxiliary_params_keys[] = {
+	[RTE_AUXILIARY_PARAM_NAME] = "name",
+};
+
+static int
+auxiliary_dev_match(const struct rte_device *dev,
+	      const void *_kvlist)
+{
+	const struct rte_kvargs *kvlist = _kvlist;
+	int ret;
+
+	ret = rte_kvargs_process(kvlist,
+			auxiliary_params_keys[RTE_AUXILIARY_PARAM_NAME],
+			rte_kvargs_strcmp, (void *)(uintptr_t)dev->name);
+
+	return ret != 0 ? -1 : 0;
+}
+
+void *
+auxiliary_dev_iterate(const void *start,
+		    const char *str,
+		    const struct rte_dev_iterator *it __rte_unused)
+{
+	rte_bus_find_device_t find_device;
+	struct rte_kvargs *kvargs = NULL;
+	struct rte_device *dev;
+
+	if (str != NULL) {
+		kvargs = rte_kvargs_parse(str, auxiliary_params_keys);
+		if (kvargs == NULL) {
+			AUXILIARY_LOG(ERR, "cannot parse argument list %s",
+				      str);
+			rte_errno = EINVAL;
+			return NULL;
+		}
+	}
+	find_device = auxiliary_bus.bus.find_device;
+	dev = find_device(start, auxiliary_dev_match, kvargs);
+	rte_kvargs_free(kvargs);
+	return dev;
+}
diff --git a/drivers/bus/auxiliary/linux/auxiliary.c b/drivers/bus/auxiliary/linux/auxiliary.c
new file mode 100644
index 0000000000..9bd4ee3295
--- /dev/null
+++ b/drivers/bus/auxiliary/linux/auxiliary.c
@@ -0,0 +1,141 @@ 
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#include <string.h>
+#include <dirent.h>
+
+#include <rte_log.h>
+#include <rte_bus.h>
+#include <rte_malloc.h>
+#include <rte_devargs.h>
+#include <rte_memcpy.h>
+#include <eal_filesystem.h>
+
+#include "../rte_bus_auxiliary.h"
+#include "../private.h"
+
+#define AUXILIARY_SYSFS_PATH "/sys/bus/auxiliary/devices"
+
+/* Scan one auxiliary sysfs entry, and fill the devices list from it. */
+static int
+auxiliary_scan_one(const char *dirname, const char *name)
+{
+	struct rte_auxiliary_device *dev;
+	struct rte_auxiliary_device *dev2;
+	char filename[PATH_MAX];
+	unsigned long tmp;
+	int ret;
+
+	dev = malloc(sizeof(*dev));
+	if (dev == NULL)
+		return -1;
+
+	memset(dev, 0, sizeof(*dev));
+	if (rte_strscpy(dev->name, name, sizeof(dev->name)) < 0) {
+		free(dev);
+		return -1;
+	}
+	dev->device.name = dev->name;
+	dev->device.bus = &auxiliary_bus.bus;
+
+	/* Get NUMA node, default to 0 if not present */
+	snprintf(filename, sizeof(filename), "%s/%s/numa_node",
+		 dirname, name);
+	if (access(filename, F_OK) != -1) {
+		if (eal_parse_sysfs_value(filename, &tmp) == 0)
+			dev->device.numa_node = tmp;
+		else
+			dev->device.numa_node = -1;
+	} else {
+		dev->device.numa_node = 0;
+	}
+
+	auxiliary_on_scan(dev);
+
+	/* Device is valid, add in list (sorted) */
+	TAILQ_FOREACH(dev2, &auxiliary_bus.device_list, next) {
+		ret = strcmp(dev->name, dev2->name);
+		if (ret > 0)
+			continue;
+		if (ret < 0) {
+			auxiliary_insert_device(dev2, dev);
+		} else { /* already registered */
+			if (rte_dev_is_probed(&dev2->device) &&
+			    dev2->device.devargs != dev->device.devargs) {
+				/* To probe device with new devargs. */
+				rte_devargs_remove(dev2->device.devargs);
+				auxiliary_on_scan(dev2);
+			}
+			free(dev);
+		}
+		return 0;
+	}
+	auxiliary_add_device(dev);
+	return 0;
+}
+
+/*
+ * Test whether the auxiliary device exist.
+ */
+bool
+auxiliary_dev_exists(const char *name)
+{
+	DIR *dir;
+	char dirname[PATH_MAX];
+
+	snprintf(dirname, sizeof(dirname), "%s/%s",
+		 AUXILIARY_SYSFS_PATH, name);
+	dir = opendir(dirname);
+	if (dir == NULL)
+		return false;
+	closedir(dir);
+	return true;
+}
+
+/*
+ * Scan the devices in the auxiliary bus.
+ */
+int
+auxiliary_scan(void)
+{
+	struct dirent *e;
+	DIR *dir;
+	char dirname[PATH_MAX];
+	struct rte_auxiliary_driver *drv;
+
+	dir = opendir(AUXILIARY_SYSFS_PATH);
+	if (dir == NULL) {
+		AUXILIARY_LOG(INFO, "%s not found, is auxiliary module loaded?",
+			      AUXILIARY_SYSFS_PATH);
+		return 0;
+	}
+
+	while ((e = readdir(dir)) != NULL) {
+		if (e->d_name[0] == '.')
+			continue;
+
+		if (auxiliary_is_ignored_device(e->d_name))
+			continue;
+
+		snprintf(dirname, sizeof(dirname), "%s/%s",
+			 AUXILIARY_SYSFS_PATH, e->d_name);
+
+		/* Ignore if no driver can handle. */
+		FOREACH_DRIVER_ON_AUXILIARY_BUS(drv) {
+			if (drv->match(e->d_name))
+				break;
+		}
+		if (drv == NULL)
+			continue;
+
+		if (auxiliary_scan_one(dirname, e->d_name) < 0)
+			goto error;
+	}
+	closedir(dir);
+	return 0;
+
+error:
+	closedir(dir);
+	return -1;
+}
diff --git a/drivers/bus/auxiliary/meson.build b/drivers/bus/auxiliary/meson.build
new file mode 100644
index 0000000000..357550eff7
--- /dev/null
+++ b/drivers/bus/auxiliary/meson.build
@@ -0,0 +1,16 @@ 
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 NVIDIA Corporation & Affiliates
+
+headers = files(
+        'rte_bus_auxiliary.h',
+)
+sources = files(
+        'auxiliary_common.c',
+        'auxiliary_params.c',
+)
+if is_linux
+    sources += files(
+        'linux/auxiliary.c',
+    )
+endif
+deps += ['kvargs']
diff --git a/drivers/bus/auxiliary/private.h b/drivers/bus/auxiliary/private.h
new file mode 100644
index 0000000000..9987e8b501
--- /dev/null
+++ b/drivers/bus/auxiliary/private.h
@@ -0,0 +1,91 @@ 
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#ifndef BUS_AUXILIARY_PRIVATE_H
+#define BUS_AUXILIARY_PRIVATE_H
+
+#include <stdbool.h>
+#include <stdio.h>
+
+#include "rte_bus_auxiliary.h"
+
+extern struct rte_auxiliary_bus auxiliary_bus;
+extern int auxiliary_bus_logtype;
+
+#define AUXILIARY_LOG(level, ...) \
+	rte_log(RTE_LOG_ ## level, auxiliary_bus_logtype, \
+		RTE_FMT("auxiliary bus: " RTE_FMT_HEAD(__VA_ARGS__,) "\n", \
+			RTE_FMT_TAIL(__VA_ARGS__,)))
+
+/* Auxiliary bus iterators */
+#define FOREACH_DEVICE_ON_AUXILIARY_BUS(p) \
+		TAILQ_FOREACH(p, &(auxiliary_bus.device_list), next)
+
+#define FOREACH_DRIVER_ON_AUXILIARY_BUS(p) \
+		TAILQ_FOREACH(p, &(auxiliary_bus.driver_list), next)
+
+/* List of auxiliary devices. */
+TAILQ_HEAD(rte_auxiliary_device_list, rte_auxiliary_device);
+/* List of auxiliary drivers. */
+TAILQ_HEAD(rte_auxiliary_driver_list, rte_auxiliary_driver);
+
+/*
+ * Structure describing the auxiliary bus
+ */
+struct rte_auxiliary_bus {
+	struct rte_bus bus;                  /* Inherit the generic class */
+	struct rte_auxiliary_device_list device_list;  /* List of devices */
+	struct rte_auxiliary_driver_list driver_list;  /* List of drivers */
+};
+
+/*
+ * Test whether the auxiliary device exist.
+ */
+bool auxiliary_dev_exists(const char *name);
+
+/*
+ * Scan the content of the auxiliary bus, and the devices in the devices
+ * list.
+ */
+int auxiliary_scan(void);
+
+/*
+ * Update a device being scanned.
+ */
+void auxiliary_on_scan(struct rte_auxiliary_device *aux_dev);
+
+/*
+ * Validate whether a device with given auxiliary device should be ignored
+ * or not.
+ */
+bool auxiliary_is_ignored_device(const char *name);
+
+/*
+ * Add an auxiliary device to the auxiliary bus (append to auxiliary device
+ * list). This function also updates the bus references of the auxiliary
+ * device and the generic device object embedded within.
+ */
+void auxiliary_add_device(struct rte_auxiliary_device *aux_dev);
+
+/*
+ * Insert an auxiliary device in the auxiliary bus at a particular location
+ * in the device list. It also updates the auxiliary bus reference of the
+ * new devices to be inserted.
+ */
+void auxiliary_insert_device(struct rte_auxiliary_device *exist_aux_dev,
+			     struct rte_auxiliary_device *new_aux_dev);
+
+/*
+ * Match the auxiliary driver and device by driver function.
+ */
+bool auxiliary_match(const struct rte_auxiliary_driver *aux_drv,
+		     const struct rte_auxiliary_device *aux_dev);
+
+/*
+ * Iterate over devices, matching any device against the provided string.
+ */
+void *auxiliary_dev_iterate(const void *start, const char *str,
+			    const struct rte_dev_iterator *it);
+
+#endif /* BUS_AUXILIARY_PRIVATE_H */
diff --git a/drivers/bus/auxiliary/rte_bus_auxiliary.h b/drivers/bus/auxiliary/rte_bus_auxiliary.h
new file mode 100644
index 0000000000..b7a8a03be4
--- /dev/null
+++ b/drivers/bus/auxiliary/rte_bus_auxiliary.h
@@ -0,0 +1,193 @@ 
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2021 NVIDIA Corporation & Affiliates
+ */
+
+#ifndef RTE_BUS_AUXILIARY_H
+#define RTE_BUS_AUXILIARY_H
+
+/**
+ * @file
+ *
+ * Auxiliary Bus Interface.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <limits.h>
+#include <errno.h>
+#include <sys/queue.h>
+#include <stdint.h>
+#include <inttypes.h>
+
+#include <rte_debug.h>
+#include <rte_interrupts.h>
+#include <rte_dev.h>
+#include <rte_bus.h>
+#include <rte_kvargs.h>
+
+#define RTE_BUS_AUXILIARY_NAME "auxiliary"
+
+/* Forward declarations */
+struct rte_auxiliary_driver;
+struct rte_auxiliary_bus;
+struct rte_auxiliary_device;
+
+/**
+ * Match function for the driver to decide if device can be handled.
+ *
+ * @param name
+ *   Pointer to the auxiliary device name.
+ * @return
+ *   Whether the driver can handle the auxiliary device.
+ */
+typedef bool(rte_auxiliary_match_t)(const char *name);
+
+/**
+ * Initialization function for the driver called during auxiliary probing.
+ *
+ * @param drv
+ *   Pointer to the auxiliary driver.
+ * @param dev
+ *   Pointer to the auxiliary device.
+ * @return
+ *   - 0 On success.
+ *   - Negative value and rte_errno is set otherwise.
+ */
+typedef int(rte_auxiliary_probe_t)(struct rte_auxiliary_driver *drv,
+				    struct rte_auxiliary_device *dev);
+
+/**
+ * Uninitialization function for the driver called during hotplugging.
+ *
+ * @param dev
+ *   Pointer to the auxiliary device.
+ * @return
+ *   - 0 On success.
+ *   - Negative value and rte_errno is set otherwise.
+ */
+typedef int (rte_auxiliary_remove_t)(struct rte_auxiliary_device *dev);
+
+/**
+ * Driver-specific DMA mapping. After a successful call the device
+ * will be able to read/write from/to this segment.
+ *
+ * @param dev
+ *   Pointer to the auxiliary device.
+ * @param addr
+ *   Starting virtual address of memory to be mapped.
+ * @param iova
+ *   Starting IOVA address of memory to be mapped.
+ * @param len
+ *   Length of memory segment being mapped.
+ * @return
+ *   - 0 On success.
+ *   - Negative value and rte_errno is set otherwise.
+ */
+typedef int (rte_auxiliary_dma_map_t)(struct rte_auxiliary_device *dev,
+				       void *addr, uint64_t iova, size_t len);
+
+/**
+ * Driver-specific DMA un-mapping. After a successful call the device
+ * will not be able to read/write from/to this segment.
+ *
+ * @param dev
+ *   Pointer to the auxiliary device.
+ * @param addr
+ *   Starting virtual address of memory to be unmapped.
+ * @param iova
+ *   Starting IOVA address of memory to be unmapped.
+ * @param len
+ *   Length of memory segment being unmapped.
+ * @return
+ *   - 0 On success.
+ *   - Negative value and rte_errno is set otherwise.
+ */
+typedef int (rte_auxiliary_dma_unmap_t)(struct rte_auxiliary_device *dev,
+					 void *addr, uint64_t iova, size_t len);
+
+/**
+ * A structure describing an auxiliary device.
+ */
+struct rte_auxiliary_device {
+	TAILQ_ENTRY(rte_auxiliary_device) next;   /**< Next probed device. */
+	struct rte_device device;                 /**< Inherit core device */
+	char name[RTE_DEV_NAME_MAX_LEN + 1];      /**< ASCII device name */
+	struct rte_intr_handle intr_handle;       /**< Interrupt handle */
+	struct rte_auxiliary_driver *driver;      /**< Device driver */
+};
+
+/**
+ * A structure describing an auxiliary driver.
+ */
+struct rte_auxiliary_driver {
+	TAILQ_ENTRY(rte_auxiliary_driver) next; /**< Next in list. */
+	struct rte_driver driver;             /**< Inherit core driver. */
+	struct rte_auxiliary_bus *bus;        /**< Auxiliary bus reference. */
+	rte_auxiliary_match_t *match;         /**< Device match function. */
+	rte_auxiliary_probe_t *probe;         /**< Device probe function. */
+	rte_auxiliary_remove_t *remove;       /**< Device remove function. */
+	rte_auxiliary_dma_map_t *dma_map;     /**< Device DMA map function. */
+	rte_auxiliary_dma_unmap_t *dma_unmap; /**< Device DMA unmap function. */
+	uint32_t drv_flags;                   /**< Flags RTE_AUXILIARY_DRV_*. */
+};
+
+/**
+ * @internal
+ * Helper macro for drivers that need to convert to struct rte_auxiliary_device.
+ */
+#define RTE_DEV_TO_AUXILIARY(ptr) \
+	container_of(ptr, struct rte_auxiliary_device, device)
+
+#define RTE_DEV_TO_AUXILIARY_CONST(ptr) \
+	container_of(ptr, const struct rte_auxiliary_device, device)
+
+#define RTE_ETH_DEV_TO_AUXILIARY(eth_dev) \
+	RTE_DEV_TO_AUXILIARY((eth_dev)->device)
+
+/** Device driver needs IOVA as VA and cannot work with IOVA as PA */
+#define RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA 0x002
+
+/**
+ * Register an auxiliary driver.
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * @param driver
+ *   A pointer to a rte_auxiliary_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_experimental
+void rte_auxiliary_register(struct rte_auxiliary_driver *driver);
+
+/** Helper for auxiliary device registration from driver instance */
+#define RTE_PMD_REGISTER_AUXILIARY(nm, auxiliary_drv) \
+	RTE_INIT(auxiliaryinitfn_ ##nm) \
+	{ \
+		(auxiliary_drv).driver.name = RTE_STR(nm); \
+		rte_auxiliary_register(&(auxiliary_drv)); \
+	} \
+	RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+/**
+ * Unregister an auxiliary driver.
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * @param driver
+ *   A pointer to a rte_auxiliary_driver structure describing the driver
+ *   to be unregistered.
+ */
+__rte_experimental
+void rte_auxiliary_unregister(struct rte_auxiliary_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_BUS_AUXILIARY_H */
diff --git a/drivers/bus/auxiliary/version.map b/drivers/bus/auxiliary/version.map
new file mode 100644
index 0000000000..a52260657c
--- /dev/null
+++ b/drivers/bus/auxiliary/version.map
@@ -0,0 +1,7 @@ 
+EXPERIMENTAL {
+	global:
+
+	# added in 21.08
+	rte_auxiliary_register;
+	rte_auxiliary_unregister;
+};
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 410058de3a..45eab5233d 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -2,6 +2,7 @@ 
 # Copyright(c) 2017 Intel Corporation
 
 drivers = [
+        'auxiliary',
         'dpaa',
         'fslmc',
         'ifpga',