[v5,1/2] eal: add API for bus close
Checks
Commit Message
From: Rohit Raj <rohit.raj@nxp.com>
As per the current code we have API for bus probe, but the
bus close API is missing. This breaks the multi process
scenarios as objects are not cleaned while terminating the
secondary processes.
This patch adds a new API rte_bus_close() for cleanup of
bus objects which were acquired during probe.
Signed-off-by: Rohit Raj <rohit.raj@nxp.com>
---
Rebased on this patch series:
https://patches.dpdk.org/project/dpdk/list/?series=21049
v5:
* Updated release notes for new feature and API change.
* Added support for error checking while closing bus.
* Added experimental banner for new API.
* Squashed changes related to freebsd and windows into single patch.
* Discarded patch to fix a bug which is already fixed on latest
release.
v4:
* Added comments to clarify responsibility of rte_bus_close.
* Added support for rte_bus_close on freebsd.
* Added support for rte_bus_close on windows.
v3:
* nit: combined nested if statements.
v2:
* Moved rte_bus_close call to rte_eal_cleanup path.
doc/guides/rel_notes/release_22_03.rst | 8 +++++++
lib/eal/common/eal_common_bus.c | 33 +++++++++++++++++++++++++-
lib/eal/freebsd/eal.c | 1 +
lib/eal/include/rte_bus.h | 30 ++++++++++++++++++++++-
lib/eal/linux/eal.c | 8 +++++++
lib/eal/version.map | 3 +++
lib/eal/windows/eal.c | 1 +
7 files changed, 82 insertions(+), 2 deletions(-)
Comments
Hi,
10/01/2022 06:26, rohit.raj@nxp.com:
> From: Rohit Raj <rohit.raj@nxp.com>
>
> As per the current code we have API for bus probe, but the
> bus close API is missing. This breaks the multi process
> scenarios as objects are not cleaned while terminating the
> secondary processes.
>
> This patch adds a new API rte_bus_close() for cleanup of
> bus objects which were acquired during probe.
I don't understand how closing all devices of a bus will help better
than just closing all devices.
As Ferruh already suggested in the past,
we could force closing all devices in rte_eal_cleanup().
And we already have the function rte_dev_remove().
Hi Thomas,
This "rte_bus_close" API is introduced to do the opposite of what "rte_bus_probe" does. Just like there are plug and unplug APIs for plugging and unplugging a single device.
The API you mentioned, "rte_dev_remove" supports only rte_device. But we also need to close/remove private devices of dpaa and fslmc buses which are not exposed directly to user (eg: mempool device).
Note that these private devices/bus objects are not associated with a particular rte_device but they are available as a resource to be used by any of the device based on these hardware specific buses.
So, to close these devices, we need a new API which can do this for us. That is why "rte_bus_close" is required.
Regards,
Rohit Raj
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, January 19, 2022 4:02 PM
> To: Rohit Raj <rohit.raj@nxp.com>
> Cc: Bruce Richardson <bruce.richardson@intel.com>; Ray Kinsella
> <mdr@ashroe.eu>; Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; Narcisa Ana
> Maria Vasile <navasile@linux.microsoft.com>; Dmitry Malloy
> <dmitrym@microsoft.com>; Pallavi Kadam <pallavi.kadam@intel.com>;
> dev@dpdk.org; Nipun Gupta <nipun.gupta@nxp.com>; Sachin Saxena
> <sachin.saxena@nxp.com>; Hemant Agrawal <hemant.agrawal@nxp.com>;
> ferruh.yigit@intel.com; david.marchand@redhat.com
> Subject: [EXT] Re: [PATCH v5 1/2] eal: add API for bus close
>
> Caution: EXT Email
>
> Hi,
>
> 10/01/2022 06:26, rohit.raj@nxp.com:
> > From: Rohit Raj <rohit.raj@nxp.com>
> >
> > As per the current code we have API for bus probe, but the bus close
> > API is missing. This breaks the multi process scenarios as objects are
> > not cleaned while terminating the secondary processes.
> >
> > This patch adds a new API rte_bus_close() for cleanup of bus objects
> > which were acquired during probe.
>
> I don't understand how closing all devices of a bus will help better than just
> closing all devices.
>
> As Ferruh already suggested in the past, we could force closing all devices in
> rte_eal_cleanup().
> And we already have the function rte_dev_remove().
>
>
20/01/2022 15:51, Rohit Raj:
> Hi Thomas,
>
> This "rte_bus_close" API is introduced to do the opposite of what "rte_bus_probe" does. Just like there are plug and unplug APIs for plugging and unplugging a single device.
>
> The API you mentioned, "rte_dev_remove" supports only rte_device. But we also need to close/remove private devices of dpaa and fslmc buses which are not exposed directly to user (eg: mempool device).
> Note that these private devices/bus objects are not associated with a particular rte_device but they are available as a resource to be used by any of the device based on these hardware specific buses.
> So, to close these devices, we need a new API which can do this for us. That is why "rte_bus_close" is required.
You mean some resources are associated to a bus but not to a device?
It lools very weird. A resource on a bus *is* a device.
PS: please avoid top-post
> From: Thomas Monjalon <thomas@monjalon.net>
> > 10/01/2022 06:26, rohit.raj@nxp.com:
> > > From: Rohit Raj <rohit.raj@nxp.com>
> > >
> > > As per the current code we have API for bus probe, but the bus close
> > > API is missing. This breaks the multi process scenarios as objects are
> > > not cleaned while terminating the secondary processes.
> > >
> > > This patch adds a new API rte_bus_close() for cleanup of bus objects
> > > which were acquired during probe.
> >
> > I don't understand how closing all devices of a bus will help better than just
> > closing all devices.
> >
> > As Ferruh already suggested in the past, we could force closing all devices in
> > rte_eal_cleanup().
> > And we already have the function rte_dev_remove().
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, January 20, 2022 8:28 PM
> To: Rohit Raj <rohit.raj@nxp.com>
> Cc: Bruce Richardson <bruce.richardson@intel.com>; Ray Kinsella
> <mdr@ashroe.eu>; Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; Narcisa Ana
> Maria Vasile <navasile@linux.microsoft.com>; Dmitry Malloy
> <dmitrym@microsoft.com>; Pallavi Kadam <pallavi.kadam@intel.com>;
> dev@dpdk.org; Nipun Gupta <nipun.gupta@nxp.com>; Sachin Saxena
> <sachin.saxena@nxp.com>; Hemant Agrawal <hemant.agrawal@nxp.com>;
> ferruh.yigit@intel.com; david.marchand@redhat.com
> Subject: Re: [EXT] Re: [PATCH v5 1/2] eal: add API for bus close
>
> Caution: EXT Email
>
> 20/01/2022 15:51, Rohit Raj:
> > Hi Thomas,
> >
> > This "rte_bus_close" API is introduced to do the opposite of what
> "rte_bus_probe" does. Just like there are plug and unplug APIs for plugging and
> unplugging a single device.
> >
> > The API you mentioned, "rte_dev_remove" supports only rte_device. But we
> also need to close/remove private devices of dpaa and fslmc buses which are
> not exposed directly to user (eg: mempool device).
> > Note that these private devices/bus objects are not associated with a
> particular rte_device but they are available as a resource to be used by any of
> the device based on these hardware specific buses.
> > So, to close these devices, we need a new API which can do this for us. That is
> why "rte_bus_close" is required.
>
> You mean some resources are associated to a bus but not to a device?
> It lools very weird. A resource on a bus *is* a device.
>
> PS: please avoid top-post
FSLMC bus has hardware resources for memory pools, queues, hardware access lock(called portal).
These are common resources, which can be associated with any device. So they don't belong to a specific device.
Eg: mempool resource can be used by both eth and crypto device. So, we cannot close mempool while closing just one of the device(It can happen in multiprocess applications). So, these resources should be specifically closed/freed with the bus instead with a device.
There is no need to expose these devices to users and their usages is limited to other devices on the bus. There is no reason to create yet another type of device in DPDK to expose these internal only resources.
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 10/01/2022 06:26, rohit.raj@nxp.com:
> > > > From: Rohit Raj <rohit.raj@nxp.com>
> > > >
> > > > As per the current code we have API for bus probe, but the bus
> > > > close API is missing. This breaks the multi process scenarios as
> > > > objects are not cleaned while terminating the secondary processes.
> > > >
> > > > This patch adds a new API rte_bus_close() for cleanup of bus
> > > > objects which were acquired during probe.
> > >
> > > I don't understand how closing all devices of a bus will help better
> > > than just closing all devices.
> > >
> > > As Ferruh already suggested in the past, we could force closing all
> > > devices in rte_eal_cleanup().
> > > And we already have the function rte_dev_remove().
>
>
01/02/2022 06:40, Rohit Raj:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 20/01/2022 15:51, Rohit Raj:
> > > Hi Thomas,
> > >
> > > This "rte_bus_close" API is introduced to do the opposite of what
> > "rte_bus_probe" does. Just like there are plug and unplug APIs for plugging and
> > unplugging a single device.
> > >
> > > The API you mentioned, "rte_dev_remove" supports only rte_device. But we
> > also need to close/remove private devices of dpaa and fslmc buses which are
> > not exposed directly to user (eg: mempool device).
> > > Note that these private devices/bus objects are not associated with a
> > particular rte_device but they are available as a resource to be used by any of
> > the device based on these hardware specific buses.
> > > So, to close these devices, we need a new API which can do this for us. That is
> > why "rte_bus_close" is required.
> >
> > You mean some resources are associated to a bus but not to a device?
> > It lools very weird. A resource on a bus *is* a device.
> >
> > PS: please avoid top-post
>
> FSLMC bus has hardware resources for memory pools, queues, hardware access lock(called portal).
> These are common resources, which can be associated with any device. So they don't belong to a specific device.
> Eg: mempool resource can be used by both eth and crypto device. So, we cannot close mempool while closing just one of the device(It can happen in multiprocess applications). So, these resources should be specifically closed/freed with the bus instead with a device.
>
> There is no need to expose these devices to users and their usages is limited to other devices on the bus. There is no reason to create yet another type of device in DPDK to expose these internal only resources.
OK I understand better now, thanks.
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > 10/01/2022 06:26, rohit.raj@nxp.com:
> > > > > From: Rohit Raj <rohit.raj@nxp.com>
> > > > >
> > > > > As per the current code we have API for bus probe, but the bus
> > > > > close API is missing. This breaks the multi process scenarios as
> > > > > objects are not cleaned while terminating the secondary processes.
> > > > >
> > > > > This patch adds a new API rte_bus_close() for cleanup of bus
> > > > > objects which were acquired during probe.
> > > >
> > > > I don't understand how closing all devices of a bus will help better
> > > > than just closing all devices.
> > > >
> > > > As Ferruh already suggested in the past, we could force closing all
> > > > devices in rte_eal_cleanup().
> > > > And we already have the function rte_dev_remove().
On Mon, Jan 10, 2022 at 6:26 AM <rohit.raj@nxp.com> wrote:
>
> From: Rohit Raj <rohit.raj@nxp.com>
>
> As per the current code we have API for bus probe, but the
> bus close API is missing. This breaks the multi process
> scenarios as objects are not cleaned while terminating the
> secondary processes.
After an application crash, how does this bus resets the associated resources?
>
> This patch adds a new API rte_bus_close() for cleanup of
> bus objects which were acquired during probe.
The patch in its current form breaks the ABI on rte_bus object.
This can be seen in GHA, or calling DPDK_ABI_REF_VERSION=v21.11
./devtools/test-meson-builds.sh.
[snip]
> diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst
> index 6d99d1eaa9..7417606a2a 100644
> --- a/doc/guides/rel_notes/release_22_03.rst
> +++ b/doc/guides/rel_notes/release_22_03.rst
> @@ -55,6 +55,11 @@ New Features
> Also, make sure to start the actual text at the margin.
> =======================================================
>
> + * **Added support to close bus.**
> +
> + Added capability to allow a user to do cleanup of bus objects which
> + were acquired during bus probe.
> +
Wrongly indented.
>
> Removed Items
> -------------
> @@ -84,6 +89,9 @@ API Changes
> Also, make sure to start the actual text at the margin.
> =======================================================
>
> + * eal: Added new API ``rte_bus_close`` to perform cleanup bus objects which
> + were acquired during bus probe.
> +
>
> ABI Changes
> -----------
> diff --git a/lib/eal/common/eal_common_bus.c b/lib/eal/common/eal_common_bus.c
> index baa5b532af..2c3c0a90d2 100644
> --- a/lib/eal/common/eal_common_bus.c
> +++ b/lib/eal/common/eal_common_bus.c
> @@ -1,5 +1,5 @@
> /* SPDX-License-Identifier: BSD-3-Clause
> - * Copyright 2016 NXP
> + * Copyright 2016,2022 NXP
> */
>
> #include <stdio.h>
> @@ -85,6 +85,37 @@ rte_bus_probe(void)
> return 0;
> }
>
> +/* Close all devices of all buses */
> +int
> +rte_bus_close(void)
> +{
> + int ret;
> + struct rte_bus *bus, *vbus = NULL;
> +
> + TAILQ_FOREACH(bus, &rte_bus_list, next) {
> + if (!strcmp(bus->name, "vdev")) {
> + vbus = bus;
> + continue;
> + }
> +
> + if (bus->close) {
> + ret = bus->close();
> + if (ret)
> + RTE_LOG(ERR, EAL, "Bus (%s) close failed.\n",
> + bus->name);
> + }
> + }
> +
> + if (vbus && vbus->close) {
> + ret = vbus->close();
> + if (ret)
> + RTE_LOG(ERR, EAL, "Bus (%s) close failed.\n",
> + vbus->name);
> + }
The vdev bus is special in that some drivers can reference objects
from other buses (see f4ce209a8ce5 ("eal: postpone vdev
initialization") and da76cc02342b ("eal: probe new virtual bus after
other bus devices")).
For this reason, I would expect that the vdev bus is closed before the
other buses.
> +
> + return 0;
> +}
> +
> /* Dump information of a single bus */
> static int
> bus_dump_one(FILE *f, struct rte_bus *bus)
> diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c
> index a1cd2462db..87d70c6898 100644
> --- a/lib/eal/freebsd/eal.c
> +++ b/lib/eal/freebsd/eal.c
> @@ -984,6 +984,7 @@ rte_eal_cleanup(void)
> {
> struct internal_config *internal_conf =
> eal_get_internal_configuration();
> + rte_bus_close();
> rte_service_finalize();
> rte_mp_channel_cleanup();
> /* after this point, any DPDK pointers will become dangling */
> diff --git a/lib/eal/include/rte_bus.h b/lib/eal/include/rte_bus.h
> index bbbb6efd28..c6211bbd95 100644
> --- a/lib/eal/include/rte_bus.h
> +++ b/lib/eal/include/rte_bus.h
> @@ -1,5 +1,5 @@
> /* SPDX-License-Identifier: BSD-3-Clause
> - * Copyright 2016 NXP
> + * Copyright 2016,2022 NXP
> */
>
> #ifndef _RTE_BUS_H_
> @@ -66,6 +66,23 @@ typedef int (*rte_bus_scan_t)(void);
> */
> typedef int (*rte_bus_probe_t)(void);
>
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Implementation specific close function which is responsible for resetting all
> + * detected devices on the bus to a default state, closing UIO nodes or VFIO
> + * groups and also freeing any memory allocated during rte_bus_probe like
> + * private resources for device list.
> + *
> + * This is called while iterating over each registered bus.
> + *
> + * @return
> + * 0 for successful close
> + * !0 for any error while closing
> + */
> +typedef int (*rte_bus_close_t)(void);
> +
> /**
> * Device iterator to find a device on a bus.
> *
> @@ -263,6 +280,7 @@ struct rte_bus {
> const char *name; /**< Name of the bus */
> rte_bus_scan_t scan; /**< Scan for devices attached to bus */
> rte_bus_probe_t probe; /**< Probe devices on bus */
> + rte_bus_close_t close; /**< Close devices on bus */
> rte_bus_find_device_t find_device; /**< Find a device on the bus */
> rte_bus_plug_t plug; /**< Probe single device for drivers */
> rte_bus_unplug_t unplug; /**< Remove single device from driver */
> @@ -317,6 +335,16 @@ int rte_bus_scan(void);
> */
> int rte_bus_probe(void);
>
> +/**
> + * For each device on the buses, call the device specific close.
> + *
> + * @return
> + * 0 for successful close
> + * !0 otherwise
> + */
> +__rte_experimental
> +int rte_bus_close(void);
> +
> /**
> * Dump information of all the buses registered with EAL.
> *
> diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
> index 60b4924838..5c60131e46 100644
> --- a/lib/eal/linux/eal.c
> +++ b/lib/eal/linux/eal.c
> @@ -1362,6 +1362,14 @@ rte_eal_cleanup(void)
>
> if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> rte_memseg_walk(mark_freeable, NULL);
> +
> + /* Close all the buses and devices/drivers on them */
> + if (rte_bus_close()) {
> + rte_eal_init_alert("Cannot close devices");
You can't call rte_eal_*init*_alert in rte_eal_cleanup.
There is not much to do if the bus close fails, I'd rather leave the
cleanup continue.
> + rte_errno = ENOTSUP;
> + return -1;
> + }
> +
> rte_service_finalize();
> rte_mp_channel_cleanup();
> /* after this point, any DPDK pointers will become dangling */
> diff --git a/lib/eal/version.map b/lib/eal/version.map
> index ab28c22791..39882dbbd5 100644
> --- a/lib/eal/version.map
> +++ b/lib/eal/version.map
> @@ -420,6 +420,9 @@ EXPERIMENTAL {
> rte_intr_instance_free;
> rte_intr_type_get;
> rte_intr_type_set;
> +
> + # added in 22.03
> + rte_bus_close;
> };
>
> INTERNAL {
> diff --git a/lib/eal/windows/eal.c b/lib/eal/windows/eal.c
> index 67db7f099a..5915ab6291 100644
> --- a/lib/eal/windows/eal.c
> +++ b/lib/eal/windows/eal.c
> @@ -260,6 +260,7 @@ rte_eal_cleanup(void)
> struct internal_config *internal_conf =
> eal_get_internal_configuration();
>
> + rte_bus_close();
> eal_intr_thread_cancel();
> eal_mem_virt2iova_cleanup();
> /* after this point, any DPDK pointers will become dangling */
> --
> 2.17.1
>
09/02/2022 12:04, David Marchand:
> On Mon, Jan 10, 2022 at 6:26 AM <rohit.raj@nxp.com> wrote:
> >
> > From: Rohit Raj <rohit.raj@nxp.com>
> >
> > As per the current code we have API for bus probe, but the
> > bus close API is missing. This breaks the multi process
> > scenarios as objects are not cleaned while terminating the
> > secondary processes.
>
> After an application crash, how does this bus resets the associated resources?
>
> > This patch adds a new API rte_bus_close() for cleanup of
> > bus objects which were acquired during probe.
>
> The patch in its current form breaks the ABI on rte_bus object.
> This can be seen in GHA, or calling DPDK_ABI_REF_VERSION=v21.11
> ./devtools/test-meson-builds.sh.
[...]
> > +/* Close all devices of all buses */
> > +int
> > +rte_bus_close(void)
> > +{
> > + int ret;
> > + struct rte_bus *bus, *vbus = NULL;
> > +
> > + TAILQ_FOREACH(bus, &rte_bus_list, next) {
> > + if (!strcmp(bus->name, "vdev")) {
Please do an explicit comparison "== 0".
> > + vbus = bus;
> > + continue;
> > + }
> > +
> > + if (bus->close) {
Please do an explicit comparison with "!= NULL".
We can also completely remove this check and implement the callback
in all buses. It should loop in all remaining devices and remove them.
> > + ret = bus->close();
> > + if (ret)
> > + RTE_LOG(ERR, EAL, "Bus (%s) close failed.\n",
> > + bus->name);
> > + }
> > + }
> > +
> > + if (vbus && vbus->close) {
> > + ret = vbus->close();
> > + if (ret)
> > + RTE_LOG(ERR, EAL, "Bus (%s) close failed.\n",
> > + vbus->name);
> > + }
>
> The vdev bus is special in that some drivers can reference objects
> from other buses (see f4ce209a8ce5 ("eal: postpone vdev
> initialization") and da76cc02342b ("eal: probe new virtual bus after
> other bus devices")).
> For this reason, I would expect that the vdev bus is closed before the
> other buses.
Yes, good catch.
We don't have to expose this function as API.
This can be an internal function called only in rte_eal_cleanup().
Instead, it would be more useful to have a public function
to close a single bus by its name.
> > @@ -263,6 +280,7 @@ struct rte_bus {
> > const char *name; /**< Name of the bus */
> > rte_bus_scan_t scan; /**< Scan for devices attached to bus */
> > rte_bus_probe_t probe; /**< Probe devices on bus */
> > + rte_bus_close_t close; /**< Close devices on bus */
As David said, it is breaking the ABI.
[...]
> > @@ -1362,6 +1362,14 @@ rte_eal_cleanup(void)
> >
> > if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> > rte_memseg_walk(mark_freeable, NULL);
> > +
> > + /* Close all the buses and devices/drivers on them */
> > + if (rte_bus_close()) {
> > + rte_eal_init_alert("Cannot close devices");
>
> You can't call rte_eal_*init*_alert in rte_eal_cleanup.
>
> There is not much to do if the bus close fails, I'd rather leave the
> cleanup continue.
+1, just log and save the error code for return at the end.
@@ -55,6 +55,11 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+ * **Added support to close bus.**
+
+ Added capability to allow a user to do cleanup of bus objects which
+ were acquired during bus probe.
+
Removed Items
-------------
@@ -84,6 +89,9 @@ API Changes
Also, make sure to start the actual text at the margin.
=======================================================
+ * eal: Added new API ``rte_bus_close`` to perform cleanup bus objects which
+ were acquired during bus probe.
+
ABI Changes
-----------
@@ -1,5 +1,5 @@
/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright 2016 NXP
+ * Copyright 2016,2022 NXP
*/
#include <stdio.h>
@@ -85,6 +85,37 @@ rte_bus_probe(void)
return 0;
}
+/* Close all devices of all buses */
+int
+rte_bus_close(void)
+{
+ int ret;
+ struct rte_bus *bus, *vbus = NULL;
+
+ TAILQ_FOREACH(bus, &rte_bus_list, next) {
+ if (!strcmp(bus->name, "vdev")) {
+ vbus = bus;
+ continue;
+ }
+
+ if (bus->close) {
+ ret = bus->close();
+ if (ret)
+ RTE_LOG(ERR, EAL, "Bus (%s) close failed.\n",
+ bus->name);
+ }
+ }
+
+ if (vbus && vbus->close) {
+ ret = vbus->close();
+ if (ret)
+ RTE_LOG(ERR, EAL, "Bus (%s) close failed.\n",
+ vbus->name);
+ }
+
+ return 0;
+}
+
/* Dump information of a single bus */
static int
bus_dump_one(FILE *f, struct rte_bus *bus)
@@ -984,6 +984,7 @@ rte_eal_cleanup(void)
{
struct internal_config *internal_conf =
eal_get_internal_configuration();
+ rte_bus_close();
rte_service_finalize();
rte_mp_channel_cleanup();
/* after this point, any DPDK pointers will become dangling */
@@ -1,5 +1,5 @@
/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright 2016 NXP
+ * Copyright 2016,2022 NXP
*/
#ifndef _RTE_BUS_H_
@@ -66,6 +66,23 @@ typedef int (*rte_bus_scan_t)(void);
*/
typedef int (*rte_bus_probe_t)(void);
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Implementation specific close function which is responsible for resetting all
+ * detected devices on the bus to a default state, closing UIO nodes or VFIO
+ * groups and also freeing any memory allocated during rte_bus_probe like
+ * private resources for device list.
+ *
+ * This is called while iterating over each registered bus.
+ *
+ * @return
+ * 0 for successful close
+ * !0 for any error while closing
+ */
+typedef int (*rte_bus_close_t)(void);
+
/**
* Device iterator to find a device on a bus.
*
@@ -263,6 +280,7 @@ struct rte_bus {
const char *name; /**< Name of the bus */
rte_bus_scan_t scan; /**< Scan for devices attached to bus */
rte_bus_probe_t probe; /**< Probe devices on bus */
+ rte_bus_close_t close; /**< Close devices on bus */
rte_bus_find_device_t find_device; /**< Find a device on the bus */
rte_bus_plug_t plug; /**< Probe single device for drivers */
rte_bus_unplug_t unplug; /**< Remove single device from driver */
@@ -317,6 +335,16 @@ int rte_bus_scan(void);
*/
int rte_bus_probe(void);
+/**
+ * For each device on the buses, call the device specific close.
+ *
+ * @return
+ * 0 for successful close
+ * !0 otherwise
+ */
+__rte_experimental
+int rte_bus_close(void);
+
/**
* Dump information of all the buses registered with EAL.
*
@@ -1362,6 +1362,14 @@ rte_eal_cleanup(void)
if (rte_eal_process_type() == RTE_PROC_PRIMARY)
rte_memseg_walk(mark_freeable, NULL);
+
+ /* Close all the buses and devices/drivers on them */
+ if (rte_bus_close()) {
+ rte_eal_init_alert("Cannot close devices");
+ rte_errno = ENOTSUP;
+ return -1;
+ }
+
rte_service_finalize();
rte_mp_channel_cleanup();
/* after this point, any DPDK pointers will become dangling */
@@ -420,6 +420,9 @@ EXPERIMENTAL {
rte_intr_instance_free;
rte_intr_type_get;
rte_intr_type_set;
+
+ # added in 22.03
+ rte_bus_close;
};
INTERNAL {
@@ -260,6 +260,7 @@ rte_eal_cleanup(void)
struct internal_config *internal_conf =
eal_get_internal_configuration();
+ rte_bus_close();
eal_intr_thread_cancel();
eal_mem_virt2iova_cleanup();
/* after this point, any DPDK pointers will become dangling */