pci: fix missing pci bus with shared library build
Checks
Commit Message
If DPDK is built as a shared library, then any application linked
with rte.app.mk will not find any PCI devices. When the application
is started no ethernet devices are found.
This is because the link order of libraries on the command line matters.
And PCI is before EAL. That causes there to be no dependency on PCI
so linker ignores linking the library.
Swapping the order fixes this.
Fixes: c752998b5e2e ("pci: introduce library and driver")
Cc: stable@dpdk.org
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
mk/rte.app.mk | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Comments
On Mon, 15 Jul 2019 16:41:36 -0700
Stephen Hemminger <stephen@networkplumber.org> wrote:
> If DPDK is built as a shared library, then any application linked
> with rte.app.mk will not find any PCI devices. When the application
> is started no ethernet devices are found.
>
> This is because the link order of libraries on the command line matters.
> And PCI is before EAL. That causes there to be no dependency on PCI
> so linker ignores linking the library.
> Swapping the order fixes this.
>
> Fixes: c752998b5e2e ("pci: introduce library and driver")
> Cc: stable@dpdk.org
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> mk/rte.app.mk | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> index a277c808ed8e..470b92e4d73e 100644
> --- a/mk/rte.app.mk
> +++ b/mk/rte.app.mk
> @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack
> _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
> _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring
> -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal
> +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline
> _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
> _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
This only seems to happen on 18.11 and if configuration is reduced.
On Mon, 15 Jul 2019 16:41:36 -0700
Stephen Hemminger <stephen@networkplumber.org> wrote:
> If DPDK is built as a shared library, then any application linked
> with rte.app.mk will not find any PCI devices. When the application
> is started no ethernet devices are found.
>
> This is because the link order of libraries on the command line matters.
> And PCI is before EAL. That causes there to be no dependency on PCI
> so linker ignores linking the library.
> Swapping the order fixes this.
>
> Fixes: c752998b5e2e ("pci: introduce library and driver")
> Cc: stable@dpdk.org
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> mk/rte.app.mk | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> index a277c808ed8e..470b92e4d73e 100644
> --- a/mk/rte.app.mk
> +++ b/mk/rte.app.mk
> @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack
> _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
> _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring
> -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal
> +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline
> _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
> _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
It still happens with 19.08. Testpmd works but only because it is
linked with so many things. But l3fwd fails...
# ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: failed to parse device "02:00.0"
EAL: Unable to parse device '02:00.0'
EAL: Error - exiting with code: 1
Cause: Invalid EAL parameters
On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> On Mon, 15 Jul 2019 16:41:36 -0700
> Stephen Hemminger <stephen@networkplumber.org> wrote:
>
> > If DPDK is built as a shared library, then any application linked
> > with rte.app.mk will not find any PCI devices. When the application
> > is started no ethernet devices are found.
> >
> > This is because the link order of libraries on the command line matters.
> > And PCI is before EAL. That causes there to be no dependency on PCI
> > so linker ignores linking the library.
> > Swapping the order fixes this.
> >
> > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > Cc: stable@dpdk.org
> > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > ---
> > mk/rte.app.mk | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > index a277c808ed8e..470b92e4d73e 100644
> > --- a/mk/rte.app.mk
> > +++ b/mk/rte.app.mk
> > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack
> > _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
> > _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring
> > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal
> > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline
> > _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
> > _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
>
> It still happens with 19.08. Testpmd works but only because it is
> linked with so many things. But l3fwd fails...
>
> # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> EAL: Detected 8 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: failed to parse device "02:00.0"
> EAL: Unable to parse device '02:00.0'
> EAL: Error - exiting with code: 1
> Cause: Invalid EAL parameters
I don't think the position of these is going to be the cause here, the more
likely cause is that the pci bus driver - and all other drivers - are not
linked into apps for shared library builds. You always need to pass "-d"
parameter to load drivers at init time (or have them installed in the
correct driver path). For example, for me with a shared library build the
following gives a no ports error:
sudo ./build/l2fwd -c F00000 -- -p 3
while this succeeds and runs fine
sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
On Tue, 16 Jul 2019 09:46:04 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:
> On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> > On Mon, 15 Jul 2019 16:41:36 -0700
> > Stephen Hemminger <stephen@networkplumber.org> wrote:
> >
> > > If DPDK is built as a shared library, then any application linked
> > > with rte.app.mk will not find any PCI devices. When the application
> > > is started no ethernet devices are found.
> > >
> > > This is because the link order of libraries on the command line matters.
> > > And PCI is before EAL. That causes there to be no dependency on PCI
> > > so linker ignores linking the library.
> > > Swapping the order fixes this.
> > >
> > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > > Cc: stable@dpdk.org
> > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > > ---
> > > mk/rte.app.mk | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > index a277c808ed8e..470b92e4d73e 100644
> > > --- a/mk/rte.app.mk
> > > +++ b/mk/rte.app.mk
> > > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack
> > > _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring
> > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal
> > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
> >
> > It still happens with 19.08. Testpmd works but only because it is
> > linked with so many things. But l3fwd fails...
> >
> > # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> > EAL: Detected 8 lcore(s)
> > EAL: Detected 1 NUMA nodes
> > EAL: failed to parse device "02:00.0"
> > EAL: Unable to parse device '02:00.0'
> > EAL: Error - exiting with code: 1
> > Cause: Invalid EAL parameters
>
> I don't think the position of these is going to be the cause here, the more
> likely cause is that the pci bus driver - and all other drivers - are not
> linked into apps for shared library builds. You always need to pass "-d"
> parameter to load drivers at init time (or have them installed in the
> correct driver path). For example, for me with a shared library build the
> following gives a no ports error:
>
> sudo ./build/l2fwd -c F00000 -- -p 3
>
> while this succeeds and runs fine
>
> sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
It is broken with original link order.
With the patch it works fine.
On Tue, 16 Jul 2019 09:46:04 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:
> I don't think the position of these is going to be the cause here, the more
> likely cause is that the pci bus driver - and all other drivers - are not
> linked into apps for shared library builds. You always need to pass "-d"
> parameter to load drivers at init time (or have them installed in the
> correct driver path). For example, for me with a shared library build the
> following gives a no ports error:
>
> sudo ./build/l2fwd -c F00000 -- -p 3
>
> while this succeeds and runs fine
>
> sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
Application should never have to use -d to force driver load!
The purpose of all the pmdinfo, and probing logic is to not do this.
On Tue, 16 Jul 2019 09:46:04 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:
> On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> > On Mon, 15 Jul 2019 16:41:36 -0700
> > Stephen Hemminger <stephen@networkplumber.org> wrote:
> >
> > > If DPDK is built as a shared library, then any application linked
> > > with rte.app.mk will not find any PCI devices. When the application
> > > is started no ethernet devices are found.
> > >
> > > This is because the link order of libraries on the command line matters.
> > > And PCI is before EAL. That causes there to be no dependency on PCI
> > > so linker ignores linking the library.
> > > Swapping the order fixes this.
> > >
> > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > > Cc: stable@dpdk.org
> > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > > ---
> > > mk/rte.app.mk | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > index a277c808ed8e..470b92e4d73e 100644
> > > --- a/mk/rte.app.mk
> > > +++ b/mk/rte.app.mk
> > > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack
> > > _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring
> > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal
> > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
> >
> > It still happens with 19.08. Testpmd works but only because it is
> > linked with so many things. But l3fwd fails...
> >
> > # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> > EAL: Detected 8 lcore(s)
> > EAL: Detected 1 NUMA nodes
> > EAL: failed to parse device "02:00.0"
> > EAL: Unable to parse device '02:00.0'
> > EAL: Error - exiting with code: 1
> > Cause: Invalid EAL parameters
>
> I don't think the position of these is going to be the cause here, the more
> likely cause is that the pci bus driver - and all other drivers - are not
> linked into apps for shared library builds. You always need to pass "-d"
> parameter to load drivers at init time (or have them installed in the
> correct driver path). For example, for me with a shared library build the
> following gives a no ports error:
>
> sudo ./build/l2fwd -c F00000 -- -p 3
>
> while this succeeds and runs fine
>
> sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
There is something else going on.
This patch isn't fixing the problem consistently.
The suggestion of using -d is still wrong. We need to figure out how to get applications
to run the same whether using shared library or statically linked.
On Tue, 16 Jul 2019 09:46:04 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:
> On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> > On Mon, 15 Jul 2019 16:41:36 -0700
> > Stephen Hemminger <stephen@networkplumber.org> wrote:
> >
> > > If DPDK is built as a shared library, then any application linked
> > > with rte.app.mk will not find any PCI devices. When the application
> > > is started no ethernet devices are found.
> > >
> > > This is because the link order of libraries on the command line matters.
> > > And PCI is before EAL. That causes there to be no dependency on PCI
> > > so linker ignores linking the library.
> > > Swapping the order fixes this.
> > >
> > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > > Cc: stable@dpdk.org
> > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > > ---
> > > mk/rte.app.mk | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > index a277c808ed8e..470b92e4d73e 100644
> > > --- a/mk/rte.app.mk
> > > +++ b/mk/rte.app.mk
> > > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack
> > > _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring
> > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal
> > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
> > > _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
> >
> > It still happens with 19.08. Testpmd works but only because it is
> > linked with so many things. But l3fwd fails...
> >
> > # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> > EAL: Detected 8 lcore(s)
> > EAL: Detected 1 NUMA nodes
> > EAL: failed to parse device "02:00.0"
> > EAL: Unable to parse device '02:00.0'
> > EAL: Error - exiting with code: 1
> > Cause: Invalid EAL parameters
>
> I don't think the position of these is going to be the cause here, the more
> likely cause is that the pci bus driver - and all other drivers - are not
> linked into apps for shared library builds. You always need to pass "-d"
> parameter to load drivers at init time (or have them installed in the
> correct driver path). For example, for me with a shared library build the
> following gives a no ports error:
>
> sudo ./build/l2fwd -c F00000 -- -p 3
>
> while this succeeds and runs fine
>
> sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
The root cause is that recent gcc won't run constructor on unused libraries.
Testing a patch to take --as-needed off of PCI library.
See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
19/07/2019 22:55, Stephen Hemminger:
> On Tue, 16 Jul 2019 09:46:04 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
>
> > On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> > > On Mon, 15 Jul 2019 16:41:36 -0700
> > > Stephen Hemminger <stephen@networkplumber.org> wrote:
> > >
> > > > If DPDK is built as a shared library, then any application linked
> > > > with rte.app.mk will not find any PCI devices. When the application
> > > > is started no ethernet devices are found.
> > > >
> > > > This is because the link order of libraries on the command line matters.
> > > > And PCI is before EAL. That causes there to be no dependency on PCI
> > > > so linker ignores linking the library.
> > > > Swapping the order fixes this.
> > > >
> > > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > > > Cc: stable@dpdk.org
> > > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > > > ---
> > > > mk/rte.app.mk | 2 +-
> > > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > > index a277c808ed8e..470b92e4d73e 100644
> > > > --- a/mk/rte.app.mk
> > > > +++ b/mk/rte.app.mk
> > > > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack
> > > > _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
> > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring
> > > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal
> > > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline
> > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
> > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
> > >
> > > It still happens with 19.08. Testpmd works but only because it is
> > > linked with so many things. But l3fwd fails...
> > >
> > > # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> > > EAL: Detected 8 lcore(s)
> > > EAL: Detected 1 NUMA nodes
> > > EAL: failed to parse device "02:00.0"
> > > EAL: Unable to parse device '02:00.0'
> > > EAL: Error - exiting with code: 1
> > > Cause: Invalid EAL parameters
> >
> > I don't think the position of these is going to be the cause here, the more
> > likely cause is that the pci bus driver - and all other drivers - are not
> > linked into apps for shared library builds. You always need to pass "-d"
> > parameter to load drivers at init time (or have them installed in the
> > correct driver path). For example, for me with a shared library build the
> > following gives a no ports error:
> >
> > sudo ./build/l2fwd -c F00000 -- -p 3
> >
> > while this succeeds and runs fine
> >
> > sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
>
> The root cause is that recent gcc won't run constructor on unused libraries.
> Testing a patch to take --as-needed off of PCI library.
>
> See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
The constructor is run when calling dlopen, right?
Note: dlopen with -d is a feature.
The original idea was to be able to specify which driver we want to use.
If we want an automatic dlopen, like modprobe, then we need more scripts.
But I understand you are against the whole dlopen idea.
On Mon, Jul 22, 2019 at 09:38:27AM +0200, Thomas Monjalon wrote:
> 19/07/2019 22:55, Stephen Hemminger:
> > On Tue, 16 Jul 2019 09:46:04 +0100
> > Bruce Richardson <bruce.richardson@intel.com> wrote:
> >
> > > On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> > > > On Mon, 15 Jul 2019 16:41:36 -0700
> > > > Stephen Hemminger <stephen@networkplumber.org> wrote:
> > > >
> > > > > If DPDK is built as a shared library, then any application linked
> > > > > with rte.app.mk will not find any PCI devices. When the application
> > > > > is started no ethernet devices are found.
> > > > >
> > > > > This is because the link order of libraries on the command line matters.
> > > > > And PCI is before EAL. That causes there to be no dependency on PCI
> > > > > so linker ignores linking the library.
> > > > > Swapping the order fixes this.
> > > > >
> > > > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > > > > Cc: stable@dpdk.org
> > > > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > > > > ---
> > > > > mk/rte.app.mk | 2 +-
> > > > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > > > index a277c808ed8e..470b92e4d73e 100644
> > > > > --- a/mk/rte.app.mk
> > > > > +++ b/mk/rte.app.mk
> > > > > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack
> > > > > _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
> > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring
> > > > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal
> > > > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline
> > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
> > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
> > > >
> > > > It still happens with 19.08. Testpmd works but only because it is
> > > > linked with so many things. But l3fwd fails...
> > > >
> > > > # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> > > > EAL: Detected 8 lcore(s)
> > > > EAL: Detected 1 NUMA nodes
> > > > EAL: failed to parse device "02:00.0"
> > > > EAL: Unable to parse device '02:00.0'
> > > > EAL: Error - exiting with code: 1
> > > > Cause: Invalid EAL parameters
> > >
> > > I don't think the position of these is going to be the cause here, the more
> > > likely cause is that the pci bus driver - and all other drivers - are not
> > > linked into apps for shared library builds. You always need to pass "-d"
> > > parameter to load drivers at init time (or have them installed in the
> > > correct driver path). For example, for me with a shared library build the
> > > following gives a no ports error:
> > >
> > > sudo ./build/l2fwd -c F00000 -- -p 3
> > >
> > > while this succeeds and runs fine
> > >
> > > sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
> >
> > The root cause is that recent gcc won't run constructor on unused libraries.
> > Testing a patch to take --as-needed off of PCI library.
> >
> > See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
>
> The constructor is run when calling dlopen, right?
>
> Note: dlopen with -d is a feature.
> The original idea was to be able to specify which driver we want to use.
> If we want an automatic dlopen, like modprobe, then we need more scripts.
> But I understand you are against the whole dlopen idea.
>
This issue is more of a problem for development systems where we EAL path
is not really usable for finding the drivers. For a properly deployed
system where we use DPDK installed to /usr/local or /usr, the EAL PMD path
will be correctly configured and properly probe all drivers.
On Mon, 22 Jul 2019 10:06:11 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:
> On Mon, Jul 22, 2019 at 09:38:27AM +0200, Thomas Monjalon wrote:
> > 19/07/2019 22:55, Stephen Hemminger:
> > > On Tue, 16 Jul 2019 09:46:04 +0100
> > > Bruce Richardson <bruce.richardson@intel.com> wrote:
> > >
> > > > On Mon, Jul 15, 2019 at 05:19:12PM -0700, Stephen Hemminger wrote:
> > > > > On Mon, 15 Jul 2019 16:41:36 -0700
> > > > > Stephen Hemminger <stephen@networkplumber.org> wrote:
> > > > >
> > > > > > If DPDK is built as a shared library, then any application linked
> > > > > > with rte.app.mk will not find any PCI devices. When the application
> > > > > > is started no ethernet devices are found.
> > > > > >
> > > > > > This is because the link order of libraries on the command line matters.
> > > > > > And PCI is before EAL. That causes there to be no dependency on PCI
> > > > > > so linker ignores linking the library.
> > > > > > Swapping the order fixes this.
> > > > > >
> > > > > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > > > > > Cc: stable@dpdk.org
> > > > > > Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> > > > > > ---
> > > > > > mk/rte.app.mk | 2 +-
> > > > > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > > > > >
> > > > > > diff --git a/mk/rte.app.mk b/mk/rte.app.mk
> > > > > > index a277c808ed8e..470b92e4d73e 100644
> > > > > > --- a/mk/rte.app.mk
> > > > > > +++ b/mk/rte.app.mk
> > > > > > @@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack
> > > > > > _LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
> > > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
> > > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring
> > > > > > -_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal
> > > > > > +_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
> > > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline
> > > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
> > > > > > _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
> > > > >
> > > > > It still happens with 19.08. Testpmd works but only because it is
> > > > > linked with so many things. But l3fwd fails...
> > > > >
> > > > > # ./examples/l3fwd/build/l3fwd -n4 -l0-3 -w 02:00.0
> > > > > EAL: Detected 8 lcore(s)
> > > > > EAL: Detected 1 NUMA nodes
> > > > > EAL: failed to parse device "02:00.0"
> > > > > EAL: Unable to parse device '02:00.0'
> > > > > EAL: Error - exiting with code: 1
> > > > > Cause: Invalid EAL parameters
> > > >
> > > > I don't think the position of these is going to be the cause here, the more
> > > > likely cause is that the pci bus driver - and all other drivers - are not
> > > > linked into apps for shared library builds. You always need to pass "-d"
> > > > parameter to load drivers at init time (or have them installed in the
> > > > correct driver path). For example, for me with a shared library build the
> > > > following gives a no ports error:
> > > >
> > > > sudo ./build/l2fwd -c F00000 -- -p 3
> > > >
> > > > while this succeeds and runs fine
> > > >
> > > > sudo ./build/l2fwd -c F00000 -d $RTE_SDK/$RTE_TARGET/lib/librte_pmd_i40e.so -- -p 3
> > >
> > > The root cause is that recent gcc won't run constructor on unused libraries.
> > > Testing a patch to take --as-needed off of PCI library.
> > >
> > > See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
> >
> > The constructor is run when calling dlopen, right?
> >
> > Note: dlopen with -d is a feature.
> > The original idea was to be able to specify which driver we want to use.
> > If we want an automatic dlopen, like modprobe, then we need more scripts.
> > But I understand you are against the whole dlopen idea.
> >
>
> This issue is more of a problem for development systems where we EAL path
> is not really usable for finding the drivers. For a properly deployed
> system where we use DPDK installed to /usr/local or /usr, the EAL PMD path
> will be correctly configured and properly probe all drivers.
The problem is that bus drivers register themselves in constructors and
these construtors are not run with as-needed.
One part of fixing this is:
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index df917f946497..46bdff8ec5e8 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -130,6 +130,9 @@ ifeq ($(CONFIG_RTE_LIBRTE_FSLMC_BUS),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_COMMON_FSLMC) += -lrte_common_fslmc
endif
+# Bus devices register in constructor so always link
+_LDLIBS-y += --no-as-needed
+
_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += -lrte_bus_pci
_LDLIBS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += -lrte_bus_vdev
_LDLIBS-$(CONFIG_RTE_LIBRTE_DPAA_BUS) += -lrte_bus_dpaa
@@ -137,6 +140,8 @@ ifeq ($(CONFIG_RTE_EAL_VFIO),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += -lrte_bus_fslmc
endif
+_LDLIBS-y += --as-needed
+
ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),n)
# plugins (link only if static libraries)
22/07/2019 18:43, Stephen Hemminger:
> On Mon, 22 Jul 2019 10:06:11 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
> > On Mon, Jul 22, 2019 at 09:38:27AM +0200, Thomas Monjalon wrote:
> > > 19/07/2019 22:55, Stephen Hemminger:
> > > > The root cause is that recent gcc won't run constructor on unused libraries.
> > > > Testing a patch to take --as-needed off of PCI library.
> > > >
> > > > See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
> > >
> > > The constructor is run when calling dlopen, right?
> > >
> > > Note: dlopen with -d is a feature.
> > > The original idea was to be able to specify which driver we want to use.
> > > If we want an automatic dlopen, like modprobe, then we need more scripts.
> > > But I understand you are against the whole dlopen idea.
> >
> > This issue is more of a problem for development systems where we EAL path
> > is not really usable for finding the drivers. For a properly deployed
> > system where we use DPDK installed to /usr/local or /usr, the EAL PMD path
> > will be correctly configured and properly probe all drivers.
>
> The problem is that bus drivers register themselves in constructors and
> these construtors are not run with as-needed.
Are the constructors run on dlopen of the bus driver?
On Mon, 22 Jul 2019 19:04:55 +0200
Thomas Monjalon <thomas@monjalon.net> wrote:
> 22/07/2019 18:43, Stephen Hemminger:
> > On Mon, 22 Jul 2019 10:06:11 +0100
> > Bruce Richardson <bruce.richardson@intel.com> wrote:
> > > On Mon, Jul 22, 2019 at 09:38:27AM +0200, Thomas Monjalon wrote:
> > > > 19/07/2019 22:55, Stephen Hemminger:
> > > > > The root cause is that recent gcc won't run constructor on unused libraries.
> > > > > Testing a patch to take --as-needed off of PCI library.
> > > > >
> > > > > See: https://stackoverflow.com/questions/11631161/force-to-link-against-unused-shared-library
> > > >
> > > > The constructor is run when calling dlopen, right?
> > > >
> > > > Note: dlopen with -d is a feature.
> > > > The original idea was to be able to specify which driver we want to use.
> > > > If we want an automatic dlopen, like modprobe, then we need more scripts.
> > > > But I understand you are against the whole dlopen idea.
> > >
> > > This issue is more of a problem for development systems where we EAL path
> > > is not really usable for finding the drivers. For a properly deployed
> > > system where we use DPDK installed to /usr/local or /usr, the EAL PMD path
> > > will be correctly configured and properly probe all drivers.
> >
> > The problem is that bus drivers register themselves in constructors and
> > these construtors are not run with as-needed.
>
> Are the constructors run on dlopen of the bus driver?
>
>
>
Yes, constructors are run on dlopen.
But application should not have to ask DPDK to dlopen the bus devices.
The core principle is that dynamic build of DPDK should act the same as old
statically linked DPDK. Otherwise, the user experience is even worse, and all
the example documentation is wrong.
22/07/2019 19:13, Stephen Hemminger:
> Thomas Monjalon <thomas@monjalon.net> wrote:
> > Are the constructors run on dlopen of the bus driver?
>
> Yes, constructors are run on dlopen.
> But application should not have to ask DPDK to dlopen the bus devices.
>
> The core principle is that dynamic build of DPDK should act the same as old
> statically linked DPDK. Otherwise, the user experience is even worse, and all
> the example documentation is wrong.
OK, this is where I wanted to bring the discussion.
You are arguing against a design which is in DPDK from some early days.
So this is an interesting discussion to have.
Do we want to change the "plugin model" we have?
Or do we want to simply drop this model (dlopen calls)
and replace it with strong dynamic linking?
On Mon, 22 Jul 2019 19:31:08 +0200
Thomas Monjalon <thomas@monjalon.net> wrote:
> 22/07/2019 19:13, Stephen Hemminger:
> > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > Are the constructors run on dlopen of the bus driver?
> >
> > Yes, constructors are run on dlopen.
> > But application should not have to ask DPDK to dlopen the bus devices.
> >
> > The core principle is that dynamic build of DPDK should act the same as old
> > statically linked DPDK. Otherwise, the user experience is even worse, and all
> > the example documentation is wrong.
>
> OK, this is where I wanted to bring the discussion.
> You are arguing against a design which is in DPDK from some early days.
> So this is an interesting discussion to have.
> Do we want to change the "plugin model" we have?
> Or do we want to simply drop this model (dlopen calls)
> and replace it with strong dynamic linking?
I argue that examples should work the same with dynamic linking.
This used to work before the break out of the bus model, so it is a bug.
For distributions, this also matters. Linking with -ldpdk which is a linker
script should work.
On Mon, 22 Jul 2019 19:31:08 +0200
Thomas Monjalon <thomas@monjalon.net> wrote:
> 22/07/2019 19:13, Stephen Hemminger:
> > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > Are the constructors run on dlopen of the bus driver?
> >
> > Yes, constructors are run on dlopen.
> > But application should not have to ask DPDK to dlopen the bus devices.
> >
> > The core principle is that dynamic build of DPDK should act the same as old
> > statically linked DPDK. Otherwise, the user experience is even worse, and all
> > the example documentation is wrong.
>
> OK, this is where I wanted to bring the discussion.
> You are arguing against a design which is in DPDK from some early days.
> So this is an interesting discussion to have.
> Do we want to change the "plugin model" we have?
> Or do we want to simply drop this model (dlopen calls)
> and replace it with strong dynamic linking?
>
>
What I think should happen (and isn't is):
1. The PCI bus library is linked with --whole-archive, and --no-as-needed.
This causes constructor to be called and register the bus.
2. As part of the build process all the PCI drivers pmdinfo would get
constructed into a table of vendor/device to PMD shared library name.
3. PMD's are linked as --whole-archive, and --as-needed.
4. New code in PCI probe which looks for existing entries (static or -d)
for devices. If device is still not found it refers to the table of PMD's
(from #2) and calls dlopen for that device (and adds it to static table).
This would allow examples and customer applications to Just Work without
having to know the PMD that is present. It would also solve the problem
that currently if applications is linked with -ldpdk linker script then
all PMD's get pulled into the application address space.
22/07/2019 20:34, Stephen Hemminger:
> On Mon, 22 Jul 2019 19:31:08 +0200
> Thomas Monjalon <thomas@monjalon.net> wrote:
>
> > 22/07/2019 19:13, Stephen Hemminger:
> > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > Are the constructors run on dlopen of the bus driver?
> > >
> > > Yes, constructors are run on dlopen.
> > > But application should not have to ask DPDK to dlopen the bus devices.
> > >
> > > The core principle is that dynamic build of DPDK should act the same as old
> > > statically linked DPDK. Otherwise, the user experience is even worse, and all
> > > the example documentation is wrong.
> >
> > OK, this is where I wanted to bring the discussion.
> > You are arguing against a design which is in DPDK from some early days.
> > So this is an interesting discussion to have.
> > Do we want to change the "plugin model" we have?
> > Or do we want to simply drop this model (dlopen calls)
> > and replace it with strong dynamic linking?
>
> I argue that examples should work the same with dynamic linking.
> This used to work before the break out of the bus model, so it is a bug.
The PCI support was part of EAL, yes, but the device drivers
were plugins and already required the -d option.
> For distributions, this also matters. Linking with -ldpdk which is a linker
> script should work.
There is no longer this linker script with meson.
On Mon, Jul 22, 2019 at 11:53:26AM -0700, Stephen Hemminger wrote:
> On Mon, 22 Jul 2019 19:31:08 +0200
> Thomas Monjalon <thomas@monjalon.net> wrote:
>
> > 22/07/2019 19:13, Stephen Hemminger:
> > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > Are the constructors run on dlopen of the bus driver?
> > >
> > > Yes, constructors are run on dlopen.
> > > But application should not have to ask DPDK to dlopen the bus devices.
> > >
> > > The core principle is that dynamic build of DPDK should act the same as old
> > > statically linked DPDK. Otherwise, the user experience is even worse, and all
> > > the example documentation is wrong.
> >
> > OK, this is where I wanted to bring the discussion.
> > You are arguing against a design which is in DPDK from some early days.
> > So this is an interesting discussion to have.
> > Do we want to change the "plugin model" we have?
> > Or do we want to simply drop this model (dlopen calls)
> > and replace it with strong dynamic linking?
> >
> >
>
> What I think should happen (and isn't is):
>
> 1. The PCI bus library is linked with --whole-archive, and --no-as-needed.
> This causes constructor to be called and register the bus.
>
This should be applied to the whole of the bus drivers, not just the PCI
bus.
> 2. As part of the build process all the PCI drivers pmdinfo would get
> constructed into a table of vendor/device to PMD shared library name.
>
> 3. PMD's are linked as --whole-archive, and --as-needed.
>
I'm not sure I agree with this change to always link in all the PMDs. It
prevents an app from being used with just a subset of the drivers needed.
> 4. New code in PCI probe which looks for existing entries (static or -d)
> for devices. If device is still not found it refers to the table of PMD's
> (from #2) and calls dlopen for that device (and adds it to static table).
>
> This would allow examples and customer applications to Just Work without
> having to know the PMD that is present. It would also solve the problem
> that currently if applications is linked with -ldpdk linker script then
> all PMD's get pulled into the application address space.
>
In all this you seem to be assuming that the drivers are not picked up at
runtime from the RTE_EAL_PMD_PATH. In real world cases where a user is
building an app, and not developing DPDK itself, the DPDK libraries should
be installed in /usr(/local)/lib64 and the drivers in
.../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD
drivers are all loaded at runtime for each app, without having any
dependency on having a specific one be present, allowing a user to remove
any drivers unnecessary for the current hardware.
Did you try installing DPDK using "ninja install" or "make install" before
running any apps using it?
/Bruce
On Tue, 23 Jul 2019 13:30:33 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:
> On Mon, Jul 22, 2019 at 11:53:26AM -0700, Stephen Hemminger wrote:
> > On Mon, 22 Jul 2019 19:31:08 +0200
> > Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > > 22/07/2019 19:13, Stephen Hemminger:
> > > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > > Are the constructors run on dlopen of the bus driver?
> > > >
> > > > Yes, constructors are run on dlopen.
> > > > But application should not have to ask DPDK to dlopen the bus devices.
> > > >
> > > > The core principle is that dynamic build of DPDK should act the same as old
> > > > statically linked DPDK. Otherwise, the user experience is even worse, and all
> > > > the example documentation is wrong.
> > >
> > > OK, this is where I wanted to bring the discussion.
> > > You are arguing against a design which is in DPDK from some early days.
> > > So this is an interesting discussion to have.
> > > Do we want to change the "plugin model" we have?
> > > Or do we want to simply drop this model (dlopen calls)
> > > and replace it with strong dynamic linking?
> > >
> > >
> >
> > What I think should happen (and isn't is):
> >
> > 1. The PCI bus library is linked with --whole-archive, and --no-as-needed.
> > This causes constructor to be called and register the bus.
> >
>
> This should be applied to the whole of the bus drivers, not just the PCI
> bus.
>
> > 2. As part of the build process all the PCI drivers pmdinfo would get
> > constructed into a table of vendor/device to PMD shared library name.
> >
> > 3. PMD's are linked as --whole-archive, and --as-needed.
> >
>
> I'm not sure I agree with this change to always link in all the PMDs. It
> prevents an app from being used with just a subset of the drivers needed.
>
> > 4. New code in PCI probe which looks for existing entries (static or -d)
> > for devices. If device is still not found it refers to the table of PMD's
> > (from #2) and calls dlopen for that device (and adds it to static table).
> >
> > This would allow examples and customer applications to Just Work without
> > having to know the PMD that is present. It would also solve the problem
> > that currently if applications is linked with -ldpdk linker script then
> > all PMD's get pulled into the application address space.
> >
>
> In all this you seem to be assuming that the drivers are not picked up at
> runtime from the RTE_EAL_PMD_PATH. In real world cases where a user is
> building an app, and not developing DPDK itself, the DPDK libraries should
> be installed in /usr(/local)/lib64 and the drivers in
> .../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD
> drivers are all loaded at runtime for each app, without having any
> dependency on having a specific one be present, allowing a user to remove
> any drivers unnecessary for the current hardware.
>
> Did you try installing DPDK using "ninja install" or "make install" before
> running any apps using it?
>
> /Bruce
I was using "make install-runtime" into a local chroot directory like
a distribution package builder does.
There are multiple use cases:
1. Developer build DPDK and application together on one machine (and running on another).
Or software appliance being built on one machine and run in many environments.
Also cross builds are like this.
2. Distribution building a package (and installing in standard library locations /lib etc).
3. Demo machine where build is local and native.
DPDK seems to always focus on #3 which is least interesting for real production.
On Tue, 23 Jul 2019 09:59:04 +0200
Thomas Monjalon <thomas@monjalon.net> wrote:
> 22/07/2019 20:34, Stephen Hemminger:
> > On Mon, 22 Jul 2019 19:31:08 +0200
> > Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > > 22/07/2019 19:13, Stephen Hemminger:
> > > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > > Are the constructors run on dlopen of the bus driver?
> > > >
> > > > Yes, constructors are run on dlopen.
> > > > But application should not have to ask DPDK to dlopen the bus devices.
> > > >
> > > > The core principle is that dynamic build of DPDK should act the same as old
> > > > statically linked DPDK. Otherwise, the user experience is even worse, and all
> > > > the example documentation is wrong.
> > >
> > > OK, this is where I wanted to bring the discussion.
> > > You are arguing against a design which is in DPDK from some early days.
> > > So this is an interesting discussion to have.
> > > Do we want to change the "plugin model" we have?
> > > Or do we want to simply drop this model (dlopen calls)
> > > and replace it with strong dynamic linking?
> >
> > I argue that examples should work the same with dynamic linking.
> > This used to work before the break out of the bus model, so it is a bug.
>
> The PCI support was part of EAL, yes, but the device drivers
> were plugins and already required the -d option.
>
> > For distributions, this also matters. Linking with -ldpdk which is a linker
> > script should work.
>
> There is no longer this linker script with meson.
>
>
Ok, for usability that is a problem.
Requiring user to figure out which DPDK libraries to link with is a serious
waste of time. It should be possible to just link with -ldpdk and
distribution packages and just get the necessary libraries for the application
(no extra rte_foo_bar .so loaded at run time), and the application should
just work.
The idea that the user should link with 20 shared libraries, in the right
order and pass -d flags to eal_init to load the right PMD is user hostile.
It only makes sense if you want to invent yet another layer to manage the
ugly stuff hidden underneath. Think virt-manager versus raw KVM/QEMU.
I know it is hard, and I know not all this will make it into 19.08
but let's try and do better. The DPDK already has a reputation as being
like a super car, (ie unreliable and hard to drive). It doesn't have to be that way.
23/07/2019 20:29, Stephen Hemminger:
> On Tue, 23 Jul 2019 09:59:04 +0200
> Thomas Monjalon <thomas@monjalon.net> wrote:
>
> > 22/07/2019 20:34, Stephen Hemminger:
> > > On Mon, 22 Jul 2019 19:31:08 +0200
> > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > >
> > > > 22/07/2019 19:13, Stephen Hemminger:
> > > > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > > > Are the constructors run on dlopen of the bus driver?
> > > > >
> > > > > Yes, constructors are run on dlopen.
> > > > > But application should not have to ask DPDK to dlopen the bus devices.
> > > > >
> > > > > The core principle is that dynamic build of DPDK should act the same as old
> > > > > statically linked DPDK. Otherwise, the user experience is even worse, and all
> > > > > the example documentation is wrong.
> > > >
> > > > OK, this is where I wanted to bring the discussion.
> > > > You are arguing against a design which is in DPDK from some early days.
> > > > So this is an interesting discussion to have.
> > > > Do we want to change the "plugin model" we have?
> > > > Or do we want to simply drop this model (dlopen calls)
> > > > and replace it with strong dynamic linking?
> > >
> > > I argue that examples should work the same with dynamic linking.
> > > This used to work before the break out of the bus model, so it is a bug.
> >
> > The PCI support was part of EAL, yes, but the device drivers
> > were plugins and already required the -d option.
> >
> > > For distributions, this also matters. Linking with -ldpdk which is a linker
> > > script should work.
> >
> > There is no longer this linker script with meson.
>
> Ok, for usability that is a problem.
> Requiring user to figure out which DPDK libraries to link with is a serious
> waste of time. It should be possible to just link with -ldpdk and
> distribution packages and just get the necessary libraries for the application
> (no extra rte_foo_bar .so loaded at run time), and the application should
> just work.
>
> The idea that the user should link with 20 shared libraries, in the right
> order and pass -d flags to eal_init to load the right PMD is user hostile.
> It only makes sense if you want to invent yet another layer to manage the
> ugly stuff hidden underneath. Think virt-manager versus raw KVM/QEMU.
>
> I know it is hard, and I know not all this will make it into 19.08
> but let's try and do better. The DPDK already has a reputation as being
> like a super car, (ie unreliable and hard to drive). It doesn't have to be that way.
About the build-time link, it is already improved thanks
to the pkgconfig file generated by meson.
The only issue I understand is the runtime need for -d,
which is a feature, and could be removed.
On Tue, 23 Jul 2019 13:30:33 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:
> On Mon, Jul 22, 2019 at 11:53:26AM -0700, Stephen Hemminger wrote:
> > On Mon, 22 Jul 2019 19:31:08 +0200
> > Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > > 22/07/2019 19:13, Stephen Hemminger:
> > > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > > Are the constructors run on dlopen of the bus driver?
> > > >
> > > > Yes, constructors are run on dlopen.
> > > > But application should not have to ask DPDK to dlopen the bus devices.
> > > >
> > > > The core principle is that dynamic build of DPDK should act the same as old
> > > > statically linked DPDK. Otherwise, the user experience is even worse, and all
> > > > the example documentation is wrong.
> > >
> > > OK, this is where I wanted to bring the discussion.
> > > You are arguing against a design which is in DPDK from some early days.
> > > So this is an interesting discussion to have.
> > > Do we want to change the "plugin model" we have?
> > > Or do we want to simply drop this model (dlopen calls)
> > > and replace it with strong dynamic linking?
> > >
> > >
> >
> > What I think should happen (and isn't is):
> >
> > 1. The PCI bus library is linked with --whole-archive, and --no-as-needed.
> > This causes constructor to be called and register the bus.
> >
>
> This should be applied to the whole of the bus drivers, not just the PCI
> bus.
>
> > 2. As part of the build process all the PCI drivers pmdinfo would get
> > constructed into a table of vendor/device to PMD shared library name.
> >
> > 3. PMD's are linked as --whole-archive, and --as-needed.
> >
>
> I'm not sure I agree with this change to always link in all the PMDs. It
> prevents an app from being used with just a subset of the drivers needed.
>
> > 4. New code in PCI probe which looks for existing entries (static or -d)
> > for devices. If device is still not found it refers to the table of PMD's
> > (from #2) and calls dlopen for that device (and adds it to static table).
> >
> > This would allow examples and customer applications to Just Work without
> > having to know the PMD that is present. It would also solve the problem
> > that currently if applications is linked with -ldpdk linker script then
> > all PMD's get pulled into the application address space.
> >
>
> In all this you seem to be assuming that the drivers are not picked up at
> runtime from the RTE_EAL_PMD_PATH. In real world cases where a user is
> building an app, and not developing DPDK itself, the DPDK libraries should
> be installed in /usr(/local)/lib64 and the drivers in
> .../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD
> drivers are all loaded at runtime for each app, without having any
> dependency on having a specific one be present, allowing a user to remove
> any drivers unnecessary for the current hardware.
Looking at the plugin loading, the problem is it loads every PMD not just
those that are going to be used. Isn't this a problem with a distribution
model on an embedded system? Not everyone has virtual memory space to burn.
On Tue, Jul 23, 2019 at 11:11:14AM -0700, Stephen Hemminger wrote:
> On Tue, 23 Jul 2019 13:30:33 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
>
> > On Mon, Jul 22, 2019 at 11:53:26AM -0700, Stephen Hemminger wrote:
> > > On Mon, 22 Jul 2019 19:31:08 +0200
> > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > >
> > > > 22/07/2019 19:13, Stephen Hemminger:
> > > > > Thomas Monjalon <thomas@monjalon.net> wrote:
> > > > > > Are the constructors run on dlopen of the bus driver?
> > > > >
> > > > > Yes, constructors are run on dlopen.
> > > > > But application should not have to ask DPDK to dlopen the bus devices.
> > > > >
> > > > > The core principle is that dynamic build of DPDK should act the same as old
> > > > > statically linked DPDK. Otherwise, the user experience is even worse, and all
> > > > > the example documentation is wrong.
> > > >
> > > > OK, this is where I wanted to bring the discussion.
> > > > You are arguing against a design which is in DPDK from some early days.
> > > > So this is an interesting discussion to have.
> > > > Do we want to change the "plugin model" we have?
> > > > Or do we want to simply drop this model (dlopen calls)
> > > > and replace it with strong dynamic linking?
> > > >
> > > >
> > >
> > > What I think should happen (and isn't is):
> > >
> > > 1. The PCI bus library is linked with --whole-archive, and --no-as-needed.
> > > This causes constructor to be called and register the bus.
> > >
> >
> > This should be applied to the whole of the bus drivers, not just the PCI
> > bus.
> >
> > > 2. As part of the build process all the PCI drivers pmdinfo would get
> > > constructed into a table of vendor/device to PMD shared library name.
> > >
> > > 3. PMD's are linked as --whole-archive, and --as-needed.
> > >
> >
> > I'm not sure I agree with this change to always link in all the PMDs. It
> > prevents an app from being used with just a subset of the drivers needed.
> >
> > > 4. New code in PCI probe which looks for existing entries (static or -d)
> > > for devices. If device is still not found it refers to the table of PMD's
> > > (from #2) and calls dlopen for that device (and adds it to static table).
> > >
> > > This would allow examples and customer applications to Just Work without
> > > having to know the PMD that is present. It would also solve the problem
> > > that currently if applications is linked with -ldpdk linker script then
> > > all PMD's get pulled into the application address space.
> > >
> >
> > In all this you seem to be assuming that the drivers are not picked up at
> > runtime from the RTE_EAL_PMD_PATH. In real world cases where a user is
> > building an app, and not developing DPDK itself, the DPDK libraries should
> > be installed in /usr(/local)/lib64 and the drivers in
> > .../lib64/dpdk/dpdk-19.08. In that case, the bus drivers and the PMD
> > drivers are all loaded at runtime for each app, without having any
> > dependency on having a specific one be present, allowing a user to remove
> > any drivers unnecessary for the current hardware.
> >
> > Did you try installing DPDK using "ninja install" or "make install" before
> > running any apps using it?
> >
> > /Bruce
>
> I was using "make install-runtime" into a local chroot directory like
> a distribution package builder does.
>
>
> There are multiple use cases:
> 1. Developer build DPDK and application together on one machine (and running on another).
> Or software appliance being built on one machine and run in many environments.
> Also cross builds are like this.
>
Sure, but the expectation here is that the resultant build is installed in
system locations, or a known location. As part of the build EAL_PMD_PATH
needs to be set to the final driver location and then the driver loading
issue requiring "-d" flag goes away.
> 2. Distribution building a package (and installing in standard library locations /lib etc).
>
No issue here, since package building is not going to be a problem and when
installed the drivers are all picked up from the EAL_PMD_PATH.
> 3. Demo machine where build is local and native.
>
> DPDK seems to always focus on #3 which is least interesting for real production.
You are missing our primary target use-case for the future -
* developer builds application using distro-supplied DPDK.
Individual developers building their own DPDK alongside their app should be
a thing of the past. DPDK needs to be come like other libraries e.g. zlib,
openssl, where app developers just pick it up from their linux distro and
not feel the need to roll their own.
Yes, there will be cases where things are more memory constrained etc. and
this model won't work, but despite this, it should be our primary target.
To get there, though, there is still work to be done, e.g. in increasing
ABI compatibility to allow easier adoption by distros, etc.
/Bruce
@@ -90,8 +90,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_STACK) += -lrte_stack
_LDLIBS-$(CONFIG_RTE_DRIVER_MEMPOOL_RING) += -lrte_mempool_ring
_LDLIBS-$(CONFIG_RTE_LIBRTE_OCTEONTX2_MEMPOOL) += -lrte_mempool_octeontx2
_LDLIBS-$(CONFIG_RTE_LIBRTE_RING) += -lrte_ring
-_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
_LDLIBS-$(CONFIG_RTE_LIBRTE_EAL) += -lrte_eal
+_LDLIBS-$(CONFIG_RTE_LIBRTE_PCI) += -lrte_pci
_LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline
_LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
_LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched