devtools/test-meson-builds: allow custom set of examples
Checks
Commit Message
To test the installation process of DPDK using "ninja install"
test-meson-builds.sh builds a subset of the examples using "make". To allow
more flexibility for people testing, allow the set of examples chosen for
this make test to be overridden using variable "DPDK_BUILD_TEST_EXAMPLES"
in the environment.
Since a number of example apps link against drivers directly even for
shared builds, we need to ensure that LD_LIBRARY_PATH points to the main
DPDK lib folder so any dependencies of those drivers can be found e.g. that
the PCI/vdev bus driver .so is found. [All drivers are symlinked from
drivers dir back to lib dir on install, so only one dir rather than two is
needed in the path.]
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
devtools/test-meson-builds.sh | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
Comments
On Tue, Oct 27, 2020 at 6:39 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> To test the installation process of DPDK using "ninja install"
> test-meson-builds.sh builds a subset of the examples using "make". To allow
> more flexibility for people testing, allow the set of examples chosen for
> this make test to be overridden using variable "DPDK_BUILD_TEST_EXAMPLES"
> in the environment.
>
> Since a number of example apps link against drivers directly even for
> shared builds, we need to ensure that LD_LIBRARY_PATH points to the main
> DPDK lib folder so any dependencies of those drivers can be found e.g. that
> the PCI/vdev bus driver .so is found. [All drivers are symlinked from
> drivers dir back to lib dir on install, so only one dir rather than two is
> needed in the path.]
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
> devtools/test-meson-builds.sh | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/devtools/test-meson-builds.sh b/devtools/test-meson-builds.sh
> index a87de635a2..b9cde5b366 100755
> --- a/devtools/test-meson-builds.sh
> +++ b/devtools/test-meson-builds.sh
> @@ -238,11 +238,15 @@ fi
> load_env cc
> pc_file=$(find $DESTDIR -name libdpdk.pc)
> export PKG_CONFIG_PATH=$(dirname $pc_file):$PKG_CONFIG_PATH
> +libdir=$(dirname $(find $DESTDIR -name librte_eal.so))
> +export LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
> +
> +examples_to_test=${DPDK_BUILD_TEST_EXAMPLES:-"cmdline helloworld l2fwd l3fwd skeleton timer"}
>
> # if pkg-config defines the necessary flags, test building some examples
> if pkg-config --define-prefix libdpdk >/dev/null 2>&1; then
> export PKGCONF="pkg-config --define-prefix"
> - for example in cmdline helloworld l2fwd l3fwd skeleton timer; do
> + for example in $examples_to_test; do
> echo "## Building $example"
> $MAKE -C $DESTDIR/usr/local/share/dpdk/examples/$example clean shared static
> done
> --
> 2.25.1
>
I had something similar in store for a while, happy to get a better fix.
Acked-by: David Marchand <david.marchand@redhat.com>
Caught more issues when compiling examples externally, I will send
fixes on kni, l2fwd-crypto, vhost and vhost_blk.
27/10/2020 18:38, Bruce Richardson:
> To test the installation process of DPDK using "ninja install"
> test-meson-builds.sh builds a subset of the examples using "make". To allow
> more flexibility for people testing, allow the set of examples chosen for
> this make test to be overridden using variable "DPDK_BUILD_TEST_EXAMPLES"
> in the environment.
>
> Since a number of example apps link against drivers directly even for
> shared builds, we need to ensure that LD_LIBRARY_PATH points to the main
> DPDK lib folder so any dependencies of those drivers can be found e.g. that
> the PCI/vdev bus driver .so is found. [All drivers are symlinked from
> drivers dir back to lib dir on install, so only one dir rather than two is
> needed in the path.]
[...]
> +libdir=$(dirname $(find $DESTDIR -name librte_eal.so))
> +export LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
I don't get why libdir is required for some examples,
and not for others? The pkg-config file is not enough?
> +examples_to_test=${DPDK_BUILD_TEST_EXAMPLES:-"cmdline helloworld l2fwd l3fwd skeleton timer"}
It makes me think that we should rename TEST_MESON_BUILD_VERY_VERBOSE
to DPDK_BUILD_TEST_VERY_VERBOSE for consistency.
On Mon, Nov 09, 2020 at 06:09:51PM +0100, Thomas Monjalon wrote:
> 27/10/2020 18:38, Bruce Richardson:
> > To test the installation process of DPDK using "ninja install"
> > test-meson-builds.sh builds a subset of the examples using "make". To allow
> > more flexibility for people testing, allow the set of examples chosen for
> > this make test to be overridden using variable "DPDK_BUILD_TEST_EXAMPLES"
> > in the environment.
> >
> > Since a number of example apps link against drivers directly even for
> > shared builds, we need to ensure that LD_LIBRARY_PATH points to the main
> > DPDK lib folder so any dependencies of those drivers can be found e.g. that
> > the PCI/vdev bus driver .so is found. [All drivers are symlinked from
> > drivers dir back to lib dir on install, so only one dir rather than two is
> > needed in the path.]
> [...]
> > +libdir=$(dirname $(find $DESTDIR -name librte_eal.so))
> > +export LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
>
> I don't get why libdir is required for some examples,
> and not for others? The pkg-config file is not enough?
>
It's only needed for examples that link against drivers directly.
I believe it's needed in those cases, because app linker flags (including
e.g. -lrte_pmd_bond) occur before the pkg-config flags, which means that
the linker at that point does not have the path to find the dependencies of
the driver. [In a normal build, this wouldn't be necessary because the
library directory would be a standard path]
> > +examples_to_test=${DPDK_BUILD_TEST_EXAMPLES:-"cmdline helloworld l2fwd l3fwd skeleton timer"}
>
> It makes me think that we should rename TEST_MESON_BUILD_VERY_VERBOSE
> to DPDK_BUILD_TEST_VERY_VERBOSE for consistency.
>
Sure.
09/11/2020 19:02, Bruce Richardson:
> On Mon, Nov 09, 2020 at 06:09:51PM +0100, Thomas Monjalon wrote:
> > 27/10/2020 18:38, Bruce Richardson:
> > > To test the installation process of DPDK using "ninja install"
> > > test-meson-builds.sh builds a subset of the examples using "make". To allow
> > > more flexibility for people testing, allow the set of examples chosen for
> > > this make test to be overridden using variable "DPDK_BUILD_TEST_EXAMPLES"
> > > in the environment.
> > >
> > > Since a number of example apps link against drivers directly even for
> > > shared builds, we need to ensure that LD_LIBRARY_PATH points to the main
> > > DPDK lib folder so any dependencies of those drivers can be found e.g. that
> > > the PCI/vdev bus driver .so is found. [All drivers are symlinked from
> > > drivers dir back to lib dir on install, so only one dir rather than two is
> > > needed in the path.]
> > [...]
> > > +libdir=$(dirname $(find $DESTDIR -name librte_eal.so))
> > > +export LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
> >
> > I don't get why libdir is required for some examples,
> > and not for others? The pkg-config file is not enough?
> >
>
> It's only needed for examples that link against drivers directly.
>
> I believe it's needed in those cases, because app linker flags (including
> e.g. -lrte_pmd_bond) occur before the pkg-config flags, which means that
> the linker at that point does not have the path to find the dependencies of
> the driver. [In a normal build, this wouldn't be necessary because the
> library directory would be a standard path]
If it's just a matter of ordering,
it would be a better example to fix the ordering in the Makefile,
isn't it?
On Mon, Nov 09, 2020 at 08:26:10PM +0100, Thomas Monjalon wrote:
> 09/11/2020 19:02, Bruce Richardson:
> > On Mon, Nov 09, 2020 at 06:09:51PM +0100, Thomas Monjalon wrote:
> > > 27/10/2020 18:38, Bruce Richardson:
> > > > To test the installation process of DPDK using "ninja install"
> > > > test-meson-builds.sh builds a subset of the examples using "make".
> > > > To allow more flexibility for people testing, allow the set of
> > > > examples chosen for this make test to be overridden using variable
> > > > "DPDK_BUILD_TEST_EXAMPLES" in the environment.
> > > >
> > > > Since a number of example apps link against drivers directly even
> > > > for shared builds, we need to ensure that LD_LIBRARY_PATH points to
> > > > the main DPDK lib folder so any dependencies of those drivers can
> > > > be found e.g. that the PCI/vdev bus driver .so is found. [All
> > > > drivers are symlinked from drivers dir back to lib dir on install,
> > > > so only one dir rather than two is needed in the path.]
> > > [...]
> > > > +libdir=$(dirname $(find $DESTDIR -name librte_eal.so)) +export
> > > > LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
> > >
> > > I don't get why libdir is required for some examples, and not for
> > > others? The pkg-config file is not enough?
> > >
> >
> > It's only needed for examples that link against drivers directly.
> >
> > I believe it's needed in those cases, because app linker flags
> > (including e.g. -lrte_pmd_bond) occur before the pkg-config flags,
> > which means that the linker at that point does not have the path to
> > find the dependencies of the driver. [In a normal build, this wouldn't
> > be necessary because the library directory would be a standard path]
>
> If it's just a matter of ordering, it would be a better example to fix
> the ordering in the Makefile, isn't it?
>
I thought about that, but it seems strange to have the DPDK linker flags
appear before the application specific flags. The general style is to have
the apps own linker flags apply first, and then the list of dependencies,
so that any app linker flags take precedence. The other option is to have
two separate LDFLAG variables for the app, one for before pkg-config flags
and the other afterwards, but that is an untidy solution.
Therefore, I think it's better to keep the ordering in the examples as-is
and just set the library path in the script. It's only a single extra
assignment that is necessary because we are installing to a non-standard
path using DESTDIR.
/Bruce
10/11/2020 11:08, Bruce Richardson:
> On Mon, Nov 09, 2020 at 08:26:10PM +0100, Thomas Monjalon wrote:
> > 09/11/2020 19:02, Bruce Richardson:
> > > On Mon, Nov 09, 2020 at 06:09:51PM +0100, Thomas Monjalon wrote:
> > > > 27/10/2020 18:38, Bruce Richardson:
> > > > > To test the installation process of DPDK using "ninja install"
> > > > > test-meson-builds.sh builds a subset of the examples using "make".
> > > > > To allow more flexibility for people testing, allow the set of
> > > > > examples chosen for this make test to be overridden using variable
> > > > > "DPDK_BUILD_TEST_EXAMPLES" in the environment.
> > > > >
> > > > > Since a number of example apps link against drivers directly even
> > > > > for shared builds, we need to ensure that LD_LIBRARY_PATH points to
> > > > > the main DPDK lib folder so any dependencies of those drivers can
> > > > > be found e.g. that the PCI/vdev bus driver .so is found. [All
> > > > > drivers are symlinked from drivers dir back to lib dir on install,
> > > > > so only one dir rather than two is needed in the path.]
> > > > [...]
> > > > > +libdir=$(dirname $(find $DESTDIR -name librte_eal.so)) +export
> > > > > LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
> > > >
> > > > I don't get why libdir is required for some examples, and not for
> > > > others? The pkg-config file is not enough?
> > > >
> > >
> > > It's only needed for examples that link against drivers directly.
> > >
> > > I believe it's needed in those cases, because app linker flags
> > > (including e.g. -lrte_pmd_bond) occur before the pkg-config flags,
> > > which means that the linker at that point does not have the path to
> > > find the dependencies of the driver. [In a normal build, this wouldn't
> > > be necessary because the library directory would be a standard path]
> >
> > If it's just a matter of ordering, it would be a better example to fix
> > the ordering in the Makefile, isn't it?
> >
>
> I thought about that, but it seems strange to have the DPDK linker flags
> appear before the application specific flags. The general style is to have
> the apps own linker flags apply first, and then the list of dependencies,
> so that any app linker flags take precedence. The other option is to have
> two separate LDFLAG variables for the app, one for before pkg-config flags
> and the other afterwards, but that is an untidy solution.
>
> Therefore, I think it's better to keep the ordering in the examples as-is
> and just set the library path in the script. It's only a single extra
> assignment that is necessary because we are installing to a non-standard
> path using DESTDIR.
It is OK to add LD_LIBRARY_PATH in test-meson-builds.sh because
DPDK is "installed" with a DESTDIR prefix.
But this change is unrelated to test more examples,
it is a general fix for examples compilation.
The thing I'm missing, as I said above,
"why libdir is required for some examples, and not for others?"
I feel something wrong made linking work where it should not.
On Tue, Nov 10, 2020 at 12:25:13PM +0100, Thomas Monjalon wrote:
> 10/11/2020 11:08, Bruce Richardson:
> > On Mon, Nov 09, 2020 at 08:26:10PM +0100, Thomas Monjalon wrote:
> > > 09/11/2020 19:02, Bruce Richardson:
> > > > On Mon, Nov 09, 2020 at 06:09:51PM +0100, Thomas Monjalon wrote:
> > > > > 27/10/2020 18:38, Bruce Richardson:
> > > > > > To test the installation process of DPDK using "ninja install"
> > > > > > test-meson-builds.sh builds a subset of the examples using "make".
> > > > > > To allow more flexibility for people testing, allow the set of
> > > > > > examples chosen for this make test to be overridden using variable
> > > > > > "DPDK_BUILD_TEST_EXAMPLES" in the environment.
> > > > > >
> > > > > > Since a number of example apps link against drivers directly even
> > > > > > for shared builds, we need to ensure that LD_LIBRARY_PATH points to
> > > > > > the main DPDK lib folder so any dependencies of those drivers can
> > > > > > be found e.g. that the PCI/vdev bus driver .so is found. [All
> > > > > > drivers are symlinked from drivers dir back to lib dir on install,
> > > > > > so only one dir rather than two is needed in the path.]
> > > > > [...]
> > > > > > +libdir=$(dirname $(find $DESTDIR -name librte_eal.so)) +export
> > > > > > LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
> > > > >
> > > > > I don't get why libdir is required for some examples, and not for
> > > > > others? The pkg-config file is not enough?
> > > > >
> > > >
> > > > It's only needed for examples that link against drivers directly.
> > > >
> > > > I believe it's needed in those cases, because app linker flags
> > > > (including e.g. -lrte_pmd_bond) occur before the pkg-config flags,
> > > > which means that the linker at that point does not have the path to
> > > > find the dependencies of the driver. [In a normal build, this wouldn't
> > > > be necessary because the library directory would be a standard path]
> > >
> > > If it's just a matter of ordering, it would be a better example to fix
> > > the ordering in the Makefile, isn't it?
> > >
> >
> > I thought about that, but it seems strange to have the DPDK linker flags
> > appear before the application specific flags. The general style is to have
> > the apps own linker flags apply first, and then the list of dependencies,
> > so that any app linker flags take precedence. The other option is to have
> > two separate LDFLAG variables for the app, one for before pkg-config flags
> > and the other afterwards, but that is an untidy solution.
> >
> > Therefore, I think it's better to keep the ordering in the examples as-is
> > and just set the library path in the script. It's only a single extra
> > assignment that is necessary because we are installing to a non-standard
> > path using DESTDIR.
>
> It is OK to add LD_LIBRARY_PATH in test-meson-builds.sh because
> DPDK is "installed" with a DESTDIR prefix.
>
> But this change is unrelated to test more examples,
> it is a general fix for examples compilation.
> The thing I'm missing, as I said above,
> "why libdir is required for some examples, and not for others?"
> I feel something wrong made linking work where it should not.
>
As I explained above, libdir is required for examples that link directly
against DPDK drivers in shared builds. There are only a few examples that
do so, which is why the majority worked fine, and why this was not needed
before.
/Bruce
10/11/2020 12:34, Bruce Richardson:
> On Tue, Nov 10, 2020 at 12:25:13PM +0100, Thomas Monjalon wrote:
> > 10/11/2020 11:08, Bruce Richardson:
> > > On Mon, Nov 09, 2020 at 08:26:10PM +0100, Thomas Monjalon wrote:
> > > > 09/11/2020 19:02, Bruce Richardson:
> > > > > On Mon, Nov 09, 2020 at 06:09:51PM +0100, Thomas Monjalon wrote:
> > > > > > 27/10/2020 18:38, Bruce Richardson:
> > > > > > > To test the installation process of DPDK using "ninja install"
> > > > > > > test-meson-builds.sh builds a subset of the examples using "make".
> > > > > > > To allow more flexibility for people testing, allow the set of
> > > > > > > examples chosen for this make test to be overridden using variable
> > > > > > > "DPDK_BUILD_TEST_EXAMPLES" in the environment.
> > > > > > >
> > > > > > > Since a number of example apps link against drivers directly even
> > > > > > > for shared builds, we need to ensure that LD_LIBRARY_PATH points to
> > > > > > > the main DPDK lib folder so any dependencies of those drivers can
> > > > > > > be found e.g. that the PCI/vdev bus driver .so is found. [All
> > > > > > > drivers are symlinked from drivers dir back to lib dir on install,
> > > > > > > so only one dir rather than two is needed in the path.]
> > > > > > [...]
> > > > > > > +libdir=$(dirname $(find $DESTDIR -name librte_eal.so)) +export
> > > > > > > LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
> > > > > >
> > > > > > I don't get why libdir is required for some examples, and not for
> > > > > > others? The pkg-config file is not enough?
> > > > > >
> > > > >
> > > > > It's only needed for examples that link against drivers directly.
> > > > >
> > > > > I believe it's needed in those cases, because app linker flags
> > > > > (including e.g. -lrte_pmd_bond) occur before the pkg-config flags,
> > > > > which means that the linker at that point does not have the path to
> > > > > find the dependencies of the driver. [In a normal build, this wouldn't
> > > > > be necessary because the library directory would be a standard path]
> > > >
> > > > If it's just a matter of ordering, it would be a better example to fix
> > > > the ordering in the Makefile, isn't it?
> > > >
> > >
> > > I thought about that, but it seems strange to have the DPDK linker flags
> > > appear before the application specific flags. The general style is to have
> > > the apps own linker flags apply first, and then the list of dependencies,
> > > so that any app linker flags take precedence. The other option is to have
> > > two separate LDFLAG variables for the app, one for before pkg-config flags
> > > and the other afterwards, but that is an untidy solution.
> > >
> > > Therefore, I think it's better to keep the ordering in the examples as-is
> > > and just set the library path in the script. It's only a single extra
> > > assignment that is necessary because we are installing to a non-standard
> > > path using DESTDIR.
> >
> > It is OK to add LD_LIBRARY_PATH in test-meson-builds.sh because
> > DPDK is "installed" with a DESTDIR prefix.
> >
> > But this change is unrelated to test more examples,
> > it is a general fix for examples compilation.
> > The thing I'm missing, as I said above,
> > "why libdir is required for some examples, and not for others?"
> > I feel something wrong made linking work where it should not.
> >
>
> As I explained above, libdir is required for examples that link directly
> against DPDK drivers in shared builds. There are only a few examples that
> do so, which is why the majority worked fine, and why this was not needed
> before.
Sorry I don't understand how we link other libraries without LD_LIBRARY_PATH.
Where the library path was taken from before this change?
On Tue, Nov 10, 2020 at 02:04:21PM +0100, Thomas Monjalon wrote:
> 10/11/2020 12:34, Bruce Richardson:
> > On Tue, Nov 10, 2020 at 12:25:13PM +0100, Thomas Monjalon wrote:
> > > 10/11/2020 11:08, Bruce Richardson:
> > > > On Mon, Nov 09, 2020 at 08:26:10PM +0100, Thomas Monjalon wrote:
> > > > > 09/11/2020 19:02, Bruce Richardson:
> > > > > > On Mon, Nov 09, 2020 at 06:09:51PM +0100, Thomas Monjalon wrote:
> > > > > > > 27/10/2020 18:38, Bruce Richardson:
> > > > > > > > To test the installation process of DPDK using "ninja install"
> > > > > > > > test-meson-builds.sh builds a subset of the examples using "make".
> > > > > > > > To allow more flexibility for people testing, allow the set of
> > > > > > > > examples chosen for this make test to be overridden using variable
> > > > > > > > "DPDK_BUILD_TEST_EXAMPLES" in the environment.
> > > > > > > >
> > > > > > > > Since a number of example apps link against drivers directly even
> > > > > > > > for shared builds, we need to ensure that LD_LIBRARY_PATH points to
> > > > > > > > the main DPDK lib folder so any dependencies of those drivers can
> > > > > > > > be found e.g. that the PCI/vdev bus driver .so is found. [All
> > > > > > > > drivers are symlinked from drivers dir back to lib dir on install,
> > > > > > > > so only one dir rather than two is needed in the path.]
> > > > > > > [...]
> > > > > > > > +libdir=$(dirname $(find $DESTDIR -name librte_eal.so)) +export
> > > > > > > > LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
> > > > > > >
> > > > > > > I don't get why libdir is required for some examples, and not for
> > > > > > > others? The pkg-config file is not enough?
> > > > > > >
> > > > > >
> > > > > > It's only needed for examples that link against drivers directly.
> > > > > >
> > > > > > I believe it's needed in those cases, because app linker flags
> > > > > > (including e.g. -lrte_pmd_bond) occur before the pkg-config flags,
> > > > > > which means that the linker at that point does not have the path to
> > > > > > find the dependencies of the driver. [In a normal build, this wouldn't
> > > > > > be necessary because the library directory would be a standard path]
> > > > >
> > > > > If it's just a matter of ordering, it would be a better example to fix
> > > > > the ordering in the Makefile, isn't it?
> > > > >
> > > >
> > > > I thought about that, but it seems strange to have the DPDK linker flags
> > > > appear before the application specific flags. The general style is to have
> > > > the apps own linker flags apply first, and then the list of dependencies,
> > > > so that any app linker flags take precedence. The other option is to have
> > > > two separate LDFLAG variables for the app, one for before pkg-config flags
> > > > and the other afterwards, but that is an untidy solution.
> > > >
> > > > Therefore, I think it's better to keep the ordering in the examples as-is
> > > > and just set the library path in the script. It's only a single extra
> > > > assignment that is necessary because we are installing to a non-standard
> > > > path using DESTDIR.
> > >
> > > It is OK to add LD_LIBRARY_PATH in test-meson-builds.sh because
> > > DPDK is "installed" with a DESTDIR prefix.
> > >
> > > But this change is unrelated to test more examples,
> > > it is a general fix for examples compilation.
> > > The thing I'm missing, as I said above,
> > > "why libdir is required for some examples, and not for others?"
> > > I feel something wrong made linking work where it should not.
> > >
> >
> > As I explained above, libdir is required for examples that link directly
> > against DPDK drivers in shared builds. There are only a few examples that
> > do so, which is why the majority worked fine, and why this was not needed
> > before.
>
> Sorry I don't understand how we link other libraries without LD_LIBRARY_PATH.
> Where the library path was taken from before this change?
>
pkg-config outputs the library path via -L flag for all the other
libraries, which is why they are picked up. Right now the flags passed are
(in the case of the bonding example app):
LDFLAGS += -lrte_net_bond
...
LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
...
build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
which resolves to (roughly):
"-lrte_net_bond -L/path/to/dpdk/libs -lrte_ethdev ..."
This means that all the regular DPDK libs are found based off the "-L"
flag, but the rte_net_bond and its dependencies are not as they occur
before the -L flag.
The alternative, as you suggested previously, is to move the -lrte_net_bond
to the end of the list, but that strikes me as less elegant as the LDFLAGS
for the app need to be split into two.
A third choice is to modify the makefiles to use pkg-config "--libs-only-L"
flag to put in the relevant pkg-config path flag before the rte_net_bond.
(However, I have yet to check that all versions of pkg-config support this
flag, since we've been bitten in the past of using flags supported by only
some versions on some distros!)
LDFLAGS += $(shell $(PKGCONF) --libs-only-L libdpdk) -lrte_net_bond
Overall, given we are looking at a test script where we install to a
non-standard location, putting that location in the LD_LIBRARY_PATH seems
the easiest, sane solution. It also most closely matches real-world use,
where if one installs DPDK to a novel location, the path to the library
folder does need to be set somewhere for the runtime loader to find the
libs.
/Bruce
10/11/2020 14:19, Bruce Richardson:
> On Tue, Nov 10, 2020 at 02:04:21PM +0100, Thomas Monjalon wrote:
> > 10/11/2020 12:34, Bruce Richardson:
> > > On Tue, Nov 10, 2020 at 12:25:13PM +0100, Thomas Monjalon wrote:
> > > > 10/11/2020 11:08, Bruce Richardson:
> > > > > On Mon, Nov 09, 2020 at 08:26:10PM +0100, Thomas Monjalon wrote:
> > > > > > 09/11/2020 19:02, Bruce Richardson:
> > > > > > > On Mon, Nov 09, 2020 at 06:09:51PM +0100, Thomas Monjalon wrote:
> > > > > > > > 27/10/2020 18:38, Bruce Richardson:
> > > > > > > > > To test the installation process of DPDK using "ninja install"
> > > > > > > > > test-meson-builds.sh builds a subset of the examples using "make".
> > > > > > > > > To allow more flexibility for people testing, allow the set of
> > > > > > > > > examples chosen for this make test to be overridden using variable
> > > > > > > > > "DPDK_BUILD_TEST_EXAMPLES" in the environment.
> > > > > > > > >
> > > > > > > > > Since a number of example apps link against drivers directly even
> > > > > > > > > for shared builds, we need to ensure that LD_LIBRARY_PATH points to
> > > > > > > > > the main DPDK lib folder so any dependencies of those drivers can
> > > > > > > > > be found e.g. that the PCI/vdev bus driver .so is found. [All
> > > > > > > > > drivers are symlinked from drivers dir back to lib dir on install,
> > > > > > > > > so only one dir rather than two is needed in the path.]
> > > > > > > > [...]
> > > > > > > > > +libdir=$(dirname $(find $DESTDIR -name librte_eal.so)) +export
> > > > > > > > > LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
> > > > > > > >
> > > > > > > > I don't get why libdir is required for some examples, and not for
> > > > > > > > others? The pkg-config file is not enough?
> > > > > > > >
> > > > > > >
> > > > > > > It's only needed for examples that link against drivers directly.
> > > > > > >
> > > > > > > I believe it's needed in those cases, because app linker flags
> > > > > > > (including e.g. -lrte_pmd_bond) occur before the pkg-config flags,
> > > > > > > which means that the linker at that point does not have the path to
> > > > > > > find the dependencies of the driver. [In a normal build, this wouldn't
> > > > > > > be necessary because the library directory would be a standard path]
> > > > > >
> > > > > > If it's just a matter of ordering, it would be a better example to fix
> > > > > > the ordering in the Makefile, isn't it?
> > > > > >
> > > > >
> > > > > I thought about that, but it seems strange to have the DPDK linker flags
> > > > > appear before the application specific flags. The general style is to have
> > > > > the apps own linker flags apply first, and then the list of dependencies,
> > > > > so that any app linker flags take precedence. The other option is to have
> > > > > two separate LDFLAG variables for the app, one for before pkg-config flags
> > > > > and the other afterwards, but that is an untidy solution.
> > > > >
> > > > > Therefore, I think it's better to keep the ordering in the examples as-is
> > > > > and just set the library path in the script. It's only a single extra
> > > > > assignment that is necessary because we are installing to a non-standard
> > > > > path using DESTDIR.
> > > >
> > > > It is OK to add LD_LIBRARY_PATH in test-meson-builds.sh because
> > > > DPDK is "installed" with a DESTDIR prefix.
> > > >
> > > > But this change is unrelated to test more examples,
> > > > it is a general fix for examples compilation.
> > > > The thing I'm missing, as I said above,
> > > > "why libdir is required for some examples, and not for others?"
> > > > I feel something wrong made linking work where it should not.
> > > >
> > >
> > > As I explained above, libdir is required for examples that link directly
> > > against DPDK drivers in shared builds. There are only a few examples that
> > > do so, which is why the majority worked fine, and why this was not needed
> > > before.
> >
> > Sorry I don't understand how we link other libraries without LD_LIBRARY_PATH.
> > Where the library path was taken from before this change?
> >
>
> pkg-config outputs the library path via -L flag for all the other
> libraries, which is why they are picked up. Right now the flags passed are
> (in the case of the bonding example app):
>
> LDFLAGS += -lrte_net_bond
> ...
> LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
> ...
> build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
> $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
>
> which resolves to (roughly):
> "-lrte_net_bond -L/path/to/dpdk/libs -lrte_ethdev ..."
This is the part I still don't understand.
How it can resolve to path including DESTDIR?
In my test, it resolves to -L/usr/local/lib so it cannot work.
> This means that all the regular DPDK libs are found based off the "-L"
> flag, but the rte_net_bond and its dependencies are not as they occur
> before the -L flag.
>
> The alternative, as you suggested previously, is to move the -lrte_net_bond
> to the end of the list, but that strikes me as less elegant as the LDFLAGS
> for the app need to be split into two.
>
> A third choice is to modify the makefiles to use pkg-config "--libs-only-L"
> flag to put in the relevant pkg-config path flag before the rte_net_bond.
> (However, I have yet to check that all versions of pkg-config support this
> flag, since we've been bitten in the past of using flags supported by only
> some versions on some distros!)
>
> LDFLAGS += $(shell $(PKGCONF) --libs-only-L libdpdk) -lrte_net_bond
>
> Overall, given we are looking at a test script where we install to a
> non-standard location, putting that location in the LD_LIBRARY_PATH seems
> the easiest, sane solution. It also most closely matches real-world use,
> where if one installs DPDK to a novel location, the path to the library
> folder does need to be set somewhere for the runtime loader to find the
> libs.
I agree with setting LD_LIBRARY_PATH in the test script
because of the "non-standard location".
This is not a question.
On Tue, Nov 10, 2020 at 02:53:03PM +0100, Thomas Monjalon wrote:
> 10/11/2020 14:19, Bruce Richardson:
> > On Tue, Nov 10, 2020 at 02:04:21PM +0100, Thomas Monjalon wrote:
> > > 10/11/2020 12:34, Bruce Richardson:
> > > > On Tue, Nov 10, 2020 at 12:25:13PM +0100, Thomas Monjalon wrote:
> > > > > 10/11/2020 11:08, Bruce Richardson:
> > > > > > On Mon, Nov 09, 2020 at 08:26:10PM +0100, Thomas Monjalon wrote:
> > > > > > > 09/11/2020 19:02, Bruce Richardson:
> > > > > > > > On Mon, Nov 09, 2020 at 06:09:51PM +0100, Thomas Monjalon wrote:
> > > > > > > > > 27/10/2020 18:38, Bruce Richardson:
> > > > > > > > > > To test the installation process of DPDK using "ninja install"
> > > > > > > > > > test-meson-builds.sh builds a subset of the examples using "make".
> > > > > > > > > > To allow more flexibility for people testing, allow the set of
> > > > > > > > > > examples chosen for this make test to be overridden using variable
> > > > > > > > > > "DPDK_BUILD_TEST_EXAMPLES" in the environment.
> > > > > > > > > >
> > > > > > > > > > Since a number of example apps link against drivers directly even
> > > > > > > > > > for shared builds, we need to ensure that LD_LIBRARY_PATH points to
> > > > > > > > > > the main DPDK lib folder so any dependencies of those drivers can
> > > > > > > > > > be found e.g. that the PCI/vdev bus driver .so is found. [All
> > > > > > > > > > drivers are symlinked from drivers dir back to lib dir on install,
> > > > > > > > > > so only one dir rather than two is needed in the path.]
> > > > > > > > > [...]
> > > > > > > > > > +libdir=$(dirname $(find $DESTDIR -name librte_eal.so)) +export
> > > > > > > > > > LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
> > > > > > > > >
> > > > > > > > > I don't get why libdir is required for some examples, and not for
> > > > > > > > > others? The pkg-config file is not enough?
> > > > > > > > >
> > > > > > > >
> > > > > > > > It's only needed for examples that link against drivers directly.
> > > > > > > >
> > > > > > > > I believe it's needed in those cases, because app linker flags
> > > > > > > > (including e.g. -lrte_pmd_bond) occur before the pkg-config flags,
> > > > > > > > which means that the linker at that point does not have the path to
> > > > > > > > find the dependencies of the driver. [In a normal build, this wouldn't
> > > > > > > > be necessary because the library directory would be a standard path]
> > > > > > >
> > > > > > > If it's just a matter of ordering, it would be a better example to fix
> > > > > > > the ordering in the Makefile, isn't it?
> > > > > > >
> > > > > >
> > > > > > I thought about that, but it seems strange to have the DPDK linker flags
> > > > > > appear before the application specific flags. The general style is to have
> > > > > > the apps own linker flags apply first, and then the list of dependencies,
> > > > > > so that any app linker flags take precedence. The other option is to have
> > > > > > two separate LDFLAG variables for the app, one for before pkg-config flags
> > > > > > and the other afterwards, but that is an untidy solution.
> > > > > >
> > > > > > Therefore, I think it's better to keep the ordering in the examples as-is
> > > > > > and just set the library path in the script. It's only a single extra
> > > > > > assignment that is necessary because we are installing to a non-standard
> > > > > > path using DESTDIR.
> > > > >
> > > > > It is OK to add LD_LIBRARY_PATH in test-meson-builds.sh because
> > > > > DPDK is "installed" with a DESTDIR prefix.
> > > > >
> > > > > But this change is unrelated to test more examples,
> > > > > it is a general fix for examples compilation.
> > > > > The thing I'm missing, as I said above,
> > > > > "why libdir is required for some examples, and not for others?"
> > > > > I feel something wrong made linking work where it should not.
> > > > >
> > > >
> > > > As I explained above, libdir is required for examples that link directly
> > > > against DPDK drivers in shared builds. There are only a few examples that
> > > > do so, which is why the majority worked fine, and why this was not needed
> > > > before.
> > >
> > > Sorry I don't understand how we link other libraries without LD_LIBRARY_PATH.
> > > Where the library path was taken from before this change?
> > >
> >
> > pkg-config outputs the library path via -L flag for all the other
> > libraries, which is why they are picked up. Right now the flags passed are
> > (in the case of the bonding example app):
> >
> > LDFLAGS += -lrte_net_bond
> > ...
> > LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
> > ...
> > build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
> > $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
> >
> > which resolves to (roughly):
> > "-lrte_net_bond -L/path/to/dpdk/libs -lrte_ethdev ..."
>
> This is the part I still don't understand.
> How it can resolve to path including DESTDIR?
> In my test, it resolves to -L/usr/local/lib so it cannot work.
>
It's the --define-prefix parameter we pass to pkg-config that does the
trick.
$ export PKG_CONFIG_LIBDIR=$(pwd)/usr/local/lib/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig
$ pkg-config --libs-only-L libdpdk
-L/usr/local/lib -L/usr/lib/x86_64-linux-gnu
$ pkg-config --define-prefix --libs-only-L libdpdk
-L/home/bruce/dpdk.org/__BUILDS/build-x86-default/install/usr/local/lib -L/usr/lib/lib/x86_64-linux-gnu
This is done in test-meson-builds right before we call make for each
example:
# if pkg-config defines the necessary flags, test building some examples
if pkg-config --define-prefix libdpdk >/dev/null 2>&1; then
export PKGCONF="pkg-config --define-prefix"
...
fi
10/11/2020 15:19, Bruce Richardson:
> On Tue, Nov 10, 2020 at 02:53:03PM +0100, Thomas Monjalon wrote:
> > 10/11/2020 14:19, Bruce Richardson:
> > > On Tue, Nov 10, 2020 at 02:04:21PM +0100, Thomas Monjalon wrote:
> > > > 10/11/2020 12:34, Bruce Richardson:
> > > > > On Tue, Nov 10, 2020 at 12:25:13PM +0100, Thomas Monjalon wrote:
> > > > > > 10/11/2020 11:08, Bruce Richardson:
> > > > > > > On Mon, Nov 09, 2020 at 08:26:10PM +0100, Thomas Monjalon wrote:
> > > > > > > > 09/11/2020 19:02, Bruce Richardson:
> > > > > > > > > On Mon, Nov 09, 2020 at 06:09:51PM +0100, Thomas Monjalon wrote:
> > > > > > > > > > 27/10/2020 18:38, Bruce Richardson:
> > > > > > > > > > > To test the installation process of DPDK using "ninja install"
> > > > > > > > > > > test-meson-builds.sh builds a subset of the examples using "make".
> > > > > > > > > > > To allow more flexibility for people testing, allow the set of
> > > > > > > > > > > examples chosen for this make test to be overridden using variable
> > > > > > > > > > > "DPDK_BUILD_TEST_EXAMPLES" in the environment.
> > > > > > > > > > >
> > > > > > > > > > > Since a number of example apps link against drivers directly even
> > > > > > > > > > > for shared builds, we need to ensure that LD_LIBRARY_PATH points to
> > > > > > > > > > > the main DPDK lib folder so any dependencies of those drivers can
> > > > > > > > > > > be found e.g. that the PCI/vdev bus driver .so is found. [All
> > > > > > > > > > > drivers are symlinked from drivers dir back to lib dir on install,
> > > > > > > > > > > so only one dir rather than two is needed in the path.]
> > > > > > > > > > [...]
> > > > > > > > > > > +libdir=$(dirname $(find $DESTDIR -name librte_eal.so)) +export
> > > > > > > > > > > LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
> > > > > > > > > >
> > > > > > > > > > I don't get why libdir is required for some examples, and not for
> > > > > > > > > > others? The pkg-config file is not enough?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > It's only needed for examples that link against drivers directly.
> > > > > > > > >
> > > > > > > > > I believe it's needed in those cases, because app linker flags
> > > > > > > > > (including e.g. -lrte_pmd_bond) occur before the pkg-config flags,
> > > > > > > > > which means that the linker at that point does not have the path to
> > > > > > > > > find the dependencies of the driver. [In a normal build, this wouldn't
> > > > > > > > > be necessary because the library directory would be a standard path]
> > > > > > > >
> > > > > > > > If it's just a matter of ordering, it would be a better example to fix
> > > > > > > > the ordering in the Makefile, isn't it?
> > > > > > > >
> > > > > > >
> > > > > > > I thought about that, but it seems strange to have the DPDK linker flags
> > > > > > > appear before the application specific flags. The general style is to have
> > > > > > > the apps own linker flags apply first, and then the list of dependencies,
> > > > > > > so that any app linker flags take precedence. The other option is to have
> > > > > > > two separate LDFLAG variables for the app, one for before pkg-config flags
> > > > > > > and the other afterwards, but that is an untidy solution.
> > > > > > >
> > > > > > > Therefore, I think it's better to keep the ordering in the examples as-is
> > > > > > > and just set the library path in the script. It's only a single extra
> > > > > > > assignment that is necessary because we are installing to a non-standard
> > > > > > > path using DESTDIR.
> > > > > >
> > > > > > It is OK to add LD_LIBRARY_PATH in test-meson-builds.sh because
> > > > > > DPDK is "installed" with a DESTDIR prefix.
> > > > > >
> > > > > > But this change is unrelated to test more examples,
> > > > > > it is a general fix for examples compilation.
> > > > > > The thing I'm missing, as I said above,
> > > > > > "why libdir is required for some examples, and not for others?"
> > > > > > I feel something wrong made linking work where it should not.
> > > > > >
> > > > >
> > > > > As I explained above, libdir is required for examples that link directly
> > > > > against DPDK drivers in shared builds. There are only a few examples that
> > > > > do so, which is why the majority worked fine, and why this was not needed
> > > > > before.
> > > >
> > > > Sorry I don't understand how we link other libraries without LD_LIBRARY_PATH.
> > > > Where the library path was taken from before this change?
> > > >
> > >
> > > pkg-config outputs the library path via -L flag for all the other
> > > libraries, which is why they are picked up. Right now the flags passed are
> > > (in the case of the bonding example app):
> > >
> > > LDFLAGS += -lrte_net_bond
> > > ...
> > > LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
> > > ...
> > > build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
> > > $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
> > >
> > > which resolves to (roughly):
> > > "-lrte_net_bond -L/path/to/dpdk/libs -lrte_ethdev ..."
> >
> > This is the part I still don't understand.
> > How it can resolve to path including DESTDIR?
> > In my test, it resolves to -L/usr/local/lib so it cannot work.
> >
> It's the --define-prefix parameter we pass to pkg-config that does the
> trick.
>
> $ export PKG_CONFIG_LIBDIR=$(pwd)/usr/local/lib/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig
>
> $ pkg-config --libs-only-L libdpdk
> -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu
>
> $ pkg-config --define-prefix --libs-only-L libdpdk
> -L/home/bruce/dpdk.org/__BUILDS/build-x86-default/install/usr/local/lib -L/usr/lib/lib/x86_64-linux-gnu
>
> This is done in test-meson-builds right before we call make for each
> example:
>
> # if pkg-config defines the necessary flags, test building some examples
> if pkg-config --define-prefix libdpdk >/dev/null 2>&1; then
> export PKGCONF="pkg-config --define-prefix"
> ...
> fi
Yes this is what David suggested to me,
but it doesn't work well for me (while test-meson-builds.sh works):
PKG_CONFIG_PATH=../dpdk-build/build-x86-default/meson-private \
pkg-config --define-prefix --libs-only-L libdpdk
-L/usr/local/lib
Oh! I am testing on the build directory instead of installed one.
PKG_CONFIG_PATH=../dpdk-build/build-x86-default/install/usr/local/lib/pkgconfig \
pkg-config --define-prefix --libs-only-L libdpdk
-L../dpdk-build/build-x86-default/install/usr/local/lib
Good :)
Now THE question: what is the difference between these two .pc files?
On Tue, Nov 10, 2020 at 3:36 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> PKG_CONFIG_PATH=../dpdk-build/build-x86-default/meson-private \
> pkg-config --define-prefix --libs-only-L libdpdk
> -L/usr/local/lib
>
> Oh! I am testing on the build directory instead of installed one.
>
> PKG_CONFIG_PATH=../dpdk-build/build-x86-default/install/usr/local/lib/pkgconfig \
> pkg-config --define-prefix --libs-only-L libdpdk
> -L../dpdk-build/build-x86-default/install/usr/local/lib
>
> Good :)
> Now THE question: what is the difference between these two .pc files?
I moved libdpdk{,-libs}.pc files in different folders.
pkg-config seems to check the path to the .pc files to decide what to return.
There seems to be a combination of having $prefix in the path and
/pkgconfig as the last directory.
09/11/2020 11:01, David Marchand:
> On Tue, Oct 27, 2020 at 6:39 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > To test the installation process of DPDK using "ninja install"
> > test-meson-builds.sh builds a subset of the examples using "make". To allow
> > more flexibility for people testing, allow the set of examples chosen for
> > this make test to be overridden using variable "DPDK_BUILD_TEST_EXAMPLES"
> > in the environment.
> >
> > Since a number of example apps link against drivers directly even for
> > shared builds, we need to ensure that LD_LIBRARY_PATH points to the main
> > DPDK lib folder so any dependencies of those drivers can be found e.g. that
> > the PCI/vdev bus driver .so is found. [All drivers are symlinked from
> > drivers dir back to lib dir on install, so only one dir rather than two is
> > needed in the path.]
> >
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
>
> I had something similar in store for a while, happy to get a better fix.
> Acked-by: David Marchand <david.marchand@redhat.com>
The behaviour of pkg-config --define-prefix is kind of magic,
but that's unrelated to this patch or this script :)
Acked-by: Thomas Monjalon <thomas@monjalon.net>
Applied, thanks
@@ -238,11 +238,15 @@ fi
load_env cc
pc_file=$(find $DESTDIR -name libdpdk.pc)
export PKG_CONFIG_PATH=$(dirname $pc_file):$PKG_CONFIG_PATH
+libdir=$(dirname $(find $DESTDIR -name librte_eal.so))
+export LD_LIBRARY_PATH=$libdir:$LD_LIBRARY_PATH
+
+examples_to_test=${DPDK_BUILD_TEST_EXAMPLES:-"cmdline helloworld l2fwd l3fwd skeleton timer"}
# if pkg-config defines the necessary flags, test building some examples
if pkg-config --define-prefix libdpdk >/dev/null 2>&1; then
export PKGCONF="pkg-config --define-prefix"
- for example in cmdline helloworld l2fwd l3fwd skeleton timer; do
+ for example in $examples_to_test; do
echo "## Building $example"
$MAKE -C $DESTDIR/usr/local/share/dpdk/examples/$example clean shared static
done