[v2,2/4] buildtools: get static mlx dependencies for meson

  The shell script options-ibverbs-static.sh was used with make
in forcing static linkage of ibverbs libraries.

When choosing to link with a static dependency in meson,
the generated .pc file will not force such static linkage.
The solution will rely on using this script in meson.

If linking with libraries installed in a non-standard path,
an option -L is provided via EXTRA_LDFLAGS in case of using make.
With meson, tuning PKG_CONFIG_PATH for pkg-config should be enough.
When statically linking an application, the .pc file must save the
-L path so the application link will work without any extra option.
That's why --libs-only-l is replaced with --libs which includes -L.
Options which are neither -l or -L are filtered out because not needed
and can cause compilation issues with the legacy system using make.

The other change in this script is to move the main library file
(libiverbs.a) at the end of the list of dependencies.
It fixes some undefined references when linking a static application
using libdpdk.pc.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
 buildtools/options-ibverbs-static.sh | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
  

On Mon, Jan 27, 2020 at 04:44:00PM +0100, Thomas Monjalon wrote:
> The shell script options-ibverbs-static.sh was used with make
> in forcing static linkage of ibverbs libraries.
> 
> When choosing to link with a static dependency in meson,
> the generated .pc file will not force such static linkage.
> The solution will rely on using this script in meson.
> 
> If linking with libraries installed in a non-standard path,
> an option -L is provided via EXTRA_LDFLAGS in case of using make.
> With meson, tuning PKG_CONFIG_PATH for pkg-config should be enough.
> When statically linking an application, the .pc file must save the
> -L path so the application link will work without any extra option.
> That's why --libs-only-l is replaced with --libs which includes -L.
> Options which are neither -l or -L are filtered out because not needed
> and can cause compilation issues with the legacy system using make.
> 
> The other change in this script is to move the main library file
> (libiverbs.a) at the end of the list of dependencies.
> It fixes some undefined references when linking a static application
> using libdpdk.pc.
> 
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
>  buildtools/options-ibverbs-static.sh | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/buildtools/options-ibverbs-static.sh b/buildtools/options-ibverbs-static.sh
> index 0f285a343b..43578a37f3 100755
> --- a/buildtools/options-ibverbs-static.sh
> +++ b/buildtools/options-ibverbs-static.sh
> @@ -9,6 +9,12 @@
>  #
>  # PKG_CONFIG_PATH may be required to be set if libibverbs.pc is not installed.
>  
> -pkg-config --libs-only-l --static libibverbs |
> +lib='libibverbs'
> +deps='pthread|nl'
> +
> +pkg-config --libs --static $lib |
>  	tr '[:space:]' '\n' |
> -	sed -r '/^-l(pthread|nl)/! s,(^-l)(.*),\1:lib\2.a,'
> +	sed -r "/^-l($deps)/! s,(^-l)(.*),\1:lib\2.a," |   # explicit .a
> +	sed -n '/^-[Ll]/p' |   # extra link options may break with make
> +	sed "/$lib/d"   # move main lib at the end
> +echo -l:$lib.a

To improve my understanding of the desired end result with this patchset
applied, I tried running different build configs with make with this patch
applied. I get a build error on my system with the following settings
adjusted from default (i.e. shared build with static link of ibverbs):

diff .config.orig .config
32c32
< CONFIG_RTE_BUILD_SHARED_LIB=n
---
> CONFIG_RTE_BUILD_SHARED_LIB=y
219c219
< CONFIG_RTE_LIBRTE_MLX5_PMD=n
---
> CONFIG_RTE_LIBRTE_MLX5_PMD=y
225c225
< CONFIG_RTE_IBVERBS_LINK_STATIC=n
---
> CONFIG_RTE_IBVERBS_LINK_STATIC=y

Error message I got is below. This is on Ubuntu 19.10 with gcc 9.2:

  LD librte_pmd_mlx5.so.20.0.1
/usr/bin/ld: /lib/x86_64-linux-gnu/libmlx5.a(mlx5.c.o): relocation R_X86_64_PC32 against symbol `stderr@@GL
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make[5]: *** [/home/bruce/dpdk.org/mk/rte.lib.mk:100: librte_pmd_mlx5.so.20.0.1] Error 1
make[4]: *** [/home/bruce/dpdk.org/mk/rte.subdir.mk:35: mlx5] Error 2
make[3]: *** [/home/bruce/dpdk.org/mk/rte.subdir.mk:35: net] Error 2
make[2]: *** [/home/bruce/dpdk.org/mk/rte.sdkbuild.mk:48: drivers] Error 2
make[1]: *** [/home/bruce/dpdk.org/mk/rte.sdkroot.mk:99: all] Error 2
make: *** [Makefile:12: all] Error 2
  

29/01/2020 16:37, Bruce Richardson:
> Error message I got is below. This is on Ubuntu 19.10 with gcc 9.2:
> 
>   LD librte_pmd_mlx5.so.20.0.1
> /usr/bin/ld: /lib/x86_64-linux-gnu/libmlx5.a(mlx5.c.o): relocation R_X86_64_PC32 against symbol `stderr@@GL

I think -fPIC is missing.
Which version of rdma-core is it?
As documented, you may have to build the static libraries yourself:
	http://doc.dpdk.org/guides/nics/mlx5.html#installation
	CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja .. && ninja
  

On Wed, Jan 29, 2020 at 03:37:51PM +0000, Bruce Richardson wrote:
> On Mon, Jan 27, 2020 at 04:44:00PM +0100, Thomas Monjalon wrote:
> > The shell script options-ibverbs-static.sh was used with make
> > in forcing static linkage of ibverbs libraries.
> > 
> > When choosing to link with a static dependency in meson,
> > the generated .pc file will not force such static linkage.
> > The solution will rely on using this script in meson.
> > 
> > If linking with libraries installed in a non-standard path,
> > an option -L is provided via EXTRA_LDFLAGS in case of using make.
> > With meson, tuning PKG_CONFIG_PATH for pkg-config should be enough.
> > When statically linking an application, the .pc file must save the
> > -L path so the application link will work without any extra option.
> > That's why --libs-only-l is replaced with --libs which includes -L.
> > Options which are neither -l or -L are filtered out because not needed
> > and can cause compilation issues with the legacy system using make.
> > 
> > The other change in this script is to move the main library file
> > (libiverbs.a) at the end of the list of dependencies.
> > It fixes some undefined references when linking a static application
> > using libdpdk.pc.
> > 
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
> >  buildtools/options-ibverbs-static.sh | 10 ++++++++--
> >  1 file changed, 8 insertions(+), 2 deletions(-)
> > 
> > diff --git a/buildtools/options-ibverbs-static.sh b/buildtools/options-ibverbs-static.sh
> > index 0f285a343b..43578a37f3 100755
> > --- a/buildtools/options-ibverbs-static.sh
> > +++ b/buildtools/options-ibverbs-static.sh
> > @@ -9,6 +9,12 @@
> >  #
> >  # PKG_CONFIG_PATH may be required to be set if libibverbs.pc is not installed.
> >  
> > -pkg-config --libs-only-l --static libibverbs |
> > +lib='libibverbs'
> > +deps='pthread|nl'
> > +
> > +pkg-config --libs --static $lib |
> >  	tr '[:space:]' '\n' |
> > -	sed -r '/^-l(pthread|nl)/! s,(^-l)(.*),\1:lib\2.a,'
> > +	sed -r "/^-l($deps)/! s,(^-l)(.*),\1:lib\2.a," |   # explicit .a
> > +	sed -n '/^-[Ll]/p' |   # extra link options may break with make
> > +	sed "/$lib/d"   # move main lib at the end
> > +echo -l:$lib.a
> 
> To improve my understanding of the desired end result with this patchset
> applied, I tried running different build configs with make with this patch
> applied. I get a build error on my system with the following settings
> adjusted from default (i.e. shared build with static link of ibverbs):
> 
> diff .config.orig .config
> 32c32
> < CONFIG_RTE_BUILD_SHARED_LIB=n
> ---
> > CONFIG_RTE_BUILD_SHARED_LIB=y
> 219c219
> < CONFIG_RTE_LIBRTE_MLX5_PMD=n
> ---
> > CONFIG_RTE_LIBRTE_MLX5_PMD=y
> 225c225
> < CONFIG_RTE_IBVERBS_LINK_STATIC=n
> ---
> > CONFIG_RTE_IBVERBS_LINK_STATIC=y
> 
> Error message I got is below. This is on Ubuntu 19.10 with gcc 9.2:
> 
>   LD librte_pmd_mlx5.so.20.0.1
> /usr/bin/ld: /lib/x86_64-linux-gnu/libmlx5.a(mlx5.c.o): relocation R_X86_64_PC32 against symbol `stderr@@GL
> /usr/bin/ld: final link failed: bad value
> collect2: error: ld returned 1 exit status
> make[5]: *** [/home/bruce/dpdk.org/mk/rte.lib.mk:100: librte_pmd_mlx5.so.20.0.1] Error 1
> make[4]: *** [/home/bruce/dpdk.org/mk/rte.subdir.mk:35: mlx5] Error 2
> make[3]: *** [/home/bruce/dpdk.org/mk/rte.subdir.mk:35: net] Error 2
> make[2]: *** [/home/bruce/dpdk.org/mk/rte.sdkbuild.mk:48: drivers] Error 2
> make[1]: *** [/home/bruce/dpdk.org/mk/rte.sdkroot.mk:99: all] Error 2
> make: *** [Makefile:12: all] Error 2
> 

After applying all 4 patches, and testing with meson, I see the same issue.
It appears that the mlx4/5.a files in Ubuntu are not relocatable for
including in a shared library.

/Bruce

[1/2] Linking target drivers/librte_pmd_mlx4.so.20.0.1.
FAILED: drivers/librte_pmd_mlx4.so.20.0.1
cc  -o drivers/librte_pmd_mlx4.so.20.0.1 'drivers/a715181@@rte_pmd_mlx4@sha/meson-generated_.._rte_pmd_mlx4.pmd.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx4@sta/net_mlx4_mlx4.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx4@sta/net_mlx4_mlx4_ethdev.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx4@sta/net_mlx4_mlx4_flow.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx4@sta/net_mlx4_mlx4_intr.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx4@sta/net_mlx4_mlx4_mp.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx4@sta/net_mlx4_mlx4_mr.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx4@sta/net_mlx4_mlx4_rxq.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx4@sta/net_mlx4_mlx4_rxtx.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx4@sta/net_mlx4_mlx4_txq.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx4@sta/net_mlx4_mlx4_utils.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx4@sta/net_mlx4_mlx4_glue.c.o' -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group -Wl,-soname,librte_pmd_mlx4.so.20.0 -Wl,--no-as-needed -pthread -lm -ldl -lnuma lib/librte_ethdev.so.20.0.1 lib/librte_eal.so.20.0.1 lib/librte_kvargs.so.20.0.1 lib/librte_net.so.20.0.1 lib/librte_mbuf.so.20.0.1 lib/librte_mempool.so.20.0.1 lib/librte_ring.so.20.0.1 lib/librte_meter.so.20.0.1 drivers/librte_bus_pci.so.20.0.1 lib/librte_pci.so.20.0.1 drivers/librte_bus_vdev.so.20.0.1 -Wl,--version-script=/home/bruce/dpdk.org/drivers/net/mlx4/rte_pmd_mlx4_version.map -lpthread -l:libbnxt_re-rdmav22.a -l:libcxgb3-rdmav22.a -l:libcxgb4-rdmav22.a -l:libefa.a -l:libhns-rdmav22.a -l:libi40iw-rdmav22.a -l:libmlx4.a -l:libmlx5.a -l:libmthca-rdmav22.a -l:libnes-rdmav22.a -l:libocrdma-rdmav22.a -l:libqedr-rdmav22.a -l:libvmw_pvrdma-rdmav22.a -l:libhfi1verbs-rdmav22.a -l:libipathverbs-rdmav22.a -l:librxe-rdmav22.a -lnl-route-3 -lnl-3 -l:libibverbs.a /usr/lib/x86_64-linux-gnu/libbsd.so -Wl,--end-group '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/' -Wl,-rpath-link,/home/bruce/dpdk.org/build/lib -Wl,-rpath-link,/home/bruce/dpdk.org/build/drivers
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/libmlx4.a(mlx4.c.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
[2/2] Linking target drivers/librte_pmd_mlx5.so.20.0.1.
FAILED: drivers/librte_pmd_mlx5.so.20.0.1
cc  -o drivers/librte_pmd_mlx5.so.20.0.1 'drivers/a715181@@rte_pmd_mlx5@sha/meson-generated_.._rte_pmd_mlx5.pmd.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_ethdev.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_flow.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_flow_meter.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_flow_dv.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_flow_verbs.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_mac.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_mr.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_nl.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_rss.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_rxmode.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_rxq.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_rxtx.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_mp.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_stats.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_trigger.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_txq.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_vlan.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_devx_cmds.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_utils.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_socket.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_rxtx_vec.c.o' 'drivers/a715181@@tmp_rte_pmd_mlx5@sta/net_mlx5_mlx5_glue.c.o' -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group -Wl,-soname,librte_pmd_mlx5.so.20.0 -Wl,--no-as-needed -pthread -lm -ldl -lnuma lib/librte_ethdev.so.20.0.1 lib/librte_eal.so.20.0.1 lib/librte_kvargs.so.20.0.1 lib/librte_net.so.20.0.1 lib/librte_mbuf.so.20.0.1 lib/librte_mempool.so.20.0.1 lib/librte_ring.so.20.0.1 lib/librte_meter.so.20.0.1 drivers/librte_bus_pci.so.20.0.1 lib/librte_pci.so.20.0.1 drivers/librte_bus_vdev.so.20.0.1 lib/librte_hash.so.20.0.1 -Wl,--version-script=/home/bruce/dpdk.org/drivers/net/mlx5/rte_pmd_mlx5_version.map -lpthread -l:libbnxt_re-rdmav22.a -l:libcxgb3-rdmav22.a -l:libcxgb4-rdmav22.a -l:libefa.a -l:libhns-rdmav22.a -l:libi40iw-rdmav22.a -l:libmlx4.a -l:libmlx5.a -l:libmthca-rdmav22.a -l:libnes-rdmav22.a -l:libocrdma-rdmav22.a -l:libqedr-rdmav22.a -l:libvmw_pvrdma-rdmav22.a -l:libhfi1verbs-rdmav22.a -l:libipathverbs-rdmav22.a -l:librxe-rdmav22.a -lnl-route-3 -lnl-3 -l:libibverbs.a /usr/lib/x86_64-linux-gnu/libbsd.so -Wl,--end-group '-Wl,-rpath,$ORIGIN/../lib:$ORIGIN/' -Wl,-rpath-link,/home/bruce/dpdk.org/build/lib -Wl,-rpath-link,/home/bruce/dpdk.org/build/drivers
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/libmlx5.a(mlx5.c.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
  

On Wed, Jan 29, 2020 at 06:48:14PM +0100, Thomas Monjalon wrote:
> 29/01/2020 16:37, Bruce Richardson:
> > Error message I got is below. This is on Ubuntu 19.10 with gcc 9.2:
> > 
> >   LD librte_pmd_mlx5.so.20.0.1
> > /usr/bin/ld: /lib/x86_64-linux-gnu/libmlx5.a(mlx5.c.o): relocation R_X86_64_PC32 against symbol `stderr@@GL
> 
> I think -fPIC is missing.
> Which version of rdma-core is it?
> As documented, you may have to build the static libraries yourself:
> 	http://doc.dpdk.org/guides/nics/mlx5.html#installation
> 	CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja .. && ninja
> 
Yes, that appears to be necessary. :-(

What is the big advantage of doing this over default linking using standard
packages, especially since for end apps the pkg-config file is taking care
of providing all the correct linker args?

/Bruce
  

29/01/2020 18:50, Bruce Richardson:
> On Wed, Jan 29, 2020 at 06:48:14PM +0100, Thomas Monjalon wrote:
> > 29/01/2020 16:37, Bruce Richardson:
> > > Error message I got is below. This is on Ubuntu 19.10 with gcc 9.2:
> > > 
> > >   LD librte_pmd_mlx5.so.20.0.1
> > > /usr/bin/ld: /lib/x86_64-linux-gnu/libmlx5.a(mlx5.c.o): relocation R_X86_64_PC32 against symbol `stderr@@GL
> > 
> > I think -fPIC is missing.
> > Which version of rdma-core is it?
> > As documented, you may have to build the static libraries yourself:
> > 	http://doc.dpdk.org/guides/nics/mlx5.html#installation
> > 	CFLAGS=-fPIC cmake -DIN_PLACE=1 -DENABLE_STATIC=1 -GNinja .. && ninja
> > 
> Yes, that appears to be necessary. :-(
> 
> What is the big advantage of doing this over default linking using standard
> packages, especially since for end apps the pkg-config file is taking care
> of providing all the correct linker args?

It does not change anything for compilation. It's all about packaging.
It may be easier to package in some environments, by removing one dependency.