[6/6] doc: guide for Windows build using MinGW-w64

  Instructions for different toolchains presented as options on the
corresponging steps of the guide, so that common parts may be reused.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
  

31/01/2020 04:07, Dmitry Kozlyuk:
> Instructions for different toolchains presented as options on the
> corresponging steps of the guide, so that common parts may be reused.
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
[...]
> --- a/doc/guides/windows_gsg/build_dpdk.rst
> +++ b/doc/guides/windows_gsg/build_dpdk.rst
> -The DPDK and its applications require the Clang-LLVM C compiler
> -and Microsoft MSVC linker.
> +Building the DPDK and its applications requires one of the following
> +environments:
> +
> +* The Clang-LLVM C compiler and Microsoft MSVC linker.
> +* The MinGW-w64 toolchain (either native or cross).
[...]
> +Option 1. Native Build on Windows
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>  
> -Specifying the compiler might be required to complete the meson command.
> +When using Clang-LLVM, specifying the compiler might be required to complete
> +the meson command:
[...]
> +When using MinGW-w64, it is sufficient to have toolchain executables in PATH:
[...]
> +Option 2. Cross-Compile with MinGW-w64
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I think having the choice between GCC and Clang for Windows is very cool.

When starting the Windows port, I asked the fundamental question of the
supported compilers. The answer was MinGW adds "non-standard" DLLs:
	https://mails.dpdk.org/archives/dev/2019-January/124236.html

Now the question is to know which one is the easiest to use?
If I understand well, MinGW brings the missing parts we are trying
to add in the DPDK repository for compliance with POSIX libraries.
  

> I think having the choice between GCC and Clang for Windows is very cool.
> 
> When starting the Windows port, I asked the fundamental question of the
> supported compilers. The answer was MinGW adds "non-standard" DLLs:
> 	https://mails.dpdk.org/archives/dev/2019-January/124236.html

Indeed, it does add a pthread implementation DLL, I even documented it. It
presents no technical or legal trouble though.

> Now the question is to know which one is the easiest to use?

The issues are a few (almost all in this message), after they are resolved,
there is no significant difference. With MinGW, more things work out of the
box. It also allows for immediate cross-compilation from Linux, which is
important while Windows drivers are unstable and can crash the system hard.

> If I understand well, MinGW brings the missing parts we are trying
> to add in the DPDK repository for compliance with POSIX libraries.

Yes, having pthread implementation out-of-the-box helps with porting EAL.
Clang can use external pthread, e. g. winpthreads [0] (not to be confused with
winpthread without S). MinGW doesn't aim to provide POSIX support, it only
brings the parts required for GCC toolchain.

Clang has structure alignment issue, which is a show-stopper for rte_mbuf,
because its carefully crafted layout is broken and becomes 3 cache-lines
instead of two:

    https://bugs.llvm.org/show_bug.cgi?id=24383

For GCC, there is a way force rte_mbuf layout:

    https://github.com/PlushBeaver/dpdk/commit/37f052cb18d1d5d425818196d5e1d15a7ada0de0

GCC, in its turn, has an AVX bug, although it can be worked around by
force-inlining functions in rte_acl (as I did):

    https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412

[0]:
https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-libraries/winpthreads/
  

05/02/2020 00:57, Dmitry Kozlyuk:
> > I think having the choice between GCC and Clang for Windows is very cool.
> > 
> > When starting the Windows port, I asked the fundamental question of the
> > supported compilers. The answer was MinGW adds "non-standard" DLLs:
> > 	https://mails.dpdk.org/archives/dev/2019-January/124236.html
> 
> Indeed, it does add a pthread implementation DLL, I even documented it. It
> presents no technical or legal trouble though.
> 
> > Now the question is to know which one is the easiest to use?
> 
> The issues are a few (almost all in this message), after they are resolved,
> there is no significant difference. With MinGW, more things work out of the
> box. It also allows for immediate cross-compilation from Linux, which is
> important while Windows drivers are unstable and can crash the system hard.
> 
> > If I understand well, MinGW brings the missing parts we are trying
> > to add in the DPDK repository for compliance with POSIX libraries.
> 
> Yes, having pthread implementation out-of-the-box helps with porting EAL.
> Clang can use external pthread, e. g. winpthreads [0] (not to be confused with
> winpthread without S). MinGW doesn't aim to provide POSIX support, it only
> brings the parts required for GCC toolchain.
> 
> Clang has structure alignment issue, which is a show-stopper for rte_mbuf,
> because its carefully crafted layout is broken and becomes 3 cache-lines
> instead of two:
> 
>     https://bugs.llvm.org/show_bug.cgi?id=24383
> 
> For GCC, there is a way force rte_mbuf layout:
> 
>     https://github.com/PlushBeaver/dpdk/commit/37f052cb18d1d5d425818196d5e1d15a7ada0de0
> 
> GCC, in its turn, has an AVX bug, although it can be worked around by
> force-inlining functions in rte_acl (as I did):
> 
>     https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412
> 
> [0]:
> https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-libraries/winpthreads/

Thanks a lot for all these valuable informations.

I think the strategy should be to progress on both GCC and Clang
at the same time.

Please Dmitry, would you like to review Pallavi's patches
in order to make them coexist with yours?
  

> I think the strategy should be to progress on both GCC and Clang
> at the same time.

Remembered another issue: thread-local storage (TLS) with shared libraries.
Windows PE doesn't support TLS via special sections, so compilers use TLS
emulation layer. With static libraries, there are no issues described below.

The first aspect is a build-time issue of MinGW. When linking to DPDK shared
libraries, errors occur:

	undefined reference to `__emutls_v.per_lcore__rte_errno'
	undefined reference to `__emutls_v.per_lcore__rte_lcore_id'

DPDK declares per_lcore__XXX in a map file, but GCC places __thread symbols
in __emutls_v section, so the proper name to export becomes __emutls_v.XXX.
This can be worked around by using an additional version script with MinGW,
as I do in my port [0], however, the proper solution would be fixing the bug
on MinGW side [1]. MinGW already converts TLS variable names when generating
DEF files with `-Wl,--output-def` option (not used by DPDK, just a hint).

The second aspect is performance. Per [2], Win32 API TLS functions are ~10%
slower than non-emulated access on Linux, and MinGW emulation layer slows
access by another 20% (30% total). Clang emulation code is similar to
MinGW's [3], although I wasn't able to find any benchmarks. As a DPDK user, I
know that rte_lcore_id() is heavily used on the data-path, so this is severe.
My opinion is, for the first time Windows port should use whatever TLS
implementation the compiler provides, then attempt optimization. It should be
possible, because we don't need a general-purpose TLS, but only TLS for EAL
threads.

[0]: https://github.com/PlushBeaver/dpdk/blob/windows/lib/meson.build#L174
[1]: https://sourceforge.net/p/mingw-w64/bugs/827/
[2]:
https://sourceforge.net/p/mingw-w64/mailman/mingw-w64-public/thread/d72aad95-b6aa-af03-667b-5898456a5a63@gmx.com/
[3]:
https://github.com/llvm-mirror/compiler-rt/blob/master/lib/builtins/emutls.c


> Please Dmitry, would you like to review Pallavi's patches
> in order to make them coexist with yours?

Done. There are no logical conflicts, if code conflicts occur after merging
Pallivi's patchset, I'll resolve them before sending v3.
  

> Remembered another issue: thread-local storage (TLS) with shared libraries.
> Windows PE doesn't support TLS via special sections, so compilers use TLS
> emulation layer. With static libraries, there are no issues described below.
> 
> The first aspect is a build-time issue of MinGW. When linking to DPDK shared
> libraries, errors occur:
> 
> 	undefined reference to `__emutls_v.per_lcore__rte_errno'
> 	undefined reference to `__emutls_v.per_lcore__rte_lcore_id'
> 
> DPDK declares per_lcore__XXX in a map file, but GCC places __thread symbols
> in __emutls_v section, so the proper name to export becomes __emutls_v.XXX.
> This can be worked around by using an additional version script with MinGW,
> as I do in my port [0], however, the proper solution would be fixing the bug
> on MinGW side [1]. MinGW already converts TLS variable names when generating
> DEF files with `-Wl,--output-def` option (not used by DPDK, just a hint).

Did some research and AFAICT, there is not effortless solution for
efficient per-lcore variables on Windows. While MinGW-w64 has aforementioned
issues (actually, GCC on Windows does), Clang with default TLS options just
generates wrong results when exporting variables from dynamic libraries.
Demo: https://github.com/PlushBeaver/tlstest

Thread [0] claims this is a fundamental problem with PE-COFF executable
format, but I honestly lack expertise to tell if this is valid. Microsoft
docs [1] suggests that exporting __thread variables won't just work. Can
someone from Microsoft or from UNH Lab comment further?

[0]: https://sourceforge.net/p/mingw-w64/mailman/message/31777672/
[1]:
https://docs.microsoft.com/en-us/windows/win32/dlls/using-thread-local-storage-in-a-dynamic-link-library
  

Hi,

We didn't get a confirmation from Microsoft about the TLS problem.
Please could we have a status?

Should we mark shared library linkage as not supported in Windows DPDK?
Is there a difference between clang and MinGW?


17/02/2020 07:27, Dmitry Kozlyuk:
> > Remembered another issue: thread-local storage (TLS) with shared libraries.
> > Windows PE doesn't support TLS via special sections, so compilers use TLS
> > emulation layer. With static libraries, there are no issues described below.
> > 
> > The first aspect is a build-time issue of MinGW. When linking to DPDK shared
> > libraries, errors occur:
> > 
> > 	undefined reference to `__emutls_v.per_lcore__rte_errno'
> > 	undefined reference to `__emutls_v.per_lcore__rte_lcore_id'
> > 
> > DPDK declares per_lcore__XXX in a map file, but GCC places __thread symbols
> > in __emutls_v section, so the proper name to export becomes __emutls_v.XXX.
> > This can be worked around by using an additional version script with MinGW,
> > as I do in my port [0], however, the proper solution would be fixing the bug
> > on MinGW side [1]. MinGW already converts TLS variable names when generating
> > DEF files with `-Wl,--output-def` option (not used by DPDK, just a hint).
> 
> Did some research and AFAICT, there is not effortless solution for
> efficient per-lcore variables on Windows. While MinGW-w64 has aforementioned
> issues (actually, GCC on Windows does), Clang with default TLS options just
> generates wrong results when exporting variables from dynamic libraries.
> Demo: https://github.com/PlushBeaver/tlstest
> 
> Thread [0] claims this is a fundamental problem with PE-COFF executable
> format, but I honestly lack expertise to tell if this is valid. Microsoft
> docs [1] suggests that exporting __thread variables won't just work. Can
> someone from Microsoft or from UNH Lab comment further?
> 
> [0]: https://sourceforge.net/p/mingw-w64/mailman/message/31777672/
> [1]:
> https://docs.microsoft.com/en-us/windows/win32/dlls/using-thread-local-storage-in-a-dynamic-link-library