[6/6] doc: guide for Windows build using MinGW-w64
diff mbox series

Message ID 20200131030744.19596-7-dmitry.kozliuk@gmail.com
State Superseded, archived
Delegated to: Thomas Monjalon
Headers show
Series
  • MinGW-w64 support
Related show

Checks

Context Check Description
ci/Intel-compilation success Compilation OK
ci/travis-robot success Travis build: passed
ci/checkpatch success coding style OK

Commit Message

Dmitry Kozlyuk Jan. 31, 2020, 3:07 a.m. UTC
Instructions for different toolchains presented as options on the
corresponging steps of the guide, so that common parts may be reused.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>

Comments

Thomas Monjalon Feb. 4, 2020, 10:34 p.m. UTC | #1
31/01/2020 04:07, Dmitry Kozlyuk:
> Instructions for different toolchains presented as options on the
> corresponging steps of the guide, so that common parts may be reused.
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
[...]
> --- a/doc/guides/windows_gsg/build_dpdk.rst
> +++ b/doc/guides/windows_gsg/build_dpdk.rst
> -The DPDK and its applications require the Clang-LLVM C compiler
> -and Microsoft MSVC linker.
> +Building the DPDK and its applications requires one of the following
> +environments:
> +
> +* The Clang-LLVM C compiler and Microsoft MSVC linker.
> +* The MinGW-w64 toolchain (either native or cross).
[...]
> +Option 1. Native Build on Windows
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>  
> -Specifying the compiler might be required to complete the meson command.
> +When using Clang-LLVM, specifying the compiler might be required to complete
> +the meson command:
[...]
> +When using MinGW-w64, it is sufficient to have toolchain executables in PATH:
[...]
> +Option 2. Cross-Compile with MinGW-w64
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I think having the choice between GCC and Clang for Windows is very cool.

When starting the Windows port, I asked the fundamental question of the
supported compilers. The answer was MinGW adds "non-standard" DLLs:
	https://mails.dpdk.org/archives/dev/2019-January/124236.html

Now the question is to know which one is the easiest to use?
If I understand well, MinGW brings the missing parts we are trying
to add in the DPDK repository for compliance with POSIX libraries.
Dmitry Kozlyuk Feb. 4, 2020, 11:57 p.m. UTC | #2
> I think having the choice between GCC and Clang for Windows is very cool.
> 
> When starting the Windows port, I asked the fundamental question of the
> supported compilers. The answer was MinGW adds "non-standard" DLLs:
> 	https://mails.dpdk.org/archives/dev/2019-January/124236.html

Indeed, it does add a pthread implementation DLL, I even documented it. It
presents no technical or legal trouble though.

> Now the question is to know which one is the easiest to use?

The issues are a few (almost all in this message), after they are resolved,
there is no significant difference. With MinGW, more things work out of the
box. It also allows for immediate cross-compilation from Linux, which is
important while Windows drivers are unstable and can crash the system hard.

> If I understand well, MinGW brings the missing parts we are trying
> to add in the DPDK repository for compliance with POSIX libraries.

Yes, having pthread implementation out-of-the-box helps with porting EAL.
Clang can use external pthread, e. g. winpthreads [0] (not to be confused with
winpthread without S). MinGW doesn't aim to provide POSIX support, it only
brings the parts required for GCC toolchain.

Clang has structure alignment issue, which is a show-stopper for rte_mbuf,
because its carefully crafted layout is broken and becomes 3 cache-lines
instead of two:

    https://bugs.llvm.org/show_bug.cgi?id=24383

For GCC, there is a way force rte_mbuf layout:

    https://github.com/PlushBeaver/dpdk/commit/37f052cb18d1d5d425818196d5e1d15a7ada0de0

GCC, in its turn, has an AVX bug, although it can be worked around by
force-inlining functions in rte_acl (as I did):

    https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412

[0]:
https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-libraries/winpthreads/
Thomas Monjalon Feb. 5, 2020, 2:20 a.m. UTC | #3
05/02/2020 00:57, Dmitry Kozlyuk:
> > I think having the choice between GCC and Clang for Windows is very cool.
> > 
> > When starting the Windows port, I asked the fundamental question of the
> > supported compilers. The answer was MinGW adds "non-standard" DLLs:
> > 	https://mails.dpdk.org/archives/dev/2019-January/124236.html
> 
> Indeed, it does add a pthread implementation DLL, I even documented it. It
> presents no technical or legal trouble though.
> 
> > Now the question is to know which one is the easiest to use?
> 
> The issues are a few (almost all in this message), after they are resolved,
> there is no significant difference. With MinGW, more things work out of the
> box. It also allows for immediate cross-compilation from Linux, which is
> important while Windows drivers are unstable and can crash the system hard.
> 
> > If I understand well, MinGW brings the missing parts we are trying
> > to add in the DPDK repository for compliance with POSIX libraries.
> 
> Yes, having pthread implementation out-of-the-box helps with porting EAL.
> Clang can use external pthread, e. g. winpthreads [0] (not to be confused with
> winpthread without S). MinGW doesn't aim to provide POSIX support, it only
> brings the parts required for GCC toolchain.
> 
> Clang has structure alignment issue, which is a show-stopper for rte_mbuf,
> because its carefully crafted layout is broken and becomes 3 cache-lines
> instead of two:
> 
>     https://bugs.llvm.org/show_bug.cgi?id=24383
> 
> For GCC, there is a way force rte_mbuf layout:
> 
>     https://github.com/PlushBeaver/dpdk/commit/37f052cb18d1d5d425818196d5e1d15a7ada0de0
> 
> GCC, in its turn, has an AVX bug, although it can be worked around by
> force-inlining functions in rte_acl (as I did):
> 
>     https://gcc.gnu.org/bugzilla/show_bug.cgi?id=54412
> 
> [0]:
> https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-libraries/winpthreads/

Thanks a lot for all these valuable informations.

I think the strategy should be to progress on both GCC and Clang
at the same time.

Please Dmitry, would you like to review Pallavi's patches
in order to make them coexist with yours?
Dmitry Kozlyuk Feb. 9, 2020, 9:39 p.m. UTC | #4
> I think the strategy should be to progress on both GCC and Clang
> at the same time.

Remembered another issue: thread-local storage (TLS) with shared libraries.
Windows PE doesn't support TLS via special sections, so compilers use TLS
emulation layer. With static libraries, there are no issues described below.

The first aspect is a build-time issue of MinGW. When linking to DPDK shared
libraries, errors occur:

	undefined reference to `__emutls_v.per_lcore__rte_errno'
	undefined reference to `__emutls_v.per_lcore__rte_lcore_id'

DPDK declares per_lcore__XXX in a map file, but GCC places __thread symbols
in __emutls_v section, so the proper name to export becomes __emutls_v.XXX.
This can be worked around by using an additional version script with MinGW,
as I do in my port [0], however, the proper solution would be fixing the bug
on MinGW side [1]. MinGW already converts TLS variable names when generating
DEF files with `-Wl,--output-def` option (not used by DPDK, just a hint).

The second aspect is performance. Per [2], Win32 API TLS functions are ~10%
slower than non-emulated access on Linux, and MinGW emulation layer slows
access by another 20% (30% total). Clang emulation code is similar to
MinGW's [3], although I wasn't able to find any benchmarks. As a DPDK user, I
know that rte_lcore_id() is heavily used on the data-path, so this is severe.
My opinion is, for the first time Windows port should use whatever TLS
implementation the compiler provides, then attempt optimization. It should be
possible, because we don't need a general-purpose TLS, but only TLS for EAL
threads.

[0]: https://github.com/PlushBeaver/dpdk/blob/windows/lib/meson.build#L174
[1]: https://sourceforge.net/p/mingw-w64/bugs/827/
[2]:
https://sourceforge.net/p/mingw-w64/mailman/mingw-w64-public/thread/d72aad95-b6aa-af03-667b-5898456a5a63@gmx.com/
[3]:
https://github.com/llvm-mirror/compiler-rt/blob/master/lib/builtins/emutls.c


> Please Dmitry, would you like to review Pallavi's patches
> in order to make them coexist with yours?

Done. There are no logical conflicts, if code conflicts occur after merging
Pallivi's patchset, I'll resolve them before sending v3.
Dmitry Kozlyuk Feb. 17, 2020, 6:27 a.m. UTC | #5
> Remembered another issue: thread-local storage (TLS) with shared libraries.
> Windows PE doesn't support TLS via special sections, so compilers use TLS
> emulation layer. With static libraries, there are no issues described below.
> 
> The first aspect is a build-time issue of MinGW. When linking to DPDK shared
> libraries, errors occur:
> 
> 	undefined reference to `__emutls_v.per_lcore__rte_errno'
> 	undefined reference to `__emutls_v.per_lcore__rte_lcore_id'
> 
> DPDK declares per_lcore__XXX in a map file, but GCC places __thread symbols
> in __emutls_v section, so the proper name to export becomes __emutls_v.XXX.
> This can be worked around by using an additional version script with MinGW,
> as I do in my port [0], however, the proper solution would be fixing the bug
> on MinGW side [1]. MinGW already converts TLS variable names when generating
> DEF files with `-Wl,--output-def` option (not used by DPDK, just a hint).

Did some research and AFAICT, there is not effortless solution for
efficient per-lcore variables on Windows. While MinGW-w64 has aforementioned
issues (actually, GCC on Windows does), Clang with default TLS options just
generates wrong results when exporting variables from dynamic libraries.
Demo: https://github.com/PlushBeaver/tlstest

Thread [0] claims this is a fundamental problem with PE-COFF executable
format, but I honestly lack expertise to tell if this is valid. Microsoft
docs [1] suggests that exporting __thread variables won't just work. Can
someone from Microsoft or from UNH Lab comment further?

[0]: https://sourceforge.net/p/mingw-w64/mailman/message/31777672/
[1]:
https://docs.microsoft.com/en-us/windows/win32/dlls/using-thread-local-storage-in-a-dynamic-link-library

Patch
diff mbox series

diff --git a/doc/guides/windows_gsg/build_dpdk.rst b/doc/guides/windows_gsg/build_dpdk.rst
index 6711e07e2..eabc459fb 100644
--- a/doc/guides/windows_gsg/build_dpdk.rst
+++ b/doc/guides/windows_gsg/build_dpdk.rst
@@ -7,15 +7,22 @@  Compiling the DPDK Target from Source
 System Requirements
 -------------------
 
-The DPDK and its applications require the Clang-LLVM C compiler
-and Microsoft MSVC linker.
+Building the DPDK and its applications requires one of the following
+environments:
+
+* The Clang-LLVM C compiler and Microsoft MSVC linker.
+* The MinGW-w64 toolchain (either native or cross).
+
 The Meson Build system is used to prepare the sources for compilation
 with the Ninja backend.
 The installation of these tools is covered in this section.
 
 
+Option 1. Clang-LLVM C Compiler and Microsoft MSVC Linker
+---------------------------------------------------------
+
 Install the Compiler
---------------------
+~~~~~~~~~~~~~~~~~~~~
 
 Download and install the clang compiler from
 `LLVM website <http://releases.llvm.org/download.html>`_.
@@ -25,7 +32,7 @@  For example, Clang-LLVM direct download link::
 
 
 Install the Linker
-------------------
+~~~~~~~~~~~~~~~~~~
 
 Download and install the Build Tools for Visual Studio to link and build the
 files on windows,
@@ -34,6 +41,15 @@  When installing build tools, select the "Visual C++ build tools" option
 and ensure the Windows SDK is selected.
 
 
+Option 2. MinGW-w64 Toolchain
+-----------------------------
+
+Obtain the latest version from
+`MinGW-w64 website <http://mingw-w64.org/doku.php/download>`_.
+On Windows, install to a folder without spaces in its name, like ``C:\MinGW``.
+This path is assumed for the rest of this guide.
+
+
 Install the Build System
 ------------------------
 
@@ -56,23 +72,41 @@  Build the code
 The build environment is setup to build the EAL and the helloworld example by
 default.
 
-Using the ninja backend
-~~~~~~~~~~~~~~~~~~~~~~~~
+Option 1. Native Build on Windows
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Specifying the compiler might be required to complete the meson command.
+When using Clang-LLVM, specifying the compiler might be required to complete
+the meson command:
 
 .. code-block:: console
 
     set CC=clang
 
+When using MinGW-w64, it is sufficient to have toolchain executables in PATH:
+
+.. code-block:: console
+
+    set PATH=C:\MinGW\mingw64\bin;%PATH%
+
 To compile the examples, the flag ``-Dexamples`` is required.
 
 .. code-block:: console
 
     cd C:\Users\me\dpdk
     meson -Dexamples=helloworld build
-    cd build
-    ninja
+    ninja -C build
+
+Option 2. Cross-Compile with MinGW-w64
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The cross-file option must be specified for Meson.
+Depending on the distribution, paths in this file may need adjustments.
+
+.. code-block:: console
+
+    meson --cross-file meson_mingw.txt -Dexamples=helloworld build
+    ninja -C build
+
 
 Run the helloworld example
 ==========================
@@ -87,3 +121,8 @@  Navigate to the examples in the build directory and run `dpdk-helloworld.exe`.
     hello from core 3
     hello from core 0
     hello from core 2
+
+Note for MinGW-w64: applications are linked to ``libwinpthread-1.dll``
+by default. To run the example, either add toolchain executables directory
+to the PATH or copy the library to the working directory.
+Alternatively, static linking may be used (mind the LGPLv2.1 license).