diff mbox series

[v2] build: optional NUMA and cpu counts detection

Message ID 1617181602-18293-1-git-send-email-juraj.linkes@pantheon.tech (mailing list archive)
State Superseded, archived
Delegated to: Thomas Monjalon
Headers
Series [v2] build: optional NUMA and cpu counts detection |

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/travis-robot success travis build: passed
ci/github-robot success github build: passed
ci/Intel-compilation success Compilation OK
ci/intel-Testing success Testing PASS
ci/iol-intel-Performance success Performance Testing PASS
ci/iol-abi-testing success Testing PASS
ci/iol-mellanox-Performance success Performance Testing PASS
ci/iol-testing success Testing PASS

Commit Message

Juraj Linkeš March 31, 2021, 9:06 a.m. UTC 
  Add an option to automatically discover the host's numa and cpu counts
and use those values for a non cross-build.
Give users the option to override the per-arch default values or values
from cross files by specifying them on the command line with -Dmax_lcores
and -Dmax_numa_nodes.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 MAINTAINERS                  |  2 ++
 buildtools/get-cpu-count.py  |  7 ++++++
 buildtools/get-numa-count.py | 22 +++++++++++++++++
 buildtools/meson.build       |  2 ++
 config/meson.build           | 47 ++++++++++++++++++++++++++++++++++--
 config/x86/meson.build       |  2 ++
 meson_options.txt            |  8 +++---
 7 files changed, 84 insertions(+), 6 deletions(-)
 create mode 100644 buildtools/get-cpu-count.py
 create mode 100644 buildtools/get-numa-count.py

Comments

David Christensen March 31, 2021, 7:07 p.m. UTC | #1 
On 3/31/21 2:06 AM, Juraj Linkeš wrote:
> Add an option to automatically discover the host's numa and cpu counts
> and use those values for a non cross-build.
> Give users the option to override the per-arch default values or values
> from cross files by specifying them on the command line with -Dmax_lcores
> and -Dmax_numa_nodes.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> ---
>   MAINTAINERS                  |  2 ++
>   buildtools/get-cpu-count.py  |  7 ++++++
>   buildtools/get-numa-count.py | 22 +++++++++++++++++
>   buildtools/meson.build       |  2 ++
>   config/meson.build           | 47 ++++++++++++++++++++++++++++++++++--
>   config/x86/meson.build       |  2 ++
>   meson_options.txt            |  8 +++---
>   7 files changed, 84 insertions(+), 6 deletions(-)
>   create mode 100644 buildtools/get-cpu-count.py
>   create mode 100644 buildtools/get-numa-count.py
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 0ec5588540..7270f33cf5 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -99,6 +99,8 @@ F: meson_options.txt
>   F: config/
>   F: buildtools/chkincs/
>   F: buildtools/call-sphinx-build.py
> +F: buildtools/get-cpu-count.py
> +F: buildtools/get-numa-count.py
>   F: buildtools/list-dir-globs.py
>   F: buildtools/pkg-config/
>   F: buildtools/symlink-drivers-solibs.sh
> diff --git a/buildtools/get-cpu-count.py b/buildtools/get-cpu-count.py
> new file mode 100644
> index 0000000000..317b32088f
> --- /dev/null
> +++ b/buildtools/get-cpu-count.py
> @@ -0,0 +1,7 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2021 PANTHEON.tech s.r.o.
> +
> +import os
> +
> +print(os.cpu_count())

Returns the expected value on a P9 system.

> diff --git a/buildtools/get-numa-count.py b/buildtools/get-numa-count.py
> new file mode 100644
> index 0000000000..77ef2b9f24
> --- /dev/null
> +++ b/buildtools/get-numa-count.py
> @@ -0,0 +1,22 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2021 PANTHEON.tech s.r.o.
> +
> +import ctypes
> +import glob
> +import os
> +import subprocess
> +
> +if os.name == 'posix':
> +    if os.path.isdir('/sys/devices/system/node'):
> +        print(len(glob.glob('/sys/devices/system/node/node*')))
> +    else:
> +        subprocess.run(['sysctl', '-n', 'vm.ndomains'])
> +
> +elif os.name == 'nt':
> +    libkernel32 = ctypes.windll.kernel32
> +
> +    count = ctypes.c_ulong()
> +
> +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> +    print(count.value + 1)

Does not return the expected value on my P9 system (NUMA nodes are not 
contiguous).  Got 6, expect to see at least 8 otherwise I can't use 
lcores on NUMA node 8.

$ python3 ./get-numa-count.py
6
$ lscpu
Architecture:        ppc64le
Byte Order:          Little Endian
CPU(s):              128
On-line CPU(s) list: 0-127
Thread(s) per core:  4
Core(s) per socket:  16
Socket(s):           2
NUMA node(s):        6
Model:               2.3 (pvr 004e 1203)
Model name:          POWER9, altivec supported
CPU max MHz:         3800.0000
CPU min MHz:         2300.0000
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            10240K
NUMA node0 CPU(s):   0-63
NUMA node8 CPU(s):   64-127
NUMA node252 CPU(s):
NUMA node253 CPU(s):
NUMA node254 CPU(s):
NUMA node255 CPU(s):

See attached ls-node.txt.gz for full directory structure on a P9 system.

Dave
Juraj Linkeš April 16, 2021, 12:43 p.m. UTC | #2 
I didn't notice this e-mail, since it was sent just to dev@dpdk.org and not to me explicitly, which left it unnoticed by my filters, unfortunately.

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of David Christensen
> Sent: Wednesday, March 31, 2021 9:07 PM
> To: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2] build: optional NUMA and cpu counts
> detection
> 
> 
> 
> On 3/31/21 2:06 AM, Juraj Linkeš wrote:
> > Add an option to automatically discover the host's numa and cpu counts
> > and use those values for a non cross-build.
> > Give users the option to override the per-arch default values or
> > values from cross files by specifying them on the command line with
> > -Dmax_lcores and -Dmax_numa_nodes.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > ---
> >   MAINTAINERS                  |  2 ++
> >   buildtools/get-cpu-count.py  |  7 ++++++
> >   buildtools/get-numa-count.py | 22 +++++++++++++++++
> >   buildtools/meson.build       |  2 ++
> >   config/meson.build           | 47 ++++++++++++++++++++++++++++++++++--
> >   config/x86/meson.build       |  2 ++
> >   meson_options.txt            |  8 +++---
> >   7 files changed, 84 insertions(+), 6 deletions(-)
> >   create mode 100644 buildtools/get-cpu-count.py
> >   create mode 100644 buildtools/get-numa-count.py
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS index 0ec5588540..7270f33cf5
> > 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -99,6 +99,8 @@ F: meson_options.txt
> >   F: config/
> >   F: buildtools/chkincs/
> >   F: buildtools/call-sphinx-build.py
> > +F: buildtools/get-cpu-count.py
> > +F: buildtools/get-numa-count.py
> >   F: buildtools/list-dir-globs.py
> >   F: buildtools/pkg-config/
> >   F: buildtools/symlink-drivers-solibs.sh
> > diff --git a/buildtools/get-cpu-count.py b/buildtools/get-cpu-count.py
> > new file mode 100644 index 0000000000..317b32088f
> > --- /dev/null
> > +++ b/buildtools/get-cpu-count.py
> > @@ -0,0 +1,7 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2021
> > +PANTHEON.tech s.r.o.
> > +
> > +import os
> > +
> > +print(os.cpu_count())
> 
> Returns the expected value on a P9 system.
> 
> > diff --git a/buildtools/get-numa-count.py
> > b/buildtools/get-numa-count.py new file mode 100644 index
> > 0000000000..77ef2b9f24
> > --- /dev/null
> > +++ b/buildtools/get-numa-count.py
> > @@ -0,0 +1,22 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2021
> > +PANTHEON.tech s.r.o.
> > +
> > +import ctypes
> > +import glob
> > +import os
> > +import subprocess
> > +
> > +if os.name == 'posix':
> > +    if os.path.isdir('/sys/devices/system/node'):
> > +        print(len(glob.glob('/sys/devices/system/node/node*')))
> > +    else:
> > +        subprocess.run(['sysctl', '-n', 'vm.ndomains'])
> > +
> > +elif os.name == 'nt':
> > +    libkernel32 = ctypes.windll.kernel32
> > +
> > +    count = ctypes.c_ulong()
> > +
> > +    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > +    print(count.value + 1)
> 
> Does not return the expected value on my P9 system (NUMA nodes are not
> contiguous).  Got 6, expect to see at least 8 otherwise I can't use lcores on
> NUMA node 8.
> 

I guess the problem here is that we're not really setting the number of NUMA nodes that are usable, but the highest NUMA that is usable?

In that case, the proper config in should be 256 so that even node255 is usable, is that right?

> $ python3 ./get-numa-count.py
> 6
> $ lscpu
> Architecture:        ppc64le
> Byte Order:          Little Endian
> CPU(s):              128
> On-line CPU(s) list: 0-127
> Thread(s) per core:  4
> Core(s) per socket:  16
> Socket(s):           2
> NUMA node(s):        6
> Model:               2.3 (pvr 004e 1203)
> Model name:          POWER9, altivec supported
> CPU max MHz:         3800.0000
> CPU min MHz:         2300.0000
> L1d cache:           32K
> L1i cache:           32K
> L2 cache:            512K
> L3 cache:            10240K
> NUMA node0 CPU(s):   0-63
> NUMA node8 CPU(s):   64-127
> NUMA node252 CPU(s):
> NUMA node253 CPU(s):
> NUMA node254 CPU(s):
> NUMA node255 CPU(s):
> 
> See attached ls-node.txt.gz for full directory structure on a P9 system.
> 
> Dave
diff mbox series

Patch

diff --git a/MAINTAINERS b/MAINTAINERS
index 0ec5588540..7270f33cf5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -99,6 +99,8 @@  F: meson_options.txt
 F: config/
 F: buildtools/chkincs/
 F: buildtools/call-sphinx-build.py
+F: buildtools/get-cpu-count.py
+F: buildtools/get-numa-count.py
 F: buildtools/list-dir-globs.py
 F: buildtools/pkg-config/
 F: buildtools/symlink-drivers-solibs.sh
diff --git a/buildtools/get-cpu-count.py b/buildtools/get-cpu-count.py
new file mode 100644
index 0000000000..317b32088f
--- /dev/null
+++ b/buildtools/get-cpu-count.py
@@ -0,0 +1,7 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 PANTHEON.tech s.r.o.
+
+import os
+
+print(os.cpu_count())
diff --git a/buildtools/get-numa-count.py b/buildtools/get-numa-count.py
new file mode 100644
index 0000000000..77ef2b9f24
--- /dev/null
+++ b/buildtools/get-numa-count.py
@@ -0,0 +1,22 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 PANTHEON.tech s.r.o.
+
+import ctypes
+import glob
+import os
+import subprocess
+
+if os.name == 'posix':
+    if os.path.isdir('/sys/devices/system/node'):
+        print(len(glob.glob('/sys/devices/system/node/node*')))
+    else:
+        subprocess.run(['sysctl', '-n', 'vm.ndomains'])
+
+elif os.name == 'nt':
+    libkernel32 = ctypes.windll.kernel32
+
+    count = ctypes.c_ulong()
+
+    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
+    print(count.value + 1)
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 9c9347457a..acbcec0cc9 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -16,6 +16,8 @@  else
 endif
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+get_cpu_count_cmd = py3 + files('get-cpu-count.py')
+get_numa_count_cmd = py3 + files('get-numa-count.py')
 
 # select library and object file format
 pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
diff --git a/config/meson.build b/config/meson.build
index 66a2edcc47..65fc1c1d38 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -243,8 +243,6 @@  foreach arg: warning_flags
 endforeach
 
 # set other values pulled from the build options
-dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
-dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
 dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
 dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
 dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
@@ -278,6 +276,51 @@  if meson.is_cross_build()
 	endif
 endif
 
+max_lcores = get_option('max_lcores')
+if max_lcores > 0
+	# Overwrite the default value from arch_subdir with user input
+	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+elif max_lcores == -1
+	# Overwrite the default value with discovered values
+	if meson.is_cross_build()
+		error('Discovery of max_lcores is not supported for cross-compilation.')
+	endif
+	# Discovery makes sense only for non-cross builds
+	max_lcores = run_command(get_cpu_count_cmd).stdout().to_int()
+	min_lcores = 2
+	# DPDK must be build for at least 2 cores
+	if max_lcores < min_lcores
+		message('Found less than @0@ cores, building for @0@ cores'.format(min_lcores))
+		max_lcores = min_lcores
+	else
+		message('Found @0@ cores'.format(max_lcores))
+	endif
+	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+endif
+
+max_numa_nodes = get_option('max_numa_nodes')
+if max_numa_nodes > 0
+	# Overwrite the default value from arch_subdir with user input
+	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+elif max_numa_nodes == -1
+	# Overwrite the default value with discovered values
+	if meson.is_cross_build()
+		error('Discovery of max_numa_nodes not supported for cross-compilation.')
+	endif
+	# Discovery makes sense only for non-cross builds
+	max_numa_nodes = run_command(get_numa_count_cmd).stdout().to_int()
+	message('Found @0@ numa nodes'.format(max_numa_nodes))
+	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+endif
+
+# check that cpu and numa count is set and error out if it's not set
+if not dpdk_conf.has('RTE_MAX_LCORE')
+	error('Number of cores not specified.')
+endif
+if not dpdk_conf.has('RTE_MAX_NUMA_NODES')
+	error('Number of numa nodes not specified.')
+endif
+
 # set the install path for the drivers
 dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
 
diff --git a/config/x86/meson.build b/config/x86/meson.build
index 31bfa63b1c..c9f129a20e 100644
--- a/config/x86/meson.build
+++ b/config/x86/meson.build
@@ -57,3 +57,5 @@  else
 endif
 
 dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
+dpdk_conf.set('RTE_MAX_LCORE', 128)
+dpdk_conf.set('RTE_MAX_NUMA_NODES', 32)
diff --git a/meson_options.txt b/meson_options.txt
index 3b8c5d316d..39f530924e 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -24,10 +24,10 @@  option('machine', type: 'string', value: 'native',
 	description: 'set the target machine type')
 option('max_ethports', type: 'integer', value: 32,
 	description: 'maximum number of Ethernet devices')
-option('max_lcores', type: 'integer', value: 128,
-	description: 'maximum number of cores/threads supported by EAL')
-option('max_numa_nodes', type: 'integer', value: 32,
-	description: 'maximum number of NUMA nodes supported by EAL')
+option('max_lcores', type: 'integer', value: 0,
+	description: 'A positive integer sets the maximum number of cores/threads supported by EAL. The default is per-arch (or cross-compilation) default. Set to -1 to detect the number of cores on the build machine.')
+option('max_numa_nodes', type: 'integer', value: 0,
+	description: 'A positive integer sets maximum number of NUMA nodes supported by EAL. The default is per-arch (or cross-compilation) default. Set to -1 to detect the number of numa nodes on the build machine.')
 option('enable_trace_fp', type: 'boolean', value: false,
 	description: 'enable fast path trace points.')
 option('tests', type: 'boolean', value: true,