[v2] build: optional NUMA and cpu counts detection
Checks
Commit Message
Add an option to automatically discover the host's numa and cpu counts
and use those values for a non cross-build.
Give users the option to override the per-arch default values or values
from cross files by specifying them on the command line with -Dmax_lcores
and -Dmax_numa_nodes.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
MAINTAINERS | 2 ++
buildtools/get-cpu-count.py | 7 ++++++
buildtools/get-numa-count.py | 22 +++++++++++++++++
buildtools/meson.build | 2 ++
config/meson.build | 47 ++++++++++++++++++++++++++++++++++--
config/x86/meson.build | 2 ++
meson_options.txt | 8 +++---
7 files changed, 84 insertions(+), 6 deletions(-)
create mode 100644 buildtools/get-cpu-count.py
create mode 100644 buildtools/get-numa-count.py
Comments
On 3/31/21 2:06 AM, Juraj Linkeš wrote:
> Add an option to automatically discover the host's numa and cpu counts
> and use those values for a non cross-build.
> Give users the option to override the per-arch default values or values
> from cross files by specifying them on the command line with -Dmax_lcores
> and -Dmax_numa_nodes.
>
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> ---
> MAINTAINERS | 2 ++
> buildtools/get-cpu-count.py | 7 ++++++
> buildtools/get-numa-count.py | 22 +++++++++++++++++
> buildtools/meson.build | 2 ++
> config/meson.build | 47 ++++++++++++++++++++++++++++++++++--
> config/x86/meson.build | 2 ++
> meson_options.txt | 8 +++---
> 7 files changed, 84 insertions(+), 6 deletions(-)
> create mode 100644 buildtools/get-cpu-count.py
> create mode 100644 buildtools/get-numa-count.py
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 0ec5588540..7270f33cf5 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -99,6 +99,8 @@ F: meson_options.txt
> F: config/
> F: buildtools/chkincs/
> F: buildtools/call-sphinx-build.py
> +F: buildtools/get-cpu-count.py
> +F: buildtools/get-numa-count.py
> F: buildtools/list-dir-globs.py
> F: buildtools/pkg-config/
> F: buildtools/symlink-drivers-solibs.sh
> diff --git a/buildtools/get-cpu-count.py b/buildtools/get-cpu-count.py
> new file mode 100644
> index 0000000000..317b32088f
> --- /dev/null
> +++ b/buildtools/get-cpu-count.py
> @@ -0,0 +1,7 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2021 PANTHEON.tech s.r.o.
> +
> +import os
> +
> +print(os.cpu_count())
Returns the expected value on a P9 system.
> diff --git a/buildtools/get-numa-count.py b/buildtools/get-numa-count.py
> new file mode 100644
> index 0000000000..77ef2b9f24
> --- /dev/null
> +++ b/buildtools/get-numa-count.py
> @@ -0,0 +1,22 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2021 PANTHEON.tech s.r.o.
> +
> +import ctypes
> +import glob
> +import os
> +import subprocess
> +
> +if os.name == 'posix':
> + if os.path.isdir('/sys/devices/system/node'):
> + print(len(glob.glob('/sys/devices/system/node/node*')))
> + else:
> + subprocess.run(['sysctl', '-n', 'vm.ndomains'])
> +
> +elif os.name == 'nt':
> + libkernel32 = ctypes.windll.kernel32
> +
> + count = ctypes.c_ulong()
> +
> + libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> + print(count.value + 1)
Does not return the expected value on my P9 system (NUMA nodes are not
contiguous). Got 6, expect to see at least 8 otherwise I can't use
lcores on NUMA node 8.
$ python3 ./get-numa-count.py
6
$ lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 4
Core(s) per socket: 16
Socket(s): 2
NUMA node(s): 6
Model: 2.3 (pvr 004e 1203)
Model name: POWER9, altivec supported
CPU max MHz: 3800.0000
CPU min MHz: 2300.0000
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 10240K
NUMA node0 CPU(s): 0-63
NUMA node8 CPU(s): 64-127
NUMA node252 CPU(s):
NUMA node253 CPU(s):
NUMA node254 CPU(s):
NUMA node255 CPU(s):
See attached ls-node.txt.gz for full directory structure on a P9 system.
Dave
I didn't notice this e-mail, since it was sent just to dev@dpdk.org and not to me explicitly, which left it unnoticed by my filters, unfortunately.
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of David Christensen
> Sent: Wednesday, March 31, 2021 9:07 PM
> To: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2] build: optional NUMA and cpu counts
> detection
>
>
>
> On 3/31/21 2:06 AM, Juraj Linkeš wrote:
> > Add an option to automatically discover the host's numa and cpu counts
> > and use those values for a non cross-build.
> > Give users the option to override the per-arch default values or
> > values from cross files by specifying them on the command line with
> > -Dmax_lcores and -Dmax_numa_nodes.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > ---
> > MAINTAINERS | 2 ++
> > buildtools/get-cpu-count.py | 7 ++++++
> > buildtools/get-numa-count.py | 22 +++++++++++++++++
> > buildtools/meson.build | 2 ++
> > config/meson.build | 47 ++++++++++++++++++++++++++++++++++--
> > config/x86/meson.build | 2 ++
> > meson_options.txt | 8 +++---
> > 7 files changed, 84 insertions(+), 6 deletions(-)
> > create mode 100644 buildtools/get-cpu-count.py
> > create mode 100644 buildtools/get-numa-count.py
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS index 0ec5588540..7270f33cf5
> > 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -99,6 +99,8 @@ F: meson_options.txt
> > F: config/
> > F: buildtools/chkincs/
> > F: buildtools/call-sphinx-build.py
> > +F: buildtools/get-cpu-count.py
> > +F: buildtools/get-numa-count.py
> > F: buildtools/list-dir-globs.py
> > F: buildtools/pkg-config/
> > F: buildtools/symlink-drivers-solibs.sh
> > diff --git a/buildtools/get-cpu-count.py b/buildtools/get-cpu-count.py
> > new file mode 100644 index 0000000000..317b32088f
> > --- /dev/null
> > +++ b/buildtools/get-cpu-count.py
> > @@ -0,0 +1,7 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2021
> > +PANTHEON.tech s.r.o.
> > +
> > +import os
> > +
> > +print(os.cpu_count())
>
> Returns the expected value on a P9 system.
>
> > diff --git a/buildtools/get-numa-count.py
> > b/buildtools/get-numa-count.py new file mode 100644 index
> > 0000000000..77ef2b9f24
> > --- /dev/null
> > +++ b/buildtools/get-numa-count.py
> > @@ -0,0 +1,22 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright (c) 2021
> > +PANTHEON.tech s.r.o.
> > +
> > +import ctypes
> > +import glob
> > +import os
> > +import subprocess
> > +
> > +if os.name == 'posix':
> > + if os.path.isdir('/sys/devices/system/node'):
> > + print(len(glob.glob('/sys/devices/system/node/node*')))
> > + else:
> > + subprocess.run(['sysctl', '-n', 'vm.ndomains'])
> > +
> > +elif os.name == 'nt':
> > + libkernel32 = ctypes.windll.kernel32
> > +
> > + count = ctypes.c_ulong()
> > +
> > + libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
> > + print(count.value + 1)
>
> Does not return the expected value on my P9 system (NUMA nodes are not
> contiguous). Got 6, expect to see at least 8 otherwise I can't use lcores on
> NUMA node 8.
>
I guess the problem here is that we're not really setting the number of NUMA nodes that are usable, but the highest NUMA that is usable?
In that case, the proper config in should be 256 so that even node255 is usable, is that right?
> $ python3 ./get-numa-count.py
> 6
> $ lscpu
> Architecture: ppc64le
> Byte Order: Little Endian
> CPU(s): 128
> On-line CPU(s) list: 0-127
> Thread(s) per core: 4
> Core(s) per socket: 16
> Socket(s): 2
> NUMA node(s): 6
> Model: 2.3 (pvr 004e 1203)
> Model name: POWER9, altivec supported
> CPU max MHz: 3800.0000
> CPU min MHz: 2300.0000
> L1d cache: 32K
> L1i cache: 32K
> L2 cache: 512K
> L3 cache: 10240K
> NUMA node0 CPU(s): 0-63
> NUMA node8 CPU(s): 64-127
> NUMA node252 CPU(s):
> NUMA node253 CPU(s):
> NUMA node254 CPU(s):
> NUMA node255 CPU(s):
>
> See attached ls-node.txt.gz for full directory structure on a P9 system.
>
> Dave
@@ -99,6 +99,8 @@ F: meson_options.txt
F: config/
F: buildtools/chkincs/
F: buildtools/call-sphinx-build.py
+F: buildtools/get-cpu-count.py
+F: buildtools/get-numa-count.py
F: buildtools/list-dir-globs.py
F: buildtools/pkg-config/
F: buildtools/symlink-drivers-solibs.sh
new file mode 100644
@@ -0,0 +1,7 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 PANTHEON.tech s.r.o.
+
+import os
+
+print(os.cpu_count())
new file mode 100644
@@ -0,0 +1,22 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 PANTHEON.tech s.r.o.
+
+import ctypes
+import glob
+import os
+import subprocess
+
+if os.name == 'posix':
+ if os.path.isdir('/sys/devices/system/node'):
+ print(len(glob.glob('/sys/devices/system/node/node*')))
+ else:
+ subprocess.run(['sysctl', '-n', 'vm.ndomains'])
+
+elif os.name == 'nt':
+ libkernel32 = ctypes.windll.kernel32
+
+ count = ctypes.c_ulong()
+
+ libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
+ print(count.value + 1)
@@ -16,6 +16,8 @@ else
endif
map_to_win_cmd = py3 + files('map_to_win.py')
sphinx_wrapper = py3 + files('call-sphinx-build.py')
+get_cpu_count_cmd = py3 + files('get-cpu-count.py')
+get_numa_count_cmd = py3 + files('get-numa-count.py')
# select library and object file format
pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
@@ -243,8 +243,6 @@ foreach arg: warning_flags
endforeach
# set other values pulled from the build options
-dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
-dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
@@ -278,6 +276,51 @@ if meson.is_cross_build()
endif
endif
+max_lcores = get_option('max_lcores')
+if max_lcores > 0
+ # Overwrite the default value from arch_subdir with user input
+ dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+elif max_lcores == -1
+ # Overwrite the default value with discovered values
+ if meson.is_cross_build()
+ error('Discovery of max_lcores is not supported for cross-compilation.')
+ endif
+ # Discovery makes sense only for non-cross builds
+ max_lcores = run_command(get_cpu_count_cmd).stdout().to_int()
+ min_lcores = 2
+ # DPDK must be build for at least 2 cores
+ if max_lcores < min_lcores
+ message('Found less than @0@ cores, building for @0@ cores'.format(min_lcores))
+ max_lcores = min_lcores
+ else
+ message('Found @0@ cores'.format(max_lcores))
+ endif
+ dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+endif
+
+max_numa_nodes = get_option('max_numa_nodes')
+if max_numa_nodes > 0
+ # Overwrite the default value from arch_subdir with user input
+ dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+elif max_numa_nodes == -1
+ # Overwrite the default value with discovered values
+ if meson.is_cross_build()
+ error('Discovery of max_numa_nodes not supported for cross-compilation.')
+ endif
+ # Discovery makes sense only for non-cross builds
+ max_numa_nodes = run_command(get_numa_count_cmd).stdout().to_int()
+ message('Found @0@ numa nodes'.format(max_numa_nodes))
+ dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+endif
+
+# check that cpu and numa count is set and error out if it's not set
+if not dpdk_conf.has('RTE_MAX_LCORE')
+ error('Number of cores not specified.')
+endif
+if not dpdk_conf.has('RTE_MAX_NUMA_NODES')
+ error('Number of numa nodes not specified.')
+endif
+
# set the install path for the drivers
dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
@@ -57,3 +57,5 @@ else
endif
dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
+dpdk_conf.set('RTE_MAX_LCORE', 128)
+dpdk_conf.set('RTE_MAX_NUMA_NODES', 32)
@@ -24,10 +24,10 @@ option('machine', type: 'string', value: 'native',
description: 'set the target machine type')
option('max_ethports', type: 'integer', value: 32,
description: 'maximum number of Ethernet devices')
-option('max_lcores', type: 'integer', value: 128,
- description: 'maximum number of cores/threads supported by EAL')
-option('max_numa_nodes', type: 'integer', value: 32,
- description: 'maximum number of NUMA nodes supported by EAL')
+option('max_lcores', type: 'integer', value: 0,
+ description: 'A positive integer sets the maximum number of cores/threads supported by EAL. The default is per-arch (or cross-compilation) default. Set to -1 to detect the number of cores on the build machine.')
+option('max_numa_nodes', type: 'integer', value: 0,
+ description: 'A positive integer sets maximum number of NUMA nodes supported by EAL. The default is per-arch (or cross-compilation) default. Set to -1 to detect the number of numa nodes on the build machine.')
option('enable_trace_fp', type: 'boolean', value: false,
description: 'enable fast path trace points.')
option('tests', type: 'boolean', value: true,