diff mbox series

[v5] build: optional NUMA and cpu counts detection

Message ID 1626872655-27496-1-git-send-email-juraj.linkes@pantheon.tech (mailing list archive)
State Deferred
Delegated to: Thomas Monjalon
Headers show
Series [v5] build: optional NUMA and cpu counts detection | expand

Checks

Context Check Description
ci/intel-Testing success Testing PASS
ci/Intel-compilation success Compilation OK
ci/iol-testing success Testing PASS
ci/iol-abi-testing success Testing PASS
ci/iol-intel-Performance success Performance Testing PASS
ci/iol-intel-Functional success Functional Testing PASS
ci/github-robot success github build: passed
ci/checkpatch success coding style OK

Commit Message

Juraj Linkeš July 21, 2021, 1:04 p.m. UTC
Add an option to automatically discover the host's numa and cpu counts
and use those values for a non cross-build.
Give users the option to override the per-arch default values or values
from cross files by specifying them on the command line with -Dmax_lcores
and -Dmax_numa_nodes.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: David Christensen <drc@linux.vnet.ibm.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
 MAINTAINERS                  |  2 ++
 buildtools/get-cpu-count.py  |  7 ++++++
 buildtools/get-numa-count.py | 24 ++++++++++++++++++
 buildtools/meson.build       |  2 ++
 config/meson.build           | 47 ++++++++++++++++++++++++++++++++++--
 config/x86/meson.build       |  2 ++
 meson_options.txt            |  8 +++---
 7 files changed, 86 insertions(+), 6 deletions(-)
 create mode 100644 buildtools/get-cpu-count.py
 create mode 100644 buildtools/get-numa-count.py

Comments

Juraj Linkeš Aug. 2, 2021, 12:44 p.m. UTC | #1
> +if os.name == 'posix':
> +    if os.path.isdir('/sys/devices/system/node'):
> +        numa_nodes = glob.glob('/sys/devices/system/node/node*')
> +        numa_nodes.sort()
> +        print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
> +    else:
> +        subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
> +

Bruce, David, Thomas,

Is DPDK actually supported on Power9 FreeBSD? Is anyone using this combination? How can we address the open question of what exactly does sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or should we just leave it as is? Or maybe add 1 to the output (as we do in other cases)?
David Christensen Aug. 2, 2021, 11:29 p.m. UTC | #2
On 8/2/21 5:44 AM, Juraj Linkeš wrote:
>> +if os.name == 'posix':
>> +    if os.path.isdir('/sys/devices/system/node'):
>> +        numa_nodes = glob.glob('/sys/devices/system/node/node*')
>> +        numa_nodes.sort()
>> +        print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
>> +    else:
>> +        subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
>> +
> 
> Bruce, David, Thomas,
> 
> Is DPDK actually supported on Power9 FreeBSD? Is anyone using this combination? How can we address the open question of what exactly does sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or should we just leave it as is? Or maybe add 1 to the output (as we do in other cases)?

Not supported within IBM, but you can buy OpenPOWER boxes from 3rd 
parties such as Raptor Computing Systems so there may be customers using 
DPDK on POWER with FreeBSD that I don't track.  Adding Piotr Kubaj who 
has commented on POWER/FreeBSD issues in this past.

Dave
Juraj Linkeš Aug. 3, 2021, 10:21 a.m. UTC | #3
> -----Original Message-----
> From: David Christensen <drc@linux.vnet.ibm.com>
> Sent: Tuesday, August 3, 2021 1:29 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; thomas@monjalon.net;
> david.marchand@redhat.com; bruce.richardson@intel.com;
> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> stephen@networkplumber.org; Piotr Kubaj <pkubaj@FreeBSD.org>
> Cc: dev@dpdk.org
> Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> 
> 
> 
> On 8/2/21 5:44 AM, Juraj Linkeš wrote:
> >> +if os.name == 'posix':
> >> +    if os.path.isdir('/sys/devices/system/node'):
> >> +        numa_nodes = glob.glob('/sys/devices/system/node/node*')
> >> +        numa_nodes.sort()
> >> +        print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
> >> +    else:
> >> +        subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
> >> +
> >
> > Bruce, David, Thomas,
> >
> > Is DPDK actually supported on Power9 FreeBSD? Is anyone using this
> combination? How can we address the open question of what exactly does
> sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or should we just
> leave it as is? Or maybe add 1 to the output (as we do in other cases)?
> 
> Not supported within IBM, but you can buy OpenPOWER boxes from 3rd parties
> such as Raptor Computing Systems so there may be customers using DPDK on
> POWER with FreeBSD that I don't track.  Adding Piotr Kubaj who has commented
> on POWER/FreeBSD issues in this past.
> 
> Dave

Thanks, David.

Piotr, to provide more context, we're trying to figure out what the highest NUMA node on a system is.
On P9 systems, here's how NUMA nodes look like in Linux:
NUMA node0 CPU(s):   0-63
NUMA node8 CPU(s):   64-127
NUMA node252 CPU(s):
NUMA node253 CPU(s):
NUMA node254 CPU(s):
NUMA node255 CPU(s):
The highest NUMA with CPUs is node8.

We're trying to get the highest NUMA with CPUs on P9 FreeBSD systems, but we don't know whether FreeBSD NUMA layout looks the same (does FreeBSD report non-contiguous NUMA nodes as Linxu above, or does it renumerate) a what does "systemctl -n vm.ndomains" return. Could you check these for us?
diff mbox series

Patch

diff --git a/MAINTAINERS b/MAINTAINERS
index af2a91d7c4..f0e82598aa 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -102,6 +102,8 @@  F: meson_options.txt
 F: config/
 F: buildtools/chkincs/
 F: buildtools/call-sphinx-build.py
+F: buildtools/get-cpu-count.py
+F: buildtools/get-numa-count.py
 F: buildtools/list-dir-globs.py
 F: buildtools/pkg-config/
 F: buildtools/symlink-drivers-solibs.sh
diff --git a/buildtools/get-cpu-count.py b/buildtools/get-cpu-count.py
new file mode 100644
index 0000000000..317b32088f
--- /dev/null
+++ b/buildtools/get-cpu-count.py
@@ -0,0 +1,7 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 PANTHEON.tech s.r.o.
+
+import os
+
+print(os.cpu_count())
diff --git a/buildtools/get-numa-count.py b/buildtools/get-numa-count.py
new file mode 100644
index 0000000000..1b7787787f
--- /dev/null
+++ b/buildtools/get-numa-count.py
@@ -0,0 +1,24 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 PANTHEON.tech s.r.o.
+
+import ctypes
+import glob
+import os
+import subprocess
+
+if os.name == 'posix':
+    if os.path.isdir('/sys/devices/system/node'):
+        numa_nodes = glob.glob('/sys/devices/system/node/node*')
+        numa_nodes.sort()
+        print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
+    else:
+        subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
+
+elif os.name == 'nt':
+    libkernel32 = ctypes.windll.kernel32
+
+    numa_count = ctypes.c_ulong()
+
+    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(numa_count))
+    print(numa_count.value + 1)
diff --git a/buildtools/meson.build b/buildtools/meson.build
index bd460e3e00..f776316da1 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -16,6 +16,8 @@  echo = py3 + ['-c', 'import sys; print(*sys.argv[1:])']
 list_dir_globs = py3 + files('list-dir-globs.py')
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+get_cpu_count_cmd = py3 + files('get-cpu-count.py')
+get_numa_count_cmd = py3 + files('get-numa-count.py')
 
 # select library and object file format
 pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
diff --git a/config/meson.build b/config/meson.build
index e80421003b..364788c32d 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -286,8 +286,6 @@  foreach arg: warning_flags
 endforeach
 
 # set other values pulled from the build options
-dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
-dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
 dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
 dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
 dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
@@ -321,6 +319,51 @@  if meson.is_cross_build()
     endif
 endif
 
+max_lcores = get_option('max_lcores')
+if max_lcores == 'detect'
+	# Discovery makes sense only for non-cross builds
+	if meson.is_cross_build()
+		error('Discovery of max_lcores is not supported for cross-compilation.')
+	endif
+	# Overwrite the default value with discovered values
+	max_lcores = run_command(get_cpu_count_cmd).stdout().to_int()
+	min_lcores = 2
+	# DPDK must be build for at least 2 cores
+	if max_lcores < min_lcores
+		message('Found less than @0@ cores, building for @0@ cores'.format(min_lcores))
+		max_lcores = min_lcores
+	else
+		message('Found @0@ cores'.format(max_lcores))
+	endif
+	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+elif max_lcores != 'default'
+	# Overwrite the default value from arch_subdir with user input
+	dpdk_conf.set('RTE_MAX_LCORE', max_lcores.to_int())
+endif
+
+max_numa_nodes = get_option('max_numa_nodes')
+if max_numa_nodes == 'detect'
+	# Discovery makes sense only for non-cross builds
+	if meson.is_cross_build()
+		error('Discovery of max_numa_nodes not supported for cross-compilation.')
+	endif
+	# Overwrite the default value with discovered values
+	max_numa_nodes = run_command(get_numa_count_cmd).stdout().to_int()
+	message('Found @0@ numa nodes'.format(max_numa_nodes))
+	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+elif max_numa_nodes != 'default'
+	# Overwrite the default value from arch_subdir with user input
+	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes.to_int())
+endif
+
+# check that cpu and numa count is set and error out if it's not set
+if not dpdk_conf.has('RTE_MAX_LCORE')
+	error('Number of cores not specified.')
+endif
+if not dpdk_conf.has('RTE_MAX_NUMA_NODES')
+	error('Number of numa nodes not specified.')
+endif
+
 # set the install path for the drivers
 dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
 
diff --git a/config/x86/meson.build b/config/x86/meson.build
index b9348c44de..872dda9585 100644
--- a/config/x86/meson.build
+++ b/config/x86/meson.build
@@ -57,3 +57,5 @@  else
 endif
 
 dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
+dpdk_conf.set('RTE_MAX_LCORE', 128)
+dpdk_conf.set('RTE_MAX_NUMA_NODES', 32)
diff --git a/meson_options.txt b/meson_options.txt
index 0e92734c49..b56e3b83c7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -34,10 +34,10 @@  option('machine', type: 'string', value: 'auto', description:
        'Alias of cpu_instruction_set.')
 option('max_ethports', type: 'integer', value: 32, description:
        'maximum number of Ethernet devices')
-option('max_lcores', type: 'integer', value: 128, description:
-       'maximum number of cores/threads supported by EAL')
-option('max_numa_nodes', type: 'integer', value: 32, description:
-       'maximum number of NUMA nodes supported by EAL')
+option('max_lcores', type: 'string', value: 'default', description:
+       'Set maximum number of cores/threads supported by EAL; "default" is different per-arch, "detect" detects the number of cores on the build machine.')
+option('max_numa_nodes', type: 'string', value: 'default', description:
+       'Set the highest NUMA node supported by EAL; "default" is different per-arch, "detect" detects the highest numa node on the build machine.')
 option('platform', type: 'string', value: 'native', description:
        'Platform to build, either "native", "generic" or a SoC. Please refer to the Linux build guide for more information.')
 option('enable_trace_fp', type: 'boolean', value: false, description: