diff mbox series

[v4] build: optional NUMA and cpu counts detection

Message ID 1624964105-6525-1-git-send-email-juraj.linkes@pantheon.tech (mailing list archive)
State Superseded, archived
Delegated to: Thomas Monjalon
Headers show
Series [v4] build: optional NUMA and cpu counts detection | expand

Checks

Context Check Description
ci/intel-Testing success Testing PASS
ci/Intel-compilation success Compilation OK
ci/iol-intel-Performance fail Performance Testing issues
ci/iol-mellanox-Functional fail Functional Testing issues
ci/iol-testing success Testing PASS
ci/iol-intel-Functional success Functional Testing PASS
ci/github-robot success github build: passed
ci/checkpatch success coding style OK

Commit Message

Juraj Linkeš June 29, 2021, 10:55 a.m. UTC
Add an option to automatically discover the host's numa and cpu counts
and use those values for a non cross-build.
Give users the option to override the per-arch default values or values
from cross files by specifying them on the command line with -Dmax_lcores
and -Dmax_numa_nodes.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
---
 MAINTAINERS                  |  2 ++
 buildtools/get-cpu-count.py  |  7 ++++++
 buildtools/get-numa-count.py | 24 ++++++++++++++++++
 buildtools/meson.build       |  2 ++
 config/meson.build           | 47 ++++++++++++++++++++++++++++++++++--
 config/x86/meson.build       |  2 ++
 meson_options.txt            |  8 +++---
 7 files changed, 86 insertions(+), 6 deletions(-)
 create mode 100644 buildtools/get-cpu-count.py
 create mode 100644 buildtools/get-numa-count.py

Comments

Bruce Richardson June 29, 2021, 11:28 a.m. UTC | #1
On Tue, Jun 29, 2021 at 12:55:05PM +0200, Juraj Linkeš wrote:
> Add an option to automatically discover the host's numa and cpu counts
> and use those values for a non cross-build.
> Give users the option to override the per-arch default values or values
> from cross files by specifying them on the command line with -Dmax_lcores
> and -Dmax_numa_nodes.
> 
> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> ---
Two very minor suggestions inline below.

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

>  
<snip>
> +max_lcores = get_option('max_lcores')
> +if max_lcores == 'auto'

Rather than "auto", would "detect" be a clearer name for this option value?

<snip>
> +option('max_lcores', type: 'string', value: 'default', description:
> +       'Set maximum number of cores/threads supported by EAL. The default is different per-arch. Set to auto to detect the number of cores on the build machine.')
> +option('max_numa_nodes', type: 'string', value: 'default', description:
> +       'Set highest NUMA node supported by EAL. The default is different per-arch. Set to auto to detect the highest numa node on the build machine.')

I'd put the explicit values of "default" and "auto"(or "detect") in
quotes "" to make clear they are literal values.
Stephen Hemminger June 29, 2021, 6 p.m. UTC | #2
On Tue, 29 Jun 2021 12:55:05 +0200
Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:

> diff --git a/buildtools/get-numa-count.py b/buildtools/get-numa-count.py
> new file mode 100644
> index 0000000000..3b67564fd4
> --- /dev/null
> +++ b/buildtools/get-numa-count.py
> @@ -0,0 +1,24 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright (c) 2021 PANTHEON.tech s.r.o.
> +
> +import ctypes
> +import glob
> +import os
> +import subprocess
> +
> +if os.name == 'posix':
> +    if os.path.isdir('/sys/devices/system/node'):
> +        numa_nodes = glob.glob('/sys/devices/system/node/node*')
> +        numa_nodes.sort()
> +        print(os.path.basename(numa_nodes[-1])[4:])
> +    else:
> +        subprocess.run(['sysctl', '-n', 'vm.ndomains'])
> +

python lint has warning here
buildtools/get-numa-count.py:16:8: W1510: Using subprocess.run without explicitly set `check` is not recommended. (subprocess-run-check)
Juraj Linkeš July 6, 2021, 8:56 a.m. UTC | #3
> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Tuesday, June 29, 2021 1:29 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
> Subject: Re: [PATCH v4] build: optional NUMA and cpu counts detection
> 
> On Tue, Jun 29, 2021 at 12:55:05PM +0200, Juraj Linkeš wrote:
> > Add an option to automatically discover the host's numa and cpu counts
> > and use those values for a non cross-build.
> > Give users the option to override the per-arch default values or
> > values from cross files by specifying them on the command line with
> > -Dmax_lcores and -Dmax_numa_nodes.
> >
> > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > ---
> Two very minor suggestions inline below.
> 
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> >
> <snip>
> > +max_lcores = get_option('max_lcores') if max_lcores == 'auto'
> 
> Rather than "auto", would "detect" be a clearer name for this option value?
> 
> <snip>
> > +option('max_lcores', type: 'string', value: 'default', description:
> > +       'Set maximum number of cores/threads supported by EAL. The
> > +default is different per-arch. Set to auto to detect the number of cores on the
> build machine.') option('max_numa_nodes', type: 'string', value: 'default',
> description:
> > +       'Set highest NUMA node supported by EAL. The default is
> > +different per-arch. Set to auto to detect the highest numa node on
> > +the build machine.')
> 
> I'd put the explicit values of "default" and "auto"(or "detect") in quotes "" to
> make clear they are literal values.
> 

Thanks, Bruce, I'll change it. I have one extra question now that I'm looking at the patch:
What does subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False) return exactly? Is the the number of NUMA nodes (looks like it) or the highest NUMA node on the system (the highest number of all NUMA nodes)? I'm asking because of how NUMA works on P9:
NUMA node0 CPU(s):   0-63
NUMA node8 CPU(s):   64-127
NUMA node252 CPU(s):
NUMA node253 CPU(s):
NUMA node254 CPU(s):
NUMA node255 CPU(s):

Here we need not just two NUMA nodes, but at least 9 (0-8). Linux and Windows should return the highest NUMA, not sure about FreeBSD. Or maybe we should return the highest NUMA on which there are actual CPUs?
Bruce Richardson July 6, 2021, 9:08 a.m. UTC | #4
On Tue, Jul 06, 2021 at 08:56:37AM +0000, Juraj Linkeš wrote:
> 
> 
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Tuesday, June 29, 2021 1:29 PM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Cc: thomas@monjalon.net; david.marchand@redhat.com;
> > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
> > Subject: Re: [PATCH v4] build: optional NUMA and cpu counts detection
> > 
> > On Tue, Jun 29, 2021 at 12:55:05PM +0200, Juraj Linkeš wrote:
> > > Add an option to automatically discover the host's numa and cpu counts
> > > and use those values for a non cross-build.
> > > Give users the option to override the per-arch default values or
> > > values from cross files by specifying them on the command line with
> > > -Dmax_lcores and -Dmax_numa_nodes.
> > >
> > > Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > > ---
> > Two very minor suggestions inline below.
> > 
> > Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> > 
> > >
> > <snip>
> > > +max_lcores = get_option('max_lcores') if max_lcores == 'auto'
> > 
> > Rather than "auto", would "detect" be a clearer name for this option value?
> > 
> > <snip>
> > > +option('max_lcores', type: 'string', value: 'default', description:
> > > +       'Set maximum number of cores/threads supported by EAL. The
> > > +default is different per-arch. Set to auto to detect the number of cores on the
> > build machine.') option('max_numa_nodes', type: 'string', value: 'default',
> > description:
> > > +       'Set highest NUMA node supported by EAL. The default is
> > > +different per-arch. Set to auto to detect the highest numa node on
> > > +the build machine.')
> > 
> > I'd put the explicit values of "default" and "auto"(or "detect") in quotes "" to
> > make clear they are literal values.
> > 
> 
> Thanks, Bruce, I'll change it. I have one extra question now that I'm looking at the patch:
> What does subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False) return exactly? Is the the number of NUMA nodes (looks like it) or the highest NUMA node on the system (the highest number of all NUMA nodes)? I'm asking because of how NUMA works on P9:
> NUMA node0 CPU(s):   0-63
> NUMA node8 CPU(s):   64-127
> NUMA node252 CPU(s):
> NUMA node253 CPU(s):
> NUMA node254 CPU(s):
> NUMA node255 CPU(s):
> 
> Here we need not just two NUMA nodes, but at least 9 (0-8). Linux and Windows should return the highest NUMA, not sure about FreeBSD. Or maybe we should return the highest NUMA on which there are actual CPUs?

I'm not sure, and I think to be really sure we'd need it tested on a P9
system. The help text for the sysctl node says "Number of physical memory
domains available", which would imply 2 in the case above. [However, we also
would need to find out how BSD numbers the domains, too, as it's possible
an OS could just call them 0 and 1, rather than 0 and 8 if it wanted to.]

In short, we'd need to test to be sure. Is FreeBSD on P9 a supported
config, and if so can the P9 maintainer perhaps help out with testing?

/Bruce
David Christensen July 6, 2021, 6:10 p.m. UTC | #5
On 7/6/21 2:08 AM, Bruce Richardson wrote:
> On Tue, Jul 06, 2021 at 08:56:37AM +0000, Juraj Linkeš wrote:
>>
>>
>>> -----Original Message-----
>>> From: Bruce Richardson <bruce.richardson@intel.com>
>>> Sent: Tuesday, June 29, 2021 1:29 PM
>>> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
>>> Cc: thomas@monjalon.net; david.marchand@redhat.com;
>>> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
>>> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
>>> Subject: Re: [PATCH v4] build: optional NUMA and cpu counts detection
>>>
>>> On Tue, Jun 29, 2021 at 12:55:05PM +0200, Juraj Linkeš wrote:
>>>> Add an option to automatically discover the host's numa and cpu counts
>>>> and use those values for a non cross-build.
>>>> Give users the option to override the per-arch default values or
>>>> values from cross files by specifying them on the command line with
>>>> -Dmax_lcores and -Dmax_numa_nodes.
>>>>
>>>> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>>>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>>>> ---
>>> Two very minor suggestions inline below.
>>>
>>> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
>>>
>>>>
>>> <snip>
>>>> +max_lcores = get_option('max_lcores') if max_lcores == 'auto'
>>>
>>> Rather than "auto", would "detect" be a clearer name for this option value?
>>>
>>> <snip>
>>>> +option('max_lcores', type: 'string', value: 'default', description:
>>>> +       'Set maximum number of cores/threads supported by EAL. The
>>>> +default is different per-arch. Set to auto to detect the number of cores on the
>>> build machine.') option('max_numa_nodes', type: 'string', value: 'default',
>>> description:
>>>> +       'Set highest NUMA node supported by EAL. The default is
>>>> +different per-arch. Set to auto to detect the highest numa node on
>>>> +the build machine.')
>>>
>>> I'd put the explicit values of "default" and "auto"(or "detect") in quotes "" to
>>> make clear they are literal values.
>>>
>>
>> Thanks, Bruce, I'll change it. I have one extra question now that I'm looking at the patch:
>> What does subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False) return exactly? Is the the number of NUMA nodes (looks like it) or the highest NUMA node on the system (the highest number of all NUMA nodes)? I'm asking because of how NUMA works on P9:
>> NUMA node0 CPU(s):   0-63
>> NUMA node8 CPU(s):   64-127
>> NUMA node252 CPU(s):
>> NUMA node253 CPU(s):
>> NUMA node254 CPU(s):
>> NUMA node255 CPU(s):
>>
>> Here we need not just two NUMA nodes, but at least 9 (0-8). Linux and Windows should return the highest NUMA, not sure about FreeBSD. Or maybe we should return the highest NUMA on which there are actual CPUs?
> 
> I'm not sure, and I think to be really sure we'd need it tested on a P9
> system. The help text for the sysctl node says "Number of physical memory
> domains available", which would imply 2 in the case above. [However, we also
> would need to find out how BSD numbers the domains, too, as it's possible
> an OS could just call them 0 and 1, rather than 0 and 8 if it wanted to.]
> 
> In short, we'd need to test to be sure. Is FreeBSD on P9 a supported
> config, and if so can the P9 maintainer perhaps help out with testing?

Results of the v4 patch on an IBM AC922 P9 system with Linux:

$ python3 get-cpu-count.py
128
$ python3 get-numa-count.py
8
$ lscpu
Architecture:        ppc64le
Byte Order:          Little Endian
CPU(s):              128
On-line CPU(s) list: 0-127
Thread(s) per core:  4
Core(s) per socket:  16
Socket(s):           2
NUMA node(s):        6
Model:               2.3 (pvr 004e 1203)
Model name:          POWER9, altivec supported
CPU max MHz:         3800.0000
CPU min MHz:         2300.0000
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            10240K
NUMA node0 CPU(s):   0-63
NUMA node8 CPU(s):   64-127
NUMA node252 CPU(s):
NUMA node253 CPU(s):
NUMA node254 CPU(s):
NUMA node255 CPU(s):

OVS has a problem with requiring contiguous NUMA nodes that we've 
submitted patches to fix, so we need to ensure that's not a problem in DPDK.

I've been doing things like "--socket-mem=2048,0,0,0,0,0,0,0,2048" so 
far to manage the memory-to-NUMA mapping which has been working fine, so 
it depends on how vm.ndomains will be.

Dave
Juraj Linkeš July 16, 2021, 1:53 p.m. UTC | #6
> -----Original Message-----
> From: David Christensen <drc@linux.vnet.ibm.com>
> Sent: Tuesday, July 6, 2021 8:11 PM
> To: Bruce Richardson <bruce.richardson@intel.com>; Juraj Linkeš
> <juraj.linkes@pantheon.tech>
> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v4] build: optional NUMA and cpu counts
> detection
> 
> 
> 
> On 7/6/21 2:08 AM, Bruce Richardson wrote:
> > On Tue, Jul 06, 2021 at 08:56:37AM +0000, Juraj Linkeš wrote:
> >>
> >>
> >>> -----Original Message-----
> >>> From: Bruce Richardson <bruce.richardson@intel.com>
> >>> Sent: Tuesday, June 29, 2021 1:29 PM
> >>> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> >>> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> >>> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> >>> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
> >>> Subject: Re: [PATCH v4] build: optional NUMA and cpu counts
> >>> detection
> >>>
> >>> On Tue, Jun 29, 2021 at 12:55:05PM +0200, Juraj Linkeš wrote:
> >>>> Add an option to automatically discover the host's numa and cpu
> >>>> counts and use those values for a non cross-build.
> >>>> Give users the option to override the per-arch default values or
> >>>> values from cross files by specifying them on the command line with
> >>>> -Dmax_lcores and -Dmax_numa_nodes.
> >>>>
> >>>> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> >>>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> >>>> ---
> >>> Two very minor suggestions inline below.
> >>>
> >>> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> >>>
> >>>>
> >>> <snip>
> >>>> +max_lcores = get_option('max_lcores') if max_lcores == 'auto'
> >>>
> >>> Rather than "auto", would "detect" be a clearer name for this option value?
> >>>
> >>> <snip>
> >>>> +option('max_lcores', type: 'string', value: 'default', description:
> >>>> +       'Set maximum number of cores/threads supported by EAL. The
> >>>> +default is different per-arch. Set to auto to detect the number of
> >>>> +cores on the
> >>> build machine.') option('max_numa_nodes', type: 'string', value:
> >>> 'default',
> >>> description:
> >>>> +       'Set highest NUMA node supported by EAL. The default is
> >>>> +different per-arch. Set to auto to detect the highest numa node on
> >>>> +the build machine.')
> >>>
> >>> I'd put the explicit values of "default" and "auto"(or "detect") in
> >>> quotes "" to make clear they are literal values.
> >>>
> >>
> >> Thanks, Bruce, I'll change it. I have one extra question now that I'm looking
> at the patch:
> >> What does subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False) return
> exactly? Is the the number of NUMA nodes (looks like it) or the highest NUMA
> node on the system (the highest number of all NUMA nodes)? I'm asking
> because of how NUMA works on P9:
> >> NUMA node0 CPU(s):   0-63
> >> NUMA node8 CPU(s):   64-127
> >> NUMA node252 CPU(s):
> >> NUMA node253 CPU(s):
> >> NUMA node254 CPU(s):
> >> NUMA node255 CPU(s):
> >>
> >> Here we need not just two NUMA nodes, but at least 9 (0-8). Linux and
> Windows should return the highest NUMA, not sure about FreeBSD. Or maybe
> we should return the highest NUMA on which there are actual CPUs?
> >
> > I'm not sure, and I think to be really sure we'd need it tested on a
> > P9 system. The help text for the sysctl node says "Number of physical
> > memory domains available", which would imply 2 in the case above.
> > [However, we also would need to find out how BSD numbers the domains,
> > too, as it's possible an OS could just call them 0 and 1, rather than
> > 0 and 8 if it wanted to.]
> >
> > In short, we'd need to test to be sure. Is FreeBSD on P9 a supported
> > config, and if so can the P9 maintainer perhaps help out with testing?
> 
> Results of the v4 patch on an IBM AC922 P9 system with Linux:
> 

Can you get results from FreeBSD as well?

> $ python3 get-numa-count.py
> 8
> NUMA node0 CPU(s):   0-63
> NUMA node8 CPU(s):   64-127
<snip>
Is this the right number for your case, i.e. are you able to use both numa nodes when RTE_MAX_NUMA_NODES=8?

Or maybe this is a question for Bruce or Thomas - what do we need to set in RTE_MAX_NUMA_NODES to be able to use all numa nodes on the system? The highest numa node number or that + 1? In linux, with 4 numa nodes, there will be node0-node3 under /sys/devices/system/node - do we need to set RTE_MAX_NUMA_NODES to 3 or 4?

> OVS has a problem with requiring contiguous NUMA nodes that we've submitted
> patches to fix, so we need to ensure that's not a problem in DPDK.
> 
> I've been doing things like "--socket-mem=2048,0,0,0,0,0,0,0,2048" so far to
> manage the memory-to-NUMA mapping which has been working fine, so it
> depends on how vm.ndomains will be.
> 
> Dave
Bruce Richardson July 16, 2021, 2:24 p.m. UTC | #7
On Fri, Jul 16, 2021 at 01:53:18PM +0000, Juraj Linkeš wrote:
> 
> 
> > -----Original Message-----
> > From: David Christensen <drc@linux.vnet.ibm.com>
> > Sent: Tuesday, July 6, 2021 8:11 PM
> > To: Bruce Richardson <bruce.richardson@intel.com>; Juraj Linkeš
> > <juraj.linkes@pantheon.tech>
> > Cc: thomas@monjalon.net; david.marchand@redhat.com;
> > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH v4] build: optional NUMA and cpu counts
> > detection
> > 
> > 
> > 
> > On 7/6/21 2:08 AM, Bruce Richardson wrote:
> > > On Tue, Jul 06, 2021 at 08:56:37AM +0000, Juraj Linkeš wrote:
> > >>
> > >>
> > >>> -----Original Message-----
> > >>> From: Bruce Richardson <bruce.richardson@intel.com>
> > >>> Sent: Tuesday, June 29, 2021 1:29 PM
> > >>> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > >>> Cc: thomas@monjalon.net; david.marchand@redhat.com;
> > >>> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > >>> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
> > >>> Subject: Re: [PATCH v4] build: optional NUMA and cpu counts
> > >>> detection
> > >>>
> > >>> On Tue, Jun 29, 2021 at 12:55:05PM +0200, Juraj Linkeš wrote:
> > >>>> Add an option to automatically discover the host's numa and cpu
> > >>>> counts and use those values for a non cross-build.
> > >>>> Give users the option to override the per-arch default values or
> > >>>> values from cross files by specifying them on the command line with
> > >>>> -Dmax_lcores and -Dmax_numa_nodes.
> > >>>>
> > >>>> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > >>>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > >>>> ---
> > >>> Two very minor suggestions inline below.
> > >>>
> > >>> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> > >>>
> > >>>>
> > >>> <snip>
> > >>>> +max_lcores = get_option('max_lcores') if max_lcores == 'auto'
> > >>>
> > >>> Rather than "auto", would "detect" be a clearer name for this option value?
> > >>>
> > >>> <snip>
> > >>>> +option('max_lcores', type: 'string', value: 'default', description:
> > >>>> +       'Set maximum number of cores/threads supported by EAL. The
> > >>>> +default is different per-arch. Set to auto to detect the number of
> > >>>> +cores on the
> > >>> build machine.') option('max_numa_nodes', type: 'string', value:
> > >>> 'default',
> > >>> description:
> > >>>> +       'Set highest NUMA node supported by EAL. The default is
> > >>>> +different per-arch. Set to auto to detect the highest numa node on
> > >>>> +the build machine.')
> > >>>
> > >>> I'd put the explicit values of "default" and "auto"(or "detect") in
> > >>> quotes "" to make clear they are literal values.
> > >>>
> > >>
> > >> Thanks, Bruce, I'll change it. I have one extra question now that I'm looking
> > at the patch:
> > >> What does subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False) return
> > exactly? Is the the number of NUMA nodes (looks like it) or the highest NUMA
> > node on the system (the highest number of all NUMA nodes)? I'm asking
> > because of how NUMA works on P9:
> > >> NUMA node0 CPU(s):   0-63
> > >> NUMA node8 CPU(s):   64-127
> > >> NUMA node252 CPU(s):
> > >> NUMA node253 CPU(s):
> > >> NUMA node254 CPU(s):
> > >> NUMA node255 CPU(s):
> > >>
> > >> Here we need not just two NUMA nodes, but at least 9 (0-8). Linux and
> > Windows should return the highest NUMA, not sure about FreeBSD. Or maybe
> > we should return the highest NUMA on which there are actual CPUs?
> > >
> > > I'm not sure, and I think to be really sure we'd need it tested on a
> > > P9 system. The help text for the sysctl node says "Number of physical
> > > memory domains available", which would imply 2 in the case above.
> > > [However, we also would need to find out how BSD numbers the domains,
> > > too, as it's possible an OS could just call them 0 and 1, rather than
> > > 0 and 8 if it wanted to.]
> > >
> > > In short, we'd need to test to be sure. Is FreeBSD on P9 a supported
> > > config, and if so can the P9 maintainer perhaps help out with testing?
> > 
> > Results of the v4 patch on an IBM AC922 P9 system with Linux:
> > 
> 
> Can you get results from FreeBSD as well?
> 
> > $ python3 get-numa-count.py
> > 8
> > NUMA node0 CPU(s):   0-63
> > NUMA node8 CPU(s):   64-127
> <snip>
> Is this the right number for your case, i.e. are you able to use both numa nodes when RTE_MAX_NUMA_NODES=8?
> 
> Or maybe this is a question for Bruce or Thomas - what do we need to set in RTE_MAX_NUMA_NODES to be able to use all numa nodes on the system? The highest numa node number or that + 1? In linux, with 4 numa nodes, there will be node0-node3 under /sys/devices/system/node - do we need to set RTE_MAX_NUMA_NODES to 3 or 4?
>

4, I believe.
David Christensen July 20, 2021, 8:49 p.m. UTC | #8
On 7/16/21 6:53 AM, Juraj Linkeš wrote:
> 
> 
>> -----Original Message-----
>> From: David Christensen <drc@linux.vnet.ibm.com>
>> Sent: Tuesday, July 6, 2021 8:11 PM
>> To: Bruce Richardson <bruce.richardson@intel.com>; Juraj Linkeš
>> <juraj.linkes@pantheon.tech>
>> Cc: thomas@monjalon.net; david.marchand@redhat.com;
>> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
>> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH v4] build: optional NUMA and cpu counts
>> detection
>>
>>
>>
>> On 7/6/21 2:08 AM, Bruce Richardson wrote:
>>> On Tue, Jul 06, 2021 at 08:56:37AM +0000, Juraj Linkeš wrote:
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Bruce Richardson <bruce.richardson@intel.com>
>>>>> Sent: Tuesday, June 29, 2021 1:29 PM
>>>>> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
>>>>> Cc: thomas@monjalon.net; david.marchand@redhat.com;
>>>>> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
>>>>> ferruh.yigit@intel.com; jerinjacobk@gmail.com; dev@dpdk.org
>>>>> Subject: Re: [PATCH v4] build: optional NUMA and cpu counts
>>>>> detection
>>>>>
>>>>> On Tue, Jun 29, 2021 at 12:55:05PM +0200, Juraj Linkeš wrote:
>>>>>> Add an option to automatically discover the host's numa and cpu
>>>>>> counts and use those values for a non cross-build.
>>>>>> Give users the option to override the per-arch default values or
>>>>>> values from cross files by specifying them on the command line with
>>>>>> -Dmax_lcores and -Dmax_numa_nodes.
>>>>>>
>>>>>> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>>>>>> Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
>>>>>> ---
>>>>> Two very minor suggestions inline below.
>>>>>
>>>>> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
>>>>>
>>>>>>
>>>>> <snip>
>>>>>> +max_lcores = get_option('max_lcores') if max_lcores == 'auto'
>>>>>
>>>>> Rather than "auto", would "detect" be a clearer name for this option value?
>>>>>
>>>>> <snip>
>>>>>> +option('max_lcores', type: 'string', value: 'default', description:
>>>>>> +       'Set maximum number of cores/threads supported by EAL. The
>>>>>> +default is different per-arch. Set to auto to detect the number of
>>>>>> +cores on the
>>>>> build machine.') option('max_numa_nodes', type: 'string', value:
>>>>> 'default',
>>>>> description:
>>>>>> +       'Set highest NUMA node supported by EAL. The default is
>>>>>> +different per-arch. Set to auto to detect the highest numa node on
>>>>>> +the build machine.')
>>>>>
>>>>> I'd put the explicit values of "default" and "auto"(or "detect") in
>>>>> quotes "" to make clear they are literal values.
>>>>>
>>>>
>>>> Thanks, Bruce, I'll change it. I have one extra question now that I'm looking
>> at the patch:
>>>> What does subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False) return
>> exactly? Is the the number of NUMA nodes (looks like it) or the highest NUMA
>> node on the system (the highest number of all NUMA nodes)? I'm asking
>> because of how NUMA works on P9:
>>>> NUMA node0 CPU(s):   0-63
>>>> NUMA node8 CPU(s):   64-127
>>>> NUMA node252 CPU(s):
>>>> NUMA node253 CPU(s):
>>>> NUMA node254 CPU(s):
>>>> NUMA node255 CPU(s):
>>>>
>>>> Here we need not just two NUMA nodes, but at least 9 (0-8). Linux and
>> Windows should return the highest NUMA, not sure about FreeBSD. Or maybe
>> we should return the highest NUMA on which there are actual CPUs?
>>>
>>> I'm not sure, and I think to be really sure we'd need it tested on a
>>> P9 system. The help text for the sysctl node says "Number of physical
>>> memory domains available", which would imply 2 in the case above.
>>> [However, we also would need to find out how BSD numbers the domains,
>>> too, as it's possible an OS could just call them 0 and 1, rather than
>>> 0 and 8 if it wanted to.]
>>>
>>> In short, we'd need to test to be sure. Is FreeBSD on P9 a supported
>>> config, and if so can the P9 maintainer perhaps help out with testing?
>>
>> Results of the v4 patch on an IBM AC922 P9 system with Linux:
>>
> 
> Can you get results from FreeBSD as well?

I can't help with FreeBSD here, I only have Linux on systems within IBM.

> 
>> $ python3 get-numa-count.py
>> 8
>> NUMA node0 CPU(s):   0-63
>> NUMA node8 CPU(s):   64-127
> <snip>
> Is this the right number for your case, i.e. are you able to use both numa nodes when RTE_MAX_NUMA_NODES=8?

node8 above is the ninth NUMA node so I'd need to use 
RTE_MAX_NUMA_NODES=9 as a minimum so that any arrays using that value 
are adequately sized.

Dave
Juraj Linkeš July 21, 2021, 12:41 p.m. UTC | #9
> >>> [However, we also would need to find out how BSD numbers the
> >>> domains, too, as it's possible an OS could just call them 0 and 1,
> >>> rather than
> >>> 0 and 8 if it wanted to.]
> >>>
> >>> In short, we'd need to test to be sure. Is FreeBSD on P9 a supported
> >>> config, and if so can the P9 maintainer perhaps help out with testing?
> >>
> >> Results of the v4 patch on an IBM AC922 P9 system with Linux:
> >>
> >
> > Can you get results from FreeBSD as well?
> 
> I can't help with FreeBSD here, I only have Linux on systems within IBM.
> 

This is an open question still then. I guess we'll have to go with sysctl -n vm.ndomains.

> >
> >> $ python3 get-numa-count.py
> >> 8
> >> NUMA node0 CPU(s):   0-63
> >> NUMA node8 CPU(s):   64-127
> > <snip>
> > Is this the right number for your case, i.e. are you able to use both numa nodes
> when RTE_MAX_NUMA_NODES=8?
> 
> node8 above is the ninth NUMA node so I'd need to use
> RTE_MAX_NUMA_NODES=9 as a minimum so that any arrays using that value
> are adequately sized.
> 
> Dave

Ok, the proper value to return is highest numa node + 1 then.
diff mbox series

Patch

diff --git a/MAINTAINERS b/MAINTAINERS
index 5877a16971..0ca3d4e9f5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -102,6 +102,8 @@  F: meson_options.txt
 F: config/
 F: buildtools/chkincs/
 F: buildtools/call-sphinx-build.py
+F: buildtools/get-cpu-count.py
+F: buildtools/get-numa-count.py
 F: buildtools/list-dir-globs.py
 F: buildtools/pkg-config/
 F: buildtools/symlink-drivers-solibs.sh
diff --git a/buildtools/get-cpu-count.py b/buildtools/get-cpu-count.py
new file mode 100644
index 0000000000..317b32088f
--- /dev/null
+++ b/buildtools/get-cpu-count.py
@@ -0,0 +1,7 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 PANTHEON.tech s.r.o.
+
+import os
+
+print(os.cpu_count())
diff --git a/buildtools/get-numa-count.py b/buildtools/get-numa-count.py
new file mode 100644
index 0000000000..3b67564fd4
--- /dev/null
+++ b/buildtools/get-numa-count.py
@@ -0,0 +1,24 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 PANTHEON.tech s.r.o.
+
+import ctypes
+import glob
+import os
+import subprocess
+
+if os.name == 'posix':
+    if os.path.isdir('/sys/devices/system/node'):
+        numa_nodes = glob.glob('/sys/devices/system/node/node*')
+        numa_nodes.sort()
+        print(os.path.basename(numa_nodes[-1])[4:])
+    else:
+        subprocess.run(['sysctl', '-n', 'vm.ndomains'])
+
+elif os.name == 'nt':
+    libkernel32 = ctypes.windll.kernel32
+
+    count = ctypes.c_ulong()
+
+    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(count))
+    print(count.value + 1)
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 0f24b15297..887882e6a5 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -16,6 +16,8 @@  endif
 list_dir_globs = py3 + files('list-dir-globs.py')
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+get_cpu_count_cmd = py3 + files('get-cpu-count.py')
+get_numa_count_cmd = py3 + files('get-numa-count.py')
 
 # select library and object file format
 pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
diff --git a/config/meson.build b/config/meson.build
index 017bb2efbb..2d1db40127 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -247,8 +247,6 @@  foreach arg: warning_flags
 endforeach
 
 # set other values pulled from the build options
-dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
-dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
 dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
 dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
 dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
@@ -282,6 +280,51 @@  if meson.is_cross_build()
     endif
 endif
 
+max_lcores = get_option('max_lcores')
+if max_lcores == 'auto'
+	# Discovery makes sense only for non-cross builds
+	if meson.is_cross_build()
+		error('Discovery of max_lcores is not supported for cross-compilation.')
+	endif
+	# Overwrite the default value with discovered values
+	max_lcores = run_command(get_cpu_count_cmd).stdout().to_int()
+	min_lcores = 2
+	# DPDK must be build for at least 2 cores
+	if max_lcores < min_lcores
+		message('Found less than @0@ cores, building for @0@ cores'.format(min_lcores))
+		max_lcores = min_lcores
+	else
+		message('Found @0@ cores'.format(max_lcores))
+	endif
+	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+elif max_lcores != 'default'
+	# Overwrite the default value from arch_subdir with user input
+	dpdk_conf.set('RTE_MAX_LCORE', max_lcores.to_int())
+endif
+
+max_numa_nodes = get_option('max_numa_nodes')
+if max_numa_nodes == 'auto'
+	# Discovery makes sense only for non-cross builds
+	if meson.is_cross_build()
+		error('Discovery of max_numa_nodes not supported for cross-compilation.')
+	endif
+	# Overwrite the default value with discovered values
+	max_numa_nodes = run_command(get_numa_count_cmd).stdout().to_int()
+	message('Found @0@ numa nodes'.format(max_numa_nodes))
+	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+elif max_numa_nodes != 'default'
+	# Overwrite the default value from arch_subdir with user input
+	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes.to_int())
+endif
+
+# check that cpu and numa count is set and error out if it's not set
+if not dpdk_conf.has('RTE_MAX_LCORE')
+	error('Number of cores not specified.')
+endif
+if not dpdk_conf.has('RTE_MAX_NUMA_NODES')
+	error('Number of numa nodes not specified.')
+endif
+
 # set the install path for the drivers
 dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
 
diff --git a/config/x86/meson.build b/config/x86/meson.build
index b9348c44de..872dda9585 100644
--- a/config/x86/meson.build
+++ b/config/x86/meson.build
@@ -57,3 +57,5 @@  else
 endif
 
 dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
+dpdk_conf.set('RTE_MAX_LCORE', 128)
+dpdk_conf.set('RTE_MAX_NUMA_NODES', 32)
diff --git a/meson_options.txt b/meson_options.txt
index 56bdfd0f0a..649a60abc9 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -32,10 +32,10 @@  option('machine', type: 'string', value: 'native', description:
        'set the target machine type or "generic", a build usable on all machines of the build machine architecture or "native", which lets the compiler pick the architecture of the build machine.')
 option('max_ethports', type: 'integer', value: 32, description:
        'maximum number of Ethernet devices')
-option('max_lcores', type: 'integer', value: 128, description:
-       'maximum number of cores/threads supported by EAL')
-option('max_numa_nodes', type: 'integer', value: 32, description:
-       'maximum number of NUMA nodes supported by EAL')
+option('max_lcores', type: 'string', value: 'default', description:
+       'Set maximum number of cores/threads supported by EAL. The default is different per-arch. Set to auto to detect the number of cores on the build machine.')
+option('max_numa_nodes', type: 'string', value: 'default', description:
+       'Set highest NUMA node supported by EAL. The default is different per-arch. Set to auto to detect the highest numa node on the build machine.')
 option('platform', type: 'string', value: '', description:
        'use configuration for a particular platform (such as a SoC).')
 option('enable_trace_fp', type: 'boolean', value: false, description: