diff mbox series

[v5] build: optional NUMA and cpu counts detection

Message ID 1626872655-27496-1-git-send-email-juraj.linkes@pantheon.tech (mailing list archive)
State Superseded, archived
Delegated to: Thomas Monjalon
Headers show
Series [v5] build: optional NUMA and cpu counts detection | expand

Checks

Context Check Description
ci/intel-Testing success Testing PASS
ci/Intel-compilation success Compilation OK
ci/iol-testing success Testing PASS
ci/iol-abi-testing success Testing PASS
ci/iol-intel-Performance success Performance Testing PASS
ci/iol-intel-Functional success Functional Testing PASS
ci/github-robot success github build: passed
ci/checkpatch success coding style OK

Commit Message

Juraj Linkeš July 21, 2021, 1:04 p.m. UTC
Add an option to automatically discover the host's numa and cpu counts
and use those values for a non cross-build.
Give users the option to override the per-arch default values or values
from cross files by specifying them on the command line with -Dmax_lcores
and -Dmax_numa_nodes.

Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: David Christensen <drc@linux.vnet.ibm.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
 MAINTAINERS                  |  2 ++
 buildtools/get-cpu-count.py  |  7 ++++++
 buildtools/get-numa-count.py | 24 ++++++++++++++++++
 buildtools/meson.build       |  2 ++
 config/meson.build           | 47 ++++++++++++++++++++++++++++++++++--
 config/x86/meson.build       |  2 ++
 meson_options.txt            |  8 +++---
 7 files changed, 86 insertions(+), 6 deletions(-)
 create mode 100644 buildtools/get-cpu-count.py
 create mode 100644 buildtools/get-numa-count.py

Comments

Juraj Linkeš Aug. 2, 2021, 12:44 p.m. UTC | #1
> +if os.name == 'posix':
> +    if os.path.isdir('/sys/devices/system/node'):
> +        numa_nodes = glob.glob('/sys/devices/system/node/node*')
> +        numa_nodes.sort()
> +        print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
> +    else:
> +        subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
> +

Bruce, David, Thomas,

Is DPDK actually supported on Power9 FreeBSD? Is anyone using this combination? How can we address the open question of what exactly does sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or should we just leave it as is? Or maybe add 1 to the output (as we do in other cases)?
David Christensen Aug. 2, 2021, 11:29 p.m. UTC | #2
On 8/2/21 5:44 AM, Juraj Linkeš wrote:
>> +if os.name == 'posix':
>> +    if os.path.isdir('/sys/devices/system/node'):
>> +        numa_nodes = glob.glob('/sys/devices/system/node/node*')
>> +        numa_nodes.sort()
>> +        print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
>> +    else:
>> +        subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
>> +
> 
> Bruce, David, Thomas,
> 
> Is DPDK actually supported on Power9 FreeBSD? Is anyone using this combination? How can we address the open question of what exactly does sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or should we just leave it as is? Or maybe add 1 to the output (as we do in other cases)?

Not supported within IBM, but you can buy OpenPOWER boxes from 3rd 
parties such as Raptor Computing Systems so there may be customers using 
DPDK on POWER with FreeBSD that I don't track.  Adding Piotr Kubaj who 
has commented on POWER/FreeBSD issues in this past.

Dave
Juraj Linkeš Aug. 3, 2021, 10:21 a.m. UTC | #3
> -----Original Message-----
> From: David Christensen <drc@linux.vnet.ibm.com>
> Sent: Tuesday, August 3, 2021 1:29 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; thomas@monjalon.net;
> david.marchand@redhat.com; bruce.richardson@intel.com;
> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> stephen@networkplumber.org; Piotr Kubaj <pkubaj@FreeBSD.org>
> Cc: dev@dpdk.org
> Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> 
> 
> 
> On 8/2/21 5:44 AM, Juraj Linkeš wrote:
> >> +if os.name == 'posix':
> >> +    if os.path.isdir('/sys/devices/system/node'):
> >> +        numa_nodes = glob.glob('/sys/devices/system/node/node*')
> >> +        numa_nodes.sort()
> >> +        print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
> >> +    else:
> >> +        subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
> >> +
> >
> > Bruce, David, Thomas,
> >
> > Is DPDK actually supported on Power9 FreeBSD? Is anyone using this
> combination? How can we address the open question of what exactly does
> sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or should we just
> leave it as is? Or maybe add 1 to the output (as we do in other cases)?
> 
> Not supported within IBM, but you can buy OpenPOWER boxes from 3rd parties
> such as Raptor Computing Systems so there may be customers using DPDK on
> POWER with FreeBSD that I don't track.  Adding Piotr Kubaj who has commented
> on POWER/FreeBSD issues in this past.
> 
> Dave

Thanks, David.

Piotr, to provide more context, we're trying to figure out what the highest NUMA node on a system is.
On P9 systems, here's how NUMA nodes look like in Linux:
NUMA node0 CPU(s):   0-63
NUMA node8 CPU(s):   64-127
NUMA node252 CPU(s):
NUMA node253 CPU(s):
NUMA node254 CPU(s):
NUMA node255 CPU(s):
The highest NUMA with CPUs is node8.

We're trying to get the highest NUMA with CPUs on P9 FreeBSD systems, but we don't know whether FreeBSD NUMA layout looks the same (does FreeBSD report non-contiguous NUMA nodes as Linxu above, or does it renumerate) a what does "systemctl -n vm.ndomains" return. Could you check these for us?
Piotr Kubaj Aug. 31, 2021, 12:54 a.m. UTC | #4
Hi,

sorry for the late answer.

I suppose you mean sysctl command, not systemctl.

On dual CPU systems, it returns 2. On single CPU ones, 1.

On 21-08-03 10:21:50, Juraj Linkeš wrote:
> 
> 
> > -----Original Message-----
> > From: David Christensen <drc@linux.vnet.ibm.com>
> > Sent: Tuesday, August 3, 2021 1:29 AM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>; thomas@monjalon.net;
> > david.marchand@redhat.com; bruce.richardson@intel.com;
> > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > stephen@networkplumber.org; Piotr Kubaj <pkubaj@FreeBSD.org>
> > Cc: dev@dpdk.org
> > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> > 
> > 
> > 
> > On 8/2/21 5:44 AM, Juraj Linkeš wrote:
> > >> +if os.name == 'posix':
> > >> +    if os.path.isdir('/sys/devices/system/node'):
> > >> +        numa_nodes = glob.glob('/sys/devices/system/node/node*')
> > >> +        numa_nodes.sort()
> > >> +        print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
> > >> +    else:
> > >> +        subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
> > >> +
> > >
> > > Bruce, David, Thomas,
> > >
> > > Is DPDK actually supported on Power9 FreeBSD? Is anyone using this
> > combination? How can we address the open question of what exactly does
> > sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or should we just
> > leave it as is? Or maybe add 1 to the output (as we do in other cases)?
> > 
> > Not supported within IBM, but you can buy OpenPOWER boxes from 3rd parties
> > such as Raptor Computing Systems so there may be customers using DPDK on
> > POWER with FreeBSD that I don't track.  Adding Piotr Kubaj who has commented
> > on POWER/FreeBSD issues in this past.
> > 
> > Dave
> 
> Thanks, David.
> 
> Piotr, to provide more context, we're trying to figure out what the highest NUMA node on a system is.
> On P9 systems, here's how NUMA nodes look like in Linux:
> NUMA node0 CPU(s):   0-63
> NUMA node8 CPU(s):   64-127
> NUMA node252 CPU(s):
> NUMA node253 CPU(s):
> NUMA node254 CPU(s):
> NUMA node255 CPU(s):
> The highest NUMA with CPUs is node8.
> 
> We're trying to get the highest NUMA with CPUs on P9 FreeBSD systems, but we don't know whether FreeBSD NUMA layout looks the same (does FreeBSD report non-contiguous NUMA nodes as Linxu above, or does it renumerate) a what does "systemctl -n vm.ndomains" return. Could you check these for us?
Juraj Linkeš Aug. 31, 2021, 7:54 a.m. UTC | #5
> -----Original Message-----
> From: Piotr Kubaj <pkubaj@anongoth.pl>
> Sent: Tuesday, August 31, 2021 2:55 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: David Christensen <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> david.marchand@redhat.com; bruce.richardson@intel.com;
> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> stephen@networkplumber.org; dev@dpdk.org
> Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> 
> Hi,
> 
> sorry for the late answer.

Thanks for the answer anyway, better late than never.

> 
> I suppose you mean sysctl command, not systemctl.
> 

That's right. What does lscpu say? Are the NUMA nodes non-contiguous like this?:
NUMA node0 CPU(s):   0-63
NUMA node8 CPU(s):   64-127
NUMA node252 CPU(s):
NUMA node253 CPU(s):
NUMA node254 CPU(s):
NUMA node255 CPU(s):

> On dual CPU systems, it returns 2. On single CPU ones, 1.

I asked the previous question so that we know the actual numa node number of the second CPU. If it's 8, then sysctl does some renumeration and we can't use it.

Bruce, maybe we should just parse lscpu output? That introduces a dependency, but that may not be such a big deal as lscpu is pretty common.

> 
> On 21-08-03 10:21:50, Juraj Linkeš wrote:
> >
> >
> > > -----Original Message-----
> > > From: David Christensen <drc@linux.vnet.ibm.com>
> > > Sent: Tuesday, August 3, 2021 1:29 AM
> > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>; thomas@monjalon.net;
> > > david.marchand@redhat.com; bruce.richardson@intel.com;
> > > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > > stephen@networkplumber.org; Piotr Kubaj <pkubaj@FreeBSD.org>
> > > Cc: dev@dpdk.org
> > > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts
> > > detection
> > >
> > >
> > >
> > > On 8/2/21 5:44 AM, Juraj Linkeš wrote:
> > > >> +if os.name == 'posix':
> > > >> +    if os.path.isdir('/sys/devices/system/node'):
> > > >> +        numa_nodes = glob.glob('/sys/devices/system/node/node*')
> > > >> +        numa_nodes.sort()
> > > >> +        print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
> > > >> +    else:
> > > >> +        subprocess.run(['sysctl', '-n', 'vm.ndomains'],
> > > >> +check=False)
> > > >> +
> > > >
> > > > Bruce, David, Thomas,
> > > >
> > > > Is DPDK actually supported on Power9 FreeBSD? Is anyone using this
> > > combination? How can we address the open question of what exactly
> > > does sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or
> > > should we just leave it as is? Or maybe add 1 to the output (as we do in other
> cases)?
> > >
> > > Not supported within IBM, but you can buy OpenPOWER boxes from 3rd
> > > parties such as Raptor Computing Systems so there may be customers
> > > using DPDK on POWER with FreeBSD that I don't track.  Adding Piotr
> > > Kubaj who has commented on POWER/FreeBSD issues in this past.
> > >
> > > Dave
> >
> > Thanks, David.
> >
> > Piotr, to provide more context, we're trying to figure out what the highest
> NUMA node on a system is.
> > On P9 systems, here's how NUMA nodes look like in Linux:
> > NUMA node0 CPU(s):   0-63
> > NUMA node8 CPU(s):   64-127
> > NUMA node252 CPU(s):
> > NUMA node253 CPU(s):
> > NUMA node254 CPU(s):
> > NUMA node255 CPU(s):
> > The highest NUMA with CPUs is node8.
> >
> > We're trying to get the highest NUMA with CPUs on P9 FreeBSD systems, but
> we don't know whether FreeBSD NUMA layout looks the same (does FreeBSD
> report non-contiguous NUMA nodes as Linxu above, or does it renumerate) a
> what does "systemctl -n vm.ndomains" return. Could you check these for us?
Bruce Richardson Aug. 31, 2021, 8:02 a.m. UTC | #6
On Tue, Aug 31, 2021 at 07:54:16AM +0000, Juraj Linkeš wrote:
> 
> 
> > -----Original Message-----
> > From: Piotr Kubaj <pkubaj@anongoth.pl>
> > Sent: Tuesday, August 31, 2021 2:55 AM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Cc: David Christensen <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> > david.marchand@redhat.com; bruce.richardson@intel.com;
> > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > stephen@networkplumber.org; dev@dpdk.org
> > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> > 
> > Hi,
> > 
> > sorry for the late answer.
> 
> Thanks for the answer anyway, better late than never.
> 
> > 
> > I suppose you mean sysctl command, not systemctl.
> > 
> 
> That's right. What does lscpu say? Are the NUMA nodes non-contiguous like this?:
> NUMA node0 CPU(s):   0-63
> NUMA node8 CPU(s):   64-127
> NUMA node252 CPU(s):
> NUMA node253 CPU(s):
> NUMA node254 CPU(s):
> NUMA node255 CPU(s):
> 
> > On dual CPU systems, it returns 2. On single CPU ones, 1.
> 
> I asked the previous question so that we know the actual numa node number of the second CPU. If it's 8, then sysctl does some renumeration and we can't use it.
> 
> Bruce, maybe we should just parse lscpu output? That introduces a dependency, but that may not be such a big deal as lscpu is pretty common.
> 
Until we are sure that we need it, can we just keep things simple? Perhaps
we can use lscpu if present, and fallback to sysctl output if not.
Piotr Kubaj Aug. 31, 2021, 12:32 p.m. UTC | #7
On 21-08-31 07:54:16, Juraj Linkeš wrote:
> 
> 
> > -----Original Message-----
> > From: Piotr Kubaj <pkubaj@anongoth.pl>
> > Sent: Tuesday, August 31, 2021 2:55 AM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Cc: David Christensen <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> > david.marchand@redhat.com; bruce.richardson@intel.com;
> > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > stephen@networkplumber.org; dev@dpdk.org
> > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> > 
> > Hi,
> > 
> > sorry for the late answer.
> 
> Thanks for the answer anyway, better late than never.
> 
> > 
> > I suppose you mean sysctl command, not systemctl.
> > 
> 
> That's right. What does lscpu say? Are the NUMA nodes non-contiguous like this?:
> NUMA node0 CPU(s):   0-63
> NUMA node8 CPU(s):   64-127
> NUMA node252 CPU(s):
> NUMA node253 CPU(s):
> NUMA node254 CPU(s):
> NUMA node255 CPU(s):
> 
> > On dual CPU systems, it returns 2. On single CPU ones, 1.
> 
> I asked the previous question so that we know the actual numa node number of the second CPU. If it's 8, then sysctl does some renumeration and we can't use it.
> 
> Bruce, maybe we should just parse lscpu output? That introduces a dependency, but that may not be such a big deal as lscpu is pretty common.
There's no lscpu on FreeBSD in the base system. There is one in packages, but it's written only for amd64 / i386 since it uses CPUID. https://github.com/NanXiao/lscpu


> 
> > 
> > On 21-08-03 10:21:50, Juraj Linkeš wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: David Christensen <drc@linux.vnet.ibm.com>
> > > > Sent: Tuesday, August 3, 2021 1:29 AM
> > > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>; thomas@monjalon.net;
> > > > david.marchand@redhat.com; bruce.richardson@intel.com;
> > > > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > > > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > > > stephen@networkplumber.org; Piotr Kubaj <pkubaj@FreeBSD.org>
> > > > Cc: dev@dpdk.org
> > > > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts
> > > > detection
> > > >
> > > >
> > > >
> > > > On 8/2/21 5:44 AM, Juraj Linkeš wrote:
> > > > >> +if os.name == 'posix':
> > > > >> +    if os.path.isdir('/sys/devices/system/node'):
> > > > >> +        numa_nodes = glob.glob('/sys/devices/system/node/node*')
> > > > >> +        numa_nodes.sort()
> > > > >> +        print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
> > > > >> +    else:
> > > > >> +        subprocess.run(['sysctl', '-n', 'vm.ndomains'],
> > > > >> +check=False)
> > > > >> +
> > > > >
> > > > > Bruce, David, Thomas,
> > > > >
> > > > > Is DPDK actually supported on Power9 FreeBSD? Is anyone using this
> > > > combination? How can we address the open question of what exactly
> > > > does sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or
> > > > should we just leave it as is? Or maybe add 1 to the output (as we do in other
> > cases)?
> > > >
> > > > Not supported within IBM, but you can buy OpenPOWER boxes from 3rd
> > > > parties such as Raptor Computing Systems so there may be customers
> > > > using DPDK on POWER with FreeBSD that I don't track.  Adding Piotr
> > > > Kubaj who has commented on POWER/FreeBSD issues in this past.
> > > >
> > > > Dave
> > >
> > > Thanks, David.
> > >
> > > Piotr, to provide more context, we're trying to figure out what the highest
> > NUMA node on a system is.
> > > On P9 systems, here's how NUMA nodes look like in Linux:
> > > NUMA node0 CPU(s):   0-63
> > > NUMA node8 CPU(s):   64-127
> > > NUMA node252 CPU(s):
> > > NUMA node253 CPU(s):
> > > NUMA node254 CPU(s):
> > > NUMA node255 CPU(s):
> > > The highest NUMA with CPUs is node8.
> > >
> > > We're trying to get the highest NUMA with CPUs on P9 FreeBSD systems, but
> > we don't know whether FreeBSD NUMA layout looks the same (does FreeBSD
> > report non-contiguous NUMA nodes as Linxu above, or does it renumerate) a
> > what does "systemctl -n vm.ndomains" return. Could you check these for us?

--
Juraj Linkeš Sept. 9, 2021, 7:20 a.m. UTC | #8
> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Tuesday, August 31, 2021 10:03 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: Piotr Kubaj <pkubaj@anongoth.pl>; David Christensen
> <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> david.marchand@redhat.com; Honnappa.Nagarahalli@arm.com;
> Ruifeng.Wang@arm.com; ferruh.yigit@intel.com; jerinjacobk@gmail.com;
> jerinj@marvell.com; stephen@networkplumber.org; dev@dpdk.org
> Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> 
> On Tue, Aug 31, 2021 at 07:54:16AM +0000, Juraj Linkeš wrote:
> >
> >
> > > -----Original Message-----
> > > From: Piotr Kubaj <pkubaj@anongoth.pl>
> > > Sent: Tuesday, August 31, 2021 2:55 AM
> > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > Cc: David Christensen <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> > > david.marchand@redhat.com; bruce.richardson@intel.com;
> > > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > > stephen@networkplumber.org; dev@dpdk.org
> > > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts
> > > detection
> > >
> > > Hi,
> > >
> > > sorry for the late answer.
> >
> > Thanks for the answer anyway, better late than never.
> >
> > >
> > > I suppose you mean sysctl command, not systemctl.
> > >
> >
> > That's right. What does lscpu say? Are the NUMA nodes non-contiguous like
> this?:
> > NUMA node0 CPU(s):   0-63
> > NUMA node8 CPU(s):   64-127
> > NUMA node252 CPU(s):
> > NUMA node253 CPU(s):
> > NUMA node254 CPU(s):
> > NUMA node255 CPU(s):
> >
> > > On dual CPU systems, it returns 2. On single CPU ones, 1.
> >
> > I asked the previous question so that we know the actual numa node number
> of the second CPU. If it's 8, then sysctl does some renumeration and we can't
> use it.
> >
> > Bruce, maybe we should just parse lscpu output? That introduces a
> dependency, but that may not be such a big deal as lscpu is pretty common.
> >
> Until we are sure that we need it, can we just keep things simple? Perhaps we
> can use lscpu if present, and fallback to sysctl output if not.

Piotr says lscpu isn't available on ppc FreeBSD. I think this leaves us with two options:
1. either use sysctl with the risk that it may not return an accurate value
2. not support PPC for numa detection

What do you think?
Bruce Richardson Sept. 9, 2021, 8:01 a.m. UTC | #9
On Thu, Sep 09, 2021 at 07:20:27AM +0000, Juraj Linkeš wrote:
> 
> 
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Tuesday, August 31, 2021 10:03 AM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Cc: Piotr Kubaj <pkubaj@anongoth.pl>; David Christensen
> > <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> > david.marchand@redhat.com; Honnappa.Nagarahalli@arm.com;
> > Ruifeng.Wang@arm.com; ferruh.yigit@intel.com; jerinjacobk@gmail.com;
> > jerinj@marvell.com; stephen@networkplumber.org; dev@dpdk.org
> > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> > 
> > On Tue, Aug 31, 2021 at 07:54:16AM +0000, Juraj Linkeš wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: Piotr Kubaj <pkubaj@anongoth.pl>
> > > > Sent: Tuesday, August 31, 2021 2:55 AM
> > > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > Cc: David Christensen <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> > > > david.marchand@redhat.com; bruce.richardson@intel.com;
> > > > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > > > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > > > stephen@networkplumber.org; dev@dpdk.org
> > > > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts
> > > > detection
> > > >
> > > > Hi,
> > > >
> > > > sorry for the late answer.
> > >
> > > Thanks for the answer anyway, better late than never.
> > >
> > > >
> > > > I suppose you mean sysctl command, not systemctl.
> > > >
> > >
> > > That's right. What does lscpu say? Are the NUMA nodes non-contiguous like
> > this?:
> > > NUMA node0 CPU(s):   0-63
> > > NUMA node8 CPU(s):   64-127
> > > NUMA node252 CPU(s):
> > > NUMA node253 CPU(s):
> > > NUMA node254 CPU(s):
> > > NUMA node255 CPU(s):
> > >
> > > > On dual CPU systems, it returns 2. On single CPU ones, 1.
> > >
> > > I asked the previous question so that we know the actual numa node number
> > of the second CPU. If it's 8, then sysctl does some renumeration and we can't
> > use it.
> > >
> > > Bruce, maybe we should just parse lscpu output? That introduces a
> > dependency, but that may not be such a big deal as lscpu is pretty common.
> > >
> > Until we are sure that we need it, can we just keep things simple? Perhaps we
> > can use lscpu if present, and fallback to sysctl output if not.
> 
> Piotr says lscpu isn't available on ppc FreeBSD. I think this leaves us with two options:
> 1. either use sysctl with the risk that it may not return an accurate value
> 2. not support PPC for numa detection
> 
> What do you think?

I think point #2 is more not supporting FreeBSD on PPC for numa detection
rather than not supporting PPC generally, right? Personally, I am just in
favour of #1 for the sake of moving forward and giving us best support with
simplest code. If or when someone does hit this (FreeBSD+PPC) as a problem,
then we can work with them to find best solution for fixing it. Right now
we don't have anyone with the PPC+FreeBSD setup, so I'd rather ignore it
than implement more complicated solutions that we can't test "just in
case".

/Bruce
Thomas Monjalon Sept. 9, 2021, 8:08 a.m. UTC | #10
09/09/2021 10:01, Bruce Richardson:
> On Thu, Sep 09, 2021 at 07:20:27AM +0000, Juraj Linkeš wrote:
> > Piotr says lscpu isn't available on ppc FreeBSD. I think this leaves us with two options:
> > 1. either use sysctl with the risk that it may not return an accurate value
> > 2. not support PPC for numa detection
> > 
> > What do you think?
> 
> I think point #2 is more not supporting FreeBSD on PPC for numa detection
> rather than not supporting PPC generally, right? Personally, I am just in
> favour of #1 for the sake of moving forward and giving us best support with
> simplest code. If or when someone does hit this (FreeBSD+PPC) as a problem,
> then we can work with them to find best solution for fixing it. Right now
> we don't have anyone with the PPC+FreeBSD setup, so I'd rather ignore it
> than implement more complicated solutions that we can't test "just in
> case".

+1, FreeBSD + PPC is not worth the complication
Juraj Linkeš Sept. 9, 2021, 8:42 a.m. UTC | #11
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, September 9, 2021 10:09 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; Bruce Richardson
> <bruce.richardson@intel.com>
> Cc: Piotr Kubaj <pkubaj@anongoth.pl>; David Christensen
> <drc@linux.vnet.ibm.com>; david.marchand@redhat.com;
> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> stephen@networkplumber.org; dev@dpdk.org
> Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> 
> 09/09/2021 10:01, Bruce Richardson:
> > On Thu, Sep 09, 2021 at 07:20:27AM +0000, Juraj Linkeš wrote:
> > > Piotr says lscpu isn't available on ppc FreeBSD. I think this leaves us with two
> options:
> > > 1. either use sysctl with the risk that it may not return an
> > > accurate value 2. not support PPC for numa detection
> > >
> > > What do you think?
> >
> > I think point #2 is more not supporting FreeBSD on PPC for numa
> > detection rather than not supporting PPC generally, right? Personally,
> > I am just in favour of #1 for the sake of moving forward and giving us
> > best support with simplest code. If or when someone does hit this
> > (FreeBSD+PPC) as a problem, then we can work with them to find best
> > solution for fixing it. Right now we don't have anyone with the
> > PPC+FreeBSD setup, so I'd rather ignore it than implement more
> > complicated solutions that we can't test "just in case".
> 
> +1, FreeBSD + PPC is not worth the complication
> 
> 

Thanks for the feedback, makes sense to me. I think v6 has all the features we want then - linux, windows and freebsd using sysctl.
David Christensen Sept. 9, 2021, 4:44 p.m. UTC | #12
>> Piotr says lscpu isn't available on ppc FreeBSD. I think this leaves us with two options:
>> 1. either use sysctl with the risk that it may not return an accurate value
>> 2. not support PPC for numa detection
>>
>> What do you think?
> 
> I think point #2 is more not supporting FreeBSD on PPC for numa detection
> rather than not supporting PPC generally, right? Personally, I am just in
> favour of #1 for the sake of moving forward and giving us best support with
> simplest code. If or when someone does hit this (FreeBSD+PPC) as a problem,
> then we can work with them to find best solution for fixing it. Right now
> we don't have anyone with the PPC+FreeBSD setup, so I'd rather ignore it
> than implement more complicated solutions that we can't test "just in
> case".

Fully agree here.  Can't expect a user app to completely resolve all 
limitations in the underlying OS.

Dave
diff mbox series

Patch

diff --git a/MAINTAINERS b/MAINTAINERS
index af2a91d7c4..f0e82598aa 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -102,6 +102,8 @@  F: meson_options.txt
 F: config/
 F: buildtools/chkincs/
 F: buildtools/call-sphinx-build.py
+F: buildtools/get-cpu-count.py
+F: buildtools/get-numa-count.py
 F: buildtools/list-dir-globs.py
 F: buildtools/pkg-config/
 F: buildtools/symlink-drivers-solibs.sh
diff --git a/buildtools/get-cpu-count.py b/buildtools/get-cpu-count.py
new file mode 100644
index 0000000000..317b32088f
--- /dev/null
+++ b/buildtools/get-cpu-count.py
@@ -0,0 +1,7 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 PANTHEON.tech s.r.o.
+
+import os
+
+print(os.cpu_count())
diff --git a/buildtools/get-numa-count.py b/buildtools/get-numa-count.py
new file mode 100644
index 0000000000..1b7787787f
--- /dev/null
+++ b/buildtools/get-numa-count.py
@@ -0,0 +1,24 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 PANTHEON.tech s.r.o.
+
+import ctypes
+import glob
+import os
+import subprocess
+
+if os.name == 'posix':
+    if os.path.isdir('/sys/devices/system/node'):
+        numa_nodes = glob.glob('/sys/devices/system/node/node*')
+        numa_nodes.sort()
+        print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
+    else:
+        subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
+
+elif os.name == 'nt':
+    libkernel32 = ctypes.windll.kernel32
+
+    numa_count = ctypes.c_ulong()
+
+    libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(numa_count))
+    print(numa_count.value + 1)
diff --git a/buildtools/meson.build b/buildtools/meson.build
index bd460e3e00..f776316da1 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -16,6 +16,8 @@  echo = py3 + ['-c', 'import sys; print(*sys.argv[1:])']
 list_dir_globs = py3 + files('list-dir-globs.py')
 map_to_win_cmd = py3 + files('map_to_win.py')
 sphinx_wrapper = py3 + files('call-sphinx-build.py')
+get_cpu_count_cmd = py3 + files('get-cpu-count.py')
+get_numa_count_cmd = py3 + files('get-numa-count.py')
 
 # select library and object file format
 pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
diff --git a/config/meson.build b/config/meson.build
index e80421003b..364788c32d 100644
--- a/config/meson.build
+++ b/config/meson.build
@@ -286,8 +286,6 @@  foreach arg: warning_flags
 endforeach
 
 # set other values pulled from the build options
-dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
-dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
 dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
 dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
 dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
@@ -321,6 +319,51 @@  if meson.is_cross_build()
     endif
 endif
 
+max_lcores = get_option('max_lcores')
+if max_lcores == 'detect'
+	# Discovery makes sense only for non-cross builds
+	if meson.is_cross_build()
+		error('Discovery of max_lcores is not supported for cross-compilation.')
+	endif
+	# Overwrite the default value with discovered values
+	max_lcores = run_command(get_cpu_count_cmd).stdout().to_int()
+	min_lcores = 2
+	# DPDK must be build for at least 2 cores
+	if max_lcores < min_lcores
+		message('Found less than @0@ cores, building for @0@ cores'.format(min_lcores))
+		max_lcores = min_lcores
+	else
+		message('Found @0@ cores'.format(max_lcores))
+	endif
+	dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+elif max_lcores != 'default'
+	# Overwrite the default value from arch_subdir with user input
+	dpdk_conf.set('RTE_MAX_LCORE', max_lcores.to_int())
+endif
+
+max_numa_nodes = get_option('max_numa_nodes')
+if max_numa_nodes == 'detect'
+	# Discovery makes sense only for non-cross builds
+	if meson.is_cross_build()
+		error('Discovery of max_numa_nodes not supported for cross-compilation.')
+	endif
+	# Overwrite the default value with discovered values
+	max_numa_nodes = run_command(get_numa_count_cmd).stdout().to_int()
+	message('Found @0@ numa nodes'.format(max_numa_nodes))
+	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+elif max_numa_nodes != 'default'
+	# Overwrite the default value from arch_subdir with user input
+	dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes.to_int())
+endif
+
+# check that cpu and numa count is set and error out if it's not set
+if not dpdk_conf.has('RTE_MAX_LCORE')
+	error('Number of cores not specified.')
+endif
+if not dpdk_conf.has('RTE_MAX_NUMA_NODES')
+	error('Number of numa nodes not specified.')
+endif
+
 # set the install path for the drivers
 dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
 
diff --git a/config/x86/meson.build b/config/x86/meson.build
index b9348c44de..872dda9585 100644
--- a/config/x86/meson.build
+++ b/config/x86/meson.build
@@ -57,3 +57,5 @@  else
 endif
 
 dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
+dpdk_conf.set('RTE_MAX_LCORE', 128)
+dpdk_conf.set('RTE_MAX_NUMA_NODES', 32)
diff --git a/meson_options.txt b/meson_options.txt
index 0e92734c49..b56e3b83c7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -34,10 +34,10 @@  option('machine', type: 'string', value: 'auto', description:
        'Alias of cpu_instruction_set.')
 option('max_ethports', type: 'integer', value: 32, description:
        'maximum number of Ethernet devices')
-option('max_lcores', type: 'integer', value: 128, description:
-       'maximum number of cores/threads supported by EAL')
-option('max_numa_nodes', type: 'integer', value: 32, description:
-       'maximum number of NUMA nodes supported by EAL')
+option('max_lcores', type: 'string', value: 'default', description:
+       'Set maximum number of cores/threads supported by EAL; "default" is different per-arch, "detect" detects the number of cores on the build machine.')
+option('max_numa_nodes', type: 'string', value: 'default', description:
+       'Set the highest NUMA node supported by EAL; "default" is different per-arch, "detect" detects the highest numa node on the build machine.')
 option('platform', type: 'string', value: 'native', description:
        'Platform to build, either "native", "generic" or a SoC. Please refer to the Linux build guide for more information.')
 option('enable_trace_fp', type: 'boolean', value: false, description: