[v5] build: optional NUMA and cpu counts detection
Checks
Commit Message
Add an option to automatically discover the host's numa and cpu counts
and use those values for a non cross-build.
Give users the option to override the per-arch default values or values
from cross files by specifying them on the command line with -Dmax_lcores
and -Dmax_numa_nodes.
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: David Christensen <drc@linux.vnet.ibm.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
---
MAINTAINERS | 2 ++
buildtools/get-cpu-count.py | 7 ++++++
buildtools/get-numa-count.py | 24 ++++++++++++++++++
buildtools/meson.build | 2 ++
config/meson.build | 47 ++++++++++++++++++++++++++++++++++--
config/x86/meson.build | 2 ++
meson_options.txt | 8 +++---
7 files changed, 86 insertions(+), 6 deletions(-)
create mode 100644 buildtools/get-cpu-count.py
create mode 100644 buildtools/get-numa-count.py
Comments
> +if os.name == 'posix':
> + if os.path.isdir('/sys/devices/system/node'):
> + numa_nodes = glob.glob('/sys/devices/system/node/node*')
> + numa_nodes.sort()
> + print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
> + else:
> + subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
> +
Bruce, David, Thomas,
Is DPDK actually supported on Power9 FreeBSD? Is anyone using this combination? How can we address the open question of what exactly does sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or should we just leave it as is? Or maybe add 1 to the output (as we do in other cases)?
On 8/2/21 5:44 AM, Juraj Linkeš wrote:
>> +if os.name == 'posix':
>> + if os.path.isdir('/sys/devices/system/node'):
>> + numa_nodes = glob.glob('/sys/devices/system/node/node*')
>> + numa_nodes.sort()
>> + print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
>> + else:
>> + subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
>> +
>
> Bruce, David, Thomas,
>
> Is DPDK actually supported on Power9 FreeBSD? Is anyone using this combination? How can we address the open question of what exactly does sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or should we just leave it as is? Or maybe add 1 to the output (as we do in other cases)?
Not supported within IBM, but you can buy OpenPOWER boxes from 3rd
parties such as Raptor Computing Systems so there may be customers using
DPDK on POWER with FreeBSD that I don't track. Adding Piotr Kubaj who
has commented on POWER/FreeBSD issues in this past.
Dave
> -----Original Message-----
> From: David Christensen <drc@linux.vnet.ibm.com>
> Sent: Tuesday, August 3, 2021 1:29 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; thomas@monjalon.net;
> david.marchand@redhat.com; bruce.richardson@intel.com;
> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> stephen@networkplumber.org; Piotr Kubaj <pkubaj@FreeBSD.org>
> Cc: dev@dpdk.org
> Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
>
>
>
> On 8/2/21 5:44 AM, Juraj Linkeš wrote:
> >> +if os.name == 'posix':
> >> + if os.path.isdir('/sys/devices/system/node'):
> >> + numa_nodes = glob.glob('/sys/devices/system/node/node*')
> >> + numa_nodes.sort()
> >> + print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
> >> + else:
> >> + subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
> >> +
> >
> > Bruce, David, Thomas,
> >
> > Is DPDK actually supported on Power9 FreeBSD? Is anyone using this
> combination? How can we address the open question of what exactly does
> sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or should we just
> leave it as is? Or maybe add 1 to the output (as we do in other cases)?
>
> Not supported within IBM, but you can buy OpenPOWER boxes from 3rd parties
> such as Raptor Computing Systems so there may be customers using DPDK on
> POWER with FreeBSD that I don't track. Adding Piotr Kubaj who has commented
> on POWER/FreeBSD issues in this past.
>
> Dave
Thanks, David.
Piotr, to provide more context, we're trying to figure out what the highest NUMA node on a system is.
On P9 systems, here's how NUMA nodes look like in Linux:
NUMA node0 CPU(s): 0-63
NUMA node8 CPU(s): 64-127
NUMA node252 CPU(s):
NUMA node253 CPU(s):
NUMA node254 CPU(s):
NUMA node255 CPU(s):
The highest NUMA with CPUs is node8.
We're trying to get the highest NUMA with CPUs on P9 FreeBSD systems, but we don't know whether FreeBSD NUMA layout looks the same (does FreeBSD report non-contiguous NUMA nodes as Linxu above, or does it renumerate) a what does "systemctl -n vm.ndomains" return. Could you check these for us?
Hi,
sorry for the late answer.
I suppose you mean sysctl command, not systemctl.
On dual CPU systems, it returns 2. On single CPU ones, 1.
On 21-08-03 10:21:50, Juraj Linkeš wrote:
>
>
> > -----Original Message-----
> > From: David Christensen <drc@linux.vnet.ibm.com>
> > Sent: Tuesday, August 3, 2021 1:29 AM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>; thomas@monjalon.net;
> > david.marchand@redhat.com; bruce.richardson@intel.com;
> > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > stephen@networkplumber.org; Piotr Kubaj <pkubaj@FreeBSD.org>
> > Cc: dev@dpdk.org
> > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> >
> >
> >
> > On 8/2/21 5:44 AM, Juraj Linkeš wrote:
> > >> +if os.name == 'posix':
> > >> + if os.path.isdir('/sys/devices/system/node'):
> > >> + numa_nodes = glob.glob('/sys/devices/system/node/node*')
> > >> + numa_nodes.sort()
> > >> + print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
> > >> + else:
> > >> + subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
> > >> +
> > >
> > > Bruce, David, Thomas,
> > >
> > > Is DPDK actually supported on Power9 FreeBSD? Is anyone using this
> > combination? How can we address the open question of what exactly does
> > sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or should we just
> > leave it as is? Or maybe add 1 to the output (as we do in other cases)?
> >
> > Not supported within IBM, but you can buy OpenPOWER boxes from 3rd parties
> > such as Raptor Computing Systems so there may be customers using DPDK on
> > POWER with FreeBSD that I don't track. Adding Piotr Kubaj who has commented
> > on POWER/FreeBSD issues in this past.
> >
> > Dave
>
> Thanks, David.
>
> Piotr, to provide more context, we're trying to figure out what the highest NUMA node on a system is.
> On P9 systems, here's how NUMA nodes look like in Linux:
> NUMA node0 CPU(s): 0-63
> NUMA node8 CPU(s): 64-127
> NUMA node252 CPU(s):
> NUMA node253 CPU(s):
> NUMA node254 CPU(s):
> NUMA node255 CPU(s):
> The highest NUMA with CPUs is node8.
>
> We're trying to get the highest NUMA with CPUs on P9 FreeBSD systems, but we don't know whether FreeBSD NUMA layout looks the same (does FreeBSD report non-contiguous NUMA nodes as Linxu above, or does it renumerate) a what does "systemctl -n vm.ndomains" return. Could you check these for us?
> -----Original Message-----
> From: Piotr Kubaj <pkubaj@anongoth.pl>
> Sent: Tuesday, August 31, 2021 2:55 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: David Christensen <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> david.marchand@redhat.com; bruce.richardson@intel.com;
> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> stephen@networkplumber.org; dev@dpdk.org
> Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
>
> Hi,
>
> sorry for the late answer.
Thanks for the answer anyway, better late than never.
>
> I suppose you mean sysctl command, not systemctl.
>
That's right. What does lscpu say? Are the NUMA nodes non-contiguous like this?:
NUMA node0 CPU(s): 0-63
NUMA node8 CPU(s): 64-127
NUMA node252 CPU(s):
NUMA node253 CPU(s):
NUMA node254 CPU(s):
NUMA node255 CPU(s):
> On dual CPU systems, it returns 2. On single CPU ones, 1.
I asked the previous question so that we know the actual numa node number of the second CPU. If it's 8, then sysctl does some renumeration and we can't use it.
Bruce, maybe we should just parse lscpu output? That introduces a dependency, but that may not be such a big deal as lscpu is pretty common.
>
> On 21-08-03 10:21:50, Juraj Linkeš wrote:
> >
> >
> > > -----Original Message-----
> > > From: David Christensen <drc@linux.vnet.ibm.com>
> > > Sent: Tuesday, August 3, 2021 1:29 AM
> > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>; thomas@monjalon.net;
> > > david.marchand@redhat.com; bruce.richardson@intel.com;
> > > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > > stephen@networkplumber.org; Piotr Kubaj <pkubaj@FreeBSD.org>
> > > Cc: dev@dpdk.org
> > > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts
> > > detection
> > >
> > >
> > >
> > > On 8/2/21 5:44 AM, Juraj Linkeš wrote:
> > > >> +if os.name == 'posix':
> > > >> + if os.path.isdir('/sys/devices/system/node'):
> > > >> + numa_nodes = glob.glob('/sys/devices/system/node/node*')
> > > >> + numa_nodes.sort()
> > > >> + print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
> > > >> + else:
> > > >> + subprocess.run(['sysctl', '-n', 'vm.ndomains'],
> > > >> +check=False)
> > > >> +
> > > >
> > > > Bruce, David, Thomas,
> > > >
> > > > Is DPDK actually supported on Power9 FreeBSD? Is anyone using this
> > > combination? How can we address the open question of what exactly
> > > does sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or
> > > should we just leave it as is? Or maybe add 1 to the output (as we do in other
> cases)?
> > >
> > > Not supported within IBM, but you can buy OpenPOWER boxes from 3rd
> > > parties such as Raptor Computing Systems so there may be customers
> > > using DPDK on POWER with FreeBSD that I don't track. Adding Piotr
> > > Kubaj who has commented on POWER/FreeBSD issues in this past.
> > >
> > > Dave
> >
> > Thanks, David.
> >
> > Piotr, to provide more context, we're trying to figure out what the highest
> NUMA node on a system is.
> > On P9 systems, here's how NUMA nodes look like in Linux:
> > NUMA node0 CPU(s): 0-63
> > NUMA node8 CPU(s): 64-127
> > NUMA node252 CPU(s):
> > NUMA node253 CPU(s):
> > NUMA node254 CPU(s):
> > NUMA node255 CPU(s):
> > The highest NUMA with CPUs is node8.
> >
> > We're trying to get the highest NUMA with CPUs on P9 FreeBSD systems, but
> we don't know whether FreeBSD NUMA layout looks the same (does FreeBSD
> report non-contiguous NUMA nodes as Linxu above, or does it renumerate) a
> what does "systemctl -n vm.ndomains" return. Could you check these for us?
On Tue, Aug 31, 2021 at 07:54:16AM +0000, Juraj Linkeš wrote:
>
>
> > -----Original Message-----
> > From: Piotr Kubaj <pkubaj@anongoth.pl>
> > Sent: Tuesday, August 31, 2021 2:55 AM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Cc: David Christensen <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> > david.marchand@redhat.com; bruce.richardson@intel.com;
> > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > stephen@networkplumber.org; dev@dpdk.org
> > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> >
> > Hi,
> >
> > sorry for the late answer.
>
> Thanks for the answer anyway, better late than never.
>
> >
> > I suppose you mean sysctl command, not systemctl.
> >
>
> That's right. What does lscpu say? Are the NUMA nodes non-contiguous like this?:
> NUMA node0 CPU(s): 0-63
> NUMA node8 CPU(s): 64-127
> NUMA node252 CPU(s):
> NUMA node253 CPU(s):
> NUMA node254 CPU(s):
> NUMA node255 CPU(s):
>
> > On dual CPU systems, it returns 2. On single CPU ones, 1.
>
> I asked the previous question so that we know the actual numa node number of the second CPU. If it's 8, then sysctl does some renumeration and we can't use it.
>
> Bruce, maybe we should just parse lscpu output? That introduces a dependency, but that may not be such a big deal as lscpu is pretty common.
>
Until we are sure that we need it, can we just keep things simple? Perhaps
we can use lscpu if present, and fallback to sysctl output if not.
On 21-08-31 07:54:16, Juraj Linkeš wrote:
>
>
> > -----Original Message-----
> > From: Piotr Kubaj <pkubaj@anongoth.pl>
> > Sent: Tuesday, August 31, 2021 2:55 AM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Cc: David Christensen <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> > david.marchand@redhat.com; bruce.richardson@intel.com;
> > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > stephen@networkplumber.org; dev@dpdk.org
> > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> >
> > Hi,
> >
> > sorry for the late answer.
>
> Thanks for the answer anyway, better late than never.
>
> >
> > I suppose you mean sysctl command, not systemctl.
> >
>
> That's right. What does lscpu say? Are the NUMA nodes non-contiguous like this?:
> NUMA node0 CPU(s): 0-63
> NUMA node8 CPU(s): 64-127
> NUMA node252 CPU(s):
> NUMA node253 CPU(s):
> NUMA node254 CPU(s):
> NUMA node255 CPU(s):
>
> > On dual CPU systems, it returns 2. On single CPU ones, 1.
>
> I asked the previous question so that we know the actual numa node number of the second CPU. If it's 8, then sysctl does some renumeration and we can't use it.
>
> Bruce, maybe we should just parse lscpu output? That introduces a dependency, but that may not be such a big deal as lscpu is pretty common.
There's no lscpu on FreeBSD in the base system. There is one in packages, but it's written only for amd64 / i386 since it uses CPUID. https://github.com/NanXiao/lscpu
>
> >
> > On 21-08-03 10:21:50, Juraj Linkeš wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: David Christensen <drc@linux.vnet.ibm.com>
> > > > Sent: Tuesday, August 3, 2021 1:29 AM
> > > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>; thomas@monjalon.net;
> > > > david.marchand@redhat.com; bruce.richardson@intel.com;
> > > > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > > > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > > > stephen@networkplumber.org; Piotr Kubaj <pkubaj@FreeBSD.org>
> > > > Cc: dev@dpdk.org
> > > > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts
> > > > detection
> > > >
> > > >
> > > >
> > > > On 8/2/21 5:44 AM, Juraj Linkeš wrote:
> > > > >> +if os.name == 'posix':
> > > > >> + if os.path.isdir('/sys/devices/system/node'):
> > > > >> + numa_nodes = glob.glob('/sys/devices/system/node/node*')
> > > > >> + numa_nodes.sort()
> > > > >> + print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
> > > > >> + else:
> > > > >> + subprocess.run(['sysctl', '-n', 'vm.ndomains'],
> > > > >> +check=False)
> > > > >> +
> > > > >
> > > > > Bruce, David, Thomas,
> > > > >
> > > > > Is DPDK actually supported on Power9 FreeBSD? Is anyone using this
> > > > combination? How can we address the open question of what exactly
> > > > does sysctl -n vm.ndomains return on a Power9 FreeBSD system? Or
> > > > should we just leave it as is? Or maybe add 1 to the output (as we do in other
> > cases)?
> > > >
> > > > Not supported within IBM, but you can buy OpenPOWER boxes from 3rd
> > > > parties such as Raptor Computing Systems so there may be customers
> > > > using DPDK on POWER with FreeBSD that I don't track. Adding Piotr
> > > > Kubaj who has commented on POWER/FreeBSD issues in this past.
> > > >
> > > > Dave
> > >
> > > Thanks, David.
> > >
> > > Piotr, to provide more context, we're trying to figure out what the highest
> > NUMA node on a system is.
> > > On P9 systems, here's how NUMA nodes look like in Linux:
> > > NUMA node0 CPU(s): 0-63
> > > NUMA node8 CPU(s): 64-127
> > > NUMA node252 CPU(s):
> > > NUMA node253 CPU(s):
> > > NUMA node254 CPU(s):
> > > NUMA node255 CPU(s):
> > > The highest NUMA with CPUs is node8.
> > >
> > > We're trying to get the highest NUMA with CPUs on P9 FreeBSD systems, but
> > we don't know whether FreeBSD NUMA layout looks the same (does FreeBSD
> > report non-contiguous NUMA nodes as Linxu above, or does it renumerate) a
> > what does "systemctl -n vm.ndomains" return. Could you check these for us?
--
> -----Original Message-----
> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Tuesday, August 31, 2021 10:03 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: Piotr Kubaj <pkubaj@anongoth.pl>; David Christensen
> <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> david.marchand@redhat.com; Honnappa.Nagarahalli@arm.com;
> Ruifeng.Wang@arm.com; ferruh.yigit@intel.com; jerinjacobk@gmail.com;
> jerinj@marvell.com; stephen@networkplumber.org; dev@dpdk.org
> Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
>
> On Tue, Aug 31, 2021 at 07:54:16AM +0000, Juraj Linkeš wrote:
> >
> >
> > > -----Original Message-----
> > > From: Piotr Kubaj <pkubaj@anongoth.pl>
> > > Sent: Tuesday, August 31, 2021 2:55 AM
> > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > Cc: David Christensen <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> > > david.marchand@redhat.com; bruce.richardson@intel.com;
> > > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > > stephen@networkplumber.org; dev@dpdk.org
> > > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts
> > > detection
> > >
> > > Hi,
> > >
> > > sorry for the late answer.
> >
> > Thanks for the answer anyway, better late than never.
> >
> > >
> > > I suppose you mean sysctl command, not systemctl.
> > >
> >
> > That's right. What does lscpu say? Are the NUMA nodes non-contiguous like
> this?:
> > NUMA node0 CPU(s): 0-63
> > NUMA node8 CPU(s): 64-127
> > NUMA node252 CPU(s):
> > NUMA node253 CPU(s):
> > NUMA node254 CPU(s):
> > NUMA node255 CPU(s):
> >
> > > On dual CPU systems, it returns 2. On single CPU ones, 1.
> >
> > I asked the previous question so that we know the actual numa node number
> of the second CPU. If it's 8, then sysctl does some renumeration and we can't
> use it.
> >
> > Bruce, maybe we should just parse lscpu output? That introduces a
> dependency, but that may not be such a big deal as lscpu is pretty common.
> >
> Until we are sure that we need it, can we just keep things simple? Perhaps we
> can use lscpu if present, and fallback to sysctl output if not.
Piotr says lscpu isn't available on ppc FreeBSD. I think this leaves us with two options:
1. either use sysctl with the risk that it may not return an accurate value
2. not support PPC for numa detection
What do you think?
On Thu, Sep 09, 2021 at 07:20:27AM +0000, Juraj Linkeš wrote:
>
>
> > -----Original Message-----
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Tuesday, August 31, 2021 10:03 AM
> > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > Cc: Piotr Kubaj <pkubaj@anongoth.pl>; David Christensen
> > <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> > david.marchand@redhat.com; Honnappa.Nagarahalli@arm.com;
> > Ruifeng.Wang@arm.com; ferruh.yigit@intel.com; jerinjacobk@gmail.com;
> > jerinj@marvell.com; stephen@networkplumber.org; dev@dpdk.org
> > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
> >
> > On Tue, Aug 31, 2021 at 07:54:16AM +0000, Juraj Linkeš wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: Piotr Kubaj <pkubaj@anongoth.pl>
> > > > Sent: Tuesday, August 31, 2021 2:55 AM
> > > > To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> > > > Cc: David Christensen <drc@linux.vnet.ibm.com>; thomas@monjalon.net;
> > > > david.marchand@redhat.com; bruce.richardson@intel.com;
> > > > Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> > > > ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> > > > stephen@networkplumber.org; dev@dpdk.org
> > > > Subject: Re: [PATCH v5] build: optional NUMA and cpu counts
> > > > detection
> > > >
> > > > Hi,
> > > >
> > > > sorry for the late answer.
> > >
> > > Thanks for the answer anyway, better late than never.
> > >
> > > >
> > > > I suppose you mean sysctl command, not systemctl.
> > > >
> > >
> > > That's right. What does lscpu say? Are the NUMA nodes non-contiguous like
> > this?:
> > > NUMA node0 CPU(s): 0-63
> > > NUMA node8 CPU(s): 64-127
> > > NUMA node252 CPU(s):
> > > NUMA node253 CPU(s):
> > > NUMA node254 CPU(s):
> > > NUMA node255 CPU(s):
> > >
> > > > On dual CPU systems, it returns 2. On single CPU ones, 1.
> > >
> > > I asked the previous question so that we know the actual numa node number
> > of the second CPU. If it's 8, then sysctl does some renumeration and we can't
> > use it.
> > >
> > > Bruce, maybe we should just parse lscpu output? That introduces a
> > dependency, but that may not be such a big deal as lscpu is pretty common.
> > >
> > Until we are sure that we need it, can we just keep things simple? Perhaps we
> > can use lscpu if present, and fallback to sysctl output if not.
>
> Piotr says lscpu isn't available on ppc FreeBSD. I think this leaves us with two options:
> 1. either use sysctl with the risk that it may not return an accurate value
> 2. not support PPC for numa detection
>
> What do you think?
I think point #2 is more not supporting FreeBSD on PPC for numa detection
rather than not supporting PPC generally, right? Personally, I am just in
favour of #1 for the sake of moving forward and giving us best support with
simplest code. If or when someone does hit this (FreeBSD+PPC) as a problem,
then we can work with them to find best solution for fixing it. Right now
we don't have anyone with the PPC+FreeBSD setup, so I'd rather ignore it
than implement more complicated solutions that we can't test "just in
case".
/Bruce
09/09/2021 10:01, Bruce Richardson:
> On Thu, Sep 09, 2021 at 07:20:27AM +0000, Juraj Linkeš wrote:
> > Piotr says lscpu isn't available on ppc FreeBSD. I think this leaves us with two options:
> > 1. either use sysctl with the risk that it may not return an accurate value
> > 2. not support PPC for numa detection
> >
> > What do you think?
>
> I think point #2 is more not supporting FreeBSD on PPC for numa detection
> rather than not supporting PPC generally, right? Personally, I am just in
> favour of #1 for the sake of moving forward and giving us best support with
> simplest code. If or when someone does hit this (FreeBSD+PPC) as a problem,
> then we can work with them to find best solution for fixing it. Right now
> we don't have anyone with the PPC+FreeBSD setup, so I'd rather ignore it
> than implement more complicated solutions that we can't test "just in
> case".
+1, FreeBSD + PPC is not worth the complication
> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, September 9, 2021 10:09 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; Bruce Richardson
> <bruce.richardson@intel.com>
> Cc: Piotr Kubaj <pkubaj@anongoth.pl>; David Christensen
> <drc@linux.vnet.ibm.com>; david.marchand@redhat.com;
> Honnappa.Nagarahalli@arm.com; Ruifeng.Wang@arm.com;
> ferruh.yigit@intel.com; jerinjacobk@gmail.com; jerinj@marvell.com;
> stephen@networkplumber.org; dev@dpdk.org
> Subject: Re: [PATCH v5] build: optional NUMA and cpu counts detection
>
> 09/09/2021 10:01, Bruce Richardson:
> > On Thu, Sep 09, 2021 at 07:20:27AM +0000, Juraj Linkeš wrote:
> > > Piotr says lscpu isn't available on ppc FreeBSD. I think this leaves us with two
> options:
> > > 1. either use sysctl with the risk that it may not return an
> > > accurate value 2. not support PPC for numa detection
> > >
> > > What do you think?
> >
> > I think point #2 is more not supporting FreeBSD on PPC for numa
> > detection rather than not supporting PPC generally, right? Personally,
> > I am just in favour of #1 for the sake of moving forward and giving us
> > best support with simplest code. If or when someone does hit this
> > (FreeBSD+PPC) as a problem, then we can work with them to find best
> > solution for fixing it. Right now we don't have anyone with the
> > PPC+FreeBSD setup, so I'd rather ignore it than implement more
> > complicated solutions that we can't test "just in case".
>
> +1, FreeBSD + PPC is not worth the complication
>
>
Thanks for the feedback, makes sense to me. I think v6 has all the features we want then - linux, windows and freebsd using sysctl.
>> Piotr says lscpu isn't available on ppc FreeBSD. I think this leaves us with two options:
>> 1. either use sysctl with the risk that it may not return an accurate value
>> 2. not support PPC for numa detection
>>
>> What do you think?
>
> I think point #2 is more not supporting FreeBSD on PPC for numa detection
> rather than not supporting PPC generally, right? Personally, I am just in
> favour of #1 for the sake of moving forward and giving us best support with
> simplest code. If or when someone does hit this (FreeBSD+PPC) as a problem,
> then we can work with them to find best solution for fixing it. Right now
> we don't have anyone with the PPC+FreeBSD setup, so I'd rather ignore it
> than implement more complicated solutions that we can't test "just in
> case".
Fully agree here. Can't expect a user app to completely resolve all
limitations in the underlying OS.
Dave
@@ -102,6 +102,8 @@ F: meson_options.txt
F: config/
F: buildtools/chkincs/
F: buildtools/call-sphinx-build.py
+F: buildtools/get-cpu-count.py
+F: buildtools/get-numa-count.py
F: buildtools/list-dir-globs.py
F: buildtools/pkg-config/
F: buildtools/symlink-drivers-solibs.sh
new file mode 100644
@@ -0,0 +1,7 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 PANTHEON.tech s.r.o.
+
+import os
+
+print(os.cpu_count())
new file mode 100644
@@ -0,0 +1,24 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (c) 2021 PANTHEON.tech s.r.o.
+
+import ctypes
+import glob
+import os
+import subprocess
+
+if os.name == 'posix':
+ if os.path.isdir('/sys/devices/system/node'):
+ numa_nodes = glob.glob('/sys/devices/system/node/node*')
+ numa_nodes.sort()
+ print(int(os.path.basename(numa_nodes[-1])[4:]) + 1)
+ else:
+ subprocess.run(['sysctl', '-n', 'vm.ndomains'], check=False)
+
+elif os.name == 'nt':
+ libkernel32 = ctypes.windll.kernel32
+
+ numa_count = ctypes.c_ulong()
+
+ libkernel32.GetNumaHighestNodeNumber(ctypes.pointer(numa_count))
+ print(numa_count.value + 1)
@@ -16,6 +16,8 @@ echo = py3 + ['-c', 'import sys; print(*sys.argv[1:])']
list_dir_globs = py3 + files('list-dir-globs.py')
map_to_win_cmd = py3 + files('map_to_win.py')
sphinx_wrapper = py3 + files('call-sphinx-build.py')
+get_cpu_count_cmd = py3 + files('get-cpu-count.py')
+get_numa_count_cmd = py3 + files('get-numa-count.py')
# select library and object file format
pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
@@ -286,8 +286,6 @@ foreach arg: warning_flags
endforeach
# set other values pulled from the build options
-dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
-dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))
dpdk_conf.set('RTE_MAX_ETHPORTS', get_option('max_ethports'))
dpdk_conf.set('RTE_LIBEAL_USE_HPET', get_option('use_hpet'))
dpdk_conf.set('RTE_ENABLE_TRACE_FP', get_option('enable_trace_fp'))
@@ -321,6 +319,51 @@ if meson.is_cross_build()
endif
endif
+max_lcores = get_option('max_lcores')
+if max_lcores == 'detect'
+ # Discovery makes sense only for non-cross builds
+ if meson.is_cross_build()
+ error('Discovery of max_lcores is not supported for cross-compilation.')
+ endif
+ # Overwrite the default value with discovered values
+ max_lcores = run_command(get_cpu_count_cmd).stdout().to_int()
+ min_lcores = 2
+ # DPDK must be build for at least 2 cores
+ if max_lcores < min_lcores
+ message('Found less than @0@ cores, building for @0@ cores'.format(min_lcores))
+ max_lcores = min_lcores
+ else
+ message('Found @0@ cores'.format(max_lcores))
+ endif
+ dpdk_conf.set('RTE_MAX_LCORE', max_lcores)
+elif max_lcores != 'default'
+ # Overwrite the default value from arch_subdir with user input
+ dpdk_conf.set('RTE_MAX_LCORE', max_lcores.to_int())
+endif
+
+max_numa_nodes = get_option('max_numa_nodes')
+if max_numa_nodes == 'detect'
+ # Discovery makes sense only for non-cross builds
+ if meson.is_cross_build()
+ error('Discovery of max_numa_nodes not supported for cross-compilation.')
+ endif
+ # Overwrite the default value with discovered values
+ max_numa_nodes = run_command(get_numa_count_cmd).stdout().to_int()
+ message('Found @0@ numa nodes'.format(max_numa_nodes))
+ dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes)
+elif max_numa_nodes != 'default'
+ # Overwrite the default value from arch_subdir with user input
+ dpdk_conf.set('RTE_MAX_NUMA_NODES', max_numa_nodes.to_int())
+endif
+
+# check that cpu and numa count is set and error out if it's not set
+if not dpdk_conf.has('RTE_MAX_LCORE')
+ error('Number of cores not specified.')
+endif
+if not dpdk_conf.has('RTE_MAX_NUMA_NODES')
+ error('Number of numa nodes not specified.')
+endif
+
# set the install path for the drivers
dpdk_conf.set_quoted('RTE_EAL_PMD_PATH', eal_pmd_path)
@@ -57,3 +57,5 @@ else
endif
dpdk_conf.set('RTE_CACHE_LINE_SIZE', 64)
+dpdk_conf.set('RTE_MAX_LCORE', 128)
+dpdk_conf.set('RTE_MAX_NUMA_NODES', 32)
@@ -34,10 +34,10 @@ option('machine', type: 'string', value: 'auto', description:
'Alias of cpu_instruction_set.')
option('max_ethports', type: 'integer', value: 32, description:
'maximum number of Ethernet devices')
-option('max_lcores', type: 'integer', value: 128, description:
- 'maximum number of cores/threads supported by EAL')
-option('max_numa_nodes', type: 'integer', value: 32, description:
- 'maximum number of NUMA nodes supported by EAL')
+option('max_lcores', type: 'string', value: 'default', description:
+ 'Set maximum number of cores/threads supported by EAL; "default" is different per-arch, "detect" detects the number of cores on the build machine.')
+option('max_numa_nodes', type: 'string', value: 'default', description:
+ 'Set the highest NUMA node supported by EAL; "default" is different per-arch, "detect" detects the highest numa node on the build machine.')
option('platform', type: 'string', value: 'native', description:
'Platform to build, either "native", "generic" or a SoC. Please refer to the Linux build guide for more information.')
option('enable_trace_fp', type: 'boolean', value: false, description: