[v1,1/6] build: increase default of max lcores to 512

  Modern processors are coming with an ever increasing number of cores,
and 128 does not seem like a sensible max limit any more, especially
when you consider multi-socket systems with Hyper-Threading enabled.

This patch increases max_lcores default from 128 to 512.

Signed-off-by: David Hunt <david.hunt@intel.com>
---
 meson_options.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
  

On Thu, Sep 09, 2021 at 02:45:06PM +0100, David Hunt wrote:
> Modern processors are coming with an ever increasing number of cores,
> and 128 does not seem like a sensible max limit any more, especially
> when you consider multi-socket systems with Hyper-Threading enabled.
> 
> This patch increases max_lcores default from 128 to 512.
> 
> Signed-off-by: David Hunt <david.hunt@intel.com>
> ---
Thanks Dave, I think this is needed for future-proofing. Question is,
though, is 512 enough, or should we push it higher to 768 or 1k even?
  

On Thu, Sep 9, 2021 at 4:38 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Thu, Sep 09, 2021 at 02:45:06PM +0100, David Hunt wrote:
> > Modern processors are coming with an ever increasing number of cores,
> > and 128 does not seem like a sensible max limit any more, especially
> > when you consider multi-socket systems with Hyper-Threading enabled.
> >
> > This patch increases max_lcores default from 128 to 512.
> >
> > Signed-off-by: David Hunt <david.hunt@intel.com>

Why should we need this?

--lcores makes it possible to pin 128 lcores to any physical core on
your system.
And for applications that have their own thread management, they can
pin thread, then use rte_thread_register.

Do you have applications that require more than 128 lcores?
  

On Fri, Sep 10, 2021 at 08:51:04AM +0200, David Marchand wrote:
> On Thu, Sep 9, 2021 at 4:38 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Thu, Sep 09, 2021 at 02:45:06PM +0100, David Hunt wrote:
> > > Modern processors are coming with an ever increasing number of cores,
> > > and 128 does not seem like a sensible max limit any more, especially
> > > when you consider multi-socket systems with Hyper-Threading enabled.
> > >
> > > This patch increases max_lcores default from 128 to 512.
> > >
> > > Signed-off-by: David Hunt <david.hunt@intel.com>
> 
> Why should we need this?
> 
> --lcores makes it possible to pin 128 lcores to any physical core on
> your system.
> And for applications that have their own thread management, they can
> pin thread, then use rte_thread_register.
> 
> Do you have applications that require more than 128 lcores?
>
The trouble is that using the --lcores syntax for mapping high core numbers
to low lcore ids is much more awkward to use. Every case of DPDK use I've
seen uses -c with a coremask, or -l with just giving a few core numbers on
it. This simple scheme won't work with core numbers greater than 128, and
there are already systems available with more than that number of cores.

Apart from the memory footprint issues - which this patch is already making
a good start in addressing, why would we not increase the default
max_lcores to that seen on real systems?

/Bruce
  

On Fri, Sep 10, 2021 at 9:54 AM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Fri, Sep 10, 2021 at 08:51:04AM +0200, David Marchand wrote:
> > On Thu, Sep 9, 2021 at 4:38 PM Bruce Richardson
> > <bruce.richardson@intel.com> wrote:
> > >
> > > On Thu, Sep 09, 2021 at 02:45:06PM +0100, David Hunt wrote:
> > > > Modern processors are coming with an ever increasing number of cores,
> > > > and 128 does not seem like a sensible max limit any more, especially
> > > > when you consider multi-socket systems with Hyper-Threading enabled.
> > > >
> > > > This patch increases max_lcores default from 128 to 512.
> > > >
> > > > Signed-off-by: David Hunt <david.hunt@intel.com>
> >
> > Why should we need this?
> >
> > --lcores makes it possible to pin 128 lcores to any physical core on
> > your system.
> > And for applications that have their own thread management, they can
> > pin thread, then use rte_thread_register.
> >
> > Do you have applications that require more than 128 lcores?
> >
> The trouble is that using the --lcores syntax for mapping high core numbers
> to low lcore ids is much more awkward to use. Every case of DPDK use I've
> seen uses -c with a coremask, or -l with just giving a few core numbers on
> it. This simple scheme won't work with core numbers greater than 128, and
> there are already systems available with more than that number of cores.
>
> Apart from the memory footprint issues - which this patch is already making
> a good start in addressing, why would we not increase the default
> max_lcores to that seen on real systems?

The memory footprint is a major issue to me, and reserving all those
lcores won't be needed in any system.
We will also have to decide on a "640k ought to be enough" value to
avoid ABI issue with the next processor that comes out and has more
than 512 cores.

Could we wire the -c / -l options to --lcores behavior ?
It breaks the 1:1 lcore/physical core assumption, but it solves your
usability issue.
  

10/09/2021 10:06, David Marchand:
> On Fri, Sep 10, 2021 at 9:54 AM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Fri, Sep 10, 2021 at 08:51:04AM +0200, David Marchand wrote:
> > > On Thu, Sep 9, 2021 at 4:38 PM Bruce Richardson
> > > <bruce.richardson@intel.com> wrote:
> > > >
> > > > On Thu, Sep 09, 2021 at 02:45:06PM +0100, David Hunt wrote:
> > > > > Modern processors are coming with an ever increasing number of cores,
> > > > > and 128 does not seem like a sensible max limit any more, especially
> > > > > when you consider multi-socket systems with Hyper-Threading enabled.
> > > > >
> > > > > This patch increases max_lcores default from 128 to 512.
> > > > >
> > > > > Signed-off-by: David Hunt <david.hunt@intel.com>
> > >
> > > Why should we need this?
> > >
> > > --lcores makes it possible to pin 128 lcores to any physical core on
> > > your system.
> > > And for applications that have their own thread management, they can
> > > pin thread, then use rte_thread_register.
> > >
> > > Do you have applications that require more than 128 lcores?
> > >
> > The trouble is that using the --lcores syntax for mapping high core numbers
> > to low lcore ids is much more awkward to use. Every case of DPDK use I've
> > seen uses -c with a coremask, or -l with just giving a few core numbers on
> > it. This simple scheme won't work with core numbers greater than 128, and
> > there are already systems available with more than that number of cores.
> >
> > Apart from the memory footprint issues - which this patch is already making
> > a good start in addressing, why would we not increase the default
> > max_lcores to that seen on real systems?
> 
> The memory footprint is a major issue to me, and reserving all those
> lcores won't be needed in any system.
> We will also have to decide on a "640k ought to be enough" value to
> avoid ABI issue with the next processor that comes out and has more
> than 512 cores.
> 
> Could we wire the -c / -l options to --lcores behavior ?
> It breaks the 1:1 lcore/physical core assumption, but it solves your
> usability issue.

Why would we change existing options while we already have an option
(--lcores) which solves the issue above?
I think the only issue is to educate users.
Is there something to improve in the documentation?
  

On 10/9/2021 9:24 AM, Thomas Monjalon wrote:
> 10/09/2021 10:06, David Marchand:
>> On Fri, Sep 10, 2021 at 9:54 AM Bruce Richardson
>> <bruce.richardson@intel.com> wrote:
>>> On Fri, Sep 10, 2021 at 08:51:04AM +0200, David Marchand wrote:
>>>> On Thu, Sep 9, 2021 at 4:38 PM Bruce Richardson
>>>> <bruce.richardson@intel.com> wrote:
>>>>> On Thu, Sep 09, 2021 at 02:45:06PM +0100, David Hunt wrote:
>>>>>> Modern processors are coming with an ever increasing number of cores,
>>>>>> and 128 does not seem like a sensible max limit any more, especially
>>>>>> when you consider multi-socket systems with Hyper-Threading enabled.
>>>>>>
>>>>>> This patch increases max_lcores default from 128 to 512.
>>>>>>
>>>>>> Signed-off-by: David Hunt <david.hunt@intel.com>
>>>> Why should we need this?
>>>>
>>>> --lcores makes it possible to pin 128 lcores to any physical core on
>>>> your system.
>>>> And for applications that have their own thread management, they can
>>>> pin thread, then use rte_thread_register.
>>>>
>>>> Do you have applications that require more than 128 lcores?
>>>>
>>> The trouble is that using the --lcores syntax for mapping high core numbers
>>> to low lcore ids is much more awkward to use. Every case of DPDK use I've
>>> seen uses -c with a coremask, or -l with just giving a few core numbers on
>>> it. This simple scheme won't work with core numbers greater than 128, and
>>> there are already systems available with more than that number of cores.
>>>
>>> Apart from the memory footprint issues - which this patch is already making
>>> a good start in addressing, why would we not increase the default
>>> max_lcores to that seen on real systems?
>> The memory footprint is a major issue to me, and reserving all those
>> lcores won't be needed in any system.
>> We will also have to decide on a "640k ought to be enough" value to
>> avoid ABI issue with the next processor that comes out and has more
>> than 512 cores.
>>
>> Could we wire the -c / -l options to --lcores behavior ?
>> It breaks the 1:1 lcore/physical core assumption, but it solves your
>> usability issue.
> Why would we change existing options while we already have an option
> (--lcores) which solves the issue above?
> I think the only issue is to educate users.
> Is there something to improve in the documentation?
>

Hi all,
I agree that it’s a good idea to switch to using the “--lcrores” option 
for cores above the default, that’s already future proofed.
However, I’m still a little concerned about usability, if our users are 
accustomed to the “-c” and “-l” options, I suggest that we add a warning 
to suggest using the “--lcores” option if any of the cores provided on 
the command line are above RTE_MAX_LCORE. That would help them with the 
solution to using physical cores above 128 (or whatever the compiled 
default is).

Example:

“ERROR: logical core 212 is above the maximum lcore number permitted.
Please use the --lcores option to map lcores onto physical cores, e.g. 
--lcores="(0-3)@(212-215).”

I’ll replace the first patch in the set with a patch that adds the 
additional information in the error message.

Thanks,
Dave.
  

On Tue, Sep 14, 2021 at 11:34 AM David Hunt <david.hunt@intel.com> wrote:
>
>
> On 10/9/2021 9:24 AM, Thomas Monjalon wrote:
> > 10/09/2021 10:06, David Marchand:
> >> On Fri, Sep 10, 2021 at 9:54 AM Bruce Richardson
> >> <bruce.richardson@intel.com> wrote:
> >>> On Fri, Sep 10, 2021 at 08:51:04AM +0200, David Marchand wrote:
> >>>> On Thu, Sep 9, 2021 at 4:38 PM Bruce Richardson
> >>>> <bruce.richardson@intel.com> wrote:
> >>>>> On Thu, Sep 09, 2021 at 02:45:06PM +0100, David Hunt wrote:
> >>>>>> Modern processors are coming with an ever increasing number of cores,
> >>>>>> and 128 does not seem like a sensible max limit any more, especially
> >>>>>> when you consider multi-socket systems with Hyper-Threading enabled.
> >>>>>>
> >>>>>> This patch increases max_lcores default from 128 to 512.
> >>>>>>
> >>>>>> Signed-off-by: David Hunt <david.hunt@intel.com>
> >>>> Why should we need this?
> >>>>
> >>>> --lcores makes it possible to pin 128 lcores to any physical core on
> >>>> your system.
> >>>> And for applications that have their own thread management, they can
> >>>> pin thread, then use rte_thread_register.
> >>>>
> >>>> Do you have applications that require more than 128 lcores?
> >>>>
> >>> The trouble is that using the --lcores syntax for mapping high core numbers
> >>> to low lcore ids is much more awkward to use. Every case of DPDK use I've
> >>> seen uses -c with a coremask, or -l with just giving a few core numbers on
> >>> it. This simple scheme won't work with core numbers greater than 128, and
> >>> there are already systems available with more than that number of cores.
> >>>
> >>> Apart from the memory footprint issues - which this patch is already making
> >>> a good start in addressing, why would we not increase the default
> >>> max_lcores to that seen on real systems?
> >> The memory footprint is a major issue to me, and reserving all those
> >> lcores won't be needed in any system.
> >> We will also have to decide on a "640k ought to be enough" value to
> >> avoid ABI issue with the next processor that comes out and has more
> >> than 512 cores.
> >>
> >> Could we wire the -c / -l options to --lcores behavior ?
> >> It breaks the 1:1 lcore/physical core assumption, but it solves your
> >> usability issue.
> > Why would we change existing options while we already have an option
> > (--lcores) which solves the issue above?
> > I think the only issue is to educate users.
> > Is there something to improve in the documentation?
> >
>
> Hi all,
> I agree that it’s a good idea to switch to using the “--lcrores” option

Let's avoid typo in the error message you'll add :-).


> for cores above the default, that’s already future proofed.
> However, I’m still a little concerned about usability, if our users are
> accustomed to the “-c” and “-l” options, I suggest that we add a warning
> to suggest using the “--lcores” option if any of the cores provided on
> the command line are above RTE_MAX_LCORE. That would help them with the
> solution to using physical cores above 128 (or whatever the compiled
> default is).
>
> Example:
>
> “ERROR: logical core 212 is above the maximum lcore number permitted.
> Please use the --lcores option to map lcores onto physical cores, e.g.
> --lcores="(0-3)@(212-215).”

If you could directly provide the right --lcores syntax based on what
user provided with -c or -l, it would be even better.
This should be not that difficult.


>
> I’ll replace the first patch in the set with a patch that adds the
> additional information in the error message.
  

On 14/9/2021 11:00 AM, David Marchand wrote:
> On Tue, Sep 14, 2021 at 11:34 AM David Hunt <david.hunt@intel.com> wrote:
>>
>> On 10/9/2021 9:24 AM, Thomas Monjalon wrote:
>>> 10/09/2021 10:06, David Marchand:
>>>> On Fri, Sep 10, 2021 at 9:54 AM Bruce Richardson
>>>> <bruce.richardson@intel.com> wrote:
>>>>> On Fri, Sep 10, 2021 at 08:51:04AM +0200, David Marchand wrote:
>>>>>> On Thu, Sep 9, 2021 at 4:38 PM Bruce Richardson
>>>>>> <bruce.richardson@intel.com> wrote:
>>>>>>> On Thu, Sep 09, 2021 at 02:45:06PM +0100, David Hunt wrote:
>>>>>>>> Modern processors are coming with an ever increasing number of cores,
>>>>>>>> and 128 does not seem like a sensible max limit any more, especially
>>>>>>>> when you consider multi-socket systems with Hyper-Threading enabled.
>>>>>>>>
>>>>>>>> This patch increases max_lcores default from 128 to 512.
>>>>>>>>
>>>>>>>> Signed-off-by: David Hunt <david.hunt@intel.com>
>>>>>> Why should we need this?
>>>>>>
>>>>>> --lcores makes it possible to pin 128 lcores to any physical core on
>>>>>> your system.
>>>>>> And for applications that have their own thread management, they can
>>>>>> pin thread, then use rte_thread_register.
>>>>>>
>>>>>> Do you have applications that require more than 128 lcores?
>>>>>>
>>>>> The trouble is that using the --lcores syntax for mapping high core numbers
>>>>> to low lcore ids is much more awkward to use. Every case of DPDK use I've
>>>>> seen uses -c with a coremask, or -l with just giving a few core numbers on
>>>>> it. This simple scheme won't work with core numbers greater than 128, and
>>>>> there are already systems available with more than that number of cores.
>>>>>
>>>>> Apart from the memory footprint issues - which this patch is already making
>>>>> a good start in addressing, why would we not increase the default
>>>>> max_lcores to that seen on real systems?
>>>> The memory footprint is a major issue to me, and reserving all those
>>>> lcores won't be needed in any system.
>>>> We will also have to decide on a "640k ought to be enough" value to
>>>> avoid ABI issue with the next processor that comes out and has more
>>>> than 512 cores.
>>>>
>>>> Could we wire the -c / -l options to --lcores behavior ?
>>>> It breaks the 1:1 lcore/physical core assumption, but it solves your
>>>> usability issue.
>>> Why would we change existing options while we already have an option
>>> (--lcores) which solves the issue above?
>>> I think the only issue is to educate users.
>>> Is there something to improve in the documentation?
>>>
>> Hi all,
>> I agree that it’s a good idea to switch to using the “--lcrores” option
> Let's avoid typo in the error message you'll add :-).
>
>
>> for cores above the default, that’s already future proofed.
>> However, I’m still a little concerned about usability, if our users are
>> accustomed to the “-c” and “-l” options, I suggest that we add a warning
>> to suggest using the “--lcores” option if any of the cores provided on
>> the command line are above RTE_MAX_LCORE. That would help them with the
>> solution to using physical cores above 128 (or whatever the compiled
>> default is).
>>
>> Example:
>>
>> “ERROR: logical core 212 is above the maximum lcore number permitted.
>> Please use the --lcores option to map lcores onto physical cores, e.g.
>> --lcores="(0-3)@(212-215).”
> If you could directly provide the right --lcores syntax based on what
> user provided with -c or -l, it would be even better.
> This should be not that difficult.


Agreed. I now have something working that when given "-l 12-16,130,132", 
will output the following:

EAL: One of the 7 cores provided exceeds RTE_MAX_LCORE (128)
EAL: Please use --lcores instead, e.g. --lcores "(0-6)@(12-16,130,132)"

So you can just cut-and-paste that option into your command line. Makes 
it very easy for users to migrate.


>
>> I’ll replace the first patch in the set with a patch that adds the
>> additional information in the error message.
>
>
  

On Tue, Sep 14, 2021 at 1:07 PM David Hunt <david.hunt@intel.com> wrote:
> >> “ERROR: logical core 212 is above the maximum lcore number permitted.
> >> Please use the --lcores option to map lcores onto physical cores, e.g.
> >> --lcores="(0-3)@(212-215).”
> > If you could directly provide the right --lcores syntax based on what
> > user provided with -c or -l, it would be even better.
> > This should be not that difficult.
>
>
> Agreed. I now have something working that when given "-l 12-16,130,132",
> will output the following:
>
> EAL: One of the 7 cores provided exceeds RTE_MAX_LCORE (128)
> EAL: Please use --lcores instead, e.g. --lcores "(0-6)@(12-16,130,132)"

That's not equivalent.

(0-6)@(12-16,130,132) means 7 lcores with each lcore running on the
same group of physical cores.
-l 12-16,130,132 means 7 lcores running on dedicated physical cores.
I would expect 0@12,1@13,2@14,3@15,4@16,5@130,6@132


You can see with debug logs:

$ echo quit | ./build/app/dpdk-testpmd --log-level=*:debug --no-huge
-m 512 --lcores '(0-2)@(0-2)' -- --total-num-mbufs 2048 |& grep
lcore.*is.ready
EAL: Main lcore 0 is ready (tid=7feb9550bc00;cpuset=[0,1,2])
EAL: lcore 1 is ready (tid=7feb909ce700;cpuset=[0,1,2])
EAL: lcore 2 is ready (tid=7feb901cd700;cpuset=[0,1,2])

vs

$ echo quit | ./build/app/dpdk-testpmd --log-level=*:debug --no-huge
-m 512 --lcores 0@0,1@1,2@2 -- --total-num-mbufs 2048 |& grep
lcore.*is.ready
EAL: Main lcore 0 is ready (tid=7fba1cd1ac00;cpuset=[0])
EAL: lcore 2 is ready (tid=7fba179dc700;cpuset=[2])
EAL: lcore 1 is ready (tid=7fba181dd700;cpuset=[1])
  

On 14/9/2021 12:29 PM, David Marchand wrote:
> On Tue, Sep 14, 2021 at 1:07 PM David Hunt <david.hunt@intel.com> wrote:
>>>> “ERROR: logical core 212 is above the maximum lcore number permitted.
>>>> Please use the --lcores option to map lcores onto physical cores, e.g.
>>>> --lcores="(0-3)@(212-215).”
>>> If you could directly provide the right --lcores syntax based on what
>>> user provided with -c or -l, it would be even better.
>>> This should be not that difficult.
>>
>> Agreed. I now have something working that when given "-l 12-16,130,132",
>> will output the following:
>>
>> EAL: One of the 7 cores provided exceeds RTE_MAX_LCORE (128)
>> EAL: Please use --lcores instead, e.g. --lcores "(0-6)@(12-16,130,132)"
> That's not equivalent.
>
> (0-6)@(12-16,130,132) means 7 lcores with each lcore running on the
> same group of physical cores.
> -l 12-16,130,132 means 7 lcores running on dedicated physical cores.
> I would expect 0@12,1@13,2@14,3@15,4@16,5@130,6@132
>
>
> You can see with debug logs:
>
> $ echo quit | ./build/app/dpdk-testpmd --log-level=*:debug --no-huge
> -m 512 --lcores '(0-2)@(0-2)' -- --total-num-mbufs 2048 |& grep
> lcore.*is.ready
> EAL: Main lcore 0 is ready (tid=7feb9550bc00;cpuset=[0,1,2])
> EAL: lcore 1 is ready (tid=7feb909ce700;cpuset=[0,1,2])
> EAL: lcore 2 is ready (tid=7feb901cd700;cpuset=[0,1,2])
>
> vs
>
> $ echo quit | ./build/app/dpdk-testpmd --log-level=*:debug --no-huge
> -m 512 --lcores 0@0,1@1,2@2 -- --total-num-mbufs 2048 |& grep
> lcore.*is.ready
> EAL: Main lcore 0 is ready (tid=7fba1cd1ac00;cpuset=[0])
> EAL: lcore 2 is ready (tid=7fba179dc700;cpuset=[2])
> EAL: lcore 1 is ready (tid=7fba181dd700;cpuset=[1])
>

Hi David,

    Thanks for the clarification. I've made the relevant changes and 
submitted a v2. Hopefully the suggested parameters are correct this time! :)

Regards,
Dave.
  

My microphone was not working during the Techboard meeting, so here goes instead:

Please consider if the default of max 128 assigned lcores suffices for the expected lifetime of the 21.11 LTS, or if the default should be bumped to 512 as suggested.

@David Hunt, do you have idea about the memory cost of increasing from 128 to 512? This seems to be the primary objection.

-Morten
  

On 17/11/2021 3:55 PM, Morten Brørup wrote:
> My microphone was not working during the Techboard meeting, so here goes instead:
>
> Please consider if the default of max 128 assigned lcores suffices for the expected lifetime of the 21.11 LTS, or if the default should be bumped to 512 as suggested.
>
> @David Hunt, do you have idea about the memory cost of increasing from 128 to 512? This seems to be the primary objection.
>
> -Morten


Hi Morten,

I seem to recall that a simple increase from 128 to 512 cores would 
result in an increased static memory footprint of 2.5MBs per 
application, but there were additional patches in the opriginal patch 
set that switch from static allocation to dynamic allocation reducing 
this to a few hundred kilobytes. This was not deemed the best direction, 
so we went with the improvement of informing the user more clearly on 
how to use the command line syntax to use the physical core numbers 
above 128 as lcores in your application.

I believe the thinking is that 128 lcores is plenty for a single 
application, and if there is a case where more than that is needed (for 
a single application), then the default can be over-ridden by changing 
the max_lcores setting and recompiling.

Rgds,
Dave.