[2/3] power: defer lcore variable allocation

The lcore variable in this code unit is only used through
rte_power_ethdev_pmgmt_queue_*() public symbols.

Defer the unconditional lcore variable allocation in those symbols.

Fixes: 130643319579 ("power: keep per-lcore state in lcore variable")
Cc: stable@dpdk.org

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 lib/power/rte_power_pmd_mgmt.c | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)
  

On 2024-12-05 18:57, David Marchand wrote:
> The lcore variable in this code unit is only used through
> rte_power_ethdev_pmgmt_queue_*() public symbols.
> 
> Defer the unconditional lcore variable allocation in those symbols.
> 
> Fixes: 130643319579 ("power: keep per-lcore state in lcore variable")
> Cc: stable@dpdk.org
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
>   lib/power/rte_power_pmd_mgmt.c | 27 +++++++++++++++++++--------
>   1 file changed, 19 insertions(+), 8 deletions(-)
> 
> diff --git a/lib/power/rte_power_pmd_mgmt.c b/lib/power/rte_power_pmd_mgmt.c
> index a2fff3b765..29e2d438a3 100644
> --- a/lib/power/rte_power_pmd_mgmt.c
> +++ b/lib/power/rte_power_pmd_mgmt.c
> @@ -72,6 +72,19 @@ struct __rte_cache_aligned pmd_core_cfg {
>   };
>   static RTE_LCORE_VAR_HANDLE(struct pmd_core_cfg, lcore_cfgs);
>   
> +static void
> +alloc_lcore_cfgs(void)
> +{
> +	struct pmd_core_cfg *lcore_cfg;
> +	unsigned int lcore_id;
> +
> +	RTE_LCORE_VAR_ALLOC(lcore_cfgs);
> +
> +	/* initialize all tailqs */
> +	RTE_LCORE_VAR_FOREACH(lcore_id, lcore_cfg, lcore_cfgs)
> +		TAILQ_INIT(&lcore_cfg->head);
> +}
> +
>   static inline bool
>   queue_equal(const union queue *l, const union queue *r)
>   {
> @@ -517,6 +530,9 @@ rte_power_ethdev_pmgmt_queue_enable(unsigned int lcore_id, uint16_t port_id,
>   		goto end;
>   	}
>   
> +	if (lcore_cfgs == NULL)
> +		alloc_lcore_cfgs();
> +


I would wrap all RTE_LCORE_VAR_LCORE() and RTE_LCORE_VAR().

static struct pmd_core_cfg *
get_cfg_lcore(unsigned int lcore_id)
{
	assure_lcore_cfgs_alloced();
	return RTE_LCORE_VAR_LCORE(lcore_cfgs, lcore_id);
}

static struct pmd_core_cfg *
get_cfg(void)
{
	get_cfg_lcore(rte_lcore_id());
}

Add

static void
assure_lcore_cfgs_alloced(unsigned int lcore_id)
{
	if (lcore_cfgs != NULL)
		lcore_cfgs_alloc();
}

..or maybe better merge assure_lcore_cfgs_alloced() and lcore_cfgs_alloc().

Makes it a little harder to make mistakes.

A somewhat unrelated question: why is pmd_core_cfg cache-line aligned? I 
don't think it should be.

>   	lcore_cfg = RTE_LCORE_VAR_LCORE(lcore_id, lcore_cfgs);
>   
>   	/* check if other queues are stopped as well */
> @@ -617,6 +633,9 @@ rte_power_ethdev_pmgmt_queue_disable(unsigned int lcore_id,
>   		return ret < 0 ? -EINVAL : -EBUSY;
>   	}
>   
> +	if (lcore_cfgs == NULL)
> +		alloc_lcore_cfgs();
> +
>   	/* no need to check queue id as wrong queue id would not be enabled */
>   	lcore_cfg = RTE_LCORE_VAR_LCORE(lcore_id, lcore_cfgs);
>   
> @@ -768,16 +787,8 @@ rte_power_pmd_mgmt_get_scaling_freq_max(unsigned int lcore)
>   }
>   
>   RTE_INIT(rte_power_ethdev_pmgmt_init) {
> -	unsigned int lcore_id;
> -	struct pmd_core_cfg *lcore_cfg;
>   	int i;
>   
> -	RTE_LCORE_VAR_ALLOC(lcore_cfgs);
> -
> -	/* initialize all tailqs */
> -	RTE_LCORE_VAR_FOREACH(lcore_id, lcore_cfg, lcore_cfgs)
> -		TAILQ_INIT(&lcore_cfg->head);
> -
>   	/* initialize config defaults */
>   	emptypoll_max = 512;
>   	pause_duration = 1;
  

On Fri, Dec 6, 2024 at 12:29 PM Mattias Rönnblom <hofors@lysator.liu.se> wrote:
> I would wrap all RTE_LCORE_VAR_LCORE() and RTE_LCORE_VAR().
>
> static struct pmd_core_cfg *
> get_cfg_lcore(unsigned int lcore_id)
> {
>         assure_lcore_cfgs_alloced();
>         return RTE_LCORE_VAR_LCORE(lcore_cfgs, lcore_id);
> }
>
> static struct pmd_core_cfg *
> get_cfg(void)
> {
>         get_cfg_lcore(rte_lcore_id());
> }
>
> Add
>
> static void
> assure_lcore_cfgs_alloced(unsigned int lcore_id)
> {
>         if (lcore_cfgs != NULL)

==

>                 lcore_cfgs_alloc();
> }
>
> ..or maybe better merge assure_lcore_cfgs_alloced() and lcore_cfgs_alloc().
>
> Makes it a little harder to make mistakes.

clb_multiwait, clb_pause and clb_scale_freq callbacks can only be
reached after a successful call to
rte_power_ethdev_pmgmt_queue_enable.
Triggering an allocation in them means we are hiding a (internal)
programatic error as allocation and use of a lcore variable are
clearly separated atm.
If we keep the lcore var api as is, I would add an assert() (maybe
under a debug build option) in RTE_LCORE_VAR macros themselves, as
calling with a NULL handle means the initialisation path in some
code/RTE_LCORE_VAR API use was incorrect.


Or because you propose to add the same type of helpers in both this
patch and the next, I am considering the other way: hide the
allocation in the RTE_LCORE_VAR* macros.
Checking for already allocated var in RTE_LCORE_VAR_ALLOC seems fine.
But the "fast path" RTE_LCORE_VAR would have an unwanted branch in most cases.


> A somewhat unrelated question: why is pmd_core_cfg cache-line aligned? I
> don't think it should be.

Before the conversion to per lcore variable, it was probably useful
(avoiding false sharing).
With the conversion, indeed, it looks like a waste of space.
It seems worth a separate fix.
  

On 2024-12-12 08:57, David Marchand wrote:
> On Fri, Dec 6, 2024 at 12:29 PM Mattias Rönnblom <hofors@lysator.liu.se> wrote:
>> I would wrap all RTE_LCORE_VAR_LCORE() and RTE_LCORE_VAR().
>>
>> static struct pmd_core_cfg *
>> get_cfg_lcore(unsigned int lcore_id)
>> {
>>          assure_lcore_cfgs_alloced();
>>          return RTE_LCORE_VAR_LCORE(lcore_cfgs, lcore_id);
>> }
>>
>> static struct pmd_core_cfg *
>> get_cfg(void)
>> {
>>          get_cfg_lcore(rte_lcore_id());
>> }
>>
>> Add
>>
>> static void
>> assure_lcore_cfgs_alloced(unsigned int lcore_id)
>> {
>>          if (lcore_cfgs != NULL)
> 
> ==
> 

Oops.

>>                  lcore_cfgs_alloc();
>> }
>>
>> ..or maybe better merge assure_lcore_cfgs_alloced() and lcore_cfgs_alloc().
>>
>> Makes it a little harder to make mistakes.
> 
> clb_multiwait, clb_pause and clb_scale_freq callbacks can only be
> reached after a successful call to
> rte_power_ethdev_pmgmt_queue_enable.
> Triggering an allocation in them means we are hiding a (internal)
> programatic error as allocation and use of a lcore variable are
> clearly separated atm.
> If we keep the lcore var api as is, I would add an assert() (maybe
> under a debug build option) in RTE_LCORE_VAR macros themselves, as
> calling with a NULL handle means the initialisation path in some
> code/RTE_LCORE_VAR API use was incorrect.
> 

Sure, that would make sense. RTE_ASSERT(), that is. RTE_VERIFY() would 
be too expensive.

> 
> Or because you propose to add the same type of helpers in both this
> patch and the next, I am considering the other way: hide the
> allocation in the RTE_LCORE_VAR* macros.
> Checking for already allocated var in RTE_LCORE_VAR_ALLOC seems fine.
> But the "fast path" RTE_LCORE_VAR would have an unwanted branch in most cases.
> 

I would prefer to have the ALLOC() macro with semantics most people 
expect from a macro (or function) with that name, which is, I would 
argue, an unconditional allocation.

It would make sense to have another macro, which performs an allocation 
only if the handle is NULL.

RTE_LCORE_VAR_ASSURE_ALLOCATED(), or just RTE_LCORE_VAR_ASSURE() 
(although the latter sounds a little like an assertion, and not an 
allocation).

RTE_LCORE_VAR_LAZY_ALLOC()

I don't know. Something like that.

> 
>> A somewhat unrelated question: why is pmd_core_cfg cache-line aligned? I
>> don't think it should be.
> 
> Before the conversion to per lcore variable, it was probably useful
> (avoiding false sharing).
> With the conversion, indeed, it looks like a waste of space.
> It seems worth a separate fix.
> 
> 

You will include it, or should I submit a separate patch?
  

On Fri, Dec 13, 2024 at 7:58 AM Mattias Rönnblom <hofors@lysator.liu.se> wrote:
> On 2024-12-12 08:57, David Marchand wrote:
> > clb_multiwait, clb_pause and clb_scale_freq callbacks can only be
> > reached after a successful call to
> > rte_power_ethdev_pmgmt_queue_enable.
> > Triggering an allocation in them means we are hiding a (internal)
> > programatic error as allocation and use of a lcore variable are
> > clearly separated atm.
> > If we keep the lcore var api as is, I would add an assert() (maybe
> > under a debug build option) in RTE_LCORE_VAR macros themselves, as
> > calling with a NULL handle means the initialisation path in some
> > code/RTE_LCORE_VAR API use was incorrect.
> >
>
> Sure, that would make sense. RTE_ASSERT(), that is. RTE_VERIFY() would
> be too expensive.

Yes, I'll send in next revision.


>
> >
> > Or because you propose to add the same type of helpers in both this
> > patch and the next, I am considering the other way: hide the
> > allocation in the RTE_LCORE_VAR* macros.
> > Checking for already allocated var in RTE_LCORE_VAR_ALLOC seems fine.
> > But the "fast path" RTE_LCORE_VAR would have an unwanted branch in most cases.
> >
>
> I would prefer to have the ALLOC() macro with semantics most people
> expect from a macro (or function) with that name, which is, I would
> argue, an unconditional allocation.
> It would make sense to have another macro, which performs an allocation
> only if the handle is NULL.
>
> RTE_LCORE_VAR_ASSURE_ALLOCATED(), or just RTE_LCORE_VAR_ASSURE()
> (although the latter sounds a little like an assertion, and not an
> allocation).
>
> RTE_LCORE_VAR_LAZY_ALLOC()
>
> I don't know. Something like that.

- In the power libary case, allocating the lcore variable is followed
by the initialisation of the lcore variable internals (per lcore
tailqs).
For this patch, I will rename the alloc_lcore_cfgs helper I had in v1 as:

static void
+init_lcore_cfgs(void)
+{
+    struct pmd_core_cfg *lcore_cfg;
+    unsigned int lcore_id;
+
+    if (lcore_cfgs != NULL)
+        return;
+
+    RTE_LCORE_VAR_ALLOC(lcore_cfgs);
+
+    /* initialize all tailqs */
+    RTE_LCORE_VAR_FOREACH(lcore_id, lcore_cfg, lcore_cfgs)
+        TAILQ_INIT(&lcore_cfg->head);
+}

and only keep those checks in the public symbols.


- About more macros, I am wondering if this is needed in the end.
Adding assertions in the lcore var accessor should catch incorrect
initialisation path.


> >
> >> A somewhat unrelated question: why is pmd_core_cfg cache-line aligned? I
> >> don't think it should be.
> >
> > Before the conversion to per lcore variable, it was probably useful
> > (avoiding false sharing).
> > With the conversion, indeed, it looks like a waste of space.
> > It seems worth a separate fix.
> >
> >
>
> You will include it, or should I submit a separate patch?

I'll send it in next revision.