vdpa/mlx5: use master core Id for vhost thread
Checks
Commit Message
With latest rte api, vdpa example failed on vq setup, the api to get
event queue of specified core failed.
The api to get event queue needs core id as input, but after
commit 67ae5936c4fc36 ("eal: fix lcore accessors for non-EAL threads"),
code of "vhost-event" thread to get current core returns -1, an invalid
core id.
As vhost thread created on same core of eal master core, this patch uses
master core ID as a workaround.
Signed-off-by: Xueming Li <xuemingl@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Comments
On Tue, Jul 28, 2020 at 8:11 AM Xueming Li <xuemingl@mellanox.com> wrote:
>
> With latest rte api, vdpa example failed on vq setup, the api to get
> event queue of specified core failed.
>
> The api to get event queue needs core id as input, but after
> commit 67ae5936c4fc36 ("eal: fix lcore accessors for non-EAL threads"),
> code of "vhost-event" thread to get current core returns -1, an invalid
> core id.
>
> As vhost thread created on same core of eal master core, this patch uses
> master core ID as a workaround.
rte_lcore_to_cpu_id(-1) is supposed to return the current lcore id.
As explained in the commitlog of 67ae5936c4fc36, for a non-EAL thread
(which has no associated lcore by default) calling this function means
lcore_config[-1].core_id.
Did I miss something?
If not, this is not a workaround but a fix.
>
> Signed-off-by: Xueming Li <xuemingl@mellanox.com>
> Acked-by: Matan Azrad <matan@mellanox.com>
> ---
> drivers/vdpa/mlx5/mlx5_vdpa_event.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> index 13ad43611e..ff6db8e345 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> @@ -55,7 +55,7 @@ mlx5_vdpa_event_qp_global_prepare(struct mlx5_vdpa_priv *priv)
>
> if (priv->eventc)
> return 0;
> - lcore = (uint32_t)rte_lcore_to_cpu_id(-1);
> + lcore = (uint32_t)rte_get_master_lcore();
> if (mlx5_glue->devx_query_eqn(priv->ctx, lcore, &priv->eqn)) {
> rte_errno = errno;
> DRV_LOG(ERR, "Failed to query EQ number %d.", rte_errno);
> --
> 2.17.1
>
From: David Marchand
> On Tue, Jul 28, 2020 at 8:11 AM Xueming Li <xuemingl@mellanox.com>
> wrote:
> >
> > With latest rte api, vdpa example failed on vq setup, the api to get
> > event queue of specified core failed.
> >
> > The api to get event queue needs core id as input, but after commit
> > 67ae5936c4fc36 ("eal: fix lcore accessors for non-EAL threads"), code
> > of "vhost-event" thread to get current core returns -1, an invalid
> > core id.
> >
> > As vhost thread created on same core of eal master core, this patch
> > uses master core ID as a workaround.
>
> rte_lcore_to_cpu_id(-1) is supposed to return the current lcore id.
> As explained in the commitlog of 67ae5936c4fc36, for a non-EAL thread
> (which has no associated lcore by default) calling this function means
> lcore_config[-1].core_id.
Deosn't lcore mean core?
So, if the private thread run on a core that opened by EAL,
Shouldn't the API return the lcore of the private thread core?
>
> Did I miss something?
> If not, this is not a workaround but a fix.
I'm not sure yet.
> >
> > Signed-off-by: Xueming Li <xuemingl@mellanox.com>
> > Acked-by: Matan Azrad <matan@mellanox.com>
> > ---
> > drivers/vdpa/mlx5/mlx5_vdpa_event.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> > b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> > index 13ad43611e..ff6db8e345 100644
> > --- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> > +++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> > @@ -55,7 +55,7 @@ mlx5_vdpa_event_qp_global_prepare(struct
> > mlx5_vdpa_priv *priv)
> >
> > if (priv->eventc)
> > return 0;
> > - lcore = (uint32_t)rte_lcore_to_cpu_id(-1);
> > + lcore = (uint32_t)rte_get_master_lcore();
> > if (mlx5_glue->devx_query_eqn(priv->ctx, lcore, &priv->eqn)) {
> > rte_errno = errno;
> > DRV_LOG(ERR, "Failed to query EQ number %d.",
> > rte_errno);
> > --
> > 2.17.1
> >
>
>
> --
> David Marchand
On Tue, Jul 28, 2020 at 9:20 AM Matan Azrad <matan@mellanox.com> wrote:
>
>
>
> From: David Marchand
> > On Tue, Jul 28, 2020 at 8:11 AM Xueming Li <xuemingl@mellanox.com>
> > wrote:
> > >
> > > With latest rte api, vdpa example failed on vq setup, the api to get
> > > event queue of specified core failed.
> > >
> > > The api to get event queue needs core id as input, but after commit
> > > 67ae5936c4fc36 ("eal: fix lcore accessors for non-EAL threads"), code
> > > of "vhost-event" thread to get current core returns -1, an invalid
> > > core id.
> > >
> > > As vhost thread created on same core of eal master core, this patch
> > > uses master core ID as a workaround.
> >
> > rte_lcore_to_cpu_id(-1) is supposed to return the current lcore id.
> > As explained in the commitlog of 67ae5936c4fc36, for a non-EAL thread
> > (which has no associated lcore by default) calling this function means
> > lcore_config[-1].core_id.
>
> Deosn't lcore mean core?
No.
> So, if the private thread run on a core that opened by EAL,
> Shouldn't the API return the lcore of the private thread core?
vhost-events is a control thread.
Was it working just by chance and returning 0 before?
From: David Marchand
> On Tue, Jul 28, 2020 at 9:20 AM Matan Azrad <matan@mellanox.com> wrote:
> >
> >
> >
> > From: David Marchand
> > > On Tue, Jul 28, 2020 at 8:11 AM Xueming Li <xuemingl@mellanox.com>
> > > wrote:
> > > >
> > > > With latest rte api, vdpa example failed on vq setup, the api to
> > > > get event queue of specified core failed.
> > > >
> > > > The api to get event queue needs core id as input, but after
> > > > commit
> > > > 67ae5936c4fc36 ("eal: fix lcore accessors for non-EAL threads"),
> > > > code of "vhost-event" thread to get current core returns -1, an
> > > > invalid core id.
> > > >
> > > > As vhost thread created on same core of eal master core, this
> > > > patch uses master core ID as a workaround.
> > >
> > > rte_lcore_to_cpu_id(-1) is supposed to return the current lcore id.
> > > As explained in the commitlog of 67ae5936c4fc36, for a non-EAL
> > > thread (which has no associated lcore by default) calling this
> > > function means lcore_config[-1].core_id.
> >
> > Deosn't lcore mean core?
>
> No.
So, it may be that we have more than one lcore working on a specific cpu_id?
Can you extend here?
>
> > So, if the private thread run on a core that opened by EAL, Shouldn't
> > the API return the lcore of the private thread core?
>
> vhost-events is a control thread.
> Was it working just by chance and returning 0 before?
Probably yes.
> --
> David Marchand
On Tue, Jul 28, 2020 at 9:30 AM Matan Azrad <matan@mellanox.com> wrote:
> > > > rte_lcore_to_cpu_id(-1) is supposed to return the current lcore id.
> > > > As explained in the commitlog of 67ae5936c4fc36, for a non-EAL
> > > > thread (which has no associated lcore by default) calling this
> > > > function means lcore_config[-1].core_id.
> > >
> > > Deosn't lcore mean core?
> >
> > No.
>
> So, it may be that we have more than one lcore working on a specific cpu_id?
> Can you extend here?
A control thread is a non-EAL thread.
A non-EAL thread has no lcore associated (unless you call the newly
added rte_thread_register API).
Not sure why you need the physical core identifier in the driver.
It is possible that non-EAL threads run on one physical core if you
start your application with the right cpuset.
But without checking, you can't make such an assumption.
>
> >
> > > So, if the private thread run on a core that opened by EAL, Shouldn't
> > > the API return the lcore of the private thread core?
> >
> > vhost-events is a control thread.
> > Was it working just by chance and returning 0 before?
>
> Probably yes.
(gdb) info symbol &lcore_config[-1].core_id
vfio_cfgs + 444820 in section .bss of
/home/dmarchan/git/pub/dpdk.org/build/app/../lib/librte_eal.so.20.0
Please, update the commitlog.
Thanks.
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, July 28, 2020 3:53 PM
> To: Matan Azrad <matan@mellanox.com>
> Cc: Xueming(Steven) Li <xuemingl@mellanox.com>; dev <dev@dpdk.org>; Asaf
> Penso <asafp@mellanox.com>
> Subject: Re: [dpdk-dev] [PATCH] vdpa/mlx5: use master core Id for vhost
> thread
>
> On Tue, Jul 28, 2020 at 9:30 AM Matan Azrad <matan@mellanox.com> wrote:
> > > > > rte_lcore_to_cpu_id(-1) is supposed to return the current lcore id.
> > > > > As explained in the commitlog of 67ae5936c4fc36, for a non-EAL
> > > > > thread (which has no associated lcore by default) calling this
> > > > > function means lcore_config[-1].core_id.
> > > >
> > > > Deosn't lcore mean core?
> > >
> > > No.
> >
> > So, it may be that we have more than one lcore working on a specific cpu_id?
> > Can you extend here?
>
> A control thread is a non-EAL thread.
Is the default thread a control thread? After eal init, rte_lcore_to_cpu_id(-1) returns 0.
In function rte_ctrl_thread_create(), the new thread is set cpu affinity with control cpu set,
bind it same core of default control thread.
> A non-EAL thread has no lcore associated (unless you call the newly added
> rte_thread_register API).
>
> Not sure why you need the physical core identifier in the driver.
>
> It is possible that non-EAL threads run on one physical core if you start your
> application with the right cpuset.
> But without checking, you can't make such an assumption.
>
>
> >
> > >
> > > > So, if the private thread run on a core that opened by EAL,
> > > > Shouldn't the API return the lcore of the private thread core?
> > >
> > > vhost-events is a control thread.
> > > Was it working just by chance and returning 0 before?
> >
> > Probably yes.
>
> (gdb) info symbol &lcore_config[-1].core_id vfio_cfgs + 444820 in section .bss
> of
> /home/dmarchan/git/pub/dpdk.org/build/app/../lib/librte_eal.so.20.0
>
> Please, update the commitlog.
> Thanks.
>
>
> --
> David Marchand
From internal discussion, we found a misuse of the vector parameter in devx_query_eqn
Function, no cpu id is required, I'll come up with another version.
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Xueming Li
> Sent: Tuesday, July 28, 2020 2:11 PM
> Cc: dev@dpdk.org; Asaf Penso <asafp@mellanox.com>
> Subject: [dpdk-dev] [PATCH] vdpa/mlx5: use master core Id for vhost thread
>
> With latest rte api, vdpa example failed on vq setup, the api to get event queue
> of specified core failed.
>
> The api to get event queue needs core id as input, but after commit
> 67ae5936c4fc36 ("eal: fix lcore accessors for non-EAL threads"), code of "vhost-
> event" thread to get current core returns -1, an invalid core id.
>
> As vhost thread created on same core of eal master core, this patch uses
> master core ID as a workaround.
>
> Signed-off-by: Xueming Li <xuemingl@mellanox.com>
> Acked-by: Matan Azrad <matan@mellanox.com>
> ---
> drivers/vdpa/mlx5/mlx5_vdpa_event.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> index 13ad43611e..ff6db8e345 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa_event.c
> @@ -55,7 +55,7 @@ mlx5_vdpa_event_qp_global_prepare(struct
> mlx5_vdpa_priv *priv)
>
> if (priv->eventc)
> return 0;
> - lcore = (uint32_t)rte_lcore_to_cpu_id(-1);
> + lcore = (uint32_t)rte_get_master_lcore();
> if (mlx5_glue->devx_query_eqn(priv->ctx, lcore, &priv->eqn)) {
> rte_errno = errno;
> DRV_LOG(ERR, "Failed to query EQ number %d.", rte_errno);
> --
> 2.17.1
Hi David
We found that our kernel driver doesn't really need it and we didn't understand
the internal API correctly.
We will create v2 as fix to the vdpa code.
Thank you for the review!
Matan
From: Xueming(Steven) Li:
> > -----Original Message-----
> > From: David Marchand <david.marchand@redhat.com>
> > Sent: Tuesday, July 28, 2020 3:53 PM
> > To: Matan Azrad <matan@mellanox.com>
> > Cc: Xueming(Steven) Li <xuemingl@mellanox.com>; dev <dev@dpdk.org>;
> > Asaf Penso <asafp@mellanox.com>
> > Subject: Re: [dpdk-dev] [PATCH] vdpa/mlx5: use master core Id for
> > vhost thread
> >
> > On Tue, Jul 28, 2020 at 9:30 AM Matan Azrad <matan@mellanox.com>
> wrote:
> > > > > > rte_lcore_to_cpu_id(-1) is supposed to return the current lcore id.
> > > > > > As explained in the commitlog of 67ae5936c4fc36, for a non-EAL
> > > > > > thread (which has no associated lcore by default) calling this
> > > > > > function means lcore_config[-1].core_id.
> > > > >
> > > > > Deosn't lcore mean core?
> > > >
> > > > No.
> > >
> > > So, it may be that we have more than one lcore working on a specific
> cpu_id?
> > > Can you extend here?
> >
> > A control thread is a non-EAL thread.
>
> Is the default thread a control thread? After eal init, rte_lcore_to_cpu_id(-1)
> returns 0.
>
> In function rte_ctrl_thread_create(), the new thread is set cpu affinity with
> control cpu set, bind it same core of default control thread.
>
> > A non-EAL thread has no lcore associated (unless you call the newly
> > added rte_thread_register API).
> >
> > Not sure why you need the physical core identifier in the driver.
> >
> > It is possible that non-EAL threads run on one physical core if you
> > start your application with the right cpuset.
> > But without checking, you can't make such an assumption.
> >
> >
> > >
> > > >
> > > > > So, if the private thread run on a core that opened by EAL,
> > > > > Shouldn't the API return the lcore of the private thread core?
> > > >
> > > > vhost-events is a control thread.
> > > > Was it working just by chance and returning 0 before?
> > >
> > > Probably yes.
> >
> > (gdb) info symbol &lcore_config[-1].core_id vfio_cfgs + 444820 in
> > section .bss of
> > /home/dmarchan/git/pub/dpdk.org/build/app/../lib/librte_eal.so.20.0
> >
> > Please, update the commitlog.
> > Thanks.
> >
> >
> > --
> > David Marchand
On Tue, Jul 28, 2020 at 10:27 AM Matan Azrad <matan@mellanox.com> wrote:
> We found that our kernel driver doesn't really need it and we didn't understand
> the internal API correctly.
>
> We will create v2 as fix to the vdpa code.
Ok cool.
>
> Thank you for the review!
No problem, I was scared of a regression in EAL :-).
@@ -55,7 +55,7 @@ mlx5_vdpa_event_qp_global_prepare(struct mlx5_vdpa_priv *priv)
if (priv->eventc)
return 0;
- lcore = (uint32_t)rte_lcore_to_cpu_id(-1);
+ lcore = (uint32_t)rte_get_master_lcore();
if (mlx5_glue->devx_query_eqn(priv->ctx, lcore, &priv->eqn)) {
rte_errno = errno;
DRV_LOG(ERR, "Failed to query EQ number %d.", rte_errno);