mbox series

[0/7] Register external threads as lcore

Message ID 20200610144506.30505-1-david.marchand@redhat.com (mailing list archive)
Headers show
Series Register external threads as lcore | expand

Message

David Marchand June 10, 2020, 2:44 p.m. UTC
OVS and some other applications have been hacking into DPDK internals to
fake EAL threads and avoid performance penalty of only having non-EAL
threads.

This series proposes to add a new type of lcores and maps those external
threads to such lcores.
Those threads won't run the DPDK eal mainloop and as a consequence part of
the EAL threads API cannot work.

Having new lcores appearing during the process lifetime is not expected
by some DPDK components. This is addressed by notifying of such lcore
hotplug.

This patchset has still some more work (like refusing new lcore type in
incompatible EAL threads API, updating the documentation and adding unit
tests) but I am sending it anyway as I would like to get this in for
20.08.

For the interested parties, I have a patch [1] against dpdk-latest OVS
branch that makes use of this series.

1: https://github.com/david-marchand/ovs/commit/dpdk-latest-external-lcores

Comments

Jerin Jacob June 10, 2020, 3:09 p.m. UTC | #1
On Wed, Jun 10, 2020 at 8:15 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> OVS and some other applications have been hacking into DPDK internals to
> fake EAL threads and avoid performance penalty of only having non-EAL
> threads.
>
> This series proposes to add a new type of lcores and maps those external
> threads to such lcores.
> Those threads won't run the DPDK eal mainloop and as a consequence part of
> the EAL threads API cannot work.
>
> Having new lcores appearing during the process lifetime is not expected
> by some DPDK components. This is addressed by notifying of such lcore
> hotplug.
>
> This patchset has still some more work (like refusing new lcore type in
> incompatible EAL threads API, updating the documentation and adding unit
> tests) but I am sending it anyway as I would like to get this in for
> 20.08.

Cool feature.

Is mempool's lcore local cache working for external cores with this scheme?


>
> For the interested parties, I have a patch [1] against dpdk-latest OVS
> branch that makes use of this series.
>
> 1: https://github.com/david-marchand/ovs/commit/dpdk-latest-external-lcores

Usage is clear from the example code,
Looking forward to having documentation and UT for the new API
in the next version.



>
> --
> David Marchand
>
> David Marchand (7):
>   eal: relocate per thread symbols to common
>   eal: fix multiple definition of per lcore thread id
>   eal: introduce thread init helper
>   eal: introduce thread uninit helper
>   eal: register non-EAL threads as lcore
>   eal: dump lcores
>   eal: add lcore hotplug notifications
>
>  doc/guides/howto/debug_troubleshoot.rst   |   5 +-
>  lib/librte_eal/common/eal_common_lcore.c  | 167 +++++++++++++++++++++-
>  lib/librte_eal/common/eal_common_thread.c | 143 ++++++++++++++----
>  lib/librte_eal/common/eal_common_trace.c  |  49 ++++++-
>  lib/librte_eal/common/eal_private.h       |  43 ++++++
>  lib/librte_eal/common/eal_thread.h        |  26 +++-
>  lib/librte_eal/common/eal_trace.h         |   1 +
>  lib/librte_eal/freebsd/eal.c              |  16 ++-
>  lib/librte_eal/freebsd/eal_thread.c       |  38 +----
>  lib/librte_eal/include/rte_eal.h          |  12 +-
>  lib/librte_eal/include/rte_lcore.h        | 112 +++++++++++++--
>  lib/librte_eal/linux/eal.c                |  17 ++-
>  lib/librte_eal/linux/eal_thread.c         |  38 +----
>  lib/librte_eal/rte_eal_version.map        |  14 ++
>  lib/librte_eal/windows/eal.c              |   3 +-
>  lib/librte_eal/windows/eal_thread.c       |  14 +-
>  16 files changed, 536 insertions(+), 162 deletions(-)
>
> --
> 2.23.0
>
Bruce Richardson June 10, 2020, 3:13 p.m. UTC | #2
On Wed, Jun 10, 2020 at 08:39:30PM +0530, Jerin Jacob wrote:
> On Wed, Jun 10, 2020 at 8:15 PM David Marchand
> <david.marchand@redhat.com> wrote:
> >
> > OVS and some other applications have been hacking into DPDK internals to
> > fake EAL threads and avoid performance penalty of only having non-EAL
> > threads.
> >
> > This series proposes to add a new type of lcores and maps those external
> > threads to such lcores.
> > Those threads won't run the DPDK eal mainloop and as a consequence part of
> > the EAL threads API cannot work.
> >
> > Having new lcores appearing during the process lifetime is not expected
> > by some DPDK components. This is addressed by notifying of such lcore
> > hotplug.
> >
> > This patchset has still some more work (like refusing new lcore type in
> > incompatible EAL threads API, updating the documentation and adding unit
> > tests) but I am sending it anyway as I would like to get this in for
> > 20.08.
> 
> Cool feature.
> 

+1 for this, this is something that I think has been needed for a long
time. Thanks!

> Is mempool's lcore local cache working for external cores with this scheme?
> 
> 
> >
> > For the interested parties, I have a patch [1] against dpdk-latest OVS
> > branch that makes use of this series.
> >
> > 1: https://github.com/david-marchand/ovs/commit/dpdk-latest-external-lcores
> 
> Usage is clear from the example code,
> Looking forward to having documentation and UT for the new API
> in the next version.
> 
Yes, I think we need a doc update explaining usage. The EAL chapter of the
programmers guide needs an update [1], and I think a description in the
howto would be good also [2].

[1] https://doc.dpdk.org/guides-20.05/prog_guide/env_abstraction_layer.html
[2] https://doc.dpdk.org/guides-20.05/howto/index.html

Regards,
/Bruce
David Marchand June 10, 2020, 3:18 p.m. UTC | #3
On Wed, Jun 10, 2020 at 5:09 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> On Wed, Jun 10, 2020 at 8:15 PM David Marchand
> <david.marchand@redhat.com> wrote:
> >
> > OVS and some other applications have been hacking into DPDK internals to
> > fake EAL threads and avoid performance penalty of only having non-EAL
> > threads.
> >
> > This series proposes to add a new type of lcores and maps those external
> > threads to such lcores.
> > Those threads won't run the DPDK eal mainloop and as a consequence part of
> > the EAL threads API cannot work.
> >
> > Having new lcores appearing during the process lifetime is not expected
> > by some DPDK components. This is addressed by notifying of such lcore
> > hotplug.
> >
> > This patchset has still some more work (like refusing new lcore type in
> > incompatible EAL threads API, updating the documentation and adding unit
> > tests) but I am sending it anyway as I would like to get this in for
> > 20.08.
>
> Cool feature.
>
> Is mempool's lcore local cache working for external cores with this scheme?

Yes, as it is stateless, all we need is a unique lcore_id in [0,
RTE_MAX_LCORE-1] range.
We could imagine flushing such caches on unregistering.

And we can fix other mempool drivers like the bucket driver.
Jerin Jacob June 10, 2020, 3:33 p.m. UTC | #4
On Wed, Jun 10, 2020 at 8:48 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Wed, Jun 10, 2020 at 5:09 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> >
> > On Wed, Jun 10, 2020 at 8:15 PM David Marchand
> > <david.marchand@redhat.com> wrote:
> > >
> > > OVS and some other applications have been hacking into DPDK internals to
> > > fake EAL threads and avoid performance penalty of only having non-EAL
> > > threads.
> > >
> > > This series proposes to add a new type of lcores and maps those external
> > > threads to such lcores.
> > > Those threads won't run the DPDK eal mainloop and as a consequence part of
> > > the EAL threads API cannot work.
> > >
> > > Having new lcores appearing during the process lifetime is not expected
> > > by some DPDK components. This is addressed by notifying of such lcore
> > > hotplug.
> > >
> > > This patchset has still some more work (like refusing new lcore type in
> > > incompatible EAL threads API, updating the documentation and adding unit
> > > tests) but I am sending it anyway as I would like to get this in for
> > > 20.08.
> >
> > Cool feature.
> >
> > Is mempool's lcore local cache working for external cores with this scheme?
>
> Yes, as it is stateless, all we need is a unique lcore_id in [0,
> RTE_MAX_LCORE-1] range.

Was it the case when lcore registered and then mempool created? What
about other case, mempool created and then lcore registered?


> We could imagine flushing such caches on unregistering.
>
> And we can fix other mempool drivers like the bucket driver.
>
> --
> David Marchand
>
David Marchand June 15, 2020, 7:11 a.m. UTC | #5
On Wed, Jun 10, 2020 at 5:33 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> On Wed, Jun 10, 2020 at 8:48 PM David Marchand
> <david.marchand@redhat.com> wrote:
> >
> > On Wed, Jun 10, 2020 at 5:09 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
> > >
> > > On Wed, Jun 10, 2020 at 8:15 PM David Marchand
> > > <david.marchand@redhat.com> wrote:
> > > >
> > > > OVS and some other applications have been hacking into DPDK internals to
> > > > fake EAL threads and avoid performance penalty of only having non-EAL
> > > > threads.
> > > >
> > > > This series proposes to add a new type of lcores and maps those external
> > > > threads to such lcores.
> > > > Those threads won't run the DPDK eal mainloop and as a consequence part of
> > > > the EAL threads API cannot work.
> > > >
> > > > Having new lcores appearing during the process lifetime is not expected
> > > > by some DPDK components. This is addressed by notifying of such lcore
> > > > hotplug.
> > > >
> > > > This patchset has still some more work (like refusing new lcore type in
> > > > incompatible EAL threads API, updating the documentation and adding unit
> > > > tests) but I am sending it anyway as I would like to get this in for
> > > > 20.08.
> > >
> > > Cool feature.
> > >
> > > Is mempool's lcore local cache working for external cores with this scheme?
> >
> > Yes, as it is stateless, all we need is a unique lcore_id in [0,
> > RTE_MAX_LCORE-1] range.
>
> Was it the case when lcore registered and then mempool created? What
> about other case, mempool created and then lcore registered?

- All caches are initialised for all possible lcores for each mempool.
https://git.dpdk.org/dpdk/tree/lib/librte_mempool/rte_mempool.c#n965
So any order is fine wrt the local mempool cache.

- If the mempool drivers want to initialise per lcore data on demand
[1], the driver have to register a lcore notifier per mempool.
1: https://git.dpdk.org/dpdk/tree/drivers/mempool/bucket/rte_mempool_bucket.c#n437

But this current series implementation does not handle registering in any order.
I will fix this in v2 (and rework the locking which is quite ugly)
hopefully this week.