[0/8] Remove IPC threads

  As previously discussed [1], IPC threads need to be removed and their
workload moved to interrupt thread.

FreeBSD did not have an interrupt thread, nor did it support alarm
API. This patchset adds support for both on FreeBSD. FreeBSD interrupt
thread is based on kevent, FreeBSD's native event multiplexing
mechanism similar to Linux's epoll.

The patchset makes FreeBSD's interrupts and alarm work just enough to
suffice for purposes of IPC, however there are really weird problems
observed. Specifically, FreeBSD's kevent timers are really slow to
trigger for some reason, sleeping on a 10ms timer as much as 200ms
before waking up. Interrupt handling on fd's is also a bit flaky.

It has also been observed that both problems go away if we do not
affinitize master lcore (by commenting relevant code out [2]). It is
not known why these problems are observed, nor it is clear what a
solution might entail.

For the purposes of making IPC work and having rudimentary support
for alarm and interrupt API's, this patchset works fine. However,
because of the above described issues, documentation will not be
updated to indicate support for interrupts on FreeBSD at this time.

[1] http://dpdk.org/dev/patchwork/patch/36579/
[2] http://dpdk.org/browse/dpdk/tree/lib/librte_eal/bsdapp/eal/eal.c#n729

Anatoly Burakov (4):
  ipc: remove IPC thread for async requests
  eal/bsdapp: add interrupt thread
  eal/bsdapp: add alarm support
  ipc: remove main IPC thread

Jianfeng Tan (4):
  eal/linux: use glibc malloc in alarm
  eal/linux: use glibc malloc in interrupt handling
  eal: bring forward init of interrupt handling
  eal: add IPC type for interrupt thread

 lib/librte_eal/bsdapp/eal/eal.c               |  10 +-
 lib/librte_eal/bsdapp/eal/eal_alarm.c         | 299 +++++++++++-
 lib/librte_eal/bsdapp/eal/eal_alarm_private.h |  19 +
 lib/librte_eal/bsdapp/eal/eal_interrupts.c    | 460 +++++++++++++++++-
 lib/librte_eal/common/eal_common_proc.c       | 243 ++++-----
 .../common/include/rte_eal_interrupts.h       |   1 +
 lib/librte_eal/linuxapp/eal/eal.c             |  10 +-
 lib/librte_eal/linuxapp/eal/eal_alarm.c       |   9 +-
 lib/librte_eal/linuxapp/eal/eal_interrupts.c  |  19 +-
 test/test/test_interrupts.c                   |  29 +-
 10 files changed, 899 insertions(+), 200 deletions(-)
 create mode 100644 lib/librte_eal/bsdapp/eal/eal_alarm_private.h
  

Hi Anatoly and Thomas:

Sorry for raise this late, but seems merge mp thread into interrupt thread gives problem to enable hotplug on secondary [1].

The issue is, when secondary want to attach a share device, it send request to primary
Then primary is running in mp callback (mp thread) to attach device, it will call rte_malloc which get chance to increase heap that will do sync IPC,
You know, this is the limitation we can't do sync IPC in mp thread itself. so the solution is try to move real work to a separate thread which has no limitation to do sync IPC, 
and interrupt thread is the good candidate, because we just need to call rte_eal_set_alarm and we don't need to worry about the execution sequence.

But if we merge mp thread into interrupt thread, the solution will not work, we may need to create specific temporal thread to handle callback, but this looks like some re-build which we already have.
So I think we need to revisit if we need to merge the thread before we have a good solution for such kind of issue.

Thanks
Qi

[1] https://mails.dpdk.org/archives/dev/2018-June/105018.html


> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Anatoly Burakov
> Sent: Friday, June 15, 2018 10:25 PM
> To: dev@dpdk.org
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> thomas@monjalon.net; Richardson, Bruce <bruce.richardson@intel.com>
> Subject: [dpdk-dev] [PATCH 0/8] Remove IPC threads
> 
> As previously discussed [1], IPC threads need to be removed and their
> workload moved to interrupt thread.
> 
> FreeBSD did not have an interrupt thread, nor did it support alarm API. This
> patchset adds support for both on FreeBSD. FreeBSD interrupt thread is based
> on kevent, FreeBSD's native event multiplexing mechanism similar to Linux's
> epoll.
> 
> The patchset makes FreeBSD's interrupts and alarm work just enough to
> suffice for purposes of IPC, however there are really weird problems observed.
> Specifically, FreeBSD's kevent timers are really slow to trigger for some reason,
> sleeping on a 10ms timer as much as 200ms before waking up. Interrupt
> handling on fd's is also a bit flaky.
> 
> It has also been observed that both problems go away if we do not affinitize
> master lcore (by commenting relevant code out [2]). It is not known why these
> problems are observed, nor it is clear what a solution might entail.
> 
> For the purposes of making IPC work and having rudimentary support for
> alarm and interrupt API's, this patchset works fine. However, because of the
> above described issues, documentation will not be updated to indicate
> support for interrupts on FreeBSD at this time.
> 
> [1] http://dpdk.org/dev/patchwork/patch/36579/
> [2] http://dpdk.org/browse/dpdk/tree/lib/librte_eal/bsdapp/eal/eal.c#n729
> 
> Anatoly Burakov (4):
>   ipc: remove IPC thread for async requests
>   eal/bsdapp: add interrupt thread
>   eal/bsdapp: add alarm support
>   ipc: remove main IPC thread
> 
> Jianfeng Tan (4):
>   eal/linux: use glibc malloc in alarm
>   eal/linux: use glibc malloc in interrupt handling
>   eal: bring forward init of interrupt handling
>   eal: add IPC type for interrupt thread
> 
>  lib/librte_eal/bsdapp/eal/eal.c               |  10 +-
>  lib/librte_eal/bsdapp/eal/eal_alarm.c         | 299 +++++++++++-
>  lib/librte_eal/bsdapp/eal/eal_alarm_private.h |  19 +
>  lib/librte_eal/bsdapp/eal/eal_interrupts.c    | 460 +++++++++++++++++-
>  lib/librte_eal/common/eal_common_proc.c       | 243 ++++-----
>  .../common/include/rte_eal_interrupts.h       |   1 +
>  lib/librte_eal/linuxapp/eal/eal.c             |  10 +-
>  lib/librte_eal/linuxapp/eal/eal_alarm.c       |   9 +-
>  lib/librte_eal/linuxapp/eal/eal_interrupts.c  |  19 +-
>  test/test/test_interrupts.c                   |  29 +-
>  10 files changed, 899 insertions(+), 200 deletions(-)  create mode 100644
> lib/librte_eal/bsdapp/eal/eal_alarm_private.h
> 
> --
> 2.17.1
  

> -----Original Message-----
> From: Zhang, Qi Z
> Sent: Tuesday, June 26, 2018 9:19 AM
> To: 'Anatoly Burakov' <anatoly.burakov@intel.com>; dev@dpdk.org
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> thomas@monjalon.net; Richardson, Bruce <bruce.richardson@intel.com>
> Subject: RE: [dpdk-dev] [PATCH 0/8] Remove IPC threads
> 
> Hi Anatoly and Thomas:
> 
> Sorry for raise this late, but seems merge mp thread into interrupt thread gives
> problem to enable hotplug on secondary [1].
> 
> The issue is, when secondary want to attach a share device, it send request to
> primary Then primary is running in mp callback (mp thread) to attach device, it
> will call rte_malloc which get chance to increase heap that will do sync IPC, You
> know, this is the limitation we can't do sync IPC in mp thread itself. so the
> solution is try to move real work to a separate thread which has no limitation
> to do sync IPC, and interrupt thread is the good candidate, because we just
> need to call rte_eal_set_alarm and we don't need to worry about the
> execution sequence.
> 
> But if we merge mp thread into interrupt thread, the solution will not work, we
> may need to create specific temporal thread to handle callback, but this looks
> like some re-build which we already have.

Ok, actually this method looks good to me :), I will send v4 to have this,
But please let me know if you have better idea.

> So I think we need to revisit if we need to merge the thread before we have a
> good solution for such kind of issue.
> 
> Thanks
> Qi
> 
> [1] https://mails.dpdk.org/archives/dev/2018-June/105018.html
> 
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Anatoly Burakov
> > Sent: Friday, June 15, 2018 10:25 PM
> > To: dev@dpdk.org
> > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> > thomas@monjalon.net; Richardson, Bruce <bruce.richardson@intel.com>
> > Subject: [dpdk-dev] [PATCH 0/8] Remove IPC threads
> >
> > As previously discussed [1], IPC threads need to be removed and their
> > workload moved to interrupt thread.
> >
> > FreeBSD did not have an interrupt thread, nor did it support alarm
> > API. This patchset adds support for both on FreeBSD. FreeBSD interrupt
> > thread is based on kevent, FreeBSD's native event multiplexing
> > mechanism similar to Linux's epoll.
> >
> > The patchset makes FreeBSD's interrupts and alarm work just enough to
> > suffice for purposes of IPC, however there are really weird problems
> observed.
> > Specifically, FreeBSD's kevent timers are really slow to trigger for
> > some reason, sleeping on a 10ms timer as much as 200ms before waking
> > up. Interrupt handling on fd's is also a bit flaky.
> >
> > It has also been observed that both problems go away if we do not
> > affinitize master lcore (by commenting relevant code out [2]). It is
> > not known why these problems are observed, nor it is clear what a solution
> might entail.
> >
> > For the purposes of making IPC work and having rudimentary support for
> > alarm and interrupt API's, this patchset works fine. However, because
> > of the above described issues, documentation will not be updated to
> > indicate support for interrupts on FreeBSD at this time.
> >
> > [1] http://dpdk.org/dev/patchwork/patch/36579/
> > [2]
> > http://dpdk.org/browse/dpdk/tree/lib/librte_eal/bsdapp/eal/eal.c#n729
> >
> > Anatoly Burakov (4):
> >   ipc: remove IPC thread for async requests
> >   eal/bsdapp: add interrupt thread
> >   eal/bsdapp: add alarm support
> >   ipc: remove main IPC thread
> >
> > Jianfeng Tan (4):
> >   eal/linux: use glibc malloc in alarm
> >   eal/linux: use glibc malloc in interrupt handling
> >   eal: bring forward init of interrupt handling
> >   eal: add IPC type for interrupt thread
> >
> >  lib/librte_eal/bsdapp/eal/eal.c               |  10 +-
> >  lib/librte_eal/bsdapp/eal/eal_alarm.c         | 299 +++++++++++-
> >  lib/librte_eal/bsdapp/eal/eal_alarm_private.h |  19 +
> >  lib/librte_eal/bsdapp/eal/eal_interrupts.c    | 460 +++++++++++++++++-
> >  lib/librte_eal/common/eal_common_proc.c       | 243 ++++-----
> >  .../common/include/rte_eal_interrupts.h       |   1 +
> >  lib/librte_eal/linuxapp/eal/eal.c             |  10 +-
> >  lib/librte_eal/linuxapp/eal/eal_alarm.c       |   9 +-
> >  lib/librte_eal/linuxapp/eal/eal_interrupts.c  |  19 +-
> >  test/test/test_interrupts.c                   |  29 +-
> >  10 files changed, 899 insertions(+), 200 deletions(-)  create mode
> > 100644 lib/librte_eal/bsdapp/eal/eal_alarm_private.h
> >
> > --
> > 2.17.1