Message ID | cover.1529071026.git.anatoly.burakov@intel.com (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B27621D63D; Fri, 15 Jun 2018 16:25:13 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 54FC61D616 for <dev@dpdk.org>; Fri, 15 Jun 2018 16:25:11 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Jun 2018 07:25:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,227,1526367600"; d="scan'208";a="50178064" Received: from irvmail001.ir.intel.com ([163.33.26.43]) by orsmga006.jf.intel.com with ESMTP; 15 Jun 2018 07:25:08 -0700 Received: from sivswdev01.ir.intel.com (sivswdev01.ir.intel.com [10.237.217.45]) by irvmail001.ir.intel.com (8.14.3/8.13.6/MailSET/Hub) with ESMTP id w5FEP8m5028172; Fri, 15 Jun 2018 15:25:08 +0100 Received: from sivswdev01.ir.intel.com (localhost [127.0.0.1]) by sivswdev01.ir.intel.com with ESMTP id w5FEP8DU019435; Fri, 15 Jun 2018 15:25:08 +0100 Received: (from aburakov@localhost) by sivswdev01.ir.intel.com with LOCAL id w5FEP8xB019431; Fri, 15 Jun 2018 15:25:08 +0100 From: Anatoly Burakov <anatoly.burakov@intel.com> To: dev@dpdk.org Cc: konstantin.ananyev@intel.com, thomas@monjalon.net, bruce.richardson@intel.com Date: Fri, 15 Jun 2018 15:25:00 +0100 Message-Id: <cover.1529071026.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 1.7.0.7 Subject: [dpdk-dev] [PATCH 0/8] Remove IPC threads X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
Series |
Remove IPC threads
|
|
Message
Anatoly Burakov
June 15, 2018, 2:25 p.m. UTC
As previously discussed [1], IPC threads need to be removed and their workload moved to interrupt thread. FreeBSD did not have an interrupt thread, nor did it support alarm API. This patchset adds support for both on FreeBSD. FreeBSD interrupt thread is based on kevent, FreeBSD's native event multiplexing mechanism similar to Linux's epoll. The patchset makes FreeBSD's interrupts and alarm work just enough to suffice for purposes of IPC, however there are really weird problems observed. Specifically, FreeBSD's kevent timers are really slow to trigger for some reason, sleeping on a 10ms timer as much as 200ms before waking up. Interrupt handling on fd's is also a bit flaky. It has also been observed that both problems go away if we do not affinitize master lcore (by commenting relevant code out [2]). It is not known why these problems are observed, nor it is clear what a solution might entail. For the purposes of making IPC work and having rudimentary support for alarm and interrupt API's, this patchset works fine. However, because of the above described issues, documentation will not be updated to indicate support for interrupts on FreeBSD at this time. [1] http://dpdk.org/dev/patchwork/patch/36579/ [2] http://dpdk.org/browse/dpdk/tree/lib/librte_eal/bsdapp/eal/eal.c#n729 Anatoly Burakov (4): ipc: remove IPC thread for async requests eal/bsdapp: add interrupt thread eal/bsdapp: add alarm support ipc: remove main IPC thread Jianfeng Tan (4): eal/linux: use glibc malloc in alarm eal/linux: use glibc malloc in interrupt handling eal: bring forward init of interrupt handling eal: add IPC type for interrupt thread lib/librte_eal/bsdapp/eal/eal.c | 10 +- lib/librte_eal/bsdapp/eal/eal_alarm.c | 299 +++++++++++- lib/librte_eal/bsdapp/eal/eal_alarm_private.h | 19 + lib/librte_eal/bsdapp/eal/eal_interrupts.c | 460 +++++++++++++++++- lib/librte_eal/common/eal_common_proc.c | 243 ++++----- .../common/include/rte_eal_interrupts.h | 1 + lib/librte_eal/linuxapp/eal/eal.c | 10 +- lib/librte_eal/linuxapp/eal/eal_alarm.c | 9 +- lib/librte_eal/linuxapp/eal/eal_interrupts.c | 19 +- test/test/test_interrupts.c | 29 +- 10 files changed, 899 insertions(+), 200 deletions(-) create mode 100644 lib/librte_eal/bsdapp/eal/eal_alarm_private.h
Comments
Hi Anatoly and Thomas: Sorry for raise this late, but seems merge mp thread into interrupt thread gives problem to enable hotplug on secondary [1]. The issue is, when secondary want to attach a share device, it send request to primary Then primary is running in mp callback (mp thread) to attach device, it will call rte_malloc which get chance to increase heap that will do sync IPC, You know, this is the limitation we can't do sync IPC in mp thread itself. so the solution is try to move real work to a separate thread which has no limitation to do sync IPC, and interrupt thread is the good candidate, because we just need to call rte_eal_set_alarm and we don't need to worry about the execution sequence. But if we merge mp thread into interrupt thread, the solution will not work, we may need to create specific temporal thread to handle callback, but this looks like some re-build which we already have. So I think we need to revisit if we need to merge the thread before we have a good solution for such kind of issue. Thanks Qi [1] https://mails.dpdk.org/archives/dev/2018-June/105018.html > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Anatoly Burakov > Sent: Friday, June 15, 2018 10:25 PM > To: dev@dpdk.org > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; > thomas@monjalon.net; Richardson, Bruce <bruce.richardson@intel.com> > Subject: [dpdk-dev] [PATCH 0/8] Remove IPC threads > > As previously discussed [1], IPC threads need to be removed and their > workload moved to interrupt thread. > > FreeBSD did not have an interrupt thread, nor did it support alarm API. This > patchset adds support for both on FreeBSD. FreeBSD interrupt thread is based > on kevent, FreeBSD's native event multiplexing mechanism similar to Linux's > epoll. > > The patchset makes FreeBSD's interrupts and alarm work just enough to > suffice for purposes of IPC, however there are really weird problems observed. > Specifically, FreeBSD's kevent timers are really slow to trigger for some reason, > sleeping on a 10ms timer as much as 200ms before waking up. Interrupt > handling on fd's is also a bit flaky. > > It has also been observed that both problems go away if we do not affinitize > master lcore (by commenting relevant code out [2]). It is not known why these > problems are observed, nor it is clear what a solution might entail. > > For the purposes of making IPC work and having rudimentary support for > alarm and interrupt API's, this patchset works fine. However, because of the > above described issues, documentation will not be updated to indicate > support for interrupts on FreeBSD at this time. > > [1] http://dpdk.org/dev/patchwork/patch/36579/ > [2] http://dpdk.org/browse/dpdk/tree/lib/librte_eal/bsdapp/eal/eal.c#n729 > > Anatoly Burakov (4): > ipc: remove IPC thread for async requests > eal/bsdapp: add interrupt thread > eal/bsdapp: add alarm support > ipc: remove main IPC thread > > Jianfeng Tan (4): > eal/linux: use glibc malloc in alarm > eal/linux: use glibc malloc in interrupt handling > eal: bring forward init of interrupt handling > eal: add IPC type for interrupt thread > > lib/librte_eal/bsdapp/eal/eal.c | 10 +- > lib/librte_eal/bsdapp/eal/eal_alarm.c | 299 +++++++++++- > lib/librte_eal/bsdapp/eal/eal_alarm_private.h | 19 + > lib/librte_eal/bsdapp/eal/eal_interrupts.c | 460 +++++++++++++++++- > lib/librte_eal/common/eal_common_proc.c | 243 ++++----- > .../common/include/rte_eal_interrupts.h | 1 + > lib/librte_eal/linuxapp/eal/eal.c | 10 +- > lib/librte_eal/linuxapp/eal/eal_alarm.c | 9 +- > lib/librte_eal/linuxapp/eal/eal_interrupts.c | 19 +- > test/test/test_interrupts.c | 29 +- > 10 files changed, 899 insertions(+), 200 deletions(-) create mode 100644 > lib/librte_eal/bsdapp/eal/eal_alarm_private.h > > -- > 2.17.1
> -----Original Message----- > From: Zhang, Qi Z > Sent: Tuesday, June 26, 2018 9:19 AM > To: 'Anatoly Burakov' <anatoly.burakov@intel.com>; dev@dpdk.org > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; > thomas@monjalon.net; Richardson, Bruce <bruce.richardson@intel.com> > Subject: RE: [dpdk-dev] [PATCH 0/8] Remove IPC threads > > Hi Anatoly and Thomas: > > Sorry for raise this late, but seems merge mp thread into interrupt thread gives > problem to enable hotplug on secondary [1]. > > The issue is, when secondary want to attach a share device, it send request to > primary Then primary is running in mp callback (mp thread) to attach device, it > will call rte_malloc which get chance to increase heap that will do sync IPC, You > know, this is the limitation we can't do sync IPC in mp thread itself. so the > solution is try to move real work to a separate thread which has no limitation > to do sync IPC, and interrupt thread is the good candidate, because we just > need to call rte_eal_set_alarm and we don't need to worry about the > execution sequence. > > But if we merge mp thread into interrupt thread, the solution will not work, we > may need to create specific temporal thread to handle callback, but this looks > like some re-build which we already have. Ok, actually this method looks good to me :), I will send v4 to have this, But please let me know if you have better idea. > So I think we need to revisit if we need to merge the thread before we have a > good solution for such kind of issue. > > Thanks > Qi > > [1] https://mails.dpdk.org/archives/dev/2018-June/105018.html > > > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Anatoly Burakov > > Sent: Friday, June 15, 2018 10:25 PM > > To: dev@dpdk.org > > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; > > thomas@monjalon.net; Richardson, Bruce <bruce.richardson@intel.com> > > Subject: [dpdk-dev] [PATCH 0/8] Remove IPC threads > > > > As previously discussed [1], IPC threads need to be removed and their > > workload moved to interrupt thread. > > > > FreeBSD did not have an interrupt thread, nor did it support alarm > > API. This patchset adds support for both on FreeBSD. FreeBSD interrupt > > thread is based on kevent, FreeBSD's native event multiplexing > > mechanism similar to Linux's epoll. > > > > The patchset makes FreeBSD's interrupts and alarm work just enough to > > suffice for purposes of IPC, however there are really weird problems > observed. > > Specifically, FreeBSD's kevent timers are really slow to trigger for > > some reason, sleeping on a 10ms timer as much as 200ms before waking > > up. Interrupt handling on fd's is also a bit flaky. > > > > It has also been observed that both problems go away if we do not > > affinitize master lcore (by commenting relevant code out [2]). It is > > not known why these problems are observed, nor it is clear what a solution > might entail. > > > > For the purposes of making IPC work and having rudimentary support for > > alarm and interrupt API's, this patchset works fine. However, because > > of the above described issues, documentation will not be updated to > > indicate support for interrupts on FreeBSD at this time. > > > > [1] http://dpdk.org/dev/patchwork/patch/36579/ > > [2] > > http://dpdk.org/browse/dpdk/tree/lib/librte_eal/bsdapp/eal/eal.c#n729 > > > > Anatoly Burakov (4): > > ipc: remove IPC thread for async requests > > eal/bsdapp: add interrupt thread > > eal/bsdapp: add alarm support > > ipc: remove main IPC thread > > > > Jianfeng Tan (4): > > eal/linux: use glibc malloc in alarm > > eal/linux: use glibc malloc in interrupt handling > > eal: bring forward init of interrupt handling > > eal: add IPC type for interrupt thread > > > > lib/librte_eal/bsdapp/eal/eal.c | 10 +- > > lib/librte_eal/bsdapp/eal/eal_alarm.c | 299 +++++++++++- > > lib/librte_eal/bsdapp/eal/eal_alarm_private.h | 19 + > > lib/librte_eal/bsdapp/eal/eal_interrupts.c | 460 +++++++++++++++++- > > lib/librte_eal/common/eal_common_proc.c | 243 ++++----- > > .../common/include/rte_eal_interrupts.h | 1 + > > lib/librte_eal/linuxapp/eal/eal.c | 10 +- > > lib/librte_eal/linuxapp/eal/eal_alarm.c | 9 +- > > lib/librte_eal/linuxapp/eal/eal_interrupts.c | 19 +- > > test/test/test_interrupts.c | 29 +- > > 10 files changed, 899 insertions(+), 200 deletions(-) create mode > > 100644 lib/librte_eal/bsdapp/eal/eal_alarm_private.h > > > > -- > > 2.17.1