| Message ID | 20200610144506.30505-1-david.marchand@redhat.com (mailing list archive) |
|---|---|
| Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8E21BA051F; Wed, 10 Jun 2020 16:45:24 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DFD701BF86; Wed, 10 Jun 2020 16:45:23 +0200 (CEST) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) by dpdk.org (Postfix) with ESMTP id C731B1BF80 for <dev@dpdk.org>; Wed, 10 Jun 2020 16:45:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591800322; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Gtf8GQQiQ5cI5mgUJdR5XtYW9/tJCCFhUVfXzp6k6f0=; b=XrId+geoy/gbAGfdL4QPEvHzf2R6qZa4D4yuWOGoef0kjV4Y6SfqWhQaUQCNAjH/0W2skj jaOSqww3JZJDjrmEAznkNYOHKnrSwhfo/8gi0KWAmYn5CfpPrR2i8+xIMRmjXMIXIycbDB mxJ2Pb1e19ZhtQK9McMLUJmOs3l6sYc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-209-GYIoE3-ZMrSv-cPEb0Efng-1; Wed, 10 Jun 2020 10:45:19 -0400 X-MC-Unique: GYIoE3-ZMrSv-cPEb0Efng-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8FA39800053 for <dev@dpdk.org>; Wed, 10 Jun 2020 14:45:18 +0000 (UTC) Received: from dmarchan.remote.csb (unknown [10.40.192.116]) by smtp.corp.redhat.com (Postfix) with ESMTP id EACE060E3E for <dev@dpdk.org>; Wed, 10 Jun 2020 14:45:17 +0000 (UTC) From: David Marchand <david.marchand@redhat.com> To: dev@dpdk.org Date: Wed, 10 Jun 2020 16:44:59 +0200 Message-Id: <20200610144506.30505-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH 0/7] Register external threads as lcore X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
| Series |
Register external threads as lcore
|
|
Message
David Marchand
June 10, 2020, 2:44 p.m. UTC
OVS and some other applications have been hacking into DPDK internals to fake EAL threads and avoid performance penalty of only having non-EAL threads. This series proposes to add a new type of lcores and maps those external threads to such lcores. Those threads won't run the DPDK eal mainloop and as a consequence part of the EAL threads API cannot work. Having new lcores appearing during the process lifetime is not expected by some DPDK components. This is addressed by notifying of such lcore hotplug. This patchset has still some more work (like refusing new lcore type in incompatible EAL threads API, updating the documentation and adding unit tests) but I am sending it anyway as I would like to get this in for 20.08. For the interested parties, I have a patch [1] against dpdk-latest OVS branch that makes use of this series. 1: https://github.com/david-marchand/ovs/commit/dpdk-latest-external-lcores
Comments
On Wed, Jun 10, 2020 at 8:15 PM David Marchand <david.marchand@redhat.com> wrote: > > OVS and some other applications have been hacking into DPDK internals to > fake EAL threads and avoid performance penalty of only having non-EAL > threads. > > This series proposes to add a new type of lcores and maps those external > threads to such lcores. > Those threads won't run the DPDK eal mainloop and as a consequence part of > the EAL threads API cannot work. > > Having new lcores appearing during the process lifetime is not expected > by some DPDK components. This is addressed by notifying of such lcore > hotplug. > > This patchset has still some more work (like refusing new lcore type in > incompatible EAL threads API, updating the documentation and adding unit > tests) but I am sending it anyway as I would like to get this in for > 20.08. Cool feature. Is mempool's lcore local cache working for external cores with this scheme? > > For the interested parties, I have a patch [1] against dpdk-latest OVS > branch that makes use of this series. > > 1: https://github.com/david-marchand/ovs/commit/dpdk-latest-external-lcores Usage is clear from the example code, Looking forward to having documentation and UT for the new API in the next version. > > -- > David Marchand > > David Marchand (7): > eal: relocate per thread symbols to common > eal: fix multiple definition of per lcore thread id > eal: introduce thread init helper > eal: introduce thread uninit helper > eal: register non-EAL threads as lcore > eal: dump lcores > eal: add lcore hotplug notifications > > doc/guides/howto/debug_troubleshoot.rst | 5 +- > lib/librte_eal/common/eal_common_lcore.c | 167 +++++++++++++++++++++- > lib/librte_eal/common/eal_common_thread.c | 143 ++++++++++++++---- > lib/librte_eal/common/eal_common_trace.c | 49 ++++++- > lib/librte_eal/common/eal_private.h | 43 ++++++ > lib/librte_eal/common/eal_thread.h | 26 +++- > lib/librte_eal/common/eal_trace.h | 1 + > lib/librte_eal/freebsd/eal.c | 16 ++- > lib/librte_eal/freebsd/eal_thread.c | 38 +---- > lib/librte_eal/include/rte_eal.h | 12 +- > lib/librte_eal/include/rte_lcore.h | 112 +++++++++++++-- > lib/librte_eal/linux/eal.c | 17 ++- > lib/librte_eal/linux/eal_thread.c | 38 +---- > lib/librte_eal/rte_eal_version.map | 14 ++ > lib/librte_eal/windows/eal.c | 3 +- > lib/librte_eal/windows/eal_thread.c | 14 +- > 16 files changed, 536 insertions(+), 162 deletions(-) > > -- > 2.23.0 >
On Wed, Jun 10, 2020 at 08:39:30PM +0530, Jerin Jacob wrote: > On Wed, Jun 10, 2020 at 8:15 PM David Marchand > <david.marchand@redhat.com> wrote: > > > > OVS and some other applications have been hacking into DPDK internals to > > fake EAL threads and avoid performance penalty of only having non-EAL > > threads. > > > > This series proposes to add a new type of lcores and maps those external > > threads to such lcores. > > Those threads won't run the DPDK eal mainloop and as a consequence part of > > the EAL threads API cannot work. > > > > Having new lcores appearing during the process lifetime is not expected > > by some DPDK components. This is addressed by notifying of such lcore > > hotplug. > > > > This patchset has still some more work (like refusing new lcore type in > > incompatible EAL threads API, updating the documentation and adding unit > > tests) but I am sending it anyway as I would like to get this in for > > 20.08. > > Cool feature. > +1 for this, this is something that I think has been needed for a long time. Thanks! > Is mempool's lcore local cache working for external cores with this scheme? > > > > > > For the interested parties, I have a patch [1] against dpdk-latest OVS > > branch that makes use of this series. > > > > 1: https://github.com/david-marchand/ovs/commit/dpdk-latest-external-lcores > > Usage is clear from the example code, > Looking forward to having documentation and UT for the new API > in the next version. > Yes, I think we need a doc update explaining usage. The EAL chapter of the programmers guide needs an update [1], and I think a description in the howto would be good also [2]. [1] https://doc.dpdk.org/guides-20.05/prog_guide/env_abstraction_layer.html [2] https://doc.dpdk.org/guides-20.05/howto/index.html Regards, /Bruce
On Wed, Jun 10, 2020 at 5:09 PM Jerin Jacob <jerinjacobk@gmail.com> wrote: > > On Wed, Jun 10, 2020 at 8:15 PM David Marchand > <david.marchand@redhat.com> wrote: > > > > OVS and some other applications have been hacking into DPDK internals to > > fake EAL threads and avoid performance penalty of only having non-EAL > > threads. > > > > This series proposes to add a new type of lcores and maps those external > > threads to such lcores. > > Those threads won't run the DPDK eal mainloop and as a consequence part of > > the EAL threads API cannot work. > > > > Having new lcores appearing during the process lifetime is not expected > > by some DPDK components. This is addressed by notifying of such lcore > > hotplug. > > > > This patchset has still some more work (like refusing new lcore type in > > incompatible EAL threads API, updating the documentation and adding unit > > tests) but I am sending it anyway as I would like to get this in for > > 20.08. > > Cool feature. > > Is mempool's lcore local cache working for external cores with this scheme? Yes, as it is stateless, all we need is a unique lcore_id in [0, RTE_MAX_LCORE-1] range. We could imagine flushing such caches on unregistering. And we can fix other mempool drivers like the bucket driver.
On Wed, Jun 10, 2020 at 8:48 PM David Marchand <david.marchand@redhat.com> wrote: > > On Wed, Jun 10, 2020 at 5:09 PM Jerin Jacob <jerinjacobk@gmail.com> wrote: > > > > On Wed, Jun 10, 2020 at 8:15 PM David Marchand > > <david.marchand@redhat.com> wrote: > > > > > > OVS and some other applications have been hacking into DPDK internals to > > > fake EAL threads and avoid performance penalty of only having non-EAL > > > threads. > > > > > > This series proposes to add a new type of lcores and maps those external > > > threads to such lcores. > > > Those threads won't run the DPDK eal mainloop and as a consequence part of > > > the EAL threads API cannot work. > > > > > > Having new lcores appearing during the process lifetime is not expected > > > by some DPDK components. This is addressed by notifying of such lcore > > > hotplug. > > > > > > This patchset has still some more work (like refusing new lcore type in > > > incompatible EAL threads API, updating the documentation and adding unit > > > tests) but I am sending it anyway as I would like to get this in for > > > 20.08. > > > > Cool feature. > > > > Is mempool's lcore local cache working for external cores with this scheme? > > Yes, as it is stateless, all we need is a unique lcore_id in [0, > RTE_MAX_LCORE-1] range. Was it the case when lcore registered and then mempool created? What about other case, mempool created and then lcore registered? > We could imagine flushing such caches on unregistering. > > And we can fix other mempool drivers like the bucket driver. > > -- > David Marchand >
On Wed, Jun 10, 2020 at 5:33 PM Jerin Jacob <jerinjacobk@gmail.com> wrote: > > On Wed, Jun 10, 2020 at 8:48 PM David Marchand > <david.marchand@redhat.com> wrote: > > > > On Wed, Jun 10, 2020 at 5:09 PM Jerin Jacob <jerinjacobk@gmail.com> wrote: > > > > > > On Wed, Jun 10, 2020 at 8:15 PM David Marchand > > > <david.marchand@redhat.com> wrote: > > > > > > > > OVS and some other applications have been hacking into DPDK internals to > > > > fake EAL threads and avoid performance penalty of only having non-EAL > > > > threads. > > > > > > > > This series proposes to add a new type of lcores and maps those external > > > > threads to such lcores. > > > > Those threads won't run the DPDK eal mainloop and as a consequence part of > > > > the EAL threads API cannot work. > > > > > > > > Having new lcores appearing during the process lifetime is not expected > > > > by some DPDK components. This is addressed by notifying of such lcore > > > > hotplug. > > > > > > > > This patchset has still some more work (like refusing new lcore type in > > > > incompatible EAL threads API, updating the documentation and adding unit > > > > tests) but I am sending it anyway as I would like to get this in for > > > > 20.08. > > > > > > Cool feature. > > > > > > Is mempool's lcore local cache working for external cores with this scheme? > > > > Yes, as it is stateless, all we need is a unique lcore_id in [0, > > RTE_MAX_LCORE-1] range. > > Was it the case when lcore registered and then mempool created? What > about other case, mempool created and then lcore registered? - All caches are initialised for all possible lcores for each mempool. https://git.dpdk.org/dpdk/tree/lib/librte_mempool/rte_mempool.c#n965 So any order is fine wrt the local mempool cache. - If the mempool drivers want to initialise per lcore data on demand [1], the driver have to register a lcore notifier per mempool. 1: https://git.dpdk.org/dpdk/tree/drivers/mempool/bucket/rte_mempool_bucket.c#n437 But this current series implementation does not handle registering in any order. I will fix this in v2 (and rework the locking which is quite ugly) hopefully this week.