From patchwork Thu Mar 12 07:44:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 66563 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0F071A0569; Thu, 12 Mar 2020 08:45:42 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id EF9E11C014; Thu, 12 Mar 2020 08:45:33 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 6CBDB1C014 for ; Thu, 12 Mar 2020 08:45:32 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E2EE0FEC; Thu, 12 Mar 2020 00:45:31 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.150]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4A1683F534; Thu, 12 Mar 2020 00:45:28 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, harry.van.haaren@intel.com, konstantin.ananyev@intel.com, stephen@networkplumber.org, maxime.coquelin@redhat.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, hemant.agrawal@nxp.com, Honnappa.Nagarahalli@arm.com, gavin.hu@arm.com, ruifeng.wang@arm.com, joyce.kong@arm.com, nd@arm.com Date: Thu, 12 Mar 2020 15:44:22 +0800 Message-Id: <1583999071-22872-2-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1583999071-22872-1-git-send-email-phil.yang@arm.com> References: <1583862551-2049-1-git-send-email-phil.yang@arm.com> <1583999071-22872-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v2 01/10] doc: add generic atomic deprecation section X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add deprecating the generic rte_atomic_xx APIs to c11 atomic built-ins guide and examples. Suggested-by: Honnappa Nagarahalli Signed-off-by: Phil Yang Reviewed-by: Gavin Hu --- doc/guides/prog_guide/writing_efficient_code.rst | 60 +++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-) diff --git a/doc/guides/prog_guide/writing_efficient_code.rst b/doc/guides/prog_guide/writing_efficient_code.rst index 849f63e..b278bc6 100644 --- a/doc/guides/prog_guide/writing_efficient_code.rst +++ b/doc/guides/prog_guide/writing_efficient_code.rst @@ -167,7 +167,13 @@ but with the added cost of lower throughput. Locks and Atomic Operations --------------------------- -Atomic operations imply a lock prefix before the instruction, +This section describes some key considerations when using locks and atomic +operations in the DPDK environment. + +Locks +~~~~~ + +On x86, atomic operations imply a lock prefix before the instruction, causing the processor's LOCK# signal to be asserted during execution of the following instruction. This has a big impact on performance in a multicore environment. @@ -176,6 +182,58 @@ It can often be replaced by other solutions like per-lcore variables. Also, some locking techniques are more efficient than others. For instance, the Read-Copy-Update (RCU) algorithm can frequently replace simple rwlocks. +Atomic Operations: Use C11 Atomic Built-ins +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +DPDK `generic rte_atomic `_ operations are +implemented by `__sync built-ins `_. +These __sync built-ins result in full barriers on aarch64, which are unnecessary +in many use cases. They can be replaced by `__atomic built-ins `_ that +conform to the C11 memory model and provide finer memory order control. + +So replacing the rte_atomic operations with __atomic built-ins might improve +performance for aarch64 machines. `More details `_. + +Some typical optimization cases are listed below: + +Atomicity +^^^^^^^^^ + +Some use cases require atomicity alone, the ordering of the memory operations +does not matter. For example the packets statistics in the `vhost `_ example application. + +It just updates the number of transmitted packets, no subsequent logic depends +on these counters. So the RELAXED memory ordering is sufficient: + +.. code-block:: c + + static __rte_always_inline void + virtio_xmit(struct vhost_dev *dst_vdev, struct vhost_dev *src_vdev, + struct rte_mbuf *m) + { + ... + ... + if (enable_stats) { + __atomic_add_fetch(&dst_vdev->stats.rx_total_atomic, 1, __ATOMIC_RELAXED); + __atomic_add_fetch(&dst_vdev->stats.rx_atomic, ret, __ATOMIC_RELAXED); + ... + } + } + +One-way Barrier +^^^^^^^^^^^^^^^ + +Some use cases allow for memory reordering in one way while requiring memory +ordering in the other direction. + +For example, the memory operations before the `lock `_ can move to the +critical section, but the memory operations in the critical section cannot move +above the lock. In this case, the full memory barrier in the CAS operation can +be replaced to ACQUIRE. On the other hand, the memory operations after the +`unlock `_ can move to the critical section, but the memory operations in the +critical section cannot move below the unlock. So the full barrier in the STORE +operation can be replaced with RELEASE. + Coding Considerations --------------------- From patchwork Thu Mar 12 07:44:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 66564 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9D16AA0569; Thu, 12 Mar 2020 08:45:52 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id ABD6C1C020; Thu, 12 Mar 2020 08:45:37 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 71D2C1C01F for ; Thu, 12 Mar 2020 08:45:36 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0CF16106F; Thu, 12 Mar 2020 00:45:36 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.150]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 584343F534; Thu, 12 Mar 2020 00:45:32 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, harry.van.haaren@intel.com, konstantin.ananyev@intel.com, stephen@networkplumber.org, maxime.coquelin@redhat.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, hemant.agrawal@nxp.com, Honnappa.Nagarahalli@arm.com, gavin.hu@arm.com, ruifeng.wang@arm.com, joyce.kong@arm.com, nd@arm.com Date: Thu, 12 Mar 2020 15:44:23 +0800 Message-Id: <1583999071-22872-3-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1583999071-22872-1-git-send-email-phil.yang@arm.com> References: <1583862551-2049-1-git-send-email-phil.yang@arm.com> <1583999071-22872-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v2 02/10] devtools: prevent use of rte atomic APIs in future patches X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" In order to deprecate the rte_atomic APIs, prevent the patches from using rte_atomic APIs. Signed-off-by: Phil Yang Reviewed-by: Gavin Hu Reviewed-by: Honnappa Nagarahalli --- devtools/checkpatches.sh | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/devtools/checkpatches.sh b/devtools/checkpatches.sh index 1794468..493f48e 100755 --- a/devtools/checkpatches.sh +++ b/devtools/checkpatches.sh @@ -61,6 +61,15 @@ check_forbidden_additions() { # -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ "$1" || res=1 + # refrain from new additions of 16/32/64 bits rte_atomic_xxx() + # multiple folders and expressions are separated by spaces + awk -v FOLDERS="lib drivers app examples" \ + -v EXPRESSIONS="rte_atomic[0-9][0-9]_.*\\\(" \ + -v RET_ON_FAIL=1 \ + -v MESSAGE='Using c11 atomic built-ins instead of rte_atomic' \ + -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ + "$1" || res=1 + # svg figures must be included with wildcard extension # because of png conversion for pdf docs awk -v FOLDERS='doc' \ From patchwork Thu Mar 12 07:44:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 66565 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1FB10A0569; Thu, 12 Mar 2020 08:46:04 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0653D1C029; Thu, 12 Mar 2020 08:45:42 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 7E9171C010 for ; Thu, 12 Mar 2020 08:45:40 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F30A430E; Thu, 12 Mar 2020 00:45:39 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.150]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7633B3F534; Thu, 12 Mar 2020 00:45:36 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, harry.van.haaren@intel.com, konstantin.ananyev@intel.com, stephen@networkplumber.org, maxime.coquelin@redhat.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, hemant.agrawal@nxp.com, Honnappa.Nagarahalli@arm.com, gavin.hu@arm.com, ruifeng.wang@arm.com, joyce.kong@arm.com, nd@arm.com Date: Thu, 12 Mar 2020 15:44:24 +0800 Message-Id: <1583999071-22872-4-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1583999071-22872-1-git-send-email-phil.yang@arm.com> References: <1583862551-2049-1-git-send-email-phil.yang@arm.com> <1583999071-22872-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v2 03/10] vhost: optimize broadcast rarp sync with c11 atomic X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The rarp packet broadcast flag is synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier. Signed-off-by: Phil Yang Reviewed-by: Gavin Hu Reviewed-by: Honnappa Nagarahalli Reviewed-by: Joyce Kong --- lib/librte_vhost/vhost.h | 2 +- lib/librte_vhost/vhost_user.c | 7 +++---- lib/librte_vhost/virtio_net.c | 16 +++++++++------- 3 files changed, 13 insertions(+), 12 deletions(-) diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h index 2087d14..0e22125 100644 --- a/lib/librte_vhost/vhost.h +++ b/lib/librte_vhost/vhost.h @@ -350,7 +350,7 @@ struct virtio_net { uint32_t flags; uint16_t vhost_hlen; /* to tell if we need broadcast rarp packet */ - rte_atomic16_t broadcast_rarp; + int16_t broadcast_rarp; uint32_t nr_vring; int dequeue_zero_copy; int extbuf; diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c index bd1be01..857187d 100644 --- a/lib/librte_vhost/vhost_user.c +++ b/lib/librte_vhost/vhost_user.c @@ -2145,11 +2145,10 @@ vhost_user_send_rarp(struct virtio_net **pdev, struct VhostUserMsg *msg, * Set the flag to inject a RARP broadcast packet at * rte_vhost_dequeue_burst(). * - * rte_smp_wmb() is for making sure the mac is copied - * before the flag is set. + * __ATOMIC_RELEASE ordering is for making sure the mac is + * copied before the flag is set. */ - rte_smp_wmb(); - rte_atomic16_set(&dev->broadcast_rarp, 1); + __atomic_store_n(&dev->broadcast_rarp, 1, __ATOMIC_RELEASE); did = dev->vdpa_dev_id; vdpa_dev = rte_vdpa_get_device(did); if (vdpa_dev && vdpa_dev->ops->migration_done) diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c index 37c47c7..fa10deb 100644 --- a/lib/librte_vhost/virtio_net.c +++ b/lib/librte_vhost/virtio_net.c @@ -2203,6 +2203,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id, struct virtio_net *dev; struct rte_mbuf *rarp_mbuf = NULL; struct vhost_virtqueue *vq; + int16_t success = 1; dev = get_device(vid); if (!dev) @@ -2249,16 +2250,17 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id, * * broadcast_rarp shares a cacheline in the virtio_net structure * with some fields that are accessed during enqueue and - * rte_atomic16_cmpset() causes a write if using cmpxchg. This could - * result in false sharing between enqueue and dequeue. + * __atomic_compare_exchange_n causes a write if performed compare + * and exchange. This could result in false sharing between enqueue + * and dequeue. * * Prevent unnecessary false sharing by reading broadcast_rarp first - * and only performing cmpset if the read indicates it is likely to - * be set. + * and only performing compare and exchange if the read indicates it + * is likely to be set. */ - if (unlikely(rte_atomic16_read(&dev->broadcast_rarp) && - rte_atomic16_cmpset((volatile uint16_t *) - &dev->broadcast_rarp.cnt, 1, 0))) { + if (unlikely(__atomic_load_n(&dev->broadcast_rarp, __ATOMIC_ACQUIRE) && + __atomic_compare_exchange_n(&dev->broadcast_rarp, + &success, 0, 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED))) { rarp_mbuf = rte_net_make_rarp_packet(mbuf_pool, &dev->mac); if (rarp_mbuf == NULL) { From patchwork Thu Mar 12 07:44:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 66566 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C9718A0569; Thu, 12 Mar 2020 08:46:16 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3AF241C030; Thu, 12 Mar 2020 08:45:46 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id C71CD1C030 for ; Thu, 12 Mar 2020 08:45:44 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5915130E; Thu, 12 Mar 2020 00:45:44 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.150]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 67BAC3F534; Thu, 12 Mar 2020 00:45:40 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, harry.van.haaren@intel.com, konstantin.ananyev@intel.com, stephen@networkplumber.org, maxime.coquelin@redhat.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, hemant.agrawal@nxp.com, Honnappa.Nagarahalli@arm.com, gavin.hu@arm.com, ruifeng.wang@arm.com, joyce.kong@arm.com, nd@arm.com Date: Thu, 12 Mar 2020 15:44:25 +0800 Message-Id: <1583999071-22872-5-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1583999071-22872-1-git-send-email-phil.yang@arm.com> References: <1583862551-2049-1-git-send-email-phil.yang@arm.com> <1583999071-22872-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v2 04/10] ipsec: optimize with c11 atomic for sa outbound sqn update X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" For SA outbound packets, rte_atomic64_add_return is used to generate SQN atomically. This introduced an unnecessary full barrier by calling the '__sync' builtin implemented rte_atomic_XX API on aarch64. This patch optimized it with c11 atomic and eliminated the expensive barrier for aarch64. Signed-off-by: Phil Yang Reviewed-by: Ruifeng Wang Reviewed-by: Gavin Hu --- lib/librte_ipsec/ipsec_sqn.h | 3 ++- lib/librte_ipsec/sa.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/librte_ipsec/ipsec_sqn.h b/lib/librte_ipsec/ipsec_sqn.h index 0c2f76a..e884af7 100644 --- a/lib/librte_ipsec/ipsec_sqn.h +++ b/lib/librte_ipsec/ipsec_sqn.h @@ -128,7 +128,8 @@ esn_outb_update_sqn(struct rte_ipsec_sa *sa, uint32_t *num) n = *num; if (SQN_ATOMIC(sa)) - sqn = (uint64_t)rte_atomic64_add_return(&sa->sqn.outb.atom, n); + sqn = __atomic_add_fetch(&sa->sqn.outb.atom, n, + __ATOMIC_RELAXED); else { sqn = sa->sqn.outb.raw + n; sa->sqn.outb.raw = sqn; diff --git a/lib/librte_ipsec/sa.h b/lib/librte_ipsec/sa.h index d22451b..cab9a2e 100644 --- a/lib/librte_ipsec/sa.h +++ b/lib/librte_ipsec/sa.h @@ -120,7 +120,7 @@ struct rte_ipsec_sa { */ union { union { - rte_atomic64_t atom; + uint64_t atom; uint64_t raw; } outb; struct { From patchwork Thu Mar 12 07:44:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 66567 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8F51CA0569; Thu, 12 Mar 2020 08:46:29 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id A66271C066; Thu, 12 Mar 2020 08:45:50 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 0BD1D1C05C; Thu, 12 Mar 2020 08:45:49 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 80379FEC; Thu, 12 Mar 2020 00:45:48 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.150]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C300B3F534; Thu, 12 Mar 2020 00:45:44 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, harry.van.haaren@intel.com, konstantin.ananyev@intel.com, stephen@networkplumber.org, maxime.coquelin@redhat.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, hemant.agrawal@nxp.com, Honnappa.Nagarahalli@arm.com, gavin.hu@arm.com, ruifeng.wang@arm.com, joyce.kong@arm.com, nd@arm.com, stable@dpdk.org Date: Thu, 12 Mar 2020 15:44:26 +0800 Message-Id: <1583999071-22872-6-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1583999071-22872-1-git-send-email-phil.yang@arm.com> References: <1583862551-2049-1-git-send-email-phil.yang@arm.com> <1583999071-22872-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v2 05/10] service: remove rte prefix from static functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Fixes: 3cf5eb1546ed ("service: fix and refactor atomic service accesses") Fixes: 21698354c832 ("service: introduce service cores concept") Cc: stable@dpdk.org Signed-off-by: Phil Yang Reviewed-by: Honnappa Nagarahalli --- lib/librte_eal/common/rte_service.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/lib/librte_eal/common/rte_service.c b/lib/librte_eal/common/rte_service.c index 7e537b8..a691f5d 100644 --- a/lib/librte_eal/common/rte_service.c +++ b/lib/librte_eal/common/rte_service.c @@ -333,7 +333,7 @@ rte_service_runstate_get(uint32_t id) } static inline void -rte_service_runner_do_callback(struct rte_service_spec_impl *s, +service_runner_do_callback(struct rte_service_spec_impl *s, struct core_state *cs, uint32_t service_idx) { void *userdata = s->spec.callback_userdata; @@ -376,10 +376,10 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1)) return -EBUSY; - rte_service_runner_do_callback(s, cs, i); + service_runner_do_callback(s, cs, i); rte_atomic32_clear(&s->execute_lock); } else - rte_service_runner_do_callback(s, cs, i); + service_runner_do_callback(s, cs, i); return 0; } @@ -433,7 +433,7 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) } static int32_t -rte_service_runner_func(void *arg) +service_runner_func(void *arg) { RTE_SET_USED(arg); uint32_t i; @@ -703,7 +703,7 @@ rte_service_lcore_start(uint32_t lcore) */ lcore_states[lcore].runstate = RUNSTATE_RUNNING; - int ret = rte_eal_remote_launch(rte_service_runner_func, 0, lcore); + int ret = rte_eal_remote_launch(service_runner_func, 0, lcore); /* returns -EBUSY if the core is already launched, 0 on success */ return ret; } @@ -782,7 +782,7 @@ rte_service_lcore_attr_get(uint32_t lcore, uint32_t attr_id, } static void -rte_service_dump_one(FILE *f, struct rte_service_spec_impl *s, +service_dump_one(FILE *f, struct rte_service_spec_impl *s, uint64_t all_cycles, uint32_t reset) { /* avoid divide by zero */ @@ -815,7 +815,7 @@ rte_service_attr_reset_all(uint32_t id) SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL); int reset = 1; - rte_service_dump_one(NULL, s, 0, reset); + service_dump_one(NULL, s, 0, reset); return 0; } @@ -873,7 +873,7 @@ rte_service_dump(FILE *f, uint32_t id) SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL); fprintf(f, "Service %s Summary\n", s->spec.name); uint32_t reset = 0; - rte_service_dump_one(f, s, total_cycles, reset); + service_dump_one(f, s, total_cycles, reset); return 0; } @@ -883,7 +883,7 @@ rte_service_dump(FILE *f, uint32_t id) if (!service_valid(i)) continue; uint32_t reset = 0; - rte_service_dump_one(f, &rte_services[i], total_cycles, reset); + service_dump_one(f, &rte_services[i], total_cycles, reset); } fprintf(f, "Service Cores Summary\n"); From patchwork Thu Mar 12 07:44:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 66568 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id AAD9EA0569; Thu, 12 Mar 2020 08:46:44 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 188091C07C; Thu, 12 Mar 2020 08:45:55 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 1D1ED1C00D; Thu, 12 Mar 2020 08:45:53 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A2B4130E; Thu, 12 Mar 2020 00:45:52 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.150]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E9F0D3F534; Thu, 12 Mar 2020 00:45:48 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, harry.van.haaren@intel.com, konstantin.ananyev@intel.com, stephen@networkplumber.org, maxime.coquelin@redhat.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, hemant.agrawal@nxp.com, Honnappa.Nagarahalli@arm.com, gavin.hu@arm.com, ruifeng.wang@arm.com, joyce.kong@arm.com, nd@arm.com, Stable@dpdk.org Date: Thu, 12 Mar 2020 15:44:27 +0800 Message-Id: <1583999071-22872-7-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1583999071-22872-1-git-send-email-phil.yang@arm.com> References: <1583862551-2049-1-git-send-email-phil.yang@arm.com> <1583999071-22872-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v2 06/10] service: remove redundant code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The service id validation is verified in the calling function, remove the redundant code inside the service_update function. Fixes: 21698354c832 ("service: introduce service cores concept") Cc: Stable@dpdk.org Signed-off-by: Phil Yang Reviewed-by: Honnappa Nagarahalli --- lib/librte_eal/common/rte_service.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-) diff --git a/lib/librte_eal/common/rte_service.c b/lib/librte_eal/common/rte_service.c index a691f5d..6990dc2 100644 --- a/lib/librte_eal/common/rte_service.c +++ b/lib/librte_eal/common/rte_service.c @@ -549,21 +549,10 @@ rte_service_start_with_defaults(void) } static int32_t -service_update(struct rte_service_spec *service, uint32_t lcore, +service_update(uint32_t sid, uint32_t lcore, uint32_t *set, uint32_t *enabled) { - uint32_t i; - int32_t sid = -1; - - for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) { - if ((struct rte_service_spec *)&rte_services[i] == service && - service_valid(i)) { - sid = i; - break; - } - } - - if (sid == -1 || lcore >= RTE_MAX_LCORE) + if (lcore >= RTE_MAX_LCORE) return -EINVAL; if (!lcore_states[lcore].is_service_core) @@ -595,19 +584,23 @@ service_update(struct rte_service_spec *service, uint32_t lcore, int32_t rte_service_map_lcore_set(uint32_t id, uint32_t lcore, uint32_t enabled) { - struct rte_service_spec_impl *s; - SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL); + /* validate ID, or return error value */ + if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id)) + return -EINVAL; + uint32_t on = enabled > 0; - return service_update(&s->spec, lcore, &on, 0); + return service_update(id, lcore, &on, 0); } int32_t rte_service_map_lcore_get(uint32_t id, uint32_t lcore) { - struct rte_service_spec_impl *s; - SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL); + /* validate ID, or return error value */ + if (id >= RTE_SERVICE_NUM_MAX || !service_valid(id)) + return -EINVAL; + uint32_t enabled; - int ret = service_update(&s->spec, lcore, 0, &enabled); + int ret = service_update(id, lcore, 0, &enabled); if (ret == 0) return enabled; return ret; From patchwork Thu Mar 12 07:44:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 66569 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2D2C6A0569; Thu, 12 Mar 2020 08:46:58 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1F81F1C065; Thu, 12 Mar 2020 08:45:59 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 146721C05C; Thu, 12 Mar 2020 08:45:58 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9344E30E; Thu, 12 Mar 2020 00:45:57 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.150]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1C68F3F534; Thu, 12 Mar 2020 00:45:52 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, harry.van.haaren@intel.com, konstantin.ananyev@intel.com, stephen@networkplumber.org, maxime.coquelin@redhat.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, hemant.agrawal@nxp.com, Honnappa.Nagarahalli@arm.com, gavin.hu@arm.com, ruifeng.wang@arm.com, joyce.kong@arm.com, nd@arm.com, Honnappa Nagarahalli , stable@dpdk.org Date: Thu, 12 Mar 2020 15:44:28 +0800 Message-Id: <1583999071-22872-8-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1583999071-22872-1-git-send-email-phil.yang@arm.com> References: <1583862551-2049-1-git-send-email-phil.yang@arm.com> <1583999071-22872-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v2 07/10] service: avoid race condition for MT unsafe service X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Honnappa Nagarahalli There has possible that a MT unsafe service might get configured to run on another core while the service is running currently. This might result in the MT unsafe service running on multiple cores simultaneously. Use 'execute_lock' always when the service is MT unsafe. Fixes: e9139a32f6e8 ("service: add function to run on app lcore") Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli Reviewed-by: Phil Yang Reviewed-by: Gavin Hu --- lib/librte_eal/common/rte_service.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/lib/librte_eal/common/rte_service.c b/lib/librte_eal/common/rte_service.c index 6990dc2..b37fc56 100644 --- a/lib/librte_eal/common/rte_service.c +++ b/lib/librte_eal/common/rte_service.c @@ -50,6 +50,10 @@ struct rte_service_spec_impl { uint8_t internal_flags; /* per service statistics */ + /* Indicates how many cores the service is mapped to run on. + * It does not indicate the number of cores the service is running + * on currently. + */ rte_atomic32_t num_mapped_cores; uint64_t calls; uint64_t cycles_spent; @@ -367,12 +371,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] = 1; - /* check do we need cmpset, if MT safe or <= 1 core - * mapped, atomic ops are not required. - */ - const int use_atomics = (service_mt_safe(s) == 0) && - (rte_atomic32_read(&s->num_mapped_cores) > 1); - if (use_atomics) { + if (service_mt_safe(s) == 0) { if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1)) return -EBUSY; From patchwork Thu Mar 12 07:44:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 66570 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id ED108A0569; Thu, 12 Mar 2020 08:47:14 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0FB021C06D; Thu, 12 Mar 2020 08:46:04 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 90B101C0AE; Thu, 12 Mar 2020 08:46:02 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 22978FEC; Thu, 12 Mar 2020 00:46:02 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.150]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 092B53F534; Thu, 12 Mar 2020 00:45:57 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, harry.van.haaren@intel.com, konstantin.ananyev@intel.com, stephen@networkplumber.org, maxime.coquelin@redhat.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, hemant.agrawal@nxp.com, Honnappa.Nagarahalli@arm.com, gavin.hu@arm.com, ruifeng.wang@arm.com, joyce.kong@arm.com, nd@arm.com, Honnappa Nagarahalli , stable@dpdk.org Date: Thu, 12 Mar 2020 15:44:29 +0800 Message-Id: <1583999071-22872-9-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1583999071-22872-1-git-send-email-phil.yang@arm.com> References: <1583862551-2049-1-git-send-email-phil.yang@arm.com> <1583999071-22872-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v2 08/10] service: identify service running on another core correctly X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Honnappa Nagarahalli The logic to identify if the MT unsafe service is running on another core can return -EBUSY spuriously. In such cases, running the service becomes costlier than using atomic operations. Assume that the application passes the right parameters and reduces the number of instructions for all cases. Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli Reviewed-by: Phil Yang Reviewed-by: Gavin Hu --- lib/librte_eal/common/rte_service.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-) diff --git a/lib/librte_eal/common/rte_service.c b/lib/librte_eal/common/rte_service.c index b37fc56..670f5a9 100644 --- a/lib/librte_eal/common/rte_service.c +++ b/lib/librte_eal/common/rte_service.c @@ -357,7 +357,7 @@ service_runner_do_callback(struct rte_service_spec_impl *s, /* Expects the service 's' is valid. */ static int32_t service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, - struct rte_service_spec_impl *s) + struct rte_service_spec_impl *s, uint32_t serialize_mt_unsafe) { if (!s) return -EINVAL; @@ -371,7 +371,7 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] = 1; - if (service_mt_safe(s) == 0) { + if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) { if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1)) return -EBUSY; @@ -409,24 +409,14 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL); - /* Atomically add this core to the mapped cores first, then examine if - * we can run the service. This avoids a race condition between - * checking the value, and atomically adding to the mapped count. + /* Increment num_mapped_cores to indicate that the service + * is running on a core. */ - if (serialize_mt_unsafe) - rte_atomic32_inc(&s->num_mapped_cores); + rte_atomic32_inc(&s->num_mapped_cores); - if (service_mt_safe(s) == 0 && - rte_atomic32_read(&s->num_mapped_cores) > 1) { - if (serialize_mt_unsafe) - rte_atomic32_dec(&s->num_mapped_cores); - return -EBUSY; - } - - int ret = service_run(id, cs, UINT64_MAX, s); + int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe); - if (serialize_mt_unsafe) - rte_atomic32_dec(&s->num_mapped_cores); + rte_atomic32_dec(&s->num_mapped_cores); return ret; } @@ -446,7 +436,7 @@ service_runner_func(void *arg) if (!service_valid(i)) continue; /* return value ignored as no change to code flow */ - service_run(i, cs, service_mask, service_get(i)); + service_run(i, cs, service_mask, service_get(i), 1); } cs->loops++; From patchwork Thu Mar 12 07:44:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 66571 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5590FA0569; Thu, 12 Mar 2020 08:47:32 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 142D41C0B5; Thu, 12 Mar 2020 08:46:08 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 932AC1C0BD for ; Thu, 12 Mar 2020 08:46:06 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1D26030E; Thu, 12 Mar 2020 00:46:06 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.150]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8B8013F534; Thu, 12 Mar 2020 00:46:02 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, harry.van.haaren@intel.com, konstantin.ananyev@intel.com, stephen@networkplumber.org, maxime.coquelin@redhat.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, hemant.agrawal@nxp.com, Honnappa.Nagarahalli@arm.com, gavin.hu@arm.com, ruifeng.wang@arm.com, joyce.kong@arm.com, nd@arm.com Date: Thu, 12 Mar 2020 15:44:30 +0800 Message-Id: <1583999071-22872-10-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1583999071-22872-1-git-send-email-phil.yang@arm.com> References: <1583862551-2049-1-git-send-email-phil.yang@arm.com> <1583999071-22872-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v2 09/10] service: optimize with c11 one-way barrier X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The num_mapped_cores and execute_lock are synchronized with rte_atomic_XX APIs which is a full barrier, DMB, on aarch64. This patch optimized it with c11 atomic one-way barrier. Signed-off-by: Phil Yang Reviewed-by: Ruifeng Wang Reviewed-by: Gavin Hu Reviewed-by: Honnappa Nagarahalli --- lib/librte_eal/common/rte_service.c | 50 ++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 15 deletions(-) diff --git a/lib/librte_eal/common/rte_service.c b/lib/librte_eal/common/rte_service.c index 670f5a9..96a59b6 100644 --- a/lib/librte_eal/common/rte_service.c +++ b/lib/librte_eal/common/rte_service.c @@ -42,7 +42,7 @@ struct rte_service_spec_impl { * running this service callback. When not set, a core may take the * lock and then run the service callback. */ - rte_atomic32_t execute_lock; + uint32_t execute_lock; /* API set/get-able variables */ int8_t app_runstate; @@ -54,7 +54,7 @@ struct rte_service_spec_impl { * It does not indicate the number of cores the service is running * on currently. */ - rte_atomic32_t num_mapped_cores; + int32_t num_mapped_cores; uint64_t calls; uint64_t cycles_spent; } __rte_cache_aligned; @@ -329,7 +329,8 @@ rte_service_runstate_get(uint32_t id) rte_smp_rmb(); int check_disabled = !(s->internal_flags & SERVICE_F_START_CHECK); - int lcore_mapped = (rte_atomic32_read(&s->num_mapped_cores) > 0); + int lcore_mapped = (__atomic_load_n(&s->num_mapped_cores, + __ATOMIC_RELAXED) > 0); return (s->app_runstate == RUNSTATE_RUNNING) && (s->comp_runstate == RUNSTATE_RUNNING) && @@ -372,11 +373,20 @@ service_run(uint32_t i, struct core_state *cs, uint64_t service_mask, cs->service_active_on_lcore[i] = 1; if ((service_mt_safe(s) == 0) && (serialize_mt_unsafe == 1)) { - if (!rte_atomic32_cmpset((uint32_t *)&s->execute_lock, 0, 1)) + uint32_t expected = 0; + /* ACQUIRE ordering here is to prevent the callback + * function from hoisting up before the execute_lock + * setting. + */ + if (!__atomic_compare_exchange_n(&s->execute_lock, &expected, 1, + 0, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED)) return -EBUSY; service_runner_do_callback(s, cs, i); - rte_atomic32_clear(&s->execute_lock); + /* RELEASE ordering here is used to pair with ACQUIRE + * above to achieve lock semantic. + */ + __atomic_store_n(&s->execute_lock, 0, __ATOMIC_RELEASE); } else service_runner_do_callback(s, cs, i); @@ -412,11 +422,11 @@ rte_service_run_iter_on_app_lcore(uint32_t id, uint32_t serialize_mt_unsafe) /* Increment num_mapped_cores to indicate that the service * is running on a core. */ - rte_atomic32_inc(&s->num_mapped_cores); + __atomic_add_fetch(&s->num_mapped_cores, 1, __ATOMIC_ACQUIRE); int ret = service_run(id, cs, UINT64_MAX, s, serialize_mt_unsafe); - rte_atomic32_dec(&s->num_mapped_cores); + __atomic_sub_fetch(&s->num_mapped_cores, 1, __ATOMIC_RELEASE); return ret; } @@ -549,24 +559,32 @@ service_update(uint32_t sid, uint32_t lcore, uint64_t sid_mask = UINT64_C(1) << sid; if (set) { - uint64_t lcore_mapped = lcore_states[lcore].service_mask & - sid_mask; + /* When multiple threads try to update the same lcore + * service concurrently, e.g. set lcore map followed + * by clear lcore map, the unsynchronized service_mask + * values have issues on the num_mapped_cores value + * consistency. So we use ACQUIRE ordering to pair with + * the RELEASE ordering to synchronize the service_mask. + */ + uint64_t lcore_mapped = __atomic_load_n( + &lcore_states[lcore].service_mask, + __ATOMIC_ACQUIRE) & sid_mask; if (*set && !lcore_mapped) { lcore_states[lcore].service_mask |= sid_mask; - rte_atomic32_inc(&rte_services[sid].num_mapped_cores); + __atomic_add_fetch(&rte_services[sid].num_mapped_cores, + 1, __ATOMIC_RELEASE); } if (!*set && lcore_mapped) { lcore_states[lcore].service_mask &= ~(sid_mask); - rte_atomic32_dec(&rte_services[sid].num_mapped_cores); + __atomic_sub_fetch(&rte_services[sid].num_mapped_cores, + 1, __ATOMIC_RELEASE); } } if (enabled) *enabled = !!(lcore_states[lcore].service_mask & (sid_mask)); - rte_smp_wmb(); - return 0; } @@ -622,7 +640,8 @@ rte_service_lcore_reset_all(void) } } for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) - rte_atomic32_set(&rte_services[i].num_mapped_cores, 0); + __atomic_store_n(&rte_services[i].num_mapped_cores, 0, + __ATOMIC_RELAXED); rte_smp_wmb(); @@ -705,7 +724,8 @@ rte_service_lcore_stop(uint32_t lcore) int32_t enabled = service_mask & (UINT64_C(1) << i); int32_t service_running = rte_service_runstate_get(i); int32_t only_core = (1 == - rte_atomic32_read(&rte_services[i].num_mapped_cores)); + __atomic_load_n(&rte_services[i].num_mapped_cores, + __ATOMIC_RELAXED)); /* if the core is mapped, and the service is running, and this * is the only core that is mapped, the service would cease to From patchwork Thu Mar 12 07:44:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 66572 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 267FEA0569; Thu, 12 Mar 2020 08:47:45 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C470D1C0BE; Thu, 12 Mar 2020 08:46:11 +0100 (CET) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 7EF351C0B0 for ; Thu, 12 Mar 2020 08:46:10 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 14CC1FEC; Thu, 12 Mar 2020 00:46:10 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.150]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8F4A13F534; Thu, 12 Mar 2020 00:46:06 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, harry.van.haaren@intel.com, konstantin.ananyev@intel.com, stephen@networkplumber.org, maxime.coquelin@redhat.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, hemant.agrawal@nxp.com, Honnappa.Nagarahalli@arm.com, gavin.hu@arm.com, ruifeng.wang@arm.com, joyce.kong@arm.com, nd@arm.com Date: Thu, 12 Mar 2020 15:44:31 +0800 Message-Id: <1583999071-22872-11-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1583999071-22872-1-git-send-email-phil.yang@arm.com> References: <1583862551-2049-1-git-send-email-phil.yang@arm.com> <1583999071-22872-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v2 10/10] service: relax barriers with C11 atomic operations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" To guarantee the inter-threads visibility of the shareable domain, it uses a lot of rte_smp_r/wmb in the service library. This patch relaxed these barriers for service by using c11 atomic one-way barrier operations. Signed-off-by: Phil Yang Reviewed-by: Ruifeng Wang Reviewed-by: Gavin Hu --- lib/librte_eal/common/rte_service.c | 45 ++++++++++++++++++++----------------- 1 file changed, 25 insertions(+), 20 deletions(-) diff --git a/lib/librte_eal/common/rte_service.c b/lib/librte_eal/common/rte_service.c index 96a59b6..9c02c24 100644 --- a/lib/librte_eal/common/rte_service.c +++ b/lib/librte_eal/common/rte_service.c @@ -176,9 +176,11 @@ rte_service_set_stats_enable(uint32_t id, int32_t enabled) SERVICE_VALID_GET_OR_ERR_RET(id, s, 0); if (enabled) - s->internal_flags |= SERVICE_F_STATS_ENABLED; + __atomic_or_fetch(&s->internal_flags, SERVICE_F_STATS_ENABLED, + __ATOMIC_RELEASE); else - s->internal_flags &= ~(SERVICE_F_STATS_ENABLED); + __atomic_and_fetch(&s->internal_flags, + ~(SERVICE_F_STATS_ENABLED), __ATOMIC_RELEASE); return 0; } @@ -190,9 +192,11 @@ rte_service_set_runstate_mapped_check(uint32_t id, int32_t enabled) SERVICE_VALID_GET_OR_ERR_RET(id, s, 0); if (enabled) - s->internal_flags |= SERVICE_F_START_CHECK; + __atomic_or_fetch(&s->internal_flags, SERVICE_F_START_CHECK, + __ATOMIC_RELEASE); else - s->internal_flags &= ~(SERVICE_F_START_CHECK); + __atomic_and_fetch(&s->internal_flags, ~(SERVICE_F_START_CHECK), + __ATOMIC_RELEASE); return 0; } @@ -261,8 +265,8 @@ rte_service_component_register(const struct rte_service_spec *spec, s->spec = *spec; s->internal_flags |= SERVICE_F_REGISTERED | SERVICE_F_START_CHECK; - rte_smp_wmb(); - rte_service_count++; + /* make sure the counter update after the state change. */ + __atomic_add_fetch(&rte_service_count, 1, __ATOMIC_RELEASE); if (id_ptr) *id_ptr = free_slot; @@ -278,9 +282,10 @@ rte_service_component_unregister(uint32_t id) SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL); rte_service_count--; - rte_smp_wmb(); - s->internal_flags &= ~(SERVICE_F_REGISTERED); + /* make sure the counter update before the state change. */ + __atomic_and_fetch(&s->internal_flags, ~(SERVICE_F_REGISTERED), + __ATOMIC_RELEASE); /* clear the run-bit in all cores */ for (i = 0; i < RTE_MAX_LCORE; i++) @@ -298,11 +303,12 @@ rte_service_component_runstate_set(uint32_t id, uint32_t runstate) SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL); if (runstate) - s->comp_runstate = RUNSTATE_RUNNING; + __atomic_store_n(&s->comp_runstate, RUNSTATE_RUNNING, + __ATOMIC_RELEASE); else - s->comp_runstate = RUNSTATE_STOPPED; + __atomic_store_n(&s->comp_runstate, RUNSTATE_STOPPED, + __ATOMIC_RELEASE); - rte_smp_wmb(); return 0; } @@ -313,11 +319,12 @@ rte_service_runstate_set(uint32_t id, uint32_t runstate) SERVICE_VALID_GET_OR_ERR_RET(id, s, -EINVAL); if (runstate) - s->app_runstate = RUNSTATE_RUNNING; + __atomic_store_n(&s->app_runstate, RUNSTATE_RUNNING, + __ATOMIC_RELEASE); else - s->app_runstate = RUNSTATE_STOPPED; + __atomic_store_n(&s->app_runstate, RUNSTATE_STOPPED, + __ATOMIC_RELEASE); - rte_smp_wmb(); return 0; } @@ -439,7 +446,8 @@ service_runner_func(void *arg) const int lcore = rte_lcore_id(); struct core_state *cs = &lcore_states[lcore]; - while (lcore_states[lcore].runstate == RUNSTATE_RUNNING) { + while (__atomic_load_n(&cs->runstate, + __ATOMIC_ACQUIRE) == RUNSTATE_RUNNING) { const uint64_t service_mask = cs->service_mask; for (i = 0; i < RTE_SERVICE_NUM_MAX; i++) { @@ -450,8 +458,6 @@ service_runner_func(void *arg) } cs->loops++; - - rte_smp_rmb(); } lcore_config[lcore].state = WAIT; @@ -660,9 +666,8 @@ rte_service_lcore_add(uint32_t lcore) /* ensure that after adding a core the mask and state are defaults */ lcore_states[lcore].service_mask = 0; - lcore_states[lcore].runstate = RUNSTATE_STOPPED; - - rte_smp_wmb(); + __atomic_store_n(&lcore_states[lcore].runstate, RUNSTATE_STOPPED, + __ATOMIC_RELEASE); return rte_eal_wait_lcore(lcore); }