From patchwork Fri Jul 17 05:08:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 74277 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C1B47A0528; Fri, 17 Jul 2020 07:09:23 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 98B6E1BED2; Fri, 17 Jul 2020 07:09:20 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id F3F2F1BED1 for ; Fri, 17 Jul 2020 07:09:18 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5ED7D12FC; Thu, 16 Jul 2020 22:09:18 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.108.167]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2E5343F66E; Thu, 16 Jul 2020 22:09:13 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, john.mcnamara@intel.com, Honnappa.Nagarahalli@arm.com, drc@linux.vnet.ibm.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, konstantin.ananyev@intel.com, Ola.Liljedahl@arm.com, bruce.richardson@intel.com, Ruifeng.Wang@arm.com, nd@arm.com, Honnappa Nagarahalli , Marko Kovacevic Date: Fri, 17 Jul 2020 13:08:37 +0800 Message-Id: <1594962519-20619-2-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1594962519-20619-1-git-send-email-phil.yang@arm.com> References: <1594875225-5850-1-git-send-email-phil.yang@arm.com> <1594962519-20619-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v9 1/3] doc: add optimizations using C11 atomic builtins X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add information about possible optimizations using C11 atomic builtins. Signed-off-by: Phil Yang Signed-off-by: Honnappa Nagarahalli Reviewed-by: Honnappa Nagarahalli --- doc/guides/prog_guide/writing_efficient_code.rst | 59 +++++++++++++++++++++++- 1 file changed, 58 insertions(+), 1 deletion(-) diff --git a/doc/guides/prog_guide/writing_efficient_code.rst b/doc/guides/prog_guide/writing_efficient_code.rst index 849f63e..8dd5439 100644 --- a/doc/guides/prog_guide/writing_efficient_code.rst +++ b/doc/guides/prog_guide/writing_efficient_code.rst @@ -167,7 +167,13 @@ but with the added cost of lower throughput. Locks and Atomic Operations --------------------------- -Atomic operations imply a lock prefix before the instruction, +This section describes some key considerations when using locks and atomic +operations in the DPDK environment. + +Locks +~~~~~ + +On x86, atomic operations imply a lock prefix before the instruction, causing the processor's LOCK# signal to be asserted during execution of the following instruction. This has a big impact on performance in a multicore environment. @@ -176,6 +182,57 @@ It can often be replaced by other solutions like per-lcore variables. Also, some locking techniques are more efficient than others. For instance, the Read-Copy-Update (RCU) algorithm can frequently replace simple rwlocks. +Atomic Operations: Use C11 Atomic Builtins +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +DPDK generic rte_atomic operations are implemented by __sync builtins. These +__sync builtins result in full barriers on aarch64, which are unnecessary +in many use cases. They can be replaced by __atomic builtins that conform to +the C11 memory model and provide finer memory order control. + +So replacing the rte_atomic operations with __atomic builtins might improve +performance for aarch64 machines. + +Some typical optimization cases are listed below: + +Atomicity +^^^^^^^^^ + +Some use cases require atomicity alone, the ordering of the memory operations +does not matter. For example, the packet statistics counters need to be +incremented atomically but do not need any particular memory ordering. +So, RELAXED memory ordering is sufficient. + +One-way Barrier +^^^^^^^^^^^^^^^ + +Some use cases allow for memory reordering in one way while requiring memory +ordering in the other direction. + +For example, the memory operations before the spinlock lock are allowed to +move to the critical section, but the memory operations in the critical section +are not allowed to move above the lock. In this case, the full memory barrier +in the compare-and-swap operation can be replaced with ACQUIRE memory order. +On the other hand, the memory operations after the spinlock unlock are allowed +to move to the critical section, but the memory operations in the critical +section are not allowed to move below the unlock. So the full barrier in the +store operation can use RELEASE memory order. + +Reader-Writer Concurrency +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Lock-free reader-writer concurrency is one of the common use cases in DPDK. + +The payload or the data that the writer wants to communicate to the reader, +can be written with RELAXED memory order. However, the guard variable should +be written with RELEASE memory order. This ensures that the store to guard +variable is observable only after the store to payload is observable. + +Correspondingly, on the reader side, the guard variable should be read +with ACQUIRE memory order. The payload or the data the writer communicated, +can be read with RELAXED memory order. This ensures that, if the store to +guard variable is observable, the store to payload is also observable. + Coding Considerations --------------------- From patchwork Fri Jul 17 05:08:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 74278 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7C81FA0528; Fri, 17 Jul 2020 07:09:32 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D53761BED9; Fri, 17 Jul 2020 07:09:24 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 31B501BED8 for ; Fri, 17 Jul 2020 07:09:23 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AAEB61045; Thu, 16 Jul 2020 22:09:22 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.108.167]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 09DD03F66E; Thu, 16 Jul 2020 22:09:18 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, john.mcnamara@intel.com, Honnappa.Nagarahalli@arm.com, drc@linux.vnet.ibm.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, konstantin.ananyev@intel.com, Ola.Liljedahl@arm.com, bruce.richardson@intel.com, Ruifeng.Wang@arm.com, nd@arm.com Date: Fri, 17 Jul 2020 13:08:38 +0800 Message-Id: <1594962519-20619-3-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1594962519-20619-1-git-send-email-phil.yang@arm.com> References: <1594875225-5850-1-git-send-email-phil.yang@arm.com> <1594962519-20619-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v9 2/3] devtools: prevent use of rte atomic APIs in future patches X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" In order to deprecate the rte_atomic and rte_smp barrier APIs, prevent the patches from using these APIs and __sync builtins in new code. On x86 the __atomic_thread_fence(__ATOMIC_SEQ_CST) is quite expensive for SMP case. Flag the new code which use __atomic_thread_fence API. Signed-off-by: Phil Yang Reviewed-by: Ruifeng Wang --- devtools/checkpatches.sh | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/devtools/checkpatches.sh b/devtools/checkpatches.sh index 58021aa..990a4d0 100755 --- a/devtools/checkpatches.sh +++ b/devtools/checkpatches.sh @@ -77,6 +77,41 @@ check_forbidden_additions() { # -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ "$1" || res=1 + # refrain from new additions of 16/32/64 bits rte_atomic_xxx() + # multiple folders and expressions are separated by spaces + awk -v FOLDERS="lib drivers app examples" \ + -v EXPRESSIONS="rte_atomic[0-9][0-9]_.*\\\(" \ + -v RET_ON_FAIL=1 \ + -v MESSAGE='Using rte_atomicNN_xxx' \ + -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ + "$1" || res=1 + + # refrain from new additions of rte_smp_XXmb() + # multiple folders and expressions are separated by spaces + awk -v FOLDERS="lib drivers app examples" \ + -v EXPRESSIONS="rte_smp_(r|w)?mb\\\(" \ + -v RET_ON_FAIL=1 \ + -v MESSAGE='Using rte_smp_[r/w]mb' \ + -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ + "$1" || res=1 + + # refrain from using compiler __sync builtins + awk -v FOLDERS="lib drivers app examples" \ + -v EXPRESSIONS="__sync_.*\\\(" \ + -v RET_ON_FAIL=1 \ + -v MESSAGE='Using __sync_xxx builtins' \ + -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ + "$1" || res=1 + + # refrain from using compiler __atomic_thread_fence() + # It should be avoided on x86 for SMP case. + awk -v FOLDERS="lib drivers app examples" \ + -v EXPRESSIONS="__atomic_thread_fence\\\(" \ + -v RET_ON_FAIL=1 \ + -v MESSAGE='Using __atomic_thread_fence' \ + -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ + "$1" || res=1 + # svg figures must be included with wildcard extension # because of png conversion for pdf docs awk -v FOLDERS='doc' \ From patchwork Fri Jul 17 05:08:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 74279 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id E697FA0528; Fri, 17 Jul 2020 07:09:43 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 23EAE1BEE1; Fri, 17 Jul 2020 07:09:29 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 178991BEE1 for ; Fri, 17 Jul 2020 07:09:28 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 992AD1045; Thu, 16 Jul 2020 22:09:27 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.108.167]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5F7ED3F66E; Thu, 16 Jul 2020 22:09:23 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, john.mcnamara@intel.com, Honnappa.Nagarahalli@arm.com, drc@linux.vnet.ibm.com, dev@dpdk.org Cc: david.marchand@redhat.com, jerinj@marvell.com, konstantin.ananyev@intel.com, Ola.Liljedahl@arm.com, bruce.richardson@intel.com, Ruifeng.Wang@arm.com, nd@arm.com, Jan Viktorin , Ruifeng Wang Date: Fri, 17 Jul 2020 13:08:39 +0800 Message-Id: <1594962519-20619-4-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1594962519-20619-1-git-send-email-phil.yang@arm.com> References: <1594875225-5850-1-git-send-email-phil.yang@arm.com> <1594962519-20619-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v9 3/3] eal/atomic: add wrapper for C11 atomic thread fence X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Provide a wrapper for __atomic_thread_fence builtins to support optimized code for __ATOMIC_SEQ_CST memory order for x86 platforms. Suggested-by: Honnappa Nagarahalli Signed-off-by: Phil Yang Reviewed-by: Ola Liljedahl Acked-by: Konstantin Ananyev --- lib/librte_eal/arm/include/rte_atomic_32.h | 6 ++++++ lib/librte_eal/arm/include/rte_atomic_64.h | 6 ++++++ lib/librte_eal/include/generic/rte_atomic.h | 6 ++++++ lib/librte_eal/ppc/include/rte_atomic.h | 6 ++++++ lib/librte_eal/x86/include/rte_atomic.h | 17 +++++++++++++++++ 5 files changed, 41 insertions(+) diff --git a/lib/librte_eal/arm/include/rte_atomic_32.h b/lib/librte_eal/arm/include/rte_atomic_32.h index 7dc0d06..dbe7cc6 100644 --- a/lib/librte_eal/arm/include/rte_atomic_32.h +++ b/lib/librte_eal/arm/include/rte_atomic_32.h @@ -37,6 +37,12 @@ extern "C" { #define rte_cio_rmb() rte_rmb() +static __rte_always_inline void +rte_atomic_thread_fence(int mo) +{ + __atomic_thread_fence(mo); +} + #ifdef __cplusplus } #endif diff --git a/lib/librte_eal/arm/include/rte_atomic_64.h b/lib/librte_eal/arm/include/rte_atomic_64.h index e42f69e..bf885ad 100644 --- a/lib/librte_eal/arm/include/rte_atomic_64.h +++ b/lib/librte_eal/arm/include/rte_atomic_64.h @@ -41,6 +41,12 @@ extern "C" { #define rte_cio_rmb() rte_rmb() +static __rte_always_inline void +rte_atomic_thread_fence(int mo) +{ + __atomic_thread_fence(mo); +} + /*------------------------ 128 bit atomic operations -------------------------*/ #if defined(__ARM_FEATURE_ATOMICS) || defined(RTE_ARM_FEATURE_ATOMICS) diff --git a/lib/librte_eal/include/generic/rte_atomic.h b/lib/librte_eal/include/generic/rte_atomic.h index e6ab15a..5b941db 100644 --- a/lib/librte_eal/include/generic/rte_atomic.h +++ b/lib/librte_eal/include/generic/rte_atomic.h @@ -158,6 +158,12 @@ static inline void rte_cio_rmb(void); asm volatile ("" : : : "memory"); \ } while(0) +/** + * Synchronization fence between threads based on the specified + * memory order. + */ +static inline void rte_atomic_thread_fence(int mo); + /*------------------------- 16 bit atomic operations -------------------------*/ /** diff --git a/lib/librte_eal/ppc/include/rte_atomic.h b/lib/librte_eal/ppc/include/rte_atomic.h index 7e3e131..91c5f30 100644 --- a/lib/librte_eal/ppc/include/rte_atomic.h +++ b/lib/librte_eal/ppc/include/rte_atomic.h @@ -40,6 +40,12 @@ extern "C" { #define rte_cio_rmb() rte_rmb() +static __rte_always_inline void +rte_atomic_thread_fence(int mo) +{ + __atomic_thread_fence(mo); +} + /*------------------------- 16 bit atomic operations -------------------------*/ /* To be compatible with Power7, use GCC built-in functions for 16 bit * operations */ diff --git a/lib/librte_eal/x86/include/rte_atomic.h b/lib/librte_eal/x86/include/rte_atomic.h index b9dcd30..bd256e7 100644 --- a/lib/librte_eal/x86/include/rte_atomic.h +++ b/lib/librte_eal/x86/include/rte_atomic.h @@ -83,6 +83,23 @@ rte_smp_mb(void) #define rte_cio_rmb() rte_compiler_barrier() +/** + * Synchronization fence between threads based on the specified + * memory order. + * + * On x86 the __atomic_thread_fence(__ATOMIC_SEQ_CST) generates + * full 'mfence' which is quite expensive. The optimized + * implementation of rte_smp_mb is used instead. + */ +static __rte_always_inline void +rte_atomic_thread_fence(int mo) +{ + if (mo == __ATOMIC_SEQ_CST) + rte_smp_mb(); + else + __atomic_thread_fence(mo); +} + /*------------------------- 16 bit atomic operations -------------------------*/ #ifndef RTE_FORCE_INTRINSICS