From patchwork Tue May 12 08:03:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 70098 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1AEABA04A2; Tue, 12 May 2020 10:04:10 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DEE3B1C06A; Tue, 12 May 2020 10:04:09 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 60DEE1C06A for ; Tue, 12 May 2020 10:04:08 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D9A7431B; Tue, 12 May 2020 01:04:07 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.147]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 40C0E3F305; Tue, 12 May 2020 01:03:57 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, dev@dpdk.org Cc: bruce.richardson@intel.com, ferruh.yigit@intel.com, hemant.agrawal@nxp.com, honnappa.nagarahalli@arm.com, jerinj@marvell.com, ktraynor@redhat.com, konstantin.ananyev@intel.com, maxime.coquelin@redhat.com, olivier.matz@6wind.com, stephen@networkplumber.org, mb@smartsharesystems.com, mattias.ronnblom@ericsson.com, harry.van.haaren@intel.com, erik.g.carrillo@intel.com, phil.yang@arm.com, nd@arm.com Date: Tue, 12 May 2020 16:03:03 +0800 Message-Id: <1589270586-4480-2-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1589270586-4480-1-git-send-email-phil.yang@arm.com> References: <1584407863-774-1-git-send-email-phil.yang@arm.com> <1589270586-4480-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v4 1/4] doc: add generic atomic deprecation section X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add deprecating the generic rte_atomic_xx APIs to c11 atomic built-ins guide and examples. Signed-off-by: Phil Yang Signed-off-by: Honnappa Nagarahalli --- doc/guides/prog_guide/writing_efficient_code.rst | 139 ++++++++++++++++++++++- 1 file changed, 138 insertions(+), 1 deletion(-) diff --git a/doc/guides/prog_guide/writing_efficient_code.rst b/doc/guides/prog_guide/writing_efficient_code.rst index 849f63e..3bd2601 100644 --- a/doc/guides/prog_guide/writing_efficient_code.rst +++ b/doc/guides/prog_guide/writing_efficient_code.rst @@ -167,7 +167,13 @@ but with the added cost of lower throughput. Locks and Atomic Operations --------------------------- -Atomic operations imply a lock prefix before the instruction, +This section describes some key considerations when using locks and atomic +operations in the DPDK environment. + +Locks +~~~~~ + +On x86, atomic operations imply a lock prefix before the instruction, causing the processor's LOCK# signal to be asserted during execution of the following instruction. This has a big impact on performance in a multicore environment. @@ -176,6 +182,137 @@ It can often be replaced by other solutions like per-lcore variables. Also, some locking techniques are more efficient than others. For instance, the Read-Copy-Update (RCU) algorithm can frequently replace simple rwlocks. +Atomic Operations: Use C11 Atomic Built-ins +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +DPDK `generic rte_atomic `_ operations are +implemented by `__sync built-ins `_. +These __sync built-ins result in full barriers on aarch64, which are unnecessary +in many use cases. They can be replaced by `__atomic built-ins `_ that +conform to the C11 memory model and provide finer memory order control. + +So replacing the rte_atomic operations with __atomic built-ins might improve +performance for aarch64 machines. `More details `_. + +Some typical optimization cases are listed below: + +Atomicity +^^^^^^^^^ + +Some use cases require atomicity alone, the ordering of the memory operations +does not matter. For example the packets statistics in the `vhost `_ example application. + +It just updates the number of transmitted packets, no subsequent logic depends +on these counters. So the RELAXED memory ordering is sufficient: + +.. code-block:: c + + static __rte_always_inline void + virtio_xmit(struct vhost_dev *dst_vdev, struct vhost_dev *src_vdev, + struct rte_mbuf *m) + { + ... + ... + if (enable_stats) { + __atomic_add_fetch(&dst_vdev->stats.rx_total_atomic, 1, __ATOMIC_RELAXED); + __atomic_add_fetch(&dst_vdev->stats.rx_atomic, ret, __ATOMIC_RELAXED); + ... + } + } + +One-way Barrier +^^^^^^^^^^^^^^^ + +Some use cases allow for memory reordering in one way while requiring memory +ordering in the other direction. + +For example, the memory operations before the `lock `_ can move to the +critical section, but the memory operations in the critical section cannot move +above the lock. In this case, the full memory barrier in the CAS operation can +be replaced to ACQUIRE. On the other hand, the memory operations after the +`unlock `_ can move to the critical section, but the memory operations in the +critical section cannot move below the unlock. So the full barrier in the STORE +operation can be replaced with RELEASE. + +Reader-Writer Concurrency +^^^^^^^^^^^^^^^^^^^^^^^^^ +Lock-free reader-writer concurrency is one of the common use cases in DPDK. + +The payload or the data that the writer wants to communicate to the reader, +can be written with RELAXED memory order. However, the guard variable should +be written with RELEASE memory order. This ensures that the store to guard +variable is observable only after the store to payload is observable. +Refer to `rte_hash insert `_ for an example. + +.. code-block:: c + + static inline int32_t + rte_hash_cuckoo_insert_mw(const struct rte_hash *h, + ... + int32_t *ret_val) + { + ... + ... + + /* Insert new entry if there is room in the primary + * bucket. + */ + for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { + /* Check if slot is available */ + if (likely(prim_bkt->key_idx[i] == EMPTY_SLOT)) { + prim_bkt->sig_current[i] = sig; + /* Store to signature and key should not + * leak after the store to key_idx. i.e. + * key_idx is the guard variable for signature + * and key. + */ + __atomic_store_n(&prim_bkt->key_idx[i], + new_idx, + __ATOMIC_RELEASE); + break; + } + } + + ... + } + +Correspondingly, on the reader side, the guard variable should be read +with ACQUIRE memory order. The payload or the data the writer communicated, +can be read with RELAXED memory order. This ensures that, if the store to +guard variable is observable, the store to payload is also observable. Refer to `rte_hash lookup `_ for an example. + +.. code-block:: c + + static inline int32_t + search_one_bucket_lf(const struct rte_hash *h, const void *key, uint16_t sig, + void **data, const struct rte_hash_bucket *bkt) + { + ... + + for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { + .... + if (bkt->sig_current[i] == sig) { + key_idx = __atomic_load_n(&bkt->key_idx[i], + __ATOMIC_ACQUIRE); + if (key_idx != EMPTY_SLOT) { + k = (struct rte_hash_key *) ((char *)keys + + key_idx * h->key_entry_size); + + if (rte_hash_cmp_eq(key, k->key, h) == 0) { + if (data != NULL) { + *data = __atomic_load_n(&k->pdata, + __ATOMIC_ACQUIRE); + } + + /* + * Return index where key is stored, + * subtracting the first dummy index + */ + return key_idx - 1; + } + ... + } + Coding Considerations --------------------- From patchwork Tue May 12 08:03:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 70099 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 73349A04A2; Tue, 12 May 2020 10:04:20 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 5E0C01C18E; Tue, 12 May 2020 10:04:20 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 7D1B01C118 for ; Tue, 12 May 2020 10:04:19 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E32E11FB; Tue, 12 May 2020 01:04:18 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.147]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 38A543F305; Tue, 12 May 2020 01:04:08 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, dev@dpdk.org Cc: bruce.richardson@intel.com, ferruh.yigit@intel.com, hemant.agrawal@nxp.com, honnappa.nagarahalli@arm.com, jerinj@marvell.com, ktraynor@redhat.com, konstantin.ananyev@intel.com, maxime.coquelin@redhat.com, olivier.matz@6wind.com, stephen@networkplumber.org, mb@smartsharesystems.com, mattias.ronnblom@ericsson.com, harry.van.haaren@intel.com, erik.g.carrillo@intel.com, phil.yang@arm.com, nd@arm.com Date: Tue, 12 May 2020 16:03:04 +0800 Message-Id: <1589270586-4480-3-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1589270586-4480-1-git-send-email-phil.yang@arm.com> References: <1584407863-774-1-git-send-email-phil.yang@arm.com> <1589270586-4480-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v4 2/4] maintainers: claim maintainers of c11 atomics code X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add the maintainership of c11 atomics code. Signed-off-by: Phil Yang Reviewed-by: Honnappa Nagarahalli --- MAINTAINERS | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 6a14622..4435ae5 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -266,6 +266,10 @@ F: lib/librte_eal/include/rte_random.h F: lib/librte_eal/common/rte_random.c F: app/test/test_rand_perf.c +C11 Code Maintainer +M: Honnappa Nagarahalli +M: David Christensen + ARM v7 M: Jan Viktorin M: Ruifeng Wang From patchwork Tue May 12 08:03:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 70100 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 48242A04A2; Tue, 12 May 2020 10:04:30 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id A45D51C19C; Tue, 12 May 2020 10:04:28 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 642611C195 for ; Tue, 12 May 2020 10:04:27 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E2F7D1FB; Tue, 12 May 2020 01:04:26 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.147]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 83F6C3F305; Tue, 12 May 2020 01:04:19 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, dev@dpdk.org Cc: bruce.richardson@intel.com, ferruh.yigit@intel.com, hemant.agrawal@nxp.com, honnappa.nagarahalli@arm.com, jerinj@marvell.com, ktraynor@redhat.com, konstantin.ananyev@intel.com, maxime.coquelin@redhat.com, olivier.matz@6wind.com, stephen@networkplumber.org, mb@smartsharesystems.com, mattias.ronnblom@ericsson.com, harry.van.haaren@intel.com, erik.g.carrillo@intel.com, phil.yang@arm.com, nd@arm.com Date: Tue, 12 May 2020 16:03:05 +0800 Message-Id: <1589270586-4480-4-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1589270586-4480-1-git-send-email-phil.yang@arm.com> References: <1584407863-774-1-git-send-email-phil.yang@arm.com> <1589270586-4480-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v4 3/4] devtools: prevent use of rte atomic APIs in future patches X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" In order to deprecate the rte_atomic APIs, prevent the patches from using rte_atomic APIs in the converted modules and compilers __sync built-ins in all modules. The converted modules: lib/librte_distributor lib/librte_hash lib/librte_kni lib/librte_lpm lib/librte_rcu lib/librte_ring lib/librte_stack lib/librte_vhost lib/librte_timer lib/librte_ipsec drivers/event/octeontx drivers/event/octeontx2 drivers/event/opdl drivers/net/bnx2x drivers/net/hinic drivers/net/hns3 drivers/net/memif drivers/net/thunderx drivers/net/virtio examples/l2fwd-event Signed-off-by: Phil Yang Reviewed-by: Honnappa Nagarahalli --- devtools/checkpatches.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/devtools/checkpatches.sh b/devtools/checkpatches.sh index 42b833e..002586d 100755 --- a/devtools/checkpatches.sh +++ b/devtools/checkpatches.sh @@ -69,6 +69,29 @@ check_forbidden_additions() { # -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ "$1" || res=1 + # refrain from new additions of 16/32/64 bits rte_atomic_xxx() + # multiple folders and expressions are separated by spaces + awk -v FOLDERS="lib/librte_distributor lib/librte_hash lib/librte_kni + lib/librte_lpm lib/librte_rcu lib/librte_ring + lib/librte_stack lib/librte_vhost drivers/event/octeontx + drivers/event/octeontx2 drivers/event/opdl + drivers/net/bnx2x drivers/net/hinic drivers/net/hns3 + drivers/net/memif drivers/net/thunderx + drivers/net/virtio examples/l2fwd-event" \ + -v EXPRESSIONS="rte_atomic[0-9][0-9]_.*\\\(" \ + -v RET_ON_FAIL=1 \ + -v MESSAGE='Use of rte_atomicNN_xxx APIs not allowed, use rte_atomic_xxx APIs' \ + -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ + "$1" || res=1 + + # refrain from using compiler __sync built-ins + awk -v FOLDERS="lib drivers app examples" \ + -v EXPRESSIONS="__sync_.*\\\(" \ + -v RET_ON_FAIL=1 \ + -v MESSAGE='Use of __sync_xxx built-ins not allowed, use rte_atomic_xxx APIs' \ + -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \ + "$1" || res=1 + # svg figures must be included with wildcard extension # because of png conversion for pdf docs awk -v FOLDERS='doc' \ From patchwork Tue May 12 08:03:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phil Yang X-Patchwork-Id: 70101 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id B29FFA04A2; Tue, 12 May 2020 10:04:41 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E3B4B1C1C6; Tue, 12 May 2020 10:04:38 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 216921C1C3 for ; Tue, 12 May 2020 10:04:37 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9F31D1FB; Tue, 12 May 2020 01:04:36 -0700 (PDT) Received: from phil-VirtualBox.shanghai.arm.com (phil-VirtualBox.shanghai.arm.com [10.169.109.147]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D79883F305; Tue, 12 May 2020 01:04:27 -0700 (PDT) From: Phil Yang To: thomas@monjalon.net, dev@dpdk.org Cc: bruce.richardson@intel.com, ferruh.yigit@intel.com, hemant.agrawal@nxp.com, honnappa.nagarahalli@arm.com, jerinj@marvell.com, ktraynor@redhat.com, konstantin.ananyev@intel.com, maxime.coquelin@redhat.com, olivier.matz@6wind.com, stephen@networkplumber.org, mb@smartsharesystems.com, mattias.ronnblom@ericsson.com, harry.van.haaren@intel.com, erik.g.carrillo@intel.com, phil.yang@arm.com, nd@arm.com Date: Tue, 12 May 2020 16:03:06 +0800 Message-Id: <1589270586-4480-5-git-send-email-phil.yang@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1589270586-4480-1-git-send-email-phil.yang@arm.com> References: <1584407863-774-1-git-send-email-phil.yang@arm.com> <1589270586-4480-1-git-send-email-phil.yang@arm.com> Subject: [dpdk-dev] [PATCH v4 4/4] eal/atomic: add wrapper for c11 atomics X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Wraps up compiler c11 atomic built-ins with explicit memory ordering parameter. Signed-off-by: Phil Yang --- lib/librte_eal/include/generic/rte_atomic_c11.h | 139 ++++++++++++++++++++++++ lib/librte_eal/include/meson.build | 1 + 2 files changed, 140 insertions(+) create mode 100644 lib/librte_eal/include/generic/rte_atomic_c11.h diff --git a/lib/librte_eal/include/generic/rte_atomic_c11.h b/lib/librte_eal/include/generic/rte_atomic_c11.h new file mode 100644 index 0000000..20490f4 --- /dev/null +++ b/lib/librte_eal/include/generic/rte_atomic_c11.h @@ -0,0 +1,139 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2020 Arm Limited + */ + +#ifndef _RTE_ATOMIC_C11_H_ +#define _RTE_ATOMIC_C11_H_ + +#include + +/** + * @file + * c11 atomic operations + * + * This file wraps up compiler (GCC) c11 atomic built-ins. + * https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html + */ + +#define memory_order_relaxed __ATOMIC_RELAXED +#define memory_order_consume __ATOMIC_CONSUME +#define memory_order_acquire __ATOMIC_ACQUIRE +#define memory_order_release __ATOMIC_RELEASE +#define memory_order_acq_rel __ATOMIC_ACQ_REL +#define memory_order_seq_cst __ATOMIC_SEQ_CST + +/* Generic atomic load. + * It returns the contents of *PTR. + * + * The valid memory order variants are: + * memory_order_relaxed + * memory_order_consume + * memory_order_acquire + * memory_order_seq_cst + */ +#define rte_atomic_load(PTR, MO) \ + (__extension__ ({ \ + typeof(PTR) _ptr = (PTR); \ + typeof(*_ptr) _ret; \ + __atomic_load(_ptr, &_ret, (MO)); \ + _ret; \ + })) + +/* Generic atomic store. + * It stores the value of VAL into *PTR. + * + * The valid memory order variants are: + * memory_order_relaxed + * memory_order_release + * memory_order_seq_cst + */ +#define rte_atomic_store(PTR, VAL, MO) \ + (__extension__ ({ \ + typeof(PTR) _ptr = (PTR); \ + typeof(*_ptr) _val = (VAL); \ + __atomic_store(_ptr, &_val, (MO)); \ + })) + +/* Generic atomic exchange. + * It stores the value of VAL into *PTR. + * It returns the original value of *PTR. + * + * The valid memory order variants are: + * memory_order_relaxed + * memory_order_acquire + * memory_order_release + * memory_order_acq_rel + * memory_order_seq_cst + */ +#define rte_atomic_exchange(PTR, VAL, MO) \ + (__extension__ ({ \ + typeof(PTR) _ptr = (PTR); \ + typeof(*_ptr) _val = (VAL); \ + typeof(*_ptr) _ret; \ + __atomic_exchange(_ptr, &_val, &_ret, (MO)); \ + _ret; \ + })) + +/* Generic atomic compare and exchange. + * It compares the contents of *PTR with the contents of *EXP. + * If equal, the operation is a read-modify-write operation that + * writes DES into *PTR. + * If they are not equal, the operation is a read and the current + * contents of *PTR are written into *EXP. + * + * The weak compare_exchange may fail spuriously and the strong + * variation will never fails spuriously. + * + * If DES is written into *PTR then true is returned and memory is + * affected according to the memory order specified by SUC_MO. + * There are no restrictions on what memory order can be used here. + * + * Otherwise, false is returned and memory is affected according to + * FAIL_MO. This memory order cannot be memory_order_release nor + * memory_order_acq_rel. It also cannot be a stronger order than that + * specified by SUC_MO. + */ +#define rte_atomic_compare_exchange_weak(PTR, EXP, DES, SUC_MO, FAIL_MO) \ + (__extension__ ({ \ + typeof(PTR) _ptr = (PTR); \ + typeof(*_ptr) _des = (DES); \ + __atomic_compare_exchange(_ptr, (EXP), &_des, 1, \ + (SUC_MO), (FAIL_MO)); \ + })) + +#define rte_atomic_compare_exchange_strong(PTR, EXP, DES, SUC_MO, FAIL_MO) \ + (__extension__ ({ \ + typeof(PTR) _ptr = (PTR); \ + typeof(*_ptr) _des = (DES); \ + __atomic_compare_exchange(_ptr, (EXP), &_des, 0, \ + (SUC_MO), (FAIL_MO)); \ + })) + +#define rte_atomic_fetch_add(PTR, VAL, MO) \ + __atomic_fetch_add((PTR), (VAL), (MO)) +#define rte_atomic_fetch_sub(PTR, VAL, MO) \ + __atomic_fetch_sub((PTR), (VAL), (MO)) +#define rte_atomic_fetch_or(PTR, VAL, MO) \ + __atomic_fetch_or((PTR), (VAL), (MO)) +#define rte_atomic_fetch_xor(PTR, VAL, MO) \ + __atomic_fetch_xor((PTR), (VAL), (MO)) +#define rte_atomic_fetch_and(PTR, VAL, MO) \ + __atomic_fetch_and((PTR), (VAL), (MO)) + +#define rte_atomic_add_fetch(PTR, VAL, MO) \ + __atomic_add_fetch((PTR), (VAL), (MO)) +#define rte_atomic_sub_fetch(PTR, VAL, MO) \ + __atomic_sub_fetch((PTR), (VAL), (MO)) +#define rte_atomic_or_fetch(PTR, VAL, MO) \ + __atomic_or_fetch((PTR), (VAL), (MO)) +#define rte_atomic_xor_fetch(PTR, VAL, MO) \ + __atomic_xor_fetch((PTR), (VAL), (MO)) +#define rte_atomic_and_fetch(PTR, VAL, MO) \ + __atomic_and_fetch((PTR), (VAL), (MO)) + +/* Synchronization fence between threads based on + * the specified memory order. + */ +#define rte_atomic_thread_fence(MO) __atomic_thread_fence((MO)) + +#endif /* _RTE_ATOMIC_C11_H_ */ diff --git a/lib/librte_eal/include/meson.build b/lib/librte_eal/include/meson.build index bc73ec2..dac1aac 100644 --- a/lib/librte_eal/include/meson.build +++ b/lib/librte_eal/include/meson.build @@ -51,6 +51,7 @@ headers += files( # special case install the generic headers, since they go in a subdir generic_headers = files( 'generic/rte_atomic.h', + 'generic/rte_atomic_c11.h', 'generic/rte_byteorder.h', 'generic/rte_cpuflags.h', 'generic/rte_cycles.h',