From patchwork Sat Mar 2 13:53:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Mattias_R=C3=B6nnblom?= X-Patchwork-Id: 137805 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 320EA43C3A; Sat, 2 Mar 2024 15:02:57 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 06B9242DDB; Sat, 2 Mar 2024 15:02:24 +0100 (CET) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2085.outbound.protection.outlook.com [40.107.21.85]) by mails.dpdk.org (Postfix) with ESMTP id F015F42DB0 for ; Sat, 2 Mar 2024 15:02:19 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UOmfaW5pvo85a4uaXRWcEDWZRstnVfZkx3VJ3Fo/bzyyH1cM9sAQ0h/pkiLgtzFzEwxm7jDJ+T0cgTJ00y2ZCWn/CEw9Xpt1jNytlHXvWTAorFpyqLiKneOG6L95YIkn3rt/rt56kt3vHBkL04ND42n1jAX1YHhMGKlCCJnUGoyD3g6w4dlze0+5B66+1+Q6l2sbXrt7VaMZCZFIYrO7s8niwCTviwejJo7hsAAgDzVuupiRwMUZ8WokI1p80bxKfDKLTGOhKrouQ0IA2PcgPhKpUdpKGE5EyEyAtgxDIf6Nl/Sv9oq3J1MDkHgdsmb5bD3olEVmtlIeRNztNmSkJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=k8Nj4gASYzE77LHZeDtN4V8e8HlSVIoCI/rhcl/RAfI=; b=n2Bn1/DPuz7N3KXl+dUXzXf6HufWlL2uC+x7rU1WYnF3SG41J5VuotZnDg/VXJdRLI0bphSGyMqzgvF2jw0wAdwmyDERZ5JiurwEKA7hcWFtL6HOBDbdxcqURbQXdYJa+kO1cheElBcsjApRRqpVI6qcnBC/I/nk0BQyMEdejCYR8t4KYUg+HajXvfkiKGoZ5RyQgWQLV6CvTrWmmwuBgguKHxqePoqK5CdbXw2AwYBj2Cg1HwQ1k2B5lixiIBOmAYvZ8J7BqDlt22s7aRqccD/29uwUxRiKBaLwrHFHlsHyCI7NlClJCeTQ6He7Zp/9f0S1GOWjfy9/bLMjBR8PlA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=dpdk.org smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=k8Nj4gASYzE77LHZeDtN4V8e8HlSVIoCI/rhcl/RAfI=; b=ARE2vi0VZfXKZWiFMygN8M6SZbzvLCW69ZyMzRa1j6BQtnF7soUhSLe9H2xhElJFINSyrsT2+7wglHRcccEjldmEvHqmoVnT+EnZI/Ytc9xPoHgNS2XDT/eSS+SoLrWCAirquv7j1mAppX90rt6wrMuBwgSEzgIs6lRRzeabWNVr3IX56Gek3fAzTYU2RiLzqKLWrffsO3wEAGlGv2jiaekeRs7HvbLokAt5UpeUmbcgzlub3aIIhqyZnNmvXZ7uiHYrNgsJBgKEm5ASYEa+dZ3eLZvEIPSq33N3GZ4+TlQne1WFpJC7vBScl5oVT1kr01Eiy9rbFzxYwgnN3GDgUA== Received: from AM6PR0502CA0048.eurprd05.prod.outlook.com (2603:10a6:20b:56::25) by DUZPR07MB9814.eurprd07.prod.outlook.com (2603:10a6:10:4dd::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7339.34; Sat, 2 Mar 2024 14:02:18 +0000 Received: from AM4PEPF00025F9B.EURPRD83.prod.outlook.com (2603:10a6:20b:56:cafe::8f) by AM6PR0502CA0048.outlook.office365.com (2603:10a6:20b:56::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7339.36 via Frontend Transport; Sat, 2 Mar 2024 14:02:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by AM4PEPF00025F9B.mail.protection.outlook.com (10.167.16.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7386.0 via Frontend Transport; Sat, 2 Mar 2024 14:02:17 +0000 Received: from seliicinfr00050.seli.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.60) with Microsoft SMTP Server id 15.2.1258.12; Sat, 2 Mar 2024 15:02:16 +0100 Received: from breslau.. (seliicwb00002.seli.gic.ericsson.se [10.156.25.100]) by seliicinfr00050.seli.gic.ericsson.se (Postfix) with ESMTP id DD83D1C006A; Sat, 2 Mar 2024 15:02:16 +0100 (CET) From: =?utf-8?q?Mattias_R=C3=B6nnblom?= To: CC: , Heng Wang , =?utf-8?q?M?= =?utf-8?q?attias_R=C3=B6nnblom?= Subject: [RFC 5/7] eal: add atomic bit operations Date: Sat, 2 Mar 2024 14:53:26 +0100 Message-ID: <20240302135328.531940-6-mattias.ronnblom@ericsson.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240302135328.531940-1-mattias.ronnblom@ericsson.com> References: <20240302135328.531940-1-mattias.ronnblom@ericsson.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM4PEPF00025F9B:EE_|DUZPR07MB9814:EE_ X-MS-Office365-Filtering-Correlation-Id: c67f2b62-0698-4682-e8bd-08dc3ac15ef2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uZQNlMbRGF41+EGkY5J0JVdlhEX8EH8oz0GpYchZulm6M/p3x3xjtLItdzIXsnK6rCTglMidmJhIXCfGvahRu+qDaF5M2tHEvnPk1Wv/sIXmZwQS8FD9npHzn64YYM0ODp5puv1OwrUGR+2Tuar+XAP+zJoz3sV7mISz6fJyo8ajXgBuC7/3bOk+Yg0d+Vyd5GZCkwwdqcFW9E5M+ZwhUb8v/JkxlvgGbQfbyd9woTAIDTKBxY2VZh+qYoF+iLAL8FxFdY/ecIQ5Ro/y3SgOhuQJvN/A96g2zlcSQ3PwReEbliU/TJNpRUkzBekr/KMGfl+NkHFSq0C4nqyoSpeziQTQxrR3TFqn1aVLQPk4Xj1avHEsv/FhfCgeJ8X1iQREldFwDRQTz/hjuoVekcD9Kehr2a15xqFfCoEIQ0F4lsyR+HvCEkjl8ArfBePPh9T5JRT8J81Rn+tQFEjA5zmPOFZxoQ6u8kab10GeQzJzlXOFSe/6knkvA2CNwjV4sDtWeHCQzXVadGTPXEbUr3b8NNGgVAhA1s85263ocUnpB7q9R/5Tfm37ztzAAqB6mjAjbs+0E+hyCak2gNTl8y+3BV/eplsm1HiQ7BhI+RM1tRiOShmMLAPlECIzRJtjgJYkqlY7bavNIeD+N6kupzuUmmXvuI9QWlAidLTkj62kNijJR0VUrm8UHJwGY9N+IXvfWhqA739RY1AMSV5R/oI0Pj+MEDK2YvDbgCea2gD5wYU= X-Forefront-Antispam-Report: CIP:192.176.1.74; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:oa.msg.ericsson.com; PTR:office365.se.ericsson.net; CAT:NONE; SFS:(13230031)(376005)(82310400014)(36860700004); DIR:OUT; SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Mar 2024 14:02:17.5219 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c67f2b62-0698-4682-e8bd-08dc3ac15ef2 X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f; Ip=[192.176.1.74]; Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: AM4PEPF00025F9B.EURPRD83.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DUZPR07MB9814 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add atomic bit test/set/clear/assign and test-and-set/clear functions. All atomic bit functions allow (and indeed, require) the caller to specify a memory order. Signed-off-by: Mattias Rönnblom --- lib/eal/include/rte_bitops.h | 337 +++++++++++++++++++++++++++++++++++ 1 file changed, 337 insertions(+) diff --git a/lib/eal/include/rte_bitops.h b/lib/eal/include/rte_bitops.h index 450334c751..7eb08bc768 100644 --- a/lib/eal/include/rte_bitops.h +++ b/lib/eal/include/rte_bitops.h @@ -20,6 +20,7 @@ #include #include +#include #ifdef __cplusplus extern "C" { @@ -706,6 +707,342 @@ __RTE_GEN_BIT_TEST(rte_bit_once_test64, 64, volatile) __RTE_GEN_BIT_SET(rte_bit_once_set64, 64, volatile) __RTE_GEN_BIT_CLEAR(rte_bit_once_clear64, 64, volatile) +/** + * Test if a particular bit in a 32-bit word is set with a particular + * memory order. + * + * Test a bit with the resulting memory load ordered as per the + * specified memory order. + * + * @param addr + * A pointer to the 32-bit word to query. + * @param nr + * The index of the bit (0-31). + * @param memory_order + * The memory order to use. See for details. + * @return + * Returns true if the bit is set, and false otherwise. + */ +static inline bool +rte_bit_atomic_test32(const uint32_t *addr, unsigned int nr, int memory_order); + +/** + * Atomically set bit in 32-bit word. + * + * Atomically bit specified by @c nr in the 32-bit word pointed to by + * @c addr to '1', with the memory ordering as specified by @c + * memory_order. + * + * @param addr + * A pointer to the 32-bit word to modify. + * @param nr + * The index of the bit (0-31). + * @param memory_order + * The memory order to use. See for details. + */ +static inline void +rte_bit_atomic_set32(uint32_t *addr, unsigned int nr, int memory_order); + +/** + * Atomically clear bit in 32-bit word. + * + * Atomically set bit specified by @c nr in the 32-bit word pointed to + * by @c addr to '0', with the memory ordering as specified by @c + * memory_order. + * + * @param addr + * A pointer to the 32-bit word to modify. + * @param nr + * The index of the bit (0-31). + * @param memory_order + * The memory order to use. See for details. + */ +static inline void +rte_bit_atomic_clear32(uint32_t *addr, unsigned int nr, int memory_order); + +/** + * Atomically assign a value to bit in a 32-bit word. + * + * Atomically set bit specified by @c nr in the 32-bit word pointed to + * by @c addr to the value indicated by @c value, with the memory + * ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the 32-bit word to modify. + * @param nr + * The index of the bit (0-31). + * @param value + * The new value of the bit - true for '1', or false for '0'. + * @param memory_order + * The memory order to use. See for details. + */ +static inline void +rte_bit_atomic_assign32(uint32_t *addr, unsigned int nr, bool value, + int memory_order); + +/* + * Atomic test-and-assign is not considered useful-enough to document + * and expose in the public API. + */ +static inline bool +__rte_bit_atomic_test_and_assign32(uint32_t *addr, unsigned int nr, bool value, + int memory_order); + +/** + * Atomically test and set a bit in a 32-bit word. + * + * Atomically test and set bit specified by @c nr in the 32-bit word + * pointed to by @c addr to the value indicated by @c value, with the + * memory ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the 32-bit word to modify. + * @param nr + * The index of the bit (0-31). + * @param memory_order + * The memory order to use. See for details. + * @return + * Returns true if the bit was set, and false otherwise. + */ +static inline bool +rte_bit_atomic_test_and_set32(uint32_t *addr, unsigned int nr, + int memory_order) +{ + return __rte_bit_atomic_test_and_assign32(addr, nr, true, memory_order); +} + +/** + * Atomically test and clear a bit in a 32-bit word. + * + * Atomically test and clear bit specified by @c nr in the 32-bit word + * pointed to by @c addr to the value indicated by @c value, with the + * memory ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the 32-bit word to modify. + * @param nr + * The index of the bit (0-31). + * @param memory_order + * The memory order to use. See for details. + * @return + * Returns true if the bit was set, and false otherwise. + */ +static inline bool +rte_bit_atomic_test_and_clear32(uint32_t *addr, unsigned int nr, + int memory_order) +{ + return __rte_bit_atomic_test_and_assign32(addr, nr, false, memory_order); +} + +/** + * Test if a particular bit in a 32-bit word is set with a particular + * memory order. + * + * Test a bit with the resulting memory load ordered as per the + * specified memory order. + * + * @param addr + * A pointer to the 32-bit word to query. + * @param nr + * The index of the bit (0-31). + * @param memory_order + * The memory order to use. See for details. + * @return + * Returns true if the bit is set, and false otherwise. + */ +static inline bool +rte_bit_atomic_test64(const uint64_t *addr, unsigned int nr, int memory_order); + +/** + * Atomically set bit in 64-bit word. + * + * Atomically bit specified by @c nr in the 64-bit word pointed to by + * @c addr to '1', with the memory ordering as specified by @c + * memory_order. + * + * @param addr + * A pointer to the 64-bit word to modify. + * @param nr + * The index of the bit (0-63). + * @param memory_order + * The memory order to use. See for details. + */ +static inline void +rte_bit_atomic_set64(uint64_t *addr, unsigned int nr, int memory_order); + +/** + * Atomically clear bit in 64-bit word. + * + * Atomically set bit specified by @c nr in the 64-bit word pointed to + * by @c addr to '0', with the memory ordering as specified by @c + * memory_order. + * + * @param addr + * A pointer to the 64-bit word to modify. + * @param nr + * The index of the bit (0-63). + * @param memory_order + * The memory order to use. See for details. + */ +static inline void +rte_bit_atomic_clear64(uint64_t *addr, unsigned int nr, int memory_order); + +/** + * Atomically assign a value to bit in a 64-bit word. + * + * Atomically set bit specified by @c nr in the 64-bit word pointed to + * by @c addr to the value indicated by @c value, with the memory + * ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the 64-bit word to modify. + * @param nr + * The index of the bit (0-63). + * @param value + * The new value of the bit - true for '1', or false for '0'. + * @param memory_order + * The memory order to use. See for details. + */ +static inline void +rte_bit_atomic_assign64(uint64_t *addr, unsigned int nr, bool value, + int memory_order); + +/* + * Atomic test-and-assign is not considered useful-enough to document + * and expose in the public API. + */ +static inline bool +__rte_bit_atomic_test_and_assign64(uint64_t *addr, unsigned int nr, bool value, + int memory_order); +/** + * Atomically test and set a bit in a 64-bit word. + * + * Atomically test and set bit specified by @c nr in the 64-bit word + * pointed to by @c addr to the value indicated by @c value, with the + * memory ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the 64-bit word to modify. + * @param nr + * The index of the bit (0-63). + * @param memory_order + * The memory order to use. See for details. + * @return + * Returns true if the bit was set, and false otherwise. + */ +static inline bool +rte_bit_atomic_test_and_set64(uint64_t *addr, unsigned int nr, + int memory_order) +{ + return __rte_bit_atomic_test_and_assign64(addr, nr, true, memory_order); +} + +/** + * Atomically test and clear a bit in a 64-bit word. + * + * Atomically test and clear bit specified by @c nr in the 64-bit word + * pointed to by @c addr to the value indicated by @c value, with the + * memory ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the 64-bit word to modify. + * @param nr + * The index of the bit (0-63). + * @param memory_order + * The memory order to use. See for details. + * @return + * Returns true if the bit was set, and false otherwise. + */ +static inline bool +rte_bit_atomic_test_and_clear64(uint64_t *addr, unsigned int nr, + int memory_order) +{ + return __rte_bit_atomic_test_and_assign64(addr, nr, false, memory_order); +} + +#ifndef RTE_ENABLE_STDATOMIC + +#define __RTE_GEN_BIT_ATOMIC_TEST(size) \ + static inline bool \ + rte_bit_atomic_test ## size(const uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + return __atomic_load_n(addr, memory_order) & mask; \ + } + +#define __RTE_GEN_BIT_ATOMIC_SET(size) \ + static inline void \ + rte_bit_atomic_set ## size(uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + __atomic_fetch_or(addr, mask, memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_CLEAR(size) \ + static inline void \ + rte_bit_atomic_clear ## size(uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + __atomic_fetch_and(addr, ~mask, memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_ASSIGN(size) \ + static inline void \ + rte_bit_atomic_assign ## size(uint ## size ## _t *addr, \ + unsigned int nr, bool value, \ + int memory_order) \ + { \ + if (value) \ + rte_bit_atomic_set ## size(addr, nr, memory_order); \ + else \ + rte_bit_atomic_clear ## size(addr, nr, memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_TEST_AND_ASSIGN(size) \ + static inline bool \ + __rte_bit_atomic_test_and_assign ## size(uint ## size ## _t *addr, \ + unsigned int nr, \ + bool value, \ + int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + uint ## size ## _t before; \ + uint ## size ## _t after; \ + \ + before = __atomic_load_n(addr, __ATOMIC_RELAXED); \ + \ + do { \ + rte_bit_assign ## size(&before, nr, value); \ + } while(!__atomic_compare_exchange_n(addr, &before, after, \ + true, __ATOMIC_RELAXED, \ + memory_order)); \ + return rte_bit_test ## size(&before, nr); \ + } + +#else +#error "C11 atomics (MSVC) not supported in this RFC version" +#endif + +#define __RTE_GEN_BIT_ATOMIC_OPS(size) \ + __RTE_GEN_BIT_ATOMIC_TEST(size) \ + __RTE_GEN_BIT_ATOMIC_SET(size) \ + __RTE_GEN_BIT_ATOMIC_CLEAR(size) \ + __RTE_GEN_BIT_ATOMIC_ASSIGN(size) \ + __RTE_GEN_BIT_ATOMIC_TEST_AND_ASSIGN(size) + +__RTE_GEN_BIT_ATOMIC_OPS(32) +__RTE_GEN_BIT_ATOMIC_OPS(64) + /*------------------------ 32-bit relaxed operations ------------------------*/ /**