From patchwork Tue Apr 30 09:55:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Mattias_R=C3=B6nnblom?= X-Patchwork-Id: 139732 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C385F43F3A; Tue, 30 Apr 2024 12:06:54 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7A4FE4064A; Tue, 30 Apr 2024 12:06:04 +0200 (CEST) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on2054.outbound.protection.outlook.com [40.107.14.54]) by mails.dpdk.org (Postfix) with ESMTP id DC08F402EB for ; Tue, 30 Apr 2024 12:06:01 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hk0Ba68VDp9Nl1c/a6aXVSwEAPOZpA//cnq618hrb0AAaAJOduFeverk3zlQWjCjmZUh42fsEB2pIQXHKmAVA+dYGWizidQhDbLxn1g10I2JJijMt3kzn55vrLQSp/RwWe9b4TNOlkQ7wNZSjmr3TEULjhNK/drkzXzaXAXs5UNXjP7eyZT+UK/3OSboR06hN4YEhrGrgZwDbLnv2OkNP3YUJbmG1Leiu0Yrc/e9+rqFBv2H/a0Ag1HL8lSlVgFehkLPmQ/fR8UoFNPjP8XhXc9K5pA2TvCYs0IPABmnk/87ZF7TlLtbUGdU5g3UQs3NXEhCQN0QkCikVVRWjZ2bzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=S7lEMJk/WgNgYNKA+tfHbv7buTOuFqDk0YnVV6mMMn0=; b=dUndGY+ifkidIeaXY9EKC5tS0fftIlNpNIxffZZ+250mWUDtJRF7Ao95hOJO+sD1yV2WmDS9gR1Zlo2nH/czk3XlO/KOi/nlIKrIc31i3vt++SDgGINtA2caXz6pscZZ8AHBt/0joIOG7RC4PetBBuCT67rzGxb3DAy4lmk7iy+8+dDx2JNTFiGBCOFRi2exukpyDO+AzYj7slo+utWkUQkiCAzRKahZYU7Tnqtn4WHjr0nHX+EWZ0XJn1ZOAZaBniNXhn2dMHhRnz1kttvBCPO23O5AxgWk7ftNXut4vZUNpbP4q3D+4siM8LJ9hpCneywJ/qpb4zGkmTu8qHyAjg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=dpdk.org smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=S7lEMJk/WgNgYNKA+tfHbv7buTOuFqDk0YnVV6mMMn0=; b=Tn5YDEtrjJ25uALcSBR6r8R2BLTunG/6vHJ0MoV/okZbwmqNKwoJTo1cuA1wIqTxSORk0sOf7uSpsECyw8Cx3iKuXGbZplZuUYE2vUNKfrlJQVlkuPhh/7i2dnuet+LwlurWHdxLX/KEhwqI3tkItd542pqAYrNnNYjGgV+VmMC1bAzIxQUm/omnKQEc9sRdWwZsle5gYZYIRV6UUOqWz7TMSZ6bBKNqIxT/ADlm5c5QZ9mUkcwm/zhH2n4SI08MUeht5q1o/FVPU+CgiyuLjLbjFV9uJAv5joxwpENSTqxpDZl1eG5lqbBXQD9hEvLAKHE6aUooT3Iff7sdBhoTWg== Received: from DUZPR01CA0236.eurprd01.prod.exchangelabs.com (2603:10a6:10:4b5::15) by AS2PR07MB9328.eurprd07.prod.outlook.com (2603:10a6:20b:608::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.34; Tue, 30 Apr 2024 10:05:57 +0000 Received: from DB3PEPF0000885A.eurprd02.prod.outlook.com (2603:10a6:10:4b5:cafe::8f) by DUZPR01CA0236.outlook.office365.com (2603:10a6:10:4b5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.34 via Frontend Transport; Tue, 30 Apr 2024 10:05:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by DB3PEPF0000885A.mail.protection.outlook.com (10.167.242.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7544.18 via Frontend Transport; Tue, 30 Apr 2024 10:05:54 +0000 Received: from seliicinfr00050.seli.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.69) with Microsoft SMTP Server id 15.2.1544.9; Tue, 30 Apr 2024 12:05:53 +0200 Received: from breslau.. (seliicwb00002.seli.gic.ericsson.se [10.156.25.100]) by seliicinfr00050.seli.gic.ericsson.se (Postfix) with ESMTP id B914B1C006A; Tue, 30 Apr 2024 12:05:53 +0200 (CEST) From: =?utf-8?q?Mattias_R=C3=B6nnblom?= To: CC: , Heng Wang , "Stephen Hemminger" , Tyler Retzlaff , =?utf-8?q?Morten_Br=C3=B8rup?= , =?utf-8?q?Mattia?= =?utf-8?q?s_R=C3=B6nnblom?= Subject: [RFC v4 4/6] eal: add unit tests for exactly-once bit access functions Date: Tue, 30 Apr 2024 11:55:21 +0200 Message-ID: <20240430095523.108688-5-mattias.ronnblom@ericsson.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240430095523.108688-1-mattias.ronnblom@ericsson.com> References: <20240429095138.106849-2-mattias.ronnblom@ericsson.com> <20240430095523.108688-1-mattias.ronnblom@ericsson.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DB3PEPF0000885A:EE_|AS2PR07MB9328:EE_ X-MS-Office365-Filtering-Correlation-Id: bf57ba10-05dc-4358-48ab-08dc68fd1f7d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230031|82310400014|376005|1800799015|36860700004; X-Microsoft-Antispam-Message-Info: =?utf-8?q?gWdEAwWmhcWRfaK43feQhpIgW2PZG7d?= =?utf-8?q?9IUdjf4fNYvaX/Op+1OnJqIjKbiRqFxigED6hzvwFXgZZxCgZZYubNyv0/Anozs2R?= =?utf-8?q?8CtNFAFbI3oLf5nqw2RNxjtMb0+rCNgrfN2CVWDJcgBkoOLs+02qx6l6LinD1ayBV?= =?utf-8?q?URa3eI8jtJELotuYa415xmlJ6bO/+uAK4EAVPDaw2odafNQcb2KBE23dBJCFqduEl?= =?utf-8?q?YP6JfHfP2fWhKUjXsT9X9EyA15MY2/SzaIerUjiT0XroVhjUhU+x3pXqq1ZDJobA8?= =?utf-8?q?M4MSSMOMLTTlYrqNj3IfvtsPBjKo9TS99lV83wUx3rYtjTFnpqwT29usE2nl2j5/M?= =?utf-8?q?PMgGepxA3XNittdebQsUUL4gEyS1RWPjQS8YhdtCuPCY18tPYmGH5HS8V39r3iF9d?= =?utf-8?q?k3aWzfyOppcVgb17RFMYa1Mve1VHUV6tIIj8HxEfXEAbflsONjiFsaxHSMNDJZ9ht?= =?utf-8?q?mogFqUy4zNemmLpx+Dj34CGL94/p6aPIqWQnG3vAwSCCcTyxg1YcOtOdhFbnjYfX2?= =?utf-8?q?WGEnK/wkcai2QsFQ6Iz0TkV+D8baSz4xBSpqs5YtUm/NpuBRt4XU6QvEDByvblRQw?= =?utf-8?q?/7WO0NIu3iYT1chxQIiuKjH/NA+45QEQJRQVBafs9iAFuGTNIWRVFHzq8REMS1zO5?= =?utf-8?q?MNg34VPq90TMVn2YcRa15NUcKsw1Yeh0wnvL+LWVhvoSfClBfHM5+S7vPmh6dNmCt?= =?utf-8?q?ST4hJtKxlxhHOSangE8sSxO5XfsGElYoxn9x0/7kuvYa5W5By3R9G0wV9c8deevvT?= =?utf-8?q?Es9mJoPzTUTQi/P9sOitdV/9UL6m5DGqm9wxP2wJ8txiIWWftSZyyWFWiQvZodaY5?= =?utf-8?q?rdWOqkDxFiLNB8BB5hCC3bRGGYQb2F/jL5zhMANB+/FW08KOIA2rAxMVh28qOsEnZ?= =?utf-8?q?gfyq+wXvewL6bNK1jpIQ9XasnFhLzQ7WDXqFp7+kv10EhUynmYIZuwdYxKzyTdHZo?= =?utf-8?q?XfXHkslYHeIMSUfsq5d5nLWF8JaOyuDHcBfBFjYewG+1tRTMQuo5CfRBcl3e8ADhY?= =?utf-8?q?qhtDYI2xFubG9EiFSv99SpF2Q4EPPgm5WDmCvpbNjYfi8LxTA2XgJ0XCMXhe/9bI/?= =?utf-8?q?+LpR86WK3tNY36tMsKOZVpTidMdV6QJ1fGCRyhHbI5n8TSVbqC8QuI57G+5ByMASy?= =?utf-8?q?g2LfvkPMVz+0fveD7jll8YbMY8IwSGeAa7TBOmHHMHGCdXwlOHYZske9i9E728Jlj?= =?utf-8?q?tbEv0K8i6mO9V/bzaU9bMyamAcCOePX1n/CrL1Cbb9WiPhuO2WGI+6xbG5XzzMeuL?= =?utf-8?q?MIf7WI7OHujXnIEtP9Vy13Pc2dbkFRlueTzgA32n36x7UK5nXR7i189upcLJBaE8Q?= =?utf-8?q?uV7gyACqrQ5t?= X-Forefront-Antispam-Report: CIP:192.176.1.74; CTRY:SE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:oa.msg.ericsson.com; PTR:office365.se.ericsson.net; CAT:NONE; SFS:(13230031)(82310400014)(376005)(1800799015)(36860700004); DIR:OUT; SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Apr 2024 10:05:54.3305 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bf57ba10-05dc-4358-48ab-08dc68fd1f7d X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f; Ip=[192.176.1.74]; Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: DB3PEPF0000885A.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS2PR07MB9328 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Extend bitops tests to cover the rte_bit_once_[set|clear|assign|test]() family of functions. RFC v4: * Remove redundant continuations. Signed-off-by: Mattias Rönnblom Acked-by: Morten Brørup Acked-by: Tyler Retzlaff --- app/test/test_bitops.c | 10 + lib/eal/include/rte_bitops.h | 425 +++++++++++++++++++++++++++++++++++ 2 files changed, 435 insertions(+) diff --git a/app/test/test_bitops.c b/app/test/test_bitops.c index 111f9b328e..615ec6e563 100644 --- a/app/test/test_bitops.c +++ b/app/test/test_bitops.c @@ -56,6 +56,14 @@ GEN_TEST_BIT_ACCESS(test_bit_access32, rte_bit_set, rte_bit_clear, GEN_TEST_BIT_ACCESS(test_bit_access64, rte_bit_set, rte_bit_clear, rte_bit_assign, rte_bit_flip, rte_bit_test, 64) +GEN_TEST_BIT_ACCESS(test_bit_once_access32, rte_bit_once_set, + rte_bit_once_clear, rte_bit_once_assign, + rte_bit_once_flip, rte_bit_once_test, 32) + +GEN_TEST_BIT_ACCESS(test_bit_once_access64, rte_bit_once_set, + rte_bit_once_clear, rte_bit_once_assign, + rte_bit_once_flip, rte_bit_once_test, 64) + static uint32_t val32; static uint64_t val64; @@ -172,6 +180,8 @@ static struct unit_test_suite test_suite = { .unit_test_cases = { TEST_CASE(test_bit_access32), TEST_CASE(test_bit_access64), + TEST_CASE(test_bit_once_access32), + TEST_CASE(test_bit_once_access64), TEST_CASE(test_bit_relaxed_set), TEST_CASE(test_bit_relaxed_clear), TEST_CASE(test_bit_relaxed_test_set_clear), diff --git a/lib/eal/include/rte_bitops.h b/lib/eal/include/rte_bitops.h index f77bd83e97..abfe96d531 100644 --- a/lib/eal/include/rte_bitops.h +++ b/lib/eal/include/rte_bitops.h @@ -21,6 +21,7 @@ #include #include +#include #ifdef __cplusplus extern "C" { @@ -395,6 +396,199 @@ extern "C" { uint32_t *: __rte_bit_once_flip32, \ uint64_t *: __rte_bit_once_flip64)(addr, nr) +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Test if a particular bit in a word is set with a particular memory + * order. + * + * Test a bit with the resulting memory load ordered as per the + * specified memory order. + * + * @param addr + * A pointer to the word to query. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. See for details. + * @return + * Returns true if the bit is set, and false otherwise. + */ +#define rte_bit_atomic_test(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_test32, \ + uint64_t *: __rte_bit_atomic_test64)(addr, nr, memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically set bit in word. + * + * Atomically set bit specified by @c nr in the word pointed to by @c + * addr to '1', with the memory ordering as specified by @c + * memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. See for details. + */ +#define rte_bit_atomic_set(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_set32, \ + uint64_t *: __rte_bit_atomic_set64)(addr, nr, memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically clear bit in word. + * + * Atomically set bit specified by @c nr in the word pointed to by @c + * addr to '0', with the memory ordering as specified by @c + * memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. See for details. + */ +#define rte_bit_atomic_clear(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_clear32, \ + uint64_t *: __rte_bit_atomic_clear64)(addr, nr, memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically assign a value to bit in word. + * + * Atomically set bit specified by @c nr in the word pointed to by @c + * addr to the value indicated by @c value, with the memory ordering + * as specified with @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param value + * The new value of the bit - true for '1', or false for '0'. + * @param memory_order + * The memory order to use. See for details. + */ +#define rte_bit_atomic_assign(addr, nr, value, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_assign32, \ + uint64_t *: __rte_bit_atomic_assign64)(addr, nr, value, \ + memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically flip bit in word. + * + * Atomically negate the value of the bit specified by @c nr in the + * word pointed to by @c addr to the value indicated by @c value, with + * the memory ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. See for details. + */ +#define rte_bit_atomic_flip(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_flip32, \ + uint64_t *: __rte_bit_atomic_flip64)(addr, nr, memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically test and set a bit in word. + * + * Atomically test and set bit specified by @c nr in the word pointed + * to by @c addr to '1', with the memory ordering as specified with @c + * memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. See for details. + * @return + * Returns true if the bit was set, and false otherwise. + */ +#define rte_bit_atomic_test_and_set(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_test_and_set32, \ + uint64_t *: __rte_bit_atomic_test_and_set64)(addr, nr, \ + memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically test and clear a bit in word. + * + * Atomically test and clear bit specified by @c nr in the word + * pointed to by @c addr to '0', with the memory ordering as specified + * with @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param memory_order + * The memory order to use. See for details. + * @return + * Returns true if the bit was set, and false otherwise. + */ +#define rte_bit_atomic_test_and_clear(addr, nr, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_test_and_clear32, \ + uint64_t *: __rte_bit_atomic_test_and_clear64)(addr, nr, \ + memory_order) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Atomically test and assign a bit in word. + * + * Atomically test and assign bit specified by @c nr in the word + * pointed to by @c addr the value specified by @c value, with the + * memory ordering as specified with @c memory_order. + * + * @param addr + * A pointer to the word to modify. + * @param nr + * The index of the bit. + * @param value + * The new value of the bit - true for '1', or false for '0'. + * @param memory_order + * The memory order to use. See for details. + * @return + * Returns true if the bit was set, and false otherwise. + */ +#define rte_bit_atomic_test_and_assign(addr, nr, value, memory_order) \ + _Generic((addr), \ + uint32_t *: __rte_bit_atomic_test_and_assign32, \ + uint64_t *: __rte_bit_atomic_test_and_assign64)(addr, nr, \ + value, \ + memory_order) + #define __RTE_GEN_BIT_TEST(family, fun, qualifier, size) \ __rte_experimental \ static inline bool \ @@ -479,6 +673,162 @@ __RTE_GEN_BIT_CLEAR(once_, clear, volatile, 64) __RTE_GEN_BIT_ASSIGN(once_, assign, volatile, 64) __RTE_GEN_BIT_FLIP(once_, flip, volatile, 64) +#define __RTE_GEN_BIT_ATOMIC_TEST(size) \ + __rte_experimental \ + static inline bool \ + __rte_bit_atomic_test ## size(const uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + const RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (const RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + return rte_atomic_load_explicit(a_addr, memory_order) & mask; \ + } + +#define __RTE_GEN_BIT_ATOMIC_SET(size) \ + __rte_experimental \ + static inline void \ + __rte_bit_atomic_set ## size(uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + rte_atomic_fetch_or_explicit(a_addr, mask, memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_CLEAR(size) \ + __rte_experimental \ + static inline void \ + __rte_bit_atomic_clear ## size(uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t mask = (uint ## size ## _t)1 << nr; \ + rte_atomic_fetch_and_explicit(a_addr, ~mask, memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_ASSIGN(size) \ + __rte_experimental \ + static inline void \ + __rte_bit_atomic_assign ## size(uint ## size ## _t *addr, \ + unsigned int nr, bool value, \ + int memory_order) \ + { \ + if (value) \ + __rte_bit_atomic_set ## size(addr, nr, memory_order); \ + else \ + __rte_bit_atomic_clear ## size(addr, nr, \ + memory_order); \ + } + +#define __RTE_GEN_BIT_ATOMIC_TEST_AND_ASSIGN(size) \ + __rte_experimental \ + static inline bool \ + __rte_bit_atomic_test_and_assign ## size(uint ## size ## _t *addr, \ + unsigned int nr, \ + bool value, \ + int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t before; \ + uint ## size ## _t target; \ + \ + before = rte_atomic_load_explicit(a_addr, \ + rte_memory_order_relaxed); \ + \ + do { \ + target = before; \ + __rte_bit_assign ## size(&target, nr, value); \ + } while (!rte_atomic_compare_exchange_weak_explicit( \ + a_addr, &before, target, \ + rte_memory_order_relaxed, \ + memory_order)); \ + return __rte_bit_test ## size(&before, nr); \ + } + +#define __RTE_GEN_BIT_ATOMIC_FLIP(size) \ + __rte_experimental \ + static inline void \ + __rte_bit_atomic_flip ## size(uint ## size ## _t *addr, \ + unsigned int nr, int memory_order) \ + { \ + RTE_ASSERT(nr < size); \ + \ + RTE_ATOMIC(uint ## size ## _t) *a_addr = \ + (RTE_ATOMIC(uint ## size ## _t) *)addr; \ + uint ## size ## _t before; \ + uint ## size ## _t target; \ + \ + before = rte_atomic_load_explicit(a_addr, \ + rte_memory_order_relaxed); \ + \ + do { \ + target = before; \ + __rte_bit_flip ## size(&target, nr); \ + } while (!rte_atomic_compare_exchange_weak_explicit( \ + a_addr, &before, target, \ + rte_memory_order_relaxed, \ + memory_order)); \ + } + +#define __RTE_GEN_BIT_ATOMIC_OPS(size) \ + __RTE_GEN_BIT_ATOMIC_TEST(size) \ + __RTE_GEN_BIT_ATOMIC_SET(size) \ + __RTE_GEN_BIT_ATOMIC_CLEAR(size) \ + __RTE_GEN_BIT_ATOMIC_ASSIGN(size) \ + __RTE_GEN_BIT_ATOMIC_TEST_AND_ASSIGN(size) \ + __RTE_GEN_BIT_ATOMIC_FLIP(size) + +__RTE_GEN_BIT_ATOMIC_OPS(32) +__RTE_GEN_BIT_ATOMIC_OPS(64) + +__rte_experimental +static inline bool +__rte_bit_atomic_test_and_set32(uint32_t *addr, unsigned int nr, + int memory_order) +{ + return __rte_bit_atomic_test_and_assign32(addr, nr, true, + memory_order); +} + +__rte_experimental +static inline bool +__rte_bit_atomic_test_and_clear32(uint32_t *addr, unsigned int nr, + int memory_order) +{ + return __rte_bit_atomic_test_and_assign32(addr, nr, false, + memory_order); +} + +__rte_experimental +static inline bool +__rte_bit_atomic_test_and_set64(uint64_t *addr, unsigned int nr, + int memory_order) +{ + return __rte_bit_atomic_test_and_assign64(addr, nr, true, + memory_order); +} + +__rte_experimental +static inline bool +__rte_bit_atomic_test_and_clear64(uint64_t *addr, unsigned int nr, + int memory_order) +{ + return __rte_bit_atomic_test_and_assign64(addr, nr, false, + memory_order); +} + /*------------------------ 32-bit relaxed operations ------------------------*/ /** @@ -1180,6 +1530,14 @@ rte_log2_u64(uint64_t v) #undef rte_bit_once_assign #undef rte_bit_once_flip +#undef rte_bit_atomic_test +#undef rte_bit_atomic_set +#undef rte_bit_atomic_clear +#undef rte_bit_atomic_assign +#undef rte_bit_atomic_test_and_set +#undef rte_bit_atomic_test_and_clear +#undef rte_bit_atomic_test_and_assign + #define __RTE_BIT_OVERLOAD_SZ_2(fun, qualifier, size, arg1_type, arg1_name) \ static inline void \ rte_bit_ ## fun(qualifier uint ## size ## _t *addr, \ @@ -1223,6 +1581,59 @@ rte_log2_u64(uint64_t v) __RTE_BIT_OVERLOAD_SZ_3(fun, qualifier, 64, arg1_type, arg1_name, \ arg2_type, arg2_name) +#define __RTE_BIT_OVERLOAD_SZ_3R(fun, qualifier, size, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name) \ + static inline ret_type \ + rte_bit_ ## fun(uint ## size ## _t *addr, arg1_type arg1_name, \ + arg2_type arg2_name) \ + { \ + return __rte_bit_ ## fun ## size(addr, arg1_name, arg2_name); \ + } + +#define __RTE_BIT_OVERLOAD_3R(fun, qualifier, ret_type, arg1_type, arg1_name, \ + arg2_type, arg2_name) \ + __RTE_BIT_OVERLOAD_SZ_3R(fun, qualifier, 32, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name) \ + __RTE_BIT_OVERLOAD_SZ_3R(fun, qualifier, 64, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name) + +#define __RTE_BIT_OVERLOAD_SZ_4(fun, qualifier, size, arg1_type, arg1_name, \ + arg2_type, arg2_name, arg3_type, arg3_name) \ + static inline void \ + rte_bit_ ## fun(uint ## size ## _t *addr, arg1_type arg1_name, \ + arg2_type arg2_name, arg3_type arg3_name) \ + { \ + __rte_bit_ ## fun ## size(addr, arg1_name, arg2_name, \ + arg3_name); \ + } + +#define __RTE_BIT_OVERLOAD_4(fun, qualifier, arg1_type, arg1_name, arg2_type, \ + arg2_name, arg3_type, arg3_name) \ + __RTE_BIT_OVERLOAD_SZ_4(fun, qualifier, 32, arg1_type, arg1_name, \ + arg2_type, arg2_name, arg3_type, arg3_name) \ + __RTE_BIT_OVERLOAD_SZ_4(fun, qualifier, 64, arg1_type, arg1_name, \ + arg2_type, arg2_name, arg3_type, arg3_name) + +#define __RTE_BIT_OVERLOAD_SZ_4R(fun, qualifier, size, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name, arg3_type, \ + arg3_name) \ + static inline ret_type \ + rte_bit_ ## fun(uint ## size ## _t *addr, arg1_type arg1_name, \ + arg2_type arg2_name, arg3_type arg3_name) \ + { \ + return __rte_bit_ ## fun ## size(addr, arg1_name, arg2_name, \ + arg3_name); \ + } + +#define __RTE_BIT_OVERLOAD_4R(fun, qualifier, ret_type, arg1_type, arg1_name, \ + arg2_type, arg2_name, arg3_type, arg3_name) \ + __RTE_BIT_OVERLOAD_SZ_4R(fun, qualifier, 32, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name, arg3_type, \ + arg3_name) \ + __RTE_BIT_OVERLOAD_SZ_4R(fun, qualifier, 64, ret_type, arg1_type, \ + arg1_name, arg2_type, arg2_name, arg3_type, \ + arg3_name) + __RTE_BIT_OVERLOAD_2R(test, const, bool, unsigned int, nr) __RTE_BIT_OVERLOAD_2(set,, unsigned int, nr) __RTE_BIT_OVERLOAD_2(clear,, unsigned int, nr) @@ -1235,6 +1646,20 @@ __RTE_BIT_OVERLOAD_2(once_clear, volatile, unsigned int, nr) __RTE_BIT_OVERLOAD_3(once_assign, volatile, unsigned int, nr, bool, value) __RTE_BIT_OVERLOAD_2(once_flip, volatile, unsigned int, nr) +__RTE_BIT_OVERLOAD_3R(atomic_test, const, bool, unsigned int, nr, + int, memory_order) +__RTE_BIT_OVERLOAD_3(atomic_set,, unsigned int, nr, int, memory_order) +__RTE_BIT_OVERLOAD_3(atomic_clear,, unsigned int, nr, int, memory_order) +__RTE_BIT_OVERLOAD_4(atomic_assign,, unsigned int, nr, bool, value, + int, memory_order) +__RTE_BIT_OVERLOAD_3(atomic_flip,, unsigned int, nr, int, memory_order) +__RTE_BIT_OVERLOAD_3R(atomic_test_and_set,, bool, unsigned int, nr, + int, memory_order) +__RTE_BIT_OVERLOAD_3R(atomic_test_and_clear,, bool, unsigned int, nr, + int, memory_order) +__RTE_BIT_OVERLOAD_4R(atomic_test_and_assign,, bool, unsigned int, nr, + bool, value, int, memory_order) + #endif #endif /* _RTE_BITOPS_H_ */