From patchwork Fri Sep 25 17:43:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Lariau X-Patchwork-Id: 78861 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5BFE9A04C0; Fri, 25 Sep 2020 19:44:31 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 730591E9D9; Fri, 25 Sep 2020 19:44:15 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 445C41E9D0 for ; Fri, 25 Sep 2020 19:44:10 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B2069101E; Fri, 25 Sep 2020 10:44:08 -0700 (PDT) Received: from localhost.localdomain (unknown [10.57.54.55]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 02B443F718; Fri, 25 Sep 2020 10:44:07 -0700 (PDT) From: Steven Lariau To: Gage Eads , Olivier Matz Cc: dev@dpdk.org, nd@arm.com, Steven Lariau Date: Fri, 25 Sep 2020 18:43:35 +0100 Message-Id: <20200925174340.10014-2-steven.lariau@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200925174340.10014-1-steven.lariau@arm.com> References: <20200911152938.8019-1-steven.lariau@arm.com> <20200925174340.10014-1-steven.lariau@arm.com> Subject: [dpdk-dev] [PATCH v2 1/5] lib/stack: fix inconsistent weak / strong cas X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Fix cmpexchange usage of weak / strong. The generated code is the same on x86 and ARM (there is no weak cmpexchange), but the old usage was inconsistent. For push and pop update size, weak is used because cmpexchange is inside a loop. For pop update root, strong is used even though cmpexchange is inside a loop, because there may be a lot of operations to do in a loop iteration (locate the new head). Signed-off-by: Steven Lariau Reviewed-by: Dharmik Thakkar Reviewed-by: Ruifeng Wang Acked-by: Gage Eads --- lib/librte_stack/rte_stack_lf_c11.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/librte_stack/rte_stack_lf_c11.h b/lib/librte_stack/rte_stack_lf_c11.h index 999359f08..1e0ea0bef 100644 --- a/lib/librte_stack/rte_stack_lf_c11.h +++ b/lib/librte_stack/rte_stack_lf_c11.h @@ -96,7 +96,7 @@ __rte_stack_lf_pop_elems(struct rte_stack_lf_list *list, /* len is updated on failure */ if (__atomic_compare_exchange_n(&list->len, &len, len - num, - 0, __ATOMIC_ACQUIRE, + 1, __ATOMIC_ACQUIRE, __ATOMIC_ACQUIRE)) break; } @@ -149,7 +149,7 @@ __rte_stack_lf_pop_elems(struct rte_stack_lf_list *list, (rte_int128_t *)&list->head, (rte_int128_t *)&old_head, (rte_int128_t *)&new_head, - 1, __ATOMIC_RELEASE, + 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED); } while (success == 0); From patchwork Fri Sep 25 17:43:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Lariau X-Patchwork-Id: 78862 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 87050A04C0; Fri, 25 Sep 2020 19:44:47 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 831CF1E9EA; Fri, 25 Sep 2020 19:44:17 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 634611E9D1 for ; Fri, 25 Sep 2020 19:44:12 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D26A01396; Fri, 25 Sep 2020 10:44:10 -0700 (PDT) Received: from localhost.localdomain (unknown [10.57.54.55]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D967A3F718; Fri, 25 Sep 2020 10:44:09 -0700 (PDT) From: Steven Lariau To: Gage Eads , Olivier Matz Cc: dev@dpdk.org, nd@arm.com, Steven Lariau Date: Fri, 25 Sep 2020 18:43:36 +0100 Message-Id: <20200925174340.10014-3-steven.lariau@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200925174340.10014-1-steven.lariau@arm.com> References: <20200911152938.8019-1-steven.lariau@arm.com> <20200925174340.10014-1-steven.lariau@arm.com> Subject: [dpdk-dev] [PATCH v2 2/5] lib/stack: remove push acquire fence X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" An acquire fence is used to make sure loads after the fence can observe all store operations before a specific store-release. But push doesn't read any data, except for the head which is part of a CAS operation (the items on the list are not read). So there is no need for the acquire barrier. Signed-off-by: Steven Lariau Reviewed-by: Dharmik Thakkar Reviewed-by: Ruifeng Wang Acked-by: Gage Eads --- lib/librte_stack/rte_stack_lf_c11.h | 6 ------ 1 file changed, 6 deletions(-) diff --git a/lib/librte_stack/rte_stack_lf_c11.h b/lib/librte_stack/rte_stack_lf_c11.h index 1e0ea0bef..82b7287f1 100644 --- a/lib/librte_stack/rte_stack_lf_c11.h +++ b/lib/librte_stack/rte_stack_lf_c11.h @@ -44,12 +44,6 @@ __rte_stack_lf_push_elems(struct rte_stack_lf_list *list, do { struct rte_stack_lf_head new_head; - /* Use an acquire fence to establish a synchronized-with - * relationship between the list->head load and store-release - * operations (as part of the rte_atomic128_cmp_exchange()). - */ - __atomic_thread_fence(__ATOMIC_ACQUIRE); - /* Swing the top pointer to the first element in the list and * make the last element point to the old top. */ From patchwork Fri Sep 25 17:43:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Lariau X-Patchwork-Id: 78863 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id B7D84A04C0; Fri, 25 Sep 2020 19:45:11 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BC34A1E9F9; Fri, 25 Sep 2020 19:44:19 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 198501E9D6 for ; Fri, 25 Sep 2020 19:44:14 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 99D2F101E; Fri, 25 Sep 2020 10:44:12 -0700 (PDT) Received: from localhost.localdomain (unknown [10.57.54.55]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C651B3F718; Fri, 25 Sep 2020 10:44:11 -0700 (PDT) From: Steven Lariau To: Gage Eads , Olivier Matz Cc: dev@dpdk.org, nd@arm.com, Steven Lariau Date: Fri, 25 Sep 2020 18:43:37 +0100 Message-Id: <20200925174340.10014-4-steven.lariau@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200925174340.10014-1-steven.lariau@arm.com> References: <20200911152938.8019-1-steven.lariau@arm.com> <20200925174340.10014-1-steven.lariau@arm.com> Subject: [dpdk-dev] [PATCH v2 3/5] lib/stack: remove redundant orderings for list->len X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The load-acquire of list->len on pop function is redundant. Only the CAS success needs to be load-acquire. It synchronizes with the store release in push, to ensure that the updated head is visible when the new length is visible. Without this, one thread in pop could see the increased length but the old list, which doesn't have enough items yet for pop to succeed. Signed-off-by: Steven Lariau Reviewed-by: Dharmik Thakkar Reviewed-by: Ruifeng Wang Acked-by: Gage Eads --- lib/librte_stack/rte_stack_lf_c11.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/librte_stack/rte_stack_lf_c11.h b/lib/librte_stack/rte_stack_lf_c11.h index 82b7287f1..2bc639419 100644 --- a/lib/librte_stack/rte_stack_lf_c11.h +++ b/lib/librte_stack/rte_stack_lf_c11.h @@ -80,7 +80,7 @@ __rte_stack_lf_pop_elems(struct rte_stack_lf_list *list, int success; /* Reserve num elements, if available */ - len = __atomic_load_n(&list->len, __ATOMIC_ACQUIRE); + len = __atomic_load_n(&list->len, __ATOMIC_RELAXED); while (1) { /* Does the list contain enough elements? */ @@ -91,7 +91,7 @@ __rte_stack_lf_pop_elems(struct rte_stack_lf_list *list, if (__atomic_compare_exchange_n(&list->len, &len, len - num, 1, __ATOMIC_ACQUIRE, - __ATOMIC_ACQUIRE)) + __ATOMIC_RELAXED)) break; } From patchwork Fri Sep 25 17:43:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Lariau X-Patchwork-Id: 78864 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id CFA0FA04C0; Fri, 25 Sep 2020 19:45:32 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id A73E81EA02; Fri, 25 Sep 2020 19:44:29 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 7F2521E9E9; Fri, 25 Sep 2020 19:44:17 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CADDF101E; Fri, 25 Sep 2020 10:44:14 -0700 (PDT) Received: from localhost.localdomain (unknown [10.57.54.55]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CD3F63F718; Fri, 25 Sep 2020 10:44:13 -0700 (PDT) From: Steven Lariau To: Gage Eads , Olivier Matz , Honnappa Nagarahalli Cc: dev@dpdk.org, nd@arm.com, Steven Lariau , stable@dpdk.org Date: Fri, 25 Sep 2020 18:43:38 +0100 Message-Id: <20200925174340.10014-5-steven.lariau@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200925174340.10014-1-steven.lariau@arm.com> References: <20200911152938.8019-1-steven.lariau@arm.com> <20200925174340.10014-1-steven.lariau@arm.com> Subject: [dpdk-dev] [PATCH v2 4/5] lib/stack: reload head when pop fails X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" List head must be loaded right before continue (when failed to find the new head). Without this, one thread might keep trying and failing to pop items without ever loading the new correct head. Fixes: 7e6e609939a8 ("stack: add C11 atomic implementation") Cc: gage.eads@intel.com Cc: stable@dpdk.org Signed-off-by: Steven Lariau Reviewed-by: Dharmik Thakkar Reviewed-by: Ruifeng Wang Acked-by: Gage Eads --- lib/librte_stack/rte_stack_lf_c11.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/librte_stack/rte_stack_lf_c11.h b/lib/librte_stack/rte_stack_lf_c11.h index 2bc639419..adb9f590d 100644 --- a/lib/librte_stack/rte_stack_lf_c11.h +++ b/lib/librte_stack/rte_stack_lf_c11.h @@ -133,8 +133,10 @@ __rte_stack_lf_pop_elems(struct rte_stack_lf_list *list, /* If NULL was encountered, the list was modified while * traversing it. Retry. */ - if (i != num) + if (i != num) { + old_head = list->head; continue; + } new_head.top = tmp; new_head.cnt = old_head.cnt + 1; From patchwork Fri Sep 25 17:43:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Lariau X-Patchwork-Id: 78865 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 83BA6A04C0; Fri, 25 Sep 2020 19:45:50 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 43AFC1EA16; Fri, 25 Sep 2020 19:44:31 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id E2FEE1E9F5 for ; Fri, 25 Sep 2020 19:44:18 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CBB9A1396; Fri, 25 Sep 2020 10:44:16 -0700 (PDT) Received: from localhost.localdomain (unknown [10.57.54.55]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EEBCA3F718; Fri, 25 Sep 2020 10:44:15 -0700 (PDT) From: Steven Lariau To: Gage Eads , Olivier Matz Cc: dev@dpdk.org, nd@arm.com, Steven Lariau Date: Fri, 25 Sep 2020 18:43:39 +0100 Message-Id: <20200925174340.10014-6-steven.lariau@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200925174340.10014-1-steven.lariau@arm.com> References: <20200911152938.8019-1-steven.lariau@arm.com> <20200925174340.10014-1-steven.lariau@arm.com> Subject: [dpdk-dev] [PATCH v2 5/5] lib/stack: remove pop cas release ordering X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Replace the store-release by relaxed for the CAS success at the end of pop. Release isn't needed, because there is not write to data that need to be synchronized. The only preceding write is when the length is decreased, but the length CAS loop already ensures the right synchronization. The situation to avoid is when a thread sees the old length but the new list, that doesn't have enough items for pop to success. But the CAS success on length before the pop loop ensures any core reads and updates the latest length, preventing this situation. The store-release is also used to make sure that the items are read before the head is updated, in order to prevent a core in pop to read an incorrect value because another core rewrites it with push. But this isn't needed, because items are read only when removed from the used list. Right after this, they are pushed to the free list, and the store-release in push makes sure the items are read before they are visible in the free list. Signed-off-by: Steven Lariau Reviewed-by: Dharmik Thakkar Reviewed-by: Ruifeng Wang Acked-by: Gage Eads --- lib/librte_stack/rte_stack_lf_c11.h | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/lib/librte_stack/rte_stack_lf_c11.h b/lib/librte_stack/rte_stack_lf_c11.h index adb9f590d..8403196d5 100644 --- a/lib/librte_stack/rte_stack_lf_c11.h +++ b/lib/librte_stack/rte_stack_lf_c11.h @@ -141,11 +141,25 @@ __rte_stack_lf_pop_elems(struct rte_stack_lf_list *list, new_head.top = tmp; new_head.cnt = old_head.cnt + 1; + /* + * The CAS should have release semantics to ensure that + * items are read before the head is updated. + * But this function is internal, and items are read + * only when __rte_stack_lf_pop calls this function to + * pop items from used list. + * Then, those items are pushed to the free list. + * Push uses a CAS store-release on head, which makes + * sure that items are read before they are pushed to + * the free list, without need for a CAS release here. + * This CAS could also be used to ensure that the new + * length is visible before the head update, but + * acquire semantics on the length update is enough. + */ success = rte_atomic128_cmp_exchange( (rte_int128_t *)&list->head, (rte_int128_t *)&old_head, (rte_int128_t *)&new_head, - 0, __ATOMIC_RELEASE, + 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED); } while (success == 0);