From patchwork Fri Jul 10 02:22:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ruifeng Wang X-Patchwork-Id: 73674 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0963CA0526; Fri, 10 Jul 2020 04:22:49 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 293931DC51; Fri, 10 Jul 2020 04:22:45 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 16E6E1DC46 for ; Fri, 10 Jul 2020 04:22:44 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 87C3931B; Thu, 9 Jul 2020 19:22:43 -0700 (PDT) Received: from net-arm-thunderx2-02.shanghai.arm.com (net-arm-thunderx2-02.shanghai.arm.com [10.169.210.119]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DB3DC3F9AB; Thu, 9 Jul 2020 19:22:39 -0700 (PDT) From: Ruifeng Wang To: Bruce Richardson , Vladimir Medvedkin , John McNamara , Marko Kovacevic , Ray Kinsella , Neil Horman Cc: dev@dpdk.org, konstantin.ananyev@intel.com, honnappa.nagarahalli@arm.com, nd@arm.com, Ruifeng Wang Date: Fri, 10 Jul 2020 10:22:25 +0800 Message-Id: <20200710022227.103963-2-ruifeng.wang@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200710022227.103963-1-ruifeng.wang@arm.com> References: <20190906094534.36060-1-ruifeng.wang@arm.com> <20200710022227.103963-1-ruifeng.wang@arm.com> Subject: [dpdk-dev] [PATCH v10 1/3] lib/lpm: integrate RCU QSBR X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Currently, the tbl8 group is freed even though the readers might be using the tbl8 group entries. The freed tbl8 group can be reallocated quickly. This results in incorrect lookup results. RCU QSBR process is integrated for safe tbl8 group reclaim. Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries. To avoid ABI breakage, a struct __rte_lpm is created for lpm library internal use. This struct wraps rte_lpm that has been exposed and also includes members that don't need to be exposed such as RCU related config. Signed-off-by: Ruifeng Wang Reviewed-by: Honnappa Nagarahalli Acked-by: Ray Kinsella Acked-by: Vladimir Medvedkin --- doc/guides/prog_guide/lpm_lib.rst | 32 ++++++ lib/librte_lpm/Makefile | 2 +- lib/librte_lpm/meson.build | 1 + lib/librte_lpm/rte_lpm.c | 165 +++++++++++++++++++++++++---- lib/librte_lpm/rte_lpm.h | 53 +++++++++ lib/librte_lpm/rte_lpm_version.map | 6 ++ 6 files changed, 237 insertions(+), 22 deletions(-) diff --git a/doc/guides/prog_guide/lpm_lib.rst b/doc/guides/prog_guide/lpm_lib.rst index 1609a57d0..03945904b 100644 --- a/doc/guides/prog_guide/lpm_lib.rst +++ b/doc/guides/prog_guide/lpm_lib.rst @@ -145,6 +145,38 @@ depending on whether we need to move to the next table or not. Prefix expansion is one of the keys of this algorithm, since it improves the speed dramatically by adding redundancy. +Deletion +~~~~~~~~ + +When deleting a rule, a replacement rule is searched for. Replacement rule is an existing rule that has +the longest prefix match with the rule to be deleted, but has shorter prefix. + +If a replacement rule is found, target tbl24 and tbl8 entries are updated to have the same depth and next hop +value with the replacement rule. + +If no replacement rule can be found, target tbl24 and tbl8 entries will be cleared. + +Prefix expansion is performed if the rule's depth is not exactly 24 bits or 32 bits. + +After deleting a rule, a group of tbl8s that belongs to the same tbl24 entry are freed in following cases: + +* All tbl8s in the group are empty . + +* All tbl8s in the group have the same values and with depth no greater than 24. + +Free of tbl8s have different behaviors: + +* If RCU is not used, tbl8s are cleared and reclaimed immediately. + +* If RCU is used, tbl8s are reclaimed when readers are in quiescent state. + +When the LPM is not using RCU, tbl8 group can be freed immediately even though the readers might be using +the tbl8 group entries. This might result in incorrect lookup results. + +RCU QSBR process is integrated for safe tbl8 group reclamation. Application has certain responsibilities +while using this feature. Please refer to resource reclamation framework of :ref:`RCU library ` +for more details. + Lookup ~~~~~~ diff --git a/lib/librte_lpm/Makefile b/lib/librte_lpm/Makefile index d682785b6..6f06c5c03 100644 --- a/lib/librte_lpm/Makefile +++ b/lib/librte_lpm/Makefile @@ -8,7 +8,7 @@ LIB = librte_lpm.a CFLAGS += -O3 CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -LDLIBS += -lrte_eal -lrte_hash +LDLIBS += -lrte_eal -lrte_hash -lrte_rcu EXPORT_MAP := rte_lpm_version.map diff --git a/lib/librte_lpm/meson.build b/lib/librte_lpm/meson.build index 021ac6d8d..6cfc083c5 100644 --- a/lib/librte_lpm/meson.build +++ b/lib/librte_lpm/meson.build @@ -7,3 +7,4 @@ headers = files('rte_lpm.h', 'rte_lpm6.h') # without worrying about which architecture we actually need headers += files('rte_lpm_altivec.h', 'rte_lpm_neon.h', 'rte_lpm_sse.h') deps += ['hash'] +deps += ['rcu'] diff --git a/lib/librte_lpm/rte_lpm.c b/lib/librte_lpm/rte_lpm.c index 38ab512a4..2d687c372 100644 --- a/lib/librte_lpm/rte_lpm.c +++ b/lib/librte_lpm/rte_lpm.c @@ -1,5 +1,6 @@ /* SPDX-License-Identifier: BSD-3-Clause * Copyright(c) 2010-2014 Intel Corporation + * Copyright(c) 2020 Arm Limited */ #include @@ -39,6 +40,17 @@ enum valid_flag { VALID }; +/** @internal LPM structure. */ +struct __rte_lpm { + /* LPM metadata. */ + struct rte_lpm lpm; + + /* RCU config. */ + struct rte_rcu_qsbr *v; /* RCU QSBR variable. */ + enum rte_lpm_qsbr_mode rcu_mode;/* Blocking, defer queue. */ + struct rte_rcu_qsbr_dq *dq; /* RCU QSBR defer queue. */ +}; + /* Macro to enable/disable run-time checks. */ #if defined(RTE_LIBRTE_LPM_DEBUG) #include @@ -122,6 +134,7 @@ rte_lpm_create(const char *name, int socket_id, const struct rte_lpm_config *config) { char mem_name[RTE_LPM_NAMESIZE]; + struct __rte_lpm *internal_lpm; struct rte_lpm *lpm = NULL; struct rte_tailq_entry *te; uint32_t mem_size, rules_size, tbl8s_size; @@ -140,12 +153,6 @@ rte_lpm_create(const char *name, int socket_id, snprintf(mem_name, sizeof(mem_name), "LPM_%s", name); - /* Determine the amount of memory to allocate. */ - mem_size = sizeof(*lpm); - rules_size = sizeof(struct rte_lpm_rule) * config->max_rules; - tbl8s_size = (sizeof(struct rte_lpm_tbl_entry) * - RTE_LPM_TBL8_GROUP_NUM_ENTRIES * config->number_tbl8s); - rte_mcfg_tailq_write_lock(); /* guarantee there's no existing */ @@ -161,6 +168,12 @@ rte_lpm_create(const char *name, int socket_id, goto exit; } + /* Determine the amount of memory to allocate. */ + mem_size = sizeof(*internal_lpm); + rules_size = sizeof(struct rte_lpm_rule) * config->max_rules; + tbl8s_size = (sizeof(struct rte_lpm_tbl_entry) * + RTE_LPM_TBL8_GROUP_NUM_ENTRIES * config->number_tbl8s); + /* allocate tailq entry */ te = rte_zmalloc("LPM_TAILQ_ENTRY", sizeof(*te), 0); if (te == NULL) { @@ -170,21 +183,23 @@ rte_lpm_create(const char *name, int socket_id, } /* Allocate memory to store the LPM data structures. */ - lpm = rte_zmalloc_socket(mem_name, mem_size, + internal_lpm = rte_zmalloc_socket(mem_name, mem_size, RTE_CACHE_LINE_SIZE, socket_id); - if (lpm == NULL) { + if (internal_lpm == NULL) { RTE_LOG(ERR, LPM, "LPM memory allocation failed\n"); rte_free(te); rte_errno = ENOMEM; goto exit; } + lpm = &internal_lpm->lpm; lpm->rules_tbl = rte_zmalloc_socket(NULL, (size_t)rules_size, RTE_CACHE_LINE_SIZE, socket_id); if (lpm->rules_tbl == NULL) { RTE_LOG(ERR, LPM, "LPM rules_tbl memory allocation failed\n"); - rte_free(lpm); + rte_free(internal_lpm); + internal_lpm = NULL; lpm = NULL; rte_free(te); rte_errno = ENOMEM; @@ -197,7 +212,8 @@ rte_lpm_create(const char *name, int socket_id, if (lpm->tbl8 == NULL) { RTE_LOG(ERR, LPM, "LPM tbl8 memory allocation failed\n"); rte_free(lpm->rules_tbl); - rte_free(lpm); + rte_free(internal_lpm); + internal_lpm = NULL; lpm = NULL; rte_free(te); rte_errno = ENOMEM; @@ -225,6 +241,7 @@ rte_lpm_create(const char *name, int socket_id, void rte_lpm_free(struct rte_lpm *lpm) { + struct __rte_lpm *internal_lpm; struct rte_lpm_list *lpm_list; struct rte_tailq_entry *te; @@ -246,12 +263,84 @@ rte_lpm_free(struct rte_lpm *lpm) rte_mcfg_tailq_write_unlock(); + internal_lpm = container_of(lpm, struct __rte_lpm, lpm); + if (internal_lpm->dq) + rte_rcu_qsbr_dq_delete(internal_lpm->dq); rte_free(lpm->tbl8); rte_free(lpm->rules_tbl); rte_free(lpm); rte_free(te); } +static void +__lpm_rcu_qsbr_free_resource(void *p, void *data, unsigned int n) +{ + struct rte_lpm_tbl_entry zero_tbl8_entry = {0}; + uint32_t tbl8_group_index = *(uint32_t *)data; + struct rte_lpm_tbl_entry *tbl8 = ((struct rte_lpm *)p)->tbl8; + + RTE_SET_USED(n); + /* Set tbl8 group invalid */ + __atomic_store(&tbl8[tbl8_group_index], &zero_tbl8_entry, + __ATOMIC_RELAXED); +} + +/* Associate QSBR variable with an LPM object. + */ +int +rte_lpm_rcu_qsbr_add(struct rte_lpm *lpm, struct rte_lpm_rcu_config *cfg, + struct rte_rcu_qsbr_dq **dq) +{ + struct __rte_lpm *internal_lpm; + char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE]; + struct rte_rcu_qsbr_dq_parameters params = {0}; + + if (lpm == NULL || cfg == NULL) { + rte_errno = EINVAL; + return 1; + } + + internal_lpm = container_of(lpm, struct __rte_lpm, lpm); + if (internal_lpm->v != NULL) { + rte_errno = EEXIST; + return 1; + } + + if (cfg->mode == RTE_LPM_QSBR_MODE_SYNC) { + /* No other things to do. */ + } else if (cfg->mode == RTE_LPM_QSBR_MODE_DQ) { + /* Init QSBR defer queue. */ + snprintf(rcu_dq_name, sizeof(rcu_dq_name), + "LPM_RCU_%s", lpm->name); + params.name = rcu_dq_name; + params.size = cfg->dq_size; + if (params.size == 0) + params.size = lpm->number_tbl8s; + params.trigger_reclaim_limit = cfg->reclaim_thd; + params.max_reclaim_size = cfg->reclaim_max; + if (params.max_reclaim_size == 0) + params.max_reclaim_size = RTE_LPM_RCU_DQ_RECLAIM_MAX; + params.esize = sizeof(uint32_t); /* tbl8 group index */ + params.free_fn = __lpm_rcu_qsbr_free_resource; + params.p = lpm; + params.v = cfg->v; + internal_lpm->dq = rte_rcu_qsbr_dq_create(¶ms); + if (internal_lpm->dq == NULL) { + RTE_LOG(ERR, LPM, "LPM defer queue creation failed\n"); + return 1; + } + if (dq) + *dq = internal_lpm->dq; + } else { + rte_errno = EINVAL; + return 1; + } + internal_lpm->rcu_mode = cfg->mode; + internal_lpm->v = cfg->v; + + return 0; +} + /* * Adds a rule to the rule table. * @@ -394,14 +483,15 @@ rule_find(struct rte_lpm *lpm, uint32_t ip_masked, uint8_t depth) * Find, clean and allocate a tbl8. */ static int32_t -tbl8_alloc(struct rte_lpm_tbl_entry *tbl8, uint32_t number_tbl8s) +_tbl8_alloc(struct rte_lpm *lpm) { uint32_t group_idx; /* tbl8 group index. */ struct rte_lpm_tbl_entry *tbl8_entry; /* Scan through tbl8 to find a free (i.e. INVALID) tbl8 group. */ - for (group_idx = 0; group_idx < number_tbl8s; group_idx++) { - tbl8_entry = &tbl8[group_idx * RTE_LPM_TBL8_GROUP_NUM_ENTRIES]; + for (group_idx = 0; group_idx < lpm->number_tbl8s; group_idx++) { + tbl8_entry = &lpm->tbl8[group_idx * + RTE_LPM_TBL8_GROUP_NUM_ENTRIES]; /* If a free tbl8 group is found clean it and set as VALID. */ if (!tbl8_entry->valid_group) { struct rte_lpm_tbl_entry new_tbl8_entry = { @@ -427,14 +517,47 @@ tbl8_alloc(struct rte_lpm_tbl_entry *tbl8, uint32_t number_tbl8s) return -ENOSPC; } +static int32_t +tbl8_alloc(struct rte_lpm *lpm) +{ + struct __rte_lpm *internal_lpm = container_of(lpm, + struct __rte_lpm, lpm); + int32_t group_idx; /* tbl8 group index. */ + + group_idx = _tbl8_alloc(lpm); + if (group_idx == -ENOSPC && internal_lpm->dq != NULL) { + /* If there are no tbl8 groups try to reclaim one. */ + if (rte_rcu_qsbr_dq_reclaim(internal_lpm->dq, 1, + NULL, NULL, NULL) == 0) + group_idx = _tbl8_alloc(lpm); + } + + return group_idx; +} + static void -tbl8_free(struct rte_lpm_tbl_entry *tbl8, uint32_t tbl8_group_start) +tbl8_free(struct rte_lpm *lpm, uint32_t tbl8_group_start) { - /* Set tbl8 group invalid*/ + struct __rte_lpm *internal_lpm = container_of(lpm, + struct __rte_lpm, lpm); struct rte_lpm_tbl_entry zero_tbl8_entry = {0}; - __atomic_store(&tbl8[tbl8_group_start], &zero_tbl8_entry, - __ATOMIC_RELAXED); + if (internal_lpm->v == NULL) { + /* Set tbl8 group invalid*/ + __atomic_store(&lpm->tbl8[tbl8_group_start], &zero_tbl8_entry, + __ATOMIC_RELAXED); + } else if (internal_lpm->rcu_mode == RTE_LPM_QSBR_MODE_SYNC) { + /* Wait for quiescent state change. */ + rte_rcu_qsbr_synchronize(internal_lpm->v, + RTE_QSBR_THRID_INVALID); + /* Set tbl8 group invalid*/ + __atomic_store(&lpm->tbl8[tbl8_group_start], &zero_tbl8_entry, + __ATOMIC_RELAXED); + } else if (internal_lpm->rcu_mode == RTE_LPM_QSBR_MODE_DQ) { + /* Push into QSBR defer queue. */ + rte_rcu_qsbr_dq_enqueue(internal_lpm->dq, + (void *)&tbl8_group_start); + } } static __rte_noinline int32_t @@ -523,7 +646,7 @@ add_depth_big(struct rte_lpm *lpm, uint32_t ip_masked, uint8_t depth, if (!lpm->tbl24[tbl24_index].valid) { /* Search for a free tbl8 group. */ - tbl8_group_index = tbl8_alloc(lpm->tbl8, lpm->number_tbl8s); + tbl8_group_index = tbl8_alloc(lpm); /* Check tbl8 allocation was successful. */ if (tbl8_group_index < 0) { @@ -569,7 +692,7 @@ add_depth_big(struct rte_lpm *lpm, uint32_t ip_masked, uint8_t depth, } /* If valid entry but not extended calculate the index into Table8. */ else if (lpm->tbl24[tbl24_index].valid_group == 0) { /* Search for free tbl8 group. */ - tbl8_group_index = tbl8_alloc(lpm->tbl8, lpm->number_tbl8s); + tbl8_group_index = tbl8_alloc(lpm); if (tbl8_group_index < 0) { return tbl8_group_index; @@ -977,7 +1100,7 @@ delete_depth_big(struct rte_lpm *lpm, uint32_t ip_masked, */ lpm->tbl24[tbl24_index].valid = 0; __atomic_thread_fence(__ATOMIC_RELEASE); - tbl8_free(lpm->tbl8, tbl8_group_start); + tbl8_free(lpm, tbl8_group_start); } else if (tbl8_recycle_index > -1) { /* Update tbl24 entry. */ struct rte_lpm_tbl_entry new_tbl24_entry = { @@ -993,7 +1116,7 @@ delete_depth_big(struct rte_lpm *lpm, uint32_t ip_masked, __atomic_store(&lpm->tbl24[tbl24_index], &new_tbl24_entry, __ATOMIC_RELAXED); __atomic_thread_fence(__ATOMIC_RELEASE); - tbl8_free(lpm->tbl8, tbl8_group_start); + tbl8_free(lpm, tbl8_group_start); } #undef group_idx return 0; diff --git a/lib/librte_lpm/rte_lpm.h b/lib/librte_lpm/rte_lpm.h index b9d49ac87..a9568fcdd 100644 --- a/lib/librte_lpm/rte_lpm.h +++ b/lib/librte_lpm/rte_lpm.h @@ -1,5 +1,6 @@ /* SPDX-License-Identifier: BSD-3-Clause * Copyright(c) 2010-2014 Intel Corporation + * Copyright(c) 2020 Arm Limited */ #ifndef _RTE_LPM_H_ @@ -20,6 +21,7 @@ #include #include #include +#include #ifdef __cplusplus extern "C" { @@ -62,6 +64,17 @@ extern "C" { /** Bitmask used to indicate successful lookup */ #define RTE_LPM_LOOKUP_SUCCESS 0x01000000 +/** @internal Default RCU defer queue entries to reclaim in one go. */ +#define RTE_LPM_RCU_DQ_RECLAIM_MAX 16 + +/** RCU reclamation modes */ +enum rte_lpm_qsbr_mode { + /** Create defer queue for reclaim. */ + RTE_LPM_QSBR_MODE_DQ = 0, + /** Use blocking mode reclaim. No defer queue created. */ + RTE_LPM_QSBR_MODE_SYNC +}; + #if RTE_BYTE_ORDER == RTE_LITTLE_ENDIAN /** @internal Tbl24 entry structure. */ __extension__ @@ -132,6 +145,22 @@ struct rte_lpm { struct rte_lpm_rule *rules_tbl; /**< LPM rules. */ }; +/** LPM RCU QSBR configuration structure. */ +struct rte_lpm_rcu_config { + struct rte_rcu_qsbr *v; /* RCU QSBR variable. */ + /* Mode of RCU QSBR. RTE_LPM_QSBR_MODE_xxx + * '0' for default: create defer queue for reclaim. + */ + enum rte_lpm_qsbr_mode mode; + uint32_t dq_size; /* RCU defer queue size. + * default: lpm->number_tbl8s. + */ + uint32_t reclaim_thd; /* Threshold to trigger auto reclaim. */ + uint32_t reclaim_max; /* Max entries to reclaim in one go. + * default: RTE_LPM_RCU_DQ_RECLAIM_MAX. + */ +}; + /** * Create an LPM object. * @@ -179,6 +208,30 @@ rte_lpm_find_existing(const char *name); void rte_lpm_free(struct rte_lpm *lpm); +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Associate RCU QSBR variable with an LPM object. + * + * @param lpm + * the lpm object to add RCU QSBR + * @param cfg + * RCU QSBR configuration + * @param dq + * handler of created RCU QSBR defer queue + * @return + * On success - 0 + * On error - 1 with error code set in rte_errno. + * Possible rte_errno codes are: + * - EINVAL - invalid pointer + * - EEXIST - already added QSBR + * - ENOMEM - memory allocation failure + */ +__rte_experimental +int rte_lpm_rcu_qsbr_add(struct rte_lpm *lpm, struct rte_lpm_rcu_config *cfg, + struct rte_rcu_qsbr_dq **dq); + /** * Add a rule to the LPM table. * diff --git a/lib/librte_lpm/rte_lpm_version.map b/lib/librte_lpm/rte_lpm_version.map index 500f58b80..bfccd7eac 100644 --- a/lib/librte_lpm/rte_lpm_version.map +++ b/lib/librte_lpm/rte_lpm_version.map @@ -21,3 +21,9 @@ DPDK_20.0 { local: *; }; + +EXPERIMENTAL { + global: + + rte_lpm_rcu_qsbr_add; +}; From patchwork Fri Jul 10 02:22:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ruifeng Wang X-Patchwork-Id: 73675 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 54965A0526; Fri, 10 Jul 2020 04:22:57 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7151C1DC44; Fri, 10 Jul 2020 04:22:49 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 5C18A1D702 for ; Fri, 10 Jul 2020 04:22:48 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DC425D6E; Thu, 9 Jul 2020 19:22:47 -0700 (PDT) Received: from net-arm-thunderx2-02.shanghai.arm.com (net-arm-thunderx2-02.shanghai.arm.com [10.169.210.119]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4B1CD3F9AB; Thu, 9 Jul 2020 19:22:45 -0700 (PDT) From: Ruifeng Wang To: Bruce Richardson , Vladimir Medvedkin Cc: dev@dpdk.org, mdr@ashroe.eu, konstantin.ananyev@intel.com, honnappa.nagarahalli@arm.com, nd@arm.com, Ruifeng Wang Date: Fri, 10 Jul 2020 10:22:26 +0800 Message-Id: <20200710022227.103963-3-ruifeng.wang@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200710022227.103963-1-ruifeng.wang@arm.com> References: <20190906094534.36060-1-ruifeng.wang@arm.com> <20200710022227.103963-1-ruifeng.wang@arm.com> Subject: [dpdk-dev] [PATCH v10 2/3] test/lpm: add LPM RCU integration functional tests X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add positive and negative tests for API rte_lpm_rcu_qsbr_add. Also test LPM library behavior when RCU QSBR is enabled. Signed-off-by: Ruifeng Wang Reviewed-by: Gavin Hu Reviewed-by: Honnappa Nagarahalli Acked-by: Vladimir Medvedkin --- app/test/test_lpm.c | 291 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 290 insertions(+), 1 deletion(-) diff --git a/app/test/test_lpm.c b/app/test/test_lpm.c index 3a3fd097f..8330501f0 100644 --- a/app/test/test_lpm.c +++ b/app/test/test_lpm.c @@ -8,6 +8,7 @@ #include #include +#include #include "test.h" #include "test_xmmt_ops.h" @@ -40,6 +41,9 @@ static int32_t test15(void); static int32_t test16(void); static int32_t test17(void); static int32_t test18(void); +static int32_t test19(void); +static int32_t test20(void); +static int32_t test21(void); rte_lpm_test tests[] = { /* Test Cases */ @@ -61,7 +65,10 @@ rte_lpm_test tests[] = { test15, test16, test17, - test18 + test18, + test19, + test20, + test21 }; #define MAX_DEPTH 32 @@ -1265,6 +1272,288 @@ test18(void) return PASS; } +/* + * rte_lpm_rcu_qsbr_add positive and negative tests. + * - Add RCU QSBR variable to LPM + * - Add another RCU QSBR variable to LPM + * - Check returns + */ +int32_t +test19(void) +{ + struct rte_lpm *lpm = NULL; + struct rte_lpm_config config; + size_t sz; + struct rte_rcu_qsbr *qsv; + struct rte_rcu_qsbr *qsv2; + int32_t status; + struct rte_lpm_rcu_config rcu_cfg = {0}; + + config.max_rules = MAX_RULES; + config.number_tbl8s = NUMBER_TBL8S; + config.flags = 0; + + lpm = rte_lpm_create(__func__, SOCKET_ID_ANY, &config); + TEST_LPM_ASSERT(lpm != NULL); + + /* Create RCU QSBR variable */ + sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE); + qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz, + RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY); + TEST_LPM_ASSERT(qsv != NULL); + + status = rte_rcu_qsbr_init(qsv, RTE_MAX_LCORE); + TEST_LPM_ASSERT(status == 0); + + rcu_cfg.v = qsv; + /* Invalid QSBR mode */ + rcu_cfg.mode = 2; + status = rte_lpm_rcu_qsbr_add(lpm, &rcu_cfg, NULL); + TEST_LPM_ASSERT(status != 0); + + rcu_cfg.mode = RTE_LPM_QSBR_MODE_DQ; + /* Attach RCU QSBR to LPM table */ + status = rte_lpm_rcu_qsbr_add(lpm, &rcu_cfg, NULL); + TEST_LPM_ASSERT(status == 0); + + /* Create and attach another RCU QSBR to LPM table */ + qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz, + RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY); + TEST_LPM_ASSERT(qsv2 != NULL); + + rcu_cfg.v = qsv2; + rcu_cfg.mode = RTE_LPM_QSBR_MODE_SYNC; + status = rte_lpm_rcu_qsbr_add(lpm, &rcu_cfg, NULL); + TEST_LPM_ASSERT(status != 0); + + rte_lpm_free(lpm); + rte_free(qsv); + rte_free(qsv2); + + return PASS; +} + +/* + * rte_lpm_rcu_qsbr_add DQ mode functional test. + * Reader and writer are in the same thread in this test. + * - Create LPM which supports 1 tbl8 group at max + * - Add RCU QSBR variable to LPM + * - Add a rule with depth=28 (> 24) + * - Register a reader thread (not a real thread) + * - Reader lookup existing rule + * - Writer delete the rule + * - Reader lookup the rule + * - Writer re-add the rule (no available tbl8 group) + * - Reader report quiescent state and unregister + * - Writer re-add the rule + * - Reader lookup the rule + */ +int32_t +test20(void) +{ + struct rte_lpm *lpm = NULL; + struct rte_lpm_config config; + size_t sz; + struct rte_rcu_qsbr *qsv; + int32_t status; + uint32_t ip, next_hop, next_hop_return; + uint8_t depth; + struct rte_lpm_rcu_config rcu_cfg = {0}; + + config.max_rules = MAX_RULES; + config.number_tbl8s = 1; + config.flags = 0; + + lpm = rte_lpm_create(__func__, SOCKET_ID_ANY, &config); + TEST_LPM_ASSERT(lpm != NULL); + + /* Create RCU QSBR variable */ + sz = rte_rcu_qsbr_get_memsize(1); + qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz, + RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY); + TEST_LPM_ASSERT(qsv != NULL); + + status = rte_rcu_qsbr_init(qsv, 1); + TEST_LPM_ASSERT(status == 0); + + rcu_cfg.v = qsv; + rcu_cfg.mode = RTE_LPM_QSBR_MODE_DQ; + /* Attach RCU QSBR to LPM table */ + status = rte_lpm_rcu_qsbr_add(lpm, &rcu_cfg, NULL); + TEST_LPM_ASSERT(status == 0); + + ip = RTE_IPV4(192, 0, 2, 100); + depth = 28; + next_hop = 1; + status = rte_lpm_add(lpm, ip, depth, next_hop); + TEST_LPM_ASSERT(status == 0); + TEST_LPM_ASSERT(lpm->tbl24[ip>>8].valid_group); + + /* Register pseudo reader */ + status = rte_rcu_qsbr_thread_register(qsv, 0); + TEST_LPM_ASSERT(status == 0); + rte_rcu_qsbr_thread_online(qsv, 0); + + status = rte_lpm_lookup(lpm, ip, &next_hop_return); + TEST_LPM_ASSERT(status == 0); + TEST_LPM_ASSERT(next_hop_return == next_hop); + + /* Writer update */ + status = rte_lpm_delete(lpm, ip, depth); + TEST_LPM_ASSERT(status == 0); + TEST_LPM_ASSERT(!lpm->tbl24[ip>>8].valid); + + status = rte_lpm_lookup(lpm, ip, &next_hop_return); + TEST_LPM_ASSERT(status != 0); + + status = rte_lpm_add(lpm, ip, depth, next_hop); + TEST_LPM_ASSERT(status != 0); + + /* Reader quiescent */ + rte_rcu_qsbr_quiescent(qsv, 0); + + status = rte_lpm_add(lpm, ip, depth, next_hop); + TEST_LPM_ASSERT(status == 0); + + rte_rcu_qsbr_thread_offline(qsv, 0); + status = rte_rcu_qsbr_thread_unregister(qsv, 0); + TEST_LPM_ASSERT(status == 0); + + status = rte_lpm_lookup(lpm, ip, &next_hop_return); + TEST_LPM_ASSERT(status == 0); + TEST_LPM_ASSERT(next_hop_return == next_hop); + + rte_lpm_free(lpm); + rte_free(qsv); + + return PASS; +} + +static struct rte_lpm *g_lpm; +static struct rte_rcu_qsbr *g_v; +static uint32_t g_ip = RTE_IPV4(192, 0, 2, 100); +static volatile uint8_t writer_done; +/* Report quiescent state interval every 1024 lookups. Larger critical + * sections in reader will result in writer polling multiple times. + */ +#define QSBR_REPORTING_INTERVAL 1024 +#define WRITER_ITERATIONS 512 + +/* + * Reader thread using rte_lpm data structure with RCU. + */ +static int +test_lpm_rcu_qsbr_reader(void *arg) +{ + int i; + uint32_t next_hop_return = 0; + + RTE_SET_USED(arg); + /* Register this thread to report quiescent state */ + rte_rcu_qsbr_thread_register(g_v, 0); + rte_rcu_qsbr_thread_online(g_v, 0); + + do { + for (i = 0; i < QSBR_REPORTING_INTERVAL; i++) + rte_lpm_lookup(g_lpm, g_ip, &next_hop_return); + + /* Update quiescent state */ + rte_rcu_qsbr_quiescent(g_v, 0); + } while (!writer_done); + + rte_rcu_qsbr_thread_offline(g_v, 0); + rte_rcu_qsbr_thread_unregister(g_v, 0); + + return 0; +} + +/* + * rte_lpm_rcu_qsbr_add sync mode functional test. + * 1 Reader and 1 writer. They cannot be in the same thread in this test. + * - Create LPM which supports 1 tbl8 group at max + * - Add RCU QSBR variable with sync mode to LPM + * - Register a reader thread. Reader keeps looking up a specific rule. + * - Writer keeps adding and deleting a specific rule with depth=28 (> 24) + */ +int32_t +test21(void) +{ + struct rte_lpm_config config; + size_t sz; + int32_t status; + uint32_t i, next_hop; + uint8_t depth; + struct rte_lpm_rcu_config rcu_cfg = {0}; + + if (rte_lcore_count() < 2) { + printf("Not enough cores for %s, expecting at least 2\n", + __func__); + return TEST_SKIPPED; + } + + config.max_rules = MAX_RULES; + config.number_tbl8s = 1; + config.flags = 0; + + g_lpm = rte_lpm_create(__func__, SOCKET_ID_ANY, &config); + TEST_LPM_ASSERT(g_lpm != NULL); + + /* Create RCU QSBR variable */ + sz = rte_rcu_qsbr_get_memsize(1); + g_v = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz, + RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY); + TEST_LPM_ASSERT(g_v != NULL); + + status = rte_rcu_qsbr_init(g_v, 1); + TEST_LPM_ASSERT(status == 0); + + rcu_cfg.v = g_v; + rcu_cfg.mode = RTE_LPM_QSBR_MODE_SYNC; + /* Attach RCU QSBR to LPM table */ + status = rte_lpm_rcu_qsbr_add(g_lpm, &rcu_cfg, NULL); + TEST_LPM_ASSERT(status == 0); + + writer_done = 0; + /* Launch reader thread */ + rte_eal_remote_launch(test_lpm_rcu_qsbr_reader, NULL, + rte_get_next_lcore(-1, 1, 0)); + + depth = 28; + next_hop = 1; + status = rte_lpm_add(g_lpm, g_ip, depth, next_hop); + if (status != 0) { + printf("%s: Failed to add rule\n", __func__); + goto error; + } + + /* Writer update */ + for (i = 0; i < WRITER_ITERATIONS; i++) { + status = rte_lpm_delete(g_lpm, g_ip, depth); + if (status != 0) { + printf("%s: Failed to delete rule at iteration %d\n", + __func__, i); + goto error; + } + + status = rte_lpm_add(g_lpm, g_ip, depth, next_hop); + if (status != 0) { + printf("%s: Failed to add rule at iteration %d\n", + __func__, i); + goto error; + } + } + +error: + writer_done = 1; + /* Wait until reader exited. */ + rte_eal_mp_wait_lcore(); + + rte_lpm_free(g_lpm); + rte_free(g_v); + + return (status == 0) ? PASS : -1; +} + /* * Do all unit tests. */ From patchwork Fri Jul 10 02:22:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ruifeng Wang X-Patchwork-Id: 73676 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9C08CA0526; Fri, 10 Jul 2020 04:23:05 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C16AE1DD02; Fri, 10 Jul 2020 04:22:53 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 7A7DC1DD02 for ; Fri, 10 Jul 2020 04:22:52 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DAE56D6E; Thu, 9 Jul 2020 19:22:51 -0700 (PDT) Received: from net-arm-thunderx2-02.shanghai.arm.com (net-arm-thunderx2-02.shanghai.arm.com [10.169.210.119]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8959F3F9AB; Thu, 9 Jul 2020 19:22:49 -0700 (PDT) From: Ruifeng Wang To: Bruce Richardson , Vladimir Medvedkin Cc: dev@dpdk.org, mdr@ashroe.eu, konstantin.ananyev@intel.com, honnappa.nagarahalli@arm.com, nd@arm.com Date: Fri, 10 Jul 2020 10:22:27 +0800 Message-Id: <20200710022227.103963-4-ruifeng.wang@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200710022227.103963-1-ruifeng.wang@arm.com> References: <20190906094534.36060-1-ruifeng.wang@arm.com> <20200710022227.103963-1-ruifeng.wang@arm.com> Subject: [dpdk-dev] [PATCH v10 3/3] test/lpm: add RCU integration performance tests X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Honnappa Nagarahalli Add performance tests for RCU integration. The performance difference with and without RCU integration is very small (~1% to ~2%) on both Arm and x86 platforms. Signed-off-by: Honnappa Nagarahalli Reviewed-by: Gavin Hu Reviewed-by: Ruifeng Wang Acked-by: Vladimir Medvedkin --- app/test/test_lpm_perf.c | 492 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 489 insertions(+), 3 deletions(-) diff --git a/app/test/test_lpm_perf.c b/app/test/test_lpm_perf.c index 489719c40..dfe186426 100644 --- a/app/test/test_lpm_perf.c +++ b/app/test/test_lpm_perf.c @@ -1,5 +1,6 @@ /* SPDX-License-Identifier: BSD-3-Clause * Copyright(c) 2010-2014 Intel Corporation + * Copyright(c) 2020 Arm Limited */ #include @@ -10,12 +11,27 @@ #include #include #include +#include #include #include #include "test.h" #include "test_xmmt_ops.h" +struct rte_lpm *lpm; +static struct rte_rcu_qsbr *rv; +static volatile uint8_t writer_done; +static volatile uint32_t thr_id; +static uint64_t gwrite_cycles; +static uint64_t gwrites; +/* LPM APIs are not thread safe, use mutex to provide thread safety */ +static pthread_mutex_t lpm_mutex = PTHREAD_MUTEX_INITIALIZER; + +/* Report quiescent state interval every 1024 lookups. Larger critical + * sections in reader will result in writer polling multiple times. + */ +#define QSBR_REPORTING_INTERVAL 1024 + #define TEST_LPM_ASSERT(cond) do { \ if (!(cond)) { \ printf("Error at line %d: \n", __LINE__); \ @@ -24,6 +40,7 @@ } while(0) #define ITERATIONS (1 << 10) +#define RCU_ITERATIONS 10 #define BATCH_SIZE (1 << 12) #define BULK_SIZE 32 @@ -35,9 +52,13 @@ struct route_rule { }; static struct route_rule large_route_table[MAX_RULE_NUM]; +/* Route table for routes with depth > 24 */ +struct route_rule large_ldepth_route_table[MAX_RULE_NUM]; static uint32_t num_route_entries; +static uint32_t num_ldepth_route_entries; #define NUM_ROUTE_ENTRIES num_route_entries +#define NUM_LDEPTH_ROUTE_ENTRIES num_ldepth_route_entries enum { IP_CLASS_A, @@ -191,7 +212,7 @@ static void generate_random_rule_prefix(uint32_t ip_class, uint8_t depth) uint32_t ip_head_mask; uint32_t rule_num; uint32_t k; - struct route_rule *ptr_rule; + struct route_rule *ptr_rule, *ptr_ldepth_rule; if (ip_class == IP_CLASS_A) { /* IP Address class A */ fixed_bit_num = IP_HEAD_BIT_NUM_A; @@ -236,10 +257,20 @@ static void generate_random_rule_prefix(uint32_t ip_class, uint8_t depth) */ start = lrand48() & mask; ptr_rule = &large_route_table[num_route_entries]; + ptr_ldepth_rule = &large_ldepth_route_table[num_ldepth_route_entries]; for (k = 0; k < rule_num; k++) { ptr_rule->ip = (start << (RTE_LPM_MAX_DEPTH - depth)) | ip_head_mask; ptr_rule->depth = depth; + /* If the depth of the route is more than 24, store it + * in another table as well. + */ + if (depth > 24) { + ptr_ldepth_rule->ip = ptr_rule->ip; + ptr_ldepth_rule->depth = ptr_rule->depth; + ptr_ldepth_rule++; + num_ldepth_route_entries++; + } ptr_rule++; start = (start + step) & mask; } @@ -273,6 +304,7 @@ static void generate_large_route_rule_table(void) uint8_t depth; num_route_entries = 0; + num_ldepth_route_entries = 0; memset(large_route_table, 0, sizeof(large_route_table)); for (ip_class = IP_CLASS_A; ip_class <= IP_CLASS_C; ip_class++) { @@ -316,10 +348,460 @@ print_route_distribution(const struct route_rule *table, uint32_t n) printf("\n"); } +/* Check condition and return an error if true. */ +static uint16_t enabled_core_ids[RTE_MAX_LCORE]; +static unsigned int num_cores; + +/* Simple way to allocate thread ids in 0 to RTE_MAX_LCORE space */ +static inline uint32_t +alloc_thread_id(void) +{ + uint32_t tmp_thr_id; + + tmp_thr_id = __atomic_fetch_add(&thr_id, 1, __ATOMIC_RELAXED); + if (tmp_thr_id >= RTE_MAX_LCORE) + printf("Invalid thread id %u\n", tmp_thr_id); + + return tmp_thr_id; +} + +/* + * Reader thread using rte_lpm data structure without RCU. + */ +static int +test_lpm_reader(void *arg) +{ + int i; + uint32_t ip_batch[QSBR_REPORTING_INTERVAL]; + uint32_t next_hop_return = 0; + + RTE_SET_USED(arg); + do { + for (i = 0; i < QSBR_REPORTING_INTERVAL; i++) + ip_batch[i] = rte_rand(); + + for (i = 0; i < QSBR_REPORTING_INTERVAL; i++) + rte_lpm_lookup(lpm, ip_batch[i], &next_hop_return); + + } while (!writer_done); + + return 0; +} + +/* + * Reader thread using rte_lpm data structure with RCU. + */ +static int +test_lpm_rcu_qsbr_reader(void *arg) +{ + int i; + uint32_t thread_id = alloc_thread_id(); + uint32_t ip_batch[QSBR_REPORTING_INTERVAL]; + uint32_t next_hop_return = 0; + + RTE_SET_USED(arg); + /* Register this thread to report quiescent state */ + rte_rcu_qsbr_thread_register(rv, thread_id); + rte_rcu_qsbr_thread_online(rv, thread_id); + + do { + for (i = 0; i < QSBR_REPORTING_INTERVAL; i++) + ip_batch[i] = rte_rand(); + + for (i = 0; i < QSBR_REPORTING_INTERVAL; i++) + rte_lpm_lookup(lpm, ip_batch[i], &next_hop_return); + + /* Update quiescent state */ + rte_rcu_qsbr_quiescent(rv, thread_id); + } while (!writer_done); + + rte_rcu_qsbr_thread_offline(rv, thread_id); + rte_rcu_qsbr_thread_unregister(rv, thread_id); + + return 0; +} + +/* + * Writer thread using rte_lpm data structure with RCU. + */ +static int +test_lpm_rcu_qsbr_writer(void *arg) +{ + unsigned int i, j, si, ei; + uint64_t begin, total_cycles; + uint8_t core_id = (uint8_t)((uintptr_t)arg); + uint32_t next_hop_add = 0xAA; + + RTE_SET_USED(arg); + /* 2 writer threads are used */ + if (core_id % 2 == 0) { + si = 0; + ei = NUM_LDEPTH_ROUTE_ENTRIES / 2; + } else { + si = NUM_LDEPTH_ROUTE_ENTRIES / 2; + ei = NUM_LDEPTH_ROUTE_ENTRIES; + } + + /* Measure add/delete. */ + begin = rte_rdtsc_precise(); + for (i = 0; i < RCU_ITERATIONS; i++) { + /* Add all the entries */ + for (j = si; j < ei; j++) { + pthread_mutex_lock(&lpm_mutex); + if (rte_lpm_add(lpm, large_ldepth_route_table[j].ip, + large_ldepth_route_table[j].depth, + next_hop_add) != 0) { + printf("Failed to add iteration %d, route# %d\n", + i, j); + } + pthread_mutex_unlock(&lpm_mutex); + } + + /* Delete all the entries */ + for (j = si; j < ei; j++) { + pthread_mutex_lock(&lpm_mutex); + if (rte_lpm_delete(lpm, large_ldepth_route_table[j].ip, + large_ldepth_route_table[j].depth) != 0) { + printf("Failed to delete iteration %d, route# %d\n", + i, j); + } + pthread_mutex_unlock(&lpm_mutex); + } + } + + total_cycles = rte_rdtsc_precise() - begin; + + __atomic_fetch_add(&gwrite_cycles, total_cycles, __ATOMIC_RELAXED); + __atomic_fetch_add(&gwrites, + 2 * NUM_LDEPTH_ROUTE_ENTRIES * RCU_ITERATIONS, + __ATOMIC_RELAXED); + + return 0; +} + +/* + * Functional test: + * 2 writers, rest are readers + */ +static int +test_lpm_rcu_perf_multi_writer(void) +{ + struct rte_lpm_config config; + size_t sz; + unsigned int i; + uint16_t core_id; + struct rte_lpm_rcu_config rcu_cfg = {0}; + + if (rte_lcore_count() < 3) { + printf("Not enough cores for lpm_rcu_perf_autotest, expecting at least 3\n"); + return TEST_SKIPPED; + } + + num_cores = 0; + RTE_LCORE_FOREACH_SLAVE(core_id) { + enabled_core_ids[num_cores] = core_id; + num_cores++; + } + + printf("\nPerf test: 2 writers, %d readers, RCU integration enabled\n", + num_cores - 2); + + /* Create LPM table */ + config.max_rules = NUM_LDEPTH_ROUTE_ENTRIES; + config.number_tbl8s = NUM_LDEPTH_ROUTE_ENTRIES; + config.flags = 0; + lpm = rte_lpm_create(__func__, SOCKET_ID_ANY, &config); + TEST_LPM_ASSERT(lpm != NULL); + + /* Init RCU variable */ + sz = rte_rcu_qsbr_get_memsize(num_cores); + rv = (struct rte_rcu_qsbr *)rte_zmalloc("rcu0", sz, + RTE_CACHE_LINE_SIZE); + rte_rcu_qsbr_init(rv, num_cores); + + rcu_cfg.v = rv; + /* Assign the RCU variable to LPM */ + if (rte_lpm_rcu_qsbr_add(lpm, &rcu_cfg, NULL) != 0) { + printf("RCU variable assignment failed\n"); + goto error; + } + + writer_done = 0; + __atomic_store_n(&gwrite_cycles, 0, __ATOMIC_RELAXED); + __atomic_store_n(&gwrites, 0, __ATOMIC_RELAXED); + + __atomic_store_n(&thr_id, 0, __ATOMIC_SEQ_CST); + + /* Launch reader threads */ + for (i = 2; i < num_cores; i++) + rte_eal_remote_launch(test_lpm_rcu_qsbr_reader, NULL, + enabled_core_ids[i]); + + /* Launch writer threads */ + for (i = 0; i < 2; i++) + rte_eal_remote_launch(test_lpm_rcu_qsbr_writer, + (void *)(uintptr_t)i, + enabled_core_ids[i]); + + /* Wait for writer threads */ + for (i = 0; i < 2; i++) + if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0) + goto error; + + printf("Total LPM Adds: %d\n", + 2 * ITERATIONS * NUM_LDEPTH_ROUTE_ENTRIES); + printf("Total LPM Deletes: %d\n", + 2 * ITERATIONS * NUM_LDEPTH_ROUTE_ENTRIES); + printf("Average LPM Add/Del: %"PRIu64" cycles\n", + __atomic_load_n(&gwrite_cycles, __ATOMIC_RELAXED) / + __atomic_load_n(&gwrites, __ATOMIC_RELAXED) + ); + + /* Wait and check return value from reader threads */ + writer_done = 1; + for (i = 2; i < num_cores; i++) + if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0) + goto error; + + rte_lpm_free(lpm); + rte_free(rv); + lpm = NULL; + rv = NULL; + + /* Test without RCU integration */ + printf("\nPerf test: 2 writers, %d readers, RCU integration disabled\n", + num_cores - 2); + + /* Create LPM table */ + config.max_rules = NUM_LDEPTH_ROUTE_ENTRIES; + config.number_tbl8s = NUM_LDEPTH_ROUTE_ENTRIES; + config.flags = 0; + lpm = rte_lpm_create(__func__, SOCKET_ID_ANY, &config); + TEST_LPM_ASSERT(lpm != NULL); + + writer_done = 0; + __atomic_store_n(&gwrite_cycles, 0, __ATOMIC_RELAXED); + __atomic_store_n(&gwrites, 0, __ATOMIC_RELAXED); + __atomic_store_n(&thr_id, 0, __ATOMIC_SEQ_CST); + + /* Launch reader threads */ + for (i = 2; i < num_cores; i++) + rte_eal_remote_launch(test_lpm_reader, NULL, + enabled_core_ids[i]); + + /* Launch writer threads */ + for (i = 0; i < 2; i++) + rte_eal_remote_launch(test_lpm_rcu_qsbr_writer, + (void *)(uintptr_t)i, + enabled_core_ids[i]); + + /* Wait for writer threads */ + for (i = 0; i < 2; i++) + if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0) + goto error; + + printf("Total LPM Adds: %d\n", + 2 * ITERATIONS * NUM_LDEPTH_ROUTE_ENTRIES); + printf("Total LPM Deletes: %d\n", + 2 * ITERATIONS * NUM_LDEPTH_ROUTE_ENTRIES); + printf("Average LPM Add/Del: %"PRIu64" cycles\n", + __atomic_load_n(&gwrite_cycles, __ATOMIC_RELAXED) / + __atomic_load_n(&gwrites, __ATOMIC_RELAXED) + ); + + writer_done = 1; + /* Wait and check return value from reader threads */ + for (i = 2; i < num_cores; i++) + if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0) + goto error; + + rte_lpm_free(lpm); + + return 0; + +error: + writer_done = 1; + /* Wait until all readers have exited */ + rte_eal_mp_wait_lcore(); + + rte_lpm_free(lpm); + rte_free(rv); + + return -1; +} + +/* + * Functional test: + * Single writer, rest are readers + */ +static int +test_lpm_rcu_perf(void) +{ + struct rte_lpm_config config; + uint64_t begin, total_cycles; + size_t sz; + unsigned int i, j; + uint16_t core_id; + uint32_t next_hop_add = 0xAA; + struct rte_lpm_rcu_config rcu_cfg = {0}; + + if (rte_lcore_count() < 2) { + printf("Not enough cores for lpm_rcu_perf_autotest, expecting at least 2\n"); + return TEST_SKIPPED; + } + + num_cores = 0; + RTE_LCORE_FOREACH_SLAVE(core_id) { + enabled_core_ids[num_cores] = core_id; + num_cores++; + } + + printf("\nPerf test: 1 writer, %d readers, RCU integration enabled\n", + num_cores); + + /* Create LPM table */ + config.max_rules = NUM_LDEPTH_ROUTE_ENTRIES; + config.number_tbl8s = NUM_LDEPTH_ROUTE_ENTRIES; + config.flags = 0; + lpm = rte_lpm_create(__func__, SOCKET_ID_ANY, &config); + TEST_LPM_ASSERT(lpm != NULL); + + /* Init RCU variable */ + sz = rte_rcu_qsbr_get_memsize(num_cores); + rv = (struct rte_rcu_qsbr *)rte_zmalloc("rcu0", sz, + RTE_CACHE_LINE_SIZE); + rte_rcu_qsbr_init(rv, num_cores); + + rcu_cfg.v = rv; + /* Assign the RCU variable to LPM */ + if (rte_lpm_rcu_qsbr_add(lpm, &rcu_cfg, NULL) != 0) { + printf("RCU variable assignment failed\n"); + goto error; + } + + writer_done = 0; + __atomic_store_n(&thr_id, 0, __ATOMIC_SEQ_CST); + + /* Launch reader threads */ + for (i = 0; i < num_cores; i++) + rte_eal_remote_launch(test_lpm_rcu_qsbr_reader, NULL, + enabled_core_ids[i]); + + /* Measure add/delete. */ + begin = rte_rdtsc_precise(); + for (i = 0; i < RCU_ITERATIONS; i++) { + /* Add all the entries */ + for (j = 0; j < NUM_LDEPTH_ROUTE_ENTRIES; j++) + if (rte_lpm_add(lpm, large_ldepth_route_table[j].ip, + large_ldepth_route_table[j].depth, + next_hop_add) != 0) { + printf("Failed to add iteration %d, route# %d\n", + i, j); + goto error; + } + + /* Delete all the entries */ + for (j = 0; j < NUM_LDEPTH_ROUTE_ENTRIES; j++) + if (rte_lpm_delete(lpm, large_ldepth_route_table[j].ip, + large_ldepth_route_table[j].depth) != 0) { + printf("Failed to delete iteration %d, route# %d\n", + i, j); + goto error; + } + } + total_cycles = rte_rdtsc_precise() - begin; + + printf("Total LPM Adds: %d\n", ITERATIONS * NUM_LDEPTH_ROUTE_ENTRIES); + printf("Total LPM Deletes: %d\n", + ITERATIONS * NUM_LDEPTH_ROUTE_ENTRIES); + printf("Average LPM Add/Del: %g cycles\n", + (double)total_cycles / (NUM_LDEPTH_ROUTE_ENTRIES * ITERATIONS)); + + writer_done = 1; + /* Wait and check return value from reader threads */ + for (i = 0; i < num_cores; i++) + if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0) + goto error; + + rte_lpm_free(lpm); + rte_free(rv); + lpm = NULL; + rv = NULL; + + /* Test without RCU integration */ + printf("\nPerf test: 1 writer, %d readers, RCU integration disabled\n", + num_cores); + + /* Create LPM table */ + config.max_rules = NUM_LDEPTH_ROUTE_ENTRIES; + config.number_tbl8s = NUM_LDEPTH_ROUTE_ENTRIES; + config.flags = 0; + lpm = rte_lpm_create(__func__, SOCKET_ID_ANY, &config); + TEST_LPM_ASSERT(lpm != NULL); + + writer_done = 0; + __atomic_store_n(&thr_id, 0, __ATOMIC_SEQ_CST); + + /* Launch reader threads */ + for (i = 0; i < num_cores; i++) + rte_eal_remote_launch(test_lpm_reader, NULL, + enabled_core_ids[i]); + + /* Measure add/delete. */ + begin = rte_rdtsc_precise(); + for (i = 0; i < RCU_ITERATIONS; i++) { + /* Add all the entries */ + for (j = 0; j < NUM_LDEPTH_ROUTE_ENTRIES; j++) + if (rte_lpm_add(lpm, large_ldepth_route_table[j].ip, + large_ldepth_route_table[j].depth, + next_hop_add) != 0) { + printf("Failed to add iteration %d, route# %d\n", + i, j); + goto error; + } + + /* Delete all the entries */ + for (j = 0; j < NUM_LDEPTH_ROUTE_ENTRIES; j++) + if (rte_lpm_delete(lpm, large_ldepth_route_table[j].ip, + large_ldepth_route_table[j].depth) != 0) { + printf("Failed to delete iteration %d, route# %d\n", + i, j); + goto error; + } + } + total_cycles = rte_rdtsc_precise() - begin; + + printf("Total LPM Adds: %d\n", ITERATIONS * NUM_LDEPTH_ROUTE_ENTRIES); + printf("Total LPM Deletes: %d\n", + ITERATIONS * NUM_LDEPTH_ROUTE_ENTRIES); + printf("Average LPM Add/Del: %g cycles\n", + (double)total_cycles / (NUM_LDEPTH_ROUTE_ENTRIES * ITERATIONS)); + + writer_done = 1; + /* Wait and check return value from reader threads */ + for (i = 0; i < num_cores; i++) + if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0) + printf("Warning: lcore %u not finished.\n", + enabled_core_ids[i]); + + rte_lpm_free(lpm); + + return 0; + +error: + writer_done = 1; + /* Wait until all readers have exited */ + rte_eal_mp_wait_lcore(); + + rte_lpm_free(lpm); + rte_free(rv); + + return -1; +} + static int test_lpm_perf(void) { - struct rte_lpm *lpm = NULL; struct rte_lpm_config config; config.max_rules = 2000000; @@ -343,7 +825,7 @@ test_lpm_perf(void) lpm = rte_lpm_create(__func__, SOCKET_ID_ANY, &config); TEST_LPM_ASSERT(lpm != NULL); - /* Measue add. */ + /* Measure add. */ begin = rte_rdtsc(); for (i = 0; i < NUM_ROUTE_ENTRIES; i++) { @@ -478,6 +960,10 @@ test_lpm_perf(void) rte_lpm_delete_all(lpm); rte_lpm_free(lpm); + test_lpm_rcu_perf(); + + test_lpm_rcu_perf_multi_writer(); + return 0; }