From patchwork Tue Jul 6 13:32:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suanming Mou X-Patchwork-Id: 95400 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7855BA0C47; Tue, 6 Jul 2021 15:35:11 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 26E1541424; Tue, 6 Jul 2021 15:33:46 +0200 (CEST) Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam07on2040.outbound.protection.outlook.com [40.107.95.40]) by mails.dpdk.org (Postfix) with ESMTP id C1F5C413C2 for ; Tue, 6 Jul 2021 15:33:43 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ert8gHKt/fCJXliI8yoM5olOFa1SHtb+LtQnP8DLSlOlSc/xeNsVG93b9HwqjgAEiJv8jWcGL34QRAojphFGvdfHqcqlwhf5WYK0PQcKwdfzbqc7GzWVSy7tkVf2pGhK3Hbqx2DyfEECCRayR6j1a2AppTJgh3S+pTr0LvaaRW5mK9HBKDmSpZpUTfLXktsEL2rRQsDONU6yHUP1pjPDziPqSHq6hVPjYYQR5q4xnw3G5FQHWqjv19pu6x68sABdsG+cy3+11Qycnk6bhyFR2YDqiM4UPOLMsVqVjCrPLmAHiCDqv7qXeEhcwF1jsjKPId9GAkjxQ5EZwMKRDldwBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W8FCoQHhTuwZnTOQA6gXUjvHy58uwpA8JY7dUz3OC58=; b=d7QBKtf4N2q9wRVUxz/TLR5urAseo/4uUgaCdtWf/RrEGvgrcsNOcGyfaazM2VCSWRbVk/Yw5o/2endLdvlOlQw1R9CzKmA4SA5pmjdL7WikL+b/MDkVENGZgSYXARmCq3gs3lmAHMJ6Eq7tiYPTtcMNUABkHhWgK5neEe9SZq0O6ZPei0Hd6n0XbU/R/4rC76gYmFmot0t73B/gpLYNhysW0ldbWvBVfSnJZhDyLUu80QtpcrozRVxK/EjZ9A19jmz0tWRJGJDHfe1r8CfqbkN8JiCZxAtjQLsVfPLnCfaU67d+ocoZpvUXAmwZOQ53kBkveImpMxU5xSvh11n9Sg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W8FCoQHhTuwZnTOQA6gXUjvHy58uwpA8JY7dUz3OC58=; b=swVJb/KJjMtPNJbeJQ5n9vlHJmvitDUPQWoI3m1gTlkzXFD3NZOVab42Ln3SHp/Ks/jyXQpt3IE56SQc5BrWP4K+K08mWYibJsYvJutP2GxNGtyTh20D0X2ZZrKhwZp3DCOyjm/xM+XnCcWLRnnzWxC+r+HLnU04DoIgbrRNVvlTHo1TyhFjlNCNypAQd2R7wgTQgU5iD38lRyIEGMwXWwNfUnbFSSLLNM5Eg7RddX4fEBrcenqQguzH/1MChfjJGY1zfjs6lfU1jFs24by+UzmtaKjlhgLRErZXM3zwrXOcHqtQAT+vj40BeLcQzq7Zx4V6udX4kLKw0jwsHA9xVw== Received: from BN9PR03CA0569.namprd03.prod.outlook.com (2603:10b6:408:138::34) by MN2PR12MB4389.namprd12.prod.outlook.com (2603:10b6:208:262::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4287.33; Tue, 6 Jul 2021 13:33:42 +0000 Received: from BN8NAM11FT062.eop-nam11.prod.protection.outlook.com (2603:10b6:408:138:cafe::ea) by BN9PR03CA0569.outlook.office365.com (2603:10b6:408:138::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4308.19 via Frontend Transport; Tue, 6 Jul 2021 13:33:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT062.mail.protection.outlook.com (10.13.177.34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4287.22 via Frontend Transport; Tue, 6 Jul 2021 13:33:41 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 6 Jul 2021 13:33:39 +0000 From: Suanming Mou To: , CC: , , Date: Tue, 6 Jul 2021 16:32:44 +0300 Message-ID: <20210706133257.3353-14-suanmingm@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20210706133257.3353-1-suanmingm@nvidia.com> References: <20210527093403.1153127-1-suanmingm@nvidia.com> <20210706133257.3353-1-suanmingm@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL101.nvidia.com (172.20.187.10) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b7ca0c2e-81ae-465d-2404-08d94082abb1 X-MS-TrafficTypeDiagnostic: MN2PR12MB4389: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:33; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: krEUKk44r9J46B7pkLVvEom+R9kdeSu0z41COvBT4kOL2EQWLlrcunWcdwmCjat+1ijyZAoD5D/BZAE1qZbL0bfaicox/5JtrzMga1GtKc10HsIptvGIxbTY6oGcA7F2XIinWwAKfJ2YO0v8PRzlHTwUq/AIm1mYf03YwQ7F8TEIEooYWM20YmMPnrIgee6elcLlrvG7daLRWeWkO4/fql4RAOOB4ToOqG9pumcFxr9qpk7p5fu+wZSbFVdqtuoei/AxGB3iKkffN4aGEAep+NXMUBEIV2XBuVXZGRAganEHhDiUXIaJNsB5C41DSBh4p3sLU+uy6v5rg5zPeY9De6X78Dm2AFBhETaJ6PmciXExjSmQ3qzzbA4g01/jdoUxWtTQydZLxqWOTAO5nskwytzMOd2BEPP3IpRlD2Ikslpc6vcRdXIHIp5KkcaTdeId+gCNY1UjPls3eN2xrUWHIc0JpqQo3F+WGENBYJ8oQclyFkyUfpJaXcI4lMJWM2G6qxJf26GmVlHeh45iKqXEdDROQU7wnKzeiAc6/nytedVG0Nm2dIbyc1ugX17mqFGUyi2EKw2sqFBXZH03copavRhkfRwe9agS10ec8/Qsv1OTf0L71Kmil/p0nh3lpYfS9S2Bjwcadp+UlwReUKjNtDllZYmyIS6XHQT3H1bCQXafxmmYbrZCxy/ojgaj4gxmX8t+niBzJtRuTrlBxCKUCg== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(39860400002)(136003)(346002)(376002)(36840700001)(46966006)(7696005)(86362001)(426003)(8936002)(6286002)(83380400001)(6666004)(55016002)(478600001)(336012)(82310400003)(30864003)(82740400003)(1076003)(6636002)(70586007)(47076005)(4326008)(54906003)(2906002)(356005)(110136005)(316002)(26005)(7636003)(36756003)(36860700001)(36906005)(5660300002)(2616005)(186003)(70206006)(16526019)(8676002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jul 2021 13:33:41.8316 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b7ca0c2e-81ae-465d-2404-08d94082abb1 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT062.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4389 Subject: [dpdk-dev] [PATCH v4 13/26] common/mlx5: move list utility to common X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hash list is planned to be implemented with the cache list code. This commit moves the list utility to common directory. Signed-off-by: Suanming Mou Acked-by: Matan Azrad --- drivers/common/mlx5/mlx5_common.h | 2 + drivers/common/mlx5/mlx5_common_utils.c | 250 +++++++++++++++++++++++ drivers/common/mlx5/mlx5_common_utils.h | 205 +++++++++++++++++++ drivers/common/mlx5/version.map | 7 + drivers/net/mlx5/mlx5_utils.c | 251 ------------------------ drivers/net/mlx5/mlx5_utils.h | 197 ------------------- 6 files changed, 464 insertions(+), 448 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 306f2f1ab7..7fb7d40b38 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -14,6 +14,8 @@ #include #include #include +#include +#include #include #include "mlx5_prm.h" diff --git a/drivers/common/mlx5/mlx5_common_utils.c b/drivers/common/mlx5/mlx5_common_utils.c index ad2011e858..8bb8a6016d 100644 --- a/drivers/common/mlx5/mlx5_common_utils.c +++ b/drivers/common/mlx5/mlx5_common_utils.c @@ -11,6 +11,256 @@ #include "mlx5_common_utils.h" #include "mlx5_common_log.h" +/********************* mlx5 list ************************/ + +struct mlx5_list * +mlx5_list_create(const char *name, void *ctx, + mlx5_list_create_cb cb_create, + mlx5_list_match_cb cb_match, + mlx5_list_remove_cb cb_remove, + mlx5_list_clone_cb cb_clone, + mlx5_list_clone_free_cb cb_clone_free) +{ + struct mlx5_list *list; + int i; + + if (!cb_match || !cb_create || !cb_remove || !cb_clone || + !cb_clone_free) { + rte_errno = EINVAL; + return NULL; + } + list = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*list), 0, SOCKET_ID_ANY); + if (!list) + return NULL; + if (name) + snprintf(list->name, sizeof(list->name), "%s", name); + list->ctx = ctx; + list->cb_create = cb_create; + list->cb_match = cb_match; + list->cb_remove = cb_remove; + list->cb_clone = cb_clone; + list->cb_clone_free = cb_clone_free; + rte_rwlock_init(&list->lock); + DRV_LOG(DEBUG, "mlx5 list %s initialized.", list->name); + for (i = 0; i <= RTE_MAX_LCORE; i++) + LIST_INIT(&list->cache[i].h); + return list; +} + +static struct mlx5_list_entry * +__list_lookup(struct mlx5_list *list, int lcore_index, void *ctx, bool reuse) +{ + struct mlx5_list_entry *entry = LIST_FIRST(&list->cache[lcore_index].h); + uint32_t ret; + + while (entry != NULL) { + if (list->cb_match(list, entry, ctx) == 0) { + if (reuse) { + ret = __atomic_add_fetch(&entry->ref_cnt, 1, + __ATOMIC_RELAXED) - 1; + DRV_LOG(DEBUG, "mlx5 list %s entry %p ref: %u.", + list->name, (void *)entry, + entry->ref_cnt); + } else if (lcore_index < RTE_MAX_LCORE) { + ret = __atomic_load_n(&entry->ref_cnt, + __ATOMIC_RELAXED); + } + if (likely(ret != 0 || lcore_index == RTE_MAX_LCORE)) + return entry; + if (reuse && ret == 0) + entry->ref_cnt--; /* Invalid entry. */ + } + entry = LIST_NEXT(entry, next); + } + return NULL; +} + +struct mlx5_list_entry * +mlx5_list_lookup(struct mlx5_list *list, void *ctx) +{ + struct mlx5_list_entry *entry = NULL; + int i; + + rte_rwlock_read_lock(&list->lock); + for (i = 0; i < RTE_MAX_LCORE; i++) { + entry = __list_lookup(list, i, ctx, false); + if (entry) + break; + } + rte_rwlock_read_unlock(&list->lock); + return entry; +} + +static struct mlx5_list_entry * +mlx5_list_cache_insert(struct mlx5_list *list, int lcore_index, + struct mlx5_list_entry *gentry, void *ctx) +{ + struct mlx5_list_entry *lentry = list->cb_clone(list, gentry, ctx); + + if (unlikely(!lentry)) + return NULL; + lentry->ref_cnt = 1u; + lentry->gentry = gentry; + lentry->lcore_idx = (uint32_t)lcore_index; + LIST_INSERT_HEAD(&list->cache[lcore_index].h, lentry, next); + return lentry; +} + +static void +__list_cache_clean(struct mlx5_list *list, int lcore_index) +{ + struct mlx5_list_cache *c = &list->cache[lcore_index]; + struct mlx5_list_entry *entry = LIST_FIRST(&c->h); + uint32_t inv_cnt = __atomic_exchange_n(&c->inv_cnt, 0, + __ATOMIC_RELAXED); + + while (inv_cnt != 0 && entry != NULL) { + struct mlx5_list_entry *nentry = LIST_NEXT(entry, next); + + if (__atomic_load_n(&entry->ref_cnt, __ATOMIC_RELAXED) == 0) { + LIST_REMOVE(entry, next); + list->cb_clone_free(list, entry); + inv_cnt--; + } + entry = nentry; + } +} + +struct mlx5_list_entry * +mlx5_list_register(struct mlx5_list *list, void *ctx) +{ + struct mlx5_list_entry *entry, *local_entry; + volatile uint32_t prev_gen_cnt = 0; + int lcore_index = rte_lcore_index(rte_lcore_id()); + + MLX5_ASSERT(list); + MLX5_ASSERT(lcore_index < RTE_MAX_LCORE); + if (unlikely(lcore_index == -1)) { + rte_errno = ENOTSUP; + return NULL; + } + /* 0. Free entries that was invalidated by other lcores. */ + __list_cache_clean(list, lcore_index); + /* 1. Lookup in local cache. */ + local_entry = __list_lookup(list, lcore_index, ctx, true); + if (local_entry) + return local_entry; + /* 2. Lookup with read lock on global list, reuse if found. */ + rte_rwlock_read_lock(&list->lock); + entry = __list_lookup(list, RTE_MAX_LCORE, ctx, true); + if (likely(entry)) { + rte_rwlock_read_unlock(&list->lock); + return mlx5_list_cache_insert(list, lcore_index, entry, ctx); + } + prev_gen_cnt = list->gen_cnt; + rte_rwlock_read_unlock(&list->lock); + /* 3. Prepare new entry for global list and for cache. */ + entry = list->cb_create(list, entry, ctx); + if (unlikely(!entry)) + return NULL; + local_entry = list->cb_clone(list, entry, ctx); + if (unlikely(!local_entry)) { + list->cb_remove(list, entry); + return NULL; + } + entry->ref_cnt = 1u; + local_entry->ref_cnt = 1u; + local_entry->gentry = entry; + local_entry->lcore_idx = (uint32_t)lcore_index; + rte_rwlock_write_lock(&list->lock); + /* 4. Make sure the same entry was not created before the write lock. */ + if (unlikely(prev_gen_cnt != list->gen_cnt)) { + struct mlx5_list_entry *oentry = __list_lookup(list, + RTE_MAX_LCORE, + ctx, true); + + if (unlikely(oentry)) { + /* 4.5. Found real race!!, reuse the old entry. */ + rte_rwlock_write_unlock(&list->lock); + list->cb_remove(list, entry); + list->cb_clone_free(list, local_entry); + return mlx5_list_cache_insert(list, lcore_index, oentry, + ctx); + } + } + /* 5. Update lists. */ + LIST_INSERT_HEAD(&list->cache[RTE_MAX_LCORE].h, entry, next); + list->gen_cnt++; + rte_rwlock_write_unlock(&list->lock); + LIST_INSERT_HEAD(&list->cache[lcore_index].h, local_entry, next); + __atomic_add_fetch(&list->count, 1, __ATOMIC_RELAXED); + DRV_LOG(DEBUG, "mlx5 list %s entry %p new: %u.", list->name, + (void *)entry, entry->ref_cnt); + return local_entry; +} + +int +mlx5_list_unregister(struct mlx5_list *list, + struct mlx5_list_entry *entry) +{ + struct mlx5_list_entry *gentry = entry->gentry; + int lcore_idx; + + if (__atomic_sub_fetch(&entry->ref_cnt, 1, __ATOMIC_RELAXED) != 0) + return 1; + lcore_idx = rte_lcore_index(rte_lcore_id()); + MLX5_ASSERT(lcore_idx < RTE_MAX_LCORE); + if (entry->lcore_idx == (uint32_t)lcore_idx) { + LIST_REMOVE(entry, next); + list->cb_clone_free(list, entry); + } else if (likely(lcore_idx != -1)) { + __atomic_add_fetch(&list->cache[entry->lcore_idx].inv_cnt, 1, + __ATOMIC_RELAXED); + } else { + return 0; + } + if (__atomic_sub_fetch(&gentry->ref_cnt, 1, __ATOMIC_RELAXED) != 0) + return 1; + rte_rwlock_write_lock(&list->lock); + if (likely(gentry->ref_cnt == 0)) { + LIST_REMOVE(gentry, next); + rte_rwlock_write_unlock(&list->lock); + list->cb_remove(list, gentry); + __atomic_sub_fetch(&list->count, 1, __ATOMIC_RELAXED); + DRV_LOG(DEBUG, "mlx5 list %s entry %p removed.", + list->name, (void *)gentry); + return 0; + } + rte_rwlock_write_unlock(&list->lock); + return 1; +} + +void +mlx5_list_destroy(struct mlx5_list *list) +{ + struct mlx5_list_entry *entry; + int i; + + MLX5_ASSERT(list); + for (i = 0; i <= RTE_MAX_LCORE; i++) { + while (!LIST_EMPTY(&list->cache[i].h)) { + entry = LIST_FIRST(&list->cache[i].h); + LIST_REMOVE(entry, next); + if (i == RTE_MAX_LCORE) { + list->cb_remove(list, entry); + DRV_LOG(DEBUG, "mlx5 list %s entry %p " + "destroyed.", list->name, + (void *)entry); + } else { + list->cb_clone_free(list, entry); + } + } + } + mlx5_free(list); +} + +uint32_t +mlx5_list_get_entry_num(struct mlx5_list *list) +{ + MLX5_ASSERT(list); + return __atomic_load_n(&list->count, __ATOMIC_RELAXED); +} + /********************* Hash List **********************/ static struct mlx5_hlist_entry * diff --git a/drivers/common/mlx5/mlx5_common_utils.h b/drivers/common/mlx5/mlx5_common_utils.h index ed378ce9bd..96add6d003 100644 --- a/drivers/common/mlx5/mlx5_common_utils.h +++ b/drivers/common/mlx5/mlx5_common_utils.h @@ -7,6 +7,211 @@ #include "mlx5_common.h" +/************************ mlx5 list *****************************/ + +/** Maximum size of string for naming. */ +#define MLX5_NAME_SIZE 32 + +struct mlx5_list; + +/** + * Structure of the entry in the mlx5 list, user should define its own struct + * that contains this in order to store the data. + */ +struct mlx5_list_entry { + LIST_ENTRY(mlx5_list_entry) next; /* Entry pointers in the list. */ + uint32_t ref_cnt; /* 0 means, entry is invalid. */ + uint32_t lcore_idx; + struct mlx5_list_entry *gentry; +}; + +struct mlx5_list_cache { + LIST_HEAD(mlx5_list_head, mlx5_list_entry) h; + uint32_t inv_cnt; /* Invalid entries counter. */ +} __rte_cache_aligned; + +/** + * Type of callback function for entry removal. + * + * @param list + * The mlx5 list. + * @param entry + * The entry in the list. + */ +typedef void (*mlx5_list_remove_cb)(struct mlx5_list *list, + struct mlx5_list_entry *entry); + +/** + * Type of function for user defined matching. + * + * @param list + * The mlx5 list. + * @param entry + * The entry in the list. + * @param ctx + * The pointer to new entry context. + * + * @return + * 0 if matching, non-zero number otherwise. + */ +typedef int (*mlx5_list_match_cb)(struct mlx5_list *list, + struct mlx5_list_entry *entry, void *ctx); + +typedef struct mlx5_list_entry *(*mlx5_list_clone_cb) + (struct mlx5_list *list, + struct mlx5_list_entry *entry, void *ctx); + +typedef void (*mlx5_list_clone_free_cb)(struct mlx5_list *list, + struct mlx5_list_entry *entry); + +/** + * Type of function for user defined mlx5 list entry creation. + * + * @param list + * The mlx5 list. + * @param entry + * The new allocated entry, NULL if list entry size unspecified, + * New entry has to be allocated in callback and return. + * @param ctx + * The pointer to new entry context. + * + * @return + * Pointer of entry on success, NULL otherwise. + */ +typedef struct mlx5_list_entry *(*mlx5_list_create_cb) + (struct mlx5_list *list, + struct mlx5_list_entry *entry, + void *ctx); + +/** + * Linked mlx5 list structure. + * + * Entry in mlx5 list could be reused if entry already exists, + * reference count will increase and the existing entry returns. + * + * When destroy an entry from list, decrease reference count and only + * destroy when no further reference. + * + * Linked list is designed for limited number of entries, + * read mostly, less modification. + * + * For huge amount of entries, please consider hash list. + * + */ +struct mlx5_list { + char name[MLX5_NAME_SIZE]; /**< Name of the mlx5 list. */ + volatile uint32_t gen_cnt; + /* List modification will update generation count. */ + volatile uint32_t count; /* number of entries in list. */ + void *ctx; /* user objects target to callback. */ + rte_rwlock_t lock; /* read/write lock. */ + mlx5_list_create_cb cb_create; /**< entry create callback. */ + mlx5_list_match_cb cb_match; /**< entry match callback. */ + mlx5_list_remove_cb cb_remove; /**< entry remove callback. */ + mlx5_list_clone_cb cb_clone; /**< entry clone callback. */ + mlx5_list_clone_free_cb cb_clone_free; + struct mlx5_list_cache cache[RTE_MAX_LCORE + 1]; + /* Lcore cache, last index is the global cache. */ +}; + +/** + * Create a mlx5 list. + * + * @param list + * Pointer to the hast list table. + * @param name + * Name of the mlx5 list. + * @param ctx + * Pointer to the list context data. + * @param cb_create + * Callback function for entry create. + * @param cb_match + * Callback function for entry match. + * @param cb_remove + * Callback function for entry remove. + * @return + * List pointer on success, otherwise NULL. + */ +__rte_internal +struct mlx5_list *mlx5_list_create(const char *name, void *ctx, + mlx5_list_create_cb cb_create, + mlx5_list_match_cb cb_match, + mlx5_list_remove_cb cb_remove, + mlx5_list_clone_cb cb_clone, + mlx5_list_clone_free_cb cb_clone_free); + +/** + * Search an entry matching the key. + * + * Result returned might be destroyed by other thread, must use + * this function only in main thread. + * + * @param list + * Pointer to the mlx5 list. + * @param ctx + * Common context parameter used by entry callback function. + * + * @return + * Pointer of the list entry if found, NULL otherwise. + */ +__rte_internal +struct mlx5_list_entry *mlx5_list_lookup(struct mlx5_list *list, + void *ctx); + +/** + * Reuse or create an entry to the mlx5 list. + * + * @param list + * Pointer to the hast list table. + * @param ctx + * Common context parameter used by callback function. + * + * @return + * registered entry on success, NULL otherwise + */ +__rte_internal +struct mlx5_list_entry *mlx5_list_register(struct mlx5_list *list, + void *ctx); + +/** + * Remove an entry from the mlx5 list. + * + * User should guarantee the validity of the entry. + * + * @param list + * Pointer to the hast list. + * @param entry + * Entry to be removed from the mlx5 list table. + * @return + * 0 on entry removed, 1 on entry still referenced. + */ +__rte_internal +int mlx5_list_unregister(struct mlx5_list *list, + struct mlx5_list_entry *entry); + +/** + * Destroy the mlx5 list. + * + * @param list + * Pointer to the mlx5 list. + */ +__rte_internal +void mlx5_list_destroy(struct mlx5_list *list); + +/** + * Get entry number from the mlx5 list. + * + * @param list + * Pointer to the hast list. + * @return + * mlx5 list entry number. + */ +__rte_internal +uint32_t +mlx5_list_get_entry_num(struct mlx5_list *list); + +/************************ Hash list *****************************/ + #define MLX5_HLIST_DIRECT_KEY 0x0001 /* Use the key directly as hash index. */ #define MLX5_HLIST_WRITE_MOST 0x0002 /* List mostly used for append new. */ diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index b8be73a77b..e6586d6f6f 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -73,6 +73,13 @@ INTERNAL { mlx5_glue; + mlx5_list_create; + mlx5_list_register; + mlx5_list_unregister; + mlx5_list_lookup; + mlx5_list_get_entry_num; + mlx5_list_destroy; + mlx5_hlist_create; mlx5_hlist_lookup; mlx5_hlist_register; diff --git a/drivers/net/mlx5/mlx5_utils.c b/drivers/net/mlx5/mlx5_utils.c index 0be778935f..e4e66ae4c5 100644 --- a/drivers/net/mlx5/mlx5_utils.c +++ b/drivers/net/mlx5/mlx5_utils.c @@ -8,257 +8,6 @@ #include "mlx5_utils.h" - -/********************* mlx5 list ************************/ - -struct mlx5_list * -mlx5_list_create(const char *name, void *ctx, - mlx5_list_create_cb cb_create, - mlx5_list_match_cb cb_match, - mlx5_list_remove_cb cb_remove, - mlx5_list_clone_cb cb_clone, - mlx5_list_clone_free_cb cb_clone_free) -{ - struct mlx5_list *list; - int i; - - if (!cb_match || !cb_create || !cb_remove || !cb_clone || - !cb_clone_free) { - rte_errno = EINVAL; - return NULL; - } - list = mlx5_malloc(MLX5_MEM_ZERO, sizeof(*list), 0, SOCKET_ID_ANY); - if (!list) - return NULL; - if (name) - snprintf(list->name, sizeof(list->name), "%s", name); - list->ctx = ctx; - list->cb_create = cb_create; - list->cb_match = cb_match; - list->cb_remove = cb_remove; - list->cb_clone = cb_clone; - list->cb_clone_free = cb_clone_free; - rte_rwlock_init(&list->lock); - DRV_LOG(DEBUG, "mlx5 list %s initialized.", list->name); - for (i = 0; i <= RTE_MAX_LCORE; i++) - LIST_INIT(&list->cache[i].h); - return list; -} - -static struct mlx5_list_entry * -__list_lookup(struct mlx5_list *list, int lcore_index, void *ctx, bool reuse) -{ - struct mlx5_list_entry *entry = LIST_FIRST(&list->cache[lcore_index].h); - uint32_t ret; - - while (entry != NULL) { - if (list->cb_match(list, entry, ctx) == 0) { - if (reuse) { - ret = __atomic_add_fetch(&entry->ref_cnt, 1, - __ATOMIC_RELAXED) - 1; - DRV_LOG(DEBUG, "mlx5 list %s entry %p ref: %u.", - list->name, (void *)entry, - entry->ref_cnt); - } else if (lcore_index < RTE_MAX_LCORE) { - ret = __atomic_load_n(&entry->ref_cnt, - __ATOMIC_RELAXED); - } - if (likely(ret != 0 || lcore_index == RTE_MAX_LCORE)) - return entry; - if (reuse && ret == 0) - entry->ref_cnt--; /* Invalid entry. */ - } - entry = LIST_NEXT(entry, next); - } - return NULL; -} - -struct mlx5_list_entry * -mlx5_list_lookup(struct mlx5_list *list, void *ctx) -{ - struct mlx5_list_entry *entry = NULL; - int i; - - rte_rwlock_read_lock(&list->lock); - for (i = 0; i < RTE_MAX_LCORE; i++) { - entry = __list_lookup(list, i, ctx, false); - if (entry) - break; - } - rte_rwlock_read_unlock(&list->lock); - return entry; -} - -static struct mlx5_list_entry * -mlx5_list_cache_insert(struct mlx5_list *list, int lcore_index, - struct mlx5_list_entry *gentry, void *ctx) -{ - struct mlx5_list_entry *lentry = list->cb_clone(list, gentry, ctx); - - if (unlikely(!lentry)) - return NULL; - lentry->ref_cnt = 1u; - lentry->gentry = gentry; - lentry->lcore_idx = (uint32_t)lcore_index; - LIST_INSERT_HEAD(&list->cache[lcore_index].h, lentry, next); - return lentry; -} - -static void -__list_cache_clean(struct mlx5_list *list, int lcore_index) -{ - struct mlx5_list_cache *c = &list->cache[lcore_index]; - struct mlx5_list_entry *entry = LIST_FIRST(&c->h); - uint32_t inv_cnt = __atomic_exchange_n(&c->inv_cnt, 0, - __ATOMIC_RELAXED); - - while (inv_cnt != 0 && entry != NULL) { - struct mlx5_list_entry *nentry = LIST_NEXT(entry, next); - - if (__atomic_load_n(&entry->ref_cnt, __ATOMIC_RELAXED) == 0) { - LIST_REMOVE(entry, next); - list->cb_clone_free(list, entry); - inv_cnt--; - } - entry = nentry; - } -} - -struct mlx5_list_entry * -mlx5_list_register(struct mlx5_list *list, void *ctx) -{ - struct mlx5_list_entry *entry, *local_entry; - volatile uint32_t prev_gen_cnt = 0; - int lcore_index = rte_lcore_index(rte_lcore_id()); - - MLX5_ASSERT(list); - MLX5_ASSERT(lcore_index < RTE_MAX_LCORE); - if (unlikely(lcore_index == -1)) { - rte_errno = ENOTSUP; - return NULL; - } - /* 0. Free entries that was invalidated by other lcores. */ - __list_cache_clean(list, lcore_index); - /* 1. Lookup in local cache. */ - local_entry = __list_lookup(list, lcore_index, ctx, true); - if (local_entry) - return local_entry; - /* 2. Lookup with read lock on global list, reuse if found. */ - rte_rwlock_read_lock(&list->lock); - entry = __list_lookup(list, RTE_MAX_LCORE, ctx, true); - if (likely(entry)) { - rte_rwlock_read_unlock(&list->lock); - return mlx5_list_cache_insert(list, lcore_index, entry, ctx); - } - prev_gen_cnt = list->gen_cnt; - rte_rwlock_read_unlock(&list->lock); - /* 3. Prepare new entry for global list and for cache. */ - entry = list->cb_create(list, entry, ctx); - if (unlikely(!entry)) - return NULL; - local_entry = list->cb_clone(list, entry, ctx); - if (unlikely(!local_entry)) { - list->cb_remove(list, entry); - return NULL; - } - entry->ref_cnt = 1u; - local_entry->ref_cnt = 1u; - local_entry->gentry = entry; - local_entry->lcore_idx = (uint32_t)lcore_index; - rte_rwlock_write_lock(&list->lock); - /* 4. Make sure the same entry was not created before the write lock. */ - if (unlikely(prev_gen_cnt != list->gen_cnt)) { - struct mlx5_list_entry *oentry = __list_lookup(list, - RTE_MAX_LCORE, - ctx, true); - - if (unlikely(oentry)) { - /* 4.5. Found real race!!, reuse the old entry. */ - rte_rwlock_write_unlock(&list->lock); - list->cb_remove(list, entry); - list->cb_clone_free(list, local_entry); - return mlx5_list_cache_insert(list, lcore_index, oentry, - ctx); - } - } - /* 5. Update lists. */ - LIST_INSERT_HEAD(&list->cache[RTE_MAX_LCORE].h, entry, next); - list->gen_cnt++; - rte_rwlock_write_unlock(&list->lock); - LIST_INSERT_HEAD(&list->cache[lcore_index].h, local_entry, next); - __atomic_add_fetch(&list->count, 1, __ATOMIC_RELAXED); - DRV_LOG(DEBUG, "mlx5 list %s entry %p new: %u.", list->name, - (void *)entry, entry->ref_cnt); - return local_entry; -} - -int -mlx5_list_unregister(struct mlx5_list *list, - struct mlx5_list_entry *entry) -{ - struct mlx5_list_entry *gentry = entry->gentry; - int lcore_idx; - - if (__atomic_sub_fetch(&entry->ref_cnt, 1, __ATOMIC_RELAXED) != 0) - return 1; - lcore_idx = rte_lcore_index(rte_lcore_id()); - MLX5_ASSERT(lcore_idx < RTE_MAX_LCORE); - if (entry->lcore_idx == (uint32_t)lcore_idx) { - LIST_REMOVE(entry, next); - list->cb_clone_free(list, entry); - } else if (likely(lcore_idx != -1)) { - __atomic_add_fetch(&list->cache[entry->lcore_idx].inv_cnt, 1, - __ATOMIC_RELAXED); - } else { - return 0; - } - if (__atomic_sub_fetch(&gentry->ref_cnt, 1, __ATOMIC_RELAXED) != 0) - return 1; - rte_rwlock_write_lock(&list->lock); - if (likely(gentry->ref_cnt == 0)) { - LIST_REMOVE(gentry, next); - rte_rwlock_write_unlock(&list->lock); - list->cb_remove(list, gentry); - __atomic_sub_fetch(&list->count, 1, __ATOMIC_RELAXED); - DRV_LOG(DEBUG, "mlx5 list %s entry %p removed.", - list->name, (void *)gentry); - return 0; - } - rte_rwlock_write_unlock(&list->lock); - return 1; -} - -void -mlx5_list_destroy(struct mlx5_list *list) -{ - struct mlx5_list_entry *entry; - int i; - - MLX5_ASSERT(list); - for (i = 0; i <= RTE_MAX_LCORE; i++) { - while (!LIST_EMPTY(&list->cache[i].h)) { - entry = LIST_FIRST(&list->cache[i].h); - LIST_REMOVE(entry, next); - if (i == RTE_MAX_LCORE) { - list->cb_remove(list, entry); - DRV_LOG(DEBUG, "mlx5 list %s entry %p " - "destroyed.", list->name, - (void *)entry); - } else { - list->cb_clone_free(list, entry); - } - } - } - mlx5_free(list); -} - -uint32_t -mlx5_list_get_entry_num(struct mlx5_list *list) -{ - MLX5_ASSERT(list); - return __atomic_load_n(&list->count, __ATOMIC_RELAXED); -} - /********************* Indexed pool **********************/ static inline void diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h index ea64bb75c9..cf3db89403 100644 --- a/drivers/net/mlx5/mlx5_utils.h +++ b/drivers/net/mlx5/mlx5_utils.h @@ -297,203 +297,6 @@ log2above(unsigned int v) return l + r; } -/************************ mlx5 list *****************************/ - -/** Maximum size of string for naming. */ -#define MLX5_NAME_SIZE 32 - -struct mlx5_list; - -/** - * Structure of the entry in the mlx5 list, user should define its own struct - * that contains this in order to store the data. - */ -struct mlx5_list_entry { - LIST_ENTRY(mlx5_list_entry) next; /* Entry pointers in the list. */ - uint32_t ref_cnt; /* 0 means, entry is invalid. */ - uint32_t lcore_idx; - struct mlx5_list_entry *gentry; -}; - -struct mlx5_list_cache { - LIST_HEAD(mlx5_list_head, mlx5_list_entry) h; - uint32_t inv_cnt; /* Invalid entries counter. */ -} __rte_cache_aligned; - -/** - * Type of callback function for entry removal. - * - * @param list - * The mlx5 list. - * @param entry - * The entry in the list. - */ -typedef void (*mlx5_list_remove_cb)(struct mlx5_list *list, - struct mlx5_list_entry *entry); - -/** - * Type of function for user defined matching. - * - * @param list - * The mlx5 list. - * @param entry - * The entry in the list. - * @param ctx - * The pointer to new entry context. - * - * @return - * 0 if matching, non-zero number otherwise. - */ -typedef int (*mlx5_list_match_cb)(struct mlx5_list *list, - struct mlx5_list_entry *entry, void *ctx); - -typedef struct mlx5_list_entry *(*mlx5_list_clone_cb) - (struct mlx5_list *list, - struct mlx5_list_entry *entry, void *ctx); - -typedef void (*mlx5_list_clone_free_cb)(struct mlx5_list *list, - struct mlx5_list_entry *entry); - -/** - * Type of function for user defined mlx5 list entry creation. - * - * @param list - * The mlx5 list. - * @param entry - * The new allocated entry, NULL if list entry size unspecified, - * New entry has to be allocated in callback and return. - * @param ctx - * The pointer to new entry context. - * - * @return - * Pointer of entry on success, NULL otherwise. - */ -typedef struct mlx5_list_entry *(*mlx5_list_create_cb) - (struct mlx5_list *list, - struct mlx5_list_entry *entry, - void *ctx); - -/** - * Linked mlx5 list structure. - * - * Entry in mlx5 list could be reused if entry already exists, - * reference count will increase and the existing entry returns. - * - * When destroy an entry from list, decrease reference count and only - * destroy when no further reference. - * - * Linked list is designed for limited number of entries, - * read mostly, less modification. - * - * For huge amount of entries, please consider hash list. - * - */ -struct mlx5_list { - char name[MLX5_NAME_SIZE]; /**< Name of the mlx5 list. */ - volatile uint32_t gen_cnt; - /* List modification will update generation count. */ - volatile uint32_t count; /* number of entries in list. */ - void *ctx; /* user objects target to callback. */ - rte_rwlock_t lock; /* read/write lock. */ - mlx5_list_create_cb cb_create; /**< entry create callback. */ - mlx5_list_match_cb cb_match; /**< entry match callback. */ - mlx5_list_remove_cb cb_remove; /**< entry remove callback. */ - mlx5_list_clone_cb cb_clone; /**< entry clone callback. */ - mlx5_list_clone_free_cb cb_clone_free; - struct mlx5_list_cache cache[RTE_MAX_LCORE + 1]; - /* Lcore cache, last index is the global cache. */ -}; - -/** - * Create a mlx5 list. - * - * @param list - * Pointer to the hast list table. - * @param name - * Name of the mlx5 list. - * @param ctx - * Pointer to the list context data. - * @param cb_create - * Callback function for entry create. - * @param cb_match - * Callback function for entry match. - * @param cb_remove - * Callback function for entry remove. - * @return - * List pointer on success, otherwise NULL. - */ -struct mlx5_list *mlx5_list_create(const char *name, void *ctx, - mlx5_list_create_cb cb_create, - mlx5_list_match_cb cb_match, - mlx5_list_remove_cb cb_remove, - mlx5_list_clone_cb cb_clone, - mlx5_list_clone_free_cb cb_clone_free); - -/** - * Search an entry matching the key. - * - * Result returned might be destroyed by other thread, must use - * this function only in main thread. - * - * @param list - * Pointer to the mlx5 list. - * @param ctx - * Common context parameter used by entry callback function. - * - * @return - * Pointer of the list entry if found, NULL otherwise. - */ -struct mlx5_list_entry *mlx5_list_lookup(struct mlx5_list *list, - void *ctx); - -/** - * Reuse or create an entry to the mlx5 list. - * - * @param list - * Pointer to the hast list table. - * @param ctx - * Common context parameter used by callback function. - * - * @return - * registered entry on success, NULL otherwise - */ -struct mlx5_list_entry *mlx5_list_register(struct mlx5_list *list, - void *ctx); - -/** - * Remove an entry from the mlx5 list. - * - * User should guarantee the validity of the entry. - * - * @param list - * Pointer to the hast list. - * @param entry - * Entry to be removed from the mlx5 list table. - * @return - * 0 on entry removed, 1 on entry still referenced. - */ -int mlx5_list_unregister(struct mlx5_list *list, - struct mlx5_list_entry *entry); - -/** - * Destroy the mlx5 list. - * - * @param list - * Pointer to the mlx5 list. - */ -void mlx5_list_destroy(struct mlx5_list *list); - -/** - * Get entry number from the mlx5 list. - * - * @param list - * Pointer to the hast list. - * @return - * mlx5 list entry number. - */ -uint32_t -mlx5_list_get_entry_num(struct mlx5_list *list); - /********************************* indexed pool *************************/ /**