From patchwork Fri Jul 2 06:18:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suanming Mou X-Patchwork-Id: 95162 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 21BA5A0A0C; Fri, 2 Jul 2021 08:19:50 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EF56341376; Fri, 2 Jul 2021 08:18:53 +0200 (CEST) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2072.outbound.protection.outlook.com [40.107.236.72]) by mails.dpdk.org (Postfix) with ESMTP id 4FBAF4135B for ; Fri, 2 Jul 2021 08:18:50 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gP+Sal7KvKZBv3V+KTVZjXp+Doh0moVEoyQLNT6RV+BgeGVjK4/5HNLcdmk2Fby49P1BQDgdwVoPYRe4fgPsstDXtX7jwvqvXHzdcGX/T4vVAd48slGkF4TiSGmiXUTnlB9c1coP58R9s1Ek29vX5c7LEt93g/1ERzfCOFbn5qWsmcqXuj3oq7HrEZucLXkyQKs0iA299MC6PW63iDiKJYdmI8dgkRxX/QtkOP0oZO7asihQdwFqrR1VzD+11C+HuUjsI0qfvwcvPMr3zipgwTx3xA8mKRm9RXfpxgsBcICr7S7pZVHFP7kUoIXXTNDkGbxOMnlZFUDPZFKnnRfoBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wrUW7GrTNGe2j3O0k+er+bpDBYzHMA3HhlzYmRGx6jY=; b=Jnqv3qv2EqvmJz8punNMe+tqnNoAHtrn2047XWFUlAhkzsvkI3c4/izS4i1k9TIKhxbMfPeoHO88mAtU8gidyMxa8LLxKRm/177NNhjY991SZp3IRXB525zbVUDL+rIe+fnQ5eQKpVt3Vdss2bPAvG1ftVnK1cCGTd+gbttSDiC23AxkEQUMjoJxaZjpbRQ/tX+CAIz7c+OH8VFUnVVBpGjhDLOyY3fwJjoYv5dDL/mmwWki48MBdAF99R3ZrsxXaJQT3qZzHXqEIDFryFvlIqACKmvSOffuehjeN8eFMw+bOa+1H3If+bjetAcu0OM4KEDzr4XVM38vpJjv2LHOaA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wrUW7GrTNGe2j3O0k+er+bpDBYzHMA3HhlzYmRGx6jY=; b=d8qqMP53sWtgwpJG60Zs+Y4D3F+Au3cWMa+GqLlKBoD6IWSskl05DhXHgbdhChFrlpmeN49S2O7/JS9F5tBR5/gshJZ4HWkiQJcvF0AyqVy7mTAdO8bWzhgt8B9QCut9nrr7VrzTsFOdnMm7lvEnPPRRcNLQM8X1W7yQI/0zDbVS1JVntQYn48soK+h5MFxSy7xjEmTF5SW+zuIJJJJJsnQbZMi9PzO1IC71MP3OuWlH3lRpi1yqe6hujQwWoxb7d7LUUvL9agici26UxTGCk2STKplU3xYkyFDb7XihRicha9MTVE+e5tqlWKTlID4SwBg1vVtvFQZdJpPV1JJKgg== Received: from MW4PR03CA0100.namprd03.prod.outlook.com (2603:10b6:303:b7::15) by MWHPR12MB1311.namprd12.prod.outlook.com (2603:10b6:300:13::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4287.23; Fri, 2 Jul 2021 06:18:46 +0000 Received: from CO1NAM11FT022.eop-nam11.prod.protection.outlook.com (2603:10b6:303:b7:cafe::60) by MW4PR03CA0100.outlook.office365.com (2603:10b6:303:b7::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4287.21 via Frontend Transport; Fri, 2 Jul 2021 06:18:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by CO1NAM11FT022.mail.protection.outlook.com (10.13.175.199) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4287.22 via Frontend Transport; Fri, 2 Jul 2021 06:18:46 +0000 Received: from nvidia.com (172.20.187.5) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 2 Jul 2021 06:18:43 +0000 From: Suanming Mou To: , CC: , , Date: Fri, 2 Jul 2021 09:18:02 +0300 Message-ID: <20210702061816.10454-9-suanmingm@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20210702061816.10454-1-suanmingm@nvidia.com> References: <20210527093403.1153127-1-suanmingm@nvidia.com> <20210702061816.10454-1-suanmingm@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1e23963e-40be-43fc-e7d9-08d93d21400d X-MS-TrafficTypeDiagnostic: MWHPR12MB1311: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:883; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mYyE7IMsJqdrav52VwJW7V+ap1EaqeawT7JLjOb0y/4/4d66s9dXPYBsUPPdASrdBjIwF5j2JsTLdYp4J1CrolUdtpr7uuKcw9/NMCU0Kx1Oll+FS+azbkfTWNkTqWsiuRREGPoySEKcnkegMCHM0yZBgk1WTL3wdj9M9ylSAau2RcYbfd6dBf8qv8ReDeR1GeUfzzHf0Q6VJcdf3Esn+qU9uR6u9v27IBS3Gf5hpmDk0/nkDMRugavtC1Hpj8tPKx7F/TjOhWWHzi6NH9D1h55C7Ha8+z9m3rw+8urNEQyncEMFIMxjxYH2puDhF6gbyRMUPXqw/P8xwA2xYtS9sVPv0rUb76S/9aFMraUJdu3dHs5xC37BZHzkbnl6BJQ4O/jdTiVZ1ZCu1wVp8RQ4eaG/w6Hg2DXlaWqfVXTDJfRx5YmMAMF4mDTVmrs8ouXYTmqrdqS7i5OqdTvpc+jbIOQVHhRldz0wbeKwYt1cmXuLLpkc9z41fU4Am+5Tx/ckDEFAc4tpTWmMgDqgWfwRliZMc5dwCYICJ39ocZDWQ1vyItDSC7PXf5XPGvK3cUyInpsjur6aP9mPcbCd9SV6oism7qOBkwMAyj8nGcEzwxSC1LW2Y1G4KXhdG+TIwbY/Duzx+AmEr8I5O8EJBuhKSJxA2Jloik4HfT73sY/S9MJbE1/YhjKkDWSONh4CfjgI8vGWcZesSIYL8pFDrPpuLg== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(396003)(39860400002)(376002)(346002)(136003)(36840700001)(46966006)(36906005)(8936002)(82310400003)(82740400003)(6636002)(7696005)(47076005)(70586007)(70206006)(8676002)(26005)(316002)(6286002)(110136005)(54906003)(2616005)(1076003)(336012)(36860700001)(55016002)(478600001)(7636003)(16526019)(426003)(5660300002)(186003)(2906002)(356005)(4326008)(36756003)(83380400001)(6666004)(86362001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jul 2021 06:18:46.6414 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1e23963e-40be-43fc-e7d9-08d93d21400d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT022.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1311 Subject: [dpdk-dev] [PATCH v3 08/22] net/mlx5: minimize list critical sections X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Matan Azrad The mlx5 internal list utility is thread safe. In order to synchronize list access between the threads, a RW lock is taken for the critical sections. The create\remove\clone\clone_free operations are in the critical sections. These operations are heavy and make the critical sections heavy because they are used for memory and other resources allocations\deallocations. Moved out the operations from the critical sections and use generation counter in order to detect parallel allocations. Signed-off-by: Matan Azrad Acked-by: Suanming Mou --- drivers/net/mlx5/mlx5_utils.c | 86 ++++++++++++++++++----------------- drivers/net/mlx5/mlx5_utils.h | 5 +- 2 files changed, 48 insertions(+), 43 deletions(-) diff --git a/drivers/net/mlx5/mlx5_utils.c b/drivers/net/mlx5/mlx5_utils.c index 51cca68ea9..772b352af5 100644 --- a/drivers/net/mlx5/mlx5_utils.c +++ b/drivers/net/mlx5/mlx5_utils.c @@ -101,7 +101,7 @@ mlx5_list_cache_insert(struct mlx5_list *list, int lcore_index, { struct mlx5_list_entry *lentry = list->cb_clone(list, gentry, ctx); - if (!lentry) + if (unlikely(!lentry)) return NULL; lentry->ref_cnt = 1u; lentry->gentry = gentry; @@ -112,8 +112,8 @@ mlx5_list_cache_insert(struct mlx5_list *list, int lcore_index, struct mlx5_list_entry * mlx5_list_register(struct mlx5_list *list, void *ctx) { - struct mlx5_list_entry *entry, *lentry; - uint32_t prev_gen_cnt = 0; + struct mlx5_list_entry *entry, *local_entry; + volatile uint32_t prev_gen_cnt = 0; int lcore_index = rte_lcore_index(rte_lcore_id()); MLX5_ASSERT(list); @@ -122,51 +122,56 @@ mlx5_list_register(struct mlx5_list *list, void *ctx) rte_errno = ENOTSUP; return NULL; } - /* Lookup in local cache. */ - lentry = __list_lookup(list, lcore_index, ctx, true); - if (lentry) - return lentry; - /* Lookup with read lock, reuse if found. */ + /* 1. Lookup in local cache. */ + local_entry = __list_lookup(list, lcore_index, ctx, true); + if (local_entry) + return local_entry; + /* 2. Lookup with read lock on global list, reuse if found. */ rte_rwlock_read_lock(&list->lock); entry = __list_lookup(list, RTE_MAX_LCORE, ctx, true); - if (entry == NULL) { - prev_gen_cnt = __atomic_load_n(&list->gen_cnt, - __ATOMIC_ACQUIRE); - rte_rwlock_read_unlock(&list->lock); - } else { + if (likely(entry)) { rte_rwlock_read_unlock(&list->lock); return mlx5_list_cache_insert(list, lcore_index, entry, ctx); } - /* Not found, append with write lock - block read from other threads. */ + prev_gen_cnt = list->gen_cnt; + rte_rwlock_read_unlock(&list->lock); + /* 3. Prepare new entry for global list and for cache. */ + entry = list->cb_create(list, entry, ctx); + if (unlikely(!entry)) + return NULL; + local_entry = list->cb_clone(list, entry, ctx); + if (unlikely(!local_entry)) { + list->cb_remove(list, entry); + return NULL; + } + entry->ref_cnt = 1u; + local_entry->ref_cnt = 1u; + local_entry->gentry = entry; rte_rwlock_write_lock(&list->lock); - /* If list changed by other threads before lock, search again. */ - if (prev_gen_cnt != __atomic_load_n(&list->gen_cnt, __ATOMIC_ACQUIRE)) { - /* Lookup and reuse w/o read lock. */ - entry = __list_lookup(list, RTE_MAX_LCORE, ctx, true); - if (entry) { + /* 4. Make sure the same entry was not created before the write lock. */ + if (unlikely(prev_gen_cnt != list->gen_cnt)) { + struct mlx5_list_entry *oentry = __list_lookup(list, + RTE_MAX_LCORE, + ctx, true); + + if (unlikely(oentry)) { + /* 4.5. Found real race!!, reuse the old entry. */ rte_rwlock_write_unlock(&list->lock); - return mlx5_list_cache_insert(list, lcore_index, entry, - ctx); - } - } - entry = list->cb_create(list, entry, ctx); - if (entry) { - lentry = mlx5_list_cache_insert(list, lcore_index, entry, ctx); - if (!lentry) { list->cb_remove(list, entry); - } else { - entry->ref_cnt = 1u; - LIST_INSERT_HEAD(&list->cache[RTE_MAX_LCORE].h, entry, - next); - __atomic_add_fetch(&list->gen_cnt, 1, __ATOMIC_RELEASE); - __atomic_add_fetch(&list->count, 1, __ATOMIC_ACQUIRE); - DRV_LOG(DEBUG, "mlx5 list %s entry %p new: %u.", - list->name, (void *)entry, entry->ref_cnt); + list->cb_clone_free(list, local_entry); + return mlx5_list_cache_insert(list, lcore_index, oentry, + ctx); } - } + /* 5. Update lists. */ + LIST_INSERT_HEAD(&list->cache[RTE_MAX_LCORE].h, entry, next); + list->gen_cnt++; rte_rwlock_write_unlock(&list->lock); - return lentry; + LIST_INSERT_HEAD(&list->cache[lcore_index].h, local_entry, next); + __atomic_add_fetch(&list->count, 1, __ATOMIC_ACQUIRE); + DRV_LOG(DEBUG, "mlx5 list %s entry %p new: %u.", + list->name, (void *)entry, entry->ref_cnt); + return local_entry; } int @@ -180,12 +185,11 @@ mlx5_list_unregister(struct mlx5_list *list, if (__atomic_sub_fetch(&gentry->ref_cnt, 1, __ATOMIC_ACQUIRE) != 0) return 1; rte_rwlock_write_lock(&list->lock); - if (__atomic_load_n(&gentry->ref_cnt, __ATOMIC_ACQUIRE) == 0) { - __atomic_add_fetch(&list->gen_cnt, 1, __ATOMIC_ACQUIRE); - __atomic_sub_fetch(&list->count, 1, __ATOMIC_ACQUIRE); + if (likely(gentry->ref_cnt == 0)) { LIST_REMOVE(gentry, next); - list->cb_remove(list, gentry); rte_rwlock_write_unlock(&list->lock); + list->cb_remove(list, gentry); + __atomic_sub_fetch(&list->count, 1, __ATOMIC_ACQUIRE); DRV_LOG(DEBUG, "mlx5 list %s entry %p removed.", list->name, (void *)gentry); return 0; diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h index 24ae2b2ccb..684d1e8a2a 100644 --- a/drivers/net/mlx5/mlx5_utils.h +++ b/drivers/net/mlx5/mlx5_utils.h @@ -387,8 +387,9 @@ typedef struct mlx5_list_entry *(*mlx5_list_create_cb) */ struct mlx5_list { char name[MLX5_NAME_SIZE]; /**< Name of the mlx5 list. */ - uint32_t gen_cnt; /* List modification will update generation count. */ - uint32_t count; /* number of entries in list. */ + volatile uint32_t gen_cnt; + /* List modification will update generation count. */ + volatile uint32_t count; /* number of entries in list. */ void *ctx; /* user objects target to callback. */ rte_rwlock_t lock; /* read/write lock. */ mlx5_list_create_cb cb_create; /**< entry create callback. */