From patchwork Tue Jul 6 13:32:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suanming Mou X-Patchwork-Id: 95396 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 73F14A0C47; Tue, 6 Jul 2021 15:34:33 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 63E7D413EF; Tue, 6 Jul 2021 15:33:40 +0200 (CEST) Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam07on2057.outbound.protection.outlook.com [40.107.95.57]) by mails.dpdk.org (Postfix) with ESMTP id 0F88A413CE for ; Tue, 6 Jul 2021 15:33:37 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QIdthcfmYqSmh/TofypmCq+TQ7q5cW0fT1+Ecb/tnAn8v3P2CwK6GGjg4oRih3bSvTNoZIkOI5qKP3r0l3TGiUXOgoYSqDdZpgzOGBDpVCZUfdCqoQ9JpLewin6S+WLgH05taEQ98tn+qgHcXS+bKGWMzNy4QnnjTwXY1PTJ8upGiGWhwkYwkZGJ7BEn5TGGW7Ju7rwc/rdUm/wVFRbGLkVi7dng44qxgEVXT7izUDnaF0JxJm4yyiocUrSF2FAgk7QjHTABsCzyr9+nzhbnSHBy7kzof7d+QzkSssqBaAdtst6tR1TJWT6i7lDQBjMLNVPK5yRSdmt4cl4HaL74rw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uV93wypaIqcIjR6KAkI2lwhTr9fA7fZQQi0UUl2XMbY=; b=hVNbSS5AvpuUqLDSjIkbDAW9yw1T8aif2yLUg+h2FtBWLu3WAcz++kyJRp8YanvD/XMUPORCtiHD4rk8vAWQcwvF+nkVnrPxb9++6SqqHMuGYFSk9YQtvb5etCIc/NKOei7azNU6kAh0K5CXcrgqYJSBRDhLI9HIXg3WKthCy8qHub5s4wWpDG3TdzFulrPObBSUwXS5+WuCwg175m6ghHKkyaF0Kth2rGY9X7VTVHVBAgDfUPjArKu0lJuLvIMURjVAcHqmvozmLCcgVnoDRCWpqpnRqNrXmiczYD8oIR0b1tvKPdt4r+u9KN5bW10+fu3YSyNjOPI98veZX4OzYw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uV93wypaIqcIjR6KAkI2lwhTr9fA7fZQQi0UUl2XMbY=; b=KZOGJJJJOvmHKR/scDe4otTNoVRZiuDQXwCx4g/5hbKRPDadQqJN6NvliFy5HDtottGnnGzfk02EsfC3GIzgxoVRPf8WgymztqBhh2BdtSMZt5bTnVMIgQ7gF61dxlVsxcCx6oPUt2afkw7LUfXFA/ZuyMFAJOacrN/psZuFljCrfEtVHaOx9MoTx61R7vIrQllZnIHgUHof+DY11zWzZHkPDTI9p65sfVKhNbh3eZ+cb6O0jem8jP69QHqul2ueDGjG/SZTUGh+X41cBM6e6eTcExiuur8/Qf0D/3sRrCfpyPp2xbKLYA9H752kuBMjSbBIu1XCuq5A5CfLDaSp2A== Received: from BN0PR02CA0035.namprd02.prod.outlook.com (2603:10b6:408:e5::10) by BN8PR12MB3364.namprd12.prod.outlook.com (2603:10b6:408:40::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4287.27; Tue, 6 Jul 2021 13:33:35 +0000 Received: from BN8NAM11FT047.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e5:cafe::44) by BN0PR02CA0035.outlook.office365.com (2603:10b6:408:e5::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4308.19 via Frontend Transport; Tue, 6 Jul 2021 13:33:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT047.mail.protection.outlook.com (10.13.177.220) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4287.22 via Frontend Transport; Tue, 6 Jul 2021 13:33:34 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 6 Jul 2021 13:33:32 +0000 From: Suanming Mou To: , CC: , , Date: Tue, 6 Jul 2021 16:32:40 +0300 Message-ID: <20210706133257.3353-10-suanmingm@nvidia.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20210706133257.3353-1-suanmingm@nvidia.com> References: <20210527093403.1153127-1-suanmingm@nvidia.com> <20210706133257.3353-1-suanmingm@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL101.nvidia.com (172.20.187.10) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6574b013-162c-469b-a734-08d94082a78e X-MS-TrafficTypeDiagnostic: BN8PR12MB3364: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:883; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1VY5YtEMutNYzOExlH5KfkDsedst0oNAG4YiFc5Gj3nBO1S2wPTDEzRBZlEgu7gjUQIQiQIYacMGqhxXM8Pyr3dBcPDIrLVFuKtqKgZ/cWc3/e89pOcUmNafCo+Kye6TcdZrwI3XvWPOlndXKQFafoGH6b4hIXHiof9boDFp49vE9P1n/1VhyLNNimwcla4q5JYo68nnmsKxN1cek6DGJT+o43G/zpvuzIfPAM5QsSmQ/OBDtGGMxS7L7SUzcrWhbNOg2nH/OVlU6BW4BNy3GtXGPQS3wtRDFR5IjiVLfF+owVQO5HDiiqAUprPaTqL4RVwzgFjOMJlz0YmsyMTx/CBKSl1c3qwM5hFeVjsTyqKdyZQ5+Ln7pqhV41xtBbNwVEwMdLVyz0T2yEUDI8+HAtxqLXuCRZlvRMNMesaD0JI9pq9wj16lwLozqRSlWjyP2apG8D+8gPoV/aDO1cOFdfk8C1UodY4e/2IWQyjVBvnb62D7/jkOcWQYcNuEstoclqbSKlqV3rUtF8LVXBozz4dDL1pMH64SNUorZRGjUqtN8d8zEsAsdQbeZwZCG1ioK4Ao99wljW0/tLq0IEQq5K68Bjexl850FQDrWrR7RDtBKy2esuVEw7wUmln77pL5YilmeJAK2Y/QRHDqwBr2WCcNm+rTZctK4WGD1Stu85mQIqZZcCF6RhnCDf5q4pN94jkTnSqPC+wHcYhYpd/jaw== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(39860400002)(376002)(396003)(136003)(346002)(36840700001)(46966006)(426003)(16526019)(36756003)(86362001)(82310400003)(8936002)(70586007)(6286002)(83380400001)(6636002)(5660300002)(70206006)(47076005)(82740400003)(2616005)(55016002)(7696005)(2906002)(8676002)(7636003)(336012)(26005)(36860700001)(1076003)(54906003)(36906005)(6666004)(356005)(4326008)(110136005)(478600001)(316002)(186003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jul 2021 13:33:34.9006 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6574b013-162c-469b-a734-08d94082a78e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT047.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB3364 Subject: [dpdk-dev] [PATCH v4 09/26] net/mlx5: minimize list critical sections X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Matan Azrad The mlx5 internal list utility is thread safe. In order to synchronize list access between the threads, a RW lock is taken for the critical sections. The create\remove\clone\clone_free operations are in the critical sections. These operations are heavy and make the critical sections heavy because they are used for memory and other resources allocations\deallocations. Moved out the operations from the critical sections and use generation counter in order to detect parallel allocations. Signed-off-by: Matan Azrad Acked-by: Suanming Mou --- drivers/net/mlx5/mlx5_utils.c | 86 ++++++++++++++++++----------------- drivers/net/mlx5/mlx5_utils.h | 5 +- 2 files changed, 48 insertions(+), 43 deletions(-) diff --git a/drivers/net/mlx5/mlx5_utils.c b/drivers/net/mlx5/mlx5_utils.c index f505caed4e..c4c9adb039 100644 --- a/drivers/net/mlx5/mlx5_utils.c +++ b/drivers/net/mlx5/mlx5_utils.c @@ -101,7 +101,7 @@ mlx5_list_cache_insert(struct mlx5_list *list, int lcore_index, { struct mlx5_list_entry *lentry = list->cb_clone(list, gentry, ctx); - if (!lentry) + if (unlikely(!lentry)) return NULL; lentry->ref_cnt = 1u; lentry->gentry = gentry; @@ -112,8 +112,8 @@ mlx5_list_cache_insert(struct mlx5_list *list, int lcore_index, struct mlx5_list_entry * mlx5_list_register(struct mlx5_list *list, void *ctx) { - struct mlx5_list_entry *entry, *lentry; - uint32_t prev_gen_cnt = 0; + struct mlx5_list_entry *entry, *local_entry; + volatile uint32_t prev_gen_cnt = 0; int lcore_index = rte_lcore_index(rte_lcore_id()); MLX5_ASSERT(list); @@ -122,51 +122,56 @@ mlx5_list_register(struct mlx5_list *list, void *ctx) rte_errno = ENOTSUP; return NULL; } - /* Lookup in local cache. */ - lentry = __list_lookup(list, lcore_index, ctx, true); - if (lentry) - return lentry; - /* Lookup with read lock, reuse if found. */ + /* 1. Lookup in local cache. */ + local_entry = __list_lookup(list, lcore_index, ctx, true); + if (local_entry) + return local_entry; + /* 2. Lookup with read lock on global list, reuse if found. */ rte_rwlock_read_lock(&list->lock); entry = __list_lookup(list, RTE_MAX_LCORE, ctx, true); - if (entry == NULL) { - prev_gen_cnt = __atomic_load_n(&list->gen_cnt, - __ATOMIC_ACQUIRE); - rte_rwlock_read_unlock(&list->lock); - } else { + if (likely(entry)) { rte_rwlock_read_unlock(&list->lock); return mlx5_list_cache_insert(list, lcore_index, entry, ctx); } - /* Not found, append with write lock - block read from other threads. */ + prev_gen_cnt = list->gen_cnt; + rte_rwlock_read_unlock(&list->lock); + /* 3. Prepare new entry for global list and for cache. */ + entry = list->cb_create(list, entry, ctx); + if (unlikely(!entry)) + return NULL; + local_entry = list->cb_clone(list, entry, ctx); + if (unlikely(!local_entry)) { + list->cb_remove(list, entry); + return NULL; + } + entry->ref_cnt = 1u; + local_entry->ref_cnt = 1u; + local_entry->gentry = entry; rte_rwlock_write_lock(&list->lock); - /* If list changed by other threads before lock, search again. */ - if (prev_gen_cnt != __atomic_load_n(&list->gen_cnt, __ATOMIC_ACQUIRE)) { - /* Lookup and reuse w/o read lock. */ - entry = __list_lookup(list, RTE_MAX_LCORE, ctx, true); - if (entry) { + /* 4. Make sure the same entry was not created before the write lock. */ + if (unlikely(prev_gen_cnt != list->gen_cnt)) { + struct mlx5_list_entry *oentry = __list_lookup(list, + RTE_MAX_LCORE, + ctx, true); + + if (unlikely(oentry)) { + /* 4.5. Found real race!!, reuse the old entry. */ rte_rwlock_write_unlock(&list->lock); - return mlx5_list_cache_insert(list, lcore_index, entry, - ctx); - } - } - entry = list->cb_create(list, entry, ctx); - if (entry) { - lentry = mlx5_list_cache_insert(list, lcore_index, entry, ctx); - if (!lentry) { list->cb_remove(list, entry); - } else { - entry->ref_cnt = 1u; - LIST_INSERT_HEAD(&list->cache[RTE_MAX_LCORE].h, entry, - next); - __atomic_add_fetch(&list->gen_cnt, 1, __ATOMIC_RELEASE); - __atomic_add_fetch(&list->count, 1, __ATOMIC_ACQUIRE); - DRV_LOG(DEBUG, "mlx5 list %s entry %p new: %u.", - list->name, (void *)entry, entry->ref_cnt); + list->cb_clone_free(list, local_entry); + return mlx5_list_cache_insert(list, lcore_index, oentry, + ctx); } - } + /* 5. Update lists. */ + LIST_INSERT_HEAD(&list->cache[RTE_MAX_LCORE].h, entry, next); + list->gen_cnt++; rte_rwlock_write_unlock(&list->lock); - return lentry; + LIST_INSERT_HEAD(&list->cache[lcore_index].h, local_entry, next); + __atomic_add_fetch(&list->count, 1, __ATOMIC_ACQUIRE); + DRV_LOG(DEBUG, "mlx5 list %s entry %p new: %u.", + list->name, (void *)entry, entry->ref_cnt); + return local_entry; } int @@ -180,12 +185,11 @@ mlx5_list_unregister(struct mlx5_list *list, if (__atomic_sub_fetch(&gentry->ref_cnt, 1, __ATOMIC_ACQUIRE) != 0) return 1; rte_rwlock_write_lock(&list->lock); - if (__atomic_load_n(&gentry->ref_cnt, __ATOMIC_ACQUIRE) == 0) { - __atomic_add_fetch(&list->gen_cnt, 1, __ATOMIC_ACQUIRE); - __atomic_sub_fetch(&list->count, 1, __ATOMIC_ACQUIRE); + if (likely(gentry->ref_cnt == 0)) { LIST_REMOVE(gentry, next); - list->cb_remove(list, gentry); rte_rwlock_write_unlock(&list->lock); + list->cb_remove(list, gentry); + __atomic_sub_fetch(&list->count, 1, __ATOMIC_ACQUIRE); DRV_LOG(DEBUG, "mlx5 list %s entry %p removed.", list->name, (void *)gentry); return 0; diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h index 9e3fe0cb85..6dade8238d 100644 --- a/drivers/net/mlx5/mlx5_utils.h +++ b/drivers/net/mlx5/mlx5_utils.h @@ -388,8 +388,9 @@ typedef struct mlx5_list_entry *(*mlx5_list_create_cb) */ struct mlx5_list { char name[MLX5_NAME_SIZE]; /**< Name of the mlx5 list. */ - uint32_t gen_cnt; /* List modification will update generation count. */ - uint32_t count; /* number of entries in list. */ + volatile uint32_t gen_cnt; + /* List modification will update generation count. */ + volatile uint32_t count; /* number of entries in list. */ void *ctx; /* user objects target to callback. */ rte_rwlock_t lock; /* read/write lock. */ mlx5_list_create_cb cb_create; /**< entry create callback. */