From patchwork Thu Oct 26 09:14:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bing Zhao X-Patchwork-Id: 133383 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 85D2743206; Thu, 26 Oct 2023 11:15:16 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C3DE642DA6; Thu, 26 Oct 2023 11:15:15 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2045.outbound.protection.outlook.com [40.107.92.45]) by mails.dpdk.org (Postfix) with ESMTP id F20B8402C5; Thu, 26 Oct 2023 11:15:13 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cdm1TCeqMMOGv/aM7NOYJTmRQunMwUjNpr4M4zkGHs8NDcm8HXC1n5xofMK4dBe2vOW8W28TBfpbiGVGuyIvTghX/Jm8AaPps5Nfim+RunGUzACrrq9MWaY6Mb/TY9CmSd+1Fx3An72J3dZRUiNqpFY/F1gb6LozxfjXb0JY65W7+2BQ4znB7JTUKgfuNIzC4pe+OeYQI60KnOhxaPRuYQr7vp1YDiEKxv2inDylnCDHVkpJn7IV7+1+t7Fi81Y19N4rEtLLSGAzivqec64FepOEJbN5TJKsjWSohaM/sXnR3KIE3xYh4FH/V+nsJdT5/tAXv6vTsd43Q6/EC9iGIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DxOJON5QvQogMrjgbnOyRAlaAKIrR7pdv2t6UR4UXds=; b=Ai0zHwjfqPsJPW4MCRIO6UYQGGj2bWakMRFCZl/dWvyYeSI3615aLXsUeVT4UKoFPj27BS/yoSgPS6P04JbJ+zvNkk/aZBhSiC21ggDtVj2UfSe3gXue+4DjJpfWa/m2hjbrSre//I3Xq0G+ksp9mhyh6m0ONrJ+T40S24NOpql/bkRXT40Rqse9Bp3rAsrx/t67jZurIZuZVpqdqY9N76kKTFC2eL5y68ohfhIHqBWjW687Ft8uCpn8R3DN3teLAKiec/dRcz69zRk/oU3yR9bT6V4RJwALIVfo4K5YzH+I/AEG9H90vwaPZQUBSqvDbrfnYuOj6ndNAdOvNdC5Og== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DxOJON5QvQogMrjgbnOyRAlaAKIrR7pdv2t6UR4UXds=; b=buDnFgsCwz6Vqqj8vyRMaRWI+KsnNZ87MPs5/FToKSv7lnyqgwM+B66Amyhjo+E7Wc6r9y7aqbuCEjGgJAg+Ve8JAa2hkfn7pF2RB5zIiWUztWzNd18b5q2++Nyw7zO6dqzwrePV+RJvouGzoAgoCVeMOWrDXOQQzOIYM8kM4gwfNxqk9PMhmEGyvwg8LYgCwwp+aCakDVxUCykDG+ugxc81G0AJtLmZKsZoQOXcGQiorhdCKbIcAiHRVbULeH117qniHn64qfEUZVt3gDfJhyuQZuEM44H2+apka8VsOZTvVeLWpuciFexgAVnghSDLm+inouqKkmFjPYVJ06zK+A== Received: from SA9PR13CA0138.namprd13.prod.outlook.com (2603:10b6:806:27::23) by DM6PR12MB4370.namprd12.prod.outlook.com (2603:10b6:5:2aa::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.19; Thu, 26 Oct 2023 09:15:11 +0000 Received: from SA2PEPF00001509.namprd04.prod.outlook.com (2603:10b6:806:27:cafe::77) by SA9PR13CA0138.outlook.office365.com (2603:10b6:806:27::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.8 via Frontend Transport; Thu, 26 Oct 2023 09:15:10 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SA2PEPF00001509.mail.protection.outlook.com (10.167.242.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.15 via Frontend Transport; Thu, 26 Oct 2023 09:15:10 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 26 Oct 2023 02:15:02 -0700 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 26 Oct 2023 02:15:00 -0700 From: Bing Zhao To: , , , , CC: , , Subject: [PATCH] net/mlx5: fix the workspace double free in Windows Date: Thu, 26 Oct 2023 12:14:47 +0300 Message-ID: <20231026091447.10991-1-bingz@nvidia.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001509:EE_|DM6PR12MB4370:EE_ X-MS-Office365-Filtering-Correlation-Id: d971481a-724d-42a8-0090-08dbd6040e00 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CkbgSnzWSmJt7Wqlbmv3UwQG+LWDwfjUU8bVLD2aZjNkEdSd+wflJp2DtEPFGLMocpqJAVD+Q6LwfOIdMuyugYS8sKnZB1MC1Ct18g3FoISkfNGKmq4qKPjfyDaRaT/hqoumLWE5VLUbCMVLakUfqR+1YbwTMYB9xFZ/Y59oHL5AV86VZtTCC+znjp1tiu6FPA/UvMcbUY/MLGYsP+/+wl+BvXK5tGDvi/eV9O0HOZqqBZhY+F4hbyeTJBo/f2W2e35FMsLwAqtr68/AowG3PFpL7Xf85KIlXE2rShJPtgtTpFmExCP2goeVxP7w3gwOo8uFywYKMx8l5ykWvmFUaG/HDhgwsNeX7aoXTniqoII9qnFTu6fAWlhYHMuK6uN+QVA9XmF2chiJHrXx8fDCe6QYzQ/pwzhLpofs4u792vk7Jy+c5+/50ezx1IYLFphFS/K22ho8fS8CWQwf4QywjCy7XZ6JA3EFDACiQeiySpnWLt6bfgk/+tvTjQEvEiZU4rRjs2+HCc9jAc6bi2bd0KgWp7ARwQnugJbfVyH4Y6vlYEUzWnfXbJ4+zUSVsU7gRiLjeQEfo8+0t8rQ+MvphAUw68pLDf1+B1hhgjqkUZvqeG3M1RYF48k4wonmcFLnCDZfL736xwl0hhrjoJ4GlzMjzpG/jjTKmwWDqk003o4WNOX+ApG0hGZHKF4y62/F/AfVglKCic9ivxF8fDQdEgNm77P7LCMVR0FlLqpIFuN6QYPK2REE0ArBUTW5XzmF X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(39860400002)(376002)(396003)(346002)(136003)(230922051799003)(64100799003)(82310400011)(451199024)(1800799009)(186009)(36840700001)(40470700004)(46966006)(2906002)(41300700001)(4326008)(5660300002)(8676002)(6636002)(8936002)(316002)(450100002)(55016003)(70206006)(54906003)(110136005)(40460700003)(478600001)(2616005)(70586007)(6666004)(7696005)(426003)(83380400001)(26005)(336012)(40480700001)(6286002)(16526019)(47076005)(36860700001)(36756003)(7636003)(86362001)(1076003)(82740400003)(356005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Oct 2023 09:15:10.4432 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d971481a-724d-42a8-0090-08dbd6040e00 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001509.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4370 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The thread specific variable workspace indicated by "key_workspace" should be freed explicitly when closing a device. For example, in Linux, when exiting an application, the thread will not exit explicitly and the thread resources will not be deconstructed. The commit to solve this introduced a global list to manage the workspace resources as a garbage collector. It will also be executed in Windows, but the workspaces have already been freed in the function mlx5_flow_os_release_workspace(). With this commit, the garbage collector will only be executed in Linux. The workspace resources management in Windows will remain the same with some stub function when needed. Fixes: dc7c5e0aa905 ("net/mlx5: fix flow workspace destruction") Cc: getelson@nvidia.com Cc: stable@dpdk.org Signed-off-by: Bing Zhao Acked-by: Matan Azrad --- drivers/net/mlx5/linux/mlx5_flow_os.c | 25 +++++++++++++++++++++++++ drivers/net/mlx5/linux/mlx5_flow_os.h | 8 ++++++++ drivers/net/mlx5/mlx5.c | 1 - drivers/net/mlx5/mlx5_flow.c | 25 +------------------------ drivers/net/mlx5/mlx5_flow.h | 1 - drivers/net/mlx5/windows/mlx5_flow_os.c | 6 ++++++ drivers/net/mlx5/windows/mlx5_flow_os.h | 8 ++++++++ 7 files changed, 48 insertions(+), 26 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_flow_os.c b/drivers/net/mlx5/linux/mlx5_flow_os.c index b139bb75b9..2767b11708 100644 --- a/drivers/net/mlx5/linux/mlx5_flow_os.c +++ b/drivers/net/mlx5/linux/mlx5_flow_os.c @@ -8,6 +8,10 @@ /* Key of thread specific flow workspace data. */ static rte_thread_key key_workspace; +/* Flow workspace global list head for garbage collector. */ +static struct mlx5_flow_workspace *gc_head; +/* Spinlock for operating flow workspace list. */ +static rte_spinlock_t mlx5_flow_workspace_lock = RTE_SPINLOCK_INITIALIZER; int mlx5_flow_os_validate_item_esp(const struct rte_flow_item *item, @@ -48,6 +52,26 @@ mlx5_flow_os_validate_item_esp(const struct rte_flow_item *item, return 0; } +void +mlx5_flow_os_workspace_gc_add(struct mlx5_flow_workspace *ws) +{ + rte_spinlock_lock(&mlx5_flow_workspace_lock); + ws->gc = gc_head; + gc_head = ws; + rte_spinlock_unlock(&mlx5_flow_workspace_lock); +} + +static void +mlx5_flow_os_workspace_gc_release(void) +{ + while (gc_head) { + struct mlx5_flow_workspace *wks = gc_head; + + gc_head = wks->gc; + flow_release_workspace(wks); + } +} + int mlx5_flow_os_init_workspace_once(void) { @@ -75,4 +99,5 @@ void mlx5_flow_os_release_workspace(void) { rte_thread_key_delete(key_workspace); + mlx5_flow_os_workspace_gc_release(); } diff --git a/drivers/net/mlx5/linux/mlx5_flow_os.h b/drivers/net/mlx5/linux/mlx5_flow_os.h index ed71289322..3f7a94c9ee 100644 --- a/drivers/net/mlx5/linux/mlx5_flow_os.h +++ b/drivers/net/mlx5/linux/mlx5_flow_os.h @@ -526,4 +526,12 @@ mlx5_flow_os_validate_item_esp(const struct rte_flow_item *item, uint8_t target_protocol, struct rte_flow_error *error); +/** + * Add per thread workspace to the global list for garbage collection. + * + * @param[in] ws + * Pointer to the flow workspace. + */ +void mlx5_flow_os_workspace_gc_add(struct mlx5_flow_workspace *ws); + #endif /* RTE_PMD_MLX5_FLOW_OS_H_ */ diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 997df595d0..a673d626ac 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1851,7 +1851,6 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh) if (LIST_EMPTY(&mlx5_dev_ctx_list)) { mlx5_os_net_cleanup(); mlx5_flow_os_release_workspace(); - mlx5_flow_workspace_gc_release(); } pthread_mutex_unlock(&mlx5_dev_ctx_list_mutex); if (sh->flex_parsers_dv) { diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c index 772f6afb66..c3a611517b 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c @@ -8093,29 +8093,6 @@ flow_release_workspace(void *data) } } -static struct mlx5_flow_workspace *gc_head; -static rte_spinlock_t mlx5_flow_workspace_lock = RTE_SPINLOCK_INITIALIZER; - -static void -mlx5_flow_workspace_gc_add(struct mlx5_flow_workspace *ws) -{ - rte_spinlock_lock(&mlx5_flow_workspace_lock); - ws->gc = gc_head; - gc_head = ws; - rte_spinlock_unlock(&mlx5_flow_workspace_lock); -} - -void -mlx5_flow_workspace_gc_release(void) -{ - while (gc_head) { - struct mlx5_flow_workspace *wks = gc_head; - - gc_head = wks->gc; - flow_release_workspace(wks); - } -} - /** * Get thread specific current flow workspace. * @@ -8172,7 +8149,7 @@ mlx5_flow_push_thread_workspace(void) data = flow_alloc_thread_workspace(); if (!data) return NULL; - mlx5_flow_workspace_gc_add(data); + mlx5_flow_os_workspace_gc_add(data); } else if (!curr->inuse) { data = curr; } else if (curr->next) { diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index 903ff66d72..4edca39936 100644 --- a/drivers/net/mlx5/mlx5_flow.h +++ b/drivers/net/mlx5/mlx5_flow.h @@ -2032,7 +2032,6 @@ struct mlx5_flow_driver_ops { struct mlx5_flow_workspace *mlx5_flow_push_thread_workspace(void); void mlx5_flow_pop_thread_workspace(void); struct mlx5_flow_workspace *mlx5_flow_get_thread_workspace(void); -void mlx5_flow_workspace_gc_release(void); __extension__ struct flow_grp_info { diff --git a/drivers/net/mlx5/windows/mlx5_flow_os.c b/drivers/net/mlx5/windows/mlx5_flow_os.c index 5013e9f012..f907b21ecc 100644 --- a/drivers/net/mlx5/windows/mlx5_flow_os.c +++ b/drivers/net/mlx5/windows/mlx5_flow_os.c @@ -417,6 +417,12 @@ mlx5_flow_os_set_specific_workspace(struct mlx5_flow_workspace *data) return err; } +void +mlx5_flow_os_workspace_gc_add(struct mlx5_flow_workspace *ws) +{ + RTE_SET_USED(ws); +} + int mlx5_flow_os_validate_item_esp(const struct rte_flow_item *item, uint64_t item_flags, diff --git a/drivers/net/mlx5/windows/mlx5_flow_os.h b/drivers/net/mlx5/windows/mlx5_flow_os.h index 1c1c17fc41..856d8ba948 100644 --- a/drivers/net/mlx5/windows/mlx5_flow_os.h +++ b/drivers/net/mlx5/windows/mlx5_flow_os.h @@ -473,4 +473,12 @@ mlx5_flow_os_validate_item_esp(const struct rte_flow_item *item, uint8_t target_protocol, struct rte_flow_error *error); +/** + * Add per thread workspace to the global list for garbage collection. + * + * @param[in] ws + * Pointer to the flow workspace. + */ +void mlx5_flow_os_workspace_gc_add(struct mlx5_flow_workspace *ws); + #endif /* RTE_PMD_MLX5_FLOW_OS_H_ */