From patchwork Thu Oct 26 09:11:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bing Zhao X-Patchwork-Id: 133382 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3304843206; Thu, 26 Oct 2023 11:12:12 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A8B9F402CE; Thu, 26 Oct 2023 11:12:11 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2079.outbound.protection.outlook.com [40.107.223.79]) by mails.dpdk.org (Postfix) with ESMTP id 00E97402C5 for ; Thu, 26 Oct 2023 11:12:10 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=J0hy00aDL0yAWGTdkWcF4L2g8St+tvbeBA8ssUZb4l9plLKrBTNuU2dxcKSh01ql9lDPnYh9PDeqN/kpBP2bN6nFA2a87IH0cLwhs4ePKsaZMV4sBVmHmnoj9bOiAjP92GZeHAHJNTpIUlpT1q17KD2RlSaTr8HIXbT+VUaOsQCq4Y54SjGn2IwBbEG25N5cooyf+c7jz39nDSaKpschaovXI2mvEO24L7EJ7t3XUGEJv/2Ver8Q+RX7uHP39/vTRRqrfBxI5qcDqoXC7y9/FwetOWGXcQ0z2eVxA5sWPP7aPQIEze8QBPAEG1cICHM2dqaAgReOBYubr6+1XAdXgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oxpPOHe31noKLSJfmiLJGo8c/Os1o812kQw+C6n+Ggo=; b=U97/Y/ImQbGWUPkdAg0spz4lCzCi2Oo4LJyN1VpFSom1GHzgofdM1k0ksIkdgGhZOXcga2bKvM/S6qZabtR9DT1ytnlsx0B60Ae/5WQNEhV61UzlPvA+zxbMbaB3o6H9k/Zfgi1h4e4QD+KDBsbDm3AfEV8uGaQZ++KK4BskvSGHpuz4Vu82o4T3781IH8b/1xLdaulaYuI46PAHOVEFfzp0wgG4Ec/TB0uhGg49gt2gxSakUtB0f6hSBlI6pPHnbEY+Xek9iTbuOWEYeTDe03g2OQavSPx7k7PfGp079XRVf7gkqOqSQSfb1lHgPLc2SkT62pZEpIjs/KOa+A6w/A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oxpPOHe31noKLSJfmiLJGo8c/Os1o812kQw+C6n+Ggo=; b=UA55rsazsKhRFiCX9/HX4tDnZCwWfzN52YeevcdQEdnHebd7sD4Wwa86fR4fFs16OJYh/YmKwCb9NSVfSeuGiF/ZTBBk5gv6jEtVf9Ba4Z8p1vwOecgs2oMZxE0CgVg808DK4E1QIB6mpPLJth0IvkElhM21BnVYiryq1o9O5viw7YPLYFrWkKtx+SHKBRTvHAN1W8B/ketfJVMh1fuokNArq4Xy5bInp9GsZkoDuO3eVYmvmzG5EyQT5x7wDpIulBLwrOXD4509oomt7pjMIMsMa39rRUsolRCRQsQboEsLIZcvJEwDzginr75YoxIy4UIJrbNI8ig0uFPpgVWQlg== Received: from BL1PR13CA0319.namprd13.prod.outlook.com (2603:10b6:208:2c1::24) by MW3PR12MB4554.namprd12.prod.outlook.com (2603:10b6:303:55::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.19; Thu, 26 Oct 2023 09:12:08 +0000 Received: from MN1PEPF0000ECD4.namprd02.prod.outlook.com (2603:10b6:208:2c1:cafe::a3) by BL1PR13CA0319.outlook.office365.com (2603:10b6:208:2c1::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.8 via Frontend Transport; Thu, 26 Oct 2023 09:12:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECD4.mail.protection.outlook.com (10.167.242.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.19 via Frontend Transport; Thu, 26 Oct 2023 09:12:07 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 26 Oct 2023 02:11:54 -0700 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 26 Oct 2023 02:11:51 -0700 From: Bing Zhao To: , , CC: , , , , Subject: [PATCH] net/mlx5: fix the workspace double free in Windows Date: Thu, 26 Oct 2023 12:11:27 +0300 Message-ID: <20231026091127.10966-1-bingz@nvidia.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECD4:EE_|MW3PR12MB4554:EE_ X-MS-Office365-Filtering-Correlation-Id: 410288eb-7268-460b-4a70-08dbd603a13b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: aREQeHUwQkC/zksWFfRQ8xc9tUegnF0IPKl895CwwmZ7C5nxwXm2YvbSqNX9/i8G2DHz0M0Tr8dR+4OKkT0AWQFOBUz9E7wMSjHHhuGVb8QJjAkbiKJ3GK0d4FbvjGlZDYAICbIhECeZbVV4ufl0az41oat19tN5YM6N1QLpeveWRtpej3MACKwkonq/iBsOPZ2MNjv1K1jl/ZKQd0tFGKe85zfAGcJR1u7PdlLfC6g/zLQZ2XWx9Ye0QDSTOzhaNMgPORWW+Hzzxn5FCDrfhiC0rrXgsYGhbSc038o48mF9AVHtEKbzGzKfyTxAyn+rqWofEDXaMJLq6xekRu8veo6/qj0MGTpjwehYw6BfhICJIXuQYvpmaFSPujI7yNr3XlGoU3kJ8UHiRJOVNNS+AB3vCHiTu701abL9Eo3HOalS6OkMzKeFd7Wbs1nlEZDSt9ebLtKqxpioBImz8MMNsLouB31CHWzTKkmbinMpm9svDyQdI52sAMITroZwLxwNabadYpiIGB/xzRrrcqqDYpQw6TL340PAnOh467QehXoqMeH4tghUqcoVAtQAf6s6/1JBiaLqdI/G/q5xKVK/O5YkxmbVa5QCc4XmpkCQdrMEmUnQutWa4jbGcs+2FkJnuxjg/bmp3B6uZgFdZC5dQu1SjGUDwQ+8zotPKBYVyC/cm0xT2igNDrDkK56IcN3XwSiIMnL6T65xsCsEqC4khcnEIueMvhDFfgr/pHpe52Lw3drzIuhMjFaIKwo9Rq1K X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230031)(4636009)(396003)(376002)(136003)(346002)(39860400002)(230922051799003)(451199024)(64100799003)(186009)(1800799009)(82310400011)(40470700004)(36840700001)(46966006)(1076003)(2616005)(110136005)(82740400003)(40480700001)(70206006)(7636003)(70586007)(356005)(55016003)(54906003)(16526019)(316002)(6286002)(26005)(6636002)(336012)(40460700003)(478600001)(7696005)(6666004)(426003)(41300700001)(47076005)(36860700001)(2906002)(86362001)(5660300002)(36756003)(4326008)(83380400001)(8936002)(8676002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Oct 2023 09:12:07.9289 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 410288eb-7268-460b-4a70-08dbd603a13b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECD4.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR12MB4554 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The thread specific variable workspace indicated by "key_workspace" should be freed explicitly when closing a device. For example, in Linux, when exiting an application, the thread will not exit explicitly and the thread resources will not be deconstructed. The commit to solve this introduced a global list to manage the workspace resources as a garbage collector. It will also be executed in Windows, but the workspaces have already been freed in the function mlx5_flow_os_release_workspace(). With this commit, the garbage collector will only be executed in Linux. The workspace resources management in Windows will remain the same with some stub function when needed. Fixes: dc7c5e0aa905 ("net/mlx5: fix flow workspace destruction") Cc: getelson@nvidia.com Cc: dpdk@stable.org Signed-off-by: Bing Zhao Acked-by: Matan Azrad --- drivers/net/mlx5/linux/mlx5_flow_os.c | 25 +++++++++++++++++++++++++ drivers/net/mlx5/linux/mlx5_flow_os.h | 8 ++++++++ drivers/net/mlx5/mlx5.c | 1 - drivers/net/mlx5/mlx5_flow.c | 25 +------------------------ drivers/net/mlx5/mlx5_flow.h | 1 - drivers/net/mlx5/windows/mlx5_flow_os.c | 6 ++++++ drivers/net/mlx5/windows/mlx5_flow_os.h | 8 ++++++++ 7 files changed, 48 insertions(+), 26 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_flow_os.c b/drivers/net/mlx5/linux/mlx5_flow_os.c index b139bb75b9..2767b11708 100644 --- a/drivers/net/mlx5/linux/mlx5_flow_os.c +++ b/drivers/net/mlx5/linux/mlx5_flow_os.c @@ -8,6 +8,10 @@ /* Key of thread specific flow workspace data. */ static rte_thread_key key_workspace; +/* Flow workspace global list head for garbage collector. */ +static struct mlx5_flow_workspace *gc_head; +/* Spinlock for operating flow workspace list. */ +static rte_spinlock_t mlx5_flow_workspace_lock = RTE_SPINLOCK_INITIALIZER; int mlx5_flow_os_validate_item_esp(const struct rte_flow_item *item, @@ -48,6 +52,26 @@ mlx5_flow_os_validate_item_esp(const struct rte_flow_item *item, return 0; } +void +mlx5_flow_os_workspace_gc_add(struct mlx5_flow_workspace *ws) +{ + rte_spinlock_lock(&mlx5_flow_workspace_lock); + ws->gc = gc_head; + gc_head = ws; + rte_spinlock_unlock(&mlx5_flow_workspace_lock); +} + +static void +mlx5_flow_os_workspace_gc_release(void) +{ + while (gc_head) { + struct mlx5_flow_workspace *wks = gc_head; + + gc_head = wks->gc; + flow_release_workspace(wks); + } +} + int mlx5_flow_os_init_workspace_once(void) { @@ -75,4 +99,5 @@ void mlx5_flow_os_release_workspace(void) { rte_thread_key_delete(key_workspace); + mlx5_flow_os_workspace_gc_release(); } diff --git a/drivers/net/mlx5/linux/mlx5_flow_os.h b/drivers/net/mlx5/linux/mlx5_flow_os.h index ed71289322..3f7a94c9ee 100644 --- a/drivers/net/mlx5/linux/mlx5_flow_os.h +++ b/drivers/net/mlx5/linux/mlx5_flow_os.h @@ -526,4 +526,12 @@ mlx5_flow_os_validate_item_esp(const struct rte_flow_item *item, uint8_t target_protocol, struct rte_flow_error *error); +/** + * Add per thread workspace to the global list for garbage collection. + * + * @param[in] ws + * Pointer to the flow workspace. + */ +void mlx5_flow_os_workspace_gc_add(struct mlx5_flow_workspace *ws); + #endif /* RTE_PMD_MLX5_FLOW_OS_H_ */ diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 997df595d0..a673d626ac 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1851,7 +1851,6 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh) if (LIST_EMPTY(&mlx5_dev_ctx_list)) { mlx5_os_net_cleanup(); mlx5_flow_os_release_workspace(); - mlx5_flow_workspace_gc_release(); } pthread_mutex_unlock(&mlx5_dev_ctx_list_mutex); if (sh->flex_parsers_dv) { diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c index 772f6afb66..c3a611517b 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c @@ -8093,29 +8093,6 @@ flow_release_workspace(void *data) } } -static struct mlx5_flow_workspace *gc_head; -static rte_spinlock_t mlx5_flow_workspace_lock = RTE_SPINLOCK_INITIALIZER; - -static void -mlx5_flow_workspace_gc_add(struct mlx5_flow_workspace *ws) -{ - rte_spinlock_lock(&mlx5_flow_workspace_lock); - ws->gc = gc_head; - gc_head = ws; - rte_spinlock_unlock(&mlx5_flow_workspace_lock); -} - -void -mlx5_flow_workspace_gc_release(void) -{ - while (gc_head) { - struct mlx5_flow_workspace *wks = gc_head; - - gc_head = wks->gc; - flow_release_workspace(wks); - } -} - /** * Get thread specific current flow workspace. * @@ -8172,7 +8149,7 @@ mlx5_flow_push_thread_workspace(void) data = flow_alloc_thread_workspace(); if (!data) return NULL; - mlx5_flow_workspace_gc_add(data); + mlx5_flow_os_workspace_gc_add(data); } else if (!curr->inuse) { data = curr; } else if (curr->next) { diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h index 903ff66d72..4edca39936 100644 --- a/drivers/net/mlx5/mlx5_flow.h +++ b/drivers/net/mlx5/mlx5_flow.h @@ -2032,7 +2032,6 @@ struct mlx5_flow_driver_ops { struct mlx5_flow_workspace *mlx5_flow_push_thread_workspace(void); void mlx5_flow_pop_thread_workspace(void); struct mlx5_flow_workspace *mlx5_flow_get_thread_workspace(void); -void mlx5_flow_workspace_gc_release(void); __extension__ struct flow_grp_info { diff --git a/drivers/net/mlx5/windows/mlx5_flow_os.c b/drivers/net/mlx5/windows/mlx5_flow_os.c index 5013e9f012..f907b21ecc 100644 --- a/drivers/net/mlx5/windows/mlx5_flow_os.c +++ b/drivers/net/mlx5/windows/mlx5_flow_os.c @@ -417,6 +417,12 @@ mlx5_flow_os_set_specific_workspace(struct mlx5_flow_workspace *data) return err; } +void +mlx5_flow_os_workspace_gc_add(struct mlx5_flow_workspace *ws) +{ + RTE_SET_USED(ws); +} + int mlx5_flow_os_validate_item_esp(const struct rte_flow_item *item, uint64_t item_flags, diff --git a/drivers/net/mlx5/windows/mlx5_flow_os.h b/drivers/net/mlx5/windows/mlx5_flow_os.h index 1c1c17fc41..856d8ba948 100644 --- a/drivers/net/mlx5/windows/mlx5_flow_os.h +++ b/drivers/net/mlx5/windows/mlx5_flow_os.h @@ -473,4 +473,12 @@ mlx5_flow_os_validate_item_esp(const struct rte_flow_item *item, uint8_t target_protocol, struct rte_flow_error *error); +/** + * Add per thread workspace to the global list for garbage collection. + * + * @param[in] ws + * Pointer to the flow workspace. + */ +void mlx5_flow_os_workspace_gc_add(struct mlx5_flow_workspace *ws); + #endif /* RTE_PMD_MLX5_FLOW_OS_H_ */