From patchwork Mon May 30 13:47:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhichao Zeng X-Patchwork-Id: 112027 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 35A98A00C2; Mon, 30 May 2022 07:48:09 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D843B40A89; Mon, 30 May 2022 07:48:08 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id BECEF40A82; Mon, 30 May 2022 07:48:06 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653889687; x=1685425687; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ETGY/5vCmvZXdVF/rOcDSCz9X7pqdn4ybHH0siHy6yM=; b=QhcTdFex2B9rYd5XxaSa2Ql31hgd7gvn5nIEOWTBRDx4MgLc+fE4MUDO t2I4VJ1WuPH91awaC1FTUz9lMn2j3Mgdi2ts4+wyYDmrruiBAm1bDIVzn dwgE7CxIJbK5TM8R4WvcDxIlccLll2ZLRo2WQzfEKIHUyhClUk5WE6C8p 48CT0rNPX/dpUY4YBuU+OdCw8h02yv8VQKxq5Fsq3Cm6FfvSFdLoTx5Lc EC8UC9Nt6Xa+v6buQAgE782rJ8ryT1MIGYEjsCjNq6WYQmDE7P7rFBOb5 HKroDtlhGzMxAhkKs7hsuN2Ye9sXHK4T4zVGkP7D7r+W8Scz9lOid0VcF g==; X-IronPort-AV: E=McAfee;i="6400,9594,10362"; a="274991404" X-IronPort-AV: E=Sophos;i="5.91,262,1647327600"; d="scan'208";a="274991404" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 May 2022 22:48:05 -0700 X-IronPort-AV: E=Sophos;i="5.91,262,1647327600"; d="scan'208";a="575778796" Received: from unknown (HELO localhost.localdomain) ([10.239.251.103]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 May 2022 22:48:01 -0700 From: zhichaox.zeng@intel.com To: dev@dpdk.org, qiming.yang@intel.com, bruce.richardson@intel.com, hkalra@marvell.com, david.marchand@redhat.com Cc: aconole@redhat.com, olivier.matz@6wind.com, thomas@monjalon.net, stable@dpdk.org, Zhichao Zeng Subject: [PATCH v3] lib/eal: fix segfaults due to thread exit order Date: Mon, 30 May 2022 13:47:38 +0000 Message-Id: <20220530134738.488602-1-zhichaox.zeng@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220523111642.10406-1-zhichaox.zeng@intel.com> References: <20220523111642.10406-1-zhichaox.zeng@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Zhichao Zeng The eal-intr-thread is not closed before memory cleanup in the process of exiting. There is a small probability that when the eal-intr-thread is about to use some pointers, the memory were just cleaned, which cause the segment fault error caught by ASan. This patch close the eal-intr-thread before memory cleanup when exiting to avoid segment fault. --- v2: add the same API for FreeBSD --- v3: fix rte_eal_cleanup crash in debug_autotest Signed-off-by: Zhichao Zeng --- lib/eal/common/eal_private.h | 7 +++++++ lib/eal/freebsd/eal.c | 22 +++++++++++++++++++++- lib/eal/freebsd/eal_interrupts.c | 12 ++++++++++++ lib/eal/linux/eal.c | 21 ++++++++++++++++++++- lib/eal/linux/eal_interrupts.c | 12 ++++++++++++ 5 files changed, 72 insertions(+), 2 deletions(-) diff --git a/lib/eal/common/eal_private.h b/lib/eal/common/eal_private.h index 44d14241f0..7adf41b7d7 100644 --- a/lib/eal/common/eal_private.h +++ b/lib/eal/common/eal_private.h @@ -152,6 +152,13 @@ int rte_eal_tailqs_init(void); */ int rte_eal_intr_init(void); +/** + * Destroy interrupt handling thread. + * + * This function is private to EAL. + */ +void rte_eal_intr_destroy(void); + /** * Close the default log stream * diff --git a/lib/eal/freebsd/eal.c b/lib/eal/freebsd/eal.c index a6b20960f2..6875a9f6d2 100644 --- a/lib/eal/freebsd/eal.c +++ b/lib/eal/freebsd/eal.c @@ -72,6 +72,8 @@ struct lcore_config lcore_config[RTE_MAX_LCORE]; /* used by rte_rdtsc() */ int rte_cycles_vmware_tsc_map; +/* used to judge the running status of the eal */ +static uint32_t run_once; int eal_clean_runtime_dir(void) @@ -574,12 +576,23 @@ static void rte_eal_init_alert(const char *msg) RTE_LOG(ERR, EAL, "%s\n", msg); } +static void warn_parent(void) +{ + RTE_LOG(WARNING, EAL, "fork() was called, DPDK won't work in the child " + "process unless it calls rte_eal_init()\n"); +} + +static void scratch_child(void) +{ + /* Scratch run_once so that a call to rte_eal_cleanup won't crash... */ + __atomic_store_n(&run_once, 0, __ATOMIC_RELAXED); +} + /* Launch threads, called at application init(). */ int rte_eal_init(int argc, char **argv) { int i, fctret, ret; - static uint32_t run_once; uint32_t has_run = 0; char cpuset[RTE_CPU_AFFINITY_STR_LEN]; char thread_name[RTE_MAX_THREAD_NAME_LEN]; @@ -883,6 +896,8 @@ rte_eal_init(int argc, char **argv) eal_mcfg_complete(); + pthread_atfork(NULL, warn_parent, scratch_child); + return fctret; } @@ -891,8 +906,13 @@ rte_eal_cleanup(void) { struct internal_config *internal_conf = eal_get_internal_configuration(); + + if (__atomic_load_n(&run_once, __ATOMIC_RELAXED) == 0) + return 0; + rte_service_finalize(); rte_mp_channel_cleanup(); + rte_eal_intr_destroy(); /* after this point, any DPDK pointers will become dangling */ rte_eal_memory_detach(); rte_eal_alarm_cleanup(); diff --git a/lib/eal/freebsd/eal_interrupts.c b/lib/eal/freebsd/eal_interrupts.c index 9f720bdc8f..cac3859b06 100644 --- a/lib/eal/freebsd/eal_interrupts.c +++ b/lib/eal/freebsd/eal_interrupts.c @@ -648,6 +648,18 @@ rte_eal_intr_init(void) return ret; } +void +rte_eal_intr_destroy(void) +{ + /* cancel the host thread to wait/handle the interrupt */ + pthread_cancel(intr_thread); + pthread_join(intr_thread, NULL); + + /* close kqueue */ + close(kq); + kq = -1; +} + int rte_intr_rx_ctl(struct rte_intr_handle *intr_handle, int epfd, int op, unsigned int vec, void *data) diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c index 1ef263434a..32c7adaa52 100644 --- a/lib/eal/linux/eal.c +++ b/lib/eal/linux/eal.c @@ -76,6 +76,8 @@ struct lcore_config lcore_config[RTE_MAX_LCORE]; /* used by rte_rdtsc() */ int rte_cycles_vmware_tsc_map; +/* used to judge the running status of the eal */ +static uint32_t run_once; int eal_clean_runtime_dir(void) @@ -857,12 +859,23 @@ is_iommu_enabled(void) return n > 2; } +static void warn_parent(void) +{ + RTE_LOG(WARNING, EAL, "fork() was called, DPDK won't work in the child " + "process unless it calls rte_eal_init()\n"); +} + +static void scratch_child(void) +{ + /* Scratch run_once so that a call to rte_eal_cleanup won't crash... */ + __atomic_store_n(&run_once, 0, __ATOMIC_RELAXED); +} + /* Launch threads, called at application init(). */ int rte_eal_init(int argc, char **argv) { int i, fctret, ret; - static uint32_t run_once; uint32_t has_run = 0; const char *p; static char logid[PATH_MAX]; @@ -1228,6 +1241,8 @@ rte_eal_init(int argc, char **argv) eal_mcfg_complete(); + pthread_atfork(NULL, warn_parent, scratch_child); + return fctret; } @@ -1257,6 +1272,9 @@ rte_eal_cleanup(void) struct internal_config *internal_conf = eal_get_internal_configuration(); + if (__atomic_load_n(&run_once, __ATOMIC_RELAXED) == 0) + return 0; + if (rte_eal_process_type() == RTE_PROC_PRIMARY && internal_conf->hugepage_file.unlink_existing) rte_memseg_walk(mark_freeable, NULL); @@ -1266,6 +1284,7 @@ rte_eal_cleanup(void) vfio_mp_sync_cleanup(); #endif rte_mp_channel_cleanup(); + rte_eal_intr_destroy(); /* after this point, any DPDK pointers will become dangling */ rte_eal_memory_detach(); eal_mp_dev_hotplug_cleanup(); diff --git a/lib/eal/linux/eal_interrupts.c b/lib/eal/linux/eal_interrupts.c index d52ec8eb4c..7e9853e8e7 100644 --- a/lib/eal/linux/eal_interrupts.c +++ b/lib/eal/linux/eal_interrupts.c @@ -1199,6 +1199,18 @@ rte_eal_intr_init(void) return ret; } +void +rte_eal_intr_destroy(void) +{ + /* cancel the host thread to wait/handle the interrupt */ + pthread_cancel(intr_thread); + pthread_join(intr_thread, NULL); + + /* close the pipe used by epoll */ + close(intr_pipe.writefd); + close(intr_pipe.readfd); +} + static void eal_intr_proc_rxtx_intr(int fd, const struct rte_intr_handle *intr_handle) {