From patchwork Mon Mar 13 23:31:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tyler Retzlaff X-Patchwork-Id: 125082 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 429B941E88; Tue, 14 Mar 2023 00:31:16 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EAEF541611; Tue, 14 Mar 2023 00:31:12 +0100 (CET) Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mails.dpdk.org (Postfix) with ESMTP id D4135410D1 for ; Tue, 14 Mar 2023 00:31:08 +0100 (CET) Received: by linux.microsoft.com (Postfix, from userid 1086) id 2F6CD2056866; Mon, 13 Mar 2023 16:31:08 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 2F6CD2056866 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1678750268; bh=QdVmYb/o7Aq8A11P9wbdx3I8kxxqizdMblRmgtj6ZXc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kZ4rkXpeWAQlbsSoVnmvNS89D5fVQQtNINv4eO5PBgpnnxg8xMF8sMjgn/Lt0iPYg FtdxMjJgnWBbix8E/hYp03dSkDLizh4cZXUIp6bVCktUddsRxFr2uv1iKtpwHEy0kW EE6ROVZSH2FUqz+8QmZnz36SoYw1oxbcHkUCxxBQ= From: Tyler Retzlaff To: dev@dpdk.org Cc: thomas@monjalon.net, david.marchand@redhat.com, Tyler Retzlaff Subject: [PATCH v2 1/2] eal: make cpusetp to rte thread set affinity const Date: Mon, 13 Mar 2023 16:31:06 -0700 Message-Id: <1678750267-3829-2-git-send-email-roretzla@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1678750267-3829-1-git-send-email-roretzla@linux.microsoft.com> References: <1677782682-27200-1-git-send-email-roretzla@linux.microsoft.com> <1678750267-3829-1-git-send-email-roretzla@linux.microsoft.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org const qualify rte_cpuset_t *cpusetp parameter in the rte_thread_set_affinity API. Signed-off-by: Tyler Retzlaff --- lib/eal/common/eal_common_thread.c | 6 +++--- lib/eal/include/rte_thread.h | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/eal/common/eal_common_thread.c b/lib/eal/common/eal_common_thread.c index 079a385..c9e5619 100644 --- a/lib/eal/common/eal_common_thread.c +++ b/lib/eal/common/eal_common_thread.c @@ -34,7 +34,7 @@ unsigned rte_socket_id(void) } static int -eal_cpuset_socket_id(rte_cpuset_t *cpusetp) +eal_cpuset_socket_id(const rte_cpuset_t *cpusetp) { unsigned cpu = 0; int socket_id = SOCKET_ID_ANY; @@ -62,7 +62,7 @@ unsigned rte_socket_id(void) } static void -thread_update_affinity(rte_cpuset_t *cpusetp) +thread_update_affinity(const rte_cpuset_t *cpusetp) { unsigned int lcore_id = rte_lcore_id(); @@ -83,7 +83,7 @@ unsigned rte_socket_id(void) } int -rte_thread_set_affinity(rte_cpuset_t *cpusetp) +rte_thread_set_affinity(const rte_cpuset_t *cpusetp) { if (rte_thread_set_affinity_by_id(rte_thread_self(), cpusetp) != 0) { RTE_LOG(ERR, EAL, "rte_thread_set_affinity_by_id failed\n"); diff --git a/lib/eal/include/rte_thread.h b/lib/eal/include/rte_thread.h index ddd411e..e461354 100644 --- a/lib/eal/include/rte_thread.h +++ b/lib/eal/include/rte_thread.h @@ -348,7 +348,7 @@ int rte_thread_get_affinity_by_id(rte_thread_t thread_id, * @return * On success, return 0; otherwise return -1; */ -int rte_thread_set_affinity(rte_cpuset_t *cpusetp); +int rte_thread_set_affinity(const rte_cpuset_t *cpusetp); /** * Get core affinity of the current thread. From patchwork Mon Mar 13 23:31:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tyler Retzlaff X-Patchwork-Id: 125083 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3193F41E88; Tue, 14 Mar 2023 00:31:23 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0FF9D42B71; Tue, 14 Mar 2023 00:31:14 +0100 (CET) Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mails.dpdk.org (Postfix) with ESMTP id E448E410DE; Tue, 14 Mar 2023 00:31:08 +0100 (CET) Received: by linux.microsoft.com (Postfix, from userid 1086) id 3B856205686B; Mon, 13 Mar 2023 16:31:08 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 3B856205686B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1678750268; bh=jC8Xhk7BCJbwB3cHMprOhwPQWF2udfbRC4kdd4p0Ftg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oGTK9qBuBy5+szogcYtcao503q4C9Aq6Ap+OTDria1j1qX73UpSNK5hdRd2U4nOJP jsV8D4EaWD1W5isPUEy6VDeRbcPV5IM084AZi2dgNO0LP0luL3tAS0nuWh2yc0+MWB U7VXth/HPuKEfhopZwwiyA3/YhGDTT7aX/OIlXkI= From: Tyler Retzlaff To: dev@dpdk.org Cc: thomas@monjalon.net, david.marchand@redhat.com, Tyler Retzlaff , stable@dpdk.org Subject: [PATCH v2 2/2] eal: fix failure path race setting new thread affinity Date: Mon, 13 Mar 2023 16:31:07 -0700 Message-Id: <1678750267-3829-3-git-send-email-roretzla@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1678750267-3829-1-git-send-email-roretzla@linux.microsoft.com> References: <1677782682-27200-1-git-send-email-roretzla@linux.microsoft.com> <1678750267-3829-1-git-send-email-roretzla@linux.microsoft.com> X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In rte_thread_create setting affinity after pthread_create may fail. Such a failure should result in the entire rte_thread_create failing but doesn't. Additionally if there is a failure to set affinity a race exists where the creating thread will free ctx and depending on scheduling of the new thread it may also free ctx (double free). Resolve both of the above issues by setting the affinity from the newly created thread instead of after thread creation. To achieve this modify the existing thread wrapper to allow the creating thread to wait on the result of the set affinity operation. Fixes: ce6e911d20f6 ("eal: add thread lifetime API") Cc: stable@dpdk.org Signed-off-by: Tyler Retzlaff --- lib/eal/unix/rte_thread.c | 52 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 42 insertions(+), 10 deletions(-) diff --git a/lib/eal/unix/rte_thread.c b/lib/eal/unix/rte_thread.c index 37ebfcf..a09fb08 100644 --- a/lib/eal/unix/rte_thread.c +++ b/lib/eal/unix/rte_thread.c @@ -8,6 +8,7 @@ #include #include +#include #include #include #include @@ -16,8 +17,17 @@ struct eal_tls_key { pthread_key_t thread_index; }; +enum __rte_thread_wrapper_status { + THREAD_WRAPPER_LAUNCHING, /* Yet to call thread_func function */ + THREAD_WRAPPER_RUNNING, /* Thread is running successfully */ + THREAD_WRAPPER_ERROR /* Thread thread_wrapper encountered an error */ +}; + struct thread_routine_ctx { rte_thread_func thread_func; + const rte_thread_attr_t *thread_attr; + int thread_wrapper_ret; + enum __rte_thread_wrapper_status thread_wrapper_status; void *routine_args; }; @@ -83,11 +93,24 @@ struct thread_routine_ctx { static void * thread_func_wrapper(void *arg) { - struct thread_routine_ctx ctx = *(struct thread_routine_ctx *)arg; + struct thread_routine_ctx *ctx = (struct thread_routine_ctx *)arg; + rte_thread_func thread_func = ctx->thread_func; + void *thread_args = ctx->routine_args; + + if (ctx->thread_attr != NULL && CPU_COUNT(&ctx->thread_attr->cpuset) > 0) { + ctx->thread_wrapper_ret = rte_thread_set_affinity(&ctx->thread_attr->cpuset); + if (ctx->thread_wrapper_ret != 0) { + RTE_LOG(DEBUG, EAL, "rte_thread_set_affinity failed\n"); + __atomic_store_n(&ctx->thread_wrapper_status, + THREAD_WRAPPER_ERROR, __ATOMIC_RELEASE); + } + } + __atomic_store_n(&ctx->thread_wrapper_status, + THREAD_WRAPPER_RUNNING, __ATOMIC_RELEASE); free(arg); - return (void *)(uintptr_t)ctx.thread_func(ctx.routine_args); + return (void *)(uintptr_t)thread_func(thread_args); } int @@ -98,6 +121,7 @@ struct thread_routine_ctx { int ret = 0; pthread_attr_t attr; pthread_attr_t *attrp = NULL; + enum __rte_thread_wrapper_status thread_wrapper_status; struct thread_routine_ctx *ctx; struct sched_param param = { .sched_priority = 0, @@ -111,7 +135,10 @@ struct thread_routine_ctx { goto cleanup; } ctx->routine_args = args; + ctx->thread_attr = thread_attr; ctx->thread_func = thread_func; + ctx->thread_wrapper_ret = 0; + ctx->thread_wrapper_status = THREAD_WRAPPER_LAUNCHING; if (thread_attr != NULL) { ret = pthread_attr_init(&attr); @@ -164,16 +191,21 @@ struct thread_routine_ctx { goto cleanup; } - if (thread_attr != NULL && CPU_COUNT(&thread_attr->cpuset) > 0) { - ret = rte_thread_set_affinity_by_id(*thread_id, - &thread_attr->cpuset); - if (ret != 0) { - RTE_LOG(DEBUG, EAL, "rte_thread_set_affinity_by_id failed\n"); - goto cleanup; - } + /* Wait for the thread wrapper to initialize thread successfully */ + while ((thread_wrapper_status = + __atomic_load_n(&ctx->thread_wrapper_status, + __ATOMIC_ACQUIRE)) == THREAD_WRAPPER_LAUNCHING) + sched_yield(); + + /* Check if the control thread encountered an error */ + if (thread_wrapper_status == THREAD_WRAPPER_ERROR) { + /* thread wrapper is exiting */ + pthread_join((pthread_t)thread_id->opaque_id, NULL); + ret = ctx->thread_wrapper_ret; + free(ctx); } - ctx = NULL; + cleanup: free(ctx); if (attrp != NULL)