From patchwork Tue Nov 6 06:07:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 47874 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 36CFB493D; Tue, 6 Nov 2018 07:03:44 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id E05AB2082 for ; Tue, 6 Nov 2018 07:03:41 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Nov 2018 22:03:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,470,1534834800"; d="scan'208";a="277430820" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by fmsmga005.fm.intel.com with ESMTP; 05 Nov 2018 22:03:39 -0800 From: Jeff Guo To: konstantin.ananyev@intel.com, anatoly.burakov@intel.com, thomas@monjalon.net, bernard.iremonger@intel.com, jingjing.wu@intel.com, wenzhuo.lu@intel.com Cc: ferruh.yigit@intel.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com, matan@mellanox.com, shaopeng.he@intel.com Date: Tue, 6 Nov 2018 14:07:13 +0800 Message-Id: <1541484436-91320-2-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1541484436-91320-1-git-send-email-jia.guo@intel.com> References: <1541484436-91320-1-git-send-email-jia.guo@intel.com> Subject: [dpdk-dev] [PATCH 1/3] eal: fix lock issue for hot-unplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch will add missing unlock for hot-unplug handler, without this patch potential dead lock will occur when device be hotplug-in after device be hot-unplugged. Fixes: 0fc54536b14a ("eal: add failure handling for hot-unplug") Signed-off-by: Jeff Guo --- lib/librte_eal/linuxapp/eal/eal_dev.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/lib/librte_eal/linuxapp/eal/eal_dev.c b/lib/librte_eal/linuxapp/eal/eal_dev.c index d589c69..2830c86 100644 --- a/lib/librte_eal/linuxapp/eal/eal_dev.c +++ b/lib/librte_eal/linuxapp/eal/eal_dev.c @@ -258,7 +258,7 @@ dev_uev_handler(__rte_unused void *param) if (bus == NULL) { RTE_LOG(ERR, EAL, "Cannot find bus (%s)\n", busname); - return; + goto failure_handle_err; } dev = bus->find_device(NULL, cmp_dev_name, @@ -266,19 +266,23 @@ dev_uev_handler(__rte_unused void *param) if (dev == NULL) { RTE_LOG(ERR, EAL, "Cannot find device (%s) on " "bus (%s)\n", uevent.devname, busname); - return; + goto failure_handle_err; } ret = bus->hot_unplug_handler(dev); - rte_spinlock_unlock(&failure_handle_lock); if (ret) { RTE_LOG(ERR, EAL, "Can not handle hot-unplug " "for device (%s)\n", dev->name); - return; } + rte_spinlock_unlock(&failure_handle_lock); } rte_dev_event_callback_process(uevent.devname, uevent.type); } + + return; + +failure_handle_err: + rte_spinlock_unlock(&failure_handle_lock); } int __rte_experimental From patchwork Tue Nov 6 06:07:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 47875 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id CF11D4C92; Tue, 6 Nov 2018 07:03:45 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 1C74D31FC for ; Tue, 6 Nov 2018 07:03:43 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Nov 2018 22:03:43 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,470,1534834800"; d="scan'208";a="277430830" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by fmsmga005.fm.intel.com with ESMTP; 05 Nov 2018 22:03:41 -0800 From: Jeff Guo To: konstantin.ananyev@intel.com, anatoly.burakov@intel.com, thomas@monjalon.net, bernard.iremonger@intel.com, jingjing.wu@intel.com, wenzhuo.lu@intel.com Cc: ferruh.yigit@intel.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com, matan@mellanox.com, shaopeng.he@intel.com Date: Tue, 6 Nov 2018 14:07:14 +0800 Message-Id: <1541484436-91320-3-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1541484436-91320-1-git-send-email-jia.guo@intel.com> References: <1541484436-91320-1-git-send-email-jia.guo@intel.com> Subject: [dpdk-dev] [PATCH 2/3] vfio: fix to add handler lock for hot-unplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch add hot-unplug handler lock and unlock in device request handler when process bus and device resource, in order to avoid the synchronization issue when device be hot-unplugged. Fixes: c115fd000c32 ("vfio: handle hotplug request notifier") Signed-off-by: Jeff Guo --- drivers/bus/pci/linux/pci_vfio.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c index 305cc06..d2c8410 100644 --- a/drivers/bus/pci/linux/pci_vfio.c +++ b/drivers/bus/pci/linux/pci_vfio.c @@ -19,6 +19,7 @@ #include #include #include +#include #include "eal_filesystem.h" @@ -35,6 +36,14 @@ * This file is only compiled if CONFIG_RTE_EAL_VFIO is set to "y". */ +/* + * spinlock for device hot-unplug failure handling. If it try to access bus or + * device, such as handle sigbus on bus or handle memory failure for device + * just need to use this lock. It could protect the bus and the device to avoid + * race condition. + */ +static rte_spinlock_t failure_handle_lock = RTE_SPINLOCK_INITIALIZER; + #ifdef VFIO_PRESENT #ifndef PAGE_SIZE @@ -289,11 +298,12 @@ pci_vfio_req_handler(void *param) int ret; struct rte_device *device = (struct rte_device *)param; + rte_spinlock_lock(&failure_handle_lock); bus = rte_bus_find_by_device(device); if (bus == NULL) { RTE_LOG(ERR, EAL, "Cannot find bus for device (%s)\n", device->name); - return; + goto handle_end; } /* @@ -306,6 +316,8 @@ pci_vfio_req_handler(void *param) RTE_LOG(ERR, EAL, "Can not handle hot-unplug for device (%s)\n", device->name); +handle_end: + rte_spinlock_unlock(&failure_handle_lock); } /* enable notifier (only enable req now) */ From patchwork Tue Nov 6 06:07:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 47876 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 9A73E4CE4; Tue, 6 Nov 2018 07:03:47 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 367134CA5 for ; Tue, 6 Nov 2018 07:03:46 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Nov 2018 22:03:45 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,470,1534834800"; d="scan'208";a="277430835" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by fmsmga005.fm.intel.com with ESMTP; 05 Nov 2018 22:03:43 -0800 From: Jeff Guo To: konstantin.ananyev@intel.com, anatoly.burakov@intel.com, thomas@monjalon.net, bernard.iremonger@intel.com, jingjing.wu@intel.com, wenzhuo.lu@intel.com Cc: ferruh.yigit@intel.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com, matan@mellanox.com, shaopeng.he@intel.com Date: Tue, 6 Nov 2018 14:07:15 +0800 Message-Id: <1541484436-91320-4-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1541484436-91320-1-git-send-email-jia.guo@intel.com> References: <1541484436-91320-1-git-send-email-jia.guo@intel.com> Subject: [dpdk-dev] [PATCH 3/3] app/testpmd: fix callback issue for hot-unplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Before detach device when device be hot-unplugged, the failure process in user space and kernel space both need to be finished, such as eal interrupt callback need to be inactive before the callback be unregistered when device is being cleaned. This patch add rte alarm for device detaching, with that it could finish interrupt callback soon and give time to let the failure process done before detaching. Fixes: 2049c5113fe8 ("app/testpmd: use hotplug failure handler") Signed-off-by: Jeff Guo --- app/test-pmd/testpmd.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 9c0edca..9c673cf 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -2620,7 +2620,18 @@ eth_dev_event_callback(const char *device_name, enum rte_dev_event_type type, device_name); return; } - rmv_event_callback((void *)(intptr_t)port_id); + /* + * Before detach device, the hot-unplug failure process in + * user space and kernel space both need to be finished, + * such as eal interrupt callback need to be inactive before + * the callback be unregistered when device is being cleaned. + * So finished interrupt callback soon here and give time to + * let the work done before detaching. + */ + if (rte_eal_alarm_set(100000, + rmv_event_callback, (void *)(intptr_t)port_id)) + RTE_LOG(ERR, EAL, + "Could not set up deferred device removal\n"); break; case RTE_DEV_EVENT_ADD: RTE_LOG(ERR, EAL, "The device: %s has been added!\n",