From patchwork Sun Sep 30 10:24:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 45703 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 643AF5B40; Sun, 30 Sep 2018 12:21:20 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 760A15911 for ; Sun, 30 Sep 2018 12:21:13 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Sep 2018 03:21:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,323,1534834800"; d="scan'208";a="84583567" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by FMSMGA003.fm.intel.com with ESMTP; 30 Sep 2018 03:20:25 -0700 From: Jeff Guo To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com, bernard.iremonger@intel.com, arybchenko@solarflare.com, wenzhuo.lu@intel.com, anatoly.burakov@intel.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Sun, 30 Sep 2018 18:24:07 +0800 Message-Id: <1538303053-54728-2-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1538303053-54728-1-git-send-email-jia.guo@intel.com> References: <1498711073-42917-1-git-send-email-jia.guo@intel.com> <1538303053-54728-1-git-send-email-jia.guo@intel.com> Subject: [dpdk-dev] [PATCH v11 1/7] bus: add hot-unplug handler X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" A hot-unplug failure and app crash can be caused, when a device is hot-unplugged but the application still try to access the device by reading or writing from the BARs, which is already invalid but still not timely be unmap or released. This patch introduces bus ops to handle hot-unplug failures. Each bus can implement its own case-dependent logic to handle the failures. Signed-off-by: Jeff Guo Acked-by: Shaopeng He --- v11->v10: change the ops name, since both uio and vfio will use the hot-unplug ops. --- lib/librte_eal/common/include/rte_bus.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h index b7b5b08..1bb53dc 100644 --- a/lib/librte_eal/common/include/rte_bus.h +++ b/lib/librte_eal/common/include/rte_bus.h @@ -168,6 +168,20 @@ typedef int (*rte_bus_unplug_t)(struct rte_device *dev); typedef int (*rte_bus_parse_t)(const char *name, void *addr); /** + * Implement a specific hot-unplug handler, which is responsible for + * handle the failure when device be hot-unplugged. When the event of + * hot-unplug be detected, it could call this function to handle + * the hot-unplug failure and avoid app crash. + * @param dev + * Pointer of the device structure. + * + * @return + * 0 on success. + * !0 on error. + */ +typedef int (*rte_bus_hot_unplug_handler_t)(struct rte_device *dev); + +/** * Bus scan policies */ enum rte_bus_scan_mode { @@ -212,6 +226,8 @@ struct rte_bus { struct rte_bus_conf conf; /**< Bus configuration */ rte_bus_get_iommu_class_t get_iommu_class; /**< Get iommu class */ rte_dev_iterate_t dev_iterate; /**< Device iterator. */ + rte_bus_hot_unplug_handler_t hot_unplug_handler; + /**< handle hot-unplug failure on the bus */ }; /** From patchwork Sun Sep 30 10:24:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 45701 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id A9EDF5A6E; Sun, 30 Sep 2018 12:21:15 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id DA287569B for ; Sun, 30 Sep 2018 12:21:12 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Sep 2018 03:21:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,323,1534834800"; d="scan'208";a="84583569" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by FMSMGA003.fm.intel.com with ESMTP; 30 Sep 2018 03:20:28 -0700 From: Jeff Guo To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com, bernard.iremonger@intel.com, arybchenko@solarflare.com, wenzhuo.lu@intel.com, anatoly.burakov@intel.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Sun, 30 Sep 2018 18:24:08 +0800 Message-Id: <1538303053-54728-3-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1538303053-54728-1-git-send-email-jia.guo@intel.com> References: <1498711073-42917-1-git-send-email-jia.guo@intel.com> <1538303053-54728-1-git-send-email-jia.guo@intel.com> Subject: [dpdk-dev] [PATCH v11 2/7] bus/pci: implement hot-unplug handler ops X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch implements the ops to handle hot-unplug on the PCI bus. For UIO PCI, it could avoids BARs read/write errors by creating a new dummy memory to remap the memory where the failure is. For VFIO or other kernel driver, it could specific implement function to handle hot-unplug case by case. Signed-off-by: Jeff Guo Acked-by: Shaopeng He --- v11->v10: change the ops name --- drivers/bus/pci/pci_common.c | 28 ++++++++++++++++++++++++++++ drivers/bus/pci/pci_common_uio.c | 33 +++++++++++++++++++++++++++++++++ drivers/bus/pci/private.h | 12 ++++++++++++ 3 files changed, 73 insertions(+) diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c index 7736b3f..d286234 100644 --- a/drivers/bus/pci/pci_common.c +++ b/drivers/bus/pci/pci_common.c @@ -406,6 +406,33 @@ pci_find_device(const struct rte_device *start, rte_dev_cmp_t cmp, } static int +pci_hot_unplug_handler(struct rte_device *dev) +{ + struct rte_pci_device *pdev = NULL; + int ret = 0; + + pdev = RTE_DEV_TO_PCI(dev); + if (!pdev) + return -1; + + switch (pdev->kdrv) { + case RTE_KDRV_IGB_UIO: + case RTE_KDRV_UIO_GENERIC: + case RTE_KDRV_NIC_UIO: + /* BARs resource is invalid, remap it to be safe. */ + ret = pci_uio_remap_resource(pdev); + break; + default: + RTE_LOG(DEBUG, EAL, + "Not managed by a supported kernel driver, skipped\n"); + ret = -1; + break; + } + + return ret; +} + +static int pci_plug(struct rte_device *dev) { return pci_probe_all_drivers(RTE_DEV_TO_PCI(dev)); @@ -435,6 +462,7 @@ struct rte_pci_bus rte_pci_bus = { .unplug = pci_unplug, .parse = pci_parse, .get_iommu_class = rte_pci_get_iommu_class, + .hot_unplug_handler = pci_hot_unplug_handler, }, .device_list = TAILQ_HEAD_INITIALIZER(rte_pci_bus.device_list), .driver_list = TAILQ_HEAD_INITIALIZER(rte_pci_bus.driver_list), diff --git a/drivers/bus/pci/pci_common_uio.c b/drivers/bus/pci/pci_common_uio.c index 54bc20b..7ea73db 100644 --- a/drivers/bus/pci/pci_common_uio.c +++ b/drivers/bus/pci/pci_common_uio.c @@ -146,6 +146,39 @@ pci_uio_unmap(struct mapped_pci_resource *uio_res) } } +/* remap the PCI resource of a PCI device in anonymous virtual memory */ +int +pci_uio_remap_resource(struct rte_pci_device *dev) +{ + int i; + void *map_address; + + if (dev == NULL) + return -1; + + /* Remap all BARs */ + for (i = 0; i != PCI_MAX_RESOURCE; i++) { + /* skip empty BAR */ + if (dev->mem_resource[i].phys_addr == 0) + continue; + map_address = mmap(dev->mem_resource[i].addr, + (size_t)dev->mem_resource[i].len, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0); + if (map_address == MAP_FAILED) { + RTE_LOG(ERR, EAL, + "Cannot remap resource for device %s\n", + dev->name); + return -1; + } + RTE_LOG(INFO, EAL, + "Successful remap resource for device %s\n", + dev->name); + } + + return 0; +} + static struct mapped_pci_resource * pci_uio_find_resource(struct rte_pci_device *dev) { diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h index 8ddd03e..6b312e5 100644 --- a/drivers/bus/pci/private.h +++ b/drivers/bus/pci/private.h @@ -123,6 +123,18 @@ void pci_uio_free_resource(struct rte_pci_device *dev, struct mapped_pci_resource *uio_res); /** + * Remap the PCI resource of a PCI device in anonymous virtual memory. + * + * @param dev + * Point to the struct rte pci device. + * @return + * - On success, zero. + * - On failure, a negative value. + */ +int +pci_uio_remap_resource(struct rte_pci_device *dev); + +/** * Map device memory to uio resource * * This function is private to EAL. From patchwork Sun Sep 30 10:24:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 45702 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 931155B38; Sun, 30 Sep 2018 12:21:18 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 0C42658FA for ; Sun, 30 Sep 2018 12:21:12 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Sep 2018 03:21:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,323,1534834800"; d="scan'208";a="84583573" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by FMSMGA003.fm.intel.com with ESMTP; 30 Sep 2018 03:20:31 -0700 From: Jeff Guo To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com, bernard.iremonger@intel.com, arybchenko@solarflare.com, wenzhuo.lu@intel.com, anatoly.burakov@intel.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Sun, 30 Sep 2018 18:24:09 +0800 Message-Id: <1538303053-54728-4-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1538303053-54728-1-git-send-email-jia.guo@intel.com> References: <1498711073-42917-1-git-send-email-jia.guo@intel.com> <1538303053-54728-1-git-send-email-jia.guo@intel.com> Subject: [dpdk-dev] [PATCH v11 3/7] bus: add sigbus handler X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When a device is hot-unplugged, a sigbus error will occur of the datapath can still read/write to the device. A handler is required here to capture the sigbus signal and handle it appropriately. This patch introduces a bus ops to handle sigbus errors. Each bus can implement its own case-dependent logic to handle the sigbus errors. Signed-off-by: Jeff Guo Acked-by: Shaopeng He --- v11->v10: change some commit log --- lib/librte_eal/common/include/rte_bus.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h index 1bb53dc..201454a 100644 --- a/lib/librte_eal/common/include/rte_bus.h +++ b/lib/librte_eal/common/include/rte_bus.h @@ -182,6 +182,21 @@ typedef int (*rte_bus_parse_t)(const char *name, void *addr); typedef int (*rte_bus_hot_unplug_handler_t)(struct rte_device *dev); /** + * Implement a specific sigbus handler, which is responsible for handling + * the sigbus error which is either original memory error, or specific memory + * error that caused of device be hot-unplugged. When sigbus error be captured, + * it could call this function to handle sigbus error. + * @param failure_addr + * Pointer of the fault address of the sigbus error. + * + * @return + * 0 for success handle the sigbus. + * 1 for no bus handle the sigbus. + * -1 for failed to handle the sigbus + */ +typedef int (*rte_bus_sigbus_handler_t)(const void *failure_addr); + +/** * Bus scan policies */ enum rte_bus_scan_mode { @@ -228,6 +243,9 @@ struct rte_bus { rte_dev_iterate_t dev_iterate; /**< Device iterator. */ rte_bus_hot_unplug_handler_t hot_unplug_handler; /**< handle hot-unplug failure on the bus */ + rte_bus_sigbus_handler_t sigbus_handler; + /**< handle sigbus error on the bus */ + }; /** From patchwork Sun Sep 30 10:24:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 45706 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E06835F1C; Sun, 30 Sep 2018 12:22:19 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 65A135920 for ; Sun, 30 Sep 2018 12:22:15 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Sep 2018 03:22:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,323,1534834800"; d="scan'208";a="84583577" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by FMSMGA003.fm.intel.com with ESMTP; 30 Sep 2018 03:20:35 -0700 From: Jeff Guo To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com, bernard.iremonger@intel.com, arybchenko@solarflare.com, wenzhuo.lu@intel.com, anatoly.burakov@intel.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Sun, 30 Sep 2018 18:24:10 +0800 Message-Id: <1538303053-54728-5-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1538303053-54728-1-git-send-email-jia.guo@intel.com> References: <1498711073-42917-1-git-send-email-jia.guo@intel.com> <1538303053-54728-1-git-send-email-jia.guo@intel.com> Subject: [dpdk-dev] [PATCH v11 4/7] bus/pci: implement sigbus handler ops X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch implements the ops for the PCI bus sigbus handler. It finds the PCI device that is being hot-unplugged and calls the relevant ops of the hot-unplug handler to handle the hot-unplug failure of the device. Signed-off-by: Jeff Guo Acked-by: Shaopeng He --- v11->v10: change commit log. --- drivers/bus/pci/pci_common.c | 53 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c index d286234..f313fe9 100644 --- a/drivers/bus/pci/pci_common.c +++ b/drivers/bus/pci/pci_common.c @@ -405,6 +405,36 @@ pci_find_device(const struct rte_device *start, rte_dev_cmp_t cmp, return NULL; } +/** + * find the device which encounter the failure, by iterate over all device on + * PCI bus to check if the memory failure address is located in the range + * of the BARs of the device. + */ +static struct rte_pci_device * +pci_find_device_by_addr(const void *failure_addr) +{ + struct rte_pci_device *pdev = NULL; + int i; + + FOREACH_DEVICE_ON_PCIBUS(pdev) { + for (i = 0; i != RTE_DIM(pdev->mem_resource); i++) { + if ((uint64_t)(uintptr_t)failure_addr >= + (uint64_t)(uintptr_t)pdev->mem_resource[i].addr && + (uint64_t)(uintptr_t)failure_addr < + (uint64_t)(uintptr_t)pdev->mem_resource[i].addr + + pdev->mem_resource[i].len) { + RTE_LOG(INFO, EAL, "Failure address " + "%16.16"PRIx64" belongs to " + "device %s!\n", + (uint64_t)(uintptr_t)failure_addr, + pdev->device.name); + return pdev; + } + } + } + return NULL; +} + static int pci_hot_unplug_handler(struct rte_device *dev) { @@ -433,6 +463,28 @@ pci_hot_unplug_handler(struct rte_device *dev) } static int +pci_sigbus_handler(const void *failure_addr) +{ + struct rte_pci_device *pdev = NULL; + int ret = 0; + + pdev = pci_find_device_by_addr(failure_addr); + if (!pdev) { + /* It is a generic sigbus error, no bus would handle it. */ + ret = 1; + } else { + /* The sigbus error is caused of hot-unplug. */ + ret = pci_hot_unplug_handler(&pdev->device); + if (ret) { + RTE_LOG(ERR, EAL, "Failed to handle hot-unplug for " + "device %s", pdev->name); + ret = -1; + } + } + return ret; +} + +static int pci_plug(struct rte_device *dev) { return pci_probe_all_drivers(RTE_DEV_TO_PCI(dev)); @@ -463,6 +515,7 @@ struct rte_pci_bus rte_pci_bus = { .parse = pci_parse, .get_iommu_class = rte_pci_get_iommu_class, .hot_unplug_handler = pci_hot_unplug_handler, + .sigbus_handler = pci_sigbus_handler, }, .device_list = TAILQ_HEAD_INITIALIZER(rte_pci_bus.device_list), .driver_list = TAILQ_HEAD_INITIALIZER(rte_pci_bus.driver_list), From patchwork Sun Sep 30 10:24:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 45707 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C460A5F24; Sun, 30 Sep 2018 12:22:21 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 3FE2358FE for ; Sun, 30 Sep 2018 12:22:15 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Sep 2018 03:22:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,323,1534834800"; d="scan'208";a="84583580" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by FMSMGA003.fm.intel.com with ESMTP; 30 Sep 2018 03:20:38 -0700 From: Jeff Guo To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com, bernard.iremonger@intel.com, arybchenko@solarflare.com, wenzhuo.lu@intel.com, anatoly.burakov@intel.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Sun, 30 Sep 2018 18:24:11 +0800 Message-Id: <1538303053-54728-6-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1538303053-54728-1-git-send-email-jia.guo@intel.com> References: <1498711073-42917-1-git-send-email-jia.guo@intel.com> <1538303053-54728-1-git-send-email-jia.guo@intel.com> Subject: [dpdk-dev] [PATCH v11 5/7] bus: add helper to handle sigbus X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch aims to add a helper to iterate over all buses to find the relevant bus to handle the sigbus error. Signed-off-by: Jeff Guo Acked-by: Shaopeng He --- v11->v10: change some words. --- lib/librte_eal/common/eal_common_bus.c | 43 ++++++++++++++++++++++++++++++++++ lib/librte_eal/common/eal_private.h | 13 ++++++++++ 2 files changed, 56 insertions(+) diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c index 0943851..62b7318 100644 --- a/lib/librte_eal/common/eal_common_bus.c +++ b/lib/librte_eal/common/eal_common_bus.c @@ -37,6 +37,7 @@ #include #include #include +#include #include "eal_private.h" @@ -242,3 +243,45 @@ rte_bus_get_iommu_class(void) } return mode; } + +static int +bus_handle_sigbus(const struct rte_bus *bus, + const void *failure_addr) +{ + int ret; + + if (!bus->sigbus_handler) + return -1; + + ret = bus->sigbus_handler(failure_addr); + + /* find bus but handle failed, keep the errno be set. */ + if (ret < 0 && rte_errno == 0) + rte_errno = ENOTSUP; + + return ret > 0; +} + +int +rte_bus_sigbus_handler(const void *failure_addr) +{ + struct rte_bus *bus; + + int ret = 0; + int old_errno = rte_errno; + + rte_errno = 0; + + bus = rte_bus_find(NULL, bus_handle_sigbus, failure_addr); + /* can not find bus. */ + if (!bus) + return 1; + /* find bus but handle failed, pass on the new errno. */ + else if (rte_errno != 0) + return -1; + + /* restore the old errno. */ + rte_errno = old_errno; + + return ret; +} diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h index 4f809a8..a2d1528 100644 --- a/lib/librte_eal/common/eal_private.h +++ b/lib/librte_eal/common/eal_private.h @@ -304,4 +304,17 @@ int rte_devargs_layers_parse(struct rte_devargs *devargs, const char *devstr); +/** + * Iterate over all buses to find the corresponding bus to handle the sigbus + * error. + * @param failure_addr + * Pointer of the fault address of the sigbus error. + * + * @return + * 0 success to handle the sigbus. + * -1 failed to handle the sigbus + * 1 no bus can handler the sigbus + */ +int rte_bus_sigbus_handler(const void *failure_addr); + #endif /* _EAL_PRIVATE_H_ */ From patchwork Sun Sep 30 10:24:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 45704 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4D8345920; Sun, 30 Sep 2018 12:22:16 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id D5DD558FE for ; Sun, 30 Sep 2018 12:22:13 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Sep 2018 03:22:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,323,1534834800"; d="scan'208";a="84583583" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by FMSMGA003.fm.intel.com with ESMTP; 30 Sep 2018 03:20:41 -0700 From: Jeff Guo To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com, bernard.iremonger@intel.com, arybchenko@solarflare.com, wenzhuo.lu@intel.com, anatoly.burakov@intel.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Sun, 30 Sep 2018 18:24:12 +0800 Message-Id: <1538303053-54728-7-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1538303053-54728-1-git-send-email-jia.guo@intel.com> References: <1498711073-42917-1-git-send-email-jia.guo@intel.com> <1538303053-54728-1-git-send-email-jia.guo@intel.com> Subject: [dpdk-dev] [PATCH v11 6/7] eal: add failure handle mechanism for hot-unplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The mechanism can initially register the sigbus handler after the device event monitor is enabled. When a sigbus event is captured, it will check the failure address and accordingly handle the memory failure of the corresponding device by invoke the hot-unplug handler. It could prevent the application from crashing when a device is hot-unplugged. By this patch, users could call below new added APIs to enable/disable the device hotplug handle mechanism. Note that it just implement the hot-unplug handler in these functions, the other handler of hotplug, such as handler for hotplug binding, could be add in the future if need: - rte_dev_hotplug_handle_enable - rte_dev_hotplug_handle_disable Signed-off-by: Jeff Guo Acked-by: Shaopeng He --- v11->v10: change some words and change the invoked func name. --- doc/guides/rel_notes/release_18_08.rst | 5 + lib/librte_eal/bsdapp/eal/eal_dev.c | 14 +++ lib/librte_eal/common/eal_private.h | 26 +++++ lib/librte_eal/common/include/rte_dev.h | 26 +++++ lib/librte_eal/linuxapp/eal/eal_dev.c | 162 +++++++++++++++++++++++++++++++- lib/librte_eal/rte_eal_version.map | 2 + 6 files changed, 234 insertions(+), 1 deletion(-) diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst index 321fa84..fe0e60f 100644 --- a/doc/guides/rel_notes/release_18_08.rst +++ b/doc/guides/rel_notes/release_18_08.rst @@ -117,6 +117,11 @@ New Features Added support for chained mbufs (input and output). +* **Added hot-unplug handle mechanism.** + + ``rte_dev_hotplug_handle_enable`` and ``rte_dev_hotplug_handle_disable`` are + for enabling or disabling hotplug handle mechanism. + API Changes ----------- diff --git a/lib/librte_eal/bsdapp/eal/eal_dev.c b/lib/librte_eal/bsdapp/eal/eal_dev.c index 1c6c51b..ae1c558 100644 --- a/lib/librte_eal/bsdapp/eal/eal_dev.c +++ b/lib/librte_eal/bsdapp/eal/eal_dev.c @@ -19,3 +19,17 @@ rte_dev_event_monitor_stop(void) RTE_LOG(ERR, EAL, "Device event is not supported for FreeBSD\n"); return -1; } + +int +rte_dev_hotplug_handle_enable(void) +{ + RTE_LOG(ERR, EAL, "Device event is not supported for FreeBSD\n"); + return -1; +} + +int +rte_dev_hotplug_handle_disable(void) +{ + RTE_LOG(ERR, EAL, "Device event is not supported for FreeBSD\n"); + return -1; +} diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h index a2d1528..637f20d 100644 --- a/lib/librte_eal/common/eal_private.h +++ b/lib/librte_eal/common/eal_private.h @@ -317,4 +317,30 @@ rte_devargs_layers_parse(struct rte_devargs *devargs, */ int rte_bus_sigbus_handler(const void *failure_addr); +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Register the sigbus handler. + * + * @return + * - On success, zero. + * - On failure, a negative value. + */ +int +rte_dev_sigbus_handler_register(void); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Unregister the sigbus handler. + * + * @return + * - On success, zero. + * - On failure, a negative value. + */ +int +rte_dev_sigbus_handler_unregister(void); + #endif /* _EAL_PRIVATE_H_ */ diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/common/include/rte_dev.h index b80a805..ff580a0 100644 --- a/lib/librte_eal/common/include/rte_dev.h +++ b/lib/librte_eal/common/include/rte_dev.h @@ -460,4 +460,30 @@ rte_dev_event_monitor_start(void); int __rte_experimental rte_dev_event_monitor_stop(void); +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Enable hotplug handling for devices. + * + * @return + * - On success, zero. + * - On failure, a negative value. + */ +int __rte_experimental +rte_dev_hotplug_handle_enable(void); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Disable hotplug handling for devices. + * + * @return + * - On success, zero. + * - On failure, a negative value. + */ +int __rte_experimental +rte_dev_hotplug_handle_disable(void); + #endif /* _RTE_DEV_H_ */ diff --git a/lib/librte_eal/linuxapp/eal/eal_dev.c b/lib/librte_eal/linuxapp/eal/eal_dev.c index 1cf6aeb..9f9e1cf 100644 --- a/lib/librte_eal/linuxapp/eal/eal_dev.c +++ b/lib/librte_eal/linuxapp/eal/eal_dev.c @@ -4,6 +4,8 @@ #include #include +#include +#include #include #include @@ -14,15 +16,32 @@ #include #include #include +#include +#include +#include +#include #include "eal_private.h" static struct rte_intr_handle intr_handle = {.fd = -1 }; static bool monitor_started; +static bool hotplug_handle; #define EAL_UEV_MSG_LEN 4096 #define EAL_UEV_MSG_ELEM_LEN 128 +/* + * spinlock for device hot-unplug failure handling. If it try to access bus or + * device, such as handle sigbus on bus or handle memory failure for device + * just need to use this lock. It could protect the bus and the device to avoid + * race condition. + */ +static rte_spinlock_t failure_handle_lock = RTE_SPINLOCK_INITIALIZER; + +static struct sigaction sigbus_action_old; + +static int sigbus_need_recover; + static void dev_uev_handler(__rte_unused void *param); /* identify the system layer which reports this event. */ @@ -33,6 +52,49 @@ enum eal_dev_event_subsystem { EAL_DEV_EVENT_SUBSYSTEM_MAX }; +static void +sigbus_action_recover(void) +{ + if (sigbus_need_recover) { + sigaction(SIGBUS, &sigbus_action_old, NULL); + sigbus_need_recover = 0; + } +} + +static void sigbus_handler(int signum, siginfo_t *info, + void *ctx __rte_unused) +{ + int ret; + + RTE_LOG(INFO, EAL, "Thread[%d] catch SIGBUS, fault address:%p\n", + (int)pthread_self(), info->si_addr); + + rte_spinlock_lock(&failure_handle_lock); + ret = rte_bus_sigbus_handler(info->si_addr); + rte_spinlock_unlock(&failure_handle_lock); + if (ret == -1) { + rte_exit(EXIT_FAILURE, + "Failed to handle SIGBUS for hot-unplug, " + "(rte_errno: %s)!", strerror(rte_errno)); + } else if (ret == 1) { + if (sigbus_action_old.sa_handler) + (*(sigbus_action_old.sa_handler))(signum); + else + rte_exit(EXIT_FAILURE, + "Failed to handle generic SIGBUS!"); + } + + RTE_LOG(INFO, EAL, "Success to handle SIGBUS for hot-unplug!\n"); +} + +static int cmp_dev_name(const struct rte_device *dev, + const void *_name) +{ + const char *name = _name; + + return strcmp(dev->name, name); +} + static int dev_uev_socket_fd_create(void) { @@ -147,6 +209,9 @@ dev_uev_handler(__rte_unused void *param) struct rte_dev_event uevent; int ret; char buf[EAL_UEV_MSG_LEN]; + struct rte_bus *bus; + struct rte_device *dev; + const char *busname = ""; memset(&uevent, 0, sizeof(struct rte_dev_event)); memset(buf, 0, EAL_UEV_MSG_LEN); @@ -171,8 +236,43 @@ dev_uev_handler(__rte_unused void *param) RTE_LOG(DEBUG, EAL, "receive uevent(name:%s, type:%d, subsystem:%d)\n", uevent.devname, uevent.type, uevent.subsystem); - if (uevent.devname) + switch (uevent.subsystem) { + case EAL_DEV_EVENT_SUBSYSTEM_PCI: + case EAL_DEV_EVENT_SUBSYSTEM_UIO: + busname = "pci"; + break; + default: + break; + } + + if (uevent.devname) { + if (uevent.type == RTE_DEV_EVENT_REMOVE && hotplug_handle) { + rte_spinlock_lock(&failure_handle_lock); + bus = rte_bus_find_by_name(busname); + if (bus == NULL) { + RTE_LOG(ERR, EAL, "Cannot find bus (%s)\n", + busname); + return; + } + + dev = bus->find_device(NULL, cmp_dev_name, + uevent.devname); + if (dev == NULL) { + RTE_LOG(ERR, EAL, "Cannot find device (%s) on " + "bus (%s)\n", uevent.devname, busname); + return; + } + + ret = bus->hot_unplug_handler(dev); + rte_spinlock_unlock(&failure_handle_lock); + if (ret) { + RTE_LOG(ERR, EAL, "Can not handle hot-unplug " + "for device (%s)\n", dev->name); + return; + } + } dev_callback_process(uevent.devname, uevent.type); + } } int __rte_experimental @@ -220,5 +320,65 @@ rte_dev_event_monitor_stop(void) close(intr_handle.fd); intr_handle.fd = -1; monitor_started = false; + return 0; } + +int +rte_dev_sigbus_handler_register(void) +{ + sigset_t mask; + struct sigaction action; + + rte_errno = 0; + + sigemptyset(&mask); + sigaddset(&mask, SIGBUS); + action.sa_flags = SA_SIGINFO; + action.sa_mask = mask; + action.sa_sigaction = sigbus_handler; + sigbus_need_recover = !sigaction(SIGBUS, &action, &sigbus_action_old); + + return rte_errno; +} + +int +rte_dev_sigbus_handler_unregister(void) +{ + rte_errno = 0; + sigbus_need_recover = 1; + + sigbus_action_recover(); + + return rte_errno; +} + +int +rte_dev_hotplug_handle_enable(void) +{ + int ret = 0; + + ret = rte_dev_sigbus_handler_register(); + if (ret < 0) + RTE_LOG(ERR, EAL, "fail to register sigbus handler for " + "devices.\n"); + + hotplug_handle = true; + + return ret; +} + +int +rte_dev_hotplug_handle_disable(void) +{ + int ret = 0; + + ret = rte_dev_sigbus_handler_unregister(); + if (ret < 0) + RTE_LOG(ERR, EAL, "fail to unregister sigbus handler for " + "devices.\n"); + + hotplug_handle = false; + + return ret; +} diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map index 73282bb..a3255aa 100644 --- a/lib/librte_eal/rte_eal_version.map +++ b/lib/librte_eal/rte_eal_version.map @@ -281,6 +281,8 @@ EXPERIMENTAL { rte_dev_event_callback_unregister; rte_dev_event_monitor_start; rte_dev_event_monitor_stop; + rte_dev_hotplug_handle_enable; + rte_dev_hotplug_handle_disable; rte_dev_iterator_init; rte_dev_iterator_next; rte_devargs_add; From patchwork Sun Sep 30 10:24:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 45705 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 685705B38; Sun, 30 Sep 2018 12:22:18 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id C41D458FE for ; Sun, 30 Sep 2018 12:22:14 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Sep 2018 03:22:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,323,1534834800"; d="scan'208";a="84583589" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by FMSMGA003.fm.intel.com with ESMTP; 30 Sep 2018 03:20:45 -0700 From: Jeff Guo To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com, bernard.iremonger@intel.com, arybchenko@solarflare.com, wenzhuo.lu@intel.com, anatoly.burakov@intel.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Sun, 30 Sep 2018 18:24:13 +0800 Message-Id: <1538303053-54728-8-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1538303053-54728-1-git-send-email-jia.guo@intel.com> References: <1498711073-42917-1-git-send-email-jia.guo@intel.com> <1538303053-54728-1-git-send-email-jia.guo@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v11 7/7] testpmd: use hot-unplug failure handle mechanism X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch use testpmd for example, to show how an app smoothly handle failure when device be hot-unplug. Except app should enabled the device event monitor and register the hotplug event’s callback, it also need enable hotplug handle mechanism before running. Once app detect the removal event, the hot-unplug callback would be called. It will first stop the packet forwarding, then stop the port, close the port, and finally detach the port to clean the device and release the resources. Signed-off-by: Jeff Guo --- v11->v10: since we plan to abandon RTE_ETH_EVENT_INTR_RMV, change to use RTE_DEV_EVENT_REMOVE, so modify the hotplug event and callback usage. --- app/test-pmd/testpmd.c | 39 +++++++++++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 001f0e5..bfef483 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -2093,14 +2093,22 @@ pmd_test_exit(void) if (hot_plug) { ret = rte_dev_event_monitor_stop(); - if (ret) + if (ret) { RTE_LOG(ERR, EAL, "fail to stop device event monitor."); + return; + } ret = eth_dev_event_callback_unregister(); if (ret) + return; + + ret = rte_dev_hotplug_handle_disable(); + if (ret) { RTE_LOG(ERR, EAL, - "fail to unregister all event callbacks."); + "fail to disable hotplug handling."); + return; + } } printf("\nBye...\n"); @@ -2244,6 +2252,9 @@ static void eth_dev_event_callback(char *device_name, enum rte_dev_event_type type, __rte_unused void *arg) { + uint16_t port_id; + int ret; + if (type >= RTE_DEV_EVENT_MAX) { fprintf(stderr, "%s called upon invalid event %d\n", __func__, type); @@ -2254,9 +2265,12 @@ eth_dev_event_callback(char *device_name, enum rte_dev_event_type type, case RTE_DEV_EVENT_REMOVE: RTE_LOG(ERR, EAL, "The device: %s has been removed!\n", device_name); - /* TODO: After finish failure handle, begin to stop - * packet forward, stop port, close port, detach port. - */ + ret = rte_eth_dev_get_port_by_name(device_name, &port_id); + if (ret) { + printf("can not get port by device %s!\n", device_name); + return; + } + rmv_event_callback((void *)(intptr_t)port_id); break; case RTE_DEV_EVENT_ADD: RTE_LOG(ERR, EAL, "The device: %s has been added!\n", @@ -2779,14 +2793,23 @@ main(int argc, char** argv) init_config(); if (hot_plug) { - /* enable hot plug monitoring */ + ret = rte_dev_hotplug_handle_enable(); + if (ret) { + RTE_LOG(ERR, EAL, + "fail to enable hotplug handling."); + return -1; + } + ret = rte_dev_event_monitor_start(); if (ret) { - rte_errno = EINVAL; + RTE_LOG(ERR, EAL, + "fail to start device event monitoring."); return -1; } - eth_dev_event_callback_register(); + ret = eth_dev_event_callback_register(); + if (ret) + return -1; } if (start_port(RTE_PORT_ALL) != 0)