From patchwork Fri Aug 17 10:48:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 43761 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DFD99FEB; Fri, 17 Aug 2018 12:51:22 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 100EF98 for ; Fri, 17 Aug 2018 12:51:20 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Aug 2018 03:51:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,250,1531810800"; d="scan'208";a="63131811" Received: from jeffguo-z170x-ud5.sh.intel.com (HELO localhost.localdomain) ([10.67.104.10]) by fmsmga007.fm.intel.com with ESMTP; 17 Aug 2018 03:51:16 -0700 From: Jeff Guo To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com, bernard.iremonger@intel.com, arybchenko@solarflare.com, wenzhuo.lu@intel.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Fri, 17 Aug 2018 18:48:27 +0800 Message-Id: <1534502916-31636-1-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1498711073-42917-1-git-send-email-jia.guo@intel.com> References: <1498711073-42917-1-git-send-email-jia.guo@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v10 0/8] hotplug failure handle mechanism X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hotplug is an important feature for use-cases like the datacenter device's fail-safe and for SRIOV Live Migration in SDN/NFV. It could bring higher flexibility and continuality to networking services in multiple use-cases in the industry. So let's see how DPDK can help users implement hotplug solutions. We already have a general device-event monitor mechanism, failsafe driver, and hot plug/unplug API in DPDK. We have already got the solution of “ethdev event + kernel PMD hotplug handler + failsafe”, but we still not got “eal event + hotplug handler for pci PMD + failsafe” implement, and we need to considerate 2 different solutions between uio and vfio. In the case of hotplug for igb_uio, when a hardware device be removed physically or disabled in software, the application needs to be notified and detach the device out of the bus, and then make the device invalidate. The problem is that, the removal of the device is not instantaneous in software. If the application data path tries to read/write to the device when removal is still in process, it will cause an MMIO error and application will crash. In this patch set, we propose a PCIe bus failure handler mechanism for hotplug in igb_uio. It aims to guarantee that, when a hot unplug occurs, the system will not crash and the application will not break out. The mechanism should work as below: First, the application enables the device event monitor, registers the hotplug event’s callback and enable hotplug handling before running the data path. Once the hot unplug occurs, the mechanism will detect the removal event and then accordingly do the failure handling. In order to do that, the below functionality will be required: - Add a new bus ops “memory_failure_handler” to handle bus read/write errors. - Implement pci bus specific ops “pci_memory_failure_handler”. It will be based on the failure address to remap memory for the corresponding device that unplugged. For the data path or other unexpected behaviors from the control path when a hot unplug occurs: - Add a new bus ops “sigbus_handler”, that is responsible for handling the sigbus error which is either an original memory error, or a specific memory error that is caused by a hot unplug. When a sigbus error is captured, it will call this function to handle sigbus error. - Implement PCI bus specific ops “pci_sigbus_handler”. It will iterate all device on PCI bus to find which device encounter the failure. - Implement a "rte_bus_sigbus_handler" to iterate all buses to find a bus to handle the failure. - Add a couple of APIs “rte_dev_hotplug_handle_enable” and “rte_dev_hotplug_handle_diable” to enable/disable hotplug handling. It will monitor the sigbus error by a handler which is per-process. Based on the signal event principle, the control path thread and the data path thread will randomly receive the sigbus error, but will call the common sigbus process. When sigbus be captured, it will call the above API to find bus to handle it. The mechanism could be used by app or PMDs. For example, the whole process of hotplug in testpmd is: - Enable device event monitor->Enable hotplug handle->Register event callback ->attach port->start port->start forwarding->Device unplug->failure handle ->stop forwarding->stop port->close port->detach port. This patch set would not cover hotplug insert and binding, and it is only implement the igb_uio failure handler, the vfio hotplug failure handler will be in next coming patch set. patchset history: v10->v9: modify the api name and exposure out for public use. add hotplug handle enable/disable APIs refine commit log v9->v8: refine commit log to be more readable. v8->v7: refine errno process in sigbus handler. refine igb uio release process v7->v6: delete some unused part v6->v5: refine some description about bus ops refine commit log add some entry check. v5->v4: split patches to focus on the failure handle, remove the event usage by testpmd to another patch. change the hotplug failure handler name. refine the sigbus handle logic. add lock for udev state in igb uio driver. v4->v3: split patches to be small and clear. change to use new parameter "--hotplug-mode" in testpmd to identify the eal hotplug and ethdev hotplug. v3->v2: change bus ops name to bus_hotplug_handler. add new API and bus ops of bus_signal_handler distingush handle generic. sigbus and hotplug sigbus. v2->v1(v21): refine some doc and commit log. fix igb uio kernel issue for control path failure rebase testpmd code. Since the hot plug solution be discussed serval around in the public, the scope be changed and the patch set be split into many times. Coming to the recently RFC and feature design, it just focus on the hot unplug failure handler at this patch set, so in order let this topic more clear and focus, summarize privours patch set in history “v1(v21)”, the v2 here go ahead for further track. "v1(21)" == v21 as below: v21->v20: split function in hot unplug ops. sync failure hanlde to fix multiple process issue fix attach port issue for multiple devices case. combind rmv callback function to be only one. v20->v19: clean the code. refine the remap logic for multiple device. remove the auto binding. v19->18: note for limitation of multiple hotplug, fix some typo, sqeeze patch. v18->v15: add document, add signal bus handler, refine the code to be more clear. the prior patch history please check the patch set "add device event monitor framework". Jeff Guo (8): bus: add memory failure handler bus/pci: implement memory failure handler ops bus: add sigbus handler bus/pci: implement sigbus handler ops bus: add helper to handle sigbus eal: add failure handle mechanism for hotplug igb_uio: fix unexpected remove issue for hotplug testpmd: use hotplug failure handle mechanism app/test-pmd/testpmd.c | 27 +++++- doc/guides/rel_notes/release_18_08.rst | 5 + drivers/bus/pci/pci_common.c | 81 ++++++++++++++++ drivers/bus/pci/pci_common_uio.c | 33 +++++++ drivers/bus/pci/private.h | 12 +++ kernel/linux/igb_uio/igb_uio.c | 69 +++++++++++--- lib/librte_eal/bsdapp/eal/eal_dev.c | 14 +++ lib/librte_eal/common/eal_common_bus.c | 43 +++++++++ lib/librte_eal/common/eal_private.h | 38 ++++++++ lib/librte_eal/common/include/rte_bus.h | 34 +++++++ lib/librte_eal/common/include/rte_dev.h | 26 ++++++ lib/librte_eal/linuxapp/eal/eal_dev.c | 159 +++++++++++++++++++++++++++++++- lib/librte_eal/rte_eal_version.map | 2 + 13 files changed, 523 insertions(+), 20 deletions(-)