Cover Detail
Show a cover letter.
GET /api/covers/42293/?format=api
{ "id": 42293, "url": "http://patches.dpdk.org/api/covers/42293/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/cover/1530776333-30318-1-git-send-email-jia.guo@intel.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<1530776333-30318-1-git-send-email-jia.guo@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/1530776333-30318-1-git-send-email-jia.guo@intel.com", "date": "2018-07-05T07:38:47", "name": "[V5,0/7] hot plug failure handle mechanism", "submitter": { "id": 507, "url": "http://patches.dpdk.org/api/people/507/?format=api", "name": "Guo, Jia", "email": "jia.guo@intel.com" }, "mbox": "http://patches.dpdk.org/project/dpdk/cover/1530776333-30318-1-git-send-email-jia.guo@intel.com/mbox/", "series": [ { "id": 412, "url": "http://patches.dpdk.org/api/series/412/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=412", "date": "2018-07-05T07:38:47", "name": "hot plug failure handle mechanism", "version": 5, "mbox": "http://patches.dpdk.org/series/412/mbox/" } ], "comments": "http://patches.dpdk.org/api/covers/42293/comments/", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 9D60D1BE26;\n\tThu, 5 Jul 2018 09:41:07 +0200 (CEST)", "from mga09.intel.com (mga09.intel.com [134.134.136.24])\n\tby dpdk.org (Postfix) with ESMTP id A78FD1B47E\n\tfor <dev@dpdk.org>; Thu, 5 Jul 2018 09:41:05 +0200 (CEST)", "from fmsmga007.fm.intel.com ([10.253.24.52])\n\tby orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t05 Jul 2018 00:41:04 -0700", "from jeffguo-z170x-ud5.sh.intel.com (HELO localhost.localdomain)\n\t([10.67.104.10])\n\tby fmsmga007.fm.intel.com with ESMTP; 05 Jul 2018 00:41:00 -0700" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.51,311,1526367600\"; d=\"scan'208\";a=\"52144023\"", "From": "Jeff Guo <jia.guo@intel.com>", "To": "stephen@networkplumber.org, bruce.richardson@intel.com,\n\tferruh.yigit@intel.com, konstantin.ananyev@intel.com,\n\tgaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net,\n\tmotih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com,\n\tqi.z.zhang@intel.com, shaopeng.he@intel.com, bernard.iremonger@intel.com", "Cc": "jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org,\n\tjia.guo@intel.com, helin.zhang@intel.com", "Date": "Thu, 5 Jul 2018 15:38:47 +0800", "Message-Id": "<1530776333-30318-1-git-send-email-jia.guo@intel.com>", "X-Mailer": "git-send-email 2.7.4", "In-Reply-To": "<1498711073-42917-1-git-send-email-jia.guo@intel.com>", "References": "<1498711073-42917-1-git-send-email-jia.guo@intel.com>", "MIME-Version": "1.0", "Content-Type": "text/plain; charset=UTF-8", "Content-Transfer-Encoding": "8bit", "Subject": "[dpdk-dev] [PATCH V5 0/7] hot plug failure handle mechanism", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "As we know, hot plug is an importance feature, either use for the datacenter\ndevice’s fail-safe, or use for SRIOV Live Migration in SDN/NFV. It could bring\nthe higher flexibility and continuality to the networking services in multiple\nuse cases in industry. So let we see, dpdk as an importance networking\nframework, what can it help to implement hot plug solution for users.\n\nWe already have a general device event detect mechanism, failsafe driver,\nbonding driver and hot plug/unplug api in framework, app could use these to\ndevelop their hot plug solution.\n\nlet’s see the case of hot unplug, it can happen when a hardware device is\nbe removed physically, or when the software disables it. App need to call\nether dev API to detach the device, to unplug the device at the bus level and\nmake access to the device invalid. But the problem is that, the removal of the\ndevice from the software lists is not going to be instantaneous, at this time\nif the data(fast) path still read/write the device, it will cause MMIO error\nand result of the app crash out.\n\nSeems that we have got fail-safe driver(or app) + RTE_ETH_EVENT_INTR_RMV +\nkernel core driver solution to handle it, but still not have failsafe driver\n(or app) + RTE_DEV_EVENT_REMOVE + PCIe pmd driver failure handle solution. So\nthere is an absence in dpdk hot plug solution right now.\n\nAlso, we know that kernel only guaranty hot plug on the kernel side, but not for\nthe user mode side. Firstly we can hardly have a gatekeeper for any MMIO for\nmultiple PMD driver. Secondly, no more specific 3rd tools such as udev/driverctl\nhave especially cover these hot plug failure processing. Third, the feasibility\nof app’s implement for multiple user mode PMD driver is still a problem. Here,\na general hot plug failure handle mechanism in dpdk framework would be proposed,\nit aim to guaranty that, when hot unplug occur, the system will not crash and\napp will not be break out, and user space can normally stop and release any\nrelevant resources, then unplug of the device at the bus level cleanly.\n\nThe mechanism should be come across as bellow:\n\nFirstly, app enabled the device event monitor and register the hot plug event’s\ncallback before running data path. Once the hot unplug behave occur, the\nmechanism will detect the removal event and then accordingly do the failure\nhandle. In order to do that, below functional will be bring in.\n - Add a new bus ops “handle_hot_unplug” to handle bus read/write error, it is\n bus-specific and each kind of bus can implement its own logic.\n - Implement pci bus specific ops “pci_handle_hot_unplug”. It will base on the\n failure address to remap memory for the corresponding device that unplugged.\n\nFor the data path or other unexpected control from the control path when hot\nunplug occur.\n - Implement a new sigbus handler, it is registered when start device even\n monitoring. The handler is per process. Base on the signal event principle,\n control path thread and data path thread will randomly receive the sigbus\n error, but will go to the common sigbus handler. Once the MMIO sigbus error\n exposure, it will trigger the above hot unplug operation. The sigbus will be\n check if it is cause of the hot unplug or not, if not will info exception as\n the original sigbus handler. If yes, will do memory remapping.\n\nFor the control path and the igb uio release:\n - When hot unplug device, the kernel will release the device resource in the\n kernel side, such as the fd sys file will disappear, and the irq will be\n released. At this time, if igb uio driver still try to release this resource,\n it will cause kernel crash.\n On the other hand, something like interrupt disable do not automatically\n process in kernel side. If not handler it, this redundancy and dirty thing\n will affect the interrupt resource be used by other device.\n So the igb_uio driver have to check the hot plug status and corresponding\n process should be taken in igb uio deriver.\n This patch propose to add structure of rte_udev_state into rte_uio_pci_dev\n of igb_uio kernel driver, which will record the state of uio device, such as\n probed/opened/released/removed/unplug. When detect the unexpected removal\n which cause of hot unplug behavior, it will corresponding disable interrupt\n resource, while for the part of releasement which kernel have already handle,\n just skip it to avoid double free or null pointer kernel crash issue.\n\nThe mechanism could be use for fail-safe driver and app which want to use hot\nplug solution. let testpmd for example:\n - Enable device event monitor->device unplug->failure handle->stop forwarding->\n stop port->close port->detach port.\n\nThis process will not breaking the app/fail-safe running, and will not break\nother irrelevance device. And app could plug in the device and restart the date\npath again by below.\n - Device plug in->bind igb_uio driver ->attached device->start port->\n start forwarding.\n\npatchset history:\nv5->v4:\nsplit patches to focus on the failure handle, remove the event usage by testpmd\nto another patch.\nchange the hotplug failure handler name\nrefine the sigbus handle logic\nadd lock for udev state in igb uio driver \n\nv4->v3:\nsplit patches to be small and clear\nchange to use new parameter \"--hotplug-mode\" in testpmd\nto identify the eal hotplug and ethdev hotplug\n\nv3->v2:\nchange bus ops name to bus_hotplug_handler.\nadd new API and bus ops of bus_signal_handler\ndistingush handle generic sigbus and hotplug sigbus\n\nv2->v1(v21):\nrefine some doc and commit log\nfix igb uio kernel issue for control path failure\nrebase testpmd code\n\nSince the hot plug solution be discussed serval around in the public, the\nscope be changed and the patch set be split into many times. Coming to the\nrecently RFC and feature design, it just focus on the hot unplug failure\nhandler at this patch set, so in order let this topic more clear and focus,\nsummarize privours patch set in history “v1(v21)”, the v2 here go ahead\nfor further track.\n\n\"v1(21)\" == v21 as below:\nv21->v20:\nsplit function in hot unplug ops\nsync failure hanlde to fix multiple process issue fix attach port issue for multiple devices case.\ncombind rmv callback function to be only one.\n\nv20->v19:\nclean the code\nrefine the remap logic for multiple device.\nremove the auto binding\n\nv19->18:\nnote for limitation of multiple hotplug,fix some typo, sqeeze patch.\n\nv18->v15:\nadd document, add signal bus handler, refine the code to be more clear.\n\nthe prior patch history please check the patch set \"add device event monitor framework\"\n\nJeff Guo (7):\n bus: add hotplug failure handler\n bus/pci: implement hotplug failure handler ops\n bus: add sigbus handler\n bus/pci: implement sigbus handler operation\n bus: add helper to handle sigbus\n eal: add failure handle mechanism for hotplug\n igb_uio: fix uio release issue when hot unplug\n\n drivers/bus/pci/pci_common.c | 77 ++++++++++++++++++++++\n drivers/bus/pci/pci_common_uio.c | 33 ++++++++++\n drivers/bus/pci/private.h | 12 ++++\n kernel/linux/igb_uio/igb_uio.c | 51 ++++++++++++++-\n lib/librte_eal/common/eal_common_bus.c | 36 ++++++++++-\n lib/librte_eal/common/eal_private.h | 12 ++++\n lib/librte_eal/common/include/rte_bus.h | 31 +++++++++\n lib/librte_eal/linuxapp/eal/eal_dev.c | 111 +++++++++++++++++++++++++++++++-\n 8 files changed, 358 insertions(+), 5 deletions(-)" }