Cover Detail
Show a cover letter.
GET /api/covers/41388/?format=api
{ "id": 41388, "url": "http://patches.dpdk.org/api/covers/41388/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/cover/1529668268-7462-1-git-send-email-jia.guo@intel.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<1529668268-7462-1-git-send-email-jia.guo@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/1529668268-7462-1-git-send-email-jia.guo@intel.com", "date": "2018-06-22T11:51:04", "name": "[v2,0/4] hot plug failure handle mechanism", "submitter": { "id": 507, "url": "http://patches.dpdk.org/api/people/507/?format=api", "name": "Guo, Jia", "email": "jia.guo@intel.com" }, "mbox": "http://patches.dpdk.org/project/dpdk/cover/1529668268-7462-1-git-send-email-jia.guo@intel.com/mbox/", "series": [ { "id": 205, "url": "http://patches.dpdk.org/api/series/205/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=205", "date": "2018-06-22T11:51:04", "name": "hot plug failure handle mechanism", "version": 2, "mbox": "http://patches.dpdk.org/series/205/mbox/" } ], "comments": "http://patches.dpdk.org/api/covers/41388/comments/", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 73B071BB15;\n\tFri, 22 Jun 2018 13:48:06 +0200 (CEST)", "from mga09.intel.com (mga09.intel.com [134.134.136.24])\n\tby dpdk.org (Postfix) with ESMTP id 6B1191BB12\n\tfor <dev@dpdk.org>; Fri, 22 Jun 2018 13:48:03 +0200 (CEST)", "from fmsmga007.fm.intel.com ([10.253.24.52])\n\tby orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t22 Jun 2018 04:48:02 -0700", "from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain)\n\t([10.67.110.10])\n\tby fmsmga007.fm.intel.com with ESMTP; 22 Jun 2018 04:47:59 -0700" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.51,256,1526367600\"; d=\"scan'208\";a=\"49078325\"", "From": "Jeff Guo <jia.guo@intel.com>", "To": "stephen@networkplumber.org, bruce.richardson@intel.com,\n\tferruh.yigit@intel.com, konstantin.ananyev@intel.com,\n\tgaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net,\n\tmotih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com,\n\tqi.z.zhang@intel.com, shaopeng.he@intel.com", "Cc": "jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org,\n\tjia.guo@intel.com, helin.zhang@intel.com", "Date": "Fri, 22 Jun 2018 19:51:04 +0800", "Message-Id": "<1529668268-7462-1-git-send-email-jia.guo@intel.com>", "X-Mailer": "git-send-email 2.7.4", "In-Reply-To": "<1498711073-42917-1-git-send-email-jia.guo@intel.com>", "References": "<1498711073-42917-1-git-send-email-jia.guo@intel.com>", "MIME-Version": "1.0", "Content-Type": "text/plain; charset=UTF-8", "Content-Transfer-Encoding": "8bit", "Subject": "[dpdk-dev] [PATCH v2 0/4] hot plug failure handle mechanism", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "As we know, hot plug is an importance feature, either use for the datacenter\ndevice’s fail-safe, or use for SRIOV Live Migration in SDN/NFV. It could bring\nthe higher flexibility and continuality to the networking services in multiple\nuse cases in industry. So let we see, dpdk as an importance networking\nframework, what can it help to implement hot plug solution for users.\n\nWe already have a general device event detect mechanism, failsafe driver,\nbonding driver and hot plug/unplug api in framework, app could use these to\ndevelop their hot plug solution.\n\nlet’s see the case of hot unplug, it can happen when a hardware device is\nbe removed physically, or when the software disables it. App need to call\nether dev API to detach the device, to unplug the device at the bus level and\nmake access to the device invalid. But the problem is that, the removal of the\ndevice from the software lists is not going to be instantaneous, at this time\nif the data(fast) path still read/write the device, it will cause MMIO error\nand result of the app crash out.\n\nSeems that we have got fail-safe driver(or app) + RTE_ETH_EVENT_INTR_RMV +\nkernel core driver solution to handle it, but still not have failsafe driver\n(or app) + RTE_DEV_EVENT_REMOVE + PCIe pmd driver failure handle solution. So\nthere is an absence in dpdk hot plug solution right now.\n\nAlso, we know that kernel only guaranty hot plug on the kernel side, but not for\nthe user mode side. Firstly we can hardly have a gatekeeper for any MMIO for\nmultiple PMD driver. Secondly, no more specific 3rd tools such as udev/driverctl\nhave especially cover these hot plug failure processing. Third, the feasibility\nof app’s implement for multiple user mode PMD driver is still a problem. Here,\na general hot plug failure handle mechanism in dpdk framework would be proposed,\nit aim to guaranty that, when hot unplug occur, the system will not crash and\napp will not be break out, and user space can normally stop and release any\nrelevant resources, then unplug of the device at the bus level cleanly.\n\nThe mechanism should be come across as bellow:\n\nFirstly, app enabled the device event monitor and register the hot plug event’s\ncallback before running data path. Once the hot unplug behave occur, the\nmechanism will detect the removal event and then accordingly do the failure\nhandle. In order to do that, below functional will be bring in.\n - Add a new bus ops “handle_hot_unplug” to handle bus read/write error, it is\n bus-specific and each kind of bus can implement its own logic.\n - Implement pci bus specific ops “pci_handle_hot_unplug”. It will base on the\n failure address to remap memory for the corresponding device that unplugged.\n\nFor the data path or other unexpected control from the control path when hot\nunplug occur.\n - Implement a new sigbus handler, it is registered when start device even\n monitoring. The handler is per process. Base on the signal event principle,\n control path thread and data path thread will randomly receive the sigbus\n error, but will go to the common sigbus handler. Once the MMIO sigbus error\n exposure, it will trigger the above hot unplug operation. The sigbus will be\n check if it is cause of the hot unplug or not, if not will info exception as\n the original sigbus handler. If yes, will do memory remapping.\n\nFor the control path and the igb uio release:\n - When hot unplug device, the kernel will release the device resource in the\n kernel side, such as the fd sys file will disappear, and the irq will be\n released. At this time, if igb uio driver still try to release this resource,\n it will cause kernel crash.\n On the other hand, something like interrupt disable do not automatically\n process in kernel side. If not handler it, this redundancy and dirty thing\n will affect the interrupt resource be used by other device.\n So the igb_uio driver have to check the hot plug status and corresponding\n process should be taken in igb uio deriver.\n This patch propose to add structure of rte_udev_state into rte_uio_pci_dev\n of igb_uio kernel driver, which will record the state of uio device, such as\n probed/opened/released/removed/unplug. When detect the unexpected removal\n which cause of hot unplug behavior, it will corresponding disable interrupt\n resource, while for the part of releasement which kernel have already handle,\n just skip it to avoid double free or null pointer kernel crash issue.\n\nThe mechanism could be use for fail-safe driver and app which want to use hot\nplug solution. At this stage, will only use testpmd as reference to show how to\nuse the mechanism.\n - Enable device event monitor->device unplug->failure handle->stop forwarding->\n stop port->close port->detach port.\n\nThis process will not breaking the app/fail-safe running, and will not break\nother irrelevance device. And app could plug in the device and restart the date\npath again by below.\n - Device plug in->bind igb_uio driver ->attached device->start port->\n start forwarding.\n\npatchset history:\n\nv2->v1(v21):\nrefine some doc and commit log\nfix igb uio kernel issue for control path failure\nrebase testpmd code \n\nSince the hot plug solution be discussed serval around in the public, the\nscope be changed and the patch set be split into many times. Coming to the\nrecently RFC and feature design, it just focus on the hot unplug failure\nhandler at this patch set, so in order let this topic more clear and focus,\nsummarize privours patch set in history “v1(v21)”, the v2 here go ahead\nfor further track. \n\n\"v1(21)\" == v21 as below:\nv21->v20:\nsplit function in hot unplug ops\nsync failure hanlde to fix multiple process issue fix attach port issue for multiple devices case.\ncombind rmv callback function to be only one.\n\nv20->v19:\nclean the code\nrefine the remap logic for multiple device.\nremove the auto binding\n\nv19->18:\nnote for limitation of multiple hotplug,fix some typo, sqeeze patch.\n\nv18->v15:\nadd document, add signal bus handler, refine the code to be more clear.\n\nthe prior patch history please check the patch set \"add device event monitor framework\"\n\nJeff Guo (4):\n bus/pci: handle device hot unplug\n eal: add failure handle mechanism for hot plug\n igb_uio: fix uio release issue when hot unplug\n app/testpmd: show example to handle hot unplug\n\n app/test-pmd/testpmd.c | 25 ++++--\n drivers/bus/pci/pci_common.c | 65 ++++++++++++++\n drivers/bus/pci/pci_common_uio.c | 33 +++++++\n drivers/bus/pci/private.h | 12 +++\n kernel/linux/igb_uio/igb_uio.c | 50 +++++++++--\n lib/librte_eal/common/include/rte_bus.h | 16 ++++\n lib/librte_eal/linuxapp/eal/eal_dev.c | 154 +++++++++++++++++++++++++++++++-\n 7 files changed, 344 insertions(+), 11 deletions(-)" }