From patchwork Thu Jun 21 02:00:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zhang X-Patchwork-Id: 41327 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3014B1B8C1; Thu, 21 Jun 2018 04:10:23 +0200 (CEST) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id 6C8681B61B for ; Thu, 21 Jun 2018 04:00:38 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Jun 2018 19:00:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,249,1526367600"; d="scan'208";a="64993791" Received: from dpdk51.sh.intel.com ([10.67.110.190]) by fmsmga004.fm.intel.com with ESMTP; 20 Jun 2018 19:00:26 -0700 From: Qi Zhang To: thomas@monjalon.net, anatoly.burakov@intel.com Cc: konstantin.ananyev@intel.com, dev@dpdk.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, benjamin.h.shelton@intel.com, narender.vangati@intel.com, Qi Zhang Date: Thu, 21 Jun 2018 10:00:37 +0800 Message-Id: <20180621020059.1198-1-qi.z.zhang@intel.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20180607123849.14439-1-qi.z.zhang@intel.com> References: <20180607123849.14439-1-qi.z.zhang@intel.com> Subject: [dpdk-dev] [PATCH v2 00/22] enable hotplug on multi-process X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" v2: - rename rte_ethdev_mp.* to ethdev_mp.* - rename rte_ethdev_lock.* to ethdev_lock.* - move internal funciton to ethdev_private.h - separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and rte_eth_dev_[un]lock_with_callback - lock callbacks will be removed automatically after device is detached. - add experimental tag for all new APIs. - fix coding style issue. - fix wrong lisence header in sample code. - fix spelling - fix meson.build. - improve comments. Background: =========== Currently secondary process will only sync ethdev from primary process at init stage, but it will not be aware if device is attached/detached on primary process at runtime. While there is the requirement from application that take primary-secondary process model. The primary process work as a resource management process, it will create/destroy virtual device at runtime, while the secondary process deal with the network stuff with these devices. Solution: ========= So the orignial intention is to fix this gap, but beyond that the patch set provide a more comprehesive solution to handle different hotplug cases in multi-process situation, it cover below scenario: 1. Attach a share device from primary 2. Detach a share device from primary 3. Attach a share device from secondary 4. Detach a share device from secondary 5. Attach a private device from secondary 6. Detach a private device from secondary 7. Detach a share device from secondary privately 8. Attach a share device from secondary privately In primary-secondary process model, we assume ethernet devices are shared by default. that means attach or detach a device on any process will broadcast to all other processes through mp channel then device information will be synchronized on all processes. Any failure during attaching process will cause inconsistent status between processes, so proper rollback action should be considered. Also, it is not safe to detach a share device when other process still use it, so a handshake mechanism is introduced. Scenario for Case 1, 2: attach device from primary a) primary attach the new device if failed goto h). b) primary send attach sync request to all secondary. c) secondary receive request and attach device and send reply. d) primary check the reply if all success go to i). e) primary send attach rollback sync request to all secondary. f) secondary receive the request and detach device and send reply. g) primary receive the reply and detach device as rollback action. h) attach fail i) attach success detach device from primary a) primary perform pre-detach check, if device is locked, goto i). b) primary send pre-detach sync request to all secondary. c) secondary perform pre-detach check and send reply. d) primary check the reply if any fail goto i). e) primary send detach sync request to all secondary f) secondary detach the device and send reply (assume no fail) g) primary detach the device. h) detach success i) detach failed Scenario for case 3, 4: attach device from secondary: a) seconary send asycn request to primary and wait on a condition which will be released by matched response from primary. b) primary receive the request and attach the new device if failed goto i). c) primary forward attach request to all secondary as async request (because this in mp thread context, use sync request will deadlock, same reason for all following async request.) d) secondary receive request and attach device and send reply. e) primary check the reply if all success go to j). f) primary send attach rollback async request to all secondary. g) secondary receive the request and detach device and send reply. h) primary receive the reply and detach device as rollback action. i) send fail response to secondary, goto k). j) send success response to secondary. k) secondary process receive response and return. detach device from secondary: a) secondary send async request to primary and wait on a condition which will be released by matched response from primary. b) primary receive the request and perform pre-detach check, if device is locked, goto j). c) primary send pre-detach async request to all secondary. d) secondary perform pre-detach check and send reply. e) primary check the reply if any fail goto j). f) primary send detach async request to all secondary g) secondary detach the device and send reply h) primary detach the device. i) send success response to secondary, goto k). j) send fail response to secondary. k) secondary process receive response and return. Case 5, 6: Secondary process can attach private device which only visible to itself, in this case no IPC is involved, primary process is not allowed to have private device so far. Case 7, 8: Secondary process can also temporally to detach a share device "privately" then attach it back later, this action also not impact other processes. APIs chenages: ============== rte_eth_dev_attach and rte_eth_dev_attach are extended to support share device attach/detach in primary-secondary process model, it will be called in case 1,2,3,4. New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are introduced to cover case 5,6,7,8, this API can only be invoked in secondary process. New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let application lock or unlock on specific ethdev, a locked device can't be detached. This help applicaiton to prevent unexpected device detaching, especially in multi-process envrionment. Aslo the new API let application to register a callback function which will be invoked before a device is going to be detached, the return value of the function will decide if device will continue be detached or not, this support application to do condition check at runtime. PMD Impact: =========== Currently device removing is not handled well in secondary process on most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up primary process since it reset all shared data. So we introduced new API rte_eth_dev_release_port_local which only reset ethdev's state to unsued but not touch shared data so other process will not be impacted. Since not all device driver is target to support primary-secondary process model, so the patch set only fix this on all Intel devices and vdev, it can be refereneced by other driver when equevalent fix is required Limitation: =========== 1. The solution does not cover the case that primary process exit while secondary processes still be active. Though this is not a typial use case, but if this happens: a. secondary process can't attach / detach any shared device since no primary exist. b. secondary process still can attach / detach private device. c. secondary process still can detach a share device privately but may not attach it back, that ethdev slot will become zombie slot. 2. So for, for PCI bus, case 5,6 is not supported. PCI bus scan/probe mechanism can be improved to support attach private device on secondary process, but this is not the scope of this patchset. Example: ======== The patchset also contains a example to demonstrate device management in multi-process model, below are detail instructions. /* start sample code as primary then secondary */ ./devmgm_mp --proc-type=auto Command Line Example: >help >list /* attach a af_packet vdev */ >attach net_af_packet,iface=eth0 /* detach port 0 */ >detach 0 /* attach a private af_packet vdev (secondary process only)*/ >attachp net_af_packet,iface=eth0 /* detach a private device (secondary process only) */ >detachp 0 /* lock port 0 */ >lock 0 /* unlock port 0 */ >unlock 0 Qi Zhang (22): eal: introduce one device scan bus/vdev: enable one device scan ethdev: add function to release port in local process ethdev: enable hotplug on multi-process ethdev: introduce device lock ethdev: support attach or detach share device from secondary net/i40e: enable port detach on secondary process net/ixgbe: enable port detach on secondary process net/e1000: enable port detach on secondary process net/igb: enable port detach on secondary process net/fm10k: enable port detach on secondary process net/af_packet: enable port detach on secondary process net/bonding: enable port detach on secondary process net/failsafe: enable port detach on secondary process net/kni: enable port detach on secondary process net/null: enable port detach on secondary process net/octeontx: enable port detach on secondary process net/pcap: enable port detach on secondary process net/softnic: enable port detach on secondary process net/tap: enable port detach on secondary process net/vhost: enable port detach on secondary process examples/devmgm_mp: add simple device management sample drivers/bus/vdev/vdev.c | 30 ++ drivers/net/af_packet/rte_eth_af_packet.c | 11 + drivers/net/bonding/rte_eth_bond_pmd.c | 11 + drivers/net/e1000/em_ethdev.c | 9 + drivers/net/e1000/igb_ethdev.c | 9 + drivers/net/failsafe/failsafe.c | 16 + drivers/net/fm10k/fm10k_ethdev.c | 9 + drivers/net/i40e/i40e_ethdev.c | 2 + drivers/net/i40e/i40e_ethdev_vf.c | 9 + drivers/net/ixgbe/ixgbe_ethdev.c | 12 + drivers/net/kni/rte_eth_kni.c | 11 + drivers/net/null/rte_eth_null.c | 16 +- drivers/net/octeontx/octeontx_ethdev.c | 16 + drivers/net/pcap/rte_eth_pcap.c | 15 +- drivers/net/softnic/rte_eth_softnic.c | 19 +- drivers/net/tap/rte_eth_tap.c | 17 +- drivers/net/vhost/rte_eth_vhost.c | 11 + examples/devmgm_mp/Makefile | 64 +++ examples/devmgm_mp/commands.c | 381 +++++++++++++++++ examples/devmgm_mp/commands.h | 10 + examples/devmgm_mp/main.c | 39 ++ examples/devmgm_mp/meson.build | 11 + lib/librte_eal/common/eal_common_dev.c | 17 +- lib/librte_eal/common/eal_private.h | 8 + lib/librte_eal/common/include/rte_bus.h | 16 + lib/librte_eal/linuxapp/eal/eal.c | 7 + lib/librte_ethdev/Makefile | 2 + lib/librte_ethdev/ethdev_lock.c | 139 +++++++ lib/librte_ethdev/ethdev_lock.h | 31 ++ lib/librte_ethdev/ethdev_mp.c | 662 ++++++++++++++++++++++++++++++ lib/librte_ethdev/ethdev_mp.h | 45 ++ lib/librte_ethdev/ethdev_private.h | 39 ++ lib/librte_ethdev/meson.build | 2 + lib/librte_ethdev/rte_ethdev.c | 266 +++++++++++- lib/librte_ethdev/rte_ethdev.h | 169 ++++++++ lib/librte_ethdev/rte_ethdev_core.h | 5 + lib/librte_ethdev/rte_ethdev_driver.h | 13 + 37 files changed, 2118 insertions(+), 31 deletions(-) create mode 100644 examples/devmgm_mp/Makefile create mode 100644 examples/devmgm_mp/commands.c create mode 100644 examples/devmgm_mp/commands.h create mode 100644 examples/devmgm_mp/main.c create mode 100644 examples/devmgm_mp/meson.build create mode 100644 lib/librte_ethdev/ethdev_lock.c create mode 100644 lib/librte_ethdev/ethdev_lock.h create mode 100644 lib/librte_ethdev/ethdev_mp.c create mode 100644 lib/librte_ethdev/ethdev_mp.h create mode 100644 lib/librte_ethdev/ethdev_private.h