From patchwork Wed Aug 8 07:00:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zhang X-Patchwork-Id: 43625 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4EC762BAC; Wed, 8 Aug 2018 09:00:10 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id E130F100C for ; Wed, 8 Aug 2018 09:00:06 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Aug 2018 00:00:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,456,1526367600"; d="scan'208";a="79500573" Received: from dpdk51.sh.intel.com ([10.67.110.190]) by fmsmga001.fm.intel.com with ESMTP; 08 Aug 2018 00:00:00 -0700 From: Qi Zhang To: thomas@monjalon.net, konstantin.ananyev@intel.com, declan.doherty@intel.com, ferruh.yigit@intel.com Cc: dev@dpdk.org, benjamin.h.shelton@intel.com, narender.vangati@intel.com, beilei.xing@intel.com, wenzhuo.lu@intel.com, 0000-cover-letter.patch@dpdk.org, Qi Zhang Date: Wed, 8 Aug 2018 15:00:42 +0800 Message-Id: <20180808070045.13334-1-qi.z.zhang@intel.com> X-Mailer: git-send-email 2.13.6 Subject: [dpdk-dev] [RFC 1/4] ethdev: claim device reset as async X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" rte_eth_dev_reset should be implemented in an async way since it is possible be invoked in interrupt thread and sometimes to reset a device need to wait for some dependency, for example, a VF expects for PF ready, or a NIC function as part of a SOC wait for the whole system reset complete, all these time consuming task will block the the interrupt thread. The patch claims rte_eth_dev_reset is an async function and introduce a new event RTE_ETH_EVENT_RESET_COMPLETE. PMD should raise this event when finish reset in background. The applicaiton should always wait for this event before continue to configure and restart the device. Signed-off-by: Qi Zhang --- lib/librte_ethdev/rte_ethdev.h | 48 ++++++++++++++++++++++++++---------------- 1 file changed, 30 insertions(+), 18 deletions(-) diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h index 7070e9ab4..541b5161d 100644 --- a/lib/librte_ethdev/rte_ethdev.h +++ b/lib/librte_ethdev/rte_ethdev.h @@ -1814,21 +1814,34 @@ void rte_eth_dev_close(uint16_t port_id); * RTE_ETH_EVENT_INTR_RESET event is detected, but can also use it to start * a port reset in other circumstances. * - * When this function is called, it first stops the port and then calls the - * PMD specific dev_uninit( ) and dev_init( ) to return the port to initial - * state, in which no Tx and Rx queues are setup, as if the port has been - * reset and not started. The port keeps the port id it had before the - * function call. - * - * After calling rte_eth_dev_reset( ), the application should use - * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ), - * rte_eth_tx_queue_setup( ), and rte_eth_dev_start( ) - * to reconfigure the device as appropriate. - * - * Note: To avoid unexpected behavior, the application should stop calling - * Tx and Rx functions before calling rte_eth_dev_reset( ). For thread - * safety, all these controlling functions should be called from the same - * thread. + * @note + * Device reset may have the dependency, for example, a VF reset expects + * PF ready, or a NIC function as a part of a SOC need to wait for other + * parts of the system be ready, these are time-consuming tasks and will + * block current thread. + * + * So we claimed rte_eth_dev_reset as an async API, that makes things easy + * for an application that what to reset the device from the interrupt + * thread since typically a RTE_ETH_EVENT_INTR_RESET handler is invoked in + * interrupt thread. + * + * PMD is responsrible to implement ops->dev_reset in an async way, it can + * offload the whole task into a separate thread, or maybe just pending on + * hardware interrupt as reset dependency ready or start a timely alarm + * to poll register status as a background daemon. PMD is also responsible + * to raise the RTE_ETH_EVENT_RESET_COMPLETE event to notify the application + * when reset is complete. + * + * Application should not assume device reset is finished after + * rte_eth_dev_reset return, it should always wait for a + * RTE_ETH_EVENT_RESET_COMPLETE event and check the reset result. + * If reset success, application should call rte_eth_dev_configure( ), + * rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ), + * and rte_eth_dev_start( ) to reconfigure the device as appropriate. + * + * @Note + * To avoid unexpected behavior, the application should stop calling + * Tx and Rx functions before calling rte_eth_dev_reset( ). * * @param port_id * The port identifier of the Ethernet device. @@ -1838,9 +1851,6 @@ void rte_eth_dev_close(uint16_t port_id); * - (-EINVAL) if port identifier is invalid. * - (-ENOTSUP) if hardware doesn't support this function. * - (-EPERM) if not ran from the primary process. - * - (-EIO) if re-initialisation failed or device is removed. - * - (-ENOMEM) if the reset failed due to OOM. - * - (-EAGAIN) if the reset temporarily failed and should be retried later. */ int rte_eth_dev_reset(uint16_t port_id); @@ -2574,6 +2584,8 @@ enum rte_eth_event_type { /**< queue state event (enabled/disabled) */ RTE_ETH_EVENT_INTR_RESET, /**< reset interrupt event, sent to VF on PF reset */ + RTE_ETH_EVENT_RESET_COMPLETE, + /**< inform application that reset is completed */ RTE_ETH_EVENT_VF_MBOX, /**< message from the VF received by PF */ RTE_ETH_EVENT_MACSEC, /**< MACsec offload related event */ RTE_ETH_EVENT_INTR_RMV, /**< device removal event */ From patchwork Wed Aug 8 07:00:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zhang X-Patchwork-Id: 43624 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 9F5FB25D9; Wed, 8 Aug 2018 09:00:08 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id 1211E100C for ; Wed, 8 Aug 2018 09:00:05 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Aug 2018 00:00:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,456,1526367600"; d="scan'208";a="79500628" Received: from dpdk51.sh.intel.com ([10.67.110.190]) by fmsmga001.fm.intel.com with ESMTP; 08 Aug 2018 00:00:02 -0700 From: Qi Zhang To: thomas@monjalon.net, konstantin.ananyev@intel.com, declan.doherty@intel.com, ferruh.yigit@intel.com Cc: dev@dpdk.org, benjamin.h.shelton@intel.com, narender.vangati@intel.com, beilei.xing@intel.com, wenzhuo.lu@intel.com, 0000-cover-letter.patch@dpdk.org, Qi Zhang Date: Wed, 8 Aug 2018 15:00:43 +0800 Message-Id: <20180808070045.13334-2-qi.z.zhang@intel.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20180808070045.13334-1-qi.z.zhang@intel.com> References: <20180808070045.13334-1-qi.z.zhang@intel.com> Subject: [dpdk-dev] [RFC 2/4] net/i40e: enable async device reset X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Handle device reset in a separate thread and raise RTE_ETH_EVENT_RESET_COMPLETE event when it is done. Signed-off-by: Qi Zhang --- drivers/net/i40e/i40e_ethdev.c | 28 +++++++++++++++++++++++----- drivers/net/i40e/i40e_ethdev_vf.c | 28 +++++++++++++++++++++++----- 2 files changed, 46 insertions(+), 10 deletions(-) diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c index 85a6a867f..611570159 100644 --- a/drivers/net/i40e/i40e_ethdev.c +++ b/drivers/net/i40e/i40e_ethdev.c @@ -2439,12 +2439,29 @@ i40e_dev_close(struct rte_eth_dev *dev) I40E_WRITE_FLUSH(hw); } +static void * +do_dev_reset(void *args) +{ + struct rte_eth_dev *dev = args; + int ret; + + ret = eth_i40e_dev_uninit(dev); + if (!ret) + ret = eth_i40e_dev_init(dev, NULL); + + _rte_eth_dev_callback_process(dev, + RTE_ETH_EVENT_RESET_COMPLETE, + &ret); + return NULL; +} + /* * Reset PF device only to re-initialize resources in PMD layer */ static int i40e_dev_reset(struct rte_eth_dev *dev) { + pthread_t tid; int ret; /* When a DPDK PMD PF begin to reset PF port, it should notify all @@ -2456,11 +2473,12 @@ i40e_dev_reset(struct rte_eth_dev *dev) if (dev->data->sriov.active) return -ENOTSUP; - ret = eth_i40e_dev_uninit(dev); - if (ret) - return ret; - - ret = eth_i40e_dev_init(dev, NULL); + /** + * Since dev_reset should be implemented as async, do + * reset in a separate thread. + */ + ret = rte_ctrl_thread_create(&tid, "i40e_dev_reset", + NULL, do_dev_reset, dev); return ret; } diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c index 001c301b9..e10e81792 100644 --- a/drivers/net/i40e/i40e_ethdev_vf.c +++ b/drivers/net/i40e/i40e_ethdev_vf.c @@ -2284,19 +2284,37 @@ i40evf_dev_close(struct rte_eth_dev *dev) i40evf_disable_irq0(hw); } +static void * +do_dev_reset(void *args) +{ + struct rte_eth_dev *dev = args; + int ret; + + ret = i40evf_dev_uninit(dev); + if (!ret) + ret = i40evf_dev_init(dev); + + _rte_eth_dev_callback_process(dev, + RTE_ETH_EVENT_RESET_COMPLETE, + &ret); + return NULL; +} + /* * Reset VF device only to re-initialize resources in PMD layer */ static int i40evf_dev_reset(struct rte_eth_dev *dev) { + pthread_t tid; int ret; - ret = i40evf_dev_uninit(dev); - if (ret) - return ret; - - ret = i40evf_dev_init(dev); + /** + * Since dev_reset should be implemented as async, do + * reset in a separate thread. + */ + ret = rte_ctrl_thread_create(&tid, "i40evf_dev_reset", + NULL, do_dev_reset, dev); return ret; } From patchwork Wed Aug 8 07:00:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zhang X-Patchwork-Id: 43626 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0659F2BF1; Wed, 8 Aug 2018 09:00:12 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id E5D2D100C for ; Wed, 8 Aug 2018 09:00:07 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Aug 2018 00:00:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,456,1526367600"; d="scan'208";a="79500671" Received: from dpdk51.sh.intel.com ([10.67.110.190]) by fmsmga001.fm.intel.com with ESMTP; 08 Aug 2018 00:00:04 -0700 From: Qi Zhang To: thomas@monjalon.net, konstantin.ananyev@intel.com, declan.doherty@intel.com, ferruh.yigit@intel.com Cc: dev@dpdk.org, benjamin.h.shelton@intel.com, narender.vangati@intel.com, beilei.xing@intel.com, wenzhuo.lu@intel.com, 0000-cover-letter.patch@dpdk.org, Qi Zhang Date: Wed, 8 Aug 2018 15:00:44 +0800 Message-Id: <20180808070045.13334-3-qi.z.zhang@intel.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20180808070045.13334-1-qi.z.zhang@intel.com> References: <20180808070045.13334-1-qi.z.zhang@intel.com> Subject: [dpdk-dev] [RFC 3/4] net/ixgbe: enable async device reset X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Handle device reset in a separate thread and raise RTE_ETH_EVENT_RESET_COMPLETE event when it is done. Signed-off-by: Qi Zhang --- drivers/net/ixgbe/ixgbe_ethdev.c | 48 +++++++++++++++++++++++++++++++--------- 1 file changed, 38 insertions(+), 10 deletions(-) diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c index 26b192737..6e2d6fc1a 100644 --- a/drivers/net/ixgbe/ixgbe_ethdev.c +++ b/drivers/net/ixgbe/ixgbe_ethdev.c @@ -2923,12 +2923,29 @@ ixgbe_dev_close(struct rte_eth_dev *dev) ixgbe_set_rar(hw, 0, hw->mac.addr, 0, IXGBE_RAH_AV); } +static void * +do_dev_reset(void *args) +{ + struct rte_eth_dev *dev = args; + int ret; + + ret = eth_ixgbe_dev_uninit(dev); + if (!ret) + ret = eth_ixgbe_dev_init(dev, NULL); + + _rte_eth_dev_callback_process(dev, + RTE_ETH_EVENT_RESET_COMPLETE, + &ret); + return NULL; +} + /* * Reset PF device. */ static int ixgbe_dev_reset(struct rte_eth_dev *dev) { + pthread_t tid; int ret; /* When a DPDK PMD PF begin to reset PF port, it should notify all @@ -2940,11 +2957,8 @@ ixgbe_dev_reset(struct rte_eth_dev *dev) if (dev->data->sriov.active) return -ENOTSUP; - ret = eth_ixgbe_dev_uninit(dev); - if (ret) - return ret; - - ret = eth_ixgbe_dev_init(dev, NULL); + ret = rte_ctrl_thread_create(&tid, "ixgbe_dev_reset", + NULL, do_dev_reset, dev); return ret; } @@ -5173,19 +5187,33 @@ ixgbevf_dev_close(struct rte_eth_dev *dev) ixgbevf_remove_mac_addr(dev, 0); } +static void * +do_dev_vf_reset(void *args) +{ + struct rte_eth_dev *dev = args; + int ret; + + ret = eth_ixgbevf_dev_uninit(dev); + if (!ret) + ret = eth_ixgbevf_dev_init(dev); + + _rte_eth_dev_callback_process(dev, + RTE_ETH_EVENT_RESET_COMPLETE, + &ret); + return NULL; +} + /* * Reset VF device */ static int ixgbevf_dev_reset(struct rte_eth_dev *dev) { + pthread_t tid; int ret; - ret = eth_ixgbevf_dev_uninit(dev); - if (ret) - return ret; - - ret = eth_ixgbevf_dev_init(dev); + ret = rte_ctrl_thread_create(&tid, "ixgbevf_dev_reset", + NULL, do_dev_vf_reset, dev); return ret; } From patchwork Wed Aug 8 07:00:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zhang X-Patchwork-Id: 43627 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3D14E2C39; Wed, 8 Aug 2018 09:00:13 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id 7F0CD100C for ; Wed, 8 Aug 2018 09:00:08 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Aug 2018 00:00:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,456,1526367600"; d="scan'208";a="79500694" Received: from dpdk51.sh.intel.com ([10.67.110.190]) by fmsmga001.fm.intel.com with ESMTP; 08 Aug 2018 00:00:06 -0700 From: Qi Zhang To: thomas@monjalon.net, konstantin.ananyev@intel.com, declan.doherty@intel.com, ferruh.yigit@intel.com Cc: dev@dpdk.org, benjamin.h.shelton@intel.com, narender.vangati@intel.com, beilei.xing@intel.com, wenzhuo.lu@intel.com, 0000-cover-letter.patch@dpdk.org, Qi Zhang Date: Wed, 8 Aug 2018 15:00:45 +0800 Message-Id: <20180808070045.13334-4-qi.z.zhang@intel.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20180808070045.13334-1-qi.z.zhang@intel.com> References: <20180808070045.13334-1-qi.z.zhang@intel.com> Subject: [dpdk-dev] [RFC 4/4] testpmd: enable async device reset X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" rte_eth_dev_reset is claimed as an async API, so re-work on the device reset handling. Signed-off-by: Qi Zhang --- app/test-pmd/testpmd.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index ee48db2a3..24d5c5d9c 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -1918,6 +1918,59 @@ close_port(portid_t pid) printf("Done\n"); } +static pthread_mutex_t dev_reset_lock; +static pthread_cond_t dev_reset_cond; +static int reset_status; + +static int +on_reset_complete(__rte_unused uint16_t port_id, + __rte_unused enum rte_eth_event_type event, + __rte_unused void *cb_arg, + void *ret_param) +{ + RTE_ASSERT(event == RTE_ETH_EVENT_RESET_COMPLETE); + + pthread_cond_broadcast(&dev_reset_cond); + + reset_status = *(int *)ret_param; + return 0; +} + +static int +do_dev_reset_sync(portid_t pid) +{ + int ret; + + pthread_mutex_init(&dev_reset_lock, NULL); + pthread_cond_init(&dev_reset_cond, NULL); + + ret = rte_eth_dev_callback_register(pid, + RTE_ETH_EVENT_RESET_COMPLETE, + on_reset_complete, NULL); + + if (ret) { + printf("Fail to reigster callback function\n"); + return ret; + } + + ret = rte_eth_dev_reset(pid); + if (ret) + goto finish; + + pthread_mutex_lock(&dev_reset_lock); + pthread_cond_wait(&dev_reset_cond, &dev_reset_lock); + pthread_mutex_unlock(&dev_reset_lock); + + ret = reset_status; + +finish: + rte_eth_dev_callback_unregister(pid, + RTE_ETH_EVENT_RESET_COMPLETE, + on_reset_complete, NULL); + + return ret; +} + void reset_port(portid_t pid) { @@ -1946,7 +1999,7 @@ reset_port(portid_t pid) continue; } - diag = rte_eth_dev_reset(pi); + diag = do_dev_reset_sync(pi); if (diag == 0) { port = &ports[pi]; port->need_reconfig = 1;