From patchwork Fri Aug 11 09:40:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Shiyang He X-Patchwork-Id: 130105 X-Patchwork-Delegate: qi.z.zhang@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9370743028; Fri, 11 Aug 2023 04:13:50 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8159F40E03; Fri, 11 Aug 2023 04:13:50 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.100]) by mails.dpdk.org (Postfix) with ESMTP id AAD8740DFB for ; Fri, 11 Aug 2023 04:13:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691720027; x=1723256027; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=JxtCqvLHjaF3ukbJ6dBAblYSonkyQUwrlq3954mHm58=; b=U/613hN2H57HzzcAS9Oh9wllb/ii0yAjt2PFp9XH+DbwNoxjw294JI06 TxWt04pTGpT5MsdX2flhnqhR+dE+sAOnxr2jtBG19UDF6gHIcno4iE0ai +EV4sJRc/NrPC3XyUzb0g2LeEjzRx1gocIMqNnf4yi4JkHN8VGPN6z/Ll DKLaaaKb9S33zV7xfeZWaTeJGNuJAZOzAEgSGOrPmAcpuBur6Nc5G5U5p FMdSdLmTaQK76s5RU02ymw8wJsWohhRvyIKMR1X3Arg43iCQJYuKcjn/v PCCBxnHCcIZTTZq/ETSEQpEQITRWcisYSGr7fhGoWR4h4wgNUoW2tH7tS A==; X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="437911428" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="437911428" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:13:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="735617512" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="735617512" Received: from unknown (HELO root..) ([10.239.252.115]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:13:43 -0700 From: Shiyang He To: dev@dpdk.org Cc: yidingx.zhou@intel.com, Shiyang He , Qiming Yang , Wenjun Wu , Simei Su , Yuying Zhang , Beilei Xing , Jingjing Wu Subject: [PATCH v2] net/iavf: add devargs to enable vf auto-reset Date: Fri, 11 Aug 2023 09:40:43 +0000 Message-Id: <20230811094043.200995-1-shiyangx.he@intel.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230801094815.4028257-1-shiyangx.he@intel.com> References: <20230801094815.4028257-1-shiyangx.he@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Originally, the iavf PMD does not perform special actions when it receives a PF-to-VF reset event, resulting in vf being offline and unavailable. This patch enables vf auto-reset by setting 'watchdog_period' devargs to true. The iavf PMD will perform an automatic reset to bring the vf back online when it receives a PF-to-VF event. v2: using event handler to handle reset Signed-off-by: Shiyang He --- doc/guides/nics/intel_vf.rst | 3 + doc/guides/rel_notes/release_23_11.rst | 3 + drivers/net/iavf/iavf.h | 31 +++++ drivers/net/iavf/iavf_ethdev.c | 177 ++++++++++++++++++++++++- drivers/net/iavf/iavf_rxtx.c | 25 ++++ drivers/net/iavf/iavf_rxtx.h | 1 + drivers/net/iavf/iavf_vchnl.c | 11 +- 7 files changed, 248 insertions(+), 3 deletions(-) diff --git a/doc/guides/nics/intel_vf.rst b/doc/guides/nics/intel_vf.rst index d365dbc185..c0acd2a7f5 100644 --- a/doc/guides/nics/intel_vf.rst +++ b/doc/guides/nics/intel_vf.rst @@ -101,6 +101,9 @@ For more detail on SR-IOV, please refer to the following documents: Set ``devargs`` parameter ``watchdog_period`` to adjust the watchdog period in microseconds, or set it to 0 to disable the watchdog, for example, ``-a 18:01.0,watchdog_period=5000`` or ``-a 18:01.0,watchdog_period=0``. + Enable vf auto-reset by setting the ``devargs`` parameter like ``-a 18:01.0,enable_auto_reset=1`` when IAVF is backed + by an Intel® E810 device or an Intel® 700 Series Ethernet device. + The PCIE host-interface of Intel Ethernet Switch FM10000 Series VF infrastructure ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/doc/guides/rel_notes/release_23_11.rst b/doc/guides/rel_notes/release_23_11.rst index 4411bb32c1..80e2d7672f 100644 --- a/doc/guides/rel_notes/release_23_11.rst +++ b/doc/guides/rel_notes/release_23_11.rst @@ -72,6 +72,9 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Updated Intel iavf driver.** + + Added support for iavf auto-reset. Removed Items ------------- diff --git a/drivers/net/iavf/iavf.h b/drivers/net/iavf/iavf.h index 98861e4242..e78bcda962 100644 --- a/drivers/net/iavf/iavf.h +++ b/drivers/net/iavf/iavf.h @@ -95,6 +95,24 @@ #define IAVF_L2TPV2_FLAGS_LEN 0x4000 +#ifndef DIV_ROUND_UP +#define DIV_ROUND_UP(n, d) ( \ +{ \ + const typeof(d) __d = d; \ + (((n) + (__d) - 1) / (__d));\ +} \ +) +#endif +#ifndef DELAY +#define DELAY(x) rte_delay_us(x) +#endif +#ifndef msleep +#define msleep(x) DELAY(1000 * (x)) +#endif +#ifndef usleep_range +#define usleep_range(min, max) msleep(DIV_ROUND_UP(min, 1000)) +#endif + struct iavf_adapter; struct iavf_rx_queue; struct iavf_tx_queue; @@ -305,6 +323,7 @@ struct iavf_devargs { uint8_t proto_xtr[IAVF_MAX_QUEUE_NUM]; uint16_t quanta_size; uint32_t watchdog_period; + uint8_t enable_auto_reset; }; struct iavf_security_ctx; @@ -424,8 +443,19 @@ _atomic_set_async_response_cmd(struct iavf_info *vf, enum virtchnl_ops ops) return !ret; } + +static inline bool +iavf_is_reset(struct iavf_hw *hw) +{ + return !(IAVF_READ_REG(hw, IAVF_VF_ARQLEN1) & + IAVF_VF_ARQLEN1_ARQENABLE_MASK); +} + int iavf_check_api_version(struct iavf_adapter *adapter); int iavf_get_vf_resource(struct iavf_adapter *adapter); +void iavf_dev_event_post(struct rte_eth_dev *dev, + enum rte_eth_event_type event, + void *param, size_t param_alloc_size); void iavf_dev_event_handler_fini(void); int iavf_dev_event_handler_init(void); void iavf_handle_virtchnl_msg(struct rte_eth_dev *dev); @@ -501,4 +531,5 @@ int iavf_flow_sub_check(struct iavf_adapter *adapter, struct iavf_fsub_conf *filter); void iavf_dev_watchdog_enable(struct iavf_adapter *adapter); void iavf_dev_watchdog_disable(struct iavf_adapter *adapter); +int iavf_handle_hw_reset(struct rte_eth_dev *dev); #endif /* _IAVF_ETHDEV_H_ */ diff --git a/drivers/net/iavf/iavf_ethdev.c b/drivers/net/iavf/iavf_ethdev.c index f2fc5a5621..9ac612e12b 100644 --- a/drivers/net/iavf/iavf_ethdev.c +++ b/drivers/net/iavf/iavf_ethdev.c @@ -37,6 +37,7 @@ #define IAVF_PROTO_XTR_ARG "proto_xtr" #define IAVF_QUANTA_SIZE_ARG "quanta_size" #define IAVF_RESET_WATCHDOG_ARG "watchdog_period" +#define IAVF_ENABLE_AUTO_RESET_ARG "enable_auto_reset" uint64_t iavf_timestamp_dynflag; int iavf_timestamp_dynfield_offset = -1; @@ -45,6 +46,7 @@ static const char * const iavf_valid_args[] = { IAVF_PROTO_XTR_ARG, IAVF_QUANTA_SIZE_ARG, IAVF_RESET_WATCHDOG_ARG, + IAVF_ENABLE_AUTO_RESET_ARG, NULL }; @@ -305,8 +307,8 @@ iavf_dev_watchdog(void *cb_arg) adapter->vf.vf_reset = true; adapter->vf.link_up = false; - rte_eth_dev_callback_process(adapter->vf.eth_dev, - RTE_ETH_EVENT_INTR_RESET, NULL); + iavf_dev_event_post(adapter->vf.eth_dev, RTE_ETH_EVENT_INTR_RESET, + NULL, 0); } } @@ -2210,6 +2212,26 @@ parse_u16(__rte_unused const char *key, const char *value, void *args) return 0; } +static int +parse_bool(const char *key, const char *value, void *args) +{ + int *i = (int *)args; + char *end; + int num; + + num = strtoul(value, &end, 10); + + if (num != 0 && num != 1) { + PMD_DRV_LOG(WARNING, "invalid value:\"%s\" for key:\"%s\", " + "value must be 0 or 1", + value, key); + return -1; + } + + *i = num; + return 0; +} + static int iavf_parse_watchdog_period(__rte_unused const char *key, const char *value, void *args) { @@ -2278,6 +2300,11 @@ static int iavf_parse_devargs(struct rte_eth_dev *dev) goto bail; } + ret = rte_kvargs_process(kvlist, IAVF_ENABLE_AUTO_RESET_ARG, + &parse_bool, &ad->devargs.enable_auto_reset); + if (ret) + goto bail; + bail: rte_kvargs_free(kvlist); return ret; @@ -2879,6 +2906,152 @@ iavf_dev_reset(struct rte_eth_dev *dev) return iavf_dev_init(dev); } +static bool +iavf_is_reset_detected(struct iavf_adapter *adapter) +{ + struct iavf_hw *hw = &adapter->hw; + int i; + + /* poll until we see the reset actually happen */ + for (i = 0; i < IAVF_RESET_WAIT_CNT; i++) { + if (iavf_is_reset(hw)) + return true; + usleep_range(5000, 10000); + } + + return false; +} + +static int +iavf_uninit_hw(struct rte_eth_dev *dev, struct iavf_hw *hw) +{ + struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev); + struct rte_intr_handle *intr_handle = dev->intr_handle; + struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); + struct iavf_adapter *adapter = dev->data->dev_private; + + iavf_reset_queues(dev); + + /* Disable the interrupt for Rx */ + rte_intr_efd_disable(intr_handle); + /* Rx interrupt vector mapping free */ + rte_intr_vec_list_free(intr_handle); + + adapter->stopped = 1; + dev->data->dev_started = 0; + + adapter->closed = true; + + /* free iAVF security device context all related resources */ + iavf_security_ctx_destroy(adapter); + + iavf_flow_flush(dev, NULL); + iavf_flow_uninit(adapter); + + if (vf->promisc_unicast_enabled || vf->promisc_multicast_enabled) + iavf_config_promisc(adapter, false, false); + + iavf_shutdown_adminq(hw); + if (vf->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_WB_ON_ITR) { + /* disable uio intr before callback unregister */ + rte_intr_disable(intr_handle); + + /* unregister callback func from eal lib */ + rte_intr_callback_unregister(intr_handle, + iavf_dev_interrupt_handler, dev); + } + iavf_disable_irq0(hw); + + if (vf->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_QOS) + iavf_tm_conf_uninit(dev); + + if (vf->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_RSS_PF) { + if (vf->rss_lut) { + rte_free(vf->rss_lut); + vf->rss_lut = NULL; + } + if (vf->rss_key) { + rte_free(vf->rss_key); + vf->rss_key = NULL; + } + } + + rte_free(vf->vf_res); + vf->vsi_res = NULL; + vf->vf_res = NULL; + + rte_free(vf->aq_resp); + vf->aq_resp = NULL; + + if (vf->vf_reset && !rte_pci_set_bus_master(pci_dev, true)) + vf->vf_reset = false; + + /* disable watchdog */ + iavf_dev_watchdog_disable(adapter); + + return 0; +} + +static int +iavf_reset_hw(struct rte_eth_dev *dev, struct iavf_hw *hw) +{ + iavf_uninit_hw(dev, hw); + + return iavf_dev_init(dev); +} + +/* + * Handle hardware reset + */ +int +iavf_handle_hw_reset(struct rte_eth_dev *dev) +{ + struct iavf_adapter *adapter = dev->data->dev_private; + struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private); + struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); + int ret; + + ret = iavf_is_reset_detected(adapter); + if (!ret) { + PMD_DRV_LOG(ERR, "Did not detect vf reset!\n"); + return ret; + } + + ret = iavf_check_vf_reset_done(hw); + if (ret) { + PMD_DRV_LOG(ERR, "Wait too long for reset done!\n"); + return ret; + } + + ret = iavf_reset_hw(dev, hw); + if (ret) { + PMD_DRV_LOG(ERR, "Failed to reset vf!\n"); + return ret; + } + + /* + * This may return a failure due to the vf may not able to get + * a response from the pf. + */ + ret = iavf_dev_configure(dev); + if (ret) + PMD_DRV_LOG(ERR, "Failed to configure dev!"); + + iavf_dev_xstats_reset(dev); + + /* start the device */ + ret = iavf_dev_start(dev); + if (ret) { + PMD_DRV_LOG(ERR, "Failed to start dev!"); + return ret; + } + + dev->data->dev_started = 1; + vf->vf_reset = false; + + return 0; +} + static int iavf_dcf_cap_check_handler(__rte_unused const char *key, const char *value, __rte_unused void *opaque) diff --git a/drivers/net/iavf/iavf_rxtx.c b/drivers/net/iavf/iavf_rxtx.c index f7df4665d1..a1f556d1ce 100644 --- a/drivers/net/iavf/iavf_rxtx.c +++ b/drivers/net/iavf/iavf_rxtx.c @@ -1094,6 +1094,31 @@ iavf_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid) rte_free(q); } +void +iavf_reset_queues(struct rte_eth_dev *dev) +{ + struct iavf_rx_queue *rxq; + struct iavf_tx_queue *txq; + int i; + + for (i = 0; i < dev->data->nb_tx_queues; i++) { + txq = dev->data->tx_queues[i]; + if (!txq) + continue; + iavf_txq_release_mbufs_ops[txq->rel_mbufs_type].release_mbufs(txq); + reset_tx_queue(txq); + dev->data->tx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED; + } + for (i = 0; i < dev->data->nb_rx_queues; i++) { + rxq = dev->data->rx_queues[i]; + if (!rxq) + continue; + iavf_rxq_release_mbufs_ops[rxq->rel_mbufs_type].release_mbufs(rxq); + reset_rx_queue(rxq); + dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED; + } +} + void iavf_stop_queues(struct rte_eth_dev *dev) { diff --git a/drivers/net/iavf/iavf_rxtx.h b/drivers/net/iavf/iavf_rxtx.h index 605ea3f824..237022338e 100644 --- a/drivers/net/iavf/iavf_rxtx.h +++ b/drivers/net/iavf/iavf_rxtx.h @@ -650,6 +650,7 @@ int iavf_dev_tx_queue_start(struct rte_eth_dev *dev, uint16_t tx_queue_id); int iavf_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id); int iavf_dev_tx_done_cleanup(void *txq, uint32_t free_cnt); void iavf_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid); +void iavf_reset_queues(struct rte_eth_dev *dev); void iavf_stop_queues(struct rte_eth_dev *dev); uint16_t iavf_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); diff --git a/drivers/net/iavf/iavf_vchnl.c b/drivers/net/iavf/iavf_vchnl.c index 524732f67d..dffe91f8c9 100644 --- a/drivers/net/iavf/iavf_vchnl.c +++ b/drivers/net/iavf/iavf_vchnl.c @@ -79,6 +79,15 @@ iavf_dev_event_handle(void *param __rte_unused) struct iavf_event_element *pos, *save_next; TAILQ_FOREACH_SAFE(pos, &pending, next, save_next) { TAILQ_REMOVE(&pending, pos, next); + + struct iavf_adapter *adapter = pos->dev->data->dev_private; + if (pos->event == RTE_ETH_EVENT_INTR_RESET && + adapter->devargs.enable_auto_reset) { + iavf_handle_hw_reset(pos->dev); + rte_free(pos); + continue; + } + rte_eth_dev_callback_process(pos->dev, pos->event, pos->param); rte_free(pos); } @@ -87,7 +96,7 @@ iavf_dev_event_handle(void *param __rte_unused) return 0; } -static void +void iavf_dev_event_post(struct rte_eth_dev *dev, enum rte_eth_event_type event, void *param, size_t param_alloc_size)