From patchwork Fri Oct 20 10:07:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fengchengwen X-Patchwork-Id: 133068 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AF945431BB; Fri, 20 Oct 2023 12:11:03 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F018E4069D; Fri, 20 Oct 2023 12:10:59 +0200 (CEST) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by mails.dpdk.org (Postfix) with ESMTP id CB1E240284 for ; Fri, 20 Oct 2023 12:10:57 +0200 (CEST) Received: from dggpeml100024.china.huawei.com (unknown [172.30.72.55]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4SBgHM48xhzVlmZ; Fri, 20 Oct 2023 18:07:11 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by dggpeml100024.china.huawei.com (7.185.36.115) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Fri, 20 Oct 2023 18:10:54 +0800 From: Chengwen Feng To: , , , , Andrew Rybchenko , Somnath Kotur , Kalesh AP CC: , Subject: [PATCH v2 1/7] ethdev: fix race-condition of proactive error handling mode Date: Fri, 20 Oct 2023 10:07:40 +0000 Message-ID: <20231020100746.31520-2-fengchengwen@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231020100746.31520-1-fengchengwen@huawei.com> References: <20230301030610.49468-1-fengchengwen@huawei.com> <20231020100746.31520-1-fengchengwen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpeml100024.china.huawei.com (7.185.36.115) X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org In the proactive error handling mode, the PMD will set the data path pointers to dummy functions and then try recovery, in this period the application may still invoking data path API. This will introduce a race-condition with data path which may lead to crash [1]. Although the PMD added delay after setting data path pointers to cover the above race-condition, it reduces the probability, but it doesn't solve the problem. To solve the race-condition problem fundamentally, the following requirements are added: 1. The PMD should set the data path pointers to dummy functions after report RTE_ETH_EVENT_ERR_RECOVERING event. 2. The application should stop data path API invocation when process the RTE_ETH_EVENT_ERR_RECOVERING event. 3. The PMD should set the data path pointers to valid functions before report RTE_ETH_EVENT_RECOVERY_SUCCESS event. 4. The application should enable data path API invocation when process the RTE_ETH_EVENT_RECOVERY_SUCCESS event. Also, this patch introduce a driver internal function rte_eth_fp_ops_setup which used as an help function for PMD. [1] http://patchwork.dpdk.org/project/dpdk/patch/20230220060839.1267349-2-ashok.k.kaladi@intel.com/ Fixes: eb0d471a8941 ("ethdev: add proactive error handling mode") Cc: stable@dpdk.org Signed-off-by: Chengwen Feng Acked-by: Konstantin Ananyev Acked-by: Huisong Li --- doc/guides/prog_guide/poll_mode_drv.rst | 20 +++++++--------- lib/ethdev/ethdev_driver.c | 8 +++++++ lib/ethdev/ethdev_driver.h | 10 ++++++++ lib/ethdev/rte_ethdev.h | 32 +++++++++++++++---------- lib/ethdev/version.map | 1 + 5 files changed, 46 insertions(+), 25 deletions(-) diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst index c145a9066c..e380ff135a 100644 --- a/doc/guides/prog_guide/poll_mode_drv.rst +++ b/doc/guides/prog_guide/poll_mode_drv.rst @@ -638,14 +638,9 @@ different from the application invokes recovery in PASSIVE mode, the PMD automatically recovers from error in PROACTIVE mode, and only a small amount of work is required for the application. -During error detection and automatic recovery, -the PMD sets the data path pointers to dummy functions -(which will prevent the crash), -and also make sure the control path operations fail with a return code ``-EBUSY``. - -Because the PMD recovers automatically, -the application can only sense that the data flow is disconnected for a while -and the control API returns an error in this period. +During error detection and automatic recovery, the PMD sets the data path +pointers to dummy functions and also make sure the control path operations +failed with a return code ``-EBUSY``. In order to sense the error happening/recovering, as well as to restore some additional configuration, @@ -653,9 +648,9 @@ three events are available: ``RTE_ETH_EVENT_ERR_RECOVERING`` Notify the application that an error is detected - and the recovery is being started. + and the recovery is about to start. Upon receiving the event, the application should not invoke - any control path function until receiving + any control and data path API until receiving ``RTE_ETH_EVENT_RECOVERY_SUCCESS`` or ``RTE_ETH_EVENT_RECOVERY_FAILED`` event. .. note:: @@ -666,8 +661,9 @@ three events are available: ``RTE_ETH_EVENT_RECOVERY_SUCCESS`` Notify the application that the recovery from error is successful, - the PMD already re-configures the port, - and the effect is the same as a restart operation. + the PMD already re-configures the port. + The application should restore some additional configuration, and then + enable data path API invocation. ``RTE_ETH_EVENT_RECOVERY_FAILED`` Notify the application that the recovery from error failed, diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c index fff4b7b4cd..65ead7b910 100644 --- a/lib/ethdev/ethdev_driver.c +++ b/lib/ethdev/ethdev_driver.c @@ -537,6 +537,14 @@ rte_eth_dma_zone_free(const struct rte_eth_dev *dev, const char *ring_name, return rc; } +void +rte_eth_fp_ops_setup(struct rte_eth_dev *dev) +{ + if (dev == NULL) + return; + eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev); +} + const struct rte_memzone * rte_eth_dma_zone_reserve(const struct rte_eth_dev *dev, const char *ring_name, uint16_t queue_id, size_t size, unsigned int align, diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h index deb23ada18..8567b96f53 100644 --- a/lib/ethdev/ethdev_driver.h +++ b/lib/ethdev/ethdev_driver.h @@ -1636,6 +1636,16 @@ int rte_eth_dma_zone_free(const struct rte_eth_dev *eth_dev, const char *name, uint16_t queue_id); +/** + * @internal + * Setup eth fast-path API to ethdev values. + * + * @param dev + * Pointer to struct rte_eth_dev. + */ +__rte_internal +void rte_eth_fp_ops_setup(struct rte_eth_dev *dev); + /** * @internal * Atomically set the link status for the specific device. diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index 85b9af7a02..dbe2d9c745 100644 --- a/lib/ethdev/rte_ethdev.h +++ b/lib/ethdev/rte_ethdev.h @@ -3994,25 +3994,28 @@ enum rte_eth_event_type { */ RTE_ETH_EVENT_RX_AVAIL_THRESH, /** Port recovering from a hardware or firmware error. - * If PMD supports proactive error recovery, - * it should trigger this event to notify application - * that it detected an error and the recovery is being started. - * Upon receiving the event, the application should not invoke any control path API - * (such as rte_eth_dev_configure/rte_eth_dev_stop...) until receiving - * RTE_ETH_EVENT_RECOVERY_SUCCESS or RTE_ETH_EVENT_RECOVERY_FAILED event. - * The PMD will set the data path pointers to dummy functions, - * and re-set the data path pointers to non-dummy functions - * before reporting RTE_ETH_EVENT_RECOVERY_SUCCESS event. - * It means that the application cannot send or receive any packets - * during this period. + * + * If PMD supports proactive error recovery, it should trigger this + * event to notify application that it detected an error and the + * recovery is about to start. + * + * Upon receiving the event, the application should not invoke any + * control and data path API until receiving + * RTE_ETH_EVENT_RECOVERY_SUCCESS or RTE_ETH_EVENT_RECOVERY_FAILED + * event. + * + * Once this event is reported, the PMD will set the data path pointers + * to dummy functions, and re-set the data path pointers to valid + * functions before reporting RTE_ETH_EVENT_RECOVERY_SUCCESS event. + * * @note Before the PMD reports the recovery result, * the PMD may report the RTE_ETH_EVENT_ERR_RECOVERING event again, * because a larger error may occur during the recovery. */ RTE_ETH_EVENT_ERR_RECOVERING, /** Port recovers successfully from the error. - * The PMD already re-configured the port, - * and the effect is the same as a restart operation. + * + * The PMD already re-configured the port: * a) The following operation will be retained: (alphabetically) * - DCB configuration * - FEC configuration @@ -4039,6 +4042,9 @@ enum rte_eth_event_type { * (@see RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEEP) * c) Any other configuration will not be stored * and will need to be re-configured. + * + * The application should restore some additional configuration + * (see above case b/c), and then enable data path API invocation. */ RTE_ETH_EVENT_RECOVERY_SUCCESS, /** Port recovery failed. diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map index 919ba5b8e6..1e6ee0a6f1 100644 --- a/lib/ethdev/version.map +++ b/lib/ethdev/version.map @@ -338,6 +338,7 @@ INTERNAL { rte_eth_devices; rte_eth_dma_zone_free; rte_eth_dma_zone_reserve; + rte_eth_fp_ops_setup; rte_eth_hairpin_queue_peer_bind; rte_eth_hairpin_queue_peer_unbind; rte_eth_hairpin_queue_peer_update; From patchwork Fri Oct 20 10:07:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fengchengwen X-Patchwork-Id: 133069 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 18BDB431BB; Fri, 20 Oct 2023 12:11:10 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2FC2B40ED0; Fri, 20 Oct 2023 12:11:01 +0200 (CEST) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by mails.dpdk.org (Postfix) with ESMTP id 14F2540697 for ; Fri, 20 Oct 2023 12:10:57 +0200 (CEST) Received: from dggpeml100024.china.huawei.com (unknown [172.30.72.56]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4SBgH25KtpzFr8D; Fri, 20 Oct 2023 18:06:54 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by dggpeml100024.china.huawei.com (7.185.36.115) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Fri, 20 Oct 2023 18:10:55 +0800 From: Chengwen Feng To: , , , , Jie Hai , Yisen Zhuang CC: , , , Subject: [PATCH v2 2/7] net/hns3: replace fp ops config function Date: Fri, 20 Oct 2023 10:07:41 +0000 Message-ID: <20231020100746.31520-3-fengchengwen@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231020100746.31520-1-fengchengwen@huawei.com> References: <20230301030610.49468-1-fengchengwen@huawei.com> <20231020100746.31520-1-fengchengwen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpeml100024.china.huawei.com (7.185.36.115) X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch replace hns3_eth_dev_fp_ops_config() with rte_eth_fp_ops_setup(). Cc: stable@dpdk.org Signed-off-by: Chengwen Feng Acked-by: Dongdong Liu Acked-by: Huisong Li Acked-by: Konstantin Ananyev --- drivers/net/hns3/hns3_rxtx.c | 21 +++------------------ 1 file changed, 3 insertions(+), 18 deletions(-) diff --git a/drivers/net/hns3/hns3_rxtx.c b/drivers/net/hns3/hns3_rxtx.c index f3c3b38c55..f43f1eb9ad 100644 --- a/drivers/net/hns3/hns3_rxtx.c +++ b/drivers/net/hns3/hns3_rxtx.c @@ -4434,21 +4434,6 @@ hns3_trace_rxtx_function(struct rte_eth_dev *dev) rx_mode.info, tx_mode.info); } -static void -hns3_eth_dev_fp_ops_config(const struct rte_eth_dev *dev) -{ - struct rte_eth_fp_ops *fpo = rte_eth_fp_ops; - uint16_t port_id = dev->data->port_id; - - fpo[port_id].rx_pkt_burst = dev->rx_pkt_burst; - fpo[port_id].tx_pkt_burst = dev->tx_pkt_burst; - fpo[port_id].tx_pkt_prepare = dev->tx_pkt_prepare; - fpo[port_id].rx_descriptor_status = dev->rx_descriptor_status; - fpo[port_id].tx_descriptor_status = dev->tx_descriptor_status; - fpo[port_id].rxq.data = dev->data->rx_queues; - fpo[port_id].txq.data = dev->data->tx_queues; -} - void hns3_set_rxtx_function(struct rte_eth_dev *eth_dev) { @@ -4471,7 +4456,7 @@ hns3_set_rxtx_function(struct rte_eth_dev *eth_dev) } hns3_trace_rxtx_function(eth_dev); - hns3_eth_dev_fp_ops_config(eth_dev); + rte_eth_fp_ops_setup(eth_dev); } void @@ -4824,7 +4809,7 @@ hns3_stop_tx_datapath(struct rte_eth_dev *dev) { dev->tx_pkt_burst = rte_eth_pkt_burst_dummy; dev->tx_pkt_prepare = NULL; - hns3_eth_dev_fp_ops_config(dev); + rte_eth_fp_ops_setup(dev); if (rte_eal_process_type() == RTE_PROC_SECONDARY) return; @@ -4841,7 +4826,7 @@ hns3_start_tx_datapath(struct rte_eth_dev *dev) { dev->tx_pkt_burst = hns3_get_tx_function(dev); dev->tx_pkt_prepare = hns3_get_tx_prepare(dev); - hns3_eth_dev_fp_ops_config(dev); + rte_eth_fp_ops_setup(dev); if (rte_eal_process_type() == RTE_PROC_SECONDARY) return; From patchwork Fri Oct 20 10:07:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fengchengwen X-Patchwork-Id: 133067 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BE9D9431BB; Fri, 20 Oct 2023 12:10:58 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A8B7D40697; Fri, 20 Oct 2023 12:10:58 +0200 (CEST) Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by mails.dpdk.org (Postfix) with ESMTP id C694840283 for ; Fri, 20 Oct 2023 12:10:57 +0200 (CEST) Received: from dggpeml100024.china.huawei.com (unknown [172.30.72.54]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4SBgJT1p6YzrT5j; Fri, 20 Oct 2023 18:08:09 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by dggpeml100024.china.huawei.com (7.185.36.115) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Fri, 20 Oct 2023 18:10:55 +0800 From: Chengwen Feng To: , , , , Somnath Kotur , Kalesh AP CC: , , Subject: [PATCH v2 3/7] net/bnxt: fix race-condition when report error recovery Date: Fri, 20 Oct 2023 10:07:42 +0000 Message-ID: <20231020100746.31520-4-fengchengwen@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231020100746.31520-1-fengchengwen@huawei.com> References: <20230301030610.49468-1-fengchengwen@huawei.com> <20231020100746.31520-1-fengchengwen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpeml100024.china.huawei.com (7.185.36.115) X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org If set data path functions to dummy functions before reports error recovering event, there maybe a race-condition with data path threads, this patch fixes it by setting data path functions to dummy functions only after reports such event. Fixes: e11052f3a46f ("net/bnxt: support proactive error handling mode") Cc: stable@dpdk.org Signed-off-by: Chengwen Feng Acked-by: Konstantin Ananyev Acked-by: Ajit Khaparde --- drivers/net/bnxt/bnxt_cpr.c | 13 +++++++------ drivers/net/bnxt/bnxt_ethdev.c | 4 ++-- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/drivers/net/bnxt/bnxt_cpr.c b/drivers/net/bnxt/bnxt_cpr.c index 0733cf4df2..d8947d5b5f 100644 --- a/drivers/net/bnxt/bnxt_cpr.c +++ b/drivers/net/bnxt/bnxt_cpr.c @@ -168,14 +168,9 @@ void bnxt_handle_async_event(struct bnxt *bp, PMD_DRV_LOG(INFO, "Port conn async event\n"); break; case HWRM_ASYNC_EVENT_CMPL_EVENT_ID_RESET_NOTIFY: - /* - * Avoid any rx/tx packet processing during firmware reset - * operation. - */ - bnxt_stop_rxtx(bp->eth_dev); - /* Ignore reset notify async events when stopping the port */ if (!bp->eth_dev->data->dev_started) { + bnxt_stop_rxtx(bp->eth_dev); bp->flags |= BNXT_FLAG_FATAL_ERROR; return; } @@ -184,6 +179,12 @@ void bnxt_handle_async_event(struct bnxt *bp, RTE_ETH_EVENT_ERR_RECOVERING, NULL); + /* + * Avoid any rx/tx packet processing during firmware reset + * operation. + */ + bnxt_stop_rxtx(bp->eth_dev); + pthread_mutex_lock(&bp->err_recovery_lock); event_data = data1; /* timestamp_lo/hi values are in units of 100ms */ diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c index 5c4d96d4b1..003a6eec11 100644 --- a/drivers/net/bnxt/bnxt_ethdev.c +++ b/drivers/net/bnxt/bnxt_ethdev.c @@ -4616,14 +4616,14 @@ static void bnxt_check_fw_health(void *arg) bp->flags |= BNXT_FLAG_FATAL_ERROR; bp->flags |= BNXT_FLAG_FW_RESET; - bnxt_stop_rxtx(bp->eth_dev); - PMD_DRV_LOG(ERR, "Detected FW dead condition\n"); rte_eth_dev_callback_process(bp->eth_dev, RTE_ETH_EVENT_ERR_RECOVERING, NULL); + bnxt_stop_rxtx(bp->eth_dev); + if (bnxt_is_primary_func(bp)) wait_msec = info->primary_func_wait_period; else From patchwork Fri Oct 20 10:07:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fengchengwen X-Patchwork-Id: 133070 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 56C0D431BB; Fri, 20 Oct 2023 12:11:16 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 54AC940EE7; Fri, 20 Oct 2023 12:11:02 +0200 (CEST) Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by mails.dpdk.org (Postfix) with ESMTP id 1A8E34069D for ; Fri, 20 Oct 2023 12:10:57 +0200 (CEST) Received: from dggpeml100024.china.huawei.com (unknown [172.30.72.56]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4SBgHN24qmzVlm8; Fri, 20 Oct 2023 18:07:12 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by dggpeml100024.china.huawei.com (7.185.36.115) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Fri, 20 Oct 2023 18:10:55 +0800 From: Chengwen Feng To: , , , , Somnath Kotur CC: , , , Subject: [PATCH v2 4/7] net/bnxt: use fp ops setup function Date: Fri, 20 Oct 2023 10:07:43 +0000 Message-ID: <20231020100746.31520-5-fengchengwen@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231020100746.31520-1-fengchengwen@huawei.com> References: <20230301030610.49468-1-fengchengwen@huawei.com> <20231020100746.31520-1-fengchengwen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpeml100024.china.huawei.com (7.185.36.115) X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Use rte_eth_fp_ops_setup() instead of directly manipulating rte_eth_fp_ops variable. Cc: stable@dpdk.org Signed-off-by: Chengwen Feng Acked-by: Huisong Li Acked-by: Konstantin Ananyev Acked-by: Ajit Khaparde --- drivers/net/bnxt/bnxt_cpr.c | 5 +---- drivers/net/bnxt/bnxt_ethdev.c | 5 +---- 2 files changed, 2 insertions(+), 8 deletions(-) diff --git a/drivers/net/bnxt/bnxt_cpr.c b/drivers/net/bnxt/bnxt_cpr.c index d8947d5b5f..3a08028331 100644 --- a/drivers/net/bnxt/bnxt_cpr.c +++ b/drivers/net/bnxt/bnxt_cpr.c @@ -416,10 +416,7 @@ void bnxt_stop_rxtx(struct rte_eth_dev *eth_dev) eth_dev->rx_pkt_burst = rte_eth_pkt_burst_dummy; eth_dev->tx_pkt_burst = rte_eth_pkt_burst_dummy; - rte_eth_fp_ops[eth_dev->data->port_id].rx_pkt_burst = - eth_dev->rx_pkt_burst; - rte_eth_fp_ops[eth_dev->data->port_id].tx_pkt_burst = - eth_dev->tx_pkt_burst; + rte_eth_fp_ops_setup(eth_dev); rte_mb(); /* Allow time for threads to exit the real burst functions. */ diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c index 003a6eec11..9d9b9ae8cf 100644 --- a/drivers/net/bnxt/bnxt_ethdev.c +++ b/drivers/net/bnxt/bnxt_ethdev.c @@ -4428,10 +4428,7 @@ static void bnxt_dev_recover(void *arg) if (rc) goto err_start; - rte_eth_fp_ops[bp->eth_dev->data->port_id].rx_pkt_burst = - bp->eth_dev->rx_pkt_burst; - rte_eth_fp_ops[bp->eth_dev->data->port_id].tx_pkt_burst = - bp->eth_dev->tx_pkt_burst; + rte_eth_fp_ops_setup(bp->eth_dev); rte_mb(); PMD_DRV_LOG(INFO, "Port: %u Recovered from FW reset\n", From patchwork Fri Oct 20 10:07:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fengchengwen X-Patchwork-Id: 133071 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 47D83431BB; Fri, 20 Oct 2023 12:11:23 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7A80C410F1; Fri, 20 Oct 2023 12:11:03 +0200 (CEST) Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by mails.dpdk.org (Postfix) with ESMTP id B744040A84 for ; Fri, 20 Oct 2023 12:10:58 +0200 (CEST) Received: from dggpeml100024.china.huawei.com (unknown [172.30.72.55]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4SBgG86jhPzcdQG; Fri, 20 Oct 2023 18:06:08 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by dggpeml100024.china.huawei.com (7.185.36.115) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Fri, 20 Oct 2023 18:10:55 +0800 From: Chengwen Feng To: , , , , Aman Singh , Yuying Zhang CC: , , , Subject: [PATCH v2 5/7] app/testpmd: add error recovery usage demo Date: Fri, 20 Oct 2023 10:07:44 +0000 Message-ID: <20231020100746.31520-6-fengchengwen@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231020100746.31520-1-fengchengwen@huawei.com> References: <20230301030610.49468-1-fengchengwen@huawei.com> <20231020100746.31520-1-fengchengwen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpeml100024.china.huawei.com (7.185.36.115) X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch adds error recovery usage demo which will: 1. stop packet forwarding when the RTE_ETH_EVENT_ERR_RECOVERING event is received. 2. restart packet forwarding when the RTE_ETH_EVENT_RECOVERY_SUCCESS event is received. 3. prompt the ports that fail to recovery and need to be removed when the RTE_ETH_EVENT_RECOVERY_FAILED event is received. In addition, a message is added to the printed information, requiring no command to be executed during the error recovery. Signed-off-by: Chengwen Feng Acked-by: Konstantin Ananyev --- app/test-pmd/testpmd.c | 80 ++++++++++++++++++++++++++++++++++++++++++ app/test-pmd/testpmd.h | 4 ++- 2 files changed, 83 insertions(+), 1 deletion(-) diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 595b77748c..39a25238e5 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -3942,6 +3942,77 @@ rmv_port_callback(void *arg) start_packet_forwarding(0); } +static int need_start_when_recovery_over; + +static bool +has_port_in_err_recovering(void) +{ + struct rte_port *port; + portid_t pid; + + RTE_ETH_FOREACH_DEV(pid) { + port = &ports[pid]; + if (port->err_recovering) + return true; + } + + return false; +} + +static void +err_recovering_callback(portid_t port_id) +{ + if (!has_port_in_err_recovering()) + printf("Please stop executing any commands until recovery result events are received!\n"); + + ports[port_id].err_recovering = 1; + ports[port_id].recover_failed = 0; + + /* To simplify implementation, stop forwarding regardless of whether the port is used. */ + if (!test_done) { + printf("Stop packet forwarding because some ports are in error recovering!\n"); + stop_packet_forwarding(); + need_start_when_recovery_over = 1; + } +} + +static void +recover_success_callback(portid_t port_id) +{ + ports[port_id].err_recovering = 0; + if (has_port_in_err_recovering()) + return; + + if (need_start_when_recovery_over) { + printf("Recovery success! Restart packet forwarding!\n"); + start_packet_forwarding(0); + need_start_when_recovery_over = 0; + } else { + printf("Recovery success!\n"); + } +} + +static void +recover_failed_callback(portid_t port_id) +{ + struct rte_port *port; + portid_t pid; + + ports[port_id].err_recovering = 0; + ports[port_id].recover_failed = 1; + if (has_port_in_err_recovering()) + return; + + need_start_when_recovery_over = 0; + printf("The ports:"); + RTE_ETH_FOREACH_DEV(pid) { + port = &ports[pid]; + if (port->recover_failed) + printf(" %u", pid); + } + printf(" recovery failed! Please remove them!\n"); +} + /* This function is used by the interrupt thread */ static int eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param, @@ -3997,6 +4068,15 @@ eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param, } break; } + case RTE_ETH_EVENT_ERR_RECOVERING: + err_recovering_callback(port_id); + break; + case RTE_ETH_EVENT_RECOVERY_SUCCESS: + recover_success_callback(port_id); + break; + case RTE_ETH_EVENT_RECOVERY_FAILED: + recover_failed_callback(port_id); + break; default: break; } diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h index 09a36b90b8..42782d5a05 100644 --- a/app/test-pmd/testpmd.h +++ b/app/test-pmd/testpmd.h @@ -342,7 +342,9 @@ struct rte_port { uint8_t member_flag : 1, /**< bonding member port */ bond_flag : 1, /**< port is bond device */ fwd_mac_swap : 1, /**< swap packet MAC before forward */ - update_conf : 1; /**< need to update bonding device configuration */ + update_conf : 1, /**< need to update bonding device configuration */ + err_recovering : 1, /**< port is in error recovering */ + recover_failed : 1; /**< port recover failed */ struct port_template *pattern_templ_list; /**< Pattern templates. */ struct port_template *actions_templ_list; /**< Actions templates. */ struct port_table *table_list; /**< Flow tables. */ From patchwork Fri Oct 20 10:07:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fengchengwen X-Patchwork-Id: 133072 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 50C2F431BB; Fri, 20 Oct 2023 12:11:29 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8AFB04111C; Fri, 20 Oct 2023 12:11:04 +0200 (CEST) Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by mails.dpdk.org (Postfix) with ESMTP id 59F5340283 for ; Fri, 20 Oct 2023 12:10:58 +0200 (CEST) Received: from dggpeml100024.china.huawei.com (unknown [172.30.72.53]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4SBgJV5XL1z15NkK; Fri, 20 Oct 2023 18:08:10 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by dggpeml100024.china.huawei.com (7.185.36.115) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Fri, 20 Oct 2023 18:10:55 +0800 From: Chengwen Feng To: , , , , Aman Singh , Yuying Zhang CC: , , , Subject: [PATCH v2 6/7] app/testpmd: extract event handling to event.c Date: Fri, 20 Oct 2023 10:07:45 +0000 Message-ID: <20231020100746.31520-7-fengchengwen@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231020100746.31520-1-fengchengwen@huawei.com> References: <20230301030610.49468-1-fengchengwen@huawei.com> <20231020100746.31520-1-fengchengwen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpeml100024.china.huawei.com (7.185.36.115) X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch extract event handling (including eth-event and dev-event) to a new file 'event.c'. Signed-off-by: Chengwen Feng Acked-by: Huisong Li --- app/test-pmd/event.c | 390 ++++++++++++++++++++++++++++++++++++++ app/test-pmd/meson.build | 1 + app/test-pmd/parameters.c | 36 +--- app/test-pmd/testpmd.c | 327 +------------------------------- app/test-pmd/testpmd.h | 6 + 5 files changed, 407 insertions(+), 353 deletions(-) create mode 100644 app/test-pmd/event.c diff --git a/app/test-pmd/event.c b/app/test-pmd/event.c new file mode 100644 index 0000000000..8393e105d7 --- /dev/null +++ b/app/test-pmd/event.c @@ -0,0 +1,390 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2023 HiSilicon Limited + */ + +#include + +#include +#include +#include +#include +#ifdef RTE_NET_MLX5 +#include "mlx5_testpmd.h" +#endif + +#include "testpmd.h" + +/* Pretty printing of ethdev events */ +static const char * const eth_event_desc[] = { + [RTE_ETH_EVENT_UNKNOWN] = "unknown", + [RTE_ETH_EVENT_INTR_LSC] = "link state change", + [RTE_ETH_EVENT_QUEUE_STATE] = "queue state", + [RTE_ETH_EVENT_INTR_RESET] = "reset", + [RTE_ETH_EVENT_VF_MBOX] = "VF mbox", + [RTE_ETH_EVENT_IPSEC] = "IPsec", + [RTE_ETH_EVENT_MACSEC] = "MACsec", + [RTE_ETH_EVENT_INTR_RMV] = "device removal", + [RTE_ETH_EVENT_NEW] = "device probed", + [RTE_ETH_EVENT_DESTROY] = "device released", + [RTE_ETH_EVENT_FLOW_AGED] = "flow aged", + [RTE_ETH_EVENT_RX_AVAIL_THRESH] = "RxQ available descriptors threshold reached", + [RTE_ETH_EVENT_ERR_RECOVERING] = "error recovering", + [RTE_ETH_EVENT_RECOVERY_SUCCESS] = "error recovery successful", + [RTE_ETH_EVENT_RECOVERY_FAILED] = "error recovery failed", + [RTE_ETH_EVENT_MAX] = NULL, +}; + +/* + * Display or mask ether events + * Default to all events except VF_MBOX + */ +uint32_t event_print_mask = (UINT32_C(1) << RTE_ETH_EVENT_UNKNOWN) | + (UINT32_C(1) << RTE_ETH_EVENT_INTR_LSC) | + (UINT32_C(1) << RTE_ETH_EVENT_QUEUE_STATE) | + (UINT32_C(1) << RTE_ETH_EVENT_INTR_RESET) | + (UINT32_C(1) << RTE_ETH_EVENT_IPSEC) | + (UINT32_C(1) << RTE_ETH_EVENT_MACSEC) | + (UINT32_C(1) << RTE_ETH_EVENT_INTR_RMV) | + (UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED) | + (UINT32_C(1) << RTE_ETH_EVENT_ERR_RECOVERING) | + (UINT32_C(1) << RTE_ETH_EVENT_RECOVERY_SUCCESS) | + (UINT32_C(1) << RTE_ETH_EVENT_RECOVERY_FAILED); + +int +get_event_name_mask(const char *name, uint32_t *mask) +{ + if (!strcmp(name, "unknown")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_UNKNOWN; + else if (!strcmp(name, "intr_lsc")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_INTR_LSC; + else if (!strcmp(name, "queue_state")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_QUEUE_STATE; + else if (!strcmp(name, "intr_reset")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_INTR_RESET; + else if (!strcmp(name, "vf_mbox")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_VF_MBOX; + else if (!strcmp(name, "ipsec")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_IPSEC; + else if (!strcmp(name, "macsec")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_MACSEC; + else if (!strcmp(name, "intr_rmv")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_INTR_RMV; + else if (!strcmp(name, "dev_probed")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_NEW; + else if (!strcmp(name, "dev_released")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_DESTROY; + else if (!strcmp(name, "flow_aged")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED; + else if (!strcmp(name, "err_recovering")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_ERR_RECOVERING; + else if (!strcmp(name, "recovery_success")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_RECOVERY_SUCCESS; + else if (!strcmp(name, "recovery_failed")) + *mask = UINT32_C(1) << RTE_ETH_EVENT_RECOVERY_FAILED; + else if (!strcmp(name, "all")) + *mask = ~UINT32_C(0); + else + return -1; + + return 0; +} + +static void +rmv_port_callback(void *arg) +{ + int need_to_start = 0; + int org_no_link_check = no_link_check; + portid_t port_id = (intptr_t)arg; + struct rte_eth_dev_info dev_info; + int ret; + + RTE_ETH_VALID_PORTID_OR_RET(port_id); + + if (!test_done && port_is_forwarding(port_id)) { + need_to_start = 1; + stop_packet_forwarding(); + } + no_link_check = 1; + stop_port(port_id); + no_link_check = org_no_link_check; + + ret = eth_dev_info_get_print_err(port_id, &dev_info); + if (ret != 0) + TESTPMD_LOG(ERR, + "Failed to get device info for port %d, not detaching\n", + port_id); + else { + struct rte_device *device = dev_info.device; + close_port(port_id); + detach_device(device); /* might be already removed or have more ports */ + } + if (need_to_start) + start_packet_forwarding(0); +} + +static int need_start_when_recovery_over; + +static bool +has_port_in_err_recovering(void) +{ + struct rte_port *port; + portid_t pid; + + RTE_ETH_FOREACH_DEV(pid) { + port = &ports[pid]; + if (port->err_recovering) + return true; + } + + return false; +} + +static void +err_recovering_callback(portid_t port_id) +{ + if (!has_port_in_err_recovering()) + printf("Please stop executing any commands until recovery result events are received!\n"); + + ports[port_id].err_recovering = 1; + ports[port_id].recover_failed = 0; + + /* To simplify implementation, stop forwarding regardless of whether the port is used. */ + if (!test_done) { + printf("Stop packet forwarding because some ports are in error recovering!\n"); + stop_packet_forwarding(); + need_start_when_recovery_over = 1; + } +} + +static void +recover_success_callback(portid_t port_id) +{ + ports[port_id].err_recovering = 0; + if (has_port_in_err_recovering()) + return; + + if (need_start_when_recovery_over) { + printf("Recovery success! Restart packet forwarding!\n"); + start_packet_forwarding(0); + need_start_when_recovery_over = 0; + } else { + printf("Recovery success!\n"); + } +} + +static void +recover_failed_callback(portid_t port_id) +{ + struct rte_port *port; + portid_t pid; + + ports[port_id].err_recovering = 0; + ports[port_id].recover_failed = 1; + if (has_port_in_err_recovering()) + return; + + need_start_when_recovery_over = 0; + printf("The ports:"); + RTE_ETH_FOREACH_DEV(pid) { + port = &ports[pid]; + if (port->recover_failed) + printf(" %u", pid); + } + printf(" recovery failed! Please remove them!\n"); +} + +/* This function is used by the interrupt thread */ +static int +eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param, + void *ret_param) +{ + RTE_SET_USED(param); + RTE_SET_USED(ret_param); + + if (type >= RTE_ETH_EVENT_MAX) { + fprintf(stderr, + "\nPort %" PRIu16 ": %s called upon invalid event %d\n", + port_id, __func__, type); + fflush(stderr); + } else if (event_print_mask & (UINT32_C(1) << type)) { + printf("\nPort %" PRIu16 ": %s event\n", port_id, + eth_event_desc[type]); + fflush(stdout); + } + + switch (type) { + case RTE_ETH_EVENT_NEW: + ports[port_id].need_setup = 1; + ports[port_id].port_status = RTE_PORT_HANDLING; + break; + case RTE_ETH_EVENT_INTR_RMV: + if (port_id_is_invalid(port_id, DISABLED_WARN)) + break; + if (rte_eal_alarm_set(100000, + rmv_port_callback, (void *)(intptr_t)port_id)) + fprintf(stderr, + "Could not set up deferred device removal\n"); + break; + case RTE_ETH_EVENT_DESTROY: + ports[port_id].port_status = RTE_PORT_CLOSED; + printf("Port %u is closed\n", port_id); + break; + case RTE_ETH_EVENT_RX_AVAIL_THRESH: { + uint16_t rxq_id; + int ret; + + /* avail_thresh query API rewinds rxq_id, no need to check max RxQ num */ + for (rxq_id = 0; ; rxq_id++) { + ret = rte_eth_rx_avail_thresh_query(port_id, &rxq_id, + NULL); + if (ret <= 0) + break; + printf("Received avail_thresh event, port: %u, rxq_id: %u\n", + port_id, rxq_id); + +#ifdef RTE_NET_MLX5 + mlx5_test_avail_thresh_event_handler(port_id, rxq_id); +#endif + } + break; + } + case RTE_ETH_EVENT_ERR_RECOVERING: + err_recovering_callback(port_id); + break; + case RTE_ETH_EVENT_RECOVERY_SUCCESS: + recover_success_callback(port_id); + break; + case RTE_ETH_EVENT_RECOVERY_FAILED: + recover_failed_callback(port_id); + break; + default: + break; + } + return 0; +} + +int +register_eth_event_callback(void) +{ + int ret; + enum rte_eth_event_type event; + + for (event = RTE_ETH_EVENT_UNKNOWN; + event < RTE_ETH_EVENT_MAX; event++) { + ret = rte_eth_dev_callback_register(RTE_ETH_ALL, + event, + eth_event_callback, + NULL); + if (ret != 0) { + TESTPMD_LOG(ERR, "Failed to register callback for " + "%s event\n", eth_event_desc[event]); + return -1; + } + } + + return 0; +} + +int +unregister_eth_event_callback(void) +{ + int ret; + enum rte_eth_event_type event; + + for (event = RTE_ETH_EVENT_UNKNOWN; + event < RTE_ETH_EVENT_MAX; event++) { + ret = rte_eth_dev_callback_unregister(RTE_ETH_ALL, + event, + eth_event_callback, + NULL); + if (ret != 0) { + TESTPMD_LOG(ERR, "Failed to unregister callback for " + "%s event\n", eth_event_desc[event]); + return -1; + } + } + + return 0; +} + +/* This function is used by the interrupt thread */ +static void +dev_event_callback(const char *device_name, enum rte_dev_event_type type, + __rte_unused void *arg) +{ + uint16_t port_id; + int ret; + + if (type >= RTE_DEV_EVENT_MAX) { + fprintf(stderr, "%s called upon invalid event %d\n", + __func__, type); + fflush(stderr); + } + + switch (type) { + case RTE_DEV_EVENT_REMOVE: + RTE_LOG(DEBUG, EAL, "The device: %s has been removed!\n", + device_name); + ret = rte_eth_dev_get_port_by_name(device_name, &port_id); + if (ret) { + RTE_LOG(ERR, EAL, "can not get port by device %s!\n", + device_name); + return; + } + /* + * Because the user's callback is invoked in eal interrupt + * callback, the interrupt callback need to be finished before + * it can be unregistered when detaching device. So finish + * callback soon and use a deferred removal to detach device + * is need. It is a workaround, once the device detaching be + * moved into the eal in the future, the deferred removal could + * be deleted. + */ + if (rte_eal_alarm_set(100000, + rmv_port_callback, (void *)(intptr_t)port_id)) + RTE_LOG(ERR, EAL, + "Could not set up deferred device removal\n"); + break; + case RTE_DEV_EVENT_ADD: + RTE_LOG(ERR, EAL, "The device: %s has been added!\n", + device_name); + /* TODO: After finish kernel driver binding, + * begin to attach port. + */ + break; + default: + break; + } +} + +int +register_dev_event_callback(void) +{ + int ret; + + ret = rte_dev_event_callback_register(NULL, + dev_event_callback, NULL); + if (ret != 0) { + RTE_LOG(ERR, EAL, + "fail to register device event callback\n"); + return -1; + } + + return 0; +} + +int +unregister_dev_event_callback(void) +{ + int ret; + + ret = rte_dev_event_callback_unregister(NULL, + dev_event_callback, NULL); + if (ret < 0) { + RTE_LOG(ERR, EAL, + "fail to unregister device event callback.\n"); + return -1; + } + + return 0; +} diff --git a/app/test-pmd/meson.build b/app/test-pmd/meson.build index 719f875be0..b7860f3ab0 100644 --- a/app/test-pmd/meson.build +++ b/app/test-pmd/meson.build @@ -14,6 +14,7 @@ sources = files( 'cmd_flex_item.c', 'config.c', 'csumonly.c', + 'event.c', 'flowgen.c', 'icmpecho.c', 'ieee1588fwd.c', diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c index a9ca58339d..504315da8b 100644 --- a/app/test-pmd/parameters.c +++ b/app/test-pmd/parameters.c @@ -434,45 +434,19 @@ static int parse_event_printing_config(const char *optarg, int enable) { uint32_t mask = 0; + int ret; - if (!strcmp(optarg, "unknown")) - mask = UINT32_C(1) << RTE_ETH_EVENT_UNKNOWN; - else if (!strcmp(optarg, "intr_lsc")) - mask = UINT32_C(1) << RTE_ETH_EVENT_INTR_LSC; - else if (!strcmp(optarg, "queue_state")) - mask = UINT32_C(1) << RTE_ETH_EVENT_QUEUE_STATE; - else if (!strcmp(optarg, "intr_reset")) - mask = UINT32_C(1) << RTE_ETH_EVENT_INTR_RESET; - else if (!strcmp(optarg, "vf_mbox")) - mask = UINT32_C(1) << RTE_ETH_EVENT_VF_MBOX; - else if (!strcmp(optarg, "ipsec")) - mask = UINT32_C(1) << RTE_ETH_EVENT_IPSEC; - else if (!strcmp(optarg, "macsec")) - mask = UINT32_C(1) << RTE_ETH_EVENT_MACSEC; - else if (!strcmp(optarg, "intr_rmv")) - mask = UINT32_C(1) << RTE_ETH_EVENT_INTR_RMV; - else if (!strcmp(optarg, "dev_probed")) - mask = UINT32_C(1) << RTE_ETH_EVENT_NEW; - else if (!strcmp(optarg, "dev_released")) - mask = UINT32_C(1) << RTE_ETH_EVENT_DESTROY; - else if (!strcmp(optarg, "flow_aged")) - mask = UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED; - else if (!strcmp(optarg, "err_recovering")) - mask = UINT32_C(1) << RTE_ETH_EVENT_ERR_RECOVERING; - else if (!strcmp(optarg, "recovery_success")) - mask = UINT32_C(1) << RTE_ETH_EVENT_RECOVERY_SUCCESS; - else if (!strcmp(optarg, "recovery_failed")) - mask = UINT32_C(1) << RTE_ETH_EVENT_RECOVERY_FAILED; - else if (!strcmp(optarg, "all")) - mask = ~UINT32_C(0); - else { + ret = get_event_name_mask(optarg, &mask); + if (ret != 0) { fprintf(stderr, "Invalid event: %s\n", optarg); return -1; } + if (enable) event_print_mask |= mask; else event_print_mask &= ~mask; + return 0; } diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 39a25238e5..3a664fec66 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -435,41 +435,6 @@ uint8_t clear_ptypes = true; /* Hairpin ports configuration mode. */ uint32_t hairpin_mode; -/* Pretty printing of ethdev events */ -static const char * const eth_event_desc[] = { - [RTE_ETH_EVENT_UNKNOWN] = "unknown", - [RTE_ETH_EVENT_INTR_LSC] = "link state change", - [RTE_ETH_EVENT_QUEUE_STATE] = "queue state", - [RTE_ETH_EVENT_INTR_RESET] = "reset", - [RTE_ETH_EVENT_VF_MBOX] = "VF mbox", - [RTE_ETH_EVENT_IPSEC] = "IPsec", - [RTE_ETH_EVENT_MACSEC] = "MACsec", - [RTE_ETH_EVENT_INTR_RMV] = "device removal", - [RTE_ETH_EVENT_NEW] = "device probed", - [RTE_ETH_EVENT_DESTROY] = "device released", - [RTE_ETH_EVENT_FLOW_AGED] = "flow aged", - [RTE_ETH_EVENT_RX_AVAIL_THRESH] = "RxQ available descriptors threshold reached", - [RTE_ETH_EVENT_ERR_RECOVERING] = "error recovering", - [RTE_ETH_EVENT_RECOVERY_SUCCESS] = "error recovery successful", - [RTE_ETH_EVENT_RECOVERY_FAILED] = "error recovery failed", - [RTE_ETH_EVENT_MAX] = NULL, -}; - -/* - * Display or mask ether events - * Default to all events except VF_MBOX - */ -uint32_t event_print_mask = (UINT32_C(1) << RTE_ETH_EVENT_UNKNOWN) | - (UINT32_C(1) << RTE_ETH_EVENT_INTR_LSC) | - (UINT32_C(1) << RTE_ETH_EVENT_QUEUE_STATE) | - (UINT32_C(1) << RTE_ETH_EVENT_INTR_RESET) | - (UINT32_C(1) << RTE_ETH_EVENT_IPSEC) | - (UINT32_C(1) << RTE_ETH_EVENT_MACSEC) | - (UINT32_C(1) << RTE_ETH_EVENT_INTR_RMV) | - (UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED) | - (UINT32_C(1) << RTE_ETH_EVENT_ERR_RECOVERING) | - (UINT32_C(1) << RTE_ETH_EVENT_RECOVERY_SUCCESS) | - (UINT32_C(1) << RTE_ETH_EVENT_RECOVERY_FAILED); /* * Decide if all memory are locked for performance. */ @@ -701,12 +666,6 @@ eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu) /* Forward function declarations */ static void setup_attached_port(portid_t pi); static void check_all_ports_link_status(uint32_t port_mask); -static int eth_event_callback(portid_t port_id, - enum rte_eth_event_type type, - void *param, void *ret_param); -static void dev_event_callback(const char *device_name, - enum rte_dev_event_type type, - void *param); static void fill_xstats_display_info(void); /* @@ -3672,7 +3631,7 @@ setup_attached_port(portid_t pi) printf("Done\n"); } -static void +void detach_device(struct rte_device *dev) { portid_t sibling; @@ -3818,13 +3777,9 @@ pmd_test_exit(void) return; } - ret = rte_dev_event_callback_unregister(NULL, - dev_event_callback, NULL); - if (ret < 0) { - RTE_LOG(ERR, EAL, - "fail to unregister device event callback.\n"); + ret = unregister_dev_event_callback(); + if (ret != 0) return; - } ret = rte_dev_hotplug_handle_disable(); if (ret) { @@ -3909,274 +3864,6 @@ check_all_ports_link_status(uint32_t port_mask) } } -static void -rmv_port_callback(void *arg) -{ - int need_to_start = 0; - int org_no_link_check = no_link_check; - portid_t port_id = (intptr_t)arg; - struct rte_eth_dev_info dev_info; - int ret; - - RTE_ETH_VALID_PORTID_OR_RET(port_id); - - if (!test_done && port_is_forwarding(port_id)) { - need_to_start = 1; - stop_packet_forwarding(); - } - no_link_check = 1; - stop_port(port_id); - no_link_check = org_no_link_check; - - ret = eth_dev_info_get_print_err(port_id, &dev_info); - if (ret != 0) - TESTPMD_LOG(ERR, - "Failed to get device info for port %d, not detaching\n", - port_id); - else { - struct rte_device *device = dev_info.device; - close_port(port_id); - detach_device(device); /* might be already removed or have more ports */ - } - if (need_to_start) - start_packet_forwarding(0); -} - -static int need_start_when_recovery_over; - -static bool -has_port_in_err_recovering(void) -{ - struct rte_port *port; - portid_t pid; - - RTE_ETH_FOREACH_DEV(pid) { - port = &ports[pid]; - if (port->err_recovering) - return true; - } - - return false; -} - -static void -err_recovering_callback(portid_t port_id) -{ - if (!has_port_in_err_recovering()) - printf("Please stop executing any commands until recovery result events are received!\n"); - - ports[port_id].err_recovering = 1; - ports[port_id].recover_failed = 0; - - /* To simplify implementation, stop forwarding regardless of whether the port is used. */ - if (!test_done) { - printf("Stop packet forwarding because some ports are in error recovering!\n"); - stop_packet_forwarding(); - need_start_when_recovery_over = 1; - } -} - -static void -recover_success_callback(portid_t port_id) -{ - ports[port_id].err_recovering = 0; - if (has_port_in_err_recovering()) - return; - - if (need_start_when_recovery_over) { - printf("Recovery success! Restart packet forwarding!\n"); - start_packet_forwarding(0); - need_start_when_recovery_over = 0; - } else { - printf("Recovery success!\n"); - } -} - -static void -recover_failed_callback(portid_t port_id) -{ - struct rte_port *port; - portid_t pid; - - ports[port_id].err_recovering = 0; - ports[port_id].recover_failed = 1; - if (has_port_in_err_recovering()) - return; - - need_start_when_recovery_over = 0; - printf("The ports:"); - RTE_ETH_FOREACH_DEV(pid) { - port = &ports[pid]; - if (port->recover_failed) - printf(" %u", pid); - } - printf(" recovery failed! Please remove them!\n"); -} - -/* This function is used by the interrupt thread */ -static int -eth_event_callback(portid_t port_id, enum rte_eth_event_type type, void *param, - void *ret_param) -{ - RTE_SET_USED(param); - RTE_SET_USED(ret_param); - - if (type >= RTE_ETH_EVENT_MAX) { - fprintf(stderr, - "\nPort %" PRIu16 ": %s called upon invalid event %d\n", - port_id, __func__, type); - fflush(stderr); - } else if (event_print_mask & (UINT32_C(1) << type)) { - printf("\nPort %" PRIu16 ": %s event\n", port_id, - eth_event_desc[type]); - fflush(stdout); - } - - switch (type) { - case RTE_ETH_EVENT_NEW: - ports[port_id].need_setup = 1; - ports[port_id].port_status = RTE_PORT_HANDLING; - break; - case RTE_ETH_EVENT_INTR_RMV: - if (port_id_is_invalid(port_id, DISABLED_WARN)) - break; - if (rte_eal_alarm_set(100000, - rmv_port_callback, (void *)(intptr_t)port_id)) - fprintf(stderr, - "Could not set up deferred device removal\n"); - break; - case RTE_ETH_EVENT_DESTROY: - ports[port_id].port_status = RTE_PORT_CLOSED; - printf("Port %u is closed\n", port_id); - break; - case RTE_ETH_EVENT_RX_AVAIL_THRESH: { - uint16_t rxq_id; - int ret; - - /* avail_thresh query API rewinds rxq_id, no need to check max RxQ num */ - for (rxq_id = 0; ; rxq_id++) { - ret = rte_eth_rx_avail_thresh_query(port_id, &rxq_id, - NULL); - if (ret <= 0) - break; - printf("Received avail_thresh event, port: %u, rxq_id: %u\n", - port_id, rxq_id); - -#ifdef RTE_NET_MLX5 - mlx5_test_avail_thresh_event_handler(port_id, rxq_id); -#endif - } - break; - } - case RTE_ETH_EVENT_ERR_RECOVERING: - err_recovering_callback(port_id); - break; - case RTE_ETH_EVENT_RECOVERY_SUCCESS: - recover_success_callback(port_id); - break; - case RTE_ETH_EVENT_RECOVERY_FAILED: - recover_failed_callback(port_id); - break; - default: - break; - } - return 0; -} - -static int -register_eth_event_callback(void) -{ - int ret; - enum rte_eth_event_type event; - - for (event = RTE_ETH_EVENT_UNKNOWN; - event < RTE_ETH_EVENT_MAX; event++) { - ret = rte_eth_dev_callback_register(RTE_ETH_ALL, - event, - eth_event_callback, - NULL); - if (ret != 0) { - TESTPMD_LOG(ERR, "Failed to register callback for " - "%s event\n", eth_event_desc[event]); - return -1; - } - } - - return 0; -} - -static int -unregister_eth_event_callback(void) -{ - int ret; - enum rte_eth_event_type event; - - for (event = RTE_ETH_EVENT_UNKNOWN; - event < RTE_ETH_EVENT_MAX; event++) { - ret = rte_eth_dev_callback_unregister(RTE_ETH_ALL, - event, - eth_event_callback, - NULL); - if (ret != 0) { - TESTPMD_LOG(ERR, "Failed to unregister callback for " - "%s event\n", eth_event_desc[event]); - return -1; - } - } - - return 0; -} - -/* This function is used by the interrupt thread */ -static void -dev_event_callback(const char *device_name, enum rte_dev_event_type type, - __rte_unused void *arg) -{ - uint16_t port_id; - int ret; - - if (type >= RTE_DEV_EVENT_MAX) { - fprintf(stderr, "%s called upon invalid event %d\n", - __func__, type); - fflush(stderr); - } - - switch (type) { - case RTE_DEV_EVENT_REMOVE: - RTE_LOG(DEBUG, EAL, "The device: %s has been removed!\n", - device_name); - ret = rte_eth_dev_get_port_by_name(device_name, &port_id); - if (ret) { - RTE_LOG(ERR, EAL, "can not get port by device %s!\n", - device_name); - return; - } - /* - * Because the user's callback is invoked in eal interrupt - * callback, the interrupt callback need to be finished before - * it can be unregistered when detaching device. So finish - * callback soon and use a deferred removal to detach device - * is need. It is a workaround, once the device detaching be - * moved into the eal in the future, the deferred removal could - * be deleted. - */ - if (rte_eal_alarm_set(100000, - rmv_port_callback, (void *)(intptr_t)port_id)) - RTE_LOG(ERR, EAL, - "Could not set up deferred device removal\n"); - break; - case RTE_DEV_EVENT_ADD: - RTE_LOG(ERR, EAL, "The device: %s has been added!\n", - device_name); - /* TODO: After finish kernel driver binding, - * begin to attach port. - */ - break; - default: - break; - } -} - static void rxtx_port_config(portid_t pid) { @@ -4725,13 +4412,9 @@ main(int argc, char** argv) return -1; } - ret = rte_dev_event_callback_register(NULL, - dev_event_callback, NULL); - if (ret) { - RTE_LOG(ERR, EAL, - "fail to register device event callback\n"); + ret = register_dev_event_callback(); + if (ret != 0) return -1; - } } if (!no_device_start && start_port(RTE_PORT_ALL) != 0) { diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h index 42782d5a05..5c8a052b43 100644 --- a/app/test-pmd/testpmd.h +++ b/app/test-pmd/testpmd.h @@ -1109,6 +1109,11 @@ void set_nb_pkt_per_burst(uint16_t pkt_burst); char *list_pkt_forwarding_modes(void); char *list_pkt_forwarding_retry_modes(void); void set_pkt_forwarding_mode(const char *fwd_mode); +int get_event_name_mask(const char *name, uint32_t *mask); +int register_eth_event_callback(void); +int unregister_eth_event_callback(void); +int register_dev_event_callback(void); +int unregister_dev_event_callback(void); void start_packet_forwarding(int with_tx_first); void fwd_stats_display(void); void fwd_stats_reset(void); @@ -1128,6 +1133,7 @@ void stop_port(portid_t pid); void close_port(portid_t pid); void reset_port(portid_t pid); void attach_port(char *identifier); +void detach_device(struct rte_device *dev); void detach_devargs(char *identifier); void detach_port_device(portid_t port_id); int all_ports_stopped(void); From patchwork Fri Oct 20 10:07:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: fengchengwen X-Patchwork-Id: 133073 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 302A7431BB; Fri, 20 Oct 2023 12:11:38 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F20E3427E4; Fri, 20 Oct 2023 12:11:05 +0200 (CEST) Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by mails.dpdk.org (Postfix) with ESMTP id AE8674069D for ; Fri, 20 Oct 2023 12:10:58 +0200 (CEST) Received: from dggpeml100024.china.huawei.com (unknown [172.30.72.57]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4SBgG93VYxzcdPD; Fri, 20 Oct 2023 18:06:09 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by dggpeml100024.china.huawei.com (7.185.36.115) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Fri, 20 Oct 2023 18:10:56 +0800 From: Chengwen Feng To: , , , , Aman Singh , Yuying Zhang CC: , , , Subject: [PATCH v2 7/7] doc: testpmd support event handling section Date: Fri, 20 Oct 2023 10:07:46 +0000 Message-ID: <20231020100746.31520-8-fengchengwen@huawei.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231020100746.31520-1-fengchengwen@huawei.com> References: <20230301030610.49468-1-fengchengwen@huawei.com> <20231020100746.31520-1-fengchengwen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpeml100024.china.huawei.com (7.185.36.115) X-CFilter-Loop: Reflected X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add new section of event handling, which documented the ethdev and device events. Signed-off-by: Chengwen Feng --- doc/guides/testpmd_app_ug/event_handling.rst | 80 ++++++++++++++++++++ doc/guides/testpmd_app_ug/index.rst | 1 + 2 files changed, 81 insertions(+) create mode 100644 doc/guides/testpmd_app_ug/event_handling.rst diff --git a/doc/guides/testpmd_app_ug/event_handling.rst b/doc/guides/testpmd_app_ug/event_handling.rst new file mode 100644 index 0000000000..c116753ad0 --- /dev/null +++ b/doc/guides/testpmd_app_ug/event_handling.rst @@ -0,0 +1,80 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2023 HiSilicon Limited. + +Event Handling +============== + +The ``testpmd`` application supports following two type event handling: + +ethdev events +------------- + +The ``testpmd`` provide options "--print-event" and "--mask-event" to control +whether display such as "Port x y event" when received "y" event on port "x". +This is named as default processing. + +This section details the support events, unless otherwise specified, only the +default processing is support. + +- ``RTE_ETH_EVENT_INTR_LSC``: + If device started with lsc enabled, the PMD will launch this event when it + detect link status changes. + +- ``RTE_ETH_EVENT_QUEUE_STATE``: + Used only within vhost PMD to report vring whether enabled. + +- ``RTE_ETH_EVENT_INTR_RESET``: + Used to report reset interrupt happened, this event only reported when the + PMD supports ``RTE_ETH_ERROR_HANDLE_MODE_PASSIVE``. + +- ``RTE_ETH_EVENT_VF_MBOX``: + Used as a PF to process mailbox messages of the VFs to which the PF belongs. + +- ``RTE_ETH_EVENT_INTR_RMV``: + Used to report device removal event. The ``testpmd`` will remove the port + later. + +- ``RTE_ETH_EVENT_NEW``: + Used to report port was probed event. The ``testpmd`` will setup the port + later. + +- ``RTE_ETH_EVENT_DESTROY``: + Used to report port was released event. The ``testpmd`` will changes the + port's status. + +- ``RTE_ETH_EVENT_MACSEC``: + Used to report MACsec offload related event. + +- ``RTE_ETH_EVENT_IPSEC``: + Used to report IPsec offload related event. + +- ``RTE_ETH_EVENT_FLOW_AGED``: + Used to report new aged-out flows was detected. Only valid with mlx5 PMD. + +- ``RTE_ETH_EVENT_RX_AVAIL_THRESH``: + Used to report available Rx descriptors was smaller than the threshold. Only + valid with mlx5 PMD. + +- ``RTE_ETH_EVENT_ERR_RECOVERING``: + Used to report error happened, and PMD will do recover after report this + event. The ``testpmd`` will stop packet forwarding when received the event. + +- ``RTE_ETH_EVENT_RECOVERY_SUCCESS``: + Used to report error recovery success. The ``testpmd`` will restart packet + forwarding when received the event. + +- ``RTE_ETH_EVENT_RECOVERY_FAILED``: + Used to report error recovery failed. The ``testpmd`` will display one + message to show which ports failed. + +.. note:: + + The ``RTE_ETH_EVENT_ERR_RECOVERING``, ``RTE_ETH_EVENT_RECOVERY_SUCCESS`` and + ``RTE_ETH_EVENT_RECOVERY_FAILED`` only reported when the PMD supports + ``RTE_ETH_ERROR_HANDLE_MODE_PROACTIVE``. + +device events +------------- + +Including two events ``RTE_DEV_EVENT_ADD`` and ``RTE_DEV_EVENT_ADD``, and +enabled only when the ``testpmd`` stated with options "--hot-plug". diff --git a/doc/guides/testpmd_app_ug/index.rst b/doc/guides/testpmd_app_ug/index.rst index 1ac0d25d57..3c09448c4e 100644 --- a/doc/guides/testpmd_app_ug/index.rst +++ b/doc/guides/testpmd_app_ug/index.rst @@ -14,3 +14,4 @@ Testpmd Application User Guide build_app run_app testpmd_funcs + event_handling