From patchwork Sun May 23 11:46:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Haiyue" X-Patchwork-Id: 93375 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1C72FA0547; Sun, 23 May 2021 14:07:07 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BD781410FC; Sun, 23 May 2021 14:07:01 +0200 (CEST) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id DA35F4003C for ; Sun, 23 May 2021 14:06:58 +0200 (CEST) IronPort-SDR: rpD+itF/PsuGIObQkCDlJFaE4/WMm442kPx4IuHfx5GyYSzNeIfRGe37ERIKCyT+0rPiwfZ61S 01I8CAvakQiw== X-IronPort-AV: E=McAfee;i="6200,9189,9992"; a="199847490" X-IronPort-AV: E=Sophos;i="5.82,319,1613462400"; d="scan'208";a="199847490" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2021 05:06:56 -0700 IronPort-SDR: SizbguBILRdxA164bz6uEHmhT/i+FEST4WbD3ZLtYiHcplX0pWRZzMVMQcsyP3CBKK65J8/V5q JhnjZgePwBng== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,319,1613462400"; d="scan'208";a="475390078" Received: from npg-dpdk-haiyue-1.sh.intel.com ([10.67.118.220]) by orsmga001.jf.intel.com with ESMTP; 23 May 2021 05:06:52 -0700 From: Haiyue Wang To: dev@dpdk.org Cc: qi.z.zhang@intel.com, liang-min.wang@intel.com, david.marchand@redhat.com, Haiyue Wang , Ray Kinsella , Neil Horman , Gaetan Rivet Date: Sun, 23 May 2021 19:46:06 +0800 Message-Id: <20210523114609.448092-2-haiyue.wang@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210523114609.448092-1-haiyue.wang@intel.com> References: <20210421050243.130585-1-haiyue.wang@intel.com> <20210523114609.448092-1-haiyue.wang@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v6 1/3] bus/pci: set PCI master in command register X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add the API to set 'Bus Master Enable' bit to be enabled or disabled in the PCI command register. Signed-off-by: Haiyue Wang Acked-by: Ray Kinsella --- drivers/bus/pci/pci_common.c | 28 ++++++++++++++++++++++++++++ drivers/bus/pci/rte_bus_pci.h | 14 ++++++++++++++ drivers/bus/pci/version.map | 3 +++ lib/pci/rte_pci.h | 4 ++++ 4 files changed, 49 insertions(+) diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c index ee7f966358..35d7d092d1 100644 --- a/drivers/bus/pci/pci_common.c +++ b/drivers/bus/pci/pci_common.c @@ -746,6 +746,34 @@ rte_pci_find_ext_capability(struct rte_pci_device *dev, uint32_t cap) return 0; } +int +rte_pci_set_bus_master(struct rte_pci_device *dev, bool enable) +{ + uint16_t old_cmd, cmd; + + if (rte_pci_read_config(dev, &old_cmd, sizeof(old_cmd), + RTE_PCI_COMMAND) < 0) { + RTE_LOG(ERR, EAL, "error in reading PCI command register\n"); + return -1; + } + + if (enable) + cmd = old_cmd | RTE_PCI_COMMAND_MASTER; + else + cmd = old_cmd & ~RTE_PCI_COMMAND_MASTER; + + if (cmd == old_cmd) + return 0; + + if (rte_pci_write_config(dev, &cmd, sizeof(cmd), + RTE_PCI_COMMAND) < 0) { + RTE_LOG(ERR, EAL, "error in writing PCI command register\n"); + return -1; + } + + return 0; +} + struct rte_pci_bus rte_pci_bus = { .bus = { .scan = rte_pci_scan, diff --git a/drivers/bus/pci/rte_bus_pci.h b/drivers/bus/pci/rte_bus_pci.h index 64886b4731..976c33c921 100644 --- a/drivers/bus/pci/rte_bus_pci.h +++ b/drivers/bus/pci/rte_bus_pci.h @@ -249,6 +249,20 @@ void rte_pci_dump(FILE *f); __rte_experimental off_t rte_pci_find_ext_capability(struct rte_pci_device *dev, uint32_t cap); +/** + * Enables/Disables Bus Master for device's PCI command register. + * + * @param dev + * A pointer to rte_pci_device structure. + * @param enable + * Enable or disable Bus Master. + * + * @return + * 0 on success, -1 on error in PCI config space read/write. + */ +__rte_experimental +int rte_pci_set_bus_master(struct rte_pci_device *dev, bool enable); + /** * Register a PCI driver. * diff --git a/drivers/bus/pci/version.map b/drivers/bus/pci/version.map index f33ed0abd1..00fac8864c 100644 --- a/drivers/bus/pci/version.map +++ b/drivers/bus/pci/version.map @@ -21,4 +21,7 @@ EXPERIMENTAL { global: rte_pci_find_ext_capability; + + # added in 21.08 + rte_pci_set_bus_master; }; diff --git a/lib/pci/rte_pci.h b/lib/pci/rte_pci.h index a8f8e404a9..1f33d687f4 100644 --- a/lib/pci/rte_pci.h +++ b/lib/pci/rte_pci.h @@ -32,6 +32,10 @@ extern "C" { #define RTE_PCI_VENDOR_ID 0x00 /* 16 bits */ #define RTE_PCI_DEVICE_ID 0x02 /* 16 bits */ +#define RTE_PCI_COMMAND 0x04 /* 16 bits */ + +/* PCI Command Register */ +#define RTE_PCI_COMMAND_MASTER 0x4 /* Bus Master Enable */ /* PCI Express capability registers */ #define RTE_PCI_EXP_DEVCTL 8 /* Device Control */ From patchwork Sun May 23 11:46:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Haiyue" X-Patchwork-Id: 93376 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ADA37A0547; Sun, 23 May 2021 14:07:13 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 02B8641109; Sun, 23 May 2021 14:07:03 +0200 (CEST) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id 7AF314003C for ; Sun, 23 May 2021 14:06:59 +0200 (CEST) IronPort-SDR: kDWEk7pNtmQ4n377enhcFCjfME6/sz79PT+/wiVKkNUkBU1j6D5+aTz7SOcyShGWCMADWKXsjZ us9P9wEkII3w== X-IronPort-AV: E=McAfee;i="6200,9189,9992"; a="199847493" X-IronPort-AV: E=Sophos;i="5.82,319,1613462400"; d="scan'208";a="199847493" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2021 05:06:58 -0700 IronPort-SDR: JytySES1of7HErY11qSa26dKF3wVyeHSwNkvLVsM6ABDpwnrJPJWQUlMio7i4XYdV9I//qFhj2 wcQJNaDfF81Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,319,1613462400"; d="scan'208";a="475390084" Received: from npg-dpdk-haiyue-1.sh.intel.com ([10.67.118.220]) by orsmga001.jf.intel.com with ESMTP; 23 May 2021 05:06:56 -0700 From: Haiyue Wang To: dev@dpdk.org Cc: qi.z.zhang@intel.com, liang-min.wang@intel.com, david.marchand@redhat.com, Haiyue Wang , Jingjing Wu , Beilei Xing Date: Sun, 23 May 2021 19:46:07 +0800 Message-Id: <20210523114609.448092-3-haiyue.wang@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210523114609.448092-1-haiyue.wang@intel.com> References: <20210421050243.130585-1-haiyue.wang@intel.com> <20210523114609.448092-1-haiyue.wang@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v6 2/3] net/iavf: enable PCI bus master after reset X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The VF reset can be triggerred by the PF reset event, in this case, the PCI bus master will be cleared, then the VF is not allowed to issue any Memory or I/O Requests. So after the reset event is detected, always enable the PCI bus master. And if failed, the device or system may be in an invalid state, so keep the VF reset state to mark it as I/O error. Signed-off-by: Haiyue Wang --- drivers/net/iavf/iavf_ethdev.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/net/iavf/iavf_ethdev.c b/drivers/net/iavf/iavf_ethdev.c index d688c31cfb..a7ef7a6d4d 100644 --- a/drivers/net/iavf/iavf_ethdev.c +++ b/drivers/net/iavf/iavf_ethdev.c @@ -2356,7 +2356,15 @@ iavf_dev_close(struct rte_eth_dev *dev) rte_free(vf->aq_resp); vf->aq_resp = NULL; - vf->vf_reset = false; + /* + * If the VF is reset via VFLR, the device will be knocked out of bus + * master mode, and the driver will fail to recover from the reset. Fix + * this by enabling bus mastering after every reset. In a non-VFLR case, + * the bus master bit will not be disabled, and this call will have no + * effect. + */ + if (vf->vf_reset && !rte_pci_set_bus_master(pci_dev, true)) + vf->vf_reset = false; return ret; } From patchwork Sun May 23 11:46:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Haiyue" X-Patchwork-Id: 93377 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B38D7A0547; Sun, 23 May 2021 14:07:19 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 357BA41110; Sun, 23 May 2021 14:07:04 +0200 (CEST) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mails.dpdk.org (Postfix) with ESMTP id C63DD410FE for ; Sun, 23 May 2021 14:07:01 +0200 (CEST) IronPort-SDR: mNLRcr0uEh3eFDZsfYz99+QRAEMZTDMZbNhAFrYjpKc7g6ju+jgnqftXq4G/5FHrdskdaImOHh xzf9PYOZzsow== X-IronPort-AV: E=McAfee;i="6200,9189,9992"; a="199847498" X-IronPort-AV: E=Sophos;i="5.82,319,1613462400"; d="scan'208";a="199847498" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2021 05:07:01 -0700 IronPort-SDR: 8SY4P28ecs1T2+uxMtYNQi85x8Lqiy6TAFiCybGm1z16UlrFolEbWqBAFYVO9sRtDayi1Wvxrg rW3rJeBoQjcA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,319,1613462400"; d="scan'208";a="475390095" Received: from npg-dpdk-haiyue-1.sh.intel.com ([10.67.118.220]) by orsmga001.jf.intel.com with ESMTP; 23 May 2021 05:06:59 -0700 From: Haiyue Wang To: dev@dpdk.org Cc: qi.z.zhang@intel.com, liang-min.wang@intel.com, david.marchand@redhat.com, Haiyue Wang , Beilei Xing Date: Sun, 23 May 2021 19:46:08 +0800 Message-Id: <20210523114609.448092-4-haiyue.wang@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210523114609.448092-1-haiyue.wang@intel.com> References: <20210421050243.130585-1-haiyue.wang@intel.com> <20210523114609.448092-1-haiyue.wang@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v6 3/3] net/i40e: enable PCI bus master after reset X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The VF reset can be triggerred by the PF reset event, in this case, the PCI bus master will be cleared, then the VF is not allowed to issue any Memory or I/O Requests. So after the reset event is detected, always enable the PCI bus master. And if failed, the device or system may be in an invalid state, so keep the VF reset state to mark it as I/O error. Signed-off-by: Haiyue Wang --- drivers/net/i40e/i40e_ethdev_vf.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c index cb898bdb68..385ebedcd3 100644 --- a/drivers/net/i40e/i40e_ethdev_vf.c +++ b/drivers/net/i40e/i40e_ethdev_vf.c @@ -1213,7 +1213,6 @@ i40evf_check_vf_reset_done(struct rte_eth_dev *dev) if (i >= MAX_RESET_WAIT_CNT) return -1; - vf->vf_reset = false; vf->pend_msg &= ~PFMSG_RESET_IMPENDING; return 0; @@ -1392,6 +1391,7 @@ i40evf_handle_pf_event(struct rte_eth_dev *dev, uint8_t *msg, switch (pf_msg->event) { case VIRTCHNL_EVENT_RESET_IMPENDING: PMD_DRV_LOG(DEBUG, "VIRTCHNL_EVENT_RESET_IMPENDING event"); + vf->vf_reset = true; rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_RESET, NULL); break; @@ -2468,6 +2468,7 @@ i40evf_dev_close(struct rte_eth_dev *dev) { struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private); struct i40e_vf *vf = I40EVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); + struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev); int ret; if (rte_eal_process_type() != RTE_PROC_PRIMARY) @@ -2490,6 +2491,16 @@ i40evf_dev_close(struct rte_eth_dev *dev) i40e_shutdown_adminq(hw); i40evf_disable_irq0(hw); + /* + * If the VF is reset via VFLR, the device will be knocked out of bus + * master mode, and the driver will fail to recover from the reset. Fix + * this by enabling bus mastering after every reset. In a non-VFLR case, + * the bus master bit will not be disabled, and this call will have no + * effect. + */ + if (vf->vf_reset && !rte_pci_set_bus_master(pci_dev, true)) + vf->vf_reset = false; + rte_free(vf->vf_res); vf->vf_res = NULL; rte_free(vf->aq_resp);