From patchwork Mon May 24 01:23:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Haiyue" X-Patchwork-Id: 93379 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 55F6FA0547; Mon, 24 May 2021 03:44:45 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 167AA410FE; Mon, 24 May 2021 03:44:39 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 8A6184003C for ; Mon, 24 May 2021 03:44:36 +0200 (CEST) IronPort-SDR: d4o9PQfyk8io38DUAYnCoajOhR8daT7ergMOvQRVFWxOLmKcr5oiJOgj5cLy2dniagBGkiTimf xWYsMRB0GL5w== X-IronPort-AV: E=McAfee;i="6200,9189,9993"; a="201573598" X-IronPort-AV: E=Sophos;i="5.82,319,1613462400"; d="scan'208";a="201573598" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2021 18:44:36 -0700 IronPort-SDR: rPofawZJYPVu+mjhraznTMBEsEj74x+TCcSO0NRrr6xS0OZHoh64JEKZsB0hI1qhd1xcc1XP7y CZ3W2zAmq6Tw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,319,1613462400"; d="scan'208";a="441824220" Received: from npg-dpdk-haiyue-1.sh.intel.com ([10.67.118.220]) by orsmga008.jf.intel.com with ESMTP; 23 May 2021 18:44:33 -0700 From: Haiyue Wang To: dev@dpdk.org Cc: qi.z.zhang@intel.com, liang-min.wang@intel.com, david.marchand@redhat.com, Haiyue Wang , Ray Kinsella , Neil Horman , Gaetan Rivet Date: Mon, 24 May 2021 09:23:43 +0800 Message-Id: <20210524012346.496560-2-haiyue.wang@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210524012346.496560-1-haiyue.wang@intel.com> References: <20210421050243.130585-1-haiyue.wang@intel.com> <20210524012346.496560-1-haiyue.wang@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v7 1/3] bus/pci: set PCI master in command register X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add the API to set 'Bus Master Enable' bit to be enabled or disabled in the PCI command register. Signed-off-by: Haiyue Wang Acked-by: Ray Kinsella --- drivers/bus/pci/pci_common.c | 28 ++++++++++++++++++++++++++++ drivers/bus/pci/rte_bus_pci.h | 14 ++++++++++++++ drivers/bus/pci/version.map | 3 +++ lib/pci/rte_pci.h | 4 ++++ 4 files changed, 49 insertions(+) diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c index ee7f966358..35d7d092d1 100644 --- a/drivers/bus/pci/pci_common.c +++ b/drivers/bus/pci/pci_common.c @@ -746,6 +746,34 @@ rte_pci_find_ext_capability(struct rte_pci_device *dev, uint32_t cap) return 0; } +int +rte_pci_set_bus_master(struct rte_pci_device *dev, bool enable) +{ + uint16_t old_cmd, cmd; + + if (rte_pci_read_config(dev, &old_cmd, sizeof(old_cmd), + RTE_PCI_COMMAND) < 0) { + RTE_LOG(ERR, EAL, "error in reading PCI command register\n"); + return -1; + } + + if (enable) + cmd = old_cmd | RTE_PCI_COMMAND_MASTER; + else + cmd = old_cmd & ~RTE_PCI_COMMAND_MASTER; + + if (cmd == old_cmd) + return 0; + + if (rte_pci_write_config(dev, &cmd, sizeof(cmd), + RTE_PCI_COMMAND) < 0) { + RTE_LOG(ERR, EAL, "error in writing PCI command register\n"); + return -1; + } + + return 0; +} + struct rte_pci_bus rte_pci_bus = { .bus = { .scan = rte_pci_scan, diff --git a/drivers/bus/pci/rte_bus_pci.h b/drivers/bus/pci/rte_bus_pci.h index 64886b4731..976c33c921 100644 --- a/drivers/bus/pci/rte_bus_pci.h +++ b/drivers/bus/pci/rte_bus_pci.h @@ -249,6 +249,20 @@ void rte_pci_dump(FILE *f); __rte_experimental off_t rte_pci_find_ext_capability(struct rte_pci_device *dev, uint32_t cap); +/** + * Enables/Disables Bus Master for device's PCI command register. + * + * @param dev + * A pointer to rte_pci_device structure. + * @param enable + * Enable or disable Bus Master. + * + * @return + * 0 on success, -1 on error in PCI config space read/write. + */ +__rte_experimental +int rte_pci_set_bus_master(struct rte_pci_device *dev, bool enable); + /** * Register a PCI driver. * diff --git a/drivers/bus/pci/version.map b/drivers/bus/pci/version.map index f33ed0abd1..00fac8864c 100644 --- a/drivers/bus/pci/version.map +++ b/drivers/bus/pci/version.map @@ -21,4 +21,7 @@ EXPERIMENTAL { global: rte_pci_find_ext_capability; + + # added in 21.08 + rte_pci_set_bus_master; }; diff --git a/lib/pci/rte_pci.h b/lib/pci/rte_pci.h index a8f8e404a9..1f33d687f4 100644 --- a/lib/pci/rte_pci.h +++ b/lib/pci/rte_pci.h @@ -32,6 +32,10 @@ extern "C" { #define RTE_PCI_VENDOR_ID 0x00 /* 16 bits */ #define RTE_PCI_DEVICE_ID 0x02 /* 16 bits */ +#define RTE_PCI_COMMAND 0x04 /* 16 bits */ + +/* PCI Command Register */ +#define RTE_PCI_COMMAND_MASTER 0x4 /* Bus Master Enable */ /* PCI Express capability registers */ #define RTE_PCI_EXP_DEVCTL 8 /* Device Control */ From patchwork Mon May 24 01:23:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Haiyue" X-Patchwork-Id: 93380 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5A33EA0547; Mon, 24 May 2021 03:44:52 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 41F0C41109; Mon, 24 May 2021 03:44:42 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 6725A41108 for ; Mon, 24 May 2021 03:44:40 +0200 (CEST) IronPort-SDR: Dsx/MSKr7qwfiAr5Xhf6WgyZS2+rbR7I5clsLhRs72NrXwggSpy0M70yBCWAkl/P9QhZunGdWU jzvrPoA12DuQ== X-IronPort-AV: E=McAfee;i="6200,9189,9993"; a="201573599" X-IronPort-AV: E=Sophos;i="5.82,319,1613462400"; d="scan'208";a="201573599" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2021 18:44:38 -0700 IronPort-SDR: H+u+8QBDvOyXdOt9k/n9J3nof18J+qxlUkGFaFCuT7GOHVmhz8mLCsv2veHjHU8w+zH+bHlsTp xsftFIbcO4YA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,319,1613462400"; d="scan'208";a="441824235" Received: from npg-dpdk-haiyue-1.sh.intel.com ([10.67.118.220]) by orsmga008.jf.intel.com with ESMTP; 23 May 2021 18:44:36 -0700 From: Haiyue Wang To: dev@dpdk.org Cc: qi.z.zhang@intel.com, liang-min.wang@intel.com, david.marchand@redhat.com, Haiyue Wang , Jingjing Wu , Beilei Xing Date: Mon, 24 May 2021 09:23:44 +0800 Message-Id: <20210524012346.496560-3-haiyue.wang@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210524012346.496560-1-haiyue.wang@intel.com> References: <20210421050243.130585-1-haiyue.wang@intel.com> <20210524012346.496560-1-haiyue.wang@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v7 2/3] net/iavf: enable PCI bus master after reset X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The VF reset can be triggered by the PF reset event, then the PCI bus master will be cleared, the VF will be not allowed to issue any Memory or I/O Requests. So after the reset event is detected, always enable the PCI bus master. And if failed, the device or system may be in an invalid state, so keep the VF reset state to mark it as I/O error. Signed-off-by: Haiyue Wang Acked-by: Beilei Xing --- drivers/net/iavf/iavf_ethdev.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/net/iavf/iavf_ethdev.c b/drivers/net/iavf/iavf_ethdev.c index d688c31cfb..a7ef7a6d4d 100644 --- a/drivers/net/iavf/iavf_ethdev.c +++ b/drivers/net/iavf/iavf_ethdev.c @@ -2356,7 +2356,15 @@ iavf_dev_close(struct rte_eth_dev *dev) rte_free(vf->aq_resp); vf->aq_resp = NULL; - vf->vf_reset = false; + /* + * If the VF is reset via VFLR, the device will be knocked out of bus + * master mode, and the driver will fail to recover from the reset. Fix + * this by enabling bus mastering after every reset. In a non-VFLR case, + * the bus master bit will not be disabled, and this call will have no + * effect. + */ + if (vf->vf_reset && !rte_pci_set_bus_master(pci_dev, true)) + vf->vf_reset = false; return ret; } From patchwork Mon May 24 01:23:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Haiyue" X-Patchwork-Id: 93381 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DBB5DA0547; Mon, 24 May 2021 03:44:57 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7A98C41110; Mon, 24 May 2021 03:44:43 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 2EFAA41108 for ; Mon, 24 May 2021 03:44:41 +0200 (CEST) IronPort-SDR: AKil6bG/NZXdoPuHepezgDX8LGiMPnyXqHvm0dkZTUaGHQ9WA3BP9Rl0JeAHyKw60zWRJi1gXU qqUdwbNvPMKA== X-IronPort-AV: E=McAfee;i="6200,9189,9993"; a="201573601" X-IronPort-AV: E=Sophos;i="5.82,319,1613462400"; d="scan'208";a="201573601" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2021 18:44:40 -0700 IronPort-SDR: VXqSq2bOodmtK+Nn08chO91183ZdEh4GWHy4faaqsn7WmZXdQsyCXHufD/i8VFe96GZqCptVTw rBJJjIlF0Jog== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,319,1613462400"; d="scan'208";a="441824244" Received: from npg-dpdk-haiyue-1.sh.intel.com ([10.67.118.220]) by orsmga008.jf.intel.com with ESMTP; 23 May 2021 18:44:38 -0700 From: Haiyue Wang To: dev@dpdk.org Cc: qi.z.zhang@intel.com, liang-min.wang@intel.com, david.marchand@redhat.com, Haiyue Wang , Beilei Xing Date: Mon, 24 May 2021 09:23:45 +0800 Message-Id: <20210524012346.496560-4-haiyue.wang@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210524012346.496560-1-haiyue.wang@intel.com> References: <20210421050243.130585-1-haiyue.wang@intel.com> <20210524012346.496560-1-haiyue.wang@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v7 3/3] net/i40e: enable PCI bus master after reset X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The VF reset can be triggered by the PF reset event, then the PCI bus master will be cleared, the VF will be not allowed to issue any Memory or I/O Requests. So after the reset event is detected, always enable the PCI bus master. And if failed, the device or system may be in an invalid state, so keep the VF reset state to mark it as I/O error. Signed-off-by: Haiyue Wang Acked-by: Beilei Xing --- drivers/net/i40e/i40e_ethdev_vf.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c index cb898bdb68..385ebedcd3 100644 --- a/drivers/net/i40e/i40e_ethdev_vf.c +++ b/drivers/net/i40e/i40e_ethdev_vf.c @@ -1213,7 +1213,6 @@ i40evf_check_vf_reset_done(struct rte_eth_dev *dev) if (i >= MAX_RESET_WAIT_CNT) return -1; - vf->vf_reset = false; vf->pend_msg &= ~PFMSG_RESET_IMPENDING; return 0; @@ -1392,6 +1391,7 @@ i40evf_handle_pf_event(struct rte_eth_dev *dev, uint8_t *msg, switch (pf_msg->event) { case VIRTCHNL_EVENT_RESET_IMPENDING: PMD_DRV_LOG(DEBUG, "VIRTCHNL_EVENT_RESET_IMPENDING event"); + vf->vf_reset = true; rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_RESET, NULL); break; @@ -2468,6 +2468,7 @@ i40evf_dev_close(struct rte_eth_dev *dev) { struct i40e_hw *hw = I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private); struct i40e_vf *vf = I40EVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); + struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev); int ret; if (rte_eal_process_type() != RTE_PROC_PRIMARY) @@ -2490,6 +2491,16 @@ i40evf_dev_close(struct rte_eth_dev *dev) i40e_shutdown_adminq(hw); i40evf_disable_irq0(hw); + /* + * If the VF is reset via VFLR, the device will be knocked out of bus + * master mode, and the driver will fail to recover from the reset. Fix + * this by enabling bus mastering after every reset. In a non-VFLR case, + * the bus master bit will not be disabled, and this call will have no + * effect. + */ + if (vf->vf_reset && !rte_pci_set_bus_master(pci_dev, true)) + vf->vf_reset = false; + rte_free(vf->vf_res); vf->vf_res = NULL; rte_free(vf->aq_resp);