From patchwork Thu Sep 24 13:02:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Rybchenko X-Patchwork-Id: 78741 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id DE50AA04B1; Thu, 24 Sep 2020 15:03:01 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 909671DDB8; Thu, 24 Sep 2020 15:03:01 +0200 (CEST) Received: from dispatch1-us1.ppe-hosted.com (dispatch1-us1.ppe-hosted.com [67.231.154.164]) by dpdk.org (Postfix) with ESMTP id 12A8C1C068 for ; Thu, 24 Sep 2020 15:03:00 +0200 (CEST) Received: from mx1-us1.ppe-hosted.com (unknown [10.110.50.137]) by dispatch1-us1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTP id 9DC74200CD for ; Thu, 24 Sep 2020 13:02:59 +0000 (UTC) Received: from us4-mdac16-59.at1.mdlocal (unknown [10.110.50.152]) by mx1-us1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTP id 9C496600A7 for ; Thu, 24 Sep 2020 13:02:59 +0000 (UTC) X-Virus-Scanned: Proofpoint Essentials engine Received: from mx1-us1.ppe-hosted.com (unknown [10.110.49.109]) by mx1-us1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTPS id B1AA9220064 for ; Thu, 24 Sep 2020 13:02:57 +0000 (UTC) Received: from webmail.solarflare.com (uk.solarflare.com [193.34.186.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mx1-us1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTPS id 6676EB4008D for ; Thu, 24 Sep 2020 13:02:57 +0000 (UTC) Received: from ukex01.SolarFlarecom.com (10.17.10.4) by ukex01.SolarFlarecom.com (10.17.10.4) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 24 Sep 2020 14:02:52 +0100 Received: from opal.uk.solarflarecom.com (10.17.10.1) by ukex01.SolarFlarecom.com (10.17.10.4) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 24 Sep 2020 14:02:52 +0100 Received: from ukv-loginhost.uk.solarflarecom.com (ukv-loginhost.uk.solarflarecom.com [10.17.10.39]) by opal.uk.solarflarecom.com (8.13.8/8.13.8) with ESMTP id 08OD2qsL008219 for ; Thu, 24 Sep 2020 14:02:52 +0100 Received: from ukv-loginhost.uk.solarflarecom.com (localhost [127.0.0.1]) by ukv-loginhost.uk.solarflarecom.com (Postfix) with ESMTP id A16541613A9 for ; Thu, 24 Sep 2020 14:02:52 +0100 (BST) From: Andrew Rybchenko To: Date: Thu, 24 Sep 2020 14:02:35 +0100 Message-ID: <1600952555-5030-1-git-send-email-arybchenko@solarflare.com> X-Mailer: git-send-email 1.8.3.1 MIME-Version: 1.0 X-TM-AS-Product-Ver: SMEX-12.5.0.1300-8.6.1012-25674.003 X-TM-AS-Result: No-8.803600-8.000000-10 X-TMASE-MatchedRID: CrpnShPcGhGFCX92EIwegsPRlYRmv+e2nKpQna4coUDRmpwtWlELp9nf JrUSEbFDMdqMWOXDAJTgLc5WU85ONq0iin8P0KjVT7O/YHJhINDr3E41VlKsfThdESD0qLXT94d o8m0JE5L6rhTXN9WjCHQkkeQfHinAMdl2bvZ9hrPP/uumIUCPf+RjZuXE0WlHv5ndmnZN3URQ85 OjNEKQWC/jBgcz9kPzawhmKfgtnaUTYWf1ycjul9E+Q9UKuqQn3FYvKmZiVnOHX0cDZiY+DaEGK hm9baaNEWWjdUQziNE0+nwJxXqsVw8NdVEi9GXAFJx56/xm2k99dEBz6zdEV3mvr3EYUSdAkxr3 kgeRkMKpX6UtbIzNSyIx3X9pCBYZMDvZPMbvTD4vLP1C8DIeOnHv4qNcyiZVpkdx/UTH0kprcDq Fw27ycIZcaAuRgsdJxVuJ7nmLWiPrbEGb+5Q/eehsg0dmQfnG9fvWztwgm5MC/nuPl5P0Fqyaw9 ZbX5wGzJMcyL1Q2P6xLt+iCL7g3HDbgq6Qs1XKSHCU59h5KrFC61vu00HIv0oMqYzZgxDr1Is5G vhmGbx9Kr9zyswmaKFLKr480mEqwZuT56fuU0+KYdYQLbymTd1eFEoaE12njCcYoCggoksnbHEx J2kKrv0NI6Bi46OOr8186pfsJcZzKfT2b6qrDmFdfLBMkul8SkWlzdsmIx6EWB8iuyGqCcZDwd3 jv9Fu4vM1YF6AJbbCCfuIMF6xLSAHAopEd76vnB+C87yJt1y9CMtAJzoY4rc8MsvbL1BlzYgu5i BH7Ujmu7p4/d9o5g== X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--8.803600-8.000000 X-TMASE-Version: SMEX-12.5.0.1300-8.6.1012-25674.003 X-MDID: 1600952578-y5zwLAAW8R61 Subject: [dpdk-dev] [PATCH] net/sfc: create virtual switch to enable VFs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" PF driver is responsible for vSwitch creation and vPorts allocation for VFs. Signed-off-by: Andrew Rybchenko Reviewed-by: Andy Moreton --- doc/guides/nics/sfc_efx.rst | 2 + doc/guides/rel_notes/release_20_11.rst | 6 + drivers/common/sfc_efx/efsys.h | 2 +- .../sfc_efx/rte_common_sfc_efx_version.map | 9 + drivers/net/sfc/meson.build | 1 + drivers/net/sfc/sfc.c | 36 ++++ drivers/net/sfc/sfc.h | 4 +- drivers/net/sfc/sfc_ethdev.c | 2 + drivers/net/sfc/sfc_sriov.c | 168 ++++++++++++++++++ drivers/net/sfc/sfc_sriov.h | 39 ++++ 10 files changed, 267 insertions(+), 2 deletions(-) create mode 100644 drivers/net/sfc/sfc_sriov.c create mode 100644 drivers/net/sfc/sfc_sriov.h diff --git a/doc/guides/nics/sfc_efx.rst b/doc/guides/nics/sfc_efx.rst index be1c2fe1d6..ab44ce66c8 100644 --- a/doc/guides/nics/sfc_efx.rst +++ b/doc/guides/nics/sfc_efx.rst @@ -71,6 +71,8 @@ SFC EFX PMD has support for: - Loopback +- SR-IOV PF + Non-supported Features ---------------------- diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst index c6642f5f94..56b4ac1515 100644 --- a/doc/guides/rel_notes/release_20_11.rst +++ b/doc/guides/rel_notes/release_20_11.rst @@ -62,6 +62,12 @@ New Features * Added support for non-zero priorities for group 0 flows * Added support for VXLAN decap combined with VLAN pop +* **Updated Solarflare network PMD.** + + Updated the Solarflare ``sfc_efx`` driver with changes including: + + * Added SR-IOV PF support + * **Extended flow-perf application.** * Started supporting user order instead of bit mask: diff --git a/drivers/common/sfc_efx/efsys.h b/drivers/common/sfc_efx/efsys.h index 46b08f9789..3d769d32ae 100644 --- a/drivers/common/sfc_efx/efsys.h +++ b/drivers/common/sfc_efx/efsys.h @@ -156,7 +156,7 @@ prefetch_read_once(const volatile void *addr) #define EFSYS_OPT_FW_SUBVARIANT_AWARE 1 -#define EFSYS_OPT_EVB 0 +#define EFSYS_OPT_EVB 1 #define EFSYS_OPT_MCDI_PROXY_AUTH_SERVER 0 diff --git a/drivers/common/sfc_efx/rte_common_sfc_efx_version.map b/drivers/common/sfc_efx/rte_common_sfc_efx_version.map index 5e2a7ad919..1049e633c8 100644 --- a/drivers/common/sfc_efx/rte_common_sfc_efx_version.map +++ b/drivers/common/sfc_efx/rte_common_sfc_efx_version.map @@ -14,6 +14,15 @@ INTERNAL { efx_ev_qprime; efx_ev_usecs_to_ticks; + efx_evb_fini; + efx_evb_init; + efx_evb_vport_mac_set; + efx_evb_vport_reset; + efx_evb_vport_stats; + efx_evb_vport_vlan_set; + efx_evb_vswitch_create; + efx_evb_vswitch_destroy; + efx_evq_nbufs; efx_evq_size; diff --git a/drivers/net/sfc/meson.build b/drivers/net/sfc/meson.build index 26f0323738..1c6451938a 100644 --- a/drivers/net/sfc/meson.build +++ b/drivers/net/sfc/meson.build @@ -39,6 +39,7 @@ sources = files( 'sfc_kvargs.c', 'sfc.c', 'sfc_mcdi.c', + 'sfc_sriov.c', 'sfc_intr.c', 'sfc_ev.c', 'sfc_port.c', diff --git a/drivers/net/sfc/sfc.c b/drivers/net/sfc/sfc.c index 03ea5dc128..493a72f3ba 100644 --- a/drivers/net/sfc/sfc.c +++ b/drivers/net/sfc/sfc.c @@ -436,7 +436,20 @@ sfc_start(struct sfc_adapter *sa) sa->state = SFC_ADAPTER_STARTING; + rc = 0; do { + /* + * FIXME Try to recreate vSwitch on start retry. + * vSwitch is absent after MC reboot like events and + * we should recreate it. May be we need proper + * indication instead of guessing. + */ + if (rc != 0) { + sfc_sriov_vswitch_destroy(sa); + rc = sfc_sriov_vswitch_create(sa); + if (rc != 0) + goto fail_sriov_vswitch_create; + } rc = sfc_try_start(sa); } while ((--start_tries > 0) && (rc == EIO || rc == EAGAIN || rc == ENOENT || rc == EINVAL)); @@ -449,6 +462,7 @@ sfc_start(struct sfc_adapter *sa) return 0; fail_try_start: +fail_sriov_vswitch_create: sa->state = SFC_ADAPTER_CONFIGURED; fail_bad_state: sfc_log_init(sa, "failed %d", rc); @@ -727,6 +741,10 @@ sfc_attach(struct sfc_adapter *sa) if (rc != 0) goto fail_nic_reset; + rc = sfc_sriov_attach(sa); + if (rc != 0) + goto fail_sriov_attach; + /* * Probed NIC is sufficient for tunnel init. * Initialize tunnel support to be able to use libefx @@ -809,11 +827,24 @@ sfc_attach(struct sfc_adapter *sa) sfc_flow_init(sa); + /* + * Create vSwitch to be able to use VFs when PF is not started yet + * as DPDK port. VFs should be able to talk to each other even + * if PF is down. + */ + rc = sfc_sriov_vswitch_create(sa); + if (rc != 0) + goto fail_sriov_vswitch_create; + sa->state = SFC_ADAPTER_INITIALIZED; sfc_log_init(sa, "done"); return 0; +fail_sriov_vswitch_create: + sfc_flow_fini(sa); + sfc_filter_detach(sa); + fail_filter_attach: sfc_rss_detach(sa); @@ -832,7 +863,9 @@ sfc_attach(struct sfc_adapter *sa) fail_estimate_rsrc_limits: fail_tunnel_init: efx_tunnel_fini(sa->nic); + sfc_sriov_detach(sa); +fail_sriov_attach: fail_nic_reset: sfc_log_init(sa, "failed %d", rc); @@ -846,6 +879,8 @@ sfc_detach(struct sfc_adapter *sa) SFC_ASSERT(sfc_adapter_is_locked(sa)); + sfc_sriov_vswitch_destroy(sa); + sfc_flow_fini(sa); sfc_filter_detach(sa); @@ -854,6 +889,7 @@ sfc_detach(struct sfc_adapter *sa) sfc_ev_detach(sa); sfc_intr_detach(sa); efx_tunnel_fini(sa->nic); + sfc_sriov_detach(sa); sa->state = SFC_ADAPTER_UNINITIALIZED; } diff --git a/drivers/net/sfc/sfc.h b/drivers/net/sfc/sfc.h index fa7c79b80b..0c32527ad5 100644 --- a/drivers/net/sfc/sfc.h +++ b/drivers/net/sfc/sfc.h @@ -1,5 +1,5 @@ /* SPDX-License-Identifier: BSD-3-Clause - * +* * Copyright(c) 2019-2020 Xilinx, Inc. * Copyright(c) 2016-2019 Solarflare Communications Inc. * @@ -26,6 +26,7 @@ #include "sfc_debug.h" #include "sfc_log.h" #include "sfc_filter.h" +#include "sfc_sriov.h" #ifdef __cplusplus extern "C" { @@ -224,6 +225,7 @@ struct sfc_adapter { rte_atomic32_t restart_required; struct sfc_efx_mcdi mcdi; + struct sfc_sriov sriov; struct sfc_intr intr; struct sfc_port port; struct sfc_filter filter; diff --git a/drivers/net/sfc/sfc_ethdev.c b/drivers/net/sfc/sfc_ethdev.c index 6c96118007..600eaa23ea 100644 --- a/drivers/net/sfc/sfc_ethdev.c +++ b/drivers/net/sfc/sfc_ethdev.c @@ -102,6 +102,8 @@ sfc_dev_infos_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) dev_info->max_rx_pktlen = EFX_MAC_PDU_MAX; + dev_info->max_vfs = sa->sriov.num_vfs; + /* Autonegotiation may be disabled */ dev_info->speed_capa = ETH_LINK_SPEED_FIXED; if (sa->port.phy_adv_cap_mask & (1u << EFX_PHY_CAP_1000FDX)) diff --git a/drivers/net/sfc/sfc_sriov.c b/drivers/net/sfc/sfc_sriov.c new file mode 100644 index 0000000000..6120840bb4 --- /dev/null +++ b/drivers/net/sfc/sfc_sriov.c @@ -0,0 +1,168 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * + * Copyright(c) 2019-2020 Xilinx, Inc. + * Copyright(c) 2019 Solarflare Communications Inc. + * + * This software was jointly developed between OKTET Labs (under contract + * for Solarflare) and Solarflare Communications, Inc. + */ + +#include +#include + +#include "sfc.h" +#include "sfc_log.h" + +#include "efx.h" + + +/* + * Check if a MAC address is already assigned to to one of previously + * configured vPorts (either PF itself or one of already configured VFs). + * + * Typically the first vPort which corresponds to PF has globally + * administered unicast address, but it still could be locally + * administered if user assigned it or in the case of unconfigured NIC. + * So, it is safer to include it as well in uniqueness check. + */ +static bool +sriov_mac_addr_assigned(const efx_vport_config_t *vport_config, + unsigned int num, const uint8_t *mac_addr) +{ + unsigned int i; + + /* Check PF's MAC address as well as explained above */ + for (i = 0; i < num; ++i) { + if (memcmp(mac_addr, vport_config[i].evc_mac_addr, + sizeof(vport_config[i].evc_mac_addr)) == 0) + return true; + } + + return false; +} + +int +sfc_sriov_attach(struct sfc_adapter *sa) +{ + const struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(sa->eth_dev); + struct sfc_sriov *sriov = &sa->sriov; + efx_vport_config_t *vport_config; + unsigned int i; + int rc; + + sfc_log_init(sa, "entry"); + + sriov->num_vfs = pci_dev->max_vfs; + if (sriov->num_vfs == 0) + goto done; + + vport_config = calloc(sriov->num_vfs + 1, sizeof(*vport_config)); + if (vport_config == NULL) { + rc = ENOMEM; + goto fail_alloc_vport_config; + } + + vport_config[0].evc_function = 0xffff; + vport_config[0].evc_vid = EFX_VF_VID_DEFAULT; + vport_config[0].evc_vlan_restrict = B_FALSE; + + for (i = 1; i <= sriov->num_vfs; ++i) { + vport_config[i].evc_function = i - 1; + vport_config[i].evc_vid = EFX_VF_VID_DEFAULT; + vport_config[i].evc_vlan_restrict = B_FALSE; + do { + rte_eth_random_addr(vport_config[i].evc_mac_addr); + } while (sriov_mac_addr_assigned(vport_config, i, + vport_config[i].evc_mac_addr)); + } + + sriov->vport_config = vport_config; + +done: + sfc_log_init(sa, "done"); + return 0; + +fail_alloc_vport_config: + sriov->num_vfs = 0; + return rc; +} + +void +sfc_sriov_detach(struct sfc_adapter *sa) +{ + struct sfc_sriov *sriov = &sa->sriov; + + sfc_log_init(sa, "entry"); + + free(sriov->vport_config); + sriov->vport_config = NULL; + sriov->num_vfs = 0; + + sfc_log_init(sa, "done"); +} + +int +sfc_sriov_vswitch_create(struct sfc_adapter *sa) +{ + struct sfc_sriov *sriov = &sa->sriov; + efx_vport_config_t *vport_config = sriov->vport_config; + int rc; + + sfc_log_init(sa, "entry"); + + if (sriov->num_vfs == 0) { + sfc_log_init(sa, "no VFs enabled"); + goto done; + } + + rc = efx_evb_init(sa->nic); + if (rc != 0) { + sfc_err(sa, "EVB init failed %d", rc); + goto fail_evb_init; + } + + RTE_BUILD_BUG_ON(sizeof(sa->port.default_mac_addr) != + sizeof(vport_config[0].evc_mac_addr)); + rte_ether_addr_copy(&sa->port.default_mac_addr, + (struct rte_ether_addr *)vport_config[0].evc_mac_addr); + + rc = efx_evb_vswitch_create(sa->nic, sriov->num_vfs + 1, + vport_config, &sriov->vswitch); + if (rc != 0) { + sfc_err(sa, "EVB vSwitch create failed %d", rc); + goto fail_evb_vswitch_create; + } + +done: + sfc_log_init(sa, "done"); + return 0; + +fail_evb_vswitch_create: + efx_evb_fini(sa->nic); + +fail_evb_init: + return rc; +} + +void +sfc_sriov_vswitch_destroy(struct sfc_adapter *sa) +{ + struct sfc_sriov *sriov = &sa->sriov; + int rc; + + sfc_log_init(sa, "entry"); + + if (sriov->num_vfs == 0) + goto done; + + rc = efx_evb_vswitch_destroy(sa->nic, sriov->vswitch); + if (rc != 0) + sfc_err(sa, "efx_evb_vswitch_destroy() failed %d", rc); + + sriov->vswitch = NULL; + + efx_evb_fini(sa->nic); + +done: + sfc_log_init(sa, "done"); +} diff --git a/drivers/net/sfc/sfc_sriov.h b/drivers/net/sfc/sfc_sriov.h new file mode 100644 index 0000000000..2e2e1c69f1 --- /dev/null +++ b/drivers/net/sfc/sfc_sriov.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * + * Copyright(c) 2019-2020 Xilinx, Inc. + * Copyright(c) 2019 Solarflare Communications Inc. + * + * This software was jointly developed between OKTET Labs (under contract + * for Solarflare) and Solarflare Communications, Inc. + */ + +#ifndef _SFC_SRIOV_H +#define _SFC_SRIOV_H + +#include "efx.h" + +#ifdef __cplusplus +extern "C" { +#endif + +struct sfc_sriov { + /** Number of enabled virtual functions */ + unsigned int num_vfs; + /** PF and VFs vPorts configuration */ + efx_vport_config_t *vport_config; + /** vSwitch handle */ + efx_vswitch_t *vswitch; +}; + +struct sfc_adapter; + +int sfc_sriov_attach(struct sfc_adapter *sa); +void sfc_sriov_detach(struct sfc_adapter *sa); + +int sfc_sriov_vswitch_create(struct sfc_adapter *sa); +void sfc_sriov_vswitch_destroy(struct sfc_adapter *sa); + +#ifdef __cplusplus +} +#endif +#endif /* _SFC_SRIOV_H */