From patchwork Thu Nov 2 02:23:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chaoyong He X-Patchwork-Id: 133757 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4A99043267; Thu, 2 Nov 2023 03:25:09 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0E64542E90; Thu, 2 Nov 2023 03:24:07 +0100 (CET) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2134.outbound.protection.outlook.com [40.107.220.134]) by mails.dpdk.org (Postfix) with ESMTP id 968F842E80 for ; Thu, 2 Nov 2023 03:24:04 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=keJNo11GgXucn3fjI9Q+ijJCeo1W1nOguFHdlnEnaMZvRUWv6Cb1sY9eAU0O6pF7W1EdhPB8MBJRZQ7Ds0aoppyZY19OitFJFf+iKpcFne09u65KK7xML6PkE7w4drf8PiLCUMtL/O9MMwo3HG71IHY7GEMsAevjoLcXIoOBOcOR5JZ8jfqC5LHWpxiO9Ft1KtFRSwQRgJzqo86TVTOv0NRkHbCx/q+0S6FvAG5nNGyGyZiZtkrBjBANRyYaGLs/X9aRpXwFQQI5LT9bUkGQ9XkbbwsIfJx/GDu7VKpWZQSDIlqSiCizOD0D1l0v4NpGnEdQIA/bkQrcGlCSOUCYig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VzQZ4OilXKglucq38wwvqpuoEyuGYuLzQ3T6ZF39X7E=; b=Ed4YCjy1ofDdkT3eddFgdGTQNic6r9REzd1aNIMoa4kDKqUw7AAkM/Tw77yhDQRwQXpHnUS/RS17GdZ4/a32AysRQ1EO6t0V67ceetH1bk3LmCfRzS4n1zVFPI9Sfb7dD354Chs2ki9jbhIIXJK2UicNJ3TDLUntEuGQOSS5jIso0hJkuPz4kuhb3CYX+qa7WrFpIRI9O22J/4dc2+ecVCWXPeer5kECpt0hVih+sHNXeejDEZwolVmHpCKqk0rQxxnqgaQcv+zWh6tpH+qixX6/Bo7zHpMwqVVR0NfqJz/bsVYKZuvmcwb86OxdNdQmjc4VXSVzbEItVULuBO/qyQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VzQZ4OilXKglucq38wwvqpuoEyuGYuLzQ3T6ZF39X7E=; b=gz2KeBYK62KlKCOl2kqlkVarGtOMoKxWAZoYmu9Kf/u3QWXfh3MnTJUSvLOI7jifOH6T62yMOBJHl17GCmgEu4SVkBjFyIIK7lvX7Lm67gnJfJOBuTCMLikSzCSjMANZAtgoOz18uD3/s5a72Fnc9zYX3MW6E25XmeewdfgzOfI= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) by MN2PR13MB3709.namprd13.prod.outlook.com (2603:10b6:208:1e1::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.19; Thu, 2 Nov 2023 02:24:02 +0000 Received: from SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::45b3:d83c:cff2:4a1b]) by SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::45b3:d83c:cff2:4a1b%6]) with mapi id 15.20.6933.027; Thu, 2 Nov 2023 02:24:02 +0000 From: Chaoyong He To: dev@dpdk.org Cc: oss-drivers@corigine.com, Peng Zhang , Chaoyong He , Long Wu Subject: [PATCH 10/11] net/nfp: introduce keepalive mechanism for multiple PF Date: Thu, 2 Nov 2023 10:23:20 +0800 Message-Id: <20231102022321.2254224-11-chaoyong.he@corigine.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20231102022321.2254224-1-chaoyong.he@corigine.com> References: <20231102022321.2254224-1-chaoyong.he@corigine.com> X-ClientProxiedBy: BY5PR20CA0030.namprd20.prod.outlook.com (2603:10b6:a03:1f4::43) To SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR13MB5545:EE_|MN2PR13MB3709:EE_ X-MS-Office365-Filtering-Correlation-Id: e9065ef6-b94e-4681-88dc-08dbdb4ac772 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2xTEvfzpYFSZQNm/+G+MJqSFIcqdMfWApCDTdj+L8CxMxUB8N/1EqJD06UPD2J93WwRuAGqd+bEXFD2rPLRJ1xznSYgpUG34HQrSPSBC1ZSLoPMScF79waQFmQE3FNizzRKs+6pPAZ6eLBUN9xMyQXbbekE1tnuVI0zPKD6CY/35h+XFy9ru3mQQ73y5Qvuivf0V/mWbiCkKI1HrEdCN6Nt5jnHqSgykRnMnbzDANoAJzPl+75kxIswtaJLGHZuksl4HudFCr0kUVWPzh/QBMNmia9v+gA/AiKBTW02hDGXBXFpqbAHkqePfrLgKBdLsPuNv6a/RHPv9MuDwvev2xcgRhYPHSkejsDom1AwIUl3soS2D+w6dPYYBjSp/zrjL4mQieNTexyHH93J5UXuR5GRuAs24ztPDujZZUIxBNQzsntQC9GVk6BW4PeOnlMZ2fI/o9wptwBwG0RSLHHvXwMws/bjoMI7Cn6nQ1uG/lq2R2XyzieAvx6340kdsiMuMsO6JAHaWOUXeSf2rRlSsPHTx9fuBZMSu7u5f2KIzHBu57rf+UEMXVIKNfUFwa7WpqYuknxvTcOt8zmdkhUGr2lJkP4PeIUUMfmUp5/w/ZkDeJwtTZaE6MhhA+aegX8ssSafLvP2iJxKTexAAkLogszLrxk+SUEE4hjpnIRmpaKU= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR13MB5545.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(346002)(136003)(376002)(366004)(39840400004)(396003)(230922051799003)(186009)(1800799009)(64100799003)(451199024)(38350700005)(478600001)(52116002)(36756003)(6486002)(6506007)(107886003)(38100700002)(83380400001)(41300700001)(6666004)(4326008)(8676002)(8936002)(2616005)(44832011)(2906002)(316002)(1076003)(6916009)(86362001)(26005)(66946007)(54906003)(6512007)(66556008)(66476007)(5660300002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: GMeJisCMahy3R20fceqyTWCtCtlNKbSN7cP5CX59h9zgGcCxaP0EaM/5cr1GuI35JpMB/8qhDcmkb+Vf2ZR6YAK5XGm0EGWWMlhVedekXAA0p+9S9p0xgy58xJY+TYbqhk/lGLWKnWpcS8AzXjz4VHUbn1m7CqmJfXYCdyPSLd6M/uGf0YXSfxiQfq2r/RFHejTN3WHdLaV7lGEhTAPPRac5EVV1SzKoRmUvqrtfh4vjkYFKQhmmuq7GkkicTJHp4BakFjpi63eNwEOtpA/7m4BnUFbRdteDMLlWi2K15BYvOHo6FUiVtEtTV5mTjOknAeErm7v0wMr600pb2WeNZT5oNbtfLk/QQC1d5uS3RWbQb41y/R68xqk984Ca0fhoKyIJjRRF+D3sAXO5zPBPhboZoajfE/F/IH77mXO9rP+Om9k0f026pSTwYYsAGPFHdLgc6hl9MBNxrBuF8BcG+y5QaDa5h0ROxaLlvf0CNXFPpjhj5u9pWyFJHwRjnYEY0tN2DY3pcsZQamZUIlpLS7dO/bk77HH/wNbRer3VlhRy8UHfgm+we7RjB6FIBCkLGp7bl67INVB9geHvXT/kFDXjlnNkCmNMavi83iggUD5B3vhFWOjc4fQ6FOE38mPjSjZd1l3G4hxbOE922HQUoqQBO/yxQ09vB1SwAAhPhL8yh6TbC9SFNWkkFWOYWGPjYAyKKUKLEavnFOtW1bdU4v0JpA+1u3Z1txSmT2ylVRfWlVusYlo1tvwZmjhysvvU/krJDnD2eZQkyBn2ppZ9UzxrPb3tCWRWNSCfuWa7OeRnDrfIADWR1d0B2WeRg6CRR5raZGQnOb3z1UxD1suFNouDHNWDB6MOLA2GEHQ/QM1awJ7wVeqtQCT86D7iNn9+fZoUKlv5f5CD52Yd2P7lBpCUtVNG0KfY+0audrnSM+JtL3a5YeB8zT8KHkpZA9r/lWmmmy7i6T5+rz/Nv3CPikvtze0EuM+qZOPrDqjHyTwhQYKmXYAFmV9qKGnIpZHDLLG5nV+eNbrWj35KxC1JWRS5Qci7CyiczOApsqEH+lJqNy9T0vgQLRnbIZdZqLyU9oLT7zAoQT9gA2N1g1/MP+KkoyTSo+v11Tx6AdjrC1O1vD8uMH0jdzTesUHIqLppnQFX4BIONeeII2OHvSJf0hCjYhedJ6s7SPFAnOqbDLPv0Z6hk1sCY0MPRBJeSoVcpdm4bqFdxG2g6k2U2nfrLamivGxMnVW6ou+9FVQbls52UDU2Q+Q4Q2SEFZX1Fk9aGRtkLtG04lNLnLhY9sFsXSgJKYcN9dKZeRi90iNGt1teXT4GwL8Oz1tvjNMm85La3qjn7Irrk+AKA9J7U610TSAbCxEV7WIoOvSze1MOfE09TIB7CEt5VuVPPaXbY5uZ1FVVgY4BiPHOAmrWXvMcCf6bBNqobPKg//dd51oAHRjot0PudNViSQbhZTAOBjRRi8qhVtNDkzEcnEDpve7xaH1QaymQLzqpBz0qt34HRDh9FE6temlkfypZDp9nQ0bZoMQV3ScU0Bp2slbBAFDHI5UUYPHa8S2fcbijzE3ghPAhqpe4+VIjbhUqvI3iUvYo3vMep3D/ol5qSySjENyEmA== X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: e9065ef6-b94e-4681-88dc-08dbdb4ac772 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR13MB5545.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2023 02:24:02.6307 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: pBDdOtXgtojc2Gg+hshir3yvwm3CvIaAY/XNFKLQ7B1twgpcN7Q/hL54uXYhdbVe6CQQVZH7lHvcAp9DMUHn+2PqP9AxTp9EcjDeonyilQA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR13MB3709 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Peng Zhang In multiple PF scenario, management firmware is in charge of application firmware unloading instead of driver by keepalive mechanism. A new NSP resource area is allocated for keepalive use with name "nfp.beat". Driver sets periodically updates the PFs' corresponding word in "nfp.beat". Management firmware checks these PF's words to learn whether and which PF are alive, and will unload the application firmware if no PF is running. Signed-off-by: Peng Zhang Reviewed-by: Chaoyong He Reviewed-by: Long Wu --- drivers/net/nfp/nfp_ethdev.c | 173 ++++++++++++++++++++++++- drivers/net/nfp/nfp_net_common.h | 17 +++ drivers/net/nfp/nfpcore/nfp_resource.h | 3 + 3 files changed, 189 insertions(+), 4 deletions(-) diff --git a/drivers/net/nfp/nfp_ethdev.c b/drivers/net/nfp/nfp_ethdev.c index 96f0ae3fe3..bbc0109f5f 100644 --- a/drivers/net/nfp/nfp_ethdev.c +++ b/drivers/net/nfp/nfp_ethdev.c @@ -5,6 +5,8 @@ * Small portions derived from code Copyright(c) 2010-2015 Intel Corporation. */ +#include + #include #include @@ -16,6 +18,7 @@ #include "nfpcore/nfp_rtsym.h" #include "nfpcore/nfp_nsp.h" #include "nfpcore/nfp6000_pcie.h" +#include "nfpcore/nfp_resource.h" #include "nfp_cpp_bridge.h" #include "nfp_ipsec.h" @@ -234,6 +237,79 @@ nfp_function_id_get(const struct nfp_pf_dev *pf_dev, return phy_port; } +static void +nfp_net_beat_timer(void *arg) +{ + uint64_t cur_sec; + struct nfp_multi_pf *multi_pf = arg; + + cur_sec = rte_rdtsc(); + nn_writeq(cur_sec, multi_pf->beat_addr + NFP_BEAT_OFFSET(multi_pf->function_id)); + + /* Beat once per second. */ + if (rte_eal_alarm_set(1000 * 1000, nfp_net_beat_timer, + (void *)multi_pf) < 0) { + PMD_DRV_LOG(ERR, "Error setting alarm"); + } +} + +static int +nfp_net_keepalive_init(struct nfp_cpp *cpp, + struct nfp_multi_pf *multi_pf) +{ + uint8_t *base; + uint64_t addr; + uint32_t size; + uint32_t cpp_id; + struct nfp_resource *res; + + res = nfp_resource_acquire(cpp, NFP_RESOURCE_KEEPALIVE); + if (res == NULL) + return -EIO; + + cpp_id = nfp_resource_cpp_id(res); + addr = nfp_resource_address(res); + size = nfp_resource_size(res); + + nfp_resource_release(res); + + /* Allocate a fixed area for keepalive. */ + base = nfp_cpp_map_area(cpp, cpp_id, addr, size, &multi_pf->beat_area); + if (base == NULL) { + PMD_DRV_LOG(ERR, "Failed to map area for keepalive."); + return -EIO; + } + + multi_pf->beat_addr = base; + + return 0; +} + +static void +nfp_net_keepalive_uninit(struct nfp_multi_pf *multi_pf) +{ + nfp_cpp_area_release_free(multi_pf->beat_area); +} + +static int +nfp_net_keepalive_start(struct nfp_multi_pf *multi_pf) +{ + if (rte_eal_alarm_set(1000 * 1000, nfp_net_beat_timer, + (void *)multi_pf) < 0) { + PMD_DRV_LOG(ERR, "Error setting alarm"); + return -EIO; + } + + return 0; +} + +static void +nfp_net_keepalive_stop(struct nfp_multi_pf *multi_pf) +{ + /* Cancel keepalive for multiple PF setup */ + rte_eal_alarm_cancel(nfp_net_beat_timer, (void *)multi_pf); +} + /* Reset and stop device. The device can not be restarted. */ static int nfp_net_close(struct rte_eth_dev *dev) @@ -284,6 +360,10 @@ nfp_net_close(struct rte_eth_dev *dev) /* Now it is safe to free all PF resources */ PMD_INIT_LOG(INFO, "Freeing PF resources"); + if (pf_dev->multi_pf.enabled) { + nfp_net_keepalive_stop(&pf_dev->multi_pf); + nfp_net_keepalive_uninit(&pf_dev->multi_pf); + } nfp_cpp_area_free(pf_dev->ctrl_area); nfp_cpp_area_free(pf_dev->qc_area); free(pf_dev->hwinfo); @@ -693,11 +773,92 @@ nfp_fw_unload(struct nfp_cpp *cpp) nfp_nsp_close(nsp); } +static int +nfp_fw_reload(struct rte_pci_device *dev, + struct nfp_nsp *nsp, + char *card_desc) +{ + int err; + + nfp_nsp_device_soft_reset(nsp); + err = nfp_fw_upload(dev, nsp, card_desc); + if (err != 0) + PMD_DRV_LOG(ERR, "NFP firmware load failed"); + + return err; +} + +static int +nfp_fw_loaded_check_alive(struct rte_pci_device *dev, + struct nfp_nsp *nsp, + char *card_desc, + const struct nfp_dev_info *dev_info, + struct nfp_multi_pf *multi_pf) +{ + int offset; + uint32_t i; + uint64_t beat; + uint32_t port_num; + + /* + * If the beats of any other port changed in 3s, + * we should not reload the firmware. + */ + for (port_num = 0; port_num < dev_info->pf_num_per_unit; port_num++) { + if (port_num == multi_pf->function_id) + continue; + + offset = NFP_BEAT_OFFSET(port_num); + beat = nn_readq(multi_pf->beat_addr + offset); + for (i = 0; i < 3; i++) { + sleep(1); + if (nn_readq(multi_pf->beat_addr + offset) != beat) + return 0; + } + } + + return nfp_fw_reload(dev, nsp, card_desc); +} + +static int +nfp_fw_reload_for_multipf(struct rte_pci_device *dev, + struct nfp_nsp *nsp, + char *card_desc, + struct nfp_cpp *cpp, + const struct nfp_dev_info *dev_info, + struct nfp_multi_pf *multi_pf) +{ + int err; + + err = nfp_net_keepalive_init(cpp, multi_pf); + if (err != 0) + PMD_DRV_LOG(ERR, "NFP write beat failed"); + + if (nfp_nsp_fw_loaded(nsp)) + err = nfp_fw_loaded_check_alive(dev, nsp, card_desc, dev_info, multi_pf); + else + err = nfp_fw_reload(dev, nsp, card_desc); + if (err != 0) { + nfp_net_keepalive_uninit(multi_pf); + return err; + } + + err = nfp_net_keepalive_start(multi_pf); + if (err != 0) { + nfp_net_keepalive_uninit(multi_pf); + PMD_DRV_LOG(ERR, "NFP write beat failed"); + } + + return err; +} + static int nfp_fw_setup(struct rte_pci_device *dev, struct nfp_cpp *cpp, struct nfp_eth_table *nfp_eth_table, - struct nfp_hwinfo *hwinfo) + struct nfp_hwinfo *hwinfo, + const struct nfp_dev_info *dev_info, + struct nfp_multi_pf *multi_pf) { int err; char card_desc[100]; @@ -736,8 +897,10 @@ nfp_fw_setup(struct rte_pci_device *dev, return -EIO; } - nfp_nsp_device_soft_reset(nsp); - err = nfp_fw_upload(dev, nsp, card_desc); + if (multi_pf->enabled) + err = nfp_fw_reload_for_multipf(dev, nsp, card_desc, cpp, dev_info, multi_pf); + else + err = nfp_fw_reload(dev, nsp, card_desc); nfp_nsp_close(nsp); return err; @@ -1007,7 +1170,8 @@ nfp_pf_init(struct rte_pci_device *pci_dev) nfp_eth_set_configured(cpp, index, 0); } - if (nfp_fw_setup(pci_dev, cpp, nfp_eth_table, hwinfo) != 0) { + if (nfp_fw_setup(pci_dev, cpp, nfp_eth_table, hwinfo, + dev_info, &pf_dev->multi_pf) != 0) { PMD_INIT_LOG(ERR, "Error when uploading firmware"); ret = -EIO; goto eth_table_cleanup; @@ -1092,6 +1256,7 @@ nfp_pf_init(struct rte_pci_device *pci_dev) free(sym_tbl); fw_cleanup: nfp_fw_unload(cpp); + nfp_net_keepalive_stop(&pf_dev->multi_pf); eth_table_cleanup: free(nfp_eth_table); hwinfo_cleanup: diff --git a/drivers/net/nfp/nfp_net_common.h b/drivers/net/nfp/nfp_net_common.h index a40ddfd985..db2e5842ca 100644 --- a/drivers/net/nfp/nfp_net_common.h +++ b/drivers/net/nfp/nfp_net_common.h @@ -36,6 +36,19 @@ /* Number of supported physical ports */ #define NFP_MAX_PHYPORTS 12 +#define NFP_BEAT_LENGTH 8 + +/* + * Each PF has corresponding word to beat: + * Offset | Usage + * 0 | magic number + * 8 | beat of Pf0 + * 16 | beat of Pf1 + * 24 | beat of Pf2 + * 32 | beat of Pf3 + */ +#define NFP_BEAT_OFFSET(_x) (((_x) + 1) * NFP_BEAT_LENGTH) + /* Firmware application ID's */ enum nfp_app_fw_id { NFP_APP_FW_CORE_NIC = 0x1, @@ -59,6 +72,10 @@ struct nfp_multi_pf { bool enabled; /** Function index */ uint8_t function_id; + /** Pointer to CPP area for beat to keepalive */ + struct nfp_cpp_area *beat_area; + /** Pointer to mapped beat address used for keepalive */ + uint8_t *beat_addr; }; struct nfp_pf_dev { diff --git a/drivers/net/nfp/nfpcore/nfp_resource.h b/drivers/net/nfp/nfpcore/nfp_resource.h index f49c99e462..7ce4727b37 100644 --- a/drivers/net/nfp/nfpcore/nfp_resource.h +++ b/drivers/net/nfp/nfpcore/nfp_resource.h @@ -17,6 +17,9 @@ /* Service Processor */ #define NFP_RESOURCE_NSP "nfp.sp" +/* Keepalive */ +#define NFP_RESOURCE_KEEPALIVE "nfp.beat" + /* Opaque handle to a NFP Resource */ struct nfp_resource;