From patchwork Mon Aug 5 02:12:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chaoyong He X-Patchwork-Id: 142887 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 17FA14573B; Mon, 5 Aug 2024 04:13:57 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8B4FD402F1; Mon, 5 Aug 2024 04:13:21 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2138.outbound.protection.outlook.com [40.107.237.138]) by mails.dpdk.org (Postfix) with ESMTP id E068D402E5 for ; Mon, 5 Aug 2024 04:13:18 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=GlpxL3RUHyb6pzGK7vSaOf0ln+0JBdsni7naV/foZ2ckyv19PwS5KExNZ4xVVjhjKV6fFMVZGT7Jj4tsuhNfdZlHNmIbZ0esC6lN69JjBzu+htJ8jUURjTjgxBJb+uADSCFE2nfUNziQt/tM8rdaw8VUu2Hjg0T5yAz/Qi1ImJUvEn5U3ixKirGFT0armsC+Uvmnd3cHeaQgwP7T6mKhMEUOa4rzMoM/PMML0FwxJZTcp7dOK2vxyTFACMgLVzz8D5mGeQtUjPCvx+Zhveu2wPwR+o/3gQ3E+de0JHCGU7PFPnYz4Cp2V0WU/7+nkmvs4SDmGeO2D+buJcTlKfY62w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ePhDxkh7AWVCZoBOCiTps1/sYFQr1dzJ1ogaRPsdvFc=; b=Q3M/WP3UkcCTAnxa6dEc1OUbIpeX5K6OPq0HY+wVhQn0D16/2rsgTXk+ObPVruyjRsEvhw+Di62rYlSvhnY0VOSOrMyeaGj+vK288yidfn+Banpw3/mMY9QvfMiD4JjCGRJdf4/pR+ioFHkjSnSxj5ySXoowlyJKuL4Yz61EAP93HYVEW40cl2ToHS7K2Eu5AV0KkJhzNdkoCtiIw6KR1PWhwUcCmTbywPePxI4GeNpoQ9eSNKNapfuMq0ScroYp6tFFs9OvOUUxKgbyCQh0puWg3H36D6XWA7UsEtx3I40xyAA/t5YPpZKqy+WK5soDrGgzrYNfkGs60jR1qQQMQg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ePhDxkh7AWVCZoBOCiTps1/sYFQr1dzJ1ogaRPsdvFc=; b=pqaDTsR3Z5qYhR13mYCQ3DZWSzLXnz2xbXMu8/LIk5u0ly6zOfSZZpoWqMOcowbV3pvt+QI3lxPYjePJDAnc3QarJqxrURHMZU2L8dWSYmHKqWXaTpge2HMnkjwZxSF6BW4/CHkq31c0ItK6/M8WkvQXa3KIPw7ujV2WPp0Vkf8= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) by SJ0PR13MB5896.namprd13.prod.outlook.com (2603:10b6:a03:43a::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7828.25; Mon, 5 Aug 2024 02:13:17 +0000 Received: from SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::b900:5f05:766f:833]) by SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::b900:5f05:766f:833%4]) with mapi id 15.20.7828.023; Mon, 5 Aug 2024 02:13:17 +0000 From: Chaoyong He To: dev@dpdk.org Cc: oss-drivers@corigine.com, Xinying Yu , Chaoyong He , Long Wu , Peng Zhang , Maxime Coquelin Subject: [PATCH v4 05/11] vdpa/nfp: add the live migration logic Date: Mon, 5 Aug 2024 10:12:42 +0800 Message-Id: <20240805021248.1051198-6-chaoyong.he@corigine.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240805021248.1051198-1-chaoyong.he@corigine.com> References: <20240617062708.2932037-1-chaoyong.he@corigine.com> <20240805021248.1051198-1-chaoyong.he@corigine.com> X-ClientProxiedBy: BYAPR01CA0050.prod.exchangelabs.com (2603:10b6:a03:94::27) To SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR13MB5545:EE_|SJ0PR13MB5896:EE_ X-MS-Office365-Filtering-Correlation-Id: 0d87cc32-e8c8-4236-8c01-08dcb4f42b62 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|366016|1800799024|376014|52116014|38350700014; X-Microsoft-Antispam-Message-Info: tWNe67Sy18qq1713aZJbmUrDsnXpMk+RXQFC+vYmWT4rsjMNT4QqmeYyrgBpYdWwOptV4seDY0T0qrshKRGIBqfQSar9/0pw0n/1Xr3+GH3NjTONeZUYiOolnLsFC5iVayI74bKvZcFtxixIj6mFuoBNiJg3acG+2wMs6f2el8wcdkQDWVXCmZ8jFXQRRfVJwn5E2CFsrdeearEBGhKVGZcUj7hjLEZllTcyIi1NudXcDSkXMcFG6coIz3lun6eeWd6VG2kIComWGQeFyIoKL+JBZgLEElqRg0K8RdUPDFcyoWv9bFa2pqswdjf6UY+vVzOKpT9b4J1Z0qXDnoT9kgaf2sblyNb/hNIjHar7WiK2z44P4kjX3jNYJdxWEHqZfjeoDifbjnGCAmVVELP6IwU65ZFaXQIaEdRg2w+tqhVROvsQ0PGwMgw+tS06nZBniwLX9ataWceV1fSo27FDOrPFjnVhWS24nC1Wo+vHgW2xb1V5qnCsfB+h3t1vdBPyRmON23MWfBWRvJSov+k0X0GXgZYktV+meeI+DUEjvIrDBeMSMp4qq0TH5cLonbpOOmQe/SkCsiqlTcg3WmNoOQNAZ6MbcwbFQPmvDCKI03jRzDiD4uOP3hXGDk0kB3lQUTEFjqGDMX04/UkkbfZyn3GBek7kdOtMsOGJchn7Go74QUjPY48Y+YQxKEMPrMTjFtLC8B2X4ppOXjFDhcBzg4H2Rl17BHQAUIoifW3nz7rOp69gSCMSlRwu4xlEJmJ6IvwspJKxqeFI8QkTniQcQtikT7Sj3iWFP36W8se+iDj+VS04LER0un9BFM1q8xSDfhnppMIWJeiozSsdwMDlXPp7Mjzi3aJQzTpZHY45SPl+GwmZJparAdj1bXvwdYG0N6Hjf/mNOqJW5Dn2YKUMsAvcbmCUbnecnrf9lVsAKt+l9r/Kt2uRM+ti84RXePFICqF/kadF5jt8DTBGRe15RoHmmvSbAwM+3aG2XWatrvKSxS5b9a34unzJAXXzJh30Kw9xtK4eZ2yxjqNLSERqpplTjWb9kFCe3xPKJlfww1dLSLa44x13yQUgTQ3/mxF86SqqHAXejoAgbj6ESPVZYC67RMQbsrYQaphCzLoZ6XfW1LNtHbvR0UdfF9bGUZQJ9bmok2sL/lButAvjZAYkkyCtSGw4RRMruc5RtlFHiGp1wtu/Wi9MqkyWPXEKFDUT1BvrTtcyay2Z1l//R2+Zw6Fwd16/U2b9FuKmaFBatogr/daTKIMWDKG6Y9Pgp3RmNDtQ771fBUSz3SaKoPkckiDg64SVLQX/iWeqBImaE91J5Q06FWRCPvq0zoFDcKIWwhs7bZbIaMNPTN+cAb4t/TgJUulTemWAS4fWkgCWqnQuHz6cgi0mIvVufTUijWTRSI1aMmnieko/rfJF+vPtRQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR13MB5545.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(1800799024)(376014)(52116014)(38350700014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 4+wZCC8bs5JaSqGzI/w7xO8I4E/IzfRng9iP6kkPdOnIzkm7LSNtYlmd4vnX9XZSUYeufJymnxFr7QpqWdbp8dDSP+iadQqxE1GQpYg3kGcYPUIu8lQgDZHp6mB7d21mXtYyb1WoBVn+RsGNpU9oyreWKB1H2Dg7CmLPnuxneiT2B+36UqKB8g1lYqcw+acfFhjoJSkKOEArWOW6lcaLYY5niNo9OpGyKIImwa8drSD6dMlSeyoHuFHRb7I+6CVOwm4XhRk4R29xU4UyGWcgBGQJzkaxpStSn7hbnX+ZRzJHPBgqM5Uc2kSz146xKV1HmbG0JmyqUY0J9pFWxJcTYa4cyUxbw+z0LUpYvS48iQjhRqGOx9RBv3uMQG+TV5BiISPzxPpJPdHairCTIrH8Y13i7clOWO2eCCFOilgrcXiko4VSo2J7iAr/FJpCjFHx7CZiO2EmgsNr+HPxCQZvJkjICZlOKJtbeWOsRVqa3Kzznmr9VsRn2Jh+W+E8fm/8oCuvchjGNmyq0rX7HacRsHViIOyBAxTvWb+2OWKSckQHURkvgv0ceSopPKrj+B7JFHviNQGsKlN3Siaqtfyzbi4lb9R8NQViYnHzzQjV6EMFiqN+tnZcDCe+85w64AtKcxIhdZZOKNzDZaYoT3ihEazOCwlHK8pe0HdRP6txeb7OL+g1t1wU84sy6d0Ghd//1Fw+zqp/RGU38TFyqLazzcMb4zxkJWatlJqdXvgrclD9camb5af4QeWNd02IwEn4ZXSNz2QVQ6eAzyzvyrkSoKAI6WBYPZx/Ip2CIyVezbc4MiWKzEAbxL0zizhm9hEB4grtOqb0QJ5hi72T1kQGdz7404Swfl3Xa0aTyvqVZYavOTRbAQfhZBHlWRGL5B0t76SKev+B9z/ERrngUvjzlsC4PptmkXCgtIk44DnNBCpex8qAInp0M/9RB64QzMMqqeU2JlkujlASGSLOWoL5BtRwIiTRUIU0hpqDTBJ1h74ejqcT4NldeLC+S574yhTspl/CPcgCp2fDT25aAASyiN4RR3XMKZVokC37PgJCQi0z5bBkaEogStScksPCQqzVtROxF0Bu/tCnZJ974An8qJO0gRERXdqldXWUTJ9WSfKxrze4uvqYPKC4Kr6kgq9FYdUFw2NFjjIB2FvdUszhfIRA0WOqZP0djKCjjUxPuSoq74jIXCC0SIoYehy0u5OYww8TsMCW6yrrJjtLguNynR00gRjR1QgTauGIhcUgAlugTqZOBEUoVbM8XZYcLY66z8syK5BfVVLSJ21rMRCJO/7lWnmpeX8vKVjOJgaVsFtpJpqBKEzGYCZQJzBPmqSbCuzFBWEWYAUnnAMw6mPgPHAPJk9O0UUfCvfjljjwcxke10A4txVtvYtT8jz9FmFdLFUqMQFjbU20MBH2UDDOuiLuN7URSa+DR+OGPtHKUJ1+yr+2A5XSBkXpMzQQ8HxeCF+VtmRTzFE5qSOeiHjBlw+KRsYNRggyepvQ4+3W7NqUSXzQsoeRxjLVj2cPTOkZ30QOA+Ctmkw+P2sIaUIb5MkhpRdlGUxcMWwr3yDrMZ5RazkytvCLW6Y2WgJaaJvmG43gzGxAzyRuELv9MjfL2Q== X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0d87cc32-e8c8-4236-8c01-08dcb4f42b62 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR13MB5545.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Aug 2024 02:13:17.5298 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MQIpYdET4/RjauEe0iX9twsiL7lpv7+wKJyqWLH6NNtP6j1QVVhlsVR9KMzucNr8hpIQv8HKQMmAeoYKzRcMeFRkb+3wllWF5OMjlq67QVo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR13MB5896 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Xinying Yu Add the basic logic of software live migration. Unset the ring notify area to stop the direct IO datapath if the device support, then we can setup the vring relay to help the live migration. Signed-off-by: Xinying Yu Reviewed-by: Chaoyong He Reviewed-by: Long Wu Reviewed-by: Peng Zhang Reviewed-by: Maxime Coquelin --- drivers/vdpa/nfp/nfp_vdpa.c | 66 +++++++++++++++++++++++++++++++- drivers/vdpa/nfp/nfp_vdpa_core.c | 2 + drivers/vdpa/nfp/nfp_vdpa_core.h | 4 ++ 3 files changed, 70 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c index cef80b5476..45092cb0af 100644 --- a/drivers/vdpa/nfp/nfp_vdpa.c +++ b/drivers/vdpa/nfp/nfp_vdpa.c @@ -603,6 +603,30 @@ update_datapath(struct nfp_vdpa_dev *device) return ret; } +static int +nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device) +{ + int ret; + int vid = device->vid; + + /* Stop the direct IO data path */ + nfp_vdpa_unset_notify_relay(device); + nfp_vdpa_disable_vfio_intr(device); + + ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); + if ((ret != 0) && (ret != -ENOTSUP)) { + DRV_VDPA_LOG(ERR, "Unset the host notifier failed."); + goto error; + } + + device->hw.sw_fallback_running = true; + + return 0; + +error: + return ret; +} + static int nfp_vdpa_dev_config(int vid) { @@ -646,8 +670,18 @@ nfp_vdpa_dev_close(int vid) } device = node->device; - rte_atomic_store_explicit(&device->dev_attached, 0, rte_memory_order_relaxed); - update_datapath(device); + if (device->hw.sw_fallback_running) { + device->hw.sw_fallback_running = false; + + rte_atomic_store_explicit(&device->dev_attached, 0, + rte_memory_order_relaxed); + rte_atomic_store_explicit(&device->running, 0, + rte_memory_order_relaxed); + } else { + rte_atomic_store_explicit(&device->dev_attached, 0, + rte_memory_order_relaxed); + update_datapath(device); + } return 0; } @@ -770,7 +804,35 @@ nfp_vdpa_get_protocol_features(struct rte_vdpa_device *vdev __rte_unused, static int nfp_vdpa_set_features(int32_t vid) { + int ret; + uint64_t features = 0; + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + DRV_VDPA_LOG(DEBUG, "Start vid=%d", vid); + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + rte_vhost_get_negotiated_features(vid, &features); + + if (RTE_VHOST_NEED_LOG(features) == 0) + return 0; + + device = node->device; + if (device->hw.sw_lm) { + ret = nfp_vdpa_sw_fallback(device); + if (ret != 0) { + DRV_VDPA_LOG(ERR, "Software fallback start failed"); + return -1; + } + } + return 0; } diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c index 79ecd2b4fc..50eda4cb2c 100644 --- a/drivers/vdpa/nfp/nfp_vdpa_core.c +++ b/drivers/vdpa/nfp/nfp_vdpa_core.c @@ -91,6 +91,8 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off; hw->qcp_cfg = tx_bar + NFP_QCP_QUEUE_ADDR_SZ; + vdpa_hw->sw_lm = true; + vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) | (1ULL << VIRTIO_F_IN_ORDER) | (1ULL << VHOST_USER_F_PROTOCOL_FEATURES); diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h index a8e0d6dd70..0f880fc0c6 100644 --- a/drivers/vdpa/nfp/nfp_vdpa_core.h +++ b/drivers/vdpa/nfp/nfp_vdpa_core.h @@ -36,6 +36,10 @@ struct nfp_vdpa_hw { uint8_t mac_addr[RTE_ETHER_ADDR_LEN]; uint8_t notify_region; uint8_t nr_vring; + + /** Software Live Migration */ + bool sw_lm; + bool sw_fallback_running; }; int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct rte_pci_device *dev);