From patchwork Fri Oct 27 02:59:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chaoyong He X-Patchwork-Id: 133463 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D37EC4320F; Fri, 27 Oct 2023 05:03:40 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7FFA442E69; Fri, 27 Oct 2023 05:01:16 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2129.outbound.protection.outlook.com [40.107.220.129]) by mails.dpdk.org (Postfix) with ESMTP id 603DD42E27 for ; Fri, 27 Oct 2023 05:01:05 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cEaFqHQ7bH22u3NFCnB3c9oOXYFo6AvVkZT9/4MyulBzcUg6eh5+VmPAm8j3IsImP+LrLqECGFYZO/wNVt3D+eLcEpI8VTVqIUugOF+Ku2A/s19TgYtdehuAWIGjVKzSc8vhmgys+Ahekgkt7MKxdOdholc7ID4QEgisdzI2qf70Bvc7uemPq8mR9MaCg98D+m0pHvzBuT24vZnogKnFJY/ZZvcD5iWUOAF2IptFl8A6DLMnahtskSUMg7B8E/k1s0/HRe9WUGq144EUoPL0u3YKmWvpMHUHsI32ypYv9mtGL6hWRiO2V2V8Q0BsFUjGUF+hf/qN68GiIB5ZaOuwKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jk/CbMQztnVDf2I4hFCWS4AwJwyfhhq7mTjnyIunuEA=; b=VMLtWsQ264cLCAtnJXFuaaznRYd5xJpqBb1DiTjxIAG7boKgOrZmW0GpcAqyeBi9v4octRiIMh6ZUpwN620A2G8pwHvyhH0VRgqs+bbQO2WgUNGYGG2oB0CbbC4RnpaezicJn2UXX6p6Gs1YA+BHhqltHRsRoAn4clxy5/rHwx/xxBYLCgbVKUbwzWJ+ge4e0tQ2AWPWu98T/laYi6wPyJU1ioEJhk60brm+zljnQPAHFqil7uqyUiApwrap3sqVT0wW7KNgDtDoo4pWOhWuJDeFPi+BMCizUwXDcikZJENVT3IqF/BTb7EmPk7eSoeBqi535pPno0fA6bC9KREbew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jk/CbMQztnVDf2I4hFCWS4AwJwyfhhq7mTjnyIunuEA=; b=GTGl9NczMeInaIYssl/q93ULarO5bfg2AygXAtdHjyAZYdDqVXMddHoI2is7cVdThByRXP+2jdezQaLvz9XepXYN6DZsvLnCsvFtuzgKPyaYrhr17jSNdDwOWrf47Sm+Td64uI8WM/gqGtpvzUPu1uh/gEgztYpsTsANj3GmhDQ= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) by DM6PR13MB4493.namprd13.prod.outlook.com (2603:10b6:5:1b9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.33; Fri, 27 Oct 2023 03:01:03 +0000 Received: from SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::45b3:d83c:cff2:4a1b]) by SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::45b3:d83c:cff2:4a1b%6]) with mapi id 15.20.6933.022; Fri, 27 Oct 2023 03:01:03 +0000 From: Chaoyong He To: dev@dpdk.org Cc: oss-drivers@corigine.com, Chaoyong He , Shujing Dong , Long Wu , Peng Zhang Subject: [PATCH v4 20/24] vdpa/nfp: add remap PCI memory Date: Fri, 27 Oct 2023 10:59:57 +0800 Message-Id: <20231027030001.602639-21-chaoyong.he@corigine.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20231027030001.602639-1-chaoyong.he@corigine.com> References: <20231026064324.177531-1-chaoyong.he@corigine.com> <20231027030001.602639-1-chaoyong.he@corigine.com> X-ClientProxiedBy: BYAPR08CA0065.namprd08.prod.outlook.com (2603:10b6:a03:117::42) To SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR13MB5545:EE_|DM6PR13MB4493:EE_ X-MS-Office365-Filtering-Correlation-Id: 8785ec08-cc9a-4fde-c4ca-08dbd698f4d9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jxDmGhtlpYBo41hXmRexbF3oszKILLpKcfjgmfX+dFnBKnHERlFqoW5pK04/CKiAbq22Bq+PJ0uroKZk/WfeeUuQx6qeKai4BHfCNooYFT4GsmV8g73G1Onuv1vyYkiaE39L4k+8IpQqmAIDve2oc8RVM3Bvc1Bi1JVVOOV8GFZ2Fbd/Rh1J5e9QrR3JzxvplA+ppA0IoGdIGl4FAqQC3cu0ZhzM7Fw1+XyIH5lFuFwclBlWpb/mnkCCvbISFOUQmCQw2q2t9h3rLmxi9JX0fEovl5hjRi1jZ06WbfJYXGDdCAhOMzgm2ac7XPjSU4wFD6yEDR+MRBzexeRTz4ieIoypki0LO/dtQu32KKocXvdPOHJTcCqORYesRTzCeM9c/JlRweMq/pPg3jZafRG1IXHKCnaui7qUiGqSHe7HdttkMg/s41JWu+axjJPm6dTIEAMoq2qTpl3B/oipAZa87S24y4nIrgNB0Sl1byWDDhBVABGsewfyfQSyZaRyK0sgqf/lu9/CBMY0L7kdoFh/Jp0SEiUiT7O0Tr0KoCpob/dTnI53+whNO/I0mW6M2ieE2jbJKZ8RvnD1AC9+KJ1vhRaWeFIJHPbUdd182XKvyCBvvUPOcqONUyVAGXRrHV71Hnt/OlFzzXDapHSnDTzSTfHg1XNTvK5zBu1KBFiqL+k= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR13MB5545.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(396003)(136003)(366004)(346002)(376002)(39840400004)(230922051799003)(186009)(1800799009)(64100799003)(451199024)(38350700005)(83380400001)(4326008)(52116002)(2906002)(38100700002)(107886003)(44832011)(8936002)(26005)(2616005)(1076003)(36756003)(8676002)(54906003)(6666004)(5660300002)(66476007)(6916009)(6506007)(316002)(6486002)(6512007)(41300700001)(66556008)(66946007)(86362001)(478600001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /jMHX6Lju//53W0YXCcYFY+6Q+7R4ojdPZd3peRMnsjVkVWwa0dI8ij4BpnSb2uyFsQg/M5fvE8xko1EPDlJT+cjh6JbwkmKm9zrGB+9s7iR9RXR88s/PCrKBpphJxPC+eRRbPs8Pjd2W1kks8T+nOyP8ZqY+Z1Ov0WyzbmdK0vfEACLi51eO2j/2507HsojfRF9fOfBVTd1/NQlGl7wH0qHlQlkWljtu3Jxjvx2R6Giv28zaHRQ2OlOglA1XTGzbiiCiuPcwoJ0bihHq2QT/zYH7nHHlvJSM8AXZg+4duwn9Wxj7Wk/8z4A9myXe7d+TM6EVEeQHp625ezKgEz14t+/ZVKOOJm0bwBgPKSMAAIaxoaRfnaN9DGKBBH41Gt9q7jYNwX5B62HL8GFTwZS+I8kEgfHCA8LJaBWoqkL/M0DetMu0rNyC2Jdk/yD+M2QgYYniRs7fOX5DJmcY1bWH3iOKyWYOU+ChvmB9UnfBvSAUlPgA7DVWYhbP0VVprqHIzeXlsrcU2rZtvcw5BzfU4PePNCRIu1E5dxNY4zx2tfgDUT6g8EbvL2Uw2R1/AFcS5hUpnnkm7xscmVKzCL3JZmWtA05caGyJg/ysanAmdRJ57Ii0pJ2Z2XepBzHlKleRT/u1hH9vIBSLixMBve7NsY/NFPdPUj+neFPlXdKCipjSjSFgmmZAfhiz0GP4hMh/HSPf1RgVJPxPu4XO91UvmlQIwfi7QizyfeuN/ppQ2Sy9c9N+rxNfORBNFIeCbm1GbYNPJEhWoD0eUvYuF2aL/5q3RR3eEJ0+BiUnBg9nemAG1KBEiO1kW/8lkUr1EeyuPV4U91q8PUeEaCm4gN5x39HM16F+/FTqW+vhPiVsANuUqQOD2Kt4GWeWkZ/2iWr/57LqtsY2TwlsTZZUA9ZNm5TXrrICxoxV0zwmHBmppLn9Y5ZNYPPNmkR9O2GTzlYEeZKFs3/UzLiPSPGKMF08rxQQ4zsnjGjakIcKFYUQMjIf/L5buQNW6e0VyElaKqwFes8HsJFcPuGANjnV6B7BhU91NkBvoZOml4utbPl94vwKPnpX8PuDvgeEE6NyWLQUTCBWZeYKXQ2CkZMv6UboB+yoLy/qmpVwfTz4cttAs0YpK0ajD5edhlODz2YaN6R98ATftw/C3GwAzzFKqJcZuJqKBdS+Pyjq36eJnVQGqIzEHmnQBsFTMWRI0Joi+Qp2OzZ1GojY5LGK7w1KYVyIK/Z3zcCTAdb+QJHVvFlsb08IJcqyIsqR0VWO9hqir1UxiSFKNmUXkz1i4IBcmRStBN9Z+WjfjXrFchvVkz1TwwMQkLVSwARI3WBKFu5U7vG9doyPhT+CCW6vSkpsHYg081CNO3Kio8zCgWe/CLQHdmSHmeVquVa812QMrUrnM+ceO4FIDI2WMZjgsutlwfTIYF8dJkdn9Hl5AD98Gf+AajfH5OeeJatSVs07vnDwNCZtSipxo2vDhv8nxB5oInP4tckR9ylC5P90PWN2htxwz0TZaUP8lvHsXQIJHYG+LMqOBhLAw4woBph3CjWoZUekDxwjfrjqExmBPoY9YLv8DIKBUlxYu2XEYumRRTePDBBnRPM1v8UKl2QjRaNzif43A== X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8785ec08-cc9a-4fde-c4ca-08dbd698f4d9 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR13MB5545.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Oct 2023 03:01:03.7221 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: c2qo7C7fTo1DQMkmZJu7UA/IzOTrmnRbp92+bTAvsiLK5KJUlLg906UKnTYajicVOKDoSYmlyVAB5tqjZzwkwdnKZWQmZGGnPWpDDziOR2o= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR13MB4493 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add the logic of remap PCI memory. Signed-off-by: Chaoyong He Signed-off-by: Shujing Dong Reviewed-by: Long Wu Reviewed-by: Peng Zhang --- drivers/vdpa/nfp/nfp_vdpa.c | 65 ++++++++++++++++++++++++++++++++++++- 1 file changed, 64 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c index ef4ae4946a..824b0d681a 100644 --- a/drivers/vdpa/nfp/nfp_vdpa.c +++ b/drivers/vdpa/nfp/nfp_vdpa.c @@ -7,6 +7,7 @@ #include #include +#include #include #include "nfp_vdpa_log.h" @@ -16,6 +17,11 @@ struct nfp_vdpa_dev { struct rte_pci_device *pci_dev; struct rte_vdpa_device *vdev; + + int vfio_container_fd; + int vfio_group_fd; + int vfio_dev_fd; + int iommu_group; }; struct nfp_vdpa_dev_node { @@ -53,12 +59,62 @@ nfp_vdpa_find_node_by_pdev(struct rte_pci_device *pdev) return NULL; } +static int +nfp_vdpa_vfio_setup(struct nfp_vdpa_dev *device) +{ + int ret; + char dev_name[RTE_DEV_NAME_MAX_LEN] = {0}; + struct rte_pci_device *pci_dev = device->pci_dev; + + rte_pci_unmap_device(pci_dev); + + rte_pci_device_name(&pci_dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN); + rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name, + &device->iommu_group); + + device->vfio_container_fd = rte_vfio_container_create(); + if (device->vfio_container_fd < 0) + return -1; + + device->vfio_group_fd = rte_vfio_container_group_bind( + device->vfio_container_fd, device->iommu_group); + if (device->vfio_group_fd < 0) + goto container_destroy; + + DRV_VDPA_LOG(DEBUG, "container_fd=%d, group_fd=%d,\n", + device->vfio_container_fd, device->vfio_group_fd); + + ret = rte_pci_map_device(pci_dev); + if (ret != 0) + goto group_unbind; + + device->vfio_dev_fd = rte_intr_dev_fd_get(pci_dev->intr_handle); + + return 0; + +group_unbind: + rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group); +container_destroy: + rte_vfio_container_destroy(device->vfio_container_fd); + + return -1; +} + +static void +nfp_vdpa_vfio_teardown(struct nfp_vdpa_dev *device) +{ + rte_pci_unmap_device(device->pci_dev); + rte_vfio_container_group_unbind(device->vfio_container_fd, device->iommu_group); + rte_vfio_container_destroy(device->vfio_container_fd); +} + struct rte_vdpa_dev_ops nfp_vdpa_ops = { }; static int nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) { + int ret; struct nfp_vdpa_dev *device; struct nfp_vdpa_dev_node *node; @@ -75,10 +131,14 @@ nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) device->pci_dev = pci_dev; + ret = nfp_vdpa_vfio_setup(device); + if (ret != 0) + goto free_device; + device->vdev = rte_vdpa_register_device(&pci_dev->device, &nfp_vdpa_ops); if (device->vdev == NULL) { DRV_VDPA_LOG(ERR, "Failed to register device %s", pci_dev->name); - goto free_device; + goto vfio_teardown; } node->device = device; @@ -88,6 +148,8 @@ nfp_vdpa_pci_probe(struct rte_pci_device *pci_dev) return 0; +vfio_teardown: + nfp_vdpa_vfio_teardown(device); free_device: free(device); free_node: @@ -118,6 +180,7 @@ nfp_vdpa_pci_remove(struct rte_pci_device *pci_dev) pthread_mutex_unlock(&vdpa_list_lock); rte_vdpa_unregister_device(device->vdev); + nfp_vdpa_vfio_teardown(device); free(device); free(node);