From patchwork Tue Oct 24 02:28:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chaoyong He X-Patchwork-Id: 133229 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4DCAD431EB; Tue, 24 Oct 2023 04:33:04 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 694FD42E1A; Tue, 24 Oct 2023 04:30:30 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2130.outbound.protection.outlook.com [40.107.223.130]) by mails.dpdk.org (Postfix) with ESMTP id CA80542DE5 for ; Tue, 24 Oct 2023 04:30:24 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eiD1pcwwsGJf8dwFDEwk9Nv5DTOqIELk4XHRwF63fcuPDuwf7EdCm0kp9KXpLEkik74M+Hz7hxUwqXqGrsbSq6kXzilNZNiKJI/Vg4bwOfz2cB7nxDKc/8jBhbFK2hVgoC27TevfvDlakMf+5JPJuE50oIidUyHSEO2w0QHuJgr2Vxi78U1z0t+mSf8iFq8vio9tp/hbvrcL/57bDU54+dC++EX7hje3dSs7yJMkKMR0mhaxfH7FIrRamjgDrYZw/qmYFU6dsP1IYlc71AT1RWMmXZdUegfR6r9QAYUb+ApK237heIyBheWWFs12PdLvXZ4sMRM0v2NDnBRYBIIQSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=s6/uD0XlZ5EznMcVag8a7+X3NNWpa++VHqzaMZ/6RM0=; b=DWfEBj82INwO/jog4HWyqWbjZ64ypwpY4gFMY37hunUPGBxspUjTllHQWx0JfPqN9YtVblcwbuIHGtGL8vJVtoFgWxNRNcEkEKKF/pVfOqYcwB0reTAVLQXqCiwgWH1zdL6zsKtgM5Kfl32Ttu4uAun3m160nEC44GTfMB61orBjpQhHIMHGvjSxqcAqj+JQ1J+DZ/ZX2j8yZIAgfYN7PEYtKnDp6Spovi0URNMoD1zLbuZuWEwMD8bfnUNLoTXybovqqCBHY5eUEgOYooa5eAY+gLEKWG56cF81jUy4ZuxtPzYsWpWQVyhBjNhzzyTKvEo7GMFru/zeu1OQ83n9tw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=s6/uD0XlZ5EznMcVag8a7+X3NNWpa++VHqzaMZ/6RM0=; b=fb7EvzqnNJ/8Ipr2c7wgabTeGs5zSSAqC2WB0rRRO1nmaZPk2F70eOOScG48uI0n7rFhTm5GKOt+61PzxqCrJdoAO8PYNN+B9sK3mUxqLydZZNaiWIr0kG3PHZeAxf0MQEnjHBL3CWsv2NtQySF2XMV3qoOc4SIi3TAOJJnKmYo= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) by PH8PR13MB6182.namprd13.prod.outlook.com (2603:10b6:510:259::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21; Tue, 24 Oct 2023 02:30:23 +0000 Received: from SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::142e:d47c:7320:8a9d]) by SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::142e:d47c:7320:8a9d%4]) with mapi id 15.20.6907.021; Tue, 24 Oct 2023 02:30:23 +0000 From: Chaoyong He To: dev@dpdk.org Cc: oss-drivers@corigine.com, Chaoyong He , Shujing Dong , Long Wu , Peng Zhang Subject: [PATCH v2 24/25] vdpa/nfp: add nfp vDPA device operations Date: Tue, 24 Oct 2023 10:28:48 +0800 Message-Id: <20231024022849.3092506-25-chaoyong.he@corigine.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20231024022849.3092506-1-chaoyong.he@corigine.com> References: <20231017054545.1692509-1-chaoyong.he@corigine.com> <20231024022849.3092506-1-chaoyong.he@corigine.com> X-ClientProxiedBy: BYAPR04CA0015.namprd04.prod.outlook.com (2603:10b6:a03:40::28) To SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR13MB5545:EE_|PH8PR13MB6182:EE_ X-MS-Office365-Filtering-Correlation-Id: da531f3e-d306-45f7-2920-08dbd4392c94 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mTKk3WhvweZcllgny3kNPWCIF7xtrNPcXkzXYdVkNLjbaaiRltRtRUAfm4r6zCPAEmK0oUuKD5GxYQJlTDMWcBYfqPBGPAktvrekfDUrNnR6w4SOMlRr3OBNQp8EBUR3mPkyeUbQSTVQt25uJgCcvhTmwy4HuWiHsAxZ1QXCzEsEgb2235jSh8p3xN8BiHlD2w9v419Z8LrN8Z1BanEXp0wo6wi1ti4XUDnjT492RuKaA5Ue69qWiu5M/iry9rtWE49eI9byaV0+rI+BDx+tG5N56SYwP6X1WW2SJpn/tyoAIKnyZy2jh8NwoqnizprFvgHV0mLhLRgyedEWLtvVV7Sv3us/BvTiYw8CDg8asxLt3AqkweXn/kT4qASxVlRqAJ+b/tv2r5LgQExNQmfF9LBxWzRhcErHfUPGb3w0LHLNHaKjdgnLJEF+CGblOis0NYuNVSJmrdubPnY0bfnz0UN9aXLk4NNov6KZ4weZ2e58BBmTiLt2+n8Gh0ODl1PMuSJQaJL4Oo9WESTBFzSIzx1utdYpL6exsw+XJVOIYyQgVauNZF7e/dbnW0g/DZSA0Kku2cFNkPLW1k6UAoG/UpL93iFTt69ouQUUzMOwi/mJPL/uERa6b2CZRxwQn6PBcX9Hft/1uPmvRVC8MTef1i5HVbWWh9XwUDKCJOLCFcU= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR13MB5545.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(366004)(396003)(376002)(39830400003)(346002)(136003)(230922051799003)(1800799009)(64100799003)(186009)(451199024)(86362001)(66946007)(316002)(2906002)(54906003)(66476007)(6916009)(66556008)(5660300002)(8936002)(44832011)(8676002)(4326008)(36756003)(41300700001)(2616005)(38100700002)(478600001)(1076003)(52116002)(6506007)(107886003)(6666004)(38350700005)(6512007)(26005)(83380400001)(6486002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pAAaSbcVHiXBBhJ/p2yQZWW0qDqanskPls8CvlbkPvp2U8dijTVHJYViqVObrqH9nXO6q9IV5bo1Z7lAAAj+cuMk7emH7f7hIlkfEshUsUGD+bKgZFBmapgMGoweKDZ0OGqjisPplcAUKVQo4BHIvsQLUHJz4XUedVmMDVPbV2p5rXx8jq9w4dcs46Qz2JSoUAyWFL05wIoyZD0N1xwsa/LL88tlwStpI6/H6+1GTKXsWTfS+sQaoeA7/72vq2jHEh6bkOpXD4EOM4FWjYEl2fdhbxfdpd1aL0rvk8xEfgqGH6JgbrVHm4ZsQS9wd8sGI7VBYkt1iVRIXiNN4WknZ3OPsXCaFVrlHalt/Gf/boIyE/ghxyCtAsHEgoKhQMxYSSqYblZzWd91XQB0ZtlyiPwkDpya/OCKH3vDxHCX5Bj9KoyOiEGuhYva3fbzB3J5LQmb3rLhKDCW8knCgkdkIEw59wJg6s4Z/aFB2NuiO+j5Grh/ja+SSWSVRacGw7SO9X3UwtUqlfQXWi4xEwv0y/GSgyW6LQl3ZBXJgB95aD4m32+KKtIQDsUqxbyI0fegTGjVftqvyLLdWLRf+uQP52VXy9ah0LnjFsBgZGpIU9mwRAia0Hymv28G+6Nhup3kGqaOGYFNn2Ci+ipgiX3eIJZ4DqFYgWUE/UPX/DST901r/LiOIEg0x72w4pk8ia818KItBMMqkBG4+38EWZo9eq0U2nXaJabvzBAsU+0YkI42YOdL9ZhNJFWNwL+JfTK/6or83miVqLis7Cm8ZaErmQ5eUXUGHzOJ06BPaqSULosyW6tfc5xJil8D+Gb0SKmcz8nkraHPPdmnktjSMVmsbThwM1Txnr2UgpslEWuiG5XfcR5Vcos933Bwk6bK/z6QtonSmsEMsYEHVul1oslYXKCVBsviNg74qWWuy2kO0kYulw95YV6sOvJGfOhD9+iAgBFuvJs15kw4b2uhbQCAWmx93sqvLBq81MiQN5nOZVdYM0eWYsvx2jvs3Ic18oxfV/Z3VnfkDlgYkUHCYYl68kCB5biA0HWYMvz/7oZ/MzhDmlP86jt5P7ad1q9ajFo1gGY6yco7UuzgwShUsQUB9mnZZT0YzWV6xxTXEbWykyXr/fBHNw/b5vHzdv7aiSr+vb1PLyhMrxE2iRMF+UnxfY9VYs+qwa/UxvDxSVNP08buPQzwPMG3tgovvoO0VDY82mSQbCqsATPjZp+TGvI56DFf3WuA76dM7AMTIZtC+XV6J943WXYU5fbeB9NcmYeGfuLQ5x1yMIjwyOGxgRIOvVMvwicPUGPCjvJjT7OLicBjq/mfoG1cdFp+OhdsRW9z3QakbhXHBLqIexXrg/NGO6yMg8B8rHPp5R+8vlPYKEs7ljrJJncnSulcLi4vuXqQlyPfmDzsxz8EIKlGetLMa+iAtHiykJ6cSTXqpKuCVW1nxt6nTk4VyKYT3zBoYMlG7hauQOw5GnL1GoZ2PtgeDYW5qi1I9FCM76dfd7J7H+wi8pWp+zMlrrZf/D+zKzDAs9U2XQ7adrjrAgQ9Nbpdbl60r24avkKqli2sapALMkHe6xMryU3MRpqMb8tQqhM1n2K19cfOFowi8uRVA+I2mw== X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: da531f3e-d306-45f7-2920-08dbd4392c94 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR13MB5545.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Oct 2023 02:30:23.1863 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: QAob0/gTNSEtn8/smhBjwbjSGDus839xCRMuVIERv2syJw0uFIaaTD4pa8IGUzgdHi4yGypp/LJDX+dfW7sJmtscFLaOJIKahlac0UGD7o0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR13MB6182 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Implement the corresponding nfp vDPA opetation functions. Signed-off-by: Shujing Dong Signed-off-by: Chaoyong He Reviewed-by: Long Wu Reviewed-by: Peng Zhang --- drivers/vdpa/nfp/nfp_vdpa.c | 213 ++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+) diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c index 01a59e2cfd..860b5cce63 100644 --- a/drivers/vdpa/nfp/nfp_vdpa.c +++ b/drivers/vdpa/nfp/nfp_vdpa.c @@ -57,6 +57,29 @@ static struct vdpa_dev_list_head vdpa_dev_list = static pthread_mutex_t vdpa_list_lock = PTHREAD_MUTEX_INITIALIZER; +static struct nfp_vdpa_dev_node * +nfp_vdpa_find_node_by_vdev(struct rte_vdpa_device *vdev) +{ + bool found = false; + struct nfp_vdpa_dev_node *node; + + pthread_mutex_lock(&vdpa_list_lock); + + TAILQ_FOREACH(node, &vdpa_dev_list, next) { + if (vdev == node->device->vdev) { + found = true; + break; + } + } + + pthread_mutex_unlock(&vdpa_list_lock); + + if (found) + return node; + + return NULL; +} + static struct nfp_vdpa_dev_node * nfp_vdpa_find_node_by_pdev(struct rte_pci_device *pdev) { @@ -575,7 +598,197 @@ update_datapath(struct nfp_vdpa_dev *device) return ret; } +static int +nfp_vdpa_dev_config(int vid) +{ + int ret; + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + device = node->device; + device->vid = vid; + __atomic_store_n(&device->dev_attached, 1, __ATOMIC_RELAXED); + update_datapath(device); + + ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, true); + if (ret != 0) + DRV_VDPA_LOG(INFO, "vDPA (%s): software relay is used.", + vdev->device->name); + + return 0; +} + +static int +nfp_vdpa_dev_close(int vid) +{ + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + device = node->device; + __atomic_store_n(&device->dev_attached, 0, __ATOMIC_RELAXED); + update_datapath(device); + + return 0; +} + +static int +nfp_vdpa_get_vfio_group_fd(int vid) +{ + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + return node->device->vfio_group_fd; +} + +static int +nfp_vdpa_get_vfio_device_fd(int vid) +{ + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + return node->device->vfio_dev_fd; +} + +static int +nfp_vdpa_get_notify_area(int vid, + int qid, + uint64_t *offset, + uint64_t *size) +{ + int ret; + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + struct vfio_region_info region = { + .argsz = sizeof(region) + }; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + device = node->device; + region.index = device->hw.notify_region; + + ret = ioctl(device->vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®ion); + if (ret != 0) { + DRV_VDPA_LOG(ERR, "Get not get device region info."); + return -EIO; + } + + *offset = nfp_vdpa_get_queue_notify_offset(&device->hw, qid) + region.offset; + *size = NFP_VDPA_NOTIFY_ADDR_INTERVAL; + + return 0; +} + +static int +nfp_vdpa_get_queue_num(struct rte_vdpa_device *vdev, + uint32_t *queue_num) +{ + struct nfp_vdpa_dev_node *node; + + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + *queue_num = node->device->max_queues; + + return 0; +} + +static int +nfp_vdpa_get_vdpa_features(struct rte_vdpa_device *vdev, + uint64_t *features) +{ + struct nfp_vdpa_dev_node *node; + + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + *features = node->device->hw.features; + + return 0; +} + +static int +nfp_vdpa_get_protocol_features(struct rte_vdpa_device *vdev __rte_unused, + uint64_t *features) +{ + *features = 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | + 1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK | + 1ULL << VHOST_USER_PROTOCOL_F_BACKEND_REQ | + 1ULL << VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD | + 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER; + + return 0; +} + +static int +nfp_vdpa_set_features(int32_t vid) +{ + DRV_VDPA_LOG(DEBUG, "Start vid=%d", vid); + return 0; +} + +static int +nfp_vdpa_set_vring_state(int vid, + int vring, + int state) +{ + DRV_VDPA_LOG(DEBUG, "Start vid=%d, vring=%d, state=%d", vid, vring, state); + return 0; +} + struct rte_vdpa_dev_ops nfp_vdpa_ops = { + .get_queue_num = nfp_vdpa_get_queue_num, + .get_features = nfp_vdpa_get_vdpa_features, + .get_protocol_features = nfp_vdpa_get_protocol_features, + .dev_conf = nfp_vdpa_dev_config, + .dev_close = nfp_vdpa_dev_close, + .set_vring_state = nfp_vdpa_set_vring_state, + .set_features = nfp_vdpa_set_features, + .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd, + .get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd, + .get_notify_area = nfp_vdpa_get_notify_area, }; static int