From patchwork Tue Oct 17 05:45:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chaoyong He X-Patchwork-Id: 132714 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ABE2D43186; Tue, 17 Oct 2023 07:49:13 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 539DD42D91; Tue, 17 Oct 2023 07:47:29 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2122.outbound.protection.outlook.com [40.107.94.122]) by mails.dpdk.org (Postfix) with ESMTP id 9F70342DA3 for ; Tue, 17 Oct 2023 07:47:27 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=R9IADCnUMl8Ep/BNo7ZV1bvgKSolWoxgmql++eZO+nkmNaQ2hXKxVjte5wtrmTQY96DqUI/M2aom6FMGsYZouixGZRg68QZSHVkVx7gxTToay1sPCECTz/fEYAp4lN4/6NKF2sbR7tLOQfzQdfk/MufLy+fVDtK77yYJDotaZ7AeZte2M2l2AV/Re4MSVqlsuUPZ1YEQ5M0MihUn31d/+pWtPqCKbTtU6C95Wcbq2yYCGx49rXMg0xK/v0Z3qDjxWjjZSL/oUN/rWAmPf4oLgIKAriyEPwjwdOuAE4jRK4zo+A9muwXRloEXNJUL8/LvDM05qVi2S7/pYx3DX2LwCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=s6/uD0XlZ5EznMcVag8a7+X3NNWpa++VHqzaMZ/6RM0=; b=jp/IbnmqzIvk52yltbdROut5ctfCT326G4EVKeQ/7wi5964vdCEGiO1phJCZSbm8qe3pb2Mpimm+CmfX8oVKBrz17R3jYhU5enXU8uRZlbND89OcLcnzi1mKX1pVz3RFkZzjYexIqdEwmKYF2nmuQd7hn2IIeD0VBISFx0RJ4glvg71sZbgt363ScF1yyOifwuUgwzd45wrcmILvoTxzH0UKg1BwB5EJ5gwvexEcCwiEkcOyh71n/V42LME/YF0r0GajBmviOZ4BXcdm1L7TVpggB/twoiWo956BN+M7LPNPBrPCNNW0W2vdWMnIRKfUQQDg8w+onQA5wPZERswc9w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=s6/uD0XlZ5EznMcVag8a7+X3NNWpa++VHqzaMZ/6RM0=; b=BVt4BLLACpOJFTogzS+U3lTZIjPxMyMpr1n7yTvTyObxnEK00pq4U7o1LgxZYqDW0WJM32o+0PkZyIg8ycIYKVgM5BVOgttxfvWX8J9kIWRORbMGVGiGuQiE3HUO9AjPLdqVLDuLeKvF7XIGg0b1m7d7QajO4RxzJ2hs9HphqLc= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from PH0PR13MB5568.namprd13.prod.outlook.com (2603:10b6:510:12b::16) by SJ0PR13MB5272.namprd13.prod.outlook.com (2603:10b6:a03:3e3::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.46; Tue, 17 Oct 2023 05:47:25 +0000 Received: from PH0PR13MB5568.namprd13.prod.outlook.com ([fe80::b070:92e1:931e:fee7]) by PH0PR13MB5568.namprd13.prod.outlook.com ([fe80::b070:92e1:931e:fee7%4]) with mapi id 15.20.6863.047; Tue, 17 Oct 2023 05:47:25 +0000 From: Chaoyong He To: dev@dpdk.org Cc: oss-drivers@corigine.com, Chaoyong He , Shujing Dong , Long Wu , Peng Zhang Subject: [PATCH 24/25] vdpa/nfp: add nfp vDPA device operations Date: Tue, 17 Oct 2023 13:45:44 +0800 Message-Id: <20231017054545.1692509-25-chaoyong.he@corigine.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20231017054545.1692509-1-chaoyong.he@corigine.com> References: <20231017054545.1692509-1-chaoyong.he@corigine.com> X-ClientProxiedBy: PH7PR17CA0069.namprd17.prod.outlook.com (2603:10b6:510:325::29) To PH0PR13MB5568.namprd13.prod.outlook.com (2603:10b6:510:12b::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH0PR13MB5568:EE_|SJ0PR13MB5272:EE_ X-MS-Office365-Filtering-Correlation-Id: 18186f39-2a8e-4339-8b90-08dbced48a0b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MjOJ6K0c5dmhNG5+/7N7kwGrVv9Khw01mD225eKgZ3DoAshPmJzSwxpi0bNLRtFE4IacPqN7MnmfHNdZcVYSdjZ58t92pPxcfHP1Kz6pqs1C5WNY59TTo2YTD7Ps5FleCZzZO52tY1QDxyUKQD7Hra/gNi4um/bMTYGhX+ltiBJ2Z1h/A7OnHLyHgH5/Qk/+qzh1nK4fwNQkyHa9W4cuZYlYbfFgx+3vCZIzRfuaCOY6S/+bMsU3ud+VjxnIlghH+sns++T2EAl8LxoPvwiKVOxGUxFl0tCmKoTQ39OjNSudNHhdEDaQPa1ZMVXtC1cQD/qEnFZUAatKgWHxzCrjg86805dG798LZguETbTD+jKUC4L7TFPfKpKEaAL12YSzO/FQ5bnOefRWEpabVq/6fXW4cvSQmBl7jdDUTufQtKAF3ec96gyIdRcmOob6mjtbzevhDcYWJT0w1gktOzt177WNiTtaU9s4JK7x8xhJmjekf/8+NOrUsKzNK8XsdUDa4krjEeZHI/LVrTm2edGdjuMUpd0MdhYNoc/P0+w5Yn4GGUPjoyQfccaF3pSVsw/WA1tpKp5QfcJrV6Rgjwj8hh7swzrjW0XL4H8tAgoceoGivxTVmbPDek3kERIMZ0+D7xiF4+z4z7Rlmn/0sWKa7xRueuHypplIqrGgn38PNsU= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR13MB5568.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(366004)(136003)(376002)(346002)(396003)(39830400003)(230922051799003)(451199024)(64100799003)(1800799009)(186009)(38350700005)(6916009)(66476007)(6666004)(66556008)(66946007)(6486002)(86362001)(54906003)(2616005)(83380400001)(478600001)(6512007)(38100700002)(316002)(1076003)(52116002)(26005)(107886003)(5660300002)(6506007)(2906002)(4326008)(8676002)(36756003)(44832011)(41300700001)(8936002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: wYnL/RiK74f87xD6JA/bwRMeZ6aOeRL0fPhXmo1doeLC961PKMa/wyUwUN0jA5RQSXZtNyp99LEw+nnpsb++B3OotVoJa57bIggocO+2R8vvsPnxSL0pukevIFREIKPi51uTR/l7PFEAH0+mnMO7jmmmyoDWI7fiBye9xRS4U7Ib7Cra5fBQ+Pkd4KdWYTVYOVIe3+PU4UuZKRHCPEE9fZ7TvL9QQC8EAHkUPdEh2r5XSfTHW0ibbomdeWAGpWQwTRhy87w2cf+8wzqLeI3OpMfeHvCGSJpRMRuLLhlXrmJsDYqNPbr+Wk8eLunNe9M5xUTwQg8K/c6LVLfBvMEA46W+thbSXBJuyXUcI9Xdg5ekoK+e9JAqpKiUb3TIJBLQfnj4SYkPTmPbuclERoaCB60V3GiGYIYgnPsjL1jB0xgWyiMTKNvDAz0tENvblV1MQw7uHzceeMKcYVSbRDpnOZgZYksH+QbFK7ryr5LQpwTIkBLqK0n/TQiZurfvwXloFac1rXkSvsFAKHOEuPPkWfJXs906y1Ub3eOUCF15C6UjEMyFSKAjfwvnDMTbF9a6U0VlqeA4E0xd2SCgjWsNsuZxEWc6krdpb8N7mhpWJnNHQra5rDUbT5wpXYVvhdQRJtRj2Vd1S9k/G4PtrIGKKLd/HjNTBbBHjtQr9cOG/2iwU9LmIcVjtgKgbjdxVSA2N2H47BgwIfbMdRzf3XAtimuntSogD2KwgoMYqx+lS6zBvorzdTb4EMKlwc4jwdEysK7cWdI1nIT9DqiIvNktQN8nRaPhGZYfJWUYlZE51E5AejF2S+964UBSF8+BU24KB6QMBeIqK9uViEnv5LPgoc4v4pAbvaeUhHW5LJOGO3Jnkzl+0l+Fh/VBoIVCOaZxxaDqZVHjUjmcqul6jgaAaDUKwdd3pZswb0EPgsNjqKUe1NcccH94FehcOwmjHWUvmzBovG+3E1VstJ4VURUU2oGpGPsdy5Ly3N+Z/M5Ozg6K3XjWWoL3hgOVkBHO/kuOIVEG/4vFIHvFqSddf0UpQtc34WWbPnr7lBQi6DjLb+AE8IZOT1KE1V+gmpOk2S6lt4zvZhObyHeM11jBpNc/0/URDwVBUOdscEPilfiBYs2L2QDxyD6+yz+FNh5vIxwDPjhdNhPRH+OPMBRGtHmC0B9e/CY+epcaS9r49x0nO0JF6a3CerE7WjOcm3MklBwg+KryTq50ezzEtnVARtJbMclk8E4HSHiJKU04ZWYIquTXqSJvJXaUt1Z6EjA8MhoEWAq0VH945AGzYycdoLMYSu25Pj/liqIdCTdZVnRrSdZaDWNvnUz4dxmlLuAfPxguoM6GxzfTVvo0jiDNi0xfbzEKOEWu+aJ6UoiQeWJJ8SCd+dMwu98s11VSGESk4H/3FeVMTImfOdjjyn0nz0/3gNvSy+tlO88SXtLKPWojkYY27gHuQH9W2+t/ggEanEk3lq+ZS1dK5eAhw93pd80QxRkT2ErWwJldCLqSDnnOt2ZnJSI6LlD9A5INrtRlkkuQH68kbgZsPNA0gbJkwt8kQUM5RtZpDKgtzoBSqGRWIf79hkPepaRKqOEoLwVwCtnrSnV+SvAU/4itMm+gaa6DRg== X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: 18186f39-2a8e-4339-8b90-08dbced48a0b X-MS-Exchange-CrossTenant-AuthSource: PH0PR13MB5568.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Oct 2023 05:47:25.6827 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: V2m1tgjiOq0+nVPBg6qj/LNHicBnCoEgOHvwGwU34SvJmx6l2gHEpDET6KJxJRgIPjQY9zvWSXPhTYq3bYtBNKHog8UvCiDy23izc1OC7kM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR13MB5272 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Implement the corresponding nfp vDPA opetation functions. Signed-off-by: Shujing Dong Signed-off-by: Chaoyong He Reviewed-by: Long Wu Reviewed-by: Peng Zhang --- drivers/vdpa/nfp/nfp_vdpa.c | 213 ++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+) diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c index 01a59e2cfd..860b5cce63 100644 --- a/drivers/vdpa/nfp/nfp_vdpa.c +++ b/drivers/vdpa/nfp/nfp_vdpa.c @@ -57,6 +57,29 @@ static struct vdpa_dev_list_head vdpa_dev_list = static pthread_mutex_t vdpa_list_lock = PTHREAD_MUTEX_INITIALIZER; +static struct nfp_vdpa_dev_node * +nfp_vdpa_find_node_by_vdev(struct rte_vdpa_device *vdev) +{ + bool found = false; + struct nfp_vdpa_dev_node *node; + + pthread_mutex_lock(&vdpa_list_lock); + + TAILQ_FOREACH(node, &vdpa_dev_list, next) { + if (vdev == node->device->vdev) { + found = true; + break; + } + } + + pthread_mutex_unlock(&vdpa_list_lock); + + if (found) + return node; + + return NULL; +} + static struct nfp_vdpa_dev_node * nfp_vdpa_find_node_by_pdev(struct rte_pci_device *pdev) { @@ -575,7 +598,197 @@ update_datapath(struct nfp_vdpa_dev *device) return ret; } +static int +nfp_vdpa_dev_config(int vid) +{ + int ret; + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + device = node->device; + device->vid = vid; + __atomic_store_n(&device->dev_attached, 1, __ATOMIC_RELAXED); + update_datapath(device); + + ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, true); + if (ret != 0) + DRV_VDPA_LOG(INFO, "vDPA (%s): software relay is used.", + vdev->device->name); + + return 0; +} + +static int +nfp_vdpa_dev_close(int vid) +{ + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + device = node->device; + __atomic_store_n(&device->dev_attached, 0, __ATOMIC_RELAXED); + update_datapath(device); + + return 0; +} + +static int +nfp_vdpa_get_vfio_group_fd(int vid) +{ + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + return node->device->vfio_group_fd; +} + +static int +nfp_vdpa_get_vfio_device_fd(int vid) +{ + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + return node->device->vfio_dev_fd; +} + +static int +nfp_vdpa_get_notify_area(int vid, + int qid, + uint64_t *offset, + uint64_t *size) +{ + int ret; + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + struct vfio_region_info region = { + .argsz = sizeof(region) + }; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + device = node->device; + region.index = device->hw.notify_region; + + ret = ioctl(device->vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®ion); + if (ret != 0) { + DRV_VDPA_LOG(ERR, "Get not get device region info."); + return -EIO; + } + + *offset = nfp_vdpa_get_queue_notify_offset(&device->hw, qid) + region.offset; + *size = NFP_VDPA_NOTIFY_ADDR_INTERVAL; + + return 0; +} + +static int +nfp_vdpa_get_queue_num(struct rte_vdpa_device *vdev, + uint32_t *queue_num) +{ + struct nfp_vdpa_dev_node *node; + + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + *queue_num = node->device->max_queues; + + return 0; +} + +static int +nfp_vdpa_get_vdpa_features(struct rte_vdpa_device *vdev, + uint64_t *features) +{ + struct nfp_vdpa_dev_node *node; + + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + *features = node->device->hw.features; + + return 0; +} + +static int +nfp_vdpa_get_protocol_features(struct rte_vdpa_device *vdev __rte_unused, + uint64_t *features) +{ + *features = 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | + 1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK | + 1ULL << VHOST_USER_PROTOCOL_F_BACKEND_REQ | + 1ULL << VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD | + 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER; + + return 0; +} + +static int +nfp_vdpa_set_features(int32_t vid) +{ + DRV_VDPA_LOG(DEBUG, "Start vid=%d", vid); + return 0; +} + +static int +nfp_vdpa_set_vring_state(int vid, + int vring, + int state) +{ + DRV_VDPA_LOG(DEBUG, "Start vid=%d, vring=%d, state=%d", vid, vring, state); + return 0; +} + struct rte_vdpa_dev_ops nfp_vdpa_ops = { + .get_queue_num = nfp_vdpa_get_queue_num, + .get_features = nfp_vdpa_get_vdpa_features, + .get_protocol_features = nfp_vdpa_get_protocol_features, + .dev_conf = nfp_vdpa_dev_config, + .dev_close = nfp_vdpa_dev_close, + .set_vring_state = nfp_vdpa_set_vring_state, + .set_features = nfp_vdpa_set_features, + .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd, + .get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd, + .get_notify_area = nfp_vdpa_get_notify_area, }; static int