From patchwork Thu Oct 26 06:43:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chaoyong He X-Patchwork-Id: 133364 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8A80243204; Thu, 26 Oct 2023 08:48:02 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A3A9A42E62; Thu, 26 Oct 2023 08:44:40 +0200 (CEST) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2095.outbound.protection.outlook.com [40.107.93.95]) by mails.dpdk.org (Postfix) with ESMTP id B8A7542E45 for ; Thu, 26 Oct 2023 08:44:32 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Evshyel0061qXjhE2wvL14JxzTEK0qdp2xNlYxtdsKrpRyEVF1FoPX3lQ2UnkMQkjGhnUaEtH1UqkuKdFkHFDgWgfFih1N5VWhYDFD8RU/DM8iw7IV2VsBrEqLREGQ7VTDe0nnFh1PCPbvqIUoOUh7rYg3jy7FljY9WrgmwtSXAXQDlAW5Ihb3fJhyfmKdBEbhbSIxOwpS1AcP70oAFQ7DpHbCI+Fsiv9zUkRPMYr5tJTIYMPd6w1W+C1kDd4bcSkNSgfnaVhq8SDbOFXHkQEeMdacZAstIoyloWpV9EpWfwa+XDI+AKDJUwync+QtCNrtE5nmdq4v96sdyfuhgi0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wiURsf+vlS538RAt2eNSJjZmTmEO4kYbx9XWM297ei0=; b=DfCNBSzxBwgSU3hm/JG1GEBe+MaY1mneScmX+HSaPpRsPWSv7yV3PbG9q+zMxFhavwBRpNYcMV0cQCvqM5Fp3Gk2xtp1Rio6D7q5+qxiu2m+YJEBPJvVihHuRWbCnYzi16gPBTe6qNYPG0DeNAw03kPzuY+BqJuqwoU839OnFvIm6lv7ZoWEq9CG+kDKXOCPJorco1o75XyACkMc3phqL+COvIjESfCPqDPEtrwHh+G8sV481ba8I/5q7j1572WTCL1JQNTgUIE48+tg73WDE2O2EvV4U+q/MIgY9mTfeVmNEPv6E6I1Tib+6h6Kr3gIkdD2eIF+zivlNtwR4ZseNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wiURsf+vlS538RAt2eNSJjZmTmEO4kYbx9XWM297ei0=; b=cIyX2dIIA/eZeMYAhhHg0DGg+eRA8y+gB78fec7Jh38oH4G6M2PEmjcwfj3z4Ic4a4ONXsk5RQkjBCa28ZyHnysh1JLvWiy3zeedbDyskjyo1qjWYwIanQGkYEtdgxnfaCYeqsASEe9IXpagI2wFxr3MH2t77sD8k7mZrLM5X+Q= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) by DM4PR13MB5908.namprd13.prod.outlook.com (2603:10b6:8:50::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.19; Thu, 26 Oct 2023 06:44:31 +0000 Received: from SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::45b3:d83c:cff2:4a1b]) by SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::45b3:d83c:cff2:4a1b%6]) with mapi id 15.20.6933.022; Thu, 26 Oct 2023 06:44:31 +0000 From: Chaoyong He To: dev@dpdk.org Cc: oss-drivers@corigine.com, Chaoyong He , Shujing Dong , Long Wu , Peng Zhang Subject: [PATCH v3 24/25] vdpa/nfp: add nfp vDPA device operations Date: Thu, 26 Oct 2023 14:43:23 +0800 Message-Id: <20231026064324.177531-25-chaoyong.he@corigine.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20231026064324.177531-1-chaoyong.he@corigine.com> References: <20231024022849.3092506-1-chaoyong.he@corigine.com> <20231026064324.177531-1-chaoyong.he@corigine.com> X-ClientProxiedBy: BYAPR07CA0034.namprd07.prod.outlook.com (2603:10b6:a02:bc::47) To SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR13MB5545:EE_|DM4PR13MB5908:EE_ X-MS-Office365-Filtering-Correlation-Id: ded28d47-8a03-45a6-bf61-08dbd5ef01da X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: XWADqd2Qm9l9xCwDAlwYtoq6Lda5EgoHCsfnh18f4SVgCC2HpTfuJxqSvHTBvmJ7ZKV1dTYdziGU6ulMR0lb1H3Aq54tzuO8f8VlDvh3PH+alWVuBfCOFV8ue1sgZJE1aoT7OXNhWsgefGd3MsLjVm9YZ9Ks05HH+l3BAdxF2SYoAtCXXuXvVpfBMd0Ulhd06LoCysrszedQlTzBVe1EPAainNFjTNc3ik48JgtgOqhTsOTDV4MtE53qUcgiPR9dIQBl4YmznhYzH21VcRb+7tuR2UitDYb+F6Z6Of+T68DJ8tlv5794zc1kyki1pFmrNj4HJwzohVeUZO4RAEslhG7jQisQPnbvb334vpI9PqWlx8+qHB9guNBmX6Ko3BfN/d1xXbY7Q23bXLWSt6L8bzEiGNOkszpBQocNO0ujarSpYZ0lTdORa22+REKOeGwZjXgMy7c+/WbXncYAy6jy4YN/T/txZkPSvEqJRubzVgmxAWVeBbSgntH3XTFFUYPriOOu7ZQlHbKWC5n0LwvDEbp6tpllHruWUoIOkAxd454AbOwM3ySUz4MOFwVKwoSIN9q65RlRtMzyfPvzY01A/h9gSkjmEb0RlqImodFJDZxX48BtOhqx5V1YOaO2BUPunMNXy4Dc1kDIDgDaRlOnMgk7rEiRFKqziPwJvj30A+o= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR13MB5545.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(136003)(376002)(366004)(396003)(346002)(39840400004)(230922051799003)(186009)(1800799009)(451199024)(64100799003)(26005)(478600001)(1076003)(316002)(6666004)(6512007)(86362001)(6506007)(6916009)(66946007)(66556008)(38100700002)(66476007)(54906003)(36756003)(38350700005)(2616005)(6486002)(107886003)(44832011)(2906002)(83380400001)(4326008)(8676002)(8936002)(5660300002)(41300700001)(52116002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: F+RFApoOU2tgAepzUUwNEnQ48FR83FBcPAaDhP8Mg1wHuYTaPpPNJN/NjA985lgdxZf/omhJtZyOfFP4kD5zNXGPDPyf932d+VEhQw2k44h/rXAOMxEIyX7TmubHo5P/bHzFHRWFN+4TMZ7BiyimH25g86HzHlL6crRS0lJ18D+dqmiBgwu4yBWgU7wzJV+VY6lUBRcUE4z3F7IrJ7q6HiTFq6+ldiKjx+pam8Urb6jeGgfCp1DyZXY6fM3OORBpvyHyDxrLEBHqSNmP1MsgPgy0T91yBVT9udK77O6jOo0/x9HDxLAKk7qIDAdy/e5wNdG/J3L+P9XiL92qrlilbnypEuxpHe8UMoOGvo9kLK9SahVnV3/+QsbdQjB1dcPlxwrkjDyy5Ss9qiq/RurjK9wwb4ggfGdBgBbFHtf3zNCadhY9dOw+c2JJuE89mwTblZXkgJCABPaYokDH+EcOz/o2iPNSPgPnDkeoailx2aGyYFaKThDZkP3p2Okoz2fxL4tueUNNDo9V+JA1b5MxWQIJ1gplpjIEyQoZDtLckL1/BZvkvDkq/+wd0xYxTki3V0IO55Mow+gHEASVdalhYUcDgX//FNNnBqQeFXIwW/b1XNtwwjNQosmE0WnJjizfXN6dcza1s5vZmEiD5+ukp+SfFHur4i7nWD6vddXXQkQrqArrZpryX5L8pR1K+ciEqQBx3PqtoPJGehGVF0UXi8NMqE5Hi0MIkLGeruM8gUcQTqOyp7dwvOAFVKnvaJw/+wkw8uL5GI5EE2WfUupylOHWj/8CrGsNPyWDdX0wnsirns8i1lwJiEhIDHuJNjiMQOhbzDyz2aMXKIUHp4Rn+Os/Zmzrs3Sgzld4jYuzUPdGKwcQNhp3jobV0j21X69DcISzHOan8ufjkjIonZ2M+xx3vD8dAftZ+l5hQ6UlsIRwTYvjX1xv94mCeWGv30KPYZEYe4wPPjrVLKVxOfgUo0OMJqZUjNAfTSj0r7sbSGpC0+tSuqSrmtSRV6n8UnqZrDAG5cWqCmCcjb4lXmSdsfXwbrtn/FuC4RtRsEVayz74/X2T9EQN2p4xEqnQXkC50uUF2wzpCgD8ustnHrzEqJxoegWGG/JGNqwJFM/5v1ipYn3o7HnhDQkiw8rIuSO3v/vHwcgfzd3MUZsJ0BpzPK/rwgdtlTDCYo5CW1CYIHs/J3McxcK7h4BZSSv91ZzEWpRygofpXt9lNAJu8xFw7NkYBLnhxyv/M/9mMYHHQdITPNX1tnT7OQJlME8Kxrntv6NdsK4JxyvOLe+ICoLVZyWPvMFrJmDx2h/Fcf5N5iiAgSWvqeH3QahQrlSnMCEHYd+iXF3lTLIbiWUCD6XJSrsSQVBrvwf8CsFiN7w4aCiPtPcUUoHc2RuPiMboETGfHqmjiKOaPBRK1lAaA7tVXHM0n/WHMXJMDnE0pavrS0565qeb9cdHR0RxhZ0seA/fdhOfw/Mj5EoQBVoTIQzm9oRi2x9d/ZurS1SsJ9s9elqja5o4vRuL6rs05qi2JWaE81LRRaVFMQXtpccQsP/kkjDV0Ol4mTanxG42Dp3fUL8A5FGiEfKfBsaVvWba0R8jL5SyUe91biPD3cbKpKs3Eg== X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: ded28d47-8a03-45a6-bf61-08dbd5ef01da X-MS-Exchange-CrossTenant-AuthSource: SJ0PR13MB5545.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Oct 2023 06:44:31.0810 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6uqOLo5bIgSOlbpdLeL5+DO2acD/6kr4ZrYgASf0qk+vYvG/CfMD33ujokOoQDZKVsnXK3argahNXMcjicyt00Wm9hVxJOdGgKpBrzqPsLo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR13MB5908 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Implement the corresponding nfp vDPA operation functions. Signed-off-by: Shujing Dong Signed-off-by: Chaoyong He Reviewed-by: Long Wu Reviewed-by: Peng Zhang --- drivers/vdpa/nfp/nfp_vdpa.c | 213 ++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+) diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c index f4d63b8ba9..181fee58c2 100644 --- a/drivers/vdpa/nfp/nfp_vdpa.c +++ b/drivers/vdpa/nfp/nfp_vdpa.c @@ -57,6 +57,29 @@ static struct vdpa_dev_list_head vdpa_dev_list = static pthread_mutex_t vdpa_list_lock = PTHREAD_MUTEX_INITIALIZER; +static struct nfp_vdpa_dev_node * +nfp_vdpa_find_node_by_vdev(struct rte_vdpa_device *vdev) +{ + bool found = false; + struct nfp_vdpa_dev_node *node; + + pthread_mutex_lock(&vdpa_list_lock); + + TAILQ_FOREACH(node, &vdpa_dev_list, next) { + if (vdev == node->device->vdev) { + found = true; + break; + } + } + + pthread_mutex_unlock(&vdpa_list_lock); + + if (found) + return node; + + return NULL; +} + static struct nfp_vdpa_dev_node * nfp_vdpa_find_node_by_pdev(struct rte_pci_device *pdev) { @@ -578,7 +601,197 @@ update_datapath(struct nfp_vdpa_dev *device) return ret; } +static int +nfp_vdpa_dev_config(int vid) +{ + int ret; + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + device = node->device; + device->vid = vid; + rte_atomic_store_explicit(&device->dev_attached, 1, rte_memory_order_relaxed); + update_datapath(device); + + ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, true); + if (ret != 0) + DRV_VDPA_LOG(INFO, "vDPA (%s): software relay is used.", + vdev->device->name); + + return 0; +} + +static int +nfp_vdpa_dev_close(int vid) +{ + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + device = node->device; + rte_atomic_store_explicit(&device->dev_attached, 0, rte_memory_order_relaxed); + update_datapath(device); + + return 0; +} + +static int +nfp_vdpa_get_vfio_group_fd(int vid) +{ + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + return node->device->vfio_group_fd; +} + +static int +nfp_vdpa_get_vfio_device_fd(int vid) +{ + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + return node->device->vfio_dev_fd; +} + +static int +nfp_vdpa_get_notify_area(int vid, + int qid, + uint64_t *offset, + uint64_t *size) +{ + int ret; + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + struct vfio_region_info region = { + .argsz = sizeof(region) + }; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + device = node->device; + region.index = device->hw.notify_region; + + ret = ioctl(device->vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®ion); + if (ret != 0) { + DRV_VDPA_LOG(ERR, "Get not get device region info."); + return -EIO; + } + + *offset = nfp_vdpa_get_queue_notify_offset(&device->hw, qid) + region.offset; + *size = NFP_VDPA_NOTIFY_ADDR_INTERVAL; + + return 0; +} + +static int +nfp_vdpa_get_queue_num(struct rte_vdpa_device *vdev, + uint32_t *queue_num) +{ + struct nfp_vdpa_dev_node *node; + + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + *queue_num = node->device->max_queues; + + return 0; +} + +static int +nfp_vdpa_get_vdpa_features(struct rte_vdpa_device *vdev, + uint64_t *features) +{ + struct nfp_vdpa_dev_node *node; + + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + *features = node->device->hw.features; + + return 0; +} + +static int +nfp_vdpa_get_protocol_features(struct rte_vdpa_device *vdev __rte_unused, + uint64_t *features) +{ + *features = 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | + 1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK | + 1ULL << VHOST_USER_PROTOCOL_F_BACKEND_REQ | + 1ULL << VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD | + 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER; + + return 0; +} + +static int +nfp_vdpa_set_features(int32_t vid) +{ + DRV_VDPA_LOG(DEBUG, "Start vid=%d", vid); + return 0; +} + +static int +nfp_vdpa_set_vring_state(int vid, + int vring, + int state) +{ + DRV_VDPA_LOG(DEBUG, "Start vid=%d, vring=%d, state=%d", vid, vring, state); + return 0; +} + struct rte_vdpa_dev_ops nfp_vdpa_ops = { + .get_queue_num = nfp_vdpa_get_queue_num, + .get_features = nfp_vdpa_get_vdpa_features, + .get_protocol_features = nfp_vdpa_get_protocol_features, + .dev_conf = nfp_vdpa_dev_config, + .dev_close = nfp_vdpa_dev_close, + .set_vring_state = nfp_vdpa_set_vring_state, + .set_features = nfp_vdpa_set_features, + .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd, + .get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd, + .get_notify_area = nfp_vdpa_get_notify_area, }; static int