From patchwork Wed Nov 3 13:57:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijay Srivastava X-Patchwork-Id: 103659 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7B382A0C53; Wed, 3 Nov 2021 15:00:49 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 80D5842721; Wed, 3 Nov 2021 15:00:43 +0100 (CET) Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam08on2068.outbound.protection.outlook.com [40.107.100.68]) by mails.dpdk.org (Postfix) with ESMTP id A650E41134 for ; Wed, 3 Nov 2021 15:00:41 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iwANE18L5Vs/r6qCHyJRbrKxX3a+xM8Ew7ku0vf4jKwfd9CstRXzgAyI+yT8Af+86CsXUsZOH+xWH0ZObWXDQ8XyvKqzf89DpwJOEDAFnF70YgXlzCnafsZ6W4aeMttMeB3xQMvtwwtp+gyEHjlFPuQUyaKANphyzSPTmo1irtPCQKeAh/1TRe/igLkidge/CIXc+qa8BoDZGtr/e4FQg7JvQDSA9A1wIH/AeTjt63qsSvXD+h8TsrGuN1wqSsBkL/Qn7/7a/KN+hfPhjzGbCkudIVjJvpB9Wkx6eISLt5vgC96F4z3NymfkG0I6FIIoC5C6h8oaslAzL82BXWjS1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mNlLgs5SJBl9ZDkoJLXmSAg5RUMdKiT2bbhDDELS5iE=; b=D2hMFIfck8g6JUYpo38AcOYC2uMgay3sfoiUdtuRdP4puLWU+9wLbBuF75hM7+4wx8qbhdZ0ecrcPkQRjunyW0DZukTjXT7yxM3U0gRz7ddsscQzuoZ90/tOXy5CVBhM1ZKhzNWXuRSALHJW6rWzzwCHoVVAm1xSaGUBK0oHM29D3VoTKlfg0zFQLZEaKrXe5JIFo4DkF/q9spo/pVjTYUcXszXYH31qeUtpXPAU9diUi1QgyxxFwNpdoNxOWhtydRyfo6jEq7mHubOWbVSaGx9x0hzy8YLhlhecgVxwHc0cdToErCTZNkIgUOTgxPIsYr/DsT5Uy17z7nY5oAbIHA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=dpdk.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mNlLgs5SJBl9ZDkoJLXmSAg5RUMdKiT2bbhDDELS5iE=; b=AUJt5BunSWzYt4uwKaP7htGsDdKZCrECvFcMSKMaU5LrmQtiq5rR8n1JyUSD+TO1GlvaygeaPBJjpp9edoE0GoezK/mznr2GzJFooCZD8JsYKrgr8m/PSon9LWJ8O5mSgNYlKbo/2LstxLJfNizXP68/O9HyChI9R6VdVo/ivMU= Received: from BN9PR03CA0353.namprd03.prod.outlook.com (2603:10b6:408:f6::28) by DM6PR02MB4186.namprd02.prod.outlook.com (2603:10b6:5:9a::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.19; Wed, 3 Nov 2021 14:00:39 +0000 Received: from BN1NAM02FT030.eop-nam02.prod.protection.outlook.com (2603:10b6:408:f6:cafe::e0) by BN9PR03CA0353.outlook.office365.com (2603:10b6:408:f6::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.10 via Frontend Transport; Wed, 3 Nov 2021 14:00:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch01.xlnx.xilinx.com; Received: from xsj-pvapexch01.xlnx.xilinx.com (149.199.62.198) by BN1NAM02FT030.mail.protection.outlook.com (10.13.2.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4649.14 via Frontend Transport; Wed, 3 Nov 2021 14:00:39 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch01.xlnx.xilinx.com (172.19.86.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Wed, 3 Nov 2021 07:00:35 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Wed, 3 Nov 2021 07:00:35 -0700 Envelope-to: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, andrew.rybchenko@oktetlabs.ru Received: from [10.170.66.108] (port=38886 helo=xndengvm004108.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1miGoc-0003SS-Sc; Wed, 03 Nov 2021 07:00:35 -0700 From: Vijay Srivastava To: CC: , , , Vijay Kumar Srivastava Date: Wed, 3 Nov 2021 19:27:45 +0530 Message-ID: <20211103135754.17411-2-vsrivast@xilinx.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20211103135754.17411-1-vsrivast@xilinx.com> References: <20210706164418.32615-1-vsrivast@xilinx.com> <20211103135754.17411-1-vsrivast@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0a0ff08d-ef27-45a4-a3a9-08d99ed2513d X-MS-TrafficTypeDiagnostic: DM6PR02MB4186: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:236; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SU0qKI2BjgpFdNr5GrcutPtd7gfWsjJ5gdZHQ/x65GNmqBweiFwh0swF3v0BY1PXU+fxEOTbQFcu0taf6NnzXz1UAMjpoY7nn2DoPHrnx3fk9DrTsLndpHNQST84D9gsu3u4lgZewt0xwW417A8Jpm/0VNqWE2eU4faSDpNhIdfm6MFOsIfV6ri/LzltxEXOQNrgK3MXi/HhQ5F8+0miaQjDJwHVKRQlVCQEcl6wdWNfsQ3Gd6rMyhyBnvsPkZSw1pl6qBZJNBxQzo0eJTl9mu3fOj1jTU3UniIairgk0cdvX2UGx/dIHAhSVp7ShiHFtzn7mrPyDKawiMD/J7lytheNtlIT0lFqLgS1E+nXdiI11it4jrApJEUxtg31wUHjnnR9/ZBOJ3bPFQgfNI6pWU1AIDJd71/fC+NdZSOCSFnpnsUQi86OeutR03qfWSjAZZVa2HFFZAIQstfJ/VXvvphSU+UNyTPpthFJjGCTa1dbKyJGRtXBslbfVnRDIFv9wxojdmA4MbQ98BlksMfbp/DtR02QyMMo/Mww/KaDu21yU004KdxlzWdcFMSTBKZKP13LHwlitunYTljbB/Iyl7PwM4EzCSV93WtvCbNhnZ33vxFWM8A5C7zYLaVlDNW3ZQpCtlIryS/rENp3Py7aEuhTDqNiT80FOvoFCRtsYJQmW+h1SD82CcsPtftQkrSMz+VT9zeslLZ2FZ5jJET5dJT3sceVmRX3y+AwnhF00qpd4PaPEUBMgFQ9UmpSD6paTW7WI3DUkGjJPeojpdyNWt2P4l2r6TY6k92OyQTVbjkYatKzZECB7PF04J1rMRsDehkbCtXGP3xZMAZVcQZumhfF6lxHMPC906M6+9RZ7oQ= X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch01.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(46966006)(36840700001)(2616005)(83380400001)(7636003)(82310400003)(70206006)(8676002)(1076003)(186003)(508600001)(36756003)(4326008)(356005)(30864003)(70586007)(336012)(2906002)(966005)(44832011)(47076005)(316002)(5660300002)(7696005)(426003)(54906003)(36860700001)(6666004)(26005)(8936002)(36906005)(9786002)(6916009)(107886003)(102446001); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 14:00:39.1541 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0a0ff08d-ef27-45a4-a3a9-08d99ed2513d X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch01.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: BN1NAM02FT030.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR02MB4186 Subject: [dpdk-dev] [PATCH v4 01/10] vdpa/sfc: introduce Xilinx vDPA driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vijay Kumar Srivastava Add new vDPA PMD to support vDPA operations of Xilinx devices. This patch implements probe and remove functions. Signed-off-by: Vijay Kumar Srivastava Acked-by: Andrew Rybchenko Reviewed-by: Maxime Coquelin --- v2: * Updated logging macros to remove redundant code. v3: * Replaced depreciated whitelist with allowlist. * Text corrections in the sfc.rst and commit msg. * Added sfc in the toctree of doc/guides/vdpadevs/index.rst. * Removed extra compiler flags. v4: * Removed redundant NULL checks. * Removed unused pci_addr from adapter structure. * Updated format specifiers in log messages. MAINTAINERS | 6 + doc/guides/rel_notes/release_21_11.rst | 5 + doc/guides/vdpadevs/features/sfc.ini | 9 ++ doc/guides/vdpadevs/index.rst | 1 + doc/guides/vdpadevs/sfc.rst | 97 ++++++++++++ drivers/vdpa/meson.build | 1 + drivers/vdpa/sfc/meson.build | 22 +++ drivers/vdpa/sfc/sfc_vdpa.c | 282 +++++++++++++++++++++++++++++++++ drivers/vdpa/sfc/sfc_vdpa.h | 39 +++++ drivers/vdpa/sfc/sfc_vdpa_log.h | 56 +++++++ drivers/vdpa/sfc/version.map | 3 + 11 files changed, 521 insertions(+) create mode 100644 doc/guides/vdpadevs/features/sfc.ini create mode 100644 doc/guides/vdpadevs/sfc.rst create mode 100644 drivers/vdpa/sfc/meson.build create mode 100644 drivers/vdpa/sfc/sfc_vdpa.c create mode 100644 drivers/vdpa/sfc/sfc_vdpa.h create mode 100644 drivers/vdpa/sfc/sfc_vdpa_log.h create mode 100644 drivers/vdpa/sfc/version.map diff --git a/MAINTAINERS b/MAINTAINERS index be2c9b6..5d12c49 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1236,6 +1236,12 @@ F: drivers/vdpa/mlx5/ F: doc/guides/vdpadevs/mlx5.rst F: doc/guides/vdpadevs/features/mlx5.ini +Xilinx sfc vDPA +M: Vijay Kumar Srivastava +F: drivers/vdpa/sfc/ +F: doc/guides/vdpadevs/sfc.rst +F: doc/guides/vdpadevs/features/sfc.ini + Eventdev Drivers ---------------- diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index c68f122..0859b9b 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -324,6 +324,11 @@ New Features with the capability for the application to configure the hash key, the RETA and the hash types. Virtio hash reporting is yet to be added. +* **Added new vDPA PMD based on Xilinx devices.** + + Added a new Xilinx vDPA (``sfc_vdpa``) PMD. + See the :doc:`../vdpadevs/sfc` guide for more details on this driver. + Removed Items ------------- diff --git a/doc/guides/vdpadevs/features/sfc.ini b/doc/guides/vdpadevs/features/sfc.ini new file mode 100644 index 0000000..71b6158 --- /dev/null +++ b/doc/guides/vdpadevs/features/sfc.ini @@ -0,0 +1,9 @@ +; +; Supported features of the 'sfc' vDPA driver. +; +; Refer to default.ini for the full list of available driver features. +; +[Features] +Linux = Y +x86-64 = Y +Usage doc = Y diff --git a/doc/guides/vdpadevs/index.rst b/doc/guides/vdpadevs/index.rst index 1a13efe..cc2f858 100644 --- a/doc/guides/vdpadevs/index.rst +++ b/doc/guides/vdpadevs/index.rst @@ -14,3 +14,4 @@ which can be used from an application through vhost API. features_overview ifc mlx5 + sfc diff --git a/doc/guides/vdpadevs/sfc.rst b/doc/guides/vdpadevs/sfc.rst new file mode 100644 index 0000000..44e694f --- /dev/null +++ b/doc/guides/vdpadevs/sfc.rst @@ -0,0 +1,97 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2021 Xilinx Corporation. + +Xilinx vDPA driver +================== + +The Xilinx vDPA (vhost data path acceleration) driver (**librte_pmd_sfc_vdpa**) +provides support for the Xilinx SN1022 SmartNICs family of 10/25/40/50/100 Gbps +adapters that have support for latest Linux and FreeBSD operating systems. + +More information can be found at Xilinx website https://www.xilinx.com. + + +Xilinx vDPA implementation +-------------------------- + +ef100 device can be configured in the net device or vDPA mode. +Adding "class=vdpa" parameter helps to specify that this +device is to be used in vDPA mode. If this parameter is not specified, device +will be probed by net/sfc driver and will used as a net device. + +This PMD uses libefx (common/sfc_efx) code to access the device firmware. + + +Supported NICs +-------------- + +- Xilinx SN1022 SmartNICs + + +Features +-------- + +Features of the Xilinx vDPA driver are: + +- Compatibility with virtio 0.95 and 1.0 + + +Non-supported Features +---------------------- + +- Control Queue +- Multi queue +- Live Migration + + +Prerequisites +------------- + +Requires firmware version: v1.0.7.0 or higher + +Visit `Xilinx Support Downloads `_ +to get Xilinx Utilities with the latest firmware. +Follow instructions from Alveo SN1000 SmartNICs User Guide to +update firmware and configure the adapter. + + +Per-Device Parameters +~~~~~~~~~~~~~~~~~~~~~ + +The following per-device parameters can be passed via EAL PCI device +allowlist option like "-a 02:00.0,arg1=value1,...". + +Case-insensitive 1/y/yes/on or 0/n/no/off may be used to specify +boolean parameters value. + +- ``class`` [net|vdpa] (default **net**) + + Choose the mode of operation of ef100 device. + **net** device will work as network device and will be probed by net/sfc driver. + **vdpa** device will work as vdpa device and will be probed by vdpa/sfc driver. + If this parameter is not specified then ef100 device will operate as network device. + + +Dynamic Logging Parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +One may leverage EAL option "--log-level" to change default levels +for the log types supported by the driver. The option is used with +an argument typically consisting of two parts separated by a colon. + +Level value is the last part which takes a symbolic name (or integer). +Log type is the former part which may shell match syntax. +Depending on the choice of the expression, the given log level may +be used either for some specific log type or for a subset of types. + +SFC vDPA PMD provides the following log types available for control: + +- ``pmd.vdpa.sfc.driver`` (default level is **notice**) + + Affects driver-wide messages unrelated to any particular devices. + +- ``pmd.vdpa.sfc.main`` (default level is **notice**) + + Matches a subset of per-port log types registered during runtime. + A full name for a particular type may be obtained by appending a + dot and a PCI device identifier (``XXXX:XX:XX.X``) to the prefix. diff --git a/drivers/vdpa/meson.build b/drivers/vdpa/meson.build index f765fe3..77412c7 100644 --- a/drivers/vdpa/meson.build +++ b/drivers/vdpa/meson.build @@ -8,6 +8,7 @@ endif drivers = [ 'ifc', 'mlx5', + 'sfc', ] std_deps = ['bus_pci', 'kvargs'] std_deps += ['vhost'] diff --git a/drivers/vdpa/sfc/meson.build b/drivers/vdpa/sfc/meson.build new file mode 100644 index 0000000..4255d65 --- /dev/null +++ b/drivers/vdpa/sfc/meson.build @@ -0,0 +1,22 @@ +# SPDX-License-Identifier: BSD-3-Clause +# +# Copyright(c) 2020-2021 Xilinx, Inc. + +if (arch_subdir != 'x86' or not dpdk_conf.get('RTE_ARCH_64')) and (arch_subdir != 'arm' or not host_machine.cpu_family().startswith('aarch64')) + build = false + reason = 'only supported on x86_64 and aarch64' +endif + +fmt_name = 'sfc_vdpa' +extra_flags = [] + +foreach flag: extra_flags + if cc.has_argument(flag) + cflags += flag + endif +endforeach + +deps += ['common_sfc_efx', 'bus_pci'] +sources = files( + 'sfc_vdpa.c', +) diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c new file mode 100644 index 0000000..f087944 --- /dev/null +++ b/drivers/vdpa/sfc/sfc_vdpa.c @@ -0,0 +1,282 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * + * Copyright(c) 2020-2021 Xilinx, Inc. + */ + +#include +#include +#include + +#include +#include +#include +#include +#include + +#include "efx.h" +#include "sfc_efx.h" +#include "sfc_vdpa.h" + +TAILQ_HEAD(sfc_vdpa_adapter_list_head, sfc_vdpa_adapter); +static struct sfc_vdpa_adapter_list_head sfc_vdpa_adapter_list = + TAILQ_HEAD_INITIALIZER(sfc_vdpa_adapter_list); + +static pthread_mutex_t sfc_vdpa_adapter_list_lock = PTHREAD_MUTEX_INITIALIZER; + +struct sfc_vdpa_adapter * +sfc_vdpa_get_adapter_by_dev(struct rte_pci_device *pdev) +{ + bool found = false; + struct sfc_vdpa_adapter *sva; + + pthread_mutex_lock(&sfc_vdpa_adapter_list_lock); + + TAILQ_FOREACH(sva, &sfc_vdpa_adapter_list, next) { + if (pdev == sva->pdev) { + found = true; + break; + } + } + + pthread_mutex_unlock(&sfc_vdpa_adapter_list_lock); + + return found ? sva : NULL; +} + +static int +sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva) +{ + struct rte_pci_device *dev = sva->pdev; + char dev_name[RTE_DEV_NAME_MAX_LEN] = {0}; + int rc; + + rte_pci_device_name(&dev->addr, dev_name, RTE_DEV_NAME_MAX_LEN); + + sva->vfio_container_fd = rte_vfio_container_create(); + if (sva->vfio_container_fd < 0) { + sfc_vdpa_err(sva, "failed to create VFIO container"); + goto fail_container_create; + } + + rc = rte_vfio_get_group_num(rte_pci_get_sysfs_path(), dev_name, + &sva->iommu_group_num); + if (rc <= 0) { + sfc_vdpa_err(sva, "failed to get IOMMU group for %s : %s", + dev_name, rte_strerror(-rc)); + goto fail_get_group_num; + } + + sva->vfio_group_fd = + rte_vfio_container_group_bind(sva->vfio_container_fd, + sva->iommu_group_num); + if (sva->vfio_group_fd < 0) { + sfc_vdpa_err(sva, + "failed to bind IOMMU group %d to container %d", + sva->iommu_group_num, sva->vfio_container_fd); + goto fail_group_bind; + } + + if (rte_pci_map_device(dev) != 0) { + sfc_vdpa_err(sva, "failed to map PCI device %s : %s", + dev_name, rte_strerror(rte_errno)); + goto fail_pci_map_device; + } + + sva->vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle); + + return 0; + +fail_pci_map_device: + if (rte_vfio_container_group_unbind(sva->vfio_container_fd, + sva->iommu_group_num) != 0) { + sfc_vdpa_err(sva, + "failed to unbind IOMMU group %d from container %d", + sva->iommu_group_num, sva->vfio_container_fd); + } + +fail_group_bind: +fail_get_group_num: + if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) { + sfc_vdpa_err(sva, "failed to destroy container %d", + sva->vfio_container_fd); + } + +fail_container_create: + return -1; +} + +static void +sfc_vdpa_vfio_teardown(struct sfc_vdpa_adapter *sva) +{ + rte_pci_unmap_device(sva->pdev); + + if (rte_vfio_container_group_unbind(sva->vfio_container_fd, + sva->iommu_group_num) != 0) { + sfc_vdpa_err(sva, + "failed to unbind IOMMU group %d from container %d", + sva->iommu_group_num, sva->vfio_container_fd); + } + + if (rte_vfio_container_destroy(sva->vfio_container_fd) != 0) { + sfc_vdpa_err(sva, + "failed to destroy container %d", + sva->vfio_container_fd); + } +} + +static int +sfc_vdpa_set_log_prefix(struct sfc_vdpa_adapter *sva) +{ + struct rte_pci_device *pci_dev = sva->pdev; + int ret; + + ret = snprintf(sva->log_prefix, sizeof(sva->log_prefix), + "PMD: sfc_vdpa " PCI_PRI_FMT " : ", + pci_dev->addr.domain, pci_dev->addr.bus, + pci_dev->addr.devid, pci_dev->addr.function); + + if (ret < 0 || ret >= (int)sizeof(sva->log_prefix)) { + SFC_VDPA_GENERIC_LOG(ERR, + "reserved log prefix is too short for " PCI_PRI_FMT, + pci_dev->addr.domain, pci_dev->addr.bus, + pci_dev->addr.devid, pci_dev->addr.function); + return -EINVAL; + } + + return 0; +} + +uint32_t +sfc_vdpa_register_logtype(const struct rte_pci_addr *pci_addr, + const char *lt_prefix_str, uint32_t ll_default) +{ + size_t lt_prefix_str_size = strlen(lt_prefix_str); + size_t lt_str_size_max; + char *lt_str = NULL; + int ret; + + if (SIZE_MAX - PCI_PRI_STR_SIZE - 1 > lt_prefix_str_size) { + ++lt_prefix_str_size; /* Reserve space for prefix separator */ + lt_str_size_max = lt_prefix_str_size + PCI_PRI_STR_SIZE + 1; + } else { + return RTE_LOGTYPE_PMD; + } + + lt_str = rte_zmalloc("logtype_str", lt_str_size_max, 0); + if (lt_str == NULL) + return RTE_LOGTYPE_PMD; + + strncpy(lt_str, lt_prefix_str, lt_prefix_str_size); + lt_str[lt_prefix_str_size - 1] = '.'; + rte_pci_device_name(pci_addr, lt_str + lt_prefix_str_size, + lt_str_size_max - lt_prefix_str_size); + lt_str[lt_str_size_max - 1] = '\0'; + + ret = rte_log_register_type_and_pick_level(lt_str, ll_default); + rte_free(lt_str); + + return ret < 0 ? RTE_LOGTYPE_PMD : ret; +} + +static struct rte_pci_id pci_id_sfc_vdpa_efx_map[] = { + { RTE_PCI_DEVICE(EFX_PCI_VENID_XILINX, EFX_PCI_DEVID_RIVERHEAD_VF) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static int +sfc_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct sfc_vdpa_adapter *sva = NULL; + uint32_t logtype_main; + int ret = 0; + + if (sfc_efx_dev_class_get(pci_dev->device.devargs) != + SFC_EFX_DEV_CLASS_VDPA) { + SFC_VDPA_GENERIC_LOG(INFO, + "Incompatible device class: skip probing, should be probed by other sfc driver."); + return 1; + } + + /* + * It will not be probed in the secondary process. As device class + * is vdpa so return 0 to avoid probe by other sfc driver + */ + if (rte_eal_process_type() != RTE_PROC_PRIMARY) + return 0; + + logtype_main = sfc_vdpa_register_logtype(&pci_dev->addr, + SFC_VDPA_LOGTYPE_MAIN_STR, + RTE_LOG_NOTICE); + + sva = rte_zmalloc("sfc_vdpa", sizeof(struct sfc_vdpa_adapter), 0); + if (sva == NULL) + goto fail_zmalloc; + + sva->pdev = pci_dev; + sva->logtype_main = logtype_main; + + ret = sfc_vdpa_set_log_prefix(sva); + if (ret != 0) + goto fail_set_log_prefix; + + sfc_vdpa_log_init(sva, "entry"); + + sfc_vdpa_log_init(sva, "vfio init"); + if (sfc_vdpa_vfio_setup(sva) < 0) { + sfc_vdpa_err(sva, "failed to setup device %s", pci_dev->name); + goto fail_vfio_setup; + } + + pthread_mutex_lock(&sfc_vdpa_adapter_list_lock); + TAILQ_INSERT_TAIL(&sfc_vdpa_adapter_list, sva, next); + pthread_mutex_unlock(&sfc_vdpa_adapter_list_lock); + + sfc_vdpa_log_init(sva, "done"); + + return 0; + +fail_vfio_setup: +fail_set_log_prefix: + rte_free(sva); + +fail_zmalloc: + return -1; +} + +static int +sfc_vdpa_pci_remove(struct rte_pci_device *pci_dev) +{ + struct sfc_vdpa_adapter *sva = NULL; + + if (rte_eal_process_type() != RTE_PROC_PRIMARY) + return -1; + + sva = sfc_vdpa_get_adapter_by_dev(pci_dev); + if (sva == NULL) { + sfc_vdpa_info(sva, "invalid device: %s", pci_dev->name); + return -1; + } + + pthread_mutex_lock(&sfc_vdpa_adapter_list_lock); + TAILQ_REMOVE(&sfc_vdpa_adapter_list, sva, next); + pthread_mutex_unlock(&sfc_vdpa_adapter_list_lock); + + sfc_vdpa_vfio_teardown(sva); + + rte_free(sva); + + return 0; +} + +static struct rte_pci_driver rte_sfc_vdpa = { + .id_table = pci_id_sfc_vdpa_efx_map, + .drv_flags = 0, + .probe = sfc_vdpa_pci_probe, + .remove = sfc_vdpa_pci_remove, +}; + +RTE_PMD_REGISTER_PCI(net_sfc_vdpa, rte_sfc_vdpa); +RTE_PMD_REGISTER_PCI_TABLE(net_sfc_vdpa, pci_id_sfc_vdpa_efx_map); +RTE_PMD_REGISTER_KMOD_DEP(net_sfc_vdpa, "* vfio-pci"); +RTE_LOG_REGISTER_SUFFIX(sfc_vdpa_logtype_driver, driver, NOTICE); diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h new file mode 100644 index 0000000..2ae9ec3 --- /dev/null +++ b/drivers/vdpa/sfc/sfc_vdpa.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * + * Copyright(c) 2020-2021 Xilinx, Inc. + */ + +#ifndef _SFC_VDPA_H +#define _SFC_VDPA_H + +#include +#include + +#include + +#include "sfc_vdpa_log.h" + +/* Adapter private data */ +struct sfc_vdpa_adapter { + TAILQ_ENTRY(sfc_vdpa_adapter) next; + struct rte_pci_device *pdev; + + char log_prefix[SFC_VDPA_LOG_PREFIX_MAX]; + uint32_t logtype_main; + + int vfio_group_fd; + int vfio_dev_fd; + int vfio_container_fd; + int iommu_group_num; +}; + +uint32_t +sfc_vdpa_register_logtype(const struct rte_pci_addr *pci_addr, + const char *lt_prefix_str, + uint32_t ll_default); + +struct sfc_vdpa_adapter * +sfc_vdpa_get_adapter_by_dev(struct rte_pci_device *pdev); + +#endif /* _SFC_VDPA_H */ + diff --git a/drivers/vdpa/sfc/sfc_vdpa_log.h b/drivers/vdpa/sfc/sfc_vdpa_log.h new file mode 100644 index 0000000..858e5ee --- /dev/null +++ b/drivers/vdpa/sfc/sfc_vdpa_log.h @@ -0,0 +1,56 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * + * Copyright(c) 2020-2021 Xilinx, Inc. + */ + +#ifndef _SFC_VDPA_LOG_H_ +#define _SFC_VDPA_LOG_H_ + +/** Generic driver log type */ +extern int sfc_vdpa_logtype_driver; + +/** Common log type name prefix */ +#define SFC_VDPA_LOGTYPE_PREFIX "pmd.vdpa.sfc." + +/** Log PMD generic message, add a prefix and a line break */ +#define SFC_VDPA_GENERIC_LOG(level, ...) \ + rte_log(RTE_LOG_ ## level, sfc_vdpa_logtype_driver, \ + RTE_FMT("PMD: " RTE_FMT_HEAD(__VA_ARGS__ ,) "\n", \ + RTE_FMT_TAIL(__VA_ARGS__ ,))) + +/** Name prefix for the per-device log type used to report basic information */ +#define SFC_VDPA_LOGTYPE_MAIN_STR SFC_VDPA_LOGTYPE_PREFIX "main" + +#define SFC_VDPA_LOG_PREFIX_MAX 32 + +/* Log PMD message, automatically add prefix and \n */ +#define SFC_VDPA_LOG(sva, level, ...) \ + do { \ + const struct sfc_vdpa_adapter *_sva = (sva); \ + \ + rte_log(RTE_LOG_ ## level, _sva->logtype_main, \ + RTE_FMT("%s" RTE_FMT_HEAD(__VA_ARGS__ ,) "\n", \ + _sva->log_prefix, \ + RTE_FMT_TAIL(__VA_ARGS__ ,))); \ + } while (0) + +#define sfc_vdpa_err(sva, ...) \ + SFC_VDPA_LOG(sva, ERR, __VA_ARGS__) + +#define sfc_vdpa_warn(sva, ...) \ + SFC_VDPA_LOG(sva, WARNING, __VA_ARGS__) + +#define sfc_vdpa_notice(sva, ...) \ + SFC_VDPA_LOG(sva, NOTICE, __VA_ARGS__) + +#define sfc_vdpa_info(sva, ...) \ + SFC_VDPA_LOG(sva, INFO, __VA_ARGS__) + +#define sfc_vdpa_log_init(sva, ...) \ + SFC_VDPA_LOG(sva, INFO, \ + RTE_FMT("%s(): " \ + RTE_FMT_HEAD(__VA_ARGS__ ,), \ + __func__, \ + RTE_FMT_TAIL(__VA_ARGS__ ,))) + +#endif /* _SFC_VDPA_LOG_H_ */ diff --git a/drivers/vdpa/sfc/version.map b/drivers/vdpa/sfc/version.map new file mode 100644 index 0000000..4a76d1d --- /dev/null +++ b/drivers/vdpa/sfc/version.map @@ -0,0 +1,3 @@ +DPDK_21 { + local: *; +}; From patchwork Wed Nov 3 13:57:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijay Srivastava X-Patchwork-Id: 103660 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B2310A0C53; Wed, 3 Nov 2021 15:01:00 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EF3EF42731; Wed, 3 Nov 2021 15:00:44 +0100 (CET) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2051.outbound.protection.outlook.com [40.107.92.51]) by mails.dpdk.org (Postfix) with ESMTP id 0F36E426DD for ; Wed, 3 Nov 2021 15:00:41 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NzjCjGrVQn9bZ8TouCdwe+25GoAy6gRHtRfjYmZIfzWiWBXbz2CRkMzbElUM7/pXiUxBaruK8JxQxfRimIrQRgnfpmII/Fg+TsjfzSdlktQ74m+42tdz70krfSKMd00gfT0+a4sApruj7OtCbSIZ6BdOEwqHFJc/3Y4JexrFkQ74jL5Cl+rLvZH0bUdutCoUN1o5g5Mx7E8WX900a7CIPpmQSuI8tjMY5s0EtpnuQz6Zxw8IKuLECwNNQ+YJKstFOdMJjnuAXqeqgupckuAZY1s6RQc4gCJ0QEGIlPXAHU41w6g5l1FitQ3ojZ+UvDzXHaevbbcDZvfIyOstRVmf8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Gv5fhO/rijTo3nX3LLY+9ortff1h6X4bU2hwFskGbdk=; b=CG42wKDTYMx8BHVH4pAmo6Z40SDkNtBkMDguG2/+i85JQeRMJ0zOCoYAeYELFuKC5DkdV2sIWS13ok56aSfPLzq//08Zi41+HbwzPkERTUxPy/ZZMevMjHFJRCj+BWahBXkh5HMdBPC2fd/hsjYssc08+WgBIK76UwQYdsC/BwfcetJuXtztplvU4vZ7ksqhgit0zIN4/H4L8A9fU7VmHKIteECYrgbeQHgxaUfsKQQdwB2QyJ0wceWV5Pc/W5ZRAavw1ehGHm+As7deBO9dRGlGWntBgUaenCmK5wylZajf9HBm4qlgTFzKMEFxiLcCPXCWZ2nADHc/1o+R/3rk/A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=dpdk.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Gv5fhO/rijTo3nX3LLY+9ortff1h6X4bU2hwFskGbdk=; b=rNAiGTo5Kk3MNYKVrHcCOMXNdIJYyt+QYt/VUfrUmeDj2OWqUWXodrkv1V3UH98gwLdKlMJ61uzi8SJI4ZNL0xZfsNucpNP8wfB53P7EgEaGESp+xuAqcT8ghuRg18+HWLJZSTzh7vB8q9rvjLbUJ2ITKR+Y+wxNL+jalj1Hpi4= Received: from SA9PR13CA0004.namprd13.prod.outlook.com (2603:10b6:806:21::9) by CY4PR0201MB3410.namprd02.prod.outlook.com (2603:10b6:910:8a::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.10; Wed, 3 Nov 2021 14:00:39 +0000 Received: from SN1NAM02FT0008.eop-nam02.prod.protection.outlook.com (2603:10b6:806:21:cafe::36) by SA9PR13CA0004.outlook.office365.com (2603:10b6:806:21::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.4 via Frontend Transport; Wed, 3 Nov 2021 14:00:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by SN1NAM02FT0008.mail.protection.outlook.com (10.97.5.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4649.14 via Frontend Transport; Wed, 3 Nov 2021 14:00:38 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Wed, 3 Nov 2021 07:00:37 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Wed, 3 Nov 2021 07:00:37 -0700 Envelope-to: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, andrew.rybchenko@oktetlabs.ru Received: from [10.170.66.108] (port=38886 helo=xndengvm004108.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1miGof-0003SS-4o; Wed, 03 Nov 2021 07:00:37 -0700 From: Vijay Srivastava To: CC: , , , Vijay Kumar Srivastava Date: Wed, 3 Nov 2021 19:27:46 +0530 Message-ID: <20211103135754.17411-3-vsrivast@xilinx.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20211103135754.17411-1-vsrivast@xilinx.com> References: <20210706164418.32615-1-vsrivast@xilinx.com> <20211103135754.17411-1-vsrivast@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9c5186f6-2fdf-4e4e-a075-08d99ed25116 X-MS-TrafficTypeDiagnostic: CY4PR0201MB3410: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1227; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: f/m/WjT+Q6a+h2BXp/KM5pH9UUqmhst7mOj8VUWQeq1ofhnOKOHs/DB3Uq1KVcUs9joBef4sX7tjUaQtEm1LLSoMfupLQvSqKm5gODfeJCXuVfLfK0guZxfRu+/rMSrMDF3eoYKy2t4XmiVhKECqZz7X4f1AOMm1t65HxKxwqJ2s+QspGpsAWPAq2hQftoP9ecSzD7op/uWriDM6epBQrMM5EQajh0M5qXwTTQbIGFhHEOb/4amO27I8PfHSwXpy8MrUH2/+qqAyWS2IazBrhkL2SJ4/6SfLGi+hI1coOnib8iYnwapVahoFpcNru64oRJQXv9F57fP1vS99s2ZwpaZFy+RkXGbV9YOYiyPHtZfy97617ZYNQuz0QYT+4qjSfyGEzY+pPNfZemO0XLVDPUv9SUf/lsdkE0He4B5o1lN/uTn8lKfdqD+gmWXemI1thAgpgCyF+7GmcOV36cxF72a3xKT3Dwzs2r2NZ0xoYRzcsU0mcMQmQqC8Fpe3OuV0AtMCKkzZKo8Kn+PilVRr1lwIOX/m/VMXeAuTnUOqkwXpzUe7bGq07yP3WlaJ+9kZEpYtYIXG35yW/hx9xVzo0MRuwcv+hiabfv3pBNK3/JWWn14JI27sf/DMkVtzGL45PexDywl4M0d67wmNGRwlQwyHsrO95TnUgpwTV8+etN5QDa08avITOWRxkErPdvkBJ7ywBeoXOpvo9GHX+mf6tPvhjwYkRCzMBOwtYv3QOIY= X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(46966006)(36840700001)(4326008)(6666004)(2616005)(36860700001)(1076003)(316002)(54906003)(36906005)(2906002)(47076005)(30864003)(83380400001)(5660300002)(70586007)(7696005)(186003)(82310400003)(70206006)(356005)(44832011)(26005)(6916009)(508600001)(9786002)(107886003)(7636003)(36756003)(8676002)(8936002)(336012)(426003)(102446001); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 14:00:38.9584 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9c5186f6-2fdf-4e4e-a075-08d99ed25116 X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: SN1NAM02FT0008.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR0201MB3410 Subject: [dpdk-dev] [PATCH v4 02/10] vdpa/sfc: add support for device initialization X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vijay Kumar Srivastava Add HW initialization and vDPA device registration support. Signed-off-by: Vijay Kumar Srivastava Acked-by: Andrew Rybchenko Reviewed-by: Maxime Coquelin --- v2: * Used rte_memzone_reserve_aligned for mcdi buffer allocation. * Freeing mcdi buff when DMA map fails. * Fixed one typo. v4: * Removed unused enum fields. * Updated format specifiers in log messages. * Changed do/while loop used during mcdi buffer mapping into for() loop. doc/guides/vdpadevs/sfc.rst | 6 + drivers/vdpa/sfc/meson.build | 3 + drivers/vdpa/sfc/sfc_vdpa.c | 23 +++ drivers/vdpa/sfc/sfc_vdpa.h | 49 +++++- drivers/vdpa/sfc/sfc_vdpa_debug.h | 21 +++ drivers/vdpa/sfc/sfc_vdpa_hw.c | 326 ++++++++++++++++++++++++++++++++++++++ drivers/vdpa/sfc/sfc_vdpa_log.h | 3 + drivers/vdpa/sfc/sfc_vdpa_mcdi.c | 74 +++++++++ drivers/vdpa/sfc/sfc_vdpa_ops.c | 129 +++++++++++++++ drivers/vdpa/sfc/sfc_vdpa_ops.h | 34 ++++ 10 files changed, 667 insertions(+), 1 deletion(-) create mode 100644 drivers/vdpa/sfc/sfc_vdpa_debug.h create mode 100644 drivers/vdpa/sfc/sfc_vdpa_hw.c create mode 100644 drivers/vdpa/sfc/sfc_vdpa_mcdi.c create mode 100644 drivers/vdpa/sfc/sfc_vdpa_ops.c create mode 100644 drivers/vdpa/sfc/sfc_vdpa_ops.h diff --git a/doc/guides/vdpadevs/sfc.rst b/doc/guides/vdpadevs/sfc.rst index 44e694f..d06c427 100644 --- a/doc/guides/vdpadevs/sfc.rst +++ b/doc/guides/vdpadevs/sfc.rst @@ -95,3 +95,9 @@ SFC vDPA PMD provides the following log types available for control: Matches a subset of per-port log types registered during runtime. A full name for a particular type may be obtained by appending a dot and a PCI device identifier (``XXXX:XX:XX.X``) to the prefix. + +- ``pmd.vdpa.sfc.mcdi`` (default level is **notice**) + + Extra logging of the communication with the NIC's management CPU. + The format of the log is consumed by the netlogdecode cross-platform + tool. May be managed per-port, as explained above. diff --git a/drivers/vdpa/sfc/meson.build b/drivers/vdpa/sfc/meson.build index 4255d65..dc333de 100644 --- a/drivers/vdpa/sfc/meson.build +++ b/drivers/vdpa/sfc/meson.build @@ -19,4 +19,7 @@ endforeach deps += ['common_sfc_efx', 'bus_pci'] sources = files( 'sfc_vdpa.c', + 'sfc_vdpa_hw.c', + 'sfc_vdpa_mcdi.c', + 'sfc_vdpa_ops.c', ) diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c index f087944..da75ba6 100644 --- a/drivers/vdpa/sfc/sfc_vdpa.c +++ b/drivers/vdpa/sfc/sfc_vdpa.c @@ -228,6 +228,19 @@ struct sfc_vdpa_adapter * goto fail_vfio_setup; } + sfc_vdpa_log_init(sva, "hw init"); + if (sfc_vdpa_hw_init(sva) != 0) { + sfc_vdpa_err(sva, "failed to init HW %s", pci_dev->name); + goto fail_hw_init; + } + + sfc_vdpa_log_init(sva, "dev init"); + sva->ops_data = sfc_vdpa_device_init(sva, SFC_VDPA_AS_VF); + if (sva->ops_data == NULL) { + sfc_vdpa_err(sva, "failed vDPA dev init %s", pci_dev->name); + goto fail_dev_init; + } + pthread_mutex_lock(&sfc_vdpa_adapter_list_lock); TAILQ_INSERT_TAIL(&sfc_vdpa_adapter_list, sva, next); pthread_mutex_unlock(&sfc_vdpa_adapter_list_lock); @@ -236,6 +249,12 @@ struct sfc_vdpa_adapter * return 0; +fail_dev_init: + sfc_vdpa_hw_fini(sva); + +fail_hw_init: + sfc_vdpa_vfio_teardown(sva); + fail_vfio_setup: fail_set_log_prefix: rte_free(sva); @@ -262,6 +281,10 @@ struct sfc_vdpa_adapter * TAILQ_REMOVE(&sfc_vdpa_adapter_list, sva, next); pthread_mutex_unlock(&sfc_vdpa_adapter_list_lock); + sfc_vdpa_device_fini(sva->ops_data); + + sfc_vdpa_hw_fini(sva); + sfc_vdpa_vfio_teardown(sva); rte_free(sva); diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h index 2ae9ec3..bd3c437 100644 --- a/drivers/vdpa/sfc/sfc_vdpa.h +++ b/drivers/vdpa/sfc/sfc_vdpa.h @@ -11,13 +11,37 @@ #include +#include "sfc_efx.h" +#include "sfc_efx_mcdi.h" +#include "sfc_vdpa_debug.h" #include "sfc_vdpa_log.h" +#include "sfc_vdpa_ops.h" + +#define SFC_VDPA_DEFAULT_MCDI_IOVA 0x200000000000 /* Adapter private data */ struct sfc_vdpa_adapter { TAILQ_ENTRY(sfc_vdpa_adapter) next; + /* + * PMD setup and configuration is not thread safe. Since it is not + * performance sensitive, it is better to guarantee thread-safety + * and add device level lock. vDPA control operations which + * change its state should acquire the lock. + */ + rte_spinlock_t lock; struct rte_pci_device *pdev; + efx_family_t family; + efx_nic_t *nic; + rte_spinlock_t nic_lock; + + efsys_bar_t mem_bar; + + struct sfc_efx_mcdi mcdi; + size_t mcdi_buff_size; + + uint32_t max_queue_count; + char log_prefix[SFC_VDPA_LOG_PREFIX_MAX]; uint32_t logtype_main; @@ -25,6 +49,7 @@ struct sfc_vdpa_adapter { int vfio_dev_fd; int vfio_container_fd; int iommu_group_num; + struct sfc_vdpa_ops_data *ops_data; }; uint32_t @@ -35,5 +60,27 @@ struct sfc_vdpa_adapter { struct sfc_vdpa_adapter * sfc_vdpa_get_adapter_by_dev(struct rte_pci_device *pdev); -#endif /* _SFC_VDPA_H */ +int +sfc_vdpa_hw_init(struct sfc_vdpa_adapter *sva); +void +sfc_vdpa_hw_fini(struct sfc_vdpa_adapter *sva); +int +sfc_vdpa_mcdi_init(struct sfc_vdpa_adapter *sva); +void +sfc_vdpa_mcdi_fini(struct sfc_vdpa_adapter *sva); + +int +sfc_vdpa_dma_alloc(struct sfc_vdpa_adapter *sva, const char *name, + size_t len, efsys_mem_t *esmp); + +void +sfc_vdpa_dma_free(struct sfc_vdpa_adapter *sva, efsys_mem_t *esmp); + +static inline struct sfc_vdpa_adapter * +sfc_vdpa_adapter_by_dev_handle(void *dev_handle) +{ + return (struct sfc_vdpa_adapter *)dev_handle; +} + +#endif /* _SFC_VDPA_H */ diff --git a/drivers/vdpa/sfc/sfc_vdpa_debug.h b/drivers/vdpa/sfc/sfc_vdpa_debug.h new file mode 100644 index 0000000..cfa8cc5 --- /dev/null +++ b/drivers/vdpa/sfc/sfc_vdpa_debug.h @@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * + * Copyright(c) 2020-2021 Xilinx, Inc. + */ + +#ifndef _SFC_VDPA_DEBUG_H_ +#define _SFC_VDPA_DEBUG_H_ + +#include + +#ifdef RTE_LIBRTE_SFC_VDPA_DEBUG +/* Avoid dependency from RTE_LOG_DP_LEVEL to be able to enable debug check + * in the driver only. + */ +#define SFC_VDPA_ASSERT(exp) RTE_VERIFY(exp) +#else +/* If the driver debug is not enabled, follow DPDK debug/non-debug */ +#define SFC_VDPA_ASSERT(exp) RTE_ASSERT(exp) +#endif + +#endif /* _SFC_VDPA_DEBUG_H_ */ diff --git a/drivers/vdpa/sfc/sfc_vdpa_hw.c b/drivers/vdpa/sfc/sfc_vdpa_hw.c new file mode 100644 index 0000000..dfb6d4f --- /dev/null +++ b/drivers/vdpa/sfc/sfc_vdpa_hw.c @@ -0,0 +1,326 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * + * Copyright(c) 2020-2021 Xilinx, Inc. + */ + +#include + +#include +#include +#include + +#include "efx.h" +#include "sfc_vdpa.h" +#include "sfc_vdpa_ops.h" + +extern uint32_t sfc_logtype_driver; + +#ifndef PAGE_SIZE +#define PAGE_SIZE (sysconf(_SC_PAGESIZE)) +#endif + +int +sfc_vdpa_dma_alloc(struct sfc_vdpa_adapter *sva, const char *name, + size_t len, efsys_mem_t *esmp) +{ + uint64_t mcdi_iova; + size_t mcdi_buff_size; + const struct rte_memzone *mz = NULL; + int numa_node = sva->pdev->device.numa_node; + int ret; + + mcdi_buff_size = RTE_ALIGN_CEIL(len, PAGE_SIZE); + + sfc_vdpa_log_init(sva, "name=%s, len=%zu", name, len); + + mz = rte_memzone_reserve_aligned(name, mcdi_buff_size, + numa_node, + RTE_MEMZONE_IOVA_CONTIG, + PAGE_SIZE); + if (mz == NULL) { + sfc_vdpa_err(sva, "cannot reserve memory for %s: len=%#x: %s", + name, (unsigned int)len, rte_strerror(rte_errno)); + return -ENOMEM; + } + + /* IOVA address for MCDI would be re-calculated if mapping + * using default IOVA would fail. + * TODO: Earlier there was no way to get valid IOVA range. + * Recently a patch has been submitted to get the IOVA range + * using ioctl. VFIO_IOMMU_GET_INFO. This patch is available + * in the kernel version >= 5.4. Support to get the default + * IOVA address for MCDI buffer using available IOVA range + * would be added later. Meanwhile default IOVA for MCDI buffer + * is kept at high mem at 2TB. In case of overlap new available + * addresses would be searched and same would be used. + */ + mcdi_iova = SFC_VDPA_DEFAULT_MCDI_IOVA; + + for (;;) { + ret = rte_vfio_container_dma_map(sva->vfio_container_fd, + (uint64_t)mz->addr, mcdi_iova, + mcdi_buff_size); + if (ret == 0) + break; + + mcdi_iova = mcdi_iova >> 1; + if (mcdi_iova < mcdi_buff_size) { + sfc_vdpa_err(sva, + "DMA mapping failed for MCDI : %s", + rte_strerror(rte_errno)); + rte_memzone_free(mz); + return ret; + } + } + + esmp->esm_addr = mcdi_iova; + esmp->esm_base = mz->addr; + sva->mcdi_buff_size = mcdi_buff_size; + + sfc_vdpa_info(sva, + "DMA name=%s len=%zu => virt=%p iova=0x%" PRIx64, + name, len, esmp->esm_base, esmp->esm_addr); + + return 0; +} + +void +sfc_vdpa_dma_free(struct sfc_vdpa_adapter *sva, efsys_mem_t *esmp) +{ + int ret; + + sfc_vdpa_log_init(sva, "name=%s", esmp->esm_mz->name); + + ret = rte_vfio_container_dma_unmap(sva->vfio_container_fd, + (uint64_t)esmp->esm_base, + esmp->esm_addr, sva->mcdi_buff_size); + if (ret < 0) + sfc_vdpa_err(sva, "DMA unmap failed for MCDI : %s", + rte_strerror(rte_errno)); + + sfc_vdpa_info(sva, + "DMA free name=%s => virt=%p iova=0x%" PRIx64, + esmp->esm_mz->name, esmp->esm_base, esmp->esm_addr); + + rte_free((void *)(esmp->esm_base)); + + sva->mcdi_buff_size = 0; + memset(esmp, 0, sizeof(*esmp)); +} + +static int +sfc_vdpa_mem_bar_init(struct sfc_vdpa_adapter *sva, + const efx_bar_region_t *mem_ebrp) +{ + struct rte_pci_device *pci_dev = sva->pdev; + efsys_bar_t *ebp = &sva->mem_bar; + struct rte_mem_resource *res = + &pci_dev->mem_resource[mem_ebrp->ebr_index]; + + SFC_BAR_LOCK_INIT(ebp, pci_dev->name); + ebp->esb_rid = mem_ebrp->ebr_index; + ebp->esb_dev = pci_dev; + ebp->esb_base = res->addr; + + return 0; +} + +static void +sfc_vdpa_mem_bar_fini(struct sfc_vdpa_adapter *sva) +{ + efsys_bar_t *ebp = &sva->mem_bar; + + SFC_BAR_LOCK_DESTROY(ebp); + memset(ebp, 0, sizeof(*ebp)); +} + +static int +sfc_vdpa_nic_probe(struct sfc_vdpa_adapter *sva) +{ + efx_nic_t *enp = sva->nic; + int rc; + + rc = efx_nic_probe(enp, EFX_FW_VARIANT_DONT_CARE); + if (rc != 0) + sfc_vdpa_err(sva, "nic probe failed: %s", rte_strerror(rc)); + + return rc; +} + +static int +sfc_vdpa_estimate_resource_limits(struct sfc_vdpa_adapter *sva) +{ + efx_drv_limits_t limits; + int rc; + uint32_t evq_allocated; + uint32_t rxq_allocated; + uint32_t txq_allocated; + uint32_t max_queue_cnt; + + memset(&limits, 0, sizeof(limits)); + + /* Request at least one Rx and Tx queue */ + limits.edl_min_rxq_count = 1; + limits.edl_min_txq_count = 1; + /* Management event queue plus event queue for Tx/Rx queue */ + limits.edl_min_evq_count = + 1 + RTE_MAX(limits.edl_min_rxq_count, limits.edl_min_txq_count); + + limits.edl_max_rxq_count = SFC_VDPA_MAX_QUEUE_PAIRS; + limits.edl_max_txq_count = SFC_VDPA_MAX_QUEUE_PAIRS; + limits.edl_max_evq_count = 1 + SFC_VDPA_MAX_QUEUE_PAIRS; + + SFC_VDPA_ASSERT(limits.edl_max_evq_count >= limits.edl_min_rxq_count); + SFC_VDPA_ASSERT(limits.edl_max_rxq_count >= limits.edl_min_rxq_count); + SFC_VDPA_ASSERT(limits.edl_max_txq_count >= limits.edl_min_rxq_count); + + /* Configure the minimum required resources needed for the + * driver to operate, and the maximum desired resources that the + * driver is capable of using. + */ + sfc_vdpa_log_init(sva, "set drv limit"); + efx_nic_set_drv_limits(sva->nic, &limits); + + sfc_vdpa_log_init(sva, "init nic"); + rc = efx_nic_init(sva->nic); + if (rc != 0) { + sfc_vdpa_err(sva, "nic init failed: %s", rte_strerror(rc)); + goto fail_nic_init; + } + + /* Find resource dimensions assigned by firmware to this function */ + rc = efx_nic_get_vi_pool(sva->nic, &evq_allocated, &rxq_allocated, + &txq_allocated); + if (rc != 0) { + sfc_vdpa_err(sva, "vi pool get failed: %s", rte_strerror(rc)); + goto fail_get_vi_pool; + } + + /* It still may allocate more than maximum, ensure limit */ + evq_allocated = RTE_MIN(evq_allocated, limits.edl_max_evq_count); + rxq_allocated = RTE_MIN(rxq_allocated, limits.edl_max_rxq_count); + txq_allocated = RTE_MIN(txq_allocated, limits.edl_max_txq_count); + + + max_queue_cnt = RTE_MIN(rxq_allocated, txq_allocated); + /* Subtract management EVQ not used for traffic */ + max_queue_cnt = RTE_MIN(evq_allocated - 1, max_queue_cnt); + + SFC_VDPA_ASSERT(max_queue_cnt > 0); + + sva->max_queue_count = max_queue_cnt; + + return 0; + +fail_get_vi_pool: + efx_nic_fini(sva->nic); +fail_nic_init: + sfc_vdpa_log_init(sva, "failed: %s", rte_strerror(rc)); + return rc; +} + +int +sfc_vdpa_hw_init(struct sfc_vdpa_adapter *sva) +{ + efx_bar_region_t mem_ebr; + efx_nic_t *enp; + int rc; + + sfc_vdpa_log_init(sva, "entry"); + + sfc_vdpa_log_init(sva, "get family"); + rc = sfc_efx_family(sva->pdev, &mem_ebr, &sva->family); + if (rc != 0) + goto fail_family; + sfc_vdpa_log_init(sva, + "family is %u, membar is %d," + "function control window offset is %#" PRIx64, + sva->family, mem_ebr.ebr_index, mem_ebr.ebr_offset); + + sfc_vdpa_log_init(sva, "init mem bar"); + rc = sfc_vdpa_mem_bar_init(sva, &mem_ebr); + if (rc != 0) + goto fail_mem_bar_init; + + sfc_vdpa_log_init(sva, "create nic"); + rte_spinlock_init(&sva->nic_lock); + rc = efx_nic_create(sva->family, (efsys_identifier_t *)sva, + &sva->mem_bar, mem_ebr.ebr_offset, + &sva->nic_lock, &enp); + if (rc != 0) { + sfc_vdpa_err(sva, "nic create failed: %s", rte_strerror(rc)); + goto fail_nic_create; + } + sva->nic = enp; + + sfc_vdpa_log_init(sva, "init mcdi"); + rc = sfc_vdpa_mcdi_init(sva); + if (rc != 0) { + sfc_vdpa_err(sva, "mcdi init failed: %s", rte_strerror(rc)); + goto fail_mcdi_init; + } + + sfc_vdpa_log_init(sva, "probe nic"); + rc = sfc_vdpa_nic_probe(sva); + if (rc != 0) + goto fail_nic_probe; + + sfc_vdpa_log_init(sva, "reset nic"); + rc = efx_nic_reset(enp); + if (rc != 0) { + sfc_vdpa_err(sva, "nic reset failed: %s", rte_strerror(rc)); + goto fail_nic_reset; + } + + sfc_vdpa_log_init(sva, "estimate resource limits"); + rc = sfc_vdpa_estimate_resource_limits(sva); + if (rc != 0) + goto fail_estimate_rsrc_limits; + + sfc_vdpa_log_init(sva, "done"); + + return 0; + +fail_estimate_rsrc_limits: +fail_nic_reset: + efx_nic_unprobe(enp); + +fail_nic_probe: + sfc_vdpa_mcdi_fini(sva); + +fail_mcdi_init: + sfc_vdpa_log_init(sva, "destroy nic"); + sva->nic = NULL; + efx_nic_destroy(enp); + +fail_nic_create: + sfc_vdpa_mem_bar_fini(sva); + +fail_mem_bar_init: +fail_family: + sfc_vdpa_log_init(sva, "failed: %s", rte_strerror(rc)); + return rc; +} + +void +sfc_vdpa_hw_fini(struct sfc_vdpa_adapter *sva) +{ + efx_nic_t *enp = sva->nic; + + sfc_vdpa_log_init(sva, "entry"); + + sfc_vdpa_log_init(sva, "unprobe nic"); + efx_nic_unprobe(enp); + + sfc_vdpa_log_init(sva, "mcdi fini"); + sfc_vdpa_mcdi_fini(sva); + + sfc_vdpa_log_init(sva, "nic fini"); + efx_nic_fini(enp); + + sfc_vdpa_log_init(sva, "destroy nic"); + sva->nic = NULL; + efx_nic_destroy(enp); + + sfc_vdpa_mem_bar_fini(sva); +} diff --git a/drivers/vdpa/sfc/sfc_vdpa_log.h b/drivers/vdpa/sfc/sfc_vdpa_log.h index 858e5ee..4e7a84f 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_log.h +++ b/drivers/vdpa/sfc/sfc_vdpa_log.h @@ -21,6 +21,9 @@ /** Name prefix for the per-device log type used to report basic information */ #define SFC_VDPA_LOGTYPE_MAIN_STR SFC_VDPA_LOGTYPE_PREFIX "main" +/** Device MCDI log type name prefix */ +#define SFC_VDPA_LOGTYPE_MCDI_STR SFC_VDPA_LOGTYPE_PREFIX "mcdi" + #define SFC_VDPA_LOG_PREFIX_MAX 32 /* Log PMD message, automatically add prefix and \n */ diff --git a/drivers/vdpa/sfc/sfc_vdpa_mcdi.c b/drivers/vdpa/sfc/sfc_vdpa_mcdi.c new file mode 100644 index 0000000..961d2d3 --- /dev/null +++ b/drivers/vdpa/sfc/sfc_vdpa_mcdi.c @@ -0,0 +1,74 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * + * Copyright(c) 2020-2021 Xilinx, Inc. + */ + +#include "sfc_efx_mcdi.h" + +#include "sfc_vdpa.h" +#include "sfc_vdpa_debug.h" +#include "sfc_vdpa_log.h" + +static sfc_efx_mcdi_dma_alloc_cb sfc_vdpa_mcdi_dma_alloc; +static int +sfc_vdpa_mcdi_dma_alloc(void *cookie, const char *name, size_t len, + efsys_mem_t *esmp) +{ + struct sfc_vdpa_adapter *sva = cookie; + + return sfc_vdpa_dma_alloc(sva, name, len, esmp); +} + +static sfc_efx_mcdi_dma_free_cb sfc_vdpa_mcdi_dma_free; +static void +sfc_vdpa_mcdi_dma_free(void *cookie, efsys_mem_t *esmp) +{ + struct sfc_vdpa_adapter *sva = cookie; + + sfc_vdpa_dma_free(sva, esmp); +} + +static sfc_efx_mcdi_sched_restart_cb sfc_vdpa_mcdi_sched_restart; +static void +sfc_vdpa_mcdi_sched_restart(void *cookie) +{ + RTE_SET_USED(cookie); +} + +static sfc_efx_mcdi_mgmt_evq_poll_cb sfc_vdpa_mcdi_mgmt_evq_poll; +static void +sfc_vdpa_mcdi_mgmt_evq_poll(void *cookie) +{ + RTE_SET_USED(cookie); +} + +static const struct sfc_efx_mcdi_ops sfc_vdpa_mcdi_ops = { + .dma_alloc = sfc_vdpa_mcdi_dma_alloc, + .dma_free = sfc_vdpa_mcdi_dma_free, + .sched_restart = sfc_vdpa_mcdi_sched_restart, + .mgmt_evq_poll = sfc_vdpa_mcdi_mgmt_evq_poll, + +}; + +int +sfc_vdpa_mcdi_init(struct sfc_vdpa_adapter *sva) +{ + uint32_t logtype; + + sfc_vdpa_log_init(sva, "entry"); + + logtype = sfc_vdpa_register_logtype(&(sva->pdev->addr), + SFC_VDPA_LOGTYPE_MCDI_STR, + RTE_LOG_NOTICE); + + return sfc_efx_mcdi_init(&sva->mcdi, logtype, + sva->log_prefix, sva->nic, + &sfc_vdpa_mcdi_ops, sva); +} + +void +sfc_vdpa_mcdi_fini(struct sfc_vdpa_adapter *sva) +{ + sfc_vdpa_log_init(sva, "entry"); + sfc_efx_mcdi_fini(&sva->mcdi); +} diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.c b/drivers/vdpa/sfc/sfc_vdpa_ops.c new file mode 100644 index 0000000..71696be --- /dev/null +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.c @@ -0,0 +1,129 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * + * Copyright(c) 2020-2021 Xilinx, Inc. + */ + +#include +#include +#include +#include + +#include "sfc_vdpa_ops.h" +#include "sfc_vdpa.h" + +/* Dummy functions for mandatory vDPA ops to pass vDPA device registration. + * In subsequent patches these ops would be implemented. + */ +static int +sfc_vdpa_get_queue_num(struct rte_vdpa_device *vdpa_dev, uint32_t *queue_num) +{ + RTE_SET_USED(vdpa_dev); + RTE_SET_USED(queue_num); + + return -1; +} + +static int +sfc_vdpa_get_features(struct rte_vdpa_device *vdpa_dev, uint64_t *features) +{ + RTE_SET_USED(vdpa_dev); + RTE_SET_USED(features); + + return -1; +} + +static int +sfc_vdpa_get_protocol_features(struct rte_vdpa_device *vdpa_dev, + uint64_t *features) +{ + RTE_SET_USED(vdpa_dev); + RTE_SET_USED(features); + + return -1; +} + +static int +sfc_vdpa_dev_config(int vid) +{ + RTE_SET_USED(vid); + + return -1; +} + +static int +sfc_vdpa_dev_close(int vid) +{ + RTE_SET_USED(vid); + + return -1; +} + +static int +sfc_vdpa_set_vring_state(int vid, int vring, int state) +{ + RTE_SET_USED(vid); + RTE_SET_USED(vring); + RTE_SET_USED(state); + + return -1; +} + +static int +sfc_vdpa_set_features(int vid) +{ + RTE_SET_USED(vid); + + return -1; +} + +static struct rte_vdpa_dev_ops sfc_vdpa_ops = { + .get_queue_num = sfc_vdpa_get_queue_num, + .get_features = sfc_vdpa_get_features, + .get_protocol_features = sfc_vdpa_get_protocol_features, + .dev_conf = sfc_vdpa_dev_config, + .dev_close = sfc_vdpa_dev_close, + .set_vring_state = sfc_vdpa_set_vring_state, + .set_features = sfc_vdpa_set_features, +}; + +struct sfc_vdpa_ops_data * +sfc_vdpa_device_init(void *dev_handle, enum sfc_vdpa_context context) +{ + struct sfc_vdpa_ops_data *ops_data; + struct rte_pci_device *pci_dev; + + /* Create vDPA ops context */ + ops_data = rte_zmalloc("vdpa", sizeof(struct sfc_vdpa_ops_data), 0); + if (ops_data == NULL) + return NULL; + + ops_data->vdpa_context = context; + ops_data->dev_handle = dev_handle; + + pci_dev = sfc_vdpa_adapter_by_dev_handle(dev_handle)->pdev; + + /* Register vDPA Device */ + sfc_vdpa_log_init(dev_handle, "register vDPA device"); + ops_data->vdpa_dev = + rte_vdpa_register_device(&pci_dev->device, &sfc_vdpa_ops); + if (ops_data->vdpa_dev == NULL) { + sfc_vdpa_err(dev_handle, "vDPA device registration failed"); + goto fail_register_device; + } + + ops_data->state = SFC_VDPA_STATE_INITIALIZED; + + return ops_data; + +fail_register_device: + rte_free(ops_data); + return NULL; +} + +void +sfc_vdpa_device_fini(struct sfc_vdpa_ops_data *ops_data) +{ + rte_vdpa_unregister_device(ops_data->vdpa_dev); + + rte_free(ops_data); +} diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.h b/drivers/vdpa/sfc/sfc_vdpa_ops.h new file mode 100644 index 0000000..baa4293 --- /dev/null +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * + * Copyright(c) 2020-2021 Xilinx, Inc. + */ + +#ifndef _SFC_VDPA_OPS_H +#define _SFC_VDPA_OPS_H + +#include + +#define SFC_VDPA_MAX_QUEUE_PAIRS 1 + +enum sfc_vdpa_context { + SFC_VDPA_AS_VF +}; + +enum sfc_vdpa_state { + SFC_VDPA_STATE_UNINITIALIZED = 0, + SFC_VDPA_STATE_INITIALIZED, +}; + +struct sfc_vdpa_ops_data { + void *dev_handle; + struct rte_vdpa_device *vdpa_dev; + enum sfc_vdpa_context vdpa_context; + enum sfc_vdpa_state state; +}; + +struct sfc_vdpa_ops_data * +sfc_vdpa_device_init(void *adapter, enum sfc_vdpa_context context); +void +sfc_vdpa_device_fini(struct sfc_vdpa_ops_data *ops_data); + +#endif /* _SFC_VDPA_OPS_H */ From patchwork Wed Nov 3 13:57:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijay Srivastava X-Patchwork-Id: 103661 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CCDD0A0C53; Wed, 3 Nov 2021 15:01:09 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1114042737; Wed, 3 Nov 2021 15:00:46 +0100 (CET) Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam08on2057.outbound.protection.outlook.com [40.107.102.57]) by mails.dpdk.org (Postfix) with ESMTP id 8DB8042722 for ; Wed, 3 Nov 2021 15:00:43 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=US50NO0ApO01oUWX1Nuycn8k8JrFUPqMahbsHqYLFA+3ZxJPPN2q7obhdHM3e4Db9kJtHz0Gn9kRZnRIiKZpRn46siayiJjEQN61m+oh2b6Nq2rQk0/GGrYnqcysWllgsoNnRc/hPBvdiu2CuVSuJkmUVIAb/RzhD6CjV/FZH1tfrnV8Xh0q6RLcbSbZqWeASaYAvfdlF9xkan1QEGTBQp4yzmgLCBXNkWReArQG06LUKyuVUvsOWsPgRSKp08gFMBvJLr4aOWYSvZW9nFr35+joyzhUGa3t6qXzJFyMh0OQw/26t9z5uFJlwDPlpbpb71h38/6MxuAb+Wk6Vka4Tg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WqW/2glLI9n90psUhWdD2hyl0OYvTU38HVZQ6puFmcE=; b=NUKKiR4F6K5r7TrK/PrVBTpBa1JTdjNOJn83EnrQEiYHETK6yRgpkuU+IjzH/60/GypCF1tCS/Pm93ALWu6zARue2/I7SDAStUS/feGLnxrhAFh/Aok6Idhikp0+T1k87rMNtRU/O1Lze0feNVpW9xIF1AhuTxKpKe7KTsmC0aVc44DpKDvCAkr29DX0TpwvxAQMS2/cHk4siEeuL9XCmyETsIvIzFjveosDCxG3/1cthCMWkUMz1pN59rbiAe/cUsGQEhubUH/I0VTFYfkmV+6oLoOTJ0tSCk5da2jjX9LNnCsjBcHQ4XBGHDUrEWk57YbBTov7NMTOi0aJh8cOqQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=dpdk.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WqW/2glLI9n90psUhWdD2hyl0OYvTU38HVZQ6puFmcE=; b=eQq39PLjvcrbi0W13MdtYKqzXAzay79TS3ZZTSwZmsKtgK7q0X9FwUo99pETMDDLglo6lb/O4T5P5ICZcwEbFGK6aobY+N7icMO7jm84IkYyA4/qriIj2ys461GczLNmqjeGHraWXXI19eOdxu46Cs4BS+A8xEBYGsMbRsQWZa8= Received: from SA9PR13CA0022.namprd13.prod.outlook.com (2603:10b6:806:21::27) by PH0PR02MB7784.namprd02.prod.outlook.com (2603:10b6:510:58::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.10; Wed, 3 Nov 2021 14:00:41 +0000 Received: from SN1NAM02FT0008.eop-nam02.prod.protection.outlook.com (2603:10b6:806:21:cafe::86) by SA9PR13CA0022.outlook.office365.com (2603:10b6:806:21::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.6 via Frontend Transport; Wed, 3 Nov 2021 14:00:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by SN1NAM02FT0008.mail.protection.outlook.com (10.97.5.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4649.14 via Frontend Transport; Wed, 3 Nov 2021 14:00:40 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Wed, 3 Nov 2021 07:00:40 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Wed, 3 Nov 2021 07:00:40 -0700 Envelope-to: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, andrew.rybchenko@oktetlabs.ru Received: from [10.170.66.108] (port=38886 helo=xndengvm004108.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1miGoh-0003SS-Kw; Wed, 03 Nov 2021 07:00:40 -0700 From: Vijay Srivastava To: CC: , , , Vijay Kumar Srivastava Date: Wed, 3 Nov 2021 19:27:47 +0530 Message-ID: <20211103135754.17411-4-vsrivast@xilinx.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20211103135754.17411-1-vsrivast@xilinx.com> References: <20210706164418.32615-1-vsrivast@xilinx.com> <20211103135754.17411-1-vsrivast@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: df2bfa4e-4306-425b-5663-08d99ed25227 X-MS-TrafficTypeDiagnostic: PH0PR02MB7784: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1751; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5nDqNKCINJe76gL1K4F7xcHrYMrvdhha/+/lt3vIydj9nKFzC4EoayT/qKDSj1YRtZDpdaSNwSuLH2pZe6FQmZ88puZ429zNDAVqIw5fW8e6RajIeVHuETR4aP5q5J1TT/P8ZGTGettFdEvemPuo8Uxk84ibheRGBS24Sa5wJAdXsfD5mnZv3Vave3FikqsH7mDHYPD26eA5O0X5HXZg9jOb2NbOiAqvDKirRq7CAEbD3bdb5HnHxseuhP4vjngHhCPVBqlTGLcu6XzrpSk6m4b0NVgdtdDCA+qEqTe2k+IliwJ3NiPwjboTe4A967GWOyi47InPkBSC42LrlRFrDwb6gvyEqobnX6uJ98OPkt3gsbM2COGTB3hOcQWiZ5vb2VAgrUttGypfdiKiBhsJj7MA1Tte1pMpuVVUSOrUew/SrSirJ9uv5/TtA8nzc5kpKvRzTpbMnsBvvCe+QZZb9f4Eik+XY6+u57jGAtVC6xv39XNGhYsFqHUiyyCZ42kCYcQhRCJKsvl+3HSfp2M1AwSZ+Y8/MG23vLvagGvwhhM9Ic8hYVhGzF13TZ/dmKsNy6HQ7R4b/V5imRcp9rVt8KQ6hftfFht8adr8jkenQR67lab75BsoJFlIVoC46KNOho4nffHluH9Gc/FynB2n3IkDqVZFW8P+SyW86XTHMB7BGgRhrK4mmUMxM4B64TBb0XXD8JQjr1LqrroS+70t8VKeBmqvzi/QNTO96oe7MT8= X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(36840700001)(46966006)(54906003)(7636003)(26005)(8936002)(5660300002)(4326008)(36860700001)(36906005)(36756003)(6916009)(336012)(44832011)(107886003)(2906002)(70586007)(356005)(1076003)(47076005)(316002)(186003)(8676002)(7696005)(82310400003)(508600001)(83380400001)(2616005)(70206006)(9786002)(426003)(102446001); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 14:00:40.7456 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: df2bfa4e-4306-425b-5663-08d99ed25227 X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: SN1NAM02FT0008.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR02MB7784 Subject: [dpdk-dev] [PATCH v4 03/10] vdpa/sfc: add support to get device and protocol features X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vijay Kumar Srivastava Implement vDPA ops get_feature and get_protocol_features. This patch retrieves device supported features and enables protocol features. Signed-off-by: Vijay Kumar Srivastava Acked-by: Andrew Rybchenko Reviewed-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- doc/guides/vdpadevs/features/sfc.ini | 10 ++++ drivers/common/sfc_efx/efsys.h | 2 +- drivers/common/sfc_efx/version.map | 10 ++++ drivers/vdpa/sfc/sfc_vdpa.c | 20 ++++++++ drivers/vdpa/sfc/sfc_vdpa.h | 2 + drivers/vdpa/sfc/sfc_vdpa_hw.c | 13 ++++++ drivers/vdpa/sfc/sfc_vdpa_ops.c | 91 ++++++++++++++++++++++++++++++++---- drivers/vdpa/sfc/sfc_vdpa_ops.h | 3 ++ 8 files changed, 142 insertions(+), 9 deletions(-) diff --git a/doc/guides/vdpadevs/features/sfc.ini b/doc/guides/vdpadevs/features/sfc.ini index 71b6158..700d061 100644 --- a/doc/guides/vdpadevs/features/sfc.ini +++ b/doc/guides/vdpadevs/features/sfc.ini @@ -4,6 +4,16 @@ ; Refer to default.ini for the full list of available driver features. ; [Features] +csum = Y +guest csum = Y +host tso4 = Y +host tso6 = Y +version 1 = Y +mrg rxbuf = Y +any layout = Y +in_order = Y +proto host notifier = Y +IOMMU platform = Y Linux = Y x86-64 = Y Usage doc = Y diff --git a/drivers/common/sfc_efx/efsys.h b/drivers/common/sfc_efx/efsys.h index d133d61..37ec6b9 100644 --- a/drivers/common/sfc_efx/efsys.h +++ b/drivers/common/sfc_efx/efsys.h @@ -187,7 +187,7 @@ #define EFSYS_OPT_MAE 1 -#define EFSYS_OPT_VIRTIO 0 +#define EFSYS_OPT_VIRTIO 1 /* ID */ diff --git a/drivers/common/sfc_efx/version.map b/drivers/common/sfc_efx/version.map index 642a62e..ec86220 100644 --- a/drivers/common/sfc_efx/version.map +++ b/drivers/common/sfc_efx/version.map @@ -247,6 +247,16 @@ INTERNAL { efx_txq_nbufs; efx_txq_size; + efx_virtio_fini; + efx_virtio_get_doorbell_offset; + efx_virtio_get_features; + efx_virtio_init; + efx_virtio_qcreate; + efx_virtio_qdestroy; + efx_virtio_qstart; + efx_virtio_qstop; + efx_virtio_verify_features; + sfc_efx_dev_class_get; sfc_efx_family; diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c index da75ba6..7b9ebcc 100644 --- a/drivers/vdpa/sfc/sfc_vdpa.c +++ b/drivers/vdpa/sfc/sfc_vdpa.c @@ -43,6 +43,26 @@ struct sfc_vdpa_adapter * return found ? sva : NULL; } +struct sfc_vdpa_ops_data * +sfc_vdpa_get_data_by_dev(struct rte_vdpa_device *vdpa_dev) +{ + bool found = false; + struct sfc_vdpa_adapter *sva; + + pthread_mutex_lock(&sfc_vdpa_adapter_list_lock); + + TAILQ_FOREACH(sva, &sfc_vdpa_adapter_list, next) { + if (vdpa_dev == sva->ops_data->vdpa_dev) { + found = true; + break; + } + } + + pthread_mutex_unlock(&sfc_vdpa_adapter_list_lock); + + return found ? sva->ops_data : NULL; +} + static int sfc_vdpa_vfio_setup(struct sfc_vdpa_adapter *sva) { diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h index bd3c437..dd3ca80 100644 --- a/drivers/vdpa/sfc/sfc_vdpa.h +++ b/drivers/vdpa/sfc/sfc_vdpa.h @@ -59,6 +59,8 @@ struct sfc_vdpa_adapter { struct sfc_vdpa_adapter * sfc_vdpa_get_adapter_by_dev(struct rte_pci_device *pdev); +struct sfc_vdpa_ops_data * +sfc_vdpa_get_data_by_dev(struct rte_vdpa_device *vdpa_dev); int sfc_vdpa_hw_init(struct sfc_vdpa_adapter *sva); diff --git a/drivers/vdpa/sfc/sfc_vdpa_hw.c b/drivers/vdpa/sfc/sfc_vdpa_hw.c index dfb6d4f..dca7034 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_hw.c +++ b/drivers/vdpa/sfc/sfc_vdpa_hw.c @@ -277,10 +277,20 @@ if (rc != 0) goto fail_estimate_rsrc_limits; + sfc_vdpa_log_init(sva, "init virtio"); + rc = efx_virtio_init(enp); + if (rc != 0) { + sfc_vdpa_err(sva, "virtio init failed: %s", rte_strerror(rc)); + goto fail_virtio_init; + } + sfc_vdpa_log_init(sva, "done"); return 0; +fail_virtio_init: + efx_nic_fini(enp); + fail_estimate_rsrc_limits: fail_nic_reset: efx_nic_unprobe(enp); @@ -309,6 +319,9 @@ sfc_vdpa_log_init(sva, "entry"); + sfc_vdpa_log_init(sva, "virtio fini"); + efx_virtio_fini(enp); + sfc_vdpa_log_init(sva, "unprobe nic"); efx_nic_unprobe(enp); diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.c b/drivers/vdpa/sfc/sfc_vdpa_ops.c index 71696be..5750944 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_ops.c +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.c @@ -3,17 +3,31 @@ * Copyright(c) 2020-2021 Xilinx, Inc. */ +#include #include #include #include #include +#include "efx.h" #include "sfc_vdpa_ops.h" #include "sfc_vdpa.h" -/* Dummy functions for mandatory vDPA ops to pass vDPA device registration. - * In subsequent patches these ops would be implemented. +/* These protocol features are needed to enable notifier ctrl */ +#define SFC_VDPA_PROTOCOL_FEATURES \ + ((1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK) | \ + (1ULL << VHOST_USER_PROTOCOL_F_SLAVE_REQ) | \ + (1ULL << VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD) | \ + (1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER) | \ + (1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD)) + +/* + * Set of features which are enabled by default. + * Protocol feature bit is needed to enable notification notifier ctrl. */ +#define SFC_VDPA_DEFAULT_FEATURES \ + (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) + static int sfc_vdpa_get_queue_num(struct rte_vdpa_device *vdpa_dev, uint32_t *queue_num) { @@ -24,22 +38,67 @@ } static int +sfc_vdpa_get_device_features(struct sfc_vdpa_ops_data *ops_data) +{ + int rc; + uint64_t dev_features; + efx_nic_t *nic; + + nic = sfc_vdpa_adapter_by_dev_handle(ops_data->dev_handle)->nic; + + rc = efx_virtio_get_features(nic, EFX_VIRTIO_DEVICE_TYPE_NET, + &dev_features); + if (rc != 0) { + sfc_vdpa_err(ops_data->dev_handle, + "could not read device feature: %s", + rte_strerror(rc)); + return rc; + } + + ops_data->dev_features = dev_features; + + sfc_vdpa_info(ops_data->dev_handle, + "device supported virtio features : 0x%" PRIx64, + ops_data->dev_features); + + return 0; +} + +static int sfc_vdpa_get_features(struct rte_vdpa_device *vdpa_dev, uint64_t *features) { - RTE_SET_USED(vdpa_dev); - RTE_SET_USED(features); + struct sfc_vdpa_ops_data *ops_data; - return -1; + ops_data = sfc_vdpa_get_data_by_dev(vdpa_dev); + if (ops_data == NULL) + return -1; + + *features = ops_data->drv_features; + + sfc_vdpa_info(ops_data->dev_handle, + "vDPA ops get_feature :: features : 0x%" PRIx64, + *features); + + return 0; } static int sfc_vdpa_get_protocol_features(struct rte_vdpa_device *vdpa_dev, uint64_t *features) { - RTE_SET_USED(vdpa_dev); - RTE_SET_USED(features); + struct sfc_vdpa_ops_data *ops_data; - return -1; + ops_data = sfc_vdpa_get_data_by_dev(vdpa_dev); + if (ops_data == NULL) + return -1; + + *features = SFC_VDPA_PROTOCOL_FEATURES; + + sfc_vdpa_info(ops_data->dev_handle, + "vDPA ops get_protocol_feature :: features : 0x%" PRIx64, + *features); + + return 0; } static int @@ -91,6 +150,7 @@ struct sfc_vdpa_ops_data * { struct sfc_vdpa_ops_data *ops_data; struct rte_pci_device *pci_dev; + int rc; /* Create vDPA ops context */ ops_data = rte_zmalloc("vdpa", sizeof(struct sfc_vdpa_ops_data), 0); @@ -111,10 +171,25 @@ struct sfc_vdpa_ops_data * goto fail_register_device; } + /* Read supported device features */ + sfc_vdpa_log_init(dev_handle, "get device feature"); + rc = sfc_vdpa_get_device_features(ops_data); + if (rc != 0) + goto fail_get_dev_feature; + + /* Driver features are superset of device supported feature + * and any additional features supported by the driver. + */ + ops_data->drv_features = + ops_data->dev_features | SFC_VDPA_DEFAULT_FEATURES; + ops_data->state = SFC_VDPA_STATE_INITIALIZED; return ops_data; +fail_get_dev_feature: + rte_vdpa_unregister_device(ops_data->vdpa_dev); + fail_register_device: rte_free(ops_data); return NULL; diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.h b/drivers/vdpa/sfc/sfc_vdpa_ops.h index baa4293..7b77df0 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_ops.h +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.h @@ -24,6 +24,9 @@ struct sfc_vdpa_ops_data { struct rte_vdpa_device *vdpa_dev; enum sfc_vdpa_context vdpa_context; enum sfc_vdpa_state state; + + uint64_t dev_features; + uint64_t drv_features; }; struct sfc_vdpa_ops_data * From patchwork Wed Nov 3 13:57:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijay Srivastava X-Patchwork-Id: 103662 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 67E47A0C53; Wed, 3 Nov 2021 15:01:22 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6E67142740; Wed, 3 Nov 2021 15:00:47 +0100 (CET) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1anam02on2085.outbound.protection.outlook.com [40.107.96.85]) by mails.dpdk.org (Postfix) with ESMTP id 13F1342738 for ; Wed, 3 Nov 2021 15:00:46 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cYgoBZDUgJVs++pzBxx9ussQbZj14UbUZsUt9YW129bvOzKktoTUhWWeipuPFKCKLg6sryOo8qNph0FQ2TbCGR2SHTO/kGKwQMpyyZgTYJmbzobQKpawmQ0w2WUHYQtqmIZTQfMxiogVYvVemAzLqBy10WEv6bFkxG2LIczVtFN+OWeZ7sPzGw0GIBuS+g25k1Xso7wehEFzzAkNzrzCmVRLcWoq24nSJ1wTyzhCmDfVUGhDOfYglMcwLZd5L3sA2YBjKW5sll13Kub1F3K3CozYNYrD6bq1xW0zYiD5kWJu82J++PI7dqi5Zbbuggk86WdzimSPBT1lhXKcl86vxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KWMl+euQBRVv5ofLOHuGCue2bLL9w4GZq1LV5Czb5t4=; b=HSnL4e34zUescIrDq7v3FSsO6Wi2edwaj9O+z6hjEAh+m1q4BpFXAKwfSs+xnof5w4+GYL20Y3ivK9O0ah/TQkyijMIm03nvS0zIAdckkT+7cMnMy47imSASYiJ132NqC5on0pKCjGt/fwk6+Ybqorl6saUEhGdE1NozbE6lY2q+2DeaJwnmp73Gp3MrihJdCnddgc0lsNE+vxxqRnszexk2hsq61P+Q4Pd3qCmWUKBG9K6awZ3U7B1UNJvN/s4sBaUZZBnxXgjdMfbOSwpBon+pgKXRHS1+3Q1w/A1A3GFR7pLuUdp4VN5ke7v2l+dE50YOVYxGAgoTxUKmKhFr+Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=dpdk.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KWMl+euQBRVv5ofLOHuGCue2bLL9w4GZq1LV5Czb5t4=; b=RBJ/hdRVdyzAeqrUV7YWdAk6OFxiaQ9wHAw99M5YF1HQCptxkqwmvYORVnJj/FejA76heLHSEgjkFtt/hFJWzGZp0+JBWJAjiVcLoIgfbwRf/J2Eza2gaVlvfqodn+JAxJnCIpHF2q1dh/FQlw8DzKb7igMGBxZ6wCMkscUoIAg= Received: from SA9PR13CA0004.namprd13.prod.outlook.com (2603:10b6:806:21::9) by SJ0PR02MB7438.namprd02.prod.outlook.com (2603:10b6:a03:294::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.15; Wed, 3 Nov 2021 14:00:43 +0000 Received: from SN1NAM02FT0008.eop-nam02.prod.protection.outlook.com (2603:10b6:806:21:cafe::5c) by SA9PR13CA0004.outlook.office365.com (2603:10b6:806:21::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.4 via Frontend Transport; Wed, 3 Nov 2021 14:00:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by SN1NAM02FT0008.mail.protection.outlook.com (10.97.5.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4649.14 via Frontend Transport; Wed, 3 Nov 2021 14:00:43 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Wed, 3 Nov 2021 07:00:42 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Wed, 3 Nov 2021 07:00:42 -0700 Envelope-to: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, andrew.rybchenko@oktetlabs.ru Received: from [10.170.66.108] (port=38886 helo=xndengvm004108.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1miGoj-0003SS-Sa; Wed, 03 Nov 2021 07:00:42 -0700 From: Vijay Srivastava To: CC: , , , Vijay Kumar Srivastava Date: Wed, 3 Nov 2021 19:27:48 +0530 Message-ID: <20211103135754.17411-5-vsrivast@xilinx.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20211103135754.17411-1-vsrivast@xilinx.com> References: <20210706164418.32615-1-vsrivast@xilinx.com> <20211103135754.17411-1-vsrivast@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cd95a57d-0a7d-46bb-f057-08d99ed253f5 X-MS-TrafficTypeDiagnostic: SJ0PR02MB7438: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2582; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wKUrVAL6Or75xXkB5XOUc4yeYKvpf5vusPCxxKGvGjGTPGOtJyqToa4n/za90fMCoWvliW/JqTyEl02uB5dhAEWSyqi2g2MHKaCcJpBTNrFTwYnstcFZHvZIuREPSl0jvgnsstunJlwsyGpKMXGKc+owklx96ivpx5msZuNitzZut5+iQE8kKIsYZHQN8zJN1bTpGr2/D39lyUwihzabkQdDlFvHf438tmfNae8CWJzV/6YDDp7nLcJ0K+27vZKF4sO2AE0AtNLScGNmpcfieBBHe8GRAp+7WNzzc5FZ24vLED3anRYbQTZn5trpfRiXLqeAY/7r6wIZcq1aNJE+TMfHNNPKKsF28JLOPdbuumMyoH5nHFqV5NFBlAHHTCnKquLw49u4U/UDRjmLCtZuabyXJCx2Kz/Aia4eL6wmKt+SAxLmV/oLh2ONQCwXmHh6m/4NNSNy20ZNf/ZdSrkuXR7G0gBcowk1JTpsDcwP5rlHnqd56CIZaztOiTLqS3m12eH+bilyNyaIzSFA/0L5aJtNkF1yS58MNpnlZB6AEs9dcbMLyPcGvxyUe5o6rd8U6PavHx0yuOxb4uBDHhchxXt6UpGC7XtA7u3I7CFD/w5ommefOqGqpIl+7f7NO8bF+Qvx3MAyxLePRC+bv3rt2MCJTOKU6HwzYghQ2hSaEebo0cK03QLSDfI+qALqvov/82fsTEQwW5FilDNSlXw4j3epV8En/HTJpC7WItEds0M= X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(46966006)(36840700001)(8936002)(7636003)(186003)(36906005)(82310400003)(8676002)(4326008)(26005)(7696005)(336012)(356005)(107886003)(316002)(54906003)(9786002)(70586007)(2906002)(44832011)(47076005)(426003)(2616005)(36756003)(508600001)(5660300002)(1076003)(6916009)(36860700001)(83380400001)(70206006)(102446001); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 14:00:43.7773 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cd95a57d-0a7d-46bb-f057-08d99ed253f5 X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: SN1NAM02FT0008.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR02MB7438 Subject: [dpdk-dev] [PATCH v4 04/10] vdpa/sfc: get device supported max queue count X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vijay Kumar Srivastava Implement vDPA ops get_queue_num to get the maximum number of queues supported by the device. Signed-off-by: Vijay Kumar Srivastava Acked-by: Andrew Rybchenko Reviewed-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- v4: * Updated format specifier in log message. drivers/vdpa/sfc/sfc_vdpa_ops.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.c b/drivers/vdpa/sfc/sfc_vdpa_ops.c index 5750944..9a95211 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_ops.c +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.c @@ -31,10 +31,20 @@ static int sfc_vdpa_get_queue_num(struct rte_vdpa_device *vdpa_dev, uint32_t *queue_num) { - RTE_SET_USED(vdpa_dev); - RTE_SET_USED(queue_num); + struct sfc_vdpa_ops_data *ops_data; + void *dev; - return -1; + ops_data = sfc_vdpa_get_data_by_dev(vdpa_dev); + if (ops_data == NULL) + return -1; + + dev = ops_data->dev_handle; + *queue_num = sfc_vdpa_adapter_by_dev_handle(dev)->max_queue_count; + + sfc_vdpa_info(dev, "vDPA ops get_queue_num :: supported queue num : %u", + *queue_num); + + return 0; } static int From patchwork Wed Nov 3 13:57:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijay Srivastava X-Patchwork-Id: 103663 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BD1BAA0C53; Wed, 3 Nov 2021 15:01:30 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7394642744; Wed, 3 Nov 2021 15:00:49 +0100 (CET) Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam08on2088.outbound.protection.outlook.com [40.107.101.88]) by mails.dpdk.org (Postfix) with ESMTP id 09EB942744 for ; Wed, 3 Nov 2021 15:00:48 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bXf32hxengujLO9HAX3+ldfYNOub90Yvcpb0HraFz6u9u+CnUb5RUEyYXxJtTJmOckPkXW3YIMjBS6RJSDNg2fGFtAVLMG4r2qkLO7q/+2CE7+E0Q01MT2xoony+Zt21tkKyzJDN/znrJ+xXMzJfDXfLaJ1SxWvKTioS9v4TYsKHjTJfVBfyPZFpUr2I7NK/DnqFMIGxZYW9UK7QXr3hJg9ZEPhRiGo5eJcoX2wt8WDJOoXSoCe4nj88fiL+nWGCiQfDzk3fRV6tlFELis1a4hVpRn1LpIceFx+FO3EUV8PX1pQBstKuSAd5a9cDePoSo/uj6WIYdUo12EBVH6Luuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cTpYg7UFlLGhD5kdyYDCtR9nVyNzCMJeFWQQ31LySpQ=; b=VLgBMKCCJvkfIZe44SluuOSsvg8G8psYNUhRhCGdu47KzKbTHxStfQZDUab3EDDCSRC2Xl3eQsc+jnpN5sVB68pVmDo5nOq7XWJOxaDJ1G/9uK3+qCMpasZtv4Uybo7Q+BFGQfXIRohKcFQyJ6Wl/IEZ8t65fVdlWmoP5ox5P9/MdAB4wUnC8mqX8kw8+/YDLEHPo3H3INNdcLX0lQ0BohwDeN+4eVmbQm/mCjyznDNJeRkd/ioIwBYsNZgnr8N/fn+ghxSADXwZEIRWMYrsH0Knu3QMbf4idvjbOpB+Bb27crEBXtT7s4Uw6jhLI4eewQQjCLTnrVk0zxdX0CJRsg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=dpdk.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cTpYg7UFlLGhD5kdyYDCtR9nVyNzCMJeFWQQ31LySpQ=; b=rDC7cZ6U8+lAH8fmGt7ixlQ+JqDAiredipeiCGQJzZjl/XgvNi0c6uL8WR+Sd+vSi7WlXwjZrTqYwyt2QFwQQk+vQiPKEVN943gmwHcE/Fg86sdTQEEY3Wbmb91fJqMhCn2I24UWlgGI4GAolu3spea2thflz0+R+l/ayQsNrsE= Received: from SA9PR13CA0029.namprd13.prod.outlook.com (2603:10b6:806:21::34) by PH0PR02MB7416.namprd02.prod.outlook.com (2603:10b6:510:18::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.14; Wed, 3 Nov 2021 14:00:45 +0000 Received: from SN1NAM02FT0008.eop-nam02.prod.protection.outlook.com (2603:10b6:806:21:cafe::1) by SA9PR13CA0029.outlook.office365.com (2603:10b6:806:21::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.4 via Frontend Transport; Wed, 3 Nov 2021 14:00:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by SN1NAM02FT0008.mail.protection.outlook.com (10.97.5.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4649.14 via Frontend Transport; Wed, 3 Nov 2021 14:00:45 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Wed, 3 Nov 2021 07:00:44 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Wed, 3 Nov 2021 07:00:44 -0700 Envelope-to: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, andrew.rybchenko@oktetlabs.ru Received: from [10.170.66.108] (port=38886 helo=xndengvm004108.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1miGom-0003SS-43; Wed, 03 Nov 2021 07:00:44 -0700 From: Vijay Srivastava To: CC: , , , Vijay Kumar Srivastava Date: Wed, 3 Nov 2021 19:27:49 +0530 Message-ID: <20211103135754.17411-6-vsrivast@xilinx.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20211103135754.17411-1-vsrivast@xilinx.com> References: <20210706164418.32615-1-vsrivast@xilinx.com> <20211103135754.17411-1-vsrivast@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 321ea52f-4920-4a95-8c1e-08d99ed25509 X-MS-TrafficTypeDiagnostic: PH0PR02MB7416: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1051; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Jl4NIjORIzJdeTjQG4Xk1lrUhMvZ24hXLrzOEuLaF5E3QTgXg/GFzry0qeCbdscCrQB/PSOt6Zi9XkFgMFl/lBInnjlPZGT2fI10DxEeOgA5uqvpxpNV8zWcndKYPUDFeISsQblUH6+K9JEq6B0FqBZFVXtYfT9L/UYY8+xYAo9Thrr2M3bqnjynbxm2HoTVVEnJ1KiXzDwTcRmVJ1Lne9kxi6TetiHGuyrLmTN8ND0wjYv/+QL/6G10dkA8+w7yxN1KUT6pvMdgRs/Lk3r/hTbCG4xrxvVKGVivIZ+qdOXv7AQt3KFPwlIjCNekE50u9Jnw2uvsHfogauUhP1afcw5QsC9dMbUNNDdn6vYVNrE/bDYBBGBAYrWNBZ8acOZR5UFZGW+i8Yu84uaF3myPUgSJ0pHVMcsy8VRv+U7N0dhSASjUZSuP1hCiroJiaHZEwqf/0GVUi5k88gKbf9+QDaMkQPYTm/gI6a/TIC5kj+uMmYYifaFSazDMyO2qUH/X1lXSMHEOHjfBtVnbrS+sfKC0ef8bv4TIICxtQHmH9sH2taRrKk/JQAlcpTf4JAKkPoROk0g7Vmq8nf0bue6u8Sd+AO5Ah3pVlIU0cmdB66qr8lSBaZLsAHfiJ4o2DbhjROLSuJdcnn20aXWDlJGWUM1cwrO+aRZw2i2MbrIiUBR2IDDhk+yOOAyGrHqjCd07teFr8VBbUvCqG6Yew3F+6oyu9UypEyA+/2pNN7A2bh4= X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(36840700001)(46966006)(36756003)(7636003)(9786002)(356005)(54906003)(7696005)(107886003)(426003)(82310400003)(36860700001)(508600001)(336012)(83380400001)(1076003)(316002)(6916009)(70206006)(8936002)(36906005)(5660300002)(2616005)(44832011)(186003)(47076005)(70586007)(8676002)(2906002)(26005)(4326008)(102446001); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 14:00:45.5856 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 321ea52f-4920-4a95-8c1e-08d99ed25509 X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: SN1NAM02FT0008.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR02MB7416 Subject: [dpdk-dev] [PATCH v4 05/10] vdpa/sfc: add support to get VFIO device fd X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vijay Kumar Srivastava Implement vDPA ops get_vfio_device_fd to get the VFIO device fd. Signed-off-by: Vijay Kumar Srivastava Acked-by: Andrew Rybchenko Reviewed-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- drivers/vdpa/sfc/sfc_vdpa_ops.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.c b/drivers/vdpa/sfc/sfc_vdpa_ops.c index 9a95211..b5815df 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_ops.c +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.c @@ -145,6 +145,29 @@ return -1; } +static int +sfc_vdpa_get_vfio_device_fd(int vid) +{ + struct rte_vdpa_device *vdpa_dev; + struct sfc_vdpa_ops_data *ops_data; + int vfio_dev_fd; + void *dev; + + vdpa_dev = rte_vhost_get_vdpa_device(vid); + + ops_data = sfc_vdpa_get_data_by_dev(vdpa_dev); + if (ops_data == NULL) + return -1; + + dev = ops_data->dev_handle; + vfio_dev_fd = sfc_vdpa_adapter_by_dev_handle(dev)->vfio_dev_fd; + + sfc_vdpa_info(dev, "vDPA ops get_vfio_device_fd :: vfio fd : %d", + vfio_dev_fd); + + return vfio_dev_fd; +} + static struct rte_vdpa_dev_ops sfc_vdpa_ops = { .get_queue_num = sfc_vdpa_get_queue_num, .get_features = sfc_vdpa_get_features, @@ -153,6 +176,7 @@ .dev_close = sfc_vdpa_dev_close, .set_vring_state = sfc_vdpa_set_vring_state, .set_features = sfc_vdpa_set_features, + .get_vfio_device_fd = sfc_vdpa_get_vfio_device_fd, }; struct sfc_vdpa_ops_data * From patchwork Wed Nov 3 13:57:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijay Srivastava X-Patchwork-Id: 103668 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C6361A0C53; Wed, 3 Nov 2021 15:02:18 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DB93B42721; Wed, 3 Nov 2021 15:01:29 +0100 (CET) Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam08on2080.outbound.protection.outlook.com [40.107.100.80]) by mails.dpdk.org (Postfix) with ESMTP id CC7C542725 for ; Wed, 3 Nov 2021 15:01:27 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FhEJW4oDdikKM1Ri0APFAuSj4O1H+HzaIYF+qPYRIaJYW+6TKaTQOg9oB1jT3YDkedmKKnqjW2qDFjGDJHjY8QWXG4glFO7GEmqBDb/dzuoXqwr0dp5KMMOUDgSNH2jb/S8V53o0t99hdZRPFjSiDb0CONR8Cr9JeMa5YI7baqsJTHo6KA79rUryhka2wXmPXS9ryvYYnX5sbQmeiYgltxk3ebomdBwyQFrNMqXFGmVOBZO1MnQnEWCuwwt5sCH4RJxKd93Wk0OXjvIyaOcfhx85eq2QXfo6qtATQLoPNF+kDJ+Tprezv+nPwX8pBlIlf635FnViuIJQvi8/08elmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KFlbwRPNxzD4/W4VfCvEVn6z0seIP3CB4rB/Ju2GP0M=; b=eSLX46PE0TxF+VybnOqyYCMOCoAl6f+m92YBGpvekZMaRQ/iIawB/aDcEnQbXnhVlP3xit1/6hPKftY/V+ZqPvBDZSc37iVcpkcs5kJ7MkXrrqtdvGYWiWUeAh2RoSAbWlV42b6j6OxYQzVdSXW40HvyZhCoOGUjTKyvlQtTpg9wsLTZRAEGTvQwTB/AbzoKXHcJMRbUqRwUXc4zF8872HiXp3ezYLsAO9EpenyJ8g54nRunlfLoTg1W8HNIPMw9/fJg6z+Rpj/272JABKzx1LJRXCNIXO4R7asvtgqwGUdTpQwN5KieXokptge3O4Dz0BR9zZ6Y8Oo/Lro8yDzPuw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=dpdk.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KFlbwRPNxzD4/W4VfCvEVn6z0seIP3CB4rB/Ju2GP0M=; b=AKaWRNjtcICfRZo1pUhMFGZyo5zAYQK4ilPM+YZXJchQLtJE1N1c+jgKx2nyze2z90TVmjmM8hfHXgaSNZDufwPbA/zGeeVKeAHeI4QqKHbJ0C8acKw+VM/p64gcVz5iYCdFjhqLNSQd1PEHNcKgW68tYxDmxfNYozLjYa8J1Uo= Received: from BN9PR03CA0031.namprd03.prod.outlook.com (2603:10b6:408:fb::6) by MN2PR02MB7102.namprd02.prod.outlook.com (2603:10b6:208:20d::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.11; Wed, 3 Nov 2021 14:01:24 +0000 Received: from BN1NAM02FT062.eop-nam02.prod.protection.outlook.com (2603:10b6:408:fb:cafe::9d) by BN9PR03CA0031.outlook.office365.com (2603:10b6:408:fb::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.17 via Frontend Transport; Wed, 3 Nov 2021 14:01:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch01.xlnx.xilinx.com; Received: from xsj-pvapexch01.xlnx.xilinx.com (149.199.62.198) by BN1NAM02FT062.mail.protection.outlook.com (10.13.2.168) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4669.10 via Frontend Transport; Wed, 3 Nov 2021 14:01:24 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch01.xlnx.xilinx.com (172.19.86.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Wed, 3 Nov 2021 07:00:46 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Wed, 3 Nov 2021 07:00:46 -0700 Envelope-to: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, andrew.rybchenko@oktetlabs.ru Received: from [10.170.66.108] (port=38886 helo=xndengvm004108.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1miGoo-0003SS-Cf; Wed, 03 Nov 2021 07:00:46 -0700 From: Vijay Srivastava To: CC: , , , Vijay Kumar Srivastava Date: Wed, 3 Nov 2021 19:27:50 +0530 Message-ID: <20211103135754.17411-7-vsrivast@xilinx.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20211103135754.17411-1-vsrivast@xilinx.com> References: <20210706164418.32615-1-vsrivast@xilinx.com> <20211103135754.17411-1-vsrivast@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3c093f56-bf3a-48d5-5e88-08d99ed26c48 X-MS-TrafficTypeDiagnostic: MN2PR02MB7102: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:127; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +sFAzmznM/+D9jksgj4BI10h4B8vQ22p94ZS2KSm7I8phP3RFrv6qipo8XdrCHgguwPpBU3zhqLebIF+QM/lrgv575yK7mjlJ2bVmR//MpFGtVeHXET/RFkS417PZiYpuE4p3LKuIH3u+FmwoNIx6a/CjkxdsEmAbjnmTtGBzocM++8q/IpkSz0//mB47TNv+8NM+V5KX1lIKxU6U+D/B4tgr0oFDjurzfXh37bcHLdchffEL0o0HyilEHF/7ETm3UxL+b0dx+KYZCO2/BawVMm1bXQPNsK4f1ylUC66iKUWBPz2WKObjQ8hY+YQXZlTUbpT0M5Nkhy5dVSzOGXo0yxXRsyMHIZa3Wp6lbajROQ0xS3iHRiP1t3BKD5SinBmvKhm0oAKKVj3EydnPCPmOUOB+8EtowwD2C9k+wbyGUB66yFRefbvDP+4xbLpz8mWTrDqz3cs9Iz+dhII9dteFgaZBZ0DyYm0gE4glIvY5lPNsNtRHaoxgoThbMWPP4qnDCkdcCmMheb6ORApAf3cVOym6K1iyL/8zcPAtSHXpdwXXglB4JWiYOLicOJbfT0Glk9sjX48zJnillQeoTMytuokwPx7isBCAtvpQ9OP2wuwS0q3f8qpmKzWg8tr2ldZjwRV6UPqup4TUYDF/87y3EhnFxfDEF4Ju+uJccduBrNuWzuksWEQQhKt0ktbJbAvXId9lolgUMSfqUoo0CpBDNtDJK9yoipxjHEr+CneM74= X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch01.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(36840700001)(46966006)(1076003)(47076005)(30864003)(9786002)(36860700001)(426003)(36756003)(336012)(82310400003)(5660300002)(2616005)(44832011)(2906002)(6916009)(8936002)(508600001)(7696005)(7636003)(26005)(70206006)(186003)(70586007)(8676002)(4326008)(6666004)(54906003)(83380400001)(107886003)(36906005)(356005)(316002)(102446001); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 14:01:24.5229 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3c093f56-bf3a-48d5-5e88-08d99ed26c48 X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch01.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: BN1NAM02FT062.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR02MB7102 Subject: [dpdk-dev] [PATCH v4 06/10] vdpa/sfc: add support for dev conf and dev close ops X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vijay Kumar Srivastava Implement vDPA ops dev_conf and dev_close for DMA mapping, interrupt and virtqueue configurations. Signed-off-by: Vijay Kumar Srivastava Acked-by: Andrew Rybchenko Reviewed-by: Maxime Coquelin --- v2: * Removed redundant null check while calling free(). * Added error handling for rte_vhost_get_vhost_vring(). v4: * Removed rc for rte_vfio_container_dma_unmap from label failed_vfio_dma_map. * Updated VFIO_DEVICE_SET_IRQS parameter fields for intr disable. * Removed goto label fail_vring_map. drivers/vdpa/sfc/sfc_vdpa.c | 6 + drivers/vdpa/sfc/sfc_vdpa.h | 43 ++++ drivers/vdpa/sfc/sfc_vdpa_hw.c | 69 ++++++ drivers/vdpa/sfc/sfc_vdpa_ops.c | 525 ++++++++++++++++++++++++++++++++++++++-- drivers/vdpa/sfc/sfc_vdpa_ops.h | 28 +++ 5 files changed, 651 insertions(+), 20 deletions(-) diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c index 7b9ebcc..f548c1e 100644 --- a/drivers/vdpa/sfc/sfc_vdpa.c +++ b/drivers/vdpa/sfc/sfc_vdpa.c @@ -242,6 +242,8 @@ struct sfc_vdpa_ops_data * sfc_vdpa_log_init(sva, "entry"); + sfc_vdpa_adapter_lock_init(sva); + sfc_vdpa_log_init(sva, "vfio init"); if (sfc_vdpa_vfio_setup(sva) < 0) { sfc_vdpa_err(sva, "failed to setup device %s", pci_dev->name); @@ -276,6 +278,8 @@ struct sfc_vdpa_ops_data * sfc_vdpa_vfio_teardown(sva); fail_vfio_setup: + sfc_vdpa_adapter_lock_fini(sva); + fail_set_log_prefix: rte_free(sva); @@ -307,6 +311,8 @@ struct sfc_vdpa_ops_data * sfc_vdpa_vfio_teardown(sva); + sfc_vdpa_adapter_lock_fini(sva); + rte_free(sva); return 0; diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h index dd3ca80..38fd5de 100644 --- a/drivers/vdpa/sfc/sfc_vdpa.h +++ b/drivers/vdpa/sfc/sfc_vdpa.h @@ -79,10 +79,53 @@ struct sfc_vdpa_ops_data * void sfc_vdpa_dma_free(struct sfc_vdpa_adapter *sva, efsys_mem_t *esmp); +int +sfc_vdpa_dma_map(struct sfc_vdpa_ops_data *vdpa_data, bool do_map); + static inline struct sfc_vdpa_adapter * sfc_vdpa_adapter_by_dev_handle(void *dev_handle) { return (struct sfc_vdpa_adapter *)dev_handle; } +/* + * Add wrapper functions to acquire/release lock to be able to remove or + * change the lock in one place. + */ +static inline void +sfc_vdpa_adapter_lock_init(struct sfc_vdpa_adapter *sva) +{ + rte_spinlock_init(&sva->lock); +} + +static inline int +sfc_vdpa_adapter_is_locked(struct sfc_vdpa_adapter *sva) +{ + return rte_spinlock_is_locked(&sva->lock); +} + +static inline void +sfc_vdpa_adapter_lock(struct sfc_vdpa_adapter *sva) +{ + rte_spinlock_lock(&sva->lock); +} + +static inline int +sfc_vdpa_adapter_trylock(struct sfc_vdpa_adapter *sva) +{ + return rte_spinlock_trylock(&sva->lock); +} + +static inline void +sfc_vdpa_adapter_unlock(struct sfc_vdpa_adapter *sva) +{ + rte_spinlock_unlock(&sva->lock); +} + +static inline void +sfc_vdpa_adapter_lock_fini(__rte_unused struct sfc_vdpa_adapter *sva) +{ + /* Just for symmetry of the API */ +} + #endif /* _SFC_VDPA_H */ diff --git a/drivers/vdpa/sfc/sfc_vdpa_hw.c b/drivers/vdpa/sfc/sfc_vdpa_hw.c index dca7034..d3459ee 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_hw.c +++ b/drivers/vdpa/sfc/sfc_vdpa_hw.c @@ -8,6 +8,7 @@ #include #include #include +#include #include "efx.h" #include "sfc_vdpa.h" @@ -108,6 +109,74 @@ memset(esmp, 0, sizeof(*esmp)); } +int +sfc_vdpa_dma_map(struct sfc_vdpa_ops_data *ops_data, bool do_map) +{ + uint32_t i, j; + int rc; + struct rte_vhost_memory *vhost_mem = NULL; + struct rte_vhost_mem_region *mem_reg = NULL; + int vfio_container_fd; + void *dev; + + dev = ops_data->dev_handle; + vfio_container_fd = + sfc_vdpa_adapter_by_dev_handle(dev)->vfio_container_fd; + + rc = rte_vhost_get_mem_table(ops_data->vid, &vhost_mem); + if (rc < 0) { + sfc_vdpa_err(dev, + "failed to get VM memory layout"); + goto error; + } + + for (i = 0; i < vhost_mem->nregions; i++) { + mem_reg = &vhost_mem->regions[i]; + + if (do_map) { + rc = rte_vfio_container_dma_map(vfio_container_fd, + mem_reg->host_user_addr, + mem_reg->guest_phys_addr, + mem_reg->size); + if (rc < 0) { + sfc_vdpa_err(dev, + "DMA map failed : %s", + rte_strerror(rte_errno)); + goto failed_vfio_dma_map; + } + } else { + rc = rte_vfio_container_dma_unmap(vfio_container_fd, + mem_reg->host_user_addr, + mem_reg->guest_phys_addr, + mem_reg->size); + if (rc < 0) { + sfc_vdpa_err(dev, + "DMA unmap failed : %s", + rte_strerror(rte_errno)); + goto error; + } + } + } + + free(vhost_mem); + + return 0; + +failed_vfio_dma_map: + for (j = 0; j < i; j++) { + mem_reg = &vhost_mem->regions[j]; + rte_vfio_container_dma_unmap(vfio_container_fd, + mem_reg->host_user_addr, + mem_reg->guest_phys_addr, + mem_reg->size); + } + +error: + free(vhost_mem); + + return rc; +} + static int sfc_vdpa_mem_bar_init(struct sfc_vdpa_adapter *sva, const efx_bar_region_t *mem_ebrp) diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.c b/drivers/vdpa/sfc/sfc_vdpa_ops.c index b5815df..f0be652 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_ops.c +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.c @@ -3,10 +3,13 @@ * Copyright(c) 2020-2021 Xilinx, Inc. */ +#include + #include #include #include #include +#include #include #include "efx.h" @@ -28,24 +31,12 @@ #define SFC_VDPA_DEFAULT_FEATURES \ (1ULL << VHOST_USER_F_PROTOCOL_FEATURES) -static int -sfc_vdpa_get_queue_num(struct rte_vdpa_device *vdpa_dev, uint32_t *queue_num) -{ - struct sfc_vdpa_ops_data *ops_data; - void *dev; - - ops_data = sfc_vdpa_get_data_by_dev(vdpa_dev); - if (ops_data == NULL) - return -1; - - dev = ops_data->dev_handle; - *queue_num = sfc_vdpa_adapter_by_dev_handle(dev)->max_queue_count; +#define SFC_VDPA_MSIX_IRQ_SET_BUF_LEN \ + (sizeof(struct vfio_irq_set) + \ + sizeof(int) * (SFC_VDPA_MAX_QUEUE_PAIRS * 2 + 1)) - sfc_vdpa_info(dev, "vDPA ops get_queue_num :: supported queue num : %u", - *queue_num); - - return 0; -} +/* It will be used for target VF when calling function is not PF */ +#define SFC_VDPA_VF_NULL 0xFFFF static int sfc_vdpa_get_device_features(struct sfc_vdpa_ops_data *ops_data) @@ -74,6 +65,436 @@ return 0; } +static uint64_t +hva_to_gpa(int vid, uint64_t hva) +{ + struct rte_vhost_memory *vhost_mem = NULL; + struct rte_vhost_mem_region *mem_reg = NULL; + uint32_t i; + uint64_t gpa = 0; + + if (rte_vhost_get_mem_table(vid, &vhost_mem) < 0) + goto error; + + for (i = 0; i < vhost_mem->nregions; i++) { + mem_reg = &vhost_mem->regions[i]; + + if (hva >= mem_reg->host_user_addr && + hva < mem_reg->host_user_addr + mem_reg->size) { + gpa = (hva - mem_reg->host_user_addr) + + mem_reg->guest_phys_addr; + break; + } + } + +error: + free(vhost_mem); + return gpa; +} + +static int +sfc_vdpa_enable_vfio_intr(struct sfc_vdpa_ops_data *ops_data) +{ + int rc; + int *irq_fd_ptr; + int vfio_dev_fd; + uint32_t i, num_vring; + struct rte_vhost_vring vring; + struct vfio_irq_set *irq_set; + struct rte_pci_device *pci_dev; + char irq_set_buf[SFC_VDPA_MSIX_IRQ_SET_BUF_LEN]; + void *dev; + + num_vring = rte_vhost_get_vring_num(ops_data->vid); + dev = ops_data->dev_handle; + vfio_dev_fd = sfc_vdpa_adapter_by_dev_handle(dev)->vfio_dev_fd; + pci_dev = sfc_vdpa_adapter_by_dev_handle(dev)->pdev; + + irq_set = (struct vfio_irq_set *)irq_set_buf; + irq_set->argsz = sizeof(irq_set_buf); + irq_set->count = num_vring + 1; + irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | + VFIO_IRQ_SET_ACTION_TRIGGER; + irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; + irq_set->start = 0; + irq_fd_ptr = (int *)&irq_set->data; + irq_fd_ptr[RTE_INTR_VEC_ZERO_OFFSET] = + rte_intr_fd_get(pci_dev->intr_handle); + + for (i = 0; i < num_vring; i++) { + rc = rte_vhost_get_vhost_vring(ops_data->vid, i, &vring); + if (rc) + return -1; + + irq_fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd; + } + + rc = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set); + if (rc) { + sfc_vdpa_err(ops_data->dev_handle, + "error enabling MSI-X interrupts: %s", + strerror(errno)); + return -1; + } + + return 0; +} + +static int +sfc_vdpa_disable_vfio_intr(struct sfc_vdpa_ops_data *ops_data) +{ + int rc; + int vfio_dev_fd; + struct vfio_irq_set irq_set; + void *dev; + + dev = ops_data->dev_handle; + vfio_dev_fd = sfc_vdpa_adapter_by_dev_handle(dev)->vfio_dev_fd; + + irq_set.argsz = sizeof(irq_set); + irq_set.count = 0; + irq_set.flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_TRIGGER; + irq_set.index = VFIO_PCI_MSIX_IRQ_INDEX; + irq_set.start = 0; + + rc = ioctl(vfio_dev_fd, VFIO_DEVICE_SET_IRQS, &irq_set); + if (rc) { + sfc_vdpa_err(ops_data->dev_handle, + "error disabling MSI-X interrupts: %s", + strerror(errno)); + return -1; + } + + return 0; +} + +static int +sfc_vdpa_get_vring_info(struct sfc_vdpa_ops_data *ops_data, + int vq_num, struct sfc_vdpa_vring_info *vring) +{ + int rc; + uint64_t gpa; + struct rte_vhost_vring vq; + + rc = rte_vhost_get_vhost_vring(ops_data->vid, vq_num, &vq); + if (rc < 0) { + sfc_vdpa_err(ops_data->dev_handle, + "get vhost vring failed: %s", rte_strerror(rc)); + return rc; + } + + gpa = hva_to_gpa(ops_data->vid, (uint64_t)(uintptr_t)vq.desc); + if (gpa == 0) { + sfc_vdpa_err(ops_data->dev_handle, + "fail to get GPA for descriptor ring."); + return -1; + } + vring->desc = gpa; + + gpa = hva_to_gpa(ops_data->vid, (uint64_t)(uintptr_t)vq.avail); + if (gpa == 0) { + sfc_vdpa_err(ops_data->dev_handle, + "fail to get GPA for available ring."); + return -1; + } + vring->avail = gpa; + + gpa = hva_to_gpa(ops_data->vid, (uint64_t)(uintptr_t)vq.used); + if (gpa == 0) { + sfc_vdpa_err(ops_data->dev_handle, + "fail to get GPA for used ring."); + return -1; + } + vring->used = gpa; + + vring->size = vq.size; + + rc = rte_vhost_get_vring_base(ops_data->vid, vq_num, + &vring->last_avail_idx, + &vring->last_used_idx); + + return rc; +} + +static int +sfc_vdpa_virtq_start(struct sfc_vdpa_ops_data *ops_data, int vq_num) +{ + int rc; + efx_virtio_vq_t *vq; + struct sfc_vdpa_vring_info vring; + efx_virtio_vq_cfg_t vq_cfg; + efx_virtio_vq_dyncfg_t vq_dyncfg; + + vq = ops_data->vq_cxt[vq_num].vq; + if (vq == NULL) + return -1; + + rc = sfc_vdpa_get_vring_info(ops_data, vq_num, &vring); + if (rc < 0) { + sfc_vdpa_err(ops_data->dev_handle, + "get vring info failed: %s", rte_strerror(rc)); + goto fail_vring_info; + } + + vq_cfg.evvc_target_vf = SFC_VDPA_VF_NULL; + + /* even virtqueue for RX and odd for TX */ + if (vq_num % 2) { + vq_cfg.evvc_type = EFX_VIRTIO_VQ_TYPE_NET_TXQ; + sfc_vdpa_info(ops_data->dev_handle, + "configure virtqueue # %d (TXQ)", vq_num); + } else { + vq_cfg.evvc_type = EFX_VIRTIO_VQ_TYPE_NET_RXQ; + sfc_vdpa_info(ops_data->dev_handle, + "configure virtqueue # %d (RXQ)", vq_num); + } + + vq_cfg.evvc_vq_num = vq_num; + vq_cfg.evvc_desc_tbl_addr = vring.desc; + vq_cfg.evvc_avail_ring_addr = vring.avail; + vq_cfg.evvc_used_ring_addr = vring.used; + vq_cfg.evvc_vq_size = vring.size; + + vq_dyncfg.evvd_vq_pidx = vring.last_used_idx; + vq_dyncfg.evvd_vq_cidx = vring.last_avail_idx; + + /* MSI-X vector is function-relative */ + vq_cfg.evvc_msix_vector = RTE_INTR_VEC_RXTX_OFFSET + vq_num; + if (ops_data->vdpa_context == SFC_VDPA_AS_VF) + vq_cfg.evvc_pas_id = 0; + vq_cfg.evcc_features = ops_data->dev_features & + ops_data->req_features; + + /* Start virtqueue */ + rc = efx_virtio_qstart(vq, &vq_cfg, &vq_dyncfg); + if (rc != 0) { + /* destroy virtqueue */ + sfc_vdpa_err(ops_data->dev_handle, + "virtqueue start failed: %s", + rte_strerror(rc)); + efx_virtio_qdestroy(vq); + goto fail_virtio_qstart; + } + + sfc_vdpa_info(ops_data->dev_handle, + "virtqueue started successfully for vq_num %d", vq_num); + + ops_data->vq_cxt[vq_num].enable = B_TRUE; + + return rc; + +fail_virtio_qstart: +fail_vring_info: + return rc; +} + +static int +sfc_vdpa_virtq_stop(struct sfc_vdpa_ops_data *ops_data, int vq_num) +{ + int rc; + efx_virtio_vq_dyncfg_t vq_idx; + efx_virtio_vq_t *vq; + + if (ops_data->vq_cxt[vq_num].enable != B_TRUE) + return -1; + + vq = ops_data->vq_cxt[vq_num].vq; + if (vq == NULL) + return -1; + + /* stop the vq */ + rc = efx_virtio_qstop(vq, &vq_idx); + if (rc == 0) { + ops_data->vq_cxt[vq_num].cidx = vq_idx.evvd_vq_cidx; + ops_data->vq_cxt[vq_num].pidx = vq_idx.evvd_vq_pidx; + } + ops_data->vq_cxt[vq_num].enable = B_FALSE; + + return rc; +} + +static int +sfc_vdpa_configure(struct sfc_vdpa_ops_data *ops_data) +{ + int rc, i; + int nr_vring; + int max_vring_cnt; + efx_virtio_vq_t *vq; + efx_nic_t *nic; + void *dev; + + dev = ops_data->dev_handle; + nic = sfc_vdpa_adapter_by_dev_handle(dev)->nic; + + SFC_EFX_ASSERT(ops_data->state == SFC_VDPA_STATE_INITIALIZED); + + ops_data->state = SFC_VDPA_STATE_CONFIGURING; + + nr_vring = rte_vhost_get_vring_num(ops_data->vid); + max_vring_cnt = + (sfc_vdpa_adapter_by_dev_handle(dev)->max_queue_count * 2); + + /* number of vring should not be more than supported max vq count */ + if (nr_vring > max_vring_cnt) { + sfc_vdpa_err(dev, + "nr_vring (%d) is > max vring count (%d)", + nr_vring, max_vring_cnt); + goto fail_vring_num; + } + + rc = sfc_vdpa_dma_map(ops_data, true); + if (rc) { + sfc_vdpa_err(dev, + "DMA map failed: %s", rte_strerror(rc)); + goto fail_dma_map; + } + + for (i = 0; i < nr_vring; i++) { + rc = efx_virtio_qcreate(nic, &vq); + if ((rc != 0) || (vq == NULL)) { + sfc_vdpa_err(dev, + "virtqueue create failed: %s", + rte_strerror(rc)); + goto fail_vq_create; + } + + /* store created virtqueue context */ + ops_data->vq_cxt[i].vq = vq; + } + + ops_data->vq_count = i; + + ops_data->state = SFC_VDPA_STATE_CONFIGURED; + + return 0; + +fail_vq_create: + sfc_vdpa_dma_map(ops_data, false); + +fail_dma_map: +fail_vring_num: + ops_data->state = SFC_VDPA_STATE_INITIALIZED; + + return -1; +} + +static void +sfc_vdpa_close(struct sfc_vdpa_ops_data *ops_data) +{ + int i; + + if (ops_data->state != SFC_VDPA_STATE_CONFIGURED) + return; + + ops_data->state = SFC_VDPA_STATE_CLOSING; + + for (i = 0; i < ops_data->vq_count; i++) { + if (ops_data->vq_cxt[i].vq == NULL) + continue; + + efx_virtio_qdestroy(ops_data->vq_cxt[i].vq); + } + + sfc_vdpa_dma_map(ops_data, false); + + ops_data->state = SFC_VDPA_STATE_INITIALIZED; +} + +static void +sfc_vdpa_stop(struct sfc_vdpa_ops_data *ops_data) +{ + int i; + int rc; + + if (ops_data->state != SFC_VDPA_STATE_STARTED) + return; + + ops_data->state = SFC_VDPA_STATE_STOPPING; + + for (i = 0; i < ops_data->vq_count; i++) { + rc = sfc_vdpa_virtq_stop(ops_data, i); + if (rc != 0) + continue; + } + + sfc_vdpa_disable_vfio_intr(ops_data); + + ops_data->state = SFC_VDPA_STATE_CONFIGURED; +} + +static int +sfc_vdpa_start(struct sfc_vdpa_ops_data *ops_data) +{ + int i, j; + int rc; + + SFC_EFX_ASSERT(ops_data->state == SFC_VDPA_STATE_CONFIGURED); + + sfc_vdpa_log_init(ops_data->dev_handle, "entry"); + + ops_data->state = SFC_VDPA_STATE_STARTING; + + sfc_vdpa_log_init(ops_data->dev_handle, "enable interrupts"); + rc = sfc_vdpa_enable_vfio_intr(ops_data); + if (rc < 0) { + sfc_vdpa_err(ops_data->dev_handle, + "vfio intr allocation failed: %s", + rte_strerror(rc)); + goto fail_enable_vfio_intr; + } + + rte_vhost_get_negotiated_features(ops_data->vid, + &ops_data->req_features); + + sfc_vdpa_info(ops_data->dev_handle, + "negotiated feature : 0x%" PRIx64, + ops_data->req_features); + + for (i = 0; i < ops_data->vq_count; i++) { + sfc_vdpa_log_init(ops_data->dev_handle, + "starting vq# %d", i); + rc = sfc_vdpa_virtq_start(ops_data, i); + if (rc != 0) + goto fail_vq_start; + } + + ops_data->state = SFC_VDPA_STATE_STARTED; + + sfc_vdpa_log_init(ops_data->dev_handle, "done"); + + return 0; + +fail_vq_start: + /* stop already started virtqueues */ + for (j = 0; j < i; j++) + sfc_vdpa_virtq_stop(ops_data, j); + sfc_vdpa_disable_vfio_intr(ops_data); + +fail_enable_vfio_intr: + ops_data->state = SFC_VDPA_STATE_CONFIGURED; + + return rc; +} + +static int +sfc_vdpa_get_queue_num(struct rte_vdpa_device *vdpa_dev, uint32_t *queue_num) +{ + struct sfc_vdpa_ops_data *ops_data; + void *dev; + + ops_data = sfc_vdpa_get_data_by_dev(vdpa_dev); + if (ops_data == NULL) + return -1; + + dev = ops_data->dev_handle; + *queue_num = sfc_vdpa_adapter_by_dev_handle(dev)->max_queue_count; + + sfc_vdpa_info(dev, "vDPA ops get_queue_num :: supported queue num : %u", + *queue_num); + + return 0; +} + static int sfc_vdpa_get_features(struct rte_vdpa_device *vdpa_dev, uint64_t *features) { @@ -114,7 +535,53 @@ static int sfc_vdpa_dev_config(int vid) { - RTE_SET_USED(vid); + struct rte_vdpa_device *vdpa_dev; + int rc; + struct sfc_vdpa_ops_data *ops_data; + + vdpa_dev = rte_vhost_get_vdpa_device(vid); + + ops_data = sfc_vdpa_get_data_by_dev(vdpa_dev); + if (ops_data == NULL) { + sfc_vdpa_err(ops_data->dev_handle, + "invalid vDPA device : %p, vid : %d", + vdpa_dev, vid); + return -1; + } + + sfc_vdpa_log_init(ops_data->dev_handle, "entry"); + + ops_data->vid = vid; + + sfc_vdpa_adapter_lock(ops_data->dev_handle); + + sfc_vdpa_log_init(ops_data->dev_handle, "configuring"); + rc = sfc_vdpa_configure(ops_data); + if (rc != 0) + goto fail_vdpa_config; + + sfc_vdpa_log_init(ops_data->dev_handle, "starting"); + rc = sfc_vdpa_start(ops_data); + if (rc != 0) + goto fail_vdpa_start; + + sfc_vdpa_adapter_unlock(ops_data->dev_handle); + + sfc_vdpa_log_init(ops_data->dev_handle, "vhost notifier ctrl"); + if (rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, true) != 0) + sfc_vdpa_info(ops_data->dev_handle, + "vDPA (%s): software relay for notify is used.", + vdpa_dev->device->name); + + sfc_vdpa_log_init(ops_data->dev_handle, "done"); + + return 0; + +fail_vdpa_start: + sfc_vdpa_close(ops_data); + +fail_vdpa_config: + sfc_vdpa_adapter_unlock(ops_data->dev_handle); return -1; } @@ -122,9 +589,27 @@ static int sfc_vdpa_dev_close(int vid) { - RTE_SET_USED(vid); + struct rte_vdpa_device *vdpa_dev; + struct sfc_vdpa_ops_data *ops_data; - return -1; + vdpa_dev = rte_vhost_get_vdpa_device(vid); + + ops_data = sfc_vdpa_get_data_by_dev(vdpa_dev); + if (ops_data == NULL) { + sfc_vdpa_err(ops_data->dev_handle, + "invalid vDPA device : %p, vid : %d", + vdpa_dev, vid); + return -1; + } + + sfc_vdpa_adapter_lock(ops_data->dev_handle); + + sfc_vdpa_stop(ops_data); + sfc_vdpa_close(ops_data); + + sfc_vdpa_adapter_unlock(ops_data->dev_handle); + + return 0; } static int diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.h b/drivers/vdpa/sfc/sfc_vdpa_ops.h index 7b77df0..d20d219 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_ops.h +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.h @@ -17,16 +17,44 @@ enum sfc_vdpa_context { enum sfc_vdpa_state { SFC_VDPA_STATE_UNINITIALIZED = 0, SFC_VDPA_STATE_INITIALIZED, + SFC_VDPA_STATE_CONFIGURING, + SFC_VDPA_STATE_CONFIGURED, + SFC_VDPA_STATE_CLOSING, + SFC_VDPA_STATE_CLOSED, + SFC_VDPA_STATE_STARTING, + SFC_VDPA_STATE_STARTED, + SFC_VDPA_STATE_STOPPING, }; +struct sfc_vdpa_vring_info { + uint64_t desc; + uint64_t avail; + uint64_t used; + uint64_t size; + uint16_t last_avail_idx; + uint16_t last_used_idx; +}; + +typedef struct sfc_vdpa_vq_context_s { + uint8_t enable; + uint32_t pidx; + uint32_t cidx; + efx_virtio_vq_t *vq; +} sfc_vdpa_vq_context_t; + struct sfc_vdpa_ops_data { void *dev_handle; + int vid; struct rte_vdpa_device *vdpa_dev; enum sfc_vdpa_context vdpa_context; enum sfc_vdpa_state state; uint64_t dev_features; uint64_t drv_features; + uint64_t req_features; + + uint16_t vq_count; + struct sfc_vdpa_vq_context_s vq_cxt[SFC_VDPA_MAX_QUEUE_PAIRS * 2]; }; struct sfc_vdpa_ops_data * From patchwork Wed Nov 3 13:57:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijay Srivastava X-Patchwork-Id: 103664 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 89046A0C53; Wed, 3 Nov 2021 15:01:39 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B037E4271E; Wed, 3 Nov 2021 15:01:18 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2064.outbound.protection.outlook.com [40.107.244.64]) by mails.dpdk.org (Postfix) with ESMTP id AB7F241134 for ; Wed, 3 Nov 2021 15:01:16 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OcCknOpHK4L5fPv0giwdkl2wcnC89iNGTSseEvcAWjpHWMMQQTbhnLIspVZYBIx8JjP8V2cNJOLs2OB2MfGByDje0LNwP2jYAOT7A/ESrmuaoTmgu+BbxNYqbDP7fSs/ucDd80XyQr7zi5BJw/T6P+3pmt5Pe8qK5ZC4B6wGtBwCJgpoNkap3btoKDKbeLBVwpRnVX9105kM6dyvETUOgHMwHl57lIjjdSzn2GdPylY7vmbpS1CwR9vRtFvtD7Y+ATmJHA1YvKAsE+btLFe7JSUs81bnASTPRTUn/HFDWpu+kR5jN/OL2dEj6EEdLM3Popkwk5bRc6LQgpVTxWWNWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VoxnIBD/xcwXi8MpULEaPQ/0PNGZNqApOqvN4SVtOVg=; b=Taq7YIF1ObSBRqmbbLkyL2v1d6qD+eYrCUylZpWeufCh6NKz1UiF44hGfC4KA3BXOgLZDMCHuh/1Tl0qO+oyuRD7taGWvvyGTWHwUjj4ZTgYffMs/fOfeeRQOitlNB7ayGNf636y/v/ab8o6EW5pviMEISE5Rpn78OEBmTQ68kWCfkhj+Q3cff7kVozMPzPJ5Wode5Pm7+tcXdv/nFLKkrRSL2VD/FkA1J/ahgS1FZ0djpGO4g9HCcmXLZ2lDTA7pXIbnuXZawYOj52tqWwusAC3Zgrh5cDC9UYsS1z3dYD0j1CXfabzUBIzPXqFI06Nfb73gZEz4pBM6rBWZGPVvw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=dpdk.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VoxnIBD/xcwXi8MpULEaPQ/0PNGZNqApOqvN4SVtOVg=; b=K32tyigaDXJCPoQ8RFVAVpnoxVRpFR9+y40XivnEbvyGDM5+3l+bJrb77XI5TwZlS5bHrO0Ai09AhTiq/P0jtP4sEnIEIp8N9cb5Wub0SmPmd6dhcrA1m0xKcBLe1gI9/SOOMu1g7NohYXrbF8MwhzOv6i8IA/iWYLuUDGxcNfQ= Received: from SA9PR10CA0004.namprd10.prod.outlook.com (2603:10b6:806:a7::9) by BN7PR02MB4148.namprd02.prod.outlook.com (2603:10b6:406:f6::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.20; Wed, 3 Nov 2021 14:01:11 +0000 Received: from SN1NAM02FT0055.eop-nam02.prod.protection.outlook.com (2603:10b6:806:a7:cafe::e8) by SA9PR10CA0004.outlook.office365.com (2603:10b6:806:a7::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.10 via Frontend Transport; Wed, 3 Nov 2021 14:01:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by SN1NAM02FT0055.mail.protection.outlook.com (10.97.5.78) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4669.10 via Frontend Transport; Wed, 3 Nov 2021 14:01:11 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Wed, 3 Nov 2021 07:00:49 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Wed, 3 Nov 2021 07:00:49 -0700 Envelope-to: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, andrew.rybchenko@oktetlabs.ru Received: from [10.170.66.108] (port=38886 helo=xndengvm004108.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1miGoq-0003SS-KM; Wed, 03 Nov 2021 07:00:49 -0700 From: Vijay Srivastava To: CC: , , , Vijay Kumar Srivastava Date: Wed, 3 Nov 2021 19:27:51 +0530 Message-ID: <20211103135754.17411-8-vsrivast@xilinx.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20211103135754.17411-1-vsrivast@xilinx.com> References: <20210706164418.32615-1-vsrivast@xilinx.com> <20211103135754.17411-1-vsrivast@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 64167ccc-0d3c-4c8f-68da-08d99ed2646b X-MS-TrafficTypeDiagnostic: BN7PR02MB4148: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3044; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: eB9wnxxdWGbm5IYcI8ikGylNE7fldhUwR+leSDL6vrWMQktKW1YSn/Olqx17X3MJ1Xv+dFtZpADbzmDt3gXnnrADpGClba5qdY0po/L1Zptx3efnRBe+Br26tJxPRiX1RsInB2XjQ+0TBRMvkwksOwowDeY33v/5RHZ6Omm/BbQ8ddsgp4VW9hlwiycYC98L/7dgvsM70QOFTtKL0uO4tOE5hmhH7LGoPw0Xks9VDvWvWHaBKM7q+RMKNiJR9CrpLyUpLmAFveagmaHjiVy1mgFVo0buJzw1woChNehLV9Cj4THyZSKAGwhJOIGH4mFqS6ZGQYS8Uyy2Pbz2UbwcdRQ7B226Aau/jUfX7zjNWTIL8bEKmG/36iVzrjpdwt16L87CMqEBA3USh92vpq8tcpstXrrthK0sQJdkLEE/+R4D/ednDhyv1TS1UmAdncmCmZj9KWvBYwfZB5T0yqA1F2FfW/UGU/B55iHdXMMfLHE2JwXoeCi8v0b0cWCSbNV2o/2kmqxKC4awfR4YgyI3kmxYYqR+J6aUOSoOELt2Mg/31csAGcdX52wmq7UxywmvQsjztLN+h7m5PPMDvn1LDQY0bHNcEc6OgIIQDjYaKHRKBD/AL8UvN9SU9CsYrv+a6iMG6kuiBoe0DxPp1CAjfnzUEHeGgWJsVeXoraqhvqVcNyKb56pgRk58xSC/curCR/9im1TPfLVJRbqkDcVcFEyk/pUjfQ+sHaXWCDNllOw= X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(46966006)(36840700001)(107886003)(498600001)(26005)(70206006)(36860700001)(54906003)(6916009)(7636003)(356005)(36906005)(7696005)(70586007)(47076005)(2616005)(426003)(83380400001)(1076003)(36756003)(186003)(4326008)(8936002)(8676002)(2906002)(9786002)(82310400003)(336012)(44832011)(5660300002)(6666004)(102446001); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 14:01:11.3917 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 64167ccc-0d3c-4c8f-68da-08d99ed2646b X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: SN1NAM02FT0055.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR02MB4148 Subject: [dpdk-dev] [PATCH v4 07/10] vdpa/sfc: add support to get queue notify area info X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vijay Kumar Srivastava Implement the vDPA ops get_notify_area to get the notify area info of the queue. Signed-off-by: Vijay Kumar Srivastava Acked-by: Andrew Rybchenko Reviewed-by: Maxime Coquelin --- v2: * Added error log in sfc_vdpa_get_notify_area. v4: * Updated parameter for sfc_vdpa_setup_notify_ctrl drivers/vdpa/sfc/sfc_vdpa_ops.c | 156 ++++++++++++++++++++++++++++++++++++++-- drivers/vdpa/sfc/sfc_vdpa_ops.h | 2 + 2 files changed, 152 insertions(+), 6 deletions(-) diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.c b/drivers/vdpa/sfc/sfc_vdpa_ops.c index f0be652..1725597 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_ops.c +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.c @@ -3,6 +3,8 @@ * Copyright(c) 2020-2021 Xilinx, Inc. */ +#include +#include #include #include @@ -532,6 +534,55 @@ return 0; } +static void * +sfc_vdpa_notify_ctrl(void *arg) +{ + struct sfc_vdpa_ops_data *ops_data; + int vid; + + ops_data = arg; + if (ops_data == NULL) + return NULL; + + sfc_vdpa_adapter_lock(ops_data->dev_handle); + + vid = ops_data->vid; + + if (rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, true) != 0) + sfc_vdpa_info(ops_data->dev_handle, + "vDPA (%s): Notifier could not get configured", + ops_data->vdpa_dev->device->name); + + sfc_vdpa_adapter_unlock(ops_data->dev_handle); + + return NULL; +} + +static int +sfc_vdpa_setup_notify_ctrl(struct sfc_vdpa_ops_data *ops_data) +{ + int ret; + + ops_data->is_notify_thread_started = false; + + /* + * Use rte_vhost_host_notifier_ctrl in a thread to avoid + * dead lock scenario when multiple VFs are used in single vdpa + * application and multiple VFs are passed to a single VM. + */ + ret = pthread_create(&ops_data->notify_tid, NULL, + sfc_vdpa_notify_ctrl, ops_data); + if (ret != 0) { + sfc_vdpa_err(ops_data->dev_handle, + "failed to create notify_ctrl thread: %s", + rte_strerror(ret)); + return -1; + } + ops_data->is_notify_thread_started = true; + + return 0; +} + static int sfc_vdpa_dev_config(int vid) { @@ -565,18 +616,19 @@ if (rc != 0) goto fail_vdpa_start; - sfc_vdpa_adapter_unlock(ops_data->dev_handle); + rc = sfc_vdpa_setup_notify_ctrl(ops_data); + if (rc != 0) + goto fail_vdpa_notify; - sfc_vdpa_log_init(ops_data->dev_handle, "vhost notifier ctrl"); - if (rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, true) != 0) - sfc_vdpa_info(ops_data->dev_handle, - "vDPA (%s): software relay for notify is used.", - vdpa_dev->device->name); + sfc_vdpa_adapter_unlock(ops_data->dev_handle); sfc_vdpa_log_init(ops_data->dev_handle, "done"); return 0; +fail_vdpa_notify: + sfc_vdpa_stop(ops_data); + fail_vdpa_start: sfc_vdpa_close(ops_data); @@ -589,6 +641,7 @@ static int sfc_vdpa_dev_close(int vid) { + int ret; struct rte_vdpa_device *vdpa_dev; struct sfc_vdpa_ops_data *ops_data; @@ -603,6 +656,23 @@ } sfc_vdpa_adapter_lock(ops_data->dev_handle); + if (ops_data->is_notify_thread_started == true) { + void *status; + ret = pthread_cancel(ops_data->notify_tid); + if (ret != 0) { + sfc_vdpa_err(ops_data->dev_handle, + "failed to cancel notify_ctrl thread: %s", + rte_strerror(ret)); + } + + ret = pthread_join(ops_data->notify_tid, &status); + if (ret != 0) { + sfc_vdpa_err(ops_data->dev_handle, + "failed to join terminated notify_ctrl thread: %s", + rte_strerror(ret)); + } + } + ops_data->is_notify_thread_started = false; sfc_vdpa_stop(ops_data); sfc_vdpa_close(ops_data); @@ -653,6 +723,79 @@ return vfio_dev_fd; } +static int +sfc_vdpa_get_notify_area(int vid, int qid, uint64_t *offset, uint64_t *size) +{ + int ret; + efx_nic_t *nic; + int vfio_dev_fd; + efx_rc_t rc; + unsigned int bar_offset; + struct rte_vdpa_device *vdpa_dev; + struct sfc_vdpa_ops_data *ops_data; + struct vfio_region_info reg = { .argsz = sizeof(reg) }; + const efx_nic_cfg_t *encp; + int max_vring_cnt; + int64_t len; + void *dev; + + vdpa_dev = rte_vhost_get_vdpa_device(vid); + + ops_data = sfc_vdpa_get_data_by_dev(vdpa_dev); + if (ops_data == NULL) + return -1; + + dev = ops_data->dev_handle; + + vfio_dev_fd = sfc_vdpa_adapter_by_dev_handle(dev)->vfio_dev_fd; + max_vring_cnt = + (sfc_vdpa_adapter_by_dev_handle(dev)->max_queue_count * 2); + + nic = sfc_vdpa_adapter_by_dev_handle(ops_data->dev_handle)->nic; + encp = efx_nic_cfg_get(nic); + + if (qid >= max_vring_cnt) { + sfc_vdpa_err(dev, "invalid qid : %d", qid); + return -1; + } + + if (ops_data->vq_cxt[qid].enable != B_TRUE) { + sfc_vdpa_err(dev, "vq is not enabled"); + return -1; + } + + rc = efx_virtio_get_doorbell_offset(ops_data->vq_cxt[qid].vq, + &bar_offset); + if (rc != 0) { + sfc_vdpa_err(dev, "failed to get doorbell offset: %s", + rte_strerror(rc)); + return rc; + } + + reg.index = sfc_vdpa_adapter_by_dev_handle(dev)->mem_bar.esb_rid; + ret = ioctl(vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®); + if (ret != 0) { + sfc_vdpa_err(dev, "could not get device region info: %s", + strerror(errno)); + return ret; + } + + *offset = reg.offset + bar_offset; + + len = (1U << encp->enc_vi_window_shift) / 2; + if (len >= sysconf(_SC_PAGESIZE)) { + *size = sysconf(_SC_PAGESIZE); + } else { + sfc_vdpa_err(dev, "invalid VI window size : 0x%" PRIx64, len); + return -1; + } + + sfc_vdpa_info(dev, "vDPA ops get_notify_area :: offset : 0x%" PRIx64, + *offset); + + return 0; +} + static struct rte_vdpa_dev_ops sfc_vdpa_ops = { .get_queue_num = sfc_vdpa_get_queue_num, .get_features = sfc_vdpa_get_features, @@ -662,6 +805,7 @@ .set_vring_state = sfc_vdpa_set_vring_state, .set_features = sfc_vdpa_set_features, .get_vfio_device_fd = sfc_vdpa_get_vfio_device_fd, + .get_notify_area = sfc_vdpa_get_notify_area, }; struct sfc_vdpa_ops_data * diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.h b/drivers/vdpa/sfc/sfc_vdpa_ops.h index d20d219..9b94aca 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_ops.h +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.h @@ -48,6 +48,8 @@ struct sfc_vdpa_ops_data { struct rte_vdpa_device *vdpa_dev; enum sfc_vdpa_context vdpa_context; enum sfc_vdpa_state state; + pthread_t notify_tid; + bool is_notify_thread_started; uint64_t dev_features; uint64_t drv_features; From patchwork Wed Nov 3 13:57:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijay Srivastava X-Patchwork-Id: 103665 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A92EBA0C53; Wed, 3 Nov 2021 15:01:49 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BE43C4273F; Wed, 3 Nov 2021 15:01:19 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2064.outbound.protection.outlook.com [40.107.244.64]) by mails.dpdk.org (Postfix) with ESMTP id 260AC41134 for ; Wed, 3 Nov 2021 15:01:17 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=D7ANyss8ow5q4YA89L9a5cqtbuEsBMR9vSF0b7P5QfEN2aVVLXWahihYX+Ns9rtJTWuEqbbhZHIUSIYhytDw+rFC5Kkpd4C/M0ADRV4MpTia4xYlXAKXvK3YohxIkDvXudOT7yoJA7Nx5iA9tEDmBCp2FD+LWh6B2n+PmPdqIiT7iqY1Rp4MMKxGD1bEkB2o7JezUhQXKu/RsuAfayqCDarmkkdnOjcdvZ77SNQL7WuwirEudlx8CyRc6ercHMpMuaodqk582KxV16LCej23WYHx2RthyusRwq1i0hQPXyky2+4gEY2ChrNsMy5abXyzUG/I6yOFF1v9iPUcXfskkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=R/jfimkq0SoDwhsI+hT465IsACawjTb5DxHRpDgXTn4=; b=HwL3uBCcwZuQayMXgdvlLmrp3jHz37SPr1cTdbKphmeTZjoeXVH8pUQVf/UIukv0R/JFZHbVj4EFwRm4z+SPOmHZ4SvEWijo+cvOMyIpEPvNsCeQe+HKUKNNMhEtAgvbk7ptAcW+whu2+gXZmaOoBd8C9SHmtSJ8cEIlSXI7bVvZcmohaoChsKIl8R5ty/lfKtFdk3Qj+XPQSSfD1JEJK1EJ8iEQE/T4JNHdX2LDoLeYIALx1pbKrBq6Xswlepghg9Ktj+6ic26pbKTbV+6egFBF3IdPHIjOD75/+Rbxg+1J4nbbUfnJ6Pen+N6RpMfzi9n3hpLxS1ArNECvdLRbhw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=dpdk.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=R/jfimkq0SoDwhsI+hT465IsACawjTb5DxHRpDgXTn4=; b=Tq6On21+lwg+XE6h3OXT/7xVQlMQrC9yt1g+b+0/EtyJXKtLfDSlpZeBRzdKHVxbl5sGwfCQkjCgb+BmoxYLQ8ADwLSAypzFVNoi8GVH7QGTIjKvYhe7655DMfgabw7AqtpdJcTepkmEc8tcg/F6nYRhLrsumG+XSpP/iW6bVkY= Received: from SA9PR10CA0004.namprd10.prod.outlook.com (2603:10b6:806:a7::9) by BN7PR02MB4148.namprd02.prod.outlook.com (2603:10b6:406:f6::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.20; Wed, 3 Nov 2021 14:01:15 +0000 Received: from SN1NAM02FT0055.eop-nam02.prod.protection.outlook.com (2603:10b6:806:a7:cafe::2a) by SA9PR10CA0004.outlook.office365.com (2603:10b6:806:a7::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.10 via Frontend Transport; Wed, 3 Nov 2021 14:01:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by SN1NAM02FT0055.mail.protection.outlook.com (10.97.5.78) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4669.10 via Frontend Transport; Wed, 3 Nov 2021 14:01:13 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Wed, 3 Nov 2021 07:00:51 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Wed, 3 Nov 2021 07:00:51 -0700 Envelope-to: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, andrew.rybchenko@oktetlabs.ru Received: from [10.170.66.108] (port=38886 helo=xndengvm004108.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1miGos-0003SS-SS; Wed, 03 Nov 2021 07:00:51 -0700 From: Vijay Srivastava To: CC: , , , Vijay Kumar Srivastava Date: Wed, 3 Nov 2021 19:27:52 +0530 Message-ID: <20211103135754.17411-9-vsrivast@xilinx.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20211103135754.17411-1-vsrivast@xilinx.com> References: <20210706164418.32615-1-vsrivast@xilinx.com> <20211103135754.17411-1-vsrivast@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9fefe4f4-2a00-4571-5310-08d99ed265c5 X-MS-TrafficTypeDiagnostic: BN7PR02MB4148: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:240; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZDHeN5kF0giq0ZOso3cBOYB0yfGU83LkkCUI4lR+gfD+WmyRNF4qbxgx29SqSf7VVhEHd4KZ4WxCGWdMOb7mK1hRoIzgdvhBvnN9snLiq1wSEenGS2dQB3L92XxZtxivKmU7QnG7zncICnhx1cfoAjPRkEyBu99ZfeIe1kkPys+xpXCjzvHuMc5Y8KZwmB0yE3OQtGEGHwr5HRHbjer5CZjKCADvYI75Pim3w7wi3vUxpsOzp9mXECZQ5giIuQ+gP3Jg+YOEeENb5xjmXnQ2WwOF2Mk5mf/n1G6/9c9UVXE2tlQOWzJdMLGEozQ1T2Y06hYuFej/cDg+53XIFqw+UgbeGULNvI6RylBk4eD9Z/QQs8zXjgtZCbCI1/fP4Y1/Zqyu3sZabGFg++jpxvhs1rLpv5Dih+mnkEz5HptPgPzhmh4SE9lO09lvGCzDiDlzQE/mEQe5uXEb4Fo5SmzS9pwPYn5FRL8UfNm3B3M/QqFAypVpx9U+bEt18klkxqk4TSAQ1x9FU1iejFEUbw2DDAwjcijjlNjM2QxKU07TrDIL3BAE1RMPb3h3hHvb4lY5QXaFHkKf5kTTKaKnyJaDgtiRm6cTAKhPqbULMfPOjEEZ//QrEaTWXh1VtWU/xY4H9ZFhCvQQxUQpljU2s1z858uONOr+s1CEblUMEGFbrsz/e6/Wl8ca9ayfSSRkLv8Bapj0JvEyiXlEfaO/arlmy2IGcTOibviHlPCPD4jcLyU= X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(46966006)(36840700001)(107886003)(26005)(70206006)(36860700001)(54906003)(508600001)(6916009)(7636003)(356005)(36906005)(7696005)(316002)(70586007)(47076005)(2616005)(426003)(83380400001)(1076003)(36756003)(186003)(4326008)(8936002)(8676002)(2906002)(9786002)(82310400003)(336012)(44832011)(5660300002)(102446001); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 14:01:13.6597 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9fefe4f4-2a00-4571-5310-08d99ed265c5 X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: SN1NAM02FT0055.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR02MB4148 Subject: [dpdk-dev] [PATCH v4 08/10] vdpa/sfc: add support for MAC filter config X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vijay Kumar Srivastava Add support for unicast and broadcast MAC filter configuration. Signed-off-by: Vijay Kumar Srivastava Acked-by: Andrew Rybchenko Reviewed-by: Maxime Coquelin --- v4: * Removed redundant NULL check. * Improved line usage by rearranging function params. doc/guides/vdpadevs/sfc.rst | 4 ++ drivers/vdpa/sfc/meson.build | 1 + drivers/vdpa/sfc/sfc_vdpa.c | 32 +++++++++ drivers/vdpa/sfc/sfc_vdpa.h | 30 ++++++++ drivers/vdpa/sfc/sfc_vdpa_filter.c | 136 +++++++++++++++++++++++++++++++++++++ drivers/vdpa/sfc/sfc_vdpa_hw.c | 10 +++ drivers/vdpa/sfc/sfc_vdpa_ops.c | 17 +++++ 7 files changed, 230 insertions(+) create mode 100644 drivers/vdpa/sfc/sfc_vdpa_filter.c diff --git a/doc/guides/vdpadevs/sfc.rst b/doc/guides/vdpadevs/sfc.rst index d06c427..512f23e 100644 --- a/doc/guides/vdpadevs/sfc.rst +++ b/doc/guides/vdpadevs/sfc.rst @@ -71,6 +71,10 @@ boolean parameters value. **vdpa** device will work as vdpa device and will be probed by vdpa/sfc driver. If this parameter is not specified then ef100 device will operate as network device. +- ``mac`` [mac address] + + Configures MAC address which would be used to setup MAC filters. + Dynamic Logging Parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/vdpa/sfc/meson.build b/drivers/vdpa/sfc/meson.build index dc333de..2ca33bc 100644 --- a/drivers/vdpa/sfc/meson.build +++ b/drivers/vdpa/sfc/meson.build @@ -22,4 +22,5 @@ sources = files( 'sfc_vdpa_hw.c', 'sfc_vdpa_mcdi.c', 'sfc_vdpa_ops.c', + 'sfc_vdpa_filter.c', ) diff --git a/drivers/vdpa/sfc/sfc_vdpa.c b/drivers/vdpa/sfc/sfc_vdpa.c index f548c1e..78b280e 100644 --- a/drivers/vdpa/sfc/sfc_vdpa.c +++ b/drivers/vdpa/sfc/sfc_vdpa.c @@ -8,7 +8,9 @@ #include #include +#include #include +#include #include #include #include @@ -198,6 +200,31 @@ struct sfc_vdpa_ops_data * return ret < 0 ? RTE_LOGTYPE_PMD : ret; } +static int +sfc_vdpa_kvargs_parse(struct sfc_vdpa_adapter *sva) +{ + struct rte_pci_device *pci_dev = sva->pdev; + struct rte_devargs *devargs = pci_dev->device.devargs; + /* + * To get the device class a mandatory param 'class' is being + * used so included SFC_EFX_KVARG_DEV_CLASS in the param list. + */ + const char **params = (const char *[]){ + RTE_DEVARGS_KEY_CLASS, + SFC_VDPA_MAC_ADDR, + NULL, + }; + + if (devargs == NULL) + return 0; + + sva->kvargs = rte_kvargs_parse(devargs->args, params); + if (sva->kvargs == NULL) + return -EINVAL; + + return 0; +} + static struct rte_pci_id pci_id_sfc_vdpa_efx_map[] = { { RTE_PCI_DEVICE(EFX_PCI_VENID_XILINX, EFX_PCI_DEVID_RIVERHEAD_VF) }, { .vendor_id = 0, /* sentinel */ }, @@ -240,6 +267,10 @@ struct sfc_vdpa_ops_data * if (ret != 0) goto fail_set_log_prefix; + ret = sfc_vdpa_kvargs_parse(sva); + if (ret != 0) + goto fail_kvargs_parse; + sfc_vdpa_log_init(sva, "entry"); sfc_vdpa_adapter_lock_init(sva); @@ -280,6 +311,7 @@ struct sfc_vdpa_ops_data * fail_vfio_setup: sfc_vdpa_adapter_lock_fini(sva); +fail_kvargs_parse: fail_set_log_prefix: rte_free(sva); diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h index 38fd5de..81fc39d 100644 --- a/drivers/vdpa/sfc/sfc_vdpa.h +++ b/drivers/vdpa/sfc/sfc_vdpa.h @@ -17,8 +17,29 @@ #include "sfc_vdpa_log.h" #include "sfc_vdpa_ops.h" +#define SFC_VDPA_MAC_ADDR "mac" #define SFC_VDPA_DEFAULT_MCDI_IOVA 0x200000000000 +/* Broadcast & Unicast MAC filters are supported */ +#define SFC_MAX_SUPPORTED_FILTERS 2 + +/* + * Get function-local index of the associated VI from the + * virtqueue number. Queue 0 is reserved for MCDI + */ +#define SFC_VDPA_GET_VI_INDEX(vq_num) (((vq_num) / 2) + 1) + +enum sfc_vdpa_filter_type { + SFC_VDPA_BCAST_MAC_FILTER = 0, + SFC_VDPA_UCAST_MAC_FILTER = 1, + SFC_VDPA_FILTER_NTYPE +}; + +typedef struct sfc_vdpa_filter_s { + int filter_cnt; + efx_filter_spec_t spec[SFC_MAX_SUPPORTED_FILTERS]; +} sfc_vdpa_filter_t; + /* Adapter private data */ struct sfc_vdpa_adapter { TAILQ_ENTRY(sfc_vdpa_adapter) next; @@ -31,6 +52,8 @@ struct sfc_vdpa_adapter { rte_spinlock_t lock; struct rte_pci_device *pdev; + struct rte_kvargs *kvargs; + efx_family_t family; efx_nic_t *nic; rte_spinlock_t nic_lock; @@ -45,6 +68,8 @@ struct sfc_vdpa_adapter { char log_prefix[SFC_VDPA_LOG_PREFIX_MAX]; uint32_t logtype_main; + sfc_vdpa_filter_t filters; + int vfio_group_fd; int vfio_dev_fd; int vfio_container_fd; @@ -82,6 +107,11 @@ struct sfc_vdpa_ops_data * int sfc_vdpa_dma_map(struct sfc_vdpa_ops_data *vdpa_data, bool do_map); +int +sfc_vdpa_filter_remove(struct sfc_vdpa_ops_data *ops_data); +int +sfc_vdpa_filter_config(struct sfc_vdpa_ops_data *ops_data); + static inline struct sfc_vdpa_adapter * sfc_vdpa_adapter_by_dev_handle(void *dev_handle) { diff --git a/drivers/vdpa/sfc/sfc_vdpa_filter.c b/drivers/vdpa/sfc/sfc_vdpa_filter.c new file mode 100644 index 0000000..9687278 --- /dev/null +++ b/drivers/vdpa/sfc/sfc_vdpa_filter.c @@ -0,0 +1,136 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * + * Copyright(c) 2020-2021 Xilinx, Inc. + */ + +#include +#include +#include + +#include "efx.h" +#include "efx_impl.h" +#include "sfc_vdpa.h" + +static inline int +sfc_vdpa_get_eth_addr(const char *key __rte_unused, + const char *value, void *extra_args) +{ + struct rte_ether_addr *mac_addr = extra_args; + + if (value == NULL || extra_args == NULL) + return -EINVAL; + + /* Convert string with Ethernet address to an ether_addr */ + rte_ether_unformat_addr(value, mac_addr); + + return 0; +} + +static int +sfc_vdpa_set_mac_filter(efx_nic_t *nic, efx_filter_spec_t *spec, + int qid, uint8_t *eth_addr) +{ + int rc; + + if (nic == NULL || spec == NULL) + return -1; + + spec->efs_priority = EFX_FILTER_PRI_MANUAL; + spec->efs_flags = EFX_FILTER_FLAG_RX; + spec->efs_dmaq_id = qid; + + rc = efx_filter_spec_set_eth_local(spec, EFX_FILTER_SPEC_VID_UNSPEC, + eth_addr); + if (rc != 0) + return rc; + + rc = efx_filter_insert(nic, spec); + if (rc != 0) + return rc; + + return rc; +} + +int sfc_vdpa_filter_config(struct sfc_vdpa_ops_data *ops_data) +{ + int rc; + int qid; + efx_nic_t *nic; + struct rte_ether_addr bcast_eth_addr; + struct rte_ether_addr ucast_eth_addr; + struct sfc_vdpa_adapter *sva = ops_data->dev_handle; + efx_filter_spec_t *spec; + + sfc_vdpa_log_init(sva, "entry"); + + nic = sva->nic; + + sfc_vdpa_log_init(sva, "process kvarg"); + + /* skip MAC filter configuration if mac address is not provided */ + if (rte_kvargs_count(sva->kvargs, SFC_VDPA_MAC_ADDR) == 0) { + sfc_vdpa_warn(sva, + "MAC address is not provided, skipping MAC Filter Config"); + return -1; + } + + rc = rte_kvargs_process(sva->kvargs, SFC_VDPA_MAC_ADDR, + &sfc_vdpa_get_eth_addr, + &ucast_eth_addr); + if (rc < 0) + return -1; + + /* create filters on the base queue */ + qid = SFC_VDPA_GET_VI_INDEX(0); + + sfc_vdpa_log_init(sva, "insert broadcast mac filter"); + + EFX_MAC_BROADCAST_ADDR_SET(bcast_eth_addr.addr_bytes); + spec = &sva->filters.spec[SFC_VDPA_BCAST_MAC_FILTER]; + + rc = sfc_vdpa_set_mac_filter(nic, spec, qid, + bcast_eth_addr.addr_bytes); + if (rc != 0) + sfc_vdpa_err(ops_data->dev_handle, + "broadcast MAC filter insertion failed: %s", + rte_strerror(rc)); + else + sva->filters.filter_cnt++; + + sfc_vdpa_log_init(sva, "insert unicast mac filter"); + spec = &sva->filters.spec[SFC_VDPA_UCAST_MAC_FILTER]; + + rc = sfc_vdpa_set_mac_filter(nic, spec, qid, + ucast_eth_addr.addr_bytes); + if (rc != 0) + sfc_vdpa_err(sva, + "unicast MAC filter insertion failed: %s", + rte_strerror(rc)); + else + sva->filters.filter_cnt++; + + sfc_vdpa_log_init(sva, "done"); + + return rc; +} + +int sfc_vdpa_filter_remove(struct sfc_vdpa_ops_data *ops_data) +{ + int i, rc = 0; + struct sfc_vdpa_adapter *sva = ops_data->dev_handle; + efx_nic_t *nic; + + nic = sva->nic; + + for (i = 0; i < sva->filters.filter_cnt; i++) { + rc = efx_filter_remove(nic, &(sva->filters.spec[i])); + if (rc != 0) + sfc_vdpa_err(sva, + "remove HW filter failed for entry %d: %s", + i, rte_strerror(rc)); + } + + sva->filters.filter_cnt = 0; + + return rc; +} diff --git a/drivers/vdpa/sfc/sfc_vdpa_hw.c b/drivers/vdpa/sfc/sfc_vdpa_hw.c index d3459ee..463498c 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_hw.c +++ b/drivers/vdpa/sfc/sfc_vdpa_hw.c @@ -353,10 +353,20 @@ goto fail_virtio_init; } + sfc_vdpa_log_init(sva, "init filter"); + rc = efx_filter_init(enp); + if (rc != 0) { + sfc_vdpa_err(sva, "filter init failed: %s", rte_strerror(rc)); + goto fail_filter_init; + } + sfc_vdpa_log_init(sva, "done"); return 0; +fail_filter_init: + efx_virtio_fini(enp); + fail_virtio_init: efx_nic_fini(enp); diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.c b/drivers/vdpa/sfc/sfc_vdpa_ops.c index 1725597..9d4ed34 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_ops.c +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.c @@ -421,6 +421,8 @@ sfc_vdpa_disable_vfio_intr(ops_data); + sfc_vdpa_filter_remove(ops_data); + ops_data->state = SFC_VDPA_STATE_CONFIGURED; } @@ -460,12 +462,27 @@ goto fail_vq_start; } + ops_data->vq_count = i; + + sfc_vdpa_log_init(ops_data->dev_handle, + "configure MAC filters"); + rc = sfc_vdpa_filter_config(ops_data); + if (rc != 0) { + sfc_vdpa_err(ops_data->dev_handle, + "MAC filter config failed: %s", + rte_strerror(rc)); + goto fail_filter_cfg; + } + ops_data->state = SFC_VDPA_STATE_STARTED; sfc_vdpa_log_init(ops_data->dev_handle, "done"); return 0; +fail_filter_cfg: + /* remove already created filters */ + sfc_vdpa_filter_remove(ops_data); fail_vq_start: /* stop already started virtqueues */ for (j = 0; j < i; j++) From patchwork Wed Nov 3 13:57:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijay Srivastava X-Patchwork-Id: 103666 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4ACB7A0C53; Wed, 3 Nov 2021 15:02:00 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 45D8B42753; Wed, 3 Nov 2021 15:01:21 +0100 (CET) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2068.outbound.protection.outlook.com [40.107.237.68]) by mails.dpdk.org (Postfix) with ESMTP id DC55742736 for ; Wed, 3 Nov 2021 15:01:18 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=agEaQ4qzaUqJSC41Lv81p6e/ma/suNAvNhRsy56ZxFhEQJnDDHVgo+ZrHZLzYMX3Ius+h8js535BgIgCulxC0ag8OyRf1MSojvdd9fSYr1JpBQ1OgQRRpwcZrP/7SdUmU5nQF8WjYfpTqHCNhwwxWJONxRetEClIOtz/prWU+AkaTwHmxwMPERRRMMyvQvEvhfTPe7rUh/vsIyprpyaC671gWNCYZxcEj3EXnaF0QVlEDZYneoER4T0pxBw4w1rAcK9IUqIiCGKBfGOgv4wIkjEtPcYUysnZ8PAe++PH0ltAyCIvc224fNQBPXodmAB45HcxqxsSapvr8Ab3EKXJ4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qpe52jk3sY6Dp721p8uN88ZJYwPfry5FY8JwI0eSCoI=; b=aoE3PiQjBddhIsZGhvMbAKkFVkrD6pir+dwyAvgVTVtzEG2XYPmHXCH3ikWYRPZDDzycOaPR+PL7JpyoujMCCx5uoc3mwt5D1T/xctjKBqLygqmxTBVXrn1GH8KWzQtX11dbEidzDJIlmateLqb/Jm9O4iSZqwjVgUYbEQMAmpTOJvNxje3swz7Z1BI/Z1lX1RM+fvIJEq96lkm5JQiIRA0B3q41Sm+0P924bRzwxmgKbtPEm1RMOt0kyvtru/8OlI/K5ZIu+bCS1WaJNv8jAfQaEqFP9/7fhaTd0/2uEWtmlY1ALoF/4dGxJiC+2rWXA7goGX1em6oYNaZjWqSZpA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=dpdk.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qpe52jk3sY6Dp721p8uN88ZJYwPfry5FY8JwI0eSCoI=; b=rpgx3vE+gpfrKKf002muHN+po2PKO9x76A/H5bVbE2HgSvW8F5eOcAlXSE1EScUq8DnnswlUaWiV9r/pY3Y3hBPUJEiQtXXhAHdC5Ou5ba5tyIW0QjmUTbaqhLhlq+jovIaei+54PwxZYO6ym3p9PT6OJKdXB3H8MVV8lPlikiI= Received: from SA9PR10CA0013.namprd10.prod.outlook.com (2603:10b6:806:a7::18) by SN6PR02MB5264.namprd02.prod.outlook.com (2603:10b6:805:71::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.15; Wed, 3 Nov 2021 14:01:16 +0000 Received: from SN1NAM02FT0055.eop-nam02.prod.protection.outlook.com (2603:10b6:806:a7:cafe::9f) by SA9PR10CA0013.outlook.office365.com (2603:10b6:806:a7::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.11 via Frontend Transport; Wed, 3 Nov 2021 14:01:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by SN1NAM02FT0055.mail.protection.outlook.com (10.97.5.78) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4669.10 via Frontend Transport; Wed, 3 Nov 2021 14:01:15 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Wed, 3 Nov 2021 07:00:53 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Wed, 3 Nov 2021 07:00:53 -0700 Envelope-to: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, andrew.rybchenko@oktetlabs.ru Received: from [10.170.66.108] (port=38886 helo=xndengvm004108.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1miGov-0003SS-3o; Wed, 03 Nov 2021 07:00:53 -0700 From: Vijay Srivastava To: CC: , , , Vijay Kumar Srivastava Date: Wed, 3 Nov 2021 19:27:53 +0530 Message-ID: <20211103135754.17411-10-vsrivast@xilinx.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20211103135754.17411-1-vsrivast@xilinx.com> References: <20210706164418.32615-1-vsrivast@xilinx.com> <20211103135754.17411-1-vsrivast@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9b2c5591-9bf0-40cf-0d72-08d99ed26723 X-MS-TrafficTypeDiagnostic: SN6PR02MB5264: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:124; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3kLargMvgfLJx9uQ/rRN7AftCCyH5J+MKJYOLYmpiPN9PfA8S0AoIlJBa/PG/PFnRw31FzoPy5ZKuqdLbu9MIxmUjuIeGNPyKinvdrrsHRjIbWZkkcrakzvMgwFdMduEbIZLuS04lNwokF5r9Y7oEnozOMIYNCXRBtufyqOfTLmNuCiMOGKujTOhPlnPxz3FxDEro8uxf0GEn9LORP3Le3sfUN/B19/BK0LlYrRiOO7tBkZt07Bd/1y2Qrc6OsT9JtCib6hVf4DoWN+c/g7LGP5Bravyizq7tYFP8PGRzCmgEjumovNeZtgQezphNcnSsqs4Tu/AQeH9nIhsE/TNTIorhjLuVERCThnr7EMeYg4KrbDDwaiMCY/rxc3X4QjAKoLwIJsLHO/16YvCOLqWDeBhndAWEDkitwewptiO2uA+umZ5Y095LO631FA0G7obqyVFOsTZGKDn/kSfwr+0VvxxnCUEvpBGvGp8+6XpQHm2gEV92g4DSE8ULIngzOZHg4lLdNhtIIXXY8DnL24Jc4nD6wzRuZskFgq9+bgfyXYGCB6uKDfjTCS831f0/opKXmGguMh32epOIh/H8TR55ogtv9h8stw5KMjS7POT0BTGW5U+IEAgadu+lW7bQE/e48/kvkPtjIrP49huGefTot9u06rZ1UZTCqD+K1FTgN9EQ6frbQ9c+tchcGNeJY8eJuqU7o/VY0gSU1+IvQNkXMK+QWLswnfM9cO9fFQbRDI= X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(46966006)(36840700001)(2906002)(83380400001)(36860700001)(54906003)(316002)(426003)(5660300002)(2616005)(82310400003)(1076003)(186003)(8676002)(7636003)(70586007)(36906005)(26005)(356005)(70206006)(6916009)(47076005)(336012)(36756003)(8936002)(107886003)(508600001)(44832011)(7696005)(4326008)(9786002)(102446001); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 14:01:15.9517 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9b2c5591-9bf0-40cf-0d72-08d99ed26723 X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: SN1NAM02FT0055.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR02MB5264 Subject: [dpdk-dev] [PATCH v4 09/10] vdpa/sfc: add support to set vring state X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vijay Kumar Srivastava Implements vDPA ops set_vring_state to configure vring state. Signed-off-by: Vijay Kumar Srivastava Acked-by: Andrew Rybchenko Reviewed-by: Maxime Coquelin Reviewed-by: Chenbo Xia --- drivers/vdpa/sfc/sfc_vdpa_ops.c | 54 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 50 insertions(+), 4 deletions(-) diff --git a/drivers/vdpa/sfc/sfc_vdpa_ops.c b/drivers/vdpa/sfc/sfc_vdpa_ops.c index 9d4ed34..c5c97b8 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_ops.c +++ b/drivers/vdpa/sfc/sfc_vdpa_ops.c @@ -702,11 +702,57 @@ static int sfc_vdpa_set_vring_state(int vid, int vring, int state) { - RTE_SET_USED(vid); - RTE_SET_USED(vring); - RTE_SET_USED(state); + struct sfc_vdpa_ops_data *ops_data; + struct rte_vdpa_device *vdpa_dev; + efx_rc_t rc; + int vring_max; + void *dev; - return -1; + vdpa_dev = rte_vhost_get_vdpa_device(vid); + + ops_data = sfc_vdpa_get_data_by_dev(vdpa_dev); + if (ops_data == NULL) + return -1; + + dev = ops_data->dev_handle; + + sfc_vdpa_info(dev, + "vDPA ops set_vring_state: vid: %d, vring: %d, state:%d", + vid, vring, state); + + vring_max = (sfc_vdpa_adapter_by_dev_handle(dev)->max_queue_count * 2); + + if (vring < 0 || vring > vring_max) { + sfc_vdpa_err(dev, "received invalid vring id : %d to set state", + vring); + return -1; + } + + /* + * Skip if device is not yet started. virtqueues state can be + * changed once it is created and other configurations are done. + */ + if (ops_data->state != SFC_VDPA_STATE_STARTED) + return 0; + + if (ops_data->vq_cxt[vring].enable == state) + return 0; + + if (state == 0) { + rc = sfc_vdpa_virtq_stop(ops_data, vring); + if (rc != 0) { + sfc_vdpa_err(dev, "virtqueue stop failed: %s", + rte_strerror(rc)); + } + } else { + rc = sfc_vdpa_virtq_start(ops_data, vring); + if (rc != 0) { + sfc_vdpa_err(dev, "virtqueue start failed: %s", + rte_strerror(rc)); + } + } + + return rc; } static int From patchwork Wed Nov 3 13:57:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vijay Srivastava X-Patchwork-Id: 103667 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7678EA0C53; Wed, 3 Nov 2021 15:02:09 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 52C904275D; Wed, 3 Nov 2021 15:01:22 +0100 (CET) Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam07on2041.outbound.protection.outlook.com [40.107.95.41]) by mails.dpdk.org (Postfix) with ESMTP id ABBEA42756 for ; Wed, 3 Nov 2021 15:01:21 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XiqyAV6Ng2owNmqk0RMLuyCXsVyiaY+RoqeNn/Lu4+y8+iV1vce0mf1OUzWyt2YNPXDFUoyzizF1lN+Pwo3O9cbjuhm4rkDgAeqkjU4TWLPpnABQk9zqOm8vGL4VeHD8q187vA8/Bk/87qGEDbbPzrpIaKSpOyJmlkyM6kLr5Vn+FP0l3d3j99UBHbdX9oWCgfnm4/7zH0vOf1sOqHrI5XjZrh18Nvqgsjz5byV5NK8kfwvlGP9J8Qez0ACWoZi43aY5HCQ+GXimchyt444OEUXj7Y38zgfEF/SZDA3NoObaf4WjE+ImJ6oDfiI+l/WrZV07o+bm7FCVvXiyT8dh7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FDqmHez17t6BGlZl0B1rgyzIIol0MeRdLhaMtox4doI=; b=ahRrx7ob4EkwqnE8G08x89dwxB7VRFD2imvXwPZjfNd/LbQHmuhhyow5kCmJWIc86Nm5V7aamSVtAVrQc6HWP3TRG8cDavJAoCRMQZvhuGJ5wNuYuaiR3SJzWAr3s2EolTstg35MAvAmHGHCMgAJ0qD1hrURyiu+Fp5gkfvgFOoxsz4Rtw2X+KIX+5skqii1KwoptD8kCrQQ0ubai3UBnBmhdukjsSRkzNvJHySNcxRnQ/fFsoL7a8mF+kGJhFhDdeXOBz+ETl14KMZudRFavlbJkg1Q/39YfsW3c+1zQZtjhzMXTfeEIAKW6VTaftZQzUdkPhI00OkFBwpwuDdhBw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 149.199.62.198) smtp.rcpttodomain=dpdk.org smtp.mailfrom=xilinx.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=xilinx.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FDqmHez17t6BGlZl0B1rgyzIIol0MeRdLhaMtox4doI=; b=bvdA/fE+ECr1BEzPltwHbFVogTWK4eVGAC73nsz+GG+kchYCcTOy8G5RyI3EiXWd4DYAOBYEB+ej+iut095+qKKTO3w1JbO5zvcDXNw0SVUhVkJj9YvNzKJunHeKo1dRlQM8NgznQl1ZBfQPzWviOJLQOqlMpgFSMC8xVEtuUUo= Received: from SA0PR11CA0163.namprd11.prod.outlook.com (2603:10b6:806:1bb::18) by DM6PR02MB6587.namprd02.prod.outlook.com (2603:10b6:5:220::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.15; Wed, 3 Nov 2021 14:01:19 +0000 Received: from SN1NAM02FT0029.eop-nam02.prod.protection.outlook.com (2603:10b6:806:1bb:cafe::11) by SA0PR11CA0163.outlook.office365.com (2603:10b6:806:1bb::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.15 via Frontend Transport; Wed, 3 Nov 2021 14:01:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 149.199.62.198) smtp.mailfrom=xilinx.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=xilinx.com; Received-SPF: Pass (protection.outlook.com: domain of xilinx.com designates 149.199.62.198 as permitted sender) receiver=protection.outlook.com; client-ip=149.199.62.198; helo=xsj-pvapexch02.xlnx.xilinx.com; Received: from xsj-pvapexch02.xlnx.xilinx.com (149.199.62.198) by SN1NAM02FT0029.mail.protection.outlook.com (10.97.4.175) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4669.10 via Frontend Transport; Wed, 3 Nov 2021 14:01:19 +0000 Received: from xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Wed, 3 Nov 2021 07:00:55 -0700 Received: from smtp.xilinx.com (172.19.127.96) by xsj-pvapexch02.xlnx.xilinx.com (172.19.86.41) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Wed, 3 Nov 2021 07:00:55 -0700 Envelope-to: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, andrew.rybchenko@oktetlabs.ru Received: from [10.170.66.108] (port=38886 helo=xndengvm004108.xilinx.com) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1miGox-0003SS-BX; Wed, 03 Nov 2021 07:00:55 -0700 From: Vijay Srivastava To: CC: , , , Vijay Kumar Srivastava Date: Wed, 3 Nov 2021 19:27:54 +0530 Message-ID: <20211103135754.17411-11-vsrivast@xilinx.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20211103135754.17411-1-vsrivast@xilinx.com> References: <20210706164418.32615-1-vsrivast@xilinx.com> <20211103135754.17411-1-vsrivast@xilinx.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8d57f3c7-32be-4fda-2df4-08d99ed2694f X-MS-TrafficTypeDiagnostic: DM6PR02MB6587: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:101; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: L3AP+RTa4eskTzioN6CeGJ5BUHROPbLJ/4PDp69emItdIdZqODaHCkJo+zI1GOUWsbtapXF0Ye1fpUivAbJSetFeuvDY7mLDbuaWZk+9M8nSfs7QhGs9cQaZs8m8JAd/ZxEukH6iNMw9MeoWiaqvuH6pF46FPPQ9726clG4qfD6neZxK7NGxtxz6B8q1gS/ldATyVNcjHUZ1Z+vT8Bd0vAi0Gc8VVhXr2g/nw+BHlP5mNGe9s6zqtu1Ju0L3q8/xxwUe2a6YPJ4+WHU6J+eFMXYYF0LFW0Mh6uxy5GoxRKuz60iqAY0rWjp9Y8FFkvbzSkoENtd1vExD6++u9pbgaOr+SsCp53Oi2/c18DQ6/e8ubSoEfYSm0CV/LpcxNH3J+dGmpVjIhUKavuSdMUPNtmuefk4nFEp280U5e14mmVU5jhnr/AeppF+O1h4lINa1pvVUZ+ZV9xdsUdXIkjy1FvhIeR9mGNtCGhiACHaUbDk3TYGvKa+onP+7rRfj92CpXg3HOB3RDS5LWZX5RtaIeOqjwr4PFr/JjYftLkys3PtG/M8f3uD0kFix/0f9p9wviQOwmsdatp55jFnzDsgiLX7p8k+7WgTCXzct1xLH4TLSIPIBuf2BiSINr1hYd+fBIlODIJJ7XqJGuigOKWO6EsL+bq5NDbm8otg6QczK9eE+B8l6YDp1dAT++33SFs0TGc5W1GnasPxJ/dZh60pbYgtxLQVWDBelZZF82amyGJc= X-Forefront-Antispam-Report: CIP:149.199.62.198; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:xsj-pvapexch02.xlnx.xilinx.com; PTR:unknown-62-198.xilinx.com; CAT:NONE; SFS:(36840700001)(46966006)(426003)(47076005)(82310400003)(54906003)(316002)(36860700001)(2906002)(36906005)(336012)(5660300002)(1076003)(8676002)(9786002)(6916009)(107886003)(4326008)(83380400001)(7636003)(2616005)(70206006)(356005)(44832011)(70586007)(36756003)(8936002)(7696005)(186003)(26005)(508600001)(102446001); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2021 14:01:19.5963 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8d57f3c7-32be-4fda-2df4-08d99ed2694f X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.62.198]; Helo=[xsj-pvapexch02.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: SN1NAM02FT0029.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR02MB6587 Subject: [dpdk-dev] [PATCH v4 10/10] vdpa/sfc: set a multicast filter during vDPA init X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vijay Kumar Srivastava Insert unknown multicast filter to allow IPv6 neighbor discovery Signed-off-by: Vijay Kumar Srivastava Acked-by: Andrew Rybchenko Reviewed-by: Chenbo Xia Reviewed-by: Maxime Coquelin --- v4: * Improved line usage by rearranging function params. drivers/vdpa/sfc/sfc_vdpa.h | 3 ++- drivers/vdpa/sfc/sfc_vdpa_filter.c | 20 +++++++++++++++++--- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/drivers/vdpa/sfc/sfc_vdpa.h b/drivers/vdpa/sfc/sfc_vdpa.h index 81fc39d..080f2d6 100644 --- a/drivers/vdpa/sfc/sfc_vdpa.h +++ b/drivers/vdpa/sfc/sfc_vdpa.h @@ -21,7 +21,7 @@ #define SFC_VDPA_DEFAULT_MCDI_IOVA 0x200000000000 /* Broadcast & Unicast MAC filters are supported */ -#define SFC_MAX_SUPPORTED_FILTERS 2 +#define SFC_MAX_SUPPORTED_FILTERS 3 /* * Get function-local index of the associated VI from the @@ -32,6 +32,7 @@ enum sfc_vdpa_filter_type { SFC_VDPA_BCAST_MAC_FILTER = 0, SFC_VDPA_UCAST_MAC_FILTER = 1, + SFC_VDPA_MCAST_DST_FILTER = 2, SFC_VDPA_FILTER_NTYPE }; diff --git a/drivers/vdpa/sfc/sfc_vdpa_filter.c b/drivers/vdpa/sfc/sfc_vdpa_filter.c index 9687278..4c133aa 100644 --- a/drivers/vdpa/sfc/sfc_vdpa_filter.c +++ b/drivers/vdpa/sfc/sfc_vdpa_filter.c @@ -39,8 +39,12 @@ spec->efs_flags = EFX_FILTER_FLAG_RX; spec->efs_dmaq_id = qid; - rc = efx_filter_spec_set_eth_local(spec, EFX_FILTER_SPEC_VID_UNSPEC, - eth_addr); + if (eth_addr == NULL) + rc = efx_filter_spec_set_mc_def(spec); + else + rc = efx_filter_spec_set_eth_local(spec, + EFX_FILTER_SPEC_VID_UNSPEC, + eth_addr); if (rc != 0) return rc; @@ -103,8 +107,18 @@ int sfc_vdpa_filter_config(struct sfc_vdpa_ops_data *ops_data) rc = sfc_vdpa_set_mac_filter(nic, spec, qid, ucast_eth_addr.addr_bytes); if (rc != 0) + sfc_vdpa_err(sva, "unicast MAC filter insertion failed: %s", + rte_strerror(rc)); + else + sva->filters.filter_cnt++; + + sfc_vdpa_log_init(sva, "insert unknown mcast filter"); + spec = &sva->filters.spec[SFC_VDPA_MCAST_DST_FILTER]; + + rc = sfc_vdpa_set_mac_filter(nic, spec, qid, NULL); + if (rc != 0) sfc_vdpa_err(sva, - "unicast MAC filter insertion failed: %s", + "mcast filter insertion failed: %s", rte_strerror(rc)); else sva->filters.filter_cnt++;