From patchwork Mon Sep 28 16:42:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 79034 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id CDDC4A04DB; Mon, 28 Sep 2020 18:46:57 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id A26B01D8FD; Mon, 28 Sep 2020 18:43:28 +0200 (CEST) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by dpdk.org (Postfix) with ESMTP id 4C71F1D719 for ; Mon, 28 Sep 2020 18:43:20 +0200 (CEST) IronPort-SDR: z1qfO5YqOrXjZKS/pxN5QR/E18e9sg+eskSh1pYtq2GrsZd/6rrbvsH5EIHHJY+4Ky4kcJQzxn cu3SPW6tO/eQ== X-IronPort-AV: E=McAfee;i="6000,8403,9758"; a="223619811" X-IronPort-AV: E=Sophos;i="5.77,313,1596524400"; d="scan'208";a="223619811" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Sep 2020 09:43:19 -0700 IronPort-SDR: b1xxbKmqONv7dQx1AdOuiJ1NB9vKjcET9WYcPuAk20I8Y5JqPQ4STe4nSf56ZHchUvlwUzKMcH Of+PAoWsLRVA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,313,1596524400"; d="scan'208";a="338250534" Received: from silpixa00399126.ir.intel.com ([10.237.222.4]) by fmsmga004.fm.intel.com with ESMTP; 28 Sep 2020 09:43:17 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: patrick.fu@intel.com, Bruce Richardson , Kevin Laatz Date: Mon, 28 Sep 2020 17:42:31 +0100 Message-Id: <20200928164245.84997-12-bruce.richardson@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200928164245.84997-1-bruce.richardson@intel.com> References: <20200721095140.719297-1-bruce.richardson@intel.com> <20200928164245.84997-1-bruce.richardson@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 11/25] raw/ioat: add skeleton for VFIO/UIO based DSA device X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add in the basic probe/remove skeleton code for DSA devices which are bound directly to vfio or uio driver. The kernel module for supporting these uses the "idxd" name, so that name is used as function and file prefix to avoid conflict with existing "ioat" prefixed functions. Since we are adding new files to the driver and there will be common definitions shared between the various files, we create a new internal header file ioat_private.h to hold common macros and function prototypes. Signed-off-by: Bruce Richardson Reviewed-by: Kevin Laatz --- doc/guides/rawdevs/ioat.rst | 69 ++++++++++----------------------- drivers/raw/ioat/idxd_pci.c | 56 ++++++++++++++++++++++++++ drivers/raw/ioat/ioat_private.h | 27 +++++++++++++ drivers/raw/ioat/ioat_rawdev.c | 9 +---- drivers/raw/ioat/meson.build | 6 ++- 5 files changed, 108 insertions(+), 59 deletions(-) create mode 100644 drivers/raw/ioat/idxd_pci.c create mode 100644 drivers/raw/ioat/ioat_private.h diff --git a/doc/guides/rawdevs/ioat.rst b/doc/guides/rawdevs/ioat.rst index 71bca0b28f..b898f98d5f 100644 --- a/doc/guides/rawdevs/ioat.rst +++ b/doc/guides/rawdevs/ioat.rst @@ -3,10 +3,12 @@ .. include:: -IOAT Rawdev Driver for Intel\ |reg| QuickData Technology -====================================================================== +IOAT Rawdev Driver +=================== The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg| +Data Streaming Accelerator `(Intel DSA) +`_ and for Intel\ |reg| QuickData Technology, part of Intel\ |reg| I/O Acceleration Technology `(Intel I/OAT) `_. @@ -17,61 +19,30 @@ be done by software, freeing up CPU cycles for other tasks. Hardware Requirements ---------------------- -On Linux, the presence of an Intel\ |reg| QuickData Technology hardware can -be detected by checking the output of the ``lspci`` command, where the -hardware will be often listed as "Crystal Beach DMA" or "CBDMA". For -example, on a system with Intel\ |reg| Xeon\ |reg| CPU E5-2699 v4 @2.20GHz, -lspci shows: - -.. code-block:: console - - # lspci | grep DMA - 00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01) - 00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01) - 00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01) - 00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01) - 00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01) - 00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01) - 00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01) - 00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01) - -On a system with Intel\ |reg| Xeon\ |reg| Gold 6154 CPU @ 3.00GHz, lspci -shows: - -.. code-block:: console - - # lspci | grep DMA - 00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - 00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) - +The ``dpdk-devbind.py`` script, included with DPDK, +can be used to show the presence of supported hardware. +Running ``dpdk-devbind.py --status-dev misc`` will show all the miscellaneous, +or rawdev-based devices on the system. +For Intel\ |reg| QuickData Technology devices, the hardware will be often listed as "Crystal Beach DMA", +or "CBDMA". +For Intel\ |reg| DSA devices, they are currently (at time of writing) appearing as devices with type "0b25", +due to the absence of pci-id database entries for them at this point. Compilation ------------ -For builds done with ``make``, the driver compilation is enabled by the -``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option. This is -enabled by default in builds for x86 platforms, and disabled in other -configurations. - -For builds using ``meson`` and ``ninja``, the driver will be built when the -target platform is x86-based. +For builds using ``meson`` and ``ninja``, the driver will be built when the target platform is x86-based. +No additional compilation steps are necessary. Device Setup ------------- -The Intel\ |reg| QuickData Technology HW devices will need to be bound to a -user-space IO driver for use. The script ``dpdk-devbind.py`` script -included with DPDK can be used to view the state of the devices and to bind -them to a suitable DPDK-supported kernel driver. When querying the status -of the devices, they will appear under the category of "Misc (rawdev) -devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be -used to see the state of those devices alone. +The HW devices to be used will need to be bound to a user-space IO driver for use. +The ``dpdk-devbind.py`` script can be used to view the state of the devices +and to bind them to a suitable DPDK-supported kernel driver, such as ``vfio-pci``. +For example:: + + $ dpdk-devbind.py -b vfio-pci 00:04.0 00:04.1 Device Probing and Initialization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/raw/ioat/idxd_pci.c b/drivers/raw/ioat/idxd_pci.c new file mode 100644 index 0000000000..1a30e9c316 --- /dev/null +++ b/drivers/raw/ioat/idxd_pci.c @@ -0,0 +1,56 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2020 Intel Corporation + */ + +#include + +#include "ioat_private.h" + +#define IDXD_VENDOR_ID 0x8086 +#define IDXD_DEVICE_ID_SPR 0x0B25 + +#define IDXD_PMD_RAWDEV_NAME_PCI rawdev_idxd_pci + +const struct rte_pci_id pci_id_idxd_map[] = { + { RTE_PCI_DEVICE(IDXD_VENDOR_ID, IDXD_DEVICE_ID_SPR) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static int +idxd_rawdev_probe_pci(struct rte_pci_driver *drv, struct rte_pci_device *dev) +{ + int ret = 0; + char name[PCI_PRI_STR_SIZE]; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); + dev->device.driver = &drv->driver; + + return ret; +} + +static int +idxd_rawdev_remove_pci(struct rte_pci_device *dev) +{ + char name[PCI_PRI_STR_SIZE]; + int ret = 0; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + + IOAT_PMD_INFO("Closing %s on NUMA node %d", + name, dev->device.numa_node); + + return ret; +} + +struct rte_pci_driver idxd_pmd_drv_pci = { + .id_table = pci_id_idxd_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING, + .probe = idxd_rawdev_probe_pci, + .remove = idxd_rawdev_remove_pci, +}; + +RTE_PMD_REGISTER_PCI(IDXD_PMD_RAWDEV_NAME_PCI, idxd_pmd_drv_pci); +RTE_PMD_REGISTER_PCI_TABLE(IDXD_PMD_RAWDEV_NAME_PCI, pci_id_idxd_map); +RTE_PMD_REGISTER_KMOD_DEP(IDXD_PMD_RAWDEV_NAME_PCI, + "* igb_uio | uio_pci_generic | vfio-pci"); diff --git a/drivers/raw/ioat/ioat_private.h b/drivers/raw/ioat/ioat_private.h new file mode 100644 index 0000000000..d87d4d055e --- /dev/null +++ b/drivers/raw/ioat/ioat_private.h @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2020 Intel Corporation + */ + +#ifndef _IOAT_PRIVATE_H_ +#define _IOAT_PRIVATE_H_ + +/** + * @file idxd_private.h + * + * Private data structures for the idxd/DSA part of ioat device driver + * + * @warning + * @b EXPERIMENTAL: these structures and APIs may change without prior notice + */ + +extern int ioat_pmd_logtype; + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +#endif /* _IOAT_PRIVATE_H_ */ diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c index aa59b731fd..1fe32278d2 100644 --- a/drivers/raw/ioat/ioat_rawdev.c +++ b/drivers/raw/ioat/ioat_rawdev.c @@ -10,6 +10,7 @@ #include "rte_ioat_rawdev.h" #include "ioat_spec.h" +#include "ioat_private.h" static struct rte_pci_driver ioat_pmd_drv; @@ -29,14 +30,6 @@ static struct rte_pci_driver ioat_pmd_drv; RTE_LOG_REGISTER(ioat_pmd_logtype, rawdev.ioat, INFO); -#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ - ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args) - -#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) -#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) -#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) -#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) - #define DESC_SZ sizeof(struct rte_ioat_generic_hw_desc) #define COMPLETION_SZ sizeof(__m128i) diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build index 06636f8a9f..3529635e9c 100644 --- a/drivers/raw/ioat/meson.build +++ b/drivers/raw/ioat/meson.build @@ -3,8 +3,10 @@ build = dpdk_conf.has('RTE_ARCH_X86') reason = 'only supported on x86' -sources = files('ioat_rawdev.c', - 'ioat_rawdev_test.c') +sources = files( + 'idxd_pci.c', + 'ioat_rawdev.c', + 'ioat_rawdev_test.c') deps += ['rawdev', 'bus_pci', 'mbuf'] install_headers('rte_ioat_rawdev.h',