From patchwork Thu Jun 27 10:40:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 55465 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id A74782956; Thu, 27 Jun 2019 12:41:07 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 025F12956 for ; Thu, 27 Jun 2019 12:41:04 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Jun 2019 03:41:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,423,1557212400"; d="scan'208";a="156198707" Received: from silpixa00399126.ir.intel.com (HELO silpixa00399126.ger.corp.intel.com) ([10.237.223.2]) by orsmga008.jf.intel.com with ESMTP; 27 Jun 2019 03:41:02 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: thomas@monjalon.net, jerinj@marvell.com, Bruce Richardson Date: Thu, 27 Jun 2019 11:40:48 +0100 Message-Id: <20190627104055.8244-2-bruce.richardson@intel.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190627104055.8244-1-bruce.richardson@intel.com> References: <20190530212525.40370-1-bruce.richardson@intel.com> <20190627104055.8244-1-bruce.richardson@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v3 1/8] raw/ioat: add initial support for ioat rawdev driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add stubs for ioat rawdev driver support in DPDK, specifically: * makefile and meson build hooks * initial public header file * rawdev main C file, with probe and release functions * release note update announcing the driver * initial documentation for the new section in the rawdev doc * unit test stubs for device unit tests Signed-off-by: Bruce Richardson Acked-by: Anatoly Burakov Signed-off-by: Bruce Richardson Signed-off-by: Bruce Richardson --- V2: don't create a new file for unit testing, add to existing rawdev test file, and place test cases in the driver selftest routine (added later in set) Add new section in document about identifying hardware using lspci --- MAINTAINERS | 6 +- app/test/test_rawdev.c | 8 ++ config/common_armv8a_linux | 1 + config/common_base | 5 ++ config/defconfig_arm-armv7a-linuxapp-gcc | 1 + config/defconfig_ppc_64-power8-linuxapp-gcc | 1 + doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ioat_rawdev.rst | 63 ++++++++++++++ doc/guides/rel_notes/release_19_08.rst | 11 +++ drivers/raw/Makefile | 1 + drivers/raw/ioat/Makefile | 28 +++++++ drivers/raw/ioat/ioat_rawdev.c | 93 +++++++++++++++++++++ drivers/raw/ioat/meson.build | 8 ++ drivers/raw/ioat/rte_ioat_rawdev.h | 24 ++++++ drivers/raw/ioat/rte_pmd_ioat_version.map | 4 + drivers/raw/meson.build | 4 +- mk/rte.app.mk | 1 + 17 files changed, 258 insertions(+), 2 deletions(-) create mode 100644 doc/guides/rawdevs/ioat_rawdev.rst create mode 100644 drivers/raw/ioat/Makefile create mode 100644 drivers/raw/ioat/ioat_rawdev.c create mode 100644 drivers/raw/ioat/meson.build create mode 100644 drivers/raw/ioat/rte_ioat_rawdev.h create mode 100644 drivers/raw/ioat/rte_pmd_ioat_version.map diff --git a/MAINTAINERS b/MAINTAINERS index 0c3b48920..f28c526bc 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1068,6 +1068,11 @@ M: Tianfei zhang F: drivers/raw/ifpga_rawdev/ F: doc/guides/rawdevs/ifpga_rawdev.rst +IOAT Rawdev +M: Bruce Richardson +F: drivers/raw/ioat/ +F: doc/guides/rawdevs/ioat_rawdev.rst + NXP DPAA2 QDMA M: Nipun Gupta F: drivers/raw/dpaa2_qdma/ @@ -1078,7 +1083,6 @@ M: Nipun Gupta F: drivers/raw/dpaa2_cmdif/ F: doc/guides/rawdevs/dpaa2_cmdif.rst - Packet processing ----------------- diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c index 043a38a13..88549fb61 100644 --- a/app/test/test_rawdev.c +++ b/app/test/test_rawdev.c @@ -25,3 +25,11 @@ test_rawdev_selftest_skeleton(void) } REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton); + +static int +test_rawdev_selftest_ioat(void) +{ + return TEST_SKIPPED; +} + +REGISTER_TEST_COMMAND(ioat_rawdev_autotest, test_rawdev_selftest_ioat); diff --git a/config/common_armv8a_linux b/config/common_armv8a_linux index 72091de1c..481712ebc 100644 --- a/config/common_armv8a_linux +++ b/config/common_armv8a_linux @@ -34,5 +34,6 @@ CONFIG_RTE_ARCH_ARM64_MEMCPY=n CONFIG_RTE_LIBRTE_FM10K_PMD=n CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n CONFIG_RTE_LIBRTE_AVP_PMD=n +CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n CONFIG_RTE_SCHED_VECTOR=n diff --git a/config/common_base b/config/common_base index fa1ae249a..e6b830923 100644 --- a/config/common_base +++ b/config/common_base @@ -747,6 +747,11 @@ CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n # CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y +# +# Compile PMD for Intel IOAT raw device +# +CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=y + # # Compile librte_ring # diff --git a/config/defconfig_arm-armv7a-linuxapp-gcc b/config/defconfig_arm-armv7a-linuxapp-gcc index c9509b274..ee158ef9d 100644 --- a/config/defconfig_arm-armv7a-linuxapp-gcc +++ b/config/defconfig_arm-armv7a-linuxapp-gcc @@ -54,3 +54,4 @@ CONFIG_RTE_LIBRTE_QEDE_PMD=n CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n CONFIG_RTE_LIBRTE_AVP_PMD=n CONFIG_RTE_LIBRTE_NFP_PMD=n +CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n diff --git a/config/defconfig_ppc_64-power8-linuxapp-gcc b/config/defconfig_ppc_64-power8-linuxapp-gcc index 7e248b755..9f3670ec0 100644 --- a/config/defconfig_ppc_64-power8-linuxapp-gcc +++ b/config/defconfig_ppc_64-power8-linuxapp-gcc @@ -56,3 +56,4 @@ CONFIG_RTE_LIBRTE_ENIC_PMD=n CONFIG_RTE_LIBRTE_FM10K_PMD=n CONFIG_RTE_LIBRTE_SFC_EFX_PMD=n CONFIG_RTE_LIBRTE_AVP_PMD=n +CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=n diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst index 7c3bd9586..0a21989e4 100644 --- a/doc/guides/rawdevs/index.rst +++ b/doc/guides/rawdevs/index.rst @@ -14,3 +14,4 @@ application through rawdev API. dpaa2_cmdif dpaa2_qdma ifpga_rawdev + ioat_rawdev diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst new file mode 100644 index 000000000..0c612e73a --- /dev/null +++ b/doc/guides/rawdevs/ioat_rawdev.rst @@ -0,0 +1,63 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +.. include:: + +IOAT Rawdev Driver for Intel\ |reg| QuickData Technology +====================================================================== + +The ``ioat`` rawdev driver provides a poll-mode driver (PMD) for Intel\ |reg| +QuickData Technology, part of Intel\ |reg| I/O Acceleration Technology +`(Intel I/OAT) +`_. +This PMD, when used on supported hardware, allows data copies, for example, +cloning packet data, to be accelerated by that hardware rather than having to +be done by software, freeing up CPU cycles for other tasks. + +Hardware Requirements +---------------------- + +On Linux, the presence of an Intel\ |reg| QuickData Technology hardware can +be detected by checking the output of the ``lspci`` command, where the +hardware will be often listed as "Crystal Beach DMA" or "CBDMA". For +example, on a system with Intel\ |reg| Xeon\ |reg| CPU E5-2699 v4 @2.20GHz, +lspci shows: + +.. code-block:: console + + # lspci | grep DMA + 00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01) + 00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01) + 00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01) + 00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01) + 00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01) + 00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01) + 00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01) + 00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01) + +On a system with Intel\ |reg| Xeon\ |reg| Gold 6154 CPU @ 3.00GHz, lspci +shows: + +.. code-block:: console + + # lspci | grep DMA + 00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) + 00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) + 00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) + 00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) + 00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) + 00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) + 00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) + 00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 04) + + +Compilation +------------ + +For builds done with ``make``, the driver compilation is enabled by the +``CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV`` build configuration option. This is +enabled by default in builds for x86 platforms, and disabled in other +configurations. + +For builds using ``meson`` and ``ninja``, the driver will be built when the +target platform is x86-based. diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst index 3da266705..720c13f8b 100644 --- a/doc/guides/rel_notes/release_19_08.rst +++ b/doc/guides/rel_notes/release_19_08.rst @@ -1,6 +1,8 @@ .. SPDX-License-Identifier: BSD-3-Clause Copyright 2019 The DPDK contributors +.. include:: + DPDK Release 19.08 ================== @@ -99,6 +101,15 @@ New Features Updated ``librte_telemetry`` to fetch the global metrics from the ``librte_metrics`` library. +* **Added Intel QuickData Technology PMD** + + The PMD for Intel\ |reg| QuickData Technology, part of + Intel\ |reg| I/O Acceleration Technology `(Intel I/OAT) + `_, + allows data copies to be done by hardware instead + of via software, reducing cycles spent copying large blocks of data in + applications. + Removed Items ------------- diff --git a/drivers/raw/Makefile b/drivers/raw/Makefile index 8e29b4a56..c1b85c8c7 100644 --- a/drivers/raw/Makefile +++ b/drivers/raw/Makefile @@ -10,5 +10,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV) += dpaa2_cmdif DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev +DIRS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile new file mode 100644 index 000000000..7726e310a --- /dev/null +++ b/drivers/raw/ioat/Makefile @@ -0,0 +1,28 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_pmd_ioat_rawdev.a + +# build flags +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) + +LDLIBS += -lrte_eal -lrte_rawdev +LDLIBS += -lrte_pci -lrte_bus_pci + +# library version +LIBABIVER := 1 + +# versioning export map +EXPORT_MAP := rte_pmd_ioat_version.map + +# library source files +SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c + +# export include files +SYMLINK-y-include += rte_ioat_rawdev.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c new file mode 100644 index 000000000..d9fc3091a --- /dev/null +++ b/drivers/raw/ioat/ioat_rawdev.c @@ -0,0 +1,93 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation + */ + +#include +#include + +#include "rte_ioat_rawdev.h" + +/* Dynamic log type identifier */ +int ioat_pmd_logtype; + +static struct rte_pci_driver ioat_pmd_drv; + +#define IOAT_VENDOR_ID 0x8086 +#define IOAT_DEVICE_ID 0x2021 + +#define IOAT_PMD_LOG(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + ioat_pmd_logtype, "%s(): " fmt "\n", __func__, ##args) + +#define IOAT_PMD_DEBUG(fmt, args...) IOAT_PMD_LOG(DEBUG, fmt, ## args) +#define IOAT_PMD_INFO(fmt, args...) IOAT_PMD_LOG(INFO, fmt, ## args) +#define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) +#define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) + +static int +ioat_rawdev_create(const char *name, struct rte_pci_device *dev) +{ + RTE_SET_USED(name); + RTE_SET_USED(dev); + return 0; +} + +static int +ioat_rawdev_destroy(const char *name) +{ + RTE_SET_USED(name); + return 0; +} + +static int +ioat_rawdev_probe(struct rte_pci_driver *drv, struct rte_pci_device *dev) +{ + char name[32]; + int ret = 0; + + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + IOAT_PMD_INFO("Init %s on NUMA node %d", name, dev->device.numa_node); + + dev->device.driver = &drv->driver; + ret = ioat_rawdev_create(name, dev); + return ret; +} + +static int +ioat_rawdev_remove(struct rte_pci_device *dev) +{ + char name[32]; + int ret; + + rte_pci_device_name(&dev->addr, name, sizeof(name)); + + IOAT_PMD_INFO("Closing %s on NUMA node %d", + name, dev->device.numa_node); + + ret = ioat_rawdev_destroy(name); + return ret; +} + +static const struct rte_pci_id pci_id_ioat_map[] = { + { RTE_PCI_DEVICE(IOAT_VENDOR_ID, IOAT_DEVICE_ID) }, + { .vendor_id = 0, /* sentinel */ }, +}; + +static struct rte_pci_driver ioat_pmd_drv = { + .id_table = pci_id_ioat_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC | + RTE_PCI_DRV_IOVA_AS_VA, + .probe = ioat_rawdev_probe, + .remove = ioat_rawdev_remove, +}; + +RTE_PMD_REGISTER_PCI(IOAT_PMD_RAWDEV_NAME, ioat_pmd_drv); +RTE_PMD_REGISTER_PCI_TABLE(IOAT_PMD_RAWDEV_NAME, pci_id_ioat_map); +RTE_PMD_REGISTER_KMOD_DEP(IOAT_PMD_RAWDEV_NAME, "* igb_uio | uio_pci_generic"); + +RTE_INIT(ioat_pmd_init_log) +{ + ioat_pmd_logtype = rte_log_register(IOAT_PMD_LOG_NAME); + if (ioat_pmd_logtype >= 0) + rte_log_set_level(ioat_pmd_logtype, RTE_LOG_INFO); +} diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build new file mode 100644 index 000000000..ba7620a68 --- /dev/null +++ b/drivers/raw/ioat/meson.build @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2019 Intel Corporation + +build = dpdk_conf.has('RTE_ARCH_X86') +sources = files('ioat_rawdev.c') +deps += ['rawdev', 'bus_pci'] + +install_headers('rte_ioat_rawdev.h') diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h new file mode 100644 index 000000000..e77406403 --- /dev/null +++ b/drivers/raw/ioat/rte_ioat_rawdev.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation + */ + +#ifndef _RTE_IOAT_RAWDEV_H_ +#define _RTE_IOAT_RAWDEV_H_ + +/** + * @file rte_ioat_rawdev.h + * + * Definitions for using the ioat rawdev device driver + * + * @warning + * @b EXPERIMENTAL: these structures and APIs may change without prior notice + */ + +/** Name of the device driver */ +#define IOAT_PMD_RAWDEV_NAME rawdev_ioat +/** String reported as the device driver name by rte_rawdev_info_get() */ +#define IOAT_PMD_RAWDEV_NAME_STR "rawdev_ioat" +/** Name used to adjust the log level for this driver */ +#define IOAT_PMD_LOG_NAME "rawdev.ioat" + +#endif diff --git a/drivers/raw/ioat/rte_pmd_ioat_version.map b/drivers/raw/ioat/rte_pmd_ioat_version.map new file mode 100644 index 000000000..9a61188cd --- /dev/null +++ b/drivers/raw/ioat/rte_pmd_ioat_version.map @@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +}; diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build index a61cdccef..2af8a70d4 100644 --- a/drivers/raw/meson.build +++ b/drivers/raw/meson.build @@ -1,7 +1,9 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2018 NXP -drivers = ['skeleton_rawdev', 'dpaa2_cmdif', 'dpaa2_qdma', 'ifpga_rawdev'] +drivers = ['dpaa2_cmdif', 'dpaa2_qdma', + 'ifpga_rawdev', 'ioat', + 'skeleton_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' driver_name_fmt = 'rte_pmd_@0@' diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 81be289a8..2a534796f 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -306,6 +306,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_IFPGA_BUS),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += -lrte_pmd_ioat_rawdev endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS From patchwork Thu Jun 27 10:40:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 55466 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id DB0CF2BB5; Thu, 27 Jun 2019 12:41:09 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 652B31E20 for ; Thu, 27 Jun 2019 12:41:06 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Jun 2019 03:41:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,423,1557212400"; d="scan'208";a="156198717" Received: from silpixa00399126.ir.intel.com (HELO silpixa00399126.ger.corp.intel.com) ([10.237.223.2]) by orsmga008.jf.intel.com with ESMTP; 27 Jun 2019 03:41:04 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: thomas@monjalon.net, jerinj@marvell.com, Bruce Richardson Date: Thu, 27 Jun 2019 11:40:49 +0100 Message-Id: <20190627104055.8244-3-bruce.richardson@intel.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190627104055.8244-1-bruce.richardson@intel.com> References: <20190530212525.40370-1-bruce.richardson@intel.com> <20190627104055.8244-1-bruce.richardson@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v3 2/8] usertools/dpdk-devbind.py: add support for IOAT devices X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" In order to allow binding/unbinding of devices for use by the ioat_rawdev, we need to update the devbind script to add a new class of device, and add device ids for the specific HW instances. Signed-off-by: Bruce Richardson Acked-by: Anatoly Burakov --- V2: rather than adding a DMA section, add to "misc (rawdev)" section where other device types, e.g. ntb can also do. NOTE: this set largely overlaps with the equivalent changes made by the patchset adding NTB support [1]. Since it's unclear which set will be added first, this set is based off the latest head. [1] http://patches.dpdk.org/patch/55127/ --- doc/guides/rawdevs/ioat_rawdev.rst | 11 +++++++++++ usertools/dpdk-devbind.py | 10 ++++++++++ 2 files changed, 21 insertions(+) diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst index 0c612e73a..1a4b0e03e 100644 --- a/doc/guides/rawdevs/ioat_rawdev.rst +++ b/doc/guides/rawdevs/ioat_rawdev.rst @@ -61,3 +61,14 @@ configurations. For builds using ``meson`` and ``ninja``, the driver will be built when the target platform is x86-based. + +Device Setup +------------- + +The Intel\ |reg| QuickData Technology HW devices will need to be bound to a +user-space IO driver for use. The script ``dpdk-devbind.py`` script +included with DPDK can be used to view the state of the devices and to bind +them to a suitable DPDK-supported kernel driver. When querying the status +of the devices, they will appear under the category of "Misc (rawdev) +devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be +used to see the state of those devices alone. diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index 9e79f0d28..5c1cd3548 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -36,11 +36,17 @@ octeontx2_npa = {'Class': '08', 'Vendor': '177d', 'Device': 'a0fb,a0fc', 'SVendor': None, 'SDevice': None} +intel_ioat_bdw = {'Class': '08', 'Vendor': '8086', 'Device': '6f20,6f21,6f22,6f23,6f24,6f25,6f26,6f27,6f2e,6f2f', + 'SVendor': None, 'SDevice': None} +intel_ioat_skx = {'Class': '08', 'Vendor': '8086', 'Device': '2021', + 'SVendor': None, 'SDevice': None} + network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] +misc_devices = [intel_ioat_bdw, intel_ioat_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties @@ -595,6 +601,8 @@ def show_status(): if status_dev == "compress" or status_dev == "all": show_device_status(compress_devices , "Compress") + if status_dev == "misc" or status_dev == "all": + show_device_status(misc_devices, "Misc (rawdev)") def parse_args(): '''Parses the command-line arguments given by the user and takes the @@ -670,6 +678,7 @@ def do_arg_actions(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) show_status() @@ -690,6 +699,7 @@ def main(): get_device_details(eventdev_devices) get_device_details(mempool_devices) get_device_details(compress_devices) + get_device_details(misc_devices) do_arg_actions() if __name__ == "__main__": From patchwork Thu Jun 27 10:40:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 55467 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4C3A32BF2; Thu, 27 Jun 2019 12:41:12 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id E69762B98 for ; Thu, 27 Jun 2019 12:41:08 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Jun 2019 03:41:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,423,1557212400"; d="scan'208";a="156198725" Received: from silpixa00399126.ir.intel.com (HELO silpixa00399126.ger.corp.intel.com) ([10.237.223.2]) by orsmga008.jf.intel.com with ESMTP; 27 Jun 2019 03:41:07 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: thomas@monjalon.net, jerinj@marvell.com, Bruce Richardson Date: Thu, 27 Jun 2019 11:40:50 +0100 Message-Id: <20190627104055.8244-4-bruce.richardson@intel.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190627104055.8244-1-bruce.richardson@intel.com> References: <20190530212525.40370-1-bruce.richardson@intel.com> <20190627104055.8244-1-bruce.richardson@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v3 3/8] raw/ioat: add register definition file X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add in the list of registers for the device. File is taken from the SPDK project: https://github.com/spdk/spdk/blob/master/include/spdk/ioat_spec.h Signed-off-by: Bruce Richardson Acked-by: Anatoly Burakov --- drivers/raw/ioat/Makefile | 1 + drivers/raw/ioat/meson.build | 3 +- drivers/raw/ioat/rte_ioat_spec.h | 301 +++++++++++++++++++++++++++++++ 3 files changed, 304 insertions(+), 1 deletion(-) create mode 100644 drivers/raw/ioat/rte_ioat_spec.h diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile index 7726e310a..1e10938f3 100644 --- a/drivers/raw/ioat/Makefile +++ b/drivers/raw/ioat/Makefile @@ -24,5 +24,6 @@ SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c # export include files SYMLINK-y-include += rte_ioat_rawdev.h +SYMLINK-y-include += rte_ioat_spec.h include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build index ba7620a68..ca23e23fc 100644 --- a/drivers/raw/ioat/meson.build +++ b/drivers/raw/ioat/meson.build @@ -5,4 +5,5 @@ build = dpdk_conf.has('RTE_ARCH_X86') sources = files('ioat_rawdev.c') deps += ['rawdev', 'bus_pci'] -install_headers('rte_ioat_rawdev.h') +install_headers('rte_ioat_rawdev.h', + 'rte_ioat_spec.h') diff --git a/drivers/raw/ioat/rte_ioat_spec.h b/drivers/raw/ioat/rte_ioat_spec.h new file mode 100644 index 000000000..305e36ded --- /dev/null +++ b/drivers/raw/ioat/rte_ioat_spec.h @@ -0,0 +1,301 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) Intel Corporation + */ + +/** + * \file + * I/OAT specification definitions + * + * Taken from ioat_spec.h from SPDK project, with prefix renames and + * other minor changes. + */ + +#ifndef RTE_IOAT_SPEC_H +#define RTE_IOAT_SPEC_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include + +#define RTE_IOAT_PCI_CHANERR_INT_OFFSET 0x180 + +#define RTE_IOAT_INTRCTRL_MASTER_INT_EN 0x01 + +#define RTE_IOAT_VER_3_0 0x30 +#define RTE_IOAT_VER_3_3 0x33 + +/* DMA Channel Registers */ +#define RTE_IOAT_CHANCTRL_CHANNEL_PRIORITY_MASK 0xF000 +#define RTE_IOAT_CHANCTRL_COMPL_DCA_EN 0x0200 +#define RTE_IOAT_CHANCTRL_CHANNEL_IN_USE 0x0100 +#define RTE_IOAT_CHANCTRL_DESCRIPTOR_ADDR_SNOOP_CONTROL 0x0020 +#define RTE_IOAT_CHANCTRL_ERR_INT_EN 0x0010 +#define RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN 0x0008 +#define RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 +#define RTE_IOAT_CHANCTRL_INT_REARM 0x0001 + +/* DMA Channel Capabilities */ +#define RTE_IOAT_DMACAP_PB (1 << 0) +#define RTE_IOAT_DMACAP_DCA (1 << 4) +#define RTE_IOAT_DMACAP_BFILL (1 << 6) +#define RTE_IOAT_DMACAP_XOR (1 << 8) +#define RTE_IOAT_DMACAP_PQ (1 << 9) +#define RTE_IOAT_DMACAP_DMA_DIF (1 << 10) + +struct rte_ioat_registers { + uint8_t chancnt; + uint8_t xfercap; + uint8_t genctrl; + uint8_t intrctrl; + uint32_t attnstatus; + uint8_t cbver; /* 0x08 */ + uint8_t reserved4[0x3]; /* 0x09 */ + uint16_t intrdelay; /* 0x0C */ + uint16_t cs_status; /* 0x0E */ + uint32_t dmacapability; /* 0x10 */ + uint8_t reserved5[0x6C]; /* 0x14 */ + uint16_t chanctrl; /* 0x80 */ + uint8_t reserved6[0x2]; /* 0x82 */ + uint8_t chancmd; /* 0x84 */ + uint8_t reserved3[1]; /* 0x85 */ + uint16_t dmacount; /* 0x86 */ + uint64_t chansts; /* 0x88 */ + uint64_t chainaddr; /* 0x90 */ + uint64_t chancmp; /* 0x98 */ + uint8_t reserved2[0x8]; /* 0xA0 */ + uint32_t chanerr; /* 0xA8 */ + uint32_t chanerrmask; /* 0xAC */ +} __attribute__((packed)); + +#define RTE_IOAT_CHANCMD_RESET 0x20 +#define RTE_IOAT_CHANCMD_SUSPEND 0x04 + +#define RTE_IOAT_CHANSTS_STATUS 0x7ULL +#define RTE_IOAT_CHANSTS_ACTIVE 0x0 +#define RTE_IOAT_CHANSTS_IDLE 0x1 +#define RTE_IOAT_CHANSTS_SUSPENDED 0x2 +#define RTE_IOAT_CHANSTS_HALTED 0x3 +#define RTE_IOAT_CHANSTS_ARMED 0x4 + +#define RTE_IOAT_CHANSTS_UNAFFILIATED_ERROR 0x8ULL +#define RTE_IOAT_CHANSTS_SOFT_ERROR 0x10ULL + +#define RTE_IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK (~0x3FULL) + +#define RTE_IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ + +struct rte_ioat_generic_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t reserved2: 1; + uint32_t src_page_break: 1; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t op_specific[4]; +}; + +struct rte_ioat_dma_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t null: 1; + uint32_t src_page_break: 1; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define RTE_IOAT_OP_COPY 0x00 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t reserved2; + uint64_t user1; + uint64_t user2; +}; + +struct rte_ioat_fill_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t reserved: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t reserved2: 2; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t reserved3: 15; +#define RTE_IOAT_OP_FILL 0x01 + uint32_t op: 8; + } control; + } u; + uint64_t src_data; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t next_dest_addr; + uint64_t user1; + uint64_t user2; +}; + +struct rte_ioat_xor_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define RTE_IOAT_OP_XOR 0x87 +#define RTE_IOAT_OP_XOR_VAL 0x88 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint64_t src_addr4; + uint64_t src_addr5; +}; + +struct rte_ioat_xor_ext_hw_desc { + uint64_t src_addr6; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t next; + uint64_t reserved[4]; +}; + +struct rte_ioat_pq_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 11; +#define RTE_IOAT_OP_PQ 0x89 +#define RTE_IOAT_OP_PQ_VAL 0x8a + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint8_t coef[8]; + uint64_t q_addr; +}; + +struct rte_ioat_pq_ext_hw_desc { + uint64_t src_addr4; + uint64_t src_addr5; + uint64_t src_addr6; + uint64_t next; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t reserved[2]; +}; + +struct rte_ioat_pq_update_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_cnt: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 3; + uint32_t coef: 8; +#define RTE_IOAT_OP_PQ_UP 0x8b + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t p_src; + uint64_t q_src; + uint64_t q_addr; +}; + +struct rte_ioat_raw_hw_desc { + uint64_t field[8]; +}; + +union rte_ioat_hw_desc { + struct rte_ioat_raw_hw_desc raw; + struct rte_ioat_generic_hw_desc generic; + struct rte_ioat_dma_hw_desc dma; + struct rte_ioat_fill_hw_desc fill; + struct rte_ioat_xor_hw_desc xor_desc; + struct rte_ioat_xor_ext_hw_desc xor_ext; + struct rte_ioat_pq_hw_desc pq; + struct rte_ioat_pq_ext_hw_desc pq_ext; + struct rte_ioat_pq_update_hw_desc pq_update; +}; + +#ifdef __cplusplus +} +#endif + +#endif /* RTE_IOAT_SPEC_H */ From patchwork Thu Jun 27 10:40:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 55468 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E5B2C3237; Thu, 27 Jun 2019 12:41:15 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id D887B1E20 for ; Thu, 27 Jun 2019 12:41:10 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Jun 2019 03:41:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,423,1557212400"; d="scan'208";a="156198729" Received: from silpixa00399126.ir.intel.com (HELO silpixa00399126.ger.corp.intel.com) ([10.237.223.2]) by orsmga008.jf.intel.com with ESMTP; 27 Jun 2019 03:41:09 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: thomas@monjalon.net, jerinj@marvell.com, Bruce Richardson Date: Thu, 27 Jun 2019 11:40:51 +0100 Message-Id: <20190627104055.8244-5-bruce.richardson@intel.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190627104055.8244-1-bruce.richardson@intel.com> References: <20190530212525.40370-1-bruce.richardson@intel.com> <20190627104055.8244-1-bruce.richardson@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v3 4/8] raw/ioat: create device on probe and destroy on release X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add the create/destroy driver functions so that we can actually allocate a rawdev and destroy it when done. No rawdev API functions are actually implemented at this point. Signed-off-by: Bruce Richardson --- doc/guides/rawdevs/ioat_rawdev.rst | 11 ++++ drivers/raw/ioat/ioat_rawdev.c | 93 +++++++++++++++++++++++++++++- drivers/raw/ioat/rte_ioat_rawdev.h | 20 +++++++ 3 files changed, 121 insertions(+), 3 deletions(-) diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst index 1a4b0e03e..4b7fe8a8f 100644 --- a/doc/guides/rawdevs/ioat_rawdev.rst +++ b/doc/guides/rawdevs/ioat_rawdev.rst @@ -72,3 +72,14 @@ them to a suitable DPDK-supported kernel driver. When querying the status of the devices, they will appear under the category of "Misc (rawdev) devices", i.e. the command ``dpdk-devbind.py --status-dev misc`` can be used to see the state of those devices alone. + +Device Probing and Initialization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once bound to a suitable kernel device driver, the HW devices will be found +as part of the PCI scan done at application initialization time. No vdev +parameters need to be passed to create or initialize the device. + +Once probed successfully, the device will appear as a ``rawdev``, that is a +"raw device type" inside DPDK, and can be accessed using APIs from the +``rte_rawdev`` library. diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c index d9fc3091a..d13391dd5 100644 --- a/drivers/raw/ioat/ioat_rawdev.c +++ b/drivers/raw/ioat/ioat_rawdev.c @@ -2,6 +2,7 @@ * Copyright(c) 2019 Intel Corporation */ +#include #include #include @@ -26,15 +27,101 @@ static struct rte_pci_driver ioat_pmd_drv; static int ioat_rawdev_create(const char *name, struct rte_pci_device *dev) { - RTE_SET_USED(name); - RTE_SET_USED(dev); + static const struct rte_rawdev_ops ioat_rawdev_ops = { + }; + + struct rte_rawdev *rawdev = NULL; + struct rte_ioat_rawdev *ioat = NULL; + int ret = 0; + int retry = 0; + + if (!name) { + IOAT_PMD_ERR("Invalid name of the device!"); + ret = -EINVAL; + goto cleanup; + } + + /* Allocate device structure */ + rawdev = rte_rawdev_pmd_allocate(name, sizeof(struct rte_ioat_rawdev), + dev->device.numa_node); + if (rawdev == NULL) { + IOAT_PMD_ERR("Unable to allocate raw device"); + ret = -EINVAL; + goto cleanup; + } + + rawdev->dev_ops = &ioat_rawdev_ops; + rawdev->device = &dev->device; + rawdev->driver_name = dev->device.driver->name; + + ioat = rawdev->dev_private; + ioat->rawdev = rawdev; + ioat->regs = dev->mem_resource[0].addr; + ioat->ring_size = 0; + ioat->desc_ring = NULL; + ioat->status_addr = rte_malloc_virt2iova(ioat) + + offsetof(struct rte_ioat_rawdev, status); + + /* do device initialization - reset and set error behaviour */ + if (ioat->regs->chancnt != 1) + IOAT_PMD_ERR("%s: Channel count == %d\n", __func__, + ioat->regs->chancnt); + + if (ioat->regs->chanctrl & 0x100) { /* locked by someone else */ + IOAT_PMD_WARN("%s: Channel appears locked\n", __func__); + ioat->regs->chanctrl = 0; + } + + ioat->regs->chancmd = RTE_IOAT_CHANCMD_SUSPEND; + rte_delay_ms(1); + ioat->regs->chancmd = RTE_IOAT_CHANCMD_RESET; + rte_delay_ms(1); + while (ioat->regs->chancmd & RTE_IOAT_CHANCMD_RESET) { + ioat->regs->chainaddr = 0; + rte_delay_ms(1); + if (++retry >= 200) { + IOAT_PMD_ERR("%s: cannot reset device. CHANCMD=0x%"PRIx8", CHANSTS=0x%"PRIx64", CHANERR=0x%"PRIx32"\n", + __func__, + ioat->regs->chancmd, + ioat->regs->chansts, + ioat->regs->chanerr); + ret = -EIO; + } + } + ioat->regs->chanctrl = RTE_IOAT_CHANCTRL_ANY_ERR_ABORT_EN | + RTE_IOAT_CHANCTRL_ERR_COMPLETION_EN; + return 0; + +cleanup: + if (rawdev) + rte_rawdev_pmd_release(rawdev); + + return ret; } static int ioat_rawdev_destroy(const char *name) { - RTE_SET_USED(name); + int ret; + struct rte_rawdev *rdev; + + if (!name) { + IOAT_PMD_ERR("Invalid device name"); + return -EINVAL; + } + + rdev = rte_rawdev_pmd_get_named_dev(name); + if (!rdev) { + IOAT_PMD_ERR("Invalid device name (%s)", name); + return -EINVAL; + } + + /* rte_rawdev_close is called by pmd_release */ + ret = rte_rawdev_pmd_release(rdev); + if (ret) + IOAT_PMD_DEBUG("Device cleanup failed"); + return 0; } diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h index e77406403..c3216a174 100644 --- a/drivers/raw/ioat/rte_ioat_rawdev.h +++ b/drivers/raw/ioat/rte_ioat_rawdev.h @@ -14,6 +14,9 @@ * @b EXPERIMENTAL: these structures and APIs may change without prior notice */ +#include +#include + /** Name of the device driver */ #define IOAT_PMD_RAWDEV_NAME rawdev_ioat /** String reported as the device driver name by rte_rawdev_info_get() */ @@ -21,4 +24,21 @@ /** Name used to adjust the log level for this driver */ #define IOAT_PMD_LOG_NAME "rawdev.ioat" +/** + * @internal + * Structure representing a device instance + */ +struct rte_ioat_rawdev { + struct rte_rawdev *rawdev; + volatile struct rte_ioat_registers *regs; + phys_addr_t status_addr; + phys_addr_t ring_addr; + + unsigned short ring_size; + struct rte_ioat_desc *desc_ring; + + /* to report completions, the device will write status back here */ + volatile uint64_t status __rte_cache_aligned; +}; + #endif From patchwork Thu Jun 27 10:40:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 55469 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 739E7378E; Thu, 27 Jun 2019 12:41:19 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id EC83F2C12 for ; Thu, 27 Jun 2019 12:41:12 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Jun 2019 03:41:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,423,1557212400"; d="scan'208";a="156198738" Received: from silpixa00399126.ir.intel.com (HELO silpixa00399126.ger.corp.intel.com) ([10.237.223.2]) by orsmga008.jf.intel.com with ESMTP; 27 Jun 2019 03:41:11 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: thomas@monjalon.net, jerinj@marvell.com, Bruce Richardson Date: Thu, 27 Jun 2019 11:40:52 +0100 Message-Id: <20190627104055.8244-6-bruce.richardson@intel.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190627104055.8244-1-bruce.richardson@intel.com> References: <20190530212525.40370-1-bruce.richardson@intel.com> <20190627104055.8244-1-bruce.richardson@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v3 5/8] raw/ioat: add device info function X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add in the "info_get" function to the driver, to allow us to query the device. This allows us to have the unit test pick up the presence of supported hardware or not. Signed-off-by: Bruce Richardson --- V2: Test case is placed in driver self-test routine --- app/test/test_rawdev.c | 11 ++++++++++ doc/guides/rawdevs/ioat_rawdev.rst | 34 ++++++++++++++++++++++++++++++ drivers/raw/ioat/ioat_rawdev.c | 11 ++++++++++ drivers/raw/ioat/rte_ioat_rawdev.h | 11 ++++++++++ 4 files changed, 67 insertions(+) diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c index 88549fb61..4db762b4c 100644 --- a/app/test/test_rawdev.c +++ b/app/test/test_rawdev.c @@ -29,6 +29,17 @@ REGISTER_TEST_COMMAND(rawdev_autotest, test_rawdev_selftest_skeleton); static int test_rawdev_selftest_ioat(void) { + const int count = rte_rawdev_count(); + int i; + + for (i = 0; i < count; i++) { + struct rte_rawdev_info info = { .dev_private = NULL }; + if (rte_rawdev_info_get(i, &info) == 0 && + strstr(info.driver_name, "ioat") != NULL) + return 0; + } + + printf("No IOAT rawdev found, skipping tests\n"); return TEST_SKIPPED; } diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst index 4b7fe8a8f..0ce984490 100644 --- a/doc/guides/rawdevs/ioat_rawdev.rst +++ b/doc/guides/rawdevs/ioat_rawdev.rst @@ -83,3 +83,37 @@ parameters need to be passed to create or initialize the device. Once probed successfully, the device will appear as a ``rawdev``, that is a "raw device type" inside DPDK, and can be accessed using APIs from the ``rte_rawdev`` library. + +Using IOAT Rawdev Devices +-------------------------- + +To use the devices from an application, the rawdev API can be used, along +with definitions taken from the device-specific header file +``rte_ioat_rawdev.h``. This header is needed to get the definition of +structure parameters used by some of the rawdev APIs for IOAT rawdev +devices, as well as providing key functions for using the device for memory +copies. + +Getting Device Information +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Basic information about each rawdev device can be got using the +``rte_rawdev_info_get()`` API. For most applications, this API will be +needed to verify that the rawdev in question is of the expected type. For +example, the following code in ``test_ioat_rawdev.c`` is used to identify +the IOAT rawdev device for use for the tests: + +.. code-block:: C + + for (i = 0; i < count && !found; i++) { + struct rte_rawdev_info info = { .dev_private = NULL }; + found = (rte_rawdev_info_get(i, &info) == 0 && + strcmp(info.driver_name, + IOAT_PMD_RAWDEV_NAME_STR) == 0); + } + +When calling the ``rte_rawdev_info_get()`` API for an IOAT rawdev device, +the ``dev_private`` field in the ``rte_rawdev_info`` struct should either +be NULL, or else be set to point to a structure of type +``rte_ioat_rawdev_config``, in which case the size of the configured device +input ring will be returned in that structure. diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c index d13391dd5..08e7586c6 100644 --- a/drivers/raw/ioat/ioat_rawdev.c +++ b/drivers/raw/ioat/ioat_rawdev.c @@ -24,10 +24,21 @@ static struct rte_pci_driver ioat_pmd_drv; #define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) #define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) +static void +ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info) +{ + struct rte_ioat_rawdev_config *cfg = dev_info; + struct rte_ioat_rawdev *ioat = dev->dev_private; + + if (cfg != NULL) + cfg->ring_size = ioat->ring_size; +} + static int ioat_rawdev_create(const char *name, struct rte_pci_device *dev) { static const struct rte_rawdev_ops ioat_rawdev_ops = { + .dev_info_get = ioat_dev_info_get, }; struct rte_rawdev *rawdev = NULL; diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h index c3216a174..7e0d72ca3 100644 --- a/drivers/raw/ioat/rte_ioat_rawdev.h +++ b/drivers/raw/ioat/rte_ioat_rawdev.h @@ -24,6 +24,17 @@ /** Name used to adjust the log level for this driver */ #define IOAT_PMD_LOG_NAME "rawdev.ioat" +/** + * Configuration structure for an ioat rawdev instance + * + * This structure is to be passed as the ".dev_private" parameter when + * calling the rte_rawdev_get_info() and rte_rawdev_configure() APIs on + * an ioat rawdev instance. + */ +struct rte_ioat_rawdev_config { + unsigned short ring_size; +}; + /** * @internal * Structure representing a device instance From patchwork Thu Jun 27 10:40:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 55470 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id AC3DD37B4; Thu, 27 Jun 2019 12:41:21 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 0E7342C4F for ; Thu, 27 Jun 2019 12:41:14 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Jun 2019 03:41:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,423,1557212400"; d="scan'208";a="156198747" Received: from silpixa00399126.ir.intel.com (HELO silpixa00399126.ger.corp.intel.com) ([10.237.223.2]) by orsmga008.jf.intel.com with ESMTP; 27 Jun 2019 03:41:13 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: thomas@monjalon.net, jerinj@marvell.com, Bruce Richardson Date: Thu, 27 Jun 2019 11:40:53 +0100 Message-Id: <20190627104055.8244-7-bruce.richardson@intel.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190627104055.8244-1-bruce.richardson@intel.com> References: <20190530212525.40370-1-bruce.richardson@intel.com> <20190627104055.8244-1-bruce.richardson@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v3 6/8] raw/ioat: add configure, start and stop functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Allow initializing a driver instance. Include selftest to validate these functions. Signed-off-by: Bruce Richardson --- V3: don't add a new descriptor format struct, reuse from rte_ioat_spec.h V2: test cases placed in self-test routine --- app/test/test_rawdev.c | 2 +- doc/guides/rawdevs/ioat_rawdev.rst | 32 ++++++++++++ drivers/raw/ioat/Makefile | 1 + drivers/raw/ioat/ioat_rawdev.c | 78 +++++++++++++++++++++++++++++ drivers/raw/ioat/ioat_rawdev_test.c | 41 +++++++++++++++ drivers/raw/ioat/meson.build | 3 +- drivers/raw/ioat/rte_ioat_rawdev.h | 4 +- 7 files changed, 158 insertions(+), 3 deletions(-) create mode 100644 drivers/raw/ioat/ioat_rawdev_test.c diff --git a/app/test/test_rawdev.c b/app/test/test_rawdev.c index 4db762b4c..731e51717 100644 --- a/app/test/test_rawdev.c +++ b/app/test/test_rawdev.c @@ -36,7 +36,7 @@ test_rawdev_selftest_ioat(void) struct rte_rawdev_info info = { .dev_private = NULL }; if (rte_rawdev_info_get(i, &info) == 0 && strstr(info.driver_name, "ioat") != NULL) - return 0; + return rte_rawdev_selftest(i); } printf("No IOAT rawdev found, skipping tests\n"); diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst index 0ce984490..9ab97e2aa 100644 --- a/doc/guides/rawdevs/ioat_rawdev.rst +++ b/doc/guides/rawdevs/ioat_rawdev.rst @@ -117,3 +117,35 @@ the ``dev_private`` field in the ``rte_rawdev_info`` struct should either be NULL, or else be set to point to a structure of type ``rte_ioat_rawdev_config``, in which case the size of the configured device input ring will be returned in that structure. + +Device Configuration +~~~~~~~~~~~~~~~~~~~~~ + +Configuring an IOAT rawdev device is done using the +``rte_rawdev_configure()`` API, which takes the same structure parameters +as the, previously referenced, ``rte_rawdev_info_get()`` API. The main +difference is that, because the parameter is used as input rather than +output, the ``dev_private`` structure element cannot be NULL, and must +point to a valid ``rte_ioat_rawdev_config`` structure, containing the ring +size to be used by the device. The ring size must be a power of two, +between 64 and 4096. + +The following code shows how the device is configured in +``test_ioat_rawdev.c``: + +.. code-block:: C + + #define IOAT_TEST_RINGSIZE 512 + struct rte_ioat_rawdev_config p = { .ring_size = -1 }; + struct rte_rawdev_info info = { .dev_private = &p }; + + /* ... */ + + p.ring_size = IOAT_TEST_RINGSIZE; + if (rte_rawdev_configure(dev_id, &info) != 0) { + printf("Error with rte_rawdev_configure()\n"); + return -1; + } + +Once configured, the device can then be made ready for use by calling the +``rte_rawdev_start()`` API. diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile index 1e10938f3..b1af9c666 100644 --- a/drivers/raw/ioat/Makefile +++ b/drivers/raw/ioat/Makefile @@ -21,6 +21,7 @@ EXPORT_MAP := rte_pmd_ioat_version.map # library source files SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat_rawdev_test.c # export include files SYMLINK-y-include += rte_ioat_rawdev.h diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c index 08e7586c6..fd1653da2 100644 --- a/drivers/raw/ioat/ioat_rawdev.c +++ b/drivers/raw/ioat/ioat_rawdev.c @@ -24,6 +24,78 @@ static struct rte_pci_driver ioat_pmd_drv; #define IOAT_PMD_ERR(fmt, args...) IOAT_PMD_LOG(ERR, fmt, ## args) #define IOAT_PMD_WARN(fmt, args...) IOAT_PMD_LOG(WARNING, fmt, ## args) +#define DESC_SZ sizeof(struct rte_ioat_generic_hw_desc) +#define COMPLETION_SZ sizeof(__m128i) + +static int +ioat_dev_configure(const struct rte_rawdev *dev, rte_rawdev_obj_t config) +{ + struct rte_ioat_rawdev_config *params = config; + struct rte_ioat_rawdev *ioat = dev->dev_private; + unsigned short i; + + if (dev->started) + return -EBUSY; + + if (params == NULL) + return -EINVAL; + + if (params->ring_size > 4096 || params->ring_size < 64 || + !rte_is_power_of_2(params->ring_size)) + return -EINVAL; + + ioat->ring_size = params->ring_size; + if (ioat->desc_ring != NULL) { + rte_free(ioat->desc_ring); + ioat->desc_ring = NULL; + } + + /* allocate one block of memory for both descriptors + * and completion handles. + */ + ioat->desc_ring = rte_zmalloc_socket(NULL, + (DESC_SZ + COMPLETION_SZ) * ioat->ring_size, + 0, /* alignment, default to 64Byte */ + dev->device->numa_node); + if (ioat->desc_ring == NULL) + return -ENOMEM; + ioat->hdls = (void *)&ioat->desc_ring[ioat->ring_size]; + + ioat->ring_addr = rte_malloc_virt2iova(ioat->desc_ring); + + /* configure descriptor ring - each one points to next */ + for (i = 0; i < ioat->ring_size; i++) { + ioat->desc_ring[i].next = ioat->ring_addr + + (((i + 1) % ioat->ring_size) * DESC_SZ); + } + + return 0; +} + +static int +ioat_dev_start(struct rte_rawdev *dev) +{ + struct rte_ioat_rawdev *ioat = dev->dev_private; + + if (ioat->ring_size == 0 || ioat->desc_ring == NULL) + return -EBUSY; + + /* inform hardware of where the descriptor ring is */ + ioat->regs->chainaddr = ioat->ring_addr; + /* inform hardware of where to write the status/completions */ + ioat->regs->chancmp = ioat->status_addr; + + /* prime the status register to be set to the last element */ + ioat->status = ioat->ring_addr + ((ioat->ring_size - 1) * DESC_SZ); + return 0; +} + +static void +ioat_dev_stop(struct rte_rawdev *dev) +{ + RTE_SET_USED(dev); +} + static void ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info) { @@ -34,11 +106,17 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info) cfg->ring_size = ioat->ring_size; } +int ioat_rawdev_test(uint16_t dev_id); + static int ioat_rawdev_create(const char *name, struct rte_pci_device *dev) { static const struct rte_rawdev_ops ioat_rawdev_ops = { + .dev_configure = ioat_dev_configure, + .dev_start = ioat_dev_start, + .dev_stop = ioat_dev_stop, .dev_info_get = ioat_dev_info_get, + .dev_selftest = ioat_rawdev_test, }; struct rte_rawdev *rawdev = NULL; diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c new file mode 100644 index 000000000..5375da26c --- /dev/null +++ b/drivers/raw/ioat/ioat_rawdev_test.c @@ -0,0 +1,41 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation + */ + +#include "rte_rawdev.h" +#include "rte_ioat_rawdev.h" + +int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */ + +int +ioat_rawdev_test(uint16_t dev_id) +{ +#define IOAT_TEST_RINGSIZE 512 + struct rte_ioat_rawdev_config p = { .ring_size = -1 }; + struct rte_rawdev_info info = { .dev_private = &p }; + + rte_rawdev_info_get(dev_id, &info); + if (p.ring_size != 0) { + printf("Error, initial ring size is non-zero (%d)\n", + (int)p.ring_size); + return -1; + } + + p.ring_size = IOAT_TEST_RINGSIZE; + if (rte_rawdev_configure(dev_id, &info) != 0) { + printf("Error with rte_rawdev_configure()\n"); + return -1; + } + rte_rawdev_info_get(dev_id, &info); + if (p.ring_size != IOAT_TEST_RINGSIZE) { + printf("Error, ring size is not %d (%d)\n", + IOAT_TEST_RINGSIZE, (int)p.ring_size); + return -1; + } + + if (rte_rawdev_start(dev_id) != 0) { + printf("Error with rte_rawdev_start()\n"); + return -1; + } + return 0; +} diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build index ca23e23fc..40fff6654 100644 --- a/drivers/raw/ioat/meson.build +++ b/drivers/raw/ioat/meson.build @@ -2,7 +2,8 @@ # Copyright 2019 Intel Corporation build = dpdk_conf.has('RTE_ARCH_X86') -sources = files('ioat_rawdev.c') +sources = files('ioat_rawdev.c', + 'ioat_rawdev_test.c') deps += ['rawdev', 'bus_pci'] install_headers('rte_ioat_rawdev.h', diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h index 7e0d72ca3..c09fd0791 100644 --- a/drivers/raw/ioat/rte_ioat_rawdev.h +++ b/drivers/raw/ioat/rte_ioat_rawdev.h @@ -14,6 +14,7 @@ * @b EXPERIMENTAL: these structures and APIs may change without prior notice */ +#include #include #include @@ -46,7 +47,8 @@ struct rte_ioat_rawdev { phys_addr_t ring_addr; unsigned short ring_size; - struct rte_ioat_desc *desc_ring; + struct rte_ioat_generic_hw_desc *desc_ring; + __m128i *hdls; /* completion handles for returning to user */ /* to report completions, the device will write status back here */ volatile uint64_t status __rte_cache_aligned; From patchwork Thu Jun 27 10:40:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 55471 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0AC644C6C; Thu, 27 Jun 2019 12:41:24 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 247C9378E for ; Thu, 27 Jun 2019 12:41:16 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Jun 2019 03:41:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,423,1557212400"; d="scan'208";a="156198754" Received: from silpixa00399126.ir.intel.com (HELO silpixa00399126.ger.corp.intel.com) ([10.237.223.2]) by orsmga008.jf.intel.com with ESMTP; 27 Jun 2019 03:41:15 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: thomas@monjalon.net, jerinj@marvell.com, Bruce Richardson Date: Thu, 27 Jun 2019 11:40:54 +0100 Message-Id: <20190627104055.8244-8-bruce.richardson@intel.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190627104055.8244-1-bruce.richardson@intel.com> References: <20190530212525.40370-1-bruce.richardson@intel.com> <20190627104055.8244-1-bruce.richardson@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v3 7/8] raw/ioat: add statistics functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add stats functions to track what is happening in the driver, and put unit tests to check those. Signed-off-by: Bruce Richardson --- V2: test cases moved to self-test routine --- doc/guides/rawdevs/ioat_rawdev.rst | 14 +++++++++ drivers/raw/ioat/ioat_rawdev.c | 44 +++++++++++++++++++++++++++++ drivers/raw/ioat/ioat_rawdev_test.c | 39 +++++++++++++++++++++++++ drivers/raw/ioat/rte_ioat_rawdev.h | 6 ++++ 4 files changed, 103 insertions(+) diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst index 9ab97e2aa..b908f31e0 100644 --- a/doc/guides/rawdevs/ioat_rawdev.rst +++ b/doc/guides/rawdevs/ioat_rawdev.rst @@ -149,3 +149,17 @@ The following code shows how the device is configured in Once configured, the device can then be made ready for use by calling the ``rte_rawdev_start()`` API. + +Querying Device Statistics +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The statistics from the IOAT rawdev device can be got via the xstats +functions in the ``rte_rawdev`` library, i.e. +``rte_rawdev_xstats_names_get()``, ``rte_rawdev_xstats_get()`` and +``rte_rawdev_xstats_by_name_get``. The statistics returned for each device +instance are: + +* ``failed_enqueues`` +* ``successful_enqueues`` +* ``copies_started`` +* ``copies_completed`` diff --git a/drivers/raw/ioat/ioat_rawdev.c b/drivers/raw/ioat/ioat_rawdev.c index fd1653da2..0dde0e820 100644 --- a/drivers/raw/ioat/ioat_rawdev.c +++ b/drivers/raw/ioat/ioat_rawdev.c @@ -4,6 +4,7 @@ #include #include +#include #include #include "rte_ioat_rawdev.h" @@ -106,6 +107,47 @@ ioat_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info) cfg->ring_size = ioat->ring_size; } +static const char * const xstat_names[] = { + "failed_enqueues", "successful_enqueues", + "copies_started", "copies_completed" +}; + +static int +ioat_xstats_get(const struct rte_rawdev *dev, const unsigned int ids[], + uint64_t values[], unsigned int n) +{ + const struct rte_ioat_rawdev *ioat = dev->dev_private; + unsigned int i; + + for (i = 0; i < n; i++) { + switch (ids[i]) { + case 0: values[i] = ioat->enqueue_failed; break; + case 1: values[i] = ioat->enqueued; break; + case 2: values[i] = ioat->started; break; + case 3: values[i] = ioat->completed; break; + default: values[i] = 0; break; + } + } + return n; +} + +static int +ioat_xstats_get_names(const struct rte_rawdev *dev, + struct rte_rawdev_xstats_name *names, + unsigned int size) +{ + unsigned int i; + + RTE_SET_USED(dev); + if (size < RTE_DIM(xstat_names)) + return RTE_DIM(xstat_names); + + for (i = 0; i < RTE_DIM(xstat_names); i++) + strlcpy(names[i].name, xstat_names[i], sizeof(names[i])); + + return RTE_DIM(xstat_names); +} + int ioat_rawdev_test(uint16_t dev_id); static int @@ -116,6 +158,8 @@ ioat_rawdev_create(const char *name, struct rte_pci_device *dev) .dev_start = ioat_dev_start, .dev_stop = ioat_dev_stop, .dev_info_get = ioat_dev_info_get, + .xstats_get = ioat_xstats_get, + .xstats_get_names = ioat_xstats_get_names, .dev_selftest = ioat_rawdev_test, }; diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c index 5375da26c..5098d71ae 100644 --- a/drivers/raw/ioat/ioat_rawdev_test.c +++ b/drivers/raw/ioat/ioat_rawdev_test.c @@ -2,6 +2,7 @@ * Copyright(c) 2019 Intel Corporation */ +#include #include "rte_rawdev.h" #include "rte_ioat_rawdev.h" @@ -13,6 +14,11 @@ ioat_rawdev_test(uint16_t dev_id) #define IOAT_TEST_RINGSIZE 512 struct rte_ioat_rawdev_config p = { .ring_size = -1 }; struct rte_rawdev_info info = { .dev_private = &p }; + struct rte_rawdev_xstats_name *snames = NULL; + uint64_t *stats = NULL; + unsigned int *ids = NULL; + unsigned int nb_xstats; + unsigned int i; rte_rawdev_info_get(dev_id, &info); if (p.ring_size != 0) { @@ -37,5 +43,38 @@ ioat_rawdev_test(uint16_t dev_id) printf("Error with rte_rawdev_start()\n"); return -1; } + + /* allocate memory for xstats names and values */ + nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0); + + snames = malloc(sizeof(*snames) * nb_xstats); + if (snames == NULL) { + printf("Error allocating xstat names memory\n"); + return -1; + } + rte_rawdev_xstats_names_get(dev_id, snames, nb_xstats); + + ids = malloc(sizeof(*ids) * nb_xstats); + if (ids == NULL) { + printf("Error allocating xstat ids memory\n"); + return -1; + } + for (i = 0; i < nb_xstats; i++) + ids[i] = i; + + stats = malloc(sizeof(*stats) * nb_xstats); + if (stats == NULL) { + printf("Error allocating xstat memory\n"); + return -1; + } + + rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats); + for (i = 0; i < nb_xstats; i++) + printf("%s: %"PRIu64" ", snames[i].name, stats[i]); + printf("\n"); + + free(snames); + free(stats); + free(ids); return 0; } diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h index c09fd0791..aa534f9cf 100644 --- a/drivers/raw/ioat/rte_ioat_rawdev.h +++ b/drivers/raw/ioat/rte_ioat_rawdev.h @@ -50,6 +50,12 @@ struct rte_ioat_rawdev { struct rte_ioat_generic_hw_desc *desc_ring; __m128i *hdls; /* completion handles for returning to user */ + /* some statistics for tracking, if added/changed update xstats fns*/ + uint64_t enqueue_failed __rte_cache_aligned; + uint64_t enqueued; + uint64_t started; + uint64_t completed; + /* to report completions, the device will write status back here */ volatile uint64_t status __rte_cache_aligned; }; From patchwork Thu Jun 27 10:40:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 55472 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7E63A4CC0; Thu, 27 Jun 2019 12:41:26 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 282262E8F for ; Thu, 27 Jun 2019 12:41:19 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Jun 2019 03:41:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.63,423,1557212400"; d="scan'208";a="156198759" Received: from silpixa00399126.ir.intel.com (HELO silpixa00399126.ger.corp.intel.com) ([10.237.223.2]) by orsmga008.jf.intel.com with ESMTP; 27 Jun 2019 03:41:17 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: thomas@monjalon.net, jerinj@marvell.com, Bruce Richardson Date: Thu, 27 Jun 2019 11:40:55 +0100 Message-Id: <20190627104055.8244-9-bruce.richardson@intel.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190627104055.8244-1-bruce.richardson@intel.com> References: <20190530212525.40370-1-bruce.richardson@intel.com> <20190627104055.8244-1-bruce.richardson@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v3 8/8] raw/ioat: add local API to perform copies X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add local APIs to trigger data copies, and retrieve handle values once those copies are completed. Included are unit tests to validate the data is copies correctly. Signed-off-by: Bruce Richardson Acked-by: Anatoly Burakov --- V3: updated to use descriptor format in rte_ioat_spec.h V2: test cases moved to self-test routine --- doc/guides/rawdevs/ioat_rawdev.rst | 100 ++++++++++++++++ drivers/raw/ioat/Makefile | 1 + drivers/raw/ioat/ioat_rawdev_test.c | 161 +++++++++++++++++++++++++- drivers/raw/ioat/meson.build | 2 +- drivers/raw/ioat/rte_ioat_rawdev.h | 169 +++++++++++++++++++++++++++- 5 files changed, 428 insertions(+), 5 deletions(-) diff --git a/doc/guides/rawdevs/ioat_rawdev.rst b/doc/guides/rawdevs/ioat_rawdev.rst index b908f31e0..22bb8a22c 100644 --- a/doc/guides/rawdevs/ioat_rawdev.rst +++ b/doc/guides/rawdevs/ioat_rawdev.rst @@ -150,6 +150,106 @@ The following code shows how the device is configured in Once configured, the device can then be made ready for use by calling the ``rte_rawdev_start()`` API. +Performing Data Copies +~~~~~~~~~~~~~~~~~~~~~~~ + +To perform data copies using IOAT rawdev devices, the functions +``rte_ioat_enqueue_copy()`` and ``rte_ioat_do_copies()`` should be used. +Once copies have been completed, the completion will be reported back when +the application calls ``rte_ioat_completed_copies()``. + +The ``rte_ioat_enqueue_copy()`` function enqueues a single copy to the +device ring for copying at a later point. The parameters to that function +include the physical addresses of both the source and destination buffers, +as well as two "handles" to be returned to the user when the copy is +completed. These handles can be arbitrary values, but two are provided so +that the library can track handles for both source and destination on +behalf of the user, e.g. virtual addresses for the buffers, or mbuf +pointers if packet data is being copied. + +While the ``rte_ioat_enqueue_copy()`` function enqueues a copy operation on +the device ring, the copy will not actually be performed until after the +application calls the ``rte_ioat_do_copies()`` function. This function +informs the device hardware of the elements enqueued on the ring, and the +device will begin to process them. It is expected that, for efficiency +reasons, a burst of operations will be enqueued to the device via multiple +enqueue calls between calls to the ``rte_ioat_do_copies()`` function. + +The following code from ``test_ioat_rawdev.c`` demonstrates how to enqueue +a burst of copies to the device and start the hardware processing of them: + +.. code-block:: C + + struct rte_mbuf *srcs[32], *dsts[32]; + unsigned int j; + + for (i = 0; i < RTE_DIM(srcs); i++) { + char *src_data; + + srcs[i] = rte_pktmbuf_alloc(pool); + dsts[i] = rte_pktmbuf_alloc(pool); + srcs[i]->data_len = srcs[i]->pkt_len = length; + dsts[i]->data_len = dsts[i]->pkt_len = length; + src_data = rte_pktmbuf_mtod(srcs[i], char *); + + for (j = 0; j < length; j++) + src_data[j] = rand() & 0xFF; + + if (rte_ioat_enqueue_copy(dev_id, + srcs[i]->buf_iova + srcs[i]->data_off, + dsts[i]->buf_iova + dsts[i]->data_off, + length, + (uintptr_t)srcs[i], + (uintptr_t)dsts[i], + 0 /* nofence */) != 1) { + printf("Error with rte_ioat_enqueue_copy for buffer %u\n", + i); + return -1; + } + } + rte_ioat_do_copies(dev_id); + +To retrieve information about completed copies, the API +``rte_ioat_completed_copies()`` should be used. This API will return to the +application a set of completion handles passed in when the relevant copies +were enqueued. + +The following code from ``test_ioat_rawdev.c`` shows the test code +retrieving information about the completed copies and validating the data +is correct before freeing the data buffers using the returned handles: + +.. code-block:: C + + if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src, + (void *)completed_dst) != RTE_DIM(srcs)) { + printf("Error with rte_ioat_completed_copies\n"); + return -1; + } + for (i = 0; i < RTE_DIM(srcs); i++) { + char *src_data, *dst_data; + + if (completed_src[i] != srcs[i]) { + printf("Error with source pointer %u\n", i); + return -1; + } + if (completed_dst[i] != dsts[i]) { + printf("Error with dest pointer %u\n", i); + return -1; + } + + src_data = rte_pktmbuf_mtod(srcs[i], char *); + dst_data = rte_pktmbuf_mtod(dsts[i], char *); + for (j = 0; j < length; j++) + if (src_data[j] != dst_data[j]) { + printf("Error with copy of packet %u, byte %u\n", + i, j); + return -1; + } + rte_pktmbuf_free(srcs[i]); + rte_pktmbuf_free(dsts[i]); + } + + Querying Device Statistics ~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/drivers/raw/ioat/Makefile b/drivers/raw/ioat/Makefile index b1af9c666..32f079845 100644 --- a/drivers/raw/ioat/Makefile +++ b/drivers/raw/ioat/Makefile @@ -12,6 +12,7 @@ CFLAGS += $(WERROR_FLAGS) LDLIBS += -lrte_eal -lrte_rawdev LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_mbuf -lrte_mempool # library version LIBABIVER := 1 diff --git a/drivers/raw/ioat/ioat_rawdev_test.c b/drivers/raw/ioat/ioat_rawdev_test.c index 5098d71ae..8e57cfd7c 100644 --- a/drivers/raw/ioat/ioat_rawdev_test.c +++ b/drivers/raw/ioat/ioat_rawdev_test.c @@ -2,12 +2,139 @@ * Copyright(c) 2019 Intel Corporation */ +#include #include +#include #include "rte_rawdev.h" #include "rte_ioat_rawdev.h" int ioat_rawdev_test(uint16_t dev_id); /* pre-define to keep compiler happy */ +static struct rte_mempool *pool; + +static int +test_enqueue_copies(int dev_id) +{ + const unsigned int length = 1024; + unsigned int i; + + do { + struct rte_mbuf *src, *dst; + char *src_data, *dst_data; + struct rte_mbuf *completed[2] = {0}; + + /* test doing a single copy */ + src = rte_pktmbuf_alloc(pool); + dst = rte_pktmbuf_alloc(pool); + src->data_len = src->pkt_len = length; + dst->data_len = dst->pkt_len = length; + src_data = rte_pktmbuf_mtod(src, char *); + dst_data = rte_pktmbuf_mtod(dst, char *); + + for (i = 0; i < length; i++) + src_data[i] = rand() & 0xFF; + + if (rte_ioat_enqueue_copy(dev_id, + src->buf_iova + src->data_off, + dst->buf_iova + dst->data_off, + length, + (uintptr_t)src, + (uintptr_t)dst, + 0 /* no fence */) != 1) { + printf("Error with rte_ioat_enqueue_copy\n"); + return -1; + } + rte_ioat_do_copies(dev_id); + usleep(10); + + if (rte_ioat_completed_copies(dev_id, 1, (void *)&completed[0], + (void *)&completed[1]) != 1) { + printf("Error with rte_ioat_completed_copies\n"); + return -1; + } + if (completed[0] != src || completed[1] != dst) { + printf("Error with completions: got (%p, %p), not (%p,%p)\n", + completed[0], completed[1], src, dst); + return -1; + } + + for (i = 0; i < length; i++) + if (dst_data[i] != src_data[i]) { + printf("Data mismatch at char %u\n", i); + return -1; + } + rte_pktmbuf_free(src); + rte_pktmbuf_free(dst); + } while (0); + + /* test doing multiple copies */ + do { + struct rte_mbuf *srcs[32], *dsts[32]; + struct rte_mbuf *completed_src[64]; + struct rte_mbuf *completed_dst[64]; + unsigned int j; + + for (i = 0; i < RTE_DIM(srcs); i++) { + char *src_data; + + srcs[i] = rte_pktmbuf_alloc(pool); + dsts[i] = rte_pktmbuf_alloc(pool); + srcs[i]->data_len = srcs[i]->pkt_len = length; + dsts[i]->data_len = dsts[i]->pkt_len = length; + src_data = rte_pktmbuf_mtod(srcs[i], char *); + + for (j = 0; j < length; j++) + src_data[j] = rand() & 0xFF; + + if (rte_ioat_enqueue_copy(dev_id, + srcs[i]->buf_iova + srcs[i]->data_off, + dsts[i]->buf_iova + dsts[i]->data_off, + length, + (uintptr_t)srcs[i], + (uintptr_t)dsts[i], + 0 /* nofence */) != 1) { + printf("Error with rte_ioat_enqueue_copy for buffer %u\n", + i); + return -1; + } + } + rte_ioat_do_copies(dev_id); + usleep(100); + + if (rte_ioat_completed_copies(dev_id, 64, (void *)completed_src, + (void *)completed_dst) != RTE_DIM(srcs)) { + printf("Error with rte_ioat_completed_copies\n"); + return -1; + } + for (i = 0; i < RTE_DIM(srcs); i++) { + char *src_data, *dst_data; + + if (completed_src[i] != srcs[i]) { + printf("Error with source pointer %u\n", i); + return -1; + } + if (completed_dst[i] != dsts[i]) { + printf("Error with dest pointer %u\n", i); + return -1; + } + + src_data = rte_pktmbuf_mtod(srcs[i], char *); + dst_data = rte_pktmbuf_mtod(dsts[i], char *); + for (j = 0; j < length; j++) + if (src_data[j] != dst_data[j]) { + printf("Error with copy of packet %u, byte %u\n", + i, j); + return -1; + } + rte_pktmbuf_free(srcs[i]); + rte_pktmbuf_free(dsts[i]); + } + + } while (0); + + return 0; +} + int ioat_rawdev_test(uint16_t dev_id) { @@ -44,6 +171,17 @@ ioat_rawdev_test(uint16_t dev_id) return -1; } + pool = rte_pktmbuf_pool_create("TEST_IOAT_POOL", + 256, /* n == num elements */ + 32, /* cache size */ + 0, /* priv size */ + 2048, /* data room size */ + info.socket_id); + if (pool == NULL) { + printf("Error with mempool creation\n"); + return -1; + } + /* allocate memory for xstats names and values */ nb_xstats = rte_rawdev_xstats_names_get(dev_id, NULL, 0); @@ -68,13 +206,30 @@ ioat_rawdev_test(uint16_t dev_id) return -1; } - rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats); - for (i = 0; i < nb_xstats; i++) - printf("%s: %"PRIu64" ", snames[i].name, stats[i]); + /* run the test cases */ + for (i = 0; i < 100; i++) { + unsigned int j; + + if (test_enqueue_copies(dev_id) != 0) + goto err; + + rte_rawdev_xstats_get(dev_id, ids, stats, nb_xstats); + for (j = 0; j < nb_xstats; j++) + printf("%s: %"PRIu64" ", snames[j].name, stats[j]); + printf("\r"); + } printf("\n"); + rte_mempool_free(pool); free(snames); free(stats); free(ids); return 0; + +err: + rte_mempool_free(pool); + free(snames); + free(stats); + free(ids); + return -1; } diff --git a/drivers/raw/ioat/meson.build b/drivers/raw/ioat/meson.build index 40fff6654..247ff88bf 100644 --- a/drivers/raw/ioat/meson.build +++ b/drivers/raw/ioat/meson.build @@ -4,7 +4,7 @@ build = dpdk_conf.has('RTE_ARCH_X86') sources = files('ioat_rawdev.c', 'ioat_rawdev_test.c') -deps += ['rawdev', 'bus_pci'] +deps += ['rawdev', 'bus_pci', 'mbuf'] install_headers('rte_ioat_rawdev.h', 'rte_ioat_spec.h') diff --git a/drivers/raw/ioat/rte_ioat_rawdev.h b/drivers/raw/ioat/rte_ioat_rawdev.h index aa534f9cf..5aeecb26a 100644 --- a/drivers/raw/ioat/rte_ioat_rawdev.h +++ b/drivers/raw/ioat/rte_ioat_rawdev.h @@ -15,7 +15,9 @@ */ #include +#include #include +#include #include /** Name of the device driver */ @@ -50,6 +52,10 @@ struct rte_ioat_rawdev { struct rte_ioat_generic_hw_desc *desc_ring; __m128i *hdls; /* completion handles for returning to user */ + + unsigned short next_read; + unsigned short next_write; + /* some statistics for tracking, if added/changed update xstats fns*/ uint64_t enqueue_failed __rte_cache_aligned; uint64_t enqueued; @@ -60,4 +66,165 @@ struct rte_ioat_rawdev { volatile uint64_t status __rte_cache_aligned; }; -#endif +/** + * Enqueue a copy operation onto the ioat device + * + * This queues up a copy operation to be performed by hardware, but does not + * trigger hardware to begin that operation. + * + * @param dev_id + * The rawdev device id of the ioat instance + * @param src + * The physical address of the source buffer + * @param dst + * The physical address of the destination buffer + * @param length + * The length of the data to be copied + * @param src_hdl + * An opaque handle for the source data, to be returned when this operation + * has been completed and the user polls for the completion details + * @param dst_hdl + * An opaque handle for the destination data, to be returned when this + * operation has been completed and the user polls for the completion details + * @param fence + * A flag parameter indicating that hardware should not begin to perform any + * subsequently enqueued copy operations until after this operation has + * completed + * @return + * Number of operations enqueued, either 0 or 1 + */ +static inline int +rte_ioat_enqueue_copy(int dev_id, phys_addr_t src, phys_addr_t dst, + unsigned int length, uintptr_t src_hdl, uintptr_t dst_hdl, + int fence) +{ + struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private; + unsigned short read = ioat->next_read; + unsigned short write = ioat->next_write; + unsigned short mask = ioat->ring_size - 1; + unsigned short space = mask + read - write; + struct rte_ioat_generic_hw_desc *desc; + + if (space == 0) { + ioat->enqueue_failed++; + return 0; + } + + ioat->next_write = write + 1; + write &= mask; + + desc = &ioat->desc_ring[write]; + desc->size = length; + /* set descriptor write-back every 16th descriptor */ + desc->u.control_raw = (uint32_t)((!!fence << 4) | (!(write & 0xF)) << 3); + desc->src_addr = src; + desc->dest_addr = dst; + + ioat->hdls[write] = _mm_set_epi64((__m64)((uint64_t)dst_hdl), + (__m64)((uint64_t)src_hdl)); + rte_prefetch0(&ioat->desc_ring[ioat->next_write & mask]); + + ioat->enqueued++; + return 1; +} + +/** + * Trigger hardware to begin performing enqueued copy operations + * + * This API is used to write the "doorbell" to the hardware to trigger it + * to begin the copy operations previously enqueued by rte_ioat_enqueue_copy() + * + * @param dev_id + * The rawdev device id of the ioat instance + */ +static inline void +rte_ioat_do_copies(int dev_id) +{ + struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private; + ioat->desc_ring[(ioat->next_write - 1) & (ioat->ring_size - 1)].u + .control.completion_update = 1; + rte_compiler_barrier(); + ioat->regs->dmacount = ioat->next_write; + ioat->started = ioat->enqueued; +} + +/** + * @internal + * Returns the index of the last completed operation. + */ +static inline int +rte_ioat_get_last_completed(struct rte_ioat_rawdev *ioat, int *error) +{ + uint64_t status = ioat->status; + + /* lower 3 bits indicate "transfer status" : active, idle, halted. + * We can ignore bit 0. + */ + *error = status & (RTE_IOAT_CHANSTS_SUSPENDED | RTE_IOAT_CHANSTS_ARMED); + return (status - ioat->ring_addr) >> 6; +} + +/** + * Returns details of copy operations that have been completed + * + * Returns to the caller the user-provided "handles" for the copy operations + * which have been completed by the hardware, and not already returned by + * a previous call to this API. + * + * @param dev_id + * The rawdev device id of the ioat instance + * @param max_copies + * The number of entries which can fit in the src_hdls and dst_hdls + * arrays, i.e. max number of completed operations to report + * @param src_hdls + * Array to hold the source handle parameters of the completed copies + * @param dst_hdls + * Array to hold the destination handle parameters of the completed copies + * @return + * -1 on error, with rte_errno set appropriately. + * Otherwise number of completed operations i.e. number of entries written + * to the src_hdls and dst_hdls array parameters. + */ +static inline int +rte_ioat_completed_copies(int dev_id, uint8_t max_copies, + uintptr_t *src_hdls, uintptr_t *dst_hdls) +{ + struct rte_ioat_rawdev *ioat = rte_rawdevs[dev_id].dev_private; + unsigned short mask = (ioat->ring_size - 1); + unsigned short read = ioat->next_read; + unsigned short end_read, count; + int error; + int i = 0; + + end_read = (rte_ioat_get_last_completed(ioat, &error) + 1) & mask; + count = (end_read - (read & mask)) & mask; + + if (error) { + rte_errno = EIO; + return -1; + } + + if (count > max_copies) + count = max_copies; + + for (; i < count - 1; i += 2, read += 2) { + __m128i hdls0 = _mm_load_si128(&ioat->hdls[read & mask]); + __m128i hdls1 = _mm_load_si128(&ioat->hdls[(read + 1) & mask]); + + _mm_storeu_si128((void *)&src_hdls[i], + _mm_unpacklo_epi64(hdls0, hdls1)); + _mm_storeu_si128((void *)&dst_hdls[i], + _mm_unpackhi_epi64(hdls0, hdls1)); + } + for (; i < count; i++, read++) { + uintptr_t *hdls = (void *)&ioat->hdls[read & mask]; + src_hdls[i] = hdls[0]; + dst_hdls[i] = hdls[1]; + } + + ioat->next_read = read; + ioat->completed += count; + return count; +} + +#endif /* _RTE_IOAT_RAWDEV_H_ */