From patchwork Thu Jul 4 12:29:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Monjalon X-Patchwork-Id: 56084 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8E6501BE31; Thu, 4 Jul 2019 14:30:39 +0200 (CEST) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by dpdk.org (Postfix) with ESMTP id 15FEB1BE2A for ; Thu, 4 Jul 2019 14:30:38 +0200 (CEST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id A19F721550; Thu, 4 Jul 2019 08:30:37 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Thu, 04 Jul 2019 08:30:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=mesmtp; bh=EbX0bXnDj7 T/UoB/AlErGfQWh+Av0BUXDVyLhjCaxIM=; b=r2+sqFiK5nPCyE5yqivgtD6BRz ocIfvWKFs28uQMjXO96u+AKNZAQ340XsZ/mDnJgbJ02k9H2XhKcZsjcbMhQkueDq 9c6X3QG4STkGAjMSYABDhT9l6+gzlCM7uVXrrUYzm6HuTC1pGC+1T3Lu6jIuoqUh B5rZkb/YV5mOJmKF8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=EbX0bXnDj7T/UoB/AlErGfQWh+Av0BUXDVyLhjCaxIM=; b=NxmzsRz2 5xWgemIaJQxZcgP8eqsEORcUExaJjxigAdjt2Xsml+j5NDGePsJUhLwBDIXQ7cVk btRVQei6/8ZQb87cPS4sI5tn5p7/bRtAnWAzo7QXMSQEOODxuvBTkbOveDSkZVsT wQu9mX7rB6/SVJQtYNyXJz9zz6ri64eNw+6O+83Bdd9nIGaDbAyEPFjg2CWSyhQg DkoWBiXBT/mjxTgQpvVljDyLwuqPI33/I4BjT+ADkkow8hoz0piLe4ZaGYkSoUz/ Wm7rbMWXpF/tg3pSbgIqGTjyO5wuLl196YNTD9U5KhlDRTWfw1bOLykFEem7jGZR Z3338/lSO5GzLw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrfedvgdehhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucffoh hmrghinhepshhusgguihhrrdhmkhdpvhgrrhhsrdhmkhdplhhisgdrmhhknecukfhppeej jedrudefgedrvddtfedrudekgeenucfrrghrrghmpehmrghilhhfrhhomhepthhhohhmrg hssehmohhnjhgrlhhonhdrnhgvthenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from xps.monjalon.net (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 576F2380074; Thu, 4 Jul 2019 08:30:36 -0400 (EDT) From: Thomas Monjalon To: dev@dpdk.org, John McNamara , Marko Kovacevic , Xiaoyun Li , Jingjing Wu Cc: Xiaolong Ye Date: Thu, 4 Jul 2019 14:29:56 +0200 Message-Id: <20190704122959.18919-2-thomas@monjalon.net> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190704122959.18919-1-thomas@monjalon.net> References: <20190628025346.31312-1-xiaoyun.li@intel.com> <20190704122959.18919-1-thomas@monjalon.net> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v11 1/4] raw/ntb: introduce NTB raw device driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Xiaoyun Li Introduce rawdev driver support for NTB (Non-transparent Bridge) which can help to connect two separate hosts with each other. Signed-off-by: Xiaoyun Li Acked-by: Jingjing Wu Reviewed-by: Xiaolong Ye --- MAINTAINERS | 6 + config/common_base | 5 + doc/guides/rawdevs/index.rst | 1 + doc/guides/rawdevs/ntb.rst | 22 ++ doc/guides/rel_notes/release_19_08.rst | 6 + drivers/raw/Makefile | 1 + drivers/raw/meson.build | 2 +- drivers/raw/ntb/Makefile | 27 ++ drivers/raw/ntb/meson.build | 7 + drivers/raw/ntb/ntb.c | 488 ++++++++++++++++++++++++ drivers/raw/ntb/ntb.h | 164 ++++++++ drivers/raw/ntb/rte_pmd_ntb_version.map | 4 + mk/rte.app.mk | 1 + 13 files changed, 733 insertions(+), 1 deletion(-) create mode 100644 doc/guides/rawdevs/ntb.rst create mode 100644 drivers/raw/ntb/Makefile create mode 100644 drivers/raw/ntb/meson.build create mode 100644 drivers/raw/ntb/ntb.c create mode 100644 drivers/raw/ntb/ntb.h create mode 100644 drivers/raw/ntb/rte_pmd_ntb_version.map diff --git a/MAINTAINERS b/MAINTAINERS index 605422077..9e6308aa4 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1105,6 +1105,12 @@ M: Nipun Gupta F: drivers/raw/dpaa2_cmdif/ F: doc/guides/rawdevs/dpaa2_cmdif.rst +NTB +M: Xiaoyun Li +M: Jingjing Wu +F: drivers/raw/ntb/ +F: doc/guides/rawdevs/ntb.rst + Packet processing ----------------- diff --git a/config/common_base b/config/common_base index f68107e7a..75e130a43 100644 --- a/config/common_base +++ b/config/common_base @@ -762,6 +762,11 @@ CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y # CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV=y +# +# Compile PMD for NTB raw device +# +CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV=y + # # Compile librte_ring # diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst index 0a21989e4..e19c4ec26 100644 --- a/doc/guides/rawdevs/index.rst +++ b/doc/guides/rawdevs/index.rst @@ -15,3 +15,4 @@ application through rawdev API. dpaa2_qdma ifpga_rawdev ioat_rawdev + ntb diff --git a/doc/guides/rawdevs/ntb.rst b/doc/guides/rawdevs/ntb.rst new file mode 100644 index 000000000..9559ea4b0 --- /dev/null +++ b/doc/guides/rawdevs/ntb.rst @@ -0,0 +1,22 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +NTB Rawdev Driver +================= + +The ``ntb`` rawdev driver provides a non-transparent bridge between two +separate hosts so that they can communicate with each other. Thus, many +user cases can benefit from this, such as fault tolerance and visual +acceleration. + +Build Options +------------- + +- ``CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV`` (default ``y``) + + Toggle compilation of the ``ntb`` driver. + +Limitation +---------- + +- The FIFO hasn't been introduced and will come in 19.11 release. diff --git a/doc/guides/rel_notes/release_19_08.rst b/doc/guides/rel_notes/release_19_08.rst index 21934bf01..8cd90e0c3 100644 --- a/doc/guides/rel_notes/release_19_08.rst +++ b/doc/guides/rel_notes/release_19_08.rst @@ -129,6 +129,12 @@ New Features of via software, reducing cycles spent copying large blocks of data in applications. +* **Introduced NTB PMD.** + + Added a PMD for Intel NTB (Non-transparent Bridge). This PMD implemented + handshake between two separate hosts and can share local memory for peer + host to directly access. + * **Updated telemetry library for global metrics support.** Updated ``librte_telemetry`` to fetch the global metrics from the diff --git a/drivers/raw/Makefile b/drivers/raw/Makefile index c1b85c8c7..84a13b697 100644 --- a/drivers/raw/Makefile +++ b/drivers/raw/Makefile @@ -11,5 +11,6 @@ DIRS-$(CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV) += dpaa2_qdma endif DIRS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += ifpga_rawdev DIRS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += ioat +DIRS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/raw/meson.build b/drivers/raw/meson.build index 2af8a70d4..47f0a1d5f 100644 --- a/drivers/raw/meson.build +++ b/drivers/raw/meson.build @@ -2,7 +2,7 @@ # Copyright 2018 NXP drivers = ['dpaa2_cmdif', 'dpaa2_qdma', - 'ifpga_rawdev', 'ioat', + 'ifpga_rawdev', 'ioat', 'ntb', 'skeleton_rawdev'] std_deps = ['rawdev'] config_flag_fmt = 'RTE_LIBRTE_PMD_@0@_RAWDEV' diff --git a/drivers/raw/ntb/Makefile b/drivers/raw/ntb/Makefile new file mode 100644 index 000000000..ec53edd33 --- /dev/null +++ b/drivers/raw/ntb/Makefile @@ -0,0 +1,27 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_pmd_ntb.a + +CFLAGS += -DALLOW_EXPERIMENTAL_API +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool +LDLIBS += -lrte_pci -lrte_bus_pci +LDLIBS += -lrte_rawdev + +EXPORT_MAP := rte_pmd_ntb_version.map + +LIBABIVER := 1 + +# +# all source are stored in SRCS-y +# +SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb.c + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/raw/ntb/meson.build b/drivers/raw/ntb/meson.build new file mode 100644 index 000000000..0fb44452c --- /dev/null +++ b/drivers/raw/ntb/meson.build @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation. + +deps += ['rawdev', 'mbuf', 'mempool', + 'pci', 'bus_pci'] +sources = files('ntb.c') +allow_experimental_apis = true diff --git a/drivers/raw/ntb/ntb.c b/drivers/raw/ntb/ntb.c new file mode 100644 index 000000000..2ae3cbea0 --- /dev/null +++ b/drivers/raw/ntb/ntb.c @@ -0,0 +1,488 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation. + */ +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ntb.h" + +int ntb_logtype; + +static const struct rte_pci_id pci_id_ntb_map[] = { + { .vendor_id = 0, /* sentinel */ }, +}; + +static void +ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, + uint16_t queue_id __rte_unused, + rte_rawdev_obj_t queue_conf __rte_unused) +{ +} + +static int +ntb_queue_setup(struct rte_rawdev *dev __rte_unused, + uint16_t queue_id __rte_unused, + rte_rawdev_obj_t queue_conf __rte_unused) +{ + return 0; +} + +static int +ntb_queue_release(struct rte_rawdev *dev __rte_unused, + uint16_t queue_id __rte_unused) +{ + return 0; +} + +static uint16_t +ntb_queue_count(struct rte_rawdev *dev) +{ + struct ntb_hw *hw = dev->dev_private; + return hw->queue_pairs; +} + +static int +ntb_enqueue_bufs(struct rte_rawdev *dev, + struct rte_rawdev_buf **buffers, + unsigned int count, + rte_rawdev_obj_t context) +{ + RTE_SET_USED(dev); + RTE_SET_USED(buffers); + RTE_SET_USED(count); + RTE_SET_USED(context); + + return 0; +} + +static int +ntb_dequeue_bufs(struct rte_rawdev *dev, + struct rte_rawdev_buf **buffers, + unsigned int count, + rte_rawdev_obj_t context) +{ + RTE_SET_USED(dev); + RTE_SET_USED(buffers); + RTE_SET_USED(count); + RTE_SET_USED(context); + + return 0; +} + +static void +ntb_dev_info_get(struct rte_rawdev *dev, rte_rawdev_obj_t dev_info) +{ + struct ntb_hw *hw = dev->dev_private; + struct ntb_attr *ntb_attrs = dev_info; + + strncpy(ntb_attrs[NTB_TOPO_ID].name, NTB_TOPO_NAME, NTB_ATTR_NAME_LEN); + switch (hw->topo) { + case NTB_TOPO_B2B_DSD: + strncpy(ntb_attrs[NTB_TOPO_ID].value, "B2B DSD", + NTB_ATTR_VAL_LEN); + break; + case NTB_TOPO_B2B_USD: + strncpy(ntb_attrs[NTB_TOPO_ID].value, "B2B USD", + NTB_ATTR_VAL_LEN); + break; + default: + strncpy(ntb_attrs[NTB_TOPO_ID].value, "Unsupported", + NTB_ATTR_VAL_LEN); + } + + strncpy(ntb_attrs[NTB_LINK_STATUS_ID].name, NTB_LINK_STATUS_NAME, + NTB_ATTR_NAME_LEN); + snprintf(ntb_attrs[NTB_LINK_STATUS_ID].value, NTB_ATTR_VAL_LEN, + "%d", hw->link_status); + + strncpy(ntb_attrs[NTB_SPEED_ID].name, NTB_SPEED_NAME, + NTB_ATTR_NAME_LEN); + snprintf(ntb_attrs[NTB_SPEED_ID].value, NTB_ATTR_VAL_LEN, + "%d", hw->link_speed); + + strncpy(ntb_attrs[NTB_WIDTH_ID].name, NTB_WIDTH_NAME, + NTB_ATTR_NAME_LEN); + snprintf(ntb_attrs[NTB_WIDTH_ID].value, NTB_ATTR_VAL_LEN, + "%d", hw->link_width); + + strncpy(ntb_attrs[NTB_MW_CNT_ID].name, NTB_MW_CNT_NAME, + NTB_ATTR_NAME_LEN); + snprintf(ntb_attrs[NTB_MW_CNT_ID].value, NTB_ATTR_VAL_LEN, + "%d", hw->mw_cnt); + + strncpy(ntb_attrs[NTB_DB_CNT_ID].name, NTB_DB_CNT_NAME, + NTB_ATTR_NAME_LEN); + snprintf(ntb_attrs[NTB_DB_CNT_ID].value, NTB_ATTR_VAL_LEN, + "%d", hw->db_cnt); + + strncpy(ntb_attrs[NTB_SPAD_CNT_ID].name, NTB_SPAD_CNT_NAME, + NTB_ATTR_NAME_LEN); + snprintf(ntb_attrs[NTB_SPAD_CNT_ID].value, NTB_ATTR_VAL_LEN, + "%d", hw->spad_cnt); +} + +static int +ntb_dev_configure(const struct rte_rawdev *dev __rte_unused, + rte_rawdev_obj_t config __rte_unused) +{ + return 0; +} + +static int +ntb_dev_start(struct rte_rawdev *dev) +{ + /* TODO: init queues and start queues. */ + dev->started = 1; + + return 0; +} + +static void +ntb_dev_stop(struct rte_rawdev *dev) +{ + /* TODO: stop rx/tx queues. */ + dev->started = 0; +} + +static int +ntb_dev_close(struct rte_rawdev *dev) +{ + int ret = 0; + + if (dev->started) + ntb_dev_stop(dev); + + /* TODO: free queues. */ + + return ret; +} + +static int +ntb_dev_reset(struct rte_rawdev *rawdev __rte_unused) +{ + return 0; +} + +static int +ntb_attr_set(struct rte_rawdev *dev, const char *attr_name, + uint64_t attr_value) +{ + struct ntb_hw *hw = dev->dev_private; + int index = 0; + + if (dev == NULL || attr_name == NULL) { + NTB_LOG(ERR, "Invalid arguments for setting attributes"); + return -EINVAL; + } + + if (!strncmp(attr_name, NTB_SPAD_USER, NTB_SPAD_USER_LEN)) { + if (hw->ntb_ops->spad_write == NULL) + return -ENOTSUP; + index = atoi(&attr_name[NTB_SPAD_USER_LEN]); + (*hw->ntb_ops->spad_write)(dev, hw->spad_user_list[index], + 1, attr_value); + NTB_LOG(INFO, "Set attribute (%s) Value (%" PRIu64 ")", + attr_name, attr_value); + return 0; + } + + /* Attribute not found. */ + NTB_LOG(ERR, "Attribute not found."); + return -EINVAL; +} + +static int +ntb_attr_get(struct rte_rawdev *dev, const char *attr_name, + uint64_t *attr_value) +{ + struct ntb_hw *hw = dev->dev_private; + int index = 0; + + if (dev == NULL || attr_name == NULL || attr_value == NULL) { + NTB_LOG(ERR, "Invalid arguments for getting attributes"); + return -EINVAL; + } + + if (!strncmp(attr_name, NTB_TOPO_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->topo; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_LINK_STATUS_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->link_status; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_SPEED_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->link_speed; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_WIDTH_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->link_width; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_MW_CNT_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->mw_cnt; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_DB_CNT_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->db_cnt; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_SPAD_CNT_NAME, NTB_ATTR_NAME_LEN)) { + *attr_value = hw->spad_cnt; + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + if (!strncmp(attr_name, NTB_SPAD_USER, NTB_SPAD_USER_LEN)) { + if (hw->ntb_ops->spad_read == NULL) + return -ENOTSUP; + index = atoi(&attr_name[NTB_SPAD_USER_LEN]); + *attr_value = (*hw->ntb_ops->spad_read)(dev, + hw->spad_user_list[index], 0); + NTB_LOG(INFO, "Attribute (%s) Value (%" PRIu64 ")", + attr_name, *attr_value); + return 0; + } + + /* Attribute not found. */ + NTB_LOG(ERR, "Attribute not found."); + return -EINVAL; +} + +static int +ntb_xstats_get(const struct rte_rawdev *dev __rte_unused, + const unsigned int ids[] __rte_unused, + uint64_t values[] __rte_unused, + unsigned int n __rte_unused) +{ + return 0; +} + +static int +ntb_xstats_get_names(const struct rte_rawdev *dev __rte_unused, + struct rte_rawdev_xstats_name *xstats_names __rte_unused, + unsigned int size __rte_unused) +{ + return 0; +} + +static uint64_t +ntb_xstats_get_by_name(const struct rte_rawdev *dev __rte_unused, + const char *name __rte_unused, + unsigned int *id __rte_unused) +{ + return 0; +} + +static int +ntb_xstats_reset(struct rte_rawdev *dev __rte_unused, + const uint32_t ids[] __rte_unused, + uint32_t nb_ids __rte_unused) +{ + return 0; +} + +static const struct rte_rawdev_ops ntb_ops = { + .dev_info_get = ntb_dev_info_get, + .dev_configure = ntb_dev_configure, + .dev_start = ntb_dev_start, + .dev_stop = ntb_dev_stop, + .dev_close = ntb_dev_close, + .dev_reset = ntb_dev_reset, + + .queue_def_conf = ntb_queue_conf_get, + .queue_setup = ntb_queue_setup, + .queue_release = ntb_queue_release, + .queue_count = ntb_queue_count, + + .enqueue_bufs = ntb_enqueue_bufs, + .dequeue_bufs = ntb_dequeue_bufs, + + .attr_get = ntb_attr_get, + .attr_set = ntb_attr_set, + + .xstats_get = ntb_xstats_get, + .xstats_get_names = ntb_xstats_get_names, + .xstats_get_by_name = ntb_xstats_get_by_name, + .xstats_reset = ntb_xstats_reset, +}; + +static int +ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) +{ + struct ntb_hw *hw = dev->dev_private; + int ret; + + hw->pci_dev = pci_dev; + hw->peer_dev_up = 0; + hw->link_status = NTB_LINK_DOWN; + hw->link_speed = NTB_SPEED_NONE; + hw->link_width = NTB_WIDTH_NONE; + + switch (pci_dev->id.device_id) { + default: + NTB_LOG(ERR, "Not supported device."); + return -EINVAL; + } + + if (hw->ntb_ops->ntb_dev_init == NULL) + return -ENOTSUP; + ret = (*hw->ntb_ops->ntb_dev_init)(dev); + if (ret) { + NTB_LOG(ERR, "Unable to init ntb dev."); + return ret; + } + + if (hw->ntb_ops->set_link == NULL) + return -ENOTSUP; + ret = (*hw->ntb_ops->set_link)(dev, 1); + if (ret) + return ret; + + return ret; +} + +static int +ntb_create(struct rte_pci_device *pci_dev, int socket_id) +{ + char name[RTE_RAWDEV_NAME_MAX_LEN]; + struct rte_rawdev *rawdev = NULL; + int ret; + + if (pci_dev == NULL) { + NTB_LOG(ERR, "Invalid pci_dev."); + ret = -EINVAL; + } + + memset(name, 0, sizeof(name)); + snprintf(name, RTE_RAWDEV_NAME_MAX_LEN, "NTB:%x:%02x.%x", + pci_dev->addr.bus, pci_dev->addr.devid, + pci_dev->addr.function); + + NTB_LOG(INFO, "Init %s on NUMA node %d", name, socket_id); + + /* Allocate device structure. */ + rawdev = rte_rawdev_pmd_allocate(name, sizeof(struct ntb_hw), + socket_id); + if (rawdev == NULL) { + NTB_LOG(ERR, "Unable to allocate rawdev."); + ret = -EINVAL; + } + + rawdev->dev_ops = &ntb_ops; + rawdev->device = &pci_dev->device; + rawdev->driver_name = pci_dev->driver->driver.name; + + ret = ntb_init_hw(rawdev, pci_dev); + if (ret < 0) { + NTB_LOG(ERR, "Unable to init ntb hw."); + goto fail; + } + + return ret; + +fail: + if (rawdev) + rte_rawdev_pmd_release(rawdev); + + return ret; +} + +static int +ntb_destroy(struct rte_pci_device *pci_dev) +{ + char name[RTE_RAWDEV_NAME_MAX_LEN]; + struct rte_rawdev *rawdev; + int ret; + + if (pci_dev == NULL) { + NTB_LOG(ERR, "Invalid pci_dev."); + ret = -EINVAL; + return ret; + } + + memset(name, 0, sizeof(name)); + snprintf(name, RTE_RAWDEV_NAME_MAX_LEN, "NTB:%x:%02x.%x", + pci_dev->addr.bus, pci_dev->addr.devid, + pci_dev->addr.function); + + NTB_LOG(INFO, "Closing %s on NUMA node %d", name, rte_socket_id()); + + rawdev = rte_rawdev_pmd_get_named_dev(name); + if (rawdev == NULL) { + NTB_LOG(ERR, "Invalid device name (%s)", name); + ret = -EINVAL; + return ret; + } + + ret = rte_rawdev_pmd_release(rawdev); + if (ret) + NTB_LOG(ERR, "Failed to destroy ntb rawdev."); + + return ret; +} + +static int +ntb_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + return ntb_create(pci_dev, rte_socket_id()); +} + +static int +ntb_remove(struct rte_pci_device *pci_dev) +{ + return ntb_destroy(pci_dev); +} + + +static struct rte_pci_driver rte_ntb_pmd = { + .id_table = pci_id_ntb_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING, + .probe = ntb_probe, + .remove = ntb_remove, +}; + +RTE_PMD_REGISTER_PCI(raw_ntb, rte_ntb_pmd); +RTE_PMD_REGISTER_PCI_TABLE(raw_ntb, pci_id_ntb_map); +RTE_PMD_REGISTER_KMOD_DEP(raw_ntb, "* igb_uio | uio_pci_generic | vfio-pci"); + +RTE_INIT(ntb_init_log) +{ + ntb_logtype = rte_log_register("pmd.raw.ntb"); + if (ntb_logtype >= 0) + rte_log_set_level(ntb_logtype, RTE_LOG_DEBUG); +} diff --git a/drivers/raw/ntb/ntb.h b/drivers/raw/ntb/ntb.h new file mode 100644 index 000000000..d355231b0 --- /dev/null +++ b/drivers/raw/ntb/ntb.h @@ -0,0 +1,164 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation. + */ + +#ifndef _NTB_RAWDEV_H_ +#define _NTB_RAWDEV_H_ + +#include + +extern int ntb_logtype; + +#define NTB_LOG(level, fmt, args...) \ + rte_log(RTE_LOG_ ## level, ntb_logtype, "%s(): " fmt "\n", \ + __func__, ##args) + +/* Vendor ID */ +#define NTB_INTEL_VENDOR_ID 0x8086 + +/* Device IDs */ +#define NTB_INTEL_DEV_ID_B2B_SKX 0x201C + +#define NTB_TOPO_NAME "topo" +#define NTB_LINK_STATUS_NAME "link_status" +#define NTB_SPEED_NAME "speed" +#define NTB_WIDTH_NAME "width" +#define NTB_MW_CNT_NAME "mw_count" +#define NTB_DB_CNT_NAME "db_count" +#define NTB_SPAD_CNT_NAME "spad_count" +/* Reserved to app to use. */ +#define NTB_SPAD_USER "spad_user_" +#define NTB_SPAD_USER_LEN (sizeof(NTB_SPAD_USER) - 1) +#define NTB_SPAD_USER_MAX_NUM 10 +#define NTB_ATTR_NAME_LEN 30 +#define NTB_ATTR_VAL_LEN 30 +#define NTB_ATTR_MAX 20 + +/* NTB Attributes */ +struct ntb_attr { + /**< Name of the attribute */ + char name[NTB_ATTR_NAME_LEN]; + /**< Value or reference of value of attribute */ + char value[NTB_ATTR_NAME_LEN]; +}; + +enum ntb_attr_idx { + NTB_TOPO_ID = 0, + NTB_LINK_STATUS_ID, + NTB_SPEED_ID, + NTB_WIDTH_ID, + NTB_MW_CNT_ID, + NTB_DB_CNT_ID, + NTB_SPAD_CNT_ID, +}; + +enum ntb_topo { + NTB_TOPO_NONE = 0, + NTB_TOPO_B2B_USD, + NTB_TOPO_B2B_DSD, +}; + +enum ntb_link { + NTB_LINK_DOWN = 0, + NTB_LINK_UP, +}; + +enum ntb_speed { + NTB_SPEED_NONE = 0, + NTB_SPEED_GEN1 = 1, + NTB_SPEED_GEN2 = 2, + NTB_SPEED_GEN3 = 3, + NTB_SPEED_GEN4 = 4, +}; + +enum ntb_width { + NTB_WIDTH_NONE = 0, + NTB_WIDTH_1 = 1, + NTB_WIDTH_2 = 2, + NTB_WIDTH_4 = 4, + NTB_WIDTH_8 = 8, + NTB_WIDTH_12 = 12, + NTB_WIDTH_16 = 16, + NTB_WIDTH_32 = 32, +}; + +/* Define spad registers usage. 0 is reserved. */ +enum ntb_spad_idx { + SPAD_NUM_MWS = 1, + SPAD_NUM_QPS, + SPAD_Q_SZ, + SPAD_MW0_SZ_H, + SPAD_MW0_SZ_L, + SPAD_MW1_SZ_H, + SPAD_MW1_SZ_L, +}; + +/** + * NTB device operations + * @ntb_dev_init: Init ntb dev. + * @get_peer_mw_addr: To get the addr of peer mw[mw_idx]. + * @mw_set_trans: Set translation of internal memory that remote can access. + * @get_link_status: get link status, link speed and link width. + * @set_link: Set local side up/down. + * @spad_read: Read local/peer spad register val. + * @spad_write: Write val to local/peer spad register. + * @db_read: Read doorbells status. + * @db_clear: Clear local doorbells. + * @db_set_mask: Set bits in db mask, preventing db interrpts generated + * for those db bits. + * @peer_db_set: Set doorbell bit to generate peer interrupt for that bit. + * @vector_bind: Bind vector source [intr] to msix vector [msix]. + */ +struct ntb_dev_ops { + int (*ntb_dev_init)(struct rte_rawdev *dev); + void *(*get_peer_mw_addr)(struct rte_rawdev *dev, int mw_idx); + int (*mw_set_trans)(struct rte_rawdev *dev, int mw_idx, + uint64_t addr, uint64_t size); + int (*get_link_status)(struct rte_rawdev *dev); + int (*set_link)(struct rte_rawdev *dev, bool up); + uint32_t (*spad_read)(struct rte_rawdev *dev, int spad, bool peer); + int (*spad_write)(struct rte_rawdev *dev, int spad, + bool peer, uint32_t spad_v); + uint64_t (*db_read)(struct rte_rawdev *dev); + int (*db_clear)(struct rte_rawdev *dev, uint64_t db_bits); + int (*db_set_mask)(struct rte_rawdev *dev, uint64_t db_mask); + int (*peer_db_set)(struct rte_rawdev *dev, uint8_t db_bit); + int (*vector_bind)(struct rte_rawdev *dev, uint8_t intr, uint8_t msix); +}; + +/* ntb private data. */ +struct ntb_hw { + uint8_t mw_cnt; + uint8_t peer_mw_cnt; + uint8_t db_cnt; + uint8_t spad_cnt; + + uint64_t db_valid_mask; + uint64_t db_mask; + + enum ntb_topo topo; + + enum ntb_link link_status; + enum ntb_speed link_speed; + enum ntb_width link_width; + + const struct ntb_dev_ops *ntb_ops; + + struct rte_pci_device *pci_dev; + char *hw_addr; + + uint64_t *mw_size; + uint64_t *peer_mw_size; + uint8_t peer_dev_up; + + uint16_t queue_pairs; + uint16_t queue_size; + + /**< mem zone to populate RX ring. */ + const struct rte_memzone **mz; + + /* Reserve several spad for app to use. */ + int spad_user_list[NTB_SPAD_USER_MAX_NUM]; +}; + +#endif /* _NTB_RAWDEV_H_ */ diff --git a/drivers/raw/ntb/rte_pmd_ntb_version.map b/drivers/raw/ntb/rte_pmd_ntb_version.map new file mode 100644 index 000000000..8861484fb --- /dev/null +++ b/drivers/raw/ntb/rte_pmd_ntb_version.map @@ -0,0 +1,4 @@ +DPDK_19.08 { + + local: *; +}; diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 30f9a12da..d3d12350b 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -310,6 +310,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV) += -lrte_pmd_ifpga_rawdev _LDLIBS-$(CONFIG_RTE_LIBRTE_IPN3KE_PMD) += -lrte_pmd_ipn3ke endif # CONFIG_RTE_LIBRTE_IFPGA_BUS _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_IOAT_RAWDEV) += -lrte_pmd_ioat_rawdev +_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += -lrte_pmd_ntb endif # CONFIG_RTE_LIBRTE_RAWDEV endif # !CONFIG_RTE_BUILD_SHARED_LIBS From patchwork Thu Jul 4 12:29:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Monjalon X-Patchwork-Id: 56085 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E481A1BE37; Thu, 4 Jul 2019 14:30:46 +0200 (CEST) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by dpdk.org (Postfix) with ESMTP id 73C5E1BE2A for ; Thu, 4 Jul 2019 14:30:45 +0200 (CEST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 2023F21B42; Thu, 4 Jul 2019 08:30:45 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Thu, 04 Jul 2019 08:30:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=mesmtp; bh=pihPqhKQvJ PjdCrD1c6FRvdcNPQAxM4DLAAk2PjPqus=; b=T8UvIoegKl7dbaZBWFm/0/ZaM3 NQ1Uh03yylB4oAc9QJv+OUxiQnbFQlefl1OW0vmPPUaFliaHrahoCSVYvHmrGGTG vX8www+q43l9DyOKr55aWEnxvaCXIRlJW3V7quRViZJa19j/IwEizntLEDlHpkr/ 8WVFg3nwg8lWtP2EI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=pihPqhKQvJPjdCrD1c6FRvdcNPQAxM4DLAAk2PjPqus=; b=fXxt59ud vwHZUVYVj4LjsvpDFtzMUeQ0uq0K5MqFF0kPO/XgHk+rpiyex4KCshsLmX8kVgiQ Vdg2CYqjxYXdcYjCNsjcP6YZ9xIg1t65FvAaN7whP+wE20VA3feDNzbRq79UaNow Fzxn2Zhpj19It4j6/KQ1bf6BwM2YBBH6KhbLqgEITRvjCNt6jzgrI+KKevbFwRlU Mp6vortjicSKSVH5dCs9s2s5uimCzeVhvmQq9A2et2t039NFoN4dcZoaOFmZP9TK jsurrnrjBMIP7t56Zf46kldm5Z3fRKSGirOIkuKgqWXpiWdOlY+7xeYsR6YwPnFv Su6JdH2T/uFHtQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrfedvgdehhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucffoh hmrghinhepihhnthgvlhdrtghomhdplhhisgdrmhhknecukfhppeejjedrudefgedrvddt fedrudekgeenucfrrghrrghmpehmrghilhhfrhhomhepthhhohhmrghssehmohhnjhgrlh honhdrnhgvthenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from xps.monjalon.net (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 03CDC380086; Thu, 4 Jul 2019 08:30:43 -0400 (EDT) From: Thomas Monjalon To: dev@dpdk.org, Xiaoyun Li , Jingjing Wu , John McNamara , Marko Kovacevic Cc: Xiaolong Ye Date: Thu, 4 Jul 2019 14:29:57 +0200 Message-Id: <20190704122959.18919-3-thomas@monjalon.net> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190704122959.18919-1-thomas@monjalon.net> References: <20190628025346.31312-1-xiaoyun.li@intel.com> <20190704122959.18919-1-thomas@monjalon.net> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v11 2/4] raw/ntb: support Intel NTB X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Xiaoyun Li Add in the list of registers for the device. And enable NTB device ops for Intel Skylake platform. Signed-off-by: Xiaoyun Li Acked-by: Jingjing Wu Reviewed-by: Xiaolong Ye --- doc/guides/rawdevs/ntb.rst | 25 +++ drivers/raw/ntb/Makefile | 1 + drivers/raw/ntb/meson.build | 3 +- drivers/raw/ntb/ntb.c | 5 + drivers/raw/ntb/ntb_hw_intel.c | 369 +++++++++++++++++++++++++++++++++ drivers/raw/ntb/ntb_hw_intel.h | 86 ++++++++ usertools/dpdk-devbind.py | 4 +- 7 files changed, 491 insertions(+), 2 deletions(-) create mode 100644 drivers/raw/ntb/ntb_hw_intel.c create mode 100644 drivers/raw/ntb/ntb_hw_intel.h diff --git a/doc/guides/rawdevs/ntb.rst b/doc/guides/rawdevs/ntb.rst index 9559ea4b0..ff0795b47 100644 --- a/doc/guides/rawdevs/ntb.rst +++ b/doc/guides/rawdevs/ntb.rst @@ -9,6 +9,22 @@ separate hosts so that they can communicate with each other. Thus, many user cases can benefit from this, such as fault tolerance and visual acceleration. +BIOS setting on Intel Skylake +----------------------------- + +Intel Non-transparent Bridge needs special BIOS setting. Since the PMD only +supports Intel Skylake platform, introduce BIOS setting here. The referencce +is https://www.intel.com/content/dam/support/us/en/documents/server-products/Intel_Xeon_Processor_Scalable_Family_BIOS_User_Guide.pdf + +- Set the needed PCIe port as NTB to NTB mode on both hosts. +- Enable NTB bars and set bar size of bar 23 and bar 45 as 12-29 (2K-512M) + on both hosts. Note that bar size on both hosts should be the same. +- Disable split bars for both hosts. +- Set crosslink control override as DSD/USP on one host, USD/DSP on + another host. +- Disable PCIe PII SSC (Spread Spectrum Clocking) for both hosts. This + is a hardware requirement. + Build Options ------------- @@ -16,7 +32,16 @@ Build Options Toggle compilation of the ``ntb`` driver. +Device Setup +------------ + +The Intel NTB devices need to be bound to a DPDK-supported kernel driver +to use, i.e. igb_uio, vfio. The ``dpdk-devbind.py`` script can be used to +show devices status and to bind them to a suitable kernel driver. They will +appear under the category of "Misc (rawdev) devices". + Limitation ---------- - The FIFO hasn't been introduced and will come in 19.11 release. +- This PMD only supports Intel Skylake platform. diff --git a/drivers/raw/ntb/Makefile b/drivers/raw/ntb/Makefile index ec53edd33..edd49fe75 100644 --- a/drivers/raw/ntb/Makefile +++ b/drivers/raw/ntb/Makefile @@ -23,5 +23,6 @@ LIBABIVER := 1 # all source are stored in SRCS-y # SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb.c +SRCS-$(CONFIG_RTE_LIBRTE_PMD_NTB_RAWDEV) += ntb_hw_intel.c include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/raw/ntb/meson.build b/drivers/raw/ntb/meson.build index 0fb44452c..7f39437f8 100644 --- a/drivers/raw/ntb/meson.build +++ b/drivers/raw/ntb/meson.build @@ -3,5 +3,6 @@ deps += ['rawdev', 'mbuf', 'mempool', 'pci', 'bus_pci'] -sources = files('ntb.c') +sources = files('ntb.c', + 'ntb_hw_intel.c') allow_experimental_apis = true diff --git a/drivers/raw/ntb/ntb.c b/drivers/raw/ntb/ntb.c index 2ae3cbea0..5fde704b7 100644 --- a/drivers/raw/ntb/ntb.c +++ b/drivers/raw/ntb/ntb.c @@ -18,11 +18,13 @@ #include #include +#include "ntb_hw_intel.h" #include "ntb.h" int ntb_logtype; static const struct rte_pci_id pci_id_ntb_map[] = { + { RTE_PCI_DEVICE(NTB_INTEL_VENDOR_ID, NTB_INTEL_DEV_ID_B2B_SKX) }, { .vendor_id = 0, /* sentinel */ }, }; @@ -353,6 +355,9 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) hw->link_width = NTB_WIDTH_NONE; switch (pci_dev->id.device_id) { + case NTB_INTEL_DEV_ID_B2B_SKX: + hw->ntb_ops = &intel_ntb_ops; + break; default: NTB_LOG(ERR, "Not supported device."); return -EINVAL; diff --git a/drivers/raw/ntb/ntb_hw_intel.c b/drivers/raw/ntb/ntb_hw_intel.c new file mode 100644 index 000000000..21eaa8511 --- /dev/null +++ b/drivers/raw/ntb/ntb_hw_intel.c @@ -0,0 +1,369 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation. + */ +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "ntb.h" +#include "ntb_hw_intel.h" + +enum xeon_ntb_bar { + XEON_NTB_BAR23 = 2, + XEON_NTB_BAR45 = 4, +}; + +static enum xeon_ntb_bar intel_ntb_bar[] = { + XEON_NTB_BAR23, + XEON_NTB_BAR45, +}; + +static int +intel_ntb_dev_init(struct rte_rawdev *dev) +{ + struct ntb_hw *hw = dev->dev_private; + uint8_t reg_val, bar; + int ret, i; + + if (hw == NULL) { + NTB_LOG(ERR, "Invalid device."); + return -EINVAL; + } + + ret = rte_pci_read_config(hw->pci_dev, ®_val, + sizeof(reg_val), XEON_PPD_OFFSET); + if (ret < 0) { + NTB_LOG(ERR, "Cannot get NTB PPD (PCIe port definition)."); + return -EIO; + } + + /* Check connection topo type. Only support B2B. */ + switch (reg_val & XEON_PPD_CONN_MASK) { + case XEON_PPD_CONN_B2B: + NTB_LOG(INFO, "Topo B2B (back to back) is using."); + break; + case XEON_PPD_CONN_TRANSPARENT: + case XEON_PPD_CONN_RP: + default: + NTB_LOG(ERR, "Not supported conn topo. Please use B2B."); + return -EINVAL; + } + + /* Check device type. */ + if (reg_val & XEON_PPD_DEV_DSD) { + NTB_LOG(INFO, "DSD, Downstream Device."); + hw->topo = NTB_TOPO_B2B_DSD; + } else { + NTB_LOG(INFO, "USD, Upstream device."); + hw->topo = NTB_TOPO_B2B_USD; + } + + /* Check if bar4 is split. Do not support split bar. */ + if (reg_val & XEON_PPD_SPLIT_BAR_MASK) { + NTB_LOG(ERR, "Do not support split bar."); + return -EINVAL; + } + + hw->hw_addr = (char *)hw->pci_dev->mem_resource[0].addr; + + hw->mw_cnt = XEON_MW_COUNT; + hw->db_cnt = XEON_DB_COUNT; + hw->spad_cnt = XEON_SPAD_COUNT; + + hw->mw_size = rte_zmalloc("uint64_t", + hw->mw_cnt * sizeof(uint64_t), 0); + for (i = 0; i < hw->mw_cnt; i++) { + bar = intel_ntb_bar[i]; + hw->mw_size[i] = hw->pci_dev->mem_resource[bar].len; + } + + /* Reserve the last 2 spad registers for users. */ + for (i = 0; i < NTB_SPAD_USER_MAX_NUM; i++) + hw->spad_user_list[i] = hw->spad_cnt; + hw->spad_user_list[0] = hw->spad_cnt - 2; + hw->spad_user_list[1] = hw->spad_cnt - 1; + + return 0; +} + +static void * +intel_ntb_get_peer_mw_addr(struct rte_rawdev *dev, int mw_idx) +{ + struct ntb_hw *hw = dev->dev_private; + uint8_t bar; + + if (hw == NULL) { + NTB_LOG(ERR, "Invalid device."); + return 0; + } + + if (mw_idx < 0 || mw_idx >= hw->mw_cnt) { + NTB_LOG(ERR, "Invalid memory window index (0 - %u).", + hw->mw_cnt - 1); + return 0; + } + + bar = intel_ntb_bar[mw_idx]; + + return hw->pci_dev->mem_resource[bar].addr; +} + +static int +intel_ntb_mw_set_trans(struct rte_rawdev *dev, int mw_idx, + uint64_t addr, uint64_t size) +{ + struct ntb_hw *hw = dev->dev_private; + void *xlat_addr, *limit_addr; + uint64_t xlat_off, limit_off; + uint64_t base, limit; + uint8_t bar; + + if (hw == NULL) { + NTB_LOG(ERR, "Invalid device."); + return -EINVAL; + } + + if (mw_idx < 0 || mw_idx >= hw->mw_cnt) { + NTB_LOG(ERR, "Invalid memory window index (0 - %u).", + hw->mw_cnt - 1); + return -EINVAL; + } + + bar = intel_ntb_bar[mw_idx]; + + xlat_off = XEON_IMBAR1XBASE_OFFSET + mw_idx * XEON_BAR_INTERVAL_OFFSET; + limit_off = XEON_IMBAR1XLMT_OFFSET + mw_idx * XEON_BAR_INTERVAL_OFFSET; + xlat_addr = hw->hw_addr + xlat_off; + limit_addr = hw->hw_addr + limit_off; + + /* Limit reg val should be EMBAR base address plus MW size. */ + base = addr; + limit = hw->pci_dev->mem_resource[bar].phys_addr + size; + rte_write64(base, xlat_addr); + rte_write64(limit, limit_addr); + + /* Setup the external point so that remote can access. */ + xlat_off = XEON_EMBAR1_OFFSET + 8 * mw_idx; + xlat_addr = hw->hw_addr + xlat_off; + limit_off = XEON_EMBAR1XLMT_OFFSET + mw_idx * XEON_BAR_INTERVAL_OFFSET; + limit_addr = hw->hw_addr + limit_off; + base = rte_read64(xlat_addr); + base &= ~0xf; + limit = base + size; + rte_write64(limit, limit_addr); + + return 0; +} + +static int +intel_ntb_get_link_status(struct rte_rawdev *dev) +{ + struct ntb_hw *hw = dev->dev_private; + uint16_t reg_val; + int ret; + + if (hw == NULL) { + NTB_LOG(ERR, "Invalid device."); + return -EINVAL; + } + + ret = rte_pci_read_config(hw->pci_dev, ®_val, + sizeof(reg_val), XEON_LINK_STATUS_OFFSET); + if (ret < 0) { + NTB_LOG(ERR, "Unable to get link status."); + return -EIO; + } + + hw->link_status = NTB_LNK_STA_ACTIVE(reg_val); + + if (hw->link_status) { + hw->link_speed = NTB_LNK_STA_SPEED(reg_val); + hw->link_width = NTB_LNK_STA_WIDTH(reg_val); + } else { + hw->link_speed = NTB_SPEED_NONE; + hw->link_width = NTB_WIDTH_NONE; + } + + return 0; +} + +static int +intel_ntb_set_link(struct rte_rawdev *dev, bool up) +{ + struct ntb_hw *hw = dev->dev_private; + uint32_t ntb_ctrl, reg_off; + void *reg_addr; + + reg_off = XEON_NTBCNTL_OFFSET; + reg_addr = hw->hw_addr + reg_off; + ntb_ctrl = rte_read32(reg_addr); + + if (up) { + ntb_ctrl &= ~(NTB_CTL_DISABLE | NTB_CTL_CFG_LOCK); + ntb_ctrl |= NTB_CTL_P2S_BAR2_SNOOP | NTB_CTL_S2P_BAR2_SNOOP; + ntb_ctrl |= NTB_CTL_P2S_BAR4_SNOOP | NTB_CTL_S2P_BAR4_SNOOP; + } else { + ntb_ctrl &= ~(NTB_CTL_P2S_BAR2_SNOOP | NTB_CTL_S2P_BAR2_SNOOP); + ntb_ctrl &= ~(NTB_CTL_P2S_BAR4_SNOOP | NTB_CTL_S2P_BAR4_SNOOP); + ntb_ctrl |= NTB_CTL_DISABLE | NTB_CTL_CFG_LOCK; + } + + rte_write32(ntb_ctrl, reg_addr); + + return 0; +} + +static uint32_t +intel_ntb_spad_read(struct rte_rawdev *dev, int spad, bool peer) +{ + struct ntb_hw *hw = dev->dev_private; + uint32_t spad_v, reg_off; + void *reg_addr; + + if (spad < 0 || spad >= hw->spad_cnt) { + NTB_LOG(ERR, "Invalid spad reg index."); + return 0; + } + + /* When peer is true, read peer spad reg */ + reg_off = peer ? XEON_B2B_SPAD_OFFSET : XEON_IM_SPAD_OFFSET; + reg_addr = hw->hw_addr + reg_off + (spad << 2); + spad_v = rte_read32(reg_addr); + + return spad_v; +} + +static int +intel_ntb_spad_write(struct rte_rawdev *dev, int spad, + bool peer, uint32_t spad_v) +{ + struct ntb_hw *hw = dev->dev_private; + uint32_t reg_off; + void *reg_addr; + + if (spad < 0 || spad >= hw->spad_cnt) { + NTB_LOG(ERR, "Invalid spad reg index."); + return -EINVAL; + } + + /* When peer is true, write peer spad reg */ + reg_off = peer ? XEON_B2B_SPAD_OFFSET : XEON_IM_SPAD_OFFSET; + reg_addr = hw->hw_addr + reg_off + (spad << 2); + + rte_write32(spad_v, reg_addr); + + return 0; +} + +static uint64_t +intel_ntb_db_read(struct rte_rawdev *dev) +{ + struct ntb_hw *hw = dev->dev_private; + uint64_t db_off, db_bits; + void *db_addr; + + db_off = XEON_IM_INT_STATUS_OFFSET; + db_addr = hw->hw_addr + db_off; + + db_bits = rte_read64(db_addr); + + return db_bits; +} + +static int +intel_ntb_db_clear(struct rte_rawdev *dev, uint64_t db_bits) +{ + struct ntb_hw *hw = dev->dev_private; + uint64_t db_off; + void *db_addr; + + db_off = XEON_IM_INT_STATUS_OFFSET; + db_addr = hw->hw_addr + db_off; + + rte_write64(db_bits, db_addr); + + return 0; +} + +static int +intel_ntb_db_set_mask(struct rte_rawdev *dev, uint64_t db_mask) +{ + struct ntb_hw *hw = dev->dev_private; + uint64_t db_m_off; + void *db_m_addr; + + db_m_off = XEON_IM_INT_DISABLE_OFFSET; + db_m_addr = hw->hw_addr + db_m_off; + + db_mask |= hw->db_mask; + + rte_write64(db_mask, db_m_addr); + + hw->db_mask = db_mask; + + return 0; +} + +static int +intel_ntb_peer_db_set(struct rte_rawdev *dev, uint8_t db_idx) +{ + struct ntb_hw *hw = dev->dev_private; + uint32_t db_off; + void *db_addr; + + if (((uint64_t)1 << db_idx) & ~hw->db_valid_mask) { + NTB_LOG(ERR, "Invalid doorbell."); + return -EINVAL; + } + + db_off = XEON_IM_DOORBELL_OFFSET + db_idx * 4; + db_addr = hw->hw_addr + db_off; + + rte_write32(1, db_addr); + + return 0; +} + +static int +intel_ntb_vector_bind(struct rte_rawdev *dev, uint8_t intr, uint8_t msix) +{ + struct ntb_hw *hw = dev->dev_private; + uint8_t reg_off; + void *reg_addr; + + if (intr >= hw->db_cnt) { + NTB_LOG(ERR, "Invalid intr source."); + return -EINVAL; + } + + /* Bind intr source to msix vector */ + reg_off = XEON_INTVEC_OFFSET; + reg_addr = hw->hw_addr + reg_off + intr; + + rte_write8(msix, reg_addr); + + return 0; +} + +/* operations for primary side of local ntb */ +const struct ntb_dev_ops intel_ntb_ops = { + .ntb_dev_init = intel_ntb_dev_init, + .get_peer_mw_addr = intel_ntb_get_peer_mw_addr, + .mw_set_trans = intel_ntb_mw_set_trans, + .get_link_status = intel_ntb_get_link_status, + .set_link = intel_ntb_set_link, + .spad_read = intel_ntb_spad_read, + .spad_write = intel_ntb_spad_write, + .db_read = intel_ntb_db_read, + .db_clear = intel_ntb_db_clear, + .db_set_mask = intel_ntb_db_set_mask, + .peer_db_set = intel_ntb_peer_db_set, + .vector_bind = intel_ntb_vector_bind, +}; diff --git a/drivers/raw/ntb/ntb_hw_intel.h b/drivers/raw/ntb/ntb_hw_intel.h new file mode 100644 index 000000000..4d1e64504 --- /dev/null +++ b/drivers/raw/ntb/ntb_hw_intel.h @@ -0,0 +1,86 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation. + */ + +#ifndef _NTB_HW_INTEL_H_ +#define _NTB_HW_INTEL_H_ + +/* Ntb control and link status */ +#define NTB_CTL_CFG_LOCK 1 +#define NTB_CTL_DISABLE 2 +#define NTB_CTL_S2P_BAR2_SNOOP (1 << 2) +#define NTB_CTL_P2S_BAR2_SNOOP (1 << 4) +#define NTB_CTL_S2P_BAR4_SNOOP (1 << 6) +#define NTB_CTL_P2S_BAR4_SNOOP (1 << 8) +#define NTB_CTL_S2P_BAR5_SNOOP (1 << 12) +#define NTB_CTL_P2S_BAR5_SNOOP (1 << 14) + +#define NTB_LNK_STA_ACTIVE_BIT 0x2000 +#define NTB_LNK_STA_SPEED_MASK 0x000f +#define NTB_LNK_STA_WIDTH_MASK 0x03f0 +#define NTB_LNK_STA_ACTIVE(x) (!!((x) & NTB_LNK_STA_ACTIVE_BIT)) +#define NTB_LNK_STA_SPEED(x) ((x) & NTB_LNK_STA_SPEED_MASK) +#define NTB_LNK_STA_WIDTH(x) (((x) & NTB_LNK_STA_WIDTH_MASK) >> 4) + +/* Intel Skylake Xeon hardware */ +#define XEON_IMBAR1SZ_OFFSET 0x00d0 +#define XEON_IMBAR2SZ_OFFSET 0x00d1 +#define XEON_EMBAR1SZ_OFFSET 0x00d2 +#define XEON_EMBAR2SZ_OFFSET 0x00d3 +#define XEON_DEVCTRL_OFFSET 0x0098 +#define XEON_DEVSTS_OFFSET 0x009a +#define XEON_UNCERRSTS_OFFSET 0x014c +#define XEON_CORERRSTS_OFFSET 0x0158 +#define XEON_LINK_STATUS_OFFSET 0x01a2 + +#define XEON_NTBCNTL_OFFSET 0x0000 +#define XEON_BAR_INTERVAL_OFFSET 0x0010 +#define XEON_IMBAR1XBASE_OFFSET 0x0010 /* SBAR2XLAT */ +#define XEON_IMBAR1XLMT_OFFSET 0x0018 /* SBAR2LMT */ +#define XEON_IMBAR2XBASE_OFFSET 0x0020 /* SBAR4XLAT */ +#define XEON_IMBAR2XLMT_OFFSET 0x0028 /* SBAR4LMT */ +#define XEON_IM_INT_STATUS_OFFSET 0x0040 +#define XEON_IM_INT_DISABLE_OFFSET 0x0048 +#define XEON_IM_SPAD_OFFSET 0x0080 /* SPAD */ +#define XEON_USMEMMISS_OFFSET 0x0070 +#define XEON_INTVEC_OFFSET 0x00d0 +#define XEON_IM_DOORBELL_OFFSET 0x0100 /* SDOORBELL0 */ +#define XEON_B2B_SPAD_OFFSET 0x0180 /* B2B SPAD */ +#define XEON_EMBAR0XBASE_OFFSET 0x4008 /* B2B_XLAT */ +#define XEON_EMBAR1XBASE_OFFSET 0x4010 /* PBAR2XLAT */ +#define XEON_EMBAR1XLMT_OFFSET 0x4018 /* PBAR2LMT */ +#define XEON_EMBAR2XBASE_OFFSET 0x4020 /* PBAR4XLAT */ +#define XEON_EMBAR2XLMT_OFFSET 0x4028 /* PBAR4LMT */ +#define XEON_EM_INT_STATUS_OFFSET 0x4040 +#define XEON_EM_INT_DISABLE_OFFSET 0x4048 +#define XEON_EM_SPAD_OFFSET 0x4080 /* remote SPAD */ +#define XEON_EM_DOORBELL_OFFSET 0x4100 /* PDOORBELL0 */ +#define XEON_SPCICMD_OFFSET 0x4504 /* SPCICMD */ +#define XEON_EMBAR0_OFFSET 0x4510 /* SBAR0BASE */ +#define XEON_EMBAR1_OFFSET 0x4518 /* SBAR23BASE */ +#define XEON_EMBAR2_OFFSET 0x4520 /* SBAR45BASE */ + +#define XEON_PPD_OFFSET 0x00d4 +#define XEON_PPD_CONN_MASK 0x03 +#define XEON_PPD_CONN_TRANSPARENT 0x00 +#define XEON_PPD_CONN_B2B 0x01 +#define XEON_PPD_CONN_RP 0x02 +#define XEON_PPD_DEV_MASK 0x10 +#define XEON_PPD_DEV_USD 0x00 +#define XEON_PPD_DEV_DSD 0x10 +#define XEON_PPD_SPLIT_BAR_MASK 0x40 + + +#define XEON_MW_COUNT 2 + +#define XEON_DB_COUNT 32 +#define XEON_DB_LINK 32 +#define XEON_DB_LINK_BIT (1ULL << XEON_DB_LINK) +#define XEON_DB_MSIX_VECTOR_COUNT 33 +#define XEON_DB_MSIX_VECTOR_SHIFT 1 +#define XEON_DB_TOTAL_SHIFT 33 +#define XEON_SPAD_COUNT 16 + +extern const struct ntb_dev_ops intel_ntb_ops; + +#endif /* _NTB_HW_INTEL_H_ */ diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index 5c1cd3548..13026858d 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -40,13 +40,15 @@ 'SVendor': None, 'SDevice': None} intel_ioat_skx = {'Class': '08', 'Vendor': '8086', 'Device': '2021', 'SVendor': None, 'SDevice': None} +intel_ntb_skx = {'Class': '06', 'Vendor': '8086', 'Device': '201c', + 'SVendor': None, 'SDevice': None} network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class] crypto_devices = [encryption_class, intel_processor_class] eventdev_devices = [cavium_sso, cavium_tim, octeontx2_sso] mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] -misc_devices = [intel_ioat_bdw, intel_ioat_skx] +misc_devices = [intel_ioat_bdw, intel_ioat_skx, intel_ntb_skx] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties From patchwork Thu Jul 4 12:29:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Monjalon X-Patchwork-Id: 56086 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BF2791BE3F; Thu, 4 Jul 2019 14:30:56 +0200 (CEST) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by dpdk.org (Postfix) with ESMTP id A6B8A1BE32 for ; Thu, 4 Jul 2019 14:30:54 +0200 (CEST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 2045F21FC3; Thu, 4 Jul 2019 08:30:54 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Thu, 04 Jul 2019 08:30:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=mesmtp; bh=6mN1Zfe1sM H+pAJiKJhZ88NIHpQZiM6w3VmhoBNaQ6w=; b=OnqwNiIZAMbb4GIP8MLXpucX4q 6ia7lENs01uMFvHyvVBF3KSUNr8WLNtWUb1AIbSSni/eLe9sCqm1wjOhmUWbGPE5 bPoNWEPQraPApHRbzbZEkiXRYKoZVxh/uagdXCBmgmqsJ/PthoycDRrO8vhChNG1 tdLDkvwCq2PWZsjBY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=6mN1Zfe1sMH+pAJiKJhZ88NIHpQZiM6w3VmhoBNaQ6w=; b=SEaWL7vj dL3r5D8mXlXPnTmNs1KX1dHKtSrUa8g1sXxtgFHqyDJ3mu2vjgrwK+cAiue3JmuL w3wEQ+55WJfpj7L2Jzq+jpgkLIYpT5YWKz00pI6TBThQ7m5KRZ+g7moouwVW1PHa BIzGRHw6Sd8CpvYIiChL29OpxJFH90mRDYgrBwjpihIXBNWKZmEnUo7h1HMJ4+vc uvqP2uybz9WQ99OEiVe+sucAhGCVWblPJODxoP9Wglu/+fRck4q+BxOJqAXMBSxp pSGnK+xbpuauRtfZpg10OFWdOUE388JzZrPDw0memyx5wIH28IjHQoCrk6GjaH1S pgIKpaZgoh7FMg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrfedvgdehiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucfkph epjeejrddufeegrddvtdefrddukeegnecurfgrrhgrmhepmhgrihhlfhhrohhmpehthhho mhgrshesmhhonhhjrghlohhnrdhnvghtnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from xps.monjalon.net (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 0043B38008C; Thu, 4 Jul 2019 08:30:52 -0400 (EDT) From: Thomas Monjalon To: dev@dpdk.org, Xiaoyun Li , Jingjing Wu , John McNamara , Marko Kovacevic Cc: Xiaolong Ye Date: Thu, 4 Jul 2019 14:29:58 +0200 Message-Id: <20190704122959.18919-4-thomas@monjalon.net> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190704122959.18919-1-thomas@monjalon.net> References: <20190628025346.31312-1-xiaoyun.li@intel.com> <20190704122959.18919-1-thomas@monjalon.net> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v11 3/4] raw/ntb: add handshake process X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Xiaoyun Li Add handshake process using doorbell so that two hosts can communicate to start and stop. Signed-off-by: Xiaoyun Li Acked-by: Jingjing Wu Reviewed-by: Xiaolong Ye --- doc/guides/rawdevs/ntb.rst | 5 + drivers/raw/ntb/ntb.c | 336 ++++++++++++++++++++++++++++++++++++- 2 files changed, 340 insertions(+), 1 deletion(-) diff --git a/doc/guides/rawdevs/ntb.rst b/doc/guides/rawdevs/ntb.rst index ff0795b47..0a61ec03d 100644 --- a/doc/guides/rawdevs/ntb.rst +++ b/doc/guides/rawdevs/ntb.rst @@ -9,6 +9,11 @@ separate hosts so that they can communicate with each other. Thus, many user cases can benefit from this, such as fault tolerance and visual acceleration. +This PMD allows two hosts to handshake for device start and stop, memory +allocation for the peer to access and read/write allocated memory from peer. +Also, the PMD allows to use doorbell registers to notify the peer and share +some information by using scratchpad registers. + BIOS setting on Intel Skylake ----------------------------- diff --git a/drivers/raw/ntb/ntb.c b/drivers/raw/ntb/ntb.c index 5fde704b7..c83b8d964 100644 --- a/drivers/raw/ntb/ntb.c +++ b/drivers/raw/ntb/ntb.c @@ -28,6 +28,183 @@ static const struct rte_pci_id pci_id_ntb_map[] = { { .vendor_id = 0, /* sentinel */ }, }; +static int +ntb_set_mw(struct rte_rawdev *dev, int mw_idx, uint64_t mw_size) +{ + struct ntb_hw *hw = dev->dev_private; + char mw_name[RTE_MEMZONE_NAMESIZE]; + const struct rte_memzone *mz; + int ret = 0; + + if (hw->ntb_ops->mw_set_trans == NULL) { + NTB_LOG(ERR, "Not supported to set mw."); + return -ENOTSUP; + } + + snprintf(mw_name, sizeof(mw_name), "ntb_%d_mw_%d", + dev->dev_id, mw_idx); + + mz = rte_memzone_lookup(mw_name); + if (mz) + return 0; + + /** + * Hardware requires that mapped memory base address should be + * aligned with EMBARSZ and needs continuous memzone. + */ + mz = rte_memzone_reserve_aligned(mw_name, mw_size, dev->socket_id, + RTE_MEMZONE_IOVA_CONTIG, hw->mw_size[mw_idx]); + if (!mz) { + NTB_LOG(ERR, "Cannot allocate aligned memzone."); + return -EIO; + } + hw->mz[mw_idx] = mz; + + ret = (*hw->ntb_ops->mw_set_trans)(dev, mw_idx, mz->iova, mw_size); + if (ret) { + NTB_LOG(ERR, "Cannot set mw translation."); + return ret; + } + + return ret; +} + +static void +ntb_link_cleanup(struct rte_rawdev *dev) +{ + struct ntb_hw *hw = dev->dev_private; + int status, i; + + if (hw->ntb_ops->spad_write == NULL || + hw->ntb_ops->mw_set_trans == NULL) { + NTB_LOG(ERR, "Not supported to clean up link."); + return; + } + + /* Clean spad registers. */ + for (i = 0; i < hw->spad_cnt; i++) { + status = (*hw->ntb_ops->spad_write)(dev, i, 0, 0); + if (status) + NTB_LOG(ERR, "Failed to clean local spad."); + } + + /* Clear mw so that peer cannot access local memory.*/ + for (i = 0; i < hw->mw_cnt; i++) { + status = (*hw->ntb_ops->mw_set_trans)(dev, i, 0, 0); + if (status) + NTB_LOG(ERR, "Failed to clean mw."); + } +} + +static void +ntb_dev_intr_handler(void *param) +{ + struct rte_rawdev *dev = (struct rte_rawdev *)param; + struct ntb_hw *hw = dev->dev_private; + uint32_t mw_size_h, mw_size_l; + uint64_t db_bits = 0; + int i = 0; + + if (hw->ntb_ops->db_read == NULL || + hw->ntb_ops->db_clear == NULL || + hw->ntb_ops->peer_db_set == NULL) { + NTB_LOG(ERR, "Doorbell is not supported."); + return; + } + + db_bits = (*hw->ntb_ops->db_read)(dev); + if (!db_bits) + NTB_LOG(ERR, "No doorbells"); + + /* Doorbell 0 is for peer device ready. */ + if (db_bits & 1) { + NTB_LOG(DEBUG, "DB0: Peer device is up."); + /* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, 1); + + /** + * Peer dev is already up. All mw settings are already done. + * Skip them. + */ + if (hw->peer_dev_up) + return; + + if (hw->ntb_ops->spad_read == NULL || + hw->ntb_ops->spad_write == NULL) { + NTB_LOG(ERR, "Scratchpad is not supported."); + return; + } + + hw->peer_mw_cnt = (*hw->ntb_ops->spad_read) + (dev, SPAD_NUM_MWS, 0); + hw->peer_mw_size = rte_zmalloc("uint64_t", + hw->peer_mw_cnt * sizeof(uint64_t), 0); + for (i = 0; i < hw->mw_cnt; i++) { + mw_size_h = (*hw->ntb_ops->spad_read) + (dev, SPAD_MW0_SZ_H + 2 * i, 0); + mw_size_l = (*hw->ntb_ops->spad_read) + (dev, SPAD_MW0_SZ_L + 2 * i, 0); + hw->peer_mw_size[i] = ((uint64_t)mw_size_h << 32) | + mw_size_l; + NTB_LOG(DEBUG, "Peer %u mw size: 0x%"PRIx64"", i, + hw->peer_mw_size[i]); + } + + hw->peer_dev_up = 1; + + /** + * Handshake with peer. Spad_write only works when both + * devices are up. So write spad again when db is received. + * And set db again for the later device who may miss + * the 1st db. + */ + for (i = 0; i < hw->mw_cnt; i++) { + (*hw->ntb_ops->spad_write)(dev, SPAD_NUM_MWS, + 1, hw->mw_cnt); + mw_size_h = hw->mw_size[i] >> 32; + (*hw->ntb_ops->spad_write)(dev, SPAD_MW0_SZ_H + 2 * i, + 1, mw_size_h); + + mw_size_l = hw->mw_size[i]; + (*hw->ntb_ops->spad_write)(dev, SPAD_MW0_SZ_L + 2 * i, + 1, mw_size_l); + } + (*hw->ntb_ops->peer_db_set)(dev, 0); + + /* To get the link info. */ + if (hw->ntb_ops->get_link_status == NULL) { + NTB_LOG(ERR, "Not supported to get link status."); + return; + } + (*hw->ntb_ops->get_link_status)(dev); + NTB_LOG(INFO, "Link is up. Link speed: %u. Link width: %u", + hw->link_speed, hw->link_width); + return; + } + + if (db_bits & (1 << 1)) { + NTB_LOG(DEBUG, "DB1: Peer device is down."); + /* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, 2); + + /* Peer device will be down, So clean local side too. */ + ntb_link_cleanup(dev); + + hw->peer_dev_up = 0; + /* Response peer's dev_stop request. */ + (*hw->ntb_ops->peer_db_set)(dev, 2); + return; + } + + if (db_bits & (1 << 2)) { + NTB_LOG(DEBUG, "DB2: Peer device agrees dev to be down."); + /* Clear received doorbell. */ + (*hw->ntb_ops->db_clear)(dev, (1 << 2)); + hw->peer_dev_up = 0; + return; + } +} + static void ntb_queue_conf_get(struct rte_rawdev *dev __rte_unused, uint16_t queue_id __rte_unused, @@ -147,7 +324,22 @@ ntb_dev_configure(const struct rte_rawdev *dev __rte_unused, static int ntb_dev_start(struct rte_rawdev *dev) { + struct ntb_hw *hw = dev->dev_private; + int ret, i; + /* TODO: init queues and start queues. */ + + /* Map memory of bar_size to remote. */ + hw->mz = rte_zmalloc("struct rte_memzone *", + hw->mw_cnt * sizeof(struct rte_memzone *), 0); + for (i = 0; i < hw->mw_cnt; i++) { + ret = ntb_set_mw(dev, i, hw->mw_size[i]); + if (ret) { + NTB_LOG(ERR, "Fail to set mw."); + return ret; + } + } + dev->started = 1; return 0; @@ -156,13 +348,59 @@ ntb_dev_start(struct rte_rawdev *dev) static void ntb_dev_stop(struct rte_rawdev *dev) { + struct ntb_hw *hw = dev->dev_private; + uint32_t time_out; + int status; + /* TODO: stop rx/tx queues. */ + + if (!hw->peer_dev_up) + goto clean; + + ntb_link_cleanup(dev); + + /* Notify the peer that device will be down. */ + if (hw->ntb_ops->peer_db_set == NULL) { + NTB_LOG(ERR, "Peer doorbell setting is not supported."); + return; + } + status = (*hw->ntb_ops->peer_db_set)(dev, 1); + if (status) { + NTB_LOG(ERR, "Failed to tell peer device is down."); + return; + } + + /* + * Set time out as 1s in case that the peer is stopped accidently + * without any notification. + */ + time_out = 1000000; + + /* Wait for cleanup work down before db mask clear. */ + while (hw->peer_dev_up && time_out) { + time_out -= 10; + rte_delay_us(10); + } + +clean: + /* Clear doorbells mask. */ + if (hw->ntb_ops->db_set_mask == NULL) { + NTB_LOG(ERR, "Doorbell mask setting is not supported."); + return; + } + status = (*hw->ntb_ops->db_set_mask)(dev, + (((uint64_t)1 << hw->db_cnt) - 1)); + if (status) + NTB_LOG(ERR, "Failed to clear doorbells."); + dev->started = 0; } static int ntb_dev_close(struct rte_rawdev *dev) { + struct ntb_hw *hw = dev->dev_private; + struct rte_intr_handle *intr_handle; int ret = 0; if (dev->started) @@ -170,6 +408,20 @@ ntb_dev_close(struct rte_rawdev *dev) /* TODO: free queues. */ + intr_handle = &hw->pci_dev->intr_handle; + /* Clean datapath event and vec mapping */ + rte_intr_efd_disable(intr_handle); + if (intr_handle->intr_vec) { + rte_free(intr_handle->intr_vec); + intr_handle->intr_vec = NULL; + } + /* Disable uio intr before callback unregister */ + rte_intr_disable(intr_handle); + + /* Unregister callback func to eal lib */ + rte_intr_callback_unregister(intr_handle, + ntb_dev_intr_handler, dev); + return ret; } @@ -346,7 +598,9 @@ static int ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) { struct ntb_hw *hw = dev->dev_private; - int ret; + struct rte_intr_handle *intr_handle; + uint32_t val; + int ret, i; hw->pci_dev = pci_dev; hw->peer_dev_up = 0; @@ -377,6 +631,86 @@ ntb_init_hw(struct rte_rawdev *dev, struct rte_pci_device *pci_dev) if (ret) return ret; + /* Init doorbell. */ + hw->db_valid_mask = RTE_LEN2MASK(hw->db_cnt, uint64_t); + + intr_handle = &pci_dev->intr_handle; + /* Register callback func to eal lib */ + rte_intr_callback_register(intr_handle, + ntb_dev_intr_handler, dev); + + ret = rte_intr_efd_enable(intr_handle, hw->db_cnt); + if (ret) + return ret; + + /* To clarify, the interrupt for each doorbell is already mapped + * by default for intel gen3. They are mapped to msix vec 1-32, + * and hardware intr is mapped to 0. Map all to 0 for uio. + */ + if (!rte_intr_cap_multiple(intr_handle)) { + for (i = 0; i < hw->db_cnt; i++) { + if (hw->ntb_ops->vector_bind == NULL) + return -ENOTSUP; + ret = (*hw->ntb_ops->vector_bind)(dev, i, 0); + if (ret) + return ret; + } + } + + if (hw->ntb_ops->db_set_mask == NULL || + hw->ntb_ops->peer_db_set == NULL) { + NTB_LOG(ERR, "Doorbell is not supported."); + return -ENOTSUP; + } + hw->db_mask = 0; + ret = (*hw->ntb_ops->db_set_mask)(dev, hw->db_mask); + if (ret) { + NTB_LOG(ERR, "Unable to enable intr for all dbs."); + return ret; + } + + /* enable uio intr after callback register */ + rte_intr_enable(intr_handle); + + if (hw->ntb_ops->spad_write == NULL) { + NTB_LOG(ERR, "Scratchpad is not supported."); + return -ENOTSUP; + } + /* Tell peer the mw_cnt of local side. */ + ret = (*hw->ntb_ops->spad_write)(dev, SPAD_NUM_MWS, 1, hw->mw_cnt); + if (ret) { + NTB_LOG(ERR, "Failed to tell peer mw count."); + return ret; + } + + /* Tell peer each mw size on local side. */ + for (i = 0; i < hw->mw_cnt; i++) { + NTB_LOG(DEBUG, "Local %u mw size: 0x%"PRIx64"", i, + hw->mw_size[i]); + val = hw->mw_size[i] >> 32; + ret = (*hw->ntb_ops->spad_write) + (dev, SPAD_MW0_SZ_H + 2 * i, 1, val); + if (ret) { + NTB_LOG(ERR, "Failed to tell peer mw size."); + return ret; + } + + val = hw->mw_size[i]; + ret = (*hw->ntb_ops->spad_write) + (dev, SPAD_MW0_SZ_L + 2 * i, 1, val); + if (ret) { + NTB_LOG(ERR, "Failed to tell peer mw size."); + return ret; + } + } + + /* Ring doorbell 0 to tell peer the device is ready. */ + ret = (*hw->ntb_ops->peer_db_set)(dev, 0); + if (ret) { + NTB_LOG(ERR, "Failed to tell peer device is probed."); + return ret; + } + return ret; } From patchwork Thu Jul 4 12:29:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Monjalon X-Patchwork-Id: 56087 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E84C11BE38; Thu, 4 Jul 2019 14:30:59 +0200 (CEST) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by dpdk.org (Postfix) with ESMTP id B22091BE43 for ; Thu, 4 Jul 2019 14:30:57 +0200 (CEST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 2EA0521550; Thu, 4 Jul 2019 08:30:57 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Thu, 04 Jul 2019 08:30:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=mesmtp; bh=PwoUDP4ooc kJFoSh7gEZ6LX2mlA/LaTFXnMnjrY1ehg=; b=cuqd1snDwYt4Kfah3Ek9pLUh4a o1RMY7Q57evoC1pq2GSXVBsNIpd6+EZCK1+A3GjKqzt4zbRBhod9eOQ5fy6Lv9al M/42+RARjUEvGLx2T+a4e4WDwhj3z1gdH+tlG/fJR+xD/ATYRvpSj3A73O3v0P0G 80U2hnFTufMR6OUlk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=PwoUDP4oockJFoSh7gEZ6LX2mlA/LaTFXnMnjrY1ehg=; b=pwPy/x98 u1uVVJNPchHrA1H78Qm5CeuVpEsje5ZPCDrjA8q9IKdE0RydQ5+siJvsU4y3COmY ZlK5qoSBmZNGpdrpqq6G5wlnLfRLGduQGif42TASueBSdLtgFYTMbj7GrP0FrPqT zVml7D9gfqGVOeF7mFhx2fp9xPLIVJcG6/O8LowknUKHKyT6LmT0aVFsT43HEIWF jYE9yggaP6ltlfN2xym5c3qR9jzkkTv5vv5dVRkpk0TmN+EpWDM7tfgHTQTcPjlz NiluzjLCV8u2sL8XE4hrl7hnZx4V6S2Pw1WjIDMrg+ZT5ubK3AgjaJYPrZiq/2+X 6DU2nt/7NFH1xw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrfedvgdehiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucffoh hmrghinhepvhgrrhhsrdhmkhdpvgigthgrphhprdhmkhenucfkphepjeejrddufeegrddv tdefrddukeegnecurfgrrhgrmhepmhgrihhlfhhrohhmpehthhhomhgrshesmhhonhhjrg hlohhnrdhnvghtnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from xps.monjalon.net (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id E9C1F380088; Thu, 4 Jul 2019 08:30:55 -0400 (EDT) From: Thomas Monjalon To: dev@dpdk.org, John McNamara , Marko Kovacevic , Xiaoyun Li , Jingjing Wu Cc: Xiaolong Ye Date: Thu, 4 Jul 2019 14:29:59 +0200 Message-Id: <20190704122959.18919-5-thomas@monjalon.net> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190704122959.18919-1-thomas@monjalon.net> References: <20190628025346.31312-1-xiaoyun.li@intel.com> <20190704122959.18919-1-thomas@monjalon.net> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v11 4/4] examples/ntb: add example for NTB X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Xiaoyun Li Enable an example for rawdev ntb. Support interactive mode to send file on one host and receive file from another host. The command line would be 'send [filepath]' and 'receive [filepath]'. But since the FIFO is not enabled right now, use rte_memcpy as the enqueue and dequeue functions and only support transmitting file no more than 4M. Signed-off-by: Xiaoyun Li Acked-by: Jingjing Wu Reviewed-by: Xiaolong Ye --- MAINTAINERS | 2 + doc/guides/sample_app_ug/index.rst | 1 + doc/guides/sample_app_ug/ntb.rst | 47 ++++ drivers/raw/ntb/ntb.c | 28 ++- examples/Makefile | 1 + examples/meson.build | 2 +- examples/ntb/Makefile | 68 ++++++ examples/ntb/meson.build | 16 ++ examples/ntb/ntb_fwd.c | 377 +++++++++++++++++++++++++++++ 9 files changed, 533 insertions(+), 9 deletions(-) create mode 100644 doc/guides/sample_app_ug/ntb.rst create mode 100644 examples/ntb/Makefile create mode 100644 examples/ntb/meson.build create mode 100644 examples/ntb/ntb_fwd.c diff --git a/MAINTAINERS b/MAINTAINERS index 9e6308aa4..5667a4369 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1110,6 +1110,8 @@ M: Xiaoyun Li M: Jingjing Wu F: drivers/raw/ntb/ F: doc/guides/rawdevs/ntb.rst +F: examples/ntb/ +F: doc/guides/sample_app_ug/ntb.rst Packet processing diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst index 2945be08f..f23f8f59e 100644 --- a/doc/guides/sample_app_ug/index.rst +++ b/doc/guides/sample_app_ug/index.rst @@ -58,3 +58,4 @@ Sample Applications User Guides fips_validation ipsec_secgw bbdev_app + ntb diff --git a/doc/guides/sample_app_ug/ntb.rst b/doc/guides/sample_app_ug/ntb.rst new file mode 100644 index 000000000..079242175 --- /dev/null +++ b/doc/guides/sample_app_ug/ntb.rst @@ -0,0 +1,47 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +NTB Sample Application +====================== + +The ntb sample application shows how to use ntb rawdev driver. +This sample provides interactive mode to transmit file between +two hosts. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``ntb`` sub-directory. + +Running the Application +----------------------- + +The application requires an available core for each port, plus one. +The only available options are the standard ones for the EAL: + +.. code-block:: console + + ./build/ntb_fwd -c 0xf -n 6 -- -i + +Refer to the *DPDK Getting Started Guide* for general information on +running applications and the Environment Abstraction Layer (EAL) +options. + +Using the application +--------------------- + +The application is console-driven using the cmdline DPDK interface: + +.. code-block:: console + + ntb> + +From this interface the available commands and descriptions of what +they do as as follows: + +* ``send [filepath]``: Send file to the peer host. +* ``receive [filepath]``: Receive file to [filepath]. Need the peer + to send file successfully first. +* ``quit``: Exit program diff --git a/drivers/raw/ntb/ntb.c b/drivers/raw/ntb/ntb.c index c83b8d964..4ba2f3a38 100644 --- a/drivers/raw/ntb/ntb.c +++ b/drivers/raw/ntb/ntb.c @@ -240,11 +240,19 @@ ntb_enqueue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { - RTE_SET_USED(dev); - RTE_SET_USED(buffers); - RTE_SET_USED(count); - RTE_SET_USED(context); + /* Not FIFO right now. Just for testing memory write. */ + struct ntb_hw *hw = dev->dev_private; + unsigned int i; + void *bar_addr; + size_t size; + if (hw->ntb_ops->get_peer_mw_addr == NULL) + return -ENOTSUP; + bar_addr = (*hw->ntb_ops->get_peer_mw_addr)(dev, 0); + size = (size_t)context; + + for (i = 0; i < count; i++) + rte_memcpy(bar_addr, buffers[i]->buf_addr, size); return 0; } @@ -254,11 +262,15 @@ ntb_dequeue_bufs(struct rte_rawdev *dev, unsigned int count, rte_rawdev_obj_t context) { - RTE_SET_USED(dev); - RTE_SET_USED(buffers); - RTE_SET_USED(count); - RTE_SET_USED(context); + /* Not FIFO. Just for testing memory read. */ + struct ntb_hw *hw = dev->dev_private; + unsigned int i; + size_t size; + size = (size_t)context; + + for (i = 0; i < count; i++) + rte_memcpy(buffers[i]->buf_addr, hw->mz[i]->addr, size); return 0; } diff --git a/examples/Makefile b/examples/Makefile index 7562424d9..de11dd487 100644 --- a/examples/Makefile +++ b/examples/Makefile @@ -53,6 +53,7 @@ DIRS-y += link_status_interrupt DIRS-$(CONFIG_RTE_LIBRTE_LPM) += load_balancer DIRS-y += multi_process DIRS-y += netmap_compat/bridge +DIRS-y += ntb DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += packet_ordering ifeq ($(CONFIG_RTE_ARCH_X86_64),y) DIRS-y += performance-thread diff --git a/examples/meson.build b/examples/meson.build index 87113bd70..a046b74ad 100644 --- a/examples/meson.build +++ b/examples/meson.build @@ -30,7 +30,7 @@ all_examples = [ 'multi_process/hotplug_mp', 'multi_process/simple_mp', 'multi_process/symmetric_mp', - 'netmap_compat', 'packet_ordering', + 'netmap_compat', 'ntb', 'packet_ordering', 'performance-thread', 'ptpclient', 'qos_meter', 'qos_sched', 'quota_watermark', 'rxtx_callbacks', diff --git a/examples/ntb/Makefile b/examples/ntb/Makefile new file mode 100644 index 000000000..5ddd9b95f --- /dev/null +++ b/examples/ntb/Makefile @@ -0,0 +1,68 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# binary name +APP = ntb_fwd + +# all source are stored in SRCS-y +SRCS-y := ntb_fwd.c + +# Build using pkg-config variables if possible +$(shell pkg-config --exists libdpdk) +ifeq ($(.SHELLSTATUS),0) + +all: shared +.PHONY: shared static +shared: build/$(APP)-shared + ln -sf $(APP)-shared build/$(APP) +static: build/$(APP)-static + ln -sf $(APP)-static build/$(APP) + +CFLAGS += -D_FILE_OFFSET_BITS=64 +LDFLAGS += -pthread + +PC_FILE := $(shell pkg-config --path libdpdk) +CFLAGS += -O3 $(shell pkg-config --cflags libdpdk) +LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk) +LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk) + +build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build + $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED) + +build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build + $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC) + +build: + @mkdir -p $@ + +.PHONY: clean +clean: + rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared + rmdir --ignore-fail-on-non-empty build + +else # Build using legacy build system + +ifeq ($(RTE_SDK),) +$(error "Please define RTE_SDK environment variable") +endif + +# Default target, can be overridden by command line or environment +RTE_TARGET ?= x86_64-native-linuxapp-gcc + +include $(RTE_SDK)/mk/rte.vars.mk + +ifneq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) +$(info This application can only operate in a linuxapp environment, \ +please change the definition of the RTE_TARGET environment variable) +all: +else + +CFLAGS += -D_FILE_OFFSET_BITS=64 +CFLAGS += -O2 +CFLAGS += $(WERROR_FLAGS) +CFLAGS += -DALLOW_EXPERIMENTAL_API + +include $(RTE_SDK)/mk/rte.extapp.mk + +endif +endif diff --git a/examples/ntb/meson.build b/examples/ntb/meson.build new file mode 100644 index 000000000..9a6288f4f --- /dev/null +++ b/examples/ntb/meson.build @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2019 Intel Corporation + +# meson file, for building this example as part of a main DPDK build. +# +# To build this example as a standalone application with an already-installed +# DPDK instance, use 'make' + +if host_machine.system() != 'linux' + build = false +endif +deps += 'rawdev' +cflags += ['-D_FILE_OFFSET_BITS=64'] +sources = files( + 'ntb_fwd.c' +) diff --git a/examples/ntb/ntb_fwd.c b/examples/ntb/ntb_fwd.c new file mode 100644 index 000000000..c169f01a3 --- /dev/null +++ b/examples/ntb/ntb_fwd.c @@ -0,0 +1,377 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2019 Intel Corporation + */ +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#define NTB_DRV_NAME_LEN 7 +static uint64_t max_file_size = 0x400000; +static uint8_t interactive = 1; +static uint16_t dev_id; + +/* *** Help command with introduction. *** */ +struct cmd_help_result { + cmdline_fixed_string_t help; +}; + +static void cmd_help_parsed(__attribute__((unused)) void *parsed_result, + struct cmdline *cl, + __attribute__((unused)) void *data) +{ + cmdline_printf( + cl, + "\n" + "The following commands are currently available:\n\n" + "Control:\n" + " quit :" + " Quit the application.\n" + "\nFile transmit:\n" + " send [path] :" + " Send [path] file. (No more than %"PRIu64")\n" + " recv [path] :" + " Receive file to [path]. Make sure sending is done" + " on the other side.\n", + max_file_size + ); + +} + +cmdline_parse_token_string_t cmd_help_help = + TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help"); + +cmdline_parse_inst_t cmd_help = { + .f = cmd_help_parsed, + .data = NULL, + .help_str = "show help", + .tokens = { + (void *)&cmd_help_help, + NULL, + }, +}; + +/* *** QUIT *** */ +struct cmd_quit_result { + cmdline_fixed_string_t quit; +}; + +static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result, + struct cmdline *cl, + __attribute__((unused)) void *data) +{ + /* Stop traffic and Close port. */ + rte_rawdev_stop(dev_id); + rte_rawdev_close(dev_id); + + cmdline_quit(cl); +} + +cmdline_parse_token_string_t cmd_quit_quit = + TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit"); + +cmdline_parse_inst_t cmd_quit = { + .f = cmd_quit_parsed, + .data = NULL, + .help_str = "exit application", + .tokens = { + (void *)&cmd_quit_quit, + NULL, + }, +}; + +/* *** SEND FILE PARAMETERS *** */ +struct cmd_sendfile_result { + cmdline_fixed_string_t send_string; + char filepath[]; +}; + +static void +cmd_sendfile_parsed(void *parsed_result, + __attribute__((unused)) struct cmdline *cl, + __attribute__((unused)) void *data) +{ + struct cmd_sendfile_result *res = parsed_result; + struct rte_rawdev_buf *pkts_send[1]; + uint64_t rsize, size, link; + uint8_t *buff; + uint32_t val; + FILE *file; + + if (!rte_rawdevs[dev_id].started) { + printf("Device needs to be up first. Try later.\n"); + return; + } + + rte_rawdev_get_attr(dev_id, "link_status", &link); + if (!link) { + printf("Link is not up, cannot send file.\n"); + return; + } + + file = fopen(res->filepath, "r"); + if (file == NULL) { + printf("Fail to open the file.\n"); + return; + } + + fseek(file, 0, SEEK_END); + size = ftell(file); + fseek(file, 0, SEEK_SET); + + /** + * No FIFO now. Only test memory. Limit sending file + * size <= max_file_size. + */ + if (size > max_file_size) { + printf("Warning: The file is too large. Only send first" + " %"PRIu64" bits.\n", max_file_size); + size = max_file_size; + } + + buff = (uint8_t *)malloc(size); + rsize = fread(buff, size, 1, file); + if (rsize != 1) { + printf("Fail to read file.\n"); + fclose(file); + free(buff); + return; + } + + /* Tell remote about the file size. */ + val = size >> 32; + rte_rawdev_set_attr(dev_id, "spad_user_0", val); + val = size; + rte_rawdev_set_attr(dev_id, "spad_user_1", val); + + pkts_send[0] = (struct rte_rawdev_buf *)malloc + (sizeof(struct rte_rawdev_buf)); + pkts_send[0]->buf_addr = buff; + + if (rte_rawdev_enqueue_buffers(dev_id, pkts_send, 1, + (void *)(size_t)size)) { + printf("Fail to enqueue.\n"); + goto clean; + } + printf("Done sending file.\n"); + +clean: + fclose(file); + free(buff); + free(pkts_send[0]); +} + +cmdline_parse_token_string_t cmd_send_file_send = + TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, send_string, + "send"); +cmdline_parse_token_string_t cmd_send_file_filepath = + TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, filepath, NULL); + + +cmdline_parse_inst_t cmd_send_file = { + .f = cmd_sendfile_parsed, + .data = NULL, + .help_str = "send ", + .tokens = { + (void *)&cmd_send_file_send, + (void *)&cmd_send_file_filepath, + NULL, + }, +}; + +/* *** RECEIVE FILE PARAMETERS *** */ +struct cmd_recvfile_result { + cmdline_fixed_string_t recv_string; + char filepath[]; +}; + +static void +cmd_recvfile_parsed(void *parsed_result, + __attribute__((unused)) struct cmdline *cl, + __attribute__((unused)) void *data) +{ + struct cmd_sendfile_result *res = parsed_result; + struct rte_rawdev_buf *pkts_recv[1]; + uint8_t *buff; + uint64_t val; + size_t size; + FILE *file; + + if (!rte_rawdevs[dev_id].started) { + printf("Device needs to be up first. Try later.\n"); + return; + } + + rte_rawdev_get_attr(dev_id, "link_status", &val); + if (!val) { + printf("Link is not up, cannot receive file.\n"); + return; + } + + file = fopen(res->filepath, "w"); + if (file == NULL) { + printf("Fail to open the file.\n"); + return; + } + + rte_rawdev_get_attr(dev_id, "spad_user_0", &val); + size = val << 32; + rte_rawdev_get_attr(dev_id, "spad_user_1", &val); + size |= val; + + buff = (uint8_t *)malloc(size); + pkts_recv[0] = (struct rte_rawdev_buf *)malloc + (sizeof(struct rte_rawdev_buf)); + pkts_recv[0]->buf_addr = buff; + + if (rte_rawdev_dequeue_buffers(dev_id, pkts_recv, 1, (void *)size)) { + printf("Fail to dequeue.\n"); + goto clean; + } + + fwrite(buff, size, 1, file); + printf("Done receiving to file.\n"); + +clean: + fclose(file); + free(buff); + free(pkts_recv[0]); +} + +cmdline_parse_token_string_t cmd_recv_file_recv = + TOKEN_STRING_INITIALIZER(struct cmd_recvfile_result, recv_string, + "recv"); +cmdline_parse_token_string_t cmd_recv_file_filepath = + TOKEN_STRING_INITIALIZER(struct cmd_recvfile_result, filepath, NULL); + + +cmdline_parse_inst_t cmd_recv_file = { + .f = cmd_recvfile_parsed, + .data = NULL, + .help_str = "recv ", + .tokens = { + (void *)&cmd_recv_file_recv, + (void *)&cmd_recv_file_filepath, + NULL, + }, +}; + +/* list of instructions */ +cmdline_parse_ctx_t main_ctx[] = { + (cmdline_parse_inst_t *)&cmd_help, + (cmdline_parse_inst_t *)&cmd_send_file, + (cmdline_parse_inst_t *)&cmd_recv_file, + (cmdline_parse_inst_t *)&cmd_quit, + NULL, +}; + +/* prompt function, called from main on MASTER lcore */ +static void +prompt(void) +{ + struct cmdline *cl; + + cl = cmdline_stdin_new(main_ctx, "ntb> "); + if (cl == NULL) + return; + + cmdline_interact(cl); + cmdline_stdin_exit(cl); +} + +static void +signal_handler(int signum) +{ + if (signum == SIGINT || signum == SIGTERM) { + printf("\nSignal %d received, preparing to exit...\n", signum); + signal(signum, SIG_DFL); + kill(getpid(), signum); + } +} + +static void +ntb_usage(const char *prgname) +{ + printf("%s [EAL options] -- [options]\n" + "-i : run in interactive mode (default value is 1)\n", + prgname); +} + +static int +parse_args(int argc, char **argv) +{ + char *prgname = argv[0], **argvopt = argv; + int opt, ret; + + /* Only support interactive mode to send/recv file first. */ + while ((opt = getopt(argc, argvopt, "i")) != EOF) { + switch (opt) { + case 'i': + printf("Interactive-mode selected\n"); + interactive = 1; + break; + + default: + ntb_usage(prgname); + return -1; + } + } + + if (optind >= 0) + argv[optind-1] = prgname; + + ret = optind-1; + optind = 1; /* reset getopt lib */ + return ret; +} + +int +main(int argc, char **argv) +{ + int ret, i; + + signal(SIGINT, signal_handler); + signal(SIGTERM, signal_handler); + + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Error with EAL initialization.\n"); + + /* Find 1st ntb rawdev. */ + for (i = 0; i < RTE_RAWDEV_MAX_DEVS; i++) + if (rte_rawdevs[i].driver_name && + (strncmp(rte_rawdevs[i].driver_name, "raw_ntb", + NTB_DRV_NAME_LEN) == 0) && (rte_rawdevs[i].attached == 1)) + break; + + if (i == RTE_RAWDEV_MAX_DEVS) + rte_exit(EXIT_FAILURE, "Cannot find any ntb device.\n"); + + dev_id = i; + + argc -= ret; + argv += ret; + + ret = parse_args(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Invalid arguments\n"); + + rte_rawdev_start(dev_id); + + if (interactive) { + sleep(1); + prompt(); + } + + return 0; +}