From patchwork Wed Mar 16 13:45:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108738 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 299F7A0032; Wed, 16 Mar 2022 14:46:11 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0583E41151; Wed, 16 Mar 2022 14:46:07 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id CCDF540395; Wed, 16 Mar 2022 14:46:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647438364; x=1678974364; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=w42y0b+vp/x5rgqV86/n/k3hG7cBj8/I10n/LtBAeSk=; b=SsMnsR8UqCFWZEaQ7ZQ+g2qffUzrOR6hmfnd3b8G+R4jKcPqGSsJAAhp P7p3c+JPVQDmIau22TIy3i4KnPflJKlNktwwwoZfUCNV7FJJ5NMJWcrLh Xo2ppktfptukQod3TtmZNRWnV2Z3fZtfYrEgvSFJdrnQb0GFM4OkXojcI lhPgHMsq2jpsFBwGXzOtSDbip7ZguaRSHRyHiVS26nS/bWjrSX17x9771 NhWRCbzZoZtpTxyvmBe5lCDEvrmp+gS7WaEbuFddDMMPF+U/HC0bROruj VGW8N3GdKzHOcLK+0XX/XT1lkVmNu/0pv7xOf9omBUm9PxshVDeW/+A5x g==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="256780535" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="256780535" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 06:46:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="714606475" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga005.jf.intel.com with ESMTP; 16 Mar 2022 06:46:01 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Bruce Richardson , stable@dpdk.org Subject: [PATCH v2 1/9] doc: replace special characters for (R) symbol in Linux GSG Date: Wed, 16 Mar 2022 13:45:43 +0000 Message-Id: <20220316134551.1099599-2-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220316134551.1099599-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> <20220316134551.1099599-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Some IDEs, such as eclipse, complained on save about the use of special characters in the (R) symbol in linux GSG doc. We can replace those with the equivalent "|reg|" text, and including isonum.txt. Cc: stable@dpdk.org Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/enable_func.rst | 8 +++++--- doc/guides/linux_gsg/linux_drivers.rst | 6 ++++-- doc/guides/linux_gsg/sys_reqs.rst | 6 ++++-- 3 files changed, 13 insertions(+), 7 deletions(-) -- 2.32.0 diff --git a/doc/guides/linux_gsg/enable_func.rst b/doc/guides/linux_gsg/enable_func.rst index 44c3b05130..1df3ab0255 100644 --- a/doc/guides/linux_gsg/enable_func.rst +++ b/doc/guides/linux_gsg/enable_func.rst @@ -1,6 +1,8 @@ .. SPDX-License-Identifier: BSD-3-Clause Copyright(c) 2010-2014 Intel Corporation. +.. include:: + .. _Enabling_Additional_Functionality: Enabling Additional Functionality @@ -66,15 +68,15 @@ system objects' permissions should be adjusted: Power Management and Power Saving Functionality ----------------------------------------------- -Enhanced Intel SpeedStep® Technology must be enabled in the platform BIOS if the power management feature of DPDK is to be used. +Enhanced Intel SpeedStep\ |reg| Technology must be enabled in the platform BIOS if the power management feature of DPDK is to be used. Otherwise, the sys file folder ``/sys/devices/system/cpu/cpu0/cpufreq`` will not exist, and the CPU frequency- based power management cannot be used. Consult the relevant BIOS documentation to determine how these settings can be accessed. -For example, on some Intel reference platform BIOS variants, the path to Enhanced Intel SpeedStep® Technology is:: +For example, on some Intel reference platform BIOS variants, the path to Enhanced Intel SpeedStep\ |reg| Technology is:: Advanced -> Processor Configuration - -> Enhanced Intel SpeedStep® Tech + -> Enhanced Intel SpeedStep\ |reg| Tech In addition, C3 and C6 should be enabled as well for power management. The path of C3 and C6 on the same platform BIOS is:: diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index 006a9df83e..ef6fec10d7 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -3,6 +3,8 @@ Copyright 2017 Mellanox Technologies, Ltd All rights reserved. +.. include:: + .. _linux_gsg_linux_drivers: Linux Drivers @@ -99,7 +101,7 @@ The token will be used for all PF and VF ports within the application. To make use of full VFIO functionality, both kernel and BIOS must support and be configured -to use IO virtualization (such as Intel® VT-d). +to use IO virtualization (such as Intel\ |reg| VT-d). .. note:: @@ -335,7 +337,7 @@ Please refer to earlier sections on how to configure kernel parameters correctly for your system. If the kernel is configured correctly, one also has to make sure that -the BIOS configuration has virtualization features (such as Intel® VT-d). +the BIOS configuration has virtualization features (such as Intel\ |reg| VT-d). There is no standard way to check if the platform is configured correctly, so please check with your platform documentation to see if it has such features, and how to enable them. diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst index b2eacac6dc..08d45898f0 100644 --- a/doc/guides/linux_gsg/sys_reqs.rst +++ b/doc/guides/linux_gsg/sys_reqs.rst @@ -1,6 +1,8 @@ .. SPDX-License-Identifier: BSD-3-Clause Copyright(c) 2010-2014 Intel Corporation. +.. include:: + System Requirements =================== @@ -68,10 +70,10 @@ Compilation of the DPDK **Optional Tools:** -* Intel® C++ Compiler (icc). For installation, additional libraries may be required. +* Intel\ |reg| C++ Compiler (icc). For installation, additional libraries may be required. See the icc Installation Guide found in the Documentation directory under the compiler installation. -* IBM® Advance ToolChain for Powerlinux. This is a set of open source development tools and runtime libraries +* IBM\ |reg| Advance ToolChain for Powerlinux. This is a set of open source development tools and runtime libraries which allows users to take leading edge advantage of IBM's latest POWER hardware features on Linux. To install it, see the IBM official installation document. From patchwork Wed Mar 16 13:45:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108739 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 87E6AA0032; Wed, 16 Mar 2022 14:46:17 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 43C8241157; Wed, 16 Mar 2022 14:46:08 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id BC5E8410EC; Wed, 16 Mar 2022 14:46:04 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647438365; x=1678974365; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=a5DksXrHV3EeBwq0MdM8HDawNxNFjRbGVHhGy34QaLs=; b=l1Dl2Y4WG0lZZ+ifytDunLZ47DCF8CbGPMK5sJC37VcpMvpC3GD4e5Qo x1noSYaiQ3/36gHx9qAB9ga0Ycg3Q56laM5mVeXZfM51O+/GWx8fBeMKj Ud+Do6SMjbdFe6uEOu226ocLd0fQ5+xXRu8VKAF/lD0ImURRNc+A+2OD1 jHHS5vxf1SbmuDHCktqJMK7ZBSrA3G9VyUt+yzn0pKBN9ql+nzmhUnf+S kN1VQGWQNsW5YTMdp8tfH1tUvtZtVHCfQNU1lPRlQ9CGZFyJ8lS31qhOR qwpR3BEipqlGtRBCnJfM61YjKuKBpdJENAPtnsrCRjdOmDfqulA4Sdydt A==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="256780545" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="256780545" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 06:46:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="714606480" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga005.jf.intel.com with ESMTP; 16 Mar 2022 06:46:02 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Bruce Richardson , thomas@monjalon.net, stable@dpdk.org Subject: [PATCH v2 2/9] doc: fix missing note on UIO module in Linux GSG Date: Wed, 16 Mar 2022 13:45:44 +0000 Message-Id: <20220316134551.1099599-3-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220316134551.1099599-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> <20220316134551.1099599-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The docs on binding drivers was updated as part of the removal of the igb_uio module from the main DPDK repo. As part of that update, a note about uio_pci_generic requiring legacy interrupts was removed, but should have been kept. Fixes: 56bb5841fd06 ("kernel/linux: remove igb_uio") Cc: thomas@monjalon.net Cc: stable@dpdk.org Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index ef6fec10d7..bd983b4d81 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -174,6 +174,11 @@ It can be loaded as shown below: sudo modprobe uio sudo insmod igb_uio.ko +.. note:: + + For some devices which lack support for legacy interrupts, e.g. virtual function + (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``. + .. note:: If UEFI secure boot is enabled, From patchwork Wed Mar 16 13:45:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108740 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6ED84A0032; Wed, 16 Mar 2022 14:46:24 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5E8764115F; Wed, 16 Mar 2022 14:46:09 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 98802410EC; Wed, 16 Mar 2022 14:46:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647438365; x=1678974365; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=I0SywmE4kawcto0DjwgmPxfnu/HH9iE033uSzVrxRI8=; b=G/AqMW4tfB2IAhZXHjwrXm/TARV/5hEVvlaai5Iiu1DJ3ZU+ek58xYMf ++9GfkRl4xjH/thrFZZKZKL/oeMUBbIaYEA7A5V+NR3ueuAGkfZVjJhQk zpgTCD2qZKc++td1wgPdiuT9hlqTFBDp0jLKqAeG98POxPaVDk6KBs9cp 8a1s30kTyJkVs9UKj/dDF7ubE64FgXkSBXTb0I1SmpoH70tddvpXFnoSU g9urC9DF3G9iQ1iES+QghShz9KS15NoRoSh/N8co1ae0u9qHaZ+sFxUEe nVnxxew8D863A9MHxSCrpWN0idku2oPkdsePjS6R5+hvxLhFhmsN1J9Kb w==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="256780555" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="256780555" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 06:46:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="714606485" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga005.jf.intel.com with ESMTP; 16 Mar 2022 06:46:03 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Bruce Richardson , stable@dpdk.org Subject: [PATCH v2 3/9] doc: make UIO safety warning more visible in Linux GSG Date: Wed, 16 Mar 2022 13:45:45 +0000 Message-Id: <20220316134551.1099599-4-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220316134551.1099599-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> <20220316134551.1099599-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The GSG has a note warning that use of UIO is inherently unsafe due to lack of IOMMU protection. However, this was only flagged as a "NOTE", meaning it could easily be missed. Changing the rst tag from "note" to "warning" and moving it to the top of the UIO subsection makes this a lot more visible to users. Cc: stable@dpdk.org Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) -- 2.32.0 diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index bd983b4d81..f0a274df6a 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -153,6 +153,11 @@ After that, VFIO can be used with hardware devices as usual. UIO --- +.. warning:: + + Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection, + and can only be done by root user. + In situations where using VFIO is not an option, there are alternative drivers one can use. In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel can be used as a substitute for VFIO. This module can be loaded using the command: @@ -195,11 +200,6 @@ It can be loaded as shown below: in GRUB command line on x86_64 systems, or add ``iommu.passthrough=1`` on aarch64 systems. -.. note:: - - Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection, - and can only be done by root user. - .. _bifurcated_driver: Bifurcated Driver From patchwork Wed Mar 16 13:45:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108741 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1FF19A0032; Wed, 16 Mar 2022 14:46:32 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AAE444116D; Wed, 16 Mar 2022 14:46:11 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 81023410F6; Wed, 16 Mar 2022 14:46:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647438366; x=1678974366; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=kUyiUYiQ4N5ICcUFZJFBFjUUdD7hNw1MZ2gzwc3dmTw=; b=gckCfsKJQmgqgrRHEPUIy8Vvz8z6RIzHVFfp/2JfWKuJ73J0yzJpil5s D+zBKYH/z7o/Il32ORN7vN5Cv1E8m0N3XVqVf1m9+WSsC4DK5/IcufrXO YGP4vyWOjqxB/RDQf2lEUZ+OtPQ43L25wGjDCNR2RMKdEPMyfBeDuQutK OY+PJp6ovwZpBAifzpH8eWhHbv+KomOeblgi+XizGNmgC0VyA0ktIcaIl 9lixrclKtJj0pgy0PLFphteV9cIi1JZbxXasu+fuz80wSxNVpk/L3z3Am ySpCy4G9WMh/qb8sOT+++LQfecgUCzJllMwUfosdR+4YUBm2D9gWM+nhY A==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="256780565" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="256780565" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 06:46:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="714606495" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga005.jf.intel.com with ESMTP; 16 Mar 2022 06:46:04 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Bruce Richardson , stable@dpdk.org Subject: [PATCH v2 4/9] doc: move section on binding drivers up the page in GSG Date: Wed, 16 Mar 2022 13:45:46 +0000 Message-Id: <20220316134551.1099599-5-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220316134551.1099599-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> <20220316134551.1099599-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org While the details of VFIO and UIO may be of interest to some, most users of the doc are likely primarily interested in how to bind their devices to the kernel driver and then move on to running the app. Therefore, the most important part of the "Linux Drivers" section of the GSG is the subsection on "Binding and Unbinding", so put that first. Cc: stable@dpdk.org Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 179 +++++++++++++------------ 1 file changed, 90 insertions(+), 89 deletions(-) -- 2.32.0 diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index f0a274df6a..b93feae7a1 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -14,6 +14,96 @@ Different PMDs may require different kernel drivers in order to work properly. Depending on the PMD being used, a corresponding kernel driver should be loaded, and network ports should be bound to that driver. +.. _linux_gsg_binding_kernel: + +Binding and Unbinding Network Ports to/from the Kernel Modules +-------------------------------------------------------------- + +.. note:: + + PMDs which use the bifurcated driver should not be unbound from their kernel drivers. + This section is for PMDs which use the UIO or VFIO drivers. + See :ref:`bifurcated_driver` section for more details. + +As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use. +Instead, in case the PMD being used use the VFIO or UIO drivers, +all ports that are to be used by a DPDK application must be bound to +the ``vfio-pci``, ``uio_pci_generic``, or ``igb_uio`` module +before the application is run. +For such PMDs, any network ports under Linux* control will be ignored and cannot be used by the application. + +To bind ports to the ``vfio-pci``, ``uio_pci_generic`` or ``igb_uio`` module +for DPDK use, or to return ports to Linux control, +a utility script called ``dpdk-devbind.py`` is provided in the ``usertools`` subdirectory. +This utility can be used to provide a view of the current state of the network ports on the system, +and to bind and unbind those ports from the different kernel modules, +including the VFIO and UIO modules. +The following are some examples of how the script can be used. +A full description of the script and its parameters can be obtained +by calling the script with the ``--help`` or ``--usage`` options. +Note that the UIO or VFIO kernel modules to be used, +should be loaded into the kernel before running the ``dpdk-devbind.py`` script. + +.. warning:: + + Due to the way VFIO works, there are certain limitations + to which devices can be used with VFIO. + Mainly it comes down to how IOMMU groups work. + Any Virtual Function device can usually be used with VFIO on its own, + but physical devices may require either all ports bound to VFIO, + or some of them bound to VFIO while others not being bound to anything at all. + + If your device is behind a PCI-to-PCI bridge, + the bridge will then be part of the IOMMU group in which your device is in. + Therefore, the bridge driver should also be unbound from the bridge PCI device + for VFIO to work with devices behind the bridge. + +.. warning:: + + While any user can run the ``dpdk-devbind.py`` script + to view the status of the network ports, + binding or unbinding network ports requires root privileges. + +To see the status of all network ports on the system: + +.. code-block:: console + + ./usertools/dpdk-devbind.py --status + + Network devices using DPDK-compatible driver + ============================================ + 0000:82:00.0 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe + 0000:82:00.1 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe + + Network devices using kernel driver + =================================== + 0000:04:00.0 'I350 1-GbE NIC' if=em0 drv=igb unused=uio_pci_generic *Active* + 0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=uio_pci_generic + 0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=uio_pci_generic + 0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=uio_pci_generic + + Other network devices + ===================== + + +To bind device ``eth1``,``04:00.1``, to the ``uio_pci_generic`` driver: + +.. code-block:: console + + ./usertools/dpdk-devbind.py --bind=uio_pci_generic 04:00.1 + +or, alternatively, + +.. code-block:: console + + ./usertools/dpdk-devbind.py --bind=uio_pci_generic eth1 + +To restore device ``82:00.0`` to its original kernel binding: + +.. code-block:: console + + ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.0 + VFIO ---- @@ -225,95 +315,6 @@ More about the bifurcated driver can be found in `Mellanox Bifurcated DPDK PMD `__. -.. _linux_gsg_binding_kernel: - -Binding and Unbinding Network Ports to/from the Kernel Modules --------------------------------------------------------------- - -.. note:: - - PMDs which use the bifurcated driver should not be unbound from their kernel drivers. - This section is for PMDs which use the UIO or VFIO drivers. - -As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use. -Instead, in case the PMD being used use the VFIO or UIO drivers, -all ports that are to be used by a DPDK application must be bound to -the ``vfio-pci``, ``uio_pci_generic``, or ``igb_uio`` module -before the application is run. -For such PMDs, any network ports under Linux* control will be ignored and cannot be used by the application. - -To bind ports to the ``vfio-pci``, ``uio_pci_generic`` or ``igb_uio`` module -for DPDK use, or to return ports to Linux control, -a utility script called ``dpdk-devbind.py`` is provided in the ``usertools`` subdirectory. -This utility can be used to provide a view of the current state of the network ports on the system, -and to bind and unbind those ports from the different kernel modules, -including the VFIO and UIO modules. -The following are some examples of how the script can be used. -A full description of the script and its parameters can be obtained -by calling the script with the ``--help`` or ``--usage`` options. -Note that the UIO or VFIO kernel modules to be used, -should be loaded into the kernel before running the ``dpdk-devbind.py`` script. - -.. warning:: - - Due to the way VFIO works, there are certain limitations - to which devices can be used with VFIO. - Mainly it comes down to how IOMMU groups work. - Any Virtual Function device can usually be used with VFIO on its own, - but physical devices may require either all ports bound to VFIO, - or some of them bound to VFIO while others not being bound to anything at all. - - If your device is behind a PCI-to-PCI bridge, - the bridge will then be part of the IOMMU group in which your device is in. - Therefore, the bridge driver should also be unbound from the bridge PCI device - for VFIO to work with devices behind the bridge. - -.. warning:: - - While any user can run the ``dpdk-devbind.py`` script - to view the status of the network ports, - binding or unbinding network ports requires root privileges. - -To see the status of all network ports on the system: - -.. code-block:: console - - ./usertools/dpdk-devbind.py --status - - Network devices using DPDK-compatible driver - ============================================ - 0000:82:00.0 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe - 0000:82:00.1 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe - - Network devices using kernel driver - =================================== - 0000:04:00.0 'I350 1-GbE NIC' if=em0 drv=igb unused=uio_pci_generic *Active* - 0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=uio_pci_generic - 0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=uio_pci_generic - 0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=uio_pci_generic - - Other network devices - ===================== - - -To bind device ``eth1``,``04:00.1``, to the ``uio_pci_generic`` driver: - -.. code-block:: console - - ./usertools/dpdk-devbind.py --bind=uio_pci_generic 04:00.1 - -or, alternatively, - -.. code-block:: console - - ./usertools/dpdk-devbind.py --bind=uio_pci_generic eth1 - -To restore device ``82:00.0`` to its original kernel binding: - -.. code-block:: console - - ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.0 - Troubleshooting VFIO -------------------- From patchwork Wed Mar 16 13:45:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108742 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4DCEDA0032; Wed, 16 Mar 2022 14:46:39 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E2EF741174; Wed, 16 Mar 2022 14:46:14 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id C3104410F6; Wed, 16 Mar 2022 14:46:07 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647438368; x=1678974368; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6T7m2SDFr6Ll49aO5uLStzvMbZCAJFecSt/Wpj1zHcA=; b=UB85sJSnsBEw3fba8IKIC1WW3DhQGr9tXPMfYi/tE5Awn9zyBUfIBeRF M4xKcY5zYRN6/MTKfQm4zIb0E0k2bQzUSkH+Q0QXOyGXbBP01Ahoh/BR2 yCZrUTtxyuEEcY0XBjhsob0QtXT/Q0p2OFUtwrIH0sMLun/TevPppCp1C C1Mvw/AOStzb3L+lTu2Mcjdf7OyRg3s3qlRPNsOOAN/HFNEOyLv+0fY8b yhJCzTkry8ENiKlkDivFTO4wIJIiNto9n7GAUAJw13S3UmeNFtXc1BE8w 7urzgZ7A0yH/EbW3iAHsMM+XWvcGmXi07tnCsWRwcivTJHX6g7vpM3OIT A==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="256780570" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="256780570" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 06:46:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="714606501" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga005.jf.intel.com with ESMTP; 16 Mar 2022 06:46:06 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Bruce Richardson , stable@dpdk.org Subject: [PATCH v2 5/9] doc: emphasise VFIO over UIO-based modules in GSG Date: Wed, 16 Mar 2022 13:45:47 +0000 Message-Id: <20220316134551.1099599-6-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220316134551.1099599-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> <20220316134551.1099599-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org VFIO is to be strongly preferred over uio-based modules, so update our text and examples to only refer to vfio, giving an initial reference at the start to uio as a fallback option. Cc: stable@dpdk.org Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 47 +++++++++++++++----------- 1 file changed, 28 insertions(+), 19 deletions(-) -- 2.32.0 diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index b93feae7a1..bd649db929 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -25,14 +25,18 @@ Binding and Unbinding Network Ports to/from the Kernel Modules This section is for PMDs which use the UIO or VFIO drivers. See :ref:`bifurcated_driver` section for more details. -As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use. -Instead, in case the PMD being used use the VFIO or UIO drivers, -all ports that are to be used by a DPDK application must be bound to -the ``vfio-pci``, ``uio_pci_generic``, or ``igb_uio`` module -before the application is run. -For such PMDs, any network ports under Linux* control will be ignored and cannot be used by the application. - -To bind ports to the ``vfio-pci``, ``uio_pci_generic`` or ``igb_uio`` module +.. note:: + + It is recommended that ``vfio-pci`` be used as the kernel module for DPDK-bound ports in all cases. + If an IOMMU is unavailable, the ``vfio-pci`` can be used in :ref:`no-iommu` mode. + If, for some reason, vfio is unavailable, then UIO-based modules, ``igb_uio`` and ``uio_pci_generic`` may be used. + See section :ref:`uio` for details. + +Most devices require that the hardware to be used by DPDK be unbound from the kernel driver it uses, +and instead be bound to the ``vfio-pci`` kernel module before the application is run. +For such PMDs, any network ports or other hardware under Linux* control will be ignored and cannot be used by the application. + +To bind ports to the ``vfio-pci`` module for DPDK use, or to return ports to Linux control, a utility script called ``dpdk-devbind.py`` is provided in the ``usertools`` subdirectory. This utility can be used to provide a view of the current state of the network ports on the system, @@ -72,37 +76,38 @@ To see the status of all network ports on the system: Network devices using DPDK-compatible driver ============================================ - 0000:82:00.0 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe - 0000:82:00.1 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe + 0000:82:00.0 '82599EB 10-GbE NIC' drv=vfio-pci unused=ixgbe + 0000:82:00.1 '82599EB 10-GbE NIC' drv=vfio-pci unused=ixgbe Network devices using kernel driver =================================== - 0000:04:00.0 'I350 1-GbE NIC' if=em0 drv=igb unused=uio_pci_generic *Active* - 0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=uio_pci_generic - 0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=uio_pci_generic - 0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=uio_pci_generic + 0000:04:00.0 'I350 1-GbE NIC' if=em0 drv=igb unused=vfio-pci *Active* + 0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=vfio-pci + 0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=vfio-pci + 0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=vfio-pci Other network devices ===================== -To bind device ``eth1``,``04:00.1``, to the ``uio_pci_generic`` driver: +To bind device ``eth1``,``04:00.1``, to the ``vfio-pci`` driver: .. code-block:: console - ./usertools/dpdk-devbind.py --bind=uio_pci_generic 04:00.1 + ./usertools/dpdk-devbind.py --bind=vfio-pci 04:00.1 or, alternatively, .. code-block:: console - ./usertools/dpdk-devbind.py --bind=uio_pci_generic eth1 + ./usertools/dpdk-devbind.py --bind=vfio-pci eth1 -To restore device ``82:00.0`` to its original kernel binding: +When specifying device ids, wildcards can be used for the final part of the address. +To restore device ``82:00.0`` and ``82:00.1`` to their original kernel binding: .. code-block:: console - ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.0 + ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.* VFIO ---- @@ -210,6 +215,8 @@ to use IO virtualization (such as Intel\ |reg| VT-d). For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up. For more information, please refer to :ref:`Running_Without_Root_Privileges`. +.. _vfio_noiommu: + VFIO no-IOMMU mode ------------------ @@ -240,6 +247,8 @@ After that, VFIO can be used with hardware devices as usual. to keep the degree of device access and programming that VFIO has, in situations where IOMMU is not available. +.. _uio: + UIO --- From patchwork Wed Mar 16 13:45:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108743 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CA27AA0032; Wed, 16 Mar 2022 14:46:44 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 080C441178; Wed, 16 Mar 2022 14:46:16 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id F29D641152; Wed, 16 Mar 2022 14:46:08 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647438369; x=1678974369; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9rm1Z0cfODzs2d4Rg/+54CeEHueOyU095nFeb+cikL4=; b=P2B9tUdl+trC++3g4Pi7TCZv6anjXX6x9pEh8mongQrk/PXnJAhjYhjP KmV8NgaG464b54F3tGtpeuUbm43vEc5RFmrSHPRTnGoKsflf8uCj1hPyo z3vQ11ml/JFrhffT0PhqPtEgL9QEJ/dK/poMoGHTaPgkaor1E1GB33x9i JJEhMK1aHo+STk5GGd+xIQQMs+JSbcCa9cGgeuVZQUPnlhXe5KhjZYrZa 9kCpz6SP3NSIuAMVYgoR2gs0NiPpYeJmZHCF44G3fTNKklBzs6e5KEwtH 9U2GCT1vWx8yvb8S3Sem9biXTjCrTvjju5nWVK+FFQ19LJaw82MnoONzu Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="256780574" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="256780574" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 06:46:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="714606509" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga005.jf.intel.com with ESMTP; 16 Mar 2022 06:46:07 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Bruce Richardson , stable@dpdk.org Subject: [PATCH v2 6/9] doc: split GSG VFIO section into subsections Date: Wed, 16 Mar 2022 13:45:48 +0000 Message-Id: <20220316134551.1099599-7-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220316134551.1099599-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> <20220316134551.1099599-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The VFIO section of the page about linux drivers was rather long and unstructured. This can be improved by splitting it up into subsections, to cover the specifics of memory limits and creating VFs. When moving the various text notes into the relevant subsections, we can drop the note about kernels earlier than 3.6, since DPDK no longer supports kernels that old. Cc: stable@dpdk.org Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 35 ++++++++++++++------------ 1 file changed, 19 insertions(+), 16 deletions(-) -- 2.32.0 diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index bd649db929..f9c24e9456 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -122,6 +122,22 @@ To make use of VFIO, the ``vfio-pci`` module must be loaded: VFIO kernel is usually present by default in all distributions, however please consult your distributions documentation to make sure that is the case. +To make use of full VFIO functionality, +both kernel and BIOS must support and be configured +to use IO virtualization (such as Intel\ |reg| VT-d). + +.. note:: + + In most cases, specifying "iommu=on" as kernel parameter should be enough to + configure the Linux kernel to use IOMMU. + +For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up. +For more information, please refer to :ref:`Running_Without_Root_Privileges`. + + +VFIO Memory Mapping Limits +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + For DMA mapping of either external memory or hugepages, VFIO interface is used. VFIO does not support partial unmap of once mapped memory. Hence DPDK's memory is mapped in hugepage granularity or system page granularity. Number of DMA @@ -132,6 +148,9 @@ VFIO module parameter ``dma_entry_limit`` with a default value of 64K. When application is out of DMA entries, these limits need to be adjusted to increase the allowed limit. +Creating Virtual Functions using vfio-pci +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + Since Linux version 5.7, the ``vfio-pci`` module supports the creation of virtual functions. After the PF is bound to ``vfio-pci`` module, @@ -194,27 +213,11 @@ The token will be used for all PF and VF ports within the application. /app/dpdk-testpmd -l 26-29 -n 4 -a 86:02.0 \ --vfio-vf-token=14d63f20-8445-11ea-8900-1f9ce7d5650d --file-prefix=vf0 -- -i -To make use of full VFIO functionality, -both kernel and BIOS must support and be configured -to use IO virtualization (such as Intel\ |reg| VT-d). - -.. note:: - - Linux versions earlier than version 3.6 do not support VFIO. - .. note:: Linux versions earlier than version 5.7 do not support the creation of virtual functions within the VFIO framework. -.. note:: - - In most cases, specifying "iommu=on" as kernel parameter should be enough to - configure the Linux kernel to use IOMMU. - -For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up. -For more information, please refer to :ref:`Running_Without_Root_Privileges`. - .. _vfio_noiommu: VFIO no-IOMMU mode From patchwork Wed Mar 16 13:45:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108745 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 70EA2A0032; Wed, 16 Mar 2022 14:47:00 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BE1AF41172; Wed, 16 Mar 2022 14:46:22 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 396644116E; Wed, 16 Mar 2022 14:46:14 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647438374; x=1678974374; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Cj9LcwcfQ2wRE6ZH8iBKgBeeTq3XhxZdVgW1sWmKzGQ=; b=G/oBsItoOPUd3fr5DzEh6Y8DTDxrVIjY8CqvBvBBTeZcToX9vMoqbJkm iOnOZKAHozlTNBhYJSp8cCDb6nokVqhLHKCMTUtOlHLQzJ4vQvPcFHO4p iD36xZXYMu11E/dQfnOooKxpoxNs07MV3bpD23Nm6FIYYPpWy0PtdP8gp 6U8SzZOL8KSWMp/y8go3OehvMPyCb+Ip8jFOcFgV3ArBTJGE5Pao5cC3O fnzTFaY7D5Ev8l1GGH0xYxBUcTFRu4cA/per/ecElJBdWGWeMbQPRuu0b KH3CJAwX8rn16s8mhPs6mLMIpYgMda7V3Rs9PmYETa8GbWOmmhm41e1fq w==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="256780596" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="256780596" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 06:46:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="714606516" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga005.jf.intel.com with ESMTP; 16 Mar 2022 06:46:08 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Bruce Richardson , stable@dpdk.org Subject: [PATCH v2 7/9] doc: move GSG section on UIO to the end of drivers page Date: Wed, 16 Mar 2022 13:45:49 +0000 Message-Id: <20220316134551.1099599-8-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220316134551.1099599-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> <20220316134551.1099599-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org To further de-emphasise UIO over the alternatives, we can move the UIO section of the drivers page to the end of the document, giving more prominence to VFIO and bifurcated drivers. Cc: stable@dpdk.org Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 104 ++++++++++++------------- 1 file changed, 52 insertions(+), 52 deletions(-) -- 2.32.0 diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index f9c24e9456..6ff8586940 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -250,58 +250,6 @@ After that, VFIO can be used with hardware devices as usual. to keep the degree of device access and programming that VFIO has, in situations where IOMMU is not available. -.. _uio: - -UIO ---- - -.. warning:: - - Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection, - and can only be done by root user. - -In situations where using VFIO is not an option, there are alternative drivers one can use. -In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel -can be used as a substitute for VFIO. This module can be loaded using the command: - -.. code-block:: console - - sudo modprobe uio_pci_generic - -.. note:: - - ``uio_pci_generic`` module doesn't support the creation of virtual functions. - -As an alternative to the ``uio_pci_generic``, there is the ``igb_uio`` module -which can be found in the repository `dpdk-kmods `_. -It can be loaded as shown below: - -.. code-block:: console - - sudo modprobe uio - sudo insmod igb_uio.ko - -.. note:: - - For some devices which lack support for legacy interrupts, e.g. virtual function - (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``. - -.. note:: - - If UEFI secure boot is enabled, - the Linux kernel may disallow the use of UIO on the system. - Therefore, devices for use by DPDK should be bound to the ``vfio-pci`` kernel module - rather than any UIO-based module. - For more details see :ref:`linux_gsg_binding_kernel` below. - -.. note:: - - If the devices used for DPDK are bound to a UIO-based kernel module, - please make sure that the IOMMU is disabled or is in passthrough mode. - One can add ``intel_iommu=off`` or ``amd_iommu=off`` or ``intel_iommu=on iommu=pt`` - in GRUB command line on x86_64 systems, - or add ``iommu.passthrough=1`` on aarch64 systems. - .. _bifurcated_driver: Bifurcated Driver @@ -372,3 +320,55 @@ This can be checked in the boot configuration of your system: If ``CONFIG_VFIO_NOIOMMU`` is not enabled in the kernel configuration, VFIO driver will not support the no-IOMMU mode, and other alternatives (such as UIO drivers) will have to be used. + +.. _uio: + +UIO +--- + +.. warning:: + + Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection, + and can only be done by root user. + +In situations where using VFIO is not an option, there are alternative drivers one can use. +In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel +can be used as a substitute for VFIO. This module can be loaded using the command: + +.. code-block:: console + + sudo modprobe uio_pci_generic + +.. note:: + + ``uio_pci_generic`` module doesn't support the creation of virtual functions. + +As an alternative to the ``uio_pci_generic``, there is the ``igb_uio`` module +which can be found in the repository `dpdk-kmods `_. +It can be loaded as shown below: + +.. code-block:: console + + sudo modprobe uio + sudo insmod igb_uio.ko + +.. note:: + + For some devices which lack support for legacy interrupts, e.g. virtual function + (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``. + +.. note:: + + If UEFI secure boot is enabled, + the Linux kernel may disallow the use of UIO on the system. + Therefore, devices for use by DPDK should be bound to the ``vfio-pci`` kernel module + rather than any UIO-based module. + For more details see :ref:`linux_gsg_binding_kernel` below. + +.. note:: + + If the devices used for DPDK are bound to a UIO-based kernel module, + please make sure that the IOMMU is disabled or is in passthrough mode. + One can add ``intel_iommu=off`` or ``amd_iommu=off`` or ``intel_iommu=on iommu=pt`` + in GRUB command line on x86_64 systems, + or add ``iommu.passthrough=1`` on aarch64 systems. From patchwork Wed Mar 16 13:45:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108744 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A41C6A0032; Wed, 16 Mar 2022 14:46:54 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DE7744118F; Wed, 16 Mar 2022 14:46:17 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 8897141158; Wed, 16 Mar 2022 14:46:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647438373; x=1678974373; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3YHzXpMZ9jD6DfBX+P1IcyBU8wxnvL0LoCGbgefYMLI=; b=jOrqfWFPeYC1N56cqmPl/5Z9wqKOl7ghggDoS8C+9fZSddJDb/lW1hIU ruEz427d//OOapCZArUubj159TAKLDYNbXLLI15h1KdzrpTuZ81wvWvrl VIHj+Zhspsz7WN+iLpz36CoLhslohZgkChKkrJJxtOtRKFjESAyKb5jJ/ itYhCbuyXc9vIZzowUxf6KSGq4bTkT/RJzuyA/dOboj/Ax120XmtJkPsi yUllny3jXhhSBIXXlD/k9Ce5LYaOHsShTgoHMxncQ6VzwPnyHZaBAQLOI SMfyQQ7Jf23+uMpHN/c801NKzk9Zv0Dy79mNO/OXggxjnmBCk1THFM7zO g==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="256780595" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="256780595" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 06:46:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="714606525" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga005.jf.intel.com with ESMTP; 16 Mar 2022 06:46:10 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Bruce Richardson , stable@dpdk.org Subject: [PATCH v2 8/9] doc: consolidate all VFIO content on GSG driver page Date: Wed, 16 Mar 2022 13:45:50 +0000 Message-Id: <20220316134551.1099599-9-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220316134551.1099599-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> <20220316134551.1099599-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Rather than having separate sections for VFIO and VFIO no-iommu mode, as well as a separate section further down the document on troubleshooting VFIO, we can consolidate all these as subsections into a primary VFIO section. This section starts with the basics of VFIO use, then covers no-iommu mode, before moving on to the more advanced topics such as creating VFs and ending with the troubleshooting subsection. Cc: stable@dpdk.org Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 116 ++++++++++++------------- 1 file changed, 58 insertions(+), 58 deletions(-) -- 2.32.0 diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index 6ff8586940..8320db44d9 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -135,6 +135,38 @@ For proper operation of VFIO when running DPDK applications as a non-privileged For more information, please refer to :ref:`Running_Without_Root_Privileges`. +.. _vfio_noiommu: + +VFIO no-IOMMU mode +~~~~~~~~~~~~~~~~~~ + +If there is no IOMMU available on the system, VFIO can still be used, +but it has to be loaded with an additional module parameter: + +.. code-block:: console + + modprobe vfio enable_unsafe_noiommu_mode=1 + +Alternatively, one can also enable this option in an already loaded kernel module: + +.. code-block:: console + + echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode + +After that, VFIO can be used with hardware devices as usual. + +.. note:: + + It may be required to unload all VFIO related-modules before probing + the module again with ``enable_unsafe_noiommu_mode=1`` parameter. + +.. warning:: + + Since no-IOMMU mode forgoes IOMMU protection, it is inherently unsafe. + That said, it does make it possible for the user + to keep the degree of device access and programming that VFIO has, + in situations where IOMMU is not available. + VFIO Memory Mapping Limits ~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -218,65 +250,8 @@ The token will be used for all PF and VF ports within the application. Linux versions earlier than version 5.7 do not support the creation of virtual functions within the VFIO framework. -.. _vfio_noiommu: - -VFIO no-IOMMU mode ------------------- - -If there is no IOMMU available on the system, VFIO can still be used, -but it has to be loaded with an additional module parameter: - -.. code-block:: console - - modprobe vfio enable_unsafe_noiommu_mode=1 - -Alternatively, one can also enable this option in an already loaded kernel module: - -.. code-block:: console - - echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode - -After that, VFIO can be used with hardware devices as usual. - -.. note:: - - It may be required to unload all VFIO related-modules before probing - the module again with ``enable_unsafe_noiommu_mode=1`` parameter. - -.. warning:: - - Since no-IOMMU mode forgoes IOMMU protection, it is inherently unsafe. - That said, it does make it possible for the user - to keep the degree of device access and programming that VFIO has, - in situations where IOMMU is not available. - -.. _bifurcated_driver: - -Bifurcated Driver ------------------ - -PMDs which use the bifurcated driver co-exists with the device kernel driver. -On such model the NIC is controlled by the kernel, while the data -path is performed by the PMD directly on top of the device. - -Such model has the following benefits: - - - It is secure and robust, as the memory management and isolation - is done by the kernel. - - It enables the user to use legacy linux tools such as ``ethtool`` or - ``ifconfig`` while running DPDK application on the same network ports. - - It enables the DPDK application to filter only part of the traffic, - while the rest will be directed and handled by the kernel driver. - The flow bifurcation is performed by the NIC hardware. - As an example, using :ref:`flow_isolated_mode` allows to choose - strictly what is received in DPDK. - -More about the bifurcated driver can be found in -`Mellanox Bifurcated DPDK PMD -`__. - Troubleshooting VFIO --------------------- +~~~~~~~~~~~~~~~~~~~~ In certain situations, using ``dpdk-devbind.py`` script to bind a device to VFIO driver may fail. @@ -321,6 +296,31 @@ If ``CONFIG_VFIO_NOIOMMU`` is not enabled in the kernel configuration, VFIO driver will not support the no-IOMMU mode, and other alternatives (such as UIO drivers) will have to be used. +.. _bifurcated_driver: + +Bifurcated Driver +----------------- + +PMDs which use the bifurcated driver co-exists with the device kernel driver. +On such model the NIC is controlled by the kernel, while the data +path is performed by the PMD directly on top of the device. + +Such model has the following benefits: + + - It is secure and robust, as the memory management and isolation + is done by the kernel. + - It enables the user to use legacy linux tools such as ``ethtool`` or + ``ifconfig`` while running DPDK application on the same network ports. + - It enables the DPDK application to filter only part of the traffic, + while the rest will be directed and handled by the kernel driver. + The flow bifurcation is performed by the NIC hardware. + As an example, using :ref:`flow_isolated_mode` allows to choose + strictly what is received in DPDK. + +More about the bifurcated driver can be found in +`Mellanox Bifurcated DPDK PMD +`__. + .. _uio: UIO From patchwork Wed Mar 16 13:45:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108746 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DBE2AA0032; Wed, 16 Mar 2022 14:47:07 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id ABE43411C9; Wed, 16 Mar 2022 14:46:24 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 93E0541158; Wed, 16 Mar 2022 14:46:14 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647438374; x=1678974374; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=i7sAb+4CXMVETEqtFbHMACw20Dz9ht57/zvVKBxNcAE=; b=GKwItvKIZHUaEhizxByk1LnvQVvL+B3RZNkN1445B7JQ1wQZyQpjVwXm 8n7NVa6rC94a8iMCoLU9O+Jnq9+DSeGJ3SbK1ShWanBzYl5XKkUThbC4F Gk75tB3VT1heI/bSb+byxDBLelL3N+iLmG7pfMsgA8FjlSsxdMlVHtQ+b e50yoEf5DZgawQZ3fKVs6n+plMAKbjKSUJ66v6If6nlypyFWvEkAtCv20 Xr76f4q3viDoEor5tvAIvQ7oTKVJO4upM1viSzBxUwfSP/D0MPU+/bWmA +fS4uYObZXmxvfrX4y1IKsq+DVMCHyUEo7fdTogTqdBbgEUrUauSDLkrC g==; X-IronPort-AV: E=McAfee;i="6200,9189,10286"; a="256780600" X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="256780600" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Mar 2022 06:46:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,186,1643702400"; d="scan'208";a="714606538" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga005.jf.intel.com with ESMTP; 16 Mar 2022 06:46:11 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Bruce Richardson , stable@dpdk.org Subject: [PATCH v2 9/9] doc: change informational warnings to notes in Linux GSG Date: Wed, 16 Mar 2022 13:45:51 +0000 Message-Id: <20220316134551.1099599-10-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220316134551.1099599-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> <20220316134551.1099599-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There are two warnings in the VFIO section about limitations of VFIO and limitations on who can bind/unbind devices. Since these don't actually describe any unsafe conditions, and are more informational, we can change these to notes. This also helps emphasise the other warnings in the documents which flag genuine security concerns. Cc: stable@dpdk.org Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 2.32.0 diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index 8320db44d9..2e4c80ebd3 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -48,7 +48,7 @@ by calling the script with the ``--help`` or ``--usage`` options. Note that the UIO or VFIO kernel modules to be used, should be loaded into the kernel before running the ``dpdk-devbind.py`` script. -.. warning:: +.. note:: Due to the way VFIO works, there are certain limitations to which devices can be used with VFIO. @@ -62,7 +62,7 @@ should be loaded into the kernel before running the ``dpdk-devbind.py`` script. Therefore, the bridge driver should also be unbound from the bridge PCI device for VFIO to work with devices behind the bridge. -.. warning:: +.. note:: While any user can run the ``dpdk-devbind.py`` script to view the status of the network ports,