From patchwork Wed Mar 2 17:22:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108495 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EE721A04A4; Wed, 2 Mar 2022 18:24:15 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4165842755; Wed, 2 Mar 2022 18:24:12 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id D8B0C40141; Wed, 2 Mar 2022 18:24:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646241850; x=1677777850; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=eCcywuCAUWJPmxqv89P4a0UddBACsV8aKBVoRjE5Kco=; b=bWReswJdqyxFrZx8yKzelK5nJ4RfWRrHxO4Y3qkU1/PxADfrE2AJIEhr G4tzs2EPzL6pJcgchnu6ko/4j6p7qWjbx1QxImPPib/tnHyc/sZcOozbF 7dtFzG8Y6QFOC3CUMRZBZyYKNCTAeTba++ZTRrmJmoYJRmW5veX3a+Tf8 ZH0quIxdQWEwFipF2LVOdjcu561M8cWriwlQ+ULLkJ6z4b8OGKP6jOnvQ D3cMMYhEjrFWL4pf9duoMutBhRm9AnvD6s609AEKo3y7hqI9k8p6i5qiI 6+9qK8YXGN9sfMg7c+oktC2lNjKmqIA6ejCuNFWro2ojNc0mvjo8TTjsg A==; X-IronPort-AV: E=McAfee;i="6200,9189,10274"; a="253651965" X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="253651965" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2022 09:24:09 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="551348093" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga008.jf.intel.com with ESMTP; 02 Mar 2022 09:24:07 -0800 From: Bruce Richardson To: dev@dpdk.org Cc: stable@dpdk.org, Bruce Richardson Subject: [PATCH 1/9] doc/linux_gsg: replace special characters for (R) symbol Date: Wed, 2 Mar 2022 17:22:09 +0000 Message-Id: <20220302172217.472279-2-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220302172217.472279-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Some IDEs, such as eclipse, complained on save about the use of special characters in the (R) symbol in linux GSG doc. We can replace those with the equivalent "|reg|" text, and including isonum.txt. Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/enable_func.rst | 16 +++++++++------- doc/guides/linux_gsg/linux_drivers.rst | 6 ++++-- doc/guides/linux_gsg/sys_reqs.rst | 10 ++++++---- 3 files changed, 19 insertions(+), 13 deletions(-) diff --git a/doc/guides/linux_gsg/enable_func.rst b/doc/guides/linux_gsg/enable_func.rst index 7bd6b03f10..b5c4c652d7 100644 --- a/doc/guides/linux_gsg/enable_func.rst +++ b/doc/guides/linux_gsg/enable_func.rst @@ -1,6 +1,8 @@ .. SPDX-License-Identifier: BSD-3-Clause Copyright(c) 2010-2014 Intel Corporation. +.. include:: + .. _Enabling_Additional_Functionality: Enabling Additional Functionality @@ -119,15 +121,15 @@ system objects' permissions should be adjusted: Power Management and Power Saving Functionality ----------------------------------------------- -Enhanced Intel SpeedStep® Technology must be enabled in the platform BIOS if the power management feature of DPDK is to be used. +Enhanced Intel SpeedStep\ |reg| Technology must be enabled in the platform BIOS if the power management feature of DPDK is to be used. Otherwise, the sys file folder ``/sys/devices/system/cpu/cpu0/cpufreq`` will not exist, and the CPU frequency- based power management cannot be used. Consult the relevant BIOS documentation to determine how these settings can be accessed. -For example, on some Intel reference platform BIOS variants, the path to Enhanced Intel SpeedStep® Technology is:: +For example, on some Intel reference platform BIOS variants, the path to Enhanced Intel SpeedStep\ |reg| Technology is:: Advanced -> Processor Configuration - -> Enhanced Intel SpeedStep® Tech + -> Enhanced Intel SpeedStep\ |reg| Tech In addition, C3 and C6 should be enabled as well for power management. The path of C3 and C6 on the same platform BIOS is:: @@ -165,10 +167,10 @@ It should be loaded using the insmod command:: See the "Kernel NIC Interface Sample Application" chapter in the *DPDK Sample Applications User Guide* for more details. -Using Linux IOMMU Pass-Through to Run DPDK with Intel® VT-d ------------------------------------------------------------ +Using Linux IOMMU Pass-Through to Run DPDK with Intel\ |reg| VT-d +------------------------------------------------------------------ -To enable Intel® VT-d in a Linux kernel, a number of kernel configuration options must be set. These include: +To enable Intel\ |reg| VT-d in a Linux kernel, a number of kernel configuration options must be set. These include: * ``IOMMU_SUPPORT`` @@ -176,7 +178,7 @@ To enable Intel® VT-d in a Linux kernel, a number of kernel configuration optio * ``INTEL_IOMMU`` -In addition, to run the DPDK with Intel® VT-d, the ``iommu=pt`` kernel parameter must be used when using ``igb_uio`` driver. +In addition, to run the DPDK with Intel\ |reg| VT-d, the ``iommu=pt`` kernel parameter must be used when using ``igb_uio`` driver. This results in pass-through of the DMAR (DMA Remapping) lookup in the host. Also, if ``INTEL_IOMMU_DEFAULT_ON`` is not set in the kernel, the ``intel_iommu=on`` kernel parameter must be used too. This ensures that the Intel IOMMU is being initialized as expected. diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index 2dd711bb37..56564286ac 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -3,6 +3,8 @@ Copyright 2017 Mellanox Technologies, Ltd All rights reserved. +.. include:: + .. _linux_gsg_linux_drivers: Linux Drivers @@ -99,7 +101,7 @@ The token will be used for all PF and VF ports within the application. To make use of full VFIO functionality, both kernel and BIOS must support and be configured -to use IO virtualization (such as Intel® VT-d). +to use IO virtualization (such as Intel\ |reg| VT-d). .. note:: @@ -335,7 +337,7 @@ Please refer to earlier sections on how to configure kernel parameters correctly for your system. If the kernel is configured correctly, one also has to make sure that -the BIOS configuration has virtualization features (such as Intel® VT-d). +the BIOS configuration has virtualization features (such as Intel\ |reg| VT-d). There is no standard way to check if the platform is configured correctly, so please check with your platform documentation to see if it has such features, and how to enable them. diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst index d95a78d156..9dccd54d9c 100644 --- a/doc/guides/linux_gsg/sys_reqs.rst +++ b/doc/guides/linux_gsg/sys_reqs.rst @@ -1,6 +1,8 @@ .. SPDX-License-Identifier: BSD-3-Clause Copyright(c) 2010-2014 Intel Corporation. +.. include:: + System Requirements =================== @@ -8,8 +10,8 @@ This chapter describes the packages required to compile the DPDK. .. note:: - If the DPDK is being used on an Intel® Communications Chipset 89xx Series platform, - please consult the *Intel® Communications Chipset 89xx Series Software for Linux Getting Started Guide*. + If the DPDK is being used on an Intel\ |reg| Communications Chipset 89xx Series platform, + please consult the *Intel\ |reg| Communications Chipset 89xx Series Software for Linux Getting Started Guide*. BIOS Setting Prerequisite on x86 -------------------------------- @@ -72,10 +74,10 @@ Compilation of the DPDK **Optional Tools:** -* Intel® C++ Compiler (icc). For installation, additional libraries may be required. +* Intel\ |reg| C++ Compiler (icc). For installation, additional libraries may be required. See the icc Installation Guide found in the Documentation directory under the compiler installation. -* IBM® Advance ToolChain for Powerlinux. This is a set of open source development tools and runtime libraries +* IBM\ |reg| Advance ToolChain for Powerlinux. This is a set of open source development tools and runtime libraries which allows users to take leading edge advantage of IBM's latest POWER hardware features on Linux. To install it, see the IBM official installation document. From patchwork Wed Mar 2 17:22:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108496 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 68F4DA04A4; Wed, 2 Mar 2022 18:24:21 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3DDCE4275C; Wed, 2 Mar 2022 18:24:13 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 918934274D; Wed, 2 Mar 2022 18:24:11 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646241851; x=1677777851; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XHQ9NcqZOU9e+YGqa+JCZuJenBim/EQOZXpEWOPISxE=; b=FHSVyjKpkew/QXylC/pWLyZyA7b4j8+asOoP8H95Klf0rHouwLRzNfsN XdYpO2P325BaS4z+lNgU/waWo5DZeoGNyft/uYFcJMV4Vd95/G1BSBPaF yipJVXzk7gHtYxdBiZCfDkHkP/s/LhSYpvXuKKVr/1JjbyBIqkUO2B40b r5TQM7NULDDExZ9nqkvYncegR0GIuySGKEYPo+Bb+t/5/GKUnXLQGFqMl 4iazPv2dcbmpxSAZT+n0g8upc1XVzLDE7fmndc4piG7lqwl4fwdaOyEI+ SjprBRjX6bwxTYShYaTCiy8Yu8pBP9Inpy+YA+zaNL5DOSlgbO9h40bqL w==; X-IronPort-AV: E=McAfee;i="6200,9189,10274"; a="253651969" X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="253651969" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2022 09:24:10 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="551348108" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga008.jf.intel.com with ESMTP; 02 Mar 2022 09:24:09 -0800 From: Bruce Richardson To: dev@dpdk.org Cc: stable@dpdk.org, Bruce Richardson , thomas@monjalon.net Subject: [PATCH 2/9] doc/linux_gsg: fix missing note on UIO module Date: Wed, 2 Mar 2022 17:22:10 +0000 Message-Id: <20220302172217.472279-3-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220302172217.472279-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The docs on binding drivers was updated as part of the removal of the igb_uio module from the main DPDK repo. As part of that update, a note about uio_pci_generic requiring legacy interrupts was removed, but should have been kept. Fixes: 56bb5841fd06 ("kernel/linux: remove igb_uio") Cc: thomas@monjalon.net Cc: stable@dpdk.org Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index 56564286ac..75af2f01e1 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -174,6 +174,11 @@ It can be loaded as shown below: sudo modprobe uio sudo insmod igb_uio.ko +.. note:: + + For some devices which lack support for legacy interrupts, e.g. virtual function + (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``. + .. note:: If UEFI secure boot is enabled, From patchwork Wed Mar 2 17:22:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108497 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DECB7A04A4; Wed, 2 Mar 2022 18:24:26 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 320AF42754; Wed, 2 Mar 2022 18:24:14 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 625E742758; Wed, 2 Mar 2022 18:24:12 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646241852; x=1677777852; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=W/WXqetrPzHPU1i0W4z6TE+FVsvpRPyPTwocyJ//w7U=; b=XxzHfB5PLv3HdhtF8OI02Wiw4YDdFu68k+qcV1i/O4vY/lu20kIH9FW+ zMIfzXnXU5ofXUxdcE6kQse16iv2X/n+1qcSb550QYBZ3swmEFMJxKEmt 2UVzvOaj7Aak7tveGeWQVFlfpEK9E48Z9N9Str6vkNxrJEyRkGBBs1BSH f7CDr0LhflzsRE0XlDzYG7petaZa4iYwjF9FcL1ZktvU3Xat9FLXIHgLU a8uGyenyxZTSviHjxP/gcB5QfZbVTgSSrNEMgBxdibryk1Sk0TivfHOii ZN3lMA1uQdc1zf+z66YAZW8RsLL/jZATXnIIo+IfyXi4+nAXHUBzK0gP4 w==; X-IronPort-AV: E=McAfee;i="6200,9189,10274"; a="253651976" X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="253651976" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2022 09:24:11 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="551348118" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga008.jf.intel.com with ESMTP; 02 Mar 2022 09:24:10 -0800 From: Bruce Richardson To: dev@dpdk.org Cc: stable@dpdk.org, Bruce Richardson Subject: [PATCH 3/9] doc/linux_gsg: make UIO safety warning more visible Date: Wed, 2 Mar 2022 17:22:11 +0000 Message-Id: <20220302172217.472279-4-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220302172217.472279-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The GSG has a note warning that use of UIO is inherently unsafe due to lack of IOMMU protection. However, this was only flagged as a "NOTE", meaning it could easily be missed. Changing the rst tag from "note" to "warning" and moving it to the top of the UIO subsection makes this a lot more visible to users. Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) -- 2.32.0 diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index 75af2f01e1..2cdacf0961 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -153,6 +153,11 @@ After that, VFIO can be used with hardware devices as usual. UIO --- +.. warning:: + + Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection, + and can only be done by root user. + In situations where using VFIO is not an option, there are alternative drivers one can use. In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel can be used as a substitute for VFIO. This module can be loaded using the command: @@ -195,11 +200,6 @@ It can be loaded as shown below: in GRUB command line on x86_64 systems, or add ``iommu.passthrough=1`` on aarch64 systems. -.. note:: - - Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection, - and can only be done by root user. - .. _bifurcated_driver: Bifurcated Driver From patchwork Wed Mar 2 17:22:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108498 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A33B1A04A4; Wed, 2 Mar 2022 18:24:33 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6BC9442767; Wed, 2 Mar 2022 18:24:16 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id DAD7C42752; Wed, 2 Mar 2022 18:24:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646241854; x=1677777854; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Ot3AlOFBhmNb1WdG56o1o6Nd95gQa/MNINAsrxsJZjw=; b=WVQfgV6y5CDqIeBAS2F0WMk+Nxv3CzupnSPDPuLsw4kiNMJb/Sue8EpT +hggnIfbgdce+g41jwVtOivPMZsBYToLFXyxOiuCVY8Rb3yrQLKANZH/i eEH3qTIPHUkaC8pKqqYKo2UouEhNBeeueupmCvzVAgkJ3BGBG6OqNnBCm YV8eIr2naGkVf6o8CZRSQ1geroIdg24VYUuY+NazuMejd5G50zbvHqAEH pM0aQA61ixtOfryBWRtRKRzHtaXeRkPgM8Te1JMyXCQnB/cESTvN4hyfl Nq9LpIqSUwRTf6XQrsnbkLI30x3OWA0QV9gLTfNqm8FBnn/SkV/1b4qcE w==; X-IronPort-AV: E=McAfee;i="6200,9189,10274"; a="253651982" X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="253651982" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2022 09:24:13 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="551348126" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga008.jf.intel.com with ESMTP; 02 Mar 2022 09:24:12 -0800 From: Bruce Richardson To: dev@dpdk.org Cc: stable@dpdk.org, Bruce Richardson Subject: [PATCH 4/9] doc/linux_gsg: move section on binding drivers up the page Date: Wed, 2 Mar 2022 17:22:12 +0000 Message-Id: <20220302172217.472279-5-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220302172217.472279-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org While the details of VFIO and UIO may be of interest to some, most users of the doc are likely primarily interested in how to bind their devices to the kernel driver and then move on to running the app. Therefore, the most important part of the "Linux Drivers" section of the GSG is the subsection on "Binding and Unbinding", so put that first. Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 179 +++++++++++++------------ 1 file changed, 90 insertions(+), 89 deletions(-) diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index 2cdacf0961..83f28d86fe 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -14,6 +14,96 @@ Different PMDs may require different kernel drivers in order to work properly. Depending on the PMD being used, a corresponding kernel driver should be loaded, and network ports should be bound to that driver. +.. _linux_gsg_binding_kernel: + +Binding and Unbinding Network Ports to/from the Kernel Modules +-------------------------------------------------------------- + +.. note:: + + PMDs which use the bifurcated driver should not be unbound from their kernel drivers. + This section is for PMDs which use the UIO or VFIO drivers. + See :ref:`bifurcated_driver` section for more details. + +As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use. +Instead, in case the PMD being used use the VFIO or UIO drivers, +all ports that are to be used by a DPDK application must be bound to +the ``vfio-pci``, ``uio_pci_generic``, or ``igb_uio`` module +before the application is run. +For such PMDs, any network ports under Linux* control will be ignored and cannot be used by the application. + +To bind ports to the ``vfio-pci``, ``uio_pci_generic`` or ``igb_uio`` module +for DPDK use, or to return ports to Linux control, +a utility script called ``dpdk-devbind.py`` is provided in the ``usertools`` subdirectory. +This utility can be used to provide a view of the current state of the network ports on the system, +and to bind and unbind those ports from the different kernel modules, +including the VFIO and UIO modules. +The following are some examples of how the script can be used. +A full description of the script and its parameters can be obtained +by calling the script with the ``--help`` or ``--usage`` options. +Note that the UIO or VFIO kernel modules to be used, +should be loaded into the kernel before running the ``dpdk-devbind.py`` script. + +.. warning:: + + Due to the way VFIO works, there are certain limitations + to which devices can be used with VFIO. + Mainly it comes down to how IOMMU groups work. + Any Virtual Function device can usually be used with VFIO on its own, + but physical devices may require either all ports bound to VFIO, + or some of them bound to VFIO while others not being bound to anything at all. + + If your device is behind a PCI-to-PCI bridge, + the bridge will then be part of the IOMMU group in which your device is in. + Therefore, the bridge driver should also be unbound from the bridge PCI device + for VFIO to work with devices behind the bridge. + +.. warning:: + + While any user can run the ``dpdk-devbind.py`` script + to view the status of the network ports, + binding or unbinding network ports requires root privileges. + +To see the status of all network ports on the system: + +.. code-block:: console + + ./usertools/dpdk-devbind.py --status + + Network devices using DPDK-compatible driver + ============================================ + 0000:82:00.0 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe + 0000:82:00.1 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe + + Network devices using kernel driver + =================================== + 0000:04:00.0 'I350 1-GbE NIC' if=em0 drv=igb unused=uio_pci_generic *Active* + 0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=uio_pci_generic + 0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=uio_pci_generic + 0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=uio_pci_generic + + Other network devices + ===================== + + +To bind device ``eth1``,``04:00.1``, to the ``uio_pci_generic`` driver: + +.. code-block:: console + + ./usertools/dpdk-devbind.py --bind=uio_pci_generic 04:00.1 + +or, alternatively, + +.. code-block:: console + + ./usertools/dpdk-devbind.py --bind=uio_pci_generic eth1 + +To restore device ``82:00.0`` to its original kernel binding: + +.. code-block:: console + + ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.0 + VFIO ---- @@ -225,95 +315,6 @@ More about the bifurcated driver can be found in `Mellanox Bifurcated DPDK PMD `__. -.. _linux_gsg_binding_kernel: - -Binding and Unbinding Network Ports to/from the Kernel Modules --------------------------------------------------------------- - -.. note:: - - PMDs which use the bifurcated driver should not be unbound from their kernel drivers. - This section is for PMDs which use the UIO or VFIO drivers. - -As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use. -Instead, in case the PMD being used use the VFIO or UIO drivers, -all ports that are to be used by a DPDK application must be bound to -the ``vfio-pci``, ``uio_pci_generic``, or ``igb_uio`` module -before the application is run. -For such PMDs, any network ports under Linux* control will be ignored and cannot be used by the application. - -To bind ports to the ``vfio-pci``, ``uio_pci_generic`` or ``igb_uio`` module -for DPDK use, or to return ports to Linux control, -a utility script called ``dpdk-devbind.py`` is provided in the ``usertools`` subdirectory. -This utility can be used to provide a view of the current state of the network ports on the system, -and to bind and unbind those ports from the different kernel modules, -including the VFIO and UIO modules. -The following are some examples of how the script can be used. -A full description of the script and its parameters can be obtained -by calling the script with the ``--help`` or ``--usage`` options. -Note that the UIO or VFIO kernel modules to be used, -should be loaded into the kernel before running the ``dpdk-devbind.py`` script. - -.. warning:: - - Due to the way VFIO works, there are certain limitations - to which devices can be used with VFIO. - Mainly it comes down to how IOMMU groups work. - Any Virtual Function device can usually be used with VFIO on its own, - but physical devices may require either all ports bound to VFIO, - or some of them bound to VFIO while others not being bound to anything at all. - - If your device is behind a PCI-to-PCI bridge, - the bridge will then be part of the IOMMU group in which your device is in. - Therefore, the bridge driver should also be unbound from the bridge PCI device - for VFIO to work with devices behind the bridge. - -.. warning:: - - While any user can run the ``dpdk-devbind.py`` script - to view the status of the network ports, - binding or unbinding network ports requires root privileges. - -To see the status of all network ports on the system: - -.. code-block:: console - - ./usertools/dpdk-devbind.py --status - - Network devices using DPDK-compatible driver - ============================================ - 0000:82:00.0 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe - 0000:82:00.1 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe - - Network devices using kernel driver - =================================== - 0000:04:00.0 'I350 1-GbE NIC' if=em0 drv=igb unused=uio_pci_generic *Active* - 0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=uio_pci_generic - 0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=uio_pci_generic - 0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=uio_pci_generic - - Other network devices - ===================== - - -To bind device ``eth1``,``04:00.1``, to the ``uio_pci_generic`` driver: - -.. code-block:: console - - ./usertools/dpdk-devbind.py --bind=uio_pci_generic 04:00.1 - -or, alternatively, - -.. code-block:: console - - ./usertools/dpdk-devbind.py --bind=uio_pci_generic eth1 - -To restore device ``82:00.0`` to its original kernel binding: - -.. code-block:: console - - ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.0 - Troubleshooting VFIO -------------------- From patchwork Wed Mar 2 17:22:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108499 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9C6F1A04A4; Wed, 2 Mar 2022 18:24:41 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 82B644274E; Wed, 2 Mar 2022 18:24:20 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id EFCA442760; Wed, 2 Mar 2022 18:24:17 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646241858; x=1677777858; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bFvqrLNhBGM0Fo7KXqq0nS6eDPeU99gaY/oomaeqgc4=; b=PQIOnvoMlqOjV1xMhWA8euPH51jn56Z3C+H/HP9sFavm/Bo77h5T2XKl 3H0ZyLAiewQyO1eec3bGm6Nt39o7mH/PXYFH9TWRGb7hHB7ZDVsPzS07C +ryxjUMkU0+IYnGVFB8UywJg4MXwpAeKvdYRhtdCM8vVJRknhOL2vbQlJ tcfu67BE8KESWKYw537nz2YfwB3H6lW5HxYtzZm3GJtl7bGzydMMu23iS zX9iepC5u1BUeYVN2tGsVktInxxmkx3hsruZlraIMkveaGQfAmnDDYCL4 HZyGxRWYJSEEZBR1GTjFQipuPTw8+8J51vLpdWzb7fcRVEjWbWPwKav0p w==; X-IronPort-AV: E=McAfee;i="6200,9189,10274"; a="253651989" X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="253651989" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2022 09:24:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="551348133" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga008.jf.intel.com with ESMTP; 02 Mar 2022 09:24:13 -0800 From: Bruce Richardson To: dev@dpdk.org Cc: stable@dpdk.org, Bruce Richardson Subject: [PATCH 5/9] doc/linux_gsg: emphasise VFIO over UIO-based modules Date: Wed, 2 Mar 2022 17:22:13 +0000 Message-Id: <20220302172217.472279-6-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220302172217.472279-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org VFIO is to be strongly preferred over uio-based modules, so update our text and examples to only refer to vfio, giving an initial reference at the start to uio as a fallback option. Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 47 +++++++++++++++----------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index 83f28d86fe..af7e3c4fbc 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -25,14 +25,18 @@ Binding and Unbinding Network Ports to/from the Kernel Modules This section is for PMDs which use the UIO or VFIO drivers. See :ref:`bifurcated_driver` section for more details. -As of release 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use. -Instead, in case the PMD being used use the VFIO or UIO drivers, -all ports that are to be used by a DPDK application must be bound to -the ``vfio-pci``, ``uio_pci_generic``, or ``igb_uio`` module -before the application is run. -For such PMDs, any network ports under Linux* control will be ignored and cannot be used by the application. - -To bind ports to the ``vfio-pci``, ``uio_pci_generic`` or ``igb_uio`` module +.. note:: + + It is recommended that ``vfio-pci`` be used as the kernel module for DPDK-bound ports in all cases. + If an IOMMU is unavailable, the ``vfio-pci`` can be used in :ref:`no-iommu` mode. + If, for some reason, vfio is unavailable, then UIO-based modules, ``igb_uio`` and ``uio_pci_generic`` may be used. + See section :ref:`uio` for details. + +Most devices require that the hardware to be used by DPDK be unbound from the kernel driver it uses, +and instead be bound to the ``vfio-pci`` kernel module before the application is run. +For such PMDs, any network ports or other hardware under Linux* control will be ignored and cannot be used by the application. + +To bind ports to the ``vfio-pci`` module for DPDK use, or to return ports to Linux control, a utility script called ``dpdk-devbind.py`` is provided in the ``usertools`` subdirectory. This utility can be used to provide a view of the current state of the network ports on the system, @@ -72,37 +76,38 @@ To see the status of all network ports on the system: Network devices using DPDK-compatible driver ============================================ - 0000:82:00.0 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe - 0000:82:00.1 '82599EB 10-GbE NIC' drv=uio_pci_generic unused=ixgbe + 0000:82:00.0 '82599EB 10-GbE NIC' drv=vfio-pci unused=ixgbe + 0000:82:00.1 '82599EB 10-GbE NIC' drv=vfio-pci unused=ixgbe Network devices using kernel driver =================================== - 0000:04:00.0 'I350 1-GbE NIC' if=em0 drv=igb unused=uio_pci_generic *Active* - 0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=uio_pci_generic - 0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=uio_pci_generic - 0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=uio_pci_generic + 0000:04:00.0 'I350 1-GbE NIC' if=em0 drv=igb unused=vfio-pci *Active* + 0000:04:00.1 'I350 1-GbE NIC' if=eth1 drv=igb unused=vfio-pci + 0000:04:00.2 'I350 1-GbE NIC' if=eth2 drv=igb unused=vfio-pci + 0000:04:00.3 'I350 1-GbE NIC' if=eth3 drv=igb unused=vfio-pci Other network devices ===================== -To bind device ``eth1``,``04:00.1``, to the ``uio_pci_generic`` driver: +To bind device ``eth1``,``04:00.1``, to the ``vfio-pci`` driver: .. code-block:: console - ./usertools/dpdk-devbind.py --bind=uio_pci_generic 04:00.1 + ./usertools/dpdk-devbind.py --bind=vfio-pci 04:00.1 or, alternatively, .. code-block:: console - ./usertools/dpdk-devbind.py --bind=uio_pci_generic eth1 + ./usertools/dpdk-devbind.py --bind=vfio-pci eth1 -To restore device ``82:00.0`` to its original kernel binding: +When specifying device ids, wildcards can be used for the final part of the address. +To restore device ``82:00.0`` and ``82:00.1`` to their original kernel binding: .. code-block:: console - ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.0 + ./usertools/dpdk-devbind.py --bind=ixgbe 82:00.* VFIO ---- @@ -210,6 +215,8 @@ to use IO virtualization (such as Intel\ |reg| VT-d). For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up. For more information, please refer to :ref:`Running_Without_Root_Privileges`. +.. _vfio_noiommu: + VFIO no-IOMMU mode ------------------ @@ -240,6 +247,8 @@ After that, VFIO can be used with hardware devices as usual. to keep the degree of device access and programming that VFIO has, in situations where IOMMU is not available. +.. _uio: + UIO --- From patchwork Wed Mar 2 17:22:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108500 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 794EFA04A4; Wed, 2 Mar 2022 18:24:49 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A43C842773; Wed, 2 Mar 2022 18:24:23 +0100 (CET) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id 327D442756; Wed, 2 Mar 2022 18:24:21 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646241861; x=1677777861; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qz1Tj1pQmOzs/CldsKWLIzTXnrmnOIyBk9b50iuj6CU=; b=T82c4Z70A8URBZmo5Gk0Kdk2Eq4teNPociiyVEL7Chue/YKKyCD9Lqst S7M6X2IksLWnOCNoqf8jMeI8V5zUof5HXIYFz5zaf/7zBbPvRTy3IFSRS 3vDFhoo6yQ2aHGqNK3gEa5r8tlkfk4Xih26B2kWd/UKtrlB7yeJKQ1uHk zh45xQ8TtrIvw2HFYLlFv7xxTGHyfnR5Kydy+24vUEeZyax9poBhqR5Sw QCDKEO6xUuo26Qs/2bMz54kK3CAzwNwaXCBoSwJpypOHf/Ytt4Xyi6zsu bEjDROmqxKAmkeu6PmNlqLERlJ2oYzm+WVg3BgEL5PG7uAwY3Y05jIjOZ Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10274"; a="314177962" X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="314177962" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2022 09:24:16 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="551348139" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga008.jf.intel.com with ESMTP; 02 Mar 2022 09:24:15 -0800 From: Bruce Richardson To: dev@dpdk.org Cc: stable@dpdk.org, Bruce Richardson Subject: [PATCH 6/9] doc/linux_gsg: split VFIO section into subsections Date: Wed, 2 Mar 2022 17:22:14 +0000 Message-Id: <20220302172217.472279-7-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220302172217.472279-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The VFIO section of the page about linux drivers was rather long and unstructured. This can be improved by splitting it up into subsections, to cover the specifics of memory limits and creating VFs. When moving the various text notes into the relevant subsections, we can drop the note about kernels earlier than 3.6, since DPDK no longer supports kernels that old. Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 35 ++++++++++++++------------ 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index af7e3c4fbc..e29f79e385 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -122,6 +122,22 @@ To make use of VFIO, the ``vfio-pci`` module must be loaded: VFIO kernel is usually present by default in all distributions, however please consult your distributions documentation to make sure that is the case. +To make use of full VFIO functionality, +both kernel and BIOS must support and be configured +to use IO virtualization (such as Intel\ |reg| VT-d). + +.. note:: + + In most cases, specifying "iommu=on" as kernel parameter should be enough to + configure the Linux kernel to use IOMMU. + +For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up. +For more information, please refer to :ref:`Running_Without_Root_Privileges`. + + +VFIO Memory Mapping Limits +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + For DMA mapping of either external memory or hugepages, VFIO interface is used. VFIO does not support partial unmap of once mapped memory. Hence DPDK's memory is mapped in hugepage granularity or system page granularity. Number of DMA @@ -132,6 +148,9 @@ VFIO module parameter ``dma_entry_limit`` with a default value of 64K. When application is out of DMA entries, these limits need to be adjusted to increase the allowed limit. +Creating Virtual Functions using vfio-pci +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + Since Linux version 5.7, the ``vfio-pci`` module supports the creation of virtual functions. After the PF is bound to ``vfio-pci`` module, @@ -194,27 +213,11 @@ The token will be used for all PF and VF ports within the application. /app/dpdk-testpmd -l 26-29 -n 4 -a 86:02.0 \ --vfio-vf-token=14d63f20-8445-11ea-8900-1f9ce7d5650d --file-prefix=vf0 -- -i -To make use of full VFIO functionality, -both kernel and BIOS must support and be configured -to use IO virtualization (such as Intel\ |reg| VT-d). - -.. note:: - - Linux versions earlier than version 3.6 do not support VFIO. - .. note:: Linux versions earlier than version 5.7 do not support the creation of virtual functions within the VFIO framework. -.. note:: - - In most cases, specifying "iommu=on" as kernel parameter should be enough to - configure the Linux kernel to use IOMMU. - -For proper operation of VFIO when running DPDK applications as a non-privileged user, correct permissions should also be set up. -For more information, please refer to :ref:`Running_Without_Root_Privileges`. - .. _vfio_noiommu: VFIO no-IOMMU mode From patchwork Wed Mar 2 17:22:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108501 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 91725A04A4; Wed, 2 Mar 2022 18:24:56 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 974B342777; Wed, 2 Mar 2022 18:24:24 +0100 (CET) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id 7F83442756; Wed, 2 Mar 2022 18:24:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646241862; x=1677777862; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zkqiqgTW9W5WUn2KcJ0DEdhcSJpz8wrzyTx+Tth8JWI=; b=PhI8ATloMynHZV8NINn6S91NH9zKcmO6VmDsoGBJ2nTP/Iy59nzyOh91 Z0DFHCTu6qDDNt7SRsI6d16wZcignPThQyf6pgAoDJx5XVe0hhA6wmsHk SODbAud+jMX0jrSkj8k9rZEJnLKvKHeGT//ahxTYcTX8LibL2yAmGIlfa H//dw+PbAq/Qz4iWkDVXB08K1mR2Y3nB7jiHVcQ9hlIQAbVLlKkxd8VFB lbB5CfigJNh0UKeVp1TX8BhedxXB7L8lQnm+WLa3dUoOHMSVWdDonVd/1 iFRUXMYtDsJ5YqICSsPS5W3dJyfi80aSRhCbAKGn95WIsTwZGgKJf2bwz Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10274"; a="314177965" X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="314177965" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2022 09:24:17 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="551348149" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga008.jf.intel.com with ESMTP; 02 Mar 2022 09:24:16 -0800 From: Bruce Richardson To: dev@dpdk.org Cc: stable@dpdk.org, Bruce Richardson Subject: [PATCH 7/9] doc/linux_gsg: move UIO section to the end Date: Wed, 2 Mar 2022 17:22:15 +0000 Message-Id: <20220302172217.472279-8-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220302172217.472279-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org To further de-emphasise UIO over the alternatives, we can move the UIO section of the drivers page to the end of the document, giving more prominence to VFIO and bifurcated drivers. Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 104 ++++++++++++------------- 1 file changed, 52 insertions(+), 52 deletions(-) diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index e29f79e385..b2be442d27 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -250,58 +250,6 @@ After that, VFIO can be used with hardware devices as usual. to keep the degree of device access and programming that VFIO has, in situations where IOMMU is not available. -.. _uio: - -UIO ---- - -.. warning:: - - Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection, - and can only be done by root user. - -In situations where using VFIO is not an option, there are alternative drivers one can use. -In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel -can be used as a substitute for VFIO. This module can be loaded using the command: - -.. code-block:: console - - sudo modprobe uio_pci_generic - -.. note:: - - ``uio_pci_generic`` module doesn't support the creation of virtual functions. - -As an alternative to the ``uio_pci_generic``, there is the ``igb_uio`` module -which can be found in the repository `dpdk-kmods `_. -It can be loaded as shown below: - -.. code-block:: console - - sudo modprobe uio - sudo insmod igb_uio.ko - -.. note:: - - For some devices which lack support for legacy interrupts, e.g. virtual function - (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``. - -.. note:: - - If UEFI secure boot is enabled, - the Linux kernel may disallow the use of UIO on the system. - Therefore, devices for use by DPDK should be bound to the ``vfio-pci`` kernel module - rather than any UIO-based module. - For more details see :ref:`linux_gsg_binding_kernel` below. - -.. note:: - - If the devices used for DPDK are bound to the ``uio_pci_generic`` kernel module, - please make sure that the IOMMU is disabled or is in passthrough mode. - One can add ``intel_iommu=off`` or ``amd_iommu=off`` or ``intel_iommu=on iommu=pt`` - in GRUB command line on x86_64 systems, - or add ``iommu.passthrough=1`` on aarch64 systems. - .. _bifurcated_driver: Bifurcated Driver @@ -372,3 +320,55 @@ This can be checked in the boot configuration of your system: If ``CONFIG_VFIO_NOIOMMU`` is not enabled in the kernel configuration, VFIO driver will not support the no-IOMMU mode, and other alternatives (such as UIO drivers) will have to be used. + +.. _uio: + +UIO +--- + +.. warning:: + + Using UIO drivers is inherently unsafe due to this method lacking IOMMU protection, + and can only be done by root user. + +In situations where using VFIO is not an option, there are alternative drivers one can use. +In many cases, the standard ``uio_pci_generic`` module included in the Linux kernel +can be used as a substitute for VFIO. This module can be loaded using the command: + +.. code-block:: console + + sudo modprobe uio_pci_generic + +.. note:: + + ``uio_pci_generic`` module doesn't support the creation of virtual functions. + +As an alternative to the ``uio_pci_generic``, there is the ``igb_uio`` module +which can be found in the repository `dpdk-kmods `_. +It can be loaded as shown below: + +.. code-block:: console + + sudo modprobe uio + sudo insmod igb_uio.ko + +.. note:: + + For some devices which lack support for legacy interrupts, e.g. virtual function + (VF) devices, the ``igb_uio`` module may be needed in place of ``uio_pci_generic``. + +.. note:: + + If UEFI secure boot is enabled, + the Linux kernel may disallow the use of UIO on the system. + Therefore, devices for use by DPDK should be bound to the ``vfio-pci`` kernel module + rather than any UIO-based module. + For more details see :ref:`linux_gsg_binding_kernel` below. + +.. note:: + + If the devices used for DPDK are bound to the ``uio_pci_generic`` kernel module, + please make sure that the IOMMU is disabled or is in passthrough mode. + One can add ``intel_iommu=off`` or ``amd_iommu=off`` or ``intel_iommu=on iommu=pt`` + in GRUB command line on x86_64 systems, + or add ``iommu.passthrough=1`` on aarch64 systems. From patchwork Wed Mar 2 17:22:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108502 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 05546A04A4; Wed, 2 Mar 2022 18:25:04 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 02EBE4277F; Wed, 2 Mar 2022 18:24:26 +0100 (CET) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id 002D84275B; Wed, 2 Mar 2022 18:24:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646241863; x=1677777863; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=k065+2O+3oN0bZwLP1KzaeYRA3FLtfZgUoswBzpOpI4=; b=Oone0fVIDlU7cfxike6JGisqgWEc2MiAK3mmEgwuT8t2yvtiWXbJJ+p3 2xle5OT7jobRMxK4RUMUU1GDk9wXGAfHQepzJ6S4v+Hv+np6HcMbHbK8a MTQelC89cCTX1o0CIilKFiuFgFTbTNNMCy1WsO536LHpaDJLvGjm9PqCh gewEXvAvzQKIyt3WPOC5uYh+0r8SMIM3jxldoyEwDOMry0yRGF6QbKWot LjqfU5WxEVO9juCm7FAbB0O9yVGYcLdxLHDAMTDwsXYZXjRx1vnLKzGuv FonE9PheylcC307VwzzQU6fI2LDQSZ37LaTHXR4V2guGKcddVydmgkd4M w==; X-IronPort-AV: E=McAfee;i="6200,9189,10274"; a="314177971" X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="314177971" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2022 09:24:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="551348163" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga008.jf.intel.com with ESMTP; 02 Mar 2022 09:24:18 -0800 From: Bruce Richardson To: dev@dpdk.org Cc: stable@dpdk.org, Bruce Richardson Subject: [PATCH 8/9] doc/linux_gsg: consolidate all VFIO content Date: Wed, 2 Mar 2022 17:22:16 +0000 Message-Id: <20220302172217.472279-9-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220302172217.472279-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Rather than having separate sections for VFIO and VFIO no-iommu mode, as well as a separate section further down the document on troubleshooting VFIO, we can consolidate all these as subsections into a primary VFIO section. This section starts with the basics of VFIO use, then covers no-iommu mode, before moving on to the more advanced topics such as creating VFs and ending with the troubleshooting subsection. Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 116 ++++++++++++------------- 1 file changed, 58 insertions(+), 58 deletions(-) diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index b2be442d27..ef91999dd9 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -135,6 +135,38 @@ For proper operation of VFIO when running DPDK applications as a non-privileged For more information, please refer to :ref:`Running_Without_Root_Privileges`. +.. _vfio_noiommu: + +VFIO no-IOMMU mode +~~~~~~~~~~~~~~~~~~ + +If there is no IOMMU available on the system, VFIO can still be used, +but it has to be loaded with an additional module parameter: + +.. code-block:: console + + modprobe vfio enable_unsafe_noiommu_mode=1 + +Alternatively, one can also enable this option in an already loaded kernel module: + +.. code-block:: console + + echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode + +After that, VFIO can be used with hardware devices as usual. + +.. note:: + + It may be required to unload all VFIO related-modules before probing + the module again with ``enable_unsafe_noiommu_mode=1`` parameter. + +.. warning:: + + Since no-IOMMU mode forgoes IOMMU protection, it is inherently unsafe. + That said, it does make it possible for the user + to keep the degree of device access and programming that VFIO has, + in situations where IOMMU is not available. + VFIO Memory Mapping Limits ~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -218,65 +250,8 @@ The token will be used for all PF and VF ports within the application. Linux versions earlier than version 5.7 do not support the creation of virtual functions within the VFIO framework. -.. _vfio_noiommu: - -VFIO no-IOMMU mode ------------------- - -If there is no IOMMU available on the system, VFIO can still be used, -but it has to be loaded with an additional module parameter: - -.. code-block:: console - - modprobe vfio enable_unsafe_noiommu_mode=1 - -Alternatively, one can also enable this option in an already loaded kernel module: - -.. code-block:: console - - echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode - -After that, VFIO can be used with hardware devices as usual. - -.. note:: - - It may be required to unload all VFIO related-modules before probing - the module again with ``enable_unsafe_noiommu_mode=1`` parameter. - -.. warning:: - - Since no-IOMMU mode forgoes IOMMU protection, it is inherently unsafe. - That said, it does make it possible for the user - to keep the degree of device access and programming that VFIO has, - in situations where IOMMU is not available. - -.. _bifurcated_driver: - -Bifurcated Driver ------------------ - -PMDs which use the bifurcated driver co-exists with the device kernel driver. -On such model the NIC is controlled by the kernel, while the data -path is performed by the PMD directly on top of the device. - -Such model has the following benefits: - - - It is secure and robust, as the memory management and isolation - is done by the kernel. - - It enables the user to use legacy linux tools such as ``ethtool`` or - ``ifconfig`` while running DPDK application on the same network ports. - - It enables the DPDK application to filter only part of the traffic, - while the rest will be directed and handled by the kernel driver. - The flow bifurcation is performed by the NIC hardware. - As an example, using :ref:`flow_isolated_mode` allows to choose - strictly what is received in DPDK. - -More about the bifurcated driver can be found in -`Mellanox Bifurcated DPDK PMD -`__. - Troubleshooting VFIO --------------------- +~~~~~~~~~~~~~~~~~~~~ In certain situations, using ``dpdk-devbind.py`` script to bind a device to VFIO driver may fail. @@ -321,6 +296,31 @@ If ``CONFIG_VFIO_NOIOMMU`` is not enabled in the kernel configuration, VFIO driver will not support the no-IOMMU mode, and other alternatives (such as UIO drivers) will have to be used. +.. _bifurcated_driver: + +Bifurcated Driver +----------------- + +PMDs which use the bifurcated driver co-exists with the device kernel driver. +On such model the NIC is controlled by the kernel, while the data +path is performed by the PMD directly on top of the device. + +Such model has the following benefits: + + - It is secure and robust, as the memory management and isolation + is done by the kernel. + - It enables the user to use legacy linux tools such as ``ethtool`` or + ``ifconfig`` while running DPDK application on the same network ports. + - It enables the DPDK application to filter only part of the traffic, + while the rest will be directed and handled by the kernel driver. + The flow bifurcation is performed by the NIC hardware. + As an example, using :ref:`flow_isolated_mode` allows to choose + strictly what is received in DPDK. + +More about the bifurcated driver can be found in +`Mellanox Bifurcated DPDK PMD +`__. + .. _uio: UIO From patchwork Wed Mar 2 17:22:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bruce Richardson X-Patchwork-Id: 108503 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BC24CA04A4; Wed, 2 Mar 2022 18:25:09 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DB9ED42783; Wed, 2 Mar 2022 18:24:26 +0100 (CET) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id 7B60442756; Wed, 2 Mar 2022 18:24:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1646241863; x=1677777863; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=V7P18K/pcc20AuC3DAfsJsA5uJRnQmkY5JMuQHfUfik=; b=J0b8nvX8r/hifVspRa9ryqiEde3vdxGSUT8j7C/vsyvxABgCzT5DLyJp QbjYDkbus/aPw+N+aanbwo0wIk2BuIQfzxlwTwVfvqsjmI1qnGYaY2pli D4uTac+nSbZysCFcGb4ksvZE4WK714xW9MCYUOZQKXlsBPhT3TyfZWEQB CzC/PbUDcSjJ2f7DDyTy1RdYx52XMJRth8lxMHtf5O+4B4M/4sB3doiJL iDwDGcR/OXXBE1kd/owKLwDOQ3Av7n/Yn8qPqxCU5rA7Inl3P8P5h7Sfp GXVTb9Jx/B8K5sroMldKNXSkfwEaoVJsMR8GhXHkATBuWTa8FuYwX85yN g==; X-IronPort-AV: E=McAfee;i="6200,9189,10274"; a="314177974" X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="314177974" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2022 09:24:20 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,149,1643702400"; d="scan'208";a="551348182" Received: from silpixa00399126.ir.intel.com ([10.237.223.34]) by orsmga008.jf.intel.com with ESMTP; 02 Mar 2022 09:24:19 -0800 From: Bruce Richardson To: dev@dpdk.org Cc: stable@dpdk.org, Bruce Richardson Subject: [PATCH 9/9] doc/linux_gsg: change informational warnings to notes Date: Wed, 2 Mar 2022 17:22:17 +0000 Message-Id: <20220302172217.472279-10-bruce.richardson@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220302172217.472279-1-bruce.richardson@intel.com> References: <20220302172217.472279-1-bruce.richardson@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org There are two warnings in the VFIO section about limitations of VFIO and limitations on who can bind/unbind devices. Since these don't actually describe any unsafe conditions, and are more informational, we can change these to notes. This also helps emphasise the other warnings in the documents which flag genuine security concerns. Signed-off-by: Bruce Richardson --- doc/guides/linux_gsg/linux_drivers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index ef91999dd9..03cf264a0a 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -48,7 +48,7 @@ by calling the script with the ``--help`` or ``--usage`` options. Note that the UIO or VFIO kernel modules to be used, should be loaded into the kernel before running the ``dpdk-devbind.py`` script. -.. warning:: +.. note:: Due to the way VFIO works, there are certain limitations to which devices can be used with VFIO. @@ -62,7 +62,7 @@ should be loaded into the kernel before running the ``dpdk-devbind.py`` script. Therefore, the bridge driver should also be unbound from the bridge PCI device for VFIO to work with devices behind the bridge. -.. warning:: +.. note:: While any user can run the ``dpdk-devbind.py`` script to view the status of the network ports,