From patchwork Fri Mar 27 10:18:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michal Krawczyk X-Patchwork-Id: 67301 X-Patchwork-Delegate: andrew.rybchenko@oktetlabs.ru Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id B5B96A0589; Fri, 27 Mar 2020 11:33:49 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id AE7681C1D8; Fri, 27 Mar 2020 11:30:00 +0100 (CET) Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) by dpdk.org (Postfix) with ESMTP id 01DCA1C0B6 for ; Fri, 27 Mar 2020 11:29:48 +0100 (CET) Received: by mail-lj1-f177.google.com with SMTP id g12so9669485ljj.3 for ; Fri, 27 Mar 2020 03:29:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=37TX1haI8lfTSbafjjq6dd3cZpyImpd/bvell024amo=; b=ehvA4gcB/6XDO+KmgMNMzdOZ8z4dTixyKdse8KobbsKfGOXEhSo3RzOeWkw60iqUux XPjpHitTYH1pV2BwRb/nulKDZhts/xRr0QSIZsTalpEmuwvxQJG4+RY3992nzEoN42ew Yj9vxrC4ZTCMR2oTOJui4pumjrLtCjwyC/g8CZE6gFEraWhKEpXcYvyD6ByYrQauqMRL sEpq/RJ0vkaLlfCrGZjSAam7YNc7E14EQpooF6lk1xA5mgIBvfZo/AWL02oBLR4E0Rj2 cyofSSg2XrluBoNa1qsK2y05wFrlv0kKRmTf4FBEdXCyy5HMTBr0RRa8eCfXdsfcZb15 6vrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=37TX1haI8lfTSbafjjq6dd3cZpyImpd/bvell024amo=; b=fHKYW3xGnCvkI0V/pCfqC/pg7oKFTdpBPCv9CoYnt2AXG9AlCoJJOYlSHTSU+UMBJc btNycxnOQtJOeKcFXXLZOdNPgFSjUMPB0WMsKTmRchzSjw4JNOW4LiVi3wYVrfY0ECjq FYwcG3E+7iC9+eTvcsSH4JYXxHVheQAzyKYGJiE8OLl0700hsk7LUQXRgE4OzkkXd99G Q+ilwo4+C3DYO69rOH9ZXOJUo4xuoxpfT2KKjhQ0KnML8jsierWgNqG4bHVytuV60fz9 rdHjaLoIxrSXRDCilatVs4BfGOexqlJX6JtMg1fEJhrCUkGhYlHrrca4zViOJZhQVnm7 fLxQ== X-Gm-Message-State: ANhLgQ3aU0tdoFSWUN/N2oDL13xLVnBYLUPWEhiEucZAVWf3FZKe8SNR spA/fqqW0zVmfMTfOPJUgvCDDRqGu0MDNg== X-Google-Smtp-Source: APiQypKGNWMzeidBn6e0/YK5oA+RbEBBb+05CAsK7N8VI+g0pn22eiyUXYPYhQVCB6IzYRd2tZ9Z2Q== X-Received: by 2002:a2e:3309:: with SMTP id d9mr8161552ljc.73.1585304988246; Fri, 27 Mar 2020 03:29:48 -0700 (PDT) Received: from localhost.localdomain (193-106-246-138.noc.fibertech.net.pl. [193.106.246.138]) by smtp.gmail.com with ESMTPSA id i11sm2789587lfo.84.2020.03.27.03.29.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Mar 2020 03:29:47 -0700 (PDT) From: Michal Krawczyk To: dev@dpdk.org Cc: mw@semihalf.com, mba@semihalf.com, gtzalik@amazon.com, evgenys@amazon.com, igorch@amazon.com, Michal Krawczyk Date: Fri, 27 Mar 2020 11:18:21 +0100 Message-Id: <20200327101823.12646-28-mk@semihalf.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200327101823.12646-1-mk@semihalf.com> References: <20200327101823.12646-1-mk@semihalf.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 27/29] doc: add notes on ENA usage on metal instances X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" As AWS metal instances are supporting IOMMU, the usage of igb_uio or vfio-pci can lead to a problems (when to use which module), especially that the vfio-pci isn't supporting SMMU on arm64. To clear up the problem of using those modules in various setup conditions (with or without IOMMU) on metal instances, more detailed explanation was added. Signed-off-by: Michal Krawczyk Reviewed-by: Igor Chauskin Reviewed-by: Guy Tzalik --- doc/guides/nics/ena.rst | 48 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/doc/guides/nics/ena.rst b/doc/guides/nics/ena.rst index a953068cac..62e05e62e6 100644 --- a/doc/guides/nics/ena.rst +++ b/doc/guides/nics/ena.rst @@ -184,11 +184,55 @@ Prerequisites echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode -#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module. + To use ``noiommu`` mode, the ``vfio-pci`` must be built with flag + ``CONFIG_VFIO_NOIOMMU``. +#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module. At this point the system should be ready to run DPDK applications. Once the -application runs to completion, the ENA can be detached from igb_uio if necessary. +application runs to completion, the ENA can be detached from attached module if +necessary. + +**Note about usage on \*.metal instances** + +On AWS, the metal instances are supporting IOMMU for both arm64 and x86_64 +hosts. + +* x86_64 (e.g. c5.metal, i3.metal): + IOMMU should be disabled by default. In that situation, the ``igb_uio`` can + be used as it is but ``vfio-pci`` should be working in no-IOMMU mode (please + see above). + + When IOMMU is enabled, ``igb_uio`` cannot be used as it's not supporting this + feature, while ``vfio-pci`` should work without any changes. + To enable IOMMU on those hosts, please update ``GRUB_CMDLINE_LINUX`` in file + ``/etc/default/grub`` with the below extra boot arguments:: + + iommu=1 intel_iommu=on + + Then, make the changes live by executing as a root:: + + # grub2-mkconfig > /boot/grub2/grub.cfg + + Finally, reboot should result in IOMMU being enabled. + +* arm64 (a1.metal): + IOMMU should be enabled by default. Unfortunately, ``vfio-pci`` isn't + supporting SMMU, which is implementation of IOMMU for arm64 architecture and + ``igb_uio`` isn't supporting IOMMU at all, so to use DPDK with ENA on those + hosts, one must disable IOMMU. This can be done by updating + ``GRUB_CMDLINE_LINUX`` in file ``/etc/default/grub`` with the extra boot + argument:: + + iommu.passthrough=1 + + Then, make the changes live by executing as a root:: + + # grub2-mkconfig > /boot/grub2/grub.cfg + + Finally, reboot should result in IOMMU being disabled. + Without IOMMU, ``igb_uio`` can be used as it is but ``vfio-pci`` should be + working in no-IOMMU mode (please see above). Usage example -------------