[v2,27/29] doc: add notes on ENA usage on metal instances
diff mbox series

Message ID 20200401142127.13715-28-mk@semihalf.com
State Superseded, archived
Delegated to: Ferruh Yigit
Headers show
Series
  • Update ENA driver to v2.1.0
Related show

Checks

Context Check Description
ci/Intel-compilation success Compilation OK
ci/checkpatch success coding style OK

Commit Message

Michał Krawczyk April 1, 2020, 2:21 p.m. UTC
As AWS metal instances are supporting IOMMU, the usage of igb_uio or
vfio-pci can lead to a problems (when to use which module), especially
that the vfio-pci isn't supporting SMMU on arm64.

To clear up the problem of using those modules in various setup
conditions (with or without IOMMU) on metal instances, more detailed
explanation was added.

Signed-off-by: Michal Krawczyk <mk@semihalf.com>
Reviewed-by: Igor Chauskin <igorch@amazon.com>
Reviewed-by: Guy Tzalik <gtzalik@amazon.com>
---
 doc/guides/nics/ena.rst | 48 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 46 insertions(+), 2 deletions(-)

Patch
diff mbox series

diff --git a/doc/guides/nics/ena.rst b/doc/guides/nics/ena.rst
index d5081e91e5..57749eb736 100644
--- a/doc/guides/nics/ena.rst
+++ b/doc/guides/nics/ena.rst
@@ -187,11 +187,55 @@  Prerequisites
 
      echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
 
-#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
+   To use ``noiommu`` mode, the ``vfio-pci`` must be built with flag
+   ``CONFIG_VFIO_NOIOMMU``.
 
+#. Bind the intended ENA device to ``vfio-pci`` or ``igb_uio`` module.
 
 At this point the system should be ready to run DPDK applications. Once the
-application runs to completion, the ENA can be detached from igb_uio if necessary.
+application runs to completion, the ENA can be detached from attached module if
+necessary.
+
+**Note about usage on \*.metal instances**
+
+On AWS, the metal instances are supporting IOMMU for both arm64 and x86_64
+hosts.
+
+* x86_64 (e.g. c5.metal, i3.metal):
+   IOMMU should be disabled by default. In that situation, the ``igb_uio`` can
+   be used as it is but ``vfio-pci`` should be working in no-IOMMU mode (please
+   see above).
+
+   When IOMMU is enabled, ``igb_uio`` cannot be used as it's not supporting this
+   feature, while ``vfio-pci`` should work without any changes.
+   To enable IOMMU on those hosts, please update ``GRUB_CMDLINE_LINUX`` in file
+   ``/etc/default/grub`` with the below extra boot arguments::
+
+    iommu=1 intel_iommu=on
+
+   Then, make the changes live by executing as a root::
+
+    # grub2-mkconfig > /boot/grub2/grub.cfg
+
+   Finally, reboot should result in IOMMU being enabled.
+
+* arm64 (a1.metal):
+   IOMMU should be enabled by default. Unfortunately, ``vfio-pci`` isn't
+   supporting SMMU, which is implementation of IOMMU for arm64 architecture and
+   ``igb_uio`` isn't supporting IOMMU at all, so to use DPDK with ENA on those
+   hosts, one must disable IOMMU. This can be done by updating
+   ``GRUB_CMDLINE_LINUX`` in file ``/etc/default/grub`` with the extra boot
+   argument::
+
+    iommu.passthrough=1
+
+   Then, make the changes live by executing as a root::
+
+    # grub2-mkconfig > /boot/grub2/grub.cfg
+
+   Finally, reboot should result in IOMMU being disabled.
+   Without IOMMU, ``igb_uio`` can be used as it is but ``vfio-pci`` should be
+   working in no-IOMMU mode (please see above).
 
 Usage example
 -------------