diff mbox

[dpdk-dev,13/13] net/thunderx: document secondary queue set support

Message ID 1472230448-17490-14-git-send-email-krytarowski@caviumnetworks.com (mailing list archive)
State Superseded, archived
Delegated to: Bruce Richardson
Headers

Commit Message

Kamil Rytarowski Aug. 26, 2016, 4:54 p.m. UTC 
  From: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>

Signed-off-by: Maciej Czekaj <maciej.czekaj@caviumnetworks.com>
Signed-off-by: Kamil Rytarowski <kamil.rytarowski@caviumnetworks.com>
Signed-off-by: Zyta Szpak <zyta.szpak@semihalf.com>
Signed-off-by: Slawomir Rosek <slawomir.rosek@semihalf.com>
Signed-off-by: Radoslaw Biernacki <rad@semihalf.com>
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 doc/guides/nics/thunderx.rst | 114 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 107 insertions(+), 7 deletions(-)

Comments

John McNamara Sept. 26, 2016, 8:17 p.m. UTC | #1 
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Kamil Rytarowski
> Sent: Friday, August 26, 2016 5:54 PM
> To: dev@dpdk.org
> Cc: maciej.czekaj@caviumnetworks.com; zyta.szpak@semihalf.com;
> slawomir.rosek@semihalf.com; rad@semihalf.com;
> jerin.jacob@caviumnetworks.com; Kamil Rytarowski
> <kamil.rytarowski@caviumnetworks.com>
> Subject: [dpdk-dev] [PATCH 13/13] net/thunderx: document secondary queue
> set support
> 

There are some whitespace errors in the docs:

    Applying patch #15435 using 'git am'
    Description: [dpdk-dev,13/13] net/thunderx: document secondary queue set support
    Applying: net/thunderx: document secondary queue set support
    .git/rebase-apply/patch:70: trailing whitespace.
          
    .git/rebase-apply/patch:74: trailing whitespace.
          
    .git/rebase-apply/patch:98: trailing whitespace.



Some other minor comments below.



> 
>  Supported ThunderX SoCs
>  -----------------------
> @@ -322,6 +323,112 @@ This section provides instructions to configure SR-
> IOV with Linux OS.
>  #. Refer to section :ref:`Running testpmd <thunderx_testpmd_example>` for
> instruction
>     how to launch ``testpmd`` application.
> 
> +Multiple Queue Set per DPDK port configuration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +There are two types of VFs:
> +
> +- Primary VF
> +- Secondary VF
> +
> +Each port consist of a primary VF and n secondary VF(s). Each VF provides

s/consist/consists/


> 8 Tx/Rx queues to a port.
> +In case port is configured to use more than 8 queues, then it requires
> +one (or more) secondary VF. Each secondary VF adds additional 8 queues to
> the queue set.

There are a few missing definite and indefinite articles missing in the text.


> +
> +During PMD driver initialization, the primary VF's are enumerated by
> +checking the specific flag (see sqs message in DPDK boot log - sqs
> indicates secondary queue set).
> +They are at the beginning of VF list (the remain ones are secondary
> VF's).
> +
> +The primary VFs are used as master queue sets. Secondary VFs provid

s/provid/provide/

> +additional queue sets for primary ones. If a port is configured for
> +more then
> +8 queues than it will request for additional queues from secondary VFs.
> +
> +Secondary VFs cannot be shared between primary VFs.
> +
> +Primary VFs are present on the beginning of the 'Network devices using
> +kernel driver' list, secondary VFs are on the remaining on the remaining
> part of the list.
> +
> +   .. note::
> +

This note and the following one are indented too far. They should be aligned with the margin.



> ...


> +Example device binding
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +If a system has three interfaces, a total of 18 VF devices will be
> +created on a non-NUMA machine.
> +
> +   .. note::
> +
> +      NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port.
> +
> +   .. code-block:: console

This note and code block are indented too far. They should be aligned with the margin.


> +
> +      # tools/dpdk-devbind.py --status
> +
> +      Network devices using DPDK-compatible driver
> +      ============================================
> +      <none>
> +
> +      Network devices using kernel driver
> +      ===================================
> +      0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-
> pci,uio_pci_generic
> +      0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-
> pci,uio_pci_generic
> +      0002:01:02.2 'Device 0011' if= drv=thunder-nicvf
> + unused=vfio-pci,uio_pci_generic
> +
> +      Other network devices
> +      =====================
> +      0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic
> +
> +
> +We want to bind two physical interfaces with 24 queues each device, we
> +attach two primary VFs and four secondary queues. In our example we
> choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3).
> +We will chose four secondary queue sets from the ending of the list
> (0002:01:01.7-0002:01:02.2).

s/chose/choose/

> +
> +
> +#. Bind two primary VFs to the ``vfio-pci`` driver:
> +
> +   .. code-block:: console
> +

These code blocks are indented correctly.


John.
--
Maciej Czekaj Sept. 29, 2016, 2:38 p.m. UTC | #2 
We will address all the issues in v2.


>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Kamil Rytarowski
>> Sent: Friday, August 26, 2016 5:54 PM
>> To: dev@dpdk.org
>> Cc: maciej.czekaj@caviumnetworks.com; zyta.szpak@semihalf.com;
>> slawomir.rosek@semihalf.com; rad@semihalf.com;
>> jerin.jacob@caviumnetworks.com; Kamil Rytarowski
>> <kamil.rytarowski@caviumnetworks.com>
>> Subject: [dpdk-dev] [PATCH 13/13] net/thunderx: document secondary queue
>> set support
>>
>
> There are some whitespace errors in the docs:
>
>     Applying patch #15435 using 'git am'
>     Description: [dpdk-dev,13/13] net/thunderx: document secondary queue set support
>     Applying: net/thunderx: document secondary queue set support
>     .git/rebase-apply/patch:70: trailing whitespace.
>
>     .git/rebase-apply/patch:74: trailing whitespace.
>
>     .git/rebase-apply/patch:98: trailing whitespace.
>
>
>
> Some other minor comments below.
>
>
>
>>
>>  Supported ThunderX SoCs
>>  -----------------------
>> @@ -322,6 +323,112 @@ This section provides instructions to configure SR-
>> IOV with Linux OS.
>>  #. Refer to section :ref:`Running testpmd <thunderx_testpmd_example>` for
>> instruction
>>     how to launch ``testpmd`` application.
>>
>> +Multiple Queue Set per DPDK port configuration
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +There are two types of VFs:
>> +
>> +- Primary VF
>> +- Secondary VF
>> +
>> +Each port consist of a primary VF and n secondary VF(s). Each VF provides
>
> s/consist/consists/
>
>
>> 8 Tx/Rx queues to a port.
>> +In case port is configured to use more than 8 queues, then it requires
>> +one (or more) secondary VF. Each secondary VF adds additional 8 queues to
>> the queue set.
>
> There are a few missing definite and indefinite articles missing in the text.
>
>
>> +
>> +During PMD driver initialization, the primary VF's are enumerated by
>> +checking the specific flag (see sqs message in DPDK boot log - sqs
>> indicates secondary queue set).
>> +They are at the beginning of VF list (the remain ones are secondary
>> VF's).
>> +
>> +The primary VFs are used as master queue sets. Secondary VFs provid
>
> s/provid/provide/
>
>> +additional queue sets for primary ones. If a port is configured for
>> +more then
>> +8 queues than it will request for additional queues from secondary VFs.
>> +
>> +Secondary VFs cannot be shared between primary VFs.
>> +
>> +Primary VFs are present on the beginning of the 'Network devices using
>> +kernel driver' list, secondary VFs are on the remaining on the remaining
>> part of the list.
>> +
>> +   .. note::
>> +
>
> This note and the following one are indented too far. They should be aligned with the margin.
>
>
>
>> ...
>
>
>> +Example device binding
>> +~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +If a system has three interfaces, a total of 18 VF devices will be
>> +created on a non-NUMA machine.
>> +
>> +   .. note::
>> +
>> +      NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port.
>> +
>> +   .. code-block:: console
>
> This note and code block are indented too far. They should be aligned with the margin.
>
>
>> +
>> +      # tools/dpdk-devbind.py --status
>> +
>> +      Network devices using DPDK-compatible driver
>> +      ============================================
>> +      <none>
>> +
>> +      Network devices using kernel driver
>> +      ===================================
>> +      0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-
>> pci,uio_pci_generic
>> +      0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-
>> pci,uio_pci_generic
>> +      0002:01:02.2 'Device 0011' if= drv=thunder-nicvf
>> + unused=vfio-pci,uio_pci_generic
>> +
>> +      Other network devices
>> +      =====================
>> +      0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic
>> +
>> +
>> +We want to bind two physical interfaces with 24 queues each device, we
>> +attach two primary VFs and four secondary queues. In our example we
>> choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3).
>> +We will chose four secondary queue sets from the ending of the list
>> (0002:01:01.7-0002:01:02.2).
>
> s/chose/choose/
>
>> +
>> +
>> +#. Bind two primary VFs to the ``vfio-pci`` driver:
>> +
>> +   .. code-block:: console
>> +
>
> These code blocks are indented correctly.
>
>
> John.
>
diff mbox

Patch

diff --git a/doc/guides/nics/thunderx.rst b/doc/guides/nics/thunderx.rst
index acc05a4..2b6f1f0 100644
--- a/doc/guides/nics/thunderx.rst
+++ b/doc/guides/nics/thunderx.rst
@@ -56,6 +56,7 @@  Features of the ThunderX PMD are:
 - VLAN stripping
 - SR-IOV VF
 - NUMA support
+- Multi queue set support (up to 96 queues (12 queue sets)) per port
 
 Supported ThunderX SoCs
 -----------------------
@@ -322,6 +323,112 @@  This section provides instructions to configure SR-IOV with Linux OS.
 #. Refer to section :ref:`Running testpmd <thunderx_testpmd_example>` for instruction
    how to launch ``testpmd`` application.
 
+Multiple Queue Set per DPDK port configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are two types of VFs:
+
+- Primary VF
+- Secondary VF
+
+Each port consist of a primary VF and n secondary VF(s). Each VF provides 8 Tx/Rx queues to a port.
+In case port is configured to use more than 8 queues, then it requires one (or more)
+secondary VF. Each secondary VF adds additional 8 queues to the queue set.
+
+During PMD driver initialization, the primary VF's are enumerated by checking the
+specific flag (see sqs message in DPDK boot log - sqs indicates secondary queue set).
+They are at the beginning of VF list (the remain ones are secondary VF's).
+
+The primary VFs are used as master queue sets. Secondary VFs provid
+additional queue sets for primary ones. If a port is configured for more then
+8 queues than it will request for additional queues from secondary VFs.
+
+Secondary VFs cannot be shared between primary VFs.
+
+Primary VFs are present on the beginning of the 'Network devices using kernel
+driver' list, secondary VFs are on the remaining on the remaining part of the list.
+
+   .. note::
+
+      The VNIC driver in the multiqueue setup works differently than other drivers like `ixgbe`.
+      We need to bind separately each specific queue set device with the ``tools/dpdk-devbind.py`` utility.
+
+   .. note::
+
+      Depending on the hardware used, the kernel driver sets a threshold ``vf_id``. VFs that try to attached with an id below or equal to
+      this boundary are considered primary VFs. VFs that try to attach with an id above this boundary are considered secondary VFs.
+
+
+Example device binding
+~~~~~~~~~~~~~~~~~~~~~~
+
+If a system has three interfaces, a total of 18 VF devices will be created
+on a non-NUMA machine.
+
+   .. note::
+
+      NUMA systems have 12 VFs per port and non-NUMA 6 VFs per port.
+
+   .. code-block:: console
+
+      # tools/dpdk-devbind.py --status
+      
+      Network devices using DPDK-compatible driver
+      ============================================
+      <none>
+      
+      Network devices using kernel driver
+      ===================================
+      0000:01:10.0 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic
+      0000:01:10.1 'Device a026' if= drv=thunder-BGX unused=vfio-pci,uio_pci_generic
+      0002:01:00.0 'Device a01e' if= drv=thunder-nic unused=vfio-pci,uio_pci_generic
+      0002:01:00.1 'Device 0011' if=eth0 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.2 'Device 0011' if=eth1 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.3 'Device 0011' if=eth2 drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:00.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.3 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.4 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.5 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.6 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:01.7 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:02.0 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:02.1 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      0002:01:02.2 'Device 0011' if= drv=thunder-nicvf unused=vfio-pci,uio_pci_generic
+      
+      Other network devices
+      =====================
+      0002:00:03.0 'Device a01f' unused=vfio-pci,uio_pci_generic
+
+
+We want to bind two physical interfaces with 24 queues each device, we attach two primary VFs
+and four secondary queues. In our example we choose two 10G interfaces eth1 (0002:01:00.2) and eth2 (0002:01:00.3).
+We will chose four secondary queue sets from the ending of the list (0002:01:01.7-0002:01:02.2).
+
+
+#. Bind two primary VFs to the ``vfio-pci`` driver:
+
+   .. code-block:: console
+
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:00.2
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:00.3
+
+#. Bind four primary VFs to the ``vfio-pci`` driver:
+
+   .. code-block:: console
+
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:01.7
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.0
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.1
+      tools/dpdk-devbind.py -b vfio-pci 0002:01:02.2
+
+The nicvf thunderx driver will make use of attached secondary VFs automatically during the interface configuration stage.
+
 Limitations
 -----------
 
@@ -346,10 +453,3 @@  Maximum packet segments
 The ThunderX SoC family NICs support up to 12 segments per packet when working
 in scatter/gather mode. So, setting MTU will result with ``EINVAL`` when the
 frame size does not fit in the maximum number of segments.
-
-Limited VFs
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The ThunderX SoC family NICs has 128VFs and each VF has 8/8 queues
-for RX/TX respectively. Current driver implementation has one to one mapping
-between physical port and VF hence only limited VFs can be used.