[v1,10/11] doc: add IPN3KE document
Checks
Commit Message
Add Intel FPGA Acceleration NIC IPN3KE document.
Signed-off-by: Rosen Xu <rosen.xu@intel.com>
Signed-off-by: Dan Wei <dan.wei@intel.com>
---
doc/guides/nics/features/ipn3ke.ini | 57 ++++++++++++++++++++++
doc/guides/nics/index.rst | 1 +
doc/guides/nics/ipn3ke.rst | 97 +++++++++++++++++++++++++++++++++++++
3 files changed, 155 insertions(+)
create mode 100644 doc/guides/nics/features/ipn3ke.ini
create mode 100644 doc/guides/nics/ipn3ke.rst
Comments
On 2/28/2019 7:13 AM, Rosen Xu wrote:
> Add Intel FPGA Acceleration NIC IPN3KE document.
No need to separate the doc patch, can merge into the one that adds the driver.
>
> Signed-off-by: Rosen Xu <rosen.xu@intel.com>
> Signed-off-by: Dan Wei <dan.wei@intel.com>
> ---
> doc/guides/nics/features/ipn3ke.ini | 57 ++++++++++++++++++++++
> doc/guides/nics/index.rst | 1 +
> doc/guides/nics/ipn3ke.rst | 97 +++++++++++++++++++++++++++++++++++++
> 3 files changed, 155 insertions(+)
> create mode 100644 doc/guides/nics/features/ipn3ke.ini
> create mode 100644 doc/guides/nics/ipn3ke.rst
>
> diff --git a/doc/guides/nics/features/ipn3ke.ini b/doc/guides/nics/features/ipn3ke.ini
> new file mode 100644
> index 0000000..06cfaf5
> --- /dev/null
> +++ b/doc/guides/nics/features/ipn3ke.ini
> @@ -0,0 +1,57 @@
> +;
> +; Supported features of the 'ipn3ke' network poll mode driver.
> +;
> +; Refer to default.ini for the full list of available PMD features.
> +;
> +[Features]
> +Speed capabilities = Y
> +Link status = Y
> +Link status event = Y
> +Rx interrupt = Y
> +Queue start/stop = Y
> +Runtime Rx queue setup = Y
> +Runtime Tx queue setup = Y
> +Jumbo frame = Y
> +Scattered Rx = Y
> +TSO = Y
> +Promiscuous mode = Y
> +Allmulticast mode = Y
> +Unicast MAC filter = Y
> +Multicast MAC filter = Y
> +RSS hash = Y
> +RSS key update = Y
> +RSS reta update = Y
> +VMDq = Y
> +SR-IOV = Y
> +DCB = Y
> +VLAN filter = Y
> +Ethertype filter = Y
> +Tunnel filter = Y
> +Hash filter = Y
> +Flow director = Y
> +Flow control = Y
> +Flow API = Y
> +Traffic mirroring = Y
> +CRC offload = Y
> +VLAN offload = Y
> +QinQ offload = Y
> +L3 checksum offload = Y
> +L4 checksum offload = Y
> +Inner L3 checksum = Y
> +Inner L4 checksum = Y
> +Packet type parsing = Y
> +Timesync = Y
> +Rx descriptor status = Y
> +Tx descriptor status = Y
> +Basic stats = Y
> +Extended stats = Y
> +FW version = Y
> +Module EEPROM dump = Y
> +Multiprocess aware = Y
> +BSD nic_uio = Y
> +Linux UIO = Y
> +Linux VFIO = Y
> +x86-32 = Y
> +x86-64 = Y
> +ARMv8 = Y
> +Power8 = Y
Are these all marked to 'Y' because the underlying PMD is i40e?
Is ipn3ke really verified with Power8, arm etc?
> diff --git a/doc/guides/nics/index.rst b/doc/guides/nics/index.rst
> index 5c80e3b..6671481 100644
> --- a/doc/guides/nics/index.rst
> +++ b/doc/guides/nics/index.rst
> @@ -28,6 +28,7 @@ Network Interface Controller Drivers
> fm10k
> i40e
> ice
> + ipn3ke
Please add alphabetically sorted.
> ifc
> igb
> ixgbe
> diff --git a/doc/guides/nics/ipn3ke.rst b/doc/guides/nics/ipn3ke.rst
> new file mode 100644
> index 0000000..7386075
> --- /dev/null
> +++ b/doc/guides/nics/ipn3ke.rst
> @@ -0,0 +1,97 @@
> +.. SPDX-License-Identifier: BSD-3-Clause
> + Copyright(c) 2019 Intel Corporation.
> +
> +IPN3KE Poll Mode Driver
> +=======================
> +
> +The ipn3ke PMD (librte_pmd_ipn3ke) provides poll mode driver support
> +for Intel® FPGA PAC(Programmable Acceleration Card) N3000 based on
> +the Intel Ethernet Controller X710/XXV710 and Intel Arria 10 FPGA.
> +
> +In this card, FPGA is an acceleration bridge between network interface
> +and the Intel Ethernet Controller. Although both FPGA and Ethernet
> +Controllers are connected to CPU with PCIe Gen3x16 Switch, all the
> +packet RX/TX is handled by Intel Ethernet Controller. So from application
> +point of view the data path is still the legacy Intel Ethernet Controller
> +X710/XXV710 PMD. Besides this, users can enable more acceleration
> +features by FPGA IP.
> +
> +Prerequisites
> +-------------
> +
> +- Identifying your adapter using `Intel Support
> + <http://www.intel.com/support>`_ and get the latest NVM/FW images.
> +
> +- Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup the basic DPDK environment.
> +
> +- To get better performance on Intel platforms, please follow the "How to get best performance with NICs on Intel platforms"
> + section of the :ref:`Getting Started Guide for Linux <linux_gsg>`.
> +
> +
> +Pre-Installation Configuration
> +------------------------------
> +
> +Config File Options
> +~~~~~~~~~~~~~~~~~~~
> +
> +The following options can be modified in the ``config`` file.
> +Please note that enabling debugging options may affect system performance.
> +
> +- ``CONFIG_RTE_LIBRTE_IPN3KE_PMD`` (default ``n``)
> +
> + Toggle compilation of the ``librte_pmd_ipn3ke`` driver.
> +
> +- ``CONFIG_RTE_LIBRTE_IPN3KE_DEBUG_*`` (default ``n``)
The name is wrong, there is no DEBUG_*, also it seems not used, if so please remove.
> +
> + Toggle display of generic debugging messages.
> +
> +Runtime Config Options
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +- ``Maximum Number of Queue Pairs``
> +
> + The maximum number of queue pairs is decided by HW. If not configured, APP
> + uses the number from HW. Users can check the number by calling the API
> + ``rte_eth_dev_info_get``.
> + If users want to limit the number of queues, they can set a smaller number
> + using EAL parameter like ``max_queue_pair_num=n``.
Is this copy/paste error?
There is no 'max_queue_pair_num' device argument in this driver.
And there are other devargs, seems not documented here.
> +
> +
> +Driver compilation and testing
> +------------------------------
> +
> +Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
> +for details.
> +
> +Sample Application Notes
> +------------------------
> +
> +Packet TX/RX with FPGA Pass-through image
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +FPGA Pass-through bitstream is original FPGA Image.
> +
> +To start ``testpmd``, and add I40e PF to FPGA network port:
> +
> +.. code-block:: console
> +
> + ./app/testpmd -l 0-15 -n 4 --vdev 'ifpga_rawdev_cfg0,ifpga=b3:00.0,port=0' --vdev 'ipn3ke_cfg0,afu=0|b3:00.0,i40e_pf={0000:b1:00.0|0000:b1:00.1|0000:b1:00.2|0000:b1:00.3|0000:b5:00.0|0000:b5:00.1|0000:b5:00.2|0000:b5:00.3}' -- -i --no-numa --port-topology=loop
Can you please help me understand?
raw device ifpga_rawdev_cfg0, used to get devargs and to create the AFU device.
AFU driver is already in this patch, net_ipn3ke_afu. It should be probed after
rawdev created the device.
vdev ipn3ke_cfg0, used to get devargs and link afu device and i40e devices,
so i40e devices should be already probed in this stage, and port representors
created for AFU device.
Is above correct?
Overall there is a IFPGA bus and AFU device work going on, but they are behind
rawdev and vdev.
> +
> +HQoS and flow acceleration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +HQoS and flow acceleration bitstream is used to offloading HQoS and flow classifier.
> +
> +To start ``testpmd``, and add I40e PF to FPGA network port, enable FPGA HQoS and Flow Acceleration:
> +
> +.. code-block:: console
> +
> + ./app/testpmd -l 0-15 -n 4 --vdev 'ifpga_rawdev_cfg0,ifpga=b3:00.0,port=0' --vdev 'ipn3ke_cfg0,afu=0|b3:00.0,fpga_acc={tm|flow},i40e_pf={0000:b1:00.0|0000:b1:00.1|0000:b1:00.2|0000:b1:00.3|0000:b5:00.0|0000:b5:00.1|0000:b5:00.2|0000:b5:00.3}' -- -i --no-numa --forward-mode=macswap
> +
> +Limitations or Known issues
> +---------------------------
> +
> +19.05 limitation
> +~~~~~~~~~~~~~~~~
> +
> +Ipn3ke code released in 19.05 is for evaluation only.
>
> -----Original Message-----
> From: Yigit, Ferruh
> Sent: Wednesday, March 06, 2019 20:45
> To: Xu, Rosen <rosen.xu@intel.com>; dev@dpdk.org
> Cc: Zhang, Tianfei <tianfei.zhang@intel.com>; Wei, Dan
> <dan.wei@intel.com>; Pei, Andy <andy.pei@intel.com>; Yang, Qiming
> <qiming.yang@intel.com>; Wang, Haiyue <haiyue.wang@intel.com>; Chen,
> Santos <santos.chen@intel.com>; Zhang, Zhang <zhang.zhang@intel.com>
> Subject: Re: [PATCH v1 10/11] doc: add IPN3KE document
>
> On 2/28/2019 7:13 AM, Rosen Xu wrote:
> > Add Intel FPGA Acceleration NIC IPN3KE document.
>
> No need to separate the doc patch, can merge into the one that adds the
> driver.
Okay, I will apply it in patch v2.
> >
> > Signed-off-by: Rosen Xu <rosen.xu@intel.com>
> > Signed-off-by: Dan Wei <dan.wei@intel.com>
> > ---
> > doc/guides/nics/features/ipn3ke.ini | 57 ++++++++++++++++++++++
> > doc/guides/nics/index.rst | 1 +
> > doc/guides/nics/ipn3ke.rst | 97
> +++++++++++++++++++++++++++++++++++++
> > 3 files changed, 155 insertions(+)
> > create mode 100644 doc/guides/nics/features/ipn3ke.ini
> > create mode 100644 doc/guides/nics/ipn3ke.rst
> >
> > diff --git a/doc/guides/nics/features/ipn3ke.ini
> > b/doc/guides/nics/features/ipn3ke.ini
> > new file mode 100644
> > index 0000000..06cfaf5
> > --- /dev/null
> > +++ b/doc/guides/nics/features/ipn3ke.ini
> > @@ -0,0 +1,57 @@
> > +;
> > +; Supported features of the 'ipn3ke' network poll mode driver.
> > +;
> > +; Refer to default.ini for the full list of available PMD features.
> > +;
> > +[Features]
> > +Speed capabilities = Y
> > +Link status = Y
> > +Link status event = Y
> > +Rx interrupt = Y
> > +Queue start/stop = Y
> > +Runtime Rx queue setup = Y
> > +Runtime Tx queue setup = Y
> > +Jumbo frame = Y
> > +Scattered Rx = Y
> > +TSO = Y
> > +Promiscuous mode = Y
> > +Allmulticast mode = Y
> > +Unicast MAC filter = Y
> > +Multicast MAC filter = Y
> > +RSS hash = Y
> > +RSS key update = Y
> > +RSS reta update = Y
> > +VMDq = Y
> > +SR-IOV = Y
> > +DCB = Y
> > +VLAN filter = Y
> > +Ethertype filter = Y
> > +Tunnel filter = Y
> > +Hash filter = Y
> > +Flow director = Y
> > +Flow control = Y
> > +Flow API = Y
> > +Traffic mirroring = Y
> > +CRC offload = Y
> > +VLAN offload = Y
> > +QinQ offload = Y
> > +L3 checksum offload = Y
> > +L4 checksum offload = Y
> > +Inner L3 checksum = Y
> > +Inner L4 checksum = Y
> > +Packet type parsing = Y
> > +Timesync = Y
> > +Rx descriptor status = Y
> > +Tx descriptor status = Y
> > +Basic stats = Y
> > +Extended stats = Y
> > +FW version = Y
> > +Module EEPROM dump = Y
> > +Multiprocess aware = Y
> > +BSD nic_uio = Y
> > +Linux UIO = Y
> > +Linux VFIO = Y
> > +x86-32 = Y
> > +x86-64 = Y
> > +ARMv8 = Y
> > +Power8 = Y
>
>
> Are these all marked to 'Y' because the underlying PMD is i40e?
Yes, because there are 2xFVL in Vista Creek, and currently all the CPU
packets TX/RX are handled by FVL.
> Is ipn3ke really verified with Power8, arm etc?
Just copy from FVL, because FVL has this feature and there are 2xFVL in
Vista Creek.
> > diff --git a/doc/guides/nics/index.rst b/doc/guides/nics/index.rst
> > index 5c80e3b..6671481 100644
> > --- a/doc/guides/nics/index.rst
> > +++ b/doc/guides/nics/index.rst
> > @@ -28,6 +28,7 @@ Network Interface Controller Drivers
> > fm10k
> > i40e
> > ice
> > + ipn3ke
>
> Please add alphabetically sorted.
Okay
> > ifc
> > igb
> > ixgbe
> > diff --git a/doc/guides/nics/ipn3ke.rst b/doc/guides/nics/ipn3ke.rst
> > new file mode 100644 index 0000000..7386075
> > --- /dev/null
> > +++ b/doc/guides/nics/ipn3ke.rst
> > @@ -0,0 +1,97 @@
> > +.. SPDX-License-Identifier: BSD-3-Clause
> > + Copyright(c) 2019 Intel Corporation.
> > +
> > +IPN3KE Poll Mode Driver
> > +=======================
> > +
> > +The ipn3ke PMD (librte_pmd_ipn3ke) provides poll mode driver support
> > +for Intel® FPGA PAC(Programmable Acceleration Card) N3000 based on
> > +the Intel Ethernet Controller X710/XXV710 and Intel Arria 10 FPGA.
> > +
> > +In this card, FPGA is an acceleration bridge between network
> > +interface and the Intel Ethernet Controller. Although both FPGA and
> > +Ethernet Controllers are connected to CPU with PCIe Gen3x16 Switch,
> > +all the packet RX/TX is handled by Intel Ethernet Controller. So from
> > +application point of view the data path is still the legacy Intel
> > +Ethernet Controller
> > +X710/XXV710 PMD. Besides this, users can enable more acceleration
> > +features by FPGA IP.
> > +
> > +Prerequisites
> > +-------------
> > +
> > +- Identifying your adapter using `Intel Support
> > + <http://www.intel.com/support>`_ and get the latest NVM/FW images.
> > +
> > +- Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to
> setup the basic DPDK environment.
> > +
> > +- To get better performance on Intel platforms, please follow the "How to
> get best performance with NICs on Intel platforms"
> > + section of the :ref:`Getting Started Guide for Linux <linux_gsg>`.
> > +
> > +
> > +Pre-Installation Configuration
> > +------------------------------
> > +
> > +Config File Options
> > +~~~~~~~~~~~~~~~~~~~
> > +
> > +The following options can be modified in the ``config`` file.
> > +Please note that enabling debugging options may affect system
> performance.
> > +
> > +- ``CONFIG_RTE_LIBRTE_IPN3KE_PMD`` (default ``n``)
> > +
> > + Toggle compilation of the ``librte_pmd_ipn3ke`` driver.
> > +
> > +- ``CONFIG_RTE_LIBRTE_IPN3KE_DEBUG_*`` (default ``n``)
>
> The name is wrong, there is no DEBUG_*, also it seems not used, if so please
> remove.
Fix in patch v2.
> > +
> > + Toggle display of generic debugging messages.
> > +
> > +Runtime Config Options
> > +~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +- ``Maximum Number of Queue Pairs``
> > +
> > + The maximum number of queue pairs is decided by HW. If not
> > + configured, APP uses the number from HW. Users can check the number
> > + by calling the API ``rte_eth_dev_info_get``.
> > + If users want to limit the number of queues, they can set a smaller
> > + number using EAL parameter like ``max_queue_pair_num=n``.
>
> Is this copy/paste error?
> There is no 'max_queue_pair_num' device argument in this driver.
>
> And there are other devargs, seems not documented here.
Fix in patch v2.
> > +
> > +
> > +Driver compilation and testing
> > +------------------------------
> > +
> > +Refer to the document :ref:`compiling and testing a PMD for a NIC
> > +<pmd_build_and_test>` for details.
> > +
> > +Sample Application Notes
> > +------------------------
> > +
> > +Packet TX/RX with FPGA Pass-through image
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +FPGA Pass-through bitstream is original FPGA Image.
> > +
> > +To start ``testpmd``, and add I40e PF to FPGA network port:
> > +
> > +.. code-block:: console
> > +
> > + ./app/testpmd -l 0-15 -n 4 --vdev
> > + 'ifpga_rawdev_cfg0,ifpga=b3:00.0,port=0' --vdev
This is IFPGA BUS AFU configuration, same as before.
> > + 'ipn3ke_cfg0,afu=0|b3:00.0,i40e_pf={0000:b1:00.0|0000:b1:00.1|0000:b
> > + 1:00.2|0000:b1:00.3|0000:b5:00.0|0000:b5:00.1|0000:b5:00.2|0000:b5:0
In IPN3KE, there are one2one port mapping relationship between network side(port representor)
and FVL PF, this example is to create mapping of 8 ports.
> > + 0.3}' -- -i --no-numa --port-topology=loop
>
> Can you please help me understand?
Pls see my reply.
> raw device ifpga_rawdev_cfg0, used to get devargs and to create the AFU
> device.
Yes
> AFU driver is already in this patch, net_ipn3ke_afu. It should be probed after
> rawdev created the device.
Yes
> vdev ipn3ke_cfg0, used to get devargs and link afu device and i40e devices,
> so i40e devices should be already probed in this stage, and port representors
> created for AFU device.
Yes
> Is above correct?
Yes
> Overall there is a IFPGA bus and AFU device work going on, but they are
> behind rawdev and vdev.
Yes
> > +
> > +HQoS and flow acceleration
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +HQoS and flow acceleration bitstream is used to offloading HQoS and flow
> classifier.
> > +
> > +To start ``testpmd``, and add I40e PF to FPGA network port, enable FPGA
> HQoS and Flow Acceleration:
> > +
> > +.. code-block:: console
> > +
> > + ./app/testpmd -l 0-15 -n 4 --vdev
> > + 'ifpga_rawdev_cfg0,ifpga=b3:00.0,port=0' --vdev
> > + 'ipn3ke_cfg0,afu=0|b3:00.0,fpga_acc={tm|flow},i40e_pf={0000:b1:00.0|
> > + 0000:b1:00.1|0000:b1:00.2|0000:b1:00.3|0000:b5:00.0|0000:b5:00.1|000
> > + 0:b5:00.2|0000:b5:00.3}' -- -i --no-numa --forward-mode=macswap
> > +
> > +Limitations or Known issues
> > +---------------------------
> > +
> > +19.05 limitation
> > +~~~~~~~~~~~~~~~~
> > +
> > +Ipn3ke code released in 19.05 is for evaluation only.
> >
new file mode 100644
@@ -0,0 +1,57 @@
+;
+; Supported features of the 'ipn3ke' network poll mode driver.
+;
+; Refer to default.ini for the full list of available PMD features.
+;
+[Features]
+Speed capabilities = Y
+Link status = Y
+Link status event = Y
+Rx interrupt = Y
+Queue start/stop = Y
+Runtime Rx queue setup = Y
+Runtime Tx queue setup = Y
+Jumbo frame = Y
+Scattered Rx = Y
+TSO = Y
+Promiscuous mode = Y
+Allmulticast mode = Y
+Unicast MAC filter = Y
+Multicast MAC filter = Y
+RSS hash = Y
+RSS key update = Y
+RSS reta update = Y
+VMDq = Y
+SR-IOV = Y
+DCB = Y
+VLAN filter = Y
+Ethertype filter = Y
+Tunnel filter = Y
+Hash filter = Y
+Flow director = Y
+Flow control = Y
+Flow API = Y
+Traffic mirroring = Y
+CRC offload = Y
+VLAN offload = Y
+QinQ offload = Y
+L3 checksum offload = Y
+L4 checksum offload = Y
+Inner L3 checksum = Y
+Inner L4 checksum = Y
+Packet type parsing = Y
+Timesync = Y
+Rx descriptor status = Y
+Tx descriptor status = Y
+Basic stats = Y
+Extended stats = Y
+FW version = Y
+Module EEPROM dump = Y
+Multiprocess aware = Y
+BSD nic_uio = Y
+Linux UIO = Y
+Linux VFIO = Y
+x86-32 = Y
+x86-64 = Y
+ARMv8 = Y
+Power8 = Y
@@ -28,6 +28,7 @@ Network Interface Controller Drivers
fm10k
i40e
ice
+ ipn3ke
ifc
igb
ixgbe
new file mode 100644
@@ -0,0 +1,97 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2019 Intel Corporation.
+
+IPN3KE Poll Mode Driver
+=======================
+
+The ipn3ke PMD (librte_pmd_ipn3ke) provides poll mode driver support
+for Intel® FPGA PAC(Programmable Acceleration Card) N3000 based on
+the Intel Ethernet Controller X710/XXV710 and Intel Arria 10 FPGA.
+
+In this card, FPGA is an acceleration bridge between network interface
+and the Intel Ethernet Controller. Although both FPGA and Ethernet
+Controllers are connected to CPU with PCIe Gen3x16 Switch, all the
+packet RX/TX is handled by Intel Ethernet Controller. So from application
+point of view the data path is still the legacy Intel Ethernet Controller
+X710/XXV710 PMD. Besides this, users can enable more acceleration
+features by FPGA IP.
+
+Prerequisites
+-------------
+
+- Identifying your adapter using `Intel Support
+ <http://www.intel.com/support>`_ and get the latest NVM/FW images.
+
+- Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup the basic DPDK environment.
+
+- To get better performance on Intel platforms, please follow the "How to get best performance with NICs on Intel platforms"
+ section of the :ref:`Getting Started Guide for Linux <linux_gsg>`.
+
+
+Pre-Installation Configuration
+------------------------------
+
+Config File Options
+~~~~~~~~~~~~~~~~~~~
+
+The following options can be modified in the ``config`` file.
+Please note that enabling debugging options may affect system performance.
+
+- ``CONFIG_RTE_LIBRTE_IPN3KE_PMD`` (default ``n``)
+
+ Toggle compilation of the ``librte_pmd_ipn3ke`` driver.
+
+- ``CONFIG_RTE_LIBRTE_IPN3KE_DEBUG_*`` (default ``n``)
+
+ Toggle display of generic debugging messages.
+
+Runtime Config Options
+~~~~~~~~~~~~~~~~~~~~~~
+
+- ``Maximum Number of Queue Pairs``
+
+ The maximum number of queue pairs is decided by HW. If not configured, APP
+ uses the number from HW. Users can check the number by calling the API
+ ``rte_eth_dev_info_get``.
+ If users want to limit the number of queues, they can set a smaller number
+ using EAL parameter like ``max_queue_pair_num=n``.
+
+
+Driver compilation and testing
+------------------------------
+
+Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
+for details.
+
+Sample Application Notes
+------------------------
+
+Packet TX/RX with FPGA Pass-through image
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+FPGA Pass-through bitstream is original FPGA Image.
+
+To start ``testpmd``, and add I40e PF to FPGA network port:
+
+.. code-block:: console
+
+ ./app/testpmd -l 0-15 -n 4 --vdev 'ifpga_rawdev_cfg0,ifpga=b3:00.0,port=0' --vdev 'ipn3ke_cfg0,afu=0|b3:00.0,i40e_pf={0000:b1:00.0|0000:b1:00.1|0000:b1:00.2|0000:b1:00.3|0000:b5:00.0|0000:b5:00.1|0000:b5:00.2|0000:b5:00.3}' -- -i --no-numa --port-topology=loop
+
+HQoS and flow acceleration
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+HQoS and flow acceleration bitstream is used to offloading HQoS and flow classifier.
+
+To start ``testpmd``, and add I40e PF to FPGA network port, enable FPGA HQoS and Flow Acceleration:
+
+.. code-block:: console
+
+ ./app/testpmd -l 0-15 -n 4 --vdev 'ifpga_rawdev_cfg0,ifpga=b3:00.0,port=0' --vdev 'ipn3ke_cfg0,afu=0|b3:00.0,fpga_acc={tm|flow},i40e_pf={0000:b1:00.0|0000:b1:00.1|0000:b1:00.2|0000:b1:00.3|0000:b5:00.0|0000:b5:00.1|0000:b5:00.2|0000:b5:00.3}' -- -i --no-numa --forward-mode=macswap
+
+Limitations or Known issues
+---------------------------
+
+19.05 limitation
+~~~~~~~~~~~~~~~~
+
+Ipn3ke code released in 19.05 is for evaluation only.