new file mode 100644
@@ -0,0 +1,377 @@
+Practice with TRex
+==================
+
+Download TREX
+-------------
+TREX should be installed in Tester, it could be downloaded from http://trex-tgn.cisco.com/trex/release/.
+We recommend to use the latest version v2.88.
+
+.. code-block:: console
+
+ wget http://trex-tgn.cisco.com/trex/release/v2.88.tar.gz
+
+To read more about T-Rex stateless mode, read the
+`T-Rex stateless support guide <https://trex-tgn.cisco.com/trex/doc/trex_stateless.html>`__.
+
+Configure TREX
+--------------
+
+Extract the downloaded TREX tarball
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Extract the downloaded TREX file to /opt/trex/::
+
+ root@tester:/opt# mkdir trex
+ root@tester:/opt# cd trex
+ root@tester:/opt/trex# tar -zxvf v2.88.tar.gz
+ v2.88/
+ v2.88/_t-rex-64-debug
+ v2.88/t-rex-64-debug
+ ...
+ v2.88/trex_client_v2.88.tar.gz
+
+Generate Configure File
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Identify the performance test ports in tester
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+execute the command like below to list the ports::
+
+ root@tester:/opt/trex/v2.88# ./dpdk_setup_ports.py -s
+
+ Network devices using DPDK-compatible driver
+ ============================================
+ <none>
+
+ Network devices using kernel driver
+ ===================================
+ 0000:03:00.0 'VMXNET3 Ethernet Controller' if=ens160 drv=vmxnet3 unused=igb_uio,vfio-pci,uio_pci_generic *Active*
+ 0000:0b:00.0 'Device 1592' if=ens192f0 drv=ice unused=igb_uio,vfio-pci,uio_pci_generic
+ 0000:0b:00.1 'Device 1592' if=ens192f1 drv=ice unused=igb_uio,vfio-pci,uio_pci_generic
+ Other network devices
+ =====================
+ <none>
+
+Generate Configure File
+^^^^^^^^^^^^^^^^^^^^^^^
+Assume that we use 0000:0b:00.0, 0000:0b:00.1 to connect to DUT port.
+And the ports topology as below::
+
+ trex.0 (0b:00.0/40:a6:b7:0b:55:78) <-------> DUT port 0 (05:00.0/b4:96:91:9f:63:68)
+ trex.1 (0b:00.1/40:a6:b7:0b:55:79) <-------> DUT port 1 (05:00.1/b4:96:91:9f:63:69)
+
+We need to generate the trex config file as below::
+
+ root@tester:/opt/v2.88# ./dpdk_setup_ports.py -i
+ By default, IP based configuration file will be created. Do you want to use MAC based config? (y/N)y
+ +----+------+---------+-------------------+--------------------------------------------+---------+----------+----------+
+ | ID | NUMA | PCI | MAC | Name | Driver | Linux IF | Active |
+ +====+======+=========+===================+============================================+=========+==========+==========+
+ | 0 | -1 | 03:00.0 | 00:0c:29:29:30:80 | VMXNET3 Ethernet Controller | vmxnet3 | ens160 | *Active* |
+ +----+------+---------+-------------------+--------------------------------------------+---------+----------+----------+
+ | 1 | -1 | 0b:00.0 | 40:a6:b7:0b:55:78 | Device 1592 | ice | ens192f0 | |
+ +----+------+---------+-------------------+--------------------------------------------+---------+----------+----------+
+ | 2 | -1 | 0b:00.1 | 40:a6:b7:0b:55:79 | Device 1592 | ice | ens192f1 | |
+ +----+------+---------+-------------------+--------------------------------------------+---------+----------+----------+
+ Please choose an even number of interfaces from the list above, either by ID, PCI or Linux IF
+ Stateful will use order of interfaces: Client1 Server1 Client2 Server2 etc. for flows.
+ Stateless can be in any order.
+ Enter list of interfaces separated by space (for example: 1 3) : 1 2
+ For interface 1, assuming loopback to its dual interface 2.
+ Destination MAC is 40:a6:b7:0b:55:79. Change it to MAC of DUT? (y/N).y
+ Please enter a new destination MAC of interface 1: b4:96:91:9f:63:68
+ For interface 2, assuming loopback to its dual interface 1.
+ Destination MAC is 40:a6:b7:0b:55:78. Change it to MAC of DUT? (y/N).y
+ Please enter a new destination MAC of interface 2: b4:96:91:9f:63:69
+ Print preview of generated config? (Y/n)Y
+ ### Config file generated by dpdk_setup_ports.py ###
+
+ - version: 2
+ interfaces: ['0b:00.0', '0b:00.1']
+ port_info:
+ - dest_mac: b4:96:91:9f:63:68
+ src_mac: 40:a6:b7:0b:55:78
+ - dest_mac: b4:96:91:9f:63:69
+ src_mac: 40:a6:b7:0b:55:79
+
+ platform:
+ master_thread_id: 0
+ latency_thread_id: 1
+ dual_if:
+ - socket: 0
+ threads: [2,3,4,5,6,7]
+
+ Save the config to file? (Y/n)Y
+ Default filename is /etc/trex_cfg.yaml
+ Press ENTER to confirm or enter new file:
+ File /etc/trex_cfg.yaml already exist, overwrite? (y/N)Y
+ Saved to /etc/trex_cfg.yaml.
+ root@tester:/opt/trex/v2.88#
+
+We could not modify the DUT mac during the configuration and modify it in the generated file /etc/trex_cfg.yaml.
+
+Modify Configure File
+^^^^^^^^^^^^^^^^^^^^^
+Make sure the DUT mac of the generated TREX file is correct, and add prefix and limit_memory is better::
+
+ root@tester:/opt/trex/v2.88# cat /etc/trex_cfg.yaml
+ ### Config file generated by dpdk_setup_ports.py ###
+
+ - version: 2
+ interfaces: ['0b:00.0', '0b:00.1']
+ prefix: TREX
+ limit_memory: 4096
+ port_info:
+ - dest_mac: b4:96:91:9f:63:68
+ src_mac: 40:a6:b7:0b:55:78
+ - dest_mac: b4:96:91:9f:63:69
+ src_mac: 40:a6:b7:0b:55:79
+
+ platform:
+ master_thread_id: 0
+ latency_thread_id: 1
+ dual_if:
+ - socket: 0
+ threads: [2,3,4,5,6,7]
+
+ root@tester:/opt/trex/v2.88#
+
+Replace system scapy with TREX scapy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+As the know trex issue -- trex scapy lib will be conflict with scapy installed in OS, we need the replace the scapy
+in system with trex scapy.
+
+#. backup your scapy::
+
+ cp -rf /usr/local/lib/python3.6/dist-packages/scapy /usr/local/lib/python3.6/dist-packages/scapy_backup
+
+#. unify scapy version with trex::
+
+ cp -rf /opt/trex/v2.88/external_libs/scapy-2.4.3/scapy/ /usr/local/lib/python3.6/dist-packages/scapy
+
+Configure DTS
+--------------
+
+Configure execution.cfg
+~~~~~~~~~~~~~~~~~~~~~~~
+Add the performance test suite and set perf=true::
+
+ root@tester:dts# cat execution.cfg
+ [Execution1]
+ crbs=192.168.1.1
+ drivername=vfio-pci
+ build_type=meson
+ test_suites=
+ tso,
+ targets=
+ x86_64-native-linuxapp-gcc
+ parameters=nic_type=cfg:perf=true
+ root@tester:dts#
+
+Configure $DTS_CFG_FOLDER/crbs.cfg
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Set the pktgen_group=trex or pktgen_group=TREX, this item is case insensitive::
+
+ root@tester:dts# cat $DTS_CFG_FOLDER/crbs.cfg
+ [192.168.1.1]
+ dut_ip=192.168.1.1
+ dut_user=root
+ dut_passwd=dutpwd
+ os=linux
+ dut_arch=
+ tester_ip=192.168.1.2
+ tester_passwd=testerpwd
+ ixia_group=
+ pktgen_group=trex
+ channels=4
+ bypass_core0=True
+
+Configure $DTS_CFG_FOLDER/ports.cfg
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This configuration is just same with PF function test, so if you have completed some functional test,
+you have no need to modify the $DTS_CFG_FOLDER/ports.cfg, just like below::
+
+ root@tester:dts# cat $DTS_CFG_FOLDER/ports.cfg
+ [192.168.1.1]
+ ports =
+ pci=0000:05:00.0,peer=0000:0b:00.0;
+ pci=0000:05:00.1,peer=0000:0b:00.1;
+
+In addition, it could be configured as below::
+
+ root@tester:dts# cat $DTS_CFG_FOLDER/ports.cfg
+ [192.168.1.1]
+ ports =
+ pci=0000:05:00.0,peer=TREX:0;
+ pci=0000:05:00.1,peer=TREX:1;
+
+We recommend to use the first format configuration, as it has no need to do modification when we do functional test.
+
+Configure $DTS_CFG_FOLDER/pktgen.cfg
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Fill in the $DTS_CFG_FOLDER/pktgen.cfg with your TREX setting, for the first time, you should set start_trex=yes.
+Set the IP address of the device which you installed TREX into item ‘server=’, it should be the tester IP.
+The configuration should as below::
+
+ root@tester:dts# cat $DTS_CFG_FOLDER/pktgen.cfg
+ [TREX]
+ trex_root_path=/opt/trex/v2.88
+ trex_lib_path=/opt/trex/v2.88/automation/trex_control_plane/interactive
+ config_file=/etc/trex_cfg.yaml
+ server=192.168.1.1
+ pcap_file=/opt/trex/v2.88/stl/sample.pcap
+ core_num=4
+ #core_mask=0x3
+ ip_src=16.0.0.1
+ ip_dst=10.0.0.1
+ warmup=15
+ duration=-1
+ start_trex=yes
+
+As the trex_lib_path may be different in different versions, you could find the correct path as blow command::
+
+ root@tester:/opt/trex/v2.88# find . -name trex_stl_lib
+ ./automation/trex_control_plane/interactive/trex_stl_lib
+ root@tester:/opt/trex/v2.88#
+
+
+Run DTS performance test with TREX
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now you can start DTS performance test with TREX::
+
+ root@tester:/home/zhaohy/dts# ./dts
+ dts:
+ DUT 192.168.1.1
+ tester: ssh root@192.168.1.1
+ ...
+ pktgen: ssh root@192.168.1.1
+ pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg /etc/trex_cfg.yaml -c 4
+ pktgen: Starting Scapy server..... Scapy server is started
+ Trying to bind to igb_uio ...
+ /usr/bin/python3 dpdk_nic_bind.py --bind=igb_uio 0000:0b:00.0 0000:0b:00.1
+ ...
+ TestTSO: Test Case test_perf_TSO_2ports Begin
+ dut.10.240.183.72:
+ tester:
+ TestTSO: Executing PMD using 1S/1C/2T
+ dut.10.240.183.72: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 1,45 -n 4 -a 0000:05:00.0 -a 0000:05:00.1 --file-prefix=dpdk_31529_20210324143008 -- -i --rxd=512 --txd=512 --burst=32 --rxfreet=64 --mbcache=128 --portmask=0x3 --max-pkt-len=9000 --txpt=36 --txht=0 --txwt=0 --txfreet=32 --txrst=32
+ dut.10.240.183.72: EAL: Detected 88 lcore(s)
+ ...
+ pktgen: Rx Port 0 stats:
+ rx_port: 0, rx_bps: 25354096640.000000, rx_pps: 1239130.250000
+ pktgen: throughput: pps_rx 5463897.750000, bps_rx 50961129472.000000
+ pktgen: traffic completed.
+ ...
+ TestTSO:
+ +------------+---------------+------------+
+ | Frame Size | 1S/1C/2T Mpps | % linerate |
+ +============+===============+============+
+ | 128 | 5.371 | 4 |
+ +------------+---------------+------------+
+ | 2500 | 5.464 | 56 |
+ +------------+---------------+------------+
+ TestTSO: Test Case test_perf_TSO_2ports Result PASSED
+
+FAQ
+---
+
+dpdk hugepage management conflict issue
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+trex use older dpdk version than we release cycle source code. When dpdk change
+the memory management merchanism, trex will meet the following issue.
+
+Trex should run on an independent platform. DUT/Trex should run on two platforms:
+
+* one is used as TESTER and trex server, another one is used as DUT.(dts/pktgen)
+* one is used as trex server, another one is used as DUT/TESTER.(recommended scheme)
+ This scheme can make sure that trex run on its full status capability.
+
+When trex run with dts on the same platform, trex server sometimes boot up
+failed for hugepage error.
+
+.. code-block:: console
+
+ ./t-rex-64 -i --stl -k 4
+
+ Starting Scapy server..... Scapy server is started
+ Trying to bind to igb_uio ...
+ /usr/bin/python3 dpdk_nic_bind.py --bind=igb_uio 0000:85:00.0 0000:8a:00.1
+ The ports are bound/configured.
+ Starting TRex v2.41 please wait ...
+ EAL: Can only reserve 1766 pages from 4096 requested
+ Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
+ Please either increase it or request less amount of memory.
+ EAL: FATAL: Cannot init memory
+
+ EAL: Cannot init memory
+
+ You might need to run ./trex-cfg once
+ EAL: Error - exiting with code: 1
+ Cause: Invalid EAL arguments
+
+trex quit when using Niantic
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+when bind dut NNT port to igb_uio, peer port will get a link down status, then
+trex server using NNT nic will quit.
+
+.. code-block:: console
+
+ WATCHDOG: task 'master' has not responded for more than 2.00044 seconds - timeout is 2 seconds
+
+ *** traceback follows ***
+
+ 1 0x55a7c779561a ./_t-rex-64(+0x12761a) [0x55a7c779561a]
+ 2 0x7f23da4be1b0 /lib64/libpthread.so.0(+0x121b0) [0x7f23da4be1b0]
+ 3 0x55a7c7942d40 rte_delay_us_block + 128
+ 4 0x55a7c798d731 ixgbe_setup_mac_link_multispeed_fiber + 337
+ 5 0x55a7c79a8f14 ./_t-rex-64(+0x33af14) [0x55a7c79a8f14]
+ 6 0x55a7c7954c72 rte_eth_link_get_nowait + 114
+ 7 0x55a7c776a988 DpdkTRexPortAttr::update_link_status_nowait() + 24
+ 8 0x55a7c77856a6 CGlobalTRex::handle_slow_path() + 118
+ 9 0x55a7c7785ad7 CGlobalTRex::run_in_master() + 759
+ 10 0x55a7c7785e3c ./_t-rex-64(+0x117e3c) [0x55a7c7785e3c]
+ 11 0x55a7c793efba rte_eal_mp_remote_launch + 346
+ 12 0x55a7c7789e1e main_test(int, char**) + 1038
+ 13 0x7f23d9417f2a __libc_start_main + 234
+ 14 0x55a7c7719b9d ./_t-rex-64(+0xabb9d) [0x55a7c7719b9d]
+
+
+ *** addr2line information follows ***
+
+ ??:0
+ ??:0
+ ??:0
+ ??:0
+ ??:0
+ ??:0
+ ??:0
+ ??:0
+ ??:0
+ ??:0
+ ??:0
+ ??:0
+ ??:0
+ ??:0
+
+
+ ./t-rex-64: line 80: 25870 Aborted (core dumped) ./_$(
+
+other issues
+~~~~~~~~~~~~
+
+#. linux kernel version should not be too low.
+
+#. Trex only works with even number link peers.
+
+#. Trex only works with nics, which are using the same driver.
+
+#. Before boot up trex, please make sure the peer ports are on up status.
+
+#. If you have ran dpdk on the platform which you want to deploy trex-server,
+ reboot the platform to make sure that trex-server can work fine.
+
+#. If using i40e driver, Trex v2.41 version need i40e nic firmware version newer than 5.02.
+
+#. trex will drop the received packet, which dst mac is the port mac address.