@@ -1,3 +1,35 @@
+.. Copyright (c) <2021>, Intel Corporation
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+
+ - Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+
+ - Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in
+ the documentation and/or other materials provided with the
+ distribution.
+
+ - Neither the name of Intel Corporation nor the names of its
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+ COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+ INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+ (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+ SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+ OF THE POSSIBILITY OF SUCH DAMAGE.
+
========================================================
Vswitch sample test with vhost async data path test plan
========================================================
@@ -6,17 +38,10 @@ Description
===========
Vswitch sample can leverage IOAT to accelerate vhost async data-path from dpdk 20.11. This plan test
-vhost DMA operation callbacks for CBDMAV PMD and vhost async data-path in vhost sample. Also compare
-the performance between CPU copy, CBDMA copy and Sync copy.
-CPU copy means vhost enqueue w/o cbdma channel; CBDMA copy needs vhost enqueue with cbdma channel
-using parameter '-dmas'; Sync copy needs vhost enqueue with cbdma channel, but threshold ( can be
-adjusted by change value of f.async_threshold in dpdk code) is larger than forwarding packet length.
-
-CBDMA performance indicator
-===========================
-
-(1)CPU copy vs. sync copy delta < 10% for 64B packet size
-(2)CBDMA copy vs sync copy delta > 5% for 1518 packet size
+vhost DMA operation callbacks for CBDMA PMD and vhost async data-path in vhost sample.
+From 20.11 to 21.02, only split ring support cbdma copy with vhost enqueue direction;
+from 21.05,packed ring also can support cbdma copy with vhost enqueue direction;
+from 21.08, split ring support cbdma copy both vhost enqueue and dequeue direction.
Prerequisites
=============
@@ -38,175 +63,152 @@ Modify the testpmd code as following::
Test Case1: PVP performance check with CBDMA channel using vhost async driver
=============================================================================
-1. Adjust dpdk code as below::
-
- --- a/examples/vhost/main.c
- +++ b/examples/vhost/main.c
- @@ -1343,7 +1343,7 @@ new_device(int vid)
-
- if (async_vhost_driver) {
- f.async_inorder = 1;
- - f.async_threshold = 256;
- + f.async_threshold = 1518;
- return rte_vhost_async_channel_register(vid, VIRTIO_RXQ,
- f.intval, &channel_ops);
- }
-
-2. Bind physical port to vfio-pci and CBDMA channel to igb_uio.
+1. Bind physical port to vfio-pci and CBDMA channel to igb_uio.
-3. On host, launch dpdk-vhost by below command::
+2. On host, launch dpdk-vhost by below command::
./dpdk-vhost -c 0x1c000000 -n 4 -- \
- -p 0x1 --mergeable 1 --vm2vm 1 --dma-type ioat --stats 1 --socket-file /tmp/vhost-net --dmas [txd0@00:04.0]
+ -p 0x1 --mergeable 1 --vm2vm 1 --dma-type ioat --stats 1 --socket-file /tmp/vhost-net --dmas [txd0@0000:00:04.0] --client
-4. Launch virtio-user with testpmd::
+3. Launch virtio-user with packed ring::
./dpdk-testpmd -l 29-30 -n 4 --no-pci --file-prefix=testpmd0 \
- --vdev=net_virtio_user0,mac=00:11:22:33:44:10,path=/tmp/vhost-net,queues=1 -- -i --rxq=1 --txq=1 --txd=1024 --rxd=1024 --nb-cores=1
+ --vdev=net_virtio_user0,mac=00:11:22:33:44:10,path=/tmp/vhost-net,queues=1,mrg_rxbuf=0,in_order=1,vectorized=1,packed_vq=1,server=1 -- -i --rxq=1 --txq=1 --txd=1024 --rxd=1024 --nb-cores=1
-5. Start pkts from virtio-user side to let vswitch know the mac addr::
+4. Start pkts from virtio-user side to let vswitch know the mac addr::
- testpmd>set fwd mac #if use io fwd ,ixia can't receive packets
+ testpmd>set fwd mac
testpmd>start tx_first
- testpmd>stop
- testpmd>start #start to forward
-
-6. Inject different length pkts (packets length=64,512,1024,1518) seperately with VLAN_id=1000 and dest_mac=virtio_mac_addresss ( specific in above cmd with 00:11:22:33:44:10) to NIC using packet generator, record performance number.
-
-7. Adjust vswitch code as below, re-test step 3-6 and record perfromance of different packet length::
-
- --- a/examples/vhost/main.c
- +++ b/examples/vhost/main.c
- @@ -1343,7 +1343,7 @@ new_device(int vid)
-
- if (async_vhost_driver) {
- f.async_inorder = 1;
- - f.async_threshold = 256;
- + f.async_threshold = 0;
- return rte_vhost_async_channel_register(vid, VIRTIO_RXQ,
- f.intval, &channel_ops);
- }
-8. Re-test step 3-6 except to launch dpdk-vhost by below command, record perfromance of different packet length::
+5. Inject pkts (packets length=64...1518) separately with VLAN_id=1000 and dest_mac=virtio_mac_addresss (specific in above cmd with 00:11:22:33:44:10) to NIC using packet generator, record pvp (PG>nic>vswitch>virtio-user>vswitch>nic>PG) performance number can get expected.
- ./dpdk-vhost -c 0x1c000000 -n 4 -- -p 0x1 --mergeable 1 --vm2vm 1 --stats 1 --socket-file /tmp/vhost-net
+6. Quit and re-launch virtio-user with split ring::
-9. Compare performance, check below two performance indicator::
+ ./dpdk-testpmd -l 29-30 -n 4 --no-pci --file-prefix=testpmd0 \
+ --vdev=net_virtio_user0,mac=00:11:22:33:44:10,path=/tmp/vhost-net,queues=1,mrg_rxbuf=1,in_order=1,vectorized=1,server=1 -- -i --rxq=1 --txq=1 --txd=1024 --rxd=1024 --nb-cores=1
- (1)CPU copy vs. sync copy delta < 10% for 64B packet size
- (2)CBDMA copy vs sync copy delta > 5% for 1518 packet size
+7. Re-test step 4-5, record performance of different packet length.
-Test Case2: PV test with multiple CBDMA channels using vhost async driver
-==========================================================================
+Test Case2: PVP test with two VM and two CBDMA channels using vhost async driver
+=================================================================================
1. Bind one physical ports to vfio-pci and two CBDMA channels to igb_uio.
2. On host, launch dpdk-vhost by below command::
./dpdk-vhost -l 26-28 -n 4 -- \
- -p 0x1 --mergeable 1 --vm2vm 1 --dma-type ioat --stats 1 --socket-file /tmp/vhost-net0 --socket-file /tmp/vhost-net1 --dmas [txd0@00:04.0,txd1@00:04.1] --client
+ -p 0x1 --mergeable 1 --vm2vm 1 --dma-type ioat --stats 1 --socket-file /tmp/vhost-net0 --socket-file /tmp/vhost-net1 --dmas [txd0@0000:00:04.0,txd1@0000:00:04.1] --client
3. launch two virtio-user ports::
./dpdk-testpmd -l 29-30 -n 4 --no-pci --file-prefix=testpmd0 \
- --vdev=net_virtio_user0,mac=00:11:22:33:44:10,path=/tmp/vhost-net0,queues=1,server=1 -- -i --rxq=1 --txq=1 --txd=1024 --rxd=1024 --nb-cores=1
+ --vdev=net_virtio_user0,mac=00:11:22:33:44:10,path=/tmp/vhost-net0,queues=1,server=1,mrg_rxbuf=0,in_order=1,vectorized=1,packed_vq=1 -- -i --rxq=1 --txq=1 --txd=1024 --rxd=1024 --nb-cores=1
./dpdk-testpmd -l 31-32 -n 4 --no-pci --file-prefix=testpmd1 \
- --vdev=net_virtio_user0,mac=00:11:22:33:44:11,path=/tmp/vhost-net1,queues=1,server=1,mrg_rxbuf=0,in_order=1,vectorized=1 -- -i --rxq=1 --txq=1 --txd=1024 --rxd=1024 --nb-cores=1
+ --vdev=net_virtio_user0,mac=00:11:22:33:44:11,path=/tmp/vhost-net1,queues=1,server=1,mrg_rxbuf=1,in_order=1,vectorized=1 -- -i --rxq=1 --txq=1 --txd=1024 --rxd=1024 --nb-cores=1
4. Start pkts from two virtio-user side individually to let vswitch know the mac addr::
- testpmd0>start tx_first
+ testpmd0>set fwd mac
+ testpmd1>set fwd mac
+ testpmd1>start tx_first
testpmd1>start tx_first
-5. Inject 1518B packets with VLAN_id=1000 and dest_mac=virtio_mac_addresss (00:11:22:33:44:10 and 00:11:22:33:44:11) to NIC using packet generator,check two virtio-user ports can receive according packets.
+5. Inject IMIX packets (64b...1518b) with VLAN_id=1000 and dest_mac=virtio_mac_addresss (00:11:22:33:44:10 and 00:11:22:33:44:11) to NIC using packet generator,record performance number can get expected from Packet generator rx side.
6. Stop dpdk-vhost side and relaunch it with same cmd as step2.
-7.Start pkts from two virtio-user side individually to let vswitch know the mac addr, check two virtio-user ports still can receive according packets and get same throughput as step5::
+7. Start pkts from two virtio-user side individually to let vswitch know the mac addr::
+
+ testpmd0>stop
+ testpmd0>start tx_first
+ testpmd1>stop
+ testpmd1>start tx_first
- testpmd0>stop
- testpmd0>start tx_first
- testpmd1>stop
- testpmd1>start tx_first
+8. Inject IMIX packets (64b...1518b) to NIC using packet generator, ensure get same throughput as step5.
-Test Case3: VM2VM performance test with two CBDMA channels using vhost async driver
-====================================================================================
+Test Case3: VM2VM forwarding test with two CBDMA channels
+=========================================================
1.Bind one physical ports to vfio-pci and two CBDMA channels to igb_uio.
2. On host, launch dpdk-vhost by below command::
- ./dpdk-vhost -l 26-28 -n 4 -- -p 0x1 --mergeable 1 --vm2vm 1 --dma-type ioat \
- --socket-file /tmp/vhost-net0 --socket-file /tmp/vhost-net1 --dmas [txd0@00:04.0,txd1@00:04.1]
+ ./x86_64-native-linuxapp-gcc/examples/dpdk-vhost -l 26-28 -n 4 -- -p 0x1 --mergeable 1 --vm2vm 1 --dma-type ioat \
+ --socket-file /tmp/vhost-net0 --socket-file /tmp/vhost-net1 --dmas [txd0@0000:00:04.0,txd1@0000:00:04.1] --client
3. Launch virtio-user::
- ./dpdk-testpmd -l 29-30 -n 4 --no-pci --file-prefix=testpmd0 \
- --vdev=net_virtio_user0,mac=00:11:22:33:44:10,path=/tmp/vhost-net0,queues=1 -- -i --rxq=1 --txq=1 --txd=1024 --rxd=1024 --nb-cores=1
+ ./x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 29-30 -n 4 --no-pci --file-prefix=testpmd0 \
+ --vdev=net_virtio_user0,mac=00:11:22:33:44:10,path=/tmp/vhost-net0,queues=1,server=1,mrg_rxbuf=1,in_order=1,vectorized=1,packed_vq=1 -- -i --rxq=1 --txq=1 --txd=1024 --rxd=1024 --nb-cores=1
- ./dpdk-testpmd -l 31-32 -n 4 --no-pci --file-prefix=testpmd1 \
- --vdev=net_virtio_user0,mac=00:11:22:33:44:11,path=/tmp/vhost-net0,queues=1,mrg_rxbuf=0,in_order=1,vectorized=1 -- -i --rxq=1 --txq=1 --txd=1024 --rxd=1024 --nb-cores=1
+ ./x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 31-32 -n 4 --no-pci --file-prefix=testpmd1 \
+ --vdev=net_virtio_user0,mac=00:11:22:33:44:11,path=/tmp/vhost-net1,queues=1,server=1,mrg_rxbuf=1,in_order=1,vectorized=1 -- -i --rxq=1 --txq=1 --txd=1024 --rxd=1024 --nb-cores=1
-4. Start pkts from two virtio-user sides, record performance number with txpkts=256 and 2000 from testpmd1 seperately::
+4. Loop pkts between two virtio-user sides, record performance number with 64b/2000b/8000b/IMIX pkts can get expected::
testpmd0>set fwd mac
testpmd0>start tx_first
testpmd0>stop
testpmd0>set eth-peer 0 00:11:22:33:44:11
testpmd0>start
-
testpmd1>set fwd mac
testpmd1>set eth-peer 0 00:11:22:33:44:10
+ testpmd1>set txpkts 64
+ testpmd1>start tx_first
+ testpmd1>show port stats all
+ testpmd1>stop
testpmd1>set txpkts 2000
testpmd1>start tx_first
testpmd1>show port stats all
testpmd1>stop
- testpmd1>set txpkts 256
+ testpmd1>set txpkts 2000,2000,2000,2000
+ testpmd1>start tx_first
+ testpmd1>show port stats all
+ testpmd1>stop
+ testpmd1>set txpkts 64,256,2000,64,256,2000
testpmd1>start tx_first
testpmd1>show port stats all
-5. Re-test step 2-4 except to launch dpdk-vhost by below command, record perfromance::
-
- ./dpdk-vhost -l 27-28 -n 4 -- -p 0x1 --mergeable 1 --vm2vm 1 --socket-file /tmp/vhost-net0 --socket-file /tmp/vhost-net1
+5. Stop dpdk-vhost side and relaunch it with same cmd as step2.
-6. Compare perfromance number, when txpkts=2000, first test has better performance; while txpkts=256, the second test has better performance.
+6. Rerun step 4.
-Test Case4: VM2VM test with 2 vhost device using vhost async driver
-=======================================================================
+Test Case4: VM2VM test with cbdma channels register/unregister stable check
+============================================================================
1. Bind one physical ports to vfio-pci and two CBDMA channels to igb_uio.
2. On host, launch dpdk-vhost by below command::
- ./dpdk-vhost -l 26-28 -n 4 -- -p 0x1 --mergeable 1 --vm2vm 1 --dma-type ioat \
- --socket-file /tmp/vhost-net0 --socket-file /tmp/vhost-net1 --dmas [txd0@00:04.0,txd1@00:04.1]
+ ./dpdk-vhost -l 26-28 -n 4 -- -p 0x1 --mergeable 1 --vm2vm 1 --dma-type ioat \
+ --socket-file /tmp/vhost-net0 --socket-file /tmp/vhost-net1 --dmas [txd0@0000:00:04.0,txd1@0000:00:04.1] --client
-3. Start VM0::
+3. Start VM0 with qemu-5.2.0::
- /home/qemu-install/qemu-4.2.1/bin/qemu-system-x86_64 -name vm1 -enable-kvm -cpu host -smp 4 -m 4096 \
+ qemu-system-x86_64 -name vm1 -enable-kvm -cpu host -smp 4 -m 4096 \
-object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \
-numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu20-04.img \
-chardev socket,path=/tmp/vm2_qga0.sock,server,nowait,id=vm2_qga0 -device virtio-serial \
-device virtserialport,chardev=vm2_qga0,name=org.qemu.guest_agent.2 -daemonize \
-monitor unix:/tmp/vm2_monitor.sock,server,nowait -device e1000,netdev=nttsip1 \
-netdev user,id=nttsip1,hostfwd=tcp:127.0.0.1:6002-:22 \
- -chardev socket,id=char0,path=/tmp/vhost-net0 \
+ -chardev socket,id=char0,path=/tmp/vhost-net0,server \
-netdev type=vhost-user,id=netdev0,chardev=char0,vhostforce \
- -device virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=true,mrg_rxbuf=off,csum=on,guest_csum=on,host_tso4=on,guest_tso4=on,guest_ecn=on -vnc :10
+ -device virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=true,mrg_rxbuf=on,csum=on,guest_csum=on,host_tso4=on,guest_tso4=on,guest_ecn=on -vnc :10
-4. Start VM1::
+4. Start VM1 with qemu-5.2.0::
- /home/qemu-install/qemu-4.2.1/bin/qemu-system-x86_64 -name vm2 -enable-kvm -cpu host -smp 4 -m 4096 \
+ qemu-system-x86_64 -name vm2 -enable-kvm -cpu host -smp 4 -m 4096 \
-object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \
-numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu20-04-2.img \
-chardev socket,path=/tmp/vm2_qga0.sock,server,nowait,id=vm2_qga0 -device virtio-serial \
-device virtserialport,chardev=vm2_qga0,name=org.qemu.guest_agent.2 -daemonize \
-monitor unix:/tmp/vm2_monitor.sock,server,nowait -device e1000,netdev=nttsip1 \
- -netdev user,id=nttsip1,hostfwd=tcp:10.67.119.61:6003-:22 \
- -chardev socket,id=char0,path=/tmp/vhost-net1 \
+ -netdev user,id=nttsip1,hostfwd=tcp:127.0.0.1:6003-:22 \
+ -chardev socket,id=char0,path=/tmp/vhost-net1,server \
-netdev type=vhost-user,id=netdev0,chardev=char0,vhostforce \
- -device virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:02,disable-modern=true,mrg_rxbuf=off,csum=on,guest_csum=on,host_tso4=on,guest_tso4=on,guest_ecn=on -vnc :12
+ -device virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:02,disable-modern=true,mrg_rxbuf=on,csum=on,guest_csum=on,host_tso4=on,guest_tso4=on,guest_ecn=on,packed=on -vnc :12
5. Bind virtio port to vfio-pci in both two VMs::
@@ -219,30 +221,153 @@ Test Case4: VM2VM test with 2 vhost device using vhost async driver
./dpdk-testpmd -l 1-2 -n 4 -- -i --rxq=1 --txq=1 --nb-cores=1 --txd=1024 --rxd=1024
-7. Start pkts from two virtio-pmd, record performance number with txpkts=256 and 2000 from testpmd1 seperately::
+7. Loop pkts between two virtio-user sides, record performance number with 64b/2000b/8000b/IMIX pkts can get expected::
testpmd0>set fwd mac
testpmd0>start tx_first
testpmd0>stop
testpmd0>set eth-peer 0 52:54:00:00:00:02
testpmd0>start
-
testpmd1>set fwd mac
testpmd1>set eth-peer 0 52:54:00:00:00:01
+ testpmd1>set txpkts 64
+ testpmd1>start tx_first
+ testpmd1>show port stats all
+ testpmd1>stop
testpmd1>set txpkts 2000
testpmd1>start tx_first
testpmd1>show port stats all
testpmd1>stop
- testpmd1>set txpkts 256
+ testpmd1>set txpkts 2000,2000,2000,2000
+ testpmd1>start tx_first
+ testpmd1>show port stats all
+ testpmd1>stop
+ testpmd1>set txpkts 64,256,2000,64,256,2000
testpmd1>start tx_first
testpmd1>show port stats all
-8. Inject traffic with VLAN_id=1000 and dest_mac=virtio_mac_addresss (52:54:00:00:00:02 and 52:54:00:00:00:02) to NIC using packet generator, check two virtio-pmd can receive according packets.
-
-9. Quit two testpmd in two VMs, bind virtio-pmd port to virtio-pci,then bind port back to vfio-pci, rerun below cmd 50 times::
+8. Quit two testpmd in two VMs, bind virtio-pmd port to virtio-pci,then bind port back to vfio-pci, rerun below cmd 50 times::
./usertools/dpdk-devbind.py -u 00:05.0
./usertools/dpdk-devbind.py --bind=virtio-pci 00:05.0
./usertools/dpdk-devbind.py --bind=vfio-pci 00:05.0
-10. Rerun step 6-8，check vhost can stable work.
+9. Restart vhost, then rerun step 7，check vhost can stable work and get expected throughput.
+
+Test Case5: VM2VM split ring test with iperf and reconnect stable check
+=======================================================================
+
+1. Bind one physical ports to vfio-pci and two CBDMA channels to igb_uio.
+
+2. On host, launch dpdk-vhost by below command::
+
+ ./dpdk-vhost -l 26-28 -n 4 -- -p 0x1 --mergeable 1 --vm2vm 1 --dma-type ioat \
+ --socket-file /tmp/vhost-net0 --socket-file /tmp/vhost-net1 --dmas [txd0@0000:00:04.0,txd1@0000:00:04.1] --client
+
+3. Start VM0 with qemu-5.2.0::
+
+ qemu-system-x86_64 -name vm1 -enable-kvm -cpu host -smp 4 -m 4096 \
+ -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \
+ -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu20-04.img \
+ -chardev socket,path=/tmp/vm2_qga0.sock,server,nowait,id=vm2_qga0 -device virtio-serial \
+ -device virtserialport,chardev=vm2_qga0,name=org.qemu.guest_agent.2 -daemonize \
+ -monitor unix:/tmp/vm2_monitor.sock,server,nowait -device e1000,netdev=nttsip1 \
+ -netdev user,id=nttsip1,hostfwd=tcp:127.0.0.1:6002-:22 \
+ -chardev socket,id=char0,path=/tmp/vhost-net0,server \
+ -netdev type=vhost-user,id=netdev0,chardev=char0,vhostforce \
+ -device virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=true,mrg_rxbuf=off,csum=on,guest_csum=on,host_tso4=on,guest_tso4=on,guest_ecn=on -vnc :10
+
+4. Start VM1 with qemu-5.2.0::
+
+ qemu-system-x86_64 -name vm2 -enable-kvm -cpu host -smp 4 -m 4096 \
+ -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \
+ -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu20-04-2.img \
+ -chardev socket,path=/tmp/vm2_qga0.sock,server,nowait,id=vm2_qga0 -device virtio-serial \
+ -device virtserialport,chardev=vm2_qga0,name=org.qemu.guest_agent.2 -daemonize \
+ -monitor unix:/tmp/vm2_monitor.sock,server,nowait -device e1000,netdev=nttsip1 \
+ -netdev user,id=nttsip1,hostfwd=tcp:127.0.0.1:6003-:22 \
+ -chardev socket,id=char0,path=/tmp/vhost-net1,server \
+ -netdev type=vhost-user,id=netdev0,chardev=char0,vhostforce \
+ -device virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:02,disable-modern=true,mrg_rxbuf=off,csum=on,guest_csum=on,host_tso4=on,guest_tso4=on,guest_ecn=on -vnc :12
+
+5. On VM1, set virtio device IP and run arp protocal::
+
+ ifconfig ens5 1.1.1.2
+ arp -s 1.1.1.8 52:54:00:00:00:02
+
+6. On VM2, set virtio device IP and run arp protocal::
+
+ ifconfig ens5 1.1.1.8
+ arp -s 1.1.1.2 52:54:00:00:00:01
+
+7. Check the iperf performance between two VMs by below commands::
+
+ Under VM1, run: `iperf -s -i 1`
+ Under VM2, run: `iperf -c 1.1.1.2 -i 1 -t 60`
+
+8. Check iperf throughput can get x Gbits/sec.
+
+9. Scp 1MB file form VM1 to VM2M, check packets can be forwarding success by scp::
+
+ Under VM1, run: `scp [xxx] root@1.1.1.8:/` [xxx] is the file name
+
+10. Relaunch vhost-dpdk, then rerun step 7-9 five times.
+
+Test Case6: VM2VM packed ring test with iperf and reconnect stable test
+=======================================================================
+
+1. Bind one physical ports to vfio-pci and two CBDMA channels to igb_uio.
+
+2. On host, launch dpdk-vhost by below command::
+
+ ./dpdk-vhost -l 26-28 -n 4 -- -p 0x1 --mergeable 1 --vm2vm 1 --dma-type ioat \
+ --socket-file /tmp/vhost-net0 --socket-file /tmp/vhost-net1 --dmas [txd0@0000:00:04.0,txd1@0000:00:04.1]
+
+3. Start VM0 with qemu-5.2.0::
+
+ qemu-system-x86_64 -name vm1 -enable-kvm -cpu host -smp 4 -m 4096 \
+ -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \
+ -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu20-04.img \
+ -chardev socket,path=/tmp/vm2_qga0.sock,server,nowait,id=vm2_qga0 -device virtio-serial \
+ -device virtserialport,chardev=vm2_qga0,name=org.qemu.guest_agent.2 -daemonize \
+ -monitor unix:/tmp/vm2_monitor.sock,server,nowait -device e1000,netdev=nttsip1 \
+ -netdev user,id=nttsip1,hostfwd=tcp:127.0.0.1:6002-:22 \
+ -chardev socket,id=char0,path=/tmp/vhost-net0 \
+ -netdev type=vhost-user,id=netdev0,chardev=char0,vhostforce \
+ -device virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:01,disable-modern=true,mrg_rxbuf=off,csum=on,guest_csum=on,host_tso4=on,guest_tso4=on,guest_ecn=on,packed=on -vnc :10
+
+4. Start VM1 with qemu-5.2.0::
+
+ qemu-system-x86_64 -name vm2 -enable-kvm -cpu host -smp 4 -m 4096 \
+ -object memory-backend-file,id=mem,size=4096M,mem-path=/mnt/huge,share=on \
+ -numa node,memdev=mem -mem-prealloc -drive file=/home/osimg/ubuntu20-04-2.img \
+ -chardev socket,path=/tmp/vm2_qga0.sock,server,nowait,id=vm2_qga0 -device virtio-serial \
+ -device virtserialport,chardev=vm2_qga0,name=org.qemu.guest_agent.2 -daemonize \
+ -monitor unix:/tmp/vm2_monitor.sock,server,nowait -device e1000,netdev=nttsip1 \
+ -netdev user,id=nttsip1,hostfwd=tcp:127.0.0.1:6003-:22 \
+ -chardev socket,id=char0,path=/tmp/vhost-net1 \
+ -netdev type=vhost-user,id=netdev0,chardev=char0,vhostforce \
+ -device virtio-net-pci,netdev=netdev0,mac=52:54:00:00:00:02,disable-modern=true,mrg_rxbuf=off,csum=on,guest_csum=on,host_tso4=on,guest_tso4=on,guest_ecn=on,packed=on -vnc :12
+
+5. On VM1, set virtio device IP and run arp protocal::
+
+ ifconfig ens5 1.1.1.2
+ arp -s 1.1.1.8 52:54:00:00:00:02
+
+6. On VM2, set virtio device IP and run arp protocal::
+
+ ifconfig ens5 1.1.1.8
+ arp -s 1.1.1.2 52:54:00:00:00:01
+
+7. Check the iperf performance between two VMs by below commands::
+
+ Under VM1, run: `iperf -s -i 1`
+ Under VM2, run: `iperf -c 1.1.1.2 -i 1 -t 60`
+
+8. Check iperf throughput can get x Gbits/sec.
+
+9. Scp 1MB file form VM1 to VM2M, check packets can be forwarding success by scp::
+
+ Under VM1, run: `scp [xxx] root@1.1.1.8:/` [xxx] is the file name
+
+10. Rerun step 7-9 five times.
\ No newline at end of file