[dpdk-dev] doc: introduce PVP reference benchmark
Checks
Commit Message
Having reference benchmarks is important in order to obtain
reproducible performance figures.
This patch describes required steps to configure a PVP setup
using testpmd in both host and guest.
Not relying on external vSwitch ease integration in a CI loop by
not being impacted by DPDK API changes.
Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
doc/guides/howto/img/pvp_2nics.svg | 556 +++++++++++++++++++++++++++
doc/guides/howto/index.rst | 1 +
doc/guides/howto/pvp_reference_benchmark.rst | 389 +++++++++++++++++++
3 files changed, 946 insertions(+)
create mode 100644 doc/guides/howto/img/pvp_2nics.svg
create mode 100644 doc/guides/howto/pvp_reference_benchmark.rst
Comments
First of all, thanks for the doc! It's a great one.
On Wed, Nov 23, 2016 at 10:00:06PM +0100, Maxime Coquelin wrote:
> +Qemu build
> +~~~~~~~~~~
> +
> + .. code-block:: console
> +
> + git clone git://dpdk.org/dpdk
> + cd dpdk
> + export RTE_SDK=$PWD
> + make install T=x86_64-native-linuxapp-gcc DESTDIR=install
It's actually DPDK build.
I will take a closer look at it and also render it to see how it looks
like when I get back to office next week.
--yliu
> +
> +DPDK build
> +~~~~~~~~~~
> +
> + .. code-block:: console
> +
> + git clone git://dpdk.org/dpdk
> + cd dpdk
> + export RTE_SDK=$PWD
> + make install T=x86_64-native-linuxapp-gcc DESTDIR=install
> +
On 11/24/2016 06:07 AM, Yuanhan Liu wrote:
> First of all, thanks for the doc! It's a great one.
Thanks.
I would be interested to know if you have other tuning I don't mention
in this doc.
>
> On Wed, Nov 23, 2016 at 10:00:06PM +0100, Maxime Coquelin wrote:
>> +Qemu build
>> +~~~~~~~~~~
>> +
>> + .. code-block:: console
>> +
>> + git clone git://dpdk.org/dpdk
>> + cd dpdk
>> + export RTE_SDK=$PWD
>> + make install T=x86_64-native-linuxapp-gcc DESTDIR=install
>
> It's actually DPDK build.
>
Oh right! Copy/paste mistake...
This is the Qemu build block:
Qemu build
~~~~~~~~~~
.. code-block:: console
git clone git://git.qemu.org/qemu.git
cd qemu
mkdir bin
cd bin
../configure --target-list=x86_64-softmmu
make
> I will take a closer look at it and also render it to see how it looks
> like when I get back to office next week.
>
> --yliu
>> +
>> +DPDK build
>> +~~~~~~~~~~
>> +
>> + .. code-block:: console
>> +
>> + git clone git://dpdk.org/dpdk
>> + cd dpdk
>> + export RTE_SDK=$PWD
>> + make install T=x86_64-native-linuxapp-gcc DESTDIR=install
>> +
On 11/23/2016 09:00 PM, Maxime Coquelin wrote:
> Having reference benchmarks is important in order to obtain
> reproducible performance figures.
>
> This patch describes required steps to configure a PVP setup
> using testpmd in both host and guest.
>
> Not relying on external vSwitch ease integration in a CI loop by
> not being impacted by DPDK API changes.
>
> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
A short template/hint of the main things to report after running could
be useful to help ML discussions about results e.g.
Traffic Generator: IXIA
Acceptable Loss: 100% (i.e. raw throughput test)
DPDK version/commit: v16.11
QEMU version/commit: v2.7.0
Patches applied: <link to patchwork>
CPU: E5-2680 v3, 2.8GHz
Result: x mpps
NIC: ixgbe 82599
> ---
> doc/guides/howto/img/pvp_2nics.svg | 556 +++++++++++++++++++++++++++
> doc/guides/howto/index.rst | 1 +
> doc/guides/howto/pvp_reference_benchmark.rst | 389 +++++++++++++++++++
> 3 files changed, 946 insertions(+)
> create mode 100644 doc/guides/howto/img/pvp_2nics.svg
> create mode 100644 doc/guides/howto/pvp_reference_benchmark.rst
>
<snip>
> +Host tuning
> +~~~~~~~~~~~
I would add turbo boost =disabled on BIOS.
> +
> +#. Append these options to Kernel command line:
> +
> + .. code-block:: console
> +
> + intel_pstate=disable mce=ignore_ce default_hugepagesz=1G hugepagesz=1G hugepages=6 isolcpus=2-7 rcu_nocbs=2-7 nohz_full=2-7 iommu=pt intel_iommu=on
> +
> +#. Disable hyper-threads at runtime if necessary and BIOS not accessible:
> +
> + .. code-block:: console
> +
> + cat /sys/devices/system/cpu/cpu*[0-9]/topology/thread_siblings_list \
> + | sort | uniq \
> + | awk -F, '{system("echo 0 > /sys/devices/system/cpu/cpu"$2"/online")}'
> +
> +#. Disable NMIs:
> +
> + .. code-block:: console
> +
> + echo 0 > /proc/sys/kernel/nmi_watchdog
> +
> +#. Exclude isolated CPUs from the writeback cpumask:
> +
> + .. code-block:: console
> +
> + echo ffffff03 > /sys/bus/workqueue/devices/writeback/cpumask
> +
> +#. Isolate CPUs from IRQs:
> +
> + .. code-block:: console
> +
> + clear_mask=0xfc #Isolate CPU2 to CPU7 from IRQs
> + for i in /proc/irq/*/smp_affinity
> + do
> + echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i
> + done
> +
> +Qemu build
> +~~~~~~~~~~
> +
> + .. code-block:: console
> +
> + git clone git://dpdk.org/dpdk
> + cd dpdk
> + export RTE_SDK=$PWD
> + make install T=x86_64-native-linuxapp-gcc DESTDIR=install
> +
> +DPDK build
> +~~~~~~~~~~
> +
> + .. code-block:: console
> +
> + git clone git://dpdk.org/dpdk
> + cd dpdk
> + export RTE_SDK=$PWD
> + make install T=x86_64-native-linuxapp-gcc DESTDIR=install
> +
> +Testpmd launch
> +~~~~~~~~~~~~~~
> +
> +#. Assign NICs to DPDK:
> +
> + .. code-block:: console
> +
> + modprobe vfio-pci
> + $RTE_SDK/install/sbin/dpdk-devbind -b vfio-pci 0000:11:00.0 0000:11:00.1
> +
> +*Note: Sandy Bridge family seems to have some limitations wrt its IOMMU,
> +giving poor performance results. To achieve good performance on these machines,
> +consider using UIO instead.*
> +
> +#. Launch testpmd application:
> +
> + .. code-block:: console
> +
> + $RTE_SDK/install/bin/testpmd -l 0,2,3,4,5 --socket-mem=1024 -n 4 \
> + --vdev 'net_vhost0,iface=/tmp/vhost-user1' \
> + --vdev 'net_vhost1,iface=/tmp/vhost-user2' -- \
> + --portmask=f --disable-hw-vlan -i --rxq=1 --txq=1
> + --nb-cores=4 --forward-mode=io
> +
> +#. In testpmd interactive mode, set the portlist to obtin the right chaining:
> +
> + .. code-block:: console
> +
> + set portlist 0,2,1,3
> + start
> +
> +VM launch
> +~~~~~~~~~
> +
> +The VM may be launched ezither by calling directly QEMU, or by using libvirt.
s/ezither/either
> +
> +#. Qemu way:
> +
> +Launch QEMU with two Virtio-net devices paired to the vhost-user sockets created by testpmd:
> +
> + .. code-block:: console
> +
> + <QEMU path>/bin/x86_64-softmmu/qemu-system-x86_64 \
> + -enable-kvm -cpu host -m 3072 -smp 3 \
> + -chardev socket,id=char0,path=/tmp/vhost-user1 \
> + -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
> + -device virtio-net-pci,netdev=mynet1,mac=52:54:00:02:d9:01,addr=0x10 \
> + -chardev socket,id=char1,path=/tmp/vhost-user2 \
> + -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce \
> + -device virtio-net-pci,netdev=mynet2,mac=52:54:00:02:d9:02,addr=0x11 \
> + -object memory-backend-file,id=mem,size=3072M,mem-path=/dev/hugepages,share=on \
> + -numa node,memdev=mem -mem-prealloc \
> + -net user,hostfwd=tcp::1002$1-:22 -net nic \
> + -qmp unix:/tmp/qmp.socket,server,nowait \
> + -monitor stdio <vm_image>.qcow2
Probably mergeable rx data path =off would want to be tested also when
evaluating any performance improvements/regressions.
> +
> +You can use this qmp-vcpu-pin script to pin vCPUs:
> +
> + .. code-block:: python
> +
> + #!/usr/bin/python
> + # QEMU vCPU pinning tool
> + #
> + # Copyright (C) 2016 Red Hat Inc.
> + #
> + # Authors:
> + # Maxime Coquelin <maxime.coquelin@redhat.com>
> + #
> + # This work is licensed under the terms of the GNU GPL, version 2. See
> + # the COPYING file in the top-level directory
> + import argparse
> + import json
> + import os
> +
> + from subprocess import call
> + from qmp import QEMUMonitorProtocol
> +
> + pinned = []
> +
> + parser = argparse.ArgumentParser(description='Pin QEMU vCPUs to physical CPUs')
> + parser.add_argument('-s', '--server', type=str, required=True,
> + help='QMP server path or address:port')
> + parser.add_argument('cpu', type=int, nargs='+',
> + help='Physical CPUs IDs')
> + args = parser.parse_args()
> +
> + devnull = open(os.devnull, 'w')
> +
> + srv = QEMUMonitorProtocol(args.server)
> + srv.connect()
> +
> + for vcpu in srv.command('query-cpus'):
> + vcpuid = vcpu['CPU']
> + tid = vcpu['thread_id']
> + if tid in pinned:
> + print 'vCPU{}\'s tid {} already pinned, skipping'.format(vcpuid, tid)
> + continue
> +
> + cpuid = args.cpu[vcpuid % len(args.cpu)]
> + print 'Pin vCPU {} (tid {}) to physical CPU {}'.format(vcpuid, tid, cpuid)
> + try:
> + call(['taskset', '-pc', str(cpuid), str(tid)], stdout=devnull)
> + pinned.append(tid)
> + except OSError:
> + print 'Failed to pin vCPU{} to CPU{}'.format(vcpuid, cpuid)
> +
> +
> +That can be used this way, for example to pin 3 vCPUs to CPUs 1, 6 and 7:
I think it would be good to explicitly explain the link you've made on
core numbers in this case between isolcpus, the vCPU pinning above and
the core list in the testpmd cmd line later.
> +
> + .. code-block:: console
> +
> + export PYTHONPATH=$PYTHONPATH:<QEMU path>/scripts/qmp
> + ./qmp-vcpu-pin -s /tmp/qmp.socket 1 6 7
> +
> +#. Libvirt way:
> +
> +Some initial steps are required for libvirt to be able to connect to testpmd's
> +sockets.
> +
> +First, SELinux policy needs to be set to permissiven, as testpmd is run as root
> +(reboot required):
> +
> + .. code-block:: console
> +
> + cat /etc/selinux/config
> +
> + # This file controls the state of SELinux on the system.
> + # SELINUX= can take one of these three values:
> + # enforcing - SELinux security policy is enforced.
> + # permissive - SELinux prints warnings instead of enforcing.
> + # disabled - No SELinux policy is loaded.
> + SELINUX=permissive
> + # SELINUXTYPE= can take one of three two values:
> + # targeted - Targeted processes are protected,
> + # minimum - Modification of targeted policy. Only selected processes are protected.
> + # mls - Multi Level Security protection.
> + SELINUXTYPE=targeted
> +
> +
> +Also, Qemu needs to be run as root, which has to be specified in /etc/libvirt/qemu.conf:
> +
> + .. code-block:: console
> +
> + user = "root"
> +
> +Once the domain created, following snippset is an extract of most important
> +information (hugepages, vCPU pinning, Virtio PCI devices):
> +
> + .. code-block:: xml
> +
> + <domain type='kvm'>
> + <memory unit='KiB'>3145728</memory>
> + <currentMemory unit='KiB'>3145728</currentMemory>
> + <memoryBacking>
> + <hugepages>
> + <page size='1048576' unit='KiB' nodeset='0'/>
> + </hugepages>
> + <locked/>
> + </memoryBacking>
> + <vcpu placement='static'>3</vcpu>
> + <cputune>
> + <vcpupin vcpu='0' cpuset='1'/>
> + <vcpupin vcpu='1' cpuset='6'/>
> + <vcpupin vcpu='2' cpuset='7'/>
> + <emulatorpin cpuset='0'/>
> + </cputune>
> + <numatune>
> + <memory mode='strict' nodeset='0'/>
> + </numatune>
> + <os>
> + <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
> + <boot dev='hd'/>
> + </os>
> + <cpu mode='host-passthrough'>
> + <topology sockets='1' cores='3' threads='1'/>
> + <numa>
> + <cell id='0' cpus='0-2' memory='3145728' unit='KiB' memAccess='shared'/>
> + </numa>
> + </cpu>
> + <devices>
> + <interface type='vhostuser'>
> + <mac address='56:48:4f:53:54:01'/>
> + <source type='unix' path='/tmp/vhost-user1' mode='client'/>
> + <model type='virtio'/>
> + <driver name='vhost' rx_queue_size='256' />
> + <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
> + </interface>
> + <interface type='vhostuser'>
> + <mac address='56:48:4f:53:54:02'/>
> + <source type='unix' path='/tmp/vhost-user2' mode='client'/>
> + <model type='virtio'/>
> + <driver name='vhost' rx_queue_size='256' />
> + <address type='pci' domain='0x0000' bus='0x00' slot='0x11' function='0x0'/>
> + </interface>
> + </devices>
> + </domain>
> +
> +Guest setup
> +...........
> +
> +Guest tuning
> +~~~~~~~~~~~~
> +
> +#. Append these options to Kernel command line:
> +
> + .. code-block:: console
> +
> + default_hugepagesz=1G hugepagesz=1G hugepages=1 intel_iommu=on iommu=pt isolcpus=1,2 rcu_nocbs=1,2 nohz_full=1,2
> +
> +#. Disable NMIs:
> +
> + .. code-block:: console
> +
> + echo 0 > /proc/sys/kernel/nmi_watchdog
> +
> +#. Exclude isolated CPU1 and CPU2 from the writeback wq cpumask:
> +
> + .. code-block:: console
> +
> + echo 1 > /sys/bus/workqueue/devices/writeback/cpumask
> +
> +#. Isolate CPUs from IRQs:
> +
> + .. code-block:: console
> +
> + clear_mask=0x6 #Isolate CPU1 and CPU2 from IRQs
> + for i in /proc/irq/*/smp_affinity
> + do
> + echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i
> + done
> +
> +DPDK build
> +~~~~~~~~~~
> +
> + .. code-block:: console
> +
> + git clone git://dpdk.org/dpdk
> + cd dpdk
> + export RTE_SDK=$PWD
> + make install T=x86_64-native-linuxapp-gcc DESTDIR=install
> +
> +Testpmd launch
> +~~~~~~~~~~~~~~
> +
> +Probe vfio module without iommu:
> +
> + .. code-block:: console
> +
> + modprobe -r vfio_iommu_type1
> + modprobe -r vfio
> + modprobe vfio enable_unsafe_noiommu_mode=1
> + cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
> + modprobe vfio-pci
> +
> +Bind virtio-net devices to DPDK:
> +
> + .. code-block:: console
> +
> + $RTE_SDK/tools/dpdk-devbind.py -b vfio-pci 0000:00:10.0 0000:00:11.0
> +
> +Start testpmd:
> +
> + .. code-block:: console
> +
> + $RTE_SDK/install/bin/testpmd -l 0,1,2 --socket-mem 1024 -n 4 \
> + --proc-type auto --file-prefix pg -- \
> + --portmask=3 --forward-mode=macswap --port-topology=chained \
> + --disable-hw-vlan --disable-rss -i --rxq=1 --txq=1 \
> + --rxd=256 --txd=256 --nb-cores=2 --auto-start
>
On 11/24/2016 12:58 PM, Kevin Traynor wrote:
> On 11/23/2016 09:00 PM, Maxime Coquelin wrote:
>> Having reference benchmarks is important in order to obtain
>> reproducible performance figures.
>>
>> This patch describes required steps to configure a PVP setup
>> using testpmd in both host and guest.
>>
>> Not relying on external vSwitch ease integration in a CI loop by
>> not being impacted by DPDK API changes.
>>
>> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>
> A short template/hint of the main things to report after running could
> be useful to help ML discussions about results e.g.
>
> Traffic Generator: IXIA
> Acceptable Loss: 100% (i.e. raw throughput test)
> DPDK version/commit: v16.11
> QEMU version/commit: v2.7.0
> Patches applied: <link to patchwork>
> CPU: E5-2680 v3, 2.8GHz
> Result: x mpps
> NIC: ixgbe 82599
Good idea, I'll add a section in the end providing this template.
>
>> ---
>> doc/guides/howto/img/pvp_2nics.svg | 556 +++++++++++++++++++++++++++
>> doc/guides/howto/index.rst | 1 +
>> doc/guides/howto/pvp_reference_benchmark.rst | 389 +++++++++++++++++++
>> 3 files changed, 946 insertions(+)
>> create mode 100644 doc/guides/howto/img/pvp_2nics.svg
>> create mode 100644 doc/guides/howto/pvp_reference_benchmark.rst
>>
>
> <snip>
>
>> +Host tuning
>> +~~~~~~~~~~~
>
> I would add turbo boost =disabled on BIOS.
>
+1, will be in next revision.
>> +
>> +#. Append these options to Kernel command line:
>> +
>> + .. code-block:: console
>> +
>> + intel_pstate=disable mce=ignore_ce default_hugepagesz=1G hugepagesz=1G hugepages=6 isolcpus=2-7 rcu_nocbs=2-7 nohz_full=2-7 iommu=pt intel_iommu=on
>> +
>> +#. Disable hyper-threads at runtime if necessary and BIOS not accessible:
>> +
>> + .. code-block:: console
>> +
>> + cat /sys/devices/system/cpu/cpu*[0-9]/topology/thread_siblings_list \
>> + | sort | uniq \
>> + | awk -F, '{system("echo 0 > /sys/devices/system/cpu/cpu"$2"/online")}'
>> +
>> +#. Disable NMIs:
>> +
>> + .. code-block:: console
>> +
>> + echo 0 > /proc/sys/kernel/nmi_watchdog
>> +
>> +#. Exclude isolated CPUs from the writeback cpumask:
>> +
>> + .. code-block:: console
>> +
>> + echo ffffff03 > /sys/bus/workqueue/devices/writeback/cpumask
>> +
>> +#. Isolate CPUs from IRQs:
>> +
>> + .. code-block:: console
>> +
>> + clear_mask=0xfc #Isolate CPU2 to CPU7 from IRQs
>> + for i in /proc/irq/*/smp_affinity
>> + do
>> + echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i
>> + done
>> +
>> +Qemu build
>> +~~~~~~~~~~
>> +
>> + .. code-block:: console
>> +
>> + git clone git://dpdk.org/dpdk
>> + cd dpdk
>> + export RTE_SDK=$PWD
>> + make install T=x86_64-native-linuxapp-gcc DESTDIR=install
>> +
>> +DPDK build
>> +~~~~~~~~~~
>> +
>> + .. code-block:: console
>> +
>> + git clone git://dpdk.org/dpdk
>> + cd dpdk
>> + export RTE_SDK=$PWD
>> + make install T=x86_64-native-linuxapp-gcc DESTDIR=install
>> +
>> +Testpmd launch
>> +~~~~~~~~~~~~~~
>> +
>> +#. Assign NICs to DPDK:
>> +
>> + .. code-block:: console
>> +
>> + modprobe vfio-pci
>> + $RTE_SDK/install/sbin/dpdk-devbind -b vfio-pci 0000:11:00.0 0000:11:00.1
>> +
>> +*Note: Sandy Bridge family seems to have some limitations wrt its IOMMU,
>> +giving poor performance results. To achieve good performance on these machines,
>> +consider using UIO instead.*
>> +
>> +#. Launch testpmd application:
>> +
>> + .. code-block:: console
>> +
>> + $RTE_SDK/install/bin/testpmd -l 0,2,3,4,5 --socket-mem=1024 -n 4 \
>> + --vdev 'net_vhost0,iface=/tmp/vhost-user1' \
>> + --vdev 'net_vhost1,iface=/tmp/vhost-user2' -- \
>> + --portmask=f --disable-hw-vlan -i --rxq=1 --txq=1
>> + --nb-cores=4 --forward-mode=io
>> +
>> +#. In testpmd interactive mode, set the portlist to obtin the right chaining:
>> +
>> + .. code-block:: console
>> +
>> + set portlist 0,2,1,3
>> + start
>> +
>> +VM launch
>> +~~~~~~~~~
>> +
>> +The VM may be launched ezither by calling directly QEMU, or by using libvirt.
>
> s/ezither/either
>
>> +
>> +#. Qemu way:
>> +
>> +Launch QEMU with two Virtio-net devices paired to the vhost-user sockets created by testpmd:
>> +
>> + .. code-block:: console
>> +
>> + <QEMU path>/bin/x86_64-softmmu/qemu-system-x86_64 \
>> + -enable-kvm -cpu host -m 3072 -smp 3 \
>> + -chardev socket,id=char0,path=/tmp/vhost-user1 \
>> + -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
>> + -device virtio-net-pci,netdev=mynet1,mac=52:54:00:02:d9:01,addr=0x10 \
>> + -chardev socket,id=char1,path=/tmp/vhost-user2 \
>> + -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce \
>> + -device virtio-net-pci,netdev=mynet2,mac=52:54:00:02:d9:02,addr=0x11 \
>> + -object memory-backend-file,id=mem,size=3072M,mem-path=/dev/hugepages,share=on \
>> + -numa node,memdev=mem -mem-prealloc \
>> + -net user,hostfwd=tcp::1002$1-:22 -net nic \
>> + -qmp unix:/tmp/qmp.socket,server,nowait \
>> + -monitor stdio <vm_image>.qcow2
>
> Probably mergeable rx data path =off would want to be tested also when
> evaluating any performance improvements/regressions.
Right, I'll add few lines about this.
>
>> +
>> +You can use this qmp-vcpu-pin script to pin vCPUs:
>> +
>> + .. code-block:: python
>> +
>> + #!/usr/bin/python
>> + # QEMU vCPU pinning tool
>> + #
>> + # Copyright (C) 2016 Red Hat Inc.
>> + #
>> + # Authors:
>> + # Maxime Coquelin <maxime.coquelin@redhat.com>
>> + #
>> + # This work is licensed under the terms of the GNU GPL, version 2. See
>> + # the COPYING file in the top-level directory
>> + import argparse
>> + import json
>> + import os
>> +
>> + from subprocess import call
>> + from qmp import QEMUMonitorProtocol
>> +
>> + pinned = []
>> +
>> + parser = argparse.ArgumentParser(description='Pin QEMU vCPUs to physical CPUs')
>> + parser.add_argument('-s', '--server', type=str, required=True,
>> + help='QMP server path or address:port')
>> + parser.add_argument('cpu', type=int, nargs='+',
>> + help='Physical CPUs IDs')
>> + args = parser.parse_args()
>> +
>> + devnull = open(os.devnull, 'w')
>> +
>> + srv = QEMUMonitorProtocol(args.server)
>> + srv.connect()
>> +
>> + for vcpu in srv.command('query-cpus'):
>> + vcpuid = vcpu['CPU']
>> + tid = vcpu['thread_id']
>> + if tid in pinned:
>> + print 'vCPU{}\'s tid {} already pinned, skipping'.format(vcpuid, tid)
>> + continue
>> +
>> + cpuid = args.cpu[vcpuid % len(args.cpu)]
>> + print 'Pin vCPU {} (tid {}) to physical CPU {}'.format(vcpuid, tid, cpuid)
>> + try:
>> + call(['taskset', '-pc', str(cpuid), str(tid)], stdout=devnull)
>> + pinned.append(tid)
>> + except OSError:
>> + print 'Failed to pin vCPU{} to CPU{}'.format(vcpuid, cpuid)
>> +
>> +
>> +That can be used this way, for example to pin 3 vCPUs to CPUs 1, 6 and 7:
>
> I think it would be good to explicitly explain the link you've made on
> core numbers in this case between isolcpus, the vCPU pinning above and
> the core list in the testpmd cmd line later.
Yes.
So vCPU0 doesn't run testpmd lcores, so doesn't need to be pinned to an
isolated CPU. vCPU1&2 are used as lcores, so are pinned to isolated
CPUs.
I'll add this information to next version.
Thanks,
Maxime
> -----Original Message-----
> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
> Sent: Wednesday, November 23, 2016 9:00 PM
> To: yuanhan.liu@linux.intel.com; thomas.monjalon@6wind.com; Mcnamara, John
> <john.mcnamara@intel.com>; Yang, Zhiyong <zhiyong.yang@intel.com>;
> dev@dpdk.org
> Cc: fbaudin@redhat.com; Maxime Coquelin <maxime.coquelin@redhat.com>
> Subject: [PATCH] doc: introduce PVP reference benchmark
>
> Having reference benchmarks is important in order to obtain reproducible
> performance figures.
>
> This patch describes required steps to configure a PVP setup using testpmd
> in both host and guest.
>
> Not relying on external vSwitch ease integration in a CI loop by not being
> impacted by DPDK API changes.
Hi Maxime,
Thanks for the detailed doc and this initiative. Some minor documentation
comments below.
> +
> +Setup overview
> +..............
This level header should be ---------, even if it looks like dots in the
contribution guide:
http://dpdk.org/doc/guides/contributing/documentation.html#section-headers
> +
> +.. figure:: img/pvp_2nics.svg
> +
> + PVP setup using 2 NICs
> +
The figure needs a target so it can be used with :numref:, like this:
.. _figure_pvp_2nics:
.. figure:: img/pvp_2nics.*
PVP setup using 2 NICs
> +DPDK build
> +~~~~~~~~~~
> +
Put a one line description at the start of each section, even if it is just: Build DPDK:
> +Testpmd launch
> +~~~~~~~~~~~~~~
> +
> +#. Assign NICs to DPDK:
> +
> + .. code-block:: console
> +
> + modprobe vfio-pci
> + $RTE_SDK/install/sbin/dpdk-devbind -b vfio-pci 0000:11:00.0
> + 0000:11:00.1
> +
> +*Note: Sandy Bridge family seems to have some limitations wrt its
> +IOMMU, giving poor performance results. To achieve good performance on
> +these machines, consider using UIO instead.*
This would be better as an RST note:
#. Assign NICs to DPDK:
.. code-block:: console
modprobe vfio-pci
$RTE_SDK/install/sbin/dpdk-devbind -b vfio-pci 0000:11:00.0 0000:11:00.1
.. Note::
The Sandy Bridge family seems to have some IOMMU limitations giving poor
performance results. To achieve good performance on these machines
consider using UIO instead.
> +First, SELinux policy needs to be set to permissiven, as testpmd is run
> +as root (reboot required):
s/permissiven/permissive/
There are a couple of trailing whitespace errors as well at build as well.
John
Hi John,
On 11/24/2016 06:38 PM, Mcnamara, John wrote:
>> -----Original Message-----
>> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
>> Sent: Wednesday, November 23, 2016 9:00 PM
>> To: yuanhan.liu@linux.intel.com; thomas.monjalon@6wind.com; Mcnamara, John
>> <john.mcnamara@intel.com>; Yang, Zhiyong <zhiyong.yang@intel.com>;
>> dev@dpdk.org
>> Cc: fbaudin@redhat.com; Maxime Coquelin <maxime.coquelin@redhat.com>
>> Subject: [PATCH] doc: introduce PVP reference benchmark
>>
>> Having reference benchmarks is important in order to obtain reproducible
>> performance figures.
>>
>> This patch describes required steps to configure a PVP setup using testpmd
>> in both host and guest.
>>
>> Not relying on external vSwitch ease integration in a CI loop by not being
>> impacted by DPDK API changes.
>
> Hi Maxime,
>
> Thanks for the detailed doc and this initiative. Some minor documentation
> comments below.
>
>
>
>> +
>> +Setup overview
>> +..............
>
> This level header should be ---------, even if it looks like dots in the
> contribution guide:
>
> http://dpdk.org/doc/guides/contributing/documentation.html#section-headers
>
>
>> +
>> +.. figure:: img/pvp_2nics.svg
>> +
>> + PVP setup using 2 NICs
>> +
>
> The figure needs a target so it can be used with :numref:, like this:
>
> .. _figure_pvp_2nics:
>
> .. figure:: img/pvp_2nics.*
>
> PVP setup using 2 NICs
>
>
>> +DPDK build
>> +~~~~~~~~~~
>> +
>
> Put a one line description at the start of each section, even if it is just: Build DPDK:
Ok.
>
>
>
>> +Testpmd launch
>> +~~~~~~~~~~~~~~
>> +
>> +#. Assign NICs to DPDK:
>> +
>> + .. code-block:: console
>> +
>> + modprobe vfio-pci
>> + $RTE_SDK/install/sbin/dpdk-devbind -b vfio-pci 0000:11:00.0
>> + 0000:11:00.1
>> +
>> +*Note: Sandy Bridge family seems to have some limitations wrt its
>> +IOMMU, giving poor performance results. To achieve good performance on
>> +these machines, consider using UIO instead.*
>
> This would be better as an RST note:
>
> #. Assign NICs to DPDK:
>
> .. code-block:: console
>
> modprobe vfio-pci
> $RTE_SDK/install/sbin/dpdk-devbind -b vfio-pci 0000:11:00.0 0000:11:00.1
>
> .. Note::
>
> The Sandy Bridge family seems to have some IOMMU limitations giving poor
> performance results. To achieve good performance on these machines
> consider using UIO instead.
This is indeed better, thanks for the tip!
About this note, I couldn't find official information about this
problem.
Do you confirm the issue, or I misconfigured something?
I'll also add something about security implications of using UIO.
>
>
>
>> +First, SELinux policy needs to be set to permissiven, as testpmd is run
>> +as root (reboot required):
>
> s/permissiven/permissive/
>
>
> There are a couple of trailing whitespace errors as well at build as well.
Ok, I will rework all this.
Thanks for the review,
Maxime
2016-11-23 22:00, Maxime Coquelin:
> +You can use this qmp-vcpu-pin script to pin vCPUs:
> +
> + .. code-block:: python
> +
> + #!/usr/bin/python
> + # QEMU vCPU pinning tool
> + #
> + # Copyright (C) 2016 Red Hat Inc.
> + #
> + # Authors:
> + # Maxime Coquelin <maxime.coquelin@redhat.com>
> + #
> + # This work is licensed under the terms of the GNU GPL, version 2. See
> + # the COPYING file in the top-level directory
> + import argparse
> + import json
> + import os
> +
> + from subprocess import call
> + from qmp import QEMUMonitorProtocol
> +
> + pinned = []
> +
> + parser = argparse.ArgumentParser(description='Pin QEMU vCPUs to physical CPUs')
> + parser.add_argument('-s', '--server', type=str, required=True,
> + help='QMP server path or address:port')
> + parser.add_argument('cpu', type=int, nargs='+',
> + help='Physical CPUs IDs')
> + args = parser.parse_args()
> +
> + devnull = open(os.devnull, 'w')
> +
> + srv = QEMUMonitorProtocol(args.server)
> + srv.connect()
> +
> + for vcpu in srv.command('query-cpus'):
> + vcpuid = vcpu['CPU']
> + tid = vcpu['thread_id']
> + if tid in pinned:
> + print 'vCPU{}\'s tid {} already pinned, skipping'.format(vcpuid, tid)
> + continue
> +
> + cpuid = args.cpu[vcpuid % len(args.cpu)]
> + print 'Pin vCPU {} (tid {}) to physical CPU {}'.format(vcpuid, tid, cpuid)
> + try:
> + call(['taskset', '-pc', str(cpuid), str(tid)], stdout=devnull)
> + pinned.append(tid)
> + except OSError:
> + print 'Failed to pin vCPU{} to CPU{}'.format(vcpuid, cpuid)
>
No please do not introduce such useful script in a doc.
I think it must be a separate file in the DPDK repository or
in the QEMU repository.
On 11/28/2016 12:22 PM, Thomas Monjalon wrote:
> 2016-11-23 22:00, Maxime Coquelin:
>> +You can use this qmp-vcpu-pin script to pin vCPUs:
>> +
>> + .. code-block:: python
>> +
>> + #!/usr/bin/python
>> + # QEMU vCPU pinning tool
>> + #
>> + # Copyright (C) 2016 Red Hat Inc.
>> + #
>> + # Authors:
>> + # Maxime Coquelin <maxime.coquelin@redhat.com>
>> + #
>> + # This work is licensed under the terms of the GNU GPL, version 2. See
>> + # the COPYING file in the top-level directory
>> + import argparse
>> + import json
>> + import os
>> +
>> + from subprocess import call
>> + from qmp import QEMUMonitorProtocol
>> +
>> + pinned = []
>> +
>> + parser = argparse.ArgumentParser(description='Pin QEMU vCPUs to physical CPUs')
>> + parser.add_argument('-s', '--server', type=str, required=True,
>> + help='QMP server path or address:port')
>> + parser.add_argument('cpu', type=int, nargs='+',
>> + help='Physical CPUs IDs')
>> + args = parser.parse_args()
>> +
>> + devnull = open(os.devnull, 'w')
>> +
>> + srv = QEMUMonitorProtocol(args.server)
>> + srv.connect()
>> +
>> + for vcpu in srv.command('query-cpus'):
>> + vcpuid = vcpu['CPU']
>> + tid = vcpu['thread_id']
>> + if tid in pinned:
>> + print 'vCPU{}\'s tid {} already pinned, skipping'.format(vcpuid, tid)
>> + continue
>> +
>> + cpuid = args.cpu[vcpuid % len(args.cpu)]
>> + print 'Pin vCPU {} (tid {}) to physical CPU {}'.format(vcpuid, tid, cpuid)
>> + try:
>> + call(['taskset', '-pc', str(cpuid), str(tid)], stdout=devnull)
>> + pinned.append(tid)
>> + except OSError:
>> + print 'Failed to pin vCPU{} to CPU{}'.format(vcpuid, cpuid)
>>
>
>
> No please do not introduce such useful script in a doc.
> I think it must be a separate file in the DPDK repository or
> in the QEMU repository.
Ok. The patch is under review on Qemu ML.
While it gets merged, I can add a link to its patchwork ID.
Ok for you?
Thanks,
Maxime
2016-11-28 15:02, Maxime Coquelin:
>
> On 11/28/2016 12:22 PM, Thomas Monjalon wrote:
> > 2016-11-23 22:00, Maxime Coquelin:
> >> +You can use this qmp-vcpu-pin script to pin vCPUs:
> >> +
> >> + .. code-block:: python
> >> +
> >> + #!/usr/bin/python
> >> + # QEMU vCPU pinning tool
> >> + #
> >> + # Copyright (C) 2016 Red Hat Inc.
> >> + #
> >> + # Authors:
> >> + # Maxime Coquelin <maxime.coquelin@redhat.com>
> >> + #
> >> + # This work is licensed under the terms of the GNU GPL, version 2. See
> >> + # the COPYING file in the top-level directory
> >> + import argparse
> >> + import json
> >> + import os
> >> +
> >> + from subprocess import call
> >> + from qmp import QEMUMonitorProtocol
> >> +
> >> + pinned = []
> >> +
> >> + parser = argparse.ArgumentParser(description='Pin QEMU vCPUs to physical CPUs')
> >> + parser.add_argument('-s', '--server', type=str, required=True,
> >> + help='QMP server path or address:port')
> >> + parser.add_argument('cpu', type=int, nargs='+',
> >> + help='Physical CPUs IDs')
> >> + args = parser.parse_args()
> >> +
> >> + devnull = open(os.devnull, 'w')
> >> +
> >> + srv = QEMUMonitorProtocol(args.server)
> >> + srv.connect()
> >> +
> >> + for vcpu in srv.command('query-cpus'):
> >> + vcpuid = vcpu['CPU']
> >> + tid = vcpu['thread_id']
> >> + if tid in pinned:
> >> + print 'vCPU{}\'s tid {} already pinned, skipping'.format(vcpuid, tid)
> >> + continue
> >> +
> >> + cpuid = args.cpu[vcpuid % len(args.cpu)]
> >> + print 'Pin vCPU {} (tid {}) to physical CPU {}'.format(vcpuid, tid, cpuid)
> >> + try:
> >> + call(['taskset', '-pc', str(cpuid), str(tid)], stdout=devnull)
> >> + pinned.append(tid)
> >> + except OSError:
> >> + print 'Failed to pin vCPU{} to CPU{}'.format(vcpuid, cpuid)
> >>
> >
> >
> > No please do not introduce such useful script in a doc.
> > I think it must be a separate file in the DPDK repository or
> > in the QEMU repository.
>
> Ok. The patch is under review on Qemu ML.
> While it gets merged, I can add a link to its patchwork ID.
> Ok for you?
Perfect, thanks
On Thu, Nov 24, 2016 at 08:35:51AM +0100, Maxime Coquelin wrote:
>
>
> On 11/24/2016 06:07 AM, Yuanhan Liu wrote:
> >First of all, thanks for the doc! It's a great one.
> Thanks.
> I would be interested to know if you have other tuning I don't mention
> in this doc.
I was thinking we may need doc some performance impacts by some features,
say we observed that indirect desc may be good for some cases, while may
be bad for others. Also, the non mergeable Rx path outweighs the mergeable
Rx path. If user cares about the perfomance and ascertains all packets
fits into a typical MTU, he may likely want to disable the mergeable
feature, which is enabled by default.
Maybe we could start a new doc, or maybe we could add a new section here?
--yliu
Hi Yuanhan,
On 11/29/2016 11:16 AM, Yuanhan Liu wrote:
> On Thu, Nov 24, 2016 at 08:35:51AM +0100, Maxime Coquelin wrote:
>>
>>
>> On 11/24/2016 06:07 AM, Yuanhan Liu wrote:
>>> First of all, thanks for the doc! It's a great one.
>> Thanks.
>> I would be interested to know if you have other tuning I don't mention
>> in this doc.
>
> I was thinking we may need doc some performance impacts by some features,
> say we observed that indirect desc may be good for some cases, while may
> be bad for others. Also, the non mergeable Rx path outweighs the mergeable
> Rx path. If user cares about the perfomance and ascertains all packets
> fits into a typical MTU, he may likely want to disable the mergeable
> feature, which is enabled by default.
>
> Maybe we could start a new doc, or maybe we could add a new section here?
I agree that we should documents impact of Virtio features on traffic
profile.
My opinion is that it deserves a dedicated document.
For this PVP doc, I suggest we add a section stating that one could try
with different Virtio features, and in Kevin's result template
proposal, we add a line for Virtio features enabled/disabled.
Thanks,
Maxime
>
> --yliu
>
new file mode 100644
@@ -0,0 +1,556 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+ xmlns:dc="http://purl.org/dc/elements/1.1/"
+ xmlns:cc="http://creativecommons.org/ns#"
+ xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+ xmlns:svg="http://www.w3.org/2000/svg"
+ xmlns="http://www.w3.org/2000/svg"
+ xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+ xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+ width="127.46428mm"
+ height="139.41411mm"
+ viewBox="0 0 451.64508 493.987"
+ id="svg2"
+ version="1.1"
+ inkscape:version="0.92pre2 r"
+ sodipodi:docname="pvp_2nics.svg"
+ inkscape:export-filename="/home/max/Pictures/dpdk/pvp/pvp.png"
+ inkscape:export-xdpi="90"
+ inkscape:export-ydpi="90">
+ <defs
+ id="defs4">
+ <marker
+ inkscape:stockid="Arrow1Lend"
+ orient="auto"
+ refY="0"
+ refX="0"
+ id="marker4760"
+ style="overflow:visible"
+ inkscape:isstock="true">
+ <path
+ inkscape:connector-curvature="0"
+ id="path4762"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+ transform="matrix(-0.8,0,0,-0.8,-10,0)" />
+ </marker>
+ <marker
+ inkscape:stockid="Arrow1Lend"
+ orient="auto"
+ refY="0"
+ refX="0"
+ id="marker4642"
+ style="overflow:visible"
+ inkscape:isstock="true">
+ <path
+ inkscape:connector-curvature="0"
+ id="path4644"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+ transform="matrix(-0.8,0,0,-0.8,-10,0)" />
+ </marker>
+ <marker
+ inkscape:stockid="Arrow1Lstart"
+ orient="auto"
+ refY="0"
+ refX="0"
+ id="marker10370"
+ style="overflow:visible"
+ inkscape:isstock="true">
+ <path
+ id="path10372"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+ transform="matrix(0.8,0,0,0.8,10,0)"
+ inkscape:connector-curvature="0" />
+ </marker>
+ <marker
+ inkscape:isstock="true"
+ style="overflow:visible"
+ id="marker10306"
+ refX="0"
+ refY="0"
+ orient="auto"
+ inkscape:stockid="Arrow1Lend">
+ <path
+ transform="matrix(-0.8,0,0,-0.8,-10,0)"
+ style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ id="path10308"
+ inkscape:connector-curvature="0" />
+ </marker>
+ <marker
+ inkscape:isstock="true"
+ style="overflow:visible"
+ id="marker9757"
+ refX="0"
+ refY="0"
+ orient="auto"
+ inkscape:stockid="Arrow1Lend"
+ inkscape:collect="always">
+ <path
+ transform="matrix(-0.8,0,0,-0.8,-10,0)"
+ style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ id="path9759"
+ inkscape:connector-curvature="0" />
+ </marker>
+ <marker
+ inkscape:stockid="Arrow1Lstart"
+ orient="auto"
+ refY="0"
+ refX="0"
+ id="Arrow1Lstart"
+ style="overflow:visible"
+ inkscape:isstock="true">
+ <path
+ id="path4224"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+ transform="matrix(0.8,0,0,0.8,10,0)"
+ inkscape:connector-curvature="0" />
+ </marker>
+ <marker
+ inkscape:stockid="Arrow1Lend"
+ orient="auto"
+ refY="0"
+ refX="0"
+ id="Arrow1Lend"
+ style="overflow:visible"
+ inkscape:isstock="true">
+ <path
+ id="path4227"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+ transform="matrix(-0.8,0,0,-0.8,-10,0)"
+ inkscape:connector-curvature="0" />
+ </marker>
+ <marker
+ inkscape:stockid="Arrow1Lend"
+ orient="auto"
+ refY="0"
+ refX="0"
+ id="Arrow1Lend-1"
+ style="overflow:visible"
+ inkscape:isstock="true">
+ <path
+ inkscape:connector-curvature="0"
+ id="path4227-27"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+ transform="matrix(-0.8,0,0,-0.8,-10,0)" />
+ </marker>
+ <marker
+ inkscape:stockid="Arrow1Lstart"
+ orient="auto"
+ refY="0"
+ refX="0"
+ id="Arrow1Lstart-9"
+ style="overflow:visible"
+ inkscape:isstock="true">
+ <path
+ inkscape:connector-curvature="0"
+ id="path4224-3"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+ transform="matrix(0.8,0,0,0.8,10,0)" />
+ </marker>
+ <marker
+ inkscape:isstock="true"
+ style="overflow:visible"
+ id="marker9757-0"
+ refX="0"
+ refY="0"
+ orient="auto"
+ inkscape:stockid="Arrow1Lend">
+ <path
+ transform="matrix(-0.8,0,0,-0.8,-10,0)"
+ style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ id="path9759-6"
+ inkscape:connector-curvature="0" />
+ </marker>
+ <marker
+ inkscape:stockid="Arrow1Lstart"
+ orient="auto"
+ refY="0"
+ refX="0"
+ id="Arrow1Lstart-6"
+ style="overflow:visible"
+ inkscape:isstock="true">
+ <path
+ inkscape:connector-curvature="0"
+ id="path4224-0"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+ transform="matrix(0.8,0,0,0.8,10,0)" />
+ </marker>
+ <marker
+ inkscape:stockid="Arrow1Lend"
+ orient="auto"
+ refY="0"
+ refX="0"
+ id="Arrow1Lend-62"
+ style="overflow:visible"
+ inkscape:isstock="true">
+ <path
+ inkscape:connector-curvature="0"
+ id="path4227-6"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+ transform="matrix(-0.8,0,0,-0.8,-10,0)" />
+ </marker>
+ <marker
+ inkscape:stockid="Arrow1Lstart"
+ orient="auto"
+ refY="0"
+ refX="0"
+ id="marker10370-7"
+ style="overflow:visible"
+ inkscape:isstock="true">
+ <path
+ inkscape:connector-curvature="0"
+ id="path10372-9"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+ transform="matrix(0.8,0,0,0.8,10,0)" />
+ </marker>
+ <marker
+ inkscape:isstock="true"
+ style="overflow:visible"
+ id="marker9757-2"
+ refX="0"
+ refY="0"
+ orient="auto"
+ inkscape:stockid="Arrow1Lend">
+ <path
+ transform="matrix(-0.8,0,0,-0.8,-10,0)"
+ style="fill:#ff0000;fill-opacity:1;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.00000003pt;stroke-opacity:1"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ id="path9759-0"
+ inkscape:connector-curvature="0" />
+ </marker>
+ <marker
+ inkscape:stockid="Arrow1Lstart"
+ orient="auto"
+ refY="0"
+ refX="0"
+ id="Arrow1Lstart-9-2"
+ style="overflow:visible"
+ inkscape:isstock="true">
+ <path
+ inkscape:connector-curvature="0"
+ id="path4224-3-3"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+ transform="matrix(0.8,0,0,0.8,10,0)" />
+ </marker>
+ <marker
+ inkscape:stockid="Arrow1Lend"
+ orient="auto"
+ refY="0"
+ refX="0"
+ id="Arrow1Lend-1-7"
+ style="overflow:visible"
+ inkscape:isstock="true">
+ <path
+ inkscape:connector-curvature="0"
+ id="path4227-27-5"
+ d="M 0,0 5,-5 -12.5,0 5,5 Z"
+ style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
+ transform="matrix(-0.8,0,0,-0.8,-10,0)" />
+ </marker>
+ </defs>
+ <sodipodi:namedview
+ id="base"
+ pagecolor="#ffffff"
+ bordercolor="#666666"
+ borderopacity="1.0"
+ inkscape:pageopacity="0.0"
+ inkscape:pageshadow="2"
+ inkscape:zoom="0.49497475"
+ inkscape:cx="206.7485"
+ inkscape:cy="227.93958"
+ inkscape:document-units="px"
+ inkscape:current-layer="layer1"
+ showgrid="false"
+ inkscape:object-nodes="true"
+ inkscape:window-width="1916"
+ inkscape:window-height="1040"
+ inkscape:window-x="0"
+ inkscape:window-y="38"
+ inkscape:window-maximized="0"
+ inkscape:snap-grids="true"
+ inkscape:snap-to-guides="true"
+ inkscape:snap-others="false"
+ inkscape:snap-nodes="false"
+ inkscape:snap-global="false" />
+ <metadata
+ id="metadata7">
+ <rdf:RDF>
+ <cc:Work
+ rdf:about="">
+ <dc:format>image/svg+xml</dc:format>
+ <dc:type
+ rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+ <dc:title></dc:title>
+ </cc:Work>
+ </rdf:RDF>
+ </metadata>
+ <g
+ inkscape:label="Layer 1"
+ inkscape:groupmode="layer"
+ id="layer1"
+ transform="translate(-5.3301459,-7.348317)">
+ <rect
+ style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.78969002;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+ id="rect4140"
+ width="434.38919"
+ height="75.295639"
+ x="21.691195"
+ y="404.59354" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ x="421.47873"
+ y="501.3353"
+ id="text4142"><tspan
+ sodipodi:role="line"
+ id="tspan4144"
+ x="421.47873"
+ y="501.3353"
+ style="font-size:25px;line-height:125%">TE</tspan></text>
+ <rect
+ style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+ id="rect4146"
+ width="92.934036"
+ height="32.324883"
+ x="182.57764"
+ y="372.03574" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ x="183.5878"
+ y="397.28958"
+ id="text4148"><tspan
+ sodipodi:role="line"
+ id="tspan4150"
+ x="183.5878"
+ y="397.28958"
+ style="font-size:25px">10G NIC</tspan></text>
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ x="166.92024"
+ y="451.33276"
+ id="text4152"><tspan
+ sodipodi:role="line"
+ id="tspan4154"
+ x="166.92024"
+ y="451.33276">Moongen</tspan></text>
+ <rect
+ style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.39882457;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+ id="rect4156"
+ width="449.73071"
+ height="244.32167"
+ x="6.0295582"
+ y="29.046324" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ x="405.31628"
+ y="25.048317"
+ id="text4158"><tspan
+ sodipodi:role="line"
+ id="tspan4160"
+ x="405.31628"
+ y="25.048317"
+ style="font-size:25px">DUT</tspan></text>
+ <rect
+ style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.14168489;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+ id="rect4162"
+ width="418.69415"
+ height="107.50462"
+ x="19.038134"
+ y="41.044758" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ x="395.38812"
+ y="66.496857"
+ id="text4164"><tspan
+ sodipodi:role="line"
+ id="tspan4166"
+ x="395.38812"
+ y="66.496857"
+ style="font-size:25px">VM</tspan></text>
+ <rect
+ style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+ id="rect4146-3"
+ width="92.934036"
+ height="32.324883"
+ x="183.0827"
+ y="274.05093" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ x="184.09286"
+ y="299.30475"
+ id="text4148-6"><tspan
+ sodipodi:role="line"
+ id="tspan4150-7"
+ x="184.09286"
+ y="299.30475"
+ style="font-size:25px">10G NIC</tspan></text>
+ <rect
+ style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:2.4804399;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+ id="rect4189"
+ width="398.00476"
+ height="65.451302"
+ x="26.901583"
+ y="82.647781" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:25px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ x="30.683046"
+ y="108.31288"
+ id="text4191"><tspan
+ sodipodi:role="line"
+ id="tspan4193"
+ x="30.683046"
+ y="108.31288">TestPMD</tspan><tspan
+ sodipodi:role="line"
+ x="30.683046"
+ y="139.56288"
+ id="tspan10476">(macswap)</tspan></text>
+ <rect
+ style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:2.49124122;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+ id="rect4189-5"
+ width="397.22263"
+ height="66.152573"
+ x="29.743357"
+ y="207.6543" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:25px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ x="42.720772"
+ y="231.14902"
+ id="text4191-3"><tspan
+ sodipodi:role="line"
+ id="tspan4193-5"
+ x="42.720772"
+ y="231.14902">TestPMD </tspan><tspan
+ sodipodi:role="line"
+ x="42.720772"
+ y="262.39902"
+ id="tspan9747">(io)</tspan></text>
+ <path
+ style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.97838062px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend)"
+ d="M 202.56669,371.44487 V 308.37034"
+ id="path4218"
+ inkscape:connector-curvature="0" />
+ <path
+ style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.97297633px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart)"
+ d="M 252.03098,369.63533 V 307.25568"
+ id="path4218-9"
+ inkscape:connector-curvature="0" />
+ <path
+ style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.92982113px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend-1)"
+ d="M 198.63811,207.44389 V 150.47507"
+ id="path4218-0"
+ inkscape:connector-curvature="0" />
+ <path
+ style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.95360273px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart-9)"
+ d="M 255.56859,206.9303 V 147.01008"
+ id="path4218-9-6"
+ inkscape:connector-curvature="0" />
+ <path
+ style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-opacity:1;marker-end:url(#marker9757)"
+ d="M 199.50513,271.00921 V 207.3696"
+ id="path9749"
+ inkscape:connector-curvature="0" />
+ <path
+ style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-opacity:1;marker-start:url(#marker10370)"
+ d="M 255.56859,270.56991 V 206.9303"
+ id="path9749-2"
+ inkscape:connector-curvature="0" />
+ <rect
+ style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+ id="rect4146-36"
+ width="92.934036"
+ height="32.324883"
+ x="304.05591"
+ y="372.52954" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ x="305.06607"
+ y="397.78339"
+ id="text4148-7"><tspan
+ sodipodi:role="line"
+ id="tspan4150-5"
+ x="305.06607"
+ y="397.78339"
+ style="font-size:25px">10G NIC</tspan></text>
+ <rect
+ style="fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.46599996;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1"
+ id="rect4146-3-3"
+ width="92.934036"
+ height="32.324883"
+ x="306.07623"
+ y="273.53461" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:40px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
+ x="307.0864"
+ y="298.78842"
+ id="text4148-6-5"><tspan
+ sodipodi:role="line"
+ id="tspan4150-7-6"
+ x="307.0864"
+ y="298.78842"
+ style="font-size:25px">10G NIC</tspan></text>
+ <path
+ style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.97838062px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend-62)"
+ d="M 323.7504,370.24835 V 307.17382"
+ id="path4218-1"
+ inkscape:connector-curvature="0" />
+ <path
+ style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.97297633px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart-6)"
+ d="M 373.21469,368.43881 V 306.05916"
+ id="path4218-9-8"
+ inkscape:connector-curvature="0" />
+ <path
+ style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.92982113px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend-1-7)"
+ d="M 324.93036,207.24894 V 150.28012"
+ id="path4218-0-9"
+ inkscape:connector-curvature="0" />
+ <path
+ style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.95360273px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart-9-2)"
+ d="M 381.86084,206.73535 V 146.81513"
+ id="path4218-9-6-2"
+ inkscape:connector-curvature="0" />
+ <path
+ style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-opacity:1;marker-end:url(#marker9757-2)"
+ d="M 325.79738,270.81426 V 207.17465"
+ id="path9749-28"
+ inkscape:connector-curvature="0" />
+ <path
+ style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:3, 1;stroke-dashoffset:0;stroke-opacity:1;marker-start:url(#marker10370-7)"
+ d="M 381.86084,270.37496 V 206.73535"
+ id="path9749-2-9"
+ inkscape:connector-curvature="0" />
+ <path
+ style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2, 1;stroke-dashoffset:0;stroke-opacity:1;marker-end:url(#marker4642)"
+ d="M 198.57143,148.79077 V 95.93363 h 182.85714 v 50"
+ id="path3748"
+ inkscape:connector-curvature="0" />
+ <path
+ style="fill:none;fill-rule:evenodd;stroke:#ff0000;stroke-width:1.01005316;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2.02010632, 1.01005316;stroke-dashoffset:0;stroke-opacity:1;marker-end:url(#marker4760)"
+ d="m 325.70774,148.78714 v -32.84999 h -70.7012 v 30.70761"
+ id="path4634"
+ inkscape:connector-curvature="0" />
+ </g>
+</svg>
@@ -38,3 +38,4 @@ HowTo Guides
lm_bond_virtio_sriov
lm_virtio_vhost_user
flow_bifurcation
+ pvp_reference_benchmark
new file mode 100644
@@ -0,0 +1,389 @@
+.. BSD LICENSE
+ Copyright(c) 2016 Red Hat, Inc. All rights reserved.
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in
+ the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+PVP reference benchmark setup using testpmd
+===========================================
+
+This guide lists the steps required to setup PVP benchmark using testpmd as a
+simple forwarder between NICs and Vhost interfaces. The goal of this setup is
+to have a reference PVP benchmark not using external vSwitches (OVS, VPP, ...),
+making easier to obtain reproducible results and easing continuous integration
+testing.
+
+The guide covers two way for launching the VM, either by calling directly QEMU
+command line, or by relying on libvirt. It has been tested working with DPDK
+v16.11, using RHEL7 for both host and guest.
+
+Setup overview
+..............
+
+.. figure:: img/pvp_2nics.svg
+
+ PVP setup using 2 NICs
+
+In this diagram, each red arrow represents one logical core. This use-case
+requires 6 dedicated logical cores. A setup consisting in doing forwarding
+with a single NIC is also possible, requiring 3 logical cores.
+
+Host setup
+..........
+
+In this setup, we isolate 6 cores (from CPU2 to CPU7) in a same NUMA node. Two
+cores are assigned to the VM vCPUs running testpmd, four are assigned to
+testpmd on host.
+
+Host tuning
+~~~~~~~~~~~
+
+#. Append these options to Kernel command line:
+
+ .. code-block:: console
+
+ intel_pstate=disable mce=ignore_ce default_hugepagesz=1G hugepagesz=1G hugepages=6 isolcpus=2-7 rcu_nocbs=2-7 nohz_full=2-7 iommu=pt intel_iommu=on
+
+#. Disable hyper-threads at runtime if necessary and BIOS not accessible:
+
+ .. code-block:: console
+
+ cat /sys/devices/system/cpu/cpu*[0-9]/topology/thread_siblings_list \
+ | sort | uniq \
+ | awk -F, '{system("echo 0 > /sys/devices/system/cpu/cpu"$2"/online")}'
+
+#. Disable NMIs:
+
+ .. code-block:: console
+
+ echo 0 > /proc/sys/kernel/nmi_watchdog
+
+#. Exclude isolated CPUs from the writeback cpumask:
+
+ .. code-block:: console
+
+ echo ffffff03 > /sys/bus/workqueue/devices/writeback/cpumask
+
+#. Isolate CPUs from IRQs:
+
+ .. code-block:: console
+
+ clear_mask=0xfc #Isolate CPU2 to CPU7 from IRQs
+ for i in /proc/irq/*/smp_affinity
+ do
+ echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i
+ done
+
+Qemu build
+~~~~~~~~~~
+
+ .. code-block:: console
+
+ git clone git://dpdk.org/dpdk
+ cd dpdk
+ export RTE_SDK=$PWD
+ make install T=x86_64-native-linuxapp-gcc DESTDIR=install
+
+DPDK build
+~~~~~~~~~~
+
+ .. code-block:: console
+
+ git clone git://dpdk.org/dpdk
+ cd dpdk
+ export RTE_SDK=$PWD
+ make install T=x86_64-native-linuxapp-gcc DESTDIR=install
+
+Testpmd launch
+~~~~~~~~~~~~~~
+
+#. Assign NICs to DPDK:
+
+ .. code-block:: console
+
+ modprobe vfio-pci
+ $RTE_SDK/install/sbin/dpdk-devbind -b vfio-pci 0000:11:00.0 0000:11:00.1
+
+*Note: Sandy Bridge family seems to have some limitations wrt its IOMMU,
+giving poor performance results. To achieve good performance on these machines,
+consider using UIO instead.*
+
+#. Launch testpmd application:
+
+ .. code-block:: console
+
+ $RTE_SDK/install/bin/testpmd -l 0,2,3,4,5 --socket-mem=1024 -n 4 \
+ --vdev 'net_vhost0,iface=/tmp/vhost-user1' \
+ --vdev 'net_vhost1,iface=/tmp/vhost-user2' -- \
+ --portmask=f --disable-hw-vlan -i --rxq=1 --txq=1
+ --nb-cores=4 --forward-mode=io
+
+#. In testpmd interactive mode, set the portlist to obtin the right chaining:
+
+ .. code-block:: console
+
+ set portlist 0,2,1,3
+ start
+
+VM launch
+~~~~~~~~~
+
+The VM may be launched ezither by calling directly QEMU, or by using libvirt.
+
+#. Qemu way:
+
+Launch QEMU with two Virtio-net devices paired to the vhost-user sockets created by testpmd:
+
+ .. code-block:: console
+
+ <QEMU path>/bin/x86_64-softmmu/qemu-system-x86_64 \
+ -enable-kvm -cpu host -m 3072 -smp 3 \
+ -chardev socket,id=char0,path=/tmp/vhost-user1 \
+ -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
+ -device virtio-net-pci,netdev=mynet1,mac=52:54:00:02:d9:01,addr=0x10 \
+ -chardev socket,id=char1,path=/tmp/vhost-user2 \
+ -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce \
+ -device virtio-net-pci,netdev=mynet2,mac=52:54:00:02:d9:02,addr=0x11 \
+ -object memory-backend-file,id=mem,size=3072M,mem-path=/dev/hugepages,share=on \
+ -numa node,memdev=mem -mem-prealloc \
+ -net user,hostfwd=tcp::1002$1-:22 -net nic \
+ -qmp unix:/tmp/qmp.socket,server,nowait \
+ -monitor stdio <vm_image>.qcow2
+
+You can use this qmp-vcpu-pin script to pin vCPUs:
+
+ .. code-block:: python
+
+ #!/usr/bin/python
+ # QEMU vCPU pinning tool
+ #
+ # Copyright (C) 2016 Red Hat Inc.
+ #
+ # Authors:
+ # Maxime Coquelin <maxime.coquelin@redhat.com>
+ #
+ # This work is licensed under the terms of the GNU GPL, version 2. See
+ # the COPYING file in the top-level directory
+ import argparse
+ import json
+ import os
+
+ from subprocess import call
+ from qmp import QEMUMonitorProtocol
+
+ pinned = []
+
+ parser = argparse.ArgumentParser(description='Pin QEMU vCPUs to physical CPUs')
+ parser.add_argument('-s', '--server', type=str, required=True,
+ help='QMP server path or address:port')
+ parser.add_argument('cpu', type=int, nargs='+',
+ help='Physical CPUs IDs')
+ args = parser.parse_args()
+
+ devnull = open(os.devnull, 'w')
+
+ srv = QEMUMonitorProtocol(args.server)
+ srv.connect()
+
+ for vcpu in srv.command('query-cpus'):
+ vcpuid = vcpu['CPU']
+ tid = vcpu['thread_id']
+ if tid in pinned:
+ print 'vCPU{}\'s tid {} already pinned, skipping'.format(vcpuid, tid)
+ continue
+
+ cpuid = args.cpu[vcpuid % len(args.cpu)]
+ print 'Pin vCPU {} (tid {}) to physical CPU {}'.format(vcpuid, tid, cpuid)
+ try:
+ call(['taskset', '-pc', str(cpuid), str(tid)], stdout=devnull)
+ pinned.append(tid)
+ except OSError:
+ print 'Failed to pin vCPU{} to CPU{}'.format(vcpuid, cpuid)
+
+
+That can be used this way, for example to pin 3 vCPUs to CPUs 1, 6 and 7:
+
+ .. code-block:: console
+
+ export PYTHONPATH=$PYTHONPATH:<QEMU path>/scripts/qmp
+ ./qmp-vcpu-pin -s /tmp/qmp.socket 1 6 7
+
+#. Libvirt way:
+
+Some initial steps are required for libvirt to be able to connect to testpmd's
+sockets.
+
+First, SELinux policy needs to be set to permissiven, as testpmd is run as root
+(reboot required):
+
+ .. code-block:: console
+
+ cat /etc/selinux/config
+
+ # This file controls the state of SELinux on the system.
+ # SELINUX= can take one of these three values:
+ # enforcing - SELinux security policy is enforced.
+ # permissive - SELinux prints warnings instead of enforcing.
+ # disabled - No SELinux policy is loaded.
+ SELINUX=permissive
+ # SELINUXTYPE= can take one of three two values:
+ # targeted - Targeted processes are protected,
+ # minimum - Modification of targeted policy. Only selected processes are protected.
+ # mls - Multi Level Security protection.
+ SELINUXTYPE=targeted
+
+
+Also, Qemu needs to be run as root, which has to be specified in /etc/libvirt/qemu.conf:
+
+ .. code-block:: console
+
+ user = "root"
+
+Once the domain created, following snippset is an extract of most important
+information (hugepages, vCPU pinning, Virtio PCI devices):
+
+ .. code-block:: xml
+
+ <domain type='kvm'>
+ <memory unit='KiB'>3145728</memory>
+ <currentMemory unit='KiB'>3145728</currentMemory>
+ <memoryBacking>
+ <hugepages>
+ <page size='1048576' unit='KiB' nodeset='0'/>
+ </hugepages>
+ <locked/>
+ </memoryBacking>
+ <vcpu placement='static'>3</vcpu>
+ <cputune>
+ <vcpupin vcpu='0' cpuset='1'/>
+ <vcpupin vcpu='1' cpuset='6'/>
+ <vcpupin vcpu='2' cpuset='7'/>
+ <emulatorpin cpuset='0'/>
+ </cputune>
+ <numatune>
+ <memory mode='strict' nodeset='0'/>
+ </numatune>
+ <os>
+ <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
+ <boot dev='hd'/>
+ </os>
+ <cpu mode='host-passthrough'>
+ <topology sockets='1' cores='3' threads='1'/>
+ <numa>
+ <cell id='0' cpus='0-2' memory='3145728' unit='KiB' memAccess='shared'/>
+ </numa>
+ </cpu>
+ <devices>
+ <interface type='vhostuser'>
+ <mac address='56:48:4f:53:54:01'/>
+ <source type='unix' path='/tmp/vhost-user1' mode='client'/>
+ <model type='virtio'/>
+ <driver name='vhost' rx_queue_size='256' />
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
+ </interface>
+ <interface type='vhostuser'>
+ <mac address='56:48:4f:53:54:02'/>
+ <source type='unix' path='/tmp/vhost-user2' mode='client'/>
+ <model type='virtio'/>
+ <driver name='vhost' rx_queue_size='256' />
+ <address type='pci' domain='0x0000' bus='0x00' slot='0x11' function='0x0'/>
+ </interface>
+ </devices>
+ </domain>
+
+Guest setup
+...........
+
+Guest tuning
+~~~~~~~~~~~~
+
+#. Append these options to Kernel command line:
+
+ .. code-block:: console
+
+ default_hugepagesz=1G hugepagesz=1G hugepages=1 intel_iommu=on iommu=pt isolcpus=1,2 rcu_nocbs=1,2 nohz_full=1,2
+
+#. Disable NMIs:
+
+ .. code-block:: console
+
+ echo 0 > /proc/sys/kernel/nmi_watchdog
+
+#. Exclude isolated CPU1 and CPU2 from the writeback wq cpumask:
+
+ .. code-block:: console
+
+ echo 1 > /sys/bus/workqueue/devices/writeback/cpumask
+
+#. Isolate CPUs from IRQs:
+
+ .. code-block:: console
+
+ clear_mask=0x6 #Isolate CPU1 and CPU2 from IRQs
+ for i in /proc/irq/*/smp_affinity
+ do
+ echo "obase=16;$(( 0x$(cat $i) & ~$clear_mask ))" | bc > $i
+ done
+
+DPDK build
+~~~~~~~~~~
+
+ .. code-block:: console
+
+ git clone git://dpdk.org/dpdk
+ cd dpdk
+ export RTE_SDK=$PWD
+ make install T=x86_64-native-linuxapp-gcc DESTDIR=install
+
+Testpmd launch
+~~~~~~~~~~~~~~
+
+Probe vfio module without iommu:
+
+ .. code-block:: console
+
+ modprobe -r vfio_iommu_type1
+ modprobe -r vfio
+ modprobe vfio enable_unsafe_noiommu_mode=1
+ cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
+ modprobe vfio-pci
+
+Bind virtio-net devices to DPDK:
+
+ .. code-block:: console
+
+ $RTE_SDK/tools/dpdk-devbind.py -b vfio-pci 0000:00:10.0 0000:00:11.0
+
+Start testpmd:
+
+ .. code-block:: console
+
+ $RTE_SDK/install/bin/testpmd -l 0,1,2 --socket-mem 1024 -n 4 \
+ --proc-type auto --file-prefix pg -- \
+ --portmask=3 --forward-mode=macswap --port-topology=chained \
+ --disable-hw-vlan --disable-rss -i --rxq=1 --txq=1 \
+ --rxd=256 --txd=256 --nb-cores=2 --auto-start