[v3,0/2] Enhance Async Enqueue for Small Packets

  Async enqueue offloads large copies to DMA devices, and small copies
are still performed by the CPU. However, it requires users to get
enqueue completed packets by rte_vhost_poll_enqueue_completed(), even
if they are completed by the CPU when rte_vhost_submit_enqueue_burst()
returns. This design incurs extra overheads of tracking completed
pktmbufs and function calls, thus degrading performance on small packets.

The first patch cleans up async enqueue code, and the second patch
enables rte_vhost_submit_enqueue_burst() to return completed packets.

Change log
==========
v3:
- fix incorrect ret value when DMA ring is full
- enhance description of API declaration and programmer guide
v2:
- fix typo
- rename API variables
- update programmer guide

Jiayu Hu (2):
  vhost: cleanup async enqueue
  vhost: enhance async enqueue for small packets

 doc/guides/prog_guide/vhost_lib.rst |   8 +-
 lib/librte_vhost/rte_vhost_async.h  |  32 +++--
 lib/librte_vhost/vhost.c            |  14 +-
 lib/librte_vhost/vhost.h            |   7 +-
 lib/librte_vhost/vhost_user.c       |   7 +-
 lib/librte_vhost/virtio_net.c       | 258 ++++++++++++++++++++----------------
 6 files changed, 185 insertions(+), 141 deletions(-)
  

Tested-by: Wang, Yinan <yinan.wang@intel.com>

> -----Original Message-----
> From: Hu, Jiayu <jiayu.hu@intel.com>
> Sent: 2020?12?25? 16:29
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; Xia, Chenbo <chenbo.xia@intel.com>;
> Jiang, Cheng1 <cheng1.jiang@intel.com>; Wang, Yinan
> <yinan.wang@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>
> Subject: [PATCH v3 0/2] Enhance Async Enqueue for Small Packets
> 
> Async enqueue offloads large copies to DMA devices, and small copies
> are still performed by the CPU. However, it requires users to get
> enqueue completed packets by rte_vhost_poll_enqueue_completed(), even
> if they are completed by the CPU when rte_vhost_submit_enqueue_burst()
> returns. This design incurs extra overheads of tracking completed
> pktmbufs and function calls, thus degrading performance on small packets.
> 
> The first patch cleans up async enqueue code, and the second patch
> enables rte_vhost_submit_enqueue_burst() to return completed packets.
> 
> Change log
> ==========
> v3:
> - fix incorrect ret value when DMA ring is full
> - enhance description of API declaration and programmer guide
> v2:
> - fix typo
> - rename API variables
> - update programmer guide
> 
> Jiayu Hu (2):
>   vhost: cleanup async enqueue
>   vhost: enhance async enqueue for small packets
> 
>  doc/guides/prog_guide/vhost_lib.rst |   8 +-
>  lib/librte_vhost/rte_vhost_async.h  |  32 +++--
>  lib/librte_vhost/vhost.c            |  14 +-
>  lib/librte_vhost/vhost.h            |   7 +-
>  lib/librte_vhost/vhost_user.c       |   7 +-
>  lib/librte_vhost/virtio_net.c       | 258 ++++++++++++++++++++----------------
>  6 files changed, 185 insertions(+), 141 deletions(-)
> 
> --
> 2.7.4
  

On 12/25/20 9:28 AM, Jiayu Hu wrote:
> Async enqueue offloads large copies to DMA devices, and small copies
> are still performed by the CPU. However, it requires users to get
> enqueue completed packets by rte_vhost_poll_enqueue_completed(), even
> if they are completed by the CPU when rte_vhost_submit_enqueue_burst()
> returns. This design incurs extra overheads of tracking completed
> pktmbufs and function calls, thus degrading performance on small packets.
> 
> The first patch cleans up async enqueue code, and the second patch
> enables rte_vhost_submit_enqueue_burst() to return completed packets.
> 
> Change log
> ==========
> v3:
> - fix incorrect ret value when DMA ring is full
> - enhance description of API declaration and programmer guide
> v2:
> - fix typo
> - rename API variables
> - update programmer guide
> 
> Jiayu Hu (2):
>   vhost: cleanup async enqueue
>   vhost: enhance async enqueue for small packets
> 
>  doc/guides/prog_guide/vhost_lib.rst |   8 +-
>  lib/librte_vhost/rte_vhost_async.h  |  32 +++--
>  lib/librte_vhost/vhost.c            |  14 +-
>  lib/librte_vhost/vhost.h            |   7 +-
>  lib/librte_vhost/vhost_user.c       |   7 +-
>  lib/librte_vhost/virtio_net.c       | 258 ++++++++++++++++++++----------------
>  6 files changed, 185 insertions(+), 141 deletions(-)
> 

CI reports build failure with your series, because API changes are not
done in the examples:

FAILED: examples/dpdk-vhost.p/vhost_main.c.o
cc -Iexamples/dpdk-vhost.p -Iexamples -I../examples -Iexamples/vhost
-I../examples/vhost -I. -I.. -Iconfig -I../config
-Ilib/librte_eal/include -I../lib/librte_eal/include
-Ilib/librte_eal/linux/include -I../lib/librte_eal/linux/include
-Ilib/librte_eal/x86/include -I../lib/librte_eal/x86/include
-Ilib/librte_eal/common -I../lib/librte_eal/common -Ilib/librte_eal
-I../lib/librte_eal -Ilib/librte_kvargs -I../lib/librte_kvargs
-Ilib/librte_metrics -I../lib/librte_metrics -Ilib/librte_telemetry
-I../lib/librte_telemetry -Ilib/librte_mempool -I../lib/librte_mempool
-Ilib/librte_ring -I../lib/librte_ring -Ilib/librte_net
-I../lib/librte_net -Ilib/librte_mbuf -I../lib/librte_mbuf
-Ilib/librte_ethdev -I../lib/librte_ethdev -Ilib/librte_meter
-I../lib/librte_meter -Ilib/librte_cmdline -I../lib/librte_cmdline
-Ilib/librte_vhost -I../lib/librte_vhost -Ilib/librte_cryptodev
-I../lib/librte_cryptodev -Ilib/librte_hash -I../lib/librte_hash
-Ilib/librte_rcu -I../lib/librte_rcu -Ilib/librte_pci
-I../lib/librte_pci -Idrivers/raw/ioat -I../drivers/raw/ioat
-Ilib/librte_rawdev -I../lib/librte_rawdev -Idrivers/bus/pci
-I../drivers/bus/pci -I../drivers/bus/pci/linux -Idrivers/bus/vdev
-I../drivers/bus/vdev -fdiagnostics-color=always -pipe
-D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O3 -include
rte_config.h -Wextra -Wcast-qual -Wdeprecated -Wformat
-Wformat-nonliteral -Wformat-security -Wmissing-declarations
-Wmissing-prototypes -Wnested-externs -Wold-style-definition
-Wpointer-arith -Wsign-compare -Wstrict-prototypes -Wundef
-Wwrite-strings -Wno-missing-field-initializers -D_GNU_SOURCE
-march=native -Wno-format-truncation -DALLOW_EXPERIMENTAL_API -MD -MQ
examples/dpdk-vhost.p/vhost_main.c.o -MF
examples/dpdk-vhost.p/vhost_main.c.o.d -o
examples/dpdk-vhost.p/vhost_main.c.o -c ../examples/vhost/main.c
../examples/vhost/main.c: In function 'virtio_xmit':
../examples/vhost/main.c:817:9: error: too few arguments to function
'rte_vhost_submit_enqueue_burst'
   ret = rte_vhost_submit_enqueue_burst(dst_vdev->vid, VIRTIO_RXQ,
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../examples/vhost/ioat.h:10:0,
                 from ../examples/vhost/main.c:28:
../lib/librte_vhost/rte_vhost_async.h:171:10: note: declared here
 uint16_t rte_vhost_submit_enqueue_burst(int vid, uint16_t queue_id,
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../examples/vhost/main.c: In function 'drain_eth_rx':
../examples/vhost/main.c:1126:19: error: too few arguments to function
'rte_vhost_submit_enqueue_burst'
   enqueue_count = rte_vhost_submit_enqueue_burst(vdev->vid,
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../examples/vhost/ioat.h:10:0,
                 from ../examples/vhost/main.c:28:
../lib/librte_vhost/rte_vhost_async.h:171:10: note: declared here
 uint16_t rte_vhost_submit_enqueue_burst(int vid, uint16_t queue_id,
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~