[v4,0/4] net/virtio: make virtqueue struct cache-friendly

  This series optimizes the cache usage of virtqueue struct,
by making a "fake" mbuf being dynamically allocated in Rx
virtnet struct, by removing a useless virtuque pointer
into the virtnet structs and by moving a few fields
to pack holes.

With these 3 patches, the virtqueue struct size goes from
576 bytes (9 cachelines) to 248 bytes (4 cachelines).

Changes in v4:
==============
- Fix ARM build failure after rebase (0-day robot)
- Fix fake_mbuf freeing (David)
- Applied David R-by

Changes in v3:
==============
- Use rte_zmalloc_socket for fake mbuf alloc (David)
- Fix typos in commit messages
- Remove superfluous pointer check befor freeing (David)
- Fix checkpatch warnings

Changes in v2:
==============
- Rebase on latest main
- Improve error path in virtio_init_queue
- Fix various typos in commit messages

Maxime Coquelin (4):
  net/virtio: remove reference to virtqueue in vrings
  net/virtio: improve queue init error path
  net/virtio: allocate fake mbuf in Rx queue
  net/virtio: pack virtqueue struct

 drivers/net/virtio/virtio_ethdev.c            | 64 +++++++++++--------
 drivers/net/virtio/virtio_rxtx.c              | 37 +++++------
 drivers/net/virtio/virtio_rxtx.h              |  5 +-
 drivers/net/virtio/virtio_rxtx_packed.c       |  4 +-
 drivers/net/virtio/virtio_rxtx_packed.h       |  6 +-
 drivers/net/virtio/virtio_rxtx_packed_avx.h   |  4 +-
 drivers/net/virtio/virtio_rxtx_packed_neon.h  |  4 +-
 drivers/net/virtio/virtio_rxtx_simple.h       |  2 +-
 .../net/virtio/virtio_rxtx_simple_altivec.c   |  2 +-
 drivers/net/virtio/virtio_rxtx_simple_neon.c  |  2 +-
 drivers/net/virtio/virtio_rxtx_simple_sse.c   |  2 +-
 .../net/virtio/virtio_user/virtio_user_dev.c  |  4 +-
 drivers/net/virtio/virtio_user_ethdev.c       |  2 +-
 drivers/net/virtio/virtqueue.h                | 24 ++++---
 14 files changed, 87 insertions(+), 75 deletions(-)
  

On Tue, 2021-03-16 at 10:38 +0100, Maxime Coquelin wrote:
> This series optimizes the cache usage of virtqueue struct,
> by making a "fake" mbuf being dynamically allocated in Rx
> virtnet struct, by removing a useless virtuque pointer
> into the virtnet structs and by moving a few fields
> to pack holes.
> 
> With these 3 patches, the virtqueue struct size goes from
> 576 bytes (9 cachelines) to 248 bytes (4 cachelines).
> 
> Changes in v4:
> ==============
> - Fix ARM build failure after rebase (0-day robot)
> - Fix fake_mbuf freeing (David)
> - Applied David R-by
> 
> Changes in v3:
> ==============
> - Use rte_zmalloc_socket for fake mbuf alloc (David)
> - Fix typos in commit messages
> - Remove superfluous pointer check befor freeing (David)
> - Fix checkpatch warnings
> 
> Changes in v2:
> ==============
> - Rebase on latest main
> - Improve error path in virtio_init_queue
> - Fix various typos in commit messages
> 
> Maxime Coquelin (4):
>   net/virtio: remove reference to virtqueue in vrings
>   net/virtio: improve queue init error path
>   net/virtio: allocate fake mbuf in Rx queue
>   net/virtio: pack virtqueue struct
> 
>  drivers/net/virtio/virtio_ethdev.c            | 64 +++++++++++--------
>  drivers/net/virtio/virtio_rxtx.c              | 37 +++++------
>  drivers/net/virtio/virtio_rxtx.h              |  5 +-
>  drivers/net/virtio/virtio_rxtx_packed.c       |  4 +-
>  drivers/net/virtio/virtio_rxtx_packed.h       |  6 +-
>  drivers/net/virtio/virtio_rxtx_packed_avx.h   |  4 +-
>  drivers/net/virtio/virtio_rxtx_packed_neon.h  |  4 +-
>  drivers/net/virtio/virtio_rxtx_simple.h       |  2 +-
>  .../net/virtio/virtio_rxtx_simple_altivec.c   |  2 +-
>  drivers/net/virtio/virtio_rxtx_simple_neon.c  |  2 +-
>  drivers/net/virtio/virtio_rxtx_simple_sse.c   |  2 +-
>  .../net/virtio/virtio_user/virtio_user_dev.c  |  4 +-
>  drivers/net/virtio/virtio_user_ethdev.c       |  2 +-
>  drivers/net/virtio/virtqueue.h                | 24 ++++---
>  14 files changed, 87 insertions(+), 75 deletions(-)
> 


Tested this in a PVP setup on ARM, giving a slight improvement in
performance. For the series:

Tested-by: Balazs Nemeth <bnemeth@redhat.com>
  

> -----Original Message-----
> From: Maxime Coquelin <maxime.coquelin@redhat.com>
> Sent: Tuesday, March 16, 2021 5:38 PM
> To: dev@dpdk.org; Xia, Chenbo <chenbo.xia@intel.com>; amorenoz@redhat.com;
> david.marchand@redhat.com; olivier.matz@6wind.com; bnemeth@redhat.com
> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>
> Subject: [PATCH v4 0/4] net/virtio: make virtqueue struct cache-friendly
> 
> This series optimizes the cache usage of virtqueue struct,
> by making a "fake" mbuf being dynamically allocated in Rx
> virtnet struct, by removing a useless virtuque pointer
> into the virtnet structs and by moving a few fields
> to pack holes.
> 
>  drivers/net/virtio/virtio_ethdev.c            | 64 +++++++++++--------
>  drivers/net/virtio/virtio_rxtx.c              | 37 +++++------
>  drivers/net/virtio/virtio_rxtx.h              |  5 +-
>  drivers/net/virtio/virtio_rxtx_packed.c       |  4 +-
>  drivers/net/virtio/virtio_rxtx_packed.h       |  6 +-
>  drivers/net/virtio/virtio_rxtx_packed_avx.h   |  4 +-
>  drivers/net/virtio/virtio_rxtx_packed_neon.h  |  4 +-
>  drivers/net/virtio/virtio_rxtx_simple.h       |  2 +-
>  .../net/virtio/virtio_rxtx_simple_altivec.c   |  2 +-
>  drivers/net/virtio/virtio_rxtx_simple_neon.c  |  2 +-
>  drivers/net/virtio/virtio_rxtx_simple_sse.c   |  2 +-
>  .../net/virtio/virtio_user/virtio_user_dev.c  |  4 +-
>  drivers/net/virtio/virtio_user_ethdev.c       |  2 +-
>  drivers/net/virtio/virtqueue.h                | 24 ++++---
>  14 files changed, 87 insertions(+), 75 deletions(-)
> 
> --
> 2.29.2

With below commit log change(because of commit log line too long)

    net/virtio: allocate fake mbuf in Rx queue
    [...]
    struct virtnet_rx {
     struct virtqueue *vq; /*0 8*/

     /* XXX 56 bytes hole, try to pack */

     /* --- cacheline 1 boundary (64 bytes) --- */
     struct rte_mbuf fake_mbuf __attribute__((__aligned__(64))); /*64 128*/
     /* --- cacheline 3 boundary (192 bytes) --- */

Series applied to next-virtio/main.

Thanks!
Chenbo