[v4,0/3] vhost: make virtqueue cache-friendly

  As done for Virtio PMD, this series improves cache utilization
of the vhost_virtqueue struct by removing unused field,
make the live-migration cache dynamically allocated at
live-migration setup time and by moving fields
around so that hot fields are on the first cachelines.

With this series, The struct vhost_virtqueue size goes
from 832B (13 cachelines) down to 320B (5 cachelines).

With this series and the virtio one, I measure a gain
of up to 8% in IO loop micro-benchmark with packed
ring, and 5% with split ring.

I don't have a setup at hand to run PVP testing, but
it might be interresting to get the numbers as I
suspect the cache pressure is higher in this test as
in real use-cases.

Changes in v4:
==============
- Fix missing changes to boolean (Chenbo)

Changes in v3:
==============
- Don't check pointer validity before freeing (David)
- Don't use deprecated rte_smp_wmb() (David, Checkpatch)
- Handle booleans properly (David)
- Prevent VQ size field overflow (David)
- Fix typo and indent (David)

Changes in v2:
==============
- Add log_cache freeing in free_vq (Chenbo)

Maxime Coquelin (3):
  vhost: remove unused Vhost virtqueue field
  vhost: move dirty logging cache out of the virtqueue
  vhost: optimize vhost virtqueue struct

 lib/librte_vhost/vhost.c      | 21 +++++++++----
 lib/librte_vhost/vhost.h      | 56 +++++++++++++++++------------------
 lib/librte_vhost/vhost_user.c | 44 +++++++++++++++++++--------
 lib/librte_vhost/virtio_net.c | 12 ++++----
 4 files changed, 82 insertions(+), 51 deletions(-)
  

On Tue, Mar 23, 2021 at 10:02 AM Maxime Coquelin
<maxime.coquelin@redhat.com> wrote:
>
> As done for Virtio PMD, this series improves cache utilization
> of the vhost_virtqueue struct by removing unused field,
> make the live-migration cache dynamically allocated at
> live-migration setup time and by moving fields
> around so that hot fields are on the first cachelines.
>
> With this series, The struct vhost_virtqueue size goes
> from 832B (13 cachelines) down to 320B (5 cachelines).
>
> With this series and the virtio one, I measure a gain
> of up to 8% in IO loop micro-benchmark with packed
> ring, and 5% with split ring.
>
> I don't have a setup at hand to run PVP testing, but
> it might be interresting to get the numbers as I
> suspect the cache pressure is higher in this test as
> in real use-cases.
>
> Changes in v4:
> ==============
> - Fix missing changes to boolean (Chenbo)
>

For the series,
Reviewed-by: David Marchand <david.marchand@redhat.com>

Merci !
  

On Tue, 2021-03-23 at 11:30 +0100, David Marchand wrote:
> On Tue, Mar 23, 2021 at 10:02 AM Maxime Coquelin
> <maxime.coquelin@redhat.com> wrote:
> > 
> > As done for Virtio PMD, this series improves cache utilization
> > of the vhost_virtqueue struct by removing unused field,
> > make the live-migration cache dynamically allocated at
> > live-migration setup time and by moving fields
> > around so that hot fields are on the first cachelines.
> > 
> > With this series, The struct vhost_virtqueue size goes
> > from 832B (13 cachelines) down to 320B (5 cachelines).
> > 
> > With this series and the virtio one, I measure a gain
> > of up to 8% in IO loop micro-benchmark with packed
> > ring, and 5% with split ring.
> > 
> > I don't have a setup at hand to run PVP testing, but
> > it might be interresting to get the numbers as I
> > suspect the cache pressure is higher in this test as
> > in real use-cases.
> > 
> > Changes in v4:
> > ==============
> > - Fix missing changes to boolean (Chenbo)
> > 
> 
> For the series,
> Reviewed-by: David Marchand <david.marchand@redhat.com>
> 
> Merci !
> 
> 

Tested this in a PVP setup on ARM, giving a slight improvement in
performance. For the series:

Tested-by: Balazs Nemeth <bnemeth@redhat.com>
  

> -----Original Message-----
> From: Maxime Coquelin <maxime.coquelin@redhat.com>
> Sent: Tuesday, March 23, 2021 5:02 PM
> To: dev@dpdk.org; Xia, Chenbo <chenbo.xia@intel.com>; amorenoz@redhat.com;
> david.marchand@redhat.com; olivier.matz@6wind.com; bnemeth@redhat.com
> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>
> Subject: [PATCH v4 0/3] vhost: make virtqueue cache-friendly
> 
> As done for Virtio PMD, this series improves cache utilization
> of the vhost_virtqueue struct by removing unused field,
> make the live-migration cache dynamically allocated at
> live-migration setup time and by moving fields
> around so that hot fields are on the first cachelines.
> 
> With this series, The struct vhost_virtqueue size goes
> from 832B (13 cachelines) down to 320B (5 cachelines).
> 
> With this series and the virtio one, I measure a gain
> of up to 8% in IO loop micro-benchmark with packed
> ring, and 5% with split ring.
> 
> I don't have a setup at hand to run PVP testing, but
> it might be interresting to get the numbers as I
> suspect the cache pressure is higher in this test as
> in real use-cases.
> 
> Maxime Coquelin (3):
>   vhost: remove unused Vhost virtqueue field
>   vhost: move dirty logging cache out of the virtqueue
>   vhost: optimize vhost virtqueue struct
> 
>  lib/librte_vhost/vhost.c      | 21 +++++++++----
>  lib/librte_vhost/vhost.h      | 56 +++++++++++++++++------------------
>  lib/librte_vhost/vhost_user.c | 44 +++++++++++++++++++--------
>  lib/librte_vhost/virtio_net.c | 12 ++++----
>  4 files changed, 82 insertions(+), 51 deletions(-)
> 
> --
> 2.30.2

Series applied to next-virtio/main, Thanks!