[v4,00/10] support SW assisted VDPA live migration

  In the previous VDPA implementation we have enabled live migration support
by HW accelerator doing all the stuff, including dirty page logging and
device status report/restore. In this mode VDPA sample daemon and device
driver just takes care of the control path and does not involve in data
path, so there's almost 0 CPU resource usage. This mode requires device
to have dirty page logging capability.

This patch series adds live migration support for devices without logging
capability. VDPA driver could set up a relay thread standing between the
guest and device when live migration happens, this relay intervenes into
the communication between guest virtio driver and physical virtio
accelerator, it helps device to do a vring relay and passingly log dirty
pages. Thus some CPU resource will be consumed in this scenario, percentage
depending on the network throughput.

Some new helpers are added into vhost lib for this VDPA SW fallback:
- rte_vhost_host_notifier_ctrl, to enable/disable the VDPA direct-IO
  datapath.
- rte_vdpa_relay_vring_avail, to relay the available request from guest vring
  to mediate vring.
- rte_vdpa_relay_vring_used, to relay the used response from mediate vring to
  guest vring.

Some existing helpers are also leveraged for SW fallback setup, like VFIO
interrupt configuration, IOMMU table programming, etc.

This patch enables this SW assisted VDPA live migration in ifc driver.
Since ifcvf also supports HW dirty page logging, we add a new devarg
for user to select if the SW mode is used or not.

v4:
* Add a patch to remove the unused vhost internal API: vhost_detach_vdpa_device().

v3:
* Fix indent in relay code.
* Fix the iova access mode issue of buffer check.
* Rename the relay API to be more generic, and add more API note for used
  ring handling.
* Add kvargs lib dependency in ifc driver.
* Add commit message for the doc update patch for checkpatch warning.

v2:
* Reword the vdpa host notifier control API comment.
* Make the vring relay API parameter as "void *" to accomodate the future
  potential new ring layout, e.g. packed ring.
* Add parameter check for the new API.
* Add memory barrier for ring idx update.
* Remove the used ring logging in the relay.
* Some comment fix and code cleaning according to Tiwei's comment.
* Add release note update.

Xiao Wang (10):
  vhost: remove unused internal API
  vhost: provide helper for host notifier ctrl
  vhost: provide helpers for virtio ring relay
  net/ifc: dump debug message for error
  net/ifc: store only registered device instance
  net/ifc: detect if VDPA mode is specified
  net/ifc: add devarg for LM mode
  net/ifc: use lib API for used ring logging
  net/ifc: support SW assisted VDPA live migration
  doc: update ifc NIC document

 doc/guides/nics/ifc.rst                |   8 +
 doc/guides/rel_notes/release_19_02.rst |   6 +
 drivers/net/ifc/Makefile               |   1 +
 drivers/net/ifc/base/ifcvf.h           |   1 +
 drivers/net/ifc/ifcvf_vdpa.c           | 461 ++++++++++++++++++++++++++++++---
 lib/librte_vhost/rte_vdpa.h            |  57 ++++
 lib/librte_vhost/rte_vhost_version.map |   3 +
 lib/librte_vhost/vdpa.c                | 194 ++++++++++++++
 lib/librte_vhost/vhost.c               |  13 -
 lib/librte_vhost/vhost.h               |  41 ++-
 lib/librte_vhost/vhost_user.c          |   7 +-
 lib/librte_vhost/virtio_net.c          |  39 ---
 12 files changed, 741 insertions(+), 90 deletions(-)
  

On 12/14/18 10:16 PM, Xiao Wang wrote:
> In the previous VDPA implementation we have enabled live migration support
> by HW accelerator doing all the stuff, including dirty page logging and
> device status report/restore. In this mode VDPA sample daemon and device
> driver just takes care of the control path and does not involve in data
> path, so there's almost 0 CPU resource usage. This mode requires device
> to have dirty page logging capability.
> 
> This patch series adds live migration support for devices without logging
> capability. VDPA driver could set up a relay thread standing between the
> guest and device when live migration happens, this relay intervenes into
> the communication between guest virtio driver and physical virtio
> accelerator, it helps device to do a vring relay and passingly log dirty
> pages. Thus some CPU resource will be consumed in this scenario, percentage
> depending on the network throughput.
> 
> Some new helpers are added into vhost lib for this VDPA SW fallback:
> - rte_vhost_host_notifier_ctrl, to enable/disable the VDPA direct-IO
>    datapath.
> - rte_vdpa_relay_vring_avail, to relay the available request from guest vring
>    to mediate vring.
> - rte_vdpa_relay_vring_used, to relay the used response from mediate vring to
>    guest vring.
> 
> Some existing helpers are also leveraged for SW fallback setup, like VFIO
> interrupt configuration, IOMMU table programming, etc.
> 
> This patch enables this SW assisted VDPA live migration in ifc driver.
> Since ifcvf also supports HW dirty page logging, we add a new devarg
> for user to select if the SW mode is used or not.
> 
> v4:
> * Add a patch to remove the unused vhost internal API: vhost_detach_vdpa_device().
> 
> v3:
> * Fix indent in relay code.
> * Fix the iova access mode issue of buffer check.
> * Rename the relay API to be more generic, and add more API note for used
>    ring handling.
> * Add kvargs lib dependency in ifc driver.
> * Add commit message for the doc update patch for checkpatch warning.
> 
> v2:
> * Reword the vdpa host notifier control API comment.
> * Make the vring relay API parameter as "void *" to accomodate the future
>    potential new ring layout, e.g. packed ring.
> * Add parameter check for the new API.
> * Add memory barrier for ring idx update.
> * Remove the used ring logging in the relay.
> * Some comment fix and code cleaning according to Tiwei's comment.
> * Add release note update.
> 
> Xiao Wang (10):
>    vhost: remove unused internal API
>    vhost: provide helper for host notifier ctrl
>    vhost: provide helpers for virtio ring relay
>    net/ifc: dump debug message for error
>    net/ifc: store only registered device instance
>    net/ifc: detect if VDPA mode is specified
>    net/ifc: add devarg for LM mode
>    net/ifc: use lib API for used ring logging
>    net/ifc: support SW assisted VDPA live migration
>    doc: update ifc NIC document
> 
>   doc/guides/nics/ifc.rst                |   8 +
>   doc/guides/rel_notes/release_19_02.rst |   6 +
>   drivers/net/ifc/Makefile               |   1 +
>   drivers/net/ifc/base/ifcvf.h           |   1 +
>   drivers/net/ifc/ifcvf_vdpa.c           | 461 ++++++++++++++++++++++++++++++---
>   lib/librte_vhost/rte_vdpa.h            |  57 ++++
>   lib/librte_vhost/rte_vhost_version.map |   3 +
>   lib/librte_vhost/vdpa.c                | 194 ++++++++++++++
>   lib/librte_vhost/vhost.c               |  13 -
>   lib/librte_vhost/vhost.h               |  41 ++-
>   lib/librte_vhost/vhost_user.c          |   7 +-
>   lib/librte_vhost/virtio_net.c          |  39 ---
>   12 files changed, 741 insertions(+), 90 deletions(-)
> 


Applied to dpdk-next-virtio

Thanks,
Maxime