[v2,0/9] Introduce vfio-user library

  This series enables DPDK to be an alternative I/O device emulation library of
building virtualized devices in separate processes outside QEMU. It introduces
a new library for device emulation (librte_vfio_user).

*librte_vfio_user* library is an implementation of VFIO-over-socket[1] (also
known as vfio-user) which is a protocol that allows a device to be virtualized
in a separate process outside of QEMU. 

Background & Motivation 
-----------------------
The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user
as the main transport mechanism to disaggregate IO services from QEMU[2].
Vfio-user essentially implements the VFIO device model presented to the
user process by a set of messages over a unix-domain socket. The main
difference between application using vfio-user and application using vfio
kernel module is that device manipulation is based on socket messages for
vfio-user but system calls for vfio kernel module. The vfio-user devices
consist of a generic VFIO device type, living in QEMU, which is called the
client[3], and the core device implementation (emulated device), living
outside of QEMU, which is called the server. With emulated devices removed
from QEMU enabled by vfio-user implementation, other places should be
introduced to accommodate virtualized/emulated device. This series introduces
vfio-user support in DPDK to enable DPDK as one of the living places for
emulated device except QEMU.

This series introduce the server and client implementation of vfio-user protocol.
The server plays the role as emulated devices and the client is the device
consumer. With this implementation, DPDK will be enabled to be both device
provider and consumer.

Design overview
---------------

+--------------+     +--------------+     
| +----------+ |     | +----------+ |
| | Generic  | |     | | Emulated | |
| | vfio-dev | |     | | device   | |
| +----------+ |     | +----|-----+ |
| +----------+ |     | +----|-----+ |
| | vfio-user| |     | | vfio-user| |
| | client   | |<--->| | server   | |
| +----------+ |     | +----------+ |
| QEMU/DPDK    |     | DPDK         |
+--------------+     +--------------+

- Generic vfio-dev. 
  It is the generic vfio framework in vfio applications like QEMU or DPDK.
  Applications can keep the most of vfio device management and plug in a
  vfio-user device type. Note that in current implementation, we have not
  yet integrated client vfio-user into kernel vfio in DPDK but it is viable
  and good to do so.

- vfio-user client.
  For DPDK, it is part of librte_vfio_user implementation to provide ways to
  manipulate a vfio-user based emulated devices. This manipulation is very
  similar with kernel vfio (i.e., syscalls like ioctl, mmap and pread/pwrite).
  It is a base for vfio-user device consumer.

- vfio-user server. 
  It is server part of librte_vfio_user. It provides ways to emulate your own
  device. A device provider could only care about device layout that VFIO
  defines but does not need to know how it communicates with vfio-user client.

- Emulated device.
  It is emulated device of any type (e.g., network, crypto and etc.).

References
----------
[1]: https://patchew.org/QEMU/20201130161229.23164-1-thanos.makatos@nutanix.com/
[2]: https://wiki.qemu.org/Features/MultiProcessQEMU
[3]: https://github.com/oracle/qemu/tree/vfio-user-v0.2

----------------------------------
v2:
 - Clean up non-static inline function (Stephen)
 - Naturally pack vfio-user message payload and header (Stephen)
 - Make function definiton align with coding style (Beilei)
 - Clean up duplicate code in vfio-user server APIs (Beilei)
 - Fix some typos

Chenbo Xia (9):
  lib: introduce vfio-user library
  vfio_user: implement lifecycle related APIs
  vfio_user: implement device and region related APIs
  vfio_user: implement DMA table and socket address API
  vfio_user: implement interrupt related APIs
  vfio_user: add client APIs of device attach/detach
  vfio_user: add client APIs of DMA/IRQ/region
  test/vfio_user: introduce functional test
  doc: add vfio-user library guide

 MAINTAINERS                             |    4 +
 app/test/meson.build                    |    4 +
 app/test/test_vfio_user.c               |  665 ++++++++++
 doc/guides/prog_guide/index.rst         |    1 +
 doc/guides/prog_guide/vfio_user_lib.rst |  215 +++
 doc/guides/rel_notes/release_21_02.rst  |   11 +
 lib/librte_vfio_user/meson.build        |   11 +
 lib/librte_vfio_user/rte_vfio_user.h    |  446 +++++++
 lib/librte_vfio_user/version.map        |   26 +
 lib/librte_vfio_user/vfio_user_base.c   |  223 ++++
 lib/librte_vfio_user/vfio_user_base.h   |  109 ++
 lib/librte_vfio_user/vfio_user_client.c |  700 ++++++++++
 lib/librte_vfio_user/vfio_user_client.h |   26 +
 lib/librte_vfio_user/vfio_user_server.c | 1593 +++++++++++++++++++++++
 lib/librte_vfio_user/vfio_user_server.h |   66 +
 lib/meson.build                         |    1 +
 16 files changed, 4101 insertions(+)
 create mode 100644 app/test/test_vfio_user.c
 create mode 100644 doc/guides/prog_guide/vfio_user_lib.rst
 create mode 100644 lib/librte_vfio_user/meson.build
 create mode 100644 lib/librte_vfio_user/rte_vfio_user.h
 create mode 100644 lib/librte_vfio_user/version.map
 create mode 100644 lib/librte_vfio_user/vfio_user_base.c
 create mode 100644 lib/librte_vfio_user/vfio_user_base.h
 create mode 100644 lib/librte_vfio_user/vfio_user_client.c
 create mode 100644 lib/librte_vfio_user/vfio_user_client.h
 create mode 100644 lib/librte_vfio_user/vfio_user_server.c
 create mode 100644 lib/librte_vfio_user/vfio_user_server.h
  

Hello Chenbo,

On Thu, Jan 14, 2021 at 7:19 AM Chenbo Xia <chenbo.xia@intel.com> wrote:
>
> This series enables DPDK to be an alternative I/O device emulation library of
> building virtualized devices in separate processes outside QEMU. It introduces
> a new library for device emulation (librte_vfio_user).
>
> *librte_vfio_user* library is an implementation of VFIO-over-socket[1] (also
> known as vfio-user) which is a protocol that allows a device to be virtualized
> in a separate process outside of QEMU.
>
> Background & Motivation
> -----------------------
> The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user
> as the main transport mechanism to disaggregate IO services from QEMU[2].
> Vfio-user essentially implements the VFIO device model presented to the
> user process by a set of messages over a unix-domain socket. The main
> difference between application using vfio-user and application using vfio
> kernel module is that device manipulation is based on socket messages for
> vfio-user but system calls for vfio kernel module. The vfio-user devices
> consist of a generic VFIO device type, living in QEMU, which is called the
> client[3], and the core device implementation (emulated device), living
> outside of QEMU, which is called the server. With emulated devices removed
> from QEMU enabled by vfio-user implementation, other places should be
> introduced to accommodate virtualized/emulated device. This series introduces
> vfio-user support in DPDK to enable DPDK as one of the living places for
> emulated device except QEMU.
>
> This series introduce the server and client implementation of vfio-user protocol.
> The server plays the role as emulated devices and the client is the device
> consumer. With this implementation, DPDK will be enabled to be both device
> provider and consumer.
>
> Design overview
> ---------------
>
> +--------------+     +--------------+
> | +----------+ |     | +----------+ |
> | | Generic  | |     | | Emulated | |
> | | vfio-dev | |     | | device   | |
> | +----------+ |     | +----|-----+ |
> | +----------+ |     | +----|-----+ |
> | | vfio-user| |     | | vfio-user| |
> | | client   | |<--->| | server   | |
> | +----------+ |     | +----------+ |
> | QEMU/DPDK    |     | DPDK         |
> +--------------+     +--------------+
>
> - Generic vfio-dev.
>   It is the generic vfio framework in vfio applications like QEMU or DPDK.
>   Applications can keep the most of vfio device management and plug in a
>   vfio-user device type. Note that in current implementation, we have not
>   yet integrated client vfio-user into kernel vfio in DPDK but it is viable
>   and good to do so.
>
> - vfio-user client.
>   For DPDK, it is part of librte_vfio_user implementation to provide ways to
>   manipulate a vfio-user based emulated devices. This manipulation is very
>   similar with kernel vfio (i.e., syscalls like ioctl, mmap and pread/pwrite).
>   It is a base for vfio-user device consumer.
>
> - vfio-user server.
>   It is server part of librte_vfio_user. It provides ways to emulate your own
>   device. A device provider could only care about device layout that VFIO
>   defines but does not need to know how it communicates with vfio-user client.
>
> - Emulated device.
>   It is emulated device of any type (e.g., network, crypto and etc.).
>
> References
> ----------
> [1]: https://patchew.org/QEMU/20201130161229.23164-1-thanos.makatos@nutanix.com/
> [2]: https://wiki.qemu.org/Features/MultiProcessQEMU
> [3]: https://github.com/oracle/qemu/tree/vfio-user-v0.2
>
> ----------------------------------
> v2:
>  - Clean up non-static inline function (Stephen)
>  - Naturally pack vfio-user message payload and header (Stephen)
>  - Make function definiton align with coding style (Beilei)
>  - Clean up duplicate code in vfio-user server APIs (Beilei)
>  - Fix some typos


GHA (called by the robot) caught various issues:
- doc: https://github.com/ovsrobot/dpdk/runs/1700373705?check_suite_focus=true#step:14:3407
- clang: https://github.com/ovsrobot/dpdk/runs/1700373722?check_suite_focus=true#step:14:1050
- i386: https://github.com/ovsrobot/dpdk/runs/1700373747?check_suite_focus=true#step:14:2607
- aarch64: https://github.com/ovsrobot/dpdk/runs/1700373764?check_suite_focus=true#step:14:2848
and https://github.com/ovsrobot/dpdk/runs/1700373770?check_suite_focus=true#step:14:2880
  

Hi David,

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Friday, January 15, 2021 3:58 PM
> To: Xia, Chenbo <chenbo.xia@intel.com>
> Cc: dev <dev@dpdk.org>; Thomas Monjalon <thomas@monjalon.net>; Stephen
> Hemminger <stephen@networkplumber.org>; Liang, Cunming
> <cunming.liang@intel.com>; Lu, Xiuchun <xiuchun.lu@intel.com>; Li, Miao
> <miao.li@intel.com>; Wu, Jingjing <jingjing.wu@intel.com>; Xing, Beilei
> <beilei.xing@intel.com>
> Subject: Re: [PATCH v2 0/9] Introduce vfio-user library
> 
> Hello Chenbo,
> 
> On Thu, Jan 14, 2021 at 7:19 AM Chenbo Xia <chenbo.xia@intel.com> wrote:
> >
> > This series enables DPDK to be an alternative I/O device emulation library
> of
> > building virtualized devices in separate processes outside QEMU. It
> introduces
> > a new library for device emulation (librte_vfio_user).
> >
> > *librte_vfio_user* library is an implementation of VFIO-over-socket[1] (also
> > known as vfio-user) which is a protocol that allows a device to be
> virtualized
> > in a separate process outside of QEMU.
> >
> > Background & Motivation
> > -----------------------
> > The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user
> > as the main transport mechanism to disaggregate IO services from QEMU[2].
> > Vfio-user essentially implements the VFIO device model presented to the
> > user process by a set of messages over a unix-domain socket. The main
> > difference between application using vfio-user and application using vfio
> > kernel module is that device manipulation is based on socket messages for
> > vfio-user but system calls for vfio kernel module. The vfio-user devices
> > consist of a generic VFIO device type, living in QEMU, which is called the
> > client[3], and the core device implementation (emulated device), living
> > outside of QEMU, which is called the server. With emulated devices removed
> > from QEMU enabled by vfio-user implementation, other places should be
> > introduced to accommodate virtualized/emulated device. This series
> introduces
> > vfio-user support in DPDK to enable DPDK as one of the living places for
> > emulated device except QEMU.
> >
> > This series introduce the server and client implementation of vfio-user
> protocol.
> > The server plays the role as emulated devices and the client is the device
> > consumer. With this implementation, DPDK will be enabled to be both device
> > provider and consumer.
> >
> > Design overview
> > ---------------
> >
> > +--------------+     +--------------+
> > | +----------+ |     | +----------+ |
> > | | Generic  | |     | | Emulated | |
> > | | vfio-dev | |     | | device   | |
> > | +----------+ |     | +----|-----+ |
> > | +----------+ |     | +----|-----+ |
> > | | vfio-user| |     | | vfio-user| |
> > | | client   | |<--->| | server   | |
> > | +----------+ |     | +----------+ |
> > | QEMU/DPDK    |     | DPDK         |
> > +--------------+     +--------------+
> >
> > - Generic vfio-dev.
> >   It is the generic vfio framework in vfio applications like QEMU or DPDK.
> >   Applications can keep the most of vfio device management and plug in a
> >   vfio-user device type. Note that in current implementation, we have not
> >   yet integrated client vfio-user into kernel vfio in DPDK but it is viable
> >   and good to do so.
> >
> > - vfio-user client.
> >   For DPDK, it is part of librte_vfio_user implementation to provide ways to
> >   manipulate a vfio-user based emulated devices. This manipulation is very
> >   similar with kernel vfio (i.e., syscalls like ioctl, mmap and
> pread/pwrite).
> >   It is a base for vfio-user device consumer.
> >
> > - vfio-user server.
> >   It is server part of librte_vfio_user. It provides ways to emulate your
> own
> >   device. A device provider could only care about device layout that VFIO
> >   defines but does not need to know how it communicates with vfio-user
> client.
> >
> > - Emulated device.
> >   It is emulated device of any type (e.g., network, crypto and etc.).
> >
> > References
> > ----------
> > [1]: https://patchew.org/QEMU/20201130161229.23164-1-
> thanos.makatos@nutanix.com/
> > [2]: https://wiki.qemu.org/Features/MultiProcessQEMU
> > [3]: https://github.com/oracle/qemu/tree/vfio-user-v0.2
> >
> > ----------------------------------
> > v2:
> >  - Clean up non-static inline function (Stephen)
> >  - Naturally pack vfio-user message payload and header (Stephen)
> >  - Make function definiton align with coding style (Beilei)
> >  - Clean up duplicate code in vfio-user server APIs (Beilei)
> >  - Fix some typos
> 
> 
> GHA (called by the robot) caught various issues:
> - doc:
> https://github.com/ovsrobot/dpdk/runs/1700373705?check_suite_focus=true#step:1
> 4:3407
> - clang:
> https://github.com/ovsrobot/dpdk/runs/1700373722?check_suite_focus=true#step:1
> 4:1050
> - i386:
> https://github.com/ovsrobot/dpdk/runs/1700373747?check_suite_focus=true#step:1
> 4:2607
> - aarch64:
> https://github.com/ovsrobot/dpdk/runs/1700373764?check_suite_focus=true#step:1
> 4:2848
> and
> https://github.com/ovsrobot/dpdk/runs/1700373770?check_suite_focus=true#step:1
> 4:2880

Thanks! Will fix in next version.

Chenbo
> 
> 
> --
> David Marchand