mbox series

[RFC,0/6] Improve mlx5 PMD common driver framework for multiple classes

Message ID 20200610171728.89-1-parav@mellanox.com (mailing list archive)
Headers
Series Improve mlx5 PMD common driver framework for multiple classes |

Message

Parav Pandit June 10, 2020, 5:17 p.m. UTC 
  This commit introduces mlx5 bus to support multiple class of devices
for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
a new mlx5 PCI bus is created. This bus allows binding multiple
class drivers (such as net, vdpa, regex(future)) to bind to the
mlx5 PCI bus driver.

Change description
------------------
Patch-1 prepares the code to have RTE_BIT() macro defined in a
common header.
Patch-2 Prepares the code to check for enabled class
Patch-3 Changes class value to a bit field
Patch-4 Exposes mlx5_pci class driver registration APIs
PAtch-5 Implements mlx5 PCI bus
Patch-6 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus
API instead of rte PCI bus API

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       -------------       /
         \______|   mlx5    |_____ /
                |   pci bus |
                -------------   
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD defines its
  specific PCI ID table and mlx5 bus driver probes matching
  class drivers.
- mlx5 pci bus driver is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI bus driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci bus, a common driver is
implemented which exposes class registration API.
However, bus model fits better with existing DPDK design
similar to ifpga driver.
Class registration API need to create a new callbacks
and ID signature; instead it is better to utilize current
well defined methods.

2. Enhance pci core to allow multiple driver binding to
single rte PCI device.
This approach is not taken, because peer drivers using
one PCI device won't be aware of other's presence. This
requires cross-driver syncronization of who initializes
common resources (such as irq, eq and more).
This also requires refcounting common objects etc among
peer drivers.
Instead of layered approach delivers and allows putting
common resource sharing, setup code in common bus driver.
It also eliminates peer blind zone problem as bottom pci
bus layer provides necessary setup without any reference
counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf
used between net and regex pmd so that same mbuf use across
multiple device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list
of classes at command line such as,
./testpmd -w <PCI BDF>,class=net:vdpa
./testpmd -w <PCI BDF>,class=vdpa

In future,
./testpmd -w <PCI BDF>,class=net:regex


Parav Pandit (6):
  eal: introduce macros for getting value for bit
  common/mlx5: use class enable check helper function
  common/mlx5: change mlx5 class enum values as bits
  bus/mlx5_pci: add mlx5 PCI bus
  bus/mlx5_pci: register a PCI driver
  bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver

 config/common_base                            |   6 +
 config/defconfig_arm64-bluefield-linuxapp-gcc |   6 +
 drivers/bus/Makefile                          |   3 +
 drivers/bus/meson.build                       |   2 +-
 drivers/bus/mlx5_pci/Makefile                 |  49 +++
 drivers/bus/mlx5_pci/meson.build              |   6 +
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 327 ++++++++++++++++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       |  66 ++++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |   5 +
 drivers/common/mlx5/mlx5_common.c             |  12 +-
 drivers/common/mlx5/mlx5_common.h             |   8 +-
 .../common/mlx5/rte_common_mlx5_version.map   |   2 +-
 drivers/net/mlx5/Makefile                     |   3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   4 +-
 drivers/net/mlx5/linux/mlx5_os.h              |   3 -
 drivers/net/mlx5/meson.build                  |   2 +-
 drivers/net/mlx5/mlx5.c                       |   7 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/vdpa/mlx5/Makefile                    |   3 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  12 +-
 lib/librte_eal/include/rte_bits.h             |  10 +
 mk/rte.app.mk                                 |   1 +
 23 files changed, 511 insertions(+), 29 deletions(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
 create mode 100644 lib/librte_eal/include/rte_bits.h