mbox

[v2,00/18] mlx5: sharing global MR cache between drivers

Message ID 20211006220350.2357487-1-michaelba@nvidia.com (mailing list archive)
Headers

Message

Michael Baum Oct. 6, 2021, 10:03 p.m. UTC 
From: Michael Baum <michaelba@oss.nvidia.com>

The MLNX PMD supports 5 classes (net, RegEx, vDPA, Compress and Crypto).
The various drivers are registered under the common driver, and managed
by it.
In the common driver probing, it calls in a loop the probe function of
each driver registered to it. Each driver creates for itself all the
objects required for communication with the hardware, as well as a
global MR cache that manages memory mappings.
The management of the caches separately by the different drivers, is not
very efficient. In fact the same memory is managed several times and
each caching handle is done by doubling the number of drivers, so we
want to manage this cache jointly by the common driver.
This feature will move management to common driver in two phases.

Phase 1: sharing HW objects between drivers (11 patches)
The communication with the hardware - for any MR handle - is conducted
by the Protection Domain, so we are motivated to share it between the
drivers. However, to create it we need to give the context of the
device, so the context must also be shared between the drivers.
At this point we will share between the drivers the next trio (CTX, PD,
pdn) to create an infrastructure that will allow sharing of dependent
objects, and in particular the global MR cache.
The common driver itself will create this trio individually for all
drivers, before calling their probe function. As a parameter to the
probe function, it will give them a pointer to the structure containing
the trio.

Phase 2: sharing global MR cache between drivers (7 patches)
The common driver will add to the structure containing the trio also the
structure that manages the global MR cache, and will keep a list of such
structures for memory management. In each driver, each queue will manage
its own local MR cache. If the queue does not find its cache, it will
switch to searching the global MR cache shared by all. Caching access
will be through the pointer that the driver received as a parameter in
probing.

Depends-on: series-19267 ("mempool: add event callbacks")
https://patchwork.dpdk.org/project/dpdk/list/?series=19267

v2: rebase

Michael Baum (18):
  net/mlx5/windows: fix miss callback register for mem event
  common/mlx5: share basic probing with the internal drivers
  common/mlx5: share common definitions
  common/mlx5: share memory related devargs
  net/mlx5/windows: rearrange probing code
  common/mlx5: move basic probing functions to common
  net/mlx5: remove redundant flag in device config
  common/mlx5: share device context object
  common/mlx5: add ROCE disable in context device creation
  common/mlx5: share the protection domain object
  common/mlx5: share the HCA capabilities handle
  net/mlx5: remove redundancy in MR file
  common/mlx5: add MR ctrl init function
  common/mlx5: add global MR cache create function
  common/mlx5: share MR top-half search function
  common/mlx5: share MR management
  common/mlx5: support device DMA map and unmap
  common/mlx5: share MR mempool registration

 drivers/common/mlx5/linux/mlx5_common_os.c    | 280 +++++++-
 drivers/common/mlx5/linux/mlx5_common_os.h    |  16 +-
 drivers/common/mlx5/linux/mlx5_common_verbs.c |  96 ++-
 drivers/common/mlx5/linux/mlx5_nl.c           |   2 +-
 drivers/common/mlx5/linux/mlx5_nl.h           |   6 +-
 drivers/common/mlx5/mlx5_common.c             | 596 +++++++++++++++---
 drivers/common/mlx5/mlx5_common.h             |  96 +--
 drivers/common/mlx5/mlx5_common_defs.h        |  45 ++
 drivers/common/mlx5/mlx5_common_mp.h          |  11 +
 drivers/common/mlx5/mlx5_common_mr.c          | 192 +++++-
 drivers/common/mlx5/mlx5_common_mr.h          |  76 ++-
 drivers/common/mlx5/mlx5_common_private.h     |   6 -
 drivers/common/mlx5/mlx5_devx_cmds.h          |  12 +-
 drivers/common/mlx5/mlx5_malloc.h             |   1 -
 drivers/common/mlx5/version.map               |  29 +-
 drivers/common/mlx5/windows/mlx5_common_os.c  | 245 ++++++-
 drivers/common/mlx5/windows/mlx5_common_os.h  |   9 +-
 drivers/compress/mlx5/mlx5_compress.c         | 237 ++-----
 drivers/crypto/mlx5/mlx5_crypto.c             | 201 +-----
 drivers/crypto/mlx5/mlx5_crypto.h             |   6 +-
 drivers/crypto/mlx5/mlx5_crypto_dek.c         |   5 +-
 drivers/net/mlx5/linux/mlx5_ethdev_os.c       |   6 +-
 drivers/net/mlx5/linux/mlx5_mp_os.c           |  12 +-
 drivers/net/mlx5/linux/mlx5_os.c              | 511 ++++-----------
 drivers/net/mlx5/linux/mlx5_verbs.c           |  99 +--
 drivers/net/mlx5/linux/mlx5_verbs.h           |   2 -
 drivers/net/mlx5/meson.build                  |   1 -
 drivers/net/mlx5/mlx5.c                       | 241 ++-----
 drivers/net/mlx5/mlx5.h                       |  57 +-
 drivers/net/mlx5/mlx5_defs.h                  |  22 +-
 drivers/net/mlx5/mlx5_devx.c                  |  39 +-
 drivers/net/mlx5/mlx5_flow.c                  |   6 +-
 drivers/net/mlx5/mlx5_flow_aso.c              |  50 +-
 drivers/net/mlx5/mlx5_flow_dv.c               |  66 +-
 drivers/net/mlx5/mlx5_flow_verbs.c            |   4 +-
 drivers/net/mlx5/mlx5_mr.c                    | 397 ------------
 drivers/net/mlx5/mlx5_mr.h                    |  26 -
 drivers/net/mlx5/mlx5_rx.c                    |  16 +-
 drivers/net/mlx5/mlx5_rx.h                    |  20 +-
 drivers/net/mlx5/mlx5_rxq.c                   |  11 +-
 drivers/net/mlx5/mlx5_rxtx.c                  |   1 -
 drivers/net/mlx5/mlx5_rxtx.h                  |  27 -
 drivers/net/mlx5/mlx5_rxtx_vec.h              |   1 -
 drivers/net/mlx5/mlx5_trigger.c               |   8 +-
 drivers/net/mlx5/mlx5_tx.c                    |   1 -
 drivers/net/mlx5/mlx5_tx.h                    |  30 +-
 drivers/net/mlx5/mlx5_txpp.c                  |  23 +-
 drivers/net/mlx5/mlx5_txq.c                   |  10 +-
 drivers/net/mlx5/windows/mlx5_ethdev_os.c     |  12 +-
 drivers/net/mlx5/windows/mlx5_os.c            | 402 ++----------
 drivers/regex/mlx5/mlx5_regex.c               | 144 +----
 drivers/regex/mlx5/mlx5_regex.h               |  27 +-
 drivers/regex/mlx5/mlx5_regex_control.c       |  19 +-
 drivers/regex/mlx5/mlx5_regex_fastpath.c      |  47 +-
 drivers/regex/mlx5/mlx5_rxp.c                 |  62 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 | 212 +------
 drivers/vdpa/mlx5/mlx5_vdpa.h                 |   5 +-
 drivers/vdpa/mlx5/mlx5_vdpa_event.c           |  25 +-
 drivers/vdpa/mlx5/mlx5_vdpa_lm.c              |   6 +-
 drivers/vdpa/mlx5/mlx5_vdpa_mem.c             |  13 +-
 drivers/vdpa/mlx5/mlx5_vdpa_steer.c           |  11 +-
 drivers/vdpa/mlx5/mlx5_vdpa_virtq.c           |  15 +-
 62 files changed, 2075 insertions(+), 2779 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_defs.h
 delete mode 100644 drivers/net/mlx5/mlx5_mr.c
 delete mode 100644 drivers/net/mlx5/mlx5_mr.h