mbox series

[0/4] net/mlx5: implicit mempool registration

Message ID 20210818090755.2419483-1-dkozlyuk@nvidia.com (mailing list archive)
Headers
Series net/mlx5: implicit mempool registration |

Message

Dmitry Kozlyuk Aug. 18, 2021, 9:07 a.m. UTC 
  MLX5 hardware has its internal IOMMU where PMD registers the memory.
On the data path, PMD translates VA into a key consumed by the device
IOMMU.  It is impractical for the PMD to register all allocated memory
because of increased lookup cost both in HW and SW.  Most often mbuf
memory comes from mempools, so if PMD tracks them, it can almost always
have mbuf memory registered before an mbuf hits the PMD. This patchset
adds such tracking in the PMD and internal API to support it.

Please see [1] for a more thorough explanation of the patch 2/4
and how it can be useful outside of the MLX5 PMD.

[1]: http://inbox.dpdk.org/dev/CH0PR12MB509112FADB778AB28AF3771DB9F99@CH0PR12MB5091.namprd12.prod.outlook.com/

Dmitry Kozlyuk (4):
  mempool: add event callbacks
  mempool: add non-IO flag
  common/mlx5: add mempool registration facilities
  net/mlx5: support mempool registration

 doc/guides/nics/mlx5.rst               |  11 +
 doc/guides/rel_notes/release_21_11.rst |   9 +
 drivers/common/mlx5/mlx5_common_mp.c   |  50 +++
 drivers/common/mlx5/mlx5_common_mp.h   |  14 +
 drivers/common/mlx5/mlx5_common_mr.c   | 564 +++++++++++++++++++++++++
 drivers/common/mlx5/mlx5_common_mr.h   |  17 +
 drivers/common/mlx5/version.map        |   5 +
 drivers/net/mlx5/linux/mlx5_mp_os.c    |  44 ++
 drivers/net/mlx5/linux/mlx5_os.c       |   4 +-
 drivers/net/mlx5/linux/mlx5_os.h       |   2 +
 drivers/net/mlx5/mlx5.c                | 128 ++++++
 drivers/net/mlx5/mlx5.h                |  13 +
 drivers/net/mlx5/mlx5_mr.c             |  27 ++
 drivers/net/mlx5/mlx5_trigger.c        |  10 +-
 lib/mempool/rte_mempool.c              | 153 ++++++-
 lib/mempool/rte_mempool.h              |  60 +++
 lib/mempool/version.map                |   8 +
 17 files changed, 1110 insertions(+), 9 deletions(-)

Comments

Dmitry Kozlyuk Sept. 29, 2021, 2:52 p.m. UTC | #1 
From: Dmitry Kozlyuk <dkozlyuk@oss.nvidia.com>

MLX5 hardware has its internal IOMMU where PMD registers the memory.
On the data path, PMD translates VA into a key consumed by the device
IOMMU.  It is impractical for the PMD to register all allocated memory
because of increased lookup cost both in HW and SW.  Most often mbuf
memory comes from mempools, so if PMD tracks them, it can almost always
have mbuf memory registered before an mbuf hits the PMD. This patchset
adds such tracking in the PMD and internal API to support it.

Please see [1] for a more thorough explanation of the patch 2/4
and how it can be useful outside of the MLX5 PMD.

[1]: http://inbox.dpdk.org/dev/CH0PR12MB509112FADB778AB28AF3771DB9F99@CH0PR12MB5091.namprd12.prod.outlook.com/

v2 (internal review and testing):
    1. Change tracked mempool event from being created (CREATE) to being
       fully populated (READY), which is the state PMD is interested in.
    2. Unit test the new mempool callback API.
    3. Remove bogus "error" messages in normal conditions.
    4. Fixes in PMD.

Dmitry Kozlyuk (4):
  mempool: add event callbacks
  mempool: add non-IO flag
  common/mlx5: add mempool registration facilities
  net/mlx5: support mempool registration

 app/test/test_mempool.c                |  75 ++++
 doc/guides/nics/mlx5.rst               |  11 +
 doc/guides/rel_notes/release_21_11.rst |   9 +
 drivers/common/mlx5/mlx5_common_mp.c   |  50 +++
 drivers/common/mlx5/mlx5_common_mp.h   |  14 +
 drivers/common/mlx5/mlx5_common_mr.c   | 580 +++++++++++++++++++++++++
 drivers/common/mlx5/mlx5_common_mr.h   |  17 +
 drivers/common/mlx5/version.map        |   5 +
 drivers/net/mlx5/linux/mlx5_mp_os.c    |  44 ++
 drivers/net/mlx5/linux/mlx5_os.c       |   4 +-
 drivers/net/mlx5/mlx5.c                | 152 +++++++
 drivers/net/mlx5/mlx5.h                |  10 +
 drivers/net/mlx5/mlx5_mr.c             | 120 ++---
 drivers/net/mlx5/mlx5_mr.h             |   2 -
 drivers/net/mlx5/mlx5_rx.h             |  21 +-
 drivers/net/mlx5/mlx5_rxq.c            |  13 +
 drivers/net/mlx5/mlx5_trigger.c        |  77 +++-
 drivers/net/mlx5/windows/mlx5_os.c     |   1 +
 lib/mempool/rte_mempool.c              | 143 +++++-
 lib/mempool/rte_mempool.h              |  60 +++
 lib/mempool/version.map                |   8 +
 21 files changed, 1297 insertions(+), 119 deletions(-)