Message ID | 20210818090755.2419483-1-dkozlyuk@nvidia.com (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D6CEBA0C47; Wed, 18 Aug 2021 11:08:19 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5D55E4014E; Wed, 18 Aug 2021 11:08:19 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2077.outbound.protection.outlook.com [40.107.92.77]) by mails.dpdk.org (Postfix) with ESMTP id 09C4040042 for <dev@dpdk.org>; Wed, 18 Aug 2021 11:08:18 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ki81GueWbZQ/0bKE6o7nckjxGIPGUV0nJ7dnN/JvJSoawUBZwogGRFDuPnrujM4G1q0KcEEMOkMiJ6sXyAOs7d2A9pbVYk16VP+F4kjOPkNARRdn4PUDHjk+3ozxyPBnXPLn3hq91lhdV5QF1vA2BP+YBKTcinq9A3Ea+tDH7lVSQZ0WoJzj9vgKgEvXu5VTFejd94t6duqGEX93nZzrrQFY0CLXTAzaqvTWKzO/kmALUoZb/GOef/e/IQuocNCushiZqHxa7SjdLxOCr32pF2ZOwrwdLL7/PMkqQYbcsF5mmKGQ0FESkH3WBTZmavxgIFmpb9ivLRXN6NA+KAdbFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+uZJCT2JH3SRdHMwpbOTW44w27eawhEk/1kVifEmdYM=; b=P+l/UrUB640SSfGjvG/fQlcrb3TzgEbudeDKeV+H6EeHw64WSljeFHQcuIZvD1eC4i22RHDciWzPIqPRG0aRPsWInmWFW8SCIMhMZUE2vg1Ozt1L3AwbFI000iTeVWtRhzDGmpU5LpUB8dU+OTGxmTkSXCqTeNv0CjhqS3F+IEM3klVEOobIQmz0ugP8ARe7xuhnSveKvjbwlZBbnId3qTjK6dMJrakQm58g4o02W0hRXQPRZDGvhjZNSO4oFyu6w907rkG+VXUk3dIhNphNoR2lLjkzzcbNf9RQMLbhiTrBjmc4YsXTsL+5W0J80fCO45cmgct5HEerDAAzt6I8YA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.36) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+uZJCT2JH3SRdHMwpbOTW44w27eawhEk/1kVifEmdYM=; b=MSEtJCPoZ7VB3L6xj8E1o7mJz2lzaGYjzA+ooBBMIRfT2dnmQHXDUyhzl2eTJaekq8cwf3kIQGGcKjOnS8UPVMAEcX5o6t4SnhryJnK3O0Ua2ULAKKEvi3pgq2AgXrAYp6sRZTFidUorMsXXogCfyb6pn/cXfoSoLgJdT3n7AD6bxTWGhnlbiKJfBBlDUHjUV5XLxXgmKA8i5FwUQJpS/2hTUDS90P4Pi4g315hS09aMoKG1hqOkhKwySYTWwcqLurSuqPq68bZzWvC9orb7UEyAixvYEc7cw2LOJBnbFO47sFU2Z6Zjz51qOw1y0hcPFljhLSJ6KD+7yMWxepf/wg== Received: from MW3PR06CA0020.namprd06.prod.outlook.com (2603:10b6:303:2a::25) by MN2PR12MB3325.namprd12.prod.outlook.com (2603:10b6:208:ce::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.17; Wed, 18 Aug 2021 09:08:16 +0000 Received: from CO1NAM11FT006.eop-nam11.prod.protection.outlook.com (2603:10b6:303:2a:cafe::3c) by MW3PR06CA0020.outlook.office365.com (2603:10b6:303:2a::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.19 via Frontend Transport; Wed, 18 Aug 2021 09:08:15 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.36) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.36 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.36; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.36) by CO1NAM11FT006.mail.protection.outlook.com (10.13.174.246) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4415.16 via Frontend Transport; Wed, 18 Aug 2021 09:08:15 +0000 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 18 Aug 2021 09:08:14 +0000 Received: from nvidia.com (172.20.187.5) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 18 Aug 2021 09:08:13 +0000 From: Dmitry Kozlyuk <dkozlyuk@nvidia.com> To: <dev@dpdk.org> Date: Wed, 18 Aug 2021 12:07:51 +0300 Message-ID: <20210818090755.2419483-1-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To DRHQMAIL107.nvidia.com (10.27.9.16) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4120b78d-57b3-49cb-a9f7-08d96227b687 X-MS-TrafficTypeDiagnostic: MN2PR12MB3325: X-Microsoft-Antispam-PRVS: <MN2PR12MB3325545D783967E88F27B295B9FF9@MN2PR12MB3325.namprd12.prod.outlook.com> X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: XPv0jtyAYj0lnbB3JhMngs7hBdbzLAeD01pJRWGa4ZAjeWGUaD0Lcg3su4ttnTHSnJtlrEWUORCp/ibmYoR73mhoFx3r46oMCnUS4hlxzmOaUc66SBL66nLIVPI3GAn+reDZpBPBcrlYe7vaTq3dUDO2MWGxTKGXIoHlJEMYqm4lF9XgfiY9Y4Nw0KkrKyZ9SPXOQC7IaLa5t5g2U3YupfYNTDo04+yX2ZlpcNAcuTQO6+tj37xwShdrOuB299BBix/Q5RPBkfR6H5c9sdl+Tjk1BmfWg4e3IeGq21758MeYmVSTAC5yK8IkSqz7btLlHlOcefrD/VbhUcfPgoDilq87xGPV2LcYBBRWCUQae+eunMv4Pgk/hO4ZVWp0RdQNbG+jwUyyr9vet2mg1JM1ulbpcewBp+1yrrzziB8iaacvsx+lEIYR0bro6nZAHheGlS4f03ZWN0cJitCk3mV2xXp4BQytvT/68YpcIREGAC7wFp3XNsvqqwUlV5/SKbbfyoz5t7/CYCf1nnA7NeknsvJYgJZCf3TtpO65Kb1XSXzUD/6qBxiFUpPF9goDpgeqbH7nbbnriJQTeXjOcvgtqJ1Gz04+ormdv5uzFJoy5zpPtRCj/GBnm2fUorqZDCJxcYyPDqKqmxv4wZ33mXWr2dSAslpNTw9HckQEaCkYdanX895tWoh5jQmGkaav+wdT/w3b73VXnifvBwQ4kvYDardNG2sK1uEims7tzqqCbbnTvcPAofZsL+VfCQRPiX8KmgR81+UtV7IteFPaWAvdyiDl4l3xPmZLiOxJde41EMA= X-Forefront-Antispam-Report: CIP:216.228.112.36; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid05.nvidia.com; CAT:NONE; SFS:(4636009)(346002)(396003)(136003)(376002)(39860400002)(36840700001)(46966006)(966005)(5660300002)(8676002)(47076005)(2906002)(45080400002)(8936002)(478600001)(316002)(1076003)(6916009)(7696005)(70586007)(36860700001)(336012)(86362001)(186003)(55016002)(70206006)(6666004)(2616005)(83380400001)(82740400003)(7636003)(356005)(36756003)(426003)(82310400003)(6286002)(16526019)(26005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Aug 2021 09:08:15.4137 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4120b78d-57b3-49cb-a9f7-08d96227b687 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.36]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT006.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3325 Subject: [dpdk-dev] [PATCH 0/4] net/mlx5: implicit mempool registration X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
Series |
net/mlx5: implicit mempool registration
|
|
Message
Dmitry Kozlyuk
Aug. 18, 2021, 9:07 a.m. UTC
MLX5 hardware has its internal IOMMU where PMD registers the memory. On the data path, PMD translates VA into a key consumed by the device IOMMU. It is impractical for the PMD to register all allocated memory because of increased lookup cost both in HW and SW. Most often mbuf memory comes from mempools, so if PMD tracks them, it can almost always have mbuf memory registered before an mbuf hits the PMD. This patchset adds such tracking in the PMD and internal API to support it. Please see [1] for a more thorough explanation of the patch 2/4 and how it can be useful outside of the MLX5 PMD. [1]: http://inbox.dpdk.org/dev/CH0PR12MB509112FADB778AB28AF3771DB9F99@CH0PR12MB5091.namprd12.prod.outlook.com/ Dmitry Kozlyuk (4): mempool: add event callbacks mempool: add non-IO flag common/mlx5: add mempool registration facilities net/mlx5: support mempool registration doc/guides/nics/mlx5.rst | 11 + doc/guides/rel_notes/release_21_11.rst | 9 + drivers/common/mlx5/mlx5_common_mp.c | 50 +++ drivers/common/mlx5/mlx5_common_mp.h | 14 + drivers/common/mlx5/mlx5_common_mr.c | 564 +++++++++++++++++++++++++ drivers/common/mlx5/mlx5_common_mr.h | 17 + drivers/common/mlx5/version.map | 5 + drivers/net/mlx5/linux/mlx5_mp_os.c | 44 ++ drivers/net/mlx5/linux/mlx5_os.c | 4 +- drivers/net/mlx5/linux/mlx5_os.h | 2 + drivers/net/mlx5/mlx5.c | 128 ++++++ drivers/net/mlx5/mlx5.h | 13 + drivers/net/mlx5/mlx5_mr.c | 27 ++ drivers/net/mlx5/mlx5_trigger.c | 10 +- lib/mempool/rte_mempool.c | 153 ++++++- lib/mempool/rte_mempool.h | 60 +++ lib/mempool/version.map | 8 + 17 files changed, 1110 insertions(+), 9 deletions(-)
Comments
From: Dmitry Kozlyuk <dkozlyuk@oss.nvidia.com>
MLX5 hardware has its internal IOMMU where PMD registers the memory.
On the data path, PMD translates VA into a key consumed by the device
IOMMU. It is impractical for the PMD to register all allocated memory
because of increased lookup cost both in HW and SW. Most often mbuf
memory comes from mempools, so if PMD tracks them, it can almost always
have mbuf memory registered before an mbuf hits the PMD. This patchset
adds such tracking in the PMD and internal API to support it.
Please see [1] for a more thorough explanation of the patch 2/4
and how it can be useful outside of the MLX5 PMD.
[1]: http://inbox.dpdk.org/dev/CH0PR12MB509112FADB778AB28AF3771DB9F99@CH0PR12MB5091.namprd12.prod.outlook.com/
v2 (internal review and testing):
1. Change tracked mempool event from being created (CREATE) to being
fully populated (READY), which is the state PMD is interested in.
2. Unit test the new mempool callback API.
3. Remove bogus "error" messages in normal conditions.
4. Fixes in PMD.
Dmitry Kozlyuk (4):
mempool: add event callbacks
mempool: add non-IO flag
common/mlx5: add mempool registration facilities
net/mlx5: support mempool registration
app/test/test_mempool.c | 75 ++++
doc/guides/nics/mlx5.rst | 11 +
doc/guides/rel_notes/release_21_11.rst | 9 +
drivers/common/mlx5/mlx5_common_mp.c | 50 +++
drivers/common/mlx5/mlx5_common_mp.h | 14 +
drivers/common/mlx5/mlx5_common_mr.c | 580 +++++++++++++++++++++++++
drivers/common/mlx5/mlx5_common_mr.h | 17 +
drivers/common/mlx5/version.map | 5 +
drivers/net/mlx5/linux/mlx5_mp_os.c | 44 ++
drivers/net/mlx5/linux/mlx5_os.c | 4 +-
drivers/net/mlx5/mlx5.c | 152 +++++++
drivers/net/mlx5/mlx5.h | 10 +
drivers/net/mlx5/mlx5_mr.c | 120 ++---
drivers/net/mlx5/mlx5_mr.h | 2 -
drivers/net/mlx5/mlx5_rx.h | 21 +-
drivers/net/mlx5/mlx5_rxq.c | 13 +
drivers/net/mlx5/mlx5_trigger.c | 77 +++-
drivers/net/mlx5/windows/mlx5_os.c | 1 +
lib/mempool/rte_mempool.c | 143 +++++-
lib/mempool/rte_mempool.h | 60 +++
lib/mempool/version.map | 8 +
21 files changed, 1297 insertions(+), 119 deletions(-)