From patchwork Thu Oct 6 11:00:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dariusz Sosnowski X-Patchwork-Id: 117449 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 73A97A00C2; Thu, 6 Oct 2022 13:02:09 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 562C442C0A; Thu, 6 Oct 2022 13:02:05 +0200 (CEST) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by mails.dpdk.org (Postfix) with ESMTP id 518FA42BF0 for ; Thu, 6 Oct 2022 13:02:03 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mdTplIKfEDpASufUtQLeLsFnp5LdIplq5TMDZpnZz35+S8IhPx3ck7wadRfrLW/Fj0aQV85LBkv9hjogWjmVj2oO5oFrpWFat4lmzXCH9IktJP77EMZwK7edVEzlGIlTlBXup/t+mk30jEE/aTrvDcFeCtPfn/xgwXvqHOu79RaXhOeOSsORqxEPiwupr19dGC7B0H3OdzJ4cFH7G9CnMJnnKHGD8NmSMjWIWNTSVI3Bi0Egjxu9NaJAQ11KwIQ8UEku2R55+PcSJ2CNLUrP91tT305e8TP+4vdThfyOTxDtosZuitvNghRYsUZGTp34Y0cvqC7G7xI+h2OfvJ8WgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bn/MyawROKwK8H7CmCJ0ZtXf076IgfX5kKhbnYTtaq0=; b=XZUqYA4qCu5ArrKXHLfSLf68otuNu7XwKzvyyPIGWff8QK9a6c4SOPNboOuSSrimJTRcPDDMS2YfhsoyC4A637rDjtiSDFS+l5a8f6fLY6C7wIcPq9J83qK7torjz4JS40HrBFXzUEcVYJYOF+lIlex1u44Png7onPtMjxBM2leBkmh7O2nRrpKf/0fs1nRBHlm5CsFNGHzDplWvoeLLQZ0oWTNUZwCRdZuU37OIO4MUiA18nnUw6U9tRI3L/lg0w6YDN9AD2Ypm8ewTPMGMBwoe5tOwwPk9XCqRtvQw6qnG5Le8SusNLbh03iD1zLsZYNxeOpFY30SHjlj13IYkbg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bn/MyawROKwK8H7CmCJ0ZtXf076IgfX5kKhbnYTtaq0=; b=tpxuKY91F1tooCbISACA5Kwr6dZb1OgTWFOrpjQIRxpVw5Oyu8pjPtDbn2McwO69rYHI6tvS0HFcamRxDYVLiiUWcwkejvI8nPUFiXSc352ivj7l7iYu2V9jxdYVwSGK3Eo3N6xrsPVhqvWABK5iKk6AovCvtusm+EnLI9O2C4toZqmWha6tIr01FcahAgaiy/2LFGX2Nt9D47PLCCZFCZiPB44SxwEVDatVYH+zOnx3u+7GZuQzrfaCkTV3GgnZzyfvC8d/eRMd/QP+xYQ/urGm/rOdMSRvyAjr8GIx5U8PlyZVlNWemSrql7wVfJbHjZB8oDbQ52/3OLxsnkkaiA== Received: from BN1PR13CA0002.namprd13.prod.outlook.com (2603:10b6:408:e2::7) by BY5PR12MB4871.namprd12.prod.outlook.com (2603:10b6:a03:1d1::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.23; Thu, 6 Oct 2022 11:02:01 +0000 Received: from BN8NAM11FT011.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e2:cafe::ca) by BN1PR13CA0002.outlook.office365.com (2603:10b6:408:e2::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5709.12 via Frontend Transport; Thu, 6 Oct 2022 11:02:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN8NAM11FT011.mail.protection.outlook.com (10.13.176.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5709.10 via Frontend Transport; Thu, 6 Oct 2022 11:02:01 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Thu, 6 Oct 2022 04:01:50 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Thu, 6 Oct 2022 04:01:48 -0700 From: Dariusz Sosnowski To: Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko CC: Subject: [PATCH v2 1/8] ethdev: introduce hairpin memory capabilities Date: Thu, 6 Oct 2022 11:00:58 +0000 Message-ID: <20221006110105.2986966-2-dsosnowski@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221006110105.2986966-1-dsosnowski@nvidia.com> References: <20221006110105.2986966-1-dsosnowski@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT011:EE_|BY5PR12MB4871:EE_ X-MS-Office365-Filtering-Correlation-Id: adc2a582-1e92-4f6d-ec90-08daa78a31f8 X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0SG4v7UN0zTfdiMVYrKZa5AYaVAz/UwsWP6Mj8FEGV2TsEA7guek9UklchJ/ij4o4sr5s6bo/0vxiOuxdzzQC4A0ZdK86VR0GV2zNtjuSE8yJomh3os3/0AjJ5T/J4NTkv8LEz9BaWpvrkmZEl/34Ou5Uuy4NB60tJM2Yr2o6VZUGwKEXau4gCH/bxBtcwpC2JxXPJTWUT14JSdCutYyM5BpvNJ2Wb+b9J0Nkl6tf71dS3bDWUZiuv5+TpZRrb9xbqG0agydcYWaKC8v3igkCarMjZaYdz/aq48XGBcdGqYl0QEaPdwdQqOZKmL14Wr/YC7hac2glKXGN0280q64h7ngTtiAwo3naZ4Yi6yKkPCKG0xkILFMC7x0YZinnvtFSx570lp3O5VZc0wvx42CyKM1Nj5Z/jrHDOPUh4LVHzgkRhZxtZl8VRvSEP0fjtVF1YYlXOvO9Z7ZIWn3AO6bjyRjkNhjrXYoGNwCYhR5/ottzAbFgxQKgsl3+gJFWj9tm5H/Ct6HZW/JmTj49jj5UNDvafZiljhU8XKkI6FYno8OpZt1cWSqnviop7BF4K+yjp1KqbxfxpiP1D1KXJx81acdeXNj9JUV/pr8KMBByIbVYiTxM3IwFjQ0J0Nke5WjLpk/99I+txs7v68evXo2G2BtPj4fL5cBPz8rEO3y65sVtjqfQ8QPx8u1HQUXLqpme5Z/Ua5cr6D5yia7E09Ms/Yrau46L9fPZeCfSfhkjTFwDF8gfdbFavye9EaG10hkKKB+e+prVJjbFSmDniEAQQ== X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230022)(4636009)(136003)(346002)(39860400002)(376002)(396003)(451199015)(36840700001)(46966006)(40470700004)(1076003)(186003)(2616005)(16526019)(356005)(2906002)(82310400005)(316002)(70206006)(6666004)(40460700003)(70586007)(4326008)(110136005)(8676002)(55016003)(86362001)(478600001)(41300700001)(5660300002)(6286002)(26005)(336012)(426003)(7636003)(8936002)(7696005)(40480700001)(47076005)(36756003)(82740400003)(83380400001)(36860700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2022 11:02:01.0037 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: adc2a582-1e92-4f6d-ec90-08daa78a31f8 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT011.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4871 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Before this patch, implementation details and configuration of hairpin queues were decided internally by the PMD. Applications had no control over the configuration of Rx and Tx hairpin queues, despite number of descriptors, explicit Tx flow mode and disabling automatic binding. This patch addresses that by adding: - Hairpin queue capabilities reported by PMDs. - New configuration options for Rx and Tx hairpin queues. Main goal of this patch is to allow applications to provide configuration hints regarding placement of hairpin queues. These hints specify whether buffers of hairpin queues should be placed in host memory or in dedicated device memory. Different memory options may have different performance characteristics and hairpin configuration should be fine-tuned to the specific application and use case. This patch introduces new hairpin queue configuration options through rte_eth_hairpin_conf struct, allowing to tune Rx and Tx hairpin queues memory configuration. Hairpin configuration is extended with the following fields: - use_locked_device_memory - If set, PMD will use specialized on-device memory to store RX or TX hairpin queue data. - use_rte_memory - If set, PMD will use DPDK-managed memory to store RX or TX hairpin queue data. - force_memory - If set, PMD will be forced to use provided memory settings. If no appropriate resources are available, then device start will fail. If unset and no resources are available, PMD will fallback to using default type of resource for given queue. If application chooses to use PMD default memory configuration, all of these flags should remain unset. Hairpin capabilities are also extended, to allow verification of support of given hairpin memory configurations. Struct rte_eth_hairpin_cap is extended with two additional fields of type rte_eth_hairpin_queue_cap: - rx_cap - memory capabilities of hairpin RX queues. - tx_cap - memory capabilities of hairpin TX queues. Struct rte_eth_hairpin_queue_cap exposes whether given queue type supports use_locked_device_memory and use_rte_memory flags. Signed-off-by: Dariusz Sosnowski --- doc/guides/rel_notes/release_22_11.rst | 10 ++++ lib/ethdev/rte_ethdev.c | 44 +++++++++++++++++ lib/ethdev/rte_ethdev.h | 68 +++++++++++++++++++++++++- 3 files changed, 120 insertions(+), 2 deletions(-) diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst index ac67e7e710..e5c48c6b18 100644 --- a/doc/guides/rel_notes/release_22_11.rst +++ b/doc/guides/rel_notes/release_22_11.rst @@ -66,6 +66,16 @@ New Features Added new function ``rte_flow_async_action_handle_query()``, to query the action asynchronously. +* **Added hairpin memory configurations options in ethdev API.** + + Added new configuration flags for hairpin queues in ``rte_eth_hairpin_conf``: + + * ``use_locked_device_memory`` + * ``use_rte_memory`` + * ``force_memory`` + + Each flag has a corresponding capability flag in ``rte_eth_hairpin_queue_cap`` struct. + * **Updated Intel iavf driver.** * Added flow subscription support. diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c index 2821770e2d..bece83eb91 100644 --- a/lib/ethdev/rte_ethdev.c +++ b/lib/ethdev/rte_ethdev.c @@ -1961,6 +1961,28 @@ rte_eth_rx_hairpin_queue_setup(uint16_t port_id, uint16_t rx_queue_id, conf->peer_count, cap.max_rx_2_tx); return -EINVAL; } + if (conf->use_locked_device_memory && !cap.rx_cap.locked_device_memory) { + RTE_ETHDEV_LOG(ERR, + "Attempt to use locked device memory for Rx queue, which is not supported"); + return -EINVAL; + } + if (conf->use_rte_memory && !cap.rx_cap.rte_memory) { + RTE_ETHDEV_LOG(ERR, + "Attempt to use DPDK memory for Rx queue, which is not supported"); + return -EINVAL; + } + if (conf->use_locked_device_memory && conf->use_rte_memory) { + RTE_ETHDEV_LOG(ERR, + "Attempt to use mutually exclusive memory settings for Rx queue"); + return -EINVAL; + } + if (conf->force_memory && + !conf->use_locked_device_memory && + !conf->use_rte_memory) { + RTE_ETHDEV_LOG(ERR, + "Attempt to force Rx queue memory settings, but none is set"); + return -EINVAL; + } if (conf->peer_count == 0) { RTE_ETHDEV_LOG(ERR, "Invalid value for number of peers for Rx queue(=%u), should be: > 0", @@ -2128,6 +2150,28 @@ rte_eth_tx_hairpin_queue_setup(uint16_t port_id, uint16_t tx_queue_id, conf->peer_count, cap.max_tx_2_rx); return -EINVAL; } + if (conf->use_locked_device_memory && !cap.tx_cap.locked_device_memory) { + RTE_ETHDEV_LOG(ERR, + "Attempt to use locked device memory for Tx queue, which is not supported"); + return -EINVAL; + } + if (conf->use_rte_memory && !cap.tx_cap.rte_memory) { + RTE_ETHDEV_LOG(ERR, + "Attempt to use DPDK memory for Tx queue, which is not supported"); + return -EINVAL; + } + if (conf->use_locked_device_memory && conf->use_rte_memory) { + RTE_ETHDEV_LOG(ERR, + "Attempt to use mutually exclusive memory settings for Tx queue"); + return -EINVAL; + } + if (conf->force_memory && + !conf->use_locked_device_memory && + !conf->use_rte_memory) { + RTE_ETHDEV_LOG(ERR, + "Attempt to force Tx queue memory settings, but none is set"); + return -EINVAL; + } if (conf->peer_count == 0) { RTE_ETHDEV_LOG(ERR, "Invalid value for number of peers for Tx queue(=%u), should be: > 0", diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h index a21f58b9cd..eab931d3b2 100644 --- a/lib/ethdev/rte_ethdev.h +++ b/lib/ethdev/rte_ethdev.h @@ -1092,6 +1092,28 @@ struct rte_eth_txconf { void *reserved_ptrs[2]; /**< Reserved for future fields */ }; +/** + * @warning + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice + * + * A structure used to return the Tx or Rx hairpin queue capabilities that are supported. + */ +struct rte_eth_hairpin_queue_cap { + /** + * When set, PMD supports placing descriptors and/or data buffers + * in dedicated device memory. + */ + uint32_t locked_device_memory:1; + + /** + * When set, PMD supports placing descriptors and/or data buffers + * in host memory managed by DPDK. + */ + uint32_t rte_memory:1; + + uint32_t reserved:30; /**< Reserved for future fields */ +}; + /** * @warning * @b EXPERIMENTAL: this API may change, or be removed, without prior notice @@ -1106,6 +1128,8 @@ struct rte_eth_hairpin_cap { /** Max number of Tx queues to be connected to one Rx queue. */ uint16_t max_tx_2_rx; uint16_t max_nb_desc; /**< The max num of descriptors. */ + struct rte_eth_hairpin_queue_cap rx_cap; /**< Rx hairpin queue capabilities. */ + struct rte_eth_hairpin_queue_cap tx_cap; /**< Tx hairpin queue capabilities. */ }; #define RTE_ETH_MAX_HAIRPIN_PEERS 32 @@ -1149,11 +1173,51 @@ struct rte_eth_hairpin_conf { * function after all the queues are set up properly and the ports are * started. Also, the hairpin unbind function should be called * accordingly before stopping a port that with hairpin configured. - * - When clear, the PMD will try to enable the hairpin with the queues + * - When cleared, the PMD will try to enable the hairpin with the queues * configured automatically during port start. */ uint32_t manual_bind:1; - uint32_t reserved:14; /**< Reserved bits. */ + + /** + * Use locked device memory as a backing storage. + * + * - When set, PMD will attempt place descriptors and/or data buffers + * in dedicated device memory. + * - When cleared, PMD will use default memory type as a backing storage. + * Please refer to PMD documentation for details. + * + * API user should check if PMD supports this configuration flag using + * @see rte_eth_dev_hairpin_capability_get. + */ + uint32_t use_locked_device_memory:1; + + /** + * Use DPDK memory as backing storage. + * + * - When set, PMD will attempt place descriptors and/or data buffers + * in host memory managed by DPDK. + * - When cleared, PMD will use default memory type as a backing storage. + * Please refer to PMD documentation for details. + * + * API user should check if PMD supports this configuration flag using + * @see rte_eth_dev_hairpin_capability_get. + */ + uint32_t use_rte_memory:1; + + /** + * Force usage of hairpin memory configuration. + * + * - When set, PMD will attempt to use specified memory settings. + * If resource allocation fails, then hairpin queue allocation + * will result in an error. + * - When clear, PMD will attempt to use specified memory settings. + * If resource allocation fails, then PMD will retry + * allocation with default configuration. + */ + uint32_t force_memory:1; + + uint32_t reserved:11; /**< Reserved bits. */ + struct rte_eth_hairpin_peer peers[RTE_ETH_MAX_HAIRPIN_PEERS]; };