From patchwork Mon Mar 4 12:29:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Brandes, Shai" X-Patchwork-Id: 137909 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2266543B9B; Mon, 4 Mar 2024 13:34:08 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6E8BE42E67; Mon, 4 Mar 2024 13:31:01 +0100 (CET) Received: from smtp-fw-80008.amazon.com (smtp-fw-80008.amazon.com [99.78.197.219]) by mails.dpdk.org (Postfix) with ESMTP id 485FC40ED0 for ; Mon, 4 Mar 2024 13:30:58 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1709555458; x=1741091458; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=tWOlgDV0YhYGZSQXEdxty05V+z6RxDwv9vnnB9hQBTk=; b=v05sT7WmPySLqcL1vRSF7jK0nT/nuhPSjKtxTf0qlkK6VyyNeM0+8EuX KkQsCQPsrAKcp7wGisZXdF+Ks4YCkwH9eoZ1M0HaVdVih6pTMBcTaehxm 0d6uSIs17dAplenEAFTgntxzvkv4kZLrMiJBD3hIkVQlkUALdLpk5IXNQ c=; X-IronPort-AV: E=Sophos;i="6.06,203,1705363200"; d="scan'208";a="70480928" Received: from pdx4-co-svc-p1-lb2-vlan3.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.25.36.214]) by smtp-border-fw-80008.pdx80.corp.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Mar 2024 12:30:57 +0000 Received: from EX19MTAEUC002.ant.amazon.com [10.0.10.100:29716] by smtpin.naws.eu-west-1.prod.farcaster.email.amazon.dev [10.0.41.19:2525] with esmtp (Farcaster) id dab80f79-0e6a-4835-83a7-67c799ec16a3; Mon, 4 Mar 2024 12:30:56 +0000 (UTC) X-Farcaster-Flow-ID: dab80f79-0e6a-4835-83a7-67c799ec16a3 Received: from EX19D007EUA002.ant.amazon.com (10.252.50.68) by EX19MTAEUC002.ant.amazon.com (10.252.51.245) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.28; Mon, 4 Mar 2024 12:30:56 +0000 Received: from EX19MTAUWA001.ant.amazon.com (10.250.64.204) by EX19D007EUA002.ant.amazon.com (10.252.50.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1258.28; Mon, 4 Mar 2024 12:30:55 +0000 Received: from HFA15-CG15235BS.amazon.com (10.1.212.49) by mail-relay.amazon.com (10.250.64.204) with Microsoft SMTP Server id 15.2.1258.28 via Frontend Transport; Mon, 4 Mar 2024 12:30:54 +0000 From: To: CC: , Shai Brandes Subject: [PATCH v2 31/33] net/ena: support max large llq depth from the device Date: Mon, 4 Mar 2024 14:29:40 +0200 Message-ID: <20240304122942.3496-32-shaibran@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240304122942.3496-1-shaibran@amazon.com> References: <20240304122942.3496-1-shaibran@amazon.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Shai Brandes Selected AWS instances from later generations enable large LLQ by default, allowing the transmission of packets with headers exceeding 96 bytes. Due to the overall ENA memory BAR size limitation, large LLQ has the side effect of halving the maximum number of LLQ entries (from 1024 to 512). ENA-Express, powered by AWS Scalable Reliable Datagram (SRD) technology, requires Tx queue with 1024 entries. Selected AWS instances from upcoming generations will have double the size of the ENA memory BAR, enabling ENA-Express to work with a large LLQ of 1024 entries. The initial default large LLQ size will remain 512. Signed-off-by: Shai Brandes Reviewed-by: Amit Bernstein --- doc/guides/rel_notes/release_24_03.rst | 2 + drivers/net/ena/ena_ethdev.c | 38 ++++++++++++------- drivers/net/ena/hal/ena_defs/ena_admin_defs.h | 4 +- 3 files changed, 29 insertions(+), 15 deletions(-) diff --git a/doc/guides/rel_notes/release_24_03.rst b/doc/guides/rel_notes/release_24_03.rst index 2a22bb07ed..9823616eeb 100644 --- a/doc/guides/rel_notes/release_24_03.rst +++ b/doc/guides/rel_notes/release_24_03.rst @@ -107,6 +107,8 @@ New Features * Added support for sub-optimal configuration notifications from the device. * Restructured fast release of mbufs when RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE optimization is enabled. * Replaced `enable_llq` and `large_llq_hdr` devargs with a new devarg `llq_policy`. + * Added support for LLQ header size recommendation from the device. + * Allowed large LLQ with 1024 entries when the device supports enlarged memory BAR. * **Updated Atomic Rules' Arkville driver.** diff --git a/drivers/net/ena/ena_ethdev.c b/drivers/net/ena/ena_ethdev.c index d73e321d0f..43693ee2ee 100644 --- a/drivers/net/ena/ena_ethdev.c +++ b/drivers/net/ena/ena_ethdev.c @@ -42,6 +42,8 @@ #define DECIMAL_BASE 10 +#define MAX_WIDE_LLQ_DEPTH_UNSUPPORTED 0 + /* * We should try to keep ENA_CLEANUP_BUF_SIZE lower than * RTE_MEMPOOL_CACHE_MAX_SIZE, so we can fit this in mempool local cache. @@ -1071,7 +1073,7 @@ static int ena_calc_io_queue_size(struct ena_calc_queue_size_ctx *ctx, bool use_large_llq_hdr) { - struct ena_admin_feature_llq_desc *llq = &ctx->get_feat_ctx->llq; + struct ena_admin_feature_llq_desc *dev = &ctx->get_feat_ctx->llq; struct ena_com_dev *ena_dev = ctx->ena_dev; uint32_t max_tx_queue_size; uint32_t max_rx_queue_size; @@ -1086,7 +1088,7 @@ ena_calc_io_queue_size(struct ena_calc_queue_size_ctx *ctx, if (ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_DEV) { max_tx_queue_size = RTE_MIN(max_tx_queue_size, - llq->max_llq_depth); + dev->max_llq_depth); } else { max_tx_queue_size = RTE_MIN(max_tx_queue_size, max_queue_ext->max_tx_sq_depth); @@ -1106,7 +1108,7 @@ ena_calc_io_queue_size(struct ena_calc_queue_size_ctx *ctx, if (ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_DEV) { max_tx_queue_size = RTE_MIN(max_tx_queue_size, - llq->max_llq_depth); + dev->max_llq_depth); } else { max_tx_queue_size = RTE_MIN(max_tx_queue_size, max_queues->max_sq_depth); @@ -1122,18 +1124,28 @@ ena_calc_io_queue_size(struct ena_calc_queue_size_ctx *ctx, max_rx_queue_size = rte_align32prevpow2(max_rx_queue_size); max_tx_queue_size = rte_align32prevpow2(max_tx_queue_size); - if (use_large_llq_hdr) { - if ((llq->entry_size_ctrl_supported & - ENA_ADMIN_LIST_ENTRY_SIZE_256B) && - (ena_dev->tx_mem_queue_type == - ENA_ADMIN_PLACEMENT_POLICY_DEV)) { - max_tx_queue_size /= 2; - PMD_INIT_LOG(INFO, - "Forcing large headers and decreasing maximum Tx queue size to %d\n", + if (ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_DEV && use_large_llq_hdr) { + /* intersection between driver configuration and device capabilities */ + if (dev->entry_size_ctrl_supported & ENA_ADMIN_LIST_ENTRY_SIZE_256B) { + if (dev->max_wide_llq_depth == MAX_WIDE_LLQ_DEPTH_UNSUPPORTED) { + /* Devices that do not support the double-sized ENA memory BAR will + * report max_wide_llq_depth as 0. In such case, driver halves the + * queue depth when working in large llq policy. + */ + max_tx_queue_size >>= 1; + PMD_INIT_LOG(INFO, + "large LLQ policy requires limiting Tx queue size to %u entries\n", max_tx_queue_size); + } else if (dev->max_wide_llq_depth < max_tx_queue_size) { + /* In case the queue depth that the driver calculated exceeds + * the maximal value that the device allows, it will be limited + * to that maximal value + */ + max_tx_queue_size = dev->max_wide_llq_depth; + } } else { - PMD_INIT_LOG(ERR, - "Forcing large headers failed: LLQ is disabled or device does not support large headers\n"); + PMD_INIT_LOG(INFO, + "Forcing large LLQ headers failed since device lacks this support\n"); } } diff --git a/drivers/net/ena/hal/ena_defs/ena_admin_defs.h b/drivers/net/ena/hal/ena_defs/ena_admin_defs.h index 2adce75ed3..cff6451c96 100644 --- a/drivers/net/ena/hal/ena_defs/ena_admin_defs.h +++ b/drivers/net/ena/hal/ena_defs/ena_admin_defs.h @@ -696,8 +696,8 @@ struct ena_admin_feature_llq_desc { */ uint8_t entry_size_recommended; - /* reserved */ - uint8_t reserved1[2]; + /* max depth of wide llq, or 0 for N/A */ + uint16_t max_wide_llq_depth; /* accelerated low latency queues requirement. driver needs to * support those requirements in order to use accelerated llq