From patchwork Mon Sep 7 19:32:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Kumar X-Patchwork-Id: 76738 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 976A4A04B1; Mon, 7 Sep 2020 21:32:48 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 307CF1C0CC; Mon, 7 Sep 2020 21:32:46 +0200 (CEST) Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) by dpdk.org (Postfix) with ESMTP id C6FF51C0C9 for ; Mon, 7 Sep 2020 21:32:43 +0200 (CEST) Received: by mail-pg1-f196.google.com with SMTP id j34so3052070pgi.7 for ; Mon, 07 Sep 2020 12:32:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vvdntech-in.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=UzW+fRIuVd5z3jtlfdy9A11PL3YCP5z45RPp0iao0fo=; b=LvXGmjGAOAA1t5/inUgQP0eJeEmR6IiOW690h4hf0nv0viYZDSaiu5SxqRPIkMoxxu tHy8u33tPLOC5HeXKRzuD+sMdU0v4pPgA35XswSMJsM/UASvPoEWBYBnEN0MjQqUB16w YzB4O2cwJ8f1dJc7oQTxeLFqpcPDsEALA0nPpTBR3/tK0y8fJ9FQI7mOjxCqBxZQEtD8 bp5f9UHXzNhgdg219VrhXRcZdk34GCxK9ktp6UHiUYKc8gBGvJBwYBMmir4Hhc/kfbio lUVs6lXr3lK3KZzN8SQoFLWbI4R01f5FwhBSBWJN7hfRRQervhBKmnvbknah3NSAqM0Y fmgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=UzW+fRIuVd5z3jtlfdy9A11PL3YCP5z45RPp0iao0fo=; b=jWQr3FiQ2Q3OCiY3wCZeKfj/+1QOipJuvAlNra4GILfFjFhA030n4bBPBTsamc7jgV bR22xw5kgQG3PpLEJ1mSUpwpFffRHLyLUin0yP6sYp+JFBO0+MrmbCyM6I7398bPrzBE tqxOrW8uEu76p6LMTYDK3FDkVJFl0UsXFuNr7qSShxkiuPUUbYIw5PLF/p4c6J0EUBss c+tBSTToCbG7tEV2Ge02oTWV2WwJHowHDimp6rZSGzYmGLfJsaLrx8ZmGRLTa2PEBuJJ Ek1a0pgFS7gD1+WCw/sHGzF/V2WDSLKR5h+pauNDpa6ZMKRF5N5nJwsgaDkKtwTMVGGM QVYg== X-Gm-Message-State: AOAM532SF+SDxpUuGmxnD8bIqfz78zNe2kk/CDHDHo2xMuvdQJx5cRhj Q3Og2SMKOev3R3RxNsPgvDi023gLs0SfB97QPkjMakQMwiFsyTErqZL5pIncWPvWEso0obc5/+t CaoSt X-Google-Smtp-Source: ABdhPJxEaK/ktjWmUZ4JQQN9QgmHgvHbsqyEteNBJzWepX7bZ6tellYp5diaU3m0XpM0sjA4zZsfOg== X-Received: by 2002:a63:4b47:: with SMTP id k7mr1906036pgl.437.1599507162490; Mon, 07 Sep 2020 12:32:42 -0700 (PDT) Received: from localhost.localdomain ([103.251.53.230]) by smtp.gmail.com with ESMTPSA id p4sm3021495pju.29.2020.09.07.12.32.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2020 12:32:41 -0700 (PDT) From: Aman Kumar To: dev@dpdk.org Cc: rasland@mellanox.com, keesang.song@amd.com, aman.kumar@vvdntech.in Date: Tue, 8 Sep 2020 01:02:13 +0530 Message-Id: <20200907193214.38426-2-aman.kumar@vvdntech.in> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200907193214.38426-1-aman.kumar@vvdntech.in> References: <20200907193214.38426-1-aman.kumar@vvdntech.in> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 2/3] net/mlx5: add non temporal store for WQE fields X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" add non temporal store for few WQE fields to optimize data path. This can be enable by making CONFG_RTE_LIBRTE_MLX5_NT_STORE=y in dpdk config. Signed-off-by: Aman Kumar --- config/common_base | 1 + drivers/net/mlx5/mlx5.c | 17 +++++++++++++++++ drivers/net/mlx5/mlx5.h | 4 ++++ drivers/net/mlx5/mlx5_rxq.c | 3 +++ drivers/net/mlx5/mlx5_rxtx.c | 20 +++++++++++++++++--- drivers/net/mlx5/mlx5_rxtx.h | 6 ++++++ drivers/net/mlx5/mlx5_rxtx_vec.h | 28 +++++++++++++++++++++++----- drivers/net/mlx5/mlx5_txq.c | 3 +++ 8 files changed, 74 insertions(+), 8 deletions(-) diff --git a/config/common_base b/config/common_base index 1476cf334..bbe74833b 100644 --- a/config/common_base +++ b/config/common_base @@ -372,6 +372,7 @@ CONFIG_RTE_LIBRTE_MLX4_DEBUG=n # CONFIG_RTE_LIBRTE_MLX5_PMD=n CONFIG_RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY=n +CONFIG_RTE_LIBRTE_MLX5_NT_STORE=n CONFIG_RTE_LIBRTE_MLX5_DEBUG=n # diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 6eb85dfac..8e1b7df23 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -167,6 +167,13 @@ /* mprq_tstore_memcpy */ #define MLX5_MPRQ_TSTORE_MEMCPY "mprq_tstore_memcpy" #endif +#ifdef RTE_LIBRTE_MLX5_NT_STORE +/* tx_wqe_field_ntstore */ +#define MLX5_TX_WQE_FIELD_NTSTORE "tx_wqe_field_ntstore" + +/* vec_rx_wqe_field_ntstore */ +#define MLX5_VEC_RX_WQE_FIELD_NTSTORE "vec_rx_wqe_field_ntstore" +#endif /* * Device parameter to configure the total data buffer size for a single @@ -1629,6 +1636,12 @@ mlx5_args_check(const char *key, const char *val, void *opaque) #ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY } else if (strcmp(MLX5_MPRQ_TSTORE_MEMCPY, key) == 0) { config->mprq_tstore_memcpy = tmp; +#endif +#ifdef RTE_LIBRTE_MLX5_NT_STORE + } else if (strcmp(MLX5_TX_WQE_FIELD_NTSTORE, key) == 0) { + config->tx_wqe_field_ntstore = tmp; + } else if (strcmp(MLX5_VEC_RX_WQE_FIELD_NTSTORE, key) == 0) { + config->vec_rx_wqe_field_ntstore = tmp; #endif } else { DRV_LOG(WARNING, "%s: unknown parameter", key); @@ -1692,6 +1705,10 @@ mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs) MLX5_DECAP_EN, #ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY MLX5_MPRQ_TSTORE_MEMCPY, +#endif +#ifdef RTE_LIBRTE_MLX5_NT_STORE + MLX5_TX_WQE_FIELD_NTSTORE, + MLX5_VEC_RX_WQE_FIELD_NTSTORE, #endif NULL, }; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 09dc90953..4a816cb2e 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -239,6 +239,10 @@ struct mlx5_dev_config { #ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY unsigned int mprq_tstore_memcpy:1; #endif +#ifdef RTE_LIBRTE_MLX5_NT_STORE + unsigned int tx_wqe_field_ntstore:1; + unsigned int vec_rx_wqe_field_ntstore:1; +#endif }; diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index bee5c03bc..ceb33e5c5 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -2304,6 +2304,9 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, tmpl->irq = 1; #ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY tmpl->rxq.mprq_tstore_memcpy = config->mprq_tstore_memcpy; +#endif +#ifdef RTE_LIBRTE_MLX5_NT_STORE + tmpl->rxq.vec_rx_wqe_field_ntstore = config->vec_rx_wqe_field_ntstore; #endif mprq_stride_nums = config->mprq.stride_num_n ? config->mprq.stride_num_n : MLX5_MPRQ_STRIDE_NUM_N; diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index 62ade3775..6bcdc44a5 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -2318,6 +2318,9 @@ mlx5_tx_request_completion(struct mlx5_txq_data *__rte_restrict txq, { uint16_t head = txq->elts_head; unsigned int part; +#ifdef RTE_LIBRTE_MLX5_NT_STORE + register uint32_t flags; +#endif part = MLX5_TXOFF_CONFIG(INLINE) ? 0 : loc->pkts_sent - loc->pkts_copy; @@ -2331,9 +2334,20 @@ mlx5_tx_request_completion(struct mlx5_txq_data *__rte_restrict txq, txq->elts_comp = head; if (MLX5_TXOFF_CONFIG(INLINE)) txq->wqe_comp = txq->wqe_ci; - /* Request unconditional completion on last WQE. */ - last->cseg.flags = RTE_BE32(MLX5_COMP_ALWAYS << - MLX5_COMP_MODE_OFFSET); +#ifdef RTE_LIBRTE_MLX5_NT_STORE + if (txq->tx_wqe_field_ntstore) { + flags = RTE_BE32(MLX5_COMP_ALWAYS << + MLX5_COMP_MODE_OFFSET); + _mm_stream_si32(((void *)(uintptr_t)&last->cseg.flags), + flags); + } else { +#endif + /* Request unconditional completion on last WQE. */ + last->cseg.flags = RTE_BE32(MLX5_COMP_ALWAYS << + MLX5_COMP_MODE_OFFSET); +#ifdef RTE_LIBRTE_MLX5_NT_STORE + } +#endif /* Save elts_head in dedicated free on completion queue. */ #ifdef RTE_LIBRTE_MLX5_DEBUG txq->fcqs[txq->cq_pi++ & txq->cqe_m] = head | diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 72763962f..b031eff0b 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -156,6 +156,9 @@ struct mlx5_rxq_data { #ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY unsigned int mprq_tstore_memcpy:1; #endif +#ifdef RTE_LIBRTE_MLX5_NT_STORE + unsigned int vec_rx_wqe_field_ntstore:1; +#endif } __rte_cache_aligned; enum mlx5_rxq_obj_type { @@ -324,6 +327,9 @@ struct mlx5_txq_data { int32_t ts_offset; /* Timestamp field dynamic offset. */ struct mlx5_dev_ctx_shared *sh; /* Shared context. */ struct mlx5_txq_stats stats; /* TX queue counters. */ +#ifdef RTE_LIBRTE_MLX5_NT_STORE + unsigned int tx_wqe_field_ntstore:1; +#endif #ifndef RTE_ARCH_64 rte_spinlock_t *uar_lock; /* UAR access lock required for 32bit implementations */ diff --git a/drivers/net/mlx5/mlx5_rxtx_vec.h b/drivers/net/mlx5/mlx5_rxtx_vec.h index 6ddcbfb0a..62a07ef00 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec.h @@ -86,6 +86,10 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *rxq, uint16_t n) volatile struct mlx5_wqe_data_seg *wq = &((volatile struct mlx5_wqe_data_seg *)rxq->wqes)[elts_idx]; unsigned int i; +#ifdef RTE_LIBRTE_MLX5_NT_STORE + register uint64_t buf_addr2; + register uint32_t lkey_t; +#endif MLX5_ASSERT(n >= MLX5_VPMD_RXQ_RPLNSH_THRESH(q_n)); MLX5_ASSERT(n <= (uint16_t)(q_n - (rxq->rq_ci - rxq->rq_pi))); @@ -107,11 +111,25 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *rxq, uint16_t n) * impact the performance. */ buf_addr = elts[i]->buf_addr; - wq[i].addr = rte_cpu_to_be_64((uintptr_t)buf_addr + - RTE_PKTMBUF_HEADROOM); - /* If there's only one MR, no need to replace LKey in WQE. */ - if (unlikely(mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1)) - wq[i].lkey = mlx5_rx_mb2mr(rxq, elts[i]); +#ifdef RTE_LIBRTE_MLX5_NT_STORE + if (rxq->vec_rx_wqe_field_ntstore) { + buf_addr2 = (uint64_t)rte_cpu_to_be_64((uintptr_t)buf_addr + RTE_PKTMBUF_HEADROOM); + _mm_stream_si64(((void *)(uintptr_t)&wq[i].addr), buf_addr2); + /* If there's only one MR, no need to replace LKey in WQE. */ + if (unlikely(mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1)) { + lkey_t = (uint32_t)mlx5_rx_mb2mr(rxq, elts[i]); + _mm_stream_si32(((void *)(uintptr_t)&wq[i].lkey), lkey_t); + } + } else { +#endif + wq[i].addr = rte_cpu_to_be_64((uintptr_t)buf_addr + + RTE_PKTMBUF_HEADROOM); + /* If there's only one MR, no need to replace LKey in WQE. */ + if (unlikely(mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1)) + wq[i].lkey = mlx5_rx_mb2mr(rxq, elts[i]); +#ifdef RTE_LIBRTE_MLX5_NT_STORE + } +#endif } rxq->rq_ci += n; /* Prevent overflowing into consumed mbufs. */ diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 21fe16b7e..8feac4bdc 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -1562,6 +1562,9 @@ txq_set_params(struct mlx5_txq_ctrl *txq_ctrl) DEV_TX_OFFLOAD_UDP_TNL_TSO); bool vlan_inline; unsigned int temp; +#ifdef RTE_LIBRTE_MLX5_NT_STORE + txq_ctrl->txq.tx_wqe_field_ntstore = config->tx_wqe_field_ntstore; +#endif if (config->txqs_inline == MLX5_ARG_UNSET) txqs_inline =