From patchwork Mon Sep 7 19:32:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Kumar X-Patchwork-Id: 76737 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id E0EAAA04B1; Mon, 7 Sep 2020 21:32:40 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 132801B9B7; Mon, 7 Sep 2020 21:32:40 +0200 (CEST) Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by dpdk.org (Postfix) with ESMTP id 4476D1DB8 for ; Mon, 7 Sep 2020 21:32:38 +0200 (CEST) Received: by mail-pf1-f196.google.com with SMTP id k15so9052179pfc.12 for ; Mon, 07 Sep 2020 12:32:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vvdntech-in.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:mime-version; bh=rdcGPuyPapNqZ5KrnDp/WXBComp0nfwLTfk4e8MN4V0=; b=oiypdqq4BKgX6YNF5fUCElsgfeONk+Ja/JhZxMC1YNzXc7sjKHwbzGnHODSdonN7Eb r4/NEQB1ds1Zv77RjMFqkg8qY24Lpzq1yQDpw/42Pa2LOgen9/8LBJeKJoc86JCeh+tZ 9j7eEmp0IMKLnyMSjwoHNmOlBCBDgpKiPgBAVc1C8z064lPtxc/T1oUGFFzNM4SckRCy EAd+HutW+hC2cjh5CLk+8MQqH22h4hCfysnuLisqPY6qeYwioC1UTBMy7U4w+p+50oi0 +PtmtEOZWaCoR5Xi1DrQKHZyz+reCrPc0XPUL1GbB3RaUevZi/ZmrUBsWeDAgNX38jh8 iN7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version; bh=rdcGPuyPapNqZ5KrnDp/WXBComp0nfwLTfk4e8MN4V0=; b=cb+Ri59guA4FWmvyZoXSFyQNPznel5j0+e/ZJJa1kuu9jSpEu9fIf8HF5qsGzGMegL TSvgyQLA8zyd/AyW8QWt/cyN1X7rluK7BdvODkBYMnvE/uIPNneA8hIBuNbFJsh4sgc/ xPWI74iyiP2AGSSMD9nEzCfSijRhRmaB8yJgst08wQ0E5Vz8i4isyqHMFWH06fpvLIGf j/SDiQo9I6YzwThRUSBtVpKO3hc0uQ1wIglAijr2P2Wa4+z5n5n+cFTv+YCJ+Ll9W9zu Q3cxZ5aSakLKA0Kx6CnRzFqyAySQx+MpD891Kmw6jzIkLyBIgBdEUocF/GxuPTtpNq+J SFYg== X-Gm-Message-State: AOAM533lq/nK8wYcHDpEOesfQ84F5z0RXRbP6TsFBu751Bs77pwICE0t bPW+O8z/h75M34eC/Eu4eo2olEFa4WncXLcOEHRXweDIHdsOFOVAolPRWWMLA05gOMOB19ncgjJ hYYR9 X-Google-Smtp-Source: ABdhPJz4UCj+LB3ijwf2rH2Te3SOiwrjCFxFK7opuI2Sr84faCdky0C5ogI173kMkwBN2jvnMn0pAw== X-Received: by 2002:a17:902:8bc3:: with SMTP id r3mr20414072plo.238.1599507156618; Mon, 07 Sep 2020 12:32:36 -0700 (PDT) Received: from localhost.localdomain ([103.251.53.230]) by smtp.gmail.com with ESMTPSA id p4sm3021495pju.29.2020.09.07.12.32.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2020 12:32:35 -0700 (PDT) From: Aman Kumar To: dev@dpdk.org Cc: rasland@mellanox.com, keesang.song@amd.com, aman.kumar@vvdntech.in Date: Tue, 8 Sep 2020 01:02:12 +0530 Message-Id: <20200907193214.38426-1-aman.kumar@vvdntech.in> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 1/3] net/mlx5: optimize mprq memcpy X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" add non temporal load and temporal store for mprq memcpy. Enabling CONFIG_RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY=y in dpdk config will enable this optimization. Signed-off-by: Aman Kumar --- config/common_base | 1 + drivers/net/mlx5/mlx5.c | 12 ++++ drivers/net/mlx5/mlx5.h | 3 + drivers/net/mlx5/mlx5_rxq.c | 3 + drivers/net/mlx5/mlx5_rxtx.c | 17 ++++- drivers/net/mlx5/mlx5_rxtx.h | 3 + lib/librte_eal/x86/include/rte_memcpy.h | 92 +++++++++++++++++++++++++ 7 files changed, 129 insertions(+), 2 deletions(-) diff --git a/config/common_base b/config/common_base index fbf0ee70c..1476cf334 100644 --- a/config/common_base +++ b/config/common_base @@ -371,6 +371,7 @@ CONFIG_RTE_LIBRTE_MLX4_DEBUG=n # ConnectX-6 & BlueField (MLX5) PMD # CONFIG_RTE_LIBRTE_MLX5_PMD=n +CONFIG_RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY=n CONFIG_RTE_LIBRTE_MLX5_DEBUG=n # diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 1e4c695f8..6eb85dfac 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -163,6 +163,11 @@ /* Configure timeout of LRO session (in microseconds). */ #define MLX5_LRO_TIMEOUT_USEC "lro_timeout_usec" +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY +/* mprq_tstore_memcpy */ +#define MLX5_MPRQ_TSTORE_MEMCPY "mprq_tstore_memcpy" +#endif + /* * Device parameter to configure the total data buffer size for a single * hairpin queue (logarithm value). @@ -1621,6 +1626,10 @@ mlx5_args_check(const char *key, const char *val, void *opaque) config->sys_mem_en = !!tmp; } else if (strcmp(MLX5_DECAP_EN, key) == 0) { config->decap_en = !!tmp; +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + } else if (strcmp(MLX5_MPRQ_TSTORE_MEMCPY, key) == 0) { + config->mprq_tstore_memcpy = tmp; +#endif } else { DRV_LOG(WARNING, "%s: unknown parameter", key); rte_errno = EINVAL; @@ -1681,6 +1690,9 @@ mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs) MLX5_RECLAIM_MEM, MLX5_SYS_MEM_EN, MLX5_DECAP_EN, +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + MLX5_MPRQ_TSTORE_MEMCPY, +#endif NULL, }; struct rte_kvargs *kvlist; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 78d6eb728..09dc90953 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -236,6 +236,9 @@ struct mlx5_dev_config { int tx_skew; /* Tx scheduling skew between WQE and data on wire. */ struct mlx5_hca_attr hca_attr; /* HCA attributes. */ struct mlx5_lro_config lro; /* LRO configuration. */ +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + unsigned int mprq_tstore_memcpy:1; +#endif }; diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index 79eb8f8d7..bee5c03bc 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -2302,6 +2302,9 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, tmpl->socket = socket; if (dev->data->dev_conf.intr_conf.rxq) tmpl->irq = 1; +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + tmpl->rxq.mprq_tstore_memcpy = config->mprq_tstore_memcpy; +#endif mprq_stride_nums = config->mprq.stride_num_n ? config->mprq.stride_num_n : MLX5_MPRQ_STRIDE_NUM_N; mprq_stride_size = non_scatter_min_mbuf_size <= diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index 1b71e9422..62ade3775 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -1774,8 +1774,21 @@ mlx5_rx_burst_mprq(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n) rxq->mprq_repl == NULL || (hdrm_overlap > 0 && !rxq->strd_scatter_en)) { if (likely(rte_pktmbuf_tailroom(pkt) >= len)) { - rte_memcpy(rte_pktmbuf_mtod(pkt, void *), - addr, len); +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + if ((rxq->mprq_tstore_memcpy) && + (!(((uintptr_t)(rte_pktmbuf_mtod(pkt, + void *)) | + (uintptr_t)addr) & ALIGNMENT_MASK))) { + memcpy_aligned_rx_tstore_16B( + rte_pktmbuf_mtod(pkt, void *), + addr, len); + } else { +#endif + rte_memcpy(rte_pktmbuf_mtod(pkt, void *), + addr, len); +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + } +#endif DATA_LEN(pkt) = len; } else if (rxq->strd_scatter_en) { struct rte_mbuf *prev = pkt; diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index c02a007c8..72763962f 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -153,6 +153,9 @@ struct mlx5_rxq_data { uint32_t tunnel; /* Tunnel information. */ uint64_t flow_meta_mask; int32_t flow_meta_offset; +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + unsigned int mprq_tstore_memcpy:1; +#endif } __rte_cache_aligned; enum mlx5_rxq_obj_type { diff --git a/lib/librte_eal/x86/include/rte_memcpy.h b/lib/librte_eal/x86/include/rte_memcpy.h index 9c67232df..6345572a7 100644 --- a/lib/librte_eal/x86/include/rte_memcpy.h +++ b/lib/librte_eal/x86/include/rte_memcpy.h @@ -874,6 +874,98 @@ rte_memcpy(void *dst, const void *src, size_t n) return rte_memcpy_generic(dst, src, n); } +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY +static __rte_always_inline +void copy16B_ts(void *dst, void *src) +{ + __m128i var128; + + var128 = _mm_stream_load_si128((__m128i *)src); + _mm_storeu_si128((__m128i *)dst, var128); +} + +static __rte_always_inline +void copy32B_ts(void *dst, void *src) +{ + __m256i ymm0; + + ymm0 = _mm256_stream_load_si256((const __m256i *)src); + _mm256_storeu_si256((__m256i *)dst, ymm0); +} + +static __rte_always_inline +void copy64B_ts(void *dst, void *src) +{ + __m256i ymm0, ymm1; + + ymm0 = _mm256_stream_load_si256((const __m256i *)src); + ymm1 = _mm256_stream_load_si256((const __m256i *)((uint8_t *)src + 32)); + _mm256_storeu_si256((__m256i *)dst, ymm0); + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 32), ymm1); +} + +static __rte_always_inline +void copy128B_ts(void *dst, void *src) +{ + __m256i ymm0, ymm1, ymm2, ymm3; + + ymm0 = _mm256_stream_load_si256((const __m256i *)src); + ymm1 = _mm256_stream_load_si256((const __m256i *)((uint8_t *)src + 32)); + ymm2 = _mm256_stream_load_si256((const __m256i *)((uint8_t *)src + 64)); + ymm3 = _mm256_stream_load_si256((const __m256i *)((uint8_t *)src + 96)); + _mm256_storeu_si256((__m256i *)dst, ymm0); + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 32), ymm1); + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 64), ymm2); + _mm256_storeu_si256((__m256i *)((uint8_t *)dst + 96), ymm3); +} + +static __rte_always_inline +void memcpy_aligned_rx_tstore_16B(void *dst, void *src, int len) +{ + while (len >= 128) { + copy128B_ts(dst, src); + dst = (uint8_t *)dst + 128; + src = (uint8_t *)src + 128; + len -= 128; + } + while (len >= 64) { + copy64B_ts(dst, src); + dst = (uint8_t *)dst + 64; + src = (uint8_t *)src + 64; + len -= 64; + } + while (len >= 32) { + copy32B_ts(dst, src); + dst = (uint8_t *)dst + 32; + src = (uint8_t *)src + 32; + len -= 32; + } + if (len >= 16) { + copy16B_ts(dst, src); + dst = (uint8_t *)dst + 16; + src = (uint8_t *)src + 16; + len -= 16; + } + if (len >= 8) { + *(uint64_t *)dst = *(const uint64_t *)src; + dst = (uint8_t *)dst + 8; + src = (uint8_t *)src + 8; + len -= 8; + } + if (len >= 4) { + *(uint32_t *)dst = *(const uint32_t *)src; + dst = (uint8_t *)dst + 4; + src = (uint8_t *)src + 4; + len -= 4; + } + if (len != 0) { + dst = (uint8_t *)dst - (4 - len); + src = (uint8_t *)src - (4 - len); + *(uint32_t *)dst = *(const uint32_t *)src; + } +} +#endif + #if defined(RTE_TOOLCHAIN_GCC) && (GCC_VERSION >= 100000) #pragma GCC diagnostic pop #endif From patchwork Mon Sep 7 19:32:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Kumar X-Patchwork-Id: 76738 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 976A4A04B1; Mon, 7 Sep 2020 21:32:48 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 307CF1C0CC; Mon, 7 Sep 2020 21:32:46 +0200 (CEST) Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) by dpdk.org (Postfix) with ESMTP id C6FF51C0C9 for ; Mon, 7 Sep 2020 21:32:43 +0200 (CEST) Received: by mail-pg1-f196.google.com with SMTP id j34so3052070pgi.7 for ; Mon, 07 Sep 2020 12:32:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vvdntech-in.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=UzW+fRIuVd5z3jtlfdy9A11PL3YCP5z45RPp0iao0fo=; b=LvXGmjGAOAA1t5/inUgQP0eJeEmR6IiOW690h4hf0nv0viYZDSaiu5SxqRPIkMoxxu tHy8u33tPLOC5HeXKRzuD+sMdU0v4pPgA35XswSMJsM/UASvPoEWBYBnEN0MjQqUB16w YzB4O2cwJ8f1dJc7oQTxeLFqpcPDsEALA0nPpTBR3/tK0y8fJ9FQI7mOjxCqBxZQEtD8 bp5f9UHXzNhgdg219VrhXRcZdk34GCxK9ktp6UHiUYKc8gBGvJBwYBMmir4Hhc/kfbio lUVs6lXr3lK3KZzN8SQoFLWbI4R01f5FwhBSBWJN7hfRRQervhBKmnvbknah3NSAqM0Y fmgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=UzW+fRIuVd5z3jtlfdy9A11PL3YCP5z45RPp0iao0fo=; b=jWQr3FiQ2Q3OCiY3wCZeKfj/+1QOipJuvAlNra4GILfFjFhA030n4bBPBTsamc7jgV bR22xw5kgQG3PpLEJ1mSUpwpFffRHLyLUin0yP6sYp+JFBO0+MrmbCyM6I7398bPrzBE tqxOrW8uEu76p6LMTYDK3FDkVJFl0UsXFuNr7qSShxkiuPUUbYIw5PLF/p4c6J0EUBss c+tBSTToCbG7tEV2Ge02oTWV2WwJHowHDimp6rZSGzYmGLfJsaLrx8ZmGRLTa2PEBuJJ Ek1a0pgFS7gD1+WCw/sHGzF/V2WDSLKR5h+pauNDpa6ZMKRF5N5nJwsgaDkKtwTMVGGM QVYg== X-Gm-Message-State: AOAM532SF+SDxpUuGmxnD8bIqfz78zNe2kk/CDHDHo2xMuvdQJx5cRhj Q3Og2SMKOev3R3RxNsPgvDi023gLs0SfB97QPkjMakQMwiFsyTErqZL5pIncWPvWEso0obc5/+t CaoSt X-Google-Smtp-Source: ABdhPJxEaK/ktjWmUZ4JQQN9QgmHgvHbsqyEteNBJzWepX7bZ6tellYp5diaU3m0XpM0sjA4zZsfOg== X-Received: by 2002:a63:4b47:: with SMTP id k7mr1906036pgl.437.1599507162490; Mon, 07 Sep 2020 12:32:42 -0700 (PDT) Received: from localhost.localdomain ([103.251.53.230]) by smtp.gmail.com with ESMTPSA id p4sm3021495pju.29.2020.09.07.12.32.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2020 12:32:41 -0700 (PDT) From: Aman Kumar To: dev@dpdk.org Cc: rasland@mellanox.com, keesang.song@amd.com, aman.kumar@vvdntech.in Date: Tue, 8 Sep 2020 01:02:13 +0530 Message-Id: <20200907193214.38426-2-aman.kumar@vvdntech.in> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200907193214.38426-1-aman.kumar@vvdntech.in> References: <20200907193214.38426-1-aman.kumar@vvdntech.in> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 2/3] net/mlx5: add non temporal store for WQE fields X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" add non temporal store for few WQE fields to optimize data path. This can be enable by making CONFG_RTE_LIBRTE_MLX5_NT_STORE=y in dpdk config. Signed-off-by: Aman Kumar --- config/common_base | 1 + drivers/net/mlx5/mlx5.c | 17 +++++++++++++++++ drivers/net/mlx5/mlx5.h | 4 ++++ drivers/net/mlx5/mlx5_rxq.c | 3 +++ drivers/net/mlx5/mlx5_rxtx.c | 20 +++++++++++++++++--- drivers/net/mlx5/mlx5_rxtx.h | 6 ++++++ drivers/net/mlx5/mlx5_rxtx_vec.h | 28 +++++++++++++++++++++++----- drivers/net/mlx5/mlx5_txq.c | 3 +++ 8 files changed, 74 insertions(+), 8 deletions(-) diff --git a/config/common_base b/config/common_base index 1476cf334..bbe74833b 100644 --- a/config/common_base +++ b/config/common_base @@ -372,6 +372,7 @@ CONFIG_RTE_LIBRTE_MLX4_DEBUG=n # CONFIG_RTE_LIBRTE_MLX5_PMD=n CONFIG_RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY=n +CONFIG_RTE_LIBRTE_MLX5_NT_STORE=n CONFIG_RTE_LIBRTE_MLX5_DEBUG=n # diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 6eb85dfac..8e1b7df23 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -167,6 +167,13 @@ /* mprq_tstore_memcpy */ #define MLX5_MPRQ_TSTORE_MEMCPY "mprq_tstore_memcpy" #endif +#ifdef RTE_LIBRTE_MLX5_NT_STORE +/* tx_wqe_field_ntstore */ +#define MLX5_TX_WQE_FIELD_NTSTORE "tx_wqe_field_ntstore" + +/* vec_rx_wqe_field_ntstore */ +#define MLX5_VEC_RX_WQE_FIELD_NTSTORE "vec_rx_wqe_field_ntstore" +#endif /* * Device parameter to configure the total data buffer size for a single @@ -1629,6 +1636,12 @@ mlx5_args_check(const char *key, const char *val, void *opaque) #ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY } else if (strcmp(MLX5_MPRQ_TSTORE_MEMCPY, key) == 0) { config->mprq_tstore_memcpy = tmp; +#endif +#ifdef RTE_LIBRTE_MLX5_NT_STORE + } else if (strcmp(MLX5_TX_WQE_FIELD_NTSTORE, key) == 0) { + config->tx_wqe_field_ntstore = tmp; + } else if (strcmp(MLX5_VEC_RX_WQE_FIELD_NTSTORE, key) == 0) { + config->vec_rx_wqe_field_ntstore = tmp; #endif } else { DRV_LOG(WARNING, "%s: unknown parameter", key); @@ -1692,6 +1705,10 @@ mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs) MLX5_DECAP_EN, #ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY MLX5_MPRQ_TSTORE_MEMCPY, +#endif +#ifdef RTE_LIBRTE_MLX5_NT_STORE + MLX5_TX_WQE_FIELD_NTSTORE, + MLX5_VEC_RX_WQE_FIELD_NTSTORE, #endif NULL, }; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 09dc90953..4a816cb2e 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -239,6 +239,10 @@ struct mlx5_dev_config { #ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY unsigned int mprq_tstore_memcpy:1; #endif +#ifdef RTE_LIBRTE_MLX5_NT_STORE + unsigned int tx_wqe_field_ntstore:1; + unsigned int vec_rx_wqe_field_ntstore:1; +#endif }; diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index bee5c03bc..ceb33e5c5 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -2304,6 +2304,9 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, tmpl->irq = 1; #ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY tmpl->rxq.mprq_tstore_memcpy = config->mprq_tstore_memcpy; +#endif +#ifdef RTE_LIBRTE_MLX5_NT_STORE + tmpl->rxq.vec_rx_wqe_field_ntstore = config->vec_rx_wqe_field_ntstore; #endif mprq_stride_nums = config->mprq.stride_num_n ? config->mprq.stride_num_n : MLX5_MPRQ_STRIDE_NUM_N; diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index 62ade3775..6bcdc44a5 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -2318,6 +2318,9 @@ mlx5_tx_request_completion(struct mlx5_txq_data *__rte_restrict txq, { uint16_t head = txq->elts_head; unsigned int part; +#ifdef RTE_LIBRTE_MLX5_NT_STORE + register uint32_t flags; +#endif part = MLX5_TXOFF_CONFIG(INLINE) ? 0 : loc->pkts_sent - loc->pkts_copy; @@ -2331,9 +2334,20 @@ mlx5_tx_request_completion(struct mlx5_txq_data *__rte_restrict txq, txq->elts_comp = head; if (MLX5_TXOFF_CONFIG(INLINE)) txq->wqe_comp = txq->wqe_ci; - /* Request unconditional completion on last WQE. */ - last->cseg.flags = RTE_BE32(MLX5_COMP_ALWAYS << - MLX5_COMP_MODE_OFFSET); +#ifdef RTE_LIBRTE_MLX5_NT_STORE + if (txq->tx_wqe_field_ntstore) { + flags = RTE_BE32(MLX5_COMP_ALWAYS << + MLX5_COMP_MODE_OFFSET); + _mm_stream_si32(((void *)(uintptr_t)&last->cseg.flags), + flags); + } else { +#endif + /* Request unconditional completion on last WQE. */ + last->cseg.flags = RTE_BE32(MLX5_COMP_ALWAYS << + MLX5_COMP_MODE_OFFSET); +#ifdef RTE_LIBRTE_MLX5_NT_STORE + } +#endif /* Save elts_head in dedicated free on completion queue. */ #ifdef RTE_LIBRTE_MLX5_DEBUG txq->fcqs[txq->cq_pi++ & txq->cqe_m] = head | diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 72763962f..b031eff0b 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -156,6 +156,9 @@ struct mlx5_rxq_data { #ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY unsigned int mprq_tstore_memcpy:1; #endif +#ifdef RTE_LIBRTE_MLX5_NT_STORE + unsigned int vec_rx_wqe_field_ntstore:1; +#endif } __rte_cache_aligned; enum mlx5_rxq_obj_type { @@ -324,6 +327,9 @@ struct mlx5_txq_data { int32_t ts_offset; /* Timestamp field dynamic offset. */ struct mlx5_dev_ctx_shared *sh; /* Shared context. */ struct mlx5_txq_stats stats; /* TX queue counters. */ +#ifdef RTE_LIBRTE_MLX5_NT_STORE + unsigned int tx_wqe_field_ntstore:1; +#endif #ifndef RTE_ARCH_64 rte_spinlock_t *uar_lock; /* UAR access lock required for 32bit implementations */ diff --git a/drivers/net/mlx5/mlx5_rxtx_vec.h b/drivers/net/mlx5/mlx5_rxtx_vec.h index 6ddcbfb0a..62a07ef00 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec.h @@ -86,6 +86,10 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *rxq, uint16_t n) volatile struct mlx5_wqe_data_seg *wq = &((volatile struct mlx5_wqe_data_seg *)rxq->wqes)[elts_idx]; unsigned int i; +#ifdef RTE_LIBRTE_MLX5_NT_STORE + register uint64_t buf_addr2; + register uint32_t lkey_t; +#endif MLX5_ASSERT(n >= MLX5_VPMD_RXQ_RPLNSH_THRESH(q_n)); MLX5_ASSERT(n <= (uint16_t)(q_n - (rxq->rq_ci - rxq->rq_pi))); @@ -107,11 +111,25 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *rxq, uint16_t n) * impact the performance. */ buf_addr = elts[i]->buf_addr; - wq[i].addr = rte_cpu_to_be_64((uintptr_t)buf_addr + - RTE_PKTMBUF_HEADROOM); - /* If there's only one MR, no need to replace LKey in WQE. */ - if (unlikely(mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1)) - wq[i].lkey = mlx5_rx_mb2mr(rxq, elts[i]); +#ifdef RTE_LIBRTE_MLX5_NT_STORE + if (rxq->vec_rx_wqe_field_ntstore) { + buf_addr2 = (uint64_t)rte_cpu_to_be_64((uintptr_t)buf_addr + RTE_PKTMBUF_HEADROOM); + _mm_stream_si64(((void *)(uintptr_t)&wq[i].addr), buf_addr2); + /* If there's only one MR, no need to replace LKey in WQE. */ + if (unlikely(mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1)) { + lkey_t = (uint32_t)mlx5_rx_mb2mr(rxq, elts[i]); + _mm_stream_si32(((void *)(uintptr_t)&wq[i].lkey), lkey_t); + } + } else { +#endif + wq[i].addr = rte_cpu_to_be_64((uintptr_t)buf_addr + + RTE_PKTMBUF_HEADROOM); + /* If there's only one MR, no need to replace LKey in WQE. */ + if (unlikely(mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1)) + wq[i].lkey = mlx5_rx_mb2mr(rxq, elts[i]); +#ifdef RTE_LIBRTE_MLX5_NT_STORE + } +#endif } rxq->rq_ci += n; /* Prevent overflowing into consumed mbufs. */ diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 21fe16b7e..8feac4bdc 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -1562,6 +1562,9 @@ txq_set_params(struct mlx5_txq_ctrl *txq_ctrl) DEV_TX_OFFLOAD_UDP_TNL_TSO); bool vlan_inline; unsigned int temp; +#ifdef RTE_LIBRTE_MLX5_NT_STORE + txq_ctrl->txq.tx_wqe_field_ntstore = config->tx_wqe_field_ntstore; +#endif if (config->txqs_inline == MLX5_ARG_UNSET) txqs_inline = From patchwork Mon Sep 7 19:32:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aman Kumar X-Patchwork-Id: 76739 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 76393A04B1; Mon, 7 Sep 2020 21:32:57 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 581871C0D1; Mon, 7 Sep 2020 21:32:57 +0200 (CEST) Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by dpdk.org (Postfix) with ESMTP id 7714C1C0CA for ; Mon, 7 Sep 2020 21:32:56 +0200 (CEST) Received: by mail-pg1-f194.google.com with SMTP id m8so3152284pgi.3 for ; Mon, 07 Sep 2020 12:32:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vvdntech-in.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=UEpGlwNp5jJkPh5dL1HAjpykMorVvrYCP9leSwqP+3w=; b=iwqZLnsaT2qLhZHslLBhyRWf4VjWFccF7XMY7Jp/NH10tPYFQdK9KDt2jiY/+nHo6y fPzOLFNU28ss8KnHPFtUYVHNUYxWsyeNSRMq8l/d/190MRjfiLtn8a74zFkTHAEDKbRc hayvMsJkU/7a5uBdcr68DmSdZj9zoMVG1kCEub1CGU3j4HcvyPvQN7GAc6FBX5fZkZkR SlJfUAGNVbSnjCXPQx/f5lJ4wcrjJUKNOUvKm1Co7JgFlqe4nwiDo+jcaoOexYtEsdLj 4NgS31JIl8EQesz2kyx8jlM3IxYkIscsl00aoXYqO2CZkEota4mSBrFOywaomql+ahJp uz/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=UEpGlwNp5jJkPh5dL1HAjpykMorVvrYCP9leSwqP+3w=; b=fg7Fqrxz+NwfEBH+bC4c3stcNo2U1biOCidiW4qMNA5bAu1fV2bXpVv4MJRi53MkiZ 1BLW2mLnSQDoip+CDYdcYy985pzo8Of2yoP0fPTCG3uNuU6IXQb6BWRnhYFyJ5z7Wu/6 A1dnP3aLDp4s70TMca+L2q6byugAyv3TFssP2awS9uYoGkdjhfgw9ebE41dvhiIYM7vW 4w1kcPxTPDfRiSeohnvKmATBQL1dX/EVVUv6ujzP8F0vNRy8Ya8j51prqa5jt443PhNE pjgZLpuJM9DIEKGNJv0XuA5LPkklACGWsRYCY0GhZA4x3ExyhyIp8J5ug6nQwZZC3lws RS7Q== X-Gm-Message-State: AOAM531oSwsVO06VEvfcjd5ogZPhhOwjFrEUgRH5HZFwkFGhrQGj3wxq JNB2SNQ6GblnGDAHATidRBi+tMCbge7xEW41VjqZqOHYJCpVgXz1+AsTYYiwkDSCiA4O3G7ofOZ cYzFv X-Google-Smtp-Source: ABdhPJwbFGPml9Uq+Oh0t+dSew6SIez1MDwPTuelRJ6VHhDBiHutuumdtRrPgXJdGI6mEnjK6pJqgw== X-Received: by 2002:a62:1b56:: with SMTP id b83mr19519722pfb.15.1599507175414; Mon, 07 Sep 2020 12:32:55 -0700 (PDT) Received: from localhost.localdomain ([103.251.53.230]) by smtp.gmail.com with ESMTPSA id p4sm3021495pju.29.2020.09.07.12.32.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Sep 2020 12:32:54 -0700 (PDT) From: Aman Kumar To: dev@dpdk.org Cc: rasland@mellanox.com, keesang.song@amd.com, aman.kumar@vvdntech.in Date: Tue, 8 Sep 2020 01:02:14 +0530 Message-Id: <20200907193214.38426-3-aman.kumar@vvdntech.in> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200907193214.38426-1-aman.kumar@vvdntech.in> References: <20200907193214.38426-1-aman.kumar@vvdntech.in> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 3/3] config: added build config file for AMD EPYC platform X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" add build config specific to AMD EPYC platform Signed-off-by: Aman Kumar --- config/defconfig_x86_64-amdEPYC-linux-gcc | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 config/defconfig_x86_64-amdEPYC-linux-gcc diff --git a/config/defconfig_x86_64-amdEPYC-linux-gcc b/config/defconfig_x86_64-amdEPYC-linux-gcc new file mode 100644 index 000000000..8c1aa34e2 --- /dev/null +++ b/config/defconfig_x86_64-amdEPYC-linux-gcc @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2010-2014 Intel Corporation + +#include "common_linux" + +CONFIG_RTE_MACHINE="native" + +CONFIG_RTE_ARCH="x86_64" +CONFIG_RTE_ARCH_X86_64=y +CONFIG_RTE_ARCH_X86=y +CONFIG_RTE_ARCH_64=y + +CONFIG_RTE_TOOLCHAIN="gcc" +CONFIG_RTE_TOOLCHAIN_GCC=y +CONFIG_RTE_MAX_LCORE=256 +CONFIG_RTE_LIBRTE_MLX5_PMD=n +CONFIG_RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY=n +CONFIG_RTE_LIBRTE_MLX5_NT_STORE=n