From patchwork Wed Sep 23 14:22:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Radu Nicolau X-Patchwork-Id: 78576 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id D2B61A04B1; Wed, 23 Sep 2020 16:23:51 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E8A2F1DC75; Wed, 23 Sep 2020 16:23:44 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id 6905A1DAAF for ; Wed, 23 Sep 2020 16:23:42 +0200 (CEST) IronPort-SDR: 6jUGz4lPEEPPOxCawYgs6bQzwjx7Fe02gW/Y7HbaFY/ndeqhUZ9UPx75ek/GV2YdXD6AdmHSj3 aBqjB2dSNnTQ== X-IronPort-AV: E=McAfee;i="6000,8403,9752"; a="245732141" X-IronPort-AV: E=Sophos;i="5.77,293,1596524400"; d="scan'208";a="245732141" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2020 07:23:42 -0700 IronPort-SDR: +mSsVow0tl2gA0Lorsd+V6zlAp97Ym4Sf2cOvj9lQiPvb/IJ1cMeii+oBaEIiUR89fad6UXyEJ zjhcL7fau8+g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,293,1596524400"; d="scan'208";a="511663490" Received: from silpixa00399477.ir.intel.com ([10.237.214.232]) by fmsmga005.fm.intel.com with ESMTP; 23 Sep 2020 07:23:39 -0700 From: Radu Nicolau To: dev@dpdk.org Cc: beilei.xing@intel.com, jia.guo@intel.com, bruce.richardson@intel.com, konstantin.ananyev@intel.com, jerinjacobk@gmail.com, david.marchand@redhat.com, fiona.trahe@intel.com, wei.zhao1@intel.com, ruifeng.wang@arm.com, qiming.yang@intel.com, qi.z.zhang@intel.com, Radu Nicolau Date: Wed, 23 Sep 2020 14:22:49 +0000 Message-Id: <20200923142253.18853-2-radu.nicolau@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200923142253.18853-1-radu.nicolau@intel.com> References: <1591870283-7776-1-git-send-email-radu.nicolau@intel.com> <20200923142253.18853-1-radu.nicolau@intel.com> Subject: [dpdk-dev] [PATCH v12 1/5] eal: add WC store functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add rte_write32_wc and rte_write32_wc_relaxed functions that implement 32bit stores using write combining memory protocol. Provided generic stubs and x86 implementation. Signed-off-by: Radu Nicolau Acked-by: Bruce Richardson --- doc/guides/rel_notes/release_20_11.rst | 6 ++++ lib/librte_eal/arm/include/rte_io_64.h | 12 +++++++ lib/librte_eal/include/generic/rte_io.h | 48 +++++++++++++++++++++++++ lib/librte_eal/x86/include/rte_io.h | 42 ++++++++++++++++++++++ 4 files changed, 108 insertions(+) diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst index c6642f5f9..f51577684 100644 --- a/doc/guides/rel_notes/release_20_11.rst +++ b/doc/guides/rel_notes/release_20_11.rst @@ -78,6 +78,12 @@ New Features ``--portmask=N`` where N represents the hexadecimal bitmask of ports used. +* **Added write combining store APIs.** + + Added ``rte_write32_wc`` and ``rte_write32_wc_relaxed`` APIs + that enable write combining stores (depending on architecture). + The functions are provided as a generic stubs and + x86 specific implementation. Removed Items ------------- diff --git a/lib/librte_eal/arm/include/rte_io_64.h b/lib/librte_eal/arm/include/rte_io_64.h index e5346240e..d07d9cb22 100644 --- a/lib/librte_eal/arm/include/rte_io_64.h +++ b/lib/librte_eal/arm/include/rte_io_64.h @@ -164,6 +164,18 @@ rte_write64(uint64_t value, volatile void *addr) rte_write64_relaxed(value, addr); } +static __rte_always_inline void +rte_write32_wc(uint32_t value, volatile void *addr) +{ + rte_write32(value, addr); +} + +static __rte_always_inline void +rte_write32_wc_relaxed(uint32_t value, volatile void *addr) +{ + rte_write32_relaxed(value, addr); +} + #ifdef __cplusplus } #endif diff --git a/lib/librte_eal/include/generic/rte_io.h b/lib/librte_eal/include/generic/rte_io.h index da457f7f7..0669baa0b 100644 --- a/lib/librte_eal/include/generic/rte_io.h +++ b/lib/librte_eal/include/generic/rte_io.h @@ -229,6 +229,40 @@ rte_write32(uint32_t value, volatile void *addr); static inline void rte_write64(uint64_t value, volatile void *addr); +/** + * Write a 32-bit value to I/O device memory address addr using write + * combining memory write protocol. Depending on the platform write combining + * may not be available and/or may be treated as a hint and the behavior may + * fallback to a regular store. + * + * @param value + * Value to write + * @param addr + * I/O memory address to write the value to + */ +__rte_experimental +static inline void +rte_write32_wc(uint32_t value, volatile void *addr); + +/** + * Write a 32-bit value to I/O device memory address addr using write + * combining memory write protocol. Depending on the platform write combining + * may not be available and/or may be treated as a hint and the behavior may + * fallback to a regular store. + * + * The relaxed version does not have additional I/O memory barrier, useful in + * accessing the device registers of integrated controllers which implicitly + * strongly ordered with respect to memory access. + * + * @param value + * Value to write + * @param addr + * I/O memory address to write the value to + */ +__rte_experimental +static inline void +rte_write32_wc_relaxed(uint32_t value, volatile void *addr); + #endif /* __DOXYGEN__ */ #ifndef RTE_OVERRIDE_IO_H @@ -345,6 +379,20 @@ rte_write64(uint64_t value, volatile void *addr) rte_write64_relaxed(value, addr); } +#ifndef RTE_NATIVE_WRITE32_WC +static __rte_always_inline void +rte_write32_wc(uint32_t value, volatile void *addr) +{ + rte_write32(value, addr); +} + +static __rte_always_inline void +rte_write32_wc_relaxed(uint32_t value, volatile void *addr) +{ + rte_write32_relaxed(value, addr); +} +#endif /* RTE_NATIVE_WRITE32_WC */ + #endif /* RTE_OVERRIDE_IO_H */ #endif /* _RTE_IO_H_ */ diff --git a/lib/librte_eal/x86/include/rte_io.h b/lib/librte_eal/x86/include/rte_io.h index 2db71b1b0..4f4ff8b87 100644 --- a/lib/librte_eal/x86/include/rte_io.h +++ b/lib/librte_eal/x86/include/rte_io.h @@ -9,8 +9,50 @@ extern "C" { #endif +#include "rte_cpuflags.h" + +#define RTE_NATIVE_WRITE32_WC #include "generic/rte_io.h" +/** + * @internal + * MOVDIRI wrapper. + */ +static __rte_always_inline void +_rte_x86_movdiri(uint32_t value, volatile void *addr) +{ + asm volatile( + /* MOVDIRI */ + ".byte 0x40, 0x0f, 0x38, 0xf9, 0x02" + : + : "a" (value), "d" (addr)); +} + +static __rte_always_inline void +rte_write32_wc_relaxed(uint32_t value, volatile void *addr) +{ + static int _x86_movdiri_flag = -1; + if (_x86_movdiri_flag == 1) { + _rte_x86_movdiri(value, addr); + } else if (_x86_movdiri_flag == 0) { + rte_write32_relaxed(value, addr); + } else { + _x86_movdiri_flag = + (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIRI) > 0); + if (_x86_movdiri_flag == 1) + _rte_x86_movdiri(value, addr); + else + rte_write32_relaxed(value, addr); + } +} + +static __rte_always_inline void +rte_write32_wc(uint32_t value, volatile void *addr) +{ + rte_wmb(); + rte_write32_wc_relaxed(value, addr); +} + #ifdef __cplusplus } #endif From patchwork Wed Sep 23 14:22:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Radu Nicolau X-Patchwork-Id: 78577 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id F137FA04B1; Wed, 23 Sep 2020 16:24:00 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4D40C1DC9A; Wed, 23 Sep 2020 16:23:47 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id 486DD1DC8E for ; Wed, 23 Sep 2020 16:23:45 +0200 (CEST) IronPort-SDR: Nb1URrMsOmH2o1btIIH1AVqcuyu+sBISle7j50P84wspuWPE6aVXuOdLB8RjNkyiv1/+bQ+RVs /EpKJbsOTz7A== X-IronPort-AV: E=McAfee;i="6000,8403,9752"; a="245732149" X-IronPort-AV: E=Sophos;i="5.77,293,1596524400"; d="scan'208";a="245732149" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2020 07:23:44 -0700 IronPort-SDR: zTF3x/TejMaUGGH6OjpJYinKWOOTGppXAcozn0CebX5MxkhEkPxCkaClTFQ1v9rEu4uML1gmgd +bPziVIywnBw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,293,1596524400"; d="scan'208";a="511663498" Received: from silpixa00399477.ir.intel.com ([10.237.214.232]) by fmsmga005.fm.intel.com with ESMTP; 23 Sep 2020 07:23:42 -0700 From: Radu Nicolau To: dev@dpdk.org Cc: beilei.xing@intel.com, jia.guo@intel.com, bruce.richardson@intel.com, konstantin.ananyev@intel.com, jerinjacobk@gmail.com, david.marchand@redhat.com, fiona.trahe@intel.com, wei.zhao1@intel.com, ruifeng.wang@arm.com, qiming.yang@intel.com, qi.z.zhang@intel.com, Radu Nicolau Date: Wed, 23 Sep 2020 14:22:50 +0000 Message-Id: <20200923142253.18853-3-radu.nicolau@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200923142253.18853-1-radu.nicolau@intel.com> References: <1591870283-7776-1-git-send-email-radu.nicolau@intel.com> <20200923142253.18853-1-radu.nicolau@intel.com> Subject: [dpdk-dev] [PATCH v12 2/5] net/i40e: use WC store to update queue tail registers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Performance improvement: use a write combining store instead of a regular mmio write to update queue tail registers. Signed-off-by: Radu Nicolau Acked-by: Bruce Richardson Reviewed-by: Wenzhuo Lu --- doc/guides/rel_notes/release_20_11.rst | 4 ++++ drivers/net/i40e/base/i40e_osdep.h | 5 +++++ drivers/net/i40e/i40e_rxtx.c | 8 ++++---- drivers/net/i40e/i40e_rxtx_vec_avx2.c | 4 ++-- drivers/net/i40e/i40e_rxtx_vec_sse.c | 4 ++-- 5 files changed, 17 insertions(+), 8 deletions(-) diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst index f51577684..c0307893e 100644 --- a/doc/guides/rel_notes/release_20_11.rst +++ b/doc/guides/rel_notes/release_20_11.rst @@ -85,6 +85,10 @@ New Features The functions are provided as a generic stubs and x86 specific implementation. +* **Updated Intel i40e driver.** + + Updated the Intel i40e driver to use write combining stores. + Removed Items ------------- diff --git a/drivers/net/i40e/base/i40e_osdep.h b/drivers/net/i40e/base/i40e_osdep.h index 58be39677..9b5033024 100644 --- a/drivers/net/i40e/base/i40e_osdep.h +++ b/drivers/net/i40e/base/i40e_osdep.h @@ -138,6 +138,11 @@ static inline uint32_t i40e_read_addr(volatile void *addr) #define I40E_PCI_REG_WRITE_RELAXED(reg, value) \ rte_write32_relaxed((rte_cpu_to_le_32(value)), reg) +#define I40E_PCI_REG_WC_WRITE(reg, value) \ + rte_write32_wc((rte_cpu_to_le_32(value)), reg) +#define I40E_PCI_REG_WC_WRITE_RELAXED(reg, value) \ + rte_write32_wc_relaxed((rte_cpu_to_le_32(value)), reg) + #define I40E_WRITE_FLUSH(a) I40E_READ_REG(a, I40E_GLGEN_STAT) #define I40EVF_WRITE_FLUSH(a) I40E_READ_REG(a, I40E_VFGEN_RSTAT) diff --git a/drivers/net/i40e/i40e_rxtx.c b/drivers/net/i40e/i40e_rxtx.c index 60b33d20a..7be9c8c95 100644 --- a/drivers/net/i40e/i40e_rxtx.c +++ b/drivers/net/i40e/i40e_rxtx.c @@ -760,7 +760,7 @@ i40e_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) if (nb_hold > rxq->rx_free_thresh) { rx_id = (uint16_t) ((rx_id == 0) ? (rxq->nb_rx_desc - 1) : (rx_id - 1)); - I40E_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); nb_hold = 0; } rxq->nb_rx_hold = nb_hold; @@ -938,7 +938,7 @@ i40e_recv_scattered_pkts(void *rx_queue, if (nb_hold > rxq->rx_free_thresh) { rx_id = (uint16_t)(rx_id == 0 ? (rxq->nb_rx_desc - 1) : (rx_id - 1)); - I40E_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); nb_hold = 0; } rxq->nb_rx_hold = nb_hold; @@ -1249,7 +1249,7 @@ i40e_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) (unsigned) tx_id, (unsigned) nb_tx); rte_cio_wmb(); - I40E_PCI_REG_WRITE_RELAXED(txq->qtx_tail, tx_id); + I40E_PCI_REG_WC_WRITE_RELAXED(txq->qtx_tail, tx_id); txq->tx_tail = tx_id; return nb_tx; @@ -1400,7 +1400,7 @@ tx_xmit_pkts(struct i40e_tx_queue *txq, txq->tx_tail = 0; /* Update the tx tail register */ - I40E_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + I40E_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail); return nb_pkts; } diff --git a/drivers/net/i40e/i40e_rxtx_vec_avx2.c b/drivers/net/i40e/i40e_rxtx_vec_avx2.c index 37e7db5d7..7a558fc73 100644 --- a/drivers/net/i40e/i40e_rxtx_vec_avx2.c +++ b/drivers/net/i40e/i40e_rxtx_vec_avx2.c @@ -134,7 +134,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - I40E_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); } #ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC @@ -921,7 +921,7 @@ i40e_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, txq->tx_tail = tx_id; - I40E_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + I40E_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail); return nb_pkts; } diff --git a/drivers/net/i40e/i40e_rxtx_vec_sse.c b/drivers/net/i40e/i40e_rxtx_vec_sse.c index 698518349..240ce478a 100644 --- a/drivers/net/i40e/i40e_rxtx_vec_sse.c +++ b/drivers/net/i40e/i40e_rxtx_vec_sse.c @@ -86,7 +86,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - I40E_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); } #ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC @@ -733,7 +733,7 @@ i40e_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, txq->tx_tail = tx_id; - I40E_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + I40E_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail); return nb_pkts; } From patchwork Wed Sep 23 14:22:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Radu Nicolau X-Patchwork-Id: 78578 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2678EA04B1; Wed, 23 Sep 2020 16:24:10 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BFB841DB92; Wed, 23 Sep 2020 16:23:50 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id D39861DCA5 for ; Wed, 23 Sep 2020 16:23:47 +0200 (CEST) IronPort-SDR: 5JgJLfDqj6/wGZDM8WRzZozsftuwV/Z9Sg0bIlciSHz3NjPa9Uu0fUcFkfSVSDjX++Mqrk2p0M Tj+nuwPREsgQ== X-IronPort-AV: E=McAfee;i="6000,8403,9752"; a="245732155" X-IronPort-AV: E=Sophos;i="5.77,293,1596524400"; d="scan'208";a="245732155" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2020 07:23:47 -0700 IronPort-SDR: cU2dUqksQe19ROxXUBxB0iCW4r2aoOLlWH4+kBm8U3gOV8FKWePbAPg+UgZ7+wSVAowOkjqWIL qkPumpaiEP0w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,293,1596524400"; d="scan'208";a="511663518" Received: from silpixa00399477.ir.intel.com ([10.237.214.232]) by fmsmga005.fm.intel.com with ESMTP; 23 Sep 2020 07:23:44 -0700 From: Radu Nicolau To: dev@dpdk.org Cc: beilei.xing@intel.com, jia.guo@intel.com, bruce.richardson@intel.com, konstantin.ananyev@intel.com, jerinjacobk@gmail.com, david.marchand@redhat.com, fiona.trahe@intel.com, wei.zhao1@intel.com, ruifeng.wang@arm.com, qiming.yang@intel.com, qi.z.zhang@intel.com, Radu Nicolau Date: Wed, 23 Sep 2020 14:22:51 +0000 Message-Id: <20200923142253.18853-4-radu.nicolau@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200923142253.18853-1-radu.nicolau@intel.com> References: <1591870283-7776-1-git-send-email-radu.nicolau@intel.com> <20200923142253.18853-1-radu.nicolau@intel.com> Subject: [dpdk-dev] [PATCH v12 3/5] common/qat: use WC store to update queue tail registers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Performance improvement: use a write combining store instead of a regular mmio write to update queue tail registers. Signed-off-by: Radu Nicolau Acked-by: Fiona Trahe --- doc/guides/rel_notes/release_20_11.rst | 4 ++++ drivers/common/qat/qat_adf/adf_transport_access_macros.h | 6 ++++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst index c0307893e..3d310572c 100644 --- a/doc/guides/rel_notes/release_20_11.rst +++ b/doc/guides/rel_notes/release_20_11.rst @@ -89,6 +89,10 @@ New Features Updated the Intel i40e driver to use write combining stores. +* **Updated Intel qat driver.** + + Updated the Intel qat driver to use write combining stores. + Removed Items ------------- diff --git a/drivers/common/qat/qat_adf/adf_transport_access_macros.h b/drivers/common/qat/qat_adf/adf_transport_access_macros.h index 1eef5513f..504ffb723 100644 --- a/drivers/common/qat/qat_adf/adf_transport_access_macros.h +++ b/drivers/common/qat/qat_adf/adf_transport_access_macros.h @@ -9,6 +9,8 @@ /* CSR write macro */ #define ADF_CSR_WR(csrAddr, csrOffset, val) \ rte_write32(val, (((uint8_t *)csrAddr) + csrOffset)) +#define ADF_CSR_WC_WR(csrAddr, csrOffset, val) \ + rte_write32_wc(val, (((uint8_t *)csrAddr) + csrOffset)) /* CSR read macro */ #define ADF_CSR_RD(csrAddr, csrOffset) \ @@ -110,10 +112,10 @@ do { \ ADF_RING_CSR_RING_UBASE + (ring << 2), u_base); \ } while (0) #define WRITE_CSR_RING_HEAD(csr_base_addr, bank, ring, value) \ - ADF_CSR_WR(csr_base_addr, (ADF_RING_BUNDLE_SIZE * bank) + \ + ADF_CSR_WC_WR(csr_base_addr, (ADF_RING_BUNDLE_SIZE * bank) + \ ADF_RING_CSR_RING_HEAD + (ring << 2), value) #define WRITE_CSR_RING_TAIL(csr_base_addr, bank, ring, value) \ - ADF_CSR_WR(csr_base_addr, (ADF_RING_BUNDLE_SIZE * bank) + \ + ADF_CSR_WC_WR(csr_base_addr, (ADF_RING_BUNDLE_SIZE * bank) + \ ADF_RING_CSR_RING_TAIL + (ring << 2), value) #define WRITE_CSR_INT_SRCSEL(csr_base_addr, bank) \ do { \ From patchwork Wed Sep 23 14:22:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Radu Nicolau X-Patchwork-Id: 78579 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id A89F1A04B1; Wed, 23 Sep 2020 16:24:18 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2A4E31DCB9; Wed, 23 Sep 2020 16:23:54 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id BFE621DCAC for ; Wed, 23 Sep 2020 16:23:50 +0200 (CEST) IronPort-SDR: On0DvhfyZ0Te+Uf4MgHb+AkfG0FTAgkOepflOEusJoOImTFYbHHbVRm9Top/oA8e59IbOOBDZD 95ueL8Bja28g== X-IronPort-AV: E=McAfee;i="6000,8403,9752"; a="245732162" X-IronPort-AV: E=Sophos;i="5.77,293,1596524400"; d="scan'208";a="245732162" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2020 07:23:50 -0700 IronPort-SDR: HyzppUHjPcVH6EAmW/SFqxBNfPaYPSz9wFmIsYHhZ3UsmTd5DxutClhu0fYSILl62woXTCpRkr bHOYbNPBvApw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,293,1596524400"; d="scan'208";a="511663547" Received: from silpixa00399477.ir.intel.com ([10.237.214.232]) by fmsmga005.fm.intel.com with ESMTP; 23 Sep 2020 07:23:47 -0700 From: Radu Nicolau To: dev@dpdk.org Cc: beilei.xing@intel.com, jia.guo@intel.com, bruce.richardson@intel.com, konstantin.ananyev@intel.com, jerinjacobk@gmail.com, david.marchand@redhat.com, fiona.trahe@intel.com, wei.zhao1@intel.com, ruifeng.wang@arm.com, qiming.yang@intel.com, qi.z.zhang@intel.com, Radu Nicolau Date: Wed, 23 Sep 2020 14:22:52 +0000 Message-Id: <20200923142253.18853-5-radu.nicolau@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200923142253.18853-1-radu.nicolau@intel.com> References: <1591870283-7776-1-git-send-email-radu.nicolau@intel.com> <20200923142253.18853-1-radu.nicolau@intel.com> Subject: [dpdk-dev] [PATCH v12 4/5] net/ixgbe: use WC store to update queue tail registers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Performance improvement: use a write combining store instead of a regular mmio write to update queue tail registers. Signed-off-by: Radu Nicolau Acked-by: Konstantin Ananyev Reviewed-by: Wenzhuo Lu --- doc/guides/rel_notes/release_20_11.rst | 4 ++++ drivers/net/ixgbe/base/ixgbe_osdep.h | 6 ++++++ drivers/net/ixgbe/ixgbe_rxtx.c | 15 ++++++++------- drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c | 4 ++-- 4 files changed, 20 insertions(+), 9 deletions(-) diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst index 3d310572c..caca04208 100644 --- a/doc/guides/rel_notes/release_20_11.rst +++ b/doc/guides/rel_notes/release_20_11.rst @@ -93,6 +93,10 @@ New Features Updated the Intel qat driver to use write combining stores. +* **Updated Intel ixgbe driver.** + + Updated the Intel ixgbe driver to use write combining stores. + Removed Items ------------- diff --git a/drivers/net/ixgbe/base/ixgbe_osdep.h b/drivers/net/ixgbe/base/ixgbe_osdep.h index dc712b7c0..cacf72419 100644 --- a/drivers/net/ixgbe/base/ixgbe_osdep.h +++ b/drivers/net/ixgbe/base/ixgbe_osdep.h @@ -105,6 +105,12 @@ static inline uint32_t ixgbe_read_addr(volatile void* addr) #define IXGBE_PCI_REG_WRITE_RELAXED(reg, value) \ rte_write32_relaxed((rte_cpu_to_le_32(value)), reg) +#define IXGBE_PCI_REG_WC_WRITE(reg, value) \ + rte_write32_wc((rte_cpu_to_le_32(value)), reg) + +#define IXGBE_PCI_REG_WC_WRITE_RELAXED(reg, value) \ + rte_write32_wc_relaxed((rte_cpu_to_le_32(value)), reg) + #define IXGBE_PCI_REG_ADDR(hw, reg) \ ((volatile uint32_t *)((char *)(hw)->hw_addr + (reg))) diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxtx.c index 977ecf513..29d385c06 100644 --- a/drivers/net/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/ixgbe/ixgbe_rxtx.c @@ -308,7 +308,7 @@ tx_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, /* update tail pointer */ rte_wmb(); - IXGBE_PCI_REG_WRITE_RELAXED(txq->tdt_reg_addr, txq->tx_tail); + IXGBE_PCI_REG_WC_WRITE_RELAXED(txq->tdt_reg_addr, txq->tx_tail); return nb_pkts; } @@ -946,7 +946,7 @@ ixgbe_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, PMD_TX_LOG(DEBUG, "port_id=%u queue_id=%u tx_tail=%u nb_tx=%u", (unsigned) txq->port_id, (unsigned) txq->queue_id, (unsigned) tx_id, (unsigned) nb_tx); - IXGBE_PCI_REG_WRITE_RELAXED(txq->tdt_reg_addr, tx_id); + IXGBE_PCI_REG_WC_WRITE_RELAXED(txq->tdt_reg_addr, tx_id); txq->tx_tail = tx_id; return nb_tx; @@ -1692,7 +1692,7 @@ rx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, /* update tail pointer */ rte_wmb(); - IXGBE_PCI_REG_WRITE_RELAXED(rxq->rdt_reg_addr, + IXGBE_PCI_REG_WC_WRITE_RELAXED(rxq->rdt_reg_addr, cur_free_trigger); } @@ -1918,7 +1918,7 @@ ixgbe_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, (unsigned) nb_rx); rx_id = (uint16_t) ((rx_id == 0) ? (rxq->nb_rx_desc - 1) : (rx_id - 1)); - IXGBE_PCI_REG_WRITE(rxq->rdt_reg_addr, rx_id); + IXGBE_PCI_REG_WC_WRITE(rxq->rdt_reg_addr, rx_id); nb_hold = 0; } rxq->nb_rx_hold = nb_hold; @@ -2096,8 +2096,9 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, if (!ixgbe_rx_alloc_bufs(rxq, false)) { rte_wmb(); - IXGBE_PCI_REG_WRITE_RELAXED(rxq->rdt_reg_addr, - next_rdt); + IXGBE_PCI_REG_WC_WRITE_RELAXED( + rxq->rdt_reg_addr, + next_rdt); nb_hold -= rxq->rx_free_thresh; } else { PMD_RX_LOG(DEBUG, "RX bulk alloc failed " @@ -2262,7 +2263,7 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, rxq->port_id, rxq->queue_id, rx_id, nb_hold, nb_rx); rte_wmb(); - IXGBE_PCI_REG_WRITE_RELAXED(rxq->rdt_reg_addr, prev_id); + IXGBE_PCI_REG_WC_WRITE_RELAXED(rxq->rdt_reg_addr, prev_id); nb_hold = 0; } diff --git a/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c b/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c index 517ca3166..e77a7f31c 100644 --- a/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c +++ b/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c @@ -90,7 +90,7 @@ ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - IXGBE_PCI_REG_WRITE(rxq->rdt_reg_addr, rx_id); + IXGBE_PCI_REG_WC_WRITE(rxq->rdt_reg_addr, rx_id); } #ifdef RTE_LIBRTE_SECURITY @@ -697,7 +697,7 @@ ixgbe_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, txq->tx_tail = tx_id; - IXGBE_PCI_REG_WRITE(txq->tdt_reg_addr, txq->tx_tail); + IXGBE_PCI_REG_WC_WRITE(txq->tdt_reg_addr, txq->tx_tail); return nb_pkts; } From patchwork Wed Sep 23 14:22:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Radu Nicolau X-Patchwork-Id: 78580 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3E979A04B1; Wed, 23 Sep 2020 16:24:27 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7455B1DCC3; Wed, 23 Sep 2020 16:23:56 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id A192B1DCB5 for ; Wed, 23 Sep 2020 16:23:53 +0200 (CEST) IronPort-SDR: 4aSz6zWQynNk12JV7JlmL4RqbhRwyWac1fMp2gEAxGthVbM0hwGVoE+9Ti8fr1oTXFyXOOISV/ PoIGPFzXL8JQ== X-IronPort-AV: E=McAfee;i="6000,8403,9752"; a="245732169" X-IronPort-AV: E=Sophos;i="5.77,293,1596524400"; d="scan'208";a="245732169" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Sep 2020 07:23:53 -0700 IronPort-SDR: zl836N+z+K2KLNPT3+IyxlHO8/GNtwVzeLqVJu3jvJwOAlDHvnSLAgoGvJdGXC5Ue3Zjb1VagK 7p+fMBP85uVA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,293,1596524400"; d="scan'208";a="511663560" Received: from silpixa00399477.ir.intel.com ([10.237.214.232]) by fmsmga005.fm.intel.com with ESMTP; 23 Sep 2020 07:23:50 -0700 From: Radu Nicolau To: dev@dpdk.org Cc: beilei.xing@intel.com, jia.guo@intel.com, bruce.richardson@intel.com, konstantin.ananyev@intel.com, jerinjacobk@gmail.com, david.marchand@redhat.com, fiona.trahe@intel.com, wei.zhao1@intel.com, ruifeng.wang@arm.com, qiming.yang@intel.com, qi.z.zhang@intel.com, Radu Nicolau Date: Wed, 23 Sep 2020 14:22:53 +0000 Message-Id: <20200923142253.18853-6-radu.nicolau@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200923142253.18853-1-radu.nicolau@intel.com> References: <1591870283-7776-1-git-send-email-radu.nicolau@intel.com> <20200923142253.18853-1-radu.nicolau@intel.com> Subject: [dpdk-dev] [PATCH v12 5/5] net/ice: use WC store to update queue tail registers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Performance improvement: use a write combining store instead of a regular mmio write to update queue tail registers. Signed-off-by: Radu Nicolau Reviewed-by: Wenzhuo Lu --- doc/guides/rel_notes/release_20_11.rst | 4 ++++ drivers/net/ice/base/ice_osdep.h | 1 + drivers/net/ice/ice_rxtx.c | 6 +++--- drivers/net/ice/ice_rxtx_vec_avx2.c | 4 ++-- drivers/net/ice/ice_rxtx_vec_sse.c | 4 ++-- 5 files changed, 12 insertions(+), 7 deletions(-) diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst index caca04208..9d6e07474 100644 --- a/doc/guides/rel_notes/release_20_11.rst +++ b/doc/guides/rel_notes/release_20_11.rst @@ -97,6 +97,10 @@ New Features Updated the Intel ixgbe driver to use write combining stores. +* **Updated Intel ice driver.** + + Updated the Intel ice driver to use write combining stores. + Removed Items ------------- diff --git a/drivers/net/ice/base/ice_osdep.h b/drivers/net/ice/base/ice_osdep.h index 9a170b514..c0f1e7725 100644 --- a/drivers/net/ice/base/ice_osdep.h +++ b/drivers/net/ice/base/ice_osdep.h @@ -165,6 +165,7 @@ do { \ #endif #define ICE_PCI_REG_WRITE(reg, value) writel(value, reg) +#define ICE_PCI_REG_WC_WRITE(reg, value) rte_write32_wc(value, reg) #define ICE_READ_REG(hw, reg) rd32(hw, reg) #define ICE_WRITE_REG(hw, reg, value) wr32(hw, reg, value) diff --git a/drivers/net/ice/ice_rxtx.c b/drivers/net/ice/ice_rxtx.c index fef6ad454..6bd5b4a0c 100644 --- a/drivers/net/ice/ice_rxtx.c +++ b/drivers/net/ice/ice_rxtx.c @@ -1788,7 +1788,7 @@ ice_recv_scattered_pkts(void *rx_queue, rx_id = (uint16_t)(rx_id == 0 ? (rxq->nb_rx_desc - 1) : (rx_id - 1)); /* write TAIL register */ - ICE_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + ICE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); nb_hold = 0; } rxq->nb_rx_hold = nb_hold; @@ -2178,7 +2178,7 @@ ice_recv_pkts(void *rx_queue, rx_id = (uint16_t)(rx_id == 0 ? (rxq->nb_rx_desc - 1) : (rx_id - 1)); /* write TAIL register */ - ICE_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + ICE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); nb_hold = 0; } rxq->nb_rx_hold = nb_hold; @@ -2893,7 +2893,7 @@ tx_xmit_pkts(struct ice_tx_queue *txq, txq->tx_tail = 0; /* Update the tx tail register */ - ICE_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + ICE_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail); return nb_pkts; } diff --git a/drivers/net/ice/ice_rxtx_vec_avx2.c b/drivers/net/ice/ice_rxtx_vec_avx2.c index 5969a3048..b72a9e702 100644 --- a/drivers/net/ice/ice_rxtx_vec_avx2.c +++ b/drivers/net/ice/ice_rxtx_vec_avx2.c @@ -129,7 +129,7 @@ ice_rxq_rearm(struct ice_rx_queue *rxq) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - ICE_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + ICE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); } static inline __m256i @@ -962,7 +962,7 @@ ice_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, txq->tx_tail = tx_id; - ICE_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + ICE_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail); return nb_pkts; } diff --git a/drivers/net/ice/ice_rxtx_vec_sse.c b/drivers/net/ice/ice_rxtx_vec_sse.c index c4c9a9126..1afd96ac9 100644 --- a/drivers/net/ice/ice_rxtx_vec_sse.c +++ b/drivers/net/ice/ice_rxtx_vec_sse.c @@ -97,7 +97,7 @@ ice_rxq_rearm(struct ice_rx_queue *rxq) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - ICE_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + ICE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); } static inline void @@ -689,7 +689,7 @@ ice_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, txq->tx_tail = tx_id; - ICE_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + ICE_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail); return nb_pkts; }