From patchwork Wed Aug 26 09:55:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Radu Nicolau X-Patchwork-Id: 75985 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C558DA04B2; Wed, 26 Aug 2020 11:56:12 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 35DAE1BEBA; Wed, 26 Aug 2020 11:56:05 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 3712814581 for ; Wed, 26 Aug 2020 11:56:03 +0200 (CEST) IronPort-SDR: NuYI3rLxVVWYXpjqRxHph5AoLVU0DHkwJOZc79TDF4czv1/qee9mwPk8vLNYcsqPB1pxn4UDvl cKJ0tfTQlsYw== X-IronPort-AV: E=McAfee;i="6000,8403,9724"; a="157301339" X-IronPort-AV: E=Sophos;i="5.76,355,1592895600"; d="scan'208";a="157301339" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2020 02:56:02 -0700 IronPort-SDR: JAUJ8jXHp2Wx0gvGTBEUL1+vT0oFkf6nBJP3TTe3G1Z/VS/lYjBON2zCgO46iwkyD4iYM+5RMW Ecf0+XK8QbqQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.76,355,1592895600"; d="scan'208";a="295319478" Received: from silpixa00400567.ir.intel.com ([10.237.214.190]) by orsmga003.jf.intel.com with ESMTP; 26 Aug 2020 02:55:59 -0700 From: Radu Nicolau To: dev@dpdk.org Cc: beilei.xing@intel.com, jia.guo@intel.com, bruce.richardson@intel.com, konstantin.ananyev@intel.com, jerinjacobk@gmail.com, david.marchand@redhat.com, fiona.trahe@intel.com, wei.zhao1@intel.com, ruifeng.wang@arm.com, qiming.yang@intel.com, qi.z.zhang@intel.com, Radu Nicolau Date: Wed, 26 Aug 2020 10:55:48 +0100 Message-Id: <20200826095552.82525-2-radu.nicolau@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200826095552.82525-1-radu.nicolau@intel.com> References: <1591870283-7776-1-git-send-email-radu.nicolau@intel.com> <20200826095552.82525-1-radu.nicolau@intel.com> Subject: [dpdk-dev] [PATCH v11 1/5] eal: add WC store functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add rte_write32_wc and rte_write32_wc_relaxed functions that implement 32bit stores using write combining memory protocol. Provided generic stubs and x86 implementation. Signed-off-by: Radu Nicolau Acked-by: Bruce Richardson --- lib/librte_eal/arm/include/rte_io_64.h | 12 +++++++ lib/librte_eal/include/generic/rte_io.h | 48 +++++++++++++++++++++++++ lib/librte_eal/x86/include/rte_io.h | 42 ++++++++++++++++++++++ 3 files changed, 102 insertions(+) diff --git a/lib/librte_eal/arm/include/rte_io_64.h b/lib/librte_eal/arm/include/rte_io_64.h index e5346240e..d07d9cb22 100644 --- a/lib/librte_eal/arm/include/rte_io_64.h +++ b/lib/librte_eal/arm/include/rte_io_64.h @@ -164,6 +164,18 @@ rte_write64(uint64_t value, volatile void *addr) rte_write64_relaxed(value, addr); } +static __rte_always_inline void +rte_write32_wc(uint32_t value, volatile void *addr) +{ + rte_write32(value, addr); +} + +static __rte_always_inline void +rte_write32_wc_relaxed(uint32_t value, volatile void *addr) +{ + rte_write32_relaxed(value, addr); +} + #ifdef __cplusplus } #endif diff --git a/lib/librte_eal/include/generic/rte_io.h b/lib/librte_eal/include/generic/rte_io.h index da457f7f7..0669baa0b 100644 --- a/lib/librte_eal/include/generic/rte_io.h +++ b/lib/librte_eal/include/generic/rte_io.h @@ -229,6 +229,40 @@ rte_write32(uint32_t value, volatile void *addr); static inline void rte_write64(uint64_t value, volatile void *addr); +/** + * Write a 32-bit value to I/O device memory address addr using write + * combining memory write protocol. Depending on the platform write combining + * may not be available and/or may be treated as a hint and the behavior may + * fallback to a regular store. + * + * @param value + * Value to write + * @param addr + * I/O memory address to write the value to + */ +__rte_experimental +static inline void +rte_write32_wc(uint32_t value, volatile void *addr); + +/** + * Write a 32-bit value to I/O device memory address addr using write + * combining memory write protocol. Depending on the platform write combining + * may not be available and/or may be treated as a hint and the behavior may + * fallback to a regular store. + * + * The relaxed version does not have additional I/O memory barrier, useful in + * accessing the device registers of integrated controllers which implicitly + * strongly ordered with respect to memory access. + * + * @param value + * Value to write + * @param addr + * I/O memory address to write the value to + */ +__rte_experimental +static inline void +rte_write32_wc_relaxed(uint32_t value, volatile void *addr); + #endif /* __DOXYGEN__ */ #ifndef RTE_OVERRIDE_IO_H @@ -345,6 +379,20 @@ rte_write64(uint64_t value, volatile void *addr) rte_write64_relaxed(value, addr); } +#ifndef RTE_NATIVE_WRITE32_WC +static __rte_always_inline void +rte_write32_wc(uint32_t value, volatile void *addr) +{ + rte_write32(value, addr); +} + +static __rte_always_inline void +rte_write32_wc_relaxed(uint32_t value, volatile void *addr) +{ + rte_write32_relaxed(value, addr); +} +#endif /* RTE_NATIVE_WRITE32_WC */ + #endif /* RTE_OVERRIDE_IO_H */ #endif /* _RTE_IO_H_ */ diff --git a/lib/librte_eal/x86/include/rte_io.h b/lib/librte_eal/x86/include/rte_io.h index 2db71b1b0..4f4ff8b87 100644 --- a/lib/librte_eal/x86/include/rte_io.h +++ b/lib/librte_eal/x86/include/rte_io.h @@ -9,8 +9,50 @@ extern "C" { #endif +#include "rte_cpuflags.h" + +#define RTE_NATIVE_WRITE32_WC #include "generic/rte_io.h" +/** + * @internal + * MOVDIRI wrapper. + */ +static __rte_always_inline void +_rte_x86_movdiri(uint32_t value, volatile void *addr) +{ + asm volatile( + /* MOVDIRI */ + ".byte 0x40, 0x0f, 0x38, 0xf9, 0x02" + : + : "a" (value), "d" (addr)); +} + +static __rte_always_inline void +rte_write32_wc_relaxed(uint32_t value, volatile void *addr) +{ + static int _x86_movdiri_flag = -1; + if (_x86_movdiri_flag == 1) { + _rte_x86_movdiri(value, addr); + } else if (_x86_movdiri_flag == 0) { + rte_write32_relaxed(value, addr); + } else { + _x86_movdiri_flag = + (rte_cpu_get_flag_enabled(RTE_CPUFLAG_MOVDIRI) > 0); + if (_x86_movdiri_flag == 1) + _rte_x86_movdiri(value, addr); + else + rte_write32_relaxed(value, addr); + } +} + +static __rte_always_inline void +rte_write32_wc(uint32_t value, volatile void *addr) +{ + rte_wmb(); + rte_write32_wc_relaxed(value, addr); +} + #ifdef __cplusplus } #endif From patchwork Wed Aug 26 09:55:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Radu Nicolau X-Patchwork-Id: 75986 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 66523A04B2; Wed, 26 Aug 2020 11:56:21 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4D2EB1BFE3; Wed, 26 Aug 2020 11:56:08 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 5B1C71BFE3 for ; Wed, 26 Aug 2020 11:56:06 +0200 (CEST) IronPort-SDR: /WLN4JbGD/u93hp1sNTs6jQ01LpTdIgFJakmNfs/KSgK8yKcAXzabc6+l57CPRbq7GxGroAbu8 HhBDNp+5fbMQ== X-IronPort-AV: E=McAfee;i="6000,8403,9724"; a="157301345" X-IronPort-AV: E=Sophos;i="5.76,355,1592895600"; d="scan'208";a="157301345" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2020 02:56:06 -0700 IronPort-SDR: q2aoME4Fa47f1+b0QUAfLn+vmZsS19ExCbJSJ6FDxqVs4Gf6mIyRdIUd437KsT0pcnCQ8GYPdP 3kACHwdpcuEg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.76,355,1592895600"; d="scan'208";a="295319490" Received: from silpixa00400567.ir.intel.com ([10.237.214.190]) by orsmga003.jf.intel.com with ESMTP; 26 Aug 2020 02:56:03 -0700 From: Radu Nicolau To: dev@dpdk.org Cc: beilei.xing@intel.com, jia.guo@intel.com, bruce.richardson@intel.com, konstantin.ananyev@intel.com, jerinjacobk@gmail.com, david.marchand@redhat.com, fiona.trahe@intel.com, wei.zhao1@intel.com, ruifeng.wang@arm.com, qiming.yang@intel.com, qi.z.zhang@intel.com, Radu Nicolau Date: Wed, 26 Aug 2020 10:55:49 +0100 Message-Id: <20200826095552.82525-3-radu.nicolau@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200826095552.82525-1-radu.nicolau@intel.com> References: <1591870283-7776-1-git-send-email-radu.nicolau@intel.com> <20200826095552.82525-1-radu.nicolau@intel.com> Subject: [dpdk-dev] [PATCH v11 2/5] net/i40e: use WC store to update queue tail registers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Performance improvement: use a write combining store instead of a regular mmio write to update queue tail registers. Signed-off-by: Radu Nicolau Acked-by: Bruce Richardson Reviewed-by: Wenzhuo Lu --- drivers/net/i40e/base/i40e_osdep.h | 5 +++++ drivers/net/i40e/i40e_rxtx.c | 8 ++++---- drivers/net/i40e/i40e_rxtx_vec_avx2.c | 4 ++-- drivers/net/i40e/i40e_rxtx_vec_sse.c | 4 ++-- 4 files changed, 13 insertions(+), 8 deletions(-) diff --git a/drivers/net/i40e/base/i40e_osdep.h b/drivers/net/i40e/base/i40e_osdep.h index 58be39677..9b5033024 100644 --- a/drivers/net/i40e/base/i40e_osdep.h +++ b/drivers/net/i40e/base/i40e_osdep.h @@ -138,6 +138,11 @@ static inline uint32_t i40e_read_addr(volatile void *addr) #define I40E_PCI_REG_WRITE_RELAXED(reg, value) \ rte_write32_relaxed((rte_cpu_to_le_32(value)), reg) +#define I40E_PCI_REG_WC_WRITE(reg, value) \ + rte_write32_wc((rte_cpu_to_le_32(value)), reg) +#define I40E_PCI_REG_WC_WRITE_RELAXED(reg, value) \ + rte_write32_wc_relaxed((rte_cpu_to_le_32(value)), reg) + #define I40E_WRITE_FLUSH(a) I40E_READ_REG(a, I40E_GLGEN_STAT) #define I40EVF_WRITE_FLUSH(a) I40E_READ_REG(a, I40E_VFGEN_RSTAT) diff --git a/drivers/net/i40e/i40e_rxtx.c b/drivers/net/i40e/i40e_rxtx.c index fe7f9200c..76d634091 100644 --- a/drivers/net/i40e/i40e_rxtx.c +++ b/drivers/net/i40e/i40e_rxtx.c @@ -760,7 +760,7 @@ i40e_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) if (nb_hold > rxq->rx_free_thresh) { rx_id = (uint16_t) ((rx_id == 0) ? (rxq->nb_rx_desc - 1) : (rx_id - 1)); - I40E_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); nb_hold = 0; } rxq->nb_rx_hold = nb_hold; @@ -938,7 +938,7 @@ i40e_recv_scattered_pkts(void *rx_queue, if (nb_hold > rxq->rx_free_thresh) { rx_id = (uint16_t)(rx_id == 0 ? (rxq->nb_rx_desc - 1) : (rx_id - 1)); - I40E_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); nb_hold = 0; } rxq->nb_rx_hold = nb_hold; @@ -1249,7 +1249,7 @@ i40e_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) (unsigned) tx_id, (unsigned) nb_tx); rte_cio_wmb(); - I40E_PCI_REG_WRITE_RELAXED(txq->qtx_tail, tx_id); + I40E_PCI_REG_WC_WRITE_RELAXED(txq->qtx_tail, tx_id); txq->tx_tail = tx_id; return nb_tx; @@ -1400,7 +1400,7 @@ tx_xmit_pkts(struct i40e_tx_queue *txq, txq->tx_tail = 0; /* Update the tx tail register */ - I40E_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + I40E_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail); return nb_pkts; } diff --git a/drivers/net/i40e/i40e_rxtx_vec_avx2.c b/drivers/net/i40e/i40e_rxtx_vec_avx2.c index 3bcef1363..178d8f4e2 100644 --- a/drivers/net/i40e/i40e_rxtx_vec_avx2.c +++ b/drivers/net/i40e/i40e_rxtx_vec_avx2.c @@ -134,7 +134,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - I40E_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); } #ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC @@ -921,7 +921,7 @@ i40e_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, txq->tx_tail = tx_id; - I40E_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + I40E_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail); return nb_pkts; } diff --git a/drivers/net/i40e/i40e_rxtx_vec_sse.c b/drivers/net/i40e/i40e_rxtx_vec_sse.c index 698518349..240ce478a 100644 --- a/drivers/net/i40e/i40e_rxtx_vec_sse.c +++ b/drivers/net/i40e/i40e_rxtx_vec_sse.c @@ -86,7 +86,7 @@ i40e_rxq_rearm(struct i40e_rx_queue *rxq) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - I40E_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + I40E_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); } #ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC @@ -733,7 +733,7 @@ i40e_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, txq->tx_tail = tx_id; - I40E_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + I40E_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail); return nb_pkts; } From patchwork Wed Aug 26 09:55:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Radu Nicolau X-Patchwork-Id: 75987 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5A97FA04B2; Wed, 26 Aug 2020 11:56:29 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8BFEE1C07D; Wed, 26 Aug 2020 11:56:11 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 4DA161C07D for ; Wed, 26 Aug 2020 11:56:09 +0200 (CEST) IronPort-SDR: UHU7V39j5DPEMRi1oVSkc909xQEWptMfGsKIAtuUY/OSNmMbpZDFO5ATXjJgbuOuJmO7iUwwPM nKulfv2+ATuQ== X-IronPort-AV: E=McAfee;i="6000,8403,9724"; a="157301349" X-IronPort-AV: E=Sophos;i="5.76,355,1592895600"; d="scan'208";a="157301349" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2020 02:56:09 -0700 IronPort-SDR: LNJK577B/ZMHwZpPgjJUZ5QeC0jW8/tYn9h5WcIFbdN6J0UgDkpHQezxZIRzFBoQX+WSXz51i6 Kq4dDi71L2sg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.76,355,1592895600"; d="scan'208";a="295319502" Received: from silpixa00400567.ir.intel.com ([10.237.214.190]) by orsmga003.jf.intel.com with ESMTP; 26 Aug 2020 02:56:06 -0700 From: Radu Nicolau To: dev@dpdk.org Cc: beilei.xing@intel.com, jia.guo@intel.com, bruce.richardson@intel.com, konstantin.ananyev@intel.com, jerinjacobk@gmail.com, david.marchand@redhat.com, fiona.trahe@intel.com, wei.zhao1@intel.com, ruifeng.wang@arm.com, qiming.yang@intel.com, qi.z.zhang@intel.com, Radu Nicolau Date: Wed, 26 Aug 2020 10:55:50 +0100 Message-Id: <20200826095552.82525-4-radu.nicolau@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200826095552.82525-1-radu.nicolau@intel.com> References: <1591870283-7776-1-git-send-email-radu.nicolau@intel.com> <20200826095552.82525-1-radu.nicolau@intel.com> Subject: [dpdk-dev] [PATCH v11 3/5] common/qat: use WC store to update queue tail registers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Performance improvement: use a write combining store instead of a regular mmio write to update queue tail registers. Signed-off-by: Radu Nicolau Acked-by: Fiona Trahe --- drivers/common/qat/qat_adf/adf_transport_access_macros.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/common/qat/qat_adf/adf_transport_access_macros.h b/drivers/common/qat/qat_adf/adf_transport_access_macros.h index 1eef5513f..504ffb723 100644 --- a/drivers/common/qat/qat_adf/adf_transport_access_macros.h +++ b/drivers/common/qat/qat_adf/adf_transport_access_macros.h @@ -9,6 +9,8 @@ /* CSR write macro */ #define ADF_CSR_WR(csrAddr, csrOffset, val) \ rte_write32(val, (((uint8_t *)csrAddr) + csrOffset)) +#define ADF_CSR_WC_WR(csrAddr, csrOffset, val) \ + rte_write32_wc(val, (((uint8_t *)csrAddr) + csrOffset)) /* CSR read macro */ #define ADF_CSR_RD(csrAddr, csrOffset) \ @@ -110,10 +112,10 @@ do { \ ADF_RING_CSR_RING_UBASE + (ring << 2), u_base); \ } while (0) #define WRITE_CSR_RING_HEAD(csr_base_addr, bank, ring, value) \ - ADF_CSR_WR(csr_base_addr, (ADF_RING_BUNDLE_SIZE * bank) + \ + ADF_CSR_WC_WR(csr_base_addr, (ADF_RING_BUNDLE_SIZE * bank) + \ ADF_RING_CSR_RING_HEAD + (ring << 2), value) #define WRITE_CSR_RING_TAIL(csr_base_addr, bank, ring, value) \ - ADF_CSR_WR(csr_base_addr, (ADF_RING_BUNDLE_SIZE * bank) + \ + ADF_CSR_WC_WR(csr_base_addr, (ADF_RING_BUNDLE_SIZE * bank) + \ ADF_RING_CSR_RING_TAIL + (ring << 2), value) #define WRITE_CSR_INT_SRCSEL(csr_base_addr, bank) \ do { \ From patchwork Wed Aug 26 09:55:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Radu Nicolau X-Patchwork-Id: 75988 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7B822A04B2; Wed, 26 Aug 2020 11:56:37 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B12F21C0B3; Wed, 26 Aug 2020 11:56:15 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 762311C0B0 for ; Wed, 26 Aug 2020 11:56:12 +0200 (CEST) IronPort-SDR: kKhI3ArMEBVPNqE/KhVyfIXiWvjZS+fwmS/8tNDvM2oSpcJReM0sNujq4kYtzcbkNXi5RZY8Jz dPWC07I9vuSw== X-IronPort-AV: E=McAfee;i="6000,8403,9724"; a="157301355" X-IronPort-AV: E=Sophos;i="5.76,355,1592895600"; d="scan'208";a="157301355" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2020 02:56:12 -0700 IronPort-SDR: 6oE0SX9qtM50Bp050RfzHOo5ww5SLa7DnJdRS8ne+e6wOzhwp6qdJLcZ96HytcEpZjwL6rr3/J jCpuOrwK4F6g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.76,355,1592895600"; d="scan'208";a="295319508" Received: from silpixa00400567.ir.intel.com ([10.237.214.190]) by orsmga003.jf.intel.com with ESMTP; 26 Aug 2020 02:56:09 -0700 From: Radu Nicolau To: dev@dpdk.org Cc: beilei.xing@intel.com, jia.guo@intel.com, bruce.richardson@intel.com, konstantin.ananyev@intel.com, jerinjacobk@gmail.com, david.marchand@redhat.com, fiona.trahe@intel.com, wei.zhao1@intel.com, ruifeng.wang@arm.com, qiming.yang@intel.com, qi.z.zhang@intel.com, Radu Nicolau Date: Wed, 26 Aug 2020 10:55:51 +0100 Message-Id: <20200826095552.82525-5-radu.nicolau@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200826095552.82525-1-radu.nicolau@intel.com> References: <1591870283-7776-1-git-send-email-radu.nicolau@intel.com> <20200826095552.82525-1-radu.nicolau@intel.com> Subject: [dpdk-dev] [PATCH v11 4/5] net/ixgbe: use WC store to update queue tail registers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Performance improvement: use a write combining store instead of a regular mmio write to update queue tail registers. Signed-off-by: Radu Nicolau Acked-by: Konstantin Ananyev Reviewed-by: Wenzhuo Lu --- drivers/net/ixgbe/base/ixgbe_osdep.h | 6 ++++++ drivers/net/ixgbe/ixgbe_rxtx.c | 15 ++++++++------- drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c | 4 ++-- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/drivers/net/ixgbe/base/ixgbe_osdep.h b/drivers/net/ixgbe/base/ixgbe_osdep.h index dc712b7c0..cacf72419 100644 --- a/drivers/net/ixgbe/base/ixgbe_osdep.h +++ b/drivers/net/ixgbe/base/ixgbe_osdep.h @@ -105,6 +105,12 @@ static inline uint32_t ixgbe_read_addr(volatile void* addr) #define IXGBE_PCI_REG_WRITE_RELAXED(reg, value) \ rte_write32_relaxed((rte_cpu_to_le_32(value)), reg) +#define IXGBE_PCI_REG_WC_WRITE(reg, value) \ + rte_write32_wc((rte_cpu_to_le_32(value)), reg) + +#define IXGBE_PCI_REG_WC_WRITE_RELAXED(reg, value) \ + rte_write32_wc_relaxed((rte_cpu_to_le_32(value)), reg) + #define IXGBE_PCI_REG_ADDR(hw, reg) \ ((volatile uint32_t *)((char *)(hw)->hw_addr + (reg))) diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxtx.c index 977ecf513..29d385c06 100644 --- a/drivers/net/ixgbe/ixgbe_rxtx.c +++ b/drivers/net/ixgbe/ixgbe_rxtx.c @@ -308,7 +308,7 @@ tx_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, /* update tail pointer */ rte_wmb(); - IXGBE_PCI_REG_WRITE_RELAXED(txq->tdt_reg_addr, txq->tx_tail); + IXGBE_PCI_REG_WC_WRITE_RELAXED(txq->tdt_reg_addr, txq->tx_tail); return nb_pkts; } @@ -946,7 +946,7 @@ ixgbe_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, PMD_TX_LOG(DEBUG, "port_id=%u queue_id=%u tx_tail=%u nb_tx=%u", (unsigned) txq->port_id, (unsigned) txq->queue_id, (unsigned) tx_id, (unsigned) nb_tx); - IXGBE_PCI_REG_WRITE_RELAXED(txq->tdt_reg_addr, tx_id); + IXGBE_PCI_REG_WC_WRITE_RELAXED(txq->tdt_reg_addr, tx_id); txq->tx_tail = tx_id; return nb_tx; @@ -1692,7 +1692,7 @@ rx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, /* update tail pointer */ rte_wmb(); - IXGBE_PCI_REG_WRITE_RELAXED(rxq->rdt_reg_addr, + IXGBE_PCI_REG_WC_WRITE_RELAXED(rxq->rdt_reg_addr, cur_free_trigger); } @@ -1918,7 +1918,7 @@ ixgbe_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, (unsigned) nb_rx); rx_id = (uint16_t) ((rx_id == 0) ? (rxq->nb_rx_desc - 1) : (rx_id - 1)); - IXGBE_PCI_REG_WRITE(rxq->rdt_reg_addr, rx_id); + IXGBE_PCI_REG_WC_WRITE(rxq->rdt_reg_addr, rx_id); nb_hold = 0; } rxq->nb_rx_hold = nb_hold; @@ -2096,8 +2096,9 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, if (!ixgbe_rx_alloc_bufs(rxq, false)) { rte_wmb(); - IXGBE_PCI_REG_WRITE_RELAXED(rxq->rdt_reg_addr, - next_rdt); + IXGBE_PCI_REG_WC_WRITE_RELAXED( + rxq->rdt_reg_addr, + next_rdt); nb_hold -= rxq->rx_free_thresh; } else { PMD_RX_LOG(DEBUG, "RX bulk alloc failed " @@ -2262,7 +2263,7 @@ ixgbe_recv_pkts_lro(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, rxq->port_id, rxq->queue_id, rx_id, nb_hold, nb_rx); rte_wmb(); - IXGBE_PCI_REG_WRITE_RELAXED(rxq->rdt_reg_addr, prev_id); + IXGBE_PCI_REG_WC_WRITE_RELAXED(rxq->rdt_reg_addr, prev_id); nb_hold = 0; } diff --git a/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c b/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c index 517ca3166..e77a7f31c 100644 --- a/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c +++ b/drivers/net/ixgbe/ixgbe_rxtx_vec_sse.c @@ -90,7 +90,7 @@ ixgbe_rxq_rearm(struct ixgbe_rx_queue *rxq) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - IXGBE_PCI_REG_WRITE(rxq->rdt_reg_addr, rx_id); + IXGBE_PCI_REG_WC_WRITE(rxq->rdt_reg_addr, rx_id); } #ifdef RTE_LIBRTE_SECURITY @@ -697,7 +697,7 @@ ixgbe_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, txq->tx_tail = tx_id; - IXGBE_PCI_REG_WRITE(txq->tdt_reg_addr, txq->tx_tail); + IXGBE_PCI_REG_WC_WRITE(txq->tdt_reg_addr, txq->tx_tail); return nb_pkts; } From patchwork Wed Aug 26 09:55:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Radu Nicolau X-Patchwork-Id: 75989 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5695BA04B2; Wed, 26 Aug 2020 11:56:46 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 087BD1C0BE; Wed, 26 Aug 2020 11:56:18 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 874CA1C0AF for ; Wed, 26 Aug 2020 11:56:15 +0200 (CEST) IronPort-SDR: 2hkVeadccgGOCvFJRul/+kf8vzCsVkGyTLE0+gYqZOqt9onzRDsN/G2k1hCCRU6AM2J1bo2OFr zowMA8iRbM+Q== X-IronPort-AV: E=McAfee;i="6000,8403,9724"; a="157301368" X-IronPort-AV: E=Sophos;i="5.76,355,1592895600"; d="scan'208";a="157301368" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Aug 2020 02:56:15 -0700 IronPort-SDR: UJhb8nqxcYzAW7EhYeepnp1i0mBo6f7dv6RFvFrTLwklf3fwnxsdERYI/WSafJH7Ewbcg1CvWZ Wd7mWeafdd5A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.76,355,1592895600"; d="scan'208";a="295319516" Received: from silpixa00400567.ir.intel.com ([10.237.214.190]) by orsmga003.jf.intel.com with ESMTP; 26 Aug 2020 02:56:12 -0700 From: Radu Nicolau To: dev@dpdk.org Cc: beilei.xing@intel.com, jia.guo@intel.com, bruce.richardson@intel.com, konstantin.ananyev@intel.com, jerinjacobk@gmail.com, david.marchand@redhat.com, fiona.trahe@intel.com, wei.zhao1@intel.com, ruifeng.wang@arm.com, qiming.yang@intel.com, qi.z.zhang@intel.com, Radu Nicolau Date: Wed, 26 Aug 2020 10:55:52 +0100 Message-Id: <20200826095552.82525-6-radu.nicolau@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200826095552.82525-1-radu.nicolau@intel.com> References: <1591870283-7776-1-git-send-email-radu.nicolau@intel.com> <20200826095552.82525-1-radu.nicolau@intel.com> Subject: [dpdk-dev] [PATCH v11 5/5] net/ice: use WC store to update queue tail registers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Performance improvement: use a write combining store instead of a regular mmio write to update queue tail registers. Signed-off-by: Radu Nicolau Reviewed-by: Wenzhuo Lu --- drivers/net/ice/base/ice_osdep.h | 1 + drivers/net/ice/ice_rxtx.c | 6 +++--- drivers/net/ice/ice_rxtx_vec_avx2.c | 4 ++-- drivers/net/ice/ice_rxtx_vec_sse.c | 4 ++-- 4 files changed, 8 insertions(+), 7 deletions(-) diff --git a/drivers/net/ice/base/ice_osdep.h b/drivers/net/ice/base/ice_osdep.h index 360e435b8..c793f1e38 100644 --- a/drivers/net/ice/base/ice_osdep.h +++ b/drivers/net/ice/base/ice_osdep.h @@ -165,6 +165,7 @@ do { \ #endif #define ICE_PCI_REG_WRITE(reg, value) writel(value, reg) +#define ICE_PCI_REG_WC_WRITE(reg, value) rte_write32_wc(value, reg) #define ICE_READ_REG(hw, reg) rd32(hw, reg) #define ICE_WRITE_REG(hw, reg, value) wr32(hw, reg, value) diff --git a/drivers/net/ice/ice_rxtx.c b/drivers/net/ice/ice_rxtx.c index 2e1f06d2c..d23841623 100644 --- a/drivers/net/ice/ice_rxtx.c +++ b/drivers/net/ice/ice_rxtx.c @@ -1741,7 +1741,7 @@ ice_recv_scattered_pkts(void *rx_queue, rx_id = (uint16_t)(rx_id == 0 ? (rxq->nb_rx_desc - 1) : (rx_id - 1)); /* write TAIL register */ - ICE_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + ICE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); nb_hold = 0; } rxq->nb_rx_hold = nb_hold; @@ -2131,7 +2131,7 @@ ice_recv_pkts(void *rx_queue, rx_id = (uint16_t)(rx_id == 0 ? (rxq->nb_rx_desc - 1) : (rx_id - 1)); /* write TAIL register */ - ICE_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + ICE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); nb_hold = 0; } rxq->nb_rx_hold = nb_hold; @@ -2846,7 +2846,7 @@ tx_xmit_pkts(struct ice_tx_queue *txq, txq->tx_tail = 0; /* Update the tx tail register */ - ICE_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + ICE_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail); return nb_pkts; } diff --git a/drivers/net/ice/ice_rxtx_vec_avx2.c b/drivers/net/ice/ice_rxtx_vec_avx2.c index be50677c2..d9d475863 100644 --- a/drivers/net/ice/ice_rxtx_vec_avx2.c +++ b/drivers/net/ice/ice_rxtx_vec_avx2.c @@ -129,7 +129,7 @@ ice_rxq_rearm(struct ice_rx_queue *rxq) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - ICE_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + ICE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); } static inline uint16_t @@ -810,7 +810,7 @@ ice_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, txq->tx_tail = tx_id; - ICE_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + ICE_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail); return nb_pkts; } diff --git a/drivers/net/ice/ice_rxtx_vec_sse.c b/drivers/net/ice/ice_rxtx_vec_sse.c index 382ef31f3..3de162af0 100644 --- a/drivers/net/ice/ice_rxtx_vec_sse.c +++ b/drivers/net/ice/ice_rxtx_vec_sse.c @@ -78,7 +78,7 @@ ice_rxq_rearm(struct ice_rx_queue *rxq) (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); /* Update the tail pointer on the NIC */ - ICE_PCI_REG_WRITE(rxq->qrx_tail, rx_id); + ICE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); } static inline void @@ -583,7 +583,7 @@ ice_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, txq->tx_tail = tx_id; - ICE_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + ICE_PCI_REG_WC_WRITE(txq->qtx_tail, txq->tx_tail); return nb_pkts; }