From patchwork Mon Nov 21 15:01:10 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Rybchenko X-Patchwork-Id: 17154 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id F14AFD66A; Mon, 21 Nov 2016 16:03:49 +0100 (CET) Received: from nbfkord-smmo02.seg.att.com (nbfkord-smmo02.seg.att.com [209.65.160.78]) by dpdk.org (Postfix) with ESMTP id 5CE433977 for ; Mon, 21 Nov 2016 16:01:51 +0100 (CET) Received: from unknown [12.187.104.26] (EHLO nbfkord-smmo02.seg.att.com) by nbfkord-smmo02.seg.att.com(mxl_mta-7.2.4-7) with ESMTP id f5c03385.2b9284833940.1541321.00-2472.3424293.nbfkord-smmo02.seg.att.com (envelope-from ); Mon, 21 Nov 2016 15:01:51 +0000 (UTC) X-MXL-Hash: 58330c5f7b393d7b-df8107b19eced5e0eabf34f1514cf383b0ef6e7a Received: from unknown [12.187.104.26] by nbfkord-smmo02.seg.att.com(mxl_mta-7.2.4-7) with SMTP id 55c03385.0.1541303.00-2373.3424249.nbfkord-smmo02.seg.att.com (envelope-from ); Mon, 21 Nov 2016 15:01:43 +0000 (UTC) X-MXL-Hash: 58330c5702b547ad-50c401828f477c19daabc6516cec1845095e9ac2 Received: from ocex03.SolarFlarecom.com (10.20.40.36) by ocex03.SolarFlarecom.com (10.20.40.36) with Microsoft SMTP Server (TLS) id 15.0.1044.25; Mon, 21 Nov 2016 07:01:23 -0800 Received: from opal.uk.solarflarecom.com (10.17.10.1) by ocex03.SolarFlarecom.com (10.20.40.36) with Microsoft SMTP Server (TLS) id 15.0.1044.25 via Frontend Transport; Mon, 21 Nov 2016 07:01:22 -0800 Received: from uklogin.uk.solarflarecom.com (uklogin.uk.solarflarecom.com [10.17.10.10]) by opal.uk.solarflarecom.com (8.13.8/8.13.8) with ESMTP id uALF1LS1007266; Mon, 21 Nov 2016 15:01:21 GMT Received: from uklogin.uk.solarflarecom.com (localhost.localdomain [127.0.0.1]) by uklogin.uk.solarflarecom.com (8.13.8/8.13.8) with ESMTP id uALF1J3l006765; Mon, 21 Nov 2016 15:01:21 GMT From: Andrew Rybchenko To: CC: Ivan Malov Date: Mon, 21 Nov 2016 15:01:10 +0000 Message-ID: <1479740470-6723-57-git-send-email-arybchenko@solarflare.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1479740470-6723-1-git-send-email-arybchenko@solarflare.com> References: <1479740470-6723-1-git-send-email-arybchenko@solarflare.com> MIME-Version: 1.0 X-AnalysisOut: [v=2.1 cv=UI/baXry c=1 sm=1 tr=0 a=8BlWFWvVlq5taO8ncb8nKg==] X-AnalysisOut: [:17 a=L24OOQBejmoA:10 a=pK7X0mNQAAAA:8 a=zRKbQ67AAAAA:8 a=] X-AnalysisOut: [x3NtvZPS5Aqf7GUG1cQA:9 a=XvkekhfmQLjxaZK_:21 a=gBNrHQtNUfY] X-AnalysisOut: [wr3WT:21 a=5HA-qpC1VU4iIGLgRoNS:22 a=PA03WX8tBzeizutn5_OT:] X-AnalysisOut: [22] X-Spam: [F=0.3521600375; CM=0.500; S=0.352(2015072901)] X-MAIL-FROM: X-SOURCE-IP: [12.187.104.26] Subject: [dpdk-dev] [PATCH 56/56] net/sfc: add callback to send bursts of packets X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Ivan Malov Reviewed-by: Andy Moreton Signed-off-by: Ivan Malov Signed-off-by: Andrew Rybchenko --- drivers/net/sfc/efx/sfc_ethdev.c | 2 + drivers/net/sfc/efx/sfc_ev.c | 22 +++++++- drivers/net/sfc/efx/sfc_tweak.h | 7 +++ drivers/net/sfc/efx/sfc_tx.c | 115 +++++++++++++++++++++++++++++++++++++++ drivers/net/sfc/efx/sfc_tx.h | 18 ++++++ 5 files changed, 162 insertions(+), 2 deletions(-) diff --git a/drivers/net/sfc/efx/sfc_ethdev.c b/drivers/net/sfc/efx/sfc_ethdev.c index 98e5d83..4f94653 100644 --- a/drivers/net/sfc/efx/sfc_ethdev.c +++ b/drivers/net/sfc/efx/sfc_ethdev.c @@ -386,6 +386,7 @@ sfc_eth_dev_init(struct rte_eth_dev *dev) dev->dev_ops = &sfc_eth_dev_ops; dev->rx_pkt_burst = &sfc_recv_pkts; + dev->tx_pkt_burst = &sfc_xmit_pkts; sfc_adapter_unlock(sa); @@ -424,6 +425,7 @@ sfc_eth_dev_uninit(struct rte_eth_dev *dev) dev->dev_ops = NULL; dev->rx_pkt_burst = NULL; + dev->tx_pkt_burst = NULL; sfc_kvargs_cleanup(sa); diff --git a/drivers/net/sfc/efx/sfc_ev.c b/drivers/net/sfc/efx/sfc_ev.c index b3cecbb..8b3f098 100644 --- a/drivers/net/sfc/efx/sfc_ev.c +++ b/drivers/net/sfc/efx/sfc_ev.c @@ -142,9 +142,27 @@ static boolean_t sfc_ev_tx(void *arg, uint32_t label, uint32_t id) { struct sfc_evq *evq = arg; + struct sfc_txq *txq; + unsigned int stop; + unsigned int delta; - sfc_err(evq->sa, "EVQ %u unexpected Tx event", evq->evq_index); - return B_TRUE; + txq = evq->txq; + + SFC_ASSERT(txq != NULL); + SFC_ASSERT(txq->evq == evq); + + if (unlikely((txq->state & SFC_TXQ_STARTED) == 0)) + goto done; + + stop = (id + 1) & txq->ptr_mask; + id = txq->pending & txq->ptr_mask; + + delta = (stop >= id) ? (stop - id) : (txq->ptr_mask + 1 - id + stop); + + txq->pending += delta; + +done: + return B_FALSE; } static boolean_t diff --git a/drivers/net/sfc/efx/sfc_tweak.h b/drivers/net/sfc/efx/sfc_tweak.h index 24cb9f4..60b618b 100644 --- a/drivers/net/sfc/efx/sfc_tweak.h +++ b/drivers/net/sfc/efx/sfc_tweak.h @@ -41,4 +41,11 @@ */ #define SFC_RX_REFILL_BULK (RTE_CACHE_LINE_SIZE / sizeof(efx_qword_t)) +/** + * Make the transmit path reap at least one time per a burst; + * this improves cache locality because the same mbufs may be used to send + * subsequent bursts in certain cases because of well-timed reap + */ +#define SFC_TX_XMIT_PKTS_REAP_AT_LEAST_ONCE 0 + #endif /* _SFC_TWEAK_H_ */ diff --git a/drivers/net/sfc/efx/sfc_tx.c b/drivers/net/sfc/efx/sfc_tx.c index 99d845a..92e81a8 100644 --- a/drivers/net/sfc/efx/sfc_tx.c +++ b/drivers/net/sfc/efx/sfc_tx.c @@ -32,6 +32,7 @@ #include "sfc_log.h" #include "sfc_ev.h" #include "sfc_tx.h" +#include "sfc_tweak.h" /* * Maximum number of TX queue flush attempts in case of @@ -529,3 +530,117 @@ sfc_tx_stop(struct sfc_adapter *sa) efx_tx_fini(sa->nic); } + +uint16_t +sfc_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) +{ + struct sfc_txq *txq = (struct sfc_txq *)tx_queue; + unsigned int added = txq->added; + unsigned int pushed = added; + unsigned int pkts_sent = 0; + efx_desc_t *pend = &txq->pend_desc[0]; + const unsigned int hard_max_fill = EFX_TXQ_LIMIT(txq->ptr_mask + 1); + const unsigned int soft_max_fill = hard_max_fill - + SFC_TX_MAX_PKT_DESC; + unsigned int fill_level = added - txq->completed; + boolean_t reap_done; + int rc; + struct rte_mbuf **pktp; + + if (unlikely((txq->state & SFC_TXQ_RUNNING) == 0)) + goto done; + + /* + * If insufficient space for a single packet is present, + * we should reap; otherwise, we shouldn't do that all the time + * to avoid latency increase + */ + reap_done = (fill_level > soft_max_fill); + + if (reap_done) { + sfc_tx_reap(txq); + /* + * Recalculate fill level since 'txq->completed' + * might have changed on reap + */ + fill_level = added - txq->completed; + } + + for (pkts_sent = 0, pktp = &tx_pkts[0]; + (pkts_sent < nb_pkts) && (fill_level <= soft_max_fill); + pkts_sent++, pktp++) { + struct rte_mbuf *m_seg = *pktp; + size_t pkt_len = m_seg->pkt_len; + unsigned int pkt_descs = 0; + + for (; m_seg != NULL; m_seg = m_seg->next) { + efsys_dma_addr_t next_frag; + size_t seg_len; + + seg_len = m_seg->data_len; + next_frag = rte_mbuf_data_dma_addr(m_seg); + + do { + efsys_dma_addr_t frag_addr = next_frag; + size_t frag_len; + + next_frag = RTE_ALIGN(frag_addr + 1, + SFC_TX_SEG_BOUNDARY); + frag_len = MIN(next_frag - frag_addr, seg_len); + seg_len -= frag_len; + pkt_len -= frag_len; + + efx_tx_qdesc_dma_create(txq->common, + frag_addr, frag_len, + (pkt_len == 0), + pend++); + + pkt_descs++; + } while (seg_len != 0); + } + + added += pkt_descs; + + fill_level += pkt_descs; + if (unlikely(fill_level > hard_max_fill)) { + /* + * Our estimation for maximum number of descriptors + * required to send a packet seems to be wrong. + * Try to reap (if we haven't yet). + */ + if (!reap_done) { + sfc_tx_reap(txq); + reap_done = B_TRUE; + fill_level = added - txq->completed; + if (fill_level > hard_max_fill) { + pend -= pkt_descs; + break; + } + } else { + pend -= pkt_descs; + break; + } + } + + /* Assign mbuf to the last used desc */ + txq->sw_ring[(added - 1) & txq->ptr_mask].mbuf = *pktp; + } + + if (likely(pkts_sent > 0)) { + rc = efx_tx_qdesc_post(txq->common, txq->pend_desc, + pend - &txq->pend_desc[0], + txq->completed, &txq->added); + SFC_ASSERT(rc == 0); + + if (likely(pushed != txq->added)) + efx_tx_qpush(txq->common, txq->added, pushed); + } + +#if SFC_TX_XMIT_PKTS_REAP_AT_LEAST_ONCE + if (!reap_done) + sfc_tx_reap(txq); +#endif + +done: + return pkts_sent; +} diff --git a/drivers/net/sfc/efx/sfc_tx.h b/drivers/net/sfc/efx/sfc_tx.h index 47970f9..ae13a16 100644 --- a/drivers/net/sfc/efx/sfc_tx.h +++ b/drivers/net/sfc/efx/sfc_tx.h @@ -39,6 +39,21 @@ extern "C" { #endif +/** + * Estimated maximum number of segments that transmit packet consists of; + * it is determined with respect to the expectation of a packet to consist + * of a header plus a couple of data segments one of those crossing 4K page; + * it is used by transmit path to avoid redundant reaping and, thus, + * to avoid increase of latency + */ +#define SFC_TX_MAX_PKT_DESC 4 + +/** + * A segment must not cross 4K boundary + * (this is a requirement of NIC TX descriptors) + */ +#define SFC_TX_SEG_BOUNDARY 4096 + struct sfc_adapter; struct sfc_evq; @@ -100,6 +115,9 @@ void sfc_tx_qstop(struct sfc_adapter *sa, unsigned int sw_index); int sfc_tx_start(struct sfc_adapter *sa); void sfc_tx_stop(struct sfc_adapter *sa); +uint16_t sfc_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, + uint16_t nb_pkts); + #ifdef __cplusplus } #endif