From patchwork Wed Sep 9 15:57:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Richardson X-Patchwork-Id: 77076 X-Patchwork-Delegate: ajit.khaparde@broadcom.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6A737A04B5; Wed, 9 Sep 2020 17:57:38 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 498751C115; Wed, 9 Sep 2020 17:57:38 +0200 (CEST) Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by dpdk.org (Postfix) with ESMTP id 0BA721C112 for ; Wed, 9 Sep 2020 17:57:36 +0200 (CEST) Received: by mail-pf1-f195.google.com with SMTP id w7so2541467pfi.4 for ; Wed, 09 Sep 2020 08:57:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=hISRRUNg7+Niq+Fa+X0zHfPM0eE3nGBX4GiZaG41v9g=; b=JCG3HGns1JbDoBnbvPcAukK6Jyq08V2xHQmV1p9Gl46Qr5DG+vHdMMlcixNdWKrkV6 LQXA1ERatf5vNUI2CtwCoyUBVQzZqdPDex4gbvDyzFSLHnJxmUayO5qU5dGnQdGpMQBH 2aM3zSQRcr3ryN0Mg4oTwiAxdkGuwk4Uoe1lQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=hISRRUNg7+Niq+Fa+X0zHfPM0eE3nGBX4GiZaG41v9g=; b=q0+1282pkGhPTKETX0tQu/qXjpDtnScf6I/8mkb/bgnYj7mL7VF7d7iZ5J0ZXcVv2V qG12xVTgAvAndKbbjswtp7DwhbeJzbbzqx7vS4sLpK9YsO553qHz5jNwK/+Uvu8WN3e2 KxKaK6zLkzTSSk4+vcs0w/RABMuNfH+ASSfC3lizueC7IsRsYe1AjTotM3olyae2Ibbr fWtV64IZmj7fKDlHdIndnDch5wzOKct/0jWr6plj3j20OK3LETRNvrCmqifQ6tzALqkn UBORkMFyy41+b/KLCtZzalrYco/5K4CmfQcAEbX7jkB7iuhUfEq5CCWVt0dAY5SHSrXX q5dg== X-Gm-Message-State: AOAM530wapUkr508sXtlxhADLP1sUHjhiTPKrdpRdRLbjBvNkn35pshx 4hqN6y5RcIEsdeLpjI+e8g9VHA== X-Google-Smtp-Source: ABdhPJzip5BsKG6K9FjKTjrvxzMyjDMn5VbqVeeXyPY+QeTeLhT5NvYjOCocmZdkQ/BsmqxmRcsJxQ== X-Received: by 2002:a17:902:aa8d:: with SMTP id d13mr1447191plr.124.1599667055212; Wed, 09 Sep 2020 08:57:35 -0700 (PDT) Received: from localhost.localdomain ([192.19.231.250]) by smtp.gmail.com with ESMTPSA id kt18sm77948pjb.56.2020.09.09.08.57.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Sep 2020 08:57:34 -0700 (PDT) From: Lance Richardson To: Ajit Khaparde , Somnath Kotur Cc: dev@dpdk.org Date: Wed, 9 Sep 2020 11:57:30 -0400 Message-Id: <20200909155730.29182-1-lance.richardson@broadcom.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH 12/12] net/bnxt: transmit vector mode improvements X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Improve performance of vector burst transmit function by processing multiple packets per inner loop iteration. Signed-off-by: Lance Richardson --- drivers/net/bnxt/bnxt_rxtx_vec_sse.c | 96 +++++++++++++++++++--------- 1 file changed, 66 insertions(+), 30 deletions(-) diff --git a/drivers/net/bnxt/bnxt_rxtx_vec_sse.c b/drivers/net/bnxt/bnxt_rxtx_vec_sse.c index 362992ceb2..dace2f8a16 100644 --- a/drivers/net/bnxt/bnxt_rxtx_vec_sse.c +++ b/drivers/net/bnxt/bnxt_rxtx_vec_sse.c @@ -316,13 +316,14 @@ bnxt_tx_cmp_vec(struct bnxt_tx_queue *txq, int nr_pkts) struct rte_mbuf **free = txq->free; uint16_t cons = txr->tx_cons; unsigned int blk = 0; + uint32_t ring_mask = txr->tx_ring_struct->ring_mask; while (nr_pkts--) { struct bnxt_sw_tx_bd *tx_buf; struct rte_mbuf *mbuf; tx_buf = &txr->tx_buf_ring[cons]; - cons = RING_NEXT(txr->tx_ring_struct, cons); + cons = (cons + 1) & ring_mask; mbuf = rte_pktmbuf_prefree_seg(tx_buf->mbuf); if (unlikely(mbuf == NULL)) continue; @@ -376,18 +377,40 @@ bnxt_handle_tx_cp_vec(struct bnxt_tx_queue *txq) } } +static inline void +bnxt_xmit_one(struct rte_mbuf *mbuf, struct tx_bd_long *txbd, + struct bnxt_sw_tx_bd *tx_buf) +{ + __m128i desc; + + tx_buf->mbuf = mbuf; + tx_buf->nr_bds = 1; + + desc = _mm_set_epi64x(mbuf->buf_iova + mbuf->data_off, + bnxt_xmit_flags_len(mbuf->data_len, + TX_BD_FLAGS_NOCMPL)); + desc = _mm_blend_epi16(desc, _mm_set_epi16(0, 0, 0, 0, 0, 0, + mbuf->data_len, 0), 0x02); + _mm_store_si128((void *)txbd, desc); +} + static uint16_t -bnxt_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, +bnxt_xmit_fixed_burst_vec(struct bnxt_tx_queue *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) { - struct bnxt_tx_queue *txq = tx_queue; struct bnxt_tx_ring_info *txr = txq->tx_ring; - uint16_t prod = txr->tx_prod; - struct rte_mbuf *tx_mbuf; - struct tx_bd_long *txbd = NULL; + uint16_t tx_prod = txr->tx_prod; + struct tx_bd_long *txbd; struct bnxt_sw_tx_bd *tx_buf; uint16_t to_send; + txbd = &txr->tx_desc_ring[tx_prod]; + tx_buf = &txr->tx_buf_ring[tx_prod]; + + /* Prefetch next transmit buffer descriptors. */ + rte_prefetch0(txbd); + rte_prefetch0(txbd + 3); + nb_pkts = RTE_MIN(nb_pkts, bnxt_tx_avail(txq)); if (unlikely(nb_pkts == 0)) @@ -395,33 +418,35 @@ bnxt_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, /* Handle TX burst request */ to_send = nb_pkts; - while (to_send) { - tx_mbuf = *tx_pkts++; - rte_prefetch0(tx_mbuf); - - tx_buf = &txr->tx_buf_ring[prod]; - tx_buf->mbuf = tx_mbuf; - tx_buf->nr_bds = 1; - - txbd = &txr->tx_desc_ring[prod]; - txbd->address = tx_mbuf->buf_iova + tx_mbuf->data_off; - txbd->len = tx_mbuf->data_len; - txbd->flags_type = bnxt_xmit_flags_len(tx_mbuf->data_len, - TX_BD_FLAGS_NOCMPL); - prod = RING_NEXT(txr->tx_ring_struct, prod); - to_send--; + while (to_send >= RTE_BNXT_DESCS_PER_LOOP) { + /* Prefetch next transmit buffer descriptors. */ + rte_prefetch0(txbd + 4); + rte_prefetch0(txbd + 7); + + bnxt_xmit_one(tx_pkts[0], txbd++, tx_buf++); + bnxt_xmit_one(tx_pkts[1], txbd++, tx_buf++); + bnxt_xmit_one(tx_pkts[2], txbd++, tx_buf++); + bnxt_xmit_one(tx_pkts[3], txbd++, tx_buf++); + + to_send -= RTE_BNXT_DESCS_PER_LOOP; + tx_pkts += RTE_BNXT_DESCS_PER_LOOP; } - /* Request a completion for last packet in burst */ - if (txbd) { - txbd->opaque = nb_pkts; - txbd->flags_type &= ~TX_BD_LONG_FLAGS_NO_CMPL; + while (to_send) { + bnxt_xmit_one(tx_pkts[0], txbd++, tx_buf++); + to_send--; + tx_pkts++; } + /* Request a completion for the final packet of burst. */ rte_compiler_barrier(); - bnxt_db_write(&txr->tx_db, prod); + txbd[-1].opaque = nb_pkts; + txbd[-1].flags_type &= ~TX_BD_LONG_FLAGS_NO_CMPL; - txr->tx_prod = prod; + tx_prod = RING_ADV(txr->tx_ring_struct, tx_prod, nb_pkts); + bnxt_db_write(&txr->tx_db, tx_prod); + + txr->tx_prod = tx_prod; return nb_pkts; } @@ -432,6 +457,8 @@ bnxt_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, { int nb_sent = 0; struct bnxt_tx_queue *txq = tx_queue; + struct bnxt_tx_ring_info *txr = txq->tx_ring; + uint16_t ring_size = txr->tx_ring_struct->ring_size; /* Tx queue was stopped; wait for it to be restarted */ if (unlikely(!txq->tx_started)) { @@ -446,10 +473,19 @@ bnxt_xmit_pkts_vec(void *tx_queue, struct rte_mbuf **tx_pkts, while (nb_pkts) { uint16_t ret, num; + /* + * Ensure that no more than RTE_BNXT_MAX_TX_BURST packets + * are transmitted before the next completion. + */ num = RTE_MIN(nb_pkts, RTE_BNXT_MAX_TX_BURST); - ret = bnxt_xmit_fixed_burst_vec(tx_queue, - &tx_pkts[nb_sent], - num); + + /* + * Ensure that a ring wrap does not occur within a call to + * bnxt_xmit_fixed_burst_vec(). + */ + num = RTE_MIN(num, + ring_size - (txr->tx_prod & (ring_size - 1))); + ret = bnxt_xmit_fixed_burst_vec(txq, &tx_pkts[nb_sent], num); nb_sent += ret; nb_pkts -= ret; if (ret < num)