From patchwork Tue Feb 7 10:16:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liu, Mingxia" X-Patchwork-Id: 123240 X-Patchwork-Delegate: qi.z.zhang@intel.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3456541C2D; Tue, 7 Feb 2023 12:14:25 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 854F942D30; Tue, 7 Feb 2023 12:14:19 +0100 (CET) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 18AB842D0E for ; Tue, 7 Feb 2023 12:14:16 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675768457; x=1707304457; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RGLjnAkFAWZWToBk81TeoejET+iw9yyU7ni2n4Krrkw=; b=R/W8+bZJ9iMj9/o4YNTPGMHVuqUdALqoJR+PY2cQpy6BByNjrFZ49pHP besCdg+qkDIhzxw6TIXbIwn7OqN3nb6kBfqmM1NnukVjHitkCcyy9LfT8 5HxmDeE8lt5CzG20+1GCN8Q3Jense6KKMyWmzORKurEJpeNqZRiQ+PNV3 GADa7YAJrueNWBtDwYb82xncgHAt0IrUYDaqoQ8xoxtNKtyo0IZ7IfuYC SiREYDYLjUtpynCaABCkZKDvno2v7z4b8KN+6uqpKdKQwt5HaytY9EqSx lmNOIvRY+RRNMrNAPDkM8sE2SGWYfmbz2Y3sAYZdSQjO5AFWhvFvCrD8R A==; X-IronPort-AV: E=McAfee;i="6500,9779,10613"; a="415698996" X-IronPort-AV: E=Sophos;i="5.97,278,1669104000"; d="scan'208";a="415698996" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Feb 2023 03:14:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10613"; a="809486956" X-IronPort-AV: E=Sophos;i="5.97,278,1669104000"; d="scan'208";a="809486956" Received: from dpdk-mingxial-01.sh.intel.com ([10.67.119.167]) by fmsmga001.fm.intel.com with ESMTP; 07 Feb 2023 03:14:13 -0800 From: Mingxia Liu To: dev@dpdk.org, qi.z.zhang@intel.com, jingjing.wu@intel.com, beilei.xing@intel.com Cc: Mingxia Liu , Wenjun Wu Subject: [PATCH v6 3/6] common/idpf: support single q scatter RX datapath Date: Tue, 7 Feb 2023 10:16:47 +0000 Message-Id: <20230207101650.2402452-4-mingxia.liu@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230207101650.2402452-1-mingxia.liu@intel.com> References: <20230207100859.2401709-2-mingxia.liu@intel.com> <20230207101650.2402452-1-mingxia.liu@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch add single q recv scatter rx function. Signed-off-by: Mingxia Liu Signed-off-by: Wenjun Wu --- drivers/common/idpf/idpf_common_rxtx.c | 135 +++++++++++++++++++++++++ drivers/common/idpf/idpf_common_rxtx.h | 3 + drivers/common/idpf/version.map | 1 + drivers/net/idpf/idpf_ethdev.c | 3 +- drivers/net/idpf/idpf_rxtx.c | 28 +++++ drivers/net/idpf/idpf_rxtx.h | 2 + 6 files changed, 171 insertions(+), 1 deletion(-) diff --git a/drivers/common/idpf/idpf_common_rxtx.c b/drivers/common/idpf/idpf_common_rxtx.c index fdac2c3114..9303b51cce 100644 --- a/drivers/common/idpf/idpf_common_rxtx.c +++ b/drivers/common/idpf/idpf_common_rxtx.c @@ -1146,6 +1146,141 @@ idpf_dp_singleq_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, return nb_rx; } +uint16_t +idpf_dp_singleq_recv_scatter_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, + uint16_t nb_pkts) +{ + struct idpf_rx_queue *rxq = rx_queue; + volatile union virtchnl2_rx_desc *rx_ring = rxq->rx_ring; + volatile union virtchnl2_rx_desc *rxdp; + union virtchnl2_rx_desc rxd; + struct idpf_adapter *ad; + struct rte_mbuf *first_seg = rxq->pkt_first_seg; + struct rte_mbuf *last_seg = rxq->pkt_last_seg; + struct rte_mbuf *rxm; + struct rte_mbuf *nmb; + struct rte_eth_dev *dev; + const uint32_t *ptype_tbl = rxq->adapter->ptype_tbl; + uint16_t rx_id = rxq->rx_tail; + uint16_t rx_packet_len; + uint16_t nb_hold = 0; + uint16_t rx_status0; + uint16_t nb_rx = 0; + uint64_t pkt_flags; + uint64_t dma_addr; + uint64_t ts_ns; + + ad = rxq->adapter; + + if (unlikely(!rxq) || unlikely(!rxq->q_started)) + return nb_rx; + + while (nb_rx < nb_pkts) { + rxdp = &rx_ring[rx_id]; + rx_status0 = rte_le_to_cpu_16(rxdp->flex_nic_wb.status_error0); + + /* Check the DD bit first */ + if (!(rx_status0 & (1 << VIRTCHNL2_RX_FLEX_DESC_STATUS0_DD_S))) + break; + + nmb = rte_mbuf_raw_alloc(rxq->mp); + if (unlikely(!nmb)) { + __atomic_fetch_add(&rxq->rx_stats.mbuf_alloc_failed, 1, __ATOMIC_RELAXED); + RX_LOG(DEBUG, "RX mbuf alloc failed port_id=%u " + "queue_id=%u", rxq->port_id, rxq->queue_id); + break; + } + + rxd = *rxdp; + + nb_hold++; + rxm = rxq->sw_ring[rx_id]; + rxq->sw_ring[rx_id] = nmb; + rx_id++; + if (unlikely(rx_id == rxq->nb_rx_desc)) + rx_id = 0; + + /* Prefetch next mbuf */ + rte_prefetch0(rxq->sw_ring[rx_id]); + + /* When next RX descriptor is on a cache line boundary, + * prefetch the next 4 RX descriptors and next 8 pointers + * to mbufs. + */ + if ((rx_id & 0x3) == 0) { + rte_prefetch0(&rx_ring[rx_id]); + rte_prefetch0(rxq->sw_ring[rx_id]); + } + dma_addr = + rte_cpu_to_le_64(rte_mbuf_data_iova_default(nmb)); + rxdp->read.hdr_addr = 0; + rxdp->read.pkt_addr = dma_addr; + rx_packet_len = (rte_cpu_to_le_16(rxd.flex_nic_wb.pkt_len) & + VIRTCHNL2_RX_FLEX_DESC_PKT_LEN_M); + rxm->data_len = rx_packet_len; + rxm->data_off = RTE_PKTMBUF_HEADROOM; + + /** + * If this is the first buffer of the received packet, set the + * pointer to the first mbuf of the packet and initialize its + * context. Otherwise, update the total length and the number + * of segments of the current scattered packet, and update the + * pointer to the last mbuf of the current packet. + */ + if (!first_seg) { + first_seg = rxm; + first_seg->nb_segs = 1; + first_seg->pkt_len = rx_packet_len; + } else { + first_seg->pkt_len = + (uint16_t)(first_seg->pkt_len + + rx_packet_len); + first_seg->nb_segs++; + last_seg->next = rxm; + } + + if (!(rx_status0 & (1 << VIRTCHNL2_RX_FLEX_DESC_STATUS0_EOF_S))) { + last_seg = rxm; + continue; + } + + rxm->next = NULL; + + first_seg->port = rxq->port_id; + first_seg->ol_flags = 0; + pkt_flags = idpf_rxd_to_pkt_flags(rx_status0); + first_seg->packet_type = + ptype_tbl[(uint8_t)(rte_cpu_to_le_16(rxd.flex_nic_wb.ptype_flex_flags0) & + VIRTCHNL2_RX_FLEX_DESC_PTYPE_M)]; + + if (idpf_timestamp_dynflag > 0 && + (rxq->offloads & IDPF_RX_OFFLOAD_TIMESTAMP) != 0) { + /* timestamp */ + ts_ns = idpf_tstamp_convert_32b_64b(ad, + rxq->hw_register_set, + rte_le_to_cpu_32(rxd.flex_nic_wb.flex_ts.ts_high)); + rxq->hw_register_set = 0; + *RTE_MBUF_DYNFIELD(rxm, + idpf_timestamp_dynfield_offset, + rte_mbuf_timestamp_t *) = ts_ns; + first_seg->ol_flags |= idpf_timestamp_dynflag; + } + + first_seg->ol_flags |= pkt_flags; + rte_prefetch0(RTE_PTR_ADD(first_seg->buf_addr, + first_seg->data_off)); + rx_pkts[nb_rx++] = first_seg; + first_seg = NULL; + } + rxq->rx_tail = rx_id; + rxq->pkt_first_seg = first_seg; + rxq->pkt_last_seg = last_seg; + + idpf_update_rx_tail(rxq, nb_hold, rx_id); + + return nb_rx; +} + static inline int idpf_xmit_cleanup(struct idpf_tx_queue *txq) { diff --git a/drivers/common/idpf/idpf_common_rxtx.h b/drivers/common/idpf/idpf_common_rxtx.h index 263dab061c..7e6df080e6 100644 --- a/drivers/common/idpf/idpf_common_rxtx.h +++ b/drivers/common/idpf/idpf_common_rxtx.h @@ -293,5 +293,8 @@ uint16_t idpf_dp_singleq_xmit_pkts_avx512(void *tx_queue, __rte_internal uint16_t idpf_dp_splitq_xmit_pkts_avx512(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts); +__rte_internal +uint16_t idpf_dp_singleq_recv_scatter_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, + uint16_t nb_pkts); #endif /* _IDPF_COMMON_RXTX_H_ */ diff --git a/drivers/common/idpf/version.map b/drivers/common/idpf/version.map index f6c92e7e57..e31f6ff4d9 100644 --- a/drivers/common/idpf/version.map +++ b/drivers/common/idpf/version.map @@ -7,6 +7,7 @@ INTERNAL { idpf_dp_prep_pkts; idpf_dp_singleq_recv_pkts; idpf_dp_singleq_recv_pkts_avx512; + idpf_dp_singleq_recv_scatter_pkts; idpf_dp_singleq_xmit_pkts; idpf_dp_singleq_xmit_pkts_avx512; idpf_dp_splitq_recv_pkts; diff --git a/drivers/net/idpf/idpf_ethdev.c b/drivers/net/idpf/idpf_ethdev.c index 7262109d0a..11f0ca0085 100644 --- a/drivers/net/idpf/idpf_ethdev.c +++ b/drivers/net/idpf/idpf_ethdev.c @@ -119,7 +119,8 @@ idpf_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info) RTE_ETH_RX_OFFLOAD_UDP_CKSUM | RTE_ETH_RX_OFFLOAD_TCP_CKSUM | RTE_ETH_RX_OFFLOAD_OUTER_IPV4_CKSUM | - RTE_ETH_RX_OFFLOAD_TIMESTAMP; + RTE_ETH_RX_OFFLOAD_TIMESTAMP | + RTE_ETH_RX_OFFLOAD_SCATTER; dev_info->tx_offload_capa = RTE_ETH_TX_OFFLOAD_IPV4_CKSUM | diff --git a/drivers/net/idpf/idpf_rxtx.c b/drivers/net/idpf/idpf_rxtx.c index 38d9829912..d16acd87fb 100644 --- a/drivers/net/idpf/idpf_rxtx.c +++ b/drivers/net/idpf/idpf_rxtx.c @@ -503,6 +503,8 @@ int idpf_rx_queue_init(struct rte_eth_dev *dev, uint16_t rx_queue_id) { struct idpf_rx_queue *rxq; + uint16_t max_pkt_len; + uint32_t frame_size; int err; if (rx_queue_id >= dev->data->nb_rx_queues) @@ -516,6 +518,17 @@ idpf_rx_queue_init(struct rte_eth_dev *dev, uint16_t rx_queue_id) return -EINVAL; } + frame_size = dev->data->mtu + IDPF_ETH_OVERHEAD; + + max_pkt_len = + RTE_MIN((uint32_t)IDPF_SUPPORT_CHAIN_NUM * rxq->rx_buf_len, + frame_size); + + rxq->max_pkt_len = max_pkt_len; + if ((dev->data->dev_conf.rxmode.offloads & RTE_ETH_RX_OFFLOAD_SCATTER) || + frame_size > rxq->rx_buf_len) + dev->data->scattered_rx = 1; + err = idpf_qc_ts_mbuf_register(rxq); if (err != 0) { PMD_DRV_LOG(ERR, "fail to residter timestamp mbuf %u", @@ -807,6 +820,14 @@ idpf_set_rx_function(struct rte_eth_dev *dev) } #endif /* CC_AVX512_SUPPORT */ } + + if (dev->data->scattered_rx) { + PMD_DRV_LOG(NOTICE, + "Using Single Scalar Scatterd Rx (port %d).", + dev->data->port_id); + dev->rx_pkt_burst = idpf_dp_singleq_recv_scatter_pkts; + return; + } PMD_DRV_LOG(NOTICE, "Using Single Scalar Rx (port %d).", dev->data->port_id); @@ -819,6 +840,13 @@ idpf_set_rx_function(struct rte_eth_dev *dev) dev->data->port_id); dev->rx_pkt_burst = idpf_dp_splitq_recv_pkts; } else { + if (dev->data->scattered_rx) { + PMD_DRV_LOG(NOTICE, + "Using Single Scalar Scatterd Rx (port %d).", + dev->data->port_id); + dev->rx_pkt_burst = idpf_dp_singleq_recv_scatter_pkts; + return; + } PMD_DRV_LOG(NOTICE, "Using Single Scalar Rx (port %d).", dev->data->port_id); diff --git a/drivers/net/idpf/idpf_rxtx.h b/drivers/net/idpf/idpf_rxtx.h index 3a5084dfd6..41a7495083 100644 --- a/drivers/net/idpf/idpf_rxtx.h +++ b/drivers/net/idpf/idpf_rxtx.h @@ -23,6 +23,8 @@ #define IDPF_DEFAULT_TX_RS_THRESH 32 #define IDPF_DEFAULT_TX_FREE_THRESH 32 +#define IDPF_SUPPORT_CHAIN_NUM 5 + int idpf_rx_queue_setup(struct rte_eth_dev *dev, uint16_t queue_idx, uint16_t nb_desc, unsigned int socket_id, const struct rte_eth_rxconf *rx_conf,