Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/77109/?format=api
{ "id": 77109, "url": "http://patches.dpdk.org/api/patches/77109/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/patch/1599717545-106571-4-git-send-email-wenzhuo.lu@intel.com/", "project": { "id": 1, "url": "http://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<1599717545-106571-4-git-send-email-wenzhuo.lu@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/1599717545-106571-4-git-send-email-wenzhuo.lu@intel.com", "date": "2020-09-10T05:59:05", "name": "[3/3] net/iavf: enable AVX512 for TX", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": true, "hash": "f3c0313bdf10ef4df2b070ec93eb5cc6671d9561", "submitter": { "id": 258, "url": "http://patches.dpdk.org/api/people/258/?format=api", "name": "Wenzhuo Lu", "email": "wenzhuo.lu@intel.com" }, "delegate": { "id": 1540, "url": "http://patches.dpdk.org/api/users/1540/?format=api", "username": "qzhan15", "first_name": "Qi", "last_name": "Zhang", "email": "qi.z.zhang@intel.com" }, "mbox": "http://patches.dpdk.org/project/dpdk/patch/1599717545-106571-4-git-send-email-wenzhuo.lu@intel.com/mbox/", "series": [ { "id": 12085, "url": "http://patches.dpdk.org/api/series/12085/?format=api", "web_url": "http://patches.dpdk.org/project/dpdk/list/?series=12085", "date": "2020-09-10T05:59:02", "name": "enable AVX512 for iavf", "version": 1, "mbox": "http://patches.dpdk.org/series/12085/mbox/" } ], "comments": "http://patches.dpdk.org/api/patches/77109/comments/", "check": "fail", "checks": "http://patches.dpdk.org/api/patches/77109/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from dpdk.org (dpdk.org [92.243.14.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 8A7CFA04B6;\n\tThu, 10 Sep 2020 08:00:13 +0200 (CEST)", "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 65F581C116;\n\tThu, 10 Sep 2020 07:59:50 +0200 (CEST)", "from mga03.intel.com (mga03.intel.com [134.134.136.65])\n by dpdk.org (Postfix) with ESMTP id 87F7C1C0D8\n for <dev@dpdk.org>; Thu, 10 Sep 2020 07:59:47 +0200 (CEST)", "from orsmga004.jf.intel.com ([10.7.209.38])\n by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;\n 09 Sep 2020 22:59:47 -0700", "from dpdk-wenzhuo-haswell.sh.intel.com ([10.67.111.137])\n by orsmga004.jf.intel.com with ESMTP; 09 Sep 2020 22:59:45 -0700" ], "IronPort-SDR": [ "\n agTTSkL9rfLyM3ubI85oB5hHh03jW0qNoJ3k/tgGRseH1oylG6hvSqIiFDVIBpFW4d/Y+fz/zM\n NXyMz4W1QFVw==", "\n /Ip21tbz+e3qWfVUyF1J2EdI/6x6G4md0M3vZEWW+WZpF05Y2ezM3w14mi7bq0oE2xJGQbSQ91\n HqT3mBbF+pGw==" ], "X-IronPort-AV": [ "E=McAfee;i=\"6000,8403,9739\"; a=\"158508478\"", "E=Sophos;i=\"5.76,412,1592895600\"; d=\"scan'208\";a=\"158508478\"", "E=Sophos;i=\"5.76,411,1592895600\"; d=\"scan'208\";a=\"449471229\"" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "From": "Wenzhuo Lu <wenzhuo.lu@intel.com>", "To": "dev@dpdk.org", "Cc": "Wenzhuo Lu <wenzhuo.lu@intel.com>,\n Bruce Richardson <bruce.richardson@intel.com>,\n Leyi Rong <leyi.rong@intel.com>", "Date": "Thu, 10 Sep 2020 13:59:05 +0800", "Message-Id": "<1599717545-106571-4-git-send-email-wenzhuo.lu@intel.com>", "X-Mailer": "git-send-email 1.9.3", "In-Reply-To": "<1599717545-106571-1-git-send-email-wenzhuo.lu@intel.com>", "References": "<1599717545-106571-1-git-send-email-wenzhuo.lu@intel.com>", "Subject": "[dpdk-dev] [PATCH 3/3] net/iavf: enable AVX512 for TX", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "To enhance the per-core performance, this patch adds some AVX512\ninstructions to the data path to handle the TX descriptors.\n\nSigned-off-by: Wenzhuo Lu <wenzhuo.lu@intel.com>\nSigned-off-by: Bruce Richardson <bruce.richardson@intel.com>\nSigned-off-by: Leyi Rong <leyi.rong@intel.com>\n---\n doc/guides/rel_notes/release_20_11.rst | 3 +\n drivers/net/iavf/iavf_ethdev.c | 3 +-\n drivers/net/iavf/iavf_rxtx.c | 32 +++-\n drivers/net/iavf/iavf_rxtx.h | 7 +\n drivers/net/iavf/iavf_rxtx_vec_avx512.c | 301 ++++++++++++++++++++++++++++++++\n 5 files changed, 338 insertions(+), 8 deletions(-)", "diff": "diff --git a/doc/guides/rel_notes/release_20_11.rst b/doc/guides/rel_notes/release_20_11.rst\nindex df227a1..d40b8d6 100644\n--- a/doc/guides/rel_notes/release_20_11.rst\n+++ b/doc/guides/rel_notes/release_20_11.rst\n@@ -55,6 +55,9 @@ New Features\n Also, make sure to start the actual text at the margin.\n =======================================================\n \n+ * **Added support of vector instructions on IAVF.**\n+\n+ Added support of AVX512 instructions in IAVF RX and TX path.\n \n Removed Items\n -------------\ndiff --git a/drivers/net/iavf/iavf_ethdev.c b/drivers/net/iavf/iavf_ethdev.c\nindex c3aa4cd..5bc2851 100644\n--- a/drivers/net/iavf/iavf_ethdev.c\n+++ b/drivers/net/iavf/iavf_ethdev.c\n@@ -528,7 +528,8 @@ static int iavf_config_rx_queues_irqs(struct rte_eth_dev *dev,\n \t\tDEV_TX_OFFLOAD_GRE_TNL_TSO |\n \t\tDEV_TX_OFFLOAD_IPIP_TNL_TSO |\n \t\tDEV_TX_OFFLOAD_GENEVE_TNL_TSO |\n-\t\tDEV_TX_OFFLOAD_MULTI_SEGS;\n+\t\tDEV_TX_OFFLOAD_MULTI_SEGS |\n+\t\tDEV_TX_OFFLOAD_MBUF_FAST_FREE;\n \n \tdev_info->default_rxconf = (struct rte_eth_rxconf) {\n \t\t.rx_free_thresh = IAVF_DEFAULT_RX_FREE_THRESH,\ndiff --git a/drivers/net/iavf/iavf_rxtx.c b/drivers/net/iavf/iavf_rxtx.c\nindex 0818107..04dcd48 100644\n--- a/drivers/net/iavf/iavf_rxtx.c\n+++ b/drivers/net/iavf/iavf_rxtx.c\n@@ -2206,18 +2206,18 @@\n \tstruct iavf_tx_queue *txq;\n \tint i;\n \tbool use_avx2 = false;\n+#ifdef CC_AVX512_SUPPORT\n+\tbool use_avx512 = false;\n+#endif\n \n \tif (!iavf_tx_vec_dev_check(dev)) {\n-\t\tfor (i = 0; i < dev->data->nb_tx_queues; i++) {\n-\t\t\ttxq = dev->data->tx_queues[i];\n-\t\t\tif (!txq)\n-\t\t\t\tcontinue;\n-\t\t\tiavf_txq_vec_setup(txq);\n-\t\t}\n-\n \t\tif (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2) == 1 ||\n \t\t rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F) == 1)\n \t\t\tuse_avx2 = true;\n+#ifdef CC_AVX512_SUPPORT\n+\t\tif (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F) == 1)\n+\t\t\tuse_avx512 = true;\n+#endif\n \n \t\tPMD_DRV_LOG(DEBUG, \"Using %sVector Tx (port %d).\",\n \t\t\t use_avx2 ? \"avx2 \" : \"\",\n@@ -2225,8 +2225,26 @@\n \t\tdev->tx_pkt_burst = use_avx2 ?\n \t\t\t\t iavf_xmit_pkts_vec_avx2 :\n \t\t\t\t iavf_xmit_pkts_vec;\n+#ifdef CC_AVX512_SUPPORT\n+\t\tif (use_avx512)\n+\t\t\tdev->tx_pkt_burst = iavf_xmit_pkts_vec_avx512;\n+#endif\n \t\tdev->tx_pkt_prepare = NULL;\n \n+\t\tfor (i = 0; i < dev->data->nb_tx_queues; i++) {\n+\t\t\ttxq = dev->data->tx_queues[i];\n+\t\t\tif (!txq)\n+\t\t\t\tcontinue;\n+#ifdef CC_AVX512_SUPPORT\n+\t\t\tif (use_avx512)\n+\t\t\t\tiavf_txq_vec_setup_avx512(txq);\n+\t\t\telse\n+\t\t\t\tiavf_txq_vec_setup(txq);\n+#else\n+\t\t\tiavf_txq_vec_setup(txq);\n+#endif\n+\t\t}\n+\n \t\treturn;\n \t}\n #endif\ndiff --git a/drivers/net/iavf/iavf_rxtx.h b/drivers/net/iavf/iavf_rxtx.h\nindex 9653e0c..08eebb0 100644\n--- a/drivers/net/iavf/iavf_rxtx.h\n+++ b/drivers/net/iavf/iavf_rxtx.h\n@@ -122,6 +122,10 @@ struct iavf_tx_entry {\n \tuint16_t last_id;\n };\n \n+struct iavf_tx_vec_entry {\n+\tstruct rte_mbuf *mbuf;\n+};\n+\n /* Structure associated with each TX queue. */\n struct iavf_tx_queue {\n \tconst struct rte_memzone *mz; /* memzone for Tx ring */\n@@ -448,6 +452,9 @@ uint16_t iavf_recv_scattered_pkts_vec_avx512(void *rx_queue,\n uint16_t iavf_recv_scattered_pkts_vec_avx512_flex_rxd(void *rx_queue,\n \t\t\t\t\t\t struct rte_mbuf **rx_pkts,\n \t\t\t\t\t\t uint16_t nb_pkts);\n+uint16_t iavf_xmit_pkts_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,\n+\t\t\t\t uint16_t nb_pkts);\n+int iavf_txq_vec_setup_avx512(struct iavf_tx_queue *txq);\n \n const uint32_t *iavf_get_default_ptype_table(void);\n \ndiff --git a/drivers/net/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/iavf/iavf_rxtx_vec_avx512.c\nindex 8c33661..6c75d04 100644\n--- a/drivers/net/iavf/iavf_rxtx_vec_avx512.c\n+++ b/drivers/net/iavf/iavf_rxtx_vec_avx512.c\n@@ -1417,3 +1417,304 @@\n \treturn retval + iavf_recv_scattered_burst_vec_avx512_flex_rxd(rx_queue,\n \t\t\t\trx_pkts + retval, nb_pkts);\n }\n+\n+static __rte_always_inline int\n+iavf_tx_free_bufs_avx512(struct iavf_tx_queue *txq)\n+{\n+\tstruct iavf_tx_vec_entry *txep;\n+\tuint32_t n;\n+\tuint32_t i;\n+\tint nb_free = 0;\n+\tstruct rte_mbuf *m, *free[IAVF_VPMD_TX_MAX_FREE_BUF];\n+\n+\t/* check DD bits on threshold descriptor */\n+\tif ((txq->tx_ring[txq->next_dd].cmd_type_offset_bsz &\n+\t\t\trte_cpu_to_le_64(IAVF_TXD_QW1_DTYPE_MASK)) !=\n+\t\t\trte_cpu_to_le_64(IAVF_TX_DESC_DTYPE_DESC_DONE))\n+\t\treturn 0;\n+\n+\tn = txq->rs_thresh;\n+\n+\t /* first buffer to free from S/W ring is at index\n+\t * tx_next_dd - (tx_rs_thresh-1)\n+\t */\n+\ttxep = (void *)txq->sw_ring;\n+\ttxep += txq->next_dd - (n - 1);\n+\n+\tif (txq->offloads & DEV_TX_OFFLOAD_MBUF_FAST_FREE && (n & 31) == 0) {\n+\t\tstruct rte_mempool *mp = txep[0].mbuf->pool;\n+\t\tstruct rte_mempool_cache *cache = rte_mempool_default_cache(mp,\n+\t\t\t\trte_lcore_id());\n+\t\tvoid **cache_objs = &cache->objs[cache->len];\n+\n+\t\tif (n > RTE_MEMPOOL_CACHE_MAX_SIZE) {\n+\t\t\trte_mempool_ops_enqueue_bulk(mp, (void *)txep, n);\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\t/* The cache follows the following algorithm\n+\t\t * 1. Add the objects to the cache\n+\t\t * 2. Anything greater than the cache min value (if it crosses the\n+\t\t * cache flush threshold) is flushed to the ring.\n+\t\t */\n+\t\t/* Add elements back into the cache */\n+\t\tuint32_t copied = 0;\n+\t\t/* n is multiple of 32 */\n+\t\twhile (copied < n) {\n+\t\t\tconst __m512i a = _mm512_loadu_si512(&txep[copied]);\n+\t\t\tconst __m512i b = _mm512_loadu_si512(&txep[copied + 8]);\n+\t\t\tconst __m512i c = _mm512_loadu_si512(&txep[copied + 16]);\n+\t\t\tconst __m512i d = _mm512_loadu_si512(&txep[copied + 24]);\n+\n+\t\t\t_mm512_storeu_si512(&cache_objs[copied], a);\n+\t\t\t_mm512_storeu_si512(&cache_objs[copied + 8], b);\n+\t\t\t_mm512_storeu_si512(&cache_objs[copied + 16], c);\n+\t\t\t_mm512_storeu_si512(&cache_objs[copied + 24], d);\n+\t\t\tcopied += 32;\n+\t\t}\n+\t\tcache->len += n;\n+\n+\t\tif (cache->len >= cache->flushthresh) {\n+\t\t\trte_mempool_ops_enqueue_bulk(mp,\n+\t\t\t\t\t\t &cache->objs[cache->size],\n+\t\t\t\t\t\t cache->len - cache->size);\n+\t\t\tcache->len = cache->size;\n+\t\t}\n+\t\tgoto done;\n+\t}\n+\n+\tm = rte_pktmbuf_prefree_seg(txep[0].mbuf);\n+\tif (likely(m)) {\n+\t\tfree[0] = m;\n+\t\tnb_free = 1;\n+\t\tfor (i = 1; i < n; i++) {\n+\t\t\tm = rte_pktmbuf_prefree_seg(txep[i].mbuf);\n+\t\t\tif (likely(m)) {\n+\t\t\t\tif (likely(m->pool == free[0]->pool)) {\n+\t\t\t\t\tfree[nb_free++] = m;\n+\t\t\t\t} else {\n+\t\t\t\t\trte_mempool_put_bulk(free[0]->pool,\n+\t\t\t\t\t\t\t (void *)free,\n+\t\t\t\t\t\t\t nb_free);\n+\t\t\t\t\tfree[0] = m;\n+\t\t\t\t\tnb_free = 1;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\t\trte_mempool_put_bulk(free[0]->pool, (void **)free, nb_free);\n+\t} else {\n+\t\tfor (i = 1; i < n; i++) {\n+\t\t\tm = rte_pktmbuf_prefree_seg(txep[i].mbuf);\n+\t\t\tif (m)\n+\t\t\t\trte_mempool_put(m->pool, m);\n+\t\t}\n+\t}\n+\n+done:\n+\t/* buffers were freed, update counters */\n+\ttxq->nb_free = (uint16_t)(txq->nb_free + txq->rs_thresh);\n+\ttxq->next_dd = (uint16_t)(txq->next_dd + txq->rs_thresh);\n+\tif (txq->next_dd >= txq->nb_tx_desc)\n+\t\ttxq->next_dd = (uint16_t)(txq->rs_thresh - 1);\n+\n+\treturn txq->rs_thresh;\n+}\n+\n+static __rte_always_inline void\n+tx_backlog_entry_avx512(struct iavf_tx_vec_entry *txep,\n+\t\t\tstruct rte_mbuf **tx_pkts, uint16_t nb_pkts)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < (int)nb_pkts; ++i)\n+\t\ttxep[i].mbuf = tx_pkts[i];\n+}\n+\n+static inline void\n+iavf_vtx1(volatile struct iavf_tx_desc *txdp,\n+\t struct rte_mbuf *pkt, uint64_t flags)\n+{\n+\tuint64_t high_qw =\n+\t\t(IAVF_TX_DESC_DTYPE_DATA |\n+\t\t ((uint64_t)flags << IAVF_TXD_QW1_CMD_SHIFT) |\n+\t\t ((uint64_t)pkt->data_len << IAVF_TXD_QW1_TX_BUF_SZ_SHIFT));\n+\n+\t__m128i descriptor = _mm_set_epi64x(high_qw,\n+\t\t\t\tpkt->buf_physaddr + pkt->data_off);\n+\t_mm_storeu_si128((__m128i *)txdp, descriptor);\n+}\n+\n+static inline void\n+iavf_vtx(volatile struct iavf_tx_desc *txdp,\n+\t struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags)\n+{\n+\tconst uint64_t hi_qw_tmpl = (IAVF_TX_DESC_DTYPE_DATA |\n+\t\t\t((uint64_t)flags << IAVF_TXD_QW1_CMD_SHIFT));\n+\tconst __mmask8 len_msk = _cvtu32_mask8(0xAA);\n+\tconst __mmask8 off_msk = _cvtu32_mask8(0x55);\n+\n+\t/* if unaligned on 32-bit boundary, do one to align */\n+\tif (((uintptr_t)txdp & 0x1F) != 0 && nb_pkts != 0) {\n+\t\tiavf_vtx1(txdp, *pkt, flags);\n+\t\tnb_pkts--, txdp++, pkt++;\n+\t}\n+\n+\t/* do two at a time while possible, in bursts */\n+\tfor (; nb_pkts > 3; txdp += 4, pkt += 4, nb_pkts -= 4) {\n+\t\t__m512i desc4 =\n+\t\t\t_mm512_set_epi64\n+\t\t\t\t((uint64_t)pkt[3]->data_len,\n+\t\t\t\t pkt[3]->buf_physaddr,\n+\t\t\t\t (uint64_t)pkt[2]->data_len,\n+\t\t\t\t pkt[2]->buf_physaddr,\n+\t\t\t\t (uint64_t)pkt[1]->data_len,\n+\t\t\t\t pkt[1]->buf_physaddr,\n+\t\t\t\t (uint64_t)pkt[0]->data_len,\n+\t\t\t\t pkt[0]->buf_physaddr);\n+\t\t__m512i hi_qw_tmpl_4 = _mm512_set1_epi64(hi_qw_tmpl);\n+\t\t__m512i data_off_4 =\n+\t\t\t_mm512_set_epi64\n+\t\t\t\t(0,\n+\t\t\t\t pkt[3]->data_off,\n+\t\t\t\t 0,\n+\t\t\t\t pkt[2]->data_off,\n+\t\t\t\t 0,\n+\t\t\t\t pkt[1]->data_off,\n+\t\t\t\t 0,\n+\t\t\t\t pkt[0]->data_off);\n+\n+\t\tdesc4 = _mm512_mask_slli_epi64(desc4, len_msk, desc4, IAVF_TXD_QW1_TX_BUF_SZ_SHIFT);\n+\t\tdesc4 = _mm512_mask_or_epi64(desc4, len_msk, desc4, hi_qw_tmpl_4);\n+\t\tdesc4 = _mm512_mask_add_epi64(desc4, off_msk, desc4, data_off_4);\n+\t\t_mm512_storeu_si512((void *)txdp, desc4);\n+\t}\n+\n+\t/* do any last ones */\n+\twhile (nb_pkts) {\n+\t\tiavf_vtx1(txdp, *pkt, flags);\n+\t\ttxdp++, pkt++, nb_pkts--;\n+\t}\n+}\n+\n+static inline uint16_t\n+iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,\n+\t\t\t\t uint16_t nb_pkts)\n+{\n+\tstruct iavf_tx_queue *txq = (struct iavf_tx_queue *)tx_queue;\n+\tvolatile struct iavf_tx_desc *txdp;\n+\tstruct iavf_tx_vec_entry *txep;\n+\tuint16_t n, nb_commit, tx_id;\n+\t/* bit2 is reserved and must be set to 1 according to Spec */\n+\tuint64_t flags = IAVF_TX_DESC_CMD_EOP | IAVF_TX_DESC_CMD_ICRC;\n+\tuint64_t rs = IAVF_TX_DESC_CMD_RS | flags;\n+\n+\t/* cross rx_thresh boundary is not allowed */\n+\tnb_pkts = RTE_MIN(nb_pkts, txq->rs_thresh);\n+\n+\tif (txq->nb_free < txq->free_thresh)\n+\t\tiavf_tx_free_bufs_avx512(txq);\n+\n+\tnb_commit = nb_pkts = (uint16_t)RTE_MIN(txq->nb_free, nb_pkts);\n+\tif (unlikely(nb_pkts == 0))\n+\t\treturn 0;\n+\n+\ttx_id = txq->tx_tail;\n+\ttxdp = &txq->tx_ring[tx_id];\n+\ttxep = (void *)txq->sw_ring;\n+\ttxep += tx_id;\n+\n+\ttxq->nb_free = (uint16_t)(txq->nb_free - nb_pkts);\n+\n+\tn = (uint16_t)(txq->nb_tx_desc - tx_id);\n+\tif (nb_commit >= n) {\n+\t\ttx_backlog_entry_avx512(txep, tx_pkts, n);\n+\n+\t\tiavf_vtx(txdp, tx_pkts, n - 1, flags);\n+\t\ttx_pkts += (n - 1);\n+\t\ttxdp += (n - 1);\n+\n+\t\tiavf_vtx1(txdp, *tx_pkts++, rs);\n+\n+\t\tnb_commit = (uint16_t)(nb_commit - n);\n+\n+\t\ttx_id = 0;\n+\t\ttxq->next_rs = (uint16_t)(txq->rs_thresh - 1);\n+\n+\t\t/* avoid reach the end of ring */\n+\t\ttxdp = &txq->tx_ring[tx_id];\n+\t\ttxep = (void *)txq->sw_ring;\n+\t\ttxep += tx_id;\n+\t}\n+\n+\ttx_backlog_entry_avx512(txep, tx_pkts, nb_commit);\n+\n+\tiavf_vtx(txdp, tx_pkts, nb_commit, flags);\n+\n+\ttx_id = (uint16_t)(tx_id + nb_commit);\n+\tif (tx_id > txq->next_rs) {\n+\t\ttxq->tx_ring[txq->next_rs].cmd_type_offset_bsz |=\n+\t\t\trte_cpu_to_le_64(((uint64_t)IAVF_TX_DESC_CMD_RS) <<\n+\t\t\t\t\t IAVF_TXD_QW1_CMD_SHIFT);\n+\t\ttxq->next_rs =\n+\t\t\t(uint16_t)(txq->next_rs + txq->rs_thresh);\n+\t}\n+\n+\ttxq->tx_tail = tx_id;\n+\n+\tIAVF_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail);\n+\n+\treturn nb_pkts;\n+}\n+\n+uint16_t\n+iavf_xmit_pkts_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts,\n+\t\t\t uint16_t nb_pkts)\n+{\n+\tuint16_t nb_tx = 0;\n+\tstruct iavf_tx_queue *txq = (struct iavf_tx_queue *)tx_queue;\n+\n+\twhile (nb_pkts) {\n+\t\tuint16_t ret, num;\n+\n+\t\tnum = (uint16_t)RTE_MIN(nb_pkts, txq->rs_thresh);\n+\t\tret = iavf_xmit_fixed_burst_vec_avx512(tx_queue, &tx_pkts[nb_tx],\n+\t\t\t\t\t\t num);\n+\t\tnb_tx += ret;\n+\t\tnb_pkts -= ret;\n+\t\tif (ret < num)\n+\t\t\tbreak;\n+\t}\n+\n+\treturn nb_tx;\n+}\n+\n+static inline void\n+iavf_tx_queue_release_mbufs_avx512(struct iavf_tx_queue *txq)\n+{\n+\tunsigned int i;\n+\tconst uint16_t max_desc = (uint16_t)(txq->nb_tx_desc - 1);\n+\tstruct iavf_tx_vec_entry *swr = (void *)txq->sw_ring;\n+\n+\tif (!txq->sw_ring || txq->nb_free == max_desc)\n+\t\treturn;\n+\n+\ti = txq->next_dd - txq->rs_thresh + 1;\n+\tif (txq->tx_tail < i) {\n+\t\tfor (; i < txq->nb_tx_desc; i++) {\n+\t\t\trte_pktmbuf_free_seg(swr[i].mbuf);\n+\t\t\tswr[i].mbuf = NULL;\n+\t\t}\n+\t\ti = 0;\n+\t}\n+}\n+\n+static const struct iavf_txq_ops avx512_vec_txq_ops = {\n+\t.release_mbufs = iavf_tx_queue_release_mbufs_avx512,\n+};\n+\n+int __rte_cold\n+iavf_txq_vec_setup_avx512(struct iavf_tx_queue *txq)\n+{\n+\ttxq->ops = &avx512_vec_txq_ops;\n+\treturn 0;\n+}\n", "prefixes": [ "3/3" ] }