Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/14361/?format=api
{ "id": 14361, "url": "https://patches.dpdk.org/api/patches/14361/?format=api", "web_url": "https://patches.dpdk.org/project/dpdk/patch/1466774284-20932-22-git-send-email-nelio.laranjeiro@6wind.com/", "project": { "id": 1, "url": "https://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<1466774284-20932-22-git-send-email-nelio.laranjeiro@6wind.com>", "list_archive_url": "https://inbox.dpdk.org/dev/1466774284-20932-22-git-send-email-nelio.laranjeiro@6wind.com", "date": "2016-06-24T13:18:00", "name": "[dpdk-dev,v7,21/25] mlx5: resurrect Tx gather support", "commit_ref": null, "pull_url": null, "state": "accepted", "archived": true, "hash": "9c0a7f1c56e2aebb3fa9cd1e76bddd2a043c8892", "submitter": { "id": 243, "url": "https://patches.dpdk.org/api/people/243/?format=api", "name": "Nélio Laranjeiro", "email": "nelio.laranjeiro@6wind.com" }, "delegate": { "id": 10, "url": "https://patches.dpdk.org/api/users/10/?format=api", "username": "bruce", "first_name": "Bruce", "last_name": "Richardson", "email": "bruce.richardson@intel.com" }, "mbox": "https://patches.dpdk.org/project/dpdk/patch/1466774284-20932-22-git-send-email-nelio.laranjeiro@6wind.com/mbox/", "series": [], "comments": "https://patches.dpdk.org/api/patches/14361/comments/", "check": "pending", "checks": "https://patches.dpdk.org/api/patches/14361/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [IPv6:::1])\n\tby dpdk.org (Postfix) with ESMTP id 56F45C6EE;\n\tFri, 24 Jun 2016 15:19:52 +0200 (CEST)", "from mail-wm0-f50.google.com (mail-wm0-f50.google.com\n\t[74.125.82.50]) by dpdk.org (Postfix) with ESMTP id D3E25C742\n\tfor <dev@dpdk.org>; Fri, 24 Jun 2016 15:19:21 +0200 (CEST)", "by mail-wm0-f50.google.com with SMTP id r201so25240333wme.1\n\tfor <dev@dpdk.org>; Fri, 24 Jun 2016 06:19:21 -0700 (PDT)", "from ping.vm.6wind.com (guy78-3-82-239-227-177.fbx.proxad.net.\n\t[82.239.227.177]) by smtp.gmail.com with ESMTPSA id\n\tm125sm1279533wmm.8.2016.06.24.06.19.19\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128);\n\tFri, 24 Jun 2016 06:19:20 -0700 (PDT)" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=6wind-com.20150623.gappssmtp.com; s=20150623;\n\th=from:to:cc:subject:date:message-id:in-reply-to:references;\n\tbh=o4oMMIioTXy3NucUBPBHCul7zRFDB8EjVn0D0xF7Y4A=;\n\tb=HSKtgSmbBUtNR3h0bUT+cnEdP3iXFurFSCshiP5GZ4BsDPLAPHahChi7bhsTLo9y7N\n\t7hNU8qrwP8sL8RhxfrugmssXOnK7Eu5jjr0eXHIAOcuEZFVbc2m6y83w4IWMSBL5w1fa\n\t/BZqhc3C3sGqUVA3luIxlFtqgljro1S4mmqKGIgmwCFyMCx0qpr1kjTHJg+4V+4np9+x\n\t+rT5odDwapyjEwQbEiILwdWGCwteapVozCYss47BBLU7T/QfEAO01wjMVqtQEnLfKr71\n\tI1Jd8MRhNJVi6IWVYqES/8Fpyk1yxD5vLgw1MnTcDdMTCNvJSeHwd3SYtoxjTDpOkRoi\n\t77gg==", "X-Google-DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20130820;\n\th=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to\n\t:references;\n\tbh=o4oMMIioTXy3NucUBPBHCul7zRFDB8EjVn0D0xF7Y4A=;\n\tb=ONS9kq/GuIbCtn1NU6Y046iAD9iHa1zF88vConu7ziLw7WusXJ/UgbbAFChUvDfq/4\n\tR6TQ8GPe7xIa0+aIeT2Rwaq+I/XekCUkJlr3Cyyo/TTBVbG9en+//Dx8MpytuWXqg98M\n\tC+Kpk7t23ybgnk/CKlwrP38aqe3jEe8NokXIsYnZe80odQ3w7zkllqQ8RxEUfwjuku8q\n\tCwkgIPijLxoyHvrW0p6+h6ywQdcBmdpjIli3WhLO+8mzL4aaxLDCIoyLnCS81Ch9p0Mo\n\tN+GdYoPk2cVZNU4YH4myAj0yG68NGgZE8nRkBLPoBVMiISzI2uA1uB0zvf7KVxaAokOW\n\thP/A==", "X-Gm-Message-State": "ALyK8tKK17+QA6I9X2B06wi1sEEuOgByW8Rp+2BIUl2uoJa9T76Z6oqM6mdQDRz8l8Pa/yPB", "X-Received": "by 10.28.45.142 with SMTP id t136mr5909825wmt.40.1466774361058; \n\tFri, 24 Jun 2016 06:19:21 -0700 (PDT)", "From": "Nelio Laranjeiro <nelio.laranjeiro@6wind.com>", "To": "dev@dpdk.org", "Cc": "Bruce Richardson <bruce.richardson@intel.com>,\n\tFerruh Yigit <ferruh.yigit@intel.com>,\n\tAdrien Mazarguil <adrien.mazarguil@6wind.com>", "Date": "Fri, 24 Jun 2016 15:18:00 +0200", "Message-Id": "<1466774284-20932-22-git-send-email-nelio.laranjeiro@6wind.com>", "X-Mailer": "git-send-email 2.1.4", "In-Reply-To": "<1466774284-20932-1-git-send-email-nelio.laranjeiro@6wind.com>", "References": "<1466758261-25986-1-git-send-email-nelio.laranjeiro@6wind.com>\n\t<1466774284-20932-1-git-send-email-nelio.laranjeiro@6wind.com>", "Subject": "[dpdk-dev] [PATCH v7 21/25] mlx5: resurrect Tx gather support", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "patches and discussions about DPDK <dev.dpdk.org>", "List-Unsubscribe": "<http://dpdk.org/ml/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://dpdk.org/ml/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<http://dpdk.org/ml/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "From: Adrien Mazarguil <adrien.mazarguil@6wind.com>\n\nCompared to its previous incarnation, the software limit on the number of\nmbuf segments is no more (previously MLX5_PMD_SGE_WR_N, set to 4 by\ndefault) hence no need for linearization code and related buffers that\npermanently consumed a non negligible amount of memory to handle oversized\nmbufs.\n\nThe resulting code is both lighter and faster.\n\nSigned-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>\nSigned-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>\n---\n drivers/net/mlx5/mlx5_rxtx.c | 235 +++++++++++++++++++++++++++++++++----------\n drivers/net/mlx5/mlx5_txq.c | 8 +-\n 2 files changed, 188 insertions(+), 55 deletions(-)", "diff": "diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c\nindex fadc182..c72e7ce 100644\n--- a/drivers/net/mlx5/mlx5_rxtx.c\n+++ b/drivers/net/mlx5/mlx5_rxtx.c\n@@ -303,6 +303,7 @@ mlx5_wqe_write(struct txq *txq, volatile union mlx5_wqe *wqe,\n {\n \twqe->wqe.ctrl.data[0] = htonl((txq->wqe_ci << 8) | MLX5_OPCODE_SEND);\n \twqe->wqe.ctrl.data[1] = htonl((txq->qp_num_8s) | 4);\n+\twqe->wqe.ctrl.data[2] = 0;\n \twqe->wqe.ctrl.data[3] = 0;\n \twqe->inl.eseg.rsvd0 = 0;\n \twqe->inl.eseg.rsvd1 = 0;\n@@ -348,6 +349,7 @@ mlx5_wqe_write_vlan(struct txq *txq, volatile union mlx5_wqe *wqe,\n \n \twqe->wqe.ctrl.data[0] = htonl((txq->wqe_ci << 8) | MLX5_OPCODE_SEND);\n \twqe->wqe.ctrl.data[1] = htonl((txq->qp_num_8s) | 4);\n+\twqe->wqe.ctrl.data[2] = 0;\n \twqe->wqe.ctrl.data[3] = 0;\n \twqe->inl.eseg.rsvd0 = 0;\n \twqe->inl.eseg.rsvd1 = 0;\n@@ -425,6 +427,7 @@ mlx5_wqe_write_inline(struct txq *txq, volatile union mlx5_wqe *wqe,\n \tassert(size < 64);\n \twqe->inl.ctrl.data[0] = htonl((txq->wqe_ci << 8) | MLX5_OPCODE_SEND);\n \twqe->inl.ctrl.data[1] = htonl(txq->qp_num_8s | size);\n+\twqe->inl.ctrl.data[2] = 0;\n \twqe->inl.ctrl.data[3] = 0;\n \twqe->inl.eseg.rsvd0 = 0;\n \twqe->inl.eseg.rsvd1 = 0;\n@@ -498,6 +501,7 @@ mlx5_wqe_write_inline_vlan(struct txq *txq, volatile union mlx5_wqe *wqe,\n \tassert(size < 64);\n \twqe->inl.ctrl.data[0] = htonl((txq->wqe_ci << 8) | MLX5_OPCODE_SEND);\n \twqe->inl.ctrl.data[1] = htonl(txq->qp_num_8s | size);\n+\twqe->inl.ctrl.data[2] = 0;\n \twqe->inl.ctrl.data[3] = 0;\n \twqe->inl.eseg.rsvd0 = 0;\n \twqe->inl.eseg.rsvd1 = 0;\n@@ -586,6 +590,7 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \tuint16_t elts_head = txq->elts_head;\n \tconst unsigned int elts_n = txq->elts_n;\n \tunsigned int i = 0;\n+\tunsigned int j = 0;\n \tunsigned int max;\n \tunsigned int comp;\n \tvolatile union mlx5_wqe *wqe;\n@@ -602,23 +607,27 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \tif (max > elts_n)\n \t\tmax -= elts_n;\n \tdo {\n-\t\tstruct rte_mbuf *buf;\n+\t\tstruct rte_mbuf *buf = *(pkts++);\n \t\tunsigned int elts_head_next;\n \t\tuintptr_t addr;\n \t\tuint32_t length;\n \t\tuint32_t lkey;\n+\t\tunsigned int segs_n = buf->nb_segs;\n+\t\tvolatile struct mlx5_wqe_data_seg *dseg;\n+\t\tunsigned int ds = sizeof(*wqe) / 16;\n \n \t\t/*\n \t\t * Make sure there is enough room to store this packet and\n \t\t * that one ring entry remains unused.\n \t\t */\n-\t\tif (max < 1 + 1)\n+\t\tassert(segs_n);\n+\t\tif (max < segs_n + 1)\n \t\t\tbreak;\n-\t\t--max;\n+\t\tmax -= segs_n;\n \t\t--pkts_n;\n-\t\tbuf = *(pkts++);\n \t\telts_head_next = (elts_head + 1) & (elts_n - 1);\n \t\twqe = &(*txq->wqes)[txq->wqe_ci & (txq->wqe_n - 1)];\n+\t\tdseg = &wqe->wqe.dseg;\n \t\trte_prefetch0(wqe);\n \t\tif (pkts_n)\n \t\t\trte_prefetch0(*pkts);\n@@ -638,7 +647,6 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \t\t\t\t\t buf->vlan_tci);\n \t\telse\n \t\t\tmlx5_wqe_write(txq, wqe, addr, length, lkey);\n-\t\twqe->wqe.ctrl.data[2] = 0;\n \t\t/* Should we enable HW CKSUM offload */\n \t\tif (buf->ol_flags &\n \t\t (PKT_TX_IP_CKSUM | PKT_TX_TCP_CKSUM | PKT_TX_UDP_CKSUM)) {\n@@ -648,6 +656,37 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \t\t} else {\n \t\t\twqe->wqe.eseg.cs_flags = 0;\n \t\t}\n+\t\twhile (--segs_n) {\n+\t\t\t/*\n+\t\t\t * Spill on next WQE when the current one does not have\n+\t\t\t * enough room left. Size of WQE must a be a multiple\n+\t\t\t * of data segment size.\n+\t\t\t */\n+\t\t\tassert(!(sizeof(*wqe) % sizeof(*dseg)));\n+\t\t\tif (!(ds % (sizeof(*wqe) / 16)))\n+\t\t\t\tdseg = (volatile void *)\n+\t\t\t\t\t&(*txq->wqes)[txq->wqe_ci++ &\n+\t\t\t\t\t\t (txq->wqe_n - 1)];\n+\t\t\telse\n+\t\t\t\t++dseg;\n+\t\t\t++ds;\n+\t\t\tbuf = buf->next;\n+\t\t\tassert(buf);\n+\t\t\t/* Store segment information. */\n+\t\t\tdseg->byte_count = htonl(DATA_LEN(buf));\n+\t\t\tdseg->lkey = txq_mp2mr(txq, txq_mb2mp(buf));\n+\t\t\tdseg->addr = htonll(rte_pktmbuf_mtod(buf, uintptr_t));\n+\t\t\t(*txq->elts)[elts_head_next] = buf;\n+\t\t\telts_head_next = (elts_head_next + 1) & (elts_n - 1);\n+#ifdef MLX5_PMD_SOFT_COUNTERS\n+\t\t\tlength += DATA_LEN(buf);\n+#endif\n+\t\t\t++j;\n+\t\t}\n+\t\t/* Update DS field in WQE. */\n+\t\twqe->wqe.ctrl.data[1] &= htonl(0xffffffc0);\n+\t\twqe->wqe.ctrl.data[1] |= htonl(ds & 0x3f);\n+\t\telts_head = elts_head_next;\n #ifdef MLX5_PMD_SOFT_COUNTERS\n \t\t/* Increment sent bytes counter. */\n \t\ttxq->stats.obytes += length;\n@@ -659,7 +698,7 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \tif (unlikely(i == 0))\n \t\treturn 0;\n \t/* Check whether completion threshold has been reached. */\n-\tcomp = txq->elts_comp + i;\n+\tcomp = txq->elts_comp + i + j;\n \tif (comp >= MLX5_TX_COMP_THRESH) {\n \t\t/* Request completion on last WQE. */\n \t\twqe->wqe.ctrl.data[2] = htonl(8);\n@@ -699,6 +738,7 @@ mlx5_tx_burst_inline(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \tuint16_t elts_head = txq->elts_head;\n \tconst unsigned int elts_n = txq->elts_n;\n \tunsigned int i = 0;\n+\tunsigned int j = 0;\n \tunsigned int max;\n \tunsigned int comp;\n \tvolatile union mlx5_wqe *wqe;\n@@ -716,23 +756,27 @@ mlx5_tx_burst_inline(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \tif (max > elts_n)\n \t\tmax -= elts_n;\n \tdo {\n-\t\tstruct rte_mbuf *buf;\n+\t\tstruct rte_mbuf *buf = *(pkts++);\n \t\tunsigned int elts_head_next;\n \t\tuintptr_t addr;\n \t\tuint32_t length;\n \t\tuint32_t lkey;\n+\t\tunsigned int segs_n = buf->nb_segs;\n+\t\tvolatile struct mlx5_wqe_data_seg *dseg;\n+\t\tunsigned int ds = sizeof(*wqe) / 16;\n \n \t\t/*\n \t\t * Make sure there is enough room to store this packet and\n \t\t * that one ring entry remains unused.\n \t\t */\n-\t\tif (max < 1 + 1)\n+\t\tassert(segs_n);\n+\t\tif (max < segs_n + 1)\n \t\t\tbreak;\n-\t\t--max;\n+\t\tmax -= segs_n;\n \t\t--pkts_n;\n-\t\tbuf = *(pkts++);\n \t\telts_head_next = (elts_head + 1) & (elts_n - 1);\n \t\twqe = &(*txq->wqes)[txq->wqe_ci & (txq->wqe_n - 1)];\n+\t\tdseg = &wqe->wqe.dseg;\n \t\ttx_prefetch_wqe(txq, txq->wqe_ci);\n \t\ttx_prefetch_wqe(txq, txq->wqe_ci + 1);\n \t\tif (pkts_n)\n@@ -755,13 +799,14 @@ mlx5_tx_burst_inline(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \t\tif (pkts_n)\n \t\t\trte_prefetch0(rte_pktmbuf_mtod(*pkts,\n \t\t\t\t\t\t volatile void *));\n-\t\tif (length <= max_inline) {\n+\t\tif ((length <= max_inline) && (segs_n == 1)) {\n \t\t\tif (buf->ol_flags & PKT_TX_VLAN_PKT)\n \t\t\t\tmlx5_wqe_write_inline_vlan(txq, wqe,\n \t\t\t\t\t\t\t addr, length,\n \t\t\t\t\t\t\t buf->vlan_tci);\n \t\t\telse\n \t\t\t\tmlx5_wqe_write_inline(txq, wqe, addr, length);\n+\t\t\tgoto skip_segs;\n \t\t} else {\n \t\t\t/* Retrieve Memory Region key for this memory pool. */\n \t\t\tlkey = txq_mp2mr(txq, txq_mb2mp(buf));\n@@ -771,7 +816,37 @@ mlx5_tx_burst_inline(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \t\t\telse\n \t\t\t\tmlx5_wqe_write(txq, wqe, addr, length, lkey);\n \t\t}\n-\t\twqe->inl.ctrl.data[2] = 0;\n+\t\twhile (--segs_n) {\n+\t\t\t/*\n+\t\t\t * Spill on next WQE when the current one does not have\n+\t\t\t * enough room left. Size of WQE must a be a multiple\n+\t\t\t * of data segment size.\n+\t\t\t */\n+\t\t\tassert(!(sizeof(*wqe) % sizeof(*dseg)));\n+\t\t\tif (!(ds % (sizeof(*wqe) / 16)))\n+\t\t\t\tdseg = (volatile void *)\n+\t\t\t\t\t&(*txq->wqes)[txq->wqe_ci++ &\n+\t\t\t\t\t\t (txq->wqe_n - 1)];\n+\t\t\telse\n+\t\t\t\t++dseg;\n+\t\t\t++ds;\n+\t\t\tbuf = buf->next;\n+\t\t\tassert(buf);\n+\t\t\t/* Store segment information. */\n+\t\t\tdseg->byte_count = htonl(DATA_LEN(buf));\n+\t\t\tdseg->lkey = txq_mp2mr(txq, txq_mb2mp(buf));\n+\t\t\tdseg->addr = htonll(rte_pktmbuf_mtod(buf, uintptr_t));\n+\t\t\t(*txq->elts)[elts_head_next] = buf;\n+\t\t\telts_head_next = (elts_head_next + 1) & (elts_n - 1);\n+#ifdef MLX5_PMD_SOFT_COUNTERS\n+\t\t\tlength += DATA_LEN(buf);\n+#endif\n+\t\t\t++j;\n+\t\t}\n+\t\t/* Update DS field in WQE. */\n+\t\twqe->inl.ctrl.data[1] &= htonl(0xffffffc0);\n+\t\twqe->inl.ctrl.data[1] |= htonl(ds & 0x3f);\n+skip_segs:\n \t\telts_head = elts_head_next;\n #ifdef MLX5_PMD_SOFT_COUNTERS\n \t\t/* Increment sent bytes counter. */\n@@ -783,7 +858,7 @@ mlx5_tx_burst_inline(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \tif (unlikely(i == 0))\n \t\treturn 0;\n \t/* Check whether completion threshold has been reached. */\n-\tcomp = txq->elts_comp + i;\n+\tcomp = txq->elts_comp + i + j;\n \tif (comp >= MLX5_TX_COMP_THRESH) {\n \t\t/* Request completion on last WQE. */\n \t\twqe->inl.ctrl.data[2] = htonl(8);\n@@ -890,6 +965,7 @@ mlx5_tx_burst_mpw(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \tuint16_t elts_head = txq->elts_head;\n \tconst unsigned int elts_n = txq->elts_n;\n \tunsigned int i = 0;\n+\tunsigned int j = 0;\n \tunsigned int max;\n \tunsigned int comp;\n \tstruct mlx5_mpw mpw = {\n@@ -908,48 +984,69 @@ mlx5_tx_burst_mpw(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \tif (max > elts_n)\n \t\tmax -= elts_n;\n \tdo {\n-\t\tstruct rte_mbuf *buf;\n-\t\tvolatile struct mlx5_wqe_data_seg *dseg;\n+\t\tstruct rte_mbuf *buf = *(pkts++);\n \t\tunsigned int elts_head_next;\n-\t\tuintptr_t addr;\n \t\tuint32_t length;\n+\t\tunsigned int segs_n = buf->nb_segs;\n \t\tuint32_t cs_flags = 0;\n \n \t\t/*\n \t\t * Make sure there is enough room to store this packet and\n \t\t * that one ring entry remains unused.\n \t\t */\n-\t\tif (max < 1 + 1)\n+\t\tassert(segs_n);\n+\t\tif (max < segs_n + 1)\n \t\t\tbreak;\n-\t\t--max;\n+\t\t/* Do not bother with large packets MPW cannot handle. */\n+\t\tif (segs_n > MLX5_MPW_DSEG_MAX)\n+\t\t\tbreak;\n+\t\tmax -= segs_n;\n \t\t--pkts_n;\n-\t\tbuf = *(pkts++);\n-\t\telts_head_next = (elts_head + 1) & (elts_n - 1);\n \t\t/* Should we enable HW CKSUM offload */\n \t\tif (buf->ol_flags &\n \t\t (PKT_TX_IP_CKSUM | PKT_TX_TCP_CKSUM | PKT_TX_UDP_CKSUM))\n \t\t\tcs_flags = MLX5_ETH_WQE_L3_CSUM | MLX5_ETH_WQE_L4_CSUM;\n-\t\t/* Retrieve buffer information. */\n-\t\taddr = rte_pktmbuf_mtod(buf, uintptr_t);\n-\t\tlength = DATA_LEN(buf);\n-\t\t/* Update element. */\n-\t\t(*txq->elts)[elts_head] = buf;\n+\t\t/* Retrieve packet information. */\n+\t\tlength = PKT_LEN(buf);\n+\t\tassert(length);\n \t\t/* Start new session if packet differs. */\n \t\tif ((mpw.state == MLX5_MPW_STATE_OPENED) &&\n \t\t ((mpw.len != length) ||\n+\t\t (segs_n != 1) ||\n \t\t (mpw.wqe->mpw.eseg.cs_flags != cs_flags)))\n \t\t\tmlx5_mpw_close(txq, &mpw);\n \t\tif (mpw.state == MLX5_MPW_STATE_CLOSED) {\n \t\t\tmlx5_mpw_new(txq, &mpw, length);\n \t\t\tmpw.wqe->mpw.eseg.cs_flags = cs_flags;\n \t\t}\n-\t\tdseg = mpw.data.dseg[mpw.pkts_n];\n-\t\t*dseg = (struct mlx5_wqe_data_seg){\n-\t\t\t.byte_count = htonl(length),\n-\t\t\t.lkey = txq_mp2mr(txq, txq_mb2mp(buf)),\n-\t\t\t.addr = htonll(addr),\n-\t\t};\n-\t\t++mpw.pkts_n;\n+\t\t/* Multi-segment packets must be alone in their MPW. */\n+\t\tassert((segs_n == 1) || (mpw.pkts_n == 0));\n+#if defined(MLX5_PMD_SOFT_COUNTERS) || !defined(NDEBUG)\n+\t\tlength = 0;\n+#endif\n+\t\tdo {\n+\t\t\tvolatile struct mlx5_wqe_data_seg *dseg;\n+\t\t\tuintptr_t addr;\n+\n+\t\t\telts_head_next = (elts_head + 1) & (elts_n - 1);\n+\t\t\tassert(buf);\n+\t\t\t(*txq->elts)[elts_head] = buf;\n+\t\t\tdseg = mpw.data.dseg[mpw.pkts_n];\n+\t\t\taddr = rte_pktmbuf_mtod(buf, uintptr_t);\n+\t\t\t*dseg = (struct mlx5_wqe_data_seg){\n+\t\t\t\t.byte_count = htonl(DATA_LEN(buf)),\n+\t\t\t\t.lkey = txq_mp2mr(txq, txq_mb2mp(buf)),\n+\t\t\t\t.addr = htonll(addr),\n+\t\t\t};\n+\t\t\telts_head = elts_head_next;\n+#if defined(MLX5_PMD_SOFT_COUNTERS) || !defined(NDEBUG)\n+\t\t\tlength += DATA_LEN(buf);\n+#endif\n+\t\t\tbuf = buf->next;\n+\t\t\t++mpw.pkts_n;\n+\t\t\t++j;\n+\t\t} while (--segs_n);\n+\t\tassert(length == mpw.len);\n \t\tif (mpw.pkts_n == MLX5_MPW_DSEG_MAX)\n \t\t\tmlx5_mpw_close(txq, &mpw);\n \t\telts_head = elts_head_next;\n@@ -963,7 +1060,8 @@ mlx5_tx_burst_mpw(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \tif (unlikely(i == 0))\n \t\treturn 0;\n \t/* Check whether completion threshold has been reached. */\n-\tcomp = txq->elts_comp + i;\n+\t/* \"j\" includes both packets and segments. */\n+\tcomp = txq->elts_comp + j;\n \tif (comp >= MLX5_TX_COMP_THRESH) {\n \t\tvolatile union mlx5_wqe *wqe = mpw.wqe;\n \n@@ -1067,6 +1165,7 @@ mlx5_tx_burst_mpw_inline(void *dpdk_txq, struct rte_mbuf **pkts,\n \tuint16_t elts_head = txq->elts_head;\n \tconst unsigned int elts_n = txq->elts_n;\n \tunsigned int i = 0;\n+\tunsigned int j = 0;\n \tunsigned int max;\n \tunsigned int comp;\n \tunsigned int inline_room = txq->max_inline;\n@@ -1086,38 +1185,40 @@ mlx5_tx_burst_mpw_inline(void *dpdk_txq, struct rte_mbuf **pkts,\n \tif (max > elts_n)\n \t\tmax -= elts_n;\n \tdo {\n-\t\tstruct rte_mbuf *buf;\n+\t\tstruct rte_mbuf *buf = *(pkts++);\n \t\tunsigned int elts_head_next;\n \t\tuintptr_t addr;\n \t\tuint32_t length;\n+\t\tunsigned int segs_n = buf->nb_segs;\n \t\tuint32_t cs_flags = 0;\n \n \t\t/*\n \t\t * Make sure there is enough room to store this packet and\n \t\t * that one ring entry remains unused.\n \t\t */\n-\t\tif (max < 1 + 1)\n+\t\tassert(segs_n);\n+\t\tif (max < segs_n + 1)\n+\t\t\tbreak;\n+\t\t/* Do not bother with large packets MPW cannot handle. */\n+\t\tif (segs_n > MLX5_MPW_DSEG_MAX)\n \t\t\tbreak;\n-\t\t--max;\n+\t\tmax -= segs_n;\n \t\t--pkts_n;\n-\t\tbuf = *(pkts++);\n-\t\telts_head_next = (elts_head + 1) & (elts_n - 1);\n \t\t/* Should we enable HW CKSUM offload */\n \t\tif (buf->ol_flags &\n \t\t (PKT_TX_IP_CKSUM | PKT_TX_TCP_CKSUM | PKT_TX_UDP_CKSUM))\n \t\t\tcs_flags = MLX5_ETH_WQE_L3_CSUM | MLX5_ETH_WQE_L4_CSUM;\n-\t\t/* Retrieve buffer information. */\n-\t\taddr = rte_pktmbuf_mtod(buf, uintptr_t);\n-\t\tlength = DATA_LEN(buf);\n-\t\t/* Update element. */\n-\t\t(*txq->elts)[elts_head] = buf;\n+\t\t/* Retrieve packet information. */\n+\t\tlength = PKT_LEN(buf);\n \t\t/* Start new session if packet differs. */\n \t\tif (mpw.state == MLX5_MPW_STATE_OPENED) {\n \t\t\tif ((mpw.len != length) ||\n+\t\t\t (segs_n != 1) ||\n \t\t\t (mpw.wqe->mpw.eseg.cs_flags != cs_flags))\n \t\t\t\tmlx5_mpw_close(txq, &mpw);\n \t\t} else if (mpw.state == MLX5_MPW_INL_STATE_OPENED) {\n \t\t\tif ((mpw.len != length) ||\n+\t\t\t (segs_n != 1) ||\n \t\t\t (length > inline_room) ||\n \t\t\t (mpw.wqe->mpw_inl.eseg.cs_flags != cs_flags)) {\n \t\t\t\tmlx5_mpw_inline_close(txq, &mpw);\n@@ -1125,7 +1226,8 @@ mlx5_tx_burst_mpw_inline(void *dpdk_txq, struct rte_mbuf **pkts,\n \t\t\t}\n \t\t}\n \t\tif (mpw.state == MLX5_MPW_STATE_CLOSED) {\n-\t\t\tif (length > inline_room) {\n+\t\t\tif ((segs_n != 1) ||\n+\t\t\t (length > inline_room)) {\n \t\t\t\tmlx5_mpw_new(txq, &mpw, length);\n \t\t\t\tmpw.wqe->mpw.eseg.cs_flags = cs_flags;\n \t\t\t} else {\n@@ -1133,17 +1235,36 @@ mlx5_tx_burst_mpw_inline(void *dpdk_txq, struct rte_mbuf **pkts,\n \t\t\t\tmpw.wqe->mpw_inl.eseg.cs_flags = cs_flags;\n \t\t\t}\n \t\t}\n+\t\t/* Multi-segment packets must be alone in their MPW. */\n+\t\tassert((segs_n == 1) || (mpw.pkts_n == 0));\n \t\tif (mpw.state == MLX5_MPW_STATE_OPENED) {\n-\t\t\tvolatile struct mlx5_wqe_data_seg *dseg;\n-\n \t\t\tassert(inline_room == txq->max_inline);\n-\t\t\tdseg = mpw.data.dseg[mpw.pkts_n];\n-\t\t\t*dseg = (struct mlx5_wqe_data_seg){\n-\t\t\t\t.byte_count = htonl(length),\n-\t\t\t\t.lkey = txq_mp2mr(txq, txq_mb2mp(buf)),\n-\t\t\t\t.addr = htonll(addr),\n-\t\t\t};\n-\t\t\t++mpw.pkts_n;\n+#if defined(MLX5_PMD_SOFT_COUNTERS) || !defined(NDEBUG)\n+\t\t\tlength = 0;\n+#endif\n+\t\t\tdo {\n+\t\t\t\tvolatile struct mlx5_wqe_data_seg *dseg;\n+\n+\t\t\t\telts_head_next =\n+\t\t\t\t\t(elts_head + 1) & (elts_n - 1);\n+\t\t\t\tassert(buf);\n+\t\t\t\t(*txq->elts)[elts_head] = buf;\n+\t\t\t\tdseg = mpw.data.dseg[mpw.pkts_n];\n+\t\t\t\taddr = rte_pktmbuf_mtod(buf, uintptr_t);\n+\t\t\t\t*dseg = (struct mlx5_wqe_data_seg){\n+\t\t\t\t\t.byte_count = htonl(DATA_LEN(buf)),\n+\t\t\t\t\t.lkey = txq_mp2mr(txq, txq_mb2mp(buf)),\n+\t\t\t\t\t.addr = htonll(addr),\n+\t\t\t\t};\n+\t\t\t\telts_head = elts_head_next;\n+#if defined(MLX5_PMD_SOFT_COUNTERS) || !defined(NDEBUG)\n+\t\t\t\tlength += DATA_LEN(buf);\n+#endif\n+\t\t\t\tbuf = buf->next;\n+\t\t\t\t++mpw.pkts_n;\n+\t\t\t\t++j;\n+\t\t\t} while (--segs_n);\n+\t\t\tassert(length == mpw.len);\n \t\t\tif (mpw.pkts_n == MLX5_MPW_DSEG_MAX)\n \t\t\t\tmlx5_mpw_close(txq, &mpw);\n \t\t} else {\n@@ -1151,6 +1272,10 @@ mlx5_tx_burst_mpw_inline(void *dpdk_txq, struct rte_mbuf **pkts,\n \n \t\t\tassert(mpw.state == MLX5_MPW_INL_STATE_OPENED);\n \t\t\tassert(length <= inline_room);\n+\t\t\tassert(length == DATA_LEN(buf));\n+\t\t\telts_head_next = (elts_head + 1) & (elts_n - 1);\n+\t\t\taddr = rte_pktmbuf_mtod(buf, uintptr_t);\n+\t\t\t(*txq->elts)[elts_head] = buf;\n \t\t\t/* Maximum number of bytes before wrapping. */\n \t\t\tmax = ((uintptr_t)&(*txq->wqes)[txq->wqe_n] -\n \t\t\t (uintptr_t)mpw.data.raw);\n@@ -1175,6 +1300,7 @@ mlx5_tx_burst_mpw_inline(void *dpdk_txq, struct rte_mbuf **pkts,\n \t\t\t\tmpw.data.raw =\n \t\t\t\t\t(volatile void *)&(*txq->wqes)[0];\n \t\t\t++mpw.pkts_n;\n+\t\t\t++j;\n \t\t\tif (mpw.pkts_n == MLX5_MPW_DSEG_MAX) {\n \t\t\t\tmlx5_mpw_inline_close(txq, &mpw);\n \t\t\t\tinline_room = txq->max_inline;\n@@ -1194,7 +1320,8 @@ mlx5_tx_burst_mpw_inline(void *dpdk_txq, struct rte_mbuf **pkts,\n \tif (unlikely(i == 0))\n \t\treturn 0;\n \t/* Check whether completion threshold has been reached. */\n-\tcomp = txq->elts_comp + i;\n+\t/* \"j\" includes both packets and segments. */\n+\tcomp = txq->elts_comp + j;\n \tif (comp >= MLX5_TX_COMP_THRESH) {\n \t\tvolatile union mlx5_wqe *wqe = mpw.wqe;\n \ndiff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c\nindex bae9f3d..6fe61c4 100644\n--- a/drivers/net/mlx5/mlx5_txq.c\n+++ b/drivers/net/mlx5/mlx5_txq.c\n@@ -320,7 +320,13 @@ txq_ctrl_setup(struct rte_eth_dev *dev, struct txq_ctrl *txq_ctrl,\n \t\t\t.max_send_wr = ((priv->device_attr.max_qp_wr < desc) ?\n \t\t\t\t\tpriv->device_attr.max_qp_wr :\n \t\t\t\t\tdesc),\n-\t\t\t/* Max number of scatter/gather elements in a WR. */\n+\t\t\t/*\n+\t\t\t * Max number of scatter/gather elements in a WR,\n+\t\t\t * must be 1 to prevent libmlx5 from trying to affect\n+\t\t\t * too much memory. TX gather is not impacted by the\n+\t\t\t * priv->device_attr.max_sge limit and will still work\n+\t\t\t * properly.\n+\t\t\t */\n \t\t\t.max_send_sge = 1,\n \t\t},\n \t\t.qp_type = IBV_QPT_RAW_PACKET,\n", "prefixes": [ "dpdk-dev", "v7", "21/25" ] }