Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/7432/?format=api
{ "id": 7432, "url": "https://patches.dpdk.org/api/patches/7432/?format=api", "web_url": "https://patches.dpdk.org/project/dpdk/patch/1444067589-29513-6-git-send-email-adrien.mazarguil@6wind.com/", "project": { "id": 1, "url": "https://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<1444067589-29513-6-git-send-email-adrien.mazarguil@6wind.com>", "list_archive_url": "https://inbox.dpdk.org/dev/1444067589-29513-6-git-send-email-adrien.mazarguil@6wind.com", "date": "2015-10-05T17:53:01", "name": "[dpdk-dev,05/13] mlx5: add support for scattered RX and TX buffers", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": true, "hash": "61f585c415647f54b8ec71e1765af2373bf2b1fb", "submitter": { "id": 165, "url": "https://patches.dpdk.org/api/people/165/?format=api", "name": "Adrien Mazarguil", "email": "adrien.mazarguil@6wind.com" }, "delegate": null, "mbox": "https://patches.dpdk.org/project/dpdk/patch/1444067589-29513-6-git-send-email-adrien.mazarguil@6wind.com/mbox/", "series": [], "comments": "https://patches.dpdk.org/api/patches/7432/comments/", "check": "pending", "checks": "https://patches.dpdk.org/api/patches/7432/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [IPv6:::1])\n\tby dpdk.org (Postfix) with ESMTP id 5BB719249;\n\tMon, 5 Oct 2015 19:53:48 +0200 (CEST)", "from mail-wi0-f171.google.com (mail-wi0-f171.google.com\n\t[209.85.212.171]) by dpdk.org (Postfix) with ESMTP id 1CBDC9246\n\tfor <dev@dpdk.org>; Mon, 5 Oct 2015 19:53:47 +0200 (CEST)", "by wicgb1 with SMTP id gb1so130794665wic.1\n\tfor <dev@dpdk.org>; Mon, 05 Oct 2015 10:53:47 -0700 (PDT)", "from 6wind.com (guy78-3-82-239-227-177.fbx.proxad.net.\n\t[82.239.227.177]) by smtp.gmail.com with ESMTPSA id\n\tmc20sm15792429wic.16.2015.10.05.10.53.45\n\t(version=TLSv1.2 cipher=RC4-SHA bits=128/128);\n\tMon, 05 Oct 2015 10:53:46 -0700 (PDT)" ], "X-Google-DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20130820;\n\th=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to\n\t:references;\n\tbh=i6vh6OIkZ0uof+SAgmc97Xq+PPCJSy/m6Qz7jwIg0+A=;\n\tb=VPhocdtTeIY/wBYId6CNL4x2/Vy+M0ei9m82AMgEOc+DC+xphRojcLYadUHDFHJEz/\n\tDqef6Us4FEWPaE4qtKD62+IdamkUClT0RuL5lw3u3kXR39Zoxla/1p4qk+sMd5NAmnIj\n\tcIGrxr2p+y23rtSuqcBICfwcJG8hqrPPdUh6WG4gf+hoRQAxEU1nJbPabxatveRic4fP\n\t6bP812uIPyiOheaTBe2TdjZcsCHAJzRpmCsyRhFdhgQ/0+y+PtEZS92Yex2DrekZCF/i\n\t7V/znZDKWfPFFeuNzdjvR5GB78TyHthHzO9vDvOlNtcacN3Lx5EX6Cv+Uy5/QacVwcRf\n\tN/ig==", "X-Gm-Message-State": "ALoCoQm2ZnCQv+vAPEijSXKL2fpmJptImauPsDV/TOyqETfzMytgFwAVcKTUgBcjmChVsguNt38H", "X-Received": "by 10.194.184.20 with SMTP id eq20mr2572531wjc.22.1444067626783; \n\tMon, 05 Oct 2015 10:53:46 -0700 (PDT)", "From": "Adrien Mazarguil <adrien.mazarguil@6wind.com>", "To": "dev@dpdk.org", "Date": "Mon, 5 Oct 2015 19:53:01 +0200", "Message-Id": "<1444067589-29513-6-git-send-email-adrien.mazarguil@6wind.com>", "X-Mailer": "git-send-email 2.1.0", "In-Reply-To": "<1444067589-29513-1-git-send-email-adrien.mazarguil@6wind.com>", "References": "<1444067589-29513-1-git-send-email-adrien.mazarguil@6wind.com>", "Subject": "[dpdk-dev] [PATCH 05/13] mlx5: add support for scattered RX and TX\n\tbuffers", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "patches and discussions about DPDK <dev.dpdk.org>", "List-Unsubscribe": "<http://dpdk.org/ml/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://dpdk.org/ml/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<http://dpdk.org/ml/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "A dedicated RX callback is added to handle scattered buffers. For better\nperformance, it is only used when jumbo frames are enabled and MTU is larger\nthan a single mbuf.\n\nOn the TX path, scattered buffers are also handled in a separate function.\nWhen there are more than MLX5_PMD_SGE_WR_N segments in a given mbuf, the\nremaining segments are linearized in the last SGE entry.\n\nSigned-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>\nSigned-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>\n---\n drivers/net/mlx5/mlx5_rxq.c | 175 +++++++++++++++++++-\n drivers/net/mlx5/mlx5_rxtx.c | 376 +++++++++++++++++++++++++++++++++++++++++++\n drivers/net/mlx5/mlx5_rxtx.h | 10 ++\n 3 files changed, 557 insertions(+), 4 deletions(-)", "diff": "diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c\nindex 8450fe3..1eddfc7 100644\n--- a/drivers/net/mlx5/mlx5_rxq.c\n+++ b/drivers/net/mlx5/mlx5_rxq.c\n@@ -65,6 +65,153 @@\n #include \"mlx5_defs.h\"\n \n /**\n+ * Allocate RX queue elements with scattered packets support.\n+ *\n+ * @param rxq\n+ * Pointer to RX queue structure.\n+ * @param elts_n\n+ * Number of elements to allocate.\n+ * @param[in] pool\n+ * If not NULL, fetch buffers from this array instead of allocating them\n+ * with rte_pktmbuf_alloc().\n+ *\n+ * @return\n+ * 0 on success, errno value on failure.\n+ */\n+static int\n+rxq_alloc_elts_sp(struct rxq *rxq, unsigned int elts_n,\n+\t\t struct rte_mbuf **pool)\n+{\n+\tunsigned int i;\n+\tstruct rxq_elt_sp (*elts)[elts_n] =\n+\t\trte_calloc_socket(\"RXQ elements\", 1, sizeof(*elts), 0,\n+\t\t\t\t rxq->socket);\n+\tint ret = 0;\n+\n+\tif (elts == NULL) {\n+\t\tERROR(\"%p: can't allocate packets array\", (void *)rxq);\n+\t\tret = ENOMEM;\n+\t\tgoto error;\n+\t}\n+\t/* For each WR (packet). */\n+\tfor (i = 0; (i != elts_n); ++i) {\n+\t\tunsigned int j;\n+\t\tstruct rxq_elt_sp *elt = &(*elts)[i];\n+\t\tstruct ibv_recv_wr *wr = &elt->wr;\n+\t\tstruct ibv_sge (*sges)[RTE_DIM(elt->sges)] = &elt->sges;\n+\n+\t\t/* These two arrays must have the same size. */\n+\t\tassert(RTE_DIM(elt->sges) == RTE_DIM(elt->bufs));\n+\t\t/* Configure WR. */\n+\t\twr->wr_id = i;\n+\t\twr->next = &(*elts)[(i + 1)].wr;\n+\t\twr->sg_list = &(*sges)[0];\n+\t\twr->num_sge = RTE_DIM(*sges);\n+\t\t/* For each SGE (segment). */\n+\t\tfor (j = 0; (j != RTE_DIM(elt->bufs)); ++j) {\n+\t\t\tstruct ibv_sge *sge = &(*sges)[j];\n+\t\t\tstruct rte_mbuf *buf;\n+\n+\t\t\tif (pool != NULL) {\n+\t\t\t\tbuf = *(pool++);\n+\t\t\t\tassert(buf != NULL);\n+\t\t\t\trte_pktmbuf_reset(buf);\n+\t\t\t} else\n+\t\t\t\tbuf = rte_pktmbuf_alloc(rxq->mp);\n+\t\t\tif (buf == NULL) {\n+\t\t\t\tassert(pool == NULL);\n+\t\t\t\tERROR(\"%p: empty mbuf pool\", (void *)rxq);\n+\t\t\t\tret = ENOMEM;\n+\t\t\t\tgoto error;\n+\t\t\t}\n+\t\t\telt->bufs[j] = buf;\n+\t\t\t/* Headroom is reserved by rte_pktmbuf_alloc(). */\n+\t\t\tassert(DATA_OFF(buf) == RTE_PKTMBUF_HEADROOM);\n+\t\t\t/* Buffer is supposed to be empty. */\n+\t\t\tassert(rte_pktmbuf_data_len(buf) == 0);\n+\t\t\tassert(rte_pktmbuf_pkt_len(buf) == 0);\n+\t\t\t/* sge->addr must be able to store a pointer. */\n+\t\t\tassert(sizeof(sge->addr) >= sizeof(uintptr_t));\n+\t\t\tif (j == 0) {\n+\t\t\t\t/* The first SGE keeps its headroom. */\n+\t\t\t\tsge->addr = rte_pktmbuf_mtod(buf, uintptr_t);\n+\t\t\t\tsge->length = (buf->buf_len -\n+\t\t\t\t\t RTE_PKTMBUF_HEADROOM);\n+\t\t\t} else {\n+\t\t\t\t/* Subsequent SGEs lose theirs. */\n+\t\t\t\tassert(DATA_OFF(buf) == RTE_PKTMBUF_HEADROOM);\n+\t\t\t\tSET_DATA_OFF(buf, 0);\n+\t\t\t\tsge->addr = (uintptr_t)buf->buf_addr;\n+\t\t\t\tsge->length = buf->buf_len;\n+\t\t\t}\n+\t\t\tsge->lkey = rxq->mr->lkey;\n+\t\t\t/* Redundant check for tailroom. */\n+\t\t\tassert(sge->length == rte_pktmbuf_tailroom(buf));\n+\t\t}\n+\t}\n+\t/* The last WR pointer must be NULL. */\n+\t(*elts)[(i - 1)].wr.next = NULL;\n+\tDEBUG(\"%p: allocated and configured %u WRs (%zu segments)\",\n+\t (void *)rxq, elts_n, (elts_n * RTE_DIM((*elts)[0].sges)));\n+\trxq->elts_n = elts_n;\n+\trxq->elts_head = 0;\n+\trxq->elts.sp = elts;\n+\tassert(ret == 0);\n+\treturn 0;\n+error:\n+\tif (elts != NULL) {\n+\t\tassert(pool == NULL);\n+\t\tfor (i = 0; (i != RTE_DIM(*elts)); ++i) {\n+\t\t\tunsigned int j;\n+\t\t\tstruct rxq_elt_sp *elt = &(*elts)[i];\n+\n+\t\t\tfor (j = 0; (j != RTE_DIM(elt->bufs)); ++j) {\n+\t\t\t\tstruct rte_mbuf *buf = elt->bufs[j];\n+\n+\t\t\t\tif (buf != NULL)\n+\t\t\t\t\trte_pktmbuf_free_seg(buf);\n+\t\t\t}\n+\t\t}\n+\t\trte_free(elts);\n+\t}\n+\tDEBUG(\"%p: failed, freed everything\", (void *)rxq);\n+\tassert(ret > 0);\n+\treturn ret;\n+}\n+\n+/**\n+ * Free RX queue elements with scattered packets support.\n+ *\n+ * @param rxq\n+ * Pointer to RX queue structure.\n+ */\n+static void\n+rxq_free_elts_sp(struct rxq *rxq)\n+{\n+\tunsigned int i;\n+\tunsigned int elts_n = rxq->elts_n;\n+\tstruct rxq_elt_sp (*elts)[elts_n] = rxq->elts.sp;\n+\n+\tDEBUG(\"%p: freeing WRs\", (void *)rxq);\n+\trxq->elts_n = 0;\n+\trxq->elts.sp = NULL;\n+\tif (elts == NULL)\n+\t\treturn;\n+\tfor (i = 0; (i != RTE_DIM(*elts)); ++i) {\n+\t\tunsigned int j;\n+\t\tstruct rxq_elt_sp *elt = &(*elts)[i];\n+\n+\t\tfor (j = 0; (j != RTE_DIM(elt->bufs)); ++j) {\n+\t\t\tstruct rte_mbuf *buf = elt->bufs[j];\n+\n+\t\t\tif (buf != NULL)\n+\t\t\t\trte_pktmbuf_free_seg(buf);\n+\t\t}\n+\t}\n+\trte_free(elts);\n+}\n+\n+/**\n * Allocate RX queue elements.\n *\n * @param rxq\n@@ -224,7 +371,10 @@ rxq_cleanup(struct rxq *rxq)\n \tstruct ibv_exp_release_intf_params params;\n \n \tDEBUG(\"cleaning up %p\", (void *)rxq);\n-\trxq_free_elts(rxq);\n+\tif (rxq->sp)\n+\t\trxq_free_elts_sp(rxq);\n+\telse\n+\t\trxq_free_elts(rxq);\n \tif (rxq->if_qp != NULL) {\n \t\tassert(rxq->priv != NULL);\n \t\tassert(rxq->priv->ctx != NULL);\n@@ -445,6 +595,15 @@ rxq_setup(struct rte_eth_dev *dev, struct rxq *rxq, uint16_t desc,\n \t\trte_pktmbuf_tailroom(buf)) == tmpl.mb_len);\n \tassert(rte_pktmbuf_headroom(buf) == RTE_PKTMBUF_HEADROOM);\n \trte_pktmbuf_free(buf);\n+\t/* Enable scattered packets support for this queue if necessary. */\n+\tif ((dev->data->dev_conf.rxmode.jumbo_frame) &&\n+\t (dev->data->dev_conf.rxmode.max_rx_pkt_len >\n+\t (tmpl.mb_len - RTE_PKTMBUF_HEADROOM))) {\n+\t\ttmpl.sp = 1;\n+\t\tdesc /= MLX5_PMD_SGE_WR_N;\n+\t}\n+\tDEBUG(\"%p: %s scattered packets support (%u WRs)\",\n+\t (void *)dev, (tmpl.sp ? \"enabling\" : \"disabling\"), desc);\n \t/* Use the entire RX mempool as the memory region. */\n \ttmpl.mr = ibv_reg_mr(priv->pd,\n \t\t\t (void *)mp->elt_va_start,\n@@ -528,14 +687,19 @@ skip_mr:\n \t/* Allocate descriptors for RX queues, except for the RSS parent. */\n \tif (parent)\n \t\tgoto skip_alloc;\n-\tret = rxq_alloc_elts(&tmpl, desc, NULL);\n+\tif (tmpl.sp)\n+\t\tret = rxq_alloc_elts_sp(&tmpl, desc, NULL);\n+\telse\n+\t\tret = rxq_alloc_elts(&tmpl, desc, NULL);\n \tif (ret) {\n \t\tERROR(\"%p: RXQ allocation failed: %s\",\n \t\t (void *)dev, strerror(ret));\n \t\tgoto error;\n \t}\n \tret = ibv_post_recv(tmpl.qp,\n-\t\t\t &(*tmpl.elts.no_sp)[0].wr,\n+\t\t\t (tmpl.sp ?\n+\t\t\t &(*tmpl.elts.sp)[0].wr :\n+\t\t\t &(*tmpl.elts.no_sp)[0].wr),\n \t\t\t &bad_wr);\n \tif (ret) {\n \t\tERROR(\"%p: ibv_post_recv() failed for WR %p: %s\",\n@@ -655,7 +819,10 @@ mlx5_rx_queue_setup(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc,\n \t\t (void *)dev, (void *)rxq);\n \t\t(*priv->rxqs)[idx] = rxq;\n \t\t/* Update receive callback. */\n-\t\tdev->rx_pkt_burst = mlx5_rx_burst;\n+\t\tif (rxq->sp)\n+\t\t\tdev->rx_pkt_burst = mlx5_rx_burst_sp;\n+\t\telse\n+\t\t\tdev->rx_pkt_burst = mlx5_rx_burst;\n \t}\n \tpriv_unlock(priv);\n \treturn -ret;\ndiff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c\nindex 40bddf0..5042011 100644\n--- a/drivers/net/mlx5/mlx5_rxtx.c\n+++ b/drivers/net/mlx5/mlx5_rxtx.c\n@@ -173,6 +173,154 @@ txq_mp2mr(struct txq *txq, struct rte_mempool *mp)\n \treturn txq->mp2mr[i].lkey;\n }\n \n+#if MLX5_PMD_SGE_WR_N > 1\n+\n+/**\n+ * Copy scattered mbuf contents to a single linear buffer.\n+ *\n+ * @param[out] linear\n+ * Linear output buffer.\n+ * @param[in] buf\n+ * Scattered input buffer.\n+ *\n+ * @return\n+ * Number of bytes copied to the output buffer or 0 if not large enough.\n+ */\n+static unsigned int\n+linearize_mbuf(linear_t *linear, struct rte_mbuf *buf)\n+{\n+\tunsigned int size = 0;\n+\tunsigned int offset;\n+\n+\tdo {\n+\t\tunsigned int len = DATA_LEN(buf);\n+\n+\t\toffset = size;\n+\t\tsize += len;\n+\t\tif (unlikely(size > sizeof(*linear)))\n+\t\t\treturn 0;\n+\t\tmemcpy(&(*linear)[offset],\n+\t\t rte_pktmbuf_mtod(buf, uint8_t *),\n+\t\t len);\n+\t\tbuf = NEXT(buf);\n+\t} while (buf != NULL);\n+\treturn size;\n+}\n+\n+/**\n+ * Handle scattered buffers for mlx5_tx_burst().\n+ *\n+ * @param txq\n+ * TX queue structure.\n+ * @param segs\n+ * Number of segments in buf.\n+ * @param elt\n+ * TX queue element to fill.\n+ * @param[in] buf\n+ * Buffer to process.\n+ * @param elts_head\n+ * Index of the linear buffer to use if necessary (normally txq->elts_head).\n+ * @param[out] sges\n+ * Array filled with SGEs on success.\n+ *\n+ * @return\n+ * A structure containing the processed packet size in bytes and the\n+ * number of SGEs. Both fields are set to (unsigned int)-1 in case of\n+ * failure.\n+ */\n+static struct tx_burst_sg_ret {\n+\tunsigned int length;\n+\tunsigned int num;\n+}\n+tx_burst_sg(struct txq *txq, unsigned int segs, struct txq_elt *elt,\n+\t struct rte_mbuf *buf, unsigned int elts_head,\n+\t struct ibv_sge (*sges)[MLX5_PMD_SGE_WR_N])\n+{\n+\tunsigned int sent_size = 0;\n+\tunsigned int j;\n+\tint linearize = 0;\n+\n+\t/* When there are too many segments, extra segments are\n+\t * linearized in the last SGE. */\n+\tif (unlikely(segs > RTE_DIM(*sges))) {\n+\t\tsegs = (RTE_DIM(*sges) - 1);\n+\t\tlinearize = 1;\n+\t}\n+\t/* Update element. */\n+\telt->buf = buf;\n+\t/* Register segments as SGEs. */\n+\tfor (j = 0; (j != segs); ++j) {\n+\t\tstruct ibv_sge *sge = &(*sges)[j];\n+\t\tuint32_t lkey;\n+\n+\t\t/* Retrieve Memory Region key for this memory pool. */\n+\t\tlkey = txq_mp2mr(txq, buf->pool);\n+\t\tif (unlikely(lkey == (uint32_t)-1)) {\n+\t\t\t/* MR does not exist. */\n+\t\t\tDEBUG(\"%p: unable to get MP <-> MR association\",\n+\t\t\t (void *)txq);\n+\t\t\t/* Clean up TX element. */\n+\t\t\telt->buf = NULL;\n+\t\t\tgoto stop;\n+\t\t}\n+\t\t/* Update SGE. */\n+\t\tsge->addr = rte_pktmbuf_mtod(buf, uintptr_t);\n+\t\tif (txq->priv->vf)\n+\t\t\trte_prefetch0((volatile void *)\n+\t\t\t\t (uintptr_t)sge->addr);\n+\t\tsge->length = DATA_LEN(buf);\n+\t\tsge->lkey = lkey;\n+\t\tsent_size += sge->length;\n+\t\tbuf = NEXT(buf);\n+\t}\n+\t/* If buf is not NULL here and is not going to be linearized,\n+\t * nb_segs is not valid. */\n+\tassert(j == segs);\n+\tassert((buf == NULL) || (linearize));\n+\t/* Linearize extra segments. */\n+\tif (linearize) {\n+\t\tstruct ibv_sge *sge = &(*sges)[segs];\n+\t\tlinear_t *linear = &(*txq->elts_linear)[elts_head];\n+\t\tunsigned int size = linearize_mbuf(linear, buf);\n+\n+\t\tassert(segs == (RTE_DIM(*sges) - 1));\n+\t\tif (size == 0) {\n+\t\t\t/* Invalid packet. */\n+\t\t\tDEBUG(\"%p: packet too large to be linearized.\",\n+\t\t\t (void *)txq);\n+\t\t\t/* Clean up TX element. */\n+\t\t\telt->buf = NULL;\n+\t\t\tgoto stop;\n+\t\t}\n+\t\t/* If MLX5_PMD_SGE_WR_N is 1, free mbuf immediately. */\n+\t\tif (RTE_DIM(*sges) == 1) {\n+\t\t\tdo {\n+\t\t\t\tstruct rte_mbuf *next = NEXT(buf);\n+\n+\t\t\t\trte_pktmbuf_free_seg(buf);\n+\t\t\t\tbuf = next;\n+\t\t\t} while (buf != NULL);\n+\t\t\telt->buf = NULL;\n+\t\t}\n+\t\t/* Update SGE. */\n+\t\tsge->addr = (uintptr_t)&(*linear)[0];\n+\t\tsge->length = size;\n+\t\tsge->lkey = txq->mr_linear->lkey;\n+\t\tsent_size += size;\n+\t}\n+\treturn (struct tx_burst_sg_ret){\n+\t\t.length = sent_size,\n+\t\t.num = segs,\n+\t};\n+stop:\n+\treturn (struct tx_burst_sg_ret){\n+\t\t.length = -1,\n+\t\t.num = -1,\n+\t};\n+}\n+\n+#endif /* MLX5_PMD_SGE_WR_N > 1 */\n+\n /**\n * DPDK callback for TX.\n *\n@@ -282,9 +430,28 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \t\t\tif (unlikely(err))\n \t\t\t\tgoto stop;\n \t\t} else {\n+#if MLX5_PMD_SGE_WR_N > 1\n+\t\t\tstruct ibv_sge sges[MLX5_PMD_SGE_WR_N];\n+\t\t\tstruct tx_burst_sg_ret ret;\n+\n+\t\t\tret = tx_burst_sg(txq, segs, elt, buf, elts_head,\n+\t\t\t\t\t &sges);\n+\t\t\tif (ret.length == (unsigned int)-1)\n+\t\t\t\tgoto stop;\n+\t\t\tRTE_MBUF_PREFETCH_TO_FREE(elt_next->buf);\n+\t\t\t/* Put SG list into send queue. */\n+\t\t\terr = txq->if_qp->send_pending_sg_list\n+\t\t\t\t(txq->qp,\n+\t\t\t\t sges,\n+\t\t\t\t ret.num,\n+\t\t\t\t send_flags);\n+\t\t\tif (unlikely(err))\n+\t\t\t\tgoto stop;\n+#else /* MLX5_PMD_SGE_WR_N > 1 */\n \t\t\tDEBUG(\"%p: TX scattered buffers support not\"\n \t\t\t \" compiled in\", (void *)txq);\n \t\t\tgoto stop;\n+#endif /* MLX5_PMD_SGE_WR_N > 1 */\n \t\t}\n \t\telts_head = elts_head_next;\n \t}\n@@ -307,8 +474,215 @@ stop:\n }\n \n /**\n+ * DPDK callback for RX with scattered packets support.\n+ *\n+ * @param dpdk_rxq\n+ * Generic pointer to RX queue structure.\n+ * @param[out] pkts\n+ * Array to store received packets.\n+ * @param pkts_n\n+ * Maximum number of packets in array.\n+ *\n+ * @return\n+ * Number of packets successfully received (<= pkts_n).\n+ */\n+uint16_t\n+mlx5_rx_burst_sp(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)\n+{\n+\tstruct rxq *rxq = (struct rxq *)dpdk_rxq;\n+\tstruct rxq_elt_sp (*elts)[rxq->elts_n] = rxq->elts.sp;\n+\tconst unsigned int elts_n = rxq->elts_n;\n+\tunsigned int elts_head = rxq->elts_head;\n+\tstruct ibv_recv_wr head;\n+\tstruct ibv_recv_wr **next = &head.next;\n+\tstruct ibv_recv_wr *bad_wr;\n+\tunsigned int i;\n+\tunsigned int pkts_ret = 0;\n+\tint ret;\n+\n+\tif (unlikely(!rxq->sp))\n+\t\treturn mlx5_rx_burst(dpdk_rxq, pkts, pkts_n);\n+\tif (unlikely(elts == NULL)) /* See RTE_DEV_CMD_SET_MTU. */\n+\t\treturn 0;\n+\tfor (i = 0; (i != pkts_n); ++i) {\n+\t\tstruct rxq_elt_sp *elt = &(*elts)[elts_head];\n+\t\tstruct ibv_recv_wr *wr = &elt->wr;\n+\t\tuint64_t wr_id = wr->wr_id;\n+\t\tunsigned int len;\n+\t\tunsigned int pkt_buf_len;\n+\t\tstruct rte_mbuf *pkt_buf = NULL; /* Buffer returned in pkts. */\n+\t\tstruct rte_mbuf **pkt_buf_next = &pkt_buf;\n+\t\tunsigned int seg_headroom = RTE_PKTMBUF_HEADROOM;\n+\t\tunsigned int j = 0;\n+\t\tuint32_t flags;\n+\n+\t\t/* Sanity checks. */\n+#ifdef NDEBUG\n+\t\t(void)wr_id;\n+#endif\n+\t\tassert(wr_id < rxq->elts_n);\n+\t\tassert(wr->sg_list == elt->sges);\n+\t\tassert(wr->num_sge == RTE_DIM(elt->sges));\n+\t\tassert(elts_head < rxq->elts_n);\n+\t\tassert(rxq->elts_head < rxq->elts_n);\n+\t\tret = rxq->if_cq->poll_length_flags(rxq->cq, NULL, NULL,\n+\t\t\t\t\t\t &flags);\n+\t\tif (unlikely(ret < 0)) {\n+\t\t\tstruct ibv_wc wc;\n+\t\t\tint wcs_n;\n+\n+\t\t\tDEBUG(\"rxq=%p, poll_length() failed (ret=%d)\",\n+\t\t\t (void *)rxq, ret);\n+\t\t\t/* ibv_poll_cq() must be used in case of failure. */\n+\t\t\twcs_n = ibv_poll_cq(rxq->cq, 1, &wc);\n+\t\t\tif (unlikely(wcs_n == 0))\n+\t\t\t\tbreak;\n+\t\t\tif (unlikely(wcs_n < 0)) {\n+\t\t\t\tDEBUG(\"rxq=%p, ibv_poll_cq() failed (wcs_n=%d)\",\n+\t\t\t\t (void *)rxq, wcs_n);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tassert(wcs_n == 1);\n+\t\t\tif (unlikely(wc.status != IBV_WC_SUCCESS)) {\n+\t\t\t\t/* Whatever, just repost the offending WR. */\n+\t\t\t\tDEBUG(\"rxq=%p, wr_id=%\" PRIu64 \": bad work\"\n+\t\t\t\t \" completion status (%d): %s\",\n+\t\t\t\t (void *)rxq, wc.wr_id, wc.status,\n+\t\t\t\t ibv_wc_status_str(wc.status));\n+\t\t\t\t/* Link completed WRs together for repost. */\n+\t\t\t\t*next = wr;\n+\t\t\t\tnext = &wr->next;\n+\t\t\t\tgoto repost;\n+\t\t\t}\n+\t\t\tret = wc.byte_len;\n+\t\t}\n+\t\tif (ret == 0)\n+\t\t\tbreak;\n+\t\tlen = ret;\n+\t\tpkt_buf_len = len;\n+\t\t/* Link completed WRs together for repost. */\n+\t\t*next = wr;\n+\t\tnext = &wr->next;\n+\t\t/*\n+\t\t * Replace spent segments with new ones, concatenate and\n+\t\t * return them as pkt_buf.\n+\t\t */\n+\t\twhile (1) {\n+\t\t\tstruct ibv_sge *sge = &elt->sges[j];\n+\t\t\tstruct rte_mbuf *seg = elt->bufs[j];\n+\t\t\tstruct rte_mbuf *rep;\n+\t\t\tunsigned int seg_tailroom;\n+\n+\t\t\t/*\n+\t\t\t * Fetch initial bytes of packet descriptor into a\n+\t\t\t * cacheline while allocating rep.\n+\t\t\t */\n+\t\t\trte_prefetch0(seg);\n+\t\t\trep = __rte_mbuf_raw_alloc(rxq->mp);\n+\t\t\tif (unlikely(rep == NULL)) {\n+\t\t\t\t/*\n+\t\t\t\t * Unable to allocate a replacement mbuf,\n+\t\t\t\t * repost WR.\n+\t\t\t\t */\n+\t\t\t\tDEBUG(\"rxq=%p, wr_id=%\" PRIu64 \":\"\n+\t\t\t\t \" can't allocate a new mbuf\",\n+\t\t\t\t (void *)rxq, wr_id);\n+\t\t\t\tif (pkt_buf != NULL) {\n+\t\t\t\t\t*pkt_buf_next = NULL;\n+\t\t\t\t\trte_pktmbuf_free(pkt_buf);\n+\t\t\t\t}\n+\t\t\t\t/* Increment out of memory counters. */\n+\t\t\t\t++rxq->priv->dev->data->rx_mbuf_alloc_failed;\n+\t\t\t\tgoto repost;\n+\t\t\t}\n+#ifndef NDEBUG\n+\t\t\t/* Poison user-modifiable fields in rep. */\n+\t\t\tNEXT(rep) = (void *)((uintptr_t)-1);\n+\t\t\tSET_DATA_OFF(rep, 0xdead);\n+\t\t\tDATA_LEN(rep) = 0xd00d;\n+\t\t\tPKT_LEN(rep) = 0xdeadd00d;\n+\t\t\tNB_SEGS(rep) = 0x2a;\n+\t\t\tPORT(rep) = 0x2a;\n+\t\t\trep->ol_flags = -1;\n+#endif\n+\t\t\tassert(rep->buf_len == seg->buf_len);\n+\t\t\tassert(rep->buf_len == rxq->mb_len);\n+\t\t\t/* Reconfigure sge to use rep instead of seg. */\n+\t\t\tassert(sge->lkey == rxq->mr->lkey);\n+\t\t\tsge->addr = ((uintptr_t)rep->buf_addr + seg_headroom);\n+\t\t\telt->bufs[j] = rep;\n+\t\t\t++j;\n+\t\t\t/* Update pkt_buf if it's the first segment, or link\n+\t\t\t * seg to the previous one and update pkt_buf_next. */\n+\t\t\t*pkt_buf_next = seg;\n+\t\t\tpkt_buf_next = &NEXT(seg);\n+\t\t\t/* Update seg information. */\n+\t\t\tseg_tailroom = (seg->buf_len - seg_headroom);\n+\t\t\tassert(sge->length == seg_tailroom);\n+\t\t\tSET_DATA_OFF(seg, seg_headroom);\n+\t\t\tif (likely(len <= seg_tailroom)) {\n+\t\t\t\t/* Last segment. */\n+\t\t\t\tDATA_LEN(seg) = len;\n+\t\t\t\tPKT_LEN(seg) = len;\n+\t\t\t\t/* Sanity check. */\n+\t\t\t\tassert(rte_pktmbuf_headroom(seg) ==\n+\t\t\t\t seg_headroom);\n+\t\t\t\tassert(rte_pktmbuf_tailroom(seg) ==\n+\t\t\t\t (seg_tailroom - len));\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tDATA_LEN(seg) = seg_tailroom;\n+\t\t\tPKT_LEN(seg) = seg_tailroom;\n+\t\t\t/* Sanity check. */\n+\t\t\tassert(rte_pktmbuf_headroom(seg) == seg_headroom);\n+\t\t\tassert(rte_pktmbuf_tailroom(seg) == 0);\n+\t\t\t/* Fix len and clear headroom for next segments. */\n+\t\t\tlen -= seg_tailroom;\n+\t\t\tseg_headroom = 0;\n+\t\t}\n+\t\t/* Update head and tail segments. */\n+\t\t*pkt_buf_next = NULL;\n+\t\tassert(pkt_buf != NULL);\n+\t\tassert(j != 0);\n+\t\tNB_SEGS(pkt_buf) = j;\n+\t\tPORT(pkt_buf) = rxq->port_id;\n+\t\tPKT_LEN(pkt_buf) = pkt_buf_len;\n+\n+\t\t/* Return packet. */\n+\t\t*(pkts++) = pkt_buf;\n+\t\t++pkts_ret;\n+repost:\n+\t\tif (++elts_head >= elts_n)\n+\t\t\telts_head = 0;\n+\t\tcontinue;\n+\t}\n+\tif (unlikely(i == 0))\n+\t\treturn 0;\n+\t*next = NULL;\n+\t/* Repost WRs. */\n+#ifdef DEBUG_RECV\n+\tDEBUG(\"%p: reposting %d WRs\", (void *)rxq, i);\n+#endif\n+\tret = ibv_post_recv(rxq->qp, head.next, &bad_wr);\n+\tif (unlikely(ret)) {\n+\t\t/* Inability to repost WRs is fatal. */\n+\t\tDEBUG(\"%p: ibv_post_recv(): failed for WR %p: %s\",\n+\t\t (void *)rxq->priv,\n+\t\t (void *)bad_wr,\n+\t\t strerror(ret));\n+\t\tabort();\n+\t}\n+\trxq->elts_head = elts_head;\n+\treturn pkts_ret;\n+}\n+\n+/**\n * DPDK callback for RX.\n *\n+ * The following function is the same as mlx5_rx_burst_sp(), except it doesn't\n+ * manage scattered packets. Improves performance when MRU is lower than the\n+ * size of the first segment.\n+ *\n * @param dpdk_rxq\n * Generic pointer to RX queue structure.\n * @param[out] pkts\n@@ -331,6 +705,8 @@ mlx5_rx_burst(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n)\n \tunsigned int pkts_ret = 0;\n \tint ret;\n \n+\tif (unlikely(rxq->sp))\n+\t\treturn mlx5_rx_burst_sp(dpdk_rxq, pkts, pkts_n);\n \tfor (i = 0; (i != pkts_n); ++i) {\n \t\tstruct rxq_elt *elt = &(*elts)[elts_head];\n \t\tstruct ibv_recv_wr *wr = &elt->wr;\ndiff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h\nindex 08a4911..44170d7 100644\n--- a/drivers/net/mlx5/mlx5_rxtx.h\n+++ b/drivers/net/mlx5/mlx5_rxtx.h\n@@ -60,6 +60,13 @@\n #include \"mlx5.h\"\n #include \"mlx5_defs.h\"\n \n+/* RX element (scattered packets). */\n+struct rxq_elt_sp {\n+\tstruct ibv_recv_wr wr; /* Work Request. */\n+\tstruct ibv_sge sges[MLX5_PMD_SGE_WR_N]; /* Scatter/Gather Elements. */\n+\tstruct rte_mbuf *bufs[MLX5_PMD_SGE_WR_N]; /* SGEs buffers. */\n+};\n+\n /* RX element. */\n struct rxq_elt {\n \tstruct ibv_recv_wr wr; /* Work Request. */\n@@ -87,8 +94,10 @@ struct rxq {\n \tunsigned int elts_n; /* (*elts)[] length. */\n \tunsigned int elts_head; /* Current index in (*elts)[]. */\n \tunion {\n+\t\tstruct rxq_elt_sp (*sp)[]; /* Scattered RX elements. */\n \t\tstruct rxq_elt (*no_sp)[]; /* RX elements. */\n \t} elts;\n+\tunsigned int sp:1; /* Use scattered RX elements. */\n \tuint32_t mb_len; /* Length of a mp-issued mbuf. */\n \tunsigned int socket; /* CPU socket ID for allocations. */\n \tstruct ibv_exp_res_domain *rd; /* Resource Domain. */\n@@ -154,6 +163,7 @@ void mlx5_tx_queue_release(void *);\n /* mlx5_rxtx.c */\n \n uint16_t mlx5_tx_burst(void *, struct rte_mbuf **, uint16_t);\n+uint16_t mlx5_rx_burst_sp(void *, struct rte_mbuf **, uint16_t);\n uint16_t mlx5_rx_burst(void *, struct rte_mbuf **, uint16_t);\n uint16_t removed_tx_burst(void *, struct rte_mbuf **, uint16_t);\n uint16_t removed_rx_burst(void *, struct rte_mbuf **, uint16_t);\n", "prefixes": [ "dpdk-dev", "05/13" ] }