Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/108631/?format=api
{ "id": 108631, "url": "https://patches.dpdk.org/api/patches/108631/?format=api", "web_url": "https://patches.dpdk.org/project/dpdk/patch/20220310065407.17145-2-xuan.ding@intel.com/", "project": { "id": 1, "url": "https://patches.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20220310065407.17145-2-xuan.ding@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20220310065407.17145-2-xuan.ding@intel.com", "date": "2022-03-10T06:54:06", "name": "[RFC,v3,1/2] vhost: support async dequeue for split ring", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": true, "hash": "fce794541a41516ad31630f4d1d9d382a924fa32", "submitter": { "id": 1401, "url": "https://patches.dpdk.org/api/people/1401/?format=api", "name": "Ding, Xuan", "email": "xuan.ding@intel.com" }, "delegate": { "id": 2642, "url": "https://patches.dpdk.org/api/users/2642/?format=api", "username": "mcoquelin", "first_name": "Maxime", "last_name": "Coquelin", "email": "maxime.coquelin@redhat.com" }, "mbox": "https://patches.dpdk.org/project/dpdk/patch/20220310065407.17145-2-xuan.ding@intel.com/mbox/", "series": [ { "id": 22093, "url": "https://patches.dpdk.org/api/series/22093/?format=api", "web_url": "https://patches.dpdk.org/project/dpdk/list/?series=22093", "date": "2022-03-10T06:54:05", "name": "vhost: support async dequeue data path", "version": 3, "mbox": "https://patches.dpdk.org/series/22093/mbox/" } ], "comments": "https://patches.dpdk.org/api/patches/108631/comments/", "check": "success", "checks": "https://patches.dpdk.org/api/patches/108631/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from mails.dpdk.org (mails.dpdk.org [217.70.189.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 3E4D5A0093;\n\tThu, 10 Mar 2022 07:57:07 +0100 (CET)", "from [217.70.189.124] (localhost [127.0.0.1])\n\tby mails.dpdk.org (Postfix) with ESMTP id 2BDE441144;\n\tThu, 10 Mar 2022 07:57:07 +0100 (CET)", "from mga02.intel.com (mga02.intel.com [134.134.136.20])\n by mails.dpdk.org (Postfix) with ESMTP id 16FB2410DD\n for <dev@dpdk.org>; Thu, 10 Mar 2022 07:57:04 +0100 (CET)", "from orsmga007.jf.intel.com ([10.7.209.58])\n by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;\n 09 Mar 2022 22:57:03 -0800", "from npg-dpdk-xuan-cbdma.sh.intel.com ([10.67.110.228])\n by orsmga007.jf.intel.com with ESMTP; 09 Mar 2022 22:57:00 -0800" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/simple;\n d=intel.com; i=@intel.com; q=dns/txt; s=Intel;\n t=1646895425; x=1678431425;\n h=from:to:cc:subject:date:message-id:in-reply-to: references;\n bh=o+8TwZPuVa2WRY183jwJprxVIkdaRVUhlPvhoya9ZxE=;\n b=VeYwq4vHbkwmzyyxMaj7ChQKKjYi1K8W1IqPtsk50BUl/FXeAQbzKloe\n avzu9aiiRzibaD2nXPePEr1AzX2kGIW3OfDPLWtdnHKLs3dBOPB8j+Xjr\n M4ERCirt1Kvzvt40Dd0GF5oHyoCRPLUu7KFQ7FiZFk+9bvpAsocwRPNwB\n mWvVUG0aX8rVYDU5pwBQzeH8JWbSwxfDrueJwMKMypS5ToafIUuxJ0lTf\n qqc6203YygQspjt88GBVmZnOru/BoJ22i7ceoo3J5QTqsLiBKT2VgxafV\n 3nyAX2/eAH4eXwmxZEMFL4GhyY88VHJ648K/OoEjgNgof+ENkzBYElbC/ Q==;", "X-IronPort-AV": [ "E=McAfee;i=\"6200,9189,10281\"; a=\"242621035\"", "E=Sophos;i=\"5.90,169,1643702400\"; d=\"scan'208\";a=\"242621035\"", "E=Sophos;i=\"5.90,169,1643702400\"; d=\"scan'208\";a=\"538333297\"" ], "X-ExtLoop1": "1", "From": "xuan.ding@intel.com", "To": "maxime.coquelin@redhat.com,\n\tchenbo.xia@intel.com", "Cc": "dev@dpdk.org, jiayu.hu@intel.com, cheng1.jiang@intel.com,\n sunil.pai.g@intel.com, liangma@liangbit.com,\n Xuan Ding <xuan.ding@intel.com>, Yuan Wang <yuanx.wang@intel.com>", "Subject": "[RFC,v3 1/2] vhost: support async dequeue for split ring", "Date": "Thu, 10 Mar 2022 06:54:06 +0000", "Message-Id": "<20220310065407.17145-2-xuan.ding@intel.com>", "X-Mailer": "git-send-email 2.17.1", "In-Reply-To": "<20220310065407.17145-1-xuan.ding@intel.com>", "References": "<20220101001244.90147-1-xuan.ding@intel.com>\n <20220310065407.17145-1-xuan.ding@intel.com>", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org" }, "content": "From: Xuan Ding <xuan.ding@intel.com>\n\nThis patch implements asynchronous dequeue data path for vhost split\nring, with dmadev library integrated.\n\nSigned-off-by: Xuan Ding <xuan.ding@intel.com>\nSigned-off-by: Yuan Wang <yuanx.wang@intel.com>\n---\n lib/vhost/rte_vhost_async.h | 37 ++-\n lib/vhost/version.map | 1 +\n lib/vhost/vhost.h | 1 +\n lib/vhost/virtio_net.c | 504 ++++++++++++++++++++++++++++++++++++\n 4 files changed, 541 insertions(+), 2 deletions(-)", "diff": "diff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h\nindex f1293c6a9d..b6ab0b06a2 100644\n--- a/lib/vhost/rte_vhost_async.h\n+++ b/lib/vhost/rte_vhost_async.h\n@@ -155,9 +155,9 @@ int rte_vhost_async_get_inflight(int vid, uint16_t queue_id);\n * @param count\n * Size of the packet array\n * @param dma_id\n- * the identifier of DMA device\n+ * The identifier of DMA device\n * @param vchan_id\n- * the identifier of virtual DMA channel\n+ * The identifier of virtual DMA channel\n * @return\n * Number of packets returned\n */\n@@ -187,6 +187,39 @@ uint16_t rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,\n __rte_experimental\n int rte_vhost_async_dma_configure(int16_t dma_id, uint16_t vchan_id);\n \n+/**\n+ * This function tries to receive packets from the guest with offloading\n+ * copies to the async channel. The packets that are transfer completed\n+ * are returned in \"pkts\". The other packets that their copies are submitted to\n+ * the async channel but not completed are called \"in-flight packets\".\n+ * This function will not return in-flight packets until their copies are\n+ * completed by the async channel.\n+ *\n+ * @param vid\n+ * ID of vhost device to dequeue data\n+ * @param queue_id\n+ * ID of virtqueue to dequeue data\n+ * @param mbuf_pool\n+ * Mbuf_pool where host mbuf is allocated\n+ * @param pkts\n+ * Blank array to keep successfully dequeued packets\n+ * @param count\n+ * Size of the packet array\n+ * @param nr_inflight\n+ * The amount of in-flight packets. If error occurred, its value is set to -1.\n+ * @param dma_id\n+ * The identifier of DMA device\n+ * @param vchan_id\n+ * The identifier of virtual DMA channel\n+ * @return\n+ * Number of successfully dequeued packets\n+ */\n+__rte_experimental\n+uint16_t\n+rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,\n+\tstruct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,\n+\tint *nr_inflight, uint16_t dma_id, uint16_t vchan_id);\n+\n #ifdef __cplusplus\n }\n #endif\ndiff --git a/lib/vhost/version.map b/lib/vhost/version.map\nindex 0a66c5840c..968d6d4290 100644\n--- a/lib/vhost/version.map\n+++ b/lib/vhost/version.map\n@@ -87,6 +87,7 @@ EXPERIMENTAL {\n \n \t# added in 22.03\n \trte_vhost_async_dma_configure;\n+\trte_vhost_async_try_dequeue_burst;\n };\n \n INTERNAL {\ndiff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h\nindex a9edc271aa..3799d41089 100644\n--- a/lib/vhost/vhost.h\n+++ b/lib/vhost/vhost.h\n@@ -178,6 +178,7 @@ extern struct async_dma_info dma_copy_track[RTE_DMADEV_DEFAULT_MAX];\n */\n struct async_inflight_info {\n \tstruct rte_mbuf *mbuf;\n+\tstruct virtio_net_hdr nethdr;\n \tuint16_t descs; /* num of descs inflight */\n \tuint16_t nr_buffers; /* num of buffers inflight for packed ring */\n };\ndiff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c\nindex 5f432b0d77..3816caca79 100644\n--- a/lib/vhost/virtio_net.c\n+++ b/lib/vhost/virtio_net.c\n@@ -3141,3 +3141,507 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,\n \n \treturn count;\n }\n+\n+static __rte_always_inline int\n+async_desc_to_mbuf_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,\n+\t\t\tstruct rte_mbuf *m, uint32_t mbuf_offset,\n+\t\t\tuint64_t buf_iova, uint32_t cpy_len)\n+{\n+\tstruct vhost_async *async = vq->async;\n+\tuint64_t mapped_len;\n+\tuint32_t buf_offset = 0;\n+\tvoid *host_iova;\n+\n+\twhile (cpy_len) {\n+\t\thost_iova = (void *)(uintptr_t)gpa_to_first_hpa(dev,\n+\t\t\t\t\tbuf_iova + buf_offset, cpy_len,\n+\t\t\t\t\t&mapped_len);\n+\t\tif (unlikely(!host_iova)) {\n+\t\t\tVHOST_LOG_DATA(ERR, \"(%s) %s: failed to get host_iova.\\n\",\n+\t\t\t\tdev->ifname, __func__);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (unlikely(async_iter_add_iovec(dev, async, host_iova,\n+\t\t\t\t(void *)(uintptr_t)rte_pktmbuf_iova_offset(m, mbuf_offset),\n+\t\t\t\t(size_t)mapped_len)))\n+\t\t\treturn -1;\n+\n+\t\tcpy_len -= (uint32_t)mapped_len;\n+\t\tmbuf_offset += (uint32_t)mapped_len;\n+\t\tbuf_offset += (uint32_t)mapped_len;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static __rte_always_inline int\n+async_desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,\n+\t\t struct buf_vector *buf_vec, uint16_t nr_vec,\n+\t\t struct rte_mbuf *m, struct rte_mempool *mbuf_pool,\n+\t\t struct virtio_net_hdr *nethdr)\n+{\n+\tuint64_t buf_addr, buf_iova;\n+\tuint32_t buf_avail, buf_offset, buf_len;\n+\tuint32_t mbuf_avail, mbuf_offset;\n+\tuint32_t cpy_len;\n+\t/* A counter to avoid desc dead loop chain */\n+\tuint16_t vec_idx = 0;\n+\tstruct rte_mbuf *cur = m, *prev = m;\n+\tstruct virtio_net_hdr tmp_hdr;\n+\tstruct virtio_net_hdr *hdr = NULL;\n+\tstruct vhost_async *async = vq->async;\n+\n+\tbuf_addr = buf_vec[vec_idx].buf_addr;\n+\tbuf_len = buf_vec[vec_idx].buf_len;\n+\tbuf_iova = buf_vec[vec_idx].buf_iova;\n+\n+\tif (unlikely(buf_len < dev->vhost_hlen && nr_vec <= 1))\n+\t\treturn -1;\n+\n+\tif (virtio_net_with_host_offload(dev)) {\n+\t\tif (unlikely(buf_len < sizeof(struct virtio_net_hdr))) {\n+\t\t\t/*\n+\t\t\t * No luck, the virtio-net header doesn't fit\n+\t\t\t * in a contiguous virtual area.\n+\t\t\t */\n+\t\t\tcopy_vnet_hdr_from_desc(&tmp_hdr, buf_vec);\n+\t\t\thdr = &tmp_hdr;\n+\t\t} else {\n+\t\t\thdr = (struct virtio_net_hdr *)((uintptr_t)buf_addr);\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * A virtio driver normally uses at least 2 desc buffers\n+\t * for Tx: the first for storing the header, and others\n+\t * for storing the data.\n+\t */\n+\tif (unlikely(buf_len < dev->vhost_hlen)) {\n+\t\tbuf_offset = dev->vhost_hlen - buf_len;\n+\t\tvec_idx++;\n+\t\tbuf_addr = buf_vec[vec_idx].buf_addr;\n+\t\tbuf_iova = buf_vec[vec_idx].buf_iova;\n+\t\tbuf_len = buf_vec[vec_idx].buf_len;\n+\t\tbuf_avail = buf_len - buf_offset;\n+\t} else if (buf_len == dev->vhost_hlen) {\n+\t\tif (unlikely(++vec_idx >= nr_vec))\n+\t\t\treturn -1;\n+\t\tbuf_addr = buf_vec[vec_idx].buf_addr;\n+\t\tbuf_iova = buf_vec[vec_idx].buf_iova;\n+\t\tbuf_len = buf_vec[vec_idx].buf_len;\n+\n+\t\tbuf_offset = 0;\n+\t\tbuf_avail = buf_len;\n+\t} else {\n+\t\tbuf_offset = dev->vhost_hlen;\n+\t\tbuf_avail = buf_vec[vec_idx].buf_len - dev->vhost_hlen;\n+\t}\n+\n+\tPRINT_PACKET(dev, (uintptr_t)(buf_addr + buf_offset), (uint32_t)buf_avail, 0);\n+\n+\tmbuf_offset = 0;\n+\tmbuf_avail = m->buf_len - RTE_PKTMBUF_HEADROOM;\n+\n+\tif (async_iter_initialize(dev, async))\n+\t\treturn -1;\n+\n+\twhile (1) {\n+\t\tcpy_len = RTE_MIN(buf_avail, mbuf_avail);\n+\n+\t\tif (async_desc_to_mbuf_seg(dev, vq, cur, mbuf_offset, buf_iova + buf_offset,\n+\t\t\t\t\t cpy_len) < 0)\n+\t\t\tgoto error;\n+\n+\t\tmbuf_avail -= cpy_len;\n+\t\tbuf_avail -= cpy_len;\n+\t\tmbuf_offset += cpy_len;\n+\t\tbuf_offset += cpy_len;\n+\n+\t\t/* This buf reaches to its end, get the next one */\n+\t\tif (buf_avail == 0) {\n+\t\t\tif (++vec_idx >= nr_vec)\n+\t\t\t\tbreak;\n+\n+\t\t\tbuf_addr = buf_vec[vec_idx].buf_addr;\n+\t\t\tbuf_iova = buf_vec[vec_idx].buf_iova;\n+\t\t\tbuf_len = buf_vec[vec_idx].buf_len;\n+\n+\t\t\tbuf_offset = 0;\n+\t\t\tbuf_avail = buf_len;\n+\n+\t\t\tPRINT_PACKET(dev, (uintptr_t)buf_addr, (uint32_t)buf_avail, 0);\n+\t\t}\n+\n+\t\t/*\n+\t\t * This mbuf reaches to its end, get a new one\n+\t\t * to hold more data.\n+\t\t */\n+\t\tif (mbuf_avail == 0) {\n+\t\t\tcur = rte_pktmbuf_alloc(mbuf_pool);\n+\t\t\tif (unlikely(cur == NULL)) {\n+\t\t\t\tVHOST_LOG_DATA(ERR,\n+\t\t\t\t\t\"(%s) %s: failed to allocate memory for mbuf.\\n\",\n+\t\t\t\t\tdev->ifname, __func__);\n+\t\t\t\tgoto error;\n+\t\t\t}\n+\n+\t\t\tprev->next = cur;\n+\t\t\tprev->data_len = mbuf_offset;\n+\t\t\tm->nb_segs += 1;\n+\t\t\tm->pkt_len += mbuf_offset;\n+\t\t\tprev = cur;\n+\n+\t\t\tmbuf_offset = 0;\n+\t\t\tmbuf_avail = cur->buf_len - RTE_PKTMBUF_HEADROOM;\n+\t\t}\n+\t}\n+\n+\tprev->data_len = mbuf_offset;\n+\tm->pkt_len += mbuf_offset;\n+\n+\tasync_iter_finalize(async);\n+\tif (hdr)\n+\t\t*nethdr = *hdr;\n+\n+\treturn 0;\n+\n+error:\n+\tasync_iter_cancel(async);\n+\treturn -1;\n+}\n+\n+static __rte_always_inline uint16_t\n+async_poll_dequeue_completed_split(struct virtio_net *dev, uint16_t queue_id,\n+\t\tstruct rte_mbuf **pkts, uint16_t count, uint16_t dma_id,\n+\t\tuint16_t vchan_id, bool legacy_ol_flags)\n+{\n+\tuint16_t start_idx, from, i;\n+\tuint16_t nr_cpl_pkts = 0;\n+\tstruct async_inflight_info *pkts_info;\n+\tstruct vhost_virtqueue *vq = dev->virtqueue[queue_id];\n+\n+\tpkts_info = vq->async->pkts_info;\n+\n+\tvhost_async_dma_check_completed(dev, dma_id, vchan_id, VHOST_DMA_MAX_COPY_COMPLETE);\n+\n+\tstart_idx = async_get_first_inflight_pkt_idx(vq);\n+\n+\tfrom = start_idx;\n+\twhile (vq->async->pkts_cmpl_flag[from] && count--) {\n+\t\tvq->async->pkts_cmpl_flag[from] = false;\n+\t\tfrom = (from + 1) & (vq->size - 1);\n+\t\tnr_cpl_pkts++;\n+\t}\n+\n+\tif (nr_cpl_pkts == 0)\n+\t\treturn 0;\n+\n+\tfor (i = 0; i < nr_cpl_pkts; i++) {\n+\t\tfrom = (start_idx + i) & (vq->size - 1);\n+\t\tpkts[i] = pkts_info[from].mbuf;\n+\n+\t\tif (virtio_net_with_host_offload(dev))\n+\t\t\tvhost_dequeue_offload(dev, &pkts_info[from].nethdr, pkts[i],\n+\t\t\t\t\t legacy_ol_flags);\n+\t}\n+\n+\t/* write back completed descs to used ring and update used idx */\n+\twrite_back_completed_descs_split(vq, nr_cpl_pkts);\n+\t__atomic_add_fetch(&vq->used->idx, nr_cpl_pkts, __ATOMIC_RELEASE);\n+\tvhost_vring_call_split(dev, vq);\n+\n+\tvq->async->pkts_inflight_n -= nr_cpl_pkts;\n+\n+\treturn nr_cpl_pkts;\n+}\n+\n+static __rte_always_inline uint16_t\n+virtio_dev_tx_async_split(struct virtio_net *dev, struct vhost_virtqueue *vq,\n+\t\tuint16_t queue_id, struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,\n+\t\tuint16_t count, uint16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)\n+{\n+\tstatic bool allocerr_warned;\n+\tbool dropped = false;\n+\tuint16_t free_entries;\n+\tuint16_t pkt_idx, slot_idx = 0;\n+\tuint16_t nr_done_pkts = 0;\n+\tuint16_t pkt_err = 0;\n+\tuint16_t n_xfer;\n+\tstruct vhost_async *async = vq->async;\n+\tstruct async_inflight_info *pkts_info = async->pkts_info;\n+\tstruct rte_mbuf *pkts_prealloc[MAX_PKT_BURST];\n+\tuint16_t pkts_size = count;\n+\n+\t/**\n+\t * The ordering between avail index and\n+\t * desc reads needs to be enforced.\n+\t */\n+\tfree_entries = __atomic_load_n(&vq->avail->idx, __ATOMIC_ACQUIRE) - vq->last_avail_idx;\n+\tif (free_entries == 0)\n+\t\tgoto out;\n+\n+\trte_prefetch0(&vq->avail->ring[vq->last_avail_idx & (vq->size - 1)]);\n+\n+\tasync_iter_reset(async);\n+\n+\tcount = RTE_MIN(count, MAX_PKT_BURST);\n+\tcount = RTE_MIN(count, free_entries);\n+\tVHOST_LOG_DATA(DEBUG, \"(%s) about to dequeue %u buffers\\n\", dev->ifname, count);\n+\n+\tif (rte_pktmbuf_alloc_bulk(mbuf_pool, pkts_prealloc, count))\n+\t\tgoto out;\n+\n+\tfor (pkt_idx = 0; pkt_idx < count; pkt_idx++) {\n+\t\tuint16_t head_idx = 0;\n+\t\tuint16_t nr_vec = 0;\n+\t\tuint16_t to;\n+\t\tuint32_t buf_len;\n+\t\tint err;\n+\t\tstruct buf_vector buf_vec[BUF_VECTOR_MAX];\n+\t\tstruct rte_mbuf *pkt = pkts_prealloc[pkt_idx];\n+\n+\t\tif (unlikely(fill_vec_buf_split(dev, vq, vq->last_avail_idx,\n+\t\t\t\t\t\t&nr_vec, buf_vec,\n+\t\t\t\t\t\t&head_idx, &buf_len,\n+\t\t\t\t\t\tVHOST_ACCESS_RO) < 0)) {\n+\t\t\tdropped = true;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\terr = virtio_dev_pktmbuf_prep(dev, pkt, buf_len);\n+\t\tif (unlikely(err)) {\n+\t\t\t/**\n+\t\t\t * mbuf allocation fails for jumbo packets when external\n+\t\t\t * buffer allocation is not allowed and linear buffer\n+\t\t\t * is required. Drop this packet.\n+\t\t\t */\n+\t\t\tif (!allocerr_warned) {\n+\t\t\t\tVHOST_LOG_DATA(ERR,\n+\t\t\t\t\t\"(%s) %s: Failed mbuf alloc of size %d from %s\\n\",\n+\t\t\t\t\tdev->ifname, __func__, buf_len, mbuf_pool->name);\n+\t\t\t\tallocerr_warned = true;\n+\t\t\t}\n+\t\t\tdropped = true;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tslot_idx = (async->pkts_idx + pkt_idx) & (vq->size - 1);\n+\t\terr = async_desc_to_mbuf(dev, vq, buf_vec, nr_vec, pkt, mbuf_pool,\n+\t\t\t\t\t&pkts_info[slot_idx].nethdr);\n+\t\tif (unlikely(err)) {\n+\t\t\tif (!allocerr_warned) {\n+\t\t\t\tVHOST_LOG_DATA(ERR,\n+\t\t\t\t\t\"(%s) %s: Failed to offload copies to async channel.\\n\",\n+\t\t\t\t\tdev->ifname, __func__);\n+\t\t\t\tallocerr_warned = true;\n+\t\t\t}\n+\t\t\tdropped = true;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tpkts_info[slot_idx].mbuf = pkt;\n+\n+\t\t/* store used descs */\n+\t\tto = async->desc_idx_split & (vq->size - 1);\n+\t\tasync->descs_split[to].id = head_idx;\n+\t\tasync->descs_split[to].len = 0;\n+\t\tasync->desc_idx_split++;\n+\n+\t\tvq->last_avail_idx++;\n+\t}\n+\n+\tif (unlikely(dropped))\n+\t\trte_pktmbuf_free_bulk(&pkts_prealloc[pkt_idx], count - pkt_idx);\n+\n+\tn_xfer = vhost_async_dma_transfer(dev, vq, dma_id, vchan_id, async->pkts_idx,\n+\t\t\t\t\t async->iov_iter, pkt_idx);\n+\n+\tasync->pkts_inflight_n += n_xfer;\n+\n+\tpkt_err = pkt_idx - n_xfer;\n+\tif (unlikely(pkt_err)) {\n+\t\tVHOST_LOG_DATA(DEBUG,\n+\t\t\t\"(%s) %s: failed to transfer data for queue id %d.\\n\",\n+\t\t\tdev->ifname, __func__, queue_id);\n+\n+\t\tpkt_idx = n_xfer;\n+\t\t/* recover available ring */\n+\t\tvq->last_avail_idx -= pkt_err;\n+\n+\t\t/**\n+\t\t * recover async channel copy related structures and free pktmbufs\n+\t\t * for error pkts.\n+\t\t */\n+\t\tasync->desc_idx_split -= pkt_err;\n+\t\twhile (pkt_err-- > 0) {\n+\t\t\trte_pktmbuf_free(pkts_info[slot_idx & (vq->size - 1)].mbuf);\n+\t\t\tslot_idx--;\n+\t\t}\n+\t}\n+\n+\tasync->pkts_idx += pkt_idx;\n+\tif (async->pkts_idx >= vq->size)\n+\t\tasync->pkts_idx -= vq->size;\n+\n+out:\n+\t/* DMA device may serve other queues, unconditionally check completed. */\n+\tnr_done_pkts = async_poll_dequeue_completed_split(dev, queue_id, pkts, pkts_size,\n+\t\t\t\t\t\t\t dma_id, vchan_id, legacy_ol_flags);\n+\n+\treturn nr_done_pkts;\n+}\n+\n+__rte_noinline\n+static uint16_t\n+virtio_dev_tx_async_split_legacy(struct virtio_net *dev,\n+\t\tstruct vhost_virtqueue *vq, uint16_t queue_id,\n+\t\tstruct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,\n+\t\tuint16_t count, uint16_t dma_id, uint16_t vchan_id)\n+{\n+\treturn virtio_dev_tx_async_split(dev, vq, queue_id, mbuf_pool,\n+\t\t\t\tpkts, count, dma_id, vchan_id, true);\n+}\n+\n+__rte_noinline\n+static uint16_t\n+virtio_dev_tx_async_split_compliant(struct virtio_net *dev,\n+\t\tstruct vhost_virtqueue *vq, uint16_t queue_id,\n+\t\tstruct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,\n+\t\tuint16_t count, uint16_t dma_id, uint16_t vchan_id)\n+{\n+\treturn virtio_dev_tx_async_split(dev, vq, queue_id, mbuf_pool,\n+\t\t\t\tpkts, count, dma_id, vchan_id, false);\n+}\n+\n+uint16_t\n+rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,\n+\tstruct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,\n+\tint *nr_inflight, uint16_t dma_id, uint16_t vchan_id)\n+{\n+\tstruct virtio_net *dev;\n+\tstruct rte_mbuf *rarp_mbuf = NULL;\n+\tstruct vhost_virtqueue *vq;\n+\tint16_t success = 1;\n+\n+\t*nr_inflight = -1;\n+\n+\tdev = get_device(vid);\n+\tif (!dev)\n+\t\treturn 0;\n+\n+\tif (unlikely(!(dev->flags & VIRTIO_DEV_BUILTIN_VIRTIO_NET))) {\n+\t\tVHOST_LOG_DATA(ERR,\n+\t\t\t\"(%s) %s: built-in vhost net backend is disabled.\\n\",\n+\t\t\tdev->ifname, __func__);\n+\t\treturn 0;\n+\t}\n+\n+\tif (unlikely(!is_valid_virt_queue_idx(queue_id, 1, dev->nr_vring))) {\n+\t\tVHOST_LOG_DATA(ERR,\n+\t\t\t\"(%s) %s: invalid virtqueue idx %d.\\n\",\n+\t\t\tdev->ifname, __func__, queue_id);\n+\t\treturn 0;\n+\t}\n+\n+\tif (unlikely(!dma_copy_track[dma_id].vchans ||\n+\t\t\t\t!dma_copy_track[dma_id].vchans[vchan_id].pkts_cmpl_flag_addr)) {\n+\t\tVHOST_LOG_DATA(ERR, \"(%s) %s: invalid channel %d:%u.\\n\", dev->ifname, __func__,\n+\t\t\t dma_id, vchan_id);\n+\t\treturn 0;\n+\t}\n+\n+\tvq = dev->virtqueue[queue_id];\n+\n+\tif (unlikely(rte_spinlock_trylock(&vq->access_lock) == 0))\n+\t\treturn 0;\n+\n+\tif (unlikely(vq->enabled == 0)) {\n+\t\tcount = 0;\n+\t\tgoto out_access_unlock;\n+\t}\n+\n+\tif (unlikely(!vq->async)) {\n+\t\tVHOST_LOG_DATA(ERR, \"(%s) %s: async not registered for queue id %d.\\n\",\n+\t\t\tdev->ifname, __func__, queue_id);\n+\t\tcount = 0;\n+\t\tgoto out_access_unlock;\n+\t}\n+\n+\tif (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))\n+\t\tvhost_user_iotlb_rd_lock(vq);\n+\n+\tif (unlikely(vq->access_ok == 0))\n+\t\tif (unlikely(vring_translate(dev, vq) < 0)) {\n+\t\t\tcount = 0;\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t/*\n+\t * Construct a RARP broadcast packet, and inject it to the \"pkts\"\n+\t * array, to looks like that guest actually send such packet.\n+\t *\n+\t * Check user_send_rarp() for more information.\n+\t *\n+\t * broadcast_rarp shares a cacheline in the virtio_net structure\n+\t * with some fields that are accessed during enqueue and\n+\t * __atomic_compare_exchange_n causes a write if performed compare\n+\t * and exchange. This could result in false sharing between enqueue\n+\t * and dequeue.\n+\t *\n+\t * Prevent unnecessary false sharing by reading broadcast_rarp first\n+\t * and only performing compare and exchange if the read indicates it\n+\t * is likely to be set.\n+\t */\n+\tif (unlikely(__atomic_load_n(&dev->broadcast_rarp, __ATOMIC_ACQUIRE) &&\n+\t\t\t__atomic_compare_exchange_n(&dev->broadcast_rarp,\n+\t\t\t&success, 0, 0, __ATOMIC_RELEASE, __ATOMIC_RELAXED))) {\n+\n+\t\trarp_mbuf = rte_net_make_rarp_packet(mbuf_pool, &dev->mac);\n+\t\tif (rarp_mbuf == NULL) {\n+\t\t\tVHOST_LOG_DATA(ERR, \"Failed to make RARP packet.\\n\");\n+\t\t\tcount = 0;\n+\t\t\tgoto out;\n+\t\t}\n+\t\t/*\n+\t\t * Inject it to the head of \"pkts\" array, so that switch's mac\n+\t\t * learning table will get updated first.\n+\t\t */\n+\t\tpkts[0] = rarp_mbuf;\n+\t\tpkts++;\n+\t\tcount -= 1;\n+\t}\n+\n+\tif (unlikely(vq_is_packed(dev))) {\n+\t\tstatic bool not_support_pack_log;\n+\t\tif (!not_support_pack_log) {\n+\t\t\tVHOST_LOG_DATA(ERR,\n+\t\t\t\t\"(%s) %s: async dequeue does not support packed ring.\\n\",\n+\t\t\t\tdev->ifname, __func__);\n+\t\t\tnot_support_pack_log = true;\n+\t\t}\n+\t\tcount = 0;\n+\t\tgoto out;\n+\t}\n+\n+\tif (dev->flags & VIRTIO_DEV_LEGACY_OL_FLAGS)\n+\t\tcount = virtio_dev_tx_async_split_legacy(dev, vq, queue_id,\n+\t\t\t\tmbuf_pool, pkts, count, dma_id, vchan_id);\n+\telse\n+\t\tcount = virtio_dev_tx_async_split_compliant(dev, vq, queue_id,\n+\t\t\t\tmbuf_pool, pkts, count, dma_id, vchan_id);\n+\n+\t*nr_inflight = vq->async->pkts_inflight_n;\n+\n+out:\n+\tif (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))\n+\t\tvhost_user_iotlb_rd_unlock(vq);\n+\n+out_access_unlock:\n+\trte_spinlock_unlock(&vq->access_lock);\n+\n+\tif (unlikely(rarp_mbuf != NULL))\n+\t\tcount += 1;\n+\n+\treturn count;\n+}\n", "prefixes": [ "RFC", "v3", "1/2" ] }