From patchwork Tue Jul 7 05:07:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Fu X-Patchwork-Id: 73360 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4609AA00BE; Tue, 7 Jul 2020 07:07:16 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6190D1DA3D; Tue, 7 Jul 2020 07:07:06 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by dpdk.org (Postfix) with ESMTP id B30F51D9B9 for ; Tue, 7 Jul 2020 07:07:04 +0200 (CEST) IronPort-SDR: 05YrGrvlQ7kjhXcLK3tmE1OOu9QECi4LbkgSdrYVePaE/JZn6JkAQDGVvCV0Eud+UOaxGBNLkX v4YP9GYh4h2A== X-IronPort-AV: E=McAfee;i="6000,8403,9674"; a="135005151" X-IronPort-AV: E=Sophos;i="5.75,321,1589266800"; d="scan'208";a="135005151" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Jul 2020 22:07:04 -0700 IronPort-SDR: a9pntyeCJWStM606ClgSBF3dHGGjdioIS5KPCUxNT1FHtUiyYjyow9LJiScrMGKsvmlZvSFM4b Zzow3nfzUrcg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,321,1589266800"; d="scan'208";a="427346657" Received: from npg-dpdk-patrickfu-casc2.sh.intel.com ([10.67.119.92]) by orsmga004.jf.intel.com with ESMTP; 06 Jul 2020 22:07:01 -0700 From: patrick.fu@intel.com To: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, zhihong.wang@intel.com Cc: patrick.fu@intel.com, yinan.wang@intel.com, cheng1.jiang@intel.com, cunming.liang@intel.com Date: Tue, 7 Jul 2020 13:07:08 +0800 Message-Id: <20200707050709.205480-2-patrick.fu@intel.com> X-Mailer: git-send-email 2.18.4 In-Reply-To: <20200707050709.205480-1-patrick.fu@intel.com> References: <1591869725-13331-1-git-send-email-patrick.fu@intel.com> <20200707050709.205480-1-patrick.fu@intel.com> Subject: [dpdk-dev] [PATCH v6 1/2] vhost: introduce async enqueue registration API X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Patrick Fu Performing large memory copies usually takes up a major part of CPU cycles and becomes the hot spot in vhost-user enqueue operation. To offload the large copies from CPU to the DMA devices, asynchronous APIs are introduced, with which the CPU just submits copy jobs to the DMA but without waiting for its copy completion. Thus, there is no CPU intervention during data transfer. We can save precious CPU cycles and improve the overall throughput for vhost-user based applications. This patch introduces registration/un-registration APIs for vhost async data enqueue operation. Together with the registration APIs implementations, data structures and the prototype of the async callback functions required for async enqueue data path are also defined. Signed-off-by: Patrick Fu Reviewed-by: Chenbo Xia --- lib/librte_vhost/Makefile | 2 +- lib/librte_vhost/meson.build | 2 +- lib/librte_vhost/rte_vhost.h | 1 + lib/librte_vhost/rte_vhost_async.h | 136 +++++++++++++++++++++++++ lib/librte_vhost/rte_vhost_version.map | 4 + lib/librte_vhost/socket.c | 27 +++++ lib/librte_vhost/vhost.c | 127 ++++++++++++++++++++++- lib/librte_vhost/vhost.h | 30 +++++- lib/librte_vhost/vhost_user.c | 23 ++++- 9 files changed, 345 insertions(+), 7 deletions(-) create mode 100644 lib/librte_vhost/rte_vhost_async.h diff --git a/lib/librte_vhost/Makefile b/lib/librte_vhost/Makefile index b7ff7dc4b..4f2f3e47d 100644 --- a/lib/librte_vhost/Makefile +++ b/lib/librte_vhost/Makefile @@ -42,7 +42,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_VHOST) := fd_man.c iotlb.c socket.c vhost.c \ # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_VHOST)-include += rte_vhost.h rte_vdpa.h \ - rte_vdpa_dev.h + rte_vdpa_dev.h rte_vhost_async.h # only compile vhost crypto when cryptodev is enabled ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y) diff --git a/lib/librte_vhost/meson.build b/lib/librte_vhost/meson.build index 882a0eaf4..cc9aa65c6 100644 --- a/lib/librte_vhost/meson.build +++ b/lib/librte_vhost/meson.build @@ -22,5 +22,5 @@ sources = files('fd_man.c', 'iotlb.c', 'socket.c', 'vdpa.c', 'vhost.c', 'vhost_user.c', 'virtio_net.c', 'vhost_crypto.c') headers = files('rte_vhost.h', 'rte_vdpa.h', 'rte_vdpa_dev.h', - 'rte_vhost_crypto.h') + 'rte_vhost_crypto.h', 'rte_vhost_async.h') deps += ['ethdev', 'cryptodev', 'hash', 'pci'] diff --git a/lib/librte_vhost/rte_vhost.h b/lib/librte_vhost/rte_vhost.h index 8a5c332c8..f93f9595a 100644 --- a/lib/librte_vhost/rte_vhost.h +++ b/lib/librte_vhost/rte_vhost.h @@ -35,6 +35,7 @@ extern "C" { #define RTE_VHOST_USER_EXTBUF_SUPPORT (1ULL << 5) /* support only linear buffers (no chained mbufs) */ #define RTE_VHOST_USER_LINEARBUF_SUPPORT (1ULL << 6) +#define RTE_VHOST_USER_ASYNC_COPY (1ULL << 7) /* Features. */ #ifndef VIRTIO_NET_F_GUEST_ANNOUNCE diff --git a/lib/librte_vhost/rte_vhost_async.h b/lib/librte_vhost/rte_vhost_async.h new file mode 100644 index 000000000..d5a59279a --- /dev/null +++ b/lib/librte_vhost/rte_vhost_async.h @@ -0,0 +1,136 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2020 Intel Corporation + */ + +#ifndef _RTE_VHOST_ASYNC_H_ +#define _RTE_VHOST_ASYNC_H_ + +#include "rte_vhost.h" + +/** + * iovec iterator + */ +struct rte_vhost_iov_iter { + /** offset to the first byte of interesting data */ + size_t offset; + /** total bytes of data in this iterator */ + size_t count; + /** pointer to the iovec array */ + struct iovec *iov; + /** number of iovec in this iterator */ + unsigned long nr_segs; +}; + +/** + * dma transfer descriptor pair + */ +struct rte_vhost_async_desc { + /** source memory iov_iter */ + struct rte_vhost_iov_iter *src; + /** destination memory iov_iter */ + struct rte_vhost_iov_iter *dst; +}; + +/** + * dma transfer status + */ +struct rte_vhost_async_status { + /** An array of application specific data for source memory */ + uintptr_t *src_opaque_data; + /** An array of application specific data for destination memory */ + uintptr_t *dst_opaque_data; +}; + +/** + * dma operation callbacks to be implemented by applications + */ +struct rte_vhost_async_channel_ops { + /** + * instruct async engines to perform copies for a batch of packets + * + * @param vid + * id of vhost device to perform data copies + * @param queue_id + * queue id to perform data copies + * @param descs + * an array of DMA transfer memory descriptors + * @param opaque_data + * opaque data pair sending to DMA engine + * @param count + * number of elements in the "descs" array + * @return + * -1 on failure, number of descs processed on success + */ + int (*transfer_data)(int vid, uint16_t queue_id, + struct rte_vhost_async_desc *descs, + struct rte_vhost_async_status *opaque_data, + uint16_t count); + /** + * check copy-completed packets from the async engine + * @param vid + * id of vhost device to check copy completion + * @param queue_id + * queue id to check copyp completion + * @param opaque_data + * buffer to receive the opaque data pair from DMA engine + * @param max_packets + * max number of packets could be completed + * @return + * -1 on failure, number of iov segments completed on success + */ + int (*check_completed_copies)(int vid, uint16_t queue_id, + struct rte_vhost_async_status *opaque_data, + uint16_t max_packets); +}; + +/** + * dma channel feature bit definition + */ +struct rte_vhost_async_features { + union { + uint32_t intval; + struct { + uint32_t async_inorder:1; + uint32_t resvd_0:15; + uint32_t async_threshold:12; + uint32_t resvd_1:4; + }; + }; +}; + +/** + * register a async channel for vhost + * + * @param vid + * vhost device id async channel to be attached to + * @param queue_id + * vhost queue id async channel to be attached to + * @param features + * DMA channel feature bit + * b0 : DMA supports inorder data transfer + * b1 - b15: reserved + * b16 - b27: Packet length threshold for DMA transfer + * b28 - b31: reserved + * @param ops + * DMA operation callbacks + * @return + * 0 on success, -1 on failures + */ +__rte_experimental +int rte_vhost_async_channel_register(int vid, uint16_t queue_id, + uint32_t features, struct rte_vhost_async_channel_ops *ops); + +/** + * unregister a dma channel for vhost + * + * @param vid + * vhost device id DMA channel to be detached + * @param queue_id + * vhost queue id DMA channel to be detached + * @return + * 0 on success, -1 on failures + */ +__rte_experimental +int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id); + +#endif /* _RTE_VHOST_ASYNC_H_ */ diff --git a/lib/librte_vhost/rte_vhost_version.map b/lib/librte_vhost/rte_vhost_version.map index 86784405a..13ec53b63 100644 --- a/lib/librte_vhost/rte_vhost_version.map +++ b/lib/librte_vhost/rte_vhost_version.map @@ -71,4 +71,8 @@ EXPERIMENTAL { rte_vdpa_get_queue_num; rte_vdpa_get_features; rte_vdpa_get_protocol_features; + rte_vhost_async_channel_register; + rte_vhost_async_channel_unregister; + rte_vhost_submit_enqueue_burst; + rte_vhost_poll_enqueue_completed; }; diff --git a/lib/librte_vhost/socket.c b/lib/librte_vhost/socket.c index 49267cebf..c4626d2c4 100644 --- a/lib/librte_vhost/socket.c +++ b/lib/librte_vhost/socket.c @@ -42,6 +42,7 @@ struct vhost_user_socket { bool use_builtin_virtio_net; bool extbuf; bool linearbuf; + bool async_copy; /* * The "supported_features" indicates the feature bits the @@ -205,6 +206,7 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket) size_t size; struct vhost_user_connection *conn; int ret; + struct virtio_net *dev; if (vsocket == NULL) return; @@ -236,6 +238,13 @@ vhost_user_add_connection(int fd, struct vhost_user_socket *vsocket) if (vsocket->linearbuf) vhost_enable_linearbuf(vid); + if (vsocket->async_copy) { + dev = get_device(vid); + + if (dev) + dev->async_copy = 1; + } + VHOST_LOG_CONFIG(INFO, "new device, handle is %d\n", vid); if (vsocket->notify_ops->new_connection) { @@ -881,6 +890,17 @@ rte_vhost_driver_register(const char *path, uint64_t flags) goto out_mutex; } + vsocket->async_copy = flags & RTE_VHOST_USER_ASYNC_COPY; + + if (vsocket->async_copy && + (flags & (RTE_VHOST_USER_IOMMU_SUPPORT | + RTE_VHOST_USER_POSTCOPY_SUPPORT))) { + VHOST_LOG_CONFIG(ERR, "error: enabling async copy and IOMMU " + "or post-copy feature simultaneously is not " + "supported\n"); + goto out_mutex; + } + /* * Set the supported features correctly for the builtin vhost-user * net driver. @@ -931,6 +951,13 @@ rte_vhost_driver_register(const char *path, uint64_t flags) ~(1ULL << VHOST_USER_PROTOCOL_F_PAGEFAULT); } + if (vsocket->async_copy) { + vsocket->supported_features &= ~(1ULL << VHOST_F_LOG_ALL); + vsocket->features &= ~(1ULL << VHOST_F_LOG_ALL); + VHOST_LOG_CONFIG(INFO, + "Logging feature is disabled in async copy mode\n"); + } + /* * We'll not be able to receive a buffer from guest in linear mode * without external buffer if it will not fit in a single mbuf, which is diff --git a/lib/librte_vhost/vhost.c b/lib/librte_vhost/vhost.c index 0d822d6a3..a11385f39 100644 --- a/lib/librte_vhost/vhost.c +++ b/lib/librte_vhost/vhost.c @@ -332,8 +332,13 @@ free_vq(struct virtio_net *dev, struct vhost_virtqueue *vq) { if (vq_is_packed(dev)) rte_free(vq->shadow_used_packed); - else + else { rte_free(vq->shadow_used_split); + if (vq->async_pkts_pending) + rte_free(vq->async_pkts_pending); + if (vq->async_pending_info) + rte_free(vq->async_pending_info); + } rte_free(vq->batch_copy_elems); rte_mempool_free(vq->iotlb_pool); rte_free(vq); @@ -1522,3 +1527,123 @@ RTE_INIT(vhost_log_init) if (vhost_data_log_level >= 0) rte_log_set_level(vhost_data_log_level, RTE_LOG_WARNING); } + +int rte_vhost_async_channel_register(int vid, uint16_t queue_id, + uint32_t features, + struct rte_vhost_async_channel_ops *ops) +{ + struct vhost_virtqueue *vq; + struct virtio_net *dev = get_device(vid); + struct rte_vhost_async_features f; + + if (dev == NULL || ops == NULL) + return -1; + + f.intval = features; + + vq = dev->virtqueue[queue_id]; + + if (unlikely(vq == NULL || !dev->async_copy)) + return -1; + + /* packed queue is not supported */ + if (unlikely(vq_is_packed(dev) || !f.async_inorder)) { + VHOST_LOG_CONFIG(ERR, + "async copy is not supported on packed queue or non-inorder mode " + "(vid %d, qid: %d)\n", vid, queue_id); + return -1; + } + + if (unlikely(ops->check_completed_copies == NULL || + ops->transfer_data == NULL)) + return -1; + + rte_spinlock_lock(&vq->access_lock); + + if (unlikely(vq->async_registered)) { + VHOST_LOG_CONFIG(ERR, + "async register failed: channel already registered " + "(vid %d, qid: %d)\n", vid, queue_id); + goto reg_out; + } + + vq->async_pkts_pending = rte_malloc(NULL, + vq->size * sizeof(uintptr_t), + RTE_CACHE_LINE_SIZE); + vq->async_pending_info = rte_malloc(NULL, + vq->size * sizeof(uint64_t), + RTE_CACHE_LINE_SIZE); + if (!vq->async_pkts_pending || !vq->async_pending_info) { + if (vq->async_pkts_pending) + rte_free(vq->async_pkts_pending); + + if (vq->async_pending_info) + rte_free(vq->async_pending_info); + + VHOST_LOG_CONFIG(ERR, + "async register failed: cannot allocate memory for vq data " + "(vid %d, qid: %d)\n", vid, queue_id); + goto reg_out; + } + + vq->async_ops.check_completed_copies = ops->check_completed_copies; + vq->async_ops.transfer_data = ops->transfer_data; + + vq->async_inorder = f.async_inorder; + vq->async_threshold = f.async_threshold; + + vq->async_registered = true; + +reg_out: + rte_spinlock_unlock(&vq->access_lock); + + return 0; +} + +int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id) +{ + struct vhost_virtqueue *vq; + struct virtio_net *dev = get_device(vid); + int ret = -1; + + if (dev == NULL) + return ret; + + vq = dev->virtqueue[queue_id]; + + if (vq == NULL) + return ret; + + ret = 0; + rte_spinlock_lock(&vq->access_lock); + + if (!vq->async_registered) + goto out; + + if (vq->async_pkts_inflight_n) { + VHOST_LOG_CONFIG(ERR, "Failed to unregister async channel. " + "async inflight packets must be completed before unregistration.\n"); + ret = -1; + goto out; + } + + if (vq->async_pkts_pending) { + rte_free(vq->async_pkts_pending); + vq->async_pkts_pending = NULL; + } + + if (vq->async_pending_info) { + rte_free(vq->async_pending_info); + vq->async_pending_info = NULL; + } + + vq->async_ops.transfer_data = NULL; + vq->async_ops.check_completed_copies = NULL; + vq->async_registered = false; + +out: + rte_spinlock_unlock(&vq->access_lock); + + return ret; +} + diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h index 034463699..f3731982b 100644 --- a/lib/librte_vhost/vhost.h +++ b/lib/librte_vhost/vhost.h @@ -24,6 +24,8 @@ #include "rte_vdpa.h" #include "rte_vdpa_dev.h" +#include "rte_vhost_async.h" + /* Used to indicate that the device is running on a data core */ #define VIRTIO_DEV_RUNNING 1 /* Used to indicate that the device is ready to operate */ @@ -40,6 +42,11 @@ #define VHOST_LOG_CACHE_NR 32 +#define MAX_PKT_BURST 32 + +#define VHOST_MAX_ASYNC_IT (MAX_PKT_BURST * 2) +#define VHOST_MAX_ASYNC_VEC (BUF_VECTOR_MAX * 2) + #define PACKED_DESC_ENQUEUE_USED_FLAG(w) \ ((w) ? (VRING_DESC_F_AVAIL | VRING_DESC_F_USED | VRING_DESC_F_WRITE) : \ VRING_DESC_F_WRITE) @@ -202,6 +209,25 @@ struct vhost_virtqueue { TAILQ_HEAD(, vhost_iotlb_entry) iotlb_list; int iotlb_cache_nr; TAILQ_HEAD(, vhost_iotlb_entry) iotlb_pending_list; + + /* operation callbacks for async dma */ + struct rte_vhost_async_channel_ops async_ops; + + struct rte_vhost_iov_iter it_pool[VHOST_MAX_ASYNC_IT]; + struct iovec vec_pool[VHOST_MAX_ASYNC_VEC]; + + /* async data transfer status */ + uintptr_t **async_pkts_pending; + #define ASYNC_PENDING_INFO_N_MSK 0xFFFF + #define ASYNC_PENDING_INFO_N_SFT 16 + uint64_t *async_pending_info; + uint16_t async_pkts_idx; + uint16_t async_pkts_inflight_n; + + /* vq async features */ + bool async_inorder; + bool async_registered; + uint16_t async_threshold; } __rte_cache_aligned; #define VHOST_MAX_VRING 0x100 @@ -338,6 +364,7 @@ struct virtio_net { int16_t broadcast_rarp; uint32_t nr_vring; int dequeue_zero_copy; + int async_copy; int extbuf; int linearbuf; struct vhost_virtqueue *virtqueue[VHOST_MAX_QUEUE_PAIRS * 2]; @@ -683,7 +710,8 @@ vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq) /* Don't kick guest if we don't reach index specified by guest. */ if (dev->features & (1ULL << VIRTIO_RING_F_EVENT_IDX)) { uint16_t old = vq->signalled_used; - uint16_t new = vq->last_used_idx; + uint16_t new = vq->async_pkts_inflight_n ? + vq->used->idx:vq->last_used_idx; bool signalled_used_valid = vq->signalled_used_valid; vq->signalled_used = new; diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c index 6039a8fdb..aa8605523 100644 --- a/lib/librte_vhost/vhost_user.c +++ b/lib/librte_vhost/vhost_user.c @@ -476,12 +476,14 @@ vhost_user_set_vring_num(struct virtio_net **pdev, } else { if (vq->shadow_used_split) rte_free(vq->shadow_used_split); + vq->shadow_used_split = rte_malloc(NULL, vq->size * sizeof(struct vring_used_elem), RTE_CACHE_LINE_SIZE); + if (!vq->shadow_used_split) { VHOST_LOG_CONFIG(ERR, - "failed to allocate memory for shadow used ring.\n"); + "failed to allocate memory for vq internal data.\n"); return RTE_VHOST_MSG_RESULT_ERR; } } @@ -1166,7 +1168,8 @@ vhost_user_set_mem_table(struct virtio_net **pdev, struct VhostUserMsg *msg, goto err_mmap; } - populate = (dev->dequeue_zero_copy) ? MAP_POPULATE : 0; + populate = (dev->dequeue_zero_copy || dev->async_copy) ? + MAP_POPULATE : 0; mmap_addr = mmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_SHARED | populate, fd, 0); @@ -1181,7 +1184,7 @@ vhost_user_set_mem_table(struct virtio_net **pdev, struct VhostUserMsg *msg, reg->host_user_addr = (uint64_t)(uintptr_t)mmap_addr + mmap_offset; - if (dev->dequeue_zero_copy) + if (dev->dequeue_zero_copy || dev->async_copy) if (add_guest_pages(dev, reg, alignment) < 0) { VHOST_LOG_CONFIG(ERR, "adding guest pages to region %u failed.\n", @@ -1979,6 +1982,12 @@ vhost_user_get_vring_base(struct virtio_net **pdev, } else { rte_free(vq->shadow_used_split); vq->shadow_used_split = NULL; + if (vq->async_pkts_pending) + rte_free(vq->async_pkts_pending); + if (vq->async_pending_info) + rte_free(vq->async_pending_info); + vq->async_pkts_pending = NULL; + vq->async_pending_info = NULL; } rte_free(vq->batch_copy_elems); @@ -2012,6 +2021,14 @@ vhost_user_set_vring_enable(struct virtio_net **pdev, "set queue enable: %d to qp idx: %d\n", enable, index); + if (!enable && dev->virtqueue[index]->async_registered) { + if (dev->virtqueue[index]->async_pkts_inflight_n) { + VHOST_LOG_CONFIG(ERR, "failed to disable vring. " + "async inflight packets must be completed first\n"); + return RTE_VHOST_MSG_RESULT_ERR; + } + } + /* On disable, rings have to be stopped being processed. */ if (!enable && dev->dequeue_zero_copy) drain_zmbuf_list(dev->virtqueue[index]); From patchwork Tue Jul 7 05:07:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Fu X-Patchwork-Id: 73361 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 46018A00BE; Tue, 7 Jul 2020 07:07:26 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id CB1651DA64; Tue, 7 Jul 2020 07:07:10 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by dpdk.org (Postfix) with ESMTP id B6E4F1DA60 for ; Tue, 7 Jul 2020 07:07:08 +0200 (CEST) IronPort-SDR: /EPk7OAc6OwBr2e68Z0rZNXGLUcyg3cr4XnqjCK4Qmd7C990Kxq9arffR95W3JO8EZ2jaM/lEV 8x2CSWaFfO/Q== X-IronPort-AV: E=McAfee;i="6000,8403,9674"; a="135005159" X-IronPort-AV: E=Sophos;i="5.75,321,1589266800"; d="scan'208";a="135005159" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Jul 2020 22:07:08 -0700 IronPort-SDR: Vj2MycyO34lO5gmc2xYknyLhDaBQiNY72DYOFZAf+AuQJtJlqCCuEoDAAcprAKVi6/fVR7QXGA ToCRXYXQ/K7Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,321,1589266800"; d="scan'208";a="427346681" Received: from npg-dpdk-patrickfu-casc2.sh.intel.com ([10.67.119.92]) by orsmga004.jf.intel.com with ESMTP; 06 Jul 2020 22:07:05 -0700 From: patrick.fu@intel.com To: dev@dpdk.org, maxime.coquelin@redhat.com, chenbo.xia@intel.com, zhihong.wang@intel.com Cc: patrick.fu@intel.com, yinan.wang@intel.com, cheng1.jiang@intel.com, cunming.liang@intel.com Date: Tue, 7 Jul 2020 13:07:09 +0800 Message-Id: <20200707050709.205480-3-patrick.fu@intel.com> X-Mailer: git-send-email 2.18.4 In-Reply-To: <20200707050709.205480-1-patrick.fu@intel.com> References: <1591869725-13331-1-git-send-email-patrick.fu@intel.com> <20200707050709.205480-1-patrick.fu@intel.com> Subject: [dpdk-dev] [PATCH v6 2/2] vhost: introduce async enqueue for split ring X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Patrick Fu This patch implements async enqueue data path for split ring. 2 new async data path APIs are defined, by which applications can submit and poll packets to/from async engines. The async engine is either a physical DMA device or it could also be a software emulated backend. The async enqueue data path leverages callback functions registered by applications to work with the async engine. Signed-off-by: Patrick Fu Reviewed-by: Chenbo Xia --- lib/librte_vhost/rte_vhost_async.h | 40 +++ lib/librte_vhost/virtio_net.c | 551 ++++++++++++++++++++++++++++- 2 files changed, 589 insertions(+), 2 deletions(-) diff --git a/lib/librte_vhost/rte_vhost_async.h b/lib/librte_vhost/rte_vhost_async.h index d5a59279a..c8ad8dbc7 100644 --- a/lib/librte_vhost/rte_vhost_async.h +++ b/lib/librte_vhost/rte_vhost_async.h @@ -133,4 +133,44 @@ int rte_vhost_async_channel_register(int vid, uint16_t queue_id, __rte_experimental int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id); +/** + * This function submit enqueue data to async engine. This function has + * no guranttee to the transfer completion upon return. Applications + * should poll transfer status by rte_vhost_poll_enqueue_completed() + * + * @param vid + * id of vhost device to enqueue data + * @param queue_id + * queue id to enqueue data + * @param pkts + * array of packets to be enqueued + * @param count + * packets num to be enqueued + * @return + * num of packets enqueued + */ +__rte_experimental +uint16_t rte_vhost_submit_enqueue_burst(int vid, uint16_t queue_id, + struct rte_mbuf **pkts, uint16_t count); + +/** + * This function check async completion status for a specific vhost + * device queue. Packets which finish copying (enqueue) operation + * will be returned in an array. + * + * @param vid + * id of vhost device to enqueue data + * @param queue_id + * queue id to enqueue data + * @param pkts + * blank array to get return packet pointer + * @param count + * size of the packet array + * @return + * num of packets returned + */ +__rte_experimental +uint16_t rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id, + struct rte_mbuf **pkts, uint16_t count); + #endif /* _RTE_VHOST_ASYNC_H_ */ diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c index 751c1f373..236498f71 100644 --- a/lib/librte_vhost/virtio_net.c +++ b/lib/librte_vhost/virtio_net.c @@ -17,14 +17,15 @@ #include #include #include +#include #include "iotlb.h" #include "vhost.h" -#define MAX_PKT_BURST 32 - #define MAX_BATCH_LEN 256 +#define VHOST_ASYNC_BATCH_THRESHOLD 32 + static __rte_always_inline bool rxvq_is_mergeable(struct virtio_net *dev) { @@ -116,6 +117,31 @@ flush_shadow_used_ring_split(struct virtio_net *dev, struct vhost_virtqueue *vq) sizeof(vq->used->idx)); } +static __rte_always_inline void +async_flush_shadow_used_ring_split(struct virtio_net *dev, + struct vhost_virtqueue *vq) +{ + uint16_t used_idx = vq->last_used_idx & (vq->size - 1); + + if (used_idx + vq->shadow_used_idx <= vq->size) { + do_flush_shadow_used_ring_split(dev, vq, used_idx, 0, + vq->shadow_used_idx); + } else { + uint16_t size; + + /* update used ring interval [used_idx, vq->size] */ + size = vq->size - used_idx; + do_flush_shadow_used_ring_split(dev, vq, used_idx, 0, size); + + /* update the left half used ring interval [0, left_size] */ + do_flush_shadow_used_ring_split(dev, vq, 0, size, + vq->shadow_used_idx - size); + } + + vq->last_used_idx += vq->shadow_used_idx; + vq->shadow_used_idx = 0; +} + static __rte_always_inline void update_shadow_used_ring_split(struct vhost_virtqueue *vq, uint16_t desc_idx, uint32_t len) @@ -905,6 +931,200 @@ copy_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, return error; } +static __rte_always_inline void +async_fill_vec(struct iovec *v, void *base, size_t len) +{ + v->iov_base = base; + v->iov_len = len; +} + +static __rte_always_inline void +async_fill_iter(struct rte_vhost_iov_iter *it, size_t count, + struct iovec *vec, unsigned long nr_seg) +{ + it->offset = 0; + it->count = count; + + if (count) { + it->iov = vec; + it->nr_segs = nr_seg; + } else { + it->iov = 0; + it->nr_segs = 0; + } +} + +static __rte_always_inline void +async_fill_desc(struct rte_vhost_async_desc *desc, + struct rte_vhost_iov_iter *src, struct rte_vhost_iov_iter *dst) +{ + desc->src = src; + desc->dst = dst; +} + +static __rte_always_inline int +async_mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq, + struct rte_mbuf *m, struct buf_vector *buf_vec, + uint16_t nr_vec, uint16_t num_buffers, + struct iovec *src_iovec, struct iovec *dst_iovec, + struct rte_vhost_iov_iter *src_it, + struct rte_vhost_iov_iter *dst_it) +{ + uint32_t vec_idx = 0; + uint32_t mbuf_offset, mbuf_avail; + uint32_t buf_offset, buf_avail; + uint64_t buf_addr, buf_iova, buf_len; + uint32_t cpy_len, cpy_threshold; + uint64_t hdr_addr; + struct rte_mbuf *hdr_mbuf; + struct batch_copy_elem *batch_copy = vq->batch_copy_elems; + struct virtio_net_hdr_mrg_rxbuf tmp_hdr, *hdr = NULL; + int error = 0; + + uint32_t tlen = 0; + int tvec_idx = 0; + void *hpa; + + if (unlikely(m == NULL)) { + error = -1; + goto out; + } + + cpy_threshold = vq->async_threshold; + + buf_addr = buf_vec[vec_idx].buf_addr; + buf_iova = buf_vec[vec_idx].buf_iova; + buf_len = buf_vec[vec_idx].buf_len; + + if (unlikely(buf_len < dev->vhost_hlen && nr_vec <= 1)) { + error = -1; + goto out; + } + + hdr_mbuf = m; + hdr_addr = buf_addr; + if (unlikely(buf_len < dev->vhost_hlen)) + hdr = &tmp_hdr; + else + hdr = (struct virtio_net_hdr_mrg_rxbuf *)(uintptr_t)hdr_addr; + + VHOST_LOG_DATA(DEBUG, "(%d) RX: num merge buffers %d\n", + dev->vid, num_buffers); + + if (unlikely(buf_len < dev->vhost_hlen)) { + buf_offset = dev->vhost_hlen - buf_len; + vec_idx++; + buf_addr = buf_vec[vec_idx].buf_addr; + buf_iova = buf_vec[vec_idx].buf_iova; + buf_len = buf_vec[vec_idx].buf_len; + buf_avail = buf_len - buf_offset; + } else { + buf_offset = dev->vhost_hlen; + buf_avail = buf_len - dev->vhost_hlen; + } + + mbuf_avail = rte_pktmbuf_data_len(m); + mbuf_offset = 0; + + while (mbuf_avail != 0 || m->next != NULL) { + /* done with current buf, get the next one */ + if (buf_avail == 0) { + vec_idx++; + if (unlikely(vec_idx >= nr_vec)) { + error = -1; + goto out; + } + + buf_addr = buf_vec[vec_idx].buf_addr; + buf_iova = buf_vec[vec_idx].buf_iova; + buf_len = buf_vec[vec_idx].buf_len; + + buf_offset = 0; + buf_avail = buf_len; + } + + /* done with current mbuf, get the next one */ + if (mbuf_avail == 0) { + m = m->next; + + mbuf_offset = 0; + mbuf_avail = rte_pktmbuf_data_len(m); + } + + if (hdr_addr) { + virtio_enqueue_offload(hdr_mbuf, &hdr->hdr); + if (rxvq_is_mergeable(dev)) + ASSIGN_UNLESS_EQUAL(hdr->num_buffers, + num_buffers); + + if (unlikely(hdr == &tmp_hdr)) { + copy_vnet_hdr_to_desc(dev, vq, buf_vec, hdr); + } else { + PRINT_PACKET(dev, (uintptr_t)hdr_addr, + dev->vhost_hlen, 0); + vhost_log_cache_write_iova(dev, vq, + buf_vec[0].buf_iova, + dev->vhost_hlen); + } + + hdr_addr = 0; + } + + cpy_len = RTE_MIN(buf_avail, mbuf_avail); + + if (unlikely(cpy_len >= cpy_threshold)) { + hpa = (void *)(uintptr_t)gpa_to_hpa(dev, + buf_iova + buf_offset, cpy_len); + + if (unlikely(!hpa)) { + error = -1; + goto out; + } + + async_fill_vec(src_iovec + tvec_idx, + (void *)(uintptr_t)rte_pktmbuf_iova_offset(m, + mbuf_offset), cpy_len); + + async_fill_vec(dst_iovec + tvec_idx, hpa, cpy_len); + + tlen += cpy_len; + tvec_idx++; + } else { + if (unlikely(vq->batch_copy_nb_elems >= vq->size)) { + rte_memcpy( + (void *)((uintptr_t)(buf_addr + buf_offset)), + rte_pktmbuf_mtod_offset(m, void *, mbuf_offset), + cpy_len); + + PRINT_PACKET(dev, + (uintptr_t)(buf_addr + buf_offset), + cpy_len, 0); + } else { + batch_copy[vq->batch_copy_nb_elems].dst = + (void *)((uintptr_t)(buf_addr + buf_offset)); + batch_copy[vq->batch_copy_nb_elems].src = + rte_pktmbuf_mtod_offset(m, void *, mbuf_offset); + batch_copy[vq->batch_copy_nb_elems].log_addr = + buf_iova + buf_offset; + batch_copy[vq->batch_copy_nb_elems].len = + cpy_len; + vq->batch_copy_nb_elems++; + } + } + + mbuf_avail -= cpy_len; + mbuf_offset += cpy_len; + buf_avail -= cpy_len; + buf_offset += cpy_len; + } + +out: + async_fill_iter(src_it, tlen, src_iovec, tvec_idx); + async_fill_iter(dst_it, tlen, dst_iovec, tvec_idx); + + return error; +} + static __rte_always_inline int vhost_enqueue_single_packed(struct virtio_net *dev, struct vhost_virtqueue *vq, @@ -1236,6 +1456,333 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id, return virtio_dev_rx(dev, queue_id, pkts, count); } +static __rte_always_inline uint16_t +virtio_dev_rx_async_get_info_idx(uint16_t pkts_idx, + uint16_t vq_size, uint16_t n_inflight) +{ + return pkts_idx > n_inflight ? (pkts_idx - n_inflight) : + (vq_size - n_inflight + pkts_idx) & (vq_size - 1); +} + +static __rte_always_inline void +virtio_dev_rx_async_submit_split_err(struct virtio_net *dev, + struct vhost_virtqueue *vq, uint16_t queue_id, + uint16_t last_idx, uint16_t shadow_idx) +{ + uint16_t start_idx, pkts_idx, vq_size; + uint64_t *async_pending_info; + + pkts_idx = vq->async_pkts_idx; + async_pending_info = vq->async_pending_info; + vq_size = vq->size; + start_idx = virtio_dev_rx_async_get_info_idx(pkts_idx, + vq_size, vq->async_pkts_inflight_n); + + while (likely((start_idx & (vq_size - 1)) != pkts_idx)) { + uint64_t n_seg = + async_pending_info[(start_idx) & (vq_size - 1)] >> + ASYNC_PENDING_INFO_N_SFT; + + while (n_seg) + n_seg -= vq->async_ops.check_completed_copies(dev->vid, + queue_id, 0, 1); + } + + vq->async_pkts_inflight_n = 0; + vq->batch_copy_nb_elems = 0; + + vq->shadow_used_idx = shadow_idx; + vq->last_avail_idx = last_idx; +} + +static __rte_noinline uint32_t +virtio_dev_rx_async_submit_split(struct virtio_net *dev, + struct vhost_virtqueue *vq, uint16_t queue_id, + struct rte_mbuf **pkts, uint32_t count) +{ + uint32_t pkt_idx = 0, pkt_burst_idx = 0; + uint16_t num_buffers; + struct buf_vector buf_vec[BUF_VECTOR_MAX]; + uint16_t avail_head, last_idx, shadow_idx; + + struct rte_vhost_iov_iter *it_pool = vq->it_pool; + struct iovec *vec_pool = vq->vec_pool; + struct rte_vhost_async_desc tdes[MAX_PKT_BURST]; + struct iovec *src_iovec = vec_pool; + struct iovec *dst_iovec = vec_pool + (VHOST_MAX_ASYNC_VEC >> 1); + struct rte_vhost_iov_iter *src_it = it_pool; + struct rte_vhost_iov_iter *dst_it = it_pool + 1; + uint16_t n_free_slot, slot_idx; + int n_pkts = 0; + + avail_head = __atomic_load_n(&vq->avail->idx, __ATOMIC_ACQUIRE); + last_idx = vq->last_avail_idx; + shadow_idx = vq->shadow_used_idx; + + /* + * The ordering between avail index and + * desc reads needs to be enforced. + */ + rte_smp_rmb(); + + rte_prefetch0(&vq->avail->ring[vq->last_avail_idx & (vq->size - 1)]); + + for (pkt_idx = 0; pkt_idx < count; pkt_idx++) { + uint32_t pkt_len = pkts[pkt_idx]->pkt_len + dev->vhost_hlen; + uint16_t nr_vec = 0; + + if (unlikely(reserve_avail_buf_split(dev, vq, + pkt_len, buf_vec, &num_buffers, + avail_head, &nr_vec) < 0)) { + VHOST_LOG_DATA(DEBUG, + "(%d) failed to get enough desc from vring\n", + dev->vid); + vq->shadow_used_idx -= num_buffers; + break; + } + + VHOST_LOG_DATA(DEBUG, "(%d) current index %d | end index %d\n", + dev->vid, vq->last_avail_idx, + vq->last_avail_idx + num_buffers); + + if (async_mbuf_to_desc(dev, vq, pkts[pkt_idx], + buf_vec, nr_vec, num_buffers, + src_iovec, dst_iovec, src_it, dst_it) < 0) { + vq->shadow_used_idx -= num_buffers; + break; + } + + slot_idx = (vq->async_pkts_idx + pkt_idx) & (vq->size - 1); + if (src_it->count) { + async_fill_desc(&tdes[pkt_burst_idx], src_it, dst_it); + pkt_burst_idx++; + vq->async_pending_info[slot_idx] = + num_buffers | (src_it->nr_segs << 16); + src_iovec += src_it->nr_segs; + dst_iovec += dst_it->nr_segs; + src_it += 2; + dst_it += 2; + } else { + vq->async_pending_info[slot_idx] = num_buffers; + vq->async_pkts_inflight_n++; + } + + vq->last_avail_idx += num_buffers; + + if (pkt_burst_idx >= VHOST_ASYNC_BATCH_THRESHOLD || + (pkt_idx == count - 1 && pkt_burst_idx)) { + n_pkts = vq->async_ops.transfer_data(dev->vid, + queue_id, tdes, 0, pkt_burst_idx); + src_iovec = vec_pool; + dst_iovec = vec_pool + (VHOST_MAX_ASYNC_VEC >> 1); + src_it = it_pool; + dst_it = it_pool + 1; + + if (unlikely(n_pkts < (int)pkt_burst_idx)) { + vq->async_pkts_inflight_n += + n_pkts > 0 ? n_pkts : 0; + virtio_dev_rx_async_submit_split_err(dev, + vq, queue_id, last_idx, shadow_idx); + return 0; + } + + pkt_burst_idx = 0; + vq->async_pkts_inflight_n += n_pkts; + } + } + + if (pkt_burst_idx) { + n_pkts = vq->async_ops.transfer_data(dev->vid, + queue_id, tdes, 0, pkt_burst_idx); + if (unlikely(n_pkts < (int)pkt_burst_idx)) { + vq->async_pkts_inflight_n += n_pkts > 0 ? n_pkts : 0; + virtio_dev_rx_async_submit_split_err(dev, vq, queue_id, + last_idx, shadow_idx); + return 0; + } + + vq->async_pkts_inflight_n += n_pkts; + } + + do_data_copy_enqueue(dev, vq); + + n_free_slot = vq->size - vq->async_pkts_idx; + if (n_free_slot > pkt_idx) { + rte_memcpy(&vq->async_pkts_pending[vq->async_pkts_idx], + pkts, pkt_idx * sizeof(uintptr_t)); + vq->async_pkts_idx += pkt_idx; + } else { + rte_memcpy(&vq->async_pkts_pending[vq->async_pkts_idx], + pkts, n_free_slot * sizeof(uintptr_t)); + rte_memcpy(&vq->async_pkts_pending[0], + &pkts[n_free_slot], + (pkt_idx - n_free_slot) * sizeof(uintptr_t)); + vq->async_pkts_idx = pkt_idx - n_free_slot; + } + + if (likely(vq->shadow_used_idx)) + async_flush_shadow_used_ring_split(dev, vq); + + return pkt_idx; +} + +uint16_t rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id, + struct rte_mbuf **pkts, uint16_t count) +{ + struct virtio_net *dev = get_device(vid); + struct vhost_virtqueue *vq; + uint16_t n_pkts_cpl, n_pkts_put = 0, n_descs = 0; + uint16_t start_idx, pkts_idx, vq_size; + uint64_t *async_pending_info; + + VHOST_LOG_DATA(DEBUG, "(%d) %s\n", dev->vid, __func__); + if (unlikely(!is_valid_virt_queue_idx(queue_id, 0, dev->nr_vring))) { + VHOST_LOG_DATA(ERR, "(%d) %s: invalid virtqueue idx %d.\n", + dev->vid, __func__, queue_id); + return 0; + } + + vq = dev->virtqueue[queue_id]; + + rte_spinlock_lock(&vq->access_lock); + + pkts_idx = vq->async_pkts_idx; + async_pending_info = vq->async_pending_info; + vq_size = vq->size; + start_idx = virtio_dev_rx_async_get_info_idx(pkts_idx, + vq_size, vq->async_pkts_inflight_n); + + n_pkts_cpl = + vq->async_ops.check_completed_copies(vid, queue_id, 0, count); + + rte_smp_wmb(); + + while (likely(((start_idx + n_pkts_put) & (vq_size - 1)) != pkts_idx)) { + uint64_t info = async_pending_info[ + (start_idx + n_pkts_put) & (vq_size - 1)]; + uint64_t n_segs; + n_pkts_put++; + n_descs += info & ASYNC_PENDING_INFO_N_MSK; + n_segs = info >> ASYNC_PENDING_INFO_N_SFT; + + if (n_segs) { + if (!n_pkts_cpl || n_pkts_cpl < n_segs) { + n_pkts_put--; + n_descs -= info & ASYNC_PENDING_INFO_N_MSK; + if (n_pkts_cpl) { + async_pending_info[ + (start_idx + n_pkts_put) & + (vq_size - 1)] = + ((n_segs - n_pkts_cpl) << + ASYNC_PENDING_INFO_N_SFT) | + (info & ASYNC_PENDING_INFO_N_MSK); + n_pkts_cpl = 0; + } + break; + } + n_pkts_cpl -= n_segs; + } + } + + if (n_pkts_put) { + vq->async_pkts_inflight_n -= n_pkts_put; + __atomic_add_fetch(&vq->used->idx, n_descs, __ATOMIC_RELEASE); + + vhost_vring_call_split(dev, vq); + } + + if (start_idx + n_pkts_put <= vq_size) { + rte_memcpy(pkts, &vq->async_pkts_pending[start_idx], + n_pkts_put * sizeof(uintptr_t)); + } else { + rte_memcpy(pkts, &vq->async_pkts_pending[start_idx], + (vq_size - start_idx) * sizeof(uintptr_t)); + rte_memcpy(&pkts[vq_size - start_idx], vq->async_pkts_pending, + (n_pkts_put - vq_size + start_idx) * sizeof(uintptr_t)); + } + + rte_spinlock_unlock(&vq->access_lock); + + return n_pkts_put; +} + +static __rte_always_inline uint32_t +virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id, + struct rte_mbuf **pkts, uint32_t count) +{ + struct vhost_virtqueue *vq; + uint32_t nb_tx = 0; + bool drawback = false; + + VHOST_LOG_DATA(DEBUG, "(%d) %s\n", dev->vid, __func__); + if (unlikely(!is_valid_virt_queue_idx(queue_id, 0, dev->nr_vring))) { + VHOST_LOG_DATA(ERR, "(%d) %s: invalid virtqueue idx %d.\n", + dev->vid, __func__, queue_id); + return 0; + } + + vq = dev->virtqueue[queue_id]; + + rte_spinlock_lock(&vq->access_lock); + + if (unlikely(vq->enabled == 0)) + goto out_access_unlock; + + if (unlikely(!vq->async_registered)) { + drawback = true; + goto out_access_unlock; + } + + if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)) + vhost_user_iotlb_rd_lock(vq); + + if (unlikely(vq->access_ok == 0)) + if (unlikely(vring_translate(dev, vq) < 0)) + goto out; + + count = RTE_MIN((uint32_t)MAX_PKT_BURST, count); + if (count == 0) + goto out; + + /* TODO: packed queue not implemented */ + if (vq_is_packed(dev)) + nb_tx = 0; + else + nb_tx = virtio_dev_rx_async_submit_split(dev, + vq, queue_id, pkts, count); + +out: + if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)) + vhost_user_iotlb_rd_unlock(vq); + +out_access_unlock: + rte_spinlock_unlock(&vq->access_lock); + + if (drawback) + return rte_vhost_enqueue_burst(dev->vid, queue_id, pkts, count); + + return nb_tx; +} + +uint16_t +rte_vhost_submit_enqueue_burst(int vid, uint16_t queue_id, + struct rte_mbuf **pkts, uint16_t count) +{ + struct virtio_net *dev = get_device(vid); + + if (!dev) + return 0; + + if (unlikely(!(dev->flags & VIRTIO_DEV_BUILTIN_VIRTIO_NET))) { + VHOST_LOG_DATA(ERR, + "(%d) %s: built-in vhost net backend is disabled.\n", + dev->vid, __func__); + return 0; + } + + return virtio_dev_rx_async_submit(dev, queue_id, pkts, count); +} + static inline bool virtio_net_with_host_offload(struct virtio_net *dev) {