List comments

GET /api/patches/73360/comments/
Content-Type: application/json
Vary: Accept

        "id": 115325,
        "web_url": "",
        "msgid": "<>",
        "date": "2020-07-07T08:22:08",
        "subject": "Re: [dpdk-dev] [PATCH v6 1/2] vhost: introduce async enqueue\n\tregistration API",
        "submitter": {
            "id": 1276,
            "url": "",
            "name": "Xia, Chenbo",
            "email": ""
        "content": "> -----Original Message-----\n> From: Fu, Patrick <>\n> Sent: Tuesday, July 7, 2020 1:07 PM\n> To:;; Xia, Chenbo\n> <>; Wang, Zhihong <>\n> Cc: Fu, Patrick <>; Wang, Yinan <>;\n> Jiang, Cheng1 <>; Liang, Cunming\n> <>\n> Subject: [PATCH v6 1/2] vhost: introduce async enqueue registration API\n> \n> From: Patrick Fu <>\n> \n> Performing large memory copies usually takes up a major part of CPU cycles and\n> becomes the hot spot in vhost-user enqueue operation. To offload the large\n> copies from CPU to the DMA devices, asynchronous APIs are introduced, with\n> which the CPU just submits copy jobs to the DMA but without waiting for its\n> copy completion. Thus, there is no CPU intervention during data transfer. We\n> can save precious CPU cycles and improve the overall throughput for vhost-user\n> based applications. This patch introduces registration/un-registration APIs for\n> vhost async data enqueue operation. Together with the registration APIs\n> implementations, data structures and the prototype of the async callback\n> functions required for async enqueue data path are also defined.\n> \n> Signed-off-by: Patrick Fu <>\n> ---\n>  lib/librte_vhost/Makefile              |   2 +-\n>  lib/librte_vhost/           |   2 +-\n>  lib/librte_vhost/rte_vhost.h           |   1 +\n>  lib/librte_vhost/rte_vhost_async.h     | 136 +++++++++++++++++++++++++\n>  lib/librte_vhost/ |   4 +\n>  lib/librte_vhost/socket.c              |  27 +++++\n>  lib/librte_vhost/vhost.c               | 127 ++++++++++++++++++++++-\n>  lib/librte_vhost/vhost.h               |  30 +++++-\n>  lib/librte_vhost/vhost_user.c          |  23 ++++-\n>  9 files changed, 345 insertions(+), 7 deletions(-)  create mode 100644\n> lib/librte_vhost/rte_vhost_async.h\n> \n> diff --git a/lib/librte_vhost/Makefile b/lib/librte_vhost/Makefile index\n> b7ff7dc4b..4f2f3e47d 100644\n> --- a/lib/librte_vhost/Makefile\n> +++ b/lib/librte_vhost/Makefile\n> @@ -42,7 +42,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_VHOST) := fd_man.c iotlb.c\n> socket.c vhost.c \\\n> \n>  # install includes\n>  SYMLINK-$(CONFIG_RTE_LIBRTE_VHOST)-include += rte_vhost.h rte_vdpa.h \\\n> -\t\t\t\t\t\trte_vdpa_dev.h\n> +\t\t\t\t\t\trte_vdpa_dev.h\n> rte_vhost_async.h\n> \n>  # only compile vhost crypto when cryptodev is enabled  ifeq\n> ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)\n> diff --git a/lib/librte_vhost/ b/lib/librte_vhost/ index\n> 882a0eaf4..cc9aa65c6 100644\n> --- a/lib/librte_vhost/\n> +++ b/lib/librte_vhost/\n> @@ -22,5 +22,5 @@ sources = files('fd_man.c', 'iotlb.c', 'socket.c', 'vdpa.c',\n>  \t\t'vhost.c', 'vhost_user.c',\n>  \t\t'virtio_net.c', 'vhost_crypto.c')\n>  headers = files('rte_vhost.h', 'rte_vdpa.h', 'rte_vdpa_dev.h',\n> -\t\t'rte_vhost_crypto.h')\n> +\t\t'rte_vhost_crypto.h', 'rte_vhost_async.h')\n>  deps += ['ethdev', 'cryptodev', 'hash', 'pci'] diff --git\n> a/lib/librte_vhost/rte_vhost.h b/lib/librte_vhost/rte_vhost.h index\n> 8a5c332c8..f93f9595a 100644\n> --- a/lib/librte_vhost/rte_vhost.h\n> +++ b/lib/librte_vhost/rte_vhost.h\n> @@ -35,6 +35,7 @@ extern \"C\" {\n>  #define RTE_VHOST_USER_EXTBUF_SUPPORT\t(1ULL << 5)\n>  /* support only linear buffers (no chained mbufs) */\n>  #define RTE_VHOST_USER_LINEARBUF_SUPPORT\t(1ULL << 6)\n> +#define RTE_VHOST_USER_ASYNC_COPY\t(1ULL << 7)\n> \n>  /* Features. */\n>  #ifndef VIRTIO_NET_F_GUEST_ANNOUNCE\n> diff --git a/lib/librte_vhost/rte_vhost_async.h\n> b/lib/librte_vhost/rte_vhost_async.h\n> new file mode 100644\n> index 000000000..d5a59279a\n> --- /dev/null\n> +++ b/lib/librte_vhost/rte_vhost_async.h\n> @@ -0,0 +1,136 @@\n> +/* SPDX-License-Identifier: BSD-3-Clause\n> + * Copyright(c) 2020 Intel Corporation\n> + */\n> +\n> +#ifndef _RTE_VHOST_ASYNC_H_\n> +#define _RTE_VHOST_ASYNC_H_\n> +\n> +#include \"rte_vhost.h\"\n> +\n> +/**\n> + * iovec iterator\n> + */\n> +struct rte_vhost_iov_iter {\n> +\t/** offset to the first byte of interesting data */\n> +\tsize_t offset;\n> +\t/** total bytes of data in this iterator */\n> +\tsize_t count;\n> +\t/** pointer to the iovec array */\n> +\tstruct iovec *iov;\n> +\t/** number of iovec in this iterator */\n> +\tunsigned long nr_segs;\n> +};\n> +\n> +/**\n> + * dma transfer descriptor pair\n> + */\n> +struct rte_vhost_async_desc {\n> +\t/** source memory iov_iter */\n> +\tstruct rte_vhost_iov_iter *src;\n> +\t/** destination memory iov_iter */\n> +\tstruct rte_vhost_iov_iter *dst;\n> +};\n> +\n> +/**\n> + * dma transfer status\n> + */\n> +struct rte_vhost_async_status {\n> +\t/** An array of application specific data for source memory */\n> +\tuintptr_t *src_opaque_data;\n> +\t/** An array of application specific data for destination memory */\n> +\tuintptr_t *dst_opaque_data;\n> +};\n> +\n> +/**\n> + * dma operation callbacks to be implemented by applications  */ struct\n> +rte_vhost_async_channel_ops {\n> +\t/**\n> +\t * instruct async engines to perform copies for a batch of packets\n> +\t *\n> +\t * @param vid\n> +\t *  id of vhost device to perform data copies\n> +\t * @param queue_id\n> +\t *  queue id to perform data copies\n> +\t * @param descs\n> +\t *  an array of DMA transfer memory descriptors\n> +\t * @param opaque_data\n> +\t *  opaque data pair sending to DMA engine\n> +\t * @param count\n> +\t *  number of elements in the \"descs\" array\n> +\t * @return\n> +\t *  -1 on failure, number of descs processed on success\n> +\t */\n> +\tint (*transfer_data)(int vid, uint16_t queue_id,\n> +\t\tstruct rte_vhost_async_desc *descs,\n> +\t\tstruct rte_vhost_async_status *opaque_data,\n> +\t\tuint16_t count);\n> +\t/**\n> +\t * check copy-completed packets from the async engine\n> +\t * @param vid\n> +\t *  id of vhost device to check copy completion\n> +\t * @param queue_id\n> +\t *  queue id to check copyp completion\n> +\t * @param opaque_data\n> +\t *  buffer to receive the opaque data pair from DMA engine\n> +\t * @param max_packets\n> +\t *  max number of packets could be completed\n> +\t * @return\n> +\t *  -1 on failure, number of iov segments completed on success\n> +\t */\n> +\tint (*check_completed_copies)(int vid, uint16_t queue_id,\n> +\t\tstruct rte_vhost_async_status *opaque_data,\n> +\t\tuint16_t max_packets);\n> +};\n> +\n> +/**\n> + *  dma channel feature bit definition\n> + */\n> +struct rte_vhost_async_features {\n> +\tunion {\n> +\t\tuint32_t intval;\n> +\t\tstruct {\n> +\t\t\tuint32_t async_inorder:1;\n> +\t\t\tuint32_t resvd_0:15;\n> +\t\t\tuint32_t async_threshold:12;\n> +\t\t\tuint32_t resvd_1:4;\n> +\t\t};\n> +\t};\n> +};\n> +\n> +/**\n> + * register a async channel for vhost\n> + *\n> + * @param vid\n> + *  vhost device id async channel to be attached to\n> + * @param queue_id\n> + *  vhost queue id async channel to be attached to\n> + * @param features\n> + *  DMA channel feature bit\n> + *    b0       : DMA supports inorder data transfer\n> + *    b1  - b15: reserved\n> + *    b16 - b27: Packet length threshold for DMA transfer\n> + *    b28 - b31: reserved\n> + * @param ops\n> + *  DMA operation callbacks\n> + * @return\n> + *  0 on success, -1 on failures\n> + */\n> +__rte_experimental\n> +int rte_vhost_async_channel_register(int vid, uint16_t queue_id,\n> +\tuint32_t features, struct rte_vhost_async_channel_ops *ops);\n> +\n> +/**\n> + * unregister a dma channel for vhost\n> + *\n> + * @param vid\n> + *  vhost device id DMA channel to be detached\n> + * @param queue_id\n> + *  vhost queue id DMA channel to be detached\n> + * @return\n> + *  0 on success, -1 on failures\n> + */\n> +__rte_experimental\n> +int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id);\n> +\n> +#endif /* _RTE_VHOST_ASYNC_H_ */\n> diff --git a/lib/librte_vhost/\n> b/lib/librte_vhost/\n> index 86784405a..13ec53b63 100644\n> --- a/lib/librte_vhost/\n> +++ b/lib/librte_vhost/\n> @@ -71,4 +71,8 @@ EXPERIMENTAL {\n>  \trte_vdpa_get_queue_num;\n>  \trte_vdpa_get_features;\n>  \trte_vdpa_get_protocol_features;\n> +\trte_vhost_async_channel_register;\n> +\trte_vhost_async_channel_unregister;\n> +\trte_vhost_submit_enqueue_burst;\n> +\trte_vhost_poll_enqueue_completed;\n>  };\n> diff --git a/lib/librte_vhost/socket.c b/lib/librte_vhost/socket.c index\n> 49267cebf..c4626d2c4 100644\n> --- a/lib/librte_vhost/socket.c\n> +++ b/lib/librte_vhost/socket.c\n> @@ -42,6 +42,7 @@ struct vhost_user_socket {\n>  \tbool use_builtin_virtio_net;\n>  \tbool extbuf;\n>  \tbool linearbuf;\n> +\tbool async_copy;\n> \n>  \t/*\n>  \t * The \"supported_features\" indicates the feature bits the @@ -205,6\n> +206,7 @@ vhost_user_add_connection(int fd, struct vhost_user_socket\n> *vsocket)\n>  \tsize_t size;\n>  \tstruct vhost_user_connection *conn;\n>  \tint ret;\n> +\tstruct virtio_net *dev;\n> \n>  \tif (vsocket == NULL)\n>  \t\treturn;\n> @@ -236,6 +238,13 @@ vhost_user_add_connection(int fd, struct\n> vhost_user_socket *vsocket)\n>  \tif (vsocket->linearbuf)\n>  \t\tvhost_enable_linearbuf(vid);\n> \n> +\tif (vsocket->async_copy) {\n> +\t\tdev = get_device(vid);\n> +\n> +\t\tif (dev)\n> +\t\t\tdev->async_copy = 1;\n> +\t}\n> +\n>  \tVHOST_LOG_CONFIG(INFO, \"new device, handle is %d\\n\", vid);\n> \n>  \tif (vsocket->notify_ops->new_connection) { @@ -881,6 +890,17 @@\n> rte_vhost_driver_register(const char *path, uint64_t flags)\n>  \t\tgoto out_mutex;\n>  \t}\n> \n> +\tvsocket->async_copy = flags & RTE_VHOST_USER_ASYNC_COPY;\n> +\n> +\tif (vsocket->async_copy &&\n> +\t\t(flags & (RTE_VHOST_USER_IOMMU_SUPPORT |\n> +\t\tRTE_VHOST_USER_POSTCOPY_SUPPORT))) {\n> +\t\tVHOST_LOG_CONFIG(ERR, \"error: enabling async copy and\n> IOMMU \"\n> +\t\t\t\"or post-copy feature simultaneously is not \"\n> +\t\t\t\"supported\\n\");\n> +\t\tgoto out_mutex;\n> +\t}\n> +\n>  \t/*\n>  \t * Set the supported features correctly for the builtin vhost-user\n>  \t * net driver.\n> @@ -931,6 +951,13 @@ rte_vhost_driver_register(const char *path, uint64_t\n> flags)\n>  \t\t\t~(1ULL << VHOST_USER_PROTOCOL_F_PAGEFAULT);\n>  \t}\n> \n> +\tif (vsocket->async_copy) {\n> +\t\tvsocket->supported_features &= ~(1ULL << VHOST_F_LOG_ALL);\n> +\t\tvsocket->features &= ~(1ULL << VHOST_F_LOG_ALL);\n> +\t\tVHOST_LOG_CONFIG(INFO,\n> +\t\t\t\"Logging feature is disabled in async copy mode\\n\");\n> +\t}\n> +\n>  \t/*\n>  \t * We'll not be able to receive a buffer from guest in linear mode\n>  \t * without external buffer if it will not fit in a single mbuf, which is diff --\n> git a/lib/librte_vhost/vhost.c b/lib/librte_vhost/vhost.c index\n> 0d822d6a3..a11385f39 100644\n> --- a/lib/librte_vhost/vhost.c\n> +++ b/lib/librte_vhost/vhost.c\n> @@ -332,8 +332,13 @@ free_vq(struct virtio_net *dev, struct vhost_virtqueue\n> *vq)  {\n>  \tif (vq_is_packed(dev))\n>  \t\trte_free(vq->shadow_used_packed);\n> -\telse\n> +\telse {\n>  \t\trte_free(vq->shadow_used_split);\n> +\t\tif (vq->async_pkts_pending)\n> +\t\t\trte_free(vq->async_pkts_pending);\n> +\t\tif (vq->async_pending_info)\n> +\t\t\trte_free(vq->async_pending_info);\n> +\t}\n>  \trte_free(vq->batch_copy_elems);\n>  \trte_mempool_free(vq->iotlb_pool);\n>  \trte_free(vq);\n> @@ -1522,3 +1527,123 @@ RTE_INIT(vhost_log_init)\n>  \tif (vhost_data_log_level >= 0)\n>  \t\trte_log_set_level(vhost_data_log_level,\n> RTE_LOG_WARNING);  }\n> +\n> +int rte_vhost_async_channel_register(int vid, uint16_t queue_id,\n> +\t\t\t\t\tuint32_t features,\n> +\t\t\t\t\tstruct rte_vhost_async_channel_ops\n> *ops) {\n> +\tstruct vhost_virtqueue *vq;\n> +\tstruct virtio_net *dev = get_device(vid);\n> +\tstruct rte_vhost_async_features f;\n> +\n> +\tif (dev == NULL || ops == NULL)\n> +\t\treturn -1;\n> +\n> +\tf.intval = features;\n> +\n> +\tvq = dev->virtqueue[queue_id];\n> +\n> +\tif (unlikely(vq == NULL || !dev->async_copy))\n> +\t\treturn -1;\n> +\n> +\t/* packed queue is not supported */\n> +\tif (unlikely(vq_is_packed(dev) || !f.async_inorder)) {\n> +\t\tVHOST_LOG_CONFIG(ERR,\n> +\t\t\t\"async copy is not supported on packed queue or non-\n> inorder mode \"\n> +\t\t\t\"(vid %d, qid: %d)\\n\", vid, queue_id);\n> +\t\treturn -1;\n> +\t}\n> +\n> +\tif (unlikely(ops->check_completed_copies == NULL ||\n> +\t\tops->transfer_data == NULL))\n> +\t\treturn -1;\n> +\n> +\trte_spinlock_lock(&vq->access_lock);\n> +\n> +\tif (unlikely(vq->async_registered)) {\n> +\t\tVHOST_LOG_CONFIG(ERR,\n> +\t\t\t\"async register failed: channel already registered \"\n> +\t\t\t\"(vid %d, qid: %d)\\n\", vid, queue_id);\n> +\t\tgoto reg_out;\n> +\t}\n> +\n> +\tvq->async_pkts_pending = rte_malloc(NULL,\n> +\t\t\tvq->size * sizeof(uintptr_t),\n> +\t\t\tRTE_CACHE_LINE_SIZE);\n> +\tvq->async_pending_info = rte_malloc(NULL,\n> +\t\t\tvq->size * sizeof(uint64_t),\n> +\t\t\tRTE_CACHE_LINE_SIZE);\n> +\tif (!vq->async_pkts_pending || !vq->async_pending_info) {\n> +\t\tif (vq->async_pkts_pending)\n> +\t\t\trte_free(vq->async_pkts_pending);\n> +\n> +\t\tif (vq->async_pending_info)\n> +\t\t\trte_free(vq->async_pending_info);\n> +\n> +\t\tVHOST_LOG_CONFIG(ERR,\n> +\t\t\t\t\"async register failed: cannot allocate memory\n> for vq data \"\n> +\t\t\t\t\"(vid %d, qid: %d)\\n\", vid, queue_id);\n> +\t\tgoto reg_out;\n> +\t}\n> +\n> +\tvq->async_ops.check_completed_copies = ops-\n> >check_completed_copies;\n> +\tvq->async_ops.transfer_data = ops->transfer_data;\n> +\n> +\tvq->async_inorder = f.async_inorder;\n> +\tvq->async_threshold = f.async_threshold;\n> +\n> +\tvq->async_registered = true;\n> +\n> +reg_out:\n> +\trte_spinlock_unlock(&vq->access_lock);\n> +\n> +\treturn 0;\n> +}\n> +\n> +int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id) {\n> +\tstruct vhost_virtqueue *vq;\n> +\tstruct virtio_net *dev = get_device(vid);\n> +\tint ret = -1;\n> +\n> +\tif (dev == NULL)\n> +\t\treturn ret;\n> +\n> +\tvq = dev->virtqueue[queue_id];\n> +\n> +\tif (vq == NULL)\n> +\t\treturn ret;\n> +\n> +\tret = 0;\n> +\trte_spinlock_lock(&vq->access_lock);\n> +\n> +\tif (!vq->async_registered)\n> +\t\tgoto out;\n> +\n> +\tif (vq->async_pkts_inflight_n) {\n> +\t\tVHOST_LOG_CONFIG(ERR, \"Failed to unregister async channel.\n> \"\n> +\t\t\t\"async inflight packets must be completed before\n> unregistration.\\n\");\n> +\t\tret = -1;\n> +\t\tgoto out;\n> +\t}\n> +\n> +\tif (vq->async_pkts_pending) {\n> +\t\trte_free(vq->async_pkts_pending);\n> +\t\tvq->async_pkts_pending = NULL;\n> +\t}\n> +\n> +\tif (vq->async_pending_info) {\n> +\t\trte_free(vq->async_pending_info);\n> +\t\tvq->async_pending_info = NULL;\n> +\t}\n> +\n> +\tvq->async_ops.transfer_data = NULL;\n> +\tvq->async_ops.check_completed_copies = NULL;\n> +\tvq->async_registered = false;\n> +\n> +out:\n> +\trte_spinlock_unlock(&vq->access_lock);\n> +\n> +\treturn ret;\n> +}\n> +\n> diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h index\n> 034463699..f3731982b 100644\n> --- a/lib/librte_vhost/vhost.h\n> +++ b/lib/librte_vhost/vhost.h\n> @@ -24,6 +24,8 @@\n>  #include \"rte_vdpa.h\"\n>  #include \"rte_vdpa_dev.h\"\n> \n> +#include \"rte_vhost_async.h\"\n> +\n>  /* Used to indicate that the device is running on a data core */  #define\n> VIRTIO_DEV_RUNNING 1\n>  /* Used to indicate that the device is ready to operate */ @@ -40,6 +42,11 @@\n> \n>  #define VHOST_LOG_CACHE_NR 32\n> \n> +#define MAX_PKT_BURST 32\n> +\n> +#define VHOST_MAX_ASYNC_IT (MAX_PKT_BURST * 2) #define\n> +VHOST_MAX_ASYNC_VEC (BUF_VECTOR_MAX * 2)\n> +\n>  #define PACKED_DESC_ENQUEUE_USED_FLAG(w)\t\\\n>  \t((w) ? (VRING_DESC_F_AVAIL | VRING_DESC_F_USED |\n> VRING_DESC_F_WRITE) : \\\n>  \t\tVRING_DESC_F_WRITE)\n> @@ -202,6 +209,25 @@ struct vhost_virtqueue {\n>  \tTAILQ_HEAD(, vhost_iotlb_entry) iotlb_list;\n>  \tint\t\t\t\tiotlb_cache_nr;\n>  \tTAILQ_HEAD(, vhost_iotlb_entry) iotlb_pending_list;\n> +\n> +\t/* operation callbacks for async dma */\n> +\tstruct rte_vhost_async_channel_ops\tasync_ops;\n> +\n> +\tstruct rte_vhost_iov_iter it_pool[VHOST_MAX_ASYNC_IT];\n> +\tstruct iovec vec_pool[VHOST_MAX_ASYNC_VEC];\n> +\n> +\t/* async data transfer status */\n> +\tuintptr_t\t**async_pkts_pending;\n> +\t#define\t\tASYNC_PENDING_INFO_N_MSK 0xFFFF\n> +\t#define\t\tASYNC_PENDING_INFO_N_SFT 16\n> +\tuint64_t\t*async_pending_info;\n> +\tuint16_t\tasync_pkts_idx;\n> +\tuint16_t\tasync_pkts_inflight_n;\n> +\n> +\t/* vq async features */\n> +\tbool\t\tasync_inorder;\n> +\tbool\t\tasync_registered;\n> +\tuint16_t\tasync_threshold;\n>  } __rte_cache_aligned;\n> \n>  #define VHOST_MAX_VRING\t\t\t0x100\n> @@ -338,6 +364,7 @@ struct virtio_net {\n>  \tint16_t\t\t\tbroadcast_rarp;\n>  \tuint32_t\t\tnr_vring;\n>  \tint\t\t\tdequeue_zero_copy;\n> +\tint\t\t\tasync_copy;\n>  \tint\t\t\textbuf;\n>  \tint\t\t\tlinearbuf;\n>  \tstruct vhost_virtqueue\t*virtqueue[VHOST_MAX_QUEUE_PAIRS * 2];\n> @@ -683,7 +710,8 @@ vhost_vring_call_split(struct virtio_net *dev, struct\n> vhost_virtqueue *vq)\n>  \t/* Don't kick guest if we don't reach index specified by guest. */\n>  \tif (dev->features & (1ULL << VIRTIO_RING_F_EVENT_IDX)) {\n>  \t\tuint16_t old = vq->signalled_used;\n> -\t\tuint16_t new = vq->last_used_idx;\n> +\t\tuint16_t new = vq->async_pkts_inflight_n ?\n> +\t\t\t\t\tvq->used->idx:vq->last_used_idx;\n>  \t\tbool signalled_used_valid = vq->signalled_used_valid;\n> \n>  \t\tvq->signalled_used = new;\n> diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c index\n> 6039a8fdb..aa8605523 100644\n> --- a/lib/librte_vhost/vhost_user.c\n> +++ b/lib/librte_vhost/vhost_user.c\n> @@ -476,12 +476,14 @@ vhost_user_set_vring_num(struct virtio_net **pdev,\n>  \t} else {\n>  \t\tif (vq->shadow_used_split)\n>  \t\t\trte_free(vq->shadow_used_split);\n> +\n>  \t\tvq->shadow_used_split = rte_malloc(NULL,\n>  \t\t\t\tvq->size * sizeof(struct vring_used_elem),\n>  \t\t\t\tRTE_CACHE_LINE_SIZE);\n> +\n>  \t\tif (!vq->shadow_used_split) {\n>  \t\t\tVHOST_LOG_CONFIG(ERR,\n> -\t\t\t\t\t\"failed to allocate memory for shadow\n> used ring.\\n\");\n> +\t\t\t\t\t\"failed to allocate memory for vq\n> internal data.\\n\");\n>  \t\t\treturn RTE_VHOST_MSG_RESULT_ERR;\n>  \t\t}\n>  \t}\n> @@ -1166,7 +1168,8 @@ vhost_user_set_mem_table(struct virtio_net **pdev,\n> struct VhostUserMsg *msg,\n>  \t\t\tgoto err_mmap;\n>  \t\t}\n> \n> -\t\tpopulate = (dev->dequeue_zero_copy) ? MAP_POPULATE : 0;\n> +\t\tpopulate = (dev->dequeue_zero_copy || dev->async_copy) ?\n> +\t\t\tMAP_POPULATE : 0;\n>  \t\tmmap_addr = mmap(NULL, mmap_size, PROT_READ |\n> PROT_WRITE,\n>  \t\t\t\t MAP_SHARED | populate, fd, 0);\n> \n> @@ -1181,7 +1184,7 @@ vhost_user_set_mem_table(struct virtio_net **pdev,\n> struct VhostUserMsg *msg,\n>  \t\treg->host_user_addr = (uint64_t)(uintptr_t)mmap_addr +\n>  \t\t\t\t      mmap_offset;\n> \n> -\t\tif (dev->dequeue_zero_copy)\n> +\t\tif (dev->dequeue_zero_copy || dev->async_copy)\n>  \t\t\tif (add_guest_pages(dev, reg, alignment) < 0) {\n>  \t\t\t\tVHOST_LOG_CONFIG(ERR,\n>  \t\t\t\t\t\"adding guest pages to region %u\n> failed.\\n\", @@ -1979,6 +1982,12 @@ vhost_user_get_vring_base(struct\n> virtio_net **pdev,\n>  \t} else {\n>  \t\trte_free(vq->shadow_used_split);\n>  \t\tvq->shadow_used_split = NULL;\n> +\t\tif (vq->async_pkts_pending)\n> +\t\t\trte_free(vq->async_pkts_pending);\n> +\t\tif (vq->async_pending_info)\n> +\t\t\trte_free(vq->async_pending_info);\n> +\t\tvq->async_pkts_pending = NULL;\n> +\t\tvq->async_pending_info = NULL;\n>  \t}\n> \n>  \trte_free(vq->batch_copy_elems);\n> @@ -2012,6 +2021,14 @@ vhost_user_set_vring_enable(struct virtio_net\n> **pdev,\n>  \t\t\"set queue enable: %d to qp idx: %d\\n\",\n>  \t\tenable, index);\n> \n> +\tif (!enable && dev->virtqueue[index]->async_registered) {\n> +\t\tif (dev->virtqueue[index]->async_pkts_inflight_n) {\n> +\t\t\tVHOST_LOG_CONFIG(ERR, \"failed to disable vring. \"\n> +\t\t\t\"async inflight packets must be completed first\\n\");\n> +\t\t\treturn RTE_VHOST_MSG_RESULT_ERR;\n> +\t\t}\n> +\t}\n> +\n>  \t/* On disable, rings have to be stopped being processed. */\n>  \tif (!enable && dev->dequeue_zero_copy)\n>  \t\tdrain_zmbuf_list(dev->virtqueue[index]);\n> --\n> 2.18.4\n\nReviewed-by: Chenbo Xia <>",
        "headers": {
            "x-microsoft-antispam-message-info": "\n hLW/VPLaN0dwSDb3CJWzxzs3SU2FyPudt69PI0i3XSSsqK7T5vzuIfsA4Lf/6Z7SBiVZaBR1cRpEY5PEhQ9w0b7+VNkW0jq0qeto7LyFmNmHHI0DT/jMKaR5zZCTBc0QJqS9hxy7M65cVmcd68oJ0q5MyFSpadDz6DT+RMF9M+k6sHILrjURAA7FxUla44Wx3Xj9Qg87qA2HYRzWxeIVYc9WTytNW3nmvNm373buPwPrdgOzmG6g4VcRZ4ZgFYQIxF261beIkIAQyRwpGLSJjW87tY6TJKKQ8H21GN5dDgJjPkqWFzjxXn9FsMk81FYk",
            "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;;\n s=selector2-intel-onmicrosoft-com;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=CPfaPmfIf8escIeJ1V4+tbDMUuvkSbfYk9glyH1QxOA=;\n b=crzxdFB5e5sv85GoE9vgiIPgqmRXXCkjvkAayIGJSwRCHOb8ZollxlTWkE7DiPdMtY/HbQCthEB85kvBS3Okr323+aiueHHBNWVQwUnmrHoBSju3rrERtvo6ItLr7ZoW761lGys0eDHQPN9wiv33TMi7ahQovTkpN/AMliEWghg=",
            "X-Mailman-Version": "2.1.15",
            "X-MS-Exchange-CrossTenant-AuthAs": "Internal",
            "List-Archive": "<>",
            "x-ms-traffictypediagnostic": "MN2PR11MB3712:",
            "Accept-Language": "zh-CN, en-US",
            "X-MS-Has-Attach": "",
            "x-forefront-prvs": "0457F11EAF",
            "CC": "\"Wang, Yinan\" <>, \"Jiang, Cheng1\"\n <>, \"Liang, Cunming\" <>",
            "X-MS-TNEF-Correlator": "",
            "X-MS-Exchange-Transport-CrossTenantHeadersStamped": "MN2PR11MB3712",
            "X-MS-Exchange-CrossTenant-userprincipalname": "\n LXpakB95Gts2zFTJMtQQ9L4WbWze3Uhr121ExHaO5rJ6bW9blZk/WGknYkI/303Fm0dW4T9/9tmcfMfBelScDw==",
            "References": "<>\n <>\n <>",
            "Return-Path": "<>",
            "X-MS-Exchange-CrossTenant-AuthSource": "",
            "X-IronPort-AV": [
                "E=McAfee;i=\"6000,8403,9674\"; a=\"127150403\"",
                "E=Sophos;i=\"5.75,323,1589266800\"; d=\"scan'208\";a=\"127150403\"",
                "E=Sophos;i=\"5.75,323,1589266800\"; d=\"scan'208\";a=\"305579834\""
            "List-Post": "<>",
            "ARC-Authentication-Results": "i=1; 1; spf=pass\n; dmarc=pass action=none;\n dkim=pass; arc=none",
            "Date": "Tue, 7 Jul 2020 08:22:08 +0000",
            "authentication-results": "; dkim=none (message not signed)\n header.d=none;; dmarc=none action=none;",
            "X-MS-Exchange-CrossTenant-originalarrivaltime": "07 Jul 2020 08:22:08.4400 (UTC)",
            "IronPort-SDR": [
                "\n T495kX83oElcNNM5w0N7p3WlJyfU6s31y/3z3dy2RldS1P3EWGU+T/O8yGPeaHdU019Irmbk7K\n f7o3AjxWukhQ==",
                "\n aKvl0h87wBSBDsknK01idc9S05IXDcIUqiUte9ERzOq3fbq8+mZ2IFwXBEj9/+B7GHbRGNeFje\n 1/M5+J79hESg=="
            "Subject": "Re: [dpdk-dev] [PATCH v6 1/2] vhost: introduce async enqueue\n\tregistration API",
            "In-Reply-To": "<>",
            "X-OriginatorOrg": "",
            "x-ms-office365-filtering-correlation-id": "44790960-7990-4c10-4ba6-08d8224ed72f",
            "X-BeenThere": "",
            "Thread-Topic": "[PATCH v6 1/2] vhost: introduce async enqueue registration API",
            "X-MS-Exchange-CrossTenant-Network-Message-Id": "\n 44790960-7990-4c10-4ba6-08d8224ed72f",
            "Content-Language": "en-US",
            "List-Id": "DPDK patches and discussions <>",
            "From": "\"Xia, Chenbo\" <>",
            "x-ms-exchange-senderadcheck": "1",
            "x-ms-publictraffictype": "Email",
            "Delivered-To": "",
            "x-ms-exchange-transport-forked": "True",
            "x-originating-ip": "[]",
            "List-Subscribe": "<>,\n <>",
            "X-Amp-Result": "SKIPPED(no attachment in message)",
            "X-MS-Exchange-CrossTenant-mailboxtype": "HOSTED",
            "To": "\"Fu, Patrick\" <>, \"\" <>,\n \"\" <>, \"Wang, Zhihong\"\n <>",
            "X-MS-Exchange-CrossTenant-fromentityheader": "Hosted",
            "List-Help": "<>",
            "Sender": "\"dev\" <>",
            "x-forefront-antispam-report": "CIP:; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM;; PTR:; CAT:NONE;\n SFTY:;\n SFS:(4636009)(39860400002)(136003)(366004)(346002)(396003)(376002)(26005)(2906002)(110136005)(316002)(6636002)(478600001)(66946007)(64756008)(66446008)(66556008)(66476007)(7696005)(4326008)(6506007)(54906003)(76116006)(107886003)(30864003)(53546011)(33656002)(52536014)(8936002)(186003)(8676002)(86362001)(55016002)(5660300002)(9686003)(83380400001)(71200400001);\n DIR:OUT; SFP:1102;",
            "x-microsoft-antispam-prvs": "\n <>",
            "Errors-To": "",
            "List-Unsubscribe": "<>,\n <>",
            "ARC-Seal": "i=1; a=rsa-sha256; s=arcselector9901;; cv=none;\n b=COmBaEbdZXYiQQcByTydzUSOCAq825gs6CRnkT9yRl1XnNI/bs0CeN8/o93bMrp/uTOTF/QCwSFhx3o/AfV09eTYLh0CY8Lq1+68cTSXfrG3g6kc8ui1UkrJW1kC19YFnnK7H+8Q/pIGyMXZMnwhLxWZzwxwRV/DIlDr2Xsq/YujqL4Rl9jsA5wfFDOjPJgvONJHt/krVCU/vNnO9+4ZIUYdbTU6+4SQqdXVBBOKPO789sQpTgpRmJJicrMF5wn3+WP9cF8StEEhEBJ79eRqKegR/hnVLqSdLeLLqJTOL/vz1Oea1ISWIDdSm9a4pLQOZYcgaXyWO2/LnwedWW/Low==",
            "Received": [
                "from ( [])\n\tby (Postfix) with ESMTP id 0AFB5A00BE;\n\tTue,  7 Jul 2020 10:22:18 +0200 (CEST)",
                "from [] (localhost [])\n\tby (Postfix) with ESMTP id E58451D9E7;\n\tTue,  7 Jul 2020 10:22:16 +0200 (CEST)",
                "from ( [])\n by (Postfix) with ESMTP id 0F1481D619\n for <>; Tue,  7 Jul 2020 10:22:14 +0200 (CEST)",
                "from ([])\n by with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;\n 07 Jul 2020 01:22:14 -0700",
                "from ([])\n by with ESMTP; 07 Jul 2020 01:22:13 -0700",
                "from ( by\n ( with Microsoft SMTP Server (TLS)\n id 14.3.439.0; Tue, 7 Jul 2020 01:22:11 -0700",
                "from ( by\n ( with Microsoft SMTP Server\n (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id\n 15.1.1713.5; Tue, 7 Jul 2020 01:22:10 -0700",
                "from ( by\n ( with Microsoft SMTP Server\n (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1713.5\n via Frontend Transport; Tue, 7 Jul 2020 01:22:10 -0700",
                "from (\n by ( with Microsoft SMTP Server (TLS) id\n 14.3.439.0; Tue, 7 Jul 2020 01:22:10 -0700",
                "from (2603:10b6:208:13f::22)\n by (2603:10b6:208:f6::29)\n with Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.20; Tue, 7 Jul\n 2020 08:22:08 +0000",
                "from\n ([fe80::7cde:8326:5010:c47e]) by\n ([fe80::7cde:8326:5010:c47e%7]) with mapi id 15.20.3153.029; Tue, 7 Jul 2020\n 08:22:08 +0000"
            "Thread-Index": "AQHWVBx+H+h2Y9MdREmD5ClBeN8Niqj7xulQ",
            "Content-Type": "text/plain; charset=\"us-ascii\"",
            "x-ms-oob-tlc-oobclassifiers": "OLM:10000;",
            "X-Amp-File-Uploaded": "False",
            "MIME-Version": "1.0",
            "x-ms-exchange-antispam-messagedata": "\n gofslYkbLsQ1xUIJ0FuhokRe29XuehzmG7VJEEOt/XVzhkSYrNcygNZKQ4RMV+plhJi06c/BNhDDVmHummrqNUZjCA56ohhwRck7Bxt3xtMpSsGLftGgnI9Z9NKMZn8LjSWrWJRBbg1Q6Wze3F5vpRZ29jNZYV7oMEUeFeu1KJzrwtBAJRn6t3sajOxKI6wdJaMhEDydscHL7pK7s78MxoLtEuzj+6Eb3lrl/X6TUJQVHbXcaIdsvhf35wWluRdX70Zhy2TATOI28E+kjYLVbbzRDM7T1lnCJRYV7TV85yc4rQSyGsDFXDZUV0QnLfdqMUBIIvW++VL8PQY1K8Vlswo3HmzgSR2YVVySn1HNh33oVva5c7jgKHdT0kSWaqywjUVbDSqW+mp4IhYtCN4W51emStbLoMaJdgPqNsetVpCUZzsVhCYi8n8AppvdeC3ccMMh8Spa4GGd87heEbUYycTm7COf5xYdI9mQH0cjXO5c5ahxrk/2CXR4ag2s3n9+",
            "Message-ID": "\n <>",
            "Precedence": "list",
            "X-ExtLoop1": "1",
            "x-microsoft-antispam": "BCL:0;",
            "Content-Transfer-Encoding": "quoted-printable",
            "X-MS-Exchange-CrossTenant-id": "46c98d88-e344-4ed4-8496-4ed7712e255d",
            "X-Original-To": "",
            "ARC-Message-Signature": "i=1; a=rsa-sha256; c=relaxed/relaxed;;\n s=arcselector9901;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=CPfaPmfIf8escIeJ1V4+tbDMUuvkSbfYk9glyH1QxOA=;\n b=dzgtStQUNNXteQXvgL86YI5SVd4hSRQ24aJXuDqFthuzfy3W88FssASJUTekCGABBcuCDOGdjH5bYdsxspx+MtEX/5U1b1P7k4QadRPkR3NJQ5ZMmDoW7OKz+plkbirGiOyy/6BYf27fZJArg0fj/8h1prUUYgYdZcqzbpA/THyIiSTxaWCT1l0OOHuqNL1LHjK7lBvXvztVOi2e+xDn56NahCOr9lZLwXhrTwpvwLCWoEJSBeocUs7W6/9JG0uLCDKsqnzVqE8hC4+ilzxp+E1koDG5swKjEGsbLIQ63mo9PaXheCjFOdhwpowN5U6YlkYinn1hP7i3nnqwyg9Bgw=="