Patch Detail

HTTP 200 OK
Allow: GET, PUT, PATCH, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "id": 105528,
    "url": "https://patches.dpdk.org/api/patches/105528/?format=api",
    "web_url": "https://patches.dpdk.org/project/dpdk/patch/20211230215505.329674-2-jiayu.hu@intel.com/",
    "project": {
        "id": 1,
        "url": "https://patches.dpdk.org/api/projects/1/?format=api",
        "name": "DPDK",
        "link_name": "dpdk",
        "list_id": "dev.dpdk.org",
        "list_email": "dev@dpdk.org",
        "web_url": "http://core.dpdk.org",
        "scm_url": "git://dpdk.org/dpdk",
        "webscm_url": "http://git.dpdk.org/dpdk",
        "list_archive_url": "https://inbox.dpdk.org/dev",
        "list_archive_url_format": "https://inbox.dpdk.org/dev/{}",
        "commit_url_format": ""
    },
    "msgid": "<20211230215505.329674-2-jiayu.hu@intel.com>",
    "list_archive_url": "https://inbox.dpdk.org/dev/20211230215505.329674-2-jiayu.hu@intel.com",
    "date": "2021-12-30T21:55:05",
    "name": "[v1,1/1] vhost: integrate dmadev in asynchronous datapath",
    "commit_ref": null,
    "pull_url": null,
    "state": "superseded",
    "archived": true,
    "hash": "de746e4e26e3063564f68d4c539463869f5792e7",
    "submitter": {
        "id": 539,
        "url": "https://patches.dpdk.org/api/people/539/?format=api",
        "name": "Hu, Jiayu",
        "email": "jiayu.hu@intel.com"
    },
    "delegate": {
        "id": 2642,
        "url": "https://patches.dpdk.org/api/users/2642/?format=api",
        "username": "mcoquelin",
        "first_name": "Maxime",
        "last_name": "Coquelin",
        "email": "maxime.coquelin@redhat.com"
    },
    "mbox": "https://patches.dpdk.org/project/dpdk/patch/20211230215505.329674-2-jiayu.hu@intel.com/mbox/",
    "series": [
        {
            "id": 21041,
            "url": "https://patches.dpdk.org/api/series/21041/?format=api",
            "web_url": "https://patches.dpdk.org/project/dpdk/list/?series=21041",
            "date": "2021-12-30T21:55:04",
            "name": "integrate dmadev in vhost",
            "version": 1,
            "mbox": "https://patches.dpdk.org/series/21041/mbox/"
        }
    ],
    "comments": "https://patches.dpdk.org/api/patches/105528/comments/",
    "check": "warning",
    "checks": "https://patches.dpdk.org/api/patches/105528/checks/",
    "tags": {},
    "related": [],
    "headers": {
        "Return-Path": "<dev-bounces@dpdk.org>",
        "X-Original-To": "patchwork@inbox.dpdk.org",
        "Delivered-To": "patchwork@inbox.dpdk.org",
        "Received": [
            "from mails.dpdk.org (mails.dpdk.org [217.70.189.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 97E80A0352;\n\tThu, 30 Dec 2021 14:51:05 +0100 (CET)",
            "from [217.70.189.124] (localhost [127.0.0.1])\n\tby mails.dpdk.org (Postfix) with ESMTP id 7D8C641141;\n\tThu, 30 Dec 2021 14:51:05 +0100 (CET)",
            "from mga14.intel.com (mga14.intel.com [192.55.52.115])\n by mails.dpdk.org (Postfix) with ESMTP id 585A541141\n for <dev@dpdk.org>; Thu, 30 Dec 2021 14:51:02 +0100 (CET)",
            "from orsmga008.jf.intel.com ([10.7.209.65])\n by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;\n 30 Dec 2021 05:51:01 -0800",
            "from npgdpdkvirtiojiayuhu117.sh.intel.com ([10.67.119.202])\n by orsmga008.jf.intel.com with ESMTP; 30 Dec 2021 05:50:58 -0800"
        ],
        "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/simple;\n d=intel.com; i=@intel.com; q=dns/txt; s=Intel;\n t=1640872262; x=1672408262;\n h=from:to:cc:subject:date:message-id:in-reply-to:\n references:mime-version:content-transfer-encoding;\n bh=noDy7DyjhWuqShyTRxCK1xp44WIndw7IpIIdPlIc/3w=;\n b=catkezZLR88cHutPIArCr9WodBT+SP2Pvl3GelBvMU7ksz7+tXAreMs0\n 0YU847t0A4+BPOlF2fYkp3MGNAaqNhRrr1V32w8ayHFHdXzdmkLfb5m8J\n 9p9fSWPGQGzEaRvthGb4/TQWb9PCdsr4JQ9wv68QqmjBcXA67cMRMsna3\n c6sDrhcg2pBT5Z55SOIeRN/PErfpF4xjP/wX5U8Rt0uViNp+Hh/ds5MUQ\n OSHiZ7luhWkxw5FGqU+Ad5SjzLCMXL7RBdw7RjA4OJfz5xqQwEiqxzj0G\n Hio3uUXkC1Jc8LPdAOGlDGJt39+5bz/ZTD3drcBynNj6xk/sEiAYMohgj g==;",
        "X-IronPort-AV": [
            "E=McAfee;i=\"6200,9189,10212\"; a=\"241881260\"",
            "E=Sophos;i=\"5.88,248,1635231600\"; d=\"scan'208\";a=\"241881260\"",
            "E=Sophos;i=\"5.88,248,1635231600\"; d=\"scan'208\";a=\"524397250\""
        ],
        "X-ExtLoop1": "1",
        "From": "Jiayu Hu <jiayu.hu@intel.com>",
        "To": "dev@dpdk.org",
        "Cc": "maxime.coquelin@redhat.com, i.maximets@ovn.org, chenbo.xia@intel.com,\n bruce.richardson@intel.com, harry.van.haaren@intel.com,\n sunil.pai.g@intel.com, john.mcnamara@intel.com, xuan.ding@intel.com,\n cheng1.jiang@intel.com, liangma@liangbit.com, Jiayu Hu <jiayu.hu@intel.com>",
        "Subject": "[PATCH v1 1/1] vhost: integrate dmadev in asynchronous datapath",
        "Date": "Thu, 30 Dec 2021 16:55:05 -0500",
        "Message-Id": "<20211230215505.329674-2-jiayu.hu@intel.com>",
        "X-Mailer": "git-send-email 2.25.1",
        "In-Reply-To": "<20211230215505.329674-1-jiayu.hu@intel.com>",
        "References": "<20211122105437.3534231-1-jiayu.hu@intel.com>\n <20211230215505.329674-1-jiayu.hu@intel.com>",
        "MIME-Version": "1.0",
        "Content-Transfer-Encoding": "8bit",
        "X-BeenThere": "dev@dpdk.org",
        "X-Mailman-Version": "2.1.29",
        "Precedence": "list",
        "List-Id": "DPDK patches and discussions <dev.dpdk.org>",
        "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>",
        "List-Archive": "<http://mails.dpdk.org/archives/dev/>",
        "List-Post": "<mailto:dev@dpdk.org>",
        "List-Help": "<mailto:dev-request@dpdk.org?subject=help>",
        "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>",
        "Errors-To": "dev-bounces@dpdk.org"
    },
    "content": "Since dmadev is introduced in 21.11, to avoid the overhead of vhost DMA\nabstraction layer and simplify application logics, this patch integrates\ndmadev in asynchronous data path.\n\nSigned-off-by: Jiayu Hu <jiayu.hu@intel.com>\nSigned-off-by: Sunil Pai G <sunil.pai.g@intel.com>\n---\n doc/guides/prog_guide/vhost_lib.rst |  70 ++++-----\n examples/vhost/Makefile             |   2 +-\n examples/vhost/ioat.c               | 218 --------------------------\n examples/vhost/ioat.h               |  63 --------\n examples/vhost/main.c               | 230 +++++++++++++++++++++++-----\n examples/vhost/main.h               |  11 ++\n examples/vhost/meson.build          |   6 +-\n lib/vhost/meson.build               |   3 +-\n lib/vhost/rte_vhost_async.h         | 121 +++++----------\n lib/vhost/version.map               |   3 +\n lib/vhost/vhost.c                   | 130 +++++++++++-----\n lib/vhost/vhost.h                   |  53 ++++++-\n lib/vhost/virtio_net.c              | 206 +++++++++++++++++++------\n 13 files changed, 587 insertions(+), 529 deletions(-)\n delete mode 100644 examples/vhost/ioat.c\n delete mode 100644 examples/vhost/ioat.h",
    "diff": "diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst\nindex 76f5d303c9..bdce7cbf02 100644\n--- a/doc/guides/prog_guide/vhost_lib.rst\n+++ b/doc/guides/prog_guide/vhost_lib.rst\n@@ -218,38 +218,12 @@ The following is an overview of some key Vhost API functions:\n \n   Enable or disable zero copy feature of the vhost crypto backend.\n \n-* ``rte_vhost_async_channel_register(vid, queue_id, config, ops)``\n+* ``rte_vhost_async_channel_register(vid, queue_id)``\n \n   Register an async copy device channel for a vhost queue after vring\n-  is enabled. Following device ``config`` must be specified together\n-  with the registration:\n+  is enabled.\n \n-  * ``features``\n-\n-    This field is used to specify async copy device features.\n-\n-    ``RTE_VHOST_ASYNC_INORDER`` represents the async copy device can\n-    guarantee the order of copy completion is the same as the order\n-    of copy submission.\n-\n-    Currently, only ``RTE_VHOST_ASYNC_INORDER`` capable device is\n-    supported by vhost.\n-\n-  Applications must provide following ``ops`` callbacks for vhost lib to\n-  work with the async copy devices:\n-\n-  * ``transfer_data(vid, queue_id, descs, opaque_data, count)``\n-\n-    vhost invokes this function to submit copy data to the async devices.\n-    For non-async_inorder capable devices, ``opaque_data`` could be used\n-    for identifying the completed packets.\n-\n-  * ``check_completed_copies(vid, queue_id, opaque_data, max_packets)``\n-\n-    vhost invokes this function to get the copy data completed by async\n-    devices.\n-\n-* ``rte_vhost_async_channel_register_thread_unsafe(vid, queue_id, config, ops)``\n+* ``rte_vhost_async_channel_register_thread_unsafe(vid, queue_id)``\n \n   Register an async copy device channel for a vhost queue without\n   performing any locking.\n@@ -277,18 +251,13 @@ The following is an overview of some key Vhost API functions:\n   This function is only safe to call in vhost callback functions\n   (i.e., struct rte_vhost_device_ops).\n \n-* ``rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count, comp_pkts, comp_count)``\n+* ``rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count, dma_id, dma_vchan)``\n \n   Submit an enqueue request to transmit ``count`` packets from host to guest\n-  by async data path. Successfully enqueued packets can be transfer completed\n-  or being occupied by DMA engines; transfer completed packets are returned in\n-  ``comp_pkts``, but others are not guaranteed to finish, when this API\n-  call returns.\n+  by async data path. Applications must not free the packets submitted for\n+  enqueue until the packets are completed.\n \n-  Applications must not free the packets submitted for enqueue until the\n-  packets are completed.\n-\n-* ``rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count)``\n+* ``rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count, dma_id, dma_vchan)``\n \n   Poll enqueue completion status from async data path. Completed packets\n   are returned to applications through ``pkts``.\n@@ -298,7 +267,7 @@ The following is an overview of some key Vhost API functions:\n   This function returns the amount of in-flight packets for the vhost\n   queue using async acceleration.\n \n-* ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count)``\n+* ``rte_vhost_clear_queue_thread_unsafe(vid, queue_id, **pkts, count, dma_id, dma_vchan)``\n \n   Clear inflight packets which are submitted to DMA engine in vhost async data\n   path. Completed packets are returned to applications through ``pkts``.\n@@ -442,3 +411,26 @@ Finally, a set of device ops is defined for device specific operations:\n * ``get_notify_area``\n \n   Called to get the notify area info of the queue.\n+\n+Vhost asynchronous data path\n+----------------------------\n+\n+Vhost asynchronous data path leverages DMA devices to offload memory\n+copies from the CPU and it is implemented in an asynchronous way. It\n+enables applcations, like OVS, to save CPU cycles and hide memory copy\n+overhead, thus achieving higher throughput.\n+\n+Vhost doesn't manage DMA devices and applications, like OVS, need to\n+manage and configure DMA devices. Applications need to tell vhost what\n+DMA devices to use in every data path function call. This design enables\n+the flexibility for applications to dynamically use DMA channels in\n+different function modules, not limited in vhost.\n+\n+In addition, vhost supports M:N mapping between vrings and DMA virtual\n+channels. Specifically, one vring can use multiple different DMA channels\n+and one DMA channel can be shared by multiple vrings at the same time.\n+The reason of enabling one vring to use multiple DMA channels is that\n+it's possible that more than one dataplane threads enqueue packets to\n+the same vring with their own DMA virtual channels. Besides, the number\n+of DMA devices is limited. For the purpose of scaling, it's necessary to\n+support sharing DMA channels among vrings.\ndiff --git a/examples/vhost/Makefile b/examples/vhost/Makefile\nindex 587ea2ab47..975a5dfe40 100644\n--- a/examples/vhost/Makefile\n+++ b/examples/vhost/Makefile\n@@ -5,7 +5,7 @@\n APP = vhost-switch\n \n # all source are stored in SRCS-y\n-SRCS-y := main.c virtio_net.c ioat.c\n+SRCS-y := main.c virtio_net.c\n \n PKGCONF ?= pkg-config\n \ndiff --git a/examples/vhost/ioat.c b/examples/vhost/ioat.c\ndeleted file mode 100644\nindex 9aeeb12fd9..0000000000\n--- a/examples/vhost/ioat.c\n+++ /dev/null\n@@ -1,218 +0,0 @@\n-/* SPDX-License-Identifier: BSD-3-Clause\n- * Copyright(c) 2010-2020 Intel Corporation\n- */\n-\n-#include <sys/uio.h>\n-#ifdef RTE_RAW_IOAT\n-#include <rte_rawdev.h>\n-#include <rte_ioat_rawdev.h>\n-\n-#include \"ioat.h\"\n-#include \"main.h\"\n-\n-struct dma_for_vhost dma_bind[MAX_VHOST_DEVICE];\n-\n-struct packet_tracker {\n-\tunsigned short size_track[MAX_ENQUEUED_SIZE];\n-\tunsigned short next_read;\n-\tunsigned short next_write;\n-\tunsigned short last_remain;\n-\tunsigned short ioat_space;\n-};\n-\n-struct packet_tracker cb_tracker[MAX_VHOST_DEVICE];\n-\n-int\n-open_ioat(const char *value)\n-{\n-\tstruct dma_for_vhost *dma_info = dma_bind;\n-\tchar *input = strndup(value, strlen(value) + 1);\n-\tchar *addrs = input;\n-\tchar *ptrs[2];\n-\tchar *start, *end, *substr;\n-\tint64_t vid, vring_id;\n-\tstruct rte_ioat_rawdev_config config;\n-\tstruct rte_rawdev_info info = { .dev_private = &config };\n-\tchar name[32];\n-\tint dev_id;\n-\tint ret = 0;\n-\tuint16_t i = 0;\n-\tchar *dma_arg[MAX_VHOST_DEVICE];\n-\tint args_nr;\n-\n-\twhile (isblank(*addrs))\n-\t\taddrs++;\n-\tif (*addrs == '\\0') {\n-\t\tret = -1;\n-\t\tgoto out;\n-\t}\n-\n-\t/* process DMA devices within bracket. */\n-\taddrs++;\n-\tsubstr = strtok(addrs, \";]\");\n-\tif (!substr) {\n-\t\tret = -1;\n-\t\tgoto out;\n-\t}\n-\targs_nr = rte_strsplit(substr, strlen(substr),\n-\t\t\tdma_arg, MAX_VHOST_DEVICE, ',');\n-\tif (args_nr <= 0) {\n-\t\tret = -1;\n-\t\tgoto out;\n-\t}\n-\twhile (i < args_nr) {\n-\t\tchar *arg_temp = dma_arg[i];\n-\t\tuint8_t sub_nr;\n-\t\tsub_nr = rte_strsplit(arg_temp, strlen(arg_temp), ptrs, 2, '@');\n-\t\tif (sub_nr != 2) {\n-\t\t\tret = -1;\n-\t\t\tgoto out;\n-\t\t}\n-\n-\t\tstart = strstr(ptrs[0], \"txd\");\n-\t\tif (start == NULL) {\n-\t\t\tret = -1;\n-\t\t\tgoto out;\n-\t\t}\n-\n-\t\tstart += 3;\n-\t\tvid = strtol(start, &end, 0);\n-\t\tif (end == start) {\n-\t\t\tret = -1;\n-\t\t\tgoto out;\n-\t\t}\n-\n-\t\tvring_id = 0 + VIRTIO_RXQ;\n-\t\tif (rte_pci_addr_parse(ptrs[1],\n-\t\t\t\t&(dma_info + vid)->dmas[vring_id].addr) < 0) {\n-\t\t\tret = -1;\n-\t\t\tgoto out;\n-\t\t}\n-\n-\t\trte_pci_device_name(&(dma_info + vid)->dmas[vring_id].addr,\n-\t\t\t\tname, sizeof(name));\n-\t\tdev_id = rte_rawdev_get_dev_id(name);\n-\t\tif (dev_id == (uint16_t)(-ENODEV) ||\n-\t\tdev_id == (uint16_t)(-EINVAL)) {\n-\t\t\tret = -1;\n-\t\t\tgoto out;\n-\t\t}\n-\n-\t\tif (rte_rawdev_info_get(dev_id, &info, sizeof(config)) < 0 ||\n-\t\tstrstr(info.driver_name, \"ioat\") == NULL) {\n-\t\t\tret = -1;\n-\t\t\tgoto out;\n-\t\t}\n-\n-\t\t(dma_info + vid)->dmas[vring_id].dev_id = dev_id;\n-\t\t(dma_info + vid)->dmas[vring_id].is_valid = true;\n-\t\tconfig.ring_size = IOAT_RING_SIZE;\n-\t\tconfig.hdls_disable = true;\n-\t\tif (rte_rawdev_configure(dev_id, &info, sizeof(config)) < 0) {\n-\t\t\tret = -1;\n-\t\t\tgoto out;\n-\t\t}\n-\t\trte_rawdev_start(dev_id);\n-\t\tcb_tracker[dev_id].ioat_space = IOAT_RING_SIZE - 1;\n-\t\tdma_info->nr++;\n-\t\ti++;\n-\t}\n-out:\n-\tfree(input);\n-\treturn ret;\n-}\n-\n-int32_t\n-ioat_transfer_data_cb(int vid, uint16_t queue_id,\n-\t\tstruct rte_vhost_iov_iter *iov_iter,\n-\t\tstruct rte_vhost_async_status *opaque_data, uint16_t count)\n-{\n-\tuint32_t i_iter;\n-\tuint16_t dev_id = dma_bind[vid].dmas[queue_id * 2 + VIRTIO_RXQ].dev_id;\n-\tstruct rte_vhost_iov_iter *iter = NULL;\n-\tunsigned long i_seg;\n-\tunsigned short mask = MAX_ENQUEUED_SIZE - 1;\n-\tunsigned short write = cb_tracker[dev_id].next_write;\n-\n-\tif (!opaque_data) {\n-\t\tfor (i_iter = 0; i_iter < count; i_iter++) {\n-\t\t\titer = iov_iter + i_iter;\n-\t\t\ti_seg = 0;\n-\t\t\tif (cb_tracker[dev_id].ioat_space < iter->nr_segs)\n-\t\t\t\tbreak;\n-\t\t\twhile (i_seg < iter->nr_segs) {\n-\t\t\t\trte_ioat_enqueue_copy(dev_id,\n-\t\t\t\t\t(uintptr_t)(iter->iov[i_seg].src_addr),\n-\t\t\t\t\t(uintptr_t)(iter->iov[i_seg].dst_addr),\n-\t\t\t\t\titer->iov[i_seg].len,\n-\t\t\t\t\t0,\n-\t\t\t\t\t0);\n-\t\t\t\ti_seg++;\n-\t\t\t}\n-\t\t\twrite &= mask;\n-\t\t\tcb_tracker[dev_id].size_track[write] = iter->nr_segs;\n-\t\t\tcb_tracker[dev_id].ioat_space -= iter->nr_segs;\n-\t\t\twrite++;\n-\t\t}\n-\t} else {\n-\t\t/* Opaque data is not supported */\n-\t\treturn -1;\n-\t}\n-\t/* ring the doorbell */\n-\trte_ioat_perform_ops(dev_id);\n-\tcb_tracker[dev_id].next_write = write;\n-\treturn i_iter;\n-}\n-\n-int32_t\n-ioat_check_completed_copies_cb(int vid, uint16_t queue_id,\n-\t\tstruct rte_vhost_async_status *opaque_data,\n-\t\tuint16_t max_packets)\n-{\n-\tif (!opaque_data) {\n-\t\tuintptr_t dump[255];\n-\t\tint n_seg;\n-\t\tunsigned short read, write;\n-\t\tunsigned short nb_packet = 0;\n-\t\tunsigned short mask = MAX_ENQUEUED_SIZE - 1;\n-\t\tunsigned short i;\n-\n-\t\tuint16_t dev_id = dma_bind[vid].dmas[queue_id * 2\n-\t\t\t\t+ VIRTIO_RXQ].dev_id;\n-\t\tn_seg = rte_ioat_completed_ops(dev_id, 255, NULL, NULL, dump, dump);\n-\t\tif (n_seg < 0) {\n-\t\t\tRTE_LOG(ERR,\n-\t\t\t\tVHOST_DATA,\n-\t\t\t\t\"fail to poll completed buf on IOAT device %u\",\n-\t\t\t\tdev_id);\n-\t\t\treturn 0;\n-\t\t}\n-\t\tif (n_seg == 0)\n-\t\t\treturn 0;\n-\n-\t\tcb_tracker[dev_id].ioat_space += n_seg;\n-\t\tn_seg += cb_tracker[dev_id].last_remain;\n-\n-\t\tread = cb_tracker[dev_id].next_read;\n-\t\twrite = cb_tracker[dev_id].next_write;\n-\t\tfor (i = 0; i < max_packets; i++) {\n-\t\t\tread &= mask;\n-\t\t\tif (read == write)\n-\t\t\t\tbreak;\n-\t\t\tif (n_seg >= cb_tracker[dev_id].size_track[read]) {\n-\t\t\t\tn_seg -= cb_tracker[dev_id].size_track[read];\n-\t\t\t\tread++;\n-\t\t\t\tnb_packet++;\n-\t\t\t} else {\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t}\n-\t\tcb_tracker[dev_id].next_read = read;\n-\t\tcb_tracker[dev_id].last_remain = n_seg;\n-\t\treturn nb_packet;\n-\t}\n-\t/* Opaque data is not supported */\n-\treturn -1;\n-}\n-\n-#endif /* RTE_RAW_IOAT */\ndiff --git a/examples/vhost/ioat.h b/examples/vhost/ioat.h\ndeleted file mode 100644\nindex d9bf717e8d..0000000000\n--- a/examples/vhost/ioat.h\n+++ /dev/null\n@@ -1,63 +0,0 @@\n-/* SPDX-License-Identifier: BSD-3-Clause\n- * Copyright(c) 2010-2020 Intel Corporation\n- */\n-\n-#ifndef _IOAT_H_\n-#define _IOAT_H_\n-\n-#include <rte_vhost.h>\n-#include <rte_pci.h>\n-#include <rte_vhost_async.h>\n-\n-#define MAX_VHOST_DEVICE 1024\n-#define IOAT_RING_SIZE 4096\n-#define MAX_ENQUEUED_SIZE 4096\n-\n-struct dma_info {\n-\tstruct rte_pci_addr addr;\n-\tuint16_t dev_id;\n-\tbool is_valid;\n-};\n-\n-struct dma_for_vhost {\n-\tstruct dma_info dmas[RTE_MAX_QUEUES_PER_PORT * 2];\n-\tuint16_t nr;\n-};\n-\n-#ifdef RTE_RAW_IOAT\n-int open_ioat(const char *value);\n-\n-int32_t\n-ioat_transfer_data_cb(int vid, uint16_t queue_id,\n-\t\tstruct rte_vhost_iov_iter *iov_iter,\n-\t\tstruct rte_vhost_async_status *opaque_data, uint16_t count);\n-\n-int32_t\n-ioat_check_completed_copies_cb(int vid, uint16_t queue_id,\n-\t\tstruct rte_vhost_async_status *opaque_data,\n-\t\tuint16_t max_packets);\n-#else\n-static int open_ioat(const char *value __rte_unused)\n-{\n-\treturn -1;\n-}\n-\n-static int32_t\n-ioat_transfer_data_cb(int vid __rte_unused, uint16_t queue_id __rte_unused,\n-\t\tstruct rte_vhost_iov_iter *iov_iter __rte_unused,\n-\t\tstruct rte_vhost_async_status *opaque_data __rte_unused,\n-\t\tuint16_t count __rte_unused)\n-{\n-\treturn -1;\n-}\n-\n-static int32_t\n-ioat_check_completed_copies_cb(int vid __rte_unused,\n-\t\tuint16_t queue_id __rte_unused,\n-\t\tstruct rte_vhost_async_status *opaque_data __rte_unused,\n-\t\tuint16_t max_packets __rte_unused)\n-{\n-\treturn -1;\n-}\n-#endif\n-#endif /* _IOAT_H_ */\ndiff --git a/examples/vhost/main.c b/examples/vhost/main.c\nindex 33d023aa39..44073499bc 100644\n--- a/examples/vhost/main.c\n+++ b/examples/vhost/main.c\n@@ -24,8 +24,9 @@\n #include <rte_ip.h>\n #include <rte_tcp.h>\n #include <rte_pause.h>\n+#include <rte_dmadev.h>\n+#include <rte_vhost_async.h>\n \n-#include \"ioat.h\"\n #include \"main.h\"\n \n #ifndef MAX_QUEUES\n@@ -56,6 +57,14 @@\n #define RTE_TEST_TX_DESC_DEFAULT 512\n \n #define INVALID_PORT_ID 0xFF\n+#define INVALID_DMA_ID -1\n+\n+#define MAX_VHOST_DEVICE 1024\n+#define DMA_RING_SIZE 4096\n+\n+struct dma_for_vhost dma_bind[MAX_VHOST_DEVICE];\n+struct rte_vhost_async_dma_info dma_config[RTE_DMADEV_DEFAULT_MAX];\n+static int dma_count;\n \n /* mask of enabled ports */\n static uint32_t enabled_port_mask = 0;\n@@ -96,8 +105,6 @@ static int builtin_net_driver;\n \n static int async_vhost_driver;\n \n-static char *dma_type;\n-\n /* Specify timeout (in useconds) between retries on RX. */\n static uint32_t burst_rx_delay_time = BURST_RX_WAIT_US;\n /* Specify the number of retries on RX. */\n@@ -196,13 +203,134 @@ struct vhost_bufftable *vhost_txbuff[RTE_MAX_LCORE * MAX_VHOST_DEVICE];\n #define MBUF_TABLE_DRAIN_TSC\t((rte_get_tsc_hz() + US_PER_S - 1) \\\n \t\t\t\t / US_PER_S * BURST_TX_DRAIN_US)\n \n+static inline bool\n+is_dma_configured(int16_t dev_id)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < dma_count; i++) {\n+\t\tif (dma_config[i].dev_id == dev_id) {\n+\t\t\treturn true;\n+\t\t}\n+\t}\n+\treturn false;\n+}\n+\n static inline int\n open_dma(const char *value)\n {\n-\tif (dma_type != NULL && strncmp(dma_type, \"ioat\", 4) == 0)\n-\t\treturn open_ioat(value);\n+\tstruct dma_for_vhost *dma_info = dma_bind;\n+\tchar *input = strndup(value, strlen(value) + 1);\n+\tchar *addrs = input;\n+\tchar *ptrs[2];\n+\tchar *start, *end, *substr;\n+\tint64_t vid, vring_id;\n+\n+\tstruct rte_dma_info info;\n+\tstruct rte_dma_conf dev_config = { .nb_vchans = 1 };\n+\tstruct rte_dma_vchan_conf qconf = {\n+\t\t.direction = RTE_DMA_DIR_MEM_TO_MEM,\n+\t\t.nb_desc = DMA_RING_SIZE\n+\t};\n+\n+\tint dev_id;\n+\tint ret = 0;\n+\tuint16_t i = 0;\n+\tchar *dma_arg[MAX_VHOST_DEVICE];\n+\tint args_nr;\n+\n+\twhile (isblank(*addrs))\n+\t\taddrs++;\n+\tif (*addrs == '\\0') {\n+\t\tret = -1;\n+\t\tgoto out;\n+\t}\n+\n+\t/* process DMA devices within bracket. */\n+\taddrs++;\n+\tsubstr = strtok(addrs, \";]\");\n+\tif (!substr) {\n+\t\tret = -1;\n+\t\tgoto out;\n+\t}\n+\n+\targs_nr = rte_strsplit(substr, strlen(substr),\n+\t\t\tdma_arg, MAX_VHOST_DEVICE, ',');\n+\tif (args_nr <= 0) {\n+\t\tret = -1;\n+\t\tgoto out;\n+\t}\n+\n+\twhile (i < args_nr) {\n+\t\tchar *arg_temp = dma_arg[i];\n+\t\tuint8_t sub_nr;\n+\n+\t\tsub_nr = rte_strsplit(arg_temp, strlen(arg_temp), ptrs, 2, '@');\n+\t\tif (sub_nr != 2) {\n+\t\t\tret = -1;\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tstart = strstr(ptrs[0], \"txd\");\n+\t\tif (start == NULL) {\n+\t\t\tret = -1;\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tstart += 3;\n+\t\tvid = strtol(start, &end, 0);\n+\t\tif (end == start) {\n+\t\t\tret = -1;\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tvring_id = 0 + VIRTIO_RXQ;\n+\n+\t\tdev_id = rte_dma_get_dev_id_by_name(ptrs[1]);\n+\t\tif (dev_id < 0) {\n+\t\t\tRTE_LOG(ERR, VHOST_CONFIG, \"Fail to find DMA %s.\\n\", ptrs[1]);\n+\t\t\tret = -1;\n+\t\t\tgoto out;\n+\t\t} else if (is_dma_configured(dev_id)) {\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tif (rte_dma_configure(dev_id, &dev_config) != 0) {\n+\t\t\tRTE_LOG(ERR, VHOST_CONFIG, \"Fail to configure DMA %d.\\n\", dev_id);\n+\t\t\tret = -1;\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tif (rte_dma_vchan_setup(dev_id, 0, &qconf) != 0) {\n+\t\t\tRTE_LOG(ERR, VHOST_CONFIG, \"Fail to set up DMA %d.\\n\", dev_id);\n+\t\t\tret = -1;\n+\t\t\tgoto out;\n+\t\t}\n \n-\treturn -1;\n+\t\trte_dma_info_get(dev_id, &info);\n+\t\tif (info.nb_vchans != 1) {\n+\t\t\tRTE_LOG(ERR, VHOST_CONFIG, \"DMA %d has no queues.\\n\", dev_id);\n+\t\t\tret = -1;\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tif (rte_dma_start(dev_id) != 0) {\n+\t\t\tRTE_LOG(ERR, VHOST_CONFIG, \"Fail to start DMA %u.\\n\", dev_id);\n+\t\t\tret = -1;\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tdma_config[dma_count].dev_id = dev_id;\n+\t\tdma_config[dma_count].max_vchans = 1;\n+\t\tdma_config[dma_count++].max_desc = DMA_RING_SIZE;\n+\n+done:\n+\t\t(dma_info + vid)->dmas[vring_id].dev_id = dev_id;\n+\t\ti++;\n+\t}\n+out:\n+\tfree(input);\n+\treturn ret;\n }\n \n /*\n@@ -500,8 +628,6 @@ enum {\n \tOPT_CLIENT_NUM,\n #define OPT_BUILTIN_NET_DRIVER  \"builtin-net-driver\"\n \tOPT_BUILTIN_NET_DRIVER_NUM,\n-#define OPT_DMA_TYPE            \"dma-type\"\n-\tOPT_DMA_TYPE_NUM,\n #define OPT_DMAS                \"dmas\"\n \tOPT_DMAS_NUM,\n };\n@@ -539,8 +665,6 @@ us_vhost_parse_args(int argc, char **argv)\n \t\t\t\tNULL, OPT_CLIENT_NUM},\n \t\t{OPT_BUILTIN_NET_DRIVER, no_argument,\n \t\t\t\tNULL, OPT_BUILTIN_NET_DRIVER_NUM},\n-\t\t{OPT_DMA_TYPE, required_argument,\n-\t\t\t\tNULL, OPT_DMA_TYPE_NUM},\n \t\t{OPT_DMAS, required_argument,\n \t\t\t\tNULL, OPT_DMAS_NUM},\n \t\t{NULL, 0, 0, 0},\n@@ -661,10 +785,6 @@ us_vhost_parse_args(int argc, char **argv)\n \t\t\t}\n \t\t\tbreak;\n \n-\t\tcase OPT_DMA_TYPE_NUM:\n-\t\t\tdma_type = optarg;\n-\t\t\tbreak;\n-\n \t\tcase OPT_DMAS_NUM:\n \t\t\tif (open_dma(optarg) == -1) {\n \t\t\t\tRTE_LOG(INFO, VHOST_CONFIG,\n@@ -841,9 +961,10 @@ complete_async_pkts(struct vhost_dev *vdev)\n {\n \tstruct rte_mbuf *p_cpl[MAX_PKT_BURST];\n \tuint16_t complete_count;\n+\tint16_t dma_id = dma_bind[vdev->vid].dmas[VIRTIO_RXQ].dev_id;\n \n \tcomplete_count = rte_vhost_poll_enqueue_completed(vdev->vid,\n-\t\t\t\t\tVIRTIO_RXQ, p_cpl, MAX_PKT_BURST);\n+\t\t\t\t\tVIRTIO_RXQ, p_cpl, MAX_PKT_BURST, dma_id, 0);\n \tif (complete_count) {\n \t\tfree_pkts(p_cpl, complete_count);\n \t\t__atomic_sub_fetch(&vdev->pkts_inflight, complete_count, __ATOMIC_SEQ_CST);\n@@ -883,11 +1004,12 @@ drain_vhost(struct vhost_dev *vdev)\n \n \tif (builtin_net_driver) {\n \t\tret = vs_enqueue_pkts(vdev, VIRTIO_RXQ, m, nr_xmit);\n-\t} else if (async_vhost_driver) {\n+\t} else if (dma_bind[vdev->vid].dmas[VIRTIO_RXQ].async_enabled) {\n \t\tuint16_t enqueue_fail = 0;\n+\t\tint16_t dma_id = dma_bind[vdev->vid].dmas[VIRTIO_RXQ].dev_id;\n \n \t\tcomplete_async_pkts(vdev);\n-\t\tret = rte_vhost_submit_enqueue_burst(vdev->vid, VIRTIO_RXQ, m, nr_xmit);\n+\t\tret = rte_vhost_submit_enqueue_burst(vdev->vid, VIRTIO_RXQ, m, nr_xmit, dma_id, 0);\n \t\t__atomic_add_fetch(&vdev->pkts_inflight, ret, __ATOMIC_SEQ_CST);\n \n \t\tenqueue_fail = nr_xmit - ret;\n@@ -905,7 +1027,7 @@ drain_vhost(struct vhost_dev *vdev)\n \t\t\t\t__ATOMIC_SEQ_CST);\n \t}\n \n-\tif (!async_vhost_driver)\n+\tif (!dma_bind[vdev->vid].dmas[VIRTIO_RXQ].async_enabled)\n \t\tfree_pkts(m, nr_xmit);\n }\n \n@@ -1211,12 +1333,13 @@ drain_eth_rx(struct vhost_dev *vdev)\n \tif (builtin_net_driver) {\n \t\tenqueue_count = vs_enqueue_pkts(vdev, VIRTIO_RXQ,\n \t\t\t\t\t\tpkts, rx_count);\n-\t} else if (async_vhost_driver) {\n+\t} else if (dma_bind[vdev->vid].dmas[VIRTIO_RXQ].async_enabled) {\n \t\tuint16_t enqueue_fail = 0;\n+\t\tint16_t dma_id = dma_bind[vdev->vid].dmas[VIRTIO_RXQ].dev_id;\n \n \t\tcomplete_async_pkts(vdev);\n \t\tenqueue_count = rte_vhost_submit_enqueue_burst(vdev->vid,\n-\t\t\t\t\tVIRTIO_RXQ, pkts, rx_count);\n+\t\t\t\t\tVIRTIO_RXQ, pkts, rx_count, dma_id, 0);\n \t\t__atomic_add_fetch(&vdev->pkts_inflight, enqueue_count, __ATOMIC_SEQ_CST);\n \n \t\tenqueue_fail = rx_count - enqueue_count;\n@@ -1235,7 +1358,7 @@ drain_eth_rx(struct vhost_dev *vdev)\n \t\t\t\t__ATOMIC_SEQ_CST);\n \t}\n \n-\tif (!async_vhost_driver)\n+\tif (!dma_bind[vdev->vid].dmas[VIRTIO_RXQ].async_enabled)\n \t\tfree_pkts(pkts, rx_count);\n }\n \n@@ -1387,18 +1510,20 @@ destroy_device(int vid)\n \t\t\"(%d) device has been removed from data core\\n\",\n \t\tvdev->vid);\n \n-\tif (async_vhost_driver) {\n+\tif (dma_bind[vid].dmas[VIRTIO_RXQ].async_enabled) {\n \t\tuint16_t n_pkt = 0;\n+\t\tint16_t dma_id = dma_bind[vid].dmas[VIRTIO_RXQ].dev_id;\n \t\tstruct rte_mbuf *m_cpl[vdev->pkts_inflight];\n \n \t\twhile (vdev->pkts_inflight) {\n \t\t\tn_pkt = rte_vhost_clear_queue_thread_unsafe(vid, VIRTIO_RXQ,\n-\t\t\t\t\t\tm_cpl, vdev->pkts_inflight);\n+\t\t\t\t\t\tm_cpl, vdev->pkts_inflight, dma_id, 0);\n \t\t\tfree_pkts(m_cpl, n_pkt);\n \t\t\t__atomic_sub_fetch(&vdev->pkts_inflight, n_pkt, __ATOMIC_SEQ_CST);\n \t\t}\n \n \t\trte_vhost_async_channel_unregister(vid, VIRTIO_RXQ);\n+\t\tdma_bind[vid].dmas[VIRTIO_RXQ].async_enabled = false;\n \t}\n \n \trte_free(vdev);\n@@ -1468,20 +1593,14 @@ new_device(int vid)\n \t\t\"(%d) device has been added to data core %d\\n\",\n \t\tvid, vdev->coreid);\n \n-\tif (async_vhost_driver) {\n-\t\tstruct rte_vhost_async_config config = {0};\n-\t\tstruct rte_vhost_async_channel_ops channel_ops;\n-\n-\t\tif (dma_type != NULL && strncmp(dma_type, \"ioat\", 4) == 0) {\n-\t\t\tchannel_ops.transfer_data = ioat_transfer_data_cb;\n-\t\t\tchannel_ops.check_completed_copies =\n-\t\t\t\tioat_check_completed_copies_cb;\n-\n-\t\t\tconfig.features = RTE_VHOST_ASYNC_INORDER;\n+\tif (dma_bind[vid].dmas[VIRTIO_RXQ].dev_id != INVALID_DMA_ID) {\n+\t\tint ret;\n \n-\t\t\treturn rte_vhost_async_channel_register(vid, VIRTIO_RXQ,\n-\t\t\t\tconfig, &channel_ops);\n+\t\tret = rte_vhost_async_channel_register(vid, VIRTIO_RXQ);\n+\t\tif (ret == 0) {\n+\t\t\tdma_bind[vid].dmas[VIRTIO_RXQ].async_enabled = true;\n \t\t}\n+\t\treturn ret;\n \t}\n \n \treturn 0;\n@@ -1502,14 +1621,15 @@ vring_state_changed(int vid, uint16_t queue_id, int enable)\n \tif (queue_id != VIRTIO_RXQ)\n \t\treturn 0;\n \n-\tif (async_vhost_driver) {\n+\tif (dma_bind[vid].dmas[queue_id].async_enabled) {\n \t\tif (!enable) {\n \t\t\tuint16_t n_pkt = 0;\n+\t\t\tint16_t dma_id = dma_bind[vid].dmas[VIRTIO_RXQ].dev_id;\n \t\t\tstruct rte_mbuf *m_cpl[vdev->pkts_inflight];\n \n \t\t\twhile (vdev->pkts_inflight) {\n \t\t\t\tn_pkt = rte_vhost_clear_queue_thread_unsafe(vid, queue_id,\n-\t\t\t\t\t\t\tm_cpl, vdev->pkts_inflight);\n+\t\t\t\t\t\t\tm_cpl, vdev->pkts_inflight, dma_id, 0);\n \t\t\t\tfree_pkts(m_cpl, n_pkt);\n \t\t\t\t__atomic_sub_fetch(&vdev->pkts_inflight, n_pkt, __ATOMIC_SEQ_CST);\n \t\t\t}\n@@ -1657,6 +1777,25 @@ create_mbuf_pool(uint16_t nr_port, uint32_t nr_switch_core, uint32_t mbuf_size,\n \t\trte_exit(EXIT_FAILURE, \"Cannot create mbuf pool\\n\");\n }\n \n+static void\n+init_dma(void)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < MAX_VHOST_DEVICE; i++) {\n+\t\tint j;\n+\n+\t\tfor (j = 0; j < RTE_MAX_QUEUES_PER_PORT * 2; j++) {\n+\t\t\tdma_bind[i].dmas[j].dev_id = INVALID_DMA_ID;\n+\t\t\tdma_bind[i].dmas[j].async_enabled = false;\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < RTE_DMADEV_DEFAULT_MAX; i++) {\n+\t\tdma_config[i].dev_id = INVALID_DMA_ID;\n+\t}\n+}\n+\n /*\n  * Main function, does initialisation and calls the per-lcore functions.\n  */\n@@ -1679,6 +1818,9 @@ main(int argc, char *argv[])\n \targc -= ret;\n \targv += ret;\n \n+\t/* initialize dma structures */\n+\tinit_dma();\n+\n \t/* parse app arguments */\n \tret = us_vhost_parse_args(argc, argv);\n \tif (ret < 0)\n@@ -1754,6 +1896,20 @@ main(int argc, char *argv[])\n \tif (client_mode)\n \t\tflags |= RTE_VHOST_USER_CLIENT;\n \n+\tif (async_vhost_driver) {\n+\t\tif (rte_vhost_async_dma_configure(dma_config, dma_count) < 0) {\n+\t\t\tRTE_LOG(ERR, VHOST_PORT, \"Failed to configure DMA in vhost.\\n\");\n+\t\t\tfor (i = 0; i < dma_count; i++) {\n+\t\t\t\tif (dma_config[i].dev_id != INVALID_DMA_ID) {\n+\t\t\t\t\trte_dma_stop(dma_config[i].dev_id);\n+\t\t\t\t\tdma_config[i].dev_id = INVALID_DMA_ID;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tdma_count = 0;\n+\t\t\tasync_vhost_driver = false;\n+\t\t}\n+\t}\n+\n \t/* Register vhost user driver to handle vhost messages. */\n \tfor (i = 0; i < nb_sockets; i++) {\n \t\tchar *file = socket_files + i * PATH_MAX;\ndiff --git a/examples/vhost/main.h b/examples/vhost/main.h\nindex e7b1ac60a6..b4a453e77e 100644\n--- a/examples/vhost/main.h\n+++ b/examples/vhost/main.h\n@@ -8,6 +8,7 @@\n #include <sys/queue.h>\n \n #include <rte_ether.h>\n+#include <rte_pci.h>\n \n /* Macros for printing using RTE_LOG */\n #define RTE_LOGTYPE_VHOST_CONFIG RTE_LOGTYPE_USER1\n@@ -79,6 +80,16 @@ struct lcore_info {\n \tstruct vhost_dev_tailq_list vdev_list;\n };\n \n+struct dma_info {\n+\tstruct rte_pci_addr addr;\n+\tint16_t dev_id;\n+\tbool async_enabled;\n+};\n+\n+struct dma_for_vhost {\n+\tstruct dma_info dmas[RTE_MAX_QUEUES_PER_PORT * 2];\n+};\n+\n /* we implement non-extra virtio net features */\n #define VIRTIO_NET_FEATURES\t0\n \ndiff --git a/examples/vhost/meson.build b/examples/vhost/meson.build\nindex 3efd5e6540..87a637f83f 100644\n--- a/examples/vhost/meson.build\n+++ b/examples/vhost/meson.build\n@@ -12,13 +12,9 @@ if not is_linux\n endif\n \n deps += 'vhost'\n+deps += 'dmadev'\n allow_experimental_apis = true\n sources = files(\n         'main.c',\n         'virtio_net.c',\n )\n-\n-if dpdk_conf.has('RTE_RAW_IOAT')\n-    deps += 'raw_ioat'\n-    sources += files('ioat.c')\n-endif\ndiff --git a/lib/vhost/meson.build b/lib/vhost/meson.build\nindex cdb37a4814..8107329400 100644\n--- a/lib/vhost/meson.build\n+++ b/lib/vhost/meson.build\n@@ -33,7 +33,8 @@ headers = files(\n         'rte_vhost_async.h',\n         'rte_vhost_crypto.h',\n )\n+\n driver_sdk_headers = files(\n         'vdpa_driver.h',\n )\n-deps += ['ethdev', 'cryptodev', 'hash', 'pci']\n+deps += ['ethdev', 'cryptodev', 'hash', 'pci', 'dmadev']\ndiff --git a/lib/vhost/rte_vhost_async.h b/lib/vhost/rte_vhost_async.h\nindex a87ea6ba37..23a7a2d8b3 100644\n--- a/lib/vhost/rte_vhost_async.h\n+++ b/lib/vhost/rte_vhost_async.h\n@@ -27,70 +27,12 @@ struct rte_vhost_iov_iter {\n };\n \n /**\n- * dma transfer status\n+ * DMA device information\n  */\n-struct rte_vhost_async_status {\n-\t/** An array of application specific data for source memory */\n-\tuintptr_t *src_opaque_data;\n-\t/** An array of application specific data for destination memory */\n-\tuintptr_t *dst_opaque_data;\n-};\n-\n-/**\n- * dma operation callbacks to be implemented by applications\n- */\n-struct rte_vhost_async_channel_ops {\n-\t/**\n-\t * instruct async engines to perform copies for a batch of packets\n-\t *\n-\t * @param vid\n-\t *  id of vhost device to perform data copies\n-\t * @param queue_id\n-\t *  queue id to perform data copies\n-\t * @param iov_iter\n-\t *  an array of IOV iterators\n-\t * @param opaque_data\n-\t *  opaque data pair sending to DMA engine\n-\t * @param count\n-\t *  number of elements in the \"descs\" array\n-\t * @return\n-\t *  number of IOV iterators processed, negative value means error\n-\t */\n-\tint32_t (*transfer_data)(int vid, uint16_t queue_id,\n-\t\tstruct rte_vhost_iov_iter *iov_iter,\n-\t\tstruct rte_vhost_async_status *opaque_data,\n-\t\tuint16_t count);\n-\t/**\n-\t * check copy-completed packets from the async engine\n-\t * @param vid\n-\t *  id of vhost device to check copy completion\n-\t * @param queue_id\n-\t *  queue id to check copy completion\n-\t * @param opaque_data\n-\t *  buffer to receive the opaque data pair from DMA engine\n-\t * @param max_packets\n-\t *  max number of packets could be completed\n-\t * @return\n-\t *  number of async descs completed, negative value means error\n-\t */\n-\tint32_t (*check_completed_copies)(int vid, uint16_t queue_id,\n-\t\tstruct rte_vhost_async_status *opaque_data,\n-\t\tuint16_t max_packets);\n-};\n-\n-/**\n- *  async channel features\n- */\n-enum {\n-\tRTE_VHOST_ASYNC_INORDER = 1U << 0,\n-};\n-\n-/**\n- *  async channel configuration\n- */\n-struct rte_vhost_async_config {\n-\tuint32_t features;\n-\tuint32_t rsvd[2];\n+struct rte_vhost_async_dma_info {\n+\tint16_t dev_id;\t/* DMA device ID */\n+\tuint16_t max_vchans;\t/* max number of vchan */\n+\tuint16_t max_desc;\t/* max desc number of vchan */\n };\n \n /**\n@@ -100,17 +42,11 @@ struct rte_vhost_async_config {\n  *  vhost device id async channel to be attached to\n  * @param queue_id\n  *  vhost queue id async channel to be attached to\n- * @param config\n- *  Async channel configuration structure\n- * @param ops\n- *  Async channel operation callbacks\n  * @return\n  *  0 on success, -1 on failures\n  */\n __rte_experimental\n-int rte_vhost_async_channel_register(int vid, uint16_t queue_id,\n-\tstruct rte_vhost_async_config config,\n-\tstruct rte_vhost_async_channel_ops *ops);\n+int rte_vhost_async_channel_register(int vid, uint16_t queue_id);\n \n /**\n  * Unregister an async channel for a vhost queue\n@@ -136,17 +72,11 @@ int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id);\n  *  vhost device id async channel to be attached to\n  * @param queue_id\n  *  vhost queue id async channel to be attached to\n- * @param config\n- *  Async channel configuration\n- * @param ops\n- *  Async channel operation callbacks\n  * @return\n  *  0 on success, -1 on failures\n  */\n __rte_experimental\n-int rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id,\n-\tstruct rte_vhost_async_config config,\n-\tstruct rte_vhost_async_channel_ops *ops);\n+int rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id);\n \n /**\n  * Unregister an async channel for a vhost queue without performing any\n@@ -179,12 +109,17 @@ int rte_vhost_async_channel_unregister_thread_unsafe(int vid,\n  *  array of packets to be enqueued\n  * @param count\n  *  packets num to be enqueued\n+ * @param dma_id\n+ *  the identifier of the DMA device\n+ * @param vchan\n+ *  the identifier of virtual DMA channel\n  * @return\n  *  num of packets enqueued\n  */\n __rte_experimental\n uint16_t rte_vhost_submit_enqueue_burst(int vid, uint16_t queue_id,\n-\t\tstruct rte_mbuf **pkts, uint16_t count);\n+\t\tstruct rte_mbuf **pkts, uint16_t count, int16_t dma_id,\n+\t\tuint16_t vchan);\n \n /**\n  * This function checks async completion status for a specific vhost\n@@ -199,12 +134,17 @@ uint16_t rte_vhost_submit_enqueue_burst(int vid, uint16_t queue_id,\n  *  blank array to get return packet pointer\n  * @param count\n  *  size of the packet array\n+ * @param dma_id\n+ *  the identifier of the DMA device\n+ * @param vchan\n+ *  the identifier of virtual DMA channel\n  * @return\n  *  num of packets returned\n  */\n __rte_experimental\n uint16_t rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,\n-\t\tstruct rte_mbuf **pkts, uint16_t count);\n+\t\tstruct rte_mbuf **pkts, uint16_t count, int16_t dma_id,\n+\t\tuint16_t vchan);\n \n /**\n  * This function returns the amount of in-flight packets for the vhost\n@@ -235,11 +175,32 @@ int rte_vhost_async_get_inflight(int vid, uint16_t queue_id);\n  *  Blank array to get return packet pointer\n  * @param count\n  *  Size of the packet array\n+ * @param dma_id\n+ *  the identifier of the DMA device\n+ * @param vchan\n+ *  the identifier of virtual DMA channel\n  * @return\n  *  Number of packets returned\n  */\n __rte_experimental\n uint16_t rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,\n-\t\tstruct rte_mbuf **pkts, uint16_t count);\n+\t\tstruct rte_mbuf **pkts, uint16_t count, int16_t dma_id,\n+\t\tuint16_t vchan);\n+/**\n+ * The DMA vChannels used in asynchronous data path must be configured\n+ * first. So this function needs to be called before enabling DMA\n+ * acceleration for vring. If this function fails, asynchronous data path\n+ * cannot be enabled for any vring further.\n+ *\n+ * @param dmas\n+ *  DMA information\n+ * @param count\n+ *  Element number of 'dmas'\n+ * @return\n+ *  0 on success, and -1 on failure\n+ */\n+__rte_experimental\n+int rte_vhost_async_dma_configure(struct rte_vhost_async_dma_info *dmas,\n+\t\tuint16_t count);\n \n #endif /* _RTE_VHOST_ASYNC_H_ */\ndiff --git a/lib/vhost/version.map b/lib/vhost/version.map\nindex a7ef7f1976..1202ba9c1a 100644\n--- a/lib/vhost/version.map\n+++ b/lib/vhost/version.map\n@@ -84,6 +84,9 @@ EXPERIMENTAL {\n \n \t# added in 21.11\n \trte_vhost_get_monitor_addr;\n+\n+\t# added in 22.03\n+\trte_vhost_async_dma_configure;\n };\n \n INTERNAL {\ndiff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c\nindex 13a9bb9dd1..32f37f4851 100644\n--- a/lib/vhost/vhost.c\n+++ b/lib/vhost/vhost.c\n@@ -344,6 +344,7 @@ vhost_free_async_mem(struct vhost_virtqueue *vq)\n \t\treturn;\n \n \trte_free(vq->async->pkts_info);\n+\trte_free(vq->async->pkts_cmpl_flag);\n \n \trte_free(vq->async->buffers_packed);\n \tvq->async->buffers_packed = NULL;\n@@ -1626,8 +1627,7 @@ rte_vhost_extern_callback_register(int vid,\n }\n \n static __rte_always_inline int\n-async_channel_register(int vid, uint16_t queue_id,\n-\t\tstruct rte_vhost_async_channel_ops *ops)\n+async_channel_register(int vid, uint16_t queue_id)\n {\n \tstruct virtio_net *dev = get_device(vid);\n \tstruct vhost_virtqueue *vq = dev->virtqueue[queue_id];\n@@ -1656,6 +1656,14 @@ async_channel_register(int vid, uint16_t queue_id,\n \t\tgoto out_free_async;\n \t}\n \n+\tasync->pkts_cmpl_flag = rte_zmalloc_socket(NULL, vq->size * sizeof(bool),\n+\t\t\tRTE_CACHE_LINE_SIZE, node);\n+\tif (!async->pkts_cmpl_flag) {\n+\t\tVHOST_LOG_CONFIG(ERR, \"failed to allocate async pkts_cmpl_flag (vid %d, qid: %d)\\n\",\n+\t\t\t\tvid, queue_id);\n+\t\tgoto out_free_async;\n+\t}\n+\n \tif (vq_is_packed(dev)) {\n \t\tasync->buffers_packed = rte_malloc_socket(NULL,\n \t\t\t\tvq->size * sizeof(struct vring_used_elem_packed),\n@@ -1676,9 +1684,6 @@ async_channel_register(int vid, uint16_t queue_id,\n \t\t}\n \t}\n \n-\tasync->ops.check_completed_copies = ops->check_completed_copies;\n-\tasync->ops.transfer_data = ops->transfer_data;\n-\n \tvq->async = async;\n \n \treturn 0;\n@@ -1691,15 +1696,13 @@ async_channel_register(int vid, uint16_t queue_id,\n }\n \n int\n-rte_vhost_async_channel_register(int vid, uint16_t queue_id,\n-\t\tstruct rte_vhost_async_config config,\n-\t\tstruct rte_vhost_async_channel_ops *ops)\n+rte_vhost_async_channel_register(int vid, uint16_t queue_id)\n {\n \tstruct vhost_virtqueue *vq;\n \tstruct virtio_net *dev = get_device(vid);\n \tint ret;\n \n-\tif (dev == NULL || ops == NULL)\n+\tif (dev == NULL)\n \t\treturn -1;\n \n \tif (queue_id >= VHOST_MAX_VRING)\n@@ -1710,33 +1713,20 @@ rte_vhost_async_channel_register(int vid, uint16_t queue_id,\n \tif (unlikely(vq == NULL || !dev->async_copy))\n \t\treturn -1;\n \n-\tif (unlikely(!(config.features & RTE_VHOST_ASYNC_INORDER))) {\n-\t\tVHOST_LOG_CONFIG(ERR,\n-\t\t\t\"async copy is not supported on non-inorder mode \"\n-\t\t\t\"(vid %d, qid: %d)\\n\", vid, queue_id);\n-\t\treturn -1;\n-\t}\n-\n-\tif (unlikely(ops->check_completed_copies == NULL ||\n-\t\tops->transfer_data == NULL))\n-\t\treturn -1;\n-\n \trte_spinlock_lock(&vq->access_lock);\n-\tret = async_channel_register(vid, queue_id, ops);\n+\tret = async_channel_register(vid, queue_id);\n \trte_spinlock_unlock(&vq->access_lock);\n \n \treturn ret;\n }\n \n int\n-rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id,\n-\t\tstruct rte_vhost_async_config config,\n-\t\tstruct rte_vhost_async_channel_ops *ops)\n+rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id)\n {\n \tstruct vhost_virtqueue *vq;\n \tstruct virtio_net *dev = get_device(vid);\n \n-\tif (dev == NULL || ops == NULL)\n+\tif (dev == NULL)\n \t\treturn -1;\n \n \tif (queue_id >= VHOST_MAX_VRING)\n@@ -1747,18 +1737,7 @@ rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id,\n \tif (unlikely(vq == NULL || !dev->async_copy))\n \t\treturn -1;\n \n-\tif (unlikely(!(config.features & RTE_VHOST_ASYNC_INORDER))) {\n-\t\tVHOST_LOG_CONFIG(ERR,\n-\t\t\t\"async copy is not supported on non-inorder mode \"\n-\t\t\t\"(vid %d, qid: %d)\\n\", vid, queue_id);\n-\t\treturn -1;\n-\t}\n-\n-\tif (unlikely(ops->check_completed_copies == NULL ||\n-\t\tops->transfer_data == NULL))\n-\t\treturn -1;\n-\n-\treturn async_channel_register(vid, queue_id, ops);\n+\treturn async_channel_register(vid, queue_id);\n }\n \n int\n@@ -1835,6 +1814,83 @@ rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t queue_id)\n \treturn 0;\n }\n \n+static __rte_always_inline void\n+vhost_free_async_dma_mem(void)\n+{\n+\tuint16_t i;\n+\n+\tfor (i = 0; i < RTE_DMADEV_DEFAULT_MAX; i++) {\n+\t\tstruct async_dma_info *dma = &dma_copy_track[i];\n+\t\tint16_t j;\n+\n+\t\tif (dma->max_vchans == 0) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (j = 0; j < dma->max_vchans; j++) {\n+\t\t\trte_free(dma->vchans[j].metadata);\n+\t\t}\n+\t\trte_free(dma->vchans);\n+\t\tdma->vchans = NULL;\n+\t\tdma->max_vchans = 0;\n+\t}\n+}\n+\n+int\n+rte_vhost_async_dma_configure(struct rte_vhost_async_dma_info *dmas, uint16_t count)\n+{\n+\tuint16_t i;\n+\n+\tif (!dmas) {\n+\t\tVHOST_LOG_CONFIG(ERR, \"Invalid DMA configuration parameter.\\n\");\n+\t\treturn -1;\n+\t}\n+\n+\tfor (i = 0; i < count; i++) {\n+\t\tstruct async_dma_vchan_info *vchans;\n+\t\tint16_t dev_id;\n+\t\tuint16_t max_vchans;\n+\t\tuint16_t max_desc;\n+\t\tuint16_t j;\n+\n+\t\tdev_id = dmas[i].dev_id;\n+\t\tmax_vchans = dmas[i].max_vchans;\n+\t\tmax_desc = dmas[i].max_desc;\n+\n+\t\tif (!rte_is_power_of_2(max_desc)) {\n+\t\t\tmax_desc = rte_align32pow2(max_desc);\n+\t\t}\n+\n+\t\tvchans = rte_zmalloc(NULL, sizeof(struct async_dma_vchan_info) * max_vchans,\n+\t\t\t\tRTE_CACHE_LINE_SIZE);\n+\t\tif (vchans == NULL) {\n+\t\t\tVHOST_LOG_CONFIG(ERR, \"Failed to allocate vchans for dma-%d.\"\n+\t\t\t\t\t\" Cannot enable async data-path.\\n\", dev_id);\n+\t\t\tvhost_free_async_dma_mem();\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tfor (j = 0; j < max_vchans; j++) {\n+\t\t\tvchans[j].metadata = rte_zmalloc(NULL, sizeof(bool *) * max_desc,\n+\t\t\t\t\tRTE_CACHE_LINE_SIZE);\n+\t\t\tif (!vchans[j].metadata) {\n+\t\t\t\tVHOST_LOG_CONFIG(ERR, \"Failed to allocate metadata for \"\n+\t\t\t\t\t\t\"dma-%d vchan-%u\\n\", dev_id, j);\n+\t\t\t\tvhost_free_async_dma_mem();\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\n+\t\t\tvchans[j].ring_size = max_desc;\n+\t\t\tvchans[j].ring_mask = max_desc - 1;\n+\t\t}\n+\n+\t\tdma_copy_track[dev_id].vchans = vchans;\n+\t\tdma_copy_track[dev_id].max_vchans = max_vchans;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int\n rte_vhost_async_get_inflight(int vid, uint16_t queue_id)\n {\ndiff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h\nindex 7085e0885c..d9bda34e11 100644\n--- a/lib/vhost/vhost.h\n+++ b/lib/vhost/vhost.h\n@@ -19,6 +19,7 @@\n #include <rte_ether.h>\n #include <rte_rwlock.h>\n #include <rte_malloc.h>\n+#include <rte_dmadev.h>\n \n #include \"rte_vhost.h\"\n #include \"rte_vdpa.h\"\n@@ -50,6 +51,7 @@\n \n #define VHOST_MAX_ASYNC_IT (MAX_PKT_BURST)\n #define VHOST_MAX_ASYNC_VEC 2048\n+#define VHOST_ASYNC_DMA_BATCHING_SIZE 32\n \n #define PACKED_DESC_ENQUEUE_USED_FLAG(w)\t\\\n \t((w) ? (VRING_DESC_F_AVAIL | VRING_DESC_F_USED | VRING_DESC_F_WRITE) : \\\n@@ -119,6 +121,41 @@ struct vring_used_elem_packed {\n \tuint32_t count;\n };\n \n+struct async_dma_vchan_info {\n+\t/* circular array to track copy metadata */\n+\tbool **metadata;\n+\n+\t/* max elements in 'metadata' */\n+\tuint16_t ring_size;\n+\t/* ring index mask for 'metadata' */\n+\tuint16_t ring_mask;\n+\n+\t/* batching copies before a DMA doorbell */\n+\tuint16_t nr_batching;\n+\n+\t/**\n+\t * DMA virtual channel lock. Although it is able to bind DMA\n+\t * virtual channels to data plane threads, vhost control plane\n+\t * thread could call data plane functions too, thus causing\n+\t * DMA device contention.\n+\t *\n+\t * For example, in VM exit case, vhost control plane thread needs\n+\t * to clear in-flight packets before disable vring, but there could\n+\t * be anotther data plane thread is enqueuing packets to the same\n+\t * vring with the same DMA virtual channel. But dmadev PMD functions\n+\t * are lock-free, so the control plane and data plane threads\n+\t * could operate the same DMA virtual channel at the same time.\n+\t */\n+\trte_spinlock_t dma_lock;\n+};\n+\n+struct async_dma_info {\n+\tuint16_t max_vchans;\n+\tstruct async_dma_vchan_info *vchans;\n+};\n+\n+extern struct async_dma_info dma_copy_track[RTE_DMADEV_DEFAULT_MAX];\n+\n /**\n  * inflight async packet information\n  */\n@@ -129,9 +166,6 @@ struct async_inflight_info {\n };\n \n struct vhost_async {\n-\t/* operation callbacks for DMA */\n-\tstruct rte_vhost_async_channel_ops ops;\n-\n \tstruct rte_vhost_iov_iter iov_iter[VHOST_MAX_ASYNC_IT];\n \tstruct rte_vhost_iovec iovec[VHOST_MAX_ASYNC_VEC];\n \tuint16_t iter_idx;\n@@ -139,6 +173,19 @@ struct vhost_async {\n \n \t/* data transfer status */\n \tstruct async_inflight_info *pkts_info;\n+\t/**\n+\t * packet reorder array. \"true\" indicates that DMA\n+\t * device completes all copies for the packet.\n+\t *\n+\t * Note that this array could be written by multiple\n+\t * threads at the same time. For example, two threads\n+\t * enqueue packets to the same virtqueue with their\n+\t * own DMA devices. However, since offloading is\n+\t * per-packet basis, each packet flag will only be\n+\t * written by one thread. And single byte write is\n+\t * atomic, so no lock is needed.\n+\t */\n+\tbool *pkts_cmpl_flag;\n \tuint16_t pkts_idx;\n \tuint16_t pkts_inflight_n;\n \tunion {\ndiff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c\nindex b3d954aab4..9f81fc9733 100644\n--- a/lib/vhost/virtio_net.c\n+++ b/lib/vhost/virtio_net.c\n@@ -11,6 +11,7 @@\n #include <rte_net.h>\n #include <rte_ether.h>\n #include <rte_ip.h>\n+#include <rte_dmadev.h>\n #include <rte_vhost.h>\n #include <rte_tcp.h>\n #include <rte_udp.h>\n@@ -25,6 +26,9 @@\n \n #define MAX_BATCH_LEN 256\n \n+/* DMA device copy operation tracking array. */\n+struct async_dma_info dma_copy_track[RTE_DMADEV_DEFAULT_MAX];\n+\n static  __rte_always_inline bool\n rxvq_is_mergeable(struct virtio_net *dev)\n {\n@@ -43,6 +47,108 @@ is_valid_virt_queue_idx(uint32_t idx, int is_tx, uint32_t nr_vring)\n \treturn (is_tx ^ (idx & 1)) == 0 && idx < nr_vring;\n }\n \n+static __rte_always_inline uint16_t\n+vhost_async_dma_transfer(struct vhost_virtqueue *vq, int16_t dma_id,\n+\t\tuint16_t vchan, uint16_t head_idx,\n+\t\tstruct rte_vhost_iov_iter *pkts, uint16_t nr_pkts)\n+{\n+\tstruct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan];\n+\tuint16_t ring_mask = dma_info->ring_mask;\n+\tuint16_t pkt_idx;\n+\n+\trte_spinlock_lock(&dma_info->dma_lock);\n+\n+\tfor (pkt_idx = 0; pkt_idx < nr_pkts; pkt_idx++) {\n+\t\tstruct rte_vhost_iovec *iov = pkts[pkt_idx].iov;\n+\t\tint copy_idx = 0;\n+\t\tuint16_t nr_segs = pkts[pkt_idx].nr_segs;\n+\t\tuint16_t i;\n+\n+\t\tif (rte_dma_burst_capacity(dma_id, vchan) < nr_segs) {\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tfor (i = 0; i < nr_segs; i++) {\n+\t\t\t/**\n+\t\t\t * We have checked the available space before submit copies to DMA\n+\t\t\t * vChannel, so we don't handle error here.\n+\t\t\t */\n+\t\t\tcopy_idx = rte_dma_copy(dma_id, vchan, (rte_iova_t)iov[i].src_addr,\n+\t\t\t\t\t(rte_iova_t)iov[i].dst_addr, iov[i].len,\n+\t\t\t\t\tRTE_DMA_OP_FLAG_LLC);\n+\n+\t\t\t/**\n+\t\t\t * Only store packet completion flag address in the last copy's\n+\t\t\t * slot, and other slots are set to NULL.\n+\t\t\t */\n+\t\t\tif (unlikely(i == (nr_segs - 1))) {\n+\t\t\t\tdma_info->metadata[copy_idx & ring_mask] =\n+\t\t\t\t\t&vq->async->pkts_cmpl_flag[head_idx % vq->size];\n+\t\t\t}\n+\t\t}\n+\n+\t\tdma_info->nr_batching += nr_segs;\n+\t\tif (unlikely(dma_info->nr_batching >= VHOST_ASYNC_DMA_BATCHING_SIZE)) {\n+\t\t\trte_dma_submit(dma_id, vchan);\n+\t\t\tdma_info->nr_batching = 0;\n+\t\t}\n+\n+\t\thead_idx++;\n+\t}\n+\n+out:\n+\tif (dma_info->nr_batching > 0) {\n+\t\trte_dma_submit(dma_id, vchan);\n+\t\tdma_info->nr_batching = 0;\n+\t}\n+\trte_spinlock_unlock(&dma_info->dma_lock);\n+\n+\treturn pkt_idx;\n+}\n+\n+static __rte_always_inline uint16_t\n+vhost_async_dma_check_completed(int16_t dma_id, uint16_t vchan, uint16_t max_pkts)\n+{\n+\tstruct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan];\n+\tuint16_t ring_mask = dma_info->ring_mask;\n+\tuint16_t last_idx = 0;\n+\tuint16_t nr_copies;\n+\tuint16_t copy_idx;\n+\tuint16_t i;\n+\n+\trte_spinlock_lock(&dma_info->dma_lock);\n+\n+\t/**\n+\t * Since all memory is pinned and addresses should be valid,\n+\t * we don't check errors.\n+\t */\n+\tnr_copies = rte_dma_completed(dma_id, vchan, max_pkts, &last_idx, NULL);\n+\tif (nr_copies == 0) {\n+\t\tgoto out;\n+\t}\n+\n+\tcopy_idx = last_idx - nr_copies + 1;\n+\tfor (i = 0; i < nr_copies; i++) {\n+\t\tbool *flag;\n+\n+\t\tflag = dma_info->metadata[copy_idx & ring_mask];\n+\t\tif (flag) {\n+\t\t\t/**\n+\t\t\t * Mark the packet flag as received. The flag\n+\t\t\t * could belong to another virtqueue but write\n+\t\t\t * is atomic.\n+\t\t\t */\n+\t\t\t*flag = true;\n+\t\t\tdma_info->metadata[copy_idx & ring_mask] = NULL;\n+\t\t}\n+\t\tcopy_idx++;\n+\t}\n+\n+out:\n+\trte_spinlock_unlock(&dma_info->dma_lock);\n+\treturn nr_copies;\n+}\n+\n static inline void\n do_data_copy_enqueue(struct virtio_net *dev, struct vhost_virtqueue *vq)\n {\n@@ -1449,9 +1555,9 @@ store_dma_desc_info_packed(struct vring_used_elem_packed *s_ring,\n }\n \n static __rte_noinline uint32_t\n-virtio_dev_rx_async_submit_split(struct virtio_net *dev,\n-\tstruct vhost_virtqueue *vq, uint16_t queue_id,\n-\tstruct rte_mbuf **pkts, uint32_t count)\n+virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue *vq,\n+\t\tuint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,\n+\t\tint16_t dma_id, uint16_t vchan)\n {\n \tstruct buf_vector buf_vec[BUF_VECTOR_MAX];\n \tuint32_t pkt_idx = 0;\n@@ -1503,17 +1609,16 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev,\n \tif (unlikely(pkt_idx == 0))\n \t\treturn 0;\n \n-\tn_xfer = async->ops.transfer_data(dev->vid, queue_id, async->iov_iter, 0, pkt_idx);\n-\tif (unlikely(n_xfer < 0)) {\n-\t\tVHOST_LOG_DATA(ERR, \"(%d) %s: failed to transfer data for queue id %d.\\n\",\n-\t\t\t\tdev->vid, __func__, queue_id);\n-\t\tn_xfer = 0;\n-\t}\n+\tn_xfer = vhost_async_dma_transfer(vq, dma_id, vchan, async->pkts_idx, async->iov_iter,\n+\t\t\tpkt_idx);\n \n \tpkt_err = pkt_idx - n_xfer;\n \tif (unlikely(pkt_err)) {\n \t\tuint16_t num_descs = 0;\n \n+\t\tVHOST_LOG_DATA(DEBUG, \"(%d) %s: failed to transfer %u packets for queue %u.\\n\",\n+\t\t\t\tdev->vid, __func__, pkt_err, queue_id);\n+\n \t\t/* update number of completed packets */\n \t\tpkt_idx = n_xfer;\n \n@@ -1656,13 +1761,13 @@ dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx,\n }\n \n static __rte_noinline uint32_t\n-virtio_dev_rx_async_submit_packed(struct virtio_net *dev,\n-\tstruct vhost_virtqueue *vq, uint16_t queue_id,\n-\tstruct rte_mbuf **pkts, uint32_t count)\n+virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,\n+\t\tuint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,\n+\t\tint16_t dma_id, uint16_t vchan)\n {\n \tuint32_t pkt_idx = 0;\n \tuint32_t remained = count;\n-\tint32_t n_xfer;\n+\tuint16_t n_xfer;\n \tuint16_t num_buffers;\n \tuint16_t num_descs;\n \n@@ -1670,6 +1775,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev,\n \tstruct async_inflight_info *pkts_info = async->pkts_info;\n \tuint32_t pkt_err = 0;\n \tuint16_t slot_idx = 0;\n+\tuint16_t head_idx = async->pkts_idx % vq->size;\n \n \tdo {\n \t\trte_prefetch0(&vq->desc_packed[vq->last_avail_idx]);\n@@ -1694,19 +1800,17 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev,\n \tif (unlikely(pkt_idx == 0))\n \t\treturn 0;\n \n-\tn_xfer = async->ops.transfer_data(dev->vid, queue_id, async->iov_iter, 0, pkt_idx);\n-\tif (unlikely(n_xfer < 0)) {\n-\t\tVHOST_LOG_DATA(ERR, \"(%d) %s: failed to transfer data for queue id %d.\\n\",\n-\t\t\t\tdev->vid, __func__, queue_id);\n-\t\tn_xfer = 0;\n-\t}\n-\n-\tpkt_err = pkt_idx - n_xfer;\n+\tn_xfer = vhost_async_dma_transfer(vq, dma_id, vchan, head_idx,\n+\t\t\tasync->iov_iter, pkt_idx);\n \n \tasync_iter_reset(async);\n \n-\tif (unlikely(pkt_err))\n+\tpkt_err = pkt_idx - n_xfer;\n+\tif (unlikely(pkt_err)) {\n+\t\tVHOST_LOG_DATA(DEBUG, \"(%d) %s: failed to transfer %u packets for queue %u.\\n\",\n+\t\t\t\tdev->vid, __func__, pkt_err, queue_id);\n \t\tdma_error_handler_packed(vq, slot_idx, pkt_err, &pkt_idx);\n+\t}\n \n \tif (likely(vq->shadow_used_idx)) {\n \t\t/* keep used descriptors. */\n@@ -1826,28 +1930,37 @@ write_back_completed_descs_packed(struct vhost_virtqueue *vq,\n \n static __rte_always_inline uint16_t\n vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id,\n-\t\tstruct rte_mbuf **pkts, uint16_t count)\n+\t\tstruct rte_mbuf **pkts, uint16_t count, int16_t dma_id,\n+\t\tuint16_t vchan)\n {\n \tstruct vhost_virtqueue *vq = dev->virtqueue[queue_id];\n \tstruct vhost_async *async = vq->async;\n \tstruct async_inflight_info *pkts_info = async->pkts_info;\n-\tint32_t n_cpl;\n+\tuint16_t nr_cpl_pkts = 0;\n \tuint16_t n_descs = 0, n_buffers = 0;\n \tuint16_t start_idx, from, i;\n \n-\tn_cpl = async->ops.check_completed_copies(dev->vid, queue_id, 0, count);\n-\tif (unlikely(n_cpl < 0)) {\n-\t\tVHOST_LOG_DATA(ERR, \"(%d) %s: failed to check completed copies for queue id %d.\\n\",\n-\t\t\t\tdev->vid, __func__, queue_id);\n-\t\treturn 0;\n-\t}\n-\n-\tif (n_cpl == 0)\n-\t\treturn 0;\n+\t/* Check completed copies for the given DMA vChannel */\n+\tvhost_async_dma_check_completed(dma_id, vchan, count);\n \n \tstart_idx = async_get_first_inflight_pkt_idx(vq);\n \n-\tfor (i = 0; i < n_cpl; i++) {\n+\t/**\n+\t * Calculate the number of copy completed packets.\n+\t * Note that there may be completed packets even if\n+\t * no copies are reported done by the given DMA vChannel,\n+\t * as DMA vChannels could be shared by other threads.\n+\t */\n+\tfrom = start_idx;\n+\twhile (vq->async->pkts_cmpl_flag[from] && count--) {\n+\t\tvq->async->pkts_cmpl_flag[from] = false;\n+\t\tfrom++;\n+\t\tif (from >= vq->size)\n+\t\t\tfrom -= vq->size;\n+\t\tnr_cpl_pkts++;\n+\t}\n+\n+\tfor (i = 0; i < nr_cpl_pkts; i++) {\n \t\tfrom = (start_idx + i) % vq->size;\n \t\t/* Only used with packed ring */\n \t\tn_buffers += pkts_info[from].nr_buffers;\n@@ -1856,7 +1969,7 @@ vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id,\n \t\tpkts[i] = pkts_info[from].mbuf;\n \t}\n \n-\tasync->pkts_inflight_n -= n_cpl;\n+\tasync->pkts_inflight_n -= nr_cpl_pkts;\n \n \tif (likely(vq->enabled && vq->access_ok)) {\n \t\tif (vq_is_packed(dev)) {\n@@ -1877,12 +1990,13 @@ vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id,\n \t\t}\n \t}\n \n-\treturn n_cpl;\n+\treturn nr_cpl_pkts;\n }\n \n uint16_t\n rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,\n-\t\tstruct rte_mbuf **pkts, uint16_t count)\n+\t\tstruct rte_mbuf **pkts, uint16_t count, int16_t dma_id,\n+\t\tuint16_t vchan)\n {\n \tstruct virtio_net *dev = get_device(vid);\n \tstruct vhost_virtqueue *vq;\n@@ -1908,7 +2022,7 @@ rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,\n \n \trte_spinlock_lock(&vq->access_lock);\n \n-\tn_pkts_cpl = vhost_poll_enqueue_completed(dev, queue_id, pkts, count);\n+\tn_pkts_cpl = vhost_poll_enqueue_completed(dev, queue_id, pkts, count, dma_id, vchan);\n \n \trte_spinlock_unlock(&vq->access_lock);\n \n@@ -1917,7 +2031,8 @@ rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,\n \n uint16_t\n rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,\n-\t\tstruct rte_mbuf **pkts, uint16_t count)\n+\t\tstruct rte_mbuf **pkts, uint16_t count, int16_t dma_id,\n+\t\tuint16_t vchan)\n {\n \tstruct virtio_net *dev = get_device(vid);\n \tstruct vhost_virtqueue *vq;\n@@ -1941,14 +2056,14 @@ rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,\n \t\treturn 0;\n \t}\n \n-\tn_pkts_cpl = vhost_poll_enqueue_completed(dev, queue_id, pkts, count);\n+\tn_pkts_cpl = vhost_poll_enqueue_completed(dev, queue_id, pkts, count, dma_id, vchan);\n \n \treturn n_pkts_cpl;\n }\n \n static __rte_always_inline uint32_t\n virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id,\n-\tstruct rte_mbuf **pkts, uint32_t count)\n+\tstruct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan)\n {\n \tstruct vhost_virtqueue *vq;\n \tuint32_t nb_tx = 0;\n@@ -1980,10 +2095,10 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id,\n \n \tif (vq_is_packed(dev))\n \t\tnb_tx = virtio_dev_rx_async_submit_packed(dev, vq, queue_id,\n-\t\t\t\tpkts, count);\n+\t\t\t\tpkts, count, dma_id, vchan);\n \telse\n \t\tnb_tx = virtio_dev_rx_async_submit_split(dev, vq, queue_id,\n-\t\t\t\tpkts, count);\n+\t\t\t\tpkts, count, dma_id, vchan);\n \n out:\n \tif (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))\n@@ -1997,7 +2112,8 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id,\n \n uint16_t\n rte_vhost_submit_enqueue_burst(int vid, uint16_t queue_id,\n-\t\tstruct rte_mbuf **pkts, uint16_t count)\n+\t\tstruct rte_mbuf **pkts, uint16_t count, int16_t dma_id,\n+\t\tuint16_t vchan)\n {\n \tstruct virtio_net *dev = get_device(vid);\n \n@@ -2011,7 +2127,7 @@ rte_vhost_submit_enqueue_burst(int vid, uint16_t queue_id,\n \t\treturn 0;\n \t}\n \n-\treturn virtio_dev_rx_async_submit(dev, queue_id, pkts, count);\n+\treturn virtio_dev_rx_async_submit(dev, queue_id, pkts, count, dma_id, vchan);\n }\n \n static inline bool\n",
    "prefixes": [
        "v1",
        "1/1"
    ]
}