From patchwork Fri Dec 16 07:38:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Jiang, Cheng1" X-Patchwork-Id: 120969 X-Patchwork-Delegate: maxime.coquelin@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B7E69A0542; Fri, 16 Dec 2022 09:25:03 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AB46C42D0B; Fri, 16 Dec 2022 09:25:03 +0100 (CET) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id 7E8B140685 for ; Fri, 16 Dec 2022 09:25:01 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1671179101; x=1702715101; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=q81ewjW6FncRZY78Hc3PvEVRytBnhct6NiLnFS0AXVY=; b=Og2r13+NFNQ3XuYtmG+USNw5haQge4ntkNeyI1e2aZ0JObj7EUFnbeZe BMzc+SIS7tttKoA4NdqlUOCoETtCaDO1XNGRvcCpmeMz2N9nyHbVCZmA6 2ugIUdMP688PcNBvry7vpTgd2LQNdq44DPSrS1jlShQQKOVAJrX8Op7r8 iSnheHAf1cIaiAz76v+8Ap/DqrDzt+u01GMP+RoGKDnQ+AwVR4qaUiU3n j359ioum2L5Cg1pZxe1fm3k19lSFDVJewSUZy4TJn4TysTuXUhe/4FW1b ledQ7QYxWWRal6MOCDrBNvU/ra0IQyxroghyFT36cdqxPRxFmrvAEIJ56 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10562"; a="381137613" X-IronPort-AV: E=Sophos;i="5.96,249,1665471600"; d="scan'208";a="381137613" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Dec 2022 00:25:00 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10562"; a="682221164" X-IronPort-AV: E=Sophos;i="5.96,249,1665471600"; d="scan'208";a="682221164" Received: from dpdk_jiangcheng.sh.intel.com ([10.67.118.237]) by orsmga001.jf.intel.com with ESMTP; 16 Dec 2022 00:24:57 -0800 From: Cheng Jiang To: maxime.coquelin@redhat.com, chenbo.xia@intel.com Cc: dev@dpdk.org, jiayu.hu@intel.com, xuan.ding@intel.com, wenwux.ma@intel.com, yuanx.wang@intel.com, yvonnex.yang@intel.com, xingguang.he@intel.com, Cheng Jiang Subject: [RFC 2/3] vhost: add batch enqueue in async vhost packed ring Date: Fri, 16 Dec 2022 07:38:43 +0000 Message-Id: <20221216073844.24448-3-cheng1.jiang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221216073844.24448-1-cheng1.jiang@intel.com> References: <20221216073844.24448-1-cheng1.jiang@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add batch enqueue function in asynchronous vhost packed ring to improve the performance. Chained mbufs are not supported, it will be handled in single enqueue function. Signed-off-by: Cheng Jiang --- lib/vhost/virtio_net.c | 159 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 157 insertions(+), 2 deletions(-) diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index 22f97d4f77..b87405ba54 100644 --- a/lib/vhost/virtio_net.c +++ b/lib/vhost/virtio_net.c @@ -432,6 +432,24 @@ vhost_flush_enqueue_batch_packed(struct virtio_net *dev, vq_inc_last_used_packed(vq, PACKED_BATCH_SIZE); } +static __rte_always_inline void +vhost_async_shadow_enqueue_packed_batch(struct vhost_virtqueue *vq, + uint64_t *lens, + uint16_t *ids) +{ + uint16_t i; + struct vhost_async *async = vq->async; + + vhost_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) { + async->buffers_packed[async->buffer_idx_packed].id = ids[i]; + async->buffers_packed[async->buffer_idx_packed].len = lens[i]; + async->buffers_packed[async->buffer_idx_packed].count = 1; + async->buffer_idx_packed++; + if (async->buffer_idx_packed >= vq->size) + async->buffer_idx_packed -= vq->size; + } +} + static __rte_always_inline void vhost_shadow_dequeue_batch_packed_inorder(struct vhost_virtqueue *vq, uint16_t id) @@ -1451,6 +1469,58 @@ virtio_dev_rx_sync_batch_check(struct virtio_net *dev, return 0; } +static __rte_always_inline int +virtio_dev_rx_async_batch_check(struct vhost_virtqueue *vq, + struct rte_mbuf **pkts, + uint64_t *desc_addrs, + uint64_t *lens, + int16_t dma_id, + uint16_t vchan_id) +{ + bool wrap_counter = vq->avail_wrap_counter; + struct vring_packed_desc *descs = vq->desc_packed; + uint16_t avail_idx = vq->last_avail_idx; + uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf); + uint16_t i; + + if (unlikely(avail_idx & PACKED_BATCH_MASK)) + return -1; + + if (unlikely((avail_idx + PACKED_BATCH_SIZE) > vq->size)) + return -1; + + vhost_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) { + if (unlikely(pkts[i]->next != NULL)) + return -1; + if (unlikely(!desc_is_avail(&descs[avail_idx + i], + wrap_counter))) + return -1; + } + + vhost_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) + lens[i] = descs[avail_idx + i].len; + + vhost_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) { + if (unlikely(pkts[i]->pkt_len > (lens[i] - buf_offset))) + return -1; + } + + vhost_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) + desc_addrs[i] = descs[avail_idx + i].addr; + + vhost_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) { + if (unlikely(!desc_addrs[i])) + return -1; + if (unlikely(lens[i] != descs[avail_idx + i].len)) + return -1; + } + + if (rte_dma_burst_capacity(dma_id, vchan_id) < PACKED_BATCH_SIZE) + return -1; + + return 0; +} + static __rte_always_inline void virtio_dev_rx_batch_packed_copy(struct virtio_net *dev, struct vhost_virtqueue *vq, @@ -1850,6 +1920,78 @@ virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq, return 0; } +static __rte_always_inline void +virtio_dev_rx_async_packed_batch_enqueue(struct virtio_net *dev, + struct vhost_virtqueue *vq, + struct rte_mbuf **pkts, + uint64_t *desc_addrs, + uint64_t *lens) +{ + uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf); + struct virtio_net_hdr_mrg_rxbuf *hdrs[PACKED_BATCH_SIZE]; + struct vring_packed_desc *descs = vq->desc_packed; + struct vhost_async *async = vq->async; + uint16_t avail_idx = vq->last_avail_idx; + uint32_t mbuf_offset = 0; + uint16_t ids[PACKED_BATCH_SIZE]; + uint64_t mapped_len[PACKED_BATCH_SIZE]; + void *host_iova[PACKED_BATCH_SIZE]; + uintptr_t desc; + uint16_t i; + + vhost_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) { + rte_prefetch0((void *)(uintptr_t)desc_addrs[i]); + desc = vhost_iova_to_vva(dev, vq, desc_addrs[i], &lens[i], VHOST_ACCESS_RW); + hdrs[i] = (struct virtio_net_hdr_mrg_rxbuf *)(uintptr_t)desc; + lens[i] = pkts[i]->pkt_len + + sizeof(struct virtio_net_hdr_mrg_rxbuf); + } + + vhost_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) + virtio_enqueue_offload(pkts[i], &hdrs[i]->hdr); + + vq_inc_last_avail_packed(vq, PACKED_BATCH_SIZE); + + vhost_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) { + host_iova[i] = (void *)(uintptr_t)gpa_to_first_hpa(dev, + desc_addrs[i] + buf_offset, lens[i], &mapped_len[i]); + } + + vhost_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) { + async_iter_initialize(dev, async); + async_iter_add_iovec(dev, async, + (void *)(uintptr_t)rte_pktmbuf_iova_offset(pkts[i], mbuf_offset), + host_iova[i], + mapped_len[i]); + async->iter_idx++; + } + + vhost_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) + vhost_log_cache_write_iova(dev, vq, descs[avail_idx + i].addr, lens[i]); + + vhost_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) + ids[i] = descs[avail_idx + i].id; + + vhost_async_shadow_enqueue_packed_batch(vq, lens, ids); +} + +static __rte_always_inline int +virtio_dev_rx_async_packed_batch(struct virtio_net *dev, + struct vhost_virtqueue *vq, + struct rte_mbuf **pkts, + int16_t dma_id, uint16_t vchan_id) +{ + uint64_t desc_addrs[PACKED_BATCH_SIZE]; + uint64_t lens[PACKED_BATCH_SIZE]; + + if (virtio_dev_rx_async_batch_check(vq, pkts, desc_addrs, lens, dma_id, vchan_id) == -1) + return -1; + + virtio_dev_rx_async_packed_batch_enqueue(dev, vq, pkts, desc_addrs, lens); + + return 0; +} + static __rte_always_inline void dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx, uint32_t nr_err, uint32_t *pkt_idx) @@ -1885,7 +2027,6 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id) { uint32_t pkt_idx = 0; - uint32_t remained = count; uint16_t n_xfer; uint16_t num_buffers; uint16_t num_descs; @@ -1894,10 +2035,25 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue struct async_inflight_info *pkts_info = async->pkts_info; uint32_t pkt_err = 0; uint16_t slot_idx = 0; + uint16_t i; do { rte_prefetch0(&vq->desc_packed[vq->last_avail_idx]); + if (count - pkt_idx >= PACKED_BATCH_SIZE) { + if (!virtio_dev_rx_async_packed_batch(dev, vq, &pkts[pkt_idx], + dma_id, vchan_id)) { + for (i = 0; i < PACKED_BATCH_SIZE; i++) { + slot_idx = (async->pkts_idx + pkt_idx) % vq->size; + pkts_info[slot_idx].descs = 1; + pkts_info[slot_idx].nr_buffers = 1; + pkts_info[slot_idx].mbuf = pkts[pkt_idx]; + pkt_idx++; + } + continue; + } + } + num_buffers = 0; num_descs = 0; if (unlikely(virtio_dev_rx_async_packed(dev, vq, pkts[pkt_idx], @@ -1911,7 +2067,6 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue pkts_info[slot_idx].mbuf = pkts[pkt_idx]; pkt_idx++; - remained--; vq_inc_last_avail_packed(vq, num_descs); } while (pkt_idx < count);