From patchwork Fri Apr 19 06:43:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anoob Joseph X-Patchwork-Id: 139539 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9022643EAC; Fri, 19 Apr 2024 08:44:10 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5D45040698; Fri, 19 Apr 2024 08:43:45 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 1DD5940697 for ; Fri, 19 Apr 2024 08:43:44 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.24/8.17.1.24) with ESMTP id 43ILYmCp010590; Thu, 18 Apr 2024 23:43:43 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s= pfpt0220; bh=/JYbzfDCKEv1ytqV/l4sKG+IeuvClHlA6BIkmawnX2s=; b=kvW WF6BVEZvZhh8bcOuMLSv5uoRnn/1mJOJIjEogu320AViiF7YlBtprLn+0oaFmQ/C p+SUp7zFL2hi4YjpsslGVTVMHcIY2jBpUX+Ea8m/uxRwGj63l7T1L1DPU04/hGoz /XAnCBFa4dzlDHpGDBB2f3bYne0YsRhv/qPTC4qTaPd5iKzMGsRRtyyHNH+UbKvU mJNZfCO63OKOGa6jItrQKfgP4FKJtkBlK8iTKcezU5FOMvK9XHvOPT4XiEPH0BxR OI9EWucOqvQWrsoxbp5ZmWbDRnvWTWwPlusKuNnj/ZNzt5JDKs8YEczgg/6mnSk+ mONbcr16GpjipxzKL0g== Received: from dc5-exch05.marvell.com ([199.233.59.128]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3xjhecq8u9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 18 Apr 2024 23:43:43 -0700 (PDT) Received: from DC5-EXCH05.marvell.com (10.69.176.209) by DC5-EXCH05.marvell.com (10.69.176.209) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Thu, 18 Apr 2024 23:43:41 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH05.marvell.com (10.69.176.209) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Thu, 18 Apr 2024 23:43:41 -0700 Received: from BG-LT92004.corp.innovium.com (BG-LT92004.marvell.com [10.28.163.189]) by maili.marvell.com (Postfix) with ESMTP id 4441A5B6933; Thu, 18 Apr 2024 23:43:39 -0700 (PDT) From: Anoob Joseph To: Chengwen Feng , Kevin Laatz , Bruce Richardson , "Jerin Jacob" , Thomas Monjalon CC: Vidya Sagar Velumuri , Gowrishankar Muthukrishnan , Subject: [PATCH v3 6/7] dma/odm: add copy and copy sg ops Date: Fri, 19 Apr 2024 12:13:18 +0530 Message-ID: <20240419064319.149-7-anoobj@marvell.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240419064319.149-1-anoobj@marvell.com> References: <20240417072708.322-1-anoobj@marvell.com> <20240419064319.149-1-anoobj@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: hr2P28_a-7PasZGa-vMpHyU-NEMspRSe X-Proofpoint-GUID: hr2P28_a-7PasZGa-vMpHyU-NEMspRSe X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-04-19_04,2024-04-17_01,2023-05-22_02 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Vidya Sagar Velumuri Add ODM copy and copy SG ops. Signed-off-by: Anoob Joseph Signed-off-by: Gowrishankar Muthukrishnan Signed-off-by: Vidya Sagar Velumuri --- drivers/dma/odm/odm_dmadev.c | 236 +++++++++++++++++++++++++++++++++++ 1 file changed, 236 insertions(+) diff --git a/drivers/dma/odm/odm_dmadev.c b/drivers/dma/odm/odm_dmadev.c index 13b2588246..b21be83a89 100644 --- a/drivers/dma/odm/odm_dmadev.c +++ b/drivers/dma/odm/odm_dmadev.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include "odm.h" @@ -87,6 +88,238 @@ odm_dmadev_close(struct rte_dma_dev *dev) return 0; } +static int +odm_dmadev_copy(void *dev_private, uint16_t vchan, rte_iova_t src, rte_iova_t dst, uint32_t length, + uint64_t flags) +{ + uint16_t pending_submit_len, pending_submit_cnt, iring_sz_available, iring_head; + const int num_words = ODM_IRING_ENTRY_SIZE_MIN; + struct odm_dev *odm = dev_private; + uint64_t *iring_head_ptr; + struct odm_queue *vq; + uint64_t h; + + const union odm_instr_hdr_s hdr = { + .s.ct = ODM_HDR_CT_CW_NC, + .s.xtype = ODM_XTYPE_INTERNAL, + .s.nfst = 1, + .s.nlst = 1, + }; + + vq = &odm->vq[vchan]; + + h = length; + h |= ((uint64_t)length << 32); + + const uint16_t max_iring_words = vq->iring_max_words; + + iring_sz_available = vq->iring_sz_available; + pending_submit_len = vq->pending_submit_len; + pending_submit_cnt = vq->pending_submit_cnt; + iring_head_ptr = vq->iring_mz->addr; + iring_head = vq->iring_head; + + if (iring_sz_available < num_words) + return -ENOSPC; + + if ((iring_head + num_words) >= max_iring_words) { + + iring_head_ptr[iring_head] = hdr.u; + iring_head = (iring_head + 1) % max_iring_words; + + iring_head_ptr[iring_head] = h; + iring_head = (iring_head + 1) % max_iring_words; + + iring_head_ptr[iring_head] = src; + iring_head = (iring_head + 1) % max_iring_words; + + iring_head_ptr[iring_head] = dst; + iring_head = (iring_head + 1) % max_iring_words; + } else { + iring_head_ptr[iring_head++] = hdr.u; + iring_head_ptr[iring_head++] = h; + iring_head_ptr[iring_head++] = src; + iring_head_ptr[iring_head++] = dst; + } + + pending_submit_len += num_words; + + if (flags & RTE_DMA_OP_FLAG_SUBMIT) { + rte_wmb(); + odm_write64(pending_submit_len, odm->rbase + ODM_VDMA_DBELL(vchan)); + vq->stats.submitted += pending_submit_cnt + 1; + vq->pending_submit_len = 0; + vq->pending_submit_cnt = 0; + } else { + vq->pending_submit_len = pending_submit_len; + vq->pending_submit_cnt++; + } + + vq->iring_head = iring_head; + + vq->iring_sz_available = iring_sz_available - num_words; + + /* No extra space to save. Skip entry in extra space ring. */ + vq->ins_ring_head = (vq->ins_ring_head + 1) % vq->cring_max_entry; + + return vq->desc_idx++; +} + +static inline void +odm_dmadev_fill_sg(uint64_t *cmd, const struct rte_dma_sge *src, const struct rte_dma_sge *dst, + uint16_t nb_src, uint16_t nb_dst, union odm_instr_hdr_s *hdr) +{ + int i = 0, j = 0; + uint64_t h = 0; + + cmd[j++] = hdr->u; + /* When nb_src is even */ + if (!(nb_src & 0x1)) { + /* Fill the iring with src pointers */ + for (i = 1; i < nb_src; i += 2) { + h = ((uint64_t)src[i].length << 32) | src[i - 1].length; + cmd[j++] = h; + cmd[j++] = src[i - 1].addr; + cmd[j++] = src[i].addr; + } + + /* Fill the iring with dst pointers */ + for (i = 1; i < nb_dst; i += 2) { + h = ((uint64_t)dst[i].length << 32) | dst[i - 1].length; + cmd[j++] = h; + cmd[j++] = dst[i - 1].addr; + cmd[j++] = dst[i].addr; + } + + /* Handle the last dst pointer when nb_dst is odd */ + if (nb_dst & 0x1) { + h = dst[nb_dst - 1].length; + cmd[j++] = h; + cmd[j++] = dst[nb_dst - 1].addr; + cmd[j++] = 0; + } + } else { + /* When nb_src is odd */ + + /* Fill the iring with src pointers */ + for (i = 1; i < nb_src; i += 2) { + h = ((uint64_t)src[i].length << 32) | src[i - 1].length; + cmd[j++] = h; + cmd[j++] = src[i - 1].addr; + cmd[j++] = src[i].addr; + } + + /* Handle the last src pointer */ + h = ((uint64_t)dst[0].length << 32) | src[nb_src - 1].length; + cmd[j++] = h; + cmd[j++] = src[nb_src - 1].addr; + cmd[j++] = dst[0].addr; + + /* Fill the iring with dst pointers */ + for (i = 2; i < nb_dst; i += 2) { + h = ((uint64_t)dst[i].length << 32) | dst[i - 1].length; + cmd[j++] = h; + cmd[j++] = dst[i - 1].addr; + cmd[j++] = dst[i].addr; + } + + /* Handle the last dst pointer when nb_dst is even */ + if (!(nb_dst & 0x1)) { + h = dst[nb_dst - 1].length; + cmd[j++] = h; + cmd[j++] = dst[nb_dst - 1].addr; + cmd[j++] = 0; + } + } +} + +static int +odm_dmadev_copy_sg(void *dev_private, uint16_t vchan, const struct rte_dma_sge *src, + const struct rte_dma_sge *dst, uint16_t nb_src, uint16_t nb_dst, uint64_t flags) +{ + uint16_t pending_submit_len, pending_submit_cnt, iring_head, ins_ring_head; + uint16_t iring_sz_available, i, nb, num_words; + uint64_t cmd[ODM_IRING_ENTRY_SIZE_MAX]; + struct odm_dev *odm = dev_private; + uint32_t s_sz = 0, d_sz = 0; + uint64_t *iring_head_ptr; + struct odm_queue *vq; + union odm_instr_hdr_s hdr = { + .s.ct = ODM_HDR_CT_CW_NC, + .s.xtype = ODM_XTYPE_INTERNAL, + }; + + vq = &odm->vq[vchan]; + const uint16_t max_iring_words = vq->iring_max_words; + + iring_head_ptr = vq->iring_mz->addr; + iring_head = vq->iring_head; + iring_sz_available = vq->iring_sz_available; + ins_ring_head = vq->ins_ring_head; + pending_submit_len = vq->pending_submit_len; + pending_submit_cnt = vq->pending_submit_cnt; + + if (unlikely(nb_src > 4 || nb_dst > 4)) + return -EINVAL; + + for (i = 0; i < nb_src; i++) + s_sz += src[i].length; + + for (i = 0; i < nb_dst; i++) + d_sz += dst[i].length; + + if (s_sz != d_sz) + return -EINVAL; + + nb = nb_src + nb_dst; + hdr.s.nfst = nb_src; + hdr.s.nlst = nb_dst; + num_words = 1 + 3 * (nb / 2 + (nb & 0x1)); + + if (iring_sz_available < num_words) + return -ENOSPC; + + if ((iring_head + num_words) >= max_iring_words) { + uint16_t words_avail = max_iring_words - iring_head; + uint16_t words_pend = num_words - words_avail; + + if (unlikely(words_avail + words_pend > ODM_IRING_ENTRY_SIZE_MAX)) + return -ENOSPC; + + odm_dmadev_fill_sg(cmd, src, dst, nb_src, nb_dst, &hdr); + rte_memcpy((void *)&iring_head_ptr[iring_head], (void *)cmd, words_avail * 8); + rte_memcpy((void *)iring_head_ptr, (void *)&cmd[words_avail], words_pend * 8); + iring_head = words_pend; + } else { + odm_dmadev_fill_sg(&iring_head_ptr[iring_head], src, dst, nb_src, nb_dst, &hdr); + iring_head += num_words; + } + + pending_submit_len += num_words; + + if (flags & RTE_DMA_OP_FLAG_SUBMIT) { + rte_wmb(); + odm_write64(pending_submit_len, odm->rbase + ODM_VDMA_DBELL(vchan)); + vq->stats.submitted += pending_submit_cnt + 1; + vq->pending_submit_len = 0; + vq->pending_submit_cnt = 0; + } else { + vq->pending_submit_len = pending_submit_len; + vq->pending_submit_cnt++; + } + + vq->iring_head = iring_head; + + vq->iring_sz_available = iring_sz_available - num_words; + + /* Save extra space used for the instruction. */ + vq->extra_ins_sz[ins_ring_head] = num_words - 4; + + vq->ins_ring_head = (ins_ring_head + 1) % vq->cring_max_entry; + + return vq->desc_idx++; +} + static int odm_stats_get(const struct rte_dma_dev *dev, uint16_t vchan, struct rte_dma_stats *rte_stats, uint32_t size) @@ -184,6 +417,9 @@ odm_dmadev_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_dev dmadev->fp_obj->dev_private = odm; dmadev->dev_ops = &odm_dmadev_ops; + dmadev->fp_obj->copy = odm_dmadev_copy; + dmadev->fp_obj->copy_sg = odm_dmadev_copy_sg; + odm->pci_dev = pci_dev; rc = odm_dev_init(odm);