From patchwork Fri Aug 5 18:50:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Radha Chintakuntla X-Patchwork-Id: 114675 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BC684A00C4; Fri, 5 Aug 2022 20:50:46 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 37FD642C0C; Fri, 5 Aug 2022 20:50:46 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 96DF2400D6 for ; Fri, 5 Aug 2022 20:50:44 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 275HFIHR028393; Fri, 5 Aug 2022 11:50:43 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : mime-version : content-type; s=pfpt0220; bh=Y8oWzmHps5oyiUiuejgml4ZmMJXtLThVZAxxYYrq66c=; b=kd2398j81SN/2Enh+gxIf0qwYFrPoG2BphSN8D1hprTQUhIEpIGmBBwrGdaNLPe4wD3P vwy2fcj6ekPAcUF8fKfn1jZBT8Z9kIFLpv/nzt8ANrfJizyTVTFi49UV1X96AwjdL9tV JCan1Rj7eJxA36GnHNoswfzeGXdvZge9LqxvXIvYBmb3kRdx3sdIm833z6mdD7gl6UIp HVSyDYxlefb7/X7bZXMewnVtu2RdwSi/Dpjy8pkHzoUXHUmDNZCDPZ8VPfuFD/9TUqsQ Ziu/6i7lPSklfSa3k+LgIpo6tVXH2y5QtiG0c6w516IEq9CjFDWcjUHwqefRP2WjCDnR JA== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3hrvtwtewt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Fri, 05 Aug 2022 11:50:43 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 5 Aug 2022 11:50:41 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Fri, 5 Aug 2022 11:50:41 -0700 Received: from rchintakuntla-lnx3.caveonetworks.com (unknown [10.111.140.81]) by maili.marvell.com (Postfix) with ESMTP id D4C693F70A3; Fri, 5 Aug 2022 11:50:39 -0700 (PDT) From: Radha Mohan Chintakuntla To: CC: , Radha Mohan Chintakuntla , Veerasenareddy Burru Subject: [PATCH] dma/cnxk: add support for CN10K DMA engine Date: Fri, 5 Aug 2022 11:50:34 -0700 Message-ID: <20220805185035.22084-1-radhac@marvell.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: GURoCvThbqhCcbVbYDgel9mzDLhdXbtU X-Proofpoint-GUID: GURoCvThbqhCcbVbYDgel9mzDLhdXbtU X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-08-05_09,2022-08-05_01,2022-06-22_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added support for CN10K SoC DMA engine to dmadev. Signed-off-by: Radha Mohan Chintakuntla Change-Id: Ib119b3ceab0810724820e9af1d29fdd3f95d5cc9 Reviewed-on: https://sj1git1.cavium.com/c/IP/SW/dataplane/dpdk/+/68103 Tested-by: sa_ip-toolkits-Jenkins Reviewed-by: Jerin Jacob Kollanukkaran --- drivers/dma/cnxk/cnxk_dmadev.c | 248 +++++++++++++++++++++++++++++---- drivers/dma/cnxk/cnxk_dmadev.h | 2 +- 2 files changed, 220 insertions(+), 30 deletions(-) diff --git a/drivers/dma/cnxk/cnxk_dmadev.c b/drivers/dma/cnxk/cnxk_dmadev.c index 2824c1b44f..a67bcba9d4 100644 --- a/drivers/dma/cnxk/cnxk_dmadev.c +++ b/drivers/dma/cnxk/cnxk_dmadev.c @@ -70,31 +70,31 @@ cnxk_dmadev_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan, RTE_SET_USED(vchan); RTE_SET_USED(conf_sz); - header->s.pt = DPI_HDR_PT_ZBW_CA; + header->cn9k.pt = DPI_HDR_PT_ZBW_CA; switch (conf->direction) { case RTE_DMA_DIR_DEV_TO_MEM: - header->s.xtype = DPI_XTYPE_INBOUND; - header->s.lport = conf->src_port.pcie.coreid; - header->s.fport = 0; - header->s.pvfe = 1; + header->cn9k.xtype = DPI_XTYPE_INBOUND; + header->cn9k.lport = conf->src_port.pcie.coreid; + header->cn9k.fport = 0; + header->cn9k.pvfe = 1; break; case RTE_DMA_DIR_MEM_TO_DEV: - header->s.xtype = DPI_XTYPE_OUTBOUND; - header->s.lport = 0; - header->s.fport = conf->dst_port.pcie.coreid; - header->s.pvfe = 1; + header->cn9k.xtype = DPI_XTYPE_OUTBOUND; + header->cn9k.lport = 0; + header->cn9k.fport = conf->dst_port.pcie.coreid; + header->cn9k.pvfe = 1; break; case RTE_DMA_DIR_MEM_TO_MEM: - header->s.xtype = DPI_XTYPE_INTERNAL_ONLY; - header->s.lport = 0; - header->s.fport = 0; - header->s.pvfe = 0; + header->cn9k.xtype = DPI_XTYPE_INTERNAL_ONLY; + header->cn9k.lport = 0; + header->cn9k.fport = 0; + header->cn9k.pvfe = 0; break; case RTE_DMA_DIR_DEV_TO_DEV: - header->s.xtype = DPI_XTYPE_EXTERNAL_ONLY; - header->s.lport = conf->src_port.pcie.coreid; - header->s.fport = conf->dst_port.pcie.coreid; + header->cn9k.xtype = DPI_XTYPE_EXTERNAL_ONLY; + header->cn9k.lport = conf->src_port.pcie.coreid; + header->cn9k.fport = conf->dst_port.pcie.coreid; }; for (i = 0; i < conf->nb_desc; i++) { @@ -103,6 +103,63 @@ cnxk_dmadev_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan, plt_err("Failed to allocate for comp_data"); return -ENOMEM; } + comp_data->cdata = DPI_REQ_CDATA; + dpivf->conf.c_desc.compl_ptr[i] = comp_data; + }; + dpivf->conf.c_desc.max_cnt = DPI_MAX_DESC; + dpivf->conf.c_desc.head = 0; + dpivf->conf.c_desc.tail = 0; + + return 0; +} + +static int +cn10k_dmadev_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan, + const struct rte_dma_vchan_conf *conf, + uint32_t conf_sz) +{ + struct cnxk_dpi_vf_s *dpivf = dev->fp_obj->dev_private; + struct cnxk_dpi_compl_s *comp_data; + union dpi_instr_hdr_s *header = &dpivf->conf.hdr; + int i; + + RTE_SET_USED(vchan); + RTE_SET_USED(conf_sz); + + header->cn10k.pt = DPI_HDR_PT_ZBW_CA; + + switch (conf->direction) { + case RTE_DMA_DIR_DEV_TO_MEM: + header->cn10k.xtype = DPI_XTYPE_INBOUND; + header->cn10k.lport = conf->src_port.pcie.coreid; + header->cn10k.fport = 0; + header->cn10k.pvfe = 1; + break; + case RTE_DMA_DIR_MEM_TO_DEV: + header->cn10k.xtype = DPI_XTYPE_OUTBOUND; + header->cn10k.lport = 0; + header->cn10k.fport = conf->dst_port.pcie.coreid; + header->cn10k.pvfe = 1; + break; + case RTE_DMA_DIR_MEM_TO_MEM: + header->cn10k.xtype = DPI_XTYPE_INTERNAL_ONLY; + header->cn10k.lport = 0; + header->cn10k.fport = 0; + header->cn10k.pvfe = 0; + break; + case RTE_DMA_DIR_DEV_TO_DEV: + header->cn10k.xtype = DPI_XTYPE_EXTERNAL_ONLY; + header->cn10k.lport = conf->src_port.pcie.coreid; + header->cn10k.fport = conf->dst_port.pcie.coreid; + }; + + for (i = 0; i < conf->nb_desc; i++) { + comp_data = rte_zmalloc(NULL, sizeof(*comp_data), 0); + if (comp_data == NULL) { + plt_err("Failed to allocate for comp_data"); + return -ENOMEM; + } + comp_data->cdata = DPI_REQ_CDATA; dpivf->conf.c_desc.compl_ptr[i] = comp_data; }; dpivf->conf.c_desc.max_cnt = DPI_MAX_DESC; @@ -237,17 +294,17 @@ cnxk_dmadev_copy(void *dev_private, uint16_t vchan, rte_iova_t src, comp_ptr = dpivf->conf.c_desc.compl_ptr[dpivf->conf.c_desc.tail]; comp_ptr->cdata = DPI_REQ_CDATA; - header->s.ptr = (uint64_t)comp_ptr; + header->cn9k.ptr = (uint64_t)comp_ptr; STRM_INC(dpivf->conf.c_desc); - header->s.nfst = 1; - header->s.nlst = 1; + header->cn9k.nfst = 1; + header->cn9k.nlst = 1; /* * For inbound case, src pointers are last pointers. * For all other cases, src pointers are first pointers. */ - if (header->s.xtype == DPI_XTYPE_INBOUND) { + if (header->cn9k.xtype == DPI_XTYPE_INBOUND) { fptr = dst; lptr = src; } else { @@ -296,21 +353,21 @@ cnxk_dmadev_copy_sg(void *dev_private, uint16_t vchan, comp_ptr = dpivf->conf.c_desc.compl_ptr[dpivf->conf.c_desc.tail]; comp_ptr->cdata = DPI_REQ_CDATA; - header->s.ptr = (uint64_t)comp_ptr; + header->cn9k.ptr = (uint64_t)comp_ptr; STRM_INC(dpivf->conf.c_desc); /* * For inbound case, src pointers are last pointers. * For all other cases, src pointers are first pointers. */ - if (header->s.xtype == DPI_XTYPE_INBOUND) { - header->s.nfst = nb_dst & 0xf; - header->s.nlst = nb_src & 0xf; + if (header->cn9k.xtype == DPI_XTYPE_INBOUND) { + header->cn9k.nfst = nb_dst & 0xf; + header->cn9k.nlst = nb_src & 0xf; fptr = &dst[0]; lptr = &src[0]; } else { - header->s.nfst = nb_src & 0xf; - header->s.nlst = nb_dst & 0xf; + header->cn9k.nfst = nb_src & 0xf; + header->cn9k.nlst = nb_dst & 0xf; fptr = &src[0]; lptr = &dst[0]; } @@ -319,13 +376,13 @@ cnxk_dmadev_copy_sg(void *dev_private, uint16_t vchan, dpivf->cmd[1] = header->u[1]; dpivf->cmd[2] = header->u[2]; num_words += 4; - for (i = 0; i < header->s.nfst; i++) { + for (i = 0; i < header->cn9k.nfst; i++) { dpivf->cmd[num_words++] = (uint64_t)fptr->length; dpivf->cmd[num_words++] = fptr->addr; fptr++; } - for (i = 0; i < header->s.nlst; i++) { + for (i = 0; i < header->cn9k.nlst; i++) { dpivf->cmd[num_words++] = (uint64_t)lptr->length; dpivf->cmd[num_words++] = lptr->addr; lptr++; @@ -342,9 +399,113 @@ cnxk_dmadev_copy_sg(void *dev_private, uint16_t vchan, dpivf->num_words += num_words; } + return (rc < 0) ? rc : dpivf->desc_idx++; +} + +static int +cn10k_dmadev_copy(void *dev_private, uint16_t vchan, rte_iova_t src, + rte_iova_t dst, uint32_t length, uint64_t flags) +{ + struct cnxk_dpi_vf_s *dpivf = dev_private; + union dpi_instr_hdr_s *header = &dpivf->conf.hdr; + struct cnxk_dpi_compl_s *comp_ptr; + rte_iova_t fptr, lptr; + int num_words = 0; + int rc; + + RTE_SET_USED(vchan); + + comp_ptr = dpivf->conf.c_desc.compl_ptr[dpivf->conf.c_desc.tail]; + comp_ptr->cdata = DPI_REQ_CDATA; + header->cn10k.ptr = (uint64_t)comp_ptr; + STRM_INC(dpivf->conf.c_desc); + + header->cn10k.nfst = 1; + header->cn10k.nlst = 1; + + fptr = src; + lptr = dst; + + dpivf->cmd[0] = header->u[0]; + dpivf->cmd[1] = header->u[1]; + dpivf->cmd[2] = header->u[2]; + /* word3 is always 0 */ + num_words += 4; + dpivf->cmd[num_words++] = length; + dpivf->cmd[num_words++] = fptr; + dpivf->cmd[num_words++] = length; + dpivf->cmd[num_words++] = lptr; + + rc = __dpi_queue_write(&dpivf->rdpi, dpivf->cmd, num_words); + if (!rc) { + if (flags & RTE_DMA_OP_FLAG_SUBMIT) { + rte_wmb(); + plt_write64(num_words, + dpivf->rdpi.rbase + DPI_VDMA_DBELL); + dpivf->stats.submitted++; + } + dpivf->num_words += num_words; + } + return dpivf->desc_idx++; } +static int +cn10k_dmadev_copy_sg(void *dev_private, uint16_t vchan, + const struct rte_dma_sge *src, + const struct rte_dma_sge *dst, uint16_t nb_src, + uint16_t nb_dst, uint64_t flags) +{ + struct cnxk_dpi_vf_s *dpivf = dev_private; + union dpi_instr_hdr_s *header = &dpivf->conf.hdr; + const struct rte_dma_sge *fptr, *lptr; + struct cnxk_dpi_compl_s *comp_ptr; + int num_words = 0; + int i, rc; + + RTE_SET_USED(vchan); + + comp_ptr = dpivf->conf.c_desc.compl_ptr[dpivf->conf.c_desc.tail]; + comp_ptr->cdata = DPI_REQ_CDATA; + header->cn10k.ptr = (uint64_t)comp_ptr; + STRM_INC(dpivf->conf.c_desc); + + header->cn10k.nfst = nb_src & 0xf; + header->cn10k.nlst = nb_dst & 0xf; + fptr = &src[0]; + lptr = &dst[0]; + + dpivf->cmd[0] = header->u[0]; + dpivf->cmd[1] = header->u[1]; + dpivf->cmd[2] = header->u[2]; + num_words += 4; + + for (i = 0; i < header->cn10k.nfst; i++) { + dpivf->cmd[num_words++] = (uint64_t)fptr->length; + dpivf->cmd[num_words++] = fptr->addr; + fptr++; + } + + for (i = 0; i < header->cn10k.nlst; i++) { + dpivf->cmd[num_words++] = (uint64_t)lptr->length; + dpivf->cmd[num_words++] = lptr->addr; + lptr++; + } + + rc = __dpi_queue_write(&dpivf->rdpi, dpivf->cmd, num_words); + if (!rc) { + if (flags & RTE_DMA_OP_FLAG_SUBMIT) { + rte_wmb(); + plt_write64(num_words, + dpivf->rdpi.rbase + DPI_VDMA_DBELL); + dpivf->stats.submitted += nb_src; + } + dpivf->num_words += num_words; + } + + return (rc < 0) ? rc : dpivf->desc_idx++; +} + static uint16_t cnxk_dmadev_completed(void *dev_private, uint16_t vchan, const uint16_t nb_cpls, uint16_t *last_idx, bool *has_error) @@ -353,11 +514,17 @@ cnxk_dmadev_completed(void *dev_private, uint16_t vchan, const uint16_t nb_cpls, int cnt; RTE_SET_USED(vchan); + + if (dpivf->stats.submitted == dpivf->stats.completed) + return 0; + for (cnt = 0; cnt < nb_cpls; cnt++) { struct cnxk_dpi_compl_s *comp_ptr = dpivf->conf.c_desc.compl_ptr[cnt]; if (comp_ptr->cdata) { + if (comp_ptr->cdata == DPI_REQ_CDATA) + break; *has_error = 1; dpivf->stats.errors++; break; @@ -385,8 +552,12 @@ cnxk_dmadev_completed_status(void *dev_private, uint16_t vchan, struct cnxk_dpi_compl_s *comp_ptr = dpivf->conf.c_desc.compl_ptr[cnt]; status[cnt] = comp_ptr->cdata; - if (comp_ptr->cdata) + if (status[cnt]) { + if (status[cnt] == DPI_REQ_CDATA) + break; + dpivf->stats.errors++; + } } *last_idx = cnt - 1; @@ -435,6 +606,17 @@ cnxk_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) return 0; } +static const struct rte_dma_dev_ops cn10k_dmadev_ops = { + .dev_close = cnxk_dmadev_close, + .dev_configure = cnxk_dmadev_configure, + .dev_info_get = cnxk_dmadev_info_get, + .dev_start = cnxk_dmadev_start, + .dev_stop = cnxk_dmadev_stop, + .stats_get = cnxk_stats_get, + .stats_reset = cnxk_stats_reset, + .vchan_setup = cn10k_dmadev_vchan_setup, +}; + static const struct rte_dma_dev_ops cnxk_dmadev_ops = { .dev_close = cnxk_dmadev_close, .dev_configure = cnxk_dmadev_configure, @@ -486,6 +668,14 @@ cnxk_dmadev_probe(struct rte_pci_driver *pci_drv __rte_unused, dmadev->fp_obj->completed = cnxk_dmadev_completed; dmadev->fp_obj->completed_status = cnxk_dmadev_completed_status; + if (pci_dev->id.subsystem_device_id == PCI_SUBSYSTEM_DEVID_CN10KA || + pci_dev->id.subsystem_device_id == PCI_SUBSYSTEM_DEVID_CNF10KA || + pci_dev->id.subsystem_device_id == PCI_SUBSYSTEM_DEVID_CN10KB) { + dmadev->dev_ops = &cn10k_dmadev_ops; + dmadev->fp_obj->copy = cn10k_dmadev_copy; + dmadev->fp_obj->copy_sg = cn10k_dmadev_copy_sg; + } + rdpi = &dpivf->rdpi; rdpi->pci_dev = pci_dev; diff --git a/drivers/dma/cnxk/cnxk_dmadev.h b/drivers/dma/cnxk/cnxk_dmadev.h index 5fc241b55e..e1f5694f50 100644 --- a/drivers/dma/cnxk/cnxk_dmadev.h +++ b/drivers/dma/cnxk/cnxk_dmadev.h @@ -8,7 +8,7 @@ #define DPI_QUEUE_STOP 0x0 #define DPI_QUEUE_START 0x1 #define STRM_INC(s) ((s).tail = ((s).tail + 1) % (s).max_cnt) -#define DPI_MAX_DESC DPI_MAX_POINTER +#define DPI_MAX_DESC 1024 /* Set Completion data to 0xFF when request submitted, * upon successful request completion engine reset to completion status