From patchwork Thu Sep 2 12:22:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anoob Joseph X-Patchwork-Id: 97800 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3BB70A0C47; Thu, 2 Sep 2021 14:23:34 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 79802410FF; Thu, 2 Sep 2021 14:23:30 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 1EEB340686 for ; Thu, 2 Sep 2021 14:23:28 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1825LHeJ028342 for ; Thu, 2 Sep 2021 05:23:28 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=JKgwjqMrQa/+2SIODoN/hTS7U6mNCsYwDixF7aI0pm4=; b=ZMo7f7Db6OtH3tU77uEEXwRNBLhXBNxQAUsCdZEbB4ynnaps93ulFEVkojrxcbzLwXB5 zauDiJqeBQDrkA1NiMHSzQBBlqsf4gb0oV96GchFztChKJN4CXaKtxE7SgFQBxljpwjL +ANNquFjyffxmU5clWQx3eaUIDEvwoK4OIDFTRL80pWaZSwcDMyMh0Q95+j6jzzbsL9B J08xOyAqUXsyWCMY4S7ADgBay4BZtFrD/txBTuIfVsmtEpXiYo78Wh7bSFicl4ckRfU8 VLVQU8E3FWVXq5RZEo6uMy31v/Ob7HMsKheupgzpCUZW3XfoeCSl0Qg+qiAZWfovpjb0 xw== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com with ESMTP id 3atrd2hefc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Thu, 02 Sep 2021 05:23:27 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Thu, 2 Sep 2021 05:23:26 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Thu, 2 Sep 2021 05:23:26 -0700 Received: from HY-LT1002.marvell.com (HY-LT1002.marvell.com [10.28.176.218]) by maili.marvell.com (Postfix) with ESMTP id 36DC53F705E; Thu, 2 Sep 2021 05:23:22 -0700 (PDT) From: Anoob Joseph To: Akhil Goyal , Jerin Jacob CC: Anoob Joseph , Archana Muniganti , Tejasree Kondoj , Date: Thu, 2 Sep 2021 17:52:34 +0530 Message-ID: <1630585354-1136-8-git-send-email-anoobj@marvell.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1630585354-1136-1-git-send-email-anoobj@marvell.com> References: <1630585354-1136-1-git-send-email-anoobj@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: 4pwFH_JD-zgi-VYU32tircLnq_pHli1r X-Proofpoint-ORIG-GUID: 4pwFH_JD-zgi-VYU32tircLnq_pHli1r X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-02_04,2021-09-02_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 7/7] crypto/cnxk: add dual submission in crypto_cn9k X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Submit two instructions with one LMTST operation. Also updated dequeue path to have local var for constants. Signed-off-by: Anoob Joseph --- drivers/crypto/cnxk/cn9k_cryptodev_ops.c | 155 +++++++++++++++++++++++++------ 1 file changed, 127 insertions(+), 28 deletions(-) diff --git a/drivers/crypto/cnxk/cn9k_cryptodev_ops.c b/drivers/crypto/cnxk/cn9k_cryptodev_ops.c index 4c0eb12..8ade197 100644 --- a/drivers/crypto/cnxk/cn9k_cryptodev_ops.c +++ b/drivers/crypto/cnxk/cn9k_cryptodev_ops.c @@ -5,6 +5,7 @@ #include #include #include +#include #include "cn9k_cryptodev.h" #include "cn9k_cryptodev_ops.h" @@ -64,9 +65,8 @@ cn9k_cpt_sym_temp_sess_create(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op) } static inline int -cn9k_cpt_prepare_instruction(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op, - struct cpt_inflight_req *infl_req, - struct cpt_inst_s *inst) +cn9k_cpt_inst_prep(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op, + struct cpt_inflight_req *infl_req, struct cpt_inst_s *inst) { int ret; @@ -118,8 +118,8 @@ cn9k_cpt_prepare_instruction(struct cnxk_cpt_qp *qp, struct rte_crypto_op *op, } static inline void -cn9k_cpt_submit_instruction(struct cpt_inst_s *inst, uint64_t lmtline, - uint64_t io_addr) +cn9k_cpt_inst_submit(struct cpt_inst_s *inst, uint64_t lmtline, + uint64_t io_addr) { uint64_t lmt_status; @@ -138,46 +138,144 @@ cn9k_cpt_submit_instruction(struct cpt_inst_s *inst, uint64_t lmtline, } while (lmt_status == 0); } +static __plt_always_inline void +cn9k_cpt_inst_submit_dual(struct cpt_inst_s *inst, uint64_t lmtline, + uint64_t io_addr) +{ + uint64_t lmt_status; + + do { + /* Copy 2 CPT inst_s to LMTLINE */ +#if defined(RTE_ARCH_ARM64) + uint64_t *s = (uint64_t *)inst; + uint64_t *d = (uint64_t *)lmtline; + + vst1q_u64(&d[0], vld1q_u64(&s[0])); + vst1q_u64(&d[2], vld1q_u64(&s[2])); + vst1q_u64(&d[4], vld1q_u64(&s[4])); + vst1q_u64(&d[6], vld1q_u64(&s[6])); + vst1q_u64(&d[8], vld1q_u64(&s[8])); + vst1q_u64(&d[10], vld1q_u64(&s[10])); + vst1q_u64(&d[12], vld1q_u64(&s[12])); + vst1q_u64(&d[14], vld1q_u64(&s[14])); +#else + roc_lmt_mov_seg((void *)lmtline, inst, 8); +#endif + + /* + * Make sure compiler does not reorder memcpy and ldeor. + * LMTST transactions are always flushed from the write + * buffer immediately, a DMB is not required to push out + * LMTSTs. + */ + rte_io_wmb(); + lmt_status = roc_lmt_submit_ldeor(io_addr); + } while (lmt_status == 0); +} + static uint16_t cn9k_cpt_enqueue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) { - struct cpt_inflight_req *infl_req; + struct cpt_inflight_req *infl_req_1, *infl_req_2; + struct cpt_inst_s inst[2] __rte_cache_aligned; + struct rte_crypto_op *op_1, *op_2; uint16_t nb_allowed, count = 0; struct cnxk_cpt_qp *qp = qptr; struct pending_queue *pend_q; - struct rte_crypto_op *op; - struct cpt_inst_s inst; + uint64_t enq_tail; int ret; + const uint32_t nb_desc = qp->lf.nb_desc; + const uint64_t lmt_base = qp->lf.lmt_base; + const uint64_t io_addr = qp->lf.io_addr; + pend_q = &qp->pend_q; - inst.w0.u64 = 0; - inst.w2.u64 = 0; - inst.w3.u64 = 0; + /* Clear w0, w2, w3 of both inst */ + + inst[0].w0.u64 = 0; + inst[0].w2.u64 = 0; + inst[0].w3.u64 = 0; + inst[1].w0.u64 = 0; + inst[1].w2.u64 = 0; + inst[1].w3.u64 = 0; nb_allowed = qp->lf.nb_desc - pend_q->pending_count; nb_ops = RTE_MIN(nb_ops, nb_allowed); - for (count = 0; count < nb_ops; count++) { - op = ops[count]; - infl_req = &pend_q->req_queue[pend_q->enq_tail]; - infl_req->op_flags = 0; + enq_tail = pend_q->enq_tail; + + if (unlikely(nb_ops & 1)) { + op_1 = ops[0]; + infl_req_1 = &pend_q->req_queue[enq_tail]; + infl_req_1->op_flags = 0; - ret = cn9k_cpt_prepare_instruction(qp, op, infl_req, &inst); + ret = cn9k_cpt_inst_prep(qp, op_1, infl_req_1, &inst[0]); if (unlikely(ret)) { - plt_dp_err("Could not process op: %p", op); + plt_dp_err("Could not process op: %p", op_1); + return 0; + } + + infl_req_1->cop = op_1; + infl_req_1->res.cn9k.compcode = CPT_COMP_NOT_DONE; + inst[0].res_addr = (uint64_t)&infl_req_1->res; + + cn9k_cpt_inst_submit(&inst[0], lmt_base, io_addr); + MOD_INC(enq_tail, nb_desc); + count++; + } + + while (count < nb_ops) { + op_1 = ops[count]; + op_2 = ops[count + 1]; + + infl_req_1 = &pend_q->req_queue[enq_tail]; + MOD_INC(enq_tail, nb_desc); + infl_req_2 = &pend_q->req_queue[enq_tail]; + MOD_INC(enq_tail, nb_desc); + + infl_req_1->cop = op_1; + infl_req_2->cop = op_2; + infl_req_1->op_flags = 0; + infl_req_2->op_flags = 0; + + infl_req_1->res.cn9k.compcode = CPT_COMP_NOT_DONE; + inst[0].res_addr = (uint64_t)&infl_req_1->res; + + infl_req_2->res.cn9k.compcode = CPT_COMP_NOT_DONE; + inst[1].res_addr = (uint64_t)&infl_req_2->res; + + ret = cn9k_cpt_inst_prep(qp, op_1, infl_req_1, &inst[0]); + if (unlikely(ret)) { + plt_dp_err("Could not process op: %p", op_1); + if (enq_tail == 0) + enq_tail = nb_desc - 2; + else if (enq_tail == 1) + enq_tail = nb_desc - 1; + else + enq_tail--; + break; + } + + ret = cn9k_cpt_inst_prep(qp, op_2, infl_req_2, &inst[1]); + if (unlikely(ret)) { + plt_dp_err("Could not process op: %p", op_2); + if (enq_tail == 0) + enq_tail = nb_desc - 1; + else + enq_tail--; + + cn9k_cpt_inst_submit(&inst[0], lmt_base, io_addr); + count++; break; } - infl_req->cop = op; - infl_req->res.cn9k.compcode = CPT_COMP_NOT_DONE; - inst.res_addr = (uint64_t)&infl_req->res; + cn9k_cpt_inst_submit_dual(&inst[0], lmt_base, io_addr); - cn9k_cpt_submit_instruction(&inst, qp->lmtline.lmt_base, - qp->lmtline.io_addr); - MOD_INC(pend_q->enq_tail, qp->lf.nb_desc); + count += 2; } + pend_q->enq_tail = enq_tail; pend_q->pending_count += count; pend_q->time_out = rte_get_timer_cycles() + DEFAULT_COMMAND_TIMEOUT * rte_get_timer_hz(); @@ -219,7 +317,7 @@ cn9k_cpt_crypto_adapter_enqueue(uintptr_t tag_op, struct rte_crypto_op *op) } infl_req->op_flags = 0; - ret = cn9k_cpt_prepare_instruction(qp, op, infl_req, &inst); + ret = cn9k_cpt_inst_prep(qp, op, infl_req, &inst); if (unlikely(ret)) { plt_dp_err("Could not process op: %p", op); rte_mempool_put(qp->ca.req_mp, infl_req); @@ -245,8 +343,7 @@ cn9k_cpt_crypto_adapter_enqueue(uintptr_t tag_op, struct rte_crypto_op *op) if (!rsp_info->sched_type) roc_sso_hws_head_wait(tag_op); - cn9k_cpt_submit_instruction(&inst, qp->lmtline.lmt_base, - qp->lmtline.io_addr); + cn9k_cpt_inst_submit(&inst, qp->lmtline.lmt_base, qp->lmtline.io_addr); return 1; } @@ -347,14 +444,16 @@ cn9k_cpt_crypto_adapter_dequeue(uintptr_t get_work1) static uint16_t cn9k_cpt_dequeue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) { + struct cpt_inflight_req *infl_req; struct cnxk_cpt_qp *qp = qptr; struct pending_queue *pend_q; - struct cpt_inflight_req *infl_req; struct cpt_cn9k_res_s *res; struct rte_crypto_op *cop; uint32_t pq_deq_head; int i; + const uint32_t nb_desc = qp->lf.nb_desc; + pend_q = &qp->pend_q; nb_ops = RTE_MIN(nb_ops, pend_q->pending_count); @@ -377,7 +476,7 @@ cn9k_cpt_dequeue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) break; } - MOD_INC(pq_deq_head, qp->lf.nb_desc); + MOD_INC(pq_deq_head, nb_desc); cop = infl_req->cop;