From patchwork Thu Jul 16 08:39:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tejasree Kondoj X-Patchwork-Id: 74188 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id DDC03A0546; Thu, 16 Jul 2020 09:46:49 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id CD1551BED7; Thu, 16 Jul 2020 09:46:17 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id 2A4381BEBF for ; Thu, 16 Jul 2020 09:46:16 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 06G7YkgG004597; Thu, 16 Jul 2020 00:46:15 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0818; bh=a33ddeoBxUNHR+fURBAtoZilmwUm6RhheJHKqoo8Qfw=; b=t63JTT3r69dvVc7wwBIMg2bWn9F2Yl3Q6k03A+/nV54mV6gutmSCVXEQxSMe9ynGqOKD iwnr3NNo3VqTBWwl9sZn8FSo2o/kjLyh0KEBXn1WyT7Xm01qh8cg3ICt76Z1PkYm5wTx 0yYToxr0P6MWrTIvV74dZeEUPJawevBoPbZx4fo4ugDjW4QoYfIms0wE+2WkX2srSy6g TPqOovTRlSsPc0pa3Jhv6gCPzh+yOA/xQcNLHBCEsGZmXmd1bfeltp2ZvRUyJ81QP2Aq i4Zzt+Og5UW87jpqgnaAT3O2t7SkLeLf8+iKoC8GZ0CxK/I1sL269tTpJLfkkYWbhoyw 8Q== Received: from sc-exch02.marvell.com ([199.233.58.182]) by mx0b-0016f401.pphosted.com with ESMTP id 328mmhxma7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 16 Jul 2020 00:46:15 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by SC-EXCH02.marvell.com (10.93.176.82) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 16 Jul 2020 00:46:14 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 16 Jul 2020 00:46:12 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 16 Jul 2020 00:46:12 -0700 Received: from hyd1554T5810.caveonetworks.com.com (unknown [10.29.57.11]) by maili.marvell.com (Postfix) with ESMTP id 84B2A3F7041; Thu, 16 Jul 2020 00:46:10 -0700 (PDT) From: Tejasree Kondoj To: Akhil Goyal , Radu Nicolau CC: Vamsi Attunuru , Narayana Prasad , Anoob Joseph , Tejasree Kondoj , Date: Thu, 16 Jul 2020 14:09:31 +0530 Message-ID: <20200716083931.29092-9-ktejasree@marvell.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200716083931.29092-1-ktejasree@marvell.com> References: <20200716083931.29092-1-ktejasree@marvell.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235, 18.0.687 definitions=2020-07-16_04:2020-07-16, 2020-07-16 signatures=0 Subject: [dpdk-dev] [PATCH v3 8/8] crypto/octeontx2: add cryptodev sec enqueue and dequeue routines X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Vamsi Attunuru This patch adds lookaside IPsec enqueue and dequeue routines. Signed-off-by: Vamsi Attunuru Signed-off-by: Tejasree Kondoj --- doc/guides/cryptodevs/octeontx2.rst | 21 +++ doc/guides/rel_notes/release_20_08.rst | 5 + drivers/crypto/octeontx2/otx2_cryptodev.h | 8 + drivers/crypto/octeontx2/otx2_cryptodev_ops.c | 73 +++++++- drivers/crypto/octeontx2/otx2_ipsec_po.h | 8 + drivers/crypto/octeontx2/otx2_ipsec_po_ops.h | 175 ++++++++++++++++++ 6 files changed, 289 insertions(+), 1 deletion(-) create mode 100644 drivers/crypto/octeontx2/otx2_ipsec_po_ops.h diff --git a/doc/guides/cryptodevs/octeontx2.rst b/doc/guides/cryptodevs/octeontx2.rst index 085d669e49..432146db04 100644 --- a/doc/guides/cryptodevs/octeontx2.rst +++ b/doc/guides/cryptodevs/octeontx2.rst @@ -158,3 +158,24 @@ application: ./test RTE>>cryptodev_octeontx2_asym_autotest + + +Lookaside IPsec Support +----------------------- + +The OCTEON TX2 SoC can accelerate IPsec traffic in lookaside protocol mode, +with its **cryptographic accelerator (CPT)**. ``OCTEON TX2 crypto PMD`` implements +this as an ``RTE_SECURITY_ACTION_TYPE_LOOKASIDE_PROTOCOL`` offload. + +Refer to :doc:`../prog_guide/rte_security` for more details on protocol offloads. + +This feature can be tested with ipsec-secgw sample application. + + +Features supported +~~~~~~~~~~~~~~~~~~ + +* IPv4 +* ESP +* Tunnel mode +* AES-128/192/256-GCM diff --git a/doc/guides/rel_notes/release_20_08.rst b/doc/guides/rel_notes/release_20_08.rst index f19b748728..53cd13455f 100644 --- a/doc/guides/rel_notes/release_20_08.rst +++ b/doc/guides/rel_notes/release_20_08.rst @@ -196,6 +196,11 @@ New Features Added Chacha20-Poly1305 AEAD algorithm support in OCTEON TX2 crypto PMD. +* **Updated the OCTEON TX2 crypto PMD to support rte_security.** + + Updated the OCTEON TX2 crypto PMD to support ``rte_security`` lookaside + protocol offload for IPsec. + * **Added support for BPF_ABS/BPF_IND load instructions.** Added support for two BPF non-generic instructions: diff --git a/drivers/crypto/octeontx2/otx2_cryptodev.h b/drivers/crypto/octeontx2/otx2_cryptodev.h index e7a1730b22..f329741b38 100644 --- a/drivers/crypto/octeontx2/otx2_cryptodev.h +++ b/drivers/crypto/octeontx2/otx2_cryptodev.h @@ -6,6 +6,7 @@ #define _OTX2_CRYPTODEV_H_ #include "cpt_common.h" +#include "cpt_hw_types.h" #include "otx2_dev.h" @@ -33,6 +34,13 @@ struct otx2_cpt_vf { /**< CPT device capabilities */ }; +struct cpt_meta_info { + uint64_t deq_op_info[4]; + uint64_t comp_code_sz; + union cpt_res_s cpt_res __rte_aligned(16); + struct cpt_request_info cpt_req __rte_aligned(8); +}; + #define CPT_LOGTYPE otx2_cpt_logtype extern int otx2_cpt_logtype; diff --git a/drivers/crypto/octeontx2/otx2_cryptodev_ops.c b/drivers/crypto/octeontx2/otx2_cryptodev_ops.c index 229b719b42..9d51b17ddd 100644 --- a/drivers/crypto/octeontx2/otx2_cryptodev_ops.c +++ b/drivers/crypto/octeontx2/otx2_cryptodev_ops.c @@ -13,8 +13,10 @@ #include "otx2_cryptodev_hw_access.h" #include "otx2_cryptodev_mbox.h" #include "otx2_cryptodev_ops.h" +#include "otx2_ipsec_po_ops.h" #include "otx2_mbox.h" #include "otx2_sec_idev.h" +#include "otx2_security.h" #include "cpt_hw_types.h" #include "cpt_pmd_logs.h" @@ -606,6 +608,36 @@ otx2_cpt_enqueue_sym(struct otx2_cpt_qp *qp, struct rte_crypto_op *op, return ret; } +static __rte_always_inline int __rte_hot +otx2_cpt_enqueue_sec(struct otx2_cpt_qp *qp, struct rte_crypto_op *op, + struct pending_queue *pend_q) +{ + struct otx2_sec_session_ipsec_lp *sess; + struct otx2_ipsec_po_sa_ctl *ctl_wrd; + struct otx2_sec_session *priv; + struct cpt_request_info *req; + int ret; + + priv = get_sec_session_private_data(op->sym->sec_session); + sess = &priv->ipsec.lp; + + ctl_wrd = &sess->in_sa.ctl; + + if (ctl_wrd->direction == OTX2_IPSEC_PO_SA_DIRECTION_OUTBOUND) + ret = process_outb_sa(op, sess, &qp->meta_info, (void **)&req); + else + ret = process_inb_sa(op, sess, &qp->meta_info, (void **)&req); + + if (unlikely(ret)) { + otx2_err("Crypto req : op %p, ret 0x%x", op, ret); + return ret; + } + + ret = otx2_cpt_enqueue_req(qp, pend_q, req); + + return ret; +} + static __rte_always_inline int __rte_hot otx2_cpt_enqueue_sym_sessless(struct otx2_cpt_qp *qp, struct rte_crypto_op *op, struct pending_queue *pend_q) @@ -659,7 +691,9 @@ otx2_cpt_enqueue_burst(void *qptr, struct rte_crypto_op **ops, uint16_t nb_ops) for (count = 0; count < nb_ops; count++) { op = ops[count]; if (op->type == RTE_CRYPTO_OP_TYPE_SYMMETRIC) { - if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) + if (op->sess_type == RTE_CRYPTO_OP_SECURITY_SESSION) + ret = otx2_cpt_enqueue_sec(qp, op, pend_q); + else if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) ret = otx2_cpt_enqueue_sym(qp, op, pend_q); else ret = otx2_cpt_enqueue_sym_sessless(qp, op, @@ -801,11 +835,48 @@ otx2_cpt_asym_post_process(struct rte_crypto_op *cop, } } +static void +otx2_cpt_sec_post_process(struct rte_crypto_op *cop, uintptr_t *rsp) +{ + struct cpt_request_info *req = (struct cpt_request_info *)rsp[2]; + vq_cmd_word0_t *word0 = (vq_cmd_word0_t *)&req->ist.ei0; + struct rte_crypto_sym_op *sym_op = cop->sym; + struct rte_mbuf *m = sym_op->m_src; + struct rte_ipv4_hdr *ip; + uint16_t m_len; + int mdata_len; + char *data; + + mdata_len = (int)rsp[3]; + rte_pktmbuf_trim(m, mdata_len); + + if ((word0->s.opcode & 0xff) == OTX2_IPSEC_PO_PROCESS_IPSEC_INB) { + data = rte_pktmbuf_mtod(m, char *); + ip = (struct rte_ipv4_hdr *)(data + OTX2_IPSEC_PO_INB_RPTR_HDR); + + m_len = rte_be_to_cpu_16(ip->total_length); + + m->data_len = m_len; + m->pkt_len = m_len; + m->data_off += OTX2_IPSEC_PO_INB_RPTR_HDR; + } +} + static inline void otx2_cpt_dequeue_post_process(struct otx2_cpt_qp *qp, struct rte_crypto_op *cop, uintptr_t *rsp, uint8_t cc) { if (cop->type == RTE_CRYPTO_OP_TYPE_SYMMETRIC) { + if (cop->sess_type == RTE_CRYPTO_OP_SECURITY_SESSION) { + if (likely(cc == OTX2_IPSEC_PO_CC_SUCCESS)) { + otx2_cpt_sec_post_process(cop, rsp); + cop->status = RTE_CRYPTO_OP_STATUS_SUCCESS; + } else + cop->status = RTE_CRYPTO_OP_STATUS_ERROR; + + return; + } + if (likely(cc == NO_ERR)) { /* Verify authentication data if required */ if (unlikely(rsp[2])) diff --git a/drivers/crypto/octeontx2/otx2_ipsec_po.h b/drivers/crypto/octeontx2/otx2_ipsec_po.h index f2167f220a..020748609e 100644 --- a/drivers/crypto/octeontx2/otx2_ipsec_po.h +++ b/drivers/crypto/octeontx2/otx2_ipsec_po.h @@ -22,6 +22,8 @@ #define OTX2_IPSEC_PO_PROCESS_IPSEC_OUTB 0x23 #define OTX2_IPSEC_PO_PROCESS_IPSEC_INB 0x24 +#define OTX2_IPSEC_PO_INB_RPTR_HDR 0x8 + enum otx2_ipsec_po_comp_e { OTX2_IPSEC_PO_CC_SUCCESS = 0x00, OTX2_IPSEC_PO_CC_AUTH_UNSUPPORTED = 0xB0, @@ -86,6 +88,12 @@ enum { OTX2_IPSEC_PO_SA_ENCAP_UDP = 1, }; +struct otx2_ipsec_po_out_hdr { + uint32_t ip_id; + uint32_t seq; + uint8_t iv[16]; +}; + union otx2_ipsec_po_bit_perfect_iv { uint8_t aes_iv[16]; uint8_t des_iv[8]; diff --git a/drivers/crypto/octeontx2/otx2_ipsec_po_ops.h b/drivers/crypto/octeontx2/otx2_ipsec_po_ops.h new file mode 100644 index 0000000000..dd29c413d3 --- /dev/null +++ b/drivers/crypto/octeontx2/otx2_ipsec_po_ops.h @@ -0,0 +1,175 @@ + +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2019 Marvell International Ltd. + */ + +#ifndef __OTX2_IPSEC_PO_OPS_H__ +#define __OTX2_IPSEC_PO_OPS_H__ + +#include +#include + +#include "otx2_cryptodev.h" +#include "otx2_security.h" + +static __rte_always_inline int32_t +otx2_ipsec_po_out_rlen_get(struct otx2_sec_session_ipsec_lp *sess, + uint32_t plen) +{ + uint32_t enc_payload_len; + + enc_payload_len = RTE_ALIGN_CEIL(plen + sess->roundup_len, + sess->roundup_byte); + + return sess->partial_len + enc_payload_len; +} + +static __rte_always_inline struct cpt_request_info * +alloc_request_struct(char *maddr, void *cop, int mdata_len) +{ + struct cpt_request_info *req; + struct cpt_meta_info *meta; + uint8_t *resp_addr; + uintptr_t *op; + + meta = (void *)RTE_PTR_ALIGN((uint8_t *)maddr, 16); + + op = (uintptr_t *)meta->deq_op_info; + req = &meta->cpt_req; + resp_addr = (uint8_t *)&meta->cpt_res; + + req->completion_addr = (uint64_t *)((uint8_t *)resp_addr); + *req->completion_addr = COMPLETION_CODE_INIT; + req->comp_baddr = rte_mem_virt2iova(resp_addr); + req->op = op; + + op[0] = (uintptr_t)((uint64_t)meta | 1ull); + op[1] = (uintptr_t)cop; + op[2] = (uintptr_t)req; + op[3] = mdata_len; + + return req; +} + +static __rte_always_inline int +process_outb_sa(struct rte_crypto_op *cop, + struct otx2_sec_session_ipsec_lp *sess, + struct cpt_qp_meta_info *m_info, void **prep_req) +{ + uint32_t dlen, rlen, extend_head, extend_tail; + struct rte_crypto_sym_op *sym_op = cop->sym; + struct rte_mbuf *m_src = sym_op->m_src; + struct otx2_ipsec_po_sa_ctl *ctl_wrd; + struct cpt_request_info *req = NULL; + struct otx2_ipsec_po_out_hdr *hdr; + struct otx2_ipsec_po_out_sa *sa; + int hdr_len, mdata_len, ret = 0; + vq_cmd_word0_t word0; + char *mdata, *data; + + sa = &sess->out_sa; + ctl_wrd = &sa->ctl; + hdr_len = sizeof(*hdr); + + dlen = rte_pktmbuf_pkt_len(m_src) + hdr_len; + rlen = otx2_ipsec_po_out_rlen_get(sess, dlen - hdr_len); + + extend_head = hdr_len + RTE_ETHER_HDR_LEN; + extend_tail = rlen - dlen; + mdata_len = m_info->lb_mlen + 8; + + mdata = rte_pktmbuf_append(m_src, extend_tail + mdata_len); + if (unlikely(mdata == NULL)) { + otx2_err("Not enough tail room\n"); + ret = -ENOMEM; + goto exit; + } + + mdata += extend_tail; /* mdata follows encrypted data */ + req = alloc_request_struct(mdata, (void *)cop, mdata_len); + + data = rte_pktmbuf_prepend(m_src, extend_head); + if (unlikely(data == NULL)) { + otx2_err("Not enough head room\n"); + ret = -ENOMEM; + goto exit; + } + + /* + * Move the Ethernet header, to insert otx2_ipsec_po_out_hdr prior + * to the IP header + */ + memcpy(data, data + hdr_len, RTE_ETHER_HDR_LEN); + + hdr = (struct otx2_ipsec_po_out_hdr *)rte_pktmbuf_adj(m_src, + RTE_ETHER_HDR_LEN); + + if (ctl_wrd->enc_type == OTX2_IPSEC_FP_SA_ENC_AES_GCM) { + memcpy(&hdr->iv[0], &sa->iv.gcm.nonce, 4); + memcpy(&hdr->iv[4], rte_crypto_op_ctod_offset(cop, uint8_t *, + sess->iv_offset), sess->iv_length); + } else if (ctl_wrd->auth_type == OTX2_IPSEC_FP_SA_ENC_AES_CBC) { + memcpy(&hdr->iv[0], rte_crypto_op_ctod_offset(cop, uint8_t *, + sess->iv_offset), sess->iv_length); + } + + /* Prepare CPT instruction */ + word0.u64 = sess->ucmd_w0; + word0.s.dlen = dlen; + + req->ist.ei0 = word0.u64; + req->ist.ei1 = rte_pktmbuf_iova(m_src); + req->ist.ei2 = req->ist.ei1; + req->ist.ei3 = sess->ucmd_w3; + + hdr->seq = rte_cpu_to_be_32(sess->seq_lo); + hdr->ip_id = rte_cpu_to_be_32(sess->ip_id); + + sess->ip_id++; + sess->esn++; + +exit: + *prep_req = req; + + return ret; +} + +static __rte_always_inline int +process_inb_sa(struct rte_crypto_op *cop, + struct otx2_sec_session_ipsec_lp *sess, + struct cpt_qp_meta_info *m_info, void **prep_req) +{ + struct rte_crypto_sym_op *sym_op = cop->sym; + struct rte_mbuf *m_src = sym_op->m_src; + struct cpt_request_info *req = NULL; + int mdata_len, ret = 0; + vq_cmd_word0_t word0; + uint32_t dlen; + char *mdata; + + dlen = rte_pktmbuf_pkt_len(m_src); + mdata_len = m_info->lb_mlen + 8; + + mdata = rte_pktmbuf_append(m_src, mdata_len); + if (unlikely(mdata == NULL)) { + otx2_err("Not enough tail room\n"); + ret = -ENOMEM; + goto exit; + } + + req = alloc_request_struct(mdata, (void *)cop, mdata_len); + + /* Prepare CPT instruction */ + word0.u64 = sess->ucmd_w0; + word0.s.dlen = dlen; + + req->ist.ei0 = word0.u64; + req->ist.ei1 = rte_pktmbuf_iova(m_src); + req->ist.ei2 = req->ist.ei1; + req->ist.ei3 = sess->ucmd_w3; + +exit: + *prep_req = req; + return ret; +} +#endif /* __OTX2_IPSEC_PO_OPS_H__ */