From patchwork Thu Sep 2 02:14:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97727 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 32F7DA0C4C; Thu, 2 Sep 2021 04:16:54 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1B94A40698; Thu, 2 Sep 2021 04:16:54 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 397FA40696 for ; Thu, 2 Sep 2021 04:16:53 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQBNn011731; Wed, 1 Sep 2021 19:16:51 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=6pvlpxrIQQpbibBBfgI7qe4xb5Iet4DEcFQLqK/k5l8=; b=g9vSeBiaiVsVyrrKpcscfSMgAgD48evnmZyzbbl6Hwm3obS5RzCSkDpLr8a/U7ifVZR+ 1AheoN0uusGDpkbTfcLcAoI3DY8jqpUMbFzSKBt0XYWETKP7afCs8Ks8RXeo47zhdNgl kIA8kqY3jvmvwYUU9rJWnFvni4l75/NzwTfjnhJQYQ7fuDlbUaDmcahotHm12Yb+NhKP QVa6kc4e/s7KMgjcJDVDTINzPHu1KmV+BjgLLnO+zqPjw2vR5fQXV2JJrWIGLRnjbkhA cVokJZn8V6ffG1Exs8mGi8VJSX+nOh/H8hTyuf1bgeDKXFXhbH/GIDyh5XJry3voIJg1 Kg== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hpu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 19:16:49 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:16:47 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:16:47 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id DEA923F704B; Wed, 1 Sep 2021 19:16:44 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao , Ray Kinsella CC: , , Date: Thu, 2 Sep 2021 07:44:39 +0530 Message-ID: <20210902021505.17607-2-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: qTjmChuHoaJdmzut40hKpPrGyEz7UIF9 X-Proofpoint-GUID: qTjmChuHoaJdmzut40hKpPrGyEz7UIF9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 01/27] common/cnxk: add security support for cn9k fast path X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Srujana Challa Add security support to init cn9k fast path SA data for AES GCM and AES CBC + HMAC SHA1. Signed-off-by: Srujana Challa Signed-off-by: Nithin Dabilpuram --- drivers/common/cnxk/cnxk_security.c | 211 ++++++++++++++++++++++++++++++++++++ drivers/common/cnxk/cnxk_security.h | 12 ++ drivers/common/cnxk/version.map | 4 + 3 files changed, 227 insertions(+) diff --git a/drivers/common/cnxk/cnxk_security.c b/drivers/common/cnxk/cnxk_security.c index 4f7fd1b..c25b3fd 100644 --- a/drivers/common/cnxk/cnxk_security.c +++ b/drivers/common/cnxk/cnxk_security.c @@ -383,6 +383,217 @@ cnxk_ot_ipsec_outb_sa_valid(struct roc_ot_ipsec_outb_sa *sa) return !!sa->w2.s.valid; } +static inline int +ipsec_xfrm_verify(struct rte_security_ipsec_xform *ipsec_xfrm, + struct rte_crypto_sym_xform *crypto_xfrm) +{ + if (crypto_xfrm->next == NULL) + return -EINVAL; + + if (ipsec_xfrm->direction == RTE_SECURITY_IPSEC_SA_DIR_INGRESS) { + if (crypto_xfrm->type != RTE_CRYPTO_SYM_XFORM_AUTH || + crypto_xfrm->next->type != RTE_CRYPTO_SYM_XFORM_CIPHER) + return -EINVAL; + } else { + if (crypto_xfrm->type != RTE_CRYPTO_SYM_XFORM_CIPHER || + crypto_xfrm->next->type != RTE_CRYPTO_SYM_XFORM_AUTH) + return -EINVAL; + } + + return 0; +} + +static int +onf_ipsec_sa_common_param_fill(struct roc_ie_onf_sa_ctl *ctl, uint8_t *salt, + uint8_t *cipher_key, uint8_t *hmac_opad_ipad, + struct rte_security_ipsec_xform *ipsec_xfrm, + struct rte_crypto_sym_xform *crypto_xfrm) +{ + struct rte_crypto_sym_xform *auth_xfrm, *cipher_xfrm; + int rc, length, auth_key_len; + const uint8_t *key = NULL; + + /* Set direction */ + switch (ipsec_xfrm->direction) { + case RTE_SECURITY_IPSEC_SA_DIR_INGRESS: + ctl->direction = ROC_IE_SA_DIR_INBOUND; + auth_xfrm = crypto_xfrm; + cipher_xfrm = crypto_xfrm->next; + break; + case RTE_SECURITY_IPSEC_SA_DIR_EGRESS: + ctl->direction = ROC_IE_SA_DIR_OUTBOUND; + cipher_xfrm = crypto_xfrm; + auth_xfrm = crypto_xfrm->next; + break; + default: + return -EINVAL; + } + + /* Set protocol - ESP vs AH */ + switch (ipsec_xfrm->proto) { + case RTE_SECURITY_IPSEC_SA_PROTO_ESP: + ctl->ipsec_proto = ROC_IE_SA_PROTOCOL_ESP; + break; + case RTE_SECURITY_IPSEC_SA_PROTO_AH: + return -ENOTSUP; + default: + return -EINVAL; + } + + /* Set mode - transport vs tunnel */ + switch (ipsec_xfrm->mode) { + case RTE_SECURITY_IPSEC_SA_MODE_TRANSPORT: + ctl->ipsec_mode = ROC_IE_SA_MODE_TRANSPORT; + break; + case RTE_SECURITY_IPSEC_SA_MODE_TUNNEL: + ctl->ipsec_mode = ROC_IE_SA_MODE_TUNNEL; + break; + default: + return -EINVAL; + } + + /* Set encryption algorithm */ + if (crypto_xfrm->type == RTE_CRYPTO_SYM_XFORM_AEAD) { + length = crypto_xfrm->aead.key.length; + + switch (crypto_xfrm->aead.algo) { + case RTE_CRYPTO_AEAD_AES_GCM: + ctl->enc_type = ROC_IE_ON_SA_ENC_AES_GCM; + ctl->auth_type = ROC_IE_ON_SA_AUTH_NULL; + memcpy(salt, &ipsec_xfrm->salt, 4); + key = crypto_xfrm->aead.key.data; + break; + default: + return -ENOTSUP; + } + + } else { + rc = ipsec_xfrm_verify(ipsec_xfrm, crypto_xfrm); + if (rc) + return rc; + + switch (cipher_xfrm->cipher.algo) { + case RTE_CRYPTO_CIPHER_AES_CBC: + ctl->enc_type = ROC_IE_ON_SA_ENC_AES_CBC; + break; + default: + return -ENOTSUP; + } + + switch (auth_xfrm->auth.algo) { + case RTE_CRYPTO_AUTH_SHA1_HMAC: + ctl->auth_type = ROC_IE_ON_SA_AUTH_SHA1; + break; + default: + return -ENOTSUP; + } + auth_key_len = auth_xfrm->auth.key.length; + if (auth_key_len < 20 || auth_key_len > 64) + return -ENOTSUP; + + key = cipher_xfrm->cipher.key.data; + length = cipher_xfrm->cipher.key.length; + + ipsec_hmac_opad_ipad_gen(auth_xfrm, hmac_opad_ipad); + } + + switch (length) { + case ROC_CPT_AES128_KEY_LEN: + ctl->aes_key_len = ROC_IE_SA_AES_KEY_LEN_128; + break; + case ROC_CPT_AES192_KEY_LEN: + ctl->aes_key_len = ROC_IE_SA_AES_KEY_LEN_192; + break; + case ROC_CPT_AES256_KEY_LEN: + ctl->aes_key_len = ROC_IE_SA_AES_KEY_LEN_256; + break; + default: + return -EINVAL; + } + + memcpy(cipher_key, key, length); + + if (ipsec_xfrm->options.esn) + ctl->esn_en = 1; + + ctl->spi = rte_cpu_to_be_32(ipsec_xfrm->spi); + return 0; +} + +int +cnxk_onf_ipsec_inb_sa_fill(struct roc_onf_ipsec_inb_sa *sa, + struct rte_security_ipsec_xform *ipsec_xfrm, + struct rte_crypto_sym_xform *crypto_xfrm) +{ + struct roc_ie_onf_sa_ctl *ctl = &sa->ctl; + int rc; + + rc = onf_ipsec_sa_common_param_fill(ctl, sa->nonce, sa->cipher_key, + sa->hmac_key, ipsec_xfrm, + crypto_xfrm); + if (rc) + return rc; + + rte_wmb(); + + /* Enable SA */ + ctl->valid = 1; + return 0; +} + +int +cnxk_onf_ipsec_outb_sa_fill(struct roc_onf_ipsec_outb_sa *sa, + struct rte_security_ipsec_xform *ipsec_xfrm, + struct rte_crypto_sym_xform *crypto_xfrm) +{ + struct rte_security_ipsec_tunnel_param *tunnel = &ipsec_xfrm->tunnel; + struct roc_ie_onf_sa_ctl *ctl = &sa->ctl; + int rc; + + /* Fill common params */ + rc = onf_ipsec_sa_common_param_fill(ctl, sa->nonce, sa->cipher_key, + sa->hmac_key, ipsec_xfrm, + crypto_xfrm); + if (rc) + return rc; + + if (ipsec_xfrm->mode != RTE_SECURITY_IPSEC_SA_MODE_TUNNEL) + goto skip_tunnel_info; + + /* Tunnel header info */ + switch (tunnel->type) { + case RTE_SECURITY_IPSEC_TUNNEL_IPV4: + memcpy(&sa->ip_src, &tunnel->ipv4.src_ip, + sizeof(struct in_addr)); + memcpy(&sa->ip_dst, &tunnel->ipv4.dst_ip, + sizeof(struct in_addr)); + break; + case RTE_SECURITY_IPSEC_TUNNEL_IPV6: + return -ENOTSUP; + default: + return -EINVAL; + } + +skip_tunnel_info: + rte_wmb(); + + /* Enable SA */ + ctl->valid = 1; + return 0; +} + +bool +cnxk_onf_ipsec_inb_sa_valid(struct roc_onf_ipsec_inb_sa *sa) +{ + return !!sa->ctl.valid; +} + +bool +cnxk_onf_ipsec_outb_sa_valid(struct roc_onf_ipsec_outb_sa *sa) +{ + return !!sa->ctl.valid; +} + uint8_t cnxk_ipsec_ivlen_get(enum rte_crypto_cipher_algorithm c_algo, enum rte_crypto_auth_algorithm a_algo, diff --git a/drivers/common/cnxk/cnxk_security.h b/drivers/common/cnxk/cnxk_security.h index 602f583..db97887 100644 --- a/drivers/common/cnxk/cnxk_security.h +++ b/drivers/common/cnxk/cnxk_security.h @@ -46,4 +46,16 @@ cnxk_ot_ipsec_outb_sa_fill(struct roc_ot_ipsec_outb_sa *sa, bool __roc_api cnxk_ot_ipsec_inb_sa_valid(struct roc_ot_ipsec_inb_sa *sa); bool __roc_api cnxk_ot_ipsec_outb_sa_valid(struct roc_ot_ipsec_outb_sa *sa); +/* [CN9K, CN10K) */ +int __roc_api +cnxk_onf_ipsec_inb_sa_fill(struct roc_onf_ipsec_inb_sa *sa, + struct rte_security_ipsec_xform *ipsec_xfrm, + struct rte_crypto_sym_xform *crypto_xfrm); +int __roc_api +cnxk_onf_ipsec_outb_sa_fill(struct roc_onf_ipsec_outb_sa *sa, + struct rte_security_ipsec_xform *ipsec_xfrm, + struct rte_crypto_sym_xform *crypto_xfrm); +bool __roc_api cnxk_onf_ipsec_inb_sa_valid(struct roc_onf_ipsec_inb_sa *sa); +bool __roc_api cnxk_onf_ipsec_outb_sa_valid(struct roc_onf_ipsec_outb_sa *sa); + #endif /* _CNXK_SECURITY_H__ */ diff --git a/drivers/common/cnxk/version.map b/drivers/common/cnxk/version.map index 34a844b..7814b60 100644 --- a/drivers/common/cnxk/version.map +++ b/drivers/common/cnxk/version.map @@ -14,6 +14,10 @@ INTERNAL { cnxk_logtype_sso; cnxk_logtype_tim; cnxk_logtype_tm; + cnxk_onf_ipsec_inb_sa_fill; + cnxk_onf_ipsec_outb_sa_fill; + cnxk_onf_ipsec_inb_sa_valid; + cnxk_onf_ipsec_outb_sa_valid; cnxk_ot_ipsec_inb_sa_fill; cnxk_ot_ipsec_outb_sa_fill; cnxk_ot_ipsec_inb_sa_valid; From patchwork Thu Sep 2 02:14:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97728 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C4EB2A0C4C; Thu, 2 Sep 2021 04:16:59 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 54C7F40E09; Thu, 2 Sep 2021 04:16:57 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 73BD340E01 for ; Thu, 2 Sep 2021 04:16:55 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181K5SYh027320; Wed, 1 Sep 2021 19:16:52 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=mN96fFRLPN5aWFYD0xtxbfgO91nwFyfilkOiJZCxjpI=; b=gfxfx8Qh+SEo4b0y6q4/8nfhAXXy6n1+Bik6mt2XpUh+2Wyt5B9byPkfpyAEZM8QAJZC WlHrT1cReJ/Hf+5Q8QAQJUAgxUU8PHD6sJY0JoAY5NXoBPrWlQ/XvO668/UQeSH7kjz4 J1seOOLUAp0+2PC8hUEXGxe+BCv9SVTGj8Mcoq0mQs4iAk5EqPSXRQxAyAI8S2qfLT5L wZAomZMBwYCXwnAdhguP0K9r7Xcv+2KxQMmWxjZYnay5lVjm/1Q+Py7DormW7221YWhK bgGL8hCoiEYjUW3/wFkX350wEIUSRvEHcNftP5uEO+deOoXpFt6U8n7IOpzKgo8CMdrF hg== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com with ESMTP id 3atg8a91fq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 19:16:51 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:16:50 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:16:50 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id DA06D3F7052; Wed, 1 Sep 2021 19:16:47 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao , Ray Kinsella CC: , , Date: Thu, 2 Sep 2021 07:44:40 +0530 Message-ID: <20210902021505.17607-3-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: vjgZwRr6idMEPX-i714Q7Xq17wMNawcU X-Proofpoint-ORIG-GUID: vjgZwRr6idMEPX-i714Q7Xq17wMNawcU X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 02/27] common/cnxk: add helper API to dump cpt parse header X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add helper API to dump cpt parse header. Signed-off-by: Nithin Dabilpuram --- drivers/common/cnxk/roc_cpt.h | 2 ++ drivers/common/cnxk/roc_cpt_debug.c | 31 +++++++++++++++++++++++++++++++ drivers/common/cnxk/version.map | 1 + 3 files changed, 34 insertions(+) diff --git a/drivers/common/cnxk/roc_cpt.h b/drivers/common/cnxk/roc_cpt.h index f0f505a..9b55303 100644 --- a/drivers/common/cnxk/roc_cpt.h +++ b/drivers/common/cnxk/roc_cpt.h @@ -154,4 +154,6 @@ void __roc_api roc_cpt_iq_enable(struct roc_cpt_lf *lf); int __roc_api roc_cpt_lmtline_init(struct roc_cpt *roc_cpt, struct roc_cpt_lmtline *lmtline, int lf_id); +void __roc_api roc_cpt_parse_hdr_dump(const struct cpt_parse_hdr_s *cpth); + #endif /* _ROC_CPT_H_ */ diff --git a/drivers/common/cnxk/roc_cpt_debug.c b/drivers/common/cnxk/roc_cpt_debug.c index 9a9dcba..a6c9004 100644 --- a/drivers/common/cnxk/roc_cpt_debug.c +++ b/drivers/common/cnxk/roc_cpt_debug.c @@ -5,6 +5,37 @@ #include "roc_api.h" #include "roc_priv.h" +void +roc_cpt_parse_hdr_dump(const struct cpt_parse_hdr_s *cpth) +{ + plt_print("CPT_PARSE \t0x%p:", cpth); + + /* W0 */ + plt_print("W0: cookie \t0x%x\t\tmatch_id \t0x%04x\t\terr_sum \t%u \t", + cpth->w0.cookie, cpth->w0.match_id, cpth->w0.err_sum); + plt_print("W0: reas_sts \t0x%x\t\tet_owr \t%u\t\tpkt_fmt \t%u \t", + cpth->w0.reas_sts, cpth->w0.et_owr, cpth->w0.pkt_fmt); + plt_print("W0: pad_len \t%u\t\tnum_frags \t%u\t\tpkt_out \t%u \t", + cpth->w0.pad_len, cpth->w0.num_frags, cpth->w0.pkt_out); + + /* W1 */ + plt_print("W1: wqe_ptr \t0x%016lx\t", cpth->wqe_ptr); + + /* W2 */ + plt_print("W2: frag_age \t0x%x\t\torig_pf_func \t0x%04x", + cpth->w2.frag_age, cpth->w2.orig_pf_func); + plt_print("W2: il3_off \t0x%x\t\tfi_pad \t0x%x\t\tfi_offset \t0x%x \t", + cpth->w2.il3_off, cpth->w2.fi_pad, cpth->w2.fi_offset); + + /* W3 */ + plt_print("W3: hw_ccode \t0x%x\t\tuc_ccode \t0x%x\t\tspi \t0x%08x", + cpth->w3.hw_ccode, cpth->w3.uc_ccode, cpth->w3.spi); + + /* W4 */ + plt_print("W4: esn \t%" PRIx64 " \t OR frag1_wqe_ptr \t0x%" PRIx64, + cpth->esn, cpth->frag1_wqe_ptr); +} + static int cpt_af_reg_read(struct roc_cpt *roc_cpt, uint64_t reg, uint64_t *val) { diff --git a/drivers/common/cnxk/version.map b/drivers/common/cnxk/version.map index 7814b60..5dbb21c 100644 --- a/drivers/common/cnxk/version.map +++ b/drivers/common/cnxk/version.map @@ -66,6 +66,7 @@ INTERNAL { roc_cpt_lf_fini; roc_cpt_lfs_print; roc_cpt_lmtline_init; + roc_cpt_parse_hdr_dump; roc_cpt_rxc_time_cfg; roc_error_msg_get; roc_hash_sha1_gen; From patchwork Thu Sep 2 02:14:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97729 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A6D96A0C4C; Thu, 2 Sep 2021 04:17:05 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 71F70410E9; Thu, 2 Sep 2021 04:16:58 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 95EC540E01 for ; Thu, 2 Sep 2021 04:16:56 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQCpe011801 for ; Wed, 1 Sep 2021 19:16:56 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=JkqJuxejoQXhQnC7JC+9XJVQizmQvfo4DS9/FvHOT00=; b=TSYQSFG0AhYbSB/osVMhEZ+7klUHaBe6ZVda5h0XUD85n3PmRgJiAeXLP5zVMCjMaxen azItY57YLEzdCRjIX5xCZVGqmlk9D+Sjd4C4GfnFy+VMV8WZ06VwHfV1rowCu9aCz5nz Hwb3LjOfCDCiAAiRzCKwMDeSfz7nJwdifCty7eT/y56PeEDWQCES39wodVQcNOoC39iQ M9cVUpEpdFPZCmTvJY7Ndq5XuPFXrlmp4b1RFvXWhxR/rpV1Sp0N50B/FfWJaKmQOdRL QA1A67uR9njxhd4a2vQqVBseVV0R8+f0H2ksBrHQKo6OiV1xJD/EFsZXvTZ6HDjMWJIu mA== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hq9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:16:55 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:16:52 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:16:52 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id CFB623F7040; Wed, 1 Sep 2021 19:16:50 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:41 +0530 Message-ID: <20210902021505.17607-4-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: y55-lSHx8PxDPl5rGgzO1UXduCnTz6VR X-Proofpoint-GUID: y55-lSHx8PxDPl5rGgzO1UXduCnTz6VR X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 03/27] common/cnxk: allow reuse of SSO API for inline dev X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Rework interface of sso internal functions to use for nix inline dev's sso LF's. Signed-off-by: Nithin Dabilpuram --- drivers/common/cnxk/roc_sso.c | 52 ++++++++++++++++++++++++-------------- drivers/common/cnxk/roc_sso_priv.h | 9 +++++++ 2 files changed, 42 insertions(+), 19 deletions(-) diff --git a/drivers/common/cnxk/roc_sso.c b/drivers/common/cnxk/roc_sso.c index 1ccf262..bdf973f 100644 --- a/drivers/common/cnxk/roc_sso.c +++ b/drivers/common/cnxk/roc_sso.c @@ -6,11 +6,10 @@ #include "roc_priv.h" /* Private functions. */ -static int -sso_lf_alloc(struct roc_sso *roc_sso, enum sso_lf_type lf_type, uint16_t nb_lf, +int +sso_lf_alloc(struct dev *dev, enum sso_lf_type lf_type, uint16_t nb_lf, void **rsp) { - struct dev *dev = &roc_sso_to_sso_priv(roc_sso)->dev; int rc = -ENOSPC; switch (lf_type) { @@ -41,10 +40,9 @@ sso_lf_alloc(struct roc_sso *roc_sso, enum sso_lf_type lf_type, uint16_t nb_lf, return 0; } -static int -sso_lf_free(struct roc_sso *roc_sso, enum sso_lf_type lf_type, uint16_t nb_lf) +int +sso_lf_free(struct dev *dev, enum sso_lf_type lf_type, uint16_t nb_lf) { - struct dev *dev = &roc_sso_to_sso_priv(roc_sso)->dev; int rc = -ENOSPC; switch (lf_type) { @@ -152,7 +150,7 @@ sso_rsrc_get(struct roc_sso *roc_sso) return 0; } -static void +void sso_hws_link_modify(uint8_t hws, uintptr_t base, struct plt_bitmap *bmp, uint16_t hwgrp[], uint16_t n, uint16_t enable) { @@ -172,8 +170,10 @@ sso_hws_link_modify(uint8_t hws, uintptr_t base, struct plt_bitmap *bmp, k = k ? k : 4; for (j = 0; j < k; j++) { mask[j] = hwgrp[i + j] | enable << 14; - enable ? plt_bitmap_set(bmp, hwgrp[i + j]) : - plt_bitmap_clear(bmp, hwgrp[i + j]); + if (bmp) { + enable ? plt_bitmap_set(bmp, hwgrp[i + j]) : + plt_bitmap_clear(bmp, hwgrp[i + j]); + } plt_sso_dbg("HWS %d Linked to HWGRP %d", hws, hwgrp[i + j]); } @@ -388,10 +388,8 @@ roc_sso_hwgrp_qos_config(struct roc_sso *roc_sso, struct roc_sso_hwgrp_qos *qos, } int -roc_sso_hwgrp_alloc_xaq(struct roc_sso *roc_sso, uint32_t npa_aura_id, - uint16_t hwgrps) +sso_hwgrp_alloc_xaq(struct dev *dev, uint32_t npa_aura_id, uint16_t hwgrps) { - struct dev *dev = &roc_sso_to_sso_priv(roc_sso)->dev; struct sso_hw_setconfig *req; int rc = -ENOSPC; @@ -406,9 +404,17 @@ roc_sso_hwgrp_alloc_xaq(struct roc_sso *roc_sso, uint32_t npa_aura_id, } int -roc_sso_hwgrp_release_xaq(struct roc_sso *roc_sso, uint16_t hwgrps) +roc_sso_hwgrp_alloc_xaq(struct roc_sso *roc_sso, uint32_t npa_aura_id, + uint16_t hwgrps) { struct dev *dev = &roc_sso_to_sso_priv(roc_sso)->dev; + + return sso_hwgrp_alloc_xaq(dev, npa_aura_id, hwgrps); +} + +int +sso_hwgrp_release_xaq(struct dev *dev, uint16_t hwgrps) +{ struct sso_hw_xaq_release *req; req = mbox_alloc_msg_sso_hw_release_xaq_aura(dev->mbox); @@ -420,6 +426,14 @@ roc_sso_hwgrp_release_xaq(struct roc_sso *roc_sso, uint16_t hwgrps) } int +roc_sso_hwgrp_release_xaq(struct roc_sso *roc_sso, uint16_t hwgrps) +{ + struct dev *dev = &roc_sso_to_sso_priv(roc_sso)->dev; + + return sso_hwgrp_release_xaq(dev, hwgrps); +} + +int roc_sso_hwgrp_set_priority(struct roc_sso *roc_sso, uint16_t hwgrp, uint8_t weight, uint8_t affinity, uint8_t priority) { @@ -468,13 +482,13 @@ roc_sso_rsrc_init(struct roc_sso *roc_sso, uint8_t nb_hws, uint16_t nb_hwgrp) goto hwgrp_atch_fail; } - rc = sso_lf_alloc(roc_sso, SSO_LF_TYPE_HWS, nb_hws, NULL); + rc = sso_lf_alloc(&sso->dev, SSO_LF_TYPE_HWS, nb_hws, NULL); if (rc < 0) { plt_err("Unable to alloc SSO HWS LFs"); goto hws_alloc_fail; } - rc = sso_lf_alloc(roc_sso, SSO_LF_TYPE_HWGRP, nb_hwgrp, + rc = sso_lf_alloc(&sso->dev, SSO_LF_TYPE_HWGRP, nb_hwgrp, (void **)&rsp_hwgrp); if (rc < 0) { plt_err("Unable to alloc SSO HWGRP Lfs"); @@ -503,9 +517,9 @@ roc_sso_rsrc_init(struct roc_sso *roc_sso, uint8_t nb_hws, uint16_t nb_hwgrp) return 0; sso_msix_fail: - sso_lf_free(roc_sso, SSO_LF_TYPE_HWGRP, nb_hwgrp); + sso_lf_free(&sso->dev, SSO_LF_TYPE_HWGRP, nb_hwgrp); hwgrp_alloc_fail: - sso_lf_free(roc_sso, SSO_LF_TYPE_HWS, nb_hws); + sso_lf_free(&sso->dev, SSO_LF_TYPE_HWS, nb_hws); hws_alloc_fail: sso_rsrc_detach(roc_sso, SSO_LF_TYPE_HWGRP); hwgrp_atch_fail: @@ -523,8 +537,8 @@ roc_sso_rsrc_fini(struct roc_sso *roc_sso) sso_unregister_irqs_priv(roc_sso, &sso->pci_dev->intr_handle, roc_sso->nb_hws, roc_sso->nb_hwgrp); - sso_lf_free(roc_sso, SSO_LF_TYPE_HWS, roc_sso->nb_hws); - sso_lf_free(roc_sso, SSO_LF_TYPE_HWGRP, roc_sso->nb_hwgrp); + sso_lf_free(&sso->dev, SSO_LF_TYPE_HWS, roc_sso->nb_hws); + sso_lf_free(&sso->dev, SSO_LF_TYPE_HWGRP, roc_sso->nb_hwgrp); sso_rsrc_detach(roc_sso, SSO_LF_TYPE_HWS); sso_rsrc_detach(roc_sso, SSO_LF_TYPE_HWGRP); diff --git a/drivers/common/cnxk/roc_sso_priv.h b/drivers/common/cnxk/roc_sso_priv.h index 5361d4f..8dffa3f 100644 --- a/drivers/common/cnxk/roc_sso_priv.h +++ b/drivers/common/cnxk/roc_sso_priv.h @@ -39,6 +39,15 @@ roc_sso_to_sso_priv(struct roc_sso *roc_sso) return (struct sso *)&roc_sso->reserved[0]; } +/* SSO LF ops */ +int sso_lf_alloc(struct dev *dev, enum sso_lf_type lf_type, uint16_t nb_lf, + void **rsp); +int sso_lf_free(struct dev *dev, enum sso_lf_type lf_type, uint16_t nb_lf); +void sso_hws_link_modify(uint8_t hws, uintptr_t base, struct plt_bitmap *bmp, + uint16_t hwgrp[], uint16_t n, uint16_t enable); +int sso_hwgrp_alloc_xaq(struct dev *dev, uint32_t npa_aura_id, uint16_t hwgrps); +int sso_hwgrp_release_xaq(struct dev *dev, uint16_t hwgrps); + /* SSO IRQ */ int sso_register_irqs_priv(struct roc_sso *roc_sso, struct plt_intr_handle *handle, uint16_t nb_hws, From patchwork Thu Sep 2 02:14:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97730 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B931AA0C4C; Thu, 2 Sep 2021 04:17:12 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DDF394111E; Thu, 2 Sep 2021 04:16:59 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 3EB71410D8 for ; Thu, 2 Sep 2021 04:16:58 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181K549U025746 for ; Wed, 1 Sep 2021 19:16:57 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=jz/1ny09h49jED0nS+clwfTLCFwxiGiO7o1q6Of7CAU=; b=NgPIXHRAahozRFnZNfIqS3elEz6dlPKdIAu1qrMRzVoby+NHY/xnyRlW1oyLKodQ1jLo f6CB04j2Pe3AUtnNzhfNZOFc2tRhYHusjgKo3afLQq4KfYfcbtQW6n3+N/ZO+aBCSCbD a3SYF5f4cEWakQDXRB946mi48QG8X5Ozz5t6nGwRNxibWGZhtjuwvJRIdjrs6ZPDb/Rq V6EhJKTcuAx11sT40viwsxpQbXKhbBwnigI49vBoO0s5n1Zd3Je2bLLehhX8HGdpFhkE KQVw4B8xXp0d1kSNzD04zNxdcahSvBmzCUW/JKM8soHLnb9KgfB2L0tVZHvpxa6Almvf 8w== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com with ESMTP id 3atg8a91g9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:16:57 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:16:55 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:16:55 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 9E1325B692B; Wed, 1 Sep 2021 19:16:53 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:42 +0530 Message-ID: <20210902021505.17607-5-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: 0QM5hqwlifJZTAQr_R-2e1EpEPQ4VkLM X-Proofpoint-ORIG-GUID: 0QM5hqwlifJZTAQr_R-2e1EpEPQ4VkLM X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 04/27] common/cnxk: change nix debug API and queue API interface X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Change nix debug API and queue API interface for use by internal nix inline device initialization. Signed-off-by: Nithin Dabilpuram --- drivers/common/cnxk/roc_nix.c | 2 +- drivers/common/cnxk/roc_nix_debug.c | 118 +++++++++++++++++++++++++++--------- drivers/common/cnxk/roc_nix_priv.h | 16 +++++ drivers/common/cnxk/roc_nix_queue.c | 89 +++++++++++++++------------ 4 files changed, 159 insertions(+), 66 deletions(-) diff --git a/drivers/common/cnxk/roc_nix.c b/drivers/common/cnxk/roc_nix.c index 23d508b..3ab954e 100644 --- a/drivers/common/cnxk/roc_nix.c +++ b/drivers/common/cnxk/roc_nix.c @@ -300,7 +300,7 @@ sdp_lbk_id_update(struct plt_pci_device *pci_dev, struct nix *nix) } } -static inline uint64_t +uint64_t nix_get_blkaddr(struct dev *dev) { uint64_t reg; diff --git a/drivers/common/cnxk/roc_nix_debug.c b/drivers/common/cnxk/roc_nix_debug.c index 6e56513..9539bb9 100644 --- a/drivers/common/cnxk/roc_nix_debug.c +++ b/drivers/common/cnxk/roc_nix_debug.c @@ -110,17 +110,12 @@ roc_nix_lf_get_reg_count(struct roc_nix *roc_nix) } int -roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) +nix_lf_gen_reg_dump(uintptr_t nix_lf_base, uint64_t *data) { - struct nix *nix = roc_nix_to_nix_priv(roc_nix); - uintptr_t nix_lf_base = nix->base; bool dump_stdout; uint64_t reg; uint32_t i; - if (roc_nix == NULL) - return NIX_ERR_PARAM; - dump_stdout = data ? 0 : 1; for (i = 0; i < PLT_DIM(nix_lf_reg); i++) { @@ -131,8 +126,21 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) *data++ = reg; } + return i; +} + +int +nix_lf_stat_reg_dump(uintptr_t nix_lf_base, uint64_t *data, uint8_t lf_tx_stats, + uint8_t lf_rx_stats) +{ + uint32_t i, count = 0; + bool dump_stdout; + uint64_t reg; + + dump_stdout = data ? 0 : 1; + /* NIX_LF_TX_STATX */ - for (i = 0; i < nix->lf_tx_stats; i++) { + for (i = 0; i < lf_tx_stats; i++) { reg = plt_read64(nix_lf_base + NIX_LF_TX_STATX(i)); if (dump_stdout && reg) nix_dump("%32s_%d = 0x%" PRIx64, "NIX_LF_TX_STATX", i, @@ -140,9 +148,10 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) if (data) *data++ = reg; } + count += i; /* NIX_LF_RX_STATX */ - for (i = 0; i < nix->lf_rx_stats; i++) { + for (i = 0; i < lf_rx_stats; i++) { reg = plt_read64(nix_lf_base + NIX_LF_RX_STATX(i)); if (dump_stdout && reg) nix_dump("%32s_%d = 0x%" PRIx64, "NIX_LF_RX_STATX", i, @@ -151,8 +160,21 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) *data++ = reg; } + return count + i; +} + +int +nix_lf_int_reg_dump(uintptr_t nix_lf_base, uint64_t *data, uint16_t qints, + uint16_t cints) +{ + uint32_t i, count = 0; + bool dump_stdout; + uint64_t reg; + + dump_stdout = data ? 0 : 1; + /* NIX_LF_QINTX_CNT*/ - for (i = 0; i < nix->qints; i++) { + for (i = 0; i < qints; i++) { reg = plt_read64(nix_lf_base + NIX_LF_QINTX_CNT(i)); if (dump_stdout && reg) nix_dump("%32s_%d = 0x%" PRIx64, "NIX_LF_QINTX_CNT", i, @@ -160,9 +182,10 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) if (data) *data++ = reg; } + count += i; /* NIX_LF_QINTX_INT */ - for (i = 0; i < nix->qints; i++) { + for (i = 0; i < qints; i++) { reg = plt_read64(nix_lf_base + NIX_LF_QINTX_INT(i)); if (dump_stdout && reg) nix_dump("%32s_%d = 0x%" PRIx64, "NIX_LF_QINTX_INT", i, @@ -170,9 +193,10 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) if (data) *data++ = reg; } + count += i; /* NIX_LF_QINTX_ENA_W1S */ - for (i = 0; i < nix->qints; i++) { + for (i = 0; i < qints; i++) { reg = plt_read64(nix_lf_base + NIX_LF_QINTX_ENA_W1S(i)); if (dump_stdout && reg) nix_dump("%32s_%d = 0x%" PRIx64, "NIX_LF_QINTX_ENA_W1S", @@ -180,9 +204,10 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) if (data) *data++ = reg; } + count += i; /* NIX_LF_QINTX_ENA_W1C */ - for (i = 0; i < nix->qints; i++) { + for (i = 0; i < qints; i++) { reg = plt_read64(nix_lf_base + NIX_LF_QINTX_ENA_W1C(i)); if (dump_stdout && reg) nix_dump("%32s_%d = 0x%" PRIx64, "NIX_LF_QINTX_ENA_W1C", @@ -190,9 +215,10 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) if (data) *data++ = reg; } + count += i; /* NIX_LF_CINTX_CNT */ - for (i = 0; i < nix->cints; i++) { + for (i = 0; i < cints; i++) { reg = plt_read64(nix_lf_base + NIX_LF_CINTX_CNT(i)); if (dump_stdout && reg) nix_dump("%32s_%d = 0x%" PRIx64, "NIX_LF_CINTX_CNT", i, @@ -200,9 +226,10 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) if (data) *data++ = reg; } + count += i; /* NIX_LF_CINTX_WAIT */ - for (i = 0; i < nix->cints; i++) { + for (i = 0; i < cints; i++) { reg = plt_read64(nix_lf_base + NIX_LF_CINTX_WAIT(i)); if (dump_stdout && reg) nix_dump("%32s_%d = 0x%" PRIx64, "NIX_LF_CINTX_WAIT", i, @@ -210,9 +237,10 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) if (data) *data++ = reg; } + count += i; /* NIX_LF_CINTX_INT */ - for (i = 0; i < nix->cints; i++) { + for (i = 0; i < cints; i++) { reg = plt_read64(nix_lf_base + NIX_LF_CINTX_INT(i)); if (dump_stdout && reg) nix_dump("%32s_%d = 0x%" PRIx64, "NIX_LF_CINTX_INT", i, @@ -220,9 +248,10 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) if (data) *data++ = reg; } + count += i; /* NIX_LF_CINTX_INT_W1S */ - for (i = 0; i < nix->cints; i++) { + for (i = 0; i < cints; i++) { reg = plt_read64(nix_lf_base + NIX_LF_CINTX_INT_W1S(i)); if (dump_stdout && reg) nix_dump("%32s_%d = 0x%" PRIx64, "NIX_LF_CINTX_INT_W1S", @@ -230,9 +259,10 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) if (data) *data++ = reg; } + count += i; /* NIX_LF_CINTX_ENA_W1S */ - for (i = 0; i < nix->cints; i++) { + for (i = 0; i < cints; i++) { reg = plt_read64(nix_lf_base + NIX_LF_CINTX_ENA_W1S(i)); if (dump_stdout && reg) nix_dump("%32s_%d = 0x%" PRIx64, "NIX_LF_CINTX_ENA_W1S", @@ -240,9 +270,10 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) if (data) *data++ = reg; } + count += i; /* NIX_LF_CINTX_ENA_W1C */ - for (i = 0; i < nix->cints; i++) { + for (i = 0; i < cints; i++) { reg = plt_read64(nix_lf_base + NIX_LF_CINTX_ENA_W1C(i)); if (dump_stdout && reg) nix_dump("%32s_%d = 0x%" PRIx64, "NIX_LF_CINTX_ENA_W1C", @@ -250,12 +281,40 @@ roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) if (data) *data++ = reg; } + + return count + i; +} + +int +roc_nix_lf_reg_dump(struct roc_nix *roc_nix, uint64_t *data) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + bool dump_stdout = data ? 0 : 1; + uintptr_t nix_base; + uint32_t i; + + if (roc_nix == NULL) + return NIX_ERR_PARAM; + + nix_base = nix->base; + /* General registers */ + i = nix_lf_gen_reg_dump(nix_base, data); + + /* Rx, Tx stat registers */ + i += nix_lf_stat_reg_dump(nix_base, dump_stdout ? NULL : &data[i], + nix->lf_tx_stats, nix->lf_rx_stats); + + /* Intr registers */ + i += nix_lf_int_reg_dump(nix_base, dump_stdout ? NULL : &data[i], + nix->qints, nix->cints); + return 0; } -static int -nix_q_ctx_get(struct mbox *mbox, uint8_t ctype, uint16_t qid, __io void **ctx_p) +int +nix_q_ctx_get(struct dev *dev, uint8_t ctype, uint16_t qid, __io void **ctx_p) { + struct mbox *mbox = dev->mbox; int rc; if (roc_model_is_cn9k()) { @@ -485,7 +544,7 @@ nix_cn9k_lf_rq_dump(__io struct nix_rq_ctx_s *ctx) nix_dump("W10: re_pkts \t\t\t0x%" PRIx64 "\n", (uint64_t)ctx->re_pkts); } -static inline void +void nix_lf_rq_dump(__io struct nix_cn10k_rq_ctx_s *ctx) { nix_dump("W0: wqe_aura \t\t\t%d\nW0: len_ol3_dis \t\t\t%d", @@ -595,12 +654,12 @@ roc_nix_queues_ctx_dump(struct roc_nix *roc_nix) { struct nix *nix = roc_nix_to_nix_priv(roc_nix); int rc = -1, q, rq = nix->nb_rx_queues; - struct mbox *mbox = (&nix->dev)->mbox; struct npa_aq_enq_rsp *npa_rsp; struct npa_aq_enq_req *npa_aq; - volatile void *ctx; + struct dev *dev = &nix->dev; int sq = nix->nb_tx_queues; struct npa_lf *npa_lf; + volatile void *ctx; uint32_t sqb_aura; npa_lf = idev_npa_obj_get(); @@ -608,7 +667,7 @@ roc_nix_queues_ctx_dump(struct roc_nix *roc_nix) return NPA_ERR_DEVICE_NOT_BOUNDED; for (q = 0; q < rq; q++) { - rc = nix_q_ctx_get(mbox, NIX_AQ_CTYPE_CQ, q, &ctx); + rc = nix_q_ctx_get(dev, NIX_AQ_CTYPE_CQ, q, &ctx); if (rc) { plt_err("Failed to get cq context"); goto fail; @@ -619,7 +678,7 @@ roc_nix_queues_ctx_dump(struct roc_nix *roc_nix) } for (q = 0; q < rq; q++) { - rc = nix_q_ctx_get(mbox, NIX_AQ_CTYPE_RQ, q, &ctx); + rc = nix_q_ctx_get(dev, NIX_AQ_CTYPE_RQ, q, &ctx); if (rc) { plt_err("Failed to get rq context"); goto fail; @@ -633,7 +692,7 @@ roc_nix_queues_ctx_dump(struct roc_nix *roc_nix) } for (q = 0; q < sq; q++) { - rc = nix_q_ctx_get(mbox, NIX_AQ_CTYPE_SQ, q, &ctx); + rc = nix_q_ctx_get(dev, NIX_AQ_CTYPE_SQ, q, &ctx); if (rc) { plt_err("Failed to get sq context"); goto fail; @@ -686,11 +745,13 @@ roc_nix_cqe_dump(const struct nix_cqe_hdr_s *cq) { const union nix_rx_parse_u *rx = (const union nix_rx_parse_u *)((const uint64_t *)cq + 1); + const uint64_t *sgs = (const uint64_t *)(rx + 1); + int i; nix_dump("tag \t\t0x%x\tq \t\t%d\t\tnode \t\t%d\tcqe_type \t%d", cq->tag, cq->q, cq->node, cq->cqe_type); - nix_dump("W0: chan \t%d\t\tdesc_sizem1 \t%d", rx->chan, + nix_dump("W0: chan \t0x%x\t\tdesc_sizem1 \t%d", rx->chan, rx->desc_sizem1); nix_dump("W0: imm_copy \t%d\t\texpress \t%d", rx->imm_copy, rx->express); @@ -731,6 +792,9 @@ roc_nix_cqe_dump(const struct nix_cqe_hdr_s *cq) nix_dump("W5: vtag0_ptr \t%d\t\tvtag1_ptr \t%d\t\tflow_key_alg \t%d", rx->vtag0_ptr, rx->vtag1_ptr, rx->flow_key_alg); + + for (i = 0; i < (rx->desc_sizem1 + 1) << 1; i++) + nix_dump("sg[%u] = %p", i, (void *)sgs[i]); } void diff --git a/drivers/common/cnxk/roc_nix_priv.h b/drivers/common/cnxk/roc_nix_priv.h index 9dc0c88..79c15ea 100644 --- a/drivers/common/cnxk/roc_nix_priv.h +++ b/drivers/common/cnxk/roc_nix_priv.h @@ -348,6 +348,12 @@ int nix_tm_sq_sched_conf(struct nix *nix, struct nix_tm_node *node, bool rr_quantum_only); int nix_tm_prepare_rate_limited_tree(struct roc_nix *roc_nix); +int nix_rq_cn9k_cfg(struct dev *dev, struct roc_nix_rq *rq, uint16_t qints, + bool cfg, bool ena); +int nix_rq_cfg(struct dev *dev, struct roc_nix_rq *rq, uint16_t qints, bool cfg, + bool ena); +int nix_rq_ena_dis(struct dev *dev, struct roc_nix_rq *rq, bool enable); + /* * TM priv utils. */ @@ -393,4 +399,14 @@ void nix_tm_node_free(struct nix_tm_node *node); struct nix_tm_shaper_profile *nix_tm_shaper_profile_alloc(void); void nix_tm_shaper_profile_free(struct nix_tm_shaper_profile *profile); +uint64_t nix_get_blkaddr(struct dev *dev); +void nix_lf_rq_dump(__io struct nix_cn10k_rq_ctx_s *ctx); +int nix_lf_gen_reg_dump(uintptr_t nix_lf_base, uint64_t *data); +int nix_lf_stat_reg_dump(uintptr_t nix_lf_base, uint64_t *data, + uint8_t lf_tx_stats, uint8_t lf_rx_stats); +int nix_lf_int_reg_dump(uintptr_t nix_lf_base, uint64_t *data, uint16_t qints, + uint16_t cints); +int nix_q_ctx_get(struct dev *dev, uint8_t ctype, uint16_t qid, + __io void **ctx_p); + #endif /* _ROC_NIX_PRIV_H_ */ diff --git a/drivers/common/cnxk/roc_nix_queue.c b/drivers/common/cnxk/roc_nix_queue.c index 7e2f86e..de63361 100644 --- a/drivers/common/cnxk/roc_nix_queue.c +++ b/drivers/common/cnxk/roc_nix_queue.c @@ -27,46 +27,54 @@ nix_qsize_clampup(uint32_t val) } int +nix_rq_ena_dis(struct dev *dev, struct roc_nix_rq *rq, bool enable) +{ + struct mbox *mbox = dev->mbox; + + /* Pkts will be dropped silently if RQ is disabled */ + if (roc_model_is_cn9k()) { + struct nix_aq_enq_req *aq; + + aq = mbox_alloc_msg_nix_aq_enq(mbox); + aq->qidx = rq->qid; + aq->ctype = NIX_AQ_CTYPE_RQ; + aq->op = NIX_AQ_INSTOP_WRITE; + + aq->rq.ena = enable; + aq->rq_mask.ena = ~(aq->rq_mask.ena); + } else { + struct nix_cn10k_aq_enq_req *aq; + + aq = mbox_alloc_msg_nix_cn10k_aq_enq(mbox); + aq->qidx = rq->qid; + aq->ctype = NIX_AQ_CTYPE_RQ; + aq->op = NIX_AQ_INSTOP_WRITE; + + aq->rq.ena = enable; + aq->rq_mask.ena = ~(aq->rq_mask.ena); + } + + return mbox_process(mbox); +} + +int roc_nix_rq_ena_dis(struct roc_nix_rq *rq, bool enable) { struct nix *nix = roc_nix_to_nix_priv(rq->roc_nix); - struct mbox *mbox = (&nix->dev)->mbox; int rc; - /* Pkts will be dropped silently if RQ is disabled */ - if (roc_model_is_cn9k()) { - struct nix_aq_enq_req *aq; - - aq = mbox_alloc_msg_nix_aq_enq(mbox); - aq->qidx = rq->qid; - aq->ctype = NIX_AQ_CTYPE_RQ; - aq->op = NIX_AQ_INSTOP_WRITE; - - aq->rq.ena = enable; - aq->rq_mask.ena = ~(aq->rq_mask.ena); - } else { - struct nix_cn10k_aq_enq_req *aq; - - aq = mbox_alloc_msg_nix_cn10k_aq_enq(mbox); - aq->qidx = rq->qid; - aq->ctype = NIX_AQ_CTYPE_RQ; - aq->op = NIX_AQ_INSTOP_WRITE; - - aq->rq.ena = enable; - aq->rq_mask.ena = ~(aq->rq_mask.ena); - } - - rc = mbox_process(mbox); + rc = nix_rq_ena_dis(&nix->dev, rq, enable); if (roc_model_is_cn10k()) plt_write64(rq->qid, nix->base + NIX_LF_OP_VWQE_FLUSH); return rc; } -static int -rq_cn9k_cfg(struct nix *nix, struct roc_nix_rq *rq, bool cfg, bool ena) +int +nix_rq_cn9k_cfg(struct dev *dev, struct roc_nix_rq *rq, uint16_t qints, + bool cfg, bool ena) { - struct mbox *mbox = (&nix->dev)->mbox; + struct mbox *mbox = dev->mbox; struct nix_aq_enq_req *aq; aq = mbox_alloc_msg_nix_aq_enq(mbox); @@ -116,7 +124,7 @@ rq_cn9k_cfg(struct nix *nix, struct roc_nix_rq *rq, bool cfg, bool ena) aq->rq.xqe_imm_size = 0; /* No pkt data copy to CQE */ aq->rq.rq_int_ena = 0; /* Many to one reduction */ - aq->rq.qint_idx = rq->qid % nix->qints; + aq->rq.qint_idx = rq->qid % qints; aq->rq.xqe_drop_ena = 1; /* If RED enabled, then fill enable for all cases */ @@ -177,11 +185,12 @@ rq_cn9k_cfg(struct nix *nix, struct roc_nix_rq *rq, bool cfg, bool ena) return 0; } -static int -rq_cfg(struct nix *nix, struct roc_nix_rq *rq, bool cfg, bool ena) +int +nix_rq_cfg(struct dev *dev, struct roc_nix_rq *rq, uint16_t qints, bool cfg, + bool ena) { - struct mbox *mbox = (&nix->dev)->mbox; struct nix_cn10k_aq_enq_req *aq; + struct mbox *mbox = dev->mbox; aq = mbox_alloc_msg_nix_cn10k_aq_enq(mbox); aq->qidx = rq->qid; @@ -218,8 +227,10 @@ rq_cfg(struct nix *nix, struct roc_nix_rq *rq, bool cfg, bool ena) aq->rq.cq = rq->qid; } - if (rq->ipsech_ena) + if (rq->ipsech_ena) { aq->rq.ipsech_ena = 1; + aq->rq.ipsecd_drop_en = 1; + } aq->rq.lpb_aura = roc_npa_aura_handle_to_aura(rq->aura_handle); @@ -258,7 +269,7 @@ rq_cfg(struct nix *nix, struct roc_nix_rq *rq, bool cfg, bool ena) aq->rq.xqe_imm_size = 0; /* No pkt data copy to CQE */ aq->rq.rq_int_ena = 0; /* Many to one reduction */ - aq->rq.qint_idx = rq->qid % nix->qints; + aq->rq.qint_idx = rq->qid % qints; aq->rq.xqe_drop_ena = 1; /* If RED enabled, then fill enable for all cases */ @@ -357,6 +368,7 @@ roc_nix_rq_init(struct roc_nix *roc_nix, struct roc_nix_rq *rq, bool ena) struct nix *nix = roc_nix_to_nix_priv(roc_nix); struct mbox *mbox = (&nix->dev)->mbox; bool is_cn9k = roc_model_is_cn9k(); + struct dev *dev = &nix->dev; int rc; if (roc_nix == NULL || rq == NULL) @@ -368,9 +380,9 @@ roc_nix_rq_init(struct roc_nix *roc_nix, struct roc_nix_rq *rq, bool ena) rq->roc_nix = roc_nix; if (is_cn9k) - rc = rq_cn9k_cfg(nix, rq, false, ena); + rc = nix_rq_cn9k_cfg(dev, rq, nix->qints, false, ena); else - rc = rq_cfg(nix, rq, false, ena); + rc = nix_rq_cfg(dev, rq, nix->qints, false, ena); if (rc) return rc; @@ -384,6 +396,7 @@ roc_nix_rq_modify(struct roc_nix *roc_nix, struct roc_nix_rq *rq, bool ena) struct nix *nix = roc_nix_to_nix_priv(roc_nix); struct mbox *mbox = (&nix->dev)->mbox; bool is_cn9k = roc_model_is_cn9k(); + struct dev *dev = &nix->dev; int rc; if (roc_nix == NULL || rq == NULL) @@ -395,9 +408,9 @@ roc_nix_rq_modify(struct roc_nix *roc_nix, struct roc_nix_rq *rq, bool ena) rq->roc_nix = roc_nix; if (is_cn9k) - rc = rq_cn9k_cfg(nix, rq, true, ena); + rc = nix_rq_cn9k_cfg(dev, rq, nix->qints, true, ena); else - rc = rq_cfg(nix, rq, true, ena); + rc = nix_rq_cfg(dev, rq, nix->qints, true, ena); if (rc) return rc; From patchwork Thu Sep 2 02:14:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97731 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2D3A3A0C4C; Thu, 2 Sep 2021 04:17:19 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 086FB41135; Thu, 2 Sep 2021 04:17:03 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 54E00410FF for ; Thu, 2 Sep 2021 04:17:01 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQSf6012709 for ; Wed, 1 Sep 2021 19:17:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=XtoxIGCYV0JlXkigAP7eTr7Qcs3yKGcPhy+oiLE+LuA=; b=hdXuB3YbrHdg8ljrKhpWJ+9viXg8KJB2+vvBB1hFIqziP5jbYSPyiMmZXzpv4zl3F4XZ FnLwrWjgr2wCAvUdDAoEkUWp+3oYRf+bBUBXMJfPtZjZbIAyToDSaKL5sGLUUGoGn5Z8 Gr2tLFI04qxzk8Yl8Vb2EnlsTHUeNC21v7Nq2oeGzKIrygRTCBsWLtGn6qYJdmLG67Js jN+J0/ntE5SY7x79lhxflTNKku2bQGXMniF7gGcyszwGSZqtj6vVdrPLGQqzi6F0Sl/U a21SFRyuBR7TWzwT9QvFWWpLqZKFRExyinzwjMa12zGqyImyfnIoxRI2b/Ea2gDPoQAs wA== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hqx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:00 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:16:58 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:16:58 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 61BD93F7040; Wed, 1 Sep 2021 19:16:56 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:43 +0530 Message-ID: <20210902021505.17607-6-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: xnc0t7npr_XVNKl9jinkpiPzj6JvoqgA X-Proofpoint-GUID: xnc0t7npr_XVNKl9jinkpiPzj6JvoqgA X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 05/27] common/cnxk: add nix inline device irq API X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add API to setup nix inline device IRQ's. Signed-off-by: Nithin Dabilpuram --- drivers/common/cnxk/meson.build | 1 + drivers/common/cnxk/roc_api.h | 3 + drivers/common/cnxk/roc_irq.c | 7 +- drivers/common/cnxk/roc_nix_inl.h | 10 + drivers/common/cnxk/roc_nix_inl_dev_irq.c | 359 ++++++++++++++++++++++++++++++ drivers/common/cnxk/roc_nix_inl_priv.h | 57 +++++ drivers/common/cnxk/roc_platform.h | 9 +- drivers/common/cnxk/roc_priv.h | 3 + 8 files changed, 442 insertions(+), 7 deletions(-) create mode 100644 drivers/common/cnxk/roc_nix_inl.h create mode 100644 drivers/common/cnxk/roc_nix_inl_dev_irq.c create mode 100644 drivers/common/cnxk/roc_nix_inl_priv.h diff --git a/drivers/common/cnxk/meson.build b/drivers/common/cnxk/meson.build index 8a551d1..207ca00 100644 --- a/drivers/common/cnxk/meson.build +++ b/drivers/common/cnxk/meson.build @@ -28,6 +28,7 @@ sources = files( 'roc_nix_debug.c', 'roc_nix_fc.c', 'roc_nix_irq.c', + 'roc_nix_inl_dev_irq.c', 'roc_nix_mac.c', 'roc_nix_mcast.c', 'roc_nix_npc.c', diff --git a/drivers/common/cnxk/roc_api.h b/drivers/common/cnxk/roc_api.h index 7dec845..c1af95e 100644 --- a/drivers/common/cnxk/roc_api.h +++ b/drivers/common/cnxk/roc_api.h @@ -129,4 +129,7 @@ /* HASH computation */ #include "roc_hash.h" +/* NIX Inline dev */ +#include "roc_nix_inl.h" + #endif /* _ROC_API_H_ */ diff --git a/drivers/common/cnxk/roc_irq.c b/drivers/common/cnxk/roc_irq.c index 4c2b4c3..28fe691 100644 --- a/drivers/common/cnxk/roc_irq.c +++ b/drivers/common/cnxk/roc_irq.c @@ -138,9 +138,10 @@ dev_irq_register(struct plt_intr_handle *intr_handle, plt_intr_callback_fn cb, irq_init(intr_handle); } - if (vec > intr_handle->max_intr) { - plt_err("Vector=%d greater than max_intr=%d", vec, - intr_handle->max_intr); + if (vec > intr_handle->max_intr || vec >= PLT_DIM(intr_handle->efds)) { + plt_err("Vector=%d greater than max_intr=%d or " + "max_efd=%" PRIu64, + vec, intr_handle->max_intr, PLT_DIM(intr_handle->efds)); return -EINVAL; } diff --git a/drivers/common/cnxk/roc_nix_inl.h b/drivers/common/cnxk/roc_nix_inl.h new file mode 100644 index 0000000..1ec3dda --- /dev/null +++ b/drivers/common/cnxk/roc_nix_inl.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ +#ifndef _ROC_NIX_INL_H_ +#define _ROC_NIX_INL_H_ + +/* Inline device SSO Work callback */ +typedef void (*roc_nix_inl_sso_work_cb_t)(uint64_t *gw, void *args); + +#endif /* _ROC_NIX_INL_H_ */ diff --git a/drivers/common/cnxk/roc_nix_inl_dev_irq.c b/drivers/common/cnxk/roc_nix_inl_dev_irq.c new file mode 100644 index 0000000..25ed42f --- /dev/null +++ b/drivers/common/cnxk/roc_nix_inl_dev_irq.c @@ -0,0 +1,359 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "roc_api.h" +#include "roc_priv.h" + +static void +nix_inl_sso_work_cb(struct nix_inl_dev *inl_dev) +{ + uintptr_t getwrk_op = inl_dev->ssow_base + SSOW_LF_GWS_OP_GET_WORK0; + uintptr_t tag_wqe_op = inl_dev->ssow_base + SSOW_LF_GWS_WQE0; + uint32_t wdata = BIT(16) | 1; + union { + __uint128_t get_work; + uint64_t u64[2]; + } gw; + uint64_t work; + +again: + /* Try to do get work */ + gw.get_work = wdata; + plt_write64(gw.u64[0], getwrk_op); + do { + roc_load_pair(gw.u64[0], gw.u64[1], tag_wqe_op); + } while (gw.u64[0] & BIT_ULL(63)); + + work = gw.u64[1]; + /* Do we have any work? */ + if (work) { + if (inl_dev->work_cb) + inl_dev->work_cb(gw.u64, inl_dev->cb_args); + else + plt_warn("Undelivered inl dev work gw0: %p gw1: %p", + (void *)gw.u64[0], (void *)gw.u64[1]); + goto again; + } + + plt_atomic_thread_fence(__ATOMIC_ACQ_REL); +} + +static int +nix_inl_nix_reg_dump(struct nix_inl_dev *inl_dev) +{ + uintptr_t nix_base = inl_dev->nix_base; + + /* General registers */ + nix_lf_gen_reg_dump(nix_base, NULL); + + /* Rx, Tx stat registers */ + nix_lf_stat_reg_dump(nix_base, NULL, inl_dev->lf_tx_stats, + inl_dev->lf_rx_stats); + + /* Intr registers */ + nix_lf_int_reg_dump(nix_base, NULL, inl_dev->qints, inl_dev->cints); + + return 0; +} + +static void +nix_inl_sso_hwgrp_irq(void *param) +{ + struct nix_inl_dev *inl_dev = (struct nix_inl_dev *)param; + uintptr_t sso_base = inl_dev->sso_base; + uint64_t intr; + + intr = plt_read64(sso_base + SSO_LF_GGRP_INT); + if (intr == 0) + return; + + /* Check for work executable interrupt */ + if (intr & BIT(1)) + nix_inl_sso_work_cb(inl_dev); + + if (!(intr & BIT(1))) + plt_err("GGRP 0 GGRP_INT=0x%" PRIx64 "", intr); + + /* Clear interrupt */ + plt_write64(intr, sso_base + SSO_LF_GGRP_INT); +} + +static void +nix_inl_sso_hws_irq(void *param) +{ + struct nix_inl_dev *inl_dev = (struct nix_inl_dev *)param; + uintptr_t ssow_base = inl_dev->ssow_base; + uint64_t intr; + + intr = plt_read64(ssow_base + SSOW_LF_GWS_INT); + if (intr == 0) + return; + + plt_err("GWS 0 GWS_INT=0x%" PRIx64 "", intr); + + /* Clear interrupt */ + plt_write64(intr, ssow_base + SSOW_LF_GWS_INT); +} + +int +nix_inl_sso_register_irqs(struct nix_inl_dev *inl_dev) +{ + struct plt_intr_handle *handle = &inl_dev->pci_dev->intr_handle; + uintptr_t ssow_base = inl_dev->ssow_base; + uintptr_t sso_base = inl_dev->sso_base; + uint16_t sso_msixoff, ssow_msixoff; + int rc; + + ssow_msixoff = inl_dev->ssow_msixoff; + sso_msixoff = inl_dev->sso_msixoff; + if (sso_msixoff == MSIX_VECTOR_INVALID || + ssow_msixoff == MSIX_VECTOR_INVALID) { + plt_err("Invalid SSO/SSOW MSIX offsets (0x%x, 0x%x)", + sso_msixoff, ssow_msixoff); + return -EINVAL; + } + + /* + * Setup SSOW interrupt + */ + + /* Clear SSOW interrupt enable */ + plt_write64(~0ull, ssow_base + SSOW_LF_GWS_INT_ENA_W1C); + /* Register interrupt with vfio */ + rc = dev_irq_register(handle, nix_inl_sso_hws_irq, inl_dev, + ssow_msixoff + SSOW_LF_INT_VEC_IOP); + /* Set SSOW interrupt enable */ + plt_write64(~0ull, ssow_base + SSOW_LF_GWS_INT_ENA_W1S); + + /* + * Setup SSO/HWGRP interrupt + */ + + /* Clear SSO interrupt enable */ + plt_write64(~0ull, sso_base + SSO_LF_GGRP_INT_ENA_W1C); + /* Register IRQ */ + rc |= dev_irq_register(handle, nix_inl_sso_hwgrp_irq, (void *)inl_dev, + sso_msixoff + SSO_LF_INT_VEC_GRP); + /* Enable hw interrupt */ + plt_write64(~0ull, sso_base + SSO_LF_GGRP_INT_ENA_W1S); + + /* Setup threshold for work exec interrupt to 1 wqe in IAQ */ + plt_write64(0x1ull, sso_base + SSO_LF_GGRP_INT_THR); + + return rc; +} + +void +nix_inl_sso_unregister_irqs(struct nix_inl_dev *inl_dev) +{ + struct plt_intr_handle *handle = &inl_dev->pci_dev->intr_handle; + uintptr_t ssow_base = inl_dev->ssow_base; + uintptr_t sso_base = inl_dev->sso_base; + uint16_t sso_msixoff, ssow_msixoff; + + ssow_msixoff = inl_dev->ssow_msixoff; + sso_msixoff = inl_dev->sso_msixoff; + + /* Clear SSOW interrupt enable */ + plt_write64(~0ull, ssow_base + SSOW_LF_GWS_INT_ENA_W1C); + /* Clear SSO/HWGRP interrupt enable */ + plt_write64(~0ull, sso_base + SSO_LF_GGRP_INT_ENA_W1C); + /* Clear SSO threshold */ + plt_write64(0, sso_base + SSO_LF_GGRP_INT_THR); + + /* Unregister IRQ */ + dev_irq_unregister(handle, nix_inl_sso_hws_irq, (void *)inl_dev, + ssow_msixoff + SSOW_LF_INT_VEC_IOP); + dev_irq_unregister(handle, nix_inl_sso_hwgrp_irq, (void *)inl_dev, + sso_msixoff + SSO_LF_INT_VEC_GRP); +} + +static void +nix_inl_nix_q_irq(void *param) +{ + struct nix_inl_dev *inl_dev = (struct nix_inl_dev *)param; + uintptr_t nix_base = inl_dev->nix_base; + struct dev *dev = &inl_dev->dev; + volatile void *ctx; + uint64_t reg, intr; + uint8_t irq; + int rc; + + intr = plt_read64(nix_base + NIX_LF_QINTX_INT(0)); + if (intr == 0) + return; + + plt_err("Queue_intr=0x%" PRIx64 " qintx 0 pf=%d, vf=%d", intr, dev->pf, + dev->vf); + + /* Get and clear RQ0 interrupt */ + reg = roc_atomic64_add_nosync(0, + (int64_t *)(nix_base + NIX_LF_RQ_OP_INT)); + if (reg & BIT_ULL(42) /* OP_ERR */) { + plt_err("Failed to get rq_int"); + return; + } + irq = reg & 0xff; + plt_write64(0 | irq, nix_base + NIX_LF_RQ_OP_INT); + + if (irq & BIT_ULL(NIX_RQINT_DROP)) + plt_err("RQ=0 NIX_RQINT_DROP"); + + if (irq & BIT_ULL(NIX_RQINT_RED)) + plt_err("RQ=0 NIX_RQINT_RED"); + + /* Clear interrupt */ + plt_write64(intr, nix_base + NIX_LF_QINTX_INT(0)); + + /* Dump registers to std out */ + nix_inl_nix_reg_dump(inl_dev); + + /* Dump RQ 0 */ + rc = nix_q_ctx_get(dev, NIX_AQ_CTYPE_RQ, 0, &ctx); + if (rc) { + plt_err("Failed to get rq context"); + return; + } + nix_lf_rq_dump(ctx); +} + +static void +nix_inl_nix_ras_irq(void *param) +{ + struct nix_inl_dev *inl_dev = (struct nix_inl_dev *)param; + uintptr_t nix_base = inl_dev->nix_base; + struct dev *dev = &inl_dev->dev; + volatile void *ctx; + uint64_t intr; + int rc; + + intr = plt_read64(nix_base + NIX_LF_RAS); + if (intr == 0) + return; + + plt_err("Ras_intr=0x%" PRIx64 " pf=%d, vf=%d", intr, dev->pf, dev->vf); + /* Clear interrupt */ + plt_write64(intr, nix_base + NIX_LF_RAS); + + /* Dump registers to std out */ + nix_inl_nix_reg_dump(inl_dev); + + /* Dump RQ 0 */ + rc = nix_q_ctx_get(dev, NIX_AQ_CTYPE_RQ, 0, &ctx); + if (rc) { + plt_err("Failed to get rq context"); + return; + } + nix_lf_rq_dump(ctx); +} + +static void +nix_inl_nix_err_irq(void *param) +{ + struct nix_inl_dev *inl_dev = (struct nix_inl_dev *)param; + uintptr_t nix_base = inl_dev->nix_base; + struct dev *dev = &inl_dev->dev; + volatile void *ctx; + uint64_t intr; + int rc; + + intr = plt_read64(nix_base + NIX_LF_ERR_INT); + if (intr == 0) + return; + + plt_err("Err_irq=0x%" PRIx64 " pf=%d, vf=%d", intr, dev->pf, dev->vf); + + /* Clear interrupt */ + plt_write64(intr, nix_base + NIX_LF_ERR_INT); + + /* Dump registers to std out */ + nix_inl_nix_reg_dump(inl_dev); + + /* Dump RQ 0 */ + rc = nix_q_ctx_get(dev, NIX_AQ_CTYPE_RQ, 0, &ctx); + if (rc) { + plt_err("Failed to get rq context"); + return; + } + nix_lf_rq_dump(ctx); +} + +int +nix_inl_nix_register_irqs(struct nix_inl_dev *inl_dev) +{ + struct plt_intr_handle *handle = &inl_dev->pci_dev->intr_handle; + uintptr_t nix_base = inl_dev->nix_base; + uint16_t msixoff; + int rc; + + msixoff = inl_dev->nix_msixoff; + if (msixoff == MSIX_VECTOR_INVALID) { + plt_err("Invalid NIXLF MSIX vector offset: 0x%x", msixoff); + return -EINVAL; + } + + /* Disable err interrupts */ + plt_write64(~0ull, nix_base + NIX_LF_ERR_INT_ENA_W1C); + /* DIsable RAS interrupts */ + plt_write64(~0ull, nix_base + NIX_LF_RAS_ENA_W1C); + + /* Register err irq */ + rc = dev_irq_register(handle, nix_inl_nix_err_irq, inl_dev, + msixoff + NIX_LF_INT_VEC_ERR_INT); + rc |= dev_irq_register(handle, nix_inl_nix_ras_irq, inl_dev, + msixoff + NIX_LF_INT_VEC_POISON); + + /* Enable all nix lf error irqs except RQ_DISABLED and CQ_DISABLED */ + plt_write64(~(BIT_ULL(11) | BIT_ULL(24)), + nix_base + NIX_LF_ERR_INT_ENA_W1S); + /* Enable RAS interrupts */ + plt_write64(~0ull, nix_base + NIX_LF_RAS_ENA_W1S); + + /* Setup queue irq for RQ 0 */ + + /* Clear QINT CNT, interrupt */ + plt_write64(0, nix_base + NIX_LF_QINTX_CNT(0)); + plt_write64(~0ull, nix_base + NIX_LF_QINTX_ENA_W1C(0)); + + /* Register queue irq vector */ + rc |= dev_irq_register(handle, nix_inl_nix_q_irq, inl_dev, + msixoff + NIX_LF_INT_VEC_QINT_START); + + plt_write64(0, nix_base + NIX_LF_QINTX_CNT(0)); + plt_write64(0, nix_base + NIX_LF_QINTX_INT(0)); + /* Enable QINT interrupt */ + plt_write64(~0ull, nix_base + NIX_LF_QINTX_ENA_W1S(0)); + + return rc; +} + +void +nix_inl_nix_unregister_irqs(struct nix_inl_dev *inl_dev) +{ + struct plt_intr_handle *handle = &inl_dev->pci_dev->intr_handle; + uintptr_t nix_base = inl_dev->nix_base; + uint16_t msixoff; + + msixoff = inl_dev->nix_msixoff; + /* Disable err interrupts */ + plt_write64(~0ull, nix_base + NIX_LF_ERR_INT_ENA_W1C); + /* DIsable RAS interrupts */ + plt_write64(~0ull, nix_base + NIX_LF_RAS_ENA_W1C); + + dev_irq_unregister(handle, nix_inl_nix_err_irq, inl_dev, + msixoff + NIX_LF_INT_VEC_ERR_INT); + dev_irq_unregister(handle, nix_inl_nix_ras_irq, inl_dev, + msixoff + NIX_LF_INT_VEC_POISON); + + /* Clear QINT CNT */ + plt_write64(0, nix_base + NIX_LF_QINTX_CNT(0)); + plt_write64(0, nix_base + NIX_LF_QINTX_INT(0)); + + /* Disable QINT interrupt */ + plt_write64(~0ull, nix_base + NIX_LF_QINTX_ENA_W1C(0)); + + /* Unregister queue irq vector */ + dev_irq_unregister(handle, nix_inl_nix_q_irq, inl_dev, + msixoff + NIX_LF_INT_VEC_QINT_START); +} diff --git a/drivers/common/cnxk/roc_nix_inl_priv.h b/drivers/common/cnxk/roc_nix_inl_priv.h new file mode 100644 index 0000000..f424009 --- /dev/null +++ b/drivers/common/cnxk/roc_nix_inl_priv.h @@ -0,0 +1,57 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ +#ifndef _ROC_NIX_INL_PRIV_H_ +#define _ROC_NIX_INL_PRIV_H_ + +struct nix_inl_dev { + /* Base device object */ + struct dev dev; + + /* PCI device */ + struct plt_pci_device *pci_dev; + + /* LF specific BAR2 regions */ + uintptr_t nix_base; + uintptr_t ssow_base; + uintptr_t sso_base; + + /* MSIX vector offsets */ + uint16_t nix_msixoff; + uint16_t ssow_msixoff; + uint16_t sso_msixoff; + + /* SSO data */ + uint32_t xaq_buf_size; + uint32_t xae_waes; + uint32_t iue; + uint64_t xaq_aura; + void *xaq_mem; + roc_nix_inl_sso_work_cb_t work_cb; + void *cb_args; + + /* NIX data */ + uint8_t lf_tx_stats; + uint8_t lf_rx_stats; + uint16_t cints; + uint16_t qints; + struct roc_nix_rq rq; + uint16_t rq_refs; + bool is_nix1; + + /* NIX/CPT data */ + void *inb_sa_base; + uint16_t inb_sa_sz; + + /* Device arguments */ + uint8_t selftest; + uint16_t ipsec_in_max_spi; +}; + +int nix_inl_sso_register_irqs(struct nix_inl_dev *inl_dev); +void nix_inl_sso_unregister_irqs(struct nix_inl_dev *inl_dev); + +int nix_inl_nix_register_irqs(struct nix_inl_dev *inl_dev); +void nix_inl_nix_unregister_irqs(struct nix_inl_dev *inl_dev); + +#endif /* _ROC_NIX_INL_PRIV_H_ */ diff --git a/drivers/common/cnxk/roc_platform.h b/drivers/common/cnxk/roc_platform.h index 285b24b..177db3d 100644 --- a/drivers/common/cnxk/roc_platform.h +++ b/drivers/common/cnxk/roc_platform.h @@ -113,10 +113,11 @@ #define plt_write64(val, addr) \ rte_write64_relaxed((val), (volatile void *)(addr)) -#define plt_wmb() rte_wmb() -#define plt_rmb() rte_rmb() -#define plt_io_wmb() rte_io_wmb() -#define plt_io_rmb() rte_io_rmb() +#define plt_wmb() rte_wmb() +#define plt_rmb() rte_rmb() +#define plt_io_wmb() rte_io_wmb() +#define plt_io_rmb() rte_io_rmb() +#define plt_atomic_thread_fence rte_atomic_thread_fence #define plt_mmap mmap #define PLT_PROT_READ PROT_READ diff --git a/drivers/common/cnxk/roc_priv.h b/drivers/common/cnxk/roc_priv.h index 7494b8d..f72bbd5 100644 --- a/drivers/common/cnxk/roc_priv.h +++ b/drivers/common/cnxk/roc_priv.h @@ -38,4 +38,7 @@ /* CPT */ #include "roc_cpt_priv.h" +/* NIX Inline dev */ +#include "roc_nix_inl_priv.h" + #endif /* _ROC_PRIV_H_ */ From patchwork Thu Sep 2 02:14:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97732 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CE675A0C4C; Thu, 2 Sep 2021 04:17:26 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 83C324113B; Thu, 2 Sep 2021 04:17:07 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 855CD40140 for ; Thu, 2 Sep 2021 04:17:06 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQtfT012898; Wed, 1 Sep 2021 19:17:04 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=6yj2uCJXX29VgE92mUX4uWKoBukHdCP37Gu896/HukQ=; b=LUvj+Qqmy9AB1cn77zVCLvg4ub90cGR/xa6UjyEpncEG4PW9Qy9CkgtYUtJBgMQ/lA6W pXfRoHOCJUhtsUXnbSbVqMqwSpyVs8DKeqpds7sWHFp1pUKTHIboa+BwostGdNDbklmQ 73NLNP+dWdQ79r9uGQfXV5wfHWtqITkr6fvs5Qak8Ze5FBTVh2cUgx/ilPscbOmh2gV+ ZfH4RPfCM53U1cxkY/WrSROyDQMpPyutcCe01UcoBYeS3qSFjL5M+whZHSLta7WRmI3Y jD1Vn5vNMX7VaoJX2voXPAAsgXOVx9JyYuVeu8zbMsqs3Mqw50uRAw25ePUFfa5o5d3P OQ== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hr5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 19:17:04 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:02 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:01 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 3A3AD3F704B; Wed, 1 Sep 2021 19:16:59 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao , Ray Kinsella CC: , , Date: Thu, 2 Sep 2021 07:44:44 +0530 Message-ID: <20210902021505.17607-7-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: zfWwOwbffjQNOzRux2CmPemDODtKf729 X-Proofpoint-GUID: zfWwOwbffjQNOzRux2CmPemDODtKf729 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 06/27] common/cnxk: add nix inline device init and fini X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add support to init and fini nix inline device with NIX LF, SSO LF and SSOW LF for inline inbound IPSec in CN10K. Signed-off-by: Nithin Dabilpuram --- drivers/common/cnxk/meson.build | 1 + drivers/common/cnxk/roc_api.h | 2 + drivers/common/cnxk/roc_cpt.c | 7 +- drivers/common/cnxk/roc_idev.c | 2 + drivers/common/cnxk/roc_idev_priv.h | 3 + drivers/common/cnxk/roc_nix_debug.c | 35 +++ drivers/common/cnxk/roc_nix_inl.h | 55 ++++ drivers/common/cnxk/roc_nix_inl_dev.c | 544 +++++++++++++++++++++++++++++++++ drivers/common/cnxk/roc_nix_inl_priv.h | 2 + drivers/common/cnxk/roc_platform.h | 2 + drivers/common/cnxk/version.map | 3 + 11 files changed, 653 insertions(+), 3 deletions(-) create mode 100644 drivers/common/cnxk/roc_nix_inl_dev.c diff --git a/drivers/common/cnxk/meson.build b/drivers/common/cnxk/meson.build index 207ca00..e8940d7 100644 --- a/drivers/common/cnxk/meson.build +++ b/drivers/common/cnxk/meson.build @@ -28,6 +28,7 @@ sources = files( 'roc_nix_debug.c', 'roc_nix_fc.c', 'roc_nix_irq.c', + 'roc_nix_inl_dev.c', 'roc_nix_inl_dev_irq.c', 'roc_nix_mac.c', 'roc_nix_mcast.c', diff --git a/drivers/common/cnxk/roc_api.h b/drivers/common/cnxk/roc_api.h index c1af95e..53f4e4b 100644 --- a/drivers/common/cnxk/roc_api.h +++ b/drivers/common/cnxk/roc_api.h @@ -53,6 +53,8 @@ #define PCI_DEVID_CNXK_RVU_SDP_PF 0xA0f6 #define PCI_DEVID_CNXK_RVU_SDP_VF 0xA0f7 #define PCI_DEVID_CNXK_BPHY 0xA089 +#define PCI_DEVID_CNXK_RVU_NIX_INL_PF 0xA0F0 +#define PCI_DEVID_CNXK_RVU_NIX_INL_VF 0xA0F1 #define PCI_DEVID_CN9K_CGX 0xA059 #define PCI_DEVID_CN10K_RPM 0xA060 diff --git a/drivers/common/cnxk/roc_cpt.c b/drivers/common/cnxk/roc_cpt.c index 5e35d1b..3222b3e 100644 --- a/drivers/common/cnxk/roc_cpt.c +++ b/drivers/common/cnxk/roc_cpt.c @@ -381,11 +381,12 @@ cpt_lfs_alloc(struct dev *dev, uint8_t eng_grpmsk, uint8_t blkaddr, if (blkaddr != RVU_BLOCK_ADDR_CPT0 && blkaddr != RVU_BLOCK_ADDR_CPT1) return -EINVAL; - PLT_SET_USED(inl_dev_sso); - req = mbox_alloc_msg_cpt_lf_alloc(mbox); req->nix_pf_func = 0; - req->sso_pf_func = idev_sso_pffunc_get(); + if (inl_dev_sso && nix_inl_dev_pffunc_get()) + req->sso_pf_func = nix_inl_dev_pffunc_get(); + else + req->sso_pf_func = idev_sso_pffunc_get(); req->eng_grpmsk = eng_grpmsk; req->blkaddr = blkaddr; diff --git a/drivers/common/cnxk/roc_idev.c b/drivers/common/cnxk/roc_idev.c index 1494187..648f37b 100644 --- a/drivers/common/cnxk/roc_idev.c +++ b/drivers/common/cnxk/roc_idev.c @@ -38,6 +38,8 @@ idev_set_defaults(struct idev_cfg *idev) idev->num_lmtlines = 0; idev->bphy = NULL; idev->cpt = NULL; + idev->nix_inl_dev = NULL; + plt_spinlock_init(&idev->nix_inl_dev_lock); __atomic_store_n(&idev->npa_refcnt, 0, __ATOMIC_RELEASE); } diff --git a/drivers/common/cnxk/roc_idev_priv.h b/drivers/common/cnxk/roc_idev_priv.h index 84e6f1e..2c8309b 100644 --- a/drivers/common/cnxk/roc_idev_priv.h +++ b/drivers/common/cnxk/roc_idev_priv.h @@ -9,6 +9,7 @@ struct npa_lf; struct roc_bphy; struct roc_cpt; +struct nix_inl_dev; struct idev_cfg { uint16_t sso_pf_func; uint16_t npa_pf_func; @@ -20,6 +21,8 @@ struct idev_cfg { uint64_t lmt_base_addr; struct roc_bphy *bphy; struct roc_cpt *cpt; + struct nix_inl_dev *nix_inl_dev; + plt_spinlock_t nix_inl_dev_lock; }; /* Generic */ diff --git a/drivers/common/cnxk/roc_nix_debug.c b/drivers/common/cnxk/roc_nix_debug.c index 9539bb9..582f5a3 100644 --- a/drivers/common/cnxk/roc_nix_debug.c +++ b/drivers/common/cnxk/roc_nix_debug.c @@ -1213,3 +1213,38 @@ roc_nix_dump(struct roc_nix *roc_nix) nix_dump(" \trss_alg_idx = %d", nix->rss_alg_idx); nix_dump(" \ttx_pause = %d", nix->tx_pause); } + +void +roc_nix_inl_dev_dump(struct roc_nix_inl_dev *roc_inl_dev) +{ + struct nix_inl_dev *inl_dev = + (struct nix_inl_dev *)&roc_inl_dev->reserved; + struct dev *dev = &inl_dev->dev; + + nix_dump("nix_inl_dev@%p", inl_dev); + nix_dump(" pf = %d", dev_get_pf(dev->pf_func)); + nix_dump(" vf = %d", dev_get_vf(dev->pf_func)); + nix_dump(" bar2 = 0x%" PRIx64, dev->bar2); + nix_dump(" bar4 = 0x%" PRIx64, dev->bar4); + + nix_dump(" \tpci_dev = %p", inl_dev->pci_dev); + nix_dump(" \tnix_base = 0x%" PRIxPTR "", inl_dev->nix_base); + nix_dump(" \tsso_base = 0x%" PRIxPTR "", inl_dev->sso_base); + nix_dump(" \tssow_base = 0x%" PRIxPTR "", inl_dev->ssow_base); + nix_dump(" \tnix_msixoff = %d", inl_dev->nix_msixoff); + nix_dump(" \tsso_msixoff = %d", inl_dev->sso_msixoff); + nix_dump(" \tssow_msixoff = %d", inl_dev->ssow_msixoff); + nix_dump(" \tnix_cints = %d", inl_dev->cints); + nix_dump(" \tnix_qints = %d", inl_dev->qints); + nix_dump(" \trq_refs = %d", inl_dev->rq_refs); + nix_dump(" \tinb_sa_base = 0x%p", inl_dev->inb_sa_base); + nix_dump(" \tinb_sa_sz = %d", inl_dev->inb_sa_sz); + nix_dump(" \txaq_buf_size = %u", inl_dev->xaq_buf_size); + nix_dump(" \txae_waes = %u", inl_dev->xae_waes); + nix_dump(" \tiue = %u", inl_dev->iue); + nix_dump(" \txaq_aura = 0x%" PRIx64, inl_dev->xaq_aura); + nix_dump(" \txaq_mem = 0x%p", inl_dev->xaq_mem); + + nix_dump(" \tinl_dev_rq:"); + roc_nix_rq_dump(&inl_dev->rq); +} diff --git a/drivers/common/cnxk/roc_nix_inl.h b/drivers/common/cnxk/roc_nix_inl.h index 1ec3dda..f1fe4a2 100644 --- a/drivers/common/cnxk/roc_nix_inl.h +++ b/drivers/common/cnxk/roc_nix_inl.h @@ -4,7 +4,62 @@ #ifndef _ROC_NIX_INL_H_ #define _ROC_NIX_INL_H_ +/* ONF INB HW area */ +#define ROC_NIX_INL_ONF_IPSEC_INB_HW_SZ \ + PLT_ALIGN(sizeof(struct roc_onf_ipsec_inb_sa), ROC_ALIGN) +/* ONF INB SW reserved area */ +#define ROC_NIX_INL_ONF_IPSEC_INB_SW_RSVD 384 +#define ROC_NIX_INL_ONF_IPSEC_INB_SA_SZ \ + (ROC_NIX_INL_ONF_IPSEC_INB_HW_SZ + ROC_NIX_INL_ONF_IPSEC_INB_SW_RSVD) +#define ROC_NIX_INL_ONF_IPSEC_INB_SA_SZ_LOG2 9 + +/* ONF OUTB HW area */ +#define ROC_NIX_INL_ONF_IPSEC_OUTB_HW_SZ \ + PLT_ALIGN(sizeof(struct roc_onf_ipsec_outb_sa), ROC_ALIGN) +/* ONF OUTB SW reserved area */ +#define ROC_NIX_INL_ONF_IPSEC_OUTB_SW_RSVD 128 +#define ROC_NIX_INL_ONF_IPSEC_OUTB_SA_SZ \ + (ROC_NIX_INL_ONF_IPSEC_OUTB_HW_SZ + ROC_NIX_INL_ONF_IPSEC_OUTB_SW_RSVD) +#define ROC_NIX_INL_ONF_IPSEC_OUTB_SA_SZ_LOG2 8 + +/* OT INB HW area */ +#define ROC_NIX_INL_OT_IPSEC_INB_HW_SZ \ + PLT_ALIGN(sizeof(struct roc_ot_ipsec_inb_sa), ROC_ALIGN) +/* OT INB SW reserved area */ +#define ROC_NIX_INL_OT_IPSEC_INB_SW_RSVD 128 +#define ROC_NIX_INL_OT_IPSEC_INB_SA_SZ \ + (ROC_NIX_INL_OT_IPSEC_INB_HW_SZ + ROC_NIX_INL_OT_IPSEC_INB_SW_RSVD) +#define ROC_NIX_INL_OT_IPSEC_INB_SA_SZ_LOG2 10 + +/* OT OUTB HW area */ +#define ROC_NIX_INL_OT_IPSEC_OUTB_HW_SZ \ + PLT_ALIGN(sizeof(struct roc_ot_ipsec_outb_sa), ROC_ALIGN) +/* OT OUTB SW reserved area */ +#define ROC_NIX_INL_OT_IPSEC_OUTB_SW_RSVD 128 +#define ROC_NIX_INL_OT_IPSEC_OUTB_SA_SZ \ + (ROC_NIX_INL_OT_IPSEC_OUTB_HW_SZ + ROC_NIX_INL_OT_IPSEC_OUTB_SW_RSVD) +#define ROC_NIX_INL_OT_IPSEC_OUTB_SA_SZ_LOG2 9 + +/* Alignment of SA Base */ +#define ROC_NIX_INL_SA_BASE_ALIGN BIT_ULL(16) + /* Inline device SSO Work callback */ typedef void (*roc_nix_inl_sso_work_cb_t)(uint64_t *gw, void *args); +struct roc_nix_inl_dev { + /* Input parameters */ + struct plt_pci_device *pci_dev; + uint16_t ipsec_in_max_spi; + bool selftest; + /* End of input parameters */ + +#define ROC_NIX_INL_MEM_SZ (1024) + uint8_t reserved[ROC_NIX_INL_MEM_SZ] __plt_cache_aligned; +} __plt_cache_aligned; + +/* NIX Inline Device API */ +int __roc_api roc_nix_inl_dev_init(struct roc_nix_inl_dev *roc_inl_dev); +int __roc_api roc_nix_inl_dev_fini(struct roc_nix_inl_dev *roc_inl_dev); +void __roc_api roc_nix_inl_dev_dump(struct roc_nix_inl_dev *roc_inl_dev); + #endif /* _ROC_NIX_INL_H_ */ diff --git a/drivers/common/cnxk/roc_nix_inl_dev.c b/drivers/common/cnxk/roc_nix_inl_dev.c new file mode 100644 index 0000000..214f183 --- /dev/null +++ b/drivers/common/cnxk/roc_nix_inl_dev.c @@ -0,0 +1,544 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "roc_api.h" +#include "roc_priv.h" + +#define XAQ_CACHE_CNT 0x7 + +/* Default Rx Config for Inline NIX LF */ +#define NIX_INL_LF_RX_CFG \ + (ROC_NIX_LF_RX_CFG_DROP_RE | ROC_NIX_LF_RX_CFG_L2_LEN_ERR | \ + ROC_NIX_LF_RX_CFG_IP6_UDP_OPT | ROC_NIX_LF_RX_CFG_DIS_APAD | \ + ROC_NIX_LF_RX_CFG_CSUM_IL4 | ROC_NIX_LF_RX_CFG_CSUM_OL4 | \ + ROC_NIX_LF_RX_CFG_LEN_IL4 | ROC_NIX_LF_RX_CFG_LEN_IL3 | \ + ROC_NIX_LF_RX_CFG_LEN_OL4 | ROC_NIX_LF_RX_CFG_LEN_OL3) + +uint16_t +nix_inl_dev_pffunc_get(void) +{ + struct idev_cfg *idev = idev_get_cfg(); + struct nix_inl_dev *inl_dev; + + if (idev != NULL) { + inl_dev = idev->nix_inl_dev; + if (inl_dev) + return inl_dev->dev.pf_func; + } + return 0; +} + +static void +nix_inl_selftest_work_cb(uint64_t *gw, void *args) +{ + uintptr_t work = gw[1]; + + *((uintptr_t *)args + (gw[0] & 0x1)) = work; + + plt_atomic_thread_fence(__ATOMIC_ACQ_REL); +} + +static int +nix_inl_selftest(void) +{ + struct idev_cfg *idev = idev_get_cfg(); + roc_nix_inl_sso_work_cb_t save_cb; + static uintptr_t work_arr[2]; + struct nix_inl_dev *inl_dev; + void *save_cb_args; + uint64_t add_work0; + int rc = 0; + + if (idev == NULL) + return -ENOTSUP; + + inl_dev = idev->nix_inl_dev; + if (inl_dev == NULL) + return -ENOTSUP; + + plt_info("Performing nix inl self test"); + + /* Save and update cb to test cb */ + save_cb = inl_dev->work_cb; + save_cb_args = inl_dev->cb_args; + inl_dev->work_cb = nix_inl_selftest_work_cb; + inl_dev->cb_args = work_arr; + + plt_atomic_thread_fence(__ATOMIC_ACQ_REL); + +#define WORK_MAGIC1 0x335577ff0 +#define WORK_MAGIC2 0xdeadbeef0 + + /* Add work */ + add_work0 = ((uint64_t)(SSO_TT_ORDERED) << 32) | 0x0; + roc_store_pair(add_work0, WORK_MAGIC1, inl_dev->sso_base); + add_work0 = ((uint64_t)(SSO_TT_ORDERED) << 32) | 0x1; + roc_store_pair(add_work0, WORK_MAGIC2, inl_dev->sso_base); + + plt_delay_ms(10000); + + /* Check if we got expected work */ + if (work_arr[0] != WORK_MAGIC1 || work_arr[1] != WORK_MAGIC2) { + plt_err("Failed to get expected work, [0]=%p [1]=%p", + (void *)work_arr[0], (void *)work_arr[1]); + rc = -EFAULT; + goto exit; + } + + plt_info("Work, [0]=%p [1]=%p", (void *)work_arr[0], + (void *)work_arr[1]); + +exit: + /* Restore state */ + inl_dev->work_cb = save_cb; + inl_dev->cb_args = save_cb_args; + return rc; +} + +static int +nix_inl_nix_ipsec_cfg(struct nix_inl_dev *inl_dev, bool ena) +{ + struct nix_inline_ipsec_lf_cfg *lf_cfg; + struct mbox *mbox = (&inl_dev->dev)->mbox; + uint32_t sa_w; + + lf_cfg = mbox_alloc_msg_nix_inline_ipsec_lf_cfg(mbox); + if (lf_cfg == NULL) + return -ENOSPC; + + if (ena) { + sa_w = plt_align32pow2(inl_dev->ipsec_in_max_spi + 1); + sa_w = plt_log2_u32(sa_w); + + lf_cfg->enable = 1; + lf_cfg->sa_base_addr = (uintptr_t)inl_dev->inb_sa_base; + lf_cfg->ipsec_cfg1.sa_idx_w = sa_w; + /* CN9K SA size is different */ + if (roc_model_is_cn9k()) + lf_cfg->ipsec_cfg0.lenm1_max = NIX_CN9K_MAX_HW_FRS - 1; + else + lf_cfg->ipsec_cfg0.lenm1_max = NIX_RPM_MAX_HW_FRS - 1; + lf_cfg->ipsec_cfg1.sa_idx_max = inl_dev->ipsec_in_max_spi; + lf_cfg->ipsec_cfg0.sa_pow2_size = + plt_log2_u32(inl_dev->inb_sa_sz); + + lf_cfg->ipsec_cfg0.tag_const = 0; + lf_cfg->ipsec_cfg0.tt = SSO_TT_ORDERED; + } else { + lf_cfg->enable = 0; + } + + return mbox_process(mbox); +} + +static int +nix_inl_sso_setup(struct nix_inl_dev *inl_dev) +{ + struct sso_lf_alloc_rsp *sso_rsp; + struct dev *dev = &inl_dev->dev; + uint32_t xaq_cnt, count, aura; + uint16_t hwgrp[1] = {0}; + struct npa_pool_s pool; + uintptr_t iova; + int rc; + + /* Alloc SSOW LF */ + rc = sso_lf_alloc(dev, SSO_LF_TYPE_HWS, 1, NULL); + if (rc) { + plt_err("Failed to alloc SSO HWS, rc=%d", rc); + return rc; + } + + /* Alloc HWGRP LF */ + rc = sso_lf_alloc(dev, SSO_LF_TYPE_HWGRP, 1, (void **)&sso_rsp); + if (rc) { + plt_err("Failed to alloc SSO HWGRP, rc=%d", rc); + goto free_ssow; + } + + inl_dev->xaq_buf_size = sso_rsp->xaq_buf_size; + inl_dev->xae_waes = sso_rsp->xaq_wq_entries; + inl_dev->iue = sso_rsp->in_unit_entries; + + /* Create XAQ pool */ + xaq_cnt = XAQ_CACHE_CNT; + xaq_cnt += inl_dev->iue / inl_dev->xae_waes; + plt_sso_dbg("Configuring %d xaq buffers", xaq_cnt); + + inl_dev->xaq_mem = plt_zmalloc(inl_dev->xaq_buf_size * xaq_cnt, + inl_dev->xaq_buf_size); + if (!inl_dev->xaq_mem) { + rc = NIX_ERR_NO_MEM; + plt_err("Failed to alloc xaq buf mem"); + goto free_sso; + } + + memset(&pool, 0, sizeof(struct npa_pool_s)); + pool.nat_align = 1; + rc = roc_npa_pool_create(&inl_dev->xaq_aura, inl_dev->xaq_buf_size, + xaq_cnt, NULL, &pool); + if (rc) { + plt_err("Failed to alloc aura for XAQ, rc=%d", rc); + goto free_mem; + } + + /* Fill the XAQ buffers */ + iova = (uint64_t)inl_dev->xaq_mem; + for (count = 0; count < xaq_cnt; count++) { + roc_npa_aura_op_free(inl_dev->xaq_aura, 0, iova); + iova += inl_dev->xaq_buf_size; + } + roc_npa_aura_op_range_set(inl_dev->xaq_aura, (uint64_t)inl_dev->xaq_mem, + iova); + + aura = roc_npa_aura_handle_to_aura(inl_dev->xaq_aura); + + /* Setup xaq for hwgrps */ + rc = sso_hwgrp_alloc_xaq(dev, aura, 1); + if (rc) { + plt_err("Failed to setup hwgrp xaq aura, rc=%d", rc); + goto destroy_pool; + } + + /* Register SSO, SSOW error and work irq's */ + rc = nix_inl_sso_register_irqs(inl_dev); + if (rc) { + plt_err("Failed to register sso irq's, rc=%d", rc); + goto release_xaq; + } + + /* Setup hwgrp->hws link */ + sso_hws_link_modify(0, inl_dev->ssow_base, NULL, hwgrp, 1, true); + + /* Enable HWGRP */ + plt_write64(0x1, inl_dev->sso_base + SSO_LF_GGRP_QCTL); + + return 0; + +release_xaq: + sso_hwgrp_release_xaq(&inl_dev->dev, 1); +destroy_pool: + roc_npa_pool_destroy(inl_dev->xaq_aura); + inl_dev->xaq_aura = 0; +free_mem: + plt_free(inl_dev->xaq_mem); + inl_dev->xaq_mem = NULL; +free_sso: + sso_lf_free(dev, SSO_LF_TYPE_HWGRP, 1); +free_ssow: + sso_lf_free(dev, SSO_LF_TYPE_HWS, 1); + return rc; +} + +static int +nix_inl_sso_release(struct nix_inl_dev *inl_dev) +{ + uint16_t hwgrp[1] = {0}; + + /* Disable HWGRP */ + plt_write64(0, inl_dev->sso_base + SSO_LF_GGRP_QCTL); + + /* Unregister SSO/SSOW IRQ's */ + nix_inl_sso_unregister_irqs(inl_dev); + + /* Unlink hws */ + sso_hws_link_modify(0, inl_dev->ssow_base, NULL, hwgrp, 1, false); + + /* Release XAQ aura */ + sso_hwgrp_release_xaq(&inl_dev->dev, 1); + + /* Free SSO, SSOW LF's */ + sso_lf_free(&inl_dev->dev, SSO_LF_TYPE_HWS, 1); + sso_lf_free(&inl_dev->dev, SSO_LF_TYPE_HWGRP, 1); + + return 0; +} + +static int +nix_inl_nix_setup(struct nix_inl_dev *inl_dev) +{ + uint16_t ipsec_in_max_spi = inl_dev->ipsec_in_max_spi; + struct dev *dev = &inl_dev->dev; + struct mbox *mbox = dev->mbox; + struct nix_lf_alloc_rsp *rsp; + struct nix_lf_alloc_req *req; + size_t inb_sa_sz; + int rc = -ENOSPC; + + /* Alloc NIX LF needed for single RQ */ + req = mbox_alloc_msg_nix_lf_alloc(mbox); + if (req == NULL) + return rc; + req->rq_cnt = 1; + req->sq_cnt = 1; + req->cq_cnt = 1; + /* XQESZ is W16 */ + req->xqe_sz = NIX_XQESZ_W16; + /* RSS size does not matter as this RQ is only for UCAST_IPSEC action */ + req->rss_sz = ROC_NIX_RSS_RETA_SZ_64; + req->rss_grps = ROC_NIX_RSS_GRPS; + req->npa_func = idev_npa_pffunc_get(); + req->sso_func = dev->pf_func; + req->rx_cfg = NIX_INL_LF_RX_CFG; + req->flags = NIX_LF_RSS_TAG_LSB_AS_ADDER; + + rc = mbox_process_msg(mbox, (void *)&rsp); + if (rc) { + plt_err("Failed to alloc lf, rc=%d", rc); + return rc; + } + + inl_dev->lf_tx_stats = rsp->lf_tx_stats; + inl_dev->lf_rx_stats = rsp->lf_rx_stats; + inl_dev->qints = rsp->qints; + inl_dev->cints = rsp->cints; + + /* Register nix interrupts */ + rc = nix_inl_nix_register_irqs(inl_dev); + if (rc) { + plt_err("Failed to register nix irq's, rc=%d", rc); + goto lf_free; + } + + /* CN9K SA is different */ + if (roc_model_is_cn9k()) + inb_sa_sz = ROC_NIX_INL_ONF_IPSEC_INB_SA_SZ; + else + inb_sa_sz = ROC_NIX_INL_OT_IPSEC_INB_SA_SZ; + + /* Alloc contiguous memory for Inbound SA's */ + inl_dev->inb_sa_sz = inb_sa_sz; + inl_dev->inb_sa_base = plt_zmalloc(inb_sa_sz * ipsec_in_max_spi, + ROC_NIX_INL_SA_BASE_ALIGN); + if (!inl_dev->inb_sa_base) { + plt_err("Failed to allocate memory for Inbound SA"); + rc = -ENOMEM; + goto unregister_irqs; + } + + /* Setup device specific inb SA table */ + rc = nix_inl_nix_ipsec_cfg(inl_dev, true); + if (rc) { + plt_err("Failed to setup NIX Inbound SA conf, rc=%d", rc); + goto free_mem; + } + + return 0; +free_mem: + plt_free(inl_dev->inb_sa_base); + inl_dev->inb_sa_base = NULL; +unregister_irqs: + nix_inl_nix_unregister_irqs(inl_dev); +lf_free: + mbox_alloc_msg_nix_lf_free(mbox); + rc |= mbox_process(mbox); + return rc; +} + +static int +nix_inl_nix_release(struct nix_inl_dev *inl_dev) +{ + struct dev *dev = &inl_dev->dev; + struct mbox *mbox = dev->mbox; + struct nix_lf_free_req *req; + struct ndc_sync_op *ndc_req; + int rc = -ENOSPC; + + /* Disable Inbound processing */ + rc = nix_inl_nix_ipsec_cfg(inl_dev, false); + if (rc) + plt_err("Failed to disable Inbound IPSec, rc=%d", rc); + + /* Sync NDC-NIX for LF */ + ndc_req = mbox_alloc_msg_ndc_sync_op(mbox); + if (ndc_req == NULL) + return rc; + ndc_req->nix_lf_rx_sync = 1; + rc = mbox_process(mbox); + if (rc) + plt_err("Error on NDC-NIX-RX LF sync, rc %d", rc); + + /* Unregister IRQs */ + nix_inl_nix_unregister_irqs(inl_dev); + + /* By default all associated mcam rules are deleted */ + req = mbox_alloc_msg_nix_lf_free(mbox); + if (req == NULL) + return -ENOSPC; + + return mbox_process(mbox); +} + +static int +nix_inl_lf_attach(struct nix_inl_dev *inl_dev) +{ + struct msix_offset_rsp *msix_rsp; + struct dev *dev = &inl_dev->dev; + struct mbox *mbox = dev->mbox; + struct rsrc_attach_req *req; + uint64_t nix_blkaddr; + int rc = -ENOSPC; + + req = mbox_alloc_msg_attach_resources(mbox); + if (req == NULL) + return rc; + req->modify = true; + /* Attach 1 NIXLF, SSO HWS and SSO HWGRP */ + req->nixlf = true; + req->ssow = 1; + req->sso = 1; + + rc = mbox_process(dev->mbox); + if (rc) + return rc; + + /* Get MSIX vector offsets */ + mbox_alloc_msg_msix_offset(mbox); + rc = mbox_process_msg(dev->mbox, (void **)&msix_rsp); + if (rc) + return rc; + + inl_dev->nix_msixoff = msix_rsp->nix_msixoff; + inl_dev->ssow_msixoff = msix_rsp->ssow_msixoff[0]; + inl_dev->sso_msixoff = msix_rsp->sso_msixoff[0]; + + nix_blkaddr = nix_get_blkaddr(dev); + inl_dev->is_nix1 = (nix_blkaddr == RVU_BLOCK_ADDR_NIX1); + + /* Update base addresses for LF's */ + inl_dev->nix_base = dev->bar2 + (nix_blkaddr << 20); + inl_dev->ssow_base = dev->bar2 + (RVU_BLOCK_ADDR_SSOW << 20); + inl_dev->sso_base = dev->bar2 + (RVU_BLOCK_ADDR_SSO << 20); + return 0; +} + +static int +nix_inl_lf_detach(struct nix_inl_dev *inl_dev) +{ + struct dev *dev = &inl_dev->dev; + struct mbox *mbox = dev->mbox; + struct rsrc_detach_req *req; + int rc = -ENOSPC; + + req = mbox_alloc_msg_detach_resources(mbox); + if (req == NULL) + return rc; + req->partial = true; + req->nixlf = true; + req->ssow = true; + req->sso = true; + + return mbox_process(dev->mbox); +} + +int +roc_nix_inl_dev_init(struct roc_nix_inl_dev *roc_inl_dev) +{ + struct plt_pci_device *pci_dev; + struct nix_inl_dev *inl_dev; + struct idev_cfg *idev; + int rc; + + pci_dev = roc_inl_dev->pci_dev; + + /* Skip probe if already done */ + idev = idev_get_cfg(); + if (idev == NULL) + return -ENOTSUP; + + if (idev->nix_inl_dev) { + plt_info("Skipping device %s, inline device already probed", + pci_dev->name); + return -EEXIST; + } + + PLT_STATIC_ASSERT(sizeof(struct nix_inl_dev) <= ROC_NIX_INL_MEM_SZ); + + inl_dev = (struct nix_inl_dev *)roc_inl_dev->reserved; + memset(inl_dev, 0, sizeof(*inl_dev)); + + inl_dev->pci_dev = pci_dev; + inl_dev->ipsec_in_max_spi = roc_inl_dev->ipsec_in_max_spi; + inl_dev->selftest = roc_inl_dev->selftest; + + /* Initialize base device */ + rc = dev_init(&inl_dev->dev, pci_dev); + if (rc) { + plt_err("Failed to init roc device"); + goto error; + } + + /* Attach LF resources */ + rc = nix_inl_lf_attach(inl_dev); + if (rc) { + plt_err("Failed to attach LF resources, rc=%d", rc); + goto dev_cleanup; + } + + /* Setup NIX LF */ + rc = nix_inl_nix_setup(inl_dev); + if (rc) + goto lf_detach; + + /* Setup SSO LF */ + rc = nix_inl_sso_setup(inl_dev); + if (rc) + goto nix_release; + + idev->nix_inl_dev = inl_dev; + + /* Perform selftest if asked for */ + if (inl_dev->selftest) { + rc = nix_inl_selftest(); + if (rc) + goto nix_release; + } + + return 0; +nix_release: + rc |= nix_inl_nix_release(inl_dev); +lf_detach: + rc |= nix_inl_lf_detach(inl_dev); +dev_cleanup: + rc |= dev_fini(&inl_dev->dev, pci_dev); +error: + return rc; +} + +int +roc_nix_inl_dev_fini(struct roc_nix_inl_dev *roc_inl_dev) +{ + struct plt_pci_device *pci_dev; + struct nix_inl_dev *inl_dev; + struct idev_cfg *idev; + int rc; + + idev = idev_get_cfg(); + if (idev == NULL) + return 0; + + if (!idev->nix_inl_dev || + PLT_PTR_DIFF(roc_inl_dev->reserved, idev->nix_inl_dev)) + return 0; + + inl_dev = idev->nix_inl_dev; + pci_dev = inl_dev->pci_dev; + + /* Release SSO */ + rc = nix_inl_sso_release(inl_dev); + + /* Release NIX */ + rc |= nix_inl_nix_release(inl_dev); + + /* Detach LF's */ + rc |= nix_inl_lf_detach(inl_dev); + + /* Cleanup mbox */ + rc |= dev_fini(&inl_dev->dev, pci_dev); + if (rc) + return rc; + + idev->nix_inl_dev = NULL; + return 0; +} diff --git a/drivers/common/cnxk/roc_nix_inl_priv.h b/drivers/common/cnxk/roc_nix_inl_priv.h index f424009..ab38062 100644 --- a/drivers/common/cnxk/roc_nix_inl_priv.h +++ b/drivers/common/cnxk/roc_nix_inl_priv.h @@ -54,4 +54,6 @@ void nix_inl_sso_unregister_irqs(struct nix_inl_dev *inl_dev); int nix_inl_nix_register_irqs(struct nix_inl_dev *inl_dev); void nix_inl_nix_unregister_irqs(struct nix_inl_dev *inl_dev); +uint16_t nix_inl_dev_pffunc_get(void); + #endif /* _ROC_NIX_INL_PRIV_H_ */ diff --git a/drivers/common/cnxk/roc_platform.h b/drivers/common/cnxk/roc_platform.h index 177db3d..241655b 100644 --- a/drivers/common/cnxk/roc_platform.h +++ b/drivers/common/cnxk/roc_platform.h @@ -37,6 +37,7 @@ #define PLT_MEMZONE_NAMESIZE RTE_MEMZONE_NAMESIZE #define PLT_STD_C11 RTE_STD_C11 #define PLT_PTR_ADD RTE_PTR_ADD +#define PLT_PTR_DIFF RTE_PTR_DIFF #define PLT_MAX_RXTX_INTR_VEC_ID RTE_MAX_RXTX_INTR_VEC_ID #define PLT_INTR_VEC_RXTX_OFFSET RTE_INTR_VEC_RXTX_OFFSET #define PLT_MIN RTE_MIN @@ -77,6 +78,7 @@ #define plt_cpu_to_be_64 rte_cpu_to_be_64 #define plt_be_to_cpu_64 rte_be_to_cpu_64 +#define plt_align32pow2 rte_align32pow2 #define plt_align32prevpow2 rte_align32prevpow2 #define plt_bitmap rte_bitmap diff --git a/drivers/common/cnxk/version.map b/drivers/common/cnxk/version.map index 5dbb21c..3a35233 100644 --- a/drivers/common/cnxk/version.map +++ b/drivers/common/cnxk/version.map @@ -100,6 +100,9 @@ INTERNAL { roc_nix_get_pf_func; roc_nix_get_vf; roc_nix_get_vwqe_interval; + roc_nix_inl_dev_dump; + roc_nix_inl_dev_fini; + roc_nix_inl_dev_init; roc_nix_is_lbk; roc_nix_is_pf; roc_nix_is_sdp; From patchwork Thu Sep 2 02:14:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97734 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D9B7DA0C4C; Thu, 2 Sep 2021 04:17:39 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D5A854114B; Thu, 2 Sep 2021 04:17:12 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 7AE82410EA for ; Thu, 2 Sep 2021 04:17:10 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181K5Ja0026506; Wed, 1 Sep 2021 19:17:06 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=kZtJMszxr5VqzhebgShmLB4QCCd6PkCNBgb3TXgOVSQ=; b=geLGvE4xibIvK9LGdvzlCoNzwDN4DJgpTA/a0YbcMCQTf8ni5ifzb8wcah+70ZipWaif NGHblrHVqFquHwgkO/2lBcbWfFqCl09MCEG3F0YNVlpVNAKokolMc7TINtH5YaG/5Meq La5nSNN9x40WaL459cz/UM+j8y174JAv3AC+bmlesY/fE6lcfhrbDFE8udF0qaLTtelw lce49Av+nyOrM+Hs3PUYNbg4EePASHPgdLj7+sGNkjegX4Eq4HfL0sj5oZwmLek0V3Z4 i3RB0vRNyWXxEnS12O3VV14GWwqai0Wu4tROVrCU/OWBJQMOvXLxFfcpmRAGf95qrkAy AQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com with ESMTP id 3atg8a91h0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 19:17:06 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:04 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:04 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 57E723F7052; Wed, 1 Sep 2021 19:17:02 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao , Ray Kinsella CC: , , Date: Thu, 2 Sep 2021 07:44:45 +0530 Message-ID: <20210902021505.17607-8-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: ZdhlkMgYXZt4DeQOZKwT_gKsrZrbX46q X-Proofpoint-ORIG-GUID: ZdhlkMgYXZt4DeQOZKwT_gKsrZrbX46q X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 07/27] common/cnxk: add nix inline inbound and outbound support API X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add API to support setting up nix inline inbound and nix inline outbound. Signed-off-by: Nithin Dabilpuram --- drivers/common/cnxk/hw/cpt.h | 8 + drivers/common/cnxk/meson.build | 1 + drivers/common/cnxk/roc_api.h | 48 +-- drivers/common/cnxk/roc_constants.h | 58 +++ drivers/common/cnxk/roc_io.h | 9 + drivers/common/cnxk/roc_io_generic.h | 3 +- drivers/common/cnxk/roc_nix.h | 5 + drivers/common/cnxk/roc_nix_debug.c | 15 + drivers/common/cnxk/roc_nix_inl.c | 739 +++++++++++++++++++++++++++++++++++ drivers/common/cnxk/roc_nix_inl.h | 100 +++++ drivers/common/cnxk/roc_nix_priv.h | 15 + drivers/common/cnxk/roc_npc.c | 27 +- drivers/common/cnxk/version.map | 25 ++ 13 files changed, 996 insertions(+), 57 deletions(-) create mode 100644 drivers/common/cnxk/roc_constants.h create mode 100644 drivers/common/cnxk/roc_nix_inl.c diff --git a/drivers/common/cnxk/hw/cpt.h b/drivers/common/cnxk/hw/cpt.h index 84ebf2d..975139f 100644 --- a/drivers/common/cnxk/hw/cpt.h +++ b/drivers/common/cnxk/hw/cpt.h @@ -40,6 +40,7 @@ #define CPT_LF_CTX_ENC_PKT_CNT (0x540ull) #define CPT_LF_CTX_DEC_BYTE_CNT (0x550ull) #define CPT_LF_CTX_DEC_PKT_CNT (0x560ull) +#define CPT_LF_CTX_RELOAD (0x570ull) #define CPT_AF_LFX_CTL(a) (0x27000ull | (uint64_t)(a) << 3) #define CPT_AF_LFX_CTL2(a) (0x29000ull | (uint64_t)(a) << 3) @@ -68,6 +69,13 @@ union cpt_lf_ctx_flush { } s; }; +union cpt_lf_ctx_reload { + uint64_t u; + struct { + uint64_t cptr : 46; + } s; +}; + union cpt_lf_inprog { uint64_t u; struct cpt_lf_inprog_s { diff --git a/drivers/common/cnxk/meson.build b/drivers/common/cnxk/meson.build index e8940d7..cd19ad2 100644 --- a/drivers/common/cnxk/meson.build +++ b/drivers/common/cnxk/meson.build @@ -28,6 +28,7 @@ sources = files( 'roc_nix_debug.c', 'roc_nix_fc.c', 'roc_nix_irq.c', + 'roc_nix_inl.c', 'roc_nix_inl_dev.c', 'roc_nix_inl_dev_irq.c', 'roc_nix_mac.c', diff --git a/drivers/common/cnxk/roc_api.h b/drivers/common/cnxk/roc_api.h index 53f4e4b..b8f3667 100644 --- a/drivers/common/cnxk/roc_api.h +++ b/drivers/common/cnxk/roc_api.h @@ -9,28 +9,21 @@ #include #include -/* Alignment */ -#define ROC_ALIGN 128 - /* Bits manipulation */ #include "roc_bits.h" /* Bitfields manipulation */ #include "roc_bitfield.h" +/* ROC Constants */ +#include "roc_constants.h" + /* Constants */ #define PLT_ETHER_ADDR_LEN 6 /* Platform definition */ #include "roc_platform.h" -#define ROC_LMT_LINE_SZ 128 -#define ROC_NUM_LMT_LINES 2048 -#define ROC_LMT_LINES_PER_CORE_LOG2 5 -#define ROC_LMT_LINE_SIZE_LOG2 7 -#define ROC_LMT_BASE_PER_CORE_LOG2 \ - (ROC_LMT_LINES_PER_CORE_LOG2 + ROC_LMT_LINE_SIZE_LOG2) - /* IO */ #if defined(__aarch64__) #include "roc_io.h" @@ -38,41 +31,6 @@ #include "roc_io_generic.h" #endif -/* PCI IDs */ -#define PCI_VENDOR_ID_CAVIUM 0x177D -#define PCI_DEVID_CNXK_RVU_PF 0xA063 -#define PCI_DEVID_CNXK_RVU_VF 0xA064 -#define PCI_DEVID_CNXK_RVU_AF 0xA065 -#define PCI_DEVID_CNXK_RVU_SSO_TIM_PF 0xA0F9 -#define PCI_DEVID_CNXK_RVU_SSO_TIM_VF 0xA0FA -#define PCI_DEVID_CNXK_RVU_NPA_PF 0xA0FB -#define PCI_DEVID_CNXK_RVU_NPA_VF 0xA0FC -#define PCI_DEVID_CNXK_RVU_AF_VF 0xA0f8 -#define PCI_DEVID_CNXK_DPI_VF 0xA081 -#define PCI_DEVID_CNXK_EP_VF 0xB203 -#define PCI_DEVID_CNXK_RVU_SDP_PF 0xA0f6 -#define PCI_DEVID_CNXK_RVU_SDP_VF 0xA0f7 -#define PCI_DEVID_CNXK_BPHY 0xA089 -#define PCI_DEVID_CNXK_RVU_NIX_INL_PF 0xA0F0 -#define PCI_DEVID_CNXK_RVU_NIX_INL_VF 0xA0F1 - -#define PCI_DEVID_CN9K_CGX 0xA059 -#define PCI_DEVID_CN10K_RPM 0xA060 - -#define PCI_DEVID_CN9K_RVU_CPT_PF 0xA0FD -#define PCI_DEVID_CN9K_RVU_CPT_VF 0xA0FE -#define PCI_DEVID_CN10K_RVU_CPT_PF 0xA0F2 -#define PCI_DEVID_CN10K_RVU_CPT_VF 0xA0F3 - -#define PCI_SUBSYSTEM_DEVID_CN10KA 0xB900 -#define PCI_SUBSYSTEM_DEVID_CN10KAS 0xB900 - -#define PCI_SUBSYSTEM_DEVID_CN9KA 0x0000 -#define PCI_SUBSYSTEM_DEVID_CN9KB 0xb400 -#define PCI_SUBSYSTEM_DEVID_CN9KC 0x0200 -#define PCI_SUBSYSTEM_DEVID_CN9KD 0xB200 -#define PCI_SUBSYSTEM_DEVID_CN9KE 0xB100 - /* HW structure definition */ #include "hw/cpt.h" #include "hw/nix.h" diff --git a/drivers/common/cnxk/roc_constants.h b/drivers/common/cnxk/roc_constants.h new file mode 100644 index 0000000..1e6427c --- /dev/null +++ b/drivers/common/cnxk/roc_constants.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ +#ifndef _ROC_CONSTANTS_H_ +#define _ROC_CONSTANTS_H_ + +/* Alignment */ +#define ROC_ALIGN 128 + +/* LMTST constants */ +/* [CN10K, .) */ +#define ROC_LMT_LINE_SZ 128 +#define ROC_NUM_LMT_LINES 2048 +#define ROC_LMT_LINES_PER_CORE_LOG2 5 +#define ROC_LMT_LINE_SIZE_LOG2 7 +#define ROC_LMT_BASE_PER_CORE_LOG2 \ + (ROC_LMT_LINES_PER_CORE_LOG2 + ROC_LMT_LINE_SIZE_LOG2) +#define ROC_LMT_MAX_THREADS 42UL +#define ROC_LMT_CPT_LINES_PER_CORE_LOG2 4 +#define ROC_LMT_CPT_BASE_ID_OFF \ + (ROC_LMT_MAX_THREADS << ROC_LMT_LINES_PER_CORE_LOG2) + +/* PCI IDs */ +#define PCI_VENDOR_ID_CAVIUM 0x177D +#define PCI_DEVID_CNXK_RVU_PF 0xA063 +#define PCI_DEVID_CNXK_RVU_VF 0xA064 +#define PCI_DEVID_CNXK_RVU_AF 0xA065 +#define PCI_DEVID_CNXK_RVU_SSO_TIM_PF 0xA0F9 +#define PCI_DEVID_CNXK_RVU_SSO_TIM_VF 0xA0FA +#define PCI_DEVID_CNXK_RVU_NPA_PF 0xA0FB +#define PCI_DEVID_CNXK_RVU_NPA_VF 0xA0FC +#define PCI_DEVID_CNXK_RVU_AF_VF 0xA0f8 +#define PCI_DEVID_CNXK_DPI_VF 0xA081 +#define PCI_DEVID_CNXK_EP_VF 0xB203 +#define PCI_DEVID_CNXK_RVU_SDP_PF 0xA0f6 +#define PCI_DEVID_CNXK_RVU_SDP_VF 0xA0f7 +#define PCI_DEVID_CNXK_BPHY 0xA089 +#define PCI_DEVID_CNXK_RVU_NIX_INL_PF 0xA0F0 +#define PCI_DEVID_CNXK_RVU_NIX_INL_VF 0xA0F1 + +#define PCI_DEVID_CN9K_CGX 0xA059 +#define PCI_DEVID_CN10K_RPM 0xA060 + +#define PCI_DEVID_CN9K_RVU_CPT_PF 0xA0FD +#define PCI_DEVID_CN9K_RVU_CPT_VF 0xA0FE +#define PCI_DEVID_CN10K_RVU_CPT_PF 0xA0F2 +#define PCI_DEVID_CN10K_RVU_CPT_VF 0xA0F3 + +#define PCI_SUBSYSTEM_DEVID_CN10KA 0xB900 +#define PCI_SUBSYSTEM_DEVID_CN10KAS 0xB900 + +#define PCI_SUBSYSTEM_DEVID_CN9KA 0x0000 +#define PCI_SUBSYSTEM_DEVID_CN9KB 0xb400 +#define PCI_SUBSYSTEM_DEVID_CN9KC 0x0200 +#define PCI_SUBSYSTEM_DEVID_CN9KD 0xB200 +#define PCI_SUBSYSTEM_DEVID_CN9KE 0xB100 + +#endif /* _ROC_CONSTANTS_H_ */ diff --git a/drivers/common/cnxk/roc_io.h b/drivers/common/cnxk/roc_io.h index aee8c7f..fe5f7f4 100644 --- a/drivers/common/cnxk/roc_io.h +++ b/drivers/common/cnxk/roc_io.h @@ -13,6 +13,15 @@ (lmt_addr) += ((uint64_t)lmt_id << ROC_LMT_LINE_SIZE_LOG2); \ } while (0) +#define ROC_LMT_CPT_BASE_ID_GET(lmt_addr, lmt_id) \ + do { \ + /* 16 Lines per core */ \ + lmt_id = ROC_LMT_CPT_BASE_ID_OFF; \ + lmt_id += (plt_lcore_id() << ROC_LMT_CPT_LINES_PER_CORE_LOG2); \ + /* Each line is of 128B */ \ + (lmt_addr) += ((uint64_t)lmt_id << ROC_LMT_LINE_SIZE_LOG2); \ + } while (0) + #define roc_load_pair(val0, val1, addr) \ ({ \ asm volatile("ldp %x[x0], %x[x1], [%x[p1]]" \ diff --git a/drivers/common/cnxk/roc_io_generic.h b/drivers/common/cnxk/roc_io_generic.h index 28cb096..ceaa3a3 100644 --- a/drivers/common/cnxk/roc_io_generic.h +++ b/drivers/common/cnxk/roc_io_generic.h @@ -5,7 +5,8 @@ #ifndef _ROC_IO_GENERIC_H_ #define _ROC_IO_GENERIC_H_ -#define ROC_LMT_BASE_ID_GET(lmt_addr, lmt_id) (lmt_id = 0) +#define ROC_LMT_BASE_ID_GET(lmt_addr, lmt_id) (lmt_id = 0) +#define ROC_LMT_CPT_BASE_ID_GET(lmt_addr, lmt_id) (lmt_id = 0) #define roc_load_pair(val0, val1, addr) \ do { \ diff --git a/drivers/common/cnxk/roc_nix.h b/drivers/common/cnxk/roc_nix.h index 822c190..ed6e721 100644 --- a/drivers/common/cnxk/roc_nix.h +++ b/drivers/common/cnxk/roc_nix.h @@ -171,6 +171,7 @@ struct roc_nix_rq { uint8_t spb_red_pass; /* End of Input parameters */ struct roc_nix *roc_nix; + bool inl_dev_ref; }; struct roc_nix_cq { @@ -254,6 +255,10 @@ struct roc_nix { bool enable_loop; bool hw_vlan_ins; uint8_t lock_rx_ctx; + uint32_t outb_nb_desc; + uint16_t outb_nb_crypto_qs; + uint16_t ipsec_in_max_spi; + uint16_t ipsec_out_max_sa; /* End of input parameters */ /* LMT line base for "Per Core Tx LMT line" mode*/ uintptr_t lmt_base; diff --git a/drivers/common/cnxk/roc_nix_debug.c b/drivers/common/cnxk/roc_nix_debug.c index 582f5a3..266935a 100644 --- a/drivers/common/cnxk/roc_nix_debug.c +++ b/drivers/common/cnxk/roc_nix_debug.c @@ -818,6 +818,7 @@ roc_nix_rq_dump(struct roc_nix_rq *rq) nix_dump(" vwqe_wait_tmo = %ld", rq->vwqe_wait_tmo); nix_dump(" vwqe_aura_handle = %ld", rq->vwqe_aura_handle); nix_dump(" roc_nix = %p", rq->roc_nix); + nix_dump(" inl_dev_ref = %d", rq->inl_dev_ref); } void @@ -1160,6 +1161,7 @@ roc_nix_dump(struct roc_nix *roc_nix) { struct nix *nix = roc_nix_to_nix_priv(roc_nix); struct dev *dev = &nix->dev; + int i; nix_dump("nix@%p", nix); nix_dump(" pf = %d", dev_get_pf(dev->pf_func)); @@ -1169,6 +1171,7 @@ roc_nix_dump(struct roc_nix *roc_nix) nix_dump(" port_id = %d", roc_nix->port_id); nix_dump(" rss_tag_as_xor = %d", roc_nix->rss_tag_as_xor); nix_dump(" rss_tag_as_xor = %d", roc_nix->max_sqb_count); + nix_dump(" outb_nb_desc = %u", roc_nix->outb_nb_desc); nix_dump(" \tpci_dev = %p", nix->pci_dev); nix_dump(" \tbase = 0x%" PRIxPTR "", nix->base); @@ -1206,12 +1209,24 @@ roc_nix_dump(struct roc_nix *roc_nix) nix_dump(" \ttx_link = %d", nix->tx_link); nix_dump(" \tsqb_size = %d", nix->sqb_size); nix_dump(" \tmsixoff = %d", nix->msixoff); + for (i = 0; i < nix->nb_cpt_lf; i++) + nix_dump(" \tcpt_msixoff[%d] = %d", i, nix->cpt_msixoff[i]); nix_dump(" \tcints = %d", nix->cints); nix_dump(" \tqints = %d", nix->qints); nix_dump(" \tsdp_link = %d", nix->sdp_link); nix_dump(" \tptp_en = %d", nix->ptp_en); nix_dump(" \trss_alg_idx = %d", nix->rss_alg_idx); nix_dump(" \ttx_pause = %d", nix->tx_pause); + nix_dump(" \tinl_inb_ena = %d", nix->inl_inb_ena); + nix_dump(" \tinl_outb_ena = %d", nix->inl_outb_ena); + nix_dump(" \tinb_sa_base = 0x%p", nix->inb_sa_base); + nix_dump(" \tinb_sa_sz = %" PRIu64, nix->inb_sa_sz); + nix_dump(" \toutb_sa_base = 0x%p", nix->outb_sa_base); + nix_dump(" \toutb_sa_sz = %" PRIu64, nix->outb_sa_sz); + nix_dump(" \toutb_err_sso_pffunc = 0x%x", nix->outb_err_sso_pffunc); + nix_dump(" \tcpt_lf_base = 0x%p", nix->cpt_lf_base); + nix_dump(" \tnb_cpt_lf = %d", nix->nb_cpt_lf); + nix_dump(" \tinb_inl_dev = %d", nix->inb_inl_dev); } void diff --git a/drivers/common/cnxk/roc_nix_inl.c b/drivers/common/cnxk/roc_nix_inl.c new file mode 100644 index 0000000..d144b19 --- /dev/null +++ b/drivers/common/cnxk/roc_nix_inl.c @@ -0,0 +1,739 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "roc_api.h" +#include "roc_priv.h" + +PLT_STATIC_ASSERT(ROC_NIX_INL_ONF_IPSEC_INB_SA_SZ == + 1UL << ROC_NIX_INL_ONF_IPSEC_INB_SA_SZ_LOG2); +PLT_STATIC_ASSERT(ROC_NIX_INL_ONF_IPSEC_INB_SA_SZ == 512); +PLT_STATIC_ASSERT(ROC_NIX_INL_ONF_IPSEC_OUTB_SA_SZ == + 1UL << ROC_NIX_INL_ONF_IPSEC_OUTB_SA_SZ_LOG2); +PLT_STATIC_ASSERT(ROC_NIX_INL_OT_IPSEC_INB_SA_SZ == + 1UL << ROC_NIX_INL_OT_IPSEC_INB_SA_SZ_LOG2); +PLT_STATIC_ASSERT(ROC_NIX_INL_OT_IPSEC_INB_SA_SZ == 1024); +PLT_STATIC_ASSERT(ROC_NIX_INL_OT_IPSEC_OUTB_SA_SZ == + 1UL << ROC_NIX_INL_OT_IPSEC_OUTB_SA_SZ_LOG2); + +static int +nix_inl_inb_sa_tbl_setup(struct roc_nix *roc_nix) +{ + uint16_t ipsec_in_max_spi = roc_nix->ipsec_in_max_spi; + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + struct roc_nix_ipsec_cfg cfg; + size_t inb_sa_sz; + int rc; + + /* CN9K SA size is different */ + if (roc_model_is_cn9k()) + inb_sa_sz = ROC_NIX_INL_ONF_IPSEC_INB_SA_SZ; + else + inb_sa_sz = ROC_NIX_INL_OT_IPSEC_INB_SA_SZ; + + /* Alloc contiguous memory for Inbound SA's */ + nix->inb_sa_sz = inb_sa_sz; + nix->inb_sa_base = plt_zmalloc(inb_sa_sz * ipsec_in_max_spi, + ROC_NIX_INL_SA_BASE_ALIGN); + if (!nix->inb_sa_base) { + plt_err("Failed to allocate memory for Inbound SA"); + return -ENOMEM; + } + + memset(&cfg, 0, sizeof(cfg)); + cfg.sa_size = inb_sa_sz; + cfg.iova = (uintptr_t)nix->inb_sa_base; + cfg.max_sa = ipsec_in_max_spi + 1; + cfg.tt = SSO_TT_ORDERED; + + /* Setup device specific inb SA table */ + rc = roc_nix_lf_inl_ipsec_cfg(roc_nix, &cfg, true); + if (rc) { + plt_err("Failed to setup NIX Inbound SA conf, rc=%d", rc); + goto free_mem; + } + + return 0; +free_mem: + plt_free(nix->inb_sa_base); + nix->inb_sa_base = NULL; + return rc; +} + +static int +nix_inl_sa_tbl_release(struct roc_nix *roc_nix) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + int rc; + + rc = roc_nix_lf_inl_ipsec_cfg(roc_nix, NULL, false); + if (rc) { + plt_err("Failed to disable Inbound inline ipsec, rc=%d", rc); + return rc; + } + + plt_free(nix->inb_sa_base); + nix->inb_sa_base = NULL; + return 0; +} + +struct roc_cpt_lf * +roc_nix_inl_outb_lf_base_get(struct roc_nix *roc_nix) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + + /* NIX Inline config needs to be done */ + if (!nix->inl_outb_ena || !nix->cpt_lf_base) + return NULL; + + return (struct roc_cpt_lf *)nix->cpt_lf_base; +} + +uintptr_t +roc_nix_inl_outb_sa_base_get(struct roc_nix *roc_nix) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + + return (uintptr_t)nix->outb_sa_base; +} + +uintptr_t +roc_nix_inl_inb_sa_base_get(struct roc_nix *roc_nix, bool inb_inl_dev) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + struct idev_cfg *idev = idev_get_cfg(); + struct nix_inl_dev *inl_dev; + + if (idev == NULL) + return 0; + + if (!nix->inl_inb_ena) + return 0; + + inl_dev = idev->nix_inl_dev; + if (inb_inl_dev) { + /* Return inline dev sa base */ + if (inl_dev) + return (uintptr_t)inl_dev->inb_sa_base; + return 0; + } + + return (uintptr_t)nix->inb_sa_base; +} + +uint32_t +roc_nix_inl_inb_sa_max_spi(struct roc_nix *roc_nix, bool inb_inl_dev) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + struct idev_cfg *idev = idev_get_cfg(); + struct nix_inl_dev *inl_dev; + + if (idev == NULL) + return 0; + + if (!nix->inl_inb_ena) + return 0; + + inl_dev = idev->nix_inl_dev; + if (inb_inl_dev) { + if (inl_dev) + return inl_dev->ipsec_in_max_spi; + return 0; + } + + return roc_nix->ipsec_in_max_spi; +} + +uint32_t +roc_nix_inl_inb_sa_sz(struct roc_nix *roc_nix, bool inl_dev_sa) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + struct idev_cfg *idev = idev_get_cfg(); + struct nix_inl_dev *inl_dev; + + if (idev == NULL) + return 0; + + if (!inl_dev_sa) + return nix->inb_sa_sz; + + inl_dev = idev->nix_inl_dev; + if (inl_dev_sa && inl_dev) + return inl_dev->inb_sa_sz; + + /* On error */ + return 0; +} + +uintptr_t +roc_nix_inl_inb_sa_get(struct roc_nix *roc_nix, bool inb_inl_dev, uint32_t spi) +{ + uintptr_t sa_base; + uint32_t max_spi; + uint64_t sz; + + sa_base = roc_nix_inl_inb_sa_base_get(roc_nix, inb_inl_dev); + /* Check if SA base exists */ + if (!sa_base) + return 0; + + /* Check if SPI is in range */ + max_spi = roc_nix_inl_inb_sa_max_spi(roc_nix, inb_inl_dev); + if (spi > max_spi) { + plt_err("Inbound SA SPI %u exceeds max %u", spi, max_spi); + return 0; + } + + /* Get SA size */ + sz = roc_nix_inl_inb_sa_sz(roc_nix, inb_inl_dev); + if (!sz) + return 0; + + /* Basic logic of SPI->SA for now */ + return (sa_base + (spi * sz)); +} + +int +roc_nix_inl_inb_init(struct roc_nix *roc_nix) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + struct idev_cfg *idev = idev_get_cfg(); + struct roc_cpt *roc_cpt; + uint16_t param1; + int rc; + + if (idev == NULL) + return -ENOTSUP; + + /* Unless we have another mechanism to trigger + * onetime Inline config in CPTPF, we cannot + * support without CPT being probed. + */ + roc_cpt = idev->cpt; + if (!roc_cpt) { + plt_err("Cannot support inline inbound, cryptodev not probed"); + return -ENOTSUP; + } + + if (roc_model_is_cn9k()) { + param1 = ROC_ONF_IPSEC_INB_MAX_L2_SZ; + } else { + union roc_ot_ipsec_inb_param1 u; + + u.u16 = 0; + u.s.esp_trailer_disable = 1; + param1 = u.u16; + } + + /* Do onetime Inbound Inline config in CPTPF */ + rc = roc_cpt_inline_ipsec_inb_cfg(roc_cpt, param1, 0); + if (rc && rc != -EEXIST) { + plt_err("Failed to setup inbound lf, rc=%d", rc); + return rc; + } + + /* Setup Inbound SA table */ + rc = nix_inl_inb_sa_tbl_setup(roc_nix); + if (rc) + return rc; + + nix->inl_inb_ena = true; + return 0; +} + +int +roc_nix_inl_inb_fini(struct roc_nix *roc_nix) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + + if (!nix->inl_inb_ena) + return 0; + + nix->inl_inb_ena = false; + + /* Disable Inbound SA */ + return nix_inl_sa_tbl_release(roc_nix); +} + +int +roc_nix_inl_outb_init(struct roc_nix *roc_nix) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + struct idev_cfg *idev = idev_get_cfg(); + struct roc_cpt_lf *lf_base, *lf; + struct dev *dev = &nix->dev; + struct msix_offset_rsp *rsp; + struct nix_inl_dev *inl_dev; + uint16_t sso_pffunc; + uint8_t eng_grpmask; + uint64_t blkaddr; + uint16_t nb_lf; + void *sa_base; + size_t sa_sz; + int i, j, rc; + + if (idev == NULL) + return -ENOTSUP; + + nb_lf = roc_nix->outb_nb_crypto_qs; + blkaddr = nix->is_nix1 ? RVU_BLOCK_ADDR_CPT1 : RVU_BLOCK_ADDR_CPT0; + + /* Retrieve inline device if present */ + inl_dev = idev->nix_inl_dev; + sso_pffunc = inl_dev ? inl_dev->dev.pf_func : idev_sso_pffunc_get(); + if (!sso_pffunc) { + plt_err("Failed to setup inline outb, need either " + "inline device or sso device"); + return -ENOTSUP; + } + + /* Attach CPT LF for outbound */ + rc = cpt_lfs_attach(dev, blkaddr, true, nb_lf); + if (rc) { + plt_err("Failed to attach CPT LF for inline outb, rc=%d", rc); + return rc; + } + + /* Alloc CPT LF */ + eng_grpmask = (1ULL << ROC_CPT_DFLT_ENG_GRP_SE | + 1ULL << ROC_CPT_DFLT_ENG_GRP_SE_IE | + 1ULL << ROC_CPT_DFLT_ENG_GRP_AE); + rc = cpt_lfs_alloc(dev, eng_grpmask, blkaddr, true); + if (rc) { + plt_err("Failed to alloc CPT LF resources, rc=%d", rc); + goto lf_detach; + } + + /* Get msix offsets */ + rc = cpt_get_msix_offset(dev, &rsp); + if (rc) { + plt_err("Failed to get CPT LF msix offset, rc=%d", rc); + goto lf_free; + } + + mbox_memcpy(nix->cpt_msixoff, + nix->is_nix1 ? rsp->cpt1_lf_msixoff : rsp->cptlf_msixoff, + sizeof(nix->cpt_msixoff)); + + /* Alloc required num of cpt lfs */ + lf_base = plt_zmalloc(nb_lf * sizeof(struct roc_cpt_lf), 0); + if (!lf_base) { + plt_err("Failed to alloc cpt lf memory"); + rc = -ENOMEM; + goto lf_free; + } + + /* Initialize CPT LF's */ + for (i = 0; i < nb_lf; i++) { + lf = &lf_base[i]; + + lf->lf_id = i; + lf->nb_desc = roc_nix->outb_nb_desc; + lf->dev = &nix->dev; + lf->msixoff = nix->cpt_msixoff[i]; + lf->pci_dev = nix->pci_dev; + + /* Setup CPT LF instruction queue */ + rc = cpt_lf_init(lf); + if (rc) { + plt_err("Failed to initialize CPT LF, rc=%d", rc); + goto lf_fini; + } + + /* Associate this CPT LF with NIX PFFUNC */ + rc = cpt_lf_outb_cfg(dev, sso_pffunc, nix->dev.pf_func, i, + true); + if (rc) { + plt_err("Failed to setup CPT LF->(NIX,SSO) link, rc=%d", + rc); + goto lf_fini; + } + + /* Enable IQ */ + roc_cpt_iq_enable(lf); + } + + if (!roc_nix->ipsec_out_max_sa) + goto skip_sa_alloc; + + /* CN9K SA size is different */ + if (roc_model_is_cn9k()) + sa_sz = ROC_NIX_INL_ONF_IPSEC_OUTB_SA_SZ; + else + sa_sz = ROC_NIX_INL_OT_IPSEC_OUTB_SA_SZ; + /* Alloc contiguous memory of outbound SA */ + sa_base = plt_zmalloc(sa_sz * roc_nix->ipsec_out_max_sa, + ROC_NIX_INL_SA_BASE_ALIGN); + if (!sa_base) { + plt_err("Outbound SA base alloc failed"); + goto lf_fini; + } + nix->outb_sa_base = sa_base; + nix->outb_sa_sz = sa_sz; + nix->cpt_lf_base = lf_base; + nix->nb_cpt_lf = nb_lf; + nix->outb_err_sso_pffunc = sso_pffunc; + +skip_sa_alloc: + nix->inl_outb_ena = true; + return 0; +lf_fini: + for (j = i - 1; j >= 0; j--) + cpt_lf_fini(&lf_base[j]); + plt_free(lf_base); +lf_free: + rc |= cpt_lfs_free(dev); +lf_detach: + rc |= cpt_lfs_detach(dev); + return rc; +} + +int +roc_nix_inl_outb_fini(struct roc_nix *roc_nix) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + struct roc_cpt_lf *lf_base = nix->cpt_lf_base; + struct dev *dev = &nix->dev; + int i, rc, ret = 0; + + if (!nix->inl_outb_ena) + return 0; + + nix->inl_outb_ena = false; + + /* Cleanup CPT LF instruction queue */ + for (i = 0; i < nix->nb_cpt_lf; i++) + cpt_lf_fini(&lf_base[i]); + + /* Free LF resources */ + rc = cpt_lfs_free(dev); + if (rc) + plt_err("Failed to free CPT LF resources, rc=%d", rc); + ret |= rc; + + /* Detach LF */ + rc = cpt_lfs_detach(dev); + if (rc) + plt_err("Failed to detach CPT LF, rc=%d", rc); + + /* Free LF memory */ + plt_free(lf_base); + nix->cpt_lf_base = NULL; + nix->nb_cpt_lf = 0; + + /* Free outbound SA base */ + plt_free(nix->outb_sa_base); + nix->outb_sa_base = NULL; + + ret |= rc; + return ret; +} + +bool +roc_nix_inl_dev_is_probed(void) +{ + struct idev_cfg *idev = idev_get_cfg(); + + if (idev == NULL) + return 0; + + return !!idev->nix_inl_dev; +} + +bool +roc_nix_inl_inb_is_enabled(struct roc_nix *roc_nix) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + + return nix->inl_inb_ena; +} + +bool +roc_nix_inl_outb_is_enabled(struct roc_nix *roc_nix) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + + return nix->inl_outb_ena; +} + +int +roc_nix_inl_dev_rq_get(struct roc_nix_rq *rq) +{ + struct idev_cfg *idev = idev_get_cfg(); + struct nix_inl_dev *inl_dev; + struct roc_nix_rq *inl_rq; + struct dev *dev; + int rc; + + if (idev == NULL) + return 0; + + inl_dev = idev->nix_inl_dev; + /* Nothing to do if no inline device */ + if (!inl_dev) + return 0; + + /* Just take reference if already inited */ + if (inl_dev->rq_refs) { + inl_dev->rq_refs++; + rq->inl_dev_ref = true; + return 0; + } + + dev = &inl_dev->dev; + inl_rq = &inl_dev->rq; + memset(inl_rq, 0, sizeof(struct roc_nix_rq)); + + /* Take RQ pool attributes from the first ethdev RQ */ + inl_rq->qid = 0; + inl_rq->aura_handle = rq->aura_handle; + inl_rq->first_skip = rq->first_skip; + inl_rq->later_skip = rq->later_skip; + inl_rq->lpb_size = rq->lpb_size; + + /* Enable IPSec */ + inl_rq->ipsech_ena = true; + + inl_rq->flow_tag_width = 20; + /* Special tag mask */ + inl_rq->tag_mask = 0xFFF00000; + inl_rq->tt = SSO_TT_ORDERED; + inl_rq->hwgrp = 0; + inl_rq->wqe_skip = 1; + inl_rq->sso_ena = true; + + /* Prepare and send RQ init mbox */ + if (roc_model_is_cn9k()) + rc = nix_rq_cn9k_cfg(dev, inl_rq, inl_dev->qints, false, true); + else + rc = nix_rq_cfg(dev, inl_rq, inl_dev->qints, false, true); + if (rc) { + plt_err("Failed to prepare aq_enq msg, rc=%d", rc); + return rc; + } + + rc = mbox_process(dev->mbox); + if (rc) { + plt_err("Failed to send aq_enq msg, rc=%d", rc); + return rc; + } + + inl_dev->rq_refs++; + rq->inl_dev_ref = true; + return 0; +} + +int +roc_nix_inl_dev_rq_put(struct roc_nix_rq *rq) +{ + struct idev_cfg *idev = idev_get_cfg(); + struct nix_inl_dev *inl_dev; + struct roc_nix_rq *inl_rq; + struct dev *dev; + int rc; + + if (idev == NULL) + return 0; + + if (!rq->inl_dev_ref) + return 0; + + inl_dev = idev->nix_inl_dev; + /* Inline device should be there if we have ref */ + if (!inl_dev) { + plt_err("Failed to find inline device with refs"); + return -EFAULT; + } + + rq->inl_dev_ref = false; + inl_dev->rq_refs--; + if (inl_dev->rq_refs) + return 0; + + dev = &inl_dev->dev; + inl_rq = &inl_dev->rq; + /* There are no more references, disable RQ */ + rc = nix_rq_ena_dis(dev, inl_rq, false); + if (rc) + plt_err("Failed to disable inline device rq, rc=%d", rc); + + /* Flush NIX LF for CN10K */ + if (roc_model_is_cn10k()) + plt_write64(0, inl_dev->nix_base + NIX_LF_OP_VWQE_FLUSH); + + return rc; +} + +void +roc_nix_inb_mode_set(struct roc_nix *roc_nix, bool use_inl_dev) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + + /* Info used by NPC flow rule add */ + nix->inb_inl_dev = use_inl_dev; +} + +bool +roc_nix_inb_is_with_inl_dev(struct roc_nix *roc_nix) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + + return nix->inb_inl_dev; +} + +struct roc_nix_rq * +roc_nix_inl_dev_rq(void) +{ + struct idev_cfg *idev = idev_get_cfg(); + struct nix_inl_dev *inl_dev; + + if (idev != NULL) { + inl_dev = idev->nix_inl_dev; + if (inl_dev != NULL) + return &inl_dev->rq; + } + + return NULL; +} + +uint16_t __roc_api +roc_nix_inl_outb_sso_pffunc_get(struct roc_nix *roc_nix) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + + return nix->outb_err_sso_pffunc; +} + +int +roc_nix_inl_cb_register(roc_nix_inl_sso_work_cb_t cb, void *args) +{ + struct idev_cfg *idev = idev_get_cfg(); + struct nix_inl_dev *inl_dev; + + if (idev == NULL) + return -EIO; + + inl_dev = idev->nix_inl_dev; + if (!inl_dev) + return -EIO; + + /* Be silent if registration called with same cb and args */ + if (inl_dev->work_cb == cb && inl_dev->cb_args == args) + return 0; + + /* Don't allow registration again if registered with different cb */ + if (inl_dev->work_cb) + return -EBUSY; + + inl_dev->work_cb = cb; + inl_dev->cb_args = args; + return 0; +} + +int +roc_nix_inl_cb_unregister(roc_nix_inl_sso_work_cb_t cb, void *args) +{ + struct idev_cfg *idev = idev_get_cfg(); + struct nix_inl_dev *inl_dev; + + if (idev == NULL) + return -ENOENT; + + inl_dev = idev->nix_inl_dev; + if (!inl_dev) + return -ENOENT; + + if (inl_dev->work_cb != cb || inl_dev->cb_args != args) + return -EINVAL; + + inl_dev->work_cb = NULL; + inl_dev->cb_args = NULL; + return 0; +} + +int +roc_nix_inl_inb_tag_update(struct roc_nix *roc_nix, uint32_t tag_const, + uint8_t tt) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + struct roc_nix_ipsec_cfg cfg; + + /* Be silent if inline inbound not enabled */ + if (!nix->inl_inb_ena) + return 0; + + memset(&cfg, 0, sizeof(cfg)); + cfg.sa_size = nix->inb_sa_sz; + cfg.iova = (uintptr_t)nix->inb_sa_base; + cfg.max_sa = roc_nix->ipsec_in_max_spi + 1; + cfg.tt = tt; + cfg.tag_const = tag_const; + + return roc_nix_lf_inl_ipsec_cfg(roc_nix, &cfg, true); +} + +int +roc_nix_inl_sa_sync(struct roc_nix *roc_nix, void *sa, bool inb, + enum roc_nix_inl_sa_sync_op op) +{ + struct nix *nix = roc_nix_to_nix_priv(roc_nix); + struct roc_cpt_lf *outb_lf = nix->cpt_lf_base; + union cpt_lf_ctx_reload reload; + union cpt_lf_ctx_flush flush; + uintptr_t rbase; + + /* Nothing much to do on cn9k */ + if (roc_model_is_cn9k()) { + plt_atomic_thread_fence(__ATOMIC_ACQ_REL); + return 0; + } + + if (!inb && !outb_lf) + return -EINVAL; + + /* Performing op via outbound lf is enough + * when inline dev is not in use. + */ + if (outb_lf && !nix->inb_inl_dev) { + rbase = outb_lf->rbase; + + flush.u = 0; + reload.u = 0; + switch (op) { + case ROC_NIX_INL_SA_OP_FLUSH_INVAL: + flush.s.inval = 1; + /* fall through */ + case ROC_NIX_INL_SA_OP_FLUSH: + flush.s.cptr = ((uintptr_t)sa) >> 7; + plt_write64(flush.u, rbase + CPT_LF_CTX_FLUSH); + break; + case ROC_NIX_INL_SA_OP_RELOAD: + reload.s.cptr = ((uintptr_t)sa) >> 7; + plt_write64(reload.u, rbase + CPT_LF_CTX_RELOAD); + break; + default: + return -EINVAL; + } + return 0; + } + + return -ENOTSUP; +} + +void +roc_nix_inl_dev_lock(void) +{ + struct idev_cfg *idev = idev_get_cfg(); + + if (idev != NULL) + plt_spinlock_lock(&idev->nix_inl_dev_lock); +} + +void +roc_nix_inl_dev_unlock(void) +{ + struct idev_cfg *idev = idev_get_cfg(); + + if (idev != NULL) + plt_spinlock_unlock(&idev->nix_inl_dev_lock); +} diff --git a/drivers/common/cnxk/roc_nix_inl.h b/drivers/common/cnxk/roc_nix_inl.h index f1fe4a2..efc5a19 100644 --- a/drivers/common/cnxk/roc_nix_inl.h +++ b/drivers/common/cnxk/roc_nix_inl.h @@ -43,6 +43,62 @@ /* Alignment of SA Base */ #define ROC_NIX_INL_SA_BASE_ALIGN BIT_ULL(16) +static inline struct roc_onf_ipsec_inb_sa * +roc_nix_inl_onf_ipsec_inb_sa(uintptr_t base, uint64_t idx) +{ + uint64_t off = idx << ROC_NIX_INL_ONF_IPSEC_INB_SA_SZ_LOG2; + + return PLT_PTR_ADD(base, off); +} + +static inline struct roc_onf_ipsec_outb_sa * +roc_nix_inl_onf_ipsec_outb_sa(uintptr_t base, uint64_t idx) +{ + uint64_t off = idx << ROC_NIX_INL_ONF_IPSEC_OUTB_SA_SZ_LOG2; + + return PLT_PTR_ADD(base, off); +} + +static inline void * +roc_nix_inl_onf_ipsec_inb_sa_sw_rsvd(void *sa) +{ + return PLT_PTR_ADD(sa, ROC_NIX_INL_ONF_IPSEC_INB_HW_SZ); +} + +static inline void * +roc_nix_inl_onf_ipsec_outb_sa_sw_rsvd(void *sa) +{ + return PLT_PTR_ADD(sa, ROC_NIX_INL_ONF_IPSEC_OUTB_HW_SZ); +} + +static inline struct roc_ot_ipsec_inb_sa * +roc_nix_inl_ot_ipsec_inb_sa(uintptr_t base, uint64_t idx) +{ + uint64_t off = idx << ROC_NIX_INL_OT_IPSEC_INB_SA_SZ_LOG2; + + return PLT_PTR_ADD(base, off); +} + +static inline struct roc_ot_ipsec_outb_sa * +roc_nix_inl_ot_ipsec_outb_sa(uintptr_t base, uint64_t idx) +{ + uint64_t off = idx << ROC_NIX_INL_OT_IPSEC_OUTB_SA_SZ_LOG2; + + return PLT_PTR_ADD(base, off); +} + +static inline void * +roc_nix_inl_ot_ipsec_inb_sa_sw_rsvd(void *sa) +{ + return PLT_PTR_ADD(sa, ROC_NIX_INL_OT_IPSEC_INB_HW_SZ); +} + +static inline void * +roc_nix_inl_ot_ipsec_outb_sa_sw_rsvd(void *sa) +{ + return PLT_PTR_ADD(sa, ROC_NIX_INL_OT_IPSEC_OUTB_HW_SZ); +} + /* Inline device SSO Work callback */ typedef void (*roc_nix_inl_sso_work_cb_t)(uint64_t *gw, void *args); @@ -61,5 +117,49 @@ struct roc_nix_inl_dev { int __roc_api roc_nix_inl_dev_init(struct roc_nix_inl_dev *roc_inl_dev); int __roc_api roc_nix_inl_dev_fini(struct roc_nix_inl_dev *roc_inl_dev); void __roc_api roc_nix_inl_dev_dump(struct roc_nix_inl_dev *roc_inl_dev); +bool __roc_api roc_nix_inl_dev_is_probed(void); +void __roc_api roc_nix_inl_dev_lock(void); +void __roc_api roc_nix_inl_dev_unlock(void); + +/* NIX Inline Inbound API */ +int __roc_api roc_nix_inl_inb_init(struct roc_nix *roc_nix); +int __roc_api roc_nix_inl_inb_fini(struct roc_nix *roc_nix); +bool __roc_api roc_nix_inl_inb_is_enabled(struct roc_nix *roc_nix); +uintptr_t __roc_api roc_nix_inl_inb_sa_base_get(struct roc_nix *roc_nix, + bool inl_dev_sa); +uint32_t __roc_api roc_nix_inl_inb_sa_max_spi(struct roc_nix *roc_nix, + bool inl_dev_sa); +uint32_t __roc_api roc_nix_inl_inb_sa_sz(struct roc_nix *roc_nix, + bool inl_dev_sa); +uintptr_t __roc_api roc_nix_inl_inb_sa_get(struct roc_nix *roc_nix, + bool inl_dev_sa, uint32_t spi); +void __roc_api roc_nix_inb_mode_set(struct roc_nix *roc_nix, bool use_inl_dev); +int __roc_api roc_nix_inl_dev_rq_get(struct roc_nix_rq *rq); +int __roc_api roc_nix_inl_dev_rq_put(struct roc_nix_rq *rq); +bool __roc_api roc_nix_inb_is_with_inl_dev(struct roc_nix *roc_nix); +struct roc_nix_rq *__roc_api roc_nix_inl_dev_rq(void); +int __roc_api roc_nix_inl_inb_tag_update(struct roc_nix *roc_nix, + uint32_t tag_const, uint8_t tt); + +/* NIX Inline Outbound API */ +int __roc_api roc_nix_inl_outb_init(struct roc_nix *roc_nix); +int __roc_api roc_nix_inl_outb_fini(struct roc_nix *roc_nix); +bool __roc_api roc_nix_inl_outb_is_enabled(struct roc_nix *roc_nix); +uintptr_t __roc_api roc_nix_inl_outb_sa_base_get(struct roc_nix *roc_nix); +struct roc_cpt_lf *__roc_api +roc_nix_inl_outb_lf_base_get(struct roc_nix *roc_nix); +uint16_t __roc_api roc_nix_inl_outb_sso_pffunc_get(struct roc_nix *roc_nix); +int __roc_api roc_nix_inl_cb_register(roc_nix_inl_sso_work_cb_t cb, void *args); +int __roc_api roc_nix_inl_cb_unregister(roc_nix_inl_sso_work_cb_t cb, + void *args); +/* NIX Inline/Outbound API */ +enum roc_nix_inl_sa_sync_op { + ROC_NIX_INL_SA_OP_FLUSH, + ROC_NIX_INL_SA_OP_FLUSH_INVAL, + ROC_NIX_INL_SA_OP_RELOAD, +}; + +int __roc_api roc_nix_inl_sa_sync(struct roc_nix *roc_nix, void *sa, bool inb, + enum roc_nix_inl_sa_sync_op op); #endif /* _ROC_NIX_INL_H_ */ diff --git a/drivers/common/cnxk/roc_nix_priv.h b/drivers/common/cnxk/roc_nix_priv.h index 79c15ea..2cd5a72 100644 --- a/drivers/common/cnxk/roc_nix_priv.h +++ b/drivers/common/cnxk/roc_nix_priv.h @@ -162,6 +162,21 @@ struct nix { uint16_t tm_link_cfg_lvl; uint16_t contig_rsvd[NIX_TXSCH_LVL_CNT]; uint16_t discontig_rsvd[NIX_TXSCH_LVL_CNT]; + + /* Ipsec info */ + uint16_t cpt_msixoff[MAX_RVU_BLKLF_CNT]; + bool inl_inb_ena; + bool inl_outb_ena; + void *inb_sa_base; + size_t inb_sa_sz; + void *outb_sa_base; + size_t outb_sa_sz; + uint16_t outb_err_sso_pffunc; + struct roc_cpt_lf *cpt_lf_base; + uint16_t nb_cpt_lf; + /* Mode provided by driver */ + bool inb_inl_dev; + } __plt_cache_aligned; enum nix_err_status { diff --git a/drivers/common/cnxk/roc_npc.c b/drivers/common/cnxk/roc_npc.c index aff4eef..f13331f 100644 --- a/drivers/common/cnxk/roc_npc.c +++ b/drivers/common/cnxk/roc_npc.c @@ -340,10 +340,11 @@ roc_npc_fini(struct roc_npc *roc_npc) } static int -npc_parse_actions(struct npc *npc, const struct roc_npc_attr *attr, +npc_parse_actions(struct roc_npc *roc_npc, const struct roc_npc_attr *attr, const struct roc_npc_action actions[], struct roc_npc_flow *flow) { + struct npc *npc = roc_npc_to_npc_priv(roc_npc); const struct roc_npc_action_mark *act_mark; const struct roc_npc_action_queue *act_q; const struct roc_npc_action_vf *vf_act; @@ -425,15 +426,16 @@ npc_parse_actions(struct npc *npc, const struct roc_npc_attr *attr, * NPC_SECURITY_ACTION_TYPE_INLINE_PROTOCOL && * session_protocol == * NPC_SECURITY_PROTOCOL_IPSEC - * - * RSS is not supported with inline ipsec. Get the - * rq from associated conf, or make - * ROC_NPC_ACTION_TYPE_QUEUE compulsory with this - * action. - * Currently, rq = 0 is assumed. */ req_act |= ROC_NPC_ACTION_TYPE_SEC; rq = 0; + + /* Special processing when with inline device */ + if (roc_nix_inb_is_with_inl_dev(roc_npc->roc_nix) && + roc_nix_inl_dev_is_probed()) { + rq = 0; + pf_func = nix_inl_dev_pffunc_get(); + } break; case ROC_NPC_ACTION_TYPE_VLAN_STRIP: req_act |= ROC_NPC_ACTION_TYPE_VLAN_STRIP; @@ -660,11 +662,12 @@ npc_parse_attr(struct npc *npc, const struct roc_npc_attr *attr, } static int -npc_parse_rule(struct npc *npc, const struct roc_npc_attr *attr, +npc_parse_rule(struct roc_npc *roc_npc, const struct roc_npc_attr *attr, const struct roc_npc_item_info pattern[], const struct roc_npc_action actions[], struct roc_npc_flow *flow, struct npc_parse_state *pst) { + struct npc *npc = roc_npc_to_npc_priv(roc_npc); int err; /* Check attr */ @@ -678,7 +681,7 @@ npc_parse_rule(struct npc *npc, const struct roc_npc_attr *attr, return err; /* Check action */ - err = npc_parse_actions(npc, attr, actions, flow); + err = npc_parse_actions(roc_npc, attr, actions, flow); if (err) return err; return 0; @@ -694,7 +697,8 @@ roc_npc_flow_parse(struct roc_npc *roc_npc, const struct roc_npc_attr *attr, struct npc_parse_state parse_state = {0}; int rc; - rc = npc_parse_rule(npc, attr, pattern, actions, flow, &parse_state); + rc = npc_parse_rule(roc_npc, attr, pattern, actions, flow, + &parse_state); if (rc) return rc; @@ -1018,7 +1022,8 @@ roc_npc_flow_create(struct roc_npc *roc_npc, const struct roc_npc_attr *attr, } memset(flow, 0, sizeof(*flow)); - rc = npc_parse_rule(npc, attr, pattern, actions, flow, &parse_state); + rc = npc_parse_rule(roc_npc, attr, pattern, actions, flow, + &parse_state); if (rc != 0) { *errcode = rc; goto err_exit; diff --git a/drivers/common/cnxk/version.map b/drivers/common/cnxk/version.map index 3a35233..9fcc677 100644 --- a/drivers/common/cnxk/version.map +++ b/drivers/common/cnxk/version.map @@ -100,9 +100,34 @@ INTERNAL { roc_nix_get_pf_func; roc_nix_get_vf; roc_nix_get_vwqe_interval; + roc_nix_inl_cb_register; + roc_nix_inl_cb_unregister; roc_nix_inl_dev_dump; roc_nix_inl_dev_fini; roc_nix_inl_dev_init; + roc_nix_inl_dev_is_probed; + roc_nix_inl_dev_lock; + roc_nix_inl_dev_unlock; + roc_nix_inl_dev_rq; + roc_nix_inl_dev_rq_get; + roc_nix_inl_dev_rq_put; + roc_nix_inl_inb_is_enabled; + roc_nix_inl_inb_init; + roc_nix_inl_inb_sa_base_get; + roc_nix_inl_inb_sa_get; + roc_nix_inl_inb_sa_max_spi; + roc_nix_inl_inb_sa_sz; + roc_nix_inl_inb_tag_update; + roc_nix_inl_inb_fini; + roc_nix_inb_is_with_inl_dev; + roc_nix_inb_mode_set; + roc_nix_inl_outb_fini; + roc_nix_inl_outb_init; + roc_nix_inl_outb_lf_base_get; + roc_nix_inl_outb_sa_base_get; + roc_nix_inl_outb_sso_pffunc_get; + roc_nix_inl_outb_is_enabled; + roc_nix_inl_sa_sync; roc_nix_is_lbk; roc_nix_is_pf; roc_nix_is_sdp; From patchwork Thu Sep 2 02:14:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97733 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AF54EA0C4C; Thu, 2 Sep 2021 04:17:33 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BF8FD410EA; Thu, 2 Sep 2021 04:17:11 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id C342D410EA for ; Thu, 2 Sep 2021 04:17:09 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181K5SYj027320 for ; Wed, 1 Sep 2021 19:17:09 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=eHZsMW8bkw1WaiP4DG9B9rVEcb6KEgERNiS8rMahLsY=; b=KO5aX+32/cjl7I1GEAapNr5DReJjgBJTdIEwDS8ll6addgamcUn1uNLPVz8duMxxj6dZ xALIdiPSfSuLTjKEApbpAQkv8gQB53FUZVHuUlkwMcyeuOA59hoEkJfaTpTM0tcCN6MC hsTGuMGiCe4idojgq9mBcuSbcUr1Qu8xFcMgNn0OPQLvq+hCkMLXemDht/VPlBJX5yZe dc7GPwQCAHI7iECmF7eFYvegpZ7RWTvLxsSHC4ydCiFRcrbHzAggo4R+CWYwRGInTnZX fFHhIM2GdNskPne0oSgg8TA2+c9r6MhzZK6mjnrC3vU15Do91DTsld5CGwRBH35BKWzj GA== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com with ESMTP id 3atg8a91h4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:08 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:07 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:07 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 4DE4B3F704D; Wed, 1 Sep 2021 19:17:05 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:46 +0530 Message-ID: <20210902021505.17607-9-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: 10pWd-VwpXagIGlrmlv9dmU1R3hKo9u- X-Proofpoint-ORIG-GUID: 10pWd-VwpXagIGlrmlv9dmU1R3hKo9u- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 08/27] common/cnxk: dump cpt lf registers on error intr X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Dump CPT LF registers on error interrupt. Signed-off-by: Nithin Dabilpuram --- drivers/common/cnxk/roc_cpt.c | 5 ++++- drivers/common/cnxk/roc_cpt_debug.c | 32 ++++++++++++++++++++++++++++++-- drivers/common/cnxk/roc_cpt_priv.h | 1 + 3 files changed, 35 insertions(+), 3 deletions(-) diff --git a/drivers/common/cnxk/roc_cpt.c b/drivers/common/cnxk/roc_cpt.c index 3222b3e..f08b5d0 100644 --- a/drivers/common/cnxk/roc_cpt.c +++ b/drivers/common/cnxk/roc_cpt.c @@ -51,6 +51,9 @@ cpt_lf_misc_irq(void *param) plt_err("Err_irq=0x%" PRIx64 " pf=%d, vf=%d", intr, dev->pf, dev->vf); + /* Dump lf registers */ + cpt_lf_print(lf); + /* Clear interrupt */ plt_write64(intr, lf->rbase + CPT_LF_MISC_INT); } @@ -203,7 +206,7 @@ cpt_lf_dump(struct roc_cpt_lf *lf) plt_cpt_dbg("CPT LF REG:"); plt_cpt_dbg("LF_CTL[0x%016llx]: 0x%016" PRIx64, CPT_LF_CTL, plt_read64(lf->rbase + CPT_LF_CTL)); - plt_cpt_dbg("Q_SIZE[0x%016llx]: 0x%016" PRIx64, CPT_LF_INPROG, + plt_cpt_dbg("LF_INPROG[0x%016llx]: 0x%016" PRIx64, CPT_LF_INPROG, plt_read64(lf->rbase + CPT_LF_INPROG)); plt_cpt_dbg("Q_BASE[0x%016llx]: 0x%016" PRIx64, CPT_LF_Q_BASE, diff --git a/drivers/common/cnxk/roc_cpt_debug.c b/drivers/common/cnxk/roc_cpt_debug.c index a6c9004..847d969 100644 --- a/drivers/common/cnxk/roc_cpt_debug.c +++ b/drivers/common/cnxk/roc_cpt_debug.c @@ -157,11 +157,40 @@ roc_cpt_afs_print(struct roc_cpt *roc_cpt) return 0; } -static void +void cpt_lf_print(struct roc_cpt_lf *lf) { uint64_t reg_val; + reg_val = plt_read64(lf->rbase + CPT_LF_Q_BASE); + plt_print(" CPT_LF_Q_BASE:\t%016lx", reg_val); + + reg_val = plt_read64(lf->rbase + CPT_LF_Q_SIZE); + plt_print(" CPT_LF_Q_SIZE:\t%016lx", reg_val); + + reg_val = plt_read64(lf->rbase + CPT_LF_Q_INST_PTR); + plt_print(" CPT_LF_Q_INST_PTR:\t%016lx", reg_val); + + reg_val = plt_read64(lf->rbase + CPT_LF_Q_GRP_PTR); + plt_print(" CPT_LF_Q_GRP_PTR:\t%016lx", reg_val); + + reg_val = plt_read64(lf->rbase + CPT_LF_CTL); + plt_print(" CPT_LF_CTL:\t%016lx", reg_val); + + reg_val = plt_read64(lf->rbase + CPT_LF_MISC_INT_ENA_W1S); + plt_print(" CPT_LF_MISC_INT_ENA_W1S:\t%016lx", reg_val); + + reg_val = plt_read64(lf->rbase + CPT_LF_MISC_INT); + plt_print(" CPT_LF_MISC_INT:\t%016lx", reg_val); + + reg_val = plt_read64(lf->rbase + CPT_LF_INPROG); + plt_print(" CPT_LF_INPROG:\t%016lx", reg_val); + + if (roc_model_is_cn9k()) + return; + + plt_print("Count registers for CPT LF%d:", lf->lf_id); + reg_val = plt_read64(lf->rbase + CPT_LF_CTX_ENC_BYTE_CNT); plt_print(" Encrypted byte count:\t%" PRIu64, reg_val); @@ -190,7 +219,6 @@ roc_cpt_lfs_print(struct roc_cpt *roc_cpt) if (lf == NULL) continue; - plt_print("Count registers for CPT LF%d:", lf_id); cpt_lf_print(lf); } diff --git a/drivers/common/cnxk/roc_cpt_priv.h b/drivers/common/cnxk/roc_cpt_priv.h index 21911e5..61dec9a 100644 --- a/drivers/common/cnxk/roc_cpt_priv.h +++ b/drivers/common/cnxk/roc_cpt_priv.h @@ -31,5 +31,6 @@ int cpt_lf_outb_cfg(struct dev *dev, uint16_t sso_pf_func, uint16_t nix_pf_func, uint8_t lf_id, bool ena); int cpt_get_msix_offset(struct dev *dev, struct msix_offset_rsp **msix_rsp); uint64_t cpt_get_blkaddr(struct dev *dev); +void cpt_lf_print(struct roc_cpt_lf *lf); #endif /* _ROC_CPT_PRIV_H_ */ From patchwork Thu Sep 2 02:14:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97735 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 521C2A0C4C; Thu, 2 Sep 2021 04:17:50 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B4D1F4115E; Thu, 2 Sep 2021 04:17:14 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id E63FF4114E for ; Thu, 2 Sep 2021 04:17:12 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQSfB012709 for ; Wed, 1 Sep 2021 19:17:12 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=MlliJfOB98bbeqKpWHM5td6qAumeFPvgTlrjWE2/mMY=; b=V6IOxwN+PkBbrYx/n4wKEbgh0RfqoKqEU5c/EwX/oF1Ic4pmffwTz41ixtby6hE9xF/u cZxWxDo1cJAfQJxBBd9qML22BbBxom1+ZusFULAYndeRja5J1jKlGHGgt6eHEYDCXQ2Y G+h47pakNtTWlsPR+ZGuek2FbTIIv0Jn3qLYU0EF2T2Z5cAP+fuZ4DjLq+Fb+P1Sed6d 6xeK4jweWJJgoewDcf6GDLXm8x2DZ2pp2Zg+wgRzlymp0aVWXb4ktWFpeWl4V5PSvtou YvvQmG6yIHIES/95Vi20QPgP6XAIDQVssGqgtGOmtt7mpHiGxP47yPFOXLVbWh1heu30 nQ== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hs2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:12 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:10 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:10 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 0933E3F7040; Wed, 1 Sep 2021 19:17:07 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:47 +0530 Message-ID: <20210902021505.17607-10-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 4BTilk5xaZ-ZGJsRfLvLH0yZwP17-zGM X-Proofpoint-GUID: 4BTilk5xaZ-ZGJsRfLvLH0yZwP17-zGM X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 09/27] common/cnxk: align cpt lf enable/disable sequence X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" For CPT LF IQ enable, set CPT_LF_CTL[ENA] before setting CPT_LF_INPROG[EENA] to true. For CPT LF IQ disable, align sequence to that of HRM. Also this patch aligns space for instructions in CPT LF to ROC_ALIGN to make complete memory cache aligned and has other minor fixes/additions. Signed-off-by: Nithin Dabilpuram --- drivers/common/cnxk/hw/cpt.h | 11 +++++++++++ drivers/common/cnxk/roc_cpt.c | 42 ++++++++++++++++++++++++++++++++++-------- drivers/common/cnxk/roc_cpt.h | 8 ++++++++ 3 files changed, 53 insertions(+), 8 deletions(-) diff --git a/drivers/common/cnxk/hw/cpt.h b/drivers/common/cnxk/hw/cpt.h index 975139f..4d9df59 100644 --- a/drivers/common/cnxk/hw/cpt.h +++ b/drivers/common/cnxk/hw/cpt.h @@ -124,6 +124,17 @@ union cpt_lf_misc_int { } s; }; +union cpt_lf_q_grp_ptr { + uint64_t u; + struct { + uint64_t dq_ptr : 15; + uint64_t reserved_31_15 : 17; + uint64_t nq_ptr : 15; + uint64_t reserved_47_62 : 16; + uint64_t xq_xor : 1; + } s; +}; + union cpt_inst_w4 { uint64_t u64; struct { diff --git a/drivers/common/cnxk/roc_cpt.c b/drivers/common/cnxk/roc_cpt.c index f08b5d0..b30f44e 100644 --- a/drivers/common/cnxk/roc_cpt.c +++ b/drivers/common/cnxk/roc_cpt.c @@ -437,8 +437,10 @@ cpt_lf_iq_mem_calc(uint32_t nb_desc) len += CPT_IQ_FC_LEN; /* For instruction queues */ - len += CPT_IQ_NB_DESC_SIZE_DIV40(nb_desc) * CPT_IQ_NB_DESC_MULTIPLIER * - sizeof(struct cpt_inst_s); + len += PLT_ALIGN(CPT_IQ_NB_DESC_SIZE_DIV40(nb_desc) * + CPT_IQ_NB_DESC_MULTIPLIER * + sizeof(struct cpt_inst_s), + ROC_ALIGN); return len; } @@ -550,6 +552,7 @@ cpt_lf_init(struct roc_cpt_lf *lf) iq_mem = plt_zmalloc(cpt_lf_iq_mem_calc(lf->nb_desc), ROC_ALIGN); if (iq_mem == NULL) return -ENOMEM; + plt_atomic_thread_fence(__ATOMIC_ACQ_REL); blkaddr = cpt_get_blkaddr(dev); lf->rbase = dev->bar2 + ((blkaddr << 20) | (lf->lf_id << 12)); @@ -634,7 +637,7 @@ roc_cpt_dev_init(struct roc_cpt *roc_cpt) } /* Reserve 1 CPT LF for inline inbound */ - nb_lf_avail = PLT_MIN(nb_lf_avail, ROC_CPT_MAX_LFS - 1); + nb_lf_avail = PLT_MIN(nb_lf_avail, (uint16_t)(ROC_CPT_MAX_LFS - 1)); roc_cpt->nb_lf_avail = nb_lf_avail; @@ -770,8 +773,10 @@ void roc_cpt_iq_disable(struct roc_cpt_lf *lf) { union cpt_lf_ctl lf_ctl = {.u = 0x0}; + union cpt_lf_q_grp_ptr grp_ptr; union cpt_lf_inprog lf_inprog; int timeout = 20; + int cnt; /* Disable instructions enqueuing */ plt_write64(lf_ctl.u, lf->rbase + CPT_LF_CTL); @@ -795,6 +800,27 @@ roc_cpt_iq_disable(struct roc_cpt_lf *lf) */ lf_inprog.s.eena = 0x0; plt_write64(lf_inprog.u, lf->rbase + CPT_LF_INPROG); + + /* Wait for instruction queue to become empty */ + cnt = 0; + do { + lf_inprog.u = plt_read64(lf->rbase + CPT_LF_INPROG); + if (lf_inprog.s.grb_partial) + cnt = 0; + else + cnt++; + grp_ptr.u = plt_read64(lf->rbase + CPT_LF_Q_GRP_PTR); + } while ((cnt < 10) && (grp_ptr.s.nq_ptr != grp_ptr.s.dq_ptr)); + + cnt = 0; + do { + lf_inprog.u = plt_read64(lf->rbase + CPT_LF_INPROG); + if ((lf_inprog.s.inflight == 0) && (lf_inprog.s.gwb_cnt < 40) && + ((lf_inprog.s.grb_cnt == 0) || (lf_inprog.s.grb_cnt == 40))) + cnt++; + else + cnt = 0; + } while (cnt < 10); } void @@ -806,11 +832,6 @@ roc_cpt_iq_enable(struct roc_cpt_lf *lf) /* Disable command queue */ roc_cpt_iq_disable(lf); - /* Enable command queue execution */ - lf_inprog.u = plt_read64(lf->rbase + CPT_LF_INPROG); - lf_inprog.s.eena = 1; - plt_write64(lf_inprog.u, lf->rbase + CPT_LF_INPROG); - /* Enable instruction queue enqueuing */ lf_ctl.u = plt_read64(lf->rbase + CPT_LF_CTL); lf_ctl.s.ena = 1; @@ -819,6 +840,11 @@ roc_cpt_iq_enable(struct roc_cpt_lf *lf) lf_ctl.s.fc_hyst_bits = lf->fc_hyst_bits; plt_write64(lf_ctl.u, lf->rbase + CPT_LF_CTL); + /* Enable command queue execution */ + lf_inprog.u = plt_read64(lf->rbase + CPT_LF_INPROG); + lf_inprog.s.eena = 1; + plt_write64(lf_inprog.u, lf->rbase + CPT_LF_INPROG); + cpt_lf_dump(lf); } diff --git a/drivers/common/cnxk/roc_cpt.h b/drivers/common/cnxk/roc_cpt.h index 9b55303..2ac3197 100644 --- a/drivers/common/cnxk/roc_cpt.h +++ b/drivers/common/cnxk/roc_cpt.h @@ -75,6 +75,14 @@ #define ROC_CPT_TUNNEL_IPV4_HDR_LEN 20 #define ROC_CPT_TUNNEL_IPV6_HDR_LEN 40 +#define ROC_CPT_CCM_AAD_DATA 1 +#define ROC_CPT_CCM_MSG_LEN 4 +#define ROC_CPT_CCM_ICV_LEN 16 +#define ROC_CPT_CCM_FLAGS \ + ((ROC_CPT_CCM_AAD_DATA << 6) | \ + (((ROC_CPT_CCM_ICV_LEN - 2) / 2) << 3) | (ROC_CPT_CCM_MSG_LEN - 1)) +#define ROC_CPT_CCM_SALT_LEN 3 + struct roc_cpt_lmtline { uint64_t io_addr; uint64_t *fc_addr; From patchwork Thu Sep 2 02:14:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97736 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F23F1A0C4C; Thu, 2 Sep 2021 04:17:56 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DAC024114E; Thu, 2 Sep 2021 04:17:16 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 81C2441162 for ; Thu, 2 Sep 2021 04:17:15 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQCpj011801 for ; Wed, 1 Sep 2021 19:17:15 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=hMqXPREh75b1XXWD3AOoJTM4ErKoxPV55fG0Yq1KXNg=; b=T1+zkSrWy4fOhOoXrZi5dCieIGiboYF9IlkD9U2/D2521tAuv2Yy0bSebOj8fAsXnLmk dqSQ6seHLQeF9w//HLqc/f7UlHprX/OmJiBiMI0hp5M13P6yjvo2jBGBfRpeSXpyQ/+z bqc1t/zjj7FCr18bml6aW6XtXFyvyO6gBHx/gGMiX/1kJnOcF6dqnFfElCIw01luJzce nYdSzpfYFiTDi4MV3LaaUr0PerEui1nDNdvkEdnNsQRjIsANRyzhL/gPmx5iZFxaj5Ci wooADgPrgE1lNTYKzJPuY6gd72gE0wxPVdIRm3Dt3Z3N4maadeU2iyuV9HO0oYTmUMCa TQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hs7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:14 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:12 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:12 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id BD1753F704D; Wed, 1 Sep 2021 19:17:10 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:48 +0530 Message-ID: <20210902021505.17607-11-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 7HuZtkmTVRH8dEY1NmO7rPACnyJ5TE1H X-Proofpoint-GUID: 7HuZtkmTVRH8dEY1NmO7rPACnyJ5TE1H X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 10/27] common/cnxk: restore nix sqb pool limit before destroy X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Restore SQB aura/pool limit before destroying SQB to be able to drain all the buffers from the aura. Signed-off-by: Nithin Dabilpuram --- drivers/common/cnxk/roc_nix_queue.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/common/cnxk/roc_nix_queue.c b/drivers/common/cnxk/roc_nix_queue.c index de63361..03a3a06 100644 --- a/drivers/common/cnxk/roc_nix_queue.c +++ b/drivers/common/cnxk/roc_nix_queue.c @@ -921,6 +921,11 @@ roc_nix_sq_fini(struct roc_nix_sq *sq) rc |= NIX_ERR_NDC_SYNC; rc |= nix_tm_sq_flush_post(sq); + + /* Restore limit to max SQB count that the pool was created + * for aura drain to succeed. + */ + roc_npa_aura_limit_modify(sq->aura_handle, NIX_MAX_SQB); rc |= roc_npa_pool_destroy(sq->aura_handle); plt_free(sq->fc); plt_free(sq->sqe_mem); From patchwork Thu Sep 2 02:14:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97737 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 610CDA0C4C; Thu, 2 Sep 2021 04:18:02 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1479E41154; Thu, 2 Sep 2021 04:17:20 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 3024B4113A for ; Thu, 2 Sep 2021 04:17:18 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181K5Dsk025890 for ; Wed, 1 Sep 2021 19:17:17 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=uKQrvS7i7sB5KBbDNxdj/7VhwTqa636bI7il4Y895G4=; b=XqWwUNc8SVDQUNzCYQzQVeSz+tRgpjzDnMVnr8rr9oqXtpU5hFWloMYJlfU20/6yOIFS vUJQIq5hKfKzINB9SyWs8DHJSflXMbBE02jL6/EryVRukBz8oW8nDxEPloCZCutgCYsu eP5ygs695U9hepdOseVoSlUrubEM9PHOIJZuprXGUEhVvaBtRq2o1iCQ737mIzEi/cun lavNNCsrj1YvfPUAPuW2w16kvTNPrKQ3EUYPCZvRw5V2gt+6s92PrDm0CUKy2H7z/cun Z66ivewZDP9SETzjbK0wOnMKHoVceyF7zLkymFi+717zegZGCdq0OqHnFCC8mCw+C8sc 2A== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com with ESMTP id 3atg8a91hs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:17 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:15 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:15 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 7CFBF3F704B; Wed, 1 Sep 2021 19:17:13 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , , "Kommula Shiva Shankar" Date: Thu, 2 Sep 2021 07:44:49 +0530 Message-ID: <20210902021505.17607-12-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: eOyDsf_4ZXxQLjfeG6j23zfzgKHsL7Ds X-Proofpoint-ORIG-GUID: eOyDsf_4ZXxQLjfeG6j23zfzgKHsL7Ds X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 11/27] common/cnxk: add cq enable support in nix Tx path X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Kommula Shiva Shankar This patch provides applications to add cq support in Tx path Signed-off-by: Kommula Shiva Shankar --- drivers/common/cnxk/roc_nix.h | 2 ++ drivers/common/cnxk/roc_nix_queue.c | 4 ++++ 2 files changed, 6 insertions(+) diff --git a/drivers/common/cnxk/roc_nix.h b/drivers/common/cnxk/roc_nix.h index ed6e721..95ea4e0 100644 --- a/drivers/common/cnxk/roc_nix.h +++ b/drivers/common/cnxk/roc_nix.h @@ -194,7 +194,9 @@ struct roc_nix_sq { enum roc_nix_sq_max_sqe_sz max_sqe_sz; uint32_t nb_desc; uint16_t qid; + uint16_t cqid; bool sso_ena; + bool cq_ena; /* End of Input parameters */ uint16_t sqes_per_sqb_log2; struct roc_nix *roc_nix; diff --git a/drivers/common/cnxk/roc_nix_queue.c b/drivers/common/cnxk/roc_nix_queue.c index 03a3a06..8fbb13e 100644 --- a/drivers/common/cnxk/roc_nix_queue.c +++ b/drivers/common/cnxk/roc_nix_queue.c @@ -648,6 +648,8 @@ sq_cn9k_init(struct nix *nix, struct roc_nix_sq *sq, uint32_t rr_quantum, aq->sq.sqe_stype = NIX_STYPE_STF; aq->sq.ena = 1; aq->sq.sso_ena = !!sq->sso_ena; + aq->sq.cq_ena = !!sq->cq_ena; + aq->sq.cq = sq->cqid; if (aq->sq.max_sqe_size == NIX_MAXSQESZ_W8) aq->sq.sqe_stype = NIX_STYPE_STP; aq->sq.sqb_aura = roc_npa_aura_handle_to_aura(sq->aura_handle); @@ -746,6 +748,8 @@ sq_init(struct nix *nix, struct roc_nix_sq *sq, uint32_t rr_quantum, aq->sq.sqe_stype = NIX_STYPE_STF; aq->sq.ena = 1; aq->sq.sso_ena = !!sq->sso_ena; + aq->sq.cq_ena = !!sq->cq_ena; + aq->sq.cq = sq->cqid; if (aq->sq.max_sqe_size == NIX_MAXSQESZ_W8) aq->sq.sqe_stype = NIX_STYPE_STP; aq->sq.sqb_aura = roc_npa_aura_handle_to_aura(sq->aura_handle); From patchwork Thu Sep 2 02:14:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97738 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1AB66A0C4C; Thu, 2 Sep 2021 04:18:08 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3CEBC41167; Thu, 2 Sep 2021 04:17:23 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 32C3841166 for ; Thu, 2 Sep 2021 04:17:21 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQCpk011801 for ; Wed, 1 Sep 2021 19:17:20 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=mUk8xEuKp7zuF72p22RvkRI/ZC2GbRCiRcVACY3dq4A=; b=RnwGE8Ch9xKPeTo6Sq4VRd5rmV3I8Gk2zeaeFjWOyeeWzcO1000vHx1K1Y1hl0JoPTAd uQ4AwMRDpB/p9KSkKR8qKclE+56yMT11fqzT6vwwBbUTapAIB96y2bXd9DfskuWvuSo8 g6tvJotnWRlWhgw1CjqAeibLK8yHBwLT2vY6VK5vxOwUB/spo0cn4fefyCus6bHOxjUZ +n7IwL0sceNyw6WlAhm6kS1d51u1Q0n7ddRslKvh1h7Q6XtfynWYHlirdqb0Delspx7W TUmVdxW+h+qO7Tf/mZkUJO6cavLTon2nKrP9CTmPDEcVTysHEiphADoC2G+gFEZfD6c0 Ng== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hsh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:20 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:18 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:18 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 7686F3F7050; Wed, 1 Sep 2021 19:17:16 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:50 +0530 Message-ID: <20210902021505.17607-13-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: sZRE5Kw9gLg6W7YjFEXbwlZY3bMmopFo X-Proofpoint-GUID: sZRE5Kw9gLg6W7YjFEXbwlZY3bMmopFo X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 12/27] common/cnxk: setup aura bp conf based on nix X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Currently only nix0 conf is setup in Aura for backpressure. This patch adds support for nix1 as well. Signed-off-by: Nithin Dabilpuram --- drivers/common/cnxk/roc_nix_fc.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/drivers/common/cnxk/roc_nix_fc.c b/drivers/common/cnxk/roc_nix_fc.c index f17eba4..7eac7d0 100644 --- a/drivers/common/cnxk/roc_nix_fc.c +++ b/drivers/common/cnxk/roc_nix_fc.c @@ -284,8 +284,18 @@ rox_nix_fc_npa_bp_cfg(struct roc_nix *roc_nix, uint64_t pool_id, uint8_t ena, limit = rsp->aura.limit; /* BP is already enabled. */ if (rsp->aura.bp_ena) { + uint16_t bpid; + bool nix1; + + nix1 = !!(rsp->aura.bp_ena & 0x2); + if (nix1) + bpid = rsp->aura.nix1_bpid; + else + bpid = rsp->aura.nix0_bpid; + /* If BP ids don't match disable BP. */ - if ((rsp->aura.nix0_bpid != nix->bpid[0]) && !force) { + if (((nix1 != nix->is_nix1) || (bpid != nix->bpid[0])) && + !force) { req = mbox_alloc_msg_npa_aq_enq(mbox); if (req == NULL) return; @@ -315,14 +325,19 @@ rox_nix_fc_npa_bp_cfg(struct roc_nix *roc_nix, uint64_t pool_id, uint8_t ena, req->op = NPA_AQ_INSTOP_WRITE; if (ena) { - req->aura.nix0_bpid = nix->bpid[0]; - req->aura_mask.nix0_bpid = ~(req->aura_mask.nix0_bpid); + if (nix->is_nix1) { + req->aura.nix1_bpid = nix->bpid[0]; + req->aura_mask.nix1_bpid = ~(req->aura_mask.nix1_bpid); + } else { + req->aura.nix0_bpid = nix->bpid[0]; + req->aura_mask.nix0_bpid = ~(req->aura_mask.nix0_bpid); + } req->aura.bp = NIX_RQ_AURA_THRESH( limit > 128 ? 256 : limit); /* 95% of size*/ req->aura_mask.bp = ~(req->aura_mask.bp); } - req->aura.bp_ena = !!ena; + req->aura.bp_ena = (!!ena << nix->is_nix1); req->aura_mask.bp_ena = ~(req->aura_mask.bp_ena); mbox_process(mbox); From patchwork Thu Sep 2 02:14:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97739 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E7050A0C4C; Thu, 2 Sep 2021 04:18:13 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 699974116B; Thu, 2 Sep 2021 04:17:26 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 5175441159 for ; Thu, 2 Sep 2021 04:17:24 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQ3fJ011587 for ; Wed, 1 Sep 2021 19:17:23 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=RmfNko7/8XvQzbAZi7sEgjTJSHE/vfW8o7k2YT8GwIM=; b=MOaTuOPh+dlr/pHeNDobF3OuUNcKXZ2RJRscydqQLd7T6hRBVE881uGCzRGLufOyZtrF fI7YKFy129FT3J/xOLa0O8RQ7H+OgshH8g5hpu+ILq7E5Erz7Jq5vQs4p4P0KDOfzqJ/ qT1894yDr1hM/7U7tADLKHErD/tVOHR37TtjplH5dWYpVQ7W/79/tFxHhlCphwbB3MGD ue55CtPoVjPqQBwFfLpyzOEhKmctAC6nI4MIUcUfaWXglHiblVjVGtAGB1fW4ad6yoPe ZIKNaCxva1BT0YcsYZo/Bfo+4tpmqWSpkeF5mBYA4ZRAkOIPFU3xCFzzvHwLO8MFRO7B rA== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hsw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:23 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:21 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:21 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 3E2363F7040; Wed, 1 Sep 2021 19:17:19 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:51 +0530 Message-ID: <20210902021505.17607-14-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: qpGE7m6mKETJWiFUqXtzWdp7JXvIFPWo X-Proofpoint-GUID: qpGE7m6mKETJWiFUqXtzWdp7JXvIFPWo X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 13/27] common/cnxk: add anti-replay check implementation for cn9k X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Srujana Challa Adds anti replay helper for cn9k platform. Signed-off-by: Srujana Challa --- drivers/common/cnxk/cnxk_security_ar.h | 184 +++++++++++++++++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 drivers/common/cnxk/cnxk_security_ar.h diff --git a/drivers/common/cnxk/cnxk_security_ar.h b/drivers/common/cnxk/cnxk_security_ar.h new file mode 100644 index 0000000..6bc517c --- /dev/null +++ b/drivers/common/cnxk/cnxk_security_ar.h @@ -0,0 +1,184 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#ifndef __CNXK_SECURITY_AR_H__ +#define __CNXK_SECURITY_AR_H__ + +#include + +#include "cnxk_security.h" + +#define CNXK_ON_AR_WIN_SIZE_MAX 1024 + +/* u64 array size to fit anti replay window bits */ +#define AR_WIN_ARR_SZ \ + (PLT_ALIGN_CEIL(CNXK_ON_AR_WIN_SIZE_MAX, BITS_PER_LONG_LONG) / \ + BITS_PER_LONG_LONG) + +#define WORD_SHIFT 6 +#define WORD_SIZE (1 << WORD_SHIFT) +#define WORD_MASK (WORD_SIZE - 1) + +#define IPSEC_ANTI_REPLAY_FAILED (-1) + +struct cnxk_on_ipsec_ar { + rte_spinlock_t lock; + uint32_t winb; + uint32_t wint; + uint64_t base; /**< base of the anti-replay window */ + uint64_t window[AR_WIN_ARR_SZ]; /**< anti-replay window */ +}; + +static inline int +cnxk_on_anti_replay_check(uint64_t seq, struct cnxk_on_ipsec_ar *ar, + uint32_t winsz) +{ + uint64_t ex_winsz = winsz + WORD_SIZE; + uint64_t *window = &ar->window[0]; + uint64_t seqword, shiftwords; + uint64_t base = ar->base; + uint32_t winb = ar->winb; + uint32_t wint = ar->wint; + uint64_t winwords; + uint64_t bit_pos; + uint64_t shift; + uint64_t *wptr; + uint64_t tmp; + + winwords = ex_winsz >> WORD_SHIFT; + if (winsz > 64) + goto slow_shift; + /* Check if the seq is the biggest one yet */ + if (likely(seq > base)) { + shift = seq - base; + if (shift < winsz) { /* In window */ + /* + * If more than 64-bit anti-replay window, + * use slow shift routine + */ + wptr = window + (shift >> WORD_SHIFT); + *wptr <<= shift; + *wptr |= 1ull; + } else { + /* No special handling of window size > 64 */ + wptr = window + ((winsz - 1) >> WORD_SHIFT); + /* + * Zero out the whole window (especially for + * bigger than 64b window) till the last 64b word + * as the incoming sequence number minus + * base sequence is more than the window size. + */ + while (window != wptr) + *window++ = 0ull; + /* + * Set the last bit (of the window) to 1 + * as that corresponds to the base sequence number. + * Now any incoming sequence number which is + * (base - window size - 1) will pass anti-replay check + */ + *wptr = 1ull; + } + /* + * Set the base to incoming sequence number as + * that is the biggest sequence number seen yet + */ + ar->base = seq; + return 0; + } + + bit_pos = base - seq; + + /* If seq falls behind the window, return failure */ + if (bit_pos >= winsz) + return IPSEC_ANTI_REPLAY_FAILED; + + /* seq is within anti-replay window */ + wptr = window + ((winsz - bit_pos - 1) >> WORD_SHIFT); + bit_pos &= WORD_MASK; + + /* Check if this is a replayed packet */ + if (*wptr & ((1ull) << bit_pos)) + return IPSEC_ANTI_REPLAY_FAILED; + + /* mark as seen */ + *wptr |= ((1ull) << bit_pos); + return 0; + +slow_shift: + if (likely(seq > base)) { + uint32_t i; + + shift = seq - base; + if (unlikely(shift >= winsz)) { + /* + * shift is bigger than the window, + * so just zero out everything + */ + for (i = 0; i < winwords; i++) + window[i] = 0; +winupdate: + /* Find out the word */ + seqword = ((seq - 1) % ex_winsz) >> WORD_SHIFT; + + /* Find out the bit in the word */ + bit_pos = (seq - 1) & WORD_MASK; + + /* + * Set the bit corresponding to sequence number + * in window to mark it as received + */ + window[seqword] |= (1ull << (63 - bit_pos)); + + /* wint and winb range from 1 to ex_winsz */ + ar->wint = ((wint + shift - 1) % ex_winsz) + 1; + ar->winb = ((winb + shift - 1) % ex_winsz) + 1; + + ar->base = seq; + return 0; + } + + /* + * New sequence number is bigger than the base but + * it's not bigger than base + window size + */ + + shiftwords = ((wint + shift - 1) >> WORD_SHIFT) - + ((wint - 1) >> WORD_SHIFT); + if (unlikely(shiftwords)) { + tmp = (wint + WORD_SIZE - 1) / WORD_SIZE; + for (i = 0; i < shiftwords; i++) { + tmp %= winwords; + window[tmp++] = 0; + } + } + + goto winupdate; + } + + /* Sequence number is before the window */ + if (unlikely((seq + winsz) <= base)) + return IPSEC_ANTI_REPLAY_FAILED; + + /* Sequence number is within the window */ + + /* Find out the word */ + seqword = ((seq - 1) % ex_winsz) >> WORD_SHIFT; + + /* Find out the bit in the word */ + bit_pos = (seq - 1) & WORD_MASK; + + /* Check if this is a replayed packet */ + if (window[seqword] & (1ull << (63 - bit_pos))) + return IPSEC_ANTI_REPLAY_FAILED; + + /* + * Set the bit corresponding to sequence number + * in window to mark it as received + */ + window[seqword] |= (1ull << (63 - bit_pos)); + + return 0; +} + +#endif /* __CNXK_SECURITY_AR_H__ */ From patchwork Thu Sep 2 02:14:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97740 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4BFE3A0C4C; Thu, 2 Sep 2021 04:18:21 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DFBCA41176; Thu, 2 Sep 2021 04:17:27 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 217E741144 for ; Thu, 2 Sep 2021 04:17:27 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQCpm011801 for ; Wed, 1 Sep 2021 19:17:26 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=bGE2mtAAGFuYtVYQ+swggPOmR0vd/9apncuXPZ7/bKc=; b=HMejgsT0WnLr+++WRmfAOkD948snczHnUwM2nTcobVu1BPF7whmpJzg3R+XJFG572sGu aX+XcqmKt0naa4pzRcTRuv2PwW2IPRHMUDCPFLrtPP8YR6748Tc2QK/8VbKfPWQ5Q2uz R5/P3S8VLKBlGL+qNxn6X3d445v5TBwDjH2JLJjzAxwYIHMmxUSh2jrUuK3Bvvey9Kp+ uLPqUKTSgAeJBCJRu8Su9Z2iqFLbnnM4bLQoCWUYSi+JBIfKFsWQ5BVesn83q4UJHHzZ 5SvI/kwkhFKem1bu6MbAl0U65WmxG7Qss1JQY7QKbtGS88+7ZZeGSs0UUtdRqcgL24lQ rA== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9ht5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:26 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:24 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:24 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id F331B3F704D; Wed, 1 Sep 2021 19:17:21 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , , Satheesh Paul Date: Thu, 2 Sep 2021 07:44:52 +0530 Message-ID: <20210902021505.17607-15-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 5R18UrpMCxQNS-P7t8KLXab1CnFniXEs X-Proofpoint-GUID: 5R18UrpMCxQNS-P7t8KLXab1CnFniXEs X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 14/27] common/cnxk: add inline IPsec support in rte flow X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Satheesh Paul Add support to configure flow rules with inline IPsec action. Signed-off-by: Satheesh Paul --- drivers/common/cnxk/roc_nix_inl.h | 4 ++++ drivers/common/cnxk/roc_nix_inl_dev.c | 3 +++ drivers/common/cnxk/roc_nix_inl_priv.h | 3 +++ drivers/common/cnxk/roc_npc_mcam.c | 28 ++++++++++++++++++++++++++-- 4 files changed, 36 insertions(+), 2 deletions(-) diff --git a/drivers/common/cnxk/roc_nix_inl.h b/drivers/common/cnxk/roc_nix_inl.h index efc5a19..1f7ec4f 100644 --- a/drivers/common/cnxk/roc_nix_inl.h +++ b/drivers/common/cnxk/roc_nix_inl.h @@ -107,6 +107,10 @@ struct roc_nix_inl_dev { struct plt_pci_device *pci_dev; uint16_t ipsec_in_max_spi; bool selftest; + bool is_multi_channel; + uint16_t channel; + uint16_t chan_mask; + /* End of input parameters */ #define ROC_NIX_INL_MEM_SZ (1024) diff --git a/drivers/common/cnxk/roc_nix_inl_dev.c b/drivers/common/cnxk/roc_nix_inl_dev.c index 214f183..2dc2188 100644 --- a/drivers/common/cnxk/roc_nix_inl_dev.c +++ b/drivers/common/cnxk/roc_nix_inl_dev.c @@ -461,6 +461,9 @@ roc_nix_inl_dev_init(struct roc_nix_inl_dev *roc_inl_dev) inl_dev->pci_dev = pci_dev; inl_dev->ipsec_in_max_spi = roc_inl_dev->ipsec_in_max_spi; inl_dev->selftest = roc_inl_dev->selftest; + inl_dev->is_multi_channel = roc_inl_dev->is_multi_channel; + inl_dev->channel = roc_inl_dev->channel; + inl_dev->chan_mask = roc_inl_dev->chan_mask; /* Initialize base device */ rc = dev_init(&inl_dev->dev, pci_dev); diff --git a/drivers/common/cnxk/roc_nix_inl_priv.h b/drivers/common/cnxk/roc_nix_inl_priv.h index ab38062..a302118 100644 --- a/drivers/common/cnxk/roc_nix_inl_priv.h +++ b/drivers/common/cnxk/roc_nix_inl_priv.h @@ -45,6 +45,9 @@ struct nix_inl_dev { /* Device arguments */ uint8_t selftest; + uint16_t channel; + uint16_t chan_mask; + bool is_multi_channel; uint16_t ipsec_in_max_spi; }; diff --git a/drivers/common/cnxk/roc_npc_mcam.c b/drivers/common/cnxk/roc_npc_mcam.c index 8ccaaad..4985d22 100644 --- a/drivers/common/cnxk/roc_npc_mcam.c +++ b/drivers/common/cnxk/roc_npc_mcam.c @@ -503,8 +503,11 @@ npc_mcam_alloc_and_write(struct npc *npc, struct roc_npc_flow *flow, { int use_ctr = (flow->ctr_id == NPC_COUNTER_NONE ? 0 : 1); struct npc_mcam_write_entry_req *req; + struct nix_inl_dev *inl_dev = NULL; struct mbox *mbox = npc->mbox; struct mbox_msghdr *rsp; + struct idev_cfg *idev; + uint16_t pf_func = 0; uint16_t ctr = ~(0); int rc, idx; int entry; @@ -553,9 +556,30 @@ npc_mcam_alloc_and_write(struct npc *npc, struct roc_npc_flow *flow, req->entry_data.kw_mask[idx] = flow->mcam_mask[idx]; } + idev = idev_get_cfg(); + if (idev) + inl_dev = idev->nix_inl_dev; + if (flow->nix_intf == NIX_INTF_RX) { - req->entry_data.kw[0] |= (uint64_t)npc->channel; - req->entry_data.kw_mask[0] |= (BIT_ULL(12) - 1); + if (inl_dev && inl_dev->is_multi_channel && + (flow->npc_action & NIX_RX_ACTIONOP_UCAST_IPSEC)) { + req->entry_data.kw[0] |= (uint64_t)inl_dev->channel; + req->entry_data.kw_mask[0] |= + (uint64_t)inl_dev->chan_mask; + pf_func = nix_inl_dev_pffunc_get(); + req->entry_data.action &= ~(GENMASK(19, 4)); + req->entry_data.action |= (uint64_t)pf_func << 4; + + flow->npc_action &= ~(GENMASK(19, 4)); + flow->npc_action |= (uint64_t)pf_func << 4; + flow->mcam_data[0] |= (uint64_t)inl_dev->channel; + flow->mcam_mask[0] |= (uint64_t)inl_dev->chan_mask; + } else { + req->entry_data.kw[0] |= (uint64_t)npc->channel; + req->entry_data.kw_mask[0] |= (BIT_ULL(12) - 1); + flow->mcam_data[0] |= (uint64_t)npc->channel; + flow->mcam_mask[0] |= (BIT_ULL(12) - 1); + } } else { uint16_t pf_func = (flow->npc_action >> 4) & 0xffff; From patchwork Thu Sep 2 02:14:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97741 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E020BA0C4C; Thu, 2 Sep 2021 04:18:26 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2125241134; Thu, 2 Sep 2021 04:17:34 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id E852E41144 for ; Thu, 2 Sep 2021 04:17:32 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181K5Dsm025890; Wed, 1 Sep 2021 19:17:29 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=0P24A4vAHMg79ZQzx8TKPqmKM4H77Yc5saFJiFYAVaE=; b=kEjbWBBy9yVxuthOxEKBd4zOmT23T2Jq6GWMHVF3nUr3/GfzMq/nFGOuKHxeUbDzp+gO 3n80wjQE3K2SF7lLUZByeq5Engw1OHGg5pCm2JkvyPiHMcuiUIezPxc45KZthpGmYmsf 365ImBPsTmqlqzS5m754wpHZ1e6swKu6DlId0gcbg2Y/hScm3s3Hn4ljTMwNIBMj5sIc HqG2GvsqFZ9uGl2CgXTBBH1mH5MaykQ9DSsf1EL1NY7aopkIli3Pzl8gmrpC27N80Cbz Dc30nkw+eeOclVvQbdQ7HNyT+tH431LtGuU09DNPl+WIFKHLYBzztTFm6YLZ/rJciggD tA== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com with ESMTP id 3atg8a91jx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 19:17:29 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:27 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:27 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 0935B3F704B; Wed, 1 Sep 2021 19:17:24 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao , Ray Kinsella , Anatoly Burakov CC: , , Date: Thu, 2 Sep 2021 07:44:53 +0530 Message-ID: <20210902021505.17607-16-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: tNrvjEINmJQ4F0hXQVtKUL_dmVntHl93 X-Proofpoint-ORIG-GUID: tNrvjEINmJQ4F0hXQVtKUL_dmVntHl93 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 15/27] net/cnxk: add inline security support for cn9k X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add support for inline inbound and outbound IPSec for SA create, destroy and other NIX / CPT LF configurations. Signed-off-by: Nithin Dabilpuram --- drivers/net/cnxk/cn9k_ethdev.c | 23 +++ drivers/net/cnxk/cn9k_ethdev.h | 61 +++++++ drivers/net/cnxk/cn9k_ethdev_sec.c | 313 +++++++++++++++++++++++++++++++++ drivers/net/cnxk/cn9k_rx.h | 1 + drivers/net/cnxk/cn9k_tx.h | 1 + drivers/net/cnxk/cnxk_ethdev.c | 214 +++++++++++++++++++++- drivers/net/cnxk/cnxk_ethdev.h | 121 ++++++++++++- drivers/net/cnxk/cnxk_ethdev_devargs.c | 88 ++++++++- drivers/net/cnxk/cnxk_ethdev_sec.c | 278 +++++++++++++++++++++++++++++ drivers/net/cnxk/cnxk_lookup.c | 50 +++++- drivers/net/cnxk/meson.build | 2 + drivers/net/cnxk/version.map | 5 + 12 files changed, 1146 insertions(+), 11 deletions(-) create mode 100644 drivers/net/cnxk/cn9k_ethdev_sec.c create mode 100644 drivers/net/cnxk/cnxk_ethdev_sec.c diff --git a/drivers/net/cnxk/cn9k_ethdev.c b/drivers/net/cnxk/cn9k_ethdev.c index 115e678..08c86f9 100644 --- a/drivers/net/cnxk/cn9k_ethdev.c +++ b/drivers/net/cnxk/cn9k_ethdev.c @@ -36,6 +36,9 @@ nix_rx_offload_flags(struct rte_eth_dev *eth_dev) if (!dev->ptype_disable) flags |= NIX_RX_OFFLOAD_PTYPE_F; + if (dev->rx_offloads & DEV_RX_OFFLOAD_SECURITY) + flags |= NIX_RX_OFFLOAD_SECURITY_F; + return flags; } @@ -101,6 +104,9 @@ nix_tx_offload_flags(struct rte_eth_dev *eth_dev) if ((dev->rx_offloads & DEV_RX_OFFLOAD_TIMESTAMP)) flags |= NIX_TX_OFFLOAD_TSTAMP_F; + if (dev->tx_offloads & DEV_TX_OFFLOAD_SECURITY) + flags |= NIX_TX_OFFLOAD_SECURITY_F; + return flags; } @@ -179,8 +185,10 @@ cn9k_nix_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, const struct rte_eth_txconf *tx_conf) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + struct roc_cpt_lf *inl_lf; struct cn9k_eth_txq *txq; struct roc_nix_sq *sq; + uint16_t crypto_qid; int rc; RTE_SET_USED(socket); @@ -200,6 +208,19 @@ cn9k_nix_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, txq->nb_sqb_bufs_adj = sq->nb_sqb_bufs_adj; txq->sqes_per_sqb_log2 = sq->sqes_per_sqb_log2; + /* Fetch CPT LF info for outbound if present */ + if (dev->outb.lf_base) { + crypto_qid = qid % dev->outb.nb_crypto_qs; + inl_lf = dev->outb.lf_base + crypto_qid; + + txq->cpt_io_addr = inl_lf->io_addr; + txq->cpt_fc = inl_lf->fc_addr; + txq->cpt_desc = inl_lf->nb_desc * 0.7; + txq->sa_base = (uint64_t)dev->outb.sa_base; + txq->sa_base |= eth_dev->data->port_id; + PLT_STATIC_ASSERT(BIT_ULL(16) == ROC_NIX_INL_SA_BASE_ALIGN); + } + nix_form_default_desc(dev, txq, qid); txq->lso_tun_fmt = dev->lso_tun_fmt; return 0; @@ -508,6 +529,8 @@ cn9k_nix_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev) nix_eth_dev_ops_override(); npc_flow_ops_override(); + cn9k_eth_sec_ops_override(); + /* Common probe */ rc = cnxk_nix_probe(pci_drv, pci_dev); if (rc) diff --git a/drivers/net/cnxk/cn9k_ethdev.h b/drivers/net/cnxk/cn9k_ethdev.h index 3d4a206..f8818b8 100644 --- a/drivers/net/cnxk/cn9k_ethdev.h +++ b/drivers/net/cnxk/cn9k_ethdev.h @@ -5,6 +5,7 @@ #define __CN9K_ETHDEV_H__ #include +#include struct cn9k_eth_txq { uint64_t cmd[8]; @@ -15,6 +16,10 @@ struct cn9k_eth_txq { uint64_t lso_tun_fmt; uint16_t sqes_per_sqb_log2; int16_t nb_sqb_bufs_adj; + rte_iova_t cpt_io_addr; + uint64_t sa_base; + uint64_t *cpt_fc; + uint16_t cpt_desc; } __plt_cache_aligned; struct cn9k_eth_rxq { @@ -32,8 +37,64 @@ struct cn9k_eth_rxq { struct cnxk_timesync_info *tstamp; } __plt_cache_aligned; +/* Private data in sw rsvd area of struct roc_onf_ipsec_inb_sa */ +struct cn9k_inb_priv_data { + void *userdata; + struct cnxk_eth_sec_sess *eth_sec; +}; + +/* Private data in sw rsvd area of struct roc_onf_ipsec_outb_sa */ +struct cn9k_outb_priv_data { + union { + uint64_t esn; + struct { + uint32_t seq; + uint32_t esn_hi; + }; + }; + + /* Rlen computation data */ + struct cnxk_ipsec_outb_rlens rlens; + + /* IP identifier */ + uint16_t ip_id; + + /* SA index */ + uint32_t sa_idx; + + /* Flags */ + uint16_t copy_salt : 1; + + /* Salt */ + uint32_t nonce; + + /* User data pointer */ + void *userdata; + + /* Back pointer to eth sec session */ + struct cnxk_eth_sec_sess *eth_sec; +}; + +struct cn9k_sec_sess_priv { + union { + struct { + uint32_t sa_idx; + uint8_t inb_sa : 1; + uint8_t rsvd1 : 2; + uint8_t roundup_byte : 5; + uint8_t roundup_len; + uint16_t partial_len; + }; + + uint64_t u64; + }; +} __rte_packed; + /* Rx and Tx routines */ void cn9k_eth_set_rx_function(struct rte_eth_dev *eth_dev); void cn9k_eth_set_tx_function(struct rte_eth_dev *eth_dev); +/* Security context setup */ +void cn9k_eth_sec_ops_override(void); + #endif /* __CN9K_ETHDEV_H__ */ diff --git a/drivers/net/cnxk/cn9k_ethdev_sec.c b/drivers/net/cnxk/cn9k_ethdev_sec.c new file mode 100644 index 0000000..3ec7497 --- /dev/null +++ b/drivers/net/cnxk/cn9k_ethdev_sec.c @@ -0,0 +1,313 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include +#include +#include + +#include +#include + +static struct rte_cryptodev_capabilities cn9k_eth_sec_crypto_caps[] = { + { /* AES GCM */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AEAD, + {.aead = { + .algo = RTE_CRYPTO_AEAD_AES_GCM, + .block_size = 16, + .key_size = { + .min = 16, + .max = 32, + .increment = 8 + }, + .digest_size = { + .min = 16, + .max = 16, + .increment = 0 + }, + .aad_size = { + .min = 8, + .max = 12, + .increment = 4 + }, + .iv_size = { + .min = 12, + .max = 12, + .increment = 0 + } + }, } + }, } + }, + RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() +}; + +static const struct rte_security_capability cn9k_eth_sec_capabilities[] = { + { /* IPsec Inline Protocol ESP Tunnel Ingress */ + .action = RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL, + .protocol = RTE_SECURITY_PROTOCOL_IPSEC, + .ipsec = { + .proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP, + .mode = RTE_SECURITY_IPSEC_SA_MODE_TUNNEL, + .direction = RTE_SECURITY_IPSEC_SA_DIR_INGRESS, + .options = { 0 } + }, + .crypto_capabilities = cn9k_eth_sec_crypto_caps, + .ol_flags = RTE_SECURITY_TX_OLOAD_NEED_MDATA + }, + { /* IPsec Inline Protocol ESP Tunnel Egress */ + .action = RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL, + .protocol = RTE_SECURITY_PROTOCOL_IPSEC, + .ipsec = { + .proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP, + .mode = RTE_SECURITY_IPSEC_SA_MODE_TUNNEL, + .direction = RTE_SECURITY_IPSEC_SA_DIR_EGRESS, + .options = { 0 } + }, + .crypto_capabilities = cn9k_eth_sec_crypto_caps, + .ol_flags = RTE_SECURITY_TX_OLOAD_NEED_MDATA + }, + { + .action = RTE_SECURITY_ACTION_TYPE_NONE + } +}; + +static int +cn9k_eth_sec_session_create(void *device, + struct rte_security_session_conf *conf, + struct rte_security_session *sess, + struct rte_mempool *mempool) +{ + struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)device; + struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + struct rte_security_ipsec_xform *ipsec; + struct cn9k_sec_sess_priv sess_priv; + struct rte_crypto_sym_xform *crypto; + struct cnxk_eth_sec_sess *eth_sec; + bool inbound; + int rc = 0; + + if (conf->action_type != RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL) + return -ENOTSUP; + + if (conf->protocol != RTE_SECURITY_PROTOCOL_IPSEC) + return -ENOTSUP; + + if (rte_security_dynfield_register() < 0) + return -ENOTSUP; + + ipsec = &conf->ipsec; + crypto = conf->crypto_xform; + inbound = !!(ipsec->direction == RTE_SECURITY_IPSEC_SA_DIR_INGRESS); + + /* Search if a session already exists */ + if (cnxk_eth_sec_sess_get_by_spi(dev, ipsec->spi, inbound)) { + plt_err("%s SA with SPI %u already in use", + inbound ? "Inbound" : "Outbound", ipsec->spi); + return -EEXIST; + } + + if (rte_mempool_get(mempool, (void **)ð_sec)) { + plt_err("Could not allocate security session private data"); + return -ENOMEM; + } + + memset(eth_sec, 0, sizeof(struct cnxk_eth_sec_sess)); + sess_priv.u64 = 0; + + if (inbound) { + struct cn9k_inb_priv_data *inb_priv; + struct roc_onf_ipsec_inb_sa *inb_sa; + + PLT_STATIC_ASSERT(sizeof(struct cn9k_inb_priv_data) < + ROC_NIX_INL_ONF_IPSEC_INB_SW_RSVD); + + /* Get Inbound SA from NIX_RX_IPSEC_SA_BASE. Assume no inline + * device always for CN9K. + */ + inb_sa = (struct roc_onf_ipsec_inb_sa *) + roc_nix_inl_inb_sa_get(&dev->nix, false, ipsec->spi); + if (!inb_sa) { + plt_err("Failed to create ingress sa"); + rc = -EFAULT; + goto mempool_put; + } + + /* Check if SA is already in use */ + if (inb_sa->ctl.valid) { + plt_err("Inbound SA with SPI %u already in use", + ipsec->spi); + rc = -EBUSY; + goto mempool_put; + } + + memset(inb_sa, 0, sizeof(struct roc_onf_ipsec_inb_sa)); + + /* Fill inbound sa params */ + rc = cnxk_onf_ipsec_inb_sa_fill(inb_sa, ipsec, crypto); + if (rc) { + plt_err("Failed to init inbound sa, rc=%d", rc); + goto mempool_put; + } + + inb_priv = roc_nix_inl_onf_ipsec_inb_sa_sw_rsvd(inb_sa); + /* Back pointer to get eth_sec */ + inb_priv->eth_sec = eth_sec; + + /* Save userdata in inb private area */ + inb_priv->userdata = conf->userdata; + + sess_priv.inb_sa = 1; + sess_priv.sa_idx = ipsec->spi; + + /* Pointer from eth_sec -> inb_sa */ + eth_sec->sa = inb_sa; + eth_sec->sess = sess; + eth_sec->sa_idx = ipsec->spi; + eth_sec->spi = ipsec->spi; + eth_sec->inb = true; + + TAILQ_INSERT_TAIL(&dev->inb.list, eth_sec, entry); + dev->inb.nb_sess++; + } else { + struct cn9k_outb_priv_data *outb_priv; + struct roc_onf_ipsec_outb_sa *outb_sa; + uintptr_t sa_base = dev->outb.sa_base; + struct cnxk_ipsec_outb_rlens *rlens; + uint32_t sa_idx; + + PLT_STATIC_ASSERT(sizeof(struct cn9k_outb_priv_data) < + ROC_NIX_INL_ONF_IPSEC_OUTB_SW_RSVD); + + /* Alloc an sa index */ + rc = cnxk_eth_outb_sa_idx_get(dev, &sa_idx); + if (rc) + goto mempool_put; + + outb_sa = roc_nix_inl_onf_ipsec_outb_sa(sa_base, sa_idx); + outb_priv = roc_nix_inl_onf_ipsec_outb_sa_sw_rsvd(outb_sa); + rlens = &outb_priv->rlens; + + memset(outb_sa, 0, sizeof(struct roc_onf_ipsec_outb_sa)); + + /* Fill outbound sa params */ + rc = cnxk_onf_ipsec_outb_sa_fill(outb_sa, ipsec, crypto); + if (rc) { + plt_err("Failed to init outbound sa, rc=%d", rc); + rc |= cnxk_eth_outb_sa_idx_put(dev, sa_idx); + goto mempool_put; + } + + /* Save userdata */ + outb_priv->userdata = conf->userdata; + outb_priv->sa_idx = sa_idx; + outb_priv->eth_sec = eth_sec; + /* Start sequence number with 1 */ + outb_priv->seq = 1; + + memcpy(&outb_priv->nonce, outb_sa->nonce, 4); + if (outb_sa->ctl.enc_type == ROC_IE_ON_SA_ENC_AES_GCM) + outb_priv->copy_salt = 1; + + /* Save rlen info */ + cnxk_ipsec_outb_rlens_get(rlens, ipsec, crypto); + + sess_priv.sa_idx = outb_priv->sa_idx; + sess_priv.roundup_byte = rlens->roundup_byte; + sess_priv.roundup_len = rlens->roundup_len; + sess_priv.partial_len = rlens->partial_len; + + /* Pointer from eth_sec -> outb_sa */ + eth_sec->sa = outb_sa; + eth_sec->sess = sess; + eth_sec->sa_idx = sa_idx; + eth_sec->spi = ipsec->spi; + + TAILQ_INSERT_TAIL(&dev->outb.list, eth_sec, entry); + dev->outb.nb_sess++; + } + + /* Sync SA content */ + plt_atomic_thread_fence(__ATOMIC_ACQ_REL); + + plt_nix_dbg("Created %s session with spi=%u, sa_idx=%u", + inbound ? "inbound" : "outbound", eth_sec->spi, + eth_sec->sa_idx); + /* + * Update fast path info in priv area. + */ + set_sec_session_private_data(sess, (void *)sess_priv.u64); + + return 0; +mempool_put: + rte_mempool_put(mempool, eth_sec); + return rc; +} + +static int +cn9k_eth_sec_session_destroy(void *device, struct rte_security_session *sess) +{ + struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)device; + struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + struct roc_onf_ipsec_outb_sa *outb_sa; + struct roc_onf_ipsec_inb_sa *inb_sa; + struct cnxk_eth_sec_sess *eth_sec; + struct rte_mempool *mp; + + eth_sec = cnxk_eth_sec_sess_get_by_sess(dev, sess); + if (!eth_sec) + return -ENOENT; + + if (eth_sec->inb) { + inb_sa = eth_sec->sa; + /* Disable SA */ + inb_sa->ctl.valid = 0; + + TAILQ_REMOVE(&dev->inb.list, eth_sec, entry); + dev->inb.nb_sess--; + } else { + outb_sa = eth_sec->sa; + /* Disable SA */ + outb_sa->ctl.valid = 0; + + /* Release Outbound SA index */ + cnxk_eth_outb_sa_idx_put(dev, eth_sec->sa_idx); + TAILQ_REMOVE(&dev->outb.list, eth_sec, entry); + dev->outb.nb_sess--; + } + + /* Sync SA content */ + plt_atomic_thread_fence(__ATOMIC_ACQ_REL); + + plt_nix_dbg("Destroyed %s session with spi=%u, sa_idx=%u", + eth_sec->inb ? "inbound" : "outbound", eth_sec->spi, + eth_sec->sa_idx); + + /* Put eth_sec object back to pool */ + mp = rte_mempool_from_obj(eth_sec); + set_sec_session_private_data(sess, NULL); + rte_mempool_put(mp, eth_sec); + return 0; +} + +static const struct rte_security_capability * +cn9k_eth_sec_capabilities_get(void *device __rte_unused) +{ + return cn9k_eth_sec_capabilities; +} + +void +cn9k_eth_sec_ops_override(void) +{ + static int init_once; + + if (init_once) + return; + init_once = 1; + + /* Update platform specific ops */ + cnxk_eth_sec_ops.session_create = cn9k_eth_sec_session_create; + cnxk_eth_sec_ops.session_destroy = cn9k_eth_sec_session_destroy; + cnxk_eth_sec_ops.capabilities_get = cn9k_eth_sec_capabilities_get; +} diff --git a/drivers/net/cnxk/cn9k_rx.h b/drivers/net/cnxk/cn9k_rx.h index a3bf4e0..59545af 100644 --- a/drivers/net/cnxk/cn9k_rx.h +++ b/drivers/net/cnxk/cn9k_rx.h @@ -17,6 +17,7 @@ #define NIX_RX_OFFLOAD_MARK_UPDATE_F BIT(3) #define NIX_RX_OFFLOAD_TSTAMP_F BIT(4) #define NIX_RX_OFFLOAD_VLAN_STRIP_F BIT(5) +#define NIX_RX_OFFLOAD_SECURITY_F BIT(6) /* Flags to control cqe_to_mbuf conversion function. * Defining it from backwards to denote its been diff --git a/drivers/net/cnxk/cn9k_tx.h b/drivers/net/cnxk/cn9k_tx.h index ed65cd3..a27ff76 100644 --- a/drivers/net/cnxk/cn9k_tx.h +++ b/drivers/net/cnxk/cn9k_tx.h @@ -13,6 +13,7 @@ #define NIX_TX_OFFLOAD_MBUF_NOFF_F BIT(3) #define NIX_TX_OFFLOAD_TSO_F BIT(4) #define NIX_TX_OFFLOAD_TSTAMP_F BIT(5) +#define NIX_TX_OFFLOAD_SECURITY_F BIT(6) /* Flags to control xmit_prepare function. * Defining it from backwards to denote its been diff --git a/drivers/net/cnxk/cnxk_ethdev.c b/drivers/net/cnxk/cnxk_ethdev.c index 0e3652e..60a4df5 100644 --- a/drivers/net/cnxk/cnxk_ethdev.c +++ b/drivers/net/cnxk/cnxk_ethdev.c @@ -38,6 +38,159 @@ nix_get_speed_capa(struct cnxk_eth_dev *dev) return speed_capa; } +int +cnxk_nix_inb_mode_set(struct cnxk_eth_dev *dev, bool use_inl_dev) +{ + struct roc_nix *nix = &dev->nix; + + if (dev->inb.inl_dev == use_inl_dev) + return 0; + + plt_nix_dbg("Security sessions(%u) still active, inl=%u!!!", + dev->inb.nb_sess, !!dev->inb.inl_dev); + + /* Change the mode */ + dev->inb.inl_dev = use_inl_dev; + + /* Update RoC for NPC rule insertion */ + roc_nix_inb_mode_set(nix, use_inl_dev); + + /* Setup lookup mem */ + return cnxk_nix_lookup_mem_sa_base_set(dev); +} + +static int +nix_security_setup(struct cnxk_eth_dev *dev) +{ + struct roc_nix *nix = &dev->nix; + int i, rc = 0; + + if (dev->rx_offloads & DEV_RX_OFFLOAD_SECURITY) { + /* Setup Inline Inbound */ + rc = roc_nix_inl_inb_init(nix); + if (rc) { + plt_err("Failed to initialize nix inline inb, rc=%d", + rc); + return rc; + } + + /* By default pick using inline device for poll mode. + * Will be overridden when event mode rq's are setup. + */ + cnxk_nix_inb_mode_set(dev, true); + } + + /* Setup Inline outbound */ + if (dev->tx_offloads & DEV_TX_OFFLOAD_SECURITY) { + struct plt_bitmap *bmap; + size_t bmap_sz; + void *mem; + + /* Cannot ask for Tx Inline without SAs */ + if (!dev->outb.max_sa) + return -EINVAL; + + /* Setup enough descriptors for all tx queues */ + nix->outb_nb_desc = dev->outb.nb_desc; + nix->outb_nb_crypto_qs = dev->outb.nb_crypto_qs; + + /* Setup Inline Outbound */ + rc = roc_nix_inl_outb_init(nix); + if (rc) { + plt_err("Failed to initialize nix inline outb, rc=%d", + rc); + goto cleanup; + } + + rc = -ENOMEM; + /* Allocate a bitmap to alloc and free sa indexes */ + bmap_sz = plt_bitmap_get_memory_footprint(dev->outb.max_sa); + mem = plt_zmalloc(bmap_sz, PLT_CACHE_LINE_SIZE); + if (mem == NULL) { + plt_err("Outbound SA bmap alloc failed"); + + rc |= roc_nix_inl_outb_fini(nix); + goto cleanup; + } + + rc = -EIO; + bmap = plt_bitmap_init(dev->outb.max_sa, mem, bmap_sz); + if (!bmap) { + plt_err("Outbound SA bmap init failed"); + + rc |= roc_nix_inl_outb_fini(nix); + plt_free(mem); + goto cleanup; + } + + for (i = 0; i < dev->outb.max_sa; i++) + plt_bitmap_set(bmap, i); + + dev->outb.sa_base = roc_nix_inl_outb_sa_base_get(nix); + dev->outb.sa_bmap_mem = mem; + dev->outb.sa_bmap = bmap; + dev->outb.lf_base = roc_nix_inl_outb_lf_base_get(nix); + } + + return 0; +cleanup: + if (dev->rx_offloads & DEV_RX_OFFLOAD_SECURITY) + rc |= roc_nix_inl_inb_fini(nix); + return rc; +} + +static int +nix_security_release(struct cnxk_eth_dev *dev) +{ + struct rte_eth_dev *eth_dev = dev->eth_dev; + struct cnxk_eth_sec_sess *eth_sec, *tvar; + struct roc_nix *nix = &dev->nix; + int rc, ret = 0; + + /* Cleanup Inline inbound */ + if (dev->rx_offloads & DEV_RX_OFFLOAD_SECURITY) { + /* Destroy inbound sessions */ + tvar = NULL; + TAILQ_FOREACH_SAFE(eth_sec, &dev->inb.list, entry, tvar) + cnxk_eth_sec_ops.session_destroy(eth_dev, + eth_sec->sess); + + /* Clear lookup mem */ + cnxk_nix_lookup_mem_sa_base_clear(dev); + + rc = roc_nix_inl_inb_fini(nix); + if (rc) + plt_err("Failed to cleanup nix inline inb, rc=%d", rc); + ret |= rc; + } + + /* Cleanup Inline outbound */ + if (dev->tx_offloads & DEV_TX_OFFLOAD_SECURITY) { + /* Destroy outbound sessions */ + tvar = NULL; + TAILQ_FOREACH_SAFE(eth_sec, &dev->outb.list, entry, tvar) + cnxk_eth_sec_ops.session_destroy(eth_dev, + eth_sec->sess); + + rc = roc_nix_inl_outb_fini(nix); + if (rc) + plt_err("Failed to cleanup nix inline outb, rc=%d", rc); + ret |= rc; + + plt_bitmap_free(dev->outb.sa_bmap); + plt_free(dev->outb.sa_bmap_mem); + dev->outb.sa_bmap = NULL; + dev->outb.sa_bmap_mem = NULL; + } + + dev->inb.inl_dev = false; + roc_nix_inb_mode_set(nix, false); + dev->nb_rxq_sso = 0; + dev->inb.nb_sess = 0; + dev->outb.nb_sess = 0; + return ret; +} + static void nix_enable_mseg_on_jumbo(struct cnxk_eth_rxq_sp *rxq) { @@ -194,6 +347,12 @@ cnxk_nix_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, eth_dev->data->tx_queues[qid] = NULL; } + /* When Tx Security offload is enabled, increase tx desc count by + * max possible outbound desc count. + */ + if (dev->tx_offloads & DEV_TX_OFFLOAD_SECURITY) + nb_desc += dev->outb.nb_desc; + /* Setup ROC SQ */ sq = &dev->sqs[qid]; sq->qid = qid; @@ -266,6 +425,7 @@ cnxk_nix_rx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, struct rte_mempool *mp) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + struct roc_nix *nix = &dev->nix; struct cnxk_eth_rxq_sp *rxq_sp; struct rte_mempool_ops *ops; const char *platform_ops; @@ -328,6 +488,10 @@ cnxk_nix_rx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, rq->later_skip = sizeof(struct rte_mbuf); rq->lpb_size = mp->elt_size; + /* Enable Inline IPSec on RQ, will not be used for Poll mode */ + if (roc_nix_inl_inb_is_enabled(nix)) + rq->ipsech_ena = true; + rc = roc_nix_rq_init(&dev->nix, rq, !!eth_dev->data->dev_started); if (rc) { plt_err("Failed to init roc rq for rq=%d, rc=%d", qid, rc); @@ -350,6 +514,13 @@ cnxk_nix_rx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, rxq_sp->qconf.nb_desc = nb_desc; rxq_sp->qconf.mp = mp; + if (dev->rx_offloads & DEV_RX_OFFLOAD_SECURITY) { + /* Setup rq reference for inline dev if present */ + rc = roc_nix_inl_dev_rq_get(rq); + if (rc) + goto free_mem; + } + plt_nix_dbg("rq=%d pool=%s nb_desc=%d->%d", qid, mp->name, nb_desc, cq->nb_desc); @@ -370,6 +541,8 @@ cnxk_nix_rx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, } return 0; +free_mem: + plt_free(rxq_sp); rq_fini: rc |= roc_nix_rq_fini(rq); cq_fini: @@ -394,11 +567,15 @@ cnxk_nix_rx_queue_release(void *rxq) rxq_sp = cnxk_eth_rxq_to_sp(rxq); dev = rxq_sp->dev; qid = rxq_sp->qid; + rq = &dev->rqs[qid]; plt_nix_dbg("Releasing rxq %u", qid); + /* Release rq reference for inline dev if present */ + if (dev->rx_offloads & DEV_RX_OFFLOAD_SECURITY) + roc_nix_inl_dev_rq_put(rq); + /* Cleanup ROC RQ */ - rq = &dev->rqs[qid]; rc = roc_nix_rq_fini(rq); if (rc) plt_err("Failed to cleanup rq, rc=%d", rc); @@ -804,6 +981,12 @@ cnxk_nix_configure(struct rte_eth_dev *eth_dev) rc = nix_store_queue_cfg_and_then_release(eth_dev); if (rc) goto fail_configure; + + /* Cleanup security support */ + rc = nix_security_release(dev); + if (rc) + goto fail_configure; + roc_nix_tm_fini(nix); roc_nix_lf_free(nix); } @@ -958,6 +1141,12 @@ cnxk_nix_configure(struct rte_eth_dev *eth_dev) plt_err("Failed to initialize flow control rc=%d", rc); goto cq_fini; } + + /* Setup Inline security support */ + rc = nix_security_setup(dev); + if (rc) + goto cq_fini; + /* * Restore queue config when reconfigure followed by * reconfigure and no queue configure invoked from application case. @@ -965,7 +1154,7 @@ cnxk_nix_configure(struct rte_eth_dev *eth_dev) if (dev->configured == 1) { rc = nix_restore_queue_cfg(eth_dev); if (rc) - goto cq_fini; + goto sec_release; } /* Update the mac address */ @@ -987,6 +1176,8 @@ cnxk_nix_configure(struct rte_eth_dev *eth_dev) dev->nb_txq = data->nb_tx_queues; return 0; +sec_release: + rc |= nix_security_release(dev); cq_fini: roc_nix_unregister_cq_irqs(nix); q_irq_fini: @@ -1282,12 +1473,25 @@ static int cnxk_eth_dev_init(struct rte_eth_dev *eth_dev) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + struct rte_security_ctx *sec_ctx; struct roc_nix *nix = &dev->nix; struct rte_pci_device *pci_dev; int rc, max_entries; eth_dev->dev_ops = &cnxk_eth_dev_ops; + /* Alloc security context */ + sec_ctx = plt_zmalloc(sizeof(struct rte_security_ctx), 0); + if (!sec_ctx) + return -ENOMEM; + sec_ctx->device = eth_dev; + sec_ctx->ops = &cnxk_eth_sec_ops; + sec_ctx->flags = + (RTE_SEC_CTX_F_FAST_SET_MDATA | RTE_SEC_CTX_F_FAST_GET_UDATA); + eth_dev->security_ctx = sec_ctx; + TAILQ_INIT(&dev->inb.list); + TAILQ_INIT(&dev->outb.list); + /* For secondary processes, the primary has done all the work */ if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1400,6 +1604,9 @@ cnxk_eth_dev_uninit(struct rte_eth_dev *eth_dev, bool reset) struct roc_nix *nix = &dev->nix; int rc, i; + plt_free(eth_dev->security_ctx); + eth_dev->security_ctx = NULL; + /* Nothing to be done for secondary processes */ if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; @@ -1429,6 +1636,9 @@ cnxk_eth_dev_uninit(struct rte_eth_dev *eth_dev, bool reset) } eth_dev->data->nb_rx_queues = 0; + /* Free security resources */ + nix_security_release(dev); + /* Free tm resources */ roc_nix_tm_fini(nix); diff --git a/drivers/net/cnxk/cnxk_ethdev.h b/drivers/net/cnxk/cnxk_ethdev.h index 2528b3c..5ae791f 100644 --- a/drivers/net/cnxk/cnxk_ethdev.h +++ b/drivers/net/cnxk/cnxk_ethdev.h @@ -13,6 +13,9 @@ #include #include #include +#include +#include +#include #include #include "roc_api.h" @@ -70,14 +73,14 @@ DEV_TX_OFFLOAD_SCTP_CKSUM | DEV_TX_OFFLOAD_TCP_TSO | \ DEV_TX_OFFLOAD_VXLAN_TNL_TSO | DEV_TX_OFFLOAD_GENEVE_TNL_TSO | \ DEV_TX_OFFLOAD_GRE_TNL_TSO | DEV_TX_OFFLOAD_MULTI_SEGS | \ - DEV_TX_OFFLOAD_IPV4_CKSUM) + DEV_TX_OFFLOAD_IPV4_CKSUM | DEV_TX_OFFLOAD_SECURITY) #define CNXK_NIX_RX_OFFLOAD_CAPA \ (DEV_RX_OFFLOAD_CHECKSUM | DEV_RX_OFFLOAD_SCTP_CKSUM | \ DEV_RX_OFFLOAD_OUTER_IPV4_CKSUM | DEV_RX_OFFLOAD_SCATTER | \ DEV_RX_OFFLOAD_JUMBO_FRAME | DEV_RX_OFFLOAD_OUTER_UDP_CKSUM | \ DEV_RX_OFFLOAD_RSS_HASH | DEV_RX_OFFLOAD_TIMESTAMP | \ - DEV_RX_OFFLOAD_VLAN_STRIP) + DEV_RX_OFFLOAD_VLAN_STRIP | DEV_RX_OFFLOAD_SECURITY) #define RSS_IPV4_ENABLE \ (ETH_RSS_IPV4 | ETH_RSS_FRAG_IPV4 | ETH_RSS_NONFRAG_IPV4_UDP | \ @@ -112,6 +115,11 @@ #define PTYPE_TUNNEL_ARRAY_SZ BIT(PTYPE_TUNNEL_WIDTH) #define PTYPE_ARRAY_SZ \ ((PTYPE_NON_TUNNEL_ARRAY_SZ + PTYPE_TUNNEL_ARRAY_SZ) * sizeof(uint16_t)) + +/* NIX_RX_PARSE_S's ERRCODE + ERRLEV (12 bits) */ +#define ERRCODE_ERRLEN_WIDTH 12 +#define ERR_ARRAY_SZ ((BIT(ERRCODE_ERRLEN_WIDTH)) * sizeof(uint32_t)) + /* Fastpath lookup */ #define CNXK_NIX_FASTPATH_LOOKUP_MEM "cnxk_nix_fastpath_lookup_mem" @@ -119,6 +127,9 @@ ((1ull << (PKT_TX_TUNNEL_VXLAN >> 45)) | \ (1ull << (PKT_TX_TUNNEL_GENEVE >> 45))) +/* Subtype from inline outbound error event */ +#define CNXK_ETHDEV_SEC_OUTB_EV_SUB 0xFFUL + struct cnxk_fc_cfg { enum rte_eth_fc_mode mode; uint8_t rx_pause; @@ -144,6 +155,82 @@ struct cnxk_timesync_info { uint64_t *tx_tstamp; } __plt_cache_aligned; +/* Security session private data */ +struct cnxk_eth_sec_sess { + /* List entry */ + TAILQ_ENTRY(cnxk_eth_sec_sess) entry; + + /* Inbound SA is from NIX_RX_IPSEC_SA_BASE or + * Outbound SA from roc_nix_inl_outb_sa_base_get() + */ + void *sa; + + /* SA index */ + uint32_t sa_idx; + + /* SPI */ + uint32_t spi; + + /* Back pointer to session */ + struct rte_security_session *sess; + + /* Inbound */ + bool inb; + + /* Inbound session on inl dev */ + bool inl_dev; +}; + +TAILQ_HEAD(cnxk_eth_sec_sess_list, cnxk_eth_sec_sess); + +/* Inbound security data */ +struct cnxk_eth_dev_sec_inb { + /* IPSec inbound max SPI */ + uint16_t max_spi; + + /* Using inbound with inline device */ + bool inl_dev; + + /* Device argument to force inline device for inb */ + bool force_inl_dev; + + /* Active sessions */ + uint16_t nb_sess; + + /* List of sessions */ + struct cnxk_eth_sec_sess_list list; +}; + +/* Outbound security data */ +struct cnxk_eth_dev_sec_outb { + /* IPSec outbound max SA */ + uint16_t max_sa; + + /* Per CPT LF descriptor count */ + uint32_t nb_desc; + + /* SA Bitmap */ + struct plt_bitmap *sa_bmap; + + /* SA bitmap memory */ + void *sa_bmap_mem; + + /* SA base */ + uint64_t sa_base; + + /* CPT LF base */ + struct roc_cpt_lf *lf_base; + + /* Crypto queues => CPT lf count */ + uint16_t nb_crypto_qs; + + /* Active sessions */ + uint16_t nb_sess; + + /* List of sessions */ + struct cnxk_eth_sec_sess_list list; +}; + struct cnxk_eth_dev { /* ROC NIX */ struct roc_nix nix; @@ -159,6 +246,7 @@ struct cnxk_eth_dev { /* Configured queue count */ uint16_t nb_rxq; uint16_t nb_txq; + uint16_t nb_rxq_sso; uint8_t configured; /* Max macfilter entries */ @@ -223,6 +311,10 @@ struct cnxk_eth_dev { /* Per queue statistics counters */ uint32_t txq_stat_map[RTE_ETHDEV_QUEUE_STAT_CNTRS]; uint32_t rxq_stat_map[RTE_ETHDEV_QUEUE_STAT_CNTRS]; + + /* Security data */ + struct cnxk_eth_dev_sec_inb inb; + struct cnxk_eth_dev_sec_outb outb; }; struct cnxk_eth_rxq_sp { @@ -261,6 +353,9 @@ extern struct eth_dev_ops cnxk_eth_dev_ops; /* Common flow ops */ extern struct rte_flow_ops cnxk_flow_ops; +/* Common security ops */ +extern struct rte_security_ops cnxk_eth_sec_ops; + /* Ops */ int cnxk_nix_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev); @@ -383,6 +478,18 @@ int cnxk_ethdev_parse_devargs(struct rte_devargs *devargs, /* Debug */ int cnxk_nix_dev_get_reg(struct rte_eth_dev *eth_dev, struct rte_dev_reg_info *regs); +/* Security */ +int cnxk_eth_outb_sa_idx_get(struct cnxk_eth_dev *dev, uint32_t *idx_p); +int cnxk_eth_outb_sa_idx_put(struct cnxk_eth_dev *dev, uint32_t idx); +int cnxk_nix_lookup_mem_sa_base_set(struct cnxk_eth_dev *dev); +int cnxk_nix_lookup_mem_sa_base_clear(struct cnxk_eth_dev *dev); +__rte_internal +int cnxk_nix_inb_mode_set(struct cnxk_eth_dev *dev, bool use_inl_dev); +struct cnxk_eth_sec_sess *cnxk_eth_sec_sess_get_by_spi(struct cnxk_eth_dev *dev, + uint32_t spi, bool inb); +struct cnxk_eth_sec_sess * +cnxk_eth_sec_sess_get_by_sess(struct cnxk_eth_dev *dev, + struct rte_security_session *sess); /* Other private functions */ int nix_recalc_mtu(struct rte_eth_dev *eth_dev); @@ -493,4 +600,14 @@ cnxk_nix_mbuf_to_tstamp(struct rte_mbuf *mbuf, } } +static __rte_always_inline uintptr_t +cnxk_nix_sa_base_get(uint16_t port, const void *lookup_mem) +{ + uintptr_t sa_base_tbl; + + sa_base_tbl = (uintptr_t)lookup_mem; + sa_base_tbl += PTYPE_ARRAY_SZ + ERR_ARRAY_SZ; + return *((const uintptr_t *)sa_base_tbl + port); +} + #endif /* __CNXK_ETHDEV_H__ */ diff --git a/drivers/net/cnxk/cnxk_ethdev_devargs.c b/drivers/net/cnxk/cnxk_ethdev_devargs.c index 37720fb..c0b949e 100644 --- a/drivers/net/cnxk/cnxk_ethdev_devargs.c +++ b/drivers/net/cnxk/cnxk_ethdev_devargs.c @@ -8,6 +8,61 @@ #include "cnxk_ethdev.h" static int +parse_outb_nb_desc(const char *key, const char *value, void *extra_args) +{ + RTE_SET_USED(key); + uint32_t val; + + val = atoi(value); + + *(uint16_t *)extra_args = val; + + return 0; +} + +static int +parse_outb_nb_crypto_qs(const char *key, const char *value, void *extra_args) +{ + RTE_SET_USED(key); + uint32_t val; + + val = atoi(value); + + if (val < 1 || val > 64) + return -EINVAL; + + *(uint16_t *)extra_args = val; + + return 0; +} + +static int +parse_ipsec_in_max_spi(const char *key, const char *value, void *extra_args) +{ + RTE_SET_USED(key); + uint32_t val; + + val = atoi(value); + + *(uint16_t *)extra_args = val; + + return 0; +} + +static int +parse_ipsec_out_max_sa(const char *key, const char *value, void *extra_args) +{ + RTE_SET_USED(key); + uint32_t val; + + val = atoi(value); + + *(uint16_t *)extra_args = val; + + return 0; +} + +static int parse_flow_max_priority(const char *key, const char *value, void *extra_args) { RTE_SET_USED(key); @@ -117,15 +172,25 @@ parse_switch_header_type(const char *key, const char *value, void *extra_args) #define CNXK_SWITCH_HEADER_TYPE "switch_header" #define CNXK_RSS_TAG_AS_XOR "tag_as_xor" #define CNXK_LOCK_RX_CTX "lock_rx_ctx" +#define CNXK_IPSEC_IN_MAX_SPI "ipsec_in_max_spi" +#define CNXK_IPSEC_OUT_MAX_SA "ipsec_out_max_sa" +#define CNXK_OUTB_NB_DESC "outb_nb_desc" +#define CNXK_FORCE_INB_INL_DEV "force_inb_inl_dev" +#define CNXK_OUTB_NB_CRYPTO_QS "outb_nb_crypto_qs" int cnxk_ethdev_parse_devargs(struct rte_devargs *devargs, struct cnxk_eth_dev *dev) { uint16_t reta_sz = ROC_NIX_RSS_RETA_SZ_64; uint16_t sqb_count = CNXK_NIX_TX_MAX_SQB; + uint16_t ipsec_in_max_spi = BIT(8) - 1; + uint16_t ipsec_out_max_sa = BIT(12); uint16_t flow_prealloc_size = 1; uint16_t switch_header_type = 0; uint16_t flow_max_priority = 3; + uint16_t force_inb_inl_dev = 0; + uint16_t outb_nb_crypto_qs = 1; + uint16_t outb_nb_desc = 8200; uint16_t rss_tag_as_xor = 0; uint16_t scalar_enable = 0; uint8_t lock_rx_ctx = 0; @@ -153,10 +218,27 @@ cnxk_ethdev_parse_devargs(struct rte_devargs *devargs, struct cnxk_eth_dev *dev) rte_kvargs_process(kvlist, CNXK_RSS_TAG_AS_XOR, &parse_flag, &rss_tag_as_xor); rte_kvargs_process(kvlist, CNXK_LOCK_RX_CTX, &parse_flag, &lock_rx_ctx); + rte_kvargs_process(kvlist, CNXK_IPSEC_IN_MAX_SPI, + &parse_ipsec_in_max_spi, &ipsec_in_max_spi); + rte_kvargs_process(kvlist, CNXK_IPSEC_OUT_MAX_SA, + &parse_ipsec_out_max_sa, &ipsec_out_max_sa); + rte_kvargs_process(kvlist, CNXK_OUTB_NB_DESC, &parse_outb_nb_desc, + &outb_nb_desc); + rte_kvargs_process(kvlist, CNXK_OUTB_NB_CRYPTO_QS, + &parse_outb_nb_crypto_qs, &outb_nb_crypto_qs); + rte_kvargs_process(kvlist, CNXK_FORCE_INB_INL_DEV, &parse_flag, + &force_inb_inl_dev); rte_kvargs_free(kvlist); null_devargs: dev->scalar_ena = !!scalar_enable; + dev->inb.force_inl_dev = !!force_inb_inl_dev; + dev->inb.max_spi = ipsec_in_max_spi; + dev->outb.max_sa = ipsec_out_max_sa; + dev->outb.nb_desc = outb_nb_desc; + dev->outb.nb_crypto_qs = outb_nb_crypto_qs; + dev->nix.ipsec_in_max_spi = ipsec_in_max_spi; + dev->nix.ipsec_out_max_sa = ipsec_out_max_sa; dev->nix.rss_tag_as_xor = !!rss_tag_as_xor; dev->nix.max_sqb_count = sqb_count; dev->nix.reta_sz = reta_sz; @@ -177,4 +259,8 @@ RTE_PMD_REGISTER_PARAM_STRING(net_cnxk, CNXK_FLOW_PREALLOC_SIZE "=<1-32>" CNXK_FLOW_MAX_PRIORITY "=<1-32>" CNXK_SWITCH_HEADER_TYPE "=" - CNXK_RSS_TAG_AS_XOR "=1"); + CNXK_RSS_TAG_AS_XOR "=1" + CNXK_IPSEC_IN_MAX_SPI "=<1-65535>" + CNXK_OUTB_NB_DESC "=<1-65535>" + CNXK_OUTB_NB_CRYPTO_QS "=<1-64>" + CNXK_FORCE_INB_INL_DEV "=1"); diff --git a/drivers/net/cnxk/cnxk_ethdev_sec.c b/drivers/net/cnxk/cnxk_ethdev_sec.c new file mode 100644 index 0000000..c002c30 --- /dev/null +++ b/drivers/net/cnxk/cnxk_ethdev_sec.c @@ -0,0 +1,278 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include + +#define CNXK_NIX_INL_SELFTEST "selftest" +#define CNXK_NIX_INL_IPSEC_IN_MAX_SPI "ipsec_in_max_spi" + +#define CNXK_NIX_INL_DEV_NAME RTE_STR(cnxk_nix_inl_dev_) +#define CNXK_NIX_INL_DEV_NAME_LEN \ + (sizeof(CNXK_NIX_INL_DEV_NAME) + PCI_PRI_STR_SIZE) + +static inline int +bitmap_ctzll(uint64_t slab) +{ + if (slab == 0) + return 0; + + return __builtin_ctzll(slab); +} + +int +cnxk_eth_outb_sa_idx_get(struct cnxk_eth_dev *dev, uint32_t *idx_p) +{ + uint32_t pos, idx; + uint64_t slab; + int rc; + + if (!dev->outb.max_sa) + return -ENOTSUP; + + pos = 0; + slab = 0; + /* Scan from the beginning */ + plt_bitmap_scan_init(dev->outb.sa_bmap); + /* Scan bitmap to get the free sa index */ + rc = plt_bitmap_scan(dev->outb.sa_bmap, &pos, &slab); + /* Empty bitmap */ + if (rc == 0) { + plt_err("Outbound SA' exhausted, use 'ipsec_out_max_sa' " + "devargs to increase"); + return -ERANGE; + } + + /* Get free SA index */ + idx = pos + bitmap_ctzll(slab); + plt_bitmap_clear(dev->outb.sa_bmap, idx); + *idx_p = idx; + return 0; +} + +int +cnxk_eth_outb_sa_idx_put(struct cnxk_eth_dev *dev, uint32_t idx) +{ + if (idx >= dev->outb.max_sa) + return -EINVAL; + + /* Check if it is already free */ + if (plt_bitmap_get(dev->outb.sa_bmap, idx)) + return -EINVAL; + + /* Mark index as free */ + plt_bitmap_set(dev->outb.sa_bmap, idx); + return 0; +} + +struct cnxk_eth_sec_sess * +cnxk_eth_sec_sess_get_by_spi(struct cnxk_eth_dev *dev, uint32_t spi, bool inb) +{ + struct cnxk_eth_sec_sess_list *list; + struct cnxk_eth_sec_sess *eth_sec; + + list = inb ? &dev->inb.list : &dev->outb.list; + TAILQ_FOREACH(eth_sec, list, entry) { + if (eth_sec->spi == spi) + return eth_sec; + } + + return NULL; +} + +struct cnxk_eth_sec_sess * +cnxk_eth_sec_sess_get_by_sess(struct cnxk_eth_dev *dev, + struct rte_security_session *sess) +{ + struct cnxk_eth_sec_sess *eth_sec = NULL; + + /* Search in inbound list */ + TAILQ_FOREACH(eth_sec, &dev->inb.list, entry) { + if (eth_sec->sess == sess) + return eth_sec; + } + + /* Search in outbound list */ + TAILQ_FOREACH(eth_sec, &dev->outb.list, entry) { + if (eth_sec->sess == sess) + return eth_sec; + } + + return NULL; +} + +static unsigned int +cnxk_eth_sec_session_get_size(void *device __rte_unused) +{ + return sizeof(struct cnxk_eth_sec_sess); +} + +struct rte_security_ops cnxk_eth_sec_ops = { + .session_get_size = cnxk_eth_sec_session_get_size +}; + +static int +parse_ipsec_in_max_spi(const char *key, const char *value, void *extra_args) +{ + RTE_SET_USED(key); + uint32_t val; + + val = atoi(value); + + *(uint16_t *)extra_args = val; + + return 0; +} + +static int +parse_selftest(const char *key, const char *value, void *extra_args) +{ + RTE_SET_USED(key); + uint32_t val; + + val = atoi(value); + + *(uint8_t *)extra_args = !!(val == 1); + return 0; +} + +static int +nix_inl_parse_devargs(struct rte_devargs *devargs, + struct roc_nix_inl_dev *inl_dev) +{ + uint32_t ipsec_in_max_spi = BIT(8) - 1; + struct rte_kvargs *kvlist; + uint8_t selftest = 0; + + if (devargs == NULL) + goto null_devargs; + + kvlist = rte_kvargs_parse(devargs->args, NULL); + if (kvlist == NULL) + goto exit; + + rte_kvargs_process(kvlist, CNXK_NIX_INL_SELFTEST, &parse_selftest, + &selftest); + rte_kvargs_process(kvlist, CNXK_NIX_INL_IPSEC_IN_MAX_SPI, + &parse_ipsec_in_max_spi, &ipsec_in_max_spi); + rte_kvargs_free(kvlist); + +null_devargs: + inl_dev->ipsec_in_max_spi = ipsec_in_max_spi; + inl_dev->selftest = selftest; + return 0; +exit: + return -EINVAL; +} + +static inline char * +nix_inl_dev_to_name(struct rte_pci_device *pci_dev, char *name) +{ + snprintf(name, CNXK_NIX_INL_DEV_NAME_LEN, + CNXK_NIX_INL_DEV_NAME PCI_PRI_FMT, pci_dev->addr.domain, + pci_dev->addr.bus, pci_dev->addr.devid, + pci_dev->addr.function); + + return name; +} + +static int +cnxk_nix_inl_dev_remove(struct rte_pci_device *pci_dev) +{ + char name[CNXK_NIX_INL_DEV_NAME_LEN]; + const struct rte_memzone *mz; + struct roc_nix_inl_dev *dev; + int rc; + + if (rte_eal_process_type() != RTE_PROC_PRIMARY) + return 0; + + mz = rte_memzone_lookup(nix_inl_dev_to_name(pci_dev, name)); + if (!mz) + return 0; + + dev = mz->addr; + + /* Cleanup inline dev */ + rc = roc_nix_inl_dev_fini(dev); + if (rc) { + plt_err("Failed to cleanup inl dev, rc=%d(%s)", rc, + roc_error_msg_get(rc)); + return rc; + } + + rte_memzone_free(mz); + return 0; +} + +static int +cnxk_nix_inl_dev_probe(struct rte_pci_driver *pci_drv, + struct rte_pci_device *pci_dev) +{ + char name[CNXK_NIX_INL_DEV_NAME_LEN]; + struct roc_nix_inl_dev *inl_dev; + const struct rte_memzone *mz; + int rc = -ENOMEM; + + RTE_SET_USED(pci_drv); + + rc = roc_plt_init(); + if (rc) { + plt_err("Failed to initialize platform model, rc=%d", rc); + return rc; + } + + if (rte_eal_process_type() != RTE_PROC_PRIMARY) + return 0; + + mz = rte_memzone_reserve_aligned(nix_inl_dev_to_name(pci_dev, name), + sizeof(*inl_dev), SOCKET_ID_ANY, 0, + RTE_CACHE_LINE_SIZE); + if (mz == NULL) + return rc; + + inl_dev = mz->addr; + inl_dev->pci_dev = pci_dev; + + /* Parse devargs string */ + rc = nix_inl_parse_devargs(pci_dev->device.devargs, inl_dev); + if (rc) { + plt_err("Failed to parse devargs rc=%d", rc); + goto free_mem; + } + + rc = roc_nix_inl_dev_init(inl_dev); + if (rc) { + plt_err("Failed to init nix inl device, rc=%d(%s)", rc, + roc_error_msg_get(rc)); + goto free_mem; + } + + return 0; +free_mem: + rte_memzone_free(mz); + return rc; +} + +static const struct rte_pci_id cnxk_nix_inl_pci_map[] = { + {RTE_PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVID_CNXK_RVU_NIX_INL_PF)}, + {RTE_PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVID_CNXK_RVU_NIX_INL_VF)}, + { + .vendor_id = 0, + }, +}; + +static struct rte_pci_driver cnxk_nix_inl_pci = { + .id_table = cnxk_nix_inl_pci_map, + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_NEED_IOVA_AS_VA, + .probe = cnxk_nix_inl_dev_probe, + .remove = cnxk_nix_inl_dev_remove, +}; + +RTE_PMD_REGISTER_PCI(cnxk_nix_inl, cnxk_nix_inl_pci); +RTE_PMD_REGISTER_PCI_TABLE(cnxk_nix_inl, cnxk_nix_inl_pci_map); +RTE_PMD_REGISTER_KMOD_DEP(cnxk_nix_inl, "vfio-pci"); + +RTE_PMD_REGISTER_PARAM_STRING(cnxk_nix_inl, + CNXK_NIX_INL_SELFTEST "=1" + CNXK_NIX_INL_IPSEC_IN_MAX_SPI "=<1-65535>"); diff --git a/drivers/net/cnxk/cnxk_lookup.c b/drivers/net/cnxk/cnxk_lookup.c index 0152ad9..f6ec768 100644 --- a/drivers/net/cnxk/cnxk_lookup.c +++ b/drivers/net/cnxk/cnxk_lookup.c @@ -7,12 +7,8 @@ #include "cnxk_ethdev.h" -/* NIX_RX_PARSE_S's ERRCODE + ERRLEV (12 bits) */ -#define ERRCODE_ERRLEN_WIDTH 12 -#define ERR_ARRAY_SZ ((BIT(ERRCODE_ERRLEN_WIDTH)) * sizeof(uint32_t)) - -#define SA_TBL_SZ (RTE_MAX_ETHPORTS * sizeof(uint64_t)) -#define LOOKUP_ARRAY_SZ (PTYPE_ARRAY_SZ + ERR_ARRAY_SZ + SA_TBL_SZ) +#define SA_BASE_TBL_SZ (RTE_MAX_ETHPORTS * sizeof(uintptr_t)) +#define LOOKUP_ARRAY_SZ (PTYPE_ARRAY_SZ + ERR_ARRAY_SZ + SA_BASE_TBL_SZ) const uint32_t * cnxk_nix_supported_ptypes_get(struct rte_eth_dev *eth_dev) { @@ -324,3 +320,45 @@ cnxk_nix_fastpath_lookup_mem_get(void) } return NULL; } + +int +cnxk_nix_lookup_mem_sa_base_set(struct cnxk_eth_dev *dev) +{ + void *lookup_mem = cnxk_nix_fastpath_lookup_mem_get(); + uint16_t port = dev->eth_dev->data->port_id; + uintptr_t sa_base_tbl; + uintptr_t sa_base; + uint8_t sa_w; + + if (!lookup_mem) + return -EIO; + + sa_base = roc_nix_inl_inb_sa_base_get(&dev->nix, dev->inb.inl_dev); + if (!sa_base) + return -ENOTSUP; + + sa_w = plt_log2_u32(dev->nix.ipsec_in_max_spi + 1); + + /* Set SA Base in lookup mem */ + sa_base_tbl = (uintptr_t)lookup_mem; + sa_base_tbl += PTYPE_ARRAY_SZ + ERR_ARRAY_SZ; + *((uintptr_t *)sa_base_tbl + port) = sa_base | sa_w; + return 0; +} + +int +cnxk_nix_lookup_mem_sa_base_clear(struct cnxk_eth_dev *dev) +{ + void *lookup_mem = cnxk_nix_fastpath_lookup_mem_get(); + uint16_t port = dev->eth_dev->data->port_id; + uintptr_t sa_base_tbl; + + if (!lookup_mem) + return -EIO; + + /* Set SA Base in lookup mem */ + sa_base_tbl = (uintptr_t)lookup_mem; + sa_base_tbl += PTYPE_ARRAY_SZ + ERR_ARRAY_SZ; + *((uintptr_t *)sa_base_tbl + port) = 0; + return 0; +} diff --git a/drivers/net/cnxk/meson.build b/drivers/net/cnxk/meson.build index d4cdd17..6cc30c3 100644 --- a/drivers/net/cnxk/meson.build +++ b/drivers/net/cnxk/meson.build @@ -12,6 +12,7 @@ sources = files( 'cnxk_ethdev.c', 'cnxk_ethdev_devargs.c', 'cnxk_ethdev_ops.c', + 'cnxk_ethdev_sec.c', 'cnxk_link.c', 'cnxk_lookup.c', 'cnxk_ptp.c', @@ -22,6 +23,7 @@ sources = files( # CN9K sources += files( 'cn9k_ethdev.c', + 'cn9k_ethdev_sec.c', 'cn9k_rte_flow.c', 'cn9k_rx.c', 'cn9k_rx_mseg.c', diff --git a/drivers/net/cnxk/version.map b/drivers/net/cnxk/version.map index c2e0723..b9da6b1 100644 --- a/drivers/net/cnxk/version.map +++ b/drivers/net/cnxk/version.map @@ -1,3 +1,8 @@ DPDK_22 { local: *; }; + +INTERNAL { + global: + cnxk_nix_inb_mode_set; +}; From patchwork Thu Sep 2 02:14:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97742 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C35C4A0C4C; Thu, 2 Sep 2021 04:18:34 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C3BD641183; Thu, 2 Sep 2021 04:17:35 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id A2FFD40698 for ; Thu, 2 Sep 2021 04:17:34 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQCpp011801; Wed, 1 Sep 2021 19:17:34 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=9DTCrhW2O6z5G/cXb1TSwRdqvaYX04Pg8ExMrBrSLgw=; b=JT5a+0tOUhgbzayR+/LEiaoUYjEsPkZUJ+1UsS2f0dhbsV1/q2Qn+jpxWXPgS7CvRGLf 1FC3/zj7/trr1ovgFmD63cLqK6ov6WyXtOEk0fyFais6A4cNbRg8m4qetwTIF80ZgmX0 QCjcyGN7WYfjW2TfU9EgThPbErRnzy4F9W8DIOCZ1CAvtf+DQT5OhfmUi469eq3J3zEa MXZae/PKJR+ve+p6bF78vXdXW0WQiBKHx4e+M+ZlZXyfeZq1sIQSRAxun7AJiobzs2+J lc/SfmP4oel+mYzk42+FvLJQ3CwVByiaoRGU+e9/XJXISL0gmU4o4++HyhqJO4MzFxdI wQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9htp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 01 Sep 2021 19:17:33 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:31 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:31 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 8AEBB3F7050; Wed, 1 Sep 2021 19:17:28 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao , Pavan Nikhilesh , "Shijith Thotton" , Anatoly Burakov CC: , , Date: Thu, 2 Sep 2021 07:44:54 +0530 Message-ID: <20210902021505.17607-17-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: dzPfWECBRdulXd0dGXIFPnpLhwaIvtI1 X-Proofpoint-GUID: dzPfWECBRdulXd0dGXIFPnpLhwaIvtI1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 16/27] net/cnxk: add inline security support for cn10k X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add support for inline inbound and outbound IPSec for SA create, destroy and other NIX / CPT LF configurations. This patch also changes dpdk-devbind.py to list new inline device as misc device. Signed-off-by: Nithin Dabilpuram --- doc/guides/nics/cnxk.rst | 102 ++++++++ drivers/event/cnxk/cnxk_eventdev_adptr.c | 36 ++- drivers/net/cnxk/cn10k_ethdev.c | 36 ++- drivers/net/cnxk/cn10k_ethdev.h | 43 ++++ drivers/net/cnxk/cn10k_ethdev_sec.c | 426 +++++++++++++++++++++++++++++++ drivers/net/cnxk/cn10k_rx.h | 1 + drivers/net/cnxk/cn10k_tx.h | 1 + drivers/net/cnxk/meson.build | 1 + usertools/dpdk-devbind.py | 8 +- 9 files changed, 649 insertions(+), 5 deletions(-) create mode 100644 drivers/net/cnxk/cn10k_ethdev_sec.c diff --git a/doc/guides/nics/cnxk.rst b/doc/guides/nics/cnxk.rst index 90d27db..b542437 100644 --- a/doc/guides/nics/cnxk.rst +++ b/doc/guides/nics/cnxk.rst @@ -34,6 +34,7 @@ Features of the CNXK Ethdev PMD are: - Vector Poll mode driver - Debug utilities - Context dump and error interrupt support - Support Rx interrupt +- Inline IPsec processing support Prerequisites ------------- @@ -185,6 +186,74 @@ Runtime Config Options -a 0002:02:00.0,tag_as_xor=1 +- ``Max SPI for inbound inline IPsec`` (default ``255``) + + Max SPI supported for inbound inline IPsec processing can be specified by + ``ipsec_in_max_spi`` ``devargs`` parameter. + + For example:: + + -a 0002:02:00.0,ipsec_in_max_spi=128 + + With the above configuration, application can enable inline IPsec processing + for 128 inbound SAs (SPI 0-127). + +- ``Max SA's for outbound inline IPsec`` (default ``4096``) + + Max number of SA's supported for outbound inline IPsec processing can be + specified by ``ipsec_out_max_sa`` ``devargs`` parameter. + + For example:: + + -a 0002:02:00.0,ipsec_out_max_sa=128 + + With the above configuration, application can enable inline IPsec processing + for 128 outbound SAs. + +- ``Outbound CPT LF queue size`` (default ``8200``) + + Size of Outbound CPT LF queue in number of descriptors can be specified by + ``outb_nb_desc`` ``devargs`` parameter. + + For example:: + + -a 0002:02:00.0,outb_nb_desc=16384 + + With the above configuration, Outbound CPT LF will be created to accommodate + at max 16384 descriptors at any given time. + +- ``Outbound CPT LF count`` (default ``1``) + + Number of CPT LF's to attach for Outbound processing can be specified by + ``outb_nb_crypto_qs`` ``devargs`` parameter. + + For example:: + + -a 0002:02:00.0,outb_nb_crypto_qs=2 + + With the above confiuration, two CPT LF's are setup and distributed among + all the Tx queues for outbound processing. + +- ``Force using inline ipsec device for inbound`` (default ``0``) + + In CN10K, in event mode, driver can work in two modes, + + 1. Inbound encrypted traffic received by probed ipsec inline device while + plain traffic post decryption is received by ethdev. + + 2. Both Inbound encrypted traffic and plain traffic post decryption are + received by ethdev. + + By default event mode works without using inline device i.e mode ``2``. + This behaviour can be changed to pick mode ``1`` by using + ``force_inb_inl_dev`` ``devargs`` parameter. + + For example:: + + -a 0002:02:00.0,force_inb_inl_dev=1 -a 0002:03:00.0,force_inb_inl_dev=1 + + With the above configuration, inbound encrypted traffic from both the ports + is received by ipsec inline device. .. note:: @@ -250,6 +319,39 @@ Example usage in testpmd:: testpmd> flow create 0 ingress pattern eth / raw relative is 0 pattern \ spec ab pattern mask ab offset is 4 / end actions queue index 1 / end +Inline device support for CN10K +------------------------------- + +CN10K HW provides a misc device Inline device that supports ethernet devices in +providing following features. + + - Aggregate all the inline IPsec inbound traffic from all the CN10K ethernet + devices to be processed by the single inline IPSec device. This allows + single rte security session to accept traffic from multiple ports. + + - Support for event generation on outbound inline IPsec processing errors. + + - Support CN106xx poll mode of operation for inline IPSec inbound processing. + +Inline IPsec device is identified by PCI PF vendid:devid ``177D:A0F0`` or +VF ``177D:A0F1``. + +Runtime Config Options for inline device +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- ``Max SPI for inbound inline IPsec`` (default ``255``) + + Max SPI supported for inbound inline IPsec processing can be specified by + ``ipsec_in_max_spi`` ``devargs`` parameter. + + For example:: + + -a 0002:1d:00.0,ipsec_in_max_spi=128 + + With the above configuration, application can enable inline IPsec processing + for 128 inbound SAs (SPI 0-127) for traffic aggregated on inline device. + + Debugging Options ----------------- diff --git a/drivers/event/cnxk/cnxk_eventdev_adptr.c b/drivers/event/cnxk/cnxk_eventdev_adptr.c index baf2f2a..a34efbb 100644 --- a/drivers/event/cnxk/cnxk_eventdev_adptr.c +++ b/drivers/event/cnxk/cnxk_eventdev_adptr.c @@ -123,7 +123,9 @@ cnxk_sso_rxq_enable(struct cnxk_eth_dev *cnxk_eth_dev, uint16_t rq_id, uint16_t port_id, const struct rte_event *ev, uint8_t custom_flowid) { + struct roc_nix *nix = &cnxk_eth_dev->nix; struct roc_nix_rq *rq; + int rc; rq = &cnxk_eth_dev->rqs[rq_id]; rq->sso_ena = 1; @@ -140,7 +142,24 @@ cnxk_sso_rxq_enable(struct cnxk_eth_dev *cnxk_eth_dev, uint16_t rq_id, rq->tag_mask |= ev->flow_id; } - return roc_nix_rq_modify(&cnxk_eth_dev->nix, rq, 0); + rc = roc_nix_rq_modify(&cnxk_eth_dev->nix, rq, 0); + if (rc) + return rc; + + if (rq_id == 0 && roc_nix_inl_inb_is_enabled(nix)) { + uint32_t sec_tag_const; + + /* IPSec tag const is 8-bit left shifted value of tag_mask + * as it applies to bit 32:8 of tag only. + */ + sec_tag_const = rq->tag_mask >> 8; + rc = roc_nix_inl_inb_tag_update(nix, sec_tag_const, + ev->sched_type); + if (rc) + plt_err("Failed to set tag conf for ipsec, rc=%d", rc); + } + + return rc; } static int @@ -186,6 +205,7 @@ cnxk_sso_rx_adapter_queue_add( rox_nix_fc_npa_bp_cfg(&cnxk_eth_dev->nix, rxq_sp->qconf.mp->pool_id, true, dev->force_ena_bp); + cnxk_eth_dev->nb_rxq_sso++; } if (rc < 0) { @@ -196,6 +216,14 @@ cnxk_sso_rx_adapter_queue_add( dev->rx_offloads |= cnxk_eth_dev->rx_offload_flags; + /* Switch to use PF/VF's NIX LF instead of inline device for inbound + * when all the RQ's are switched to event dev mode. We do this only + * when using inline device is not forced by dev args. + */ + if (!cnxk_eth_dev->inb.force_inl_dev && + cnxk_eth_dev->nb_rxq_sso == cnxk_eth_dev->nb_rxq) + cnxk_nix_inb_mode_set(cnxk_eth_dev, false); + return 0; } @@ -220,12 +248,18 @@ cnxk_sso_rx_adapter_queue_del(const struct rte_eventdev *event_dev, rox_nix_fc_npa_bp_cfg(&cnxk_eth_dev->nix, rxq_sp->qconf.mp->pool_id, false, dev->force_ena_bp); + cnxk_eth_dev->nb_rxq_sso--; } if (rc < 0) plt_err("Failed to clear Rx adapter config port=%d, q=%d", eth_dev->data->port_id, rx_queue_id); + /* Removing RQ from Rx adapter implies need to use + * inline device for CQ/Poll mode. + */ + cnxk_nix_inb_mode_set(cnxk_eth_dev, true); + return rc; } diff --git a/drivers/net/cnxk/cn10k_ethdev.c b/drivers/net/cnxk/cn10k_ethdev.c index 7caec6c..fa2343c 100644 --- a/drivers/net/cnxk/cn10k_ethdev.c +++ b/drivers/net/cnxk/cn10k_ethdev.c @@ -36,6 +36,9 @@ nix_rx_offload_flags(struct rte_eth_dev *eth_dev) if (!dev->ptype_disable) flags |= NIX_RX_OFFLOAD_PTYPE_F; + if (dev->rx_offloads & DEV_RX_OFFLOAD_SECURITY) + flags |= NIX_RX_OFFLOAD_SECURITY_F; + return flags; } @@ -101,6 +104,9 @@ nix_tx_offload_flags(struct rte_eth_dev *eth_dev) if ((dev->rx_offloads & DEV_RX_OFFLOAD_TIMESTAMP)) flags |= NIX_TX_OFFLOAD_TSTAMP_F; + if (conf & DEV_TX_OFFLOAD_SECURITY) + flags |= NIX_TX_OFFLOAD_SECURITY_F; + return flags; } @@ -181,8 +187,11 @@ cn10k_nix_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, const struct rte_eth_txconf *tx_conf) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + struct roc_nix *nix = &dev->nix; + struct roc_cpt_lf *inl_lf; struct cn10k_eth_txq *txq; struct roc_nix_sq *sq; + uint16_t crypto_qid; int rc; RTE_SET_USED(socket); @@ -198,11 +207,24 @@ cn10k_nix_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, txq = eth_dev->data->tx_queues[qid]; txq->fc_mem = sq->fc; /* Store lmt base in tx queue for easy access */ - txq->lmt_base = dev->nix.lmt_base; + txq->lmt_base = nix->lmt_base; txq->io_addr = sq->io_addr; txq->nb_sqb_bufs_adj = sq->nb_sqb_bufs_adj; txq->sqes_per_sqb_log2 = sq->sqes_per_sqb_log2; + /* Fetch CPT LF info for outbound if present */ + if (dev->outb.lf_base) { + crypto_qid = qid % dev->outb.nb_crypto_qs; + inl_lf = dev->outb.lf_base + crypto_qid; + + txq->cpt_io_addr = inl_lf->io_addr; + txq->cpt_fc = inl_lf->fc_addr; + txq->cpt_desc = inl_lf->nb_desc * 0.7; + txq->sa_base = (uint64_t)dev->outb.sa_base; + txq->sa_base |= eth_dev->data->port_id; + PLT_STATIC_ASSERT(ROC_NIX_INL_SA_BASE_ALIGN == BIT_ULL(16)); + } + nix_form_default_desc(dev, txq, qid); txq->lso_tun_fmt = dev->lso_tun_fmt; return 0; @@ -215,6 +237,7 @@ cn10k_nix_rx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, struct rte_mempool *mp) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + struct cnxk_eth_rxq_sp *rxq_sp; struct cn10k_eth_rxq *rxq; struct roc_nix_rq *rq; struct roc_nix_cq *cq; @@ -250,6 +273,15 @@ cn10k_nix_rx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, rxq->data_off = rq->first_skip; rxq->mbuf_initializer = cnxk_nix_rxq_mbuf_setup(dev); + /* Setup security related info */ + if (dev->rx_offload_flags & NIX_RX_OFFLOAD_SECURITY_F) { + rxq->lmt_base = dev->nix.lmt_base; + rxq->sa_base = roc_nix_inl_inb_sa_base_get(&dev->nix, + dev->inb.inl_dev); + } + rxq_sp = cnxk_eth_rxq_to_sp(rxq); + rxq->aura_handle = rxq_sp->qconf.mp->pool_id; + /* Lookup mem */ rxq->lookup_mem = cnxk_nix_fastpath_lookup_mem_get(); return 0; @@ -500,6 +532,8 @@ cn10k_nix_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev) nix_eth_dev_ops_override(); npc_flow_ops_override(); + cn10k_eth_sec_ops_override(); + /* Common probe */ rc = cnxk_nix_probe(pci_drv, pci_dev); if (rc) diff --git a/drivers/net/cnxk/cn10k_ethdev.h b/drivers/net/cnxk/cn10k_ethdev.h index 8b6e0f2..a888364 100644 --- a/drivers/net/cnxk/cn10k_ethdev.h +++ b/drivers/net/cnxk/cn10k_ethdev.h @@ -5,6 +5,7 @@ #define __CN10K_ETHDEV_H__ #include +#include struct cn10k_eth_txq { uint64_t send_hdr_w0; @@ -15,6 +16,10 @@ struct cn10k_eth_txq { rte_iova_t io_addr; uint16_t sqes_per_sqb_log2; int16_t nb_sqb_bufs_adj; + rte_iova_t cpt_io_addr; + uint64_t sa_base; + uint64_t *cpt_fc; + uint16_t cpt_desc; uint64_t cmd[4]; uint64_t lso_tun_fmt; } __plt_cache_aligned; @@ -30,12 +35,50 @@ struct cn10k_eth_rxq { uint32_t qmask; uint32_t available; uint16_t data_off; + uint64_t sa_base; + uint64_t lmt_base; + uint64_t aura_handle; uint16_t rq; struct cnxk_timesync_info *tstamp; } __plt_cache_aligned; +/* Private data in sw rsvd area of struct roc_ot_ipsec_inb_sa */ +struct cn10k_inb_priv_data { + void *userdata; + struct cnxk_eth_sec_sess *eth_sec; +}; + +/* Private data in sw rsvd area of struct roc_ot_ipsec_outb_sa */ +struct cn10k_outb_priv_data { + void *userdata; + /* Rlen computation data */ + struct cnxk_ipsec_outb_rlens rlens; + /* Back pinter to eth sec session */ + struct cnxk_eth_sec_sess *eth_sec; + /* SA index */ + uint32_t sa_idx; +}; + +struct cn10k_sec_sess_priv { + union { + struct { + uint32_t sa_idx; + uint8_t inb_sa : 1; + uint8_t rsvd1 : 2; + uint8_t roundup_byte : 5; + uint8_t roundup_len; + uint16_t partial_len; + }; + + uint64_t u64; + }; +} __rte_packed; + /* Rx and Tx routines */ void cn10k_eth_set_rx_function(struct rte_eth_dev *eth_dev); void cn10k_eth_set_tx_function(struct rte_eth_dev *eth_dev); +/* Security context setup */ +void cn10k_eth_sec_ops_override(void); + #endif /* __CN10K_ETHDEV_H__ */ diff --git a/drivers/net/cnxk/cn10k_ethdev_sec.c b/drivers/net/cnxk/cn10k_ethdev_sec.c new file mode 100644 index 0000000..3ffd824 --- /dev/null +++ b/drivers/net/cnxk/cn10k_ethdev_sec.c @@ -0,0 +1,426 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include +#include +#include +#include + +#include +#include + +static struct rte_cryptodev_capabilities cn10k_eth_sec_crypto_caps[] = { + { /* AES GCM */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AEAD, + {.aead = { + .algo = RTE_CRYPTO_AEAD_AES_GCM, + .block_size = 16, + .key_size = { + .min = 16, + .max = 32, + .increment = 8 + }, + .digest_size = { + .min = 16, + .max = 16, + .increment = 0 + }, + .aad_size = { + .min = 8, + .max = 12, + .increment = 4 + }, + .iv_size = { + .min = 12, + .max = 12, + .increment = 0 + } + }, } + }, } + }, + RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() +}; + +static const struct rte_security_capability cn10k_eth_sec_capabilities[] = { + { /* IPsec Inline Protocol ESP Tunnel Ingress */ + .action = RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL, + .protocol = RTE_SECURITY_PROTOCOL_IPSEC, + .ipsec = { + .proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP, + .mode = RTE_SECURITY_IPSEC_SA_MODE_TUNNEL, + .direction = RTE_SECURITY_IPSEC_SA_DIR_INGRESS, + .options = { 0 } + }, + .crypto_capabilities = cn10k_eth_sec_crypto_caps, + .ol_flags = RTE_SECURITY_TX_OLOAD_NEED_MDATA + }, + { /* IPsec Inline Protocol ESP Tunnel Egress */ + .action = RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL, + .protocol = RTE_SECURITY_PROTOCOL_IPSEC, + .ipsec = { + .proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP, + .mode = RTE_SECURITY_IPSEC_SA_MODE_TUNNEL, + .direction = RTE_SECURITY_IPSEC_SA_DIR_EGRESS, + .options = { 0 } + }, + .crypto_capabilities = cn10k_eth_sec_crypto_caps, + .ol_flags = RTE_SECURITY_TX_OLOAD_NEED_MDATA + }, + { + .action = RTE_SECURITY_ACTION_TYPE_NONE + } +}; + +static void +cn10k_eth_sec_sso_work_cb(uint64_t *gw, void *args) +{ + struct rte_eth_event_ipsec_desc desc; + struct cn10k_sec_sess_priv sess_priv; + struct cn10k_outb_priv_data *priv; + struct roc_ot_ipsec_outb_sa *sa; + struct cpt_cn10k_res_s *res; + struct rte_eth_dev *eth_dev; + struct cnxk_eth_dev *dev; + uint16_t dlen_adj, rlen; + struct rte_mbuf *mbuf; + uintptr_t sa_base; + uintptr_t nixtx; + uint8_t port; + + RTE_SET_USED(args); + + switch ((gw[0] >> 28) & 0xF) { + case RTE_EVENT_TYPE_ETHDEV: + /* Event from inbound inline dev due to IPSEC packet bad L4 */ + mbuf = (struct rte_mbuf *)(gw[1] - sizeof(struct rte_mbuf)); + plt_nix_dbg("Received mbuf %p from inline dev inbound", mbuf); + rte_pktmbuf_free(mbuf); + return; + case RTE_EVENT_TYPE_CPU: + /* Check for subtype */ + if (((gw[0] >> 20) & 0xFF) == CNXK_ETHDEV_SEC_OUTB_EV_SUB) { + /* Event from outbound inline error */ + mbuf = (struct rte_mbuf *)gw[1]; + break; + } + /* Fall through */ + default: + plt_err("Unknown event gw[0] = 0x%016lx, gw[1] = 0x%016lx", + gw[0], gw[1]); + return; + } + + /* Get ethdev port from tag */ + port = gw[0] & 0xFF; + eth_dev = &rte_eth_devices[port]; + dev = cnxk_eth_pmd_priv(eth_dev); + + sess_priv.u64 = *rte_security_dynfield(mbuf); + /* Calculate dlen adj */ + dlen_adj = mbuf->pkt_len - mbuf->l2_len; + rlen = (dlen_adj + sess_priv.roundup_len) + + (sess_priv.roundup_byte - 1); + rlen &= ~(uint64_t)(sess_priv.roundup_byte - 1); + rlen += sess_priv.partial_len; + dlen_adj = rlen - dlen_adj; + + /* Find the res area residing on next cacheline after end of data */ + nixtx = rte_pktmbuf_mtod(mbuf, uintptr_t) + mbuf->pkt_len + dlen_adj; + nixtx += BIT_ULL(7); + nixtx = (nixtx - 1) & ~(BIT_ULL(7) - 1); + res = (struct cpt_cn10k_res_s *)nixtx; + + plt_nix_dbg("Outbound error, mbuf %p, sa_index %u, compcode %x uc %x", + mbuf, sess_priv.sa_idx, res->compcode, res->uc_compcode); + + sess_priv.u64 = *rte_security_dynfield(mbuf); + + sa_base = dev->outb.sa_base; + sa = roc_nix_inl_ot_ipsec_outb_sa(sa_base, sess_priv.sa_idx); + priv = roc_nix_inl_ot_ipsec_outb_sa_sw_rsvd(sa); + + memset(&desc, 0, sizeof(desc)); + + switch (res->uc_compcode) { + case ROC_IE_OT_UCC_ERR_SA_OVERFLOW: + desc.subtype = RTE_ETH_EVENT_IPSEC_ESN_OVERFLOW; + break; + default: + plt_warn("Outbound error, mbuf %p, sa_index %u, " + "compcode %x uc %x", mbuf, sess_priv.sa_idx, + res->compcode, res->uc_compcode); + desc.subtype = RTE_ETH_EVENT_IPSEC_UNKNOWN; + break; + } + + desc.metadata = (uint64_t)priv->userdata; + rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_IPSEC, &desc); + rte_pktmbuf_free(mbuf); +} + +static int +cn10k_eth_sec_session_create(void *device, + struct rte_security_session_conf *conf, + struct rte_security_session *sess, + struct rte_mempool *mempool) +{ + struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)device; + struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + struct rte_security_ipsec_xform *ipsec; + struct cn10k_sec_sess_priv sess_priv; + struct rte_crypto_sym_xform *crypto; + struct cnxk_eth_sec_sess *eth_sec; + bool inbound, inl_dev; + int rc = 0; + + if (conf->action_type != RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL) + return -ENOTSUP; + + if (conf->protocol != RTE_SECURITY_PROTOCOL_IPSEC) + return -ENOTSUP; + + if (rte_security_dynfield_register() < 0) + return -ENOTSUP; + + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + roc_nix_inl_cb_register(cn10k_eth_sec_sso_work_cb, NULL); + + ipsec = &conf->ipsec; + crypto = conf->crypto_xform; + inbound = !!(ipsec->direction == RTE_SECURITY_IPSEC_SA_DIR_INGRESS); + inl_dev = !!dev->inb.inl_dev; + + /* Search if a session already exits */ + if (cnxk_eth_sec_sess_get_by_spi(dev, ipsec->spi, inbound)) { + plt_err("%s SA with SPI %u already in use", + inbound ? "Inbound" : "Outbound", ipsec->spi); + return -EEXIST; + } + + if (rte_mempool_get(mempool, (void **)ð_sec)) { + plt_err("Could not allocate security session private data"); + return -ENOMEM; + } + + memset(eth_sec, 0, sizeof(struct cnxk_eth_sec_sess)); + sess_priv.u64 = 0; + + /* Acquire lock on inline dev for inbound */ + if (inbound && inl_dev) + roc_nix_inl_dev_lock(); + + if (inbound) { + struct cn10k_inb_priv_data *inb_priv; + struct roc_ot_ipsec_inb_sa *inb_sa; + uintptr_t sa; + + PLT_STATIC_ASSERT(sizeof(struct cn10k_inb_priv_data) < + ROC_NIX_INL_OT_IPSEC_INB_SW_RSVD); + + /* Get Inbound SA from NIX_RX_IPSEC_SA_BASE */ + sa = roc_nix_inl_inb_sa_get(&dev->nix, inl_dev, ipsec->spi); + if (!sa && dev->inb.inl_dev) { + plt_err("Failed to create ingress sa, inline dev " + "not found or spi not in range"); + rc = -ENOTSUP; + goto mempool_put; + } else if (!sa) { + plt_err("Failed to create ingress sa"); + rc = -EFAULT; + goto mempool_put; + } + + inb_sa = (struct roc_ot_ipsec_inb_sa *)sa; + + /* Check if SA is already in use */ + if (inb_sa->w2.s.valid) { + plt_err("Inbound SA with SPI %u already in use", + ipsec->spi); + rc = -EBUSY; + goto mempool_put; + } + + memset(inb_sa, 0, sizeof(struct roc_ot_ipsec_inb_sa)); + + /* Fill inbound sa params */ + rc = cnxk_ot_ipsec_inb_sa_fill(inb_sa, ipsec, crypto); + if (rc) { + plt_err("Failed to init inbound sa, rc=%d", rc); + goto mempool_put; + } + + inb_priv = roc_nix_inl_ot_ipsec_inb_sa_sw_rsvd(inb_sa); + /* Back pointer to get eth_sec */ + inb_priv->eth_sec = eth_sec; + /* Save userdata in inb private area */ + inb_priv->userdata = conf->userdata; + + /* Save SA index/SPI in cookie for now */ + inb_sa->w1.s.cookie = rte_cpu_to_be_32(ipsec->spi); + + /* Prepare session priv */ + sess_priv.inb_sa = 1; + sess_priv.sa_idx = ipsec->spi; + + /* Pointer from eth_sec -> inb_sa */ + eth_sec->sa = inb_sa; + eth_sec->sess = sess; + eth_sec->sa_idx = ipsec->spi; + eth_sec->spi = ipsec->spi; + eth_sec->inl_dev = !!dev->inb.inl_dev; + eth_sec->inb = true; + + TAILQ_INSERT_TAIL(&dev->inb.list, eth_sec, entry); + dev->inb.nb_sess++; + } else { + struct cn10k_outb_priv_data *outb_priv; + struct roc_ot_ipsec_outb_sa *outb_sa; + struct cnxk_ipsec_outb_rlens *rlens; + uint64_t sa_base = dev->outb.sa_base; + uint32_t sa_idx; + + PLT_STATIC_ASSERT(sizeof(struct cn10k_outb_priv_data) < + ROC_NIX_INL_OT_IPSEC_OUTB_SW_RSVD); + + /* Alloc an sa index */ + rc = cnxk_eth_outb_sa_idx_get(dev, &sa_idx); + if (rc) + goto mempool_put; + + outb_sa = roc_nix_inl_ot_ipsec_outb_sa(sa_base, sa_idx); + outb_priv = roc_nix_inl_ot_ipsec_outb_sa_sw_rsvd(outb_sa); + rlens = &outb_priv->rlens; + + memset(outb_sa, 0, sizeof(struct roc_ot_ipsec_outb_sa)); + + /* Fill outbound sa params */ + rc = cnxk_ot_ipsec_outb_sa_fill(outb_sa, ipsec, crypto); + if (rc) { + plt_err("Failed to init outbound sa, rc=%d", rc); + rc |= cnxk_eth_outb_sa_idx_put(dev, sa_idx); + goto mempool_put; + } + + /* Save userdata */ + outb_priv->userdata = conf->userdata; + outb_priv->sa_idx = sa_idx; + outb_priv->eth_sec = eth_sec; + + /* Save rlen info */ + cnxk_ipsec_outb_rlens_get(rlens, ipsec, crypto); + + /* Prepare session priv */ + sess_priv.sa_idx = outb_priv->sa_idx; + sess_priv.roundup_byte = rlens->roundup_byte; + sess_priv.roundup_len = rlens->roundup_len; + sess_priv.partial_len = rlens->partial_len; + + /* Pointer from eth_sec -> outb_sa */ + eth_sec->sa = outb_sa; + eth_sec->sess = sess; + eth_sec->sa_idx = sa_idx; + eth_sec->spi = ipsec->spi; + + TAILQ_INSERT_TAIL(&dev->outb.list, eth_sec, entry); + dev->outb.nb_sess++; + } + + /* Sync session in context cache */ + roc_nix_inl_sa_sync(&dev->nix, eth_sec->sa, eth_sec->inb, + ROC_NIX_INL_SA_OP_RELOAD); + + if (inbound && inl_dev) + roc_nix_inl_dev_unlock(); + + plt_nix_dbg("Created %s session with spi=%u, sa_idx=%u inl_dev=%u", + inbound ? "inbound" : "outbound", eth_sec->spi, + eth_sec->sa_idx, eth_sec->inl_dev); + /* + * Update fast path info in priv area. + */ + set_sec_session_private_data(sess, (void *)sess_priv.u64); + + return 0; +mempool_put: + if (inbound && inl_dev) + roc_nix_inl_dev_unlock(); + rte_mempool_put(mempool, eth_sec); + return rc; +} + +static int +cn10k_eth_sec_session_destroy(void *device, struct rte_security_session *sess) +{ + struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)device; + struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + struct roc_ot_ipsec_inb_sa *inb_sa; + struct roc_ot_ipsec_outb_sa *outb_sa; + struct cnxk_eth_sec_sess *eth_sec; + struct rte_mempool *mp; + + eth_sec = cnxk_eth_sec_sess_get_by_sess(dev, sess); + if (!eth_sec) + return -ENOENT; + + if (eth_sec->inl_dev) + roc_nix_inl_dev_lock(); + + if (eth_sec->inb) { + inb_sa = eth_sec->sa; + /* Disable SA */ + inb_sa->w2.s.valid = 0; + + TAILQ_REMOVE(&dev->inb.list, eth_sec, entry); + dev->inb.nb_sess--; + } else { + outb_sa = eth_sec->sa; + /* Disable SA */ + outb_sa->w2.s.valid = 0; + + /* Release Outbound SA index */ + cnxk_eth_outb_sa_idx_put(dev, eth_sec->sa_idx); + TAILQ_REMOVE(&dev->outb.list, eth_sec, entry); + dev->outb.nb_sess--; + } + + /* Sync session in context cache */ + roc_nix_inl_sa_sync(&dev->nix, eth_sec->sa, eth_sec->inb, + ROC_NIX_INL_SA_OP_RELOAD); + + if (eth_sec->inl_dev) + roc_nix_inl_dev_unlock(); + + plt_nix_dbg("Destroyed %s session with spi=%u, sa_idx=%u, inl_dev=%u", + eth_sec->inb ? "inbound" : "outbound", eth_sec->spi, + eth_sec->sa_idx, eth_sec->inl_dev); + + /* Put eth_sec object back to pool */ + mp = rte_mempool_from_obj(eth_sec); + set_sec_session_private_data(sess, NULL); + rte_mempool_put(mp, eth_sec); + return 0; +} + +static const struct rte_security_capability * +cn10k_eth_sec_capabilities_get(void *device __rte_unused) +{ + return cn10k_eth_sec_capabilities; +} + +void +cn10k_eth_sec_ops_override(void) +{ + static int init_once; + + if (init_once) + return; + init_once = 1; + + /* Update platform specific ops */ + cnxk_eth_sec_ops.session_create = cn10k_eth_sec_session_create; + cnxk_eth_sec_ops.session_destroy = cn10k_eth_sec_session_destroy; + cnxk_eth_sec_ops.capabilities_get = cn10k_eth_sec_capabilities_get; +} diff --git a/drivers/net/cnxk/cn10k_rx.h b/drivers/net/cnxk/cn10k_rx.h index 68219b8..d27a231 100644 --- a/drivers/net/cnxk/cn10k_rx.h +++ b/drivers/net/cnxk/cn10k_rx.h @@ -16,6 +16,7 @@ #define NIX_RX_OFFLOAD_MARK_UPDATE_F BIT(3) #define NIX_RX_OFFLOAD_TSTAMP_F BIT(4) #define NIX_RX_OFFLOAD_VLAN_STRIP_F BIT(5) +#define NIX_RX_OFFLOAD_SECURITY_F BIT(6) /* Flags to control cqe_to_mbuf conversion function. * Defining it from backwards to denote its been diff --git a/drivers/net/cnxk/cn10k_tx.h b/drivers/net/cnxk/cn10k_tx.h index f75cae0..8577a7b 100644 --- a/drivers/net/cnxk/cn10k_tx.h +++ b/drivers/net/cnxk/cn10k_tx.h @@ -13,6 +13,7 @@ #define NIX_TX_OFFLOAD_MBUF_NOFF_F BIT(3) #define NIX_TX_OFFLOAD_TSO_F BIT(4) #define NIX_TX_OFFLOAD_TSTAMP_F BIT(5) +#define NIX_TX_OFFLOAD_SECURITY_F BIT(6) /* Flags to control xmit_prepare function. * Defining it from backwards to denote its been diff --git a/drivers/net/cnxk/meson.build b/drivers/net/cnxk/meson.build index 6cc30c3..d1d4b4e 100644 --- a/drivers/net/cnxk/meson.build +++ b/drivers/net/cnxk/meson.build @@ -37,6 +37,7 @@ sources += files( # CN10K sources += files( 'cn10k_ethdev.c', + 'cn10k_ethdev_sec.c', 'cn10k_rte_flow.c', 'cn10k_rx.c', 'cn10k_rx_mseg.c', diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index 74d16e4..5f0e817 100755 --- a/usertools/dpdk-devbind.py +++ b/usertools/dpdk-devbind.py @@ -49,6 +49,8 @@ 'SVendor': None, 'SDevice': None} cnxk_bphy_cgx = {'Class': '08', 'Vendor': '177d', 'Device': 'a059,a060', 'SVendor': None, 'SDevice': None} +cnxk_inl_dev = {'Class': '08', 'Vendor': '177d', 'Device': 'a0f0,a0f1', + 'SVendor': None, 'SDevice': None} intel_dlb = {'Class': '0b', 'Vendor': '8086', 'Device': '270b,2710,2714', 'SVendor': None, 'SDevice': None} @@ -73,9 +75,9 @@ mempool_devices = [cavium_fpa, octeontx2_npa] compress_devices = [cavium_zip] regex_devices = [octeontx2_ree] -misc_devices = [cnxk_bphy, cnxk_bphy_cgx, intel_ioat_bdw, intel_ioat_skx, intel_ioat_icx, intel_idxd_spr, - intel_ntb_skx, intel_ntb_icx, - octeontx2_dma] +misc_devices = [cnxk_bphy, cnxk_bphy_cgx, cnxk_inl_dev, intel_ioat_bdw, + intel_ioat_skx, intel_ioat_icx, intel_idxd_spr, intel_ntb_skx, + intel_ntb_icx, octeontx2_dma] # global dict ethernet devices present. Dictionary indexed by PCI address. # Each device within this is itself a dictionary of device properties From patchwork Thu Sep 2 02:14:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97743 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AC906A0C4C; Thu, 2 Sep 2021 04:18:40 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 898DE4115C; Thu, 2 Sep 2021 04:17:39 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 119A340E64 for ; Thu, 2 Sep 2021 04:17:37 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQ219011572 for ; Wed, 1 Sep 2021 19:17:37 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=O5xLnMsfttDeYf0KhonOCiqEsUOvjZjI5radC/DcdOc=; b=YIH98ST8cVU9vzwczqOHEDZ0vLZUeyiWNTCyiwmBbafUjsmIa7P33nhFSkFvp4OmwHPW +AyyiblmfN497YXyfNw5S9f7utWCHLPepugWKHyPAOPAJ5v5DZxERCzWF9GHG6s3lL5H XgTCyxbvYNpHVCcDeUPXaGcvAPJocmqta8Eov0SzTrkculfRakZELwzXWb2fYfysFWwv iTxQRVrYY4n0QJPXxnJzS0FXg9e4xgxxmnQS1alONx51Up5y3PzValCDkKNQW+lUkGiT G1IRtOnkY+WiezDSwPsuUAfQqYAJGiyWUuS8imw1dywTCmRa3x4HweXwqBDocluUQ2BV RA== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9htx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:37 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:34 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:34 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 39D0E3F7040; Wed, 1 Sep 2021 19:17:32 -0700 (PDT) From: Nithin Dabilpuram To: Pavan Nikhilesh , Shijith Thotton , Nithin Dabilpuram , "Kiran Kumar K" , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:55 +0530 Message-ID: <20210902021505.17607-18-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: xFHvLSwuE1CQJ7LSxcAi_E9MJ8hO37CM X-Proofpoint-GUID: xFHvLSwuE1CQJ7LSxcAi_E9MJ8hO37CM X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 17/27] net/cnxk: add cn9k Rx support for security offload X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add support to receive CPT processed packets on Rx. Signed-off-by: Nithin Dabilpuram --- drivers/event/cnxk/cn9k_eventdev.c | 153 ++++---- drivers/event/cnxk/cn9k_worker.h | 7 +- drivers/event/cnxk/cn9k_worker_deq.c | 2 +- drivers/event/cnxk/cn9k_worker_deq_burst.c | 2 +- drivers/event/cnxk/cn9k_worker_deq_ca.c | 2 +- drivers/event/cnxk/cn9k_worker_deq_tmo.c | 2 +- drivers/event/cnxk/cn9k_worker_dual_deq.c | 2 +- drivers/event/cnxk/cn9k_worker_dual_deq_burst.c | 2 +- drivers/event/cnxk/cn9k_worker_dual_deq_ca.c | 2 +- drivers/event/cnxk/cn9k_worker_dual_deq_tmo.c | 2 +- drivers/net/cnxk/cn9k_rx.c | 31 +- drivers/net/cnxk/cn9k_rx.h | 440 +++++++++++++++++++----- drivers/net/cnxk/cn9k_rx_mseg.c | 2 +- drivers/net/cnxk/cn9k_rx_vec.c | 2 +- drivers/net/cnxk/cn9k_rx_vec_mseg.c | 2 +- drivers/net/cnxk/cnxk_ethdev.h | 3 + 16 files changed, 461 insertions(+), 195 deletions(-) diff --git a/drivers/event/cnxk/cn9k_eventdev.c b/drivers/event/cnxk/cn9k_eventdev.c index 6601c44..e91234e 100644 --- a/drivers/event/cnxk/cn9k_eventdev.c +++ b/drivers/event/cnxk/cn9k_eventdev.c @@ -10,7 +10,8 @@ #define CN9K_DUAL_WS_PAIR_ID(x, id) (((x)*CN9K_DUAL_WS_NB_WS) + id) #define CN9K_SET_EVDEV_DEQ_OP(dev, deq_op, deq_ops) \ - deq_op = deq_ops[!!(dev->rx_offloads & NIX_RX_OFFLOAD_VLAN_STRIP_F)] \ + deq_op = deq_ops[!!(dev->rx_offloads & NIX_RX_OFFLOAD_SECURITY_F)] \ + [!!(dev->rx_offloads & NIX_RX_OFFLOAD_VLAN_STRIP_F)] \ [!!(dev->rx_offloads & NIX_RX_OFFLOAD_TSTAMP_F)] \ [!!(dev->rx_offloads & NIX_RX_OFFLOAD_MARK_UPDATE_F)] \ [!!(dev->rx_offloads & NIX_RX_OFFLOAD_CHECKSUM_F)] \ @@ -329,178 +330,184 @@ cn9k_sso_fp_fns_set(struct rte_eventdev *event_dev) { struct cnxk_sso_evdev *dev = cnxk_sso_pmd_priv(event_dev); /* Single WS modes */ - const event_dequeue_t sso_hws_deq[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_##name, + const event_dequeue_t sso_hws_deq[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_burst_t sso_hws_deq_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_burst_##name, + const event_dequeue_burst_t sso_hws_deq_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_tmo[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_tmo_##name, + const event_dequeue_t sso_hws_deq_tmo[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_tmo_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_burst_t sso_hws_deq_tmo_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_tmo_burst_##name, + const event_dequeue_burst_t + sso_hws_deq_tmo_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_tmo_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_ca[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_ca_##name, + const event_dequeue_t sso_hws_deq_ca[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_ca_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_burst_t sso_hws_deq_ca_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_ca_burst_##name, + const event_dequeue_burst_t + sso_hws_deq_ca_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_ca_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_seg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_seg_##name, + const event_dequeue_t sso_hws_deq_seg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_seg_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_burst_t sso_hws_deq_seg_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_seg_burst_##name, + const event_dequeue_burst_t + sso_hws_deq_seg_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_seg_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_tmo_seg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_tmo_seg_##name, + const event_dequeue_t sso_hws_deq_tmo_seg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_tmo_seg_##name, NIX_RX_FASTPATH_MODES #undef R }; const event_dequeue_burst_t - sso_hws_deq_tmo_seg_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_tmo_seg_burst_##name, + sso_hws_deq_tmo_seg_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_tmo_seg_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_ca_seg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_ca_seg_##name, + const event_dequeue_t sso_hws_deq_ca_seg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_ca_seg_##name, NIX_RX_FASTPATH_MODES #undef R }; const event_dequeue_burst_t - sso_hws_deq_ca_seg_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_ca_seg_burst_##name, + sso_hws_deq_ca_seg_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_deq_ca_seg_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; /* Dual WS modes */ - const event_dequeue_t sso_hws_dual_deq[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_##name, + const event_dequeue_t sso_hws_dual_deq[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_burst_t sso_hws_dual_deq_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_burst_##name, + const event_dequeue_burst_t + sso_hws_dual_deq_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_dual_deq_tmo[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_tmo_##name, + const event_dequeue_t sso_hws_dual_deq_tmo[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_tmo_##name, NIX_RX_FASTPATH_MODES #undef R }; const event_dequeue_burst_t - sso_hws_dual_deq_tmo_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_tmo_burst_##name, + sso_hws_dual_deq_tmo_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_tmo_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_dual_deq_ca[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_ca_##name, + const event_dequeue_t sso_hws_dual_deq_ca[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_ca_##name, NIX_RX_FASTPATH_MODES #undef R }; const event_dequeue_burst_t - sso_hws_dual_deq_ca_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_ca_burst_##name, + sso_hws_dual_deq_ca_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_ca_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_dual_deq_seg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_seg_##name, + const event_dequeue_t sso_hws_dual_deq_seg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_seg_##name, NIX_RX_FASTPATH_MODES #undef R }; const event_dequeue_burst_t - sso_hws_dual_deq_seg_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_seg_burst_##name, + sso_hws_dual_deq_seg_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_seg_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_dual_deq_tmo_seg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_tmo_seg_##name, + const event_dequeue_t sso_hws_dual_deq_tmo_seg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_tmo_seg_##name, NIX_RX_FASTPATH_MODES #undef R }; const event_dequeue_burst_t - sso_hws_dual_deq_tmo_seg_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_tmo_seg_burst_##name, + sso_hws_dual_deq_tmo_seg_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = \ + cn9k_sso_hws_dual_deq_tmo_seg_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_dual_deq_ca_seg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_ca_seg_##name, + const event_dequeue_t sso_hws_dual_deq_ca_seg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_ca_seg_##name, NIX_RX_FASTPATH_MODES #undef R }; const event_dequeue_burst_t - sso_hws_dual_deq_ca_seg_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_deq_ca_seg_burst_##name, + sso_hws_dual_deq_ca_seg_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = \ + cn9k_sso_hws_dual_deq_ca_seg_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; diff --git a/drivers/event/cnxk/cn9k_worker.h b/drivers/event/cnxk/cn9k_worker.h index 3e8f214..f1d2e47 100644 --- a/drivers/event/cnxk/cn9k_worker.h +++ b/drivers/event/cnxk/cn9k_worker.h @@ -5,6 +5,9 @@ #ifndef __CN9K_WORKER_H__ #define __CN9K_WORKER_H__ +#include +#include + #include "cnxk_ethdev.h" #include "cnxk_eventdev.h" #include "cnxk_worker.h" @@ -380,7 +383,7 @@ uint16_t __rte_hot cn9k_sso_hws_ca_enq(void *port, struct rte_event ev[], uint16_t __rte_hot cn9k_sso_hws_dual_ca_enq(void *port, struct rte_event ev[], uint16_t nb_events); -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn9k_sso_hws_deq_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks); \ uint16_t __rte_hot cn9k_sso_hws_deq_burst_##name( \ @@ -415,7 +418,7 @@ uint16_t __rte_hot cn9k_sso_hws_dual_ca_enq(void *port, struct rte_event ev[], NIX_RX_FASTPATH_MODES #undef R -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn9k_sso_hws_dual_deq_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks); \ uint16_t __rte_hot cn9k_sso_hws_dual_deq_burst_##name( \ diff --git a/drivers/event/cnxk/cn9k_worker_deq.c b/drivers/event/cnxk/cn9k_worker_deq.c index 51ccaf4..d65c72a 100644 --- a/drivers/event/cnxk/cn9k_worker_deq.c +++ b/drivers/event/cnxk/cn9k_worker_deq.c @@ -6,7 +6,7 @@ #include "cnxk_eventdev.h" #include "cnxk_worker.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn9k_sso_hws_deq_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks) \ { \ diff --git a/drivers/event/cnxk/cn9k_worker_deq_burst.c b/drivers/event/cnxk/cn9k_worker_deq_burst.c index 4e28014..42dc59b 100644 --- a/drivers/event/cnxk/cn9k_worker_deq_burst.c +++ b/drivers/event/cnxk/cn9k_worker_deq_burst.c @@ -6,7 +6,7 @@ #include "cnxk_eventdev.h" #include "cnxk_worker.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn9k_sso_hws_deq_burst_##name( \ void *port, struct rte_event ev[], uint16_t nb_events, \ uint64_t timeout_ticks) \ diff --git a/drivers/event/cnxk/cn9k_worker_deq_ca.c b/drivers/event/cnxk/cn9k_worker_deq_ca.c index dde8288..6c5325f 100644 --- a/drivers/event/cnxk/cn9k_worker_deq_ca.c +++ b/drivers/event/cnxk/cn9k_worker_deq_ca.c @@ -6,7 +6,7 @@ #include "cnxk_eventdev.h" #include "cnxk_worker.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn9k_sso_hws_deq_ca_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks) \ { \ diff --git a/drivers/event/cnxk/cn9k_worker_deq_tmo.c b/drivers/event/cnxk/cn9k_worker_deq_tmo.c index 9713d1e..b41a590 100644 --- a/drivers/event/cnxk/cn9k_worker_deq_tmo.c +++ b/drivers/event/cnxk/cn9k_worker_deq_tmo.c @@ -6,7 +6,7 @@ #include "cnxk_eventdev.h" #include "cnxk_worker.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn9k_sso_hws_deq_tmo_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks) \ { \ diff --git a/drivers/event/cnxk/cn9k_worker_dual_deq.c b/drivers/event/cnxk/cn9k_worker_dual_deq.c index 709fa2d..440b66e 100644 --- a/drivers/event/cnxk/cn9k_worker_dual_deq.c +++ b/drivers/event/cnxk/cn9k_worker_dual_deq.c @@ -6,7 +6,7 @@ #include "cnxk_eventdev.h" #include "cnxk_worker.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn9k_sso_hws_dual_deq_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks) \ { \ diff --git a/drivers/event/cnxk/cn9k_worker_dual_deq_burst.c b/drivers/event/cnxk/cn9k_worker_dual_deq_burst.c index d50e1cf..4d913f9 100644 --- a/drivers/event/cnxk/cn9k_worker_dual_deq_burst.c +++ b/drivers/event/cnxk/cn9k_worker_dual_deq_burst.c @@ -6,7 +6,7 @@ #include "cnxk_eventdev.h" #include "cnxk_worker.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn9k_sso_hws_dual_deq_burst_##name( \ void *port, struct rte_event ev[], uint16_t nb_events, \ uint64_t timeout_ticks) \ diff --git a/drivers/event/cnxk/cn9k_worker_dual_deq_ca.c b/drivers/event/cnxk/cn9k_worker_dual_deq_ca.c index 26cc60f..74116a9 100644 --- a/drivers/event/cnxk/cn9k_worker_dual_deq_ca.c +++ b/drivers/event/cnxk/cn9k_worker_dual_deq_ca.c @@ -6,7 +6,7 @@ #include "cnxk_eventdev.h" #include "cnxk_worker.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn9k_sso_hws_dual_deq_ca_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks) \ { \ diff --git a/drivers/event/cnxk/cn9k_worker_dual_deq_tmo.c b/drivers/event/cnxk/cn9k_worker_dual_deq_tmo.c index a0508fd..78a4b3d 100644 --- a/drivers/event/cnxk/cn9k_worker_dual_deq_tmo.c +++ b/drivers/event/cnxk/cn9k_worker_dual_deq_tmo.c @@ -6,7 +6,7 @@ #include "cnxk_eventdev.h" #include "cnxk_worker.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn9k_sso_hws_dual_deq_tmo_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks) \ { \ diff --git a/drivers/net/cnxk/cn9k_rx.c b/drivers/net/cnxk/cn9k_rx.c index 7d9f1bd..5c4387e 100644 --- a/drivers/net/cnxk/cn9k_rx.c +++ b/drivers/net/cnxk/cn9k_rx.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_rx.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_recv_pkts_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ @@ -17,12 +17,13 @@ NIX_RX_FASTPATH_MODES static inline void pick_rx_func(struct rte_eth_dev *eth_dev, - const eth_rx_burst_t rx_burst[2][2][2][2][2][2]) + const eth_rx_burst_t rx_burst[2][2][2][2][2][2][2]) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); /* [TSP] [MARK] [VLAN] [CKSUM] [PTYPE] [RSS] */ eth_dev->rx_pkt_burst = rx_burst + [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_SECURITY_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_VLAN_STRIP_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_TSTAMP_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_MARK_UPDATE_F)] @@ -38,33 +39,33 @@ cn9k_eth_set_rx_function(struct rte_eth_dev *eth_dev) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - const eth_rx_burst_t nix_eth_rx_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_nix_recv_pkts_##name, + const eth_rx_burst_t nix_eth_rx_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_nix_recv_pkts_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_burst_mseg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_nix_recv_pkts_mseg_##name, + const eth_rx_burst_t nix_eth_rx_burst_mseg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_nix_recv_pkts_mseg_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_vec_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_nix_recv_pkts_vec_##name, + const eth_rx_burst_t nix_eth_rx_vec_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_nix_recv_pkts_vec_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_vec_burst_mseg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_nix_recv_pkts_vec_mseg_##name, + const eth_rx_burst_t nix_eth_rx_vec_burst_mseg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_nix_recv_pkts_vec_mseg_##name, NIX_RX_FASTPATH_MODES #undef R @@ -73,7 +74,7 @@ cn9k_eth_set_rx_function(struct rte_eth_dev *eth_dev) /* Copy multi seg version with no offload for tear down sequence */ if (rte_eal_process_type() == RTE_PROC_PRIMARY) dev->rx_pkt_burst_no_offload = - nix_eth_rx_burst_mseg[0][0][0][0][0][0]; + nix_eth_rx_burst_mseg[0][0][0][0][0][0][0]; if (dev->scalar_ena) { if (dev->rx_offloads & DEV_RX_OFFLOAD_SCATTER) diff --git a/drivers/net/cnxk/cn9k_rx.h b/drivers/net/cnxk/cn9k_rx.h index 59545af..bdedeab 100644 --- a/drivers/net/cnxk/cn9k_rx.h +++ b/drivers/net/cnxk/cn9k_rx.h @@ -166,24 +166,104 @@ nix_cqe_xtract_mseg(const union nix_rx_parse_u *rx, struct rte_mbuf *mbuf, mbuf->next = NULL; } +static __rte_always_inline uint64_t +nix_rx_sec_mbuf_update(const struct nix_cqe_hdr_s *cq, struct rte_mbuf *m, + uintptr_t sa_base, uint64_t *rearm_val, uint16_t *len) +{ + uintptr_t res_sg0 = ((uintptr_t)cq + ROC_ONF_IPSEC_INB_RES_OFF - 8); + const union nix_rx_parse_u *rx = + (const union nix_rx_parse_u *)((const uint64_t *)cq + 1); + struct cn9k_inb_priv_data *sa_priv; + struct roc_onf_ipsec_inb_sa *sa; + uint8_t lcptr = rx->lcptr; + struct rte_ipv4_hdr *ipv4; + uint16_t data_off, res; + uint32_t spi_mask; + uint32_t spi; + uintptr_t data; + __uint128_t dw; + uint8_t sa_w; + + res = *(uint64_t *)(res_sg0 + 8); + data_off = *rearm_val & (BIT_ULL(16) - 1); + data = (uintptr_t)m->buf_addr; + data += data_off; + + rte_prefetch0((void *)data); + + if (unlikely(res != (CPT_COMP_GOOD | ROC_IE_ONF_UCC_SUCCESS << 8))) + return PKT_RX_SEC_OFFLOAD | PKT_RX_SEC_OFFLOAD_FAILED; + + data += lcptr; + /* 20 bits of tag would have the SPI */ + spi = cq->tag & CNXK_ETHDEV_SPI_TAG_MASK; + + /* Get SA */ + sa_w = sa_base & (ROC_NIX_INL_SA_BASE_ALIGN - 1); + sa_base &= ~(ROC_NIX_INL_SA_BASE_ALIGN - 1); + spi_mask = (1ULL << sa_w) - 1; + sa = roc_nix_inl_onf_ipsec_inb_sa(sa_base, spi & spi_mask); + + /* Update dynamic field with userdata */ + sa_priv = roc_nix_inl_onf_ipsec_inb_sa_sw_rsvd(sa); + dw = *(__uint128_t *)sa_priv; + *rte_security_dynfield(m) = (uint64_t)dw; + + /* Get total length from IPv4 header. We can assume only IPv4 */ + ipv4 = (struct rte_ipv4_hdr *)(data + ROC_ONF_IPSEC_INB_SPI_SEQ_SZ + + ROC_ONF_IPSEC_INB_MAX_L2_SZ); + + /* Update data offset */ + data_off += (ROC_ONF_IPSEC_INB_SPI_SEQ_SZ + + ROC_ONF_IPSEC_INB_MAX_L2_SZ); + *rearm_val = *rearm_val & ~(BIT_ULL(16) - 1); + *rearm_val |= data_off; + + *len = rte_be_to_cpu_16(ipv4->total_length) + lcptr; + return PKT_RX_SEC_OFFLOAD; +} + static __rte_always_inline void cn9k_nix_cqe_to_mbuf(const struct nix_cqe_hdr_s *cq, const uint32_t tag, struct rte_mbuf *mbuf, const void *lookup_mem, - const uint64_t val, const uint16_t flag) + uint64_t val, const uint16_t flag) { const union nix_rx_parse_u *rx = (const union nix_rx_parse_u *)((const uint64_t *)cq + 1); - const uint16_t len = rx->cn9k.pkt_lenm1 + 1; + uint16_t len = rx->cn9k.pkt_lenm1 + 1; const uint64_t w1 = *(const uint64_t *)rx; + uint32_t packet_type; uint64_t ol_flags = 0; /* Mark mempool obj as "get" as it is alloc'ed by NIX */ __mempool_check_cookies(mbuf->pool, (void **)&mbuf, 1, 1); if (flag & NIX_RX_OFFLOAD_PTYPE_F) - mbuf->packet_type = nix_ptype_get(lookup_mem, w1); + packet_type = nix_ptype_get(lookup_mem, w1); else - mbuf->packet_type = 0; + packet_type = 0; + + if ((flag & NIX_RX_OFFLOAD_SECURITY_F) && + cq->cqe_type == NIX_XQE_TYPE_RX_IPSECH) { + uint16_t port = val >> 48; + uintptr_t sa_base; + + /* Get SA Base from lookup mem */ + sa_base = cnxk_nix_sa_base_get(port, lookup_mem); + + ol_flags |= nix_rx_sec_mbuf_update(cq, mbuf, sa_base, &val, + &len); + + /* Only Tunnel inner IPv4 is supported */ + packet_type = (packet_type & + ~(RTE_PTYPE_L3_MASK | RTE_PTYPE_TUNNEL_MASK)); + packet_type |= RTE_PTYPE_L3_IPV4_EXT_UNKNOWN; + mbuf->packet_type = packet_type; + goto skip_parse; + } + + if (flag & NIX_RX_OFFLOAD_PTYPE_F) + mbuf->packet_type = packet_type; if (flag & NIX_RX_OFFLOAD_RSS_F) { mbuf->hash.rss = tag; @@ -193,6 +273,7 @@ cn9k_nix_cqe_to_mbuf(const struct nix_cqe_hdr_s *cq, const uint32_t tag, if (flag & NIX_RX_OFFLOAD_CHECKSUM_F) ol_flags |= nix_rx_olflags_get(lookup_mem, w1); +skip_parse: if (flag & NIX_RX_OFFLOAD_VLAN_STRIP_F) { if (rx->cn9k.vtag0_gone) { ol_flags |= PKT_RX_VLAN | PKT_RX_VLAN_STRIPPED; @@ -208,11 +289,12 @@ cn9k_nix_cqe_to_mbuf(const struct nix_cqe_hdr_s *cq, const uint32_t tag, ol_flags = nix_update_match_id(rx->cn9k.match_id, ol_flags, mbuf); - mbuf->ol_flags = ol_flags; mbuf->pkt_len = len; mbuf->data_len = len; *(uint64_t *)(&mbuf->rearm_data) = val; + mbuf->ol_flags = ol_flags; + if (flag & NIX_RX_MULTI_SEG_F) nix_cqe_xtract_mseg(rx, mbuf, val, flag); else @@ -670,98 +752,268 @@ cn9k_nix_recv_pkts_vector(void *rx_queue, struct rte_mbuf **rx_pkts, #define MARK_F NIX_RX_OFFLOAD_MARK_UPDATE_F #define TS_F NIX_RX_OFFLOAD_TSTAMP_F #define RX_VLAN_F NIX_RX_OFFLOAD_VLAN_STRIP_F +#define R_SEC_F NIX_RX_OFFLOAD_SECURITY_F -/* [RX_VLAN_F] [TS] [MARK] [CKSUM] [PTYPE] [RSS] */ +/* [R_SEC_F] [RX_VLAN_F] [TS] [MARK] [CKSUM] [PTYPE] [RSS] */ #define NIX_RX_FASTPATH_MODES \ -R(no_offload, 0, 0, 0, 0, 0, 0, NIX_RX_OFFLOAD_NONE) \ -R(rss, 0, 0, 0, 0, 0, 1, RSS_F) \ -R(ptype, 0, 0, 0, 0, 1, 0, PTYPE_F) \ -R(ptype_rss, 0, 0, 0, 0, 1, 1, PTYPE_F | RSS_F) \ -R(cksum, 0, 0, 0, 1, 0, 0, CKSUM_F) \ -R(cksum_rss, 0, 0, 0, 1, 0, 1, CKSUM_F | RSS_F) \ -R(cksum_ptype, 0, 0, 0, 1, 1, 0, CKSUM_F | PTYPE_F) \ -R(cksum_ptype_rss, 0, 0, 0, 1, 1, 1, CKSUM_F | PTYPE_F | RSS_F) \ -R(mark, 0, 0, 1, 0, 0, 0, MARK_F) \ -R(mark_rss, 0, 0, 1, 0, 0, 1, MARK_F | RSS_F) \ -R(mark_ptype, 0, 0, 1, 0, 1, 0, MARK_F | PTYPE_F) \ -R(mark_ptype_rss, 0, 0, 1, 0, 1, 1, MARK_F | PTYPE_F | RSS_F) \ -R(mark_cksum, 0, 0, 1, 1, 0, 0, MARK_F | CKSUM_F) \ -R(mark_cksum_rss, 0, 0, 1, 1, 0, 1, MARK_F | CKSUM_F | RSS_F) \ -R(mark_cksum_ptype, 0, 0, 1, 1, 1, 0, MARK_F | CKSUM_F | PTYPE_F) \ -R(mark_cksum_ptype_rss, 0, 0, 1, 1, 1, 1, \ - MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(ts, 0, 1, 0, 0, 0, 0, TS_F) \ -R(ts_rss, 0, 1, 0, 0, 0, 1, TS_F | RSS_F) \ -R(ts_ptype, 0, 1, 0, 0, 1, 0, TS_F | PTYPE_F) \ -R(ts_ptype_rss, 0, 1, 0, 0, 1, 1, TS_F | PTYPE_F | RSS_F) \ -R(ts_cksum, 0, 1, 0, 1, 0, 0, TS_F | CKSUM_F) \ -R(ts_cksum_rss, 0, 1, 0, 1, 0, 1, TS_F | CKSUM_F | RSS_F) \ -R(ts_cksum_ptype, 0, 1, 0, 1, 1, 0, TS_F | CKSUM_F | PTYPE_F) \ -R(ts_cksum_ptype_rss, 0, 1, 0, 1, 1, 1, \ - TS_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(ts_mark, 0, 1, 1, 0, 0, 0, TS_F | MARK_F) \ -R(ts_mark_rss, 0, 1, 1, 0, 0, 1, TS_F | MARK_F | RSS_F) \ -R(ts_mark_ptype, 0, 1, 1, 0, 1, 0, TS_F | MARK_F | PTYPE_F) \ -R(ts_mark_ptype_rss, 0, 1, 1, 0, 1, 1, \ - TS_F | MARK_F | PTYPE_F | RSS_F) \ -R(ts_mark_cksum, 0, 1, 1, 1, 0, 0, TS_F | MARK_F | CKSUM_F) \ -R(ts_mark_cksum_rss, 0, 1, 1, 1, 0, 1, \ - TS_F | MARK_F | CKSUM_F | RSS_F) \ -R(ts_mark_cksum_ptype, 0, 1, 1, 1, 1, 0, \ - TS_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(ts_mark_cksum_ptype_rss, 0, 1, 1, 1, 1, 1, \ - TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(vlan, 1, 0, 0, 0, 0, 0, RX_VLAN_F) \ -R(vlan_rss, 1, 0, 0, 0, 0, 1, RX_VLAN_F | RSS_F) \ -R(vlan_ptype, 1, 0, 0, 0, 1, 0, RX_VLAN_F | PTYPE_F) \ -R(vlan_ptype_rss, 1, 0, 0, 0, 1, 1, RX_VLAN_F | PTYPE_F | RSS_F) \ -R(vlan_cksum, 1, 0, 0, 1, 0, 0, RX_VLAN_F | CKSUM_F) \ -R(vlan_cksum_rss, 1, 0, 0, 1, 0, 1, RX_VLAN_F | CKSUM_F | RSS_F) \ -R(vlan_cksum_ptype, 1, 0, 0, 1, 1, 0, \ - RX_VLAN_F | CKSUM_F | PTYPE_F) \ -R(vlan_cksum_ptype_rss, 1, 0, 0, 1, 1, 1, \ - RX_VLAN_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(vlan_mark, 1, 0, 1, 0, 0, 0, RX_VLAN_F | MARK_F) \ -R(vlan_mark_rss, 1, 0, 1, 0, 0, 1, RX_VLAN_F | MARK_F | RSS_F) \ -R(vlan_mark_ptype, 1, 0, 1, 0, 1, 0, RX_VLAN_F | MARK_F | PTYPE_F)\ -R(vlan_mark_ptype_rss, 1, 0, 1, 0, 1, 1, \ - RX_VLAN_F | MARK_F | PTYPE_F | RSS_F) \ -R(vlan_mark_cksum, 1, 0, 1, 1, 0, 0, RX_VLAN_F | MARK_F | CKSUM_F)\ -R(vlan_mark_cksum_rss, 1, 0, 1, 1, 0, 1, \ - RX_VLAN_F | MARK_F | CKSUM_F | RSS_F) \ -R(vlan_mark_cksum_ptype, 1, 0, 1, 1, 1, 0, \ - RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(vlan_mark_cksum_ptype_rss, 1, 0, 1, 1, 1, 1, \ - RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(vlan_ts, 1, 1, 0, 0, 0, 0, RX_VLAN_F | TS_F) \ -R(vlan_ts_rss, 1, 1, 0, 0, 0, 1, RX_VLAN_F | TS_F | RSS_F) \ -R(vlan_ts_ptype, 1, 1, 0, 0, 1, 0, RX_VLAN_F | TS_F | PTYPE_F) \ -R(vlan_ts_ptype_rss, 1, 1, 0, 0, 1, 1, \ - RX_VLAN_F | TS_F | PTYPE_F | RSS_F) \ -R(vlan_ts_cksum, 1, 1, 0, 1, 0, 0, RX_VLAN_F | TS_F | CKSUM_F) \ -R(vlan_ts_cksum_rss, 1, 1, 0, 1, 0, 1, \ - RX_VLAN_F | TS_F | CKSUM_F | RSS_F) \ -R(vlan_ts_cksum_ptype, 1, 1, 0, 1, 1, 0, \ - RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F) \ -R(vlan_ts_cksum_ptype_rss, 1, 1, 0, 1, 1, 1, \ - RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(vlan_ts_mark, 1, 1, 1, 0, 0, 0, RX_VLAN_F | TS_F | MARK_F) \ -R(vlan_ts_mark_rss, 1, 1, 1, 0, 0, 1, \ - RX_VLAN_F | TS_F | MARK_F | RSS_F) \ -R(vlan_ts_mark_ptype, 1, 1, 1, 0, 1, 0, \ - RX_VLAN_F | TS_F | MARK_F | PTYPE_F) \ -R(vlan_ts_mark_ptype_rss, 1, 1, 1, 0, 1, 1, \ - RX_VLAN_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ -R(vlan_ts_mark_cksum, 1, 1, 1, 1, 0, 0, \ - RX_VLAN_F | TS_F | MARK_F | CKSUM_F) \ -R(vlan_ts_mark_cksum_rss, 1, 1, 1, 1, 0, 1, \ - RX_VLAN_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ -R(vlan_ts_mark_cksum_ptype, 1, 1, 1, 1, 1, 0, \ - RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(vlan_ts_mark_cksum_ptype_rss, 1, 1, 1, 1, 1, 1, \ - RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) +R(no_offload, 0, 0, 0, 0, 0, 0, 0, \ + NIX_RX_OFFLOAD_NONE) \ +R(rss, 0, 0, 0, 0, 0, 0, 1, \ + RSS_F) \ +R(ptype, 0, 0, 0, 0, 0, 1, 0, \ + PTYPE_F) \ +R(ptype_rss, 0, 0, 0, 0, 0, 1, 1, \ + PTYPE_F | RSS_F) \ +R(cksum, 0, 0, 0, 0, 1, 0, 0, \ + CKSUM_F) \ +R(cksum_rss, 0, 0, 0, 0, 1, 0, 1, \ + CKSUM_F | RSS_F) \ +R(cksum_ptype, 0, 0, 0, 0, 1, 1, 0, \ + CKSUM_F | PTYPE_F) \ +R(cksum_ptype_rss, 0, 0, 0, 0, 1, 1, 1, \ + CKSUM_F | PTYPE_F | RSS_F) \ +R(mark, 0, 0, 0, 1, 0, 0, 0, \ + MARK_F) \ +R(mark_rss, 0, 0, 0, 1, 0, 0, 1, \ + MARK_F | RSS_F) \ +R(mark_ptype, 0, 0, 0, 1, 0, 1, 0, \ + MARK_F | PTYPE_F) \ +R(mark_ptype_rss, 0, 0, 0, 1, 0, 1, 1, \ + MARK_F | PTYPE_F | RSS_F) \ +R(mark_cksum, 0, 0, 0, 1, 1, 0, 0, \ + MARK_F | CKSUM_F) \ +R(mark_cksum_rss, 0, 0, 0, 1, 1, 0, 1, \ + MARK_F | CKSUM_F | RSS_F) \ +R(mark_cksum_ptype, 0, 0, 0, 1, 1, 1, 0, \ + MARK_F | CKSUM_F | PTYPE_F) \ +R(mark_cksum_ptype_rss, 0, 0, 0, 1, 1, 1, 1, \ + MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(ts, 0, 0, 1, 0, 0, 0, 0, \ + TS_F) \ +R(ts_rss, 0, 0, 1, 0, 0, 0, 1, \ + TS_F | RSS_F) \ +R(ts_ptype, 0, 0, 1, 0, 0, 1, 0, \ + TS_F | PTYPE_F) \ +R(ts_ptype_rss, 0, 0, 1, 0, 0, 1, 1, \ + TS_F | PTYPE_F | RSS_F) \ +R(ts_cksum, 0, 0, 1, 0, 1, 0, 0, \ + TS_F | CKSUM_F) \ +R(ts_cksum_rss, 0, 0, 1, 0, 1, 0, 1, \ + TS_F | CKSUM_F | RSS_F) \ +R(ts_cksum_ptype, 0, 0, 1, 0, 1, 1, 0, \ + TS_F | CKSUM_F | PTYPE_F) \ +R(ts_cksum_ptype_rss, 0, 0, 1, 0, 1, 1, 1, \ + TS_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(ts_mark, 0, 0, 1, 1, 0, 0, 0, \ + TS_F | MARK_F) \ +R(ts_mark_rss, 0, 0, 1, 1, 0, 0, 1, \ + TS_F | MARK_F | RSS_F) \ +R(ts_mark_ptype, 0, 0, 1, 1, 0, 1, 0, \ + TS_F | MARK_F | PTYPE_F) \ +R(ts_mark_ptype_rss, 0, 0, 1, 1, 0, 1, 1, \ + TS_F | MARK_F | PTYPE_F | RSS_F) \ +R(ts_mark_cksum, 0, 0, 1, 1, 1, 0, 0, \ + TS_F | MARK_F | CKSUM_F) \ +R(ts_mark_cksum_rss, 0, 0, 1, 1, 1, 0, 1, \ + TS_F | MARK_F | CKSUM_F | RSS_F) \ +R(ts_mark_cksum_ptype, 0, 0, 1, 1, 1, 1, 0, \ + TS_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(ts_mark_cksum_ptype_rss, 0, 0, 1, 1, 1, 1, 1, \ + TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(vlan, 0, 1, 0, 0, 0, 0, 0, \ + RX_VLAN_F) \ +R(vlan_rss, 0, 1, 0, 0, 0, 0, 1, \ + RX_VLAN_F | RSS_F) \ +R(vlan_ptype, 0, 1, 0, 0, 0, 1, 0, \ + RX_VLAN_F | PTYPE_F) \ +R(vlan_ptype_rss, 0, 1, 0, 0, 0, 1, 1, \ + RX_VLAN_F | PTYPE_F | RSS_F) \ +R(vlan_cksum, 0, 1, 0, 0, 1, 0, 0, \ + RX_VLAN_F | CKSUM_F) \ +R(vlan_cksum_rss, 0, 1, 0, 0, 1, 0, 1, \ + RX_VLAN_F | CKSUM_F | RSS_F) \ +R(vlan_cksum_ptype, 0, 1, 0, 0, 1, 1, 0, \ + RX_VLAN_F | CKSUM_F | PTYPE_F) \ +R(vlan_cksum_ptype_rss, 0, 1, 0, 0, 1, 1, 1, \ + RX_VLAN_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(vlan_mark, 0, 1, 0, 1, 0, 0, 0, \ + RX_VLAN_F | MARK_F) \ +R(vlan_mark_rss, 0, 1, 0, 1, 0, 0, 1, \ + RX_VLAN_F | MARK_F | RSS_F) \ +R(vlan_mark_ptype, 0, 1, 0, 1, 0, 1, 0, \ + RX_VLAN_F | MARK_F | PTYPE_F) \ +R(vlan_mark_ptype_rss, 0, 1, 0, 1, 0, 1, 1, \ + RX_VLAN_F | MARK_F | PTYPE_F | RSS_F) \ +R(vlan_mark_cksum, 0, 1, 0, 1, 1, 0, 0, \ + RX_VLAN_F | MARK_F | CKSUM_F) \ +R(vlan_mark_cksum_rss, 0, 1, 0, 1, 1, 0, 1, \ + RX_VLAN_F | MARK_F | CKSUM_F | RSS_F) \ +R(vlan_mark_cksum_ptype, 0, 1, 0, 1, 1, 1, 0, \ + RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(vlan_mark_cksum_ptype_rss, 0, 1, 0, 1, 1, 1, 1, \ + RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(vlan_ts, 0, 1, 1, 0, 0, 0, 0, \ + RX_VLAN_F | TS_F) \ +R(vlan_ts_rss, 0, 1, 1, 0, 0, 0, 1, \ + RX_VLAN_F | TS_F | RSS_F) \ +R(vlan_ts_ptype, 0, 1, 1, 0, 0, 1, 0, \ + RX_VLAN_F | TS_F | PTYPE_F) \ +R(vlan_ts_ptype_rss, 0, 1, 1, 0, 0, 1, 1, \ + RX_VLAN_F | TS_F | PTYPE_F | RSS_F) \ +R(vlan_ts_cksum, 0, 1, 1, 0, 1, 0, 0, \ + RX_VLAN_F | TS_F | CKSUM_F) \ +R(vlan_ts_cksum_rss, 0, 1, 1, 0, 1, 0, 1, \ + RX_VLAN_F | TS_F | CKSUM_F | RSS_F) \ +R(vlan_ts_cksum_ptype, 0, 1, 1, 0, 1, 1, 0, \ + RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F) \ +R(vlan_ts_cksum_ptype_rss, 0, 1, 1, 0, 1, 1, 1, \ + RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(vlan_ts_mark, 0, 1, 1, 1, 0, 0, 0, \ + RX_VLAN_F | TS_F | MARK_F) \ +R(vlan_ts_mark_rss, 0, 1, 1, 1, 0, 0, 1, \ + RX_VLAN_F | TS_F | MARK_F | RSS_F) \ +R(vlan_ts_mark_ptype, 0, 1, 1, 1, 0, 1, 0, \ + RX_VLAN_F | TS_F | MARK_F | PTYPE_F) \ +R(vlan_ts_mark_ptype_rss, 0, 1, 1, 1, 0, 1, 1, \ + RX_VLAN_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ +R(vlan_ts_mark_cksum, 0, 1, 1, 1, 1, 0, 0, \ + RX_VLAN_F | TS_F | MARK_F | CKSUM_F) \ +R(vlan_ts_mark_cksum_rss, 0, 1, 1, 1, 1, 0, 1, \ + RX_VLAN_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ +R(vlan_ts_mark_cksum_ptype, 0, 1, 1, 1, 1, 1, 0, \ + RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(vlan_ts_mark_cksum_ptype_rss, 0, 1, 1, 1, 1, 1, 1, \ + RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec, 1, 0, 0, 0, 0, 0, 0, \ + R_SEC_F) \ +R(sec_rss, 1, 0, 0, 0, 0, 0, 1, \ + RSS_F) \ +R(sec_ptype, 1, 0, 0, 0, 0, 1, 0, \ + R_SEC_F | PTYPE_F) \ +R(sec_ptype_rss, 1, 0, 0, 0, 0, 1, 1, \ + R_SEC_F | PTYPE_F | RSS_F) \ +R(sec_cksum, 1, 0, 0, 0, 1, 0, 0, \ + R_SEC_F | CKSUM_F) \ +R(sec_cksum_rss, 1, 0, 0, 0, 1, 0, 1, \ + R_SEC_F | CKSUM_F | RSS_F) \ +R(sec_cksum_ptype, 1, 0, 0, 0, 1, 1, 0, \ + R_SEC_F | CKSUM_F | PTYPE_F) \ +R(sec_cksum_ptype_rss, 1, 0, 0, 0, 1, 1, 1, \ + R_SEC_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_mark, 1, 0, 0, 1, 0, 0, 0, \ + R_SEC_F | MARK_F) \ +R(sec_mark_rss, 1, 0, 0, 1, 0, 0, 1, \ + R_SEC_F | MARK_F | RSS_F) \ +R(sec_mark_ptype, 1, 0, 0, 1, 0, 1, 0, \ + R_SEC_F | MARK_F | PTYPE_F) \ +R(sec_mark_ptype_rss, 1, 0, 0, 1, 0, 1, 1, \ + R_SEC_F | MARK_F | PTYPE_F | RSS_F) \ +R(sec_mark_cksum, 1, 0, 0, 1, 1, 0, 0, \ + R_SEC_F | MARK_F | CKSUM_F) \ +R(sec_mark_cksum_rss, 1, 0, 0, 1, 1, 0, 1, \ + R_SEC_F | MARK_F | CKSUM_F | RSS_F) \ +R(sec_mark_cksum_ptype, 1, 0, 0, 1, 1, 1, 0, \ + R_SEC_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(sec_mark_cksum_ptype_rss, 1, 0, 0, 1, 1, 1, 1, \ + R_SEC_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_ts, 1, 0, 1, 0, 0, 0, 0, \ + R_SEC_F | TS_F) \ +R(sec_ts_rss, 1, 0, 1, 0, 0, 0, 1, \ + R_SEC_F | TS_F | RSS_F) \ +R(sec_ts_ptype, 1, 0, 1, 0, 0, 1, 0, \ + R_SEC_F | TS_F | PTYPE_F) \ +R(sec_ts_ptype_rss, 1, 0, 1, 0, 0, 1, 1, \ + R_SEC_F | TS_F | PTYPE_F | RSS_F) \ +R(sec_ts_cksum, 1, 0, 1, 0, 1, 0, 0, \ + R_SEC_F | TS_F | CKSUM_F) \ +R(sec_ts_cksum_rss, 1, 0, 1, 0, 1, 0, 1, \ + R_SEC_F | TS_F | CKSUM_F | RSS_F) \ +R(sec_ts_cksum_ptype, 1, 0, 1, 0, 1, 1, 0, \ + R_SEC_F | TS_F | CKSUM_F | PTYPE_F) \ +R(sec_ts_cksum_ptype_rss, 1, 0, 1, 0, 1, 1, 1, \ + R_SEC_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_ts_mark, 1, 0, 1, 1, 0, 0, 0, \ + R_SEC_F | TS_F | MARK_F) \ +R(sec_ts_mark_rss, 1, 0, 1, 1, 0, 0, 1, \ + R_SEC_F | TS_F | MARK_F | RSS_F) \ +R(sec_ts_mark_ptype, 1, 0, 1, 1, 0, 1, 0, \ + R_SEC_F | TS_F | MARK_F | PTYPE_F) \ +R(sec_ts_mark_ptype_rss, 1, 0, 1, 1, 0, 1, 1, \ + R_SEC_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ +R(sec_ts_mark_cksum, 1, 0, 1, 1, 1, 0, 0, \ + R_SEC_F | TS_F | MARK_F | CKSUM_F) \ +R(sec_ts_mark_cksum_rss, 1, 0, 1, 1, 1, 0, 1, \ + R_SEC_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ +R(sec_ts_mark_cksum_ptype, 1, 0, 1, 1, 1, 1, 0, \ + R_SEC_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(sec_ts_mark_cksum_ptype_rss, 1, 0, 1, 1, 1, 1, 1, \ + R_SEC_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_vlan, 1, 1, 0, 0, 0, 0, 0, \ + R_SEC_F | RX_VLAN_F) \ +R(sec_vlan_rss, 1, 1, 0, 0, 0, 0, 1, \ + R_SEC_F | RX_VLAN_F | RSS_F) \ +R(sec_vlan_ptype, 1, 1, 0, 0, 0, 1, 0, \ + R_SEC_F | RX_VLAN_F | PTYPE_F) \ +R(sec_vlan_ptype_rss, 1, 1, 0, 0, 0, 1, 1, \ + R_SEC_F | RX_VLAN_F | PTYPE_F | RSS_F) \ +R(sec_vlan_cksum, 1, 1, 0, 0, 1, 0, 0, \ + R_SEC_F | RX_VLAN_F | CKSUM_F) \ +R(sec_vlan_cksum_rss, 1, 1, 0, 0, 1, 0, 1, \ + R_SEC_F | RX_VLAN_F | CKSUM_F | RSS_F) \ +R(sec_vlan_cksum_ptype, 1, 1, 0, 0, 1, 1, 0, \ + R_SEC_F | RX_VLAN_F | CKSUM_F | PTYPE_F) \ +R(sec_vlan_cksum_ptype_rss, 1, 1, 0, 0, 1, 1, 1, \ + R_SEC_F | RX_VLAN_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_vlan_mark, 1, 1, 0, 1, 0, 0, 0, \ + R_SEC_F | RX_VLAN_F | MARK_F) \ +R(sec_vlan_mark_rss, 1, 1, 0, 1, 0, 0, 1, \ + R_SEC_F | RX_VLAN_F | MARK_F | RSS_F) \ +R(sec_vlan_mark_ptype, 1, 1, 0, 1, 0, 1, 0, \ + R_SEC_F | RX_VLAN_F | MARK_F | PTYPE_F) \ +R(sec_vlan_mark_ptype_rss, 1, 1, 0, 1, 0, 1, 1, \ + R_SEC_F | RX_VLAN_F | MARK_F | PTYPE_F | RSS_F) \ +R(sec_vlan_mark_cksum, 1, 1, 0, 1, 1, 0, 0, \ + R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F) \ +R(sec_vlan_mark_cksum_rss, 1, 1, 0, 1, 1, 0, 1, \ + R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | RSS_F) \ +R(sec_vlan_mark_cksum_ptype, 1, 1, 0, 1, 1, 1, 0, \ + R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(sec_vlan_mark_cksum_ptype_rss, 1, 1, 0, 1, 1, 1, 1, \ + R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_vlan_ts, 1, 1, 1, 0, 0, 0, 0, \ + R_SEC_F | RX_VLAN_F | TS_F) \ +R(sec_vlan_ts_rss, 1, 1, 1, 0, 0, 0, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | RSS_F) \ +R(sec_vlan_ts_ptype, 1, 1, 1, 0, 0, 1, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | PTYPE_F) \ +R(sec_vlan_ts_ptype_rss, 1, 1, 1, 0, 0, 1, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | PTYPE_F | RSS_F) \ +R(sec_vlan_ts_cksum, 1, 1, 1, 0, 1, 0, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F) \ +R(sec_vlan_ts_cksum_rss, 1, 1, 1, 0, 1, 0, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | RSS_F) \ +R(sec_vlan_ts_cksum_ptype, 1, 1, 1, 0, 1, 1, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F) \ +R(sec_vlan_ts_cksum_ptype_rss, 1, 1, 1, 0, 1, 1, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_vlan_ts_mark, 1, 1, 1, 1, 0, 0, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F) \ +R(sec_vlan_ts_mark_rss, 1, 1, 1, 1, 0, 0, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | RSS_F) \ +R(sec_vlan_ts_mark_ptype, 1, 1, 1, 1, 0, 1, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | PTYPE_F) \ +R(sec_vlan_ts_mark_ptype_rss, 1, 1, 1, 1, 0, 1, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ +R(sec_vlan_ts_mark_cksum, 1, 1, 1, 1, 1, 0, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F) \ +R(sec_vlan_ts_mark_cksum_rss, 1, 1, 1, 1, 1, 0, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ +R(sec_vlan_ts_mark_cksum_ptype, 1, 1, 1, 1, 1, 1, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(sec_vlan_ts_mark_cksum_ptype_rss, 1, 1, 1, 1, 1, 1, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_recv_pkts_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts); \ \ diff --git a/drivers/net/cnxk/cn9k_rx_mseg.c b/drivers/net/cnxk/cn9k_rx_mseg.c index d7e19b1..06509e8 100644 --- a/drivers/net/cnxk/cn9k_rx_mseg.c +++ b/drivers/net/cnxk/cn9k_rx_mseg.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_rx.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_recv_pkts_mseg_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ diff --git a/drivers/net/cnxk/cn9k_rx_vec.c b/drivers/net/cnxk/cn9k_rx_vec.c index ef5f771..c96f61c 100644 --- a/drivers/net/cnxk/cn9k_rx_vec.c +++ b/drivers/net/cnxk/cn9k_rx_vec.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_rx.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_recv_pkts_vec_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ diff --git a/drivers/net/cnxk/cn9k_rx_vec_mseg.c b/drivers/net/cnxk/cn9k_rx_vec_mseg.c index e46d8a4..938b1c0 100644 --- a/drivers/net/cnxk/cn9k_rx_vec_mseg.c +++ b/drivers/net/cnxk/cn9k_rx_vec_mseg.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_rx.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_recv_pkts_vec_mseg_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ diff --git a/drivers/net/cnxk/cnxk_ethdev.h b/drivers/net/cnxk/cnxk_ethdev.h index 5ae791f..cfdc493 100644 --- a/drivers/net/cnxk/cnxk_ethdev.h +++ b/drivers/net/cnxk/cnxk_ethdev.h @@ -130,6 +130,9 @@ /* Subtype from inline outbound error event */ #define CNXK_ETHDEV_SEC_OUTB_EV_SUB 0xFFUL +/* SPI will be in 20 bits of tag */ +#define CNXK_ETHDEV_SPI_TAG_MASK 0xFFFFFUL + struct cnxk_fc_cfg { enum rte_eth_fc_mode mode; uint8_t rx_pause; From patchwork Thu Sep 2 02:14:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97744 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 728FDA0C4C; Thu, 2 Sep 2021 04:18:48 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 15D364118A; Thu, 2 Sep 2021 04:17:42 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id A459C4115D for ; Thu, 2 Sep 2021 04:17:40 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181K5Q9j027038 for ; Wed, 1 Sep 2021 19:17:39 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=8Gi7ezmrbnj3GseLp0ejrYPdXzcoE+1eyQWAmM1kESU=; b=cuzdvtlBsZe8fGl4IsP6a/ghUPRPkiwt7Ztf2wzEGrOYiSEGrx2a7GzS5e+EgwMb/mnj eSN1TXClTQKCxmeXJ6FAqt0MSzCbXHrBMUvco96jRRuiQSIKsqbhDs0JQpGWfz5gVX+Z CrqTLTjZ4E/01T9hw/eGDq8IPxx/arEnD831tmSBl2/FOvqoc3fT/Btjdg6BQKxhQ50g xPYhMWl57lO8SoDX1YU+RrMhRpTJMWUpT38H5lI34QhdFTQPImUcuIUPFR1wpGf2jjEX iX+2cmU9XDv5puqGX0ICG4HVtDnpNdrFQyeMYLNzoD3KxTWiiyHwCSucTnlXlV36elRn eg== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com with ESMTP id 3atg8a91kq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:39 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:37 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:37 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 787E33F704B; Wed, 1 Sep 2021 19:17:35 -0700 (PDT) From: Nithin Dabilpuram To: Pavan Nikhilesh , Shijith Thotton , Nithin Dabilpuram , "Kiran Kumar K" , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:56 +0530 Message-ID: <20210902021505.17607-19-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: VL7-8G-eyleFv6n6-MyzjUjYR12xNVb8 X-Proofpoint-ORIG-GUID: VL7-8G-eyleFv6n6-MyzjUjYR12xNVb8 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 18/27] net/cnxk: add cn9k Tx support for security offload X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add support to create and submit CPT instructions on Tx. Signed-off-by: Nithin Dabilpuram Signed-off-by: Pavan Nikhilesh --- drivers/event/cnxk/cn9k_eventdev.c | 27 +- drivers/event/cnxk/cn9k_worker.h | 164 +++++++++- drivers/event/cnxk/cn9k_worker_dual_tx_enq.c | 2 +- drivers/event/cnxk/cn9k_worker_dual_tx_enq_seg.c | 2 +- drivers/event/cnxk/cn9k_worker_tx_enq.c | 2 +- drivers/event/cnxk/cn9k_worker_tx_enq_seg.c | 2 +- drivers/net/cnxk/cn9k_tx.c | 29 +- drivers/net/cnxk/cn9k_tx.h | 392 +++++++++++++++-------- drivers/net/cnxk/cn9k_tx_mseg.c | 2 +- drivers/net/cnxk/cn9k_tx_vec.c | 2 +- drivers/net/cnxk/cn9k_tx_vec_mseg.c | 2 +- 11 files changed, 459 insertions(+), 167 deletions(-) diff --git a/drivers/event/cnxk/cn9k_eventdev.c b/drivers/event/cnxk/cn9k_eventdev.c index e91234e..0c7206c 100644 --- a/drivers/event/cnxk/cn9k_eventdev.c +++ b/drivers/event/cnxk/cn9k_eventdev.c @@ -19,7 +19,8 @@ [!!(dev->rx_offloads & NIX_RX_OFFLOAD_RSS_F)] #define CN9K_SET_EVDEV_ENQ_OP(dev, enq_op, enq_ops) \ - enq_op = enq_ops[!!(dev->tx_offloads & NIX_TX_OFFLOAD_TSTAMP_F)] \ + enq_op = enq_ops[!!(dev->tx_offloads & NIX_TX_OFFLOAD_SECURITY_F)] \ + [!!(dev->tx_offloads & NIX_TX_OFFLOAD_TSTAMP_F)] \ [!!(dev->tx_offloads & NIX_TX_OFFLOAD_TSO_F)] \ [!!(dev->tx_offloads & NIX_TX_OFFLOAD_MBUF_NOFF_F)] \ [!!(dev->tx_offloads & NIX_TX_OFFLOAD_VLAN_QINQ_F)] \ @@ -514,33 +515,33 @@ cn9k_sso_fp_fns_set(struct rte_eventdev *event_dev) /* Tx modes */ const event_tx_adapter_enqueue - sso_hws_tx_adptr_enq[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_tx_adptr_enq_##name, + sso_hws_tx_adptr_enq[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_tx_adptr_enq_##name, NIX_TX_FASTPATH_MODES #undef T }; const event_tx_adapter_enqueue - sso_hws_tx_adptr_enq_seg[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_tx_adptr_enq_seg_##name, + sso_hws_tx_adptr_enq_seg[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_tx_adptr_enq_seg_##name, NIX_TX_FASTPATH_MODES #undef T }; const event_tx_adapter_enqueue - sso_hws_dual_tx_adptr_enq[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_tx_adptr_enq_##name, + sso_hws_dual_tx_adptr_enq[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_tx_adptr_enq_##name, NIX_TX_FASTPATH_MODES #undef T }; const event_tx_adapter_enqueue - sso_hws_dual_tx_adptr_enq_seg[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_tx_adptr_enq_seg_##name, + sso_hws_dual_tx_adptr_enq_seg[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_sso_hws_dual_tx_adptr_enq_seg_##name, NIX_TX_FASTPATH_MODES #undef T }; diff --git a/drivers/event/cnxk/cn9k_worker.h b/drivers/event/cnxk/cn9k_worker.h index f1d2e47..6b4837e 100644 --- a/drivers/event/cnxk/cn9k_worker.h +++ b/drivers/event/cnxk/cn9k_worker.h @@ -478,6 +478,146 @@ cn9k_sso_hws_prepare_pkt(const struct cn9k_eth_txq *txq, struct rte_mbuf *m, cn9k_nix_xmit_prepare(m, cmd, flags, txq->lso_tun_fmt); } +#if defined(RTE_ARCH_ARM64) + +static __rte_always_inline void +cn9k_sso_hws_xmit_sec_one(const struct cn9k_eth_txq *txq, uint64_t base, + struct rte_mbuf *m, uint64_t *cmd, + uint32_t flags) +{ + struct cn9k_outb_priv_data *outb_priv; + rte_iova_t io_addr = txq->cpt_io_addr; + uint64_t *lmt_addr = txq->lmt_addr; + struct cn9k_sec_sess_priv mdata; + struct nix_send_hdr_s *send_hdr; + uint64_t sa_base = txq->sa_base; + uint32_t pkt_len, dlen_adj, rlen; + uint64x2_t cmd01, cmd23; + uint64_t lmt_status, sa; + union nix_send_sg_s *sg; + uintptr_t dptr, nixtx; + uint64_t ucode_cmd[4]; + uint64_t esn, *iv; + uint8_t l2_len; + + mdata.u64 = *rte_security_dynfield(m); + send_hdr = (struct nix_send_hdr_s *)cmd; + if (flags & NIX_TX_NEED_EXT_HDR) { + sg = (union nix_send_sg_s *)&cmd[4]; + } else { + sg = (union nix_send_sg_s *)&cmd[2]; + } + + if (flags & NIX_TX_NEED_SEND_HDR_W1) + l2_len = cmd[1] & 0xFF; + else + l2_len = m->l2_len; + + /* Retrieve DPTR */ + dptr = *(uint64_t *)(sg + 1); + pkt_len = send_hdr->w0.total; + + /* Calculate rlen */ + rlen = pkt_len - l2_len; + rlen = (rlen + mdata.roundup_len) + (mdata.roundup_byte - 1); + rlen &= ~(uint64_t)(mdata.roundup_byte - 1); + rlen += mdata.partial_len; + dlen_adj = rlen - pkt_len + l2_len; + + /* Update send descriptors. Security is single segment only */ + send_hdr->w0.total = pkt_len + dlen_adj; + sg->seg1_size = pkt_len + dlen_adj; + + /* Get area where NIX descriptor needs to be stored */ + nixtx = dptr + pkt_len + dlen_adj; + nixtx += BIT_ULL(7); + nixtx = (nixtx - 1) & ~(BIT_ULL(7) - 1); + + roc_lmt_mov((void *)(nixtx + 16), cmd, cn9k_nix_tx_ext_subs(flags)); + + /* Load opcode and cptr already prepared at pkt metadata set */ + pkt_len -= l2_len; + pkt_len += sizeof(struct roc_onf_ipsec_outb_hdr) + + ROC_ONF_IPSEC_OUTB_MAX_L2_INFO_SZ; + sa_base &= ~(ROC_NIX_INL_SA_BASE_ALIGN - 1); + + sa = (uintptr_t)roc_nix_inl_onf_ipsec_outb_sa(sa_base, mdata.sa_idx); + ucode_cmd[3] = (ROC_CPT_DFLT_ENG_GRP_SE_IE << 61 | sa); + ucode_cmd[0] = (ROC_IE_ONF_MAJOR_OP_PROCESS_OUTBOUND_IPSEC << 48 | + 0x40UL << 48 | pkt_len); + + /* CPT Word 0 and Word 1 */ + cmd01 = vdupq_n_u64((nixtx + 16) | (cn9k_nix_tx_ext_subs(flags) + 1)); + /* CPT_RES_S is 16B above NIXTX */ + cmd01 = vsetq_lane_u8(nixtx & BIT_ULL(7), cmd01, 8); + + /* CPT word 2 and 3 */ + cmd23 = vdupq_n_u64(0); + cmd23 = vsetq_lane_u64((((uint64_t)RTE_EVENT_TYPE_CPU << 28) | + CNXK_ETHDEV_SEC_OUTB_EV_SUB << 20), cmd23, 0); + cmd23 = vsetq_lane_u64((uintptr_t)m | 1, cmd23, 1); + + dptr += l2_len - ROC_ONF_IPSEC_OUTB_MAX_L2_INFO_SZ - + sizeof(struct roc_onf_ipsec_outb_hdr); + ucode_cmd[1] = dptr; + ucode_cmd[2] = dptr; + + /* Update IV to zero and l2 sz */ + *(uint16_t *)(dptr + sizeof(struct roc_onf_ipsec_outb_hdr)) = + rte_cpu_to_be_16(ROC_ONF_IPSEC_OUTB_MAX_L2_INFO_SZ); + iv = (uint64_t *)(dptr + 8); + iv[0] = 0; + iv[1] = 0; + + /* Head wait if needed */ + if (base) + roc_sso_hws_head_wait(base + SSOW_LF_GWS_TAG); + + /* ESN */ + outb_priv = roc_nix_inl_onf_ipsec_outb_sa_sw_rsvd((void *)sa); + esn = outb_priv->esn; + outb_priv->esn = esn + 1; + + ucode_cmd[0] |= (esn >> 32) << 16; + esn = rte_cpu_to_be_32(esn & (BIT_ULL(32) - 1)); + + /* Update ESN and IPID and IV */ + *(uint64_t *)dptr = esn << 32 | esn; + + rte_io_wmb(); + cn9k_sso_txq_fc_wait(txq); + + /* Write CPT instruction to lmt line */ + vst1q_u64(lmt_addr, cmd01); + vst1q_u64(lmt_addr + 2, cmd23); + + roc_lmt_mov_seg(lmt_addr + 4, ucode_cmd, 2); + + if (roc_lmt_submit_ldeor(io_addr) == 0) { + do { + vst1q_u64(lmt_addr, cmd01); + vst1q_u64(lmt_addr + 2, cmd23); + roc_lmt_mov_seg(lmt_addr + 4, ucode_cmd, 2); + + lmt_status = roc_lmt_submit_ldeor(io_addr); + } while (lmt_status == 0); + } +} +#else + +static inline void +cn9k_sso_hws_xmit_sec_one(const struct cn9k_eth_txq *txq, uint64_t base, + struct rte_mbuf *m, uint64_t *cmd, + uint32_t flags) +{ + RTE_SET_USED(txq); + RTE_SET_USED(base); + RTE_SET_USED(m); + RTE_SET_USED(cmd); + RTE_SET_USED(flags); +} +#endif + static __rte_always_inline uint16_t cn9k_sso_hws_event_tx(uint64_t base, struct rte_event *ev, uint64_t *cmd, const uint64_t txq_data[][RTE_MAX_QUEUES_PER_PORT], @@ -494,11 +634,30 @@ cn9k_sso_hws_event_tx(uint64_t base, struct rte_event *ev, uint64_t *cmd, * In case of fast free is not set, both cn9k_nix_prepare_mseg() * and cn9k_nix_xmit_prepare() has a barrier after refcnt update. */ - if (!(flags & NIX_TX_OFFLOAD_MBUF_NOFF_F)) + if (!(flags & NIX_TX_OFFLOAD_MBUF_NOFF_F) && + !(flags & NIX_TX_OFFLOAD_SECURITY_F)) rte_io_wmb(); txq = cn9k_sso_hws_xtract_meta(m, txq_data); cn9k_sso_hws_prepare_pkt(txq, m, cmd, flags); + if (flags & NIX_TX_OFFLOAD_SECURITY_F) { + uint64_t ol_flags = m->ol_flags; + + if (ol_flags & PKT_TX_SEC_OFFLOAD) { + uintptr_t ssow_base = base; + + if (ev->sched_type) + ssow_base = 0; + + cn9k_sso_hws_xmit_sec_one(txq, ssow_base, m, cmd, + flags); + goto done; + } + + if (!(flags & NIX_TX_OFFLOAD_MBUF_NOFF_F)) + rte_io_wmb(); + } + if (flags & NIX_TX_MULTI_SEG_F) { const uint16_t segdw = cn9k_nix_prepare_mseg(m, cmd, flags); if (!CNXK_TT_FROM_EVENT(ev->event)) { @@ -526,6 +685,7 @@ cn9k_sso_hws_event_tx(uint64_t base, struct rte_event *ev, uint64_t *cmd, } } +done: if (flags & NIX_TX_OFFLOAD_MBUF_NOFF_F) { if (ref_cnt > 1) return 1; @@ -537,7 +697,7 @@ cn9k_sso_hws_event_tx(uint64_t base, struct rte_event *ev, uint64_t *cmd, return 1; } -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_hot cn9k_sso_hws_tx_adptr_enq_##name( \ void *port, struct rte_event ev[], uint16_t nb_events); \ uint16_t __rte_hot cn9k_sso_hws_tx_adptr_enq_seg_##name( \ diff --git a/drivers/event/cnxk/cn9k_worker_dual_tx_enq.c b/drivers/event/cnxk/cn9k_worker_dual_tx_enq.c index 92e2981..db045d0 100644 --- a/drivers/event/cnxk/cn9k_worker_dual_tx_enq.c +++ b/drivers/event/cnxk/cn9k_worker_dual_tx_enq.c @@ -4,7 +4,7 @@ #include "cn9k_worker.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_hot cn9k_sso_hws_dual_tx_adptr_enq_##name( \ void *port, struct rte_event ev[], uint16_t nb_events) \ { \ diff --git a/drivers/event/cnxk/cn9k_worker_dual_tx_enq_seg.c b/drivers/event/cnxk/cn9k_worker_dual_tx_enq_seg.c index dfb574c..95d711f 100644 --- a/drivers/event/cnxk/cn9k_worker_dual_tx_enq_seg.c +++ b/drivers/event/cnxk/cn9k_worker_dual_tx_enq_seg.c @@ -4,7 +4,7 @@ #include "cn9k_worker.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_hot cn9k_sso_hws_dual_tx_adptr_enq_seg_##name( \ void *port, struct rte_event ev[], uint16_t nb_events) \ { \ diff --git a/drivers/event/cnxk/cn9k_worker_tx_enq.c b/drivers/event/cnxk/cn9k_worker_tx_enq.c index 3df649c..026cef8 100644 --- a/drivers/event/cnxk/cn9k_worker_tx_enq.c +++ b/drivers/event/cnxk/cn9k_worker_tx_enq.c @@ -4,7 +4,7 @@ #include "cn9k_worker.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_hot cn9k_sso_hws_tx_adptr_enq_##name( \ void *port, struct rte_event ev[], uint16_t nb_events) \ { \ diff --git a/drivers/event/cnxk/cn9k_worker_tx_enq_seg.c b/drivers/event/cnxk/cn9k_worker_tx_enq_seg.c index 0efe291..97cd7c7 100644 --- a/drivers/event/cnxk/cn9k_worker_tx_enq_seg.c +++ b/drivers/event/cnxk/cn9k_worker_tx_enq_seg.c @@ -4,7 +4,7 @@ #include "cn9k_worker.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_hot cn9k_sso_hws_tx_adptr_enq_seg_##name( \ void *port, struct rte_event ev[], uint16_t nb_events) \ { \ diff --git a/drivers/net/cnxk/cn9k_tx.c b/drivers/net/cnxk/cn9k_tx.c index 763f9a1..e5691a2 100644 --- a/drivers/net/cnxk/cn9k_tx.c +++ b/drivers/net/cnxk/cn9k_tx.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_tx.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_xmit_pkts_##name( \ void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts) \ { \ @@ -23,12 +23,13 @@ NIX_TX_FASTPATH_MODES static inline void pick_tx_func(struct rte_eth_dev *eth_dev, - const eth_tx_burst_t tx_burst[2][2][2][2][2][2]) + const eth_tx_burst_t tx_burst[2][2][2][2][2][2][2]) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); /* [TS] [TSO] [NOFF] [VLAN] [OL3_OL4_CSUM] [IL3_IL4_CSUM] */ eth_dev->tx_pkt_burst = tx_burst + [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_SECURITY_F)] [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_TSTAMP_F)] [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_TSO_F)] [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_MBUF_NOFF_F)] @@ -42,33 +43,33 @@ cn9k_eth_set_tx_function(struct rte_eth_dev *eth_dev) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - const eth_tx_burst_t nix_eth_tx_burst[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_##name, + const eth_tx_burst_t nix_eth_tx_burst[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_##name, NIX_TX_FASTPATH_MODES #undef T }; - const eth_tx_burst_t nix_eth_tx_burst_mseg[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_mseg_##name, + const eth_tx_burst_t nix_eth_tx_burst_mseg[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_mseg_##name, NIX_TX_FASTPATH_MODES #undef T }; - const eth_tx_burst_t nix_eth_tx_vec_burst[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_vec_##name, + const eth_tx_burst_t nix_eth_tx_vec_burst[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_vec_##name, NIX_TX_FASTPATH_MODES #undef T }; - const eth_tx_burst_t nix_eth_tx_vec_burst_mseg[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_vec_mseg_##name, + const eth_tx_burst_t nix_eth_tx_vec_burst_mseg[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_vec_mseg_##name, NIX_TX_FASTPATH_MODES #undef T diff --git a/drivers/net/cnxk/cn9k_tx.h b/drivers/net/cnxk/cn9k_tx.h index a27ff76..44273ec 100644 --- a/drivers/net/cnxk/cn9k_tx.h +++ b/drivers/net/cnxk/cn9k_tx.h @@ -1819,139 +1819,269 @@ cn9k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, #define NOFF_F NIX_TX_OFFLOAD_MBUF_NOFF_F #define TSO_F NIX_TX_OFFLOAD_TSO_F #define TSP_F NIX_TX_OFFLOAD_TSTAMP_F +#define T_SEC_F NIX_TX_OFFLOAD_SECURITY_F -/* [TSP] [TSO] [NOFF] [VLAN] [OL3OL4CSUM] [L3L4CSUM] */ -#define NIX_TX_FASTPATH_MODES \ -T(no_offload, 0, 0, 0, 0, 0, 0, 4, \ - NIX_TX_OFFLOAD_NONE) \ -T(l3l4csum, 0, 0, 0, 0, 0, 1, 4, \ - L3L4CSUM_F) \ -T(ol3ol4csum, 0, 0, 0, 0, 1, 0, 4, \ - OL3OL4CSUM_F) \ -T(ol3ol4csum_l3l4csum, 0, 0, 0, 0, 1, 1, 4, \ - OL3OL4CSUM_F | L3L4CSUM_F) \ -T(vlan, 0, 0, 0, 1, 0, 0, 6, \ - VLAN_F) \ -T(vlan_l3l4csum, 0, 0, 0, 1, 0, 1, 6, \ - VLAN_F | L3L4CSUM_F) \ -T(vlan_ol3ol4csum, 0, 0, 0, 1, 1, 0, 6, \ - VLAN_F | OL3OL4CSUM_F) \ -T(vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 1, 1, 6, \ - VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(noff, 0, 0, 1, 0, 0, 0, 4, \ - NOFF_F) \ -T(noff_l3l4csum, 0, 0, 1, 0, 0, 1, 4, \ - NOFF_F | L3L4CSUM_F) \ -T(noff_ol3ol4csum, 0, 0, 1, 0, 1, 0, 4, \ - NOFF_F | OL3OL4CSUM_F) \ -T(noff_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 1, 1, 4, \ - NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(noff_vlan, 0, 0, 1, 1, 0, 0, 6, \ - NOFF_F | VLAN_F) \ -T(noff_vlan_l3l4csum, 0, 0, 1, 1, 0, 1, 6, \ - NOFF_F | VLAN_F | L3L4CSUM_F) \ -T(noff_vlan_ol3ol4csum, 0, 0, 1, 1, 1, 0, 6, \ - NOFF_F | VLAN_F | OL3OL4CSUM_F) \ -T(noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 1, 1, 6, \ - NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso, 0, 1, 0, 0, 0, 0, 6, \ - TSO_F) \ -T(tso_l3l4csum, 0, 1, 0, 0, 0, 1, 6, \ - TSO_F | L3L4CSUM_F) \ -T(tso_ol3ol4csum, 0, 1, 0, 0, 1, 0, 6, \ - TSO_F | OL3OL4CSUM_F) \ -T(tso_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 1, 1, 6, \ - TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso_vlan, 0, 1, 0, 1, 0, 0, 6, \ - TSO_F | VLAN_F) \ -T(tso_vlan_l3l4csum, 0, 1, 0, 1, 0, 1, 6, \ - TSO_F | VLAN_F | L3L4CSUM_F) \ -T(tso_vlan_ol3ol4csum, 0, 1, 0, 1, 1, 0, 6, \ - TSO_F | VLAN_F | OL3OL4CSUM_F) \ -T(tso_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 1, 1, 6, \ - TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso_noff, 0, 1, 1, 0, 0, 0, 6, \ - TSO_F | NOFF_F) \ -T(tso_noff_l3l4csum, 0, 1, 1, 0, 0, 1, 6, \ - TSO_F | NOFF_F | L3L4CSUM_F) \ -T(tso_noff_ol3ol4csum, 0, 1, 1, 0, 1, 0, 6, \ - TSO_F | NOFF_F | OL3OL4CSUM_F) \ -T(tso_noff_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 1, 1, 6, \ - TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso_noff_vlan, 0, 1, 1, 1, 0, 0, 6, \ - TSO_F | NOFF_F | VLAN_F) \ -T(tso_noff_vlan_l3l4csum, 0, 1, 1, 1, 0, 1, 6, \ - TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ -T(tso_noff_vlan_ol3ol4csum, 0, 1, 1, 1, 1, 0, 6, \ - TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ -T(tso_noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 1, 1, 6, \ - TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts, 1, 0, 0, 0, 0, 0, 8, \ - TSP_F) \ -T(ts_l3l4csum, 1, 0, 0, 0, 0, 1, 8, \ - TSP_F | L3L4CSUM_F) \ -T(ts_ol3ol4csum, 1, 0, 0, 0, 1, 0, 8, \ - TSP_F | OL3OL4CSUM_F) \ -T(ts_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 1, 1, 8, \ - TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_vlan, 1, 0, 0, 1, 0, 0, 8, \ - TSP_F | VLAN_F) \ -T(ts_vlan_l3l4csum, 1, 0, 0, 1, 0, 1, 8, \ - TSP_F | VLAN_F | L3L4CSUM_F) \ -T(ts_vlan_ol3ol4csum, 1, 0, 0, 1, 1, 0, 8, \ - TSP_F | VLAN_F | OL3OL4CSUM_F) \ -T(ts_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 1, 1, 8, \ - TSP_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_noff, 1, 0, 1, 0, 0, 0, 8, \ - TSP_F | NOFF_F) \ -T(ts_noff_l3l4csum, 1, 0, 1, 0, 0, 1, 8, \ - TSP_F | NOFF_F | L3L4CSUM_F) \ -T(ts_noff_ol3ol4csum, 1, 0, 1, 0, 1, 0, 8, \ - TSP_F | NOFF_F | OL3OL4CSUM_F) \ -T(ts_noff_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 1, 1, 8, \ - TSP_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_noff_vlan, 1, 0, 1, 1, 0, 0, 8, \ - TSP_F | NOFF_F | VLAN_F) \ -T(ts_noff_vlan_l3l4csum, 1, 0, 1, 1, 0, 1, 8, \ - TSP_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ -T(ts_noff_vlan_ol3ol4csum, 1, 0, 1, 1, 1, 0, 8, \ - TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ -T(ts_noff_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 1, 1, 8, \ - TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_tso, 1, 1, 0, 0, 0, 0, 8, \ - TSP_F | TSO_F) \ -T(ts_tso_l3l4csum, 1, 1, 0, 0, 0, 1, 8, \ - TSP_F | TSO_F | L3L4CSUM_F) \ -T(ts_tso_ol3ol4csum, 1, 1, 0, 0, 1, 0, 8, \ - TSP_F | TSO_F | OL3OL4CSUM_F) \ -T(ts_tso_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 1, 1, 8, \ - TSP_F | TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_tso_vlan, 1, 1, 0, 1, 0, 0, 8, \ - TSP_F | TSO_F | VLAN_F) \ -T(ts_tso_vlan_l3l4csum, 1, 1, 0, 1, 0, 1, 8, \ - TSP_F | TSO_F | VLAN_F | L3L4CSUM_F) \ -T(ts_tso_vlan_ol3ol4csum, 1, 1, 0, 1, 1, 0, 8, \ - TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F) \ -T(ts_tso_vlan_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 1, 1, 8, \ - TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_tso_noff, 1, 1, 1, 0, 0, 0, 8, \ - TSP_F | TSO_F | NOFF_F) \ -T(ts_tso_noff_l3l4csum, 1, 1, 1, 0, 0, 1, 8, \ - TSP_F | TSO_F | NOFF_F | L3L4CSUM_F) \ -T(ts_tso_noff_ol3ol4csum, 1, 1, 1, 0, 1, 0, 8, \ - TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F) \ -T(ts_tso_noff_ol3ol4csum_l3l4csum, 1, 1, 1, 0, 1, 1, 8, \ - TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_tso_noff_vlan, 1, 1, 1, 1, 0, 0, 8, \ - TSP_F | TSO_F | NOFF_F | VLAN_F) \ -T(ts_tso_noff_vlan_l3l4csum, 1, 1, 1, 1, 0, 1, 8, \ - TSP_F | TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ -T(ts_tso_noff_vlan_ol3ol4csum, 1, 1, 1, 1, 1, 0, 8, \ - TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ -T(ts_tso_noff_vlan_ol3ol4csum_l3l4csum, 1, 1, 1, 1, 1, 1, 8, \ - TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) +/* [T_SEC_F] [TSP] [TSO] [NOFF] [VLAN] [OL3OL4CSUM] [L3L4CSUM] */ +#define NIX_TX_FASTPATH_MODES \ +T(no_offload, 0, 0, 0, 0, 0, 0, 0, 4, \ + NIX_TX_OFFLOAD_NONE) \ +T(l3l4csum, 0, 0, 0, 0, 0, 0, 1, 4, \ + L3L4CSUM_F) \ +T(ol3ol4csum, 0, 0, 0, 0, 0, 1, 0, 4, \ + OL3OL4CSUM_F) \ +T(ol3ol4csum_l3l4csum, 0, 0, 0, 0, 0, 1, 1, 4, \ + OL3OL4CSUM_F | L3L4CSUM_F) \ +T(vlan, 0, 0, 0, 0, 1, 0, 0, 6, \ + VLAN_F) \ +T(vlan_l3l4csum, 0, 0, 0, 0, 1, 0, 1, 6, \ + VLAN_F | L3L4CSUM_F) \ +T(vlan_ol3ol4csum, 0, 0, 0, 0, 1, 1, 0, 6, \ + VLAN_F | OL3OL4CSUM_F) \ +T(vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 0, 1, 1, 1, 6, \ + VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(noff, 0, 0, 0, 1, 0, 0, 0, 4, \ + NOFF_F) \ +T(noff_l3l4csum, 0, 0, 0, 1, 0, 0, 1, 4, \ + NOFF_F | L3L4CSUM_F) \ +T(noff_ol3ol4csum, 0, 0, 0, 1, 0, 1, 0, 4, \ + NOFF_F | OL3OL4CSUM_F) \ +T(noff_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 0, 1, 1, 4, \ + NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(noff_vlan, 0, 0, 0, 1, 1, 0, 0, 6, \ + NOFF_F | VLAN_F) \ +T(noff_vlan_l3l4csum, 0, 0, 0, 1, 1, 0, 1, 6, \ + NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(noff_vlan_ol3ol4csum, 0, 0, 0, 1, 1, 1, 0, 6, \ + NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 1, 1, 1, 6, \ + NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(tso, 0, 0, 1, 0, 0, 0, 0, 6, \ + TSO_F) \ +T(tso_l3l4csum, 0, 0, 1, 0, 0, 0, 1, 6, \ + TSO_F | L3L4CSUM_F) \ +T(tso_ol3ol4csum, 0, 0, 1, 0, 0, 1, 0, 6, \ + TSO_F | OL3OL4CSUM_F) \ +T(tso_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 0, 1, 1, 6, \ + TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(tso_vlan, 0, 0, 1, 0, 1, 0, 0, 6, \ + TSO_F | VLAN_F) \ +T(tso_vlan_l3l4csum, 0, 0, 1, 0, 1, 0, 1, 6, \ + TSO_F | VLAN_F | L3L4CSUM_F) \ +T(tso_vlan_ol3ol4csum, 0, 0, 1, 0, 1, 1, 0, 6, \ + TSO_F | VLAN_F | OL3OL4CSUM_F) \ +T(tso_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 1, 1, 1, 6, \ + TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(tso_noff, 0, 0, 1, 1, 0, 0, 0, 6, \ + TSO_F | NOFF_F) \ +T(tso_noff_l3l4csum, 0, 0, 1, 1, 0, 0, 1, 6, \ + TSO_F | NOFF_F | L3L4CSUM_F) \ +T(tso_noff_ol3ol4csum, 0, 0, 1, 1, 0, 1, 0, 6, \ + TSO_F | NOFF_F | OL3OL4CSUM_F) \ +T(tso_noff_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 0, 1, 1, 6, \ + TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(tso_noff_vlan, 0, 0, 1, 1, 1, 0, 0, 6, \ + TSO_F | NOFF_F | VLAN_F) \ +T(tso_noff_vlan_l3l4csum, 0, 0, 1, 1, 1, 0, 1, 6, \ + TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(tso_noff_vlan_ol3ol4csum, 0, 0, 1, 1, 1, 1, 0, 6, \ + TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(tso_noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 1, 1, 1, 6, \ + TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts, 0, 1, 0, 0, 0, 0, 0, 8, \ + TSP_F) \ +T(ts_l3l4csum, 0, 1, 0, 0, 0, 0, 1, 8, \ + TSP_F | L3L4CSUM_F) \ +T(ts_ol3ol4csum, 0, 1, 0, 0, 0, 1, 0, 8, \ + TSP_F | OL3OL4CSUM_F) \ +T(ts_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 0, 1, 1, 8, \ + TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_vlan, 0, 1, 0, 0, 1, 0, 0, 8, \ + TSP_F | VLAN_F) \ +T(ts_vlan_l3l4csum, 0, 1, 0, 0, 1, 0, 1, 8, \ + TSP_F | VLAN_F | L3L4CSUM_F) \ +T(ts_vlan_ol3ol4csum, 0, 1, 0, 0, 1, 1, 0, 8, \ + TSP_F | VLAN_F | OL3OL4CSUM_F) \ +T(ts_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 1, 1, 1, 8, \ + TSP_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_noff, 0, 1, 0, 1, 0, 0, 0, 8, \ + TSP_F | NOFF_F) \ +T(ts_noff_l3l4csum, 0, 1, 0, 1, 0, 0, 1, 8, \ + TSP_F | NOFF_F | L3L4CSUM_F) \ +T(ts_noff_ol3ol4csum, 0, 1, 0, 1, 0, 1, 0, 8, \ + TSP_F | NOFF_F | OL3OL4CSUM_F) \ +T(ts_noff_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 0, 1, 1, 8, \ + TSP_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_noff_vlan, 0, 1, 0, 1, 1, 0, 0, 8, \ + TSP_F | NOFF_F | VLAN_F) \ +T(ts_noff_vlan_l3l4csum, 0, 1, 0, 1, 1, 0, 1, 8, \ + TSP_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(ts_noff_vlan_ol3ol4csum, 0, 1, 0, 1, 1, 1, 0, 8, \ + TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(ts_noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 1, 1, 1, 8, \ + TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_tso, 0, 1, 1, 0, 0, 0, 0, 8, \ + TSP_F | TSO_F) \ +T(ts_tso_l3l4csum, 0, 1, 1, 0, 0, 0, 1, 8, \ + TSP_F | TSO_F | L3L4CSUM_F) \ +T(ts_tso_ol3ol4csum, 0, 1, 1, 0, 0, 1, 0, 8, \ + TSP_F | TSO_F | OL3OL4CSUM_F) \ +T(ts_tso_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 0, 1, 1, 8, \ + TSP_F | TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_tso_vlan, 0, 1, 1, 0, 1, 0, 0, 8, \ + TSP_F | TSO_F | VLAN_F) \ +T(ts_tso_vlan_l3l4csum, 0, 1, 1, 0, 1, 0, 1, 8, \ + TSP_F | TSO_F | VLAN_F | L3L4CSUM_F) \ +T(ts_tso_vlan_ol3ol4csum, 0, 1, 1, 0, 1, 1, 0, 8, \ + TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F) \ +T(ts_tso_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 1, 1, 1, 8, \ + TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_tso_noff, 0, 1, 1, 1, 0, 0, 0, 8, \ + TSP_F | TSO_F | NOFF_F) \ +T(ts_tso_noff_l3l4csum, 0, 1, 1, 1, 0, 0, 1, 8, \ + TSP_F | TSO_F | NOFF_F | L3L4CSUM_F) \ +T(ts_tso_noff_ol3ol4csum, 0, 1, 1, 1, 0, 1, 0, 8, \ + TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F) \ +T(ts_tso_noff_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 0, 1, 1, 8, \ + TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_tso_noff_vlan, 0, 1, 1, 1, 1, 0, 0, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F) \ +T(ts_tso_noff_vlan_l3l4csum, 0, 1, 1, 1, 1, 0, 1, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(ts_tso_noff_vlan_ol3ol4csum, 0, 1, 1, 1, 1, 1, 0, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(ts_tso_noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 1, 1, 1, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F)\ +T(sec, 1, 0, 0, 0, 0, 0, 0, 4, \ + T_SEC_F) \ +T(sec_l3l4csum, 1, 0, 0, 0, 0, 0, 1, 4, \ + T_SEC_F | L3L4CSUM_F) \ +T(sec_ol3ol4csum, 1, 0, 0, 0, 0, 1, 0, 4, \ + T_SEC_F | OL3OL4CSUM_F) \ +T(sec_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 0, 1, 1, 4, \ + T_SEC_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_vlan, 1, 0, 0, 0, 1, 0, 0, 6, \ + T_SEC_F | VLAN_F) \ +T(sec_vlan_l3l4csum, 1, 0, 0, 0, 1, 0, 1, 6, \ + T_SEC_F | VLAN_F | L3L4CSUM_F) \ +T(sec_vlan_ol3ol4csum, 1, 0, 0, 0, 1, 1, 0, 6, \ + T_SEC_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 1, 1, 1, 6, \ + T_SEC_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_noff, 1, 0, 0, 1, 0, 0, 0, 4, \ + T_SEC_F | NOFF_F) \ +T(sec_noff_l3l4csum, 1, 0, 0, 1, 0, 0, 1, 4, \ + T_SEC_F | NOFF_F | L3L4CSUM_F) \ +T(sec_noff_ol3ol4csum, 1, 0, 0, 1, 0, 1, 0, 4, \ + T_SEC_F | NOFF_F | OL3OL4CSUM_F) \ +T(sec_noff_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 0, 1, 1, 4, \ + T_SEC_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_noff_vlan, 1, 0, 0, 1, 1, 0, 0, 6, \ + T_SEC_F | NOFF_F | VLAN_F) \ +T(sec_noff_vlan_l3l4csum, 1, 0, 0, 1, 1, 0, 1, 6, \ + T_SEC_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(sec_noff_vlan_ol3ol4csum, 1, 0, 0, 1, 1, 1, 0, 6, \ + T_SEC_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_noff_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 1, 1, 1, 6, \ + T_SEC_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_tso, 1, 0, 1, 0, 0, 0, 0, 6, \ + T_SEC_F | TSO_F) \ +T(sec_tso_l3l4csum, 1, 0, 1, 0, 0, 0, 1, 6, \ + T_SEC_F | TSO_F | L3L4CSUM_F) \ +T(sec_tso_ol3ol4csum, 1, 0, 1, 0, 0, 1, 0, 6, \ + T_SEC_F | TSO_F | OL3OL4CSUM_F) \ +T(sec_tso_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 0, 1, 1, 6, \ + T_SEC_F | TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_tso_vlan, 1, 0, 1, 0, 1, 0, 0, 6, \ + T_SEC_F | TSO_F | VLAN_F) \ +T(sec_tso_vlan_l3l4csum, 1, 0, 1, 0, 1, 0, 1, 6, \ + T_SEC_F | TSO_F | VLAN_F | L3L4CSUM_F) \ +T(sec_tso_vlan_ol3ol4csum, 1, 0, 1, 0, 1, 1, 0, 6, \ + T_SEC_F | TSO_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_tso_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 1, 1, 1, 6, \ + T_SEC_F | TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_tso_noff, 1, 0, 1, 1, 0, 0, 0, 6, \ + T_SEC_F | TSO_F | NOFF_F) \ +T(sec_tso_noff_l3l4csum, 1, 0, 1, 1, 0, 0, 1, 6, \ + T_SEC_F | TSO_F | NOFF_F | L3L4CSUM_F) \ +T(sec_tso_noff_ol3ol4csum, 1, 0, 1, 1, 0, 1, 0, 6, \ + T_SEC_F | TSO_F | NOFF_F | OL3OL4CSUM_F) \ +T(sec_tso_noff_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 0, 1, 1, 6, \ + T_SEC_F | TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_tso_noff_vlan, 1, 0, 1, 1, 1, 0, 0, 6, \ + T_SEC_F | TSO_F | NOFF_F | VLAN_F) \ +T(sec_tso_noff_vlan_l3l4csum, 1, 0, 1, 1, 1, 0, 1, 6, \ + T_SEC_F | TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(sec_tso_noff_vlan_ol3ol4csum, 1, 0, 1, 1, 1, 1, 0, 6, \ + T_SEC_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_tso_noff_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 1, 1, 1, 6, \ + T_SEC_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F)\ +T(sec_ts, 1, 1, 0, 0, 0, 0, 0, 8, \ + T_SEC_F | TSP_F) \ +T(sec_ts_l3l4csum, 1, 1, 0, 0, 0, 0, 1, 8, \ + T_SEC_F | TSP_F | L3L4CSUM_F) \ +T(sec_ts_ol3ol4csum, 1, 1, 0, 0, 0, 1, 0, 8, \ + T_SEC_F | TSP_F | OL3OL4CSUM_F) \ +T(sec_ts_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 0, 1, 1, 8, \ + T_SEC_F | TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_ts_vlan, 1, 1, 0, 0, 1, 0, 0, 8, \ + T_SEC_F | TSP_F | VLAN_F) \ +T(sec_ts_vlan_l3l4csum, 1, 1, 0, 0, 1, 0, 1, 8, \ + T_SEC_F | TSP_F | VLAN_F | L3L4CSUM_F) \ +T(sec_ts_vlan_ol3ol4csum, 1, 1, 0, 0, 1, 1, 0, 8, \ + T_SEC_F | TSP_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_ts_vlan_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 1, 1, 1, 8, \ + T_SEC_F | TSP_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_ts_noff, 1, 1, 0, 1, 0, 0, 0, 8, \ + T_SEC_F | TSP_F | NOFF_F) \ +T(sec_ts_noff_l3l4csum, 1, 1, 0, 1, 0, 0, 1, 8, \ + T_SEC_F | TSP_F | NOFF_F | L3L4CSUM_F) \ +T(sec_ts_noff_ol3ol4csum, 1, 1, 0, 1, 0, 1, 0, 8, \ + T_SEC_F | TSP_F | NOFF_F | OL3OL4CSUM_F) \ +T(sec_ts_noff_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 0, 1, 1, 8, \ + T_SEC_F | TSP_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_ts_noff_vlan, 1, 1, 0, 1, 1, 0, 0, 8, \ + T_SEC_F | TSP_F | NOFF_F | VLAN_F) \ +T(sec_ts_noff_vlan_l3l4csum, 1, 1, 0, 1, 1, 0, 1, 8, \ + T_SEC_F | TSP_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(sec_ts_noff_vlan_ol3ol4csum, 1, 1, 0, 1, 1, 1, 0, 8, \ + T_SEC_F | TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_ts_noff_vlan_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 1, 1, 1, 8, \ + T_SEC_F | TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F)\ +T(sec_ts_tso, 1, 1, 1, 0, 0, 0, 0, 8, \ + T_SEC_F | TSP_F | TSO_F) \ +T(sec_ts_tso_l3l4csum, 1, 1, 1, 0, 0, 0, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | L3L4CSUM_F) \ +T(sec_ts_tso_ol3ol4csum, 1, 1, 1, 0, 0, 1, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | OL3OL4CSUM_F) \ +T(sec_ts_tso_ol3ol4csum_l3l4csum, 1, 1, 1, 0, 0, 1, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_ts_tso_vlan, 1, 1, 1, 0, 1, 0, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | VLAN_F) \ +T(sec_ts_tso_vlan_l3l4csum, 1, 1, 1, 0, 1, 0, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | VLAN_F | L3L4CSUM_F) \ +T(sec_ts_tso_vlan_ol3ol4csum, 1, 1, 1, 0, 1, 1, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_ts_tso_vlan_ol3ol4csum_l3l4csum, 1, 1, 1, 0, 1, 1, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_ts_tso_noff, 1, 1, 1, 1, 0, 0, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F) \ +T(sec_ts_tso_noff_l3l4csum, 1, 1, 1, 1, 0, 0, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | L3L4CSUM_F) \ +T(sec_ts_tso_noff_ol3ol4csum, 1, 1, 1, 1, 0, 1, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F) \ +T(sec_ts_tso_noff_ol3ol4csum_l3l4csum, 1, 1, 1, 1, 0, 1, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_ts_tso_noff_vlan, 1, 1, 1, 1, 1, 0, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | VLAN_F) \ +T(sec_ts_tso_noff_vlan_l3l4csum, 1, 1, 1, 1, 1, 0, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(sec_ts_tso_noff_vlan_ol3ol4csum, 1, 1, 1, 1, 1, 1, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F)\ +T(sec_ts_tso_noff_vlan_ol3ol4csum_l3l4csum, 1, 1, 1, 1, 1, 1, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | \ + L3L4CSUM_F) -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_xmit_pkts_##name( \ void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts); \ \ diff --git a/drivers/net/cnxk/cn9k_tx_mseg.c b/drivers/net/cnxk/cn9k_tx_mseg.c index f3c427c..37cba78 100644 --- a/drivers/net/cnxk/cn9k_tx_mseg.c +++ b/drivers/net/cnxk/cn9k_tx_mseg.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_tx.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot \ cn9k_nix_xmit_pkts_mseg_##name(void *tx_queue, \ struct rte_mbuf **tx_pkts, \ diff --git a/drivers/net/cnxk/cn9k_tx_vec.c b/drivers/net/cnxk/cn9k_tx_vec.c index 56a3e25..b424f95 100644 --- a/drivers/net/cnxk/cn9k_tx_vec.c +++ b/drivers/net/cnxk/cn9k_tx_vec.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_tx.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot \ cn9k_nix_xmit_pkts_vec_##name(void *tx_queue, \ struct rte_mbuf **tx_pkts, \ diff --git a/drivers/net/cnxk/cn9k_tx_vec_mseg.c b/drivers/net/cnxk/cn9k_tx_vec_mseg.c index 0256efd..5fdf0a9 100644 --- a/drivers/net/cnxk/cn9k_tx_vec_mseg.c +++ b/drivers/net/cnxk/cn9k_tx_vec_mseg.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_tx.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_xmit_pkts_vec_mseg_##name( \ void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts) \ { \ From patchwork Thu Sep 2 02:14:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97745 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D77A2A0C4C; Thu, 2 Sep 2021 04:18:55 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 90C4E41104; Thu, 2 Sep 2021 04:17:46 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 95935410FE for ; Thu, 2 Sep 2021 04:17:44 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQ203011573 for ; Wed, 1 Sep 2021 19:17:44 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=UuPtBQ4Jzvd3UK+/5CxFM8Ry3pLxW/oddD14A88c+Hs=; b=UNARFcr7N1RpEPKe6ch02TR6MLBCADiWINZWdS47qntUQIuLVZCJ16l/Q3MFPH2QEyeF CI7Xib/DmCUmieeg6cfVh9ocgsLGpbul2DeMkIdokSjgue3md1XjLGCRrIcAMHVK9N6w HsBf/ffD1hJX7VYPUumuDxKnAdVH8L4SUJ74gVTeQwAFo4jW2pw6BKqhbR/9ZlKOOohJ AaK6mtm+uCxSB6t59tpUkWSRPIL3U+OHNUzTRfk/DYaIz7mJjmnca509v3Smfu33MRxd jgYVlRjwkQs8YJtUG+Dm5FzGG3cqhC/mJGk1tC33vTG3WKAbL5agpqFSS9L9cMWRL1l4 1A== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hu3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:43 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:41 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:41 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id B95753F7040; Wed, 1 Sep 2021 19:17:38 -0700 (PDT) From: Nithin Dabilpuram To: Pavan Nikhilesh , Shijith Thotton , Nithin Dabilpuram , "Kiran Kumar K" , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:57 +0530 Message-ID: <20210902021505.17607-20-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: iKLyED_EcZb5NX-PZklNP5D7S6EyHlic X-Proofpoint-GUID: iKLyED_EcZb5NX-PZklNP5D7S6EyHlic X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 19/27] net/cnxk: add cn10k Rx support for security offload X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add support to receive CPT processed packets on Rx via second pass. Signed-off-by: Nithin Dabilpuram --- drivers/event/cnxk/cn10k_eventdev.c | 80 ++-- drivers/event/cnxk/cn10k_worker.h | 73 +++- drivers/event/cnxk/cn10k_worker_deq.c | 2 +- drivers/event/cnxk/cn10k_worker_deq_burst.c | 2 +- drivers/event/cnxk/cn10k_worker_deq_ca.c | 2 +- drivers/event/cnxk/cn10k_worker_deq_tmo.c | 2 +- drivers/net/cnxk/cn10k_ethdev.h | 4 + drivers/net/cnxk/cn10k_rx.c | 31 +- drivers/net/cnxk/cn10k_rx.h | 648 +++++++++++++++++++++++----- drivers/net/cnxk/cn10k_rx_mseg.c | 2 +- drivers/net/cnxk/cn10k_rx_vec.c | 4 +- drivers/net/cnxk/cn10k_rx_vec_mseg.c | 4 +- drivers/net/cnxk/cn10k_tx.h | 3 - 13 files changed, 688 insertions(+), 169 deletions(-) diff --git a/drivers/event/cnxk/cn10k_eventdev.c b/drivers/event/cnxk/cn10k_eventdev.c index bfb6f1a..2f2e7f8 100644 --- a/drivers/event/cnxk/cn10k_eventdev.c +++ b/drivers/event/cnxk/cn10k_eventdev.c @@ -7,7 +7,8 @@ #include "cnxk_worker.h" #define CN10K_SET_EVDEV_DEQ_OP(dev, deq_op, deq_ops) \ - deq_op = deq_ops[!!(dev->rx_offloads & NIX_RX_OFFLOAD_VLAN_STRIP_F)] \ + deq_op = deq_ops[!!(dev->rx_offloads & NIX_RX_OFFLOAD_SECURITY_F)] \ + [!!(dev->rx_offloads & NIX_RX_OFFLOAD_VLAN_STRIP_F)] \ [!!(dev->rx_offloads & NIX_RX_OFFLOAD_TSTAMP_F)] \ [!!(dev->rx_offloads & NIX_RX_OFFLOAD_MARK_UPDATE_F)] \ [!!(dev->rx_offloads & NIX_RX_OFFLOAD_CHECKSUM_F)] \ @@ -287,88 +288,91 @@ static void cn10k_sso_fp_fns_set(struct rte_eventdev *event_dev) { struct cnxk_sso_evdev *dev = cnxk_sso_pmd_priv(event_dev); - const event_dequeue_t sso_hws_deq[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_##name, + const event_dequeue_t sso_hws_deq[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_burst_t sso_hws_deq_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_burst_##name, + const event_dequeue_burst_t sso_hws_deq_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_tmo[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_tmo_##name, + const event_dequeue_t sso_hws_deq_tmo[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_tmo_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_burst_t sso_hws_deq_tmo_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_tmo_burst_##name, + const event_dequeue_burst_t + sso_hws_deq_tmo_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_tmo_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_ca[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_ca_##name, + const event_dequeue_t sso_hws_deq_ca[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_ca_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_burst_t sso_hws_deq_ca_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_ca_burst_##name, + const event_dequeue_burst_t + sso_hws_deq_ca_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_ca_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_seg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_seg_##name, + const event_dequeue_t sso_hws_deq_seg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_seg_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_burst_t sso_hws_deq_seg_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_seg_burst_##name, + const event_dequeue_burst_t + sso_hws_deq_seg_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_seg_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_tmo_seg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_tmo_seg_##name, + const event_dequeue_t sso_hws_deq_tmo_seg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_tmo_seg_##name, NIX_RX_FASTPATH_MODES #undef R }; const event_dequeue_burst_t - sso_hws_deq_tmo_seg_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_tmo_seg_burst_##name, + sso_hws_deq_tmo_seg_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_tmo_seg_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_ca_seg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_ca_seg_##name, + const event_dequeue_t sso_hws_deq_ca_seg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_ca_seg_##name, NIX_RX_FASTPATH_MODES #undef R }; const event_dequeue_burst_t - sso_hws_deq_ca_seg_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_ca_seg_burst_##name, + sso_hws_deq_ca_seg_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_ca_seg_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; @@ -384,7 +388,7 @@ cn10k_sso_fp_fns_set(struct rte_eventdev *event_dev) const event_tx_adapter_enqueue sso_hws_tx_adptr_enq_seg[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_tx_adptr_enq_seg_##name, NIX_TX_FASTPATH_MODES #undef T diff --git a/drivers/event/cnxk/cn10k_worker.h b/drivers/event/cnxk/cn10k_worker.h index e5ed043..b79bd90 100644 --- a/drivers/event/cnxk/cn10k_worker.h +++ b/drivers/event/cnxk/cn10k_worker.h @@ -106,12 +106,17 @@ cn10k_wqe_to_mbuf(uint64_t wqe, const uint64_t mbuf, uint8_t port_id, static __rte_always_inline void cn10k_process_vwqe(uintptr_t vwqe, uint16_t port_id, const uint32_t flags, - void *lookup_mem, void *tstamp) + void *lookup_mem, void *tstamp, uintptr_t lbase) { uint64_t mbuf_init = 0x100010000ULL | RTE_PKTMBUF_HEADROOM | (flags & NIX_RX_OFFLOAD_TSTAMP_F ? 8 : 0); struct rte_event_vector *vec; + uint64_t aura_handle, laddr; uint16_t nb_mbufs, non_vec; + uint16_t lmt_id, d_off; + struct rte_mbuf *mbuf; + uint8_t loff = 0; + uint64_t sa_base; uint64_t **wqe; mbuf_init |= ((uint64_t)port_id) << 48; @@ -121,17 +126,41 @@ cn10k_process_vwqe(uintptr_t vwqe, uint16_t port_id, const uint32_t flags, nb_mbufs = RTE_ALIGN_FLOOR(vec->nb_elem, NIX_DESCS_PER_LOOP); nb_mbufs = cn10k_nix_recv_pkts_vector(&mbuf_init, vec->mbufs, nb_mbufs, flags | NIX_RX_VWQE_F, lookup_mem, - tstamp); + tstamp, lbase); wqe += nb_mbufs; non_vec = vec->nb_elem - nb_mbufs; + if (flags & NIX_RX_OFFLOAD_SECURITY_F && non_vec) { + mbuf = (struct rte_mbuf *)((uintptr_t)wqe[0] - + sizeof(struct rte_mbuf)); + /* Pick first mbuf's aura handle assuming all + * mbufs are from a vec and are from same RQ. + */ + aura_handle = mbuf->pool->pool_id; + ROC_LMT_BASE_ID_GET(lbase, lmt_id); + laddr = lbase; + laddr += 8; + d_off = ((uintptr_t)mbuf->buf_addr - (uintptr_t)mbuf); + d_off += (mbuf_init & 0xFFFF); + sa_base = cnxk_nix_sa_base_get(mbuf_init >> 48, lookup_mem); + sa_base &= ~(ROC_NIX_INL_SA_BASE_ALIGN - 1); + } + while (non_vec) { struct nix_cqe_hdr_s *cqe = (struct nix_cqe_hdr_s *)wqe[0]; - struct rte_mbuf *mbuf; uint64_t tstamp_ptr; mbuf = (struct rte_mbuf *)((char *)cqe - sizeof(struct rte_mbuf)); + + /* Translate meta to mbuf */ + if (flags & NIX_RX_OFFLOAD_SECURITY_F) { + const uint64_t cq_w1 = *((const uint64_t *)cqe + 1); + + mbuf = nix_sec_meta_to_mbuf_sc(cq_w1, sa_base, laddr, + &loff, mbuf, d_off); + } + cn10k_nix_cqe_to_mbuf(cqe, cqe->tag, mbuf, lookup_mem, mbuf_init, flags); /* Extracting tstamp, if PTP enabled*/ @@ -145,6 +174,12 @@ cn10k_process_vwqe(uintptr_t vwqe, uint16_t port_id, const uint32_t flags, non_vec--; wqe++; } + + /* Free remaining meta buffers if any */ + if (flags & NIX_RX_OFFLOAD_SECURITY_F && loff) { + nix_sec_flush_meta(laddr, lmt_id, loff, aura_handle); + plt_io_wmb(); + } } static __rte_always_inline uint16_t @@ -188,6 +223,34 @@ cn10k_sso_hws_get_work(struct cn10k_sso_hws *ws, struct rte_event *ev, RTE_EVENT_TYPE_ETHDEV) { uint8_t port = CNXK_SUB_EVENT_FROM_TAG(gw.u64[0]); + if (flags & NIX_RX_OFFLOAD_SECURITY_F) { + struct rte_mbuf *m; + uintptr_t sa_base; + uint64_t iova = 0; + uint8_t loff = 0; + uint16_t d_off; + uint64_t cq_w1; + + m = (struct rte_mbuf *)mbuf; + d_off = (uintptr_t)(m->buf_addr) - (uintptr_t)m; + d_off += RTE_PKTMBUF_HEADROOM; + + cq_w1 = *(uint64_t *)(gw.u64[1] + 8); + + sa_base = cnxk_nix_sa_base_get(port, + lookup_mem); + sa_base &= ~(ROC_NIX_INL_SA_BASE_ALIGN - 1); + + mbuf = (uint64_t)nix_sec_meta_to_mbuf_sc(cq_w1, + sa_base, (uintptr_t)&iova, + &loff, (struct rte_mbuf *)mbuf, + d_off); + if (loff) + roc_npa_aura_op_free(m->pool->pool_id, + 0, iova); + + } + gw.u64[0] = CNXK_CLR_SUB_EVENT(gw.u64[0]); cn10k_wqe_to_mbuf(gw.u64[1], mbuf, port, gw.u64[0] & 0xFFFFF, flags, @@ -212,7 +275,7 @@ cn10k_sso_hws_get_work(struct cn10k_sso_hws *ws, struct rte_event *ev, ((uint64_t)port << 32); *(uint64_t *)gw.u64[1] = (uint64_t)vwqe_hdr; cn10k_process_vwqe(gw.u64[1], port, flags, lookup_mem, - ws->tstamp); + ws->tstamp, ws->lmt_base); } } @@ -290,7 +353,7 @@ uint16_t __rte_hot cn10k_sso_hws_enq_fwd_burst(void *port, uint16_t __rte_hot cn10k_sso_hws_ca_enq(void *port, struct rte_event ev[], uint16_t nb_events); -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn10k_sso_hws_deq_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks); \ uint16_t __rte_hot cn10k_sso_hws_deq_burst_##name( \ diff --git a/drivers/event/cnxk/cn10k_worker_deq.c b/drivers/event/cnxk/cn10k_worker_deq.c index 36ec454..6083f69 100644 --- a/drivers/event/cnxk/cn10k_worker_deq.c +++ b/drivers/event/cnxk/cn10k_worker_deq.c @@ -6,7 +6,7 @@ #include "cnxk_eventdev.h" #include "cnxk_worker.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn10k_sso_hws_deq_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks) \ { \ diff --git a/drivers/event/cnxk/cn10k_worker_deq_burst.c b/drivers/event/cnxk/cn10k_worker_deq_burst.c index 29ecc55..8539d5d 100644 --- a/drivers/event/cnxk/cn10k_worker_deq_burst.c +++ b/drivers/event/cnxk/cn10k_worker_deq_burst.c @@ -6,7 +6,7 @@ #include "cnxk_eventdev.h" #include "cnxk_worker.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn10k_sso_hws_deq_burst_##name( \ void *port, struct rte_event ev[], uint16_t nb_events, \ uint64_t timeout_ticks) \ diff --git a/drivers/event/cnxk/cn10k_worker_deq_ca.c b/drivers/event/cnxk/cn10k_worker_deq_ca.c index 508d30f..0d10fc8 100644 --- a/drivers/event/cnxk/cn10k_worker_deq_ca.c +++ b/drivers/event/cnxk/cn10k_worker_deq_ca.c @@ -6,7 +6,7 @@ #include "cnxk_eventdev.h" #include "cnxk_worker.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn10k_sso_hws_deq_ca_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks) \ { \ diff --git a/drivers/event/cnxk/cn10k_worker_deq_tmo.c b/drivers/event/cnxk/cn10k_worker_deq_tmo.c index c8524a2..537ae37 100644 --- a/drivers/event/cnxk/cn10k_worker_deq_tmo.c +++ b/drivers/event/cnxk/cn10k_worker_deq_tmo.c @@ -6,7 +6,7 @@ #include "cnxk_eventdev.h" #include "cnxk_worker.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot cn10k_sso_hws_deq_tmo_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks) \ { \ diff --git a/drivers/net/cnxk/cn10k_ethdev.h b/drivers/net/cnxk/cn10k_ethdev.h index a888364..200cd93 100644 --- a/drivers/net/cnxk/cn10k_ethdev.h +++ b/drivers/net/cnxk/cn10k_ethdev.h @@ -81,4 +81,8 @@ void cn10k_eth_set_tx_function(struct rte_eth_dev *eth_dev); /* Security context setup */ void cn10k_eth_sec_ops_override(void); +#define LMT_OFF(lmt_addr, lmt_num, offset) \ + (void *)((uintptr_t)(lmt_addr) + \ + ((uint64_t)(lmt_num) << ROC_LMT_LINE_SIZE_LOG2) + (offset)) + #endif /* __CN10K_ETHDEV_H__ */ diff --git a/drivers/net/cnxk/cn10k_rx.c b/drivers/net/cnxk/cn10k_rx.c index 69e767a..d6af54b 100644 --- a/drivers/net/cnxk/cn10k_rx.c +++ b/drivers/net/cnxk/cn10k_rx.c @@ -5,7 +5,7 @@ #include "cn10k_ethdev.h" #include "cn10k_rx.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ @@ -17,12 +17,13 @@ NIX_RX_FASTPATH_MODES static inline void pick_rx_func(struct rte_eth_dev *eth_dev, - const eth_rx_burst_t rx_burst[2][2][2][2][2][2]) + const eth_rx_burst_t rx_burst[2][2][2][2][2][2][2]) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); /* [VLAN] [TSP] [MARK] [CKSUM] [PTYPE] [RSS] */ eth_dev->rx_pkt_burst = rx_burst + [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_SECURITY_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_VLAN_STRIP_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_TSTAMP_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_MARK_UPDATE_F)] @@ -38,33 +39,33 @@ cn10k_eth_set_rx_function(struct rte_eth_dev *eth_dev) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - const eth_rx_burst_t nix_eth_rx_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_##name, + const eth_rx_burst_t nix_eth_rx_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_burst_mseg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_mseg_##name, + const eth_rx_burst_t nix_eth_rx_burst_mseg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_mseg_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_vec_burst[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_vec_##name, + const eth_rx_burst_t nix_eth_rx_vec_burst[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_vec_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_vec_burst_mseg[2][2][2][2][2][2] = { -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_vec_mseg_##name, + const eth_rx_burst_t nix_eth_rx_vec_burst_mseg[2][2][2][2][2][2][2] = { +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_vec_mseg_##name, NIX_RX_FASTPATH_MODES #undef R @@ -73,7 +74,7 @@ cn10k_eth_set_rx_function(struct rte_eth_dev *eth_dev) /* Copy multi seg version with no offload for tear down sequence */ if (rte_eal_process_type() == RTE_PROC_PRIMARY) dev->rx_pkt_burst_no_offload = - nix_eth_rx_burst_mseg[0][0][0][0][0][0]; + nix_eth_rx_burst_mseg[0][0][0][0][0][0][0]; if (dev->scalar_ena) { if (dev->rx_offloads & DEV_RX_OFFLOAD_SCATTER) diff --git a/drivers/net/cnxk/cn10k_rx.h b/drivers/net/cnxk/cn10k_rx.h index d27a231..fcc451a 100644 --- a/drivers/net/cnxk/cn10k_rx.h +++ b/drivers/net/cnxk/cn10k_rx.h @@ -65,6 +65,130 @@ nix_get_mbuf_from_cqe(void *cq, const uint64_t data_off) return (struct rte_mbuf *)(buff - data_off); } +static __rte_always_inline void +nix_sec_flush_meta(uintptr_t laddr, uint16_t lmt_id, uint8_t loff, + uintptr_t aura_handle) +{ + uint64_t pa; + + /* laddr is pointing to first pointer */ + laddr -= 8; + + /* Trigger free either on lmtline full or different aura handle */ + pa = roc_npa_aura_handle_to_base(aura_handle) + NPA_LF_AURA_BATCH_FREE0; + + /* Update aura handle */ + *(uint64_t *)laddr = (((uint64_t)(loff & 0x1) << 32) | + roc_npa_aura_handle_to_aura(aura_handle)); + + pa |= ((loff >> 1) << 4); + roc_lmt_submit_steorl(lmt_id, pa); +} + +static __rte_always_inline struct rte_mbuf * +nix_sec_meta_to_mbuf_sc(uint64_t cq_w1, const uint64_t sa_base, uintptr_t laddr, + uint8_t *loff, struct rte_mbuf *mbuf, uint16_t data_off) +{ + const void *__p = (void *)((uintptr_t)mbuf + (uint16_t)data_off); + const struct cpt_parse_hdr_s *hdr = (const struct cpt_parse_hdr_s *)__p; + struct cn10k_inb_priv_data *inb_priv; + struct rte_mbuf *inner; + uint32_t sa_idx; + void *inb_sa; + uint64_t w0; + + if (cq_w1 & BIT(11)) { + inner = (struct rte_mbuf *)(rte_be_to_cpu_64(hdr->wqe_ptr) - + sizeof(struct rte_mbuf)); + + /* Get SPI from CPT_PARSE_S's cookie(already swapped) */ + w0 = hdr->w0.u64; + sa_idx = w0 >> 32; + + inb_sa = roc_nix_inl_ot_ipsec_inb_sa(sa_base, sa_idx); + inb_priv = roc_nix_inl_ot_ipsec_inb_sa_sw_rsvd(inb_sa); + + /* Update dynamic field with userdata */ + *rte_security_dynfield(inner) = (uint64_t)inb_priv->userdata; + + /* Update l2 hdr length first */ + inner->pkt_len = (hdr->w2.il3_off - + sizeof(struct cpt_parse_hdr_s) - (w0 & 0x7)); + + /* Store meta in lmtline to free + * Assume all meta's from same aura. + */ + *(uint64_t *)(laddr + (*loff << 3)) = (uint64_t)mbuf; + *loff = *loff + 1; + + return inner; + } + return mbuf; +} + +#if defined(RTE_ARCH_ARM64) + +static __rte_always_inline struct rte_mbuf * +nix_sec_meta_to_mbuf(uint64_t cq_w1, uintptr_t sa_base, uintptr_t laddr, + uint8_t *loff, struct rte_mbuf *mbuf, uint16_t data_off, + uint8x16_t *rx_desc_field1, uint64_t *ol_flags) +{ + const void *__p = (void *)((uintptr_t)mbuf + (uint16_t)data_off); + const struct cpt_parse_hdr_s *hdr = (const struct cpt_parse_hdr_s *)__p; + struct cn10k_inb_priv_data *inb_priv; + struct rte_mbuf *inner; + uint64_t *sg, res_w1; + uint32_t sa_idx; + void *inb_sa; + uint16_t len; + uint64_t w0; + + if (cq_w1 & BIT(11)) { + inner = (struct rte_mbuf *)(rte_be_to_cpu_64(hdr->wqe_ptr) - + sizeof(struct rte_mbuf)); + /* Get SPI from CPT_PARSE_S's cookie(already swapped) */ + w0 = hdr->w0.u64; + sa_idx = w0 >> 32; + + inb_sa = roc_nix_inl_ot_ipsec_inb_sa(sa_base, sa_idx); + inb_priv = roc_nix_inl_ot_ipsec_inb_sa_sw_rsvd(inb_sa); + + /* Update dynamic field with userdata */ + *rte_security_dynfield(inner) = (uint64_t)inb_priv->userdata; + + /* CPT result(struct cpt_cn10k_res_s) is at + * after first IOVA in meta + */ + sg = (uint64_t *)(inner + 1); + res_w1 = sg[10]; + + /* Clear checksum flags and update security flag */ + *ol_flags &= ~(PKT_RX_L4_CKSUM_MASK | PKT_RX_IP_CKSUM_MASK); + *ol_flags |= (((res_w1 & 0xFF) == CPT_COMP_WARN) ? + PKT_RX_SEC_OFFLOAD : + (PKT_RX_SEC_OFFLOAD | PKT_RX_SEC_OFFLOAD_FAILED)); + /* Calculate inner packet length */ + len = ((res_w1 >> 16) & 0xFFFF) + hdr->w2.il3_off - + sizeof(struct cpt_parse_hdr_s) - (w0 & 0x7); + /* Update pkt_len and data_len */ + *rx_desc_field1 = vsetq_lane_u16(len, *rx_desc_field1, 2); + *rx_desc_field1 = vsetq_lane_u16(len, *rx_desc_field1, 4); + + /* Store meta in lmtline to free + * Assume all meta's from same aura. + */ + *(uint64_t *)(laddr + (*loff << 3)) = (uint64_t)mbuf; + *loff = *loff + 1; + + /* Return inner mbuf */ + return inner; + } + + /* Return same mbuf as it is not a decrypted pkt */ + return mbuf; +} +#endif + static __rte_always_inline uint32_t nix_ptype_get(const void *const lookup_mem, const uint64_t in) { @@ -177,8 +301,8 @@ cn10k_nix_cqe_to_mbuf(const struct nix_cqe_hdr_s *cq, const uint32_t tag, { const union nix_rx_parse_u *rx = (const union nix_rx_parse_u *)((const uint64_t *)cq + 1); - const uint16_t len = rx->pkt_lenm1 + 1; const uint64_t w1 = *(const uint64_t *)rx; + uint16_t len = rx->pkt_lenm1 + 1; uint64_t ol_flags = 0; /* Mark mempool obj as "get" as it is alloc'ed by NIX */ @@ -194,8 +318,30 @@ cn10k_nix_cqe_to_mbuf(const struct nix_cqe_hdr_s *cq, const uint32_t tag, ol_flags |= PKT_RX_RSS_HASH; } - if (flag & NIX_RX_OFFLOAD_CHECKSUM_F) - ol_flags |= nix_rx_olflags_get(lookup_mem, w1); + /* Process Security packets */ + if (flag & NIX_RX_OFFLOAD_SECURITY_F) { + if (w1 & BIT(11)) { + /* CPT result(struct cpt_cn10k_res_s) is at + * after first IOVA in meta + */ + const uint64_t *sg = (const uint64_t *)(mbuf + 1); + const uint64_t res_w1 = sg[10]; + const uint16_t uc_cc = res_w1 & 0xFF; + + /* Rlen */ + len = ((res_w1 >> 16) & 0xFFFF) + mbuf->pkt_len; + ol_flags |= ((uc_cc == CPT_COMP_WARN) ? + PKT_RX_SEC_OFFLOAD : + (PKT_RX_SEC_OFFLOAD | + PKT_RX_SEC_OFFLOAD_FAILED)); + } else { + if (flag & NIX_RX_OFFLOAD_CHECKSUM_F) + ol_flags |= nix_rx_olflags_get(lookup_mem, w1); + } + } else { + if (flag & NIX_RX_OFFLOAD_CHECKSUM_F) + ol_flags |= nix_rx_olflags_get(lookup_mem, w1); + } if (flag & NIX_RX_OFFLOAD_VLAN_STRIP_F) { if (rx->vtag0_gone) { @@ -263,13 +409,28 @@ cn10k_nix_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts, const uintptr_t desc = rxq->desc; const uint64_t wdata = rxq->wdata; const uint32_t qmask = rxq->qmask; + uint64_t lbase = rxq->lmt_base; uint16_t packets = 0, nb_pkts; + uint8_t loff = 0, lnum = 0; uint32_t head = rxq->head; struct nix_cqe_hdr_s *cq; struct rte_mbuf *mbuf; + uint64_t aura_handle; + uint64_t sa_base; + uint16_t lmt_id; + uint64_t laddr; nb_pkts = nix_rx_nb_pkts(rxq, wdata, pkts, qmask); + if (flags & NIX_RX_OFFLOAD_SECURITY_F) { + aura_handle = rxq->aura_handle; + sa_base = rxq->sa_base; + sa_base &= ~(ROC_NIX_INL_SA_BASE_ALIGN - 1); + ROC_LMT_BASE_ID_GET(lbase, lmt_id); + laddr = lbase; + laddr += 8; + } + while (packets < nb_pkts) { /* Prefetch N desc ahead */ rte_prefetch_non_temporal( @@ -278,6 +439,14 @@ cn10k_nix_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts, mbuf = nix_get_mbuf_from_cqe(cq, data_off); + /* Translate meta to mbuf */ + if (flags & NIX_RX_OFFLOAD_SECURITY_F) { + const uint64_t cq_w1 = *((const uint64_t *)cq + 1); + + mbuf = nix_sec_meta_to_mbuf_sc(cq_w1, sa_base, laddr, + &loff, mbuf, data_off); + } + cn10k_nix_cqe_to_mbuf(cq, cq->tag, mbuf, lookup_mem, mbuf_init, flags); cnxk_nix_mbuf_to_tstamp(mbuf, rxq->tstamp, @@ -289,6 +458,20 @@ cn10k_nix_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts, roc_prefetch_store_keep(mbuf); head++; head &= qmask; + + if (flags & NIX_RX_OFFLOAD_SECURITY_F) { + /* Flush when we don't have space for 4 meta */ + if ((15 - loff) < 1) { + nix_sec_flush_meta(laddr, lmt_id + lnum, loff, + aura_handle); + lnum++; + lnum &= BIT_ULL(ROC_LMT_LINES_PER_CORE_LOG2) - + 1; + /* First pointer starts at 8B offset */ + laddr = (uintptr_t)LMT_OFF(lbase, lnum, 8); + loff = 0; + } + } } rxq->head = head; @@ -297,6 +480,12 @@ cn10k_nix_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts, /* Free all the CQs that we've processed */ plt_write64((wdata | nb_pkts), rxq->cq_door); + /* Free remaining meta buffers if any */ + if (flags & NIX_RX_OFFLOAD_SECURITY_F && loff) { + nix_sec_flush_meta(laddr, lmt_id + lnum, loff, aura_handle); + plt_io_wmb(); + } + return nb_pkts; } @@ -327,7 +516,8 @@ nix_qinq_update(const uint64_t w2, uint64_t ol_flags, struct rte_mbuf *mbuf) static __rte_always_inline uint16_t cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, const uint16_t flags, void *lookup_mem, - struct cnxk_timesync_info *tstamp) + struct cnxk_timesync_info *tstamp, + uintptr_t lmt_base) { struct cn10k_eth_rxq *rxq = args; const uint64_t mbuf_initializer = (flags & NIX_RX_VWQE_F) ? @@ -346,9 +536,13 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, uint64x2_t rearm2 = vdupq_n_u64(mbuf_initializer); uint64x2_t rearm3 = vdupq_n_u64(mbuf_initializer); struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3; + uint64_t aura_handle, lbase, laddr; + uint8_t loff = 0, lnum = 0; uint8x16_t f0, f1, f2, f3; + uint16_t lmt_id, d_off; uint16_t packets = 0; uint16_t pkts_left; + uintptr_t sa_base; uint32_t head; uintptr_t cq0; @@ -366,6 +560,38 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, RTE_SET_USED(head); } + if (flags & NIX_RX_OFFLOAD_SECURITY_F) { + if (flags & NIX_RX_VWQE_F) { + uint16_t port; + + mbuf0 = (struct rte_mbuf *)((uintptr_t)mbufs[0] - + sizeof(struct rte_mbuf)); + /* Pick first mbuf's aura handle assuming all + * mbufs are from a vec and are from same RQ. + */ + aura_handle = mbuf0->pool->pool_id; + /* Calculate offset from mbuf to actual data area */ + d_off = ((uintptr_t)mbuf0->buf_addr - (uintptr_t)mbuf0); + d_off += (mbuf_initializer & 0xFFFF); + + /* Get SA Base from lookup tbl using port_id */ + port = mbuf_initializer >> 48; + sa_base = cnxk_nix_sa_base_get(port, lookup_mem); + + lbase = lmt_base; + } else { + aura_handle = rxq->aura_handle; + d_off = rxq->data_off; + sa_base = rxq->sa_base; + lbase = rxq->lmt_base; + } + sa_base &= ~(ROC_NIX_INL_SA_BASE_ALIGN - 1); + ROC_LMT_BASE_ID_GET(lbase, lmt_id); + lnum = 0; + laddr = lbase; + laddr += 8; + } + while (packets < pkts) { if (!(flags & NIX_RX_VWQE_F)) { /* Exit loop if head is about to wrap and become @@ -428,6 +654,14 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, f2 = vqtbl1q_u8(cq2_w8, shuf_msk); f3 = vqtbl1q_u8(cq3_w8, shuf_msk); + if (flags & NIX_RX_OFFLOAD_SECURITY_F) { + /* Prefetch probable CPT parse header area */ + rte_prefetch_non_temporal(RTE_PTR_ADD(mbuf0, d_off)); + rte_prefetch_non_temporal(RTE_PTR_ADD(mbuf1, d_off)); + rte_prefetch_non_temporal(RTE_PTR_ADD(mbuf2, d_off)); + rte_prefetch_non_temporal(RTE_PTR_ADD(mbuf3, d_off)); + } + /* Load CQE word0 and word 1 */ const uint64_t cq0_w0 = *CQE_PTR_OFF(cq0, 0, 0, flags); const uint64_t cq0_w1 = *CQE_PTR_OFF(cq0, 0, 8, flags); @@ -474,6 +708,30 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, ol_flags3 |= nix_rx_olflags_get(lookup_mem, cq3_w1); } + /* Translate meta to mbuf */ + if (flags & NIX_RX_OFFLOAD_SECURITY_F) { + /* Checksum ol_flags will be cleared if mbuf is meta */ + mbuf0 = nix_sec_meta_to_mbuf(cq0_w1, sa_base, laddr, + &loff, mbuf0, d_off, &f0, + &ol_flags0); + mbuf01 = vsetq_lane_u64((uint64_t)mbuf0, mbuf01, 0); + + mbuf1 = nix_sec_meta_to_mbuf(cq1_w1, sa_base, laddr, + &loff, mbuf1, d_off, &f1, + &ol_flags1); + mbuf01 = vsetq_lane_u64((uint64_t)mbuf1, mbuf01, 1); + + mbuf2 = nix_sec_meta_to_mbuf(cq2_w1, sa_base, laddr, + &loff, mbuf2, d_off, &f2, + &ol_flags2); + mbuf23 = vsetq_lane_u64((uint64_t)mbuf2, mbuf23, 0); + + mbuf3 = nix_sec_meta_to_mbuf(cq3_w1, sa_base, laddr, + &loff, mbuf3, d_off, &f3, + &ol_flags3); + mbuf23 = vsetq_lane_u64((uint64_t)mbuf3, mbuf23, 1); + } + if (flags & NIX_RX_OFFLOAD_VLAN_STRIP_F) { uint64_t cq0_w2 = *(uint64_t *)(cq0 + CQE_SZ(0) + 16); uint64_t cq1_w2 = *(uint64_t *)(cq0 + CQE_SZ(1) + 16); @@ -659,6 +917,26 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, head += NIX_DESCS_PER_LOOP; head &= qmask; } + + if (flags & NIX_RX_OFFLOAD_SECURITY_F) { + /* Flush when we don't have space for 4 meta */ + if ((15 - loff) < 4) { + nix_sec_flush_meta(laddr, lmt_id + lnum, loff, + aura_handle); + lnum++; + lnum &= BIT_ULL(ROC_LMT_LINES_PER_CORE_LOG2) - + 1; + /* First pointer starts at 8B offset */ + laddr = (uintptr_t)LMT_OFF(lbase, lnum, 8); + loff = 0; + } + } + } + + if (flags & NIX_RX_OFFLOAD_SECURITY_F && loff) { + nix_sec_flush_meta(laddr, lmt_id + lnum, loff, aura_handle); + if (flags & NIX_RX_VWQE_F) + plt_io_wmb(); } if (flags & NIX_RX_VWQE_F) @@ -681,16 +959,18 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, #else static inline uint16_t -cn10k_nix_recv_pkts_vector(void *rx_queue, struct rte_mbuf **rx_pkts, - uint16_t pkts, const uint16_t flags, - void *lookup_mem, void *tstamp) +cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, + const uint16_t flags, void *lookup_mem, + struct cnxk_timesync_info *tstamp, + uintptr_t lmt_base) { - RTE_SET_USED(lookup_mem); - RTE_SET_USED(rx_queue); - RTE_SET_USED(rx_pkts); + RTE_SET_USED(args); + RTE_SET_USED(mbufs); RTE_SET_USED(pkts); RTE_SET_USED(flags); + RTE_SET_USED(lookup_mem); RTE_SET_USED(tstamp); + RTE_SET_USED(lmt_base); return 0; } @@ -704,98 +984,268 @@ cn10k_nix_recv_pkts_vector(void *rx_queue, struct rte_mbuf **rx_pkts, #define MARK_F NIX_RX_OFFLOAD_MARK_UPDATE_F #define TS_F NIX_RX_OFFLOAD_TSTAMP_F #define RX_VLAN_F NIX_RX_OFFLOAD_VLAN_STRIP_F +#define R_SEC_F NIX_RX_OFFLOAD_SECURITY_F -/* [RX_VLAN_F] [TS] [MARK] [CKSUM] [PTYPE] [RSS] */ +/* [R_SEC_F] [RX_VLAN_F] [TS] [MARK] [CKSUM] [PTYPE] [RSS] */ #define NIX_RX_FASTPATH_MODES \ -R(no_offload, 0, 0, 0, 0, 0, 0, NIX_RX_OFFLOAD_NONE) \ -R(rss, 0, 0, 0, 0, 0, 1, RSS_F) \ -R(ptype, 0, 0, 0, 0, 1, 0, PTYPE_F) \ -R(ptype_rss, 0, 0, 0, 0, 1, 1, PTYPE_F | RSS_F) \ -R(cksum, 0, 0, 0, 1, 0, 0, CKSUM_F) \ -R(cksum_rss, 0, 0, 0, 1, 0, 1, CKSUM_F | RSS_F) \ -R(cksum_ptype, 0, 0, 0, 1, 1, 0, CKSUM_F | PTYPE_F) \ -R(cksum_ptype_rss, 0, 0, 0, 1, 1, 1, CKSUM_F | PTYPE_F | RSS_F) \ -R(mark, 0, 0, 1, 0, 0, 0, MARK_F) \ -R(mark_rss, 0, 0, 1, 0, 0, 1, MARK_F | RSS_F) \ -R(mark_ptype, 0, 0, 1, 0, 1, 0, MARK_F | PTYPE_F) \ -R(mark_ptype_rss, 0, 0, 1, 0, 1, 1, MARK_F | PTYPE_F | RSS_F) \ -R(mark_cksum, 0, 0, 1, 1, 0, 0, MARK_F | CKSUM_F) \ -R(mark_cksum_rss, 0, 0, 1, 1, 0, 1, MARK_F | CKSUM_F | RSS_F) \ -R(mark_cksum_ptype, 0, 0, 1, 1, 1, 0, MARK_F | CKSUM_F | PTYPE_F) \ -R(mark_cksum_ptype_rss, 0, 0, 1, 1, 1, 1, \ - MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(ts, 0, 1, 0, 0, 0, 0, TS_F) \ -R(ts_rss, 0, 1, 0, 0, 0, 1, TS_F | RSS_F) \ -R(ts_ptype, 0, 1, 0, 0, 1, 0, TS_F | PTYPE_F) \ -R(ts_ptype_rss, 0, 1, 0, 0, 1, 1, TS_F | PTYPE_F | RSS_F) \ -R(ts_cksum, 0, 1, 0, 1, 0, 0, TS_F | CKSUM_F) \ -R(ts_cksum_rss, 0, 1, 0, 1, 0, 1, TS_F | CKSUM_F | RSS_F) \ -R(ts_cksum_ptype, 0, 1, 0, 1, 1, 0, TS_F | CKSUM_F | PTYPE_F) \ -R(ts_cksum_ptype_rss, 0, 1, 0, 1, 1, 1, \ - TS_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(ts_mark, 0, 1, 1, 0, 0, 0, TS_F | MARK_F) \ -R(ts_mark_rss, 0, 1, 1, 0, 0, 1, TS_F | MARK_F | RSS_F) \ -R(ts_mark_ptype, 0, 1, 1, 0, 1, 0, TS_F | MARK_F | PTYPE_F) \ -R(ts_mark_ptype_rss, 0, 1, 1, 0, 1, 1, \ - TS_F | MARK_F | PTYPE_F | RSS_F) \ -R(ts_mark_cksum, 0, 1, 1, 1, 0, 0, TS_F | MARK_F | CKSUM_F) \ -R(ts_mark_cksum_rss, 0, 1, 1, 1, 0, 1, \ - TS_F | MARK_F | CKSUM_F | RSS_F) \ -R(ts_mark_cksum_ptype, 0, 1, 1, 1, 1, 0, \ - TS_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(ts_mark_cksum_ptype_rss, 0, 1, 1, 1, 1, 1, \ - TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(vlan, 1, 0, 0, 0, 0, 0, RX_VLAN_F) \ -R(vlan_rss, 1, 0, 0, 0, 0, 1, RX_VLAN_F | RSS_F) \ -R(vlan_ptype, 1, 0, 0, 0, 1, 0, RX_VLAN_F | PTYPE_F) \ -R(vlan_ptype_rss, 1, 0, 0, 0, 1, 1, RX_VLAN_F | PTYPE_F | RSS_F) \ -R(vlan_cksum, 1, 0, 0, 1, 0, 0, RX_VLAN_F | CKSUM_F) \ -R(vlan_cksum_rss, 1, 0, 0, 1, 0, 1, RX_VLAN_F | CKSUM_F | RSS_F) \ -R(vlan_cksum_ptype, 1, 0, 0, 1, 1, 0, \ - RX_VLAN_F | CKSUM_F | PTYPE_F) \ -R(vlan_cksum_ptype_rss, 1, 0, 0, 1, 1, 1, \ - RX_VLAN_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(vlan_mark, 1, 0, 1, 0, 0, 0, RX_VLAN_F | MARK_F) \ -R(vlan_mark_rss, 1, 0, 1, 0, 0, 1, RX_VLAN_F | MARK_F | RSS_F) \ -R(vlan_mark_ptype, 1, 0, 1, 0, 1, 0, RX_VLAN_F | MARK_F | PTYPE_F)\ -R(vlan_mark_ptype_rss, 1, 0, 1, 0, 1, 1, \ - RX_VLAN_F | MARK_F | PTYPE_F | RSS_F) \ -R(vlan_mark_cksum, 1, 0, 1, 1, 0, 0, RX_VLAN_F | MARK_F | CKSUM_F)\ -R(vlan_mark_cksum_rss, 1, 0, 1, 1, 0, 1, \ - RX_VLAN_F | MARK_F | CKSUM_F | RSS_F) \ -R(vlan_mark_cksum_ptype, 1, 0, 1, 1, 1, 0, \ - RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(vlan_mark_cksum_ptype_rss, 1, 0, 1, 1, 1, 1, \ - RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(vlan_ts, 1, 1, 0, 0, 0, 0, RX_VLAN_F | TS_F) \ -R(vlan_ts_rss, 1, 1, 0, 0, 0, 1, RX_VLAN_F | TS_F | RSS_F) \ -R(vlan_ts_ptype, 1, 1, 0, 0, 1, 0, RX_VLAN_F | TS_F | PTYPE_F) \ -R(vlan_ts_ptype_rss, 1, 1, 0, 0, 1, 1, \ - RX_VLAN_F | TS_F | PTYPE_F | RSS_F) \ -R(vlan_ts_cksum, 1, 1, 0, 1, 0, 0, RX_VLAN_F | TS_F | CKSUM_F) \ -R(vlan_ts_cksum_rss, 1, 1, 0, 1, 0, 1, \ - RX_VLAN_F | TS_F | CKSUM_F | RSS_F) \ -R(vlan_ts_cksum_ptype, 1, 1, 0, 1, 1, 0, \ - RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F) \ -R(vlan_ts_cksum_ptype_rss, 1, 1, 0, 1, 1, 1, \ - RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(vlan_ts_mark, 1, 1, 1, 0, 0, 0, RX_VLAN_F | TS_F | MARK_F) \ -R(vlan_ts_mark_rss, 1, 1, 1, 0, 0, 1, \ - RX_VLAN_F | TS_F | MARK_F | RSS_F) \ -R(vlan_ts_mark_ptype, 1, 1, 1, 0, 1, 0, \ - RX_VLAN_F | TS_F | MARK_F | PTYPE_F) \ -R(vlan_ts_mark_ptype_rss, 1, 1, 1, 0, 1, 1, \ - RX_VLAN_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ -R(vlan_ts_mark_cksum, 1, 1, 1, 1, 0, 0, \ - RX_VLAN_F | TS_F | MARK_F | CKSUM_F) \ -R(vlan_ts_mark_cksum_rss, 1, 1, 1, 1, 0, 1, \ - RX_VLAN_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ -R(vlan_ts_mark_cksum_ptype, 1, 1, 1, 1, 1, 0, \ - RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(vlan_ts_mark_cksum_ptype_rss, 1, 1, 1, 1, 1, 1, \ - RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) +R(no_offload, 0, 0, 0, 0, 0, 0, 0, \ + NIX_RX_OFFLOAD_NONE) \ +R(rss, 0, 0, 0, 0, 0, 0, 1, \ + RSS_F) \ +R(ptype, 0, 0, 0, 0, 0, 1, 0, \ + PTYPE_F) \ +R(ptype_rss, 0, 0, 0, 0, 0, 1, 1, \ + PTYPE_F | RSS_F) \ +R(cksum, 0, 0, 0, 0, 1, 0, 0, \ + CKSUM_F) \ +R(cksum_rss, 0, 0, 0, 0, 1, 0, 1, \ + CKSUM_F | RSS_F) \ +R(cksum_ptype, 0, 0, 0, 0, 1, 1, 0, \ + CKSUM_F | PTYPE_F) \ +R(cksum_ptype_rss, 0, 0, 0, 0, 1, 1, 1, \ + CKSUM_F | PTYPE_F | RSS_F) \ +R(mark, 0, 0, 0, 1, 0, 0, 0, \ + MARK_F) \ +R(mark_rss, 0, 0, 0, 1, 0, 0, 1, \ + MARK_F | RSS_F) \ +R(mark_ptype, 0, 0, 0, 1, 0, 1, 0, \ + MARK_F | PTYPE_F) \ +R(mark_ptype_rss, 0, 0, 0, 1, 0, 1, 1, \ + MARK_F | PTYPE_F | RSS_F) \ +R(mark_cksum, 0, 0, 0, 1, 1, 0, 0, \ + MARK_F | CKSUM_F) \ +R(mark_cksum_rss, 0, 0, 0, 1, 1, 0, 1, \ + MARK_F | CKSUM_F | RSS_F) \ +R(mark_cksum_ptype, 0, 0, 0, 1, 1, 1, 0, \ + MARK_F | CKSUM_F | PTYPE_F) \ +R(mark_cksum_ptype_rss, 0, 0, 0, 1, 1, 1, 1, \ + MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(ts, 0, 0, 1, 0, 0, 0, 0, \ + TS_F) \ +R(ts_rss, 0, 0, 1, 0, 0, 0, 1, \ + TS_F | RSS_F) \ +R(ts_ptype, 0, 0, 1, 0, 0, 1, 0, \ + TS_F | PTYPE_F) \ +R(ts_ptype_rss, 0, 0, 1, 0, 0, 1, 1, \ + TS_F | PTYPE_F | RSS_F) \ +R(ts_cksum, 0, 0, 1, 0, 1, 0, 0, \ + TS_F | CKSUM_F) \ +R(ts_cksum_rss, 0, 0, 1, 0, 1, 0, 1, \ + TS_F | CKSUM_F | RSS_F) \ +R(ts_cksum_ptype, 0, 0, 1, 0, 1, 1, 0, \ + TS_F | CKSUM_F | PTYPE_F) \ +R(ts_cksum_ptype_rss, 0, 0, 1, 0, 1, 1, 1, \ + TS_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(ts_mark, 0, 0, 1, 1, 0, 0, 0, \ + TS_F | MARK_F) \ +R(ts_mark_rss, 0, 0, 1, 1, 0, 0, 1, \ + TS_F | MARK_F | RSS_F) \ +R(ts_mark_ptype, 0, 0, 1, 1, 0, 1, 0, \ + TS_F | MARK_F | PTYPE_F) \ +R(ts_mark_ptype_rss, 0, 0, 1, 1, 0, 1, 1, \ + TS_F | MARK_F | PTYPE_F | RSS_F) \ +R(ts_mark_cksum, 0, 0, 1, 1, 1, 0, 0, \ + TS_F | MARK_F | CKSUM_F) \ +R(ts_mark_cksum_rss, 0, 0, 1, 1, 1, 0, 1, \ + TS_F | MARK_F | CKSUM_F | RSS_F) \ +R(ts_mark_cksum_ptype, 0, 0, 1, 1, 1, 1, 0, \ + TS_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(ts_mark_cksum_ptype_rss, 0, 0, 1, 1, 1, 1, 1, \ + TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(vlan, 0, 1, 0, 0, 0, 0, 0, \ + RX_VLAN_F) \ +R(vlan_rss, 0, 1, 0, 0, 0, 0, 1, \ + RX_VLAN_F | RSS_F) \ +R(vlan_ptype, 0, 1, 0, 0, 0, 1, 0, \ + RX_VLAN_F | PTYPE_F) \ +R(vlan_ptype_rss, 0, 1, 0, 0, 0, 1, 1, \ + RX_VLAN_F | PTYPE_F | RSS_F) \ +R(vlan_cksum, 0, 1, 0, 0, 1, 0, 0, \ + RX_VLAN_F | CKSUM_F) \ +R(vlan_cksum_rss, 0, 1, 0, 0, 1, 0, 1, \ + RX_VLAN_F | CKSUM_F | RSS_F) \ +R(vlan_cksum_ptype, 0, 1, 0, 0, 1, 1, 0, \ + RX_VLAN_F | CKSUM_F | PTYPE_F) \ +R(vlan_cksum_ptype_rss, 0, 1, 0, 0, 1, 1, 1, \ + RX_VLAN_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(vlan_mark, 0, 1, 0, 1, 0, 0, 0, \ + RX_VLAN_F | MARK_F) \ +R(vlan_mark_rss, 0, 1, 0, 1, 0, 0, 1, \ + RX_VLAN_F | MARK_F | RSS_F) \ +R(vlan_mark_ptype, 0, 1, 0, 1, 0, 1, 0, \ + RX_VLAN_F | MARK_F | PTYPE_F) \ +R(vlan_mark_ptype_rss, 0, 1, 0, 1, 0, 1, 1, \ + RX_VLAN_F | MARK_F | PTYPE_F | RSS_F) \ +R(vlan_mark_cksum, 0, 1, 0, 1, 1, 0, 0, \ + RX_VLAN_F | MARK_F | CKSUM_F) \ +R(vlan_mark_cksum_rss, 0, 1, 0, 1, 1, 0, 1, \ + RX_VLAN_F | MARK_F | CKSUM_F | RSS_F) \ +R(vlan_mark_cksum_ptype, 0, 1, 0, 1, 1, 1, 0, \ + RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(vlan_mark_cksum_ptype_rss, 0, 1, 0, 1, 1, 1, 1, \ + RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(vlan_ts, 0, 1, 1, 0, 0, 0, 0, \ + RX_VLAN_F | TS_F) \ +R(vlan_ts_rss, 0, 1, 1, 0, 0, 0, 1, \ + RX_VLAN_F | TS_F | RSS_F) \ +R(vlan_ts_ptype, 0, 1, 1, 0, 0, 1, 0, \ + RX_VLAN_F | TS_F | PTYPE_F) \ +R(vlan_ts_ptype_rss, 0, 1, 1, 0, 0, 1, 1, \ + RX_VLAN_F | TS_F | PTYPE_F | RSS_F) \ +R(vlan_ts_cksum, 0, 1, 1, 0, 1, 0, 0, \ + RX_VLAN_F | TS_F | CKSUM_F) \ +R(vlan_ts_cksum_rss, 0, 1, 1, 0, 1, 0, 1, \ + RX_VLAN_F | TS_F | CKSUM_F | RSS_F) \ +R(vlan_ts_cksum_ptype, 0, 1, 1, 0, 1, 1, 0, \ + RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F) \ +R(vlan_ts_cksum_ptype_rss, 0, 1, 1, 0, 1, 1, 1, \ + RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(vlan_ts_mark, 0, 1, 1, 1, 0, 0, 0, \ + RX_VLAN_F | TS_F | MARK_F) \ +R(vlan_ts_mark_rss, 0, 1, 1, 1, 0, 0, 1, \ + RX_VLAN_F | TS_F | MARK_F | RSS_F) \ +R(vlan_ts_mark_ptype, 0, 1, 1, 1, 0, 1, 0, \ + RX_VLAN_F | TS_F | MARK_F | PTYPE_F) \ +R(vlan_ts_mark_ptype_rss, 0, 1, 1, 1, 0, 1, 1, \ + RX_VLAN_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ +R(vlan_ts_mark_cksum, 0, 1, 1, 1, 1, 0, 0, \ + RX_VLAN_F | TS_F | MARK_F | CKSUM_F) \ +R(vlan_ts_mark_cksum_rss, 0, 1, 1, 1, 1, 0, 1, \ + RX_VLAN_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ +R(vlan_ts_mark_cksum_ptype, 0, 1, 1, 1, 1, 1, 0, \ + RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(vlan_ts_mark_cksum_ptype_rss, 0, 1, 1, 1, 1, 1, 1, \ + RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec, 1, 0, 0, 0, 0, 0, 0, \ + R_SEC_F) \ +R(sec_rss, 1, 0, 0, 0, 0, 0, 1, \ + RSS_F) \ +R(sec_ptype, 1, 0, 0, 0, 0, 1, 0, \ + R_SEC_F | PTYPE_F) \ +R(sec_ptype_rss, 1, 0, 0, 0, 0, 1, 1, \ + R_SEC_F | PTYPE_F | RSS_F) \ +R(sec_cksum, 1, 0, 0, 0, 1, 0, 0, \ + R_SEC_F | CKSUM_F) \ +R(sec_cksum_rss, 1, 0, 0, 0, 1, 0, 1, \ + R_SEC_F | CKSUM_F | RSS_F) \ +R(sec_cksum_ptype, 1, 0, 0, 0, 1, 1, 0, \ + R_SEC_F | CKSUM_F | PTYPE_F) \ +R(sec_cksum_ptype_rss, 1, 0, 0, 0, 1, 1, 1, \ + R_SEC_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_mark, 1, 0, 0, 1, 0, 0, 0, \ + R_SEC_F | MARK_F) \ +R(sec_mark_rss, 1, 0, 0, 1, 0, 0, 1, \ + R_SEC_F | MARK_F | RSS_F) \ +R(sec_mark_ptype, 1, 0, 0, 1, 0, 1, 0, \ + R_SEC_F | MARK_F | PTYPE_F) \ +R(sec_mark_ptype_rss, 1, 0, 0, 1, 0, 1, 1, \ + R_SEC_F | MARK_F | PTYPE_F | RSS_F) \ +R(sec_mark_cksum, 1, 0, 0, 1, 1, 0, 0, \ + R_SEC_F | MARK_F | CKSUM_F) \ +R(sec_mark_cksum_rss, 1, 0, 0, 1, 1, 0, 1, \ + R_SEC_F | MARK_F | CKSUM_F | RSS_F) \ +R(sec_mark_cksum_ptype, 1, 0, 0, 1, 1, 1, 0, \ + R_SEC_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(sec_mark_cksum_ptype_rss, 1, 0, 0, 1, 1, 1, 1, \ + R_SEC_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_ts, 1, 0, 1, 0, 0, 0, 0, \ + R_SEC_F | TS_F) \ +R(sec_ts_rss, 1, 0, 1, 0, 0, 0, 1, \ + R_SEC_F | TS_F | RSS_F) \ +R(sec_ts_ptype, 1, 0, 1, 0, 0, 1, 0, \ + R_SEC_F | TS_F | PTYPE_F) \ +R(sec_ts_ptype_rss, 1, 0, 1, 0, 0, 1, 1, \ + R_SEC_F | TS_F | PTYPE_F | RSS_F) \ +R(sec_ts_cksum, 1, 0, 1, 0, 1, 0, 0, \ + R_SEC_F | TS_F | CKSUM_F) \ +R(sec_ts_cksum_rss, 1, 0, 1, 0, 1, 0, 1, \ + R_SEC_F | TS_F | CKSUM_F | RSS_F) \ +R(sec_ts_cksum_ptype, 1, 0, 1, 0, 1, 1, 0, \ + R_SEC_F | TS_F | CKSUM_F | PTYPE_F) \ +R(sec_ts_cksum_ptype_rss, 1, 0, 1, 0, 1, 1, 1, \ + R_SEC_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_ts_mark, 1, 0, 1, 1, 0, 0, 0, \ + R_SEC_F | TS_F | MARK_F) \ +R(sec_ts_mark_rss, 1, 0, 1, 1, 0, 0, 1, \ + R_SEC_F | TS_F | MARK_F | RSS_F) \ +R(sec_ts_mark_ptype, 1, 0, 1, 1, 0, 1, 0, \ + R_SEC_F | TS_F | MARK_F | PTYPE_F) \ +R(sec_ts_mark_ptype_rss, 1, 0, 1, 1, 0, 1, 1, \ + R_SEC_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ +R(sec_ts_mark_cksum, 1, 0, 1, 1, 1, 0, 0, \ + R_SEC_F | TS_F | MARK_F | CKSUM_F) \ +R(sec_ts_mark_cksum_rss, 1, 0, 1, 1, 1, 0, 1, \ + R_SEC_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ +R(sec_ts_mark_cksum_ptype, 1, 0, 1, 1, 1, 1, 0, \ + R_SEC_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(sec_ts_mark_cksum_ptype_rss, 1, 0, 1, 1, 1, 1, 1, \ + R_SEC_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_vlan, 1, 1, 0, 0, 0, 0, 0, \ + R_SEC_F | RX_VLAN_F) \ +R(sec_vlan_rss, 1, 1, 0, 0, 0, 0, 1, \ + R_SEC_F | RX_VLAN_F | RSS_F) \ +R(sec_vlan_ptype, 1, 1, 0, 0, 0, 1, 0, \ + R_SEC_F | RX_VLAN_F | PTYPE_F) \ +R(sec_vlan_ptype_rss, 1, 1, 0, 0, 0, 1, 1, \ + R_SEC_F | RX_VLAN_F | PTYPE_F | RSS_F) \ +R(sec_vlan_cksum, 1, 1, 0, 0, 1, 0, 0, \ + R_SEC_F | RX_VLAN_F | CKSUM_F) \ +R(sec_vlan_cksum_rss, 1, 1, 0, 0, 1, 0, 1, \ + R_SEC_F | RX_VLAN_F | CKSUM_F | RSS_F) \ +R(sec_vlan_cksum_ptype, 1, 1, 0, 0, 1, 1, 0, \ + R_SEC_F | RX_VLAN_F | CKSUM_F | PTYPE_F) \ +R(sec_vlan_cksum_ptype_rss, 1, 1, 0, 0, 1, 1, 1, \ + R_SEC_F | RX_VLAN_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_vlan_mark, 1, 1, 0, 1, 0, 0, 0, \ + R_SEC_F | RX_VLAN_F | MARK_F) \ +R(sec_vlan_mark_rss, 1, 1, 0, 1, 0, 0, 1, \ + R_SEC_F | RX_VLAN_F | MARK_F | RSS_F) \ +R(sec_vlan_mark_ptype, 1, 1, 0, 1, 0, 1, 0, \ + R_SEC_F | RX_VLAN_F | MARK_F | PTYPE_F) \ +R(sec_vlan_mark_ptype_rss, 1, 1, 0, 1, 0, 1, 1, \ + R_SEC_F | RX_VLAN_F | MARK_F | PTYPE_F | RSS_F) \ +R(sec_vlan_mark_cksum, 1, 1, 0, 1, 1, 0, 0, \ + R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F) \ +R(sec_vlan_mark_cksum_rss, 1, 1, 0, 1, 1, 0, 1, \ + R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | RSS_F) \ +R(sec_vlan_mark_cksum_ptype, 1, 1, 0, 1, 1, 1, 0, \ + R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(sec_vlan_mark_cksum_ptype_rss, 1, 1, 0, 1, 1, 1, 1, \ + R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_vlan_ts, 1, 1, 1, 0, 0, 0, 0, \ + R_SEC_F | RX_VLAN_F | TS_F) \ +R(sec_vlan_ts_rss, 1, 1, 1, 0, 0, 0, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | RSS_F) \ +R(sec_vlan_ts_ptype, 1, 1, 1, 0, 0, 1, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | PTYPE_F) \ +R(sec_vlan_ts_ptype_rss, 1, 1, 1, 0, 0, 1, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | PTYPE_F | RSS_F) \ +R(sec_vlan_ts_cksum, 1, 1, 1, 0, 1, 0, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F) \ +R(sec_vlan_ts_cksum_rss, 1, 1, 1, 0, 1, 0, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | RSS_F) \ +R(sec_vlan_ts_cksum_ptype, 1, 1, 1, 0, 1, 1, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F) \ +R(sec_vlan_ts_cksum_ptype_rss, 1, 1, 1, 0, 1, 1, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ +R(sec_vlan_ts_mark, 1, 1, 1, 1, 0, 0, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F) \ +R(sec_vlan_ts_mark_rss, 1, 1, 1, 1, 0, 0, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | RSS_F) \ +R(sec_vlan_ts_mark_ptype, 1, 1, 1, 1, 0, 1, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | PTYPE_F) \ +R(sec_vlan_ts_mark_ptype_rss, 1, 1, 1, 1, 0, 1, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ +R(sec_vlan_ts_mark_cksum, 1, 1, 1, 1, 1, 0, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F) \ +R(sec_vlan_ts_mark_cksum_rss, 1, 1, 1, 1, 1, 0, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ +R(sec_vlan_ts_mark_cksum_ptype, 1, 1, 1, 1, 1, 1, 0, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ +R(sec_vlan_ts_mark_cksum_ptype_rss, 1, 1, 1, 1, 1, 1, 1, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts); \ \ diff --git a/drivers/net/cnxk/cn10k_rx_mseg.c b/drivers/net/cnxk/cn10k_rx_mseg.c index 3340771..e7c2321 100644 --- a/drivers/net/cnxk/cn10k_rx_mseg.c +++ b/drivers/net/cnxk/cn10k_rx_mseg.c @@ -5,7 +5,7 @@ #include "cn10k_ethdev.h" #include "cn10k_rx.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_mseg_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ diff --git a/drivers/net/cnxk/cn10k_rx_vec.c b/drivers/net/cnxk/cn10k_rx_vec.c index 166735a..0ccc4df 100644 --- a/drivers/net/cnxk/cn10k_rx_vec.c +++ b/drivers/net/cnxk/cn10k_rx_vec.c @@ -5,14 +5,14 @@ #include "cn10k_ethdev.h" #include "cn10k_rx.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot \ cn10k_nix_recv_pkts_vec_##name(void *rx_queue, \ struct rte_mbuf **rx_pkts, \ uint16_t pkts) \ { \ return cn10k_nix_recv_pkts_vector(rx_queue, rx_pkts, pkts, \ - (flags), NULL, NULL); \ + (flags), NULL, NULL, 0); \ } NIX_RX_FASTPATH_MODES diff --git a/drivers/net/cnxk/cn10k_rx_vec_mseg.c b/drivers/net/cnxk/cn10k_rx_vec_mseg.c index 1f44ddd..38e0ec3 100644 --- a/drivers/net/cnxk/cn10k_rx_vec_mseg.c +++ b/drivers/net/cnxk/cn10k_rx_vec_mseg.c @@ -5,13 +5,13 @@ #include "cn10k_ethdev.h" #include "cn10k_rx.h" -#define R(name, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_vec_mseg_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ return cn10k_nix_recv_pkts_vector( \ rx_queue, rx_pkts, pkts, (flags) | NIX_RX_MULTI_SEG_F, \ - NULL, NULL); \ + NULL, NULL, 0); \ } NIX_RX_FASTPATH_MODES diff --git a/drivers/net/cnxk/cn10k_tx.h b/drivers/net/cnxk/cn10k_tx.h index 8577a7b..c81a612 100644 --- a/drivers/net/cnxk/cn10k_tx.h +++ b/drivers/net/cnxk/cn10k_tx.h @@ -51,9 +51,6 @@ #define NIX_NB_SEGS_TO_SEGDW(x) ((NIX_SEGDW_MAGIC >> ((x) << 2)) & 0xF) -#define LMT_OFF(lmt_addr, lmt_num, offset) \ - (void *)((lmt_addr) + ((lmt_num) << ROC_LMT_LINE_SIZE_LOG2) + (offset)) - /* Function to determine no of tx subdesc required in case ext * sub desc is enabled. */ From patchwork Thu Sep 2 02:14:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97746 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 13A01A0C4D; Thu, 2 Sep 2021 04:19:03 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2B42941170; Thu, 2 Sep 2021 04:17:49 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id E26684118F for ; Thu, 2 Sep 2021 04:17:47 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQCpr011801 for ; Wed, 1 Sep 2021 19:17:47 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=tD/uGcBpBxmqiO+M0wobyY3UqvnSv1qCUGZ53ywg8dc=; b=AQaxBhiEqh9Wr/HpH/SwTSNiZObDt58HoyiQdrfE8VGgZ4kPzW5uwsvKoOaLqcxraIlB 7k0JIFlewvFiIvFpwEEyFQ37IbLrplpV9j3nxmMpwmYJby5zN7ANydg8W+TrhqCl6xIi juEr/U4IwOP6OyEBW5C2YqPl4bhZF+LkTJC6taXWDY2nfOVOQO7c6VqO2Ft/KHLTt/Cd kpZnfX2z7I/8mHU/vS4qLd17ft+azE8YzbjbobMi1MUp0QS+RQH64JIoQ/1obl+/3gt4 8polimz8EZmlGPlKG8J9SOX1OC1vVaxsk59x4lZc5Helw29AfVx6BIGmfLZ82lnkrS5d uw== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hue-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:46 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:44 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:44 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id F200D3F704B; Wed, 1 Sep 2021 19:17:41 -0700 (PDT) From: Nithin Dabilpuram To: Pavan Nikhilesh , Shijith Thotton , Nithin Dabilpuram , "Kiran Kumar K" , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:58 +0530 Message-ID: <20210902021505.17607-21-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: sNqtDjLX1Rk_PszcTa2JhPPqa_gGWh7G X-Proofpoint-GUID: sNqtDjLX1Rk_PszcTa2JhPPqa_gGWh7G X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 20/27] net/cnxk: add cn10k Tx support for security offload X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add support to create and submit CPT instructions on Tx. Signed-off-by: Nithin Dabilpuram --- doc/guides/rel_notes/release_21_11.rst | 5 + drivers/event/cnxk/cn10k_eventdev.c | 15 +- drivers/event/cnxk/cn10k_worker.h | 74 +- drivers/event/cnxk/cn10k_worker_tx_enq.c | 2 +- drivers/event/cnxk/cn10k_worker_tx_enq_seg.c | 2 +- drivers/net/cnxk/cn10k_tx.c | 31 +- drivers/net/cnxk/cn10k_tx.h | 981 +++++++++++++++++++++++---- drivers/net/cnxk/cn10k_tx_mseg.c | 2 +- drivers/net/cnxk/cn10k_tx_vec.c | 2 +- drivers/net/cnxk/cn10k_tx_vec_mseg.c | 2 +- 10 files changed, 934 insertions(+), 182 deletions(-) diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index fb599e5..a87f6cb 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -65,6 +65,11 @@ New Features * Added event crypto adapter OP_FORWARD mode support. +* **Added support for Inline IPsec on Marvell CN10K and CN9K.** + + * Added support for Inline IPsec in net/cnxk PMD for CN9K event mode + and CN10K poll mode and event mode. + Removed Items ------------- diff --git a/drivers/event/cnxk/cn10k_eventdev.c b/drivers/event/cnxk/cn10k_eventdev.c index 2f2e7f8..bd1cf55 100644 --- a/drivers/event/cnxk/cn10k_eventdev.c +++ b/drivers/event/cnxk/cn10k_eventdev.c @@ -16,7 +16,8 @@ [!!(dev->rx_offloads & NIX_RX_OFFLOAD_RSS_F)] #define CN10K_SET_EVDEV_ENQ_OP(dev, enq_op, enq_ops) \ - enq_op = enq_ops[!!(dev->tx_offloads & NIX_TX_OFFLOAD_TSTAMP_F)] \ + enq_op = enq_ops[!!(dev->tx_offloads & NIX_TX_OFFLOAD_SECURITY_F)] \ + [!!(dev->tx_offloads & NIX_TX_OFFLOAD_TSTAMP_F)] \ [!!(dev->tx_offloads & NIX_TX_OFFLOAD_TSO_F)] \ [!!(dev->tx_offloads & NIX_TX_OFFLOAD_MBUF_NOFF_F)] \ [!!(dev->tx_offloads & NIX_TX_OFFLOAD_VLAN_QINQ_F)] \ @@ -379,17 +380,17 @@ cn10k_sso_fp_fns_set(struct rte_eventdev *event_dev) /* Tx modes */ const event_tx_adapter_enqueue - sso_hws_tx_adptr_enq[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_tx_adptr_enq_##name, + sso_hws_tx_adptr_enq[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_tx_adptr_enq_##name, NIX_TX_FASTPATH_MODES #undef T }; const event_tx_adapter_enqueue - sso_hws_tx_adptr_enq_seg[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_tx_adptr_enq_seg_##name, + sso_hws_tx_adptr_enq_seg[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_tx_adptr_enq_seg_##name, NIX_TX_FASTPATH_MODES #undef T }; diff --git a/drivers/event/cnxk/cn10k_worker.h b/drivers/event/cnxk/cn10k_worker.h index b79bd90..1255662 100644 --- a/drivers/event/cnxk/cn10k_worker.h +++ b/drivers/event/cnxk/cn10k_worker.h @@ -423,7 +423,11 @@ cn10k_sso_vwqe_split_tx(struct rte_mbuf **mbufs, uint16_t nb_mbufs, ((queue[0] ^ queue[1]) & (queue[2] ^ queue[3]))) { for (j = 0; j < 4; j++) { + uint8_t lnum = 0, loff = 0, shft = 0; struct rte_mbuf *m = mbufs[i + j]; + uintptr_t laddr; + uint16_t segdw; + bool sec; txq = (struct cn10k_eth_txq *) txq_data[port[j]][queue[j]]; @@ -434,19 +438,35 @@ cn10k_sso_vwqe_split_tx(struct rte_mbuf **mbufs, uint16_t nb_mbufs, if (flags & NIX_TX_OFFLOAD_TSO_F) cn10k_nix_xmit_prepare_tso(m, flags); - cn10k_nix_xmit_prepare(m, cmd, lmt_addr, flags, - txq->lso_tun_fmt); + cn10k_nix_xmit_prepare(m, cmd, flags, + txq->lso_tun_fmt, &sec); + + laddr = lmt_addr; + /* Prepare CPT instruction and get nixtx addr if + * it is for CPT on same lmtline. + */ + if (flags & NIX_TX_OFFLOAD_SECURITY_F && sec) + cn10k_nix_prep_sec(m, cmd, &laddr, + lmt_addr, &lnum, + &loff, &shft, + txq->sa_base, flags); + + /* Move NIX desc to LMT/NIXTX area */ + cn10k_nix_xmit_mv_lmt_base(laddr, cmd, flags); + if (flags & NIX_TX_MULTI_SEG_F) { - const uint16_t segdw = - cn10k_nix_prepare_mseg( - m, (uint64_t *)lmt_addr, - flags); - pa = txq->io_addr | ((segdw - 1) << 4); + segdw = cn10k_nix_prepare_mseg(m, + (uint64_t *)laddr, flags); } else { - pa = txq->io_addr | - (cn10k_nix_tx_ext_subs(flags) + 1) - << 4; + segdw = cn10k_nix_tx_ext_subs(flags) + + 2; } + + if (flags & NIX_TX_OFFLOAD_SECURITY_F && sec) + pa = txq->cpt_io_addr | 3 << 4; + else + pa = txq->io_addr | ((segdw - 1) << 4); + if (!sched_type) roc_sso_hws_head_wait(base + SSOW_LF_GWS_TAG); @@ -469,15 +489,19 @@ cn10k_sso_hws_event_tx(struct cn10k_sso_hws *ws, struct rte_event *ev, const uint64_t txq_data[][RTE_MAX_QUEUES_PER_PORT], const uint32_t flags) { + uint8_t lnum = 0, loff = 0, shft = 0; struct cn10k_eth_txq *txq; + uint16_t ref_cnt, segdw; struct rte_mbuf *m; uintptr_t lmt_addr; - uint16_t ref_cnt; + uintptr_t c_laddr; uint16_t lmt_id; uintptr_t pa; + bool sec; lmt_addr = ws->lmt_base; ROC_LMT_BASE_ID_GET(lmt_addr, lmt_id); + c_laddr = lmt_addr; if (ev->event_type & RTE_EVENT_TYPE_VECTOR) { struct rte_mbuf **mbufs = ev->vec->mbufs; @@ -508,14 +532,28 @@ cn10k_sso_hws_event_tx(struct cn10k_sso_hws *ws, struct rte_event *ev, if (flags & NIX_TX_OFFLOAD_TSO_F) cn10k_nix_xmit_prepare_tso(m, flags); - cn10k_nix_xmit_prepare(m, cmd, lmt_addr, flags, txq->lso_tun_fmt); + cn10k_nix_xmit_prepare(m, cmd, flags, txq->lso_tun_fmt, &sec); + + /* Prepare CPT instruction and get nixtx addr if + * it is for CPT on same lmtline. + */ + if (flags & NIX_TX_OFFLOAD_SECURITY_F && sec) + cn10k_nix_prep_sec(m, cmd, &lmt_addr, c_laddr, &lnum, &loff, + &shft, txq->sa_base, flags); + + /* Move NIX desc to LMT/NIXTX area */ + cn10k_nix_xmit_mv_lmt_base(lmt_addr, cmd, flags); if (flags & NIX_TX_MULTI_SEG_F) { - const uint16_t segdw = - cn10k_nix_prepare_mseg(m, (uint64_t *)lmt_addr, flags); + segdw = cn10k_nix_prepare_mseg(m, (uint64_t *)lmt_addr, flags); + } else { + segdw = cn10k_nix_tx_ext_subs(flags) + 2; + } + + if (flags & NIX_TX_OFFLOAD_SECURITY_F && sec) + pa = txq->cpt_io_addr | 3 << 4; + else pa = txq->io_addr | ((segdw - 1) << 4); - } else { - pa = txq->io_addr | (cn10k_nix_tx_ext_subs(flags) + 1) << 4; - } + if (!ev->sched_type) roc_sso_hws_head_wait(ws->tx_base + SSOW_LF_GWS_TAG); @@ -531,7 +569,7 @@ cn10k_sso_hws_event_tx(struct cn10k_sso_hws *ws, struct rte_event *ev, return 1; } -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_hot cn10k_sso_hws_tx_adptr_enq_##name( \ void *port, struct rte_event ev[], uint16_t nb_events); \ uint16_t __rte_hot cn10k_sso_hws_tx_adptr_enq_seg_##name( \ diff --git a/drivers/event/cnxk/cn10k_worker_tx_enq.c b/drivers/event/cnxk/cn10k_worker_tx_enq.c index f9968ac..f14c7fc 100644 --- a/drivers/event/cnxk/cn10k_worker_tx_enq.c +++ b/drivers/event/cnxk/cn10k_worker_tx_enq.c @@ -4,7 +4,7 @@ #include "cn10k_worker.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_hot cn10k_sso_hws_tx_adptr_enq_##name( \ void *port, struct rte_event ev[], uint16_t nb_events) \ { \ diff --git a/drivers/event/cnxk/cn10k_worker_tx_enq_seg.c b/drivers/event/cnxk/cn10k_worker_tx_enq_seg.c index a24fc42..2ea61e5 100644 --- a/drivers/event/cnxk/cn10k_worker_tx_enq_seg.c +++ b/drivers/event/cnxk/cn10k_worker_tx_enq_seg.c @@ -4,7 +4,7 @@ #include "cn10k_worker.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_hot cn10k_sso_hws_tx_adptr_enq_seg_##name( \ void *port, struct rte_event ev[], uint16_t nb_events) \ { \ diff --git a/drivers/net/cnxk/cn10k_tx.c b/drivers/net/cnxk/cn10k_tx.c index 0e1276c..eb962ef 100644 --- a/drivers/net/cnxk/cn10k_tx.c +++ b/drivers/net/cnxk/cn10k_tx.c @@ -5,7 +5,7 @@ #include "cn10k_ethdev.h" #include "cn10k_tx.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_xmit_pkts_##name( \ void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts) \ { \ @@ -24,12 +24,13 @@ NIX_TX_FASTPATH_MODES static inline void pick_tx_func(struct rte_eth_dev *eth_dev, - const eth_tx_burst_t tx_burst[2][2][2][2][2][2]) + const eth_tx_burst_t tx_burst[2][2][2][2][2][2][2]) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - /* [TSP] [TSO] [NOFF] [VLAN] [OL3_OL4_CSUM] [IL3_IL4_CSUM] */ + /* [SEC] [TSP] [TSO] [NOFF] [VLAN] [OL3_OL4_CSUM] [IL3_IL4_CSUM] */ eth_dev->tx_pkt_burst = tx_burst + [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_SECURITY_F)] [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_TSTAMP_F)] [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_TSO_F)] [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_MBUF_NOFF_F)] @@ -43,33 +44,33 @@ cn10k_eth_set_tx_function(struct rte_eth_dev *eth_dev) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - const eth_tx_burst_t nix_eth_tx_burst[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_##name, + const eth_tx_burst_t nix_eth_tx_burst[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_##name, NIX_TX_FASTPATH_MODES #undef T }; - const eth_tx_burst_t nix_eth_tx_burst_mseg[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_mseg_##name, + const eth_tx_burst_t nix_eth_tx_burst_mseg[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_mseg_##name, NIX_TX_FASTPATH_MODES #undef T }; - const eth_tx_burst_t nix_eth_tx_vec_burst[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_vec_##name, + const eth_tx_burst_t nix_eth_tx_vec_burst[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_vec_##name, NIX_TX_FASTPATH_MODES #undef T }; - const eth_tx_burst_t nix_eth_tx_vec_burst_mseg[2][2][2][2][2][2] = { -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ - [f5][f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_vec_mseg_##name, + const eth_tx_burst_t nix_eth_tx_vec_burst_mseg[2][2][2][2][2][2][2] = { +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_vec_mseg_##name, NIX_TX_FASTPATH_MODES #undef T diff --git a/drivers/net/cnxk/cn10k_tx.h b/drivers/net/cnxk/cn10k_tx.h index c81a612..70ba929 100644 --- a/drivers/net/cnxk/cn10k_tx.h +++ b/drivers/net/cnxk/cn10k_tx.h @@ -6,6 +6,8 @@ #include +#include + #define NIX_TX_OFFLOAD_NONE (0) #define NIX_TX_OFFLOAD_L3_L4_CSUM_F BIT(0) #define NIX_TX_OFFLOAD_OL3_OL4_CSUM_F BIT(1) @@ -57,12 +59,22 @@ static __rte_always_inline int cn10k_nix_tx_ext_subs(const uint16_t flags) { - return (flags & NIX_TX_OFFLOAD_TSTAMP_F) - ? 2 - : ((flags & - (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F)) - ? 1 - : 0); + return (flags & NIX_TX_OFFLOAD_TSTAMP_F) ? + 2 : + ((flags & + (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F)) ? + 1 : + 0); +} + +static __rte_always_inline uint8_t +cn10k_nix_tx_dwords(const uint16_t flags, const uint8_t segdw) +{ + if (!(flags & NIX_TX_MULTI_SEG_F)) + return cn10k_nix_tx_ext_subs(flags) + 2; + + /* Already everything is accounted for in segdw */ + return segdw; } static __rte_always_inline uint8_t @@ -144,6 +156,34 @@ cn10k_nix_tx_steor_vec_data(const uint16_t flags) return data; } +static __rte_always_inline uint64_t +cn10k_cpt_tx_steor_data(void) +{ + /* We have two CPT instructions per LMTLine */ + const uint64_t dw_m1 = ROC_CN10K_TWO_CPT_INST_DW_M1; + uint64_t data; + + /* This will be moved to addr area */ + data = dw_m1 << 16; + data |= dw_m1 << 19; + data |= dw_m1 << 22; + data |= dw_m1 << 25; + data |= dw_m1 << 28; + data |= dw_m1 << 31; + data |= dw_m1 << 34; + data |= dw_m1 << 37; + data |= dw_m1 << 40; + data |= dw_m1 << 43; + data |= dw_m1 << 46; + data |= dw_m1 << 49; + data |= dw_m1 << 52; + data |= dw_m1 << 55; + data |= dw_m1 << 58; + data |= dw_m1 << 61; + + return data; +} + static __rte_always_inline void cn10k_nix_tx_skeleton(const struct cn10k_eth_txq *txq, uint64_t *cmd, const uint16_t flags) @@ -165,6 +205,236 @@ cn10k_nix_tx_skeleton(const struct cn10k_eth_txq *txq, uint64_t *cmd, } static __rte_always_inline void +cn10k_nix_sec_steorl(uintptr_t io_addr, uint32_t lmt_id, uint8_t lnum, + uint8_t loff, uint8_t shft) +{ + uint64_t data; + uintptr_t pa; + + /* Check if there is any CPT instruction to submit */ + if (!lnum && !loff) + return; + + data = cn10k_cpt_tx_steor_data(); + /* Update lmtline use for partial end line */ + if (loff) { + data &= ~(0x7ULL << shft); + /* Update it to half full i.e 64B */ + data |= (0x3UL << shft); + } + + pa = io_addr | ((data >> 16) & 0x7) << 4; + data &= ~(0x7ULL << 16); + /* Update lines - 1 that contain valid data */ + data |= ((uint64_t)(lnum + loff - 1)) << 12; + data |= lmt_id; + + /* STEOR */ + roc_lmt_submit_steorl(data, pa); +} + +#if defined(RTE_ARCH_ARM64) +static __rte_always_inline void +cn10k_nix_prep_sec_vec(struct rte_mbuf *m, uint64x2_t *cmd0, uint64x2_t *cmd1, + uintptr_t *nixtx_addr, uintptr_t lbase, uint8_t *lnum, + uint8_t *loff, uint8_t *shft, uint64_t sa_base, + const uint16_t flags) +{ + struct cn10k_sec_sess_priv sess_priv; + uint32_t pkt_len, dlen_adj, rlen; + uint64x2_t cmd01, cmd23; + uintptr_t dptr, nixtx; + uint64_t ucode_cmd[4]; + uint64_t *laddr; + uint8_t l2_len; + uint16_t tag; + uint64_t sa; + + sess_priv.u64 = *rte_security_dynfield(m); + + if (flags & NIX_TX_NEED_SEND_HDR_W1) + l2_len = vgetq_lane_u8(*cmd0, 8); + else + l2_len = m->l2_len; + + /* Retrieve DPTR */ + dptr = vgetq_lane_u64(*cmd1, 1); + pkt_len = vgetq_lane_u16(*cmd0, 0); + + /* Calculate dlen adj */ + dlen_adj = pkt_len - l2_len; + rlen = (dlen_adj + sess_priv.roundup_len) + + (sess_priv.roundup_byte - 1); + rlen &= ~(uint64_t)(sess_priv.roundup_byte - 1); + rlen += sess_priv.partial_len; + dlen_adj = rlen - dlen_adj; + + /* Update send descriptors. Security is single segment only */ + *cmd0 = vsetq_lane_u16(pkt_len + dlen_adj, *cmd0, 0); + *cmd1 = vsetq_lane_u16(pkt_len + dlen_adj, *cmd1, 0); + + /* Get area where NIX descriptor needs to be stored */ + nixtx = dptr + pkt_len + dlen_adj; + nixtx += BIT_ULL(7); + nixtx = (nixtx - 1) & ~(BIT_ULL(7) - 1); + + /* Return nixtx addr */ + *nixtx_addr = (nixtx + 16); + + /* DLEN passed is excluding L2HDR */ + pkt_len -= l2_len; + tag = sa_base & 0xFFFFUL; + sa_base &= ~0xFFFFUL; + sa = (uintptr_t)roc_nix_inl_ot_ipsec_outb_sa(sa_base, sess_priv.sa_idx); + ucode_cmd[3] = (ROC_CPT_DFLT_ENG_GRP_SE_IE << 61 | 1UL << 60 | sa); + ucode_cmd[0] = + (ROC_IE_OT_MAJOR_OP_PROCESS_OUTBOUND_IPSEC << 48 | pkt_len); + + /* CPT Word 0 and Word 1 */ + cmd01 = vdupq_n_u64((nixtx + 16) | (cn10k_nix_tx_ext_subs(flags) + 1)); + /* CPT_RES_S is 16B above NIXTX */ + cmd01 = vsetq_lane_u8(nixtx & BIT_ULL(7), cmd01, 8); + + /* CPT word 2 and 3 */ + cmd23 = vdupq_n_u64(0); + cmd23 = vsetq_lane_u64((((uint64_t)RTE_EVENT_TYPE_CPU << 28) | tag | + CNXK_ETHDEV_SEC_OUTB_EV_SUB << 20), cmd23, 0); + cmd23 = vsetq_lane_u64((uintptr_t)m | 1, cmd23, 1); + + dptr += l2_len; + ucode_cmd[1] = dptr; + ucode_cmd[2] = dptr; + + /* Move to our line */ + laddr = LMT_OFF(lbase, *lnum, *loff ? 64 : 0); + + /* Write CPT instruction to lmt line */ + vst1q_u64(laddr, cmd01); + vst1q_u64((laddr + 2), cmd23); + + *(__uint128_t *)(laddr + 4) = *(__uint128_t *)ucode_cmd; + *(__uint128_t *)(laddr + 6) = *(__uint128_t *)(ucode_cmd + 2); + + /* Move to next line for every other CPT inst */ + *loff = !(*loff); + *lnum = *lnum + (*loff ? 0 : 1); + *shft = *shft + (*loff ? 0 : 3); +} + +static __rte_always_inline void +cn10k_nix_prep_sec(struct rte_mbuf *m, uint64_t *cmd, uintptr_t *nixtx_addr, + uintptr_t lbase, uint8_t *lnum, uint8_t *loff, uint8_t *shft, + uint64_t sa_base, const uint16_t flags) +{ + struct cn10k_sec_sess_priv sess_priv; + uint32_t pkt_len, dlen_adj, rlen; + struct nix_send_hdr_s *send_hdr; + uint64x2_t cmd01, cmd23; + union nix_send_sg_s *sg; + uintptr_t dptr, nixtx; + uint64_t ucode_cmd[4]; + uint64_t *laddr; + uint8_t l2_len; + uint16_t tag; + uint64_t sa; + + /* Move to our line from base */ + sess_priv.u64 = *rte_security_dynfield(m); + send_hdr = (struct nix_send_hdr_s *)cmd; + if (flags & NIX_TX_NEED_EXT_HDR) + sg = (union nix_send_sg_s *)&cmd[4]; + else + sg = (union nix_send_sg_s *)&cmd[2]; + + if (flags & NIX_TX_NEED_SEND_HDR_W1) + l2_len = cmd[1] & 0xFF; + else + l2_len = m->l2_len; + + /* Retrieve DPTR */ + dptr = *(uint64_t *)(sg + 1); + pkt_len = send_hdr->w0.total; + + /* Calculate dlen adj */ + dlen_adj = pkt_len - l2_len; + rlen = (dlen_adj + sess_priv.roundup_len) + + (sess_priv.roundup_byte - 1); + rlen &= ~(uint64_t)(sess_priv.roundup_byte - 1); + rlen += sess_priv.partial_len; + dlen_adj = rlen - dlen_adj; + + /* Update send descriptors. Security is single segment only */ + send_hdr->w0.total = pkt_len + dlen_adj; + sg->seg1_size = pkt_len + dlen_adj; + + /* Get area where NIX descriptor needs to be stored */ + nixtx = dptr + pkt_len + dlen_adj; + nixtx += BIT_ULL(7); + nixtx = (nixtx - 1) & ~(BIT_ULL(7) - 1); + + /* Return nixtx addr */ + *nixtx_addr = (nixtx + 16); + + /* DLEN passed is excluding L2HDR */ + pkt_len -= l2_len; + tag = sa_base & 0xFFFFUL; + sa_base &= ~0xFFFFUL; + sa = (uintptr_t)roc_nix_inl_ot_ipsec_outb_sa(sa_base, sess_priv.sa_idx); + ucode_cmd[3] = (ROC_CPT_DFLT_ENG_GRP_SE_IE << 61 | 1UL << 60 | sa); + ucode_cmd[0] = + (ROC_IE_OT_MAJOR_OP_PROCESS_OUTBOUND_IPSEC << 48 | pkt_len); + + /* CPT Word 0 and Word 1. Assume no multi-seg support */ + cmd01 = vdupq_n_u64((nixtx + 16) | (cn10k_nix_tx_ext_subs(flags) + 1)); + /* CPT_RES_S is 16B above NIXTX */ + cmd01 = vsetq_lane_u8(nixtx & BIT_ULL(7), cmd01, 8); + + /* CPT word 2 and 3 */ + cmd23 = vdupq_n_u64(0); + cmd23 = vsetq_lane_u64((((uint64_t)RTE_EVENT_TYPE_CPU << 28) | tag | + CNXK_ETHDEV_SEC_OUTB_EV_SUB << 20), cmd23, 0); + cmd23 = vsetq_lane_u64((uintptr_t)m | 1, cmd23, 1); + + dptr += l2_len; + ucode_cmd[1] = dptr; + ucode_cmd[2] = dptr; + + /* Move to our line */ + laddr = LMT_OFF(lbase, *lnum, *loff ? 64 : 0); + + /* Write CPT instruction to lmt line */ + vst1q_u64(laddr, cmd01); + vst1q_u64((laddr + 2), cmd23); + + *(__uint128_t *)(laddr + 4) = *(__uint128_t *)ucode_cmd; + *(__uint128_t *)(laddr + 6) = *(__uint128_t *)(ucode_cmd + 2); + + /* Move to next line for every other CPT inst */ + *loff = !(*loff); + *lnum = *lnum + (*loff ? 0 : 1); + *shft = *shft + (*loff ? 0 : 3); +} + +#else + +static __rte_always_inline void +cn10k_nix_prep_sec(struct rte_mbuf *m, uint64_t *cmd, uintptr_t *nixtx_addr, + uintptr_t lbase, uint8_t *lnum, uint8_t *loff, uint8_t *shft, + uint64_t sa_base, const uint16_t flags) +{ + RTE_SET_USED(m); + RTE_SET_USED(cmd); + RTE_SET_USED(nixtx_addr); + RTE_SET_USED(lbase); + RTE_SET_USED(lnum); + RTE_SET_USED(loff); + RTE_SET_USED(shft); + RTE_SET_USED(sa_base); + RTE_SET_USED(flags); +} +#endif + +static __rte_always_inline void cn10k_nix_xmit_prepare_tso(struct rte_mbuf *m, const uint64_t flags) { uint64_t mask, ol_flags = m->ol_flags; @@ -217,8 +487,8 @@ cn10k_nix_xmit_prepare_tso(struct rte_mbuf *m, const uint64_t flags) } static __rte_always_inline void -cn10k_nix_xmit_prepare(struct rte_mbuf *m, uint64_t *cmd, uintptr_t lmt_addr, - const uint16_t flags, const uint64_t lso_tun_fmt) +cn10k_nix_xmit_prepare(struct rte_mbuf *m, uint64_t *cmd, const uint16_t flags, + const uint64_t lso_tun_fmt, bool *sec) { struct nix_send_ext_s *send_hdr_ext; struct nix_send_hdr_s *send_hdr; @@ -237,16 +507,16 @@ cn10k_nix_xmit_prepare(struct rte_mbuf *m, uint64_t *cmd, uintptr_t lmt_addr, sg = (union nix_send_sg_s *)(cmd + 2); } - if (flags & NIX_TX_NEED_SEND_HDR_W1) { + if (flags & (NIX_TX_NEED_SEND_HDR_W1 | NIX_TX_OFFLOAD_SECURITY_F)) { ol_flags = m->ol_flags; w1.u = 0; } - if (!(flags & NIX_TX_MULTI_SEG_F)) { + if (!(flags & NIX_TX_MULTI_SEG_F)) send_hdr->w0.total = m->data_len; - send_hdr->w0.aura = - roc_npa_aura_handle_to_aura(m->pool->pool_id); - } + else + send_hdr->w0.total = m->pkt_len; + send_hdr->w0.aura = roc_npa_aura_handle_to_aura(m->pool->pool_id); /* * L3type: 2 => IPV4 @@ -376,7 +646,7 @@ cn10k_nix_xmit_prepare(struct rte_mbuf *m, uint64_t *cmd, uintptr_t lmt_addr, send_hdr->w1.u = w1.u; if (!(flags & NIX_TX_MULTI_SEG_F)) { - sg->seg1_size = m->data_len; + sg->seg1_size = send_hdr->w0.total; *(rte_iova_t *)(sg + 1) = rte_mbuf_data_iova(m); if (flags & NIX_TX_OFFLOAD_MBUF_NOFF_F) { @@ -389,17 +659,38 @@ cn10k_nix_xmit_prepare(struct rte_mbuf *m, uint64_t *cmd, uintptr_t lmt_addr, /* Mark mempool object as "put" since it is freed by NIX */ if (!send_hdr->w0.df) __mempool_check_cookies(m->pool, (void **)&m, 1, 0); + } else { + sg->seg1_size = m->data_len; + *(rte_iova_t *)(sg + 1) = rte_mbuf_data_iova(m); + + /* NOFF is handled later for multi-seg */ } + if (flags & NIX_TX_OFFLOAD_SECURITY_F) + *sec = !!(ol_flags & PKT_TX_SEC_OFFLOAD); +} + +static __rte_always_inline void +cn10k_nix_xmit_mv_lmt_base(uintptr_t lmt_addr, uint64_t *cmd, + const uint16_t flags) +{ + struct nix_send_ext_s *send_hdr_ext; + union nix_send_sg_s *sg; + /* With minimal offloads, 'cmd' being local could be optimized out to * registers. In other cases, 'cmd' will be in stack. Intent is * 'cmd' stores content from txq->cmd which is copied only once. */ - *((struct nix_send_hdr_s *)lmt_addr) = *send_hdr; + *((struct nix_send_hdr_s *)lmt_addr) = *(struct nix_send_hdr_s *)cmd; lmt_addr += 16; if (flags & NIX_TX_NEED_EXT_HDR) { + send_hdr_ext = (struct nix_send_ext_s *)(cmd + 2); *((struct nix_send_ext_s *)lmt_addr) = *send_hdr_ext; lmt_addr += 16; + + sg = (union nix_send_sg_s *)(cmd + 4); + } else { + sg = (union nix_send_sg_s *)(cmd + 2); } /* In case of multi-seg, sg template is stored here */ *((union nix_send_sg_s *)lmt_addr) = *sg; @@ -414,7 +705,7 @@ cn10k_nix_xmit_prepare_tstamp(uintptr_t lmt_addr, const uint64_t *cmd, if (flags & NIX_TX_OFFLOAD_TSTAMP_F) { const uint8_t is_ol_tstamp = !(ol_flags & PKT_TX_IEEE1588_TMST); struct nix_send_ext_s *send_hdr_ext = - (struct nix_send_ext_s *)lmt_addr + 16; + (struct nix_send_ext_s *)lmt_addr + 16; uint64_t *lmt = (uint64_t *)lmt_addr; uint16_t off = (no_segdw - 1) << 1; struct nix_send_mem_s *send_mem; @@ -457,8 +748,6 @@ cn10k_nix_prepare_mseg(struct rte_mbuf *m, uint64_t *cmd, const uint16_t flags) uint8_t off, i; send_hdr = (struct nix_send_hdr_s *)cmd; - send_hdr->w0.total = m->pkt_len; - send_hdr->w0.aura = roc_npa_aura_handle_to_aura(m->pool->pool_id); if (flags & NIX_TX_NEED_EXT_HDR) off = 2; @@ -466,13 +755,27 @@ cn10k_nix_prepare_mseg(struct rte_mbuf *m, uint64_t *cmd, const uint16_t flags) off = 0; sg = (union nix_send_sg_s *)&cmd[2 + off]; - /* Clear sg->u header before use */ - sg->u &= 0xFC00000000000000; + + /* Start from second segment, first segment is already there */ + i = 1; sg_u = sg->u; - slist = &cmd[3 + off]; + nb_segs = m->nb_segs - 1; + m_next = m->next; + slist = &cmd[3 + off + 1]; - i = 0; - nb_segs = m->nb_segs; + /* Set invert df if buffer is not to be freed by H/W */ + if (flags & NIX_TX_OFFLOAD_MBUF_NOFF_F) + sg_u |= (cnxk_nix_prefree_seg(m) << 55); + + /* Mark mempool object as "put" since it is freed by NIX */ +#ifdef RTE_LIBRTE_MEMPOOL_DEBUG + if (!(sg_u & (1ULL << 55))) + __mempool_check_cookies(m->pool, (void **)&m, 1, 0); + rte_io_wmb(); +#endif + m = m_next; + if (!m) + goto done; /* Fill mbuf segments */ do { @@ -504,6 +807,7 @@ cn10k_nix_prepare_mseg(struct rte_mbuf *m, uint64_t *cmd, const uint16_t flags) m = m_next; } while (nb_segs); +done: sg->u = sg_u; sg->segs = i; segdw = (uint64_t *)slist - (uint64_t *)&cmd[2 + off]; @@ -522,10 +826,17 @@ cn10k_nix_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts, { struct cn10k_eth_txq *txq = tx_queue; const rte_iova_t io_addr = txq->io_addr; - uintptr_t pa, lmt_addr = txq->lmt_base; + uint8_t lnum, c_lnum, c_shft, c_loff; + uintptr_t pa, lbase = txq->lmt_base; uint16_t lmt_id, burst, left, i; + uintptr_t c_lbase = lbase; + rte_iova_t c_io_addr; uint64_t lso_tun_fmt; + uint16_t c_lmt_id; + uint64_t sa_base; + uintptr_t laddr; uint64_t data; + bool sec; if (!(flags & NIX_TX_VWQE_F)) { NIX_XMIT_FC_OR_RETURN(txq, pkts); @@ -540,10 +851,24 @@ cn10k_nix_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts, lso_tun_fmt = txq->lso_tun_fmt; /* Get LMT base address and LMT ID as lcore id */ - ROC_LMT_BASE_ID_GET(lmt_addr, lmt_id); + ROC_LMT_BASE_ID_GET(lbase, lmt_id); + if (flags & NIX_TX_OFFLOAD_SECURITY_F) { + ROC_LMT_CPT_BASE_ID_GET(c_lbase, c_lmt_id); + c_io_addr = txq->cpt_io_addr; + sa_base = txq->sa_base; + } + left = pkts; again: burst = left > 32 ? 32 : left; + + lnum = 0; + if (flags & NIX_TX_OFFLOAD_SECURITY_F) { + c_lnum = 0; + c_loff = 0; + c_shft = 16; + } + for (i = 0; i < burst; i++) { /* Perform header writes for TSO, barrier at * lmt steorl will suffice. @@ -551,16 +876,39 @@ cn10k_nix_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts, if (flags & NIX_TX_OFFLOAD_TSO_F) cn10k_nix_xmit_prepare_tso(tx_pkts[i], flags); - cn10k_nix_xmit_prepare(tx_pkts[i], cmd, lmt_addr, flags, - lso_tun_fmt); - cn10k_nix_xmit_prepare_tstamp(lmt_addr, &txq->cmd[0], + cn10k_nix_xmit_prepare(tx_pkts[i], cmd, flags, lso_tun_fmt, + &sec); + + laddr = (uintptr_t)LMT_OFF(lbase, lnum, 0); + + /* Prepare CPT instruction and get nixtx addr */ + if (flags & NIX_TX_OFFLOAD_SECURITY_F && sec) + cn10k_nix_prep_sec(tx_pkts[i], cmd, &laddr, c_lbase, + &c_lnum, &c_loff, &c_shft, sa_base, + flags); + + /* Move NIX desc to LMT/NIXTX area */ + cn10k_nix_xmit_mv_lmt_base(laddr, cmd, flags); + cn10k_nix_xmit_prepare_tstamp(laddr, &txq->cmd[0], tx_pkts[i]->ol_flags, 4, flags); - lmt_addr += (1ULL << ROC_LMT_LINE_SIZE_LOG2); + if (!(flags & NIX_TX_OFFLOAD_SECURITY_F) || !sec) + lnum++; } if (flags & NIX_TX_VWQE_F) roc_sso_hws_head_wait(base); + left -= burst; + tx_pkts += burst; + + /* Submit CPT instructions if any */ + if (flags & NIX_TX_OFFLOAD_SECURITY_F) { + /* Reduce pkts to be sent to CPT */ + burst -= ((c_lnum << 1) + c_loff); + cn10k_nix_sec_steorl(c_io_addr, c_lmt_id, c_lnum, c_loff, + c_shft); + } + /* Trigger LMTST */ if (burst > 16) { data = cn10k_nix_tx_steor_data(flags); @@ -591,16 +939,9 @@ cn10k_nix_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts, roc_lmt_submit_steorl(data, pa); } - left -= burst; rte_io_wmb(); - if (left) { - /* Start processing another burst */ - tx_pkts += burst; - /* Reset lmt base addr */ - lmt_addr -= (1ULL << ROC_LMT_LINE_SIZE_LOG2); - lmt_addr &= (~(BIT_ULL(ROC_LMT_BASE_PER_CORE_LOG2) - 1)); + if (left) goto again; - } return pkts; } @@ -611,13 +952,20 @@ cn10k_nix_xmit_pkts_mseg(void *tx_queue, struct rte_mbuf **tx_pkts, const uint16_t flags) { struct cn10k_eth_txq *txq = tx_queue; - uintptr_t pa0, pa1, lmt_addr = txq->lmt_base; + uintptr_t pa0, pa1, lbase = txq->lmt_base; const rte_iova_t io_addr = txq->io_addr; uint16_t segdw, lmt_id, burst, left, i; + uint8_t lnum, c_lnum, c_loff; + uintptr_t c_lbase = lbase; uint64_t data0, data1; + rte_iova_t c_io_addr; uint64_t lso_tun_fmt; + uint8_t shft, c_shft; __uint128_t data128; - uint16_t shft; + uint16_t c_lmt_id; + uint64_t sa_base; + uintptr_t laddr; + bool sec; NIX_XMIT_FC_OR_RETURN(txq, pkts); @@ -630,12 +978,26 @@ cn10k_nix_xmit_pkts_mseg(void *tx_queue, struct rte_mbuf **tx_pkts, lso_tun_fmt = txq->lso_tun_fmt; /* Get LMT base address and LMT ID as lcore id */ - ROC_LMT_BASE_ID_GET(lmt_addr, lmt_id); + ROC_LMT_BASE_ID_GET(lbase, lmt_id); + if (flags & NIX_TX_OFFLOAD_SECURITY_F) { + ROC_LMT_CPT_BASE_ID_GET(c_lbase, c_lmt_id); + c_io_addr = txq->cpt_io_addr; + sa_base = txq->sa_base; + } + left = pkts; again: burst = left > 32 ? 32 : left; shft = 16; data128 = 0; + + lnum = 0; + if (flags & NIX_TX_OFFLOAD_SECURITY_F) { + c_lnum = 0; + c_loff = 0; + c_shft = 16; + } + for (i = 0; i < burst; i++) { /* Perform header writes for TSO, barrier at * lmt steorl will suffice. @@ -643,22 +1005,47 @@ cn10k_nix_xmit_pkts_mseg(void *tx_queue, struct rte_mbuf **tx_pkts, if (flags & NIX_TX_OFFLOAD_TSO_F) cn10k_nix_xmit_prepare_tso(tx_pkts[i], flags); - cn10k_nix_xmit_prepare(tx_pkts[i], cmd, lmt_addr, flags, - lso_tun_fmt); + cn10k_nix_xmit_prepare(tx_pkts[i], cmd, flags, lso_tun_fmt, + &sec); + + laddr = (uintptr_t)LMT_OFF(lbase, lnum, 0); + + /* Prepare CPT instruction and get nixtx addr */ + if (flags & NIX_TX_OFFLOAD_SECURITY_F && sec) + cn10k_nix_prep_sec(tx_pkts[i], cmd, &laddr, c_lbase, + &c_lnum, &c_loff, &c_shft, sa_base, + flags); + + /* Move NIX desc to LMT/NIXTX area */ + cn10k_nix_xmit_mv_lmt_base(laddr, cmd, flags); + /* Store sg list directly on lmt line */ - segdw = cn10k_nix_prepare_mseg(tx_pkts[i], (uint64_t *)lmt_addr, + segdw = cn10k_nix_prepare_mseg(tx_pkts[i], (uint64_t *)laddr, flags); - cn10k_nix_xmit_prepare_tstamp(lmt_addr, &txq->cmd[0], + cn10k_nix_xmit_prepare_tstamp(laddr, &txq->cmd[0], tx_pkts[i]->ol_flags, segdw, flags); - lmt_addr += (1ULL << ROC_LMT_LINE_SIZE_LOG2); - data128 |= (((__uint128_t)(segdw - 1)) << shft); - shft += 3; + if (!(flags & NIX_TX_OFFLOAD_SECURITY_F) || !sec) { + lnum++; + data128 |= (((__uint128_t)(segdw - 1)) << shft); + shft += 3; + } } if (flags & NIX_TX_VWQE_F) roc_sso_hws_head_wait(base); + left -= burst; + tx_pkts += burst; + + /* Submit CPT instructions if any */ + if (flags & NIX_TX_OFFLOAD_SECURITY_F) { + /* Reduce pkts to be sent to CPT */ + burst -= ((c_lnum << 1) + c_loff); + cn10k_nix_sec_steorl(c_io_addr, c_lmt_id, c_lnum, c_loff, + c_shft); + } + data0 = (uint64_t)data128; data1 = (uint64_t)(data128 >> 64); /* Make data0 similar to data1 */ @@ -695,16 +1082,9 @@ cn10k_nix_xmit_pkts_mseg(void *tx_queue, struct rte_mbuf **tx_pkts, roc_lmt_submit_steorl(data0, pa0); } - left -= burst; rte_io_wmb(); - if (left) { - /* Start processing another burst */ - tx_pkts += burst; - /* Reset lmt base addr */ - lmt_addr -= (1ULL << ROC_LMT_LINE_SIZE_LOG2); - lmt_addr &= (~(BIT_ULL(ROC_LMT_BASE_PER_CORE_LOG2) - 1)); + if (left) goto again; - } return pkts; } @@ -989,6 +1369,90 @@ cn10k_nix_prep_lmt_mseg_vector(struct rte_mbuf **mbufs, uint64x2_t *cmd0, return lmt_used; } +static __rte_always_inline void +cn10k_nix_lmt_next(uint8_t dw, uintptr_t laddr, uint8_t *lnum, uint8_t *loff, + uint8_t *shift, __uint128_t *data128, uintptr_t *next) +{ + /* Go to next line if we are out of space */ + if ((*loff + (dw << 4)) > 128) { + *data128 = *data128 | + (((__uint128_t)((*loff >> 4) - 1)) << *shift); + *shift = *shift + 3; + *loff = 0; + *lnum = *lnum + 1; + } + + *next = (uintptr_t)LMT_OFF(laddr, *lnum, *loff); + *loff = *loff + (dw << 4); +} + +static __rte_always_inline void +cn10k_nix_xmit_store(struct rte_mbuf *mbuf, uint8_t segdw, uintptr_t laddr, + uint64x2_t cmd0, uint64x2_t cmd1, uint64x2_t cmd2, + uint64x2_t cmd3, const uint16_t flags) +{ + uint8_t off; + + /* Handle no fast free when security is enabled without mseg */ + if ((flags & NIX_TX_OFFLOAD_MBUF_NOFF_F) && + (flags & NIX_TX_OFFLOAD_SECURITY_F) && + !(flags & NIX_TX_MULTI_SEG_F)) { + union nix_send_sg_s sg; + + sg.u = vgetq_lane_u64(cmd1, 0); + sg.u |= (cnxk_nix_prefree_seg(mbuf) << 55); + cmd1 = vsetq_lane_u64(sg.u, cmd1, 0); + +#ifdef RTE_LIBRTE_MEMPOOL_DEBUG + sg.u = vgetq_lane_u64(cmd1, 0); + if (!(sg.u & (1ULL << 55))) + __mempool_check_cookies(mbuf->pool, (void **)&mbuf, 1, + 0); + rte_io_wmb(); +#endif + } + if (flags & NIX_TX_MULTI_SEG_F) { + if ((flags & NIX_TX_NEED_EXT_HDR) && + (flags & NIX_TX_OFFLOAD_TSTAMP_F)) { + cn10k_nix_prepare_mseg_vec(mbuf, LMT_OFF(laddr, 0, 48), + &cmd0, &cmd1, segdw, flags); + vst1q_u64(LMT_OFF(laddr, 0, 0), cmd0); + vst1q_u64(LMT_OFF(laddr, 0, 16), cmd2); + vst1q_u64(LMT_OFF(laddr, 0, 32), cmd1); + off = segdw - 4; + off <<= 4; + vst1q_u64(LMT_OFF(laddr, 0, 48 + off), cmd3); + } else if (flags & NIX_TX_NEED_EXT_HDR) { + cn10k_nix_prepare_mseg_vec(mbuf, LMT_OFF(laddr, 0, 48), + &cmd0, &cmd1, segdw, flags); + vst1q_u64(LMT_OFF(laddr, 0, 0), cmd0); + vst1q_u64(LMT_OFF(laddr, 0, 16), cmd2); + vst1q_u64(LMT_OFF(laddr, 0, 32), cmd1); + } else { + cn10k_nix_prepare_mseg_vec(mbuf, LMT_OFF(laddr, 0, 32), + &cmd0, &cmd1, segdw, flags); + vst1q_u64(LMT_OFF(laddr, 0, 0), cmd0); + vst1q_u64(LMT_OFF(laddr, 0, 16), cmd1); + } + } else if (flags & NIX_TX_NEED_EXT_HDR) { + /* Store the prepared send desc to LMT lines */ + if (flags & NIX_TX_OFFLOAD_TSTAMP_F) { + vst1q_u64(LMT_OFF(laddr, 0, 0), cmd0); + vst1q_u64(LMT_OFF(laddr, 0, 16), cmd2); + vst1q_u64(LMT_OFF(laddr, 0, 32), cmd1); + vst1q_u64(LMT_OFF(laddr, 0, 48), cmd3); + } else { + vst1q_u64(LMT_OFF(laddr, 0, 0), cmd0); + vst1q_u64(LMT_OFF(laddr, 0, 16), cmd2); + vst1q_u64(LMT_OFF(laddr, 0, 32), cmd1); + } + } else { + /* Store the prepared send desc to LMT lines */ + vst1q_u64(LMT_OFF(laddr, 0, 0), cmd0); + vst1q_u64(LMT_OFF(laddr, 0, 16), cmd1); + } +} + static __rte_always_inline uint16_t cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts, uint64_t *cmd, uintptr_t base, @@ -998,10 +1462,10 @@ cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, uint64x2_t len_olflags0, len_olflags1, len_olflags2, len_olflags3; uint64x2_t cmd0[NIX_DESCS_PER_LOOP], cmd1[NIX_DESCS_PER_LOOP], cmd2[NIX_DESCS_PER_LOOP], cmd3[NIX_DESCS_PER_LOOP]; + uint16_t left, scalar, burst, i, lmt_id, c_lmt_id; uint64_t *mbuf0, *mbuf1, *mbuf2, *mbuf3, pa; uint64x2_t senddesc01_w0, senddesc23_w0; uint64x2_t senddesc01_w1, senddesc23_w1; - uint16_t left, scalar, burst, i, lmt_id; uint64x2_t sendext01_w0, sendext23_w0; uint64x2_t sendext01_w1, sendext23_w1; uint64x2_t sendmem01_w0, sendmem23_w0; @@ -1010,12 +1474,16 @@ cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, uint64x2_t sgdesc01_w0, sgdesc23_w0; uint64x2_t sgdesc01_w1, sgdesc23_w1; struct cn10k_eth_txq *txq = tx_queue; - uintptr_t laddr = txq->lmt_base; rte_iova_t io_addr = txq->io_addr; + uintptr_t laddr = txq->lmt_base; + uint8_t c_lnum, c_shft, c_loff; uint64x2_t ltypes01, ltypes23; uint64x2_t xtmp128, ytmp128; uint64x2_t xmask01, xmask23; - uint8_t lnum, shift; + uintptr_t c_laddr = laddr; + uint8_t lnum, shift, loff; + rte_iova_t c_io_addr; + uint64_t sa_base; union wdata { __uint128_t data128; uint64_t data[2]; @@ -1061,19 +1529,36 @@ cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, /* Get LMT base address and LMT ID as lcore id */ ROC_LMT_BASE_ID_GET(laddr, lmt_id); + if (flags & NIX_TX_OFFLOAD_SECURITY_F) { + ROC_LMT_CPT_BASE_ID_GET(c_laddr, c_lmt_id); + c_io_addr = txq->cpt_io_addr; + sa_base = txq->sa_base; + } + left = pkts; again: /* Number of packets to prepare depends on offloads enabled. */ burst = left > cn10k_nix_pkts_per_vec_brst(flags) ? cn10k_nix_pkts_per_vec_brst(flags) : left; - if (flags & NIX_TX_MULTI_SEG_F) { + if (flags & (NIX_TX_MULTI_SEG_F | NIX_TX_OFFLOAD_SECURITY_F)) { wd.data128 = 0; shift = 16; } lnum = 0; + if (NIX_TX_OFFLOAD_SECURITY_F) { + loff = 0; + c_loff = 0; + c_lnum = 0; + c_shft = 16; + } for (i = 0; i < burst; i += NIX_DESCS_PER_LOOP) { + if (flags & NIX_TX_OFFLOAD_SECURITY_F && c_lnum + 2 > 16) { + burst = i; + break; + } + if (flags & NIX_TX_MULTI_SEG_F) { uint8_t j; @@ -1833,7 +2318,8 @@ cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, } if ((flags & NIX_TX_OFFLOAD_MBUF_NOFF_F) && - !(flags & NIX_TX_MULTI_SEG_F)) { + !(flags & NIX_TX_MULTI_SEG_F) && + !(flags & NIX_TX_OFFLOAD_SECURITY_F)) { /* Set don't free bit if reference count > 1 */ xmask01 = vdupq_n_u64(0); xmask23 = xmask01; @@ -1873,7 +2359,8 @@ cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, (void **)&mbuf3, 1, 0); senddesc01_w0 = vorrq_u64(senddesc01_w0, xmask01); senddesc23_w0 = vorrq_u64(senddesc23_w0, xmask23); - } else if (!(flags & NIX_TX_MULTI_SEG_F)) { + } else if (!(flags & NIX_TX_MULTI_SEG_F) && + !(flags & NIX_TX_OFFLOAD_SECURITY_F)) { /* Move mbufs to iova */ mbuf0 = (uint64_t *)tx_pkts[0]; mbuf1 = (uint64_t *)tx_pkts[1]; @@ -1918,7 +2405,84 @@ cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, cmd2[3] = vzip2q_u64(sendext23_w0, sendext23_w1); } - if (flags & NIX_TX_MULTI_SEG_F) { + if (flags & NIX_TX_OFFLOAD_SECURITY_F) { + const uint64x2_t olf = {PKT_TX_SEC_OFFLOAD, + PKT_TX_SEC_OFFLOAD}; + uintptr_t next; + uint8_t dw; + + /* Extract ol_flags. */ + xtmp128 = vzip1q_u64(len_olflags0, len_olflags1); + ytmp128 = vzip1q_u64(len_olflags2, len_olflags3); + + xtmp128 = vtstq_u64(olf, xtmp128); + ytmp128 = vtstq_u64(olf, ytmp128); + + /* Process mbuf0 */ + dw = cn10k_nix_tx_dwords(flags, segdw[0]); + if (vgetq_lane_u64(xtmp128, 0)) + cn10k_nix_prep_sec_vec(tx_pkts[0], &cmd0[0], + &cmd1[0], &next, c_laddr, + &c_lnum, &c_loff, + &c_shft, sa_base, flags); + else + cn10k_nix_lmt_next(dw, laddr, &lnum, &loff, + &shift, &wd.data128, &next); + + /* Store mbuf0 to LMTLINE/CPT NIXTX area */ + cn10k_nix_xmit_store(tx_pkts[0], segdw[0], next, + cmd0[0], cmd1[0], cmd2[0], cmd3[0], + flags); + + /* Process mbuf1 */ + dw = cn10k_nix_tx_dwords(flags, segdw[1]); + if (vgetq_lane_u64(xtmp128, 1)) + cn10k_nix_prep_sec_vec(tx_pkts[1], &cmd0[1], + &cmd1[1], &next, c_laddr, + &c_lnum, &c_loff, + &c_shft, sa_base, flags); + else + cn10k_nix_lmt_next(dw, laddr, &lnum, &loff, + &shift, &wd.data128, &next); + + /* Store mbuf1 to LMTLINE/CPT NIXTX area */ + cn10k_nix_xmit_store(tx_pkts[1], segdw[1], next, + cmd0[1], cmd1[1], cmd2[1], cmd3[1], + flags); + + /* Process mbuf2 */ + dw = cn10k_nix_tx_dwords(flags, segdw[2]); + if (vgetq_lane_u64(ytmp128, 0)) + cn10k_nix_prep_sec_vec(tx_pkts[2], &cmd0[2], + &cmd1[2], &next, c_laddr, + &c_lnum, &c_loff, + &c_shft, sa_base, flags); + else + cn10k_nix_lmt_next(dw, laddr, &lnum, &loff, + &shift, &wd.data128, &next); + + /* Store mbuf2 to LMTLINE/CPT NIXTX area */ + cn10k_nix_xmit_store(tx_pkts[2], segdw[2], next, + cmd0[2], cmd1[2], cmd2[2], cmd3[2], + flags); + + /* Process mbuf3 */ + dw = cn10k_nix_tx_dwords(flags, segdw[3]); + if (vgetq_lane_u64(ytmp128, 1)) + cn10k_nix_prep_sec_vec(tx_pkts[3], &cmd0[3], + &cmd1[3], &next, c_laddr, + &c_lnum, &c_loff, + &c_shft, sa_base, flags); + else + cn10k_nix_lmt_next(dw, laddr, &lnum, &loff, + &shift, &wd.data128, &next); + + /* Store mbuf3 to LMTLINE/CPT NIXTX area */ + cn10k_nix_xmit_store(tx_pkts[3], segdw[3], next, + cmd0[3], cmd1[3], cmd2[3], cmd3[3], + flags); + + } else if (flags & NIX_TX_MULTI_SEG_F) { uint8_t j; segdw[4] = 8; @@ -1982,21 +2546,35 @@ cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, tx_pkts = tx_pkts + NIX_DESCS_PER_LOOP; } - if (flags & NIX_TX_MULTI_SEG_F) + /* Roundup lnum to last line if it is partial */ + if (flags & NIX_TX_OFFLOAD_SECURITY_F) { + lnum = lnum + !!loff; + wd.data128 = wd.data128 | + (((__uint128_t)(((loff >> 4) - 1) & 0x7) << shift)); + } + + if (flags & (NIX_TX_MULTI_SEG_F | NIX_TX_OFFLOAD_SECURITY_F)) wd.data[0] >>= 16; if (flags & NIX_TX_VWQE_F) roc_sso_hws_head_wait(base); + left -= burst; + + /* Submit CPT instructions if any */ + if (flags & NIX_TX_OFFLOAD_SECURITY_F) + cn10k_nix_sec_steorl(c_io_addr, c_lmt_id, c_lnum, c_loff, + c_shft); + /* Trigger LMTST */ if (lnum > 16) { - if (!(flags & NIX_TX_MULTI_SEG_F)) + if (!(flags & (NIX_TX_MULTI_SEG_F | NIX_TX_OFFLOAD_SECURITY_F))) wd.data[0] = cn10k_nix_tx_steor_vec_data(flags); pa = io_addr | (wd.data[0] & 0x7) << 4; wd.data[0] &= ~0x7ULL; - if (flags & NIX_TX_MULTI_SEG_F) + if (flags & (NIX_TX_MULTI_SEG_F | NIX_TX_OFFLOAD_SECURITY_F)) wd.data[0] <<= 16; wd.data[0] |= (15ULL << 12); @@ -2005,13 +2583,13 @@ cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, /* STEOR0 */ roc_lmt_submit_steorl(wd.data[0], pa); - if (!(flags & NIX_TX_MULTI_SEG_F)) + if (!(flags & (NIX_TX_MULTI_SEG_F | NIX_TX_OFFLOAD_SECURITY_F))) wd.data[1] = cn10k_nix_tx_steor_vec_data(flags); pa = io_addr | (wd.data[1] & 0x7) << 4; wd.data[1] &= ~0x7ULL; - if (flags & NIX_TX_MULTI_SEG_F) + if (flags & (NIX_TX_MULTI_SEG_F | NIX_TX_OFFLOAD_SECURITY_F)) wd.data[1] <<= 16; wd.data[1] |= ((uint64_t)(lnum - 17)) << 12; @@ -2020,13 +2598,13 @@ cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, /* STEOR1 */ roc_lmt_submit_steorl(wd.data[1], pa); } else if (lnum) { - if (!(flags & NIX_TX_MULTI_SEG_F)) + if (!(flags & (NIX_TX_MULTI_SEG_F | NIX_TX_OFFLOAD_SECURITY_F))) wd.data[0] = cn10k_nix_tx_steor_vec_data(flags); pa = io_addr | (wd.data[0] & 0x7) << 4; wd.data[0] &= ~0x7ULL; - if (flags & NIX_TX_MULTI_SEG_F) + if (flags & (NIX_TX_MULTI_SEG_F | NIX_TX_OFFLOAD_SECURITY_F)) wd.data[0] <<= 16; wd.data[0] |= ((uint64_t)(lnum - 1)) << 12; @@ -2036,7 +2614,6 @@ cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, roc_lmt_submit_steorl(wd.data[0], pa); } - left -= burst; rte_io_wmb(); if (left) goto again; @@ -2076,139 +2653,269 @@ cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, #define NOFF_F NIX_TX_OFFLOAD_MBUF_NOFF_F #define TSO_F NIX_TX_OFFLOAD_TSO_F #define TSP_F NIX_TX_OFFLOAD_TSTAMP_F +#define T_SEC_F NIX_TX_OFFLOAD_SECURITY_F -/* [TSP] [TSO] [NOFF] [VLAN] [OL3OL4CSUM] [L3L4CSUM] */ +/* [T_SEC_F] [TSP] [TSO] [NOFF] [VLAN] [OL3OL4CSUM] [L3L4CSUM] */ #define NIX_TX_FASTPATH_MODES \ -T(no_offload, 0, 0, 0, 0, 0, 0, 4, \ +T(no_offload, 0, 0, 0, 0, 0, 0, 0, 4, \ NIX_TX_OFFLOAD_NONE) \ -T(l3l4csum, 0, 0, 0, 0, 0, 1, 4, \ +T(l3l4csum, 0, 0, 0, 0, 0, 0, 1, 4, \ L3L4CSUM_F) \ -T(ol3ol4csum, 0, 0, 0, 0, 1, 0, 4, \ +T(ol3ol4csum, 0, 0, 0, 0, 0, 1, 0, 4, \ OL3OL4CSUM_F) \ -T(ol3ol4csum_l3l4csum, 0, 0, 0, 0, 1, 1, 4, \ +T(ol3ol4csum_l3l4csum, 0, 0, 0, 0, 0, 1, 1, 4, \ OL3OL4CSUM_F | L3L4CSUM_F) \ -T(vlan, 0, 0, 0, 1, 0, 0, 6, \ +T(vlan, 0, 0, 0, 0, 1, 0, 0, 6, \ VLAN_F) \ -T(vlan_l3l4csum, 0, 0, 0, 1, 0, 1, 6, \ +T(vlan_l3l4csum, 0, 0, 0, 0, 1, 0, 1, 6, \ VLAN_F | L3L4CSUM_F) \ -T(vlan_ol3ol4csum, 0, 0, 0, 1, 1, 0, 6, \ +T(vlan_ol3ol4csum, 0, 0, 0, 0, 1, 1, 0, 6, \ VLAN_F | OL3OL4CSUM_F) \ -T(vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 1, 1, 6, \ +T(vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 0, 1, 1, 1, 6, \ VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(noff, 0, 0, 1, 0, 0, 0, 4, \ +T(noff, 0, 0, 0, 1, 0, 0, 0, 4, \ NOFF_F) \ -T(noff_l3l4csum, 0, 0, 1, 0, 0, 1, 4, \ +T(noff_l3l4csum, 0, 0, 0, 1, 0, 0, 1, 4, \ NOFF_F | L3L4CSUM_F) \ -T(noff_ol3ol4csum, 0, 0, 1, 0, 1, 0, 4, \ +T(noff_ol3ol4csum, 0, 0, 0, 1, 0, 1, 0, 4, \ NOFF_F | OL3OL4CSUM_F) \ -T(noff_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 1, 1, 4, \ +T(noff_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 0, 1, 1, 4, \ NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(noff_vlan, 0, 0, 1, 1, 0, 0, 6, \ +T(noff_vlan, 0, 0, 0, 1, 1, 0, 0, 6, \ NOFF_F | VLAN_F) \ -T(noff_vlan_l3l4csum, 0, 0, 1, 1, 0, 1, 6, \ +T(noff_vlan_l3l4csum, 0, 0, 0, 1, 1, 0, 1, 6, \ NOFF_F | VLAN_F | L3L4CSUM_F) \ -T(noff_vlan_ol3ol4csum, 0, 0, 1, 1, 1, 0, 6, \ +T(noff_vlan_ol3ol4csum, 0, 0, 0, 1, 1, 1, 0, 6, \ NOFF_F | VLAN_F | OL3OL4CSUM_F) \ -T(noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 1, 1, 6, \ +T(noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 1, 1, 1, 6, \ NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso, 0, 1, 0, 0, 0, 0, 6, \ +T(tso, 0, 0, 1, 0, 0, 0, 0, 6, \ TSO_F) \ -T(tso_l3l4csum, 0, 1, 0, 0, 0, 1, 6, \ +T(tso_l3l4csum, 0, 0, 1, 0, 0, 0, 1, 6, \ TSO_F | L3L4CSUM_F) \ -T(tso_ol3ol4csum, 0, 1, 0, 0, 1, 0, 6, \ +T(tso_ol3ol4csum, 0, 0, 1, 0, 0, 1, 0, 6, \ TSO_F | OL3OL4CSUM_F) \ -T(tso_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 1, 1, 6, \ +T(tso_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 0, 1, 1, 6, \ TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso_vlan, 0, 1, 0, 1, 0, 0, 6, \ +T(tso_vlan, 0, 0, 1, 0, 1, 0, 0, 6, \ TSO_F | VLAN_F) \ -T(tso_vlan_l3l4csum, 0, 1, 0, 1, 0, 1, 6, \ +T(tso_vlan_l3l4csum, 0, 0, 1, 0, 1, 0, 1, 6, \ TSO_F | VLAN_F | L3L4CSUM_F) \ -T(tso_vlan_ol3ol4csum, 0, 1, 0, 1, 1, 0, 6, \ +T(tso_vlan_ol3ol4csum, 0, 0, 1, 0, 1, 1, 0, 6, \ TSO_F | VLAN_F | OL3OL4CSUM_F) \ -T(tso_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 1, 1, 6, \ +T(tso_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 1, 1, 1, 6, \ TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso_noff, 0, 1, 1, 0, 0, 0, 6, \ +T(tso_noff, 0, 0, 1, 1, 0, 0, 0, 6, \ TSO_F | NOFF_F) \ -T(tso_noff_l3l4csum, 0, 1, 1, 0, 0, 1, 6, \ +T(tso_noff_l3l4csum, 0, 0, 1, 1, 0, 0, 1, 6, \ TSO_F | NOFF_F | L3L4CSUM_F) \ -T(tso_noff_ol3ol4csum, 0, 1, 1, 0, 1, 0, 6, \ +T(tso_noff_ol3ol4csum, 0, 0, 1, 1, 0, 1, 0, 6, \ TSO_F | NOFF_F | OL3OL4CSUM_F) \ -T(tso_noff_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 1, 1, 6, \ +T(tso_noff_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 0, 1, 1, 6, \ TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso_noff_vlan, 0, 1, 1, 1, 0, 0, 6, \ +T(tso_noff_vlan, 0, 0, 1, 1, 1, 0, 0, 6, \ TSO_F | NOFF_F | VLAN_F) \ -T(tso_noff_vlan_l3l4csum, 0, 1, 1, 1, 0, 1, 6, \ +T(tso_noff_vlan_l3l4csum, 0, 0, 1, 1, 1, 0, 1, 6, \ TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ -T(tso_noff_vlan_ol3ol4csum, 0, 1, 1, 1, 1, 0, 6, \ +T(tso_noff_vlan_ol3ol4csum, 0, 0, 1, 1, 1, 1, 0, 6, \ TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ -T(tso_noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 1, 1, 6, \ +T(tso_noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 1, 1, 1, 6, \ TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts, 1, 0, 0, 0, 0, 0, 8, \ +T(ts, 0, 1, 0, 0, 0, 0, 0, 8, \ TSP_F) \ -T(ts_l3l4csum, 1, 0, 0, 0, 0, 1, 8, \ +T(ts_l3l4csum, 0, 1, 0, 0, 0, 0, 1, 8, \ TSP_F | L3L4CSUM_F) \ -T(ts_ol3ol4csum, 1, 0, 0, 0, 1, 0, 8, \ +T(ts_ol3ol4csum, 0, 1, 0, 0, 0, 1, 0, 8, \ TSP_F | OL3OL4CSUM_F) \ -T(ts_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 1, 1, 8, \ +T(ts_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 0, 1, 1, 8, \ TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_vlan, 1, 0, 0, 1, 0, 0, 8, \ +T(ts_vlan, 0, 1, 0, 0, 1, 0, 0, 8, \ TSP_F | VLAN_F) \ -T(ts_vlan_l3l4csum, 1, 0, 0, 1, 0, 1, 8, \ +T(ts_vlan_l3l4csum, 0, 1, 0, 0, 1, 0, 1, 8, \ TSP_F | VLAN_F | L3L4CSUM_F) \ -T(ts_vlan_ol3ol4csum, 1, 0, 0, 1, 1, 0, 8, \ +T(ts_vlan_ol3ol4csum, 0, 1, 0, 0, 1, 1, 0, 8, \ TSP_F | VLAN_F | OL3OL4CSUM_F) \ -T(ts_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 1, 1, 8, \ +T(ts_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 1, 1, 1, 8, \ TSP_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_noff, 1, 0, 1, 0, 0, 0, 8, \ +T(ts_noff, 0, 1, 0, 1, 0, 0, 0, 8, \ TSP_F | NOFF_F) \ -T(ts_noff_l3l4csum, 1, 0, 1, 0, 0, 1, 8, \ +T(ts_noff_l3l4csum, 0, 1, 0, 1, 0, 0, 1, 8, \ TSP_F | NOFF_F | L3L4CSUM_F) \ -T(ts_noff_ol3ol4csum, 1, 0, 1, 0, 1, 0, 8, \ +T(ts_noff_ol3ol4csum, 0, 1, 0, 1, 0, 1, 0, 8, \ TSP_F | NOFF_F | OL3OL4CSUM_F) \ -T(ts_noff_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 1, 1, 8, \ +T(ts_noff_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 0, 1, 1, 8, \ TSP_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_noff_vlan, 1, 0, 1, 1, 0, 0, 8, \ +T(ts_noff_vlan, 0, 1, 0, 1, 1, 0, 0, 8, \ TSP_F | NOFF_F | VLAN_F) \ -T(ts_noff_vlan_l3l4csum, 1, 0, 1, 1, 0, 1, 8, \ +T(ts_noff_vlan_l3l4csum, 0, 1, 0, 1, 1, 0, 1, 8, \ TSP_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ -T(ts_noff_vlan_ol3ol4csum, 1, 0, 1, 1, 1, 0, 8, \ +T(ts_noff_vlan_ol3ol4csum, 0, 1, 0, 1, 1, 1, 0, 8, \ TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ -T(ts_noff_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 1, 1, 8, \ +T(ts_noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 1, 1, 1, 8, \ TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_tso, 1, 1, 0, 0, 0, 0, 8, \ +T(ts_tso, 0, 1, 1, 0, 0, 0, 0, 8, \ TSP_F | TSO_F) \ -T(ts_tso_l3l4csum, 1, 1, 0, 0, 0, 1, 8, \ +T(ts_tso_l3l4csum, 0, 1, 1, 0, 0, 0, 1, 8, \ TSP_F | TSO_F | L3L4CSUM_F) \ -T(ts_tso_ol3ol4csum, 1, 1, 0, 0, 1, 0, 8, \ +T(ts_tso_ol3ol4csum, 0, 1, 1, 0, 0, 1, 0, 8, \ TSP_F | TSO_F | OL3OL4CSUM_F) \ -T(ts_tso_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 1, 1, 8, \ +T(ts_tso_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 0, 1, 1, 8, \ TSP_F | TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_tso_vlan, 1, 1, 0, 1, 0, 0, 8, \ +T(ts_tso_vlan, 0, 1, 1, 0, 1, 0, 0, 8, \ TSP_F | TSO_F | VLAN_F) \ -T(ts_tso_vlan_l3l4csum, 1, 1, 0, 1, 0, 1, 8, \ +T(ts_tso_vlan_l3l4csum, 0, 1, 1, 0, 1, 0, 1, 8, \ TSP_F | TSO_F | VLAN_F | L3L4CSUM_F) \ -T(ts_tso_vlan_ol3ol4csum, 1, 1, 0, 1, 1, 0, 8, \ +T(ts_tso_vlan_ol3ol4csum, 0, 1, 1, 0, 1, 1, 0, 8, \ TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F) \ -T(ts_tso_vlan_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 1, 1, 8, \ +T(ts_tso_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 1, 1, 1, 8, \ TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_tso_noff, 1, 1, 1, 0, 0, 0, 8, \ +T(ts_tso_noff, 0, 1, 1, 1, 0, 0, 0, 8, \ TSP_F | TSO_F | NOFF_F) \ -T(ts_tso_noff_l3l4csum, 1, 1, 1, 0, 0, 1, 8, \ +T(ts_tso_noff_l3l4csum, 0, 1, 1, 1, 0, 0, 1, 8, \ TSP_F | TSO_F | NOFF_F | L3L4CSUM_F) \ -T(ts_tso_noff_ol3ol4csum, 1, 1, 1, 0, 1, 0, 8, \ +T(ts_tso_noff_ol3ol4csum, 0, 1, 1, 1, 0, 1, 0, 8, \ TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F) \ -T(ts_tso_noff_ol3ol4csum_l3l4csum, 1, 1, 1, 0, 1, 1, 8, \ +T(ts_tso_noff_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 0, 1, 1, 8, \ TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(ts_tso_noff_vlan, 1, 1, 1, 1, 0, 0, 8, \ +T(ts_tso_noff_vlan, 0, 1, 1, 1, 1, 0, 0, 8, \ TSP_F | TSO_F | NOFF_F | VLAN_F) \ -T(ts_tso_noff_vlan_l3l4csum, 1, 1, 1, 1, 0, 1, 8, \ +T(ts_tso_noff_vlan_l3l4csum, 0, 1, 1, 1, 1, 0, 1, 8, \ TSP_F | TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ -T(ts_tso_noff_vlan_ol3ol4csum, 1, 1, 1, 1, 1, 0, 8, \ +T(ts_tso_noff_vlan_ol3ol4csum, 0, 1, 1, 1, 1, 1, 0, 8, \ TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ -T(ts_tso_noff_vlan_ol3ol4csum_l3l4csum, 1, 1, 1, 1, 1, 1, 8, \ - TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) +T(ts_tso_noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 1, 1, 1, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F)\ +T(sec, 1, 0, 0, 0, 0, 0, 0, 4, \ + T_SEC_F) \ +T(sec_l3l4csum, 1, 0, 0, 0, 0, 0, 1, 4, \ + T_SEC_F | L3L4CSUM_F) \ +T(sec_ol3ol4csum, 1, 0, 0, 0, 0, 1, 0, 4, \ + T_SEC_F | OL3OL4CSUM_F) \ +T(sec_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 0, 1, 1, 4, \ + T_SEC_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_vlan, 1, 0, 0, 0, 1, 0, 0, 6, \ + T_SEC_F | VLAN_F) \ +T(sec_vlan_l3l4csum, 1, 0, 0, 0, 1, 0, 1, 6, \ + T_SEC_F | VLAN_F | L3L4CSUM_F) \ +T(sec_vlan_ol3ol4csum, 1, 0, 0, 0, 1, 1, 0, 6, \ + T_SEC_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 1, 1, 1, 6, \ + T_SEC_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_noff, 1, 0, 0, 1, 0, 0, 0, 4, \ + T_SEC_F | NOFF_F) \ +T(sec_noff_l3l4csum, 1, 0, 0, 1, 0, 0, 1, 4, \ + T_SEC_F | NOFF_F | L3L4CSUM_F) \ +T(sec_noff_ol3ol4csum, 1, 0, 0, 1, 0, 1, 0, 4, \ + T_SEC_F | NOFF_F | OL3OL4CSUM_F) \ +T(sec_noff_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 0, 1, 1, 4, \ + T_SEC_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_noff_vlan, 1, 0, 0, 1, 1, 0, 0, 6, \ + T_SEC_F | NOFF_F | VLAN_F) \ +T(sec_noff_vlan_l3l4csum, 1, 0, 0, 1, 1, 0, 1, 6, \ + T_SEC_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(sec_noff_vlan_ol3ol4csum, 1, 0, 0, 1, 1, 1, 0, 6, \ + T_SEC_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_noff_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 1, 1, 1, 6, \ + T_SEC_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_tso, 1, 0, 1, 0, 0, 0, 0, 6, \ + T_SEC_F | TSO_F) \ +T(sec_tso_l3l4csum, 1, 0, 1, 0, 0, 0, 1, 6, \ + T_SEC_F | TSO_F | L3L4CSUM_F) \ +T(sec_tso_ol3ol4csum, 1, 0, 1, 0, 0, 1, 0, 6, \ + T_SEC_F | TSO_F | OL3OL4CSUM_F) \ +T(sec_tso_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 0, 1, 1, 6, \ + T_SEC_F | TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_tso_vlan, 1, 0, 1, 0, 1, 0, 0, 6, \ + T_SEC_F | TSO_F | VLAN_F) \ +T(sec_tso_vlan_l3l4csum, 1, 0, 1, 0, 1, 0, 1, 6, \ + T_SEC_F | TSO_F | VLAN_F | L3L4CSUM_F) \ +T(sec_tso_vlan_ol3ol4csum, 1, 0, 1, 0, 1, 1, 0, 6, \ + T_SEC_F | TSO_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_tso_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 1, 1, 1, 6, \ + T_SEC_F | TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_tso_noff, 1, 0, 1, 1, 0, 0, 0, 6, \ + T_SEC_F | TSO_F | NOFF_F) \ +T(sec_tso_noff_l3l4csum, 1, 0, 1, 1, 0, 0, 1, 6, \ + T_SEC_F | TSO_F | NOFF_F | L3L4CSUM_F) \ +T(sec_tso_noff_ol3ol4csum, 1, 0, 1, 1, 0, 1, 0, 6, \ + T_SEC_F | TSO_F | NOFF_F | OL3OL4CSUM_F) \ +T(sec_tso_noff_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 0, 1, 1, 6, \ + T_SEC_F | TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_tso_noff_vlan, 1, 0, 1, 1, 1, 0, 0, 6, \ + T_SEC_F | TSO_F | NOFF_F | VLAN_F) \ +T(sec_tso_noff_vlan_l3l4csum, 1, 0, 1, 1, 1, 0, 1, 6, \ + T_SEC_F | TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(sec_tso_noff_vlan_ol3ol4csum, 1, 0, 1, 1, 1, 1, 0, 6, \ + T_SEC_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_tso_noff_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 1, 1, 1, 6, \ + T_SEC_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F)\ +T(sec_ts, 1, 1, 0, 0, 0, 0, 0, 8, \ + T_SEC_F | TSP_F) \ +T(sec_ts_l3l4csum, 1, 1, 0, 0, 0, 0, 1, 8, \ + T_SEC_F | TSP_F | L3L4CSUM_F) \ +T(sec_ts_ol3ol4csum, 1, 1, 0, 0, 0, 1, 0, 8, \ + T_SEC_F | TSP_F | OL3OL4CSUM_F) \ +T(sec_ts_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 0, 1, 1, 8, \ + T_SEC_F | TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_ts_vlan, 1, 1, 0, 0, 1, 0, 0, 8, \ + T_SEC_F | TSP_F | VLAN_F) \ +T(sec_ts_vlan_l3l4csum, 1, 1, 0, 0, 1, 0, 1, 8, \ + T_SEC_F | TSP_F | VLAN_F | L3L4CSUM_F) \ +T(sec_ts_vlan_ol3ol4csum, 1, 1, 0, 0, 1, 1, 0, 8, \ + T_SEC_F | TSP_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_ts_vlan_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 1, 1, 1, 8, \ + T_SEC_F | TSP_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_ts_noff, 1, 1, 0, 1, 0, 0, 0, 8, \ + T_SEC_F | TSP_F | NOFF_F) \ +T(sec_ts_noff_l3l4csum, 1, 1, 0, 1, 0, 0, 1, 8, \ + T_SEC_F | TSP_F | NOFF_F | L3L4CSUM_F) \ +T(sec_ts_noff_ol3ol4csum, 1, 1, 0, 1, 0, 1, 0, 8, \ + T_SEC_F | TSP_F | NOFF_F | OL3OL4CSUM_F) \ +T(sec_ts_noff_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 0, 1, 1, 8, \ + T_SEC_F | TSP_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_ts_noff_vlan, 1, 1, 0, 1, 1, 0, 0, 8, \ + T_SEC_F | TSP_F | NOFF_F | VLAN_F) \ +T(sec_ts_noff_vlan_l3l4csum, 1, 1, 0, 1, 1, 0, 1, 8, \ + T_SEC_F | TSP_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(sec_ts_noff_vlan_ol3ol4csum, 1, 1, 0, 1, 1, 1, 0, 8, \ + T_SEC_F | TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_ts_noff_vlan_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 1, 1, 1, 8, \ + T_SEC_F | TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F)\ +T(sec_ts_tso, 1, 1, 1, 0, 0, 0, 0, 8, \ + T_SEC_F | TSP_F | TSO_F) \ +T(sec_ts_tso_l3l4csum, 1, 1, 1, 0, 0, 0, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | L3L4CSUM_F) \ +T(sec_ts_tso_ol3ol4csum, 1, 1, 1, 0, 0, 1, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | OL3OL4CSUM_F) \ +T(sec_ts_tso_ol3ol4csum_l3l4csum, 1, 1, 1, 0, 0, 1, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_ts_tso_vlan, 1, 1, 1, 0, 1, 0, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | VLAN_F) \ +T(sec_ts_tso_vlan_l3l4csum, 1, 1, 1, 0, 1, 0, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | VLAN_F | L3L4CSUM_F) \ +T(sec_ts_tso_vlan_ol3ol4csum, 1, 1, 1, 0, 1, 1, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F) \ +T(sec_ts_tso_vlan_ol3ol4csum_l3l4csum, 1, 1, 1, 0, 1, 1, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(sec_ts_tso_noff, 1, 1, 1, 1, 0, 0, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F) \ +T(sec_ts_tso_noff_l3l4csum, 1, 1, 1, 1, 0, 0, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | L3L4CSUM_F) \ +T(sec_ts_tso_noff_ol3ol4csum, 1, 1, 1, 1, 0, 1, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F) \ +T(sec_ts_tso_noff_ol3ol4csum_l3l4csum, 1, 1, 1, 1, 0, 1, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F)\ +T(sec_ts_tso_noff_vlan, 1, 1, 1, 1, 1, 0, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | VLAN_F) \ +T(sec_ts_tso_noff_vlan_l3l4csum, 1, 1, 1, 1, 1, 0, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(sec_ts_tso_noff_vlan_ol3ol4csum, 1, 1, 1, 1, 1, 1, 0, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F)\ +T(sec_ts_tso_noff_vlan_ol3ol4csum_l3l4csum, 1, 1, 1, 1, 1, 1, 1, 8, \ + T_SEC_F | TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | \ + L3L4CSUM_F) -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_xmit_pkts_##name( \ void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts); \ \ diff --git a/drivers/net/cnxk/cn10k_tx_mseg.c b/drivers/net/cnxk/cn10k_tx_mseg.c index 4ea4c8a..2b83409 100644 --- a/drivers/net/cnxk/cn10k_tx_mseg.c +++ b/drivers/net/cnxk/cn10k_tx_mseg.c @@ -5,7 +5,7 @@ #include "cn10k_ethdev.h" #include "cn10k_tx.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot \ cn10k_nix_xmit_pkts_mseg_##name(void *tx_queue, \ struct rte_mbuf **tx_pkts, \ diff --git a/drivers/net/cnxk/cn10k_tx_vec.c b/drivers/net/cnxk/cn10k_tx_vec.c index a035049..2789b13 100644 --- a/drivers/net/cnxk/cn10k_tx_vec.c +++ b/drivers/net/cnxk/cn10k_tx_vec.c @@ -5,7 +5,7 @@ #include "cn10k_ethdev.h" #include "cn10k_tx.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot \ cn10k_nix_xmit_pkts_vec_##name(void *tx_queue, \ struct rte_mbuf **tx_pkts, \ diff --git a/drivers/net/cnxk/cn10k_tx_vec_mseg.c b/drivers/net/cnxk/cn10k_tx_vec_mseg.c index 7f98f79..98000df 100644 --- a/drivers/net/cnxk/cn10k_tx_vec_mseg.c +++ b/drivers/net/cnxk/cn10k_tx_vec_mseg.c @@ -5,7 +5,7 @@ #include "cn10k_ethdev.h" #include "cn10k_tx.h" -#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f6, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_xmit_pkts_vec_mseg_##name( \ void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts) \ { \ From patchwork Thu Sep 2 02:14:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97747 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 82C18A0C4C; Thu, 2 Sep 2021 04:19:10 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A586C41197; Thu, 2 Sep 2021 04:17:52 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id BA8984119D for ; Thu, 2 Sep 2021 04:17:50 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQCpu011801 for ; Wed, 1 Sep 2021 19:17:50 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=dWCVK1wq45KoC4KwL037N0S4jX5zugmcBKPBw+8Ziuc=; b=OI5Rz2kR+IKo0FsmEZL5w+rsYc+NP9bTlOFYaLzaGBnFgTnCkl5+sB/q3mumxF7O4/NH wZRGodWkgrJLu0YJHp8UG6bNciJdCk/M4IeH69FnMqxmNsSfCn3bPPd61c1NP0f161qj 9J9RT8ahobCs2SzA6gEpukPdSgpoAOYrAqaouO5NUOMMI9UhCcj1RPvWD9ztDEaJ+p16 MZSZJLXNc5DhlY49R/k6aTBB934P9PpxC+ObnIN5Yba5L82C5aqJocIJi7vVn4KovaXd z3+Wn9vpLrF8/7x6nVIYc/zHiBC5VmeH7p6jiVT3A7Cc2DJ72L4f1ji4cPW3ZVUXiVnC aA== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9huq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:49 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:47 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:47 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 2C6933F7040; Wed, 1 Sep 2021 19:17:44 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:44:59 +0530 Message-ID: <20210902021505.17607-22-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: rjqddn7eXjz8Ddv_Oy7xcKm7dsle4Kjf X-Proofpoint-GUID: rjqddn7eXjz8Ddv_Oy7xcKm7dsle4Kjf X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 21/27] net/cnxk: add cn9k anti replay support for security offload X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Srujana Challa Adds anti replay support for cn9k platform. Signed-off-by: Srujana Challa --- drivers/net/cnxk/cn9k_ethdev.h | 3 +++ drivers/net/cnxk/cn9k_ethdev_sec.c | 29 ++++++++++++++++++++ drivers/net/cnxk/cn9k_rx.h | 54 +++++++++++++++++++++++++++++++++++++- 3 files changed, 85 insertions(+), 1 deletion(-) diff --git a/drivers/net/cnxk/cn9k_ethdev.h b/drivers/net/cnxk/cn9k_ethdev.h index f8818b8..2b452fe 100644 --- a/drivers/net/cnxk/cn9k_ethdev.h +++ b/drivers/net/cnxk/cn9k_ethdev.h @@ -6,6 +6,7 @@ #include #include +#include struct cn9k_eth_txq { uint64_t cmd[8]; @@ -40,6 +41,8 @@ struct cn9k_eth_rxq { /* Private data in sw rsvd area of struct roc_onf_ipsec_inb_sa */ struct cn9k_inb_priv_data { void *userdata; + uint32_t replay_win_sz; + struct cnxk_on_ipsec_ar ar; struct cnxk_eth_sec_sess *eth_sec; }; diff --git a/drivers/net/cnxk/cn9k_ethdev_sec.c b/drivers/net/cnxk/cn9k_ethdev_sec.c index 3ec7497..deb1daf 100644 --- a/drivers/net/cnxk/cn9k_ethdev_sec.c +++ b/drivers/net/cnxk/cn9k_ethdev_sec.c @@ -73,6 +73,27 @@ static const struct rte_security_capability cn9k_eth_sec_capabilities[] = { } }; +static inline int +ar_window_init(struct cn9k_inb_priv_data *inb_priv) +{ + if (inb_priv->replay_win_sz > CNXK_ON_AR_WIN_SIZE_MAX) { + plt_err("Replay window size:%u is not supported", + inb_priv->replay_win_sz); + return -ENOTSUP; + } + + rte_spinlock_init(&inb_priv->ar.lock); + /* + * Set window bottom to 1, base and top to size of + * window + */ + inb_priv->ar.winb = 1; + inb_priv->ar.wint = inb_priv->replay_win_sz; + inb_priv->ar.base = inb_priv->replay_win_sz; + + return 0; +} + static int cn9k_eth_sec_session_create(void *device, struct rte_security_session_conf *conf, @@ -158,6 +179,14 @@ cn9k_eth_sec_session_create(void *device, /* Save userdata in inb private area */ inb_priv->userdata = conf->userdata; + inb_priv->replay_win_sz = ipsec->replay_win_sz; + if (inb_priv->replay_win_sz) { + rc = ar_window_init(inb_priv); + if (rc) + goto mempool_put; + } + + /* Prepare session priv */ sess_priv.inb_sa = 1; sess_priv.sa_idx = ipsec->spi; diff --git a/drivers/net/cnxk/cn9k_rx.h b/drivers/net/cnxk/cn9k_rx.h index bdedeab..7ab415a 100644 --- a/drivers/net/cnxk/cn9k_rx.h +++ b/drivers/net/cnxk/cn9k_rx.h @@ -31,6 +31,9 @@ #define CQE_CAST(x) ((struct nix_cqe_hdr_s *)(x)) #define CQE_SZ(x) ((x) * CNXK_NIX_CQ_ENTRY_SZ) +#define IPSEC_SQ_LO_IDX 4 +#define IPSEC_SQ_HI_IDX 8 + union mbuf_initializer { struct { uint16_t data_off; @@ -166,6 +169,48 @@ nix_cqe_xtract_mseg(const union nix_rx_parse_u *rx, struct rte_mbuf *mbuf, mbuf->next = NULL; } +static inline int +ipsec_antireplay_check(struct roc_onf_ipsec_inb_sa *sa, + struct cn9k_inb_priv_data *priv, uintptr_t data, + uint32_t win_sz) +{ + struct cnxk_on_ipsec_ar *ar = &priv->ar; + uint64_t seq_in_sa; + uint32_t seqh = 0; + uint32_t seql; + uint64_t seq; + uint8_t esn; + int rc; + + esn = sa->ctl.esn_en; + seql = rte_be_to_cpu_32(*((uint32_t *)(data + IPSEC_SQ_LO_IDX))); + + if (!esn) { + seq = (uint64_t)seql; + } else { + seqh = rte_be_to_cpu_32(*((uint32_t *)(data + + IPSEC_SQ_HI_IDX))); + seq = ((uint64_t)seqh << 32) | seql; + } + + if (unlikely(seq == 0)) + return -1; + + rte_spinlock_lock(&ar->lock); + rc = cnxk_on_anti_replay_check(seq, ar, win_sz); + if (esn && !rc) { + seq_in_sa = ((uint64_t)rte_be_to_cpu_32(sa->esn_hi) << 32) | + rte_be_to_cpu_32(sa->esn_low); + if (seq > seq_in_sa) { + sa->esn_low = rte_cpu_to_be_32(seql); + sa->esn_hi = rte_cpu_to_be_32(seqh); + } + } + rte_spinlock_unlock(&ar->lock); + + return rc; +} + static __rte_always_inline uint64_t nix_rx_sec_mbuf_update(const struct nix_cqe_hdr_s *cq, struct rte_mbuf *m, uintptr_t sa_base, uint64_t *rearm_val, uint16_t *len) @@ -178,8 +223,8 @@ nix_rx_sec_mbuf_update(const struct nix_cqe_hdr_s *cq, struct rte_mbuf *m, uint8_t lcptr = rx->lcptr; struct rte_ipv4_hdr *ipv4; uint16_t data_off, res; + uint32_t spi, win_sz; uint32_t spi_mask; - uint32_t spi; uintptr_t data; __uint128_t dw; uint8_t sa_w; @@ -209,6 +254,13 @@ nix_rx_sec_mbuf_update(const struct nix_cqe_hdr_s *cq, struct rte_mbuf *m, dw = *(__uint128_t *)sa_priv; *rte_security_dynfield(m) = (uint64_t)dw; + /* Check if anti-replay is enabled */ + win_sz = (uint32_t)(dw >> 64); + if (win_sz) { + if (ipsec_antireplay_check(sa, sa_priv, data, win_sz) < 0) + return PKT_RX_SEC_OFFLOAD | PKT_RX_SEC_OFFLOAD_FAILED; + } + /* Get total length from IPv4 header. We can assume only IPv4 */ ipv4 = (struct rte_ipv4_hdr *)(data + ROC_ONF_IPSEC_INB_SPI_SEQ_SZ + ROC_ONF_IPSEC_INB_MAX_L2_SZ); From patchwork Thu Sep 2 02:15:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97748 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B633DA0C4C; Thu, 2 Sep 2021 04:19:15 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B98C4411A0; Thu, 2 Sep 2021 04:17:53 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 68D1F41159 for ; Thu, 2 Sep 2021 04:17:52 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181K5Dsq025890 for ; Wed, 1 Sep 2021 19:17:51 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=nWbyW3ae/I+u4stmZhyt53VnFU8NSq3t1fwtgTPHn1o=; b=HYJ9Rc72cGn0se0Ile42wMMGp8YT6iut8ybj1Mxe8pnPEq6saQyQ9o+sEA1LD1elZY0y 4DWcomZ9os++R07HIQJbdAEK+wvOJ3GmarOV47/rwTqd49TBjLBIBvo5F3iHNUAaPuQm dKZcJkHw7ne4TmpGbvEfwVlOdIUlu8WuRsRCKtn7V87Enx0QMRfIvMSLsGU9dt4wRlzp F8NLRFaCtX+jhu6vUAb0O35tG1Xj5H2RDq4ml9Wl9h9rN4ESemcl6W7jStCPnZabzMgg BKhuaeccz5slv2i6ESOmhB2/KH+xhu44Tb8jtldI8nhjr2HzStlzrMY/eTZIdmd5gwP0 XA== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com with ESMTP id 3atg8a91mq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:51 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:49 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:50 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id DA25E3F704D; Wed, 1 Sep 2021 19:17:47 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:45:00 +0530 Message-ID: <20210902021505.17607-23-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: CEdcKUwn_-4YBnWOwFtgMxnWVSi6hJMw X-Proofpoint-ORIG-GUID: CEdcKUwn_-4YBnWOwFtgMxnWVSi6hJMw X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 22/27] net/cnxk: add cn10k IPsec transport mode support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Srujana Challa Adds IPsec transport mode capability to rte security capabilities. Signed-off-by: Srujana Challa --- drivers/net/cnxk/cn10k_ethdev_sec.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/drivers/net/cnxk/cn10k_ethdev_sec.c b/drivers/net/cnxk/cn10k_ethdev_sec.c index 3ffd824..dae5ea7 100644 --- a/drivers/net/cnxk/cn10k_ethdev_sec.c +++ b/drivers/net/cnxk/cn10k_ethdev_sec.c @@ -69,6 +69,30 @@ static const struct rte_security_capability cn10k_eth_sec_capabilities[] = { .crypto_capabilities = cn10k_eth_sec_crypto_caps, .ol_flags = RTE_SECURITY_TX_OLOAD_NEED_MDATA }, + { /* IPsec Inline Protocol ESP Transport Egress */ + .action = RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL, + .protocol = RTE_SECURITY_PROTOCOL_IPSEC, + .ipsec = { + .proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP, + .mode = RTE_SECURITY_IPSEC_SA_MODE_TRANSPORT, + .direction = RTE_SECURITY_IPSEC_SA_DIR_EGRESS, + .options = { 0 } + }, + .crypto_capabilities = cn10k_eth_sec_crypto_caps, + .ol_flags = RTE_SECURITY_TX_OLOAD_NEED_MDATA + }, + { /* IPsec Inline Protocol ESP Transport Ingress */ + .action = RTE_SECURITY_ACTION_TYPE_INLINE_PROTOCOL, + .protocol = RTE_SECURITY_PROTOCOL_IPSEC, + .ipsec = { + .proto = RTE_SECURITY_IPSEC_SA_PROTO_ESP, + .mode = RTE_SECURITY_IPSEC_SA_MODE_TRANSPORT, + .direction = RTE_SECURITY_IPSEC_SA_DIR_INGRESS, + .options = { 0 } + }, + .crypto_capabilities = cn10k_eth_sec_crypto_caps, + .ol_flags = RTE_SECURITY_TX_OLOAD_NEED_MDATA + }, { .action = RTE_SECURITY_ACTION_TYPE_NONE } From patchwork Thu Sep 2 02:15:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97749 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1DF42A0C4C; Thu, 2 Sep 2021 04:19:21 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DF9C441175; Thu, 2 Sep 2021 04:17:56 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 4B0E9411A6 for ; Thu, 2 Sep 2021 04:17:55 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQ21G011572 for ; Wed, 1 Sep 2021 19:17:54 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=ROdomMsTmEP8ZV4RIB8wmkOiWPgiLeh/HvUdaWF1l6U=; b=AF2CMiYXYoV3B+3fE1NvtGLnTGxZnu2nArePYqPhZuoEk+b2xwjYRCL1xC/wV+5eWhjf Y7rUymV7rDCqbcsfSCLLW8zlIphaLGaO5Ia1vm0MPTXsk9Bf3v/CFB7LGGe1PVkL7H7D W9R3K1Q7ptmc4mukmojKkpVqR1laPlcUX2il6kiQviFXqudDYKGN0uOEWOFMYNgaqKNg EirOHd+dQ9+9/ZyAMuFZuwRepwSdqWRx+iXwfQhl0i7m4ZSaBtbaU7rDmZliz3gGEtnM KIbK6NtWdTBsSrTkyqyX32LG3kasnHH3gkejVUjh8TkrpyPvjwOhsVO+UmLRTH9vPJcs vg== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hv4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:54 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:52 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:52 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 928193F7040; Wed, 1 Sep 2021 19:17:50 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:45:01 +0530 Message-ID: <20210902021505.17607-24-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: xh0UKT8RkcHxllpK3BXvpI9INGgNHO0H X-Proofpoint-GUID: xh0UKT8RkcHxllpK3BXvpI9INGgNHO0H X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 23/27] net/cnxk: update ethertype for mixed IPsec tunnel versions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Srujana Challa Adds support to update ethertype for mixed IPsec tunnel versions. And also sets et_overwr for inbound IPsec. Signed-off-by: Srujana Challa --- drivers/common/cnxk/cnxk_security.c | 1 + drivers/net/cnxk/cn10k_ethdev.h | 3 ++- drivers/net/cnxk/cn10k_ethdev_sec.c | 2 ++ drivers/net/cnxk/cn10k_tx.h | 19 +++++++++++++++++++ 4 files changed, 24 insertions(+), 1 deletion(-) diff --git a/drivers/common/cnxk/cnxk_security.c b/drivers/common/cnxk/cnxk_security.c index c25b3fd..90b5205 100644 --- a/drivers/common/cnxk/cnxk_security.c +++ b/drivers/common/cnxk/cnxk_security.c @@ -239,6 +239,7 @@ cnxk_ot_ipsec_inb_sa_fill(struct roc_ot_ipsec_inb_sa *sa, /* There are two words of CPT_CTX_HW_S for ucode to skip */ sa->w0.s.ctx_hdr_size = 1; sa->w0.s.aop_valid = 1; + sa->w0.s.et_ovrwr = 1; rte_wmb(); diff --git a/drivers/net/cnxk/cn10k_ethdev.h b/drivers/net/cnxk/cn10k_ethdev.h index 200cd93..c2a46ad 100644 --- a/drivers/net/cnxk/cn10k_ethdev.h +++ b/drivers/net/cnxk/cn10k_ethdev.h @@ -64,7 +64,8 @@ struct cn10k_sec_sess_priv { struct { uint32_t sa_idx; uint8_t inb_sa : 1; - uint8_t rsvd1 : 2; + uint8_t outer_ip_ver : 1; + uint8_t mode : 1; uint8_t roundup_byte : 5; uint8_t roundup_len; uint16_t partial_len; diff --git a/drivers/net/cnxk/cn10k_ethdev_sec.c b/drivers/net/cnxk/cn10k_ethdev_sec.c index dae5ea7..c66730a 100644 --- a/drivers/net/cnxk/cn10k_ethdev_sec.c +++ b/drivers/net/cnxk/cn10k_ethdev_sec.c @@ -341,6 +341,8 @@ cn10k_eth_sec_session_create(void *device, sess_priv.roundup_byte = rlens->roundup_byte; sess_priv.roundup_len = rlens->roundup_len; sess_priv.partial_len = rlens->partial_len; + sess_priv.mode = outb_sa->w2.s.ipsec_mode; + sess_priv.outer_ip_ver = outb_sa->w2.s.outer_ip_ver; /* Pointer from eth_sec -> outb_sa */ eth_sec->sa = outb_sa; diff --git a/drivers/net/cnxk/cn10k_tx.h b/drivers/net/cnxk/cn10k_tx.h index 70ba929..f56aa8e 100644 --- a/drivers/net/cnxk/cn10k_tx.h +++ b/drivers/net/cnxk/cn10k_tx.h @@ -302,6 +302,16 @@ cn10k_nix_prep_sec_vec(struct rte_mbuf *m, uint64x2_t *cmd0, uint64x2_t *cmd1, cmd23 = vsetq_lane_u64((uintptr_t)m | 1, cmd23, 1); dptr += l2_len; + + if (sess_priv.mode == ROC_IE_SA_MODE_TUNNEL) { + if (sess_priv.outer_ip_ver == ROC_IE_SA_IP_VERSION_4) + *((uint16_t *)(dptr - 2)) = + rte_cpu_to_be_16(RTE_ETHER_TYPE_IPV4); + else + *((uint16_t *)(dptr - 2)) = + rte_cpu_to_be_16(RTE_ETHER_TYPE_IPV6); + } + ucode_cmd[1] = dptr; ucode_cmd[2] = dptr; @@ -396,6 +406,15 @@ cn10k_nix_prep_sec(struct rte_mbuf *m, uint64_t *cmd, uintptr_t *nixtx_addr, cmd23 = vsetq_lane_u64((uintptr_t)m | 1, cmd23, 1); dptr += l2_len; + + if (sess_priv.mode == ROC_IE_SA_MODE_TUNNEL) { + if (sess_priv.outer_ip_ver == ROC_IE_SA_IP_VERSION_4) + *((uint16_t *)(dptr - 2)) = + rte_cpu_to_be_16(RTE_ETHER_TYPE_IPV4); + else + *((uint16_t *)(dptr - 2)) = + rte_cpu_to_be_16(RTE_ETHER_TYPE_IPV6); + } ucode_cmd[1] = dptr; ucode_cmd[2] = dptr; From patchwork Thu Sep 2 02:15:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97750 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 405A1A0C4C; Thu, 2 Sep 2021 04:19:26 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 04E944111C; Thu, 2 Sep 2021 04:18:00 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 1B8614115F for ; Thu, 2 Sep 2021 04:17:58 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQ206011573 for ; Wed, 1 Sep 2021 19:17:57 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=wzIbY17ynw7nnBl4sztWsStrsmxaGwRQb0XaMKzd5zQ=; b=At2HUnul9ZuTfiBlMKB7QEHNqDY2bPdHVwB8ShBSzE2T3zGYiPtD+Go4PYnLhPZ6PzQ3 E9rKRmVnYOUdeTWebSGP9JRkHuuniOhsas3G8FTXAGtfpmOQcnksXtrywsyS+XD7mZyD +dCpaRE36YlmVAsXsq0+UexiLyCKOT5u22AF0h+1UnD/E1yfuex94G8viPJdu1FLIDLg /7C3n0Xne3wIrQwfGNbAKd+xpOdMt7JU87eNJyxJfqYXtCoQ+3OQLEE0oLf1UlMCbQIX ynKZ2AiWBsECQ1bSaK9a6VtksGvNQpBi6ZMxibQEBbQ0Wf5oTPSqMDH4zYRH3jLoH8Rx yQ== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hvd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:17:57 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:55 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:55 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 465753F704B; Wed, 1 Sep 2021 19:17:53 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:45:02 +0530 Message-ID: <20210902021505.17607-25-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: w0R0sePctz62BaSQeVNBgaOtSCd4f3U7 X-Proofpoint-GUID: w0R0sePctz62BaSQeVNBgaOtSCd4f3U7 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 24/27] net/cnxk: allow zero udp6 checksum for non inline device X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Srujana Challa Sets IP6_UDP_OPT in NIX RX config to allow optional UDP checksum for IPv6 in case of security offload. Signed-off-by: Srujana Challa --- drivers/net/cnxk/cnxk_ethdev.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/net/cnxk/cnxk_ethdev.c b/drivers/net/cnxk/cnxk_ethdev.c index 60a4df5..8a102aa 100644 --- a/drivers/net/cnxk/cnxk_ethdev.c +++ b/drivers/net/cnxk/cnxk_ethdev.c @@ -1005,6 +1005,9 @@ cnxk_nix_configure(struct rte_eth_dev *eth_dev) ROC_NIX_LF_RX_CFG_LEN_IL4 | ROC_NIX_LF_RX_CFG_LEN_IL3 | ROC_NIX_LF_RX_CFG_LEN_OL4 | ROC_NIX_LF_RX_CFG_LEN_OL3); + if (dev->rx_offloads & DEV_RX_OFFLOAD_SECURITY) + rx_cfg |= ROC_NIX_LF_RX_CFG_IP6_UDP_OPT; + nb_rxq = RTE_MAX(data->nb_rx_queues, 1); nb_txq = RTE_MAX(data->nb_tx_queues, 1); From patchwork Thu Sep 2 02:15:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97751 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 894CCA0C4C; Thu, 2 Sep 2021 04:19:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0C3A04117A; Thu, 2 Sep 2021 04:18:03 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 29F694117A for ; Thu, 2 Sep 2021 04:18:01 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQBO9011731 for ; Wed, 1 Sep 2021 19:18:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=oJyQ1climf/8kB5UTgqwX8xGM91qFZos9/nEO4lF5Qc=; b=hwhAK44itPf+JqeDXa+jNWOvzOKg/fJwfw4X+ahrYedqZ9G+wg1dMEeaynYsPOon3bp/ bX17p3JTc9BHVrSI9zN6+pPZXxBwnZ3crNuxuurMuKgiW6foGfaAGzlsNIJVrDhjb2B8 dRpyO9PtHlzRBbfk0SRwUkdLeR3ELP0WXbsZ1/chFMEB9SCZfrbQGsDAcm0IWb26+uqT uk3Gjaa2c6r5x4mEfKz5v0MLcWMBABC1XnM7d2Zt22YtketjcbOI6MgsxmM6IRvqKcyQ xMsdPGukDdb4NUzLftLjt0hYDMc7JB1S4uSMXSJaxDFFNjYl4S1P/GZ+sJJU7UuSszaw Zw== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hvm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:18:00 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:17:57 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:17:57 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 0EB753F704D; Wed, 1 Sep 2021 19:17:55 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:45:03 +0530 Message-ID: <20210902021505.17607-26-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 9EFAzpUUpu-esncnRPMudv1U0F4TNv30 X-Proofpoint-GUID: 9EFAzpUUpu-esncnRPMudv1U0F4TNv30 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 25/27] net/cnxk: add crypto capabilities for AES CBC and HMAC SHA1 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Srujana Challa Adds capabitlities for AES_CBC and HMAC_SHA1 for 9k security offload. Signed-off-by: Srujana Challa --- drivers/net/cnxk/cn10k_ethdev_sec.c | 40 +++++++++++++++++++++++++++++++++++++ drivers/net/cnxk/cn9k_ethdev_sec.c | 40 +++++++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+) diff --git a/drivers/net/cnxk/cn10k_ethdev_sec.c b/drivers/net/cnxk/cn10k_ethdev_sec.c index c66730a..82dc636 100644 --- a/drivers/net/cnxk/cn10k_ethdev_sec.c +++ b/drivers/net/cnxk/cn10k_ethdev_sec.c @@ -41,6 +41,46 @@ static struct rte_cryptodev_capabilities cn10k_eth_sec_crypto_caps[] = { }, } }, } }, + { /* AES CBC */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, + {.cipher = { + .algo = RTE_CRYPTO_CIPHER_AES_CBC, + .block_size = 16, + .key_size = { + .min = 16, + .max = 32, + .increment = 8 + }, + .iv_size = { + .min = 16, + .max = 16, + .increment = 0 + } + }, } + }, } + }, + { /* SHA1 HMAC */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_SHA1_HMAC, + .block_size = 64, + .key_size = { + .min = 20, + .max = 64, + .increment = 1 + }, + .digest_size = { + .min = 12, + .max = 12, + .increment = 0 + }, + }, } + }, } + }, RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() }; diff --git a/drivers/net/cnxk/cn9k_ethdev_sec.c b/drivers/net/cnxk/cn9k_ethdev_sec.c index deb1daf..b070ad5 100644 --- a/drivers/net/cnxk/cn9k_ethdev_sec.c +++ b/drivers/net/cnxk/cn9k_ethdev_sec.c @@ -40,6 +40,46 @@ static struct rte_cryptodev_capabilities cn9k_eth_sec_crypto_caps[] = { }, } }, } }, + { /* AES CBC */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_CIPHER, + {.cipher = { + .algo = RTE_CRYPTO_CIPHER_AES_CBC, + .block_size = 16, + .key_size = { + .min = 16, + .max = 32, + .increment = 8 + }, + .iv_size = { + .min = 16, + .max = 16, + .increment = 0 + } + }, } + }, } + }, + { /* SHA1 HMAC */ + .op = RTE_CRYPTO_OP_TYPE_SYMMETRIC, + {.sym = { + .xform_type = RTE_CRYPTO_SYM_XFORM_AUTH, + {.auth = { + .algo = RTE_CRYPTO_AUTH_SHA1_HMAC, + .block_size = 64, + .key_size = { + .min = 20, + .max = 64, + .increment = 1 + }, + .digest_size = { + .min = 12, + .max = 12, + .increment = 0 + }, + }, } + }, } + }, RTE_CRYPTODEV_END_OF_CAPABILITIES_LIST() }; From patchwork Thu Sep 2 02:15:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97752 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 832A5A0C4C; Thu, 2 Sep 2021 04:19:36 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 406DA411A8; Thu, 2 Sep 2021 04:18:05 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id C9507411A8 for ; Thu, 2 Sep 2021 04:18:03 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQ3fO011587 for ; Wed, 1 Sep 2021 19:18:03 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=jzDEMn3t+ZFdrEcAhQWfWxN337iBoaYbFFYe1EIHKyI=; b=GMwFY0BkGLv9rnG8FcHdbjhWuwBWFWcSFqoCfQ5+JQ+k95P78dwjrH7hBmQ2HAmjy7ZS 16TIu2VYYZp1qbaUVicQW/CXEUCcroCYRRlC1ssKJDjiBT3kXbI4OuwOQ+GIpKHTSXTA 59rRYo8gzQ4hz2YKvbycYahgndaxAcuglYK+6QEAlWvF0VAF/3xSudXP18SdUhi3WYrG 5gZZtMBldt2iDYudOXlbnJUwzKIZwrVa1Q39dKyOEvkhTHAx1ceqR32n5ovk14XTmeR/ V337iQSsLvSSpYJo/0U+0B4gnJ5HiX1VHWkW7m8Uoz9fRH8xx6ltDcDORp3nynmXZoCM eg== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hw5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:18:03 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:18:01 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:18:01 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id C001A3F7040; Wed, 1 Sep 2021 19:17:58 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , , Satheesh Paul Date: Thu, 2 Sep 2021 07:45:04 +0530 Message-ID: <20210902021505.17607-27-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: RRp0u7unvxugWSMOi_hep4LnVjp3G7-T X-Proofpoint-GUID: RRp0u7unvxugWSMOi_hep4LnVjp3G7-T X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 26/27] net/cnxk: add devargs for configuring channel mask X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Satheesh Paul This patch adds support to configure channel mask which will be used by rte flow when adding flow rules with inline IPsec action. Signed-off-by: Satheesh Paul --- doc/guides/nics/cnxk.rst | 20 +++++++++++++++++++ drivers/net/cnxk/cnxk_ethdev_sec.c | 39 +++++++++++++++++++++++++++++++++++++- 2 files changed, 58 insertions(+), 1 deletion(-) diff --git a/doc/guides/nics/cnxk.rst b/doc/guides/nics/cnxk.rst index b542437..dd955d3 100644 --- a/doc/guides/nics/cnxk.rst +++ b/doc/guides/nics/cnxk.rst @@ -255,6 +255,26 @@ Runtime Config Options With the above configuration, inbound encrypted traffic from both the ports is received by ipsec inline device. +- ``Inline IPsec device channel and mask`` (default ``none``) + + Set channel and channel mask configuration for the inline IPSec device. This + will be used when creating flow rules with RTE_FLOW_ACTION_TYPE_SECURITY + action. + + By default, RTE Flow API sets the channel number of the port on which the + rule is created in the MCAM entry and matches it exactly. This behaviour can + be modified using the ``inl_cpt_channel`` ``devargs`` parameter. + + For example:: + + -a 0002:1d:00.0,inl_cpt_channel=0x100/0xf00 + + With the above configuration, RTE Flow rules API will set the channel + and channel mask as 0x100 and 0xF00 in the MCAM entries of the flow rules + created with RTE_FLOW_ACTION_TYPE_SECURITY action. Since channel number is + set with this custom mask, inbound encrypted traffic from all ports with + matching channel number pattern will be directed to the inline IPSec device. + .. note:: Above devarg parameters are configurable per device, user needs to pass the diff --git a/drivers/net/cnxk/cnxk_ethdev_sec.c b/drivers/net/cnxk/cnxk_ethdev_sec.c index c002c30..523837a 100644 --- a/drivers/net/cnxk/cnxk_ethdev_sec.c +++ b/drivers/net/cnxk/cnxk_ethdev_sec.c @@ -6,6 +6,13 @@ #define CNXK_NIX_INL_SELFTEST "selftest" #define CNXK_NIX_INL_IPSEC_IN_MAX_SPI "ipsec_in_max_spi" +#define CNXK_INL_CPT_CHANNEL "inl_cpt_channel" + +struct inl_cpt_channel { + bool is_multi_channel; + uint16_t channel; + uint16_t mask; +}; #define CNXK_NIX_INL_DEV_NAME RTE_STR(cnxk_nix_inl_dev_) #define CNXK_NIX_INL_DEV_NAME_LEN \ @@ -137,13 +144,37 @@ parse_selftest(const char *key, const char *value, void *extra_args) } static int +parse_inl_cpt_channel(const char *key, const char *value, void *extra_args) +{ + RTE_SET_USED(key); + uint16_t chan = 0, mask = 0; + char *next = 0; + + /* next will point to the separator '/' */ + chan = strtol(value, &next, 16); + mask = strtol(++next, 0, 16); + + if (chan > GENMASK(12, 0) || mask > GENMASK(12, 0)) + return -EINVAL; + + ((struct inl_cpt_channel *)extra_args)->channel = chan; + ((struct inl_cpt_channel *)extra_args)->mask = mask; + ((struct inl_cpt_channel *)extra_args)->is_multi_channel = true; + + return 0; +} + +static int nix_inl_parse_devargs(struct rte_devargs *devargs, struct roc_nix_inl_dev *inl_dev) { uint32_t ipsec_in_max_spi = BIT(8) - 1; + struct inl_cpt_channel cpt_channel; struct rte_kvargs *kvlist; uint8_t selftest = 0; + memset(&cpt_channel, 0, sizeof(cpt_channel)); + if (devargs == NULL) goto null_devargs; @@ -155,11 +186,16 @@ nix_inl_parse_devargs(struct rte_devargs *devargs, &selftest); rte_kvargs_process(kvlist, CNXK_NIX_INL_IPSEC_IN_MAX_SPI, &parse_ipsec_in_max_spi, &ipsec_in_max_spi); + rte_kvargs_process(kvlist, CNXK_INL_CPT_CHANNEL, &parse_inl_cpt_channel, + &cpt_channel); rte_kvargs_free(kvlist); null_devargs: inl_dev->ipsec_in_max_spi = ipsec_in_max_spi; inl_dev->selftest = selftest; + inl_dev->channel = cpt_channel.channel; + inl_dev->chan_mask = cpt_channel.mask; + inl_dev->is_multi_channel = cpt_channel.is_multi_channel; return 0; exit: return -EINVAL; @@ -275,4 +311,5 @@ RTE_PMD_REGISTER_KMOD_DEP(cnxk_nix_inl, "vfio-pci"); RTE_PMD_REGISTER_PARAM_STRING(cnxk_nix_inl, CNXK_NIX_INL_SELFTEST "=1" - CNXK_NIX_INL_IPSEC_IN_MAX_SPI "=<1-65535>"); + CNXK_NIX_INL_IPSEC_IN_MAX_SPI "=<1-65535>" + CNXK_INL_CPT_CHANNEL "=<1-4095>/<1-4095>"); From patchwork Thu Sep 2 02:15:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nithin Dabilpuram X-Patchwork-Id: 97753 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 83229A0C4C; Thu, 2 Sep 2021 04:19:42 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AFE47411B2; Thu, 2 Sep 2021 04:18:07 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 8C8F7411B3 for ; Thu, 2 Sep 2021 04:18:06 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 181HQSfM012709 for ; Wed, 1 Sep 2021 19:18:06 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=o0s4yjX5G2quxbz74T2KF9FkL5QjwO0saYNOOSBIBC8=; b=IAbUaDf/igre5ZTS5kBbvfUi/RkIQkk9pvpWGuC9KgqM2Kscp11EW3csQ7ip/7pzvvwC ii1ApOggBv8XbWyoe1KUbyH/JxdMhmG97mqtwN2Fd0UQllcs8g3giJvU5Pdl1CFcqaEu R8rEZdLkwHqc56oY62UqX/lTIvVlVnuEy+A9W2pEUCLrTO30kGWyGd9OPBOLcULb22zV SRmB+A2r0oKTiFY7vsqGV0ByaWIiRGeUuBBZ0xqXq5cGe3f9Ecw4iChde//w7VPwa8ED zW5iM1F7O3zJDeQVa6l3v4QEZqagOpf7fof5FP3Zo3ax9dosS54TjRp+dAwseC1+o+g8 /g== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 3atdwq9hwa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Wed, 01 Sep 2021 19:18:05 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 1 Sep 2021 19:18:03 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 1 Sep 2021 19:18:03 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id BB0213F7050; Wed, 1 Sep 2021 19:18:01 -0700 (PDT) From: Nithin Dabilpuram To: Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao CC: , , Date: Thu, 2 Sep 2021 07:45:05 +0530 Message-ID: <20210902021505.17607-28-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210902021505.17607-1-ndabilpuram@marvell.com> References: <20210902021505.17607-1-ndabilpuram@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: PtsEch3oFNGMZtlcyZFGWTZoy-EUOkyS X-Proofpoint-GUID: PtsEch3oFNGMZtlcyZFGWTZoy-EUOkyS X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-01_05,2021-09-01_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH 27/27] net/cnxk: reflect globally enabled offloads in queue conf X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Reflect globally enabled Rx and Tx offloads in queue conf. Also fix issue with lmt data prepare for multi seg. Fixes: a24af6361e37 ("net/cnxk: add Tx queue setup and release") Fixes: a86144cd9ded ("net/cnxk: add Rx queue setup and release") Fixes: 305ca2c4c382 ("net/cnxk: support multi-segment vector Tx") Signed-off-by: Nithin Dabilpuram --- drivers/net/cnxk/cn10k_tx.h | 2 +- drivers/net/cnxk/cnxk_ethdev.c | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/net/cnxk/cn10k_tx.h b/drivers/net/cnxk/cn10k_tx.h index f56aa8e..2821f71 100644 --- a/drivers/net/cnxk/cn10k_tx.h +++ b/drivers/net/cnxk/cn10k_tx.h @@ -1280,7 +1280,7 @@ cn10k_nix_prep_lmt_mseg_vector(struct rte_mbuf **mbufs, uint64x2_t *cmd0, vst1q_u64(lmt_addr + 14, cmd1[3]); *data128 |= ((__uint128_t)7) << *shift; - shift += 3; + *shift += 3; return 1; } diff --git a/drivers/net/cnxk/cnxk_ethdev.c b/drivers/net/cnxk/cnxk_ethdev.c index 8a102aa..978ee5b 100644 --- a/drivers/net/cnxk/cnxk_ethdev.c +++ b/drivers/net/cnxk/cnxk_ethdev.c @@ -377,6 +377,8 @@ cnxk_nix_tx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, txq_sp->dev = dev; txq_sp->qid = qid; txq_sp->qconf.conf.tx = *tx_conf; + /* Queue config should reflect global offloads */ + txq_sp->qconf.conf.tx.offloads = dev->tx_offloads; txq_sp->qconf.nb_desc = nb_desc; plt_nix_dbg("sq=%d fc=%p offload=0x%" PRIx64 " lmt_addr=%p" @@ -511,6 +513,8 @@ cnxk_nix_rx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, rxq_sp->dev = dev; rxq_sp->qid = qid; rxq_sp->qconf.conf.rx = *rx_conf; + /* Queue config should reflect global offloads */ + rxq_sp->qconf.conf.rx.offloads = dev->rx_offloads; rxq_sp->qconf.nb_desc = nb_desc; rxq_sp->qconf.mp = mp;