From patchwork Wed Feb 23 12:28:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akhil Goyal X-Patchwork-Id: 108167 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E7B14A034C; Wed, 23 Feb 2022 13:28:48 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C740D426E6; Wed, 23 Feb 2022 13:28:46 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id BE9C3426E5 for ; Wed, 23 Feb 2022 13:28:44 +0100 (CET) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21N9eViw003830; Wed, 23 Feb 2022 04:28:43 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=x4eS6lHMOiTM8NcZiWngcXhfZ5CqRMDhhhB2jlTO6FI=; b=C//g92UX9HI80wOo3JtpBLsH74Csg1LahF7+0tus5orOTfrdVz/4i6hspbNp1c9bJ5/e TfeN8y30rGxqmFvgbXuFYgS5JM+moAvRv3bBo9PmM7Sb/Rd86/ArtWL9IjBY1yT/ynQT 5ioodAwEaErj40faZyd9ejr3+4AUFpkJJ7q/xL2ofz+f27hyAICMV1etajk4NssjvhFQ mU0fS+PhIJ4XhTnDNe+H/WuFtdFw2t/CXi610pIlZbdLFxUQEo9JfZg2d9wRC9BAOVf9 1ji/olUT//f+BZuE6kOWAbaQQchrqALCLHWL4cM2w3K94eHOVOjvexujDA+w27HTQlgl 2A== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3edjga0n7a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 23 Feb 2022 04:28:43 -0800 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 23 Feb 2022 04:28:42 -0800 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Wed, 23 Feb 2022 04:28:42 -0800 Received: from localhost.localdomain (unknown [10.28.48.55]) by maili.marvell.com (Postfix) with ESMTP id C8CE63F7088; Wed, 23 Feb 2022 04:28:39 -0800 (PST) From: Akhil Goyal To: CC: , , , , , , Subject: [PATCH v3 1/2] common/cnxk: configure reassembly specific params Date: Wed, 23 Feb 2022 17:58:33 +0530 Message-ID: <20220223122834.2571860-2-gakhil@marvell.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220223122834.2571860-1-gakhil@marvell.com> References: <20220120165310.4165567-1-gakhil@marvell.com> <20220223122834.2571860-1-gakhil@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: MbDVf2W19bdt-QcDP-SgZef6K6nNG0TP X-Proofpoint-GUID: MbDVf2W19bdt-QcDP-SgZef6K6nNG0TP X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-02-23_05,2022-02-23_01,2022-02-23_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Vidya Sagar Velumuri When reassembly is enabled by application, set corresponding flags in SA during creation. Provide roc API to configure reassembly unit with active and zombie limits and step size Signed-off-by: Vidya Sagar Velumuri --- drivers/common/cnxk/cnxk_security.c | 10 ++++++++++ drivers/common/cnxk/roc_nix_inl.c | 23 +++++++++++++++++++++++ drivers/common/cnxk/roc_nix_inl.h | 7 +++++++ drivers/common/cnxk/version.map | 1 + 4 files changed, 41 insertions(+) diff --git a/drivers/common/cnxk/cnxk_security.c b/drivers/common/cnxk/cnxk_security.c index d6006d3176..ec808c0033 100644 --- a/drivers/common/cnxk/cnxk_security.c +++ b/drivers/common/cnxk/cnxk_security.c @@ -339,6 +339,16 @@ cnxk_ot_ipsec_inb_sa_fill(struct roc_ot_ipsec_inb_sa *sa, if (rc) return rc; + /* Default options for pkt_out and pkt_fmt are with + * second pass meta and no defrag. + */ + sa->w0.s.pkt_format = ROC_IE_OT_SA_PKT_FMT_META; + sa->w0.s.pkt_output = ROC_IE_OT_SA_PKT_OUTPUT_NO_FRAG; + sa->w0.s.pkind = ROC_IE_OT_CPT_PKIND; + + if (ipsec_xfrm->options.ip_reassembly_en) + sa->w0.s.pkt_output = ROC_IE_OT_SA_PKT_OUTPUT_HW_BASED_DEFRAG; + /* ESN */ sa->w2.s.esn_en = !!ipsec_xfrm->options.esn; if (ipsec_xfrm->options.udp_encap) { diff --git a/drivers/common/cnxk/roc_nix_inl.c b/drivers/common/cnxk/roc_nix_inl.c index 7bf89a44c1..9db97b65df 100644 --- a/drivers/common/cnxk/roc_nix_inl.c +++ b/drivers/common/cnxk/roc_nix_inl.c @@ -231,6 +231,29 @@ roc_nix_inl_inb_sa_get(struct roc_nix *roc_nix, bool inb_inl_dev, uint32_t spi) return (sa_base + ((spi & mask) * sz)); } +int +roc_nix_reassembly_configure(uint32_t max_wait_time, uint16_t max_frags) +{ + struct idev_cfg *idev = idev_get_cfg(); + struct roc_cpt *roc_cpt; + struct roc_cpt_rxc_time_cfg cfg; + + (void)max_frags; + roc_cpt = idev->cpt; + if (!roc_cpt) { + plt_err("Cannot support inline inbound, cryptodev not probed"); + return -ENOTSUP; + } + + cfg.step = (max_wait_time * 1000 / ROC_NIX_INL_REAS_ACTIVE_LIMIT); + cfg.zombie_limit = ROC_NIX_INL_REAS_ZOMBIE_LIMIT; + cfg.zombie_thres = ROC_NIX_INL_REAS_ZOMBIE_THRESHOLD; + cfg.active_limit = ROC_NIX_INL_REAS_ACTIVE_LIMIT; + cfg.active_thres = ROC_NIX_INL_REAS_ACTIVE_THRESHOLD; + + return roc_cpt_rxc_time_cfg(roc_cpt, &cfg); +} + int roc_nix_inl_inb_init(struct roc_nix *roc_nix) { diff --git a/drivers/common/cnxk/roc_nix_inl.h b/drivers/common/cnxk/roc_nix_inl.h index 5eb1a81471..45d349acf4 100644 --- a/drivers/common/cnxk/roc_nix_inl.h +++ b/drivers/common/cnxk/roc_nix_inl.h @@ -43,6 +43,11 @@ /* Alignment of SA Base */ #define ROC_NIX_INL_SA_BASE_ALIGN BIT_ULL(16) +#define ROC_NIX_INL_REAS_ACTIVE_LIMIT 0xFFF +#define ROC_NIX_INL_REAS_ACTIVE_THRESHOLD 10 +#define ROC_NIX_INL_REAS_ZOMBIE_LIMIT 0xFFF +#define ROC_NIX_INL_REAS_ZOMBIE_THRESHOLD 10 + static inline struct roc_onf_ipsec_inb_sa * roc_nix_inl_onf_ipsec_inb_sa(uintptr_t base, uint64_t idx) { @@ -152,6 +157,8 @@ struct roc_nix_rq *__roc_api roc_nix_inl_dev_rq(void); int __roc_api roc_nix_inl_inb_tag_update(struct roc_nix *roc_nix, uint32_t tag_const, uint8_t tt); uint64_t __roc_api roc_nix_inl_dev_rq_limit_get(void); +int __roc_api roc_nix_reassembly_configure(uint32_t max_wait_time, + uint16_t max_frags); /* NIX Inline Outbound API */ int __roc_api roc_nix_inl_outb_init(struct roc_nix *roc_nix); diff --git a/drivers/common/cnxk/version.map b/drivers/common/cnxk/version.map index dbede4aeed..2e8da4c2a7 100644 --- a/drivers/common/cnxk/version.map +++ b/drivers/common/cnxk/version.map @@ -206,6 +206,7 @@ INTERNAL { roc_nix_ptp_tx_ena_dis; roc_nix_queues_ctx_dump; roc_nix_ras_intr_ena_dis; + roc_nix_reassembly_configure; roc_nix_register_cq_irqs; roc_nix_register_queue_irqs; roc_nix_rq_dump; From patchwork Wed Feb 23 12:28:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akhil Goyal X-Patchwork-Id: 108168 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 45B8DA034C; Wed, 23 Feb 2022 13:28:56 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 14F73426ED; Wed, 23 Feb 2022 13:28:52 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 6A933426F5 for ; Wed, 23 Feb 2022 13:28:49 +0100 (CET) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 21NC87Br023579; Wed, 23 Feb 2022 04:28:49 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=kJrNJCTyjdutghhPPWNyUU0oYzVRwTzw7gFgMi8KXCk=; b=C9S2AsILTq8gGRWwfNGHl3H9YesgDZn1ETh4aKLzESi3/GJ25fS1gGsNxAdzM0kIiO+F Wovc5d3ukHLoOYUIYu0/KPQD/xyIAtB/9VuUGDP630xrwLpOK8T0SQM3nL70bMNuWUDZ o885n36QwaYdfOuVBxoS7Vw3+6UW2yd/OSTCx1GNcn0LvScPe2pRdGBQrheKYoVsBNVB 1A4nWt3vN5hYF/CLW6e0LSI43gPalrGofV5+ypSu2yuDmtjknO/TNYyNOGc/bWkt/AjI 3UX29c3MCkHkIfs3qtlKUfiBQAj713WqaCxbpm5C4feQ+WrLv58iN7HKG150uE55SMJv 6g== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3edjergmuj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 23 Feb 2022 04:28:48 -0800 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 23 Feb 2022 04:28:46 -0800 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Wed, 23 Feb 2022 04:28:46 -0800 Received: from localhost.localdomain (unknown [10.28.48.55]) by maili.marvell.com (Postfix) with ESMTP id AE4813F708A; Wed, 23 Feb 2022 04:28:42 -0800 (PST) From: Akhil Goyal To: CC: , , , , , , , Akhil Goyal Subject: [PATCH v3 2/2] net/cnxk: support IP reassembly Date: Wed, 23 Feb 2022 17:58:34 +0530 Message-ID: <20220223122834.2571860-3-gakhil@marvell.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220223122834.2571860-1-gakhil@marvell.com> References: <20220120165310.4165567-1-gakhil@marvell.com> <20220223122834.2571860-1-gakhil@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: P0R0BzRbdPZbb_pQ7uc5fWTpjJeT0SAL X-Proofpoint-GUID: P0R0BzRbdPZbb_pQ7uc5fWTpjJeT0SAL X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-02-23_05,2022-02-23_01,2022-02-23_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Vidya Sagar Velumuri Add capability and support for inbound reassembly in cnxk driver. Register the dynamic field for IPsec reassembly. Attach the fragments using the dynamic field in case of incomplete reassembly Signed-off-by: Vidya Sagar Velumuri Signed-off-by: Akhil Goyal Signed-off-by: Nithin Dabilpuram --- drivers/event/cnxk/cn10k_worker.h | 16 +- drivers/event/cnxk/deq/cn10k/deq_192_207.c | 12 + .../event/cnxk/deq/cn10k/deq_192_207_burst.c | 14 + drivers/event/cnxk/deq/cn10k/deq_192_207_ca.c | 12 + .../cnxk/deq/cn10k/deq_192_207_ca_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_192_207_ca_seg.c | 12 + .../cnxk/deq/cn10k/deq_192_207_ca_seg_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_192_207_ca_tmo.c | 12 + .../cnxk/deq/cn10k/deq_192_207_ca_tmo_burst.c | 14 + .../cnxk/deq/cn10k/deq_192_207_ca_tmo_seg.c | 13 + .../deq/cn10k/deq_192_207_ca_tmo_seg_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_192_207_dual.c | 12 + .../event/cnxk/deq/cn10k/deq_192_207_seg.c | 12 + .../cnxk/deq/cn10k/deq_192_207_seg_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_192_207_tmo.c | 12 + .../cnxk/deq/cn10k/deq_192_207_tmo_burst.c | 14 + .../cnxk/deq/cn10k/deq_192_207_tmo_seg.c | 12 + .../deq/cn10k/deq_192_207_tmo_seg_burst.c | 14 + drivers/event/cnxk/deq/cn10k/deq_208_223.c | 12 + .../event/cnxk/deq/cn10k/deq_208_223_burst.c | 14 + drivers/event/cnxk/deq/cn10k/deq_208_223_ca.c | 12 + .../cnxk/deq/cn10k/deq_208_223_ca_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_208_223_ca_seg.c | 12 + .../cnxk/deq/cn10k/deq_208_223_ca_seg_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_208_223_ca_tmo.c | 12 + .../cnxk/deq/cn10k/deq_208_223_ca_tmo_burst.c | 14 + .../cnxk/deq/cn10k/deq_208_223_ca_tmo_seg.c | 13 + .../deq/cn10k/deq_208_223_ca_tmo_seg_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_208_223_dual.c | 12 + .../event/cnxk/deq/cn10k/deq_208_223_seg.c | 12 + .../cnxk/deq/cn10k/deq_208_223_seg_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_208_223_tmo.c | 12 + .../cnxk/deq/cn10k/deq_208_223_tmo_burst.c | 14 + .../cnxk/deq/cn10k/deq_208_223_tmo_seg.c | 12 + .../deq/cn10k/deq_208_223_tmo_seg_burst.c | 14 + drivers/event/cnxk/deq/cn10k/deq_224_239.c | 12 + .../event/cnxk/deq/cn10k/deq_224_239_burst.c | 14 + drivers/event/cnxk/deq/cn10k/deq_224_239_ca.c | 12 + .../cnxk/deq/cn10k/deq_224_239_ca_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_224_239_ca_seg.c | 12 + .../cnxk/deq/cn10k/deq_224_239_ca_seg_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_224_239_ca_tmo.c | 12 + .../cnxk/deq/cn10k/deq_224_239_ca_tmo_burst.c | 14 + .../cnxk/deq/cn10k/deq_224_239_ca_tmo_seg.c | 13 + .../deq/cn10k/deq_224_239_ca_tmo_seg_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_224_239_dual.c | 12 + .../event/cnxk/deq/cn10k/deq_224_239_seg.c | 12 + .../cnxk/deq/cn10k/deq_224_239_seg_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_224_239_tmo.c | 12 + .../cnxk/deq/cn10k/deq_224_239_tmo_burst.c | 14 + .../cnxk/deq/cn10k/deq_224_239_tmo_seg.c | 12 + .../deq/cn10k/deq_224_239_tmo_seg_burst.c | 14 + drivers/event/cnxk/deq/cn10k/deq_240_255.c | 12 + .../event/cnxk/deq/cn10k/deq_240_255_burst.c | 14 + drivers/event/cnxk/deq/cn10k/deq_240_255_ca.c | 12 + .../cnxk/deq/cn10k/deq_240_255_ca_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_240_255_ca_seg.c | 12 + .../cnxk/deq/cn10k/deq_240_255_ca_seg_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_240_255_ca_tmo.c | 12 + .../cnxk/deq/cn10k/deq_240_255_ca_tmo_burst.c | 14 + .../cnxk/deq/cn10k/deq_240_255_ca_tmo_seg.c | 13 + .../deq/cn10k/deq_240_255_ca_tmo_seg_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_240_255_dual.c | 12 + .../event/cnxk/deq/cn10k/deq_240_255_seg.c | 12 + .../cnxk/deq/cn10k/deq_240_255_seg_burst.c | 14 + .../event/cnxk/deq/cn10k/deq_240_255_tmo.c | 12 + .../cnxk/deq/cn10k/deq_240_255_tmo_burst.c | 14 + .../cnxk/deq/cn10k/deq_240_255_tmo_seg.c | 12 + .../deq/cn10k/deq_240_255_tmo_seg_burst.c | 14 + drivers/event/cnxk/meson.build | 64 ++ drivers/net/cnxk/cn10k_ethdev.c | 51 ++ drivers/net/cnxk/cn10k_ethdev.h | 2 + drivers/net/cnxk/cn10k_ethdev_sec.c | 14 + drivers/net/cnxk/cn10k_rx.c | 35 +- drivers/net/cnxk/cn10k_rx.h | 672 ++++++++++++++++-- drivers/net/cnxk/cnxk_ethdev.h | 4 + drivers/net/cnxk/meson.build | 16 + drivers/net/cnxk/rx/cn10k/rx_192_207.c | 11 + drivers/net/cnxk/rx/cn10k/rx_192_207_mseg.c | 11 + drivers/net/cnxk/rx/cn10k/rx_192_207_vec.c | 11 + .../net/cnxk/rx/cn10k/rx_192_207_vec_mseg.c | 12 + drivers/net/cnxk/rx/cn10k/rx_208_223.c | 11 + drivers/net/cnxk/rx/cn10k/rx_208_223_mseg.c | 11 + drivers/net/cnxk/rx/cn10k/rx_208_223_vec.c | 11 + .../net/cnxk/rx/cn10k/rx_208_223_vec_mseg.c | 12 + drivers/net/cnxk/rx/cn10k/rx_224_239.c | 11 + drivers/net/cnxk/rx/cn10k/rx_224_239_mseg.c | 11 + drivers/net/cnxk/rx/cn10k/rx_224_239_vec.c | 11 + .../net/cnxk/rx/cn10k/rx_224_239_vec_mseg.c | 12 + drivers/net/cnxk/rx/cn10k/rx_240_255.c | 11 + drivers/net/cnxk/rx/cn10k/rx_240_255_mseg.c | 11 + drivers/net/cnxk/rx/cn10k/rx_240_255_vec.c | 11 + .../net/cnxk/rx/cn10k/rx_240_255_vec_mseg.c | 12 + 93 files changed, 1864 insertions(+), 74 deletions(-) create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_ca.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_ca_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_ca_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_ca_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_dual.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_tmo.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_tmo_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_tmo_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_192_207_tmo_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_ca.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_ca_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_ca_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_ca_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_dual.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_tmo.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_tmo_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_tmo_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_208_223_tmo_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_ca.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_ca_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_ca_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_ca_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_dual.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_tmo.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_tmo_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_tmo_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_224_239_tmo_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_ca.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_ca_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_ca_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_ca_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_dual.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_seg_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_tmo.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_tmo_burst.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_tmo_seg.c create mode 100644 drivers/event/cnxk/deq/cn10k/deq_240_255_tmo_seg_burst.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_192_207.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_192_207_mseg.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_192_207_vec.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_192_207_vec_mseg.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_208_223.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_208_223_mseg.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_208_223_vec.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_208_223_vec_mseg.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_224_239.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_224_239_mseg.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_224_239_vec.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_224_239_vec_mseg.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_240_255.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_240_255_mseg.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_240_255_vec.c create mode 100644 drivers/net/cnxk/rx/cn10k/rx_240_255_vec_mseg.c diff --git a/drivers/event/cnxk/cn10k_worker.h b/drivers/event/cnxk/cn10k_worker.h index 08a3953643..4c166c5193 100644 --- a/drivers/event/cnxk/cn10k_worker.h +++ b/drivers/event/cnxk/cn10k_worker.h @@ -169,9 +169,11 @@ cn10k_process_vwqe(uintptr_t vwqe, uint16_t port_id, const uint32_t flags, /* Translate meta to mbuf */ if (flags & NIX_RX_OFFLOAD_SECURITY_F) { const uint64_t cq_w1 = *((const uint64_t *)cqe + 1); + const uint64_t cq_w5 = *((const uint64_t *)cqe + 5); - mbuf = nix_sec_meta_to_mbuf_sc(cq_w1, sa_base, laddr, - &loff, mbuf, d_off); + mbuf = nix_sec_meta_to_mbuf_sc(cq_w1, cq_w5, sa_base, laddr, + &loff, mbuf, d_off, + flags, mbuf_init); } cn10k_nix_cqe_to_mbuf(cqe, cqe->tag, mbuf, lookup_mem, @@ -237,26 +239,32 @@ cn10k_sso_hws_get_work(struct cn10k_sso_hws *ws, struct rte_event *ev, mbuf = gw.u64[1] - sizeof(struct rte_mbuf); rte_prefetch0((void *)mbuf); if (flags & NIX_RX_OFFLOAD_SECURITY_F) { + const uint64_t mbuf_init = 0x100010000ULL | + RTE_PKTMBUF_HEADROOM | + (flags & NIX_RX_OFFLOAD_TSTAMP_F ? 8 : 0); struct rte_mbuf *m; uintptr_t sa_base; uint64_t iova = 0; uint8_t loff = 0; uint16_t d_off; uint64_t cq_w1; + uint64_t cq_w5; m = (struct rte_mbuf *)mbuf; d_off = (uintptr_t)(m->buf_addr) - (uintptr_t)m; d_off += RTE_PKTMBUF_HEADROOM; cq_w1 = *(uint64_t *)(gw.u64[1] + 8); + cq_w5 = *(uint64_t *)(gw.u64[1] + 40); sa_base = cnxk_nix_sa_base_get(port, lookup_mem); sa_base &= ~(ROC_NIX_INL_SA_BASE_ALIGN - 1); mbuf = (uint64_t)nix_sec_meta_to_mbuf_sc( - cq_w1, sa_base, (uintptr_t)&iova, &loff, - (struct rte_mbuf *)mbuf, d_off); + cq_w1, cq_w5, sa_base, (uintptr_t)&iova, &loff, + (struct rte_mbuf *)mbuf, d_off, flags, + mbuf_init | ((uint64_t)port) << 48); if (loff) roc_npa_aura_op_free(m->pool->pool_id, 0, iova); diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207.c b/drivers/event/cnxk/deq/cn10k/deq_192_207.c new file mode 100644 index 0000000000..6f53dbce2a --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ(cn10k_sso_hws_deq_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_burst.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_burst.c new file mode 100644 index 0000000000..f9789efbb2 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_burst_##name, \ + cn10k_sso_hws_deq_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_ca.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca.c new file mode 100644 index 0000000000..56ed0cae38 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_CA(cn10k_sso_hws_deq_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_burst.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_burst.c new file mode 100644 index 0000000000..70ed05e6e9 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_ca_burst_##name, \ + cn10k_sso_hws_deq_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_seg.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_seg.c new file mode 100644 index 0000000000..806b9e4ef0 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_seg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_CA_SEG(cn10k_sso_hws_deq_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_seg_burst.c new file mode 100644 index 0000000000..93360814e1 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_ca_seg_burst_##name, \ + cn10k_sso_hws_deq_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo.c new file mode 100644 index 0000000000..43c735d123 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_TMO_CA(cn10k_sso_hws_deq_tmo_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo_burst.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo_burst.c new file mode 100644 index 0000000000..5f7da5500d --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_ca_burst_##name, \ + cn10k_sso_hws_deq_tmo_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo_seg.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo_seg.c new file mode 100644 index 0000000000..4c6126f13a --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo_seg.c @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_DEQ_TMO_CA_SEG(cn10k_sso_hws_deq_tmo_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo_seg_burst.c new file mode 100644 index 0000000000..81ae021b51 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_ca_tmo_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_ca_seg_burst_##name, \ + cn10k_sso_hws_deq_tmo_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_dual.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_dual.c new file mode 100644 index 0000000000..2adcdaf59c --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_dual.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DUAL_DEQ(cn10k_sso_hws_dual_deq_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_seg.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_seg.c new file mode 100644 index 0000000000..5d59f7fe23 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_seg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_SEG(cn10k_sso_hws_deq_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_seg_burst.c new file mode 100644 index 0000000000..ddebe2aa40 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_SEG_BURST(cn10k_sso_hws_deq_seg_burst_##name, \ + cn10k_sso_hws_deq_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_tmo.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_tmo.c new file mode 100644 index 0000000000..e1c5beb41d --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_tmo.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_TMO(cn10k_sso_hws_deq_tmo_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_tmo_burst.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_tmo_burst.c new file mode 100644 index 0000000000..5d733766ee --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_tmo_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_burst_##name, \ + cn10k_sso_hws_deq_tmo_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_tmo_seg.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_tmo_seg.c new file mode 100644 index 0000000000..12bff5902c --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_tmo_seg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_TMO_SEG(cn10k_sso_hws_deq_tmo_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_192_207_tmo_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_192_207_tmo_seg_burst.c new file mode 100644 index 0000000000..2d10cc6e68 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_192_207_tmo_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_seg_burst_##name, \ + cn10k_sso_hws_deq_tmo_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223.c b/drivers/event/cnxk/deq/cn10k/deq_208_223.c new file mode 100644 index 0000000000..965a9405f0 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ(cn10k_sso_hws_deq_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_burst.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_burst.c new file mode 100644 index 0000000000..de9a048a0e --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_burst_##name, \ + cn10k_sso_hws_deq_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_ca.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca.c new file mode 100644 index 0000000000..f9148acef3 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_CA(cn10k_sso_hws_deq_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_burst.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_burst.c new file mode 100644 index 0000000000..12a927a872 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_ca_burst_##name, \ + cn10k_sso_hws_deq_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_seg.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_seg.c new file mode 100644 index 0000000000..1b020fbf98 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_seg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_CA_SEG(cn10k_sso_hws_deq_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_seg_burst.c new file mode 100644 index 0000000000..2c826f2284 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_ca_seg_burst_##name, \ + cn10k_sso_hws_deq_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo.c new file mode 100644 index 0000000000..48c23b910d --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_TMO_CA(cn10k_sso_hws_deq_tmo_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo_burst.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo_burst.c new file mode 100644 index 0000000000..5381f1ce17 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_ca_burst_##name, \ + cn10k_sso_hws_deq_tmo_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo_seg.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo_seg.c new file mode 100644 index 0000000000..116d9efc2d --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo_seg.c @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_DEQ_TMO_CA_SEG(cn10k_sso_hws_deq_tmo_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo_seg_burst.c new file mode 100644 index 0000000000..86571c7027 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_ca_tmo_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_ca_seg_burst_##name, \ + cn10k_sso_hws_deq_tmo_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_dual.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_dual.c new file mode 100644 index 0000000000..0fb35b9c83 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_dual.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DUAL_DEQ(cn10k_sso_hws_dual_deq_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_seg.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_seg.c new file mode 100644 index 0000000000..df48426ff1 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_seg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_SEG(cn10k_sso_hws_deq_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_seg_burst.c new file mode 100644 index 0000000000..f1342e3232 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_SEG_BURST(cn10k_sso_hws_deq_seg_burst_##name, \ + cn10k_sso_hws_deq_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_tmo.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_tmo.c new file mode 100644 index 0000000000..6f2d909806 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_tmo.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_TMO(cn10k_sso_hws_deq_tmo_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_tmo_burst.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_tmo_burst.c new file mode 100644 index 0000000000..3c6f226c4b --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_tmo_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_burst_##name, \ + cn10k_sso_hws_deq_tmo_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_tmo_seg.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_tmo_seg.c new file mode 100644 index 0000000000..44a794233f --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_tmo_seg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_TMO_SEG(cn10k_sso_hws_deq_tmo_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_208_223_tmo_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_208_223_tmo_seg_burst.c new file mode 100644 index 0000000000..aa89930d3e --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_208_223_tmo_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_seg_burst_##name, \ + cn10k_sso_hws_deq_tmo_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239.c b/drivers/event/cnxk/deq/cn10k/deq_224_239.c new file mode 100644 index 0000000000..03f5bd3588 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ(cn10k_sso_hws_deq_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_burst.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_burst.c new file mode 100644 index 0000000000..091419eb9b --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_burst_##name, \ + cn10k_sso_hws_deq_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_ca.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca.c new file mode 100644 index 0000000000..8f4693e057 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_CA(cn10k_sso_hws_deq_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_burst.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_burst.c new file mode 100644 index 0000000000..474c9b1eee --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_ca_burst_##name, \ + cn10k_sso_hws_deq_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_seg.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_seg.c new file mode 100644 index 0000000000..478df24934 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_seg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_CA_SEG(cn10k_sso_hws_deq_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_seg_burst.c new file mode 100644 index 0000000000..3b78a0f91e --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_ca_seg_burst_##name, \ + cn10k_sso_hws_deq_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo.c new file mode 100644 index 0000000000..366ae27814 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_TMO_CA(cn10k_sso_hws_deq_tmo_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo_burst.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo_burst.c new file mode 100644 index 0000000000..db0751fece --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_ca_burst_##name, \ + cn10k_sso_hws_deq_tmo_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo_seg.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo_seg.c new file mode 100644 index 0000000000..6dc3d2f870 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo_seg.c @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_DEQ_TMO_CA_SEG(cn10k_sso_hws_deq_tmo_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo_seg_burst.c new file mode 100644 index 0000000000..11b1784868 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_ca_tmo_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_ca_seg_burst_##name, \ + cn10k_sso_hws_deq_tmo_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_dual.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_dual.c new file mode 100644 index 0000000000..35429014da --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_dual.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DUAL_DEQ(cn10k_sso_hws_dual_deq_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_seg.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_seg.c new file mode 100644 index 0000000000..19b5d2eedd --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_seg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_SEG(cn10k_sso_hws_deq_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_seg_burst.c new file mode 100644 index 0000000000..fe8f8099b0 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_SEG_BURST(cn10k_sso_hws_deq_seg_burst_##name, \ + cn10k_sso_hws_deq_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_tmo.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_tmo.c new file mode 100644 index 0000000000..4f196ddaaf --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_tmo.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_TMO(cn10k_sso_hws_deq_tmo_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_tmo_burst.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_tmo_burst.c new file mode 100644 index 0000000000..252cf4f44d --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_tmo_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_burst_##name, \ + cn10k_sso_hws_deq_tmo_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_tmo_seg.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_tmo_seg.c new file mode 100644 index 0000000000..675149ec3a --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_tmo_seg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_TMO_SEG(cn10k_sso_hws_deq_tmo_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_224_239_tmo_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_224_239_tmo_seg_burst.c new file mode 100644 index 0000000000..9347485e05 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_224_239_tmo_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_seg_burst_##name, \ + cn10k_sso_hws_deq_tmo_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255.c b/drivers/event/cnxk/deq/cn10k/deq_240_255.c new file mode 100644 index 0000000000..7890137548 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ(cn10k_sso_hws_deq_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_burst.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_burst.c new file mode 100644 index 0000000000..b1241915c8 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_burst_##name, \ + cn10k_sso_hws_deq_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_ca.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca.c new file mode 100644 index 0000000000..1f9abae4bc --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_CA(cn10k_sso_hws_deq_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_burst.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_burst.c new file mode 100644 index 0000000000..a518f5e285 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_ca_burst_##name, \ + cn10k_sso_hws_deq_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_seg.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_seg.c new file mode 100644 index 0000000000..f6408fbdd3 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_seg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_CA_SEG(cn10k_sso_hws_deq_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_seg_burst.c new file mode 100644 index 0000000000..76bfa4ac08 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_ca_seg_burst_##name, \ + cn10k_sso_hws_deq_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo.c new file mode 100644 index 0000000000..f28b909b66 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_TMO_CA(cn10k_sso_hws_deq_tmo_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo_burst.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo_burst.c new file mode 100644 index 0000000000..5c76f78dca --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_ca_burst_##name, \ + cn10k_sso_hws_deq_tmo_ca_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo_seg.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo_seg.c new file mode 100644 index 0000000000..35af75874b --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo_seg.c @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_DEQ_TMO_CA_SEG(cn10k_sso_hws_deq_tmo_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo_seg_burst.c new file mode 100644 index 0000000000..f8c6deb8ef --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_ca_tmo_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_ca_seg_burst_##name, \ + cn10k_sso_hws_deq_tmo_ca_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_dual.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_dual.c new file mode 100644 index 0000000000..0b227c2f99 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_dual.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DUAL_DEQ(cn10k_sso_hws_dual_deq_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_seg.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_seg.c new file mode 100644 index 0000000000..6af63491da --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_seg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_SEG(cn10k_sso_hws_deq_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_seg_burst.c new file mode 100644 index 0000000000..6b20efd787 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_SEG_BURST(cn10k_sso_hws_deq_seg_burst_##name, \ + cn10k_sso_hws_deq_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_tmo.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_tmo.c new file mode 100644 index 0000000000..c074e55d02 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_tmo.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_TMO(cn10k_sso_hws_deq_tmo_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_tmo_burst.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_tmo_burst.c new file mode 100644 index 0000000000..88bf1541cb --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_tmo_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_burst_##name, \ + cn10k_sso_hws_deq_tmo_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_tmo_seg.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_tmo_seg.c new file mode 100644 index 0000000000..560fc74f64 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_tmo_seg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) SSO_DEQ_TMO_SEG(cn10k_sso_hws_deq_tmo_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/deq/cn10k/deq_240_255_tmo_seg_burst.c b/drivers/event/cnxk/deq/cn10k/deq_240_255_tmo_seg_burst.c new file mode 100644 index 0000000000..92017de4e2 --- /dev/null +++ b/drivers/event/cnxk/deq/cn10k/deq_240_255_tmo_seg_burst.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_worker.h" +#include "cnxk_eventdev.h" +#include "cnxk_worker.h" + +#define R(name, flags) \ + SSO_CMN_DEQ_BURST(cn10k_sso_hws_deq_tmo_seg_burst_##name, \ + cn10k_sso_hws_deq_tmo_seg_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/event/cnxk/meson.build b/drivers/event/cnxk/meson.build index b27bae7b12..ae538f394d 100644 --- a/drivers/event/cnxk/meson.build +++ b/drivers/event/cnxk/meson.build @@ -330,6 +330,10 @@ sources += files( 'deq/cn10k/deq_80_95_burst.c', 'deq/cn10k/deq_96_111_burst.c', 'deq/cn10k/deq_112_127_burst.c', + 'deq/cn10k/deq_192_207_burst.c', + 'deq/cn10k/deq_208_223_burst.c', + 'deq/cn10k/deq_224_239_burst.c', + 'deq/cn10k/deq_240_255_burst.c', 'deq/cn10k/deq_0_15_seg_burst.c', 'deq/cn10k/deq_16_31_seg_burst.c', 'deq/cn10k/deq_32_47_seg_burst.c', @@ -338,6 +342,10 @@ sources += files( 'deq/cn10k/deq_80_95_seg_burst.c', 'deq/cn10k/deq_96_111_seg_burst.c', 'deq/cn10k/deq_112_127_seg_burst.c', + 'deq/cn10k/deq_192_207_seg_burst.c', + 'deq/cn10k/deq_208_223_seg_burst.c', + 'deq/cn10k/deq_224_239_seg_burst.c', + 'deq/cn10k/deq_240_255_seg_burst.c', 'deq/cn10k/deq_0_15.c', 'deq/cn10k/deq_16_31.c', 'deq/cn10k/deq_32_47.c', @@ -346,6 +354,10 @@ sources += files( 'deq/cn10k/deq_80_95.c', 'deq/cn10k/deq_96_111.c', 'deq/cn10k/deq_112_127.c', + 'deq/cn10k/deq_192_207.c', + 'deq/cn10k/deq_208_223.c', + 'deq/cn10k/deq_224_239.c', + 'deq/cn10k/deq_240_255.c', 'deq/cn10k/deq_0_15_seg.c', 'deq/cn10k/deq_16_31_seg.c', 'deq/cn10k/deq_32_47_seg.c', @@ -354,6 +366,10 @@ sources += files( 'deq/cn10k/deq_80_95_seg.c', 'deq/cn10k/deq_96_111_seg.c', 'deq/cn10k/deq_112_127_seg.c', + 'deq/cn10k/deq_192_207_seg.c', + 'deq/cn10k/deq_208_223_seg.c', + 'deq/cn10k/deq_224_239_seg.c', + 'deq/cn10k/deq_240_255_seg.c', 'deq/cn10k/deq_0_15_tmo.c', 'deq/cn10k/deq_16_31_tmo.c', 'deq/cn10k/deq_32_47_tmo.c', @@ -362,6 +378,10 @@ sources += files( 'deq/cn10k/deq_80_95_tmo.c', 'deq/cn10k/deq_96_111_tmo.c', 'deq/cn10k/deq_112_127_tmo.c', + 'deq/cn10k/deq_192_207_tmo.c', + 'deq/cn10k/deq_208_223_tmo.c', + 'deq/cn10k/deq_224_239_tmo.c', + 'deq/cn10k/deq_240_255_tmo.c', 'deq/cn10k/deq_0_15_tmo_burst.c', 'deq/cn10k/deq_16_31_tmo_burst.c', 'deq/cn10k/deq_32_47_tmo_burst.c', @@ -370,6 +390,10 @@ sources += files( 'deq/cn10k/deq_80_95_tmo_burst.c', 'deq/cn10k/deq_96_111_tmo_burst.c', 'deq/cn10k/deq_112_127_tmo_burst.c', + 'deq/cn10k/deq_192_207_tmo_burst.c', + 'deq/cn10k/deq_208_223_tmo_burst.c', + 'deq/cn10k/deq_224_239_tmo_burst.c', + 'deq/cn10k/deq_240_255_tmo_burst.c', 'deq/cn10k/deq_0_15_tmo_seg.c', 'deq/cn10k/deq_16_31_tmo_seg.c', 'deq/cn10k/deq_32_47_tmo_seg.c', @@ -378,6 +402,10 @@ sources += files( 'deq/cn10k/deq_80_95_tmo_seg.c', 'deq/cn10k/deq_96_111_tmo_seg.c', 'deq/cn10k/deq_112_127_tmo_seg.c', + 'deq/cn10k/deq_192_207_tmo_seg.c', + 'deq/cn10k/deq_208_223_tmo_seg.c', + 'deq/cn10k/deq_224_239_tmo_seg.c', + 'deq/cn10k/deq_240_255_tmo_seg.c', 'deq/cn10k/deq_0_15_tmo_seg_burst.c', 'deq/cn10k/deq_16_31_tmo_seg_burst.c', 'deq/cn10k/deq_32_47_tmo_seg_burst.c', @@ -386,6 +414,10 @@ sources += files( 'deq/cn10k/deq_80_95_tmo_seg_burst.c', 'deq/cn10k/deq_96_111_tmo_seg_burst.c', 'deq/cn10k/deq_112_127_tmo_seg_burst.c', + 'deq/cn10k/deq_192_207_tmo_seg_burst.c', + 'deq/cn10k/deq_208_223_tmo_seg_burst.c', + 'deq/cn10k/deq_224_239_tmo_seg_burst.c', + 'deq/cn10k/deq_240_255_tmo_seg_burst.c', 'deq/cn10k/deq_0_15_ca.c', 'deq/cn10k/deq_16_31_ca.c', 'deq/cn10k/deq_32_47_ca.c', @@ -394,6 +426,10 @@ sources += files( 'deq/cn10k/deq_80_95_ca.c', 'deq/cn10k/deq_96_111_ca.c', 'deq/cn10k/deq_112_127_ca.c', + 'deq/cn10k/deq_192_207_ca.c', + 'deq/cn10k/deq_208_223_ca.c', + 'deq/cn10k/deq_224_239_ca.c', + 'deq/cn10k/deq_240_255_ca.c', 'deq/cn10k/deq_0_15_ca_burst.c', 'deq/cn10k/deq_16_31_ca_burst.c', 'deq/cn10k/deq_32_47_ca_burst.c', @@ -402,6 +438,10 @@ sources += files( 'deq/cn10k/deq_80_95_ca_burst.c', 'deq/cn10k/deq_96_111_ca_burst.c', 'deq/cn10k/deq_112_127_ca_burst.c', + 'deq/cn10k/deq_192_207_ca_burst.c', + 'deq/cn10k/deq_208_223_ca_burst.c', + 'deq/cn10k/deq_224_239_ca_burst.c', + 'deq/cn10k/deq_240_255_ca_burst.c', 'deq/cn10k/deq_0_15_ca_seg.c', 'deq/cn10k/deq_16_31_ca_seg.c', 'deq/cn10k/deq_32_47_ca_seg.c', @@ -410,6 +450,10 @@ sources += files( 'deq/cn10k/deq_80_95_ca_seg.c', 'deq/cn10k/deq_96_111_ca_seg.c', 'deq/cn10k/deq_112_127_ca_seg.c', + 'deq/cn10k/deq_192_207_ca_seg.c', + 'deq/cn10k/deq_208_223_ca_seg.c', + 'deq/cn10k/deq_224_239_ca_seg.c', + 'deq/cn10k/deq_240_255_ca_seg.c', 'deq/cn10k/deq_0_15_ca_seg_burst.c', 'deq/cn10k/deq_16_31_ca_seg_burst.c', 'deq/cn10k/deq_32_47_ca_seg_burst.c', @@ -418,6 +462,10 @@ sources += files( 'deq/cn10k/deq_80_95_ca_seg_burst.c', 'deq/cn10k/deq_96_111_ca_seg_burst.c', 'deq/cn10k/deq_112_127_ca_seg_burst.c', + 'deq/cn10k/deq_192_207_ca_seg_burst.c', + 'deq/cn10k/deq_208_223_ca_seg_burst.c', + 'deq/cn10k/deq_224_239_ca_seg_burst.c', + 'deq/cn10k/deq_240_255_ca_seg_burst.c', 'deq/cn10k/deq_0_15_ca_tmo.c', 'deq/cn10k/deq_16_31_ca_tmo.c', 'deq/cn10k/deq_32_47_ca_tmo.c', @@ -426,6 +474,10 @@ sources += files( 'deq/cn10k/deq_80_95_ca_tmo.c', 'deq/cn10k/deq_96_111_ca_tmo.c', 'deq/cn10k/deq_112_127_ca_tmo.c', + 'deq/cn10k/deq_192_207_ca_tmo.c', + 'deq/cn10k/deq_208_223_ca_tmo.c', + 'deq/cn10k/deq_224_239_ca_tmo.c', + 'deq/cn10k/deq_240_255_ca_tmo.c', 'deq/cn10k/deq_0_15_ca_tmo_burst.c', 'deq/cn10k/deq_16_31_ca_tmo_burst.c', 'deq/cn10k/deq_32_47_ca_tmo_burst.c', @@ -434,6 +486,10 @@ sources += files( 'deq/cn10k/deq_80_95_ca_tmo_burst.c', 'deq/cn10k/deq_96_111_ca_tmo_burst.c', 'deq/cn10k/deq_112_127_ca_tmo_burst.c', + 'deq/cn10k/deq_192_207_ca_tmo_burst.c', + 'deq/cn10k/deq_208_223_ca_tmo_burst.c', + 'deq/cn10k/deq_224_239_ca_tmo_burst.c', + 'deq/cn10k/deq_240_255_ca_tmo_burst.c', 'deq/cn10k/deq_0_15_ca_tmo_seg.c', 'deq/cn10k/deq_16_31_ca_tmo_seg.c', 'deq/cn10k/deq_32_47_ca_tmo_seg.c', @@ -442,6 +498,10 @@ sources += files( 'deq/cn10k/deq_80_95_ca_tmo_seg.c', 'deq/cn10k/deq_96_111_ca_tmo_seg.c', 'deq/cn10k/deq_112_127_ca_tmo_seg.c', + 'deq/cn10k/deq_192_207_ca_tmo_seg.c', + 'deq/cn10k/deq_208_223_ca_tmo_seg.c', + 'deq/cn10k/deq_224_239_ca_tmo_seg.c', + 'deq/cn10k/deq_240_255_ca_tmo_seg.c', 'deq/cn10k/deq_0_15_ca_tmo_seg_burst.c', 'deq/cn10k/deq_16_31_ca_tmo_seg_burst.c', 'deq/cn10k/deq_32_47_ca_tmo_seg_burst.c', @@ -450,6 +510,10 @@ sources += files( 'deq/cn10k/deq_80_95_ca_tmo_seg_burst.c', 'deq/cn10k/deq_96_111_ca_tmo_seg_burst.c', 'deq/cn10k/deq_112_127_ca_tmo_seg_burst.c', + 'deq/cn10k/deq_192_207_ca_tmo_seg_burst.c', + 'deq/cn10k/deq_208_223_ca_tmo_seg_burst.c', + 'deq/cn10k/deq_224_239_ca_tmo_seg_burst.c', + 'deq/cn10k/deq_240_255_ca_tmo_seg_burst.c', ) sources += files( diff --git a/drivers/net/cnxk/cn10k_ethdev.c b/drivers/net/cnxk/cn10k_ethdev.c index e47e04d922..218334eefc 100644 --- a/drivers/net/cnxk/cn10k_ethdev.c +++ b/drivers/net/cnxk/cn10k_ethdev.c @@ -304,6 +304,10 @@ cn10k_nix_configure(struct rte_eth_dev *eth_dev) dev->rx_offload_flags = nix_rx_offload_flags(eth_dev); dev->tx_offload_flags = nix_tx_offload_flags(eth_dev); + /* reset reassembly dynfield/flag offset */ + dev->reass_dynfield_off = -1; + dev->reass_dynflag_bit = -1; + plt_nix_dbg("Configured port%d platform specific rx_offload_flags=%x" " tx_offload_flags=0x%x", eth_dev->data->port_id, dev->rx_offload_flags, @@ -478,6 +482,49 @@ cn10k_nix_rx_metadata_negotiate(struct rte_eth_dev *eth_dev, uint64_t *features) return 0; } +static int +cn10k_nix_reassembly_capability_get(struct rte_eth_dev *eth_dev, + struct rte_eth_ip_reassembly_params *reassembly_capa) +{ + struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + int rc = -ENOTSUP; + RTE_SET_USED(eth_dev); + + if (dev->rx_offloads & RTE_ETH_RX_OFFLOAD_SECURITY) { + reassembly_capa->timeout_ms = 60 * 1000; + reassembly_capa->max_frags = 4; + reassembly_capa->flags = RTE_ETH_DEV_REASSEMBLY_F_IPV4 | + RTE_ETH_DEV_REASSEMBLY_F_IPV6; + rc = 0; + } + + return rc; +} + +static int +cn10k_nix_reassembly_conf_get(struct rte_eth_dev *eth_dev, + struct rte_eth_ip_reassembly_params *conf) +{ + RTE_SET_USED(eth_dev); + RTE_SET_USED(conf); + return -ENOTSUP; +} + +static int +cn10k_nix_reassembly_conf_set(struct rte_eth_dev *eth_dev, + const struct rte_eth_ip_reassembly_params *conf) +{ + struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + int rc = 0; + + rc = roc_nix_reassembly_configure(conf->timeout_ms, + conf->max_frags); + if (!rc && dev->rx_offloads & RTE_ETH_RX_OFFLOAD_SECURITY) + dev->rx_offload_flags |= NIX_RX_OFFLOAD_REASSEMBLY_F; + + return rc; +} + /* Update platform specific eth dev ops */ static void nix_eth_dev_ops_override(void) @@ -499,6 +546,10 @@ nix_eth_dev_ops_override(void) cnxk_eth_dev_ops.timesync_disable = cn10k_nix_timesync_disable; cnxk_eth_dev_ops.rx_metadata_negotiate = cn10k_nix_rx_metadata_negotiate; + cnxk_eth_dev_ops.ip_reassembly_capability_get = + cn10k_nix_reassembly_capability_get; + cnxk_eth_dev_ops.ip_reassembly_conf_get = cn10k_nix_reassembly_conf_get; + cnxk_eth_dev_ops.ip_reassembly_conf_set = cn10k_nix_reassembly_conf_set; } static void diff --git a/drivers/net/cnxk/cn10k_ethdev.h b/drivers/net/cnxk/cn10k_ethdev.h index fd7273078a..ab51eafcb1 100644 --- a/drivers/net/cnxk/cn10k_ethdev.h +++ b/drivers/net/cnxk/cn10k_ethdev.h @@ -44,6 +44,8 @@ struct cn10k_eth_rxq { /* Private data in sw rsvd area of struct roc_ot_ipsec_inb_sa */ struct cn10k_inb_priv_data { void *userdata; + int reass_dynfield_off; + int reass_dynflag_bit; struct cnxk_eth_sec_sess *eth_sec; }; diff --git a/drivers/net/cnxk/cn10k_ethdev_sec.c b/drivers/net/cnxk/cn10k_ethdev_sec.c index a514345034..7ae926c792 100644 --- a/drivers/net/cnxk/cn10k_ethdev_sec.c +++ b/drivers/net/cnxk/cn10k_ethdev_sec.c @@ -336,6 +336,14 @@ cn10k_eth_sec_session_create(void *device, if (rte_security_dynfield_register() < 0) return -ENOTSUP; + if (conf->ipsec.options.ip_reassembly_en && + dev->reass_dynfield_off < 0) { + if (rte_eth_ip_reassembly_dynfield_register( + &dev->reass_dynfield_off, + &dev->reass_dynflag_bit) < 0) + return -rte_errno; + } + ipsec = &conf->ipsec; crypto = conf->crypto_xform; inbound = !!(ipsec->direction == RTE_SECURITY_IPSEC_SA_DIR_INGRESS); @@ -442,6 +450,12 @@ cn10k_eth_sec_session_create(void *device, sizeof(struct roc_ot_ipsec_inb_sa)); if (rc) goto mempool_put; + + if (conf->ipsec.options.ip_reassembly_en) { + inb_priv->reass_dynfield_off = dev->reass_dynfield_off; + inb_priv->reass_dynflag_bit = dev->reass_dynflag_bit; + } + } else { struct roc_ot_ipsec_outb_sa *outb_sa, *outb_sa_dptr; struct cn10k_outb_priv_data *outb_priv; diff --git a/drivers/net/cnxk/cn10k_rx.c b/drivers/net/cnxk/cn10k_rx.c index 5d603514c0..028abe8f39 100644 --- a/drivers/net/cnxk/cn10k_rx.c +++ b/drivers/net/cnxk/cn10k_rx.c @@ -5,7 +5,7 @@ #include "cn10k_ethdev.h" #include "cn10k_rx.h" -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, f7, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ @@ -17,12 +17,13 @@ NIX_RX_FASTPATH_MODES static inline void pick_rx_func(struct rte_eth_dev *eth_dev, - const eth_rx_burst_t rx_burst[2][2][2][2][2][2][2]) + const eth_rx_burst_t rx_burst[2][2][2][2][2][2][2][2]) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); /* [VLAN] [TSP] [MARK] [CKSUM] [PTYPE] [RSS] */ eth_dev->rx_pkt_burst = rx_burst + [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_REASSEMBLY_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_SECURITY_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_VLAN_STRIP_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_TSTAMP_F)] @@ -39,42 +40,42 @@ cn10k_eth_set_rx_function(struct rte_eth_dev *eth_dev) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - const eth_rx_burst_t nix_eth_rx_burst[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_##name, + const eth_rx_burst_t nix_eth_rx_burst[2][2][2][2][2][2][2][2] = { +#define R(name, f7, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f7][f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_burst_mseg[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_mseg_##name, + const eth_rx_burst_t nix_eth_rx_burst_mseg[2][2][2][2][2][2][2][2] = { +#define R(name, f7, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f7][f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_mseg_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_vec_burst[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_vec_##name, + const eth_rx_burst_t nix_eth_rx_vec_burst[2][2][2][2][2][2][2][2] = { +#define R(name, f7, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f7][f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_vec_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_vec_burst_mseg[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_vec_mseg_##name, + const eth_rx_burst_t nix_eth_rx_vec_burst_mseg[2][2][2][2][2][2][2][2] = { +#define R(name, f7, f6, f5, f4, f3, f2, f1, f0, flags) \ + [f7][f6][f5][f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_vec_mseg_##name, - NIX_RX_FASTPATH_MODES + NIX_RX_FASTPATH_MODES #undef R - }; + }; /* Copy multi seg version with no offload for tear down sequence */ if (rte_eal_process_type() == RTE_PROC_PRIMARY) dev->rx_pkt_burst_no_offload = - nix_eth_rx_burst_mseg[0][0][0][0][0][0][0]; + nix_eth_rx_burst_mseg[0][0][0][0][0][0][0][0]; if (dev->scalar_ena) { if (dev->rx_offloads & RTE_ETH_RX_OFFLOAD_SCATTER) diff --git a/drivers/net/cnxk/cn10k_rx.h b/drivers/net/cnxk/cn10k_rx.h index 6697e391ac..48f993119f 100644 --- a/drivers/net/cnxk/cn10k_rx.h +++ b/drivers/net/cnxk/cn10k_rx.h @@ -17,7 +17,8 @@ #define NIX_RX_OFFLOAD_TSTAMP_F BIT(4) #define NIX_RX_OFFLOAD_VLAN_STRIP_F BIT(5) #define NIX_RX_OFFLOAD_SECURITY_F BIT(6) -#define NIX_RX_OFFLOAD_MAX (NIX_RX_OFFLOAD_SECURITY_F << 1) +#define NIX_RX_OFFLOAD_REASSEMBLY_F BIT(7) +#define NIX_RX_OFFLOAD_MAX (NIX_RX_OFFLOAD_REASSEMBLY_F << 1) /* Flags to control cqe_to_mbuf conversion function. * Defining it from backwards to denote its been @@ -37,6 +38,17 @@ (uint64_t *)(((uintptr_t)((uint64_t *)(b))[i]) + (o)) : \ (uint64_t *)(((uintptr_t)(b)) + CQE_SZ(i) + (o))) +#define NIX_RX_SEC_REASSEMBLY_F \ + (NIX_RX_OFFLOAD_REASSEMBLY_F | NIX_RX_OFFLOAD_SECURITY_F) + +static inline rte_eth_ip_reassembly_dynfield_t * +cnxk_ip_reassembly_dynfield(struct rte_mbuf *mbuf, + int ip_reassembly_dynfield_offset) +{ + return RTE_MBUF_DYNFIELD(mbuf, ip_reassembly_dynfield_offset, + rte_eth_ip_reassembly_dynfield_t *); +} + union mbuf_initializer { struct { uint16_t data_off; @@ -104,19 +116,348 @@ nix_sec_flush_meta(uintptr_t laddr, uint16_t lmt_id, uint8_t loff, roc_lmt_submit_steorl(lmt_id, pa); } +static struct rte_mbuf * +nix_sec_attach_frags(const struct cpt_parse_hdr_s *hdr, + struct cn10k_inb_priv_data *inb_priv, + const uint64_t mbuf_init) +{ + struct rte_mbuf *head, *mbuf, *mbuf_prev; + uint32_t offset = hdr->w2.fi_offset; + union nix_rx_parse_u *frag_rx; + struct cpt_frag_info_s *finfo; + uint64_t *frag_ptr, ol_flags; + uint16_t frag_size; + uint16_t rlen; + uint64_t *wqe; + int off; + + off = inb_priv->reass_dynfield_off; + ol_flags = BIT_ULL(inb_priv->reass_dynflag_bit); + ol_flags |= RTE_MBUF_F_RX_SEC_OFFLOAD; + + /* offset of 0 implies 256B, otherwise it implies offset*8B */ + offset = (((offset - 1) & 0x1f) + 1) * 8; + finfo = RTE_PTR_ADD(hdr, offset + hdr->w2.fi_pad); + + /* Frag-0: */ + wqe = (uint64_t *)(rte_be_to_cpu_64(hdr->wqe_ptr)); + rlen = ((*(wqe + 10)) >> 16) & 0xFFFF; + + frag_rx = (union nix_rx_parse_u *)(wqe + 1); + frag_size = rlen + frag_rx->lcptr - frag_rx->laptr; + frag_rx->pkt_lenm1 = frag_size - 1; + + mbuf = (struct rte_mbuf *)((uintptr_t)wqe - sizeof(struct rte_mbuf)); + *(uint64_t *)(&mbuf->rearm_data) = mbuf_init; + mbuf->data_len = frag_size; + mbuf->pkt_len = frag_size; + mbuf->ol_flags = ol_flags; + mbuf->next = NULL; + head = mbuf; + mbuf_prev = mbuf; + /* Update dynamic field with userdata */ + *rte_security_dynfield(mbuf) = (uint64_t)inb_priv->userdata; + + cnxk_ip_reassembly_dynfield(head, off)->nb_frags = hdr->w0.num_frags - 1; + cnxk_ip_reassembly_dynfield(head, off)->next_frag = NULL; + + /* Frag-1: */ + if (hdr->w0.num_frags > 1) { + wqe = (uint64_t *)(rte_be_to_cpu_64(hdr->frag1_wqe_ptr)); + rlen = ((*(wqe + 10)) >> 16) & 0xFFFF; + + frag_rx = (union nix_rx_parse_u *)(wqe + 1); + frag_size = rlen + frag_rx->lcptr - frag_rx->laptr; + frag_rx->pkt_lenm1 = frag_size - 1; + + mbuf = (struct rte_mbuf *)((uintptr_t)wqe - + sizeof(struct rte_mbuf)); + *(uint64_t *)(&mbuf->rearm_data) = mbuf_init; + mbuf->data_len = frag_size; + mbuf->pkt_len = frag_size; + mbuf->ol_flags = ol_flags; + mbuf->next = NULL; + + /* Update dynamic field with userdata */ + *rte_security_dynfield(mbuf) = (uint64_t)inb_priv->userdata; + + cnxk_ip_reassembly_dynfield(mbuf, off)->nb_frags = + hdr->w0.num_frags - 2; + cnxk_ip_reassembly_dynfield(mbuf, off)->next_frag = NULL; + cnxk_ip_reassembly_dynfield(mbuf_prev, off)->next_frag = mbuf; + mbuf_prev = mbuf; + } + + /* Frag-2: */ + if (hdr->w0.num_frags > 2) { + frag_ptr = (uint64_t *)(finfo + 1); + wqe = (uint64_t *)(rte_be_to_cpu_64(*frag_ptr)); + rlen = ((*(wqe + 10)) >> 16) & 0xFFFF; + + frag_rx = (union nix_rx_parse_u *)(wqe + 1); + frag_size = rlen + frag_rx->lcptr - frag_rx->laptr; + frag_rx->pkt_lenm1 = frag_size - 1; + + mbuf = (struct rte_mbuf *)((uintptr_t)wqe - + sizeof(struct rte_mbuf)); + *(uint64_t *)(&mbuf->rearm_data) = mbuf_init; + mbuf->data_len = frag_size; + mbuf->pkt_len = frag_size; + mbuf->ol_flags = ol_flags; + mbuf->next = NULL; + + /* Update dynamic field with userdata */ + *rte_security_dynfield(mbuf) = (uint64_t)inb_priv->userdata; + + cnxk_ip_reassembly_dynfield(mbuf, off)->nb_frags = + hdr->w0.num_frags - 3; + cnxk_ip_reassembly_dynfield(mbuf, off)->next_frag = NULL; + cnxk_ip_reassembly_dynfield(mbuf_prev, off)->next_frag = mbuf; + mbuf_prev = mbuf; + } + + /* Frag-3: */ + if (hdr->w0.num_frags > 3) { + wqe = (uint64_t *)(rte_be_to_cpu_64(*(frag_ptr + 1))); + rlen = ((*(wqe + 10)) >> 16) & 0xFFFF; + + frag_rx = (union nix_rx_parse_u *)(wqe + 1); + frag_size = rlen + frag_rx->lcptr - frag_rx->laptr; + frag_rx->pkt_lenm1 = frag_size - 1; + + mbuf = (struct rte_mbuf *)((uintptr_t)wqe - + sizeof(struct rte_mbuf)); + *(uint64_t *)(&mbuf->rearm_data) = mbuf_init; + mbuf->data_len = frag_size; + mbuf->pkt_len = frag_size; + mbuf->ol_flags = ol_flags; + mbuf->next = NULL; + + /* Update dynamic field with userdata */ + *rte_security_dynfield(mbuf) = (uint64_t)inb_priv->userdata; + + cnxk_ip_reassembly_dynfield(mbuf, off)->nb_frags = + hdr->w0.num_frags - 4; + cnxk_ip_reassembly_dynfield(mbuf, off)->next_frag = NULL; + cnxk_ip_reassembly_dynfield(mbuf_prev, off)->next_frag = mbuf; + } + return head; +} + +static struct rte_mbuf * +nix_sec_reassemble_frags(const struct cpt_parse_hdr_s *hdr, uint64_t cq_w1, + uint64_t cq_w5, uint64_t mbuf_init) +{ + uint32_t fragx_sum, pkt_hdr_len, l3_hdr_size; + uint32_t offset = hdr->w2.fi_offset; + union nix_rx_parse_u *inner_rx; + uint16_t rlen, data_off, b_off; + union nix_rx_parse_u *frag_rx; + struct cpt_frag_info_s *finfo; + struct rte_mbuf *head, *mbuf; + rte_iova_t *inner_iova; + uint64_t *frag_ptr; + uint16_t frag_size; + uint64_t *wqe; + + /* Base data offset */ + b_off = mbuf_init & 0xFFFFUL; + mbuf_init &= ~0xFFFFUL; + + /* offset of 0 implies 256B, otherwise it implies offset*8B */ + offset = (((offset - 1) & 0x1f) + 1) * 8; + finfo = RTE_PTR_ADD(hdr, offset + hdr->w2.fi_pad); + + /* Frag-0: */ + wqe = (uint64_t *)rte_be_to_cpu_64(hdr->wqe_ptr); + inner_rx = (union nix_rx_parse_u *)(wqe + 1); + inner_iova = (rte_iova_t *)*(wqe + 9); + + /* Update only the upper 28-bits from meta pkt parse info */ + *((uint64_t *)inner_rx) = ((*((uint64_t *)inner_rx) & ((1ULL << 36) - 1)) | + (cq_w1 & ~((1ULL << 36) - 1))); + + rlen = ((*(wqe + 10)) >> 16) & 0xFFFF; + frag_size = rlen + ((cq_w5 >> 16) & 0xFF) - (cq_w5 & 0xFF); + fragx_sum = rte_be_to_cpu_16(finfo->w1.frag_size0); + pkt_hdr_len = frag_size - fragx_sum; + + mbuf = (struct rte_mbuf *)((uintptr_t)wqe - sizeof(struct rte_mbuf)); + *(uint64_t *)(&mbuf->rearm_data) = mbuf_init | b_off; + mbuf->data_len = frag_size; + head = mbuf; + + if (inner_rx->lctype == NPC_LT_LC_IP) { + struct rte_ipv4_hdr *hdr = (struct rte_ipv4_hdr *) + RTE_PTR_ADD(inner_iova, inner_rx->lcptr); + + l3_hdr_size = (hdr->version_ihl & 0xf) << 2; + } else { + struct rte_ipv6_hdr *hdr = (struct rte_ipv6_hdr *) + RTE_PTR_ADD(inner_iova, inner_rx->lcptr); + size_t ext_len = sizeof(struct rte_ipv6_hdr); + uint8_t *nxt_hdr = (uint8_t *)hdr; + int nh = hdr->proto; + + l3_hdr_size = 0; + while (nh != -EINVAL) { + nxt_hdr += ext_len; + l3_hdr_size += ext_len; + nh = rte_ipv6_get_next_ext(nxt_hdr, nh, &ext_len); + } + } + + /* Frag-1: */ + wqe = (uint64_t *)(rte_be_to_cpu_64(hdr->frag1_wqe_ptr)); + frag_size = rte_be_to_cpu_16(finfo->w1.frag_size1); + frag_rx = (union nix_rx_parse_u *)(wqe + 1); + + mbuf->next = (struct rte_mbuf *)((uintptr_t)wqe - sizeof(struct rte_mbuf)); + mbuf = mbuf->next; + data_off = b_off + frag_rx->lcptr + l3_hdr_size; + *(uint64_t *)(&mbuf->rearm_data) = mbuf_init | data_off; + mbuf->data_len = frag_size; + fragx_sum += frag_size; + + /* Frag-2: */ + if (hdr->w0.num_frags > 2) { + frag_ptr = (uint64_t *)(finfo + 1); + wqe = (uint64_t *)(rte_be_to_cpu_64(*frag_ptr)); + frag_size = rte_be_to_cpu_16(finfo->w1.frag_size2); + frag_rx = (union nix_rx_parse_u *)(wqe + 1); + + mbuf->next = (struct rte_mbuf *)((uintptr_t)wqe - sizeof(struct rte_mbuf)); + mbuf = mbuf->next; + data_off = b_off + frag_rx->lcptr + l3_hdr_size; + *(uint64_t *)(&mbuf->rearm_data) = mbuf_init | data_off; + mbuf->data_len = frag_size; + fragx_sum += frag_size; + } + + /* Frag-3: */ + if (hdr->w0.num_frags > 3) { + wqe = (uint64_t *)(rte_be_to_cpu_64(*(frag_ptr + 1))); + frag_size = rte_be_to_cpu_16(finfo->w1.frag_size3); + frag_rx = (union nix_rx_parse_u *)(wqe + 1); + + mbuf->next = (struct rte_mbuf *)((uintptr_t)wqe - sizeof(struct rte_mbuf)); + mbuf = mbuf->next; + data_off = b_off + frag_rx->lcptr + l3_hdr_size; + *(uint64_t *)(&mbuf->rearm_data) = mbuf_init | data_off; + mbuf->data_len = frag_size; + fragx_sum += frag_size; + } + + if (inner_rx->lctype == NPC_LT_LC_IP) { + struct rte_ipv4_hdr *hdr = (struct rte_ipv4_hdr *) + RTE_PTR_ADD(inner_iova, inner_rx->lcptr); + + hdr->fragment_offset = 0; + hdr->total_length = rte_cpu_to_be_16(fragx_sum + l3_hdr_size); + hdr->hdr_checksum = 0; + hdr->hdr_checksum = rte_ipv4_cksum(hdr); + + inner_rx->pkt_lenm1 = pkt_hdr_len + fragx_sum - 1; + } else { + /* Remove the frag header by moving header 8 bytes forward */ + struct rte_ipv6_hdr *hdr = (struct rte_ipv6_hdr *) + RTE_PTR_ADD(inner_iova, inner_rx->lcptr); + + hdr->payload_len = rte_cpu_to_be_16(fragx_sum + l3_hdr_size - + 8 - sizeof(struct rte_ipv6_hdr)); + + rte_memcpy(rte_pktmbuf_mtod_offset(head, void *, 8), + rte_pktmbuf_mtod(head, void *), + inner_rx->lcptr + sizeof(struct rte_ipv6_hdr)); + + inner_rx->pkt_lenm1 = pkt_hdr_len + fragx_sum - 8 - 1; + head->data_len -= 8; + head->data_off += 8; + } + mbuf->next = NULL; + head->pkt_len = inner_rx->pkt_lenm1 + 1; + head->nb_segs = hdr->w0.num_frags; + + return head; +} + static __rte_always_inline struct rte_mbuf * -nix_sec_meta_to_mbuf_sc(uint64_t cq_w1, const uint64_t sa_base, uintptr_t laddr, - uint8_t *loff, struct rte_mbuf *mbuf, uint16_t data_off) +nix_sec_meta_to_mbuf_sc(uint64_t cq_w1, uint64_t cq_w5, const uint64_t sa_base, + uintptr_t laddr, uint8_t *loff, struct rte_mbuf *mbuf, + uint16_t data_off, const uint16_t flags, + const uint64_t mbuf_init) { const void *__p = (void *)((uintptr_t)mbuf + (uint16_t)data_off); const struct cpt_parse_hdr_s *hdr = (const struct cpt_parse_hdr_s *)__p; struct cn10k_inb_priv_data *inb_priv; - struct rte_mbuf *inner; + struct rte_mbuf *inner = NULL; + uint64_t res_w1; uint32_t sa_idx; + uint16_t uc_cc; + uint32_t len; void *inb_sa; uint64_t w0; - if (cq_w1 & BIT(11)) { + if ((flags & NIX_RX_OFFLOAD_REASSEMBLY_F) && (cq_w1 & BIT(11))) { + /* Get SPI from CPT_PARSE_S's cookie(already swapped) */ + w0 = hdr->w0.u64; + sa_idx = w0 >> 32; + + inb_sa = roc_nix_inl_ot_ipsec_inb_sa(sa_base, sa_idx); + inb_priv = roc_nix_inl_ot_ipsec_inb_sa_sw_rsvd(inb_sa); + + if (!hdr->w0.num_frags) { + /* No Reassembly or inbound error */ + inner = (struct rte_mbuf *) + (rte_be_to_cpu_64(hdr->wqe_ptr) - + sizeof(struct rte_mbuf)); + + /* Update dynamic field with userdata */ + *rte_security_dynfield(inner) = + (uint64_t)inb_priv->userdata; + + /* CPT result(struct cpt_cn10k_res_s) is at + * after first IOVA in meta + */ + res_w1 = *((uint64_t *)(&inner[1]) + 10); + uc_cc = res_w1 & 0xFF; + + /* Calculate inner packet length */ + len = ((res_w1 >> 16) & 0xFFFF) + hdr->w2.il3_off - + sizeof(struct cpt_parse_hdr_s) - (w0 & 0x7); + inner->pkt_len = len; + inner->data_len = len; + *(uint64_t *)(&inner->rearm_data) = mbuf_init; + + inner->ol_flags = ((uc_cc == CPT_COMP_WARN) ? + RTE_MBUF_F_RX_SEC_OFFLOAD : + (RTE_MBUF_F_RX_SEC_OFFLOAD | + RTE_MBUF_F_RX_SEC_OFFLOAD_FAILED)); + inner->next = NULL; + } else if (!(hdr->w0.err_sum) && !(hdr->w0.reas_sts)) { + /* Reassembly success */ + inner = nix_sec_reassemble_frags(hdr, cq_w1, cq_w5, + mbuf_init); + + /* Update dynamic field with userdata */ + *rte_security_dynfield(inner) = + (uint64_t)inb_priv->userdata; + + /* Assume success */ + inner->ol_flags = RTE_MBUF_F_RX_SEC_OFFLOAD; + } else { + /* Reassembly failure */ + inner = nix_sec_attach_frags(hdr, inb_priv, mbuf_init); + } + + /* Store meta in lmtline to free + * Assume all meta's from same aura. + */ + *(uint64_t *)(laddr + (*loff << 3)) = (uint64_t)mbuf; + *loff = *loff + 1; + + return inner; + } else if (cq_w1 & BIT(11)) { inner = (struct rte_mbuf *)(rte_be_to_cpu_64(hdr->wqe_ptr) - sizeof(struct rte_mbuf)); @@ -131,8 +472,25 @@ nix_sec_meta_to_mbuf_sc(uint64_t cq_w1, const uint64_t sa_base, uintptr_t laddr, *rte_security_dynfield(inner) = (uint64_t)inb_priv->userdata; /* Update l2 hdr length first */ - inner->pkt_len = (hdr->w2.il3_off - - sizeof(struct cpt_parse_hdr_s) - (w0 & 0x7)); + + /* CPT result(struct cpt_cn10k_res_s) is at + * after first IOVA in meta + */ + res_w1 = *((uint64_t *)(&inner[1]) + 10); + uc_cc = res_w1 & 0xFF; + + /* Calculate inner packet length */ + len = ((res_w1 >> 16) & 0xFFFF) + hdr->w2.il3_off - + sizeof(struct cpt_parse_hdr_s) - (w0 & 0x7); + inner->pkt_len = len; + inner->data_len = len; + *(uint64_t *)(&inner->rearm_data) = mbuf_init; + + inner->ol_flags = ((uc_cc == CPT_COMP_WARN) ? + RTE_MBUF_F_RX_SEC_OFFLOAD : + (RTE_MBUF_F_RX_SEC_OFFLOAD | + RTE_MBUF_F_RX_SEC_OFFLOAD_FAILED)); + inner->next = NULL; /* Store meta in lmtline to free * Assume all meta's from same aura. @@ -148,18 +506,23 @@ nix_sec_meta_to_mbuf_sc(uint64_t cq_w1, const uint64_t sa_base, uintptr_t laddr, return inner; } + + mbuf->next = NULL; return mbuf; } #if defined(RTE_ARCH_ARM64) static __rte_always_inline struct rte_mbuf * -nix_sec_meta_to_mbuf(uint64_t cq_w1, uintptr_t sa_base, uintptr_t laddr, - uint8_t *loff, struct rte_mbuf *mbuf, uint16_t data_off, - uint8x16_t *rx_desc_field1, uint64_t *ol_flags) +nix_sec_meta_to_mbuf(uint64_t cq_w1, uint64_t cq_w5, uintptr_t sa_base, + uintptr_t laddr, uint8_t *loff, struct rte_mbuf *mbuf, + uint16_t data_off, uint8x16_t *rx_desc_field1, + uint64_t *ol_flags, const uint16_t flags, + uint64x2_t *rearm) { const void *__p = (void *)((uintptr_t)mbuf + (uint16_t)data_off); const struct cpt_parse_hdr_s *hdr = (const struct cpt_parse_hdr_s *)__p; + uint64_t mbuf_init = vgetq_lane_u64(*rearm, 0); struct cn10k_inb_priv_data *inb_priv; struct rte_mbuf *inner; uint64_t *sg, res_w1; @@ -168,7 +531,102 @@ nix_sec_meta_to_mbuf(uint64_t cq_w1, uintptr_t sa_base, uintptr_t laddr, uint16_t len; uint64_t w0; - if (cq_w1 & BIT(11)) { + if ((flags & NIX_RX_OFFLOAD_REASSEMBLY_F) && (cq_w1 & BIT(11))) { + w0 = hdr->w0.u64; + sa_idx = w0 >> 32; + + /* Get SPI from CPT_PARSE_S's cookie(already swapped) */ + w0 = hdr->w0.u64; + sa_idx = w0 >> 32; + + inb_sa = roc_nix_inl_ot_ipsec_inb_sa(sa_base, sa_idx); + inb_priv = roc_nix_inl_ot_ipsec_inb_sa_sw_rsvd(inb_sa); + + /* Clear checksum flags */ + *ol_flags &= ~(RTE_MBUF_F_RX_L4_CKSUM_MASK | + RTE_MBUF_F_RX_IP_CKSUM_MASK); + + if (!hdr->w0.num_frags) { + /* No Reassembly or inbound error */ + inner = (struct rte_mbuf *) + (rte_be_to_cpu_64(hdr->wqe_ptr) - + sizeof(struct rte_mbuf)); + /* Update dynamic field with userdata */ + *rte_security_dynfield(inner) = + (uint64_t)inb_priv->userdata; + + /* CPT result(struct cpt_cn10k_res_s) is at + * after first IOVA in meta + */ + sg = (uint64_t *)(inner + 1); + res_w1 = sg[10]; + + /* Clear checksum flags and update security flag + */ + *ol_flags &= ~(RTE_MBUF_F_RX_L4_CKSUM_MASK | + RTE_MBUF_F_RX_IP_CKSUM_MASK); + *ol_flags |= + (((res_w1 & 0xFF) == CPT_COMP_WARN) ? + RTE_MBUF_F_RX_SEC_OFFLOAD : + (RTE_MBUF_F_RX_SEC_OFFLOAD | + RTE_MBUF_F_RX_SEC_OFFLOAD_FAILED)); + /* Calculate inner packet length */ + len = ((res_w1 >> 16) & 0xFFFF) + + hdr->w2.il3_off - + sizeof(struct cpt_parse_hdr_s) - + (w0 & 0x7); + /* Update pkt_len and data_len */ + *rx_desc_field1 = + vsetq_lane_u16(len, *rx_desc_field1, 2); + *rx_desc_field1 = + vsetq_lane_u16(len, *rx_desc_field1, 4); + + inner->next = NULL; + } else if (!(hdr->w0.err_sum) && !(hdr->w0.reas_sts)) { + /* Reassembly success */ + inner = nix_sec_reassemble_frags(hdr, cq_w1, cq_w5, + mbuf_init); + sg = (uint64_t *)(inner + 1); + res_w1 = sg[10]; + + /* Update dynamic field with userdata */ + *rte_security_dynfield(inner) = + (uint64_t)inb_priv->userdata; + + /* Assume success */ + *ol_flags |= RTE_MBUF_F_RX_SEC_OFFLOAD; + + /* Update pkt_len and data_len */ + *rx_desc_field1 = vsetq_lane_u16(inner->pkt_len, + *rx_desc_field1, 2); + *rx_desc_field1 = vsetq_lane_u16(inner->data_len, + *rx_desc_field1, 4); + + /* Data offset might be updated */ + mbuf_init = *(uint64_t *)(&inner->rearm_data); + *rearm = vsetq_lane_u64(mbuf_init, *rearm, 0); + } else { + /* Reassembly failure */ + inner = nix_sec_attach_frags(hdr, inb_priv, mbuf_init); + *ol_flags |= inner->ol_flags; + + /* Update pkt_len and data_len */ + *rx_desc_field1 = vsetq_lane_u16(inner->pkt_len, + *rx_desc_field1, 2); + *rx_desc_field1 = vsetq_lane_u16(inner->data_len, + *rx_desc_field1, 4); + } + + /* Store meta in lmtline to free + * Assume all meta's from same aura. + */ + *(uint64_t *)(laddr + (*loff << 3)) = (uint64_t)mbuf; + *loff = *loff + 1; + + /* Return inner mbuf */ + return inner; + + } else if (cq_w1 & BIT(11)) { inner = (struct rte_mbuf *)(rte_be_to_cpu_64(hdr->wqe_ptr) - sizeof(struct rte_mbuf)); /* Get SPI from CPT_PARSE_S's cookie(already swapped) */ @@ -211,10 +669,12 @@ nix_sec_meta_to_mbuf(uint64_t cq_w1, uintptr_t sa_base, uintptr_t laddr, /* Mark inner mbuf as get */ RTE_MEMPOOL_CHECK_COOKIES(inner->pool, (void **)&inner, 1, 1); + inner->next = NULL; /* Return inner mbuf */ return inner; } + mbuf->next = NULL; /* Return same mbuf as it is not a decrypted pkt */ return mbuf; } @@ -283,7 +743,7 @@ nix_cqe_xtract_mseg(const union nix_rx_parse_u *rx, struct rte_mbuf *mbuf, sg = *(const uint64_t *)(rx + 1); nb_segs = (sg >> 48) & 0x3; - if (nb_segs == 1) { + if (nb_segs == 1 && !(flags & NIX_RX_SEC_REASSEMBLY_F)) { mbuf->next = NULL; return; } @@ -346,30 +806,10 @@ cn10k_nix_cqe_to_mbuf(const struct nix_cqe_hdr_s *cq, const uint32_t tag, ol_flags |= RTE_MBUF_F_RX_RSS_HASH; } - /* Process Security packets */ - if (flag & NIX_RX_OFFLOAD_SECURITY_F) { - if (w1 & BIT(11)) { - /* CPT result(struct cpt_cn10k_res_s) is at - * after first IOVA in meta - */ - const uint64_t *sg = (const uint64_t *)(mbuf + 1); - const uint64_t res_w1 = sg[10]; - const uint16_t uc_cc = res_w1 & 0xFF; - - /* Rlen */ - len = ((res_w1 >> 16) & 0xFFFF) + mbuf->pkt_len; - ol_flags |= ((uc_cc == CPT_COMP_WARN) ? - RTE_MBUF_F_RX_SEC_OFFLOAD : - (RTE_MBUF_F_RX_SEC_OFFLOAD | - RTE_MBUF_F_RX_SEC_OFFLOAD_FAILED)); - } else { - if (flag & NIX_RX_OFFLOAD_CHECKSUM_F) - ol_flags |= nix_rx_olflags_get(lookup_mem, w1); - } - } else { - if (flag & NIX_RX_OFFLOAD_CHECKSUM_F) - ol_flags |= nix_rx_olflags_get(lookup_mem, w1); - } + /* Skip rx ol flags extraction for Security packets */ + if ((!(flag & NIX_RX_SEC_REASSEMBLY_F) || !(w1 & BIT(11))) && + flag & NIX_RX_OFFLOAD_CHECKSUM_F) + ol_flags |= nix_rx_olflags_get(lookup_mem, w1); if (flag & NIX_RX_OFFLOAD_VLAN_STRIP_F) { if (rx->vtag0_gone) { @@ -385,14 +825,19 @@ cn10k_nix_cqe_to_mbuf(const struct nix_cqe_hdr_s *cq, const uint32_t tag, if (flag & NIX_RX_OFFLOAD_MARK_UPDATE_F) ol_flags = nix_update_match_id(rx->match_id, ol_flags, mbuf); - mbuf->ol_flags = ol_flags; - mbuf->pkt_len = len; - mbuf->data_len = len; - *(uint64_t *)(&mbuf->rearm_data) = val; + /* Packet data length and ol flags is already updated for sec */ + if (flag & NIX_RX_SEC_REASSEMBLY_F && w1 & BIT_ULL(11)) { + mbuf->ol_flags |= ol_flags; + } else { + mbuf->ol_flags = ol_flags; + mbuf->pkt_len = len; + mbuf->data_len = len; + *(uint64_t *)(&mbuf->rearm_data) = val; + } if (flag & NIX_RX_MULTI_SEG_F) nix_cqe_xtract_mseg(rx, mbuf, val, flag); - else + else if (!(flag & NIX_RX_SEC_REASSEMBLY_F)) mbuf->next = NULL; } @@ -473,9 +918,11 @@ cn10k_nix_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts, /* Translate meta to mbuf */ if (flags & NIX_RX_OFFLOAD_SECURITY_F) { const uint64_t cq_w1 = *((const uint64_t *)cq + 1); + const uint64_t cq_w5 = *((const uint64_t *)cq + 5); - mbuf = nix_sec_meta_to_mbuf_sc(cq_w1, sa_base, laddr, - &loff, mbuf, data_off); + mbuf = nix_sec_meta_to_mbuf_sc(cq_w1, cq_w5, sa_base, laddr, + &loff, mbuf, data_off, + flags, mbuf_init); } cn10k_nix_cqe_to_mbuf(cq, cq->tag, mbuf, lookup_mem, mbuf_init, @@ -747,25 +1194,40 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, /* Translate meta to mbuf */ if (flags & NIX_RX_OFFLOAD_SECURITY_F) { + uint64_t cq0_w5 = *(uint64_t *)(cq0 + CQE_SZ(0) + 40); + uint64_t cq1_w5 = *(uint64_t *)(cq0 + CQE_SZ(1) + 40); + uint64_t cq2_w5 = *(uint64_t *)(cq0 + CQE_SZ(2) + 40); + uint64_t cq3_w5 = *(uint64_t *)(cq0 + CQE_SZ(3) + 40); + + /* Initialize rearm data when reassembly is enabled as + * data offset might change. + */ + if (flags & NIX_RX_OFFLOAD_REASSEMBLY_F) { + rearm0 = vdupq_n_u64(mbuf_initializer); + rearm1 = vdupq_n_u64(mbuf_initializer); + rearm2 = vdupq_n_u64(mbuf_initializer); + rearm3 = vdupq_n_u64(mbuf_initializer); + } + /* Checksum ol_flags will be cleared if mbuf is meta */ - mbuf0 = nix_sec_meta_to_mbuf(cq0_w1, sa_base, laddr, + mbuf0 = nix_sec_meta_to_mbuf(cq0_w1, cq0_w5, sa_base, laddr, &loff, mbuf0, d_off, &f0, - &ol_flags0); + &ol_flags0, flags, &rearm0); mbuf01 = vsetq_lane_u64((uint64_t)mbuf0, mbuf01, 0); - mbuf1 = nix_sec_meta_to_mbuf(cq1_w1, sa_base, laddr, + mbuf1 = nix_sec_meta_to_mbuf(cq1_w1, cq1_w5, sa_base, laddr, &loff, mbuf1, d_off, &f1, - &ol_flags1); + &ol_flags1, flags, &rearm1); mbuf01 = vsetq_lane_u64((uint64_t)mbuf1, mbuf01, 1); - mbuf2 = nix_sec_meta_to_mbuf(cq2_w1, sa_base, laddr, + mbuf2 = nix_sec_meta_to_mbuf(cq2_w1, cq2_w5, sa_base, laddr, &loff, mbuf2, d_off, &f2, - &ol_flags2); + &ol_flags2, flags, &rearm2); mbuf23 = vsetq_lane_u64((uint64_t)mbuf2, mbuf23, 0); - mbuf3 = nix_sec_meta_to_mbuf(cq3_w1, sa_base, laddr, + mbuf3 = nix_sec_meta_to_mbuf(cq3_w1, cq3_w5, sa_base, laddr, &loff, mbuf3, d_off, &f3, - &ol_flags3); + &ol_flags3, flags, &rearm3); mbuf23 = vsetq_lane_u64((uint64_t)mbuf3, mbuf23, 1); } @@ -927,7 +1389,7 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, nix_cqe_xtract_mseg((union nix_rx_parse_u *) (CQE_PTR_OFF(cq0, 3, 8, flags)), mbuf3, mbuf_initializer, flags); - } else { + } else if (!(flags & NIX_RX_SEC_REASSEMBLY_F)) { /* Update that no more segments */ mbuf0->next = NULL; mbuf1->next = NULL; @@ -1060,6 +1522,7 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, #define TS_F NIX_RX_OFFLOAD_TSTAMP_F #define RX_VLAN_F NIX_RX_OFFLOAD_VLAN_STRIP_F #define R_SEC_F NIX_RX_OFFLOAD_SECURITY_F +#define R_REAS_F NIX_RX_OFFLOAD_REASSEMBLY_F /* [R_SEC_F] [RX_VLAN_F] [TS] [MARK] [CKSUM] [PTYPE] [RSS] */ #define NIX_RX_FASTPATH_MODES_0_15 \ @@ -1226,6 +1689,101 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, R(sec_vlan_ts_mark_cksum_ptype_rss, \ R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) +/* R_REAS_F cannot work without R_SEC_F */ +#define NIX_RX_FASTPATH_MODES_128_143 +#define NIX_RX_FASTPATH_MODES_144_159 +#define NIX_RX_FASTPATH_MODES_160_175 +#define NIX_RX_FASTPATH_MODES_176_191 + +#define NIX_RX_FASTPATH_MODES_192_207 \ + R(reas_sec, R_REAS_F | R_SEC_F) \ + R(reas_sec_rss, R_REAS_F | R_SEC_F | RSS_F) \ + R(reas_sec_ptype, R_REAS_F | R_SEC_F | PTYPE_F) \ + R(reas_sec_ptype_rss, R_REAS_F | R_SEC_F | PTYPE_F | RSS_F) \ + R(reas_sec_cksum, R_REAS_F | R_SEC_F | CKSUM_F) \ + R(reas_sec_cksum_rss, R_REAS_F | R_SEC_F | CKSUM_F | RSS_F) \ + R(reas_sec_cksum_ptype, R_REAS_F | R_SEC_F | CKSUM_F | PTYPE_F) \ + R(reas_sec_cksum_ptype_rss, R_REAS_F | R_SEC_F | CKSUM_F | PTYPE_F | RSS_F) \ + R(reas_sec_mark, R_REAS_F | R_SEC_F | MARK_F) \ + R(reas_sec_mark_rss, R_REAS_F | R_SEC_F | MARK_F | RSS_F) \ + R(reas_sec_mark_ptype, R_REAS_F | R_SEC_F | MARK_F | PTYPE_F) \ + R(reas_sec_mark_ptype_rss, R_REAS_F | R_SEC_F | MARK_F | PTYPE_F | RSS_F) \ + R(reas_sec_mark_cksum, R_REAS_F | R_SEC_F | MARK_F | CKSUM_F) \ + R(reas_sec_mark_cksum_rss, R_REAS_F | R_SEC_F | MARK_F | CKSUM_F | RSS_F) \ + R(reas_sec_mark_cksum_ptype, R_REAS_F | R_SEC_F | MARK_F | CKSUM_F | PTYPE_F) \ + R(reas_sec_mark_cksum_ptype_rss, \ + R_REAS_F | R_SEC_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) + +#define NIX_RX_FASTPATH_MODES_208_223 \ + R(reas_sec_ts, R_REAS_F | R_SEC_F | TS_F) \ + R(reas_sec_ts_rss, R_REAS_F | R_SEC_F | TS_F | RSS_F) \ + R(reas_sec_ts_ptype, R_REAS_F | R_SEC_F | TS_F | PTYPE_F) \ + R(reas_sec_ts_ptype_rss, R_REAS_F | R_SEC_F | TS_F | PTYPE_F | RSS_F) \ + R(reas_sec_ts_cksum, R_REAS_F | R_SEC_F | TS_F | CKSUM_F) \ + R(reas_sec_ts_cksum_rss, R_REAS_F | R_SEC_F | TS_F | CKSUM_F | RSS_F) \ + R(reas_sec_ts_cksum_ptype, R_REAS_F | R_SEC_F | TS_F | CKSUM_F | PTYPE_F) \ + R(reas_sec_ts_cksum_ptype_rss, R_REAS_F | R_SEC_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ + R(reas_sec_ts_mark, R_REAS_F | R_SEC_F | TS_F | MARK_F) \ + R(reas_sec_ts_mark_rss, R_REAS_F | R_SEC_F | TS_F | MARK_F | RSS_F) \ + R(reas_sec_ts_mark_ptype, R_REAS_F | R_SEC_F | TS_F | MARK_F | PTYPE_F) \ + R(reas_sec_ts_mark_ptype_rss, R_REAS_F | R_SEC_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ + R(reas_sec_ts_mark_cksum, R_REAS_F | R_SEC_F | TS_F | MARK_F | CKSUM_F) \ + R(reas_sec_ts_mark_cksum_rss, R_REAS_F | R_SEC_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ + R(reas_sec_ts_mark_cksum_ptype, \ + R_REAS_F | R_SEC_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ + R(reas_sec_ts_mark_cksum_ptype_rss, \ + R_REAS_F | R_SEC_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) + +#define NIX_RX_FASTPATH_MODES_224_239 \ + R(reas_sec_vlan, R_REAS_F | R_SEC_F | RX_VLAN_F) \ + R(reas_sec_vlan_rss, R_REAS_F | R_SEC_F | RX_VLAN_F | RSS_F) \ + R(reas_sec_vlan_ptype, R_REAS_F | R_SEC_F | RX_VLAN_F | PTYPE_F) \ + R(reas_sec_vlan_ptype_rss, R_REAS_F | R_SEC_F | RX_VLAN_F | PTYPE_F | RSS_F) \ + R(reas_sec_vlan_cksum, R_REAS_F | R_SEC_F | RX_VLAN_F | CKSUM_F) \ + R(reas_sec_vlan_cksum_rss, R_REAS_F | R_SEC_F | RX_VLAN_F | CKSUM_F | RSS_F) \ + R(reas_sec_vlan_cksum_ptype, R_REAS_F | R_SEC_F | RX_VLAN_F | CKSUM_F | PTYPE_F) \ + R(reas_sec_vlan_cksum_ptype_rss, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | CKSUM_F | PTYPE_F | RSS_F) \ + R(reas_sec_vlan_mark, R_REAS_F | R_SEC_F | RX_VLAN_F | MARK_F) \ + R(reas_sec_vlan_mark_rss, R_REAS_F | R_SEC_F | RX_VLAN_F | MARK_F | RSS_F) \ + R(reas_sec_vlan_mark_ptype, R_REAS_F | R_SEC_F | RX_VLAN_F | MARK_F | PTYPE_F) \ + R(reas_sec_vlan_mark_ptype_rss, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | MARK_F | PTYPE_F | RSS_F) \ + R(reas_sec_vlan_mark_cksum, R_REAS_F | R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F) \ + R(reas_sec_vlan_mark_cksum_rss, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | RSS_F) \ + R(reas_sec_vlan_mark_cksum_ptype, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F) \ + R(reas_sec_vlan_mark_cksum_ptype_rss, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) + +#define NIX_RX_FASTPATH_MODES_240_255 \ + R(reas_sec_vlan_ts, R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F) \ + R(reas_sec_vlan_ts_rss, R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | RSS_F) \ + R(reas_sec_vlan_ts_ptype, R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | PTYPE_F) \ + R(reas_sec_vlan_ts_ptype_rss, R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | PTYPE_F | RSS_F) \ + R(reas_sec_vlan_ts_cksum, R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F) \ + R(reas_sec_vlan_ts_cksum_rss, R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | RSS_F) \ + R(reas_sec_vlan_ts_cksum_ptype, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F) \ + R(reas_sec_vlan_ts_cksum_ptype_rss, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ + R(reas_sec_vlan_ts_mark, R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | MARK_F) \ + R(reas_sec_vlan_ts_mark_rss, R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | MARK_F | RSS_F) \ + R(reas_sec_vlan_ts_mark_ptype, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | MARK_F | PTYPE_F) \ + R(reas_sec_vlan_ts_mark_ptype_rss, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ + R(reas_sec_vlan_ts_mark_cksum, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F) \ + R(reas_sec_vlan_ts_mark_cksum_rss, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ + R(reas_sec_vlan_ts_mark_cksum_ptype, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ + R(reas_sec_vlan_ts_mark_cksum_ptype_rss, \ + R_REAS_F | R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) + + #define NIX_RX_FASTPATH_MODES \ NIX_RX_FASTPATH_MODES_0_15 \ NIX_RX_FASTPATH_MODES_16_31 \ @@ -1234,7 +1792,15 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, NIX_RX_FASTPATH_MODES_64_79 \ NIX_RX_FASTPATH_MODES_80_95 \ NIX_RX_FASTPATH_MODES_96_111 \ - NIX_RX_FASTPATH_MODES_112_127 + NIX_RX_FASTPATH_MODES_112_127 \ + NIX_RX_FASTPATH_MODES_128_143 \ + NIX_RX_FASTPATH_MODES_144_159 \ + NIX_RX_FASTPATH_MODES_160_175 \ + NIX_RX_FASTPATH_MODES_176_191 \ + NIX_RX_FASTPATH_MODES_192_207 \ + NIX_RX_FASTPATH_MODES_208_223 \ + NIX_RX_FASTPATH_MODES_224_239 \ + NIX_RX_FASTPATH_MODES_240_255 #define R(name, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_##name( \ diff --git a/drivers/net/cnxk/cnxk_ethdev.h b/drivers/net/cnxk/cnxk_ethdev.h index f1363af70f..75f33ff36d 100644 --- a/drivers/net/cnxk/cnxk_ethdev.h +++ b/drivers/net/cnxk/cnxk_ethdev.h @@ -405,6 +405,10 @@ struct cnxk_eth_dev { /* Security data */ struct cnxk_eth_dev_sec_inb inb; struct cnxk_eth_dev_sec_outb outb; + + /* Reassembly dynfield/flag offsets */ + int reass_dynfield_off; + int reass_dynflag_bit; }; struct cnxk_eth_rxq_sp { diff --git a/drivers/net/cnxk/meson.build b/drivers/net/cnxk/meson.build index cd8c13bd1c..16db47c519 100644 --- a/drivers/net/cnxk/meson.build +++ b/drivers/net/cnxk/meson.build @@ -120,6 +120,10 @@ sources += files( 'rx/cn10k/rx_80_95.c', 'rx/cn10k/rx_96_111.c', 'rx/cn10k/rx_112_127.c', + 'rx/cn10k/rx_192_207.c', + 'rx/cn10k/rx_208_223.c', + 'rx/cn10k/rx_224_239.c', + 'rx/cn10k/rx_240_255.c', 'rx/cn10k/rx_0_15_mseg.c', 'rx/cn10k/rx_16_31_mseg.c', 'rx/cn10k/rx_32_47_mseg.c', @@ -128,6 +132,10 @@ sources += files( 'rx/cn10k/rx_80_95_mseg.c', 'rx/cn10k/rx_96_111_mseg.c', 'rx/cn10k/rx_112_127_mseg.c', + 'rx/cn10k/rx_192_207_mseg.c', + 'rx/cn10k/rx_208_223_mseg.c', + 'rx/cn10k/rx_224_239_mseg.c', + 'rx/cn10k/rx_240_255_mseg.c', 'rx/cn10k/rx_0_15_vec.c', 'rx/cn10k/rx_16_31_vec.c', 'rx/cn10k/rx_32_47_vec.c', @@ -136,6 +144,10 @@ sources += files( 'rx/cn10k/rx_80_95_vec.c', 'rx/cn10k/rx_96_111_vec.c', 'rx/cn10k/rx_112_127_vec.c', + 'rx/cn10k/rx_192_207_vec.c', + 'rx/cn10k/rx_208_223_vec.c', + 'rx/cn10k/rx_224_239_vec.c', + 'rx/cn10k/rx_240_255_vec.c', 'rx/cn10k/rx_0_15_vec_mseg.c', 'rx/cn10k/rx_16_31_vec_mseg.c', 'rx/cn10k/rx_32_47_vec_mseg.c', @@ -144,6 +156,10 @@ sources += files( 'rx/cn10k/rx_80_95_vec_mseg.c', 'rx/cn10k/rx_96_111_vec_mseg.c', 'rx/cn10k/rx_112_127_vec_mseg.c', + 'rx/cn10k/rx_192_207_vec_mseg.c', + 'rx/cn10k/rx_208_223_vec_mseg.c', + 'rx/cn10k/rx_224_239_vec_mseg.c', + 'rx/cn10k/rx_240_255_vec_mseg.c', ) sources += files( diff --git a/drivers/net/cnxk/rx/cn10k/rx_192_207.c b/drivers/net/cnxk/rx/cn10k/rx_192_207.c new file mode 100644 index 0000000000..45da8e9997 --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_192_207.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) NIX_RX_RECV(cn10k_nix_recv_pkts_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_192_207_mseg.c b/drivers/net/cnxk/rx/cn10k/rx_192_207_mseg.c new file mode 100644 index 0000000000..869be355f9 --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_192_207_mseg.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) NIX_RX_RECV_MSEG(cn10k_nix_recv_pkts_mseg_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_192_207_vec.c b/drivers/net/cnxk/rx/cn10k/rx_192_207_vec.c new file mode 100644 index 0000000000..3d214158ec --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_192_207_vec.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) NIX_RX_RECV_VEC(cn10k_nix_recv_pkts_vec_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_192_207_vec_mseg.c b/drivers/net/cnxk/rx/cn10k/rx_192_207_vec_mseg.c new file mode 100644 index 0000000000..762e9f5512 --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_192_207_vec_mseg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) \ + NIX_RX_RECV_VEC_MSEG(cn10k_nix_recv_pkts_vec_mseg_##name, flags) + +NIX_RX_FASTPATH_MODES_192_207 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_208_223.c b/drivers/net/cnxk/rx/cn10k/rx_208_223.c new file mode 100644 index 0000000000..ca64ab5fde --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_208_223.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) NIX_RX_RECV(cn10k_nix_recv_pkts_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_208_223_mseg.c b/drivers/net/cnxk/rx/cn10k/rx_208_223_mseg.c new file mode 100644 index 0000000000..35e443a06e --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_208_223_mseg.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) NIX_RX_RECV_MSEG(cn10k_nix_recv_pkts_mseg_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_208_223_vec.c b/drivers/net/cnxk/rx/cn10k/rx_208_223_vec.c new file mode 100644 index 0000000000..b7f9feece2 --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_208_223_vec.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) NIX_RX_RECV_VEC(cn10k_nix_recv_pkts_vec_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_208_223_vec_mseg.c b/drivers/net/cnxk/rx/cn10k/rx_208_223_vec_mseg.c new file mode 100644 index 0000000000..b3da6ec11e --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_208_223_vec_mseg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) \ + NIX_RX_RECV_VEC_MSEG(cn10k_nix_recv_pkts_vec_mseg_##name, flags) + +NIX_RX_FASTPATH_MODES_208_223 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_224_239.c b/drivers/net/cnxk/rx/cn10k/rx_224_239.c new file mode 100644 index 0000000000..bd288f3330 --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_224_239.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) NIX_RX_RECV(cn10k_nix_recv_pkts_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_224_239_mseg.c b/drivers/net/cnxk/rx/cn10k/rx_224_239_mseg.c new file mode 100644 index 0000000000..bf5af065df --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_224_239_mseg.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) NIX_RX_RECV_MSEG(cn10k_nix_recv_pkts_mseg_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_224_239_vec.c b/drivers/net/cnxk/rx/cn10k/rx_224_239_vec.c new file mode 100644 index 0000000000..c05bb99414 --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_224_239_vec.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) NIX_RX_RECV_VEC(cn10k_nix_recv_pkts_vec_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_224_239_vec_mseg.c b/drivers/net/cnxk/rx/cn10k/rx_224_239_vec_mseg.c new file mode 100644 index 0000000000..5f7f8efdae --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_224_239_vec_mseg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) \ + NIX_RX_RECV_VEC_MSEG(cn10k_nix_recv_pkts_vec_mseg_##name, flags) + +NIX_RX_FASTPATH_MODES_224_239 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_240_255.c b/drivers/net/cnxk/rx/cn10k/rx_240_255.c new file mode 100644 index 0000000000..d72b2eec1c --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_240_255.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) NIX_RX_RECV(cn10k_nix_recv_pkts_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_240_255_mseg.c b/drivers/net/cnxk/rx/cn10k/rx_240_255_mseg.c new file mode 100644 index 0000000000..f248ad8c77 --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_240_255_mseg.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) NIX_RX_RECV_MSEG(cn10k_nix_recv_pkts_mseg_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_240_255_vec.c b/drivers/net/cnxk/rx/cn10k/rx_240_255_vec.c new file mode 100644 index 0000000000..7e81ed1883 --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_240_255_vec.c @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) NIX_RX_RECV_VEC(cn10k_nix_recv_pkts_vec_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R diff --git a/drivers/net/cnxk/rx/cn10k/rx_240_255_vec_mseg.c b/drivers/net/cnxk/rx/cn10k/rx_240_255_vec_mseg.c new file mode 100644 index 0000000000..db8aeca013 --- /dev/null +++ b/drivers/net/cnxk/rx/cn10k/rx_240_255_vec_mseg.c @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +#define R(name, flags) \ + NIX_RX_RECV_VEC_MSEG(cn10k_nix_recv_pkts_vec_mseg_##name, flags) + +NIX_RX_FASTPATH_MODES_240_255 +#undef R