From patchwork Mon Dec 13 08:22:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavan Nikhilesh Bhagavatula X-Patchwork-Id: 105099 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 56B0EA00BE; Mon, 13 Dec 2021 09:23:30 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F1C624120A; Mon, 13 Dec 2021 09:23:01 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 93D68411FB for ; Mon, 13 Dec 2021 09:23:00 +0100 (CET) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 1BD7i8N6026809; Mon, 13 Dec 2021 00:23:00 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=h3NgQ5mhsPc7cKTHcnelmzyH1Losf89o8J8bCnvSf1I=; b=FV8RCSbhDL6kINRTarKwSi2p+G/lgo7PS0OIs6foBWwtnB4rIO/Jsr8J8KWgzYYlda3t EH/k4M7q8u2UZ/4E3ASt4bfVdUCffsLohDRvKudVFZ7qt8yy9YISxHJ2kJysCmW2ul6f Sr4Z1ktq7LK8ZjW3ox7xxYZLtNKVZtvIXOrdcnCzpRuL6BIgO3UKcPNi77m+p9v1rfdb ZY3tyBS38hzOgnXMOEAiXCuHgJs1GIzz3dsxx+kfiJT5U49hCjeFzLuRowWW7WN0OPNH HJl/1ikEfNQXww08La+MGclRn3Z0I+xT8VoGcFqGXLN1DTcGBdaJ2Vk3A4pVMzft9RU8 1w== Received: from dc5-exch02.marvell.com ([199.233.59.182]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3cx21kg3t3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Mon, 13 Dec 2021 00:22:59 -0800 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Mon, 13 Dec 2021 00:22:58 -0800 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Mon, 13 Dec 2021 00:22:58 -0800 Received: from BG-LT7430.marvell.com (BG-LT7430.marvell.com [10.28.177.176]) by maili.marvell.com (Postfix) with ESMTP id 7FDD93F7045; Mon, 13 Dec 2021 00:22:54 -0800 (PST) From: To: , , , Pavan Nikhilesh , Shijith Thotton , Nithin Dabilpuram , "Kiran Kumar K" , Sunil Kumar Kori , Satha Rao , Anatoly Burakov CC: Subject: [PATCH 6/8] net/cnxk: add CN10K template Rx functions to build Date: Mon, 13 Dec 2021 13:52:23 +0530 Message-ID: <20211213082226.3646-6-pbhagavatula@marvell.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20211213082226.3646-1-pbhagavatula@marvell.com> References: <20211213082226.3646-1-pbhagavatula@marvell.com> MIME-Version: 1.0 X-Proofpoint-GUID: 52eABvhHHcdOlgB1F3657WJPVr-cELnH X-Proofpoint-ORIG-GUID: 52eABvhHHcdOlgB1F3657WJPVr-cELnH X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.790,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2021-12-13_03,2021-12-10_01,2021-12-02_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Pavan Nikhilesh Add CN10K segregated Rx and event dequeue functions to build, add macros to make future modifications simpler. Signed-off-by: Pavan Nikhilesh --- drivers/event/cnxk/cn10k_eventdev.c | 114 +++-- drivers/event/cnxk/cn10k_worker.h | 71 ++- drivers/event/cnxk/cn10k_worker_deq.c | 44 -- drivers/event/cnxk/cn10k_worker_deq_burst.c | 29 -- drivers/event/cnxk/cn10k_worker_deq_ca.c | 65 --- drivers/event/cnxk/cn10k_worker_deq_tmo.c | 72 --- drivers/event/cnxk/meson.build | 135 +++++- drivers/net/cnxk/cn10k_rx.h | 461 +++++++++----------- drivers/net/cnxk/cn10k_rx_mseg.c | 17 - drivers/net/cnxk/cn10k_rx_select.c | 67 +++ drivers/net/cnxk/cn10k_rx_vec.c | 19 - drivers/net/cnxk/cn10k_rx_vec_mseg.c | 18 - drivers/net/cnxk/meson.build | 40 +- 13 files changed, 570 insertions(+), 582 deletions(-) delete mode 100644 drivers/event/cnxk/cn10k_worker_deq.c delete mode 100644 drivers/event/cnxk/cn10k_worker_deq_burst.c delete mode 100644 drivers/event/cnxk/cn10k_worker_deq_ca.c delete mode 100644 drivers/event/cnxk/cn10k_worker_deq_tmo.c delete mode 100644 drivers/net/cnxk/cn10k_rx_mseg.c create mode 100644 drivers/net/cnxk/cn10k_rx_select.c delete mode 100644 drivers/net/cnxk/cn10k_rx_vec.c delete mode 100644 drivers/net/cnxk/cn10k_rx_vec_mseg.c diff --git a/drivers/event/cnxk/cn10k_eventdev.c b/drivers/event/cnxk/cn10k_eventdev.c index c5a8c1ae8f..02f3d8235d 100644 --- a/drivers/event/cnxk/cn10k_eventdev.c +++ b/drivers/event/cnxk/cn10k_eventdev.c @@ -7,13 +7,7 @@ #include "cnxk_worker.h" #define CN10K_SET_EVDEV_DEQ_OP(dev, deq_op, deq_ops) \ - (deq_op = deq_ops[!!(dev->rx_offloads & NIX_RX_OFFLOAD_SECURITY_F)] \ - [!!(dev->rx_offloads & NIX_RX_OFFLOAD_VLAN_STRIP_F)] \ - [!!(dev->rx_offloads & NIX_RX_OFFLOAD_TSTAMP_F)] \ - [!!(dev->rx_offloads & NIX_RX_OFFLOAD_MARK_UPDATE_F)] \ - [!!(dev->rx_offloads & NIX_RX_OFFLOAD_CHECKSUM_F)] \ - [!!(dev->rx_offloads & NIX_RX_OFFLOAD_PTYPE_F)] \ - [!!(dev->rx_offloads & NIX_RX_OFFLOAD_RSS_F)]) + deq_op = deq_ops[dev->rx_offloads & (NIX_RX_OFFLOAD_MAX - 1)] #define CN10K_SET_EVDEV_ENQ_OP(dev, enq_op, enq_ops) \ (enq_op = \ @@ -291,95 +285,109 @@ static void cn10k_sso_fp_fns_set(struct rte_eventdev *event_dev) { struct cnxk_sso_evdev *dev = cnxk_sso_pmd_priv(event_dev); - const event_dequeue_t sso_hws_deq[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_##name, + const event_dequeue_t sso_hws_deq[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_burst_t sso_hws_deq_burst[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_burst_##name, + const event_dequeue_burst_t sso_hws_deq_burst[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_tmo[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_tmo_##name, + const event_dequeue_t sso_hws_deq_tmo[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_tmo_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_burst_t - sso_hws_deq_tmo_burst[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_tmo_burst_##name, - NIX_RX_FASTPATH_MODES + const event_dequeue_burst_t sso_hws_deq_tmo_burst[NIX_RX_OFFLOAD_MAX] = + { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_tmo_burst_##name, + NIX_RX_FASTPATH_MODES #undef R - }; + }; - const event_dequeue_t sso_hws_deq_ca[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_ca_##name, + const event_dequeue_t sso_hws_deq_ca[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_ca_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_burst_t - sso_hws_deq_ca_burst[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_ca_burst_##name, + const event_dequeue_burst_t sso_hws_deq_ca_burst[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_ca_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_seg[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_seg_##name, + const event_dequeue_t sso_hws_deq_tmo_ca[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_tmo_ca_##name, NIX_RX_FASTPATH_MODES #undef R }; const event_dequeue_burst_t - sso_hws_deq_seg_burst[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_seg_burst_##name, + sso_hws_deq_tmo_ca_burst[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_tmo_ca_burst_##name, + NIX_RX_FASTPATH_MODES +#undef R + }; + + const event_dequeue_t sso_hws_deq_seg[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_seg_##name, + NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_tmo_seg[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_tmo_seg_##name, + const event_dequeue_burst_t sso_hws_deq_seg_burst[NIX_RX_OFFLOAD_MAX] = + { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_seg_burst_##name, + NIX_RX_FASTPATH_MODES +#undef R + }; + + const event_dequeue_t sso_hws_deq_tmo_seg[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_tmo_seg_##name, NIX_RX_FASTPATH_MODES #undef R }; const event_dequeue_burst_t - sso_hws_deq_tmo_seg_burst[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_tmo_seg_burst_##name, + sso_hws_deq_tmo_seg_burst[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_tmo_seg_burst_##name, NIX_RX_FASTPATH_MODES #undef R }; - const event_dequeue_t sso_hws_deq_ca_seg[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_ca_seg_##name, + const event_dequeue_t sso_hws_deq_ca_seg[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_ca_seg_##name, NIX_RX_FASTPATH_MODES #undef R }; const event_dequeue_burst_t - sso_hws_deq_ca_seg_burst[2][2][2][2][2][2][2] = { -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - [f6][f5][f4][f3][f2][f1][f0] = cn10k_sso_hws_deq_ca_seg_burst_##name, + sso_hws_deq_ca_seg_burst[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_ca_seg_burst_##name, NIX_RX_FASTPATH_MODES +#undef R + }; + + const event_dequeue_t sso_hws_deq_tmo_ca_seg[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_tmo_ca_seg_##name, + NIX_RX_FASTPATH_MODES #undef R }; + const event_dequeue_burst_t + sso_hws_deq_tmo_ca_seg_burst[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_sso_hws_deq_tmo_ca_seg_burst_##name, + NIX_RX_FASTPATH_MODES +#undef R + }; + /* Tx modes */ const event_tx_adapter_enqueue_t sso_hws_tx_adptr_enq[2][2][2][2][2][2][2] = { @@ -418,6 +426,12 @@ cn10k_sso_fp_fns_set(struct rte_eventdev *event_dev) CN10K_SET_EVDEV_DEQ_OP(dev, event_dev->dequeue_burst, sso_hws_deq_ca_seg_burst); } + if (dev->is_timeout_deq && dev->is_ca_internal_port) { + CN10K_SET_EVDEV_DEQ_OP(dev, event_dev->dequeue, + sso_hws_deq_tmo_ca_seg); + CN10K_SET_EVDEV_DEQ_OP(dev, event_dev->dequeue_burst, + sso_hws_deq_tmo_ca_seg_burst); + } } else { CN10K_SET_EVDEV_DEQ_OP(dev, event_dev->dequeue, sso_hws_deq); CN10K_SET_EVDEV_DEQ_OP(dev, event_dev->dequeue_burst, @@ -434,6 +448,12 @@ cn10k_sso_fp_fns_set(struct rte_eventdev *event_dev) CN10K_SET_EVDEV_DEQ_OP(dev, event_dev->dequeue_burst, sso_hws_deq_ca_burst); } + if (dev->is_timeout_deq && dev->is_ca_internal_port) { + CN10K_SET_EVDEV_DEQ_OP(dev, event_dev->dequeue, + sso_hws_deq_tmo_ca); + CN10K_SET_EVDEV_DEQ_OP(dev, event_dev->dequeue_burst, + sso_hws_deq_tmo_ca_burst); + } } event_dev->ca_enqueue = cn10k_sso_hws_ca_enq; diff --git a/drivers/event/cnxk/cn10k_worker.h b/drivers/event/cnxk/cn10k_worker.h index f8331e88d7..160b90aa27 100644 --- a/drivers/event/cnxk/cn10k_worker.h +++ b/drivers/event/cnxk/cn10k_worker.h @@ -352,7 +352,7 @@ uint16_t __rte_hot cn10k_sso_hws_enq_fwd_burst(void *port, uint16_t __rte_hot cn10k_sso_hws_ca_enq(void *port, struct rte_event ev[], uint16_t nb_events); -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ +#define R(name, flags) \ uint16_t __rte_hot cn10k_sso_hws_deq_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks); \ uint16_t __rte_hot cn10k_sso_hws_deq_burst_##name( \ @@ -368,6 +368,11 @@ uint16_t __rte_hot cn10k_sso_hws_ca_enq(void *port, struct rte_event ev[], uint16_t __rte_hot cn10k_sso_hws_deq_ca_burst_##name( \ void *port, struct rte_event ev[], uint16_t nb_events, \ uint64_t timeout_ticks); \ + uint16_t __rte_hot cn10k_sso_hws_deq_tmo_ca_##name( \ + void *port, struct rte_event *ev, uint64_t timeout_ticks); \ + uint16_t __rte_hot cn10k_sso_hws_deq_tmo_ca_burst_##name( \ + void *port, struct rte_event ev[], uint16_t nb_events, \ + uint64_t timeout_ticks); \ uint16_t __rte_hot cn10k_sso_hws_deq_seg_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks); \ uint16_t __rte_hot cn10k_sso_hws_deq_seg_burst_##name( \ @@ -381,12 +386,76 @@ uint16_t __rte_hot cn10k_sso_hws_ca_enq(void *port, struct rte_event ev[], uint16_t __rte_hot cn10k_sso_hws_deq_ca_seg_##name( \ void *port, struct rte_event *ev, uint64_t timeout_ticks); \ uint16_t __rte_hot cn10k_sso_hws_deq_ca_seg_burst_##name( \ + void *port, struct rte_event ev[], uint16_t nb_events, \ + uint64_t timeout_ticks); \ + uint16_t __rte_hot cn10k_sso_hws_deq_tmo_ca_seg_##name( \ + void *port, struct rte_event *ev, uint64_t timeout_ticks); \ + uint16_t __rte_hot cn10k_sso_hws_deq_tmo_ca_seg_burst_##name( \ void *port, struct rte_event ev[], uint16_t nb_events, \ uint64_t timeout_ticks); NIX_RX_FASTPATH_MODES #undef R +#define SSO_DEQ(fn, flags) \ + uint16_t __rte_hot fn(void *port, struct rte_event *ev, \ + uint64_t timeout_ticks) \ + { \ + struct cn10k_sso_hws *ws = port; \ + \ + RTE_SET_USED(timeout_ticks); \ + if (ws->swtag_req) { \ + ws->swtag_req = 0; \ + cnxk_sso_hws_swtag_wait(ws->base + SSOW_LF_GWS_WQE0); \ + return 1; \ + } \ + return cn10k_sso_hws_get_work(ws, ev, flags, ws->lookup_mem); \ + } + +#define SSO_DEQ_SEG(fn, flags) SSO_DEQ(fn, flags | NIX_RX_MULTI_SEG_F) +#define SSO_DEQ_CA(fn, flags) SSO_DEQ(fn, flags | CPT_RX_WQE_F) +#define SSO_DEQ_CA_SEG(fn, flags) SSO_DEQ_SEG(fn, flags | CPT_RX_WQE_F) + +#define SSO_DEQ_TMO(fn, flags) \ + uint16_t __rte_hot fn(void *port, struct rte_event *ev, \ + uint64_t timeout_ticks) \ + { \ + struct cn10k_sso_hws *ws = port; \ + uint16_t ret = 1; \ + uint64_t iter; \ + \ + if (ws->swtag_req) { \ + ws->swtag_req = 0; \ + cnxk_sso_hws_swtag_wait(ws->base + SSOW_LF_GWS_WQE0); \ + return ret; \ + } \ + ret = cn10k_sso_hws_get_work(ws, ev, flags, ws->lookup_mem); \ + for (iter = 1; iter < timeout_ticks && (ret == 0); iter++) \ + ret = cn10k_sso_hws_get_work(ws, ev, flags, \ + ws->lookup_mem); \ + return ret; \ + } + +#define SSO_DEQ_TMO_SEG(fn, flags) SSO_DEQ_TMO(fn, flags | NIX_RX_MULTI_SEG_F) +#define SSO_DEQ_TMO_CA(fn, flags) SSO_DEQ_TMO(fn, flags | CPT_RX_WQE_F) +#define SSO_DEQ_TMO_CA_SEG(fn, flags) SSO_DEQ_TMO_SEG(fn, flags | CPT_RX_WQE_F) + +#define SSO_CMN_DEQ_BURST(fnb, fn, flags) \ + uint16_t __rte_hot fnb(void *port, struct rte_event ev[], \ + uint16_t nb_events, uint64_t timeout_ticks) \ + { \ + RTE_SET_USED(nb_events); \ + return fn(port, ev, timeout_ticks); \ + } + +#define SSO_CMN_DEQ_SEG_BURST(fnb, fn, flags) \ + uint16_t __rte_hot fnb(void *port, struct rte_event ev[], \ + uint16_t nb_events, uint64_t timeout_ticks) \ + { \ + RTE_SET_USED(nb_events); \ + return fn(port, ev, timeout_ticks); \ + } + static __rte_always_inline struct cn10k_eth_txq * cn10k_sso_hws_xtract_meta(struct rte_mbuf *m, const uint64_t txq_data[][RTE_MAX_QUEUES_PER_PORT]) diff --git a/drivers/event/cnxk/cn10k_worker_deq.c b/drivers/event/cnxk/cn10k_worker_deq.c deleted file mode 100644 index 6083f69ffb..0000000000 --- a/drivers/event/cnxk/cn10k_worker_deq.c +++ /dev/null @@ -1,44 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(C) 2021 Marvell. - */ - -#include "cn10k_worker.h" -#include "cnxk_eventdev.h" -#include "cnxk_worker.h" - -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - uint16_t __rte_hot cn10k_sso_hws_deq_##name( \ - void *port, struct rte_event *ev, uint64_t timeout_ticks) \ - { \ - struct cn10k_sso_hws *ws = port; \ - \ - RTE_SET_USED(timeout_ticks); \ - \ - if (ws->swtag_req) { \ - ws->swtag_req = 0; \ - cnxk_sso_hws_swtag_wait(ws->base + SSOW_LF_GWS_WQE0); \ - return 1; \ - } \ - \ - return cn10k_sso_hws_get_work(ws, ev, flags, ws->lookup_mem); \ - } \ - \ - uint16_t __rte_hot cn10k_sso_hws_deq_seg_##name( \ - void *port, struct rte_event *ev, uint64_t timeout_ticks) \ - { \ - struct cn10k_sso_hws *ws = port; \ - \ - RTE_SET_USED(timeout_ticks); \ - \ - if (ws->swtag_req) { \ - ws->swtag_req = 0; \ - cnxk_sso_hws_swtag_wait(ws->base + SSOW_LF_GWS_WQE0); \ - return 1; \ - } \ - \ - return cn10k_sso_hws_get_work( \ - ws, ev, flags | NIX_RX_MULTI_SEG_F, ws->lookup_mem); \ - } - -NIX_RX_FASTPATH_MODES -#undef R diff --git a/drivers/event/cnxk/cn10k_worker_deq_burst.c b/drivers/event/cnxk/cn10k_worker_deq_burst.c deleted file mode 100644 index 8539d5d2ed..0000000000 --- a/drivers/event/cnxk/cn10k_worker_deq_burst.c +++ /dev/null @@ -1,29 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(C) 2021 Marvell. - */ - -#include "cn10k_worker.h" -#include "cnxk_eventdev.h" -#include "cnxk_worker.h" - -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - uint16_t __rte_hot cn10k_sso_hws_deq_burst_##name( \ - void *port, struct rte_event ev[], uint16_t nb_events, \ - uint64_t timeout_ticks) \ - { \ - RTE_SET_USED(nb_events); \ - \ - return cn10k_sso_hws_deq_##name(port, ev, timeout_ticks); \ - } \ - \ - uint16_t __rte_hot cn10k_sso_hws_deq_seg_burst_##name( \ - void *port, struct rte_event ev[], uint16_t nb_events, \ - uint64_t timeout_ticks) \ - { \ - RTE_SET_USED(nb_events); \ - \ - return cn10k_sso_hws_deq_seg_##name(port, ev, timeout_ticks); \ - } - -NIX_RX_FASTPATH_MODES -#undef R diff --git a/drivers/event/cnxk/cn10k_worker_deq_ca.c b/drivers/event/cnxk/cn10k_worker_deq_ca.c deleted file mode 100644 index 15c698edf8..0000000000 --- a/drivers/event/cnxk/cn10k_worker_deq_ca.c +++ /dev/null @@ -1,65 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(C) 2021 Marvell. - */ - -#include "cn10k_worker.h" -#include "cnxk_eventdev.h" -#include "cnxk_worker.h" - -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - uint16_t __rte_hot cn10k_sso_hws_deq_ca_##name( \ - void *port, struct rte_event *ev, uint64_t timeout_ticks) \ - { \ - struct cn10k_sso_hws *ws = port; \ - \ - RTE_SET_USED(timeout_ticks); \ - \ - if (ws->swtag_req) { \ - ws->swtag_req = 0; \ - cnxk_sso_hws_swtag_wait(ws->base + SSOW_LF_GWS_WQE0); \ - return 1; \ - } \ - \ - return cn10k_sso_hws_get_work(ws, ev, flags | CPT_RX_WQE_F, \ - ws->lookup_mem); \ - } \ - \ - uint16_t __rte_hot cn10k_sso_hws_deq_ca_burst_##name( \ - void *port, struct rte_event ev[], uint16_t nb_events, \ - uint64_t timeout_ticks) \ - { \ - RTE_SET_USED(nb_events); \ - \ - return cn10k_sso_hws_deq_ca_##name(port, ev, timeout_ticks); \ - } \ - \ - uint16_t __rte_hot cn10k_sso_hws_deq_ca_seg_##name( \ - void *port, struct rte_event *ev, uint64_t timeout_ticks) \ - { \ - struct cn10k_sso_hws *ws = port; \ - \ - RTE_SET_USED(timeout_ticks); \ - \ - if (ws->swtag_req) { \ - ws->swtag_req = 0; \ - cnxk_sso_hws_swtag_wait(ws->base + SSOW_LF_GWS_WQE0); \ - return 1; \ - } \ - \ - return cn10k_sso_hws_get_work( \ - ws, ev, flags | NIX_RX_MULTI_SEG_F | CPT_RX_WQE_F, \ - ws->lookup_mem); \ - } \ - \ - uint16_t __rte_hot cn10k_sso_hws_deq_ca_seg_burst_##name( \ - void *port, struct rte_event ev[], uint16_t nb_events, \ - uint64_t timeout_ticks) \ - { \ - RTE_SET_USED(nb_events); \ - \ - return cn10k_sso_hws_deq_ca_seg_##name(port, ev, \ - timeout_ticks); \ - } - -NIX_RX_FASTPATH_MODES -#undef R diff --git a/drivers/event/cnxk/cn10k_worker_deq_tmo.c b/drivers/event/cnxk/cn10k_worker_deq_tmo.c deleted file mode 100644 index 537ae37170..0000000000 --- a/drivers/event/cnxk/cn10k_worker_deq_tmo.c +++ /dev/null @@ -1,72 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(C) 2021 Marvell. - */ - -#include "cn10k_worker.h" -#include "cnxk_eventdev.h" -#include "cnxk_worker.h" - -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - uint16_t __rte_hot cn10k_sso_hws_deq_tmo_##name( \ - void *port, struct rte_event *ev, uint64_t timeout_ticks) \ - { \ - struct cn10k_sso_hws *ws = port; \ - uint16_t ret = 1; \ - uint64_t iter; \ - \ - if (ws->swtag_req) { \ - ws->swtag_req = 0; \ - cnxk_sso_hws_swtag_wait(ws->base + SSOW_LF_GWS_WQE0); \ - return ret; \ - } \ - \ - ret = cn10k_sso_hws_get_work(ws, ev, flags, ws->lookup_mem); \ - for (iter = 1; iter < timeout_ticks && (ret == 0); iter++) \ - ret = cn10k_sso_hws_get_work(ws, ev, flags, \ - ws->lookup_mem); \ - \ - return ret; \ - } \ - \ - uint16_t __rte_hot cn10k_sso_hws_deq_tmo_burst_##name( \ - void *port, struct rte_event ev[], uint16_t nb_events, \ - uint64_t timeout_ticks) \ - { \ - RTE_SET_USED(nb_events); \ - \ - return cn10k_sso_hws_deq_tmo_##name(port, ev, timeout_ticks); \ - } \ - \ - uint16_t __rte_hot cn10k_sso_hws_deq_tmo_seg_##name( \ - void *port, struct rte_event *ev, uint64_t timeout_ticks) \ - { \ - struct cn10k_sso_hws *ws = port; \ - uint16_t ret = 1; \ - uint64_t iter; \ - \ - if (ws->swtag_req) { \ - ws->swtag_req = 0; \ - cnxk_sso_hws_swtag_wait(ws->base + SSOW_LF_GWS_WQE0); \ - return ret; \ - } \ - \ - ret = cn10k_sso_hws_get_work(ws, ev, flags, ws->lookup_mem); \ - for (iter = 1; iter < timeout_ticks && (ret == 0); iter++) \ - ret = cn10k_sso_hws_get_work(ws, ev, flags, \ - ws->lookup_mem); \ - \ - return ret; \ - } \ - \ - uint16_t __rte_hot cn10k_sso_hws_deq_tmo_seg_burst_##name( \ - void *port, struct rte_event ev[], uint16_t nb_events, \ - uint64_t timeout_ticks) \ - { \ - RTE_SET_USED(nb_events); \ - \ - return cn10k_sso_hws_deq_tmo_seg_##name(port, ev, \ - timeout_ticks); \ - } - -NIX_RX_FASTPATH_MODES -#undef R diff --git a/drivers/event/cnxk/meson.build b/drivers/event/cnxk/meson.build index c39515dadc..ac452fe451 100644 --- a/drivers/event/cnxk/meson.build +++ b/drivers/event/cnxk/meson.build @@ -13,10 +13,6 @@ sources = files( 'cn9k_worker.c', 'cn10k_eventdev.c', 'cn10k_worker.c', - 'cn10k_worker_deq.c', - 'cn10k_worker_deq_burst.c', - 'cn10k_worker_deq_ca.c', - 'cn10k_worker_deq_tmo.c', 'cn10k_worker_tx_enq.c', 'cn10k_worker_tx_enq_seg.c', 'cnxk_eventdev.c', @@ -327,6 +323,137 @@ sources += files( 'tx/cn9k/tx_112_127_dual_seg.c', ) +sources += files( + 'deq/cn10k/deq_0_15_burst.c', + 'deq/cn10k/deq_16_31_burst.c', + 'deq/cn10k/deq_32_47_burst.c', + 'deq/cn10k/deq_48_63_burst.c', + 'deq/cn10k/deq_64_79_burst.c', + 'deq/cn10k/deq_80_95_burst.c', + 'deq/cn10k/deq_96_111_burst.c', + 'deq/cn10k/deq_112_127_burst.c', + 'deq/cn10k/deq_0_15_seg_burst.c', + 'deq/cn10k/deq_16_31_seg_burst.c', + 'deq/cn10k/deq_32_47_seg_burst.c', + 'deq/cn10k/deq_48_63_seg_burst.c', + 'deq/cn10k/deq_64_79_seg_burst.c', + 'deq/cn10k/deq_80_95_seg_burst.c', + 'deq/cn10k/deq_96_111_seg_burst.c', + 'deq/cn10k/deq_112_127_seg_burst.c', + 'deq/cn10k/deq_0_15.c', + 'deq/cn10k/deq_16_31.c', + 'deq/cn10k/deq_32_47.c', + 'deq/cn10k/deq_48_63.c', + 'deq/cn10k/deq_64_79.c', + 'deq/cn10k/deq_80_95.c', + 'deq/cn10k/deq_96_111.c', + 'deq/cn10k/deq_112_127.c', + 'deq/cn10k/deq_0_15_seg.c', + 'deq/cn10k/deq_16_31_seg.c', + 'deq/cn10k/deq_32_47_seg.c', + 'deq/cn10k/deq_48_63_seg.c', + 'deq/cn10k/deq_64_79_seg.c', + 'deq/cn10k/deq_80_95_seg.c', + 'deq/cn10k/deq_96_111_seg.c', + 'deq/cn10k/deq_112_127_seg.c', + 'deq/cn10k/deq_0_15_tmo.c', + 'deq/cn10k/deq_16_31_tmo.c', + 'deq/cn10k/deq_32_47_tmo.c', + 'deq/cn10k/deq_48_63_tmo.c', + 'deq/cn10k/deq_64_79_tmo.c', + 'deq/cn10k/deq_80_95_tmo.c', + 'deq/cn10k/deq_96_111_tmo.c', + 'deq/cn10k/deq_112_127_tmo.c', + 'deq/cn10k/deq_0_15_tmo_burst.c', + 'deq/cn10k/deq_16_31_tmo_burst.c', + 'deq/cn10k/deq_32_47_tmo_burst.c', + 'deq/cn10k/deq_48_63_tmo_burst.c', + 'deq/cn10k/deq_64_79_tmo_burst.c', + 'deq/cn10k/deq_80_95_tmo_burst.c', + 'deq/cn10k/deq_96_111_tmo_burst.c', + 'deq/cn10k/deq_112_127_tmo_burst.c', + 'deq/cn10k/deq_0_15_tmo_seg.c', + 'deq/cn10k/deq_16_31_tmo_seg.c', + 'deq/cn10k/deq_32_47_tmo_seg.c', + 'deq/cn10k/deq_48_63_tmo_seg.c', + 'deq/cn10k/deq_64_79_tmo_seg.c', + 'deq/cn10k/deq_80_95_tmo_seg.c', + 'deq/cn10k/deq_96_111_tmo_seg.c', + 'deq/cn10k/deq_112_127_tmo_seg.c', + 'deq/cn10k/deq_0_15_tmo_seg_burst.c', + 'deq/cn10k/deq_16_31_tmo_seg_burst.c', + 'deq/cn10k/deq_32_47_tmo_seg_burst.c', + 'deq/cn10k/deq_48_63_tmo_seg_burst.c', + 'deq/cn10k/deq_64_79_tmo_seg_burst.c', + 'deq/cn10k/deq_80_95_tmo_seg_burst.c', + 'deq/cn10k/deq_96_111_tmo_seg_burst.c', + 'deq/cn10k/deq_112_127_tmo_seg_burst.c', + 'deq/cn10k/deq_0_15_ca.c', + 'deq/cn10k/deq_16_31_ca.c', + 'deq/cn10k/deq_32_47_ca.c', + 'deq/cn10k/deq_48_63_ca.c', + 'deq/cn10k/deq_64_79_ca.c', + 'deq/cn10k/deq_80_95_ca.c', + 'deq/cn10k/deq_96_111_ca.c', + 'deq/cn10k/deq_112_127_ca.c', + 'deq/cn10k/deq_0_15_ca_burst.c', + 'deq/cn10k/deq_16_31_ca_burst.c', + 'deq/cn10k/deq_32_47_ca_burst.c', + 'deq/cn10k/deq_48_63_ca_burst.c', + 'deq/cn10k/deq_64_79_ca_burst.c', + 'deq/cn10k/deq_80_95_ca_burst.c', + 'deq/cn10k/deq_96_111_ca_burst.c', + 'deq/cn10k/deq_112_127_ca_burst.c', + 'deq/cn10k/deq_0_15_ca_seg.c', + 'deq/cn10k/deq_16_31_ca_seg.c', + 'deq/cn10k/deq_32_47_ca_seg.c', + 'deq/cn10k/deq_48_63_ca_seg.c', + 'deq/cn10k/deq_64_79_ca_seg.c', + 'deq/cn10k/deq_80_95_ca_seg.c', + 'deq/cn10k/deq_96_111_ca_seg.c', + 'deq/cn10k/deq_112_127_ca_seg.c', + 'deq/cn10k/deq_0_15_ca_seg_burst.c', + 'deq/cn10k/deq_16_31_ca_seg_burst.c', + 'deq/cn10k/deq_32_47_ca_seg_burst.c', + 'deq/cn10k/deq_48_63_ca_seg_burst.c', + 'deq/cn10k/deq_64_79_ca_seg_burst.c', + 'deq/cn10k/deq_80_95_ca_seg_burst.c', + 'deq/cn10k/deq_96_111_ca_seg_burst.c', + 'deq/cn10k/deq_112_127_ca_seg_burst.c', + 'deq/cn10k/deq_0_15_ca_tmo.c', + 'deq/cn10k/deq_16_31_ca_tmo.c', + 'deq/cn10k/deq_32_47_ca_tmo.c', + 'deq/cn10k/deq_48_63_ca_tmo.c', + 'deq/cn10k/deq_64_79_ca_tmo.c', + 'deq/cn10k/deq_80_95_ca_tmo.c', + 'deq/cn10k/deq_96_111_ca_tmo.c', + 'deq/cn10k/deq_112_127_ca_tmo.c', + 'deq/cn10k/deq_0_15_ca_tmo_burst.c', + 'deq/cn10k/deq_16_31_ca_tmo_burst.c', + 'deq/cn10k/deq_32_47_ca_tmo_burst.c', + 'deq/cn10k/deq_48_63_ca_tmo_burst.c', + 'deq/cn10k/deq_64_79_ca_tmo_burst.c', + 'deq/cn10k/deq_80_95_ca_tmo_burst.c', + 'deq/cn10k/deq_96_111_ca_tmo_burst.c', + 'deq/cn10k/deq_112_127_ca_tmo_burst.c', + 'deq/cn10k/deq_0_15_ca_tmo_seg.c', + 'deq/cn10k/deq_16_31_ca_tmo_seg.c', + 'deq/cn10k/deq_32_47_ca_tmo_seg.c', + 'deq/cn10k/deq_48_63_ca_tmo_seg.c', + 'deq/cn10k/deq_64_79_ca_tmo_seg.c', + 'deq/cn10k/deq_80_95_ca_tmo_seg.c', + 'deq/cn10k/deq_96_111_ca_tmo_seg.c', + 'deq/cn10k/deq_112_127_ca_tmo_seg.c', + 'deq/cn10k/deq_0_15_ca_tmo_seg_burst.c', + 'deq/cn10k/deq_16_31_ca_tmo_seg_burst.c', + 'deq/cn10k/deq_32_47_ca_tmo_seg_burst.c', + 'deq/cn10k/deq_48_63_ca_tmo_seg_burst.c', + 'deq/cn10k/deq_64_79_ca_tmo_seg_burst.c', + 'deq/cn10k/deq_80_95_ca_tmo_seg_burst.c', + 'deq/cn10k/deq_96_111_ca_tmo_seg_burst.c', + 'deq/cn10k/deq_112_127_ca_tmo_seg_burst.c', +) + extra_flags = ['-flax-vector-conversions', '-Wno-strict-aliasing'] foreach flag: extra_flags if cc.has_argument(flag) diff --git a/drivers/net/cnxk/cn10k_rx.h b/drivers/net/cnxk/cn10k_rx.h index fe408907a6..a2442d3726 100644 --- a/drivers/net/cnxk/cn10k_rx.h +++ b/drivers/net/cnxk/cn10k_rx.h @@ -17,6 +17,7 @@ #define NIX_RX_OFFLOAD_TSTAMP_F BIT(4) #define NIX_RX_OFFLOAD_VLAN_STRIP_F BIT(5) #define NIX_RX_OFFLOAD_SECURITY_F BIT(6) +#define NIX_RX_OFFLOAD_MAX (NIX_RX_OFFLOAD_SECURITY_F << 1) /* Flags to control cqe_to_mbuf conversion function. * Defining it from backwards to denote its been @@ -987,278 +988,214 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_mbuf **mbufs, uint16_t pkts, #define R_SEC_F NIX_RX_OFFLOAD_SECURITY_F /* [R_SEC_F] [RX_VLAN_F] [TS] [MARK] [CKSUM] [PTYPE] [RSS] */ -#define NIX_RX_FASTPATH_MODES \ -R(no_offload, 0, 0, 0, 0, 0, 0, 0, \ - NIX_RX_OFFLOAD_NONE) \ -R(rss, 0, 0, 0, 0, 0, 0, 1, \ - RSS_F) \ -R(ptype, 0, 0, 0, 0, 0, 1, 0, \ - PTYPE_F) \ -R(ptype_rss, 0, 0, 0, 0, 0, 1, 1, \ - PTYPE_F | RSS_F) \ -R(cksum, 0, 0, 0, 0, 1, 0, 0, \ - CKSUM_F) \ -R(cksum_rss, 0, 0, 0, 0, 1, 0, 1, \ - CKSUM_F | RSS_F) \ -R(cksum_ptype, 0, 0, 0, 0, 1, 1, 0, \ - CKSUM_F | PTYPE_F) \ -R(cksum_ptype_rss, 0, 0, 0, 0, 1, 1, 1, \ - CKSUM_F | PTYPE_F | RSS_F) \ -R(mark, 0, 0, 0, 1, 0, 0, 0, \ - MARK_F) \ -R(mark_rss, 0, 0, 0, 1, 0, 0, 1, \ - MARK_F | RSS_F) \ -R(mark_ptype, 0, 0, 0, 1, 0, 1, 0, \ - MARK_F | PTYPE_F) \ -R(mark_ptype_rss, 0, 0, 0, 1, 0, 1, 1, \ - MARK_F | PTYPE_F | RSS_F) \ -R(mark_cksum, 0, 0, 0, 1, 1, 0, 0, \ - MARK_F | CKSUM_F) \ -R(mark_cksum_rss, 0, 0, 0, 1, 1, 0, 1, \ - MARK_F | CKSUM_F | RSS_F) \ -R(mark_cksum_ptype, 0, 0, 0, 1, 1, 1, 0, \ - MARK_F | CKSUM_F | PTYPE_F) \ -R(mark_cksum_ptype_rss, 0, 0, 0, 1, 1, 1, 1, \ - MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(ts, 0, 0, 1, 0, 0, 0, 0, \ - TS_F) \ -R(ts_rss, 0, 0, 1, 0, 0, 0, 1, \ - TS_F | RSS_F) \ -R(ts_ptype, 0, 0, 1, 0, 0, 1, 0, \ - TS_F | PTYPE_F) \ -R(ts_ptype_rss, 0, 0, 1, 0, 0, 1, 1, \ - TS_F | PTYPE_F | RSS_F) \ -R(ts_cksum, 0, 0, 1, 0, 1, 0, 0, \ - TS_F | CKSUM_F) \ -R(ts_cksum_rss, 0, 0, 1, 0, 1, 0, 1, \ - TS_F | CKSUM_F | RSS_F) \ -R(ts_cksum_ptype, 0, 0, 1, 0, 1, 1, 0, \ - TS_F | CKSUM_F | PTYPE_F) \ -R(ts_cksum_ptype_rss, 0, 0, 1, 0, 1, 1, 1, \ - TS_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(ts_mark, 0, 0, 1, 1, 0, 0, 0, \ - TS_F | MARK_F) \ -R(ts_mark_rss, 0, 0, 1, 1, 0, 0, 1, \ - TS_F | MARK_F | RSS_F) \ -R(ts_mark_ptype, 0, 0, 1, 1, 0, 1, 0, \ - TS_F | MARK_F | PTYPE_F) \ -R(ts_mark_ptype_rss, 0, 0, 1, 1, 0, 1, 1, \ - TS_F | MARK_F | PTYPE_F | RSS_F) \ -R(ts_mark_cksum, 0, 0, 1, 1, 1, 0, 0, \ - TS_F | MARK_F | CKSUM_F) \ -R(ts_mark_cksum_rss, 0, 0, 1, 1, 1, 0, 1, \ - TS_F | MARK_F | CKSUM_F | RSS_F) \ -R(ts_mark_cksum_ptype, 0, 0, 1, 1, 1, 1, 0, \ - TS_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(ts_mark_cksum_ptype_rss, 0, 0, 1, 1, 1, 1, 1, \ - TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(vlan, 0, 1, 0, 0, 0, 0, 0, \ - RX_VLAN_F) \ -R(vlan_rss, 0, 1, 0, 0, 0, 0, 1, \ - RX_VLAN_F | RSS_F) \ -R(vlan_ptype, 0, 1, 0, 0, 0, 1, 0, \ - RX_VLAN_F | PTYPE_F) \ -R(vlan_ptype_rss, 0, 1, 0, 0, 0, 1, 1, \ - RX_VLAN_F | PTYPE_F | RSS_F) \ -R(vlan_cksum, 0, 1, 0, 0, 1, 0, 0, \ - RX_VLAN_F | CKSUM_F) \ -R(vlan_cksum_rss, 0, 1, 0, 0, 1, 0, 1, \ - RX_VLAN_F | CKSUM_F | RSS_F) \ -R(vlan_cksum_ptype, 0, 1, 0, 0, 1, 1, 0, \ - RX_VLAN_F | CKSUM_F | PTYPE_F) \ -R(vlan_cksum_ptype_rss, 0, 1, 0, 0, 1, 1, 1, \ - RX_VLAN_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(vlan_mark, 0, 1, 0, 1, 0, 0, 0, \ - RX_VLAN_F | MARK_F) \ -R(vlan_mark_rss, 0, 1, 0, 1, 0, 0, 1, \ - RX_VLAN_F | MARK_F | RSS_F) \ -R(vlan_mark_ptype, 0, 1, 0, 1, 0, 1, 0, \ - RX_VLAN_F | MARK_F | PTYPE_F) \ -R(vlan_mark_ptype_rss, 0, 1, 0, 1, 0, 1, 1, \ - RX_VLAN_F | MARK_F | PTYPE_F | RSS_F) \ -R(vlan_mark_cksum, 0, 1, 0, 1, 1, 0, 0, \ - RX_VLAN_F | MARK_F | CKSUM_F) \ -R(vlan_mark_cksum_rss, 0, 1, 0, 1, 1, 0, 1, \ - RX_VLAN_F | MARK_F | CKSUM_F | RSS_F) \ -R(vlan_mark_cksum_ptype, 0, 1, 0, 1, 1, 1, 0, \ - RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(vlan_mark_cksum_ptype_rss, 0, 1, 0, 1, 1, 1, 1, \ - RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(vlan_ts, 0, 1, 1, 0, 0, 0, 0, \ - RX_VLAN_F | TS_F) \ -R(vlan_ts_rss, 0, 1, 1, 0, 0, 0, 1, \ - RX_VLAN_F | TS_F | RSS_F) \ -R(vlan_ts_ptype, 0, 1, 1, 0, 0, 1, 0, \ - RX_VLAN_F | TS_F | PTYPE_F) \ -R(vlan_ts_ptype_rss, 0, 1, 1, 0, 0, 1, 1, \ - RX_VLAN_F | TS_F | PTYPE_F | RSS_F) \ -R(vlan_ts_cksum, 0, 1, 1, 0, 1, 0, 0, \ - RX_VLAN_F | TS_F | CKSUM_F) \ -R(vlan_ts_cksum_rss, 0, 1, 1, 0, 1, 0, 1, \ - RX_VLAN_F | TS_F | CKSUM_F | RSS_F) \ -R(vlan_ts_cksum_ptype, 0, 1, 1, 0, 1, 1, 0, \ - RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F) \ -R(vlan_ts_cksum_ptype_rss, 0, 1, 1, 0, 1, 1, 1, \ - RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(vlan_ts_mark, 0, 1, 1, 1, 0, 0, 0, \ - RX_VLAN_F | TS_F | MARK_F) \ -R(vlan_ts_mark_rss, 0, 1, 1, 1, 0, 0, 1, \ - RX_VLAN_F | TS_F | MARK_F | RSS_F) \ -R(vlan_ts_mark_ptype, 0, 1, 1, 1, 0, 1, 0, \ - RX_VLAN_F | TS_F | MARK_F | PTYPE_F) \ -R(vlan_ts_mark_ptype_rss, 0, 1, 1, 1, 0, 1, 1, \ - RX_VLAN_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ -R(vlan_ts_mark_cksum, 0, 1, 1, 1, 1, 0, 0, \ - RX_VLAN_F | TS_F | MARK_F | CKSUM_F) \ -R(vlan_ts_mark_cksum_rss, 0, 1, 1, 1, 1, 0, 1, \ - RX_VLAN_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ -R(vlan_ts_mark_cksum_ptype, 0, 1, 1, 1, 1, 1, 0, \ - RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(vlan_ts_mark_cksum_ptype_rss, 0, 1, 1, 1, 1, 1, 1, \ - RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(sec, 1, 0, 0, 0, 0, 0, 0, \ - R_SEC_F) \ -R(sec_rss, 1, 0, 0, 0, 0, 0, 1, \ - RSS_F) \ -R(sec_ptype, 1, 0, 0, 0, 0, 1, 0, \ - R_SEC_F | PTYPE_F) \ -R(sec_ptype_rss, 1, 0, 0, 0, 0, 1, 1, \ - R_SEC_F | PTYPE_F | RSS_F) \ -R(sec_cksum, 1, 0, 0, 0, 1, 0, 0, \ - R_SEC_F | CKSUM_F) \ -R(sec_cksum_rss, 1, 0, 0, 0, 1, 0, 1, \ - R_SEC_F | CKSUM_F | RSS_F) \ -R(sec_cksum_ptype, 1, 0, 0, 0, 1, 1, 0, \ - R_SEC_F | CKSUM_F | PTYPE_F) \ -R(sec_cksum_ptype_rss, 1, 0, 0, 0, 1, 1, 1, \ - R_SEC_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(sec_mark, 1, 0, 0, 1, 0, 0, 0, \ - R_SEC_F | MARK_F) \ -R(sec_mark_rss, 1, 0, 0, 1, 0, 0, 1, \ - R_SEC_F | MARK_F | RSS_F) \ -R(sec_mark_ptype, 1, 0, 0, 1, 0, 1, 0, \ - R_SEC_F | MARK_F | PTYPE_F) \ -R(sec_mark_ptype_rss, 1, 0, 0, 1, 0, 1, 1, \ - R_SEC_F | MARK_F | PTYPE_F | RSS_F) \ -R(sec_mark_cksum, 1, 0, 0, 1, 1, 0, 0, \ - R_SEC_F | MARK_F | CKSUM_F) \ -R(sec_mark_cksum_rss, 1, 0, 0, 1, 1, 0, 1, \ - R_SEC_F | MARK_F | CKSUM_F | RSS_F) \ -R(sec_mark_cksum_ptype, 1, 0, 0, 1, 1, 1, 0, \ - R_SEC_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(sec_mark_cksum_ptype_rss, 1, 0, 0, 1, 1, 1, 1, \ - R_SEC_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(sec_ts, 1, 0, 1, 0, 0, 0, 0, \ - R_SEC_F | TS_F) \ -R(sec_ts_rss, 1, 0, 1, 0, 0, 0, 1, \ - R_SEC_F | TS_F | RSS_F) \ -R(sec_ts_ptype, 1, 0, 1, 0, 0, 1, 0, \ - R_SEC_F | TS_F | PTYPE_F) \ -R(sec_ts_ptype_rss, 1, 0, 1, 0, 0, 1, 1, \ - R_SEC_F | TS_F | PTYPE_F | RSS_F) \ -R(sec_ts_cksum, 1, 0, 1, 0, 1, 0, 0, \ - R_SEC_F | TS_F | CKSUM_F) \ -R(sec_ts_cksum_rss, 1, 0, 1, 0, 1, 0, 1, \ - R_SEC_F | TS_F | CKSUM_F | RSS_F) \ -R(sec_ts_cksum_ptype, 1, 0, 1, 0, 1, 1, 0, \ - R_SEC_F | TS_F | CKSUM_F | PTYPE_F) \ -R(sec_ts_cksum_ptype_rss, 1, 0, 1, 0, 1, 1, 1, \ - R_SEC_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(sec_ts_mark, 1, 0, 1, 1, 0, 0, 0, \ - R_SEC_F | TS_F | MARK_F) \ -R(sec_ts_mark_rss, 1, 0, 1, 1, 0, 0, 1, \ - R_SEC_F | TS_F | MARK_F | RSS_F) \ -R(sec_ts_mark_ptype, 1, 0, 1, 1, 0, 1, 0, \ - R_SEC_F | TS_F | MARK_F | PTYPE_F) \ -R(sec_ts_mark_ptype_rss, 1, 0, 1, 1, 0, 1, 1, \ - R_SEC_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ -R(sec_ts_mark_cksum, 1, 0, 1, 1, 1, 0, 0, \ - R_SEC_F | TS_F | MARK_F | CKSUM_F) \ -R(sec_ts_mark_cksum_rss, 1, 0, 1, 1, 1, 0, 1, \ - R_SEC_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ -R(sec_ts_mark_cksum_ptype, 1, 0, 1, 1, 1, 1, 0, \ - R_SEC_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(sec_ts_mark_cksum_ptype_rss, 1, 0, 1, 1, 1, 1, 1, \ - R_SEC_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(sec_vlan, 1, 1, 0, 0, 0, 0, 0, \ - R_SEC_F | RX_VLAN_F) \ -R(sec_vlan_rss, 1, 1, 0, 0, 0, 0, 1, \ - R_SEC_F | RX_VLAN_F | RSS_F) \ -R(sec_vlan_ptype, 1, 1, 0, 0, 0, 1, 0, \ - R_SEC_F | RX_VLAN_F | PTYPE_F) \ -R(sec_vlan_ptype_rss, 1, 1, 0, 0, 0, 1, 1, \ - R_SEC_F | RX_VLAN_F | PTYPE_F | RSS_F) \ -R(sec_vlan_cksum, 1, 1, 0, 0, 1, 0, 0, \ - R_SEC_F | RX_VLAN_F | CKSUM_F) \ -R(sec_vlan_cksum_rss, 1, 1, 0, 0, 1, 0, 1, \ - R_SEC_F | RX_VLAN_F | CKSUM_F | RSS_F) \ -R(sec_vlan_cksum_ptype, 1, 1, 0, 0, 1, 1, 0, \ - R_SEC_F | RX_VLAN_F | CKSUM_F | PTYPE_F) \ -R(sec_vlan_cksum_ptype_rss, 1, 1, 0, 0, 1, 1, 1, \ - R_SEC_F | RX_VLAN_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(sec_vlan_mark, 1, 1, 0, 1, 0, 0, 0, \ - R_SEC_F | RX_VLAN_F | MARK_F) \ -R(sec_vlan_mark_rss, 1, 1, 0, 1, 0, 0, 1, \ - R_SEC_F | RX_VLAN_F | MARK_F | RSS_F) \ -R(sec_vlan_mark_ptype, 1, 1, 0, 1, 0, 1, 0, \ - R_SEC_F | RX_VLAN_F | MARK_F | PTYPE_F) \ -R(sec_vlan_mark_ptype_rss, 1, 1, 0, 1, 0, 1, 1, \ - R_SEC_F | RX_VLAN_F | MARK_F | PTYPE_F | RSS_F) \ -R(sec_vlan_mark_cksum, 1, 1, 0, 1, 1, 0, 0, \ - R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F) \ -R(sec_vlan_mark_cksum_rss, 1, 1, 0, 1, 1, 0, 1, \ - R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | RSS_F) \ -R(sec_vlan_mark_cksum_ptype, 1, 1, 0, 1, 1, 1, 0, \ - R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(sec_vlan_mark_cksum_ptype_rss, 1, 1, 0, 1, 1, 1, 1, \ - R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(sec_vlan_ts, 1, 1, 1, 0, 0, 0, 0, \ - R_SEC_F | RX_VLAN_F | TS_F) \ -R(sec_vlan_ts_rss, 1, 1, 1, 0, 0, 0, 1, \ - R_SEC_F | RX_VLAN_F | TS_F | RSS_F) \ -R(sec_vlan_ts_ptype, 1, 1, 1, 0, 0, 1, 0, \ - R_SEC_F | RX_VLAN_F | TS_F | PTYPE_F) \ -R(sec_vlan_ts_ptype_rss, 1, 1, 1, 0, 0, 1, 1, \ - R_SEC_F | RX_VLAN_F | TS_F | PTYPE_F | RSS_F) \ -R(sec_vlan_ts_cksum, 1, 1, 1, 0, 1, 0, 0, \ - R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F) \ -R(sec_vlan_ts_cksum_rss, 1, 1, 1, 0, 1, 0, 1, \ - R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | RSS_F) \ -R(sec_vlan_ts_cksum_ptype, 1, 1, 1, 0, 1, 1, 0, \ - R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F) \ -R(sec_vlan_ts_cksum_ptype_rss, 1, 1, 1, 0, 1, 1, 1, \ - R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ -R(sec_vlan_ts_mark, 1, 1, 1, 1, 0, 0, 0, \ - R_SEC_F | RX_VLAN_F | TS_F | MARK_F) \ -R(sec_vlan_ts_mark_rss, 1, 1, 1, 1, 0, 0, 1, \ - R_SEC_F | RX_VLAN_F | TS_F | MARK_F | RSS_F) \ -R(sec_vlan_ts_mark_ptype, 1, 1, 1, 1, 0, 1, 0, \ - R_SEC_F | RX_VLAN_F | TS_F | MARK_F | PTYPE_F) \ -R(sec_vlan_ts_mark_ptype_rss, 1, 1, 1, 1, 0, 1, 1, \ - R_SEC_F | RX_VLAN_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ -R(sec_vlan_ts_mark_cksum, 1, 1, 1, 1, 1, 0, 0, \ - R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F) \ -R(sec_vlan_ts_mark_cksum_rss, 1, 1, 1, 1, 1, 0, 1, \ - R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ -R(sec_vlan_ts_mark_cksum_ptype, 1, 1, 1, 1, 1, 1, 0, \ - R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ -R(sec_vlan_ts_mark_cksum_ptype_rss, 1, 1, 1, 1, 1, 1, 1, \ - R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) - -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ +#define NIX_RX_FASTPATH_MODES_0_15 \ + R(no_offload, NIX_RX_OFFLOAD_NONE) \ + R(rss, RSS_F) \ + R(ptype, PTYPE_F) \ + R(ptype_rss, PTYPE_F | RSS_F) \ + R(cksum, CKSUM_F) \ + R(cksum_rss, CKSUM_F | RSS_F) \ + R(cksum_ptype, CKSUM_F | PTYPE_F) \ + R(cksum_ptype_rss, CKSUM_F | PTYPE_F | RSS_F) \ + R(mark, MARK_F) \ + R(mark_rss, MARK_F | RSS_F) \ + R(mark_ptype, MARK_F | PTYPE_F) \ + R(mark_ptype_rss, MARK_F | PTYPE_F | RSS_F) \ + R(mark_cksum, MARK_F | CKSUM_F) \ + R(mark_cksum_rss, MARK_F | CKSUM_F | RSS_F) \ + R(mark_cksum_ptype, MARK_F | CKSUM_F | PTYPE_F) \ + R(mark_cksum_ptype_rss, MARK_F | CKSUM_F | PTYPE_F | RSS_F) + +#define NIX_RX_FASTPATH_MODES_16_31 \ + R(ts, TS_F) \ + R(ts_rss, TS_F | RSS_F) \ + R(ts_ptype, TS_F | PTYPE_F) \ + R(ts_ptype_rss, TS_F | PTYPE_F | RSS_F) \ + R(ts_cksum, TS_F | CKSUM_F) \ + R(ts_cksum_rss, TS_F | CKSUM_F | RSS_F) \ + R(ts_cksum_ptype, TS_F | CKSUM_F | PTYPE_F) \ + R(ts_cksum_ptype_rss, TS_F | CKSUM_F | PTYPE_F | RSS_F) \ + R(ts_mark, TS_F | MARK_F) \ + R(ts_mark_rss, TS_F | MARK_F | RSS_F) \ + R(ts_mark_ptype, TS_F | MARK_F | PTYPE_F) \ + R(ts_mark_ptype_rss, TS_F | MARK_F | PTYPE_F | RSS_F) \ + R(ts_mark_cksum, TS_F | MARK_F | CKSUM_F) \ + R(ts_mark_cksum_rss, TS_F | MARK_F | CKSUM_F | RSS_F) \ + R(ts_mark_cksum_ptype, TS_F | MARK_F | CKSUM_F | PTYPE_F) \ + R(ts_mark_cksum_ptype_rss, TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) + +#define NIX_RX_FASTPATH_MODES_32_47 \ + R(vlan, RX_VLAN_F) \ + R(vlan_rss, RX_VLAN_F | RSS_F) \ + R(vlan_ptype, RX_VLAN_F | PTYPE_F) \ + R(vlan_ptype_rss, RX_VLAN_F | PTYPE_F | RSS_F) \ + R(vlan_cksum, RX_VLAN_F | CKSUM_F) \ + R(vlan_cksum_rss, RX_VLAN_F | CKSUM_F | RSS_F) \ + R(vlan_cksum_ptype, RX_VLAN_F | CKSUM_F | PTYPE_F) \ + R(vlan_cksum_ptype_rss, RX_VLAN_F | CKSUM_F | PTYPE_F | RSS_F) \ + R(vlan_mark, RX_VLAN_F | MARK_F) \ + R(vlan_mark_rss, RX_VLAN_F | MARK_F | RSS_F) \ + R(vlan_mark_ptype, RX_VLAN_F | MARK_F | PTYPE_F) \ + R(vlan_mark_ptype_rss, RX_VLAN_F | MARK_F | PTYPE_F | RSS_F) \ + R(vlan_mark_cksum, RX_VLAN_F | MARK_F | CKSUM_F) \ + R(vlan_mark_cksum_rss, RX_VLAN_F | MARK_F | CKSUM_F | RSS_F) \ + R(vlan_mark_cksum_ptype, RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F) \ + R(vlan_mark_cksum_ptype_rss, \ + RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) + +#define NIX_RX_FASTPATH_MODES_48_63 \ + R(vlan_ts, RX_VLAN_F | TS_F) \ + R(vlan_ts_rss, RX_VLAN_F | TS_F | RSS_F) \ + R(vlan_ts_ptype, RX_VLAN_F | TS_F | PTYPE_F) \ + R(vlan_ts_ptype_rss, RX_VLAN_F | TS_F | PTYPE_F | RSS_F) \ + R(vlan_ts_cksum, RX_VLAN_F | TS_F | CKSUM_F) \ + R(vlan_ts_cksum_rss, RX_VLAN_F | TS_F | CKSUM_F | RSS_F) \ + R(vlan_ts_cksum_ptype, RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F) \ + R(vlan_ts_cksum_ptype_rss, \ + RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ + R(vlan_ts_mark, RX_VLAN_F | TS_F | MARK_F) \ + R(vlan_ts_mark_rss, RX_VLAN_F | TS_F | MARK_F | RSS_F) \ + R(vlan_ts_mark_ptype, RX_VLAN_F | TS_F | MARK_F | PTYPE_F) \ + R(vlan_ts_mark_ptype_rss, RX_VLAN_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ + R(vlan_ts_mark_cksum, RX_VLAN_F | TS_F | MARK_F | CKSUM_F) \ + R(vlan_ts_mark_cksum_rss, RX_VLAN_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ + R(vlan_ts_mark_cksum_ptype, \ + RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ + R(vlan_ts_mark_cksum_ptype_rss, \ + RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) + +#define NIX_RX_FASTPATH_MODES_64_79 \ + R(sec, R_SEC_F) \ + R(sec_rss, R_SEC_F | RSS_F) \ + R(sec_ptype, R_SEC_F | PTYPE_F) \ + R(sec_ptype_rss, R_SEC_F | PTYPE_F | RSS_F) \ + R(sec_cksum, R_SEC_F | CKSUM_F) \ + R(sec_cksum_rss, R_SEC_F | CKSUM_F | RSS_F) \ + R(sec_cksum_ptype, R_SEC_F | CKSUM_F | PTYPE_F) \ + R(sec_cksum_ptype_rss, R_SEC_F | CKSUM_F | PTYPE_F | RSS_F) \ + R(sec_mark, R_SEC_F | MARK_F) \ + R(sec_mark_rss, R_SEC_F | MARK_F | RSS_F) \ + R(sec_mark_ptype, R_SEC_F | MARK_F | PTYPE_F) \ + R(sec_mark_ptype_rss, R_SEC_F | MARK_F | PTYPE_F | RSS_F) \ + R(sec_mark_cksum, R_SEC_F | MARK_F | CKSUM_F) \ + R(sec_mark_cksum_rss, R_SEC_F | MARK_F | CKSUM_F | RSS_F) \ + R(sec_mark_cksum_ptype, R_SEC_F | MARK_F | CKSUM_F | PTYPE_F) \ + R(sec_mark_cksum_ptype_rss, \ + R_SEC_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) + +#define NIX_RX_FASTPATH_MODES_80_95 \ + R(sec_ts, R_SEC_F | TS_F) \ + R(sec_ts_rss, R_SEC_F | TS_F | RSS_F) \ + R(sec_ts_ptype, R_SEC_F | TS_F | PTYPE_F) \ + R(sec_ts_ptype_rss, R_SEC_F | TS_F | PTYPE_F | RSS_F) \ + R(sec_ts_cksum, R_SEC_F | TS_F | CKSUM_F) \ + R(sec_ts_cksum_rss, R_SEC_F | TS_F | CKSUM_F | RSS_F) \ + R(sec_ts_cksum_ptype, R_SEC_F | TS_F | CKSUM_F | PTYPE_F) \ + R(sec_ts_cksum_ptype_rss, R_SEC_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ + R(sec_ts_mark, R_SEC_F | TS_F | MARK_F) \ + R(sec_ts_mark_rss, R_SEC_F | TS_F | MARK_F | RSS_F) \ + R(sec_ts_mark_ptype, R_SEC_F | TS_F | MARK_F | PTYPE_F) \ + R(sec_ts_mark_ptype_rss, R_SEC_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ + R(sec_ts_mark_cksum, R_SEC_F | TS_F | MARK_F | CKSUM_F) \ + R(sec_ts_mark_cksum_rss, R_SEC_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ + R(sec_ts_mark_cksum_ptype, \ + R_SEC_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ + R(sec_ts_mark_cksum_ptype_rss, \ + R_SEC_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) + +#define NIX_RX_FASTPATH_MODES_96_111 \ + R(sec_vlan, R_SEC_F | RX_VLAN_F) \ + R(sec_vlan_rss, R_SEC_F | RX_VLAN_F | RSS_F) \ + R(sec_vlan_ptype, R_SEC_F | RX_VLAN_F | PTYPE_F) \ + R(sec_vlan_ptype_rss, R_SEC_F | RX_VLAN_F | PTYPE_F | RSS_F) \ + R(sec_vlan_cksum, R_SEC_F | RX_VLAN_F | CKSUM_F) \ + R(sec_vlan_cksum_rss, R_SEC_F | RX_VLAN_F | CKSUM_F | RSS_F) \ + R(sec_vlan_cksum_ptype, R_SEC_F | RX_VLAN_F | CKSUM_F | PTYPE_F) \ + R(sec_vlan_cksum_ptype_rss, \ + R_SEC_F | RX_VLAN_F | CKSUM_F | PTYPE_F | RSS_F) \ + R(sec_vlan_mark, R_SEC_F | RX_VLAN_F | MARK_F) \ + R(sec_vlan_mark_rss, R_SEC_F | RX_VLAN_F | MARK_F | RSS_F) \ + R(sec_vlan_mark_ptype, R_SEC_F | RX_VLAN_F | MARK_F | PTYPE_F) \ + R(sec_vlan_mark_ptype_rss, \ + R_SEC_F | RX_VLAN_F | MARK_F | PTYPE_F | RSS_F) \ + R(sec_vlan_mark_cksum, R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F) \ + R(sec_vlan_mark_cksum_rss, \ + R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | RSS_F) \ + R(sec_vlan_mark_cksum_ptype, \ + R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F) \ + R(sec_vlan_mark_cksum_ptype_rss, \ + R_SEC_F | RX_VLAN_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) + +#define NIX_RX_FASTPATH_MODES_112_127 \ + R(sec_vlan_ts, R_SEC_F | RX_VLAN_F | TS_F) \ + R(sec_vlan_ts_rss, R_SEC_F | RX_VLAN_F | TS_F | RSS_F) \ + R(sec_vlan_ts_ptype, R_SEC_F | RX_VLAN_F | TS_F | PTYPE_F) \ + R(sec_vlan_ts_ptype_rss, R_SEC_F | RX_VLAN_F | TS_F | PTYPE_F | RSS_F) \ + R(sec_vlan_ts_cksum, R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F) \ + R(sec_vlan_ts_cksum_rss, R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | RSS_F) \ + R(sec_vlan_ts_cksum_ptype, \ + R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F) \ + R(sec_vlan_ts_cksum_ptype_rss, \ + R_SEC_F | RX_VLAN_F | TS_F | CKSUM_F | PTYPE_F | RSS_F) \ + R(sec_vlan_ts_mark, R_SEC_F | RX_VLAN_F | TS_F | MARK_F) \ + R(sec_vlan_ts_mark_rss, R_SEC_F | RX_VLAN_F | TS_F | MARK_F | RSS_F) \ + R(sec_vlan_ts_mark_ptype, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | PTYPE_F) \ + R(sec_vlan_ts_mark_ptype_rss, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | PTYPE_F | RSS_F) \ + R(sec_vlan_ts_mark_cksum, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F) \ + R(sec_vlan_ts_mark_cksum_rss, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | RSS_F) \ + R(sec_vlan_ts_mark_cksum_ptype, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F) \ + R(sec_vlan_ts_mark_cksum_ptype_rss, \ + R_SEC_F | RX_VLAN_F | TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) + +#define NIX_RX_FASTPATH_MODES \ + NIX_RX_FASTPATH_MODES_0_15 \ + NIX_RX_FASTPATH_MODES_16_31 \ + NIX_RX_FASTPATH_MODES_32_47 \ + NIX_RX_FASTPATH_MODES_48_63 \ + NIX_RX_FASTPATH_MODES_64_79 \ + NIX_RX_FASTPATH_MODES_80_95 \ + NIX_RX_FASTPATH_MODES_96_111 \ + NIX_RX_FASTPATH_MODES_112_127 + +#define R(name, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts); \ - \ + \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_mseg_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts); \ - \ + \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_vec_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts); \ - \ + \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_vec_mseg_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts); NIX_RX_FASTPATH_MODES #undef R +#define NIX_RX_RECV(fn, flags) \ + uint16_t __rte_noinline __rte_hot fn( \ + void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ + { \ + return cn10k_nix_recv_pkts(rx_queue, rx_pkts, pkts, (flags)); \ + } + +#define NIX_RX_RECV_MSEG(fn, flags) NIX_RX_RECV(fn, flags | NIX_RX_MULTI_SEG_F) + +#define NIX_RX_RECV_VEC(fn, flags) \ + uint16_t __rte_noinline __rte_hot fn( \ + void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ + { \ + return cn10k_nix_recv_pkts_vector(rx_queue, rx_pkts, pkts, \ + (flags), NULL, NULL, 0); \ + } + +#define NIX_RX_RECV_VEC_MSEG(fn, flags) \ + NIX_RX_RECV_VEC(fn, flags | NIX_RX_MULTI_SEG_F) + #endif /* __CN10K_RX_H__ */ diff --git a/drivers/net/cnxk/cn10k_rx_mseg.c b/drivers/net/cnxk/cn10k_rx_mseg.c deleted file mode 100644 index e7c2321f6e..0000000000 --- a/drivers/net/cnxk/cn10k_rx_mseg.c +++ /dev/null @@ -1,17 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(C) 2021 Marvell. - */ - -#include "cn10k_ethdev.h" -#include "cn10k_rx.h" - -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_mseg_##name( \ - void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ - { \ - return cn10k_nix_recv_pkts(rx_queue, rx_pkts, pkts, \ - (flags) | NIX_RX_MULTI_SEG_F); \ - } - -NIX_RX_FASTPATH_MODES -#undef R diff --git a/drivers/net/cnxk/cn10k_rx_select.c b/drivers/net/cnxk/cn10k_rx_select.c new file mode 100644 index 0000000000..da9aa81fbb --- /dev/null +++ b/drivers/net/cnxk/cn10k_rx_select.c @@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cn10k_ethdev.h" +#include "cn10k_rx.h" + +static inline void +pick_rx_func(struct rte_eth_dev *eth_dev, + const eth_rx_burst_t rx_burst[NIX_RX_OFFLOAD_MAX]) +{ + struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + + /* [VLAN] [TSP] [MARK] [CKSUM] [PTYPE] [RSS] */ + eth_dev->rx_pkt_burst = + rx_burst[dev->rx_offload_flags & (NIX_RX_OFFLOAD_MAX - 1)]; + + rte_atomic_thread_fence(__ATOMIC_RELEASE); +} + +void +cn10k_eth_set_rx_function(struct rte_eth_dev *eth_dev) +{ + struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + + const eth_rx_burst_t nix_eth_rx_burst[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_nix_recv_pkts_##name, + + NIX_RX_FASTPATH_MODES +#undef R + }; + + const eth_rx_burst_t nix_eth_rx_burst_mseg[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_nix_recv_pkts_mseg_##name, + + NIX_RX_FASTPATH_MODES +#undef R + }; + + const eth_rx_burst_t nix_eth_rx_vec_burst[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_nix_recv_pkts_vec_##name, + + NIX_RX_FASTPATH_MODES +#undef R + }; + + const eth_rx_burst_t nix_eth_rx_vec_burst_mseg[NIX_RX_OFFLOAD_MAX] = { +#define R(name, flags) [flags] = cn10k_nix_recv_pkts_vec_mseg_##name, + + NIX_RX_FASTPATH_MODES +#undef R + }; + + /* Copy multi seg version with no offload for tear down sequence */ + if (rte_eal_process_type() == RTE_PROC_PRIMARY) + dev->rx_pkt_burst_no_offload = nix_eth_rx_burst_mseg[0]; + + if (dev->scalar_ena) { + if (dev->rx_offloads & DEV_RX_OFFLOAD_SCATTER) + return pick_rx_func(eth_dev, nix_eth_rx_burst_mseg); + return pick_rx_func(eth_dev, nix_eth_rx_burst); + } + + if (dev->rx_offloads & DEV_RX_OFFLOAD_SCATTER) + return pick_rx_func(eth_dev, nix_eth_rx_vec_burst_mseg); + return pick_rx_func(eth_dev, nix_eth_rx_vec_burst); +} diff --git a/drivers/net/cnxk/cn10k_rx_vec.c b/drivers/net/cnxk/cn10k_rx_vec.c deleted file mode 100644 index 0ccc4df8ac..0000000000 --- a/drivers/net/cnxk/cn10k_rx_vec.c +++ /dev/null @@ -1,19 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(C) 2021 Marvell. - */ - -#include "cn10k_ethdev.h" -#include "cn10k_rx.h" - -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - uint16_t __rte_noinline __rte_hot \ - cn10k_nix_recv_pkts_vec_##name(void *rx_queue, \ - struct rte_mbuf **rx_pkts, \ - uint16_t pkts) \ - { \ - return cn10k_nix_recv_pkts_vector(rx_queue, rx_pkts, pkts, \ - (flags), NULL, NULL, 0); \ - } - -NIX_RX_FASTPATH_MODES -#undef R diff --git a/drivers/net/cnxk/cn10k_rx_vec_mseg.c b/drivers/net/cnxk/cn10k_rx_vec_mseg.c deleted file mode 100644 index 38e0ec36a3..0000000000 --- a/drivers/net/cnxk/cn10k_rx_vec_mseg.c +++ /dev/null @@ -1,18 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(C) 2021 Marvell. - */ - -#include "cn10k_ethdev.h" -#include "cn10k_rx.h" - -#define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ - uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_vec_mseg_##name( \ - void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ - { \ - return cn10k_nix_recv_pkts_vector( \ - rx_queue, rx_pkts, pkts, (flags) | NIX_RX_MULTI_SEG_F, \ - NULL, NULL, 0); \ - } - -NIX_RX_FASTPATH_MODES -#undef R diff --git a/drivers/net/cnxk/meson.build b/drivers/net/cnxk/meson.build index 01bd5442b5..1fd388a1f0 100644 --- a/drivers/net/cnxk/meson.build +++ b/drivers/net/cnxk/meson.build @@ -107,14 +107,46 @@ sources += files( 'cn10k_ethdev.c', 'cn10k_ethdev_sec.c', 'cn10k_rte_flow.c', - 'cn10k_rx.c', - 'cn10k_rx_mseg.c', - 'cn10k_rx_vec.c', - 'cn10k_rx_vec_mseg.c', 'cn10k_tx.c', 'cn10k_tx_mseg.c', 'cn10k_tx_vec.c', 'cn10k_tx_vec_mseg.c', + 'cn10k_rx_select.c', +) + +sources += files( + 'rx/cn10k/rx_0_15.c', + 'rx/cn10k/rx_16_31.c', + 'rx/cn10k/rx_32_47.c', + 'rx/cn10k/rx_48_63.c', + 'rx/cn10k/rx_64_79.c', + 'rx/cn10k/rx_80_95.c', + 'rx/cn10k/rx_96_111.c', + 'rx/cn10k/rx_112_127.c', + 'rx/cn10k/rx_0_15_mseg.c', + 'rx/cn10k/rx_16_31_mseg.c', + 'rx/cn10k/rx_32_47_mseg.c', + 'rx/cn10k/rx_48_63_mseg.c', + 'rx/cn10k/rx_64_79_mseg.c', + 'rx/cn10k/rx_80_95_mseg.c', + 'rx/cn10k/rx_96_111_mseg.c', + 'rx/cn10k/rx_112_127_mseg.c', + 'rx/cn10k/rx_0_15_vec.c', + 'rx/cn10k/rx_16_31_vec.c', + 'rx/cn10k/rx_32_47_vec.c', + 'rx/cn10k/rx_48_63_vec.c', + 'rx/cn10k/rx_64_79_vec.c', + 'rx/cn10k/rx_80_95_vec.c', + 'rx/cn10k/rx_96_111_vec.c', + 'rx/cn10k/rx_112_127_vec.c', + 'rx/cn10k/rx_0_15_vec_mseg.c', + 'rx/cn10k/rx_16_31_vec_mseg.c', + 'rx/cn10k/rx_32_47_vec_mseg.c', + 'rx/cn10k/rx_48_63_vec_mseg.c', + 'rx/cn10k/rx_64_79_vec_mseg.c', + 'rx/cn10k/rx_80_95_vec_mseg.c', + 'rx/cn10k/rx_96_111_vec_mseg.c', + 'rx/cn10k/rx_112_127_vec_mseg.c', ) deps += ['bus_pci', 'cryptodev', 'eventdev', 'security']