From patchwork Mon Mar 16 15:29:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Harman Kalra X-Patchwork-Id: 66718 X-Patchwork-Delegate: jerinj@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id DBA0BA0559; Mon, 16 Mar 2020 16:30:11 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C58B51C069; Mon, 16 Mar 2020 16:30:01 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id 663CB1C028 for ; Mon, 16 Mar 2020 16:29:59 +0100 (CET) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 02GFKT78029483 for ; Mon, 16 Mar 2020 08:29:58 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=pfpt0818; bh=SMGPXcmtytl8D1IbwK3rv2ieYf/1tElx0Ca9LU3KZ9E=; b=kG8zs1qffPek4oMDI0f9h4BJ8XLG5RVYji9FoDZFGUJsz+VIUjr9IoAfC3OicY/wwJLT TIFs36VxRI79aJl/gVLXRNbhv4GJQI3WaAvPTujb2VTlsFex7F33Q+oko5yZxsa/klVT GUQ6xCa32Cqvk4/cEqXZPaBWEc4qIc3Qs6WHFYZksSHabSxqguKnxdZPqq6M+ou0r5et zIERt9Ya3aTX+GvuWfC2Yoq2tDShJrJUycNA4WCofK2G2ITQKOft2pZZ5xePCxGKWdXP sMn8xfrYCIwEarfglDlVxV4vOtZL3Fl83mfD2mWpRAeodt46G2eCqsBnmn/Npwbvkl/M yg== Received: from sc-exch01.marvell.com ([199.233.58.181]) by mx0b-0016f401.pphosted.com with ESMTP id 2yrxsmf9dv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Mon, 16 Mar 2020 08:29:58 -0700 Received: from SC-EXCH04.marvell.com (10.93.176.84) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 16 Mar 2020 08:29:56 -0700 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.177) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Mon, 16 Mar 2020 08:29:55 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Uo3DFWJo5EhGce4AZI6T1eZ6wbdHHSVyC5T4xsb28uNjJALRFk5dzPRouNNgY2b8y9dV8vxUTPjoqoeAvBWDKJfGCTUXskE5pAVkPUK0fnA4UKAfzsh358hMnvIp2Qv/cLBKsg0jozGIOsnvzzOIW6R1IcFoZMNc8xMhPr2ImOmY4/ThssYjL0PymEWFHF/DV1rKQFKpTYbsakzoUPz/y2xZKHyigVov3nFxCb5ZSEkIBhZAIc7GxU6zmvIft/2HF/xsE2oMSxJT5Ioii51L8g45KYYHgWECY51xIG9w2BCVNUEa8ZCSS7mb5V0hK1q9G4uV9unBJkddBq48exPPxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SMGPXcmtytl8D1IbwK3rv2ieYf/1tElx0Ca9LU3KZ9E=; b=RG7f7CprK4oXnUuhVPWQnxj92QVi2yNsIFSi+6P3puCAP2KtqzVBZ7xlhy3owzNQC5QJA5X4rKEwI+5RakYCHbEGdq5IPnUOdvYoHBvxpo5P0ONasvpCH/IfIFgaKo35hY4HrrGulonCkFo0O8IqNDONxDA7Tz6uJyn7pviBrU7KzhigtDAup6lpApuj2hFJbfZGCID19C1ZcOTy0kGIonhK8XHA4/pY9hAr3xKrosNwwUvvC5Tuf7IFRLNwBWQncwmBwl1NpzTF9+VvbrDYAufdxbrTkQFb3ruofQ85+2t4EgHtv0vCyNVkGCCs2jYvTflGJmL2O1t1xj3YFUYHug== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector1-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SMGPXcmtytl8D1IbwK3rv2ieYf/1tElx0Ca9LU3KZ9E=; b=VKck1k37VBoP7uDABMwWkYRBy7YuAp9gg9HBtEa0u/vJ+UYLosOO9T5M7bmLFQjiZVa/VBTMiSjhQsCW6fw3zNiiFPpAZSQHsdbqIJLr2jiwQaCpW0u2ABxW/s6tBSSClyjr3K4w/UebM5MUUVuI/k6o3rRnj/fgTvRfU3TqGyU= Received: from MN2PR18MB2848.namprd18.prod.outlook.com (2603:10b6:208:38::19) by MN2PR18MB3592.namprd18.prod.outlook.com (2603:10b6:208:26c::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2814.19; Mon, 16 Mar 2020 15:29:55 +0000 Received: from MN2PR18MB2848.namprd18.prod.outlook.com ([fe80::f829:3e55:94f6:4efb]) by MN2PR18MB2848.namprd18.prod.outlook.com ([fe80::f829:3e55:94f6:4efb%5]) with mapi id 15.20.2814.021; Mon, 16 Mar 2020 15:29:55 +0000 From: Harman Kalra To: CC: , , Harman Kalra Date: Mon, 16 Mar 2020 20:59:10 +0530 Message-ID: <1584372553-28710-2-git-send-email-hkalra@marvell.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1584372553-28710-1-git-send-email-hkalra@marvell.com> References: <1584372553-28710-1-git-send-email-hkalra@marvell.com> X-ClientProxiedBy: BM1PR0101CA0036.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:1a::22) To MN2PR18MB2848.namprd18.prod.outlook.com (2603:10b6:208:38::19) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from hkarlara-OptiPlex-3046.marvell.com (115.113.156.2) by BM1PR0101CA0036.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:1a::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.2814.13 via Frontend Transport; Mon, 16 Mar 2020 15:29:53 +0000 X-Mailer: git-send-email 2.7.4 X-Originating-IP: [115.113.156.2] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 282f945b-092a-478c-5778-08d7c9bee0be X-MS-TrafficTypeDiagnostic: MN2PR18MB3592: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:312; X-Forefront-PRVS: 03449D5DD1 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4636009)(366004)(39850400004)(136003)(376002)(346002)(396003)(199004)(66476007)(30864003)(956004)(478600001)(66556008)(186003)(86362001)(26005)(16526019)(36756003)(66946007)(6486002)(4326008)(6636002)(316002)(55236004)(7696005)(37006003)(52116002)(107886003)(2616005)(8676002)(8936002)(2906002)(81166006)(5660300002)(81156014)(34206002)(6666004); DIR:OUT; SFP:1101; SCL:1; SRVR:MN2PR18MB3592; H:MN2PR18MB2848.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; Received-SPF: None (protection.outlook.com: marvell.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1qL5ajNPaVFXOriQPiiik7L68eiahS3Wcw6pS1RIuPXH5WiPPqmyIyOT4ADwv5McJ6p/7poeQCLnOt/FWWIpkIXLMha0ebkgWDTL08o40EcHHaz47xV1KDlPv7YZ3URhy4a1FEGIwk7g3aPcb47dQrztVeimzgejuXVNE2gy6k92AANGe/X7HpfSB+J2VOJipS1AFbBfijZk2ZfIhLMjAMw/CVC22xOVwPejwo66/WKt7qouaJICJoZuniEcOXX01jEh4QasTepqDMeqqgldwQ9hhxh0RsbjghaBh3vlaSPuVgCmCQCgmVM3FMsubkbW6dfXJ8OTXARVrmDbE2F0VyslxeIVtbDHfqMOMDfa2ys4L9VQzMhgzNJFLTvBZvP3wurgwJz5j2NIUvgNnha7wkamhgS2OsWQwG3c/VMj5rAVG5bJOA+hoaga8BA7yErq X-MS-Exchange-AntiSpam-MessageData: Qh+5L+3rnn9iAqkULqBPoZphgEeGc71UJ+FfnOJcvA92jhndxPb67O+dKSpcmfTFwwZ9/FsMHGTGCp46rCSQLbm6YElT73aA/sAiDc1epsJ/iNq3bCN+HPDMFfxlsWwjAIL1V+XI+CWxoZ8YiI5x7Q== X-MS-Exchange-CrossTenant-Network-Message-Id: 282f945b-092a-478c-5778-08d7c9bee0be X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2020 15:29:54.9113 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MhRMr2wrZg197bKokYvN4fS9QiyzT+urnVRLJwkP75xxHQaipkIhYdydaOPqekml7tQnQxSZsO67kfBvBmCpvQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR18MB3592 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.645 definitions=2020-03-16_06:2020-03-12, 2020-03-16 signatures=0 Subject: [dpdk-dev] [PATCH 1/4] event/octeontx: add multi segment support to eventdev X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Adding support for multi segment to the eventdev PMD. Signed-off-by: Harman Kalra --- drivers/event/octeontx/ssovf_evdev.c | 33 +++++++--- drivers/event/octeontx/ssovf_evdev.h | 13 ++++ drivers/event/octeontx/ssovf_worker.c | 90 ++++++++++++++++++++++++--- drivers/event/octeontx/ssovf_worker.h | 76 +++++++++++++++++++--- 4 files changed, 189 insertions(+), 23 deletions(-) diff --git a/drivers/event/octeontx/ssovf_evdev.c b/drivers/event/octeontx/ssovf_evdev.c index f9e93244f..1024b7284 100644 --- a/drivers/event/octeontx/ssovf_evdev.c +++ b/drivers/event/octeontx/ssovf_evdev.c @@ -146,15 +146,31 @@ ssovf_fastpath_fns_set(struct rte_eventdev *dev) dev->enqueue_burst = ssows_enq_burst; dev->enqueue_new_burst = ssows_enq_new_burst; dev->enqueue_forward_burst = ssows_enq_fwd_burst; - dev->dequeue = ssows_deq; - dev->dequeue_burst = ssows_deq_burst; - dev->txa_enqueue = sso_event_tx_adapter_enqueue; - dev->txa_enqueue_same_dest = dev->txa_enqueue; - if (edev->is_timeout_deq) { - dev->dequeue = ssows_deq_timeout; - dev->dequeue_burst = ssows_deq_timeout_burst; + if (!!(edev->rx_offload_flags & OCCTX_RX_MULTI_SEG_F)) { + dev->dequeue = ssows_deq_mseg; + dev->dequeue_burst = ssows_deq_burst_mseg; + + if (edev->is_timeout_deq) { + dev->dequeue = ssows_deq_timeout_mseg; + dev->dequeue_burst = ssows_deq_timeout_burst_mseg; + } + } else { + dev->dequeue = ssows_deq; + dev->dequeue_burst = ssows_deq_burst; + + if (edev->is_timeout_deq) { + dev->dequeue = ssows_deq_timeout; + dev->dequeue_burst = ssows_deq_timeout_burst; + } } + + if (!!(edev->tx_offload_flags & OCCTX_TX_MULTI_SEG_F)) + dev->txa_enqueue = sso_event_tx_adapter_enqueue_mseg; + else + dev->txa_enqueue = sso_event_tx_adapter_enqueue; + + dev->txa_enqueue_same_dest = dev->txa_enqueue; } static void @@ -411,6 +427,7 @@ ssovf_eth_rx_adapter_queue_add(const struct rte_eventdev *dev, { int ret = 0; const struct octeontx_nic *nic = eth_dev->data->dev_private; + struct ssovf_evdev *edev = ssovf_pmd_priv(dev); pki_mod_qos_t pki_qos; RTE_SET_USED(dev); @@ -447,6 +464,8 @@ ssovf_eth_rx_adapter_queue_add(const struct rte_eventdev *dev, ssovf_log_err("failed to modify QOS, port=%d, q=%d", nic->port_id, queue_conf->ev.queue_id); + edev->rx_offload_flags = nic->rx_offload_flags; + edev->tx_offload_flags = nic->tx_offload_flags; return ret; } diff --git a/drivers/event/octeontx/ssovf_evdev.h b/drivers/event/octeontx/ssovf_evdev.h index 0e622152c..1c3ae8556 100644 --- a/drivers/event/octeontx/ssovf_evdev.h +++ b/drivers/event/octeontx/ssovf_evdev.h @@ -12,6 +12,8 @@ #include #include +#include "octeontx_rxtx.h" + #define EVENTDEV_NAME_OCTEONTX_PMD event_octeontx #define SSOVF_LOG(level, fmt, args...) \ @@ -132,6 +134,7 @@ enum ssovf_type { }; struct ssovf_evdev { + OFFLOAD_FLAGS; /*Sequence should not be changed */ uint8_t max_event_queues; uint8_t max_event_ports; uint8_t is_timeout_deq; @@ -175,6 +178,14 @@ uint16_t ssows_deq_timeout(void *port, struct rte_event *ev, uint64_t timeout_ticks); uint16_t ssows_deq_timeout_burst(void *port, struct rte_event ev[], uint16_t nb_events, uint64_t timeout_ticks); +uint16_t ssows_deq_mseg(void *port, struct rte_event *ev, + uint64_t timeout_ticks); +uint16_t ssows_deq_burst_mseg(void *port, struct rte_event ev[], + uint16_t nb_events, uint64_t timeout_ticks); +uint16_t ssows_deq_timeout_mseg(void *port, struct rte_event *ev, + uint64_t timeout_ticks); +uint16_t ssows_deq_timeout_burst_mseg(void *port, struct rte_event ev[], + uint16_t nb_events, uint64_t timeout_ticks); typedef void (*ssows_handle_event_t)(void *arg, struct rte_event ev); void ssows_flush_events(struct ssows *ws, uint8_t queue_id, @@ -182,6 +193,8 @@ void ssows_flush_events(struct ssows *ws, uint8_t queue_id, void ssows_reset(struct ssows *ws); uint16_t sso_event_tx_adapter_enqueue(void *port, struct rte_event ev[], uint16_t nb_events); +uint16_t sso_event_tx_adapter_enqueue_mseg(void *port, + struct rte_event ev[], uint16_t nb_events); int ssovf_info(struct ssovf_info *info); void *ssovf_bar(enum ssovf_type, uint8_t id, uint8_t bar); int test_eventdev_octeontx(void); diff --git a/drivers/event/octeontx/ssovf_worker.c b/drivers/event/octeontx/ssovf_worker.c index 169939d1a..951f66890 100644 --- a/drivers/event/octeontx/ssovf_worker.c +++ b/drivers/event/octeontx/ssovf_worker.c @@ -103,7 +103,7 @@ ssows_deq(void *port, struct rte_event *ev, uint64_t timeout_ticks) ssows_swtag_wait(ws); return 1; } else { - return ssows_get_work(ws, ev); + return ssows_get_work(ws, ev, OCCTX_RX_OFFLOAD_NONE); } } @@ -118,9 +118,9 @@ ssows_deq_timeout(void *port, struct rte_event *ev, uint64_t timeout_ticks) ws->swtag_req = 0; ssows_swtag_wait(ws); } else { - ret = ssows_get_work(ws, ev); + ret = ssows_get_work(ws, ev, OCCTX_RX_OFFLOAD_NONE); for (iter = 1; iter < timeout_ticks && (ret == 0); iter++) - ret = ssows_get_work(ws, ev); + ret = ssows_get_work(ws, ev, OCCTX_RX_OFFLOAD_NONE); } return ret; } @@ -143,6 +143,61 @@ ssows_deq_timeout_burst(void *port, struct rte_event ev[], uint16_t nb_events, return ssows_deq_timeout(port, ev, timeout_ticks); } +__rte_always_inline uint16_t __hot +ssows_deq_mseg(void *port, struct rte_event *ev, uint64_t timeout_ticks) +{ + struct ssows *ws = port; + + RTE_SET_USED(timeout_ticks); + + if (ws->swtag_req) { + ws->swtag_req = 0; + ssows_swtag_wait(ws); + return 1; + } else { + return ssows_get_work(ws, ev, OCCTX_RX_OFFLOAD_NONE | + OCCTX_RX_MULTI_SEG_F); + } +} + +__rte_always_inline uint16_t __hot +ssows_deq_timeout_mseg(void *port, struct rte_event *ev, uint64_t timeout_ticks) +{ + struct ssows *ws = port; + uint64_t iter; + uint16_t ret = 1; + + if (ws->swtag_req) { + ws->swtag_req = 0; + ssows_swtag_wait(ws); + } else { + ret = ssows_get_work(ws, ev, OCCTX_RX_OFFLOAD_NONE | + OCCTX_RX_MULTI_SEG_F); + for (iter = 1; iter < timeout_ticks && (ret == 0); iter++) + ret = ssows_get_work(ws, ev, OCCTX_RX_OFFLOAD_NONE | + OCCTX_RX_MULTI_SEG_F); + } + return ret; +} + +uint16_t __hot +ssows_deq_burst_mseg(void *port, struct rte_event ev[], uint16_t nb_events, + uint64_t timeout_ticks) +{ + RTE_SET_USED(nb_events); + + return ssows_deq_mseg(port, ev, timeout_ticks); +} + +uint16_t __hot +ssows_deq_timeout_burst_mseg(void *port, struct rte_event ev[], + uint16_t nb_events, uint64_t timeout_ticks) +{ + RTE_SET_USED(nb_events); + + return ssows_deq_timeout_mseg(port, ev, timeout_ticks); +} + __rte_always_inline uint16_t __hot ssows_enq(void *port, const struct rte_event *ev) { @@ -231,7 +286,9 @@ ssows_flush_events(struct ssows *ws, uint8_t queue_id, ev.event = sched_type_queue | (get_work0 & 0xffffffff); if (get_work1 && ev.event_type == RTE_EVENT_TYPE_ETHDEV) ev.mbuf = ssovf_octeontx_wqe_to_pkt(get_work1, - (ev.event >> 20) & 0x7F); + (ev.event >> 20) & 0x7F, + OCCTX_RX_OFFLOAD_NONE | + OCCTX_RX_MULTI_SEG_F); else ev.u64 = get_work1; @@ -262,9 +319,9 @@ ssows_reset(struct ssows *ws) } } -uint16_t -sso_event_tx_adapter_enqueue(void *port, - struct rte_event ev[], uint16_t nb_events) +static __rte_always_inline uint16_t +__sso_event_tx_adapter_enqueue(void *port, struct rte_event ev[], + uint16_t nb_events, const uint16_t flag) { uint16_t port_id; uint16_t queue_id; @@ -298,5 +355,22 @@ sso_event_tx_adapter_enqueue(void *port, ethdev = &rte_eth_devices[port_id]; txq = ethdev->data->tx_queues[queue_id]; - return __octeontx_xmit_pkts(txq, &m, 1, cmd, OCCTX_TX_OFFLOAD_NONE); + return __octeontx_xmit_pkts(txq, &m, 1, cmd, flag); +} + +uint16_t +sso_event_tx_adapter_enqueue(void *port, struct rte_event ev[], + uint16_t nb_events) +{ + return __sso_event_tx_adapter_enqueue(port, ev, nb_events, + OCCTX_TX_OFFLOAD_NONE); +} + +uint16_t +sso_event_tx_adapter_enqueue_mseg(void *port, struct rte_event ev[], + uint16_t nb_events) +{ + return __sso_event_tx_adapter_enqueue(port, ev, nb_events, + OCCTX_TX_OFFLOAD_NONE | + OCCTX_TX_MULTI_SEG_F); } diff --git a/drivers/event/octeontx/ssovf_worker.h b/drivers/event/octeontx/ssovf_worker.h index c4f886d63..61332304a 100644 --- a/drivers/event/octeontx/ssovf_worker.h +++ b/drivers/event/octeontx/ssovf_worker.h @@ -23,8 +23,45 @@ enum { /* SSO Operations */ +static __rte_always_inline void +ssovf_octeontx_wqe_xtract_mseg(octtx_wqe_t *wqe, + struct rte_mbuf *mbuf) +{ + octtx_pki_buflink_t *buflink; + rte_iova_t *iova_list; + uint8_t nb_segs; + uint64_t bytes_left = wqe->s.w1.len - wqe->s.w5.size; + + nb_segs = wqe->s.w0.bufs; + + buflink = (octtx_pki_buflink_t *)((uintptr_t)wqe->s.w3.addr - + sizeof(octtx_pki_buflink_t)); + + /*TODO: work with rearm data */ + + while (--nb_segs) { + iova_list = (rte_iova_t *)(uintptr_t)(buflink->w1.s.addr); + mbuf->next = (struct rte_mbuf *)(rte_iova_t *)(iova_list - 2) + - (OCTTX_PACKET_LATER_SKIP / 128); + mbuf = mbuf->next; + + mbuf->data_off = sizeof(octtx_pki_buflink_t); + + __mempool_check_cookies(mbuf->pool, (void **)&mbuf, 1, 1); + if (nb_segs == 1) + mbuf->data_len = bytes_left; + else + mbuf->data_len = buflink->w0.s.size; + + bytes_left = bytes_left - buflink->w0.s.size; + buflink = (octtx_pki_buflink_t *)(rte_iova_t *)(iova_list - 2); + + } +} + static __rte_always_inline struct rte_mbuf * -ssovf_octeontx_wqe_to_pkt(uint64_t work, uint16_t port_info) +ssovf_octeontx_wqe_to_pkt(uint64_t work, uint16_t port_info, + const uint16_t flag) { struct rte_mbuf *mbuf; octtx_wqe_t *wqe = (octtx_wqe_t *)(uintptr_t)work; @@ -35,10 +72,18 @@ ssovf_octeontx_wqe_to_pkt(uint64_t work, uint16_t port_info) mbuf->packet_type = ptype_table[wqe->s.w2.lcty][wqe->s.w2.lety][wqe->s.w2.lfty]; mbuf->data_off = RTE_PTR_DIFF(wqe->s.w3.addr, mbuf->buf_addr); - mbuf->pkt_len = wqe->s.w1.len; - mbuf->data_len = mbuf->pkt_len; - mbuf->nb_segs = 1; mbuf->ol_flags = 0; + mbuf->pkt_len = wqe->s.w1.len; + + if (!!(flag & OCCTX_RX_MULTI_SEG_F)) { + mbuf->nb_segs = wqe->s.w0.bufs; + mbuf->data_len = wqe->s.w5.size; + ssovf_octeontx_wqe_xtract_mseg(wqe, mbuf); + } else { + mbuf->nb_segs = 1; + mbuf->data_len = mbuf->pkt_len; + } + mbuf->port = rte_octeontx_pchan_map[port_info >> 4][port_info & 0xF]; rte_mbuf_refcnt_set(mbuf, 1); @@ -49,14 +94,29 @@ static __rte_always_inline void ssovf_octeontx_wqe_free(uint64_t work) { octtx_wqe_t *wqe = (octtx_wqe_t *)(uintptr_t)work; - struct rte_mbuf *mbuf; + uint8_t nb_segs = wqe->s.w0.bufs; + octtx_pki_buflink_t *buflink; + struct rte_mbuf *mbuf, *head; + rte_iova_t *iova_list; mbuf = (struct rte_mbuf *)((uintptr_t)wqe - OCTTX_PACKET_WQE_SKIP); - rte_pktmbuf_free(mbuf); + buflink = (octtx_pki_buflink_t *)((uintptr_t)wqe->s.w3.addr - + sizeof(octtx_pki_buflink_t)); + head = mbuf; + while (--nb_segs) { + iova_list = (rte_iova_t *)(uintptr_t)(buflink->w1.s.addr); + mbuf = (struct rte_mbuf *)(rte_iova_t *)(iova_list - 2) + - (OCTTX_PACKET_LATER_SKIP / 128); + + mbuf->next = NULL; + rte_pktmbuf_free(mbuf); + buflink = (octtx_pki_buflink_t *)(rte_iova_t *)(iova_list - 2); + } + rte_pktmbuf_free(head); } static __rte_always_inline uint16_t -ssows_get_work(struct ssows *ws, struct rte_event *ev) +ssows_get_work(struct ssows *ws, struct rte_event *ev, const uint16_t flag) { uint64_t get_work0, get_work1; uint64_t sched_type_queue; @@ -71,7 +131,7 @@ ssows_get_work(struct ssows *ws, struct rte_event *ev) if (get_work1 && ev->event_type == RTE_EVENT_TYPE_ETHDEV) { ev->mbuf = ssovf_octeontx_wqe_to_pkt(get_work1, - (ev->event >> 20) & 0x7F); + (ev->event >> 20) & 0x7F, flag); } else if (unlikely((get_work0 & 0xFFFFFFFF) == 0xFFFFFFFF)) { ssovf_octeontx_wqe_free(get_work1); return 0;