From patchwork Mon Jan 20 13:45:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anoob Joseph X-Patchwork-Id: 64919 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 867BAA0528; Mon, 20 Jan 2020 14:45:48 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8B3EA11A4; Mon, 20 Jan 2020 14:45:47 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by dpdk.org (Postfix) with ESMTP id 2D0B411A2 for ; Mon, 20 Jan 2020 14:45:45 +0100 (CET) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 00KDjii5018825; Mon, 20 Jan 2020 05:45:44 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0818; bh=tme/PvkavprbRAv2d8PE7mwtpwT/QVLyY89PLNlnCU4=; b=aKK+aGfDD8s5JMh8KNrYwMXj8DjJE1h7vDMFMAVPv57C/JdQMOmkXkaUyQ6jv4pzRE0Q aqKQbXdYjDOVylZLa8Q6s5DIKwE5ISFGWVK/GJOh6UVJazmvPFerSds9sot/p2mGinsi MrfSO27GIYrppTA5BebPcMjekxIYyDHCzpvmCJt257DzNAj2HwgFSMA0Wg6nzGYa+7l8 nL8eE0uEB2UR5X0QEitgCcmRfCnomFCTFkc47FHzZFAdXKxAGQ7kAiC01hT5xpBmLn9S TdxUeh2XK6kTdttO1+kcRVEXcgxYrhB0gAKk44DKyeOdOcij8IpAxctzUhTcZ7cihes7 wQ== Received: from sc-exch02.marvell.com ([199.233.58.182]) by mx0a-0016f401.pphosted.com with ESMTP id 2xm08v6jey-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Mon, 20 Jan 2020 05:45:44 -0800 Received: from SC-EXCH01.marvell.com (10.93.176.81) by SC-EXCH02.marvell.com (10.93.176.82) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 20 Jan 2020 05:45:41 -0800 Received: from maili.marvell.com (10.93.176.43) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Mon, 20 Jan 2020 05:45:41 -0800 Received: from ajoseph83.caveonetworks.com (unknown [10.29.45.60]) by maili.marvell.com (Postfix) with ESMTP id D2C533F7040; Mon, 20 Jan 2020 05:45:37 -0800 (PST) From: Anoob Joseph To: Akhil Goyal , Radu Nicolau , Thomas Monjalon CC: Anoob Joseph , Jerin Jacob , Narayana Prasad , Ankur Dwivedi , Archana Muniganti , Tejasree Kondoj , Vamsi Attunuru , "Lukasz Bartosik" , Konstantin Ananyev , Date: Mon, 20 Jan 2020 19:15:06 +0530 Message-ID: <1579527918-360-1-git-send-email-anoobj@marvell.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1575808249-31135-1-git-send-email-anoobj@marvell.com> References: <1575808249-31135-1-git-send-email-anoobj@marvell.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.572 definitions=2020-01-20_02:2020-01-20, 2020-01-20 signatures=0 Subject: [dpdk-dev] [PATCH v2 00/12] add eventmode to ipsec-secgw X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This series introduces event-mode additions to ipsec-secgw. This effort is parallel to the similar changes in l2fwd (l2fwd-event app) & l3fwd. With this series, ipsec-secgw would be able to run in eventmode. The worker thread (executing loop) would be receiving events and would be submitting it back to the eventdev after the processing. This way, multicore scaling and h/w assisted scheduling is achieved by making use of the eventdev capabilities. Since the underlying event device would be having varying capabilities, the worker thread could be drafted differently to maximize performance. This series introduces usage of multiple worker threads, among which the one to be used will be determined by the operating conditions and the underlying device capabilities. For example, if an event device - eth device pair has Tx internal port, then application can do tx_adapter_enqueue() instead of regular event_enqueue(). So a thread making an assumption that the device pair has internal port will not be the right solution for another pair. The infrastructure added with these patches aims to help application to have multiple worker threads, there by extracting maximum performance from every device without affecting existing paths/use cases. The eventmode configuration is predefined. All packets reaching one eth port will hit one event queue. All event queues will be mapped to all event ports. So all cores will be able to receive traffic from all ports. When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event device will ensure the ordering. Ordering would be lost when tried in PARALLEL. Following command line options are introduced, --transfer-mode: to choose between poll mode & event mode --schedule-type: to specify the scheduling type (RTE_SCHED_TYPE_ORDERED/ RTE_SCHED_TYPE_ATOMIC/ RTE_SCHED_TYPE_PARALLEL) Additionally the event mode introduces two modes of processing packets: Driver-mode: This mode will have bare minimum changes in the application to support ipsec. There woudn't be any lookup etc done in the application. And for inline-protocol use case, the thread would resemble l2fwd as the ipsec processing would be done entirely in the h/w. This mode can be used to benchmark the raw performance of the h/w. All the application side steps (like lookup) can be redone based on the requirement of the end user. Hence the need for a mode which would report the raw performance. App-mode: This mode will have all the features currently implemented with ipsec-secgw (non librte_ipsec mode). All the lookups etc would follow the existing methods and would report numbers that can be compared against regular ipsec-secgw benchmark numbers. The driver mode is selected with existing --single-sa option (used also by poll mode). When --single-sa option is used in conjution with event mode then index passed to --single-sa is ignored. Example commands to execute ipsec-secgw in various modes on OCTEON TX2 platform, #Inbound and outbound app mode ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x1 -- -P -p 0x3 -u 0x1 --config "(1,0,0),(0,0,0)" -f aes-gcm.cfg --transfer-mode event --schedule-type parallel #Inbound and outbound driver mode ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x1 -- -P -p 0x3 -u 0x1 --config "(1,0,0),(0,0,0)" -f aes-gcm.cfg --transfer-mode event --schedule-type parallel --single-sa 0 This series adds non burst tx internal port workers only. It provides infrastructure for non internal port workers, however does not define any. Also, only inline ipsec protocol mode is supported by the worker threads added. Following are planned features, 1. Add burst mode workers. 2. Add non internal port workers. 3. Verify support for Rx core (the support is added but lack of h/w to verify). 4. Add lookaside protocol support. Following are features that Marvell won't be attempting. 1. Inline crypto support. 2. Lookaside crypto support. For the features that Marvell won't be attempting, new workers can be introduced by the respective stake holders. This series is tested on Marvell OCTEON TX2. Changes in v2: * Remove --process-dir option. Instead use existing unprotected port mask option (-u) to decide wheter port handles inbound or outbound traffic. * Remove --process-mode option. Instead use existing --single-sa option to select between app and driver modes. * Add handling of PKT_RX_SEC_OFFLOAD_FAIL result in app worker thread. * Fix passing of req_rx_offload flags to create_default_ipsec_flow(). * Move destruction of flows to a location where eth ports are stopped and closed. * Print error and exit when event mode --schedule-type option is used in poll mode. * Reduce number of goto statements replacing them with loop constructs. * Remove sec_session_fixed table and replace it with locally build table in driver worker thread. Table is indexed by port identifier and holds first inline session pointer found for a given port. * Print error and exit when sessions other than inline are configured in event mode. * When number of event queues is less than number of eth ports then map all eth ports to one event queue. * Cleanup and minor improvements in code as suggested by Konstantin Deferred to v3: * The final patch updates the hardcoded number of buffers in a pool. Also, there was a discussion on the update of number of qp. Both the above can be handled properly, if we can remove the logic which limits one core to only use one crypto qp. If we can allow one qp per lcore_param, every eth queue can have it's own crypto qp and that would solve the requirements with OCTEON TX2 inline ipsec support as well. Patch with the mentioned change, http://patches.dpdk.org/patch/64408/ * Update ipsec-secgw documentation to describe the new options as well as event mode support. This series depends on the PMD changes submitted in the following set, http://patches.dpdk.org/project/dpdk/list/?series=8203 Ankur Dwivedi (1): examples/ipsec-secgw: add default rte_flow for inline Rx Anoob Joseph (5): examples/ipsec-secgw: add framework for eventmode helper examples/ipsec-secgw: add eventdev port-lcore link examples/ipsec-secgw: add Rx adapter support examples/ipsec-secgw: add Tx adapter support examples/ipsec-secgw: add routines to display config Lukasz Bartosik (6): examples/ipsec-secgw: add routines to launch workers examples/ipsec-secgw: add support for internal ports examples/ipsec-secgw: add eventmode to ipsec-secgw examples/ipsec-secgw: add driver mode worker examples/ipsec-secgw: add app mode worker examples/ipsec-secgw: add cmd line option for bufs examples/ipsec-secgw/Makefile | 2 + examples/ipsec-secgw/event_helper.c | 1714 +++++++++++++++++++++++++++++++++++ examples/ipsec-secgw/event_helper.h | 312 +++++++ examples/ipsec-secgw/ipsec-secgw.c | 502 ++++++++-- examples/ipsec-secgw/ipsec-secgw.h | 86 ++ examples/ipsec-secgw/ipsec.c | 7 + examples/ipsec-secgw/ipsec.h | 36 +- examples/ipsec-secgw/ipsec_worker.c | 656 ++++++++++++++ examples/ipsec-secgw/ipsec_worker.h | 39 + examples/ipsec-secgw/meson.build | 4 +- examples/ipsec-secgw/sa.c | 11 - 11 files changed, 3275 insertions(+), 94 deletions(-) create mode 100644 examples/ipsec-secgw/event_helper.c create mode 100644 examples/ipsec-secgw/event_helper.h create mode 100644 examples/ipsec-secgw/ipsec-secgw.h create mode 100644 examples/ipsec-secgw/ipsec_worker.c create mode 100644 examples/ipsec-secgw/ipsec_worker.h