Message ID | 1582820317-7333-1-git-send-email-lbartosik@marvell.com (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id D305BA055F; Thu, 27 Feb 2020 17:18:45 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7B6B737AF; Thu, 27 Feb 2020 17:18:45 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by dpdk.org (Postfix) with ESMTP id 35D7E2C4F for <dev@dpdk.org>; Thu, 27 Feb 2020 17:18:44 +0100 (CET) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 01RFtlO8011073; Thu, 27 Feb 2020 08:18:43 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0818; bh=6ZUXVaWofd4mY3TiONf64QaHyQgfJrztRySizMQOk+w=; b=EgUJXT0xwvKi8cuDaMkHA4mUEiJ++TQgr129/vCvOBCQ+nXsXfnthJli56ArnUSF/3Sq /lJqrwtHPNocAkupo1kvD2fHTm8T1iuVt0Nbh3bxw+JrkjCKvoKaDLiPE5IWqdK7Oo6E d7fr36AynCXZJLEWG2vpXqUBByXfrfQxsBU4/8bEWKiRI5lWgYBx/qo9H3PIFg7nWiv5 81MINaD/Nt5tS9gthFhbqQnTmgNDSwEZVtqiqzAqtw58fQC0Y00q/dtwV86xXdTUmFMj cA6hrTGUaLTkNeXqizE+24br6p0FH2zyvs914NNdhyQFMFsIUxXZ9yYvBXR/f9IcA4e9 Zw== Received: from sc-exch04.marvell.com ([199.233.58.184]) by mx0a-0016f401.pphosted.com with ESMTP id 2ydcm19h8j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Thu, 27 Feb 2020 08:18:43 -0800 Received: from SC-EXCH01.marvell.com (10.93.176.81) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 27 Feb 2020 08:18:41 -0800 Received: from maili.marvell.com (10.93.176.43) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 27 Feb 2020 08:18:41 -0800 Received: from luke.marvell.com (unknown [10.95.130.81]) by maili.marvell.com (Postfix) with ESMTP id 33BFD3F7041; Thu, 27 Feb 2020 08:18:38 -0800 (PST) From: Lukasz Bartosik <lbartosik@marvell.com> To: Akhil Goyal <akhil.goyal@nxp.com>, Radu Nicolau <radu.nicolau@intel.com>, Thomas Monjalon <thomas@monjalon.net> CC: Jerin Jacob <jerinj@marvell.com>, Narayana Prasad <pathreya@marvell.com>, Ankur Dwivedi <adwivedi@marvell.com>, Anoob Joseph <anoobj@marvell.com>, Archana Muniganti <marchana@marvell.com>, Tejasree Kondoj <ktejasree@marvell.com>, Vamsi Attunuru <vattunuru@marvell.com>, "Konstantin Ananyev" <konstantin.ananyev@intel.com>, <dev@dpdk.org> Date: Thu, 27 Feb 2020 17:18:22 +0100 Message-ID: <1582820317-7333-1-git-send-email-lbartosik@marvell.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1582185727-6749-1-git-send-email-lbartosik@marvell.com> References: <1582185727-6749-1-git-send-email-lbartosik@marvell.com> MIME-Version: 1.0 Content-Type: text/plain X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.572 definitions=2020-02-27_05:2020-02-26, 2020-02-27 signatures=0 Subject: [dpdk-dev] [PATCH v5 00/15] add eventmode to ipsec-secgw X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
Series |
add eventmode to ipsec-secgw
|
|
Message
Lukas Bartosik [C]
Feb. 27, 2020, 4:18 p.m. UTC
This series introduces event-mode additions to ipsec-secgw. With this series, ipsec-secgw would be able to run in eventmode. The worker thread (executing loop) would be receiving events and would be submitting it back to the eventdev after the processing. This way, multicore scaling and h/w assisted scheduling is achieved by making use of the eventdev capabilities. Since the underlying event device would be having varying capabilities, the worker thread could be drafted differently to maximize performance. This series introduces usage of multiple worker threads, among which the one to be used will be determined by the operating conditions and the underlying device capabilities. For example, if an event device - eth device pair has Tx internal port, then application can do tx_adapter_enqueue() instead of regular event_enqueue(). So a thread making an assumption that the device pair has internal port will not be the right solution for another pair. The infrastructure added with these patches aims to help application to have multiple worker threads, there by extracting maximum performance from every device without affecting existing paths/use cases. The eventmode configuration is predefined. All packets reaching one eth port will hit one event queue. All event queues will be mapped to all event ports. So all cores will be able to receive traffic from all ports. When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event device will ensure the ordering. Ordering would be lost when tried in PARALLEL. Following command line options are introduced, --transfer-mode: to choose between poll mode & event mode --event-schedule-type: to specify the scheduling type (RTE_SCHED_TYPE_ORDERED/ RTE_SCHED_TYPE_ATOMIC/ RTE_SCHED_TYPE_PARALLEL) Additionally the event mode introduces two modes of processing packets: Driver-mode: This mode will have bare minimum changes in the application to support ipsec. There woudn't be any lookup etc done in the application. And for inline-protocol use case, the thread would resemble l2fwd as the ipsec processing would be done entirely in the h/w. This mode can be used to benchmark the raw performance of the h/w. All the application side steps (like lookup) can be redone based on the requirement of the end user. Hence the need for a mode which would report the raw performance. App-mode: This mode will have all the features currently implemented with ipsec-secgw (non librte_ipsec mode). All the lookups etc would follow the existing methods and would report numbers that can be compared against regular ipsec-secgw benchmark numbers. The driver mode is selected with existing --single-sa option (used also by poll mode). When --single-sa option is used in conjution with event mode then index passed to --single-sa is ignored. Example commands to execute ipsec-secgw in various modes on OCTEON TX2 platform, #Inbound and outbound app mode ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg --transfer-mode event --event-schedule-type parallel #Inbound and outbound driver mode ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg --transfer-mode event --event-schedule-type parallel --single-sa 0 This series adds non burst tx internal port workers only. It provides infrastructure for non internal port workers, however does not define any. Also, only inline ipsec protocol mode is supported by the worker threads added. Following are planned features, 1. Add burst mode workers. 2. Add non internal port workers. 3. Verify support for Rx core (the support is added but lack of h/w to verify). 4. Add lookaside protocol support. Following are features that Marvell won't be attempting. 1. Inline crypto support. 2. Lookaside crypto support. For the features that Marvell won't be attempting, new workers can be introduced by the respective stake holders. This series is tested on Marvell OCTEON TX2. This series is targeted for 20.05 release. Changes in v5: * Rename function check_params() to check_poll_mode_params() and check_eh_conf() to check_event_mode_params() in order to make it clear what is their purpose. * Forbid usage of --config option in event mode. * Replace magic numbers on return with enum values in process_ipsec_ev_inbound() and process_ipsec_ev_outbound() functions. * Add session_priv_pool for both inbound and outbound configuration in ipsec_wrkr_non_burst_int_port_app_mode worker. * Add check of event type in ipsec_wrkr_non_burst_int_port_app_mode worker. * Update description of --config option in both ipsec-secgw help and documentation. Changes in v4: * Update ipsec-secgw documentation to describe the new options as well as event mode support. * In event mode reserve number of crypto queues equal to number of eth ports in order to meet inline protocol offload requirements. * Add calculate_nb_mbufs() function to calculate number of mbufs in a pool and include fragments table size into the calculation. * Move structures ipsec_xf and ipsec_sad to ipsec.h and remove static keyword from sa_out, nb_sa_out, sa_in and nb_sa_in in sa.c. * Update process_ipsec_ev_inbound(), process_ipsec_ev_outbound(), check_sp() and prepare_out_sessions_tbl() functions as a result of changes introduced by SAD feature. * Remove setting sa->cdev_id_qp to 0 in create_inline_session as sa_ctx is created with rte_zmalloc. * Minor cleanup enhancements: - In eh_set_default_conf_eventdev() function in event_helper.c put definition of int local vars in one line, remove invalid comment, put "eventdev_config->ev_queue_mode = RTE_EVENT_QUEUE_CFG_ALL_TYPES" in one line instead of two. - Remove extern "C" from event_helper.h. - Put local vars in reverse xmas tree order in eh_dev_has_rx_internal_port() and eh_dev_has_tx_internal_port() functions in event_helper.c. - Put #include <rte_bitmap.h> in alphabetical order in ipsec-secgw.c. - Move "extern volatile bool force_quit" and "#include <stdbool.h>" to ipsec-secgw.h, remove #include <rte_hash.h>. - Remove not needed includes in ipsec_worker.c. - Remove expired todo from ipsec_worker.h. Changes in v3: * Move eh_conf_init() and eh_conf_uninit() functions to event_helper.c including minor rework. * Rename --schedule-type option to --event-schedule-type. * Replace macro UNPROTECTED_PORT with static inline function is_unprotected_port(). * Move definitions of global variables used by multiple modules to .c files and add externs in .h headers. * Add eh_check_conf() which validates ipsec-secgw configuration for event mode. * Add dynamic calculation of number of buffers in a pool based on number of cores, ports and crypto queues. * Fix segmentation fault in event mode driver worker which happens when there are no inline outbound sessions configured. * Remove change related to updating number of crypto queues in cryptodevs_init(). The update of crypto queues will be handled in a separate patch. * Fix compilation error on 32-bit platforms by using userdata instead of udata64 from rte_mbuf. Changes in v2: * Remove --process-dir option. Instead use existing unprotected port mask option (-u) to decide wheter port handles inbound or outbound traffic. * Remove --process-mode option. Instead use existing --single-sa option to select between app and driver modes. * Add handling of PKT_RX_SEC_OFFLOAD_FAIL result in app worker thread. * Fix passing of req_rx_offload flags to create_default_ipsec_flow(). * Move destruction of flows to a location where eth ports are stopped and closed. * Print error and exit when event mode --schedule-type option is used in poll mode. * Reduce number of goto statements replacing them with loop constructs. * Remove sec_session_fixed table and replace it with locally build table in driver worker thread. Table is indexed by port identifier and holds first inline session pointer found for a given port. * Print error and exit when sessions other than inline are configured in event mode. * When number of event queues is less than number of eth ports then map all eth ports to one event queue. * Cleanup and minor improvements in code as suggested by Konstantin Ankur Dwivedi (1): examples/ipsec-secgw: add default rte flow for inline Rx Anoob Joseph (5): examples/ipsec-secgw: add framework for eventmode helper examples/ipsec-secgw: add eventdev port-lcore link examples/ipsec-secgw: add Rx adapter support examples/ipsec-secgw: add Tx adapter support examples/ipsec-secgw: add routines to display config Lukasz Bartosik (9): examples/ipsec-secgw: add routines to launch workers examples/ipsec-secgw: add support for internal ports examples/ipsec-secgw: add event helper config init/uninit examples/ipsec-secgw: add eventmode to ipsec-secgw examples/ipsec-secgw: add driver mode worker examples/ipsec-secgw: add app mode worker examples/ipsec-secgw: make number of buffers dynamic doc: add event mode support to ipsec-secgw examples/ipsec-secgw: reserve crypto queues in event mode doc/guides/sample_app_ug/ipsec_secgw.rst | 135 ++- examples/ipsec-secgw/Makefile | 2 + examples/ipsec-secgw/event_helper.c | 1812 ++++++++++++++++++++++++++++++ examples/ipsec-secgw/event_helper.h | 327 ++++++ examples/ipsec-secgw/ipsec-secgw.c | 506 +++++++-- examples/ipsec-secgw/ipsec-secgw.h | 88 ++ examples/ipsec-secgw/ipsec.c | 5 +- examples/ipsec-secgw/ipsec.h | 53 +- examples/ipsec-secgw/ipsec_worker.c | 649 +++++++++++ examples/ipsec-secgw/ipsec_worker.h | 41 + examples/ipsec-secgw/meson.build | 6 +- examples/ipsec-secgw/sa.c | 21 +- examples/ipsec-secgw/sad.h | 5 - 13 files changed, 3516 insertions(+), 134 deletions(-) create mode 100644 examples/ipsec-secgw/event_helper.c create mode 100644 examples/ipsec-secgw/event_helper.h create mode 100644 examples/ipsec-secgw/ipsec-secgw.h create mode 100644 examples/ipsec-secgw/ipsec_worker.c create mode 100644 examples/ipsec-secgw/ipsec_worker.h
Comments
Hi Akhil, Konstantin, Are there any more comments? Or can we have the patches merged? Thanks, Anoob > -----Original Message----- > From: Lukasz Bartosik <lbartosik@marvell.com> > Sent: Thursday, February 27, 2020 9:48 PM > To: Akhil Goyal <akhil.goyal@nxp.com>; Radu Nicolau > <radu.nicolau@intel.com>; Thomas Monjalon <thomas@monjalon.net> > Cc: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Narayana Prasad Raju > Athreya <pathreya@marvell.com>; Ankur Dwivedi <adwivedi@marvell.com>; > Anoob Joseph <anoobj@marvell.com>; Archana Muniganti > <marchana@marvell.com>; Tejasree Kondoj <ktejasree@marvell.com>; Vamsi > Krishna Attunuru <vattunuru@marvell.com>; Konstantin Ananyev > <konstantin.ananyev@intel.com>; dev@dpdk.org > Subject: [PATCH v5 00/15] add eventmode to ipsec-secgw > > This series introduces event-mode additions to ipsec-secgw. > > With this series, ipsec-secgw would be able to run in eventmode. The worker > thread (executing loop) would be receiving events and would be submitting it > back to the eventdev after the processing. This way, multicore scaling and h/w > assisted scheduling is achieved by making use of the eventdev capabilities. > > Since the underlying event device would be having varying capabilities, the > worker thread could be drafted differently to maximize performance. > This series introduces usage of multiple worker threads, among which the one to > be used will be determined by the operating conditions and the underlying > device capabilities. > > For example, if an event device - eth device pair has Tx internal port, then > application can do tx_adapter_enqueue() instead of regular event_enqueue(). So > a thread making an assumption that the device pair has internal port will not be > the right solution for another pair. The infrastructure added with these patches > aims to help application to have multiple worker threads, there by extracting > maximum performance from every device without affecting existing paths/use > cases. > > The eventmode configuration is predefined. All packets reaching one eth port > will hit one event queue. All event queues will be mapped to all event ports. So > all cores will be able to receive traffic from all ports. > When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event > device will ensure the ordering. Ordering would be lost when tried in PARALLEL. > > Following command line options are introduced, > > --transfer-mode: to choose between poll mode & event mode > --event-schedule-type: to specify the scheduling type > (RTE_SCHED_TYPE_ORDERED/ > RTE_SCHED_TYPE_ATOMIC/ > RTE_SCHED_TYPE_PARALLEL) > > Additionally the event mode introduces two modes of processing packets: > > Driver-mode: This mode will have bare minimum changes in the application > to support ipsec. There woudn't be any lookup etc done in > the application. And for inline-protocol use case, the > thread would resemble l2fwd as the ipsec processing would be > done entirely in the h/w. This mode can be used to benchmark > the raw performance of the h/w. All the application side > steps (like lookup) can be redone based on the requirement > of the end user. Hence the need for a mode which would > report the raw performance. > > App-mode: This mode will have all the features currently implemented with > ipsec-secgw (non librte_ipsec mode). All the lookups etc > would follow the existing methods and would report numbers > that can be compared against regular ipsec-secgw benchmark > numbers. > > The driver mode is selected with existing --single-sa option (used also by poll > mode). When --single-sa option is used in conjution with event mode then index > passed to --single-sa is ignored. > > Example commands to execute ipsec-secgw in various modes on OCTEON TX2 > platform, > > #Inbound and outbound app mode > ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w > 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log- > level=8 -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg --transfer-mode event --event- > schedule-type parallel > > #Inbound and outbound driver mode > ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w > 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log- > level=8 -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg --transfer-mode event --event- > schedule-type parallel --single-sa 0 > > This series adds non burst tx internal port workers only. It provides infrastructure > for non internal port workers, however does not define any. Also, only inline > ipsec protocol mode is supported by the worker threads added. > > Following are planned features, > 1. Add burst mode workers. > 2. Add non internal port workers. > 3. Verify support for Rx core (the support is added but lack of h/w to verify). > 4. Add lookaside protocol support. > > Following are features that Marvell won't be attempting. > 1. Inline crypto support. > 2. Lookaside crypto support. > > For the features that Marvell won't be attempting, new workers can be > introduced by the respective stake holders. > > This series is tested on Marvell OCTEON TX2. > This series is targeted for 20.05 release. > > Changes in v5: > * Rename function check_params() to check_poll_mode_params() and > check_eh_conf() to check_event_mode_params() in order to make it clear > what is their purpose. > * Forbid usage of --config option in event mode. > * Replace magic numbers on return with enum values in > process_ipsec_ev_inbound() > and process_ipsec_ev_outbound() functions. > * Add session_priv_pool for both inbound and outbound configuration in > ipsec_wrkr_non_burst_int_port_app_mode worker. > * Add check of event type in ipsec_wrkr_non_burst_int_port_app_mode worker. > * Update description of --config option in both ipsec-secgw help and > documentation. > > Changes in v4: > * Update ipsec-secgw documentation to describe the new options as well as > event mode support. > * In event mode reserve number of crypto queues equal to number of eth ports > in order to meet inline protocol offload requirements. > * Add calculate_nb_mbufs() function to calculate number of mbufs in a pool > and include fragments table size into the calculation. > * Move structures ipsec_xf and ipsec_sad to ipsec.h and remove static keyword > from sa_out, nb_sa_out, sa_in and nb_sa_in in sa.c. > * Update process_ipsec_ev_inbound(), process_ipsec_ev_outbound(), > check_sp() > and prepare_out_sessions_tbl() functions as a result of changes introduced > by SAD feature. > * Remove setting sa->cdev_id_qp to 0 in create_inline_session as sa_ctx > is created with rte_zmalloc. > * Minor cleanup enhancements: > - In eh_set_default_conf_eventdev() function in event_helper.c put definition > of int local vars in one line, remove invalid comment, put > "eventdev_config->ev_queue_mode = RTE_EVENT_QUEUE_CFG_ALL_TYPES" > in one line > instead of two. > - Remove extern "C" from event_helper.h. > - Put local vars in reverse xmas tree order in eh_dev_has_rx_internal_port() and > eh_dev_has_tx_internal_port() functions in event_helper.c. > - Put #include <rte_bitmap.h> in alphabetical order in ipsec-secgw.c. > - Move "extern volatile bool force_quit" and "#include <stdbool.h>" to ipsec- > secgw.h, > remove #include <rte_hash.h>. > - Remove not needed includes in ipsec_worker.c. > - Remove expired todo from ipsec_worker.h. > > Changes in v3: > * Move eh_conf_init() and eh_conf_uninit() functions to event_helper.c > including minor rework. > * Rename --schedule-type option to --event-schedule-type. > * Replace macro UNPROTECTED_PORT with static inline function > is_unprotected_port(). > * Move definitions of global variables used by multiple modules > to .c files and add externs in .h headers. > * Add eh_check_conf() which validates ipsec-secgw configuration > for event mode. > * Add dynamic calculation of number of buffers in a pool based > on number of cores, ports and crypto queues. > * Fix segmentation fault in event mode driver worker which happens > when there are no inline outbound sessions configured. > * Remove change related to updating number of crypto queues > in cryptodevs_init(). The update of crypto queues will be handled > in a separate patch. > * Fix compilation error on 32-bit platforms by using userdata instead > of udata64 from rte_mbuf. > > Changes in v2: > * Remove --process-dir option. Instead use existing unprotected port mask > option (-u) to decide wheter port handles inbound or outbound traffic. > * Remove --process-mode option. Instead use existing --single-sa option > to select between app and driver modes. > * Add handling of PKT_RX_SEC_OFFLOAD_FAIL result in app worker thread. > * Fix passing of req_rx_offload flags to create_default_ipsec_flow(). > * Move destruction of flows to a location where eth ports are stopped > and closed. > * Print error and exit when event mode --schedule-type option is used > in poll mode. > * Reduce number of goto statements replacing them with loop constructs. > * Remove sec_session_fixed table and replace it with locally build > table in driver worker thread. Table is indexed by port identifier > and holds first inline session pointer found for a given port. > * Print error and exit when sessions other than inline are configured > in event mode. > * When number of event queues is less than number of eth ports then > map all eth ports to one event queue. > * Cleanup and minor improvements in code as suggested by Konstantin > > Ankur Dwivedi (1): > examples/ipsec-secgw: add default rte flow for inline Rx > > Anoob Joseph (5): > examples/ipsec-secgw: add framework for eventmode helper > examples/ipsec-secgw: add eventdev port-lcore link > examples/ipsec-secgw: add Rx adapter support > examples/ipsec-secgw: add Tx adapter support > examples/ipsec-secgw: add routines to display config > > Lukasz Bartosik (9): > examples/ipsec-secgw: add routines to launch workers > examples/ipsec-secgw: add support for internal ports > examples/ipsec-secgw: add event helper config init/uninit > examples/ipsec-secgw: add eventmode to ipsec-secgw > examples/ipsec-secgw: add driver mode worker > examples/ipsec-secgw: add app mode worker > examples/ipsec-secgw: make number of buffers dynamic > doc: add event mode support to ipsec-secgw > examples/ipsec-secgw: reserve crypto queues in event mode > > doc/guides/sample_app_ug/ipsec_secgw.rst | 135 ++- > examples/ipsec-secgw/Makefile | 2 + > examples/ipsec-secgw/event_helper.c | 1812 > ++++++++++++++++++++++++++++++ > examples/ipsec-secgw/event_helper.h | 327 ++++++ > examples/ipsec-secgw/ipsec-secgw.c | 506 +++++++-- > examples/ipsec-secgw/ipsec-secgw.h | 88 ++ > examples/ipsec-secgw/ipsec.c | 5 +- > examples/ipsec-secgw/ipsec.h | 53 +- > examples/ipsec-secgw/ipsec_worker.c | 649 +++++++++++ > examples/ipsec-secgw/ipsec_worker.h | 41 + > examples/ipsec-secgw/meson.build | 6 +- > examples/ipsec-secgw/sa.c | 21 +- > examples/ipsec-secgw/sad.h | 5 - > 13 files changed, 3516 insertions(+), 134 deletions(-) create mode 100644 > examples/ipsec-secgw/event_helper.c > create mode 100644 examples/ipsec-secgw/event_helper.h > create mode 100644 examples/ipsec-secgw/ipsec-secgw.h > create mode 100644 examples/ipsec-secgw/ipsec_worker.c > create mode 100644 examples/ipsec-secgw/ipsec_worker.h > > -- > 2.7.4
Hi Anoob, I will merge this series in this week. Regards, Akhil > -----Original Message----- > From: Anoob Joseph <anoobj@marvell.com> > Sent: Monday, March 2, 2020 2:17 PM > To: Akhil Goyal <akhil.goyal@nxp.com>; Konstantin Ananyev > <konstantin.ananyev@intel.com> > Cc: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Narayana Prasad Raju > Athreya <pathreya@marvell.com>; Ankur Dwivedi <adwivedi@marvell.com>; > Archana Muniganti <marchana@marvell.com>; Tejasree Kondoj > <ktejasree@marvell.com>; Vamsi Krishna Attunuru <vattunuru@marvell.com>; > dev@dpdk.org; Radu Nicolau <radu.nicolau@intel.com>; Thomas Monjalon > <thomas@monjalon.net>; Lukas Bartosik <lbartosik@marvell.com> > Subject: RE: [PATCH v5 00/15] add eventmode to ipsec-secgw > > Hi Akhil, Konstantin, > > Are there any more comments? Or can we have the patches merged? > > Thanks, > Anoob > > > -----Original Message----- > > From: Lukasz Bartosik <lbartosik@marvell.com> > > Sent: Thursday, February 27, 2020 9:48 PM > > To: Akhil Goyal <akhil.goyal@nxp.com>; Radu Nicolau > > <radu.nicolau@intel.com>; Thomas Monjalon <thomas@monjalon.net> > > Cc: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Narayana Prasad Raju > > Athreya <pathreya@marvell.com>; Ankur Dwivedi <adwivedi@marvell.com>; > > Anoob Joseph <anoobj@marvell.com>; Archana Muniganti > > <marchana@marvell.com>; Tejasree Kondoj <ktejasree@marvell.com>; Vamsi > > Krishna Attunuru <vattunuru@marvell.com>; Konstantin Ananyev > > <konstantin.ananyev@intel.com>; dev@dpdk.org > > Subject: [PATCH v5 00/15] add eventmode to ipsec-secgw > > > > This series introduces event-mode additions to ipsec-secgw. > > > > With this series, ipsec-secgw would be able to run in eventmode. The worker > > thread (executing loop) would be receiving events and would be submitting it > > back to the eventdev after the processing. This way, multicore scaling and h/w > > assisted scheduling is achieved by making use of the eventdev capabilities. > > > > Since the underlying event device would be having varying capabilities, the > > worker thread could be drafted differently to maximize performance. > > This series introduces usage of multiple worker threads, among which the one > to > > be used will be determined by the operating conditions and the underlying > > device capabilities. > > > > For example, if an event device - eth device pair has Tx internal port, then > > application can do tx_adapter_enqueue() instead of regular event_enqueue(). > So > > a thread making an assumption that the device pair has internal port will not > be > > the right solution for another pair. The infrastructure added with these patches > > aims to help application to have multiple worker threads, there by extracting > > maximum performance from every device without affecting existing paths/use > > cases. > > > > The eventmode configuration is predefined. All packets reaching one eth port > > will hit one event queue. All event queues will be mapped to all event ports. So > > all cores will be able to receive traffic from all ports. > > When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event > > device will ensure the ordering. Ordering would be lost when tried in PARALLEL. > > > > Following command line options are introduced, > > > > --transfer-mode: to choose between poll mode & event mode > > --event-schedule-type: to specify the scheduling type > > (RTE_SCHED_TYPE_ORDERED/ > > RTE_SCHED_TYPE_ATOMIC/ > > RTE_SCHED_TYPE_PARALLEL) > > > > Additionally the event mode introduces two modes of processing packets: > > > > Driver-mode: This mode will have bare minimum changes in the application > > to support ipsec. There woudn't be any lookup etc done in > > the application. And for inline-protocol use case, the > > thread would resemble l2fwd as the ipsec processing would be > > done entirely in the h/w. This mode can be used to benchmark > > the raw performance of the h/w. All the application side > > steps (like lookup) can be redone based on the requirement > > of the end user. Hence the need for a mode which would > > report the raw performance. > > > > App-mode: This mode will have all the features currently implemented with > > ipsec-secgw (non librte_ipsec mode). All the lookups etc > > would follow the existing methods and would report numbers > > that can be compared against regular ipsec-secgw benchmark > > numbers. > > > > The driver mode is selected with existing --single-sa option (used also by poll > > mode). When --single-sa option is used in conjution with event mode then > index > > passed to --single-sa is ignored. > > > > Example commands to execute ipsec-secgw in various modes on OCTEON TX2 > > platform, > > > > #Inbound and outbound app mode > > ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w > > 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log- > > level=8 -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg --transfer-mode event --event- > > schedule-type parallel > > > > #Inbound and outbound driver mode > > ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w > > 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log- > > level=8 -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg --transfer-mode event --event- > > schedule-type parallel --single-sa 0 > > > > This series adds non burst tx internal port workers only. It provides > infrastructure > > for non internal port workers, however does not define any. Also, only inline > > ipsec protocol mode is supported by the worker threads added. > > > > Following are planned features, > > 1. Add burst mode workers. > > 2. Add non internal port workers. > > 3. Verify support for Rx core (the support is added but lack of h/w to verify). > > 4. Add lookaside protocol support. > > > > Following are features that Marvell won't be attempting. > > 1. Inline crypto support. > > 2. Lookaside crypto support. > > > > For the features that Marvell won't be attempting, new workers can be > > introduced by the respective stake holders. > > > > This series is tested on Marvell OCTEON TX2. > > This series is targeted for 20.05 release. > > > > Changes in v5: > > * Rename function check_params() to check_poll_mode_params() and > > check_eh_conf() to check_event_mode_params() in order to make it clear > > what is their purpose. > > * Forbid usage of --config option in event mode. > > * Replace magic numbers on return with enum values in > > process_ipsec_ev_inbound() > > and process_ipsec_ev_outbound() functions. > > * Add session_priv_pool for both inbound and outbound configuration in > > ipsec_wrkr_non_burst_int_port_app_mode worker. > > * Add check of event type in ipsec_wrkr_non_burst_int_port_app_mode > worker. > > * Update description of --config option in both ipsec-secgw help and > > documentation. > > > > Changes in v4: > > * Update ipsec-secgw documentation to describe the new options as well as > > event mode support. > > * In event mode reserve number of crypto queues equal to number of eth > ports > > in order to meet inline protocol offload requirements. > > * Add calculate_nb_mbufs() function to calculate number of mbufs in a pool > > and include fragments table size into the calculation. > > * Move structures ipsec_xf and ipsec_sad to ipsec.h and remove static > keyword > > from sa_out, nb_sa_out, sa_in and nb_sa_in in sa.c. > > * Update process_ipsec_ev_inbound(), process_ipsec_ev_outbound(), > > check_sp() > > and prepare_out_sessions_tbl() functions as a result of changes introduced > > by SAD feature. > > * Remove setting sa->cdev_id_qp to 0 in create_inline_session as sa_ctx > > is created with rte_zmalloc. > > * Minor cleanup enhancements: > > - In eh_set_default_conf_eventdev() function in event_helper.c put definition > > of int local vars in one line, remove invalid comment, put > > "eventdev_config->ev_queue_mode = RTE_EVENT_QUEUE_CFG_ALL_TYPES" > > in one line > > instead of two. > > - Remove extern "C" from event_helper.h. > > - Put local vars in reverse xmas tree order in eh_dev_has_rx_internal_port() > and > > eh_dev_has_tx_internal_port() functions in event_helper.c. > > - Put #include <rte_bitmap.h> in alphabetical order in ipsec-secgw.c. > > - Move "extern volatile bool force_quit" and "#include <stdbool.h>" to ipsec- > > secgw.h, > > remove #include <rte_hash.h>. > > - Remove not needed includes in ipsec_worker.c. > > - Remove expired todo from ipsec_worker.h. > > > > Changes in v3: > > * Move eh_conf_init() and eh_conf_uninit() functions to event_helper.c > > including minor rework. > > * Rename --schedule-type option to --event-schedule-type. > > * Replace macro UNPROTECTED_PORT with static inline function > > is_unprotected_port(). > > * Move definitions of global variables used by multiple modules > > to .c files and add externs in .h headers. > > * Add eh_check_conf() which validates ipsec-secgw configuration > > for event mode. > > * Add dynamic calculation of number of buffers in a pool based > > on number of cores, ports and crypto queues. > > * Fix segmentation fault in event mode driver worker which happens > > when there are no inline outbound sessions configured. > > * Remove change related to updating number of crypto queues > > in cryptodevs_init(). The update of crypto queues will be handled > > in a separate patch. > > * Fix compilation error on 32-bit platforms by using userdata instead > > of udata64 from rte_mbuf. > > > > Changes in v2: > > * Remove --process-dir option. Instead use existing unprotected port mask > > option (-u) to decide wheter port handles inbound or outbound traffic. > > * Remove --process-mode option. Instead use existing --single-sa option > > to select between app and driver modes. > > * Add handling of PKT_RX_SEC_OFFLOAD_FAIL result in app worker thread. > > * Fix passing of req_rx_offload flags to create_default_ipsec_flow(). > > * Move destruction of flows to a location where eth ports are stopped > > and closed. > > * Print error and exit when event mode --schedule-type option is used > > in poll mode. > > * Reduce number of goto statements replacing them with loop constructs. > > * Remove sec_session_fixed table and replace it with locally build > > table in driver worker thread. Table is indexed by port identifier > > and holds first inline session pointer found for a given port. > > * Print error and exit when sessions other than inline are configured > > in event mode. > > * When number of event queues is less than number of eth ports then > > map all eth ports to one event queue. > > * Cleanup and minor improvements in code as suggested by Konstantin > > > > Ankur Dwivedi (1): > > examples/ipsec-secgw: add default rte flow for inline Rx > > > > Anoob Joseph (5): > > examples/ipsec-secgw: add framework for eventmode helper > > examples/ipsec-secgw: add eventdev port-lcore link > > examples/ipsec-secgw: add Rx adapter support > > examples/ipsec-secgw: add Tx adapter support > > examples/ipsec-secgw: add routines to display config > > > > Lukasz Bartosik (9): > > examples/ipsec-secgw: add routines to launch workers > > examples/ipsec-secgw: add support for internal ports > > examples/ipsec-secgw: add event helper config init/uninit > > examples/ipsec-secgw: add eventmode to ipsec-secgw > > examples/ipsec-secgw: add driver mode worker > > examples/ipsec-secgw: add app mode worker > > examples/ipsec-secgw: make number of buffers dynamic > > doc: add event mode support to ipsec-secgw > > examples/ipsec-secgw: reserve crypto queues in event mode > > > > doc/guides/sample_app_ug/ipsec_secgw.rst | 135 ++- > > examples/ipsec-secgw/Makefile | 2 + > > examples/ipsec-secgw/event_helper.c | 1812 > > ++++++++++++++++++++++++++++++ > > examples/ipsec-secgw/event_helper.h | 327 ++++++ > > examples/ipsec-secgw/ipsec-secgw.c | 506 +++++++-- > > examples/ipsec-secgw/ipsec-secgw.h | 88 ++ > > examples/ipsec-secgw/ipsec.c | 5 +- > > examples/ipsec-secgw/ipsec.h | 53 +- > > examples/ipsec-secgw/ipsec_worker.c | 649 +++++++++++ > > examples/ipsec-secgw/ipsec_worker.h | 41 + > > examples/ipsec-secgw/meson.build | 6 +- > > examples/ipsec-secgw/sa.c | 21 +- > > examples/ipsec-secgw/sad.h | 5 - > > 13 files changed, 3516 insertions(+), 134 deletions(-) create mode 100644 > > examples/ipsec-secgw/event_helper.c > > create mode 100644 examples/ipsec-secgw/event_helper.h > > create mode 100644 examples/ipsec-secgw/ipsec-secgw.h > > create mode 100644 examples/ipsec-secgw/ipsec_worker.c > > create mode 100644 examples/ipsec-secgw/ipsec_worker.h > > > > -- > > 2.7.4
> > This series introduces event-mode additions to ipsec-secgw. > > With this series, ipsec-secgw would be able to run in eventmode. The > worker thread (executing loop) would be receiving events and would be > submitting it back to the eventdev after the processing. This way, > multicore scaling and h/w assisted scheduling is achieved by making use > of the eventdev capabilities. > > Since the underlying event device would be having varying capabilities, > the worker thread could be drafted differently to maximize performance. > This series introduces usage of multiple worker threads, among which the > one to be used will be determined by the operating conditions and the > underlying device capabilities. > > For example, if an event device - eth device pair has Tx internal port, > then application can do tx_adapter_enqueue() instead of regular > event_enqueue(). So a thread making an assumption that the device pair > has internal port will not be the right solution for another pair. The > infrastructure added with these patches aims to help application to have > multiple worker threads, there by extracting maximum performance from > every device without affecting existing paths/use cases. > > The eventmode configuration is predefined. All packets reaching one eth > port will hit one event queue. All event queues will be mapped to all > event ports. So all cores will be able to receive traffic from all ports. > When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event device > will ensure the ordering. Ordering would be lost when tried in PARALLEL. > > Following command line options are introduced, > > --transfer-mode: to choose between poll mode & event mode > --event-schedule-type: to specify the scheduling type > (RTE_SCHED_TYPE_ORDERED/ > RTE_SCHED_TYPE_ATOMIC/ > RTE_SCHED_TYPE_PARALLEL) > > Additionally the event mode introduces two modes of processing packets: > > Driver-mode: This mode will have bare minimum changes in the application > to support ipsec. There woudn't be any lookup etc done in > the application. And for inline-protocol use case, the > thread would resemble l2fwd as the ipsec processing would be > done entirely in the h/w. This mode can be used to benchmark > the raw performance of the h/w. All the application side > steps (like lookup) can be redone based on the requirement > of the end user. Hence the need for a mode which would > report the raw performance. > > App-mode: This mode will have all the features currently implemented with > ipsec-secgw (non librte_ipsec mode). All the lookups etc > would follow the existing methods and would report numbers > that can be compared against regular ipsec-secgw benchmark > numbers. > > The driver mode is selected with existing --single-sa option > (used also by poll mode). When --single-sa option is used > in conjution with event mode then index passed to --single-sa > is ignored. > > Example commands to execute ipsec-secgw in various modes on OCTEON TX2 platform, > > #Inbound and outbound app mode > ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 > -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg --transfer-mode event --event-schedule-type parallel > > #Inbound and outbound driver mode > ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 --log-level=8 > -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg --transfer-mode event --event-schedule-type parallel --single-sa 0 > > This series adds non burst tx internal port workers only. It provides infrastructure > for non internal port workers, however does not define any. Also, only inline ipsec > protocol mode is supported by the worker threads added. > > Following are planned features, > 1. Add burst mode workers. > 2. Add non internal port workers. > 3. Verify support for Rx core (the support is added but lack of h/w to verify). > 4. Add lookaside protocol support. > > Following are features that Marvell won't be attempting. > 1. Inline crypto support. > 2. Lookaside crypto support. > > For the features that Marvell won't be attempting, new workers can be > introduced by the respective stake holders. > > This series is tested on Marvell OCTEON TX2. > This series is targeted for 20.05 release. > > Changes in v5: > * Rename function check_params() to check_poll_mode_params() and > check_eh_conf() to check_event_mode_params() in order to make it clear > what is their purpose. > * Forbid usage of --config option in event mode. > * Replace magic numbers on return with enum values in process_ipsec_ev_inbound() > and process_ipsec_ev_outbound() functions. > * Add session_priv_pool for both inbound and outbound configuration in > ipsec_wrkr_non_burst_int_port_app_mode worker. > * Add check of event type in ipsec_wrkr_non_burst_int_port_app_mode worker. > * Update description of --config option in both ipsec-secgw help and documentation. > > Changes in v4: > * Update ipsec-secgw documentation to describe the new options as well as > event mode support. > * In event mode reserve number of crypto queues equal to number of eth ports > in order to meet inline protocol offload requirements. > * Add calculate_nb_mbufs() function to calculate number of mbufs in a pool > and include fragments table size into the calculation. > * Move structures ipsec_xf and ipsec_sad to ipsec.h and remove static keyword > from sa_out, nb_sa_out, sa_in and nb_sa_in in sa.c. > * Update process_ipsec_ev_inbound(), process_ipsec_ev_outbound(), check_sp() > and prepare_out_sessions_tbl() functions as a result of changes introduced > by SAD feature. > * Remove setting sa->cdev_id_qp to 0 in create_inline_session as sa_ctx > is created with rte_zmalloc. > * Minor cleanup enhancements: > - In eh_set_default_conf_eventdev() function in event_helper.c put definition > of int local vars in one line, remove invalid comment, put > "eventdev_config->ev_queue_mode = RTE_EVENT_QUEUE_CFG_ALL_TYPES" in one line > instead of two. > - Remove extern "C" from event_helper.h. > - Put local vars in reverse xmas tree order in eh_dev_has_rx_internal_port() and > eh_dev_has_tx_internal_port() functions in event_helper.c. > - Put #include <rte_bitmap.h> in alphabetical order in ipsec-secgw.c. > - Move "extern volatile bool force_quit" and "#include <stdbool.h>" to ipsec-secgw.h, > remove #include <rte_hash.h>. > - Remove not needed includes in ipsec_worker.c. > - Remove expired todo from ipsec_worker.h. > > Changes in v3: > * Move eh_conf_init() and eh_conf_uninit() functions to event_helper.c > including minor rework. > * Rename --schedule-type option to --event-schedule-type. > * Replace macro UNPROTECTED_PORT with static inline function > is_unprotected_port(). > * Move definitions of global variables used by multiple modules > to .c files and add externs in .h headers. > * Add eh_check_conf() which validates ipsec-secgw configuration > for event mode. > * Add dynamic calculation of number of buffers in a pool based > on number of cores, ports and crypto queues. > * Fix segmentation fault in event mode driver worker which happens > when there are no inline outbound sessions configured. > * Remove change related to updating number of crypto queues > in cryptodevs_init(). The update of crypto queues will be handled > in a separate patch. > * Fix compilation error on 32-bit platforms by using userdata instead > of udata64 from rte_mbuf. > > Changes in v2: > * Remove --process-dir option. Instead use existing unprotected port mask > option (-u) to decide wheter port handles inbound or outbound traffic. > * Remove --process-mode option. Instead use existing --single-sa option > to select between app and driver modes. > * Add handling of PKT_RX_SEC_OFFLOAD_FAIL result in app worker thread. > * Fix passing of req_rx_offload flags to create_default_ipsec_flow(). > * Move destruction of flows to a location where eth ports are stopped > and closed. > * Print error and exit when event mode --schedule-type option is used > in poll mode. > * Reduce number of goto statements replacing them with loop constructs. > * Remove sec_session_fixed table and replace it with locally build > table in driver worker thread. Table is indexed by port identifier > and holds first inline session pointer found for a given port. > * Print error and exit when sessions other than inline are configured > in event mode. > * When number of event queues is less than number of eth ports then > map all eth ports to one event queue. > * Cleanup and minor improvements in code as suggested by Konstantin > > Ankur Dwivedi (1): > examples/ipsec-secgw: add default rte flow for inline Rx > > Anoob Joseph (5): > examples/ipsec-secgw: add framework for eventmode helper > examples/ipsec-secgw: add eventdev port-lcore link > examples/ipsec-secgw: add Rx adapter support > examples/ipsec-secgw: add Tx adapter support > examples/ipsec-secgw: add routines to display config > > Lukasz Bartosik (9): > examples/ipsec-secgw: add routines to launch workers > examples/ipsec-secgw: add support for internal ports > examples/ipsec-secgw: add event helper config init/uninit > examples/ipsec-secgw: add eventmode to ipsec-secgw > examples/ipsec-secgw: add driver mode worker > examples/ipsec-secgw: add app mode worker > examples/ipsec-secgw: make number of buffers dynamic > doc: add event mode support to ipsec-secgw > examples/ipsec-secgw: reserve crypto queues in event mode > > doc/guides/sample_app_ug/ipsec_secgw.rst | 135 ++- > examples/ipsec-secgw/Makefile | 2 + > examples/ipsec-secgw/event_helper.c | 1812 ++++++++++++++++++++++++++++++ > examples/ipsec-secgw/event_helper.h | 327 ++++++ > examples/ipsec-secgw/ipsec-secgw.c | 506 +++++++-- > examples/ipsec-secgw/ipsec-secgw.h | 88 ++ > examples/ipsec-secgw/ipsec.c | 5 +- > examples/ipsec-secgw/ipsec.h | 53 +- > examples/ipsec-secgw/ipsec_worker.c | 649 +++++++++++ > examples/ipsec-secgw/ipsec_worker.h | 41 + > examples/ipsec-secgw/meson.build | 6 +- > examples/ipsec-secgw/sa.c | 21 +- > examples/ipsec-secgw/sad.h | 5 - > 13 files changed, 3516 insertions(+), 134 deletions(-) > create mode 100644 examples/ipsec-secgw/event_helper.c > create mode 100644 examples/ipsec-secgw/event_helper.h > create mode 100644 examples/ipsec-secgw/ipsec-secgw.h > create mode 100644 examples/ipsec-secgw/ipsec_worker.c > create mode 100644 examples/ipsec-secgw/ipsec_worker.h > > -- Have to say I didn't look extensively on event mode. My primary concern was poll-mode and common code changes. From that perspective - LGTM. Series Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com> > 2.7.4
Hi Akhil, Reminder. Do you have any further review comments? Thanks, Anoob > -----Original Message----- > From: Ananyev, Konstantin <konstantin.ananyev@intel.com> > Sent: Tuesday, March 3, 2020 11:30 PM > To: Lukas Bartosik <lbartosik@marvell.com>; Akhil Goyal > <akhil.goyal@nxp.com>; Nicolau, Radu <radu.nicolau@intel.com>; Thomas > Monjalon <thomas@monjalon.net> > Cc: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Narayana Prasad Raju > Athreya <pathreya@marvell.com>; Ankur Dwivedi <adwivedi@marvell.com>; > Anoob Joseph <anoobj@marvell.com>; Archana Muniganti > <marchana@marvell.com>; Tejasree Kondoj <ktejasree@marvell.com>; Vamsi > Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org > Subject: [EXT] RE: [PATCH v5 00/15] add eventmode to ipsec-secgw > > External Email > > ---------------------------------------------------------------------- > > > > > This series introduces event-mode additions to ipsec-secgw. > > > > With this series, ipsec-secgw would be able to run in eventmode. The > > worker thread (executing loop) would be receiving events and would be > > submitting it back to the eventdev after the processing. This way, > > multicore scaling and h/w assisted scheduling is achieved by making > > use of the eventdev capabilities. > > > > Since the underlying event device would be having varying > > capabilities, the worker thread could be drafted differently to maximize > performance. > > This series introduces usage of multiple worker threads, among which > > the one to be used will be determined by the operating conditions and > > the underlying device capabilities. > > > > For example, if an event device - eth device pair has Tx internal > > port, then application can do tx_adapter_enqueue() instead of regular > > event_enqueue(). So a thread making an assumption that the device pair > > has internal port will not be the right solution for another pair. The > > infrastructure added with these patches aims to help application to > > have multiple worker threads, there by extracting maximum performance > > from every device without affecting existing paths/use cases. > > > > The eventmode configuration is predefined. All packets reaching one > > eth port will hit one event queue. All event queues will be mapped to > > all event ports. So all cores will be able to receive traffic from all ports. > > When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event > > device will ensure the ordering. Ordering would be lost when tried in > PARALLEL. > > > > Following command line options are introduced, > > > > --transfer-mode: to choose between poll mode & event mode > > --event-schedule-type: to specify the scheduling type > > (RTE_SCHED_TYPE_ORDERED/ > > RTE_SCHED_TYPE_ATOMIC/ > > RTE_SCHED_TYPE_PARALLEL) > > > > Additionally the event mode introduces two modes of processing packets: > > > > Driver-mode: This mode will have bare minimum changes in the application > > to support ipsec. There woudn't be any lookup etc done in > > the application. And for inline-protocol use case, the > > thread would resemble l2fwd as the ipsec processing would be > > done entirely in the h/w. This mode can be used to benchmark > > the raw performance of the h/w. All the application side > > steps (like lookup) can be redone based on the requirement > > of the end user. Hence the need for a mode which would > > report the raw performance. > > > > App-mode: This mode will have all the features currently implemented with > > ipsec-secgw (non librte_ipsec mode). All the lookups etc > > would follow the existing methods and would report numbers > > that can be compared against regular ipsec-secgw benchmark > > numbers. > > > > The driver mode is selected with existing --single-sa option (used > > also by poll mode). When --single-sa option is used in conjution with > > event mode then index passed to --single-sa is ignored. > > > > Example commands to execute ipsec-secgw in various modes on OCTEON TX2 > > platform, > > > > #Inbound and outbound app mode > > ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w > > 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 > > --log-level=8 -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg > > --transfer-mode event --event-schedule-type parallel > > > > #Inbound and outbound driver mode > > ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w > > 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 > > --log-level=8 -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg > > --transfer-mode event --event-schedule-type parallel --single-sa 0 > > > > This series adds non burst tx internal port workers only. It provides > > infrastructure for non internal port workers, however does not define > > any. Also, only inline ipsec protocol mode is supported by the worker threads > added. > > > > Following are planned features, > > 1. Add burst mode workers. > > 2. Add non internal port workers. > > 3. Verify support for Rx core (the support is added but lack of h/w to verify). > > 4. Add lookaside protocol support. > > > > Following are features that Marvell won't be attempting. > > 1. Inline crypto support. > > 2. Lookaside crypto support. > > > > For the features that Marvell won't be attempting, new workers can be > > introduced by the respective stake holders. > > > > This series is tested on Marvell OCTEON TX2. > > This series is targeted for 20.05 release. > > > > Changes in v5: > > * Rename function check_params() to check_poll_mode_params() and > > check_eh_conf() to check_event_mode_params() in order to make it clear > > what is their purpose. > > * Forbid usage of --config option in event mode. > > * Replace magic numbers on return with enum values in > process_ipsec_ev_inbound() > > and process_ipsec_ev_outbound() functions. > > * Add session_priv_pool for both inbound and outbound configuration in > > ipsec_wrkr_non_burst_int_port_app_mode worker. > > * Add check of event type in ipsec_wrkr_non_burst_int_port_app_mode > worker. > > * Update description of --config option in both ipsec-secgw help and > documentation. > > > > Changes in v4: > > * Update ipsec-secgw documentation to describe the new options as well as > > event mode support. > > * In event mode reserve number of crypto queues equal to number of eth > ports > > in order to meet inline protocol offload requirements. > > * Add calculate_nb_mbufs() function to calculate number of mbufs in a pool > > and include fragments table size into the calculation. > > * Move structures ipsec_xf and ipsec_sad to ipsec.h and remove static > keyword > > from sa_out, nb_sa_out, sa_in and nb_sa_in in sa.c. > > * Update process_ipsec_ev_inbound(), process_ipsec_ev_outbound(), > check_sp() > > and prepare_out_sessions_tbl() functions as a result of changes introduced > > by SAD feature. > > * Remove setting sa->cdev_id_qp to 0 in create_inline_session as sa_ctx > > is created with rte_zmalloc. > > * Minor cleanup enhancements: > > - In eh_set_default_conf_eventdev() function in event_helper.c put definition > > of int local vars in one line, remove invalid comment, put > > "eventdev_config->ev_queue_mode = RTE_EVENT_QUEUE_CFG_ALL_TYPES" > in one line > > instead of two. > > - Remove extern "C" from event_helper.h. > > - Put local vars in reverse xmas tree order in eh_dev_has_rx_internal_port() > and > > eh_dev_has_tx_internal_port() functions in event_helper.c. > > - Put #include <rte_bitmap.h> in alphabetical order in ipsec-secgw.c. > > - Move "extern volatile bool force_quit" and "#include <stdbool.h>" to ipsec- > secgw.h, > > remove #include <rte_hash.h>. > > - Remove not needed includes in ipsec_worker.c. > > - Remove expired todo from ipsec_worker.h. > > > > Changes in v3: > > * Move eh_conf_init() and eh_conf_uninit() functions to event_helper.c > > including minor rework. > > * Rename --schedule-type option to --event-schedule-type. > > * Replace macro UNPROTECTED_PORT with static inline function > > is_unprotected_port(). > > * Move definitions of global variables used by multiple modules > > to .c files and add externs in .h headers. > > * Add eh_check_conf() which validates ipsec-secgw configuration > > for event mode. > > * Add dynamic calculation of number of buffers in a pool based > > on number of cores, ports and crypto queues. > > * Fix segmentation fault in event mode driver worker which happens > > when there are no inline outbound sessions configured. > > * Remove change related to updating number of crypto queues > > in cryptodevs_init(). The update of crypto queues will be handled > > in a separate patch. > > * Fix compilation error on 32-bit platforms by using userdata instead > > of udata64 from rte_mbuf. > > > > Changes in v2: > > * Remove --process-dir option. Instead use existing unprotected port mask > > option (-u) to decide wheter port handles inbound or outbound traffic. > > * Remove --process-mode option. Instead use existing --single-sa option > > to select between app and driver modes. > > * Add handling of PKT_RX_SEC_OFFLOAD_FAIL result in app worker thread. > > * Fix passing of req_rx_offload flags to create_default_ipsec_flow(). > > * Move destruction of flows to a location where eth ports are stopped > > and closed. > > * Print error and exit when event mode --schedule-type option is used > > in poll mode. > > * Reduce number of goto statements replacing them with loop constructs. > > * Remove sec_session_fixed table and replace it with locally build > > table in driver worker thread. Table is indexed by port identifier > > and holds first inline session pointer found for a given port. > > * Print error and exit when sessions other than inline are configured > > in event mode. > > * When number of event queues is less than number of eth ports then > > map all eth ports to one event queue. > > * Cleanup and minor improvements in code as suggested by Konstantin > > > > Ankur Dwivedi (1): > > examples/ipsec-secgw: add default rte flow for inline Rx > > > > Anoob Joseph (5): > > examples/ipsec-secgw: add framework for eventmode helper > > examples/ipsec-secgw: add eventdev port-lcore link > > examples/ipsec-secgw: add Rx adapter support > > examples/ipsec-secgw: add Tx adapter support > > examples/ipsec-secgw: add routines to display config > > > > Lukasz Bartosik (9): > > examples/ipsec-secgw: add routines to launch workers > > examples/ipsec-secgw: add support for internal ports > > examples/ipsec-secgw: add event helper config init/uninit > > examples/ipsec-secgw: add eventmode to ipsec-secgw > > examples/ipsec-secgw: add driver mode worker > > examples/ipsec-secgw: add app mode worker > > examples/ipsec-secgw: make number of buffers dynamic > > doc: add event mode support to ipsec-secgw > > examples/ipsec-secgw: reserve crypto queues in event mode > > > > doc/guides/sample_app_ug/ipsec_secgw.rst | 135 ++- > > examples/ipsec-secgw/Makefile | 2 + > > examples/ipsec-secgw/event_helper.c | 1812 > ++++++++++++++++++++++++++++++ > > examples/ipsec-secgw/event_helper.h | 327 ++++++ > > examples/ipsec-secgw/ipsec-secgw.c | 506 +++++++-- > > examples/ipsec-secgw/ipsec-secgw.h | 88 ++ > > examples/ipsec-secgw/ipsec.c | 5 +- > > examples/ipsec-secgw/ipsec.h | 53 +- > > examples/ipsec-secgw/ipsec_worker.c | 649 +++++++++++ > > examples/ipsec-secgw/ipsec_worker.h | 41 + > > examples/ipsec-secgw/meson.build | 6 +- > > examples/ipsec-secgw/sa.c | 21 +- > > examples/ipsec-secgw/sad.h | 5 - > > 13 files changed, 3516 insertions(+), 134 deletions(-) create mode > > 100644 examples/ipsec-secgw/event_helper.c > > create mode 100644 examples/ipsec-secgw/event_helper.h > > create mode 100644 examples/ipsec-secgw/ipsec-secgw.h > > create mode 100644 examples/ipsec-secgw/ipsec_worker.c > > create mode 100644 examples/ipsec-secgw/ipsec_worker.h > > > > -- > > Have to say I didn't look extensively on event mode. > My primary concern was poll-mode and common code changes. > From that perspective - LGTM. > > Series Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com> > > > 2.7.4
Hi Anoob, Please send a release note update as a reply to this mail. I will update it while merging the patchset. Regards, Akhil > > Hi Akhil, > > Reminder. > > Do you have any further review comments? > > Thanks, > Anoob > > > -----Original Message----- > > From: Ananyev, Konstantin <konstantin.ananyev@intel.com> > > Sent: Tuesday, March 3, 2020 11:30 PM > > To: Lukas Bartosik <lbartosik@marvell.com>; Akhil Goyal > > <akhil.goyal@nxp.com>; Nicolau, Radu <radu.nicolau@intel.com>; Thomas > > Monjalon <thomas@monjalon.net> > > Cc: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Narayana Prasad Raju > > Athreya <pathreya@marvell.com>; Ankur Dwivedi <adwivedi@marvell.com>; > > Anoob Joseph <anoobj@marvell.com>; Archana Muniganti > > <marchana@marvell.com>; Tejasree Kondoj <ktejasree@marvell.com>; Vamsi > > Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org > > Subject: [EXT] RE: [PATCH v5 00/15] add eventmode to ipsec-secgw > > > > External Email > > > > ---------------------------------------------------------------------- > > > > > > > > This series introduces event-mode additions to ipsec-secgw. > > > > > > With this series, ipsec-secgw would be able to run in eventmode. The > > > worker thread (executing loop) would be receiving events and would be > > > submitting it back to the eventdev after the processing. This way, > > > multicore scaling and h/w assisted scheduling is achieved by making > > > use of the eventdev capabilities. > > > > > > Since the underlying event device would be having varying > > > capabilities, the worker thread could be drafted differently to maximize > > performance. > > > This series introduces usage of multiple worker threads, among which > > > the one to be used will be determined by the operating conditions and > > > the underlying device capabilities. > > > > > > For example, if an event device - eth device pair has Tx internal > > > port, then application can do tx_adapter_enqueue() instead of regular > > > event_enqueue(). So a thread making an assumption that the device pair > > > has internal port will not be the right solution for another pair. The > > > infrastructure added with these patches aims to help application to > > > have multiple worker threads, there by extracting maximum performance > > > from every device without affecting existing paths/use cases. > > > > > > The eventmode configuration is predefined. All packets reaching one > > > eth port will hit one event queue. All event queues will be mapped to > > > all event ports. So all cores will be able to receive traffic from all ports. > > > When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event > > > device will ensure the ordering. Ordering would be lost when tried in > > PARALLEL. > > > > > > Following command line options are introduced, > > > > > > --transfer-mode: to choose between poll mode & event mode > > > --event-schedule-type: to specify the scheduling type > > > (RTE_SCHED_TYPE_ORDERED/ > > > RTE_SCHED_TYPE_ATOMIC/ > > > RTE_SCHED_TYPE_PARALLEL) > > > > > > Additionally the event mode introduces two modes of processing packets: > > > > > > Driver-mode: This mode will have bare minimum changes in the application > > > to support ipsec. There woudn't be any lookup etc done in > > > the application. And for inline-protocol use case, the > > > thread would resemble l2fwd as the ipsec processing would be > > > done entirely in the h/w. This mode can be used to benchmark > > > the raw performance of the h/w. All the application side > > > steps (like lookup) can be redone based on the requirement > > > of the end user. Hence the need for a mode which would > > > report the raw performance. > > > > > > App-mode: This mode will have all the features currently implemented with > > > ipsec-secgw (non librte_ipsec mode). All the lookups etc > > > would follow the existing methods and would report numbers > > > that can be compared against regular ipsec-secgw benchmark > > > numbers. > > > > > > The driver mode is selected with existing --single-sa option (used > > > also by poll mode). When --single-sa option is used in conjution with > > > event mode then index passed to --single-sa is ignored. > > > > > > Example commands to execute ipsec-secgw in various modes on OCTEON > TX2 > > > platform, > > > > > > #Inbound and outbound app mode > > > ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w > > > 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 > > > --log-level=8 -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg > > > --transfer-mode event --event-schedule-type parallel > > > > > > #Inbound and outbound driver mode > > > ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w > > > 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 > > > --log-level=8 -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg > > > --transfer-mode event --event-schedule-type parallel --single-sa 0 > > > > > > This series adds non burst tx internal port workers only. It provides > > > infrastructure for non internal port workers, however does not define > > > any. Also, only inline ipsec protocol mode is supported by the worker threads > > added. > > > > > > Following are planned features, > > > 1. Add burst mode workers. > > > 2. Add non internal port workers. > > > 3. Verify support for Rx core (the support is added but lack of h/w to verify). > > > 4. Add lookaside protocol support. > > > > > > Following are features that Marvell won't be attempting. > > > 1. Inline crypto support. > > > 2. Lookaside crypto support. > > > > > > For the features that Marvell won't be attempting, new workers can be > > > introduced by the respective stake holders. > > > > > > This series is tested on Marvell OCTEON TX2. > > > This series is targeted for 20.05 release. > > > > > > Changes in v5: > > > * Rename function check_params() to check_poll_mode_params() and > > > check_eh_conf() to check_event_mode_params() in order to make it clear > > > what is their purpose. > > > * Forbid usage of --config option in event mode. > > > * Replace magic numbers on return with enum values in > > process_ipsec_ev_inbound() > > > and process_ipsec_ev_outbound() functions. > > > * Add session_priv_pool for both inbound and outbound configuration in > > > ipsec_wrkr_non_burst_int_port_app_mode worker. > > > * Add check of event type in ipsec_wrkr_non_burst_int_port_app_mode > > worker. > > > * Update description of --config option in both ipsec-secgw help and > > documentation. > > > > > > Changes in v4: > > > * Update ipsec-secgw documentation to describe the new options as well as > > > event mode support. > > > * In event mode reserve number of crypto queues equal to number of eth > > ports > > > in order to meet inline protocol offload requirements. > > > * Add calculate_nb_mbufs() function to calculate number of mbufs in a pool > > > and include fragments table size into the calculation. > > > * Move structures ipsec_xf and ipsec_sad to ipsec.h and remove static > > keyword > > > from sa_out, nb_sa_out, sa_in and nb_sa_in in sa.c. > > > * Update process_ipsec_ev_inbound(), process_ipsec_ev_outbound(), > > check_sp() > > > and prepare_out_sessions_tbl() functions as a result of changes introduced > > > by SAD feature. > > > * Remove setting sa->cdev_id_qp to 0 in create_inline_session as sa_ctx > > > is created with rte_zmalloc. > > > * Minor cleanup enhancements: > > > - In eh_set_default_conf_eventdev() function in event_helper.c put > definition > > > of int local vars in one line, remove invalid comment, put > > > "eventdev_config->ev_queue_mode = > RTE_EVENT_QUEUE_CFG_ALL_TYPES" > > in one line > > > instead of two. > > > - Remove extern "C" from event_helper.h. > > > - Put local vars in reverse xmas tree order in eh_dev_has_rx_internal_port() > > and > > > eh_dev_has_tx_internal_port() functions in event_helper.c. > > > - Put #include <rte_bitmap.h> in alphabetical order in ipsec-secgw.c. > > > - Move "extern volatile bool force_quit" and "#include <stdbool.h>" to ipsec- > > secgw.h, > > > remove #include <rte_hash.h>. > > > - Remove not needed includes in ipsec_worker.c. > > > - Remove expired todo from ipsec_worker.h. > > > > > > Changes in v3: > > > * Move eh_conf_init() and eh_conf_uninit() functions to event_helper.c > > > including minor rework. > > > * Rename --schedule-type option to --event-schedule-type. > > > * Replace macro UNPROTECTED_PORT with static inline function > > > is_unprotected_port(). > > > * Move definitions of global variables used by multiple modules > > > to .c files and add externs in .h headers. > > > * Add eh_check_conf() which validates ipsec-secgw configuration > > > for event mode. > > > * Add dynamic calculation of number of buffers in a pool based > > > on number of cores, ports and crypto queues. > > > * Fix segmentation fault in event mode driver worker which happens > > > when there are no inline outbound sessions configured. > > > * Remove change related to updating number of crypto queues > > > in cryptodevs_init(). The update of crypto queues will be handled > > > in a separate patch. > > > * Fix compilation error on 32-bit platforms by using userdata instead > > > of udata64 from rte_mbuf. > > > > > > Changes in v2: > > > * Remove --process-dir option. Instead use existing unprotected port mask > > > option (-u) to decide wheter port handles inbound or outbound traffic. > > > * Remove --process-mode option. Instead use existing --single-sa option > > > to select between app and driver modes. > > > * Add handling of PKT_RX_SEC_OFFLOAD_FAIL result in app worker thread. > > > * Fix passing of req_rx_offload flags to create_default_ipsec_flow(). > > > * Move destruction of flows to a location where eth ports are stopped > > > and closed. > > > * Print error and exit when event mode --schedule-type option is used > > > in poll mode. > > > * Reduce number of goto statements replacing them with loop constructs. > > > * Remove sec_session_fixed table and replace it with locally build > > > table in driver worker thread. Table is indexed by port identifier > > > and holds first inline session pointer found for a given port. > > > * Print error and exit when sessions other than inline are configured > > > in event mode. > > > * When number of event queues is less than number of eth ports then > > > map all eth ports to one event queue. > > > * Cleanup and minor improvements in code as suggested by Konstantin > > > > > > Ankur Dwivedi (1): > > > examples/ipsec-secgw: add default rte flow for inline Rx > > > > > > Anoob Joseph (5): > > > examples/ipsec-secgw: add framework for eventmode helper > > > examples/ipsec-secgw: add eventdev port-lcore link > > > examples/ipsec-secgw: add Rx adapter support > > > examples/ipsec-secgw: add Tx adapter support > > > examples/ipsec-secgw: add routines to display config > > > > > > Lukasz Bartosik (9): > > > examples/ipsec-secgw: add routines to launch workers > > > examples/ipsec-secgw: add support for internal ports > > > examples/ipsec-secgw: add event helper config init/uninit > > > examples/ipsec-secgw: add eventmode to ipsec-secgw > > > examples/ipsec-secgw: add driver mode worker > > > examples/ipsec-secgw: add app mode worker > > > examples/ipsec-secgw: make number of buffers dynamic > > > doc: add event mode support to ipsec-secgw > > > examples/ipsec-secgw: reserve crypto queues in event mode > > > > > > doc/guides/sample_app_ug/ipsec_secgw.rst | 135 ++- > > > examples/ipsec-secgw/Makefile | 2 + > > > examples/ipsec-secgw/event_helper.c | 1812 > > ++++++++++++++++++++++++++++++ > > > examples/ipsec-secgw/event_helper.h | 327 ++++++ > > > examples/ipsec-secgw/ipsec-secgw.c | 506 +++++++-- > > > examples/ipsec-secgw/ipsec-secgw.h | 88 ++ > > > examples/ipsec-secgw/ipsec.c | 5 +- > > > examples/ipsec-secgw/ipsec.h | 53 +- > > > examples/ipsec-secgw/ipsec_worker.c | 649 +++++++++++ > > > examples/ipsec-secgw/ipsec_worker.h | 41 + > > > examples/ipsec-secgw/meson.build | 6 +- > > > examples/ipsec-secgw/sa.c | 21 +- > > > examples/ipsec-secgw/sad.h | 5 - > > > 13 files changed, 3516 insertions(+), 134 deletions(-) create mode > > > 100644 examples/ipsec-secgw/event_helper.c > > > create mode 100644 examples/ipsec-secgw/event_helper.h > > > create mode 100644 examples/ipsec-secgw/ipsec-secgw.h > > > create mode 100644 examples/ipsec-secgw/ipsec_worker.c > > > create mode 100644 examples/ipsec-secgw/ipsec_worker.h > > > > > > -- > > > > Have to say I didn't look extensively on event mode. > > My primary concern was poll-mode and common code changes. > > From that perspective - LGTM. > > > > Series Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com> > > > > > 2.7.4
Hi Akhil, This is release note proposal for event mode feature. diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst index 2190eaf..f8deda7 100644 --- a/doc/guides/rel_notes/release_20_05.rst +++ b/doc/guides/rel_notes/release_20_05.rst @@ -56,6 +56,14 @@ New Features Also, make sure to start the actual text at the margin. ========================================================= +* **Added event mode to ipsec-secgw application ** + + Added event mode to ipsec-secgw application. The ipsec-secgw worker thread(s) + would be receiving events and would be submitting it back to the event device after + the processing. This way, multicore scaling and HW assisted scheduling is achieved + by making use of the event device capabilities. The event mode currently supports + only inline IPsec protocol offload. + Removed Items ------------- Thanks, Lukasz On 12.03.2020 06:55, Akhil Goyal wrote: > External Email > > ---------------------------------------------------------------------- > Hi Anoob, > > Please send a release note update as a reply to this mail. I will update it while merging the patchset. > > Regards, > Akhil >> >> Hi Akhil, >> >> Reminder. >> >> Do you have any further review comments? >> >> Thanks, >> Anoob >> >>> -----Original Message----- >>> From: Ananyev, Konstantin <konstantin.ananyev@intel.com> >>> Sent: Tuesday, March 3, 2020 11:30 PM >>> To: Lukas Bartosik <lbartosik@marvell.com>; Akhil Goyal >>> <akhil.goyal@nxp.com>; Nicolau, Radu <radu.nicolau@intel.com>; Thomas >>> Monjalon <thomas@monjalon.net> >>> Cc: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Narayana Prasad Raju >>> Athreya <pathreya@marvell.com>; Ankur Dwivedi <adwivedi@marvell.com>; >>> Anoob Joseph <anoobj@marvell.com>; Archana Muniganti >>> <marchana@marvell.com>; Tejasree Kondoj <ktejasree@marvell.com>; Vamsi >>> Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org >>> Subject: [EXT] RE: [PATCH v5 00/15] add eventmode to ipsec-secgw >>> >>> External Email >>> >>> ---------------------------------------------------------------------- >>> >>>> >>>> This series introduces event-mode additions to ipsec-secgw. >>>> >>>> With this series, ipsec-secgw would be able to run in eventmode. The >>>> worker thread (executing loop) would be receiving events and would be >>>> submitting it back to the eventdev after the processing. This way, >>>> multicore scaling and h/w assisted scheduling is achieved by making >>>> use of the eventdev capabilities. >>>> >>>> Since the underlying event device would be having varying >>>> capabilities, the worker thread could be drafted differently to maximize >>> performance. >>>> This series introduces usage of multiple worker threads, among which >>>> the one to be used will be determined by the operating conditions and >>>> the underlying device capabilities. >>>> >>>> For example, if an event device - eth device pair has Tx internal >>>> port, then application can do tx_adapter_enqueue() instead of regular >>>> event_enqueue(). So a thread making an assumption that the device pair >>>> has internal port will not be the right solution for another pair. The >>>> infrastructure added with these patches aims to help application to >>>> have multiple worker threads, there by extracting maximum performance >>>> from every device without affecting existing paths/use cases. >>>> >>>> The eventmode configuration is predefined. All packets reaching one >>>> eth port will hit one event queue. All event queues will be mapped to >>>> all event ports. So all cores will be able to receive traffic from all ports. >>>> When schedule_type is set as RTE_SCHED_TYPE_ORDERED/ATOMIC, event >>>> device will ensure the ordering. Ordering would be lost when tried in >>> PARALLEL. >>>> >>>> Following command line options are introduced, >>>> >>>> --transfer-mode: to choose between poll mode & event mode >>>> --event-schedule-type: to specify the scheduling type >>>> (RTE_SCHED_TYPE_ORDERED/ >>>> RTE_SCHED_TYPE_ATOMIC/ >>>> RTE_SCHED_TYPE_PARALLEL) >>>> >>>> Additionally the event mode introduces two modes of processing packets: >>>> >>>> Driver-mode: This mode will have bare minimum changes in the application >>>> to support ipsec. There woudn't be any lookup etc done in >>>> the application. And for inline-protocol use case, the >>>> thread would resemble l2fwd as the ipsec processing would be >>>> done entirely in the h/w. This mode can be used to benchmark >>>> the raw performance of the h/w. All the application side >>>> steps (like lookup) can be redone based on the requirement >>>> of the end user. Hence the need for a mode which would >>>> report the raw performance. >>>> >>>> App-mode: This mode will have all the features currently implemented with >>>> ipsec-secgw (non librte_ipsec mode). All the lookups etc >>>> would follow the existing methods and would report numbers >>>> that can be compared against regular ipsec-secgw benchmark >>>> numbers. >>>> >>>> The driver mode is selected with existing --single-sa option (used >>>> also by poll mode). When --single-sa option is used in conjution with >>>> event mode then index passed to --single-sa is ignored. >>>> >>>> Example commands to execute ipsec-secgw in various modes on OCTEON >> TX2 >>>> platform, >>>> >>>> #Inbound and outbound app mode >>>> ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w >>>> 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 >>>> --log-level=8 -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg >>>> --transfer-mode event --event-schedule-type parallel >>>> >>>> #Inbound and outbound driver mode >>>> ipsec-secgw -w 0002:02:00.0,ipsec_in_max_spi=128 -w >>>> 0002:03:00.0,ipsec_in_max_spi=128 -w 0002:0e:00.0 -w 0002:10:00.1 >>>> --log-level=8 -c 0x1 -- -P -p 0x3 -u 0x1 -f aes-gcm.cfg >>>> --transfer-mode event --event-schedule-type parallel --single-sa 0 >>>> >>>> This series adds non burst tx internal port workers only. It provides >>>> infrastructure for non internal port workers, however does not define >>>> any. Also, only inline ipsec protocol mode is supported by the worker threads >>> added. >>>> >>>> Following are planned features, >>>> 1. Add burst mode workers. >>>> 2. Add non internal port workers. >>>> 3. Verify support for Rx core (the support is added but lack of h/w to verify). >>>> 4. Add lookaside protocol support. >>>> >>>> Following are features that Marvell won't be attempting. >>>> 1. Inline crypto support. >>>> 2. Lookaside crypto support. >>>> >>>> For the features that Marvell won't be attempting, new workers can be >>>> introduced by the respective stake holders. >>>> >>>> This series is tested on Marvell OCTEON TX2. >>>> This series is targeted for 20.05 release. >>>> >>>> Changes in v5: >>>> * Rename function check_params() to check_poll_mode_params() and >>>> check_eh_conf() to check_event_mode_params() in order to make it clear >>>> what is their purpose. >>>> * Forbid usage of --config option in event mode. >>>> * Replace magic numbers on return with enum values in >>> process_ipsec_ev_inbound() >>>> and process_ipsec_ev_outbound() functions. >>>> * Add session_priv_pool for both inbound and outbound configuration in >>>> ipsec_wrkr_non_burst_int_port_app_mode worker. >>>> * Add check of event type in ipsec_wrkr_non_burst_int_port_app_mode >>> worker. >>>> * Update description of --config option in both ipsec-secgw help and >>> documentation. >>>> >>>> Changes in v4: >>>> * Update ipsec-secgw documentation to describe the new options as well as >>>> event mode support. >>>> * In event mode reserve number of crypto queues equal to number of eth >>> ports >>>> in order to meet inline protocol offload requirements. >>>> * Add calculate_nb_mbufs() function to calculate number of mbufs in a pool >>>> and include fragments table size into the calculation. >>>> * Move structures ipsec_xf and ipsec_sad to ipsec.h and remove static >>> keyword >>>> from sa_out, nb_sa_out, sa_in and nb_sa_in in sa.c. >>>> * Update process_ipsec_ev_inbound(), process_ipsec_ev_outbound(), >>> check_sp() >>>> and prepare_out_sessions_tbl() functions as a result of changes introduced >>>> by SAD feature. >>>> * Remove setting sa->cdev_id_qp to 0 in create_inline_session as sa_ctx >>>> is created with rte_zmalloc. >>>> * Minor cleanup enhancements: >>>> - In eh_set_default_conf_eventdev() function in event_helper.c put >> definition >>>> of int local vars in one line, remove invalid comment, put >>>> "eventdev_config->ev_queue_mode = >> RTE_EVENT_QUEUE_CFG_ALL_TYPES" >>> in one line >>>> instead of two. >>>> - Remove extern "C" from event_helper.h. >>>> - Put local vars in reverse xmas tree order in eh_dev_has_rx_internal_port() >>> and >>>> eh_dev_has_tx_internal_port() functions in event_helper.c. >>>> - Put #include <rte_bitmap.h> in alphabetical order in ipsec-secgw.c. >>>> - Move "extern volatile bool force_quit" and "#include <stdbool.h>" to ipsec- >>> secgw.h, >>>> remove #include <rte_hash.h>. >>>> - Remove not needed includes in ipsec_worker.c. >>>> - Remove expired todo from ipsec_worker.h. >>>> >>>> Changes in v3: >>>> * Move eh_conf_init() and eh_conf_uninit() functions to event_helper.c >>>> including minor rework. >>>> * Rename --schedule-type option to --event-schedule-type. >>>> * Replace macro UNPROTECTED_PORT with static inline function >>>> is_unprotected_port(). >>>> * Move definitions of global variables used by multiple modules >>>> to .c files and add externs in .h headers. >>>> * Add eh_check_conf() which validates ipsec-secgw configuration >>>> for event mode. >>>> * Add dynamic calculation of number of buffers in a pool based >>>> on number of cores, ports and crypto queues. >>>> * Fix segmentation fault in event mode driver worker which happens >>>> when there are no inline outbound sessions configured. >>>> * Remove change related to updating number of crypto queues >>>> in cryptodevs_init(). The update of crypto queues will be handled >>>> in a separate patch. >>>> * Fix compilation error on 32-bit platforms by using userdata instead >>>> of udata64 from rte_mbuf. >>>> >>>> Changes in v2: >>>> * Remove --process-dir option. Instead use existing unprotected port mask >>>> option (-u) to decide wheter port handles inbound or outbound traffic. >>>> * Remove --process-mode option. Instead use existing --single-sa option >>>> to select between app and driver modes. >>>> * Add handling of PKT_RX_SEC_OFFLOAD_FAIL result in app worker thread. >>>> * Fix passing of req_rx_offload flags to create_default_ipsec_flow(). >>>> * Move destruction of flows to a location where eth ports are stopped >>>> and closed. >>>> * Print error and exit when event mode --schedule-type option is used >>>> in poll mode. >>>> * Reduce number of goto statements replacing them with loop constructs. >>>> * Remove sec_session_fixed table and replace it with locally build >>>> table in driver worker thread. Table is indexed by port identifier >>>> and holds first inline session pointer found for a given port. >>>> * Print error and exit when sessions other than inline are configured >>>> in event mode. >>>> * When number of event queues is less than number of eth ports then >>>> map all eth ports to one event queue. >>>> * Cleanup and minor improvements in code as suggested by Konstantin >>>> >>>> Ankur Dwivedi (1): >>>> examples/ipsec-secgw: add default rte flow for inline Rx >>>> >>>> Anoob Joseph (5): >>>> examples/ipsec-secgw: add framework for eventmode helper >>>> examples/ipsec-secgw: add eventdev port-lcore link >>>> examples/ipsec-secgw: add Rx adapter support >>>> examples/ipsec-secgw: add Tx adapter support >>>> examples/ipsec-secgw: add routines to display config >>>> >>>> Lukasz Bartosik (9): >>>> examples/ipsec-secgw: add routines to launch workers >>>> examples/ipsec-secgw: add support for internal ports >>>> examples/ipsec-secgw: add event helper config init/uninit >>>> examples/ipsec-secgw: add eventmode to ipsec-secgw >>>> examples/ipsec-secgw: add driver mode worker >>>> examples/ipsec-secgw: add app mode worker >>>> examples/ipsec-secgw: make number of buffers dynamic >>>> doc: add event mode support to ipsec-secgw >>>> examples/ipsec-secgw: reserve crypto queues in event mode >>>> >>>> doc/guides/sample_app_ug/ipsec_secgw.rst | 135 ++- >>>> examples/ipsec-secgw/Makefile | 2 + >>>> examples/ipsec-secgw/event_helper.c | 1812 >>> ++++++++++++++++++++++++++++++ >>>> examples/ipsec-secgw/event_helper.h | 327 ++++++ >>>> examples/ipsec-secgw/ipsec-secgw.c | 506 +++++++-- >>>> examples/ipsec-secgw/ipsec-secgw.h | 88 ++ >>>> examples/ipsec-secgw/ipsec.c | 5 +- >>>> examples/ipsec-secgw/ipsec.h | 53 +- >>>> examples/ipsec-secgw/ipsec_worker.c | 649 +++++++++++ >>>> examples/ipsec-secgw/ipsec_worker.h | 41 + >>>> examples/ipsec-secgw/meson.build | 6 +- >>>> examples/ipsec-secgw/sa.c | 21 +- >>>> examples/ipsec-secgw/sad.h | 5 - >>>> 13 files changed, 3516 insertions(+), 134 deletions(-) create mode >>>> 100644 examples/ipsec-secgw/event_helper.c >>>> create mode 100644 examples/ipsec-secgw/event_helper.h >>>> create mode 100644 examples/ipsec-secgw/ipsec-secgw.h >>>> create mode 100644 examples/ipsec-secgw/ipsec_worker.c >>>> create mode 100644 examples/ipsec-secgw/ipsec_worker.h >>>> >>>> -- >>> >>> Have to say I didn't look extensively on event mode. >>> My primary concern was poll-mode and common code changes. >>> From that perspective - LGTM. >>> >>> Series Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com> >>> >>>> 2.7.4 >
> Hi Akhil, > > This is release note proposal for event mode feature. > > > > diff --git a/doc/guides/rel_notes/release_20_05.rst > b/doc/guides/rel_notes/release_20_05.rst > index 2190eaf..f8deda7 100644 > --- a/doc/guides/rel_notes/release_20_05.rst > +++ b/doc/guides/rel_notes/release_20_05.rst > @@ -56,6 +56,14 @@ New Features > Also, make sure to start the actual text at the margin. > ========================================================= > > +* **Added event mode to ipsec-secgw application ** > + > + Added event mode to ipsec-secgw application. The ipsec-secgw worker > thread(s) > + would be receiving events and would be submitting it back to the event device > after > + the processing. This way, multicore scaling and HW assisted scheduling is > achieved > + by making use of the event device capabilities. The event mode currently > supports > + only inline IPsec protocol offload. > + Modified it as below. +* **Added event mode to ipsec-secgw application.** + + Updated ipsec-secgw application to add event based packet processing. The worker + thread(s) would receive events and submit them back to the event device after + the processing. This way, multicore scaling and HW assisted scheduling is achieved + by making use of the event device capabilities. The event mode currently supports + only inline IPsec protocol offload. + > >>> Series Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com> Series Acked-by: Akhil Goyal <akhil.goyal@nxp.com> Applied to dpdk-next-crypto I may do some minor changes later while submitting pull request to master. Thanks.