[00/39] adding eventmode helper library

  This series adds support for eventmode helper library and l2fwd-event
application.

First 13 patches creates a new l2fwd application (l2fwd-event). Minor
code reorganization is done to faciliate seamless integration of
eventmode.

Next 22 patches adds eventmode helper library. This library abstracts
the configuration of event device & Rx-Tx event adapters. The library
can be extended to allow users to control all the configuration
exposed by adapters and eth device.

Last 4 patches implements eventmode in l2fwd-event application. With
event device and adapters, fine tuned threads (based on dev
capabilities) can be drafted to maximize performance. Eventmode
library facilitates this and l2fwd-event demonstrates this usage.

With the introduction of eventmode helper library, any poll mode
application can be converted to an eventmode application with simple
steps, enabling multi-core scaling and dynamic load balancing to
various example applications.

Usage:
     ./l2fwd-event -- <EAL args> -- <l2fwd args> -- --transfer-mode 1

The above command would invoke eventmode and with the default conf
loaded, traffic on one port would be delivered to all enabled cores.

Planned features,
1. Eventmode helper library doesn't intialize ethdev. Since all
   applications already do this, eventmode helper would start
   from reconfiguring.
2. All features of eventdev and adapters can be exposed to the user
   using common CL arguments. The framework for achieving the same
   is already in place. It has to be extended to support more
   features.
3. Documentation is pending.

Created new app based on discussions,
http://patchwork.dpdk.org/cover/40884/
https://patches.dpdk.org/patch/40901/

Tested with nicvf eth PMD and event_octeontx event PMD on Marvell's
CN83XX platform.

Anoob Joseph (39):
  examples/l2fwd-event: create copy of l2fwd
  examples/l2fwd-event: move macros to common header
  examples/l2fwd-event: move structures to common header
  examples/l2fwd-event: move global vars to common header
  examples/l2fwd-event: move dataplane code to new file
  examples/l2fwd-event: remove unused header includes
  examples/l2fwd-event: move drain buffers to new function
  examples/l2fwd-event: optimize check for master core
  examples/l2fwd-event: move periodic tasks to new func
  examples/l2fwd-event: do timer updates only on master
  examples/l2fwd-event: move pkt send code to a new func
  examples/l2fwd-event: use fprintf in usage print
  examples/l2fwd-event: improvements to the usage print
  eventdev: add files for eventmode helper
  eventdev: add routines for logging eventmode helper
  eventdev: add eventmode CL options framework
  eventdev: allow application to set ethernet portmask
  eventdev: add framework for eventmode conf
  eventdev: add common initialize routine for eventmode devs
  eventdev: add eventdevice init for eventmode
  eventdev: add eventdev port-lcore link
  eventdev: add option to specify schedule mode for app stage
  eventdev: add placeholder for ethdev init
  eventdev: add Rx adapter init in eventmode
  eventdev: add routine to validate conf
  eventdev: add default conf for event devs field in conf
  eventdev: add default conf for Rx adapter conf
  eventdev: add default conf for event port-lcore link
  eventdev: add routines to display the eventmode conf
  eventdev: add routine to access eventmode link info
  eventdev: add routine to access event queue for eth Tx
  eventdev: add routine to launch eventmode workers
  eventdev: add Tx adapter support
  eventdev: add support for internal ports
  eventdev: display Tx adapter conf
  examples/l2fwd-event: add eventmode for l2fwd
  examples/l2fwd-event: add eventmode worker
  examples/l2fwd-event: add eventmode worker
  examples/l2fwd-event: add eventmode worker

 config/common_base                                 |    1 +
 examples/Makefile                                  |    1 +
 examples/l2fwd-event/Makefile                      |   57 +
 examples/l2fwd-event/l2fwd_common.h                |   63 +
 examples/l2fwd-event/l2fwd_worker.c                | 1121 +++++++++++++
 examples/l2fwd-event/l2fwd_worker.h                |   16 +
 examples/l2fwd-event/main.c                        |  585 +++++++
 examples/l2fwd-event/meson.build                   |   14 +
 lib/librte_eal/common/eal_common_log.c             |    1 +
 lib/librte_eal/common/include/rte_log.h            |    1 +
 lib/librte_eventdev/Makefile                       |    6 +-
 lib/librte_eventdev/meson.build                    |    3 +
 lib/librte_eventdev/rte_eventdev_version.map       |    8 +
 lib/librte_eventdev/rte_eventmode_helper.c         | 1678 ++++++++++++++++++++
 lib/librte_eventdev/rte_eventmode_helper.h         |  241 +++
 .../rte_eventmode_helper_internal.h                |  144 ++
 lib/librte_eventdev/rte_eventmode_helper_prints.c  |  223 +++
 17 files changed, 4161 insertions(+), 2 deletions(-)
 create mode 100644 examples/l2fwd-event/Makefile
 create mode 100644 examples/l2fwd-event/l2fwd_common.h
 create mode 100644 examples/l2fwd-event/l2fwd_worker.c
 create mode 100644 examples/l2fwd-event/l2fwd_worker.h
 create mode 100644 examples/l2fwd-event/main.c
 create mode 100644 examples/l2fwd-event/meson.build
 create mode 100644 lib/librte_eventdev/rte_eventmode_helper.c
 create mode 100644 lib/librte_eventdev/rte_eventmode_helper.h
 create mode 100644 lib/librte_eventdev/rte_eventmode_helper_internal.h
 create mode 100644 lib/librte_eventdev/rte_eventmode_helper_prints.c
  

> -----Original Message-----
> From: Anoob Joseph <anoobj@marvell.com>
> Sent: Monday, June 3, 2019 11:02 PM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Nikhil Rao
> <nikhil.rao@intel.com>; Erik Gabriel Carrillo <erik.g.carrillo@intel.com>;
> Abhinandan Gujjar <abhinandan.gujjar@intel.com>; Bruce Richardson
> <bruce.richardson@intel.com>; Pablo de Lara
> <pablo.de.lara.guarch@intel.com>
> Cc: Anoob Joseph <anoobj@marvell.com>; Narayana Prasad Raju Athreya
> <pathreya@marvell.com>; dev@dpdk.org; Lukas Bartosik
> <lbartosik@marvell.com>; Pavan Nikhilesh Bhagavatula
> <pbhagavatula@marvell.com>; Hemant Agrawal
> <hemant.agrawal@nxp.com>; Nipun Gupta <nipun.gupta@nxp.com>; Harry
> van Haaren <harry.van.haaren@intel.com>; Mattias Rönnblom
> <mattias.ronnblom@ericsson.com>; Liang Ma <liang.j.ma@intel.com>
> Subject: [PATCH 00/39] adding eventmode helper library
> 
> This series adds support for eventmode helper library and l2fwd-event
> application.
> 
> First 13 patches creates a new l2fwd application (l2fwd-event). Minor code
> reorganization is done to faciliate seamless integration of eventmode.
> 
> Next 22 patches adds eventmode helper library. This library abstracts the
> configuration of event device & Rx-Tx event adapters. The library can be
> extended to allow users to control all the configuration exposed by adapters
> and eth device.
> 
> Last 4 patches implements eventmode in l2fwd-event application. With
> event device and adapters, fine tuned threads (based on dev
> capabilities) can be drafted to maximize performance. Eventmode library
> facilitates this and l2fwd-event demonstrates this usage.
> 
> With the introduction of eventmode helper library, any poll mode application
> can be converted to an eventmode application with simple steps, enabling
> multi-core scaling and dynamic load balancing to various example
> applications.


Anyone planning to review this changes?
I will spend time to review this. Requesting the review from other eventdev stake holders.
  

On 2019-06-07 11:48, Jerin Jacob Kollanukkaran wrote:
>> -----Original Message-----
>> From: Anoob Joseph <anoobj@marvell.com>
>> Sent: Monday, June 3, 2019 11:02 PM
>> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Nikhil Rao
>> <nikhil.rao@intel.com>; Erik Gabriel Carrillo <erik.g.carrillo@intel.com>;
>> Abhinandan Gujjar <abhinandan.gujjar@intel.com>; Bruce Richardson
>> <bruce.richardson@intel.com>; Pablo de Lara
>> <pablo.de.lara.guarch@intel.com>
>> Cc: Anoob Joseph <anoobj@marvell.com>; Narayana Prasad Raju Athreya
>> <pathreya@marvell.com>; dev@dpdk.org; Lukas Bartosik
>> <lbartosik@marvell.com>; Pavan Nikhilesh Bhagavatula
>> <pbhagavatula@marvell.com>; Hemant Agrawal
>> <hemant.agrawal@nxp.com>; Nipun Gupta <nipun.gupta@nxp.com>; Harry
>> van Haaren <harry.van.haaren@intel.com>; Mattias Rönnblom
>> <mattias.ronnblom@ericsson.com>; Liang Ma <liang.j.ma@intel.com>
>> Subject: [PATCH 00/39] adding eventmode helper library
>>
>> This series adds support for eventmode helper library and l2fwd-event
>> application.
>>
>> First 13 patches creates a new l2fwd application (l2fwd-event). Minor code
>> reorganization is done to faciliate seamless integration of eventmode.
>>
>> Next 22 patches adds eventmode helper library. This library abstracts the
>> configuration of event device & Rx-Tx event adapters. The library can be
>> extended to allow users to control all the configuration exposed by adapters
>> and eth device.
>>
>> Last 4 patches implements eventmode in l2fwd-event application. With
>> event device and adapters, fine tuned threads (based on dev
>> capabilities) can be drafted to maximize performance. Eventmode library
>> facilitates this and l2fwd-event demonstrates this usage.
>>
>> With the introduction of eventmode helper library, any poll mode application
>> can be converted to an eventmode application with simple steps, enabling
>> multi-core scaling and dynamic load balancing to various example
>> applications.
> 
> 
> Anyone planning to review this changes?
> I will spend time to review this. Requesting the review from other eventdev stake holders.
> 

A more extensive description of the purpose of the eventmode helper 
library would be helpful.

Is this supposed to be a generic framework for real-world applications, 
or only something to simplify DPDK the implementation of DPDK example 
programs and similar?
  

Hi Mattias,
 
> A more extensive description of the purpose of the eventmode helper 
> library would be helpful.
> 
> Is this supposed to be a generic framework for real-world 
> applications, or only something to simplify DPDK the implementation of 
> DPDK example programs and similar?
 
This is intended as a generic framework, but the initial targets would be limited to DPDK example applications.
 
For any application to use an event device for dynamic load balancing, it has to configure the event device and the adapters. Configuring the adapters would involve providing various parameters based on which the dynamic scheduling should happen. But requiring the application to do all this configuration would make the application complicated as well as the same code has to be repeated for a new application. Event mode helper tries to solve that.
 
All the complex configuration would be implemented by the helper library and the helper library would provide a default conf as well. 
 
These patches facilitate event mode configuration in a easy to use manner. My idea is that, for a poll mode DPDK example to operate in event mode, a couple of helper functions and a lean worker thread should suffice. So even complex DPDK examples and real world applications will benefit from this helper library. We plan to propose a change to ipsec-secgw to operate in event mode once this proposal is merged.
 
I'll update the cover-letter to add above details when sending v2.

Thanks,
Anoob

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Mattias Rönnblom
> Sent: Tuesday, June 11, 2019 4:14 PM
> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Anoob Joseph
> <anoobj@marvell.com>; Nikhil Rao <nikhil.rao@intel.com>; Erik Gabriel
> Carrillo <erik.g.carrillo@intel.com>; Abhinandan Gujjar
> <abhinandan.gujjar@intel.com>; Bruce Richardson
> <bruce.richardson@intel.com>; Pablo de Lara
> <pablo.de.lara.guarch@intel.com>
> Cc: Narayana Prasad Raju Athreya <pathreya@marvell.com>; dev@dpdk.org;
> Lukas Bartosik <lbartosik@marvell.com>; Pavan Nikhilesh Bhagavatula
> <pbhagavatula@marvell.com>; Hemant Agrawal
> <hemant.agrawal@nxp.com>; Nipun Gupta <nipun.gupta@nxp.com>; Harry
> van Haaren <harry.van.haaren@intel.com>; Liang Ma
> <liang.j.ma@intel.com>
> Subject: [EXT] Re: [dpdk-dev] [PATCH 00/39] adding eventmode helper
> library
> 
> External Email
> 
> ----------------------------------------------------------------------
> On 2019-06-07 11:48, Jerin Jacob Kollanukkaran wrote:
> >> -----Original Message-----
> >> From: Anoob Joseph <anoobj@marvell.com>
> >> Sent: Monday, June 3, 2019 11:02 PM
> >> To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Nikhil Rao
> >> <nikhil.rao@intel.com>; Erik Gabriel Carrillo
> >> <erik.g.carrillo@intel.com>; Abhinandan Gujjar
> >> <abhinandan.gujjar@intel.com>; Bruce Richardson
> >> <bruce.richardson@intel.com>; Pablo de Lara
> >> <pablo.de.lara.guarch@intel.com>
> >> Cc: Anoob Joseph <anoobj@marvell.com>; Narayana Prasad Raju Athreya
> >> <pathreya@marvell.com>; dev@dpdk.org; Lukas Bartosik
> >> <lbartosik@marvell.com>; Pavan Nikhilesh Bhagavatula
> >> <pbhagavatula@marvell.com>; Hemant Agrawal
> <hemant.agrawal@nxp.com>;
> >> Nipun Gupta <nipun.gupta@nxp.com>; Harry van Haaren
> >> <harry.van.haaren@intel.com>; Mattias Rönnblom
> >> <mattias.ronnblom@ericsson.com>; Liang Ma <liang.j.ma@intel.com>
> >> Subject: [PATCH 00/39] adding eventmode helper library
> >>
> >> This series adds support for eventmode helper library and l2fwd-event
> >> application.
> >>
> >> First 13 patches creates a new l2fwd application (l2fwd-event). Minor
> >> code reorganization is done to faciliate seamless integration of
> eventmode.
> >>
> >> Next 22 patches adds eventmode helper library. This library abstracts
> >> the configuration of event device & Rx-Tx event adapters. The library
> >> can be extended to allow users to control all the configuration
> >> exposed by adapters and eth device.
> >>
> >> Last 4 patches implements eventmode in l2fwd-event application. With
> >> event device and adapters, fine tuned threads (based on dev
> >> capabilities) can be drafted to maximize performance. Eventmode
> >> library facilitates this and l2fwd-event demonstrates this usage.
> >>
> >> With the introduction of eventmode helper library, any poll mode
> >> application can be converted to an eventmode application with simple
> >> steps, enabling multi-core scaling and dynamic load balancing to
> >> various example applications.
> >
> >
> > Anyone planning to review this changes?
> > I will spend time to review this. Requesting the review from other
> eventdev stake holders.
> >
> 
> A more extensive description of the purpose of the eventmode helper
> library would be helpful.
> 
> Is this supposed to be a generic framework for real-world applications, or
> only something to simplify DPDK the implementation of DPDK example
> programs and similar?
  

On 2019-06-14 11:18, Anoob Joseph wrote:
> Hi Mattias,
>   
>> A more extensive description of the purpose of the eventmode helper
>> library would be helpful.
>>
>> Is this supposed to be a generic framework for real-world
>> applications, or only something to simplify DPDK the implementation of
>> DPDK example programs and similar?
>   
> This is intended as a generic framework, but the initial targets would be limited to DPDK example applications.
>   
> For any application to use an event device for dynamic load balancing, it has to configure the event device and the adapters. Configuring the adapters would involve providing various parameters based on which the dynamic scheduling should happen. But requiring the application to do all this configuration would make the application complicated as well as the same code has to be repeated for a new application. Event mode helper tries to solve that.
>   
> All the complex configuration would be implemented by the helper library and the helper library would provide a default conf as well.
> 

The task of configuring eventdev and its adaptors, and ethernet devices 
is a daunting task indeed. If we could simplify that, that would be great.

However, the flexibility and many of the parameters are there for a 
reason (those there aren't should be deprecated). I would expect a 
real-world application to tweak quite a few of them. I know our 
applications do.

I worry I have is that if you put eventmode (in its current form) 
forward as a generic framework, applications might start using it, only 
to realize it's not flexible enough, and then eventmode is just an extra 
layer, increasing rather than reducing complexity. Or even worse, the 
application's developers are forced to do a big-bang switch over to 
using the event and ethernet device APIs directly, in case they can't 
patch DPDK to work around the 
eventmode-assumption-that-didn't-hold-for-them.

You could always add flexibility to the framework (as you encounter a 
need for it), but then it will grow in complexity as well.

A less ambitious approach would be to instead do a properly modularized, 
non-trivial eventdev example application, for the applications to start 
off from, instead of a generic library.

I would expect it to be very difficult to design a truly generic 
application framework for eventdev-based applications. Such a framework 
would tie everything that's needed in a non-trivial application 
together. If successful, it would be a huge step toward making DPDK an 
operating system for packet processing applications.

What event devices have you tested with?
  

Hi Mattias,

Please see my response inline.

Thanks,
Anoob

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Mattias Rönnblom
> Sent: Monday, June 17, 2019 6:54 PM
> To: Anoob Joseph <anoobj@marvell.com>; Jerin Jacob Kollanukkaran
> <jerinj@marvell.com>; Nikhil Rao <nikhil.rao@intel.com>; Erik Gabriel Carrillo
> <erik.g.carrillo@intel.com>; Abhinandan Gujjar
> <abhinandan.gujjar@intel.com>; Bruce Richardson
> <bruce.richardson@intel.com>; Pablo de Lara
> <pablo.de.lara.guarch@intel.com>
> Cc: Narayana Prasad Raju Athreya <pathreya@marvell.com>; dev@dpdk.org;
> Lukas Bartosik <lbartosik@marvell.com>; Pavan Nikhilesh Bhagavatula
> <pbhagavatula@marvell.com>; Hemant Agrawal
> <hemant.agrawal@nxp.com>; Nipun Gupta <nipun.gupta@nxp.com>; Harry
> van Haaren <harry.van.haaren@intel.com>; Liang Ma
> <liang.j.ma@intel.com>
> Subject: Re: [dpdk-dev] [EXT] Re: [PATCH 00/39] adding eventmode helper
> library
> 
> On 2019-06-14 11:18, Anoob Joseph wrote:
> > Hi Mattias,
> >
> >> A more extensive description of the purpose of the eventmode helper
> >> library would be helpful.
> >>
> >> Is this supposed to be a generic framework for real-world
> >> applications, or only something to simplify DPDK the implementation
> >> of DPDK example programs and similar?
> >
> > This is intended as a generic framework, but the initial targets would be
> limited to DPDK example applications.
> >
> > For any application to use an event device for dynamic load balancing, it has
> to configure the event device and the adapters. Configuring the adapters
> would involve providing various parameters based on which the dynamic
> scheduling should happen. But requiring the application to do all this
> configuration would make the application complicated as well as the same
> code has to be repeated for a new application. Event mode helper tries to
> solve that.
> >
> > All the complex configuration would be implemented by the helper library
> and the helper library would provide a default conf as well.
> >
> 
> The task of configuring eventdev and its adaptors, and ethernet devices is a
> daunting task indeed. If we could simplify that, that would be great.
> 
> However, the flexibility and many of the parameters are there for a reason
> (those there aren't should be deprecated). I would expect a real-world
> application to tweak quite a few of them. I know our applications do.
> 
> I worry I have is that if you put eventmode (in its current form) forward as a
> generic framework, applications might start using it, only to realize it's not
> flexible enough, and then eventmode is just an extra layer, increasing rather
> than reducing complexity. Or even worse, the application's developers are
> forced to do a big-bang switch over to using the event and ethernet device
> APIs directly, in case they can't patch DPDK to work around the eventmode-
> assumption-that-didn't-hold-for-them.
> 
> You could always add flexibility to the framework (as you encounter a need
> for it), but then it will grow in complexity as well.
> 
> A less ambitious approach would be to instead do a properly modularized,
> non-trivial eventdev example application, for the applications to start off
> from, instead of a generic library.
> 
> I would expect it to be very difficult to design a truly generic application
> framework for eventdev-based applications. Such a framework would tie
> everything that's needed in a non-trivial application together. If successful, it
> would be a huge step toward making DPDK an operating system for packet
> processing applications.

[Anoob] The idea here is not to deprecate any event dev APIs. I do agree that all the configuration exposed by eventdev & adapters are required for various requirements in the real world applications. But the requirement to understand & use all this configuration is making the applications complicated and causes significant effort from anyone who would want to get started with event mode. The idea of helper is to allow an easy framework for applications to get started with eventmode, and then use various options from C/L or config file (both planned) to override the configuration as required. DPDK has components like crypto-scheduler which abstracts lot of configuration and simplify usage from application's perspective. This effort is on similar lines.

My patchset is a followup to http://patches.dpdk.org/patch/37955 , wherein the approach of introducing a helper library for event mode was mooted. The initial patch proposed additions in one application, and that involved huge code additions just for doing the configuration.

The helper library will be experimental while we add event-mode support for other applications like l3fwd & ipsec-secgw. I expect the helper library to be complete over the course of those applications also using the helper library.

> 
> What event devices have you tested with?

[Anoob] Eventmode helper is tested with the following combinations, 
    1. event-octeontx event PMD & nicvf eth PMD
    2. event-octeontx event PMD & eth-octeontx eth PMD