[v2,00/12] preparing l2fwd for eventmode additions

  This patchset modularizes l2fwd application to prepare it for eventmode
additions. This patchset doesn't change the code flow or logic, except
for few minor improvements. Some of the newly added functions are used
in just one place, but is added for efficient usage with eventmode.

v2:
* Merged small patches with others as directed by Pablo

v1:
* Fix all checkpatch reported issues

Anoob Joseph (12):
  examples/l2fwd: move macro definitions to common header
  examples/l2fwd: move structure definitions to common header
  examples/l2fwd: move globally accessed vars to common header
  examples/l2fwd: move dataplane code to new file
  examples/l2fwd: remove unused header includes
  examples/l2fwd: move drain buffers to new function
  examples/l2fwd: optimize check for master core
  examples/l2fwd: move periodic tasks to new function
  examples/l2fwd: skip timer updates for non master cores
  examples/l2fwd: move pkt send code to a new function
  examples/l2fwd: use fprint instead of printf for usage print
  examples/l2fwd: improvements to the usage print

 examples/l2fwd/Makefile       |   1 +
 examples/l2fwd/l2fwd_common.h |  63 ++++++++++
 examples/l2fwd/l2fwd_worker.c | 249 +++++++++++++++++++++++++++++++++++++
 examples/l2fwd/l2fwd_worker.h |  16 +++
 examples/l2fwd/main.c         | 276 ++++++------------------------------------
 5 files changed, 364 insertions(+), 241 deletions(-)
 create mode 100644 examples/l2fwd/l2fwd_common.h
 create mode 100644 examples/l2fwd/l2fwd_worker.c
 create mode 100644 examples/l2fwd/l2fwd_worker.h
  

> Anoob Joseph (12):
>   examples/l2fwd: move macro definitions to common header
>   examples/l2fwd: move structure definitions to common header
>   examples/l2fwd: move globally accessed vars to common header
>   examples/l2fwd: move dataplane code to new file
>   examples/l2fwd: remove unused header includes
>   examples/l2fwd: move drain buffers to new function
>   examples/l2fwd: optimize check for master core
>   examples/l2fwd: move periodic tasks to new function
>   examples/l2fwd: skip timer updates for non master cores
>   examples/l2fwd: move pkt send code to a new function
>   examples/l2fwd: use fprint instead of printf for usage print
>   examples/l2fwd: improvements to the usage print

Maintainers of this app look to be against adding complexity.

In order to get this series accepted, we need more discussions
with more people involved.
So it will miss 18.08.

It can be discussed in a more global discussion about examples maintenance.
If discussion does not happen, you can request it to the technical board.
  

Hi Thomas,

On 26-07-2018 22:27, Thomas Monjalon wrote:
> External Email
>
>> Anoob Joseph (12):
>>    examples/l2fwd: move macro definitions to common header
>>    examples/l2fwd: move structure definitions to common header
>>    examples/l2fwd: move globally accessed vars to common header
>>    examples/l2fwd: move dataplane code to new file
>>    examples/l2fwd: remove unused header includes
>>    examples/l2fwd: move drain buffers to new function
>>    examples/l2fwd: optimize check for master core
>>    examples/l2fwd: move periodic tasks to new function
>>    examples/l2fwd: skip timer updates for non master cores
>>    examples/l2fwd: move pkt send code to a new function
>>    examples/l2fwd: use fprint instead of printf for usage print
>>    examples/l2fwd: improvements to the usage print
> Maintainers of this app look to be against adding complexity.
>
> In order to get this series accepted, we need more discussions
> with more people involved.
> So it will miss 18.08.
>
> It can be discussed in a more global discussion about examples maintenance.
> If discussion does not happen, you can request it to the technical board.
>
Event dev framework and various adapters enable multiple packet handling 
schemes, as opposed to the traditional polling on queues. But these 
features are not integrated into any established example application. 
There are specific example applications for event dev etc, which can be 
used to analyze an event device or a particular eventdev adapter, but 
there is no standard application which can be used to compare the real 
world performance for a system when it's using event device for packet 
handling and when it's done via polling on queues.

The following patch submitted by Sunil was looking to address this issue 
with l3fwd,
https://mails.dpdk.org/archives/dev/2018-March/093131.html

Bruce & Jerin reviewed the patch and suggested the addition of helper 
functions to abstract the event mode additions in applications,
https://mails.dpdk.org/archives/dev/2018-April/096879.html

This effort of adding helper functions for eventmode was taken up 
following the above suggestion. The idea is to add eventmode without 
touching the existing code path. All the eventmode specific additions 
would go into library so that these need not be repeated for every 
application. And since there is no change in the existing code path, 
performance for any vendor should not have any impact with the additions.

The scope of this effort has increased since the submission, as now we 
have Tx adapter as well. Sunil & Konstantin had clarified their 
concerns, and gave green flag to this approach.
https://mails.dpdk.org/archives/dev/2018-June/105730.html
https://mails.dpdk.org/archives/dev/2018-July/106453.html

I guess Bruce was opening this question to the community. For compute 
intense applications like ipsec-secgw, eventmode might be the right 
approach in the first place. Such complex applications would need a 
scheduler to perform dynamic load balancing. Addition of eventmode in 
l2fwd was to float around the idea which can then be scaled for more 
complex applications.

If maintainers doesn't have any objection to this, I'm fine with adding 
this in the next release.

Thanks,
Anoob
  

On Wed, 1 Aug 2018 12:29:50 +0530
"Joseph, Anoob" <Anoob.Joseph@caviumnetworks.com> wrote:

> Hi Thomas,
> 
> On 26-07-2018 22:27, Thomas Monjalon wrote:
> > External Email
> >  
> >> Anoob Joseph (12):
> >>    examples/l2fwd: move macro definitions to common header
> >>    examples/l2fwd: move structure definitions to common header
> >>    examples/l2fwd: move globally accessed vars to common header
> >>    examples/l2fwd: move dataplane code to new file
> >>    examples/l2fwd: remove unused header includes
> >>    examples/l2fwd: move drain buffers to new function
> >>    examples/l2fwd: optimize check for master core
> >>    examples/l2fwd: move periodic tasks to new function
> >>    examples/l2fwd: skip timer updates for non master cores
> >>    examples/l2fwd: move pkt send code to a new function
> >>    examples/l2fwd: use fprint instead of printf for usage print
> >>    examples/l2fwd: improvements to the usage print  
> > Maintainers of this app look to be against adding complexity.
> >
> > In order to get this series accepted, we need more discussions
> > with more people involved.
> > So it will miss 18.08.
> >
> > It can be discussed in a more global discussion about examples maintenance.
> > If discussion does not happen, you can request it to the technical board.
> >  
> Event dev framework and various adapters enable multiple packet handling 
> schemes, as opposed to the traditional polling on queues. But these 
> features are not integrated into any established example application. 
> There are specific example applications for event dev etc, which can be 
> used to analyze an event device or a particular eventdev adapter, but 
> there is no standard application which can be used to compare the real 
> world performance for a system when it's using event device for packet 
> handling and when it's done via polling on queues.
> 
> The following patch submitted by Sunil was looking to address this issue 
> with l3fwd,
> https://mails.dpdk.org/archives/dev/2018-March/093131.html
> 
> Bruce & Jerin reviewed the patch and suggested the addition of helper 
> functions to abstract the event mode additions in applications,
> https://mails.dpdk.org/archives/dev/2018-April/096879.html
> 
> This effort of adding helper functions for eventmode was taken up 
> following the above suggestion. The idea is to add eventmode without 
> touching the existing code path. All the eventmode specific additions 
> would go into library so that these need not be repeated for every 
> application. And since there is no change in the existing code path, 
> performance for any vendor should not have any impact with the additions.
> 
> The scope of this effort has increased since the submission, as now we 
> have Tx adapter as well. Sunil & Konstantin had clarified their 
> concerns, and gave green flag to this approach.
> https://mails.dpdk.org/archives/dev/2018-June/105730.html
> https://mails.dpdk.org/archives/dev/2018-July/106453.html
> 
> I guess Bruce was opening this question to the community. For compute 
> intense applications like ipsec-secgw, eventmode might be the right 
> approach in the first place. Such complex applications would need a 
> scheduler to perform dynamic load balancing. Addition of eventmode in 
> l2fwd was to float around the idea which can then be scaled for more 
> complex applications.
> 
> If maintainers doesn't have any objection to this, I'm fine with adding 
> this in the next release.
> 
> Thanks,
> Anoob

It is important that DPDK has good examples of how to use existing
frameworks and libraries. These applications are what most customers
build their applications from and they provide basis for testing.

The DPDK needs to continue to support multiple usage models. This
is one of its strong points. I would rather leave existing l2fwd
and l3fwd alone and instead make new examples that use the frameworks.
If nothing else haveing l2fwd and l2fwd-eventdev would allow for
performance comparisons.

As the number of examples increases, probably also need to have
a roadmap or decision chart to explain the advangage/disadvantage
of each architecture.
  

-----Original Message-----
> Date: Wed, 1 Aug 2018 09:54:14 -0700
> From: Stephen Hemminger <stephen@networkplumber.org>
> To: "Joseph, Anoob" <Anoob.Joseph@caviumnetworks.com>
> Cc: Thomas Monjalon <thomas@monjalon.net>, dev@dpdk.org, Bruce Richardson
>  <bruce.richardson@intel.com>, Pablo de Lara
>  <pablo.de.lara.guarch@intel.com>, Jerin Jacob
>  <jerin.jacob@caviumnetworks.com>, Narayana Prasad
>  <narayanaprasad.athreya@caviumnetworks.com>, Hemant Agrawal
>  <hemant.agrawal@nxp.com>, "Ananyev, Konstantin"
>  <konstantin.ananyev@intel.com>, Sunil Kumar Kori <sunil.kori@nxp.com>,
>  Nikhil Rao <nikhil.rao@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 00/12] preparing l2fwd for eventmode
>  additions
> 
> 
> On Wed, 1 Aug 2018 12:29:50 +0530
> "Joseph, Anoob" <Anoob.Joseph@caviumnetworks.com> wrote:
> 
> > Hi Thomas,
> >
> > On 26-07-2018 22:27, Thomas Monjalon wrote:
> > > External Email
> > >
> > >> Anoob Joseph (12):
> > >>    examples/l2fwd: move macro definitions to common header
> > >>    examples/l2fwd: move structure definitions to common header
> > >>    examples/l2fwd: move globally accessed vars to common header
> > >>    examples/l2fwd: move dataplane code to new file
> > >>    examples/l2fwd: remove unused header includes
> > >>    examples/l2fwd: move drain buffers to new function
> > >>    examples/l2fwd: optimize check for master core
> > >>    examples/l2fwd: move periodic tasks to new function
> > >>    examples/l2fwd: skip timer updates for non master cores
> > >>    examples/l2fwd: move pkt send code to a new function
> > >>    examples/l2fwd: use fprint instead of printf for usage print
> > >>    examples/l2fwd: improvements to the usage print
> > > Maintainers of this app look to be against adding complexity.
> > >
> > > In order to get this series accepted, we need more discussions
> > > with more people involved.
> > > So it will miss 18.08.
> > >
> > > It can be discussed in a more global discussion about examples maintenance.
> > > If discussion does not happen, you can request it to the technical board.
> > >
> > Event dev framework and various adapters enable multiple packet handling
> > schemes, as opposed to the traditional polling on queues. But these
> > features are not integrated into any established example application.
> > There are specific example applications for event dev etc, which can be
> > used to analyze an event device or a particular eventdev adapter, but
> > there is no standard application which can be used to compare the real
> > world performance for a system when it's using event device for packet
> > handling and when it's done via polling on queues.
> >
> > The following patch submitted by Sunil was looking to address this issue
> > with l3fwd,
> > https://mails.dpdk.org/archives/dev/2018-March/093131.html
> >
> > Bruce & Jerin reviewed the patch and suggested the addition of helper
> > functions to abstract the event mode additions in applications,
> > https://mails.dpdk.org/archives/dev/2018-April/096879.html
> >
> > This effort of adding helper functions for eventmode was taken up
> > following the above suggestion. The idea is to add eventmode without
> > touching the existing code path. All the eventmode specific additions
> > would go into library so that these need not be repeated for every
> > application. And since there is no change in the existing code path,
> > performance for any vendor should not have any impact with the additions.
> >
> > The scope of this effort has increased since the submission, as now we
> > have Tx adapter as well. Sunil & Konstantin had clarified their
> > concerns, and gave green flag to this approach.
> > https://mails.dpdk.org/archives/dev/2018-June/105730.html
> > https://mails.dpdk.org/archives/dev/2018-July/106453.html
> >
> > I guess Bruce was opening this question to the community. For compute
> > intense applications like ipsec-secgw, eventmode might be the right
> > approach in the first place. Such complex applications would need a
> > scheduler to perform dynamic load balancing. Addition of eventmode in
> > l2fwd was to float around the idea which can then be scaled for more
> > complex applications.
> >
> > If maintainers doesn't have any objection to this, I'm fine with adding
> > this in the next release.
> >
> > Thanks,
> > Anoob
> 
> It is important that DPDK has good examples of how to use existing
> frameworks and libraries. These applications are what most customers
> build their applications from and they provide basis for testing.
> 
> The DPDK needs to continue to support multiple usage models. This
> is one of its strong points. I would rather leave existing l2fwd
> and l3fwd alone and instead make new examples that use the frameworks.
> If nothing else haveing l2fwd and l2fwd-eventdev would allow for
> performance comparisons.

Unlike other applications example, there wont be any change in packet
processing functions in eventdev vs poll mode case. Only worker
schematics will change and that can be moved to separated files.
something like worker_poll.c and worker_event.c and both of them
use common packet processing functions using mbuf.

The only disadvantage of having separate application would be packet
processing code duplication. Which is non trivial for l3fwd, IPSec
application IMO.

# Are we fine with code duplication in example application like l3fwd and
IPSec?
# if yes, Are we fine with keeping l2fwd _as is_ to reduce the 
complexity and l2fwd-eventdev supports both modes wherever possible?

> 
> As the number of examples increases, probably also need to have
> a roadmap or decision chart to explain the advangage/disadvantage
> of each architecture.
>
  

Hi everyone,

> > > >
> > > > In order to get this series accepted, we need more discussions
> > > > with more people involved.
> > > > So it will miss 18.08.
> > > >
> > > > It can be discussed in a more global discussion about examples maintenance.
> > > > If discussion does not happen, you can request it to the technical board.
> > > >
> > > Event dev framework and various adapters enable multiple packet handling
> > > schemes, as opposed to the traditional polling on queues. But these
> > > features are not integrated into any established example application.
> > > There are specific example applications for event dev etc, which can be
> > > used to analyze an event device or a particular eventdev adapter, but
> > > there is no standard application which can be used to compare the real
> > > world performance for a system when it's using event device for packet
> > > handling and when it's done via polling on queues.
> > >
> > > The following patch submitted by Sunil was looking to address this issue
> > > with l3fwd,
> > > https://mails.dpdk.org/archives/dev/2018-March/093131.html
> > >
> > > Bruce & Jerin reviewed the patch and suggested the addition of helper
> > > functions to abstract the event mode additions in applications,
> > > https://mails.dpdk.org/archives/dev/2018-April/096879.html
> > >
> > > This effort of adding helper functions for eventmode was taken up
> > > following the above suggestion. The idea is to add eventmode without
> > > touching the existing code path. All the eventmode specific additions
> > > would go into library so that these need not be repeated for every
> > > application. And since there is no change in the existing code path,
> > > performance for any vendor should not have any impact with the additions.
> > >
> > > The scope of this effort has increased since the submission, as now we
> > > have Tx adapter as well. Sunil & Konstantin had clarified their
> > > concerns, and gave green flag to this approach.
> > > https://mails.dpdk.org/archives/dev/2018-June/105730.html
> > > https://mails.dpdk.org/archives/dev/2018-July/106453.html
> > >
> > > I guess Bruce was opening this question to the community. For compute
> > > intense applications like ipsec-secgw, eventmode might be the right
> > > approach in the first place. Such complex applications would need a
> > > scheduler to perform dynamic load balancing. Addition of eventmode in
> > > l2fwd was to float around the idea which can then be scaled for more
> > > complex applications.
> > >
> > > If maintainers doesn't have any objection to this, I'm fine with adding
> > > this in the next release.
> > >
> > > Thanks,
> > > Anoob
> >
> > It is important that DPDK has good examples of how to use existing
> > frameworks and libraries. These applications are what most customers
> > build their applications from and they provide basis for testing.
> >
> > The DPDK needs to continue to support multiple usage models. This
> > is one of its strong points. I would rather leave existing l2fwd
> > and l3fwd alone and instead make new examples that use the frameworks.
> > If nothing else haveing l2fwd and l2fwd-eventdev would allow for
> > performance comparisons.
> 
> Unlike other applications example, there wont be any change in packet
> processing functions in eventdev vs poll mode case. Only worker
> schematics will change and that can be moved to separated files.
> something like worker_poll.c and worker_event.c and both of them
> use common packet processing functions using mbuf.
> 
> The only disadvantage of having separate application would be packet
> processing code duplication. Which is non trivial for l3fwd, IPSec
> application IMO.

Personally I am ok with original design suggestion: 
keep packet processing code common, that would be used by both poll and event modes. 
We could just have a command-line parameter in which mode the app will run.
Another alternative - generate two binaries l2fwd-poll, l2fwd-event (or so).
Konstantin
> 
> # Are we fine with code duplication in example application like l3fwd and
> IPSec?
> # if yes, Are we fine with keeping l2fwd _as is_ to reduce the
> complexity and l2fwd-eventdev supports both modes wherever possible?
> 
> >
> > As the number of examples increases, probably also need to have
> > a roadmap or decision chart to explain the advangage/disadvantage
> > of each architecture.
> >
  

Hi Bruce, Pablo,

If there are no more issues about the approach, can you review the 
patches and give the feedback?

Please do note that this series doesn't add any event mode specific 
code. That will come as a different patch series after incorporating 
Jerin's comments.

Thanks,
Anoob
On 02-08-2018 13:49, Ananyev, Konstantin wrote:
> External Email
>
> Hi everyone,
>
>>>>> In order to get this series accepted, we need more discussions
>>>>> with more people involved.
>>>>> So it will miss 18.08.
>>>>>
>>>>> It can be discussed in a more global discussion about examples maintenance.
>>>>> If discussion does not happen, you can request it to the technical board.
>>>>>
>>>> Event dev framework and various adapters enable multiple packet handling
>>>> schemes, as opposed to the traditional polling on queues. But these
>>>> features are not integrated into any established example application.
>>>> There are specific example applications for event dev etc, which can be
>>>> used to analyze an event device or a particular eventdev adapter, but
>>>> there is no standard application which can be used to compare the real
>>>> world performance for a system when it's using event device for packet
>>>> handling and when it's done via polling on queues.
>>>>
>>>> The following patch submitted by Sunil was looking to address this issue
>>>> with l3fwd,
>>>> https://mails.dpdk.org/archives/dev/2018-March/093131.html
>>>>
>>>> Bruce & Jerin reviewed the patch and suggested the addition of helper
>>>> functions to abstract the event mode additions in applications,
>>>> https://mails.dpdk.org/archives/dev/2018-April/096879.html
>>>>
>>>> This effort of adding helper functions for eventmode was taken up
>>>> following the above suggestion. The idea is to add eventmode without
>>>> touching the existing code path. All the eventmode specific additions
>>>> would go into library so that these need not be repeated for every
>>>> application. And since there is no change in the existing code path,
>>>> performance for any vendor should not have any impact with the additions.
>>>>
>>>> The scope of this effort has increased since the submission, as now we
>>>> have Tx adapter as well. Sunil & Konstantin had clarified their
>>>> concerns, and gave green flag to this approach.
>>>> https://mails.dpdk.org/archives/dev/2018-June/105730.html
>>>> https://mails.dpdk.org/archives/dev/2018-July/106453.html
>>>>
>>>> I guess Bruce was opening this question to the community. For compute
>>>> intense applications like ipsec-secgw, eventmode might be the right
>>>> approach in the first place. Such complex applications would need a
>>>> scheduler to perform dynamic load balancing. Addition of eventmode in
>>>> l2fwd was to float around the idea which can then be scaled for more
>>>> complex applications.
>>>>
>>>> If maintainers doesn't have any objection to this, I'm fine with adding
>>>> this in the next release.
>>>>
>>>> Thanks,
>>>> Anoob
>>> It is important that DPDK has good examples of how to use existing
>>> frameworks and libraries. These applications are what most customers
>>> build their applications from and they provide basis for testing.
>>>
>>> The DPDK needs to continue to support multiple usage models. This
>>> is one of its strong points. I would rather leave existing l2fwd
>>> and l3fwd alone and instead make new examples that use the frameworks.
>>> If nothing else haveing l2fwd and l2fwd-eventdev would allow for
>>> performance comparisons.
>> Unlike other applications example, there wont be any change in packet
>> processing functions in eventdev vs poll mode case. Only worker
>> schematics will change and that can be moved to separated files.
>> something like worker_poll.c and worker_event.c and both of them
>> use common packet processing functions using mbuf.
>>
>> The only disadvantage of having separate application would be packet
>> processing code duplication. Which is non trivial for l3fwd, IPSec
>> application IMO.
> Personally I am ok with original design suggestion:
> keep packet processing code common, that would be used by both poll and event modes.
> We could just have a command-line parameter in which mode the app will run.
> Another alternative - generate two binaries l2fwd-poll, l2fwd-event (or so).
> Konstantin
>> # Are we fine with code duplication in example application like l3fwd and
>> IPSec?
>> # if yes, Are we fine with keeping l2fwd _as is_ to reduce the
>> complexity and l2fwd-eventdev supports both modes wherever possible?
>>
>>> As the number of examples increases, probably also need to have
>>> a roadmap or decision chart to explain the advangage/disadvantage
>>> of each architecture.
>>>
  

On Mon, Aug 13, 2018 at 12:52:19PM +0530, Joseph, Anoob wrote:
> Hi Bruce, Pablo,
> 
> If there are no more issues about the approach, can you review the patches
> and give the feedback?
> 
> Please do note that this series doesn't add any event mode specific code.
> That will come as a different patch series after incorporating Jerin's
> comments.
> 
> Thanks,
> Anoob

My main concern is with l2fwd, rather than l3fwd, which is already fairly
complicated. I could see l3fwd being updated to allow an eventmode without
too many problems.

With l2fwd, the only issue I have is with the volume of code involved.
l2fwd is currently a very simple application which fits in a single file.
With these updates it's no longer such a simple, approachable app, rather
it becomes one which takes a bit of studying a switching between files to
fully understand. The data path is only a very small part of the app, so by
adding an event-based path to the same app we have very little code saving.
Therefore, I think having a separate l2fwd-eventdev would be better for
this case. Two simpler to understand apps is better than one more
complicated on IMHO.

My 2c.

/Bruce

> On 02-08-2018 13:49, Ananyev, Konstantin wrote:
> > External Email
> > 
> > Hi everyone,
> > 
> > > > > > In order to get this series accepted, we need more discussions
> > > > > > with more people involved.
> > > > > > So it will miss 18.08.
> > > > > > 
> > > > > > It can be discussed in a more global discussion about examples maintenance.
> > > > > > If discussion does not happen, you can request it to the technical board.
> > > > > > 
> > > > > Event dev framework and various adapters enable multiple packet handling
> > > > > schemes, as opposed to the traditional polling on queues. But these
> > > > > features are not integrated into any established example application.
> > > > > There are specific example applications for event dev etc, which can be
> > > > > used to analyze an event device or a particular eventdev adapter, but
> > > > > there is no standard application which can be used to compare the real
> > > > > world performance for a system when it's using event device for packet
> > > > > handling and when it's done via polling on queues.
> > > > > 
> > > > > The following patch submitted by Sunil was looking to address this issue
> > > > > with l3fwd,
> > > > > https://mails.dpdk.org/archives/dev/2018-March/093131.html
> > > > > 
> > > > > Bruce & Jerin reviewed the patch and suggested the addition of helper
> > > > > functions to abstract the event mode additions in applications,
> > > > > https://mails.dpdk.org/archives/dev/2018-April/096879.html
> > > > > 
> > > > > This effort of adding helper functions for eventmode was taken up
> > > > > following the above suggestion. The idea is to add eventmode without
> > > > > touching the existing code path. All the eventmode specific additions
> > > > > would go into library so that these need not be repeated for every
> > > > > application. And since there is no change in the existing code path,
> > > > > performance for any vendor should not have any impact with the additions.
> > > > > 
> > > > > The scope of this effort has increased since the submission, as now we
> > > > > have Tx adapter as well. Sunil & Konstantin had clarified their
> > > > > concerns, and gave green flag to this approach.
> > > > > https://mails.dpdk.org/archives/dev/2018-June/105730.html
> > > > > https://mails.dpdk.org/archives/dev/2018-July/106453.html
> > > > > 
> > > > > I guess Bruce was opening this question to the community. For compute
> > > > > intense applications like ipsec-secgw, eventmode might be the right
> > > > > approach in the first place. Such complex applications would need a
> > > > > scheduler to perform dynamic load balancing. Addition of eventmode in
> > > > > l2fwd was to float around the idea which can then be scaled for more
> > > > > complex applications.
> > > > > 
> > > > > If maintainers doesn't have any objection to this, I'm fine with adding
> > > > > this in the next release.
> > > > > 
> > > > > Thanks,
> > > > > Anoob
> > > > It is important that DPDK has good examples of how to use existing
> > > > frameworks and libraries. These applications are what most customers
> > > > build their applications from and they provide basis for testing.
> > > > 
> > > > The DPDK needs to continue to support multiple usage models. This
> > > > is one of its strong points. I would rather leave existing l2fwd
> > > > and l3fwd alone and instead make new examples that use the frameworks.
> > > > If nothing else haveing l2fwd and l2fwd-eventdev would allow for
> > > > performance comparisons.
> > > Unlike other applications example, there wont be any change in packet
> > > processing functions in eventdev vs poll mode case. Only worker
> > > schematics will change and that can be moved to separated files.
> > > something like worker_poll.c and worker_event.c and both of them
> > > use common packet processing functions using mbuf.
> > > 
> > > The only disadvantage of having separate application would be packet
> > > processing code duplication. Which is non trivial for l3fwd, IPSec
> > > application IMO.
> > Personally I am ok with original design suggestion:
> > keep packet processing code common, that would be used by both poll and event modes.
> > We could just have a command-line parameter in which mode the app will run.
> > Another alternative - generate two binaries l2fwd-poll, l2fwd-event (or so).
> > Konstantin
> > > # Are we fine with code duplication in example application like l3fwd and
> > > IPSec?
> > > # if yes, Are we fine with keeping l2fwd _as is_ to reduce the
> > > complexity and l2fwd-eventdev supports both modes wherever possible?
> > > 
> > > > As the number of examples increases, probably also need to have
> > > > a roadmap or decision chart to explain the advangage/disadvantage
> > > > of each architecture.
> > > > 
>
  

Hi Bruce,

The reason why l2fwd was chosen was to allow everyone to chip in their 
ideas while preparing the framework.
This framework would be extended to other applications, hence needed 
enough inputs before expanding to complex applications. If your 
suggestion is to make l3fwd event driven first, I'll start looking in 
that direction.

As for l2fwd, I'm fine with moving event-mode additions to a new app. 
But with the present approach, the app would run in both event mode and 
poll mode.

Your thoughts on renaming the existing app to l2fwd-poll and the 
proposed app as l2fwd?

Thanks,
Anoob
On 13-08-2018 14:57, Bruce Richardson wrote:
> External Email
>
> On Mon, Aug 13, 2018 at 12:52:19PM +0530, Joseph, Anoob wrote:
>> Hi Bruce, Pablo,
>>
>> If there are no more issues about the approach, can you review the patches
>> and give the feedback?
>>
>> Please do note that this series doesn't add any event mode specific code.
>> That will come as a different patch series after incorporating Jerin's
>> comments.
>>
>> Thanks,
>> Anoob
> My main concern is with l2fwd, rather than l3fwd, which is already fairly
> complicated. I could see l3fwd being updated to allow an eventmode without
> too many problems.
>
> With l2fwd, the only issue I have is with the volume of code involved.
> l2fwd is currently a very simple application which fits in a single file.
> With these updates it's no longer such a simple, approachable app, rather
> it becomes one which takes a bit of studying a switching between files to
> fully understand. The data path is only a very small part of the app, so by
> adding an event-based path to the same app we have very little code saving.
> Therefore, I think having a separate l2fwd-eventdev would be better for
> this case. Two simpler to understand apps is better than one more
> complicated on IMHO.
>
> My 2c.
>
> /Bruce
>
>> On 02-08-2018 13:49, Ananyev, Konstantin wrote:
>>> External Email
>>>
>>> Hi everyone,
>>>
>>>>>>> In order to get this series accepted, we need more discussions
>>>>>>> with more people involved.
>>>>>>> So it will miss 18.08.
>>>>>>>
>>>>>>> It can be discussed in a more global discussion about examples maintenance.
>>>>>>> If discussion does not happen, you can request it to the technical board.
>>>>>>>
>>>>>> Event dev framework and various adapters enable multiple packet handling
>>>>>> schemes, as opposed to the traditional polling on queues. But these
>>>>>> features are not integrated into any established example application.
>>>>>> There are specific example applications for event dev etc, which can be
>>>>>> used to analyze an event device or a particular eventdev adapter, but
>>>>>> there is no standard application which can be used to compare the real
>>>>>> world performance for a system when it's using event device for packet
>>>>>> handling and when it's done via polling on queues.
>>>>>>
>>>>>> The following patch submitted by Sunil was looking to address this issue
>>>>>> with l3fwd,
>>>>>> https://mails.dpdk.org/archives/dev/2018-March/093131.html
>>>>>>
>>>>>> Bruce & Jerin reviewed the patch and suggested the addition of helper
>>>>>> functions to abstract the event mode additions in applications,
>>>>>> https://mails.dpdk.org/archives/dev/2018-April/096879.html
>>>>>>
>>>>>> This effort of adding helper functions for eventmode was taken up
>>>>>> following the above suggestion. The idea is to add eventmode without
>>>>>> touching the existing code path. All the eventmode specific additions
>>>>>> would go into library so that these need not be repeated for every
>>>>>> application. And since there is no change in the existing code path,
>>>>>> performance for any vendor should not have any impact with the additions.
>>>>>>
>>>>>> The scope of this effort has increased since the submission, as now we
>>>>>> have Tx adapter as well. Sunil & Konstantin had clarified their
>>>>>> concerns, and gave green flag to this approach.
>>>>>> https://mails.dpdk.org/archives/dev/2018-June/105730.html
>>>>>> https://mails.dpdk.org/archives/dev/2018-July/106453.html
>>>>>>
>>>>>> I guess Bruce was opening this question to the community. For compute
>>>>>> intense applications like ipsec-secgw, eventmode might be the right
>>>>>> approach in the first place. Such complex applications would need a
>>>>>> scheduler to perform dynamic load balancing. Addition of eventmode in
>>>>>> l2fwd was to float around the idea which can then be scaled for more
>>>>>> complex applications.
>>>>>>
>>>>>> If maintainers doesn't have any objection to this, I'm fine with adding
>>>>>> this in the next release.
>>>>>>
>>>>>> Thanks,
>>>>>> Anoob
>>>>> It is important that DPDK has good examples of how to use existing
>>>>> frameworks and libraries. These applications are what most customers
>>>>> build their applications from and they provide basis for testing.
>>>>>
>>>>> The DPDK needs to continue to support multiple usage models. This
>>>>> is one of its strong points. I would rather leave existing l2fwd
>>>>> and l3fwd alone and instead make new examples that use the frameworks.
>>>>> If nothing else haveing l2fwd and l2fwd-eventdev would allow for
>>>>> performance comparisons.
>>>> Unlike other applications example, there wont be any change in packet
>>>> processing functions in eventdev vs poll mode case. Only worker
>>>> schematics will change and that can be moved to separated files.
>>>> something like worker_poll.c and worker_event.c and both of them
>>>> use common packet processing functions using mbuf.
>>>>
>>>> The only disadvantage of having separate application would be packet
>>>> processing code duplication. Which is non trivial for l3fwd, IPSec
>>>> application IMO.
>>> Personally I am ok with original design suggestion:
>>> keep packet processing code common, that would be used by both poll and event modes.
>>> We could just have a command-line parameter in which mode the app will run.
>>> Another alternative - generate two binaries l2fwd-poll, l2fwd-event (or so).
>>> Konstantin
>>>> # Are we fine with code duplication in example application like l3fwd and
>>>> IPSec?
>>>> # if yes, Are we fine with keeping l2fwd _as is_ to reduce the
>>>> complexity and l2fwd-eventdev supports both modes wherever possible?
>>>>
>>>>> As the number of examples increases, probably also need to have
>>>>> a roadmap or decision chart to explain the advangage/disadvantage
>>>>> of each architecture.
>>>>>
  

On Mon, Aug 13, 2018 at 09:29:01PM +0530, Joseph, Anoob wrote:
> Hi Bruce,
> 
> The reason why l2fwd was chosen was to allow everyone to chip in their ideas
> while preparing the framework.
> This framework would be extended to other applications, hence needed enough
> inputs before expanding to complex applications. If your suggestion is to
> make l3fwd event driven first, I'll start looking in that direction.

Seems good to me, if others don't have an issue with it.

> 
> As for l2fwd, I'm fine with moving event-mode additions to a new app. But
> with the present approach, the app would run in both event mode and poll
> mode.
> 
> Your thoughts on renaming the existing app to l2fwd-poll and the proposed
> app as l2fwd?
> 
> Thanks,
> Anoob

I'm not sure about the name "poll", I think "ethdev" and "eventdev" should be
the suffixes, if we want to move in that direction.
However, my preference would be to leave l2fwd as-is, and to have a comment
at the top of the source file, and note in the documentation along the
lines of:

"This example demonstrates basic l2 forwarding using ethdev primitives. To
see the same use-case implemented using event-based primitives, see the
'l2fwd-eventdev' example".

As I said before, my main concern is to keep the basic examples short and
readable.

/Bruce

> On 13-08-2018 14:57, Bruce Richardson wrote:
> > External Email
> > 
> > On Mon, Aug 13, 2018 at 12:52:19PM +0530, Joseph, Anoob wrote:
> > > Hi Bruce, Pablo,
> > > 
> > > If there are no more issues about the approach, can you review the patches
> > > and give the feedback?
> > > 
> > > Please do note that this series doesn't add any event mode specific code.
> > > That will come as a different patch series after incorporating Jerin's
> > > comments.
> > > 
> > > Thanks,
> > > Anoob
> > My main concern is with l2fwd, rather than l3fwd, which is already fairly
> > complicated. I could see l3fwd being updated to allow an eventmode without
> > too many problems.
> > 
> > With l2fwd, the only issue I have is with the volume of code involved.
> > l2fwd is currently a very simple application which fits in a single file.
> > With these updates it's no longer such a simple, approachable app, rather
> > it becomes one which takes a bit of studying a switching between files to
> > fully understand. The data path is only a very small part of the app, so by
> > adding an event-based path to the same app we have very little code saving.
> > Therefore, I think having a separate l2fwd-eventdev would be better for
> > this case. Two simpler to understand apps is better than one more
> > complicated on IMHO.
> > 
> > My 2c.
> > 
> > /Bruce
> > 
> > > On 02-08-2018 13:49, Ananyev, Konstantin wrote:
> > > > External Email
> > > > 
> > > > Hi everyone,
> > > > 
> > > > > > > > In order to get this series accepted, we need more discussions
> > > > > > > > with more people involved.
> > > > > > > > So it will miss 18.08.
> > > > > > > > 
> > > > > > > > It can be discussed in a more global discussion about examples maintenance.
> > > > > > > > If discussion does not happen, you can request it to the technical board.
> > > > > > > > 
> > > > > > > Event dev framework and various adapters enable multiple packet handling
> > > > > > > schemes, as opposed to the traditional polling on queues. But these
> > > > > > > features are not integrated into any established example application.
> > > > > > > There are specific example applications for event dev etc, which can be
> > > > > > > used to analyze an event device or a particular eventdev adapter, but
> > > > > > > there is no standard application which can be used to compare the real
> > > > > > > world performance for a system when it's using event device for packet
> > > > > > > handling and when it's done via polling on queues.
> > > > > > > 
> > > > > > > The following patch submitted by Sunil was looking to address this issue
> > > > > > > with l3fwd,
> > > > > > > https://mails.dpdk.org/archives/dev/2018-March/093131.html
> > > > > > > 
> > > > > > > Bruce & Jerin reviewed the patch and suggested the addition of helper
> > > > > > > functions to abstract the event mode additions in applications,
> > > > > > > https://mails.dpdk.org/archives/dev/2018-April/096879.html
> > > > > > > 
> > > > > > > This effort of adding helper functions for eventmode was taken up
> > > > > > > following the above suggestion. The idea is to add eventmode without
> > > > > > > touching the existing code path. All the eventmode specific additions
> > > > > > > would go into library so that these need not be repeated for every
> > > > > > > application. And since there is no change in the existing code path,
> > > > > > > performance for any vendor should not have any impact with the additions.
> > > > > > > 
> > > > > > > The scope of this effort has increased since the submission, as now we
> > > > > > > have Tx adapter as well. Sunil & Konstantin had clarified their
> > > > > > > concerns, and gave green flag to this approach.
> > > > > > > https://mails.dpdk.org/archives/dev/2018-June/105730.html
> > > > > > > https://mails.dpdk.org/archives/dev/2018-July/106453.html
> > > > > > > 
> > > > > > > I guess Bruce was opening this question to the community. For compute
> > > > > > > intense applications like ipsec-secgw, eventmode might be the right
> > > > > > > approach in the first place. Such complex applications would need a
> > > > > > > scheduler to perform dynamic load balancing. Addition of eventmode in
> > > > > > > l2fwd was to float around the idea which can then be scaled for more
> > > > > > > complex applications.
> > > > > > > 
> > > > > > > If maintainers doesn't have any objection to this, I'm fine with adding
> > > > > > > this in the next release.
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > Anoob
> > > > > > It is important that DPDK has good examples of how to use existing
> > > > > > frameworks and libraries. These applications are what most customers
> > > > > > build their applications from and they provide basis for testing.
> > > > > > 
> > > > > > The DPDK needs to continue to support multiple usage models. This
> > > > > > is one of its strong points. I would rather leave existing l2fwd
> > > > > > and l3fwd alone and instead make new examples that use the frameworks.
> > > > > > If nothing else haveing l2fwd and l2fwd-eventdev would allow for
> > > > > > performance comparisons.
> > > > > Unlike other applications example, there wont be any change in packet
> > > > > processing functions in eventdev vs poll mode case. Only worker
> > > > > schematics will change and that can be moved to separated files.
> > > > > something like worker_poll.c and worker_event.c and both of them
> > > > > use common packet processing functions using mbuf.
> > > > > 
> > > > > The only disadvantage of having separate application would be packet
> > > > > processing code duplication. Which is non trivial for l3fwd, IPSec
> > > > > application IMO.
> > > > Personally I am ok with original design suggestion:
> > > > keep packet processing code common, that would be used by both poll and event modes.
> > > > We could just have a command-line parameter in which mode the app will run.
> > > > Another alternative - generate two binaries l2fwd-poll, l2fwd-event (or so).
> > > > Konstantin
> > > > > # Are we fine with code duplication in example application like l3fwd and
> > > > > IPSec?
> > > > > # if yes, Are we fine with keeping l2fwd _as is_ to reduce the
> > > > > complexity and l2fwd-eventdev supports both modes wherever possible?
> > > > > 
> > > > > > As the number of examples increases, probably also need to have
> > > > > > a roadmap or decision chart to explain the advangage/disadvantage
> > > > > > of each architecture.
> > > > > > 
>
  

Hi Bruce,

On 14-08-2018 16:03, Bruce Richardson wrote:
> External Email
>
> On Mon, Aug 13, 2018 at 09:29:01PM +0530, Joseph, Anoob wrote:
>> Hi Bruce,
>>
>> The reason why l2fwd was chosen was to allow everyone to chip in their ideas
>> while preparing the framework.
>> This framework would be extended to other applications, hence needed enough
>> inputs before expanding to complex applications. If your suggestion is to
>> make l3fwd event driven first, I'll start looking in that direction.
> Seems good to me, if others don't have an issue with it.
>
>> As for l2fwd, I'm fine with moving event-mode additions to a new app. But
>> with the present approach, the app would run in both event mode and poll
>> mode.
>>
>> Your thoughts on renaming the existing app to l2fwd-poll and the proposed
>> app as l2fwd?
>>
>> Thanks,
>> Anoob
> I'm not sure about the name "poll", I think "ethdev" and "eventdev" should be
> the suffixes, if we want to move in that direction.
With new adapters, like crypto adapter, event device will be able to 
handle multiple devices at the same time. It will be able to abstract 
not just eth device but other devices too (crypto for example). For apps 
like ipsec-secgw, crypto also would be abstracted with event-device. So 
what should be the ideal naming convention taking into account all that? 
Using "eth"& "event" won't fit in for such cases.

Presently, mode was defined by the way packets were submitted to & 
received from the device. With poll mode, device(eth, crypto etc) would 
get packets directly from the core & the core would then poll for 
completion. In case of event-mode, event device handles this scheduling. 
Event device would submit (to all devices) and receive packets (from all 
devices). Core need not poll on the device, in that case. Hence the 
naming...

Your thoughts?

Anoob
> However, my preference would be to leave l2fwd as-is, and to have a comment
> at the top of the source file, and note in the documentation along the
> lines of:
>
> "This example demonstrates basic l2 forwarding using ethdev primitives. To
> see the same use-case implemented using event-based primitives, see the
> 'l2fwd-eventdev' example".
>
> As I said before, my main concern is to keep the basic examples short and
> readable.
>
> /Bruce
>
>> On 13-08-2018 14:57, Bruce Richardson wrote:
>>> External Email
>>>
>>> On Mon, Aug 13, 2018 at 12:52:19PM +0530, Joseph, Anoob wrote:
>>>> Hi Bruce, Pablo,
>>>>
>>>> If there are no more issues about the approach, can you review the patches
>>>> and give the feedback?
>>>>
>>>> Please do note that this series doesn't add any event mode specific code.
>>>> That will come as a different patch series after incorporating Jerin's
>>>> comments.
>>>>
>>>> Thanks,
>>>> Anoob
>>> My main concern is with l2fwd, rather than l3fwd, which is already fairly
>>> complicated. I could see l3fwd being updated to allow an eventmode without
>>> too many problems.
>>>
>>> With l2fwd, the only issue I have is with the volume of code involved.
>>> l2fwd is currently a very simple application which fits in a single file.
>>> With these updates it's no longer such a simple, approachable app, rather
>>> it becomes one which takes a bit of studying a switching between files to
>>> fully understand. The data path is only a very small part of the app, so by
>>> adding an event-based path to the same app we have very little code saving.
>>> Therefore, I think having a separate l2fwd-eventdev would be better for
>>> this case. Two simpler to understand apps is better than one more
>>> complicated on IMHO.
>>>
>>> My 2c.
>>>
>>> /Bruce
>>>
>>>> On 02-08-2018 13:49, Ananyev, Konstantin wrote:
>>>>> External Email
>>>>>
>>>>> Hi everyone,
>>>>>
>>>>>>>>> In order to get this series accepted, we need more discussions
>>>>>>>>> with more people involved.
>>>>>>>>> So it will miss 18.08.
>>>>>>>>>
>>>>>>>>> It can be discussed in a more global discussion about examples maintenance.
>>>>>>>>> If discussion does not happen, you can request it to the technical board.
>>>>>>>>>
>>>>>>>> Event dev framework and various adapters enable multiple packet handling
>>>>>>>> schemes, as opposed to the traditional polling on queues. But these
>>>>>>>> features are not integrated into any established example application.
>>>>>>>> There are specific example applications for event dev etc, which can be
>>>>>>>> used to analyze an event device or a particular eventdev adapter, but
>>>>>>>> there is no standard application which can be used to compare the real
>>>>>>>> world performance for a system when it's using event device for packet
>>>>>>>> handling and when it's done via polling on queues.
>>>>>>>>
>>>>>>>> The following patch submitted by Sunil was looking to address this issue
>>>>>>>> with l3fwd,
>>>>>>>> https://mails.dpdk.org/archives/dev/2018-March/093131.html
>>>>>>>>
>>>>>>>> Bruce & Jerin reviewed the patch and suggested the addition of helper
>>>>>>>> functions to abstract the event mode additions in applications,
>>>>>>>> https://mails.dpdk.org/archives/dev/2018-April/096879.html
>>>>>>>>
>>>>>>>> This effort of adding helper functions for eventmode was taken up
>>>>>>>> following the above suggestion. The idea is to add eventmode without
>>>>>>>> touching the existing code path. All the eventmode specific additions
>>>>>>>> would go into library so that these need not be repeated for every
>>>>>>>> application. And since there is no change in the existing code path,
>>>>>>>> performance for any vendor should not have any impact with the additions.
>>>>>>>>
>>>>>>>> The scope of this effort has increased since the submission, as now we
>>>>>>>> have Tx adapter as well. Sunil & Konstantin had clarified their
>>>>>>>> concerns, and gave green flag to this approach.
>>>>>>>> https://mails.dpdk.org/archives/dev/2018-June/105730.html
>>>>>>>> https://mails.dpdk.org/archives/dev/2018-July/106453.html
>>>>>>>>
>>>>>>>> I guess Bruce was opening this question to the community. For compute
>>>>>>>> intense applications like ipsec-secgw, eventmode might be the right
>>>>>>>> approach in the first place. Such complex applications would need a
>>>>>>>> scheduler to perform dynamic load balancing. Addition of eventmode in
>>>>>>>> l2fwd was to float around the idea which can then be scaled for more
>>>>>>>> complex applications.
>>>>>>>>
>>>>>>>> If maintainers doesn't have any objection to this, I'm fine with adding
>>>>>>>> this in the next release.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Anoob
>>>>>>> It is important that DPDK has good examples of how to use existing
>>>>>>> frameworks and libraries. These applications are what most customers
>>>>>>> build their applications from and they provide basis for testing.
>>>>>>>
>>>>>>> The DPDK needs to continue to support multiple usage models. This
>>>>>>> is one of its strong points. I would rather leave existing l2fwd
>>>>>>> and l3fwd alone and instead make new examples that use the frameworks.
>>>>>>> If nothing else haveing l2fwd and l2fwd-eventdev would allow for
>>>>>>> performance comparisons.
>>>>>> Unlike other applications example, there wont be any change in packet
>>>>>> processing functions in eventdev vs poll mode case. Only worker
>>>>>> schematics will change and that can be moved to separated files.
>>>>>> something like worker_poll.c and worker_event.c and both of them
>>>>>> use common packet processing functions using mbuf.
>>>>>>
>>>>>> The only disadvantage of having separate application would be packet
>>>>>> processing code duplication. Which is non trivial for l3fwd, IPSec
>>>>>> application IMO.
>>>>> Personally I am ok with original design suggestion:
>>>>> keep packet processing code common, that would be used by both poll and event modes.
>>>>> We could just have a command-line parameter in which mode the app will run.
>>>>> Another alternative - generate two binaries l2fwd-poll, l2fwd-event (or so).
>>>>> Konstantin
>>>>>> # Are we fine with code duplication in example application like l3fwd and
>>>>>> IPSec?
>>>>>> # if yes, Are we fine with keeping l2fwd _as is_ to reduce the
>>>>>> complexity and l2fwd-eventdev supports both modes wherever possible?
>>>>>>
>>>>>>> As the number of examples increases, probably also need to have
>>>>>>> a roadmap or decision chart to explain the advangage/disadvantage
>>>>>>> of each architecture.
>>>>>>>
  

18/08/2018 11:58, Joseph, Anoob:
> Hi Bruce,
> 
> On 14-08-2018 16:03, Bruce Richardson wrote:
> > On Mon, Aug 13, 2018 at 09:29:01PM +0530, Joseph, Anoob wrote:
> >> Hi Bruce,
> >>
> >> The reason why l2fwd was chosen was to allow everyone to chip in their ideas
> >> while preparing the framework.
> >> This framework would be extended to other applications, hence needed enough
> >> inputs before expanding to complex applications. If your suggestion is to
> >> make l3fwd event driven first, I'll start looking in that direction.
> > Seems good to me, if others don't have an issue with it.
> >
> >> As for l2fwd, I'm fine with moving event-mode additions to a new app. But
> >> with the present approach, the app would run in both event mode and poll
> >> mode.
> >>
> >> Your thoughts on renaming the existing app to l2fwd-poll and the proposed
> >> app as l2fwd?
> >>
> >> Thanks,
> >> Anoob
> > I'm not sure about the name "poll", I think "ethdev" and "eventdev" should be
> > the suffixes, if we want to move in that direction.
> With new adapters, like crypto adapter, event device will be able to 
> handle multiple devices at the same time. It will be able to abstract 
> not just eth device but other devices too (crypto for example). For apps 
> like ipsec-secgw, crypto also would be abstracted with event-device. So 
> what should be the ideal naming convention taking into account all that? 
> Using "eth"& "event" won't fit in for such cases.
> 
> Presently, mode was defined by the way packets were submitted to & 
> received from the device. With poll mode, device(eth, crypto etc) would 
> get packets directly from the core & the core would then poll for 
> completion. In case of event-mode, event device handles this scheduling. 
> Event device would submit (to all devices) and receive packets (from all 
> devices). Core need not poll on the device, in that case. Hence the 
> naming...
> 
> Your thoughts?

I think it is OK to have event mode in examples, in general.
The only concern of Bruce was to keep l2fwd example simple.
So for l2fwd, and only for this one, you can create a new example
called l2fwd-event.