[00/12] add packet generator library and example app

  This patchset introduces a Gen library for DPDK. This library provides an easy
way to generate traffic in order to test software based network components.

This library enables the basic functionality required in the traffic generator.
This includes: raw data setting, packet Tx and Rx, creation and destruction of a
 Gen instance and various types of data parsing.
This functionality is implemented in "lib/gen/rte_gen.c". IPv4 parsing
functionality is also added in "lib/net/rte_ip.c", this is then used in the gen
library.

A sample app is included in "examples/generator" which shows the use of the gen
library in making a traffic generator. This can be used to generate traffic by
running the dpdk-generator generator executable. This sample app supports
runtime stats reporting (/gen/stats) and line rate limiting
(/gen/mpps,<target traffic rate in mpps>) through telemetry.py.

As more features are added to the gen library, the sample application will
become more powerful through the "/gen/packet" string parameter
(currently supports IP and Ether address setting). This will allow every
application to generate more complex traffic types in the future without
changing API.

Harry van Haaren (6):
  gen: add files for initial traffic generation library
  gen: add basic Rx and Tx routines and tests
  gen: add raw packet data API and tests
  gen: add parsing infrastructure and Ether protocol
  gen: add gen IP parsing
  examples/generator: import code from basicfwd.c

Ronan Randles (6):
  net: add string to IPv4 parse function
  net: add function to pretty print IPv4
  examples/generator: enable gen library for traffic gen
  examples/generator: telemetry support
  examples/generator: link status check added
  examples/generator: line rate limiting

 app/test/meson.build           |   4 +
 app/test/test_gen.c            | 184 +++++++++++
 app/test/test_net.c            |  87 ++++++
 doc/api/doxy-api-index.md      |   3 +-
 doc/api/doxy-api.conf.in       |   1 +
 examples/generator/main.c      | 483 ++++++++++++++++++++++++++++
 examples/generator/meson.build |  13 +
 examples/meson.build           |   1 +
 lib/gen/meson.build            |   6 +
 lib/gen/rte_gen.c              | 553 +++++++++++++++++++++++++++++++++
 lib/gen/rte_gen.h              | 114 +++++++
 lib/gen/version.map            |  10 +
 lib/meson.build                |   1 +
 lib/net/meson.build            |   1 +
 lib/net/rte_ip.c               |  58 ++++
 lib/net/rte_ip.h               |  38 +++
 lib/net/version.map            |   9 +
 17 files changed, 1565 insertions(+), 1 deletion(-)
 create mode 100644 app/test/test_gen.c
 create mode 100644 app/test/test_net.c
 create mode 100644 examples/generator/main.c
 create mode 100644 examples/generator/meson.build
 create mode 100644 lib/gen/meson.build
 create mode 100644 lib/gen/rte_gen.c
 create mode 100644 lib/gen/rte_gen.h
 create mode 100644 lib/gen/version.map
 create mode 100644 lib/net/rte_ip.c
  

On Tue, Dec 14, 2021 at 02:12:30PM +0000, Ronan Randles wrote:
> This patchset introduces a Gen library for DPDK. This library provides an easy
> way to generate traffic in order to test software based network components.
> 
> This library enables the basic functionality required in the traffic generator.
> This includes: raw data setting, packet Tx and Rx, creation and destruction of a
>  Gen instance and various types of data parsing.
> This functionality is implemented in "lib/gen/rte_gen.c". IPv4 parsing
> functionality is also added in "lib/net/rte_ip.c", this is then used in the gen
> library.
> 
> A sample app is included in "examples/generator" which shows the use of the gen
> library in making a traffic generator. This can be used to generate traffic by
> running the dpdk-generator generator executable. This sample app supports
> runtime stats reporting (/gen/stats) and line rate limiting
> (/gen/mpps,<target traffic rate in mpps>) through telemetry.py.
> 
> As more features are added to the gen library, the sample application will
> become more powerful through the "/gen/packet" string parameter
> (currently supports IP and Ether address setting). This will allow every
> application to generate more complex traffic types in the future without
> changing API.
> 
> Harry van Haaren (6):
>   gen: add files for initial traffic generation library
>   gen: add basic Rx and Tx routines and tests
>   gen: add raw packet data API and tests
>   gen: add parsing infrastructure and Ether protocol
>   gen: add gen IP parsing
>   examples/generator: import code from basicfwd.c
> 
> Ronan Randles (6):
>   net: add string to IPv4 parse function
>   net: add function to pretty print IPv4
>   examples/generator: enable gen library for traffic gen
>   examples/generator: telemetry support
>   examples/generator: link status check added
>   examples/generator: line rate limiting
> 
>  app/test/meson.build           |   4 +
>  app/test/test_gen.c            | 184 +++++++++++
>  app/test/test_net.c            |  87 ++++++
>  doc/api/doxy-api-index.md      |   3 +-
>  doc/api/doxy-api.conf.in       |   1 +
>  examples/generator/main.c      | 483 ++++++++++++++++++++++++++++
>  examples/generator/meson.build |  13 +
>  examples/meson.build           |   1 +
>  lib/gen/meson.build            |   6 +
>  lib/gen/rte_gen.c              | 553 +++++++++++++++++++++++++++++++++
>  lib/gen/rte_gen.h              | 114 +++++++
>  lib/gen/version.map            |  10 +
>  lib/meson.build                |   1 +
>  lib/net/meson.build            |   1 +
>  lib/net/rte_ip.c               |  58 ++++
>  lib/net/rte_ip.h               |  38 +++
>  lib/net/version.map            |   9 +
>  17 files changed, 1565 insertions(+), 1 deletion(-)

I think this is great to see, and sounds a good addition to DPDK. One thing
to address in any v2 is to add more documentation for both the library and
the example app. You need a chapter on the lib added to the programmers
guide to help others use the library from their code, and a chapter on the
generator example in the example apps guide.

More general question - if we do have a traffic generator in DPDK, would it
be better in the "app" rather than the examples one? If it's only going to
ever stay a simple example of using the lib, examples might be fine, but I
suspect that it will get quite complicated if people start using it and
adding more features, in which case a move to the "app" folder might be
more appropriate. Thoughts?

/Bruce
  

I'd be inclined to agree with you on the move to the app dir. The example app could be turned into more of a fully-fledged app than just an example app. I think the final decision on that would fall to @Van Haaren, Harry on that

-----Original Message-----
From: Richardson, Bruce <bruce.richardson@intel.com> 
Sent: Tuesday, December 14, 2021 2:58 PM
To: Randles, Ronan <ronan.randles@intel.com>
Cc: dev@dpdk.org; Van Haaren, Harry <harry.van.haaren@intel.com>
Subject: Re: [PATCH 00/12] add packet generator library and example app

On Tue, Dec 14, 2021 at 02:12:30PM +0000, Ronan Randles wrote:
> This patchset introduces a Gen library for DPDK. This library provides 
> an easy way to generate traffic in order to test software based network components.
> 
> This library enables the basic functionality required in the traffic generator.
> This includes: raw data setting, packet Tx and Rx, creation and 
> destruction of a  Gen instance and various types of data parsing.
> This functionality is implemented in "lib/gen/rte_gen.c". IPv4 parsing 
> functionality is also added in "lib/net/rte_ip.c", this is then used 
> in the gen library.
> 
> A sample app is included in "examples/generator" which shows the use 
> of the gen library in making a traffic generator. This can be used to 
> generate traffic by running the dpdk-generator generator executable. 
> This sample app supports runtime stats reporting (/gen/stats) and line 
> rate limiting (/gen/mpps,<target traffic rate in mpps>) through telemetry.py.
> 
> As more features are added to the gen library, the sample application 
> will become more powerful through the "/gen/packet" string parameter 
> (currently supports IP and Ether address setting). This will allow 
> every application to generate more complex traffic types in the future 
> without changing API.
> 
> Harry van Haaren (6):
>   gen: add files for initial traffic generation library
>   gen: add basic Rx and Tx routines and tests
>   gen: add raw packet data API and tests
>   gen: add parsing infrastructure and Ether protocol
>   gen: add gen IP parsing
>   examples/generator: import code from basicfwd.c
> 
> Ronan Randles (6):
>   net: add string to IPv4 parse function
>   net: add function to pretty print IPv4
>   examples/generator: enable gen library for traffic gen
>   examples/generator: telemetry support
>   examples/generator: link status check added
>   examples/generator: line rate limiting
> 
>  app/test/meson.build           |   4 +
>  app/test/test_gen.c            | 184 +++++++++++
>  app/test/test_net.c            |  87 ++++++
>  doc/api/doxy-api-index.md      |   3 +-
>  doc/api/doxy-api.conf.in       |   1 +
>  examples/generator/main.c      | 483 ++++++++++++++++++++++++++++
>  examples/generator/meson.build |  13 +
>  examples/meson.build           |   1 +
>  lib/gen/meson.build            |   6 +
>  lib/gen/rte_gen.c              | 553 +++++++++++++++++++++++++++++++++
>  lib/gen/rte_gen.h              | 114 +++++++
>  lib/gen/version.map            |  10 +
>  lib/meson.build                |   1 +
>  lib/net/meson.build            |   1 +
>  lib/net/rte_ip.c               |  58 ++++
>  lib/net/rte_ip.h               |  38 +++
>  lib/net/version.map            |   9 +
>  17 files changed, 1565 insertions(+), 1 deletion(-)

I think this is great to see, and sounds a good addition to DPDK. One thing to address in any v2 is to add more documentation for both the library and the example app. You need a chapter on the lib added to the programmers guide to help others use the library from their code, and a chapter on the generator example in the example apps guide.

More general question - if we do have a traffic generator in DPDK, would it be better in the "app" rather than the examples one? If it's only going to ever stay a simple example of using the lib, examples might be fine, but I suspect that it will get quite complicated if people start using it and adding more features, in which case a move to the "app" folder might be more appropriate. Thoughts?

/Bruce
  

On Tue, Dec 14, 2021 at 7:42 PM Ronan Randles <ronan.randles@intel.com> wrote:
>
> This patchset introduces a Gen library for DPDK. This library provides an easy
> way to generate traffic in order to test software based network components.
>
> This library enables the basic functionality required in the traffic generator.
> This includes: raw data setting, packet Tx and Rx, creation and destruction of a
>  Gen instance and various types of data parsing.
> This functionality is implemented in "lib/gen/rte_gen.c". IPv4 parsing
> functionality is also added in "lib/net/rte_ip.c", this is then used in the gen
> library.
>
> A sample app is included in "examples/generator" which shows the use of the gen
> library in making a traffic generator. This can be used to generate traffic by
> running the dpdk-generator generator executable. This sample app supports
> runtime stats reporting (/gen/stats) and line rate limiting
> (/gen/mpps,<target traffic rate in mpps>) through telemetry.py.
>
>
>  lib/gen/rte_gen.h              | 114 +++++++

Please check Doxygen syntax across the file. rte_gen_create(),
rte_gen_destroy(),
rte_gen_packet_parse_string() etc missing proper doxygen synax.
  

On Wed, Dec 15, 2021 at 06:01:23PM +0530, Jerin Jacob wrote:
> On Tue, Dec 14, 2021 at 7:42 PM Ronan Randles <ronan.randles@intel.com> wrote:
> >
> > This patchset introduces a Gen library for DPDK. This library provides an easy
> > way to generate traffic in order to test software based network components.
> >
> > This library enables the basic functionality required in the traffic generator.
> > This includes: raw data setting, packet Tx and Rx, creation and destruction of a
> >  Gen instance and various types of data parsing.
> > This functionality is implemented in "lib/gen/rte_gen.c". IPv4 parsing
> > functionality is also added in "lib/net/rte_ip.c", this is then used in the gen
> > library.
> >
> > A sample app is included in "examples/generator" which shows the use of the gen
> > library in making a traffic generator. This can be used to generate traffic by
> > running the dpdk-generator generator executable. This sample app supports
> > runtime stats reporting (/gen/stats) and line rate limiting
> > (/gen/mpps,<target traffic rate in mpps>) through telemetry.py.
> >
> >
> >  lib/gen/rte_gen.h              | 114 +++++++
> 
> Please check Doxygen syntax across the file. rte_gen_create(),
> rte_gen_destroy(),
> rte_gen_packet_parse_string() etc missing proper doxygen synax.

If you do a build with "-Denable_docs=true -Dwerror" meson options set,
these should all be flagged on build. [If they aren't, we should fix!]

/Bruce
  

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Tuesday, 14 December 2021 15.58
> 
> On Tue, Dec 14, 2021 at 02:12:30PM +0000, Ronan Randles wrote:
> > This patchset introduces a Gen library for DPDK. This library
> provides an easy
> > way to generate traffic in order to test software based network
> components.
> >
> > This library enables the basic functionality required in the traffic
> generator.
> > This includes: raw data setting, packet Tx and Rx, creation and
> destruction of a
> >  Gen instance and various types of data parsing.
> > This functionality is implemented in "lib/gen/rte_gen.c". IPv4
> parsing
> > functionality is also added in "lib/net/rte_ip.c", this is then used
> in the gen
> > library.
> >
> > A sample app is included in "examples/generator" which shows the use
> of the gen
> > library in making a traffic generator. This can be used to generate
> traffic by
> > running the dpdk-generator generator executable. This sample app
> supports
> > runtime stats reporting (/gen/stats) and line rate limiting
> > (/gen/mpps,<target traffic rate in mpps>) through telemetry.py.
> >
> > As more features are added to the gen library, the sample application
> will
> > become more powerful through the "/gen/packet" string parameter
> > (currently supports IP and Ether address setting). This will allow
> every
> > application to generate more complex traffic types in the future
> without
> > changing API.
> >
> 
> I think this is great to see, and sounds a good addition to DPDK. One
> thing
> to address in any v2 is to add more documentation for both the library
> and
> the example app. You need a chapter on the lib added to the programmers
> guide to help others use the library from their code, and a chapter on
> the
> generator example in the example apps guide.
> 
> More general question - if we do have a traffic generator in DPDK,
> would it
> be better in the "app" rather than the examples one? If it's only going
> to
> ever stay a simple example of using the lib, examples might be fine,
> but I
> suspect that it will get quite complicated if people start using it and
> adding more features, in which case a move to the "app" folder might be
> more appropriate. Thoughts?
> 
> /Bruce

If adding a traffic generator lib/app to DPDK itself, it should be able to evolve freely, unencumbered by the DPDK ABI/API stability requirements.

Also, it MUST be optional when building DPDK for production purposes. Consider the security perspective: If a network appliance based on DPDK is compromised by a hacker, you don't want it to include a traffic generator.

-Morten
  

On Tue, 2021-12-14 at 14:12 +0000, Ronan Randles wrote:
> This patchset introduces a Gen library for DPDK. This library provides an easy
> way to generate traffic in order to test software based network components.
> 
> This library enables the basic functionality required in the traffic generator.
> This includes: raw data setting, packet Tx and Rx, creation and destruction of a
>  Gen instance and various types of data parsing.
> This functionality is implemented in "lib/gen/rte_gen.c". IPv4 parsing
> functionality is also added in "lib/net/rte_ip.c", this is then used in the gen
> library.
> 
> A sample app is included in "examples/generator" which shows the use of the gen
> library in making a traffic generator. This can be used to generate traffic by
> running the dpdk-generator generator executable. This sample app supports
> runtime stats reporting (/gen/stats) and line rate limiting
> (/gen/mpps,<target traffic rate in mpps>) through telemetry.py.
> 
> As more features are added to the gen library, the sample application will
> become more powerful through the "/gen/packet" string parameter
> (currently supports IP and Ether address setting). This will allow every
> application to generate more complex traffic types in the future without
> changing API.

A good start, thanks.

Testpmd can do txonly, rxonly, forwarding, but no rx+tx only :) I guess
that's the key of a GEN. testpmd also has "--stats-period" to show
statistics in non-interactive mode, very close to a GEN, but not
enough. Testpmd can toggle a lot of settings like offloading, that's an
advantage. Do you see an chance to make the lib part of testpmd?

I tried to leverage Scapy syntax into testpmd [1][2]: to set fancy
template - useful to test flows, or dump packet using different scapy
detail level. unfotunately I don't have time to finish it. Parsing
packet from string is boring, any consideration on this?

[1]: https://github.com/steevenlee/dpdk/commits/upstream_scapy
[2]: doc
https://github.com/steevenlee/dpdk/blob/875b8407f769465837513d29a495af3cc1a29765/doc/guides/howto/scapy.rst

> 
> Harry van Haaren (6):
>   gen: add files for initial traffic generation library
>   gen: add basic Rx and Tx routines and tests
>   gen: add raw packet data API and tests
>   gen: add parsing infrastructure and Ether protocol
>   gen: add gen IP parsing
>   examples/generator: import code from basicfwd.c
> 
> Ronan Randles (6):
>   net: add string to IPv4 parse function
>   net: add function to pretty print IPv4
>   examples/generator: enable gen library for traffic gen
>   examples/generator: telemetry support
>   examples/generator: link status check added
>   examples/generator: line rate limiting
> 
>  app/test/meson.build           |   4 +
>  app/test/test_gen.c            | 184 +++++++++++
>  app/test/test_net.c            |  87 ++++++
>  doc/api/doxy-api-index.md      |   3 +-
>  doc/api/doxy-api.conf.in       |   1 +
>  examples/generator/main.c      | 483 ++++++++++++++++++++++++++++
>  examples/generator/meson.build |  13 +
>  examples/meson.build           |   1 +
>  lib/gen/meson.build            |   6 +
>  lib/gen/rte_gen.c              | 553 +++++++++++++++++++++++++++++++++
>  lib/gen/rte_gen.h              | 114 +++++++
>  lib/gen/version.map            |  10 +
>  lib/meson.build                |   1 +
>  lib/net/meson.build            |   1 +
>  lib/net/rte_ip.c               |  58 ++++
>  lib/net/rte_ip.h               |  38 +++
>  lib/net/version.map            |   9 +
>  17 files changed, 1565 insertions(+), 1 deletion(-)
>  create mode 100644 app/test/test_gen.c
>  create mode 100644 app/test/test_net.c
>  create mode 100644 examples/generator/main.c
>  create mode 100644 examples/generator/meson.build
>  create mode 100644 lib/gen/meson.build
>  create mode 100644 lib/gen/rte_gen.c
>  create mode 100644 lib/gen/rte_gen.h
>  create mode 100644 lib/gen/version.map
>  create mode 100644 lib/net/rte_ip.c
>
  

> -----Original Message-----
> From: Xueming(Steven) Li <xuemingl@nvidia.com>
> Sent: Friday, January 21, 2022 2:45 PM
> To: dev@dpdk.org; Randles, Ronan <ronan.randles@intel.com>
> Cc: Van Haaren, Harry <harry.van.haaren@intel.com>
> Subject: Re: [PATCH 00/12] add packet generator library and example app
> 
> On Tue, 2021-12-14 at 14:12 +0000, Ronan Randles wrote:
> > This patchset introduces a Gen library for DPDK. This library provides an easy
> > way to generate traffic in order to test software based network components.
> >
> > This library enables the basic functionality required in the traffic generator.
> > This includes: raw data setting, packet Tx and Rx, creation and destruction of a
> >  Gen instance and various types of data parsing.
> > This functionality is implemented in "lib/gen/rte_gen.c". IPv4 parsing
> > functionality is also added in "lib/net/rte_ip.c", this is then used in the gen
> > library.
> >
> > A sample app is included in "examples/generator" which shows the use of the
> gen
> > library in making a traffic generator. This can be used to generate traffic by
> > running the dpdk-generator generator executable. This sample app supports
> > runtime stats reporting (/gen/stats) and line rate limiting
> > (/gen/mpps,<target traffic rate in mpps>) through telemetry.py.
> >
> > As more features are added to the gen library, the sample application will
> > become more powerful through the "/gen/packet" string parameter
> > (currently supports IP and Ether address setting). This will allow every
> > application to generate more complex traffic types in the future without
> > changing API.
> 
> A good start, thanks.
> 
> Testpmd can do txonly, rxonly, forwarding, but no rx+tx only :) I guess
> that's the key of a GEN. testpmd also has "--stats-period" to show
> statistics in non-interactive mode, very close to a GEN, but not
> enough. Testpmd can toggle a lot of settings like offloading, that's an
> advantage. Do you see an chance to make the lib part of testpmd?
> 
> I tried to leverage Scapy syntax into testpmd [1][2]: to set fancy
> template - useful to test flows, or dump packet using different scapy
> detail level. unfotunately I don't have time to finish it. Parsing
> packet from string is boring, any consideration on this?

Hi Steven,

Yes I recall the patchset to extend DPDK with python/scapy itself, and indeed this was my first approach too!
I build a proof-of-concept for enabling a "run-python-script-inside-eth-null-PMD-rx-function" approach.
Although this worked (in a clunky way, using packet.__bytes__() in python script..), the performance was
extremely low. An optimization was to pre-calculate e.g. 1 million packets, and leave them all in memory,
which causes memory waste, and lack of performance due to memory-boundedness in the CPU.

There is a fundamental blocking issue here - the generation/creation of packets is slow. Even updating a
single flow, or single parameter of the packet, could/would require recalculation of checksums... so leads
to the C code needing to know the L2/L3/L4 offsets. Even then, the re-calculation over 1-million packets
puts a lot of memory pressure - the packets cannot be re-used (from mempool lcore cache) if they need
to be "unique" through pre-computation, resulting in cache-trashing.

As a solution, moving to a native/C language based creation of a "base packet" (template) with "modifiers"
seemed a good fit. It allows best-case performance (mempool cache) and lowest compute per-packet.
The "modifiers" are "pay for what you need" approach to flows/updates/etc, allowing user to choose
the right balance of traffic-complexity vs CPU-cost to generate it.

All in all; handling string-parsing in C is not fun - but by paying that price we can gain many other things...

I think that a "gen" engine for TestPMD could be interesting for certain use-cases yes. As your links below,
the "expect" behaviour is a particularly nice way of working for testing! There is an implementation for
net/null PMD to create packets with Gen library, but is not ready for upstream so was not included in V2.

For multi-flow latency testing, I'm not sure if TestPMD is the ideal tool, as it does not prioritize RX/latency
at all costs: e.g lcores perform both RX and TX functionality. The dedicated "example/generator" is a sample
application that provides the basis for a more latency/jitter focused application, however there is much work
to do there to make it a production-ready tool.

Note that at the moment, the status of the Gen work is as follows;
- Upstream discussion: http://mails.dpdk.org/archives/dev/2022-January/232965.html
- V2 patch: http://patches.dpdk.org/project/dpdk/cover/20220121103122.2926856-1-ronan.randles@intel.com/

Regards, -Harry

PS: Side note around Python/Scapy: its an *awesome* tool. I like it a lot, but for DPDK levels of performance,
and in a latency/performance critical use-case, I'm not convinced it is the right tool to use.


> [1]: https://github.com/steevenlee/dpdk/commits/upstream_scapy
> [2]: doc
> https://github.com/steevenlee/dpdk/blob/875b8407f769465837513d29a495af3cc
> 1a29765/doc/guides/howto/scapy.rst
> 
> >
> > Harry van Haaren (6):
> >   gen: add files for initial traffic generation library
> >   gen: add basic Rx and Tx routines and tests
> >   gen: add raw packet data API and tests
> >   gen: add parsing infrastructure and Ether protocol
> >   gen: add gen IP parsing
> >   examples/generator: import code from basicfwd.c
> >
> > Ronan Randles (6):
> >   net: add string to IPv4 parse function
> >   net: add function to pretty print IPv4
> >   examples/generator: enable gen library for traffic gen
> >   examples/generator: telemetry support
> >   examples/generator: link status check added
> >   examples/generator: line rate limiting
> >
> >  app/test/meson.build           |   4 +
> >  app/test/test_gen.c            | 184 +++++++++++
> >  app/test/test_net.c            |  87 ++++++
> >  doc/api/doxy-api-index.md      |   3 +-
> >  doc/api/doxy-api.conf.in       |   1 +
> >  examples/generator/main.c      | 483 ++++++++++++++++++++++++++++
> >  examples/generator/meson.build |  13 +
> >  examples/meson.build           |   1 +
> >  lib/gen/meson.build            |   6 +
> >  lib/gen/rte_gen.c              | 553 +++++++++++++++++++++++++++++++++
> >  lib/gen/rte_gen.h              | 114 +++++++
> >  lib/gen/version.map            |  10 +
> >  lib/meson.build                |   1 +
> >  lib/net/meson.build            |   1 +
> >  lib/net/rte_ip.c               |  58 ++++
> >  lib/net/rte_ip.h               |  38 +++
> >  lib/net/version.map            |   9 +
> >  17 files changed, 1565 insertions(+), 1 deletion(-)
> >  create mode 100644 app/test/test_gen.c
> >  create mode 100644 app/test/test_net.c
> >  create mode 100644 examples/generator/main.c
> >  create mode 100644 examples/generator/meson.build
> >  create mode 100644 lib/gen/meson.build
> >  create mode 100644 lib/gen/rte_gen.c
> >  create mode 100644 lib/gen/rte_gen.h
> >  create mode 100644 lib/gen/version.map
> >  create mode 100644 lib/net/rte_ip.c
> >
  

> > On Tue, 2021-12-14 at 14:12 +0000, Ronan Randles wrote:
> > > This patchset introduces a Gen library for DPDK. This library provides an easy
> > > way to generate traffic in order to test software based network components.
> > >
> > > This library enables the basic functionality required in the traffic generator.
> > > This includes: raw data setting, packet Tx and Rx, creation and destruction of a
> > >  Gen instance and various types of data parsing.
> > > This functionality is implemented in "lib/gen/rte_gen.c". IPv4 parsing
> > > functionality is also added in "lib/net/rte_ip.c", this is then used in the gen
> > > library.
> > >
> > > A sample app is included in "examples/generator" which shows the use of the
> > gen
> > > library in making a traffic generator. This can be used to generate traffic by
> > > running the dpdk-generator generator executable. This sample app supports
> > > runtime stats reporting (/gen/stats) and line rate limiting
> > > (/gen/mpps,<target traffic rate in mpps>) through telemetry.py.
> > >
> > > As more features are added to the gen library, the sample application will
> > > become more powerful through the "/gen/packet" string parameter
> > > (currently supports IP and Ether address setting). This will allow every
> > > application to generate more complex traffic types in the future without
> > > changing API.
> >
> > A good start, thanks.
> >
> > Testpmd can do txonly, rxonly, forwarding, but no rx+tx only :) I guess
> > that's the key of a GEN. testpmd also has "--stats-period" to show
> > statistics in non-interactive mode, very close to a GEN, but not
> > enough. Testpmd can toggle a lot of settings like offloading, that's an
> > advantage. Do you see an chance to make the lib part of testpmd?
> >
> > I tried to leverage Scapy syntax into testpmd [1][2]: to set fancy
> > template - useful to test flows, or dump packet using different scapy
> > detail level. unfotunately I don't have time to finish it. Parsing
> > packet from string is boring, any consideration on this?
> 
> Hi Steven,
> 
> Yes I recall the patchset to extend DPDK with python/scapy itself, and indeed this was my first approach too!
> I build a proof-of-concept for enabling a "run-python-script-inside-eth-null-PMD-rx-function" approach.
> Although this worked (in a clunky way, using packet.__bytes__() in python script..), the performance was
> extremely low. An optimization was to pre-calculate e.g. 1 million packets, and leave them all in memory,
> which causes memory waste, and lack of performance due to memory-boundedness in the CPU.
> 
> There is a fundamental blocking issue here - the generation/creation of packets is slow. Even updating a
> single flow, or single parameter of the packet, could/would require recalculation of checksums... so leads
> to the C code needing to know the L2/L3/L4 offsets. Even then, the re-calculation over 1-million packets
> puts a lot of memory pressure - the packets cannot be re-used (from mempool lcore cache) if they need
> to be "unique" through pre-computation, resulting in cache-trashing.
> 
> As a solution, moving to a native/C language based creation of a "base packet" (template) with "modifiers"
> seemed a good fit. It allows best-case performance (mempool cache) and lowest compute per-packet.
> The "modifiers" are "pay for what you need" approach to flows/updates/etc, allowing user to choose
> the right balance of traffic-complexity vs CPU-cost to generate it.
> 
> All in all; handling string-parsing in C is not fun - but by paying that price we can gain many other things...

Just to throw in an alternate thought:
Instead of trying to re-implement scapy's abilities and face all related complexities (string parsings, etc.) -
might be choose another way: allow user to write his own packet fillers via eBPF?   
I.E: user write and compile eBPF program that will fill l2/l3/l4 headers, set mbuf flags, etc.
Then it will be loaded by packet-generator and executed for each outgoing packet.
That way user will have full flexibility and extensibility in defining his outgoing traffic.  
Again from developer/maintainer point of view such packet-gen app will probably require
much less effort.    
Though yes - user will need to do more work on his own - write actual packet filler code.

> 
> I think that a "gen" engine for TestPMD could be interesting for certain use-cases yes. As your links below,
> the "expect" behaviour is a particularly nice way of working for testing! There is an implementation for
> net/null PMD to create packets with Gen library, but is not ready for upstream so was not included in V2.
> 
> For multi-flow latency testing, I'm not sure if TestPMD is the ideal tool, as it does not prioritize RX/latency
> at all costs: e.g lcores perform both RX and TX functionality. The dedicated "example/generator" is a sample
> application that provides the basis for a more latency/jitter focused application, however there is much work
> to do there to make it a production-ready tool.
> 
> Note that at the moment, the status of the Gen work is as follows;
> - Upstream discussion: http://mails.dpdk.org/archives/dev/2022-January/232965.html
> - V2 patch: http://patches.dpdk.org/project/dpdk/cover/20220121103122.2926856-1-ronan.randles@intel.com/
> 
> Regards, -Harry
> 
> PS: Side note around Python/Scapy: its an *awesome* tool. I like it a lot, but for DPDK levels of performance,
> and in a latency/performance critical use-case, I'm not convinced it is the right tool to use.
> 
> 
> > [1]: https://github.com/steevenlee/dpdk/commits/upstream_scapy
> > [2]: doc
> > https://github.com/steevenlee/dpdk/blob/875b8407f769465837513d29a495af3cc
> > 1a29765/doc/guides/howto/scapy.rst
> >
> > >
> > > Harry van Haaren (6):
> > >   gen: add files for initial traffic generation library
> > >   gen: add basic Rx and Tx routines and tests
> > >   gen: add raw packet data API and tests
> > >   gen: add parsing infrastructure and Ether protocol
> > >   gen: add gen IP parsing
> > >   examples/generator: import code from basicfwd.c
> > >
> > > Ronan Randles (6):
> > >   net: add string to IPv4 parse function
> > >   net: add function to pretty print IPv4
> > >   examples/generator: enable gen library for traffic gen
> > >   examples/generator: telemetry support
> > >   examples/generator: link status check added
> > >   examples/generator: line rate limiting
> > >
> > >  app/test/meson.build           |   4 +
> > >  app/test/test_gen.c            | 184 +++++++++++
> > >  app/test/test_net.c            |  87 ++++++
> > >  doc/api/doxy-api-index.md      |   3 +-
> > >  doc/api/doxy-api.conf.in       |   1 +
> > >  examples/generator/main.c      | 483 ++++++++++++++++++++++++++++
> > >  examples/generator/meson.build |  13 +
> > >  examples/meson.build           |   1 +
> > >  lib/gen/meson.build            |   6 +
> > >  lib/gen/rte_gen.c              | 553 +++++++++++++++++++++++++++++++++
> > >  lib/gen/rte_gen.h              | 114 +++++++
> > >  lib/gen/version.map            |  10 +
> > >  lib/meson.build                |   1 +
> > >  lib/net/meson.build            |   1 +
> > >  lib/net/rte_ip.c               |  58 ++++
> > >  lib/net/rte_ip.h               |  38 +++
> > >  lib/net/version.map            |   9 +
> > >  17 files changed, 1565 insertions(+), 1 deletion(-)
> > >  create mode 100644 app/test/test_gen.c
> > >  create mode 100644 app/test/test_net.c
> > >  create mode 100644 examples/generator/main.c
> > >  create mode 100644 examples/generator/meson.build
> > >  create mode 100644 lib/gen/meson.build
> > >  create mode 100644 lib/gen/rte_gen.c
> > >  create mode 100644 lib/gen/rte_gen.h
> > >  create mode 100644 lib/gen/version.map
> > >  create mode 100644 lib/net/rte_ip.c
> > >