[v3,0/5] document and simplify use of cmdline

  The DPDK commandline library is widely used by apps and examples within
DPDK, but it is not documented in our programmers guide and it requires
a lot of boilerplate code definitions in order to used. We can improve
this situation by creating a simple python script to automatically
generate the boilerplate from a list of commands.

This patchset contains a new documentation chapter on cmdline library,
going through step-by-step how to add commands and create the necessary
token lists and parse contexts.

Following that initial doc patch, the set then contains a
boilerplate-generating script, as well as a set of three patches showing
its use, by converting three examples to use the script instead of
having the hard-coded boilerplate. Once the script is used, adding a new
command becomes as simple as adding the desired command to the .list
file, and then writing the required function which will be called for
that command. No other boilerplate coding is necessary.

Script obviously does not cover the full range of capabilities of the
commandline lib, but does cover the most used parts. The code-saving to
each of the examples by auto-generating the boilerplate is significant,
and probably more examples with commandlines can be converted over in
future.

The "cmdline" example itself, is not converted over, as it should
probably remain as a simple example of direct library use without the
script.

V3:
* Added lots of documentation
* Added support for help text for each command
* Cleaned up script a little so it passes pycodestyle and most flake8
  checks, when line-length is set to max 100.
* Removed RFC tag, as I consider this patchset stable enough for
  consideration in a release.

V2-RFC:
* Add support for IP addresses in commands
* Move to buildtools directory and make installable
* Convert 3 examples to use script, and eliminate their boilerplate

Bruce Richardson (5):
  doc/prog_guide: new chapter on cmdline library
  buildtools: script to generate cmdline boilerplate
  examples/simple_mp: auto-generate cmdline boilerplate
  examples/hotplug_mp: auto-generate cmdline boilerplate
  examples/bond: auto-generate cmdline boilerplate

 app/test/commands.c                           |   2 +
 buildtools/dpdk-cmdline-gen.py                | 167 +++++++
 buildtools/meson.build                        |   7 +
 doc/guides/prog_guide/cmdline.rst             | 466 ++++++++++++++++++
 doc/guides/prog_guide/index.rst               |   1 +
 examples/bond/Makefile                        |  12 +-
 examples/bond/commands.list                   |   6 +
 examples/bond/main.c                          | 161 +-----
 examples/bond/main.h                          |  10 -
 examples/bond/meson.build                     |   8 +
 examples/multi_process/hotplug_mp/Makefile    |  12 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 +-----
 examples/multi_process/hotplug_mp/commands.h  |  10 -
 .../multi_process/hotplug_mp/commands.list    |   5 +
 examples/multi_process/hotplug_mp/meson.build |   9 +
 examples/multi_process/simple_mp/Makefile     |  12 +-
 examples/multi_process/simple_mp/meson.build  |   9 +
 .../multi_process/simple_mp/mp_commands.c     | 106 +---
 .../multi_process/simple_mp/mp_commands.h     |  14 -
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 20 files changed, 745 insertions(+), 422 deletions(-)
 create mode 100755 buildtools/dpdk-cmdline-gen.py
 create mode 100644 doc/guides/prog_guide/cmdline.rst
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list

--
2.39.2
  

On Wed, Oct 11, 2023 at 3:34 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> The DPDK commandline library is widely used by apps and examples within
> DPDK, but it is not documented in our programmers guide and it requires
> a lot of boilerplate code definitions in order to used. We can improve
> this situation by creating a simple python script to automatically
> generate the boilerplate from a list of commands.
>
> This patchset contains a new documentation chapter on cmdline library,
> going through step-by-step how to add commands and create the necessary
> token lists and parse contexts.
>
> Following that initial doc patch, the set then contains a
> boilerplate-generating script, as well as a set of three patches showing
> its use, by converting three examples to use the script instead of
> having the hard-coded boilerplate. Once the script is used, adding a new
> command becomes as simple as adding the desired command to the .list
> file, and then writing the required function which will be called for
> that command. No other boilerplate coding is necessary.
>
> Script obviously does not cover the full range of capabilities of the
> commandline lib, but does cover the most used parts. The code-saving to
> each of the examples by auto-generating the boilerplate is significant,
> and probably more examples with commandlines can be converted over in
> future.
>
> The "cmdline" example itself, is not converted over, as it should
> probably remain as a simple example of direct library use without the
> script.
>
> V3:
> * Added lots of documentation
> * Added support for help text for each command
> * Cleaned up script a little so it passes pycodestyle and most flake8
>   checks, when line-length is set to max 100.
> * Removed RFC tag, as I consider this patchset stable enough for
>   consideration in a release.
>
> V2-RFC:
> * Add support for IP addresses in commands
> * Move to buildtools directory and make installable
> * Convert 3 examples to use script, and eliminate their boilerplate
>
> Bruce Richardson (5):
>   doc/prog_guide: new chapter on cmdline library
>   buildtools: script to generate cmdline boilerplate
>   examples/simple_mp: auto-generate cmdline boilerplate
>   examples/hotplug_mp: auto-generate cmdline boilerplate
>   examples/bond: auto-generate cmdline boilerplate
>
>  app/test/commands.c                           |   2 +
>  buildtools/dpdk-cmdline-gen.py                | 167 +++++++
>  buildtools/meson.build                        |   7 +
>  doc/guides/prog_guide/cmdline.rst             | 466 ++++++++++++++++++
>  doc/guides/prog_guide/index.rst               |   1 +
>  examples/bond/Makefile                        |  12 +-
>  examples/bond/commands.list                   |   6 +
>  examples/bond/main.c                          | 161 +-----
>  examples/bond/main.h                          |  10 -
>  examples/bond/meson.build                     |   8 +
>  examples/multi_process/hotplug_mp/Makefile    |  12 +-
>  examples/multi_process/hotplug_mp/commands.c  | 147 +-----
>  examples/multi_process/hotplug_mp/commands.h  |  10 -
>  .../multi_process/hotplug_mp/commands.list    |   5 +
>  examples/multi_process/hotplug_mp/meson.build |   9 +
>  examples/multi_process/simple_mp/Makefile     |  12 +-
>  examples/multi_process/simple_mp/meson.build  |   9 +
>  .../multi_process/simple_mp/mp_commands.c     | 106 +---
>  .../multi_process/simple_mp/mp_commands.h     |  14 -
>  .../multi_process/simple_mp/mp_commands.list  |   3 +
>  20 files changed, 745 insertions(+), 422 deletions(-)

Interesting series.
So if we remove the doc addition, this patch is removing loc, so +1 from me :-).

There is a problem though with externally building the examples.
Maybe the dpdk-cmdline-gen.py has not been exported in the install process.


>  create mode 100755 buildtools/dpdk-cmdline-gen.py
>  create mode 100644 doc/guides/prog_guide/cmdline.rst
>  create mode 100644 examples/bond/commands.list
>  delete mode 100644 examples/bond/main.h
>  delete mode 100644 examples/multi_process/hotplug_mp/commands.h
>  create mode 100644 examples/multi_process/hotplug_mp/commands.list
>  delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
>  create mode 100644 examples/multi_process/simple_mp/mp_commands.list
>
> --
> 2.39.2
>
  

On Thu, Oct 12, 2023 at 03:21:08PM +0200, David Marchand wrote:
> On Wed, Oct 11, 2023 at 3:34 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > The DPDK commandline library is widely used by apps and examples within
> > DPDK, but it is not documented in our programmers guide and it requires
> > a lot of boilerplate code definitions in order to used. We can improve
> > this situation by creating a simple python script to automatically
> > generate the boilerplate from a list of commands.
> >
> > This patchset contains a new documentation chapter on cmdline library,
> > going through step-by-step how to add commands and create the necessary
> > token lists and parse contexts.
> >
> > Following that initial doc patch, the set then contains a
> > boilerplate-generating script, as well as a set of three patches showing
> > its use, by converting three examples to use the script instead of
> > having the hard-coded boilerplate. Once the script is used, adding a new
> > command becomes as simple as adding the desired command to the .list
> > file, and then writing the required function which will be called for
> > that command. No other boilerplate coding is necessary.
> >
> > Script obviously does not cover the full range of capabilities of the
> > commandline lib, but does cover the most used parts. The code-saving to
> > each of the examples by auto-generating the boilerplate is significant,
> > and probably more examples with commandlines can be converted over in
> > future.
> >
> > The "cmdline" example itself, is not converted over, as it should
> > probably remain as a simple example of direct library use without the
> > script.
> >
> > V3:
> > * Added lots of documentation
> > * Added support for help text for each command
> > * Cleaned up script a little so it passes pycodestyle and most flake8
> >   checks, when line-length is set to max 100.
> > * Removed RFC tag, as I consider this patchset stable enough for
> >   consideration in a release.
> >
> > V2-RFC:
> > * Add support for IP addresses in commands
> > * Move to buildtools directory and make installable
> > * Convert 3 examples to use script, and eliminate their boilerplate
> >
> > Bruce Richardson (5):
> >   doc/prog_guide: new chapter on cmdline library
> >   buildtools: script to generate cmdline boilerplate
> >   examples/simple_mp: auto-generate cmdline boilerplate
> >   examples/hotplug_mp: auto-generate cmdline boilerplate
> >   examples/bond: auto-generate cmdline boilerplate
> >
> >  app/test/commands.c                           |   2 +
> >  buildtools/dpdk-cmdline-gen.py                | 167 +++++++
> >  buildtools/meson.build                        |   7 +
> >  doc/guides/prog_guide/cmdline.rst             | 466 ++++++++++++++++++
> >  doc/guides/prog_guide/index.rst               |   1 +
> >  examples/bond/Makefile                        |  12 +-
> >  examples/bond/commands.list                   |   6 +
> >  examples/bond/main.c                          | 161 +-----
> >  examples/bond/main.h                          |  10 -
> >  examples/bond/meson.build                     |   8 +
> >  examples/multi_process/hotplug_mp/Makefile    |  12 +-
> >  examples/multi_process/hotplug_mp/commands.c  | 147 +-----
> >  examples/multi_process/hotplug_mp/commands.h  |  10 -
> >  .../multi_process/hotplug_mp/commands.list    |   5 +
> >  examples/multi_process/hotplug_mp/meson.build |   9 +
> >  examples/multi_process/simple_mp/Makefile     |  12 +-
> >  examples/multi_process/simple_mp/meson.build  |   9 +
> >  .../multi_process/simple_mp/mp_commands.c     | 106 +---
> >  .../multi_process/simple_mp/mp_commands.h     |  14 -
> >  .../multi_process/simple_mp/mp_commands.list  |   3 +
> >  20 files changed, 745 insertions(+), 422 deletions(-)
> 
> Interesting series.
> So if we remove the doc addition, this patch is removing loc, so +1 from me :-).
> 
> There is a problem though with externally building the examples.
> Maybe the dpdk-cmdline-gen.py has not been exported in the install process.
> 
It has, but it's not visible to the external builds in github actions. The
following one-line fix should correct that, I believe.
I was testing it in my own github yesterday, but never sent a v4 series.

/Bruce

diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index e0b62bac90..3db9d9de6e 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -174,6 +174,7 @@ fi
 if [ "$BUILD_EXAMPLES" = "true" ]; then
     [ -d install ] || DESTDIR=$(pwd)/install meson install -C build
     export LD_LIBRARY_PATH=$(dirname $(find $(pwd)/install -name librte_eal.so)):$LD_LIBRARY_PATH
+    export PATH=$(dirname $(find $(pwd)/install -name dpdk-devbind.py)):$PATH
     export PKG_CONFIG_PATH=$(dirname $(find $(pwd)/install -name libdpdk.pc)):$PKG_CONFIG_PATH
     export PKGCONF="pkg-config --define-prefix"
     find build/examples -maxdepth 1 -type f -name "dpdk-*" |
  

On Thu, Oct 12, 2023 at 03:21:08PM +0200, David Marchand wrote:
> On Wed, Oct 11, 2023 at 3:34 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > The DPDK commandline library is widely used by apps and examples within
> > DPDK, but it is not documented in our programmers guide and it requires
> > a lot of boilerplate code definitions in order to used. We can improve
> > this situation by creating a simple python script to automatically
> > generate the boilerplate from a list of commands.
> >
> > This patchset contains a new documentation chapter on cmdline library,
> > going through step-by-step how to add commands and create the necessary
> > token lists and parse contexts.
> >
> > Following that initial doc patch, the set then contains a
> > boilerplate-generating script, as well as a set of three patches showing
> > its use, by converting three examples to use the script instead of
> > having the hard-coded boilerplate. Once the script is used, adding a new
> > command becomes as simple as adding the desired command to the .list
> > file, and then writing the required function which will be called for
> > that command. No other boilerplate coding is necessary.
> >
> > Script obviously does not cover the full range of capabilities of the
> > commandline lib, but does cover the most used parts. The code-saving to
> > each of the examples by auto-generating the boilerplate is significant,
> > and probably more examples with commandlines can be converted over in
> > future.
> >
> > The "cmdline" example itself, is not converted over, as it should
> > probably remain as a simple example of direct library use without the
> > script.
> >
> > V3:
> > * Added lots of documentation
> > * Added support for help text for each command
> > * Cleaned up script a little so it passes pycodestyle and most flake8
> >   checks, when line-length is set to max 100.
> > * Removed RFC tag, as I consider this patchset stable enough for
> >   consideration in a release.
> >
> > V2-RFC:
> > * Add support for IP addresses in commands
> > * Move to buildtools directory and make installable
> > * Convert 3 examples to use script, and eliminate their boilerplate
> >
> > Bruce Richardson (5):
> >   doc/prog_guide: new chapter on cmdline library
> >   buildtools: script to generate cmdline boilerplate
> >   examples/simple_mp: auto-generate cmdline boilerplate
> >   examples/hotplug_mp: auto-generate cmdline boilerplate
> >   examples/bond: auto-generate cmdline boilerplate
> >
> >  app/test/commands.c                           |   2 +
> >  buildtools/dpdk-cmdline-gen.py                | 167 +++++++
> >  buildtools/meson.build                        |   7 +
> >  doc/guides/prog_guide/cmdline.rst             | 466 ++++++++++++++++++
> >  doc/guides/prog_guide/index.rst               |   1 +
> >  examples/bond/Makefile                        |  12 +-
> >  examples/bond/commands.list                   |   6 +
> >  examples/bond/main.c                          | 161 +-----
> >  examples/bond/main.h                          |  10 -
> >  examples/bond/meson.build                     |   8 +
> >  examples/multi_process/hotplug_mp/Makefile    |  12 +-
> >  examples/multi_process/hotplug_mp/commands.c  | 147 +-----
> >  examples/multi_process/hotplug_mp/commands.h  |  10 -
> >  .../multi_process/hotplug_mp/commands.list    |   5 +
> >  examples/multi_process/hotplug_mp/meson.build |   9 +
> >  examples/multi_process/simple_mp/Makefile     |  12 +-
> >  examples/multi_process/simple_mp/meson.build  |   9 +
> >  .../multi_process/simple_mp/mp_commands.c     | 106 +---
> >  .../multi_process/simple_mp/mp_commands.h     |  14 -
> >  .../multi_process/simple_mp/mp_commands.list  |   3 +
> >  20 files changed, 745 insertions(+), 422 deletions(-)
> 
> Interesting series.
> So if we remove the doc addition, this patch is removing loc, so +1 from me :-).
> 
There's probably lots more LOC that can be removed beyond this, if we put
in a bit of effort looking. I just took three simple examples, and managed
to save ~100LOC each.

However, the big benefit I find is for anyone working on new apps - using a
script makes it SOOOO  much easier to add a commandline and then
subsequently to extend with new commands as you go! Just add the new command to
the list file, and write the new callback function and you're done.

Since cmdline library is not going to be going anywhere, anytime soon,
since it's so integrated into testpmd and dpdk-test, we might as well make
it useful for others.

/Bruce