[v4,0/7] dts: add per-test-suite configuration

From: Luca Vizzarro <luca.vizzarro@arm.com>

Hi there,

sending in v4 for the per-test-suite configuration patchset.

v4:
- rebased
- Changed implementation:
  - test suites configurations are now provided
    through a dedicated optional configuration file.
  - running multiple test runs hinders flexibility in loading
    configuration, therefore DTS now accepts only one test run.
  - the tests folder is no longer a package as this is no longer
    imported directly. The implementation now relies on the test
    suite globbing imports, then the configuration class is referenced
    through the annotation for TestSuite.config.
  - as a consequence of the above, a code generator is no longer needed.
- improved the way configuration errors are displayed.
- adding the test suites to the docs resulted in several warnings,
  minimally required fixes are added.

Best,
Luca

Luca Vizzarro (7):
  dts: add tests package to API docs
  dts: amend test suites docstring
  dts: fix smoke tests docstring
  dts: update autodoc sorting order
  dts: run only one test run per execution
  dts: add per-test-suite configuration
  dts: improve configuration errors

 doc/api/dts/index.rst                         |   1 +
 doc/api/dts/tests.TestSuite_blocklist.rst     |   8 ++
 .../dts/tests.TestSuite_checksum_offload.rst  |   8 ++
 doc/api/dts/tests.TestSuite_dual_vlan.rst     |   8 ++
 .../dts/tests.TestSuite_dynamic_config.rst    |   8 ++
 .../tests.TestSuite_dynamic_queue_conf.rst    |   8 ++
 doc/api/dts/tests.TestSuite_hello_world.rst   |   8 ++
 doc/api/dts/tests.TestSuite_l2fwd.rst         |   8 ++
 doc/api/dts/tests.TestSuite_mac_filter.rst    |   8 ++
 doc/api/dts/tests.TestSuite_mtu.rst           |   8 ++
 .../tests.TestSuite_pmd_buffer_scatter.rst    |   8 ++
 ...tSuite_port_restart_config_persistency.rst |   8 ++
 .../dts/tests.TestSuite_promisc_support.rst   |   8 ++
 .../dts/tests.TestSuite_queue_start_stop.rst  |   8 ++
 doc/api/dts/tests.TestSuite_smoke_tests.rst   |   8 ++
 doc/api/dts/tests.TestSuite_softnic.rst       |   8 ++
 doc/api/dts/tests.TestSuite_uni_pkt.rst       |   8 ++
 doc/api/dts/tests.TestSuite_vlan.rst          |   8 ++
 doc/api/dts/tests.rst                         |  11 ++
 doc/guides/conf.py                            |   1 +
 doc/guides/tools/dts.rst                      |  25 ++--
 dts/.gitignore                                |   2 +-
 dts/framework/config/__init__.py              | 112 ++++++++++-------
 dts/framework/config/test_run.py              |  53 ++++++--
 dts/framework/runner.py                       |  33 +++--
 dts/framework/settings.py                     |  33 +++--
 dts/framework/test_run.py                     |  23 ++--
 dts/framework/test_suite.py                   |  25 ++--
 dts/test_run.example.yaml                     |  43 +++++++
 dts/test_runs.example.yaml                    |  43 -------
 dts/tests/TestSuite_dynamic_config.py         |  10 +-
 dts/tests/TestSuite_hello_world.py            |  14 ++-
 dts/tests/TestSuite_mac_filter.py             |  42 +++----
 dts/tests/TestSuite_mtu.py                    | 114 +++++++++---------
 dts/tests/TestSuite_smoke_tests.py            |   2 -
 dts/tests_config.example.yaml                 |   2 +
 36 files changed, 495 insertions(+), 230 deletions(-)
 create mode 100644 doc/api/dts/tests.TestSuite_blocklist.rst
 create mode 100644 doc/api/dts/tests.TestSuite_checksum_offload.rst
 create mode 100644 doc/api/dts/tests.TestSuite_dual_vlan.rst
 create mode 100644 doc/api/dts/tests.TestSuite_dynamic_config.rst
 create mode 100644 doc/api/dts/tests.TestSuite_dynamic_queue_conf.rst
 create mode 100644 doc/api/dts/tests.TestSuite_hello_world.rst
 create mode 100644 doc/api/dts/tests.TestSuite_l2fwd.rst
 create mode 100644 doc/api/dts/tests.TestSuite_mac_filter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_mtu.rst
 create mode 100644 doc/api/dts/tests.TestSuite_pmd_buffer_scatter.rst
 create mode 100644 doc/api/dts/tests.TestSuite_port_restart_config_persistency.rst
 create mode 100644 doc/api/dts/tests.TestSuite_promisc_support.rst
 create mode 100644 doc/api/dts/tests.TestSuite_queue_start_stop.rst
 create mode 100644 doc/api/dts/tests.TestSuite_smoke_tests.rst
 create mode 100644 doc/api/dts/tests.TestSuite_softnic.rst
 create mode 100644 doc/api/dts/tests.TestSuite_uni_pkt.rst
 create mode 100644 doc/api/dts/tests.TestSuite_vlan.rst
 create mode 100644 doc/api/dts/tests.rst
 create mode 100644 dts/test_run.example.yaml
 delete mode 100644 dts/test_runs.example.yaml
 create mode 100644 dts/tests_config.example.yaml