[RFC,0/7] dts: revamp framework

Hi there,

This series enables the topology configuration and implements it in the
framework. Moreover, it performs quite a few refactors and a change on
how the test suites operate through the use of a Context. Finally, the
runtime internals are now isolated under the new TestRun and are further
reworked into a finite state machine for ease of handling.

Mind that unfortunately these commits may be breaking in intermediate
steps. Due to the amount of work done in one go, it was rather difficult
to have individually working commits.

I am currently requesting for comments on how we can improve this
further. In the meantime, I am going to remove node discrimination (TG
vs SUT) in the nodes configuration and allow the test run to define the
TG and SUT configurations.

I understand this is a lot of changes so please bear with me, I may have
missed some documentation changes, or even added proper documentation in
some instances. Please do point everything out.

Best,
Luca

Luca Vizzarro (7):
  dts: add port topology configuration
  dts: isolate test specification to config
  dts: revamp Topology model
  dts: improve Port model
  dts: add runtime status
  dts: add global runtime context
  dts: revamp runtime internals

 doc/api/dts/framework.context.rst             |   8 +
 doc/api/dts/framework.status.rst              |   8 +
 doc/api/dts/framework.test_run.rst            |   8 +
 doc/api/dts/index.rst                         |   3 +
 doc/guides/conf.py                            |   3 +-
 dts/framework/config/__init__.py              | 138 +++--
 dts/framework/config/node.py                  |  25 +-
 dts/framework/config/test_run.py              | 144 +++++-
 dts/framework/context.py                      | 107 ++++
 dts/framework/exception.py                    |  33 +-
 dts/framework/logger.py                       |  36 +-
 dts/framework/remote_session/dpdk_shell.py    |  53 +-
 .../single_active_interactive_shell.py        |  14 +-
 dts/framework/remote_session/testpmd_shell.py |  27 +-
 dts/framework/runner.py                       | 485 +-----------------
 dts/framework/status.py                       |  64 +++
 dts/framework/test_result.py                  | 136 +----
 dts/framework/test_run.py                     | 443 ++++++++++++++++
 dts/framework/test_suite.py                   |  73 ++-
 dts/framework/testbed_model/capability.py     |  42 +-
 dts/framework/testbed_model/linux_session.py  |  47 +-
 dts/framework/testbed_model/node.py           |  29 +-
 dts/framework/testbed_model/os_session.py     |  16 +-
 dts/framework/testbed_model/port.py           |  97 ++--
 dts/framework/testbed_model/sut_node.py       |  50 +-
 dts/framework/testbed_model/tg_node.py        |  30 +-
 dts/framework/testbed_model/topology.py       | 165 +++---
 dts/framework/utils.py                        |   8 +-
 dts/nodes.example.yaml                        |  24 +-
 dts/test_runs.example.yaml                    |   3 +
 dts/tests/TestSuite_blocklist.py              |   6 +-
 dts/tests/TestSuite_checksum_offload.py       |  14 +-
 dts/tests/TestSuite_dual_vlan.py              |   6 +-
 dts/tests/TestSuite_dynamic_config.py         |   8 +-
 dts/tests/TestSuite_dynamic_queue_conf.py     |   1 -
 dts/tests/TestSuite_hello_world.py            |   2 +-
 dts/tests/TestSuite_l2fwd.py                  |   9 +-
 dts/tests/TestSuite_mac_filter.py             |  10 +-
 dts/tests/TestSuite_mtu.py                    |  17 +-
 dts/tests/TestSuite_pmd_buffer_scatter.py     |   9 +-
 ...stSuite_port_restart_config_persistency.py |   8 +-
 dts/tests/TestSuite_promisc_support.py        |   8 +-
 dts/tests/TestSuite_smoke_tests.py            |   3 +-
 dts/tests/TestSuite_softnic.py                |   4 +-
 dts/tests/TestSuite_uni_pkt.py                |  14 +-
 dts/tests/TestSuite_vlan.py                   |   8 +-
 46 files changed, 1287 insertions(+), 1159 deletions(-)
 create mode 100644 doc/api/dts/framework.context.rst
 create mode 100644 doc/api/dts/framework.status.rst
 create mode 100644 doc/api/dts/framework.test_run.rst
 create mode 100644 dts/framework/context.py
 create mode 100644 dts/framework/status.py
 create mode 100644 dts/framework/test_run.py
  

Hi Luca,

I saw in the meeting minutes that the main purpose of this series is
to implement the separation of concern principle better in DTS. Just
wondering, what parts of the current framework did you think needed to
be separated and why? I'm taking an OOP class this semester and we
just started talking in depth about separation of concerns, so if you
wouldn't mind I'd be interested in your thought process. Working on a
review for the series currently as well, should be done relatively
soon.

Thanks,
Dean
  

Hi there,

sending some more final code, hence dropping the RFC.

v2:
- applied suggested changes to the port topology config
- re-implemented FSM from a generator function to a more traditional
  approach where each state is actually is self-contained in its own
  class.
- dropped the status enums (as a consequence of the above)
- fixed some docstrings

Best,
Luca

Luca Vizzarro (7):
  dts: add port topology configuration
  dts: isolate test specification to config
  dts: revamp Topology model
  dts: improve Port model
  dts: add global runtime context
  dts: revamp runtime internals
  dts: remove node distinction

 doc/api/dts/framework.context.rst             |   8 +
 doc/api/dts/framework.remote_session.dpdk.rst |   8 +
 doc/api/dts/framework.remote_session.rst      |   1 +
 doc/api/dts/framework.test_run.rst            |   8 +
 doc/api/dts/index.rst                         |   2 +
 doc/guides/conf.py                            |   3 +-
 dts/framework/config/__init__.py              | 138 ++--
 dts/framework/config/node.py                  |  96 +--
 dts/framework/config/test_run.py              | 214 +++++-
 dts/framework/context.py                      | 120 ++++
 dts/framework/exception.py                    |  33 +-
 dts/framework/logger.py                       |  26 +-
 .../sut_node.py => remote_session/dpdk.py}    | 462 ++++++-------
 dts/framework/remote_session/dpdk_shell.py    |  59 +-
 .../single_active_interactive_shell.py        |  12 +-
 dts/framework/remote_session/testpmd_shell.py |  27 +-
 dts/framework/runner.py                       | 547 +--------------
 dts/framework/test_result.py                  | 138 +---
 dts/framework/test_run.py                     | 641 ++++++++++++++++++
 dts/framework/test_suite.py                   |  80 ++-
 dts/framework/testbed_model/capability.py     |  70 +-
 dts/framework/testbed_model/linux_session.py  |  47 +-
 dts/framework/testbed_model/node.py           | 100 +--
 dts/framework/testbed_model/os_session.py     |  16 +-
 dts/framework/testbed_model/port.py           |  97 ++-
 dts/framework/testbed_model/tg_node.py        | 105 ---
 dts/framework/testbed_model/topology.py       | 165 ++---
 .../traffic_generator/__init__.py             |   8 +-
 .../testbed_model/traffic_generator/scapy.py  |  12 +-
 .../traffic_generator/traffic_generator.py    |   9 +-
 dts/framework/utils.py                        |   8 +-
 dts/nodes.example.yaml                        |  24 +-
 dts/test_runs.example.yaml                    |   4 +
 dts/tests/TestSuite_blocklist.py              |   6 +-
 dts/tests/TestSuite_checksum_offload.py       |  14 +-
 dts/tests/TestSuite_dual_vlan.py              |   6 +-
 dts/tests/TestSuite_dynamic_config.py         |   8 +-
 dts/tests/TestSuite_dynamic_queue_conf.py     |   1 -
 dts/tests/TestSuite_hello_world.py            |   2 +-
 dts/tests/TestSuite_l2fwd.py                  |   9 +-
 dts/tests/TestSuite_mac_filter.py             |  10 +-
 dts/tests/TestSuite_mtu.py                    |  17 +-
 dts/tests/TestSuite_pmd_buffer_scatter.py     |   9 +-
 ...stSuite_port_restart_config_persistency.py |   8 +-
 dts/tests/TestSuite_promisc_support.py        |   8 +-
 dts/tests/TestSuite_smoke_tests.py            |   9 +-
 dts/tests/TestSuite_softnic.py                |   6 +-
 dts/tests/TestSuite_uni_pkt.py                |  14 +-
 dts/tests/TestSuite_vlan.py                   |   8 +-
 49 files changed, 1726 insertions(+), 1697 deletions(-)
 create mode 100644 doc/api/dts/framework.context.rst
 create mode 100644 doc/api/dts/framework.remote_session.dpdk.rst
 create mode 100644 doc/api/dts/framework.test_run.rst
 create mode 100644 dts/framework/context.py
 rename dts/framework/{testbed_model/sut_node.py => remote_session/dpdk.py} (59%)
 create mode 100644 dts/framework/test_run.py
 delete mode 100644 dts/framework/testbed_model/tg_node.py
  

Forgot to mention that I've also removed the distinction between SUT and 
TG nodes. These are now all the same nodes. Runtime configuration for 
DPDK and TG now lies within the test run.
  

On 04/02/2025 21:08, Dean Marx wrote:
> Hi Luca,
> 
> I saw in the meeting minutes that the main purpose of this series is
> to implement the separation of concern principle better in DTS. Just
> wondering, what parts of the current framework did you think needed to
> be separated and why? I'm taking an OOP class this semester and we
> just started talking in depth about separation of concerns, so if you
> wouldn't mind I'd be interested in your thought process. Working on a
> review for the series currently as well, should be done relatively
> soon.
> 
> Thanks,
> Dean

Hi Dean,

For starters all the major logic was held within the `runner.py` file, 
which is not ideal. When some logic touches the same object too often, 
maybe its an indicator that it actually belongs to that class instead. 
For example, the test suite and cases are configured within the 
configuration. Anything configuration should be kept within the 
configuration, the runner should assume that the configuration is final 
and is ready to be fed to the classes. For this reason, I've moved the 
test suite and cases filtering logic there. Test runs are a whole 
concept on their own, and they have their own logic. For this reason it 
is a better idea to have them individually in their own dedicated class 
where they can control themselves. As opposed to having a runner.py 
basically dealing with every part of the framework in the same place. If 
you were to make a graph with all the linked classes to runner.py, I am 
sure it'd look like spaghetti. We don't want that. We want a hierarical 
and structured approach. Separation of concerns will aid this by keeping 
things where they belong, minimising import/export. This will also help 
with circular import issues.

Hope this provides some insight!

Luca