[v4,0/7] test case blocking and logging

  We currently don't record test case results that couldn't be executed
because of a previous failure, such as when a test suite setup failed,
resulting in no executed test cases.

In order to record the test cases that couldn't be executed, we must
know the lists of test suites and test cases ahead of the actual test
suite execution, as an error could occur before we even start executing
test suites.

In addition, the patch series contains two refactors and one
improvement.

The first refactor is closely related. The dts.py was renamed to
runner.py and given a clear purpose - running the test suites and all
other orchestration needed to run test suites. The logic for this was
not all in the original dts.py module and it was brought there. The
runner is also responsible for recording results, which is the blocked
test cases are recorded.

The other refactor, logging, is related to the first refactor. The
logging module was simplified while extending capabilities. Each test
suite logs into its own log file in addition to the main log file which
the runner must handle (as it knows when we start executing particular
test suites). The runner also handles the switching between execution
stages for the purposes of logging.

The one aforementioned improvement is in unifying how we specify test
cases in the conf.yaml file and in the environment variable/command line
argument.

v2:
Rebase and update of the whole patch. There are changes in all parts of
the code, mainly improving the design and logic.
Also added the last patch which improves test suite specification on the
cmdline.

v3:
Improved variables in _get_test_suite_class along with docstring.
Fixed smoke test suite not being added into the list of test suites to
be executed.

v4:
Added the checking of CamelCase convention when discovering test cases.
Also added additional test stages. The stages were split into setup and
teardown.

Juraj Linkeš (7):
  dts: convert dts.py methods to class
  dts: move test suite execution logic to DTSRunner
  dts: filter test suites in executions
  dts: reorganize test result
  dts: block all test cases when earlier setup fails
  dts: refactor logging configuration
  dts: improve test suite and case filtering

 doc/guides/tools/dts.rst                      |  14 +-
 dts/framework/config/__init__.py              |  36 +-
 dts/framework/config/conf_yaml_schema.json    |   2 +-
 dts/framework/dts.py                          | 338 ---------
 dts/framework/logger.py                       | 246 +++---
 dts/framework/remote_session/__init__.py      |   6 +-
 .../interactive_remote_session.py             |   6 +-
 .../remote_session/interactive_shell.py       |   6 +-
 .../remote_session/remote_session.py          |   8 +-
 dts/framework/runner.py                       | 711 ++++++++++++++++++
 dts/framework/settings.py                     | 188 +++--
 dts/framework/test_result.py                  | 565 ++++++++------
 dts/framework/test_suite.py                   | 239 +-----
 dts/framework/testbed_model/node.py           |  11 +-
 dts/framework/testbed_model/os_session.py     |   7 +-
 .../traffic_generator/traffic_generator.py    |   6 +-
 dts/main.py                                   |   9 +-
 dts/pyproject.toml                            |   3 +
 dts/tests/TestSuite_os_udp.py                 |   2 +-
 dts/tests/TestSuite_smoke_tests.py            |   2 +-
 20 files changed, 1375 insertions(+), 1030 deletions(-)
 delete mode 100644 dts/framework/dts.py
 create mode 100644 dts/framework/runner.py
  

The Community CI Testing Lab had an infra failure this morning and some
patches including yours were affected with false failures. The issue is now
resolved and we are rerunning the tests in question for all patches
submitted today.

On Fri, Mar 1, 2024 at 5:55 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:

> We currently don't record test case results that couldn't be executed
> because of a previous failure, such as when a test suite setup failed,
> resulting in no executed test cases.
>
> In order to record the test cases that couldn't be executed, we must
> know the lists of test suites and test cases ahead of the actual test
> suite execution, as an error could occur before we even start executing
> test suites.
>
> In addition, the patch series contains two refactors and one
> improvement.
>
> The first refactor is closely related. The dts.py was renamed to
> runner.py and given a clear purpose - running the test suites and all
> other orchestration needed to run test suites. The logic for this was
> not all in the original dts.py module and it was brought there. The
> runner is also responsible for recording results, which is the blocked
> test cases are recorded.
>
> The other refactor, logging, is related to the first refactor. The
> logging module was simplified while extending capabilities. Each test
> suite logs into its own log file in addition to the main log file which
> the runner must handle (as it knows when we start executing particular
> test suites). The runner also handles the switching between execution
> stages for the purposes of logging.
>
> The one aforementioned improvement is in unifying how we specify test
> cases in the conf.yaml file and in the environment variable/command line
> argument.
>
> v2:
> Rebase and update of the whole patch. There are changes in all parts of
> the code, mainly improving the design and logic.
> Also added the last patch which improves test suite specification on the
> cmdline.
>
> v3:
> Improved variables in _get_test_suite_class along with docstring.
> Fixed smoke test suite not being added into the list of test suites to
> be executed.
>
> v4:
> Added the checking of CamelCase convention when discovering test cases.
> Also added additional test stages. The stages were split into setup and
> teardown.
>
> Juraj Linkeš (7):
>   dts: convert dts.py methods to class
>   dts: move test suite execution logic to DTSRunner
>   dts: filter test suites in executions
>   dts: reorganize test result
>   dts: block all test cases when earlier setup fails
>   dts: refactor logging configuration
>   dts: improve test suite and case filtering
>
>  doc/guides/tools/dts.rst                      |  14 +-
>  dts/framework/config/__init__.py              |  36 +-
>  dts/framework/config/conf_yaml_schema.json    |   2 +-
>  dts/framework/dts.py                          | 338 ---------
>  dts/framework/logger.py                       | 246 +++---
>  dts/framework/remote_session/__init__.py      |   6 +-
>  .../interactive_remote_session.py             |   6 +-
>  .../remote_session/interactive_shell.py       |   6 +-
>  .../remote_session/remote_session.py          |   8 +-
>  dts/framework/runner.py                       | 711 ++++++++++++++++++
>  dts/framework/settings.py                     | 188 +++--
>  dts/framework/test_result.py                  | 565 ++++++++------
>  dts/framework/test_suite.py                   | 239 +-----
>  dts/framework/testbed_model/node.py           |  11 +-
>  dts/framework/testbed_model/os_session.py     |   7 +-
>  .../traffic_generator/traffic_generator.py    |   6 +-
>  dts/main.py                                   |   9 +-
>  dts/pyproject.toml                            |   3 +
>  dts/tests/TestSuite_os_udp.py                 |   2 +-
>  dts/tests/TestSuite_smoke_tests.py            |   2 +-
>  20 files changed, 1375 insertions(+), 1030 deletions(-)
>  delete mode 100644 dts/framework/dts.py
>  create mode 100644 dts/framework/runner.py
>
> --
> 2.34.1
>
>
  

> Juraj Linkeš (7):
>   dts: convert dts.py methods to class
>   dts: move test suite execution logic to DTSRunner
>   dts: filter test suites in executions
>   dts: reorganize test result
>   dts: block all test cases when earlier setup fails
>   dts: refactor logging configuration
>   dts: improve test suite and case filtering

Applied, thanks.