[v2] doc/guides: add details for new test structure

Message ID 20210716134022.61315-1-ciara.power@intel.com (mailing list archive)
State Superseded, archived
Delegated to: Thomas Monjalon
Headers
Series [v2] doc/guides: add details for new test structure |

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/Intel-compilation success Compilation OK
ci/intel-Testing success Testing PASS

Commit Message

Power, Ciara July 16, 2021, 1:40 p.m. UTC
  The testing guide is now updated to include details about
using sub-testsuites. Some example code is given to demonstrate how
they can be used.

A note is also added to highlight the need for using vdev EAL args when
running cryptodev tests.

Depends-on: patch-95866 ("guides: add a guide for developing unit tests")

Signed-off-by: Ciara Power <ciara.power@intel.com>

---
v2:
  - Updated to depend on v4 of testing doc patch.
  - Added examples for running with cryptodev vdev args.
  - Small formatting fixes.
---
 doc/guides/contributing/unit_test.rst | 93 ++++++++++++++++++++++++++-
 1 file changed, 91 insertions(+), 2 deletions(-)
  

Comments

Fan Zhang July 19, 2021, 11:06 a.m. UTC | #1
> -----Original Message-----
> From: Power, Ciara <ciara.power@intel.com>
> Sent: Friday, July 16, 2021 2:40 PM
> To: dev@dpdk.org
> Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty, Declan
> <declan.doherty@intel.com>; aconole@redhat.com; Power, Ciara
> <ciara.power@intel.com>
> Subject: [PATCH v2] doc/guides: add details for new test structure
> 
> The testing guide is now updated to include details about
> using sub-testsuites. Some example code is given to demonstrate how
> they can be used.
> 
> A note is also added to highlight the need for using vdev EAL args when
> running cryptodev tests.
> 
> Depends-on: patch-95866 ("guides: add a guide for developing unit tests")
> 
> Signed-off-by: Ciara Power <ciara.power@intel.com>
> 

Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
  
Thomas Monjalon July 31, 2021, 5:41 p.m. UTC | #2
16/07/2021 15:40, Ciara Power:
> The testing guide is now updated to include details about
> using sub-testsuites. Some example code is given to demonstrate how
> they can be used.

The trend is to avoid adding code in the doc,
but include some existing code with literalinclude instead.
Can it be applied here?
  
Power, Ciara Aug. 3, 2021, 12:11 p.m. UTC | #3
>-----Original Message-----
>From: Thomas Monjalon <thomas@monjalon.net>
>Sent: Saturday 31 July 2021 18:42
>To: Power, Ciara <ciara.power@intel.com>
>Cc: dev@dpdk.org; Zhang, Roy Fan <roy.fan.zhang@intel.com>; Doherty,
>Declan <declan.doherty@intel.com>; aconole@redhat.com
>Subject: Re: [dpdk-dev] [PATCH v2] doc/guides: add details for new test
>structure
>
>16/07/2021 15:40, Ciara Power:
>> The testing guide is now updated to include details about using
>> sub-testsuites. Some example code is given to demonstrate how they can
>> be used.
>
>The trend is to avoid adding code in the doc, but include some existing code
>with literalinclude instead.
>Can it be applied here?
>

Hi Thomas, thanks for the review.

I considered this when creating the patch, but chose to follow the style of the example in Aaron's doc patch for consistency.
To include existing code, it would need to be functions from the cryptodev autotest, but I feel there would be extra code that isn't needed to demonstrate using this framework.
I tried to keep the example as simple and short as possible to help with understanding.

Thanks,
Ciara
  
John McNamara Aug. 6, 2021, 10 a.m. UTC | #4
Acked-by: John McNamara <john.mcnamara@intel.com>
  

Patch

diff --git a/doc/guides/contributing/unit_test.rst b/doc/guides/contributing/unit_test.rst
index b274bd5f93..265d58d9ea 100644
--- a/doc/guides/contributing/unit_test.rst
+++ b/doc/guides/contributing/unit_test.rst
@@ -185,13 +185,21 @@  for interacting with the test harness:
 
   2. unit_test_suite_runner(struct unit_test_suite \*)
      Returns a runner for a full test suite object, which contains
-     a test suite name, setup, tear down, and vector of unit test
-     cases.
+     a test suite name, setup, tear down, a pointer to a list of
+     sub-testsuites, and vector of unit test cases.
 
 Each test suite has a setup and tear down function that runs at the
 beginning and end of the test suite execution.  Each unit test has
 a similar function for test case setup and tear down.
 
+Each testsuite may use a nested list of sub-testsuites,
+which are iterated by the unit_test_suite_runner.
+This support allows for better granularity when designing test suites.
+The sub-testsuites list can also be used in parallel with the vector
+of testcases, in this case the testcases will be run,
+and then each sub-testsuite is executed. To see an example of a
+testsuite using sub-testsuites, see *app/test/test_cryptodev.c*.
+
 Test cases are added to the `.unit_test_cases` element of the appropriate
 unit test suite structure.  An example of both a test suite and a case:
 
@@ -236,6 +244,70 @@  unit test suite structure.  An example of both a test suite and a case:
 The above code block is a small example that can be used to create a
 complete test suite with test case.
 
+Sub-testsuites can be added to the `.unit_test_suites` element of
+the unit test suite structure, for example:
+
+.. code-block:: c
+   :linenos:
+
+   static int testsuite_setup(void) { return TEST_SUCCESS; }
+   static void testsuite_teardown(void) { }
+
+   static int ut_setup(void) { return TEST_SUCCESS; }
+   static void ut_teardown(void) { }
+
+   static int test_case_first(void) { return TEST_SUCCESS; }
+
+   static struct unit_test_suite example_parent_testsuite = {
+          .suite_name = "EXAMPLE PARENT TEST SUITE",
+          .setup = testsuite_setup,
+          .teardown = testsuite_teardown,
+          .unit_test_cases = {TEST_CASES_END()}
+   };
+
+   static int sub_testsuite_setup(void) { return TEST_SUCCESS; }
+   static void sub_testsuite_teardown(void) { }
+
+   static struct unit_test_suite example_sub_testsuite = {
+          .suite_name = "EXAMPLE SUB TEST SUITE",
+          .setup = sub_testsuite_setup,
+          .teardown = sub_testsuite_teardown,
+          .unit_test_cases = {
+               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+               TEST_CASES_END(), /**< NULL terminate unit test array */
+          },
+   };
+
+   static struct unit_test_suite end_testsuite = {
+          .suite_name = NULL,
+          .setup = NULL,
+          .teardown = NULL,
+          .unit_test_suites = NULL
+   };
+
+   static int example_tests()
+   {
+       uint8_t ret, i = 0;
+       struct unit_test_suite *sub_suites[] = {
+              &example_sub_testsuite,
+              &end_testsuite /**< NULL test suite to indicate end of list */
+        };
+
+       example_parent_testsuite.unit_test_suites =
+               malloc(sizeof(struct unit_test_suite *) * RTE_DIM(sub_suites));
+
+       for (i = 0; i < RTE_DIM(sub_suites); i++)
+           example_parent_testsuite.unit_test_suites[i] = sub_suites[i];
+
+       ret = unit_test_suite_runner(&example_parent_testsuite);
+       free(example_parent_testsuite.unit_test_suites);
+
+       return ret;
+   }
+
+   REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
 
 Designing a test
 ----------------
@@ -325,3 +397,20 @@  In general, when a test is added to the `dpdk-test` application, it
 probably should be added to a meson test suite, but the choice is
 left to maintainers and individual developers.  Preference is to add
 tests to the meson test suites.
+
+
+Running Cryptodev Tests
+-----------------------
+
+When running cryptodev tests, the user must create any required virtual
+device via EAL args, as this is not automatically done by the test::
+
+  $ ./build/app/test/dpdk-test --vdev crypto_aesni_mb
+  $ meson test -C build --suite driver-tests \
+      --test-args="--vdev crypto_aesni_mb"
+
+.. note::
+
+   The cryptodev_scheduler_autotest is the only exception to this.
+   This vdev will be created automatically by the test app,
+   as it requires a more complex setup than other vdevs.