[v2,0/6] test: refactor crypto unit test framework

  v2:
  - Added macro in place of testcase/testsuite loops.
  - Added more detail in the summary output.
  - Moved testcase counts to the testsuite structure.
  - Flattened testsuite structure to remove union.
  - Added patch for fix of blockcipher test return value.
  - Squashed release note into last patch.

The current crypto unit test framework is not granular enough to
accurately track unit test results. This is caused by one testcase
in a suite actually running multiple testcases, but only returning
one result.
 
The approach taken in this patchset allows a test suite have either a
list of sub-testsuites, or a list of testcases as previously used.
The unit test suite runner can then recursively iterate and run the
sub-testsuites, until it reaches a suite with testcases,
and it then runs each testcase as it had done previously.
 
By allowing this further breakdown into sub-testsuites,
a refactor of the crypto unit tests solves the issue of inaccurate
reporting, as sub-testsuites can be used in place of the testcases
that had multiple testcases hidden on a sub level.
The blockcipher tests previously had these hidden testcases,
but are now sub-testsuites that are dynamically created and added to a
parent test suite, allowing for each testcase status to be reported
directly to the runner.
The cryptodev test suite is broken down into smaller suites that are
used as sub-testsuites, which allows for more flexibility choosing which
sub-testsuites should run for the current device autotest.
The introduction of sub-testsuites also allows for more precise
setup/teardown functions, rather than general ones loaded with
conditions as was seen with the initial setup function used for all
crypto testsuites.
 
For example, when running the cryptodev_aesni_mb_autotest, 
the AESNI MB parent test suite has its own setup function to initialise
the AESNI MB device.
Various sub-testsuites are added to the parent test suite, such as some
of the static suites that were once in the cryptodev_testsuite,
and blockcipher suites.
The unit test runner can then run the AESNI MB parent test suite,
which in turn will run the sub-testsuites.
 
Documentation will be added in a later version of the patchset,
adding to the test document that isn't yet merged. [1]

---
[1] https://patchwork.dpdk.org/project/dpdk/patch/20210309155757.615536-1-aconole@redhat.com/

Ciara Power (6):
  app/test: refactor of unit test suite runner
  test: introduce parent testsuite format
  test/crypto: refactor to use sub-testsuites
  test/crypto: move testsuite params to header file
  test/crypto: fix return value on test skipped
  test/crypto: dynamically build blockcipher suite

 app/test/test.c                        |  226 ++-
 app/test/test.h                        |   23 +-
 app/test/test_cryptodev.c              | 1964 ++++++++++++++++++------
 app/test/test_cryptodev.h              |   20 +
 app/test/test_cryptodev_asym.c         |   93 +-
 app/test/test_cryptodev_blockcipher.c  |  127 +-
 app/test/test_cryptodev_blockcipher.h  |   12 +-
 app/test/test_ipsec.c                  |   32 +-
 doc/guides/rel_notes/release_21_05.rst |    5 +
 9 files changed, 1831 insertions(+), 671 deletions(-)
  

> -----Original Message-----
> From: Ciara Power <ciara.power@intel.com>
> Sent: Friday, April 2, 2021 10:24 PM
> To: dev@dpdk.org
> Cc: declan.doherty@intel.com; gakhil@marvell.com; aconole@redhat.com;
> hemant.agrawal@nxp.com; anoobj@marvell.com; Ruifeng Wang
> <Ruifeng.Wang@arm.com>; asomalap@amd.com; Ajit Khaparde
> (ajit.khaparde@broadcom.com) <ajit.khaparde@broadcom.com>;
> g.singh@nxp.com; Ciara Power <ciara.power@intel.com>
> Subject: [PATCH v2 0/6] test: refactor crypto unit test framework
> 
> v2:
>   - Added macro in place of testcase/testsuite loops.
>   - Added more detail in the summary output.
>   - Moved testcase counts to the testsuite structure.
>   - Flattened testsuite structure to remove union.
>   - Added patch for fix of blockcipher test return value.
>   - Squashed release note into last patch.
> 
> The current crypto unit test framework is not granular enough to accurately
> track unit test results. This is caused by one testcase in a suite actually
> running multiple testcases, but only returning one result.
> 
> The approach taken in this patchset allows a test suite have either a list of
> sub-testsuites, or a list of testcases as previously used.
> The unit test suite runner can then recursively iterate and run the sub-
> testsuites, until it reaches a suite with testcases, and it then runs each
> testcase as it had done previously.
> 
> By allowing this further breakdown into sub-testsuites, a refactor of the
> crypto unit tests solves the issue of inaccurate reporting, as sub-testsuites
> can be used in place of the testcases that had multiple testcases hidden on a
> sub level.
> The blockcipher tests previously had these hidden testcases, but are now
> sub-testsuites that are dynamically created and added to a parent test suite,
> allowing for each testcase status to be reported directly to the runner.
> The cryptodev test suite is broken down into smaller suites that are used as
> sub-testsuites, which allows for more flexibility choosing which sub-
> testsuites should run for the current device autotest.
> The introduction of sub-testsuites also allows for more precise
> setup/teardown functions, rather than general ones loaded with conditions
> as was seen with the initial setup function used for all crypto testsuites.
> 
> For example, when running the cryptodev_aesni_mb_autotest, the AESNI
> MB parent test suite has its own setup function to initialise the AESNI MB
> device.
> Various sub-testsuites are added to the parent test suite, such as some of
> the static suites that were once in the cryptodev_testsuite, and blockcipher
> suites.
> The unit test runner can then run the AESNI MB parent test suite, which in
> turn will run the sub-testsuites.
> 
> Documentation will be added in a later version of the patchset, adding to the
> test document that isn't yet merged. [1]
> 
> ---
> [1]
> https://patchwork.dpdk.org/project/dpdk/patch/20210309155757.615536-1-
> aconole@redhat.com/
> 
> Ciara Power (6):
>   app/test: refactor of unit test suite runner
>   test: introduce parent testsuite format
>   test/crypto: refactor to use sub-testsuites
>   test/crypto: move testsuite params to header file
>   test/crypto: fix return value on test skipped
>   test/crypto: dynamically build blockcipher suite
> 
>  app/test/test.c                        |  226 ++-
>  app/test/test.h                        |   23 +-
>  app/test/test_cryptodev.c              | 1964 ++++++++++++++++++------
>  app/test/test_cryptodev.h              |   20 +
>  app/test/test_cryptodev_asym.c         |   93 +-
>  app/test/test_cryptodev_blockcipher.c  |  127 +-
>  app/test/test_cryptodev_blockcipher.h  |   12 +-
>  app/test/test_ipsec.c                  |   32 +-
>  doc/guides/rel_notes/release_21_05.rst |    5 +
>  9 files changed, 1831 insertions(+), 671 deletions(-)
> 
> --
> 2.25.1

Tested against armv8crypto PMD and the result looks good.
Tested-by: Ruifeng Wang <ruifeng.wang@arm.com>