app/test: don't count skipped tests as executed

Message ID 20231113150533.249808-1-bruce.richardson@intel.com (mailing list archive)
State New
Delegated to: Thomas Monjalon
Headers
Series app/test: don't count skipped tests as executed |

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/loongarch-compilation success Compilation OK
ci/loongarch-unit-testing success Unit Testing PASS
ci/github-robot: build success github build: passed
ci/iol-intel-Performance success Performance Testing PASS
ci/iol-intel-Functional success Functional Testing PASS
ci/iol-mellanox-Performance success Performance Testing PASS
ci/iol-compile-amd64-testing success Testing PASS
ci/iol-unit-arm64-testing success Testing PASS
ci/iol-unit-amd64-testing success Testing PASS
ci/iol-compile-arm64-testing success Testing PASS
ci/iol-sample-apps-testing success Testing PASS
ci/Intel-compilation success Compilation OK
ci/intel-Testing success Testing PASS
ci/iol-broadcom-Functional success Functional Testing PASS
ci/iol-broadcom-Performance success Performance Testing PASS
ci/intel-Functional success Functional PASS

Commit Message

Bruce Richardson Nov. 13, 2023, 3:05 p.m. UTC
  The logic around skipped tests is a little confusing in the unit test
runner.
* Any explicitly disabled tests are counted as skipped but not
  executed.
* Any tests that return TEST_SKIPPED are counted as both skipped and
  executed, using the same statistics counters.

This makes the stats very strange and hard to correlate, since the
totals don't add up.  One would expect that SKIPPED + EXECUTED +
UNSUPPORTED == TOTAL, and that PASSED + FAILED == EXECUTED.

To achieve this, mark any tests returning TEST_SKIPPED, or ENOTSUP as
not having executed.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/test.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
  

Patch

diff --git a/app/test/test.c b/app/test/test.c
index bfa9ea52e3..7b882a59de 100644
--- a/app/test/test.c
+++ b/app/test/test.c
@@ -375,11 +375,13 @@  unit_test_suite_runner(struct unit_test_suite *suite)
 
 			if (test_success == TEST_SUCCESS)
 				suite->succeeded++;
-			else if (test_success == TEST_SKIPPED)
+			else if (test_success == TEST_SKIPPED) {
 				suite->skipped++;
-			else if (test_success == -ENOTSUP)
+				suite->executed--;
+			} else if (test_success == -ENOTSUP) {
 				suite->unsupported++;
-			else
+				suite->executed--;
+			} else
 				suite->failed++;
 		} else if (test_success == -ENOTSUP) {
 			suite->unsupported++;