From patchwork Wed May 12 11:36:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Power, Ciara" X-Patchwork-Id: 93208 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5605DA0C43; Wed, 12 May 2021 13:37:26 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B6E3E410DB; Wed, 12 May 2021 13:37:25 +0200 (CEST) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id D7DC24003E for ; Wed, 12 May 2021 13:37:22 +0200 (CEST) IronPort-SDR: grRRL95v5jP+IkMAbJGhfIyBcXgkksL3Iw2f3R1V0FeE6MG095KxfFEPtDq4x2CrpdJgcWFtR1 Ru3XCqjIlZnA== X-IronPort-AV: E=McAfee;i="6200,9189,9981"; a="199733787" X-IronPort-AV: E=Sophos;i="5.82,293,1613462400"; d="scan'208";a="199733787" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 May 2021 04:37:12 -0700 IronPort-SDR: hyOO2r8shCZDah1wXBBxSiHoXtlKStomuc9Ad4oR13q8Ieune1/uIDlLMHus26PQZCyI6sCDD4 FgUV2noZfmDA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,293,1613462400"; d="scan'208";a="625296862" Received: from silpixa00400355.ir.intel.com (HELO silpixa00400355.ger.corp.intel.com) ([10.237.223.148]) by fmsmga005.fm.intel.com with ESMTP; 12 May 2021 04:37:09 -0700 From: Ciara Power To: dev@dpdk.org Cc: declan.doherty@intel.com, gakhil@marvell.com, aconole@redhat.com, hemant.agrawal@nxp.com, anoobj@marvell.com, ruifeng.wang@arm.com, asomalap@amd.com, ajit.khaparde@broadcom.com, g.singh@nxp.com, roy.fan.zhang@intel.com, Ciara Power Date: Wed, 12 May 2021 11:36:50 +0000 Message-Id: <20210512113655.568814-3-ciara.power@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210512113655.568814-1-ciara.power@intel.com> References: <20210512113655.568814-1-ciara.power@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 2/7] test: introduce parent testsuite format X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The current structure for unit testing only allows for running a test suite with nested test cases. This means all test cases for an autotest must be in one suite, which is not ideal. For example, in some cases we may want to run multiple lists of test cases that each require different setup, so should be in separate suites. The unit test suite struct is modified to hold a pointer to a list of sub-testsuite pointers, along with the list of testcases as before. Signed-off-by: Ciara Power Acked-by: Declan Doherty Acked-by: Hemant Agrawal Reviewed-by: Aaron Conole Tested-by: Ruifeng Wang --- v3: - Added condition check to macro that loops sub-testsuites. - Changed execution to allow for mixture of sub-testsuites and testcases, by removing the if/else condition. - Moved testcase execution above sub-testsuite execution, to improve output readability. v2: - Added macro to loop sub-testsuites. - Added sub-testsuite summary detail. --- app/test/test.c | 70 ++++++++++++++++++++++++++++++++++++++++++++++--- app/test/test.h | 1 + 2 files changed, 68 insertions(+), 3 deletions(-) diff --git a/app/test/test.c b/app/test/test.c index 2fb99d0855..ac0a66392a 100644 --- a/app/test/test.c +++ b/app/test/test.c @@ -41,6 +41,12 @@ extern cmdline_parse_ctx_t main_ctx[]; suite->unit_test_cases[iter].testcase; \ iter++, case = suite->unit_test_cases[iter]) +#define FOR_EACH_SUITE_TESTSUITE(iter, suite, sub_ts) \ + for (iter = 0, sub_ts = suite->unit_test_suites ? \ + suite->unit_test_suites[0]:NULL; sub_ts && \ + suite->unit_test_suites[iter]->suite_name != NULL; \ + iter++, sub_ts = suite->unit_test_suites[iter]) + const char *prgname; /* to be set to argv[0] */ static const char *recursive_call; /* used in linux for MP and other tests */ @@ -241,11 +247,26 @@ main(int argc, char **argv) static void unit_test_suite_count_tcs_on_setup_fail(struct unit_test_suite *suite, - int test_success) + int test_success, unsigned int *sub_ts_failed, + unsigned int *sub_ts_skipped, unsigned int *sub_ts_total) { struct unit_test_case tc; + struct unit_test_suite *ts; int i; + FOR_EACH_SUITE_TESTSUITE(i, suite, ts) { + unit_test_suite_count_tcs_on_setup_fail( + ts, test_success, sub_ts_failed, + sub_ts_skipped, sub_ts_total); + suite->total += ts->total; + suite->failed += ts->failed; + suite->skipped += ts->skipped; + if (ts->failed) + (*sub_ts_failed)++; + else + (*sub_ts_skipped)++; + (*sub_ts_total)++; + } FOR_EACH_SUITE_TESTCASE(i, suite, tc) { suite->total++; if (!tc.enabled || test_success == TEST_SKIPPED) @@ -258,6 +279,11 @@ unit_test_suite_count_tcs_on_setup_fail(struct unit_test_suite *suite, static void unit_test_suite_reset_counts(struct unit_test_suite *suite) { + struct unit_test_suite *ts; + int i; + + FOR_EACH_SUITE_TESTSUITE(i, suite, ts) + unit_test_suite_reset_counts(ts); suite->total = 0; suite->executed = 0; suite->succeeded = 0; @@ -269,9 +295,12 @@ unit_test_suite_reset_counts(struct unit_test_suite *suite) int unit_test_suite_runner(struct unit_test_suite *suite) { - int test_success; + int test_success, i, ret; const char *status; struct unit_test_case tc; + struct unit_test_suite *ts; + unsigned int sub_ts_succeeded = 0, sub_ts_failed = 0; + unsigned int sub_ts_skipped = 0, sub_ts_total = 0; unit_test_suite_reset_counts(suite); @@ -288,7 +317,8 @@ unit_test_suite_runner(struct unit_test_suite *suite) * mark them as failed/skipped */ unit_test_suite_count_tcs_on_setup_fail(suite, - test_success); + test_success, &sub_ts_failed, + &sub_ts_skipped, &sub_ts_total); goto suite_summary; } } @@ -342,6 +372,23 @@ unit_test_suite_runner(struct unit_test_suite *suite) printf(" + TestCase [%2d] : %s %s\n", suite->total, tc.name, status); } + FOR_EACH_SUITE_TESTSUITE(i, suite, ts) { + ret = unit_test_suite_runner(ts); + if (ret == TEST_SUCCESS) + sub_ts_succeeded++; + else if (ret == TEST_SKIPPED) + sub_ts_skipped++; + else + sub_ts_failed++; + sub_ts_total++; + + suite->total += ts->total; + suite->succeeded += ts->succeeded; + suite->failed += ts->failed; + suite->skipped += ts->skipped; + suite->unsupported += ts->unsupported; + suite->executed += ts->executed; + } /* Run test suite teardown */ if (suite->teardown) @@ -352,6 +399,23 @@ unit_test_suite_runner(struct unit_test_suite *suite) suite_summary: printf(" + ------------------------------------------------------- +\n"); printf(" + Test Suite Summary : %s\n", suite->suite_name); + printf(" + ------------------------------------------------------- +\n"); + + FOR_EACH_SUITE_TESTSUITE(i, suite, ts) + printf(" + %s : %d/%d passed, %d/%d skipped, " + "%d/%d failed, %d/%d unsupported\n", ts->suite_name, + ts->succeeded, ts->total, ts->skipped, ts->total, + ts->failed, ts->total, ts->unsupported, ts->total); + + if (suite->unit_test_suites) { + printf(" + ------------------------------------------------------- +\n"); + printf(" + Sub Testsuites Total : %2d\n", sub_ts_total); + printf(" + Sub Testsuites Skipped : %2d\n", sub_ts_skipped); + printf(" + Sub Testsuites Passed : %2d\n", sub_ts_succeeded); + printf(" + Sub Testsuites Failed : %2d\n", sub_ts_failed); + printf(" + ------------------------------------------------------- +\n"); + } + printf(" + Tests Total : %2d\n", suite->total); printf(" + Tests Skipped : %2d\n", suite->skipped); printf(" + Tests Executed : %2d\n", suite->executed); diff --git a/app/test/test.h b/app/test/test.h index 55f2850fed..f277df7c9d 100644 --- a/app/test/test.h +++ b/app/test/test.h @@ -144,6 +144,7 @@ struct unit_test_suite { unsigned int skipped; unsigned int failed; unsigned int unsupported; + struct unit_test_suite **unit_test_suites; struct unit_test_case unit_test_cases[]; };