[v3] app/test: add support for skipping tests
Checks
Commit Message
When called from automated tools, like meson test, it is often useful to
skip tests in a test suite, without having to alter the test build. To
do so, we add support for DPDK_TEST_SKIP environment variable, where one
can provide a comma-separated list of tests. When the test binary is
called to run one of the tests on the list via either cmdline parameter
or environment variable (as done with meson test), the test will not
actually be run, but will be reported skipped.
Example run:
$ DPDK_TEST_SKIP=dump_devargs,dump_ring meson test --suite=debug-tests
...
1/9 DPDK:debug-tests / dump_devargs SKIP 1.11s
2/9 DPDK:debug-tests / dump_log_types OK 1.06s
3/9 DPDK:debug-tests / dump_malloc_heaps OK 1.11s
4/9 DPDK:debug-tests / dump_malloc_stats OK 1.07s
5/9 DPDK:debug-tests / dump_mempool OK 1.11s
6/9 DPDK:debug-tests / dump_memzone OK 1.06s
7/9 DPDK:debug-tests / dump_physmem OK 1.13s
8/9 DPDK:debug-tests / dump_ring SKIP 1.04s
9/9 DPDK:debug-tests / dump_struct_sizes OK 1.10s
Ok: 7
Expected Fail: 0
Fail: 0
Unexpected Pass: 0
Skipped: 2
Timeout: 0
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
---
V3: Fix checkpatch whitespace issue
V2: Check return value from rte_strsplit, just in case.
---
app/test/test.c | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
Comments
On Thu, Aug 31, 2023 at 11:18:06AM +0100, Bruce Richardson wrote:
> When called from automated tools, like meson test, it is often useful to
> skip tests in a test suite, without having to alter the test build. To
> do so, we add support for DPDK_TEST_SKIP environment variable, where one
> can provide a comma-separated list of tests. When the test binary is
> called to run one of the tests on the list via either cmdline parameter
> or environment variable (as done with meson test), the test will not
> actually be run, but will be reported skipped.
>
> Example run:
> $ DPDK_TEST_SKIP=dump_devargs,dump_ring meson test --suite=debug-tests
> ...
> 1/9 DPDK:debug-tests / dump_devargs SKIP 1.11s
> 2/9 DPDK:debug-tests / dump_log_types OK 1.06s
> 3/9 DPDK:debug-tests / dump_malloc_heaps OK 1.11s
> 4/9 DPDK:debug-tests / dump_malloc_stats OK 1.07s
> 5/9 DPDK:debug-tests / dump_mempool OK 1.11s
> 6/9 DPDK:debug-tests / dump_memzone OK 1.06s
> 7/9 DPDK:debug-tests / dump_physmem OK 1.13s
> 8/9 DPDK:debug-tests / dump_ring SKIP 1.04s
> 9/9 DPDK:debug-tests / dump_struct_sizes OK 1.10s
>
> Ok: 7
> Expected Fail: 0
> Fail: 0
> Unexpected Pass: 0
> Skipped: 2
> Timeout: 0
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
>
+Tyler
I see this set is failing CI checks due to breaking Windows builds. The
issue seems to be the use of the "strdup" function. I notice in the log
library, that we have a "#define strdup _strdup" macro. Since strdup is
fairly common, widespread function, I think we should consider a more
general approach to it.
Tyler, looking for your input here: should we just globally define strdup
as _strdup for windows in DPDK? Alternatively, some googling indicates that
there is the "_CRT_NONSTDC_NO_DEPRECATE" define which could be used to
enable a whole range of POSIX functions. Should we, or could we, just set
this to ease porting of code over? I'd hate each of our C files to have a
bunch of duplicated #defines at the start to prefix standard unix functions
with "_"s.
Thoughts?
/Bruce
@@ -193,6 +193,25 @@ main(int argc, char **argv)
if (test_count > 0) {
char buf[1024];
+ char *dpdk_test_skip = getenv("DPDK_TEST_SKIP");
+ char *skip_tests[128] = {0};
+ size_t n_skip_tests = 0;
+
+ if (dpdk_test_skip != NULL && strlen(dpdk_test_skip) > 0) {
+ int split_ret;
+ char *dpdk_test_skip_cp = strdup(dpdk_test_skip);
+ if (dpdk_test_skip_cp == NULL) {
+ ret = -1;
+ goto out;
+ }
+ dpdk_test_skip = dpdk_test_skip_cp;
+ split_ret = rte_strsplit(dpdk_test_skip, strlen(dpdk_test_skip),
+ skip_tests, RTE_DIM(skip_tests), ',');
+ if (split_ret > 0)
+ n_skip_tests = split_ret;
+ else
+ free(dpdk_test_skip);
+ }
cl = cmdline_new(main_ctx, "RTE>>", 0, 1);
if (cl == NULL) {
@@ -201,6 +220,15 @@ main(int argc, char **argv)
}
for (i = 0; i < test_count; i++) {
+ /* check if test is to be skipped */
+ for (size_t j = 0; j < n_skip_tests; j++) {
+ if (strcmp(tests[i], skip_tests[j]) == 0) {
+ fprintf(stderr, "Skipping %s [DPDK_TEST_SKIP]\n", tests[i]);
+ ret = TEST_SKIPPED;
+ goto end_of_cmd;
+ }
+ }
+
snprintf(buf, sizeof(buf), "%s\n", tests[i]);
if (cmdline_parse_check(cl, buf) < 0) {
printf("Error: invalid test command: '%s'\n", tests[i]);
@@ -211,9 +239,13 @@ main(int argc, char **argv)
} else
ret = last_test_result;
+end_of_cmd:
if (ret != 0)
break;
}
+ if (n_skip_tests > 0)
+ free(dpdk_test_skip);
+
cmdline_free(cl);
goto out;
} else {