| Message ID | 20210210145559.4090684-1-aconole@redhat.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Thomas Monjalon |
| Headers | show |
| Series | [v2] guides: add a testing guide for developing tests | expand |
| Context | Check | Description |
|---|---|---|
| ci/checkpatch | success | coding style OK |
| ci/intel-Testing | success | Testing PASS |
| ci/Intel-compilation | success | Compilation OK |
On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote: > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst > new file mode 100644 > index 0000000000..86ca24ce43 > --- /dev/null > +++ b/doc/guides/contributing/testing.rst > @@ -0,0 +1,245 @@ > +.. SPDX-License-Identifier: BSD-3-Clause > + Copyright 2018 The DPDK contributors 2021? > + > +.. _testing_guidelines: I can't find a call to the testing_guidelines reference, so this can be removed. [snip] > +The suites can be selected by adding the ``--suite`` option to the > +``meson test`` command. Ex: ``meson test --suite fast-tests``:: > + > + $ meson test -C build --suite fast-tests > + ninja: Entering directory `/home/aconole/git/dpdk/build' > + [2543/2543] Linking target app/test/dpdk-test. > + 1/60 DPDK:fast-tests / acl_autotest OK 3.17 s > + 2/60 DPDK:fast-tests / bitops_autotest OK 0.22 s > + 3/60 DPDK:fast-tests / byteorder_autotest OK 0.22 s > + 4/60 DPDK:fast-tests / cmdline_autotest OK 0.28 s > + 5/60 DPDK:fast-tests / common_autotest OK 0.57 s > + 6/60 DPDK:fast-tests / cpuflags_autotest OK 0.27 s > + ... Trying this in my build env, I get all tests failing. This is because I run this as a normal user, but the system has hugepages configured. I figured this out quickly since I know the test framework (simply added a echo 0; exit at the top of has-hugepages.sh). But I am not sure a reader of this doc would be able to troubleshoot this. Not sure if this is worth explaining here, or if we can enhance the hugepage check (permissions maybe?). [snip] > +Checking code coverage > +---------------------- > +The meson build system supports generating a code coverage report > +via the `-Db_coverage=true` option, in conjunction with a package > +like **lcov**, to generate an HTML code coverage report. Example:: > + > + $ meson covered -Db_coverage=true At first, I read "covered" as a meson command :-). I prefer an explicit "meson setup covered -Db_coverage=true", but well this is more a matter of taste. > + $ meson test -C covered --suite fast-tests > + $ ninja coverage-html -C covered > + > +The above will generate an html report in the > +`covered/meson-logs/coveragereport/` directory that can be explored > +for detailed code covered information. This can be used to assist > +in test development. > + The rest of the doc lgtm, and this is a good addition, feel free to add my review tag. Thanks Aaron.
On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote: > On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote: > > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst > > new file mode 100644 > > index 0000000000..86ca24ce43 > > --- /dev/null > > +++ b/doc/guides/contributing/testing.rst > > @@ -0,0 +1,245 @@ > > +.. SPDX-License-Identifier: BSD-3-Clause > > + Copyright 2018 The DPDK contributors > > 2021? > > > + > > +.. _testing_guidelines: > > I can't find a call to the testing_guidelines reference, so this can be removed. > > > [snip] > > > +The suites can be selected by adding the ``--suite`` option to the > > +``meson test`` command. Ex: ``meson test --suite fast-tests``:: > > + > > + $ meson test -C build --suite fast-tests > > + ninja: Entering directory `/home/aconole/git/dpdk/build' > > + [2543/2543] Linking target app/test/dpdk-test. > > + 1/60 DPDK:fast-tests / acl_autotest OK 3.17 s > > + 2/60 DPDK:fast-tests / bitops_autotest OK 0.22 s > > + 3/60 DPDK:fast-tests / byteorder_autotest OK 0.22 s > > + 4/60 DPDK:fast-tests / cmdline_autotest OK 0.28 s > > + 5/60 DPDK:fast-tests / common_autotest OK 0.57 s > > + 6/60 DPDK:fast-tests / cpuflags_autotest OK 0.27 s > > + ... > > Trying this in my build env, I get all tests failing. > This is because I run this as a normal user, but the system has > hugepages configured. > I figured this out quickly since I know the test framework (simply > added a echo 0; exit at the top of has-hugepages.sh). > But I am not sure a reader of this doc would be able to troubleshoot this. > > Not sure if this is worth explaining here, or if we can enhance the > hugepage check (permissions maybe?). > > > [snip] > > > +Checking code coverage > > +---------------------- > > +The meson build system supports generating a code coverage report > > +via the `-Db_coverage=true` option, in conjunction with a package > > +like **lcov**, to generate an HTML code coverage report. Example:: > > + > > + $ meson covered -Db_coverage=true > > At first, I read "covered" as a meson command :-). > I prefer an explicit "meson setup covered -Db_coverage=true", but well > this is more a matter of taste. > > I also tend to prefer the build directory name at the end of the command, so I'd suggest: "meson setup -Db_coverage=true covered". Furthermore, while I can understand the use of "covered" as a build directory name, I think for consistency across all docs, we should just use "build" here as the directory name, which again will reduce confusion. "meson setup -Db_coverage build" /Bruce
Bruce Richardson <bruce.richardson@intel.com> writes: > On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote: >> On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote: >> > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst >> > new file mode 100644 >> > index 0000000000..86ca24ce43 >> > --- /dev/null >> > +++ b/doc/guides/contributing/testing.rst >> > @@ -0,0 +1,245 @@ >> > +.. SPDX-License-Identifier: BSD-3-Clause >> > + Copyright 2018 The DPDK contributors >> >> 2021? Whoops, I forgot to update my time machine. >> > + >> > +.. _testing_guidelines: >> >> I can't find a call to the testing_guidelines reference, so this can be removed. done. >> >> [snip] >> >> > +The suites can be selected by adding the ``--suite`` option to the >> > +``meson test`` command. Ex: ``meson test --suite fast-tests``:: >> > + >> > + $ meson test -C build --suite fast-tests >> > + ninja: Entering directory `/home/aconole/git/dpdk/build' >> > + [2543/2543] Linking target app/test/dpdk-test. >> > + 1/60 DPDK:fast-tests / acl_autotest OK 3.17 s >> > + 2/60 DPDK:fast-tests / bitops_autotest OK 0.22 s >> > + 3/60 DPDK:fast-tests / byteorder_autotest OK 0.22 s >> > + 4/60 DPDK:fast-tests / cmdline_autotest OK 0.28 s >> > + 5/60 DPDK:fast-tests / common_autotest OK 0.57 s >> > + 6/60 DPDK:fast-tests / cpuflags_autotest OK 0.27 s >> > + ... >> >> Trying this in my build env, I get all tests failing. >> This is because I run this as a normal user, but the system has >> hugepages configured. >> I figured this out quickly since I know the test framework (simply >> added a echo 0; exit at the top of has-hugepages.sh). >> But I am not sure a reader of this doc would be able to troubleshoot this. >> >> Not sure if this is worth explaining here, or if we can enhance the >> hugepage check (permissions maybe?). I prefer to fix the hugepage check to make the tests SKIP when we don't have hugepages accessible (so we need some kind of permission check in there). I will submit it separately. >> >> [snip] >> >> > +Checking code coverage >> > +---------------------- >> > +The meson build system supports generating a code coverage report >> > +via the `-Db_coverage=true` option, in conjunction with a package >> > +like **lcov**, to generate an HTML code coverage report. Example:: >> > + >> > + $ meson covered -Db_coverage=true >> >> At first, I read "covered" as a meson command :-). >> I prefer an explicit "meson setup covered -Db_coverage=true", but well >> this is more a matter of taste. >> >> > > I also tend to prefer the build directory name at the end of the command, > so I'd suggest: "meson setup -Db_coverage=true covered". Furthermore, > while I can understand the use of "covered" as a build directory name, I > think for consistency across all docs, we should just use "build" here as > the directory name, which again will reduce confusion. "meson setup > -Db_coverage build" Okay - I will paint this bikeshed like: meson setup build -Db_coverage=true It's a little inconsistent everywhere - so I guess a good janitor project would be to clean up all the places we have meson commands. Otherwise, what I've found is that the options generally come after the build directory / command is specified (ex: see octeontx, the arm64 cross build docs, etc.) so I'll keep that for consistency there. Hopefully we will consistently become more consistent :) > /Bruce
On Tue, Mar 02, 2021 at 10:26:59AM -0500, Aaron Conole wrote: > Bruce Richardson <bruce.richardson@intel.com> writes: > > > On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote: > >> On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote: > >> > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst > >> > new file mode 100644 > >> > index 0000000000..86ca24ce43 > >> > --- /dev/null > >> > +++ b/doc/guides/contributing/testing.rst > >> > @@ -0,0 +1,245 @@ > >> > +.. SPDX-License-Identifier: BSD-3-Clause > >> > + Copyright 2018 The DPDK contributors > >> > >> 2021? > > Whoops, I forgot to update my time machine. > > >> > + > >> > +.. _testing_guidelines: > >> > >> I can't find a call to the testing_guidelines reference, so this can be removed. > > done. > > >> > >> [snip] > >> > >> > +The suites can be selected by adding the ``--suite`` option to the > >> > +``meson test`` command. Ex: ``meson test --suite fast-tests``:: > >> > + > >> > + $ meson test -C build --suite fast-tests > >> > + ninja: Entering directory `/home/aconole/git/dpdk/build' > >> > + [2543/2543] Linking target app/test/dpdk-test. > >> > + 1/60 DPDK:fast-tests / acl_autotest OK 3.17 s > >> > + 2/60 DPDK:fast-tests / bitops_autotest OK 0.22 s > >> > + 3/60 DPDK:fast-tests / byteorder_autotest OK 0.22 s > >> > + 4/60 DPDK:fast-tests / cmdline_autotest OK 0.28 s > >> > + 5/60 DPDK:fast-tests / common_autotest OK 0.57 s > >> > + 6/60 DPDK:fast-tests / cpuflags_autotest OK 0.27 s > >> > + ... > >> > >> Trying this in my build env, I get all tests failing. > >> This is because I run this as a normal user, but the system has > >> hugepages configured. > >> I figured this out quickly since I know the test framework (simply > >> added a echo 0; exit at the top of has-hugepages.sh). > >> But I am not sure a reader of this doc would be able to troubleshoot this. > >> > >> Not sure if this is worth explaining here, or if we can enhance the > >> hugepage check (permissions maybe?). > > I prefer to fix the hugepage check to make the tests SKIP when we don't > have hugepages accessible (so we need some kind of permission check in > there). I will submit it separately. > > >> > >> [snip] > >> > >> > +Checking code coverage > >> > +---------------------- > >> > +The meson build system supports generating a code coverage report > >> > +via the `-Db_coverage=true` option, in conjunction with a package > >> > +like **lcov**, to generate an HTML code coverage report. Example:: > >> > + > >> > + $ meson covered -Db_coverage=true > >> > >> At first, I read "covered" as a meson command :-). > >> I prefer an explicit "meson setup covered -Db_coverage=true", but well > >> this is more a matter of taste. > >> > >> > > > > I also tend to prefer the build directory name at the end of the command, > > so I'd suggest: "meson setup -Db_coverage=true covered". Furthermore, > > while I can understand the use of "covered" as a build directory name, I > > think for consistency across all docs, we should just use "build" here as > > the directory name, which again will reduce confusion. "meson setup > > -Db_coverage build" > > Okay - I will paint this bikeshed like: > > meson setup build -Db_coverage=true > > It's a little inconsistent everywhere - so I guess a good janitor > project would be to clean up all the places we have meson commands. > > Otherwise, what I've found is that the options generally come after the > build directory / command is specified (ex: see octeontx, the arm64 > cross build docs, etc.) so I'll keep that for consistency there. > Hopefully we will consistently become more consistent :) > That's some fine bikeshed painting! And I'm fine with all your suggestion. /Bruce
Aaron Conole <aconole@redhat.com> writes: > Bruce Richardson <bruce.richardson@intel.com> writes: > >> On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote: >>> On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote: >>> > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst >>> > new file mode 100644 >>> > index 0000000000..86ca24ce43 >>> > --- /dev/null >>> > +++ b/doc/guides/contributing/testing.rst >>> > @@ -0,0 +1,245 @@ >>> > +.. SPDX-License-Identifier: BSD-3-Clause >>> > + Copyright 2018 The DPDK contributors >>> >>> 2021? > > Whoops, I forgot to update my time machine. > >>> > + >>> > +.. _testing_guidelines: >>> >>> I can't find a call to the testing_guidelines reference, so this can be removed. > > done. > >>> >>> [snip] >>> >>> > +The suites can be selected by adding the ``--suite`` option to the >>> > +``meson test`` command. Ex: ``meson test --suite fast-tests``:: >>> > + >>> > + $ meson test -C build --suite fast-tests >>> > + ninja: Entering directory `/home/aconole/git/dpdk/build' >>> > + [2543/2543] Linking target app/test/dpdk-test. >>> > + 1/60 DPDK:fast-tests / acl_autotest OK 3.17 s >>> > + 2/60 DPDK:fast-tests / bitops_autotest OK 0.22 s >>> > + 3/60 DPDK:fast-tests / byteorder_autotest OK 0.22 s >>> > + 4/60 DPDK:fast-tests / cmdline_autotest OK 0.28 s >>> > + 5/60 DPDK:fast-tests / common_autotest OK 0.57 s >>> > + 6/60 DPDK:fast-tests / cpuflags_autotest OK 0.27 s >>> > + ... >>> >>> Trying this in my build env, I get all tests failing. >>> This is because I run this as a normal user, but the system has >>> hugepages configured. >>> I figured this out quickly since I know the test framework (simply >>> added a echo 0; exit at the top of has-hugepages.sh). >>> But I am not sure a reader of this doc would be able to troubleshoot this. >>> >>> Not sure if this is worth explaining here, or if we can enhance the >>> hugepage check (permissions maybe?). > > I prefer to fix the hugepage check to make the tests SKIP when we don't > have hugepages accessible (so we need some kind of permission check in > there). I will submit it separately. > Here is my PoC for this - if you think it's good enough, I'll submit as formal PATCH. --- index d600fad319..1c3cfb665a 100755 --- a/app/test/has-hugepage.sh +++ b/app/test/has-hugepage.sh @@ -3,7 +3,17 @@ # Copyright 2020 Mellanox Technologies, Ltd if [ "$(uname)" = "Linux" ] ; then - cat /proc/sys/vm/nr_hugepages || echo 0 + nr_hugepages=$(cat /proc/sys/vm/nr_hugepages) + # Need to check if we have permissions to access hugepages + perm="" + for mount in `mount | grep hugetlbfs | awk '{ print $3; }'`; do + test ! -w $mount/. || perm="$mount" + done + if [ "$perm" = "" -o "$nr_hugepages" = "0" ]; then + echo 0 + else + echo $nr_hugepages + fi elif [ "$(uname)" = "FreeBSD" ] ; then echo 1 # assume FreeBSD always has hugepages else ---
On Tue, Mar 9, 2021 at 5:14 PM Aaron Conole <aconole@redhat.com> wrote: > >>> > +The suites can be selected by adding the ``--suite`` option to the > >>> > +``meson test`` command. Ex: ``meson test --suite fast-tests``:: > >>> > + > >>> > + $ meson test -C build --suite fast-tests > >>> > + ninja: Entering directory `/home/aconole/git/dpdk/build' > >>> > + [2543/2543] Linking target app/test/dpdk-test. > >>> > + 1/60 DPDK:fast-tests / acl_autotest OK 3.17 s > >>> > + 2/60 DPDK:fast-tests / bitops_autotest OK 0.22 s > >>> > + 3/60 DPDK:fast-tests / byteorder_autotest OK 0.22 s > >>> > + 4/60 DPDK:fast-tests / cmdline_autotest OK 0.28 s > >>> > + 5/60 DPDK:fast-tests / common_autotest OK 0.57 s > >>> > + 6/60 DPDK:fast-tests / cpuflags_autotest OK 0.27 s > >>> > + ... > >>> > >>> Trying this in my build env, I get all tests failing. > >>> This is because I run this as a normal user, but the system has > >>> hugepages configured. > >>> I figured this out quickly since I know the test framework (simply > >>> added a echo 0; exit at the top of has-hugepages.sh). > >>> But I am not sure a reader of this doc would be able to troubleshoot this. > >>> > >>> Not sure if this is worth explaining here, or if we can enhance the > >>> hugepage check (permissions maybe?). > > > > I prefer to fix the hugepage check to make the tests SKIP when we don't > > have hugepages accessible (so we need some kind of permission check in > > there). I will submit it separately. > > > > Here is my PoC for this - if you think it's good enough, I'll submit as > formal PATCH. > > --- > index d600fad319..1c3cfb665a 100755 > --- a/app/test/has-hugepage.sh > +++ b/app/test/has-hugepage.sh > @@ -3,7 +3,17 @@ > # Copyright 2020 Mellanox Technologies, Ltd > > if [ "$(uname)" = "Linux" ] ; then > - cat /proc/sys/vm/nr_hugepages || echo 0 > + nr_hugepages=$(cat /proc/sys/vm/nr_hugepages) > + # Need to check if we have permissions to access hugepages > + perm="" > + for mount in `mount | grep hugetlbfs | awk '{ print $3; }'`; do > + test ! -w $mount/. || perm="$mount" > + done > + if [ "$perm" = "" -o "$nr_hugepages" = "0" ]; then > + echo 0 > + else > + echo $nr_hugepages > + fi > elif [ "$(uname)" = "FreeBSD" ] ; then > echo 1 # assume FreeBSD always has hugepages > else > --- > I need to think more about the multiple mountpoints case (but I may be imagining too much twisted setups..). At least, this works in my env. We need tests in travis/GHA, and sending a non-RFC patch is the best way to know :-) So +1 for a patch. Thanks Aaron!
David Marchand <david.marchand@redhat.com> writes: > On Tue, Mar 9, 2021 at 5:14 PM Aaron Conole <aconole@redhat.com> wrote: >> >>> > +The suites can be selected by adding the ``--suite`` option to the >> >>> > +``meson test`` command. Ex: ``meson test --suite fast-tests``:: >> >>> > + >> >>> > + $ meson test -C build --suite fast-tests >> >>> > + ninja: Entering directory `/home/aconole/git/dpdk/build' >> >>> > + [2543/2543] Linking target app/test/dpdk-test. >> >>> > + 1/60 DPDK:fast-tests / acl_autotest OK 3.17 s >> >>> > + 2/60 DPDK:fast-tests / bitops_autotest OK 0.22 s >> >>> > + 3/60 DPDK:fast-tests / byteorder_autotest OK 0.22 s >> >>> > + 4/60 DPDK:fast-tests / cmdline_autotest OK 0.28 s >> >>> > + 5/60 DPDK:fast-tests / common_autotest OK 0.57 s >> >>> > + 6/60 DPDK:fast-tests / cpuflags_autotest OK 0.27 s >> >>> > + ... >> >>> >> >>> Trying this in my build env, I get all tests failing. >> >>> This is because I run this as a normal user, but the system has >> >>> hugepages configured. >> >>> I figured this out quickly since I know the test framework (simply >> >>> added a echo 0; exit at the top of has-hugepages.sh). >> >>> But I am not sure a reader of this doc would be able to troubleshoot this. >> >>> >> >>> Not sure if this is worth explaining here, or if we can enhance the >> >>> hugepage check (permissions maybe?). >> > >> > I prefer to fix the hugepage check to make the tests SKIP when we don't >> > have hugepages accessible (so we need some kind of permission check in >> > there). I will submit it separately. >> > >> >> Here is my PoC for this - if you think it's good enough, I'll submit as >> formal PATCH. >> >> --- >> index d600fad319..1c3cfb665a 100755 >> --- a/app/test/has-hugepage.sh >> +++ b/app/test/has-hugepage.sh >> @@ -3,7 +3,17 @@ >> # Copyright 2020 Mellanox Technologies, Ltd >> >> if [ "$(uname)" = "Linux" ] ; then >> - cat /proc/sys/vm/nr_hugepages || echo 0 >> + nr_hugepages=$(cat /proc/sys/vm/nr_hugepages) >> + # Need to check if we have permissions to access hugepages >> + perm="" >> + for mount in `mount | grep hugetlbfs | awk '{ print $3; }'`; do >> + test ! -w $mount/. || perm="$mount" >> + done >> + if [ "$perm" = "" -o "$nr_hugepages" = "0" ]; then >> + echo 0 >> + else >> + echo $nr_hugepages >> + fi >> elif [ "$(uname)" = "FreeBSD" ] ; then >> echo 1 # assume FreeBSD always has hugepages >> else >> --- >> > > I need to think more about the multiple mountpoints case (but I may be > imagining too much twisted setups..). > > At least, this works in my env. > We need tests in travis/GHA, and sending a non-RFC patch is the best > way to know :-) Sent :) > So +1 for a patch. > Thanks Aaron!
diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst index 2fefd91931..41909d949b 100644 --- a/doc/guides/contributing/index.rst +++ b/doc/guides/contributing/index.rst @@ -14,6 +14,7 @@ Contributor's Guidelines abi_versioning documentation patches + testing vulnerability stable cheatsheet diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst new file mode 100644 index 0000000000..86ca24ce43 --- /dev/null +++ b/doc/guides/contributing/testing.rst @@ -0,0 +1,245 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + +.. _testing_guidelines: + +DPDK Testing Guidelines +======================= + +This document outlines the guidelines for running and adding new +tests to the in-tree DPDK test suites. + +The DPDK test suite model is loosely based on the xunit model, where +tests are grouped into test suites, and suites are run by runners. +For a basic overview, see the basic Wikipedia article on xunit: +`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_. + + +Running a test +-------------- + +DPDK tests are run via the main test runner, the `dpdk-test` app. +The `dpdk-test` app is a command-line interface that facilitates +running various tests or test suites. + +There are two modes of operation. The first mode is as an interactive +command shell that allows launching specific test suites. This is +the default operating mode of `dpdk-test` and can be done by:: + + $ ./build/app/test/dpdk-test --dpdk-options-here + EAL: Detected 4 lcore(s) + EAL: Detected 1 NUMA nodes + EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem + EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket + EAL: Selected IOVA mode 'VA' + EAL: Probing VFIO support... + EAL: PCI device 0000:00:1f.6 on NUMA socket -1 + EAL: Invalid NUMA socket, default to 0 + EAL: probe driver: 8086:15d7 net_e1000_em + APP: HPET is not enabled, using TSC as default timer + RTE>> + +At the prompt, simply type the name of the test suite you wish to run +and it will execute. + +The second form is useful for a scripting environment, and is used by +the DPDK meson build system. This mode is invoked by assigning a +specific test suite name to the environment variable `DPDK_TEST` +before invoking the `dpdk-test` command, such as:: + + $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --dpdk-options-here + EAL: Detected 4 lcore(s) + EAL: Detected 1 NUMA nodes + EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem + EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket + EAL: Selected IOVA mode 'VA' + EAL: Probing VFIO support... + EAL: PCI device 0000:00:1f.6 on NUMA socket -1 + EAL: Invalid NUMA socket, default to 0 + EAL: probe driver: 8086:15d7 net_e1000_em + APP: HPET is not enabled, using TSC as default timer + RTE>>version_autotest + Version string: 'DPDK 20.02.0-rc0' + Test OK + RTE>>$ + +The above shows running a specific test case. On success, the return +code will be '0', otherwise it will be set to some error value (such +as '255'). + + +Running all tests +----------------- + +In order to allow developers to quickly execute all the standard +internal tests without needing to remember or look up each test suite +name, the build system includes a standard way of executing the +default test suites. After building via `ninja`, the ``meson test`` +command will execute the standard tests and report errors. + +There are four groups of default test suites. The first group is +the **fast** test suite, which is the largest group of test cases. +These are the bulk of the unit tests to validate functional blocks. +The second group is the **perf** tests. These test suites can take +longer to run and do performance evaluations. The third group is +the **driver** test suite, which is mostly for special hardware +related testing (such as `cryptodev`). The last group are the +**debug** tests. These mostly are used to dump system information. + +The suites can be selected by adding the ``--suite`` option to the +``meson test`` command. Ex: ``meson test --suite fast-tests``:: + + $ meson test -C build --suite fast-tests + ninja: Entering directory `/home/aconole/git/dpdk/build' + [2543/2543] Linking target app/test/dpdk-test. + 1/60 DPDK:fast-tests / acl_autotest OK 3.17 s + 2/60 DPDK:fast-tests / bitops_autotest OK 0.22 s + 3/60 DPDK:fast-tests / byteorder_autotest OK 0.22 s + 4/60 DPDK:fast-tests / cmdline_autotest OK 0.28 s + 5/60 DPDK:fast-tests / common_autotest OK 0.57 s + 6/60 DPDK:fast-tests / cpuflags_autotest OK 0.27 s + ... + + +Adding test suites +------------------ + +To add a test suite to the DPDK test application, create a new test +file for that suite (ex: see *app/test/test_version.c* for the +``version_autotest`` test suite). There are two important functions +for interacting with the test harness: + + 1. REGISTER_TEST_COMMAND(command_name, function_to_execute) + Registers a test command with the name `command_name` and which + runs the function `function_to_execute` when `command_name` is + invoked. + + 2. unit_test_suite_runner(struct unit_test_suite \*) + Returns a runner for a full test suite object, which contains + a test suite name, setup, tear down, and vector of unit test + cases. + +Each test suite has a setup and tear down function that runs at the +beginning and end of the test suite execution. Each unit test has +a similar function for test case setup and tear down. + +Test cases are added to the `.unit_test_cases` element of the unit +test suite structure. Ex: + +.. code-block:: c + :linenos: + + #include <time.h> + + #include <rte_common.h> + #include <rte_cycles.h> + #include <rte_hexdump.h> + #include <rte_random.h> + + #include "test.h" + + static int testsuite_setup(void) { return TEST_SUCCESS; } + static void testsuite_teardown(void) { } + + static int ut_setup(void) { return TEST_SUCCESS; } + static void ut_teardown(void) { } + + static int test_case_first(void) { return TEST_SUCCESS; } + + static struct unit_test_suite example_testsuite = { + .suite_name = "EXAMPLE TEST SUITE", + .setup = testsuite_setup, + .teardown = testsuite_teardown, + .unit_test_cases = { + TEST_CASE_ST(ut_setup, ut_teardown, test_case_first), + + TEST_CASES_END(), /**< NULL terminate unit test array */ + }, + }; + + static int example_tests() + { + return unit_test_suite_runner(&example_testsuite); + } + + REGISTER_TEST_COMMAND(example_autotest, example_tests); + +The above code block is a small example that can be used to create a +complete test suite with test case. + + +Designing a test +---------------- + +Test cases have multiple ways of indicating an error has occurred, +in order to reflect failure state back to the runner. Using the +various methods of indicating errors can assist in not only validating +the requisite functionality is working, but also to help debug when +a change in environment or code has caused things to go wrong. + +The first way to indicate a generic error is by returning a test +result failure, using the *TEST_FAILED* error code. This is the most +basic way of indicating that an error has occurred in a test routine. +It isn't very informative to the user, so it should really be used in +cases where the test has catastrophically failed. + +The preferred method of indicating an error is via the +`RTE_TEST_ASSERT` family of macros, which will immediately return +*TEST_FAILED* error condition, but will also log details about the +failure. The basic form is: + +.. code-block:: c + + RTE_TEST_ASSERT(cond, msg, ...) + +In the above macro, *cond* is the condition to evaluate to **true**. +Any generic condition can go here. The *msg* parameter will be a +message to display if *cond* evaluates to **false**. Some specialized +macros already exist. See `lib/librte_eal/include/rte_test.h` for +a list of defined test assertions. + +Sometimes it is important to indicate that a test needs to be +skipped, either because the environment isn't able to support running +the test, or because some requisite functionality isn't available. The +test suite supports returning a result of `TEST_SKIPPED` during test +case setup, or during test case execution to indicate that the +preconditions of the test aren't available. Ex:: + + $ meson test -C build --suite fast-tests + ninja: Entering directory `/home/aconole/git/dpdk/build + [2543/2543] Linking target app/test/dpdk-test. + 1/60 DPDK:fast-tests / acl_autotest OK 3.17 s + 2/60 DPDK:fast-tests / bitops_autotest OK 0.22 s + 3/60 DPDK:fast-tests / byteorder_autotest OK 0.22 s + ... + 46/60 DPDK:fast-tests / ipsec_autotest SKIP 0.22 s + ... + + +Checking code coverage +---------------------- +The meson build system supports generating a code coverage report +via the `-Db_coverage=true` option, in conjunction with a package +like **lcov**, to generate an HTML code coverage report. Example:: + + $ meson covered -Db_coverage=true + $ meson test -C covered --suite fast-tests + $ ninja coverage-html -C covered + +The above will generate an html report in the +`covered/meson-logs/coveragereport/` directory that can be explored +for detailed code covered information. This can be used to assist +in test development. + + +Adding a suite to the default +----------------------------- + +Adding to one of the default tests involves editing the appropriate +meson build file `app/test/meson.build` and adding the command to +the correct test suite class. Once added, the new test suite will +be run as part of the appropriate class (fast, perf, driver, etc.). + +Some of these default test suites are run during continuous integration +tests, making regression checking automatic for new patches submitted +to the project.
The DPDK testing infrastructure includes a comprehensive set of libraries, utilities, and CI integrations for developers to test their code changes. This isn't well documented, however. Document the basics for adding a test suite to the infrastructure and enabling that test suite for continuous integration platforms so that newer developers can understand how to develop test suites and test cases. Signed-off-by: Aaron Conole <aconole@redhat.com> --- v0->v1: Added information for TEST_SKIPPED and details about generating code coverage to help with ideas for writing unit test cases. v1->v2: Corrected some spelling, rephrased a bit after suggestions by Ray. doc/guides/contributing/index.rst | 1 + doc/guides/contributing/testing.rst | 245 ++++++++++++++++++++++++++++ 2 files changed, 246 insertions(+) create mode 100644 doc/guides/contributing/testing.rst