diff mbox series

[v2] guides: add a testing guide for developing tests

Message ID 20210210145559.4090684-1-aconole@redhat.com (mailing list archive)
State Superseded, archived
Delegated to: Thomas Monjalon
Headers show
Series [v2] guides: add a testing guide for developing tests | expand

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/intel-Testing success Testing PASS
ci/Intel-compilation success Compilation OK

Commit Message

Aaron Conole Feb. 10, 2021, 2:55 p.m. UTC
The DPDK testing infrastructure includes a comprehensive set of
libraries, utilities, and CI integrations for developers to test
their code changes.  This isn't well documented, however.

Document the basics for adding a test suite to the infrastructure
and enabling that test suite for continuous integration platforms
so that newer developers can understand how to develop test suites
and test cases.

Signed-off-by: Aaron Conole <aconole@redhat.com>
---
v0->v1: Added information for TEST_SKIPPED and details about generating
        code coverage to help with ideas for writing unit test cases.
v1->v2: Corrected some spelling, rephrased a bit after suggestions by
        Ray.

 doc/guides/contributing/index.rst   |   1 +
 doc/guides/contributing/testing.rst | 245 ++++++++++++++++++++++++++++
 2 files changed, 246 insertions(+)
 create mode 100644 doc/guides/contributing/testing.rst

Comments

David Marchand March 2, 2021, 9:07 a.m. UTC | #1
On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote:
> diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
> new file mode 100644
> index 0000000000..86ca24ce43
> --- /dev/null
> +++ b/doc/guides/contributing/testing.rst
> @@ -0,0 +1,245 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright 2018 The DPDK contributors

2021?

> +
> +.. _testing_guidelines:

I can't find a call to the testing_guidelines reference, so this can be removed.


[snip]

> +The suites can be selected by adding the ``--suite`` option to the
> +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
> +
> +  $ meson test -C build --suite fast-tests
> +  ninja: Entering directory `/home/aconole/git/dpdk/build'
> +  [2543/2543] Linking target app/test/dpdk-test.
> +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
> +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
> +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
> +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
> +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
> +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
> +  ...

Trying this in my build env, I get all tests failing.
This is because I run this as a normal user, but the system has
hugepages configured.
I figured this out quickly since I know the test framework (simply
added a echo 0; exit at the top of has-hugepages.sh).
But I am not sure a reader of this doc would be able to troubleshoot this.

Not sure if this is worth explaining here, or if we can enhance the
hugepage check (permissions maybe?).


[snip]

> +Checking code coverage
> +----------------------
> +The meson build system supports generating a code coverage report
> +via the `-Db_coverage=true` option, in conjunction with a package
> +like **lcov**, to generate an HTML code coverage report.  Example::
> +
> +  $ meson covered -Db_coverage=true

At first, I read "covered" as a meson command :-).
I prefer an explicit "meson setup covered -Db_coverage=true", but well
this is more a matter of taste.


> +  $ meson test -C covered --suite fast-tests
> +  $ ninja coverage-html -C covered
> +
> +The above will generate an html report in the
> +`covered/meson-logs/coveragereport/` directory that can be explored
> +for detailed code covered information.  This can be used to assist
> +in test development.
> +


The rest of the doc lgtm, and this is a good addition, feel free to
add my review tag.
Thanks Aaron.
Bruce Richardson March 2, 2021, 10:04 a.m. UTC | #2
On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote:
> On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote:
> > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
> > new file mode 100644
> > index 0000000000..86ca24ce43
> > --- /dev/null
> > +++ b/doc/guides/contributing/testing.rst
> > @@ -0,0 +1,245 @@
> > +..  SPDX-License-Identifier: BSD-3-Clause
> > +    Copyright 2018 The DPDK contributors
> 
> 2021?
> 
> > +
> > +.. _testing_guidelines:
> 
> I can't find a call to the testing_guidelines reference, so this can be removed.
> 
> 
> [snip]
> 
> > +The suites can be selected by adding the ``--suite`` option to the
> > +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
> > +
> > +  $ meson test -C build --suite fast-tests
> > +  ninja: Entering directory `/home/aconole/git/dpdk/build'
> > +  [2543/2543] Linking target app/test/dpdk-test.
> > +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
> > +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
> > +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
> > +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
> > +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
> > +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
> > +  ...
> 
> Trying this in my build env, I get all tests failing.
> This is because I run this as a normal user, but the system has
> hugepages configured.
> I figured this out quickly since I know the test framework (simply
> added a echo 0; exit at the top of has-hugepages.sh).
> But I am not sure a reader of this doc would be able to troubleshoot this.
> 
> Not sure if this is worth explaining here, or if we can enhance the
> hugepage check (permissions maybe?).
> 
> 
> [snip]
> 
> > +Checking code coverage
> > +----------------------
> > +The meson build system supports generating a code coverage report
> > +via the `-Db_coverage=true` option, in conjunction with a package
> > +like **lcov**, to generate an HTML code coverage report.  Example::
> > +
> > +  $ meson covered -Db_coverage=true
> 
> At first, I read "covered" as a meson command :-).
> I prefer an explicit "meson setup covered -Db_coverage=true", but well
> this is more a matter of taste.
> 
>

I also tend to prefer the build directory name at the end of the command,
so I'd suggest: "meson setup -Db_coverage=true covered".  Furthermore,
while I can understand the use of "covered" as a build directory name, I
think for consistency across all docs, we should just use "build" here as
the directory name, which again will reduce confusion.  "meson setup
-Db_coverage build"

/Bruce
Aaron Conole March 2, 2021, 3:26 p.m. UTC | #3
Bruce Richardson <bruce.richardson@intel.com> writes:

> On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote:
>> On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote:
>> > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
>> > new file mode 100644
>> > index 0000000000..86ca24ce43
>> > --- /dev/null
>> > +++ b/doc/guides/contributing/testing.rst
>> > @@ -0,0 +1,245 @@
>> > +..  SPDX-License-Identifier: BSD-3-Clause
>> > +    Copyright 2018 The DPDK contributors
>> 
>> 2021?

Whoops, I forgot to update my time machine.

>> > +
>> > +.. _testing_guidelines:
>> 
>> I can't find a call to the testing_guidelines reference, so this can be removed.

done.

>> 
>> [snip]
>> 
>> > +The suites can be selected by adding the ``--suite`` option to the
>> > +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
>> > +
>> > +  $ meson test -C build --suite fast-tests
>> > +  ninja: Entering directory `/home/aconole/git/dpdk/build'
>> > +  [2543/2543] Linking target app/test/dpdk-test.
>> > +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
>> > +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
>> > +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
>> > +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
>> > +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
>> > +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
>> > +  ...
>> 
>> Trying this in my build env, I get all tests failing.
>> This is because I run this as a normal user, but the system has
>> hugepages configured.
>> I figured this out quickly since I know the test framework (simply
>> added a echo 0; exit at the top of has-hugepages.sh).
>> But I am not sure a reader of this doc would be able to troubleshoot this.
>> 
>> Not sure if this is worth explaining here, or if we can enhance the
>> hugepage check (permissions maybe?).

I prefer to fix the hugepage check to make the tests SKIP when we don't
have hugepages accessible (so we need some kind of permission check in
there).  I will submit it separately.

>> 
>> [snip]
>> 
>> > +Checking code coverage
>> > +----------------------
>> > +The meson build system supports generating a code coverage report
>> > +via the `-Db_coverage=true` option, in conjunction with a package
>> > +like **lcov**, to generate an HTML code coverage report.  Example::
>> > +
>> > +  $ meson covered -Db_coverage=true
>> 
>> At first, I read "covered" as a meson command :-).
>> I prefer an explicit "meson setup covered -Db_coverage=true", but well
>> this is more a matter of taste.
>> 
>>
>
> I also tend to prefer the build directory name at the end of the command,
> so I'd suggest: "meson setup -Db_coverage=true covered".  Furthermore,
> while I can understand the use of "covered" as a build directory name, I
> think for consistency across all docs, we should just use "build" here as
> the directory name, which again will reduce confusion.  "meson setup
> -Db_coverage build"

Okay - I will paint this bikeshed like:

   meson setup build -Db_coverage=true

It's a little inconsistent everywhere - so I guess a good janitor
project would be to clean up all the places we have meson commands.

Otherwise, what I've found is that the options generally come after the
build directory / command is specified (ex: see octeontx, the arm64
cross build docs, etc.) so I'll keep that for consistency there.
Hopefully we will consistently become more consistent :)

> /Bruce
Bruce Richardson March 2, 2021, 4 p.m. UTC | #4
On Tue, Mar 02, 2021 at 10:26:59AM -0500, Aaron Conole wrote:
> Bruce Richardson <bruce.richardson@intel.com> writes:
> 
> > On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote:
> >> On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote:
> >> > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
> >> > new file mode 100644
> >> > index 0000000000..86ca24ce43
> >> > --- /dev/null
> >> > +++ b/doc/guides/contributing/testing.rst
> >> > @@ -0,0 +1,245 @@
> >> > +..  SPDX-License-Identifier: BSD-3-Clause
> >> > +    Copyright 2018 The DPDK contributors
> >> 
> >> 2021?
> 
> Whoops, I forgot to update my time machine.
> 
> >> > +
> >> > +.. _testing_guidelines:
> >> 
> >> I can't find a call to the testing_guidelines reference, so this can be removed.
> 
> done.
> 
> >> 
> >> [snip]
> >> 
> >> > +The suites can be selected by adding the ``--suite`` option to the
> >> > +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
> >> > +
> >> > +  $ meson test -C build --suite fast-tests
> >> > +  ninja: Entering directory `/home/aconole/git/dpdk/build'
> >> > +  [2543/2543] Linking target app/test/dpdk-test.
> >> > +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
> >> > +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
> >> > +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
> >> > +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
> >> > +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
> >> > +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
> >> > +  ...
> >> 
> >> Trying this in my build env, I get all tests failing.
> >> This is because I run this as a normal user, but the system has
> >> hugepages configured.
> >> I figured this out quickly since I know the test framework (simply
> >> added a echo 0; exit at the top of has-hugepages.sh).
> >> But I am not sure a reader of this doc would be able to troubleshoot this.
> >> 
> >> Not sure if this is worth explaining here, or if we can enhance the
> >> hugepage check (permissions maybe?).
> 
> I prefer to fix the hugepage check to make the tests SKIP when we don't
> have hugepages accessible (so we need some kind of permission check in
> there).  I will submit it separately.
> 
> >> 
> >> [snip]
> >> 
> >> > +Checking code coverage
> >> > +----------------------
> >> > +The meson build system supports generating a code coverage report
> >> > +via the `-Db_coverage=true` option, in conjunction with a package
> >> > +like **lcov**, to generate an HTML code coverage report.  Example::
> >> > +
> >> > +  $ meson covered -Db_coverage=true
> >> 
> >> At first, I read "covered" as a meson command :-).
> >> I prefer an explicit "meson setup covered -Db_coverage=true", but well
> >> this is more a matter of taste.
> >> 
> >>
> >
> > I also tend to prefer the build directory name at the end of the command,
> > so I'd suggest: "meson setup -Db_coverage=true covered".  Furthermore,
> > while I can understand the use of "covered" as a build directory name, I
> > think for consistency across all docs, we should just use "build" here as
> > the directory name, which again will reduce confusion.  "meson setup
> > -Db_coverage build"
> 
> Okay - I will paint this bikeshed like:
> 
>    meson setup build -Db_coverage=true
> 
> It's a little inconsistent everywhere - so I guess a good janitor
> project would be to clean up all the places we have meson commands.
> 
> Otherwise, what I've found is that the options generally come after the
> build directory / command is specified (ex: see octeontx, the arm64
> cross build docs, etc.) so I'll keep that for consistency there.
> Hopefully we will consistently become more consistent :)
> 

That's some fine bikeshed painting! And I'm fine with all your suggestion.

/Bruce
Aaron Conole March 9, 2021, 4:14 p.m. UTC | #5
Aaron Conole <aconole@redhat.com> writes:

> Bruce Richardson <bruce.richardson@intel.com> writes:
>
>> On Tue, Mar 02, 2021 at 10:07:26AM +0100, David Marchand wrote:
>>> On Wed, Feb 10, 2021 at 3:56 PM Aaron Conole <aconole@redhat.com> wrote:
>>> > diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
>>> > new file mode 100644
>>> > index 0000000000..86ca24ce43
>>> > --- /dev/null
>>> > +++ b/doc/guides/contributing/testing.rst
>>> > @@ -0,0 +1,245 @@
>>> > +..  SPDX-License-Identifier: BSD-3-Clause
>>> > +    Copyright 2018 The DPDK contributors
>>> 
>>> 2021?
>
> Whoops, I forgot to update my time machine.
>
>>> > +
>>> > +.. _testing_guidelines:
>>> 
>>> I can't find a call to the testing_guidelines reference, so this can be removed.
>
> done.
>
>>> 
>>> [snip]
>>> 
>>> > +The suites can be selected by adding the ``--suite`` option to the
>>> > +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
>>> > +
>>> > +  $ meson test -C build --suite fast-tests
>>> > +  ninja: Entering directory `/home/aconole/git/dpdk/build'
>>> > +  [2543/2543] Linking target app/test/dpdk-test.
>>> > +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
>>> > +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
>>> > +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
>>> > +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
>>> > +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
>>> > +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
>>> > +  ...
>>> 
>>> Trying this in my build env, I get all tests failing.
>>> This is because I run this as a normal user, but the system has
>>> hugepages configured.
>>> I figured this out quickly since I know the test framework (simply
>>> added a echo 0; exit at the top of has-hugepages.sh).
>>> But I am not sure a reader of this doc would be able to troubleshoot this.
>>> 
>>> Not sure if this is worth explaining here, or if we can enhance the
>>> hugepage check (permissions maybe?).
>
> I prefer to fix the hugepage check to make the tests SKIP when we don't
> have hugepages accessible (so we need some kind of permission check in
> there).  I will submit it separately.
>

Here is my PoC for this - if you think it's good enough, I'll submit as
formal PATCH.

---
index d600fad319..1c3cfb665a 100755
--- a/app/test/has-hugepage.sh
+++ b/app/test/has-hugepage.sh
@@ -3,7 +3,17 @@
 # Copyright 2020 Mellanox Technologies, Ltd
 
 if [ "$(uname)" = "Linux" ] ; then
-       cat /proc/sys/vm/nr_hugepages || echo 0
+       nr_hugepages=$(cat /proc/sys/vm/nr_hugepages)
+       # Need to check if we have permissions to access hugepages
+       perm=""
+       for mount in `mount | grep hugetlbfs | awk '{ print $3; }'`; do
+               test ! -w $mount/. || perm="$mount"
+       done
+       if [ "$perm" = "" -o "$nr_hugepages" = "0" ]; then
+               echo 0
+       else
+               echo $nr_hugepages
+       fi
 elif [ "$(uname)" = "FreeBSD" ] ; then
        echo 1 # assume FreeBSD always has hugepages
 else
---
David Marchand March 11, 2021, 9:25 p.m. UTC | #6
On Tue, Mar 9, 2021 at 5:14 PM Aaron Conole <aconole@redhat.com> wrote:
> >>> > +The suites can be selected by adding the ``--suite`` option to the
> >>> > +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
> >>> > +
> >>> > +  $ meson test -C build --suite fast-tests
> >>> > +  ninja: Entering directory `/home/aconole/git/dpdk/build'
> >>> > +  [2543/2543] Linking target app/test/dpdk-test.
> >>> > +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
> >>> > +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
> >>> > +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
> >>> > +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
> >>> > +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
> >>> > +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
> >>> > +  ...
> >>>
> >>> Trying this in my build env, I get all tests failing.
> >>> This is because I run this as a normal user, but the system has
> >>> hugepages configured.
> >>> I figured this out quickly since I know the test framework (simply
> >>> added a echo 0; exit at the top of has-hugepages.sh).
> >>> But I am not sure a reader of this doc would be able to troubleshoot this.
> >>>
> >>> Not sure if this is worth explaining here, or if we can enhance the
> >>> hugepage check (permissions maybe?).
> >
> > I prefer to fix the hugepage check to make the tests SKIP when we don't
> > have hugepages accessible (so we need some kind of permission check in
> > there).  I will submit it separately.
> >
>
> Here is my PoC for this - if you think it's good enough, I'll submit as
> formal PATCH.
>
> ---
> index d600fad319..1c3cfb665a 100755
> --- a/app/test/has-hugepage.sh
> +++ b/app/test/has-hugepage.sh
> @@ -3,7 +3,17 @@
>  # Copyright 2020 Mellanox Technologies, Ltd
>
>  if [ "$(uname)" = "Linux" ] ; then
> -       cat /proc/sys/vm/nr_hugepages || echo 0
> +       nr_hugepages=$(cat /proc/sys/vm/nr_hugepages)
> +       # Need to check if we have permissions to access hugepages
> +       perm=""
> +       for mount in `mount | grep hugetlbfs | awk '{ print $3; }'`; do
> +               test ! -w $mount/. || perm="$mount"
> +       done
> +       if [ "$perm" = "" -o "$nr_hugepages" = "0" ]; then
> +               echo 0
> +       else
> +               echo $nr_hugepages
> +       fi
>  elif [ "$(uname)" = "FreeBSD" ] ; then
>         echo 1 # assume FreeBSD always has hugepages
>  else
> ---
>

I need to think more about the multiple mountpoints case (but I may be
imagining too much twisted setups..).

At least, this works in my env.
We need tests in travis/GHA, and sending a non-RFC patch is the best
way to know :-)

So +1 for a patch.
Thanks Aaron!
Aaron Conole March 17, 2021, 2:44 p.m. UTC | #7
David Marchand <david.marchand@redhat.com> writes:

> On Tue, Mar 9, 2021 at 5:14 PM Aaron Conole <aconole@redhat.com> wrote:
>> >>> > +The suites can be selected by adding the ``--suite`` option to the
>> >>> > +``meson test`` command.  Ex: ``meson test --suite fast-tests``::
>> >>> > +
>> >>> > +  $ meson test -C build --suite fast-tests
>> >>> > +  ninja: Entering directory `/home/aconole/git/dpdk/build'
>> >>> > +  [2543/2543] Linking target app/test/dpdk-test.
>> >>> > +  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
>> >>> > +  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
>> >>> > +  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
>> >>> > +  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
>> >>> > +  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
>> >>> > +  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
>> >>> > +  ...
>> >>>
>> >>> Trying this in my build env, I get all tests failing.
>> >>> This is because I run this as a normal user, but the system has
>> >>> hugepages configured.
>> >>> I figured this out quickly since I know the test framework (simply
>> >>> added a echo 0; exit at the top of has-hugepages.sh).
>> >>> But I am not sure a reader of this doc would be able to troubleshoot this.
>> >>>
>> >>> Not sure if this is worth explaining here, or if we can enhance the
>> >>> hugepage check (permissions maybe?).
>> >
>> > I prefer to fix the hugepage check to make the tests SKIP when we don't
>> > have hugepages accessible (so we need some kind of permission check in
>> > there).  I will submit it separately.
>> >
>>
>> Here is my PoC for this - if you think it's good enough, I'll submit as
>> formal PATCH.
>>
>> ---
>> index d600fad319..1c3cfb665a 100755
>> --- a/app/test/has-hugepage.sh
>> +++ b/app/test/has-hugepage.sh
>> @@ -3,7 +3,17 @@
>>  # Copyright 2020 Mellanox Technologies, Ltd
>>
>>  if [ "$(uname)" = "Linux" ] ; then
>> -       cat /proc/sys/vm/nr_hugepages || echo 0
>> +       nr_hugepages=$(cat /proc/sys/vm/nr_hugepages)
>> +       # Need to check if we have permissions to access hugepages
>> +       perm=""
>> +       for mount in `mount | grep hugetlbfs | awk '{ print $3; }'`; do
>> +               test ! -w $mount/. || perm="$mount"
>> +       done
>> +       if [ "$perm" = "" -o "$nr_hugepages" = "0" ]; then
>> +               echo 0
>> +       else
>> +               echo $nr_hugepages
>> +       fi
>>  elif [ "$(uname)" = "FreeBSD" ] ; then
>>         echo 1 # assume FreeBSD always has hugepages
>>  else
>> ---
>>
>
> I need to think more about the multiple mountpoints case (but I may be
> imagining too much twisted setups..).
>
> At least, this works in my env.
> We need tests in travis/GHA, and sending a non-RFC patch is the best
> way to know :-)

Sent :)

> So +1 for a patch.
> Thanks Aaron!
diff mbox series

Patch

diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst
index 2fefd91931..41909d949b 100644
--- a/doc/guides/contributing/index.rst
+++ b/doc/guides/contributing/index.rst
@@ -14,6 +14,7 @@  Contributor's Guidelines
     abi_versioning
     documentation
     patches
+    testing
     vulnerability
     stable
     cheatsheet
diff --git a/doc/guides/contributing/testing.rst b/doc/guides/contributing/testing.rst
new file mode 100644
index 0000000000..86ca24ce43
--- /dev/null
+++ b/doc/guides/contributing/testing.rst
@@ -0,0 +1,245 @@ 
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright 2018 The DPDK contributors
+
+.. _testing_guidelines:
+
+DPDK Testing Guidelines
+=======================
+
+This document outlines the guidelines for running and adding new
+tests to the in-tree DPDK test suites.
+
+The DPDK test suite model is loosely based on the xunit model, where
+tests are grouped into test suites, and suites are run by runners.
+For a basic overview, see the basic Wikipedia article on xunit:
+`xUnit - Wikipedia <https://en.wikipedia.org/wiki/XUnit>`_.
+
+
+Running a test
+--------------
+
+DPDK tests are run via the main test runner, the `dpdk-test` app.
+The `dpdk-test` app is a command-line interface that facilitates
+running various tests or test suites.
+
+There are two modes of operation.  The first mode is as an interactive
+command shell that allows launching specific test suites.  This is
+the default operating mode of `dpdk-test` and can be done by::
+
+  $ ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>
+
+At the prompt, simply type the name of the test suite you wish to run
+and it will execute.
+
+The second form is useful for a scripting environment, and is used by
+the DPDK meson build system.  This mode is invoked by assigning a
+specific test suite name to the environment variable `DPDK_TEST`
+before invoking the `dpdk-test` command, such as::
+
+  $ DPDK_TEST=version_autotest ./build/app/test/dpdk-test --dpdk-options-here
+  EAL: Detected 4 lcore(s)
+  EAL: Detected 1 NUMA nodes
+  EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
+  EAL: Multi-process socket /run/user/26934/dpdk/rte/mp_socket
+  EAL: Selected IOVA mode 'VA'
+  EAL: Probing VFIO support...
+  EAL: PCI device 0000:00:1f.6 on NUMA socket -1
+  EAL:   Invalid NUMA socket, default to 0
+  EAL:   probe driver: 8086:15d7 net_e1000_em
+  APP: HPET is not enabled, using TSC as default timer
+  RTE>>version_autotest
+  Version string: 'DPDK 20.02.0-rc0'
+  Test OK
+  RTE>>$
+
+The above shows running a specific test case.  On success, the return
+code will be '0', otherwise it will be set to some error value (such
+as '255').
+
+
+Running all tests
+-----------------
+
+In order to allow developers to quickly execute all the standard
+internal tests without needing to remember or look up each test suite
+name, the build system includes a standard way of executing the
+default test suites.  After building via `ninja`, the ``meson test``
+command will execute the standard tests and report errors.
+
+There are four groups of default test suites.  The first group is
+the **fast** test suite, which is the largest group of test cases.
+These are the bulk of the unit tests to validate functional blocks.
+The second group is the **perf** tests.  These test suites can take
+longer to run and do performance evaluations.  The third group is
+the **driver** test suite, which is mostly for special hardware
+related testing (such as `cryptodev`).  The last group are the
+**debug** tests.  These mostly are used to dump system information.
+
+The suites can be selected by adding the ``--suite`` option to the
+``meson test`` command.  Ex: ``meson test --suite fast-tests``::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build'
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  4/60 DPDK:fast-tests / cmdline_autotest      OK       0.28 s
+  5/60 DPDK:fast-tests / common_autotest       OK       0.57 s
+  6/60 DPDK:fast-tests / cpuflags_autotest     OK       0.27 s
+  ...
+
+
+Adding test suites
+------------------
+
+To add a test suite to the DPDK test application, create a new test
+file for that suite (ex: see *app/test/test_version.c* for the
+``version_autotest`` test suite).  There are two important functions
+for interacting with the test harness:
+
+  1. REGISTER_TEST_COMMAND(command_name, function_to_execute)
+     Registers a test command with the name `command_name` and which
+     runs the function `function_to_execute` when `command_name` is
+     invoked.
+
+  2. unit_test_suite_runner(struct unit_test_suite \*)
+     Returns a runner for a full test suite object, which contains
+     a test suite name, setup, tear down, and vector of unit test
+     cases.
+
+Each test suite has a setup and tear down function that runs at the
+beginning and end of the test suite execution.  Each unit test has
+a similar function for test case setup and tear down.
+
+Test cases are added to the `.unit_test_cases` element of the unit
+test suite structure.  Ex:
+
+.. code-block:: c
+   :linenos:
+
+   #include <time.h>
+
+   #include <rte_common.h>
+   #include <rte_cycles.h>
+   #include <rte_hexdump.h>
+   #include <rte_random.h>
+
+   #include "test.h"
+
+   static int testsuite_setup(void) { return TEST_SUCCESS; }
+   static void testsuite_teardown(void) { }
+
+   static int ut_setup(void) { return TEST_SUCCESS; }
+   static void ut_teardown(void) { }
+
+   static int test_case_first(void) { return TEST_SUCCESS; }
+
+   static struct unit_test_suite example_testsuite = {
+          .suite_name = "EXAMPLE TEST SUITE",
+          .setup = testsuite_setup,
+          .teardown = testsuite_teardown,
+          .unit_test_cases = {
+               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+               TEST_CASES_END(), /**< NULL terminate unit test array */
+          },
+   };
+
+   static int example_tests()
+   {
+       return unit_test_suite_runner(&example_testsuite);
+   }
+
+   REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
+The above code block is a small example that can be used to create a
+complete test suite with test case.
+
+
+Designing a test
+----------------
+
+Test cases have multiple ways of indicating an error has occurred,
+in order to reflect failure state back to the runner.  Using the
+various methods of indicating errors can assist in not only validating
+the requisite functionality is working, but also to help debug when
+a change in environment or code has caused things to go wrong.
+
+The first way to indicate a generic error is by returning a test
+result failure, using the *TEST_FAILED* error code.  This is the most
+basic way of indicating that an error has occurred in a test routine.
+It isn't very informative to the user, so it should really be used in
+cases where the test has catastrophically failed.
+
+The preferred method of indicating an error is via the
+`RTE_TEST_ASSERT` family of macros, which will immediately return
+*TEST_FAILED* error condition, but will also log details about the
+failure.  The basic form is:
+
+.. code-block:: c
+
+   RTE_TEST_ASSERT(cond, msg, ...)
+
+In the above macro, *cond* is the condition to evaluate to **true**.
+Any generic condition can go here.  The *msg* parameter will be a
+message to display if *cond* evaluates to **false**.  Some specialized
+macros already exist.  See `lib/librte_eal/include/rte_test.h` for
+a list of defined test assertions.
+
+Sometimes it is important to indicate that a test needs to be
+skipped, either because the environment isn't able to support running
+the test, or because some requisite functionality isn't available.  The
+test suite supports returning a result of `TEST_SKIPPED` during test
+case setup, or during test case execution to indicate that the
+preconditions of the test aren't available.  Ex::
+
+  $ meson test -C build --suite fast-tests
+  ninja: Entering directory `/home/aconole/git/dpdk/build
+  [2543/2543] Linking target app/test/dpdk-test.
+  1/60 DPDK:fast-tests / acl_autotest          OK       3.17 s
+  2/60 DPDK:fast-tests / bitops_autotest       OK       0.22 s
+  3/60 DPDK:fast-tests / byteorder_autotest    OK       0.22 s
+  ...
+  46/60 DPDK:fast-tests / ipsec_autotest       SKIP     0.22 s
+  ...
+
+
+Checking code coverage
+----------------------
+The meson build system supports generating a code coverage report
+via the `-Db_coverage=true` option, in conjunction with a package
+like **lcov**, to generate an HTML code coverage report.  Example::
+
+  $ meson covered -Db_coverage=true
+  $ meson test -C covered --suite fast-tests
+  $ ninja coverage-html -C covered
+
+The above will generate an html report in the
+`covered/meson-logs/coveragereport/` directory that can be explored
+for detailed code covered information.  This can be used to assist
+in test development.
+
+
+Adding a suite to the default
+-----------------------------
+
+Adding to one of the default tests involves editing the appropriate
+meson build file `app/test/meson.build` and adding the command to
+the correct test suite class.  Once added, the new test suite will
+be run as part of the appropriate class (fast, perf, driver, etc.).
+
+Some of these default test suites are run during continuous integration
+tests, making regression checking automatic for new patches submitted
+to the project.