[v2] test: rely on EAL detection for core list

  Cores count has a direct impact on the time needed to complete unit
tests.

Currently, the core list used for unit test is enforced to "all cores on
the system" with no way for (CI) users to adapt it.
On the other hand, EAL default behavior (when no -c/-l option gets passed)
is to start threads on as many cores available in the process cpu
affinity.

Remove logic from meson: users can then select where to run the tests by
either running meson with a custom cpu affinity (using taskset/cpuset
depending on OS) or by passing a --test-args option to meson.

Example:
$ sudo meson test -C build --suite fast-tests -t 3 --test-args "-l 0-3"

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changes since v1:
- updated documentation,

---
 app/test/get-coremask.sh           | 13 -------------
 app/test/meson.build               | 10 +---------
 doc/guides/prog_guide/meson_ut.rst | 17 ++++++++++++++++-
 3 files changed, 17 insertions(+), 23 deletions(-)
 delete mode 100755 app/test/get-coremask.sh
  

On Tue, Oct 19, 2021 at 01:26:02PM +0200, David Marchand wrote:
> Cores count has a direct impact on the time needed to complete unit
> tests.
> 
> Currently, the core list used for unit test is enforced to "all cores on
> the system" with no way for (CI) users to adapt it.
> On the other hand, EAL default behavior (when no -c/-l option gets passed)
> is to start threads on as many cores available in the process cpu
> affinity.
> 
> Remove logic from meson: users can then select where to run the tests by
> either running meson with a custom cpu affinity (using taskset/cpuset
> depending on OS) or by passing a --test-args option to meson.
> 
> Example:
> $ sudo meson test -C build --suite fast-tests -t 3 --test-args "-l 0-3"
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
Tested using ring_perf_autotest. By default this ran perf tests on multiple
threads, cores and numa nodes. Passing in `--test-args="-l 0-3"` the tests
only ran on the multiple cores option as no threads on multiple numa nodes,
or no threads sharing a core were present.

Tested-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
  

<snip>

> 
> Cores count has a direct impact on the time needed to complete unit tests.
We also need to control the number of iterations with in the tests.

We could add something like "-i low/medium/high". The test cases then can use this to decide on how many iterations to run.

> 
> Currently, the core list used for unit test is enforced to "all cores on the
> system" with no way for (CI) users to adapt it.
> On the other hand, EAL default behavior (when no -c/-l option gets passed) is
> to start threads on as many cores available in the process cpu affinity.
> 
> Remove logic from meson: users can then select where to run the tests by
> either running meson with a custom cpu affinity (using taskset/cpuset
> depending on OS) or by passing a --test-args option to meson.
> 
> Example:
> $ sudo meson test -C build --suite fast-tests -t 3 --test-args "-l 0-3"
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> Changes since v1:
> - updated documentation,
> 
> ---
>  app/test/get-coremask.sh           | 13 -------------
>  app/test/meson.build               | 10 +---------
>  doc/guides/prog_guide/meson_ut.rst | 17 ++++++++++++++++-
>  3 files changed, 17 insertions(+), 23 deletions(-)  delete mode 100755
> app/test/get-coremask.sh
> 
> diff --git a/app/test/get-coremask.sh b/app/test/get-coremask.sh deleted file
> mode 100755 index bb8cf404d2..0000000000
> --- a/app/test/get-coremask.sh
> +++ /dev/null
> @@ -1,13 +0,0 @@
> -#! /bin/sh -e
> -# SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2019 Intel
> Corporation
> -
> -if [ "$(uname)" = "Linux" ] ; then
> -	cat /sys/devices/system/cpu/present
> -elif [ "$(uname)" = "FreeBSD" ] ; then
> -	ncpus=$(/sbin/sysctl -n hw.ncpu)
> -	echo 0-$(expr $ncpus - 1)
> -else
> -# fallback
> -	echo 0-3
> -fi
> diff --git a/app/test/meson.build b/app/test/meson.build index
> a16374b7a1..c7b377d52d 100644
> --- a/app/test/meson.build
> +++ b/app/test/meson.build
> @@ -463,13 +463,8 @@ message('hugepage availability:
> @0@'.format(has_hugepage))  timeout_seconds = 600  timeout_seconds_fast
> = 10
> 
> -get_coremask = find_program('get-coremask.sh') -num_cores_arg = '-l ' +
> run_command(get_coremask).stdout().strip()
> -
> -default_test_args = [num_cores_arg]
> -
>  foreach arg : fast_tests
> -    test_args = default_test_args
> +    test_args = []
>      run_test = true
>      if not has_hugepage
>          if arg[1]
> @@ -502,7 +497,6 @@ endforeach
>  foreach arg : perf_test_names
>      test(arg, dpdk_test,
>              env : ['DPDK_TEST=' + arg],
> -            args : default_test_args,
>              timeout : timeout_seconds,
>              is_parallel : false,
>              suite : 'perf-tests')
> @@ -511,7 +505,6 @@ endforeach
>  foreach arg : driver_test_names
>      test(arg, dpdk_test,
>              env : ['DPDK_TEST=' + arg],
> -            args : default_test_args,
>              timeout : timeout_seconds,
>              is_parallel : false,
>              suite : 'driver-tests')
> @@ -520,7 +513,6 @@ endforeach
>  foreach arg : dump_test_names
>      test(arg, dpdk_test,
>              env : ['DPDK_TEST=' + arg],
> -            args : default_test_args,
>              timeout : timeout_seconds,
>              is_parallel : false,
>              suite : 'debug-tests')
> diff --git a/doc/guides/prog_guide/meson_ut.rst
> b/doc/guides/prog_guide/meson_ut.rst
> index fff88655dd..78cf3f845c 100644
> --- a/doc/guides/prog_guide/meson_ut.rst
> +++ b/doc/guides/prog_guide/meson_ut.rst
> @@ -37,6 +37,14 @@ command::
> 
>      $ meson test --suite fast-tests
> 
> +If desired, additional arguments can be passed to the test run via the
> +meson ``--test-args`` option.
> +For example, tests will by default run on as many available cores as is
> +needed for the test, starting with the lowest number core - generally core 0.
> +To run the fast-tests suite using only cores 8 through 16, one can use::
> +
> +    $ meson test --suite fast-tests --test-args="-l 8-16"
> +
>  The meson command to list all available tests::
> 
>      $ meson test --list
> @@ -47,9 +55,16 @@ Arguments of ``test()`` that can be provided in
> meson.build are as below:
> 
>  * ``is_parallel`` is used to run test case either in parallel or non-parallel mode.
>  * ``timeout`` is used to specify the timeout of test case.
> -* ``args`` is used to specify test specific parameters.
> +* ``args`` is used to specify test specific parameters (see note below).
>  * ``env`` is used to specify test specific environment parameters.
> 
> +Note: the content of meson ``--test-args`` option and the content of
> +``args`` are appended when invoking the DPDK test binary.
> +Because of this, it is recommended not to set any default coremask or
> +memory configuration in per test ``args`` and rather let users select
> +what best fits their environment. If a test can't run, then it should
> +be skipped, as described below.
> +
> 
>  Dealing with skipped test cases
>  -------------------------------
> --
> 2.23.0
  

On Tue, Oct 19, 2021 at 4:46 PM Honnappa Nagarahalli
<Honnappa.Nagarahalli@arm.com> wrote:
>
> <snip>
>
> >
> > Cores count has a direct impact on the time needed to complete unit tests.
> We also need to control the number of iterations with in the tests.
>
> We could add something like "-i low/medium/high". The test cases then can use this to decide on how many iterations to run.

This patch hands control of parameters that affect all tests to the CI.

But number of iterations is a per test thing.
What would be the definition of an "iteration"?
Something that must be done in a maximum of cycles.. ?

This could be something to look at, but it goes beyond this patch.
  

<snip>

> >
> > >
> > > Cores count has a direct impact on the time needed to complete unit tests.
> > We also need to control the number of iterations with in the tests.
> >
> > We could add something like "-i low/medium/high". The test cases then can
> use this to decide on how many iterations to run.
> 
> This patch hands control of parameters that affect all tests to the CI.
> 
> But number of iterations is a per test thing.
Agree, it is a per test thing. But, multiple test files run the test case in several iterations. So, multiple test cases can make use of the same input from the user.

> What would be the definition of an "iteration"?
> Something that must be done in a maximum of cycles.. ?
You can look at test_atomic.c, we have the following #define

#define N 1000000

It is used as follows (in the simplest case) to repeat a test:

for (i = 0; i < N; i++)
                rte_atomic16_inc(&a16);

The value 1000000 defines how long this test will take.

In a CI environment, we want to reduce the time taken, whereas in a local lab machine, I might not care about the time it takes to run the test. This could be an input from the user which the test cases can make use of. For ex: for 'low', test case might set N = 10000 and so on.

> 
> This could be something to look at, but it goes beyond this patch.
Agreed, this can be a separate patch.

> 
> 
> --
> David Marchand
  

David Marchand <david.marchand@redhat.com> writes:

> Cores count has a direct impact on the time needed to complete unit
> tests.
>
> Currently, the core list used for unit test is enforced to "all cores on
> the system" with no way for (CI) users to adapt it.
> On the other hand, EAL default behavior (when no -c/-l option gets passed)
> is to start threads on as many cores available in the process cpu
> affinity.
>
> Remove logic from meson: users can then select where to run the tests by
> either running meson with a custom cpu affinity (using taskset/cpuset
> depending on OS) or by passing a --test-args option to meson.
>
> Example:
> $ sudo meson test -C build --suite fast-tests -t 3 --test-args "-l 0-3"
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---

LGTM - the spelling issue flagged seems to be a false positive.

Acked-by: Aaron Conole <aconole@redhat.com>
  

On Tue, Oct 19, 2021 at 9:09 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > Cores count has a direct impact on the time needed to complete unit
> > tests.
> >
> > Currently, the core list used for unit test is enforced to "all cores on
> > the system" with no way for (CI) users to adapt it.
> > On the other hand, EAL default behavior (when no -c/-l option gets passed)
> > is to start threads on as many cores available in the process cpu
> > affinity.
> >
> > Remove logic from meson: users can then select where to run the tests by
> > either running meson with a custom cpu affinity (using taskset/cpuset
> > depending on OS) or by passing a --test-args option to meson.
> >
> > Example:
> > $ sudo meson test -C build --suite fast-tests -t 3 --test-args "-l 0-3"
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
>
> LGTM - the spelling issue flagged seems to be a false positive.

The first spelling issue is fixed in main:
https://git.dpdk.org/dpdk/commit/?id=d0db11a82609

The other three warnings (all of them about "test-args") are
introduced by this patch, and this is a problem: once merged, I
understand any following patch will get flagged as failing this check.
There may be something better to do for the mid/long term, but the
quicker is to add test-args to UNH dictionnary.
Once done, I can merge this patch.

Lincoln, Brandon?



>
> Acked-by: Aaron Conole <aconole@redhat.com>
>

Thanks for the review.
  

On Tue, Oct 19, 2021 at 9:28 PM David Marchand
<david.marchand@redhat.com> wrote:
> The other three warnings (all of them about "test-args") are
> introduced by this patch, and this is a problem: once merged, I
> understand any following patch will get flagged as failing this check.
> There may be something better to do for the mid/long term, but the
> quicker is to add test-args to UNH dictionnary.
> Once done, I can merge this patch.

Nevermind, we have new warnings raised by this check.
I must have overlooked some test report seeing how it comes from a
patch I merged.

> Lincoln, Brandon?

This check seems picky, and rc1 is closer than ever.
I'll ignore it for now.


--
David Marchand
  

On Tue, Oct 19, 2021 at 9:09 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > Cores count has a direct impact on the time needed to complete unit
> > tests.
> >
> > Currently, the core list used for unit test is enforced to "all cores on
> > the system" with no way for (CI) users to adapt it.
> > On the other hand, EAL default behavior (when no -c/-l option gets passed)
> > is to start threads on as many cores available in the process cpu
> > affinity.
> >
> > Remove logic from meson: users can then select where to run the tests by
> > either running meson with a custom cpu affinity (using taskset/cpuset
> > depending on OS) or by passing a --test-args option to meson.
> >
> > Example:
> > $ sudo meson test -C build --suite fast-tests -t 3 --test-args "-l 0-3"
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> Tested-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Aaron Conole <aconole@redhat.com>

Applied, thanks.