[dpdk-dev,0/7] Make unit tests great again

  Previously, unit tests were running in groups. There were
technical reasons why that was the case (mostly having to do
with limiting memory), but it was hard to maintain and update
the autotest script.

In 18.05, limiting of memory at DPDK startup was no longer
necessary, as DPDK allocates memory at runtime as needed. This
has the implication that the old test grouping can now be
retired and replaced with a more sensible way of running unit
tests (using multiprocessing pool of workers and a queue of
tests). This patchset accomplishes exactly that.

This patchset conflicts with some of the earlier work on
autotests [1] [2] [3], but i think it presents a cleaner
solution for some of the problems highlighted by those patch
series. I can integrate those patches into this series if
need be.

[1] http://dpdk.org/dev/patchwork/patch/40370/
[2] http://dpdk.org/dev/patchwork/patch/40371/
[3] http://dpdk.org/dev/patchwork/patch/40372/

Anatoly Burakov (7):
  autotest: fix printing
  autotest: fix invalid code on reports
  autotest: make autotest runner python 2/3 compliant
  autotest: visually separate different test categories
  autotest: improve filtering
  autotest: remove autotest grouping
  autotest: properly parallelize unit tests

 test/test/autotest.py        |  13 +-
 test/test/autotest_data.py   | 749 ++++++++++++++---------------------
 test/test/autotest_runner.py | 519 ++++++++++++------------
 3 files changed, 586 insertions(+), 695 deletions(-)
  

+Cc Jananee

07/06/2018 23:01, Anatoly Burakov:
> Previously, unit tests were running in groups. There were
> technical reasons why that was the case (mostly having to do
> with limiting memory), but it was hard to maintain and update
> the autotest script.
> 
> In 18.05, limiting of memory at DPDK startup was no longer
> necessary, as DPDK allocates memory at runtime as needed. This
> has the implication that the old test grouping can now be
> retired and replaced with a more sensible way of running unit
> tests (using multiprocessing pool of workers and a queue of
> tests). This patchset accomplishes exactly that.
> 
> This patchset conflicts with some of the earlier work on
> autotests [1] [2] [3], but i think it presents a cleaner
> solution for some of the problems highlighted by those patch
> series. I can integrate those patches into this series if
> need be.
> 
> [1] http://dpdk.org/dev/patchwork/patch/40370/
> [2] http://dpdk.org/dev/patchwork/patch/40371/
> [3] http://dpdk.org/dev/patchwork/patch/40372/

It may be interesting to work on lists of tests as done
in the following patch:
	http://dpdk.org/dev/patchwork/patch/40373/

The idea is to split tests in several categories:
	- basic and short test
	- longer and lower priority
	- performance test
As a long term solution, we can think about making category an attribute
inside the test itself?
  

On 12-Jun-18 2:07 PM, Thomas Monjalon wrote:
> +Cc Jananee
> 
> 07/06/2018 23:01, Anatoly Burakov:
>> Previously, unit tests were running in groups. There were
>> technical reasons why that was the case (mostly having to do
>> with limiting memory), but it was hard to maintain and update
>> the autotest script.
>>
>> In 18.05, limiting of memory at DPDK startup was no longer
>> necessary, as DPDK allocates memory at runtime as needed. This
>> has the implication that the old test grouping can now be
>> retired and replaced with a more sensible way of running unit
>> tests (using multiprocessing pool of workers and a queue of
>> tests). This patchset accomplishes exactly that.
>>
>> This patchset conflicts with some of the earlier work on
>> autotests [1] [2] [3], but i think it presents a cleaner
>> solution for some of the problems highlighted by those patch
>> series. I can integrate those patches into this series if
>> need be.
>>
>> [1] http://dpdk.org/dev/patchwork/patch/40370/
>> [2] http://dpdk.org/dev/patchwork/patch/40371/
>> [3] http://dpdk.org/dev/patchwork/patch/40372/
> 
> It may be interesting to work on lists of tests as done
> in the following patch:
> 	http://dpdk.org/dev/patchwork/patch/40373/
> 
> The idea is to split tests in several categories:
> 	- basic and short test
> 	- longer and lower priority
> 	- performance test
> As a long term solution, we can think about making category an attribute
> inside the test itself?
> 

These test categories do not conflict with my patchset as they 
ultimately rely on white/blacklisting, which will continue to work as 
before.

In my view, it really boils down to two things - either tests can be run 
in parallel with others (i.e. their result won't be affected by another 
independent DPDK test app instance), or not. On top of that, we can use 
blacklisting or whitelisting to define which tests will actually be run 
(i.e. define any "categories" we want), but their (non-)parallelism must 
always be respected to get good test results.
  

On Wed, Jun 13, 2018 at 09:38:32AM +0100, Burakov, Anatoly wrote:
> On 12-Jun-18 2:07 PM, Thomas Monjalon wrote:
> > +Cc Jananee
> > 
> > 07/06/2018 23:01, Anatoly Burakov:
> > > Previously, unit tests were running in groups. There were
> > > technical reasons why that was the case (mostly having to do
> > > with limiting memory), but it was hard to maintain and update
> > > the autotest script.
> > > 
> > > In 18.05, limiting of memory at DPDK startup was no longer
> > > necessary, as DPDK allocates memory at runtime as needed. This
> > > has the implication that the old test grouping can now be
> > > retired and replaced with a more sensible way of running unit
> > > tests (using multiprocessing pool of workers and a queue of
> > > tests). This patchset accomplishes exactly that.
> > > 
> > > This patchset conflicts with some of the earlier work on
> > > autotests [1] [2] [3], but i think it presents a cleaner
> > > solution for some of the problems highlighted by those patch
> > > series. I can integrate those patches into this series if
> > > need be.
> > > 
> > > [1] http://dpdk.org/dev/patchwork/patch/40370/
> > > [2] http://dpdk.org/dev/patchwork/patch/40371/
> > > [3] http://dpdk.org/dev/patchwork/patch/40372/
> > 
> > It may be interesting to work on lists of tests as done
> > in the following patch:
> > 	http://dpdk.org/dev/patchwork/patch/40373/
> > 
> > The idea is to split tests in several categories:
> > 	- basic and short test
> > 	- longer and lower priority
> > 	- performance test
> > As a long term solution, we can think about making category an attribute
> > inside the test itself?
> > 
> 
> These test categories do not conflict with my patchset as they ultimately
> rely on white/blacklisting, which will continue to work as before.
> 
> In my view, it really boils down to two things - either tests can be run in
> parallel with others (i.e. their result won't be affected by another
> independent DPDK test app instance), or not. On top of that, we can use
> blacklisting or whitelisting to define which tests will actually be run
> (i.e. define any "categories" we want), but their (non-)parallelism must
> always be respected to get good test results.
> 
Have you looked at: http://mesonbuild.com/Unit-tests.html, at it would be
good to transition away from our own custom script infrastructure for the
tests.

There is already some support for running the unit tests using "meson
test", but it could do with some more cleanup e.g. to move the tests into
suitable suites (corresponding to the categories Thomas has suggested). We
could also do with specifying properly what tests are parallel-safe and
what aren't.