mbox series

[00/14] Unit tests fixes for CI

Message ID 1559638792-8608-1-git-send-email-david.marchand@redhat.com (mailing list archive)
Headers show
Series Unit tests fixes for CI | expand

Message

David Marchand June 4, 2019, 8:59 a.m. UTC
This is a joint effort to make the unit tests ready for CI.
The first 8 patches are fixes that I had accumulated.
Then the second part of the series focuses on skipping tests when some
requirements are not fulfilled so that we can start them in a restrained
environment like Travis virtual machines that gives us two cores and does
not have specific hw devices.

We are still not ready for enabling those tests in Travis.
At least, the following issues remain:
- some fixes on librte_acl have not been merged yet [1],
- the tests on --file-prefix are still ko, and have been isolated in a
  test that we could disable while waiting for the fixes,
- rwlock_autotest and hash_readwrite_lf_autotest are taking a little more
  than 10s,
- librte_table unit test crashes on ipv6 [2],
- the "perf" tests are taking way too long for my taste,
- the shared build unit tests all fail when depending on mempool since
  the mempool drivers are not loaded,

1: http://patchwork.dpdk.org/project/dpdk/list/?series=4242
2: https://bugs.dpdk.org/show_bug.cgi?id=285

Comments/reviews welcome!

Comments

Michael Santana June 4, 2019, 3:49 p.m. UTC | #1
On 6/4/19 4:59 AM, David Marchand wrote:
> This is a joint effort to make the unit tests ready for CI.
> The first 8 patches are fixes that I had accumulated.
> Then the second part of the series focuses on skipping tests when some
> requirements are not fulfilled so that we can start them in a restrained
> environment like Travis virtual machines that gives us two cores and does
> not have specific hw devices.
>
> We are still not ready for enabling those tests in Travis.
> At least, the following issues remain:
> - some fixes on librte_acl have not been merged yet [1],
> - the tests on --file-prefix are still ko, and have been isolated in a
>    test that we could disable while waiting for the fixes,
> - rwlock_autotest and hash_readwrite_lf_autotest are taking a little more
>    than 10s,
It would be a good idea to increase timeout. These are probably not the 
only tests that would pass with just a little bit more time
> - librte_table unit test crashes on ipv6 [2],
> - the "perf" tests are taking way too long for my taste,

We should still fix them. However I don't know if we should be running 
the perf test for every job and every patch on travis. It takes too 
long. The travis queue will be delayed too far behind for it to be of 
any use.

OTOH we could have one job as part of the travis build dedicated to 
running tests (or just perf test). It's still time consuming but better 
than running the test on every travis job. For this to work we would 
need to decreased the timeout for the perf tests as the timeout for it 
and the travis are both 10 minutes

> - the shared build unit tests all fail when depending on mempool since
>    the mempool drivers are not loaded,
>
> 1: http://patchwork.dpdk.org/project/dpdk/list/?series=4242
> 2: https://bugs.dpdk.org/show_bug.cgi?id=285
>
> Comments/reviews welcome!
>
Thomas Monjalon June 27, 2019, 4:34 p.m. UTC | #2
04/06/2019 17:49, Michael Santana Francisco:
> On 6/4/19 4:59 AM, David Marchand wrote:
> > - the "perf" tests are taking way too long for my taste,
> 
> We should still fix them. However I don't know if we should be running 
> the perf test for every job and every patch on travis. It takes too 
> long. The travis queue will be delayed too far behind for it to be of 
> any use.
> 
> OTOH we could have one job as part of the travis build dedicated to 
> running tests (or just perf test). It's still time consuming but better 
> than running the test on every travis job. For this to work we would 
> need to decreased the timeout for the perf tests as the timeout for it 
> and the travis are both 10 minutes

+Cc ci@dpdk.org

I don't think we should run the perf tests in basic CI like Travis.
We can run perf tests if the purpose is to compare the performance
with previous releases, as some other tests in the community lab.
Aaron Conole July 1, 2019, 12:17 p.m. UTC | #3
Thomas Monjalon <thomas@monjalon.net> writes:

> 04/06/2019 17:49, Michael Santana Francisco:
>> On 6/4/19 4:59 AM, David Marchand wrote:
>> > - the "perf" tests are taking way too long for my taste,

+1 here.

>> 
>> We should still fix them. However I don't know if we should be running 
>> the perf test for every job and every patch on travis. It takes too 
>> long. The travis queue will be delayed too far behind for it to be of 
>> any use.
>> 
>> OTOH we could have one job as part of the travis build dedicated to 
>> running tests (or just perf test). It's still time consuming but better 
>> than running the test on every travis job. For this to work we would 
>> need to decreased the timeout for the perf tests as the timeout for it 
>> and the travis are both 10 minutes
>
> +Cc ci@dpdk.org
>
> I don't think we should run the perf tests in basic CI like Travis.
> We can run perf tests if the purpose is to compare the performance
> with previous releases, as some other tests in the community lab.

+1 - some of the perf tests aren't going to complete in any sort of
reasonable time.  While we could claim it's a separate problem, we
should also not enable something that will make the travis runs so much
longer.

I do like the idea of running tests in the travis build, and I think it
would make sense to have just a single job for it (or maybe one for
clang and one for gcc?  maybe even that is overkill).

I would rather not do performance tests during the travis run, though.
It doesn't really make sense.  Travis isn't any kind of an 'optimized'
environment, so I don't know what 'performance' should mean.