ci: remove aarch64 from Travis jobs

  Travis is not reliable for native Arm and PPC:
https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6

In order to get reliable Travis reports,
the use of Arm machines is removed until Travis fixes it.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
 .travis.yml | 30 ------------------------------
 1 file changed, 30 deletions(-)
  

Thomas Monjalon <thomas@monjalon.net> writes:

> Travis is not reliable for native Arm and PPC:
> https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
>
> In order to get reliable Travis reports,
> the use of Arm machines is removed until Travis fixes it.
>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---

We should add back the cross-build if we do this - at least then we
could have a reliable compilation test of Arm64 code.   Does it make
sense?
  

On Thu, Apr 16, 2020 at 6:14 PM Aaron Conole <aconole@redhat.com> wrote:
>
> Thomas Monjalon <thomas@monjalon.net> writes:
>
> > Travis is not reliable for native Arm and PPC:
> > https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
> >
> > In order to get reliable Travis reports,
> > the use of Arm machines is removed until Travis fixes it.
> >
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
>
> We should add back the cross-build if we do this - at least then we
> could have a reliable compilation test of Arm64 code.   Does it make
> sense?

+1

>
  

On Thu, Apr 16, 2020 at 2:44 PM Aaron Conole <aconole@redhat.com> wrote:
>
> Thomas Monjalon <thomas@monjalon.net> writes:
>
> > Travis is not reliable for native Arm and PPC:
> > https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
> >
> > In order to get reliable Travis reports,
> > the use of Arm machines is removed until Travis fixes it.
> >
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
>
> We should add back the cross-build if we do this - at least then we
> could have a reliable compilation test of Arm64 code.   Does it make
> sense?

I don't see them removed by this patch, the two jobs are still present ?

  # x86_64 cross-compiling aarch64 jobs
  - env: DEF_LIB="static" AARCH64=1
    arch: amd64
    compiler: gcc
    addons:
      apt:
        packages:
          - *aarch64_packages
  - env: DEF_LIB="shared" AARCH64=1
    arch: amd64
    compiler: gcc
    addons:
      apt:
        packages:
          - *aarch64_packages
  

David Marchand <david.marchand@redhat.com> writes:

> On Thu, Apr 16, 2020 at 2:44 PM Aaron Conole <aconole@redhat.com> wrote:
>>
>> Thomas Monjalon <thomas@monjalon.net> writes:
>>
>> > Travis is not reliable for native Arm and PPC:
>> > https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
>> >
>> > In order to get reliable Travis reports,
>> > the use of Arm machines is removed until Travis fixes it.
>> >
>> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
>> > ---
>>
>> We should add back the cross-build if we do this - at least then we
>> could have a reliable compilation test of Arm64 code.   Does it make
>> sense?
>
> I don't see them removed by this patch, the two jobs are still present ?

Whoops - for some reason I missed them.  Nevermind :)

>   # x86_64 cross-compiling aarch64 jobs
>   - env: DEF_LIB="static" AARCH64=1
>     arch: amd64
>     compiler: gcc
>     addons:
>       apt:
>         packages:
>           - *aarch64_packages
>   - env: DEF_LIB="shared" AARCH64=1
>     arch: amd64
>     compiler: gcc
>     addons:
>       apt:
>         packages:
>           - *aarch64_packages
  

16/04/2020 15:45, Aaron Conole:
> David Marchand <david.marchand@redhat.com> writes:
> > On Thu, Apr 16, 2020 at 2:44 PM Aaron Conole <aconole@redhat.com> wrote:
> >> Thomas Monjalon <thomas@monjalon.net> writes:
> >>
> >> > Travis is not reliable for native Arm and PPC:
> >> > https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
> >> >
> >> > In order to get reliable Travis reports,
> >> > the use of Arm machines is removed until Travis fixes it.
> >> >
> >> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> >> > ---
> >>
> >> We should add back the cross-build if we do this - at least then we
> >> could have a reliable compilation test of Arm64 code.   Does it make
> >> sense?
> >
> > I don't see them removed by this patch, the two jobs are still present ?
> 
> Whoops - for some reason I missed them.  Nevermind :)

So? Acked?
  

<snip>

> Subject: Re: [PATCH] ci: remove aarch64 from Travis jobs
> 
> 16/04/2020 15:45, Aaron Conole:
> > David Marchand <david.marchand@redhat.com> writes:
> > > On Thu, Apr 16, 2020 at 2:44 PM Aaron Conole <aconole@redhat.com>
> wrote:
> > >> Thomas Monjalon <thomas@monjalon.net> writes:
> > >>
> > >> > Travis is not reliable for native Arm and PPC:
> > >> > https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
Thanks David for creating the ticket. Will escalate this through our contacts at Travis CI, hopefully it can be resolved soon.

> > >> >
> > >> > In order to get reliable Travis reports, the use of Arm machines
> > >> > is removed until Travis fixes it.
> > >> >
> > >> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > >> > ---
> > >>
> > >> We should add back the cross-build if we do this - at least then we
> > >> could have a reliable compilation test of Arm64 code.   Does it make
> > >> sense?
> > >
> > > I don't see them removed by this patch, the two jobs are still present ?
> >
> > Whoops - for some reason I missed them.  Nevermind :)
> 
> So? Acked?
> 
> 
>
  

Thomas Monjalon <thomas@monjalon.net> writes:

> 16/04/2020 15:45, Aaron Conole:
>> David Marchand <david.marchand@redhat.com> writes:
>> > On Thu, Apr 16, 2020 at 2:44 PM Aaron Conole <aconole@redhat.com> wrote:
>> >> Thomas Monjalon <thomas@monjalon.net> writes:
>> >>
>> >> > Travis is not reliable for native Arm and PPC:
>> >> > https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
>> >> >
>> >> > In order to get reliable Travis reports,
>> >> > the use of Arm machines is removed until Travis fixes it.
>> >> >
>> >> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
>> >> > ---
>> >>
>> >> We should add back the cross-build if we do this - at least then we
>> >> could have a reliable compilation test of Arm64 code.   Does it make
>> >> sense?
>> >
>> > I don't see them removed by this patch, the two jobs are still present ?
>> 
>> Whoops - for some reason I missed them.  Nevermind :)
>
> So? Acked?

Yes,

Acked-by: Aaron Conole <aconole@redhat.com>
  

Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com> writes:

> <snip>
>
>> Subject: Re: [PATCH] ci: remove aarch64 from Travis jobs
>> 
>> 16/04/2020 15:45, Aaron Conole:
>> > David Marchand <david.marchand@redhat.com> writes:
>> > > On Thu, Apr 16, 2020 at 2:44 PM Aaron Conole <aconole@redhat.com>
>> wrote:
>> > >> Thomas Monjalon <thomas@monjalon.net> writes:
>> > >>
>> > >> > Travis is not reliable for native Arm and PPC:
>> > >> > https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
> Thanks David for creating the ticket. Will escalate this through our
> contacts at Travis CI, hopefully it can be resolved soon.

I did get an email from someone at travis support acknowledging the
issue and saying that they are working on it.

>> > >> >
>> > >> > In order to get reliable Travis reports, the use of Arm machines
>> > >> > is removed until Travis fixes it.
>> > >> >
>> > >> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
>> > >> > ---
>> > >>
>> > >> We should add back the cross-build if we do this - at least then we
>> > >> could have a reliable compilation test of Arm64 code.   Does it make
>> > >> sense?
>> > >
>> > > I don't see them removed by this patch, the two jobs are still present ?
>> >
>> > Whoops - for some reason I missed them.  Nevermind :)
>> 
>> So? Acked?
>> 
>> 
>>
  

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, April 16, 2020 10:40 PM
> To: Aaron Conole <aconole@redhat.com>
> Cc: David Marchand <david.marchand@redhat.com>; dev <dev@dpdk.org>;
> Ruifeng Wang <Ruifeng.Wang@arm.com>; Gavin Hu <Gavin.Hu@arm.com>;
> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Michael Santana
> <maicolgabriel@hotmail.com>
> Subject: Re: [PATCH] ci: remove aarch64 from Travis jobs
> 
> 16/04/2020 15:45, Aaron Conole:
> > David Marchand <david.marchand@redhat.com> writes:
> > > On Thu, Apr 16, 2020 at 2:44 PM Aaron Conole <aconole@redhat.com>
> wrote:
> > >> Thomas Monjalon <thomas@monjalon.net> writes:
> > >>
> > >> > Travis is not reliable for native Arm and PPC:
> > >> > https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
> > >> >
> > >> > In order to get reliable Travis reports, the use of Arm machines
> > >> > is removed until Travis fixes it.
> > >> >
> > >> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > >> > ---
> > >>
> > >> We should add back the cross-build if we do this - at least then we
> > >> could have a reliable compilation test of Arm64 code.   Does it make
> > >> sense?
> > >
> > > I don't see them removed by this patch, the two jobs are still present ?
> >
> > Whoops - for some reason I missed them.  Nevermind :)
> 
> So? Acked?
> 
Can we achieve this by allowing failures on AArch64 jobs?
https://docs.travis-ci.com/user/build-matrix/#rows-that-are-allowed-to-fail

Add following setting:
jobs:
  allow_failures:
  - arch: arm64

So we can keep the jobs while not suffering from unstable infrastructure.
Results of these jobs will still observable. This gives us a chance to know when jobs are stable.

Thanks.
/Ruifeng
> 
>
  

17/04/2020 10:49, Ruifeng Wang:
> From: Thomas Monjalon <thomas@monjalon.net>
> > 16/04/2020 15:45, Aaron Conole:
> > > David Marchand <david.marchand@redhat.com> writes:
> > > > On Thu, Apr 16, 2020 at 2:44 PM Aaron Conole <aconole@redhat.com>
> > wrote:
> > > >> Thomas Monjalon <thomas@monjalon.net> writes:
> > > >>
> > > >> > Travis is not reliable for native Arm and PPC:
> > > >> > https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
> > > >> >
> > > >> > In order to get reliable Travis reports, the use of Arm machines
> > > >> > is removed until Travis fixes it.
> > > >> >
> > > >> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > > >> > ---
> > > >>
> > > >> We should add back the cross-build if we do this - at least then we
> > > >> could have a reliable compilation test of Arm64 code.   Does it make
> > > >> sense?
> > > >
> > > > I don't see them removed by this patch, the two jobs are still present ?
> > >
> > > Whoops - for some reason I missed them.  Nevermind :)
> > 
> > So? Acked?
> > 
> Can we achieve this by allowing failures on AArch64 jobs?
> https://docs.travis-ci.com/user/build-matrix/#rows-that-are-allowed-to-fail
> 
> Add following setting:
> jobs:
>   allow_failures:
>   - arch: arm64
> 
> So we can keep the jobs while not suffering from unstable infrastructure.
> Results of these jobs will still observable. This gives us a chance to know when jobs are stable.

I don't see the benefit. It will just make Travis reports unclear.

I wait at least one more week to give Travis a chance to fix Arm support.
Please work with them.
If no result shortly, I will apply this patch to improve DPDK CI reliability.
  

Honnappa, Ruifeng,

On Thu, Apr 16, 2020 at 5:55 PM Honnappa Nagarahalli
<Honnappa.Nagarahalli@arm.com> wrote:
>
> <snip>
>
> > Subject: Re: [PATCH] ci: remove aarch64 from Travis jobs
> >
> > 16/04/2020 15:45, Aaron Conole:
> > > David Marchand <david.marchand@redhat.com> writes:
> > > > On Thu, Apr 16, 2020 at 2:44 PM Aaron Conole <aconole@redhat.com>
> > wrote:
> > > >> Thomas Monjalon <thomas@monjalon.net> writes:
> > > >>
> > > >> > Travis is not reliable for native Arm and PPC:
> > > >> > https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
> Thanks David for creating the ticket. Will escalate this through our contacts at Travis CI, hopefully it can be resolved soon.

There were failures that were obviously because of Travis, like this
quota exceeded error.
But we have other failures on the unit tests that I reported earlier
that are not clear: it might be because of Travis or running in
containers.

Example on last master build:
https://travis-ci.com/github/DPDK/dpdk/builds/160799081

- cycles_autotest failing:
https://travis-ci.com/github/DPDK/dpdk/jobs/320630402#L3460
- some random test ending up in timeout, this time table_autotest, I
also saw eal_fs_autotest:
https://travis-ci.com/github/DPDK/dpdk/jobs/320630406#L7140
  

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Sunday, April 19, 2020 4:01 PM
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Ruifeng Wang
> <Ruifeng.Wang@arm.com>
> Cc: thomas@monjalon.net; Aaron Conole <aconole@redhat.com>; dev
> <dev@dpdk.org>; Gavin Hu <Gavin.Hu@arm.com>; Michael Santana
> <maicolgabriel@hotmail.com>; nd <nd@arm.com>
> Subject: Re: [PATCH] ci: remove aarch64 from Travis jobs
> 
> Honnappa, Ruifeng,
> 
> On Thu, Apr 16, 2020 at 5:55 PM Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com> wrote:
> >
> > <snip>
> >
> > > Subject: Re: [PATCH] ci: remove aarch64 from Travis jobs
> > >
> > > 16/04/2020 15:45, Aaron Conole:
> > > > David Marchand <david.marchand@redhat.com> writes:
> > > > > On Thu, Apr 16, 2020 at 2:44 PM Aaron Conole
> > > > > <aconole@redhat.com>
> > > wrote:
> > > > >> Thomas Monjalon <thomas@monjalon.net> writes:
> > > > >>
> > > > >> > Travis is not reliable for native Arm and PPC:
> > > > >> > https://travis-ci.community/t/disk-quota-exceeded-on-arm64/76
> > > > >> > 19/6
> > Thanks David for creating the ticket. Will escalate this through our contacts
> at Travis CI, hopefully it can be resolved soon.
> 
> There were failures that were obviously because of Travis, like this quota
> exceeded error.
> But we have other failures on the unit tests that I reported earlier that are
> not clear: it might be because of Travis or running in containers.
> 
Yes. Unit test failures are observed more frequently recently in robot's results.

> Example on last master build:
> https://travis-ci.com/github/DPDK/dpdk/builds/160799081
> 
> - cycles_autotest failing:
> https://travis-ci.com/github/DPDK/dpdk/jobs/320630402#L3460
> - some random test ending up in timeout, this time table_autotest, I also saw
> eal_fs_autotest:
> https://travis-ci.com/github/DPDK/dpdk/jobs/320630406#L7140
> 
My ideas here:
1. Modify the test cases to relax criteria for AArch64.
2. Pick test cases to run for AArch64 on Travis.

Option 2 should be better. It only adapts for CI platform and doesn't change code.

/Ruifeng
> 
> --
> David Marchand
  

16/04/2020 13:00, Thomas Monjalon:
> Travis is not reliable for native Arm and PPC:
> https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
> 
> In order to get reliable Travis reports,
> the use of Arm machines is removed until Travis fixes it.
> 
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

We managed without applying this patch.

After one year passed, what is the situation today regarding Travis?
Can we rely on Travis service?
For which workload? Which architecture?

Aaron, what do you recommend?
  

Thomas Monjalon <thomas@monjalon.net> writes:

> 16/04/2020 13:00, Thomas Monjalon:
>> Travis is not reliable for native Arm and PPC:
>> https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
>> 
>> In order to get reliable Travis reports,
>> the use of Arm machines is removed until Travis fixes it.
>> 
>> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
>
> We managed without applying this patch.
>
> After one year passed, what is the situation today regarding Travis?
> Can we rely on Travis service?

So far, yes.

> For which workload? Which architecture?

I think for all of them.  Looking at even the failures which pop up for
the latest patches, they seem like real failures.

ex:
  https://travis-ci.com/github/ovsrobot/dpdk/jobs/493722400
  https://travis-ci.com/github/ovsrobot/dpdk/jobs/493688879
  https://travis-ci.com/github/ovsrobot/dpdk/jobs/493624012
  https://travis-ci.com/github/ovsrobot/dpdk/jobs/493611597

These are ABI, and doc failures - different arches, etc.

Seems like it's quite usable.

> Aaron, what do you recommend?

I think we should drop this patch - Travis continues to be useful even
for individual developers checking their own results.  It seems the
service works quite a bit better now for the project as well, thanks to
Honnappa and other ARM folks for working with them.
  

25/03/2021 17:40, Aaron Conole:
> Thomas Monjalon <thomas@monjalon.net> writes:
> >> Travis is not reliable for native Arm and PPC:
> >> https://travis-ci.community/t/disk-quota-exceeded-on-arm64/7619/6
> >> 
> >> In order to get reliable Travis reports,
> >> the use of Arm machines is removed until Travis fixes it.
> >> 
> >> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> >
> > We managed without applying this patch.
> >
> > After one year passed, what is the situation today regarding Travis?
> > Can we rely on Travis service?
> 
> So far, yes.
> 
> > For which workload? Which architecture?
> 
> I think for all of them.  Looking at even the failures which pop up for
> the latest patches, they seem like real failures.
> 
> ex:
>   https://travis-ci.com/github/ovsrobot/dpdk/jobs/493722400
>   https://travis-ci.com/github/ovsrobot/dpdk/jobs/493688879
>   https://travis-ci.com/github/ovsrobot/dpdk/jobs/493624012
>   https://travis-ci.com/github/ovsrobot/dpdk/jobs/493611597
> 
> These are ABI, and doc failures - different arches, etc.
> 
> Seems like it's quite usable.
> 
> > Aaron, what do you recommend?
> 
> I think we should drop this patch - Travis continues to be useful even
> for individual developers checking their own results.  It seems the
> service works quite a bit better now for the project as well, thanks to
> Honnappa and other ARM folks for working with them.

Thanks all, patch classified as "Rejected".