[v2,1/2] ci: hook to GitHub Actions

  With the recent changes in terms of free access to the Travis CI, let's
offer an alternative with GitHub Actions.
Running jobs on ARM is not supported unless using external runners, so
this commit only adds builds for x86_64 and cross compiling for i386 and
aarch64.

Differences with the Travis CI integration:
- Error logs are not dumped to the console when something goes wrong.
  Instead, they are gathered in a "catch-all" step and attached as
  artifacts.
- A cache entry is stored once and for all, but if no cache is found you
  can inherit from the default branch cache. The cache is 5GB large, for
  the whole git repository.
- The maximum retention of logs and artifacts is 3 months.
- /home/runner is world writable, so a workaround has been added for
  starting dpdk processes.
- Ilya, working on OVS GHA support, noticed that jobs can run with
  processors that don't have the same capabilities. For DPDK, this
  impacts the ccache content since everything was built with
  -march=native so far, and we will end up with binaries that can't run
  in a later build. The problem has not been seen in Travis CI (?) but
  it is safer to use a fixed "-Dmachine=default" in any case.
- Scheduling jobs is part of the configuration and takes the form of a
  crontab. A build is scheduled every Monday at 0:00 (UTC) to provide a
  default ccache for the week (useful for the ovsrobot).

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changelog since v1:
- changed shell variables value in CI scripts and Travis configuration
  (s/=[^\$]*1/=\1true), this makes it easier for GHA,
- forced compilation as 'default' to avoid random unit tests issues in
  GHA,
- scheduled a run per week on Monday at 0:00 UTC,
- updated the ccache key:
  - no need to depend on the default-library parameter since this
    parameter only impacts the linking of dpdk binaries,
  - the week when the cache is generated is added so that jobs in
    other branches can benefit from a recent cache (mimicking what we had
    for the robot in Travis),
- realigned documentation generation with what is done in Travis:
  generating the doc in all jobs was a waste of resources,

---
 .ci/linux-build.sh          |  17 +++---
 .github/workflows/build.yml | 100 ++++++++++++++++++++++++++++++++++++
 .travis.yml                 |  24 ++++-----
 MAINTAINERS                 |   1 +
 4 files changed, 123 insertions(+), 19 deletions(-)
 create mode 100644 .github/workflows/build.yml
  

On 12/4/2020 5:36 PM, David Marchand wrote:
> With the recent changes in terms of free access to the Travis CI, let's
> offer an alternative with GitHub Actions.
> Running jobs on ARM is not supported unless using external runners, so
> this commit only adds builds for x86_64 and cross compiling for i386 and
> aarch64.
> 
> Differences with the Travis CI integration:
> - Error logs are not dumped to the console when something goes wrong.
>    Instead, they are gathered in a "catch-all" step and attached as
>    artifacts.
> - A cache entry is stored once and for all, but if no cache is found you
>    can inherit from the default branch cache. The cache is 5GB large, for
>    the whole git repository.
> - The maximum retention of logs and artifacts is 3 months.
> - /home/runner is world writable, so a workaround has been added for
>    starting dpdk processes.
> - Ilya, working on OVS GHA support, noticed that jobs can run with
>    processors that don't have the same capabilities. For DPDK, this
>    impacts the ccache content since everything was built with
>    -march=native so far, and we will end up with binaries that can't run
>    in a later build. The problem has not been seen in Travis CI (?) but
>    it is safer to use a fixed "-Dmachine=default" in any case.
> - Scheduling jobs is part of the configuration and takes the form of a
>    crontab. A build is scheduled every Monday at 0:00 (UTC) to provide a
>    default ccache for the week (useful for the ovsrobot).
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> Changelog since v1:
> - changed shell variables value in CI scripts and Travis configuration
>    (s/=[^\$]*1/=\1true), this makes it easier for GHA,
> - forced compilation as 'default' to avoid random unit tests issues in
>    GHA,
> - scheduled a run per week on Monday at 0:00 UTC,
> - updated the ccache key:
>    - no need to depend on the default-library parameter since this
>      parameter only impacts the linking of dpdk binaries,
>    - the week when the cache is generated is added so that jobs in
>      other branches can benefit from a recent cache (mimicking what we had
>      for the robot in Travis),
> - realigned documentation generation with what is done in Travis:
>    generating the doc in all jobs was a waste of resources,
> 

For series,
Tested-by: Ferruh Yigit <ferruh.yigit@intel.com>

Confirmed that ABI check script is detecting issues, in the absence of the 
Travis checks I am for having this alternative.
  

11/12/2020 21:07, Ferruh Yigit:
> On 12/4/2020 5:36 PM, David Marchand wrote:
> > With the recent changes in terms of free access to the Travis CI, let's
> > offer an alternative with GitHub Actions.
> > Running jobs on ARM is not supported unless using external runners, so
> > this commit only adds builds for x86_64 and cross compiling for i386 and
> > aarch64.
> > 
> > Differences with the Travis CI integration:
> > - Error logs are not dumped to the console when something goes wrong.
> >    Instead, they are gathered in a "catch-all" step and attached as
> >    artifacts.
> > - A cache entry is stored once and for all, but if no cache is found you
> >    can inherit from the default branch cache. The cache is 5GB large, for
> >    the whole git repository.
> > - The maximum retention of logs and artifacts is 3 months.
> > - /home/runner is world writable, so a workaround has been added for
> >    starting dpdk processes.
> > - Ilya, working on OVS GHA support, noticed that jobs can run with
> >    processors that don't have the same capabilities. For DPDK, this
> >    impacts the ccache content since everything was built with
> >    -march=native so far, and we will end up with binaries that can't run
> >    in a later build. The problem has not been seen in Travis CI (?) but
> >    it is safer to use a fixed "-Dmachine=default" in any case.
> > - Scheduling jobs is part of the configuration and takes the form of a
> >    crontab. A build is scheduled every Monday at 0:00 (UTC) to provide a
> >    default ccache for the week (useful for the ovsrobot).
> > 
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> > Changelog since v1:
> > - changed shell variables value in CI scripts and Travis configuration
> >    (s/=[^\$]*1/=\1true), this makes it easier for GHA,
> > - forced compilation as 'default' to avoid random unit tests issues in
> >    GHA,
> > - scheduled a run per week on Monday at 0:00 UTC,
> > - updated the ccache key:
> >    - no need to depend on the default-library parameter since this
> >      parameter only impacts the linking of dpdk binaries,
> >    - the week when the cache is generated is added so that jobs in
> >      other branches can benefit from a recent cache (mimicking what we had
> >      for the robot in Travis),
> > - realigned documentation generation with what is done in Travis:
> >    generating the doc in all jobs was a waste of resources,
> > 
> 
> For series,
> Tested-by: Ferruh Yigit <ferruh.yigit@intel.com>
> 
> Confirmed that ABI check script is detecting issues, in the absence of the 
> Travis checks I am for having this alternative.

Thanks for offering an interesting CI alternative.
For the series,
Acked-by: Thomas Monjalon <thomas@monjalon.net>
  

David Marchand <david.marchand@redhat.com> writes:

> With the recent changes in terms of free access to the Travis CI, let's
> offer an alternative with GitHub Actions.
> Running jobs on ARM is not supported unless using external runners, so
> this commit only adds builds for x86_64 and cross compiling for i386 and
> aarch64.
>
> Differences with the Travis CI integration:
> - Error logs are not dumped to the console when something goes wrong.
>   Instead, they are gathered in a "catch-all" step and attached as
>   artifacts.
> - A cache entry is stored once and for all, but if no cache is found you
>   can inherit from the default branch cache. The cache is 5GB large, for
>   the whole git repository.
> - The maximum retention of logs and artifacts is 3 months.
> - /home/runner is world writable, so a workaround has been added for
>   starting dpdk processes.
> - Ilya, working on OVS GHA support, noticed that jobs can run with
>   processors that don't have the same capabilities. For DPDK, this
>   impacts the ccache content since everything was built with
>   -march=native so far, and we will end up with binaries that can't run
>   in a later build. The problem has not been seen in Travis CI (?) but
>   it is safer to use a fixed "-Dmachine=default" in any case.

That's because the build machine and test machine are the same, but I
think GHA uses a different model, and will spawn a new environment for
the steps.  I'm not 100% sure, because it's all supposed to be a
black-box.

> - Scheduling jobs is part of the configuration and takes the form of a
>   crontab. A build is scheduled every Monday at 0:00 (UTC) to provide a
>   default ccache for the week (useful for the ovsrobot).
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---

Acked-by: Aaron Conole <aconole@redhat.com>

> Changelog since v1:
> - changed shell variables value in CI scripts and Travis configuration
>   (s/=[^\$]*1/=\1true), this makes it easier for GHA,
> - forced compilation as 'default' to avoid random unit tests issues in
>   GHA,
> - scheduled a run per week on Monday at 0:00 UTC,
> - updated the ccache key:
>   - no need to depend on the default-library parameter since this
>     parameter only impacts the linking of dpdk binaries,
>   - the week when the cache is generated is added so that jobs in
>     other branches can benefit from a recent cache (mimicking what we had
>     for the robot in Travis),
> - realigned documentation generation with what is done in Travis:
>   generating the doc in all jobs was a waste of resources,
>
> ---
>  .ci/linux-build.sh          |  17 +++---
>  .github/workflows/build.yml | 100 ++++++++++++++++++++++++++++++++++++
>  .travis.yml                 |  24 ++++-----
>  MAINTAINERS                 |   1 +
>  4 files changed, 123 insertions(+), 19 deletions(-)
>  create mode 100644 .github/workflows/build.yml
>
> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> index d079801d78..ee8d07f865 100755
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> @@ -12,7 +12,9 @@ on_error() {
>          fi
>      done
>  }
> -trap on_error EXIT
> +# We capture the error logs as artifacts in Github Actions, no need to dump
> +# them via a EXIT handler.
> +[ -n "$GITHUB_WORKFLOW" ] || trap on_error EXIT
>  
>  install_libabigail() {
>      version=$1
> @@ -28,16 +30,16 @@ install_libabigail() {
>      rm ${version}.tar.gz
>  }
>  
> -if [ "$AARCH64" = "1" ]; then
> +if [ "$AARCH64" = "true" ]; then
>      # convert the arch specifier
>      OPTS="$OPTS --cross-file config/arm/arm64_armv8_linux_gcc"
>  fi
>  
> -if [ "$BUILD_DOCS" = "1" ]; then
> +if [ "$BUILD_DOCS" = "true" ]; then
>      OPTS="$OPTS -Denable_docs=true"
>  fi
>  
> -if [ "$BUILD_32BIT" = "1" ]; then
> +if [ "$BUILD_32BIT" = "true" ]; then
>      OPTS="$OPTS -Dc_args=-m32 -Dc_link_args=-m32"
>      export PKG_CONFIG_LIBDIR="/usr/lib32/pkgconfig"
>  fi
> @@ -48,16 +50,17 @@ else
>      OPTS="$OPTS -Dexamples=all"
>  fi
>  
> +OPTS="$OPTS -Dmachine=default"
>  OPTS="$OPTS --default-library=$DEF_LIB"
>  OPTS="$OPTS --buildtype=debugoptimized"
>  meson build --werror $OPTS
>  ninja -C build
>  
> -if [ "$AARCH64" != "1" ]; then
> +if [ "$AARCH64" != "true" ]; then
>      devtools/test-null.sh
>  fi
>  
> -if [ "$ABI_CHECKS" = "1" ]; then
> +if [ "$ABI_CHECKS" = "true" ]; then
>      LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
>  
>      if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> @@ -95,6 +98,6 @@ if [ "$ABI_CHECKS" = "1" ]; then
>      devtools/check-abi.sh reference install ${ABI_CHECKS_WARN_ONLY:-}
>  fi
>  
> -if [ "$RUN_TESTS" = "1" ]; then
> +if [ "$RUN_TESTS" = "true" ]; then
>      sudo meson test -C build --suite fast-tests -t 3
>  fi
> diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
> new file mode 100644
> index 0000000000..bef6e52372
> --- /dev/null
> +++ b/.github/workflows/build.yml
> @@ -0,0 +1,100 @@
> +name: build
> +
> +on:
> +  push:
> +  schedule:
> +    - cron: '0 0 * * 1'
> +
> +defaults:
> +  run:
> +    shell: bash --noprofile --norc -exo pipefail {0}
> +
> +jobs:
> +  build:
> +    name: ${{ join(matrix.config.*, '-') }}
> +    runs-on: ${{ matrix.config.os }}
> +    env:
> +      AARCH64: ${{ matrix.config.cross == 'aarch64' }}
> +      BUILD_32BIT: ${{ matrix.config.cross == 'i386' }}
> +      BUILD_DOCS: ${{ contains(matrix.config.checks, 'doc') }}
> +      CC: ccache ${{ matrix.config.compiler }}
> +      DEF_LIB: ${{ matrix.config.library }}
> +      RUN_TESTS: ${{ contains(matrix.config.checks, 'tests') }}
> +
> +    strategy:
> +      fail-fast: false
> +      matrix:
> +        config:
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: static
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: shared
> +            checks: doc+tests
> +          - os: ubuntu-18.04
> +            compiler: clang
> +            library: static
> +          - os: ubuntu-18.04
> +            compiler: clang
> +            library: shared
> +            checks: doc+tests
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: static
> +            cross: i386
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: static
> +            cross: aarch64
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: shared
> +            cross: aarch64
> +
> +    steps:
> +    - name: Checkout sources
> +      uses: actions/checkout@v2
> +    - name: Generate cache keys
> +      id: get_ref_keys
> +      run: |
> +        echo -n '::set-output name=ccache::'
> +        echo 'ccache-${{ matrix.config.os }}-${{ matrix.config.compiler }}-${{ matrix.config.cross }}-'$(date -u +%Y-w%W)
> +    - name: Retrieve ccache cache
> +      uses: actions/cache@v2
> +      with:
> +        path: ~/.ccache
> +        key: ${{ steps.get_ref_keys.outputs.ccache }}-${{ github.ref }}
> +        restore-keys: |
> +          ${{ steps.get_ref_keys.outputs.ccache }}-refs/heads/main
> +    - name: Install packages
> +      run: sudo apt install -y ccache libnuma-dev python3-setuptools
> +        python3-wheel python3-pip ninja-build libbsd-dev libpcap-dev
> +        libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
> +    - name: Install i386 cross compiling packages
> +      if: env.BUILD_32BIT == 'true'
> +      run: sudo apt install -y gcc-multilib
> +    - name: Install aarch64 cross compiling packages
> +      if: env.AARCH64 == 'true'
> +      run: sudo apt install -y gcc-aarch64-linux-gnu libc6-dev-arm64-cross
> +        pkg-config-aarch64-linux-gnu
> +    - name: Install doc generation packages
> +      if: env.BUILD_DOCS == 'true'
> +      run: sudo apt install -y doxygen graphviz python3-sphinx
> +        python3-sphinx-rtd-theme
> +    - name: Run setup
> +      run: |
> +        .ci/linux-setup.sh
> +        # Workaround on $HOME permissions as EAL checks them for plugin loading
> +        chmod o-w $HOME
> +    - name: Build and test
> +      run: .ci/linux-build.sh
> +    - name: Upload logs on failure
> +      if: failure()
> +      uses: actions/upload-artifact@v2
> +      with:
> +        name: meson-logs-${{ join(matrix.config.*, '-') }}
> +        path: |
> +          build/meson-logs/testlog.txt
> +          build/.ninja_log
> +          build/meson-logs/meson-log.txt
> diff --git a/.travis.yml b/.travis.yml
> index 5e12db23b5..d655e286c3 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -34,10 +34,10 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: amd64
>      compiler: gcc
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      arch: amd64
>      compiler: gcc
> -  - env: DEF_LIB="shared" BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=true
>      arch: amd64
>      compiler: gcc
>      addons:
> @@ -49,10 +49,10 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: amd64
>      compiler: clang
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      arch: amd64
>      compiler: clang
> -  - env: DEF_LIB="shared" BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=true
>      arch: amd64
>      compiler: clang
>      addons:
> @@ -61,7 +61,7 @@ jobs:
>            - *required_packages
>            - *doc_packages
>    # x86_64 cross-compiling 32-bits jobs
> -  - env: DEF_LIB="static" BUILD_32BIT=1
> +  - env: DEF_LIB="static" BUILD_32BIT=true
>      arch: amd64
>      compiler: gcc
>      addons:
> @@ -69,14 +69,14 @@ jobs:
>          packages:
>            - *build_32b_packages
>    # x86_64 cross-compiling aarch64 jobs
> -  - env: DEF_LIB="static" AARCH64=1
> +  - env: DEF_LIB="static" AARCH64=true
>      arch: amd64
>      compiler: gcc
>      addons:
>        apt:
>          packages:
>            - *aarch64_packages
> -  - env: DEF_LIB="shared" AARCH64=1
> +  - env: DEF_LIB="shared" AARCH64=true
>      arch: amd64
>      compiler: gcc
>      addons:
> @@ -87,16 +87,16 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: arm64
>      compiler: gcc
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      arch: arm64
>      compiler: gcc
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      dist: focal
>      arch: arm64-graviton2
>      virt: vm
>      group: edge
>      compiler: gcc
> -  - env: DEF_LIB="shared" BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=true
>      arch: arm64
>      compiler: gcc
>      addons:
> @@ -108,10 +108,10 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: arm64
>      compiler: clang
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      arch: arm64
>      compiler: clang
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      dist: focal
>      arch: arm64-graviton2
>      virt: vm
> diff --git a/MAINTAINERS b/MAINTAINERS
> index eafe9f8c46..f45c8c1b13 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -109,6 +109,7 @@ Public CI
>  M: Aaron Conole <aconole@redhat.com>
>  M: Michael Santana <maicolgabriel@hotmail.com>
>  F: .travis.yml
> +F: .github/workflows/build.yml
>  F: .ci/
>  
>  ABI Policy & Versioning
  

On Fri, Dec 4, 2020 at 6:37 PM David Marchand <david.marchand@redhat.com> wrote:
>
> With the recent changes in terms of free access to the Travis CI, let's
> offer an alternative with GitHub Actions.
> Running jobs on ARM is not supported unless using external runners, so
> this commit only adds builds for x86_64 and cross compiling for i386 and
> aarch64.
>
> Differences with the Travis CI integration:
> - Error logs are not dumped to the console when something goes wrong.
>   Instead, they are gathered in a "catch-all" step and attached as
>   artifacts.
> - A cache entry is stored once and for all, but if no cache is found you
>   can inherit from the default branch cache. The cache is 5GB large, for
>   the whole git repository.
> - The maximum retention of logs and artifacts is 3 months.
> - /home/runner is world writable, so a workaround has been added for
>   starting dpdk processes.
> - Ilya, working on OVS GHA support, noticed that jobs can run with
>   processors that don't have the same capabilities. For DPDK, this
>   impacts the ccache content since everything was built with
>   -march=native so far, and we will end up with binaries that can't run
>   in a later build. The problem has not been seen in Travis CI (?) but
>   it is safer to use a fixed "-Dmachine=default" in any case.
> - Scheduling jobs is part of the configuration and takes the form of a
>   crontab. A build is scheduled every Monday at 0:00 (UTC) to provide a
>   default ccache for the week (useful for the ovsrobot).
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>

Tested-by: Ferruh Yigit <ferruh.yigit@intel.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: Aaron Conole <aconole@redhat.com>

Series applied.

For the time being, we can check jobs status by checking:
https://github.com/ovsrobot/dpdk/actions
Next step is to update the robot to make use of
https://docs.github.com/en/free-pro-team@latest/rest/reference/actions#list-workflow-runs-for-a-repository