From patchwork Tue Jul 11 20:21:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adam Hassick X-Patchwork-Id: 129480 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9B93C42E4A; Tue, 11 Jul 2023 22:22:25 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9629D40A7D; Tue, 11 Jul 2023 22:22:25 +0200 (CEST) Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by mails.dpdk.org (Postfix) with ESMTP id AF2824003C for ; Tue, 11 Jul 2023 22:22:23 +0200 (CEST) Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-403392ce2b9so49496241cf.2 for ; Tue, 11 Jul 2023 13:22:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1689106943; x=1691698943; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6stpyKx5LjYIJz03M1mPzv9KXr3tAZQH/rsgxzA5kZ8=; b=IUfrf4yU313vU9jxCHsLk4MsGGNaCBJzsxORgZpUZahYXyhfDxy3Gfz9mGv13uLwr9 nJfx7tnDqN14454R3VBUsz9+8666kZRGeTWfzLsQZFAr1T3X6oi5R05SXEGr3MiBHtMw QFwqpTp6hf+OhgEm53nNMBtcIqywSo//+mIhA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689106943; x=1691698943; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6stpyKx5LjYIJz03M1mPzv9KXr3tAZQH/rsgxzA5kZ8=; b=UnRAduReHVflXwgOYt0raW70mN/ecfREq5kUxZGl87NyGgc4f4+v5Rk5gVd19F8bS7 IMS0fcrDDY/6VnaY1yAJqADudCyNw4CrsTH/xB1TsUwIi4Y5cEvZZz/QIlG6X/zOR3D5 oKr0gjptLAmXgzIbki2SXPv8miC0A0J6r+TAUM+ZCdHnyAR/7FJrZtQyJI2eaMPiRAfU VnFNmcQcCEZ8omoxXCAHZPi5QRRyK1QTlTS1WHUnoYosrtq61O5vARVTzMYnDu6/QqKU 2xvLegVec4lUg5PIJAxhwGh3bhB8ybWvPUD/lUhr9CnhJUg/vPek8fPIFRuQGyThRRDA v3CA== X-Gm-Message-State: ABy/qLafBYa8NvbB7IkjNiz9nDNsrIk671FuhW0k4dY3FjN3Dh4No1WA c23l/RbCkuZzoL/QJSrlcNgGHHEmsk4W2kWOCtuZ2QszAlAeWnQ7zVxm86kmHo6OsJgFvB2nPkN 6PE3DBVb9iPryHsKxWYALhVy589d/0LfXtpkCxxrFweM6Wt49vMrUmpqxaTXgnVPo X-Google-Smtp-Source: APBJJlGaraHjLSsc51z7e9np2XQkPkOQsC0b3QaxcL08oWDMt6ONly7q7KHYWz9H/A+PqxJ2K4UJyw== X-Received: by 2002:ac8:7d16:0:b0:403:aa87:a03d with SMTP id g22-20020ac87d16000000b00403aa87a03dmr10030100qtb.48.1689106942592; Tue, 11 Jul 2023 13:22:22 -0700 (PDT) Received: from ahassick-Desktop.iol.unh.edu ([2606:4100:3880:1220:714e:bdff:bc40:f34f]) by smtp.gmail.com with ESMTPSA id s19-20020ac85293000000b004035c1062f8sm1470604qtn.10.2023.07.11.13.22.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Jul 2023 13:22:22 -0700 (PDT) From: Adam Hassick To: ci@dpdk.org Cc: alialnu@nvidia.com, aconole@redhat.com, Adam Hassick Subject: [PATCH v7 2/6] containers/inventory: Add inventory for container builder Date: Tue, 11 Jul 2023 16:21:20 -0400 Message-Id: <20230711202124.1636317-3-ahassick@iol.unh.edu> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230711202124.1636317-1-ahassick@iol.unh.edu> References: <20230711202124.1636317-1-ahassick@iol.unh.edu> MIME-Version: 1.0 X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org From: Owen Hilyard Adds a yaml file used to define what containers should be built, what packages the containers should have, what platforms to build for and how they should be tagged. Also adds a JSON schema, which can be used to validate that the inventory file is well-formed. Signed-off-by: Owen Hilyard Signed-off-by: Adam Hassick --- containers/template_engine/inventory.yaml | 470 ++++++++++++++++++ .../template_engine/inventory_schema.json | 221 ++++++++ 2 files changed, 691 insertions(+) create mode 100644 containers/template_engine/inventory.yaml create mode 100644 containers/template_engine/inventory_schema.json diff --git a/containers/template_engine/inventory.yaml b/containers/template_engine/inventory.yaml new file mode 100644 index 0000000..31dbdb6 --- /dev/null +++ b/containers/template_engine/inventory.yaml @@ -0,0 +1,470 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2022 University of New Hampshire +--- +abi: + # versions should also include the abi versions for all stable ABIs. + versions: + # - dpdk_version: # LTS - 3 + # major: 20 + # minor: 11 + # revision: 0 + # git_tag: "20.11" + # abi_version: + # major: 21 + # minor: 0 + - dpdk_version: # LTS - 2 + major: 21 + minor: 11 + revision: 0 + git_tag: "21.11" + abi_version: + major: 22 + minor: 0 + - dpdk_version: + major: 22 # LTS - 1 + minor: 11 + revision: 1 + git_tag: "tags/v22.11.1" + abi_version: + major: 23 + minor: 10 # .xml comparison + - dpdk_version: + major: 22 # LTS + minor: 11 + revision: 1 + git_tag: "tags/v22.11.1" + abi_version: + major: 23 + minor: 11 # .so comparison + +dockerfiles: + groups: + all: + package_manager_command: "NO_INSTALL_COMMAND_SET" + parent: "NONE" + packages: + - gcc + - make + - git + - sudo + - curl + - automake + - autoconf + - flex + - bison + - bc + rpm: + parent: "all" + package_manager_command: yum install -y + packages: + - diffutils + - pkg-config + - python3 + - python3-pip + - librdmacm + - rdma-core-devel + - libmnl-devel + - ccache + - zip + - autoconf + - libbpf-devel + redhat_family: + parent: "rpm" + packages: [] + fedora: + parent: "redhat_family" + packages: + - python3-pyelftools + - zip + - clang + - ninja-build + - elfutils-libelf-devel + - jansson-devel + - libatomic + - libpcap-devel + - numactl-devel + - findutils + - libtool + - elfutils-devel + - libxml2-devel + - libabigail + - openssl-devel + fedora_coverity: + parent: fedora + packages: [] + fedora_clang: + parent: fedora + packages: [] + centos8: + parent: "redhat_family" + packages: + - elfutils-libelf-devel + - jansson-devel + - libatomic + - libpcap-devel + - numactl-devel + - libabigail + - openssl-devel + centos9: + parent: "redhat_family" + package_manager_command: "yum install --allowerasing -y" + packages: + - elfutils-libelf-devel + - jansson-devel + - libatomic + - libpcap-devel + - numactl-devel + - libtool + - g++ + - elfutils-devel + - libxml2-devel + - openssl-devel + rhel: + parent: "redhat_family" + package_manager_command: "yum install -y" + packages: + - elfutils-libelf-devel + - jansson-devel + - libatomic + - libpcap-devel + - numactl-devel + - openssl-devel + - libabigail + rhel7: + parent: "rhel" + packages: + - ninja-build + rhel8: + parent: "rhel" + package_manager_command: "yum --allowerasing install -y" + packages: + - ninja-build + - libxml2-devel + rhel9: + parent: "rhel" + package_manager_command: "yum --allowerasing install -y" + packages: + - libxml2-devel + opensuse: + parent: "rpm" + package_manager_command: "zypper install -y" + packages: + - python3-pyelftools + - ninja + - gzip + - libelf-devel + - libjansson-devel + - librdmacm1 + - libatomic1 + - libnuma-devel + - libabigail-tools + - tar + - openssl-devel + alpine: + parent: "all" + package_manager_command: "apk add" + packages: + - bsd-compat-headers + - ccache + - jansson-dev + - libarchive-dev + - libbpf-dev + - libc-dev + - libpcap-dev + - linux-headers + - ninja + - numactl-dev + - openssl-dev + - py3-pip + - py3-elftools + - py3-setuptools + - py3-wheel + - zlib-dev + - python3 + - bash + - zip + - meson + arch: + parent: "all" + package_manager_command: "pacman -Syu --noconfirm" + packages: + - glibc + - cmake + - ninja + - pandoc + - python + - python-docutils + - ethtool + - file + - fakeroot + - pkg-config + - numactl + - elfutils + - gawk + - diffutils + - python-pip + - libabigail + - openssl + - libbpf + debian: + parent: "all" + package_manager_command: "apt-get update && apt-get install --no-install-recommends -y" + packages: + - libnuma-dev + - zip + - librdmacm1 + - libcrypto++-dev + - python3-pyelftools + - libxml2-dev + - libelf-dev + - libpcap-dev + - libjansson-dev + - python3-setuptools + - autoconf + - pkg-config + - libtool + - ccache + - libatomic1 + - python3-wheel + - libdw-dev + - rdma-core + - ninja-build + - automake + - librdmacm-dev + - libmnl-dev + - python3-pip + - build-essential + - aspell-en + - parallel + - python3-grpcio + - lcov + - libbpf-dev + - libssl-dev + debian10: + parent: "debian" + packages: [] + debian11: + parent: "debian" + packages: + - nasm + debian11_arm_ipsec: + parent: "debian11" + packages: [] + ubuntu: + parent: 'debian' + packages: [] + ubuntu18.04: + parent: "ubuntu" + packages: [] + ubuntu20.04: + parent: "ubuntu" + packages: + - python3-grpcio + - lcov + - jq + - default-jre + - nodejs + ubuntu22.04: + parent: "ubuntu" + packages: + - python3-grpcio + - lcov + - abigail-tools + - jq + ubuntu_cross: # x86 -> ARM + parent: "ubuntu20.04" + packages: + - clang + - llvm + - llvm-dev + - llvm-runtime + - lld + - gcc-aarch64-linux-gnu + - libgcc-9-dev-arm64-cross + - libatomic1-arm64-cross + - libc6-dev-arm64-cross + - pkg-config-aarch64-linux-gnu + - dpkg-dev + - gcc-arm-linux-gnueabihf + - libc6-dev-armhf-cross + - pkg-config-arm-linux-gnueabihf + ubuntu_sve: + parent: "ubuntu20.04" + packages: + - gcc-10 + targets: + # Alpine + - name: alpine_compile + group: "alpine" + base_image: alpine + platforms: [linux/amd64, linux/arm64] + force_disable_abi: true + minimum_dpdk_version: + major: 22 + minor: 0 + revision: 0 + extra_information: + amd64_environment_id: 45 + arm64_environment_id: 75 + + # Arch + - name: arch_compile + group: "arch" + base_image: archlinux:base + platforms: [linux/amd64] + extra_information: + amd64_environment_id: 26 + + - name: fedora37_compile + group: "fedora" + base_image: fedora:37 + platforms: [linux/amd64, linux/arm64] + extra_information: + amd64_environment_id: 69 + arm64_environment_id: 70 + + - name: fedora38_compile + group: "fedora" + base_image: fedora:38 + platforms: [linux/amd64, linux/arm64] + extra_information: + amd64_environment_id: 71 + arm64_environment_id: 72 + + - name: fedora38_clang + group: "fedora_clang" + base_image: fedora:38 + platforms: [linux/amd64, linux/arm64] + extra_information: + amd64_environment_id: 73 + arm64_environment_id: 74 + + - name: fedora37_coverity + group: "fedora_coverity" + base_image: fedora:37 + force_disable_abi: true + requires_coverity: true + platforms: [linux/amd64] + + # CentOs + - name: centosstream8_compile + group: "centos8" + base_image: quay.io/centos/centos:stream8 + platforms: [linux/amd64] + extra_information: + amd64_environment_id: 39 + + - name: centosstream9_compile + group: "centos9" + base_image: quay.io/centos/centos:stream9 + platforms: [linux/amd64, linux/arm64] + extra_information: + amd64_environment_id: 53 + arm64_environment_id: 63 + + # RHEL + - name: rhel7_compile + group: "rhel7" + base_image: registry.access.redhat.com/ubi7/ubi + platforms: [linux/amd64] + extra_information: + amd64_environment_id: 36 + + - name: rhel8_compile + group: "rhel8" + base_image: registry.access.redhat.com/ubi8/ubi + platforms: [linux/amd64] + extra_information: + amd64_environment_id: 37 + + - name: rhel9_compile + group: "rhel9" + base_image: registry.access.redhat.com/ubi9/ubi + platforms: [linux/amd64, linux/arm64] + extra_information: + amd64_environment_id: 76 + arm64_environment_id: 77 + + # OpenSuse + - name: opensuse-leap15_compile + group: "opensuse" + base_image: opensuse/leap:15 + platforms: [linux/amd64, linux/arm64] + extra_information: + amd64_environment_id: 25 + arm64_environment_id: 67 + + # Debian + - name: debian10_compile # debian 10 + group: "debian10" + base_image: "debian:buster" + libabigail_revision: libabigail-2.2 + platforms: [linux/amd64] + extra_information: + amd64_environment_id: 52 + + - name: debian11_compile # debian 11 + group: "debian11" + base_image: "debian:bullseye" + libabigail_revision: libabigail-2.2 + platforms: + - linux/amd64 + - linux/arm64 + extra_information: + amd64_environment_id: 51 + arm64_environment_id: 64 + + # Ubuntu + - name: ubuntu20.04_compile + group: "ubuntu20.04" + base_image: "ubuntu:20.04" + platforms: [linux/amd64, linux/arm64] + extra_tags: + - $R/ubuntu20.04_libabigail:latest + - $R/ubuntu20.04_libabigail:$T + - $R/ubuntu20.04_spell-check:latest + - $R/ubuntu20.04_spell-check:$T + - $R/ubuntu20.04_sonarscanner:latest + - $R/ubuntu20.04_sonarscanner:$T + extra_information: + amd64_environment_id: 38 + arm64_environment_id: 62 + + # x86 -> ARM cross compile + - name: ubuntu20.04_cross + group: "ubuntu_cross" + base_image: "ubuntu:20.04" + platforms: [linux/amd64] + cross_file: "config/arm/arm64_armv8_linux_gcc" + extra_information: + amd64_environment_id: 47 + + # Also, x86 -> ARM cross compile (for SVE test) + - name: ubuntu20.04_sve_compile + group: "ubuntu_sve" + base_image: "arm64v8/ubuntu:20.04" + platforms: [linux/arm64] + extra_information: + arm64_environment_id: 46 + + # TEMPORARY: This has the forked IPsec lib developed by ARM. This causes issues on older LTS versions of DPDK. + # Once all supported and tested LTS versions no longer break when this lib is installed, recombine with debian11_compile. + - name: debian11_arm_ipsec + group: "debian11_arm_ipsec" + base_image: "debian:bullseye" + platforms: [linux/arm64] + force_disable_abi: true + extra_tags: + - $R/debian11_zuc_compile:latest + - $R/debian11_snow3g_compile:latest + - $R/debian11_zuc_compile:$T + - $R/debian11_snow3g_compile:$T + extra_information: + arm64_environment_id: 58 + + - name: ubuntu22.04_compile + group: "ubuntu22.04" + base_image: "ubuntu:22.04" + platforms: [linux/amd64] + extra_information: + amd64_environment_id: 54 diff --git a/containers/template_engine/inventory_schema.json b/containers/template_engine/inventory_schema.json new file mode 100644 index 0000000..7b79e44 --- /dev/null +++ b/containers/template_engine/inventory_schema.json @@ -0,0 +1,221 @@ +{ + "$schema": "https://json-schema.org/draft-07/schema", + "title": "dpdklab-ci container inventory schema", + "$comment": "SPDX-License-Identifier: BSD-3-Clause\nCopyright (c) 2022 University of New Hampshire", + "type": "object", + "properties": { + "abi": { + "type": "object", + "properties": { + "versions": { + "type": "array", + "items": { + "type": "object", + "properties": { + "dpdk_version": { + "type": "object", + "properties": { + "major": { + "type": "integer", + "description": "The major version number (usually the year of release)" + }, + "minor": { + "type": "integer", + "description": "The minor version number (usually the month of release)" + }, + "revision": { + "type": "integer", + "description": "The revision version number. Starts at 0." + }, + "git_tag": { + "type": "string", + "description": "The git tag to check out to get this ABI version" + } + }, + "required": [ + "major", + "minor", + "revision", + "git_tag" + ], + "additionalProperties": false + }, + "abi_version": { + "type": "object", + "properties": { + "major": { + "type": "integer", + "description": "The major version number (usually the year of release)" + }, + "minor": { + "type": "integer", + "description": "The minor version number (usually the month of release)" + } + }, + "required": [ + "major", + "minor" + ], + "additionalProperties": false + } + }, + "required": [ + "dpdk_version", + "abi_version" + ] + }, + "minItems": 0 + } + }, + "required": [ + "versions" + ], + "additionalProperties": false + }, + "dockerfiles": { + "type": "object", + "properties": { + "groups": { + "type": "object", + "patternProperties": { + "^[a-z][a-z_0-9\\.]+$": { + "type": "object", + "properties": { + "package_manager_command": { + "type": "string", + "description": "The command to install packages without user input." + }, + "parent": { + "type": "string", + "description": "Which group to inherit from, or NONE if no parent exists" + }, + "packages": { + "type": "array", + "description": "A list of package names to install", + "items": { + "type": "string" + }, + "uniqueItems": true + } + }, + "required": [ + "parent", + "packages" + ], + "additionalProperties": false + } + }, + "minProperties": 1 + }, + "targets": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "type": "string", + "description": "The name of the docker container" + }, + "group": { + "type": "string", + "description": "Which group the target is a member of" + }, + "base_image": { + "type": "string", + "description": "The base docker image." + }, + "platforms": { + "type": "array", + "description": "What platforms to build for", + "items": { + "type": "string", + "enum": [ + "linux/amd64", + "linux/arm64", + "linux/riscv64", + "linux/ppc64le", + "linux/arm/v7", + "linux/arm/v6" + ] + }, + "minItems": 1 + }, + "cross_file": { + "type": "string", + "description": "A cross file to compile meson with. If not present, meson's automatic behavior will be used." + }, + "libabigail_revision": { + "type": "string", + "description": "A revision or branch of libabigail to check out before building for a given target." + }, + "force_disable_abi": { + "type": "boolean", + "description": "Whether to forcibly disable ABI for this target. This should be used as a temporary measure until a fix can be created.", + "default": "false" + }, + "requires_coverity": { + "type": "boolean", + "description": "Whether this target requires the Coverity Scan binaries.", + "default": "false" + }, + "minimum_dpdk_version": { + "type": "object", + "properties": { + "major": { + "type": "integer", + "description": "The major version number (usually the year of release)" + }, + "minor": { + "type": "integer", + "description": "The minor version number (usually the month of release)" + }, + "revision": { + "type": "integer", + "description": "The revision version number. Starts at 0." + } + }, + "required": [ + "major", + "minor", + "revision" + ], + "additionalProperties": false + }, + "extra_tags": { + "type": "array", + "description": "Tags to apply to this image", + "items": { + "type": "string", + "description": "$R will be replaced with the 'registry_hostname' variable in this file. $T will be replaced with a timestamp in the form of '%Y-%m-%d'. $N will be replaced by the name of the target.. The tags '$R/$N:latest' and '$R/$N:$T' will always be present" + } + }, + "extra_information": { + "type": "object", + "description": "Extra information to be stored as a JSON file at /container_info.json", + "additionalProperties": true + } + }, + "required": [ + "name", + "group", + "base_image", + "platforms" + ], + "additionalProperties": false + }, + "minItems": 1 + } + }, + "required": [ + "groups", + "targets" + ], + "additionalProperties": false + } + }, + "required": [ + "abi", + "dockerfiles" + ], + "additionalProperties": false +} \ No newline at end of file