From patchwork Tue Jul 11 20:21:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adam Hassick X-Patchwork-Id: 129478 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6512142E4A; Tue, 11 Jul 2023 22:22:00 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5E56140A7D; Tue, 11 Jul 2023 22:22:00 +0200 (CEST) Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by mails.dpdk.org (Postfix) with ESMTP id 6C91D4003C for ; Tue, 11 Jul 2023 22:21:59 +0200 (CEST) Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-4039a2b71c1so29723871cf.0 for ; Tue, 11 Jul 2023 13:21:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1689106918; x=1691698918; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=iLrvQ4GecMZkAc0igf9Ds8a8rfnrrDQvNwmM5SsmmVg=; b=MVOYh6yUWrjsiyidPWyv7hM8Ej6hc0LqUyrFQ600XXbxs0JpA987himC996dIU8hGO /lgC8awhZGy2MChAhBvXbFDmMzDRHca7WwSNp95waROz1+s275SXeoX3El+ce7fu79QS mpmNP+VsaPfnGEVcOhff/T/CI0zRgBa/1vXwM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689106918; x=1691698918; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=iLrvQ4GecMZkAc0igf9Ds8a8rfnrrDQvNwmM5SsmmVg=; b=Vt2XkV6IwV/UpUV/VQMwgm2MvLGJiKcSH/XciH9OCsaERHjFYkiRCJU3Mh9PlQTA4J lFrPGWQqFqt7s//uL6k5jH/iMUp6K7Mo61JjuNkRQBb3T87T9jBvi3DI6+Haajhn9x3W G6NBTKriZv6LD0B/OXaRGghpYxZlWUGpnjxaiRlrnxVMAAhvlCm3miIj7dioUQf7/tME SqvjCR/laoIyXmyxbySZL+SQjez3Xd7ifmnM/CLe5Skz1ESDc+1TF7DI38H8bBeDm9xP YESdi//46dwa6LHYqMhSysvLp7DdGjLagNMgBfQvwpXGLAE6ydmBudH67hO0E9U3oa3H 0dLg== X-Gm-Message-State: ABy/qLaEQ9ZjYAJO5mF4Tl7v44drlmQk2+RAURAYD/RH8TONJtIDf9E+ KfBlawFo0toeHoFZNrUr2YWC+I77dCodaPLf70NDJAw4k1KrTdMpudYPcHLr9Xnr1vDRq/ACcAF JvXiZLZiwdq2Wu9Q4tsWa7pXsM33UJ+cYD72yPoAbk2boARoP4+XCTdZ5dSJiGlQx X-Google-Smtp-Source: APBJJlGRvDi83Kngna+mFn5+ufMtiCVovIxN0DAwuRab+DYHHyauPQ62ZZfo22vSq8V7IXhcywtPpg== X-Received: by 2002:a05:622a:164b:b0:400:8082:2891 with SMTP id y11-20020a05622a164b00b0040080822891mr19138719qtj.13.1689106918495; Tue, 11 Jul 2023 13:21:58 -0700 (PDT) Received: from ahassick-Desktop.iol.unh.edu ([2606:4100:3880:1220:714e:bdff:bc40:f34f]) by smtp.gmail.com with ESMTPSA id s19-20020ac85293000000b004035c1062f8sm1470604qtn.10.2023.07.11.13.21.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Jul 2023 13:21:58 -0700 (PDT) From: Adam Hassick To: ci@dpdk.org Cc: alialnu@nvidia.com, aconole@redhat.com, Adam Hassick Subject: [PATCH v7 0/6] Community Lab Containers and Builder Engine Date: Tue, 11 Jul 2023 16:21:18 -0400 Message-Id: <20230711202124.1636317-1-ahassick@iol.unh.edu> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org This patch series contains a new version of the DPDK CI containers. The old version was tied very tightly to the Community Lab Infrastructure, so it was not suitable for general use. This version is designed to make adding new OSes or OS versions as easy as possible. The minimum functionality can easily be built on any system that can compile DPDK. It includes support for building containers for other non-native architectures (ex: arm containers on x86) and for baking ABI references into the images. The inventory file as added in this patch series defines what the community lab currently supports. If you want to build these yourself, don't try to do parallel Makefile builds on non-server hardware. Libabigail is built into the containers, and since it is not avaliable in all distros it is compiled from source for many targets. If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with the current settings, DPDK will be compiled twice for every target in the inventory file. This will become three times when DPDK main has ABI stability again. Due to how the containers are constructed, building on non-native architectures is especially expensive because DPDK is compiled with an emulated compiler in a qemu vm that only has one thread. --- v2: * Added poetry.lock * Remove duplicates in inventory file * Fix typo in base container template v3: * Libabigail is now not built by default unless DPDK_CI_CONTAINERS_BUILD_ABI=y * Made minimum podman version 4.0.0 due to dependencies on cache mounts v4: * Fix whitespace errors v5: * Updated container templates and inventory * Add support for OCI manifests * Update README to reflect changes v6: * Fix spelling errors and whitespace errors v7: * Add workaround to Arch template to install Python packages as root * Compare against v4 and polish the patchset Owen Hilyard (6): * Add README file for containers * Add pyproject file with required dependencies for building containers Adds a yaml file used to define what containers should be built, what packages the containers should have, what platforms to build for and how they should be tagged. This script will template out all of the Dockerfiles based on the definitions provided in the inventory using the jinja2 templating library. Adds a variety of extensible templates used to create the Dockerfiles for each target. All templates inherit from base, and most distros and distro families have their own templates that are used to define common functionality. Multiple versions of a distro may use a single template if they are similar enough (ex: fedora). Adds a container that can be used to run the python scripts to create the Dockerfiles for the CI containers. This removes a large number of python environment dependencies from the host requirements. The Makefile that can be used to build all of the container images using "make build", and can also be used to push them to a remote repository (for use in CI). containers/Makefile | 252 ++++++++++ containers/README.md | 178 +++++++ containers/container_builder.dockerfile | 26 + containers/template_engine/inventory.yaml | 470 ++++++++++++++++++ .../template_engine/inventory_schema.json | 221 ++++++++ containers/template_engine/make_dockerfile.py | 358 +++++++++++++ containers/template_engine/poetry.lock | 220 ++++++++ containers/template_engine/pyproject.toml | 21 + .../templates/containers.makefile.j2 | 73 +++ .../templates/containers/alpine.dockerfile.j2 | 3 + .../templates/containers/arch.dockerfile.j2 | 39 ++ .../templates/containers/base.dockerfile.j2 | 120 +++++ .../containers/centos8.dockerfile.j2 | 21 + .../containers/centos9.dockerfile.j2 | 17 + .../templates/containers/debian.dockerfile.j2 | 7 + .../containers/debian10.dockerfile.j2 | 3 + .../containers/debian11.dockerfile.j2 | 3 + .../debian11_arm_ipsec.dockerfile.j2 | 16 + .../containers/debian_bullseye.dockerfile.j2 | 3 + .../containers/debian_buster.dockerfile.j2 | 3 + .../templates/containers/fedora.dockerfile.j2 | 11 + .../containers/fedora36_clang.dockerfile.j2 | 7 + .../containers/fedora_clang.dockerfile.j2 | 8 + .../containers/fedora_coverity.dockerfile.j2 | 10 + .../containers/opensuse.dockerfile.j2 | 10 + .../containers/redhat_family.dockerfile.j2 | 5 + .../templates/containers/rhel.dockerfile.j2 | 16 + .../templates/containers/rhel7.dockerfile.j2 | 15 + .../templates/containers/rhel8.dockerfile.j2 | 15 + .../templates/containers/rhel9.dockerfile.j2 | 19 + .../templates/containers/rpm.dockerfile.j2 | 3 + .../templates/containers/ubuntu.dockerfile.j2 | 3 + .../containers/ubuntu20.04.dockerfile.j2 | 12 + .../containers/ubuntu22.04.dockerfile.j2 | 3 + .../containers/ubuntu_cross.dockerfile.j2 | 11 + .../containers/ubuntu_sve.dockerfile.j2 | 12 + 36 files changed, 2214 insertions(+) create mode 100644 containers/Makefile create mode 100644 containers/README.md create mode 100644 containers/container_builder.dockerfile create mode 100644 containers/template_engine/inventory.yaml create mode 100644 containers/template_engine/inventory_schema.json create mode 100755 containers/template_engine/make_dockerfile.py create mode 100644 containers/template_engine/poetry.lock create mode 100644 containers/template_engine/pyproject.toml create mode 100644 containers/template_engine/templates/containers.makefile.j2 create mode 100644 containers/template_engine/templates/containers/alpine.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/arch.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/base.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/centos8.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/centos9.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian10.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian11.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian_buster.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora_clang.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/opensuse.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/redhat_family.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel7.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel8.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel9.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rpm.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2