From patchwork Fri Aug 11 20:00:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adam Hassick X-Patchwork-Id: 130214 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DDCC543038; Fri, 11 Aug 2023 22:03:08 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D801A4325F; Fri, 11 Aug 2023 22:03:08 +0200 (CEST) Received: from mail-qt1-f195.google.com (mail-qt1-f195.google.com [209.85.160.195]) by mails.dpdk.org (Postfix) with ESMTP id 17CFB40E03 for ; Fri, 11 Aug 2023 22:03:06 +0200 (CEST) Received: by mail-qt1-f195.google.com with SMTP id d75a77b69052e-40fbf360a9cso16297971cf.3 for ; Fri, 11 Aug 2023 13:03:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1691784185; x=1692388985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EVAI7GINR2x3hwMo9QPtJKzfBALu3j8uWPwKx7d9/GU=; b=UgYFC1eC8DJ4x6n3xGsle3Z3yFaFt9ksVdyiXn7O2932PgxpO8EshXUvNl8L4hAfAf Zle3rjACXOLiCaBz3NVZDCQ+sUBxRmdG69Er7iCL4TCDml97+AQFVdU+J5RuQWGPFfVp 8qmmJz/8ksJdzZLASWQZGTba7wedsDwBRuJnE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691784185; x=1692388985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EVAI7GINR2x3hwMo9QPtJKzfBALu3j8uWPwKx7d9/GU=; b=OD9Hn8IVXVbNe+yiH1G0Umyy409pJONBr5HifZdc8y9vt/oOCLsZZXqzc7f05Qs0pU qZbc5u3puyryPgOM7wdGWPFRG+FIxyp0Kh61OGGwgEMg3GDnYMGHLJl2v1WkkJswOKZI nSJJ7q6tEi2oWlVQ6mAHhLjVFlOodsaXARbPMj7AN7AK8JHvqOFuv40DY06sOJJ8JPNq VK7lI2Zo6xHiHBm3BZsuh6xYHQ8n1AVDkJeZkNSBMR6QUrUEyQ3JdKGiGVdO0wXaAlmc 4A4Pv8gNmjsAyDh0Xf/hevh21pwi0OQqn8DSfeiZ81qnyXbJIjFIjRWCyUJx/JdDpPs5 QaXg== X-Gm-Message-State: AOJu0YwN+30wYH+Xi3v5E6CKhQjc5GZSo2e0zDoresAkpuLlwBLqInzO orZ60Zwl/O+Dzr5bVPWr4DA7JjRheAsaIVy76TN/scv1YTgXQtlWS/qGZKqobAdV0DKnDQr9lRJ nVLVmB7Q/6OKSi3IFBdZ7KJpFVbNPYuZgJO8V22DWMt86eYZ9YHMp1r9BcEKTaA/UlrjrJA== X-Google-Smtp-Source: AGHT+IE2ap0LUhVXmHTkWd9htK5EvGXQhuisO+vdl1ah12vuo4y0qr7RVENCavFQLkBCsNrlosSUcw== X-Received: by 2002:a05:622a:490:b0:408:483e:5f6e with SMTP id p16-20020a05622a049000b00408483e5f6emr3151176qtx.29.1691784185292; Fri, 11 Aug 2023 13:03:05 -0700 (PDT) Received: from pogmachine2.loudonlune.net ([216.212.51.182]) by smtp.gmail.com with ESMTPSA id f19-20020ac84993000000b004053bcffe49sm1399874qtq.9.2023.08.11.13.03.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Aug 2023 13:03:05 -0700 (PDT) From: Adam Hassick To: ci@dpdk.org Cc: aconole@redhat.com, alialnu@nvidia.com, Owen Hilyard , Adam Hassick Subject: [PATCH v9 6/6] containers/Makefile: Makefile to automate builds Date: Fri, 11 Aug 2023 16:00:18 -0400 Message-ID: <20230811200018.5650-7-ahassick@iol.unh.edu> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230811200018.5650-1-ahassick@iol.unh.edu> References: <20230811200018.5650-1-ahassick@iol.unh.edu> MIME-Version: 1.0 X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org From: Owen Hilyard The Makefile that can be used to build all of the container images using "make build", and can also be used to push them to a remote repository (for use in CI). Signed-off-by: Owen Hilyard Signed-off-by: Adam Hassick --- containers/Makefile | 258 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 258 insertions(+) create mode 100644 containers/Makefile diff --git a/containers/Makefile b/containers/Makefile new file mode 100644 index 0000000..047a208 --- /dev/null +++ b/containers/Makefile @@ -0,0 +1,258 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2023 University of New Hampshire + +# Recommended Resources: +# All: +# ~100 GB of disk space: These are not minimal containers, they +# have full distros in them minus the kernel. +# With ABI images, expect 10 GB per distro per +# platform. Without, it's closer to 2 GB per distro +# per platform. +# +# No ABI: +# Serial Build (make -j 1): Should run on a laptop. +# Parallel build (make -j $(nproc)): 1 GB of free ram per job should be good enough. +# ABI: +# Time: If you are building more than four distros, or you have one distro that is built under +# emulation (x86 host + arm container or the inverse), it may be best to run it overnight. +# Some versions of qemu + podman will try to compile with a single thread under emulation, +# which can cause what should be an hour-long compile to turn into a 8+ hour compile. +# Serial Build (make -j 1): 6 GB of free ram should be fine +# Parallel build (make -j $(nproc)): 6 GB of free ram per job. You may be compiling ~20 versions of DPDK at the same time, with some under emulation. +# + + +############ Arguments ############# +# Set to 'Y' to override distro detection +DPDK_CI_CONTAINERS_ON_RHEL?=$(shell (test -f /etc/redhat-release && grep -q 'Red Hat Enterprise Linux' /etc/redhat-release && echo 'Y') || echo 'N') + +# If set to 'Y' and any container is detected as unbuildable, fail the build. +DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE?=N + +# If set to 'Y', build the ABIs for specificed DPDK versions and embed them +# in the container +DPDK_CI_CONTAINERS_BUILD_ABI?=N + +# If set to 'Y', the "latest" tag for all images will be omitted. Intended for testing changes to your inventory. +DPDK_CI_CONTAINERS_NO_LATEST_TAG?=N + +# If set to 'Y', will build containers that rely on the Coverity Scan tool. +DPDK_CI_CONTAINERS_COVERITY?=N + +# The path to Coverity Scan binaries. These will be installed inside the container. +# This field is only required if the Coverity flag is enabled. +# DPDK_CI_CONTAINERS_COVERITY_PATH?=/opt/dpdklab/coverity + +# If set to a non-empty value, overrides the auto-generated date tag with the value. +# DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE=70-01-01 + +# If set to 'Y', build libabigail from source for distros that do not have it. +# If DPDK_CI_CONTAINERS_BUILD_ABI='Y', then this is always enabled. +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=Y +else +DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL?=N +endif + +# If set to an integer value, this will restrict the count of Ninja workers performing the ABI build to the given integer. +# The argument is benign if DPDK_CI_CONTANERS_BUILD_ABI is 'N'. +# DPDK_CI_CONTAINERS_NINJA_WORKERS?=16 + +# If set to 'Y', only build containers matching the host architecture +DPDK_CI_CONTAINERS_ONLY_HOST_ARCH?=N + +# If set to 'Y', assumes only host arch and disables the push_manifests target. +DPDK_CI_CONTAINERS_IS_BUILDER?=N + +# Used to set the python interpreter +DPDK_CI_CONTAINERS_PYTHON3_CMD?=python3.8 + +# The base program to use to build individual containers. +DPDK_CI_CONTAINER_BUILDER_PROGRAM?=podman + +# The url of the git repository to pull libabigail from +DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL?=git://sourceware.org/git/libabigail.git + +# The url of the git repository to pull DPDK from +DPDK_CI_CONTAINERS_DPDK_CLONE_URL?=https://dpdk.org/git/dpdk + +# The url of the git repository to pull the DPDK CI scripts from. +DPDK_CI_CONTAINERS_DPDK_CI_CLONE_URL?=https://dpdk.org/git/tools/dpdk-ci + +# The url of the git repository to pull DPDK stable from +DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL?=https://dpdk.org/git/dpdk-stable + +# The tag to apply to the built container builder image +DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG?=dpdk_ci_container_builder + +# Extra arguments to add to the push command, can be used for credentials +DPDK_CI_CONTAINERS_EXTRA_PUSH_ARGS?= + +# Provide the hostname of the registry to push up to +DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME?=localhost + +# The path to a directory to be recursively copied to $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY). Not used if unset. +#DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH?= + +DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY?=$(CURDIR)/container_context +############ End Arguments ############# + +############ Internal Variables ############# +SCRIPT_ARGS=--output-dir "$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)" + +ifeq ($(DPDK_CI_CONTAINERS_ON_RHEL), Y) + SCRIPT_ARGS +=--rhel +endif + +ifeq ($(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE), Y) + SCRIPT_ARGS +=--fail-on-unbuildable +endif + +ifeq ($(DPDK_CI_CONTAINERS_BUILD_ABI), Y) + SCRIPT_ARGS +=--build-abi +endif + +ifeq ($(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL), Y) + SCRIPT_ARGS +=--build-libabigail +endif + +ifeq ($(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH), Y) + SCRIPT_ARGS +=--host-arch-only +endif + +ifeq ($(DPDK_CI_CONTAINERS_NO_LATEST_TAG), Y) + SCRIPT_ARGS +=--omit-latest +endif + +ifeq ($(DPDK_CI_CONTAINERS_IS_BUILDER), Y) + SCRIPT_ARGS +=--builder-mode +endif + +ifneq ($(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE),) + SCRIPT_ARGS +=--date $(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE) +endif + +ifneq ($(DPDK_CI_CONTAINERS_NINJA_WORKERS),) + SCRIPT_ARGS +=--ninja-workers $(DPDK_CI_CONTAINERS_NINJA_WORKERS) +endif + +ifeq ($(DPDK_CI_CONTAINERS_COVERITY), Y) + SCRIPT_ARGS +=--coverity +endif + +SCRIPT_DIRECTORY=$(CURDIR)/template_engine + +LIBABIGAIL_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/libabigail +DPDK_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk +DPDK_CI_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-ci +DPDK_STABLE_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/dpdk-stable + +DOCKERFILE=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/Dockerfile + +MAKE_DOCKERFILE_SCRIPT=$(SCRIPT_DIRECTORY)/make_dockerfile.py +MAKE_DOCKERFILE_COMMAND=$(DPDK_CI_CONTAINERS_PYTHON3_CMD) $(MAKE_DOCKERFILE_SCRIPT) + +TEMPLATE_FILE_DIRECTORY=$(SCRIPT_DIRECTORY)/templates +DOCKER_CONTAINER_TEMPLATE_DIR=$(TEMPLATE_FILE_DIRECTORY)/containers + +INVENTORY_FILE=$(SCRIPT_DIRECTORY)/inventory.yaml + +EXTRA_SCRIPTS_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/scripts + +COVERITY_DIRECTORY=$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/coverity + +GIT_FETCH_ARGS=--all --tags + +CONTAINER_BUILDER_DOCKERFILE=$(CURDIR)/container_builder.dockerfile + +export +############ End Internal Variables ############# + +.PHONY: build push push_images push_manifests make_docker_files_in_container build_builder_container docker_deps extra_scripts libabigail dpdk dpdk-ci coverity clean_container_files clean + +build: make_docker_files_in_container external_files + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) build_all + +push_images: make_docker_files_in_container external_files + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_image_all + +push_manifests: make_docker_files_in_container + $(MAKE) -C $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) push_manifest_all + +push: push_images push_manifests + +make_docker_files_in_container: build_builder_container $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) run --rm -v $(CURDIR):/container_workspace:z -v $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY):/container_workspace/container_context:z \ + -e DPDK_CI_CONTAINERS_ON_RHEL=$(DPDK_CI_CONTAINERS_ON_RHEL) \ + -e DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL=$(DPDK_CI_CONTAINERS_BUILD_LIBABIGAIL) \ + -e DPDK_CI_CONTAINERS_BUILD_ABI=$(DPDK_CI_CONTAINERS_BUILD_ABI) \ + -e DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE=$(DPDK_CI_CONTAINERS_FAIL_ON_UNBUILDABLE) \ + -e DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME='$(DPDK_CI_CONTAINERS_REGISTRY_HOSTNAME)' \ + -e DPDK_CI_CONTAINERS_ONLY_HOST_ARCH='$(DPDK_CI_CONTAINERS_ONLY_HOST_ARCH)' \ + -e DPDK_CI_CONTAINERS_NO_LATEST_TAG='$(DPDK_CI_CONTAINERS_NO_LATEST_TAG)' \ + -e DPDK_CI_CONTAINERS_IS_BUILDER='$(DPDK_CI_CONTAINERS_IS_BUILDER)' \ + -e DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE='$(DPDK_CI_CONTAINERS_DATE_TAG_OVERRIDE)' \ + -e DPDK_CI_CONTAINERS_NINJA_WORKERS='$(DPDK_CI_CONTAINERS_NINJA_WORKERS)' \ + -e DPDK_CI_CONTAINERS_COVERITY='$(DPDK_CI_CONTAINERS_COVERITY)' \ + $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) make docker_deps + +external_files: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) extra_scripts coverity + +build_builder_container: + $(DPDK_CI_CONTAINER_BUILDER_PROGRAM) build -f $(CONTAINER_BUILDER_DOCKERFILE) -t $(DPDK_CI_CONTAINERS_CONTAINER_BUILDER_TAG) $(CURDIR) + +docker_deps: $(DOCKERFILE) deps extra_scripts + chmod 666 $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)/*.dockerfile + +$(DOCKERFILE): $(INVENTORY_FILE) $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) deps $(MAKE_DOCKERFILE_SCRIPT) $(shell find $(DOCKER_CONTAINER_TEMPLATE_DIR) -type f) + cd $(SCRIPT_DIRECTORY) && $(MAKE_DOCKERFILE_COMMAND) $(SCRIPT_ARGS) + +deps: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) dpdk dpdk-ci libabigail dpdk-stable + +extra_scripts: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(EXTRA_SCRIPTS_DIRECTORY) $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH) +ifdef DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH + cp -pur $(DPDK_CI_CONTAINERS_EXTRA_SCRIPTS_PATH)/* $(EXTRA_SCRIPTS_DIRECTORY) +endif + +# Clone libabigail source +libabigail: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) + git -C $(LIBABIGAIL_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_LIBABIGAIL_CLONE_URL) $(LIBABIGAIL_DIRECTORY) + git -C $(LIBABIGAIL_DIRECTORY) describe --tags `git -C $(LIBABIGAIL_DIRECTORY) rev-list --tags --max-count=1` | xargs -n 1 git -C $(LIBABIGAIL_DIRECTORY) checkout + +# Clone DPDK source +dpdk: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) + git -C $(DPDK_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_CLONE_URL) $(DPDK_DIRECTORY) + +# Clone DPDK CI tools source. +dpdk-ci: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) + git -C $(DPDK_CI_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_CI_CLONE_URL) $(DPDK_CI_DIRECTORY) + +# Copy the Coverity Scan binaries from a source into the context directory. +coverity: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) $(COVERITY_DIRECTORY) +ifeq ($(DPDK_CI_CONTAINERS_COVERITY),Y) +ifndef DPDK_CI_CONTAINERS_COVERITY_PATH + exit 1 # The coverity path must be set if building Coverity containers is enabled. +else + cp -pur $(DPDK_CI_CONTAINERS_COVERITY_PATH)/* $(COVERITY_DIRECTORY) +endif +else + # Coverity is disabled for this run. +endif + +dpdk-stable: $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) + git -C $(DPDK_STABLE_DIRECTORY) fetch $(GIT_FETCH_ARGS) || git clone $(DPDK_CI_CONTAINERS_DPDK_STABLE_CLONE_URL) $(DPDK_STABLE_DIRECTORY) + +$(EXTRA_SCRIPTS_DIRECTORY): + mkdir -p $(EXTRA_SCRIPTS_DIRECTORY) + +$(COVERITY_DIRECTORY): + mkdir -p $(COVERITY_DIRECTORY) + +$(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY): + mkdir -p $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY) + +clean_container_files: + rm $(DOCKERFILE) + +clean: + rm -rf $(DPDK_CI_CONTAINERS_CONTEXT_DIRECTORY)