|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [PATCH v4 1/3] automation: Create Yocto docker images
On Thu, 17 Nov 2022, Bertrand Marquis wrote:
> Add containers suitable to run yocto kirkstone build based on ubuntu
> 22.04. It contains all packages required by Yocto and a checkout of the
> layers required to build Xen with Yocto.
>
> Add a generic docker image template to be used to automatically generate
> docker files for different configurations:
> - specific yocto version
> - different targets (qemu arm, arm64 and x86)
> - different host platforms (x86 or arm64)
>
> During a call to 'make all', only the images for the current host
> platform will be generated.
> If needed, images for an other host platform can be generated manually
> by calling the right make target (see make help).
>
> Add a build script to build and run xen on qemu using Yocto.
> The script supports arm32, arm64 and x86_64 and checks that dom0 is
> properly booting. At this stage this does not run any guest on top of
> dom0. The script is to be executed in one of the docker images to build
> and run a system using a Xen source tree.
>
> Signed-off-by: Bertrand Marquis <bertrand.marquis@xxxxxxx>
This series looks great and I think it is almost ready to be committed.
Only one comment from me below.
Also, I tested it successfully both building the containers, and also
the full gitlab-ci pipeline, hurray!
The only issue is that it works for qemuarm64 and qemuarm, but not
qemux86-64: the container build doesn't work. I am attaching the error
logs from 2 different builds on 2 different arm64 machines.
To be honest, if qemux86-64 on arm64 turns out to be an issue, I'd be
happy to take the series without it for now.
> ---
> Changes in v4:
> - Rework the system to have one dockerfile template from which make will
> generate the required dockerfiles for the wanted configuration
> - add support for different host architectures
> - Merge the generation system into one single dockerfile
> - Merge patches 1 and 2 in a single patch
> - Introduce CONTAINERS_EXTRA to have extra containers not built by
> default (for those not used by CI but useful to users)
> Changes in v3:
> - limit number of jobs in yocto by default to 8 and add --num-jobs
> option to the script to set a custom number of jobs
> - do not copy anymore the build-yocto.sh script inside the main image so
> that the current one in the repository is used when running
> Changes in v2:
> - add a --dump-log command line option to build-yocto.sh script to dump
> the logs if an error occurs.
> Changes in v1:
> - add --image command line argument to build-yocto.sh to allow building
> something different than xen-image-minimal.
> - modify dockerfile to have one layer per line and make it easier to add
> other. I kept the for loop to keep the number of docker steps lower
> - update commit message to warn that no guest are tested.
> - fix build-yocto script to properly return with an error if expect
> script ends up in timeout or EOF.
> ---
> automation/build/Makefile | 14 +-
> automation/build/yocto/build-yocto.sh | 349 +++++++++++++++++++++
> automation/build/yocto/yocto.dockerfile.in | 114 +++++++
> automation/build/yocto/yocto.inc | 41 +++
> 4 files changed, 516 insertions(+), 2 deletions(-)
> create mode 100755 automation/build/yocto/build-yocto.sh
> create mode 100644 automation/build/yocto/yocto.dockerfile.in
> create mode 100644 automation/build/yocto/yocto.inc
>
> diff --git a/automation/build/Makefile b/automation/build/Makefile
> index a4b2b85178cf..72a5335baec1 100644
> --- a/automation/build/Makefile
> +++ b/automation/build/Makefile
> @@ -1,13 +1,18 @@
>
> # the base of where these containers will appear
> REGISTRY := registry.gitlab.com/xen-project/xen
> -CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
> +CONTAINERS = $(filter-out yocto/%,$(subst .dockerfile,,$(wildcard
> */*.dockerfile)))
> +CONTAINERS_EXTRA =
> DOCKER_CMD ?= docker
>
> +include yocto/yocto.inc
I think it would be good to add a comment here to say how make could be
invoked for the yocto containers, for instance "make
yocto/kirkstone-qemuarm64" or "make yocto/kirkstone-qemuarm64-arm64v8"
and explain the difference. People tend to build locally individual
containers, rather than the whole set.
> help:
> @echo "Builds containers for building Xen based on different distros"
> @echo "To build one run 'make DISTRO/VERSION'. Available containers:"
> @$(foreach file,$(sort $(CONTAINERS)),echo ${file};)
> + @echo "Extra containers (not built using make all):"
> + @$(foreach file,$(sort $(CONTAINERS_EXTRA)),echo ${file};)
> @echo "To push container builds, set the env var PUSH"
>
> %: %.dockerfile ## Builds containers
> @@ -16,5 +21,10 @@ help:
> $(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
> fi
>
> -.PHONY: all
> +.PHONY: all clean
> all: $(CONTAINERS)
> +
> +# Remove generated dockerfiles for yocto
> +clean:
> + rm -f yocto/*.dockerfiles
> +
> diff --git a/automation/build/yocto/build-yocto.sh
> b/automation/build/yocto/build-yocto.sh
> new file mode 100755
> index 000000000000..d0c93dfaffe0
> --- /dev/null
> +++ b/automation/build/yocto/build-yocto.sh
> @@ -0,0 +1,349 @@
> +#!/bin/bash
> +#
> +# Yocto meta virtualization build and run script
> +#
> +# This script is building Yocto xen-image-minimal for qemu targets and run
> +# them using runqemu inside yocto to check that dom0 is booting properly
> +# The build is using a local xen source tree so that specific patches can be
> +# tested.
> +# In order to optimize the build time, a build cache is used so that only xen
> +# packages and its dependencies are rebuilt (qemu and final image mainly).
> +#
> +
> +# Directories
> +YOCTODIR="$HOME/yocto-layers"
> +CACHEDIR="$HOME/yocto-cache"
> +LOGDIR="$HOME/logs"
> +XENDIR="$HOME/xen"
> +BUILDDIR="$HOME/build"
> +
> +# what yocto bsp we support
> +TARGET_SUPPORTED="qemuarm qemuarm64 qemux86-64"
> +VERBOSE="n"
> +TARGETLIST=""
> +BUILDJOBS="8"
> +
> +# actions to do
> +do_clean="n"
> +do_build="y"
> +do_run="y"
> +do_localsrc="n"
> +do_dump="n"
> +build_result=0
> +
> +# layers to include in the project
> +build_layerlist="poky/meta poky/meta-poky poky/meta-yocto-bsp \
> + meta-openembedded/meta-oe meta-openembedded/meta-python \
> + meta-openembedded/meta-filesystems \
> + meta-openembedded/meta-networking meta-virtualization"
> +
> +# yocto image to build
> +build_image="xen-image-minimal"
> +
> +function print_progress() {
> + echo -n "$(date +%T) $*"
> +}
> +
> +function run_task() {
> + local task_name="$1"
> + local task_target="$2"
> +
> + task_log="${task_name//project_}-${task_target}"
> +
> + mkdir -p "${LOGDIR}"
> + print_progress
> + echo -n "${task_name//project_} ${task_target}: "
> + if [ "${VERBOSE}" = "n" ]; then
> + $@ > "${LOGDIR}/${task_log}.log" 2>&1
> + else
> + $@ 2>&1 | tee "${LOGDIR}/${task_log}.log"
> + fi
> +
> + if [ ${?} -ne 0 ]; then
> + echo "Error"
> + build_result=$((build_result+1))
> + if [ "${do_dump}" = "y" ]; then
> + echo
> + echo "############ LOGS-START ############"
> + cat "${LOGDIR}/${task_log}.log"
> + echo "############ LOGS-END ############"
> + echo
> + fi
> + return 1
> + else
> + echo "OK"
> + return 0
> + fi
> +}
> +
> +function project_create() {
> + target="${1:?}"
> + destdir="${BUILDDIR}/${target}"
> +
> + (
> + # init yocto project
> + source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
> +
> + # add needed layers
> + for layer in ${build_layerlist}; do
> + bitbake-layers add-layer ${YOCTODIR}/${layer} || exit 1
> + done
> + ) || return 1
> +
> + # customize project configuration
> + cat <<EOF >> "${destdir}/conf/local.conf"
> +# Yocto BSP
> +MACHINE = "${target}"
> +
> +# Use local cache to reuse previous builds results
> +SSTATE_DIR = "${CACHEDIR}/sstate-cache"
> +DL_DIR = "${CACHEDIR}/downloads"
> +
> +# Enable xen and virtualization
> +DISTRO_FEATURES = " virtualization xen ipv4"
> +
> +# Speed up run by not generating ssh host keys
> +IMAGE_INSTALL:append:pn-xen-image-minimal = " ssh-pregen-hostkeys"
> +
> +# Save some disk space
> +INHERIT += "rm_work"
> +
> +# Reduce number of jobs
> +BB_NUMBER_THREADS="${BUILDJOBS}"
> +
> +EOF
> +
> + if [ "${do_localsrc}" = "y" ]; then
> + XENVERS=$(grep -e "^XEN_REL" \
> + "${YOCTODIR}"/meta-virtualization/recipes-extended/xen/xen_*.bb \
> + 2> /dev/null | tr -d ' ' | tr -d '?' | tr -d '"' \
> + | sed -e "s/.*=//" | sort -V | tail -n 1)
> +
> + XENBASE=$(dirname "$(realpath -m "${XENDIR}")")
> + XENSUB=$(basename "$(realpath -m "${XENDIR}")")
> +
> + cat <<EOF >> "${destdir}/conf/local.conf"
> +# Use local sources for xen and xen-tools
> +FILESEXTRAPATHS:prepend:pn-xen := "${XENBASE}:"
> +FILESEXTRAPATHS:prepend:pn-xen-tools := "${XENBASE}:"
> +
> +SRC_URI:pn-xen = "file://${XENSUB}/;subdir=local-xen/"
> +SRC_URI:pn-xen-tools = "file://${XENSUB}/;subdir=local-xen/"
> +
> +PREFERRED_VERSION:pn-xen = "${XENVERS}%"
> +PREFERRED_VERSION:pn-xen-tools = "${XENVERS}%"
> +
> +S:pn-xen = "\${WORKDIR}/local-xen/${XENSUB}"
> +S:pn-xen-tools = "\${WORKDIR}/local-xen/${XENSUB}"
> +
> +SRCREV:pn-xen = "\${AUTOREV}"
> +SRCREV:pn-xen-tools = "\${AUTOREV}"
> +
> +SRCPV:pn-xen = "1"
> +SRCPV:pn-xen-tools = "1"
> +
> +# Disable all QA errors as the recipe is not up to date with changes in Xen
> +# when we use local sources
> +ERROR_QA:pn-xen = "arch"
> +ERROR_QA:pn-xen-tools = "arch"
> +
> +EOF
> + fi
> +}
> +
> +function project_build() {
> + target="${1:?}"
> + destdir="${BUILDDIR}/${target}"
> +
> + (
> + source ${YOCTODIR}/poky/oe-init-build-env "${destdir}"
> +
> + bitbake "${build_image}" || exit 1
> + ) || return 1
> +}
> +
> +function project_clean() {
> + target="${1:?}"
> + destdir="${BUILDDIR}/${target}"
> +
> + rm -rf "${destdir}"
> +}
> +
> +function project_run() {
> + target="${1:?}"
> + destdir="${BUILDDIR}/${target}"
> + (
> + source ${YOCTODIR}/poky/oe-init-build-env "${destdir}" > /dev/null
> 2>&1
> +
> + /usr/bin/expect <<EOF
> +set timeout 100
> +spawn bash -c "runqemu serialstdio nographic slirp"
> +
> +expect_after {
> + -re "(.*)\r" {
> + exp_continue
> + }
> + timeout {send_user "ERROR-Timeout!\n"; exit 1}
> + eof {send_user "ERROR-EOF!\n"; exit 1}
> +}
> +
> +# wait initial login
> +expect -re ".* login: "
> +send "root\r"
> +expect -re "root@.*# "
> +
> +EOF
> + exit $?
> + ) || return 1
> +}
> +
> +function help() {
> + cat <<EOF
> +Usage: ${0} [TARGET1] [TARGET2]
> +
> +This script is build the yocto xen-image-minimal for different qemu targets
> +and is running it after.
> +Without any target specified, all supported targets are done.
> +
> +Options:
> + -h, --help Print this help
> + -v, --verbose Verbose build
> + --list-target List supported targets
> + --clean Clean existing project before starting
> + --no-build Do not build (to run an already built project)
> + --no-run Do not run
> + --num-jobs=NUM Define the number of parallel jobs in Yocto.
> + Default: ${BUILDJOBS}
> + --dump-log On error, dump the logs on the console
> + --image=IMG Yocto image or package to build
> + Default: xen-image-minimal
> + --xen-dir=DIR path to xen hypervisor source tree
> + if not provide, normal yocto version of xen is built
> + Default: ${XENDIR}
> + --out-dir=DIR directory where to create the projectss
> + Default: ${BUILDDIR}
> + --log-dir=DIR directory to store logs
> + Default: ${LOGDIR}
> + --cache-dir=DIR directory where to take and store build cache
> + Default: ${CACHEDIR}
> + --layer-dir=DIR directory containing the checkout of yocto layers
> + Default: ${YOCTODIR}
> +EOF
> +}
> +
> +for OPTION in "$@"
> +do
> + case ${OPTION} in
> + -h|--help)
> + help
> + exit 0
> + ;;
> + -v|--verbose)
> + VERBOSE="y"
> + ;;
> + --list-targets)
> + echo "${TARGET_SUPPORTED}"
> + exit 0
> + ;;
> + --clean)
> + do_clean="y"
> + ;;
> + --no-build)
> + do_build="n"
> + ;;
> + --no-run)
> + do_run="n"
> + ;;
> + --dump-log)
> + do_dump="y"
> + ;;
> + --num-jobs=*)
> + BUILDJOBS="${OPTION#*=}"
> + ;;
> + --image=*)
> + build_image="${OPTION#*=}"
> + ;;
> + --xen-dir=*)
> + XENDIR="${OPTION#*=}"
> + if [ ! -e "${XENDIR}/xen/Makefile" ]; then
> + echo "No Xen source tree in ${XENDIR}"
> + exit 1
> + fi
> + do_localsrc="y"
> + ;;
> + --out-dir=*)
> + BUILDDIR="${OPTION#*=}"
> + ;;
> + --log-dir=*)
> + LOGDIR="${OPTION#*=}"
> + ;;
> + --cache-dir=*)
> + CACHEDIR="${OPTION#*=}"
> + ;;
> + --layer-dir=*)
> + YOCTODIR="${OPTION#*=}"
> + ;;
> + --*)
> + echo "Invalid option ${OPTION}"
> + help
> + exit 1
> + ;;
> + *)
> + if echo "${TARGET_SUPPORTED}" | grep -q -w "${OPTION}"; then
> + TARGETLIST="${TARGETLIST} ${OPTION}"
> + else
> + echo "Unsupported target ${OPTION}"
> + exit 1
> + fi
> + ;;
> + esac
> +done
> +
> +# if no target is specified build all targets
> +if [ -z "${TARGETLIST}" ]; then
> + TARGETLIST="${TARGET_SUPPORTED}"
> +fi
> +
> +mkdir -p "${CACHEDIR}"
> +mkdir -p "${LOGDIR}"
> +mkdir -p "${BUILDDIR}"
> +
> +# Make sure we have an absolute path
> +YOCTODIR=$(realpath -m "${YOCTODIR}")
> +CACHEDIR=$(realpath -m "${CACHEDIR}")
> +BUILDDIR=$(realpath -m "${BUILDDIR}")
> +LOGDIR=$(realpath -m "${LOGDIR}")
> +if [ "${do_localsrc}" = "y" ]; then
> + XENDIR=$(realpath -m "${XENDIR}")
> +fi
> +
> +# Check that we have all the layers we need
> +for f in ${build_layerlist}; do
> + if [ ! -f "${YOCTODIR}/${f}/conf/layer.conf" ]; then
> + echo "Layer ${f} missing in ${YOCTODIR}"
> + exit 1
> + fi
> +done
> +
> +for f in ${TARGETLIST}; do
> + if [ "${do_clean}" = "y" ]; then
> + run_task project_clean ${f}
> + fi
> + if [ ! -f ${BUILDDIR}/${f}/conf/local.conf ]; then
> + run_task project_create ${f}
> + fi
> + if [ -f ${BUILDDIR}/${f}/conf/local.conf ]; then
> + if [ "${do_build}" = "y" ]; then
> + run_task project_build ${f}
> + fi
> + if [ "${do_run}" = "y" ]; then
> + run_task project_run ${f}
> + fi
> +
> + fi
> +done
> +
> +print_progress "Build Complete (${build_result} errors)"
> +echo
> +exit ${build_result}
> +
> diff --git a/automation/build/yocto/yocto.dockerfile.in
> b/automation/build/yocto/yocto.dockerfile.in
> new file mode 100644
> index 000000000000..5350bb2b87b7
> --- /dev/null
> +++ b/automation/build/yocto/yocto.dockerfile.in
> @@ -0,0 +1,114 @@
> +# Docker file to create an environment to build yocto with virtualization
> +#
> +# Arguments that can be passed during image creation using --build-arg:
> +# "host_uid=$(id -u)": to use current user uid for build user in the image
> +# "host_gid=$(id -g)": to use current user gid for build user in the image
> +# "ubuntu_version=VERS": to select the ubuntu version number
> +
> +# Use standard ubuntu minimal
> +ARG ubuntu_version=22.04
> +From ##DOCKERPLAT##ubuntu:$ubuntu_version AS base
> +LABEL maintainer.name="The Xen Project " \
> + maintainer.email="xen-devel@xxxxxxxxxxxxxxxxxxxx"
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +
> +# Install minimal ubuntu requirements for yocto and other tools we need
> +# See
> https://docs.yoctoproject.org/4.0.1/brief-yoctoprojectqs/index.html#build-host-packages
> +RUN apt-get update && \
> + apt-get --quiet --yes install \
> + gawk \
> + wget \
> + git \
> + diffstat \
> + unzip \
> + texinfo \
> + gcc \
> + build-essential \
> + chrpath \
> + socat \
> + cpio \
> + python3 \
> + python3-pip \
> + python3-pexpect \
> + xz-utils \
> + debianutils \
> + iputils-ping \
> + python3-git \
> + python3-jinja2 \
> + libegl1-mesa \
> + libsdl1.2-dev \
> + python3-subunit \
> + mesa-common-dev \
> + zstd \
> + liblz4-tool \
> + file \
> + vim \
> + bison \
> + expect \
> + locales \
> + liblz4-tool \
> + zstd \
> + openssl \
> + libssl3 \
> + ca-certificates \
> + && \
> + apt-get autoremove -y && \
> + apt-get clean && \
> + rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
> +
> +# Use bash as shell
> +RUN rm /bin/sh && ln -s bash /bin/sh
> +
> +# Fix local for yocto
> +RUN locale-gen en_US.UTF-8 && update-locale LC_ALL=en_US.UTF-8 \
> + LANG=en_US.UTF-8
> +ENV LANG en_US.UTF-8
> +ENV LC_ALL en_US.UTF-8
> +
> +# Create a user for the build (we don't want to build as root)
> +ENV USER_NAME docker-build
> +ARG host_uid=1000
> +ARG host_gid=1000
> +RUN groupadd -g $host_gid $USER_NAME && \
> + useradd -g $host_gid -m -s /bin/bash -u $host_uid $USER_NAME
> +
> +# Switch to our user instead of root and start in its home
> +USER $USER_NAME
> +WORKDIR /home/$USER_NAME
> +
> +# Create needed directories
> +RUN mkdir -p /home/$USER_NAME/yocto-layers \
> + /home/$USER_NAME/yocto-cache \
> + /home/$USER_NAME/logs \
> + /home/$USER_NAME/bin \
> + /home/$USER_NAME/xen && \
> + chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
> +
> +# clone yocto repositories we need
> +RUN for rep in \
> + https://github.com/openembedded/meta-openembedded \
> + https://git.yoctoproject.org/poky \
> + https://git.yoctoproject.org/meta-virtualization \
> + ; do \
> + git -C /home/$USER_NAME/yocto-layers \
> + clone -b ##YOCTOVERSION## --single-branch $rep; \
> + done
> +
> +# The builder stage is building an initial cache state that we include in the
> +# final image
> +From base AS builder
> +
> +# This step can take one to several hours depending on your download bandwith
> +# and the speed of your computer
> +COPY ./build-yocto.sh /
> +RUN /build-yocto.sh --dump-log ##YOCTOTARGET##
> +
> +From base
> +
> +# Only copy the cache status
> +COPY --from=builder /home/$USER_NAME/yocto-cache
> /home/$USER_NAME/yocto-cache/.
> +
> +LABEL maintainer.name="The Xen Project " \
> + maintainer.email="xen-devel@xxxxxxxxxxxxxxxxxxxx"
> +
> diff --git a/automation/build/yocto/yocto.inc
> b/automation/build/yocto/yocto.inc
> new file mode 100644
> index 000000000000..04076bc8d174
> --- /dev/null
> +++ b/automation/build/yocto/yocto.inc
> @@ -0,0 +1,41 @@
> +# This makefile generates the docker files for Yocto builds
> +# The containers for the current architecture are the one built using make
> all
> +# To build containers for a different architecture, you need to call make for
> +# the image you want explicitely
> +# The containers are named this way:
> +# YOCTOVERSION-TARGET for x86_64 hosts
> +# YOCTOVERSION-TARGET-arm64v8 for arm64 hosts
> +
> +# Yocto versions we are currently using
> +YOCTO_VERSION = kirkstone
> +
> +# Yocto BSPs we want to build for
> +YOCTO_TARGETS = qemuarm64 qemuarm qemux86-64
> +
> +# Supported Host platforms (host architecture specific ones)
> +YOCTO_HOSTS = amd64 arm64v8
> +
> +# Architecture we want to use in gitlab CI (depends on runners arch)
> +CI_ARCH = arm64v8
> +
> +define GEN_DOCKER
> +# Make all is generating what we want in the CI
> +ifeq ($(CI_ARCH),$(3))
> +CONTAINERS += yocto/$(1)-$(2)$(4)
> +else
> +CONTAINERS_EXTRA += yocto/$(1)-$(2)$(4)
> +endif
> +
> +yocto/$(1)-$(2)$(4).dockerfile: yocto/yocto.dockerfile.in
> + echo > $$@
> + cat $$< | \
> + sed -e "s,##YOCTOVERSION##,$(1),g" | \
> + sed -e "s,##YOCTOTARGET##,$(2),g" | \
> + sed -e "s,##DOCKERPLAT##,$(3)/,g" > $$@
> +
> +endef
> +
> +$(eval $(foreach vers,$(YOCTO_VERSION),\
> + $(foreach tar,$(YOCTO_TARGETS),\
> + $(foreach hst,$(YOCTO_HOSTS),\
> + $(call GEN_DOCKER,$(vers),$(tar),$(hst),$(if $(filter
> amd64,$(hst)),,-$(hst)))))))
> --
> 2.25.1
> Attachment:
build.log Attachment:
build2.log
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |