Re: [PATCH v2 0/3] Yocto Gitlab CI

[Bertrand Marquis <Bertrand.Marquis@xxxxxxx>]

Hi Stefano,

> On 19 Oct 2022, at 01:02, Stefano Stabellini <sstabellini@xxxxxxxxxx> wrote:
> 
> On Mon, 17 Oct 2022, Stefano Stabellini wrote:
>> It should be
>> 
>> BB_NUMBER_THREADS="2"
>> 
>> but that worked! Let me a couple of more tests.
> 
> I could run successfully a Yocto build test with qemuarm64 as target in
> gitlab-ci, hurray! No size issues, no build time issues, everything was
> fine. See:
> https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/3193051236
> https://gitlab.com/xen-project/people/sstabellini/xen/-/jobs/3193083119

Awesome, this is quite fast :-)

> 
> I made the appended changes in top of this series.
> 
> - I pushed registry.gitlab.com/xen-project/xen/yocto:kirkstone and
>  registry.gitlab.com/xen-project/xen/yocto:kirkstone-qemuarm64

This should already be handle by the Makefile using PUSH or did
you have to modify something ?

> - for the gitlab-ci runs, we need to run build-yocto.sh from the copy in
>  xen.git, not from a copy stored inside a container

Ok

> - when building the kirkstone-qemuarm64 container the first time
>  (outside of gitlab-ci) I used COPY and took the script from the local
>  xen.git tree

Ok

> - after a number of tests, I settled on: BB_NUMBER_THREADS="8" more than
>  this and it breaks on some workstations, please add it

I will put this by default and leave a command line argument to have a solution 
to change this.

> - I am running the yocto build on arm64 so that we can use the arm64
>  hardware to do it in gitlab-ci

I tested this when I made the patches and this works for arm64, arm32 and x86 
targets on an arm64 machine so go for it.

> 
> Please feel free to incorporate these changes in your series, and add
> corresponding changes for the qemuarm32 and qemux86 targets.

Will do and I will also add a patch to create the build.yaml entries.

> 
> I am looking forward to it! Almost there!

Me to :-)

Thanks a lot for the testing and the review.

Cheers
Bertrand

> 
> Cheers,
> 
> Stefano
> 
> 
> diff --git a/automation/build/yocto/build-yocto.sh 
> b/automation/build/yocto/build-yocto.sh
> index 0d31dad607..16f1dcc0a5 100755
> --- a/automation/build/yocto/build-yocto.sh
> +++ b/automation/build/yocto/build-yocto.sh
> @@ -107,6 +107,9 @@ IMAGE_INSTALL:append:pn-xen-image-minimal = " 
> ssh-pregen-hostkeys"
> # Save some disk space
> INHERIT += "rm_work"
> 
> +# Reduce number of jobs
> +BB_NUMBER_THREADS="8"
> +
> EOF
> 
>     if [ "${do_localsrc}" = "y" ]; then
> diff --git a/automation/build/yocto/kirkstone-qemuarm64.dockerfile 
> b/automation/build/yocto/kirkstone-qemuarm64.dockerfile
> index f279a7af92..aea3fc1f3e 100644
> --- a/automation/build/yocto/kirkstone-qemuarm64.dockerfile
> +++ b/automation/build/yocto/kirkstone-qemuarm64.dockerfile
> @@ -16,7 +16,8 @@ ARG target=qemuarm64
> 
> # This step can take one to several hours depending on your download bandwith
> # and the speed of your computer
> -RUN /home/$USER_NAME/bin/build-yocto.sh --dump-log $target
> +COPY ./build-yocto.sh /
> +RUN /build-yocto.sh --dump-log $target
> 
> FROM $from_image
> 
> diff --git a/automation/build/yocto/kirkstone.dockerfile 
> b/automation/build/yocto/kirkstone.dockerfile
> index 367a7863b6..ffbd91aa90 100644
> --- a/automation/build/yocto/kirkstone.dockerfile
> +++ b/automation/build/yocto/kirkstone.dockerfile
> @@ -84,9 +84,6 @@ RUN mkdir -p /home/$USER_NAME/yocto-layers \
>              /home/$USER_NAME/xen && \
>     chown $USER_NAME.$USER_NAME /home/$USER_NAME/*
> 
> -# Copy the build script
> -COPY build-yocto.sh /home/$USER_NAME/bin/
> -
> # clone yocto repositories we need
> ARG yocto_version="kirkstone"
> RUN for rep in \
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index ddc2234faf..4b8bcde252 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -584,6 +584,22 @@ alpine-3.12-gcc-arm64-boot-cpupools:
>     EXTRA_XEN_CONFIG: |
>       CONFIG_BOOT_TIME_CPUPOOLS=y
> 
> +yocto-kirkstone-qemuarm64:
> +  stage: build
> +  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
> +  script:
> +    - ./automation/build/yocto/build-yocto.sh -v --log-dir=./logs 
> --xen-dir=`pwd` qemuarm64
> +  variables:
> +    CONTAINER: yocto:kirkstone-qemuarm64
> +  artifacts:
> +    paths:
> +      - '*.log'
> +      - '*/*.log'
> +      - 'logs/*'
> +    when: always
> +  tags:
> +    - arm64
> +
> ## Test artifacts common
> 
> .test-jobs-artifact-common: