Re: [PATCH v3 0/4] Yocto Gitlab CI

[Bertrand Marquis <Bertrand.Marquis@xxxxxxx>]

Hi Michal,

> On 7 Nov 2022, at 07:50, Michal Orzel <michal.orzel@xxxxxxx> wrote:
> 
> Hi Bertrand and Stefano,
> 
> On 31/10/2022 16:00, Bertrand Marquis wrote:
>> 
>> 
>> Hi Michal,
>> 
>>> On 31 Oct 2022, at 14:39, Michal Orzel <michal.orzel@xxxxxxx> wrote:
>>> 
>>> Hi Bertrand,
>>> 
>>> On 31/10/2022 15:00, Bertrand Marquis wrote:
>>>> 
>>>> 
>>>> This patch series is a first attempt to check if we could use Yocto in
>>>> gitlab ci to build and run xen on qemu for arm, arm64 and x86.
>>>> 
>>>> The first patch is creating a container with all elements required to
>>>> build Yocto, a checkout of the yocto layers required and an helper
>>>> script to build and run xen on qemu with yocto.
>>>> 
>>>> The second patch is creating containers with a first build of yocto done
>>>> so that susbsequent build with those containers would only rebuild what
>>>> was changed and take the rest from the cache.
>>>> 
>>>> The third patch is adding a way to easily clean locally created
>>>> containers.
>>>> 
>>>> This is is mainly for discussion and sharing as there are still some
>>>> issues/problem to solve:
>>>> - building the qemu* containers can take several hours depending on the
>>>> network bandwith and computing power of the machine where those are
>>>> created
>>> This is not really an issue as the build of the containers occurs on the 
>>> local
>>> machines before pushing them to registry. Also, building the containers
>>> will only be required for new Yocto releases.
>>> 
>>>> - produced containers containing the cache have a size between 8 and
>>>> 12GB depending on the architecture. We might need to store the build
>>>> cache somewhere else to reduce the size. If we choose to have one
>>>> single image, the needed size is around 20GB and we need up to 40GB
>>>> during the build, which is why I splitted them.
>>>> - during the build and run, we use a bit more then 20GB of disk which is
>>>> over the allowed size in gitlab
>>> As we could see during v2 testing, we do not have any space restrictions
>>> on the Xen GitLab and I think we already decided to have the Yocto
>>> integrated into our CI.
>> 
>> Right, I should have modified this chapter to be coherent with your latest 
>> tests.
>> Sorry for that.
>> 
>>> 
>>> I will do some testing and get back to you with results + review.
> I did some testing and here are the results:
> 
> In the current form this series will fail when running CI because the Yocto 
> containers
> are based on "From ubuntu:22.04" (there is no platform prefix), which means 
> that the containers
> are built for the host architecture (in my case and in 99% of the cases of 
> the local build it will
> be x86). In Gitlab we have 2 runners (arm64 and x86_64). This means that all 
> the test jobs would need
> to specify x86_64 as a tag when keeping the current behavior.
> After I built all the containers on my x86 machine, I pushed them to registry 
> and the pipeline was successful:
> https://gitlab.com/xen-project/people/morzel/xen-orzelmichal/-/pipelines/686853939
> 
> Here is the diff on patch no. 3 to make the series work (using x86 tag and 
> small improvement to include needs: []):
> ```
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 5c620fefce59..52cccec6f904 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -65,6 +65,9 @@
>     paths:
>       - 'logs/*'
>     when: always
> +  needs: []
> +  tags:
> +    - x86_64
> 
> # Test jobs
> build-each-commit-gcc:
> @@ -206,19 +209,13 @@ yocto-qemuarm64:
>   extends: .yocto-test
>   variables:
>     YOCTO_BOARD: qemuarm64
> -  tags:
> -    - arm64
> 
> yocto-qemuarm:
>   extends: .yocto-test
>   variables:
>     YOCTO_BOARD: qemuarm
> -  tags:
> -    - arm32
> 
> yocto-qemux86-64:
>   extends: .yocto-test
>   variables:
>     YOCTO_BOARD: qemux86-64
> -  tags:
> -    - x86_64
> ```
> 
> Now, the logical way would be to build x86 yocto container for x86, arm64 for 
> arm64 and arm32 on arm64 or x86.
> I tried building the container qemuarm64 specifying target arm64 on x86. 
> After 15h, only 70% of the Yocto build
> was completed and there was an error with glibc (the local build of the 
> container for the host arch takes on my machine max 2h).
> This enormous amount of time is due to the qemu docker emulation that happens 
> behind the scenes (I checked on 2 different machines).
> 
> So we have 3 solutions:
> 1) Build and run these containers for/on x86_64:
> - local users can build the containers on local machines that are almost 
> always x86 based, in short period of time,
> - "everyone" can build/push the containers once there is a new Yocto release
> - slightly slower CI build time
> 2) Build and run these containers for specific architectures:
> - almost no go for local users using x86 machine (unless using more than 16 
> threads (which I used) and willing to wait 2 days for the build)
> - faster CI build time (arm64 runner is faster than x86 one)
> - someone with arm64 based machine (not that common) would have to build and 
> push the containers
> 3) Try to use CI to build and push the containers to registry
> - it could be possible but what about local users

I might have a solution 4.
The containers we pushed to gitlab are publicly accessible ?
Shouldn’t we push both versions there and then document how someone can 
retrieve and use those containers instead of building them ?

This way we will do the push but a normal user would just retrieve the 
containers and use them so building time would have less impact and we could 
build both flavours of the containers.

> 
> Regardless of what we chose, we need to keep in mind that the biggest 
> advantage to the Yocto build/run is that
> it allows/should allow local users to perform basic testing for all the Xen 
> supported architectures. This is because
> everything happens in one place with one command.

Agree.

Bertrand

> 
> ~Michal