[RFC XEN PATCH 0/7] automation, RFC prototype, Have GitLab CI built its own containers

[Anthony PERARD <anthony.perard@xxxxxxxxxx>]

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git 
br.gitlab-containers-auto-rebuild-v1

Hi,

I have done some research to be able to build containers in the CI. This works
only for x86 containers as I've setup only a single x86 gitlab-runner to be
able to run docker-in-docker.

The runner is setup to only accept jobs from a branch that is "protected" in
gitlab. Also, one need credential to push to the container register, those are
called "Group deploy tokens", and I've set the variables CI_DEPLOY_USER and
CI_DEPLOY_PASSWORD in the project "xen-project/xen" (variables only visible on
a pipeline running on a protected branch).

These patch introduce quite a lot of redundancies in jobs, 2 new jobs per
containers which build/push containers, and duplicate most of build.yaml.
This mean that if we go with this, we have to duplicate and keep in sync many
jobs.

To avoid having to do the duplicated jobs by hand, I could look at
creating a script that use "build.yaml" as input and produce the 3
stages needed to update a container, but that script would need to be
run by hand, as gitlab can't really use it, unless ..

I've look at generated pipeline, and they look like this in gitlab:
    https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/777665738
But the result of the generated/child pipeline doesn't seems to be taken into
account in the original pipeline, which make me think that we can't use them to
generate "build.yaml". But maybe the could be use for generating the pipeline
that will update a container.
Doc:
    
https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines

So, with all of this, is it reasonable to test containers before
pushing them to production? Or is it to much work? We could simply have jobs
tacking care of rebuilding a container and pushing them to production without
testing.

An example with the variable DO_REBUILD_CONTAINER and PUSH_CONTAINER set (and
existing build/test jobs disabled):
    https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/791711467

Cheers,

Anthony PERARD (7):
  automation: Automatically rebuild debian:unstable container
  automation: Introduce test-containers stage
  automation: Add a template per container for build jobs.
  automation: Adding containers build jobs and test of thoses
  automation: Introduce DO_REBUILD_CONTAINER, to allow to rebuild a
    container
  automation: Push container been tested
  automation: Add some more push containers jobs

 .gitlab-ci.yml                            |   6 +
 automation/build/Makefile                 |  14 +-
 automation/gitlab-ci/build.yaml           | 327 ++++++++------
 automation/gitlab-ci/containers.yaml      |  98 +++++
 automation/gitlab-ci/push-containers.yaml |  79 ++++
 automation/gitlab-ci/test-containers.yaml | 496 ++++++++++++++++++++++
 6 files changed, 894 insertions(+), 126 deletions(-)
 create mode 100644 automation/gitlab-ci/containers.yaml
 create mode 100644 automation/gitlab-ci/push-containers.yaml
 create mode 100644 automation/gitlab-ci/test-containers.yaml

-- 
Anthony PERARD