Re: [PATCH] automation: test.yaml: Introduce templates to reduce the overhead

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Wed, 19 Oct 2022, Michal Orzel wrote:
> At the moment, we define lots of test jobs in test.yaml, that make use
> of the same configuration sections like variables, tags, artifacts.
> Introduce templates (hidden jobs whose names start with a dot) to
> reduce the overhead and simplify the file (more than 100 lines saved).
> This way, the actual jobs can only specify sections that are unique
> to them.
> 
> Most of the test jobs specify the same set of prerequisite jobs under needs
> property with just one additional being unique to the job itself. Introduce
> YAML anchors for that purpose, because when using extends, the needs property
> is not being merged (the parent property overwrites the child one).

I like the patch. Replying here on top because the diff below is not
very helpful.

When you say that "extends" overwrites the properties, do you mean that
"needs" in qemu-smoke-dom0-arm64-gcc overwrites "needs" in .qemu-arm64,
when qemu-smoke-dom0-arm64-gcc includes .qemu-arm64?


If there is no way to solve the overwrite problem then it is OK to use
YAML achors but is it possible to define the anchors outside of
.qemu-arm64/.qemu-arm32 ? It would make things a lot clearer in the
code. Maybe under a top level "definitions" key? The point is that
.qemu-arm64 and .qemu-arm32 should use the anchor rather than define the
anchor.

I wouldn't call it qemu-arm64-needs because it has things
like alpine-3.12-arm64-rootfs-export and kernel-5.19-arm64-export that
are not required by qemu-system-aarch64-6.0.0-arm64-export. If anything
qemu-system-aarch64-6.0.0-arm64-export needs CONTAINER:
debian:unstable-arm64v8.

So I would call the anchor something like "arm64-test-needs". Same
comment for the arm32 anchor.


> Signed-off-by: Michal Orzel <michal.orzel@xxxxxxx>
> ---
> This patch is based on the CI next branch where we already have several
> patches (already acked) to be merged into staging after the release:
> https://gitlab.com/xen-project/people/sstabellini/xen/-/tree/next
> 
> Tested pipeline:
> https://gitlab.com/xen-project/people/morzel/xen-orzelmichal/-/pipelines/671114820
> ---
>  automation/gitlab-ci/test.yaml | 266 ++++++++++-----------------------
>  1 file changed, 80 insertions(+), 186 deletions(-)
> 
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 92e0a1f7c510..fc0884b12082 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -7,32 +7,12 @@
>      - /^coverity-tested\/.*/
>      - /^stable-.*/
>  
> -# Test jobs
> -build-each-commit-gcc:
> -  extends: .test-jobs-common
> -  variables:
> -    CONTAINER: debian:stretch
> -    XEN_TARGET_ARCH: x86_64
> -    CC: gcc
> -  script:
> -    - BASE=${BASE_SHA:-${CI_COMMIT_BEFORE_SHA}} 
> TIP=${TIP_SHA:-${CI_COMMIT_SHA}} ./automation/gitlab-ci/build-each-commit.sh 
> 2>&1 | tee ../build-each-commit-gcc.log
> -    - mv ../build-each-commit-gcc.log .
> -  artifacts:
> -    paths:
> -      - '*.log'
> -    when: always
> -  needs: []
> -  tags:
> -    - x86_64
> -
> -qemu-smoke-dom0-arm64-gcc:
> +.qemu-arm64:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:unstable-arm64v8
> -  script:
> -    - ./automation/scripts/qemu-smoke-dom0-arm64.sh 2>&1 | tee 
> qemu-smoke-arm64.log
> -  needs:
> -    - alpine-3.12-gcc-arm64
> +    LOGFILE: qemu-smoke-arm64.log
> +  needs: &qemu-arm64-needs
>      - alpine-3.12-arm64-rootfs-export
>      - kernel-5.19-arm64-export
>      - qemu-system-aarch64-6.0.0-arm64-export

LOGFILE should be listed among the artifacts (and maybe we can remove
*.log if it has become redundant?)


> @@ -44,17 +24,13 @@ qemu-smoke-dom0-arm64-gcc:
>    tags:
>      - arm64
>  
> -qemu-smoke-dom0-arm64-gcc-debug:
> +.qemu-arm32:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:unstable-arm64v8
> -  script:
> -    - ./automation/scripts/qemu-smoke-dom0-arm64.sh 2>&1 | tee 
> qemu-smoke-arm64.log
> -  needs:
> -    - alpine-3.12-gcc-debug-arm64
> -    - alpine-3.12-arm64-rootfs-export
> -    - kernel-5.19-arm64-export
> -    - qemu-system-aarch64-6.0.0-arm64-export
> +    LOGFILE: qemu-smoke-arm32.log
> +  needs: &qemu-arm32-needs
> +    - qemu-system-aarch64-6.0.0-arm32-export
>    artifacts:
>      paths:
>        - smoke.serial
> @@ -63,16 +39,11 @@ qemu-smoke-dom0-arm64-gcc-debug:
>    tags:
>      - arm64
>  
> -qemu-alpine-x86_64-gcc:
> +.qemu-x86-64:
>    extends: .test-jobs-common
>    variables:
>      CONTAINER: debian:stretch
> -  script:
> -    - ./automation/scripts/qemu-alpine-x86_64.sh 2>&1 | tee 
> qemu-smoke-x86_64.log
> -  needs:
> -    - alpine-3.12-gcc
> -    - alpine-3.12-rootfs-export
> -    - kernel-5.10.74-export
> +    LOGFILE: qemu-smoke-x86-64.log
>    artifacts:
>      paths:
>        - smoke.serial
> @@ -81,214 +52,137 @@ qemu-alpine-x86_64-gcc:
>    tags:
>      - x86_64
>  
> -qemu-smoke-dom0less-arm64-gcc:
> +# Test jobs
> +build-each-commit-gcc:
>    extends: .test-jobs-common
>    variables:
> -    CONTAINER: debian:unstable-arm64v8
> +    CONTAINER: debian:stretch
> +    XEN_TARGET_ARCH: x86_64
> +    CC: gcc
>    script:
> -    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh 2>&1 | tee 
> qemu-smoke-arm64.log
> -  needs:
> -    - alpine-3.12-gcc-arm64
> -    - alpine-3.12-arm64-rootfs-export
> -    - kernel-5.19-arm64-export
> -    - qemu-system-aarch64-6.0.0-arm64-export
> +    - BASE=${BASE_SHA:-${CI_COMMIT_BEFORE_SHA}} 
> TIP=${TIP_SHA:-${CI_COMMIT_SHA}} ./automation/gitlab-ci/build-each-commit.sh 
> 2>&1 | tee ../build-each-commit-gcc.log
> +    - mv ../build-each-commit-gcc.log .
>    artifacts:
>      paths:
> -      - smoke.serial
>        - '*.log'
>      when: always
> +  needs: []
>    tags:
> -    - arm64
> +    - x86_64
> +
> +qemu-smoke-dom0-arm64-gcc:
> +  extends: .qemu-arm64
> +  script:
> +    - ./automation/scripts/qemu-smoke-dom0-arm64.sh 2>&1 | tee ${LOGFILE}
> +  needs:
> +    - *qemu-arm64-needs
> +    - alpine-3.12-gcc-arm64
> +
> +qemu-smoke-dom0-arm64-gcc-debug:
> +  extends: .qemu-arm64
> +  script:
> +    - ./automation/scripts/qemu-smoke-dom0-arm64.sh 2>&1 | tee ${LOGFILE}
> +  needs:
> +    - *qemu-arm64-needs
> +    - alpine-3.12-gcc-debug-arm64
> +
> +qemu-smoke-dom0less-arm64-gcc:
> +  extends: .qemu-arm64
> +  script:
> +    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh 2>&1 | tee ${LOGFILE}
> +  needs:
> +    - *qemu-arm64-needs
> +    - alpine-3.12-gcc-arm64
>  
>  qemu-smoke-dom0less-arm64-gcc-debug:
> -  extends: .test-jobs-common
> -  variables:
> -    CONTAINER: debian:unstable-arm64v8
> +  extends: .qemu-arm64
>    script:
> -    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh 2>&1 | tee 
> qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh 2>&1 | tee ${LOGFILE}
>    needs:
> +    - *qemu-arm64-needs
>      - alpine-3.12-gcc-debug-arm64
> -    - alpine-3.12-arm64-rootfs-export
> -    - kernel-5.19-arm64-export
> -    - qemu-system-aarch64-6.0.0-arm64-export
> -  artifacts:
> -    paths:
> -      - smoke.serial
> -      - '*.log'
> -    when: always
> -  tags:
> -    - arm64
>  
>  qemu-smoke-dom0less-arm64-gcc-staticmem:
> -  extends: .test-jobs-common
> -  variables:
> -    CONTAINER: debian:unstable-arm64v8
> +  extends: .qemu-arm64
>    script:
> -    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh static-mem 2>&1 | 
> tee qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh static-mem 2>&1 | 
> tee ${LOGFILE}
>    needs:
> +    - *qemu-arm64-needs
>      - alpine-3.12-gcc-arm64-staticmem
> -    - alpine-3.12-arm64-rootfs-export
> -    - kernel-5.19-arm64-export
> -    - qemu-system-aarch64-6.0.0-arm64-export
> -  artifacts:
> -    paths:
> -      - smoke.serial
> -      - '*.log'
> -    when: always
> -  tags:
> -    - arm64
>  
>  qemu-smoke-dom0less-arm64-gcc-debug-staticmem:
> -  extends: .test-jobs-common
> -  variables:
> -    CONTAINER: debian:unstable-arm64v8
> +  extends: .qemu-arm64
>    script:
> -    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh static-mem 2>&1 | 
> tee qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh static-mem 2>&1 | 
> tee ${LOGFILE}
>    needs:
> +    - *qemu-arm64-needs
>      - alpine-3.12-gcc-debug-arm64-staticmem
> -    - alpine-3.12-arm64-rootfs-export
> -    - kernel-5.19-arm64-export
> -    - qemu-system-aarch64-6.0.0-arm64-export
> -  artifacts:
> -    paths:
> -      - smoke.serial
> -      - '*.log'
> -    when: always
> -  tags:
> -    - arm64
>  
>  qemu-smoke-dom0less-arm64-gcc-boot-cpupools:
> -  extends: .test-jobs-common
> -  variables:
> -    CONTAINER: debian:unstable-arm64v8
> +  extends: .qemu-arm64
>    script:
> -    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh boot-cpupools 2>&1 | 
> tee qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh boot-cpupools 2>&1 | 
> tee ${LOGFILE}
>    needs:
> +    - *qemu-arm64-needs
>      - alpine-3.12-gcc-arm64-boot-cpupools
> -    - alpine-3.12-arm64-rootfs-export
> -    - kernel-5.19-arm64-export
> -    - qemu-system-aarch64-6.0.0-arm64-export
> -  artifacts:
> -    paths:
> -      - smoke.serial
> -      - '*.log'
> -    when: always
> -  tags:
> -    - arm64
>  
>  qemu-smoke-dom0less-arm64-gcc-debug-boot-cpupools:
> -  extends: .test-jobs-common
> -  variables:
> -    CONTAINER: debian:unstable-arm64v8
> +  extends: .qemu-arm64
>    script:
> -    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh boot-cpupools 2>&1 | 
> tee qemu-smoke-arm64.log
> +    - ./automation/scripts/qemu-smoke-dom0less-arm64.sh boot-cpupools 2>&1 | 
> tee ${LOGFILE}
>    needs:
> +    - *qemu-arm64-needs
>      - alpine-3.12-gcc-debug-arm64-boot-cpupools
> -    - alpine-3.12-arm64-rootfs-export
> -    - kernel-5.19-arm64-export
> -    - qemu-system-aarch64-6.0.0-arm64-export
> -  artifacts:
> -    paths:
> -      - smoke.serial
> -      - '*.log'
> -    when: always
> -  tags:
> -    - arm64
>  
>  qemu-smoke-dom0-arm32-gcc:
> -  extends: .test-jobs-common
> -  variables:
> -    CONTAINER: debian:unstable-arm64v8
> +  extends: .qemu-arm32
>    script:
> -    - ./automation/scripts/qemu-smoke-dom0-arm32.sh 2>&1 | tee 
> qemu-smoke-arm32.log
> +    - ./automation/scripts/qemu-smoke-dom0-arm32.sh 2>&1 | tee ${LOGFILE}
>    needs:
> +    - *qemu-arm32-needs
>      - debian-unstable-gcc-arm32
> -    - qemu-system-aarch64-6.0.0-arm32-export
> -  artifacts:
> -    paths:
> -      - smoke.serial
> -      - '*.log'
> -    when: always
> -  tags:
> -    - arm64
>  
>  qemu-smoke-dom0-arm32-gcc-debug:
> -  extends: .test-jobs-common
> -  variables:
> -    CONTAINER: debian:unstable-arm64v8
> +  extends: .qemu-arm32
>    script:
> -    - ./automation/scripts/qemu-smoke-dom0-arm32.sh 2>&1 | tee 
> qemu-smoke-arm32.log
> +    - ./automation/scripts/qemu-smoke-dom0-arm32.sh 2>&1 | tee ${LOGFILE}
>    needs:
> +    - *qemu-arm32-needs
>      - debian-unstable-gcc-arm32-debug
> -    - qemu-system-aarch64-6.0.0-arm32-export
> -  artifacts:
> -    paths:
> -      - smoke.serial
> -      - '*.log'
> -    when: always
> -  tags:
> -    - arm64
> +
> +qemu-alpine-x86_64-gcc:
> +  extends: .qemu-x86-64
> +  script:
> +    - ./automation/scripts/qemu-alpine-x86_64.sh 2>&1 | tee ${LOGFILE}
> +  needs:
> +    - alpine-3.12-gcc
> +    - alpine-3.12-rootfs-export
> +    - kernel-5.10.74-export
>  
>  qemu-smoke-x86-64-gcc:
> -  extends: .test-jobs-common
> -  variables:
> -    CONTAINER: debian:stretch
> +  extends: .qemu-x86-64
>    script:
> -    - ./automation/scripts/qemu-smoke-x86-64.sh pv 2>&1 | tee 
> qemu-smoke-x86-64.log
> -  artifacts:
> -    paths:
> -      - smoke.serial
> -      - '*.log'
> -    when: always
> +    - ./automation/scripts/qemu-smoke-x86-64.sh pv 2>&1 | tee ${LOGFILE}
>    needs:
>      - debian-stretch-gcc-debug
> -  tags:
> -    - x86_64
>  
>  qemu-smoke-x86-64-clang:
> -  extends: .test-jobs-common
> -  variables:
> -    CONTAINER: debian:stretch
> +  extends: .qemu-x86-64
>    script:
> -    - ./automation/scripts/qemu-smoke-x86-64.sh pv 2>&1 | tee 
> qemu-smoke-x86-64.log
> -  artifacts:
> -    paths:
> -      - smoke.serial
> -      - '*.log'
> -    when: always
> +    - ./automation/scripts/qemu-smoke-x86-64.sh pv 2>&1 | tee ${LOGFILE}
>    needs:
>      - debian-unstable-clang-debug
> -  tags:
> -    - x86_64
>  
>  qemu-smoke-x86-64-gcc-pvh:
> -  extends: .test-jobs-common
> -  variables:
> -    CONTAINER: debian:stretch
> +  extends: .qemu-x86-64
>    script:
> -    - ./automation/scripts/qemu-smoke-x86-64.sh pvh 2>&1 | tee 
> qemu-smoke-x86-64.log
> -  artifacts:
> -    paths:
> -      - smoke.serial
> -      - '*.log'
> -    when: always
> +    - ./automation/scripts/qemu-smoke-x86-64.sh pvh 2>&1 | tee ${LOGFILE}
>    needs:
>      - debian-stretch-gcc-debug
> -  tags:
> -    - x86_64
>  
>  qemu-smoke-x86-64-clang-pvh:
> -  extends: .test-jobs-common
> -  variables:
> -    CONTAINER: debian:stretch
> +  extends: .qemu-x86-64
>    script:
> -    - ./automation/scripts/qemu-smoke-x86-64.sh pvh 2>&1 | tee 
> qemu-smoke-x86-64.log
> -  artifacts:
> -    paths:
> -      - smoke.serial
> -      - '*.log'
> -    when: always
> +    - ./automation/scripts/qemu-smoke-x86-64.sh pvh 2>&1 | tee ${LOGFILE}
>    needs:
>      - debian-unstable-clang-debug
> -  tags:
> -    - x86_64
> -- 
> 2.25.1
>