Re: [PATCH 3/9] automation: Add debug versions of Arm tests

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Fri, 23 Sep 2022, Michal Orzel wrote:
> Hi Stefano,
> 
> On 22/09/2022 23:53, Stefano Stabellini wrote:
> > 
> > 
> > On Thu, 22 Sep 2022, Michal Orzel wrote:
> >> At the moment, all the tests are executed on non-debug Xen builds.
> >> To improve the coverage (e.g. we might catch some asserts), add new
> >> test jobs using debug Xen builds.
> > 
> > This is interesting. On one hand, I agree it is good to test DEBUG and
> > non-DEBUG builds in runtime tests too, on the other hand I am worried
> > about our test capacity.
> 
> We still have plenty of capacity. We can add many more Arm tests because
> Arm machine is very powerful. I reckon that adding more x86 tests would
> have a performance impact but not the other way around. See below.
> 
> > 
> > I take you have tried a few pipelines -- are the additional tests
> > causing the pipeline to take longer? Do we still have "room" because x86
> > is slower? What's your thinking?
> > 
> 
> With the additional tests introduced by this patch one cannot observe any
> impact for the performance. I ran several tests and the pipeline finishes
> in the same time or faster.
> 
> Performance comparison:
> current master: 92 jobs (82 minutes, 19 seconds)
> master + my series: 103 jobs (79 minutes, 40 seconds)
> 
> That said, let's improve the coverage and add these non-debug tests for Arm.

ahah WOW OK then


> > One idea would be to only do:
> > - qemu-alpine-arm64-gcc-debug
> > - qemu-smoke-arm64-gcc-debug
> > 
> > and not the others
> > 
> > 
> >> Take the opportunity to increase RAM size for QEMU from 1GB to 2GB
> >> in qemu-smoke-arm64.sh as the debug builds take more space and we might
> >> end up in a situation when there is not enough free space (especially
> >> during a static memory test that reserves some region in the middle).
> >>
> >> Signed-off-by: Michal Orzel <michal.orzel@xxxxxxx>
> >> ---
> >>  automation/gitlab-ci/build.yaml        | 16 +++++
> >>  automation/gitlab-ci/test.yaml         | 93 ++++++++++++++++++++++++++
> >>  automation/scripts/qemu-smoke-arm64.sh |  6 +-
> >>  3 files changed, 112 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/automation/gitlab-ci/build.yaml 
> >> b/automation/gitlab-ci/build.yaml
> >> index d931441b910a..6ce92531f61f 100644
> >> --- a/automation/gitlab-ci/build.yaml
> >> +++ b/automation/gitlab-ci/build.yaml
> >> @@ -587,6 +587,15 @@ alpine-3.12-gcc-arm64-staticmem:
> >>        CONFIG_UNSUPPORTED=y
> >>        CONFIG_STATIC_MEMORY=y
> >>
> >> +alpine-3.12-gcc-debug-arm64-staticmem:
> >> +  extends: .gcc-arm64-build-debug
> >> +  variables:
> >> +    CONTAINER: alpine:3.12-arm64v8
> >> +    EXTRA_XEN_CONFIG: |
> >> +      CONFIG_EXPERT=y
> >> +      CONFIG_UNSUPPORTED=y
> >> +      CONFIG_STATIC_MEMORY=y
> >> +
> >>  alpine-3.12-gcc-arm64-boot-cpupools:
> >>    extends: .gcc-arm64-build
> >>    variables:
> >> @@ -594,6 +603,13 @@ alpine-3.12-gcc-arm64-boot-cpupools:
> >>      EXTRA_XEN_CONFIG: |
> >>        CONFIG_BOOT_TIME_CPUPOOLS=y
> >>
> >> +alpine-3.12-gcc-debug-arm64-boot-cpupools:
> >> +  extends: .gcc-arm64-build-debug
> >> +  variables:
> >> +    CONTAINER: alpine:3.12-arm64v8
> >> +    EXTRA_XEN_CONFIG: |
> >> +      CONFIG_BOOT_TIME_CPUPOOLS=y
> >> +
> >>  ## Test artifacts common
> >>
> >>  .test-jobs-artifact-common:
> >> diff --git a/automation/gitlab-ci/test.yaml 
> >> b/automation/gitlab-ci/test.yaml
> >> index 4f96e6e322de..1b51030c6175 100644
> >> --- a/automation/gitlab-ci/test.yaml
> >> +++ b/automation/gitlab-ci/test.yaml
> >> @@ -44,6 +44,25 @@ qemu-alpine-arm64-gcc:
> >>    tags:
> >>      - arm64
> >>
> >> +qemu-alpine-arm64-gcc-debug:
> >> +  extends: .test-jobs-common
> >> +  variables:
> >> +    CONTAINER: debian:unstable-arm64v8
> >> +  script:
> >> +    - ./automation/scripts/qemu-alpine-arm64.sh 2>&1 | tee 
> >> qemu-smoke-arm64.log
> >> +  needs:
> >> +    - alpine-3.12-gcc-debug-arm64
> >> +    - alpine-3.12-arm64-rootfs-export
> >> +    - kernel-5.19-arm64-export
> >> +    - qemu-system-aarch64-6.0.0-arm64-export
> >> +  artifacts:
> >> +    paths:
> >> +      - smoke.serial
> >> +      - '*.log'
> >> +    when: always
> >> +  tags:
> >> +    - arm64
> >> +
> >>  qemu-alpine-x86_64-gcc:
> >>    extends: .test-jobs-common
> >>    variables:
> >> @@ -81,6 +100,25 @@ qemu-smoke-arm64-gcc:
> >>    tags:
> >>      - arm64
> >>
> >> +qemu-smoke-arm64-gcc-debug:
> >> +  extends: .test-jobs-common
> >> +  variables:
> >> +    CONTAINER: debian:unstable-arm64v8
> >> +  script:
> >> +    - ./automation/scripts/qemu-smoke-arm64.sh 2>&1 | tee 
> >> qemu-smoke-arm64.log
> >> +  needs:
> >> +    - alpine-3.12-gcc-debug-arm64
> >> +    - alpine-3.12-arm64-rootfs-export
> >> +    - kernel-5.19-arm64-export
> >> +    - qemu-system-aarch64-6.0.0-arm64-export
> >> +  artifacts:
> >> +    paths:
> >> +      - smoke.serial
> >> +      - '*.log'
> >> +    when: always
> >> +  tags:
> >> +    - arm64
> >> +
> >>  qemu-smoke-arm64-gcc-staticmem:
> >>    extends: .test-jobs-common
> >>    variables:
> >> @@ -100,6 +138,25 @@ qemu-smoke-arm64-gcc-staticmem:
> >>    tags:
> >>      - arm64
> >>
> >> +qemu-smoke-arm64-gcc-debug-staticmem:
> >> +  extends: .test-jobs-common
> >> +  variables:
> >> +    CONTAINER: debian:unstable-arm64v8
> >> +  script:
> >> +    - ./automation/scripts/qemu-smoke-arm64.sh static-mem 2>&1 | tee 
> >> qemu-smoke-arm64.log
> >> +  needs:
> >> +    - alpine-3.12-gcc-debug-arm64-staticmem
> >> +    - alpine-3.12-arm64-rootfs-export
> >> +    - kernel-5.19-arm64-export
> >> +    - qemu-system-aarch64-6.0.0-arm64-export
> >> +  artifacts:
> >> +    paths:
> >> +      - smoke.serial
> >> +      - '*.log'
> >> +    when: always
> >> +  tags:
> >> +    - arm64
> >> +
> >>  qemu-smoke-arm64-gcc-boot-cpupools:
> >>    extends: .test-jobs-common
> >>    variables:
> >> @@ -119,6 +176,25 @@ qemu-smoke-arm64-gcc-boot-cpupools:
> >>    tags:
> >>      - arm64
> >>
> >> +qemu-smoke-arm64-gcc-debug-boot-cpupools:
> >> +  extends: .test-jobs-common
> >> +  variables:
> >> +    CONTAINER: debian:unstable-arm64v8
> >> +  script:
> >> +    - ./automation/scripts/qemu-smoke-arm64.sh boot-cpupools 2>&1 | tee 
> >> qemu-smoke-arm64.log
> >> +  needs:
> >> +    - alpine-3.12-gcc-debug-arm64-boot-cpupools
> >> +    - alpine-3.12-arm64-rootfs-export
> >> +    - kernel-5.19-arm64-export
> >> +    - qemu-system-aarch64-6.0.0-arm64-export
> >> +  artifacts:
> >> +    paths:
> >> +      - smoke.serial
> >> +      - '*.log'
> >> +    when: always
> >> +  tags:
> >> +    - arm64
> >> +
> >>  qemu-smoke-arm32-gcc:
> >>    extends: .test-jobs-common
> >>    variables:
> >> @@ -136,6 +212,23 @@ qemu-smoke-arm32-gcc:
> >>    tags:
> >>      - arm64
> >>
> >> +qemu-smoke-arm32-gcc-debug:
> >> +  extends: .test-jobs-common
> >> +  variables:
> >> +    CONTAINER: debian:unstable-arm64v8
> >> +  script:
> >> +    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee 
> >> qemu-smoke-arm32.log
> >> +  needs:
> >> +    - debian-unstable-gcc-arm32-debug
> >> +    - qemu-system-aarch64-6.0.0-arm32-export
> >> +  artifacts:
> >> +    paths:
> >> +      - smoke.serial
> >> +      - '*.log'
> >> +    when: always
> >> +  tags:
> >> +    - arm64
> >> +
> >>  qemu-smoke-x86-64-gcc:
> >>    extends: .test-jobs-common
> >>    variables:
> >> diff --git a/automation/scripts/qemu-smoke-arm64.sh 
> >> b/automation/scripts/qemu-smoke-arm64.sh
> >> index f803835779f4..dea26c6c0a8f 100755
> >> --- a/automation/scripts/qemu-smoke-arm64.sh
> >> +++ b/automation/scripts/qemu-smoke-arm64.sh
> >> @@ -52,7 +52,7 @@ curl -fsSLO 
> >> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqemu%2Fqemu%2Fraw%2Fv5.2.0%2Fpc-bios%2Fefi-virtio.rom&amp;data=05%7C01%7Cmichal.orzel%40amd.com%7Cf6b0b8fabb02478b009c08da9ce4eac9%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637994804254367719%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=jgGdO93kLoieg6UqajFWYAsufvvv5mZVYHD%2B7fUW7ME%3D&amp;reserved=0
> >>  ./binaries/qemu-system-aarch64 \
> >>     -machine virtualization=true \
> >>     -cpu cortex-a57 -machine type=virt \
> >> -   -m 1024 -smp 2 -display none \
> >> +   -m 2048 -smp 2 -display none \
> >>     -machine dumpdtb=binaries/virt-gicv2.dtb
> >>
> >>  # XXX disable pl061 to avoid Linux crash
> >> @@ -117,7 +117,7 @@ cd ..
> >>
> >>  # ImageBuilder
> >>  echo 'MEMORY_START="0x40000000"
> >> -MEMORY_END="0x80000000"
> >> +MEMORY_END="0xC0000000"
> >>
> >>  DEVICE_TREE="virt-gicv2.dtb"
> >>  XEN="xen"
> >> @@ -158,7 +158,7 @@ timeout -k 1 240 \
> >>  ./binaries/qemu-system-aarch64 \
> >>      -machine virtualization=true \
> >>      -cpu cortex-a57 -machine type=virt \
> >> -    -m 1024 -monitor none -serial stdio \
> >> +    -m 2048 -monitor none -serial stdio \
> >>      -smp 2 \
> >>      -no-reboot \
> >>      -device virtio-net-pci,netdev=n0 \
> > 
> > The changes to qemu-smoke-arm64.sh make sense, but could you move them
> > to a separate patch in case we need to do any backports?
> 
> Ok, I will do a prerequisite patch for that one.
> 
> > 
> > Also I would change MEMORY_END to 0xC0000000 in qemu-alpine-arm64.sh too
> > for uniformity.
> 
> I already did that in patch [6/9] automation: qemu-alpine-arm64: Cleanup and 
> fixes.
> 
> ~Michal
>