Showcase of openQA for testing on physical hardware

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

Hi all,

I have recent-ish made openQA to run tests not only inside KVM machines,
but also on physical hardware. The setup is described here:
https://www.qubes-os.org/news/2022/05/05/automated-os-testing-on-physical-laptops/

The live instance is at https://openqa.qubes-os.org. Tests running on
physical machines are marked with "@hw(number)", but from outside they don't
differ that much from virtualized ones.

This might be interesting for you, in context of extending/replacing
OSSTest. I think the above setup is an overkill for testing just Xen -
in my case the goal is to test mostly unmodified Qubes system, the way
the user would interact with it, but for Xen, running tests over SSH
and/or collecting output over serial console should be enough (you don't
need all this HID emulation, HDMI capture etc).

openQA provides nice way to schedule tests on various machines, a
utility library for writing tests (most of it is useful for GUI tests,
but still) and then collect and present results. It's main use case is
running tests in virtual machines which is visible in few places (much
more flexible scheduling, if you can take a disk image and connect it
somewhere else), but running on a bare metal is supported use case.

BTW I intentionally decided to build thing to be tested elsewhere -
gitlab specifically, and then provide it as a test input (there are
various methods for it in openQA, mostly using URLs that will be
automatically downloaded).

I use subset of this setup for `git bisect run`, but that has more to do
with remote power and boot control, than openQA.

Currently I only run there Qubes-specific tests - especially only
Xen versions we have packaged there. Running tests on xen-unstable
(master or staging) is complicated in Qubes case, because unstable ABIs
force rebuilds of several parts in that case. But I guess testing just
Xen with its native toolstack won't have this problem (and you build all
of it in gitlab already).

I'm open for any questions about this setup :)

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature