Hi all,
I had an action to put some resources regarding to testing on Xen together. I also CC’ed people who have an interest in CI systems in Xen and own or have worked on parts of the system. Following this, there is then an action by FuSa SIG
people on the TO list to put together a set of questions together and for Lars to arrange a demo/Q&A.
@Ian, @Andy, @Wei, @Doug: the intention was to look at what we do in Xen now and see whether there are any tie-ins with existing test infrastructure in Xen to build a baseline for more extensive testing for a subset of the Xen codebase.
Best Regards
Lars
Xen Source Trees
Commits to Xen trees are made to a staging tree
Changes need to pass our push gate which is controlled by OSSTEST which runs different configurations of Xen on different Hardware.
Once the tests have passed commits on staging trees are automatically merged to master trees
GitLab CI (unofficial)
URL: https://gitlab.com/xen-project/xen/pipelines
This is a skunkworks project which was started by Doug, which does the following for every commit to the Xen staging tree
- It does build tests for different combinations of Linux distros and compilers (gcc, clang)
- It also does some very basic run-time testing (aka does Xen boot using emulation via QEMU)
- It uses container based images which can be downloaded to reproduce issues locally
- It uses cloud infrastructure to execute jobs using Rackspace (x86) and Packer.io (Arm)
- The Arm support, which we are targeting as part of the SIG is somewhat limited I believe (restricted to Debian)
@Julien/@Stefano need to confirm what is implemented and what isn’t - The GitLab CI is
not currently gating commits to any Xen trees: in other words, test failures on a staging tree will not automatically prevent merges to master
There was a plan which has not yet been completed to …
- Take patches of xen-devel@, run them on the GitLab CI and have a bot report back to the list
before commits are made to staging.
This would ensure that failures are picked up before code changes are reviewed on xen-devel@ - This would also potentially provide a hook for other tools on proposed changes to the codebase, e.g. coding standard compliance hook
- I am pretty sure that there will be some discussion about this at the Developer Summit
Presentations
Relevance to SIG: This infrastructure could potentially be a hook for …
- MisraC checking on some configurations – although the Perforce set-up works totally differently and there may be licensing issues
- Other tools we may consider, e.g. if say we reverse engineer a subset of Xen with requirements statements, design documents and additional tests, this would be a good hook to check
traceability and completeness
XTF
URL: http://xenbits.xen.org/docs/xtf/
This is a framework for creating microkernel-based tests, and a suite of tests built using the framework.
It is the closest we can possibly get to Unit-type testing and is also useful for some functional tests, given the close interaction between a Hypervisor, HW and other parts of the system
Most tests are dependent on specific HW and test the presence of security vulnerabilities in the codebase
Note that XTF is run as part of OSSTEST
There is no Arm support, which is an issue
Presentations
Relevance to SIG:
-
Using XTF is probably the most feasible route for most testing that we would ever want to do in a safety context
OSSTEST (official)
OSSTEST primarily runs interoperability tests of different components in a Xen stack and performs common actions, such as migration, updates between versions, … in different environments
OSSTEST does build AND interoperability tests (as such there is some duplication between OSSTEST and the GitLab CI)
OSSTEST runs in our datacentre on lots of different Hardware – as such test runs are fairly long and also expensive)
OSSTEST gates commits to master
Presentations
