[Xen-devel] Test Meetup at Developer Summit

[Lars Kurth <lars.kurth.xen@xxxxxxxxx>]

== Attendees ==
Lars Kurth
George Dunlap
Doug Goldstein
Andrew Cooper
Paul Durrant

There were a few others, which I may have missed

I tried to transcribe from a recording we had at lunch, but due to background 
noise I didn't get everything. Please add/correct, if I got something wrong. 
There was a bit of discussion, but I tried to capture the key points only (e.g. 
I omitted clarifying questions)

== Discussion: Travis Improvements ==

Doug, wanted to discuss changes to the project's Travis set-up
* Reconfiguring Travis set-up such that it launches docker containers to do 
very basic smoke testing of individual commits on a number of different build 
configurations
* Right now this would require running a script running inside a script
* The key question is what matrix to support: e.g. which compiler versions, 
whether it should cover deb builds, ...
* It is basically just a build test, with a very basic run-test within an 
emulated qemu environment to check whether we boot. Aka booting a small image 
that is cross-compiled against various environments that produces "Hello World" 
on the first available serial-port
* If "Hello World" was produced, then Travis would decide the build was 
successful

George: over OSSTEST what value would this add? 
Doug: feedback about 7-8 minutes after a push with some confidence that a build 
was not broken, faster than OSSTEST does now. Obviously this would not replace 
OSSTEST, but should reduce the number of build-related issues in OSSTEST
Andrew: This should help quite a lot with avoiding spotting build related 
issues earlier
Doug: Would help avoid unnecessary manual build testing
George: Should help committers also 
Lars: Are there any extra requirements for people to use?
Andrew: Needs an account and set-up on GitHub

In summary: such a set-up would not compete with OSSTEST, but should decrease 
the number of build related OSSTEST failures, wasting fewer people's time if 
there are build related failures and increases OSSTEST throughput

Andrew, is not 100% sure how much extra confidence the "Hello World" test part 
will actually provide. However, Andrew also states that he does boot-testing 
before committing a major series. 

Random boot-tests on random config's (e.g. KConfigs) may also be valuable. But 
the challenge is what to test against, in OSSTEST, and how to know which test 
would succeed for which config. 

Paul: couldn't we build in something into XTF? ... there was a bit of 
discussion around this, but the audio was very hard to understand ... there was 
a discussion around skipping tests, but not quite knowing which ones should 
legitimately fail in a given configuration, or whether a skip was wrongly 
configured, or a git time-out. We would need to have more information about the 
"intent" of an OSSTEST to better help with random config tests. Andrew: thinks 
that for random build config tests, Travis CI may be more suitable than OSSTEST 
and require fewer plumbing changes, due to the faster turn-around and less 
complexity and scope for infrastructure related time-outs, etc. That is not to 
say that we shouldn't look are OSSTEST for functional config tests.

Andy: suggested a "presence test" in XTF for testing whether a specific config 
exists in a given binary, which could be used as a building block in OSSTEST 
for functional config tests. 

ASIDE: I would say that Doug and other participants, fill out this section, as 
I lost a lot of it, and may have misrepresented. 

Summary: In summary, we discussed
- Improvements to the Travis setup in particular for various build configs
- Possible improvements to OSSTEST / XTF for (random) config tests 
- There was no major objection

== Discussion: GitHub vs. GitLab for Travis ==
The second set of topics was to move from GitHub to GitLab for Build-testing 
due to the code availability issues with GitHub. Doug uses GitLab internally 
and is also more functional that GitHub. 

GitLab can be used hosted (for free projects) or host it locally on a VM

Doug: would probably require taking the github yaml and rewrite as gitlab yam. 
There are a number of advantages on OS support and docker integration, which 
makes set-up a lot easier compared to github.

Andy: if we were going to do this, the project should probably get a support 
license

Summary:
- Gather community input on GitHub vs. GitLab
- GitLab code is public, and thus should address 

There was also a bit of discussion about the review work-flow functionality in 
GitLab: just to clarify, we were not discussing any changes in this area. 
Merely to use GitLab as framework for Travis / Build testing and addressing 
some of the past issues that were raised around GitHub code availability.

== Supported / Unsupported / Preview / Experimental / ... ==

We should have some minimum tests and requirements for testing and 
configuration.

Andy: example - Intel MPX support is currently broken, because it was added 
when there was no available support. Because it is not compile-time disabled, 
the feature breaks existing users. This is a recurring theme, and we need to do 
something about it.

Proposal strawman:
- Would need a separate level of supported, it has to be manually configured in 
- At the moment the problem is that a previously working images often break on 
new HW 
- Standardise the terms: preview / experimental

Might have several rough buckets
- incomplete or no HW to test on: switch feature off via KConfig
- experimental : should be off by default in xml
- tech-preview : could be switched on by default
- to be fully supported, we need a minimum amount of tests, but we have an 
issue with grandfathering features that are supported and do not yet have them

Lars to re-start the lifecycle proposal again, taking new information and 
functionality such as KConfig into account and trying to get committer 
agreement. 


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel