Testing for the Xen Project

[Lars Kurth <lars.kurth@xxxxxxx>]

Hi all,

you probably have all heard by now that the Xen Project Advisory Board 
(a group of vendors who provide funds to the Xen Project that are 
intended to be used for the good of the community) recently created the 
Test Framework Working 
Group.http://wiki.xenproject.org/wiki/AB_WG/Test_Frameworkcontains more 
information about the group. The working group had its first meeting a 
few weeks ago and one of the actions I had was to kick off a thread on 
development lists to figure out what would help the developer community.
I was planning to kick off this thread with some questions and options, 
which reflect some discussions I had with individuals in the community, 
various meetings (WG and AB meetings), etc. which I condensed into a 
picture.
This reflects my personal opinion (not a Citrix opinion) and is merely 
intended to get a discussion going. Feel free to pick it apart: I won’t 
be upset.
First, I wanted to clear up a few misconceptions that I have heard from 
a few people:
* The Advisory Board has funds that can be used to create an 
independently hosted test infrastructure to help the developer 
community. However, funds are limited. Thus, it is important that we do 
what is right for the Xen community in the short term and the longer 
term. Otherwise, we will burn funds that could be used to help the Xen 
community in other ways.
* The Test Framework Working Group is made up of people employed by 
vendors who have some experience in testing.
* There is no intention to prescribe a test environment that you then 
have to use. Advisory Board members made clear to me that they want to 
make sure that what we end up with a solution that works for you.
* At the Xen Developer Summit two different solutions for system testing 
were presented. The intention was to explain what is there and what we 
can use going forward. A presentation on OSSTest which runs regularly 
today was given. And one for XenRT, for which there is a plan to get a 
small 3 box system up and running that can be used for you to look at. 
Citrix volunteered to set this up at its own cost.
* Just to be clear: what works for you may be one of these, none of 
these, both of them, …
* There may also be different answers in the short and the long run.
* At the end of the day, different community members will have different 
views. Also the Advisory Board members who provide the funds, will also 
have specific interests that they will push for. Thus, in all 
likelihood, we will have to find a good enough compromise.
* The vast majority of Advisory Board members care about the Hypervisor 
(and not so much about XAPI and Mirage OS). Thus, it is likely that the 
focus of the test system would be the Hypervisor.
So let me try and condense some of the arguments and opinions I heard 
and information that is around. This list may be incomplete.
== Work Flow ==

I added this section, because some members of the community and the 
working group had prior experience with attempts to introduce a test 
infrastructure for an open source community in the past, and these may 
not have worked as well as hoped. I made up some of the terminology.
*Local testing*: the basic idea here is for a developer to write their 
code and be able to run some tests on their machine locally as part of 
their development workflow. We do have same capability to do this with 
OSSTest 
(seehttp://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/), 
but the drawback is that this only works on the developers machine. This 
may be good enough. We could “emulate” different environments using 
QEMU, but that is likely to be slow.
*System testing*: both OSSTest and XenRT are essentially system test 
environments, which would be run on code that is already in the Xen 
mainline (or a staging branch). This is really great for regression 
testing on different hardware platforms, but does not match the typical 
workflow in the community:
a) code developed locally
b) code submitted to list for review
c) code reviewed and then submitted by the committer to the staging 
branch/mainline
d) only then tests are run
e) if tests fail, the whole process starts again
One of the issues here is that the elapsed time between development and 
the tests being run. Another issue we have with the current set-up is 
that OSSTest is currently hosted at Citrix, which makes difficult for 
community members to log into a machine and investigate issues. 
Admittedly this can fixed by having the system test hosted somewhere 
neutral (some sort of access control would still remain).
*Test on demand:* this would be a mixture between local testing and 
system testing. It can probably be implemented in both OSSTest and 
XenRT, although I believe XenRT is a closer to this model already. The 
workflow would look like this:
a) code developed locally
b) developer checks working prototype into their personal git branch 
(which would need to be accessible from the web)
c) developer submits a job to the test farm which checks out and builds 
code and runs a set of interesting (or new) tests on some machines of 
different architectures. New tests would probably need to be on the git 
branch
d) if the test fail, the cycle starts again
d) if all works well, code is submitted for review as normal (and test 
results could be attached)
This would only really work, if the interface into the system is easy to 
use and tests are run fast enough. Access control, etc. would be similar 
to system test. IMHO, this would be a nice mid to long-term goal, 
assuming it could be made to work with the funds we have.
== OSSTest ==

What runs now and thus easiest to get started on

More Info
*http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/
*http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/
*http://www.youtube.com/watch?v=JxTFZIwZzJ8

Problems:
* Runs on Citrix premises (thus general access is an issue)
* Ian Jackson is acting as sys-admin in his spare time. But, the 
Advisory Board could provide resource to fix this
* Basic test coverage
* Not a lot of documentation right now (which is a bit of a barrier to 
adoption)
Risks
* Not well understood (maybe you guys can fill the gaps)

== XenRT ==

Used by Citrix for XenServer testing. Tarballs have been made available 
by Citrix under a BSD license. But the code has not been put into live 
repos: my understanding is that Citrix would do this, if the Xen 
community believes this is valuable.
More Info
*http://wiki.xenproject.org/wiki/Getting_Started_with_XenRT
*http://www.youtube.com/watch?v=s11_Iw7AI_U

Problems:
* No publicly accessible demo instance (this is being worked on – to be 
hosted on a small test bed at http://osuosl.org/ – work sponsored by Citrix)
* Currently does not yet support “xl” (a “xl” connector is being worked 
on – sponsored by Citrix)
* Code not in yet public repo

Potentially Interesting Properties:
* Very large test coverage (including performance, security and other 
tests). Most of them should work once an “xl” connector is in place
* Been in production at scale for a long time: thus well understood
* XenRT has a lot of provisioning functionality and supports a 
distributed architecture: aka the ability to manage machines in 
different locations (data centres). The detail is abstracted away from 
users. This creates some interesting possibilities. For example:
** Hardware Vendors on the Advisory Board could provide hardware to the 
community on their site (assuming that these can be hosted outside a 
firewall). Some HW vendors on the AB indicated that this would indeed be 
doable.
** This would open up the opportunity to make available cutting edge or 
“unusual” HW for testing to the community.
** It would also mean that machines that would be expensive to ship and 
host by the project, could be hosted on premise by AB vendors
* XenRT has the capability to “inject” some test code on the fly (i.e. 
the test code is attached to a job that is submitted).
* I checked this with the XenRT devs and the *Test on demand* approach 
should be relatively easy to implement, but does not exist.
I do not know what of the above would apply to OSSTest.

Risks
* Complexity
* The cost of supporting such a system may be too high
* Not in use by the community today
* Not clear whether a *local test* version of XenRT is feasible

== Support and Ownership ==

Whatever solution we go for, needs to be properly funded and looked 
after. This is understood and the intention would be for the Xen Project 
(aka Advisory Board) to fund a Linux Foundation employee to do this on 
behalf of the Xen Project: this is a bit like Greg KH and others being 
LF employees working on the kernel. Some vendors on the Advisory Board 
indicated that providing Colo/hosting space and HW would be possible in 
principle, which could help keeping the cost manageable.
== Access ==

Any central system, has of course the issue of access control and 
managing users. This is obviously a barrier to entry (if we do not have 
also a local test mechanism). Am wondering how other FOSS communities 
handle this. This should certainly be the job of the Test Framework 
owner (see above).
Best Regards

Lars