Re: [Xen-devel] Testing for the Xen Project

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Tue, 2013-11-26 at 13:21 +0000, Lars Kurth wrote:
> Hi all,
> 
> you probably have all heard by now that the Xen Project Advisory Board 
> (a group of vendors who provide funds to the Xen Project that are 
> intended to be used for the good of the community) recently created the 
> Test Framework Working 
> Group.http://wiki.xenproject.org/wiki/AB_WG/Test_Frameworkcontains more 
> information about the group. The working group had its first meeting a 
> few weeks ago and one of the actions I had was to kick off a thread on 
> development lists to figure out what would help the developer community.
> 
> I was planning to kick off this thread with some questions and options, 
> which reflect some discussions I had with individuals in the community, 
> various meetings (WG and AB meetings), etc. which I condensed into a 
> picture.
> 
> This reflects my personal opinion (not a Citrix opinion) and is merely 
> intended to get a discussion going. Feel free to pick it apart: I wonât 
> be upset.
> 
> First, I wanted to clear up a few misconceptions that I have heard from 
> a few people:
> 
> * The Advisory Board has funds that can be used to create an 
> independently hosted test infrastructure to help the developer 
> community. However, funds are limited. Thus, it is important that we do 
> what is right for the Xen community in the short term and the longer 
> term. Otherwise, we will burn funds that could be used to help the Xen 
> community in other ways.
> * The Test Framework Working Group is made up of people employed by 
> vendors who have some experience in testing.
> * There is no intention to prescribe a test environment that you then 
> have to use. Advisory Board members made clear to me that they want to 
> make sure that what we end up with a solution that works for you.
> * At the Xen Developer Summit two different solutions for system testing 
> were presented. The intention was to explain what is there and what we 
> can use going forward. A presentation on OSSTest which runs regularly 
> today was given. And one for XenRT, for which there is a plan to get a 
> small 3 box system up and running that can be used for you to look at. 
> Citrix volunteered to set this up at its own cost.
> * Just to be clear: what works for you may be one of these, none of 
> these, both of them, â
> * There may also be different answers in the short and the long run.
> * At the end of the day, different community members will have different 
> views. Also the Advisory Board members who provide the funds, will also 
> have specific interests that they will push for. Thus, in all 
> likelihood, we will have to find a good enough compromise.
> * The vast majority of Advisory Board members care about the Hypervisor 
> (and not so much about XAPI and Mirage OS). Thus, it is likely that the 
> focus of the test system would be the Hypervisor.
> 
> So let me try and condense some of the arguments and opinions I heard 
> and information that is around. This list may be incomplete.
> 
> == Work Flow ==
> 
> I added this section, because some members of the community and the 
> working group had prior experience with attempts to introduce a test 
> infrastructure for an open source community in the past, and these may 
> not have worked as well as hoped. I made up some of the terminology.
> 
> *Local testing*: the basic idea here is for a developer to write their 
[...]

We have a small amount of local test suites in the tree (e.g. vif and
disk config parsing have little test suites) but it could do with tying
together with some infrastructure into something which is simple to run
(currently it requires an installed Xen system and there is no one
single way to run something).

As you correctly suggest there is a limit to how much local testing can
cover in terms of elapsed time, available resources, the configurations
which can be reasonably set up, running on real hardware etc. IMHO This
could benefit from an enthusiastic (or press-ganged by their
manager ;-)) community member putting some time into tying it all
together into something which we can ask people to run before submitting
with a straight face.


> *System testing*: both OSSTest and XenRT are essentially system test 
[...]

I think most people use "system testing" to mean testing of the
integrated whole, as opposed to e.g. unit testing. The current
"automated test" which we have covers some aspects of both whole system
and unit testing.

Anyway, terminology aside, the existing osstest stuff is *extremely*
valuable IMHO, and the system testing has been very useful over the
majority of the lifetime of the xen project, at least as long as I've
been involved. The main limitation is the amount of resources dedicated
to it, in terms of hardware (and its location within citrix
infrastructure doesn't help here) and test coverage.

Even with its current set of tests and limited hardware it already tests
far more than we could ever realistically ask people to do locally
before submitting and it catches real issues on real hardware.

Any local test stuff should obviously be integrated into the system
tests as a step as well.

I notice that your description of system test omits the targeted local
testing which we expect contributors to do before submitting a patch --
by targeted I mean you are changing $FOO therefore you should be trying
$FOO! And if you think you might have an impact on $BAR you should be
testing that too. I just mention it because your description seemed to
imply (inadvertently I expect) that there was no testing at all between
writing the code and the system tests running, which is not quite
accurate.

IMHO both local and system test are valuable. I think the local testing
situation can be improved by people working within the community to do
the work (in particular building out the infrastructure), whereas the
system testing side of things would benefit greatly from any resourcing
which the AB can provide in terms of hardware, hosting and sysadmin time
etc. There is no doubt in my mind that this would be beneficial to the
community in both the short and long term.

It might also be worth considering spending some money kickstarting the
actual tests (i.e. fleshing out the suites) in both cases, but I think
ultimately I think this needs to be driven by community member (AB or
otherwise) who care about particular functionality making sure the tests
exist, probably by writing them. So in terms of budget I think that
would be secondary to sorting out the hosting etc

> *Test on demand:* this would be a mixture between local testing and 
[...]

I think it would be nice long term goal to aim for this but short term
the other two types of testing are more important.

> IMHO, this would be a nice mid to long-term goal, 
> assuming it could be made to work with the funds we have.

Heh, I should read right to the end ;-)

> == OSSTest ==
> 
> What runs now and thus easiest to get started on
> 
> More Info
> *http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/
> *http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/
> *http://www.youtube.com/watch?v=JxTFZIwZzJ8
> 
> Problems:
> * Runs on Citrix premises (thus general access is an issue)
> * Ian Jackson is acting as sys-admin in his spare time. But, the 
> Advisory Board could provide resource to fix this
> * Basic test coverage
> * Not a lot of documentation right now (which is a bit of a barrier to 
> adoption)
> 
> Risks
> * Not well understood (maybe you guys can fill the gaps)

This is slowly changing, Wei, Roger and myself have all done development
with osstest and contributed (or are in the process of doing so) new
bits of testing. I think Dario and Anthony have played with it too.
There is certainly more which could be done here in terms of
documentation. I at least was planning to make this part of my focus on
future documentation and/or test days.

I think all of the above applies equally to XenRT, either system is
going to have a learning curve and is going to need documentation for
the community etc.

> == XenRT ==
> 
> Used by Citrix for XenServer testing. Tarballs have been made available 
> by Citrix under a BSD license. But the code has not been put into live 
> repos: my understanding is that Citrix would do this, if the Xen 
> community believes this is valuable.
> 
> More Info
> *http://wiki.xenproject.org/wiki/Getting_Started_with_XenRT
> *http://www.youtube.com/watch?v=s11_Iw7AI_U
> 
> Problems:
> * No publicly accessible demo instance (this is being worked on â to be 
> hosted on a small test bed at http://osuosl.org/ â work sponsored by Citrix)
> * Currently does not yet support âxlâ (a âxlâ connector is being worked 
> on â sponsored by Citrix)
> * Code not in yet public repo
> 
> Potentially Interesting Properties:
> * Very large test coverage (including performance, security and other 
> tests). Most of them should work once an âxlâ connector is in place

I think that's rather optimistic. I would expect that a reasonable
proportion of the interesting tests will require features of xapi to
work, e.g. pools of hosts, storage management, networking etc and/or
require some amount of reworking to function with xl.

> * Been in production at scale for a long time: thus well understood
> * XenRT has a lot of provisioning functionality and supports a 
> distributed architecture: aka the ability to manage machines in 
> different locations (data centres). The detail is abstracted away from 
> users. This creates some interesting possibilities. For example:
> ** Hardware Vendors on the Advisory Board could provide hardware to the 
> community on their site (assuming that these can be hosted outside a 
> firewall). Some HW vendors on the AB indicated that this would indeed be 
> doable.
> ** This would open up the opportunity to make available cutting edge or 
> âunusualâ HW for testing to the community.
> ** It would also mean that machines that would be expensive to ship and 
> host by the project, could be hosted on premise by AB vendors
> * XenRT has the capability to âinjectâ some test code on the fly (i.e. 
> the test code is attached to a job that is submitted).
> * I checked this with the XenRT devs and the *Test on demand* approach 
> should be relatively easy to implement, but does not exist.
> 
> I do not know what of the above would apply to OSSTest.

I think it is all equally doable for either.

> Risks
> * Complexity
> * The cost of supporting such a system may be too high
> * Not in use by the community today
> * Not clear whether a *local test* version of XenRT is feasible
> 
> == Support and Ownership ==
> 
> Whatever solution we go for, needs to be properly funded and looked 
> after.

>From the remainder of the paragraph I think you are talking specifically
about hiring a test person here I think?

I think this is essential, the current testing is done on a shoe string
and that is one of its main limiting factors.

>  This is understood and the intention would be for the Xen Project 
> (aka Advisory Board) to fund a Linux Foundation employee to do this on 
> behalf of the Xen Project: this is a bit like Greg KH and others being 
> LF employees working on the kernel. Some vendors on the Advisory Board 
> indicated that providing Colo/hosting space and HW would be possible in 
> principle, which could help keeping the cost manageable.

We should certainly be taking them up on those offers IMHO.

> == Access ==
> 
> Any central system, has of course the issue of access control and 
> managing users. This is obviously a barrier to entry (if we do not have 
> also a local test mechanism). Am wondering how other FOSS communities 
> handle this. This should certainly be the job of the Test Framework 
> owner (see above).

At a minimum it ought to be possible to allow access to any employee of
a project member, since we have the opportunity through the membership
process to put whatever paperwork and agreements (acceptable use etc) in
place.

Unfettered access for anyone who rocks up and asks is a bit trickier.
I'm quite happy to let that be the framework owner's problem ;-)

Ian.