Re: [Xen-devel] Testing for the Xen Project

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

Premise/Disclaimer: I reply because I'm one of the ones that played (is
playing, actually) with OSSTest, since when standalone mode become
available, so I think I may have an opinion on it.

Unfortunately, I don't have direct experience with XenRT, and about it,
I only know what I've learn in talks and presentations, and talking with
people that uses it at Citrix.

For that reason, I wouldn't go as far as providing any sort of
comparison, as I only know one of the twos, and that would be unfair, if
at all possible.

On mar, 2013-11-26 at 13:21 +0000, Lars Kurth wrote:
> == Work Flow ==

> *Local testing*: the basic idea here is for a developer to write their 
> code and be able to run some tests on their machine locally as part of 
> their development workflow. We do have same capability to do this with 
> OSSTest 
> (seehttp://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/),
>  
> but the drawback is that this only works on the developers machine. This 
> may be good enough. We could âemulateâ different environments using 
> QEMU, but that is likely to be slow.
> 
Honestly, I don't believe this kind of testing needs to be that broad
and/or that general, which is a good thing, as I don't think it would be
possible for it to be that way! ;-P

What I mean is, before starting to work on feature X, I usually make
sure that I will have access to a platform reasonably sensitive to
feature X. Before sending, I do test feature X on such platform, to make
sure the code serves its purpose / match with the original design, in an
testing environment (which in this case includes both the test platform
and the kind of test I execute) meant at highlight exactly that.

I don't see how the job of checking, as thoroughly as possible, hat
feature X does not break other --even completely unrelated-- features,
does what is expected on a number of platform and architecture that are
either similar or totally different from my own one can't be demanded to
'system testing'. I don't think there is a way for every developer to
have every of its patch series tested on 5 or 10 different boxes, in all
the PV, HVM and PVH variants, etc.! :-O

All this to say that, I think 'local testing' is ok to stay on the
developers' machine. What should be done, from a workflow perspective,
is (and we've started trying to pull in that direction) require that,
along with feature X, a testcase for whatever the 'system testing' will
end up being to be submited. That way, breakage of feature X on weird
archs, as well as feature X either breaking or being broken (in future)
by other features would come, again as a part of 'system testing'.

About the actual tools, I've been able to install and use OSSTest
standalone mode inside the Citrix network (i.e., the easy part :-)) and
am halfway through setting it up here at my place, for using it to
manage my (only!) testbox. It's not super easy yet, it probably can be
improved, but it's _there_ right now and it's _working_. The fact that
such a thing is possible is, I think, of paramount importance to allow
people to familiarize with the testing framework if they want, and
becomes absolutely fundamental if we want to encourage (not to mention
request, more or less formally) developers to submit test cases for
their stuff.

> *Test on demand:* this would be a mixture between local testing and 
> system testing. It can probably be implemented in both OSSTest and 
> XenRT, although I believe XenRT is a closer to this model already. 
>
Well, if this will become a real chance, considering that *a*lot* of
people would possibly wan to run complex and lengthy tests, unless the
amount of hardware we'll have is unlimited, it looks to me that a job
scheduler would be quite a nice thing to have.

As I stated already, I know next to nothing of XenRT, and what I know
about OSSTest is mostly about standalone mode, so I don't know who's
ahead, I'll just say that I really think we need something good at doing
this (test job scheduling) in he final solution.

> The 
> workflow would look like this:
> a) code developed locally
> b) developer checks working prototype into their personal git branch 
> (which would need to be accessible from the web)
> c) developer submits a job to the test farm which checks out and builds 
> code and runs a set of interesting (or new) tests on some machines of 
> different architectures. New tests would probably need to be on the git 
> branch
> d) if the test fail, the cycle starts again
> d) if all works well, code is submitted for review as normal (and test 
> results could be attached)
>
Modulo scheduling test jobs (covered above), for what I know OSSTest can
handle all of this.

> == OSSTest ==
> 
> What runs now and thus easiest to get started on
> 
> More Info
> *http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/
> *http://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/
> *http://www.youtube.com/watch?v=JxTFZIwZzJ8
> 
> Problems:
> * Runs on Citrix premises (thus general access is an issue)
> * Ian Jackson is acting as sys-admin in his spare time. But, the 
> Advisory Board could provide resource to fix this
>
Well, these are not real issues, are they? I mean, yes, that's the
current situation, but if we decide to go for OSSTest, both
infrastructure and people will be allocated properly, making these
disappear.

> * Basic test coverage
>
True, but I never checked how may and what test coverage XenRT would
provide, perhaps someone could put this information somewhere.

For OSSTest, it's all here (isn't that Ian?):
http://www.chiark.greenend.org.uk/~xensrcts/logs/22099/

> * Not a lot of documentation right now (which is a bit of a barrier to 
> adoption)
> 
Yes, that is definitely true, the biggest issue with it (but again, I
can't really compare, since I never really went checking how much docs
there is for XenRT).

I think, a similar very strict policy about improving and keeping
updated the doc, similar to what we introduced for Xen, would be
required here too, if we go for OSSTest. If we do that, I'm sure things
will improve.

> Risks
> * Not well understood (maybe you guys can fill the gaps)
> 
Again, true, although the fact that the code is there, and that is in an
actual repo, with all the history, etc., and the fact that we're
starting to see patches on xen-devel, is already and will continue to
clarify what OSSTest does and how it does it quite a lot.

Looking back at my experience, for instance, something that would have
helped me / saved me some time (and some questions to Ian on IRC) is a
clear definition of what 'flights', 'jobs', 'recipes' and 'testcases'
are, and what relationships they share among each others.

> == XenRT ==
>
And I'm not saying anything about it, as I don't like speaking of things
I don't know. :-(

> == Support and Ownership ==
> 
> Whatever solution we go for, needs to be properly funded and looked 
> after. This is understood and the intention would be for the Xen Project 
> (aka Advisory Board) to fund a Linux Foundation employee to do this on 
> behalf of the Xen Project: this is a bit like Greg KH and others being 
> LF employees working on the kernel. 
>
Or, perhaps a better example, like John Hawley which, when acting as the
chief of maintenance of the kernel.org infrastructure, is also payed by
the Linux Foundation (https://www.linkedin.com/in/warthog9).

Ok... Done. Hope my small contribution can be of help in the
discussion. :-)

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part