Re: [Xen-devel] Testing for the Xen Project

[Lars Kurth <lars.kurth@xxxxxxx>]

Dario,
that makes sense. This shouldn't be a OSSTest vs. XenRT discussion. I 
added a few of comments...
Lars

On 27/11/2013 03:56, Dario Faggioli wrote:
> > *Local testing*: the basic idea here is for a developer to write their
> > code and be able to run some tests on their machine locally as part of
> > their development workflow. We do have same capability to do this with
> > OSSTest
> > (seehttp://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/),
> > but the drawback is that this only works on the developers machine. This
> > may be good enough. We could âemulateâ different environments using
> > QEMU, but that is likely to be slow.
> Honestly, I don't believe this kind of testing needs to be that broad
> and/or that general, which is a good thing, as I don't think it would be
> possible for it to be that way! ;-P
>
> What I mean is, before starting to work on feature X, I usually make
> sure that I will have access to a platform reasonably sensitive to
> feature X. Before sending, I do test feature X on such platform, to make
> sure the code serves its purpose / match with the original design, in an
> testing environment (which in this case includes both the test platform
> and the kind of test I execute) meant at highlight exactly that.
This is certainly true, for new feature development such as NUMA.

> I don't see how the job of checking, as thoroughly as possible, hat
> feature X does not break other --even completely unrelated-- features,
> does what is expected on a number of platform and architecture that are
> either similar or totally different from my own one can't be demanded to
> 'system testing'. I don't think there is a way for every developer to
> have every of its patch series tested on 5 or 10 different boxes, in all
> the PV, HVM and PVH variants, etc.! :-O
I don't think I am suggesting that we should always run every patch 
against all machines before it is submitted.
But maybe it is a way to de-risk feature development that is inherently 
more risky, such as HVM, or other low-level stuff.
Or for areas, where we do not have as many reviewers as we would like.
It may make HW accessible to some people, that they may not normally 
have access to.
There could be other potential uses.

If we had the capability of running some tests on different hardware 
before a patch is submitted, wouldn't that speed up the review-commit 
cycle. Each bug/issue that is found after a system test failure, will 
lead to a re-review and will consume time and effort. As we are moving 
to shorter release cycles, this sounds like something that would be very 
valuable.
It is also quite conceivable that this line of argument is a "red 
herring" (and that I stated a problem which does not actually exist). I 
guess it depends on how often we break functionality unintentionally in 
practice. I do think that our community is actually quite good and 
thorough when it comes to reviewing patches.
> All this to say that, I think 'local testing' is ok to stay on the
> developers' machine. What should be done, from a workflow perspective,
> is (and we've started trying to pull in that direction) require that,
> along with feature X, a testcase for whatever the 'system testing' will
> end up being to be submited. That way, breakage of feature X on weird
> archs, as well as feature X either breaking or being broken (in future)
> by other features would come, again as a part of 'system testing'.
>
> About the actual tools, I've been able to install and use OSSTest
> standalone mode inside the Citrix network (i.e., the easy part :-)) and
> am halfway through setting it up here at my place, for using it to
> manage my (only!) testbox. It's not super easy yet, it probably can be
> improved, but it's _there_ right now and it's _working_. The fact that
> such a thing is possible is, I think, of paramount importance to allow
> people to familiarize with the testing framework if they want, and
> becomes absolutely fundamental if we want to encourage (not to mention
> request, more or less formally) developers to submit test cases for
> their stuff.
Totally agreed.

> > [snip]
> > == OSSTest ==
> >
> > What runs now and thus easiest to get started on
> >
> > [snip]
> >
> > Problems:
> > * Runs on Citrix premises (thus general access is an issue)
> > * Ian Jackson is acting as sys-admin in his spare time. But, the
> > Advisory Board could provide resource to fix this
> Well, these are not real issues, are they? I mean, yes, that's the
> current situation, but if we decide to go for OSSTest, both
> infrastructure and people will be allocated properly, making these
> disappear.
Agreed. Which is why I added this, such that it doesn't get forgotten.

> > * Basic test coverage
> True, but I never checked how may and what test coverage XenRT would
> provide, perhaps someone could put this information somewhere.
>
> For OSSTest, it's all here (isn't that Ian?):
> http://www.chiark.greenend.org.uk/~xensrcts/logs/22099/
> > * Not a lot of documentation right now (which is a bit of a barrier to
> > adoption)
> Yes, that is definitely true, the biggest issue with it (but again, I
> can't really compare, since I never really went checking how much docs
> there is for XenRT).
>
> I think, a similar very strict policy about improving and keeping
> updated the doc, similar to what we introduced for Xen, would be
> required here too, if we go for OSSTest. If we do that, I'm sure things
> will improve.
Same point: it is an issue today and it could be resolved with adequate 
funds, focus and will. But the issue ought to be mentioned.
> > Risks
> > * Not well understood (maybe you guys can fill the gaps)
> Again, true, although the fact that the code is there, and that is in an
> actual repo, with all the history, etc., and the fact that we're
> starting to see patches on xen-devel, is already and will continue to
> clarify what OSSTest does and how it does it quite a lot.
>
> Looking back at my experience, for instance, something that would have
> helped me / saved me some time (and some questions to Ian on IRC) is a
> clear definition of what 'flights', 'jobs', 'recipes' and 'testcases'
> are, and what relationships they share among each others.
Again, I raised this such that the existing users of OSSTest can flesh 
this section out and provide a bit more clarity. It's basically a 
place-holder that you guys should flesh out.