[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Testing for the Xen Project



Dario,
that makes sense. This shouldn't be a OSSTest vs. XenRT discussion. I added a few of comments...
Lars

On 27/11/2013 03:56, Dario Faggioli wrote:
*Local testing*: the basic idea here is for a developer to write their
code and be able to run some tests on their machine locally as part of
their development workflow. We do have same capability to do this with
OSSTest
(seehttp://blog.xen.org/index.php/2013/09/30/osstest-standalone-mode-step-by-step/),
but the drawback is that this only works on the developers machine. This
may be good enough. We could âemulateâ different environments using
QEMU, but that is likely to be slow.

Honestly, I don't believe this kind of testing needs to be that broad
and/or that general, which is a good thing, as I don't think it would be
possible for it to be that way! ;-P

What I mean is, before starting to work on feature X, I usually make
sure that I will have access to a platform reasonably sensitive to
feature X. Before sending, I do test feature X on such platform, to make
sure the code serves its purpose / match with the original design, in an
testing environment (which in this case includes both the test platform
and the kind of test I execute) meant at highlight exactly that.
This is certainly true, for new feature development such as NUMA.

I don't see how the job of checking, as thoroughly as possible, hat
feature X does not break other --even completely unrelated-- features,
does what is expected on a number of platform and architecture that are
either similar or totally different from my own one can't be demanded to
'system testing'. I don't think there is a way for every developer to
have every of its patch series tested on 5 or 10 different boxes, in all
the PV, HVM and PVH variants, etc.! :-O
I don't think I am suggesting that we should always run every patch against all machines before it is submitted.

But maybe it is a way to de-risk feature development that is inherently more risky, such as HVM, or other low-level stuff.
Or for areas, where we do not have as many reviewers as we would like.
It may make HW accessible to some people, that they may not normally have access to.
There could be other potential uses.

If we had the capability of running some tests on different hardware before a patch is submitted, wouldn't that speed up the review-commit cycle. Each bug/issue that is found after a system test failure, will lead to a re-review and will consume time and effort. As we are moving to shorter release cycles, this sounds like something that would be very valuable.

It is also quite conceivable that this line of argument is a "red herring" (and that I stated a problem which does not actually exist). I guess it depends on how often we break functionality unintentionally in practice. I do think that our community is actually quite good and thorough when it comes to reviewing patches.

All this to say that, I think 'local testing' is ok to stay on the
developers' machine. What should be done, from a workflow perspective,
is (and we've started trying to pull in that direction) require that,
along with feature X, a testcase for whatever the 'system testing' will
end up being to be submited. That way, breakage of feature X on weird
archs, as well as feature X either breaking or being broken (in future)
by other features would come, again as a part of 'system testing'.

About the actual tools, I've been able to install and use OSSTest
standalone mode inside the Citrix network (i.e., the easy part :-)) and
am halfway through setting it up here at my place, for using it to
manage my (only!) testbox. It's not super easy yet, it probably can be
improved, but it's _there_ right now and it's _working_. The fact that
such a thing is possible is, I think, of paramount importance to allow
people to familiarize with the testing framework if they want, and
becomes absolutely fundamental if we want to encourage (not to mention
request, more or less formally) developers to submit test cases for
their stuff.
Totally agreed.

[snip]
== OSSTest ==

What runs now and thus easiest to get started on

[snip]

Problems:
* Runs on Citrix premises (thus general access is an issue)
* Ian Jackson is acting as sys-admin in his spare time. But, the
Advisory Board could provide resource to fix this
Well, these are not real issues, are they? I mean, yes, that's the
current situation, but if we decide to go for OSSTest, both
infrastructure and people will be allocated properly, making these
disappear.
Agreed. Which is why I added this, such that it doesn't get forgotten.

* Basic test coverage

True, but I never checked how may and what test coverage XenRT would
provide, perhaps someone could put this information somewhere.

For OSSTest, it's all here (isn't that Ian?):
http://www.chiark.greenend.org.uk/~xensrcts/logs/22099/
* Not a lot of documentation right now (which is a bit of a barrier to
adoption)

Yes, that is definitely true, the biggest issue with it (but again, I
can't really compare, since I never really went checking how much docs
there is for XenRT).

I think, a similar very strict policy about improving and keeping
updated the doc, similar to what we introduced for Xen, would be
required here too, if we go for OSSTest. If we do that, I'm sure things
will improve.
Same point: it is an issue today and it could be resolved with adequate funds, focus and will. But the issue ought to be mentioned.

Risks
* Not well understood (maybe you guys can fill the gaps)
Again, true, although the fact that the code is there, and that is in an
actual repo, with all the history, etc., and the fact that we're
starting to see patches on xen-devel, is already and will continue to
clarify what OSSTest does and how it does it quite a lot.

Looking back at my experience, for instance, something that would have
helped me / saved me some time (and some questions to Ian on IRC) is a
clear definition of what 'flights', 'jobs', 'recipes' and 'testcases'
are, and what relationships they share among each others.
Again, I raised this such that the existing users of OSSTest can flesh this section out and provide a bit more clarity. It's basically a place-holder that you guys should flesh out.




 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.