Re: [Xen-devel] Testing for the Xen Project

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On ven, 2013-11-29 at 16:52 +0000, Lars Kurth wrote:
> Dario,
> that makes sense. 
>
Good to know. :-)

> This shouldn't be a OSSTest vs. XenRT discussion.
>
Sure, and I neither read/interpreted it that way, nor I wanted to turn
it into something like that. If it sounded like I did, sorry, that
wasn't my intention.

As I said, I only have experience with OSSTest, and I felt like I waned
to share thoughts and opinions coming out from such experience, about
what it can and can't (right now) do and be used for, that's it. :-)

> On 27/11/2013 03:56, Dario Faggioli wrote:
> > I don't see how the job of checking, as thoroughly as possible, hat
> > feature X does not break other --even completely unrelated-- features,
> > does what is expected on a number of platform and architecture that are
> > either similar or totally different from my own one can't be demanded to
> > 'system testing'. I don't think there is a way for every developer to
> > have every of its patch series tested on 5 or 10 different boxes, in all
> > the PV, HVM and PVH variants, etc.! :-O
> I don't think I am suggesting that we should always run every patch 
> against all machines before it is submitted.
> 
Right, I got that in the first place. Point is, as you suggest yourself
below, if we go that way, there's some either some formal policing or
'rule-of-thumb'-ing needed, and I just think it's going to be an hard
call, to the point that I'm not sure it's even worth.

As I tired to say, I'd rather invest that effort in making OSSTest (but
I suspect that would apply to XenRT too) easier to "deploy", or, if you
want, in teaching people how to do that, so that they can run it in
*their* homes/offices, perform their own testing by writing actual
testcases for it and, perhaps, submit them to the "official" testing
service / push gate (e.g., as something that runs once in a while,
instead of every night, depending on the feature being tested).

> But maybe it is a way to de-risk feature development that is inherently 
> more risky, such as HVM, or other low-level stuff.
> Or for areas, where we do not have as many reviewers as we would like.
> It may make HW accessible to some people, that they may not normally 
> have access to.
>
Exactly, and there are some nice examples of policy items. Again, I'm
certainly not against it, but I don't thing this is the main point of
this discussion (and so, sorry for making some fuss about it myself in
the first place), so let's leave it and go ahead. :-)

> >> [snip]
> >> == OSSTest ==
> >>
> >> What runs now and thus easiest to get started on
> >>
> >> [snip]
> >>
> >> Problems:
> >> * Runs on Citrix premises (thus general access is an issue)
> >> * Ian Jackson is acting as sys-admin in his spare time. But, the
> >> Advisory Board could provide resource to fix this
> > Well, these are not real issues, are they? I mean, yes, that's the
> > current situation, but if we decide to go for OSSTest, both
> > infrastructure and people will be allocated properly, making these
> > disappear.
> Agreed. Which is why I added this, such that it doesn't get forgotten.
> 
Perfect then. Given that, I probably would move this and the other
similar arguments away from the 'Problems:' section (and of course do
the same for the same arguments in the XenRT paragraph), as, although
it's important not to forget them, they're not that relevant in the
discussion/decision phase, are they?

That is, BTW, all I meant with this (and the other similar) comment(s).

> >> Risks
> >> * Not well understood (maybe you guys can fill the gaps)
> > Again, true, although the fact that the code is there, and that is in an
> > actual repo, with all the history, etc., and the fact that we're
> > starting to see patches on xen-devel, is already and will continue to
> > clarify what OSSTest does and how it does it quite a lot.
> >
> > Looking back at my experience, for instance, something that would have
> > helped me / saved me some time (and some questions to Ian on IRC) is a
> > clear definition of what 'flights', 'jobs', 'recipes' and 'testcases'
> > are, and what relationships they share among each others.
> Again, I raised this such that the existing users of OSSTest can flesh 
> this section out and provide a bit more clarity. It's basically a 
> place-holder that you guys should flesh out.
> 
Which is exactly what I tried to do with my e-mail, all of it, but
particularly here. :-)

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part