Re: [Xen-devel] osstest commits and Xen releases

[Juergen Gross <jgross@xxxxxxxx>]

On 29/01/2019 13:43, Ian Jackson wrote:
> Juergen Gross writes ("OSStest commits and Xen releases"):
>> I have found an alarming tendency regarding changes in the OSStest
>> repository: over the last 2 years (or 3 Xen versions) there has been
>> a pattern of OSStest commits being more frequent during the RC phase
>> of a Xen release. On average there were about 4 commits to osstest.git
>> per week. The numbers were significantly higher during RC-phases:
>>
>> Version   RC-phase                 OSStest commits per week
>> 4.12      2019/01/16 -             19
>> 4.11      2018/04/17 - 2018/07/09  10
>> 4.10      2017/10/16 - 2017/12/13  6
>>
>> I have looked at this as I would have liked to cut 4.12-RC2 this
>> Monday, but OSStests for xen-unstable failed over the weekend. Ian
>> suspected a change in OSStest to be blamed (needs to be verified).
>>
>> As the release manager I don't like RCs being delayed due to changes
>> in our infrastructure. For Xen we have code freeze and patches to go in
>> need the release manager's ack. Shouldn't the same apply to OSStest?
>>
>> I like OSStest very much as it helps catching bugs early. But I believe
>> the main development should not be done in the time when we need it's
>> results to be most reliable.
>>
>> Thoughts?
> 
> 
> Thanks for raising this.  I have three lines of response.
> 
> 
> Firstly, in the most general case: I think you have a point.

Thanks for the confirmation.

> (I think this effect is probably due to changes which had been starved
> of effort due to the impending Xen freeze being unblocked, but I would
> have to do a full chart to be sure.)
> 
> I suggest we improve this by adopting a release ack system for pushes
> to osstest pretest after the Xen codefreeze date.  In practice it will
> sometimes be necessary to make changes quickly (eg debian-installer
> kernel updates) so I think I (as ossteset maintainer) would need some
> discretion to waive the need for a release ack or to make one myself,
> but that would certainly involve informing you, and asking your
> opinion if you are available.

I absolutely agree. We might want to pull the stable maintainer in
here by giving him the right to opt-in for making his Ack mandatory,
too.
> Another possibility would be to arrange for xen-unstable to have its
> own separate branch of osstest, so that xen-unstable's runs can be
> detached from the rest.  I think while this is technically possible it
> is not worth the additional complexity (admin hassle, risk of
> confusion, work to reconcile branches, etc. etc.)

We should use this solution only if the release ack system isn't
working.

> Do you think a release ack should be needed for commissioning new
> hardware ?

I'd prefer that, yes. Normally the release ack would be given in
this case, but I'd like to be able to delay such commissioning e.g.
if the release is waiting only for the last OSStest run and the new
hardware is only adding risk (and bandwidth, of course) but no new
aspects.

> Secondly, on this specific set of changes, looking at it from the
> point of view of whether such a release ack ought to have been
> forthcoming:
> 
> We have been having hardware failures.  Particularly, we have been
> having PDU port failures which I am fairly sure are due to the high
> frequency with which we use the PDU relays to hard power cycle the
> machines.  We have also had a higher rate of other hardware problems
> than I think would be to be expected, which might be related.
> 
> These PDU relay problems themselves lead to osstest unreliability and
> of course the longer the situation goes on the more stuff breaks.
> 
> So I think that for these changes a release ack should probably have
> been granted although perhaps additional formal testing (or some other
> assurance) would have or should been done - see below.

Especially in the early RC-phase I have no real problem with such
breakage. I guess I would have given my Rab, as delaying those fixes
would have been worse (imagine a massive PDU breakage around the
planned release date). Later in the release schedule I'd be more
restrictive, of course.

> Thirdly, in this case these recent changes were in fact not anything
> to do with the fact that we didn't get a push over the weekend.
> Looking at the recent flights, the first of the changes I made at the
> end of last week took effect in 132504 (which reported late on
> Monday).

Thanks for the thorough analysis (I'm dropping the details here).


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel