Re: [Xen-devel] [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ...

[Ian Jackson <ian.jackson@xxxxxxxxxx>]

Doug Goldstein writes ("Re: [Xen-devel] [Notes for xen summit 2018 design 
session] Process changes: is the 6 monthly release Cadence too short, Security 
Process, ..."):
> You effectively supported my point in the end. People value their time.
> Giving them details about trends could help folks to look at test
> failures that have "interesting" failures.

Your focuse on the notion of "trend" here is interesting.

One way of looking at that is that really you are asking about change
over time.  Of course that's what the "regression" concept is in
osstest.

But that is for each individual test step.  When we have a heisenbug
which affects multiple tests, osstest is not really right now very
good at aggregating that information.

Maybe your "trend" idea is useful here.  osstest could perhaps track
the proportion of "heisen" failures somehow.  I will have to think
about how to do that.  Ie, exactly how to calculate the numerator and
denominator.

Do we want to track that over flights that got pushes, or something,
only ?  ISTM that, for example, if master==staging~2, and tests of
staging~1 were a wipeout because of a regression fixed in staging~0,
then when we report the "trend" in the test report of staging~0, that
should disregard the disaster that was the test(s) of staging~1.

And of course I should be asking a different question entirely if we
decide to move to multiple, rewinding, input branches, rather than a
single fast-forwarding one.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel