Re: [Xen-devel] [RFC] Results of Phase 1 of the Review Process study

[Lars Kurth <lars.kurth.xen@xxxxxxxxx>]

> On 15 Oct 2015, at 10:26, Ian Campbell <Ian.Campbell@xxxxxxxxxx> wrote:
> 
> On Thu, 2015-10-15 at 10:06 +0100, Ian Campbell wrote:
>> On Wed, 2015-10-14 at 18:32 +0100, Lars Kurth wrote:
>>> C1) Only 60% percent of the reviews on the mailing list could be
>>> matched
>>> to commits. This can be improved going forward, but we felt that the
>>> dataset is big enough for statical analysis and didn't want to spend
>>> too
>>> much time to get the matching perfect at this stage. See "Coverage
>>> analysis" for more details
>> 
>> How strict or fuzzy is the matching?
>> 
>> Does it account for e.g. spelling, grammar and clarity changes and things
>> like adding a subsystem ("tools: libxc:") prefix, either upon commit or
>> by
>> the author in vN+1 based on feedback?
>> 
>> I often both comment on such things during review and (with the authors
>> permission) tweak things upon commit.
>> 
>> If those changes are not being correlated then I expect that would skew
>> the
>> figures of those for whom English is not their first language (and not a
>> small portion of native speakers even!) and newcomers who e.g. might not
>> be
>> aware of the need to prefix things with the subsystem.
>> 
>> In a (smaller) number of cases a patch is abandoned in favour of a very
>> different approach, which I think would be essentially untrackable, at
>> least automatically.
> 
> Looking at the stuff in [47] marked as last reviewed in 2014 it seems the
> majority of them (at least the ones for which I am involved as a maintainer
> etc) can be explained by one of these factors, just going from my memory of
> things having been fixed in one way or another.

I think you are right: we hardly spent any time on more intelligent matching.

> There also looks to be identical titles (e.g. "x86: Full support of PAT")
> being listed there more than once.

Will have to look at this one

> Lastly there is at least one ("Introducing Xen PV block driver to OVMF")
> which was against another project.

That is correct and a case we need to look at if it's worthwhile and possible 
to fix it. However the fact remains that pretty much all the graphs cover data 
for completed reviews only, except for backlog data. We have two options
1) Spend some effort trying to fix it
2) Accept that "stalled" reviews are not that meaningful
Not sure what the answer is at this stage

> I'm afraid that on that basis I think C1 has skewed the conclusion that
> there are 600 stalled series, possibly by a considerable factor.

I agree with that and should have maybe highlighted this more clearly.

For the other data, it is of course also possible that there is some skew. But 
given that we had 60% of a sample (and that the 60% is likely higher as 
xen-devel also is CC'ed when patches are posted to QEMU, Linux, ...) the sample 
is big enough to make significant skew very unlikely.

Lars 


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel