Re: [Xen-devel] [RFC] Results of Phase 1 of the Review Process study

[Lars Kurth <lars.kurth.xen@xxxxxxxxx>]

> On 15 Oct 2015, at 10:06, Ian Campbell <Ian.Campbell@xxxxxxxxxx> wrote:
> 
> On Wed, 2015-10-14 at 18:32 +0100, Lars Kurth wrote:
>> C1) Only 60% percent of the reviews on the mailing list could be matched
>> to commits. This can be improved going forward, but we felt that the
>> dataset is big enough for statical analysis and didn't want to spend too
>> much time to get the matching perfect at this stage. See "Coverage
>> analysis" for more details
> 
> How strict or fuzzy is the matching?

Not very: it doesn't deal with extra/fewer spaces, punctuation, slash direction 
changes and capitalisation. Once we make some minor changes to accommodate for 
that, we can also look at common things such as changes to the prefix 
("xen/arm", ...) or in fact ignore them. That should get us to a *far* higher 
percentage. Whether we still need to look at spelling mistakes, then remains to 
be seen.

> 
> Does it account for e.g. spelling, grammar and clarity changes and things
> like adding a subsystem ("tools: libxc:") prefix, either upon commit or by
> the author in vN+1 based on feedback?
> 
> I often both comment on such things during review and (with the authors
> permission) tweak things upon commit.

We will have to fix this later, which was already planned. I have an action to 
follow up on this in the second phase.

> If those changes are not being correlated then I expect that would skew the
> figures of those for whom English is not their first language (and not a
> small portion of native speakers even!) and newcomers who e.g. might not be
> aware of the need to prefix things with the subsystem.
> 
> In a (smaller) number of cases a patch is abandoned in favour of a very
> different approach, which I think would be essentially untrackable, at
> least automatically.

Agreed. 

>> == Backlog Analysis ==
>> This section shows us the total of patch series reviews that could be
>> modelled (60%) over the project's life-time 
> 
> How does this interact with the 60% in caveat C1? Is it the same 60% or is
> this 60% of that 60% (i.e. 36% overall)?

I don't know: the coverage data was really only added yesterday. And I have not 
had time to look into it in more detail. Thus, this is one for @Bitergia, if 
the answer is unsatisfactopry.

However, given the data we looked at 60% (which excludes more detailed data for 
the backlog), is a big enough sample for the metrics we looked at. In 
particular, most of the data covered only completed reviews. There may be some 
skew, but it should not change the bulk of what is in section 6 in any 
significant way.

I attached a sample for 2015: there are two files 
a) commits from git not matched to mails, 
b) e-mails with patches attached that are not matched to git (which is larger 
than a).

> If it is the same 60% then how are stalled series distinguished from the
> 40% which are not mapped to a commit?

This is something we still have resolve: as I mentioned, the back-log data has 
some issues and needs to be looked at again, which is partly why we didn't yet 
graph it.

The good news is that we can re-run the report on a new dataset automatically.

> Separately, I suppose it is impossible to distinguish stalled from
> abandoned (and perhaps in some senses they are the same thing so we don't
> need to distinguish).

Agreed. Unless we come up with some sort of convention, marking a series as 
abandoned in the mail thread, there is no way to find out. Which is why we came 
up with the 7 days, <12 months and >12 months buckets. Essentially assuming 
that very old reviews are abandoned: maybe we should change "stalled" with 
"likely abandoned".

Regards
Lars

Attachment: commits_not_matched_2015.csv
Description: Text Data

Attachment: patch_series_subject_with_null_commit.csv
Description: Text Data

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel