Re: [Xen-devel] [RFC] Results of Phase 1 of the Review Process study

["Jesus M. Gonzalez-Barahona" <jgb@xxxxxxxxxxxx>]

On Thu, 2015-10-15 at 22:36 +0100, Lars Kurth wrote:
> > On 15 Oct 2015, at 10:26, Ian Campbell <Ian.Campbell@xxxxxxxxxx>
> > wrote:
> > 
> > On Thu, 2015-10-15 at 10:06 +0100, Ian Campbell wrote:
> > > On Wed, 2015-10-14 at 18:32 +0100, Lars Kurth wrote:
> > > > C1) Only 60% percent of the reviews on the mailing list could
> > > > be
> > > > matched
> > > > to commits. This can be improved going forward, but we felt
> > > > that the
> > > > dataset is big enough for statical analysis and didn't want to
> > > > spend
> > > > too
> > > > much time to get the matching perfect at this stage. See
> > > > "Coverage
> > > > analysis" for more details
> > > 
> > > How strict or fuzzy is the matching?
> > > 
> > > Does it account for e.g. spelling, grammar and clarity changes
> > > and things
> > > like adding a subsystem ("tools: libxc:") prefix, either upon
> > > commit or
> > > by
> > > the author in vN+1 based on feedback?
> > > 
> > > I often both comment on such things during review and (with the
> > > authors
> > > permission) tweak things upon commit.
> > > 
> > > If those changes are not being correlated then I expect that
> > > would skew
> > > the
> > > figures of those for whom English is not their first language
> > > (and not a
> > > small portion of native speakers even!) and newcomers who e.g.
> > > might not
> > > be
> > > aware of the need to prefix things with the subsystem.
> > > 
> > > In a (smaller) number of cases a patch is abandoned in favour of
> > > a very
> > > different approach, which I think would be essentially
> > > untrackable, at
> > > least automatically.
> > 
> > Looking at the stuff in [47] marked as last reviewed in 2014 it
> > seems the
> > majority of them (at least the ones for which I am involved as a
> > maintainer
> > etc) can be explained by one of these factors, just going from my
> > memory of
> > things having been fixed in one way or another.
> 
> I think you are right: we hardly spent any time on more intelligent
> matching.

Yes. We tried to get to a meaningful sample, assuming the skew was
small enough to draw conclusions on the duration of the review process,
which was the main target at this stage. As Lars mentions in some other
message, the nice thing is that once we improve the matching
heuristics, the rest of the analysis can be run automatically, which
means we would get more accurate results.

At this point, without more careful validation, we started to be afraid
of having false positives (relaxing the matching rules to a point where
they start matching messages and commits that are not really the same).

> > There also looks to be identical titles (e.g. "x86: Full support of
> > PAT")
> > being listed there more than once.
> 
> Will have to look at this one

Yes. Maybe some whitespace difference or something...

> [...]

        Jesus.

-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel