[Xen-devel] Survey Results - Part 2 : Improving Xen Project Governance and Conventions

[Lars Kurth <lars.kurth.xen@xxxxxxxxx>]

Hi all,
please find results of part 2, following on from 
http://lists.xenproject.org/archives/html/xen-devel/2015-10/msg00354.html

Results and comments are marked with | under each question. Followed by an 
analysis, which is marked with !

Best Regards
Lars

== 2) Trust amongst different stakeholders in the peer review process ==

Different stakeholders in the review process have different expectations. To 
remind us of them, for the purpose of the following discussion, I want to 
restate some of the key expectations (even if not complete)

 * Contributors expect that they get swift feedback on code contributions, 
   and are in turn expected to address feedback swiftly and comprehensively

 * Reviewers (aka Maintainers and Committers) act as gatekeepers upholding 
   code quality, coding and legal standards, architectural coherence and 
   other code, design and architectural questions. 

 * Committers see themselves as ultimate gate-keepers 

In the last year or so, an influx of new developers as well as increased 
number of contributions, appear to have put our existing review set-up under 
strain, causing frustration for all stakeholders. It appears, that the current 
set-up, does not reflect reality and does not scale. During a community post-
mortem, the following points were raised as possible root causes for issues. 
We would like to get your opinion


* <u>Some maintainers are not active</u>
  Some of the maintainers listed in the xen.git MAINTAINERS file are not 
  acting as maintainers and they are often not active in the project at all.   
  Yet, the project governance, states that MAINTAINERS are responsible to 
  âreview and approveâ changes made by contributors. 

Q2.1: Do you think that it would be acceptable to remove inactive maintainers 
from the MAINTAINERS file? Maybe we could also introduce a category of 
âpast/inactive maintainersâ. If not, please explain.

| 11 / 12 Agree
| 1 / 12 Maybe

Caveats:
| Only after pinging said maintainers. It must also be clear what it means for 
a 
| maintainer to be inactive. Note that there were a few comments along these 
lines. 

| Keeping an honour-roll of former maintainer is common practice (either in 
MAINTAINERS
| or a separate file). 

| If the pain is coming from a lack of active maintainers (may need to 
deprecate 
| unmaintained code). We need to be careful to avoid orphaning components.
| Again, there were a few comments along these lines. 

| Depends in part on the discussion below about what being a "maintainer" means.
| If it means "this person has agreed to tend and care for this part of the 
code", 
| then yes, people who are not actively doing that job should be removed.  
| If it means, "This person knows more about this code than others, you might 
want 
| to ask him", then it depends on how responsive that person is.

! It seems to me that this is uncontroversial, but that we need some sort of 
! protocol.

 * Another thesis that was raised, was that given the formal responsibilities 
   of MAINTAINERS, we may have awarded maintainership to easily in the past. 
   Thus undermining trust amongst maintainers (and in particular trust of 
   committers in some maintainers).

Q2.2: Do you think the above statement is correct or incorrect. Any additional 
views are welcome.

| 3 / 12 Agree
| 3 / 12 Disagree
| 6 / 12 Don't know or have no opinion

| I don't think there is anyone in the MAINTAINERS file who should not be 
there.  
| There are certainly people in MAINTAINERS who should not have commit 
| access. Their review is necessary but not sufficient.  But we shouldn't stop 
| such people being maintainers -- on the contrary they must be encouraged to 
take 
| part so they can learn more and get better!

| When large pieces of code get introduced (say VMX and SVM support years ago), 
it 
| is obvious that the contributors would be the better initial maintainers than
| those who would inherit maintainership of the code by default.

| I think that the creation of the committer role was a mistake. It created 
| different levels among maintainers. A second mistake was granting 
maintainership 
| too easily, de-valuing the maintainer role.

| I think there is certainly some truth in that, when given the formal
| responsibilities. I don't think we were necessarily wrong to do so in many of
| those cases, just that it has come into conflict somewhat with the formal 
| responsibilities.

! I think, we should probably discount the answers to this question. However,
! the point on "encouraging" people to take part is valid. I am not sure,
! how actively we do this today and whether maybe a clearer ladder with more 
! rungs (right now we have "maintainers" and "committers" with quite a big gap
! in between) would help and motivate people. It may be worthwhile to compare
! contributions / review stats after each release cycle and compare with the
! maintainer file and then invite people to become maintainers. Maybe I also 
need 
! to better report what people contribute in 
! http://wiki.xenproject.org/wiki/Xen_Project_4.6_Acknowledgements
! (e.g. add reviewers and highlight acked-by, reviewed-by and number of review 
! comments, patches/series commented on - assuming we can track this with the 
! new tools).

! The comment on creating the "committer role was a mistake" is somewhat 
worrying 
! and also confusing, given that there seems to be agreement that it is 
consensus
! that a hierarchy is a good thing. So maybe this was more about labels.

 * Other projects such as QEMU do not have formal roles for maintainers. 

   E.g. https://github.com/qemu/qemu/blob/master/MAINTAINERS &    
   http://wiki.prplfoundation.org/wiki/QEMU_Governance_Model

   <quote from https://github.com/qemu/qemu/blob/master/MAINTAINERS>

   QEMU Maintainers
   ================

   The intention of this file is not to establish who owns what portions of 
   the code base, but to provide a set of names that developers can consult 
   when they have a question about a particular subset and also to provide a 
   set of names to be CC'd when submitting a patch to obtain appropriate 
   review.

   </quote>

   Note that projects such as LIBVIRT (see http://goo.gl/nv5GdD &
   https://libvirt.org/governance.html do not have the role of MAINTAINERS 
   altogether.

   Such an approach would most likely require increasing the number of 
   Committers (as we would have too few) and would either require to reduce 
   the scope of maintainership or remove it altogether from the governance 
   document.

Q2.3: Do you believe, that a less formal approach to maintainership may be 
helpful in building more trust and help solve our scalability issues? If so, 
how and why? 

| 3 / 12 Agree
| 6 / 12 Disagree
| 3 / 12 Don't know or have no opinion

| It would probably go some way toward solving scalability assuming maintainers 
| are the bottleneck, since somebody else can step up more quickly.

| Someone must the role of deciding when a patch is ready, otherwise we will
| end up with quality issues.
| Given the complexity of Xen I don't think this would be a good idea.

! There were a few other comments along those lines. And this is also
! consistent with part 1 of the review process study.

| I think this applies to the Linux MAINTAINERS file (although not explicitly 
| stated). In fact, QEMU and Linux both only have a single committer 
| (Peter Maydel and Linus respectively). It is not clear to what extent a single
| committer is a necessary consequence of the model. In Linux the hierarchy 
| we discussed is not encoded in the MAINTAINERS file but is essentially 
understood 
| by the maintainers knowing their place in the hierarchy (i.e. know who their 
| upstream is and who their downstreams are).

! This does not appear to be actionable. There seems to be a majority that we
! have enough or maybe not enough formality. What I read from the Linux 
MAINTAINERS
! comment, is that in some sense in Linux there is actually more degree of 
formality
! than in Xen in reality, even though this is not clearly stated and evolved 
! organically. So maybe the "agree" responds are not so far off from the ones
! who "disagree".

| I think we need more well defined roles, not more blurred ones, and 
'maintainers' 
| are one of these "blurred" roles. Looking at MAINTAINERS, we have in effect
|
| 1) Someone with deep expertise in some areas (where they are expected
| to take responsibility and do detailed review etc.). They also have a broad
| understanding of the whole system, and are trusted to make difficult 
judgements,
| help set overall strategic direction, ...  
|
| 2) Someone with an interest in and expertise in an area of code, who we
| will normally be consulted, and whose ack or review should be influential.

! What was interesting to me was that the respondent who made this comment
! equated 1 with the committer role, whereas others responding to this question 
in
! part 1 of the survey felt that actually influence comes/should come from 
seniority/
! trust/length of active service as a maintainer. But I am pretty sure that a 
lot of 
! maintainers who are not committers also feel that they do some or all of 1 at 
least
! some of the time. It also strikes me that maybe 2) could be split into those 
who 
! consistently are active in the community and those which are so on an on-off 
basis.

| I don't believe we need two roles: "committer" and "maintainer". A single
| role together with a short list of active and trusted maintainers based
| on contribution/review data may be enough.

! Not explicitly stated, but I suppose the assumption there was that an active
! & trusted maintainer should have commit access.

! Personally, I believe that if there was agreement - which we don't have - 
that 
! the committer role is "secretarial" and that real influence comes from being
! a trusted and active maintainer, that would open the possibility to either
! "rotate the committer part around", "elect one or two committers for a time 
! period", or for committers to "delegate committer responsibilities" for parts 
! of the tree to other trusted maintainers on trial, etc.

! That would not be inconsistent with the concept of meritocracy and also sits
! well with the hierarchy idea we pretty much all agreed with. In fact, we
! have dabbled with the concept of delegating responsibility on a time basis 
with 
! the Release Manager role and it worked quite well. Every single Release 
Manager 
! has stepped up to the challenge and brought in some new ideas and gave that
! person a chance to prove themselves and make an impact. I think there is also
! an aspirational aspect, which leads to people putting in energy and step up.

! Again I have not thought through, whether something like this would be 
practical, 
! whether there would be unintended consequences or what this means for 
governance. 
! I just wanted to put this idea out for debate, without it being fully formed.

 * Conversely, we could strengthen the definition of maintainership and adopt 
   a more formal approach to ownership that more clearly defines who owns 
   what, and who can ACK what - in other words reflecting the de-facto 
   hierarchy of maintainers more clearly. This would require reducing the 
   number of MAINTAINERS, strengthening the maintainer role and maybe re-
   think the committer role (the implication being that we would have fewer 
   maintainers that are more like committers - possibly merging the two 
   roles).

Q2.4: Do you believe that a stronger maintainer role (but having fewer 
maintainers, who are more trusted and autonomous) would be helpful in building 
more trust and help solve our scalability issues? If so, how and why? 

| 6 / 12 Agree
| 2 / 12 Maybe, but not quite clear
| 4 / 12 Disagree

| It is clear that we need more maintainers, not fewer.  We should not be 
| raising the barrier to entry.

| Maybe there should be levels of maintainership, ... similar ideas were 
| already covered earlier.

| Yes. It's about setting expectation and making sure both the contributors 
| and the maintainers themselves (at all levels of the hierarchy) know as
| precisely as possible what should be expected, from who, what falls
| within whom responsibility, etc.

| I do not think that it is possible to write down very clearly whose
| ACK is supposed to be finally valid for what part of the code,
| particularly at the less-trusted end of the community. Trying to do
| so would be a distraction and be inaccurate and lead to friction.

| I think it strongly depends on who the maintainers are. If they are
| able to trust others / able to delegate, then it may help. If they are 
| not, then it may well make it worse.

! I think I agree with the fact that we need more maintainers and reviewers
! and that raising the bar would be counter productive. The hierarchy concept, 
! may help lower the bar while strengthening and clarifying expectations
! (see my thoughts on the previous question). I do also agree that trying 
! this carries risks. I think the point on delegation is a good one, as I 
! stated earlier.

 * Other projects, define the committer role alongside several dimensions: 
   e.g. 

   <quote from http://wiki.prplfoundation.org/wiki/QEMU_Governance_Model>

   Committers will '''regularly''' evaluate patch requests with the '''goal   
   of maximum community participation''' in development while following 
   '''code standards''' and verifying that the patches comply with the IP 
   policy.''

   </quote>

   In our case, neither the committer definition nor maintainer definition, 
   covers the goal of maximising contributions, nor state the goal of 
   evaluating contributions in a timely manner. 

Q2.5: Do you believe that changing the focus within our role definitions to 
include maximising of community participation and some degree of timeliness, 
will make a difference and make the review process more efficient.

| 2 / 12 Agree
| 3 / 12 Agree, but don't think it will make a difference
| 5 / 12 Disagree
| 2 / 12 Don't know or have no opinion

! This seems to be pretty much a non-starter : the majority of respondents
! felt that it would not make a real difference. Maybe it would set some 
! expectations. But in reality it wouldn't have an effect.

 * <u>We lack a top level maintainer / project lead</u>

   We donât have anybody in the project today that can make decisions for 
   changes that affect multiple components (for example hypervisor and 
   tools). Today both hypervisor maintainers and tools maintainers have to 
   agree on all incoming changes. Our governance relies on referees in the 
   hierarchy to deal with conflicts. In addition, we do have no individual in 
   the project which could perform that role today.

Q2.6: Do you believe that we need a referee to help resolve conflicts? If not 
- recognizing it as unlikely that a single individual can perform that role - 
what are your views to use a more committee based approach? Note: Apache 
projects tend to do this via a Project Management Committee (with a chair to 
facilitate).

| 3 / 12 Agree
| 7 / 12 Disagree
| 1 / 12 Don't know or have no opinion

| I don't think we need a project lead.  We are actually pretty good at 
| resolving conflicts on the list, once they have been raised there.

! The bulk of comments were along the lines of: 
! - refereeing is hardly needed
! - if issues are raised, we seem to be good at resolving it
! - in effect we already have a committee (the committers), but in practice
!   their referring power is never invoked
!
! Some of the "Agree votes" were not quite clear:
! 2/12 were against a single leader and stated that we had a committee 
! approach already.

! It seems to me that this is an area where we don't actually have issues
! and we all agree. The only caveat was that we don't always raise issues. 
! But a referee would not fix this issue. 

Q2.7: Given that to be able to keep up with growth, scaling the number of 
reviewers/maintainers/committers (or some similar roles defined in different 
ways) is essential for the future of the project: do you have any concrete 
suggestions how we may encourage people to step up, while not affecting code 
quality and other system qualities?

! There are no yes / no answers here. So I am pasting in some relevant 
! quotes

| Think about maintainers / committers less as gatekeepers and more like guides 
| towards the gate (a more relaxed attitude to minor problems). 

| One of the biggest problems Xen has right now is that there's really no good 
| documentation for it outside of the source code

| We should consider a Quid pro quo ("something for something") for large 
| contributions. In other words, if a vendor consistently submits large patches
| without also reviewing code, their contributions should have less priority
| than those of others.

! Note: even if this was a desirable goal, I can't see how it could be 
! implemented

| Make the idea of becoming a co-maintainer more appealing by improving our 
| community processes.

| Encourage maintainers & contributors to review patches
| People doing so regularly are probably good candidates for a future 
| maintainership (building up trust).

| Do a retrospective each Xen version like we did for 4.6, but with
| a focus on subsystems: are some maintainers over-loaded, do they need
| a co-maintainer, are they not reacting due to other reasons?

| I always say I'm willing to help contributors to become maintainers if
| they are interested. Some sort of informal guideline on how the community 
| works, what the culture looks like would also be helpful.
| 
| I think your contributor training is very good for this purpose, but I
| don't remember if it covers community culture.

| For people to step up, they need to be valued and feel respected & 
| trusted. To achieve that you need to be able to delegate tasks and make
| them feel welcome.

| I am concerned about trying to force this issue too hard. In particular about
| making drastic changes. I think this is something where a longer term view 
| needs to be taken. 

| This requires that certain companies involved in the Xen community allow
| their staff to work full-time on Xen, and not jump between projects. Without 
| staff that can dedicate more time to Xen it's hard to increase the number of
| maintainers.

! There are some interesting ideas in there. I will leave them standing and
! see whether anything else comes back. I think there is also a "generational"
! angle, which we are also seeing in Linux. A lot of people tend to be more 
! attracted to some of the projects that get a lot of media attention and are
! seen as cool. Maybe also some of the points raised in Sarah Sharp's blog
! post are worth considering: see 
http://sarah.thesharps.us/2015/10/06/what-makes-a-good-community/
!
! Much of which we are not doing today, some of which seems *very* hard
! with our set of circumstances
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel