[Xen-devel] Notes Design Session: Making Releases Lessons Learned: Improving Our Release Process and Tooling

[Lars Kurth <lars.kurth.xen@xxxxxxxxx>]

Hi all,
these are my notes from http://sched.co/AjES
These are a little crude and I may have missed some things or misrepresented

ACTIONS for
* Wei Liu
* Julien Grall
* Lars Kurth
* Ian Jackson

Feel free to correct
Lars

Releases Lessons learned
========================

Process issues in 4.9.0 (checklists, guides, templates)
-------------------------------------------------------

Ian:
* Lots of fiddly bits need to be executed (right now we have a release 
technician checklist)
* We don't have a comprehensive checklist that is useful for Release Managers
* Started the final steps for 4.9.0 on Tue of the release week
* By this tine we were committed to releasing on Wed (no point of return due to 
PR)
* Some items were release managers job, but discovered by Release Technician 
(Ian)
* Release technician checklist is in xen.git

ACTION (Wei, Julien, Lars): Have a how to be release manager file (needs to 
contain some of the stuff in Release technician checklist)
 * RM file: clear set of criteria on PR. Go/NoGo part
 * See 
https://lists.xenproject.org/archives/html/xen-devel/2017-07/threads.html#01500 
for a starting point

ACTION (Lars/Wei/Julien) to write "standard e-mail templates for common stuff" 
rather than re-doing these every single time
 
ACTION (Ian): Clean up release technician checklist after we have the how to be 
release manager file

ACTION (Wei/Julien): Additional stuff to add to the templates/RM guide
* Add hand-over of tasks for Release Manager responsibility to the "how to be 
release manager" file
* Add clear reminders in particular at the beginning of a release into e-mail 
templates: such as "put dates X,Y, Z in your calendar"
* Communicate better when tree is open again (add to checklist and templates)

ARM issues in 4.9.0
-------------------

Julien: Unreliable testing on ARM32/64 
- ongoing problems with ARM64 box (hardware issue), other ARM64 box (firmware 
issue), ARM32 Arndale hardware problem (replace)

On ARM everything is in progress and not much more can be done => no concrete 
ACTION as those already executed

x86 Heisenbugs in 4.9.0
-----------------------
- x86 migration Heisenbugs have caused us issues this release cycle: not 
completely understood the issues and hard to debug

Ian: These are most likely software problems, most likely in Xen.
ISSUE: nobody wants to debug Windows Heisenbugs
ISSUE: have nobody to do formal triage - doing back-pressure on developers

ACTION: Ian, Julien
- Manually add ignore list for specific Heisenbugs until we found another 
solution
- Requires a proposal to the list - Julien to propose


ACTION: Julien to propose 
- OSSTEST to CC people blocked/proposed on test failures instead just the list
- Release manager can say "not releasing now" because of too many bugs
  "until someone fixes these". "no more patches until XYZ" 


Improving the Process
=====================

JIRA
----
Open source projects do NOT need licenses for JIRA: these was raised by OpenXT 
folks, who use JIRA

ACTION: Lars to follow up with Atlassian - 
https://www.atlassian.com/software/views/open-source-license-request
ACTION: Lars to follow up with Intel on owner and describe simple process to 
nominate new people

We then had a brief discussion on whether to use labels or components. OpenXT 
believe components are easier to manage

ACTION: Julien - Decide on whether to use labels or components for wiki page
ACTION: Julien - Make a proposal on components on list for review (note: Lars 
could help based on 
https://lists.xenproject.org/archives/html/xen-devel/2017-07/threads.html#01590)

XSA Testing 
-----------
ACTION: include Lars'es tool into checklist

Arguments on the mailing list on XSAs
-------------------------------------
Delayed RC by two days: security patches produced by the security team - we 
don't always consult the maintainer

ACTION: committers@ + maintainer of relevant code of Advisory should be on 
pre-disclosure list 
* Explicitly say in public that security team should be informed pre-disclosure 
...
  With appropriate template

Resolved in 
https://lists.xenproject.org/archives/html/xen-devel/2017-07/threads.html#01590

Release Manager for 4.11
------------------------
Julien stepping down after 4.10 - we should find another volunteer

ACTION: Lars to sound out SUSE (Juergen Gross), Ross Phillipson (Oracle)
Note: Juergen would be happy to do this for one release







_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel