Re: [Xen-devel] RFC: LTS and stable release scheme

[George Dunlap <george.dunlap@xxxxxxxxxx>]

On 08/10/15 09:05, Jan Beulich wrote:
>>>> On 07.10.15 at 19:45, <George.Dunlap@xxxxxxxxxxxxx> wrote:
>> OK, so we have to decide on a *set* of factors, rather than just one.
>>
>> Let's try to be analytic.
>>
>> So we have so far identified 4 "parameters" that we can tweak:
>>
>> 1. Release frequency.  At the moment, this is "9 months".
>> 2. Length of time bug fixes are backported.  At the moment this is "18 
>> months".
>> 3. Length of time security fixes are backported.  At the moment this
>> is "36 months" (i.e., original 18 months of bug fixes + 18 further
>> months of security-only fixes)
>> 4. Subset of releases given treatment for #1 and #2.  At the moment
>> this is "all".
>> 5. The release frequency for maintenance release.  At the moment this
>> is every 3 months.
> 
> We switched this to 4 months not too long ago. This changes some
> of the numbers further down, but not in a way that would significantly
> affect the meat of this analysis.

Oh right -- it's still 3 months on the wiki page Wei pointed to earlier
in this thread [1]. That should probably be updated.

[1] http://wiki.xenproject.org/wiki/Xen_Project_Maintenance_Releases

> 
>> The steady state for "R9 S3 {18,36}" is to have 2 releases receiving
>> bug fixes, and 2 releases receiving security fixes at any given time;
>> and this happens every 3 months; total of 8 bug-fix and 8
>> security-only ports per year.
> 
> Not sure what the latter 8 stands for - we don't currently do any
> more releases from branches in security maintenance mode, all we
> do is provide respective backports in XSAs and apply those to the
> trees once the issues get published.

Maintenance release every 3 months -> 4 maintenance releases per year, *
2 outstanding bugfix relases = 8 total "bugfix" maintenance releases per
year.

>> It has so far been assumed in this discussion that this would be an
>> undue burden, and therefore not acceptable.
> 
> I don't think anyone said "undue". All that was said was that the
> amount of work to be put into this increases.

I think when you objected to the 6-month release cycle on the grounds
that it implies LTS releases, many people (Wei in particular)
interpreted that as saying that changing only that one parameter would
cause undue increase in load.  That is the assumption behind Wei's
commend earlier in this thread:

"With unlimited manpower we can use the current model to equally treat
each release LTS. In reality this is not the case, so I see the need for
using LTS model to effectively direct limited resources."

>>  But it's worth asking
>> whether that is actually the case.  It has been argued, for instance,
>> that the difficulty in backporting a patch scales with the distance in
>> commits that the patch has to be ported over.  Porting a patch to a
>> releases 3, 9, and 15 months ago isn't 1.5x as much work as porting it
>> to one 3 and 12 months ago.
>>
>> But I would guess that the porting *is* more work; and the building,
>> testing, and releasing certainly is 1.5x more work.
>>
>> We could also explore other options, like having more automation, or
>> spreading the work of porting patches among more people.
> 
> Now that goes slightly astray: People like me having to maintain
> older releases for much longer time need to do certain backports
> anyway (the older the release, the higher the chance that a
> difficult backport may result in a decision being made to ignore
> that fix as not important enough, due to the risk of the backport
> going wrong being higher than that of leaving the bug in place).
> I.e. at least a good part of the _backporting_ overhead can
> probably be discounted. What I consider more of an issue is the
> tedious (because purely mechanical) task of committing the
> patches to the respective stable trees, which (in my way of
> doing it) implies test builds for every commit. This is time I
> consider worthwhile spent as long as it doesn't grow too much,
> but of course this time could be used for less boring and more
> productive things.
> 
> Perhaps there's room for further automation here, yet as with
> any automation the question is how quickly getting this in place
> will amortize itself.

Aren't stable branches already tested with OSStest?  If they aren't they
certainly could be made to be tested pretty trivially, I think.
Wouldn't an osstest (which includes both build and functional tests)
plus an RC be sufficient?

>> Another thing which has been suggested is to treat releases
>> differently; for instance, every 4 releases (24 months), release an
>> "LTS" release that has longer support requirements.  If we chose to
>> have LTS releases, we may then also choose to have normal releases get
>> a shorter support window.  It's also been suggested that security
>> fixes are really what downstreams need from an LTS release; once
>> you've been using something for more than 2 years, you've probably
>> already found all the bugs which bother you.
> 
> But wouldn't that mean that the current model of 3 years of security
> support is (almost) what people want from an LTS branch?

Is there a specific proposal you are advocating here?

>> So one option would be "R6 S3 {12,12} L24 S3 {24, 48}" (Regular
>> releases every 6 months, maintenance releases every 3 months, fixes
>> backported for 12 months; LTS release every 24 months, bug-fixes
>> backported for 24 months, security fixes backported for 48 months).
>> Steady-state this gives us a max 3 releases receiving bug-fix
>> backports and 1 release receiving security backports at any given
>> time.  This gives us 12 bug-fix and 4 security-only ports per year.
>>
>> Any thoughts on this?  Anything else to add or discuss?
> 
> So using the 4 month stable release cadence I get currently 6
> stable releases a year (two maintained branches, ignoring short
> periods of overlap), compared to 6 ordinary stable releases (two
> ordinary branches) plus 6 LTS releases (two LTS branches) a year,
> i.e. doubling the amount.
> 
>> Personally I like the LTS model.  I think arguably if we use the
>> numbers I chose above, it shouldn't be any more actual work than it is
>> now (as there are still only 4 releases every 3-month maintenance
>> window), and I don't understand the concern with "treating releases
>> differently".
> 
> If what becomes an LTS release is known up front, the pressure to
> get things into such a release may (and, looking at what I got to
> notice on Linux, will) be quite a bit higher. After all the expectation
> and intention is for distros to particularly use those releases when
> caring for long term maintenance. And with that why would
> contributors bother much about getting things into non-LTS
> releases when those won't get used by many / for a reasonable
> time period anyway?
> 
> If what becomes an LTS release is to be determined only at the
> end of a branch's ordinary life cycle, the above problem can be
> avoided, but downstream consumers having picked the "wrong"
> release will be penalized. This is what we've been getting
> complaints on by openSUSE users relative to the kernel versions
> (and which keeps coming up the discussion of up-rev-ing the
> kernel version post release, which as you surely can imagine has
> all kinds of up and down sides). Plus depending on who gets to
> maintain the branch from that point on (if not handed to always
> the same person), quality and hence value to the downstreams
> may heavily vary.

OK, so the main argument in favor of 6-month release cycles is the
reduced time between check-in and a feature being in a release.  It has
been proposed that this shorter cycle will in part reduce the pressure
to get features in for any particular release.

Here is my summary of your argument above (let me know if it's accurate):

---
We have two options WRT LTS releases: Either we make LTS releases known
ahead of time, or we choose them afterwards.

There is no value to developers to getting features into non-LTS
releases; and therefore, if the LTS releases are known ahead of time,
then the LTS release will have exactly the same pressure as a normal
release to get features in.  (And of course, if we move from a 9-month
regular release to a 24-month LTS release, the pressure will be even
greater than it is now.)

[I'm not sure if you're arguing the following or not.] In fact, in the
time after an LTS release, many developers may completely stop
developing until the next LTS release comes around.

We could mitigate this developer pressure by not announcing LTS releases
ahead of time, so that looking forward, developers don't know which
release will be an LTS release.  But then we have other undesirable side
effects, namely, distros may end up settling on a version which turns
out not to be an LTS, and are then in a bind as to whether to change
versions post-release (disruptive and risky) or do all the maintenance
themselves (a lot of extra work) or just not have maintenance updates
(poor quality).
---

First of all, I completely agree that not announcing LTSes ahead of time
has unacceptable effects on distributions and other downstreams.  I
think LTSes, if we choose them, should happen on a regular, predictable
basis.

However, I don't agree that getting features into non-LTS releases is of
no value to developers.  Many users will use non-LTS releases (probably
Fedora, and a lot of our direct users, and probably anyone using
debian-testing most of the time).  Furthermore, downstreams that want
particular features are much more likely to cherry-pick them from stable
releases than from unreleased xen-unstable.

Of course there will be more desire to get a feature into an LTS than a
regular release, but I still think that this pressure will be less now,
and less frequent, than what we have now.

Ultimately we're all just making guesses here.  There is certainly a
risk that you will be proven correct (or partially correct).  I think
it's worth giving it a try.

On the other hand, if we can shorten to a 6-month release cycle while
*not* going to an LTS model (by making the maintenance releases more
automated and/or sharing the maintenance burden), I would be satisfied
with that as well.

I think at some point we may just have to throw out a couple of options
and do a 4-point survey as we've done before (i.e., "argue againts / not
happy but won't argue / happy but won't argue / argue for") and see
where we stand.

 -George



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel