[Xen-devel] Radical proposal v2: Publish Amazon's verison now, Citrix's version soon

[George Dunlap <george.dunlap@xxxxxxxxxx>]

* Executive summary

- We've agreed on a "convergence" point for PV shim functionality that
  covers as many users as possible:
 - 'HVM' functionality: boots in HVM mode, has support for Xen 3.4
   event channels, &c, booted via 'sidecar'
 - 'PVH' functionality: boots in PVH mode, booted via toolstack
   changes

- "Vixen" (the Amazon shim) and PVH shim (mostly developed by Citrix)
  each cover some users and not others; neither one (yet) covers all
  users

- Vixen is ready for immediate release; PVH shim still needs some
  minor revision, and the toolstack component

- Proposal:
 - Release "HOWTO" for Amazon shim v1 immediately (including a signed
   tag on xenbits with the requisite patches)
 - Release "HOWTO" for PVH shim as soon as it's ready (including a
   signed tag or tags on xenbits with the requisite patches)
 - Base future development on the PVH shim (porting over functionality
   from the Amazon series)

* Discussion

In our discussions, we've generally identified four kinds of users /
consumers of Xen:

1. Those for whom upgrading to a newer version of Xen is the biggest
problem

This could be people who have local changes they can't forward-port,
people whose SLAs forbid updating versions, or any number of other
things that make updating to a newer version of Xen unpalatable or
impossible.

2. Those for whom adding a QEMU instance to all PV guests is the
biggest problem

Many users I've talked to extremely dislike the idea of adding QEMU
instances to all their PV guests, and would rather do upgrades and/or
modify guest types to avoid this

3. Those for whom modifying guest parameters is the biggest problem

Many users I've talked to are using software which severely limits the
flexibility they have in terms of how guests are created

4. Those for whom some subset of features (migration, ballooning, vcpu
hotplug, &c) is the biggest problem.

This includes users who rely on these features, as well as software
providers that provide those features.

Amazon is in the first category, and have developed a "PV shim"
series that addresses their needs.  They have also very kindly
shared the patches publicly for the community to be able to use.

Citrix is in the fourth category, and have developed a "PV shim"
series that addresses their needs.  This shim also satisfies category 2,
and with some work could support category 3.

Going forward, the best solutions for new hypervisors (Xen 4.11+)
would be to avoid needing QEMU, building the sidecar, and so on.  But
we still want to be able to support people running the 'HVM shim with
sidecar' version going forward.  So the "convergence point" we've all
seemed to agree on is to have a shim capable of satisfying all groups:

- Can run in HVM mode back as far as Xen 3.4
- Can run in PVH mode, without QEMU or "sidecar", and doesn't require
  many toolstack changes
- Has feature parity with PV mode (migration, ballooning, vcpu
  migration, &c)

The issue has been made public for nearly a week now, so there is a
lot of pressure to get a solution to our users as soon as possible; by
the end of the week at absolute latest, but today if possible.

Amazon's v1 series:
- Has support for 'legacy' event channels

- Has been tested by Amazon over a wide range of their hypervisor
  versions and guest types (including pvgrub types) back as far as Xen
  3.4

- Uses HVM mode, and so has the overhead and security implications of
  running QEMU

- Requires no L0 hypervisor or toolstack changes

- Requires users to build a "boot sidecar" image for each unique guest
  boot configuration

- Is missing many features, such as migration, memory ballooning, and
  vcpu hotplug

- Has accurate 'stolen time' support

Citrix's series:
- Has been extensively tested by XenServer's XenRT for Xen 4.7 (with
  PVH backports) and 4.10
- Uses PVH mode, and so doesn't have the overhead and security
  implications of running QEMU
- No need to build "boot sidecar"
- Requires hypervisor changes for versions for versions before 4.10
  which cannot reasonably be backported by the open-source team beyond
  Xen 4.8
- Requires toolstack changes in all versions
- Has migration, memory ballooning, and vcpu hotplug (extensively
  tested by XenRT)
- Doesn't have accurate 'stolen time' support
- Currently is missing a clean toolstack interface

With some tweaks, Citrix's version has been made to boot in HVM mode.
However, this modified version:
- Doesn't support the 'legacy' event channel mode, and so won't work
  for versions as old as Amazon's
- Hasn't been tested extensively on older versions

Ian suggested that we get something and check it into 4.10-staging
as soon as possible.  But there has been a debate about whether we
should start with Amazon's series and add PVH support, or start with
Citrix's series and add HVM support (including legacy event channels,
and so on).

No matter what, individual users will need to decide whether to take
the 'HVM + sidecar' option or the 'PVH' option..

I'd like to propose a new appraoch:

1. Immediately release Amazon's v1 series for people who can / prefer
to use the HVM + sidecar option.
- Advisory and HOWTO should include who should use this option, and
  how to do it.
- Check the series into a branch on xenbits, and add a signed tag
- HOWTO-vixen, and sidecar script, to be included in the advisory.

2. As soon as possible, release Citrix's series for people who can /
prefer to use the PVH + toolstack option.
- Immediate work should focus on getting PVH + toolstack functionality
  ready to release
- When ready, advisory and HOWTO should include who should use this
  option and how
- Either check the series into a branch on xenbits, or into
  staging-4.10
- HOWTO-pvh to be included in the advisory.

This should allow us to get a solution to the widest number of users
in the shortest amount of time; it also allows us to leverage the
testing efforts of both Amazon (for breadth of Xen versions) and
Citrix (for depth of functionality).

That takes the pressure off us to check in one or the other version of
the patch series.

Going forward, we could work towards the convergence of functionality
from either patch series.  But it looks superficially at least like
the Citrix series is closer to the convergence point, and so it seems
like using that as a starting point would make the most sense.

Regardless of what we think of step 2, I think we should take step 1
immediately.

Let me know what you think.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel