Re: [Xen-devel] Xen 4.3 development update, and stock-taking

[Andres Lagar-Cavilla <andres.lagarcavilla@xxxxxxxxx>]

> Below is a summary of the projects / features being worked on for the 4.3
> time frame.  The tentative feature freeze is scheduled for 25 March, which
> is just over 2 months away.  With that in mind, I think it's time to take
> stock of the development, so we know whether to ask for more help or divert
> resources.
> 
> To that end, I've added a line called "prognosis", indicating the
> likelihood of completion in the 4.3 timeframe.  For items involving code
> hosted on the Xen.org site (including qemu-xen), that means a likelihood of
> having the feature code-complete and mostly working by the feature freeze.
> (It's OK if there are still bugs to be worked out.)  For items in Linux, I
> think it would mean having items on track to make it into the kernel
> released just after the scheduled 4.3 time frame.  Not sure what that means
> for libvirt. :-)
> 
> I've given ratings to projects that I thought I had some insight on, but I
> would appreciate if everyone could review these and let me know if they're
> not accurate.
> 
> I'd particularly appreciate it if the people listed below could give
> prognoses on the project listed.
> 
> Meanings of prognoses:
> - Excellent: It would be very unlikely for this not to be finished in time.
> - Good: Everything is on track, and is likely to make it.
> - Fair: A pretty good chance of making it, but not as certain
> - Poor: Likely not to make it unless intervention is made
> - Not for 4.3: self-explanatory
> 
> Mukesh Rathor: PVH
> 
> Wei: Event channel scalability
> 
> Daniel de Graaf: IS_PRIV->XSM
> 
> Roger Pau Monne: Persistent blk grants, openvswitch integration, backend
> scripts
> 
> Konrad Wilk: Persistent net grants, multi-page network, multi-page blk
> 
> Jim Fehlig: libvirt migration support
> 
> Matthew Fioravante: vTPM
> 
> Stefano Panella: PV Audio
> 
> Igor Kozhukov: IllumOS support
> 
> Once we know what items are "at risk", we can list them and decide what to
> do about it.
> 
> This information will be mirrored on the Xen 4.3 Roadmap wiki page:
> http://wiki.xen.org/wiki/Xen_Roadmap/4.3
> 
> = Timeline =
> 
> We are planning on a 9-month release cycle.  Based on that, below are
> our estimated dates:
> * Feature Freeze: 25 March 2013
> * First RC: 6 May 2013
> * Release: 17 June 2013
> 
> The RCs and release will of course depend on stability and bugs, and
> will therefore be fairly unpredictable.  The feature freeze may be
> slipped for especially important features which are near completion.
> 
> = Feature tracking =
> 
> Below is a list of features we're tracking for this release. Please
> respond to this mail with any updates to the status.
> 
> There are a number of items whose owners are marked as "?".  If you
> are working on this, or know who is working on it, please respond and
> let me know.  Alternately, if you would *like* to work on it, please
> let me know as well.
> 
> And if there is something you're working on you'd like tracked, please
> respond, and I will add it to the list.
> 
> NB: Several of the items on this list are from external projects:
> linux, qemu, and libvirt.  These are not part of the Xen tree, but are
> directly related to our users' experience (e.g., work in Linux or
> qemu) or to integration with other important projects (e.g., libvirt
> bindings).  Since all of these are part of the Xen community work, and
> comes from the same pool of labor, it makes sense to track the
> progress here, even though they won't explicitly be released as part
> of 4.3.
> 
> Meanings of prognoses:
> - Excellent: It would be very unlikely for this not to be finished in time.
> - Good: Everything is on track, and is likely to make it.
> - Fair: A pretty good chance of making it, but not as certain
> - Poor: Likely not to make it unless intervention is made
> 
> == Completed ==
> 
> * Serial console improvements
>  -EHCI debug port
> 
> * Default to QEMU upstream (partial)
> - pci pass-thru (external)
> - enable dirtybit tracking during migration (external)
> - xl cd-{insert,eject} (external)
> 
> * CPUID-based idle (don't rely on ACPI info f/ dom0)
> 
> == Bugs ==
> 
> * xl, compat mode, and older kernels
>  owner: ?
>  Many older 32-bit PV kernels that can run on a 64-bit hypervisor with
>  xend do not work when started with xl.  The following work-around seems to
>  work:
>    xl create -p lightning.cfg
>    xenstore-write /local/domain/$(xl domid
> lightning)/device/vbd/51713/protocol x86_32-abi
>    xl unpause lightning
>  This node is normally written by the guest kernel, but for older kernels
>  seems not to be.  xend must have a work-around; port this work-around to
> xl.
> 
> == Not yet complete ==
> 
> * PVH mode (w/ Linux)
>  owner: mukesh@oracle
>  status (Linux): 3rd draft patches posted.
>  status (Xen): RFC submitted
>  prognosis: Good
> 
> * Event channel scalability
>  owner: wei@citrix
>  status: RFC submitted
>  prognosis: Good
>  Increase limit on event channels (currently 1024 for 32-bit guests,
>  4096 for 64-bit guests)
> 
> * ARM v7 server port
>  owner: ijc@citrix
>  prognosis: Excellent
>  status: Core hypervisor and Linux patches accepted.  Tools patches
> submitted.
> 
> * ARM v8 server port (tech preview)
>  owner: ijc@citrix
>  status: ?
>  prognosis: Tech preview only
> 
> * NUMA scheduler affinity
>  critical
>  owner: dario@citrix
>  status: Patches posted
>  prognosis: Excellent
> 
> * NUMA Memory migration
>  owner: dario@citrix
>  status: in progress
>  prognosis: Fair
> 
> * blktap3
>  owner: thanos@citrix
>  status: RFCs posted
>  prognosis: Not for 4.3
> 
> * Default to QEMU upstream
>> Add "intel-hda" to xmexample file, since it works with 64-bit Win7/8
> - qemu-based stubdom (Linux or BSD libc)
>   owner: anthony@citrix
>   status: in progress
>   prognosis: ?
>   qemu-upstream needs a more fully-featured libc than exists in
>   mini-os.  Either work on a minimalist linux-based stubdom with
>   glibc, or port one of the BSD libcs to minios.
> 
> * Persistent grants for blk (external)
>  owner: roger.pau@citrix
>  status: Initial implementation posted
>  prognosis: ?
> 
> * Persistent grants for net
>  owner: annie.li@citrix
>  status: Initial implementation posted
>  prognosis: ?
> 
> * Multi-page blk rings (external)
> - blkback in kernel (konrad@oracle, ?@intel)
> - qemu blkback
>  status: Not started.
>  prognosis: UNKNOWN
> 
> * Multi-page net protocol (external)
>  owner: ijc@citrix or annie.li@oracle
>  status: Initial patches posted (by Wei Liu)
>  expand the network ring protocol to allow multiple pages for
>  increased throughput
> 
> * Scalability: 16TiB of RAM
>  owner: jan@suse
>  status: Not started
> 
> * vTPM updates
>  owner: Matthew Fioravante @ Johns Hopkins
>  prognosis: Good
>  status: some patches submitted, more in progress
>  - Allow all vTPM components to run in stub domains for increased security
>  - Update vtpm to 0.7.1 from 0.5.x
> 
> * Guest EFI booting
> - status: tianocore in-tree, some build problems.
>   prognosis: Poor.
>   Needs new owner.
> 
> * libvirt/libxl integration (external)
> - Update libvirt to 4.2
>   status: Patch accepted
> - Migration
>   owner: cyliu@suse (?)
>   status: first draft implemented, not yet submitted
>   prognosis: ?
> - Itemize other things that need work
>   To begin with, we need someone to go and make some lists:
>   - Features available in libvirt/KVM not available in libvirt/libxl
>     See http://libvirt.org/hvsupport.html
>   - Features available in xl/Xen but not available in libvirt/Xen
> 
> * Allow XSM to override IS_PRIV checks in the hypervisor
>  owner: Daniel De Graaf
>  status: patches against 4.3-unstable posted, awaiting approval
>  prognosis: Good
>  This makes it possible to allow some user domains limited access to
>  dom0-only hypercalls. This could be used to allow a user-created
>  toolstack domain to administer its own set of VMs instead of relying
>  on dom0's toolstack.
> 
> * V4V: Inter-domain communication
>  owner (Xen): jean.guyader@xxxxxxxxxx
>  status (Xen): patches submitted
>  prognosis: ?
>  owner (Linux driver):  stefano.panella@citrix
>  status (Linux driver): in progress
> 
> * Wait queues for mm
>  owner: ?
>  status: Draft posted Feb 2012; more work to do.
>  prognosis: Poor

Mostly due to lack of time availability. All I've been able to do is float a 
(bad) idea in the list.

There is some clean up work to p2m and sharing that I have in my backlog as a 
precondition for that. I.e. unshares that don't unnecessarily allocate pages in 
paths such as remove_page, better sharing stats, use remove_page in 
XENMEM_remove_from_physmap, etc. Once I get cycles I'll aim to get that into 
4.3. 
> 
> * xl PVUSB pass-through for both PV and HVM guests
>  owner: George
>  status: ?
>  prognosis: Fair
>  xm/xend supports PVUSB pass-through to guests with PVUSB drivers (both PV
> and HVM guests).
>  - port the xm/xend functionality to xl.
>  - this PVUSB feature does not require support or emulation from Qemu.
>  - upstream the Linux frontend/backend drivers. Current work-in-progress
> versions are in Konrad's git tree.
>  - James Harper's GPLPV drivers for Windows include PVUSB frontend drivers.
> 
> * xl USB pass-through for HVM guests using Qemu USB emulation
>  owner: George
>  status: Config file pass-through submitted.
>  prognosis: Good
>  xm/xend with qemu-traditional supports USB passthrough to HVM guests
> using the Qemu emulated USB controller.
>  The HVM guest does not need any special drivers for this feature.
>  So basicly the qemu cmdline needs to have:
>     -usb -usbdevice host:xxxx:yyyy
>  - port the xm/xend functionality to xl.
>  - make sure USB passthrough with xl works with both qemu-traditional and
> qemu-upstream.
> 
> * xl QXL Spice support
>  owner: Zhou Peng
>  prognosis: Fair
>  status: Patches against 4.3-unstable posted, awaiting approval
> 
> * Remove hardcoded mobprobe's in xencommons
>  owner: ?
>  status: ?
>  prognosis: Poor.
> 
> * openvswitch toostack integration
>  owner: roger@citrix
>  prognosis: ?
>  status: Sample script posted by Bastian ("[RFC] openvswitch support
> script")
> 
> * Rationalized backend scripts (incl. driver domains)
>  owner: roger@citrix
>  status: patches submitted
>  prognosis: Good
> 
> * Xen EFI boot
> - Signature checking for dom0 kernel / initrd?
> status: No owner.
> prognosis: Probably not for 4.4
> 
> * Serial console improvements
>  owner: ?
>  status: Stalled (see below)
>  prognosis: Probably not for 4.4.
>  -xHCI debug port (Needs hardware)
>  -Firewire (needs hardware)
> 
> * Make storage migration possible
>  owner: ?
>  status: none
>  prognosis: Probably delay until 4.4
>  There needs to be a way, either via command-line or via some hooks,
>  that someone can build a "storage migration" feature on top of libxl
>  or xl.
> 
> * Full-VM snapshotting
>  owner: ?
>  status: none
>  prognosis: Probably delay until 4.4
>  Have a way of coordinating the taking and restoring of VM memory and
>  disk snapshots.  This would involve some investigation into the best
>  way to accomplish this.
> 
> * VM Cloning
>  owner: ?
>  status: none
>  prognosis: Probably need 4.4
>  Again, a way of coordinating the memory and disk aspects.  Research
>  into the best way to do this would probably go along with the
>  snapshotting feature.
> 
> * xl vm-{export,import}
>  owner: ?
>  status: none
>  prognosis: Prob put off until 4.4 (or GSoC project)
>  Allow xl to import and export VMs to other formats; particularly
>  ovf, perhaps the XenServer format, or more.
> 
> * Memory: Replace PoD with paging mechanism
>  owner: george@citrix
>  status: none
>  prognosis: Prob put off until 4.4

You're the boss here :) I think we could address most concerns with a few 
extensions: 1. Create a p2m_paged_out_zero state that allocates on 
p2m_paging_populate with no upcall. 2. Wire populate_physmap with pod flag to 
set this new state. 3. Remove lots of code :)

However, I do not fully understand the accounting that goes on in p2m-pod.c 
vis-a-vis the cache, and we'd need to find a way to keep super pages in a cache 
the way PoD currently does.

My 0.02
Andres

> 
> * PV audio (audio for stubdom qemu)
>  owner: stefano.panella@citrix
>  status: ?
>  prognosis: ?
> 
> * IllumOS support
>  owner: Igor Kozhukov
>  status: In progress
>  prognosis: ?
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: 
> <http://lists.xen.org/archives/html/xen-devel/attachments/20130116/89505b0f/attachment.html>
> 
> ------------------------------
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel
> 
> 
> End of Xen-devel Digest, Vol 95, Issue 243
> ******************************************


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel