Hi everyone.
Below is a technical blog that will accompany the news release on Wednesday.ÂIf anyone has edits, please let me know and I'll share the Google doc. Apologies that I'm unable to share via the blog software. I'm I've had trouble with the login, so I'm unable to share it via WordPress.
Thanks,
Title:Â
Less is More In The New Xen Project 4.5 Release
If we used code-names, the Xen 4.5 release should be called Panda on Diet!
We have 78K new code with 141K deletion. In effect this release has -63KLOC code than the previous one.
The net effect of a skinnier Xen Project Hypervisor code base is increased usability, simplicity and innovation. This is all by design and one of many steps weâll continue to take to fine-tune our development and release cycle.
For example, we shed the Python toolstack â including xend which we deprecated in 4.3 -- which comprised the majority of the code deleted in todayâs release. That is a big boom for the developers who now have less code to maintain and can spend more time on new features. And, 4.5 is more feature-rich than any release in Xen Projectâs history. For ease of reading, ÂIâve grouped the updates into four major categories:
Hypervisor specific
Toolstack
External users of toolstack
Linux, FreeBSD, and other OSes that can utilize the new features.
Virtualization and open source are more relevant than ever in today's evolving, more software-centric data center. New developments with hyper scale-out computing, Internet of Things, NFV/SDN, embedded computing and next-generation ARM-based products are driving increased demand for improved resource sharing and better utilization with enough flexibility to efficiently grow well into the future. What isn't likely to change anytime soon is the diversity of OSes, multi-tenant architectures, security concerns and storage and network challenges that cloud providers and enterprises must contend with to run their applications. Undeniably, abstraction at the VM level is necessary for superior performance and security in these environments.
Despite these impressive and rapid changes, or perhaps because of them, Xen Project developers are motivated to continually stay ahead of the market with performance, speed, agility and security. Our traditional customers also inspire us; organizations such as Alibaba, Amazon Web Services, IBM Softlayer, Rackspace, Oracle and others who are actually some of the most savvy and innovative users around. Today we are announcing specific patches in Xen Project Hypervisor 4.5 that span from architecture (x86 and ARM ), platforms (different ARM boards, AMD or Intel), to generic code. The release also creates new opportunity to incorporate Xen virtualization into software stacks in markets like embedded computing, automotive, drones, avionics and more.
x86 Hypervisor-Specific Updates
On the x86 side, development has focused on improving performance on various fronts:
- The HPET has been modified to provide faster and better resolution values.
- Memory is scrubbed in parallel on bootup giving a huge time boost for large-scale machines (1TB or more).
- PVH initial domain support for Intel has been added and now supports running as dom0 with Linux platforms. PVH is an extension to the classic Xen Project Paravirtualization (PV) that uses the hardware virtualization extensions available on modern x86 processor servers. Requiring no additional support other than the hypervisor, PVH boots as the first guest and takes on the responsibilities of the initial domain known as dom0. This means Xen Project Hypervisor is able to take advantage of contemporary hardware features like virtual machine extensions (VMX) to significantly expedite execution of the initial domain. Instead of asking the hypervisor to handle certain operations, the dom0 can execute operations natively without compromising security. For more background, Virtualization Spectrum is an excellent introduction to PVH.
- Lower interrupt latency for PCI passthrough on large-scale machines (more than 2 sockets).
- Multiple IO-REQ services for guests, which is a technique to have many QEMUs assigned for one domain. This allows speed up of guests operation by having multiple backends (QEMUs) deal with different emulations.
We also expanded support for:Â
- vNUMA (The FOSS Outreach Program for Women sponsored this work). Virtual NUMA allows Xen to expose to the guest the NUMA topology (either based on the host or made-up) for the guest.
- Soft affinity for VCPUs (aka NUMA affinity) â In Xen 4.4 we added Automatic NUMA placement. In Xen 4.5 we build on that to allow VCPUs that go outside their NUMA silos â if it cannot run inside the NUMA silos anymore (overcommit for example) â we still want it to run â as opposed to not running.
- Guest introspection expansion. There is an an excellent video of this on Youtube or the presentation (also part of the video) Security Improvements: VM introspection using Intel EPT / AMD RVI hardware virtualization functionality builds on Xen Project Hypervisors Memory Inspection APIs introduced in 2011. This addresses a number of security issues from outside the guest OS without relying on functionality that can be rendered unreliable by advanced malware. The approach works by auditing access of sensitive memory areas using HW support in guests in an unobtrusive way (or maybe better: with minimal overhead) and allows control software running within a dedicated VM to allow or deny attempts to access sensitive memory based on policy and security heuristics. Hereâs an excellent Youtube video and or presentation with more information.Â
- Serial support for debug purposes. This covers PCIe cards (Oxford ones) and newer Broadcom ones found on blades.
- Real Time Scheduler - improved multi-core support allows users to predict timing and performance of VMs. Video at Youtube and presentation at Linux Foundation and blog.
Intel Hypervisor-Specific Updates:
Broadwell Supervisor Mode Access Prevention. The LWN article has an excellent explanation of it â but a short summary is that it restricts the kernel from accessing the user-space pages. This feature in Xen also added alternative assembler support to patch the hypervisor during run-time (so that we wonât be running these operations on older hardware).
Haswell Bridge Cache QoS Monitoring aka Intel Resource Director Technology is âa new area of architecture extension that seeks to provide better information and control of applications running on Intel processors. The first few features we will cover, documented in the Software Developersâ Manual, are to monitor application thread LLC usage, to provide a means of directing such usage and provide more information on the amount of memory traffic out of the LLC.â (from xen-devel)
SandyBridge (vAPIC) extensions. ÂIn Xen 4.3 support for VT-d Posted Interrupts was added. In Xen 4.5 we added extensions for PVHVM guests to take advantage of VT-d Posted Interrupts. Instead of using vector callback the guest can utilize the vAPIC to lower its VMEXIT overhead, leading to lower interrupt latency and performance improvements for I/O intensive workloads in PVHMM guests.
AMD hypervisor-specific updates:
Fixes in the microcode loading.
Data Breakpoint Extensions and masking MSR Support for Kabini, Kaveri and further. This allows â.. to specify cpuid masks to help with cpuid levelling across a pool of hosts. â from xen-command-line manual.
ARM Hypervisor-Specific Updates
The ARM ecosystem operates differently to the x86 architecture â in which ARM licensees design new chipsets and features and OEMs manufacture platforms based on these specifications.
OEMs designing ARM based platforms determine what they need on the SoC â that is the System On Chip. As such they can selectively enable or disable certain functionality that they consider important (or unimportant). ARM provides the Intellectual Property (IP) and standards from which OEMs can further specialize and optimize. Therefore the list of features that Xen Project Hypervisor supports on ARM is not for a specific platform â but rather for functionality SoCs provide. New updates include:
Support for more than 1TB guests.
The Generic Interrupt Controller (GIC) v3 is supported in Xen 4.5. v3 supports Message Signaled Interrupts (MSI), emulation of GICv3 for guests â and most importantly â support for more 8 CPUS.
Power State Coordination Interface (PSCI) â important in the embedded environment where power consumption needs to be kept to the absolute minimum. It allows us to power down/up CPUS, suspend them, etc.
UEFI booting. On ARM64 servers both U-Boot and UEFI can be used to boot the OS.
IOMMU support (SMMUv1). For isolation between guests, ARM hardware can come with an IOMMU chipset based on the SMMU specification.
Super Pages (2MB) support in Xen. Using super pages for the guest pseudo-physical to physical translation tables significantly improves overall guest performance.
Passthrough â the PCI passthrough features did not make on time, but doing passthrough of MMIO regions did. In the ARM world it is quite common to have no PCIe devices and to only access devices using MMIO regions. As such this feature allows us to have driver domains be in charge of network or storage devices.
Interrupt latency reduction: no maintenance interrupts. Please see Stefanoâs slides.
With these new features, the following motherboards are now supported in Xen Project Hypervisor 4.5:
AMD Seattle
Broadcom 7445D0 A15
Midway (Calxeda)
Odroid-XU
Vexpress (ARM Ltd.)
OMAP5, OMAP6, DRA7 (Texas Instrument)
Exynos5 (Samsung chip on the Arndale and various smartphones and tablets)
SunXI (AllWinner), aka A20/A21, CubieTruck, CubieBoard
Mustang (Applied Micro-X-Gene, the armv8 SoC)
McDivitt aka HP Moonshot cartridge (Applied Micro X-Gene)
The Xen Project also maintains this list of ARM boards that work with Xen Project software.
Toolstack Updates
Xen Project software is now using a C-based toolstack called xl or libxl, replacing the obsolete Python toolstack called xend. ÂBased on a this more modern architecture for easier maintenance, users will not be affected by this move since xm and xl offer feature parity. In fact, the move greatly simplifies managing Xen instances as other toolstack, such as libvirt are C based and less complex. libvirt and XAPI are now using libxl as well. For more background on this move, check out our new hands-on tutorial âXM to XL: A Short, but Necessary, Journeyâ (will embed this link in the blog).
https://www.youtube.com/watch?v=qdJi18VekEY
Additional toolstack changes include:
VM Generation ID. This is a requirement for Windows 2012 Server and later domains to properly migrate.
Remus initial support â which provides high availability by check pointing guests states at high frequency.
libxl support for JSON to keep track of guest configs (in turn changing xl to be stateful), discard support to disable/enable the guest using it.
systemd support. This allows one source base to contain the systemd files, which can be used by various distributions instead of them having to generate them.
On the libvirt side, changes include:
PCI/SR-IOV passthrough, including hot{un}plug
Migration support
Improved concurrency through job support in the libxl driver â no more locking entire driver when modifying a domain
Improved domxml-{to,from}-native support, e.g. for converting between xl config and libvirt domXML and vise-versa
PV console support
Improved qdisk support
Support for <interface type=ânetworkâ> â allows using libvirt-managed networks in the libxl driver
Support PARAVIRT and ACPI shutdown flags
Support PARAVIRT reboot flag
Support for domain lifecycle event configuration, e.g. on_crash, on_reboot, etc
A few improvements for ARM
Lots of bug fixes
QEMU Updates
Xen Project 4.5 will ship with QEMU v2.0 and SeaBIOS v1.7.5 with the following updates:
Bigger PCI hole in QEMU via the mmio_hole parameter in guest config. This allows users to pack more legacy PCI devices for passthrough in an guest.
QEMU is now built for ARM providing backend support for framebuffer (VNC).
OSes
The 4.5 release also takes advantage of new features in Linux and FreeBSD such as PVH support.
Summary
With 43 major feature updates, 4.5 includes the most updates in our projectâs history. Thatâs not even counting 22 new enablers in up-streams (Linux and QEMU). The Project is also taking a more holistic and proactive approach to managing dependencies such as Linux and QEMU, as well as downstream functionality such as libvirt. In 2015, we plan to build on this even further up the stack to include OpenStack and other key projects. For the first time, our projectâs development process is robust, active and mature enough to systematically focus on these strategic growth opportunities. It also reflects enhanced responsiveness to community feedback; for example, weâre improving usability and performing broader testing for specific use cases with new releases.
During this development and release we have also seen a steady influx of folks helping, contributing, testing and reporting. As the Release Manager I would like to thank everybody and call out major contributions coming from AMD, Bitdefender, Citrix, Fujitsu, GlobalLogic, Intel, Oracle and Cavium, as well as several individual and academic institutions.
The sources are located in the git tree or one can download the tarball:
xen: with a recent enough git (>= 1.7.8.2) just pull from the proper tag (RELEASE-4.5.0) from the main repo directly:
git clone -b RELEASE-4.5.0 git://xenbits.xen.org/xen.git
With an older git version (and/or if that does not work, e.g., complaining with a message like this: Remote branch RELEASE-4.5.0 not found in upstream origin, using HEAD instead), do the following:
git clone git://xenbits.xen.org/xen.git ; cd xen ; git checkout RELEASE-4.5.0
tarball: here it is a 4.5.0 and its signature.
