ïThis is a guest blog post by Tamas K Lengyel, a long-time open source enthusiast and Xen contributor. Tamas works as Senior Security Researcher at Novetta, while finishing his PhD on the topic of malware analysis and virtualization security at the University of Connecticut.
-------------
Hardening Xen against VENOM-style attacks
The recent disclosure of the VENOM[1] bug affecting major open-source hypervisors, such as KVM and Xen, has been circulating in the tech news as of 5/13/15, causing many to reevaluate their risks when using cloud infrastructures. That is a very good thing indeed. Virtualization and the cloud have been erroneously considered by many to be a silver bullet against intrusions and malware in general. The fact is that the cloud is anything but a safe place. There are inherent risks in the cloud, thus it is very important to put those risks in context.
Hypervisors, same as any other complex software systems, are prone to bugs and errors. VENOM in that sense is just another one in the long list of vulnerabilities that have been popping up against hypervisors[2]. While VENOM is indeed a serious bug and can result in a VM escape, which in turn can compromise all VMs on the host, it doesnât have to be. In fact, VENOM-style attacks have been known for a long time. And there are easy-to-deploy counter-measures to mitigate the risk of such exploits, natively available in both Xen and KVM (see RedHatâs blog post on the same topic[3]).
What is the root cause of VENOM? Emulation.
While modern systems come with a plethora of virtualization extensions, many components are still being emulated. The biggest component of that are usually devices. Devices such as your network card, graphics card and your hard drive. While Linux comes with paravirtual (virtualization-aware) drivers to create such devices, emulation is often the only solution to run operating systems that do not have such kernel drivers. This has been traditionally the case with Windows[4]. This emulation layer has been implemented in QEMU, which has caused VENOM and a handful of other VM-escape bugs in recent years. However, QEMU is not the only place where emulation is used within Xen. For a variety of interrupts and timers, such as RTC, PIT, HPET, PMTimer, PIC, IOAPIC and LAPIC, there is a baked-in emulator in Xen itself for performance and scalability reasons[5]. As with QEMU, this emulator has been just as exploitable[6]. This is simply because to program emulating devices and services properly is really complex and error-prone.
This fact, that emulation is complex and error-prone, has been known for a long time. Back in 2011, the Blackhat talk on Virtunoid[7] demonstrated such a VM escape attack against KVM, through QEMU. The author of Virtunoid even cautioned there will be many other bugs of that nature found by researchers. Fast-forward 4 years and we now have VENOM.
The good news is there is an easy mitigation available for all present and future QEMU bugs, also mentioned in the original Xen Security Advisory[8]. It is called stubdomains[9]. Stubdomains can nip the whole class of vulnerabilities exposed by QEMU in the bud by moving QEMU into a deprivileged domain of its own. Instead of having QEMU run as root in dom0, a stubdomain has access only to the VM it is providing emulation for. Thus, an escape through QEMU will only land an attacker in a stubdomain, without access to critical resources. Furthermore, QEMU in a stubdomain runs on MiniOS, thus an attacker would only have a very limited environment to run code in (as in return-to-libc/ROP-style), having exactly the same level of privilege as in the domain where the attack started. Nothing is to be gained for a lot of work, effectively making the system as secure as it would be if only PV drivers were used.
One would think it is a complex process to take advantage of this protection, but in fact, it is very simple. Add the following line to your Xen domain configuration and you gain immediate protection against VENOM, and the whole class of vulnerabilities brought to you by QEMU:
device_model_stubdomain_override = 1
However, as with most security systems, it comes at a cost. Running stubdomains requires a bit of extra memory as compared to the plain QEMU process in dom0. This in turn can limit the number of VMs a cloud provider may be able to sell, thus there is limited incentive on their end to enable this feature by default. Further complicating the picture is the fact that this protection works best if all VMs on the server run with stubdomains. Otherwise you may end up protecting your neighbors against an escape attack from your domain, while not being protected from an escape attack from theirs. It is thus no surprise if some users would not want to pay for this extra protection, even if it was available. However, for private clouds and exclusive servers there is no reason why it shouldnât be your default setting.
________________
[1]
http://venom.crowdstrike.com/
[2]
http://xenbits.xen.org/xsa/
[3]
https://securityblog.redhat.com/2015/05/13/venom-dont-get-bitten/.
[4] Citrix recently open-sourced its Windows PV drivers so it is no longer the case:
http://www.xenproject.org/downloads/windows-pv-drivers.html
[5]
http://www.gossamer-threads.com/lists/xen/devel/347492#347492
[6]
http://xenbits.xen.org/xsa/advisory-105.html
[7]
http://media.blackhat.com/bh-us-11/Elhage/BH_US_11_Elhage_Virtunoid_WP.pdf
[8]
http://xenbits.xen.org/xsa/advisory-133.html
[9]
http://wiki.xen.org/wiki/Device_Model_Stub_Domains--
Tamas K Lengyel
Senior Security Researcher
7921 Jones Branch Drive
McLean VA 22102
