Re: [Xen-devel] qemu-upstream stubdom - Xen can put VENOM-type attacks to bed

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>]

Hello Eric,

Thanks for the well written email.

I agree with you that securing QEMU is very important. As a matter of
fact four of the top work items of my team, the Xen Project team in
Citrix, are all related to it:


1) rump kernel stubdoms
As you know Wei Liu is actively working on this. As discussed in the
past, we decided to go for rump kernels instead of Linux because they
provide a smaller overhead.

2) PVH
PVH guests are another way to solve the same issue, at least for Linux
and BSD guests. They are also needed to improve Dom0 performance in the
long run. Unfortunately they also need more work.

3) Raisin
The new project, Raisin, was born to clean up the xen-unstable build
system, and to provide a place where we could further develop stubdomain
(rump kernel, linux or else). You have probably noticed that the current
build system for stubdoms is far from ideal, and if we introduced linux
based stubdom, we would certainly need a more powerful and cleaner build
infrastructure.

4) QEMU depriviledge
XenServer carries a patch in their QEMU patch queue to deprivilege QEMU.
It is an ugly patch, that requires a kernel change, but it also very
effective. We are investigating ways to upstream a similar, but cleaner,
solution. I am hoping to have something ready in a matter of few months.


We are doing all we can to improve the security of Xen deployments. I am
hoping that others will follow your call to arms and join the effort.



On Thu, 14 May 2015, Eric Shelton wrote:
> Yesterday's public announcement of VENOM highlights a security concern
> I raised in January with not getting qemu-upstream moved over to stub
> domains.  I have repeated my email below for your convenience.  To
> summarize my earlier email: I believe running QEMU device emulation in
> dom0 is Xen's single most significant outstanding security issue, and
> I have been long puzzled why it has not been addressed.
> 
> This week, the fuss is about a bug in QEMU's emulated FDC.  However,
> it is guaranteed that in a codebase with the scope, complexity, and
> pace of development that QEMU has, that there exist, and there will
> continue to be created, other hardware emulation bugs that enable the
> very same kinds of attacks demonstrated by VENOM.  VENOM is merely a
> single demonstration of a broader class of attacks involving breaking
> the QEMU emulation layer - a security issue that has been recognized
> for years, but nevertheless left on the shelf.  It does not make sense
> to allow QEMU to be part of the trusted base.
> 
> I again am campaigning for the Xen team to invest the resources to
> finally implement qemu-upstream stub domains.  My email from January
> covers plenty of good reasons for doing this.  Also, the Xen team can
> view it as a product differentiator over KVM and other solutions
> relying on QEMU, or maybe even other VM products that rely on
> different device emulation codebases.  Xen stub domains are a powerful
> solution to the problem that, as far as I understand, is not even on
> the table for KVM.
> 
> I have tried to "put my money where my mouth is" on this issue.  On
> February 3rd, I released a patchset updating Anthony's original
> efforts at implementing a Linux-based qemu-upstream.  With what I
> released, QEMU device emulation runs successfully under Linux in its
> own stub domain, including networking.  I could run a Linux
> distribution headless, and SSH into it.  The next major step that
> needs to be addressed is getting the display working.  After that are
> all the niceities such as suspend and resume that are currently
> enjoyed with qemu-upstream.
> 
> However, I cannot bring this task to completion by myself, as much as
> I would like to.  Although in an earlier life I was a full-time
> professional developer, I am now merely a part-time individual
> contributor to Xen.  Also, I no longer have the bandwidth I had even a
> few months ago, and unfortunately that is unlikely to change for a
> while.  This is partly because to take this further requires an
> understanding of QEMU internals and interacting with the QEMU team
> that I do not have, and do not have the time to acquire.  However, I
> believe there are already developers in the Xen community with such
> skills, or at a minimum there are professional Xen developers who can
> sensibly devote their efforts to this issue.
> 
> Thank you,
> Eric Shelton
> 
> On Thu, Jan 8, 2015 at 11:39 AM, Eric Shelton <eshelton@xxxxxxxxx> wrote:
> >
> > With the impending rollout of another Xen release lacking qemu-xen
> > stubdom, I would like to campaign for a couple of things: (1) making
> > qemu-xen stubdom a blocker for Xen 4.6; and (2) using a Linux-based
> > stubdom, at least for the time being.
> >
> > (1) Making qemu-xen a blocker for 4.6
> >
> > This security issues presented by allowing qemu to run unrestricted
> > within dom0 have been appreciated for a long time (comments about
> > qemu-dm in XSA-109 illustrate this).  Just last month, a few
> > additional qemu escalation vulnerabilities were demonstrated.  Given
> > the size, complexity, and pace of development of qemu, we can
> > reasonably assume there will always be some escalation vulnerability.
> >
> > Xen has generally taken known, and often far more obscure, security
> > issues very seriously.  I am puzzled that Xen continues to offer a
> > non-stubdom qemu.  The current qemu stubdom solution almost amounts to
> > a mitigation, rather than a solution, of a well-known issue.  The
> > options offered today are: (a) have access to all of the features and
> > bugfixes offered by upstream qemu (which have been significant thanks
> > to KVM's use of qemu, and are appreciable on and sometimes necessary
> > for some guest operating systems), at the cost of qemu running
> > unrestricted within dom0; or (b) have qemu contained in a separate
> > domain, but with what is essentially only a "good enough" six year old
> > version of qemu (ver 0.10.2).
> >
> > Additionally, once qemu-xen stubdom is realized, we can move away from
> > and stop maintaining the forked qemu-traditional codebase.  Finally,
> > all of the Xen-specific code could be mainstreamed into upstream qemu.
> >
> > If all that can managed is to release qemu-xen stub domains at the
> > level of a tech preview, it still represents a significant improvement
> > on both of the axes of functionality and security.
> >
> > (2) Using Linux to implement qemu-xen stubdom
> >
> > Efforts over the last 3 years at realizing qemu-xen stub domains seem
> > to illustrate the "perfect is the enemy of the good" phenomenon.  I
> > gather that rump kernel is the favored direction, and ultimately would
> > be more memory efficient than using Linux, but a lot of unresolved
> > technical issues lie down that path that are already solved by using
> > Linux, and resolving them has, and likely will continue to, stymie
> > getting something effective and reasonable out the door.
> >
> > It is understandable that the Xen team has prioritized other items
> > over qemu-xen stubdom.  However, rather than allowing a lower priority
> > item to be left undone, that should inform us that an "ideal," but
> > more development intensive, path is not called for.  Instead, we
> > should just adopt the more easily implemented solution and move on.
> >
> > Some reasons for picking up where Anthony left off with using Linux:
> > (a) Linux + qemu is a mature and well-tested codebase.  Although there
> > are some other environments under which qemu is built and run, far and
> > away the most common platform is Linux.
> > (b) Linux kernel + Xen is a mature and well-tested codebase.  Although
> > there was some disappointment that external patches had to be applied
> > to the Linux kernel, they were pretty minor.
> > (c) A lot things need to be done for rump kernel that already just
> > work with Linux.
> > (d) Developers are more familiar with Linux.  At one point along the
> > way, there was expressed interest in moving away from the obscure
> > mini-os.  With rump kernel, mini-os remains in the picture AND we are
> > adding NetBSD into the mix.  There are very few developers with the
> > skills needed to make rump kernel happen or contribute to it.
> > (e) The changes that need to be made to upstream qemu to run in a stub
> > domain are a fixed item - they are somewhat independent of the
> > underlying OS/execution environment.  If Xen starts with a Linux-based
> > stubdom, the transition can still be made to rump kernel.
> > (f) The memory overhead issue is overblown.  As time goes on, systems
> > have more memory and memory gets cheaper.  How hard do we work to
> > reduce the 32 MB used for Linux when we're allocating 2+ GB of memory
> > to an HVM domain anyway, and how much of a reduction would we realize
> > anyway?
> >
> > When Anthony released his patches, there was some disappointment
> > expressed with the build process (which kernel version was being used
> > and reliance on busybox, for example).  None of the raised issues
> > strike me as significant problems - just decisions that have to be
> > made.  Anyways, I find it incredible to believe that a MiniOS + NetBSD
> > rump kernel + libc + whatever else solution is going to result in a
> > cleaner build process.
> >
> > Given limited developer resources, sometimes tradeoffs have to be
> > made.  I propose that Linux-based stub domains are the right tradeoff
> > for incorporation into Xen 4.6.  Rump kernels may indeed be the
> > future, but they do not seem to be the immediate solution for qemu
> > stubdoms that we could use.
> >
> > Thank you,
> > Eric
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel