Re: [RFC 0/4] Adding Virtual Memory Fuses to Xen

[Julien Grall <julien@xxxxxxx>]

On 13/12/2022 19:48, Smith, Jackson wrote:
> Hi Xen Developers,
Hi Jackson,

Thanks for sharing the prototype with the community. Some 
questions/remarks below.
> My team at Riverside Research is currently spending IRAD funding
> to prototype next-generation secure hypervisor design ideas
> on Xen. In particular, we are prototyping the idea of Virtual
> Memory Fuses for Software Enclaves, as described in this paper:
> https://www.nspw.org/papers/2020/nspw2020-brookes.pdf. Note that
> that paper talks about OS/Process while we have implemented the idea
> for Hypervisor/VM.
>
> Our goal is to emulate something akin to Intel SGX or AMD SEV,
> but using only existing virtual memory features common in all
> processors. The basic idea is not to map guest memory into the
> hypervisor so that a compromised hypervisor cannot compromise
> (e.g. read/write) the guest. This idea has been proposed before,
> however, Virtual Memory Fuses go one step further; they delete the
> hypervisor's mappings to its own page tables, essentially locking
> the virtual memory configuration for the lifetime of the system. This
> creates what we call "Software Enclaves", ensuring that an adversary
> with arbitrary code execution in the hypervisor STILL cannot read/write
> guest memory.
I am confused, if the attacker is able to execute arbitrary code, then 
what prevent them to write code to map/unmap the page?
Skimming through the paper (pages 5-6), it looks like you would need to 
implement extra defense in Xen to be able to prevent map/unmap a page.
> With this technique, we protect the integrity and confidentiality of
> guest memory. However, a compromised hypervisor can still read/write
> register state during traps, or refuse to schedule a guest, denying
> service. We also recognize that because this technique precludes
> modifying Xen's page tables after startup, it may not be compatible
> with all of Xen's potential use cases. On the other hand, there are
> some uses cases (in particular statically defined embedded systems)
> where our technique could be adopted with minimal friction.
From what you wrote, this sounds very much like the project Citrix and 
Amazon worked on called "Secret-free hypervisor" with a twist. In your 
case, you want to prevent the hypervisor to map/unmap the guest memory.
You can find some details in [1]. The code is x86 only, but I don't see 
any major blocker to port it on arm64.
> With this in mind our goal is to work with the Xen community to
> upstream this work as an optional feature. At this point, we have
> a prototype implementation of VMF on Xen (the contents of this RFC
> patch series) that supports dom0less guests on arm 64. By sharing
> our prototype, we hope to socialize our idea, gauge interest, and
> hopefully gain useful feedback as we work toward upstreaming.
>
> ** IMPLEMENTATION **
> In our current setup we have a static configuration with dom0 and
> one or two domUs. Soon after boot, Dom0 issues a hypercall through
> the xenctrl interface to blow the fuse for the domU. In the future,
> we could also add code to support blowing the fuse automatically on
> startup, before any domains are un-paused.
>
> Our Xen/arm64 prototype creates Software Enclaves in two steps,
> represented by these two functions defined in xen/vmf.h:
> void vmf_unmap_guest(struct domain *d);
> void vmf_lock_xen_pgtables(void);
>
> In the first, the Xen removes mappings to the guest(s) On arm64, Xen
> keeps a reference to all of guest memory in the directmap. Right now,
> we simply walk all of the guest second stage tables and remove them
> from the directmap, although there is probably a more elegant method
> for this.
IIUC, you first map all the RAM and then remove the pages. What you 
could do instead is to map only the memory required for Xen use. The 
rest would be left unmapped.
This would be similar to what we are doing on arm32. We have a split 
heap. Only the xenheap is mapped. The pages from the domheap will be 
mapped ondemand.
Another approach, would be to have a single heap where pages used by Xen 
are mapped in the page-tables when allocated (this is what secret-free 
hypervisor is doing is).
If you don't map to keep the page-tables around, then it sounds like you 
want the first approach.
> Second, the Xen removes mappings to its own page tables.
> On arm64, this also involves manipulating the directmap. One challenge
> here is that as we start to unmap our tables from the directmap,
> we can't use the directmap to walk them. Our solution here is also
> bit less elegant, we temporarily insert a recursive mapping and use
> that to remove page table entries.
See above.

> ** LIMITATIONS and other closing thoughts **
> The current Xen code has obviously been implemented under the
> assumption that new pages can be mapped, and that guest virtual
> addresses can be read, so this technique will break some Xen
> features. However, in the general case
Can you clarify your definition of "general case"? From my PoV, it is a 
lot more common to have guest with PV emulated device rather than with 
device attached. So it will be mandatory to access part of the memory 
(e.g. grant table).
> (in particular for static
> workloads where the number of guest's is not changed after boot)
That very much depend on how you configure your guest. If they have 
device assigned then possibly yes. Otherwise see above.
> Finally, our initial testing suggests that Xen never reads guest memory
> (in a static, non-dom0-enchanced configuration), but have not really
> explored this thoroughly.
> We know at least these things work:
>         Dom0less virtual serial terminal
>         Domain scheduling
> We are aware that these things currently depend on accessible guest
> memory:
>         Some hypercalls take guest pointers as arguments
There are not many hypercalls that don't take guest pointers.

>         Virtualized MMIO on arm needs to decode certain load/store
>         instructions
On Arm, this can be avoided of the guest OS is not using such 
instruction. In fact they were only added to cater "broken" guest OS.
Also, this will probably be a lot more difficult on x86 as, AFAIK, there 
is no instruction syndrome. So you will need to decode the instruction 
in order to emulate the access.
> It's likely that other Xen features require guest memory access.
For Arm, guest memory access is also needed when using the GICv3 ITS 
and/or second-level SMMU (still in RFC).
For x86, if you don't want to access the guest memory, then you may need 
to restrict to PVH as for HVM we need to emulate some devices in QEMU. 
That said, I am not sure PVH is even feasible.
Cheers,

[1] 
https://www.youtube.com/watch?v=RKJOwIkCnB4&list=PLYyw7IQjL-zFYmEoZEYswoVuXrHvXAWxj&index=5
--
Julien Grall