Re: [Xen-devel] PVHv2/HVM memory layout and interface specification

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 23/07/2019 00:50, Johnson, Ethan wrote:
> Hi all,
>
> I'm interested in using Xen as a research platform for enforcing novel memory 
> protection schemes on hypervisors and guests. Specifically, I'm looking to do 
> this in an environment containing PVH(v2) and potentially HVM guests.
>
> To know what my options are for this, I need to know where the Xen hypervisor 
> "lives" in the system's memory layouts (both virtual and physical, and within 
> guests' spaces as virtualized by extended paging). I found a Google Project 
> Zero article from 2017 
> (https://googleprojectzero.blogspot.com/2017/04/pandavirtualization-exploiting-xen.html)
>  that explains the memory layout for classic PV mode, but I couldn't find 
> anything for PVH or HVM, either in the Xen documentation, the wiki, or 
> general web searches.
>
> So, I thought I'd ask here...what's the general lay of the land in terms of 
> how Xen lays out memory for itself and guests in PVH and HVM modes? Is this 
> documented anywhere, e.g. in some sort of specification for the PVHv2 
> paravirtualization interface? I looked for such a specification document but 
> couldn't find anything particularly complete; the closest I found was 
> http://xenbits.xen.org/docs/4.12-testing/misc/pvh.html, which didn't have a 
> lot of detail. (The Xen 4.7 version of the documentation had a little bit 
> more detail at https://xenbits.xen.org/docs/4.7-testing/misc/pvh.html, but it 
> still doesn't have quite what I'm looking for, and I'm not sure if that's for 
> PVHv2 or the older PVHv1.)

That document was for PVHv1 which is completely obsolete and gone now. 
As for these days...

Xen has two fundamental types of virtualisation.  Ring deprivileging
which is what PV guests use, and hardware extensions (Intel VT-x, AMD
SVM) which is what HVM guests use.  PVH is identical to HVM in this
regard; what distinguishes PVH is that there is no Qemu providing an
emulation of a IBM-compatible PC behind the scenes.

PV guests share an address space with Xen.  In this regard, they are
very similar to userspace sharing pagetables with the guest kernel.  In
our case, we distinguish guest user, guest kernel, and Xen, but the same
basic principles hold.

HVM guests get an entirely separate address space, either provided by
hardware extensions (Intel EPT, AMD NPT), or via shadow pagetables. 
Either way, Xen itself is not a part of this address space, but does
manage it from the outside.

>
> Specifically, I'm wondering:
>
> 1. Where does the Xen hypervisor itself live in the host's (VMX root) virtual 
> and physical address spaces under PVH and HVM? In classic PV it seems it 
> lived just below the guest kernel, but that was without extended paging 
> separating the host and guest's views of memory.

Where exactly Xen lives is VMX root mode is of no concern to an HVM/PVH
guest.  That said, it is in practice the same as for PV so we can run PV
and HVM/PVH guests at the same time.  (Its only recently we've introduce
the ability to selectively compile out PV or HVM support.)

http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/asm-x86/config.h;h=9ef9d03ca7199ccca416c6ea2d24adaf6dbc4e0f;hb=refs/heads/staging#l116
describes Xen's virtual memory layout, including the bits applicable to
PV guests.

Virtually, Xen lives in the first 8Tb of address space of the upper
canonical half, while physically, the only restriction is that it must
be in the bottom 4G of RAM.  The exact location is machine specific, but
defaults to highest (sufficiently sized) block of RAM below the 4G boundary.

> 2. Does Xen also live somewhere within the guest's (VMX non-root) view of 
> physical memory as provided by extended paging, in PVH mode? (I'm guessing it 
> wouldn't for HVM, but I'm wondering about PVH because it did for classic PV.)

No.  For PVH (technically v2, but we don't call it that any more), think
"HVM without Qemu".

PVHv1 was very misguided, which is why it no longer exists.

> 3. How are hypercalls handled in PVH/HVM mode? Do they use the VMCALL 
> instruction (or something comparable) to initiate a VM exit, or are they 
> sometimes handled within VMX non-root mode without a VM exit, similar to 
> classic PV?

Ah - this document is one I prepared earlier.  To answer your lower
question, our docs are currently terrible, and I'm trying to improve
things.  This is the first part of what I hope will be a document that
would eventually cover all of your questions.  Feedback welcome,
including if there is anything unclear in the few bits which currently
exist.

http://xenbits.xen.org/docs/sphinx-unstable/guest-guide/x86/hypercall-abi.html


> 4. How does Xen access the guest's memory when it needs to examine or modify 
> it? Does it just access the physical frames through a direct map in VMX root 
> mode, or are guests' views of physical memory somehow represented somewhere 
> in the host's (hypervisor's) virtual address space?

Xen has a directmap of host memory, but we're looking to reduce the use
of that as a defence against speculative sidechannels.  Xen does not
have a 1:1 directmap of guest physical address space (i.e. we don't have
anything like Qemu/KVM has).

All physical accesses into guest memory start by walking the P2M (the
EPT/NPT or Shadow pagetables) to find the target data.  For accesses
which are based on virtual addresses rather than guest physical, we
perform a pagewalk of the guests pagetables to arrive at the target.

For emulated MMIO, frames may not be present, at which case special
handling kicks in, such as forwarding the IO request to Qemu, or
terminating it with default x86 behaviour (read ~0, write discard).

> 5. How do PVH guests learn what their physical memory layout looks like? 
> Which regions of memory do they use to communicate with Xen? How is this 
> process different for HVM (non-PVH) guests?

By default, the E820 table passed at boot in the PVH start_info
structure.  There is also a hypercall which can be used to obtain a copy
of the E820 table which the toolstack produced.

> 6. How does dom0 factor into all this? Does it have any direct access to 
> domUs' physical memory pages, or does it have to ask Xen to do that on its 
> behalf with a hypercall?

Dom0 is just a plain VM, with some default permissions.  In this regard,
it is just like a root process running with more privilege than regular
userspace.

One concept Xen has is that of a "foreign memory map" which allows dom0
(or other sufficiently privileged domains) to map guest memory.  This is
used for all kinds of tasks, from booting the guest in the first place,
to live migration, and for Qemu to emulate DMA access for an HVM guest.
>
> Thanks in advance for any help anyone can offer. If there are any good 
> resources out there (books? blog posts?) for understanding the big-picture 
> structure of the Xen architecture and codebase as it exists in its modern 
> form (i.e., in the PVH/HVM era), those would be especially appreciated.

I am trying to make things better.  For now, asking here is probably
your best bet.

In some copious free time, I'll try translating this into the guest
guide which was linked above.

I hope this is all clear and easy to follow.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel