Re: [Xen-devel] PVHv2/HVM memory layout and interface specification

["Johnson, Ethan" <ejohns48@xxxxxxxxxxxxxxxx>]

On 2019-07-22 21:11:40 -0400, Andrew Cooper wrote:
> On 23/07/2019 00:50, Johnson, Ethan wrote:
> > (The Xen 4.7 version of the documentation had a little bit
> > more detail at https://xenbits.xen.org/docs/4.7-testing/misc/pvh.html, but
> > it still doesn't have quite what I'm looking for, and I'm not sure if
> > that's for PVHv2 or the older PVHv1.)
> 
> That document was for PVHv1 which is completely obsolete and gone now. 
> As for these days...
> 
> Xen has two fundamental types of virtualisation.  Ring deprivileging
> which is what PV guests use, and hardware extensions (Intel VT-x, AMD
> SVM) which is what HVM guests use.  PVH is identical to HVM in this
> regard; what distinguishes PVH is that there is no Qemu providing an
> emulation of a IBM-compatible PC behind the scenes.
> 
> PV guests share an address space with Xen.  In this regard, they are
> very similar to userspace sharing pagetables with the guest kernel.  In
> our case, we distinguish guest user, guest kernel, and Xen, but the same
> basic principles hold.
> 
> HVM guests get an entirely separate address space, either provided by
> hardware extensions (Intel EPT, AMD NPT), or via shadow pagetables. 
> Either way, Xen itself is not a part of this address space, but does
> manage it from the outside.
> 
> > 1. Where does the Xen hypervisor itself live in the host's (VMX root)
> > virtual and physical address spaces under PVH and HVM? In classic PV it
> > seems it lived just below the guest kernel, but that was without extended
> > paging separating the host and guest's views of memory.
> 
> Where exactly Xen lives is VMX root mode is of no concern to an HVM/PVH
> guest.  That said, it is in practice the same as for PV so we can run PV
> and HVM/PVH guests at the same time.  (Its only recently we've introduce
> the ability to selectively compile out PV or HVM support.)
> 
> http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/asm-x86/config.h;h=9ef9d03ca7199ccca416c6ea2d24adaf6dbc4e0f;hb=refs/heads/staging#l116
> describes Xen's virtual memory layout, including the bits applicable to
> PV guests.
> 
> Virtually, Xen lives in the first 8Tb of address space of the upper
> canonical half, while physically, the only restriction is that it must
> be in the bottom 4G of RAM.  The exact location is machine specific, but
> defaults to highest (sufficiently sized) block of RAM below the 4G boundary.
> 
> > 2. Does Xen also live somewhere within the guest's (VMX non-root) view of
> > physical memory as provided by extended paging, in PVH mode? (I'm guessing
> > it wouldn't for HVM, but I'm wondering about PVH because it did for classic
> > PV.)
> 
> No.  For PVH (technically v2, but we don't call it that any more), think
> "HVM without Qemu".
> 
> PVHv1 was very misguided, which is why it no longer exists.
> 
> > 3. How are hypercalls handled in PVH/HVM mode? Do they use the VMCALL 
> > instruction (or something comparable) to initiate a VM exit, or are they 
> > sometimes handled within VMX non-root mode without a VM exit, similar to 
> > classic PV?
> 
> Ah - this document is one I prepared earlier.  To answer your lower
> question, our docs are currently terrible, and I'm trying to improve
> things.  This is the first part of what I hope will be a document that
> would eventually cover all of your questions.  Feedback welcome,
> including if there is anything unclear in the few bits which currently
> exist.
> 
> http://xenbits.xen.org/docs/sphinx-unstable/guest-guide/x86/hypercall-abi.html
> 
> 
> > 4. How does Xen access the guest's memory when it needs to examine or
> > modify it? Does it just access the physical frames through a direct map in
> > VMX root mode, or are guests' views of physical memory somehow represented
> > somewhere in the host's (hypervisor's) virtual address space?
> 
> Xen has a directmap of host memory, but we're looking to reduce the use
> of that as a defence against speculative sidechannels.  Xen does not
> have a 1:1 directmap of guest physical address space (i.e. we don't have
> anything like Qemu/KVM has).
> 
> All physical accesses into guest memory start by walking the P2M (the
> EPT/NPT or Shadow pagetables) to find the target data.  For accesses
> which are based on virtual addresses rather than guest physical, we
> perform a pagewalk of the guests pagetables to arrive at the target.
> 
> For emulated MMIO, frames may not be present, at which case special
> handling kicks in, such as forwarding the IO request to Qemu, or
> terminating it with default x86 behaviour (read ~0, write discard).
> 
> > 5. How do PVH guests learn what their physical memory layout looks like? 
> > Which regions of memory do they use to communicate with Xen? How is this 
> > process different for HVM (non-PVH) guests?
> 
> By default, the E820 table passed at boot in the PVH start_info
> structure.  There is also a hypercall which can be used to obtain a copy
> of the E820 table which the toolstack produced.
> 
> > 6. How does dom0 factor into all this? Does it have any direct access to 
> > domUs' physical memory pages, or does it have to ask Xen to do that on its 
> > behalf with a hypercall?
> 
> Dom0 is just a plain VM, with some default permissions.  In this regard,
> it is just like a root process running with more privilege than regular
> userspace.
> 
> One concept Xen has is that of a "foreign memory map" which allows dom0
> (or other sufficiently privileged domains) to map guest memory.  This is
> used for all kinds of tasks, from booting the guest in the first place,
> to live migration, and for Qemu to emulate DMA access for an HVM guest.
> 
> > Thanks in advance for any help anyone can offer. If there are any good 
> > resources out there (books? blog posts?) for understanding the big-picture 
> > structure of the Xen architecture and codebase as it exists in its modern 
> > form (i.e., in the PVH/HVM era), those would be especially appreciated.
> 
> I am trying to make things better.  For now, asking here is probably
> your best bet.
> 
> In some copious free time, I'll try translating this into the guest
> guide which was linked above.
> 
> I hope this is all clear and easy to follow.
> 
> ~Andrew

Thanks for taking the time to write out all these great answers - this is
exactly what I was looking for!

(Apologies for not responding sooner; for whatever reason, your reply didn't
make it to my inbox and I had to find it on the list's web archives. I
suspect one of my own inbox filters I set up to winnow this high-traffic list is
to blame...)

Sincerely,
Ethan Johnson

-- 
Ethan J. Johnson
Computer Science PhD student, Systems group, University of Rochester
ejohns48@xxxxxxxxxxxxxxxx
ethanjohnson@xxxxxxx
PGP public key available from public directory or on request

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel