RE: Best way to use altp2m to support VMFUNC EPT-switching?

["Johnson, Ethan" <ejohns48@xxxxxxxxxxxxxxxx>]

-----Original Message-----
From: Andrew Cooper <andrew.cooper3@xxxxxxxxxx> 
Sent: Wednesday, March 15, 2023 5:22 AM
To: Johnson, Ethan <ejohns48@xxxxxxxxxxxxxxxx>;
xen-devel@xxxxxxxxxxxxxxxxxxxx
Subject: [EXT] Re: Best way to use altp2m to support VMFUNC EPT-switching?

> On 15/03/2023 2:01 am, Johnson, Ethan wrote:
>> Hi all,
>>
>> I'm looking for some pointers on how Xen's altp2m system works and how it's
>> meant to be used with Intel's VMFUNC EPT-switching for secure isolation
>> within an HVM/PVH guest's kernelspace.
>>
>> Specifically, I am attempting to modify Xen to create (on request by an
>> already-booted, cooperative guest with a duly modified Linux kernel) a
>> second set of extended page tables that have access to additional
>> privileged regions of host-physical memory (specifically, a page or two to
>> store some sensitive data that we don't want the guest kernel to be able to
>> overwrite, plus some host-physical MMIO ranges, specifically the xAPIC
>> region). The idea is that the guest kernel will use VMFUNC to switch to the
>> alternate EPTs and call "secure functions" provided (by the hypervisor) as
>> read-only code to be executed in non-root mode on the alternate EPT,
>> allowing certain VM-exit scenarios (namely, sending an IPI to another vCPU
>> of the same domain) to be handled without exiting non-root mode. Hence,
>> these extra privileged pages should only be visible to the alternative p2m
>> that the "secure realm" functions live in. (Transitions between the secure-
>> and insecure-realm EPTs will be through special read-only "trampoline" code
>> pages that ensure the untrusted guest kernel can only enter the secure
>> realm at designated entry points.)
>>
>> Looking at Xen's existing altp2m code, I get the sense that Xen is already
>> designed to support something at least vaguely like this. I have not,
>> however, been able to find much in the way of documentation on altp2m, so I
>> am reaching out to see if anyone can offer pointers on how to best use it.
>>
>> What is the intended workflow (either in the toolstack or within the
>> hypervisor itself) for creating and configuring an altp2m that should have
>> access to additional host-physical frames that are not present in the
>> guest's main p2m?
>>
>> FWIW, once the altp2m has been set up in this fashion, we don't anticipate
>> needing to fiddle with its mappings any further as long as the guest is
>> running (so I'm thinking *maybe* the "external" altp2m mode will suffice
>> for this). In fact, we may not even need to have any "overlap" between the
>> primary and alternative p2m except the trampoline pages themselves
>> (although this aspect of our design is still somewhat in flux).
>>
>> I've noticed a function, do_altp2m_op(), in the hypervisor
>> (xen/arch/x86/hvm/hvm.c) that seems to implement a number of altp2m-related
>> hypercalls intended to be called from the dom0. Do these hypercalls already
>> provide a straightforward way to achieve my goals described above entirely
>> via (a potentially modified version of) the dom0 toolstack? Or would I be
>> better off creating and configuring the altp2m from within the hypervisor
>> itself, since I want to map low-level stuff like xAPIC MMIO ranges into the
>> altp2m?
>>
>> Thank you in advance for your time and assistance!
>
> Hello,
> 
> There's a lot to unpack here, but before I do so, one question.  In your
> usecase, are you wanting to map any frames with reduced permissions
> (i.e. such that you'd get a #VE exception), or are you just looking to
> add new frames with RWX perms into an alternative view?
> 
> I suspect the latter, but it's not completely clear, and changes the answer.
> 
> ~Andrew

Yes, the latter is correct: I am looking to add new frames with RWX perms
into an alternative view. I don't currently envision needing #VE in any form
for this work.

(We're using a modified PVH Linux guest for this, so rather than needing to
intercept and react to EPT faults via #VE, we can expect the guest kernel to
explicitly call our secure-realm functions via VMFUNC, replacing what would
otherwise be some hypercalls out to Xen in root mode. I suppose supporting
unmodified kernels by intercepting #VE could be an interesting enhancement
for future work, but for now we're content to work with a cooperative
modified PVH guest as a proof of concept. :-))

Basically, the primary p2m will be (largely) as it is normally, and the
untrusted guest kernel and userspace will run on it as an HVM/PVH guest
normally would. The alternate p2m will have some additional private code and
data pages mapped in (RWX in the altp2m, but either read-only or completely
unmapped in the primary p2m), as well as the host's xAPIC MMIO range so it
can send IPIs to other vCPUs without having to VM-exit. To facilitate safe
transitions between these two "realms", we'll be adding a couple of
R/X-permissioned "trampoline pages" that will contain the VMFUNC instructions
themselves and will be present in both p2ms.

Thanks,

Ethan Johnson
Computer Science PhD candidate, Systems group, University of Rochester
ejohns48@xxxxxxxxxxxxxxxx