Re: Xen Memory Sharing Query

[Andrew Cooper <Andrew.Cooper3@xxxxxxxxxx>]

On 15/07/2022 16:56, Marc Bonnici wrote:
> Hi All,
>
> I was wondering if someone could help me understand some of the rules of the 
> memory sharing implementation in Xen?
>
> Specifically I'm looking to understand what restrictions Xen places on
> granting access from one Dom to another from Xen's perspective, and what 
> types 
> of grant requests would be allowed/rejected by Xen?
>
> I.e. How would the situation be handled if the same frame of memory was 
> attempted 
> to be shared multiple times?
>
> As an example scenario, DomA shares 1 physical page of memory in a 
> transaction 
> with DomB. And then without releasing any memory, DomA attempts to share
> another region of memory, which includes the same physical page of the 
> previous share
> with DomB again. This would result in two concurrent shares containing an 
> overlap.
>
> Apologies if I've missed something but is there any documentation / threat 
> model
> that would cover these types of scenarios? So far I have been trying to read 
> through 
> the code but was wondering if there is something else I could refer to help 
> me 
> with my understanding?

There's nothing adequately written down.  It ought to live in sphinx
docs, but my copious free time is non-existent for speculative security
reasons.

This all pertains to gnttab v1 which is the only supported one on ARM
right now.  gnttab v2 is horribly more complicated.  Refer to
https://github.com/xen-project/xen/blob/master/xen/include/public/grant_table.h#L132-L186

When DomA and DomB are set up and running, they each have a grant
table.  The grant table is some shared memory (of Xen's) mapped into the
guest, and is a bidirectional communication interface between the guest
kernel and Xen.

The guest kernel logically owns the grant table, and it's a simple array
of grant entries.  Entries 0 thru 7 are reserved for system use, and
indeed two entries (one for xenstore, one for xenconsole) are set up on
the guest kernel's behalf by the domain builder.  Entries 8 thru $N are
entirely under the guest's control.

A guest kernel (domA) creates a grant by filling in a grant table entry,
and passing the grant reference (the entry's index in the table) to some
other entity in the system (in this case, domB).

The grant table entry is formed of:

u16 flags
u16 domid
u32 frame

so for domA to grant a frame to domB, it would pick a free gref (any
entry in the table with flags=0) and fill in:

frame = f
domid = domB
smp_wmb()
flags = GTF_permit_access (think "grant usable")

GFT_readonly is another relevant flag that domA might choose to set.

Then, domB would take the gref it has been given by domA, and make a
gnttab_op_map() hypercall, passing {domA, gref} as an input.

Xen looks up gref in domA's grant table, checks e.g. domA granted access
to domB, and if everything is happy, sets the the GFT_{reading,writing}
flags (as appropriate) in flags.  This tells domA that the grant is
currently mapped readably and/or writeably.

Later, when domB unmaps the grant, Xen clears the GFT_{reading,writing}
bits, telling domA that the grant is no longer in use.

DomA then clears GTF_permit_access to mark this as gref as invalid, and
can then free the frame.


Now, that's the simple overview.  To answer some of your specific questions:

DomA is perfectly free to grant away the same frame multiple times. 
DomA does this by writing multiple different grefs with the same frame
field.  These grefs could be to the same, or different domains, and can
have any (valid) combination of flags.

DomB is perfectly free to map the same gref multiple times.  This is
actually a necessity for x86 PV guests, because of how we reference
count pagetable entries.  It is not necessary for any kind of guest of
HVM guest (x86 or ARM) because of how guest physical address space works.

IMO it should have been restricted when the HVM ABI was designed, but
alas.  In practice, Xen has an internal refcount which prevents a gref
being mapped more than 127 times IIRC.

While a gref is mapped, domA is not permitted to edit the associated
entry.  Doing so shouldn't cause a security violation (Xen has a local
copy of the entry in the maptrack table), but will at least confuse
diagnostics of the granted state.

Importantly, and what may come as a surprise, is that domA has no way to
revoke a currently-mapped grant.  Fixing this limitation has been
discussed several times; there are some very complicated corner cases,
and I'm not aware of any work having started in earnest.

Xen does have logic to unmap grants of VMs which have shut down (for
whatever reason) with grants still mapped.  This prevents deadlocks
(e.g. two domains grant to each other, then both crash deliberately).


From a grant perspective, Xen doesn't enforce any policy.  domA's grefs
can be mapped in the manner permitted by what domA wrote into the grant
table.

If you want to get into policies that Xen may enforce, that would be a
discussion about XSM, Xen Security Modules.

Does any of this help?

~Andrew