RE: Proposal for Porting Xen to Armv8-R64 - DraftB

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Wed, 20 Apr 2022, Wei Chen wrote:
> > On Tue, 19 Apr 2022, Wei Chen wrote:
> > > > > ### 3.2. Xen Event Channel Support
> > > > >     In Current RFC patches we haven't enabled the event channel
> > support.
> > > > >     But I think it's good opportunity to do some discussion in
> > advanced.
> > > > >     On Armv8-R, all VMs are native direct-map, because there is no
> > > > stage2
> > > > >     MMU translation. Current event channel implementation depends on
> > > > some
> > > > >     shared pages between Xen and guest: `shared_info` and per-cpu
> > > > `vcpu_info`.
> > > > >
> > > > >     For `shared_info`, in current implementation, Xen will allocate
> > a
> > > > page
> > > > >     from heap for `shared_info` to store initial meta data. When
> > guest
> > > > is
> > > > >     trying to setup `shared_info`, it will allocate a free gfn and
> > use a
> > > > >     hypercall to setup P2M mapping between gfn and `shared_info`.
> > > > >
> > > > >     For direct-mapping VM, this will break the direct-mapping
> > concept.
> > > > >     And on an MPU based system, like Armv8-R system, this operation
> > will
> > > > >     be very unfriendly. Xen need to pop `shared_info` page from Xen
> > heap
> > > > >     and insert it to VM P2M pages. If this page is in the middle of
> > > > >     Xen heap, this means Xen need to split current heap and use
> > extra
> > > > >     MPU regions. Also for the P2M part, this page is unlikely to
> > form
> > > > >     a new continuous memory region with the existing p2m pages, and
> > Xen
> > > > >     is likely to need another additional MPU region to set it up,
> > which
> > > > >     is obviously a waste for limited MPU regions. And This kind of
> > > > dynamic
> > > > >     is quite hard to imagine on an MPU system.
> > > >
> > > > Yeah, it doesn't make any sense for MPU systems
> > > >
> > > >
> > > > >     For `vcpu_info`, in current implementation, Xen will store
> > > > `vcpu_info`
> > > > >     meta data for all vCPUs in `shared_info`. When guest is trying
> > to
> > > > setup
> > > > >     `vcpu_info`, it will allocate memory for `vcpu_info` from guest
> > side.
> > > > >     And then guest will use hypercall to copy meta data from
> > > > `shared_info`
> > > > >     to guest page. After that both Xen `vcpu_info` and guest
> > `vcpu_info`
> > > > >     are pointed to the same page that allocated by guest.
> > > > >
> > > > >     This implementation has serval benifits:
> > > > >     1. There is no waste memory. No extra memory will be allocated
> > from
> > > > Xen heap.
> > > > >     2. There is no P2M remap. This will not break the direct-mapping,
> > > > and
> > > > >        is MPU system friendly.
> > > > >     So, on Armv8-R system, we can still keep current implementation
> > for
> > > > >     per-cpu `vcpu_info`.
> > > > >
> > > > >     So, our proposal is that, can we reuse current implementation
> > idea
> > > > of
> > > > >     `vcpu_info` for `shared_info`? We still allocate one page for
> > > > >     `d->shared_info` at domain construction for holding some initial
> > > > meta-data,
> > > > >     using alloc_domheap_pages instead of alloc_xenheap_pages and
> > > > >     share_xen_page_with_guest. And when guest allocates a page for
> > > > >     `shared_info` and use hypercall to setup it,  We copy the
> > initial
> > > > data from
> > > > >     `d->shared_info` to it. And after copy we can update `d-
> > > > >shared_info` to point
> > > > >     to guest allocated 'shared_info' page. In this case, we don't
> > have
> > > > to think
> > > > >     about the fragmentation of Xen heap and p2m and the extra MPU
> > > > regions.
> > > >
> > > > Yes, I think that would work.
> > > >
> > > > Also I think it should be possible to get rid of the initial
> > > > d->shared_info allocation in Xen, given that d->shared_info is for the
> > > > benefit of the guest and the guest cannot access it until it makes the
> > > > XENMAPSPACE_shared_info hypercall.
> > > >
> > >
> > > While we're working on event channel PoC work on Xen Armv8-R, we found
> > > another issue after we dropped d->shared_info allocation in Xen. Both
> > > shared_info and vcpu_info are allocated from Guest in runtime. That
> > > means the addresses of shared_info and vcpu_info are random. For MMU
> > > system, this is OK, because Xen has a full view of system memory in
> > > runtime. But for MPU system, the situation becomes a little tricky.
> > > We have to setup extra MPU regions for remote domains' shared_info
> > > and vcpu_info in event channel hypercall runtime. That's because
> > > in current Xen hypercall concept, hypercall will not cause vCPU
> > > context switch. When hypercall trap to EL2, it will keep vCPU's
> > > P2M view. For MMU system, we have vttbr_el2 for vCPU P2M view and
> > > ttbr_el2 for Xen view. So in EL2 Xen has full permissions to access
> > > any memory it wants. But for MPU system, we only have one EL2 MPU.
> > > Before entering guest, Xen will setup vCPU P2M view in EL2 MPU.
> > > In this case, when system entry EL2 through hypercall, the EL2
> > > MPU still keeps current vCPU P2M view and with Xen essential
> > > memory (code, data, heap) access permissions. But current EL2 MPU
> > > doesn't have the access permissions for EL2 to access other
> > > domain's memory. For an event channel hypercall, if we want to
> > > update the pending bitmap in remote domain's vcpu_info, it will
> > > cause a dataabort in EL2. To solve this dataabort, we may have
> > > two methods:
> > > 1. Map remote domain's whole memory or pages for shared_info +
> > >    vcpu_info in EL2 MPU temporarily for hypercall to update
> > >    pending bits or other accesses.
> > >
> > >    This method doesn't need to do context switch for EL2 MPU,
> > >    But this method has some disadvantages:
> > >    1. We have to reserve MPU regions for hypercall.
> > >    2. Different hypercall may have different reservation of
> > >       MPU regions.
> > >    3. We have to handle hypercall one by one for existed and
> > >       new in future.
> > >
> > > 2. Switch to Xen's memory view in EL2 MPU when trap from EL1 to
> > >    EL2. In this case, Xen will have full memory access permissions
> > >    to update pending bits in EL2. This only changes the EL2 MPU
> > >    context, does not need to do vCPU context switch. Because the
> > >    trapped vCPU will be used in the full flow of hypercall. After
> > >    the hypercall, before returning to EL2, the EL2 MPU will switch
> > >    to scheduled vCPU' P2M view.
> > >    This method needs to do EL2 MPU context switch, but:
> > >    1. We don't need to reserve MPU regions for Xen's memory view.
> > >       (Xen's memory view has been setup while initialization)
> > >    2. We don't need to handle pages' mapping in hypercall level.
> > >    3. Apply to other EL1 to EL2 traps, like dataabort, IRQ, etc.
> > 
> > 
> > Both approach 1) and 2) are acceptable and in fact I think we'll
> > probably have to do a combination of both.
> > 
> > We don't need to do a full MPU context switch every time we enter Xen.
> > We can be flexible. Only when Xen needs to access another guest memory,
> > if the memory is not mappable using approach 1), Xen could do a full MPU
> > context switch. Basically, try 1) first, if it is not possible, do 2).
> > 
> > This also solves the problem of "other hypercalls". We can always do 2)
> > if we cannot do 1).
> > 
> > So do we need to do 1) at all? It really depends on performance data.
> > Not all hypercalls are made equal. Some are very rare and it is fine if
> > they are slow. Some hypercalls are actually on the hot path. The event
> > channels hypercalls are on the hot path so they need to be fast. It
> > makes sense to implement 1) just for event channels hypercalls if the
> > MPU context switch is slow.
> > 
> > Data would help a lot here to make a good decision. Specifically, how
> > much more expensive is an EL2 MPU context switch compared to add/remove
> > of an MPU region in nanosec or cpu cycles?
> > 
> 
> We will do it when we get a proper platform.
> 
> > 
> > The other aspect is how many extra MPU regions do we need for each guest
> > to implement 1). Do we need one extra MPU region for each domU? If so, I
> > don't think approach 1) if feasible unless we come up with a smart
> > memory allocation scheme for shared_info and vcpu_info. For instance, if
> > shared_info and vcpu_info of all guests were part of the Xen data or
> > heap region, or 1 other special MPU region, then they could become
> > immediately accessible without need for extra mappings when switching to
> > EL2.
> > 
> 
> Allocate shared_info and vcpu_info from Xen data or heap will cause memory
> fragmentation. We have to split the Xen data or heap and populate the pages
> for shared_info and vcpu_info, And insert them to Guest P2M. Because Armv8-R
> MPU doesn't allow memory overlap, this will cause at least 2 extra MPU
> regions usage. One page could not exist in Xen MPU region and Guest P2M
> MPU region at the same time. And we definitely don't want to make the entire
> Xen data and heap accessible to EL1. And this approach does not solve the
> 100% direct mapping problem. A special MPU region might have the same issues.
> Except we make this special MPU region can be accessed in EL1 and EL2 at
> runtime (it's unsafe), and update hypercall to use pages from this special
> region for shared_info and vcpu_info (every guest can see this region, so
> it's still 1:1 mapping).
> 
> For 1), the concern is caused by our current rough PoC, we used extra MPU
> regions to map the whole memory of remote domain, whose may have serval
> memory blocks in the worst case. We have thought it further, we can reduce
> the map granularity to page. For example, Xen wants to update shared_info
> or vcpu_info, Xen must know the address of it. So we can just map this
> one page temporarily. So I think only reserve 1 MPU region for runtime
> mapping is feasible on most platforms.

Actually I think that it would be great if we can do that. It looks like
the best way forward.


> But the additional problem with this is that if the hypercall are
> modifying multiple variables, Xen may need to do multiple mappings if
> they are not on the same page (or a proper MPU region range).

There are not that many hypercalls that require Xen to map multiple
pages, and those might be OK if they are slow.