[Xen-devel] Reducing or removing direct map from xen (was Re: Ongoing/future speculative mitigation work)

[Wei Liu <wei.liu2@xxxxxxxxxx>]

Below is a summary for a discussion on this topic between Jan and me.

End goal: reduce the size of direct map or remove it completely

Constraints:

1. We want unified xenheap and domheap.
2. We want to preserve xenheap's semantics -- always globally mapped
   so that pointers can be stashed safely.
3. Performance shouldn't be heavily impacted -- to be tested.

Things need to be done:

1. Remove map domain page infra's dependency on direct map

Currently map_domain_page uses either direct map or per-domain slot.
Direct map is used when Xen runs in EFI context or before any domain is
constructed, or when in non-debug build as a fast path.

In the new world when there isn't a direct map, we have at least two
options to set aside address space to map_domain_page:

  1.1. Implement percpu infrastructure
  1.2. Statically allocate address space to each CPU

1.1 requires:

  1.1.1 Carve out some address space
  1.1.2 Adjust early setup code
  1.1.3 Use fixmap to bootstrap percpu region
  1.1.4 Change context-switching code

1.2 is a bit simpler given that it doesn't require changing
context-switching code. Yet it is also less flexible once we decide we
want to move more stuff into the percpu region.

2. Remove map_domain_page_global's dependency on direct map

This is easy.

3. Implement xenheap using vmap infrastructure

This helps preserve xenheap's "always mapped" property. At the moment,
vmap relies on xenheap, we want to turn this relationship around.

There is a loop what needs breaking in the new world:

  alloc_xenheap_pages -> vmap -> __vmap -> map_pages_to_xen ->
    virt_to_xen_l1e -> alloc_xen_pagetable -> alloc_xenheap_page -> vmap ...

Two options were proposed to break this loop:

  3.1 Pre-populate all page tables for vmap region
  3.2 Switch page table allocation to use domheap page

3.1 is wasteful since we expect vmap to grow in the future. 3.2 requires
a lot of code churn -- the assumption up until now is xen's page tables
are always mapped, and a lot of code and APIs is designed based on that.

We think that 3.2 is a worthwhile thing to do anyway. This work just
gives us a good excuse to do it.

The other work item is to track page<->virt relationship so that
conversion functions (_to_virt etc) continue to work. For PoC purpose,
putting a void * into page_info is good enough. But in the future we
want to have a separate array for tracking so that page_info stays power
of two in size.

(The other option for the conversion functions is to purge them all.
That's a lot of code churn and, more importantly, touches common code.
So that idea was discarded.)

4. Remove or reduce direct map

This is a matter of changing some constants. It is conceptually easy but
I expect quite a bit of bug fixing is needed.

Tl;dr

I have broken down this project into several sub-projects and recorded
their relationship starting from XEN-119.

  https://xenproject.atlassian.net/browse/XEN-119

Comments are welcome.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel