Re: Xen/ARM API issue (page size)

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 08/07/2021 01:32, Elliott Mitchell wrote:
> Hopefully I'm not about to show the limits of my knowledge...
>
> Quite a few values passed to Xen via hypercalls include a page number.
> This makes sense as that maps to the hardware.  Problem is, I cannot help
> but notice aarch64 allows for 4KB, 16KB and 64KB pages.

Yes - page size is a know error through the ABI, seeing as Xen started
on x86 and 4k is the only size considered at the time.

32bit frame numbers were all the rage between the Pentum 2 (1997) and
the advent of 64bit systems (~2006), because they let you efficiently
reference up to 16T of physical memory, rather than being limited at 4G
if you used byte addresses instead.

It will be addressed in ABIv2 design (if I ever get enough time to write
everything down and make a start).

> I don't know how flexible aarch64 is.  I don't know whether an aarch64
> core can support multiple page sizes.  My tentative reading of
> information seemed to suggest a typical aarch64 core /could/ allow
> multiple page sizes.
>
> What happens if a system (and Xen) is setup to support 64KB pages, but a
> particular domain has been built strictly with 4KB page support?
>
> What if a particular domain wanted to use 64KB pages (4KB being too
> granular), but Xen was set to use 4KB pages?
>
> What if a system had two domains which were set for different page sizes,
> but the two needed to interact?

I'm afraid I'll have to defer to the arm folk to answer this, but my
understanding is that it should be possible to support guests compiled
with, and using, different page sizes (given a suitable ABI).

> Then you have things like VCPUOP_register_vcpu_info.  The structure is
> setup as mfn and offset.  With the /actual/ page size being used there,
> it is troublesome.  Several places might work better if pure 64-bit
> addresses were used, but with alignment requirements specified.

The way to fix size problems is to mandate that all addresses in the ABI
are full byte addresses, not frame numbers.  When alignment is required,
and it frequently is, it is fine to use the lower bits for metadata.

Critically, what his means is that you don't need separate API/ABI's
based on page size.  e.g. "please balloon out this page" operates "on
the alignment the guest is using", rather than needing separate ops for
4k/2M/1G (to list the x86 page sizes only).

> Then there is a question of what happens when we get a core which has
> more than 64 physical address bits (seems a few years off, but for a long
> time 32 seemed high).

riscv128 is already being discussed, and current generation x86 servers
already have 52 address bits and are using them all (partly NVDIMMs
which take up huge swathes of address space, and the various encrypted
RAM technologies which steal upper address bits for key-ids).

The only sensible way to address this is to introduce new ops mirroring
existing ones, using larger integers.  e.g. get_e820 and get_e820_2
where the latter returns __uint128_t's instead of uint64_t's (or whatever).

Whenever you're talking about systems like this, Xen has to be compiled
for the widest data type, and we know the datatype used by guest kernels
(based on its control settings).  All the compatibility layer needs to
do is zero extend 64bit addresses to form 128bit ones.

~Andrew