Re: Xen/ARM API issue (page size)

[Elliott Mitchell <ehem+xen@xxxxxxx>]

On Thu, Jul 08, 2021 at 05:06:42PM +0100, Julien Grall wrote:
> On 08/07/2021 02:05, Andrew Cooper wrote:
> > On 08/07/2021 01:32, Elliott Mitchell wrote:
> >> Hopefully I'm not about to show the limits of my knowledge...
> >>
> >> Quite a few values passed to Xen via hypercalls include a page number.
> >> This makes sense as that maps to the hardware.  Problem is, I cannot help
> >> but notice aarch64 allows for 4KB, 16KB and 64KB pages.
> > 
> > Yes - page size is a know error through the ABI, seeing as Xen started
> > on x86 and 4k is the only size considered at the time.
> > 
> > 32bit frame numbers were all the rage between the Pentum 2 (1997) and
> > the advent of 64bit systems (~2006), because they let you efficiently
> > reference up to 16T of physical memory, rather than being limited at 4G
> > if you used byte addresses instead.
> > 
> > It will be addressed in ABIv2 design (if I ever get enough time to write
> > everything down and make a start).
> 
> IIRC, ABIv2 will only focus on the interface between the hypervisor and 
> the guests. However, I think we will also need to update the PV protocol 
> so two domains agree on the page granularity used.

I'm inclined to concur with Andrew Cooper here.  It makes a fair bit of
sense to consistently use full addresses across the entire ABI, just
specify alignment so the lower bits end up zeroes.


> Most of the arm64 cores supports all the page granularity. That said, 
> this is not a requirement from the Arm Arm, so it may be possible to 
> have cores only supporting a subset of the page granularity.

At which point it is possible to have a device where the page size(s)
supported by some cores are disjoint from the page size(s) supported by
other cores.

I imagine someone has plans.  An obvious use case would be a cellphone
chip with a low-power core for the modem and a high-power OS core.


> >> What happens if a system (and Xen) is setup to support 64KB pages, but a
> >> particular domain has been built strictly with 4KB page support?
> 
> If the processor only support 64KB, then you would not be able to boot a 
> 4KB kernel there.

I was being explicit about covering both cases of distinct page sizes
between Xen and domain (Xen with smaller page size, domain with smaller
page size).


> >> What if a particular domain wanted to use 64KB pages (4KB being too
> >> granular), but Xen was set to use 4KB pages?
> Today the hypercall ABI using the same page granularity as the 
> hypervisor. IOW, the domain would need to break its page in 4KB chunk to 
> talk to the hypervisor.
> 
> FWIW, this is how Linux with 64KB/16KB page granularity is able to run 
> on current Xen.

Breaking pages up is generally easier than putting them back together.
Good news is this could be handled similar to DMA operations and a few
pages reserved for interaction with small page domains.


> >> What if a system had two domains which were set for different page sizes,
> >> but the two needed to interact?
> 
> They would need to agree on the page granularity used. At the moment, 
> this is implicitely fixed to 4KB.

"implicitly" -> "undocumented" -> "guess" -> "12 hour build wasted"

For the case I'm concerned with, the history is a decent hint, but not
being explicitly documented is Bad.  In the Xen ABI there are too many
references to "page size" without the page size being defined as 4KB.

In a few years there may be someone on this list who assumed "page size"
meant whatever page size was in use and will be rather annoyed it means
4096, when both Xen and their OS were using 65536.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@xxxxxxx  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445