[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Uses of /hypervisor memory range (was: FreeBSD/Xen/ARM issues)

(+ Andrew, + Stefano)

On 15/05/2021 02:18, Elliott Mitchell wrote:
On Fri, May 14, 2021 at 09:32:10AM +0100, Julien Grall wrote:
On 14/05/2021 03:42, Elliott Mitchell wrote:

Issue is what is the intended use of the memory range allocated to
/hypervisor in the device-tree on ARM?  What do the Xen developers plan
for?  What is expected?

  From docs/misc/arm/device-tree/guest.txt:

- reg: specifies the base physical address and size of a region in
    memory where the grant table should be mapped to, using an
    HYPERVISOR_memory_op hypercall. The memory region is large enough to map
    the whole grant table (it is larger or equal to
    This property is unnecessary when booting Dom0 using ACPI.

Effectively, this is a known space in memory that is unallocated. Not
all the guests will use it if they have a better way to find unallocated

The use of "should" is generally considered strong encouragement to do
so.  A warning $something is lurking here and you may regret it if you
recklessly disobey this without knowning what is going on behind the

I thought a bit more over night. The potential trouble I can think of for a domU is the magic pages are not described in the DT.

I think every other regions should be discoverable from the DT (at least for a domU).

Whereas your language here suggests "can" is a better word since it is
simply a random unused address range.

Was the /hypervisor range intended *strictly* for mapping grant-tables?

It was introduced to tell the OS a place where the grant-table could be
conveniently mapped.

Yet this is strange.  If any $random unused address range is acceptable,
why bother suggesting a particular one?  If this is really purely the
OS's choice, why is Xen bothering to suggest a range at all?

I have added Stefano who may have more historical context than what I wrote in my previous e-mail.

Was it intended for /hypervisor to grow over the
years as hardware got cheaper?
I don't understand this question.

Going to the trouble of suggesting a range points to something going on.
I'm looking for an explanation since strange choices might hint at
something unpleasant lurking below and I should watch where I step.

Might it be better to deprecate the /hypervisor range and have domains
allocate any available address space for foreign mappings?

It may be easy for FreeBSD to find available address space but so far
this has not been the case in Linux (I haven't checked the latest
version though).

To be clear, an OS is free to not use the range provided in /hypervisor
(maybe this is not clear enough in the spec?). This was mostly
introduced to overcome some issues we saw in Linux when Xen on Arm was

Mind if I paraphrase this?

"this is a bring-up hack for Linux which hangs around since we haven't
felt any pressure to fix the underlying Linux issue"

Is that reasonable?

Yes. I have revisited the problem a few times and every time I got stuck because not all the I/O regions where reported to Linux. So Linux would not be able to find a safe unallocated space.

Should the FreeBSD implementation be treating grant tables as distinct
from other foreign mappings?

Both require unallocated addres space to work. IIRC FreeBSD is able to
find unallocated space easily, so I would recommend to use it.

That is supposed to be, but it appears there is presently a bug which has
broken the functionality on ARM.

Do you mind share some details?

As such, as a proper lazy developer if
I can abuse the /hypervisor address range for all foreign mappings, I

Are you aiming to support dom0 now?

My feeling is one of two things should happen with the /hypervisor
address range:

1>  OSes could be encouraged to use it for all foreign mappings.  The
range should be dynamic in some fashion.  There could be a handy way to
allow configuring the amount of address space thus reserved.

In the context of XSA-300 and virtio on Xen on Arm, we discussed about providing a revion for foreign mapping. The main trouble here is figuring out of the size, if you mess it up then you may break all the PV drivers.

If the problem is finding space, then I would like to suggest a different approach (I think I may have discussed it with Andrew). Xen is in maintaining the P2M for the guest and therefore now where are the unallocated space. How about introducing a new hypercall to ask for "unallocated space"?

This would not work for older hypervisor but you could use the RAM instead (as Linux does). This is has drawbacks (e.g. shattering superpage, reducing the amount of memory usuable...), but for 1> you would also need a hack for older Xen.

2>  The range should be declared deprecated.  Everyone should be put on
the same page that this was a quick hack for bringing up Xen/ARM/Linux,
and really it shouldn't have escaped.

How about relaxing the wording instead?

(is treating them the same likely to
induce buggy behavior on x86?)

I will leave this answer to Roger.

This was directed towards *you*.  There is this thing here which looks
kind of odd in a vaguely unpleasant way.  I'm trying to figure out
whether I should embrace it, versus running away.

I am not aware of any potential buggy behavior here. In both arm and x86, the requirement is to find unallocated address space (unless you want to waste RAM as Linux does).


Julien Grall



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.