Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated space

[Julien Grall <julien@xxxxxxx>]

On 09/08/2021 19:24, Oleksandr wrote:
> On 09.08.21 18:42, Julien Grall wrote:
> > Hi Oleksandr,
>
> Hi Julien.
>
>
> Thank you for the input.
>
>
> > On 07/08/2021 18:03, Oleksandr wrote:
> > > On 06.08.21 03:30, Stefano Stabellini wrote:
> > >
> > > Hi Stefano
> > >
> > > > On Wed, 4 Aug 2021, Julien Grall wrote:
> > > > > > +#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /* 
> > > > > > 128GB */
> > > > > > +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000)
> > > > > >
> > > > > > While the possible new DT bindings has not been agreed yet, I re-used
> > > > > > existing "reg" property under the hypervisor node to pass safe 
> > > > > > range as a
> > > > > > second region,
> > > > > > https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10: 
> > > > > So a single region works for a guest today, but for dom0 we will 
> > > > > need multiple
> > > > > regions because it is may be difficult to find enough contiguous 
> > > > > space for a
> > > > > single region.
> > > > >
> > > > > That said, as dom0 is mapped 1:1 (including some guest mapping), 
> > > > > there is also
> > > > > the question where to allocate the safe region. For grant table, we 
> > > > > so far
> > > > > re-use the Xen address space because it is assumed it will space 
> > > > > will always
> > > > > be bigger than the grant table.
> > > > >
> > > > > I am not sure yet where we could allocate the safe regions. 
> > > > > Stefano, do you
> > > > > have any ideas?
> > > > The safest choice would be the address range corresponding to memory
> > > > (/memory) not already allocated to Dom0.
> > > >
> > > > For instance from my last boot logs:
> > > > (XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
> > > > (XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB)
> > > > (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)
> > > >
> > > > All the other ranges could be given as unallocated space:
> > > >
> > > > - 0x0 - 0x10000000
> > > > - 0x70000000 - 0x78000000
> > > > - 0x8_0000_0000 - 0x8_8000_0000
> > > Thank you for the ideas.
> > >
> > > If I got the idea correctly, yes, as these ranges represent the real 
> > > RAM, so no I/O would be in conflict with them and as the result - no 
> > > overlaps would be expected.
> > > But, I wonder, would this work if we have IOMMU enabled for Dom0 and 
> > > need to establish 1:1 mapping for the DMA devices to work with grant 
> > > mappings...
> > > In arm_iommu_map_page() we call guest_physmap_add_entry() with gfn = 
> > > mfn, so the question is could we end up with this new gfn replacing 
> > > the valid mapping
> > > (with gfn allocated from the safe region)?
> > Right, when we enable the IOMMU for dom0, Xen will add an extra 
> > mapping with GFN == MFN for foreign and grant pages. This is because 
> > Linux is not aware that whether a device is protected by an IOMMU. 
> > Therefore it is assuming it is not and will use the MFN to configure 
> > for DMA transaction.
> > We can't remove the mapping without significant changes in Linux and 
> > Xen. I would not mandate them for this work.
> > That said, I think it would be acceptable to have different way to 
> > find the region depending on the dom0 configuration. So we could use 
> > the RAM not used by dom0 when the IOMMU is turned off.
> OK
>
>
> > > > The second best choice would be an hole: an address range not used by
> > > > anybody else (no reg property) and also not even mappable by a bus (not
> > > > covered by a ranges property). This is not the best choice because 
> > > > there
> > > > can cases where physical resources appear afterwards.
> > Are you saying that the original device-tree doesn't even describe 
> > them in any way (i.e. reserved...)?
> > > Unfortunately, yes.
> > So the decision where the safe region is located will be done by Xen. 
> > There is no involvement of the domain (it will discover the region 
> > from the DT). Therefore, I don't think we need to think about 
> > everything right now as we could adapt this is exact region is not 
> > part of the stable ABI.
> > The hotplug is one I would defer because this is not supported (and 
> > quite likely not working) in Xen upstream today.
> Sounds reasonable.
>
>
> > Now regarding the case where dom0 is using the IOMMU. The assumption 
> > is Xen will be able to figure out all the regions used from the 
> > firmware table (ACPI or DT).
> > AFAIK, this assumption would be correct for DT. However, for ACPI, I 
> > remember we were not able to find all the MMIOs region in Xen (see [1] 
> > and [2]). So even this solution would not work for ACPI.
> > If I am not mistaken, we don't support IOMMU with ACPI yet. So we 
> > could defer the problem to when this is going to be supported.
> Sounds reasonable.
>
>
> To summarize:
>
> 0. Skip ACPI case for now, implement for DT case
Just to be clear, I suggested to skip it when the IOMMU is enabled with 
ACPI. We should still support the case without IOMMU. The implementation 
would be the same as 2.
> 1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as 
> safe ranges
> I would take into the account holes >= 1MB.
May I ask why 1MB?

> I am wondering, do we need a 
> special alignment here other than a PAGE_SIZE?
It needs to be 64KB aligned so a guest using 64KB pages can use it.

> 2. If IOMMU is disabled for Dom0 -> provide RAM which not assigned to 
> Dom0 as safe ranges
> We could even provide holes here as well.
I would rather not. We likely need hack for the hotplug case. So I want 
to keep them contained to IOMMU unless there is a strong reason to do it.
Cheers,

--
Julien Grall