RE: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated space

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Oleksandr,

> -----Original Message-----
> From: Oleksandr <olekstysh@xxxxxxxxx>
> Sent: 2021年8月10日 2:25
> To: Julien Grall <julien@xxxxxxx>
> Cc: Stefano Stabellini <sstabellini@xxxxxxxxxx>; Andrew Cooper
> <andrew.cooper3@xxxxxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx; Oleksandr
> Tyshchenko <oleksandr_tyshchenko@xxxxxxxx>; Daniel De Graaf
> <dgdegra@xxxxxxxxxxxxx>; Daniel P. Smith <dpsmith@xxxxxxxxxxxxxxxxxxxx>;
> Ian Jackson <iwj@xxxxxxxxxxxxxx>; Wei Liu <wl@xxxxxxx>; George Dunlap
> <george.dunlap@xxxxxxxxxx>; Jan Beulich <jbeulich@xxxxxxxx>; Volodymyr
> Babchuk <Volodymyr_Babchuk@xxxxxxxx>; Roger Pau Monné
> <roger.pau@xxxxxxxxxx>; Bertrand Marquis <Bertrand.Marquis@xxxxxxx>; Wei
> Chen <Wei.Chen@xxxxxxx>
> Subject: Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide
> unallocated space
> 
> 
> On 09.08.21 18:42, Julien Grall wrote:
> > Hi Oleksandr,
> 
> 
> Hi Julien.
> 
> 
> Thank you for the input.
> 
> 
> >
> > On 07/08/2021 18:03, Oleksandr wrote:
> >>
> >> On 06.08.21 03:30, Stefano Stabellini wrote:
> >>
> >> Hi Stefano
> >>
> >>> On Wed, 4 Aug 2021, Julien Grall wrote:
> >>>>> +#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /*
> >>>>> 128GB */
> >>>>> +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000)
> >>>>>
> >>>>> While the possible new DT bindings has not been agreed yet, I re-
> used
> >>>>> existing "reg" property under the hypervisor node to pass safe
> >>>>> range as a
> >>>>> second region,
> >>>>> https://elixir.bootlin.com/linux/v5.14-
> rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:
> >>>>>
> >>>> So a single region works for a guest today, but for dom0 we will
> >>>> need multiple
> >>>> regions because it is may be difficult to find enough contiguous
> >>>> space for a
> >>>> single region.
> >>>>
> >>>> That said, as dom0 is mapped 1:1 (including some guest mapping),
> >>>> there is also
> >>>> the question where to allocate the safe region. For grant table, we
> >>>> so far
> >>>> re-use the Xen address space because it is assumed it will space
> >>>> will always
> >>>> be bigger than the grant table.
> >>>>
> >>>> I am not sure yet where we could allocate the safe regions.
> >>>> Stefano, do you
> >>>> have any ideas?
> >>> The safest choice would be the address range corresponding to memory
> >>> (/memory) not already allocated to Dom0.
> >>>
> >>> For instance from my last boot logs:
> >>> (XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
> >>> (XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB)
> >>> (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)
> >>>
> >>> All the other ranges could be given as unallocated space:
> >>>
> >>> - 0x0 - 0x10000000
> >>> - 0x70000000 - 0x78000000
> >>> - 0x8_0000_0000 - 0x8_8000_0000
> >>
> >> Thank you for the ideas.
> >>
> >> If I got the idea correctly, yes, as these ranges represent the real
> >> RAM, so no I/O would be in conflict with them and as the result - no
> >> overlaps would be expected.
> >> But, I wonder, would this work if we have IOMMU enabled for Dom0 and
> >> need to establish 1:1 mapping for the DMA devices to work with grant
> >> mappings...
> >> In arm_iommu_map_page() we call guest_physmap_add_entry() with gfn =
> >> mfn, so the question is could we end up with this new gfn replacing
> >> the valid mapping
> >> (with gfn allocated from the safe region)?
> >
> > Right, when we enable the IOMMU for dom0, Xen will add an extra
> > mapping with GFN == MFN for foreign and grant pages. This is because
> > Linux is not aware that whether a device is protected by an IOMMU.
> > Therefore it is assuming it is not and will use the MFN to configure
> > for DMA transaction.
> >
> > We can't remove the mapping without significant changes in Linux and
> > Xen. I would not mandate them for this work.
> >
> > That said, I think it would be acceptable to have different way to
> > find the region depending on the dom0 configuration. So we could use
> > the RAM not used by dom0 when the IOMMU is turned off.
> 
> OK
> 
> 
> >
> >>> The second best choice would be an hole: an address range not used by
> >>> anybody else (no reg property) and also not even mappable by a bus
> (not
> >>> covered by a ranges property). This is not the best choice because
> >>> there
> >>> can cases where physical resources appear afterwards.
> >
> > Are you saying that the original device-tree doesn't even describe
> > them in any way (i.e. reserved...)?
> >
> >>
> >> Unfortunately, yes.
> >
> > So the decision where the safe region is located will be done by Xen.
> > There is no involvement of the domain (it will discover the region
> > from the DT). Therefore, I don't think we need to think about
> > everything right now as we could adapt this is exact region is not
> > part of the stable ABI.
> >
> > The hotplug is one I would defer because this is not supported (and
> > quite likely not working) in Xen upstream today.
> 
> Sounds reasonable.
> 
> 
> >
> >
> > Now regarding the case where dom0 is using the IOMMU. The assumption
> > is Xen will be able to figure out all the regions used from the
> > firmware table (ACPI or DT).
> >
> > AFAIK, this assumption would be correct for DT. However, for ACPI, I
> > remember we were not able to find all the MMIOs region in Xen (see [1]
> > and [2]). So even this solution would not work for ACPI.
> >
> > If I am not mistaken, we don't support IOMMU with ACPI yet. So we
> > could defer the problem to when this is going to be supported.
> 
> Sounds reasonable.
> 
> 
> To summarize:
> 
> 0. Skip ACPI case for now, implement for DT case
> 
> 1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as
> safe ranges
> 

Does static allocation and direct mapping driver domain can be treated
as this case?

> I would take into the account holes >= 1MB. I am wondering, do we need a
> special alignment here other than a PAGE_SIZE?
> 
> 2. If IOMMU is disabled for Dom0 -> provide RAM which not assigned to
> Dom0 as safe ranges
> 
> We could even provide holes here as well.
> 
> 
> >
> >
> > Cheers,
> >
> > [1] https://marc.info/?l=linux-arm-kernel&m=148469169210500&w=2
> > [2] Xen commit 80f9c316708400cea4417e36337267d3b26591db
> >
> --
> Regards,
> 
> Oleksandr Tyshchenko