Re: [RFC] xen/x86: allow overlaps with non-RAM regions

[Jason Andryuk <jason.andryuk@xxxxxxx>]

On 2025-04-11 03:31, Roger Pau Monné wrote:
> Thanks Jason for getting back, I'm intrigued by this issue :).
>
> On Thu, Apr 10, 2025 at 04:55:54PM -0400, Jason Andryuk wrote:
> > On 2025-04-04 06:28, Roger Pau Monné wrote:
> > > On Thu, Apr 03, 2025 at 06:01:42PM -0700, Stefano Stabellini wrote:
> > > > On one Sapphire AMD x86 board, I see this:
> > > >
> > > >
> > > > (XEN) [0000003943ca6ff2]  [00000000f0000000, 00000000f7ffffff] (reserved)
> > > > (XEN) [00000039460886d9]  [00000000fd000000, 00000000ffffffff] (reserved)
> > > > [...]
> > > > (XEN) [    4.612235] 0000:02:00.0: not mapping BAR [fea00, fea03] invalid 
> > > > position
> > > >
> > > >
> > > > Linux boots fine on this platform but Linux as Dom0 on Xen does not.
> > > > This is because the pci_check_bar->is_memory_hole check fails due to the
> > > > MMIO region overlapping with the EFI reserved region.
> > > That's weird.  (Partially) the reason to not attempt to map such BAR
> > > is that it should already be mapped, because at dom0 creation time all
> > > reserved regions are added to the p2m (see arch_iommu_hwdom_init()).
> > > If that's not the case we should figure out why this reserved region
> > > is not added to dom0 p2m as part of arch_iommu_hwdom_init().
> > Victor discovered these regions are type 11 EfiMemoryMappedIO, but they get
> > converted to e820 RESERVED.  The BAR points into it.
> >
> > 00000f0000000-00000f7ffffff type=11 attr=800000000000100d
> > 00000fd000000-00000fedfffff type=11 attr=800000000000100d
> > 00000fee00000-00000fee00fff type=11 attr=8000000000000001
> > 00000fee01000-00000ffffffff type=11 attr=800000000000100d
> >
> > Xenia discovered Linux keeps small regions like this reserved, but lets
> > larger ones (>= 256kb) become holes.  See the comment in Linux
> > arch/x86/platform/efi/efi.c:efi_remove_e820_mmio() around line 301.
> Right, but whatever Linux decides to do with the reserved regions
> won't affect how Xen maps them into the p2m.  Anything that's reserved
> in the e820 should end up identity mapped in the p2m for PVH dom0,
> unless it's being exclusively used by Xen (see
> dom0_setup_permissions() use of iomem_deny_access() to deny dom0
> access to some MMIO regions).
Oh, my point was more that Baremetal Linux won't have reserved ranges in 
these regions, so there would not be any BAR conflicts.  Though I'm not 
sure if it checks.
If Xen removed them from the memory map, then pci_check_bar() -> 
is_memory_hole() would pass.
> > The description of EfiMemoryMappedIO is a little confusing, which is
> > probably why its use in unclear.
> >
> > ```
> > Table 7.5 Memory Type Usage before ExitBootServices()
> > EfiMemoryMappedIO
> >
> > Used by system firmware to request that a memory-mapped IO region be mapped
> > by the OS to a virtual address so it can be accessed by EFI runtime
> > services.
> >
> > Table 7.6 Memory Type Usage after ExitBootServices()
> > EfiMemoryMappedIO
> >
> > This memory is not used by the OS. All system memory-mapped IO information
> > should come from ACPI tables.
> > ```
> >
> > The two after ExitBootServices sentences seem contradictory.  I wonder if it
> > should be "Ignore this memory type - All system memory-mapped IO information
> > should come from ACPI tables".
> Not very helpful indeed.  The description in "before
> ExitBootServices()" seems more sensible to me: if the MMIO region is
> used by runtime services Xen should ensure it's always mapped in the
> dom0 p2m (which Xen should in principle already do).
>
> > > Can you paste the dom0 build output when booted with `iommu=verbose
> > > dom0=pvh,verbose`?
> Would it be possible to see the output of a debug=y build when booted
> with `iommu=verbose dom0=pvh,verbose` (with or without pf-fixup,
> either is fine).
>
> I'm specially interested in the ranggeset contents printed after "d0:
> identity mappings for IOMMU:", but if possible would like to see the
> full boot log (including Linux dom0).
Attached.

Regards,
Jason

Attachment: xen-efi-mmio-staging-fixup.log
Description: Text Data