Re: [RFC] xen/x86: allow overlaps with non-RAM regions

[Jason Andryuk <jason.andryuk@xxxxxxx>]

On 2025-04-04 06:28, Roger Pau Monné wrote:
> On Thu, Apr 03, 2025 at 06:01:42PM -0700, Stefano Stabellini wrote:
> > On one Sapphire AMD x86 board, I see this:
> >
> >
> > (XEN) [0000003943ca6ff2]  [00000000f0000000, 00000000f7ffffff] (reserved)
> > (XEN) [00000039460886d9]  [00000000fd000000, 00000000ffffffff] (reserved)
> > [...]
> > (XEN) [    4.612235] 0000:02:00.0: not mapping BAR [fea00, fea03] invalid 
> > position
> >
> >
> > Linux boots fine on this platform but Linux as Dom0 on Xen does not.
> > This is because the pci_check_bar->is_memory_hole check fails due to the
> > MMIO region overlapping with the EFI reserved region.
> That's weird.  (Partially) the reason to not attempt to map such BAR
> is that it should already be mapped, because at dom0 creation time all
> reserved regions are added to the p2m (see arch_iommu_hwdom_init()).
> If that's not the case we should figure out why this reserved region
> is not added to dom0 p2m as part of arch_iommu_hwdom_init().
Victor discovered these regions are type 11 EfiMemoryMappedIO, but they 
get converted to e820 RESERVED.  The BAR points into it.
00000f0000000-00000f7ffffff type=11 attr=800000000000100d
00000fd000000-00000fedfffff type=11 attr=800000000000100d
00000fee00000-00000fee00fff type=11 attr=8000000000000001
00000fee01000-00000ffffffff type=11 attr=800000000000100d

Xenia discovered Linux keeps small regions like this reserved, but lets 
larger ones (>= 256kb) become holes.  See the comment in Linux 
arch/x86/platform/efi/efi.c:efi_remove_e820_mmio() around line 301.
The description of EfiMemoryMappedIO is a little confusing, which is 
probably why its use in unclear.
```
Table 7.5 Memory Type Usage before ExitBootServices()
EfiMemoryMappedIO

Used by system firmware to request that a memory-mapped IO region be 
mapped by the OS to a virtual address so it can be accessed by EFI 
runtime services.
Table 7.6 Memory Type Usage after ExitBootServices()
EfiMemoryMappedIO

This memory is not used by the OS. All system memory-mapped IO 
information should come from ACPI tables.
```

The two after ExitBootServices sentences seem contradictory.  I wonder 
if it should be "Ignore this memory type - All system memory-mapped IO 
information should come from ACPI tables".
> Can you paste the dom0 build output when booted with `iommu=verbose
> dom0=pvh,verbose`?  Does using `dom0=pvh,verbose,pf-fixup` solve the
> boot issue? (and can you paste the output if it does)
pf-fixup did not resolve it.  The vpci pci_check_bar() check is 
independent of pf-fixup from what I can tell.
Regards,
Jason