Re: xen/arm: Virtio-PCI for dom0less on ARM

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Wed, 21 May 2025, Julien Grall wrote:
> > For static dom0less flows, we can do the same as for xl flows but we have
> > the
> > additional problem of domU's PCI bus enumeration racing with QEMU.
> > On x86, when domU's access a memory range that has not yet got IOREQ's
> > connected to it, the accesses succeeds with reads returning 0xFFFFFFFF and
> > writes ignored. This makes it easy for guests to wait for IOREQ devices to
> > pop up since guests will find an empty bus and can initiate a rescan later
> > when QEMU attaches. On ARM, we trap on these accesses.
> > > If we on ARM add support for MMIO background regions with a default 
> read value,
> > i.e mmio handlers that have lower priority than IOREQs and that are
> > read-const + writes-ignored, we could support the same flow on ARM.
> > This may be generally useful for other devices as well (e.g virtio-mmio or
> > something else). We could also use this to defer PCI enumeration.
> 
> Regardless what I wrote above, if we are going down the route of returning
> 0xFFFFFFFF, I would just do it for every IOs rather than the one specify in
> the device-tree. This could still behind a per-domain option, but it would at
> least make simpler to setup the system (AFAIU, in your current proposal, we
> would need to specify the range in multiple places).
 
This seems like a good idea.


> > So the next versions of this series I was thinking to remove the PCI
> > specifics
> > and instead add FDT bindings to ARM dom0less enabling setup of user
> > configurable (by address-range and read-value) background mmio regions.
> > Xen would then support option #2 without any PCI specifics added.
> > 
> > Thoughts?
> > 
> > Cheers,
> > Edgar
> > 
> > # References to spec
> > 
> > PCI express base specification:
> > 7.5.1.1.1 Vendor ID Register (Offset 00h)
> > The Vendor ID register is HwInit and the value in this register identifies
> > the manufacturer of the Function. In keeping with
> > PCI-SIG procedures, valid vendor identifiers must be allocated by the
> > PCI-SIG to ensure uniqueness. Each vendor must
> > have at least one Vendor ID. It is recommended that software read the Vendor
> > ID register to determine if a Function is
> > present, where a value of FFFFh indicates that no Function is present.
> > 
> > PCI Firmware Specification:
> > 3.5 Device State at Firmware/Operating System Handoff
> > Page 34:
> > The operating system is required to configure PCI subsystems:
> >  During hotplug
> >  For devices that take too long to come out of reset
> >  PCI-to-PCI bridges that are at levels below what firmware is designed to
> > configure
> > 
> > Page 36:
> > Note: The operating system does not have to walk all buses during boot. The
> > kernel can
> > automatically configure devices on request; i.e., an event can cause a scan
> > of I/O on demand.
> 
> I am not sure why you quote this. To me it reads like this is up to the OS to
> decide when to access the PCI bus. As we don't control the OS, this doesn't
> seem a behavior Xen can rely on.
> 
> > 
> > FPGA's can be programmed at runtime and appear on the ECAM bus silently.
> > An PCI rescan needs to be triggered for the OS to discover the device:
> > Intel FPGAs:
> > https://www.intel.com/content/www/us/en/docs/programmable/683190/1-3-1/how-to-rescan-bus-and-re-enable-aer.html
> 
> To clarify, you are saying the ECAM bus may be completely empty (e.g.
> everything is reading as ~0) and some part of the ECAM will return a non ~0
> value when the FPGA run.

>From what I found from my searches on the topic, the PCIe spec says when
a device is not present or not initialized, the system typically
receives a value of 0xFFFFFFFF on read. Once a device becomes active and
responds to configuration accesses, subsequent reads will return valid
data. The specifications do not require interrupts or hot-plug events
for device detection; polling the configuration space is a compliant
method.

Which actually points to your suggestion above to do this for every IOs.