Re: [PATCH v6] xen/pt: reserve PCI slot 2 for Intel igd-passthru

[Chuck Zmudzinski <brchuckz@xxxxxxx>]

On 1/2/2023 12:46 PM, Michael S. Tsirkin wrote:
> On Sun, Jan 01, 2023 at 06:52:03PM -0500, Chuck Zmudzinski wrote:
> > Intel specifies that the Intel IGD must occupy slot 2 on the PCI bus,
> > as noted in docs/igd-assign.txt in the Qemu source code.
> > 
> > Currently, when the xl toolstack is used to configure a Xen HVM guest with
> > Intel IGD passthrough to the guest with the Qemu upstream device model,
> > a Qemu emulated PCI device will occupy slot 2 and the Intel IGD will occupy
> > a different slot. This problem often prevents the guest from booting.
> > 
> > The only available workaround is not good: Configure Xen HVM guests to use
> > the old and no longer maintained Qemu traditional device model available
> > from xenbits.xen.org which does reserve slot 2 for the Intel IGD.
> > 
> > To implement this feature in the Qemu upstream device model for Xen HVM
> > guests, introduce the following new functions, types, and macros:
> > 
> > * XEN_PT_DEVICE_CLASS declaration, based on the existing TYPE_XEN_PT_DEVICE
> > * XEN_PT_DEVICE_GET_CLASS macro helper function for XEN_PT_DEVICE_CLASS
> > * typedef XenPTQdevRealize function pointer
> > * XEN_PCI_IGD_SLOT_MASK, the value of slot_reserved_mask to reserve slot 2
> > * xen_igd_reserve_slot and xen_igd_clear_slot functions
> > 
> > The new xen_igd_reserve_slot function uses the existing slot_reserved_mask
> > member of PCIBus to reserve PCI slot 2 for Xen HVM guests configured using
> > the xl toolstack with the gfx_passthru option enabled, which sets the
> > igd-passthru=on option to Qemu for the Xen HVM machine type.
> > 
> > The new xen_igd_reserve_slot function also needs to be implemented in
> > hw/xen/xen_pt_stub.c to prevent FTBFS during the link stage for the case
> > when Qemu is configured with --enable-xen and --disable-xen-pci-passthrough,
> > in which case it does nothing.
> > 
> > The new xen_igd_clear_slot function overrides qdev->realize of the parent
> > PCI device class to enable the Intel IGD to occupy slot 2 on the PCI bus
> > since slot 2 was reserved by xen_igd_reserve_slot when the PCI bus was
> > created in hw/i386/pc_piix.c for the case when igd-passthru=on.
> > 
> > Move the call to xen_host_pci_device_get, and the associated error
> > handling, from xen_pt_realize to the new xen_igd_clear_slot function to
> > initialize the device class and vendor values which enables the checks for
> > the Intel IGD to succeed. The verification that the host device is an
> > Intel IGD to be passed through is done by checking the domain, bus, slot,
> > and function values as well as by checking that gfx_passthru is enabled,
> > the device class is VGA, and the device vendor in Intel.
> > 
> > Signed-off-by: Chuck Zmudzinski <brchuckz@xxxxxxx>
>
> I'm not sure why is the issue xen specific. Can you explain?
> Doesn't it affect kvm too?

Yes, it does, and of course this only applies to using the igd in a
guest in legacy mode as described in docs/igd-assign.txt.

Searching the web, I found this successful report of legacy
igd passthrough using kvm:

https://www.reddit.com/r/VFIO/comments/i9dbyp/this_is_how_i_managed_to_passthrough_my_igd/

That user posted the virtual machine xml on pastebin:

https://pastebin.com/vYf3a1gz

For reference, details of my configuration of legacy igd passthrough on
xen are available on the xenproject wiki:

https://wiki.xenproject.org/wiki/Xen_VGA_Passthrough_Tested_Adapters#Intel_display_adapters

As I expected, with kvm, it is possible to specify the slot number
of every pci device in the guest (as well as domain, bus, and function)
in the xml configuration, but this is not easy to do with xen's
xenlight (libxl) toolstack. That is why this patch is specific to xen.

To further explain this:

On xen, the xl.cfg guest configuration file does not allow the
administrator to specify the slot number of the xen platform
pci device, or of the emulated network device and disk controller,
and one of these devices will grab slot 2 without this patch to
qemu, making it impossible to have the passed through igd at
slot 2 on xen without patching qemu.

Another way to solve this problem on xen is to extend libxl so the
administrator can specify the slot number of the emulated qemu
pci devices, or possibly by using the xl.cfg
device_model_args_hvm=[ "ARG", "ARG", ...] settings which might
allow the administrator to control the slot number of the emulated
qemu pci devices, and I tried that without success.

This solution of patching qemu to reserve slot 2 for the intel igd when
the qemu igd-passthru=on option for the xenfv machine type is set is
a more simple solution to the problem on xen than trying to manually
set all the slot numbers using the device_model_args_hvm option in
xl.cfg.

I think kvm users who desire this feature of legacy igd passthrough
would benefit from something like the qemu igd-passthru=on option
which, as far as I know, only applies to the xenfv machine type that is
enabled with the gfx_passthru setting in the xl.cfg configuration file.
Such an option for kvm could allow for qemu to take care of all the
details of configuring the vm correctly for igd legacy passthrough
on kvm instead of requiring the administrator to manually specify
all the settings correctly in the xml configuration file.

I think making igd legacy passthrough easier to configure on kvm
would be a useful patch, but it is beyond the scope of what this patch
is trying to accomplish.

Chuck