Re: [XEN RFC PATCH v5 0/5] IOMMU subsystem redesign and PV-IOMMU interface

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Tue, Jan 21, 2025 at 04:13:20PM +0000, Teddy Astie wrote:
> This work has been presented at Xen Summit 2024 during the
>   IOMMU paravirtualization and Xen IOMMU subsystem rework
> design session.
> 
> Operating systems may want to have access to a IOMMU in order to do DMA
> protection or implement certain features (e.g VFIO on Linux).
> 
> VFIO support is mandatory for framework such as SPDK, which can be useful to
> implement an alternative storage backend for virtual machines [1].
> 
> In this patch series, we introduce in Xen the ability to manage several
> contexts per domain and provide a new hypercall interface to allow guests
> to manage IOMMU contexts.
> 
> The VT-d driver is updated to support these new features.
> 
> [1] Using SPDK with the Xen hypervisor - FOSDEM 2023
> ---
> Cc: Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>
> 
> PCI Passthrough now work on my side, but things are still feels quite brittle.
> 
> Changed in v2 :
> * fixed Xen crash when dumping IOMMU contexts (using X debug key)
> with DomUs without IOMMU
> * s/dettach/detach/
> * removed some unused includes
> * fix dangling devices in contexts with detach
> 
> Changed in v3 :
> * lock entirely map/unmap in hypercall
> * prevent IOMMU operations on dying contexts (fix race condition)
> * iommu_check_context+iommu_get_context -> iommu_get_context and check for 
> NULL
> 
> Changed in v4 :
> * Part of initialization logic is moved to domain or toolstack (IOMMU_init)
>   + domain/toolstack now decides on "context count" and "pagetable pool size"
>   + for now, all domains are able to initialize PV-IOMMU
> * introduce "dom0-iommu=no-dma" to make default context block all DMA
>   (disables HAP and sync-pt), enforcing usage of PV-IOMMU for DMA
>   Can be used to expose properly "Pre-boot DMA protection"
> * redesigned locking logic for contexts
>   + contexts are accessed using iommu_get_context and released with 
> iommu_put_context
> 
> Changed in v5 :
> * various PCI Passthrough related fixes
>   + rewrote parts of PCI Passthrough logic
>   + various other related bug fixes
> * simplified VT-d DID (for hardware) management by only having one map 
> instead of two
>   (pseudo_domid map was previously used for old quarantine code then recycled 
> for PV-IOMMU
>    in addition to another map also tracing Domain<->VT-d DID, now there is 
> only one
>    map tracking both making things simpler)
> * reworked parts of Xen quarantine logic (needed for PCI Passthrough)
> * added cf_check annotations
> * some changes to PV-IOMMU headers (Alejandro)
> 
> TODO:
> * add stub implementations for bissecting needs and non-ported IOMMU 
> implementations
> * fix some issues with no-dma+PV and grants
> * complete "no-dma" mode (expose to toolstack, add documentation, ...)
> * properly define nested mode and PASID support
> 
> * make new quarantine code more unity region aware (isolate devices with
>   different reserved regions regions using separate 'contexts')
> * find a way to make PV-IOMMU work in DomUs (they don't see machine bdf)
> * there are corner cases with PV-IOMMU and to-domain Xen PCI Passthrough
>   (e.g pci-assignable-remove will reassign to context 0, while the driver
>    expects the device to to be in context X)

Thanks for the updated patches. I have run them through gitlab-ci, and
here are some observations:
- I needed to disable CONFIG_AMD_IOMMU (it fails to build, as expected at this 
point)
- I needed to disable pvshim (it fails to build)
- fails to build with clang: 
https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/8931373789/viewer#L3525
- gcc-ibt build fails: 
https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/8931373785#L1314
- fails to build for ARM (both 32 and 64) and PPC64
- QEMU smoke test panic with PV dom0, looks like it runs on AMD, so it
  may be related to the disabled CONFIG_AMD_IOMMU, but I wouldn't expect
  it to panic on _PV_ dom0 boot...
- PVH dom0 fails to boot (on real hw) with a lot of VT-d faults: 
https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/8931373875
- PCI passthrough (with PV dom0) results in a lot of VT-d faults: 
https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/8931373881

Note this uses only this series, but plain Linux (appears to be 6.1.19).
IIUC if one doesn't try to configure PV-IOMMU specifically (non-default
contexts) it should still work.

BTW Linux says it detected "Xen version 4.19." - shouldn't it report
4.20 already at this point in release cycle?

All results:
https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1637849303

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature