Re: [Xen-devel] bug suspcion and proposed modification when xen-pciback failed to map an irq (-19) to a domU

[Jan Beulich <jbeulich@xxxxxxxx>]

On 29.11.2019 09:58,  DOZ, MARC (ext)  wrote:
> Hello,
> 
> I would like to paravirtualize several instances of a minimal Linux solution 
> with xen. Guest OS have a PCI passthrough access on  the ethernet interfaces.
> I use a pvops kernel 4.4.122 , xen 4.11.2 and an x86 platform. My hardware 
> platform doesn't support IOMMU but I doesn't need hard segmentation between 
> OS with this platform.
> 
> I got a bug when the Ethernet driver configures the MSI for a domU from a 
> domU.
> pciback 0000:00:14.3: xen map irq failed -19 for 1 domain
> 
> Please find the bug description and maybe a solution :
> 
> 1) dom1 is configured to use 0000:00:14.3
> in file /etc/xen/dom1 :
>  pci=['00:14.3,permissive=1,seize=1,msitranslate=1']
> and kernel bootparam include "iommu=soft"
> 
> 2) xen-pciback is loaded in dom0 :
>> modprobe xen-pciback passthrough=1 hide='(0000:00:14.3)'
>> xl pci-assignable-list
>  0000:00:14.3
> 
> (xen-pciback is not loaded in dom1)
> 
> 3) Xen debug terminal print that all the PCI devices are linked to dom0 
> before and after xen-pciback probing
>  ...
>  (XEN) 0000:00:14.3 - dom 0   - node -1  - MSIs < >
>  ...
> 
> 4) no error are detected when I start dom1

I think if you retry with current master or staging, you'll find this
changed.

>> xl create /etc/xen/dom1
> or with
>> xl pci-attach dom1 00:14.3
> 
> 5) For each Ethernet device I see the following call to the hypervisor
>  - xc_domain_iomem_permission
>  - xc_domain_ioport_permission
>  - xc_domain_iomem_permission
>  - xc_physdev_map_pirq
>  - xc_domain_irq_permission
>  - xc_assign_device
> 
> none of them update the pdev->domain or the pdev->domain->domain_id of the 
> pdev element linked with the bfd 0000:00:14.3 (devfn 163)
> 
> 6) Xen debug terminal prints again that all the  PCI devices are linked to 
> dom0
>  ...
>  (XEN) 0000:00:14.3 - dom 0   - node -1  - MSIs < >
>  ...
> 
> I guess that this is normal because Dom0 will allow dom1 to configure the 
> device (IRQ & DMA) thanks to xen-pciback. Could you confirm me this assertion 
> ?
> 
> 7) When the Ethernet driver (igb) of dom1 configure MSI. xen failed to map 
> MSI irqs because map_domain_pirq call pci_get_pdev_by_domain with domain_id 1 
> and the device is linked with domain_id 0
>> modprobe igb
>  igb: Copyright (c) 2007-2014 Intel Corporation.
>  igb 0000:00:14.3: enabling device (0000 -> 0002)
>  igb 0000:00:14.3: Xen PCI mapped GSI23 to IRQ25
>  en_initdom_setup_msi_irqs:xen map irq map_irq.domid 3, map_irq.type 3  
> map_irq.index -1 , map_irq.pirq -1
>  xen_initdom_setup_msi_irqs:xen map irq map_irq.bus 0, map_irq.devfn 163
>  pciback 0000:00:14.3:   PCI_CAP_ID_MSIX
>          pci_seg_supported
> 
> There,HYPERVISOR_physdev_op and  PHYSDEVOP_map_pirq  call map_domain_pirq :
>  (XEN)          map_domain_pirq--> call pci_get_pdev_by_domain
>  (XEN)          pci_get_pdev_by_domain-->
>  (XEN)                          domain->domain_id 1
>  (XEN)                          seg 0 bus 0 devfn 163
>  ...
> 
> Unfortunately the device is already mapped to domain 0 and 
> pci_get_pdev_by_domain don't found the device ... So map_domain_pirq & 
> HYPERVISOR_physdev_op return -19 :
>  pciback 0000:00:14.3: xen map irq failed -19 for 2 domain
> 
> I don't find any hypervisor syscall able to assign my device to a domU.
> xc_assign_device is cancelled because iommu is disabled.
> libxl_pci.c:1137:do_pci_add: Domain 3:do_pci_add --> call xc_assign_device
>  -> hypervisor call arch_do_domctl--> iommu_do_domctl--> CANCELLED

And intentionally so.

> To solve my issue , I just need to replace in map_domain_pirq in 
> xen/arch/x86/irq.c
>  pdev = pci_get_pdev_by_domain(d, msi->seg, msi->bus, msi->devfn);
> by
>  pdev = pci_get_pdev(msi->seg, msi->bus, msi->devfn);
> 
> With this "fix" in Xen, dom1 can use the ethernet device and its MSI that I 
> see in /proc/interrupt

Except that this is not a "fix", but the introduction of a security
vulnerability (permitting interrupt setup on un-owned devices). See
XSA-237, which actually changed it in the opposite direction of
what you're proposing.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel