Re: [Xen-devel] Dealing with non-existent BDF devices in VT-d and in the hardware.

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Tue, Mar 11, 2014 at 05:36:36PM +0000, Andrew Cooper wrote:
> On 11/03/14 17:30, Konrad Rzeszutek Wilk wrote:
> > Hey,
> >
> > I am one of those lucky folks who had purchased a motherboard that has bugs.
> 
> You say this as if you expect someone has managed to find a bugfree
> motherboard :)

One can dream :-)
> 
> >
> > I figured I would post this email as way for a starting point
> > for some discussion on this - and perhaps have a similar as 'pci-phantom'
> > way of instructing the hypervisor what to do with them.
> >
> > The problem I am seeing is that this device:
> >
> > 08:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22A IEEE-1394a-2000 
> > Controller (PHY/Link) [iOHCI-Lynx]
> >
> > Can't be passed in the guest. Or rather it can - but everytime
> > the guest (or domain0) tries to access I see:
> >
> > (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> > (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> > (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:08:00.0] 
> > fault addr 0, iommu reg = ffff82c3ffd53000
> > (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
> > (XEN) print_vtd_entries: iommu ffff83043dca99b0 dev 0000:08:00.0 gmfn 0
> > (XEN)     root_entry = ffff83043dc6b000
> > (XEN)     root_entry[8] = 3326b5001
> > (XEN)     context = ffff8303326b5000
> > (XEN)     context[0] = 0_0
> > (XEN)     ctxt_entry[0] not present
> >
> >
> > Of course the '08:00.0' device does not exist. It is rather this chipset:
> > 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
> >
> > that is buggy and using the wrong BDF when forwarding DMA requests from
> > devices underneath it (like this Firewire chip).
> >
> > The hack I came up with was to create in the Xen code that deals with
> > PCI passthrough a copy of the bridge (so 07:00.0) but with a new
> > BDF: 08:00.0. And link it to the PCI device that I am passing to the
> > guest (so 08:03.0).
> >
> > The end result is that when loading the driver (hack.c) one should
> > see:
> >
> > (XEN) 0000:08:00.0 linked with 08:03.0
> > (XEN) [VT-D]iommu.c:1456: d0:PCI: map 0000:08:00.0
> > (XEN) [VT-D]iommu.c:1476: d0:PCI: map 0000:08:03.0
> > (XEN) PCI add link 0000:08:00.0
> >
> > And when launching a guest with the BDF:
> > pci = ["08:03.0"]
> >
> > the hypervisor will automatically also create an VT-d context for the
> > 08:00.0 device.
> >
> > To use this hack, apply the 
> > 0001-xen-pci-Introduce-a-way-to-deal-with-buggy-hardware-.patch
> > to your hypervisor, compile and install.
> >
> > And also compile the 'hack.c' module. There is an attached 'Makefile'
> > that will do it for you. Make sure you edit it to set the right BDF
> > entries in it.
> >
> > Once done install your new hypervisor, and insmod ./hack.ko and try
> > passing in the device to your guest (or use it normally). The
> > 'DMAR:[DMA Write]' error should go away.
> >
> > This should be generic enough for most devices. It needn't be a bridge
> > that is spewing out these DMAR errors.
> 
> 
> Do you have an lspci -tv for the system?

Yes of course:

-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller
           +-01.0-[01]--+-00.0  Intel Corporation 82576 Gigabit Network 
Connection
           |            \-00.1  Intel Corporation 82576 Gigabit Network 
Connection
           +-01.1-[02]----00.0  LSI Logic / Symbios Logic SAS2008 PCI-Express 
Fusion-MPT SAS-2 [Falcon]
           +-02.0  Intel Corporation Xeon E3-1200 v3 Processor Integrated 
Graphics Controller
           +-03.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD 
Audio Controller
           +-14.0  Intel Corporation 8 Series/C220 Series Chipset Family USB 
xHCI
           +-16.0  Intel Corporation 8 Series/C220 Series Chipset Family MEI 
Controller #1
           +-19.0  Intel Corporation Ethernet Connection I217-LM
           +-1a.0  Intel Corporation 8 Series/C220 Series Chipset Family USB 
EHCI #2
           +-1b.0  Intel Corporation 8 Series/C220 Series Chipset High 
Definition Audio Controller
           +-1c.0-[03]----00.0  Intel Corporation 82574L Gigabit Network 
Connection
           +-1c.1-[04]----00.0  Intel Corporation 82574L Gigabit Network 
Connection
           +-1c.3-[05]----00.0  Intel Corporation I210 Gigabit Network 
Connection
           +-1c.4-[06]--+-00.0  Intel Corporation 82571EB Gigabit Ethernet 
Controller
           |            \-00.1  Intel Corporation 82571EB Gigabit Ethernet 
Controller
           +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree 
Corporation Bt878 Video Capture
           |                               |            +-08.1  Brooktree 
Corporation Bt878 Audio Capture
           |                               |            +-09.0  Brooktree 
Corporation Bt878 Video Capture
           |                               |            +-09.1  Brooktree 
Corporation Bt878 Audio Capture
           |                               |            +-0a.0  Brooktree 
Corporation Bt878 Video Capture
           |                               |            +-0a.1  Brooktree 
Corporation Bt878 Audio Capture
           |                               |            +-0b.0  Brooktree 
Corporation Bt878 Video Capture
           |                               |            \-0b.1  Brooktree 
Corporation Bt878 Audio Capture
           |                               \-03.0  Texas Instruments TSB43AB22A 
IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
           +-1c.6-[0a]----00.0  Renesas Technology Corp. uPD720202 USB 3.0 Host 
Controller
           +-1c.7-[0b]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA 
Controller
           +-1d.0  Intel Corporation 8 Series/C220 Series Chipset Family USB 
EHCI #1
           +-1f.0  Intel Corporation C226 Series Chipset Family Server Advanced 
SKU LPC Controller
           +-1f.2  Intel Corporation 8 Series/C220 Series Chipset Family 6-port 
SATA Controller 1 [AHCI mode]
           +-1f.3  Intel Corporation 8 Series/C220 Series Chipset Family SMBus 
Controller
           \-1f.6  Intel Corporation 8 Series Chipset Family Thermal Management 
Controller

> 
> It is genuinely the case that the bridge doesn't exist, or simply that
> it is not correctly attributed in the DMAR table?

It does not exist. The DMAR looks correct.

(XEN) [VT-D]dmar.c:778: Host address width 39
(XEN) [VT-D]dmar.c:792: found ACPI_DMAR_DRHD:
(XEN) [VT-D]dmar.c:472:   dmaru->address = fed90000
(XEN) [VT-D]iommu.c:1158: drhd->address = fed90000 iommu->reg = ffff82c3ffd54000
(XEN) [VT-D]iommu.c:1160: cap = c0000020660462 ecap = f0101a
(XEN) [VT-D]dmar.c:383:  endpoint: 0000:00:02.0
(XEN) [VT-D]dmar.c:792: found ACPI_DMAR_DRHD:
(XEN) [VT-D]dmar.c:472:   dmaru->address = fed91000
(XEN) [VT-D]iommu.c:1158: drhd->address = fed91000 iommu->reg = ffff82c3ffd53000
(XEN) [VT-D]iommu.c:1160: cap = d2008020660462 ecap = f010da
(XEN) [VT-D]dmar.c:397:  IOAPIC: 0000:f0:1f.0
(XEN) [VT-D]dmar.c:361:  MSI HPET: 0000:f0:0f.0
(XEN) [VT-D]dmar.c:486:   flags: INCLUDE_ALL
(XEN) [VT-D]dmar.c:797: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383:  endpoint: 0000:00:1d.0
(XEN) [VT-D]dmar.c:383:  endpoint: 0000:00:1a.0
(XEN) [VT-D]dmar.c:383:  endpoint: 0000:00:14.0
(XEN) [VT-D]dmar.c:666:   RMRR region: base_addr b7530000 end_address b753cfff
(XEN) [VT-D]dmar.c:797: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:383:  endpoint: 0000:00:02.0
(XEN) [VT-D]dmar.c:666:   RMRR region: base_addr bc000000 end_address be1fffff

As it has the INCLUDE_ALL flag.
> 
> If the latter, it Xen can probably gain some DMAR[$FOO]=$BAR command
> line workarounds similar to the IVRS ones for AMD systems.
> 
> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel