Re: [Xen-users] GPU passthrough with xen 4.2.1 on ubuntu 12.10

[Matthew Dean <mcd40@xxxxxxxxx>]

My guess is that they are on different bridges.  I've read that for this board if you use only the first CPU socket then you'll only get access to pcie slots 1 through 4.  Slots 5, 6 and 7 simply won't work at all.  Passing through both cards still does not work but this time I don't get a BSOD, the display simply never turns on.  I can hear the fan speed of the card change however at the precise point where the VNC window stops showing any change so there is something going on here.  I've just pulled out the card in slot 5 and tried to pass through the one remaining card and that has the same result, with the display never turning on.

Matt

On 12/03/2013 11:41, Peter Maloney wrote:

Okay that is enormous output :D

And I'm not sure what to say about it... hopefully someone else reading it can read it better than I can.

Your bridges:

00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)
80:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)

Your GPUs:
...
  +-[0000:80]-+-00.0-[81]--
  |           +-02.0-[83]--+-00.0  Advanced Micro Devices [AMD] nee ATI Tahiti XT [Radeon HD 7970]
  |           |            \-00.1  Advanced Micro Devices [AMD] nee ATI Tahiti XT HDMI Audio [Radeon HD 7970 Series]
  \-[0000:00]-+-00.0  Intel Corporation Xeon E5/Core i7 DMI2
              +-02.0-[03]--+-00.0  Advanced Micro Devices [AMD] nee ATI Tahiti XT [Radeon HD 7970]
              |            \-00.1  Advanced Micro Devices [AMD] nee ATI Tahiti XT HDMI Audio [Radeon HD 7970 Series]

So you would be fine with PCI bridges if that 80 vs 00 is really different bridges and not just same ones seen by different CPUSs.

I can look at many older Xeons and they all show just one set of bridges, and one very new one (newer socket) has the same thing as yours. But I don't use IOMMU on those, so I don't know what effect it has.

I kinda think they are the same bridge, which would be bad. I don't know how to test it... pull out a CPU? ;) find some other detail that has IDs? disable one in the bios (unlikely)? or you could test this by passing through both cards to the same domu... This would be a good test (and matches what the other thread was about... his worked with passing both through)




On 2013-03-12 11:32, Matthew Dean wrote:

As requested,

lspci

##############################################################################################################

00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07)
00:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b (rev 07)
00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)
00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07)
00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07)
00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07)
00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07)
00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06)
00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06)
00:1b.0 Audio device: Intel Corporation C600/X79 series chipset High Definition Audio Controller (rev 06)
00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6)
00:1c.1 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 2 (rev b6)
00:1c.2 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 3 (rev b6)
00:1c.3 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 4 (rev b6)
00:1c.4 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 5 (rev b6)
00:1c.6 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 7 (rev b6)
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06)
00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06)
00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06)
03:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Tahiti XT [Radeon HD 7970]
03:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Tahiti XT HDMI Audio [Radeon HD 7970 Series]
05:00.0 Serial Attached SCSI controller: Intel Corporation C602 chipset 4-Port SATA Storage Control Unit (rev 06)
06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
08:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
09:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
0a:00.0 SATA controller: Marvell Technology Group Ltd. Device 9230 (rev 10)
0b:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire Controller (rev 01)
7f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
7f:08.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 07)
7f:08.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 07)
7f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)
7f:09.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 07)
7f:09.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 07)
7f:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07)
7f:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07)
7f:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 07)
7f:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 07)
7f:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 07)
7f:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 07)
7f:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0c.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 07)
7f:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 07)
7f:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0d.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
7f:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 07)
7f:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 07)
7f:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 07)
7f:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 07)
7f:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 07)
7f:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 07)
7f:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 07)
7f:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 07)
7f:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 07)
7f:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 07)
7f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
7f:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07)
7f:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 07)
7f:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 07)
7f:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07)
7f:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07)
7f:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 07)
7f:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 07)
7f:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07)
7f:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07)
7f:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 07)
7f:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 07)
7f:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07)
7f:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07)
80:00.0 PCI bridge: Intel Corporation Xeon E5/Core i7 DMI2 in PCI Express Mode (rev 07)
80:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07)
80:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07)
80:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07)
80:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07)
80:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07)
80:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07)
83:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Tahiti XT [Radeon HD 7970]
83:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Tahiti XT HDMI Audio [Radeon HD 7970 Series]
ff:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07)
ff:08.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 07)
ff:08.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 07)
ff:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07)
ff:09.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 07)
ff:09.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 07)
ff:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07)
ff:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07)
ff:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 07)
ff:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 07)
ff:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 07)
ff:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 07)
ff:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0c.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 07)
ff:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 07)
ff:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0d.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07)
ff:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 07)
ff:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 07)
ff:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 07)
ff:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 07)
ff:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 07)
ff:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 07)
ff:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 07)
ff:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 07)
ff:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 07)
ff:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 07)
ff:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07)
ff:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07)
ff:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 07)
ff:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 07)
ff:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07)
ff:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07)
ff:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 07)
ff:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 07)
ff:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07)
ff:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07)
ff:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 07)
ff:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 07)
ff:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07)
ff:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07)

##############################################################################################################

lspci -tv

##############################################################################################################

-+-[0000:ff]-+-08.0  Intel Corporation Xeon E5/Core i7 QPI Link 0
 |           +-08.3  Intel Corporation Xeon E5/Core i7 QPI Link Reut 0
 |           +-08.4  Intel Corporation Xeon E5/Core i7 QPI Link Reut 0
 |           +-09.0  Intel Corporation Xeon E5/Core i7 QPI Link 1
 |           +-09.3  Intel Corporation Xeon E5/Core i7 QPI Link Reut 1
 |           +-09.4  Intel Corporation Xeon E5/Core i7 QPI Link Reut 1
 |           +-0a.0  Intel Corporation Xeon E5/Core i7 Power Control Unit 0
 |           +-0a.1  Intel Corporation Xeon E5/Core i7 Power Control Unit 1
 |           +-0a.2  Intel Corporation Xeon E5/Core i7 Power Control Unit 2
 |           +-0a.3  Intel Corporation Xeon E5/Core i7 Power Control Unit 3
 |           +-0b.0  Intel Corporation Xeon E5/Core i7 Interrupt Control Registers
 |           +-0b.3  Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers
 |           +-0c.0  Intel Corporation Xeon E5/Core i7 Unicast Register 0
 |           +-0c.1  Intel Corporation Xeon E5/Core i7 Unicast Register 0
 |           +-0c.2  Intel Corporation Xeon E5/Core i7 Unicast Register 0
 |           +-0c.6  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0
 |           +-0c.7  Intel Corporation Xeon E5/Core i7 System Address Decoder
 |           +-0d.0  Intel Corporation Xeon E5/Core i7 Unicast Register 0
 |           +-0d.1  Intel Corporation Xeon E5/Core i7 Unicast Register 0
 |           +-0d.2  Intel Corporation Xeon E5/Core i7 Unicast Register 0
 |           +-0d.6  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1
 |           +-0e.0  Intel Corporation Xeon E5/Core i7 Processor Home Agent
 |           +-0e.1  Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring
 |           +-0f.0  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers
 |           +-0f.1  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers
 |           +-0f.2  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0
 |           +-0f.3  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1
 |           +-0f.4  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2
 |           +-0f.5  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3
 |           +-0f.6  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4
 |           +-10.0  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0
 |           +-10.1  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1
 |           +-10.2  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0
 |           +-10.3  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1
 |           +-10.4  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2
 |           +-10.5  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3
 |           +-10.6  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2
 |           +-10.7  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3
 |           +-11.0  Intel Corporation Xeon E5/Core i7 DDRIO
 |           +-13.0  Intel Corporation Xeon E5/Core i7 R2PCIe
 |           +-13.1  Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor
 |           +-13.4  Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers
 |           +-13.5  Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor
 |           \-13.6  Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor
 +-[0000:80]-+-00.0-[81]--
 |           +-01.0-[82]--
 |           +-02.0-[83]--+-00.0  Advanced Micro Devices [AMD] nee ATI Tahiti XT [Radeon HD 7970]
 |           |            \-00.1  Advanced Micro Devices [AMD] nee ATI Tahiti XT HDMI Audio [Radeon HD 7970 Series]
 |           +-03.0-[84]--
 |           +-05.0  Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management
 |           +-05.2  Intel Corporation Xeon E5/Core i7 Control Status and Global Errors
 |           \-05.4  Intel Corporation Xeon E5/Core i7 I/O APIC
 +-[0000:7f]-+-08.0  Intel Corporation Xeon E5/Core i7 QPI Link 0
 |           +-08.3  Intel Corporation Xeon E5/Core i7 QPI Link Reut 0
 |           +-08.4  Intel Corporation Xeon E5/Core i7 QPI Link Reut 0
 |           +-09.0  Intel Corporation Xeon E5/Core i7 QPI Link 1
 |           +-09.3  Intel Corporation Xeon E5/Core i7 QPI Link Reut 1
 |           +-09.4  Intel Corporation Xeon E5/Core i7 QPI Link Reut 1
 |           +-0a.0  Intel Corporation Xeon E5/Core i7 Power Control Unit 0
 |           +-0a.1  Intel Corporation Xeon E5/Core i7 Power Control Unit 1
 |           +-0a.2  Intel Corporation Xeon E5/Core i7 Power Control Unit 2
 |           +-0a.3  Intel Corporation Xeon E5/Core i7 Power Control Unit 3
 |           +-0b.0  Intel Corporation Xeon E5/Core i7 Interrupt Control Registers
 |           +-0b.3  Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers
 |           +-0c.0  Intel Corporation Xeon E5/Core i7 Unicast Register 0
 |           +-0c.1  Intel Corporation Xeon E5/Core i7 Unicast Register 0
 |           +-0c.2  Intel Corporation Xeon E5/Core i7 Unicast Register 0
 |           +-0c.6  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0
 |           +-0c.7  Intel Corporation Xeon E5/Core i7 System Address Decoder
 |           +-0d.0  Intel Corporation Xeon E5/Core i7 Unicast Register 0
 |           +-0d.1  Intel Corporation Xeon E5/Core i7 Unicast Register 0
 |           +-0d.2  Intel Corporation Xeon E5/Core i7 Unicast Register 0
 |           +-0d.6  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1
 |           +-0e.0  Intel Corporation Xeon E5/Core i7 Processor Home Agent
 |           +-0e.1  Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring
 |           +-0f.0  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers
 |           +-0f.1  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers
 |           +-0f.2  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0
 |           +-0f.3  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1
 |           +-0f.4  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2
 |           +-0f.5  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3
 |           +-0f.6  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4
 |           +-10.0  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0
 |           +-10.1  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1
 |           +-10.2  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0
 |           +-10.3  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1
 |           +-10.4  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2
 |           +-10.5  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3
 |           +-10.6  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2
 |           +-10.7  Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3
 |           +-11.0  Intel Corporation Xeon E5/Core i7 DDRIO
 |           +-13.0  Intel Corporation Xeon E5/Core i7 R2PCIe
 |           +-13.1  Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor
 |           +-13.4  Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers
 |           +-13.5  Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor
 |           \-13.6  Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor
 \-[0000:00]-+-00.0  Intel Corporation Xeon E5/Core i7 DMI2
             +-01.0-[01]--
             +-01.1-[02]--
             +-02.0-[03]--+-00.0  Advanced Micro Devices [AMD] nee ATI Tahiti XT [Radeon HD 7970]
             |            \-00.1  Advanced Micro Devices [AMD] nee ATI Tahiti XT HDMI Audio [Radeon HD 7970 Series]
             +-03.0-[04]--
             +-05.0  Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management
             +-05.2  Intel Corporation Xeon E5/Core i7 Control Status and Global Errors
             +-05.4  Intel Corporation Xeon E5/Core i7 I/O APIC
             +-11.0-[05]----00.0  Intel Corporation C602 chipset 4-Port SATA Storage Control Unit
             +-16.0  Intel Corporation C600/X79 series chipset MEI Controller #1
             +-16.1  Intel Corporation C600/X79 series chipset MEI Controller #2
             +-1a.0  Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2
             +-1b.0  Intel Corporation C600/X79 series chipset High Definition Audio Controller
             +-1c.0-[06]----00.0  Intel Corporation 82574L Gigabit Network Connection
             +-1c.1-[07]----00.0  Intel Corporation 82574L Gigabit Network Connection
             +-1c.2-[08]----00.0  ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
             +-1c.3-[09]----00.0  ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
             +-1c.4-[0a]----00.0  Marvell Technology Group Ltd. Device 9230
             +-1c.6-[0b]----00.0  VIA Technologies, Inc. VT6315 Series Firewire Controller
             +-1d.0  Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1
             +-1e.0-[0c]--
             +-1f.0  Intel Corporation C600/X79 series chipset LPC Controller
             +-1f.2  Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller
             \-1f.3  Intel Corporation C600/X79 series chipset SMBus Host Controller

##############################################################################################################

I'm going to try passing through both cards and see if that works as well as passing through just the one but with only one card present.  I've had a dig for the thread your talking about but I cannot find it.  If anybody else knows if it still exists somewhere I'd be grateful if you could let me know.

Thanks again for your help.

Matt

On 11/03/2013 18:16, Peter Maloney wrote:

Ah, because you have 2 cpus and 2 cards, can you post output of "lspci -tv" and "lspci"? There was one guy who could pass through one card, but when he physically added the 2nd card in the machine, the same config with the first card wouldn't work any more because the 2 cards were on the same PCI bridge. And he could pass through both and have them work, but not just one of them at a time. (I have since cleaned out my email box, and so I can't find the old thread)

So you see on my system, the "02.0" bridge is separate from "0c.0", so I don't have a problem.

-[0000:00]-+-00.0  Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (external gfx0 port B)
           +-00.2  Advanced Micro Devices [AMD] nee ATI RD990 I/O Memory Management Unit (IOMMU)
           +-02.0-[01]--+-00.0  Advanced Micro Devices [AMD] nee ATI RV370 5B60 [Radeon X300 (PCIE)]
           |            \-00.1  Advanced Micro Devices [AMD] nee ATI RV370 [Radeon X300SE]
           +-09.0-[02]----00.0  Etron Technology, Inc. EJ168 USB 3.0 Host Controller
           +-0a.0-[03]----00.0  Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller
           +-0c.0-[04]--+-00.0  Advanced Micro Devices [AMD] nee ATI Juniper XT [AMD Radeon HD 6000 Series]
           |            \-00.1  Advanced Micro Devices [AMD] nee ATI Juniper HDMI Audio [Radeon HD 5700 Series]


00:02.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port B)
00:09.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port H)
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (external gfx1 port A)
00:0c.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI Device 5a20



On 03/11/2013 06:45 PM, Matthew Dean wrote:

That was a good suggestion, I've just tried passing through a USB controller and that worked fine.  The driver installed correctly and I can use devices attached to the relevant ports.  The problem therefore seems to be to do with gpu passthrough specifically.  I'm just going to describe my system and some settings I have in the bios as this is starting to feel like a system specific problem.  The system is

Asus Z9PE-D8 WS motherboard (chipset is intel C602)
Dual Xeon 2620
64 GB non-ecc memory
Dual AMD radeon HD 7970 in slots 1 & 5 (it's the one in slot 5 I'm trying to pass through)

In the bios I have vt-d enabled.  I also have sub-feature settings for "Coherency Support" and "ATS Support" which are disabled by default.  I've tried all combinations of these enabled and disabled.  If I enable "ATS Support" I do however have to add the option "iommu=no-qinval,no-intremap" to the xen boot otherwise the system hangs as per this thread.  If the problem is to no with these options then I think I have a problem.

Has anybody got any thoughts?

Matt

On 11/03/2013 14:57, Peter Maloney wrote:

Did you try other devices, such as your onboard sound card, your onboard USB controllers, etc.? Maybe your board doesn't do passthrough properly.

Or did you try another PCIe port for the card? Maybe there are bugs where the addresses overlap with some other device, but in another slot it wouldn't happen.


On 2013-03-11 14:48, Matthew Dean wrote:

First of all, thank you for the suggestions.  I've now re-installed windows and I've tried driver versions 12.3, 12.8 and 13.1 all with and without the CCC as I read in various places that it can cause problems.  For 13.1 I never get anywhere close as per my previous messages.  For 12.3 and 12.8 I at least get a BSOD,

#######################################################################

A problem has been detected ...

Attempt to reset the display driver and recover from timeout failed.

If this is ...

Techinal information:

*** STOP: 0x00000116 (0xFFFFFA8002AAF010,0xFFFFF88003D95AE4,0x0000000000000000,0x0000000000000002)

*** atikmpag.sys - Address FFFFF88003D95AE4 base at FFFFF88003D8E000, Datestamp 4f597fca

#######################################################################

Does anybody have any further ideas?

Matt

On 07/03/2013 21:42, John Sherwood wrote:

(and I should probably mention that it has been my experience that that is the expected and correct behavior)

On Thu, Mar 7, 2013 at 1:41 PM, John Sherwood <jrs@xxxxxx> wrote:

Having the windows logo freeze in the VNC window has been my experience with GPU passthrough as well.

On Thu, Mar 7, 2013 at 1:38 PM, Andreas Falck <falck.andreas.lists@xxxxxxxxx> wrote:

Hi,

I believe that the Windows logo freezing in the vnc window is normal behaviour when secondary GPU passthrough is working as it should, since the display should be handed over to the GPU. At least that was the case before when I was passing through a HD6850 card as secondary to win7 using xl. But you should expect output on the GPU, of course.

Have you tried different versions of the catalyst driver? I think that some users have reported problems with the latest one (13-something).

Regards,
Andreas

2013/3/7 Matthew Dean <mcd40@xxxxxxxxx>

Hi all,

I've managed to get a windows 7 x64 hvm to boot and install and I'm now looking to pass through a HD 7970 as a secondary GPU.  in order to prepare the devices for passthrough I've used

xl pci-assignable-add 83:00.0
xl pci-assignable-add 83:00.1

Where I've identified the device pci ids using lspci.  This works fine and the devices appear when I call 'xl pci-assignable-list'. After adding them to the vm config (see below) I was able to get the vm to boot and the devices appeared in device manager.  Installing the ATI drivers however has meant that on further boots the  machine never gets past the windows logo in the VNC window and no output is generated from the ATI card.  It simply freezes there and the only way I can get back into windows is via safe mode, even after uninstalling the ATI software.

Has anybody else experienced similar issues or have any advice as to how to get this to work?  I'm a bit stumped and the wiki implies that this should 'just work'.

Thank you in advance for any help.

Matt

Here is my machine config

###########################################################

builder='hvm'
memory = 2048
name = 'Windows7Test'

vcpus=4
pae=1
acpi=1
apic=1

disk = [ 'file:/xen-guests/img/windows7Test.img,hda,w' ]
pci = [ '83:00.0' , '83:00.1' ]


on_reboot   = 'restart'
on_crash    = 'destroy'

boot='cd'

sdl=0
vnc=1
vncconsole=0
vncpasswd=''
vnclisten='172.24.226.84'

stdvga=0

serial='pty'
usbdevice='tablet'

gfx_passthru=0
pci_power_mgmt=1
xen_platform_pci=1
pci_msitranslate=1
viridian=1
hpet=1

############################################################

And the vm produces the following qemu log in /var/log/xen/

############################################################

domid: 4
-videoram option does not work with cirrus vga device model. Videoram set to 4M.
Strip off blktap sub-type prefix to /xen-guests/img/windows7Test.img (drv 'aio')
Using file /xen-guests/img/windows7Test.img in read-write mode
Watching /local/domain/0/device-model/4/logdirty/cmd
Watching /local/domain/0/device-model/4/command
Watching /local/domain/4/cpu
char device redirected to /dev/pts/3
qemu_map_cache_init nr_buckets = 10000 size 4194304
shared page at pfn feffd
buffered io page at pfn feffb
Guest uuid = e09f5a51-7de1-4d92-9cec-826b9d66aeb6
populating video RAM at ff000000
mapping video RAM from ff000000
Register xen platform.
Done register platform.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
xs_read(/local/domain/0/device-model/4/xen_extended_power_mgmt): read error
Log-dirty: no command yet.
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
vcpu-set: watch node error.
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
xs_read(/local/domain/4/log-throttling): read error
qemu: ignoring not-understood drive `/local/domain/4/log-throttling'
medium change watch on `/local/domain/4/log-throttling' - unknown device, ignored
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 83:00.0 ...
register_real_device: Enable MSI translation via per device option
register_real_device: Enable power management
pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: No such file or directory: 0x83:0x0.0x0
pt_register_regions: IO region registered (size=0x10000000 base_addr=0xe000000c)
pt_register_regions: IO region registered (size=0x00040000 base_addr=0xf0000004)
pt_register_regions: IO region registered (size=0x00000100 base_addr=0x0000f001)
pt_register_regions: Expansion ROM registered (size=0x00020000 base_addr=0xf0040000)
pt_msi_setup: msi mapped with pirq 67
pci_intx: intx=1
register_real_device: Real physical device 83:00.0 registered successfuly!
IRQ type = MSI-INTx
dm-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 83:00.1 ...
register_real_device: Enable MSI translation via per device option
register_real_device: Enable power management
pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: No such file or directory: 0x83:0x0.0x1
pt_register_regions: IO region registered (size=0x00004000 base_addr=0xf0060004)
pt_msi_setup: msi mapped with pirq 66
pci_intx: intx=2
register_real_device: Real physical device 83:00.1 registered successfuly!
IRQ type = MSI-INTx
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456 index=0 first_map=1
cirrus vga map change while on lfb mode
pt_iomem_map: e_phys=f3000000 maddr=f0000000 type=0 len=262144 index=2 first_map=1
pt_iomem_map: e_phys=f3060000 maddr=f0060000 type=0 len=16384 index=0 first_map=1
pt_ioport_map: e_phys=c100 pio_base=f000 len=256 index=4 first_map=1
mapping vram to f0000000 - f0400000
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro state.
pt_pci_read_config: [00:04:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:04:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:04:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:04:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:04:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:04:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:04:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f0000000 type=0 len=262144 index=2 first_map=0
pt_ioport_map: e_phys=ffff pio_base=f000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456 index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f0000000 type=0 len=262144 index=2 first_map=0
pt_ioport_map: e_phys=c100 pio_base=f000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f0060000 type=0 len=16384 index=0 first_map=0
pt_iomem_map: e_phys=f3060000 maddr=f0060000 type=0 len=16384 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=e0000000 type=8 len=268435456 index=0 first_map=0
pt_iomem_map: e_phys=ffffffff maddr=f0000000 type=0 len=262144 index=2 first_map=0
pt_ioport_map: e_phys=ffff pio_base=f000 len=256 index=4 first_map=0
pt_iomem_map: e_phys=e0000000 maddr=e0000000 type=8 len=268435456 index=0 first_map=0
pt_iomem_map: e_phys=f3000000 maddr=f0000000 type=0 len=262144 index=2 first_map=0
pt_ioport_map: e_phys=c100 pio_base=f000 len=256 index=4 first_map=0
pt_msgctrl_reg_write: guest enabling MSI, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_disable: Unmap msi with pirq 67
pt_msgctrl_reg_write: setup msi for dev 20
pt_msi_setup: msi mapped with pirq 67
pt_msi_update: Update msi with pirq 67 gvec b0 gflags 130f
pt_iomem_map: e_phys=ffffffff maddr=f0060000 type=0 len=16384 index=0 first_map=0
pt_iomem_map: e_phys=f3060000 maddr=f0060000 type=0 len=16384 index=0 first_map=0

######################################################################

_____________________

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users