Re: [Xen-users] VT-d passthrough - NMI crashes system

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Thu, Mar 04, 2010 at 10:47:09PM +0100, Thomas Prause wrote:
> I want to pass some FC adapters to the domU following the steps
> described in the VTdHowTo -> http://wiki.xensource.com/xenwiki/VTdHowTo
> using XEN 3.3.1 (SLES 11)
> 

iirc SLES11 has also Xen 3.4 available.. try updating to it? Also make 
sure you're running the latest dom0 kernel.

--  Pasi

> dom0 seems to be configured properly:
> # xm pci-list-assignable-devices
> 0000:15:00.0
> 0000:24:00.0
> 0000:1f:00.0
> 0000:1a:00.0
> #
> 
> But after starting a domU with one adapter assigned it does not show up
> to lspci in the domU. Even 'xm pci-list-assignable-devices' does not
> show this one adapter any longer.
> 
> After some seconds or minutes the whole system crashes, tries to reboot,
> fails with NMI errors in the BIOS log and finally stops in the BIOS with
> an error that 3 consecutive boot failures have occured and now the
> default settings will be used :-o
> 
> System can then be booted and the log of the dom0 shows:
> 
> Mar  2 12:52:27 manni logger: /etc/xen/scripts/block: Writing
> backend/vbd/1/5632/hotplug-status connected to xenstore.
> Mar  2 12:52:27 manni kernel: vbd vbd-1-5632: 2 creating vbd structure
> Mar  2 12:52:36 manni kernel: vif1.0: no IPv6 routers present
> Mar  2 12:52:41 manni kernel: blkback: ring-ref 8, event-channel 8,
> protocol 1 (x86_64-abi)
> Mar  2 12:52:41 manni kernel: blkback: ring-ref 9, event-channel 9,
> protocol 1 (x86_64-abi)
> Mar  2 12:52:46 manni kernel: Uhhuh. NMI received for unknown reason 00.
> Mar  2 12:52:46 manni kernel: Do you have a strange power saving mode
> enabled?
> Mar  2 12:52:46 manni kernel: Dazed and confused, but trying to continue
> Mar  2 12:53:33 manni sshd[7298]: Accepted keyboard-interactive/pam for
> root from xx.xx.xx.xx port 42896 ssh2
> Mar  3 16:17:26 manni syslog-ng[3708]: syslog-ng starting up;
> version='2.0.9'
> 
> I assumed some sort of IRQ conflict and checked:
> # dmesg | grep GSI
> IOAPIC[0]: apic_id 8, version 0, address 0xfec00000, GSI 0-23
> IOAPIC[1]: apic_id 9, version 0, address 0xfec80000, GSI 24-47
> pci 0000:00:01.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
> pci 0000:00:02.0: PCI INT A -> GSI 29 (level, low) -> IRQ 29
> pci 0000:00:03.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
> pci 0000:00:05.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
> pci 0000:00:07.0: PCI INT A -> GSI 30 (level, low) -> IRQ 30
> pci 0000:00:09.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
> pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> pci 0000:00:1c.4: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> pci 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> ata_piix 0000:00:1f.2: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> ata_piix 0000:00:1f.5: PCI INT C -> GSI 21 (level, low) -> IRQ 21
> megaraid_sas 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> pciback 0000:24:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
> pciback 0000:1f:00.0: PCI INT A -> GSI 30 (level, low) -> IRQ 30
> pciback 0000:1a:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
> pciback 0000:15:00.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
>  sda:<6>ehci_hcd 0000:00:1a.7: PCI INT C -> GSI 19 (level, low) -> IRQ 19
> ehci_hcd 0000:00:1d.7: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 18 (level, low) -> IRQ 18
> uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 18 (level, low) -> IRQ 18
> uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 19 (level, low) -> IRQ 19
> bnx2 0000:0b:00.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
> bnx2 0000:0b:00.1: PCI INT B -> GSI 40 (level, low) -> IRQ 40
> i801_smbus 0000:00:1f.3: PCI INT B -> GSI 22 (level, low) -> IRQ 22
> #
> 
> The IRQs for the FC adapters are used by other devices. These are PCI
> bridges:
> # lspci
> 00:00.0 Host bridge: Intel Corporation X58 I/O Hub to ESI Port (rev 13)
> 00:01.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
> 1 (rev 13)
> 00:02.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
> 2 (rev 13)
> 00:03.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
> 3 (rev 13)
> 00:05.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
> 5 (rev 13)
> 00:07.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
> 7 (rev 13)
> 00:09.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
> 9 (rev 13)
> 
> So I tried to passthrough the corresponding PCI with the FC adapter. But
> this failed as well:
>  # cat /sys/bus/pci/drivers/pciback/slots
> 0000:15:00.0
> 0000:1a:00.0
> 0000:1f:00.0
> 0000:24:00.0
> 0000:00:03.0
>  # xm create /etc/xen/vm/sid-sles11 pci=00:00:03.0 pci=00:15:00.0
> Using config file "/etc/xen/vm/sid-sles11".
> Error: pci: PCI Backend does not own device 0000:00:03.0
> See the pciback.hide kernel command-line parameter or
> bind your slot/device to the PCI backend using sysfs
>  #
> 
> Am I on the wrong track with the PCI bridge? Or is this some sort of
> hardware issue (IBM X3650 M2 with latest firmware)?
> 
> Any thoughts are highly appreciated.
> Thanks.
> 
> Thomas
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users