[Xen-users] Xen PCI passthrough problem

["Phil Quesinberry" <philq@xxxxxxxxxxxxxxxxxxxxxxx>]

Hi everyone,
 
I'm trying to get PCI passthrough to work under XCP and am having one Hell
of a time with it.  I've finally gotten the device to appear when doing an
lspci in the PV DomU but the VM takes forever to boot and keeps giving the
following error, which I can see in the XenCenter console for the VM or by
doing a dmesg in the VM:
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000

Pciback appears to be seizing the device in Dom0 although it still appears
when doing an lspci... I'm not sure if that's the expected behavior or not
as I'm still a little wet behind the ears with this stuff.  Any ideas on
what might be going on here?

System is a Dell PowerEdge T610 server running Xen Cloud Platform 1.5 beta.
I'm trying to allow the VM to access a tape drive.

OS boot parameters from XenCenter:  graphical utf8 iommu=soft    (I think
this is what I should be using)
Dom0 Kernel version:  2.6.32.12-0.7.1.xs1.4.90.530.170661xen
DomU Kernel version:  2.6.18-308.11.1.el5.centos.plusxen     (CentOS 5.8)

extlinux.conf (Dom0):
# location mbr
serial 0 115200
default xe
prompt 1
timeout 50

label xe
  # XCP
  kernel mboot.c32
  append /boot/xen.gz dom0_mem=752M lowmem_emergency_pool=1M
crashkernel=64M@32M console= vga=mode-0x0311 --- /boot/vmlinuz-2.6-xen
root=LABEL=root-vadwopgr ro xencons=hvc console=hvc0 console=tty0 quiet
vga=785 splash xen-pciback.hide=(07:00.0) --- /boot/initrd-2.6-xen.img

-----

initrd-2.6-xen.img -> initrd-2.6.32.12-0.7.1.xs1.4.90.530.170661xen.img
(symbolic link)

Some relevant (IMHO) initrd config excerpts related to PCI settings:
[root@localhost boot]# grep -ir pci
config-2.6.32.12-0.7.1.xs1.4.90.530.170661xen
CONFIG_PCI_QUIRKS=y
CONFIG_PCI=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_PCI_GUESTDEV=y
CONFIG_PCI_IOMULTI=y
CONFIG_PCI_RESERVE=y
CONFIG_PCI_STUB=m
CONFIG_PCI_IOV=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_XEN_PCIDEV_BACKEND=y
CONFIG_XEN_PCIDEV_BACKEND_VPCI=y
# CONFIG_XEN_PCIDEV_BACKEND_PASS is not set
# CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set
# CONFIG_XEN_PCIDEV_BACKEND_CONTROLLER is not set
# CONFIG_XEN_PCIDEV_BE_DEBUG is not set

-----

Relevant Dom0 dmesg excerpts:
[root@localhost boot]# dmesg|grep -i pci
Allocating PCI resources starting at d0000000 (gap: d0000000:10000000)
Kernel command line: root=LABEL=root-vadwopgr ro xencons=hvc console=hvc0
console=tty0 quiet vga=785 splash xen-pciback.hide=(07:00.0)
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
mptsas 0000:07:00.0: PCI INT A -> GSI 38 (level, low) -> IRQ 38
mptbase: ioc0: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total memory =
777404 kB
mptsas 0000:07:00.0: PCI INT A disabled
pciback 0000:07:00.0: seizing device
pciback 0000:07:00.0: PCI INT A -> GSI 38 (level, low) -> IRQ 38
pciback 0000:07:00.0: PCI INT A disabled
pciback: vpci: 0000:07:00.0: assign to virtual slot 0
pciback 0000:07:00.0: enabling device (0000 -> 0003)
pciback 0000:07:00.0: PCI INT A -> GSI 38 (level, low) -> IRQ 38
pciback 0000:07:00.0: setting latency timer to 64

-----

Relevant xe host-dmesg excerpts from Dom0:
[root@localhost boot]# xe host-dmesg
(XEN) Xen version 4.1.1 (root@xxxxxxxxxxxxxxxx) (gcc version 4.1.2 20080704
(Red Hat 4.1.2-46)) Mon Feb  6 19:01:42 EST 2012
(XEN) Latest ChangeSet: unavailable
(XEN) Bootloader: SYSLINUX 4.02 2010-07-21
(XEN) Command line: dom0_mem=752M lowmem_emergency_pool=1M
crashkernel=64M@32M console= vga=mode-0x0311
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed

-----

DomU dmesg excerpts (lots of errors, and I've trimmed this quite a bit):
[root@Server1 ~]# dmesg
Bootdata ok (command line is ro root=/dev/VolGroup00/LogVol00 console=xvc0
rhgb quiet  graphical utf8 iommu=soft)
Linux version 2.6.18-308.11.1.el5.centos.plusxen
(mockbuild@xxxxxxxxxxxxxxxxxxxx) (gcc version 4.1.2 20080704 (Red Hat
4.1.2-52)) #1 SMP Tue Jul 10 17:31:22 EDT 2012
No mptable found.
Kernel command line: ro root=/dev/VolGroup00/LogVol00 console=xvc0 rhgb
quiet  graphical utf8 iommu=soft
Xen reported: 2128.074 MHz processor.
PCI: setting up Xen PCI frontend stub
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI: disabled
PCI: System does not support PCI
PCI: System does not support PCI
pcifront pci-0: Installing PCI frontend
pcifront pci-0: Creating PCI Frontend Bus 0000:00
ACPI Error (tbxfroot-0512): Could not map memory at 0000040E for length 2
[20060707]
ACPI Exception (tbxfroot-0400): AE_NO_MEMORY, RSDP structure not found -
Flags=8 [20060707]
ACPI: System description tables not found
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
rtc: IRQ 8 is not free.
Xen virtual console successfully installed as xvc0
i8042.c: No controller found.
XENBUS: Device with no driver: device/vbd/51728
XENBUS: Device with no driver: device/vbd/51760
XENBUS: Device with no driver: device/vbd/51712
XENBUS: Device with no driver: device/vif/0
XENBUS: Device with no driver: device/vif/1
XENBUS: Device with no driver: device/vif/2
XENBUS: Device with no driver: device/vif/3
ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
Registering block device major 202
 xvdb: xvdb1
 xvdd: [mac] xvdd1 xvdd2
Changing capacity of (202, 16) to 524288000 sectors
 xvda: xvda1 xvda2
Changing capacity of (202, 48) to 44924 sectors
Changing capacity of (202, 0) to 83886080 sectors
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.6-ioctl (2011-02-18) initialised:
dm-devel@xxxxxxxxxx
device-mapper: dm-raid45: initialized v0.2594l
end_request: I/O error, dev xvdd, sector 44101
end_request: I/O error, dev xvdd, sector 44293
end_request: I/O error, dev xvdd, sector 25669
end_request: I/O error, dev xvdd, sector 25677
end_request: I/O error, dev xvdd, sector 25669
end_request: I/O error, dev xvdd, sector 25669
end_request: I/O error, dev xvdd, sector 44101
Buffer I/O error on device xvdd2, logical block 4608
Buffer I/O error on device xvdd2, logical block 4609
end_request: I/O error, dev xvdd, sector 44101
Buffer I/O error on device xvdd2, logical block 4608
Buffer I/O error on device xvdd2, logical block 4609
end_request: I/O error, dev xvdd, sector 25669
Buffer I/O error on device xvdd2, logical block 0
Buffer I/O error on device xvdd2, logical block 1
Buffer I/O error on device xvdd2, logical block 2
Buffer I/O error on device xvdd2, logical block 3
Buffer I/O error on device xvdd2, logical block 4
Buffer I/O error on device xvdd2, logical block 5
end_request: I/O error, dev xvdd, sector 25713
end_request: I/O error, dev xvdd, sector 25669
end_request: I/O error, dev xvdd, sector 44101
end_request: I/O error, dev xvdd, sector 44101
end_request: I/O error, dev xvdd, sector 25669
end_request: I/O error, dev xvdd, sector 25713
end_request: I/O error, dev xvdd, sector 25669
end_request: I/O error, dev xvdd, sector 25669
Buffer I/O error on device xvdd2, logical block 0
end_request: I/O error, dev xvdd, sector 25669
SCSI subsystem initialized
Fusion MPT base driver 3.04.20rh
Copyright (c) 1999-2008 LSI Corporation
Fusion MPT SAS Host driver 3.04.20rh
PCI: Enabling device 0000:00:00.0 (0000 -> 0002)
mptbase: ioc0: Initiating bringup
ioc0: LSISAS1068E B3: Capabilities={Initiator}
Failed to obtain physical IRQ 38
PCI: Setting latency timer of device 0000:00:00.0 to 64
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Attempting Retry Config request type 0x1, page 0x2, action 0
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Initiating recovery
mptbase: ioc0: Attempting Retry Config request type 0x1, page 0x2, action 0
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Initiating recovery
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Attempting Retry Config request type 0x1, page 0x1, action 0
.....
INFO: task modprobe:894 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe      D 000000450c47ff50     0   894    876
(NOTLB)
 ffff8801fd5b9c38  0000000000000286  ffff8801ff48c7a0  00000000000064fd
 0000000000000007  ffff8801feeac100  ffff8801ffc78040  0000000000001f9a
 ffff8801feeac2e8  ffff8801fd5ba000
Call Trace:
 [<ffffffff802882a9>] __wake_up_common+0x3e/0x68
 [<ffffffff80261fd7>] wait_for_completion+0x7d/0xaa
 [<ffffffff80289a9f>] default_wake_function+0x0/0xe
 [<ffffffff8029a325>] call_usermodehelper_keys+0xe3/0xf8
 [<ffffffff8029a33a>] __call_usermodehelper+0x0/0x4f
 [<ffffffff8029a716>] request_module+0x139/0x14d
 [<ffffffff80304b9a>] init_dir+0x0/0x1c
 [<ffffffff803050b9>] create_dir+0x1c3/0x1cf
 [<ffffffff803477dc>] kobject_release+0x0/0x9
 [<ffffffff80259e86>] kobject_get+0x12/0x17
 [<ffffffff803b4bfd>] class_register+0xe1/0x10f
 [<ffffffff803b4ea6>] class_create+0x53/0x70
 [<ffffffff881b857d>] :parport:parport_register_driver+0x1b/0x74
 [<ffffffff881d8201>] :lp:lp_init_module+0x201/0x261
 [<ffffffff802a446d>] sys_init_module+0xbd/0x206
 [<ffffffff8025f106>] system_call+0x86/0x8b
 [<ffffffff8025f080>] system_call+0x0/0x8b

INFO: task modprobe:901 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe      D 000000450d2c94e1     0   901    900
(NOTLB)
 ffff8801fe075d88  0000000000000286  0000000000000000  ffffffff8021acb7
 0000000000000007  ffff8801fe0b9860  ffffffff80503b80  00000000000172ea
 ffff8801fe0b9a48  0000000000000002
Call Trace:
 [<ffffffff8021acb7>] vsnprintf+0x3f8/0x627
 [<ffffffff80248f31>] sprintf+0x51/0x59
 [<ffffffff80263852>] __down+0xc3/0xd8
 [<ffffffff80289a9f>] default_wake_function+0x0/0xe
 [<ffffffff8025893e>] netlink_broadcast+0x2a9/0x2f1
 [<ffffffff80263510>] __down_failed+0x35/0x3a
 [<ffffffff803b3a81>] __driver_attach+0x0/0xb6
 [<ffffffff803b3b20>] __driver_attach+0x9f/0xb6
 [<ffffffff803b3a81>] __driver_attach+0x0/0xb6
 [<ffffffff803b32ac>] bus_for_each_dev+0x43/0x6e
 [<ffffffff803b2ee1>] bus_add_driver+0x76/0x110
 [<ffffffff80355818>] __pci_register_driver+0x51/0xa6
 [<ffffffff881eb3db>] :parport_pc:parport_pc_init+0x2fa/0x332
 [<ffffffff802a446d>] sys_init_module+0xbd/0x206
 [<ffffffff8025f106>] system_call+0x86/0x8b
 [<ffffffff8025f080>] system_call+0x0/0x8b

mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Attempting Retry Config request type 0x9, page 0x0, action 0
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Initiating recovery
mptbase: ioc0: Attempting Retry Config request type 0x9, page 0x0, action 0
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Initiating recovery
scsi0 : ioc0: LSISAS1068E B3, FwRev=00192f00h, Ports=1, MaxQ=266, IRQ=38
mptbase: ioc0: Attempting Retry Config request type 0x12, page 0x0, action 0
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Initiating recovery
lp: driver loaded but no devices found
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
device-mapper: multipath: version 1.0.6 loaded
end_request: I/O error, dev xvdd, sector 25669
EXT3 FS on dm-0, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on xvda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev xvda1, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
kjournald starting.  Commit interval 5 seconds
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3 FS on xvdb1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev xvdb1, type ext3), uses xattr
Adding 10452984k swap on /dev/VolGroup00/LogVol01.  Priority:-1 extents:1
across:10452984k
SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses
genfs_contexts
IA-32 Microcode Update Driver: v1.14-xen <tigran@xxxxxxxxxxx>
Loading iSCSI transport class v2.0-871.
802.1Q VLAN Support v1.8 Ben Greear <greearb@xxxxxxxxxxxxxxx>
All bugs added by David S. Miller <davem@xxxxxxxxxx>
libcxgbi:libcxgbi_init_module: tag itt 0x1fff, 13 bits, age 0xf, 4 bits.
libcxgbi:ddp_setup_host_page_size: system PAGE 4096, ddp idx 0.
Chelsio T3 iSCSI Driver cxgb3i v2.0.0 (Jun. 2010)
iscsi: registered transport (cxgb3i)
Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.0.3 (Aug 04, 2011)
iscsi: registered transport (bnx2i)
iscsi: registered transport (tcp)
iscsi: registered transport (iser)
iscsi: registered transport (be2iscsi)
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
i2c /dev entries driver
end_request: I/O error, dev xvdd, sector 25669
end_request: I/O error, dev xvdd, sector 25669
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
mptbase: ioc0: Attempting Retry Config request type 0x1, page 0x2, action 0
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Initiating recovery
Bridge firewalling registered
Ebtables v2.0 registered
libvirtd[2946]: segfault at 0000000000000000 rip 00000030c3c78c4b rsp
00007fff07e63788 error 4
mptbase: ioc0: Attempting Retry Config request type 0x1, page 0x2, action 0
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Initiating recovery
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Attempting Retry Config request type 0x10, page 0x0, action 0
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Initiating recovery
mptbase: ioc0: Attempting Retry Config request type 0x10, page 0x0, action 0
mptbase: ioc0: WARNING - Issuing Reset from mpt_config!!,
doorbell=0x24000000
mptbase: ioc0: Initiating recovery



Interface info in Dom0 (lspci -v):
[root@localhost boot]# lspci -v
07:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
PCI-Express Fusion-MPT SAS (rev 08)
 Subsystem: Dell SAS 6/iR Adapter RAID Controller
 Flags: bus master, fast devsel, latency 0, IRQ 38
 I/O ports at dc00 [size=256]
 Memory at dd2ec000 (64-bit, non-prefetchable) [size=16K]
 Memory at dd2f0000 (64-bit, non-prefetchable) [size=64K]
 Expansion ROM at dd100000 [disabled] [size=1M]
 Capabilities: [50] Power Management version 2
 Capabilities: [68] Express Endpoint, MSI 00
 Capabilities: [98] MSI: Enable- Count=1/1 Maskable- 64bit+
 Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
 Capabilities: [100] Advanced Error Reporting
 Kernel driver in use: pciback
 Kernel modules: mptsas

Domu lspci -v shows:
[root@Server1 ~]# lspci -v
00:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
PCI-Express Fusion-MPT SAS (rev 08)
        Subsystem: Dell SAS 6/iR Adapter RAID Controller
        Flags: bus master, fast devsel, latency 0, IRQ 38
        I/O ports at dc00 [size=256]
        Memory at dd2ec000 (64-bit, non-prefetchable) [size=16K]
        Memory at dd2f0000 (64-bit, non-prefetchable) [size=64K]
        Expansion ROM at dd100000 [disabled] [size=1M]
        Capabilities: [50] Power Management version 2
        Capabilities: [68] Express Endpoint, MSI 00
        Capabilities: [98] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: mptsas
        Kernel modules: mptsas


Many thanks, and sorry for the long post! 
 
Phil Quesinberry
Q Systems Engineering, Inc.
Electronic Controls and Embedded Systems Development
(410) 969-8002
http://www.qsystemsengineering.com
 


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users