Re: [Xen-users] PCI passthrough of HP P420i RAID controller

[Simon Waterman <simon.waterman@xxxxxxxxxxx>]

Hi,

On Mon, Jul 7, 2014 at 15:11, jacek burghardt <jaceksburghardt@xxxxxxxxx> wrote:
>> I had passed hp sas to windows 2012 R2 and it works fine. i had tried to 
>> assign
>> it to Linux but the drivers were setting the card wrng and causing lockup 
>> requiring
>> me to reboot server.

Its interesting to know that it worked fine with the Windows driver and that 
you got a
lockup too under Linux.  Was it the same P420i controller with SAS drives?

I'm not sure its the Linux hpsa driver causing the lockup though.

I spent some time trying to diagnose where in the driver the lockup was being 
caused
and I tried modifying the probe code in the driver domain to reject the device 
immediately
before doing anything with it.  When I tried to detach the device from the 
driver domain
and re-bind it to the hpsa driver in the Dom-0 it failed in the same way 
despite never
being touched by the hpsa driver in the driver domain.

Purely unbinding the device from the hpsa driver and binding it to xen-pciback 
doesn't
seem to cause the problem because if I rebind it to hpsa without doing the 
attach to
the driver domain the device appears to function normally.  In other words, 
binding and
unbinding from xen-pciback seems fine - attaching the device to the driver 
domain
causes the problem even when the hpsa driver isn't in the picture on the driver 
domain.

Best wishes,

Simon

On Mon, Jul 7, 2014 at 7:34 AM, Simon Waterman  <simon.waterman@xxxxxxxxxxx> 
wrote:
 Hi,

We've been successfully using Xen with PCI pass-through of disk and
network cards for a couple of years now.  We have this architecture working
successfully with a wide range of hardware from HP and Dell but recently
have been getting problems with the HP ProLiant DL380p Gen8 and its
P420i RAID controller.

When we bind the RAID controller to pciback and issue the appropriate
'xl pci-attach' command the controller appears on the driver domain but fails
during probing by the hpsa driver.  Interestingly, in about 1 in every 10
test runs the hpsa driver probes successfully, the controller appears in the
driver domain and the disks appear to function correctly.

In the failure cases the syslog output from the hpsa driver is as follows:

Jul  7 13:30:01 zylevel0-HVM-domU kernel: [   54.855597] bus: 'pci': driver_prob
e_device: matched device 0000:00:00.0 with driver hpsa
Jul  7 13:30:01 zylevel0-HVM-domU kernel: [   54.855599] bus: 'pci': really_prob
e: probing driver hpsa with device 0000:00:00.0
Jul  7 13:30:01 zylevel0-HVM-domU kernel: [   54.855605] HP HPSA Driver (v 3.4.4
-1)
Jul  7 13:30:01 zylevel0-HVM-domU kernel: [   54.855684] hpsa 0000:00:00.0: enab
ling device (0000 -> 0003)
Jul  7 13:30:01 zylevel0-HVM-domU kernel: [   54.855983] hpsa 0000:00:00.0: Xen
PCI mapped GSI34 to IRQ29
Jul  7 13:30:01 zylevel0-HVM-domU kernel: [   54.856089] hpsa 0000:00:00.0: enab
ling bus mastering
Jul  7 13:30:01 zylevel0-HVM-domU kernel: [   54.856659] hpsa 0000:00:00.0: MSIX
Jul  7 13:30:01 zylevel0-HVM-domU kernel: [   54.857384] hpsa 0000:00:00.0: Logi
cal aborts not supported
Jul  7 13:40:00 zylevel0-HVM-domU kernel: [  654.318297] hpsa 0000:00:00.0: 
unable
to get board into simple mode
Jul  7 13:40:00 zylevel0-HVM-domU kernel: [  654.318478] hpsa: probe of 
0000:00:00.0
rejects match -19

The error seems to originate from the hpsa_enter_simple_mode function and looks
like the first point where an interaction with the board via IO mapped memory is
expected to result in a response from the board.

We are using Xen 4.4 and a newly built Linux 3.15.0 kernel with Ubuntu trusty. 
 VTd is
disabled in the BIOS of the Dom-0.  We use the following Grub commands for the 
Dom-0 boot:

        multiboot       /boot/xen.gz placeholder dom0_mem=max:2048m iommu=0
loglvl=all guest_loglvl=all
        module  /boot/vmlinuz-3.15.0 placeholder 
root=UUID=f56cf02b-d46f-4e2a-b667-242f97ec82f5
ro  debug
        module  --nounzip   /boot/initrd.img-3.15.0

The driver domain is a PV guest running the same kernel version as the Dom-0.
The Xen configuration file contains the following kernel boot parameters:

extra="earlyprintk=xen debug max_loop=64 iommu=soft"

After we issue the pci-attach command the following additional lines appear in 
the
'xl dmesg' log:

(XEN) grant_table.c:289:d0 Increased maptrack size to 2 frames
(XEN) mm.c:809: d2: Forcing read-only access to MFN f7f02
(XEN) mm.c:809: d0: Forcing read-only access to MFN f7f02

We were wondering whether anyone in the community had got PCI pass-through of
the P420i working, or any pointers of where to look to further diagnose the 
problem.

Best wishes,

Simon

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users
 
    
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users