Re: [Xen-devel] Non-page aligned MMIO BAR error on dom-0 on Xen HVM setup

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Thu, Oct 15, 2009 at 01:00:31PM +0530, Purna Chandar wrote:
> Hi,
> I am using Neterion x3100  multi-function NIC adapter on stable Xen(2.6.18)
> in HVM mode. I think the problem I have (explained below) is a general one
> and has got nothing to do with neterion card in specific.
> I am trying to assign one of the functions on the adapter to a guest OS(FC11
> 64 bit) in pass-through mode. When I try to start the guest OS (using xm
> create), the dom-0 throws  the error saying the pass-through device has a
> "non-page aligned MMIO BAR" and cannot boot. The function infact has a BAR
> (bar 4) whose size is 256 bytes and the resources are mapped on a
> page-aligned boundary. THe guest cannot be booted for some reason because of
> this.
> 
> To get around with this, I used the following kernel command line option:
> 
> reassign_resources  guestdev=<b:d.f>,<b:d,f>...
> 
> The BDFs of all the devices whose resources we want to re-assign to a
> page-boundary are given in guestdev parameter above. As these neterion
> functions are under a bridge, the bridge's BDF is also given in the guestdev
> option (without the bridge's BDF, I had issues with loading the neterion
> driver on guest OS).
> 
> With this kernel option, I am able to boot the guest OS but I see a
> different problem in the config space of the device in dom-0. The BAR
> regions are shown up as "disabled" in lspci output shown below. Without this
> option, the BARs are enabled.
> 
> 01:00.2 Ethernet controller: Neterion Inc. X3100 Series 10 Gigabit Ethernet
> PCIe (rev 01)
>         Subsystem: Neterion Inc. X3110 Single Port
> 10GBase-CR
>         Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Interrupt: pin A routed to IRQ
> 16
>         Region 0: Memory at f9000000 (64-bit, prefetchable)
> *[disabled]*[size=8M]
>         Region 2: Memory at fc404000 (64-bit, prefetchable)
> *[disabled]*[size=4K]
>         Region 4: Memory at fc405000 (64-bit, prefetchable)
> *[disabled]*[size=256]
>         [virtual] Expansion ROM at fc100000 *[disabled]*[size=512K]
> 
> What this "disabled" means is the resources are not treated as "memory
> mapped resources" at all (as bit-2 in COMMAND register is not set in config
> space).
> Surprisingly, on the guest OS, the "disabled" flag is not seen on the BAR
> regions. I can successfully load the driver and do some data flow on
> pass-through device on the guest.
> 
> Things seem fine so far despite the problem above on dom-0.
> 
> But, when I trigger an FLR (function level reset) on the pass-through
> device/function on dom-0 (not on dom-u), I have problems on guest OS. The
> "disabled" bit appears in the lspci output on the device in the guest. The
> driver is not able to do any memory-mapped reads/writes and hence I am not
> able to load the driver. I triggered the FLR using following command:

Does it change the state in the dom0? It is still disabled, right?

What if you manually set the bit-2 of the command register in either locations
(dom0 or domU).

> # setpci -s 01:00.2 CAP_EXP+8.w=0xa814
> (Before triggering FLR, value at CAP_EXP+8.w is 0x2814. As we have to set
> last bit in PCI Express capability for FLR, value used in above command is
> 0xa814).
> 
> Does anybody have similar issues? Is "reassign_resources" the correct kernel
> option to use? If not, how do I get around the "non-page alignment problem"

The reassign_resources is the fix for non-page aligned BARs in the older
kernels. In the newer ones... still working on that.

The BAR is not page-aligned on baremetal either, right?

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel