Re: [Xen-devel] 82576 VF passthrough

[Simon Horman <horms@xxxxxxxxxxxx>]

On Thu, Sep 24, 2009 at 12:26:25AM +0530, Satish Chowdhury wrote:
> Hi,
> 
> I am trying to setup 82576 VF passthrough to pv domU non Vt-d machine . My
> dom0 is http://hg.vergenet.net/xen/linux-2.6.18-xen-igb/  kernel on FC11 and
> 3.4.1 Xen.  My domU is also
> http://hg.vergenet.net/xen/linux-2.6.18-xen-igb/kernel on Ubuntu
> junty.
> 
> 
> Passthrough of PF device (igb driver) is successful in my setup. I could see
> the PF BDF on domU, and the igb driver gets loaded on domU.
> 
> But, when I do passthrough of VF device, lspci output on DomU doesn't show
> the VF device.(CONFIG_XEN_PCIDEV_FRONTEND=y)
> 
> The steps I follow on Dom0:
>  #rmmod igb
>  #modprobe igb max_vfs=1
>  #modprobe pciback
>  #echo -n VF-BDF > /sys/bus/pci/drivers/pciback/new_slot
>  #echo -n VF-BDF > /sys/bus/pci/drivers/pciback/bind
> On domU config file has  pci = [' VF-BDF'] .
> 
> The domU starts without any errors.
> 
> #xm pci-list domU-ID
> shows the VF BDF
> 
> Am i missing some thing of VF Passthrough? Please give me pointers to
> find/debug the issue.

Hi Satish,

I'm not sure why your setup isn't working but I'm not aware of any tricks
that are required. Though I should point out that the linux-2.6.18-xen-igb
kernel does not include an igbvf driver so you will need to run a different
kernel in your domUs - I used 2.6.31 for testing purposes.

On the off chance that it is useful to you here are some details of a
working system that I have.

Hardware
--------
  Machine: HP dc7800 MT: [japanese] 
http://h50146.www5.hp.com/products/desktops/old/dc7800mt/e8300_10_160w_xpv.html
  Chipset: Q35
  82576 PCI Card: E1G42ET (dual port RJ-45 copper)


Dom0
----

The dom0 kernel config is attached as config-2.6.18-xen-igb.
I ran my test using the following revision of
http://hg.vergenet.net/xen/linux-2.6.18-xen-igb/

        changeset:   1123:f4deded581e6
        user:        Simon Horman <horms@xxxxxxxxxxxx>
        date:        Mon Aug 31 09:54:00 2009 +1000
        summary:     Merge with http://xenbits.xensource.com/linux-2.6.18-xen.hg

The hypervisor and xend comes from the following revision
of http://xenbits.xensource.com/staging/xen-unstable.hg,
which should also exist in http://xenbits.xensource.com/xen-unstable.hg

        changeset:   20010:68e8b8379244
        user:        Keir Fraser <keir.fraser@xxxxxxxxxx>
        date:        Sun Aug 02 13:43:15 2009 +0100
        summary:     Add a single trigger for all diagnostic keyhandlers

qemu-xen is the following revision of
git://xenbits.xensource.com/staging/qemu-xen-unstable.git/,
which should also exist in git://xenbits.xensource.com/qemu-xen-unstable.git/

        commit a42b67ed27a9744a2492c8ffe5c26935bd19ee27
        Author: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
        Date:   Tue Jul 21 16:08:24 2009 +0100

        use XC_PAGE_SIZE in xenfbfront (fix stubdom build on NetBSD)

        Apropos of a report from Christoph Egger about misuse of PAGE_SIZE in
        xenfbfront; change references to XC_PAGE_SIZE which is correct in our
        context.  This fix suggested by Stefano Stabillini.

        Signed-off-by: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>

The grub snippet for dom0 is as follows.  pci=assign-busses is needed
because the BIOS on my machine does not appear to fully support VT-d
and without it there is nowhere for the VFs to go. reassign_resources
and reassigndev=00:1d.7 are spurious.

title           Xen
root            (hd0,0)
kernel          /xen com1=115200,8n1,0x3f8,0 console=com1 loglvl=all 
guest_loglvl=all vtd=1 iommu dom0_mem=1G
module          /vmlinuz-2.6.18.8 root=/dev/mapper/sam-root_x86_64 
console=ttyS0 loglevel=8 ro pci=assign-busses reassign_resources 
reassigndev=00:1d.7
module          /initrd.img-2.6.18.8

The igb module was inserted into the kernel using

        modprobe igb max_vfs=4

The output of 'dmesg | grep igb' in domU is:

igb 0000:01:00.0: pci_enable_pcie_error_reporting failed 0xffffffea
igb 0000:01:00.0: 4 vfs allocated
igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:01:00.0: eth3: (PCIe:2.5Gb/s:Width x4) ffff8800389841a8M
igb 0000:01:00.0: eth3: PBA No: e43709-003
igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
igb 0000:01:00.1: pci_enable_pcie_error_reporting failed 0xffffffea
igb 0000:01:00.1: 4 vfs allocated
igb 0000:01:00.1: Intel(R) Gigabit Ethernet Network Connection
igb 0000:01:00.1: eth3: (PCIe:2.5Gb/s:Width x4) ffff8800388101a8M
igb 0000:01:00.1: eth3: PBA No: e43709-003
igb 0000:01:00.1: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)


DomU
----

The domU kernel config is attached as config-2.6.31,
and I have verified that this works with 2.6.31.

The domU grub snippet is:

title           Linux
root            (hd0,0)
kernel          /boot/vmlinuz-2.6.31 root=/dev/hda1 ro loglevel=8 
console=uart,io,0x3f8,115200n8

The domU domain config is attached as hvm and I invoke the domain using:
xm create hvm pci='02:10.0'

   Where 02:10.0 is one of the vf devices.

The output of lspci in dom0 is:

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Class ff80: XenSource, Inc. Xen Platform Device (rev 01)
00:04.0 Ethernet controller: Intel Corporation Device 10ca (rev 01)

The output of 'dmesg | grep igb' in domU is:

[    3.765661] igbvf 0000:00:04.0: irq 48 for MSI/MSI-X
[    3.768937] igbvf 0000:00:04.0: irq 49 for MSI/MSI-X
[    3.772200] igbvf 0000:00:04.0: irq 50 for MSI/MSI-X
[    3.808717] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function
[    3.813071] igbvf 0000:00:04.0: Address: da:3a:32:ee:00:30
[    3.816918] igbvf 0000:00:04.0: MAC: 1
[    7.544165] igbvf 0000:00:04.0: Link is Up 1000 Mbps Full Duplex


Other
-----

Note that if you want to see a link (assuming that everything else works)
on the igbvf device inside the domain, you'll need to bring
up the link on the PF in dom0. 'ifconfig ethX up' should be sufficient.

The dom0 and domU user-spaces are basically Debian Lenny,
though I doubt this has any influence on the results.

Attachment: config-2.6.18-xen-igb
Description: Text document

Attachment: hvm
Description: Text document

Attachment: config-2.6.31
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel