Is it possible to use MSI-X in a PV Xen DomU (2.6.18.8) for a PCI pass-through
device? I am running an Intel ixgbe NIC in DomU, but I do not seem to be able
to run it with MSI-X enabled. In fact, I do not get it to work in any interrupt
mode. The DomU/Dom0 kernel is compiled with PCI_MSI=y and I have msi=1 set in
Xen's boot options.
I do seem to get an interrupt clash when running up the ixgbe device driver in
DomU. Also, packets seem to be corrupted when I capture with tcpdump on the
interfaces.
tcpdump -i eth1 -xx gives me this:
01:46:56.770226 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui
Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
0x0030: 0000 0000 0000 0000 0000 0000
01:46:56.973564 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui
Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
0x0030: 0000 0000 0000 0000 0000 0000
01:46:57.176902 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui
Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
0x0030: 0000 0000 0000 0000 0000 0000
01:46:57.380238 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui
Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
0x0000: 0000 0000 0000 0000 0000 0000 0000 0000
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000
0x0020: 0000 0000 0000 0000 0000 0000 0000 0000
0x0030: 0000 0000 0000 0000 0000 0000
I have tried to run DomU/DomU with the irqpoll boot parameter but when I do
that then the DomU fails to boot and hangs with a kernel panic:
Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5
Copyright (c) 1999-2008 Intel Corporation.
PCI: Enabling device 0000:02:00.0 (0000 -> 0003)
ixgbe: Interrupt Type set to 1
ixgbe: 0000:02:00.0: ixgbe_check_options: Multiple queues are not supported
while MSI-X is disabled. Disabling Multiple Queues.
ixgbe: 0000:02:00.0: ixgbe_check_options: RSS is not supported while multiple
queues are disabled. Disabling RSS.
ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue
count = 1, Tx Queue count = 1
ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8)
00:21:f7:58:01:6d
ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
PCI: Enabling device 0000:02:00.1 (0000 -> 0003)
BUG: unable to handle kernel NULL pointer dereference at virtual address
0000002d
printing eip:
c01fccaa
2c8fa000 -> *pde = 00000000:60b62001
2cfc1000 -> *pme = 00000000:00000000
Oops: 0002 [#1]
SMP
Modules linked in: ixgbe aacraid usb_storage libusual ata_piix ahci libata
sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 0
EIP: 0061:[<c01fccaa>] Not tainted VLI
EFLAGS: 00210246 (2.6.18.8-xen #5)
EIP is at pci_enable_msix+0x1da/0x510
eax: ed717800 ebx: 00000011 ecx: 00000000 edx: 00000004
esi: 00000000 edi: ed717800 ebp: ec951f80 esp: ecab9d54
ds: 007b es: 007b ss: 0069
Process modprobe (pid: 1566, ti=ecab8000 task=eccd2e10 task.ti=ecab8000)
Stack: 00000098 ecfd4400 c036d220 c015f905 ecfd4400 ecfd4400 00000004 ee0e639a
00000000 00000000 00000008 ec951f60 000000b2 00000002 00030003 0000007b
0000007b 00000000 c017682e ec951f80 00000004 00000010 ecfd4400 00000000
Call Trace:
[<c015f905>] __kzalloc+0x15/0x50
[<ee0e639a>] ixgbe_alloc_queues+0x8a/0x580 [ixgbe]
[<c017682e>] __kmalloc+0xbe/0xf0
[<ee0e6a13>] ixgbe_init_interrupt_scheme+0x183/0x340 [ixgbe]
[<ee0e3b81>] ixgbe_read_pci_cfg_word+0x21/0x30 [ixgbe]
[<ee0e8c17>] ixgbe_probe+0x6e7/0x9e0 [ixgbe]
[<c017682e>] __kmalloc+0xbe/0xf0
[<c01f5ef6>] pci_device_probe+0x56/0x80
[<c023ecd4>] driver_probe_device+0x44/0xc0
[<c023ee52>] __driver_attach+0x82/0x90
[<c023e64a>] bus_for_each_dev+0x3a/0x60
[<c023ec16>] driver_attach+0x16/0x20
[<c023edd0>] __driver_attach+0x0/0x90
[<c023e2bc>] bus_add_driver+0x8c/0x140
[<c01f6097>] __pci_register_driver+0x47/0x70
[<c0143b08>] sys_init_module+0x148/0x1b40
[<c01ea55f>] prio_tree_insert+0x1f/0x260
[<c01187fc>] do_page_fault+0x10c/0xc6f
[<c010acc0>] sys_mmap2+0xd0/0xe0
[<c01059bf>] syscall_call+0x7/0xb
Code: 00 89 54 24 38 ba ed ff ff ff 89 4c 24 3c 0b 4c 24 38 0f 85 2d 01 00 00 8b 4c
24 28 89 d6 89 8f b0 01 00 00 e9 62 fe ff ff 89 f8 <89> 5b 1c e8 3e f2 ff ff 8b
7c 24 18 89 44 24 54 85 ff 0f 8e 48
EIP: [<c01fccaa>] pci_enable_msix+0x1da/0x510 SS:ESP 0069:ecab9d54
udevd-event[1564]: run_program: '/sbin/modprobe' abnormal exit
I have also tried to use legacy interrupts by setting InterruptType to 0:
Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5-lro
Copyright (c) 1999-2008 Intel Corporation.
PCI: Enabling device 0000:02:00.0 (0000 -> 0003)
PCI: Setting latency timer of device 0000:02:00.0 to 64
ixgbe: Interrupt Type set to 0
ixgbe: Multiple Queue Support Disabled
ixgbe: Receive-Side Scaling (RSS) set to 0
ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue
count = 1, Tx Queue count = 1
ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8)
00:21:f7:58:01:6d
ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
PCI: Enabling device 0000:02:00.1 (0000 -> 0003)
PCI: Setting latency timer of device 0000:02:00.1 to 64
ixgbe: Interrupt Type set to 0
ixgbe: Multiple Queue Support Disabled
ixgbe: Receive-Side Scaling (RSS) set to 0
ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
ixgbe: 0000:02:00.1: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue
count = 1, Tx Queue count = 1
ixgbe: 0000:02:00.1: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8)
00:21:f7:58:01:6c
ixgbe: eth1: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
Without MSI and MSI-X the driver loads properly. However, I still end up with
corrupted packets on tcpdump, and additionally I am getting a kernel crash when
the guest is under high network load:
BUG: unable to handle kernel NULL pointer dereference at virtual address
00000034
printing eip:
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Oops: 0002 [#1]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe
aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 0
EIP: 0061:[<c0158df7>] Not tainted VLI
EFLAGS: 00010082 (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Oops: 0002 [#2]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe
aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 0
EIP: 0061:[<c0145df6>] Not tainted VLI
EFLAGS: 00010002 (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Recursive die() failure, output suppressed
<0>Kernel panic - not syncing: Fatal exception in interrupt
BUG: warning at
/home/user/Download/linux-2.6.18-xen.hg/arch/i386/kernel/smp-xen.c:511/smp_call_function()
[<c0112541>] <1>BUG: unable to handle kernel paging request at virtual address
245c8bf1
printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Oops: 0002 [#3]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe
aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 0
EIP: 0061:[<c0145df6>] Not tainted VLI
EFLAGS: 00010002 (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Oops: 0002 [#4]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe
aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 0
EIP: 0061:[<c0145df6>] Not tainted VLI
EFLAGS: 00010002 (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Recursive die() failure, output suppressed
<0>Kernel panic - not syncing: Fatal exception in interrupt
<1>BUG: unable to handle kernel NULL pointer dereference at virtual address
00000064
printing eip:
c0158df7
2d2e1000 -> *pde = 00000000:6051f001
2d2e2000 -> *pme = 00000000:00000000
Oops: 0002 [#5]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe
aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 1
EIP: 0061:[<c0158df7>] Not tainted VLI
EFLAGS: 00010282 (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2d2e1000 -> *pde = 00000000:6051f001
2d2e2000 -> *pme = 00000000:00000000
Oops: 0002 [#6]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe
aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd usbcore
CPU: 1
EIP: 0061:[<c0145df6>] Not tainted VLI
EFLAGS: 00010002 (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
printing eip:
c0145df6
2d2e1000 -> *pde = 00000000:6051f001
2d2e2000 -> *pme = 00000000:00000000
Recursive die() failure, output suppressed
What are the correct interrupt settings for using ixgbe in PV pass-through
mode? I have tried all three possible modes but I do not seem to get the device
working properly under high network load.
I am using Xen 3.3.1.
release : 2.6.18.8-xen
version : #5 SMP Thu Mar 5 04:14:30 PST 2009
machine : i686
nr_cpus : 2
nr_nodes : 1
cores_per_socket : 2
threads_per_core : 1
cpu_mhz : 2200
virt_caps : hvm
total_memory : 2039
free_memory : 6
node_to_cpu : node0:0-1
node_to_memory : node0:6
xen_major : 3
xen_minor : 3
xen_extra : .1
xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xf5800000
xen_changeset : unavailable
cc_compiler : gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)
cc_compile_by : root
cc_compile_domain : localdomain
cc_compile_date : Sun Mar 8 05:40:51 PDT 2009
xend_config_format : 4
I am using PCI pass-through to assign the NIC to my PV guest, so this is the part of my config file:
pci = [ '02:00.0','02:00.1' ]
I am also hiding the PCI devices from Dom0, so my Dom0 GRUB config looks like
this:
title Xen 3.3
root (hd0,1)
kernel /boot/xen-3.3.1.gz console=vga msi=1
module /boot/vmlinuz-2.6.18.8-xen root=LABEL=/ ro console=tty0 rhgb quiet
pciback.permissive pciback.hide=(02:00.0)(02:00.1)
reassigndev=0000:02:00.0,0000:02:00.1
module /boot/initrd-2.6.18.8-xen.img
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
