Xen project Mailing List

Re: [Xen-devel] HVM crashes when passing second igb interface

To: Samuel Monderer <SMonderer@xxxxxxxxxxxxxxx>

From: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>

Date: Wed, 7 May 2014 13:30:15 -0400

Cc: "xen-devel@xxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxx>

Delivery-date: Wed, 07 May 2014 17:31:08 +0000

List-id: Xen developer discussion <xen-devel.lists.xen.org>

On Wed, May 07, 2014 at 04:31:02PM +0000, Samuel Monderer wrote: > Hi, > > I've upgraded my HVM kernel to 3.14.1 and know when I attach an igb PF > interface I get the following error and crash Could you give a bit more details on the guest configuration? And perhaps the 'lspci' with the previous guest? > > After attaching the first interface as following > > [ 260.042238] ACPI: Error installing CMOS-RTC region handler > [ 260.044658] pci 0000:00:00.0: no hotplug settings from platform > [ 260.044661] pci 0000:00:00.0: using default PCI settings > [ 260.044753] pci 0000:00:01.0: no hotplug settings from platform > [ 260.044754] pci 0000:00:01.0: using default PCI settings > [ 260.044881] ata_piix 0000:00:01.1: no hotplug settings from platform > [ 260.044883] ata_piix 0000:00:01.1: using default PCI settings > [ 260.044975] pci 0000:00:01.3: no hotplug settings from platform > [ 260.044976] pci 0000:00:01.3: using default PCI settings > [ 260.045100] pci 0000:00:02.0: no hotplug settings from platform > [ 260.045102] pci 0000:00:02.0: using default PCI settings > [ 260.045197] xen-platform-pci 0000:00:03.0: no hotplug settings from > platform > [ 260.045199] xen-platform-pci 0000:00:03.0: using default PCI settings > [ 260.045286] pci 0000:00:04.0: no hotplug settings from platform > [ 260.045341] pci 0000:00:00.0: no hotplug settings from platform > [ 260.045342] pci 0000:00:00.0: using default PCI settings > [ 260.045462] pci 0000:00:01.0: no hotplug settings from platform > [ 260.045464] pci 0000:00:01.0: using default PCI settings > [ 260.045556] ata_piix 0000:00:01.1: no hotplug settings from platform > [ 260.045558] ata_piix 0000:00:01.1: using default PCI settings > [ 260.045669] pci 0000:00:01.3: no hotplug settings from platform > [ 260.045671] pci 0000:00:01.3: using default PCI settings > [ 260.045833] pci 0000:00:02.0: no hotplug settings from platform > [ 260.045835] pci 0000:00:02.0: using default PCI settings > [ 260.045993] xen-platform-pci 0000:00:03.0: no hotplug settings from > platform > [ 260.045995] xen-platform-pci 0000:00:03.0: using default PCI settings > [ 260.046129] pci 0000:00:04.0: no hotplug settings from platform > [ 260.046448] dca service started, version 1.12.1 > [ 260.046451] pci 0000:00:00.0: no hotplug settings from platform > [ 260.046454] pci 0000:00:00.0: using default PCI settings > [ 260.046597] pci 0000:00:01.0: no hotplug settings from platform > [ 260.046599] pci 0000:00:01.0: using default PCI settings > [ 260.046689] ata_piix 0000:00:01.1: no hotplug settings from platform > [ 260.046690] ata_piix 0000:00:01.1: using default PCI settings > [ 260.046777] pci 0000:00:01.3: no hotplug settings from platform > [ 260.046778] pci 0000:00:01.3: using default PCI settings > [ 260.046886] pci 0000:00:02.0: no hotplug settings from platform > [ 260.046887] pci 0000:00:02.0: using default PCI settings > [ 260.046978] xen-platform-pci 0000:00:03.0: no hotplug settings from > platform > [ 260.046980] xen-platform-pci 0000:00:03.0: using default PCI settings > [ 260.047128] pci 0000:00:04.0: no hotplug settings from platform > [ 260.047174] pps_core: LinuxPPS API ver. 1 registered > [ 260.047175] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo > Giometti <giometti@xxxxxxxx> > [ 260.047859] PTP clock support registered > [ 260.050174] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.0.5-k > [ 260.050177] igb: Copyright (c) 2007-2013 Intel Corporation. > [ 260.050297] igb 0000:00:04.0: enabling device (0000 -> 0002) > [ 260.139424] xen: --> pirq=17 -> irq=33 (gsi=33) > [ 260.265105] igb 0000:00:04.0: added PHC on eth1 > [ 260.265107] igb 0000:00:04.0: Intel(R) Gigabit Ethernet Network Connection > [ 260.265109] igb 0000:00:04.0: eth1: (PCIe:5.0Gb/s:Width x4) > a0:36:9f:00:29:dd > [ 260.265707] igb 0000:00:04.0: eth1: PBA No: G13159-006 > [ 260.265708] igb 0000:00:04.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx > queue(s) > [ 260.280260] systemd-udevd[2260]: renamed network interface eth1 to eth6 > > And After I attach the second igb PF interface the kernel crashes > > [ 546.765926] igb 0000:00:05.0: Xen PCI frontend has not registered > MSI/MSI-X support! .. and this means it cannot get interrupts. What does your /var/log/xen/qemu-<guest> look like? Or 'xl dmesg'? Do you see anything about not enough GSI? > [ 546.769763] BUG: unable to handle kernel NULL pointer dereference at > 0000000000000038 > [ 546.773715] IP: [<ffffffffa038e31b>] igb_reset_q_vector+0xb/0x50 [igb] Ha! Looks like the igb driver is not able to deal with errors when it cannot allocate all of the interrupts. One workaround might be to use (on Xen command line): 'extra_guest_irqs=256' There are some fixes (if this is the problem I think it is) that I can suggest. But before I pooint you that way please provide what I had asked above - thanks! > [ 546.773720] PGD 0 > [ 546.773720] Oops: 0000 [#1] SMP > [ 546.773720] Modules linked in: igb ptp pps_core dca i2c_algo_bit nfsd > auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc loop fuse > crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel > ghash_clmulni_intel ppdev aesni_intel aes_x86_64 lrw gf128mul glue_helper > ablk_helper parport_pc cryptd joydev parport ttm drm_kms_helper drm psmouse > processor i2c_piix4 i2c_core pcspkr serio_raw thermal_sys evdev button ext4 > crc16 mbcache jbd2 sr_mod cdrom ata_generic xen_netfront xen_blkfront > ata_piix libata scsi_mod floppy > [ 546.773720] CPU: 0 PID: 53 Comm: kworker/u256:1 Not tainted 3.14-1-amd64 > #1 Debian 3.14.2-1 > [ 546.773720] Hardware name: Xen HVM domU, BIOS 4.3.0 12/04/2013 > [ 546.773720] Workqueue: kacpi_hotplug acpi_hotplug_work_fn > [ 546.773720] task: ffff880036ed55b0 ti: ffff880000054000 task.ti: > ffff880000054000 > [ 546.773720] RIP: 0010:[<ffffffffa038e31b>] [<ffffffffa038e31b>] > igb_reset_q_vector+0xb/0x50 [igb] > [ 546.773720] RSP: 0018:ffff880000055a90 EFLAGS: 00010206 > [ 546.773720] RAX: 0000000000000000 RBX: 0000000000000006 RCX: > ffff880119e41200 > [ 546.773720] RDX: ffff88001712b858 RSI: 0000000000000007 RDI: > ffff88009a48a880 > [ 546.773720] RBP: ffff88009a48a880 R08: 0000000000000004 R09: > 0000000000000008 > [ 546.773720] R10: ffffffff818a6d40 R11: 0000000000000fd4 R12: > 0000000000000008 > [ 546.773720] R13: 0000000000000000 R14: ffff88001712b000 R15: > ffff88001712b000 > [ 546.773720] FS: 0000000000000000(0000) GS:ffff88009b400000(0000) > knlGS:0000000000000000 > [ 546.773720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 546.773720] CR2: 0000000000000038 CR3: 0000000098af2000 CR4: > 00000000001406f0 > [ 546.773720] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 546.773720] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 546.773720] Stack: > [ 546.773720] ffffffffa038e40d ffff88009a48a880 ffff88009a48a880 > ffff88001712b000 > [ 546.773720] ffffffffa0391588 ffff88009a48a000 ffff88009a48a880 > 0000000000000000 > [ 546.773720] ffff88001712b000 ffff88009a48a000 ffff88009a48a880 > ffff88001712b000 > [ 546.773720] Call Trace: > [ 546.773720] [<ffffffffa038e40d>] ? > igb_reset_interrupt_capability+0x3d/0x70 [igb] > [ 546.773720] [<ffffffffa0391588>] ? igb_init_interrupt_scheme+0x3d8/0x3f0 > [igb] > [ 546.773720] [<ffffffffa0396b23>] ? igb_sriov_reinit+0x53/0xb0 [igb] > [ 546.773720] [<ffffffffa0396c96>] ? igb_pci_enable_sriov+0x116/0x1e0 [igb] > [ 546.773720] [<ffffffffa0397257>] ? igb_probe+0x4c7/0x11e0 [igb] > [ 546.773720] [<ffffffff812b02da>] ? local_pci_probe+0x3a/0xa0 > [ 546.773720] [<ffffffff812b15ea>] ? pci_device_probe+0xca/0x120 > [ 546.773720] [<ffffffff81365d26>] ? driver_probe_device+0x76/0x220 > [ 546.773720] [<ffffffff81365ed0>] ? driver_probe_device+0x220/0x220 > [ 546.773720] [<ffffffff81364163>] ? bus_for_each_drv+0x53/0x90 > [ 546.773720] [<ffffffff81365c70>] ? device_attach+0x90/0xb0 > [ 546.773720] [<ffffffff812a8474>] ? pci_bus_add_device+0x34/0x60 > [ 546.773720] [<ffffffff812a8859>] ? pci_bus_add_devices+0x39/0xa0 > [ 546.773720] [<ffffffff814b0898>] ? enable_slot+0x278/0x2f0 > [ 546.773720] [<ffffffff812c6588>] ? acpiphp_check_bridge.part.6+0xf8/0x110 > [ 546.773720] [<ffffffff812c6d62>] ? hotplug_event+0x162/0x190 > [ 546.773720] [<ffffffff812c6db0>] ? hotplug_event_work+0x20/0x60 > [ 546.773720] [<ffffffff812e75d1>] ? acpi_hotplug_work_fn+0x13/0x1c > [ 546.773720] [<ffffffff81078b9d>] ? process_one_work+0x16d/0x420 > [ 546.773720] [<ffffffff81079756>] ? worker_thread+0x116/0x3b0 > [ 546.773720] [<ffffffff81079640>] ? rescuer_thread+0x330/0x330 > [ 546.773720] [<ffffffff8107f911>] ? kthread+0xc1/0xe0 > [ 546.773720] [<ffffffff8107f850>] ? kthread_create_on_node+0x180/0x180 > [ 546.773720] [<ffffffff814c718c>] ? ret_from_fork+0x7c/0xb0 > [ 546.773720] [<ffffffff8107f850>] ? kthread_create_on_node+0x180/0x180 > [ 546.773720] Code: 44 09 c8 eb cf 0f 1f 00 b8 a1 ff ff ff c3 66 90 b8 ea ff > ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 48 63 f6 48 8b 84 f7 48 0f 00 00 <48> > 8b 50 38 48 85 d2 74 10 0f b6 52 4e 48 c7 84 d7 80 02 00 00 > [ 546.773720] RIP [<ffffffffa038e31b>] igb_reset_q_vector+0xb/0x50 [igb] > [ 546.773720] RSP <ffff880000055a90> > [ 546.773720] CR2: 0000000000000038 > [ 546.773720] ---[ end trace 8b1d98daee2f26b8 ]--- > [ 546.926235] BUG: unable to handle kernel paging request at ffffffffffffffd8 > [ 546.928690] IP: [<ffffffff8107fe77>] kthread_data+0x7/0x10 > [ 546.930199] PGD 180f067 PUD 1811067 PMD 0 > [ 546.930199] Oops: 0000 [#2] SMP > [ 546.930199] Modules linked in: igb ptp pps_core dca i2c_algo_bit nfsd > auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc loop fuse > crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel > ghash_clmulni_intel ppdev aesni_intel aes_x86_64 lrw gf128mul glue_helper > ablk_helper parport_pc cryptd joydev parport ttm drm_kms_helper drm psmouse > processor i2c_piix4 i2c_core pcspkr serio_raw thermal_sys evdev button ext4 > crc16 mbcache jbd2 sr_mod cdrom ata_generic xen_netfront xen_blkfront > ata_piix libata scsi_mod floppy > [ 546.930199] CPU: 0 PID: 53 Comm: kworker/u256:1 Tainted: G D > 3.14-1-amd64 #1 Debian 3.14.2-1 > [ 546.930199] Hardware name: Xen HVM domU, BIOS 4.3.0 12/04/2013 > [ 546.930199] task: ffff880036ed55b0 ti: ffff880000054000 task.ti: > ffff880000054000 > [ 546.930199] RIP: 0010:[<ffffffff8107fe77>] [<ffffffff8107fe77>] > kthread_data+0x7/0x10 > [ 546.930199] RSP: 0018:ffff8800000557b0 EFLAGS: 00010002 > [ 546.930199] RAX: 0000000000000000 RBX: 0000000000000000 RCX: > 000000000000000f > [ 546.930199] RDX: 0000000000000000 RSI: 0000000000000000 RDI: > ffff880036ed55b0 > [ 546.930199] RBP: ffff880036ed55b0 R08: 0000000000000001 R09: > 000000000000ba9a > [ 546.930199] R10: ffffffff81a90a60 R11: 0000000000000000 R12: > ffff88009b414380 > [ 546.930199] R13: 0000000000000000 R14: ffff880036ed55a0 R15: > ffff880036ed55b0 > [ 546.930199] FS: 0000000000000000(0000) GS:ffff88009b400000(0000) > knlGS:0000000000000000 > [ 546.930199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 546.930199] CR2: 0000000000000028 CR3: 0000000098af2000 CR4: > 00000000001406f0 > [ 546.930199] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 546.930199] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 546.930199] Stack: > [ 546.930199] ffffffff81079d68 ffff880036ed59d0 ffffffff814bbe1d > 0000000000014380 > [ 546.930199] 0000000000014380 ffff880000055fd8 ffff880036ed55b0 > ffff880036ed5bf0 > [ 546.930199] ffff880036ed58c8 ffff880036ed55a0 ffff88009eaf51b0 > ffff880036ed55a0 > [ 546.930199] Call Trace: > [ 546.930199] [<ffffffff81079d68>] ? wq_worker_sleeping+0x8/0x80 > [ 546.930199] [<ffffffff814bbe1d>] ? __schedule+0x45d/0x6e0 > [ 546.930199] [<ffffffff810617e7>] ? do_exit+0x6e7/0xa60 > [ 546.930199] [<ffffffff814c0282>] ? oops_end+0x92/0xd0 > [ 546.930199] [<ffffffff814b67a5>] ? no_context+0x272/0x27e > [ 546.930199] [<ffffffff814c26d0>] ? __do_page_fault+0x80/0x4f0 > [ 546.930199] [<ffffffff8101257b>] ? __switch_to+0x11b/0x4b0 > [ 546.930199] [<ffffffff8113e45c>] ? pcpu_free_area+0x12c/0x1a0 > [ 546.930199] [<ffffffff8113e74a>] ? free_percpu+0x7a/0x140 > [ 546.930199] [<ffffffff814bf788>] ? page_fault+0x28/0x30 > [ 546.930199] [<ffffffffa038e31b>] ? igb_reset_q_vector+0xb/0x50 [igb] > [ 546.930199] [<ffffffffa038e40d>] ? > igb_reset_interrupt_capability+0x3d/0x70 [igb] > [ 546.930199] [<ffffffffa0391588>] ? igb_init_interrupt_scheme+0x3d8/0x3f0 > [igb] > [ 546.930199] [<ffffffffa0396b23>] ? igb_sriov_reinit+0x53/0xb0 [igb] > [ 546.930199] [<ffffffffa0396c96>] ? igb_pci_enable_sriov+0x116/0x1e0 [igb] > [ 546.930199] [<ffffffffa0397257>] ? igb_probe+0x4c7/0x11e0 [igb] > [ 546.930199] [<ffffffff812b02da>] ? local_pci_probe+0x3a/0xa0 > [ 546.930199] [<ffffffff812b15ea>] ? pci_device_probe+0xca/0x120 > [ 546.930199] [<ffffffff81365d26>] ? driver_probe_device+0x76/0x220 > [ 546.930199] [<ffffffff81365ed0>] ? driver_probe_device+0x220/0x220 > [ 546.930199] [<ffffffff81364163>] ? bus_for_each_drv+0x53/0x90 > [ 546.930199] [<ffffffff81365c70>] ? device_attach+0x90/0xb0 > [ 546.930199] [<ffffffff812a8474>] ? pci_bus_add_device+0x34/0x60 > [ 546.930199] [<ffffffff812a8859>] ? pci_bus_add_devices+0x39/0xa0 > [ 546.930199] [<ffffffff814b0898>] ? enable_slot+0x278/0x2f0 > [ 546.930199] [<ffffffff812c6588>] ? acpiphp_check_bridge.part.6+0xf8/0x110 > [ 546.930199] [<ffffffff812c6d62>] ? hotplug_event+0x162/0x190 > [ 546.930199] [<ffffffff812c6db0>] ? hotplug_event_work+0x20/0x60 > [ 546.930199] [<ffffffff812e75d1>] ? acpi_hotplug_work_fn+0x13/0x1c > [ 546.930199] [<ffffffff81078b9d>] ? process_one_work+0x16d/0x420 > [ 546.930199] [<ffffffff81079756>] ? worker_thread+0x116/0x3b0 > [ 546.930199] [<ffffffff81079640>] ? rescuer_thread+0x330/0x330 > [ 546.930199] [<ffffffff8107f911>] ? kthread+0xc1/0xe0 > [ 546.930199] [<ffffffff8107f850>] ? kthread_create_on_node+0x180/0x180 > [ 546.930199] [<ffffffff814c718c>] ? ret_from_fork+0x7c/0xb0 > [ 546.930199] [<ffffffff8107f850>] ? kthread_create_on_node+0x180/0x180 > [ 546.930199] Code: 00 00 00 00 65 48 8b 04 25 80 c8 00 00 48 8b 80 c8 03 00 > 00 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 0f 1f 40 00 48 8b 87 c8 03 00 00 <48> > 8b 40 d8 c3 0f 1f 40 00 48 83 ec 08 48 8b b7 c8 03 00 00 ba > [ 546.930199] RIP [<ffffffff8107fe77>] kthread_data+0x7/0x10 > [ 546.930199] RSP <ffff8800000557b0> > [ 546.930199] CR2: ffffffffffffffd8 > [ 546.930199] ---[ end trace 8b1d98daee2f26b9 ]--- > [ 546.930199] Fixing recursive fault but reboot is needed! > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@xxxxxxxxxxxxx > http://lists.xen.org/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.