[Xen-devel] xen/master pvops kernel boot consistently hangs on i7 with xen 3.4/3.5 where 2.6.30.3 git kernel works.

[Bruce Edge <bruce.edge@xxxxxxxxx>]

I'm running out of things to try. I would really appreciate a clue as
to what to look at next. It's not a full lockup as the CTRL-A*3 xen
debug console still works, but the xen/master dom0 shows no signs of
life.

The current 2.6.31.1 pvops  xen/master kernel hangs when running under
Xen on a proliant the a supermicro, both X58 i7 boxes.

I've tried booting with these xen versions (both of which work with a
2.6.30.3 dom0)
   http://xenbits.xensource.com/xen-unstable.hg
   http://xenbits.xensource.com/xen-3.4-testing.hg

And with the 2.6.31.1 source from both (is there a difference?)
   git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
   git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git xen/master

I've attached my .config, but here are the xen related settings:
CONFIG_XEN=y
CONFIG_XEN_TIME_VSYSCALL=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=32
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_XEN_DEBUG_FS=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_DOM0_PCI=y
CONFIG_MICROCODE_XEN=y
CONFIG_PCI_XEN=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_KBDDEV_FRONTEND=m
CONFIG_HVC_XEN=y
CONFIG_XEN_FBDEV_FRONTEND=m
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_S3=y
CONFIG_XEN_GNTDEV=y

I'm building the kernels with make deb-pkg, installing the resultant 2
packages, firmware and kernel, and building the initrd. I assume that
that's no different from running make; make modules_install, etc... on
the target box?

The boot options I'm using are:
title           Xen 3.5-unstable / Ubuntu 9.04, kernel 2.6.31.1 clean
uuid            915b8d5f-dc27-4234-adac-26382dfef7c2
kernel          /xen-3.5-unstable.gz com1=115200,8n1 console=com1
module          /vmlinuz-2.6.31.1
root=UUID=e0a8abaf-ebb4-4a46-a968-36764653b87f ro console=xvc0
module          /initrd.img-2.6.31.1

as well as:

title           Xen 3.5-unstable / Ubuntu 9.04, kernel 2.6.31.1
uuid            915b8d5f-dc27-4234-adac-26382dfef7c2
kernel          /xen-3.5-unstable.gz com1=115200,8n1 console=com1
acpi=force apic=on dom0_max_vcpus=1 dom0_vcpus_pin=true
iommu=1,no-intremap,passthrough loglvl=all loglvl_guest=all
module          /vmlinuz-2.6.31.1
root=UUID=e0a8abaf-ebb4-4a46-a968-36764653b87f ro console=xvc0
pciback.hide=(07:00.0)(07:00.1)(07:00.2)(07:00.3)(14:00.0)(14:00.1)(17:00.0)(17:00.1)
acpi=force
module          /initrd.img-2.6.31.1

and everything in between.

I've tried booting the xen/master 2.6.31.1 kernel on bare metal
instead of under xen and there's zero console output.

Should this work, or does this kernel _only_ run under Xen?

I can boot both xen versions, 3.5-unstable and 3.4-testing with a
2.6.30.3 dom0. This _does_ work, but it uses the older kernel:

title           Xen 3.5-unstable / Ubuntu 9.04, kernel 2.6.30.3
uuid            915b8d5f-dc27-4234-adac-26382dfef7c2
kernel          /xen-3.5-unstable.gz com1=115200,8n1 console=com1
acpi=force apic=on dom0_max_vcpus=1 dom0_vcpus_pin=true
iommu=1,no-intremap,passthrough loglvl=all loglvl_guest=all
module          /vmlinuz-2.6.30.3
root=UUID=e0a8abaf-ebb4-4a46-a968-36764653b87f ro console=xvc0
pciback.hide=(07:00.0)(07:00.1)(07:00.2)(07:00.3)(14:00.0)(14:00.1)(17:00.0)(17:00.1)
acpi=force
module          /initrd.img-2.6.30.3

I've tried toggling these settings in the BIOS with no effect:
- hyperthreading
- exec memory protection
- disabling all unneeded PCI cards
- APIC

Any other BIOS items I should tweak to test?

This is the tail of the console output before the hang

(XEN) PCI add device 17:00.0
(XEN) PCI add device 17:00.1
(XEN) PCI add device 07:00.0
(XEN) PCI add device 07:00.1
(XEN) PCI add device 07:00.2
(XEN) PCI add device 07:00.3
(XEN) PCI add device 02:00.0
(XEN) PCI add device 02:00.1
(XEN) PCI add device 03:00.0
(XEN) PCI add device 03:00.1
(XEN) PCI add device 01:03.0
(XEN) PCI add device 01:04.0
(XEN) PCI add device 01:04.2
(XEN) PCI add device 01:04.4
(XEN) PCI add device 01:04.6
(XEN) io_apic.c:2208:
(XEN) ioapic_guest_write: apic=0, pin=20, irq=20
(XEN) ioapic_guest_write: new_entry=0001a014
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) io_apic.c:2208:
(XEN) ioapic_guest_write: apic=0, pin=23, irq=23
(XEN) ioapic_guest_write: new_entry=0001a017
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) io_apic.c:2208:
(XEN) ioapic_guest_write: apic=0, pin=22, irq=22
(XEN) ioapic_guest_write: new_entry=0001a016
(XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) *** Serial input -> Xen (type 'CTRL-a' three times to switch
input to DOM0)

The CTRL-a does still work. What output from that can I post to help debug this?

I've already posted the full boot output so I'll resist the temptation
to include it again.

-Bruce

Attachment: xen.config
Description: Binary data

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel