[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] Launching a PV Centos6 DomU crashing at "kernel BUG at fs/sysfs/group.c:65!; Kernel panic - not syncing: Fatal exception"


While working on building up our Xen server, I've had good success so
far with launching Opensuse DomU's that use the old-style, non-pv-ops

Now, I'm trying to launch a paravirtualized Centos6 DomU guest, and
having a heck of a time with it.

I found this in a Citrix forum

Re: Death of multi-os Virtualization?

        The latest Linux kernels, including the RHEL6 kernel, leverage a
        capability called paravirt_ops -- which means that the kernel is
        enabled right out of the box to work properly in a
        paravirtualized Xen environment (as well as other
        paravirtualization) -- without needing a special "xenified"
        kernel. One of our SEs has taken it and, with a small number of
        simple manual steps, gotten it up and running without a special
        kernel. Red Hat's no longer shipping a Xen hypervisor doesn't
        affect this at all.

So it looks like the CentOS kernel should be a paravirt_ops (same as
pv-ops I think) kernel.

I grabbed it's kernel

 mount -o loop CentOS-6.0-x86_64-netinstall.iso /mnt/Centos6
 cp -a /mnt/Centos6/isolinux/{vmlinuz,initrd.img} /stor/

and wrote a simple config file

 cat /stor/centos6.cfg
  name = 'centos6'
  builder = 'linux'
  kernel = '/stor/vmlinuz'
  ramdisk = '/stor/initrd.img'
  maxmem = 1024
  memory = 1024
  disk = [ 'file:/stor/CentOS-6.0-x86_64-netinstall.iso,hdc:cdrom,r',
  vif = [ '' ]
  vfb = [ 'type=vnc, vncdisplay=1, vnclisten=' ]
  extra = 'textmode=1 xencons=xvc0'
  on_crash = 'destroy'
  on_reboot = 'restart'
  on_shutdown = 'destroy'
  localtime = 0

then launched it using

 xm create -c /stor/centos6.cfg

The process crashes at "kernel BUG at fs/sysfs/group.c:65!;Kernel panic
- not syncing: Fatal exception".  The console output is below.

I'm stumped for now.  I searched online for a working config for a pv
Centos6 DomU, but so far only found configs for older kernels before
this pv-ops generation.

Is there something else I need to get this to work, or some way to debug
further what the problem is?



CONSOLE OUTPUT - "xmcreate -c /stor/centos6.cfg"

Using config file "/stor/centos6.cfg".
Started domain centos6 (id=10)
                              Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-71.el6.x86_64 (mockbuild@xxxxxxxxxxxxxxx) (gcc
version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Fri May 20
03:51:51 BST 2011
Command line: textmode=1 xencons=xvc0 noirqdebug elevator=noop
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
ACPI in unprivileged domain disabled
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000040000000 (usable)
DMI not present or invalid.
last_pfn = 0x40000 max_arch_pfn = 0x400000000
init_memory_mapping: 0000000000000000-0000000040000000
RAMDISK: 01cbf000 - 052bf000
No NUMA configuration found
Faking a node at 0000000000000000-0000000040000000
Bootmem setup node 0 0000000000000000-0000000040000000
  NODE_DATA [0000000000008000 - 000000000003bfff]
  bootmap [000000000003c000 -  0000000000043fff] pages 8
(7 early reservations) ==> bootmem [0000000000 - 0040000000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 -
  #1 [00054c2000 - 00054f1000]   XEN PAGETABLES ==> [00054c2000 -
  #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 -
  #3 [0001000000 - 0001c9eff8]    TEXT DATA BSS ==> [0001000000 -
  #4 [0001cbf000 - 00052bf000]          RAMDISK ==> [0001cbf000 -
  #5 [00052bf000 - 00054c2000]   XEN START INFO ==> [00052bf000 -
  #6 [0000100000 - 00002d0000]          PGTABLE ==> [0000100000 -
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 -> 0x000000a0
    0: 0x00000100 -> 0x00040000
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 2 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
Allocating PCI resources starting at 40000000 (gap: 40000000:c0000000)
Booting paravirtualized kernel on Xen
Xen version: 4.1.2_01-143.1 (preserve-AD)
NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
PERCPU: Embedded 31 pages/cpu @ffff880005527000 s95064 r8192 d23720
pcpu-alloc: s95064 r8192 d23720 u126976 alloc=31*4096
pcpu-alloc: [0] 0 [0] 1
Xen: using vcpu_info placement
Built 1 zonelists in Node order, mobility grouping on.  Total pages:
Policy zone: DMA32
Kernel command line: textmode=1 xencons=xvc0 noirqdebug elevator=noop
IRQ lockup detection disabled
PID hash table entries: 4096 (order: 3, 32768 bytes)
Checking aperture...
No AGP bridge found
AMD-Vi disabled by default: pass amd_iommu=on to enable
Memory: 960752k/1048576k available (4935k kernel code, 388k absent,
87436k reserved, 3927k data, 1220k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:288
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 10485760 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory
installing Xen timer for CPU 0
Detected 2806.450 MHz processor.
Calibrating delay loop (skipped), value calculated using timer
frequency.. 5612.90 BogoMIPS (lpj=2806450)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Performance Events:
no APIC, boot with the "lapic" boot parameter to force-enable it.
no hardware sampling interrupt available.
AMD PMU driver.
... version:                0
... bit width:              48
... generic registers:      4
... value mask:             0000ffffffffffff
... max period:             00007fffffffffff
... fixed-purpose events:   0
... event mask:             000000000000000f
alternatives: switching to unfair spinlock
SMP alternatives: switching to UP code
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20276 entries in 80 pages
installing Xen timer for CPU 1
SMP alternatives: switching to SMP code
Brought up 2 CPUs
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
NET: Registered protocol family 16
PCI: Fatal: No config space access function found
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 55296k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1316458768.773:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 1984
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key B5BD89AA2E85EC04
- User ID: Red Hat, Inc. (Kernel Module GPG key)
- Added public key D4A26C9CCD09BEDA
- User ID: Red Hat Enterprise Linux Driver Update Program
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered (default)
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
pci-stub: invalid id string ""
Console: switching to colour frame buffer device 100x37
console [tty0] enabled
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.0
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
input: Xen Virtual Keyboard as /devices/virtual/input/input1
input: Xen Virtual Pointer as /devices/virtual/input/input2
rtc_cmos: probe of rtc_cmos failed with error -16
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
IMA: No TPM chip found, activating TPM-bypass!
XENBUS: Device with no driver: device/vbd/5632
XENBUS: Device with no driver: device/vbd/51712
XENBUS: Device with no driver: device/vbd/51728
XENBUS: Device with no driver: device/vbd/51744
XENBUS: Device with no driver: device/vif/0
XENBUS: Device with no driver: device/console/0
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Initalizing network drop monitor service
Freeing unused kernel memory: 1220k freed
Write protecting the kernel read-only data: 7176k
------------[ cut here ]------------
kernel BUG at fs/sysfs/group.c:65!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/virtual/block/ram12/dev
Modules linked in: xen_blkfront(+) ipv6 iscsi_ibft pcspkr iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi squashfs cramfs

Modules linked in: xen_blkfront(+) ipv6 iscsi_ibft pcspkr iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi squashfs cramfs
Pid: 16, comm: xenwatch Tainted: G        W  ----------------
2.6.32-71.el6.x86_64 #1
RIP: e030:[<ffffffff811e29d7>]  [<ffffffff811e29d7>]
RSP: e02b:ffff88003eb83c60  EFLAGS: 00010246
RAX: 00000000ffffffef RBX: ffff8800041714a0 RCX: ffff88003eaf5440
RDX: ffffffff8175d160 RSI: 0000000000000000 RDI: ffff880004be8c70
RBP: ffff88003eb83cb0 R08: 0000000000000004 R09: 0000000000000000
R10: 000000000000000f R11: 0000000000000000 R12: ffff880004be8c00
R13: ffff880004be8c70 R14: ffffffff8175d160 R15: 0000000000000000
FS:  00007f5f666c4700(0000) GS:ffff880005527000(0000)
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001e61008 CR3: 0000000004d88000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process xenwatch (pid: 16, threadinfo ffff88003eb82000, task
 ffff88003eb83cc0 000000008132aae1 000000303a323032 ffff88003eb83cd0
<0> ffff88003eb83c90 ffff8800041714a0 ffff880004be8c00 ffff880004be8c60
<0> ffff880004be8c00 0000000000000000 ffff88003eb83cc0 ffffffff811e2ab3
Call Trace:
 [<ffffffff811e2ab3>] sysfs_create_group+0x13/0x20
 [<ffffffff810f8db4>] blk_trace_init_sysfs+0x14/0x20
 [<ffffffff81243bc0>] blk_register_queue+0x40/0x100
 [<ffffffff81249024>] add_disk+0x94/0x160
 [<ffffffffa00983b5>] backend_changed+0x335/0x5f0 [xen_blkfront]
 [<ffffffff8100fb62>] ? check_events+0x12/0x20
 [<ffffffff812ea9fa>] otherend_changed+0xca/0x180
 [<ffffffff812e90ea>] xenwatch_thread+0xaa/0x170
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff814caa8c>] ? _spin_unlock_irqrestore+0x1c/0x20
 [<ffffffff812e9040>] ? xenwatch_thread+0x0/0x170
 [<ffffffff81091936>] kthread+0x96/0xa0
 [<ffffffff810141ca>] child_rip+0xa/0x20
 [<ffffffff81013393>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff81013b1d>] ? retint_restore_args+0x5/0x6
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
Code: 8b 04 24 48 85 c0 74 27 41 83 c7 01 8b 55 bc 85 d2 74 b1 48 8b 30
48 89 df e8 76 be ff ff eb a4 48 83 7f 30 00 0f 85 49 ff ff ff <0f> 0b
eb fe 48 8b 5d c8 31 d2 48 85 db 74 18 f0 ff 0b 0f 94 c0
RIP  [<ffffffff811e29d7>] internal_create_group+0xf7/0x1a0
 RSP <ffff88003eb83c60>
---[ end trace d0430e72decb56f7 ]---
Kernel panic - not syncing: Fatal exception
Pid: 16, comm: xenwatch Tainted: G      D W  ----------------
2.6.32-71.el6.x86_64 #1
Call Trace:
 [<ffffffff814c7b23>] panic+0x78/0x137
 [<ffffffff814cbbf4>] oops_end+0xe4/0x100
 [<ffffffff8101733b>] die+0x5b/0x90
 [<ffffffff814cb4a4>] do_trap+0xc4/0x160
 [<ffffffff81014ee5>] do_invalid_op+0x95/0xb0
 [<ffffffff811e29d7>] ? internal_create_group+0xf7/0x1a0
 [<ffffffff8118798a>] ? ilookup5+0x4a/0x60
 [<ffffffff8100f33d>] ? xen_force_evtchn_callback+0xd/0x10
 [<ffffffff8100fb62>] ? check_events+0x12/0x20
 [<ffffffff81013f5b>] invalid_op+0x1b/0x20
 [<ffffffff811e29d7>] ? internal_create_group+0xf7/0x1a0
 [<ffffffff811e2ab3>] sysfs_create_group+0x13/0x20
 [<ffffffff810f8db4>] blk_trace_init_sysfs+0x14/0x20
 [<ffffffff81243bc0>] blk_register_queue+0x40/0x100
 [<ffffffff81249024>] add_disk+0x94/0x160
 [<ffffffffa00983b5>] backend_changed+0x335/0x5f0 [xen_blkfront]
 [<ffffffff8100fb62>] ? check_events+0x12/0x20
 [<ffffffff812ea9fa>] otherend_changed+0xca/0x180
 [<ffffffff812e90ea>] xenwatch_thread+0xaa/0x170
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff814caa8c>] ? _spin_unlock_irqrestore+0x1c/0x20
 [<ffffffff812e9040>] ? xenwatch_thread+0x0/0x170
 [<ffffffff81091936>] kthread+0x96/0xa0
 [<ffffffff810141ca>] child_rip+0xa/0x20
 [<ffffffff81013393>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff81013b1d>] ? retint_restore_args+0x5/0x6
 [<ffffffff810141c0>] ? child_rip+0x0/0x20

Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.