[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] Re: 2.6.37-rc1 mainline domU - BUG: unable to handle kernel paging request


  • To: xen-devel@xxxxxxxxxxxxxxxxxxx, Jeremy Fitzhardinge <jeremy@xxxxxxxx>, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
  • From: Bruce Edge <bruce.edge@xxxxxxxxx>
  • Date: Thu, 11 Nov 2010 07:08:03 -0800
  • Cc:
  • Delivery-date: Thu, 11 Nov 2010 07:09:38 -0800
  • Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=BBJ0ionDLeA5M+4GxUxRnI6Je7bOOV9po6uxeTz2HhSfGC/AwKiwWzL52iSNZva+hG ANqo2lvNQQw4I1Kt2KZI7aFN6XRqb1DTD+cczpjjtX5bYOdU0UdI6vBYrxxbJTVSkFc6 ouWNCayaxG+EFL1oDWOXXXHfTaABYjomprxSw=
  • List-id: Xen developer discussion <xen-devel.lists.xensource.com>


On Wed, Nov 10, 2010 at 2:30 PM, Bruce Edge <bruce.edge@xxxxxxxxx> wrote:
On Wed, Nov 10, 2010 at 2:15 PM, Bruce Edge <bruce.edge@xxxxxxxxx> wrote:
>
> This is a mainline 2.6.37-rc1 pvops domU on top of a xen-unstable build 22241 and xen/stable-2.6.32.x 2.6.32.24 pvops dom0.
>
> The domU console output hung up prior to spitting out a prompt post root login. I logged into the domU via ssh and ran an 'ls' of an NFS mounted dir and
> this immediately came out on the domU console:
>
> [  480.467525] BUG: unable to handle kernel paging request at ffff880077e9f030
> [  480.467547] IP: [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
> [  480.467564] PGD 1a04067 PUD 3dd7067 PMD 3f97067 PTE 8010000077e9f065
> [  480.467587] Oops: 0003 [#1] SMP
> [  480.467601] last sysfs file: /sys/kernel/uevent_seqnum
> [  480.467611] CPU 3
> [  480.467616] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
> [  480.467676]
> [  480.467684] Pid: 3957, comm: zsh Not tainted 2.6.37-rc1-kernel.org-domu-6.0.bedge-debug #1 /
> [  480.467695] RIP: e030:[<ffffffff810355f3>]  [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
> [  480.467710] RSP: e02b:ffff880077ebfbd8  EFLAGS: 00010202
> [  480.467719] RAX: 800000004d686167 RBX: ffff8800763ddb80 RCX: 800000004d686167
> [  480.467729] RDX: ffff880077e9f030 RSI: 00007f5cb26069d0 RDI: ffff8800763ddb80
> [  480.467739] RBP: ffff880077ebfc08 R08: 0000000000000001 R09: ffffea0001a3b2d8
> [  480.467748] R10: 0000000000000100 R11: 0000000000000000 R12: 00007f5cb26069d0
> [  480.467759] R13: 0000000000000001 R14: 0000000000000000 R15: ffff8800763ddb80
> [  480.467772] FS:  00007f5cb2606700(0000) GS:ffff88007ff94000(0000) knlGS:0000000000000000
> [  480.467783] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [  480.467792] CR2: ffff880077e9f030 CR3: 0000000077dbc000 CR4: 0000000000002660
> [  480.467802] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [  480.467812] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [  480.467822] Process zsh (pid: 3957, threadinfo ffff880077ebe000, task ffff880078b80000)
> [  480.467832] Stack:
> [  480.467838]  0000000000000100 ffffea0001a3b2d8 7000000000000000 0000000000000000
> [  480.467862]  ffff880077e9f030 ffffea0001a3b2d8 ffff880077ebfca8 ffffffff810e2200
> [  480.467886]  ffff880077ebfc38 ffffffff81007c09 ffffea0001a3b2d8 0000000000000001
> [  480.467910] Call Trace:
> [  480.467921]  [<ffffffff810e2200>] do_wp_page+0x2d8/0x6be
> [  480.467934]  [<ffffffff81007c09>] ? __spin_time_accum+0x21/0x37
> [  480.467946]  [<ffffffff81007e85>] ? __xen_spin_lock+0xb7/0xcd
> [  480.467958]  [<ffffffff810e2e60>] handle_mm_fault+0x87a/0x8c1
> [  480.467972]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
> [  480.467984]  [<ffffffff81006c3d>] ? xen_force_evtchn_callback+0xd/0xf
> [  480.467996]  [<ffffffff81006c3d>] ? xen_force_evtchn_callback+0xd/0xf
> [  480.468009]  [<ffffffff810072a2>] ? check_events+0x12/0x20
> [  480.468020]  [<ffffffff8100728f>] ? xen_restore_fl_direct_end+0x0/0x1
> [  480.468034]  [<ffffffff810a6fa4>] ? __call_rcu+0x123/0x12c
> [  480.468045]  [<ffffffff81006c3d>] ? xen_force_evtchn_callback+0xd/0xf
> [  480.468059]  [<ffffffff810072a2>] ? check_events+0x12/0x20
> [  480.468072]  [<ffffffff81443615>] page_fault+0x25/0x30
> [  480.468085]  [<ffffffff81206add>] ? __put_user_4+0x1d/0x30
> [  480.468096]  [<ffffffff810498a6>] ? schedule_tail+0x5c/0x60
> [  480.468108]  [<ffffffff8100a9c3>] ret_from_fork+0x13/0x80
> [  480.468116] Code: 41 5d c9 c3 55 48 89 e5 41 55 45 31 ed 41 54 49 89 f4 53 48 89 fb 48 83 ec 18 48 39 0a 41 0f 95 c5 45 85 c0 74 1d 45 85 ed 74 18 <48> 89 0a 48 8b 3f 0f 1f 80 00 00 00 00 4c 89 e6 48 89 df e8 b2
> [  480.468323] RIP  [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
> [  480.468323]  RSP <ffff880077ebfbd8>
> [  480.468323] CR2: ffff880077e9f030
> [  480.468323] ---[ end trace 6d1623321ecd50ac ]---
>
> The domU config is attached as config.2.6.37_domU_debug
>
>
> -Bruce

Here's another attempt from the same config with the full domU boot logs:

Started domain dpm (id=4)
                        [    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version
2.6.37-rc1-kernel.org-domu-6.0.bedge-debug (root@opal) (gcc version
4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #1 SMP Tue Nov 9 20:26:03 PST 2010
[    0.000000] Command line:
root=UUID=4d87e4cb-8317-41d4-aea2-ac2ebee699f7 ro   swiotlb=force
noirqbalance noirqbalance console=hvc0,115200n8 --flow_control=r
ip=:127.0.255.255::::eth0:dhcp
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] released 0 pages of unused memory
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000080800000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x80800 max_arch_pfn = 0x400000000
[    0.000000] Scanning 0 areas for low memory corruption
[    0.000000] init_memory_mapping: 0000000000000000-0000000080800000
[    0.000000] RAMDISK: 01d95000 - 02fa9000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at 0000000000000000-0000000080800000
[    0.000000] Initmem setup node 0 0000000000000000-0000000080800000
[    0.000000]   NODE_DATA [000000007fffb000 - 000000007fffffff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   empty
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x00080800
[    0.000000] SMP: Allowing 6 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] Allocating PCI resources starting at 80800000 (gap:
80800000:7f800000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.1-unstable (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32
nr_cpu_ids:6 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88007ff40000 s82752
r8192 d23744 u114688
[    0.000000] pcpu-alloc: s82752 r8192 d23744 u114688 alloc=28*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.
Total pages: 519022
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line:
root=UUID=4d87e4cb-8317-41d4-aea2-ac2ebee699f7 ro   swiotlb=force
noirqbalance noirqbalance console=hvc0,115200n8 --flow_control=r
ip=:127.0.255.255::::eth0:dhcp
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Placing 64MB software IO TLB between ffff880079400000 -
ffff88007d400000
[    0.000000] software IO TLB at phys 0x79400000 - 0x7d400000
[    0.000000] Memory: 1959896k/2105344k available (4398k kernel code,
448k absent, 145000k reserved, 6327k data, 756k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0,
CPUs=6, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU-based detection of stalled CPUs is disabled.
[    0.000000] NR_IRQS:4352 nr_irqs:320 16
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] installing Xen timer for CPU 0
[    0.000000] Detected 2660.064 MHz processor.
[    0.000999] Calibrating delay loop (skipped), value calculated
using timer frequency.. 5320.12 BogoMIPS (lpj=2660064)
[    0.000999] pid_max: default: 32768 minimum: 301
[    0.000999] Security Framework initialized
[    0.000999] SELinux:  Initializing.
[    0.000999] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.001408] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.001635] Mount-cache hash table entries: 256
[    0.002195] Initializing cgroup subsys ns
[    0.002208] ns_cgroup deprecated: consider using the
'clone_children' flag without the ns_cgroup.
[    0.002220] Initializing cgroup subsys cpuacct
[    0.002231] Initializing cgroup subsys freezer
[    0.002356] CPU: Unsupported number of siblings 16
[    0.002362] Performance Events: unsupported p6 CPU model 26 no PMU
driver, software events only.
[    0.002483] SMP alternatives: switching to UP code
[    0.004119] cpu 0 spinlock event irq 318
[    0.004765] NMI watchdog failed to create perf event on cpu0:
fffffffffffffffe
[    0.005050] installing Xen timer for CPU 1
[    0.005157] cpu 1 spinlock event irq 312
[    0.005194] SMP alternatives: switching to SMP code
[    0.000999] CPU: Unsupported number of siblings 16
[    0.006158] NMI watchdog failed to create perf event on cpu1:
fffffffffffffffe
[    0.006474] installing Xen timer for CPU 2
[    0.006585] cpu 2 spinlock event irq 306
[    0.000999] CPU: Unsupported number of siblings 16
[    0.006816] NMI watchdog failed to create perf event on cpu2:
fffffffffffffffe
[    0.006999] installing Xen timer for CPU 3
[    0.007037] cpu 3 spinlock event irq 300
[    0.000999] CPU: Unsupported number of siblings 16
[    0.007300] NMI watchdog failed to create perf event on cpu3:
fffffffffffffffe
[    0.007534] installing Xen timer for CPU 4
[    0.007636] cpu 4 spinlock event irq 294
[    0.000999] CPU: Unsupported number of siblings 16
[    0.007874] NMI watchdog failed to create perf event on cpu4:
fffffffffffffffe
[    0.008041] installing Xen timer for CPU 5
[    0.008083] cpu 5 spinlock event irq 288
[    0.000999] CPU: Unsupported number of siblings 16
[    0.008344] NMI watchdog failed to create perf event on cpu5:
fffffffffffffffe
[    0.008356] Brought up 6 CPUs
[    0.009181] kworker/u:0 used greatest stack depth: 5696 bytes left
[    0.012051] Grant table initialized
[    0.030978] Time: 165:165:165  Date: 165/165/65
[    0.031229] NET: Registered protocol family 16
[    0.036672] PCI: setting up Xen PCI frontend stub
[    0.037225] kworker/u:0 used greatest stack depth: 5528 bytes left
[    0.074302] bio: create slab <bio-0> at 0
[    0.075385] ACPI: Interpreter disabled.
[    0.077056] xen_balloon: Initialising balloon driver.
[    0.079323] vgaarb: loaded
[    0.081270] usbcore: registered new interface driver usbfs
[    0.081283] usbcore: registered new interface driver hub
[    0.081283] usbcore: registered new device driver usb
[    0.083183] PCI: System does not support PCI
[    0.084007] PCI: System does not support PCI
[    0.085236] NetLabel: Initializing
[    0.085236] NetLabel:  domain hash size = 128
[    0.085236] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.085313] NetLabel:  unlabeled traffic allowed by default
[    0.086544] Switching to clocksource xen
[    0.106256] pnp: PnP ACPI: disabled
[    0.129799] NET: Registered protocol family 2
[    0.130174] IP route cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.131882] TCP established hash table entries: 262144 (order: 10,
4194304 bytes)
[    0.132891] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.133179] TCP: Hash tables configured (established 262144 bind 65536)
[    0.133191] TCP reno registered
[    0.133301] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[    0.133407] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[    0.133914] NET: Registered protocol family 1
[    0.134582] RPC: Registered udp transport module.
[    0.134590] RPC: Registered tcp transport module.
[    0.134595] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.134892] Trying to unpack rootfs image as initramfs...
[    0.178824] Freeing initrd memory: 18512k freed
[    0.268798] DMA-API: preallocated 32768 debug entries
[    0.268810] DMA-API: debugging enabled by kernel config
[    0.271181] platform rtc_cmos: registered platform RTC device (no
PNP device found)
[    0.285410] Machine check injector initialized
[    0.294732] microcode: CPU0 sig=0x106a4, pf=0x2, revision=0xa
[    0.294803] microcode: CPU1 sig=0x106a4, pf=0x2, revision=0xa
[    0.294893] microcode: CPU2 sig=0x106a4, pf=0x2, revision=0xa
[    0.294944] microcode: CPU3 sig=0x106a4, pf=0x2, revision=0xa
[    0.294998] microcode: CPU4 sig=0x106a4, pf=0x2, revision=0xa
[    0.295043] microcode: CPU5 sig=0x106a4, pf=0x2, revision=0xa
[    0.295590] microcode: Microcode Update Driver: v2.00
<tigran@xxxxxxxxxxxxxxxxxxxx>, Peter Oruba
[    0.295607] Scanning for low memory corruption every 60 seconds
[    0.298813] audit: initializing netlink socket (disabled)
[    0.298940] type=2000 audit(1289403082.795:1): initialized
[    0.333762] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.376985] VFS: Disk quotas dquot_6.5.2
[    0.377853] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.383121] NTFS driver 2.1.29 [Flags: R/W].
[    0.385014] msgmni has been set to 3864
[    0.389953] Block layer SCSI generic (bsg) driver version 0.4
loaded (major 253)
[    0.389970] io scheduler noop registered
[    0.389979] io scheduler deadline registered
[    0.390666] io scheduler cfq registered (default)
[    0.392486] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.406238] pcifront pci-0: Installing PCI frontend
[    0.406967] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    0.417634] pcifront pci-0: claiming resource 0000:00:00.0/0
[    0.417634] pcifront pci-0: claiming resource 0000:00:00.0/2
[    0.417634] pcifront pci-0: claiming resource 0000:00:00.0/3
[    0.417634] pcifront pci-0: claiming resource 0000:00:00.0/4
[    0.417634] pcifront pci-0: claiming resource 0000:00:01.0/0
[    0.417634] pcifront pci-0: claiming resource 0000:00:01.0/2
[    0.417634] pcifront pci-0: claiming resource 0000:00:01.0/3
[    0.417634] pcifront pci-0: claiming resource 0000:00:01.0/4
[    0.417634] pcifront pci-0: claiming resource 0000:00:02.0/0
[    0.417634] pcifront pci-0: claiming resource 0000:00:02.0/2
[    0.417634] pcifront pci-0: claiming resource 0000:00:02.0/3
[    0.417634] pcifront pci-0: claiming resource 0000:00:02.0/4
[    0.417634] pcifront pci-0: claiming resource 0000:00:03.0/0
[    0.417634] pcifront pci-0: claiming resource 0000:00:03.0/2
[    0.417634] pcifront pci-0: claiming resource 0000:00:03.0/3
[    0.417634] pcifront pci-0: claiming resource 0000:00:03.0/4
[    0.731775] Non-volatile memory driver v1.3
[    0.731790] Linux agpgart interface v0.103
[    0.733731] [drm] Initialized drm 1.1.0 20060810
[    0.733752] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.827202] brd: module loaded
[    0.840788] loop: module loaded
[    0.841965] Fixed MDIO Bus: probed
[    0.844025] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.844651] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.845194] uhci_hcd: USB Universal Host Controller Interface driver
[    0.846985] usbcore: registered new interface driver usblp
[    0.847763] usbcore: registered new interface driver libusual
[    0.849433] PNP: No PS/2 controller found. Probing ports directly.
[    0.850279] i8042.c: No controller found.
[    0.850883] mice: PS/2 mouse device common for all mice
[    0.853446] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.854066] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.855928] cpuidle: using governor ladder
[    0.855941] cpuidle: using governor menu
[    0.856029] Netfilter messages via NETLINK v0.30.
[    0.856136] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    0.857888] ctnetlink v0.93: registering with nfnetlink.
[    0.860285] ip_tables: (C) 2000-2006 Netfilter Core Team
[    0.860677] TCP cubic registered
[    0.860687] Initializing XFRM netlink socket
[    0.863286] NET: Registered protocol family 10
[    0.867216] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    0.867612] IPv6 over IPv4 tunneling driver
[    0.871326] NET: Registered protocol family 17
[    0.871566] Registering the dns_resolver key type
[    0.873668] registered taskstats version 1
[    0.873912] kmemleak: Kernel memory leak detector initialized
[    0.873928] kmemleak: Automatic memory scanning thread started
[    0.875057] XENBUS: Device with no driver: device/vbd/51712
[    0.875057] XENBUS: Device with no driver: device/vif/0
[    0.875057] XENBUS: Device with no driver: device/console/0
[    0.875057]   Magic number: 1:252:3141
[   13.388958] Freeing unused kernel memory: 756k freed
[   13.389190] Write protecting the kernel read-only data: 10240k
[   13.398774] Freeing unused kernel memory: 1724k freed
[   13.400663] Freeing unused kernel memory: 1992k freed
[   13.494573] all_generic_ide used greatest stack depth: 5472 bytes left
[   13.519388] udevd (1111): /proc/1111/oom_adj is deprecated, please
use /proc/1111/oom_score_adj instead.
Begin: Loading essential drivers... ...
[   13.559435] blkid used greatest stack depth: 5408 bytes left
[   13.568391] blkfront: xvda: barriers disabled
[   13.574762] blkid used greatest stack depth: 5264 bytes left
[   13.580813]  xvda: xvda1 xvda2 < xvda5 >
[   13.645562] modprobe used greatest stack depth: 5088 bytes left
[   13.653160] Initialising Xen virtual ethernet driver.
[   13.659540] blkid used greatest stack depth: 4960 bytes left
[   13.735467] blkid used greatest stack depth: 4752 bytes left
Done.
Begin: Running /scripts/init-premount ...
Done.
Begin: Mounting root file system... ...
Begin: Running /scripts/local-top ...
Done.
Begin: Running /scripts/local-premount ...
Done.
[   13.903774] EXT4-fs (xvda1): INFO: recovery required on readonly filesystem
[   13.903799] EXT4-fs (xvda1): write access will be enabled during recovery
[   14.033719] EXT4-fs (xvda1): recovery complete
[   14.034966] EXT4-fs (xvda1): mounted filesystem with ordered data
mode. Opts: (null)
[   14.035510] exe used greatest stack depth: 4112 bytes left
Begin: Running /scripts/local-bottom ...
Done.
Done.
Begin: Running /scripts/init-bottom ...
Done.
init: ureadahead main process (1214) terminated with status 5
fsck from util-linux-ng 2.17.2
e2fsck 1.41.11 (14-Mar-2010)
/dev/xvda1: clean, 56314/288576 files, 287743/1153280 blocks
 * Enable login on console                                               [ OK ]
 * Check for hostname changes...                                         [ OK ]
  ...done.
 * Setting sensors limits
        No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.
No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.
                                                                        [ OK ]
 * Initializing random number generator...                               [ OK ]
 * Setting up X server socket directory /tmp/.X11-unix...                [ OK ]
 * Setting up ICE socket directory /tmp/.ICE-unix...                     [ OK ]
 * Not starting fancontrol; run pwmconfig first.
 * Starting internet superserver inetd                                   [ OK ]
Starting openntpd: ntpd.
 * Starting network management services:
 * Setting core file location...                                         [ OK ]
 * Restoring resolver state...                                           [ OK ]
 * Checking tomcat configuration
  ...done.
 * Starting Tomcat servlet engine tomcat6
        Using CATALINA_BASE:   /var/lib/tomcat6
Using CATALINA_HOME:   /usr/share/tomcat6
Using CATALINA_TMPDIR: /tmp/tomcat6-tmp
Using JRE_HOME:        /usr/lib/jvm/java-6-openjdk
Using CLASSPATH:       /usr/share/tomcat6/bin/bootstrap.jar
                                                                        [ OK ]
collectd has been disabled - see /etc/default/collectd.
 * Recording successful boot for GRUB
  ...done.
 * Running local boot scripts (/etc/rc.local)                            [ OK ]
mountall: Plymouth command failed
mountall: Disconnected from Plymouth

6.0.16-01
kaan-04-dpm login: root
Password:
[   42.646538] ------------[ cut here ]------------
[   42.646556] kernel BUG at arch/x86/xen/mmu.c:1831!
[   42.646566] invalid opcode: 0000 [#1] SMP
[   42.646582] last sysfs file: /sys/kernel/uevent_seqnum
[   42.646592] CPU 0
[   42.646598] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront
fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront
xen_netfront xen_blkfront
[   42.646665]
[   42.646674] Pid: 3916, comm: lsb_release Not tainted
2.6.37-rc1-kernel.org-domu-6.0.bedge-debug #1 /
[   42.646688] RIP: e030:[<ffffffff81004bce>]  [<ffffffff81004bce>]
pin_pagetable_pfn+0x31/0x37
[   42.646711] RSP: e02b:ffff88007595bce8  EFLAGS: 00010282
[   42.646721] RAX: 00000000ffffffea RBX: 000000000007751e RCX: 0000000000000001
[   42.646732] RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 00000000deadbeef
[   42.646744] RBP: ffff88007595bd08 R08: 00003ffffffff000 R09: ffff880000000000
[   42.646756] R10: 00000000deadbeef R11: dead000000200200 R12: 0000000000000003
[   42.646768] R13: ffff880075c90150 R14: ffff8800781d9398 R15: ffff880075161d40
[   42.646784] FS:  00007fee63b01700(0000) GS:ffff88007ff40000(0000)
knlGS:0000000000000000
[   42.646797] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[   42.646807] CR2: 00007faac54182b8 CR3: 00000000771dd000 CR4: 0000000000002660
[   42.646819] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   42.646832] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   42.646844] Process lsb_release (pid: 3916, threadinfo
ffff88007595a000, task ffff880075161d40)
[   42.646857] Stack:
[   42.646864]  ffff880000000000 00000000000484f0 ffff8800777c6d80
000000000007751e
[   42.646892]  ffff88007595bd28 ffffffff810066a8 000000000007751e
ffff8800777c6d80
[   42.646919]  ffff88007595bd38 ffffffff810066db ffff88007595bd78
ffffffff810e048f
[   42.646946] Call Trace:
[   42.646959]  [<ffffffff810066a8>] xen_alloc_ptpage+0x68/0x6d
[   42.646973]  [<ffffffff810066db>] xen_alloc_pte+0xe/0x10
[   42.646989]  [<ffffffff810e048f>] __pte_alloc+0x72/0xd7
[   42.647003]  [<ffffffff810e2717>] handle_mm_fault+0x131/0x8c1
[   42.647017]  [<ffffffff810e6e36>] ? mmap_region+0x391/0x4bf
[   42.647033]  [<ffffffff8100eb84>] ? arch_get_unmapped_area_topdown+0x4a/0x28a
[   42.647052]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
[   42.647059]  [<ffffffff8102b78a>] ? pvclock_clocksource_read+0x4b/0xb4
[   42.647059]  [<ffffffff81007c09>] ? __spin_time_accum+0x21/0x37
[   42.647059]  [<ffffffff81007e85>] ? __xen_spin_lock+0xb7/0xcd
[   42.647059]  [<ffffffff81443615>] page_fault+0x25/0x30
[   42.647059] Code: ec 20 89 7d e0 48 89 f7 e8 c0 ff ff ff 48 8d 7d
e0 48 89 45 e8 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 76 c7 ff ff
85 c0 74 04 <0f> 0b eb fe c9 c3 55 48 89 e5 41 54 49 89 fc 53 31 db 48
8b 3d
[   42.647059] RIP  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[   42.647059]  RSP <ffff88007595bce8>
[   42.647059] ---[ end trace 3fbbffee414a5166 ]---


Here's another full boot that I just let sit at the login prompt. This is using the pcifront-fixes kernel:

Started domain dpm (id=1)
                         [    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.37-rc1-pvops-pcifront-fixes-domu-6.0.bedge-debug (root@opal) (gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #1 SMP Wed Nov 10 17:11:41 PST 2010
[    0.000000] Command line: root=UUID=1e67e680-4108-4e11-8bc4-844dad448902 ro   swiotlb=force noirqbalance noirqbalance console=hvc0,115200n8 --flow_control=r ip=:127.0.255.255::::eth0:dhcp
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] released 0 pages of unused memory
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000080800000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x80800 max_arch_pfn = 0x400000000
[    0.000000] Scanning 0 areas for low memory corruption
[    0.000000] init_memory_mapping: 0000000000000000-0000000080800000
[    0.000000] RAMDISK: 01d95000 - 02db9000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at 0000000000000000-0000000080800000
[    0.000000] Initmem setup node 0 0000000000000000-0000000080800000
[    0.000000]   NODE_DATA [000000007fffb000 - 000000007fffffff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   empty
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x00080800
[    0.000000] SMP: Allowing 6 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] Allocating PCI resources starting at 80800000 (gap: 80800000:7f800000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.1-unstable (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:6 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88007ff40000 s82752 r8192 d23744 u114688
[    0.000000] pcpu-alloc: s82752 r8192 d23744 u114688 alloc=28*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 519022
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: root=UUID=1e67e680-4108-4e11-8bc4-844dad448902 ro   swiotlb=force noirqbalance noirqbalance console=hvc0,115200n8 --flow_control=r ip=:127.0.255.255::::eth0:dhcp
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Placing 64MB software IO TLB between ffff880079400000 - ffff88007d400000
[    0.000000] software IO TLB at phys 0x79400000 - 0x7d400000
[    0.000000] Memory: 1961880k/2105344k available (4398k kernel code, 448k absent, 143016k reserved, 6327k data, 756k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=6, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU-based detection of stalled CPUs is disabled.
[    0.000000] NR_IRQS:4352 nr_irqs:320 16
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] installing Xen timer for CPU 0
[    0.000000] Detected 2660.054 MHz processor.
[    0.000999] Calibrating delay loop (skipped), value calculated using timer frequency.. 5320.10 BogoMIPS (lpj=2660054)
[    0.000999] pid_max: default: 32768 minimum: 301
[    0.000999] Security Framework initialized
[    0.000999] SELinux:  Initializing.
[    0.000999] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.001410] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.001641] Mount-cache hash table entries: 256
[    0.002191] Initializing cgroup subsys ns
[    0.002204] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[    0.002216] Initializing cgroup subsys cpuacct
[    0.002227] Initializing cgroup subsys freezer
[    0.002354] CPU: Unsupported number of siblings 16
[    0.002361] Performance Events: unsupported p6 CPU model 26 no PMU driver, software events only.
[    0.002524] SMP alternatives: switching to UP code
[    0.004910] cpu 0 spinlock event irq 318
[    0.005490] NMI watchdog failed to create perf event on cpu0: fffffffffffffffe
[    0.005817] installing Xen timer for CPU 1
[    0.005857] cpu 1 spinlock event irq 312
[    0.005893] SMP alternatives: switching to SMP code
[    0.000999] CPU: Unsupported number of siblings 16
[    0.007158] NMI watchdog failed to create perf event on cpu1: fffffffffffffffe
[    0.007474] installing Xen timer for CPU 2
[    0.007585] cpu 2 spinlock event irq 306
[    0.000999] CPU: Unsupported number of siblings 16
[    0.007909] NMI watchdog failed to create perf event on cpu2: fffffffffffffffe
[    0.008078] installing Xen timer for CPU 3
[    0.008193] cpu 3 spinlock event irq 300
[    0.000999] CPU: Unsupported number of siblings 16
[    0.008450] NMI watchdog failed to create perf event on cpu3: fffffffffffffffe
[    0.008685] installing Xen timer for CPU 4
[    0.008789] cpu 4 spinlock event irq 294
[    0.000999] CPU: Unsupported number of siblings 16
[    0.008999] NMI watchdog failed to create perf event on cpu4: fffffffffffffffe
[    0.009210] installing Xen timer for CPU 5
[    0.009252] cpu 5 spinlock event irq 288
[    0.000999] CPU: Unsupported number of siblings 16
[    0.009530] NMI watchdog failed to create perf event on cpu5: fffffffffffffffe
[    0.009541] Brought up 6 CPUs
[    0.010295] kworker/u:0 used greatest stack depth: 5696 bytes left
[    0.013138] Grant table initialized
[    0.032051] Time: 165:165:165  Date: 165/165/65
[    0.032378] NET: Registered protocol family 16
[    0.037699] PCI: setting up Xen PCI frontend stub
[    0.039132] kworker/u:0 used greatest stack depth: 5528 bytes left
[    0.067272] bio: create slab <bio-0> at 0
[    0.068203] ACPI: Interpreter disabled.
[    0.069064] xen_balloon: Initialising balloon driver.
[    0.070332] vgaarb: loaded
[    0.072248] usbcore: registered new interface driver usbfs
[    0.072279] usbcore: registered new interface driver hub
[    0.073080] usbcore: registered new device driver usb
[    0.075072] PCI: System does not support PCI
[    0.075137] PCI: System does not support PCI
[    0.076112] NetLabel: Initializing
[    0.076120] NetLabel:  domain hash size = 128
[    0.076126] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.076223] NetLabel:  unlabeled traffic allowed by default
[    0.077209] Switching to clocksource xen
[    0.093954] pnp: PnP ACPI: disabled
[    0.138850] NET: Registered protocol family 2
[    0.139329] IP route cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.141816] TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
[    0.143253] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.143474] TCP: Hash tables configured (established 262144 bind 65536)
[    0.143487] TCP reno registered
[    0.143537] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[    0.143593] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[    0.144245] NET: Registered protocol family 1
[    0.145189] RPC: Registered udp transport module.
[    0.145203] RPC: Registered tcp transport module.
[    0.145213] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.145645] Trying to unpack rootfs image as initramfs...
[    0.186054] Freeing initrd memory: 16528k freed
[    0.270158] DMA-API: preallocated 32768 debug entries
[    0.270170] DMA-API: debugging enabled by kernel config
[    0.272584] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.285338] Machine check injector initialized
[    0.294148] microcode: CPU0 sig=0x106a4, pf=0x2, revision=0xa
[    0.294217] microcode: CPU1 sig=0x106a4, pf=0x2, revision=0xa
[    0.294308] microcode: CPU2 sig=0x106a4, pf=0x2, revision=0xa
[    0.294359] microcode: CPU3 sig=0x106a4, pf=0x2, revision=0xa
[    0.294427] microcode: CPU4 sig=0x106a4, pf=0x2, revision=0xa
[    0.294473] microcode: CPU5 sig=0x106a4, pf=0x2, revision=0xa
[    0.294990] microcode: Microcode Update Driver: v2.00 <tigran@xxxxxxxxxxxxxxxxxxxx>, Peter Oruba
[    0.295008] Scanning for low memory corruption every 60 seconds
[    0.297474] audit: initializing netlink socket (disabled)
[    0.297559] type=2000 audit(1289430805.716:1): initialized
[    0.331826] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.375119] VFS: Disk quotas dquot_6.5.2
[    0.375896] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.381318] NTFS driver 2.1.29 [Flags: R/W].
[    0.383311] msgmni has been set to 3864
[    0.387906] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.387924] io scheduler noop registered
[    0.387933] io scheduler deadline registered
[    0.388724] io scheduler cfq registered (default)
[    0.390543] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.403338] pcifront pci-0: Installing PCI frontend
[    0.404082] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    0.406134] pcifront pci-0: claiming resource 0000:00:00.0/0
[    0.406134] pcifront pci-0: claiming resource 0000:00:00.0/2
[    0.406134] pcifront pci-0: claiming resource 0000:00:00.0/3
[    0.406134] pcifront pci-0: claiming resource 0000:00:00.0/4
[    0.406134] pcifront pci-0: claiming resource 0000:00:01.0/0
[    0.406134] pcifront pci-0: claiming resource 0000:00:01.0/2
[    0.406134] pcifront pci-0: claiming resource 0000:00:01.0/3
[    0.406134] pcifront pci-0: claiming resource 0000:00:01.0/4
[    0.406134] pcifront pci-0: claiming resource 0000:00:02.0/0
[    0.406134] pcifront pci-0: claiming resource 0000:00:02.0/2
[    0.406134] pcifront pci-0: claiming resource 0000:00:02.0/3
[    0.406134] pcifront pci-0: claiming resource 0000:00:02.0/4
[    0.406134] pcifront pci-0: claiming resource 0000:00:03.0/0
[    0.406134] pcifront pci-0: claiming resource 0000:00:03.0/2
[    0.406134] pcifront pci-0: claiming resource 0000:00:03.0/3
[    0.406134] pcifront pci-0: claiming resource 0000:00:03.0/4
[    0.734074] Non-volatile memory driver v1.3
[    0.734089] Linux agpgart interface v0.103
[    0.735535] [drm] Initialized drm 1.1.0 20060810
[    0.735556] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.829048] brd: module loaded
[    0.842342] loop: module loaded
[    0.843465] Fixed MDIO Bus: probed
[    0.845827] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.846398] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.846882] uhci_hcd: USB Universal Host Controller Interface driver
[    0.848574] usbcore: registered new interface driver usblp
[    0.849103] usbcore: registered new interface driver libusual
[    0.850786] PNP: No PS/2 controller found. Probing ports directly.
[    0.851626] i8042.c: No controller found.
[    0.852258] mice: PS/2 mouse device common for all mice
[    0.855068] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.855717] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.857602] cpuidle: using governor ladder
[    0.857619] cpuidle: using governor menu
[    0.857713] Netfilter messages via NETLINK v0.30.
[    0.857800] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    0.859562] ctnetlink v0.93: registering with nfnetlink.
[    0.861913] ip_tables: (C) 2000-2006 Netfilter Core Team
[    0.862342] TCP cubic registered
[    0.862354] Initializing XFRM netlink socket
[    0.864995] NET: Registered protocol family 10
[    0.869020] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    0.869455] IPv6 over IPv4 tunneling driver
[    0.873134] NET: Registered protocol family 17
[    0.873381] Registering the dns_resolver key type
[    0.875443] registered taskstats version 1
[    0.875615] kmemleak: Kernel memory leak detector initialized
[    0.875634] kmemleak: Automatic memory scanning thread started
[    0.877640] XENBUS: Device with no driver: device/vbd/51712
[    0.877654] XENBUS: Device with no driver: device/vif/0
[    0.877664] XENBUS: Device with no driver: device/console/0
[    0.877692]   Magic number: 1:252:3141
[   13.391082] Freeing unused kernel memory: 756k freed
[   13.391322] Write protecting the kernel read-only data: 10240k
[   13.401075] Freeing unused kernel memory: 1724k freed
[   13.402500] Freeing unused kernel memory: 1992k freed
Loading, please wait...
mount: mounting none on /dev failed: No such device
[   13.498289] all_generic_ide used greatest stack depth: 5472 bytes left
[   13.523488] udevd (1111): /proc/1111/oom_adj is deprecated, please use /proc/1111/oom_score_adj instead.
Begin: Loading essential drivers... ...
[   13.546062] udevadm used greatest stack depth: 5088 bytes left
[   13.580639] blkfront: xvda: barriers disabled
[   13.594779]  xvda: xvda1 xvda2 < xvda5 >
[   13.669872] Initialising Xen virtual ethernet driver.
[   13.672323] blkid used greatest stack depth: 4960 bytes left
[   13.756667] blkid used greatest stack depth: 4752 bytes left
Done.
Begin: Running /scripts/init-premount ...
Done.
Begin: Mounting root file system... ...
Begin: Running /scripts/local-top ...
Done.
Begin: Running /scripts/local-premount ...
Done.
[   13.903338] EXT4-fs (xvda1): INFO: recovery required on readonly filesystem
[   13.903363] EXT4-fs (xvda1): write access will be enabled during recovery
[   14.190114] EXT4-fs (xvda1): recovery complete
[   14.191367] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null)
[   14.191951] exe used greatest stack depth: 4112 bytes left
Begin: Running /scripts/local-bottom ...
Done.
Done.
Begin: Running /scripts/init-bottom ...
Done.
init: ureadahead main process (1208) terminated with status 5
fsck from util-linux-ng 2.17.2
e2fsck 1.41.11 (14-Mar-2010)
/dev/xvda1: clean, 56107/288576 files, 296111/1153280 blocks
 * Enable login on console                                               [ OK ]
 * Check for hostname changes...                                         [ OK ]
   ...done.
 * Setting sensors limits                                                       No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.
No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.
                                                                         [ OK ]
 * Initializing random number generator...                               [ OK ]
 * Setting up X server socket directory /tmp/.X11-unix...                [ OK ]
 * Setting up ICE socket directory /tmp/.ICE-unix...                     [ OK ]
 * Not starting fancontrol; run pwmconfig first.
 * Starting internet superserver inetd                                   [ OK ]
Starting openntpd: ntpd.
 * Starting network management services:                                         * Starting the system activity data collector sadc                                                                       Cannot open /var/log/sysstat/sa10: No such file or directory
                                                                         [fail]
 * Setting core file location...                                         [ OK ]
 * Restoring resolver state...                                           [ OK ]
 * Checking tomcat configuration
   ...done.
 * Starting Tomcat servlet engine tomcat6                                       Using CATALINA_BASE:   /var/lib/tomcat6
Using CATALINA_HOME:   /usr/share/tomcat6
Using CATALINA_TMPDIR: /tmp/tomcat6-tmp
Using JRE_HOME:        /usr/lib/jvm/java-6-openjdk
Using CLASSPATH:       /usr/share/tomcat6/bin/bootstrap.jar
                                                                         [ OK ]
collectd has been disabled - see /etc/default/collectd.
 * Recording successful boot for GRUB
   ...done.
 * Running local boot scripts (/etc/rc.local)                            [ OK ]
mountall: Plymouth command failed
mountall: Disconnected from Plymouth

6.0.16-01
kaan-04-dpm login: [   47.548041] ------------[ cut here ]------------
[   47.548078] kernel BUG at arch/x86/xen/mmu.c:1831!
[   47.548093] invalid opcode: 0000 [#1] SMP
[   47.548114] last sysfs file: /sys/kernel/uevent_seqnum
[   47.548124] CPU 0
[   47.548130] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
[   47.548199]
[   47.548208] Pid: 3815, comm: java Not tainted 2.6.37-rc1-pvops-pcifront-fixes-domu-6.0.bedge-debug #1 /
[   47.548222] RIP: e030:[<ffffffff81004bce>]  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[   47.548245] RSP: e02b:ffff880077fadce8  EFLAGS: 00010282
[   47.548255] RAX: 00000000ffffffea RBX: 0000000000078312 RCX: 0000000000000001
[   47.548266] RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 00000000deadbeef
[   47.548278] RBP: ffff880077fadd08 R08: 00003ffffffff000 R09: ffff880000000000
[   47.548290] R10: 00000000deadbeef R11: dead000000200200 R12: 0000000000000003
[   47.548302] R13: ffff880077f5e838 R14: ffff88007629ab80 R15: ffff880078288750
[   47.548319] FS:  00007f2e8058c710(0000) GS:ffff88007ff40000(0000) knlGS:0000000000000000
[   47.548332] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[   47.548342] CR2: 00007f2e60fe8728 CR3: 0000000077ec9000 CR4: 0000000000002660
[   47.548354] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   47.548366] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   47.548379] Process java (pid: 3815, threadinfo ffff880077fac000, task ffff880078288750)
[   47.548391] Stack:
[   47.548398]  ffff880000000000 00000000000476fc ffff88007603ea00 0000000000078312
[   47.548426]  ffff880077fadd28 ffffffff810066a8 0000000000078312 ffff88007603ea00
[   47.548454]  ffff880077fadd38 ffffffff810066db ffff880077fadd78 ffffffff810e048f
[   47.548481] Call Trace:
[   47.548494]  [<ffffffff810066a8>] xen_alloc_ptpage+0x68/0x6d
[   47.548509]  [<ffffffff810066db>] xen_alloc_pte+0xe/0x10
[   47.548525]  [<ffffffff810e048f>] __pte_alloc+0x72/0xd7
[   47.548539]  [<ffffffff810e2717>] handle_mm_fault+0x131/0x8c1
[   47.548554]  [<ffffffff8100728f>] ? xen_restore_fl_direct_end+0x0/0x1
[   47.548571]  [<ffffffff81442ef2>] ? _raw_spin_unlock_irqrestore+0x19/0x1c
[   47.548588]  [<ffffffff81048583>] ? try_to_wake_up+0x207/0x219
[   47.548603]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
[   47.548617]  [<ffffffff8100728f>] ? xen_restore_fl_direct_end+0x0/0x1
[   47.548632]  [<ffffffff81442ef2>] ? _raw_spin_unlock_irqrestore+0x19/0x1c
[   47.548648]  [<ffffffff81202bce>] ? rwsem_wake+0x39/0x43
[   47.548661]  [<ffffffff81206bc7>] ? call_rwsem_wake+0x17/0x30
[   47.548677]  [<ffffffff8106aa3d>] ? up_write+0x18/0x1a
[   47.548690]  [<ffffffff810e7378>] ? sys_mmap_pgoff+0x126/0x159
[   47.548705]  [<ffffffff81443615>] page_fault+0x25/0x30
[   47.548715] Code: ec 20 89 7d e0 48 89 f7 e8 c0 ff ff ff 48 8d 7d e0 48 89 45 e8 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 76 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 48 89 e5 41 54 49 89 fc 53 31 db 48 8b 3d
[   47.548794] RIP  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[   47.548794]  RSP <ffff880077fadce8>
[   47.548794] ---[ end trace ceae70ee7c69e40a ]---
[   63.942338] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
[   92.806778] ------------[ cut here ]------------
[   92.806802] kernel BUG at arch/x86/xen/mmu.c:1831!
[   92.806817] invalid opcode: 0000 [#2] SMP
[   92.806841] last sysfs file: /sys/kernel/uevent_seqnum
[   92.806856] CPU 0
[   92.806865] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
[   92.806967]
[   92.806981] Pid: 3914, comm: rsyslogd Tainted: G      D     2.6.37-rc1-pvops-pcifront-fixes-domu-6.0.bedge-debug #1 /
[   92.807005] RIP: e030:[<ffffffff81004bce>]  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[   92.807036] RSP: e02b:ffff880077869ce8  EFLAGS: 00010282
[   92.807053] RAX: 00000000ffffffea RBX: 0000000000077c61 RCX: 0000000000000001
[   92.807063] RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 00000000deadbeef
[   92.807063] RBP: ffff880077869d08 R08: 00003ffffffff000 R09: ffff880000000000
[   92.807063] R10: 00000000deadbeef R11: dead000000200200 R12: 0000000000000003
[   92.807063] R13: ffff8800760e44e0 R14: ffff880075ce8958 R15: ffff880077015f10
[   92.807063] FS:  00007ffd53e39700(0000) GS:ffff88007ff40000(0000) knlGS:0000000000000000
[   92.807063] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[   92.807063] CR2: 00007ffd53801330 CR3: 000000007821e000 CR4: 0000000000002660
[   92.807063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   92.807063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   92.807063] Process rsyslogd (pid: 3914, threadinfo ffff880077868000, task ffff880077015f10)
[   92.807063] Stack:
[   92.807063]  ffff880000000000 0000000000047dad ffff880076138700 0000000000077c61
[   92.807063]  ffff880077869d28 ffffffff810066a8 0000000000077c61 ffff880076138700
[   92.807063]  ffff880077869d38 ffffffff810066db ffff880077869d78 ffffffff810e048f
[   92.807063] Call Trace:
[   92.807063]  [<ffffffff810066a8>] xen_alloc_ptpage+0x68/0x6d
[   92.807063]  [<ffffffff810066db>] xen_alloc_pte+0xe/0x10
[   92.807063]  [<ffffffff810e048f>] __pte_alloc+0x72/0xd7
[   92.807063]  [<ffffffff810e2717>] handle_mm_fault+0x131/0x8c1
[   92.807063]  [<ffffffff8102b78a>] ? pvclock_clocksource_read+0x4b/0xb4
[   92.807063]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
[   92.807063]  [<ffffffff81007bc9>] ? xen_spin_unlock+0x11/0x30
[   92.807063]  [<ffffffff8105b253>] ? do_sigaction+0x175/0x18d
[   92.807063]  [<ffffffff811039a7>] ? fput+0x1ed/0x1fc
[   92.807063]  [<ffffffff8105d855>] ? sys_rt_sigaction+0x5e/0x9d
[   92.807063]  [<ffffffff81443615>] page_fault+0x25/0x30
[   92.807063] Code: ec 20 89 7d e0 48 89 f7 e8 c0 ff ff ff 48 8d 7d e0 48 89 45 e8 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 76 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 48 89 e5 41 54 49 89 fc 53 31 db 48 8b 3d
[   92.807063] RIP  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[   92.807063]  RSP <ffff880077869ce8>
[   92.807063] ---[ end trace ceae70ee7c69e40b ]---
[  693.101054] kmemleak: 86 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
[  695.855546] ------------[ cut here ]------------
[  695.855557] kernel BUG at arch/x86/xen/mmu.c:1831!
[  695.855563] invalid opcode: 0000 [#3] SMP
[  695.855574] last sysfs file: /sys/kernel/uevent_seqnum
[  695.855581] CPU 1
[  695.855585] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
[  695.855629]
[  695.855635] Pid: 3946, comm: cron Tainted: G      D     2.6.37-rc1-pvops-pcifront-fixes-domu-6.0.bedge-debug #1 /
[  695.855645] RIP: e030:[<ffffffff81004bce>]  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[  695.855664] RSP: e02b:ffff880067a29ce8  EFLAGS: 00010282
[  695.855671] RAX: 00000000ffffffea RBX: 0000000000077c58 RCX: 0000000000000001
[  695.855680] RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 00000000deadbeef
[  695.855689] RBP: ffff880067a29d08 R08: 00003ffffffff000 R09: ffff880000000000
[  695.855696] R10: 00000000deadbeef R11: dead000000200200 R12: 0000000000000003
[  695.855703] R13: ffff880067a2cb40 R14: ffff880077c7e678 R15: ffff8800682395f0
[  695.855716] FS:  00007fa1afde67a0(0000) GS:ffff88007ff5c000(0000) knlGS:0000000000000000
[  695.855725] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  695.855734] CR2: 00007fa1ad1ac758 CR3: 0000000067890000 CR4: 0000000000002660
[  695.855742] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  695.855749] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  695.855759] Process cron (pid: 3946, threadinfo ffff880067a28000, task ffff8800682395f0)
[  695.855767] Stack:
[  695.855774]  ffff880000000000 0000000000047db6 ffff8800760e3480 0000000000077c58
[  695.855793]  ffff880067a29d28 ffffffff810066a8 0000000000077c58 ffff8800760e3480
[  695.855814]  ffff880067a29d38 ffffffff810066db ffff880067a29d78 ffffffff810e048f
[  695.855834] Call Trace:
[  695.855842]  [<ffffffff810066a8>] xen_alloc_ptpage+0x68/0x6d
[  695.855852]  [<ffffffff810066db>] xen_alloc_pte+0xe/0x10
[  695.855865]  [<ffffffff810e048f>] __pte_alloc+0x72/0xd7
[  695.855877]  [<ffffffff810e2717>] handle_mm_fault+0x131/0x8c1
[  695.855887]  [<ffffffff810e6e36>] ? mmap_region+0x391/0x4bf
[  695.855900]  [<ffffffff8100eb84>] ? arch_get_unmapped_area_topdown+0x4a/0x28a
[  695.855915]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
[  695.855925]  [<ffffffff8102b78a>] ? pvclock_clocksource_read+0x4b/0xb4
[  695.855937]  [<ffffffff81007c09>] ? __spin_time_accum+0x21/0x37
[  695.855949]  [<ffffffff81007e85>] ? __xen_spin_lock+0xb7/0xcd
[  695.855959]  [<ffffffff81443615>] page_fault+0x25/0x30
[  695.855967] Code: ec 20 89 7d e0 48 89 f7 e8 c0 ff ff ff 48 8d 7d e0 48 89 45 e8 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 76 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 48 89 e5 41 54 49 89 fc 53 31 db 48 8b 3d
[  695.856056] RIP  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[  695.856056]  RSP <ffff880067a29ce8>
[  695.856056] ---[ end trace ceae70ee7c69e40c ]---
[  696.704202] cron[3928] general protection ip:7fa1af4ce588 sp:7ffff1875200 error:0 in libc-2.11.1.so[7fa1af424000+17a000]
[  697.853416] ------------[ cut here ]------------
[  697.853427] kernel BUG at arch/x86/xen/mmu.c:1831!
[  697.853433] invalid opcode: 0000 [#4] SMP
[  697.853443] last sysfs file: /sys/kernel/uevent_seqnum
[  697.853450] CPU 4
[  697.853453] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
[  697.853497]
[  697.853503] Pid: 3947, comm: sh Tainted: G      D     2.6.37-rc1-pvops-pcifront-fixes-domu-6.0.bedge-debug #1 /
[  697.853512] RIP: e030:[<ffffffff81004bce>]  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[  697.853526] RSP: e02b:ffff880067aa5b38  EFLAGS: 00010282
[  697.853532] RAX: 00000000ffffffea RBX: 0000000000075203 RCX: 0000000000000001
[  697.853540] RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 00000000deadbeef
[  697.853547] RBP: ffff880067aa5b58 R08: 00003ffffffff000 R09: ffff880000000000
[  697.853554] R10: 00000000deadbeef R11: dead000000200200 R12: 0000000000000003
[  697.853561] R13: ffff880075204000 R14: ffff880075204000 R15: ffff880078a26ff8
[  697.853571] FS:  00007fa1afde67a0(0000) GS:ffff88007ffb0000(0000) knlGS:0000000000000000
[  697.853580] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  697.853586] CR2: 00007fa1af7bd560 CR3: 0000000077a98000 CR4: 0000000000002660
[  697.853594] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  697.853602] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  697.853609] Process sh (pid: 3947, threadinfo ffff880067aa4000, task ffff880068239d40)
[  697.853616] Stack:
[  697.853621]  ffff880000000000 000000000004a80b ffff8800760e1500 0000000000075203
[  697.853640]  ffff880067aa5b78 ffffffff810066a8 0000000000075203 ffff8800760e1500
[  697.853658]  ffff880067aa5b88 ffffffff810066db ffff880067aa5bc8 ffffffff810e048f
[  697.853676] Call Trace:
[  697.853685]  [<ffffffff810066a8>] xen_alloc_ptpage+0x68/0x6d
[  697.853693]  [<ffffffff810066db>] xen_alloc_pte+0xe/0x10
[  697.853704]  [<ffffffff810e048f>] __pte_alloc+0x72/0xd7
[  697.853713]  [<ffffffff810e7faa>] move_page_tables+0x1cb/0x4c1
[  697.853722]  [<ffffffff81007bc9>] ? xen_spin_unlock+0x11/0x30
[  697.853732]  [<ffffffff8110870a>] setup_arg_pages+0x1cb/0x355
[  697.853743]  [<ffffffff8113ce7a>] load_elf_binary+0x418/0x16ec
[  697.853752]  [<ffffffff8113b319>] ? load_misc_binary+0x5c/0x33c
[  697.853761]  [<ffffffff810e3336>] ? get_user_pages+0x44/0x46
[  697.853772]  [<ffffffff81107355>] ? get_arg_page+0x50/0xa7
[  697.853781]  [<ffffffff81107793>] search_binary_handler+0xbb/0x24d
[  697.853790]  [<ffffffff8113ca62>] ? load_elf_binary+0x0/0x16ec
[  697.853799]  [<ffffffff81108e2d>] do_execve+0x19e/0x28d
[  697.853809]  [<ffffffff810118dd>] sys_execve+0x3e/0x55
[  697.853818]  [<ffffffff8100af1c>] stub_execve+0x6c/0xc0
[  697.853824] Code: ec 20 89 7d e0 48 89 f7 e8 c0 ff ff ff 48 8d 7d e0 48 89 45 e8 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 76 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 48 89 e5 41 54 49 89 fc 53 31 db 48 8b 3d
[  697.854013] RIP  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
[  697.854024]  RSP <ffff880067aa5b38>
[  697.854031] ---[ end trace ceae70ee7c69e40d ]---
[  755.706246] BUG: unable to handle kernel paging request at ffff880075202b80
[  755.706263] IP: [<ffffffff810e24b3>] do_wp_page+0x58b/0x6be
[  755.706278] PGD 1a04067 PUD 39d9067 PMD 3b83067 PTE 8010000075202065
[  755.706294] Oops: 0003 [#5] SMP
[  755.706305] last sysfs file: /sys/kernel/uevent_seqnum
[  755.706312] CPU 4
[  755.706316] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
[  755.706364]
[  755.706370] Pid: 3948, comm: cron Tainted: G      D     2.6.37-rc1-pvops-pcifront-fixes-domu-6.0.bedge-debug #1 /
[  755.706379] RIP: e030:[<ffffffff810e24b3>]  [<ffffffff810e24b3>] do_wp_page+0x58b/0x6be
[  755.706391] RSP: e02b:ffff880067439ce8  EFLAGS: 00010246
[  755.706397] RAX: 0000000000000000 RBX: ffffea00016f7d98 RCX: 0000000000000001
[  755.706404] RDX: 6db6db6db6db6db7 RSI: ffffffff81b460c0 RDI: ffff8800760a1f80
[  755.706411] RBP: ffff880067439d78 R08: 0000000000000000 R09: 0000000000000041
[  755.706419] R10: 0000000000000100 R11: ffff880078277030 R12: 80000000407c3067
[  755.706426] R13: ffffea0001bd0068 R14: ffff880075202b80 R15: ffff880067a77508
[  755.706436] FS:  00007fa1afde67a0(0000) GS:ffff88007ffb0000(0000) knlGS:0000000000000000
[  755.706444] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  755.706451] CR2: ffff880075202b80 CR3: 0000000077ea9000 CR4: 0000000000002660
[  755.706458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  755.706466] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  755.706474] Process cron (pid: 3948, threadinfo ffff880067438000, task ffff88006823a490)
[  755.706481] Stack:
[  755.706486]  ffff880067439d08 ffffffff81007c09 ffffea000199f080 0000000000000000
[  755.706504]  01ff880067439d58 ffffea000199f080 ffff8800760a1f80 0000000000d703b0
[  755.706523]  00007fa1afde6a70 ffff880078277030 ffff880067a77508 ffff8800760a1f80
[  755.706542] Call Trace:
[  755.706552]  [<ffffffff81007c09>] ? __spin_time_accum+0x21/0x37
[  755.706562]  [<ffffffff810e2e60>] handle_mm_fault+0x87a/0x8c1
[  755.706571]  [<ffffffff8103ed4d>] ? enqueue_entity+0x2ac/0x2b8
[  755.706582]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
[  755.706590]  [<ffffffff8100728f>] ? xen_restore_fl_direct_end+0x0/0x1
[  755.706600]  [<ffffffff81004274>] ? xen_mc_flush+0x1cd/0x1f2
[  755.706609]  [<ffffffff8103ed4d>] ? enqueue_entity+0x2ac/0x2b8
[  755.706618]  [<ffffffff81007bc9>] ? xen_spin_unlock+0x11/0x30
[  755.706628]  [<ffffffff81045b20>] ? finish_task_switch+0x4f/0x94
[  755.706637]  [<ffffffff81443615>] page_fault+0x25/0x30
[  755.706644] Code: 48 0f af fa 48 c1 e7 0c 48 09 c7 ff 14 25 28 13 a1 81 49 89 c4 48 83 c8 42 49 8b 3f 49 83 cc 40 41 f6 47 30 02 4c 0f 45 e0 31 c0 <49> 87 06 48 8b 75 a8 4c 89 f2 0f 1f 80 00 00 00 00 48 8b 75 a8
[  755.706830] RIP  [<ffffffff810e24b3>] do_wp_page+0x58b/0x6be
[  755.706841]  RSP <ffff880067439ce8>
[  755.706847] CR2: ffff880075202b80
[  755.706854] ---[ end trace ceae70ee7c69e40e ]---

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.