[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] Re: 2.6.37-rc1 mainline domU - BUG: unable to handle kernel paging request


  • To: xen-devel@xxxxxxxxxxxxxxxxxxx, Jeremy Fitzhardinge <jeremy@xxxxxxxx>, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
  • From: Bruce Edge <bruce.edge@xxxxxxxxx>
  • Date: Wed, 10 Nov 2010 15:03:34 -0800
  • Cc:
  • Delivery-date: Wed, 10 Nov 2010 15:04:51 -0800
  • Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=DB1fiOlajYDgDs5nTiS0PqXYN0MR8UAME5ZeJNcUMwjW9OUX11z8vtXtxaw3CXSObr B5z4N79sArcjJuv05Fnft1SXzkZh3HEfXUmZ31p7rkT2UKv0fR9mPU9e+OD0FKI3e5ab A+Grsy6Mq13pyxbzRqfiPbZbaeJE8Ua7qzRMY=
  • List-id: Xen developer discussion <xen-devel.lists.xensource.com>

On Wed, Nov 10, 2010 at 2:30 PM, Bruce Edge <bruce.edge@xxxxxxxxx> wrote:
> On Wed, Nov 10, 2010 at 2:15 PM, Bruce Edge <bruce.edge@xxxxxxxxx> wrote:
>>
>> This is a mainline 2.6.37-rc1 pvops domU on top of a xen-unstable build 
>> 22241 and xen/stable-2.6.32.x 2.6.32.24 pvops dom0.
>>
>> The domU console output hung up prior to spitting out a prompt post root 
>> login. I logged into the domU via ssh and ran an 'ls' of an NFS mounted dir 
>> and
>> this immediately came out on the domU console:
>>
>> [  480.467525] BUG: unable to handle kernel paging request at 
>> ffff880077e9f030
>> [  480.467547] IP: [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
>> [  480.467564] PGD 1a04067 PUD 3dd7067 PMD 3f97067 PTE 8010000077e9f065
>> [  480.467587] Oops: 0003 [#1] SMP
>> [  480.467601] last sysfs file: /sys/kernel/uevent_seqnum
>> [  480.467611] CPU 3
>> [  480.467616] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront 
>> fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront 
>> xen_blkfront
>> [  480.467676]
>> [  480.467684] Pid: 3957, comm: zsh Not tainted 
>> 2.6.37-rc1-kernel.org-domu-6.0.bedge-debug #1 /
>> [  480.467695] RIP: e030:[<ffffffff810355f3>]  [<ffffffff810355f3>] 
>> ptep_set_access_flags+0x27/0x4d
>> [  480.467710] RSP: e02b:ffff880077ebfbd8  EFLAGS: 00010202
>> [  480.467719] RAX: 800000004d686167 RBX: ffff8800763ddb80 RCX: 
>> 800000004d686167
>> [  480.467729] RDX: ffff880077e9f030 RSI: 00007f5cb26069d0 RDI: 
>> ffff8800763ddb80
>> [  480.467739] RBP: ffff880077ebfc08 R08: 0000000000000001 R09: 
>> ffffea0001a3b2d8
>> [  480.467748] R10: 0000000000000100 R11: 0000000000000000 R12: 
>> 00007f5cb26069d0
>> [  480.467759] R13: 0000000000000001 R14: 0000000000000000 R15: 
>> ffff8800763ddb80
>> [  480.467772] FS:  00007f5cb2606700(0000) GS:ffff88007ff94000(0000) 
>> knlGS:0000000000000000
>> [  480.467783] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [  480.467792] CR2: ffff880077e9f030 CR3: 0000000077dbc000 CR4: 
>> 0000000000002660
>> [  480.467802] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
>> 0000000000000000
>> [  480.467812] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
>> 0000000000000400
>> [  480.467822] Process zsh (pid: 3957, threadinfo ffff880077ebe000, task 
>> ffff880078b80000)
>> [  480.467832] Stack:
>> [  480.467838]  0000000000000100 ffffea0001a3b2d8 7000000000000000 
>> 0000000000000000
>> [  480.467862]  ffff880077e9f030 ffffea0001a3b2d8 ffff880077ebfca8 
>> ffffffff810e2200
>> [  480.467886]  ffff880077ebfc38 ffffffff81007c09 ffffea0001a3b2d8 
>> 0000000000000001
>> [  480.467910] Call Trace:
>> [  480.467921]  [<ffffffff810e2200>] do_wp_page+0x2d8/0x6be
>> [  480.467934]  [<ffffffff81007c09>] ? __spin_time_accum+0x21/0x37
>> [  480.467946]  [<ffffffff81007e85>] ? __xen_spin_lock+0xb7/0xcd
>> [  480.467958]  [<ffffffff810e2e60>] handle_mm_fault+0x87a/0x8c1
>> [  480.467972]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
>> [  480.467984]  [<ffffffff81006c3d>] ? xen_force_evtchn_callback+0xd/0xf
>> [  480.467996]  [<ffffffff81006c3d>] ? xen_force_evtchn_callback+0xd/0xf
>> [  480.468009]  [<ffffffff810072a2>] ? check_events+0x12/0x20
>> [  480.468020]  [<ffffffff8100728f>] ? xen_restore_fl_direct_end+0x0/0x1
>> [  480.468034]  [<ffffffff810a6fa4>] ? __call_rcu+0x123/0x12c
>> [  480.468045]  [<ffffffff81006c3d>] ? xen_force_evtchn_callback+0xd/0xf
>> [  480.468059]  [<ffffffff810072a2>] ? check_events+0x12/0x20
>> [  480.468072]  [<ffffffff81443615>] page_fault+0x25/0x30
>> [  480.468085]  [<ffffffff81206add>] ? __put_user_4+0x1d/0x30
>> [  480.468096]  [<ffffffff810498a6>] ? schedule_tail+0x5c/0x60
>> [  480.468108]  [<ffffffff8100a9c3>] ret_from_fork+0x13/0x80
>> [  480.468116] Code: 41 5d c9 c3 55 48 89 e5 41 55 45 31 ed 41 54 49 89 f4 
>> 53 48 89 fb 48 83 ec 18 48 39 0a 41 0f 95 c5 45 85 c0 74 1d 45 85 ed 74 18 
>> <48> 89 0a 48 8b 3f 0f 1f 80 00 00 00 00 4c 89 e6 48 89 df e8 b2
>> [  480.468323] RIP  [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
>> [  480.468323]  RSP <ffff880077ebfbd8>
>> [  480.468323] CR2: ffff880077e9f030
>> [  480.468323] ---[ end trace 6d1623321ecd50ac ]---
>>
>> The domU config is attached as config.2.6.37_domU_debug
>>
>>
>> -Bruce
>
> Here's another attempt from the same config with the full domU boot logs:
>
> Started domain dpm (id=4)
>                         [    0.000000] Initializing cgroup subsys cpuset
> [    0.000000] Initializing cgroup subsys cpu
> [    0.000000] Linux version
> 2.6.37-rc1-kernel.org-domu-6.0.bedge-debug (root@opal) (gcc version
> 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #1 SMP Tue Nov 9 20:26:03 PST 2010
> [    0.000000] Command line:
> root=UUID=4d87e4cb-8317-41d4-aea2-ac2ebee699f7 ro   swiotlb=force
> noirqbalance noirqbalance console=hvc0,115200n8 --flow_control=r
> ip=:127.0.255.255::::eth0:dhcp
> [    0.000000] ACPI in unprivileged domain disabled
> [    0.000000] released 0 pages of unused memory
> [    0.000000] BIOS-provided physical RAM map:
> [    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
> [    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
> [    0.000000]  Xen: 0000000000100000 - 0000000080800000 (usable)
> [    0.000000] NX (Execute Disable) protection: active
> [    0.000000] DMI not present or invalid.
> [    0.000000] No AGP bridge found
> [    0.000000] last_pfn = 0x80800 max_arch_pfn = 0x400000000
> [    0.000000] Scanning 0 areas for low memory corruption
> [    0.000000] init_memory_mapping: 0000000000000000-0000000080800000
> [    0.000000] RAMDISK: 01d95000 - 02fa9000
> [    0.000000] No NUMA configuration found
> [    0.000000] Faking a node at 0000000000000000-0000000080800000
> [    0.000000] Initmem setup node 0 0000000000000000-0000000080800000
> [    0.000000]   NODE_DATA [000000007fffb000 - 000000007fffffff]
> [    0.000000] Zone PFN ranges:
> [    0.000000]   DMA      0x00000010 -> 0x00001000
> [    0.000000]   DMA32    0x00001000 -> 0x00100000
> [    0.000000]   Normal   empty
> [    0.000000] Movable zone start PFN for each node
> [    0.000000] early_node_map[2] active PFN ranges
> [    0.000000]     0: 0x00000010 -> 0x000000a0
> [    0.000000]     0: 0x00000100 -> 0x00080800
> [    0.000000] SMP: Allowing 6 CPUs, 0 hotplug CPUs
> [    0.000000] No local APIC present
> [    0.000000] APIC: disable apic facility
> [    0.000000] APIC: switched to apic NOOP
> [    0.000000] PM: Registered nosave memory: 00000000000a0000 - 
> 0000000000100000
> [    0.000000] Allocating PCI resources starting at 80800000 (gap:
> 80800000:7f800000)
> [    0.000000] Booting paravirtualized kernel on Xen
> [    0.000000] Xen version: 4.1-unstable (preserve-AD)
> [    0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32
> nr_cpu_ids:6 nr_node_ids:1
> [    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88007ff40000 s82752
> r8192 d23744 u114688
> [    0.000000] pcpu-alloc: s82752 r8192 d23744 u114688 alloc=28*4096
> [    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5
> [    0.000000] Built 1 zonelists in Node order, mobility grouping on.
> Total pages: 519022
> [    0.000000] Policy zone: DMA32
> [    0.000000] Kernel command line:
> root=UUID=4d87e4cb-8317-41d4-aea2-ac2ebee699f7 ro   swiotlb=force
> noirqbalance noirqbalance console=hvc0,115200n8 --flow_control=r
> ip=:127.0.255.255::::eth0:dhcp
> [    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
> [    0.000000] Placing 64MB software IO TLB between ffff880079400000 -
> ffff88007d400000
> [    0.000000] software IO TLB at phys 0x79400000 - 0x7d400000
> [    0.000000] Memory: 1959896k/2105344k available (4398k kernel code,
> 448k absent, 145000k reserved, 6327k data, 756k init)
> [    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0,
> CPUs=6, Nodes=1
> [    0.000000] Hierarchical RCU implementation.
> [    0.000000]  RCU-based detection of stalled CPUs is disabled.
> [    0.000000] NR_IRQS:4352 nr_irqs:320 16
> [    0.000000] Console: colour dummy device 80x25
> [    0.000000] console [tty0] enabled
> [    0.000000] console [hvc0] enabled
> [    0.000000] installing Xen timer for CPU 0
> [    0.000000] Detected 2660.064 MHz processor.
> [    0.000999] Calibrating delay loop (skipped), value calculated
> using timer frequency.. 5320.12 BogoMIPS (lpj=2660064)
> [    0.000999] pid_max: default: 32768 minimum: 301
> [    0.000999] Security Framework initialized
> [    0.000999] SELinux:  Initializing.
> [    0.000999] Dentry cache hash table entries: 262144 (order: 9, 2097152 
> bytes)
> [    0.001408] Inode-cache hash table entries: 131072 (order: 8, 1048576 
> bytes)
> [    0.001635] Mount-cache hash table entries: 256
> [    0.002195] Initializing cgroup subsys ns
> [    0.002208] ns_cgroup deprecated: consider using the
> 'clone_children' flag without the ns_cgroup.
> [    0.002220] Initializing cgroup subsys cpuacct
> [    0.002231] Initializing cgroup subsys freezer
> [    0.002356] CPU: Unsupported number of siblings 16
> [    0.002362] Performance Events: unsupported p6 CPU model 26 no PMU
> driver, software events only.
> [    0.002483] SMP alternatives: switching to UP code
> [    0.004119] cpu 0 spinlock event irq 318
> [    0.004765] NMI watchdog failed to create perf event on cpu0:
> fffffffffffffffe
> [    0.005050] installing Xen timer for CPU 1
> [    0.005157] cpu 1 spinlock event irq 312
> [    0.005194] SMP alternatives: switching to SMP code
> [    0.000999] CPU: Unsupported number of siblings 16
> [    0.006158] NMI watchdog failed to create perf event on cpu1:
> fffffffffffffffe
> [    0.006474] installing Xen timer for CPU 2
> [    0.006585] cpu 2 spinlock event irq 306
> [    0.000999] CPU: Unsupported number of siblings 16
> [    0.006816] NMI watchdog failed to create perf event on cpu2:
> fffffffffffffffe
> [    0.006999] installing Xen timer for CPU 3
> [    0.007037] cpu 3 spinlock event irq 300
> [    0.000999] CPU: Unsupported number of siblings 16
> [    0.007300] NMI watchdog failed to create perf event on cpu3:
> fffffffffffffffe
> [    0.007534] installing Xen timer for CPU 4
> [    0.007636] cpu 4 spinlock event irq 294
> [    0.000999] CPU: Unsupported number of siblings 16
> [    0.007874] NMI watchdog failed to create perf event on cpu4:
> fffffffffffffffe
> [    0.008041] installing Xen timer for CPU 5
> [    0.008083] cpu 5 spinlock event irq 288
> [    0.000999] CPU: Unsupported number of siblings 16
> [    0.008344] NMI watchdog failed to create perf event on cpu5:
> fffffffffffffffe
> [    0.008356] Brought up 6 CPUs
> [    0.009181] kworker/u:0 used greatest stack depth: 5696 bytes left
> [    0.012051] Grant table initialized
> [    0.030978] Time: 165:165:165  Date: 165/165/65
> [    0.031229] NET: Registered protocol family 16
> [    0.036672] PCI: setting up Xen PCI frontend stub
> [    0.037225] kworker/u:0 used greatest stack depth: 5528 bytes left
> [    0.074302] bio: create slab <bio-0> at 0
> [    0.075385] ACPI: Interpreter disabled.
> [    0.077056] xen_balloon: Initialising balloon driver.
> [    0.079323] vgaarb: loaded
> [    0.081270] usbcore: registered new interface driver usbfs
> [    0.081283] usbcore: registered new interface driver hub
> [    0.081283] usbcore: registered new device driver usb
> [    0.083183] PCI: System does not support PCI
> [    0.084007] PCI: System does not support PCI
> [    0.085236] NetLabel: Initializing
> [    0.085236] NetLabel:  domain hash size = 128
> [    0.085236] NetLabel:  protocols = UNLABELED CIPSOv4
> [    0.085313] NetLabel:  unlabeled traffic allowed by default
> [    0.086544] Switching to clocksource xen
> [    0.106256] pnp: PnP ACPI: disabled
> [    0.129799] NET: Registered protocol family 2
> [    0.130174] IP route cache hash table entries: 65536 (order: 7, 524288 
> bytes)
> [    0.131882] TCP established hash table entries: 262144 (order: 10,
> 4194304 bytes)
> [    0.132891] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
> [    0.133179] TCP: Hash tables configured (established 262144 bind 65536)
> [    0.133191] TCP reno registered
> [    0.133301] UDP hash table entries: 1024 (order: 3, 32768 bytes)
> [    0.133407] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
> [    0.133914] NET: Registered protocol family 1
> [    0.134582] RPC: Registered udp transport module.
> [    0.134590] RPC: Registered tcp transport module.
> [    0.134595] RPC: Registered tcp NFSv4.1 backchannel transport module.
> [    0.134892] Trying to unpack rootfs image as initramfs...
> [    0.178824] Freeing initrd memory: 18512k freed
> [    0.268798] DMA-API: preallocated 32768 debug entries
> [    0.268810] DMA-API: debugging enabled by kernel config
> [    0.271181] platform rtc_cmos: registered platform RTC device (no
> PNP device found)
> [    0.285410] Machine check injector initialized
> [    0.294732] microcode: CPU0 sig=0x106a4, pf=0x2, revision=0xa
> [    0.294803] microcode: CPU1 sig=0x106a4, pf=0x2, revision=0xa
> [    0.294893] microcode: CPU2 sig=0x106a4, pf=0x2, revision=0xa
> [    0.294944] microcode: CPU3 sig=0x106a4, pf=0x2, revision=0xa
> [    0.294998] microcode: CPU4 sig=0x106a4, pf=0x2, revision=0xa
> [    0.295043] microcode: CPU5 sig=0x106a4, pf=0x2, revision=0xa
> [    0.295590] microcode: Microcode Update Driver: v2.00
> <tigran@xxxxxxxxxxxxxxxxxxxx>, Peter Oruba
> [    0.295607] Scanning for low memory corruption every 60 seconds
> [    0.298813] audit: initializing netlink socket (disabled)
> [    0.298940] type=2000 audit(1289403082.795:1): initialized
> [    0.333762] HugeTLB registered 2 MB page size, pre-allocated 0 pages
> [    0.376985] VFS: Disk quotas dquot_6.5.2
> [    0.377853] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> [    0.383121] NTFS driver 2.1.29 [Flags: R/W].
> [    0.385014] msgmni has been set to 3864
> [    0.389953] Block layer SCSI generic (bsg) driver version 0.4
> loaded (major 253)
> [    0.389970] io scheduler noop registered
> [    0.389979] io scheduler deadline registered
> [    0.390666] io scheduler cfq registered (default)
> [    0.392486] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> [    0.406238] pcifront pci-0: Installing PCI frontend
> [    0.406967] pcifront pci-0: Creating PCI Frontend Bus 0000:00
> [    0.417634] pcifront pci-0: claiming resource 0000:00:00.0/0
> [    0.417634] pcifront pci-0: claiming resource 0000:00:00.0/2
> [    0.417634] pcifront pci-0: claiming resource 0000:00:00.0/3
> [    0.417634] pcifront pci-0: claiming resource 0000:00:00.0/4
> [    0.417634] pcifront pci-0: claiming resource 0000:00:01.0/0
> [    0.417634] pcifront pci-0: claiming resource 0000:00:01.0/2
> [    0.417634] pcifront pci-0: claiming resource 0000:00:01.0/3
> [    0.417634] pcifront pci-0: claiming resource 0000:00:01.0/4
> [    0.417634] pcifront pci-0: claiming resource 0000:00:02.0/0
> [    0.417634] pcifront pci-0: claiming resource 0000:00:02.0/2
> [    0.417634] pcifront pci-0: claiming resource 0000:00:02.0/3
> [    0.417634] pcifront pci-0: claiming resource 0000:00:02.0/4
> [    0.417634] pcifront pci-0: claiming resource 0000:00:03.0/0
> [    0.417634] pcifront pci-0: claiming resource 0000:00:03.0/2
> [    0.417634] pcifront pci-0: claiming resource 0000:00:03.0/3
> [    0.417634] pcifront pci-0: claiming resource 0000:00:03.0/4
> [    0.731775] Non-volatile memory driver v1.3
> [    0.731790] Linux agpgart interface v0.103
> [    0.733731] [drm] Initialized drm 1.1.0 20060810
> [    0.733752] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
> [    0.827202] brd: module loaded
> [    0.840788] loop: module loaded
> [    0.841965] Fixed MDIO Bus: probed
> [    0.844025] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
> [    0.844651] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
> [    0.845194] uhci_hcd: USB Universal Host Controller Interface driver
> [    0.846985] usbcore: registered new interface driver usblp
> [    0.847763] usbcore: registered new interface driver libusual
> [    0.849433] PNP: No PS/2 controller found. Probing ports directly.
> [    0.850279] i8042.c: No controller found.
> [    0.850883] mice: PS/2 mouse device common for all mice
> [    0.853446] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
> [    0.854066] rtc_cmos: probe of rtc_cmos failed with error -38
> [    0.855928] cpuidle: using governor ladder
> [    0.855941] cpuidle: using governor menu
> [    0.856029] Netfilter messages via NETLINK v0.30.
> [    0.856136] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
> [    0.857888] ctnetlink v0.93: registering with nfnetlink.
> [    0.860285] ip_tables: (C) 2000-2006 Netfilter Core Team
> [    0.860677] TCP cubic registered
> [    0.860687] Initializing XFRM netlink socket
> [    0.863286] NET: Registered protocol family 10
> [    0.867216] ip6_tables: (C) 2000-2006 Netfilter Core Team
> [    0.867612] IPv6 over IPv4 tunneling driver
> [    0.871326] NET: Registered protocol family 17
> [    0.871566] Registering the dns_resolver key type
> [    0.873668] registered taskstats version 1
> [    0.873912] kmemleak: Kernel memory leak detector initialized
> [    0.873928] kmemleak: Automatic memory scanning thread started
> [    0.875057] XENBUS: Device with no driver: device/vbd/51712
> [    0.875057] XENBUS: Device with no driver: device/vif/0
> [    0.875057] XENBUS: Device with no driver: device/console/0
> [    0.875057]   Magic number: 1:252:3141
> [   13.388958] Freeing unused kernel memory: 756k freed
> [   13.389190] Write protecting the kernel read-only data: 10240k
> [   13.398774] Freeing unused kernel memory: 1724k freed
> [   13.400663] Freeing unused kernel memory: 1992k freed
> [   13.494573] all_generic_ide used greatest stack depth: 5472 bytes left
> [   13.519388] udevd (1111): /proc/1111/oom_adj is deprecated, please
> use /proc/1111/oom_score_adj instead.
> Begin: Loading essential drivers... ...
> [   13.559435] blkid used greatest stack depth: 5408 bytes left
> [   13.568391] blkfront: xvda: barriers disabled
> [   13.574762] blkid used greatest stack depth: 5264 bytes left
> [   13.580813]  xvda: xvda1 xvda2 < xvda5 >
> [   13.645562] modprobe used greatest stack depth: 5088 bytes left
> [   13.653160] Initialising Xen virtual ethernet driver.
> [   13.659540] blkid used greatest stack depth: 4960 bytes left
> [   13.735467] blkid used greatest stack depth: 4752 bytes left
> Done.
> Begin: Running /scripts/init-premount ...
> Done.
> Begin: Mounting root file system... ...
> Begin: Running /scripts/local-top ...
> Done.
> Begin: Running /scripts/local-premount ...
> Done.
> [   13.903774] EXT4-fs (xvda1): INFO: recovery required on readonly filesystem
> [   13.903799] EXT4-fs (xvda1): write access will be enabled during recovery
> [   14.033719] EXT4-fs (xvda1): recovery complete
> [   14.034966] EXT4-fs (xvda1): mounted filesystem with ordered data
> mode. Opts: (null)
> [   14.035510] exe used greatest stack depth: 4112 bytes left
> Begin: Running /scripts/local-bottom ...
> Done.
> Done.
> Begin: Running /scripts/init-bottom ...
> Done.
> init: ureadahead main process (1214) terminated with status 5
> fsck from util-linux-ng 2.17.2
> e2fsck 1.41.11 (14-Mar-2010)
> /dev/xvda1: clean, 56314/288576 files, 287743/1153280 blocks
>  * Enable login on console                                               [ OK 
> ]
>  * Check for hostname changes...                                         [ OK 
> ]
>   ...done.
>  * Setting sensors limits
>         No sensors found!
> Make sure you loaded all the kernel drivers you need.
> Try sensors-detect to find out which these are.
> No sensors found!
> Make sure you loaded all the kernel drivers you need.
> Try sensors-detect to find out which these are.
>                                                                         [ OK ]
>  * Initializing random number generator...                               [ OK 
> ]
>  * Setting up X server socket directory /tmp/.X11-unix...                [ OK 
> ]
>  * Setting up ICE socket directory /tmp/.ICE-unix...                     [ OK 
> ]
>  * Not starting fancontrol; run pwmconfig first.
>  * Starting internet superserver inetd                                   [ OK 
> ]
> Starting openntpd: ntpd.
>  * Starting network management services:
>  * Setting core file location...                                         [ OK 
> ]
>  * Restoring resolver state...                                           [ OK 
> ]
>  * Checking tomcat configuration
>   ...done.
>  * Starting Tomcat servlet engine tomcat6
>         Using CATALINA_BASE:   /var/lib/tomcat6
> Using CATALINA_HOME:   /usr/share/tomcat6
> Using CATALINA_TMPDIR: /tmp/tomcat6-tmp
> Using JRE_HOME:        /usr/lib/jvm/java-6-openjdk
> Using CLASSPATH:       /usr/share/tomcat6/bin/bootstrap.jar
>                                                                         [ OK ]
> collectd has been disabled - see /etc/default/collectd.
>  * Recording successful boot for GRUB
>   ...done.
>  * Running local boot scripts (/etc/rc.local)                            [ OK 
> ]
> mountall: Plymouth command failed
> mountall: Disconnected from Plymouth
>
> 6.0.16-01
> kaan-04-dpm login: root
> Password:
> [   42.646538] ------------[ cut here ]------------
> [   42.646556] kernel BUG at arch/x86/xen/mmu.c:1831!
> [   42.646566] invalid opcode: 0000 [#1] SMP
> [   42.646582] last sysfs file: /sys/kernel/uevent_seqnum
> [   42.646592] CPU 0
> [   42.646598] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront
> fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront
> xen_netfront xen_blkfront
> [   42.646665]
> [   42.646674] Pid: 3916, comm: lsb_release Not tainted
> 2.6.37-rc1-kernel.org-domu-6.0.bedge-debug #1 /
> [   42.646688] RIP: e030:[<ffffffff81004bce>]  [<ffffffff81004bce>]
> pin_pagetable_pfn+0x31/0x37
> [   42.646711] RSP: e02b:ffff88007595bce8  EFLAGS: 00010282
> [   42.646721] RAX: 00000000ffffffea RBX: 000000000007751e RCX: 
> 0000000000000001
> [   42.646732] RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 
> 00000000deadbeef
> [   42.646744] RBP: ffff88007595bd08 R08: 00003ffffffff000 R09: 
> ffff880000000000
> [   42.646756] R10: 00000000deadbeef R11: dead000000200200 R12: 
> 0000000000000003
> [   42.646768] R13: ffff880075c90150 R14: ffff8800781d9398 R15: 
> ffff880075161d40
> [   42.646784] FS:  00007fee63b01700(0000) GS:ffff88007ff40000(0000)
> knlGS:0000000000000000
> [   42.646797] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [   42.646807] CR2: 00007faac54182b8 CR3: 00000000771dd000 CR4: 
> 0000000000002660
> [   42.646819] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> 0000000000000000
> [   42.646832] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
> 0000000000000400
> [   42.646844] Process lsb_release (pid: 3916, threadinfo
> ffff88007595a000, task ffff880075161d40)
> [   42.646857] Stack:
> [   42.646864]  ffff880000000000 00000000000484f0 ffff8800777c6d80
> 000000000007751e
> [   42.646892]  ffff88007595bd28 ffffffff810066a8 000000000007751e
> ffff8800777c6d80
> [   42.646919]  ffff88007595bd38 ffffffff810066db ffff88007595bd78
> ffffffff810e048f
> [   42.646946] Call Trace:
> [   42.646959]  [<ffffffff810066a8>] xen_alloc_ptpage+0x68/0x6d
> [   42.646973]  [<ffffffff810066db>] xen_alloc_pte+0xe/0x10
> [   42.646989]  [<ffffffff810e048f>] __pte_alloc+0x72/0xd7
> [   42.647003]  [<ffffffff810e2717>] handle_mm_fault+0x131/0x8c1
> [   42.647017]  [<ffffffff810e6e36>] ? mmap_region+0x391/0x4bf
> [   42.647033]  [<ffffffff8100eb84>] ? 
> arch_get_unmapped_area_topdown+0x4a/0x28a
> [   42.647052]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
> [   42.647059]  [<ffffffff8102b78a>] ? pvclock_clocksource_read+0x4b/0xb4
> [   42.647059]  [<ffffffff81007c09>] ? __spin_time_accum+0x21/0x37
> [   42.647059]  [<ffffffff81007e85>] ? __xen_spin_lock+0xb7/0xcd
> [   42.647059]  [<ffffffff81443615>] page_fault+0x25/0x30
> [   42.647059] Code: ec 20 89 7d e0 48 89 f7 e8 c0 ff ff ff 48 8d 7d
> e0 48 89 45 e8 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 76 c7 ff ff
> 85 c0 74 04 <0f> 0b eb fe c9 c3 55 48 89 e5 41 54 49 89 fc 53 31 db 48
> 8b 3d
> [   42.647059] RIP  [<ffffffff81004bce>] pin_pagetable_pfn+0x31/0x37
> [   42.647059]  RSP <ffff88007595bce8>
> [   42.647059] ---[ end trace 3fbbffee414a5166 ]---
>

Once more, this time with CONFIG_XEN_PVHVM=n just in case it was
related to Stefano's PVonHVM work.

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version
2.6.37-rc1-kernel.org-domu-6.0.nopvhvm.bedge-debug (root@opal) (gcc
version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #1 SMP Wed Nov 10 12:01:55 PST
2010
[    0.000000] Command line:
root=UUID=e4e223aa-080e-421b-98ac-ab0b3e0e57e5 ro   swiotlb=force
noirqbalance noirqbalance console=hvc0,115200n8 --flow_control=r
ip=:127.0.255.255::::eth0:dhcp
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] released 0 pages of unused memory
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000080800000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x80800 max_arch_pfn = 0x400000000
[    0.000000] Scanning 0 areas for low memory corruption
[    0.000000] init_memory_mapping: 0000000000000000-0000000080800000
[    0.000000] RAMDISK: 01d95000 - 02faa000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at 0000000000000000-0000000080800000
[    0.000000] Initmem setup node 0 0000000000000000-0000000080800000
[    0.000000]   NODE_DATA [000000007fffb000 - 000000007fffffff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   empty
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x00080800
[    0.000000] SMP: Allowing 6 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.1-unstable (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32
nr_cpu_ids:6 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88007ff40000 s82752
r8192 d23744 u114688
[    0.000000] pcpu-alloc: s82752 r8192 d23744 u114688 alloc=28*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.
Total pages: 519022
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line:
root=UUID=e4e223aa-080e-421b-98ac-ab0b3e0e57e5 ro   swiotlb=force
noirqbalance noirqbalance console=hvc0,115200n8 --flow_control=r
ip=:127.0.255.255::::eth0:dhcp
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Placing 64MB software IO TLB between ffff880079400000 -
ffff88007d400000
[    0.000000] software IO TLB at phys 0x79400000 - 0x7d400000
[    0.000000] Memory: 1959892k/2105344k available (4398k kernel code,
448k absent, 145004k reserved, 6327k data, 756k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0,
CPUs=6, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU-based detection of stalled CPUs is disabled.
[    0.000000] NR_IRQS:4352 nr_irqs:320 16
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] installing Xen timer for CPU 0
[    0.000000] Detected 2660.064 MHz processor.
[    0.000999] Calibrating delay loop (skipped), value calculated
using timer frequency.. 5320.12 BogoMIPS (lpj=2660064)
[    0.000999] pid_max: default: 32768 minimum: 301
[    0.000999] Security Framework initialized
[    0.000999] SELinux:  Initializing.
[    0.000999] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.001365] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.001631] Mount-cache hash table entries: 256
[    0.002177] Initializing cgroup subsys ns
[    0.002191] ns_cgroup deprecated: consider using the
'clone_children' flag without the ns_cgroup.
[    0.002203] Initializing cgroup subsys cpuacct
[    0.002214] Initializing cgroup subsys freezer
[    0.002342] CPU: Unsupported number of siblings 16
[    0.002348] Performance Events: unsupported p6 CPU model 26 no PMU
driver, software events only.
[    0.004120] cpu 0 spinlock event irq 318
[    0.004755] NMI watchdog failed to create perf event on cpu0:
fffffffffffffffe
[    0.004999] installing Xen timer for CPU 1
[    0.005056] cpu 1 spinlock event irq 312
[    0.005093] SMP alternatives: switching to SMP code
[    0.000999] CPU: Unsupported number of siblings 16
[    0.006155] NMI watchdog failed to create perf event on cpu1:
fffffffffffffffe
[    0.006471] installing Xen timer for CPU 2
[    0.006582] cpu 2 spinlock event irq 306
[    0.000999] CPU: Unsupported number of siblings 16
[    0.006810] NMI watchdog failed to create perf event on cpu2:
fffffffffffffffe
[    0.006999] installing Xen timer for CPU 3
[    0.007033] cpu 3 spinlock event irq 300
[    0.000999] CPU: Unsupported number of siblings 16
[    0.007353] NMI watchdog failed to create perf event on cpu3:
fffffffffffffffe
[    0.007587] installing Xen timer for CPU 4
[    0.007626] cpu 4 spinlock event irq 294
[    0.000999] CPU: Unsupported number of siblings 16
[    0.007854] NMI watchdog failed to create perf event on cpu4:
fffffffffffffffe
[    0.008032] installing Xen timer for CPU 5
[    0.008072] cpu 5 spinlock event irq 288
[    0.000999] CPU: Unsupported number of siblings 16
[    0.008391] NMI watchdog failed to create perf event on cpu5:
fffffffffffffffe
[    0.008402] Brought up 6 CPUs
[    0.009156] kworker/u:0 used greatest stack depth: 5696 bytes left
[    0.011444] Grant table initialized
[    0.030860] Time: 165:165:165  Date: 165/165/65
[    0.031124] NET: Registered protocol family 16
[    0.036023] PCI: setting up Xen PCI frontend stub
[    0.037149] kworker/u:0 used greatest stack depth: 5528 bytes left
[    0.063224] bio: create slab <bio-0> at 0
[    0.064258] ACPI: Interpreter disabled.
[    0.065066] xen_balloon: Initialising balloon driver.
[    0.066380] vgaarb: loaded
[    0.067219] usbcore: registered new interface driver usbfs
[    0.068076] usbcore: registered new interface driver hub
[    0.069160] PCI: System does not support PCI
[    0.070062] PCI: System does not support PCI
[    0.071229] NetLabel: Initializing
[    0.071229] NetLabel:  domain hash size = 128
[    0.071229] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.071271] NetLabel:  unlabeled traffic allowed by default
[    0.072589] Switching to clocksource xen
[    0.086952] pnp: PnP ACPI: disabled
[    0.110928] NET: Registered protocol family 2
[    0.111286] IP route cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.112870] TCP established hash table entries: 262144 (order: 10,
4194304 bytes)
[    0.113959] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.114223] TCP: Hash tables configured (established 262144 bind 65536)
[    0.114235] TCP reno registered
[    0.114348] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[    0.114454] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[    0.115033] NET: Registered protocol family 1
[    0.139648] RPC: Registered udp transport module.
[    0.139656] RPC: Registered tcp transport module.
[    0.139662] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.139898] Trying to unpack rootfs image as initramfs...
[    0.179608] Freeing initrd memory: 18516k freed
[    0.267847] DMA-API: preallocated 32768 debug entries
[    0.267860] DMA-API: debugging enabled by kernel config
[    0.270280] platform rtc_cmos: registered platform RTC device (no
PNP device found)
[    0.284949] Machine check injector initialized
[    0.294390] microcode: CPU0 sig=0x106a4, pf=0x2, revision=0xa
[    0.294463] microcode: CPU1 sig=0x106a4, pf=0x2, revision=0xa
[    0.294516] microcode: CPU2 sig=0x106a4, pf=0x2, revision=0xa
[    0.294584] microcode: CPU3 sig=0x106a4, pf=0x2, revision=0xa
[    0.294680] microcode: CPU4 sig=0x106a4, pf=0x2, revision=0xa
[    0.294768] microcode: CPU5 sig=0x106a4, pf=0x2, revision=0xa
[    0.295370] microcode: Microcode Update Driver: v2.00
<tigran@xxxxxxxxxxxxxxxxxxxx>, Peter Oruba
[    0.295387] Scanning for low memory corruption every 60 seconds
[    0.297945] audit: initializing netlink socket (disabled)
[    0.298034] type=2000 audit(1289404539.466:1): initialized
[    0.371706] VFS: Disk quotas dquot_6.5.2
[    0.372818] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.378043] NTFS driver 2.1.29 [Flags: R/W].
[    0.379946] msgmni has been set to 3864
[    0.384845] Block layer SCSI generic (bsg) driver version 0.4
loaded (major 253)
[    0.384862] io scheduler noop registered
[    0.384871] io scheduler deadline registered
[    0.385499] io scheduler cfq registered (default)
[    0.387281] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.401306] pcifront pci-0: Installing PCI frontend
[    0.402048] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    0.411423] pcifront pci-0: claiming resource 0000:00:00.0/0
[    0.411423] pcifront pci-0: claiming resource 0000:00:00.0/2
[    0.411423] pcifront pci-0: claiming resource 0000:00:00.0/3
[    0.411423] pcifront pci-0: claiming resource 0000:00:00.0/4
[    0.411423] pcifront pci-0: claiming resource 0000:00:01.0/0
[    0.411423] pcifront pci-0: claiming resource 0000:00:01.0/2
[    0.411423] pcifront pci-0: claiming resource 0000:00:01.0/3
[    0.411423] pcifront pci-0: claiming resource 0000:00:01.0/4
[    0.411423] pcifront pci-0: claiming resource 0000:00:02.0/0
[    0.411423] pcifront pci-0: claiming resource 0000:00:02.0/2
[    0.411423] pcifront pci-0: claiming resource 0000:00:02.0/3
[    0.411423] pcifront pci-0: claiming resource 0000:00:02.0/4
[    0.411423] pcifront pci-0: claiming resource 0000:00:03.0/0
[    0.411423] pcifront pci-0: claiming resource 0000:00:03.0/2
[    0.411423] pcifront pci-0: claiming resource 0000:00:03.0/3
[    0.411423] pcifront pci-0: claiming resource 0000:00:03.0/4
[    0.718875] Non-volatile memory driver v1.3
[    0.718890] Linux agpgart interface v0.103
[    0.720286] [drm] Initialized drm 1.1.0 20060810
[    0.720307] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.816097] brd: module loaded
[    0.829618] loop: module loaded
[    0.830819] Fixed MDIO Bus: probed
[    0.833373] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.833977] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.836341] usbcore: registered new interface driver usblp
[    0.836850] usbcore: registered new interface driver libusual
[    0.838519] PNP: No PS/2 controller found. Probing ports directly.
[    0.839365] i8042.c: No controller found.
[    0.840021] mice: PS/2 mouse device common for all mice
[    0.842821] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.843646] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.845545] cpuidle: using governor ladder
[    0.845558] cpuidle: using governor menu
[    0.845645] Netfilter messages via NETLINK v0.30.
[    0.845732] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    0.847527] ctnetlink v0.93: registering with nfnetlink.
[    0.849858] ip_tables: (C) 2000-2006 Netfilter Core Team
[    0.850277] TCP cubic registered
[    0.850288] Initializing XFRM netlink socket
[    0.852995] NET: Registered protocol family 10
[    0.856904] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    0.857303] IPv6 over IPv4 tunneling driver
[    0.861148] NET: Registered protocol family 17
[    0.861376] Registering the dns_resolver key type
[    0.863749] registered taskstats version 1
[    0.864020] kmemleak: Kernel memory leak detector initialized
[    0.864035] kmemleak: Automatic memory scanning thread started
[    0.866079] XENBUS: Device with no driver: device/vbd/51712
[    0.866079] XENBUS: Device with no driver: device/vif/0
[    0.866079] XENBUS: Device with no driver: device/console/0
[    0.866079]   Magic number: 1:252:3141

[   13.380040] Freeing unused kernel memory: 756k freed
[   13.380152] Write protecting the kernel read-only data: 10240k
[   13.390107] Freeing unused kernel memory: 1724k freed
[   13.391580] Freeing unused kernel memory: 1992k freed


Loading, please wait...
[   13.484058] exe used greatest stack depth: 5360 bytes left
[   13.510024] udevd (1111): /proc/1111/oom_adj is deprecated, please
use /proc/1111/oom_score_adj instead.
Begin: Loading essential drivers... ...
[   13.533300] blkid used greatest stack depth: 5088 bytes left
[   13.564959] blkfront: xvda: barriers disabled
[   13.576723]  xvda: xvda1 xvda2 < xvda5 >
[   13.639471] blkid used greatest stack depth: 4752 bytes left
[   13.662716] Initialising Xen virtual ethernet driver.
Done.
Begin: Running /scripts/init-premount ...
Done.
Begin: Mounting root file system... ...
Begin: Running /scripts/local-top ...
Done.
Begin: Running /scripts/local-premount ...
Done.
[   13.927847] EXT4-fs (xvda1): mounted filesystem with ordered data
mode. Opts: (null)
Begin: Running /scripts/local-bottom ...
Done.
Done.
Begin: Running /scripts/init-bottom ...
Done.
[   14.608840] hostname used greatest stack depth: 4464 bytes left

fsck from util-linux-ng 2.17.2
e2fsck 1.41.11 (14-Mar-2010)
Superblock last write time is in the future.
        (by less than a day, probably due to the hardware clock being
incorrectly set).  Fix? yes

/dev/xvda1: clean, 55961/288576 files, 289520/1153280 blocks
 * Enable login on console                                               [ OK ]
 * Check for hostname changes...                                         [ OK ]
   ...done.
 * Setting sensors limits
         No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.
No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.
                                                                         [ OK ]
 * Initializing random number generator...                               [ OK ]
 * Setting up X server socket directory /tmp/.X11-unix...                [ OK ]
 * Not starting fancontrol; run pwmconfig first.
 * Starting internet superserver inetd                                   [ OK ]
Starting openntpd: ntpd.
 * Starting network management services:
          * Starting the system activity data collector sadc

Cannot open /var/log/sysstat/sa10: No such file or directory
                                                                         [fail]
 * Setting core file location...                                         [ OK ]
 * Restoring resolver state...                                           [ OK ]
 * Checking tomcat configuration
   ...done.
 * Starting Tomcat servlet engine tomcat6
         Using CATALINA_BASE:   /var/lib/tomcat6
Using CATALINA_HOME:   /usr/share/tomcat6
Using CATALINA_TMPDIR: /tmp/tomcat6-tmp
Using JRE_HOME:        /usr/lib/jvm/java-6-openjdk
Using CLASSPATH:       /usr/share/tomcat6/bin/bootstrap.jar
                                                                         [ OK ]
collectd has been disabled - see /etc/default/collectd.
 * Recording successful boot for GRUB
   ...done.
 * Running local boot scripts (/etc/rc.local)                            [ OK ]
mountall: Plymouth command failed
mountall: Disconnected from Plymouth

6.0.16-01
kaan-04-dpm login: [  144.548118] BUG: unable to handle kernel paging
request at ffff88007733d508
[  144.548145] IP: [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
[  144.548167] PGD 1a04067 PUD 3dd7067 PMD 3f91067 PTE 801000007733d065
[  144.548194] Oops: 0003 [#1] SMP
[  144.548211] last sysfs file: /sys/kernel/uevent_seqnum
[  144.548222] CPU 3
[  144.548228] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront
fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront
xen_netfront xen_blkfront
[  144.548345]
[  144.548354] Pid: 3938, comm: sh Not tainted
2.6.37-rc1-kernel.org-domu-6.0.nopvhvm.bedge-debug #1 /
[  144.548367] RIP: e030:[<ffffffff810355f3>]  [<ffffffff810355f3>]
ptep_set_access_flags+0x27/0x4d
[  144.548395] RAX: 800000005adc6167 RBX: ffff880077403b80 RCX: 800000005adc6167
[  144.548408] RDX: ffff88007733d508 RSI: 00007f52324a19d0 RDI: ffff880077403b80
[  144.548420] RBP: ffff880077cabc08 R08: 0000000000000001 R09: ffffea0001a13568
[  144.548431] R10: 0000000000000100 R11: ffffffff00000001 R12: 00007f52324a19d0
[  144.548443] R13: 0000000000000001 R14: 0000000000000000 R15: ffff880077403b80
[  144.548495] FS:  00007f52324a1700(0000) GS:ffff88007ff94000(0000)
knlGS:0000000000000000
[  144.548509] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  144.548520] CR2: ffff88007733d508 CR3: 00000000783ad000 CR4: 0000000000002660
[  144.548531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  144.548543] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  144.548556] Process sh (pid: 3938, threadinfo ffff880077caa000,
task ffff880077a615f0)
[  144.548568] Stack:
[  144.548575]  0000000000000100 ffffea0001a13568 f000000000000000
0000000000000000
[  144.548602]  ffff88007733d508 ffffea0001a13568 ffff880077cabca8
ffffffff810e2200
[  144.548630]  ffff880077cabc38 ffffffff81007c09 ffffea0001a13568
0000000000000001
[  144.548657] Call Trace:
[  144.548672]  [<ffffffff810e2200>] do_wp_page+0x2d8/0x6be
[  144.548688]  [<ffffffff81007c09>] ? __spin_time_accum+0x21/0x37
[  144.548702]  [<ffffffff81007e85>] ? __xen_spin_lock+0xb7/0xcd
[  144.548716]  [<ffffffff810e2e60>] handle_mm_fault+0x87a/0x8c1
[  144.548732]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
[  144.548747]  [<ffffffff8100ba0b>] ? xen_hypervisor_callback+0x1b/0x20
[  144.548762]  [<ffffffff8144384a>] ? error_exit+0x2a/0x60
[  144.548776]  [<ffffffff8144339d>] ? retint_restore_args+0x5/0x6
[  144.548792]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1006
[  144.548806]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1006
[  144.548821]  [<ffffffff81443615>] page_fault+0x25/0x30
[  144.548837]  [<ffffffff81206add>] ? __put_user_4+0x1d/0x30
[  144.548851]  [<ffffffff810498a6>] ? schedule_tail+0x5c/0x60
[  144.548865]  [<ffffffff8100a9c3>] ret_from_fork+0x13/0x80
[  144.548875] Code: 41 5d c9 c3 55 48 89 e5 41 55 45 31 ed 41 54 49
89 f4 53 48 89 fb 48 83 ec 18 48 39 0a 41 0f 95 c5 45 85 c0 74 1d 45
85 ed 74 18 <48> 89 0a 48 8b 3f 0f 1f 80 00 00 00 00 4c 89 e6 48 89 df
e8 b2
[  144.549091] RIP  [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
[  144.549091]  RSP <ffff880077cabbd8>
[  144.549091] CR2: ffff88007733d508
[  144.549091] ---[ end trace e2156ffb7bd19b0d ]---


This time it also generated a stack dump on dom0

BUG: scheduling while atomic: xenwatch/12/0x00000002
Modules linked in: xt_physdev ipmi_msghandler ipv6 osa_mfgdom0
xen_evtchn tun bridge stp serio_raw joydev llc ioatdma dca usb_storage
e1000e
Pid: 12, comm: xenwatch Not tainted
2.6.32.24-pvops-stable-dom0-5.3.bedge-debug #1
Call Trace:
 [<ffffffff810424d7>] __schedule_bug+0x5c/0x61
 [<ffffffff814964ac>] schedule+0xea/0x84d
 [<ffffffff8100f5dd>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff8100fdff>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff814989e5>] ? _spin_unlock_irqrestore+0x1d/0x40
 [<ffffffff812b76bb>] read_reply+0x9c/0x13a
 [<ffffffff8105ea57>] ? autoremove_wake_function+0x0/0x34
 [<ffffffff812b78bf>] xs_talkv+0xab/0x16d
 [<ffffffff81010be2>] ? xen_spin_lock+0xc7/0xda
 [<ffffffff812b7c9a>] unregister_xenbus_watch+0xdb/0x1e7
 [<ffffffff812bd303>] pciback_xenbus_remove+0x33/0x75
 [<ffffffff812b8bba>] xenbus_dev_remove+0x44/0x5b
 [<ffffffff812fead4>] __device_release_driver+0x70/0xc3
 [<ffffffff812febf5>] device_release_driver+0x1e/0x2b
 [<ffffffff812fe17c>] bus_remove_device+0x96/0xa8
 [<ffffffff812fc8ca>] device_del+0x135/0x1a2
 [<ffffffff812fc948>] device_unregister+0x11/0x1e
 [<ffffffff812b89c8>] xenbus_dev_changed+0x92/0x186
 [<ffffffff812b8f0a>] backend_changed+0x16/0x18
 [<ffffffff812b75e4>] xenwatch_thread+0x123/0x15e
 [<ffffffff8105ea57>] ? autoremove_wake_function+0x0/0x34
 [<ffffffff812b74c1>] ? xenwatch_thread+0x0/0x15e
 [<ffffffff8105e7c5>] kthread+0x7a/0x82
 [<ffffffff81014bda>] child_rip+0xa/0x20
 [<ffffffff81013d61>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff81014521>] ? retint_restore_args+0x5/0x6
 [<ffffffff81014bd0>] ? child_rip+0x0/0x20
pciback 0000:01:00.0: free-ing dynamically allocated virtual
configuration space fields
pciback 0000:01:00.0: resetting virtual configuration space
pciback 0000:01:00.1: free-ing dynamically allocated virtual
configuration space fields
pciback 0000:01:00.1: resetting virtual configuration space
pciback 0000:04:00.0: free-ing dynamically allocated virtual
configuration space fields
pciback 0000:04:00.0: resetting virtual configuration space
pciback 0000:04:00.1: free-ing dynamically allocated virtual
configuration space fields
pciback 0000:04:00.1: resetting virtual configuration space

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.