Re: [Xen-devel] 2.6.37-rc1 mainline domU - BUG: unable to handle kernel paging request

[Boris Derzhavets <bderzhavets@xxxxxxxxx>]

> The domU console output hung up prior to spitting out a prompt post root login.
>  I logged into the domU via ssh
Ubuntu 10.10 HVM image has been used to test 2.6.37-rc1 pvops DomU
File /etc/init/hvc0 has been prepared in HVM mode to login from command line.

PV Guest config

root@ServerLLX:/home/boris/ubuntu# cat u10.cfg
name="Ubuntu10PV"
memory = 2048
kernel = "/home/boris/ubuntu/vmlinuz-2.6.37-rc1"
ramdisk = "/home/boris/ubuntu/initrd.img-2.6.37-rc1"
disk = ['phy:/dev/sda7,xvda,w']
vif  = ['bridge=br0']
extra = "root=/dev/xvda1 ro console=hvc0"
# vfb =  ['type=vnc,vncunused=1']

root@ServerLLX:/home/boris/ubuntu# xm create -c  u10.cfg
Using config file "./u10.cfg".
Started domain Ubuntu10PV (id=4)
                                Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.37-rc1 (root@ServerLLX) (gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #1 SMP Fri Nov 5 19:29:09 MSK 2010
Command line: root=/dev/xvda1 ro console=hvc0
ACPI in unprivileged domain disabled
released 0 pages of unused memory
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000080800000 (usable)
NX (Execute Disable) protection: active
DMI not present or invalid.
No AGP bridge found
last_pfn = 0x80800 max_arch_pfn = 0x400000000
init_memory_mapping: 0000000000000000-0000000080800000
RAMDISK: 01e98000 - 0e9fa000
No NUMA configuration found
Faking a node at 0000000000000000-0000000080800000
Initmem setup node 0 0000000000000000-0000000080800000
  NODE_DATA [000000007ffec000 - 000000007fffffff]
Zone PFN ranges:
  DMA      0x00000010 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   empty
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000010 -> 0x000000a0
    0: 0x00000100 -> 0x00080800
SMP: Allowing 1 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
APIC: switched to apic NOOP
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
Allocating PCI resources starting at 80800000 (gap: 80800000:7f800000)
Booting paravirtualized kernel on Xen
Xen version: 4.0.1 (preserve-AD)
setup_percpu: NR_CPUS:256 nr_cpumask_bits:256 nr_cpu_ids:1 nr_node_ids:1
PERCPU: Embedded 28 pages/cpu @ffff88007ffbd000 s82752 r8192 d23744 u114688
pcpu-alloc: s82752 r8192 d23744 u114688 alloc=28*4096
pcpu-alloc: [0] 0
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 519022
Policy zone: DMA32
Kernel command line: root=/dev/xvda1 ro console=hvc0
PID hash table entries: 4096 (order: 3, 32768 bytes)
Checking aperture...
No AGP bridge found
Memory: 1835560k/2105344k available (4462k kernel code, 448k absent, 269336k reserved, 6442k data, 1564k init)
Hierarchical RCU implementation.
    RCU-based detection of stalled CPUs is disabled.
NR_IRQS:16640 nr_irqs:256 16
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 22282240 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
installing Xen timer for CPU 0
Detected 2833.068 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 5666.13 BogoMIPS (lpj=2833068)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys debug
Initializing cgroup subsys ns
ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
CPU: Unsupported number of siblings 4
Performance Events: unsupported p6 CPU model 23 no PMU driver, software events only.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 20k freed
cpu 0 spinlock event irq 254
Brought up 1 CPUs
Grant table initialized
regulator: core version 0.5
regulator: dummy:
Time: 165:165:165  Date: 165/165/65
NET: Registered protocol family 16
PCI: setting up Xen PCI frontend stub
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
IP route cache hash table entries: 65536 (order: 7, 524288 bytes)
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
UDP hash table entries: 1024 (order: 3, 32768 bytes)
UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 208264k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1289472066.106:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 3991
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
pci-stub: invalid id string ""
Event-channel device installed.
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
Fixed MDIO Bus: probed
Initialising Xen virtual ethernet driver.
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
i8042.c: No controller found.
mice: PS/2 mouse device common for all mice
rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
rtc_cmos: probe of rtc_cmos failed with error -38
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.18.0-ioctl (2010-06-29) initialised: dm-devel@xxxxxxxxxx
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
IPVS: Registered protocols (TCP, UDP, AH, ESP)
IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
IPVS: ipvs loaded.
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
Bridge firewalling registered
Ebtables v2.0 registered
Registering the dns_resolver key type
registered taskstats version 1
IMA: No TPM chip found, activating TPM-bypass!
blkfront: xvda: barriers disabled
 xvda: xvda1 xvda2
XENBUS: Device with no driver: device/console/0
  Magic number: 1:252:3141
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Initalizing network drop monitor service
Freeing unused kernel memory: 1564k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 1664k freed
Freeing unused kernel memory: 1904k freed
Loading, please wait...
mount: mounting none on /dev failed: No such device
udevd (63): /proc/63/oom_adj is deprecated, please use /proc/63/oom_score_adj instead.
Begin: Loading essential drivers... ...
Done.
Begin: Running /scripts/init-premount ...
Done.
Begin: Mounting root file system... ...
Begin: Running /scripts/local-top ...
Done.
Begin: Running /scripts/local-premount ...
Done.
EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: (null)
Begin: Running /scripts/local-bottom ...
Done.
Done.
Begin: Running /scripts/init-bottom ...
Done.

Ubuntu 10.10 boris-HVM-domU hvc0

boris-HVM-domU login: root
Password:
Linux boris-HVM-domU 2.6.37-rc1 #1 SMP Fri Nov 5 19:29:09 MSK 2010 x86_64 GNU/Linux
Ubuntu 10.10

Welcome to Ubuntu!
 * Documentation:  https://help.ubuntu.com/

root@boris-HVM-domU:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1             12G  3.9G  7.1G  36% /
none                 1001M  116K 1001M   1% /dev
none                 1001M     0 1001M   0% /dev/shm
none                 1001M   84K 1001M   1% /var/run
none                 1001M     0 1001M   0% /var/lock

Boris.

--- On Wed, 11/10/10, Bruce Edge <bruce.edge@xxxxxxxxx> wrote:

From: Bruce Edge <bruce.edge@xxxxxxxxx>
Subject: [Xen-devel] 2.6.37-rc1 mainline domU - BUG: unable to handle kernel paging request
To: xen-devel@xxxxxxxxxxxxxxxxxxx, "Jeremy Fitzhardinge" <jeremy@xxxxxxxx>, "Konrad Rzeszutek Wilk" <konrad.wilk@xxxxxxxxxx>
Date: Wednesday, November 10, 2010, 5:15 PM

This is a mainline 2.6.37-rc1 pvops domU on top of a xen-unstable build 22241 and xen/stable-2.6.32.x 2.6.32.24 pvops dom0.

The domU console output hung up prior to spitting out a prompt post root login. I logged into the domU via ssh and ran an 'ls' of an NFS mounted dir and
this immediately came out on the domU console:

[  480.467525] BUG: unable to handle kernel paging request at ffff880077e9f030
[  480.467547] IP: [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
[  480.467564] PGD 1a04067 PUD 3dd7067 PMD 3f97067 PTE 8010000077e9f065
[  480.467587] Oops: 0003 [#1] SMP
[  480.467601] last sysfs file: /sys/kernel/uevent_seqnum
[  480.467611] CPU 3
[  480.467616] Modules linked in: ext4 mbcache jbd2 crc16 xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xen_netfront xen_blkfront
[  480.467676]
[  480.467684] Pid: 3957, comm: zsh Not tainted 2.6.37-rc1-kernel.org-domu-6.0.bedge-debug #1 /
[  480.467695] RIP: e030:[<ffffffff810355f3>]  [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
[  480.467710] RSP: e02b:ffff880077ebfbd8  EFLAGS: 00010202
[  480.467719] RAX: 800000004d686167 RBX: ffff8800763ddb80 RCX: 800000004d686167
[  480.467729] RDX: ffff880077e9f030 RSI: 00007f5cb26069d0 RDI: ffff8800763ddb80
[  480.467739] RBP: ffff880077ebfc08 R08: 0000000000000001 R09: ffffea0001a3b2d8
[  480.467748] R10: 0000000000000100 R11: 0000000000000000 R12: 00007f5cb26069d0
[  480.467759] R13: 0000000000000001 R14: 0000000000000000 R15: ffff8800763ddb80
[  480.467772] FS:  00007f5cb2606700(0000) GS:ffff88007ff94000(0000) knlGS:0000000000000000
[  480.467783] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  480.467792] CR2: ffff880077e9f030 CR3: 0000000077dbc000 CR4: 0000000000002660
[  480.467802] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  480.467812] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  480.467822] Process zsh (pid: 3957, threadinfo ffff880077ebe000, task ffff880078b80000)
[  480.467832] Stack:
[  480.467838]  0000000000000100 ffffea0001a3b2d8 7000000000000000 0000000000000000
[  480.467862]  ffff880077e9f030 ffffea0001a3b2d8 ffff880077ebfca8 ffffffff810e2200
[  480.467886]  ffff880077ebfc38 ffffffff81007c09 ffffea0001a3b2d8 0000000000000001
[  480.467910] Call Trace:
[  480.467921]  [<ffffffff810e2200>] do_wp_page+0x2d8/0x6be
[  480.467934]  [<ffffffff81007c09>] ? __spin_time_accum+0x21/0x37
[  480.467946]  [<ffffffff81007e85>] ? __xen_spin_lock+0xb7/0xcd
[  480.467958]  [<ffffffff810e2e60>] handle_mm_fault+0x87a/0x8c1
[  480.467972]  [<ffffffff814463fe>] do_page_fault+0x374/0x396
[  480.467984]  [<ffffffff81006c3d>] ? xen_force_evtchn_callback+0xd/0xf
[  480.467996]  [<ffffffff81006c3d>] ? xen_force_evtchn_callback+0xd/0xf
[  480.468009]  [<ffffffff810072a2>] ? check_events+0x12/0x20
[  480.468020]  [<ffffffff8100728f>] ? xen_restore_fl_direct_end+0x0/0x1
[  480.468034]  [<ffffffff810a6fa4>] ? __call_rcu+0x123/0x12c
[  480.468045]  [<ffffffff81006c3d>] ? xen_force_evtchn_callback+0xd/0xf
[  480.468059]  [<ffffffff810072a2>] ? check_events+0x12/0x20
[  480.468072]  [<ffffffff81443615>] page_fault+0x25/0x30
[  480.468085]  [<ffffffff81206add>] ? __put_user_4+0x1d/0x30
[  480.468096]  [<ffffffff810498a6>] ? schedule_tail+0x5c/0x60
[  480.468108]  [<ffffffff8100a9c3>] ret_from_fork+0x13/0x80
[  480.468116] Code: 41 5d c9 c3 55 48 89 e5 41 55 45 31 ed 41 54 49 89 f4 53 48 89 fb 48 83 ec 18 48 39 0a 41 0f 95 c5 45 85 c0 74 1d 45 85 ed 74 18 <48> 89 0a 48 8b 3f 0f 1f 80 00 00 00 00 4c 89 e6 48 89 df e8 b2
[  480.468323] RIP  [<ffffffff810355f3>] ptep_set_access_flags+0x27/0x4d
[  480.468323]  RSP <ffff880077ebfbd8>
[  480.468323] CR2: ffff880077e9f030
[  480.468323] ---[ end trace 6d1623321ecd50ac ]---

The domU config is attached as config.2.6.37_domU_debug


-Bruce

-----Inline Attachment Follows-----

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel