[Xen-devel] Re: Xen question

[Stephen Hemminger <stephen.hemminger@xxxxxxxxxx>]

It seems to happen only on i386. Could be either high PTE in config
or BIGSMP. Anyway, changing config fixed it.

----- Original Message -----
> On 02/01/2011 10:49 AM, Stephen Hemminger wrote:
> > Something is wrong with the Vyatta Xen kernel.
> > It worked with 2.6.32, but something wrong with 2.6.35.
> > Have you seen this?
> >
> > -- [ 0.000000] Reserving virtual address space above 0xf5800000
> > [ 0.000000] Initializing cgroup subsys cpuset
> > [ 0.000000] Initializing cgroup subsys cpu
> > [ 0.000000] Linux version 2.6.35-1-586-vyatta-virt (root@vyatta)
> > (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Jan 28 16:40:27 PST
> > 2011 [ 0.000000] ACPI in unprivileged domain disabled
> > [ 0.000000] BIOS-provided physical RAM map:
> > [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable)
> > [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved)
> > [ 0.000000] Xen: 0000000000100000 - 0000000020000000 (usable)
> > [ 0.000000] NX (Execute Disable) protection: active
> > [ 0.000000] DMI not present or invalid.
> > [ 0.000000] last_pfn = 0x20000 max_arch_pfn = 0x1000000
> > [ 0.000000] init_memory_mapping: 0000000000000000-0000000020000000
> > [ 0.000000] RAMDISK: 01607000 - 03148000
> > [ 0.000000] Node: 0, start_pfn: 0, end_pfn: 20000
> > [ 0.000000] Reserving total of 600 pages for numa KVA remap
> > [ 0.000000] kva_start_pfn ~ 1f400 max_low_pfn ~ 20000
> > [ 0.000000] max_pfn = 20000
> > [ 0.000000] 0MB HIGHMEM available.
> > [ 0.000000] 512MB LOWMEM available.
> > [ 0.000000] 1 multicall(s) failed: cpu 0
> > [ 0.000000] Pid: 0, comm: swapper Not tainted
> > 2.6.35-1-586-vyatta-virt #1
> > [ 0.000000] Call Trace:
> > [ 0.000000] [<c100415b>] ? xen_mc_flush+0x92/0x13d
> > [ 0.000000] [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> > [ 0.000000] [<c10053f9>] ? xen_set_pmd_hyper+0x55/0x5c
> > [ 0.000000] [<c1025d25>] ? set_pmd_pfn+0xb4/0xc3
> > [ 0.000000] [<c13df48f>] ? initmem_init+0x4e9/0x5ea
> > [ 0.000000] [<c12895ca>] ? printk+0xe/0x14
> > [ 0.000000] [<c13d151c>] ? setup_arch+0x8d1/0x989
> > [ 0.000000] [<c13e0020>] ? proc_caches_init+0x27/0xbf
> > [ 0.000000] [<c13cd61e>] ? start_kernel+0x86/0x330
> > [ 0.000000] [<c13cfabf>] ? xen_start_kernel+0x4c0/0x4c7
> 
> This is failing during the initial mapping of memory, I think. Are
> there any messages on the Xen console? Is it completely deterministic?
> 
> > [ 0.000000] ------------[ cut here ]------------
> > [ 0.000000] WARNING: at arch/x86/xen/multicalls.c:182
> > xen_mc_issue+0x11/0x1f()
> > [ 0.000000] Modules linked in:
> > [ 0.000000] Pid: 0, comm: swapper Not tainted
> > 2.6.35-1-586-vyatta-virt #1
> > [ 0.000000] Call Trace:
> > [ 0.000000] [<c10357ce>] ? warn_slowpath_common+0x6a/0x7b
> > [ 0.000000] [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> > [ 0.000000] [<c10357ec>] ? warn_slowpath_null+0xd/0x10
> > [ 0.000000] [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> > [ 0.000000] [<c10053f9>] ? xen_set_pmd_hyper+0x55/0x5c
> > [ 0.000000] [<c1025d25>] ? set_pmd_pfn+0xb4/0xc3
> > [ 0.000000] [<c13df48f>] ? initmem_init+0x4e9/0x5ea
> > [ 0.000000] [<c12895ca>] ? printk+0xe/0x14
> > [ 0.000000] [<c13d151c>] ? setup_arch+0x8d1/0x989
> > [ 0.000000] [<c13e0020>] ? proc_caches_init+0x27/0xbf
> > [ 0.000000] [<c13cd61e>] ? start_kernel+0x86/0x330
> > [ 0.000000] [<c13cfabf>] ? xen_start_kernel+0x4c0/0x4c7
> > [ 0.000000] ---[ end trace 4eaa2a86a8e2da22 ]---
> > [ 0.000000] 1 multicall(s) failed: cpu 0
> > [ 0.000000] Pid: 0, comm: swapper Tainted: G W
> > 2.6.35-1-586-vyatta-virt #1
> > [ 0.000000] Call Trace:
> > [ 0.000000] [<c100415b>] ? xen_mc_flush+0x92/0x13d
> > [ 0.000000] [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> > [ 0.000000] [<c10053f9>] ? xen_set_pmd_hyper+0x55/0x5c
> > [ 0.000000] [<c1025d25>] ? set_pmd_pfn+0xb4/0xc3
> > [ 0.000000] [<c13df48f>] ? initmem_init+0x4e9/0x5ea
> > [ 0.000000] [<c12895ca>] ? printk+0xe/0x14
> > [ 0.000000] [<c13d151c>] ? setup_arch+0x8d1/0x989
> > [ 0.000000] [<c13e0020>] ? proc_caches_init+0x27/0xbf
> > [ 0.000000] [<c13cd61e>] ? start_kernel+0x86/0x330
> > [ 0.000000] [<c13cfabf>] ? xen_start_kernel+0x4c0/0x4c7
> > [ 0.000000] ------------[ cut here ]------------
> > [ 0.000000] WARNING: at arch/x86/xen/multicalls.c:182
> > xen_mc_issue+0x11/0x1f()
> > [ 0.000000] Modules linked in:
> > [ 0.000000] Pid: 0, comm: swapper Tainted: G W
> > 2.6.35-1-586-vyatta-virt #1
> > [ 0.000000] Call Trace:
> > [ 0.000000] [<c10357ce>] ? warn_slowpath_common+0x6a/0x7b
> > [ 0.000000] [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> > [ 0.000000] [<c10357ec>] ? warn_slowpath_null+0xd/0x10
> > [ 0.000000] [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> > [ 0.000000] [<c10053f9>] ? xen_set_pmd_hyper+0x55/0x5c
> > [ 0.000000] [<c1025d25>] ? set_pmd_pfn+0xb4/0xc3
> > [ 0.000000] [<c13df48f>] ? initmem_init+0x4e9/0x5ea
> > [ 0.000000] [<c12895ca>] ? printk+0xe/0x14
> > [ 0.000000] [<c13d151c>] ? setup_arch+0x8d1/0x989
> > [ 0.000000] [<c13e0020>] ? proc_caches_init+0x27/0xbf
> > [ 0.000000] [<c13cd61e>] ? start_kernel+0x86/0x330
> > [ 0.000000] [<c13cfabf>] ? xen_start_kernel+0x4c0/0x4c7
> > [ 0.000000] ---[ end trace 4eaa2a86a8e2da23 ]---
> > [ 0.000000] 1 multicall(s) failed: cpu 0
> > [ 0.000000] Pid: 0, comm: swapper Tainted: G W
> > 2.6.35-1-586-vyatta-virt #1
> > [ 0.000000] Call Trace:
> > [ 0.000000] [<c100415b>] ? xen_mc_flush+0x92/0x13d
> > [ 0.000000] [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> > [ 0.000000] [<c10053f9>] ? xen_set_pmd_hyper+0x55/0x5c
> > [ 0.000000] [<c1025d25>] ? set_pmd_pfn+0xb4/0xc3
> > [ 0.000000] [<c13df48f>] ? initmem_init+0x4e9/0x5ea
> > [ 0.000000] [<c12895ca>] ? printk+0xe/0x14
> > [ 0.000000] [<c13d151c>] ? setup_arch+0x8d1/0x989
> > [ 0.000000] [<c13e0020>] ? proc_caches_init+0x27/0xbf
> > [ 0.000000] [<c13cd61e>] ? start_kernel+0x86/0x330
> > [ 0.000000] [<c13cfabf>] ? xen_start_kernel+0x4c0/0x4c7
> > [ 0.000000] ------------[ cut here ]------------
> > [ 0.000000] WARNING: at arch/x86/xen/multicalls.c:182
> > xen_mc_issue+0x11/0x1f()
> > [ 0.000000] Modules linked in:
> > [ 0.000000] Pid: 0, comm: swapper Tainted: G W
> > 2.6.35-1-586-vyatta-virt #1
> > [ 0.000000] Call Trace:
> > [ 0.000000] [<c10357ce>] ? warn_slowpath_common+0x6a/0x7b
> > [ 0.000000] [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> > [ 0.000000] [<c10357ec>] ? warn_slowpath_null+0xd/0x10
> > [ 0.000000] [<c1004a39>] ? xen_mc_issue+0x11/0x1f
> > [ 0.000000] [<c10053f9>] ? xen_set_pmd_hyper+0x55/0x5c
> > [ 0.000000] [<c1025d25>] ? set_pmd_pfn+0xb4/0xc3
> > [ 0.000000] [<c13df48f>] ? initmem_init+0x4e9/0x5ea
> > [ 0.000000] [<c12895ca>] ? printk+0xe/0x14
> > [ 0.000000] [<c13d151c>] ? setup_arch+0x8d1/0x989
> > [ 0.000000] [<c13e0020>] ? proc_caches_init+0x27/0xbf
> > [ 0.000000] [<c13cd61e>] ? start_kernel+0x86/0x330
> > [ 0.000000] [<c13cfabf>] ? xen_start_kernel+0x4c0/0x4c7
> > [ 0.000000] ---[ end trace 4eaa2a86a8e2da24 ]---
> > [ 0.000000] mapped low ram: 0 - 20000000
> > [ 0.000000] low ram: 0 - 20000000
> > [ 0.000000] Zone PFN ranges:
> > [ 0.000000] DMA 0x00000001 -> 0x00001000
> > [ 0.000000] Normal 0x00001000 -> 0x00020000
> > [ 0.000000] HighMem empty
> > [ 0.000000] Movable zone start PFN for each node
> > [ 0.000000] early_node_map[2] active PFN ranges
> > [ 0.000000] 0: 0x00000001 -> 0x000000a0
> > [ 0.000000] 0: 0x00000100 -> 0x0001fa00
> > [ 0.000000] Using APIC driver default
> > [ 0.000000] SFI: Simple Firmware Interface v0.81
> > http://simplefirmware.org [ 0.000000] SMP: Allowing 1 CPUs, 0
> > hotplug CPUs
> > [ 0.000000] Local APIC disabled by BIOS -- you can enable it with
> > "lapic" [ 0.000000] APIC: disable apic facility
> > [ 0.000000] APIC: switched to apic NOOP
> > [ 0.000000] PM: Registered nosave memory: 00000000000a0000 -
> > 0000000000100000 [ 0.000000] Allocating PCI resources starting at
> > 20000000 (gap: 20000000:e0000000)
> > [ 0.000000] Booting paravirtualized kernel on Xen
> > [ 0.000000] Xen version: 4.0.1 (preserve-AD)
> > [ 0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1
> > nr_node_ids:8 [ 0.000000] PERCPU: Embedded 15 pages/cpu @c14e7000
> > s37504 r0 d23936 u65536
> > [ 0.000000] pcpu-alloc: s37504 r0 d23936 u65536 alloc=16*4096
> > [ 0.000000] pcpu-alloc: [0] 0
> > [ 0.000000] Xen: using vcpu_info placement
> > [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on.
> > Total pages: 128427
> > [ 0.000000] Policy zone: Normal
> > [ 0.000000] Kernel command line: root=/dev/xvda1 ro
> > ip=:127.0.255.255::::eth0:dhcp [ 0.000000] PID hash table entries:
> > 2048 (order: 1, 8192 bytes)
> > [ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144
> > bytes) [ 0.000000] Inode-cache hash table entries: 32768 (order: 5,
> > 131072 bytes)
> > [ 0.000000] Enabling fast FPU save and restore... done.
> > [ 0.000000] Enabling unmasked SIMD FPU exception support... done.
> > [ 0.000000] Initializing CPU#0
> > [ 0.000000] Subtract (33 early reservations)
> > [ 0.000000] #1 [00031cb000 - 00031e7000] XEN PAGETABLES
> > [ 0.000000] #2 [0000001000 - 0000002000] EX TRAMPOLINE
> > [ 0.000000] #3 [0001000000 - 00014e15d0] TEXT DATA BSS
> > [ 0.000000] #4 [0001607000 - 0003148000] RAMDISK
> > [ 0.000000] #5 [0003148000 - 00031cb000] XEN START INFO
> > [ 0.000000] #6 [0000002000 - 0000003000] TRAMPOLINE
> > [ 0.000000] #7 [0000003000 - 0000007000] ACPI WAKEUP
> > [ 0.000000] #8 [0000100000 - 00001e4000] PGTABLE
> > [ 0.000000] #9 [001fa00000 - 0020000000] KVA RAM
> > [ 0.000000] #10 [001f400000 - 001fa00000] KVA PG
> > [ 0.000000] #11 [00014e2000 - 00014e3000] BOOTMEM
> > [ 0.000000] #12 [00014e1600 - 00014e1604] BOOTMEM
> > [ 0.000000] #13 [00014e1640 - 00014e1700] BOOTMEM
> > [ 0.000000] #14 [00014e1700 - 00014e1730] BOOTMEM
> > [ 0.000000] #15 [00014e3000 - 00014e4800] BOOTMEM
> > [ 0.000000] #16 [00014e1740 - 00014e17d0] BOOTMEM
> > [ 0.000000] #17 [00014e1800 - 00014e1840] BOOTMEM
> > [ 0.000000] #18 [00014e1840 - 00014e1880] BOOTMEM
> > [ 0.000000] #19 [00014e1880 - 00014e18c0] BOOTMEM
> > [ 0.000000] #20 [00014e18c0 - 00014e18d0] BOOTMEM
> > [ 0.000000] #21 [00014e1900 - 00014e1933] BOOTMEM
> > [ 0.000000] #22 [00014e1940 - 00014e1973] BOOTMEM
> > [ 0.000000] #23 [00014e7000 - 00014f6000] BOOTMEM
> > [ 0.000000] #24 [00014e1980 - 00014e1984] BOOTMEM
> > [ 0.000000] #25 [00014e19c0 - 00014e19c4] BOOTMEM
> > [ 0.000000] #26 [00014e1a00 - 00014e1a04] BOOTMEM
> > [ 0.000000] #27 [00014e1a40 - 00014e1a44] BOOTMEM
> > [ 0.000000] #28 [00014e1a80 - 00014e1b00] BOOTMEM
> > [ 0.000000] #29 [00014e1b00 - 00014e1b2c] BOOTMEM
> > [ 0.000000] #30 [00014e4800 - 00014e6800] BOOTMEM
> > [ 0.000000] #31 [00014f6000 - 0001536000] BOOTMEM
> > [ 0.000000] #32 [0001536000 - 0001556000] BOOTMEM
> > [ 0.000000] Initializing HighMem for node 0 (00000000:00000000)
> > [ 0.000000] Memory: 476668k/524288k available (2608k kernel code,
> > 41088k reserved, 1281k data, 512k init, 0k highmem)
> > [ 0.000000] virtual kernel memory layout:
> > [ 0.000000] fixmap : 0xf5536000 - 0xf57ff000 (2852 kB)
> > [ 0.000000] pkmap : 0xf5200000 - 0xf5400000 (2048 kB)
> > [ 0.000000] vmalloc : 0xe0800000 - 0xf51fe000 ( 329 MB)
> > [ 0.000000] lowmem : 0xc0000000 - 0xe0000000 ( 512 MB)
> > [ 0.000000] .init : 0xc13cd000 - 0xc144d000 ( 512 kB)
> > [ 0.000000] .data : 0xc128c1b2 - 0xc13cc918 (1281 kB)
> > [ 0.000000] .text : 0xc1000000 - 0xc128c1b2 (2608 kB)
> > [ 0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0,
> > CPUs=1, Nodes=8
> > [ 0.000000] Hierarchical RCU implementation.
> > [ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
> > [ 0.000000] RCU-based detection of stalled CPUs is disabled.
> > [ 0.000000] Verbose stalled-CPUs detection is disabled.
> > [ 0.000000] NR_IRQS:2304 nr_irqs:256
> > [ 0.000000] Console: colour dummy device 80x25
> > [ 0.000000] console [tty0] enabled
> > [ 0.000000] console [hvc0] enabled
> > [ 0.000000] installing Xen timer for CPU 0
> > [ 0.000000] Detected 1994.378 MHz processor.
> > [ 0.010000] Calibrating delay loop (skipped), value calculated using
> > timer frequency.. 3988.75 BogoMIPS (lpj=19943780)
> > [ 0.010000] pid_max: default: 32768 minimum: 301
> > [ 0.010000] Security Framework initialized
> > [ 0.010000] SELinux: Disabled at boot.
> > [ 0.010000] Mount-cache hash table entries: 512
> > [ 0.010000] Initializing cgroup subsys ns
> > [ 0.010000] Initializing cgroup subsys cpuacct
> > [ 0.010000] Initializing cgroup subsys devices
> > [ 0.010000] Initializing cgroup subsys freezer
> > [ 0.010000] Performance Events:
> > [ 0.010000] no APIC, boot with the "lapic" boot parameter to
> > force-enable it.
> > [ 0.010000] no hardware sampling interrupt available.
> > [ 0.010000] AMD PMU driver.
> > [ 0.010000] ... version: 0
> > [ 0.010000] ... bit width: 48
> > [ 0.010000] ... generic registers: 4
> > [ 0.010000] ... value mask: 0000ffffffffffff
> > [ 0.010000] ... max period: 00007fffffffffff
> > [ 0.010000] ... fixed-purpose events: 0
> > [ 0.010000] ... event mask: 000000000000000f
> > [ 0.010000] SMP alternatives: switching to UP code
> > [ 0.010000] Freeing SMP alternatives: 16k freed
> > [ 0.010121] cpu 0 spinlock event irq 1
> > [ 0.010429] Brought up 1 CPUs
> > [ 0.010467] BUG: unable to handle kernel NULL pointer dereference at
> > (null) [ 0.010482] IP: [<c102eeda>] cpu_attach_domain+0x6e/0x178
> > [ 0.010503] *pdpt = 0000000001454027 *pde = 0000000000000000
> > [ 0.010517] Oops: 0000 [#1] SMP
> > [ 0.010528] last sysfs file:
> > [ 0.010536] Modules linked in:
> > [ 0.010546]
> > [ 0.010554] Pid: 1, comm: swapper Tainted: G W
> > 2.6.35-1-586-vyatta-virt #1 /
> > [ 0.010566] EIP: 0061:[<c102eeda>] EFLAGS: 00010206 CPU: 0
> > [ 0.010576] EIP is at cpu_attach_domain+0x6e/0x178
> > [ 0.010585] EAX: 0000042f EBX: c14eb868 ECX: 00000000 EDX: c14eb940
> > [ 0.010595] ESI: c14efb80 EDI: 00000000 EBP: c14eb9ac ESP: df035f40
> > [ 0.010605] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: e021
> > [ 0.010614] Process swapper (pid: 1, ti=df034000 task=df030000
> > task.ti=df034000)
> > [ 0.010625] Stack:
> > [ 0.010631] c14eb940 0000102f 0000042f df044000 df006058 df006058
> > 00000000 c1447868
> > [ 0.010655] <0> 00000000 c102f634 00000027 c14eba08 00000000
> > 00000020 c14479ac 00000001
> > [ 0.010683] <0> c1398564 00000000 c1398564 00000000 00000001
> > 00000001 00000001 00000000
> > [ 0.010713] Call Trace:
> > [ 0.010725] [<c102f634>] ? __build_sched_domains+0x650/0x697
> 
> This might relate to BIGSMP. Maybe something going wrong with the cpu
> topology info?
> 
> What local changes do you have? What kernel last worked for you?
> 
> Thanks,
> J
> 
> > [ 0.010742] [<c13df9e2>] ? sched_init_smp+0x81/0x11f
> > [ 0.010754] [<c13cd326>] ? kernel_init+0xfd/0x1ba
> > [ 0.010764] [<c13cd229>] ? kernel_init+0x0/0x1ba
> > [ 0.010776] [<c1007d36>] ? kernel_thread_helper+0x6/0x10
> > [ 0.010784] Code: 42 38 89 44 24 04 8b 45 38 89 14 24 89 44 24 08 8d
> > 45 5c e8 94 a7 ff ff 8b 14 24 48 74 4c 8b 45 38 a9 8f 02 00 00 74 0b
> > 8b 4d 08 <3b> 09 0f 85 ef 00 00 00 a8 20 0f 85 e7 00 00 00 eb 2d 8b
> > 4d 08
> > [ 0.010938] EIP: [<c102eeda>] cpu_attach_domain+0x6e/0x178 SS:ESP
> > e021:df035f40 [ 0.010955] CR2: 0000000000000000
> > [ 0.010965] ---[ end trace 4eaa2a86a8e2da25 ]---
> > [ 0.010988] Kernel panic - not syncing: Attempted to kill init!
> > [ 0.020000] Pid: 1, comm: swapper Tainted: G D W
> > 2.6.35-1-586-vyatta-virt #1
> > [ 0.020000] Call Trace:
> > [ 0.020000] [<c128954b>] ? panic+0x4a/0xbb
> > [ 0.020000] [<c1038616>] ? do_exit+0x53/0x646
> > [ 0.020000] [<c100a0b5>] ? oops_end+0x7d/0x80
> > [ 0.020000] [<c1021cf8>] ? no_context+0x13b/0x144
> > [ 0.020000] [<c102202d>] ? do_page_fault+0x0/0x24b
> > [ 0.020000] [<c1021e06>] ? bad_area_nosemaphore+0xa/0xc
> > [ 0.020000] [<c128b4a6>] ? error_code+0x66/0x6c
> > [ 0.020000] [<c102202d>] ? do_page_fault+0x0/0x24b
> > [ 0.020000] [<c102eeda>] ? cpu_attach_domain+0x6e/0x178
> > [ 0.020000] [<c102f634>] ? __build_sched_domains+0x650/0x697
> > [ 0.020000] [<c13df9e2>] ? sched_init_smp+0x81/0x11f
> > [ 0.020000] [<c13cd326>] ? kernel_init+0xfd/0x1ba
> > [ 0.020000] [<c13cd229>] ? kernel_init+0x0/0x1ba
> > [ 0.020000] [<c1007d36>] ? kernel_thread_helper+0x6/0x10

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel