[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN unstable c/s 11134



Summary:
Changeset 11134
- 64bit SMP HVM guests crash on boot since c/s 10070 (failure.5)
  This happens with kernarg noapic.  If you remove noapic it doesn't
  crash but the guest boots very slow and the guest network does
  not come up.

Test Configuration:
Dell Precision WorkStation 380, Dual Core, 2GB, 3 SATA (Intel VT)
64bit XEN Hypervisor on a RHEL4U2 64bit root (/dev/sda)
32bit fully virtualized (HVM) guest RHEL4U2 256MB (/dev/sdb)
      pae=1(smp) pae=0(up), acpi=1, apic=1
      kernargs noapic
64bit fully virtualized (HVM) guest RHEL4U2 256MB (/dev/sdc)
      pae=1, acpi=1, apic=1
      kernargs noapic

Boot Tests:
Boot a fully virtualized (HVM) guest to the login prompt
Results are marked Pass|Fail where (n) points to a failure description

Regression Tests:
852 tests (851 ltp tests and one 30 minute user load test)
Tests are marked #Pass/#Fail where (n) points to a failure description

XEN 64bit 2 CPU Hypervisor (booted smp):
 ----------------------------------------------------------------------
| XEN      | Guest Kernel (SMP kernels booted with 2 CPUs)             |
| Changeset|-----------------------------------------------------------|
|          | 32bit UP     | 32bit SMP    | 64bit UP     | 64bit SMP    |
|          |--------------|--------------|--------------|--------------|
|          | Boot | Test  | Boot | Test  | Boot | Test  | Boot | Test  |
|----------|------|-------|------|-------|------|-------|------|-------|
| 11134    | Pass |       | Pass | 845/7 | Pass |       | Fail |       |
|          |      |       |      | (1,2) |      |       | (5)  |       |
|----------|------|-------|------|-------|------|-------|------|-------|
| 10098    | Pass |       | Pass | 845/7 | Pass |       | Fail |       |
|          |      |       |      | (1,2) |      |       | (5)  |       |
|----------|------|-------|------|-------|------|-------|------|-------|
| 10070    | Fail |       | Fail |       | Fail |       | Fail |       |
|          | (4)  |       | (3)  |       | (4)  |       | (5)  |       |
|----------|------|-------|------|-------|------|-------|------|-------|
| 10059    | Pass |       | Pass | 844/8 | Pass |       | Pass | 852/0 |
|          |      |       |      | (1,2) |      |       |      |       |
|----------|------|-------|------|-------|------|-------|------|-------|
| 11049    | Pass |       | Pass | 845/7 | Pass |       | Pass | 852/0 |
|          |      |       |      | (1,2) |      |       |      |       |
 ----------------------------------------------------------------------

Multiple Guest Boot Test
Test is a 30 minute user load on both Guests
XEN 64bit 2 CPU Hypervisor (booted smp):
 --------------------------------------------
| XEN      | Guest Kernel                    |
| Changeset|---------------------------------|
|          | 32bit 1CPU UP  | 32bit 2CPU SMP |
|          | 64bit 1CPU UP  | 64bit 2CPU SMP |
|          |----------------|----------------|
|          | Boot | Test    | Boot | Test    |
|----------|------|---------|------|---------|
| 11134    | Pass | Fail    | Fail |         |
|          |      | (2)     | (5)  |         |
|----------|------|---------|------|---------|
| 10098    | Pass | Fail    | Fail |         |
|          |      | (2)     | (5)  |         |
|----------|------|---------|------|---------|
| 11070    | Fail |         | Fail |         |
|          | (4)  |         | (3,5)|         |
|----------|------|---------|------|---------|
| 10059    | Pass | Fail    | Pass | Fail    |
|          |      | (2)     |      | (2)     |
|----------|------|---------|------|---------|
| 11049    | Pass | Fail    | Pass | Fail    |
|          |      | (2)     |      | (2)     |
 --------------------------------------------

Failures:
 1. BUG 666: 32bit guests fail ltp gettimeofday02:
    "Time is going backwards"
 2. Multiple HVM guests (32/64 SMP) crash under load:
    32bit SMP guests fail various tests with:
    "Couldn't alloc shadow page!"
    "domain_crash_sync called from shadow.c:445"
 5. 64bit SMP HVM guests (noapic) crash on boot:
    "Unable to handle kernel paging request at
     ffffffff006c8a41 RIP: <ffffffff801112d8>
     {in_exception_stack+27} PML4 103027 PGD 0
     Oops: 0000 [432] SMP
     CPU -1"

File: failure.1

Time            Level   Message
05:00:50        INFO    Reporting status: 'Test Running' for test: 
ltp_gettimeofday02
05:00:52        INFO    Preparing to run test 'ltp_gettimeofday02' using 
profile: /qa/conductor/profiles/ltp/syscalls/gettimeofday02.xml
05:00:52        INFO    Starting test 'ltp_gettimeofday02' using profile: 
/qa/conductor/profiles/ltp/syscalls/gettimeofday02.xml
05:00:52        INFO    Dispatching operation: RemoteShell
05:00:52        FINE    Client sequencer got message requesting the start of a 
new test: ltp_gettimeofday02
05:00:52        FINER   Client sequencer sent message of type: 4 with seq num: 
1 of size: 289 bytes
05:00:52        FINER   Client sequencer handling new operation from control 
sequencer
05:00:52        FINE    Client sequencer looking for class: 
com.katana.conductor.operations.RemoteShell
05:00:52        INFO    Operation RemoteShell running
05:00:52        FINE    Client sequencer was told that an operation is now 
running
05:00:52        INFO    RemoteShell: target node(s) = vs177
05:00:52        INFO    ssh: /usr/bin/ssh root@vs177 cd 
/qa/conductor/tests/ltp/testcases/bin; gettimeofday02
05:00:52        FINE    ssh: waiting for command to finish
05:00:53        INFO       ssh: gettimeofday02 0 INFO : checking if 
gettimeofday is monotonous, takes 30s
05:00:53        INFO       ssh: gettimeofday02 1 FAIL : Time is going backwards 
(old 1145696453.61428 vs new 1145696453.60660!
05:00:53        FINE    executeShellCmd(ssh): exit value is 1
05:00:53        SEVERE  RemoteShell: command failed with error = 1
05:00:53        SEVERE  Operation RemoteShell failed
05:00:53        SEVERE  Reporting status: 'Test Failed' for test: 
ltp_gettimeofday02
05:00:53        FINE    Client sequencer detected operation completed with 
status of: Fail
05:00:53        FINER   Client sequencer sent message of type: 5 with seq num: 
2 of size: 429 bytes
05:00:53        SEVERE  Crash Collection disabled for queue : RHEL4U2-32b-XEN
05:00:53        INFO    Cleaning up after test

File: failure.2
Red Hat Enterprise Linux ES release 4 (Nahant Update 2)
Kernel 2.6.16.13-xen on an x86_64

tst110 login: root
Password:
Last login: Mon Jul 31 14:48:21 on ttyS0
You have new mail.
[root@tst110 ~]# xm info
host                   : tst110
release                : 2.6.16.13-xen
version                : #1 SMP Mon Jul 31 01:37:28 EDT 2006
machine                : x86_64
nr_cpus                : 2
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 2
threads_per_core       : 1
cpu_mhz                : 2793
hw_caps                : 
bfebfbff:20100800:00000000:00000180:0000e43d:00000000:00000001
total_memory           : 1022
free_memory            : 67
xen_major              : 3
xen_minor              : 0
xen_extra              : -unstable
xen_caps               : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p 
hvm-3.0-x86_64
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : Sat Jul 29 14:05:59 2006 +0100 10868:d2bf1a7cc131
cc_compiler            : gcc version 4.0.0 20050519 (Red Hat 4.0.0-8)
cc_compile_by          : build
cc_compile_domain      : virtualiron.com
cc_compile_date        : Mon Jul 31 00:48:57 EDT 2006
[root@tst110 ~]#                                                


(XEN) Couldn't alloc shadow page! dom6 count=1859
(XEN) Shadow table counts: l1=0 l2=0 hl2=0 snapshot=0
(XEN) domain_crash_sync called from shadow.c:445
(XEN) Domain 6 (vcpu#0) crashed on cpu#1:
(XEN) ----[ Xen-3.0-unstable    Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    0060:[<00000000c011a7ef>]
(XEN) RFLAGS: 0000000000010206   CONTEXT: hvm
(XEN) rax: 0000000000000000   rbx: 00000000c1167560   rcx: 0000000000000400
(XEN) rdx: 0000000000000016   rsi: 0000000000000000   rdi: 00000000cb3ab000
(XEN) rbp: 00000000cb3ab000   rsp: 00000000cba33e98   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr3: 0000000020865000
(XEN) ds: 007b   es: 007b   fs: 0000   gs: 0000   ss: 0068   cs: 0060
(XEN) Couldn't alloc shadow page! dom7 count=147
(XEN) Shadow table counts: l1=0 l2=0 hl2=0 snapshot=0
(XEN) domain_crash_sync called from shadow.c:445
(XEN) Domain 7 (vcpu#1) crashed on cpu#1:
(XEN) ----[ Xen-3.0-unstable    Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    0010:[<ffffffff801678c0>]
(XEN) RFLAGS: 0000000000010246   CONTEXT: hvm
(XEN) rax: 6db6db6db6db6db7   rbx: 0000010001285308   rcx: 000000000b857025
(XEN) rdx: 000001000000b780   rsi: 000000000000008e   rdi: 000001000afeae38
(XEN) rbp: 000001000afeae38   rsp: 000001000a9a7d98   r8:  0000010009f19c70
(XEN) r9:  0000010008924558   r10: 0000007fbfffca01   r11: 0000000000000246
(XEN) r12: 000001000ba9b408   r13: 000001000b2296a8   r14: 000001000afeadc0
(XEN) r15: 0000002a9578e0a0   cr0: 000000008005003b   cr3: 000000000dfae000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0010
ata1: command 0x35 timeout, stat 0x50 host_stat 0x64
ata2: command 0x35 timeout, stat 0x50 host_stat 0x44
ata1: command 0x35 timeout, stat 0x50 host_stat 0x64
ata2: command 0x35 timeout, stat 0x50 host_stat 0x44
File: failure.5

Red Hat Enterprise Linux ES release 4 (Nahant Update 2)
Kernel 2.6.16.13-xen on an x86_64

tst177 login: Bridge firewalling registered
ip_tables: (C) 2000-2006 Netfilter Core Team
ip_tables: (C) 2000-2006 Netfilter Core Team
[root@tst177 ~]# xm info
host                   : tst177
release                : 2.6.16.13-xen
version                : #1 SMP Mon Aug 14 01:37:12 EDT 2006
machine                : x86_64
nr_cpus                : 2
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 2
threads_per_core       : 1
cpu_mhz                : 2793
hw_caps                : 
bfebfbff:20100800:00000000:00000180:0000e43d:00000000:00000001
total_memory           : 2046
free_memory            : 130
xen_major              : 3
xen_minor              : 0
xen_extra              : -unstable
xen_caps               : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p 
hvm-3.0-x86_64
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : Sun Aug 13 09:44:07 2006 +0100 11070:befab551b0e1
cc_compiler            : gcc version 4.0.0 20050519 (Red Hat 4.0.0-8)
cc_compile_by          : build
cc_compile_domain      : virtualiron.com
cc_compile_date        : Mon Aug 14 00:48:08 EDT 2006
[root@tst177 ~]#                                        
(XEN) (GUEST: 1) HVM Loader
(XEN) (GUEST: 1) Detected Xen v3.0
(XEN) (GUEST: 1) Loading ROMBIOS ...
(XEN) (GUEST: 1) Loading Cirrus VGABIOS ...
(XEN) (GUEST: 1) Loading ACPI ...
(XEN) (GUEST: 1) Loading VMXAssist ...
(XEN) (GUEST: 1) VMX go ...
(XEN) (GUEST: 1) VMXAssist (Aug 14 2006)
(XEN) (GUEST: 1) Memory size 256 MB
(XEN) (GUEST: 1) E820 map:
(XEN) (GUEST: 1) 0000000000000000 - 000000000009F800 (RAM)
(XEN) (GUEST: 1) 000000000009F800 - 00000000000A0000 (Reserved)
(XEN) (GUEST: 1) 00000000000A0000 - 00000000000C0000 (Type 16)
(XEN) (GUEST: 1) 00000000000F0000 - 0000000000100000 (Reserved)
(XEN) (GUEST: 1) 0000000000100000 - 000000000FFFD000 (RAM)
(XEN) (GUEST: 1) 000000000FFFD000 - 000000000FFFE000 (Type 19)
(XEN) (GUEST: 1) 000000000FFFE000 - 000000000FFFF000 (Type 18)
(XEN) (GUEST: 1) 000000000FFFF000 - 0000000010000000 (Type 17)
(XEN) (GUEST: 1) 0000000010000000 - 0000000010003000 (ACPI NVS)
(XEN) (GUEST: 1) 0000000010003000 - 000000001000D000 (ACPI Data)
(XEN) (GUEST: 1) 00000000FEC00000 - 0000000100000000 (Type 16)
(XEN) (GUEST: 1)
(XEN) (GUEST: 1) Start BIOS ...
(XEN) (GUEST: 1) Starting emulated 16-bit real-mode: ip=F000:FFF0
(XEN) (GUEST: 1)  rombios.c,v 1.138 2005/05/07 15:55:26 vruppert Exp $
(XEN) (GUEST: 1) Remapping master: ICW2 0x8 -> 0x20
(XEN) (GUEST: 1) Remapping slave: ICW2 0x70 -> 0x28
(XEN) (GUEST: 1) VGABios $Id: vgabios.c,v 1.61 2005/05/24 16:50:50 vruppert Exp 
$
(XEN) (GUEST: 1) HVMAssist BIOS, 1 cpu, $Revision: 1.138 $ $Date: 2005/05/07 
15:55:26 $
(XEN) (GUEST: 1)
(XEN) (GUEST: 1) ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
(XEN) (GUEST: 1) ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (12997 MBytes)
(XEN) (GUEST: 1) ata0  slave: Unknown device
(XEN) (GUEST: 1)
(XEN) (GUEST: 1) Booting from Hard Disk...
(XEN) (GUEST: 1) int13_harddisk: function 41, unmapped device for ELDL=81
(XEN) (GUEST: 1) int13_harddisk: function 08, unmapped device for ELDL=81
(XEN) (GUEST: 1) *** int 15h function AX=00C0, BX=0000 not yet supported!
(XEN) (GUEST: 1) int13_harddisk: function 15, unmapped device for ELDL=81
(XEN) (GUEST: 1) *** int 15h function AX=EC00, BX=0002 not yet supported!
(XEN) (GUEST: 1) KBD: unsupported int 16h function 03
(XEN) (GUEST: 1) int13_harddisk: function 15, unmapped device for ELDL=81
(XEN) (GUEST: 1) int13_harddisk: function 02, unmapped device for ELDL=81
(XEN) (GUEST: 1) int13_harddisk: function 41, unmapped device for ELDL=81
(XEN) Local APIC Write to read-only register


Guest:

Unable to handle kernel paging request at ffffffff006c8a41 RIP:
<ffffffff801112d8> {in_exception_stack+27} 
PML4 103027 PGD 0
Oops: 0000 [432] SMP
CPU -1

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.