[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2

Hi guys,
today I have upgraded from XEN 4.5.1 to XEN 4.5.2 and also upgraded the dom0 kernel from 3.18.9 to 4.1.7
After the upgrade HVM domUs appear to no longer work - regardless of the dom0 kernel (tested with both 3.18.9 and 4.1.7 as the dom0 kernel); PV domUs, however, work just fine as before on both dom0 kernels.
xl dmesg shows the following information after the first crashed HVM domU which is started as part of the machine booting up: 

===== start xl dmesg output =====
 Xen 4.5.2
(XEN) Xen version 4.5.2 (@myhomeismycastle.com) (x86_64-pc-linux-gnu-gcc (Gentoo Hardened 4.9.3 p1.2, pie-0.6.3) 4.9.3) debug=n Wed Nov 11 13:00:00 CET 2015 
(XEN) Latest ChangeSet:
(XEN) Bootloader: GRUB 2.00
(XEN) Command line: placeholder ucode=-1 loglvl=warning guest_loglvl=none/warning dom0_mem=4G,max:4G tmem=1 tmem_compress=1 tmem_dedup=1 dom0_max_vcpus=8 dom0_vcpus_pin=true cpufreq=xen cpuidle clocksource=hpet iommu=1 sched_credit_tslice_ms=5 bootscrub=0 
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 2 MBR signatures
(XEN)  Found 2 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009d800 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000020000000 (usable)
(XEN)  0000000020000000 - 0000000020200000 (reserved)
(XEN)  0000000020200000 - 0000000040000000 (usable)
(XEN)  0000000040000000 - 0000000040200000 (reserved)
(XEN)  0000000040200000 - 00000000db9f0000 (usable)
(XEN)  00000000db9f0000 - 00000000dc0da000 (reserved)
(XEN)  00000000dc0da000 - 00000000dc1f9000 (ACPI NVS)
(XEN)  00000000dc1f9000 - 00000000dc651000 (reserved)
(XEN)  00000000dc651000 - 00000000dc652000 (usable)
(XEN)  00000000dc652000 - 00000000dc695000 (ACPI NVS)
(XEN)  00000000dc695000 - 00000000dcdba000 (usable)
(XEN)  00000000dcdba000 - 00000000dcff2000 (reserved)
(XEN)  00000000dcff2000 - 00000000dd000000 (usable)
(XEN)  00000000dd800000 - 00000000dfa00000 (reserved)
(XEN)  00000000f8000000 - 00000000fc000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fed00000 - 00000000fed04000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000081e600000 (usable)
(XEN) ACPI: RSDP 000F0490, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT DC1E9078, 0074 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FACP DC1F3710, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: DSDT DC1E9188, A587 (r2 ALASKA    A M I        1 INTL 20051117)
(XEN) ACPI: FACS DC1F7F80, 0040
(XEN) ACPI: APIC DC1F3808, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FPDT DC1F38A0, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: MCFG DC1F38E8, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) ACPI: HPET DC1F3928, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) ACPI: SSDT DC1F3960, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) ACPI: SSDT DC1F3CD0, 081E (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) ACPI: SSDT DC1F44F0, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) ACPI: DMAR DC1F4F88, 00B0 (r1 INTEL      SNB         1 INTL        1)
(XEN) ACPI: ASF! DC1F5038, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) System RAM: 32674MB (33458948kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT - dc1f7f80/0000000000000000, using 32 
(XEN) Processor #0 6:10 APIC version 21
(XEN) Processor #2 6:10 APIC version 21
(XEN) Processor #4 6:10 APIC version 21
(XEN) Processor #6 6:10 APIC version 21
(XEN) Processor #1 6:10 APIC version 21
(XEN) Processor #3 6:10 APIC version 21
(XEN) Processor #5 6:10 APIC version 21
(XEN) Processor #7 6:10 APIC version 21
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2394.643 MHz processor.
(XEN) Initing memory sharing.
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
(XEN) Brought up 8 CPUs
(XEN) tmem: initialized comp=1 dedup=1 tze=0
(XEN) Dom0 has maximum 792 PIRQs
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1c00000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000800000000->0000000804000000 (1029890 pages to be allocated) 
(XEN)  Init. ramdisk: 000000081dcff000->000000081e5fc600
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff81c00000
(XEN)  Init. ramdisk: 0000000000000000->0000000000000000
(XEN)  Phys-Mach map: ffffffff81c00000->ffffffff82400000
(XEN)  Start info:    ffffffff82400000->ffffffff824004b4
(XEN)  Page tables:   ffffffff82401000->ffffffff82418000
(XEN)  Boot stack:    ffffffff82418000->ffffffff82419000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82800000
(XEN)  ENTRY ADDRESS: ffffffff818e31f0
(XEN) Dom0 has maximum 8 VCPUs
(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen) 
(XEN) Freed 316kB init memory.
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000. (XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0802db000 to 0xffffffff81646d10. (XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0802db080 to 0xffffffff81648e70. (XEN) traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010. (XEN) traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000175 from 0xffff82d0802dffc0 to 0x0000000000000000. (XEN) traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0. (XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700. (XEN) traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000. (XEN) traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000082 from 0xffff83080ca73000 to 0xffffffff81646d10. (XEN) traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000083 from 0xffff83080ca73080 to 0xffffffff81648e70. 
(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest state (0). 
(XEN) ************* VMCS Area **************
(XEN) *** Guest State ***
(XEN) CR0: actual=0x0000000000000039, shadow=0x0000000000000011, gh_mask=ffffffffffffffff (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000, gh_mask=ffffffffffffffff 
(XEN) CR3: actual=0x0000000000800000, target_count=0
(XEN)      target0=0000000000000000, target1=0000000000000000
(XEN)      target2=0000000000000000, target3=0000000000000000
(XEN) RSP = 0x0000000000006fdc (0x0000000000006fdc) RIP = 0x0000000100000000 (0x0000000100000000) (XEN) RFLAGS=0x0000000000000006 (0x0000000000000006) DR7 = 0x0000000000000400 
(XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
(XEN) CS: sel=0x0008, attr=0x0c09b, limit=0xffffffff, base=0x0000000000000000 (XEN) DS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000 (XEN) SS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000 (XEN) ES: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000 (XEN) FS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000 (XEN) GS: sel=0x0010, attr=0x0c093, limit=0xffffffff, base=0x0000000000000000 (XEN) GDTR: limit=0x00000037, base=0x00000000000f6d80 (XEN) LDTR: sel=0x0000, attr=0x00082, limit=0x00000000, base=0x0000000000000000 (XEN) IDTR: limit=0x00000000, base=0x00000000000f6dbe (XEN) TR: sel=0x0000, attr=0x0008b, limit=0x000000ff, base=0x0000000000000000 
(XEN) Guest PAT = 0x0007040600070406
(XEN) TSC Offset = ffffffe7c228bb48
(XEN) DebugCtl=0000000000000000 DebugExceptions=0000000000000000
(XEN) Interruptibility=0000 ActivityState=0000
(XEN) *** Host State ***
(XEN) RSP = 0xffff8308063fff90  RIP = 0xffff82d0801f46a0
(XEN) CS=e008 DS=0000 ES=0000 FS=0000 GS=0000 SS=0000 TR=e040
(XEN) FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff83080ca88c80 
(XEN) GDTBase=ffff83080ca7c000 IDTBase=ffff83080ca8d000
(XEN) CR0=000000008005003b CR3=000000070bf21000 CR4=00000000000426f0
(XEN) Sysenter RSP=ffff8308063fffc0 CS:RIP=e008:ffff82d080235550
(XEN) Host PAT = 0x0000050100070406
(XEN) *** Control State ***
(XEN) PinBased=0000003f CPUBased=b6a065fe SecondaryExec=000000eb
(XEN) EntryControls=000051ff ExitControls=000fefff
(XEN) ExceptionBitmap=000400c0
(XEN) VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000000
(XEN)         reason=80000021 qualification=00000000
(XEN) IDTVectoring: info=00000000 errcode=00000000
(XEN) TPR Threshold = 0x00
(XEN) EPT pointer = 0x000000070bf1f01e
(XEN) Virtual processor ID = 0x0002
(XEN) **************************************
(XEN) domain_crash called from vmx.c:2511
(XEN) Domain 1 (vcpu#0) crashed on cpu#6:
(XEN) ----[ Xen-4.5.2  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    6
(XEN) RIP:    0008:[<0000000100000000>]
(XEN) RFLAGS: 0000000000000006   CONTEXT: hvm guest (d1v0)
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: 00000000ffff1720
(XEN) rdx: 0000000000000059   rsi: 0000000000000059   rdi: 0000000000000000
(XEN) rbp: 0000000000000000   rsp: 0000000000006fdc   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000000000011   cr4: 0000000000000000
(XEN) cr3: 0000000000800000   cr2: 0000000000000000
(XEN) ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008
===== end xl dmesg output =====
It is probably also worth pointing out again that this crash happens with both kernels, 3.18.9 and 4.1.7; the only obvious differenc (other than slight address differences) between 3.18.9 and 4.1.7 are the following lines in xl dmesg above which only show up on 4.1.7:
(XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000. (XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0802db000 to 0xffffffff81646d10. (XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0802db080 to 0xffffffff81648e70. (XEN) traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010. (XEN) traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000175 from 0xffff82d0802dffc0 to 0x0000000000000000. (XEN) traps.c:2590:d0v0 Domain attempted WRMSR 0000000000000176 from 0xffff82d080235550 to 0xffffffff81648cf0. (XEN) traps.c:2590:d0v0 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700. (XEN) traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000. (XEN) traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000082 from 0xffff83080ca73000 to 0xffffffff81646d10. (XEN) traps.c:2590:d0v1 Domain attempted WRMSR 00000000c0000083 from 0xffff83080ca73080 to 0xffffffff81648e70.
The configuration file for the crashing HVM domain contains nothing fancy and looks as follows: 

===== start configuration file =====
builder                 = 'hvm'
cpus                    = '2-7'
vcpus                   = 2
cpu_weight              = 512
memory                  = 640
name                    = 'pfsense'
disk = [ 'vdev=xvda,format=raw,access=rw,target=/etc/xen/guests/disk.d/pfsense.disk' ] vif = [ 'mac=00:16:3e:a1:64:01,bridge=xenbr0,type=vif,vifname=pfsense.0,script=vif-bridge.noTXoffload' ] 
on_poweroff             = 'destroy'
on_reboot               = 'restart'
on_crash                = 'destroy'
localtime               = 0
boot                    = 'c'
vnc                     = 1
vnclisten               = '127.0.0.1'
vncpasswd               = ''
keymap                  = 'de'
nographic               = 1
serial                  = 'pty'
nx                      = 1
pci                     = [ '04:00.0', '0a:08.0', '0a:0b.0' ]
===== end configuration file =====
The crash seems to not be related to this domU; there's another HVM domU which also crashes with a similar message.
Prior to the upgrade this has all worked flawlessly for a number of months on 4.5.1. Any insight on what's going on and how to return to proper operation would greatly be appreciated. 

Many thanks in advance, Atom2

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.

Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.