[Wg-test-framework] colo outage

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

I noticed over the weekend that newcastle (the primary of the two vm
servers in the test lab) was not accessible via ssh, ping, etc.
Therefore neither was the main osstest vm and most of the service VMs

Ian J is away with limited net access so I have rebooted it via the PDU
and it has rebooted. Attempts we made to restore (rather than start) a
bunch of vms, suggesting it may have been gracefully shutdown by
someone? syslog has:

Apr 24 19:06:26 newcastle shutdown[28059]: shutting down for system halt

Anyone want to fess up? 

Perhaps we should "apt-get install molly-guard"?

During the restore several of the associated xl processes crashed with
the logs below and the domains were left paused.

It is still restoring the final domain and I have to head out for
dinner. When I get back I will destroy all of the domains and restart
the set mentioned in /etc/xen/auto.

Also, the disk was completely full. Likely with all those saved VM
images? /var/lib/xen/save is 17GB.

I'd not be surprised if a disk full error lead to the crashes too.


Ian.


Loading new save file /var/lib/xen/save/ssh (new xl fmt info 0x0/0x0/481)
 Savefile contains xl domain config
*** glibc detected *** /usr/lib/xen-4.1/bin/xl: double free or corruption 
(!prev): 0x09f4b748 ***
======= Backtrace: =========
/lib/i386-linux-gnu/i686/nosegneg/libc.so.6(+0x71841)[0xb75d4841]
/lib/i386-linux-gnu/i686/nosegneg/libc.so.6(+0x730a8)[0xb75d60a8]
/lib/i386-linux-gnu/i686/nosegneg/libc.so.6(cfree+0x6d)[0xb75d923d]
/lib/i386-linux-gnu/i686/nosegneg/libc.so.6(fclose+0x14a)[0xb75c4b4a]
/usr/lib/xen-4.1/bin/../lib/libxenlight.so(libxl_userdata_store+0x133)[0xb771f123]
======= Memory map: ========
08048000-08063000 r-xp 00000000 fe:00 3737       /usr/lib/xen-4.1/bin/xl
08063000-08064000 rw-p 0001b000 fe:00 3737       /usr/lib/xen-4.1/bin/xl
09f49000-09f4a000 rw-p 00000000 00:00 0          [heap]
09f4a000-09f4b000 rw-p 00000000 00:00 0          [heap]
09f4b000-09f4c000 rw-p 00000000 00:00 0          [heap]
09f4c000-09f4d000 rw-p 00000000 00:00 0          [heap]
09f4d000-09f6c000 rw-p 00000000 00:00 0          [heap]
b6e00000-b6e21000 rw-p 00000000 00:00 0 
b6e21000-b6f00000 ---p 00000000 00:00 0 
b6f9c000-b6fb8000 r-xp 00000000 fe:00 917508     
/lib/i386-linux-gnu/libgcc_s.so.1
b6fb8000-b6fb9000 rw-p 0001b000 fe:00 917508     
/lib/i386-linux-gnu/libgcc_s.so.1
b6fc1000-b73c2000 rw-p 00000000 00:00 0 
b74c8000-b74c9000 rw-p 00000000 00:00 0 
b74c9000-b74e0000 r-xp 00000000 fe:00 917595     
/lib/i386-linux-gnu/libz.so.1.2.7
b74e0000-b74e1000 r--p 00016000 fe:00 917595     
/lib/i386-linux-gnu/libz.so.1.2.7
b74e1000-b74e2000 rw-p 00017000 fe:00 917595     
/lib/i386-linux-gnu/libz.so.1.2.7
b74e2000-b74e3000 rw-p 00000000 00:00 0 
b74e3000-b7508000 r-xp 00000000 fe:00 917544     
/lib/i386-linux-gnu/liblzma.so.5.0.0
b7508000-b7509000 r--p 00024000 fe:00 917544     
/lib/i386-linux-gnu/liblzma.so.5.0.0
b7509000-b750a000 rw-p 00025000 fe:00 917544     
/lib/i386-linux-gnu/liblzma.so.5.0.0
b750a000-b751f000 r-xp 00000000 fe:00 2985       
/lib/i386-linux-gnu/i686/nosegneg/libpthread-2.13.so
b751f000-b7520000 ---p 00015000 fe:00 2985       
/lib/i386-linux-gnu/i686/nosegneg/libpthread-2.13.so
b7520000-b7521000 r--p 00015000 fe:00 2985       
/lib/i386-linux-gnu/i686/nosegneg/libpthread-2.13.so
b7521000-b7522000 rw-p 00016000 fe:00 2985       
/lib/i386-linux-gnu/i686/nosegneg/libpthread-2.13.so
b7522000-b7524000 rw-p 00000000 00:00 0 
b7524000-b7526000 r-xp 00000000 fe:00 3007       
/lib/i386-linux-gnu/i686/nosegneg/libutil-2.13.so
b7526000-b7527000 r--p 00001000 fe:00 3007       
/lib/i386-linux-gnu/i686/nosegneg/libutil-2.13.so
b7527000-b7528000 rw-p 00002000 fe:00 3007       
/lib/i386-linux-gnu/i686/nosegneg/libutil-2.13.so
b7528000-b752f000 r-xp 00000000 fe:00 3483       
/usr/lib/xen-4.1/lib/libblktapctl.so
b752f000-b7530000 rw-p 00006000 fe:00 3483       
/usr/lib/xen-4.1/lib/libblktapctl.so
b7530000-b7535000 r-xp 00000000 fe:00 800900     /usr/lib/libxenstore.so.3.0.0
b7535000-b7536000 rw-p 00004000 fe:00 800900     /usr/lib/libxenstore.so.3.0.0
b7536000-b753a000 rw-p 00000000 00:00 0 
b753a000-b7562000 r-xp 00000000 fe:00 800899     /usr/lib/libxenguest-4.1.so
b7562000-b7563000 rw-p 00028000 fe:00 800899     /usr/lib/libxenguest-4.1.so
b7563000-b76c5000 r-xp 00000000 fe:00 2989       
/lib/i386-linux-gnu/i686/nosegneg/libc-2.13.so
b76c5000-b76c6000 ---p 00162000 fe:00 2989       
/lib/i386-linux-gnu/i686/nosegneg/libc-2.13.so
b76c6000-b76c8000 r--p 00162000 fe:00 2989       
/lib/i386-linux-gnu/i686/nosegneg/libc-2.13.so
b76c8000-b76c9000 rw-p 00164000 fe:00 2989       
/lib/i386-linux-gnu/i686/nosegneg/libc-2.13.so
b76c9000-b76cc000 rw-p 00000000 00:00 0 
b76cc000-b76d0000 r-xp 00000000 fe:00 917574     
/lib/i386-linux-gnu/libuuid.so.1.3.0
b76d0000-b76d1000 r--p 00003000 fe:00 917574     
/lib/i386-linux-gnu/libuuid.so.1.3.0
b76d1000-b76d2000 rw-p 00004000 fe:00 917574     
/lib/i386-linux-gnu/libuuid.so.1.3.0
b76d2000-b76d4000 r-xp 00000000 fe:00 2992       
/lib/i386-linux-gnu/i686/nosegneg/libdl-2.13.so
b76d4000-b76d5000 r--p 00001000 fe:00 2992       
/lib/i386-linux-gnu/i686/nosegneg/libdl-2.13.so
b76d5000-b76d6000 rw-p 00002000 fe:00 2992       
/lib/i386-linux-gnu/i686/nosegneg/libdl-2.13.so
b76d6000-b76fb000 r-xp 00000000 fe:00 800898     /usr/lib/libxenctrl-4.1.so
b76fb000-b76fc000 rw-p 00024000 fe:00 800898     /usr/lib/libxenctrl-4.1.so
b76fc000-b76fd000 rw-p 00000000 00:00 0 
b7704000-b7705000 rw-p 00000000 00:00 0 
b7705000-b7730000 r-xp 00000000 fe:00 3481       
/usr/lib/xen-4.1/lib/libxenlight.so
b7730000-b7731000 rw-p 0002a000 fe:00 3481       
/usr/lib/xen-4.1/lib/libxenlight.so
b7731000-b7738000 r-xp 00000000 fe:00 3480       
/usr/lib/xen-4.1/lib/libxlutil.so
b7738000-b7739000 rw-p 00006000 fe:00 3480       
/usr/lib/xen-4.1/lib/libxlutil.so
b7739000-b773d000 rw-p 00000000 00:00 0 
b773d000-b773e000 r-xp 00000000 00:00 0          [vdso]
b773e000-b775a000 r-xp 00000000 fe:00 917509     /lib/i386-linux-gnu/ld-2.13.so
b775a000-b775b000 r--p 0001b000 fe:00 917509     /lib/i386-linux-gnu/ld-2.13.so
b775b000-b775c000 rw-p 0001c000 fe:00 917509     /lib/i386-linux-gnu/ld-2.13.so
bfec0000-bfee1000 rw-p 00000000 00:00 0          [stack]



_______________________________________________
Wg-test-framework mailing list
Wg-test-framework@xxxxxxxxxxxxxxxxxxxx
http://lists.xenproject.org/cgi-bin/mailman/listinfo/wg-test-framework