Re: [Xen-users] xen 3.1 - domU hangs just after "xm create"

[Сергей Лукашевич <lukash33@xxxxxxxxx>]

Thank you, Mats for carefully answering my writings.
Now I managed to get xend.log by uncommenting it at my /etc/xen/xend-config.sxp.
And I could reproduce the following case:

http://lists.xensource.com/archives/html/xen-users/2007-06/msg00766.html

More precisely:

[2007-06-25 19:53:23 17224] ERROR (XendDomainInfo:105) Domain construction 
failed
Traceback (most recent call last):
  File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 103, in 
create_from_dict
    vm.start()
  File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 420, in start
    raise XendError('VM already running')
XendError: VM already running
[2007-06-25 19:53:23 17224] DEBUG (XendDomainInfo:1660) XendDomainInfo.destroy: 
domid=26
[2007-06-25 19:53:23 17224] ERROR (XendDomainInfo:1231) Failed to restart 
domain 26.
Traceback (most recent call last):
  File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1216, in _restart
    new_dom = XendDomain.instance().domain_create_from_dict(
  File "//usr/lib64/python/xen/xend/XendDomain.py", line 932, in 
domain_create_from_dict
    dominfo = XendDomainInfo.create_from_dict(config_dict)
  File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 103, in 
create_from_dict
    vm.start()
  File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 420, in start
    raise XendError('VM already running')
XendError: VM already running


Though I can not understand the solution (with kernel recompile) mentioned in 
the Thread above.

Looks different from what I started. Now domU is crashed, not hangs.
I tried to rename it and it effectively hangs again leaving the following in 
the xend.log (sorry for long output):

[2007-06-25 20:13:40 14796] DEBUG (DevController:117) DevController: writing 
{'domain': 'sles1', 'protocol': 'vt100', 'uuid': 
'22931fbd-2160-d624-0b03-25c683fc39f5', 'frontend': 
'/local/domain/27/device/console/0', 'state': '1', 'location': '2', 'online': 
'1', 'frontend-id': '27'} to /local/domain/0/backend/console/27/0.
[2007-06-25 20:13:40 17795] DEBUG (DevController:148) Waiting for devices vif.
[2007-06-25 20:13:40 14796] DEBUG (XendDomainInfo:824) Storing domain details: 
{'console/ring-ref': '8104036', 'image/entry': '-2145386496', 'console/port': 
'2', 'store/ring-ref': '8104037', 'image/loader': 'generic', 'vm': 
'/vm/05aa56d4-5df5-d9bf-5fbd-b881074f6ee4', 
'control/platform-feature-multiprocessor-suspend': '1', 'image/guest-os': 
'linux', 'cpu/1/availability': 'online', 
'image/features/writable-descriptor-tables': '1', 'image/virt-base': 
'-2147483648', 'memory/target': '262144', 'image/guest-version': '2.6', 
'image/features/supervisor-mode-kernel': '1', 'console/limit': '1048576', 
'image/paddr-offset': '-2147483648', 'image/hypercall-page': '-2145361920', 
'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 
'image/features/pae-pgdir-above-4gb': '1', 
'image/features/writable-page-tables': '1', 
'image/features/auto-translated-physmap': '1', 'name': 'sles1', 'domid': '27', 
'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2007-06-25 20:13:40 17795] DEBUG (DevController:148) Waiting for devices usb.
[2007-06-25 20:13:40 17795] DEBUG (DevController:148) Waiting for devices vbd.
[2007-06-25 20:13:40 17795] DEBUG (DevController:153) Waiting for 769.
[2007-06-25 20:13:40 14796] DEBUG (XendDomainInfo:908) 
XendDomainInfo.handleShutdownWatch
[2007-06-25 20:13:40 14796] DEBUG (DevController:539) hotplugStatusCallback 
/local/domain/0/backend/vbd/27/769/hotplug-status.
[2007-06-25 20:13:40 14796] DEBUG (DevController:539) hotplugStatusCallback 
/local/domain/0/backend/vbd/27/769/hotplug-status.
[2007-06-25 20:13:40 14796] DEBUG (DevController:553) hotplugStatusCallback 1.
[2007-06-25 20:13:40 17795] DEBUG (DevController:153) Waiting for 5632.
[2007-06-25 20:13:40 14796] DEBUG (DevController:539) hotplugStatusCallback 
/local/domain/0/backend/vbd/27/5632/hotplug-status.
[2007-06-25 20:13:41 14796] DEBUG (DevController:539) hotplugStatusCallback 
/local/domain/0/backend/vbd/27/5632/hotplug-status.
[2007-06-25 20:13:41 14796] DEBUG (DevController:553) hotplugStatusCallback 1.
[2007-06-25 20:13:41 17795] DEBUG (DevController:148) Waiting for devices irq.
[2007-06-25 20:13:41 17795] DEBUG (DevController:148) Waiting for devices vkbd.
[2007-06-25 20:13:41 17795] DEBUG (DevController:148) Waiting for devices vfb.
[2007-06-25 20:13:41 17795] DEBUG (DevController:148) Waiting for devices 
console.
[2007-06-25 20:13:41 17795] DEBUG (DevController:153) Waiting for 0.
[2007-06-25 20:13:41 17795] DEBUG (DevController:148) Waiting for devices pci.
[2007-06-25 20:13:41 17795] DEBUG (DevController:148) Waiting for devices 
ioports.
[2007-06-25 20:13:41 17795] DEBUG (DevController:148) Waiting for devices tap.
[2007-06-25 20:13:41 17795] DEBUG (DevController:148) Waiting for devices vtpm.
[2007-06-25 20:13:41 17989] INFO (XendDomain:1108) Domain sles1 (27) unpaused.


Please advice, what would be best ways to go on debuging?



25.06.07, 20:00, Petersson, Mats <Mats.Petersson@xxxxxxx> <>:

>  
> > -----Original Message-----
> > From: Сергей Лукашевич [mailto:lukash33@xxxxxxxxx] 
> > Sent: 25 June 2007 16:30
> > To: Petersson, Mats
> > Cc: xen-users@xxxxxxxxxxxxxxxxxxx
> > Subject: Re: [Xen-users] xen 3.1 - domU hangs just after "xm create"
> > 
> > 
> > 
> > 25.06.07, 18:34, Petersson, Mats <Mats.Petersson@xxxxxxx> <>:
> > 
> > >  
> > > > -----Original Message-----
> > > > From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx 
> > > > [mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of 
> > > > ?????? ?????????
> > > > Sent: 25 June 2007 15:26
> > > > To: xen-users@xxxxxxxxxxxxxxxxxxx
> > > > Subject: [Xen-users] xen 3.1 - domU hangs just after "xm create"
> > > > 
> > > > What are methods to debug domU when it hangs? Using xen 3.1 
> > > > compiled from sources I could not manage to launch no domU. 
> > > > Fot instance, I run something like this:
> > > > 
> > > > 
> > > > 
> > > > =====8<=====================
> > > > 
> > > > disk = [ 'file:/oradata-act/sles.disk,hda1,w', ',hdc:cdrom,r' ]
> > > > 
> > > > kernel = "/boot/vmlinuz-2.6.18-xen"
> > > > 
> > > > ramdisk = "/boot/initrd-2.6.18-xen"
> > > I take it this is the same initrd you use for Dom0? Does it 
> > have the drivers for DomU too?
> > 
> > Yes, the same kernel and the same initrd. I beleive XEN 
> > allows using same here from some of 2.* versions, is not it? 
> Same kernel is no problem at all - that's normal operating procedure for most 
> people - although some will argue that the kernel is slightly larger this way 
> compared to when you have a DomU specific kerne. 
> Yes, it's just that sometimes the initrd contains "stuff" that is 
> corresponding to the management of Dom0 that conflicts with the usage on DomU 
> - typical example is LVM management, but there's other things that cause this 
> too. 
> > Initrd contains reiserfs driver - that's quite enough to 
> > start booting. I suspect something wrong with my compiling 
> > and installing XEN because domUs output NO lines to their 
> > consoles to suspect its kernel.
> I agree, there's probably something wrong in your system on a more 
> fundamental level. 
> > 
> > > You may want to mount the sles.disk and use "mkinitrd" to 
> > create a new ramdisk from inside the sles.disk instead - that 
> > has uses the fstab of the sles.disk, etc. 
> > > > 
> > > > cpus = "1"
> > > > 
> > > > vcpus = 2
> > > > 
> > > > memory = 256
> > > > 
> > > > name = "sles"
> > > > 
> > > > root="/dev/hda1 ro"
> > > > 
> > > > =====8<=====================
> > > > 
> > > > 
> > > > 
> > > > Of course, sles.disk is a reiserfs image containing unTARred 
> > > > SLES9.3 64bit OS.
> > > I hope you don't use reiserfs on reiserfs here - as if you 
> > do and you ever need to "fsck" the underlaying disk, then 
> > you'll get problems - reiserfs uses a "magic word" to 
> > indicate where it starts it's filesystem, and finding another 
> > (same) magic word in the middle of the file-system is most 
> > certainly going to make things very confused when trying to 
> > figure out what's what. 
> > 
> > I DO use reiserfs onto reiserfs. Sounds very strange - magics 
> > can confuse fsck? Do you have some additional info on this 
> > behaviour? Some link or the like? It would be interesting to look at.
> > 
> > > > 
> > > > 
> > > > 
> > > > The domain console outputs nothing but several spaces at the 
> > > > very start and the domU hangs in a few (15-25) seconds.
> > > "Interesting". Not sure what that could be. 
> > > > 
> > > > 
> > > > 
> > > > What are log files to analyze? What are 'debug' options?
> > > I'd start with "xm dmesg". If that's not saying anything 
> > useful, look at /var/log/xen/xend.log and .../xend-debug.log
> > 
> > Only following files present there:
> > 
> > domain-builder-ng.log
> > qemu-dm.25423.log
> > qemu-dm.3345.log
> > xen-hotplug.log
> Surely you should have a /var/log/xen/xend.log too. 
> By the way, what is "builder" in your config file? Qemu-dm indicates that 
> you're trying to start a HVM domain - in which case your "kernel" shouldn't 
> be a linux kernel, but rather "hvmbuilder". 
> > 
> > xm dmesg is very interesting, thank you. Seems that every 
> > attempt to launch a domU yelds the following:
> > 
> > (XEN) mm.c:636:d0 Error getting mfn 100 (pfn 
> > 5555555555555555) from L1 entry 8000000000100125 for dom32753
> This may be harmfull - not sure. 
> > 
> > Also I saw some other strange lines from "xm dmesg" like this:
> > 
> > (XEN) microcode: CPU6 not a capable Intel processor
> At first I though this was due to your processors being AMD models, but I 
> suspect that with 16 "virtual cores", you have the Xeon model of X4600 rather 
> than the AMD one. 
> I'd ignore it anyways, as you most likely have the relevant microcode loaded 
> by the BIOS. 
> > 
> > Not sure whether they appear when I experiment with HVM domUs...
> > 
> > 
> > > These may also not contain anything useful - but it would 
> > be where I'd start trying to figure out what's wrong. 
> > > > 
> > > > 
> > > > 
> > > > I managed to run dom0  which is behaving not so bad. 
> > > > 
> > > > Only several 'segfault's in dmesg confuse me.
> > > Where are those segfualts from?
> > 
> > Well, I was trying to compile dev86 there and the ncc 
> > compiler sigfaulted. Also irqbalance sigfaulted:
> > 
> > irqbalance[3258]: segfault at 0000000000528018 rip 
> > 00000000004016ba rsp 00007fff9cf9b8f0 error 4
> They shouldn't fail like that. 
> > 
> > > > 
> > > > 
> > > > 
> > > > I use SUN X4600 server which is of 64bits, 32gigs and 16 
> > virtual CPUs.
> > > I expect this to be capable of running Xen for sure. 
> > > --
> > > Mats
> > 
> > 
> > 
> > 

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users