Re: [Xen-devel] Detecting deadlocks with hypervisor..

["T S" <thileepan_@xxxxxxxxxxx>]

> From: Ewan Mellor
> To: Thileepan Subramaniam CC: xen-devel@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [Xen-devel] Detecting deadlocks with hypervisor..
> Date: Sun, 19 Mar 2006 13:17:35 +0000
>
> On Sat, Mar 18, 2006 at 06:14:09PM -0800, Thileepan Subramaniam wrote:
>
> > Hello,
> >
> > I am trying to see if the hypervisor can be used to detect deadlocks in 
> the
> > guest VMs. My goal is to detect if a guest OS is deadlocked, and if it 
> is,
> > then create a clone of the deadlocked OS without the locking condition, 
> and
> > letting the clone run. While the clone runs I am hoping to generate some
> > hints that could tell me what caused the deadlock.
> >
> > I simulated a deadlock/hang situation in a guest OS (by loading a badly
> > written module to the kernel) and when the guestOS kernel was hanging, I
> > ran "xm save" from Dom-0. But this command waits forever.
> >
> > I tried to follow the flow of the .py files (XendCheckpoint.py etc.). 
> These
> > seem to be called when I run 'xm save'. But beyond a point I am not sure
> > what the python scripts do. I also see some libxc files such as
> > xc_linux_save.c, but I am not sure who is using it (Dom-0 or Xen or the
> > XenU). Can someone help me by explaining me what happens behind the 
> scene
> > when "xm save" is called ? Is there any good documentation explaining 
> which
> > actions are done by which layers (eg: python layer, C layer etc).
>
> xc_save, the executable, calls xc_linux_save, the libxc function.  
> Depending
> upon whether this is a live or non-live save, some stuff is done (see
> xc_linux_save for details).  The Python layer is then called back, 
> requesting
> that the domain is suspended.  This request is passed through to the guest 
> by
> writing /local/domain/<domid>/control/shutdown = suspend in the store.  
> This
> is seen by the guest (a watch fires inside reboot.c) and then the guest
> suspends itself.  This is probably where you are falling down -- if the 
> guest
> kernel is completely deadlocked, it's going to struggle to suspend itself
> correctly.
>
> If a suspend completes correctly, Xend will see it (another watch will 
> fire),
> and xc_linux_save will be free to complete the save.
So, I went and experimented this: basically, I changed XendCheckpoint.py to 
NOT wait for the guest to shutdown; I also changed xc_linux_save() to 
proceed saving without waiting (essentially, suspend_and_state() returns 0 
instead of retrying repeateedly). With this I am able to save a deadlocked 
kernel smoothly.
But when I try restore, I get this error message:
Error: /usr/lib/xen/bin/xc_restore 10 19 5 34816 1 2 failed

And the log says,
[2006-03-24 13:48:42 xend] DEBUG (XendCheckpoint:152) [xc_restore]: 
/usr/lib/xen/bin/xc_restore 10 19 5 34816 1 2
[2006-03-24 13:48:42 xend] ERROR (XendCheckpoint:231) xc_linux_restore 
start: max_pfn = 8800
[2006-03-24 13:48:42 xend] ERROR (XendCheckpoint:231) Increased domain 
reservationby22000KB
[2006-03-24 13:48:42 xend] ERROR (XendCheckpoint:231) Reloading memory 
pages:   0%
[2006-03-24 13:48:54 xend] ERROR (XendCheckpoint:231) Received all pages (0 
races)
[2006-03-24 13:48:54 xend] ERROR (XendCheckpoint:231) Failed to pin batch of 
22 page tables: 22
[2006-03-24 13:48:54 xend] ERROR (XendCheckpoint:231) Restore exit with rc=1

Any clue .. so that i can overcome this and restore the kernel to its 
previous state (i.e., deadlocked state) ?
thanks,
TS

> > Also, does it seem viable to clone a copy of a deadlocked guest OS in 
> the
> > first place?
>
> If you have a byte-for-byte copy of a deadlocked guest, even if you could
> suspend it, surely it will be deadlocked when it is resumed.  How do you
> intend to break the deadlock, and how is it easier to do that from outside
> than it is to perform deadlock detection in the guest?
>
> Ewan.
_________________________________________________________________
FREE pop-up blocking with the new MSN Toolbar ? get it now! 
http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel