Re: [Xen-users] Shutting down domU causes "hda: interrupt lost" on dom0 and freezes the box.

[Mark Williamson <mark.williamson@xxxxxxxxxxxx>]

Thanks Jamie,

I think your debugging has identified the cause of the problem with a decent 
degree of certainty.  Sounds like it's more or less what I had originally 
suspected:

The guest OS has to be able to mask interrupt lines in order to operate the 
PCI device it's been given.  If that interrupt line is shared with dom0 
devices then the domU can mask a line that dom0 needs to get interrupts on.  
That's fine if the guest is well behaved but in your situation the guest was 
leaving the line masked when it was suspended - at least that's roughly what 
it sounds like, although I couldn't describe the exact chain of events.

Your pulling BIOS switches around sounds like it's distributed the IRQs across 
multiple lines, which means that the guest screwing up doesn't break dom0.  
This is also beneficial for performance in any case (imagine having to upcall 
into two entire operating systems just to find out which driver is interested 
in an interrupt!) so it's a good switch to make in any case, if it works.

> 1. Issuing a "poweroff" from within a SSH session on the domU works fine.

This would be since the guest was able to shutdown cleanly and not leave the 
hardware in a funny state.

> 3. Executing "xm destroy <domU>" causes 'irq 16: nobody cared (try booting
> with the "irqpoll" option' to be written to the console, followed by a dom0
> kernel panic. Yay!

A fully panic?  Eww!  Anyhow, sounds like the domU hadn't been able to shut 
down the hardware right, so there were unexpected IRQs coming through.  I'm 
not sure why you'd get a panic though, that sounds nastier.

> 2. Executing "xm save <domU> filename" causes the same HDD errors I
> previously mentioned.

Right.  Your init scripts were probably doing an xm save on reboot, I guess.

Did the xm save actually appear to proceed?  Did it even succeed?  It's a bit 
scary if it does since I'm not at all sure it's a supported configuration (it 
wasn't last time I heard; I had thought we'd disabled this).  Can you remind 
me what version of Xen you're running and where you got it from?

IMPORTANT NOTE: xm save-ing and xm restore-ing guests with passed-through PCI 
devices doesn't sound like it should make much sense.  I wouldn't be at all 
surprised if it was actually unsafe and I certainly wouldn't be surprised if 
it didn't actually work properly.  You should look into disabling this before 
production use; make sure they get shut down conventionally and then booted 
from scratch when dom0 next comes up.  It will save you headaches!

> Mainly out of frustration, I then just decided to reboot the physical
> server.  For some reason, that fixed that problem. It now works.  No
> conflicts with the storage controller, no problems shutting down domU, and
> the card still works fine in Asterisk.  Problem solved.

Ho hum, not sure why that helped but if it works reliably we'll just smile, 
nod and move on!

> The very first computer problem I ever solved was changing the IRQ
> selection jumpers on an internal modem because it was conflicting with the
> IO card in my 286.  I can't believe the same problem is still stumping me
> almost 20 years later.

At least we're not having to recompile Xen and put it on a special floppy 
because we don't have enough base memory free.  Some things have moved on ... 
slightly.

Anyhow, I bet when you had to change the jumpers around the problem was only 
able to take down one OS.  With modern technology, you can blow away several 
at once :-D

Cheers,
Mark

> -----Original Message-----
> From: M.A. Williamson [mailto:maw48@xxxxxxxxxxxxxxxx] On Behalf Of Mark
> Williamson
> Sent: Tuesday, April 29, 2008 2:21 PM
> To: xen-users@xxxxxxxxxxxxxxxxxxx
> Cc: Jamie J. Begin
> Subject: Re: [Xen-users] Shutting down domU causes "hda: interrupt lost" on
> dom0 and freezes the box.
>
> > When I try to reboot dom0, it switches to runlevel 6 and the xen init.d
> > script attempts to stop a domU containing an Asterisk installation.  It's
> > at that point I get an "hda: interrupt lost" on the physical console. 
> > SSH become inaccessible and eventually the system pukes up a bunch of
> > ext3 and RAID controller related errors and freezes.  I have to
> > physically power cycle the box to get it back up.
>
> Ugh, that's nasty :-(
>
> > I suspect that a PCIe telephony card that I'm passing to the domU using
> > pciback is the source of the problem.  The card is a Digium AEX800 (which
> > is actually a PCIe version of Digium's PCI-based TDM800P). Based on some
> > preliminary testing, the card seems to function just fine in the domU.
> > lspci output is:
>
> I was actually just thinking "I wonder if he's using PCI passthrough" ;-)
>
> A few thoughts spring to mind:
>
> 1) Any idea if this is happening during a normal shutdown of the domU or if
> that shutdown is timing out, resulting in the domain being rudely
> destroyed?
>
> 2) Is there any chance that the domains are being suspended rather than
> shutdown?  That might do funny things...
>
> 3) Does this happen if you manually shutdown the domain?  if you manually
> destroy the domain?
>
> It's also possible that this is some kind of bug in the PCI passthrough.  I
> didn't actually know that it worked for PCIe, but it's nice to know that it
> (sort of) does :-)
>
> I apologise for suggesting testing on your own system; I imagine that doing
> this repeatedly is not doing your filesystem consistency any good :-(  It's
> possible that you'll get more suggestions than I've been able to provide
> from
> the xen-devel list.  Still, if you feel like doing some testing it may
> help.
>
> It's also possible that there have been other reports like this, although I
> don't remember hearing of them.  Have you done a quick search of the
> mailing
>
> list archives and the bugzilla?  (or even just a google, in case someone
> grumbled about it on their blog).
>
> The Asterisk-in-a-domU configuration seems to be rather popular, which is
> cool :-)
>
> Cheers,
> Mark
>
> > 0b:08.0 Ethernet controller: Digium, Inc. Unknown device 8002 (rev 11)
> >
> >         Subsystem: Digium, Inc. Unknown device 8002
> >
> >         Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
>
> ParErr-
>
> > Stepping- SERR- FastB2B-
> >
> >         Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> > <TAbort- <MAbort- >SERR- <PERR-
> >
> >         Interrupt: pin A routed to IRQ 16
> >
> >         Region 0: I/O ports at dc00 [disabled] [size=256]
> >
> >         Region 1: Memory at fc7dfc00 (32-bit, non-prefetchable)
> > [disabled] [size=1K]
> >
> >         Expansion ROM at fc7e0000 [disabled] [size=128K]
> >
> >         Capabilities: [c0] Power Management version 2
> >
> >                 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> > PME(D0+,D1+,D2+,D3hot+,D3cold-)
> >
> >                 Status: D0 PME-Enable- DSel=0 DScale=0 PME-
> >
> >
> >
> > Any suggestions?  This is a new Dell PowerEdge 1950 with a PERC SATA RAID
>
> 1
>
> > array, running CentOS 5.1 (2.6.18-53.1.14.el5xen) in both the dom0 and
> > domU.



-- 
Push Me Pull You - Distributed SCM tool (http://www.cl.cam.ac.uk/~maw48/pmpu/)

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users