Re: [Xen-users] Crashing mythtv, mysqld and NAN errors all over the place in a domain

[Mark Williamson <mark.williamson@xxxxxxxxxxxx>]

> Im trying to build a mythtv domain - its almost working - it can see
> thje DVB cards, and can record and so on, but occasionally (usually
> when MythTV is hitting the mysql database on the same domain) im
> getting some wierd crashes in MythTV.
> When it crashes, the process (mythbackend) says 'killed' -

Weird.  So you're assigning the DVB cards to this domain...  Have you 
remembered to hide the card from dom0?  Does Xen confirm that this hiding has 
occurred when it boots up?

Can you post menu.lst, the domain's config, the output of lspci (under vanilla 
linux) and the output of xm dmesg?

> 1. is is possible the kernel is killing the process because its
> misbehaving? dmesg doesnt say anything like this

It could be...  But a kernel error could be occurring whilst this process is 
running.

> 2. The domain is allocated 256meg of ram - just before it crashes, it
> seems to have around 4 megs of ram left (using top), and loads of swap
> space (but its used NO swapspace which seems suspect). If I decrease
> the ram to 32meg, it does use the swapspace, but still crashes.

It's very strange for Linux not to use any swap space!  If you don't try 
assigning a device to the domain and use it as a fully unprivileged domain, 
does it behave more normally?  I can't see why it would make a difference 
though...

> 3. When its gone wrong, there seems to be a problem with other
> services as well - sshd wont let me connect (but recovers after 5 mins
> or so) and mysqld also crashes (with errors in the log that dont tell
> me much - see below).

Ouch!

> 6. It crashes even when I dont have the DVB cards 'passed through' to
> the domain, so its not a driver issue.

Ah OK, that's very weird.  Is it just MythTV that breaks this?  What happens 
if you try running it in another domain?

> 7. The other thing thats happening in this domain, is that im getting
> NaN (not a number) reported all over the place  (eg, in top it reports
> NaN% processor usage) and mythtv keeps building queries that say to
> insert NaN into a table. This seems associated with times more than
> anything.

That's probably a clue as to the source of the error but it's not obvious to 
me what it could be.

> 8. It looks to me a lot like the domain is running out of resources of
> some kind - perhaps memory and its refusing to use the swap space for
> some reason?

Are you sure swap space is configured OK?

> 9. I am getting errors about the hardware clock when the domain boots,
> but I expect this as I have not passed through

Yeah, that's normal.  Domains can't program the hardware clock directly, nor 
do they need to.

> 10. Also, sleep has bombed a couple of times, with a ASSERT error
> (Assert(time > 0)) - could this be related to the 'time is going
> backwards' issue?

"Time going backwards" is a Xen error and probably not exactly the problem 
here.  Nonetheless, something weird does seem to be happening to time on your 
system.

> 11. Would I benefit from upgrading Xen to the latest testing version?
> is this a known issue?

You could try it.  There are a few bug fixes in -testing but I don't know of 
anything in your version that'd cause this to happen...

Cheers,
Mark

>
> thanks
>
> George
>
> The MySQL errors are:
>
> 050516 22:43:13  mysqld restarted
> 050516 22:43:14  InnoDB: Started; log sequence number 0 43634
> /usr/sbin/mysqld: ready for connections.
> Version: '4.1.10a'  socket: '/var/lib/mysql/mysql.sock'  port: 3306
> SUSE MySQL RPM
> mysqld got signal 11;
> This could be because you hit a bug. It is also possible that this binary
> or one of the libraries it was linked against is corrupt, improperly built,
> or misconfigured. This error can also be caused by malfunctioning hardware.
> We will try our best to scrape up some info that will hopefully help
> diagnose the problem, but since we have already crashed, something is
> definitely wrong and this may fail.
>
> key_buffer_size=16777216
> read_buffer_size=258048
> max_used_connections=1
> max_connections=100
> threads_connected=1
> It is possible that mysqld could use up to
> key_buffer_size + (read_buffer_size +
> sort_buffer_size)*max_connections = 92783 K
> bytes of memory
> Hope that's ok; if not, decrease some variables in the equation.
>
> thd=0x89f3ab8
> Attempting backtrace. You can use the following information to find out
> where mysqld died. If you see no messages after this, something went
> terribly wrong...
> Cannot determine thread, fp=0x42ee4c38, backtrace may not be correct.
> Stack range sanity check OK, backtrace follows:
> 0x8168e0b
> 0x4005d105
> 0x81b01e8
> 0x81b0a09
> 0x81b3940
> 0x81b7e7f
> 0x8183306
> 0x8184fd1
> 0x8186be4
> 0x818753b
> 0x81880a7
> 0x40056e31
> 0x4028d6fa
> New value of fp=(nil) failed sanity check, terminating stack trace!
> Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html
> and follow instructions on how to resolve the stack trace. Resolved
> stack trace is much more helpful in diagnosing the problem, so please do
> resolve it
> Trying to get some variables.
> Some pointers may be invalid and cause the dump to abort...
> thd->query at 0x8a17428 = SELECT data FROM settings WHERE value =
> "RecordChanged"
> thd->thread_id=1
> The manual page at http://www.mysql.com/doc/en/Crashing.html contains
> information that should help you find out what is causing the crash.
>
> Number of processes running now: 1
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users