[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] Debugging (possible) Xen-related hang-issues without the possibility of attaching serial console to capture Xen output

Hey all!

I'm currently in the process of trying to debug a (possibly!) Xen-related issue (with Kernel 3.2.9-vanilla as Dom0, and Xen 4.1.2 [almost] vanilla, except for a patch for CVE-2012-0029), where a system freezes without me being able to ascertain a triggering event. The system console is completely unresponsive in the event of hangs (i.e., displays motd and the login prompt, but doesn't display a Dom0 traceback or anything related), and I'm unable to pinpoint the exact source of the hang condition due to not being able to reproduce the hang under laboratory-conditions (which is the worst part about these hangs) with similar hardware (so, basically, it might actually not be Xen- but [hosting-]environment-related).

I'd already written about this some time ago, and the suggestion to enable network logging (through netconsole) is implemented on the system in question, but when the system locks up I'm also not seeing any output to the netconsole connected process on the remote logging machine - it seems that the Dom0-kernel is not "the guilty party" when these hangs occur.

Generally, I'd have started to attach a serial console to the system now to check whether the hypervisor logs any fatal errors when the system goes to a hang, but as the machine in question is located at a service provider (which I can't change easily and which adamantly refuses to help with debugging in the form of attaching a serial console), I'm unable to get at Xen serial output on the host in question.

Is there any, _any_ other way to get at hypervisor output in the case of a crash? i.e., some form of memory buffering of the output messages which I might later be able to find in the host memory after rebooting to a "working" state, or something similar? I know that FreeBSD (and lately) Linux implement this kind of crash debugging helpers - is there any possibility to get something similar working with Xen? Reverting to a kdump/kexec-enabled kernel (i.e., 2.6.18) for the Dom0 is (pretty much) impossible due to the hardware of the system in use, but if someone points me at a 2.6.34+ Xen-kexec/kdump-enabled Dom0-kernel, I'd also be happy to give that a try (I've not found any, or rather: kdump/kexec also doesn't work with the OpenSUSE-up-patched xenified 2.6.38 I had in use before switching to 3.2.x-vanilla, but the problems [hangs] were similar with the older kernel).

Thanks for any answer!

--- Heiko.

Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.