[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] debugging VM startup - update



well - looks like one of my file systems was hosed (not boot, root, or swap) - seems like the Debian boot scripts don't report stuff going on in the middle of /etc/rcS.d/S35mountall.sh - they log errors, but don't report to the console (at least with stock configuration).

Had to mount the root directory and pour through logs for the DomU to find this

running an fsck on it now

still begs the question of how to do single-user, or other diagnostic boot of a domU

Miles Fidelman wrote:
A little more data:

----background-------
I have a Debian domU that is refusing to boot. Attaching to the console (xm console) it seems to be in the middle of mounting a drive, and then hanging.

....
[    2.387986] EXT3-fs: mounted filesystem with ordered data mode.
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.86 booting
Starting the hotplug events dispatcher: udevd[ 5.467529] udevd version 125 started
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...done.
Starting boot logger: bootlogd[ 6.190409] Adding 3145588k swap on /dev/xvda2. Priority:-1 extents:1 across:3145588k
[   89.796051] EXT3 FS on xvda1, internal journal
<hang>

I know how I might proceed if I were booting a physical machine - try to come up in single user mode, failing that, boot with a live CD and start examining things. But... this is a VM, with more limited "hooks" into the boot process. Any suggestions on how to proceed?

Note, I'm running what's supposed to be a 2-node high-availability configuration (DRBD-mirrored drives, pacemaker). The underlying RAID10 and DRBD arrays are reporting good status, and other VMs are running just fine. This one just refuses to come up on either node in the cluster.
------------

1. looking at another domU that's working fine, I see that its console log reports:

.....
 [  31.088678] EXT3 FS on xvda1, internal journal
[   42.316654] NET: Registered protocol family 10
----

while my broken machine is hanging between these two steps - suggesting either a disk problem or a networking problem

2. I mounted and fsck'd the volume, and it seems to be fine, so it looks like something is either broken in the network startup, or interfering with it.

Which leads to three questions:

1. any suggestions on what might spontaneously go wrong with a previously-functioning network startup?
2. any debugging suggestions?
3. specifically, any suggestions on how to insert myself into a boot process that is normally configured to be driven by pacemaker and xen's startup hooks - so I can do some step by step debugging of the boot process?

Thanks again,

Miles Fidelman


Miles Fidelman wrote:
Hi,

I have a Debian domU that is refusing to boot. Attaching to the console (xm console) it seems to be in the middle of mounting a drive, and then hanging.

....
[    2.387986] EXT3-fs: mounted filesystem with ordered data mode.
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.86 booting
Starting the hotplug events dispatcher: udevd[ 5.467529] udevd version 125 started
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...done.
Starting boot logger: bootlogd[ 6.190409] Adding 3145588k swap on /dev/xvda2. Priority:-1 extents:1 across:3145588k
[   89.796051] EXT3 FS on xvda1, internal journal
<hang>

I know how I might proceed if I were booting a physical machine - try to come up in single user mode, failing that, boot with a live CD and start examining things. But... this is a VM, with more limited "hooks" into the boot process. Any suggestions on how to proceed?

Note, I'm running what's supposed to be a 2-node high-availability configuration (DRBD-mirrored drives, pacemaker). The underlying RAID10 and DRBD arrays are reporting good status, and other VMs are running just fine. This one just refuses to come up on either node in the cluster.

Thanks,

Miles Fidelman







--
In theory, there is no difference between theory and practice.
In<fnord>  practice, there is.   .... Yogi Berra



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.