Re: [Xen-devel] can't start domU

[Steve Dobbelstein <steved@xxxxxxxxxx>]

Ewan Mellor <ewan@xxxxxxxxxxxxx> wrote on 11/15/2005 05:34:48 PM:

Thanks for your assitance, Ewan.

> On Tue, Nov 15, 2005 at 01:59:25PM -0600, Steve Dobbelstein wrote:
>
> > "Charles Coffing" <ccoffing@xxxxxxxxxx> wrote on 11/11/2005 05:13:25
PM:
> >
> > > >>> On Fri, Nov 11, 2005 at  3:44 pm, in message
> > > <43751EDA.80304@xxxxxxxxxx>,
> > > Nivedita Singhvi <niv@xxxxxxxxxx> wrote:
> > > > Charles Coffing wrote:
> > > >> I've been getting hangs while attempting to start domUs on recent
> > > >> changesets.  I get an error such as:
> > > >> Error: Device 0 (vif) could not be connected. Hotplug scripts not
> > > >> working.
> > > [snip]
> > > >
> > > > All of these are failures to create the vif devices.
> > > > Please check /var/log/messages, xend- debug.log, etc.
> > > > You can put a set - x in your /etc/xen/vif- bridge (or equivalent)
> > > > script for more verbose detail.
> > > >
> > > > Also, check to see that your install went successfully.
> > > >
> > > >  From previous debugging over the past week, most of
> > > > these cases were the lack of brctl (or execute perms
> > > > for root), or the lack of other utilities (make install
> > > > should catch these).
> > >
> > > The obvious stuff looks good:  The install went okay; brctl exists;
all
> > > scripts are executable.
> > >
> > > I'll look more closely at the bridge script.
> >
> > Has anyone found a solution to this problem?  I'm seeing the same thing
but
> > didn't want to just post a "me, too".
> >
> > My domU config file does not have a "vif" entry; the defaults are fine
for
> > me.
> >
> > /var/log/messages, /var/log/xend-debug.log, and /var/log/xend.log don't
> > contain any additional information; they look the same as the ones on a
> > system that is working.
> >
> > I added lines to /etc/xen/scripts/vif-bridge:
> > echo "Turning debugging on"
> > echo "turning debugging on">/tmp/vif-bridge.out
> > set -x
> >
> > I don't get any debug output upon running "xm create".  I don't get a
file
> > /tmp/vif-bridge, so it appears that vif-bridge isn't even called.
> >
> > hotplug is at version 058.  I tried the "echo /sbin/hotplug >
> > /proc/sys/kernel/hotplug" trick, but that didn't help.  Besides, the
system
> > that works is also running hotplug 058 and it has
/proc/sys/kernel/hotplug
> > set to /sbin/udevsend.
> >
> > Apparently something is not creating the vif for the domain.  I have
been
> > crawling through the xend scripts to try and figure out where vifs get
> > created in hopes of finding some clues as to what is going wrong, but
being
> > barely a python newbie I get lost in the code.  :(  (Time to add
another
> > programming language to my skill set.)
>
> Have you tried today's changeset?  There have been plenty of changes in
this
> area in the last week, and a few today.

I grabbed the latest source, 7826, and still have the problem.

> What distro?  (Not that this will help, but it seems a good idea to keep
track
> of problematic distro versions.)

Fedora Core 4 for both the dom0 and the domU

> Does /sbin/hotplug or /sbin/udevsend run at all?
>
> If so, does it make it as far as /etc/hotplug/xen-backend.agent?

I'm not sure how to tell if /sbin/udevsend binary gets run.  So I did the
"echo /sbin/hotplug > /proc/sys/kernel/hotplug" trick so that I could hack
debugging stuff into the /etc/hotplug/xen-backend.agent script.  It now
looks like:

-----------
#! /bin/sh

date=`date`
echo "$date $0 $*" >> /var/log/xen-backend.log
echo "XENBUS_TYPE = \"$XENBUS_TYPE\"  ACTION = \"$ACTION\"" >>
/var/log/xen-backend.log
PATH=/etc/xen/scripts:$PATH

case "$XENBUS_TYPE" in
  vbd)
    echo "/etc/xen/scripts/block $ACTION" >> /var/log/xen-backend.log
    /etc/xen/scripts/block "$ACTION"
    ;;
  vif)
    if [ -n "$script" ]; then
      echo "$script $ACTION" >> /var/log/xen-backend.log
    else
      echo "\$script is empty" >> /var/log/xen-backend.log
    fi
    [ -n "$script" ] && $script "$ACTION"
    ;;
esac

case "$ACTION" in
  add)
    ;;
  remove)
    # remove device backend store entries
    xenstore-rm -t "$XENBUS_PATH"       || true
    xenstore-rm -t "error/$XENBUS_PATH" || true
    ;;
  online)
    ;;
  offline)
    ;;
esac
-----------

After I run "xm create /tmp/dom1/cfg" /var/log/xen-backend.log has:

-----------
Wed Nov 16 10:54:37 CST 2005 /etc/hotplug/xen-backend.agent
XENBUS_TYPE = "vbd"  ACTION = "add"
/etc/xen/scripts/block add
Wed Nov 16 10:54:37 CST 2005 /etc/hotplug/xen-backend.agent
XENBUS_TYPE = "vbd"  ACTION = "add"
/etc/xen/scripts/block add
-----------

Looks like it doesn't get called to create the vif.

> If so, how about /etc/xen/scripts/vif-common.sh?

I added similar debugging statements to /etc/xen/scripts/vif-bridge and
/etc/xen/scripts/vif-common.sh.  I don't get any log file for either vif-*
script.  Looks like they are not called.

> What do you get if you run
>
> PYTHONPATH=/usr/lib/python python /usr/lib/python/xen/util/diagnose.
> py <dom ID>

-----------
[root@gir scripts]# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      768     4 r-----   112.4
dom1                               4      768     4 -b----    12.3
[root@gir scripts]# PYTHONPATH=/usr/lib/python python
/usr/lib/python/xen/util/diagnose.py 4
Domain ID is 4.
Domain name is dom1.
Found 1 device classes in use.
Found 2 vbd devices.
Found device vbd, 2049.
Backend is in state Connected.
Frontend is in state Connected.
Device vbd, 2049 hotplugging has completed successfully, and is connected
to physical device 0xfd01.
Found device vbd, 2050.
Backend is in state Connected.
Frontend is in state Connected.
Device vbd, 2050 hotplugging has completed successfully, and is connected
to physical device 0xfd00.
-----------

FWIW, /tmp/dom1.cfg says:

-----------
kernel = "/boot/vmlinuz-xenU-smp"
memory = 768
name = "dom1"
vcpus = 4
nics = 1
disk = ['phy:/dev/virt-blkdev-backend/dom1,sda1,w',
'phy:/dev/virt-blkdev-backend/usr,sda2,r']
root = "/dev/sda1 ro"
extra = " profile=1 GATEWAY=10.0.1.254 NETMASK=255.255.0.0
IPADDR=10.0.134.1 HOSTNAME=dom1"
on_poweroff = 'destroy'
on_reboot   = 'destroy'
on_crash    = 'destroy'
-----------

> Thanks for your help,

Thanks for yours.

> Ewan.

Steve D.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel