[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] PV guests hang at boot time


  • To: xen-users@xxxxxxxxxxxxxxxxxxx
  • From: Maarten Vanraes <maarten@xxxxx>
  • Date: Fri, 23 Oct 2009 12:02:18 +0200
  • Delivery-date: Fri, 23 Oct 2009 03:03:44 -0700
  • List-id: Xen user discussion <xen-users.lists.xensource.com>

i booted 8 machines:

test1: hung after "XENBUS: Device with no driver: device/console/0"
test2: hung after "XENBUS: Device with no driver: device/console/0"
test3: "Unable to mount root fs" list partitions: "sda1 sdb1" and shut down
test4: hung after "XENBUS: Device with no driver: device/console/0"
test5: booted completely, SSH works, can write files, ...
test6: booted completely, SSH works, can write files, ...
test7: "Unable to mount root fs" list partitions: "sda1 sdb1" and shut down
test8: hung after "XENBUS: Device with no driver: device/console/0"

- all 8 xm's are defunct on the host.

- i have about 15 "/var/run/tap/tapctrlwrite35 /var/run/tap/tapctrlread35" 
processes and 2 "/var/run/tap/tapctrlwrite34 /var/run/tap/tapctrlread34"

- all 6 up machines have 2 blktap processes (sda1 and sdb1)

is there ANYTHING i can try?

does anyone have any idea???

i'm completely stuck here...

i've already rebooted the host, to see if it would help.



Op Friday 23 October 2009 11:32:57 schreef Maarten Vanraes:
> Hello,
>
> i used xen 3.2 before on 2.6.21, now i'm trying 3.4.0 on 2.6.31-4.
>
> my domU's hang at booting time...
>
> I've been reading and looking around for about a week now.
>
> I've identified 3 different places to hang (tested with the same domUs):
>  * right after kernel booted, before init
>  * after "mounting swap..."
>  * after "starting crond"
>  * sometimes it boots until login and i can login fine.
>
> i'm alway starting with "xm create -c <domname>"
>
> in the host: i see that the "xm" command is always defunct.
>
> if it's after starting crond. then SSH is also started, and i can ssh fine.
> if it's before, i cannot SSH. (this also happens if i manually start with
> "xm create ", and do xm console later; the only difference is that there is
> no defunct xm process).
>
> I'm using the ebuild from the gentoo-patches and google code with kernel
> 2.6.31-r4 .
>
> both dom0 and domU kernels are that kernel, allthough they have different
> configs. (see below)
>
> my domUs are debootstrapped debian4, with udev and syslog-ng installed.
>
> I tried booting also with extra='console=xvc0' and xencons=tty1 in
> different combinations, but that was not it.
>
> I thought it was something console-related, because of the error (see
> below): "XENBUS: Device with no driver: device/console/0"
>
> the xen log files show no error, i can figure out.
>
> also something i see, i also tried different things here, but cant find it:
> PCI: Fatal: No config space access function found
> PCI: setting up Xen PCI frontend stub
>
>
> xend-debug log does have errors, but i don't think they are serious:
> ---------------------------------------------------------------------------
>------------- cat: /sys/bus/scsi/devices/host0/vendor: No such file or
> directory cat: /sys/bus/scsi/devices/host0/model: No such file or directory
> cat: /sys/bus/scsi/devices/host0/type: No such file or directory
> cat: /sys/bus/scsi/devices/host0/rev: No such file or directory
> cat: /sys/bus/scsi/devices/host0/scsi_level: No such file or directory
> cat: /sys/bus/scsi/devices/host1/vendor: No such file or directory
> cat: /sys/bus/scsi/devices/host1/model: No such file or directory
> cat: /sys/bus/scsi/devices/host1/type: No such file or directory
> cat: /sys/bus/scsi/devices/host1/rev: No such file or directory
> cat: /sys/bus/scsi/devices/host1/scsi_level: No such file or directory
> cat: /sys/bus/scsi/devices/host2/vendor: No such file or directory
> cat: /sys/bus/scsi/devices/host2/model: No such file or directory
> cat: /sys/bus/scsi/devices/host2/type: No such file or directory
> cat: /sys/bus/scsi/devices/host2/rev: No such file or directory
> cat: /sys/bus/scsi/devices/host2/scsi_level: No such file or directory
> cat: /sys/bus/scsi/devices/host3/vendor: No such file or directory
> cat: /sys/bus/scsi/devices/host3/model: No such file or directory
> cat: /sys/bus/scsi/devices/host3/type: No such file or directory
> cat: /sys/bus/scsi/devices/host3/rev: No such file or directory
> cat: /sys/bus/scsi/devices/host3/scsi_level: No such file or directory
> cat: /sys/bus/scsi/devices/host4/vendor: No such file or directory
> cat: /sys/bus/scsi/devices/host4/model: No such file or directory
> cat: /sys/bus/scsi/devices/host4/type: No such file or directory
> cat: /sys/bus/scsi/devices/host4/rev: No such file or directory
> cat: /sys/bus/scsi/devices/host4/scsi_level: No such file or directory
> cat: /sys/bus/scsi/devices/host5/vendor: No such file or directory
> cat: /sys/bus/scsi/devices/host5/model: No such file or directory
> cat: /sys/bus/scsi/devices/host5/type: No such file or directory
> cat: /sys/bus/scsi/devices/host5/rev: No such file or directory
> cat: /sys/bus/scsi/devices/host5/scsi_level: No such file or directory
> cat: /sys/bus/scsi/devices/host6/vendor: No such file or directory
> cat: /sys/bus/scsi/devices/host6/model: No such file or directory
> cat: /sys/bus/scsi/devices/host6/type: No such file or directory
> cat: /sys/bus/scsi/devices/host6/rev: No such file or directory
> cat: /sys/bus/scsi/devices/host6/scsi_level: No such file or directory
> cat: /sys/bus/scsi/devices/host7/vendor: No such file or directory
> cat: /sys/bus/scsi/devices/host7/model: No such file or directory
> cat: /sys/bus/scsi/devices/host7/type: No such file or directory
> cat: /sys/bus/scsi/devices/host7/rev: No such file or directory
> cat: /sys/bus/scsi/devices/host7/scsi_level: No such file or directory
> cat: /sys/bus/scsi/devices/host8/vendor: No such file or directory
> cat: /sys/bus/scsi/devices/host8/model: No such file or directory
> cat: /sys/bus/scsi/devices/host8/type: No such file or directory
> cat: /sys/bus/scsi/devices/host8/rev: No such file or directory
> cat: /sys/bus/scsi/devices/host8/scsi_level: No such file or directory
> cat: /sys/bus/scsi/devices/target8:0:0/vendor: No such file or directory
> cat: /sys/bus/scsi/devices/target8:0:0/model: No such file or directory
> cat: /sys/bus/scsi/devices/target8:0:0/type: No such file or directory
> cat: /sys/bus/scsi/devices/target8:0:0/rev: No such file or directory
> cat: /sys/bus/scsi/devices/target8:0:0/scsi_level: No such file or
> directory cat: /sys/bus/scsi/devices/target8:0:1/vendor: No such file or
> directory cat: /sys/bus/scsi/devices/target8:0:1/model: No such file or
> directory cat: /sys/bus/scsi/devices/target8:0:1/type: No such file or
> directory cat: /sys/bus/scsi/devices/target8:0:1/rev: No such file or
> directory cat: /sys/bus/scsi/devices/target8:0:1/scsi_level: No such file
> or directory cat: /sys/bus/scsi/devices/target8:1:0/vendor: No such file or
> directory cat: /sys/bus/scsi/devices/target8:1:0/model: No such file or
> directory cat: /sys/bus/scsi/devices/target8:1:0/type: No such file or
> directory cat: /sys/bus/scsi/devices/target8:1:0/rev: No such file or
> directory cat: /sys/bus/scsi/devices/target8:1:0/scsi_level: No such file
> or directory
>
>
>
>
> xm console output:
> -------------------------
> Using config file "/etc/xen/vm-test6".
> Started domain vm-test6 (id=15)
>                                [    0.000000] Linux version 2.6.31
> (root@sparco01) (gcc version 4.3.2 (Gentoo 4.3.2-r3 p1.6, pie-10.1.5) ) #15
> SMP Thu Oct 22 16:44:31 CEST 2009
> [    0.000000] Command line: root=/dev/sda1 ro console=xvc0 xencons=tty1
> [    0.000000] KERNEL supported cpus:
> [    0.000000]   Intel GenuineIntel
> [    0.000000]   AMD AuthenticAMD
> [    0.000000]   Centaur CentaurHauls
> [    0.000000] maxmem of 4104M is invalid for an initial allocation of
> 128M, using 4096M
> [    0.000000] Xen-provided physical RAM map:
> [    0.000000]  Xen: 0000000000000000 - 0000000100000000 (usable)
> [    0.000000] last_pfn = 0x100000 max_arch_pfn = 0x80000000
> [    0.000000] init_memory_mapping: 0000000000000000-0000000100000000
> [    0.000000] (3 early reservations) ==> bootmem [0000000000 - 0008000000]
> [    0.000000]   #0 [0001485000 - 00014d7000]     Xen provided ==>
> [0001485000 - 00014d7000]
> [    0.000000]   #1 [0001000000 - 0001464cb8]    TEXT DATA BSS ==>
> [0001000000 - 0001464cb8]
> [    0.000000]   #2 [00014d7000 - 0001cdc000]          PGTABLE ==>
> [00014d7000 - 0001cdc000]
> [    0.000000] Zone PFN ranges:
> [    0.000000]   DMA      0x00000000 -> 0x00001000
> [    0.000000]   DMA32    0x00001000 -> 0x00100000
> [    0.000000]   Normal   0x00100000 -> 0x00100000
> [    0.000000] Movable zone start PFN for each node
> [    0.000000] early_node_map[2] active PFN ranges
> [    0.000000]     0: 0x00000000 -> 0x00008000
> [    0.000000]     0: 0x00100000 -> 0x00100000
> [    0.000000] NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:1 nr_node_ids:1
> [    0.000000] PERCPU: Allocated 11 4k pages, static data 41120 bytes
> [    0.000000] Swapping MFNs for PFN 13a6 and 1493 (MFN 2606cc and 2605df)
> [    0.000000] Built 1 zonelists in Zone order, mobility grouping on. 
> Total pages: 18432
> [    0.000000] Kernel command line: root=/dev/sda1 ro console=xvc0
> xencons=tty1
> [    0.000000] PID hash table entries: 512 (order: 9, 4096 bytes)
> [    0.000000] Dentry cache hash table entries: 16384 (order: 5, 131072
> bytes) [    0.000000] Inode-cache hash table entries: 8192 (order: 4, 65536
> bytes) [    0.000000] Initializing CPU#0
> [    0.000000] Software IO TLB disabled
> [    0.000000] Memory: 52288k/4194304k available (2511k kernel code,
> 4063232k absent, 78716k reserved, 1190k data, 220k init)
> [    0.000000] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0,
> CPUs=1, Nodes=1
> [    0.000000] NR_IRQS:336
> [    0.000000] Xen reported: 2493.748 MHz processor.
> [    0.000999] Console: colour dummy device 80x25
> [    0.000999] console [tty-1] enabled
> [    0.862897] Calibrating delay using timer specific routine.. 4990.76
> BogoMIPS (lpj=2495382)
> [    0.862945] Mount-cache hash table entries: 256
> [    0.863126] CPU: L1 I cache: 32K, L1 D cache: 32K
> [    0.863131] CPU: L2 cache: 6144K
> [    0.863142] SMP alternatives: switching to UP code
> [    0.878077] Freeing SMP alternatives: 22k freed
> [    0.878172] Brought up 1 CPUs
> [    0.879684] NET: Registered protocol family 16
> [    0.880701] Brought up 1 CPUs
> [    0.880910] PCI: Fatal: No config space access function found
> [    0.880919] PCI: setting up Xen PCI frontend stub
> [    0.883166] bio: create slab <bio-0> at 0
> [    0.883483] suspend: event channel 9
> [    0.883782] xen_mem: Initialising balloon driver.
> [    1.018676] SCSI subsystem initialized
> [    1.018815] usbcore: registered new interface driver usbfs
> [    1.018853] usbcore: registered new interface driver hub
> [    1.018893] usbcore: registered new device driver usb
> [    1.019198] PCI: System does not support PCI
> [    1.019202] PCI: System does not support PCI
> [    1.019616] NET: Registered protocol family 2
> [    1.019675] IP route cache hash table entries: 4096 (order: 3, 32768
> bytes) [    1.019865] TCP established hash table entries: 16384 (order: 6,
> 262144 bytes)
> [    1.020169] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
> [    1.020426] TCP: Hash tables configured (established 16384 bind 16384)
> [    1.020431] TCP reno registered
> [    1.020489] NET: Registered protocol family 1
> [    1.020628] platform rtc_cmos: registered platform RTC device (no PNP
> device found)
> [    1.022852] VFS: Disk quotas dquot_6.5.2
> [    1.022929] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> [    1.023370] msgmni has been set to 256
> [    1.023648] alg: No test for stdrng (krng)
> [    1.023659] io scheduler noop registered
> [    1.023661] io scheduler anticipatory registered
> [    1.023664] io scheduler deadline registered
> [    1.023672] io scheduler cfq registered (default)
> [    1.023722] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> [    1.023745] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
> [    1.023895] Linux agpgart interface v0.103
> [    1.024949] brd: module loaded
> [    1.025551] loop: module loaded
> [    1.025696] nbd: registered device at major 43
> [    1.028481] Xen virtual console successfully installed as tty1
> [    1.028525] Event-channel device installed.
> [    1.050852] netfront: Initialising virtual ethernet driver.
> [    1.067106] xen-vbd: registered block device major 8
> [    1.067122] blkfront: sda1: barriers enabled
> [    1.067997] blkfront: sdb1: barriers enabled
> [    1.076451] usbmon: debugfs is not available
> [    1.076454] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
> [    1.076493] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
> [    1.076527] uhci_hcd: USB Universal Host Controller Interface driver
> [    1.076601] Initializing USB Mass Storage driver...
> [    1.076625] usbcore: registered new interface driver usb-storage
> [    1.076629] USB Mass Storage support registered.
> [    1.076653] usbcore: registered new interface driver libusual
> [    1.077546] i8042.c: No controller found.
> [    1.077574] mice: PS/2 mouse device common for all mice
> [    1.077802] TCP cubic registered
> [    1.077804] Initializing XFRM netlink socket
> [    1.077809] NET: Registered protocol family 17
> [    1.077906] XENBUS: Device with no driver: device/console/0
> [    1.152249] kjournald starting.  Commit interval 5 seconds
> [    1.152264] EXT3-fs: mounted filesystem with writeback data mode.
> [    1.152273] VFS: Mounted root (ext3 filesystem) readonly on device 8:1.
> [    1.152351] Freeing unused kernel memory: 220k freed
> INIT: version 2.86 booting
> Activating swap...[   13.319535] Adding 262136k swap on /dev/sdb1.
> Priority:-1 extents:1 across:262136k SS
> done.
> Checking root file system...fsck 1.40-WIP (14-Nov-2006)
> /dev/sda1: clean, 15166/65808 files, 73647/263168 blocks
> done.
> [   14.432238] EXT3 FS on sda1, internal journal
> Setting the system clock..
> Cannot access the Hardware Clock via any known method.
> Use the --debug option to see the details of our search for an access
> method. Cleaning up ifupdown....
> Loading kernel modules...done.
> Loading device-mapper support.
> Checking file systems...fsck 1.40-WIP (14-Nov-2006)
> done.
> Setting kernel variables...done.
> Mounting local filesystems...done.
> Activating swapfile swap...done.
> Setting up networking....
> Configuring network interfaces...done.
> INIT: Entering runlevel: 2
> Starting system logging: syslog-ng.
> Starting nagios-nrpe: nagios-nrpe.
> * Not starting internet superserver: no services enabled.
> Starting OpenBSD Secure Shell server: sshd[   22.389110] NET: Registered
> protocol family 10
> .
> Starting periodic command scheduler: crond.
>
> Debian GNU/Linux 4.0 test6.colo.ba.be tty1
>
> test6.colo.ba.be login: root
> Password:
> Linux test6.colo.ba.be 2.6.31 #15 SMP Thu Oct 22 16:44:31 CEST 2009 x86_64
>
> The programs included with the Debian GNU/Linux system are free software;
> the exact distribution terms for each program are described in the
> individual files in /usr/share/doc/*/copyright.
>
> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
> permitted by applicable law.
>
>
> dom0 kernel .config file
> -------------------------------
>
> sparco01 xen # zcat /proc/config.gz | grep XEN
> CONFIG_X86_64_XEN=y
> # CONFIG_XEN_PCIDEV_FRONTEND is not set
> CONFIG_XEN=y
> CONFIG_XEN_INTERFACE_VERSION=0x00030207
> # XEN
> CONFIG_XEN_PRIVILEGED_GUEST=y
> # CONFIG_XEN_UNPRIVILEGED_GUEST is not set
> CONFIG_XEN_PRIVCMD=y
> CONFIG_XEN_XENBUS_DEV=y
> CONFIG_XEN_BACKEND=y
> CONFIG_XEN_BLKDEV_BACKEND=y
> CONFIG_XEN_BLKDEV_TAP=y
> CONFIG_XEN_BLKDEV_TAP2=y
> CONFIG_XEN_BLKBACK_PAGEMAP=y
> CONFIG_XEN_NETDEV_BACKEND=y
> CONFIG_XEN_NETDEV_TX_SHIFT=8
> # CONFIG_XEN_NETDEV_PIPELINED_TRANSMITTER is not set
> CONFIG_XEN_NETDEV_LOOPBACK=m
> CONFIG_XEN_PCIDEV_BACKEND=y
> CONFIG_XEN_PCIDEV_BACKEND_VPCI=y
> # CONFIG_XEN_PCIDEV_BACKEND_PASS is not set
> # CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set
> # CONFIG_XEN_PCIDEV_BACKEND_CONTROLLER is not set
> # CONFIG_XEN_PCIDEV_BE_DEBUG is not set
> # CONFIG_XEN_TPMDEV_BACKEND is not set
> CONFIG_XEN_SCSI_BACKEND=y
> CONFIG_XEN_USB_BACKEND=y
> # CONFIG_XEN_BLKDEV_FRONTEND is not set
> # CONFIG_XEN_NETDEV_FRONTEND is not set
> # CONFIG_XEN_SCSI_FRONTEND is not set
> # CONFIG_XEN_USB_FRONTEND is not set
> # CONFIG_XEN_GRANT_DEV is not set
> # CONFIG_XEN_FRAMEBUFFER is not set
> CONFIG_XEN_DISABLE_SERIAL=y
> CONFIG_XEN_SYSFS=y
> CONFIG_XEN_NR_GUEST_DEVICES=256
> # CONFIG_XEN_COMPAT_030002_AND_LATER is not set
> # CONFIG_XEN_COMPAT_030004_AND_LATER is not set
> # CONFIG_XEN_COMPAT_030100_AND_LATER is not set
> CONFIG_XEN_COMPAT_030200_AND_LATER=y
> # CONFIG_XEN_COMPAT_030300_AND_LATER is not set
> # CONFIG_XEN_COMPAT_LATEST_ONLY is not set
> CONFIG_XEN_COMPAT=0x030200
> CONFIG_XEN_VCPU_INFO_PLACEMENT=y
> CONFIG_XEN_SMPBOOT=y
> CONFIG_XEN_DEVMEM=y
> CONFIG_XEN_BALLOON=y
> CONFIG_XEN_SCRUB_PAGES=y
>
>
>
> domU kernel .config file:
> --------------------------------
> test6:~# zcat /proc/config.gz | grep XEN
> CONFIG_X86_64_XEN=y
> CONFIG_XEN_PCIDEV_FRONTEND=y
> # CONFIG_XEN_PCIDEV_FE_DEBUG is not set
> CONFIG_XEN=y
> CONFIG_XEN_INTERFACE_VERSION=0x00030207
> # XEN
> # CONFIG_XEN_PRIVILEGED_GUEST is not set
> CONFIG_XEN_UNPRIVILEGED_GUEST=y
> CONFIG_XEN_PRIVCMD=y
> CONFIG_XEN_XENBUS_DEV=y
> # CONFIG_XEN_BACKEND is not set
> CONFIG_XEN_BLKDEV_FRONTEND=y
> CONFIG_XEN_NETDEV_FRONTEND=y
> # CONFIG_XEN_NETDEV_ACCEL_SFC_FRONTEND is not set
> CONFIG_XEN_SCSI_FRONTEND=y
> CONFIG_XEN_USB_FRONTEND=y
> # CONFIG_XEN_USB_FRONTEND_HCD_STATS is not set
> # CONFIG_XEN_GRANT_DEV is not set
> CONFIG_XEN_FRAMEBUFFER=y
> CONFIG_XEN_KEYBOARD=y
> CONFIG_XEN_DISABLE_SERIAL=y
> CONFIG_XEN_SYSFS=y
> CONFIG_XEN_NR_GUEST_DEVICES=256
> # CONFIG_XEN_COMPAT_030002_AND_LATER is not set
> # CONFIG_XEN_COMPAT_030004_AND_LATER is not set
> # CONFIG_XEN_COMPAT_030100_AND_LATER is not set
> # CONFIG_XEN_COMPAT_030200_AND_LATER is not set
> # CONFIG_XEN_COMPAT_030300_AND_LATER is not set
> CONFIG_XEN_COMPAT_LATEST_ONLY=y
> CONFIG_XEN_COMPAT=0xffffff
> CONFIG_XEN_VCPU_INFO_PLACEMENT=y
> CONFIG_XEN_SMPBOOT=y
> CONFIG_XEN_DEVMEM=y
> CONFIG_XEN_BALLOON=y
> CONFIG_XEN_SCRUB_PAGES=y



-- 
BA N.V. - http://www.ba.be

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.