[Xen-users] Kernel dumps when booting domU w/ PCI support - please help.

[Nathan <natemckay@xxxxxxxxx>]

Hi all,

I'm having some trouble booting unprivileged domains w/ support for
PCI devices. What happens is that the UD will boot up until it has
reported that the root filesystem has been mounted, and it has freed
unused kernel memory, but

immediately afterwards I receive a warning that it could not open an
initial console, at which point it appears to print a memory dump:

EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 148k freed
Warning: unable to open an initial console.
Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
c02a5842
*pde = ma 00000000 pa 55555000
 [<c02a5a31>] vc_allocate+0xa1/0x1b0
 [<c02a9028>] con_open+0x48/0xc0
 [<c029873d>] tty_open+0x26d/0x310
 [<c02984d0>] tty_open+0x0/0x310
 [<c016021a>] chrdev_open+0xea/0x1d0
 [<c049a0e0>] legacy_pty_init+0x1c0/0x2d0
 [<c049a0e0>] legacy_pty_init+0x1c0/0x2d0
 [<c0155d2a>] dentry_open+0x15a/0x240
 [<c049a0e0>] legacy_pty_init+0x1c0/0x2d0
 [<c0155bc8>] filp_open+0x68/0x70
 [<c013f4f9>] kmem_cache_alloc+0x69/0x70
 [<c0155e49>] get_unused_fd+0x39/0xe0
 [<c0155fb9>] sys_open+0x49/0x90
 [<c0109678>] syscall_call+0x7/0xb
Oops: 0000 [#1]
PREEMPT
Modules linked in:
CPU:    0
EIP:    0061:[<c02a5842>]    Not tainted VLI
EFLAGS: 00010246   (2.6.10-xenu-2)
EIP is at visual_init+0x72/0x1c0
eax: 00000000   ebx: c064e600   ecx: 00000000   edx: c064e600
esi: 00000000   edi: 00000001   ebp: 00000000   esp: c11e1e94
ds: 007b   es: 007b   ss: 0069
Process init (pid: 1, threadinfo=c11e0000 task=c11cd9e0)
Stack: 00000000 000000d0 c064e600 00000000 c02a5a31 00000000 00000001 00000000
       c0635380 00000000 00000000 c06da000 00400000 c02a9028 00000000 ffffffed
       c06c7c80 00000001 c029873d c06da000 c06c7c80 c11e1ef0 00000102 c06da000
Call Trace:
 [<c02a5a31>] vc_allocate+0xa1/0x1b0
 [<c02a9028>] con_open+0x48/0xc0
 [<c029873d>] tty_open+0x26d/0x310
 [<c02984d0>] tty_open+0x0/0x310
 [<c016021a>] chrdev_open+0xea/0x1d0
 [<c049a0e0>] legacy_pty_init+0x1c0/0x2d0
 [<c049a0e0>] legacy_pty_init+0x1c0/0x2d0
 [<c0155d2a>] dentry_open+0x15a/0x240
 [<c049a0e0>] legacy_pty_init+0x1c0/0x2d0
 [<c0155bc8>] filp_open+0x68/0x70
 [<c013f4f9>] kmem_cache_alloc+0x69/0x70
 [<c0155e49>] get_unused_fd+0x39/0xe0
 [<c0155fb9>] sys_open+0x49/0x90
 [<c0109678>] syscall_call+0x7/0xb
Code: c0 78 4c c0 a1 ac 78 4c c0 89 42 28 8b 14 b5 00 8a 4c c0 85 d2
74 0a 8b 04 b5 c0 78 4c c0 89 50 28 8b 14 b5 c0 78 4c c0 8b 42 28 <8b>
18 85 db 74 35 89 1c 24 e8 e0 b2 e8 ff 85 c0 0f 84 09 01 00
 <0>Kernel panic - not syncing: Attempted to kill init!


I can however, boot the same UD without PCI support without problems.
Basically what I am trying to do is export an IDE controller to a
desktop UD so that I can use that UD to burn DVDs among other things.
I have been sucessfully

running 4 other UDs for quite some time now. Initially I attempted to
take the config from one of the stable UDs (cat /proc/config.gz |
gunzip) and compile a new UD kernel where the only thing that had been
changed was the addition of

PCI bus support but that resulted in the same problem. I have also
tried a number of other things, including booting intrd's, hard disks
attached to the IDE controller in question, etc. I have also tried to
boot the domu with various 'console' and 'extra' options, but just
can't seem to make it work.



Here is the output of xm info:

system                 : Linux
host                   : <edited>
release                : 2.6.10-xen0-2
version                : #1 Tue Oct 25 20:16:14 PDT 2005
machine                : i686
cores                  : 2
hyperthreads_per_core  : 2
cpu_mhz                : 2657
memory                 : 1023
free_memory            : 878


Here is the output of booting the domu without PCI support:

Using config file "/etc/xen/ud-debtop".
Started domain debtop, console on port 9609
************ REMOTE CONSOLE: CTRL-] TO QUIT ********
Linux version 2.6.10-xenu (root@mop) (gcc version 3.3.5 (Debian
1:3.3.5-13)) #1 Thu Jul 7 19:56:52 PDT 2005
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 000000000c800000 (usable)
0MB HIGHMEM available.
200MB LOWMEM available.
DMI not present.
IRQ lockup detection disabled
Built 1 zonelists
Kernel command line:  ip=192.168.1.70:1.2.3.4::::eth0:off root=/dev/hda1 ro
Initializing CPU#0
PID hash table entries: 1024 (order: 10, 16384 bytes)
Xen reported: 2657.880 MHz processor.
Using tsc for high-res timesource
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 199136k/204800k available (2178k kernel code, 5376k reserved,
571k data, 128k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... disabled
NET: Registered protocol family 16
xen_mem: Initialising balloon driver.
Installing knfsd (copyright (C) 1996 okir@xxxxxxxxxxxx).
NTFS driver 2.1.22 [Flags: R/O].
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
elevator: using anticipatory as default io scheduler
Floppy drive(s): fd0 is unknown type 15 (usb?), fd1 is unknown type 15 (usb?)
Failed to obtain physical IRQ 6
floppy0: no floppy controllers found
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
loop: loaded (max 8 devices)
nbd: registered device at major 43
Xen virtual console successfully installed as tty
Event-channel device installed.
xen_blk: Initialising virtual block device driver
xen_net: Initialising virtual ethernet driver.
NET: Registered protocol family 2
IP: routing cache hash table of 2048 buckets, 16Kbytes
TCP: Hash tables configured (established 16384 bind 32768)
ip_conntrack version 2.1 (1600 buckets, 12800 max) - 332 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
NET: Registered protocol family 1
NET: Registered protocol family 17
Bridge firewalling registered
IP-Config: Guessing netmask 255.255.255.0
IP-Config: Complete:
      device=eth0, addr=192.168.1.70, mask=255.255.255.0, gw=255.255.255.255,
     host=192.168.1.70, domain=, nis-domain=(none),
     bootserver=1.2.3.4, rootserver=1.2.3.4, rootpath=
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 128k freed
INIT: version 2.86 booting
Activating swap...
.
Checking root file system.../dev/hda1: clean, 28652/500960 files,
162063/1000038 blocks
.
EXT3 FS on hda1, internal journal
hwclock is unable to get I/O port access:  the iopl(3) call failed.
System time was Thu Oct 27 01:24:04 UTC 2005.
Setting the System Clock using the Hardware Clock as reference...
hwclock is unable to get I/O port access:  the iopl(3) call failed.
System Clock set. System local time is now Thu Oct 27 01:24:04 UTC 2005.
Cleaning up ifupdown...done.
Checking all file systems....
Setting kernel variables ...
... done.
Mounting local filesystems... failed!
Cleaning /tmp....
Cleaning /var/run ....
Cleaning /var/lock ....
Running 0dns-down to make sure resolv.conf is ok...done.
Setting up networking...done.
* /etc/network/options is deprecated.
Setting up IP spoofing protection...done (rp_filter).
Configuring network interfaces...done.

Setting the System Clock using the Hardware Clock as reference...
hwclock is unable to get I/O port access:  the iopl(3) call failed.
System Clock set. Local time: Thu Oct 27 01:24:05 UTC 2005

Initializing random number generator....
Recovering nvi editor sessions... done.
Setting up X server socket directory /tmp/.X11-unix...done.
Setting up ICE socket directory /tmp/.ICE-unix...done.
INIT: Entering runlevel: 2
Starting system log daemon: syslogd.
Starting kernel log daemon: klogd.
Starting MTA: exim4.
Starting internet superserver: inetd.
Starting OpenBSD Secure Shell server: sshd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.

Debian GNU/Linux testing/unstable debtop tty1

debtop login:




And here is the output of booting the same domu WITH PCI support (and
the drivers for my IDE controller et al). As you can see, the root
filesystem, which is on a physical HD, is actually mounted
successfully (note the /lib/tls warning, so obviously it's working to
some degree). The IDE controller itself has been hidden from dom0, and
exported to this domu:





Using config file "/etc/xen/ud-debtop".
Started domain debtop, console on port 9612
************ REMOTE CONSOLE: CTRL-] TO QUIT ********
Linux version 2.6.10-xenu-2 (root@mop) (gcc version 3.3.5 (Debian
1:3.3.5-13)) #1 Tue Oct 25 22:16:40 PDT 2005
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 000000000c800000 (usable)
0MB HIGHMEM available.
200MB LOWMEM available.
DMI not present.
IRQ lockup detection disabled
Built 1 zonelists
Kernel command line:  ip=192.168.1.70:1.2.3.4::::eth0:off root=/dev/hda1 single
Initializing CPU#0
PID hash table entries: 1024 (order: 10, 16384 bytes)
Xen reported: 2657.893 MHz processor.
Using tsc for high-res timesource
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 198192k/204800k available (2635k kernel code, 6324k reserved,
967k data, 148k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... disabled
NET: Registered protocol family 16
PCI: Using configuration type Xen
xen_mem: Initialising balloon driver.
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 06)
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Installing knfsd (copyright (C) 1996 okir@xxxxxxxxxxxx).
NTFS driver 2.1.22 [Flags: R/O].
Initializing Cryptographic API
Real Time Clock Driver v1.12
Non-volatile memory driver v1.2
i8042.c: Can't read CTR while initializing i8042.
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
elevator: using anticipatory as default io scheduler
Floppy drive(s): fd0 is unknown type 15 (usb?), fd1 is unknown type 15 (usb?)
floppy0: no floppy controllers found
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
loop: loaded (max 8 devices)
pktcdvd: v0.2.0a 2004-07-14 Jens Axboe (axboe@xxxxxxx) and petero2@xxxxxxxxx
nbd: registered device at major 43
WARNING: Failed to register Xen virtual console driver as 'tty1'
Event-channel device installed.
xen_blk: Initialising virtual block device driver
Could not probe disks (0)
xen_net: Initialising virtual ethernet driver.
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
SiI680: IDE controller at PCI slot 0000:06:0d.0
PCI: Obtained IRQ 21 for device 0000:06:0d.0
SiI680: chipset revision 2
SiI680: BASE CLOCK == 133
SiI680: 100% native mode on irq 21
    ide0: MMIO-DMA , BIOS settings: hda:DMA, hdb:DMA
    ide1: MMIO-DMA , BIOS settings: hdc:pio, hdd:pio
hda: QUANTUM FIREBALLP LM10.2, ATA DISK drive
hdb: LITE-ON DVDRW SOHW-1633S, ATAPI CD/DVD-ROM drive
ide0 at 0xcd00ec80-0xcd00ec87,0xcd00ec8a on irq 21
hda: max request size: 64KiB
hda: 19925880 sectors (10202 MB) w/1900KiB Cache, CHS=19767/16/63, UDMA(66)
 hda: hda1 hda2
hdb: ATAPI 48X DVD-ROM DVD-R CD-R/RW drive, 2048kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
ohci1394: $Rev: 1223 $ Ben Collins <bcollins@xxxxxxxxxx>
PCI: Obtained IRQ 22 for device 0000:06:0e.0
ohci1394: fw-host0: OHCI-1394 1.0 (PCI): IRQ=[22] 
MMIO=[fe1ff000-fe1ff7ff]  Max Packet=[2048]
eth1394: $Rev: 1224 $ Ben Collins <bcollins@xxxxxxxxxx>
eth1394: eth1: IEEE-1394 IPv4 over 1394 Ethernet (fw-host0)
mice: PS/2 mouse device common for all mice
NET: Registered protocol family 2
IP: routing cache hash table of 2048 buckets, 16Kbytes
TCP: Hash tables configured (established 16384 bind 32768)
ip_conntrack version 2.1 (1600 buckets, 12800 max) - 332 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
NET: Registered protocol family 1
NET: Registered protocol family 17
Bridge firewalling registered
IP-Config: Guessing netmask 255.255.255.0
IP-Config: Complete:
      device=eth0, addr=192.168.1.70, mask=255.255.255.0, gw=255.255.255.255,
     host=192.168.1.70, domain=, nis-domain=(none),
     bootserver=1.2.3.4, rootserver=1.2.3.4, rootpath=
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda1, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem).
Freeing unused kernel memory: 148k freed
Warning: unable to open an initial console.

  ***************************************************************
  ***************************************************************
  ** WARNING: Currently emulating unsupported memory accesses  **
  **          in /lib/tls libraries. The emulation is very     **
  **          slow, and may not work correctly with all        **
  **          programs (e.g., some may 'Segmentation fault').  **
  **          TO ENSURE FULL PERFORMANCE AND CORRECT FUNCTION, **
  **          YOU MUST EXECUTE THE FOLLOWING AS ROOT:          **
  **          mv /lib/tls /lib/tls.disabled                    **
  ***************************************************************
  ***************************************************************

Continuing...

Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
c02a5842
*pde = ma 00000000 pa 55555000
 [<c02a5a31>] vc_allocate+0xa1/0x1b0
 [<c02a9028>] con_open+0x48/0xc0
 [<c029873d>] tty_open+0x26d/0x310
 [<c02984d0>] tty_open+0x0/0x310
 [<c016021a>] chrdev_open+0xea/0x1d0
 [<c049b0e0>] i8042_allocate_kbd_port+0x40/0xd0
 [<c049b0e0>] i8042_allocate_kbd_port+0x40/0xd0
 [<c0155d2a>] dentry_open+0x15a/0x240
 [<c049b0e0>] i8042_allocate_kbd_port+0x40/0xd0
 [<c0155bc8>] filp_open+0x68/0x70
 [<c013f4f9>] kmem_cache_alloc+0x69/0x70
 [<c0155e49>] get_unused_fd+0x39/0xe0
 [<c0155fb9>] sys_open+0x49/0x90
 [<c0109678>] syscall_call+0x7/0xb
Oops: 0000 [#1]
PREEMPT
Modules linked in:
CPU:    0
EIP:    0061:[<c02a5842>]    Not tainted VLI
EFLAGS: 00010246   (2.6.10-xenu-2)
EIP is at visual_init+0x72/0x1c0
eax: 00000000   ebx: c0650600   ecx: 00000000   edx: c0650600
esi: 00000000   edi: 00000001   ebp: 00000000   esp: c0035e94
ds: 007b   es: 007b   ss: 0069
Process init (pid: 1, threadinfo=c0034000 task=c11cd9e0)
Stack: 00000000 000000d0 c0650600 00000000 c02a5a31 00000000 00000001 00000000
       c0637340 00000000 00000000 c0700000 00400000 c02a9028 00000000 ffffffed
       c06c0c80 00000001 c029873d c0700000 c06c0c80 c0035ef0 00000102 c0700000
Call Trace:
 [<c02a5a31>] vc_allocate+0xa1/0x1b0
 [<c02a9028>] con_open+0x48/0xc0
 [<c029873d>] tty_open+0x26d/0x310
 [<c02984d0>] tty_open+0x0/0x310
 [<c016021a>] chrdev_open+0xea/0x1d0
 [<c049b0e0>] i8042_allocate_kbd_port+0x40/0xd0
 [<c049b0e0>] i8042_allocate_kbd_port+0x40/0xd0
 [<c0155d2a>] dentry_open+0x15a/0x240
 [<c049b0e0>] i8042_allocate_kbd_port+0x40/0xd0
 [<c0155bc8>] filp_open+0x68/0x70
 [<c013f4f9>] kmem_cache_alloc+0x69/0x70
 [<c0155e49>] get_unused_fd+0x39/0xe0
 [<c0155fb9>] sys_open+0x49/0x90
 [<c0109678>] syscall_call+0x7/0xb
Code: c0 78 4c c0 a1 ac 78 4c c0 89 42 28 8b 14 b5 00 8a 4c c0 85 d2
74 0a 8b 04 b5 c0 78 4c c0 89 50 28 8b 14 b5 c0 78 4c c0 8b 42 28 <8b>
18 85 db 74 35 89 1c 24 e8 e0 b2 e8 ff 85 c0 0f 84 09 01 00
 <0>Kernel panic - not syncing: Attempted to kill init!
 <0>Rebooting in 1 seconds..
************ REMOTE CONSOLE EXITED *****************



Don't know if this is usefull at all, but here's some system info:

Dell Precision 450
Dual Xeon 2.66 processors (HT enabled)
1 Gig PC-2100

Here is the output of lspci:

0000:00:00.0 Host bridge: Intel Corp. E7505 Memory Controller Hub (rev 03)
0000:00:1d.0 USB Controller: Intel Corp. 82801DB/DBL/DBM
(ICH4/ICH4-L/ICH4-M) USB UHCI Controller #1 (rev 01)
0000:00:1d.1 USB Controller: Intel Corp. 82801DB/DBL/DBM
(ICH4/ICH4-L/ICH4-M) USB UHCI Controller #2 (rev 01)
0000:00:1d.2 USB Controller: Intel Corp. 82801DB/DBL/DBM
(ICH4/ICH4-L/ICH4-M) USB UHCI Controller #3 (rev 01)
0000:00:1d.7 USB Controller: Intel Corp. 82801DB/DBM (ICH4/ICH4-M) USB
2.0 EHCI Controller (rev 01)
0000:00:1f.0 ISA bridge: Intel Corp. 82801DB/DBL (ICH4/ICH4-L) LPC
Bridge (rev 01)
0000:00:1f.1 IDE interface: Intel Corp. 82801DB/DBL (ICH4/ICH4-L)
UltraATA-100 IDE Controller (rev 01)
0000:00:1f.3 SMBus: Intel Corp. 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M)
SMBus Controller (rev 01)
0000:00:1f.5 Multimedia audio controller: Intel Corp. 82801DB/DBL/DBM
(ICH4/ICH4-L/ICH4-M) AC'97 Audio Controller (rev 01)
0000:01:00.0 VGA compatible controller: nVidia Corporation NV34GL
[Quadro FX 500] (rev a1)
0000:02:1c.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 04)
0000:02:1e.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 04)
0000:03:0e.0 Ethernet controller: Intel Corp. 82545EM Gigabit Ethernet
Controller (Copper) (rev 01)
0000:04:04.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro
100] (rev 05)
0000:04:05.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro
100] (rev 05)


I've really looked everywhere I can think of for a solution to this
problem (including this mailing list, my apologies if I missed a
posting that answers this), and haven't been able to make any
progress. If anyone can help I would really appreciate it. If there is
any other info that would be useful diagnosing this problem, I will
gladly make it available.

Thanks,
Nathan

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users