Re: [Xen-devel] Unable to create guest domain AND mm.c/grant_table.c errors

[Chris Bainbridge <chris.bainbridge@xxxxxxxxx>]

Hi,

I'm been struggling with this for a couple of days.. I'd assumed that
there was just something wrong with my setup rather than a bug. I've
seen the same PTE error. In trying to eliminate possible problems I've
started using the nightly binary xen builds. So I'm using the binary
precompiled hypervisor, dom0 and domU kernels.

The really weird thing is that I have another machine with the same
Xen build, same xen and kernel image binaries, and it works. I pulled
the hard disk and put it in the bad machine, and it worked. So it
appears there's something wrong with that particular domain 0 root fs
that is causing it to fail, rather than anything wrong with the Xen
binaries or hardware.

I've tried rm -rf /var/lib/xenstored but no luck - does Xen store
state anywhere else? Is there anything else in the domain 0 root fs
that could be causing this error? In fact I did find something, that
if we try to create a device (eg. /dev/hda1) in domU then it has to
exist in dom0, otherwise xend can't find a major/minor number. That
can be tricky with udev if you don't have a tarball of standard
devices, maybe it can be fixed?

The only other thing I could think of was that I built this root fs
with '-mno-tls-direct-seg-refs' in CFLAGS, and glibc with nptlonly (it
has no linuxthreads). AFAIK, this set up should work fine with Xen.
Can anyone confirm that?

domU:

Event-channel device installed.
xen_blk: Initialising virtual block device driver
xenbus: failed to write error node for device/vbd/2049 (22 writing ring-ref)
vbd: probe of vbd-2049 failed with error -22
xen_blk: Timeout connecting to device!
xen_net: Initialising virtual ethernet driver.
xenbus: failed to write error node for device/vif/1 (22 writing tx ring-ref)
vif: probe of vif-1 failed with error -22
xen_net: Timeout connecting to device!

xen console:
smp_send_stop disable_local_APIC
<Sometimes the PTE error appears after this smp msg>
(XEN) DOM0: (file=mm.c, line=2458) Could not find PTE entry for address d0d4c000
(XEN) DOM0: (file=mm.c, line=2458) Could not find PTE entry for address d0d4d000
(XEN) (file=grant_table.c, line=1054) Grant unref rd(1) ld(0) frm(eef9) flgs(0).
(XEN) (file=grant_table.c, line=1054) Grant unref rd(1) ld(0) frm(eef8) flgs(0).

xend-debug.log:
ERROR: do_evtchn_op: HYPERVISOR_event_channel_op failed: -1
ERROR: do_evtchn_op: HYPERVISOR_event_channel_op failed: -1

# xend.log
2005-09-21 11:36:44 xend] DEBUG (image:164) initDomain: cpu=-1
mem_kb=98304 ssidref=0 dom=1
[2005-09-21 11:36:44 xend] DEBUG (XendDomainInfo:702) init_domain>
Created domain=1 name=ubuntu memory=96
[2005-09-21 11:36:44 xend] INFO (image:202) buildDomain os=linux dom=1 vcpus=1
[2005-09-21 11:36:44 xend] DEBUG (image:246) dom            = 1
[2005-09-21 11:36:44 xend] DEBUG (image:247) image          =
/boot/vmlinuz-2.6-xenU
[2005-09-21 11:36:44 xend] DEBUG (image:248) store_evtchn   = 1
[2005-09-21 11:36:44 xend] DEBUG (image:249) console_evtchn = 2
[2005-09-21 11:36:44 xend] DEBUG (image:250) cmdline        =  root=/dev/sda1
[2005-09-21 11:36:44 xend] DEBUG (image:251) ramdisk        =
[2005-09-21 11:36:44 xend] DEBUG (image:252) flags          = 0
[2005-09-21 11:36:44 xend] DEBUG (image:253) vcpus          = 1
[2005-09-21 11:36:44 xend] DEBUG (DevController:181) DevController:
writing {'virtual-device': '2049', 'backend-id': '0', 'backend':
'/domain/0e9d1b5f-ae27-40c0-8f78-b1c1039815dc/backend/vbd/64dd317d-dbcf-4e\
9e-be5a-b392e83da1fa/2049'} to
/domain/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/device/vbd/2049.
[2005-09-21 11:36:44 xend] DEBUG (DevController:183) DevController:
writing {'params': '/var/tmp/ubuntu.img', 'domain': 'ubuntu', 'type':
'file', 'frontend': '/domain/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/dev\
ice/vbd/2049', 'frontend-id': '1'} to
/domain/0e9d1b5f-ae27-40c0-8f78-b1c1039815dc/backend/vbd/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/2049.
[2005-09-21 11:36:44 xend] DEBUG (DevController:181) DevController:
writing {'backend-id': '0', 'mac': 'aa:00:00:00:00:11', 'handle': '1',
'backend': '/domain/0e9d1b5f-ae27-40c0-8f78-b1c1039815dc/backend/vif\
/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/1'} to
/domain/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/device/vif/1.
[2005-09-21 11:36:44 xend] DEBUG (DevController:183) DevController:
writing {'bridge': 'xen-br0', 'mac': 'aa:00:00:00:00:11', 'handle':
'1', 'script': '/etc/xen/scripts/vif-bridge', 'frontend-id': '1',
'doma\
in': 'ubuntu', 'frontend':
'/domain/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/device/vif/1'} to
/domain/0e9d1b5f-ae27-40c0-8f78-b1c1039815dc/backend/vif/64dd317d-dbcf-4e9e-be5a-b392e83da1fa/1.
[2005-09-21 11:36:44 xend] INFO (XendRoot:141) EVENT>
xend.domain.create ['ubuntu', 1]
[2005-09-21 11:36:44 xend] INFO (XendRoot:141) EVENT>
xend.domain.unpause ['ubuntu', 1]
[2005-09-21 11:37:05 xend] DEBUG (XendDomain:232) domain died
name=ubuntu domid=1
[2005-09-21 11:37:05 xend] INFO (XendRoot:141) EVENT> xend.domain.died
['ubuntu', None]

And I once saw this bug, could be related:

xen_blk: Initialising virtual block device driver
xen_blk: Timeout connecting to device!
xen_net: Initialising virtual ethernet driver.
XENBUS ack device/vif/1/tx-ring-ref fail -5
kernel BUG at drivers/xen/xenbus/xenbus_xs.c:151 (xs_talkv)!
 [<c0244c3d>] xs_talkv+0x15d/0x170
 [<c0245880>] watch_thread+0x0/0x180
 [<c0244c95>] xs_single+0x45/0x50
 [<c0244e43>] xenbus_directory+0x33/0x120
 [<c0245880>] watch_thread+0x0/0x180
 [<c0244f4f>] xenbus_exists+0x1f/0x40
 [<c0246711>] dev_changed+0x51/0x140
 [<c0245880>] watch_thread+0x0/0x180
 [<c0244c95>] xs_single+0x45/0x50
 [<c0245880>] watch_thread+0x0/0x180
 [<c024565e>] find_watch+0x1e/0x60
 [<c0245988>] watch_thread+0x108/0x180
 [<c0136110>] autoremove_wake_function+0x0/0x60
 [<c0136110>] autoremove_wake_function+0x0/0x60
 [<c0135c1a>] kthread+0xba/0xc0
 [<c0135b60>] kthread+0x0/0xc0
 [<c0107395>] kernel_thread_helper+0x5/0x10
Kernel panic - not syncing: BUG!


Chris


On 21/09/05, Bonnell, David <David_Bonnell@xxxxxxx> wrote:
>  
>  
> 
> Since a few changesets back (currently using 6963) I am unable to create a
> guest domain.  It appears the front-end VBD cannot connect to the backend
> and as such the guest cannot mount its root filesystem and panics. 
> 
>   
> 
> The xen console log and xend logs are attached.  Below is the bottom part of
> the guest's console output 
> 
>   
> 
> … 
> 
> Event-channel device ubstakked, 
> 
> xen_blk: Initialising virtual block device driver 
> 
> *** A FEW SECONDS DELAY *** 
> 
> xen_blk: Timeout connecting to device! 
> 
> xen_net: Initialising virtual Ethernet driver. 
> 
> NET: Registered protocol family 2 
> 
> … 
> 
> Root-NFS: No NFS server available, giving up. 
> 
> VFS: Unable to mount root fs via NFS, trying floppy. 
> 
> VFS: Cannot open root device "sda1" or unknown-block(2,0) 
> 
> Please append a correct "root=" boot option 
> 
> Kernel panic – not syncing: VFS: Unable to mount root fs on
> unknown-block(2,0) 
> 
>   
> 
> The "Unable to read memory/target" console message at the end of the Xen
> console log is new for this changeset.  Also, every now and then a failed
> attempt to create a guest domain ends with the following sequence of errors
> on the Xen console and after such a failure the guest domain is not
> automatically destroyed as it is normally and needs a manual "xm destroy
> ???" command to remove it. 
> 
>   
> 
> (XEN) DOM0: (file=mm.c, line=2458) Could not find PTE entry for address
> c8cec000 
> 
> (XEN) DOM0: (file=mm.c, line=2458) Could not find PTE entry for address
> c8ced000 
> 
> (XEN) (file=grant_table.c, line=1054) Grant unref rd(7) ld(0) frm(1ff51)
> flgs(0). 
> 
> (XEN) (file=grant_table.c, line=1054) Grant unref rd(7) ld(0) frm(1ff50)
> flgs(0). 
> 
>   
> 
>   
> 
> -Dave 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel
> 
> 
> 
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel