[Xen-users] BLKTAP limit in Dom0, can't boot with pygrub

[Jim Cipar <jcipar@xxxxxxx>]

I'm trying to start a large number of domains (hopefully around 70) each 
backed by a separate qcow disk.  They are booting via pygrub.  I've 
included an example config file below.  I can start the first 16 DomUs 
just fine, but any after that fail to start.  It looks like I'm hitting 
a limit to the number of blktap devices the Dom0 can have, so pygrub is 
failing to load the new disks.  I've posted relevant sections of 
/var/log/xen/xend.log for both a successful and failed domain creation.
I'm nearly certain that the problem is with mounting the image in Dom0 
so that pygrub can find a kernel on it; if I modify the config file for 
a failed domain to boot directly from Dom0's kernel, and then retry the 
domain creation, it works.  Furthermore, if I do an "xm shutdown" on one 
of the successfully created pygrub domains, I can start one of the 
failed ones.
Could it be that there is a limit of 16 blktap devices in Dom0 and 
they're not being cleaned up properly after pygrub uses them?  It looks 
like they are.  In the log file, after pygrub returns it says 
"Unmounting /dev/xvdp from /dev/xvdp".
Thanks,
Jim Cipar





LOG FOR FAILED DOMAIN CREATION
---------------------------------------------------------------
[2008-08-30 10:06:11 5237] INFO (XendDomainInfo:2066) Mounting 
/home/jcipar/Disks/VM-17-0.qcow on /dev/xvdp.
[2008-08-30 10:06:11 5237] DEBUG (DevController:117) DevController: 
writing {'backend-id': '0', 'virtual-device': '51952', 'device-type': 
'disk', 'state': '1', 'backend': '/loca
l/domain/0/backend/tap/0/51952'} to /local/domain/0/device/vbd/51952.
[2008-08-30 10:06:11 5237] DEBUG (DevController:119) DevController: 
writing {'domain': 'Domain-0', 'frontend': 
'/local/domain/0/device/vbd/51952', 'uuid': '714fe8ee-0376-7cad-ab
0a-a3a8085166b6', 'dev': '/dev/xvdp', 'state': '1', 'params': 
'qcow:/home/jcipar/Disks/VM-17-0.qcow', 'mode': 'w', 'online': '1', 
'frontend-id': '0', 'type': 'tap'} to /local/domain/0/backend/tap/0/51952.
[2008-08-30 10:06:11 5237] DEBUG (DevController:155) Waiting for 51952.
[2008-08-30 10:06:11 5237] DEBUG (DevController:594) 
hotplugStatusCallback /local/domain/0/backend/tap/0/51952/hotplug-status.
[2008-08-30 10:06:12 5237] DEBUG (DevController:594) 
hotplugStatusCallback /local/domain/0/backend/tap/0/51952/hotplug-status.
[2008-08-30 10:06:12 5237] DEBUG (DevController:608) 
hotplugStatusCallback 1.
[2008-08-30 10:06:12 5237] DEBUG (DevController:155) Waiting for 51952.
[2008-08-30 10:06:12 5237] DEBUG (DevController:594) 
hotplugStatusCallback /local/domain/0/backend/tap/0/51952/hotplug-status.
[2008-08-30 10:06:12 5237] DEBUG (DevController:608) 
hotplugStatusCallback 1.
[2008-08-30 10:06:12 5237] ERROR (XendBootloader:43) Disk isn't accessible
[2008-08-30 10:06:12 5237] INFO (XendDomainInfo:2084) Unmounting 
/dev/xvdp from /dev/xvdp.
[2008-08-30 10:06:12 5237] DEBUG (XendDomainInfo:590) 
XendDomainInfo.destroyDevice: deviceClass = tap, device = /dev/xvdp
[2008-08-30 10:06:12 5237] ERROR (XendDomainInfo:440) VM start failed



LOG FOR SUCCESSFUL DOMAIN CREATION
-------------------------------------------------------------------------
[2008-08-30 10:06:06 5237] DEBUG (DevController:119) DevController: 
writing {'domain': 'Domain-0', 'frontend': 
'/local/domain/0/device/vbd/51952', 'uuid': '3fbc58db-daa3-b555-21
4e-14745062eb03', 'dev': '/dev/xvdp', 'state': '1', 'params': 
'qcow:/home/jcipar/Disks/VM-16-0.qcow', 'mode': 'w', 'online': '1', 
'frontend-id': '0', '
type': 'tap'} to /local/domain/0/backend/tap/0/51952.
[2008-08-30 10:06:06 5237] DEBUG (DevController:155) Waiting for 51952.
[2008-08-30 10:06:06 5237] DEBUG (DevController:594) 
hotplugStatusCallback /local/domain/0/backend/tap/0/51952/hotplug-status.
[2008-08-30 10:06:06 5237] DEBUG (DevController:594) 
hotplugStatusCallback /local/domain/0/backend/tap/0/51952/hotplug-status.
[2008-08-30 10:06:06 5237] DEBUG (DevController:608) 
hotplugStatusCallback 1.
[2008-08-30 10:06:06 5237] DEBUG (DevController:155) Waiting for 51952.
[2008-08-30 10:06:06 5237] DEBUG (DevController:594) 
hotplugStatusCallback /local/domain/0/backend/tap/0/51952/hotplug-status.
[2008-08-30 10:06:06 5237] DEBUG (DevController:608) 
hotplugStatusCallback 1.
[2008-08-30 10:06:06 12692] DEBUG (XendBootloader:100) Launching 
bootloader as ['/usr/bin/pygrub', '--args=root=/dev/xvda1 xencons=tty', 
'--output=/var/run/xend/boot/xenbl.15861
', '-q', '/dev/xvdp'].
[2008-08-30 10:06:07 5237] INFO (XendDomainInfo:2084) Unmounting 
/dev/xvdp from /dev/xvdp.
[2008-08-30 10:06:07 5237] DEBUG (XendDomainInfo:590) 
XendDomainInfo.destroyDevice: deviceClass = tap, device = /dev/xvdp


EXAMPLE CONFIG FILE
------------------------------------------
bootloader="/usr/bin/pygrub"
disk=['tap:qcow:/home/jcipar/Disks/VM-1-0.qcow,xvda1,w']
vif = ['ip=172.19.158.1']
memory=100
root="/dev/xvda1"
extra='xencons=tty'


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users