Re: [Xen-users] RFH: loopback & blktap(2) and CDROM

[Philipp Hahn <hahn@xxxxxxxxxxxxx>]

Hello Ian,

thank you for your excellent answer.

On Friday 09 November 2012 16:49:32 Ian Campbell wrote:
> > 4. I read somewhere that "mixing loobback with tapdisk is a bad idea",
> > but I can't find that again, so I'm wondering if that (wrong) claim got
> > somehow stuck in my memory. I can only imagin two scenarios
> > a) one domU accessing two disk images, one with lookback, the other one
> > with blktap. That looks okay.
> > b) two domUs accessing the same disk image, one with lookback, the other
> > one with blktap. That looks broken but neither would I expect that to
> > work (shared disk semantics aside)
>
> loopback is dangerous because it reports success before the data has
> really hit the underlying device.
...
> Probably ok for r/o devices
> though and r/w shared disks need care for plenty of other reasons!

Okay, my current problem is getting the CDROM case right.

> > 5. While experimenting with a Linux domU I noticed that the boot process
> > sometimes gets stuck when I declare one disk as "hda" and a second one
> > as "xvda". The 2.6.32 kernel detects a clash in /sys/block naming and I'm
> > stuck in the "XENBUS: Waiting for devices to initialise: 295s..." count
> > down. Is this because hda and xvda overlap because of the PVonHVM case
> > (ide-block-major has 64 minors per device for partitions, while
> > scsi-block-major and xen-block-major only have 16 minors per device
>
> When you ask for hda you actually get hda+xvda in order to allow for the
> switchover described above. So you've actually asked for 2 xvda's --
> don't do that ;-)
>
> > , that is
> > hda=xvd[abcd], hdb=xvd[efgh], ...)?
>
> Almost. hda=xvda, hda[1234...]=xvda[1234...], hdb=xvdb ... hdd=xvdd.
>
> There is no hde so xvde is a good starting point for pure PV disks if
> you are mixing ide and pure PV.

With Linux 3.2.30 as domU I get the same as you, but with 2.6.32 I get a 
different: domU is configured with hda=hdb=disk, hdc=cdrom, but inside the 
domU I get /dev/xdva and /dev/xvde for the disk, and /dev/scd0 for the cdrom. 
With "xen_emul_unplug=never" I get hda* and hdb*.

For testing I created 60 partitions (3 primary+57 extended) on hdb, but as 
most SCSI, SATA and XEN majors only support 16 minors per device, I only see 
the first 15 partitions on /dev/xvde{,1..15}. With "..unplug=never" I see 
them all, but 16..60 are provided by block-major 259 (blkext).


There also seems to be a problem when I swap hdb and hdc: If hdb=cdrom is 
sandwiched between hda=hdc=disk, the Linux-2.6.32 kernel waits in 
the "XENBUS: Waiting for devices to initialise: 295s..." count down after an 
OOPs, because /dev/block/202:0 couldn't be double-created.
Its fine with 3.2.30.

# dmesg | grep -i xen
XENBUS: Device with no driver: device/vbd/768
XENBUS: Device with no driver: device/vbd/75632
XENBUS: Device with no driver: device/vbd/832
...
blkfront: xvda: barriers enabled
 xvda:
vbd vbd-832: 19 xenbus_dev_probe on device/vbd/832
blkfront: xvda: barriers enabled
WARNING: at ... fs/sysfs/dir.c:491 sysfs_add_one+0xcc/0xe4()
sysfs: cannot create duplicate filename '/dev/block/202:0'
Pid 12, comm: xenwatch Not Tainted 2.6.32-ucs57amd64 #1
...
 ? sysfs_add_one+0xcc/0xe4
...
 ? backend_changed+0x44e/0x468 [xen_blkfront]
 ? xenwatch_thread+0x117/0x14a
...
# xenstore-ls | egrep 'tap2|vbd'
/vm/77021a09-cbcb-35d6-d480-76ff7c2cf288/device/tap2/768/frontend 
= "/local/domain/5/device/vbd/768"
/vm/77021a09-cbcb-35d6-d480-76ff7c2cf288/device/tap2/768/frontend-id = "5"
/vm/77021a09-cbcb-35d6-d480-76ff7c2cf288/device/tap2/768/backend-id = "0"
/vm/77021a09-cbcb-35d6-d480-76ff7c2cf288/device/tap2/768/backend 
= "/local/domain/0/backend/vbd/5/768"
...
/vm/77021a09-cbcb-35d6-d480-76ff7c2cf288/device/tap2/5632/frontend 
= "/local/domain/5/device/vbd/5632"
/vm/77021a09-cbcb-35d6-d480-76ff7c2cf288/device/tap2/5632/frontend-id = "5"
/vm/77021a09-cbcb-35d6-d480-76ff7c2cf288/device/tap2/5632/backend-id = "0"
/vm/77021a09-cbcb-35d6-d480-76ff7c2cf288/device/tap2/5632/backend 
= "/local/domain/0/backend/vbd/5/5632"
...
/vm/77021a09-cbcb-35d6-d480-76ff7c2cf288/device/vbd/832/frontend 
= "/local/domain/5/device/vbd/832"
/vm/77021a09-cbcb-35d6-d480-76ff7c2cf288/device/vbd/832/frontend-id = "5"
/vm/77021a09-cbcb-35d6-d480-76ff7c2cf288/device/vbd/832/backend-id = "0"
/vm/77021a09-cbcb-35d6-d480-76ff7c2cf288/device/vbd/832/backend 
= "/local/domain/0/backend/vbd/5/832"
...
/local/domain/0/backend/vbd/5/768/domain = "ucs-hv-hd"
/local/domain/0/backend/vbd/5/768/frontend = "/local/domain/5/device/vbd/768"
/local/domain/0/backend/vbd/5/768/uuid 
= "22c313da-e8bd-5f63-6246-69adbf18408c"
/local/domain/0/backend/vbd/5/768/bootable = "1"
/local/domain/0/backend/vbd/5/768/dev = "hda"
/local/domain/0/backend/vbd/5/768/state = "4"
/local/domain/0/backend/vbd/5/768/params = "/dev/xen/blktap-2/tapdev0"
/local/domain/0/backend/vbd/5/768/mode = "w"
/local/domain/0/backend/vbd/5/768/online = "1"
/local/domain/0/backend/vbd/5/768/frontend-id = "5"
/local/domain/0/backend/vbd/5/768/type = "phy"
/local/domain/0/backend/vbd/5/768/tapdisk-params 
= "aio:/var/lib/libvirt/images/ucs-hv-hd.raw"
/local/domain/0/backend/vbd/5/768/physical-device = "fd:0"
/local/domain/0/backend/vbd/5/768/hotplug-status = "connected"
/local/domain/0/backend/vbd/5/768/feature-barrier = "1"
/local/domain/0/backend/vbd/5/768/sectors = "41943040"
/local/domain/0/backend/vbd/5/768/info = "0"
/local/domain/0/backend/vbd/5/768/sector-size = "512"
...
/local/domain/0/backend/vbd/5/5632/domain = "ucs-hv-hd"
/local/domain/0/backend/vbd/5/5632/frontend 
= "/local/domain/5/device/vbd/5632"
/local/domain/0/backend/vbd/5/5632/uuid 
= "62456f87-d3c0-1cec-a15f-dfe2cc5ea22d"
/local/domain/0/backend/vbd/5/5632/bootable = "0"
/local/domain/0/backend/vbd/5/5632/dev = "hdc"
/local/domain/0/backend/vbd/5/5632/state = "4"
/local/domain/0/backend/vbd/5/5632/params = "/dev/xen/blktap-2/tapdev1"
/local/domain/0/backend/vbd/5/5632/mode = "w"
/local/domain/0/backend/vbd/5/5632/online = "1"
/local/domain/0/backend/vbd/5/5632/frontend-id = "5"
/local/domain/0/backend/vbd/5/5632/type = "phy"
/local/domain/0/backend/vbd/5/5632/tapdisk-params 
= "aio:/var/lib/libvirt/images/ucs-hv-hd2.raw"
/local/domain/0/backend/vbd/5/5632/physical-device = "fd:1"
/local/domain/0/backend/vbd/5/5632/hotplug-status = "connected"
/local/domain/0/backend/vbd/5/5632/feature-barrier = "1"
/local/domain/0/backend/vbd/5/5632/sectors = "41943040"
/local/domain/0/backend/vbd/5/5632/info = "0"
/local/domain/0/backend/vbd/5/5632/sector-size = "512"
...
/local/domain/0/backend/vbd/5/832/domain = "ucs-hv-hd"
/local/domain/0/backend/vbd/5/832/frontend = "/local/domain/5/device/vbd/832"
/local/domain/0/backend/vbd/5/832/uuid 
= "eccf7fb5-7182-fb1b-62af-bd28f7620573"
/local/domain/0/backend/vbd/5/832/bootable = "0"
/local/domain/0/backend/vbd/5/832/dev = "hdb"
/local/domain/0/backend/vbd/5/832/state = "6"
/local/domain/0/backend/vbd/5/832/params 
= "/var/lib/libvirt/images/ucs_2.4-0-sec7-20120131133624-dvd-amd64.iso"
/local/domain/0/backend/vbd/5/832/mode = "r"
/local/domain/0/backend/vbd/5/832/online = "1"
/local/domain/0/backend/vbd/5/832/frontend-id = "5"
/local/domain/0/backend/vbd/5/832/type = "file"
/local/domain/0/backend/vbd/5/832/node = "/dev/loop0"
/local/domain/0/backend/vbd/5/832/physical-device = "7:0"
/local/domain/0/backend/vbd/5/832/hotplug-status = "connected"
...
/local/domain/5/device/vbd/768/backend-id = "0"
/local/domain/5/device/vbd/768/virtual-device = "768"
/local/domain/5/device/vbd/768/device-type = "disk"
/local/domain/5/device/vbd/768/state = "4"
/local/domain/5/device/vbd/768/backend = "/local/domain/0/backend/vbd/5/768"
/local/domain/5/device/vbd/768/ring-ref = "8"
/local/domain/5/device/vbd/768/event-channel = "4"
/local/domain/5/device/vbd/768/protocol = "x86_64-abi"
...
/local/domain/5/device/vbd/5632/backend-id = "0"
/local/domain/5/device/vbd/5632/virtual-device = "5632"
/local/domain/5/device/vbd/5632/device-type = "disk"
/local/domain/5/device/vbd/5632/state = "4"
/local/domain/5/device/vbd/5632/backend = "/local/domain/0/backend/vbd/5/5632"
/local/domain/5/device/vbd/5632/ring-ref = "9"
/local/domain/5/device/vbd/5632/event-channel = "5"
/local/domain/5/device/vbd/5632/protocol = "x86_64-abi"
...
/local/domain/5/device/vbd/832/backend-id = "0"
/local/domain/5/device/vbd/832/virtual-device = "832"
/local/domain/5/device/vbd/832/device-type = "cdrom"
/local/domain/5/device/vbd/832/state = "6"
/local/domain/5/device/vbd/832/backend = "/local/domain/0/backend/vbd/5/832"
...
/local/domain/5/error/device/vbd/832/error = "19 xenbus_dev_probe on 
device/vbd/832"


> > 6. How should I make an .iso image file accassable to a domU?
> > If a use tap:/var/lib/libvirt/images/some.iso tapdisk2 claims the image
> > and passes phy:/dev/xen/blktapX to qemu-dm, which I can access fine, but
> > eject does not work, since qemu only sees the phy: device and can't open
> > another file.
> > xen-blockfront in PVonHVM and Windows-GPLPV driver both reject
> > CDROM-devices, so the CDROM remains IDE emulated.
>
> That's right -- PV drivers aren't used for HVM CD-ROM devices so that
> media change etc can be supported. I think in this case you want to use
> file:// and let qemu open the device direct. There will be no PV path in
> this case, so no need for tap etc.

Can I tell Xend to not create a lookback device for each such ISO image using 
file://? I had hoped for some "ioemu:" flag to get only the ioemu emulated 
device as it is (or was?) the case with network interfaces, but for disk 
devices the "ioemu:" seems to be stripped just everywhere without effect.

Currently all my domUs use the same ISO image, but each domU gets its own 
lookback device, so I have to set "max_loop=256" when loading the loop module 
to have enought lookback devices available. (not doing the loopback stuff 
would be the easiest solution, since xen-blockfront doesn't use it anyway).

Sincerely
Philipp

PS: Mostly tested on Xen-3.4.3 with libvirt-0.8.7 and linux-2.6.32, but mostly 
the same with Xen-4.1.3 with libvirt-0.9.12 and linux-3.2.30.
-- 
Philipp Hahn           Open Source Software Engineer      hahn@xxxxxxxxxxxxx
Univention GmbH        be open.                       fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen                 fax: +49 421 22 232-99
                                                   http://www.univention.de/

Attachment: signature.asc
Description: This is a digitally signed message part.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users