|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-users] PV DomU can't access disk from storage driver domain
On Fri, Sep 18, 2015 at 3:23 AM, Roger Pau Monnà <roger.pau@xxxxxxxxxx> wrote:
> El 17/09/15 a les 23.08, Alex Velazquez ha escrit:
>> Hi all,
>>
>> I've been trying to set up a storage driver domain that can provide a
>> disk to another DomU, but am having trouble getting it to work. If
>> anyone has experience with this and could provide some suggestions,
>> I'd be grateful.
>>
>> I'm running Xen 4.6.0-rc3. Both the storage driver domain
>> ("storagedd") and the other DomU ("client") are PV guests running
>> Ubuntu 14.04.3 (kernel 3.13.0-63-generic).
>>
>> To set up the storage domain, I followed instructions from the
>> "Storage driver domains" page on the wiki
>> (http://wiki.xenproject.org/wiki/Storage_driver_domains) and got some
>> additional ideas from "XCP storage driver domains"
>> (http://wiki.xenproject.org/wiki/XCP_storage_driver_domains). I'll
>> summarize my steps here:
>>
>> 1) clone xen.git repository (using the 4.6.0-rc3 release, same as Dom0)
>> 2) make tools & make install tools
>> 3) apt-get install blktap-utils
>> 4) mount -t xenfs xenfs /proc/xen
>> 5) modprobe xen-blkback & modprobe xen-gntalloc & modprobe xen-gntdev
>> (not sure if any/all of these are necessary, or if they're handled
>> automatically)
>>
>> My testing so far has been without PCI passthrough. I've tried with
>> files located on an NFS share mounted by storagedd (qcow2 and raw),
>> files located on storagedd's filesystem (again, qcow2 and raw), as
>> well a block device (loop device created using losetup).
>>
>> Here is the xl config file for "storagedd":
>>
>> xenuser@xenhost:~$ cat storagedd.cfg
>> name = "storagedd"
>> builder = "generic"
>> kernel = "/usr/local/lib/xen/boot/pv-grub-x86_64.gz"
>> extra = "(hd0,0)/boot/grub/menu.lst"
>> driver_domain = 1
>> vcpus = 1
>> memory = 2048
>> disk = [
>> "format=qcow2,vdev=xvda,access=rw,target=/var/lib/xen/images/storagedd.qcow2"
>> ]
>> vif = [ "mac=00:16:3e:36:00:01,bridge=xenbr0,script=vif-openvswitch" ]
>>
>> And here is the xl config file for "client":
>>
>> xenuser@xenhost:~$ cat client.cfg
>> name = "client"
>> builder = "generic"
>> kernel = "/usr/local/lib/xen/boot/pv-grub-x86_64.gz"
>> extra = "(hd0,0)/boot/grub/menu.lst"
>> vcpus = 1
>> memory = 1024
>> disk = [
>> "format=raw,backendtype=phy,backend=storagedd,vdev=xvda,target=/dev/loop0"
>> ]
>> vif = [ "mac=00:16:3e:37:00:02,bridge=xenbr0,script=vif-openvswitch" ]
>>
>> storagedd starts up and runs normally. Here are the Xen kernel modules
>> that are loaded:
>>
>> admin@storagedd:~$ sudo lsmod | grep xen
>> xen_gntalloc 13626 0
>> xen_gntdev 18675 0
>> xen_blkback 37209 0
>> xenfs 12978 1
>> xen_privcmd 13243 1 xenfs
>>
>> Initially, there are no backend entries in xenstore:
>>
>> admin@storagedd:~$ sudo xenstore-ls /local/domain/1/backend
>> xenstore-ls: xs_directory (/local/domain/1/backend): No such file or
>> directory
>>
>> Then I start the client. Here are some excerpts of the output of "xl
>> -vvv create client.cfg":
>>
>> xenuser@xenhost:~$ sudo xl -vvv create client.cfg
>> Parsing config from xlcfg/client.cfg
>> libxl: debug: libxl_create.c:1556:do_domain_create: ao 0x21d46d0:
>> create: how=(nil) callback=(nil) poller=0x21d3b10
>> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
>> vdev=xvda spec.backend=phy
>> libxl: debug: libxl_device.c:201:disk_try_backend: Disk vdev=xvda, is
>> using a storage driver domain, skipping physical device check
>> libxl: debug: libxl_create.c:944:initiate_domain_create: running bootloader
>> libxl: debug: libxl_bootloader.c:330:libxl__bootloader_run: no
>> bootloader configured, using user supplied kernel
>> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
>> w=0x21d19b0: deregister unregistered
>> domainbuilder: detail: xc_dom_allocate:
>> cmdline="(hd0,0)/boot/grub/menu.lst", features="(null)"
>> libxl: debug: libxl_dom.c:623:libxl__build_pv: pv kernel mapped 0
>> path /usr/local/lib/xen/boot/pv-grub-x86_64.gz
>> domainbuilder: detail: xc_dom_kernel_file:
>> filename="/usr/local/lib/xen/boot/pv-grub-x86_64.gz"
>> [....]
>> domainbuilder: detail: launch_vm: called, ctxt=0x7f9631c10004
>> domainbuilder: detail: xc_dom_release: called
>> libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
>> vdev=xvda spec.backend=phy
>> libxl: debug: libxl_device.c:201:disk_try_backend: Disk vdev=xvda, is
>> using a storage driver domain, skipping physical device check
>> libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
>> w=0x21d2df0 wpath=/local/domain/1/backend/vbd/3/51712/state token=3/0:
>> register slotnum=3
>> libxl: debug: libxl_create.c:1573:do_domain_create: ao 0x21d46d0:
>> inprogress: poller=0x21d3b10, flags=i
>> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x21d2df0
>> wpath=/local/domain/1/backend/vbd/3/51712/state token=3/0: event
>> epath=/local/domain/1/backend/vbd/3/51712/state
>> libxl: debug: libxl_event.c:884:devstate_callback: backend
>> /local/domain/1/backend/vbd/3/51712/state wanted state 2 still waiting
>> state 1
>> libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x21d2df0
>> wpath=/local/domain/1/backend/vbd/3/51712/state token=3/0: event
>> epath=/local/domain/1/backend/vbd/3/51712/state
>> libxl: debug: libxl_event.c:880:devstate_callback: backend
>> /local/domain/1/backend/vbd/3/51712/state wanted state 2 ok
>> libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch
>> w=0x21d2df0 wpath=/local/domain/1/backend/vbd/3/51712/state token=3/0:
>> deregister slotnum=3
>> libxl: debug: libxl_device.c:938:device_backend_callback: calling
>> device_backend_cleanup
>> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
>> w=0x21d2df0: deregister unregistered
>> libxl: debug: libxl_device.c:993:device_hotplug: Backend domid 1,
>> domid 0, assuming driver domains
>> libxl: debug: libxl_device.c:996:device_hotplug: Not a remove, not
>> executing hotplug scripts
>> libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch
>> w=0x21d2ef0: deregister unregistered
>> libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
>> w=0x21d6010 wpath=/local/domain/0/backend/vif/3/0/state token=3/1:
>> register slotnum=3
>> [....]
>>
>> Meanwhile, client's console prints the following:
>>
>> xenuser@xenhost:~$ sudo xl console client
>> Xen Minimal OS!
>> start_info: 0xba4000(VA)
>> nr_pages: 0x40000
>> shared_inf: 0xa166c000(MA)
>> pt_base: 0xba7000(VA)
>> nr_pt_frames: 0xb
>> mfn_list: 0x9a4000(VA)
>> mod_start: 0x0(VA)
>> mod_len: 0
>> flags: 0x0
>> cmd_line: (hd0,0)/boot/grub/menu.lst
>> stack: 0x9630e0-0x9830e0
>> MM: Init
>> _text: 0x0(VA)
>> _etext: 0x75374(VA)
>> _erodata: 0x90000(VA)
>> _edata: 0x95d20(VA)
>> stack start: 0x9630e0(VA)
>> _end: 0x9a36e0(VA)
>> start_pfn: bb5
>> max_pfn: 40000
>> Mapping memory range 0x1000000 - 0x40000000
>> setting 0x0-0x90000 readonly
>> skipped 1000
>> MM: Initialise page allocator for dad000(dad000)-40000000(40000000)
>> MM: done
>> Demand map pfns at 40001000-0x2040001000.
>> Heap resides at 2040002000-4040002000.
>> Initialising timer interface
>> Initialising console ... done.
>> gnttab_table mapped at 0x40001000.
>> Initialising scheduler
>> Thread "Idle": pointer: 0x0x2040002050, stack: 0x0xfc0000
>> Thread "xenstore": pointer: 0x0x2040002800, stack: 0x0xfd0000
>> xenbus initialised on irq 1 mfn 0x240fa5
>> Thread "shutdown": pointer: 0x0x2040002fb0, stack: 0x0xfe0000
>> main.c: dummy main: start_info=0x9831e0
>> Thread "main": pointer: 0x0x2040003760, stack: 0x0xff0000
>> "main" "(hd0,0)/boot/grub/menu.lst"
>> vbd 51712 is hd0
>> ******************* BLKFRONT for device/vbd/51712 **********
>>
>>
>> backend at /local/domain/1/backend/vbd/3/51712
>>
>>
>> And it never advances beyond this point. "xl list" indicates that
>> client is blocked and waiting, and it remains at 0.1s of CPU time:
>>
>> xenuser@xenhost:~$ sudo xl list
>> Name ID Mem VCPUs State Time(s)
>> Domain-0 0 13024 8 r-----
>> 123.6
>> storagedd 1 2048 1 -b----
>> 23.4
>> client 3 1024 1 -b----
>> 0.1
>>
>> "xl block-list client" yields an empty list of block devices on the
>> client. In xenstore, the storage domain does, however, have a backend
>> entry for this disk:
>>
>> admin@storagedd:~$ sudo xenstore-ls /local/domain/1/backend
>> vbd = ""
>> 3 = ""
>> 51712 = ""
>> frontend = "/local/domain/3/device/vbd/51712"
>> params = "/dev/loop0"
>> script = "/etc/xen/scripts/block"
>> frontend-id = "3"
>> online = "1"
>> removable = "0"
>> bootable = "1"
>> state = "2"
>> dev = "xvda"
>> type = "phy"
>> mode = "w"
>> device-type = "disk"
>> discard-enable = "1"
>
> Hello,
>
> Thanks for the detailed description, AFAICT you don't have 'xl devd'
> running in the driver domain right?
>
> You will need to install the Xen tools inside of the driver domain, and
> then either launch 'xl devd' manually, or use the init script
> xendriverdomain.
>
> Roger.
Hi Roger,
Thanks for your reply. I got a bit further now, but still hit some errors.
First, as you suggested, I started the xendriverdomain service via the
init script (and have it start automatically on boot). "xl devd"
starts as expected and creates a log file at /var/log/xen/xldevd.log.
When I start the client DomU, it receives the disk and is able to boot
from it. I can even log in, if I do it quickly. However, after a few
seconds, the client locks up and I see this printed to the console:
[ 9.938197] vbd vbd-51712: 16 Device in use; refusing to close
[ 9.938524] vbd vbd-51712: failed to write error node for
device/vbd/51712 (16 Device in use; refusing to close)
When I destroy the client from Dom0:
xenuser@xenhost:~$ sudo xl destroy client
libxl: error: libxl_device.c:953:device_backend_callback: unable to
remove device with path /local/domain/2/backend/vbd/3/51712
libxl: error: libxl.c:1654:devices_destroy_cb: libxl__devices_destroy
failed for 3
And finally, in storagedd, I see the same error in xldevd.log:
admin@storagedd:~$ cat /var/log/xen/xldevd.log
libxl: error: libxl_device.c:953:device_backend_callback: unable to
remove device with path /local/domain/2/backend/vbd/3/51712
Any ideas on what's causing this or how to proceed?
Thanks,
Alex
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |