Re: [Xen-devel] Create a iSCSI DomU with disks in another DomU running on the same Dom0

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On 21/12/12 15:03, Konrad Rzeszutek Wilk wrote:
> On Fri, Dec 21, 2012 at 09:29:39AM +0100, Roger Pau Monné wrote:
>> Hello,
>>
>> I'm trying to use a strange setup, that consists in having a DomU
>> serving iSCSI targets to the Dom0, that will use this targets as disks
>> for other DomUs. I've tried to set up this iSCSI target DomU using both
>> Debian Squeeze/Wheezy (with kernels 2.6.32 and 3.2) and ISCSI
>> Enterprise Target (IET), and when launching the DomU I get this messages
>> from Xen:
>>
>> (XEN) mm.c:1925:d0 Error pfn 157e68: rd=ffff83019e60c000, 
>> od=ffff830141405000, caf=8000000000000003, taf=7400000000000001
>> (XEN) Xen WARN at mm.c:1926
>> (XEN) ----[ Xen-4.3-unstable  x86_64  debug=y  Not tainted ]----
>> (XEN) CPU:    0
>> (XEN) RIP:    e008:[<ffff82c48016ea17>] get_page+0xd5/0x101
>> (XEN) RFLAGS: 0000000000010286   CONTEXT: hypervisor
>> (XEN) rax: 0000000000000000   rbx: ffff830141405000   rcx: 0000000000000000
>> (XEN) rdx: ffff82c480300920   rsi: 000000000000000a   rdi: ffff82c4802766e8
>> (XEN) rbp: ffff82c4802bfbf8   rsp: ffff82c4802bfba8   r8:  0000000000000004
>> (XEN) r9:  0000000000000004   r10: 0000000000000004   r11: 0000000000000001
>> (XEN) r12: 0000000000157e68   r13: ffff83019e60c000   r14: 7400000000000001
>> (XEN) r15: 8000000000000003   cr0: 000000008005003b   cr4: 00000000000026f0
>> (XEN) cr3: 000000011c180000   cr2: 00007f668d1eb000
>> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
>> (XEN) Xen stack trace from rsp=ffff82c4802bfba8:
>> (XEN)    ffff830141405000 8000000000000003 7400000000000001 0000000000145028
>> (XEN)    ffff82f6028a0510 ffff83019e60c000 ffff82f602afcd00 ffff82c4802bfd28
>> (XEN)    ffff82c4802bfd18 0000000000157e68 ffff82c4802bfc58 ffff82c480109ba3
>> (XEN)    ffffffffffffffff 0000000000000000 ffff83011c977fb8 0000000061dfc3f0
>> (XEN)    0000000000000001 ffffffffffff8000 0000000000000002 ffff83011d555000
>> (XEN)    ffff83019e60c000 0000000000000000 ffff82c4802bfd98 ffff82c48010c607
>> (XEN)    ffff82c4802bfd34 ffff82c4802bfd30 ffff82c400000001 000000000011cf90
>> (XEN)    0000000000000000 ffff82c4802b8000 ffff82c4802b8000 ffff82c4802b8000
>> (XEN)    ffff82c4802b8000 ffff82c4802bfd5c 000000029e60c000 ffff82c480300920
>> (XEN)    ffff82c4802b8000 ffff82c4802bfd38 00000005802bfd38 ffff82c4802b8000
>> (XEN)    ffff82c400000000 0000000000000001 ffffc90000028b10 ffffc90000028b10
>> (XEN)    ffff8300dfb03000 0000000000000000 0000000000000000 0000000000145028
>> (XEN)    000000000011cf7c 0000000000001000 0000000000157e68 0000000000007ff0
>> (XEN)    000000000000027e 000000000042000d 0000000000020b50 ffff8300dfdf0000
>> (XEN)    ffff82c4802bfd78 ffffc90000028ac0 ffffc90000028ac0 ffff880185f6fd58
>> (XEN)    ffff880185f6fd78 0000000000000005 ffff82c4802bfef8 ffff82c48010eb65
>> (XEN)    ffff82c4802bfdc8 ffff82c480300960 ffff82c4802bfe18 ffff82c480181831
>> (XEN)    000000000006df66 000032cfdc175ce6 0000000000000000 0000000000000000
>> (XEN)    0000000000000000 0000000000000005 ffff82c4802bfe28 ffff8300dfb03000
>> (XEN)    ffff8300dfdf0000 0000150e11a417f8 0000000000000002 ffff82c480300948
>> (XEN) Xen call trace:
>> (XEN)    [<ffff82c48016ea17>] get_page+0xd5/0x101
>> (XEN)    [<ffff82c480109ba3>] __get_paged_frame+0xbf/0x162
>> (XEN)    [<ffff82c48010c607>] gnttab_copy+0x4c6/0x91a
>> (XEN)    [<ffff82c48010eb65>] do_grant_table_op+0x12ad/0x1b23
>> (XEN)    [<ffff82c48022280b>] syscall_enter+0xeb/0x145
>> (XEN)    
>> (XEN) grant_table.c:2076:d0 source frame ffffffffffffffff invalid.
>> (XEN) mm.c:1925:d0 Error pfn 157e68: rd=ffff83019e60c000, 
>> od=ffff830141405000, caf=8000000000000003, taf=7400000000000001
>> (XEN) Xen WARN at mm.c:1926
>> (XEN) ----[ Xen-4.3-unstable  x86_64  debug=y  Not tainted ]----
>> (XEN) CPU:    0
>> (XEN) RIP:    e008:[<ffff82c48016ea17>] get_page+0xd5/0x101
>> (XEN) RFLAGS: 0000000000010286   CONTEXT: hypervisor
>> (XEN) rax: 0000000000000000   rbx: ffff830141405000   rcx: 0000000000000000
>> (XEN) rdx: ffff82c480300920   rsi: 000000000000000a   rdi: ffff82c4802766e8
>> (XEN) rbp: ffff82c4802bfbf8   rsp: ffff82c4802bfba8   r8:  0000000000000004
>> (XEN) r9:  0000000000000004   r10: 0000000000000004   r11: 0000000000000001
>> (XEN) r12: 0000000000157e68   r13: ffff83019e60c000   r14: 7400000000000001
>> (XEN) r15: 8000000000000003   cr0: 000000008005003b   cr4: 00000000000026f0
>> (XEN) cr3: 000000011c180000   cr2: 00007f668d1eb000
>> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
>> (XEN) Xen stack trace from rsp=ffff82c4802bfba8:
>> (XEN)    ffff830141405000 8000000000000003 7400000000000001 000000000014581d
>> (XEN)    ffff82f6028b03b0 ffff83019e60c000 ffff82f602afcd00 ffff82c4802bfd28
>> (XEN)    ffff82c4802bfd18 0000000000157e68 ffff82c4802bfc58 ffff82c480109ba3
>> (XEN)    ffffffffffffffff 0000000000000000 ffff83011c977fb8 0000000061dfc308
>> (XEN)    0000000000000000 ffffffffffff8000 0000000000000001 ffff83011d555000
>> (XEN)    ffff83019e60c000 0000000000000000 ffff82c4802bfd98 ffff82c48010c607
>> (XEN)    ffff82c4802bfd34 ffff82c4802bfd30 ffff82c400000001 000000000011cf90
>> (XEN)    0000000000000000 ffff82c4802b8000 ffff82c4802b8000 ffff82c4802b8000
>> (XEN)    ffff82c4802b8000 ffff82c4802bfd5c 000000029e60c000 ffff82c480300920
>> (XEN)    ffff82c4802b8000 ffff82c4802bfd38 00000002802bfd38 ffff82c4802b8000
>> (XEN)    ffffffff00000000 0000000000000001 ffffc90000028b60 ffffc90000028b60
>> (XEN)    ffff8300dfb03000 0000000000000000 0000000000000000 000000000014581d
>> (XEN)    00000000000deb3e 0000000000001000 0000000000157e68 000000000b507ff0
>> (XEN)    0000000000000261 000000000042000d 00000000000204b0 ffffc90000028b38
>> (XEN)    0000000000000002 ffffc90000028b38 ffffc90000028b38 ffff880185f6fd58
>> (XEN)    ffff880185f6fd78 0000000000000005 ffff82c4802bfef8 ffff82c48010eb65
>> (XEN)    ffff82c4802bfdc8 ffff82c480300960 ffff82c4802bfe18 ffff82c480181831
>> (XEN)    000000000006df66 000032cfdc175ce6 0000000000000000 0000000000000000
>> (XEN)    0000000000000000 0000000000000005 ffff82c4802bfe28 0000000000000086
>> (XEN)    ffff82c4802bfe28 ffff82c480125eae ffff83019e60c000 0000000000000286
>> (XEN) Xen call trace:
>> (XEN)    [<ffff82c48016ea17>] get_page+0xd5/0x101
>> (XEN)    [<ffff82c480109ba3>] __get_paged_frame+0xbf/0x162
>> (XEN)    [<ffff82c48010c607>] gnttab_copy+0x4c6/0x91a
>> (XEN)    [<ffff82c48010eb65>] do_grant_table_op+0x12ad/0x1b23
>> (XEN)    [<ffff82c48022280b>] syscall_enter+0xeb/0x145
>> (XEN)    
>> (XEN) grant_table.c:2076:d0 source frame ffffffffffffffff invalid.
>>
>> (Note that I've added a WARN() to mm.c:1925 to see where the
>> get_page call was coming from).
>>
>> Connecting the iSCSI disks to another Dom0 works fine, so this
>> problem only happens when trying to connect the disks to the
>> Dom0 where the DomU is running.
> 
> Is this happening when the 'disks' are exported to the domUs?
> Are they exported via QEMU or xen-blkback?

The iSCSI disks are connected to the DomUs using blkback, and this is
happening when the DomU tries to access it's disks.

>>
>> I've replaced the Linux DomU serving iSCSI targets with a
>> NetBSD DomU, and the problems disappears, and I'm able to
>> attach the targets shared by the DomU to the Dom0 without
>> issues.
>>
>> The problem seems to come from netfront/netback, does anyone
>> have a clue about what might cause this bad interaction
>> between IET and netfront/netback?
> 
> Or it might be that we are re-using the PFN for blkback/blkfront
> and using the m2p overrides and overwritting the netfront/netback
> m2p overrides?

What's strange is that this doesn't happen when the domain that has the
targets is a NetBSD PV. There are also problems when blkback is not used
(see below), so I guess the problem is between netfront/netback and IET.

> Is this with an HVM domU or PV domU?

Both domains (the domain holding the iSCSI targets, and the created
guests) are PV.

Also, I've forgot to say that in the previous email, but if I just
connect the iSCSI disks to the Dom0, I don't see any errors from Xen,
but the Dom0 kernel starts complaining:

[70272.569607] sd 14:0:0:0: [sdc]
[70272.569611] Sense Key : Medium Error [current]
[70272.569619] Info fld=0x0
[70272.569623] sd 14:0:0:0: [sdc]
[70272.569627] Add. Sense: Unrecovered read error
[70272.569633] sd 14:0:0:0: [sdc] CDB:
[70272.569637] Read(10): 28 00 00 00 00 00 00 00 08 00
[70272.569662] end_request: critical target error, dev sdc, sector 0
[70277.571208] sd 14:0:0:0: [sdc] Unhandled sense code
[70277.571220] sd 14:0:0:0: [sdc]
[70277.571224] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[70277.571229] sd 14:0:0:0: [sdc]
[70277.571233] Sense Key : Medium Error [current]
[70277.571241] Info fld=0x0
[70277.571245] sd 14:0:0:0: [sdc]
[70277.571249] Add. Sense: Unrecovered read error
[70277.571255] sd 14:0:0:0: [sdc] CDB:
[70277.571259] Read(10): 28 00 00 00 00 00 00 00 08 00
[70277.571284] end_request: critical target error, dev sdc, sector 0
[70282.572768] sd 14:0:0:0: [sdc] Unhandled sense code
[70282.572781] sd 14:0:0:0: [sdc]
[70282.572785] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[70282.572790] sd 14:0:0:0: [sdc]
[70282.572794] Sense Key : Medium Error [current]
[70282.572802] Info fld=0x0
[70282.572806] sd 14:0:0:0: [sdc]
[70282.572810] Add. Sense: Unrecovered read error
[70282.572816] sd 14:0:0:0: [sdc] CDB:
[70282.572820] Read(10): 28 00 00 00 00 00 00 00 08 00
[70282.572846] end_request: critical target error, dev sdc, sector 0
[70287.574397] sd 14:0:0:0: [sdc] Unhandled sense code
[70287.574409] sd 14:0:0:0: [sdc]
[70287.574413] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[70287.574418] sd 14:0:0:0: [sdc]
[70287.574422] Sense Key : Medium Error [current]
[70287.574430] Info fld=0x0
[70287.574434] sd 14:0:0:0: [sdc]
[70287.574438] Add. Sense: Unrecovered read error
[70287.574445] sd 14:0:0:0: [sdc] CDB:
[70287.574448] Read(10): 28 00 00 00 00 00 00 00 08 00
[70287.574474] end_request: critical target error, dev sdc, sector 0

When I try to attach the targets to another Dom0, everything works fine,
the problem only happens when the iSCSI target is a DomU and you attach
the disks from the Dom0 on the same machine.

Thanks, Roger.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel