Re: [Xen-devel] Live migration bug introduced in 2.6.32.16?

[Nathan March <nathan@xxxxxx>]

On 1/27/2011 12:36 PM, Konrad Rzeszutek Wilk wrote:
> Whoa.. nice job figuring out what is wrong. One question: did you
> try to use the git branch from Jeremy's tree as a DomU?
Unfortunately this doesn't seem to be possible, whenever I try to start 
a domU with the latest from his git it crashes immediately. I've tried 
everything I can think of, I suspect it's due to my outdated hypervisor 
(4.0.1rc6) but it's not that far behind. Tried with and without 
grub/libvirtd, same effect. Config used is the same as the one I 
attached earlier, leaving any new options disabled.
(XEN) d74:v0: unhandled page fault (ec=0000)
(XEN) Pagetable walk from ffffffff817128c8:
(XEN)  L4[0x1ff] = 0000000a27fc9067 0000000000001003
(XEN)  L3[0x1fe] = 0000000a27fc5067 0000000000001007
(XEN)  L2[0x00b] = 0000000000000000 ffffffffffffffff
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 74 (vcpu#0) crashed on cpu#11:
(XEN) ----[ Xen-4.0.1-rc6-pre  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    11
(XEN) RIP:    e033:[<ffffffff8100ba99>]
(XEN) RFLAGS: 0000000000000202   EM: 1   CONTEXT: pv guest
(XEN) rax: ffffffff81712000   rbx: 0000000001d19067   rcx: 000000000000000e
(XEN) rdx: 0000000000000000   rsi: 0010000a272b3065   rdi: 0000000000000119
(XEN) rbp: ffffffff814a1d60   rsp: ffffffff814a1d20   r8:  00000000000008c8
(XEN) r9:  00003ffffffff000   r10: ffff880000000000   r11: 0000000000000010
(XEN) r12: ffff880001d19000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 0000000a27fcb000   cr2: ffffffff817128c8
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff814a1d20:
(XEN)    000000000000000e 0000000000000010 0000000000000000 ffffffff8100ba99
(XEN)    000000010000e030 0000000000010002 ffffffff814a1d60 000000000000e02b
(XEN)    ffffffff814a1d78 ffffffff8100bb95 ffffffff81001ea0 ffffffff814a1d88
(XEN)    ffffffff8100bc41 ffffffff814a1dd8 ffffffff8100b99a 0000000000000010
(XEN)    ffff880000000000 00003ffffffff000 00000000000008c8 0000000001d19067
(XEN)    0010000a272b3065 0000000000000000 0000000000000001 ffffffff814a1e08
(XEN)    ffffffff8154f258 0000000000000001 0000000000000000 ffffea0000000000
(XEN)    0000000000000000 ffffffff814a1e68 ffffffff8154dcb2 ffffffff814a1e58
(XEN)    ffffffff8153a15d ffffea00001bffff 0000000001000000 ffffea00001c0000
(XEN)    ffffea0000000000 0000000000000000 0000000000000000 ffff8800015e6000
(XEN)    0000000000000000 ffffffff814a1e88 ffffffff8154f0d5 ffffffff814a1f80
(XEN)    0000000000000000 ffffffff814a1ed8 ffffffff8153ba63 0000000000000001
(XEN)    ffff880001918000 ffff880001d18000 ffffffff814a1ee8 ffffffffffffffff
(XEN)    ffffffff814a1f80 0000000000000000 0000000000000000 ffffffff814a1f18
(XEN)    ffffffff81536770 0000000000001000 0000000000100000 0000000000040800
(XEN)    0000000000000000 ffffffff814a1f18 00000000015e6000 ffffffff814a1f68
(XEN)    ffffffff8152b339 ffffffff814a1f38 ffffffff81375c71 ffffffff814a1f58
(XEN)    ffffffff81055316 00000000015d9e48 0000000000000000 ffffffffffffffff
(XEN)    ffffffff8154fd40 ffffffff814a1fa8 ffffffff81527998 ffffffff814a1fa8
(XEN)    ffffffff81553360 00000000015d9e48 0000000000000000 0000000000000000

> You could run 'xenctx' and get a stack trace of what the guest is doing.
> That could narrow it down.
Hopefully this means something to you =)

xen5 ~ # xm list
Name                                        ID   Mem VCPUs      State   
Time(s)
Domain-0                                     0  1017     1     
r-----     76.5
nathanxen3                                   6  1024     2     
-b----      0.0
xen5 ~ # /usr/lib64/xen/bin/xenctx --stack-trace -a 6 0
rip: ffffffff810093aa
flags: 00001246 i z p
rsp: ffffffff81497f00
rax: 0000000000000000   rcx: ffffffff810093aa   rdx: 0000000000000246
rbx: ffffffff81496000   rsi: 0000000000000000   rdi: 0000000000000001
rbp: ffffffff81497f18    r8: 0000000000000000    r9: ffffffff81497e18
r10: 000000000000000c   r11: 0000000000000246   r12: ffffffff81505350
r13: ffffffff81543b10   r14: ffffffff81546730   r15: 0000000000000000
 cs: e033        ss: e02b        ds: 0000        es: 0000
 fs: 0000 @ 00007f039010b700
 gs: 0000 @ ffff88000181f000/0000000000000000

cr0: 8005003b
cr2: 7fff4e149ff0
cr3: 8ebcd5000
cr4: 00002660

dr0: 00000000
dr1: 00000000
dr2: 00000000
dr3: 00000000
dr6: ffff0ff0
dr7: 00000400
Code (instr addr ffffffff810093aa)
cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 
59 c3 cc cc cc cc cc cc cc

Stack:
 ffff88003f8a5da0 0000000000000000 ffffffff8100d53c ffffffff81497f38
 ffffffff8100b3ef ffffffff81497f38 ffffffff81497fd8 ffffffff81497f58
 ffffffff8100f24e 0000000000000000 ffffffffffffffff ffffffff81497f68
 ffffffff81353e9d ffffffff81497fa8 ffffffff8151cbe4 ffffffff81497fa8

Stack Trace:
* [<ffffffff810093aa>] <--
    ffff88003f8a5da0
    0000000000000000
  [<ffffffff8100d53c>]
  [<ffffffff81497f38>]
  [<ffffffff8100b3ef>]
  [<ffffffff81497f38>]
  [<ffffffff81497fd8>]
  [<ffffffff81497f58>]
  [<ffffffff8100f24e>]
    0000000000000000
  [<ffffffffffffffff>]
  [<ffffffff81497f68>]
  [<ffffffff81353e9d>]
  [<ffffffff81497fa8>]
  [<ffffffff8151cbe4>]
  [<ffffffff81497fa8>]
  [<ffffffff81546730>]
    00000000015dd548
    0000000000000000
    0000000000000000
    0000000000000000
  [<ffffffff81497fc8>]
  [<ffffffff8151c295>]
  [<ffffffff81515590>]
  [<ffffffff81801000>]
  [<ffffffff81497ff8>]
  [<ffffffff8151deff>]
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
xen5 ~ # /usr/lib64/xen/bin/xenctx --stack-trace -a 6 1
rip: ffffffff810093aa
flags: 00001246 i z p
rsp: ffff88003f851ee8
rax: 0000000000000000   rcx: ffffffff810093aa   rdx: 0000002b02eb8fe8
rbx: ffff88003f850000   rsi: 0000000000000000   rdi: 0000000000000001
rbp: ffff88003f851f00    r8: 0000000000000000    r9: ffff88003f851e00
r10: 000000000000000c   r11: 0000000000000246   r12: ffffffff81505350
r13: 0000000000000000   r14: 0000000000000000   r15: 0000000000000000
 cs: e033        ss: e02b        ds: 002b        es: 002b
 fs: 0000 @ 00002b62ff2fcb20
 gs: 0000 @ ffff880001839000/0000000000000000

cr0: 8005003b
cr2: 7f302f14ce10
cr3: 8ebc46000
cr4: 00002660

dr0: 00000000
dr1: 00000000
dr2: 00000000
dr3: 00000000
dr6: ffff0ff0
dr7: 00000400
Code (instr addr ffffffff810093aa)
cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 
59 c3 cc cc cc cc cc cc cc

Stack:
 ffff88003e169da0 0000000000000000 ffffffff8100d53c ffff88003f851f20
 ffffffff8100b3ef ffff88003f851f20 ffff88003f851fd8 ffff88003f851f40
 ffffffff8100f24e 0000000000000000 0000000000000000 ffff88003f851f50
 ffffffff81365b59 0000000000000000 0000000000000000 0000000000000000

Stack Trace:
* [<ffffffff810093aa>] <--
    ffff88003e169da0
    0000000000000000
  [<ffffffff8100d53c>]
    ffff88003f851f20
  [<ffffffff8100b3ef>]
    ffff88003f851f20
    ffff88003f851fd8
    ffff88003f851f40
  [<ffffffff8100f24e>]
    0000000000000000
    0000000000000000
    ffff88003f851f50
  [<ffffffff81365b59>]
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000
    0000000000000000


- Nathan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel