[Xen-devel] GPF Heisenbug with rumprun-xen

[Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>]

Ian Campbell writes ("Re: [Xen-devel] [xen-4.5-testing test] 34157: regressions 
- FAIL"):
> > > > http://www.chiark.greenend.org.uk/~xensrcts/logs/34157/ 
...
> > > >  test-amd64-amd64-rumpuserxen-amd64 11 
> > > > rumpuserxen-demo-xenstorels/xenstorels fail REGR. vs. 34088

Guest console contains:

   device/vbd/768/protocol = "x86_64-abi"   (n2,r0)
   device/vbd/832:
   rumpxenstack:
   could not access permissions for 832: Bad file descriptor

   rumpxenstack: xs_directory (device/vbd/832): Bad file descriptor

   === ERROR: _exit(1) called ===

> It looks to be some sort of Heisenbug in the rump kernel stuff.

I agree.  We had a failure on the 16th of January which looked like
some kind of race:
 (Subject: Re: [Xen-devel] [rumpuserxen test] 33416: regressions - FAIL)


> This 
> http://www.chiark.greenend.org.uk/~xensrcts/results/history.test-amd64-amd64-rumpuserxen-amd64.html
> show a history of random failures at the xenstorels step.
>
> At least the ones as far back as 33830 (the last one with logs still
> available) all show signs of what looks like memory corruption of some
> sort.

Thanks for the digging.  (I have left the quoted text in for the
benefit of rumpkernel-users.)

The first failure in that history that looks like part of this is
flight 33690.  We don't have logs for that any more but it used
  rumpuserxen 598ceb54916b
  xen         49de0b57b853
  netbsdsrc   17a547ca2943
Failure probability after then seems about 20%.  If I go back 10
passes from 33690 I get to 33611 which used
  rumpuserxen ffcd777f8062
  xen         0d2879062076
  netbsdsrc   a7c6b12e1752

It seems unlikely that the difference is going to be due to changes
in the versions of linux, linuxfirmware, ovmf, qemu[u] or seabios.
buildrump.sh has been 47b1a5eef43c throughout.

Ian.


> http://www.chiark.greenend.org.uk/~xensrcts/logs/33830/test-amd64-amd64-rumpuserxen-amd64/11.ts-rumpuserxen-demo-xenstorels.log
>         GPF rip: 0x46ea1c, error_code=0
>         Page fault at linear address 0x0, rip 0x13563, regs 0x469ff8, sp 
> 0x46a0a8, our_sp 0x469fe0, code 0
>         Page fault in pagetable walk (access to invalid memory?).
>         
> http://www.chiark.greenend.org.uk/~xensrcts/logs/33846/test-amd64-amd64-rumpuserxen-amd64/11.ts-rumpuserxen-demo-xenstorels.log
>         $VAR1 = {
>                   'theirs' => 'STUB ``__sigaction14\'\' called
>         device/?G:
>         rumpxenstack: 
>         could not access permissions for ?G: Invalid argument
>         
>         rumpxenstack: xs_directory (device/?G): Invalid argument
> 
> http://www.chiark.greenend.org.uk/~xensrcts/logs/33925/test-amd64-amd64-rumpuserxen-amd64/11.ts-rumpuserxen-demo-xenstorels.log
>         Similar to 33846
> 
>         
> http://www.chiark.greenend.org.uk/~xensrcts/logs/34086/test-amd64-amd64-rumpuserxen-amd64/11.ts-rumpuserxen-demo-xenstorels.log
>         rumpxenstack: 
>         could not access permissions for mac: Invalid argument
>         device/vif/0/mac = "5a:36:0e:26:00:05" 
>         rumpxenstack: xs_directory (device/vif/0/mac): Bad file descriptor
> 
> http://www.chiark.greenend.org.uk/~xensrcts/logs/34127/test-amd64-amd64-rumpuserxen-amd64/11.ts-rumpuserxen-demo-xenstorels.log
>         Page fault at linear address 0x256, rip 0x33e1c, regs 0x53f458, sp 
> 0x53f500, our_sp 0x53f440, code 0
>         Thread: main
>         RIP: e030:[<0000000000033e1c>] 
>         RSP: e02b:000000000053f500  EFLAGS: 00010202
>         RAX: 000000000025a927 RBX: 0000000000000246 RCX: 000000000053fe88
>         RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000025a8b0
>         RBP: 00000000001d55df R08: 0000000000454091 R09: 0000000000000000
>         R10: 0000000000000000 R11: 0000000000000000 R12: 000000000053fe88
>         R13: 0000000000000000 R14: 000000000025a8b0 R15: 0000000000000000
>         base is 0x1d55df caller is 0x6e69616d20676e69
>         base is 0x6c6c6163203d3d3d GPF rip: 0x13e00, error_code=0
>         
> http://www.chiark.greenend.org.uk/~xensrcts/logs/34157/test-amd64-amd64-rumpuserxen-amd64/11.ts-rumpuserxen-demo-xenstorels.log
>         Another bad fd, on device/vbd but otherwise similar to 34086.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel