Re: [Xen-users] trouble with xenserver and xfs (soft lockup - CPU#0 stuck for 61s!)

[Yann Cezard <yann.cezard@xxxxxxxxxxx>]

Sebastian Gosenheimer a Ãcrit :
> Hi everybody,
>
> i hope that i'm on the right mailing list. I'm having some trouble with 
> xfs and xen. We are running the newest xenserver version with ha on dell 
> servers and a dell equalogic.
>
> I set up a fileserver (debian lenny 2.6.29-xs5.5.0.17) with one ext3 
> partiton for the os and one xfs partition for the data. Two webserver 
> are using this fileserver with nfs.
>
> Now i run the second time into following problem when rsync was running 
> for backup:
>
> [212521.428003] BUG: soft lockup - CPU#0 stuck for 61s! [rsync:29921]
> [212521.428003] Modules linked in: ipv6 nfsd nfs lockd nfs_acl 
> auth_rpcgss sunrpc xenfs xfs exportfs loop evdev xen_netfront pcspkr 
> ext3 jbd mbcache xen_blkfront thermal_sys
> [212521.428003]
> [212521.428003] Pid: 29921, comm: rsync Tainted: G      D 
> (2.6.29-xs5.5.0.17 #1)
> [212521.428003] EIP: 0061:[<c02f2221>] EFLAGS: 00000287 CPU: 0
> [212521.428003] EIP is at __write_lock_failed+0x9/0x1c
> [212521.428003] EAX: e1875cbc EBX: c1351e1a ECX: e2407880 EDX: dd004c50
> [212521.428003] ESI: 00000000 EDI: c536fd40 EBP: e1875cbc ESP: c1351dac
> [212521.428003]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
> [212521.428003] CR0: 8005003b CR2: b806e348 CR3: 20d99000 CR4: 00002620
> [212521.428003] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> [212521.428003] DR6: ffff0ff0 DR7: 00000400
> [212521.428003] Call Trace:
> [212521.428003]  [<c02f23a3>] ? _write_lock+0xe/0xf
> [212521.428003]  [<e33f7409>] ? xfs_iget+0x328/0x44f [xfs]
> [212521.428003]  [<e340e925>] ? xfs_lookup+0x69/0x97 [xfs]
> [212521.428003]  [<e341638b>] ? xfs_vn_lookup+0x36/0x6e [xfs]
> [212521.428003]  [<c0194ee1>] ? do_lookup+0xa6/0x116
> [212521.428003]  [<c019584d>] ? __link_path_walk+0x524/0x631
> [212521.428003]  [<c01057fc>] ? xen_force_evtchn_callback+0xc/0x10
> [212521.428003]  [<c0195d5c>] ? path_walk+0x4f/0xa3
> [212521.428003]  [<c0196aa0>] ? do_path_lookup+0x132/0x178
> [212521.428003]  [<c019732e>] ? getname+0x5e/0xb0
> [212521.428003]  [<c0197b05>] ? user_path_at+0x37/0x5f
> [212521.428003]  [<c0198e80>] ? filldir64+0x0/0xc5
> [212521.428003]  [<c019187f>] ? vfs_lstat_fd+0x12/0x3
> [212455.928003]  [<c0191912>] ? sys_lstat64+0xf/0x23
> [212455.928003]  [<c0199124>] ? vfs_readdir+0x7c/0x8c
> [212455.928003]  [<c0198e80>] ? filldir64+0x0/0xc5
> [212455.928003]  [<c01991cf>] ? sys_getdents64+0x9b/0xa5
> [212455.928003]  [<c0107f23>] ? sysenter_past_esp+0x3c/0x62
> [212455.928003]  [<c0107f5b>] ? sysenter_do_call+0x12/0x2f
>
> I have to force reboot to bring the vm back online. Are there any known 
> problems with xfs and xen? Because i know from another deployment that 
> they are having nearly the same issues with xfs. There are no problems 
> with the ext3 filesystem.
>
> Thank you for your help!
>
>   
Hi Sebastian,

I'm having similar issues with two Lenny domU (amd64 and i386),
both with XFS + quite heavy load (I should say normal load for
what they do : database server and fileserver).
They are running the standard Lenny Xen kernel (2.6.26-2-xen-*)

After searching a little, I found this thread :
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=28968&start=0&tstart=0

Not sure that it is connected to my (or your) problem, but the domUs
I have problems with both have XFS log version 2, i have others domUs
with XFS filesystems and quite heavy charge (mail servers basically),
but they are still in Etch and XFS log version 1, no problem with them.

So I think I will give the log version workaround a try.

Perhaps could you give it a try to ?

In any case, if you find something about that problem, I would be happy
to have feedback from you.

Cheers,

-- 
Yann CÃzard - Administrateur SystÃmes Serveurs


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users