Re: [Xen-users] trouble with xenserver and xfs (soft lockup - CPU#0 stuck for 61s!)

[Sebastian Gosenheimer <s.gosenheimer@xxxxxxxxx>]

Am 27.04.2010 10:51, schrieb Yann Cezard:
> Sebastian Gosenheimer a Ãcrit :
>> Hi everybody,
>>
>> i hope that i'm on the right mailing list. I'm having some trouble with
>> xfs and xen. We are running the newest xenserver version with ha on dell
>> servers and a dell equalogic.
>>
>> I set up a fileserver (debian lenny 2.6.29-xs5.5.0.17) with one ext3
>> partiton for the os and one xfs partition for the data. Two webserver
>> are using this fileserver with nfs.
>>
>> Now i run the second time into following problem when rsync was running
>> for backup:
>>
>> [212521.428003] BUG: soft lockup - CPU#0 stuck for 61s! [rsync:29921]
>> [212521.428003] Modules linked in: ipv6 nfsd nfs lockd nfs_acl
>> auth_rpcgss sunrpc xenfs xfs exportfs loop evdev xen_netfront pcspkr
>> ext3 jbd mbcache xen_blkfront thermal_sys
>> [212521.428003]
>> [212521.428003] Pid: 29921, comm: rsync Tainted: G      D
>> (2.6.29-xs5.5.0.17 #1)
>> [212521.428003] EIP: 0061:[<c02f2221>] EFLAGS: 00000287 CPU: 0
>> [212521.428003] EIP is at __write_lock_failed+0x9/0x1c
>> [212521.428003] EAX: e1875cbc EBX: c1351e1a ECX: e2407880 EDX: dd004c50
>> [212521.428003] ESI: 00000000 EDI: c536fd40 EBP: e1875cbc ESP: c1351dac
>> [212521.428003]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
>> [212521.428003] CR0: 8005003b CR2: b806e348 CR3: 20d99000 CR4: 00002620
>> [212521.428003] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
>> [212521.428003] DR6: ffff0ff0 DR7: 00000400
>> [212521.428003] Call Trace:
>> [212521.428003]  [<c02f23a3>] ? _write_lock+0xe/0xf
>> [212521.428003]  [<e33f7409>] ? xfs_iget+0x328/0x44f [xfs]
>> [212521.428003]  [<e340e925>] ? xfs_lookup+0x69/0x97 [xfs]
>> [212521.428003]  [<e341638b>] ? xfs_vn_lookup+0x36/0x6e [xfs]
>> [212521.428003]  [<c0194ee1>] ? do_lookup+0xa6/0x116
>> [212521.428003]  [<c019584d>] ? __link_path_walk+0x524/0x631
>> [212521.428003]  [<c01057fc>] ? xen_force_evtchn_callback+0xc/0x10
>> [212521.428003]  [<c0195d5c>] ? path_walk+0x4f/0xa3
>> [212521.428003]  [<c0196aa0>] ? do_path_lookup+0x132/0x178
>> [212521.428003]  [<c019732e>] ? getname+0x5e/0xb0
>> [212521.428003]  [<c0197b05>] ? user_path_at+0x37/0x5f
>> [212521.428003]  [<c0198e80>] ? filldir64+0x0/0xc5
>> [212521.428003]  [<c019187f>] ? vfs_lstat_fd+0x12/0x3
>> [212455.928003]  [<c0191912>] ? sys_lstat64+0xf/0x23
>> [212455.928003]  [<c0199124>] ? vfs_readdir+0x7c/0x8c
>> [212455.928003]  [<c0198e80>] ? filldir64+0x0/0xc5
>> [212455.928003]  [<c01991cf>] ? sys_getdents64+0x9b/0xa5
>> [212455.928003]  [<c0107f23>] ? sysenter_past_esp+0x3c/0x62
>> [212455.928003]  [<c0107f5b>] ? sysenter_do_call+0x12/0x2f
>>
>> I have to force reboot to bring the vm back online. Are there any known
>> problems with xfs and xen? Because i know from another deployment that
>> they are having nearly the same issues with xfs. There are no problems
>> with the ext3 filesystem.
>>
>> Thank you for your help!
>>
>>
> Hi Sebastian,
>
> I'm having similar issues with two Lenny domU (amd64 and i386),
> both with XFS + quite heavy load (I should say normal load for
> what they do : database server and fileserver).
> They are running the standard Lenny Xen kernel (2.6.26-2-xen-*)
>
> After searching a little, I found this thread :
> http://developer.amazonwebservices.com/connect/thread.jspa?threadID=28968&start=0&tstart=0
>
> Not sure that it is connected to my (or your) problem, but the domUs
> I have problems with both have XFS log version 2, i have others domUs
> with XFS filesystems and quite heavy charge (mail servers basically),
> but they are still in Etch and XFS log version 1, no problem with them.
>
> So I think I will give the log version workaround a try.
>
> Perhaps could you give it a try to ?
>
> In any case, if you find something about that problem, I would be happy
> to have feedback from you.
>
> Cheers,
>

Hi Yann,

sorry for the late answer. I was messing arround the week with some 
other things.

I have now since four days up and running a new kernel on the fileserver 
(2.6.32.12-xs5.5.0.17). Since this time everything seems to be fine.

Right now i don't know if there is a xfs or nfs problem in the kernel 
2.6.26-2-xen-*. I will update all our machines to a newer kernel know 
which are using nfs and/or xfs.

 From my opionion the thread you sent isn't connected to my problem.

Did the workaround work for you or do you still have problems.

--sg


Diese E-Mail enthÃlt vertrauliche und/oder rechtlich geschÃtzte Informationen. 
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtÃmlich erhalten 
haben, 
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. 
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht 
gestattet. 

This e-mail may contain confidential and/or privileged information. 
If you are not the intended recipient (or have received this e-mail in error) 
please notify 
the sender immediately and destroy this e-mail.  
Any unauthorized copying, disclosure or distribution of the material in this 
e-mail is strictly forbidden. 



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users