[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] XCP NFS SR issues



Hello all,

We have 2 x compute nodes running XCP and 2 x storage nodes with GlusterFS exposed via NFS to the compute nodes.

Roughly every week we're experiencing failure of VMs due to tapdisk errors like the below :

Aug 31 17:00:51 xen02 tapdisk[15678]: ERROR: errno -116 at vhd_complete: /var/run/sr-mount/0554f47c-1a12-9bd1-ea9b-d2b68984f0ed/75af4999-3447-4fa4-bdf0-b89580807a7c.vhd: op: 1, lsec: 3464605, secs: 8, nbytes: 4096, blk: 845, blk_offset: 2725095 Aug 31 17:00:51 xen02 tapdisk[15678]: ERROR: errno -116 at __tapdisk_vbd_complete_td_request: req 1: read 0x0008 secs to 0x0034dd9d

The standard fix is always to reboot the XCP host, start the VMs and repair filesystems, and so far, I'm happy/lucky to say they have repaired successfully and come back up. However, I believe it is only a matter of time before I experience an unrecoverable failure....

We have introduced NIC bonding on the storage nodes (All NICs are Gb), we have only ~8 VMs running on the compute nodes currently, but was reluctant to add NIC bonding to them as it means reconfiguring the VMs' VIFs etc.

Can anybody advise what fine tuning of NFS may have worked for you? Are there any "Usual suspects" I should be looking at here? We're currently using vanilla NFS out of the box...

The main reason we're using Gluster is so we can expose volumes greater than the size of a single physical disc and have built in redundancy. We achieved good direct I/O results on the storage nodes, hence why I'm suspecting NFS...

Any feedback would be very much appreciated - the powers that be *may* use this as an excuse to test Hyper-V as a solution which I want to avoid at all costs ;)

Many thanks in advance for any trouble taken.

Kind regards,

Ciaran Kendellen.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.