Re: [Xen-API] NFS Tuning - How To

[hook <sergey.kruchatov@xxxxxxxxxx>]

Hi!

Thanks, you give me a chance to share my expirience in straggle with poor NFS performance in XCP 1.6.x

First of all, you just need to check mounting options issued by NFS client in XEN.

You should see something like

#mount

/dev/sda1 on / type ext3 (rw)

none on /proc type proc (rw)

none on /sys type sysfs (rw)

none on /dev/pts type devpts (rw)

none on /dev/shm type tmpfs (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

10.1.0.42:/var/storage1/ISO on /var/run/sr-mount/0f61bf48-650b-41e6-183f-6231f763468f type nfs (rw,soft,timeo=133,retrans=2147483647,tcp,acregmax=1,acdirmax=1,addr=10.1.0.42)

#

In my case you see final result, which works rather fine for me.

But in case you will find noac option inside brackets - yes, it there!

Open theÂ/opt/xensource/sm/nfs.py and jump onto the 66 line.

You would seeÂnear code something likeÂ

options += ',noac' # CA-27534

You just need to comment out this line of code and reattach you NFS or NFS ISO storage to check performance.

In our case the VM protection policy archiving process to NFS storage became 10x faster.

You can do even more and google the "noac CA-27534".

I sure you will find about year old patchÂ

https://github.com/mcclurmc/xcp-storage-managers/pull/1

-ÂÂÂÂoptionsÂ+=Â',noac'Â#ÂCA-27534
Â	72	+
Â	73	+ÂÂÂÂ#ÂAttributeÂcachingÂcanÂleadÂtoÂstaleÂdata,ÂsoÂweÂcanÂtryÂtoÂminimizeÂthese
Â	74	+ÂÂÂÂ#ÂproblemsÂbyÂreducingÂtheÂcachingÂtime.ÂGoingÂrightÂdownÂtoÂzeroÂdoesÂnot
Â	75	+ÂÂÂÂ#ÂremoveÂraceÂconditionsÂ(ratherÂjustÂmakesÂthemÂmoreÂunlikely)Âwhile
Â	76	+ÂÂÂÂ#ÂintroducingÂsomeÂseriousÂperformanceÂpenaltiesÂforÂtheÂNFSÂfilesystem.
Â	77	+ÂÂÂÂ#ÂSoÂweÂgetÂ99%ÂofÂtheÂwayÂthereÂwithoutÂtheÂmassiveÂperformanceÂpenalties
Â	78	+ÂÂÂÂ#ÂbyÂgoingÂforÂ1Âsecond.
Â	79	+ÂÂÂÂ#ÂTheÂcorrectÂsolutionÂtoÂcasesÂwhereÂyouÂneedÂup-to-dateÂinformationÂisÂto
Â	80	+ÂÂÂÂ#ÂuseÂfileÂlockingÂprimitives,ÂasÂtheÂlockÂoperationÂwillÂforceÂthe
Â	81	+ÂÂÂÂ#Âgetattr()ÂandÂensureÂnoÂconcurrentÂoperations.
Â	82	+ÂÂÂÂoptionsÂ+=Â',acregmax=1'
Â	83	+ÂÂÂÂoptionsÂ+=Â',acdirmax=1'

noac option suddenly found in XCP 1.6.x as well as in XS 6.x.

I thind develepers can describe why mentionedÂaboveÂpatch was not pulled to final version.

Mentioned above noac option definately didn't present in XS 5.6. I did not check it in XCP 1.4.9.

Hope above will help.

Finally i want to note that you have to edit manually nfs.py on every member of pool.

2013/1/4 Juan Lorenzana <juan@xxxxxxxxxxxxxx>

Okay, so I have two Dell PowerEdge server and we are using with a NAS solution called TrueNAS by IX Systems. This is basically a ZFS version 28 implementation for the NAS solution.

Â

Anyway, when we mount our NFS storage repositories over our 10GB interfaces on the Dell PowerEdge server and the NAS also has 10GB, we get horrible NFS performance. If I mount a CIFS repository, we get blazing fast performance.

Â

So my question is how would I go about tuning the different NFS options to do various tests. A quick scan on Google produced nothing on how to do this. My thoughts are to either pass NFS options in Xen Center under the
Advanced Options input box or edit the nfs.py file. I tried using the Advanced Options but I do not think that worked.

Â

I am currently running XCP 1.5 but plan on migrating to 1.6 in the next few months. I would really like to tune the NFS first on 1.5 so that I can compare performance to 1.6.Â

Â

Any pointers to try to improve NFS performance?

Â

I am getting about a 6MB/s write and 20MB/s read on the NFS storage repository when doing iozone tests inside a VM. With a 10GB interface I should be in the 100âs. If I do iozone test on the TrueNAS directly, I get about 600MB/s on average. When doing iozone testing on VM running Windows using CIFS from the same storage solution, we are getting close to 180 MB/s so I know the issue is not the storage repository, so I am guessing it has to do with NFS tuning.

Any help is appreciated. Thanks.

Â

Juan

Â

Â

_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

--
WBR

Sergey Kruchatov

_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api