[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-users] read from FC lun under Xen 4 - very strange behavior
Dear Xen Users This is my first post, so sorry for maybe newbie stupid query. Hardware config is the following: ------------------------------- Server: Dell PE R815 192GB RAM, 4x 600GB SAS HD, PERC 700 internal raid controler, EMULEX LPE12002-E FC 8Gbit dual port HBA SAN: 2x Cisco MDS 9148 + OM2/O3 cables (short <3m) Storage: tested 2 arrays: Dell MD3620F & Netapp FAS 3240 with FC Software: --------- mainly Debian Squeeze & Debian Wheezy (Testing) + Debian Xen packs 4.0 & 4.1 ========Problem======================= LUNs are visible and accessible on Dom0. I can even configure DM Multipathing (Dell requires RDAC, but it work). Write performance is fine i.e. stable e.g. on Dell LUN with RAID10 on 8 disks around 250MBytes/s (streaming write with block size 32K: dd if=/dev/zero of=lum_mount_point/file bs=32k). Write performace is similar with and without Xen hypervisor. THE PROBLEM is read performace. Without Xen hypervisor read performance is stable and high e.g. streaming reading from Dell LUN with Xen with speed about 420MBytes/s (under iostat I can see speed in a range from 390MBytes/s up to 450MBytes/s) With Xen supervisor read performance is not stable and changes randomly. Sometimes it is up to 450MBytes but sometimes as low as 16MBytes/s. The low performance does not depend on file size, block size. The same file reads once with high speed, after umount/mount reads with very low speed, and after next umount/mount starts with very low, then switches to very high for few seconds, then again very low. When high speed is in effect the queue lenth is about 256, but with low speed queue drops down to about 8 or 16 ... Strangely number of IOPS seen from linux as well as from array side with low speed is higher then with high speed (e.g. 4500 vs 3000 IOPS). This is not directly DM Multipathing related. I obtain the similar relults when accessing /dev/sdX device with multipathing off (multipath -F, multipath-tools stop). It looks like Xen 4.1.2 with newer kernels (3.1 or 3.2) performs better than default squeeze 2.6.32-5-amd64-xen) but still not stable. Here is example from iostat monitoring: dev2 #> dd if=/mnt/2T/b of=/dev/null bs=32k Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sde 0.00 0.00 13.38 0.00 519.00 0.09 77.57 0.00 0.26 0.21 0.28 sde 0.00 0.00 4731.00 0.00 18924.00 0.00 8.00 0.85 0.18 0.18 84.80 sde 0.00 0.00 4680.00 0.00 18720.00 0.00 8.00 0.87 0.19 0.19 87.20 sde 0.00 0.00 4432.00 0.00 17728.00 0.00 8.00 0.80 0.19 0.18 80.40 sde 0.00 0.00 4763.00 0.00 19052.00 0.00 8.00 0.86 0.18 0.18 86.40 sde 0.00 0.00 4580.00 0.00 18320.00 0.00 8.00 0.84 0.19 0.18 83.60 sde 0.00 0.00 4732.00 0.00 18928.00 0.00 8.00 0.88 0.19 0.19 87.60 sde 0.00 0.00 4729.00 0.00 18916.00 0.00 8.00 0.87 0.19 0.18 86.80 sde 0.00 0.00 4538.00 0.00 18152.00 0.00 8.00 0.86 0.19 0.19 86.40 sde 0.00 0.00 4704.00 0.00 18816.00 0.00 8.00 0.86 0.18 0.18 86.00 sde 0.00 0.00 4700.00 0.00 18800.00 0.00 8.00 0.89 0.19 0.19 89.20 sde 0.00 0.00 4715.00 0.00 18860.00 0.00 8.00 0.90 0.19 0.19 89.60 sde 0.00 0.00 4642.00 0.00 18568.00 0.00 8.00 0.88 0.19 0.19 87.60 sde 0.00 0.00 4547.00 0.00 18188.00 0.00 8.00 0.86 0.19 0.19 86.00 sde 0.00 0.00 4608.00 0.00 18432.00 0.00 8.00 0.86 0.19 0.19 86.40 sde 0.00 0.00 4594.00 0.00 18376.00 0.00 8.00 0.83 0.18 0.18 82.80 sde 0.00 0.00 4552.00 0.00 18208.00 0.00 8.00 0.87 0.19 0.19 87.20 sde 0.00 0.00 4628.00 0.00 18512.00 0.00 8.00 0.85 0.19 0.18 84.80 sde 0.00 0.00 4462.00 0.00 17848.00 0.00 8.00 0.84 0.19 0.19 84.40 sde 0.00 0.00 4586.00 0.00 18348.00 0.00 8.00 0.83 0.18 0.18 83.20 sde 0.00 0.00 4528.71 0.00 18110.89 0.00 8.00 0.81 0.18 0.18 81.19 sde 0.00 0.00 4637.00 0.00 18548.00 0.00 8.00 0.89 0.19 0.19 88.80 sde 0.00 0.00 4619.00 0.00 18476.00 0.00 8.00 0.84 0.18 0.18 83.60 sde 0.00 0.00 4555.00 0.00 18220.00 0.00 8.00 0.88 0.19 0.19 88.00 sde 0.00 0.00 4579.00 0.00 18316.00 0.00 8.00 0.85 0.19 0.19 85.20 sde 0.00 0.00 4760.00 0.00 19040.00 0.00 8.00 0.88 0.19 0.19 88.40 sde 0.00 0.00 5342.00 0.00 21368.00 0.00 8.00 0.76 0.14 0.14 75.60 sde 0.00 0.00 5342.00 0.00 21368.00 0.00 8.00 0.72 0.14 0.13 72.00 sde 0.00 0.00 4384.00 0.00 94048.00 0.00 42.91 0.91 0.21 0.19 84.80 sde 0.00 0.00 3370.00 0.00 431360.00 0.00 256.00 1.25 0.38 0.26 86.80 sde 0.00 0.00 3434.00 0.00 439552.00 0.00 256.00 1.28 0.38 0.27 92.40 sde 0.00 0.00 3534.00 0.00 452352.00 0.00 256.00 1.30 0.37 0.27 96.40 sde 0.00 0.00 3540.00 0.00 453120.00 0.00 256.00 1.34 0.38 0.28 99.60 sde 0.00 0.00 3338.00 0.00 427264.00 0.00 256.00 1.36 0.41 0.29 95.20 sde 0.00 0.00 3405.00 0.00 435840.00 0.00 256.00 1.28 0.38 0.28 94.40 sde 0.00 0.00 3414.00 0.00 436992.00 0.00 256.00 1.36 0.40 0.28 95.60 sde 0.00 0.00 3409.00 0.00 436352.00 0.00 256.00 1.31 0.39 0.27 92.40 sde 0.00 0.00 3403.00 0.00 435584.00 0.00 256.00 1.34 0.39 0.28 93.60 sde 0.00 0.00 3238.00 0.00 414464.00 0.00 256.00 1.21 0.38 0.27 88.80 sde 0.00 0.00 3322.00 0.00 425216.00 0.00 256.00 1.29 0.39 0.26 88.00 sde 0.00 0.00 3316.00 0.00 424448.00 0.00 256.00 1.25 0.38 0.27 89.20 sde 0.00 0.00 3313.00 0.00 424064.00 0.00 256.00 1.21 0.37 0.26 86.80 sde 0.00 0.00 3318.00 0.00 424704.00 0.00 256.00 1.20 0.36 0.25 82.80 sde 0.00 0.00 3179.00 0.00 406912.00 0.00 256.00 1.24 0.39 0.27 84.80 sde 0.00 0.00 3283.00 0.00 420224.00 0.00 256.00 1.19 0.37 0.25 82.40 sde 0.00 0.00 3287.00 0.00 420736.00 0.00 256.00 1.12 0.34 0.24 80.40 sde 0.00 0.00 3308.00 0.00 423424.00 0.00 256.00 1.21 0.37 0.26 84.40 sde 0.00 0.00 3302.00 0.00 422656.00 0.00 256.00 1.27 0.39 0.26 86.40 sde 0.00 0.00 3135.00 0.00 401280.00 0.00 256.00 1.26 0.41 0.27 86.00 sde 0.00 0.00 3259.00 0.00 417152.00 0.00 256.00 1.38 0.43 0.28 91.20 sde 0.00 0.00 3250.00 0.00 416000.00 0.00 256.00 1.50 0.47 0.29 93.60 sde 0.00 0.00 3258.00 0.00 417024.00 0.00 256.00 1.35 0.41 0.28 91.60 sde 0.00 0.00 3247.00 0.00 415616.00 0.00 256.00 1.43 0.45 0.29 92.80 sde 0.00 0.00 3117.00 0.00 398976.00 0.00 256.00 1.33 0.43 0.29 89.20 sde 0.00 0.00 3217.00 0.00 411776.00 0.00 256.00 1.34 0.42 0.27 88.40 sde 0.00 0.00 3216.00 0.00 411648.00 0.00 256.00 1.20 0.38 0.27 86.00 sde 0.00 0.00 3215.00 0.00 411520.00 0.00 256.00 1.24 0.39 0.27 87.20 sde 0.00 0.00 3214.00 0.00 411392.00 0.00 256.00 1.21 0.38 0.26 84.40 sde 0.00 0.00 3092.00 0.00 395776.00 0.00 256.00 1.24 0.40 0.27 84.80 sde 0.00 0.00 3168.00 0.00 405504.00 0.00 256.00 1.22 0.39 0.28 87.60 sde 0.00 0.00 3123.00 0.00 399744.00 0.00 256.00 1.23 0.39 0.28 86.40 sde 0.00 0.00 3122.00 0.00 399616.00 0.00 256.00 1.26 0.40 0.28 88.80 sde 0.00 0.00 3112.00 0.00 398336.00 0.00 256.00 1.18 0.38 0.27 83.60 sde 0.00 0.00 2971.00 0.00 380288.00 0.00 256.00 1.24 0.42 0.29 87.60 sde 0.00 0.00 3060.00 0.00 391680.00 0.00 256.00 1.25 0.41 0.30 90.80 I did not try to access a lun from DomU Please suggest what is wrong ??? How to proceed ? GB _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxx http://lists.xen.org/xen-users
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |