[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] 4.2.1: Poor write performance for DomU.



On 21/08/13 02:48, Konrad Rzeszutek Wilk wrote:
On Mon, Mar 25, 2013 at 01:21:09PM +1100, Steven Haigh wrote:
So, based on my tests yesterday, I decided to break the RAID6 and
pull a drive out of it to test directly on the 2Tb drives in
question.

The array in question:
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md2 : active raid6 sdd[4] sdc[0] sde[1] sdf[5]
       3907026688 blocks super 1.2 level 6, 128k chunk, algorithm 2
[4/4] [UUUU]

# mdadm /dev/md2 --fail /dev/sdf
mdadm: set /dev/sdf faulty in /dev/md2
# mdadm /dev/md2 --remove /dev/sdf
mdadm: hot removed /dev/sdf from /dev/md2

So, all tests are to be done on /dev/sdf.
Model Family:     Seagate SV35
Device Model:     ST2000VX000-9YW164
Serial Number:    Z1E17C3X
LU WWN Device Id: 5 000c50 04e1bc6f0
Firmware Version: CV13
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical

 From the Dom0:
# dd if=/dev/zero of=/dev/sdf bs=1M count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 30.7691 s, 140 MB/s

Create a single partition on the drive, and format it with ext4:
Disk /dev/sdf: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x98d8baaf

    Device Boot      Start         End      Blocks   Id  System
/dev/sdf1            2048  3907029167  1953513560   83  Linux

Command (m for help): w

# mkfs.ext4 -j /dev/sdf1
......
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Mount it on the Dom0:
# mount /dev/sdf1 /mnt/esata/
# cd /mnt/esata/
# bonnie++ -d . -u 0:0
....
Version  1.96       ------Sequential Output------ --Sequential
Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr-
--Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP /sec %CP
xenhost.lan.crc. 2G   425  94 133607  24 60544  12   973  95 209114
17 296.4   6
Latency             70971us     190ms     221ms   40369us   17657us
164ms

So from the Dom0: 133Mb/sec write, 209Mb/sec read.

Now, I'll attach the full disk to a DomU:
# xm block-attach zeus.vm phy:/dev/sdf xvdc w

And we'll test from the DomU.

# dd if=/dev/zero of=/dev/xvdc bs=1M count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 32.318 s, 133 MB/s

Partition the same as in the Dom0 and create an ext4 filesystem on it:

I notice something interesting here. In the Dom0, the device is seen as:
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

In the DomU, it is seen as:
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Not sure if this could be related - but continuing testing:
     Device Boot      Start         End      Blocks   Id  System
/dev/xvdc1            2048  3907029167  1953513560   83  Linux

# mkfs.ext4 -j /dev/xvdc1
....
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

# mount /dev/xvdc1 /mnt/esata/
# cd /mnt/esata/
# bonnie++ -d . -u 0:0
....
Version  1.96       ------Sequential Output------ --Sequential
Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr-
--Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP /sec %CP
zeus.crc.id.au   2G   396  99 116530  23 50451  15  1035  99 176407
23 313.4   9
Latency             34615us     130ms     128ms   33316us   74401us
130ms

So still... 116Mb/sec write, 176Mb/sec read to the physical device
from the DomU. More than acceptable.

It leaves me to wonder.... Could there be something in the Dom0
seeing the drives as 4096 byte sectors, but the DomU seeing it as
512 byte sectors cause an issue?

There is certain overhead in it. I still have this in my mailbox
so I am not sure whether this issue got ever resolved? I know that the
indirect patches in Xen blkback and xen blkfront are meant to resolve
some of these issues - by being able to carry a bigger payload.

Did you ever try v3.11 kernel in both dom0 and domU? Thanks.

Hi Konrad,

I don't believe I ever fixed it - however I haven't tried kernel 3.11 in Dom0 OR DomU...

I'll keep this in my inbox and try to build a 3.11 kernel for both in the near future for testing...

--
Steven Haigh

Email: netwiz@xxxxxxxxx
Web: https://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.