Xen project Mailing List

Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...

To: "Barry van Someren" <barry@xxxxxxxxxxxxxxxx>

From: "Christopher Chen" <muffaleta@xxxxxxxxx>

Date: Tue, 13 Jan 2009 15:35:23 -0800

Delivery-date: Tue, 13 Jan 2009 15:36:30 -0800

Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=ovViXUDjhm/4jI0rYo6xKIWMdRZJYZZW0suE2SaPmsiz14Zt9LclkyK9rbARVkJNlH dvj9XG3US0Pl9JC8nMDo+EBd/hmzhSmxXk+ixZTsKz6Kd0rT8tywFzg7O9MzTWy2w5sO Kyhv0Cf7o8SxGWY1YXpmEm8BgVUQN3+iPAR7o=

List-id: Xen user discussion <xen-users.lists.xensource.com>

Here's my data so far: I'm just wondering what the block device layer is doing to kill performance. I'm looking for any buffering directives in the xend config that I may be able to modify. iSCSI Performance Profiling iSCSI target on Heart, Initiator is Dom0 br04-61 as LV Version 1.94 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP br04-61.adtech2. 8G 110289 29 33781 3 45053 1 1052 9 Latency 2610ms 312ms 191ms 291ms Version 1.94 ------Sequential Create------ --------Random Create-------- br04-61 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency 627us 157us 235us 506us 183us 282us 1.93c,1.94,br04-61,1,1231849837,8G,,,,110289,29,33781,3,,,450s iSCSI target on Heart, Initiator on Dom0 br04-61 as LV, bonnie++ on DomU aenu via xvd Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP aenu 4G 47100 3 15069 0 39072 0 3971 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ aenu,4G,,,47100,3,15069,0,,,39072,0,3971.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ Local Logical Volume on Dom0 br04-61, bonnie++ on Dom0 Version 1.94 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP br04-61. 8G 39000 7 21419 0 62958 0 158.0 1 Latency 16524ms 1111ms 54552us 565ms Version 1.94 ------Sequential Create------ --------Random Create-------- br04-61 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 29249 40 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency 22439us 281us 227us 589us 220us 253us 1.93c,1.94,br04-61,1,1231848192,8G,,,,39000,7,21419,0,,,62958,0,158.0,1,16,,,,,29249,40,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,16524ms,1111ms,,54552us,565ms,22439us,281us,227us,589us,220us,253us Local Logical Volume on Dom0 br04-61, bonnie++ on DomU aenu via xvd RAM was reduced to 512M on the DomU Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP aenu 1G 64860 5 23617 0 58329 0 212.3 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6038 96 +++++ +++ +++++ +++ 6333 99 +++++ +++ 19371 100 aenu,1G,,,64860,5,23617,0,,,58329,0,212.3,0,16,6038,96,+++++,+++,+++++,+++,6333,99,+++++,+++,19371,10 iSCSI target on Heart, Initiator on Dom0 br04-61 as /dev/sd, bonnie++ on br04-61 Version 1.94 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP br04-61.adtech2. 8G 111523 30 34132 3 58284 3 1184 11 Latency 4579ms 250ms 72926us 44558us Version 1.94 ------Sequential Create------ --------Random Create-------- br04-61 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency 4643us 123us 191us 7013us 153us 195us 1.93c,1.94,br04-61,1,1231852980,8G,,,,111523,30,34132,3,,,58284,3,1184,11,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,4579ms,250ms,,72926us,44558us,4643us,123us,191us,7013us,153us,195us iSCSI target on Heart, Initiator on Dom0 br04-61 as /dev/sd, bonnie++ on aenu domU as xvd Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP aenu 4G 52625 4 15065 0 46256 0 3859 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ aenu,4G,,,52625,4,15065,0,,,46256,0,3859.1,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ iSCSI target on Heart, Initiator on DomU aenu, bonnie++ on aenu DomU as /dev/sd Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP aenu 4G 129048 22 28832 2 49896 0 5174 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ aenu,4G,,,129048,22,28832,2,,,49896,0,5173.6,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ On Tue, Jan 13, 2009 at 3:17 PM, Barry van Someren <barry@xxxxxxxxxxxxxxxx> wrote: > Hi there, > > I've used Citrix Xenserver 5.0 before this and never experienced this problem. > I did however store the entire VM on the LUN instead of mounting it to the VM. > > Maybe you can trying doing that and seeing if it makes a big difference? > > Regards, > > Barry > > (Failed to send this to the entire list, so sending again, sorry) > > On Tue, Jan 13, 2009 at 11:48 PM, Christopher Chen <muffaleta@xxxxxxxxx> > wrote: >> Hi there! >> >> I've been wrestling with an issue for a little bit now-- >> >> In my test environment, I have tgtd running on a Centos 5.2 box, with >> a raid 10 array backing it. >> >> The initiators are also Centos 5.2 boxes running Xen 3.0.3 userland >> with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos). >> >> Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec reads. >> >> I've attached the iSCSI LUN to the DomU as a virtual block device, and >> I'm seeing 47MB/sec writes, and 39MB/sec reads. >> >> I've tried a few things, like running against a local disk, and >> suprisingly, writes on the DomU are faster than the Dom0--can I assume >> the writes are buffered by the Dom0. >> >> I'm going to give a shot doing the initialization from the DomU (just >> for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads. >> >> This is all with bonnie++ -d /mnt -f -u root:root >> >> Anyone seen this, or have any ideas? >> >> Is any additional latency provided by the xen virtual block device >> causing a degradation in TCP performance (i.e. a window size or >> delayed ACK problem) or is the buffering also causing pain? I'm going >> to keep looking, but I thought I'd ask all of you. >> >> Cheers >> >> cc >> >> -- >> Chris Chen <muffaleta@xxxxxxxxx> >> "I want the kind of six pack you can't drink." >> -- Micah >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@xxxxxxxxxxxxxxxxxxx >> http://lists.xensource.com/xen-users >> > > > > -- > Barry van Someren > --------------------------------------- > Email: barry@xxxxxxxxxxxxxxx > Email: goltharnl@xxxxxxxxx > Linked in: http://www.linkedin.com/in/barryvansomeren > -- Chris Chen <muffaleta@xxxxxxxxx> "I want the kind of six pack you can't drink." -- Micah _______________________________________________ Xen-users mailing list Xen-users@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-users

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.