[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...


  • To: "Barry van Someren" <barry@xxxxxxxxxxxxxxxx>
  • From: "Christopher Chen" <muffaleta@xxxxxxxxx>
  • Date: Tue, 13 Jan 2009 15:35:23 -0800
  • Cc: xen-users@xxxxxxxxxxxxxxxxxxx
  • Delivery-date: Tue, 13 Jan 2009 15:36:30 -0800
  • Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=ovViXUDjhm/4jI0rYo6xKIWMdRZJYZZW0suE2SaPmsiz14Zt9LclkyK9rbARVkJNlH dvj9XG3US0Pl9JC8nMDo+EBd/hmzhSmxXk+ixZTsKz6Kd0rT8tywFzg7O9MzTWy2w5sO Kyhv0Cf7o8SxGWY1YXpmEm8BgVUQN3+iPAR7o=
  • List-id: Xen user discussion <xen-users.lists.xensource.com>

Here's my data so far:

I'm just wondering what the block device layer is doing to kill
performance. I'm looking for any buffering directives in the xend
config that I may be able to modify.

iSCSI Performance Profiling
iSCSI target on Heart, Initiator is Dom0 br04-61 as LV

Version  1.94       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
br04-61.adtech2. 8G           110289  29 33781   3           45053   1  1052   9
Latency                        2610ms     312ms               191ms     291ms
Version  1.94       ------Sequential Create------ --------Random Create--------
br04-61 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency               627us     157us     235us     506us     183us     282us
1.93c,1.94,br04-61,1,1231849837,8G,,,,110289,29,33781,3,,,450s

iSCSI target on Heart, Initiator on Dom0 br04-61 as LV, bonnie++ on
DomU aenu via xvd

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
aenu 4G           47100   3 15069   0           39072   0  3971   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
aenu,4G,,,47100,3,15069,0,,,39072,0,3971.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

Local Logical Volume on Dom0 br04-61, bonnie++ on Dom0

Version  1.94       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
br04-61. 8G           39000   7 21419   0           62958   0 158.0   1
Latency                       16524ms    1111ms             54552us     565ms
Version  1.94       ------Sequential Create------ --------Random Create--------
br04-61 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 29249  40 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             22439us     281us     227us     589us     220us     253us
1.93c,1.94,br04-61,1,1231848192,8G,,,,39000,7,21419,0,,,62958,0,158.0,1,16,,,,,29249,40,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,16524ms,1111ms,,54552us,565ms,22439us,281us,227us,589us,220us,253us

Local Logical Volume on Dom0 br04-61, bonnie++ on DomU aenu via xvd

RAM was reduced to 512M on the DomU

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
aenu 1G           64860   5 23617   0           58329   0 212.3   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  6038  96 +++++ +++ +++++ +++  6333  99 +++++ +++ 19371 100
aenu,1G,,,64860,5,23617,0,,,58329,0,212.3,0,16,6038,96,+++++,+++,+++++,+++,6333,99,+++++,+++,19371,10

iSCSI target on Heart, Initiator on Dom0 br04-61 as /dev/sd, bonnie++ on br04-61

Version  1.94       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
br04-61.adtech2. 8G           111523  30 34132   3           58284   3  1184  11
Latency                        4579ms     250ms             72926us   44558us
Version  1.94       ------Sequential Create------ --------Random Create--------
br04-61 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              4643us     123us     191us    7013us     153us     195us
1.93c,1.94,br04-61,1,1231852980,8G,,,,111523,30,34132,3,,,58284,3,1184,11,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,4579ms,250ms,,72926us,44558us,4643us,123us,191us,7013us,153us,195us

iSCSI target on Heart, Initiator on Dom0 br04-61 as /dev/sd, bonnie++
on aenu domU as xvd

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
aenu 4G           52625   4 15065   0           46256   0  3859   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
aenu,4G,,,52625,4,15065,0,,,46256,0,3859.1,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

iSCSI target on Heart, Initiator on DomU aenu, bonnie++ on aenu DomU as /dev/sd

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
aenu 4G           129048  22 28832   2           49896   0  5174   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
aenu,4G,,,129048,22,28832,2,,,49896,0,5173.6,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

On Tue, Jan 13, 2009 at 3:17 PM, Barry van Someren
<barry@xxxxxxxxxxxxxxxx> wrote:
> Hi there,
>
> I've used Citrix Xenserver 5.0 before this and never experienced this problem.
> I did however store the entire VM on the LUN instead of mounting it to the VM.
>
> Maybe you can trying doing that and seeing if it makes a big difference?
>
> Regards,
>
> Barry
>
> (Failed to send this to the entire list, so sending again, sorry)
>
> On Tue, Jan 13, 2009 at 11:48 PM, Christopher Chen <muffaleta@xxxxxxxxx> 
> wrote:
>> Hi there!
>>
>> I've been wrestling with an issue for a little bit now--
>>
>> In my test environment, I have tgtd running on a Centos 5.2 box, with
>> a raid 10 array backing it.
>>
>> The initiators are also Centos 5.2 boxes running Xen 3.0.3 userland
>> with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos).
>>
>> Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec reads.
>>
>> I've attached the iSCSI LUN to the DomU as a virtual block device, and
>> I'm seeing 47MB/sec writes, and 39MB/sec reads.
>>
>> I've tried a few things, like running against a local disk, and
>> suprisingly, writes on the DomU are faster than the Dom0--can I assume
>> the writes are buffered by the Dom0.
>>
>> I'm going to give a shot doing the initialization from the DomU (just
>> for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads.
>>
>> This is all with bonnie++ -d /mnt -f -u root:root
>>
>> Anyone seen this, or have any ideas?
>>
>> Is any additional latency provided by the xen virtual block device
>> causing a degradation in TCP performance (i.e. a window size or
>> delayed ACK problem) or is the buffering also causing pain? I'm going
>> to keep looking, but I thought I'd ask all of you.
>>
>> Cheers
>>
>> cc
>>
>> --
>> Chris Chen <muffaleta@xxxxxxxxx>
>> "I want the kind of six pack you can't drink."
>> -- Micah
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@xxxxxxxxxxxxxxxxxxx
>> http://lists.xensource.com/xen-users
>>
>
>
>
> --
> Barry van Someren
> ---------------------------------------
> Email: barry@xxxxxxxxxxxxxxx
> Email: goltharnl@xxxxxxxxx
> Linked in: http://www.linkedin.com/in/barryvansomeren
>



-- 
Chris Chen <muffaleta@xxxxxxxxx>
"I want the kind of six pack you can't drink."
-- Micah

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.