Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...

["Christopher Chen" <muffaleta@xxxxxxxxx>]

On Tue, Jan 13, 2009 at 3:37 PM, Ross Walker <rswwalker@xxxxxxxxx> wrote:
> On Jan 13, 2009, at 5:48 PM, "Christopher Chen" <muffaleta@xxxxxxxxx> wrote:
>
>> Hi there!
>>
>> I've been wrestling with an issue for a little bit now--
>>
>> In my test environment, I have tgtd running on a Centos 5.2 box, with
>> a raid 10 array backing it.
>>
>> The initiators are also Centos 5.2 boxes running Xen 3.0.3 userland
>> with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos).
>>
>> Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec reads.
>
> That's kind of lopsided I'd expect it the  other way around.
>
> Is this hardware RAID on the backend with write-back cache?

Yes, there's a raid 10 on the backend, with a battery backed cache.

>> I've attached the iSCSI LUN to the DomU as a virtual block device, and
>> I'm seeing 47MB/sec writes, and 39MB/sec reads.
>
> How did you attach it, what Xen driver did you use phy: or file:?

I used phy--in fact, in all this testing I've been using phy. The Dom0
sees the iSCSI lun, and re-exports that block device to the DomU as a
xvd.

>> I've tried a few things, like running against a local disk, and
>> suprisingly, writes on the DomU are faster than the Dom0--can I assume
>> the writes are buffered by the Dom0.
>
> I'm confused.
>
> I thought you said above you got 110MB/s on dom0 and 45MB/s on the domU?

Not exactly--I said I was seeing 110M/s writes, and 45M/s reads on the Dom0.
I'm seeing 47M/s writes, and 39M/s reads on the DomU.

One more thing I tried was cutting iSCSI out of the picture
altogether, and exporting one of the Dom0's local block devices to the
DomU via phy:xvd. In that case, the write speeds were greater on the
DomU side, which indicate some sort of buffering going on. This makes
sense since the Dom0 has more ram than the DomU.

>
>>
>> I'm going to give a shot doing the initialization from the DomU (just
>> for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads.
>
> You've completely lost me now, what do you mean initialization? Do you mean
> boot domU off of iSCSI directly?

No. In this case, I'm just mounting a iSCSI lun directly in the DomU
with it's initiator. Sorry, I guess initiator != initializer. In that
case, performance is pretty good, because the bridging is fast.

>
>>
>> This is all with bonnie++ -d /mnt -f -u root:root
>>
>> Anyone seen this, or have any ideas?
>>
>> Is any additional latency provided by the xen virtual block device
>> causing a degradation in TCP performance (i.e. a window size or
>> delayed ACK problem) or is the buffering also causing pain? I'm going
>> to keep looking, but I thought I'd ask all of you.
>
> Any layer you add is going to create latency.
>
> If you can be a little more clearer I'm sure an accurate explanation can be
> made.

Well, of course, I know that. What confuses me is what's happening
that that block device abstraction layer that's killing performance. I
think it might be in the buffering, which on a local disk, can
probably provide some gain, but in this case, may be causing the iscsi
target and initiator to throttle the TCP connection.

Sorry if I'm confusing you.

Cheers

cc

-- 
Chris Chen <muffaleta@xxxxxxxxx>
"I want the kind of six pack you can't drink."
-- Micah

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users