Re: [Xen-users] iscsi vs nfs for xen VMs

[Bart Coninckx <bart.coninckx@xxxxxxxxxx>]

On 01/29/11 16:30, Pasi Kärkkäinen wrote:
> On Sat, Jan 29, 2011 at 04:27:52PM +0100, Bart Coninckx wrote:
>   
>> On 01/29/11 16:09, Pasi Kärkkäinen wrote:
>>     
>>> On Thu, Jan 27, 2011 at 09:35:38AM +0100, Adi Kriegisch wrote:
>>>   
>>>       
>>>> Hi!
>>>>
>>>>     
>>>>         
>>>>>> iSCSI tipically has a quite big overhead due to the protocol, FC, SAS,
>>>>>> native infiniband, AoE have very low overhead.
>>>>>>
>>>>>>         
>>>>>>             
>>>>> For iSCSI vs AoE, that isn't as true as you might think. TCP offload can
>>>>> take care of a lot of the overhead. Any server class network adapter
>>>>> these days should allow you to send 60kb packets to the network adapter
>>>>> and it will take care of the segmentation, while AoE would be limited to
>>>>> MTU sized packets. With AoE you need to checksum every packet yourself
>>>>> while with iSCSI it is taken care of by the network adapter.
>>>>>       
>>>>>           
>>>> What AoE actually does is sending a frame per block. Block size is 4K -- so
>>>> no need for fragmentation. The overhead is pretty low, because we're
>>>> talking about Ethernet frames.
>>>> Most iSCSI issues I have seen are with reordering of packages due to
>>>> transmission across several interfaces. So what most people recommend is to
>>>> keep the number of interfaces to two. To keep performance up this means you
>>>> have to use 10G, FC or similar which is quite expensive -- especially if
>>>> you'd like to have a HA SAN network (HSRP and stuff like that is required).
>>>>
>>>> AoE does not suffer from those issues: Using 6 GBit interfaces is no
>>>> problem at all, load balancing will happen automatically, as the load is
>>>> distributed equally across all available interfaces. HA is very simple:
>>>> just use two switches and connect one half of the interfaces to one switch
>>>> and the other half to the other switch. (It is recommended to use switches
>>>> that can do jumbo frames and flow control)
>>>> IMHO most of the current recommendations and practises surrounding iSCSI
>>>> are there to overcome the shortcomings of the protocol. AoE is way more
>>>> robust and easier to handle.
>>>>
>>>>     
>>>>         
>>> iSCSI does not have problems using multiple gige interfaces.
>>> Just setup multipathing properly.
>>>
>>> -- Pasi
>>>
>>>
>>> _______________________________________________
>>> Xen-users mailing list
>>> Xen-users@xxxxxxxxxxxxxxxxxxx
>>> http://lists.xensource.com/xen-users
>>>   
>>>       
>> On this subject: am using multipathing to iSCSI too, hoping to have
>> aggregated speed on top of path redundancy but the speed seems not to
>> surpass the one of a single interface.
>>
>> Is anyone successful at doing this?
>>
>>     
> You're benchmarking sequential/linear IO, using big blocksizes, right?
>
> Some questions:
>       - Are you using multipath round robin path policy?
>       - After how many IOs do you switch paths? You might need to lower the 
> min_ios.
>
>
> -- Pasi
>
>   
Hi Pasi,

the benchmarking was intuitively done, with just dd and bonnie++.

It is indeed rr, this is a part of my multipath.conf:

defaults {
        udev_dir                /dev
        polling_interval        10
        selector                "round-robin 0"
        path_grouping_policy    multibus
        getuid_callout          "/lib/udev/scsi_id --whitelisted
--device=/dev/%n"
        prio                    const
        path_checker            directio
        rr_min_io               100
        max_fds                 8192
        rr_weight               priorities
        failback                immediate
        no_path_retry           5
        user_friendly_names     no
}

should the "100" go down a bit?

thx,

bart


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users