Re: [Xen-users] iscsi vs nfs for xen VMs

[Bart Coninckx <bart.coninckx@xxxxxxxxxx>]

On 01/29/11 18:15, Pasi Kärkkäinen wrote:
> On Sat, Jan 29, 2011 at 06:12:34PM +0100, Bart Coninckx wrote:
>   
>> On 01/29/11 16:30, Pasi Kärkkäinen wrote:
>>     
>>> On Sat, Jan 29, 2011 at 04:27:52PM +0100, Bart Coninckx wrote:
>>>   
>>>       
>>>> On 01/29/11 16:09, Pasi Kärkkäinen wrote:
>>>>     
>>>>         
>>>>> On Thu, Jan 27, 2011 at 09:35:38AM +0100, Adi Kriegisch wrote:
>>>>>   
>>>>>       
>>>>>           
>>>>>> Hi!
>>>>>>
>>>>>>     
>>>>>>         
>>>>>>             
>>>>>>>> iSCSI tipically has a quite big overhead due to the protocol, FC, SAS,
>>>>>>>> native infiniband, AoE have very low overhead.
>>>>>>>>
>>>>>>>>         
>>>>>>>>             
>>>>>>>>                 
>>>>>>> For iSCSI vs AoE, that isn't as true as you might think. TCP offload can
>>>>>>> take care of a lot of the overhead. Any server class network adapter
>>>>>>> these days should allow you to send 60kb packets to the network adapter
>>>>>>> and it will take care of the segmentation, while AoE would be limited to
>>>>>>> MTU sized packets. With AoE you need to checksum every packet yourself
>>>>>>> while with iSCSI it is taken care of by the network adapter.
>>>>>>>       
>>>>>>>           
>>>>>>>               
>>>>>> What AoE actually does is sending a frame per block. Block size is 4K -- 
>>>>>> so
>>>>>> no need for fragmentation. The overhead is pretty low, because we're
>>>>>> talking about Ethernet frames.
>>>>>> Most iSCSI issues I have seen are with reordering of packages due to
>>>>>> transmission across several interfaces. So what most people recommend is 
>>>>>> to
>>>>>> keep the number of interfaces to two. To keep performance up this means 
>>>>>> you
>>>>>> have to use 10G, FC or similar which is quite expensive -- especially if
>>>>>> you'd like to have a HA SAN network (HSRP and stuff like that is 
>>>>>> required).
>>>>>>
>>>>>> AoE does not suffer from those issues: Using 6 GBit interfaces is no
>>>>>> problem at all, load balancing will happen automatically, as the load is
>>>>>> distributed equally across all available interfaces. HA is very simple:
>>>>>> just use two switches and connect one half of the interfaces to one 
>>>>>> switch
>>>>>> and the other half to the other switch. (It is recommended to use 
>>>>>> switches
>>>>>> that can do jumbo frames and flow control)
>>>>>> IMHO most of the current recommendations and practises surrounding iSCSI
>>>>>> are there to overcome the shortcomings of the protocol. AoE is way more
>>>>>> robust and easier to handle.
>>>>>>
>>>>>>     
>>>>>>         
>>>>>>             
>>>>> iSCSI does not have problems using multiple gige interfaces.
>>>>> Just setup multipathing properly.
>>>>>
>>>>> -- Pasi
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Xen-users mailing list
>>>>> Xen-users@xxxxxxxxxxxxxxxxxxx
>>>>> http://lists.xensource.com/xen-users
>>>>>   
>>>>>       
>>>>>           
>>>> On this subject: am using multipathing to iSCSI too, hoping to have
>>>> aggregated speed on top of path redundancy but the speed seems not to
>>>> surpass the one of a single interface.
>>>>
>>>> Is anyone successful at doing this?
>>>>
>>>>     
>>>>         
>>> You're benchmarking sequential/linear IO, using big blocksizes, right?
>>>
>>> Some questions:
>>>     - Are you using multipath round robin path policy?
>>>     - After how many IOs do you switch paths? You might need to lower the 
>>> min_ios.
>>>
>>>
>>> -- Pasi
>>>
>>>   
>>>       
>> Hi Pasi,
>>
>> the benchmarking was intuitively done, with just dd and bonnie++.
>>
>>     
> with dd, use "bs=1024k" or so to force big blocks.
>
>   
>> It is indeed rr, this is a part of my multipath.conf:
>>
>> defaults {
>>         udev_dir                /dev
>>         polling_interval        10
>>         selector                "round-robin 0"
>>         path_grouping_policy    multibus
>>         getuid_callout          "/lib/udev/scsi_id --whitelisted
>> --device=/dev/%n"
>>         prio                    const
>>         path_checker            directio
>>         rr_min_io               100
>>         max_fds                 8192
>>         rr_weight               priorities
>>         failback                immediate
>>         no_path_retry           5
>>         user_friendly_names     no
>> }
>>
>> should the "100" go down a bit?
>>
>>     
> Yeah, for example Equallogic enterprise iSCSI storage
> manuals recommends rr_min_io of "3".
>
> That should allow utilizing both paths.
>
> -- Pasi
>
>   
Excellent, will try this and start measuring again ...

cheers,

B.


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users