Re: [Xen-users] iscsi vs nfs for xen VMs

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Sat, Jan 29, 2011 at 06:12:34PM +0100, Bart Coninckx wrote:
> On 01/29/11 16:30, Pasi Kärkkäinen wrote:
> > On Sat, Jan 29, 2011 at 04:27:52PM +0100, Bart Coninckx wrote:
> >   
> >> On 01/29/11 16:09, Pasi Kärkkäinen wrote:
> >>     
> >>> On Thu, Jan 27, 2011 at 09:35:38AM +0100, Adi Kriegisch wrote:
> >>>   
> >>>       
> >>>> Hi!
> >>>>
> >>>>     
> >>>>         
> >>>>>> iSCSI tipically has a quite big overhead due to the protocol, FC, SAS,
> >>>>>> native infiniband, AoE have very low overhead.
> >>>>>>
> >>>>>>         
> >>>>>>             
> >>>>> For iSCSI vs AoE, that isn't as true as you might think. TCP offload can
> >>>>> take care of a lot of the overhead. Any server class network adapter
> >>>>> these days should allow you to send 60kb packets to the network adapter
> >>>>> and it will take care of the segmentation, while AoE would be limited to
> >>>>> MTU sized packets. With AoE you need to checksum every packet yourself
> >>>>> while with iSCSI it is taken care of by the network adapter.
> >>>>>       
> >>>>>           
> >>>> What AoE actually does is sending a frame per block. Block size is 4K -- 
> >>>> so
> >>>> no need for fragmentation. The overhead is pretty low, because we're
> >>>> talking about Ethernet frames.
> >>>> Most iSCSI issues I have seen are with reordering of packages due to
> >>>> transmission across several interfaces. So what most people recommend is 
> >>>> to
> >>>> keep the number of interfaces to two. To keep performance up this means 
> >>>> you
> >>>> have to use 10G, FC or similar which is quite expensive -- especially if
> >>>> you'd like to have a HA SAN network (HSRP and stuff like that is 
> >>>> required).
> >>>>
> >>>> AoE does not suffer from those issues: Using 6 GBit interfaces is no
> >>>> problem at all, load balancing will happen automatically, as the load is
> >>>> distributed equally across all available interfaces. HA is very simple:
> >>>> just use two switches and connect one half of the interfaces to one 
> >>>> switch
> >>>> and the other half to the other switch. (It is recommended to use 
> >>>> switches
> >>>> that can do jumbo frames and flow control)
> >>>> IMHO most of the current recommendations and practises surrounding iSCSI
> >>>> are there to overcome the shortcomings of the protocol. AoE is way more
> >>>> robust and easier to handle.
> >>>>
> >>>>     
> >>>>         
> >>> iSCSI does not have problems using multiple gige interfaces.
> >>> Just setup multipathing properly.
> >>>
> >>> -- Pasi
> >>>
> >>>
> >>> _______________________________________________
> >>> Xen-users mailing list
> >>> Xen-users@xxxxxxxxxxxxxxxxxxx
> >>> http://lists.xensource.com/xen-users
> >>>   
> >>>       
> >> On this subject: am using multipathing to iSCSI too, hoping to have
> >> aggregated speed on top of path redundancy but the speed seems not to
> >> surpass the one of a single interface.
> >>
> >> Is anyone successful at doing this?
> >>
> >>     
> > You're benchmarking sequential/linear IO, using big blocksizes, right?
> >
> > Some questions:
> >     - Are you using multipath round robin path policy?
> >     - After how many IOs do you switch paths? You might need to lower the 
> > min_ios.
> >
> >
> > -- Pasi
> >
> >   
> Hi Pasi,
> 
> the benchmarking was intuitively done, with just dd and bonnie++.
> 

with dd, use "bs=1024k" or so to force big blocks.

> It is indeed rr, this is a part of my multipath.conf:
> 
> defaults {
>         udev_dir                /dev
>         polling_interval        10
>         selector                "round-robin 0"
>         path_grouping_policy    multibus
>         getuid_callout          "/lib/udev/scsi_id --whitelisted
> --device=/dev/%n"
>         prio                    const
>         path_checker            directio
>         rr_min_io               100
>         max_fds                 8192
>         rr_weight               priorities
>         failback                immediate
>         no_path_retry           5
>         user_friendly_names     no
> }
> 
> should the "100" go down a bit?
> 

Yeah, for example Equallogic enterprise iSCSI storage
manuals recommends rr_min_io of "3".

That should allow utilizing both paths.

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users