Re: [Xen-users] lenny amd64 and xen.

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Fri, Nov 28, 2008 at 12:12:01AM +0100, Thomas Halinka wrote:
> Hi Pasi,
> 
> Am Donnerstag, den 27.11.2008, 21:25 +0200 schrieb Pasi Kärkkäinen:
> > On Thu, Nov 27, 2008 at 05:10:30PM +0100, Thomas Halinka wrote:
> > > Am Samstag, den 22.11.2008, 20:19 +0400 schrieb Nicolas Ruiz:
> > > > Hi all,
> > > >  
> > > > I'm pretty interesting study this kind of solution for my office.
> > > > Following questions from Pasi, i would like to know from you, Thomas,
> > > > if you are using a SAN for your cluster.
> > > 
> > > i build up my own SAN with mdadm, lvm und vblade.
> > > 
> > > > If so, what kind of data access technologies you use with.
> > > 
> > > ATAoverEthernet, which sends ATA-commands over Ethernet (Layer2). It's
> > > something like SAN over Ethernet and much faster than iscsi, since no
> > > tcp/ip is used. also failover was very tricky with iscsi....
> > > 
> > 
> > OK. 
> > 
> > > 
> > > >  
> > > > Last question, how do you manage HA, Live migration and snapshots :
> > > > owned scripts ?
> > > 
> > > heartbeat2 with crm and constraints and the rest is managed through
> > > openqrm.
> > >
> > 
> > Hmm.. so openqrm can take care of locking domU disks in dom0's for live
> > migration? ie. making sure only single dom0 accesses domU disks at a time.. 
> >  
> > > >         
> > > >         This is interesting. Want to tell more about your setup? CLVM?
> > > >         iSCSI?
> > > 
> > > nope, just AoE and LVM
> > > 
> > > >         
> > > >         What kind of physical server hardware? What kind of storage?
> > > 
> > > It s self-build. We had evaluated FC-SAN-Solutions, but they were slow,
> > > unflexible and very expensive. We 're using Standard-Server with bonding
> > > over 10Gbit-NICs
> > > 
> > > This setup transfers 1300 MB/s at the moment, is highly scaleable and
> > > was about 70% cheaper than a FC-Solution.
> > > 
> > 
> > Ok. 
> > 
> > > >         
> > > >         What exact kernel and Xen versions?
> > > 
> > > at the moment its xen 3.2 and 2.6.18-Kernel. I am evaluating 3.3 and
> > > 2.6.26 atm.
> > > 
> > > >         
> > > >         Thanks!
> > > >         
> > > >         -- Pasi
> > > 
> > > 
> > > If interested in this Setup, i could get you a overview with a small
> > > abstract, what is managed where and why... you know ;)
> > > 
> > 
> > Yeah.. picture would be nice :) 
> 
> you can get it here: http://openqrm.com/storage-cluster.png
> 
> Some words to say:
> 
> - openqrm-server has mdadm started and sees all mdX-Devices
> - openqrm-server knows a vg "data" with all mdX-Devices inside
> - openqrm-server exports "lvol" to the LAN
> - openqrm-server provides a boot-service (pxe), which: deploys a
> XEN-Image to xen_1-X and puts this ressource into a puppet-class
> 
> in this xen-image is heartbeat2 with crm and constraints implemented.
> puppet only alters the config for me...
> 
> Some explanations:
> 
> - all the storage-boxes are standard-server with 4xGB-NICs, 24-SATA on
> Areca Raid6 (areca is impressive, since write-performance of raid 5 =
> raid 6 = raid 0). Only small OS and the rest of HDD is exported through
> vblade.
> - header_a und b is heartbeat v1 cluster with drbd. drbd mirrors the
> data for openqrm and heartbeat does HA for openqrm
> - openqrm itself is the storage-header exporting all the data from the
> storage-boxes to the clients
> - openqrm-boot-service deploys a xen-image and puppet-configuration to
> this xen-servers.
> - all xen-server see all vblades and shelfes
> - xen-vms resist on aoe-blades, so snapshotting, lvextend, resize2fs is
> possible online
> 
> Scalability:
> Storage: go buy 3 new server, put a bunch of harddisk inside, install
> linux, install vblade and fire them. on the openqrm-server you only have
> to create a new-md and extend the volume-group
> Performance: buy a new Server and let him pxe-boot, create a new
> appliance and watch your server rebooting, starting xen and
> participating the cluster.
> 
> We Started the Cluster with about 110 GB-Storage - at the moment we have
> about 430 GB Data and have to extend up to 1,2 PB in Summer 2009, which
> is no problem.
> 
> 
> No - go and search for a SAN-Solution like this and ask for the price ;)
> 
> http://www.storageperformance.org/results/benchmark_results_spc2 shows
> some-fc-solutions.....
> 
> i guess that we will be in summer in this performance-regions with about
> 30 % of the costs and much more flebility.
> http://www.storageperformance.org/results/b00035_HP-XP24000_SPC2_full-disclosure.pdf
> 
> Price: Total: $ 1,635,434  
> 
> > 
> > And thanks for the answer!
> 
> ay - cheers!
> 
> i will end up this post with some words of Coraid CEO Kemp:
> "... We truly are a SAN solution, but SAN is not in the vocabulary of
> Linux people, because SAN is equated with fiber channel, and fiber
> channel is too expensive. But now, there's 'poor man SAN"  [1]
> 
> > 
> > -- Pasi
> 
> Thomas
> 
> Any Questions - ask me ;)
> 
> [1] http://www.linuxdevices.com/news/NS3189760067.html
> 

Pretty nice setup you have there :) 

Thanks for the explanation. It was nice to see details about pretty big
setup.

Have you had any problems with it? How about failovers from header a to b..
do they cause any problems? 

-- Pasi

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users