Re: [Xen-users] high availability iscsi and xen setup

[Mark Adams <mark@xxxxxxxxxxxxxxxxxx>]

Hi James,

On Fri, Nov 05, 2010 at 10:09:12AM +1100, James Harper wrote:
> > Hi All,
> > 
> > I'm thinking about the best way to set this up, and would greatly
> > appreciate your learned opinions on it. I'm thinking of the following:
> > 
> > - 2 Storage servers, running LVM, heartbeat, drbd (primary/standy) and
> iscsi.
> >   This will protect against 1 storage server failing.
> > - 2 Xen hosts, running heartbeat to ensure the domU's are available.
> If
> >   not, migrate all hosts on to other xen host. This will protect
> against
> >   1 xen host failure.
> > 
> > Any opinions on this arrangement of setup or links to resources
> > discussing it would be much appreciated. Also any alternative ways to
> > provide the same HA would be useful for comparison.
> 
> That's the way I'd do it, subject to some performance testing.
> 
> > 
> > - Any Pitfalls?
> > 
> > - Gaps in the availability? (split-brain possibilities?)
> > 
> 
> I'd have 3 networks:
> 
> 1. drbd network. 2 gigabit (or 1 10G) ports bonded on each storage
> server connected directly to the other - no switch.
> 2. storage network. 2 gigabit (or 1 10G) ports bonded on each xen server
> and storage server, connected into a switch with some decent HA features
> (redundant PSU's probably), or stacked switches.
> 3. regular networks for normal DomU traffic

I was originally planning on having 8 gigabit ports on each DRBD
(storage) server, going in to 2 seperate switches using multipathing.
each xen host would also have 2 gigabit links in to each switch also
using multipathing.

This way I would be covered for switch failure also, any thoughts on
that? This would mean running DRBD and "SAN" traffic on the same
network, is this not advisable?

alternatively direct connected 10G CX4 port for DRBD traffic and 4 ports
(2 in to each switch with multipathing) would also be good I guess?

If anyone has any links or further info about xen ha with multipathing
that would be very helpful, or other suggestions for having auto
failover for a switch failure scenario..

> 
> So the DRBD servers would need 4 gigabit ports, bonded in pairs.
> 
> I assume from what you specified that you are running 2 completely
> separate instances of HA - one to manage DRBD and iSCSI failover and the
> other to manage Xen.

Yes that's right.

> 
> Split brain problems are the worst problems you'll face. If storage
> server #1 goes down at 9pm (power failure and this machine is on a
> circuit with a dud UPS) and everything fails over to #2, then everything
> goes down at 10pm (someone forgot to fill up the generators or the UPS
> runs out of battery or something), power comes back at 11pm but #2 fails
> to boot and everything starts using storage server #1 which is out of
> date. You'd have to be having a really bad day for that to happen
> though. 

I think that's an acceptable risk, there's not much you can really do to
mitigate it? I guess stonith to make sure #1 doesnt come back, your left
with nothing but atleast not the "bad" data?
> 
> James

Cheers!
Mark

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users