Re: [Xen-users] iSCSI target - run in Dom0 or DomU?

[Thomas Harold <tgh@xxxxxxxxxxxx>]

Matthew Wild wrote:
> What I've been building is pretty much the same as this. We have 2 storage 
> servers with 5TB usable storage each, replicating through drbd. These then 
> run iscsitarget to provide lvm based iSCSI disks to a set of Xen servers 
> using open-iscsi. The vitual machines are then set up using these physical 
> disks. Because the iscsi devices can have the same /dev/disk/by-id 
> or /dev/disk/by-path labels on each Xen dom0, you can create generic config 
> files that will work across all the servers. Also, even though drbd is a 
> primary/secondary replication agent at the moment, everything is quite happy 
> for multiple Xen dom0s to connect to the disks, allowing for very quick live 
> migration.
> I haven't gone quite so far with multiple switches etc., but we are using 
> VLANs to separate the dom0 traffic (eth0), domUs (eth1), and iSCSI (eth2).
> All on Gb networking. We are also thinking of putting 10Gb links between the 
> storage servers to keep drbd happy.
Excellent news.  Did you document your setup anywhere public?

This all started because we're getting ready to add 2 new servers to our 
motley mix of individual servers with DAS and I had a bit of a 
brainflash where I finally saw how Xen + SAN could play together. 
Eventually, we should be able to pack our 6-8 servers down into 2-4 Xen 
"head" units and a pair of redundant storage units.  The firewall / 
border boxes will remain as separate units (although possibly running 
Xen + Shorewall).
For a modest amount of complexity, we'll gain an enormous amount of 
flexibility.  Which will hopefully result in a lot less stress for me. 
No more laying awake at night worrying about what happens when server 
hardware fails.
(And doing this myself forces me to learn the technology, which is 
worthwhile.)
...

My initial thought was also to use Software RAID across the two SAN 
units.  Export a block device from each SAN unit via iSCSI, then have 
the DomU manage its own RAID1 array.  But I'll look closer into DRBD 
since that seems to be the preferred method.
...

The proposed topology for us (full build-out) was:

(2) SAN units with 5 NICs
(3) gigabit switches
(4) Xen units with 3 NICs (maybe 5 NICs)

Switch A is for normal LAN traffic

Switch B & C are for iSCSI traffic,  with the two switches connected, 
possibly via 4 bonded/trunked ports for a 4 gigabit backbone, or using 
one of the expansion ports to link the two switches.
Each SAN unit will have a single link to the LAN switch for management 
(via ssh) and monitoring.  Then the other 4 NICs would be bonded into 
two pairs and attached to switch B & C for fault-tolerance and for 
serving up the iSCSI volumes.
Xen head units would have 3 or 5 NICs.  One for connecting to the LAN 
switch to provide client services to users.  The other NICs to connect 
to SAN switches B & C for fault-tolerance (with the possibility of 
bonding for more performance if we go with 5 NICs).
One change that I'm going to make since you're talking about DRBD 
wanting a faster inter-link is to add 2 more NICs to the SAN units (for 
a total of 7 NICs).  The additional 2 NICs could then be bonded together 
and linked directly to the other SAN unit via cross-over cables.
But for the start, I'll be running all iSCSI traffic over an existing 
gigabit switch.  We're going to install a 48-port gigabit switch soon 
which will free up the existing 24-port gigabit switch for use with the 
SAN.  (Our LAN is still mostly 10/100 hubs with a gigabit core.)  ETA 
for installing the 2nd switch and 2nd SAN unit is probably 6-12 months.
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users