Re: [Xen-users] Shared Storage

[Jonathan Dye <jdye@xxxxxxxxxxxxxxxxxxxxx>]

Or FCoE for that matter.  I know some very large deployments that are moving 
that way.

- Jonathan

----- Original Message -----
From: "Bart Coninckx" <bart.coninckx@xxxxxxxxxx>
To: "Jonathan Dye" <jdye@xxxxxxxxxxxxxxxxxxxxx>
Cc: "Jonathan Tripathy" <jonnyt@xxxxxxxxxxx>, xen-users@xxxxxxxxxxxxxxxxxxx
Sent: Sunday, April 24, 2011 2:04:56 PM
Subject: Re: [Xen-users] Shared Storage

I concur, in terms of performance Linux based iSCSI might not be the 
fastest, but in terms of what you are familiar with or what is flexible, 
it might be a good choice again.

Also it might be worth to look into ATAoE. Not popular, but I'm told it 
is fast as hell.

B.



On 04/24/11 22:01, Jonathan Dye wrote:
> So, linux storage servers then.  If I might interject again I would suggest 
> you try nexenta or solaris 11 express.  If not, try a NAS appliance like 
> FreeNAS or Openfiler - one of the linux based ones is likely to have done a 
> better job than you will attempting to reproduce it.  If you're brave try 
> clustered storage with Ceph since that's the way everything is headed anyways 
> (i.e. the way of isilon, luster, GPFS and the like).  After all reasonable 
> options fail, roll your own with LVM.  IMO, making a storage server out of 
> linux is inferior because the volume management, filesystem, and raid are 
> stratified instead of engineered together.  If you use any modern solaris 
> kernel based distribution, like the ones named above, and ZFS then I think 
> you'll find that it can fill your network connection with storage traffic 
> without tweaking.  The downside is you have to be careful about hardware 
> selection.
>
> - Jonathan
>
> ----- Original Message -----
> From: "Jonathan Tripathy"<jonnyt@xxxxxxxxxxx>
> To: "Bart Coninckx"<bart.coninckx@xxxxxxxxxx>, xen-users@xxxxxxxxxxxxxxxxxxx
> Sent: Sunday, April 24, 2011 1:43:46 PM
> Subject: RE: [Xen-users] Shared Storage
>
> Thanks Bart. Very helpful info
>
> I agree with you about the LVM PV issue. It is indeed very uncomfortable. I 
> am looking into CLVM (Cluster LVM) though, however this isn't very well 
> documented.
>
> So the current idea is one target per Xen node (hense one target per RAID 
> array on the storage server), and one LUN per DomU. Is it easy enough to 
> expand and shrink LUNs? This was the advantage of LVM that I loved. I guess I 
> would run LVM on the storage server and export the LVs?
>
> Thanks
>
> -----Original Message-----
> From: Bart Coninckx [mailto:bart.coninckx@xxxxxxxxxx]
> Sent: Sun 24/04/2011 20:40
> To: Jonathan Tripathy
> Cc: Jonathan Dye; Xen List
> Subject: Re: [Xen-users] Shared Storage
>
> I think you better take one target and then several LUNs on it (one per
> DomU), that would make more sense. If you don't do that and use just one
> LUN for several DomU's, you need to create PVM LV's on the newly created
> disk for each DomU on the hypervisor side, does not really sound
> comfortable. You would also close any path to HA, unless you maybe
> introduce some locking system, since every hypervisor would be wanting
> to try to write to the LUN.
>
> B.
>
> On 04/24/11 21:35, Jonathan Tripathy wrote:
>> Hi Guys,
>>
>> Please forget the "thousands" number. We would have thousands of DomUs,
>> but this would be spread over multiple storage servers, so never mind
>> about that scale.
>>
>> If I was exporting "One big LUN" per Xen node, it would contain at most
>> 80 DomU LVs (In real world usage, closer to 50). Furthermore, each LUN
>> would be exported from a seperate RAID array. Each storage server would
>> contain x number of RAID arrays, where x equals the number of Xen nodes
>> and the number of exported LUNs.
>>
>> Of course, if I went with one LUN per DomU, then each storage server
>> would contain 80x LUNs (closer to 50x though).
>>
>> With these numbers, any idea which is better?
>>
>> Thanks
>>
>>
>> -----Original Message-----
>> From: Bart Coninckx [mailto:bart.coninckx@xxxxxxxxxx]
>> Sent: Sun 24/04/2011 19:36
>> To: Jonathan Tripathy
>> Cc: Jonathan Dye; Xen List
>> Subject: Re: [Xen-users] Shared Storage
>>
>> That is completely dependent on your hardware specs and DomU's properties.
>> It sounds like a lot though. I seem to remember some time ago you also
>> stated to want to run at least 100 DomUs on one hypervisor, maybe this
>> is again pushing it.
>> With a decent RAID and 10gbit or infiniband you can go a long way
>> though. You should also consider using SCST instrad of IET as it is faster.
>>
>> B.
>>
>>
>>
>> On 04/24/11 20:31, Jonathan Tripathy wrote:
>>   >  We're talking houndreds, if not thousands of DomUs here. Will iSCSI on
>>   >  Linux scale to these large numbers?
>>   >
>>   >  Thanks
>>   >
>>   >
>>   >  On 24/04/2011 19:13, Jonathan Dye wrote:
>>   >>  Why not create one iscsi lun per vm disk instead of carving them up on
>>   >>  the hypervisor? That's more typical, and a more typical state of
>>   >>  affairs in linux is your friend. Also, you would have just one lun
>>   >>  queue if you exported one big PV, instead of one lun queue per vbd.
>>   >>  That becomes a problem at scale.
>>   >>
>>   >>  - Jonathan
>>   >>
>>   >>  ----- Original Message -----
>>   >>  From: "Jonathan Tripathy"<jonnyt@xxxxxxxxxxx>
>>   >>  To: "Xen List"<xen-users@xxxxxxxxxxxxxxxxxxx>
>>   >>  Sent: Sunday, April 24, 2011 11:25:38 AM
>>   >>  Subject: [Xen-users] Shared Storage
>>   >>
>>   >>  Hi Everyone,
>>   >>
>>   >>  I am consider such a setup where I export an iSCSI target to a Xen 
>> node.
>>   >>  This Xen node will then use the iSCSI block device as an LVM PV, and
>>   >>  create lots of LVs for DomU use.
>>   >>
>>   >>  I was wondering if anyone could make me aware of any special
>>   >>  consideration I would need to take. I've posted a similar question to
>>   >>  the LVM list to ask for further tips more specific to LVM.
>>   >>
>>   >>  Am I barking down the wrong path here? I know it would be very easy to
>>   >>  just an NFS server and use image files, but this will be for a large
>>   >>  scale DomU hosting so this isn't really an option. Additionally, if I
>>   >>  wanted to make the LVM VG visible to multiple Xen nodes, is it just a
>>   >>  matter of running CLVM on each Xen node? Please keep in mind that only
>>   >>  one Xen node will be using an LV at any one time (so no need for GFS, I
>>   >>  believe)
>>   >>
>>   >>  Any help or tips would be appreciated
>>   >>
>>   >>  Thanks
>>   >>
>>   >>  _______________________________________________
>>   >>  Xen-users mailing list
>>   >>  Xen-users@xxxxxxxxxxxxxxxxxxx
>>   >>  http://lists.xensource.com/xen-users
>>   >
>>   >  _______________________________________________
>>   >  Xen-users mailing list
>>   >  Xen-users@xxxxxxxxxxxxxxxxxxx
>>   >  http://lists.xensource.com/xen-users
>>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users