[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-API] [xs-devel] Backup solutions for XenServer

As of around CentOS 6.5 (and for sure in RHEL 6.4) that there is a thinlvm utility that seems to take care of supporting thinly-provisioned LVM volumes. There is an associated snapshot design based on LVM thin provisioning that even supports the ability to do snapshots of snapshots, etc. down the chain.

Unfortunately, from a cursory look, it doesn't look like a back port to CentOS 5 would be that easy or even possible, but that it is integrated into both CentOS 6 and CentOS 7 gives some hope for a possibly standardized support of it in a future XenServer release, doesn't it? The other big question would be how readily something like this could be integrated into XenServer.


On 1/8/2015 9:12 AM, Tobias Kreidl wrote:
Hello, David:

Thanks very much for that feedback. Simple often means good with an easier implementation and fewer complexities to worry about, so as suggested some time ago on the Xen Project Web site https://xenorg.uservoice.com/forums/172169-xen-development?filter=hot&page=1 the thin provisioning of LVM does appear to be a good contender. The running out of space issue is, of course, endemic in any storage mechanism, so it's more a question of how to prevent corruption and recover in that event. As with ext file systems, perhaps keeping a certain percentage of the volume reserved for emergency issues could help while refusing additional write commits at some point when all storage capacity is exhausted (going into "RO" mode). Of course, being able to copy out and recover lost space would be a useful means of being able to keep VMs running by doing storage Xenmotions periodically and recovering the space through some means from the original disk (if need be though a re-initialization process or an LVM resize operation). With thin provisioning, it's less of an issue for some scenarios like a XenDesktop environment containing a large number of Windows clients because Microsoft comes out with monthly patches and it is way more efficient to just apply those to your golden images and reprovision all your VMs. You're then back to your base storage.

It would also be nice to se VHDX adopted for Windows VMs at some point. And while on the topic of storage, where do things stand with Ceph? The flexibility looks tremendous, but my understanding is that there are still performance issues. It has the capability of natively supporting several different storage options, addressing your comment about plug-in support for virtually any sort of file system one might want. It would also be interesting to get an update on both NFSv4 (which works under Creedence, just isn't officially supported) as well as with potential pNFS support. Some vendors like NetApp showed early on that NFS is still a very viable option for many storage needs and our own tests of iSCSI vs. NFS were convincing enough that with an LACP bond, we have a very solid NFS connection to our NexentaStor SDS and with thin provisioning plus on-the-fly compression/decompression, get anywhere from a 20:1 to 50:1 reduction in utilized storage. And as mentioned already, this is with the equal I/O performance of iSCSI. It's hard to beat that, especially if you can convince those who hold the purse strings that you've just increased your storage capacity by a factor of 20 or more without needing any additional disks.

Again, your efforts in this area are most appreciated and thank you for continuing to investigate and address these points. Citrix has been really great about being open to suggestions and feedback from the user community, so many kudos to you and your colleagues for lending your ears.


On 1/8/2015 7:57 AM, Dave Scott wrote:
On 8 Jan 2015, at 14:26, Lorscheider Santiago <lorscheider.santiago@xxxxxxxxx> wrote:

Hi Tobias,

Congratulations analysis. Really if used LVMoSCSI or LVMoHBA will not have much gain.
That’s right — our block storage types (LVMoiSCSI and LVMoHBA) are “thick provisioned” and hence VDIs take up a lot of space. As Tobias says below, this shows how important
thin provisioning is.

Within the upstream xapi-project (cc:d xen-api) we’ve been looking into thin provisioning
options for block storage. I’ve written two draft designs:

1. thin LVHD:


This proposes to extend the existing LVM-based SRs, allowing LVs to be resized at runtime as more space is needed. Every LV could start out small (like a file) and grow as new blocks are written. The trick is to make sure allocation can still happen when there is a network partition in the pool and avoid timing out VM I/O. The design proposes to cache free blocks locally on each host, allocate from there, and replay an allocation journal against the LVM metadata periodically. [ This is all based on previous work by
Germano Percossi and Jon Ludlam, so I can’t claim credit :) ]

2. OCFS2:


This proposes to manage OCFS2 clustered filesystem instances and to store VDIs as files (format flexible) on top. Most of the design so far is to deal with the complexity of
managing the configuration and state of the O2CB cluster.

My gut feeling at the moment is that thin LVHD is simpler to implement and will cause less “behaviour churn”, since we don’t have to change how HA behaves, how maintenance mode
is used etc.

Hopefully at some point in the future Xapi will be able to use any user-managed existing
filesystem, so a Linux expert power-user could use OCFS2 if they wanted.

Dave Scott

Best Regards,


Lorscheider Santiago
Visite meu blog: www.centralcloud.info
Twitter: @lsantiagos
Antes de imprimir, pense em sua responsabilidade e compromisso com o meio ambiente

On Tue, Jan 6, 2015 at 4:40 PM, Tobias Kreidl <tobias.kreidl@xxxxxxx> wrote:
Thank you very much, Dave, for the updates about this nice addition.

Here is my one concern, namely that for incremental backups to work properly, you need to continuously keep a base snapshot on the server, hence you will -- even if sparse storage is utilized -- still take up a fair amount of storage as long as you want to maintain the snapshot and backup mechanism.

This led me to thinking, so, what happens if you have SRs with different underlying properties? What kind of space requirements are being realized?

As a test, I took both a Linux RHEL 7 and a Windows 8.1 VM and checked how much space was really being used as well as allocated on the SR, as well as in the export files. Please note that here, some extra space savings take place on the NexentaStor NFS mount because of some built-in automatic compression taking place.

For a Linux VM with 8 GB of allocated storage on a VDI and around 1.63 GB of space in use and a Windows VM with 40 GB of allocated storage and around 20 GB in actual use, here are the numbers I came up with:

RHEL 7 RHEL 7 Windows 8.1 Windows 8.1 Function LVM NFS+compression LVM NFS+compression

base used: 8.0 GB 1.1 GB 40.0 GB 20.2 GB base allocated: 8.0 GB 8.0 GB 40.0 GB 40.0 GB

virtual-size: 8589923591 8589934592 42049672960 42949672960 physical-utilisation: 8615100416 5984371200 43041947648 20136346112

snapshot used: 13.6 GB 1.1 GB 59.0 GB 20.2 GB snapshot allocated: 16.0 GB 16.0 GB 80.1 GB 80.0 GB

full snapshot
virtual-size: 8589934592 8589934592 42949672960 42949672960
physical-virtualisation:    8388608     39944704 8388608         88576

export size (VHD): 5.97 GB 5.97 GB 20.24 GB 20.22 GB

delta snapshot used: 13.6 GB 1.1 GB 59.1 GB 0.3 GB delta snap allocated: 24.0 GB 24.0 GB 120.1 GB 120.0 GB

delta snapshot
virtual-size: 8589934592 8589934592 42949672960 42949672960
physical-virtualistion:     8388608        20992 8388608         88576

delta snapshot
export size: 0.126 GB 0.044 GB 0.078 GB 0.074 GB

Not surprising is that the exports all came our roughly the same (they are, after all, creating VHD files in all cases), with minor difference due to some activity between when the exports were created. However, what is particularly interesting in the NFS-based SR is that the initial snapshot seems to have taken up virtually no discernible space at all. Why? Evidently because it creates it's own initial "difference" disk. Running "xe vdi-list uuid=(UUID_of_VDI) params=all" on the VDI yields details on the space utilization/allocation. This is actually quite interesting, as I had not tried this with an already thinly-provisioned (as opposed to sparely populated) storage before.

Noting how an NFS-based SR is so much more space efficient, it does however leave the issue that if create both a full and delta snapshot, you will no longer be able to storage Xenmotion the VM until you bring the snapshot count back down to two or lower. One option would be to immediately clean this up after the latest delta vdi-export has taken place; alternatively, you could just defer this until a storage Xenmotion or other action is called for that requires prior cleanup.

This brings us back to the LVM case. The huge difference here is that you need readily three times the size of the VDI to be allocated and at least double the size to be able to retain a base copy and triple the size to hold in addition a delta snapshot. Plus, unlike a NFS-based SR, you cannot over-commit your storage allocation on the SR. it would hence be of great benefit in space savings if one could do one of the following: (1) store a thinly-provisioned and compressed version of the initial snapshot similar to what NexentaStor does with the NFS VM, (2) in creating an incremental snapshot, access somehow an off-line VDI file containing the base, (3) had the means to efficiently temporarily pull in an offlined VDI so you'd not have to keep the base snapshot on the SR constantly, (4) be able to store the base snapshot on a different SR than the VDI you want to snapshot, or (5) some other clever, unnamed mechanism.

The NAU VMbackup mechanism we have used in-house for years is efficient mainly because it creates a full snapshot on the fly for the purpose of a full backup and once completed, deletes it. Hence, you never need more additional space at any given time than that of a copy of the largest storage associated with an particular VM. The disadvantage is that this is of course a sequential operation and hence takes quite some time if you have a lot of VMs. However, being able to snapshot and keep a spare image would still take up a large amount of extra space. While having the means to very efficiently create incremental (delta) snapshots and back them up, this still leaves the large storage requirement issue open for LVM-based storage, should one want to retain baseline snapshots over longer periods of time.

To me, this is an indication how thin provisioning (in contrast to just sparse storage) can make a huge difference. It would be really interesting to see what other options could be implemented to help address the LVM limitations outlined above.

Thanks for taking the time to look over these thoughts, and I most certainly welcome feedback, in particular if I have overlooked something blatant.


On 1/5/2015 8:09 AM, Dave Scott wrote:
On 31 Dec 2014, at 11:27, Lorscheider Santiago <lorscheider.santiago@xxxxxxxxx>

Hi Dave!

Very good xapi the project site. Thanks for the tip!

About recuros of "export and import only the blocks Which have changed between two snapshots" this is a sensational news! It's been times that I miss this feature, which brings tremendous agility to backup. From what you wrote, similar operation to VMware CBT.

By your tests, you confirmed that this resource is in the build 90383c. We can not test since the Release Candidate build is in 90239c. Although he had already noticed changes in the snapshot in earlier builds the 90239c.

Congratulations for the work and thanks for sharing this information with us!

I’m glad you liked the new site! Credit for the site itself and much of the content should also go to: (extracted from git history in no particular order)

Euan Harris
John Else
Jonathan Davies
Rob Hoes
Si Beaumont

If you find any problems with it or have suggestions for improvement (or new content), feel free to make pull requests or make issues on the tracker:




Lorscheider Santiago
Visite meu blog:

Twitter: @lsantiagos
Antes de imprimir, pense em sua responsabilidade e compromisso com o meio ambiente

On Mon, Dec 29, 2014 at 7:47 PM, Dave Scott

On 29 Dec 2014, at 15:31, Dennis Duane Booher <Duane.Booher@xxxxxxx>

Hi Dave,

This looks very interesting. Since this is in the xapi-project does this mean it is intended for some future release? Is it in the current Creedence pre-release build?

I believe the code for this is present in the recent creedence builds. Just to check I installed build number ‘90383c’ and did a bit of smoke testing— the xe commands listed in the xapi-project page ran ok and produced the correct output.



-----Original Message-----
xs-devel-request@xxxxxxxxxxxxxxxxxxx [mailto:xs-devel-request@xxxxxxxxxxxxxxxxxxx
] On Behalf Of Dave Scott
Sent: Sunday, December 28, 2014 4:05 AM

Subject: Re: [xs-devel] Backup solutions for XenServer


On 27 Dec 2014, at 12:07, Lorscheider Santiago <lorscheider.santiago@xxxxxxxxx>

Hi Duane,

I do not mean backup tools but the improvements that XenServer 6.5 in your api to facilitate the work of backup tools.

I've written up some of the recent XenAPI improvements here:


The new APIs allow you to

- export and import disks in .vhd format (previously we only supported raw). This means the files are sparse. - export and import only the blocks which have changed between two snapshots. This allows you to avoid re-copying the same data again and again.

Let me know what you think.



Lorscheider Santiago
Visite meu blog:

Twitter: @lsantiagos
Antes de imprimir, pense em sua responsabilidade e compromisso com o meio ambiente

On Fri, Dec 19, 2014 at 8:27 PM, Dennis Duane Booher
Hi Lorscheider,

We use
for all our production Xenserver VM backups. It is freely available if you would like to give it a try.


xs-devel-request@xxxxxxxxxxxxxxxxxxx [mailto:xs-devel-request@xxxxxxxxxxxxxxxxxxx
] On Behalf Of Lorscheider Santiago
Sent: Friday, December 19, 2014 3:33 AM
To: xs-devel
Subject: [xs-devel] Backup solutions for XenServer

Hi All,

Looking backup solutions available on the market, solutions with good support XenServer are limited. The changes made in creedence api will make it easier and more attractive than more backup siluçoes have support XenServer?


Lorscheider Santiago
Visite meu blog:

Twitter: @lsantiagos
Antes de imprimir, pense em sua responsabilidade e compromisso com o meio ambiente

Xen-api mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.