[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] Cheap IOMMU hardware and ECC support importance

  • To: xen-users@xxxxxxxxxxxxx
  • From: Kuba <kuba.0000@xxxxx>
  • Date: Sat, 05 Jul 2014 00:45:31 +0200
  • Delivery-date: Fri, 04 Jul 2014 22:46:52 +0000
  • List-id: Xen user discussion <xen-users.lists.xen.org>

W dniu 2014-07-04 19:11, lee pisze:
Kuba <kuba.0000@xxxxx> writes:

W dniu 2014-07-03 00:45, lee pisze:

It's tempting to try it out, and I really like the checksumming
it does, and it's also confusing: There's (at least) ZFS and OpenZFS,
and Debian requires you to use fuse if you want ZFS, adding more

You haven't done your research thoroughly enough.

No, I haven't looked into it thoroughly at all.

On Linux there is for all intents and purposes one implementation.

Where is this implementation?  Is it available by default?  I only saw
that there's a Debian package for ZFS which involves fuse.

In case you'd like to try it out, follow these steps:

and just have few minutes of fun. I'm pretty sure a livecd will
do. You can also use files instead of real disks.

Thanks!  Sooner or later I'll try it out.  How come there are no
packages in the Debian repos other than the fuse package?

Sorry, that's way beyond my knowledge.

A very long time ago, I lost data with xfs once.  It probably was my own
fault, using some mount parameters wrongly.  That taught me to be very
careful with file system and to prefer file systems that are easy to
use, that don't have many or any parameters that need to be considered
and basically just do what they are supposed to right out of the box.

Does ZFS do that?  Since it's about keeping the data safe, it might have
a good deal of protection against user errors.

Destructive operations are usually called accordingly: zfs destroy,
zfs rollback, so they quite clearly express the intention.

"Rollback" doesn't sound very destructive.

For me "rollback" always meant "revert to some previous state" and for me it sounds very destructive - at least for the "current state" from which you are reverting.

How can a file system protect you from executing a destructive

It can try by warning you.

Does "rm" sound destructive or try to warn you? It just does what you tell it to do.

Snapshots protect you from most user errors. Off-site backups protect
you from su errors. To some extent.

Off-site would be good, but it's a hassle because I'd have to carry the
disks back and forth.

You can do that over the network. And it's always pros vs cons. It's your data, your requirements, your decisions and your responsibility.

And how are snapshots better than copying the

Snapshots are just snapshots, making them does not copy your data (well, in fact, ZFS is a COW file system, so making a snapshot may result in actually copying your data later on, if it's needed, but it's not copying as in "making a backup"). Replicating a snapshot results in creation of another dataset identical to the original snapshot. It's just a one more way of making full or incremental backups.

What if I need to access a file that's in the snapshot:  Do I
need to restore the snapshot first?

Usually you can "cd .zfs" directory, which contains subdirectories named after your snapshots, and inside that directories you have complete datasets just like the ones you took the snapshots of. No rollback/restoring/mounting is necessary.

It seems that ZFS isn't sufficiently mature yet to use it.  I haven't
learned much about it yet, but that's my impression so far.

As I said above - you haven't done your research very thoroughly.

I haven't, yet all I've been reading so far makes me very careful.  When
you search for "zfs linux mature", you find more sources saying
something like "it is not really mature" and not many, if any, that
would say something like "of course you should use it, it works

"Mature" means different things to different people in different
circumstances. Is Linux mature? Is Linux 3.15 mature? If not, is 2.6
mature? Does it mean it has no bugs? If ZoL is not mature enough for
you, you can use FreeBSD or Solaris. Or you can use hardware RAID +
any other FS. I have the same feeling about ZFS as Gordan - once you
start using it, you cannot imagine making do without it.

Why exactly is that?  Are you modifying your storage system all the time
or making snapshots all the time?

Yes, I take snapshots all the time. This way it's easy for me to revert VMs to previous states, clone them, etc. Same goes with my regular data. And I replicate them a lot.

Checksumming is sure good to have, being able to fully use the disk
caches is, too, as well as not wasting space through fixed block sizes.
I've never made a snapshot and don't know what I would make one for
other than perhaps making a snapshot of the dom0 and the VMs --- which
would require booting from ZFS, figuring out how to make snapshots and
where to put them and how to restore them.

The biggest advantage would be checksumming.  I'd be trading that
against ease of use and great complexity.  So you can see how it is not
understandable to me what makes ZFS so great that I wouldn't be able to
do without anymore.

I'm not saying you will feel about ZFS as I do after you try it out. It presents you a certain set of features, advantages and disadvantages and it is up to you, and you only, to decide whether you can benefit from it or not. All I'm saying is that I personally believe ZFS is worth taking into consideration.

Does it mean you have to use it too? Of course not:) Is it wrong not
to use it? Of course not! You should do what _you_ believe is the
right thing to do. But try it out nonetheless :) Or try HAMMER
(Dragonfly BSD). Or btrfs (although this one probably really is not
mature enough).

Btrfs still needs some time, and it seems to have disadvantages compared
to ZFS (which may not even be relevant for what I'm doing).  I never
tried BSD; that would be something new to learn.

Anyway, I'll try it out.  That doesn't mean I'll jump to it right away,
especially not while I still can't tell whether the server finally runs
stable or not.  Give it some time without crashing.  I don't even know
if the disks would work as JBOD.

So you would be running ZFS on unreliable disks, with the errors being
corrected and going unnoticed, until either, without TLER, the system
goes down or, with TLER, until the errors aren't recoverable anymore and
become noticeable only when it's too late.

ZFS tells you it had problems ("zpool status"). ZFS can also check
entire pool for defects ("zpool scrub", you should do that

You're silently loosing more and more redundancy.

I'm not sure what you mean by loosing redundancy.

How do you know when
a disk needs to be replaced?

ZFS tells you it had IO or checksum failures. It may also put your pool into a degraded state (with one or more disks disconnected from the pool) with reduced redundancy (just like a regular RAID would do). SMART also tells you something wrong has happened (or is going to, probably). And, additionally, when you replace a disk and resilver (ZFS term for rebuilding) the pool, you know whether all your data was read and restored without errors.

Does ZFS maintain a list of bad sectors which are not to be used again?

Don't know, but never heard of it. I always thought it's the storage device's job. Does any file system do that?

It's also quite difficult to corrupts the file system

It shows that there are more checksum errors after the errors were
supposedly corrected.

Not "supposedly". The increasing number only shows the count of encountered checksum errors. If ZFS could not correct the error, it would say so.

Using ZFS does not mean you don't have to do backups. File system type
won't make a difference for a fire inside your enclosure:) But ZFS
makes it easy to create backups by replicating your pool or datasets
("zfs send" lets you create full or incremental backups) to another
set of disks or machine(s).

As another ZFS or as files or archives or as what?  I'm using rsync now,
and restoring a file is as simple as copying it from the backup.

Typically as another ZFS dataset. Replicating ZFS snapshots has one big advantage for me (besides checksumming, so you know you've made your backup correctly) - it's atomic, so it either happens or not. It doesn't mean it's supposed to replace rsync, though. It depends on the task at hand.


Those guys don't use ZFS.  They must have very good reasons not to.

They do:

And I believe they have lots of good reasons to do so :)

That's some laboratory experimenting with ZFS.  Backblaze uses ext4,
though ZFS would seem to be a very good choice for what they're doing.
How can they store so much data without checksumming, without using ECC
RAM and not experience a significant amount of data corruption?

That's what I found about Backblaze and ZFS (22-07-2011):

We are intrigued by it, as it would replace RAID & LVM as well. But native ZFS is not available on Linux and we're not looking to switch to OpenSolaris or FreeBSD, as our current system works great for us. For someone starting from scratch, ZFS on one of these OSes might work and we would be interested to know if someone tries it. We're more likely to switch to btrfs in the future if anything.


That's just two organizations with similarly sized storage and different approaches. One uses standard solutions, the other one ported ZFS to Linux, so they could use it. It's up to you to define your goals, solutions and level of assurance. My personal approach is "hope for the best, plan for the worst".

The corruption wouldn't go unnoticed because they won't be able to
decrypt the data. They'd have to store everything at least twice, and
if they could cut their costs in half or less by not having to do that
through simply using ZFS, why wouldn't they?

Data redundancy is not implied by ZFS itself. You either want redundancy or not, ZFS is just one way of providing it.

What is the actual rate of data corruption or loss prevented or
corrected by ZFS due to its checksumming in daily usage?

I have experienced data corruption due to hardware failures in the past. Once is often enough for me and it happened more then once. If I hadn't done the checksumming myself, I probably wouldn't even have known about it. Since I started using it, ZFS detected data corruption several times for me (within a few years). But I don't own a data center :) Actual error rates might depend on your workload, hardware, probabilities and lots of other things. Here's something you might find interesting:



Xen-users mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.