is Zevo safe enough?

Moderators: jhartley, MSR734, nola

is Zevo safe enough?

Post by jigzat » Sat Nov 03, 2012 10:07 pm

Hello everyone I have 2 2TB external firewire HDDs and a 12GB iMac with an additional 1TB, every single HDD is formatted with HFS+ of course. I have stored in those HDD's basically my whole movie collection music at loosless format plus documents from my whole life pictures and even scanned negatives at 4800dpi since the originals have begin to fade.

Of course I have backup on very unreliable DVDs and the music and movie DVD's and CD's but the task of copying back to the computers is a daunting one.

I have been witnessing some files getting damaged, pictures specially that get cut in the middle but the originals are fine, is not something to be worried about since is not frequently. I understand that HFS+ is not reliable but I also worried that ZFS would be overkill. What do you guys recommend? Should I move to ZEVO or maybe to a different file system other than HFS+?
jigzat Offline


 
Posts: 4
Joined: Sat Nov 03, 2012 9:50 pm

Re: is Zevo safe enough?

Post by grahamperrin » Sun Nov 04, 2012 12:31 am

Historically

Before ZEVO was transferred to GreenBytes, digital photography breakage was given – with screenshots – as a classic example of silent file corruption. I have archives/caches of some pages in the tenscomplement.com domain but not that page in particular.

Rewinding further, before the name ZEVO was given, here's a screenshot of part of a page that was cached for me by Diigo:

Z-410 Storage, historical, extract.png


In my opinion, the only point that was potentially misleading was automatic repair of corrupted metadata. Depending on a person's interpretation of the word metadata, the (then) beta of Z-410 might have been unable to repair automatically where a pool was limited to a single disk. YMMV.

Now

As before ZEVO was released, there's the integrity that is associated with ZFS. In this forum I see no evidence of corruption with normal usage.

ZEVO aside for a moment, I recommend reading:


Back to ZEVO: I recently began experimenting with ungraceful removal of a cache device whilst a pool is online. Results to follow, in a separate topic.

From the guide to ZEVO:

ZEVO uses the standard ZFS on-disk format (v28) and is therefore binary compatible with ZFS on other platforms. However, direct interchange with other platforms is not supported in this version.


If you use the file system with any version 28-compatible OS, then decide to move away from ZEVO for the same data on OS X:

  • as things stand, that move will be away from ZFS.

As zfs receive in general can not work with a send from a more recent version, so the version (8?) currently used by MacZFS can not receive data sent by version 28 implementations such as ZEVO.

ZEVO Community Edition 1.1.1 fitness for your use case

12 GB memory is a good fit for system requirements.

Questions

What makes and models are the FireWire drives?

With your movies, might you use Final Cut X?
You do not have the required permissions to view the files attached to this post.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: is Zevo safe enough?

Post by TomUnderhill » Sun Nov 04, 2012 4:26 am

Greetings, jigzat. I own a publishing company and deal with great amounts of scanned pictures. Personally, I've scanned more than 13,000 images from positive, negative, reflective and transparent material. I have book project files going back thirteen years, archived on RAID5 NAS. It's so nice to know that the company who made my NAS is more concerned that the disks comprising the NAS are more important than my data. Yes, I know what they tell me... if I lose a drive I can lose everything. In their words, parity is wonderful. But then why do I find files that have been sitting on the NAS for five or six years that have problems?

You ask if Zevo is safe enough. Fair question. I asked it, too.

I now am in the process of moving all our production files and archives to ZFS. I haven't decided on the best redundancy for my needs and am currently experimenting with combinations of mirrors and stripes, but all my data now sits on ZFS.

I have been witnessing some files getting damaged, pictures specially that get cut in the middle but the originals are fine, is not something to be worried about since is not frequently. I understand that HFS+ is not reliable but I also worried that ZFS would be overkill. What do you guys recommend? Should I move to ZEVO or maybe to a different file system other than HFS+?


I disagree with you on a couple points: bit rot not being something to worry about because of its frequency and ZFS being overkill. Let's not even talk about the quality of burnable media for anything other than beverage coasters.

In addition to your movie collection, it sounds like you have invested hundreds (maybe thousands) of hours compiling a life's worth of pictures. If we assume you value your time at only $10 per hour, you have invested between several thousand dollars to tens of thousands of dollars in the time compiling the data you're trusting to unreliable media and file systems. There's no such thing as overkill when it comes to protecting stuff like this.

I don't think anyone here would not suggest you move to ZFS. It's just a matter of how.

Some thoughts to help you decide on how best to proceed:
  • You have 5TB of storage, what is your dataset?
  • How fast are your storage needs growing? 1TB/year... 1TB/month...
  • What is the size of your photo collection?
  • What is the size of your "home" movie collection?
  • What is the size of your commercial movie collection?
  • What is the size of your music collection?
  • How old are your current three FireWire hard drives?

  • How much data do you need to access on a daily basis?
  • Are you editing video, or just storing video?
  • What is your budget for adding new hard drives?

Your balancing act is between
  • the amount of storage
  • speed requirements
  • budget
Only you can decide on where to place your priorities.

The ability to run ZFS on my Mac Pro (or your iMac) helps me sleep better at night. Now if I can just get one of my financial websites to stop making me change my password once a month!
TomUnderhill Offline


 
Posts: 36
Joined: Wed Oct 10, 2012 8:06 am
Location: Southern California

Re: is Zevo safe enough?

Post by jigzat » Mon Nov 05, 2012 3:27 pm

It doesn't concern me a lot because I have backup although a big part is on self burned DVD's so my data is "safe" in the short term, I don't think my information is going to grow over 5TB in the short term. My external HDD's are Lacie Quadra and an OWC miniStack enclosure which is amazing (both with FW800), I'm planing of purchasing at least 3 more enclosures and sell the Lacie.

My picture collection is around 300GB movies are around 3TB and 150GB of music (ALAC) and the rest are documents (PPTs DOCs and source code).

Regarding movies they are actual DVD Movies, not self filmed movies so I don't have to use final cut or any video edition software. I might decide to compress the movies into mpeg4 since I just repacked them into MKVs with intact streams.

Regarding the state of my HDs they seem fine, they are less than one year old and I recently made a surface scan to check for bad sectors.

Speed is not an issue since is not a critical system, I'm just using it for convenience and to stream movies to the other room who watch at least 1 movie a day. Regarding my scanned pictures right now I'm just trying to save some of my own negatives in a really bad shape and my great-uncle collection so I'm scanning them in high dpi so I can work with them latter.

I'm not thinking into a RAID right now, I just want a reliable file system. Next year I will purchase and additional 4TB (2 * 2TB) to create a RAID 1.
jigzat Offline


 
Posts: 4
Joined: Sat Nov 03, 2012 9:50 pm

correction, clarification

Post by grahamperrin » Tue Nov 06, 2012 9:48 pm

grahamperrin wrote:As zfs receive in general can not work with a send from a more recent version, so the version (8?) currently used by MacZFS can not receive data sent by version 28 implementations such as ZEVO.


Correction, clarification:

  • ZFS file system version 2, currently used by MacZFS, can not receive data sent by ZFS file system (ZPL) version 5 implementations such as FreeBSD, OpenIndiana and ZEVO – that's not an issue with MacZFS, it's the nature of the lesser version
  • the ZFS POSIX Layer (ZPL) is visualised in the diagram discussed under the quick start guide topic.

References

Frequently Asked Questions about MacZFS: What software versions and features does MacZFS support?

ZEVO Wiki Site/ZFS Pool And Filesystem Versions
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: is Zevo safe enough?

Post by TomUnderhill » Tue Nov 06, 2012 9:50 pm

A basic comparison of RAID and ZFS:
  • RAID, at best, offers device-level security.
  • RAID offers no data integrity. You can write a file today. Next year, for whatever reason, the data can develop problems. RAID does nothing to prevent this.
  • ZFS checks every block and can provide ability to reconstruct your data.
Here's a ZFS configuration to consider that gives you better data integrity...
  • You need to leave your 1TB HDD in your iMac in JHFS+ so you can boot.
  • Create a 4TB pool from your two 2TB HDDs.
  • Provides 4TB of storage striped across your two drives.
Next year when you add two more 2TB drives...
  • Create a 4TB pool from your two new 3TB HDDs.
  • Mirror the two pools (similar to RAID 10)
  • Provides 4TB of storage, fully redundant and protected against bit rot.
Year after next...
  • As you grow, add drives costing $150 to $200 in pairs. (this is the current price of a 2TB drive)
  • Create a stripe on the new pair.
  • Add the new stripe to the mirror.
  • Remove the oldest stripe.
  • As you remove the smaller stripes, you can grow the mirrors to match the new minimum stripe size.
A strategy like this allows for growth as you need it without having to replace every drive every time and with full data integrity.
TomUnderhill Offline


 
Posts: 36
Joined: Wed Oct 10, 2012 8:06 am
Location: Southern California

minor points/queries

Post by grahamperrin » Tue Nov 06, 2012 10:24 pm

> … 4TB pool from your two new 3TB …

Typo?

> • Create a … pool from your two new … HDDs.
> • Mirror the two pools (similar to RAID 10)

A better way to describe it might be:

• use the two new disks to add a mirror to the existing pool

(If I understand correctly: creation of an additional pool would be a redundant step.)

So if the original pair of disks are at /dev/disk1 and /dev/disk2
with their zfs_pool_proxy at /dev/disk3
and if new unformatted disks are at /dev/disk4 and /dev/disk5
then the command might be:

Code: Select all
zpool attach poolname /dev/disk4 /dev/disk5


(Side note: examples in man pages for zpool in ZEVO and in other implementations of ZFS could be better, IMHO.)

Edit the post above if necessary, then I might edit or delete this post. Or clarify below. Thanks.
Last edited by grahamperrin on Tue Nov 06, 2012 10:49 pm, edited 4 times in total.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: is Zevo safe enough?

Post by jigzat » Tue Nov 06, 2012 11:17 pm

Thank you very very much for your suggestions, you are way ahead of me regarding RAID although I still don't know your opinion of Zevo state :mrgreen: , mostly writing reliability, I know is an early product and is ok if need to reinstall the drivers in case of a bug but I don't want to realize one day that my external HDDs data is inaccessible and unrecoverable permanently due a bug in the ZFS implementation.
jigzat Offline


 
Posts: 4
Joined: Sat Nov 03, 2012 9:50 pm

Re: is Zevo safe enough?

Post by grahamperrin » Wed Nov 07, 2012 2:46 am

I'm still learning (and making mistakes!) but if I understand correctly, the nature of ZFS makes it extremely unlikely that an implementation such as ZEVO could spoil the consistency of the on-disk format.

At ZFS in 30 minutes the text of slide two offers two key points:

Bill Hathaway wrote:… ZFS provides safety through two mechanisms – a Copy-on-Write architecture that uses transactions to keep the on-disk format consistent and checksums that can tell if any data is corrupted. …


I'll add a third key safety mechanism: ditto blocks.

… the more "important" a filesystem block is (the closer to the root of the tree), the more replicated it becomes …

… blocks that are more critical to the health of the pool are more replicated, making them far less likely to suffer catastrophic failure. … the chance that a 3-way ditto block will fail … approaches zero …


– Ditto Blocks - The Amazing Tape Repellent (Flippin' off bits) (2006-05-12) – an old post, but it helped me to visualise ZFS allocation for these blocks.

Effects of a bug

Loosely speaking, the worst disruption could be a kernel panic.

Experiences

In my few months' use of ZEVO I have:

  • had more than one JHFS+ volume, written by Time Machine, that neither Disk Utility nor DiskWarrior could repair (during that period I ran fsck_hfs extremely frequently; it seems that the irreparable damage to these file systems was sudden)
  • never had a problem with a ZFS volume.

Here in the support forum I know of only one case where data in a ZFS pool is not (yet) accessible – a drive hardware problem is suspected. This case underlines the importance of backups.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Re: is Zevo safe enough?

Post by jigzat » Wed Nov 07, 2012 10:16 pm

Thank you for your opinion grahamperrin and for the links, I'm going to read it. Your opinions have sold me ZEVO ;)
jigzat Offline


 
Posts: 4
Joined: Sat Nov 03, 2012 9:50 pm

Next

Return to General Discussion

Who is online

Users browsing this forum: No registered users and 1 guest

cron