Got Yottabytes?

Gleb Budman at online backup provider Backblaze has some interesting analysis on how to store the Yottabytes of data the National Security Agency will have in 2015. It will only take twice the world's GDP and fill the states of Delaware and Rhode Island with data centers. Now that's a real storage challenge.

Read their post NSA might want some Backblaze pods

Is Digital Archival?

Prints are still being made from Ansel Adams' negatives that are over sixty years old because they are inherently archival.

Will the same be true of digital files?

Will your children and grandchildren be able to look through a box of your digital files sixty years from now? Will the media be readable, and will there be anything to read it with?

The answer is NO if you’ve been using Sony’s AIT drives to archive and backup your files because they are being discontinued.

For over a decade, tape has been a primary archive and backup media for many IT professionals. It is considered by many to have a longer archival shelf life than hard drives. Up until the last year or two, tape was also cheaper per GB than hard drives, although that is no longer true. I suspect that the price drop in hard drive storage, coupled with new storage approaches have caused IT professionals to change their buying patterns.

In practical terms, what this means is that if you are using AIT, you need to immediately find a new backup media. It also means that it’s only a matter of time before you won’t have a drive available to read your archive of AIT Tapes. Your archive essentially just got reset to zero, because you’ll only be able to access it for as long as you can scrounge up drives on Ebay. Basically you need to rebuild your archive all over again if you’ve been using AIT tapes.

It’s becoming clear that digital files are not an archival medium, and that it takes constant investment to protect your archive in even an imperfect way. Even the Academy of Motion Picture Arts and Sciences realizes this. They’ve calculated that it’s less expensive and more archival to write digital movies to B&W separation negatives than it is to store a digital copy. In fact, they claim there is no archival digital storage method at all!!! Yikes!

All this makes the Millenniata/Cranberry archival DVDs a whole lot more interesting, but at 4.7 GB per disk, it’s not a efficient way to back up terabytes of files. And even if the media last as long as they claim, there’s still the problem of finding a reader that can read it 25+ years from now, and the cost of $16 per blank disk.

What’s my point? That we are constantly in danger of losing our digital photographs because there is no archival storage medium that we can be sure will be readable one hundred years from now. With my own eyes, I’ve seen the negative that makes Ansel Adam’s photograph Clearing Winter Storm, made in approximately 1939, and it’s still readable with the naked eye, and can be put into an enlarger and printed.

To achieve the same thing with digital we have to have media that doesn’t degrade, we need devices to read it, and we need software that still works with Windows 2070 SP6. No one has solved this one yet. Until someone does, make prints, which will be readable for a long time to come.

What to do when the lights go out?

by Rich Seiling

This is probably the most unglamorous article that can be written about maintaining a digital imaging workstation, but believe it or not, paying attention to the electricity going into your computer is important. Computers need a constant and regular supply of electricity to maintain normal function. 

You want normal function, right?

Yes, you do, especially when you are transferring the photos from your latest amazing adventure to your hard drive.

Leave the lights on, I’m scared!

All kinds of screwy things can happen to your computer because of power outages.  Probably the worst is a hard crash, the equivalent of your computer hitting a brick wall at 50 mph with a cup of coffee in its hand.  Everything your computer was in the process of doing comes to a dead stop, and some of it goes flying all about, in a metaphorical sense. You really don’t want this to happen in the middle of a data transfer.

If you’re using a RAID to mirror your files, it makes for even more problems. A hard crash usually causes the RAID to go out of sync, which initiates a rebuild, which leaves your data without redundancy during the rebuild, and exposes you to risk of loosing your files.

What can brown-out do for you

Fortunately, there is an answer, in the form of a UPS. Not the little brown truck that delivers your online purchases, but an uninterruptible power supply. A UPS in its simplest form is a battery that continues to power your computer should there be a blackout.

The idea is to power your computer long enough to save your work and shut down it down before the battery dies. But in practice they can do even more than that. A UPS can condition power so your computer is getting a consistent voltage, and can carry you through brownouts, overvolt, or undervolt situations, all of which are probably more common they you might think. (Look at what a Main Bus B undervolt did for Apollo 13!)

Another great feature is that most UPSs can connect with your computer and turn it off before the battery runs out, so that you’re protected even when your computer is unattended.

Does this UPS make me look fat?

With a UPS, size is an important thing. If your computer and accessories draw more power than your UPS is designed for, it won’t protect you! When the power goes out, the UPS will be overloaded and it will turn off, taking your computer and devices with it!

For a UPS to be useful, you want it to be bigger than the load you intend to plug into it. To do that, you have to know how much power your computer and accessories pull. 

If you want to know the size UPS you need with great precision, buy a Kill-A-Watt watt meter that lets you measure the VA (volt amps) that your system pulls. 

Computers and other devices pull a greater load at startup than in normal operation, and your UPS has to be large enough to handle this peak load, so we need to measure it. Plug in all of your equipment, then while watching the Kill-A-Watt display, start up all of your equipment at the same time and record the peak load in VA.  (Warning! Never plug a laser printer into UPS! You’ll see this warning in the UPS instructions. The reason is that laser printers pull peak loads during use that can overload the average UPS.) Once you know your peak load, make sure that you use a UPS that is rated for at least 130% of that load.

Don’t want to do the test? OK, I’ll make it easy for you. I tested the following representative Photoshop workstation:

Mac Pro Tower

23 inch Apple Cinema Display

17 Inch CRT

It drew a peak of 345 VA at startup (or about 300VA once up and running.) This probably is close to what your workstation draws. So, in this case, you need a UPS that can handle at least  450 VA, but I’d feel much more comfortable UPS in the 750VA to 1000VA range. The price difference between the 750VA and 1000VA typically isn’t that much, so I usually go for the larger model. It also gives you room to add more accessories as your system grows.

How long will my battery last?

When you read the specs for a UPS, you’ll see an estimated runtime, which tells you how long the UPS should run without power. This measurement is usually given for a “half load”, meaning that if you are drawing half the rated load, say 500VA from a 1000VA UPS, it’s probably accurate. The larger the load, the less time the battery will last. If you are pulling anywhere close to a full load on your UPS, the battery isn’t going to last long at all, rendering it pretty much useless, so you don’t want to pull a full load!

How do you make sure you aren’t overloading your UPS? You can use the Kill-A-Watt meter, but an easier way is to choose a UPS that has a display that tells you how much load your devices are pulling. Then with a quick glance, you can tell what the load is. My personal preference is to pull no more than 50-60% of maximum power on my UPSs.

Testing your UPS

Once you have a UPS, and have everything plugged in, you need to test it. The easiest way is to use the test button (if available) but what I like to do is pull the plug of the UPS while the computer is running to see if it all still works. It may be a little dramatic, but I feel a real sense of confidence watching the computer run while I’m holding the power plug in my hand.

Where to buy

A 750-1000VA UPS isn’t the little UPS you see advertised in the Sunday circulars for $39.95. Best Buy might have them, but you may need to visit a computer store or go online to find the model you need. Expect to pay $100-$150 for a UPS this size, and watch out for shipping charges. The batteries in a UPS are made with lead, so they are heavy! You should also take care when lifting them, unless you like back pain.

Choosing bigger is usually better, but you’re not buying a UPS to run for hours at a time. If you need that kind or runtime, you need a generator and a UPS. But if you need to run multiple computers from a UPS, it’s usually cheaper to buy one big one than lots of little ones. A 1500VA battery will comfortably run two workstations, as long as they are close enough for the cords to reach the UPS.

Buying a UPS is probably the most boring $150 you’ll ever spend, but it will help you protect those thousands of dollars you’ve invested in creating priceless images. To me, it’s as vital as the $5 lens cap on my expensive lens. Not very exciting, but cheap protection for something very valuable

UPS manufacturers:

APC

Belkin

CyberPower  

Tripp Lite

SSDs for Rugged Storage

by Rich Seiling

If you are looking for super rugged storage for your photographs while in the field, you should look at the new Solid State Disks (SSD) that are hitting the market.

While there are only a few vendors for these drives right now, that will change soon, and prices should drop dramatically. This will likely make SSDs the standard for rugged field storage.

An SSD is more rugged than a hard drive because there are no moving parts, which means it’s far less susceptible damage if it’s dropped or otherwise abused. Hard disk drives, on the other hand, have many of moving parts that can go out of whack if handled roughly.

How much do they cost? Look at this pricing from Other World Computing for their Mercury On-The-Go Pro Drives using SSD:

64GB $279.99
128GB $449.99
256GB $729.99

Currently, traditional hard drives cost far less per GB, but considering the rigors of field photography, the ruggedness of an SSD may be worth it to you.

The Idiot’s Guide to Restoring a Failed RAID 1 Hard Drive

In my last post I recommended you use a RAID 1 drive like those offered by Western Digital and LaCie as the primary storage for your photos.

Taking that simple step will offer a huge measure of protection to most photographers.

But buying the drive is only part of the battle. You have to know what to do when it fails, because some day a light on that drive is going to start blinking, warning you that one of the drives in the RAID has failed, and you are back to having your files on one drive. When that happens, you need to take immediate action to restore protection of your files.

Taking Action

If you read the manuals, most RAID 1 drives have options to replace the faulty disk with a new drive, then let the RAID automatically rebuild. This means you have to open up your RAID case, find and replace the faulty drive, and then have the RAID rebuild itself. If you select the wrong drive, or make one error, it could destroy your entire RAID structure, thus compromising all of your files.

This solution is technically possible, but I think for most users it’s a process fraught with huge risks. I believe you are more likely to mess something up than fix it. That’s because you’re stepping into a mine field, and the manual is not a good enough map to get you through. One misstep means your data goes kaboom! Even experienced IT professionals can make costly mistakes here.

Furthermore, if the “good” drive fails during rebuild, you will lose all of your files.

I think there is a far simpler solution that minimizes risks, almost guarantees success, and is still cost effective. I call it “The Idiot’s Guide to Restoring a Failed RAID 1 Hard Drive”

First, do a backup so you have a current copy of the good drive. If you have multiple backup sets, update as many of them as you can. It’s cheap insurance.

Then, DO NOT ATTEMPT TO REBUILD THE DRIVE!

Take the simplest, safest approach:

1. Shut the drive down. This is the best way to prevent further drive failures.

2. Buy a new RAID 1 drive.

3. Copy all of your data from the old failed RAID to the new RAID 1.

The cost of a new RAID 1 drive is nothing compared to the cost of having a drive recovered by a professional data recovery service. Simply copying your data to a new RAID 1 drive is so simple that any digital photographer should be able to do it correctly 99.999% of the time.

If you can’t wait a 24 hours for a new RAID 1 drive to be delivered, then keep a spare on hand. A $220 drive waiting on your shelf is cheap insurance compared to the value of your files and the cost in time and money to restore them in a worst-case scenario.

That’s it! It’s really that simple. This is a solution that you, the average photographer can use...it just comes down to making the right choices.

This leaves you with one question: What do I do with my old RAID drive? Stay tuned for that in a future post...

Systems for Automatic Duplication

So I’ve convinced you to store your files on a volume that has automatic duplication capabilities...now the question is, which one should you buy?

There are more solutions available than ever, and fortunately there are some great options for the home user that don’t require an IT genius to set up.

My recommendation is for a RAID 1 Mirroring setup.

I think RAID 1 offers the best protection from the most common hard drive failures. That’s because most RAID 1 products allow you to access your data after a single hard drive failure. That allows you to make an immediate backup of your files to ensure you have an up-to-date copy, and it gives you many options for restoring your redundant copy.

There are many vendors for RAID 1 products, but Western Digital and LaCie stand out as having solutions that are priced for the home user, and are easy to set up and maintain.

Western Digital offers the My Book Studio Edition II. LaCie offers several solutions, with the Hard Drive MAX being the most attractively priced.

Between the two, the Western Digital product looks like the more attractive choice to me.

My prime motivator is price...the Western Digital product is about $100 cheaper than similar capacity LaCie models, which adds up quick if you need a lot of storage.

But I also like the fact that Western Digital is a larger company that is likely selling more units...and I think a larger user base for a product is always a good thing. I have no way of knowing if that really will make it a better product...it’s just a gut feeling. I haven’t used either product, so I’m just going by specs and company reputation.

Before you buy any storage solution, you should run it through my Storage Litmus Test:

Do you know how an successfully operate it successfully 100% of the time?

Read the manuals online before you buy and ask yourself if you know you can successfully operate it. If you don’t feel confident that you can, then don’t buy that solution.(And if you already have a solution, read the manuals now and see if it passes or fails the test.)


Bad Solutions and Non-Solutions

In our poll a few weeks back, several people said they are using RAID 0 Striping to protect their files. That’s bad news because RAID 0 provides less protection than a single hard drive. RAID 0 works by splitting up a file so half of it goes on one drive, and the other half goes to a second drive. If one hard drive fails, you lose all of your files. Even worse, if the data that tracks how the files are split up becomes corrupted, you lose access to all your files. RAID 0 is about speed, not protection. Don’t use it for protection!

RAID 5 is another solution some people are using. It’s been the darling of the IT industry for a long time, but I think it’s playing with fire.

About a year ago, a client had a RAID 5 fail. After about 4 weeks without access to their drives and considerable expense, they were able to recover all their data...and this was with a top-end system. They got lucky...they could have lost it all...and luck is not a protection strategy. Last I talked to them, they were investigating a RAID 1 system.

If you want to get techie about it, check out Why RAID 5 stops working in 2009.

The Drobo is also subject to the many of RAID 5’s shortcommings since it typically uses parity data to create a virtual duplicate. That means that if a second drive fails before a rebuild is complete, you can lose all of your data. I did one tweet recommending the Drobo, but I’ve since become jaundiced on it, and I’ve removed the tweet. I think RAID 1 is a much safer solution than the Drobo.

If you are using RAID 5 or a Drobo, I’d replace it with a RAID 1 immediately. Not having to rebuild a disk from parity data is a good thing.

What does Rich do?

What do I do? I use a custom RAID 1 solution that I’ve assembled to work on Mac servers. I use SoftRAID to make mirrored copies to two separate hard drives. SoftRAID doesn’t use proprietary formats, so both copies of a volume are readable on any computer...I don’t need a RAID card or enclosure to read them in event of a failure. I use Sonnet DP500 SATA enclosures with Sonnet SATA cards, and various brands of SATA hard drives. The servers run 24/7/365 and are hooked up to a battery backup to protect against short power outages. In this configuration, with dozens of hard drives, I have a drive or two fail every year, but I have not lost any data, and I have been able to keep accessing my files while the mirrored drives rebuilt. It is very fault tolerant, very simple, and has proven very reliable.

My solution does basically the same thing as the Western Digital and LaCie products, just on a larger scale.

Whatever solution you choose, remember that an automatic file duplication system like a RAID 1 is NOT a backup! It will not protect against catastrophic failures, i.e. when your hard drive is lost, stolen, or destroyed. It is meant to protect you as you are working, which is the time in between backups. It also makes it less likely you’ll need to use a backup to restore a hard drive, which can be a time consuming process (And you typically lose some files, since backups are almost always out-of-sync with your hard drive).

Buy one today, before your single hard drive fails.

How do you protect your files from hard drive failures?

How do you protect your files when your hard drive fails? One of our clients came face-to-face with this question when his drive started to fail this week...and then his backup copy failed!

We want to know how you protect your files. Vote in the poll and leave comments we can delve deeper into this important issue.

New Year's Backup Plan

If you are using a digital camera or making digital files for printmaking, how you backup your photos should always be on your mind.

A good file backup plan has multiple layers of protection to deal with multiple modes of failure, because if something can go wrong, it will. Lately, I’ve been thinking of adding a new layer of protection to my backup strategy. I’m calling it the New Year's Backup, and it’s so simple anyone can do it. It’s the computer equivalent of a 30-Minute Meal with Rachel Ray.

Here’s the Recipe:

Buy a new hard drive. Attach it to your computer. Make a copy of all the new photos you took in 2008. Then, detach the hard drive, label it "2008 photos," put it at an off-site location like your safe deposit box or your mom’s house, and don't touch it again until you need it.

Is that easy enough?

If you are not already doing a regular backup, you should already be ordering a hard drive and using my backup recipe.

What about the experienced chef? Do you really need to do this, on top of all the precautions you are already taking?

My answer is yes, and thinking too hard about it will be your undoing.

Here’s a list of reasons to convince you:

- Way back when, before digital, pro photographers spent thousands of dollars making dupes of their best photos. Even then, they didn’t have a backup as good as a complete copy of their year's shooting off-site.

- How much did you spend on travel last year? (gas, lodging, meals, airfare)

- How many hours did you you spend making new photographs?

- How much did you spend on new camera equipment?

- How long would the pain last from forever losing even a few of your photographs?

Now take any one of these costs and compare it to the cost of hard drive. For $75-$150 (check our dealnews.com for a good deal) you can have a complete copy of every image you made this year. Most Americans spend more on satellite TV every month.

It may seem too simple, but sometimes the simplest solutions are the ones that work when everything else fails...and it’s a New Year's Resolution you can actually keep.

Seagate Hard Drive Random Freeze Problem

If you’ve recently purchased a Seagate 1.5TB or 1TB drive you need to read this blog post from Robin Harris.

The quick scoop is that these drives can pause or hang up which causes your RAID to think it has failed, or it can cause your computer to freeze.

The reports are that the drive freeze itself doesn’t corrupt data, but failure of your RAID could lead to a loss of data, as can hard crashes and freezes of your system.
Check your drives and make sure you aren’t at risk, and if you have experienced some problems since you purchased a new Seagate hard drive, this might be why.

DriveBox from wiebeTech creates solution for storing bare hard drives

If you are like me, it seems like I’m drowning in hard drives, because they are the media of choice for storing images and creating backups. But storing bare drives can be a pain. They need to be protected from static and physical damage, yet still be easy to store. The plastic bags most drives are shipped in aren’t adequate, and the clear plastic boxes used by some manufacturers take up a lot of space and are rather thin.

The DriveBox has proved to be the perfect solution. Constructed from sturdy anti-static plastic, it provides a secure cocoon for my drives that protects them from the minor bumps, bruises, and static shocks that can occur as they are transported and filed at my storage locations. Its shape makes it easy to fit the most drives securely in a safe deposit box or fire safe. Multiple flat surfaces make it easy to label so you can find the drive you need when you need it.

The drive box was designed to accompany wiebeTECH’s RTX TrayFree enclosures, but you don’t need to buy their enclosure to use the DriveBox. (Dedicated enclosures such at the RTX or SonnetTech D500P provide an easy way for photographers to consolidate their storage from a collection of multiple external drives with their rat’s nests of cords, to an easy-to-upgrade system that is consistent and reliable.)

The DriveBox is available directly from wiebeTech for $5.95 each, or $4.95 each in quantities of 10 or more. It's an inexpensive way to protect your priceless photographs.

Director's Data Stolen

In the ongoing saga of coming to terms with the need to secure our digital photographs and other important data, let us consider the case of famous Director Francis Ford Coppola “who just lost 15 years of computer data, including writings and family photographs, when robbers raided his Argentine studio” according to this AP article.

“Coppola appealed to the bandits to return the small computer backup device, which was taken along with computers in the raid Wednesday night.”

"They stole our computers; they got all our data, many years of work," said Coppola. "If I could get the backup back, it would save me years — all the photographs of my family, all my writing."

The lesson to be learned here?
Not only do you need a backup, but you need a current copy in a separate location (mom’s house, safe deposit box, etc) at all times to protect against theft, fire, tornado, flood, earthquake...you name it, or you’ll be making the same laments Coppola is. If he had had an backup in a separate location, you'd never be reading this.

1 in every 1500 files corrupted

Capturing and storing our images with digital technology poses risks that aren’t well understood by most end-users. If you think your images are safe, you might think differently after reading a recent post on Robin Harris’s “Blog Persistence of Memory”.

The short story is this: The scientists who smash atoms together to discover new particles, tested their new computer system. They found that you can expect to have 1 in every 1500 files corrupted when you store them to disk (or hit “save”).

It’s important to note that this corruption may not completely destroy a JPEG file--but then again, it might! And Murphy’s Law dictates the one corrupted file will be the first photo of your newborn, or that once in a lifetime quadruple rainbow over Yosemite. Also, this damage is cumulative, and over time, more and more files will become corrupt.

It’s up to photographers to tell storage vendors we don’t want larger, faster or cheaper solutions at the expense of lost data. We’d rather have storage that greatly minimizes loss, even if it costs more. We are willing to pay more for that to keep our thousands of photos safe, right?

Excuse me while I go make sure my data is backed up.....