Near miss with regards to backup.

EdinburghGary

EdinburghGary

Reply not Report
Messages
19,271
Name
Gary
Edit My Images
Yes
Hi Guys,

After an absolutely horrific near miss in terms of "backup failure" recently, I have been trying to really up my game. To cut a long story short, I had let my backup routine lapse a little and a Hard Drive **** itself. It meant for 5 days I was without an entire months worth of client files. This included £10,000 worth of paid for orders and about the same in potential sales from viewings still to take place.

Basically, the MAIN external drive took a tumble to the floor and decided NOT to work on ANY PC or MAC. I thought it had been backed up in our daily routine, but for some reason we only had data dated 20th January and anything previous to that date. Everything dated 21st onwards was gone. As I said, this included the final orders for various clients, which I had been paid for, and it also included about 15 to 25 viewings still to be done.

In short, I could have gone bust as a result :shake:

Several phone calls to data storage companies, days of research on the net, and about £200 on various "we promise to recover your data" programs, I had almost given up.

I then read a site with various testimonials on it. Disk Recoup promised to ignore hard drive faults and errors where other programs would crash as a result. I was doubtful, but desperate.

In order to use the program I had to purchase two additional 1TB hard drives. My lack of backup was now close to costing £400.

Drive 1 = FUBARED.
Drive 2 = Session Drive (Disk Recoup uses this to track what it's doing)
Drive 3 = NON FUBARED CLONE of Drive 1

The program took 38 hours to run. Once finished at 6:30 this morning, ALL my data was restored to Drive 3. I felt like a lottery winning prize chimp.

I have just had the hardest week of my life, and to think what would have happened should the program have failed to save my ass.

Moral of the story? I'm a ****ing **** and backing up daily, properly, is not an optional requirement you can let slip.

I have 100% learned my lesson.

New rules as of today:

  • 2 in store backups across 4 external drives. Drive 3 and 4 are taken off site to different locations daily.
  • All photos are now uploaded to two seperate online storage spaces (*not raw files, just high res JPEGS)
  • Any edits, adjustments are uploaded to same online storage solutions

Please take this as a warning. INSPIRE was almost a goner.

G.
 
Your a lucky Man Gary, really glad you managed to recover your livelihood, working in IT for donkeys years I preach backups to my clients religiously, Ive seen people with their entire company history on a single laptop, and Ive seen people lose an entire companies history through not backing up

definatley a lesson learned, I bet your a relieved man today :)
 
EdinburghGary said:
To cut a long story short, I had let my backup routine lapse a little
Click to expand...
I think I have more invested in hard drives doing backup than doing storage in the first place. Glad you have it all back though :)
 
I'm glad you got the data back Gary.
I can remember a few years back I had to get a data recovery company involved in the FD's failed laptop drive. It cost a fortune just because he wouldn't take advice on saving to the network rather than his laptop... " it takes too long" Yeah sure it does!

But look on the bright side, you'll never cock up in that way ever again.
It's only when you've been well and truly bitten that you really learn this lesson. Oh and yes... me too :eek: A long time ago, but I've got three separate backups now.

cheers
 
Jesus Gary - Was in a similar situation myself a while ago... Not with photos, but some documents that I hadn't backed up for some reason... That feeling at the bottom of your stomach that just doesn't go away, although yours is a bit more serious cash wise than mine though!

Glad you got them back, and all is well eventually!
 
I was expecting a bit more of a bashing :D

Matty - 5 to 6 days of feeling like I wanted to go shoot myself, absolutely brutal. I downed tools here and had to cancel 10 viewings. I am calling them all back in over the next few days, but in terms of lost revenue. I am down £500. Delayed revenue, probably £5K.

I am still nervous and feeling sick about the whole situation. I don't think I will recover from this for some time. My whole life felt like it had stopped and I spent most of the last week wondering how I was going to tell my clients, how I was going to pay my clients back, and how I was going to salvage the business going forward. I'm still ****ing furious with myself and I will admit it, embarrassed and ashamed. What a risk to take :(

Gary.
 
The word 'Whoops' doesn't quite cover things, does it (been there as well, sort of)? :D


Are you using a server or workstation OS as your main storage solution? Considering the relative size of your operation, and hence value of the data, I'd say that you'd be better off with a 2003 or later data server (yes the Macs can still talk to it) or a Linux one using 4-5 RAID disks and a decent tape drive as a back up. It's not cheap (but not much more than a D3 or 1Ds) but corporates use this method for a reason!!


Now go out and buy some Andrex...

.. your hoop must be incredible sore by now! :lol:
 
Last edited:
EdinburghGary said:
I was expecting a bit more of a bashing :D

Matty - 5 to 6 days of feeling like I wanted to go shoot myself, absolutely brutal. I downed tools here and had to cancel 10 viewings. I am calling them all back in over the next few days, but in terms of lost revenue. I am down £500. Delayed revenue, probably £5K.

I am still nervous and feeling sick about the whole situation. I don't think I will recover from this for some time. My whole life felt like it had stopped and I spent most of the last week wondering how I was going to tell my clients, how I was going to pay my clients back, and how I was going to salvage the business going forward. I'm still ****ing furious with myself and I will admit it, embarrassed and ashamed. What a risk to take :(

Gary.
Click to expand...

These things happen, you've learned a very valuable lesson and you have managed to overcome a very difficult situation. Well done for getting through it, now relax! Its not worth making yourself ill over it now that the danger has passed.

Anyway, cant chat long, gotta do a backup....
 
I did the same thing last year although not as critical financially, got too busy to worry about updating the backups, thought I'd do it when I got a bit more time, three months later, still too busy.

Booted up one morning and, "hard drive not recognised".

Three days getting data recovered, £150.00 recovery charges and, three days of panic, never again.

To those of you who are thinking "I must sort out some backup", do it now, it'll save a lot of heartache later.
 
remember kids, your backup is only good if you can restore from it..

ive heard of a company that was backing up to tape for 10 years, no errors and all backup jobs completed "sucessfully" etc. until one day they needed a complete system restore and found all of their tapes to be blank.

test, test, test regularly!
 
Backups are the one thing that always worry me.

I've recently spent 55k on restoring data with a 3rd party, as when we rejigged out datacenter space the old tape drives that were supposed to be stored were junked instead. So when I got a request for mail database restores going beck 10 years for a legal investigation, I had to outsorce it as I could no longer do the restores inhouse.

As part of this process, we found that a number of the tapes that were 6 or 7 years old just snapped when they were used, so some backups are missing anyway.

The joy of backups. One of the reasons I always like to use a proper backup system rather than just copying the files is the verify and the ability to recover individual files easily. Error reporting can also be automated in them.
 
It's not only the back up process you need to plan, but what you'll do when/if it goes wrong. Managing clients that are affected is almost as improtant as getting the data back. You've also got the cash cost/hidden cost quandry. It may appear cheaper to fix it yourself but what can't you do while your time is tied up sorting it out? No easy answers but all points to consider.
 
I would also recommend an automatic online backup, such as Mozy or Carbonite.

I've currently got 195 Gb on Carbonite, but the main advantage, as well as being offsite is the automatic bit, I don't have to think about this backup - it just works. Takes a few days to make the initial backup, but after that it just ticks away.

This is purely backup though, no archiving.
 
Online backups are all well and good if you've got a quick internet connection - I just did a calculation, and it works out if I upload around 170gb of data (which is what I currently keep backed up, it would take (assuming a 1.5mb/s connection, which is actually slightly quicker than mine) around 250 hours - so over ten days, running 24 hours a day with no interuptions for web browsing.

So for those with slower connections, it's just not viable... Even if I did the first upload via disc or drive, the uploads from weddings would be around 8gb a piece, minimum, which would still take an age.

Offline and offsite are the two I'm using, simply as anything that involves the internet is no good for me living in a village away from the exchange
 
mattyh said:
Online backups are all well and good if you've got a quick internet connection - I just did a calculation, and it works out if I upload around 170gb of data (which is what I currently keep backed up, it would take (assuming a 1.5mb/s connection, which is actually slightly quicker than mine) around 250 hours - so over ten days, running 24 hours a day with no interuptions for web browsing.

So for those with slower connections, it's just not viable... Even if I did the first upload via disc or drive, the uploads from weddings would be around 8gb a piece, minimum, which would still take an age.

Offline and offsite are the two I'm using, simply as anything that involves the internet is no good for me living in a village away from the exchange
Click to expand...

you have to factor in downloading those as well in the event of total loss.. when my download is roughly 1.5mb on a good day thats not worth thinking about :lol:
 
Acronis is worth a look, you can back up O/S or full disk.

Also non-stop backup of selected folder, just set it up and it works in the background.
 
neil_g said:
nice.. :(

i said over on OCUK recently that until fibre broadband and TB's worth of online storage is cheap for me personally its not even worth thinking about.
Click to expand...

Same here... Although that said, I do keep all of my documents and other smaller files in Dropbox, as an additional bit of backing up, but for photos, it's just not realistic at the moment.
 
Hi Guys,

I haven't yet got any automated online backup going, but like a lot of people I am on a crappy connection.

That said, the manual part of our viewing process involved sending all files to ANIMOTO. This is online only, and we have been running that way for 6 months.

We have now enhanced this part of the setup. I purchased 1TB of space from Google Docs, and I signed up for a private Pro Flickr Account.

When our viewing is ready for upload, we simultaneously upload to Google and Flickr, and then we tell ANIMOTO to copy from Flickr, removing the need for a third upload.

We are upload maximum quality full res JPEGS so each time a viewing is processed I think we are going to be in a very safe position.

The period before viewings are processed we are currently relying on our old system, minus my letting it lapse.

G.
 
I should add, now have about 10 shoots online. The average time to get everything on Google, Flickr and Animoto is approx 15 minutes per upload of single shoot.

Animoto previously took about 10 minutes and is slower than Google / Flickr, I think the 5 min increase is well worth it.

G.
 
It's good that the big man up the stairs decided to give you a second chance - you must have good karma or something!

SpinRite, by Steve Gibson, is also supposed to be a miracle worker with regards to data recovery. Just have a read of some of the testimonials, and the YouTube video on how it actually works: http://www.grc.com/spinrite.htm
 
Last edited:
Reading things like this always makes my hands sweat. Like standing on the edge of a cliff does or knowing you almost killed yourself because you fell asleep at the wheel. Glad you got it sorted mate. Pretty scary.
 
Gary, glad you managed to get everything back.

I take it you now keep your external drives well away from desk edges and glued/screwed/taped down to stop gravity attacking again?
 
You must log in or register to reply here.
Back
Top