Digital paranoia anyone ???

I have worked for an OEM for too many years now. The first item you listed is very common. Even if the machine is beeping and flashing a red light, you often get an admin who will walk in, and press the ignore button. I have worked with re-selling Dell and HP equipment a lot as well. Often it can take 7+ days to get replacement disks from them. However, in anything more than a raid1 mirror, I always ensure that there is a hot-spare (and I advise the customer to take backups as well, but they rarely listen). I once had a customer with a 9 disk raid 5+hot spare. They complained when they lost data. I recovered the machine, looked at the logs, and asked why over the last 4 months they had ignored the beeping, the flashing lights, and the emails that the system had been sending to them, after 1 disk failed, 2 disks failed, then the third and final redundant disk gave up.

Unfortunately, my experience of raid 1 mirrors, and especially striped mirrors, is that the second disk in the mirror gets a pounding as soon as the first one has failed. Often the disks are from the same batch, and same age, and tend to not put up too well to the pounding. However, most of the places where I have put raid 10 has been into database stores etc., a slightly different workflow to a jpeg server.

The advantage to mirrors over raid 5 though, is that when the system comes back to me for recovery, it is often easier to reconstruct the data from a mirror than it is from a raid 5.

well, i guess me speccing an HP ***, and having a 1 day hardware turnaround was probably the correct route then...

we raid5 pretty much everything on a virtualised array - we've had hardware failures but the way the *** works handles it.
 
I am sensing there is a "need" for a business here. Works like mozy, where you have a client installed on your pc to upload files- but you can "batch upload" by sending in dvds/blurays/hard drives- which are then copied into your space and returned to you.

hrmmmmmm....
 
I am sensing there is a "need" for a business here. Works like mozy, where you have a client installed on your pc to upload files- but you can "batch upload" by sending in dvds/blurays/hard drives- which are then copied into your space and returned to you.

hrmmmmmm....

I think its the initial upload that would be the problem for most people. Took me several weeks IIRC.....
 
well, i guess me speccing an HP ***, and having a 1 day hardware turnaround was probably the correct route then...

we raid5 pretty much everything on a virtualised array - we've had hardware failures but the way the *** works handles it.

You are lucky if you get a 1 day turnaround. We have a customer with a load of machines on one order, next day hardware engineer support from HP.
Takes 1 day for HP to log the call, another day for them to respond and ask for diagnostics to be run (red light on the disk, and the disk not being there isn't enough), and then 2 days to ship the spare part. Then they start chasing for the part to be returned even before the good one has arrived sometimes!
HP currently appear to be pulling back from being helpful with support at the moment (to the extent that I am advising customers to have onsite spares).

Anyway, digressing from the point somewhat.
 
I think its the initial upload that would be the problem for most people. Took me several weeks IIRC.....

Quite.
At 50kb/s, a 300mb upload to a server took around an hour last night. I could get 10-12 of those to a DVD? Would take ages to burn the DVD, and the hassle of posting them.
 
You are not being paranoid at all I run 3 externals and back up to dvd too
I have had a drive fail but I did knock it off the table while it was running !
I have had a couple of DVD's fail and dont completely trust them
Pete
 
duplicate external 1.5 tb drives in wiebetech housings hold all my photos in month order then another 1.5tb with time machine duplicating my hard drive to be sure that if that goes I can re-boot from another drive.

one drive stored at neighbours changed weekly
and still feel paranoid....
 
Even if a hard drive dies, the data is still there not matter what people say. You can format it put 1's and 0's over it but it is still retrievable unless it gets stolen.

You can get it imaged to retrieve the data but I am not sure how expensive this is, but it has to be cheaper then 4 backup 1tb drives???
 
Even if a hard drive dies, the data is still there not matter what people say. You can format it put 1's and 0's over it but it is still retrievable unless it gets stolen.

You can get it imaged to retrieve the data but I am not sure how expensive this is, but it has to be cheaper then 4 backup 1tb drives???

Not entirely sure that's true. I believed if you overwrote a file it was lost - or it would be possible to retrieve every single bit of data ever written to the drive.

And I am not sure £2-300 would get a drive recovered, but don't know for sure as I have everything backed up :)

Graham
 
A good backup strategy for a PC not just images is a good idea in any case. It is also advisable to have a copy off-site, even if it's in the shed at the bottom of the garden in case of fire. Probably wise to put it on a high shelf as well in view of recent rain.

Agreed, it doesn't matter how many back ups you have, if they all go up in flames in the same place, you are still screwed! Most photographers have a couple of copies, but they are often in the same location - a mistake that you will get away with when a drive fails, or a dvd gets scratched, but one that will bite a small percentage of people in the **** every now and then.

My process is: I normally copy off the CF cards, rather than cut them, I backup to a DVD before I leave the job, so I have the data in 3 locations before I leave the venue. I keep the data on the working computer as long as I can, and back up to an external HD as soon as I get back to the office. The drive is 3tb and set to mirror the data (1.5tb of data duplicated onto a second drive 1.5tb drive) At each process, the properties of the original and the copied folder are checked for physical size and number of files.

Drugs or not, you can never be too paranoid when it comes to your data.........
 
You are lucky if you get a 1 day turnaround. We have a customer with a load of machines on one order, next day hardware engineer support from HP.
Takes 1 day for HP to log the call, another day for them to respond and ask for diagnostics to be run (red light on the disk, and the disk not being there isn't enough), and then 2 days to ship the spare part. Then they start chasing for the part to be returned even before the good one has arrived sometimes!
HP currently appear to be pulling back from being helpful with support at the moment (to the extent that I am advising customers to have onsite spares).

Anyway, digressing from the point somewhat.

our EVA reports errors directly to HP, then HP call me and tell me the part/engineer is on order :P
 
Not entirely sure that's true. I believed if you overwrote a file it was lost - or it would be possible to retrieve every single bit of data ever written to the drive.

And I am not sure £2-300 would get a drive recovered, but don't know for sure as I have everything backed up :)

Graham

If you can be bothered, it is possible to recover data from a hard disk after around 7-8 writes/overwrites with other files. However this requires true forensics equipment and opening the drives. There are a couple of labs in the UK which offer this, but most of them are in the US, starting at around $5k a drive, more for raid sets.


You are lucky recrudesce that you have that service. The system I am thinking of cost ~800k and they didn't get that.

Diddy, you want to burn them to film?? :runaway: :)
I suppose, you could get an archival tape. They are able to store many GB of data, and are meant to be stored for long periods, usually in a firesafe. They are slow to write to, and even slower to read from (single files anyway), as the tape has to search for the correct place to read the file from.
 
You are lucky recrudesce that you have that service. The system I am thinking of cost ~800k and they didn't get that.
srsly ?! i spent £60k on mine, and it came with an app called ISEE that monitors everything and reports problems back to HP. clever !
 
Not entirely sure that's true. I believed if you overwrote a file it was lost - or it would be possible to retrieve every single bit of data ever written to the drive.

And I am not sure £2-300 would get a drive recovered, but don't know for sure as I have everything backed up :)

Graham

Graham,

I am not completely up to speed but my Dad works for a big computer forensic company and they are able to get a lot of data from a disk, believe me I have worked there :suspect:

I was once told that when you format something you don't actually delete anything other than details on where the data is on the disk. Its like taking a book and ripping out the content page. Its all there you just don't know how to get there.
 
No, but you are going to tell us aren't you :p

Sorry forgot the thread linky this explains it kinda, I know they exist I don't know much else as I've never had anything to do with it

Hi David101, unfortunately you are not correct there. Raid 0+1, you /can/ loose two drives and not loose data, but in reality, because of the way the extra load is put on the second disk in the pair, you will more often loose data after the second disk failure.

DiddyDave. There is nothing wrong with taking lots and lots of backups. You can never be too safe. At the moment, I am considering having my main data on a raid 5 system, as my main working system. Then I will take a backup to an external disk drive. Once a month, I am going to clone that external drive to an offsite area (well, in fact I am going to retrieve the offsite disk drive, and swap it with the current external one). This need not be expensive, consider a family member or a friend who doesn't mind you popping in once a month. The reasoning behind this is if you are burgled or the premises are burnt down. Also, each month, I can check the consistency of the offsite backup

On the raid side of things, a quick explanation. These explanations are mainly based on /hardware/ raid controllers. Software raid controllers may not offer all of the benefits (hardware raid is a separate device inside the computer, software raid the operating system performs the tasks). Also assumed that all disks are the same size. With hardware raid based systems, you can usually take disks out and replace them whilst the machine is still in use. (not with a raid 0 of course)

Raid 0. This offers no redundancy, but allows you to read and write data faster. Data is interlaced between disks, You can have multiple disks in the raid set (2+). The speed benefit is gained by having twice as much cache local to the disks. Cache being a buffer memory based on the disk controller itself. Often called stripe raid. Storage size is about the size of all the disks added together.

Raid 1. Mirror. Data is sent to both disks at the same time, in a mirror. If one disk fails, the other is used immediately. Often raid controllers will continually/schedule checks to ensure that the data on the disks match, and alert you if there are any errors likely to be encountered (not an awful lot it can do to fix them though). Mirrors have to be in pairs.

Raid 0+1. This is a stripe of mirrors. This gives you the resiliency of a mirror, you can loose a disk, but with the extra speed performance of a stripe. However, experience shows that should one disk fail, the other disk in its mirror, starts to get a lot more use. This can cause this disk to fail quicker than the other two disks left, meaning that you will more often than not, loose data on a second disk failure. For raid 0+1, you have half the disk space, minimum requirement of 4 disks, any even number above that is valid.

Raid 3. This raid system is no longer used (in preference to raid 5 if it exists), it is however easier to explain raid 3 then raid 5. With raid 3, the computer writes the data to all of the disks less 1. So if you have 4 disks in a raid 3, the data would be written to disk 1, 2, 3 . For disk 4, a checksum of the data is written. This checksum allows the computer to calculate whether any of the items, 1, 2, 3 have changed, and if they have, it can re-create the value that they had (conversely, of the data on disk 4 has changed, it can recreate that). With a raid 3, if one disk fails, then there is no loss of data. You have to have a minimum of 3 disks. You loose one disk space for the parity information.
The downside found with raid 3, was that when a disk fails, the amount of extra usage put on the disk for the parity calculations, caused it to fail quicker.

Raid 5. This is a similar theory to raid 3. The difference is that instead of there being 1 disk reserved for parity, the parity is interspersed with the standard data. Similar to the below (A,B.C being the data, P being the parity).
ABCP
BCPA
CPAB
PABC

Most (proper) hardware raid systems will also perform consistency checks of the data, and should there be a problem showing with the disk, they will copy the failing disk to a spare disk within the system, usually called a hot spare. It isn't infallible, but around 80% of the returns I see are from systems which have correctly determined an upcoming failure and automatically copied and replaced the failing disks..


I do not understand RAID, I'm an engineering guy not a comp sci guy :D

my mate does comp sci and advised me to go raid 0+1 as it was fast and had a backup, soon after isn't an issue as long as its not 20 mins after as when a drive fails I'll order another sharpish to replace it. Also this is for personal use and I have a weekly off site (well during term time its off site) backup to my home server which gets built when this computer gets built
 
Even if a hard drive dies, the data is still there not matter what people say. You can format it put 1's and 0's over it but it is still retrievable unless it gets stolen.

You can get it imaged to retrieve the data but I am not sure how expensive this is, but it has to be cheaper then 4 backup 1tb drives???

No way- 4 x 1tb hard drives, talking about £80 a drive, £320. Hard drive recovery, removing platters in a clean room etc, talking £1000(ish).
 
hah, i got HP themselves to come and put my SAN in :P rawr !
 
I was once told that when you format something you don't actually delete anything other than details on where the data is on the disk. Its like taking a book and ripping out the content page. Its all there you just don't know how to get there.

But if you write 0's to each and every sector (as is an option on formatting), it might be possible but only for the specialists or NASA. OK, I don't know, but I know you can't recover files yourself if you just overwrite the whole disk.

But anyway, having a backup is way easier!

Graham
 
srsly ?! i spent £60k on mine, and it came with an app called ISEE that monitors everything and reports problems back to HP. clever !

Seriously. Had to look up ISEE on the HP website, apparently it was retired 1st June 2009, and the replacements are Windows only (which these machines cannot run). I guess that is why HP were keeping quiet.
The other difficulty being that these machines do not have a connection to the internet.

Blue_max:
if there are only 0's written to every sector of the disk, then there are 'tools' 'available' which allow a user to change the reading threshold of the heads. This would allow for a large amount of the data to be retrieved. For a lost file it is not worth bothering about. However, for a person of nefarious habits, a lot of personal information can be gleaned quite easily from a discarded disk.
 
Back
Top