Where do you guys and gals back up your data? When PCs are now running 2 to 4 TB drives where do you put all that? (I just lost all my digital photos recently and still haven't gotten over that)
A good backup prgram can normally cut your files in half if not more. I host an Off Site FTP backup Server for my clients and use ot for myself also. I have several 3 TB Dirves (Internal) that I use RAID on and then I do the same with external USB drives. The software I use allows me to use the external drives as a "mirror" for the internal ones. As of the past 8 years this has worked well. I have occassionally had to replace my drives with larger ones but this is easily done with the software. A few weeks ago I got a scrreming deal on 5 Ea. 4 TB external WD drives from Newegg.com. If I had been allowed to buy more i would have.
So, In short, drives are cheap these days compared to your data. Even using the free MS backup software is better than nothing.
How do I back up my data? Redundantly! And it's not a blanket backup, either. I have one directory whose subdirectories contain the important stuff. The rest I could survive without if I have to.
I have historical data backed up on data DVDs stored in a safe.
Every two weeks I use a program called Retrospect to do incremental backups of my entire C: drive.
I also use a synchronizing backup program almost daily to back up financial and frequently changing files to two different external drives.
I also have all the Microsoft updates for XP ever applied to my computer (over 7GB) and a Scheduled Task that backs up the SystemState (registry and certain files) daily using ntbackup.exe (into a folder holding 31 days of backups - 15GB).
May be a bit overkill but 29 years as a programmer taught me a few lessons...
(I just lost all my digital photos recently and still haven't gotten over that)
I'm not sure how I'd get over that. I used to back up my hard drive to an external raid. And the external raid got its NTFS mangled, which naturally make both hard drives equally unusable. I no longer use a raid. Instead I back up to two singleton hard drives, a manual raid1 if you will. My backup drives are 2TB each and they are around 90% full right now. My hard drive is just 500GB and only about half full, because lots of things just go straight to the external drives. This is completely unscalable!
A Dell PCIe RAID controller can be bought on ebay for 20 bucks. This is a card that was probably around $500 new, it's a true hardware RAID0/1/5/10 card supporting up to 8 drives. So I've ordered one and I'm going to move those two external drives to internal, and add two more. Not sure I have room in the case. May have to build an open frame stack of drives behind my tower PC with the Sata and power cables draping out an open empty slot.
My mac mini has a 750 GB HDD I use time machine every now and then to some exxternal 1 TB HDDs that are stored secure in town for me .
every other backup I do a fomat of the target drive and I do a bit for bit backup . I never use the same drive 2X in a row .
this way IFI mess up the backup I have a total physical sep copy ....... also every time I move like state to state and life changes I do a HUGE backup to a USB HDD . this way I have a captured time in space for all kinds of reasons of what I had Up to that point in my life
at 100 bucks a tera and the fact I have things I can never afford to loose a few franklins a year for pice of mind is cheap ...
.
Also I use DVD RAM for smaller stuff and dedicated Photo drive use ;... every now and then Illl copy my internal photo folder to a dedicated drive . again 100 bucks is cheap .
simply put .. there woud have to be one heck of a flood and fire to make my stuff lost ..
That's the thing about digital photos. An old-time photo album is like a permanent archive. Storing digital photos is more like juggling. You have to keep tossing the balls in the air to keep the data from decaying over time. Even CDs and DVDs are not archival.
I have a picture of my grandfather taken more than 100 years ago. How many photos taken today will be around 100 years hence?
In addition to Time Machine backups I also copy to a separate external HDD my photos, videos and other important docs. Also, my SmugMug account serves as yet another backup, as does DropBox.
I have done some thinking over the years and One thing I have thought of is that the backup stress on a system it self can trigger a crash !
It has happend to me once . when I backup It takes hours ..And that can really warm up the old CPU as much as a good video render .
How many photos taken today will be around 100 years hence?
I would imagine that there were few pictures taken of your grandfather, and the picture you have now was treated as something very valuable. Nowadays people have hundreds or thousands of pictures. The incredible number of photos ensures that at least some of them will be around for eternity - barring any apocalyptic events. That is, unless you keep all your pictures in one location... same deal for 100 year old pictures though, in fact you may want to make some high quality digital copies of that picture as a backup.
Depending upon how you spread your photos around they may belong to someone else by then - facebook, instagram or whatever company has terms of service that claim ownership of whatever is submitted to them.
You can (and probably should) make archival hard copies of your important photos.
Photos and the like are backed up across three machines, one of which is only used for backups. I also use an ancient Egyptian technology commonly known as paper for schematics, code and other project archiving.
My son dropped his laptop (an HP machine) in this week "because it has stopped working". The first thing I did was pull the hdd and put in a data dock and started to back it up. And a really weird thing has happened; the first partition (system) was empty. And part way through the second partition an copy error came up. And this partition was empty. And now the remaining two partitions are empty. Odd.
Depending upon how you spread your photos around they may belong to someone else by then - facebook, instagram or whatever company has terms of service that claim ownership of whatever is submitted to them.
THIS !!!!!!!!!!!!
heck for my SLRS I just buy new cards every few years ..... they get to be so cheap and unless I am doing a event or sports where I use the 8 FPS shutter .... I dont shoot more then a few GB a year! ......... S committing my fave shots to the card they were shot on is not a big deal .
right now I have a HUGE!!!! task , I need to import all my DV and HDV footage and archive it as I am gonna to tapeless with my HD cams very soon .
This is a frequent topic of discussion in my house. My wife is very concerned about potential data loss of memories of our daughter. She diligently maintains multiple forms of recordkeeping for my daughter's life. Pictures, videos, journals, scrapbooks, etc, etc.
She currently has all photos related to my daughter backed up in three places. Two separate external drives and on our 4 TB NAS. Her laptop contains what we call "originals", which just means the first copy made from the camera's card. This has progressively gotten more difficult to manage as we have improved our devices. The 172 pictures taken the day my daughter was born average 2MB in size, whereas pictures from the past 2 years average 5+MB thanks to camera upgrades. Same goes with video since we now record in full HD.
On that note, however, I haven't backed up my laptop in a very long time. Think I should do so very soon.
One thing I do not recommend is to use on-line backup .. What I mean is . don't leave your backup drives plugged in...... or worse turned on .......
I have come to the opposite conclusion. A powered down back up is a dead back up.
What I mean is:
1) I'd like to see all my backup devices on and connected all the time.
2) My media is to be connected to different machines in different locations.
3) There should be many of them.
4) They should be actively checking against each other to see if anyone of them is down, faulty, or some how corrupted a file.
5) I should be able to connect new storage to this setup and have it replicate the groups idea of correct files automatically.
How have I come to this conclusion?
I'm going to assume that drives and machines have short lifetimes or high error rates. No matter if they are switched on or not. To defend against that I need many machines each with it's own storage. That is the only way to get the probability of catastrophic loss down to vanishingly small numbers. This is all hard to manage manually so they should be actively sharing and verifying that data.
Power surge and bus surge are a huge problem !
I backup every month . so my drives see 6 cycles a Year .
If they fail with only 6 loads and unloads then I have just won the powerball jackpot of MTBF .....
We here are at the worse place to have connected backups .... Unless they are on a fiber optics cable .
with all the messing we do with USB to SER to funky breadboards ........ you can send one heck of a surge up a USB port and on to all your drives ........... I had a NI DAQ that smoked every 5 V bus trace off a dell MoBo when we used the wrong pin for GND. We shoved 48V Up the USB cable ....... and what do ya know , the 1 TB 3.5 inch WD HDD that was plugged in was smoked too !
Also any SW glitch from you doing installs or updates can mess up every copy in your system ....
I have adobe premere elements on my PC lappy its V 8,0.......... the thing wants me to update it to 8.01
I wont as i did and it broke HDV imports off my HD cam ..... No MD5 checksum auto verify program is gonna catch a glitchy updates .
I encounter people that already have a backup solution and some hardware redundancy. It's great, but there is still a problem. Not many people ever recover their backup data. They get an email or check some log file saying the backup is completed. It is not until they try to recover their data they realize hey, these files are corrupt maybe volume shadow copy wasn't running and files were open. Locked databases, users with files open, etc etc...
When recovery audits check good, your backup is good.
In my scheme I don't care. Perhaps I did not put it correctly. The idea is my storage is connected to different machines and those machines are in totally different locations, different countries even, different cloud servers with different companies. Where ever.
Now, a power surge can take out any node at any time. That's just another kind of hardware failure.
All nodes are alive and communicating all the time so they can constantly check who is alive and correct.
Software glitches due to updates or whatever are also just another kind of node failure. The rest of the nodes continue regardless.
That is another very good reason I want the system I describe. Every copy is verified all the time. Effectively there is no backup, all data is live all the time.
I can get close to this using say git for source code management. There are many git repositories with the code base and they are all individually fully fledged repos. It's easy to check they are all in sync. anyone of them could explode at anytime and it's no big deal.
Similarly unison can do that for other types of data.
git and unison work well but still require a lot of manual intervention.
Software glitches due to updates or whatever are also just another kind of node failure. The rest of the nodes continue regardless.
how then does one keep V 8.01 from then being backed up ? and spreading to the other nodes ?
with Linuux and the like you can just use the changelog . but on the more mainsteam OSes you cant as easy .
A Dell PCIe RAID controller can be bought on ebay for 20 bucks. This is a card that was probably around $500 new,
These are actually Adaptec RAID Cards. Dell just bought and branded them. However, They are the best I have ever used and yes, they use to be very expensive and some still are especially for SATSCSI. Nowadays though a large majority of motherboards have onboard SCSI controllers. Using RAID internally and external USB drives as mirrors for the RAID drives works out well for me.
One thing that you need to make absolutely sure of is that you can restore what you have backed up!!!! I had a client that took care of their own backup years ago and when it came time to do a restore it failed. It was then that they switched to my Offsite backup service. A couple years after that their Exchange Server crashed. Has anyone ever had to restore an Exhange Server or SQL Server. Not the most fun an IT guy could ever have!!!! However, because of the software I use I was able to have them back up and running after the weekend. I also recently had one of my own systems start having MS update issues so I retored using my own backup to a month earlier. I normally do not push software but I really like BFW. It allows you to do Disaster Recovery, Undelete files and best of all, File Roll Backs. This means if you have a file that you edit a lot or whatever and overwrite it, you can roll back to any date back to the first known backup. My Backup Server is just that, all it does is stores backups for myself and my clients. I do need to look in to having another off site location to backup my backup server to but that will have to wait. Trying to backup 15 TB of files over the Internet would be quite time consuming even if I were to switch to fiber Optic. I only have DSL now!!! ( I see a powdered fire extinquisher project in my future!!!!!)
I like Peter's idea of an ultra-high-density fuse-programmed ROM. In a world that seems to have an answer for every conceivable electronic whim, it is curious that there is no such thing as a sure-fire, inexpensive, and dense 100-year storage medium.
sandisk does make OTP SD cards for CSI stuff . BUT they are not any more cheap then flash .
and they need a cam firmware to be tweaked,.
but serious ! a OTP SS drive might be a idea !
I can name a few uses right now.
Use #1 ...... a forever growing Photo repo for my works .
Use #2 I want to make a clone of my XP install AFTER Its all tweaked so that when MS dumps it I have a custum distro with all the updates installed and the programs I most use readdy to run .
this way I can just re-install my OS and its apps every term to keep a fresh system . just like OEMs do .
and use #2.5 is do secure linux boot on media faster then a CD-R . !!!!!!!!!!!!!!!!!
( does any one know where to get USB sticks with HW write protect switches on them like they used to .
Use #3 archive all my old HDDs from my youth in to ROM so I can enjoy the data on them with no harm to the original.
AFAIK the only way these days to make a WORM system is with opitical media ... and that media is NOT made for boot uses .
..how then does one keep V 8.01 from then being backed up ? and spreading to the other nodes ? with Linuux and the like you can just use the changelog . but on the more mainsteam OSes you cant as easy .
I'm sorry Peter I don't know what you are talking about. Is that "V 8.01" referring to some broken update/version of Windows or something completely different?
What "change log"? My Linux has no change log. If I delete a file it's gone, if I overwrite a file the old version is gone (barring use of disk/fs recovery tools which I really don't want to get into.
I'm not sure how "mainstream" an operating system you want Linux to be given that it already runs on huge proportion of the worlds interesting machines from phones to supercomputers. Anyway no matter, the OS that is taking care of your valuable data need not be the same one facing you in your monitor everyday.
I digress. Some questions:
1) Assuming you are referring to propagating a failed OS update why are you backing up an OS anyway? I nuke my OS quite often, sometimes due to trying over adventurous updates to testing versions, sometimes due to "user error", sometimes due to disk failure etc. No matter, just reinstall it from the net, pull my data back, and continue. It's a pain to get everything back how I like but that is the price I pay for being brutal to my machine.
I would never imagine an OS update on all my machines at the same time. My OS should do it's best to prevent itself being changed by anyone except me when I say so.
2) If the software you are using is not doing the right thing, propagating bad files around, why are you using it? Tools like unison do not do that.
A couple of surprising results I take away from it:
1) Elevated temperature, within reason, does not affect disk failure rate. In fact colder can be worse.
2) Increased activity does not affect failure rate much.
Makes me even happier about my plan for always on multipe node systems that constantly sync and check with each other.
3) SMART statistics are pretty much useless. About half their failed drives had no errors reported by SMART.
4) On the other hand as soon as you see a single error in SMART you know your drive is 20 or more times likely to fail soon.
Actually I have trouble reconciling those two facts.
RDL2004
Posts: 2,554
Comments
So, In short, drives are cheap these days compared to your data. Even using the free MS backup software is better than nothing.
-Phil
Every two weeks I use a program called Retrospect to do incremental backups of my entire C: drive.
I also use a synchronizing backup program almost daily to back up financial and frequently changing files to two different external drives.
I also have all the Microsoft updates for XP ever applied to my computer (over 7GB) and a Scheduled Task that backs up the SystemState (registry and certain files) daily using ntbackup.exe (into a folder holding 31 days of backups - 15GB).
May be a bit overkill but 29 years as a programmer taught me a few lessons...
I'm not sure how I'd get over that. I used to back up my hard drive to an external raid. And the external raid got its NTFS mangled, which naturally make both hard drives equally unusable. I no longer use a raid. Instead I back up to two singleton hard drives, a manual raid1 if you will. My backup drives are 2TB each and they are around 90% full right now. My hard drive is just 500GB and only about half full, because lots of things just go straight to the external drives. This is completely unscalable!
I found an article on the web:
http://www.servethehome.com/dell-perc-5i-raid-controller-cheap-raid-5/
A Dell PCIe RAID controller can be bought on ebay for 20 bucks. This is a card that was probably around $500 new, it's a true hardware RAID0/1/5/10 card supporting up to 8 drives. So I've ordered one and I'm going to move those two external drives to internal, and add two more. Not sure I have room in the case. May have to build an open frame stack of drives behind my tower PC with the Sata and power cables draping out an open empty slot.
every other backup I do a fomat of the target drive and I do a bit for bit backup . I never use the same drive 2X in a row .
this way IFI mess up the backup I have a total physical sep copy ....... also every time I move like state to state and life changes I do a HUGE backup to a USB HDD . this way I have a captured time in space for all kinds of reasons of what I had Up to that point in my life
at 100 bucks a tera and the fact I have things I can never afford to loose a few franklins a year for pice of mind is cheap ...
.
Also I use DVD RAM for smaller stuff and dedicated Photo drive use ;... every now and then Illl copy my internal photo folder to a dedicated drive . again 100 bucks is cheap .
simply put .. there woud have to be one heck of a flood and fire to make my stuff lost ..
I have a picture of my grandfather taken more than 100 years ago. How many photos taken today will be around 100 years hence?
-Phil
and a file format that can do WORM in a way where you dont have to burn all the rom at once
It has happend to me once . when I backup It takes hours ..And that can really warm up the old CPU as much as a good video render .
I would imagine that there were few pictures taken of your grandfather, and the picture you have now was treated as something very valuable. Nowadays people have hundreds or thousands of pictures. The incredible number of photos ensures that at least some of them will be around for eternity - barring any apocalyptic events. That is, unless you keep all your pictures in one location... same deal for 100 year old pictures though, in fact you may want to make some high quality digital copies of that picture as a backup.
Depending upon how you spread your photos around they may belong to someone else by then - facebook, instagram or whatever company has terms of service that claim ownership of whatever is submitted to them.
You can (and probably should) make archival hard copies of your important photos.
My son dropped his laptop (an HP machine) in this week "because it has stopped working". The first thing I did was pull the hdd and put in a data dock and started to back it up. And a really weird thing has happened; the first partition (system) was empty. And part way through the second partition an copy error came up. And this partition was empty. And now the remaining two partitions are empty. Odd.
Will have to crack out Phoenix Recovery.
THIS !!!!!!!!!!!!
heck for my SLRS I just buy new cards every few years ..... they get to be so cheap and unless I am doing a event or sports where I use the 8 FPS shutter .... I dont shoot more then a few GB a year! ......... S committing my fave shots to the card they were shot on is not a big deal .
right now I have a HUGE!!!! task , I need to import all my DV and HDV footage and archive it as I am gonna to tapeless with my HD cams very soon .
She currently has all photos related to my daughter backed up in three places. Two separate external drives and on our 4 TB NAS. Her laptop contains what we call "originals", which just means the first copy made from the camera's card. This has progressively gotten more difficult to manage as we have improved our devices. The 172 pictures taken the day my daughter was born average 2MB in size, whereas pictures from the past 2 years average 5+MB thanks to camera upgrades. Same goes with video since we now record in full HD.
On that note, however, I haven't backed up my laptop in a very long time. Think I should do so very soon.
might as well make that MTBF go sky-high!
I am in the middle of a huge upgrade on my desktop and http://www.mdisc.com/ is on my list to do .......
Yank out the old opti drive and get one of those .
I have come to the opposite conclusion. A powered down back up is a dead back up.
What I mean is:
1) I'd like to see all my backup devices on and connected all the time.
2) My media is to be connected to different machines in different locations.
3) There should be many of them.
4) They should be actively checking against each other to see if anyone of them is down, faulty, or some how corrupted a file.
5) I should be able to connect new storage to this setup and have it replicate the groups idea of correct files automatically.
How have I come to this conclusion?
I'm going to assume that drives and machines have short lifetimes or high error rates. No matter if they are switched on or not. To defend against that I need many machines each with it's own storage. That is the only way to get the probability of catastrophic loss down to vanishingly small numbers. This is all hard to manage manually so they should be actively sharing and verifying that data.
I backup every month . so my drives see 6 cycles a Year .
If they fail with only 6 loads and unloads then I have just won the powerball jackpot of MTBF .....
We here are at the worse place to have connected backups .... Unless they are on a fiber optics cable .
with all the messing we do with USB to SER to funky breadboards ........ you can send one heck of a surge up a USB port and on to all your drives ........... I had a NI DAQ that smoked every 5 V bus trace off a dell MoBo when we used the wrong pin for GND. We shoved 48V Up the USB cable ....... and what do ya know , the 1 TB 3.5 inch WD HDD that was plugged in was smoked too !
Also any SW glitch from you doing installs or updates can mess up every copy in your system ....
I have adobe premere elements on my PC lappy its V 8,0.......... the thing wants me to update it to 8.01
I wont as i did and it broke HDV imports off my HD cam ..... No MD5 checksum auto verify program is gonna catch a glitchy updates .
Peter...
When recovery audits check good, your backup is good.
In my scheme I don't care. Perhaps I did not put it correctly. The idea is my storage is connected to different machines and those machines are in totally different locations, different countries even, different cloud servers with different companies. Where ever.
Now, a power surge can take out any node at any time. That's just another kind of hardware failure.
All nodes are alive and communicating all the time so they can constantly check who is alive and correct.
Software glitches due to updates or whatever are also just another kind of node failure. The rest of the nodes continue regardless.
Indeed, how do you your backups are good?
That is another very good reason I want the system I describe. Every copy is verified all the time. Effectively there is no backup, all data is live all the time.
I can get close to this using say git for source code management. There are many git repositories with the code base and they are all individually fully fledged repos. It's easy to check they are all in sync. anyone of them could explode at anytime and it's no big deal.
Similarly unison can do that for other types of data.
git and unison work well but still require a lot of manual intervention.
how then does one keep V 8.01 from then being backed up ? and spreading to the other nodes ?
with Linuux and the like you can just use the changelog . but on the more mainsteam OSes you cant as easy .
One thing that you need to make absolutely sure of is that you can restore what you have backed up!!!! I had a client that took care of their own backup years ago and when it came time to do a restore it failed. It was then that they switched to my Offsite backup service. A couple years after that their Exchange Server crashed. Has anyone ever had to restore an Exhange Server or SQL Server. Not the most fun an IT guy could ever have!!!! However, because of the software I use I was able to have them back up and running after the weekend. I also recently had one of my own systems start having MS update issues so I retored using my own backup to a month earlier. I normally do not push software but I really like BFW. It allows you to do Disaster Recovery, Undelete files and best of all, File Roll Backs. This means if you have a file that you edit a lot or whatever and overwrite it, you can roll back to any date back to the first known backup. My Backup Server is just that, all it does is stores backups for myself and my clients. I do need to look in to having another off site location to backup my backup server to but that will have to wait. Trying to backup 15 TB of files over the Internet would be quite time consuming even if I were to switch to fiber Optic. I only have DSL now!!! ( I see a powdered fire extinquisher project in my future!!!!!)
and they need a cam firmware to be tweaked,.
but serious ! a OTP SS drive might be a idea !
I can name a few uses right now.
Use #1 ...... a forever growing Photo repo for my works .
Use #2 I want to make a clone of my XP install AFTER Its all tweaked so that when MS dumps it I have a custum distro with all the updates installed and the programs I most use readdy to run .
this way I can just re-install my OS and its apps every term to keep a fresh system . just like OEMs do .
and use #2.5 is do secure linux boot on media faster then a CD-R . !!!!!!!!!!!!!!!!!
( does any one know where to get USB sticks with HW write protect switches on them like they used to .
Use #3 archive all my old HDDs from my youth in to ROM so I can enjoy the data on them with no harm to the original.
AFAIK the only way these days to make a WORM system is with opitical media ... and that media is NOT made for boot uses .
EDIT
Well hot darn ,,, http://www.acard.com/english/newstabpop.jsp?idno=84
that is AWSOME !
and https://www.kanguru.com/storage-accessories/flash-blu2.shtml I guess they still make them .
FYI..
http://en.wikipedia.org/wiki/Media_preservation
infact I dont like big HDDs
more eggs in one baskit
http://news.cnet.com/8301-21546_3-10309283-10253464.html
http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/disk_failures.pdf
What "change log"? My Linux has no change log. If I delete a file it's gone, if I overwrite a file the old version is gone (barring use of disk/fs recovery tools which I really don't want to get into.
I'm not sure how "mainstream" an operating system you want Linux to be given that it already runs on huge proportion of the worlds interesting machines from phones to supercomputers. Anyway no matter, the OS that is taking care of your valuable data need not be the same one facing you in your monitor everyday.
I digress. Some questions:
1) Assuming you are referring to propagating a failed OS update why are you backing up an OS anyway? I nuke my OS quite often, sometimes due to trying over adventurous updates to testing versions, sometimes due to "user error", sometimes due to disk failure etc. No matter, just reinstall it from the net, pull my data back, and continue. It's a pain to get everything back how I like but that is the price I pay for being brutal to my machine.
I would never imagine an OS update on all my machines at the same time. My OS should do it's best to prevent itself being changed by anyone except me when I say so.
2) If the software you are using is not doing the right thing, propagating bad files around, why are you using it? Tools like unison do not do that.
That's an interesting report from Google.
A couple of surprising results I take away from it:
1) Elevated temperature, within reason, does not affect disk failure rate. In fact colder can be worse.
2) Increased activity does not affect failure rate much.
Makes me even happier about my plan for always on multipe node systems that constantly sync and check with each other.
3) SMART statistics are pretty much useless. About half their failed drives had no errors reported by SMART.
4) On the other hand as soon as you see a single error in SMART you know your drive is 20 or more times likely to fail soon.
Actually I have trouble reconciling those two facts.