So I have a few 5TB external drives with all my media. I mainly just hook them up to my xbox and use kodi to play the files locally. I should probably be investing into a NAS or some sort of JBOD, but that’s a whole other issue.
So no drive is backed up but they are all new (about 6 months old). Obviously, hdd drives can have mechanical failure, be a dud, or just plain suck. I do smart check and make sure the drive health is fine. I have been reading about bit rot and not sure if that is something I have to worry about and an immediate thing (or it’s just some snake oil shit). I want to make sure the data stays readable and in best shape I can have it in for as long as possible. From the reading I have done surfin the web “refreshing” the data is usually done by rewriting the data. I guess what I am worried about is data corruption. With all this being on a HDD, mechanical failure is probably a bigger issue, and is something I should eventually get to with actual backups/parity. Drives are getting cheaper but I don’t have the cash to drop right now on better/larger/enterprise (or NAS) drives to set all that up. I don’t really want to re-download 20+ TB of stuff just to rewrite my data or shuffle data back and forth between my computer and the drive to rewrite. I could be going about this all wrong, so some pointer or input is helpful.
I use CrystalDiskInfo and HD Tune Pro to check drive health, but kinda just wanted to know if there are programs make help against data corruption, if it’s even something that I should be immediately worried about, if I’m going about this in a dumb way, or if I should just start saving and work towards bulding a NAS, JBOD, ZFS or some shit. (if I’m honest all that shit seems out of my ballpark cause I like to just download and play, but it may be time to learn more about all that shit with regards to raid, parity, and having true backups)
tl;dr : I want to keep my shit for as long as possible on my HDDs. Back ups = good, but short of doing that, how to make sure data I currently have stays healthy on current drives?
You want raid plus a filesystem that can detect and correct for bitrot like xfs, btrfs or zfs. If you’re looking to get started try getting a used PC off eBay and try using TrueNAS or something similar to mirror your data too.
I picked up an old PowerEdge on eBay for like $300. Built in raid card. Holds up to like 24 2.5" drives. Works great for what I need.
haha, nice. 2.5s are expensive per Tb though and hardware raid screws you if you want to run Truenas.
I run unraid that does a monthly parity check will that protect?
I’ve never actually used unraid.
As others have said you don‘t need a nas solution like synology. You can just have a cheap computer with enough sata ports and run a nas software, like truenas, on it. I just think that external HDD‘s are not designed to run 24/7 and could therefore be prone to mechanical failure as compareed to datacenter HDD‘s. But thats just an assumption, I have no data to back up this claim.
there’s ways to spin-down usb externals when idle. such as https://github.com/adelolmo/hd-idle
and i think seagate’s (windows) utility can set a timer in hardware on some models.
I went the easy route and bought synology, set up a raid 6, and it saved me within 1 year.
Raid 6 spreads data wide enough to survive 2 concurrent drive failures. I had one drive fail enough to degrade the raid, and while I was awaiting a replacement (warrenty replacememt), a second drive started to fail. I bought a replacement for that, inserted it, then got the first replacement and inserted that. No losses, and I now have a drive in stock waiting for the next failure. It’s been at least 5 years without failure, so I am due.
Schedule an integrity check once a quarter or month to protect against bit rot. The danger with bit rot is you won’t know you have it until drives fail. If integrity is compromised, the shit hits the fan when you are syncing up the new drive, the system cannot recover, and you lose everything. That is when you start over with your verified backup. If that’s bad, goodbye data.
Short of backups, you need checksums and the like to detect corruption, and parity data to repair any corruption that may be found. If you received the files using torrents, those have checksums (SHA-1), so you can just tell your torrent client to recheck.
However, that won’t help you recover data. If the torrent dies, you won’t be able to get the lost data again. This is where parity comes in. It can take many forms, such as ZFS or RAID 5/6. If you simply want a file/folder-level approach, you can build recovery volumes in WinRARv when you compress. But you probably want to create PAR2 recovery volumes instead. I haven’t used it in years, but QuickPAR should still work. The big limitation is that it can’t handle subdirectories, so you need a unique set for each folder.
Side note: my setup is 24/7, mostly consumer-grade, high-quality parts. I’ve never had an issue with bitrot or silent corruption from bad RAM. I have, however lost a lot of data from drive failures, human error, and a particularly stupid decision by WD that led to me having the wrong driver. Don’t try to predict drive failures with SMART and the like. Most failures won’t show in SMART until at least some of the data is already gone. But if you get a SMART error, data loss is imminent.
Look into snapRAID. It does parity based data protection (up to 6 I believe). It’s free, opensource. I use it to run a nightly sync and scrub of ~3% of my total disk space, so in a month it scrubs everything to protect against bit rot. It then shoots me a nightly email with any errors or issues it detects. There is a learning curve, but I’m happy to provide some basic scripts for you to get it running in Windows. You can also run it on top of pooling solution such as Drivepool.
I’ll look into this. I appreciate it. Probably down the road ill set up True NAS or set up some sort of thing with ZFS but have to kinda get more acquainted with all the programs and stuff associated with their use.
