next »

[walrus]

I'm building a server (hardware) for a db app I'm doing. Fairly low cpu needs, not a lot of data, but there is one specific requirement. The data cannot be lost, corrupted or anything similar over the course of a few years. I doubt anyone will be doing backups. And it would be really really nice if I didn't have to babysit it. My thoughts are get a rack server and put in a lot of hdds, set up zfs zraid and mysql (postgresql?) on a linux box. Or build 2 nas boxes that mirror eachother, each with raid on them and have the 'main' server read the data from both and compare it. It has to be really really redundant. I'm even considering SSD for storage but I don't have a clue about the lifetime of those things and google isn't really helpful.

[kurdt]

Maybe leverage Amazon's S3 for data storage? That's about as redundant as you can get with relative low budget. Of course S3 makes sense if you don't have a need to move a lot of data. Then it starts to get really expensive...

[walrus]

Thanks, but it's gotta be local storage.

[kurdt]

Thanks, but it's gotta be local storage.

Well, then you are pretty much going for RAID in the server itself and backups. You have to use at least two different locations for backups. And remember that backup isn't a backup until it's tested to be valid backup so get some sort of checksum check at least.

[nutballs]

The best way to save data is to not create it in the first place

Raid is not an option, it's a requirement on a local machine. And it is NOT a backup solution or archive. It is a panic reducer for when a drive fails.
If this data is truly critical.
Raid 10 at minimum.
If you can afford to lose some write speed, and rebuild times being longer, raid 50.
Raid 60 is even more paranoid, but slower.
Either way, the more hot spares the better. That means you don't need to panic to replace a failure. But obviously should got to it asap.

Your options for duplication are...

Rsync. But that is whole files only I think, which means if your db is huge, it might take over a day to backup. And it's on a schedule not real time. There may be a way though, but I don't think so for databases with rsync.

Drbd. Realtime block level replication to another node. This is to copy entire volumes, and changes to those volumes, real time to a drbd slave. it's a pain in the ass and picky.

If you you just need the database to replicate, then all databases have a replication system built in.

If the files can wait until each night, like the web directories for example, then I would recommend a combo of whatever the database replication is and rsync for everything else. It is s more proven method than drbd I think.

[walrus]

I'm investigating http://en.wikipedia.org/wiki/ZFS  but I haven't used this thing before and if it blows up, I won't know how to fix it.
 
Mysql can be 'rsynced' with mysqlhotcopy, but it also supports replication.

As insane as drbd looks, I'm gonna look into it because it might be a solution.

Thank you!

...
I'm getting the feeling that I'll end up with a room full of hard drives and boxes and my main concern will be cooling them down....

[vsloathe]

The problem with going redundant is that no matter how redundant you go, it's not a backup. And you're relying on sometimes only semi-competent people to pay attention to that annoying beeping noise when a drive fails, crack it open and replace it. I had my former place of employment call me a few months ago because they'd lost a lot of data. I asked how that happened when I had installed a RAID 5 array in the one small server they use. They had no idea, but said it had been making a really high pitched beeping for a while 

[Sade]

i think your storage device memory is low, not up to the mark, it can never be store more data, If it can then for just temporarily purpose..! Check it and renew.