The Cache: Technology Expert's Forum
 
*
Welcome, Guest. Please login or register. October 16, 2019, 02:33:43 PM

Login with username, password and session length


Pages: [1]
  Print  
Author Topic: How can I gurantee no data is lost?  (Read 5493 times)
walrus
Rookie
**
Offline Offline

Posts: 46


View Profile
« on: December 07, 2010, 11:35:19 PM »

I'm building a server (hardware) for a db app I'm doing. Fairly low cpu needs, not a lot of data, but there is one specific requirement. The data cannot be lost, corrupted or anything similar over the course of a few years. I doubt anyone will be doing backups. And it would be really really nice if I didn't have to babysit it. My thoughts are get a rack server and put in a lot of hdds, set up zfs zraid and mysql (postgresql?) on a linux box. Or build 2 nas boxes that mirror eachother, each with raid on them and have the 'main' server read the data from both and compare it. It has to be really really redundant. I'm even considering SSD for storage but I don't have a clue about the lifetime of those things and google isn't really helpful.
Logged
kurdt
Lifer
*****
Offline Offline

Posts: 1153


paha arkkitehti


View Profile
« Reply #1 on: December 08, 2010, 01:16:34 AM »

Maybe leverage Amazon's S3 for data storage? That's about as redundant as you can get with relative low budget. Of course S3 makes sense if you don't have a need to move a lot of data. Then it starts to get really expensive...
Logged

I met god and he had nothing to say to me.
walrus
Rookie
**
Offline Offline

Posts: 46


View Profile
« Reply #2 on: December 08, 2010, 01:19:27 AM »

Thanks, but it's gotta be local storage.
Logged
kurdt
Lifer
*****
Offline Offline

Posts: 1153


paha arkkitehti


View Profile
« Reply #3 on: December 08, 2010, 01:55:26 AM »

Thanks, but it's gotta be local storage.
Well, then you are pretty much going for RAID in the server itself and backups. You have to use at least two different locations for backups. And remember that backup isn't a backup until it's tested to be valid backup so get some sort of checksum check at least.
Logged

I met god and he had nothing to say to me.
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #4 on: December 08, 2010, 07:16:55 AM »

The best way to save data is to not create it in the first place Smiley

Raid is not an option, it's a requirement on a local machine. And it is NOT a backup solution or archive. It is a panic reducer for when a drive fails.
If this data is truly critical.
Raid 10 at minimum.
If you can afford to lose some write speed, and rebuild times being longer, raid 50.
Raid 60 is even more paranoid, but slower.
Either way, the more hot spares the better. That means you don't need to panic to replace a failure. But obviously should got to it asap.

Your options for duplication are...

Rsync. But that is whole files only I think, which means if your db is huge, it might take over a day to backup. And it's on a schedule not real time. There may be a way though, but I don't think so for databases with rsync.

Drbd. Realtime block level replication to another node. This is to copy entire volumes, and changes to those volumes, real time to a drbd slave. it's a pain in the ass and picky.

If you you just need the database to replicate, then all databases have a replication system built in.

If the files can wait until each night, like the web directories for example, then I would recommend a combo of whatever the database replication is and rsync for everything else. It is s more proven method than drbd I think.
Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
walrus
Rookie
**
Offline Offline

Posts: 46


View Profile
« Reply #5 on: December 08, 2010, 07:47:48 AM »

I'm investigating http://en.wikipedia.org/wiki/ZFS  but I haven't used this thing before and if it blows up, I won't know how to fix it.
 
Mysql can be 'rsynced' with mysqlhotcopy, but it also supports replication.

As insane as drbd looks, I'm gonna look into it because it might be a solution.

Thank you!

...
I'm getting the feeling that I'll end up with a room full of hard drives and boxes and my main concern will be cooling them down....
Logged
vsloathe
vim ftw!
Global Moderator
Lifer
*****
Offline Offline

Posts: 1669



View Profile
« Reply #6 on: December 08, 2010, 01:29:17 PM »

The problem with going redundant is that no matter how redundant you go, it's not a backup. And you're relying on sometimes only semi-competent people to pay attention to that annoying beeping noise when a drive fails, crack it open and replace it. I had my former place of employment call me a few months ago because they'd lost a lot of data. I asked how that happened when I had installed a RAID 5 array in the one small server they use. They had no idea, but said it had been making a really high pitched beeping for a while  Roll Eyes
Logged

hai
Sade
n00b
*
Offline Offline

Posts: 4


View Profile
« Reply #7 on: April 29, 2011, 12:23:53 AM »

i think your storage device memory is low, not up to the mark, it can never be store more data, If it can then for just temporarily purpose..! Check it and renew.
Logged
Pages: [1]
  Print  
 
Jump to:  

Perkiset's Place Home   Best of The Cache   phpMyIDE: MySQL Stored Procedures, Functions & Triggers
Politics @ Perkiset's   Pinkhat's Perspective   
cache
mart
coder
programmers
ajax
php
javascript
Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS!