How to stay online most of the time?

Forum Moderators: phranque

Message Too Old, No Replies

How to stay online most of the time?

Do you have a plan for those cases where a server fails?

explorador

8:16 pm on Jun 23, 2011 (gmt 0)

I wasn't sure where to put this (apache of hardware, general..) ok.

There is little we can do when a server goes offline, it involes DB, mail, http, FTP, etc. And there is no perfect isp that can offer 100% uptime, one day it will hit you and lets hope it doesn't happen in the middle of something important.

My approach: my critical sites are on two separate servers, one is live and when it fails, I can mod the DNS to point to the other server. Yes it has its issues but it has saved the day a few times. It is very easy with ONE site having subdomains or subfolders.

I now have some other webs (more domains) so its kinda hard to keep up with it (keeping the backup server up to date, always). So here comes my question, what do you do? do you have some plan b?

wheel

8:46 pm on Jun 23, 2011 (gmt 0)

Well first off I'm not amazon. If I'm offline for 3 hours in the middle of the night, it's not mission critical. If I was offline for a day, it wouldn't sink my business either. I suspect other businesses are in the same boat if they thought about it. Heck, it beats having a B&M where tear up the road out front for six months.

But I make daily incremental backups of all data on my server; pretty much everything below /home, /etc, and /var/lib/mysql (web files, server config files, database files).

I then have a cold spare of my webserver.

So if hardware fails, I've got a spare everything to pull. I can drop the hard drives into the cold spare, or pull ram from the spare, or power supply or whatever.

If the host went offline - and that's virtually impossible given where I'm located - I'd do a fresh install of linux on the cold spare, install the rest of the files from backup, then start calling colo facilities. I'd probably be offline for a long day.

It's cheap and effective since it's basically the cost of a spare server (and I'm not running fancy servers, I think they're like 6-7 years old). And covers off everything but the most remote of probable problems.

LifeinAsia

9:12 pm on Jun 23, 2011 (gmt 0)

As wheel pointed out, people have various levels of tolerance for being down. For some, being down a day or 2 wouldn't matter that much. For others, being down an hour might cost them millions.

Your DR (disaster recovery) plan should reflect that level of hurt. If being down for a day would only cost you a few thousand in lost revenue, paying several hundred dollars a month to maintain an identical hosting setup at a different hosting company may not make sense. However, if you're losing thousands of dollars every hour you're down, spending a few thousand dollars a month for a completely redundant DR site setup would be reasonable.

We have redundant servers. Last year, a bad hard drive crash took out one server, but was able to be mostly back up and running within a few hours on the other server (have since refined the setup so that downtime should be shorter if something similar happened again).

Need to refine the system for if the data center decides to burn to the ground...

explorador

2:37 pm on Jun 24, 2011 (gmt 0)

Nice, thanks, I agree that mostly this will never harm any business so hard at all (at least 1 day). I have one new client that has me a little bit worried, his business relies on email services, heavily, receiving important orders from the other side of the world, even overnight. Sometimes they can loose a sale while the client waits for two hours and gets someone else to do the job, and using alternate emails is not possible (very branded). Anyway not even G or MS servers are bulletproof on their email services.