Big Gamble, Big Loss: the Fisher Plaza Datacenter Fire

Disaster recovery, it is one of those IT 101 issues, there is no viable good reason to not have a hot/cold spare site in the day and age of cloud computing, data centers stitched across the planet, and while it is not trivial, the ability to synch all those systems into a fail over cluster.

I have held off on writing this because after 22 years in IT, Disaster recovery, practiced and well understood disaster planning is part of the game. There is no reason, no good viable reason when you are balancing uptime against downtime to ever not be prepared for a major disaster. The Fisher Plaza fire on the July 4th weekend was just such a disaster, and unfortunately it caught a lot of companies flat footed and relying on a single data center. When that data center goes down, business stops, and you have to wonder what the dollar per minute loss of revenue for those companies was, in total, for everyone impacted by the fire.

There is no viable excuse for not having a redundant system, even if your backups are in the cloud on AWS or on Azure, or any other cloud system including rack space. This is a matter of decision making on the part of all the business owners who believed that one data center was enough. The sad part is that the Fisher Plaza fire this week truly showed that there was a clear lack of thinking in terms of data recovery and disaster preparedness. That is the sad part, one of the biggest holiday weekends, and many companies were out of business, some might never recover. Penny Wise and Pound Foolish decision making, which might be the end of some of these companies.

Techflash has a great article this as well, talking to a number of disaster recovery people from around the region. In an interview with Jessie Robbins who worked at Amazon as their master of Disaster Recovery, you need to really understand this statement:

“You can expect an outage in a data center every couple of years even in the best-engineered facilities,” said Robbins, whose Amazon nickname was Master of Disaster. “The people that I hold responsible for outages are not the data centers but the people who built systems that rely on a single data centers or that depend on disaster recovery plans that they write once and never put into practice — and end up getting caught with their pants down.” Source: Techflash

Realistically then, if you are looking at this kind of issue, you do need to pay attention to what you are doing, and have a practiced and viable disaster recovery plan in place, there is no excuse, there is no reasonable reason that someone can come up with not to have a disaster recovery plan in place. There is no reasonable excuse for not practicing your data recovery plan. While you might be looking to save money, AWS is cheap, efficient, and available. This does not mean that you should not make backups from AWS, but your disaster recovery plan has to be in the cloud, and there has to be a fail over clause in your DNS. As you grow, you might find that you are sharing data between systems regularly, allowing for growth, synching, and fail over clustering, is the best thing you can do for your business.

(Cross-posted at TechWag)