Last Tuesday a ‘perfect storm’ in the form of a double fibre cable fault in Western and Central Australia left almost the entire top end of the country without telephone, internet or mobile communications for about 10 hours.
That pretty much amounts to ZERO communications in this day in age. I woke to find out that we could not call, nor receive calls from any of our family, clients or colleagues. I could not even email them not post a message to a website or twitter to say that we were out of communication.
That is a pretty scary scenario for one so involved in the IT industry, and who evangelises ‘the cloud’. There is nothing more daunting that looking at all mobile phones in the house and seeing ‘no signal’, looking at your browser and seeing ‘This page cannot be found…’ and picking up your home handset to hear…silence. All the major carriers, including Telstra, Optus and Vodafone were down.
I could not even call clients to reschedule appointments, not check on our families to ensure they were OK. We actually had to resort to firing up a radio set to get the latest news on what was happening – something I cannot remember doing for a long, long time now.
Over the past year, we have tried to move as much of our infrastructure out to the cloud where possible, so our office phone system, a hosted Asterisk PABX, was completely down. So was our Basecamp project management site and our hosted Subversion repositories, amongst some other web services that we subscribe to.
My other blogs and some of our development web servers are hosted with Slicehost and GoGrid over in the US, so they were still up for the rest of the world, but frustratingly, I was unable to post any information to them to inform my readers and friends of what was happening. Not to mention I was getting serious withdrawal symptoms from missing my morning dosage of Twitter and Facebook.
Thankfully though (?) our accounting and payroll system is (for now) locally installed on our servers. Although we use online banking for all payments etc. which had to be put on hold.
We also use a web based CRM system, but have it hosted on an internal server, so we were able to continue working in house.
Our Exchange email server is also in house, but we have a backup secondary MX in Sydney which was still faithfully picking up our emails from the rest of the world and waiting until everything got back online here.
This brought back to me the importance of having a safety net of locally accessible data of everything we do in the cloud. Case in point is our SugarCRM system. We debated going to a fully hosted solution, but if we suffered the same sort of fault, then our data would have been trapped at the hosting site, unless we had been doing daily downloads or synchronising with a local server, and had a local copy of SugarCRM ready to get up and running when our external connections were down.
We still have a way to go to perfect this dual redundancy of data, but I think working out a solution will go a long way towards making it more comfortable for more business to jump onto Cloud Computing.
Footnote: We got all comms back online at about 11:30am that morning. But while waiting, we decided to celebrate this quiet spell with a ‘fry up’ breakfast with the family. When I went down to the corner store to pick up some eggs and bacon, I habitually whipped out my debit card to pay for the groceries, only to be told that they were operating on a ‘cash only’ basis, as all EFTPOS machines in the Northern Territory of Australia were down. This included all ATM’s as well, so no one was able to get any money out. Commerce had come to a standstill!
Footnote II: Ironically, this outage came on the very morning that the Australian Prime Minister announced the AUD$42billion investment in national broadband infrastructure!
Footnote III (editor’s note): Ironically (sadly) in the meantime Silicon Valley also experienced a major communications outage, this one due to sabotage.
(Guest post by Devan Sabaratnam)
[..] I have also had anotherguest postup at CloudAve, relating to the communications outage we had here in Northern Australia last week. In future, I will be linking more to my other guest postings etc. from here. [..]
I agree…very scary stuff. It becomes harder and harder to be a cloud advocate with these types of outages.
I guess these types of outages bring attention to this situation though.
We need more auditing of data center and fiber connectivity and build in redundancy through fiber rings and between data centers.
We need this within countries and between countries.
[..] Was just reading this post about amajor Australian Internet outagethat left the entire top end of the country without telephone, internet or mobile communications for about 10 hours. [..]