The recent flooding and storm damage brought on by Hurricane Harvey in Texas, is a timely reminder of the need to have a very sound disaster recovery plan for your data centre that has been tested as much as possible, and which anticipates as much as possible what could go wrong in the event of a catastrophe.
Each natural disaster that takes place is a wakeup call for data centre operators around the globe. It is also a timely reminder to the customers of those operators, that even if they have outsourced some of their risk by contracting with a cloud or data centre provider to manage their infrastructure, assuring their data is protected and ICT services are available in the event of a disaster is ultimately their responsibility.
That said – given the magnitude of Hurricane Harvey, the first major hurricane to hit the USA in over 10 years, there were several cases of data centres successfully weathering the storm, despite absolutely devastating conditions.
- Some of the major challenges facing the Texas Data Centre Operators included:-
a. Flooding. Not just the risk to the facility, but the fact that roads were impassable. This raised issues such as access by staff (many of whom either volunteered to stay, or could not get to work or were trapped at work), customers and ultimately fuel deliveries for generators.
- b. Power outages. The biggest challenge is continuous power to the data centre, and the need for backup generators in the event of major power loss. Fuel deliveries could be delayed because of the flooded roads around the facility, despite the fact that many data centre operators have national fuel delivery contracts that put them immediately behind hospitals, first responders and other public safety organisations. Some companies arranged for staff to work from home but residential areas are normally the first to lose power and communications, and few staff have a backup generator at home!
- c. Manpower and Client assistance. One data centre in Texas moved cots, fresh water and food supplies into its facility in anticipation of critical employees being stranded at the facility due to flooded roads. Another facility had showers and washing machines installed for staff and customers being on-site for extended periods. These initiatives proved to be smart moves because of the extended period of the disaster.
- d. Customer liaison and support. The smarter operators continued to monitor and assess the situation and to keep their customers updated on the status of things. Given the unpredictable nature of events, it is important to reassure customers (or stakeholders in the case of in-house facilities) and to ensure they themselves have disaster recovery plans in place for their own local systems.
It is worth remembering that Australia’s ICT Infrastructure has also been hit by significant storm and flood events – in 2010 and 2011 in Melbourne – then in 2012 in Brisbane – each time causing extensive damage to major datacentres and clients.
There is evidence to suggest that such occurrences are likely to multiply. A recent UK climate change report highlighted the potential risks to British Infrastructure – noting that flooding as a result of changes in weather patterns and increased rainfall poses the greatest risk of all, despite redundancy measures taken by the ICT industry.
The bottom line for Australian data centres, whether in-house, colocation, or cloud based – is the need for robust, redundant, regularly tested systems, and a well- structured and prepared disaster recovery plan, able to be implemented in the event of flooding or (in Australia particularly) bushfire.
The lessons from Texas are that, even though the facility itself may not be in direct danger from a catastrophic event, the reliance on support structures including staff, roads, supply chain challenges, are critical. Almost all data centre operators assume short term chaos, but if roads and other infrastructure remain affected for an extended period, as has happened in Texas, everyone needs to anticipate and have a “PLAN B”.