DavsDisorder

I refer here to a recent article penned by Tony Pearson of IBM discussing a recent catastrophic failure of an EMC Symmetrix within the State of Virginia's IT environment. Aside from some cheap inter-vendor point scoring, Tony mentions as one of the 'lessons' from this event; "Lesson 4: This can serve as a wake-up call to consider Cloud Computing as an alternative option". There is a faint tinge of irony here in that this post of Tony's was written in Australia in early September. At the end of that month the IT community (and the broader travelling public) witnessed how a 'cloud' provider can be just as exposed to downtime as your now unfashionable internal IT team. Virgin Blue's ticketing and reservation systems were brought offline due to an as yet unidentified systems failure within the storage data path. Virgin Blue do not own/operate their own ticketing and reservation system, but source such a service from Navitaire (disclosure: Navitaire are an old client of my firm). It took Virgin Blue (and I assume Navitaire) almost 7 days to return this service to normal operation. I wont call these observations 'lessons', rather just 'comments;

1. I agree with Tony (and probably every other seasoned data storage professional)...storage systems fail, that's why we have backup systems. These systems in turn are definitely only as good as their last successful test...if regularly testing is too much of a burden then you ought to at the very least audit the environment according to some baseline.
2. Whilst the person in question at the State of Virginia may be a little ashen faced currently, I can assure you that the "service delivery manager" (as they were called in the hey day of outsourcing), at Virgin Blue for the reservation and ticketing system will be feeling the same churning in the lower part of the stomach...his contact at Navitaire probably likewise. Just because a cloud provider is 'big' or ' branded' or, (inserted alarm bells), 'multi-tenanted', does not mean for one second that they can do better/cheaper job of helping you meet your SLAs for service uptime and/or data recovery.

I advise strong governance of how storage systems are used in medium - large organisations, as well as a strict focus on 'recoverability' in governance of backup systems.

In the former instance remember that 'speeds and feeds' as Tony puts it indicate in my experience the 'bleeding edge' of what a product can reliably do...divide by 2 and set that as your peak load. The more complex the data storage layout on a disk array (fragmented RAID  groups, meta-LUNs, concatenated LUNs, etc, etc), the longer your restore/rebuild will be. Remember that in the never ending race toward better storage performance, there is a necessary compromise around recoverability.

In the latter instance, just 'think' in terms of recovery, not 'did it get backed up'...build backup systems that focus on the process of data restoration (we talk only of data availability here, compute availability is a whole different story). It is far better to have a backup run for 8-10 hours, complete, validate and be easily restorable than a backup that runs in half that time but require multi-step, error prone recovery procedures.