Ok, I know it's been some time since I posted...at least 5 months!! I have made a commitment that I will begin posting at least once, but hopefully twice a week. As a small consultancy firm with myself heading up both sales and commercial project management, not enough of my time is spend having "conversations" with customers like I used to as a sales guy working for other people's companies. So, I have decided that this blog is a good way for me to tell some stories and give my opinions on topics relevant to what our customers do day-in-day-out...and probably some irrelevant crap too! Although I do not have the 'blogosphere' prestige of some tech bloggers out there (many of whom I read daily and will share with others), I hope to generate a multi-way conversation from topics in this blog...so please sign in and comment/question or berate and insult me anonymously..at least I will know someone is reading!!

ANYWAY...on with what I was actually going to talk about.

Recently, and on two occasions, one or more of my online social media service accounts has been compromised. I am not sure of the method of attack used however I am sure of the intrusion because on both occasions postings and/or comments were published via those social media web services (and syndicated to other web services in one case) that were certainly not penned by yours truly.

The comments were the 'get-rich-quick', 'own your own successful small business for zero effort' type of thing I'm sure you have all seen or heard before. As I indicated in a genuine posting to all contacts via the compromised (and now password changed) accounts, owning a small business is definitely a bag of mixed blessings...and there is nothing "rich" or "quick" about anything!! :)

Although I had thought about the precarious nature of web services password many times in the past, these recent incidents brought the topic into stark contrast for me and I thought I would comment on the topic. 

I have noticed over the years that I re-use the same password (or set of about 3 passwords) for varying web services and applications from things like Gmail through to Amazon accounts and Twitter/LinkedIn. I was aware that this presented a risk to security of my access to these services, particularly when most of them allow email address based login names/usernames or I have chosen fairly standard guessable usernames (e.g tdavoren, timdavoren, tdav, etc). The convenience of not having to remember multiple passwords therefore means that one breach with one particular web application opens me up to multiple breaches across different applications.

Some rogue tweets or postings via Social media are unlikely to be too damaging (for me that is...for larger PR reliant firms they may be), but the possible information leakage from other 'non-social' (i.e personal/private/secure) web applications is worth pondering...banking details, client information, sales prospect information, any kind of intellectual property.

This situation obviously calls for either stronger passwords (and a way to manage/track them...I swap browsers/computers all the time so browser solutions are no good) or for a two-factor authentication access method. I will be opting for the second...and I reckon most businesses considering any 'cloud' or web based platform or application will be doing like wise. The idea of 'something you know and something you have', whilst not entirely unhackable, is a significant improvement on just strong passwords.

Of course, you might think why haven't I done this already...well of course, the cobbler's son has no shoes right?!? We will be looking to use technology that we sell/integrate for a living to solve this issue....EMC RSA SecureID, Symantec Verisign, Microsoft Access Gateway or SafeNet are amongst the primary candidates.

Stay tuned.

Just came across this quote from EMC Asia Pacific President, Steve Leonard;

Leonard said the use of one company to approach market would help to win business over competition, and provide better servicing.

“What we’re trying to do is position Vblock as an architecture, and VCE as a company that can bring that with one support number, one architectural campaign, and we think that if we can do that faster, we think we can win,” he said.

“We want to be the guys who help put the engines in the data centres." (my emphasis)

So, do we Steve, so do we...oh, its in our company name!

With thanks to my client and friend Mr Justin Kurosawa over at Security Circus.

I refer here to a recent article penned by Tony Pearson of IBM discussing a recent catastrophic failure of an EMC Symmetrix within the State of Virginia's IT environment. Aside from some cheap inter-vendor point scoring, Tony mentions as one of the 'lessons' from this event; "Lesson 4: This can serve as a wake-up call to consider Cloud Computing as an alternative option". There is a faint tinge of irony here in that this post of Tony's was written in Australia in early September. At the end of that month the IT community (and the broader travelling public) witnessed how a 'cloud' provider can be just as exposed to downtime as your now unfashionable internal IT team. Virgin Blue's ticketing and reservation systems were brought offline due to an as yet unidentified systems failure within the storage data path. Virgin Blue do not own/operate their own ticketing and reservation system, but source such a service from Navitaire (disclosure: Navitaire are an old client of my firm). It took Virgin Blue (and I assume Navitaire) almost 7 days to return this service to normal operation. I wont call these observations 'lessons', rather just 'comments;

1. I agree with Tony (and probably every other seasoned data storage professional)...storage systems fail, that's why we have backup systems. These systems in turn are definitely only as good as their last successful test...if regularly testing is too much of a burden then you ought to at the very least audit the environment according to some baseline.
2. Whilst the person in question at the State of Virginia may be a little ashen faced currently, I can assure you that the "service delivery manager" (as they were called in the hey day of outsourcing), at Virgin Blue for the reservation and ticketing system will be feeling the same churning in the lower part of the stomach...his contact at Navitaire probably likewise. Just because a cloud provider is 'big' or ' branded' or, (inserted alarm bells), 'multi-tenanted', does not mean for one second that they can do better/cheaper job of helping you meet your SLAs for service uptime and/or data recovery.

I advise strong governance of how storage systems are used in medium - large organisations, as well as a strict focus on 'recoverability' in governance of backup systems.

In the former instance remember that 'speeds and feeds' as Tony puts it indicate in my experience the 'bleeding edge' of what a product can reliably do...divide by 2 and set that as your peak load. The more complex the data storage layout on a disk array (fragmented RAID  groups, meta-LUNs, concatenated LUNs, etc, etc), the longer your restore/rebuild will be. Remember that in the never ending race toward better storage performance, there is a necessary compromise around recoverability.

In the latter instance, just 'think' in terms of recovery, not 'did it get backed up'...build backup systems that focus on the process of data restoration (we talk only of data availability here, compute availability is a whole different story). It is far better to have a backup run for 8-10 hours, complete, validate and be easily restorable than a backup that runs in half that time but require multi-step, error prone recovery procedures.

Similar to a post from a few weeks back, I found this little summary amongst some old client correspondence which you may find useful.

8 Steps to Effective DR Planning and Budget Requisition

1. Use the term Disruption Recovery rather than Disaster Recovery.
2. Ensure you currently have some kind of DR/BCP management framework no matter how rudimentary. Show business management that there are current documented processes, metrics and testing that can be optimised. Technology supports the business insurance requirements, it doesn’t necessitate it. You need to show that DR planning is an ongoing process not a point in time flight of fancy.
3. Engage the right people in your organisation. Applications support and owners, facilities management, and of course, when ready, financial and executive management.
4. Conduct a joint Business Impact Analysis or Risk Assessment. What are we insuring/protecting against...delineate the actual dangers. Threats = Impacts = $$.
5. Then proceed to a ‘costs of downtime’ calculation. Dependency mapping of all business applications is the starting point for this. This is never a clean cut equation but, the revenue that a particular application or ecosystem of applications support is the dividend, downtime impact is the divisor and roughly speaking then the cost of downtime is the quotient (which should align roughly with a budget!)
6. Position a DR investment as having some competitive market value – ROI. Present ‘best-in-class’, industry peer data. – Reputational loss, client retention may be affected.
7. Develop a DR services catalogue...align costs to system criticality (RPO/RTO). Relative costs vs. Absolute costs wherever possible. Suggest chargeback to Bus etc.
8. Align DR investment with other IT budgets. Try and link the technologies used in providing DR services to other areas of IT ‘spend’. EG Data centre consolidation, server consolidation (virtualisation), and utilising DR infrastructure for development or test purposes.

I recently read a news article about Intermedia's service level agreement 'miss' that was linked to a performance issue on an EMC CLARiiON array.

http://searchstorage.techtarget.com/news/article/0,289142,sid5_gci1510721,00.html

There have also been a couple of subsequent posts and email responses linked to this story;

http://itknowledgeexchange.techtarget.com/storage-soup/one-storage-pros-response-to-intermedias-hosted-email-outage/

http://chucksblog.emc.com/chucks_blog/2010/04/helping-to-avoid-a-really-bad-day.html

I wanted to make a few commetns myself in regards to the story and the responses shown above.

Firstly, I agree with everything Chuck Hollis at EMC says in his post, and I wanted to emphasis and elaborate on his points.

Products Fail?
Damn right they do, all the time...sometimes without causing much of a fuss, but trust me failures don't seem that common because you only hear about the big ones (like Intermedia's). It is a testament to IT hardware vendor's engineering that alot of these "failures" go unnoticed because fo the rigorous redundancy build into their systems...not to mention field support services which, in the case of EMC, are some of the best around.

A short anecdote that relates to this story; an insurance client of ours suffered a similar failure on their IBM N-Series (NetApp) devices a few years back. A controller panicked due to a power supply issue and tried to hand over its load to the other controller but due to incorrect configuration of multi-pathing, dropped all the workloads that it was serving. Result; reboots, reboots, reboots. Missed SLA.

Design for Failure
It will happen...not if, but when. You will have a component failure somewhere in your data path at some point in the future. Design for it (or insure for it!).

CLARiiON arrays (like N-Series, HDS and many other array vendors) have controllers that operate in active/active configuration, which is great when both controllers are working, and 99.99% of the time it works fine when one fails (the beauty of PowerPath). But the disadvantage of running and active/active architecture in a disk array is that, unless you religiously monitor your workloads, you can never be sure if you can meet performance demands in a degredated state (this principle applies all down the data path, even to RAID Group design and LUN layout). My favourite disk array of the last 10 years is EqualLogic's PS Series, now owned by Dell. These fellas only operate in active/passive mode to ensure customers don't accidentally find themselves in Intermedia's situation where peak load cannot be accommodated in degradated mode.

The Alarm is Ringing but Everyone's Asleep
This is an interesting point...vendors and integrators like ourselves put effort into engineering and deploying monitoring and alerting for systems in client sites. That's great but if the client doesn't put in place procedural steps that are triggered into action by these tools, all is for nought. There is no point in having a tight RPO and the ability to deliver a quick RTO unless you have the procedural surety  to act when issues are identified. EMC's DialHome feature is a good example of removing this dependency but its simply not possible (nor do you want it to be possible) for all system or component failures. In short your recovery time is only as good the weakest trigger point and usually that trigger point is simply deciding to act on a error/mis-configuration event.

Practice Failure
Great tip here...I hear clients and prospects talk about their highly redundant environments and their sub-minute failover setups and ask have they tested it...usually the answer is no. Reminds me of people who love to talk about how much their house has gone up in value...inevitably when they actually want to sell they are a little disappointed. Proof is in the pudding. You must test your failure recovery procedures. VMware's SRM product is an excellent tool for doing this non-disruptively. Clients should regularly test failover of their Tier 1/2 applications to ensure that the 'best laid plans' are also the 'tried and true' method.

Advice, especially field tested advice is not risky Happy! But I guess there's sometimes no option but to adopt a bureaucratic approach when positioned as such.

http://www.arnnet.com.au/article/344444/smb_it_supplier_advocate_named/?eid=-4152

"Easter said his first job would be to consult industry bodies and large tech companies to get a sense of the industry. SMB IT companies would need to understand how risky they appeared to Government CIOs"

I would love to get Data Engines to do more government business; it is often challenging work and it feels a little bit like we're using IT as our way to 'give back' in public service!. But...but, but but... $40mil coverage for professional services indemnity is also a regular requirement for tender respondees and that cost is a little too much alongside side a 50 page tender response document for a specification that is often written with a particular solution and/or vendor already in mind.

//TD.

This is a little reflection I wrote some years back that I just uncovered. I think most of the key points are still relevant as they ought to be. Regardless of what technology is used archiving is a generic information managment technique (like backups) that should be approached from the high level, first principles before assessing what tools an organisation can leverage.

 

Why Archive Anyway? To Manage RISK & Manage RETURN

• Reduce storage costs (acquisition and management).
• Improve email/file server performance.
• Manage unstructured information – security, business value, risk...
• User accessibility and user expiration/termination
• Simplify Discovery
• Simplify Protection (backup)

What to Think About?

 

• A long term perspective must be taken in considering an archive product. This product/mechanism/solution will be a strategic platform and is going to be there for a long time!
  • Growth – staff/data?
  • Mergers, multiple business unit integration?
  • Retention policies should match the vendor’s longevity!!
  • Data portability - can I move my archives around?
• Manageability and Security and Reporting
  • Other applications that the archive may touch/be touched by
  • Role based administration of the archive application
  • Trending feedback – scale up before you hit the wall
• Focus on Content Intelligence
  • Key organisational data types and storage locations
  • Where and how will data be created
• Protect your archive application like a Tier 1 application
  • The archive application must be deployed as if it were a Tier 1 system...it now stands as a critical link in business data access.
• Relate policy to Directory Services.
  • Don’t re-invent the wheel...you will already have IT governance policy in some basic form in Directory Services...ensure your solution can utilise and hinge from this.
• IT Budget generally grows at about 2%, but Storage grows at about 7%...Stress that archiving is a ‘storage’ application not an email application.

Its been quite some time since I posted, but am just catching up on some IT industry news so thought I would pass on second the news of VMware purchasing a bunch of software assets from its 80% + owner EMC Corp. The products seem to be the core network management part of Ionix ans may indicate either a) EMC don't really see a future in fighting the DC managment fight or b) VMware may be a better vehicle to take part in such a fight. Interestingly though I also read subsequent to that story details about the revenue growth attributed to Virtualisation by iTX Group...not just VMware though...they also sell Vizioncore, IBM Tivoli and Computer Associates...maybe b) is the right answer...VMware may end up with a very rich portfolio of systems management and reporting tools in the medium term.