Gartner is holding their Infrastructure and Operations Management Conference this week.  As valuable as this conference would be to my career, I did not choose to go this year, but I did attend Gartner’s Data Center Conference last year in Vegas.  This was my consolation for not going to VMworld.  And it turned out being a great show.  Very eye-opening as all conferences are, but this one took me into more of the IT manager’s mind set.  Attendees were by and large managers and directors or above at their companies, most of which were large companies.  My eyes were opened to many technologies and processes that were rather foreign to our rather small shop at CFF.  There were not the usual awesome technical demo’s put together by the senior engineers and product specialists – this was more theory of how to run the IT processes and technology and even people.

Why am I writing about this now?  Since Gartner IOM is going on as I write, I happened to see some tweets from the show from an ITIL guru whom I admire, George Spafford.  George is one of the Gartner analysts whom I had the great pleasure of sitting with for a one-on-one session.  At the time, I had just read the book he co-authored with Gene Kim and Kevin Behr, Visible Ops.  Great short read and great practicable advice as the subtitle suggests.  George was very personable and encouraging of his views on how to ramp up ITIL efforts in an organization.  I had also just earned my ITIL V3 certification so I was especially excited about the topic.  We discussed the importance of finding and stopping processes that stand in the way of efficient IT, baby steps needed to start ITIL, the importance of getting management onboard and the critical nature of performing a post-mortem after each major incident to continually improve.  I also reflected on the major points of his book, the first of which cut to the heart:  “Stabilize the Patient”.  In a nutshell, the authors explained the need to get absolute control over all changes.  That means all changes must stop until a proper change management process can be implemented.  The reason was based on their statistics that 80% of systems downtime in an organization result from a change that was made.

This statistic spoke loud and clear to me as we had seen this time and again when scheduled changes created downtime extending outside of maintenance hours, unathorized changes being made with lack of coordination with the impacted department or even within IT, or lack of fully testing when changes did occur.  Last year, this problem was magnified as we moved all of our production systems to a new data center.  We endured so much business disruption in the process, certainly the majority of which was expected due to the massive amount of change required in an effort like this.  But we realized that a sometimes ineffective change management process led to some unexpected downtime – lack of fully mapping out all configurations to be changed and thus lack of testing of these components.

Obviously we had some work to do within configuration management as well.  This speaks to the second tenet of Visible Ops, “Catch and Release”: Learn what you have and document it.  Unfortunately, our environment had been built in various aspects through various stages by various individuals who have gone various ways – with limited documentation.  One of my goals within my organization has been documenting our infrastructure, from the bottom up.  How does a world-class organization get by without knowing all the nuances of the environement?  It doesn’t.  Our environment has become increasingly complex over the recent years, as most organizations have.  And the only way we can become world-class is getting a handle on all components.

Thankfully, the last year was a huge learning opportunity, both from a process standpoint and a technological one.  Today, we’re taking great strides toward solid change management and configuration management.  We’ve also welcomed a new senior-level member of our staff who among other things, has brought in real-world experience in these areas and who has guided our efforts.  We are continually tweaking the processes to make them more efficient, steadily moving forward with eyes on becoming a world-class organization.