Wow… I’m back from the blogging black hole. Nothing posted since last October, but so much has happened. It’s probably fitting for me to start afresh. I recently relocated to Cary, NC from the DC metropolitan area and started working for consulting company in Raleigh, NC. I was hired as a Sr. VMware Engineer, giving me the opportunity to hone my skills in the exciting world of data center virtualization. For the last 15 years of my IT career, I’ve had the opportunity to work my way up through the ranks, spending the last six years managing a data center. I’ve designed and implemented physical and virtual infrastructures, storage, backup and DR solutions, Windows and Linux environments, Active Directory, Exchange, Citrix,… I’ve worked closely with the network and security teams to provide the right solutions for the increasing demands of our customers. I even had to learn the lingo and philosophy of ITIL in my attempt to better manage the day to day network operations. I’ve touched all parts of the infrastructure stack, but have had little time or resources to dive really deep. In this position I hope to change that. New home, new job, new opportunities for growth, all make for an exciting adventure. I’m ready to go!
Last week I had the great pleasure of visiting the NetApp campus in Research Triangle Park. I had previously missed out on the opportunity to take part in an EBC trip with our reseller when we became a NetApp customer. Things came up and the trip never happened. Since I was going to be in the Raleigh area, I decided to take advantage of the proximity and pop in to see the folks at RTP and request a tour. I also wanted to sit down with one of their desktop virtualization experts since my company is exploring the possibility of rolling out hosted desktops.
The data center tour was quite impressive. Building 4 holds NetApp’s highly acclaimed Energy Star Data Center where they proudly show off the innovations that allow such a high energy efficiency. Efficiency is measured in PUE, which is the ratio of total power used by the facility over the power used by the IT equipment. Most data centers register a PUE of 2.0 while NetApp’s facility boasts a PUE rating of 1.2. They cut the typical power usage almost in half by using some simple yet incredibly innovative features. As Dave Hitz wrote in his blog article, two of these notable features are big fans and hot air, which were the very things I noticed when I toured.
The first thing I observed when I walked into the data center was the heat. There was a heat wave last week and walking into the data center provided little relief! I was suprised that it was not cooler since data centers are by nature, chilly. It turns out the engineers discovered that outside air up to 74 degrees was sufficient enough to cool the equipment. They just had to get this moderate air to the servers without mixing with the hot air. Of course they have their hot and cold aisles, but the engineers went a step further and closed off the cold aisles with doors at each end to provide access. They call this approach cold aisle containment. Containing these cold aisles allows the moderate outside air to reach the IT equipment without being tainted by the hot air. Thus they do not need to constantly use the chillers to cool the air. The only times they have to chill the air is when the outside temperatures rise above 74 degrees. This saves power costs since the chillers are used a smaller percent of the year.
The other noticeable thing was the wind tunnel flow of air through these aisles. Inside the cold aisle, air was rushing through. It is this increased air flow that also helps the moderate air cool the equipment by drawing the heat off the systems faster. Faster air means faster cooling. When we were on the second floor and standing on the vents above the hot aisles, I was again surprised by the amount of air gushing through the vents.
There are many other “cool” features and NetApp happily welcomes visitors every day to witness the innovative design. They built an Executive Briefing Center (EBC) to welcome their customers and showcase both the data center and the storage efficiencies of their products.
This wasn’t the end of my trip. I also had the pleasure of meeting with Chris Gebhardt, NetApp’s desktop virtualization guru. It was awesome meeting him and especially meaningful as the work of my organization, the Cystic Fibrosis Foundation, has had a very personal impact for him. We are ramping up our efforts to pilot test desktop virtualization and I was eager to speak directly with Chris about properly assessing our environment and employing best practices in the initial stages. It was a very insightful discussion with him! He had great tips and tools to share. What an incredible contact to have!
Overall, it was a fabulous time spent at the NetApp campus. I was delighted to hear how NetApp is truly trying to make a difference in the community. They have been running their “Technology Saves Lives” campaign and I think it was great for them to see that our organization is in the pursuit of saving lives by curing a disease (which is a whole other story unto itself!) NetApp also encourages its own employees to get involved and volunteer their time and energy in their communities. And of course, their energy efficiency is yet one more example of a company trying to make a difference. Bravo guys!
Special thanks to Will Graham for facilitating the tour and meeting.
Gartner is holding their Infrastructure and Operations Management Conference this week. As valuable as this conference would be to my career, I did not choose to go this year, but I did attend Gartner’s Data Center Conference last year in Vegas. This was my consolation for not going to VMworld. And it turned out being a great show. Very eye-opening as all conferences are, but this one took me into more of the IT manager’s mind set. Attendees were by and large managers and directors or above at their companies, most of which were large companies. My eyes were opened to many technologies and processes that were rather foreign to our rather small shop at CFF. There were not the usual awesome technical demo’s put together by the senior engineers and product specialists – this was more theory of how to run the IT processes and technology and even people.
Why am I writing about this now? Since Gartner IOM is going on as I write, I happened to see some tweets from the show from an ITIL guru whom I admire, George Spafford. George is one of the Gartner analysts whom I had the great pleasure of sitting with for a one-on-one session. At the time, I had just read the book he co-authored with Gene Kim and Kevin Behr, Visible Ops. Great short read and great practicable advice as the subtitle suggests. George was very personable and encouraging of his views on how to ramp up ITIL efforts in an organization. I had also just earned my ITIL V3 certification so I was especially excited about the topic. We discussed the importance of finding and stopping processes that stand in the way of efficient IT, baby steps needed to start ITIL, the importance of getting management onboard and the critical nature of performing a post-mortem after each major incident to continually improve. I also reflected on the major points of his book, the first of which cut to the heart: “Stabilize the Patient”. In a nutshell, the authors explained the need to get absolute control over all changes. That means all changes must stop until a proper change management process can be implemented. The reason was based on their statistics that 80% of systems downtime in an organization result from a change that was made.
This statistic spoke loud and clear to me as we had seen this time and again when scheduled changes created downtime extending outside of maintenance hours, unathorized changes being made with lack of coordination with the impacted department or even within IT, or lack of fully testing when changes did occur. Last year, this problem was magnified as we moved all of our production systems to a new data center. We endured so much business disruption in the process, certainly the majority of which was expected due to the massive amount of change required in an effort like this. But we realized that a sometimes ineffective change management process led to some unexpected downtime – lack of fully mapping out all configurations to be changed and thus lack of testing of these components.
Obviously we had some work to do within configuration management as well. This speaks to the second tenet of Visible Ops, “Catch and Release”: Learn what you have and document it. Unfortunately, our environment had been built in various aspects through various stages by various individuals who have gone various ways – with limited documentation. One of my goals within my organization has been documenting our infrastructure, from the bottom up. How does a world-class organization get by without knowing all the nuances of the environement? It doesn’t. Our environment has become increasingly complex over the recent years, as most organizations have. And the only way we can become world-class is getting a handle on all components.
Thankfully, the last year was a huge learning opportunity, both from a process standpoint and a technological one. Today, we’re taking great strides toward solid change management and configuration management. We’ve also welcomed a new senior-level member of our staff who among other things, has brought in real-world experience in these areas and who has guided our efforts. We are continually tweaking the processes to make them more efficient, steadily moving forward with eyes on becoming a world-class organization.
Or shall I say, “Welcome, Brian” to the world of blogging. This is the beginning of my journey that so many others have taken before me. I am starting a blog. I am an IT professional in the DC area, specializing in data center operations and virtual infrastructure (and just about everything else that sys admins do!). Many times I have thought about starting a blog focusing on the challenges and observations I come across during my days as an IT admin, especially those ideas having to do with virtual infrastructure. I am a huge VMware fan, have used their enterprise products since Dec 2007, architected and built the current virtual infrastructure for my employer, helped them virtualize close to 70% of our environment within 3 years, achieved both VCP 3 and 4 certifications, and am now missing out on the biggest virtual event of the year and of the universe right now – VMworld 2011! I will confess that my missing this event is acting as the catalyst that finally motivated me to start this blog. I needed to channel my “missing VMworld blues” into something productive for myself and that maybe one day can be productive for the greater virtual and cloud community. As for now, this will be my own little repository to let my own thoughts flow, to use as a reference and to help me articulate the concepts that I come across on a daily basis.
On a very personal and almost completely separate note, the reason I am missing VMworld this year is that my brother-in-law, my wife’s brother, died unexpectedly and very tragically one week before the conference. This has been a terribly rough time for my family, a time for grieving and seeking comfort in each other’s presence. My brother-in-law, Matt Griffith, was an amazing husband and father. He left behind a strong wife, two wonderful boys, 10 and 15 years, and was so devoted to his family and raising his boys to be responsible young men. I was so touched by all the photos that we sifted through preparing for his memorial service. The togetherness he was so committed to in his family was very obvious. Between family trips, birthdays, sports leagues, and just hanging around the house, he had no greater love than time spent with each of his boys and his wife. What a tremendous loss for them! My heart weeps for each of his boys, for his wife, for his parents who have lost their oldest son, and for my wife who lost her big brother. Everything hurts right now and will for a while. I hope to be able to honor my brother-in-law in some special way one day.