IT Can Be Amazing!

Yesterday our team had a scheduled maintenance day in our datacenter. Our goals were rather large:

• Remove two retiring Dell PowerEdge Servers (R710s, 96gb RAM, 4-Core Processors)
• Add three new Dell PowerEdge Servers (R620s, 128gb RAM, 8-Core Processors)
• Increase RAM in two 1-year-old R620s to bring them up to 128gb
• Expand RAM in our VCenter Server to 96gb
• Apply firmware updates to all hardware
• Shuffle everything in the rack
• Move the entire Hypervisor system (the basis for our private cloud computing infrastructure) from VMware 5.1 to 5.5

Our intention was to do this live, with approximately 100 client servers and services running. We were in different locations around the country as we worked on this project:

• I was the only one at the datacenter, about 12 miles south of Las Vegas, NV
• Gary, our lead engineer, was in his office in Nashville, TN
• Rick, one of our engineers, was in his office in Milan, IL
• Ed, one of our engineers, was in his office in Columbia, SC

We had a plan, we had each prayed for God's grace and wisdom, and we began at about 7:15am Pacific time. We had a GoToMeeting conference call session running the entire time, so we were in constant communication via our telephone headsets and could share screens when helpful. It was as though we were all in the same room together!

I had emailed each of our clients earlier in the week, so I knew we were being prayed for. And God was merciful; he delivered! Here are our company tweets throughout the day (so clients would know how it was progressing):

1. I am in the datacenter, staging the upgrade!  (8:26am) [This was a little more than an hour after getting to the datacenter, unboxing the new servers, and getting them to our rack.]
2. One old host (#1) is out of the rack! Its space has been filled with two new hosts, currently in place and being configured.  (9:15am) [A physical server, when used the way we use them to serve clients, is referred to as a host.]
3. 1st new host is live! Servers have been moved to it and it's/ they're happy!  (11:18am)
4. 2nd old host is out of the rack! Configuration of 2nd new host in process.We're making good progress! 2 old hosts retired, 2 new hosts configured and running. 1 add'l new host in and ready to configure!  (12:14pm)
5. So far, zero downtime for our clients. w00t!!  (2:06pm)
6. Well, always plan for the unexpected in IT! One of the hosts that is surviving this update just had a redundant power supply issue! We're on it!  (4:24pm)
7. Ok... the three new hosts are running perfectly! All servers relocated to them and running happily as we resolve issues on the two 1-year-olds.  (5:36pm) [This is a great testimony to the maturity and power of VMware. There's a reason it runs more than 70% of cloud servers!]
8. I work with a GREAT team! All issues have been defeated on the two 1-year-old hosts, and we are upgrading their VMware!  (7:14pm)
9. Whew! We're almost done! Hosts 4 & 5 are nearly done! It's been a great day, and our clients were up and running the entire time! #blessed  (7:45pm)
10. What a great day! We are done! Just a little load balancing and that's it! Thanks to everyone who was praying for us!  (8:13pm)

And then I packed up my gear and began the nearly 4-hour drive home.

As I reflect, the things that stand out to me are:

• God's grace and mercy as he worked through our team. Our clients are all churches and ministries, whose mission is the most important work on Earth. He made it possible for our clients to keep about their tasks without interruption while we did all that!
• The Lord has given us a terrific team! I am truly blessed to be working with these guys.
• Technology, when it works, is terrific! The fact that we were in different parts of the country and yet were working as though were all in the same room is amazing!

New Article on Which is Better: iPhone or Android Smartphones

Yup! I'm entering emotional smartphone NeverNeverLand! If you'd like to read about my evaluation, click here to access a free advance copy of this article I just released to the publisher!