AgileRisk Management

Blackout: Don’t drop the ball when the lights go out

When the most watched sports event in the country goes dark, it’s time for smart leaders everywhere to pay attention to what happens next. Super Bowl XLVII showed us how technology can fail in a spectacular way – despite years of planning by the most seasoned professionals. First, the background:  90 seconds into the second half of the game, power was lost within the Mercedes-Benz Superdome. According to the Washington Post, a piece of equipment monitoring the electrical load sensed an abnormality – and threw a breaker to isolate the problem.

Lesson#1: Even though the equipment responded as designed, people were still inconvenienced.

As a leader, you have to recognize the end result of any risk management or issue system: how are your customers affected? The people watching the Super Bowl (and the teams playing it) didn’t necessarily care that a piece of equipment shut down the lights to isolate an electrical problem – they cared that the game was delayed 34 minutes! The Super Bowl is planned for years in advance, and no doubt organizers were aware of potential energy concerns. However, that concern didn’t translate into the real world. As with any project, there are elements that are simply outside of an organization’s control. It’s how the organization responds to those challenges that determine whether an incident simply slows or sinks an effort. We asked our resident agile expert Brian Lucas what he thought about the planning process that occurred for the Super Bowl specifically and what IT Leaders could learn from it: Don’t assume something you planned for years won’t go wrong at the last minute.  Almost lesson one in agile, or anything else today for that matter, is that ANY plan is subject to change. Today’s environment is so complex that it is impossible to anticipate all the possible permutations.  Lucas continues by noting that Alvin Toffler writes of this in his book Revolutionary Wealth and Jurgen Appelo speaks of complexity in his book Management 3.0. The question is can we successfully plan?  The answer is yes!  You need to use strategic planning techniques as a starting point.  But only a starting point!  These plans must be ubiquitous and dynamically fed and matured from ALL elements in your organization.  They are not all inclusive edicts from on high.  If anything these plans cover the vision and the goals stating what it to be accomplished.  The details are left to be continually developed, redeveloped and added to.  So in the Super Bowl plan there needed to be a goal of uninterrupted power of contingency activities to take place in case of a power interruption.  In other words even though the Super Bowl is planned for years in advance, planners should have considered the need for the show to go on despite outside environmental changes.

Lesson #2: Everyone should know the response plan.

Fortunately for Super Bowl viewers and players, the team at the Mercedez-Benz Superdome knew what needed to happen—and knew how to get that plan into motion. As discussed in an article on the Los Angeles Times, the control room personnel were calm and deliberate in determining what steps to take next. They acted professionally despite, as article author Patrick Kevin Day writes, not knowing exactly what was happening: On Monday, CBS Sports' Sean McManus told The Times that the loss of power was “a surreal situation.” He went on to say that “”We were asking everybody at every position what was happening, and the fact of the matter is we just didn't know,” he said.

Lesson #3: Fix the problem NOW.

So you’ve got a well-grounded, responsive team trying to fix the problem – and that’s exactly how it should be handled: your team should fix the problem before trying to figure out why it happened. Root cause analysis is very important, but it’s not important immediately. Get your operations up and running first, and then determine what when wrong, and why. Super Bowl goers didn’t care why the power went out—and your customers won’t care why they can’t log in or use an application—at least not at first.

Lesson #4: Mitigate the effects on clients.

Leaders should also be looking at what they can do in the meantime to make sure other systems and processes can continue – or how they can mitigate some of the effects of the error. Just because the incident itself was not expected doesn’t mean you can’t respond holistically to the possible impacts. Once again, agile expert Brian Lucas provides some real world examples of leaders thinking on their feet to minimize damaging effects: The blackout might have been unexpected, but should not have been a surprise.  Small businesses must live with surprises all the time and survive.  For example if a mom and pop neighborhood store has a power failure, from say a car hitting a pole, they’ll take some or all of the following steps: 1)   Take their frozen goods to a backup location 2)   Bring in a generator from home to provide some power 3)   Offer special sales on perishable items 4)   Give food away free to get good will rather than just have it spoil I know I have seen small stores in my neighborhood do this during Hurricane Sandy and at other times.

Lesson #5: Get the game back into full swing ASAP.

There is always a point where it seems logical to stand back and look at the whole picture of a problem – and in many cases this is a smart move. However, it’s not a smart move when it’s a business critical situation. As soon as all systems were up and running again, the players lined up and started playing. There were no announcements, no apologizing. The organizers got the show back on the road as soon as possible, and that was the best way to satisfy the fans. In much the same way, your IT organization should feel as though it needs to have a structured “we fixed it” announcement for mission critical applications. Naturally some level of communication has to occur – but it shouldn’t take up too much time from customers who are already inconvenienced. Get the application working and up first – then get on the phone to explain the problem was resolved. The final lesson here comes from an…interesting…exchange between Coach John Harbaugh and NFL vice president of operations Mike Kensil. The exchange itself was rather heated on the coach’s side, but that isn’t the point.

Lesson #6: Upper management must be involved.

The important lesson is this: know who is the face of your work. What I mean is this: your employees are meant to make your organization function – you as the CIO are meant to lead, yes, but also to be the voice and contact for your customers in time of hardships. The NFL vice president was talking to the coaches – not one of the front line technical staff trying to explain what was happening and what would happen afterward. It’s a small thing that makes a big difference: when something goes wrong, make sure you get the person your customers will know to talk to them. Oftentimes, this is one of your highest level people. The benefit is two-fold: your staff is able to work uninterrupted on the problem and your customers recognize that it’s being taken seriously (after all, the CIO is talking to them). Overall, it comes down to response, involvement, and (appropriately timed) communication. The Super Bowl blackout might have been an inconvenience to some, but to IT leaders, it should be lights-on moment.

Show More

Leave a Reply


We use cookies on our website

We use cookies to give you the best user experience. Please confirm, if you accept our tracking cookies. You can also decline the tracking, so you can continue to visit our website without any data sent to third party services.