Technical Debt

Large business-critical systems can be brought down by power failure. Who knew?

In July 2016, Southwest Airlines suffered a major disruption to service, which lasted several days. It blamed the failure on “lingering disruptions following performance issues across multiple technology systems”, apparently triggered by a power outage.

Click below for the latest update on our system and operation: https://t.co/bqV1qwahmz

— Southwest Airlines (@SouthwestAir) July 21, 2016

In August 2016 it was Delta’s turn.

New statement from Delta – power outage caused IT failure pic.twitter.com/trkQbpym05

— Rory Cellan-Jones (@ruskin147) August 8, 2016

@ruskin147 A power outage *triggered* this issue, but poor planning and no HA *caused* it. Why can Netflix get this right but airlines cant?

— Richard Price (@RichardPrice) August 8, 2016

I am no computer expert but it seems like a whole system crashing (3 separate airlines) points to bad design (single point of failure)? 3/

— Dan DePodwin (@WxDepo) August 8, 2016

Then there were major problems at British Airways (Sept 2016) and United (Oct 2016).

@razankhabour We apologize to our customers for the delay and we appreciate their patience as our IT teams work to resolve this issue.

— British Airways (@British_Airways) September 6, 2016

We’re aware of an issue with our system and are working to resolve it. We’ll update as we learn more. We apologize for the inconvenience.

— United (@united) October 14, 2016

So every @united flight is grounded because they can’t run a decent IT shop. What year is this??

— Randy Bias (@randybias) October 14, 2016

The concept of “single point of failure” is widely known and understood. And the airline industry is rightly obsessed by safety. They wouldn’t fly a plane without backup power for all systems. So what idiot runs a whole company without backup power?

We might speculate what degree of complacency or technical debt can account for this pattern of adverse incidents. I haven’t worked with any of these organizations myself. However, my guess is that some people within the organization were aware of the vulnerability, but this awareness didn’t somehow didn’t penetrate the management hierarchy. (In terms of orgintelligence, a short-sighted board of directors becomes the single point of failure!) I’m also guessing it’s not quite as simple and straightforward as the press reports and public statements imply, but that’s no excuse. Management is paid (among other things) to manage complexity. (Hopefully with the help of system architects.)

If you are the boss of one of the many airlines not mentioned in this post, you might want to schedule a conversation with a system architect. Just a suggestion.

American Airlines Gradually Restores Service After Yesterday’s Power Outage (PR Newswire, 15 August 2003)

British Airways computer outage causes flight delays (Guardian, 6 Sept 2016)

Delta: ‘Large-scale cancellations’ after crippling power outage (CNN Wire, 8 August 2016)

Gatwick Airport Christmas Eve chaos a ‘wake-up call’ (BBC News, 11 April 2014)

Simon Calder, Dozens of flights worldwide delayed by computer systems meltdown (Independent, 14 October 2016)

Jon Cox, Ask the Captain: Do vital functions on planes have backup power? (USA Today, 6 May 2013)

Jad Mouawad, American Airlines Resumes Flights After a Computer Problem (New York Times, 16 April 2013)

 Marni Pyke, Southwest Airlines apologizes for delays as it rebounds from outage (Daily Herald, 20 July 2016)

Alexandra Zaslow, Outdated Technology Likely Culprit in Southwest Airlines Outage (NBC News, Oct 12 2015)

Updated 14 October 2016.