Aggregated enterprise architecture wisdom
Enterprises accrue technical debt by hurriedly writing new code and deploying more software to keep up. If not properly managed, that technical debt will lead to poor business outcomes. Learn how technology lifecycle management can help.
Episode 5 of this season of Architecture Corner is out (I made a guest appearance in episode 1, “Good at Innovation”). In this installment, Chris the CEO falls victim to yet another temptation. The CEO has decided that no more time and money will be “wasted” on old systems. Can Joakim Lindbom convince him that […]
In one of my earlier posts about technical debt, I differentiated between intentional debt (that taken on deliberately and purposefully) and accidental debt (that which just accrues over time without rhyme or reason or record). Dealing with (in the sense of evaluating, tracking, and resolving it) technical debt is obviously a consideration for someone in […]
Large business-critical systems can be brought down by power failure. Who knew?
In July 2016, Southwest Airlines suffered a major disruption to service, which lasted several days. It blamed the failure on “lingering disruptions following performance issues across multiple technology systems”, apparently triggered by a power outage.
Click below for the latest update on our system and operation: https://t.co/bqV1qwahmz— Southwest Airlines (@SouthwestAir) July 21, 2016
In August 2016 it was Delta’s turn.
New statement from Delta – power outage caused IT failure pic.twitter.com/trkQbpym05— Rory Cellan-Jones (@ruskin147) August 8, 2016
@ruskin147 A power outage *triggered* this issue, but poor planning and no HA *caused* it. Why can Netflix get this right but airlines cant?— Richard Price (@RichardPrice) August 8, 2016
I am no computer expert but it seems like a whole system crashing (3 separate airlines) points to bad design (single point of failure)? 3/— Dan DePodwin (@WxDepo) August 8, 2016
Then there were major problems at British Airways (Sept 2016) and United (Oct 2016).
@razankhabour We apologize to our customers for the delay and we appreciate their patience as our IT teams work to resolve this issue.— British Airways (@British_Airways) September 6, 2016
We’re aware of an issue with our system and are working to resolve it. We’ll update as we learn more. We apologize for the inconvenience.— United (@united) October 14, 2016
So every @united flight is grounded because they can’t run a decent IT shop. What year is this??— Randy Bias (@randybias) October 14, 2016
The concept of “single point of failure” is widely known and understood. And the airline industry is rightly obsessed by safety. They wouldn’t fly a plane without backup power for all systems. So what idiot runs a whole company without backup power?
We might speculate what degree of complacency or technical debt can account for this pattern of adverse incidents. I haven’t worked with any of these organizations myself. However, my guess is that some people within the organization were aware of the vulnerability, but this awareness didn’t somehow didn’t penetrate the management hierarchy. (In terms of orgintelligence, a short-sighted board of directors becomes the single point of failure!) I’m also guessing it’s not quite as simple and straightforward as the press reports and public statements imply, but that’s no excuse. Management is paid (among other things) to manage complexity. (Hopefully with the help of system architects.)
If you are the boss of one of the many airlines not mentioned in this post, you might want to schedule a conversation with a system architect. Just a suggestion.
American Airlines Gradually Restores Service After Yesterday’s Power Outage (PR Newswire, 15 August 2003)
British Airways computer outage causes flight delays (Guardian, 6 Sept 2016)
Delta: ‘Large-scale cancellations’ after crippling power outage (CNN Wire, 8 August 2016)
Gatwick Airport Christmas Eve chaos a ‘wake-up call’ (BBC News, 11 April 2014)
Simon Calder, Dozens of flights worldwide delayed by computer systems meltdown (Independent, 14 October 2016)
Jon Cox, Ask the Captain: Do vital functions on planes have backup power? (USA Today, 6 May 2013)
Jad Mouawad, American Airlines Resumes Flights After a Computer Problem (New York Times, 16 April 2013)
Marni Pyke, Southwest Airlines apologizes for delays as it rebounds from outage (Daily Herald, 20 July 2016)
Alexandra Zaslow, Outdated Technology Likely Culprit in Southwest Airlines Outage (NBC News, Oct 12 2015)
Updated 14 October 2016.
Over the last few weeks, I’ve run across a flurry of articles dealing with the issue of legacy systems used by the U.S. government. An Associated Press story on the findings from the Government Accountability Office (GAO) issued in May reported that roughly three-fourths of the $80 billion IT budget was used to maintain […]
What’s the biggest problem with technical debt? In my opinion, the biggest problem is that it works. Just like the electrical outlet pictured above, systems with technical debt get the job done, even when there’s a hidden surprise or two waiting to make life interesting for us at some later date. If it flat-out failed, […]
Technical debt, in the enterprise architecture context, measures the departure of the current enterprise organization from the good practices of enterprise architecture.
The more exceptions to the good rules of practice, the …