4 years, 11 months ago

Applications, work-instructions and business-continuity

For business-continuity after disruption or failure, what’s the first thing we’ll need? Answer: an alternative way to do things. For business-process redesign, what’s the first thing we’ll need? Answer: an alternative way to do things. In both cases, we’ll need

7 years, 10 months ago

Single Point of Failure (Comms)

Large business-critical systems can be brought down by power failure. My previous post looked at Airlines. This time we turn our attention to Telecommunications.

If someone said you had to accept an unreliable electricity supply as the price of innovation in appliances, you’d laugh. #NotNeutrality

— Martin Geddes (@martingeddes) August 8, 2016

More misery for BT broadband users after new power cut. Looks like ‘no single point of failure’ is an alien concept. https://t.co/mOobFidWe4

— Chris Tripp (@ChrisJTripp) July 21, 2016

It would be interesting to know where the single point of failure was in their power protection plan. https://t.co/zuaTm1z4tK

— Robin Koffler MBA (@robin_koffler) July 21, 2016

2G and 3G data services from @EE are down after a power outage. Details: https://t.co/zEJFpgpl4n pic.twitter.com/vcUOkPVtet

— The Register (@TheRegister) September 2, 2016

Obviously a power cut is not the only possible cause of business problems. Another single-point of failure could be a single rogue employee.

That shows that management should look at automating network. Since Network is single point of failure. https://t.co/ND5UXtNntj

— Anurag Kaushik (@kaushikanuk) August 3, 2016

Gavin Clarke, Telecity’s engineers to spend SECOND night fixing web hub power outage (The Register, 18 November 2015)

Related Post: Single Point of Failure (Airlines) (August 2016)

7 years, 11 months ago

Single Point of Failure (Airlines)

Large business-critical systems can be brought down by power failure. Who knew?

In July 2016, Southwest Airlines suffered a major disruption to service, which lasted several days. It blamed the failure on “lingering disruptions following performance issues across multiple technology systems”, apparently triggered by a power outage.

Click below for the latest update on our system and operation: https://t.co/bqV1qwahmz

— Southwest Airlines (@SouthwestAir) July 21, 2016

In August 2016 it was Delta’s turn.

New statement from Delta – power outage caused IT failure pic.twitter.com/trkQbpym05

— Rory Cellan-Jones (@ruskin147) August 8, 2016

@ruskin147 A power outage *triggered* this issue, but poor planning and no HA *caused* it. Why can Netflix get this right but airlines cant?

— Richard Price (@RichardPrice) August 8, 2016

I am no computer expert but it seems like a whole system crashing (3 separate airlines) points to bad design (single point of failure)? 3/

— Dan DePodwin (@WxDepo) August 8, 2016

Then there were major problems at British Airways (Sept 2016) and United (Oct 2016).

@razankhabour We apologize to our customers for the delay and we appreciate their patience as our IT teams work to resolve this issue.

— British Airways (@British_Airways) September 6, 2016

We’re aware of an issue with our system and are working to resolve it. We’ll update as we learn more. We apologize for the inconvenience.

— United (@united) October 14, 2016

So every @united flight is grounded because they can’t run a decent IT shop. What year is this??

— Randy Bias (@randybias) October 14, 2016

The concept of “single point of failure” is widely known and understood. And the airline industry is rightly obsessed by safety. They wouldn’t fly a plane without backup power for all systems. So what idiot runs a whole company without backup power?

We might speculate what degree of complacency or technical debt can account for this pattern of adverse incidents. I haven’t worked with any of these organizations myself. However, my guess is that some people within the organization were aware of the vulnerability, but this awareness didn’t somehow didn’t penetrate the management hierarchy. (In terms of orgintelligence, a short-sighted board of directors becomes the single point of failure!) I’m also guessing it’s not quite as simple and straightforward as the press reports and public statements imply, but that’s no excuse. Management is paid (among other things) to manage complexity. (Hopefully with the help of system architects.)

If you are the boss of one of the many airlines not mentioned in this post, you might want to schedule a conversation with a system architect. Just a suggestion.

American Airlines Gradually Restores Service After Yesterday’s Power Outage (PR Newswire, 15 August 2003)

British Airways computer outage causes flight delays (Guardian, 6 Sept 2016)

Delta: ‘Large-scale cancellations’ after crippling power outage (CNN Wire, 8 August 2016)

Gatwick Airport Christmas Eve chaos a ‘wake-up call’ (BBC News, 11 April 2014)

Simon Calder, Dozens of flights worldwide delayed by computer systems meltdown (Independent, 14 October 2016)

Jon Cox, Ask the Captain: Do vital functions on planes have backup power? (USA Today, 6 May 2013)

Jad Mouawad, American Airlines Resumes Flights After a Computer Problem (New York Times, 16 April 2013)

 Marni Pyke, Southwest Airlines apologizes for delays as it rebounds from outage (Daily Herald, 20 July 2016)

Alexandra Zaslow, Outdated Technology Likely Culprit in Southwest Airlines Outage (NBC News, Oct 12 2015)

Updated 14 October 2016.