Aggregated enterprise architecture wisdom
We recently published our Risk And Compliance Tech Tide report outlining 14 core technologies to track in 2018. One of the challenging parts of this research is setting the right scope. We found risk and compliance technology everywhere, covering every…
@NilsPratley blames delusion in the boardroom (on a grand scale, he says) for Carillion’s collapse. “In the end, it comes down to judgments made in the boardroom.”A letter to the editor of the Financial Times agrees.”This situation has been caused, in …
In one of my earlier posts about technical debt, I differentiated between intentional debt (that taken on deliberately and purposefully) and accidental debt (that which just accrues over time without rhyme or reason or record). Dealing with (in the sense of evaluating, tracking, and resolving it) technical debt is obviously a consideration for someone in […]
Some things seem so logically inconsistent that you just have to check them out. Such was the title of a post on LinkedIn that I saw the other day: “Innovation In Fear-Based Cultures? Or, why hire lions to be dogs?”. In it, Michael Graber noted that “…top-down organizations have the most trouble innovating.”: In particular, […]
When Complex Event Processing (CEP) emerged around ten years ago, one of the early applications was real-time risk management. In the financial sector, there was growing recognition for the need for real-time visibility – continuous calibration of positions – in order to keep pace with the emerging importance of algorithmic trading. This is now relatively well-established in banking and trading sectors; Chemitiganti argues that the insurance industry now faces similar requirements.
In 2008, Chris Martins, then Marketing Director for CEP firm Apama, suggested considering CEP as a prospective “dog whisperer” that can help manage the risk of the technology “dog” biting its master.
But “dog bites master” works in both directions. In the case of Eliot Spitzer, the dog that bit its master was the anti money-laundering software that he had used against others.
And in the case of algorithmic trading, it seems we can no longer be sure who is master – whether black swan events are the inevitable and emergent result of excessive complexity, or whether hostile agents are engaged in a black swan breeding programme. One of the first CEP insiders to raise this concern was John Bates, first as CTO at Apama and subsequently with Software AG. (He now works for a subsidiary of SAP.)
 |
| from Dark Pools by Scott Patterson |
And in 2015, Bates wrote that “high-speed trading algorithms are an alluring target for cyber thieves”.
So if technology is capable of both generating unexpected events and amplifying hostile attacks, are we being naive to imagine we use the same technology to protect ourselves?
Perhaps, but I believe there are some productive lines of development, as I’ve discussed previously on this blog and elsewhere.
1. Organizational intelligence – not relying either on human intelligence alone or on artificial intelligence alone, but looking for establishing sociotechnical systems that allow people and algorithms to collaborate effectively.
2. Algorithmic biodiversity – maintaining multiple algorithms, developed by different teams using different datasets, in order to detect additional weak signals and generate “second opinions”.
John Borland, The Technology That Toppled Eliot Spitzer (MIT Technology Review, 19 March 2008) via Adam Shostack, Algorithms for the War on the Unexpected (19 March 2008)
Vamsi Chemitiganti, Why the Insurance Industry Needs to Learn from Banking’s Risk Management Nightmares.. (10 September 2016)
Theo Hildyard, Pillar #6 of Market Surveillance 2.0: Known and unknown threats (Trading Mesh, 2 April 2015)
Neil Johnson et al, Financial black swans driven by ultrafast machine ecology (arXiv:1202.1448 [physics.soc-ph], 7 Feb 2012)
Chris Martins, CEP and Real-Time Risk – “The Dog Whisperer” (Apama, 21 March 2008)
Scott Patterson, Dark Pools – The Rise of A. I. Trading Machines and the Looming Threat to Wall Street (Random House, 2013). See review by David Leinweber, Are Algorithmic Monsters Threatening The Global Financial System? (Forbes, 11 July 2012)
Richard Veryard, Building Organizational Intelligence (LeanPub, 2012)
Related Posts
The Shelf-Life of Algorithms (October 2016)
What does the World War II naval campaign known as the Battle of the Atlantic have to do with learning and innovation? Quite a lot, as it turns out. Early in the war, Britain found itself in a precarious position. While being an island nation provided defensive advantages, it also came with logistical challenges. Food, […]
@mrkwpalmer (TIBCO) invites us to take what he calls a Hyper-Darwinian approach to analytics. He observes that “many algorithms, once discovered, have a remarkably short shelf-life” and argues that one must be as good at “killing off weak or vanquished…
What’s the biggest problem with technical debt? In my opinion, the biggest problem is that it works. Just like the electrical outlet pictured above, systems with technical debt get the job done, even when there’s a hidden surprise or two waiting to make life interesting for us at some later date. If it flat-out failed, […]
What does an old World War II tank have to do with innovation? I’ve mentioned it before, but it bears repeating – one of benefits of having a blog is the ability to interact with and learn from people all over the world. For example, Greger Wikstrand and I have been trading blog posts on […]
Last week, in a post titled “The Flaw in All Things”, John Vincent discussed the problem of seeing “the flaw in all things”: It’s overwhelming. It’s paralyzing. I can’t finish a project because I keep finding things that could cause problems. I even mentioned this to our CTO and CEO at one point when we […]
I had the pleasure of appearing on episode #367 of Architecture Corner, “Fail fast, learn fast”, with Greger Wikstrand and Casimir Artmann. In the episode, we discuss learning, experiments, and the idea of “fail fast” in relation to the recent incident with Microsoft’s artificial intelligence chatbot, @TayandYou. I hope you enjoy the discussion as much […]
Take a couple of seconds and watch the clip in the tweet below: While it would be incredibly difficult to predict that exact outcome, it is also incredibly easy to foresee that it’s a possibility. As the saying goes, “forewarned is forearmed”. Being forewarned and forearmed is an important part of what an architect does. […]