#eavoices

The Unexpected Happens

by

When Complex Event Processing (CEP) emerged around ten years ago, one of the early applications was real-time risk management. In the financial sector, there was growing recognition for the need for real-time visibility – continuous calibration of positions – in order to keep pace with the emerging importance of algorithmic trading. This is now relatively well-established in banking and trading sectors; Chemitiganti argues that the insurance industry now faces similar requirements.

In 2008, Chris Martins, then Marketing Director for CEP firm Apama, suggested considering CEP as a prospective “dog whisperer” that can help manage the risk of the technology “dog” biting its master.

But “dog bites master” works in both directions. In the case of Eliot Spitzer, the dog that bit its master was the anti money-laundering software that he had used against others.

And in the case of algorithmic trading, it seems we can no longer be sure who is master – whether black swan events are the inevitable and emergent result of excessive complexity, or whether hostile agents are engaged in a black swan breeding programme.  One of the first CEP insiders to raise this concern was John Bates, first as CTO at Apama and subsequently with Software AG. (He now works for a subsidiary of SAP.)

from Dark Pools by Scott Patterson

And in 2015, Bates wrote that “high-speed trading algorithms are an alluring target for cyber thieves”.

So if technology is capable of both generating unexpected events and amplifying hostile attacks, are we being naive to imagine we use the same technology to protect ourselves?

Perhaps, but I believe there are some productive lines of development, as I’ve discussed previously on this blog and elsewhere.

1. Organizational intelligence – not relying either on human intelligence alone or on artificial intelligence alone, but looking for establishing sociotechnical systems that allow people and algorithms to collaborate effectively.

2. Algorithmic biodiversity – maintaining multiple algorithms, developed by different teams using different datasets, in order to detect additional weak signals and generate “second opinions”.

John Bates, Algorithmic Terrorism (Apama, 4 August 2010). To Catch an Algo Thief (Huffington Post, 26 Feb 2015)

John Borland, The Technology That Toppled Eliot Spitzer (MIT Technology Review, 19 March 2008) via Adam Shostack, Algorithms for the War on the Unexpected (19 March 2008)

Vamsi Chemitiganti, Why the Insurance Industry Needs to Learn from Banking’s Risk Management Nightmares.. (10 September 2016)

Theo Hildyard, Pillar #6 of Market Surveillance 2.0: Known and unknown threats (Trading Mesh, 2 April 2015)

Neil Johnson et al, Financial black swans driven by ultrafast machine ecology (arXiv:1202.1448 [physics.soc-ph], 7 Feb 2012)

Chris Martins, CEP and Real-Time Risk – “The Dog Whisperer” (Apama, 21 March 2008)

Scott Patterson, Dark Pools – The Rise of A. I. Trading Machines and the Looming Threat to Wall Street (Random House, 2013). See review by David Leinweber, Are Algorithmic Monsters Threatening The Global Financial System? (Forbes, 11 July 2012)

Richard Veryard, Building Organizational Intelligence (LeanPub, 2012)

Related Posts

The Shelf-Life of Algorithms (October 2016)

If You Want to Create an Enterprise Architecture; Don’t!

by
One of the last presentations I made as an Enterprise Architect for a major DoD contractor was to the Chief Architect of the US Veterans Administration.  I walked in with a fully prepared presentation that was to take about 10 minutes of the time allotted to our team only to find the Chief Architect cutting the presentation off with a question, “How do we go about creating an IT architecture for the VA?”  Even though I had a very good answer and had applied it on a couple of occasion, my mind blanked.  I want to share with you his problem and the answer I should have given.


The Problem

The problem that the Chief Architect of the VA has is the same problem that plagues CA’s of all large organization and most of medium and smaller organizations.  That question is base on the very logical idea very much the analog of the idea that before you start changing the plumbing, you should know design of the current plumbing; that is, before you can create a “to be” or “next step” architecture you need to have a “current architecture”.  Obviously, if you don’t know which pipes connect where and start making changes to the plumbing you could end up with some very interesting and exciting results for which you may need to call your insurance company.  Likewise, if you want to improve the effectiveness and/or the cost efficiency of the organizational processes and information systems, most Enterprise Architects assume they must first define and delimit the “as-is” processes and information systems for the organization.


The conundrum is that, in today’s technological environment, by the time an IT architecture team has mapped out (structured and ordered) an “as is” architecture, some, most, or all of the elements and data of the architecture will be obsolete and out of date.  For something as large as a major corporation, a department within a state or the federal government, the cost and effort involved would require a tour de force on a very large perhaps unprecedented scale.  This cost and level of effort would be such that the senior management would cut funding to the effort as a waste of time and money, since having an “as-is” architecture by itself produces little in value to the organization.


As can be found in the literature, there are many ways to “solve” or at least ameliorate the problem of creating an “as-is” architecture.  For example, one of the best, that almost works, is to chop the organization into its components and create an “as-is” architecture for each component separately.  Then try to stitch the architectures together.  I’ve tried this and it works up to a point.


There is a truism in Systems Engineering, Systems Architecture, and Enterprise Architecture, “Optimizing the sub-systems will sub-optimize the system”.  I have demonstrated this to many people many times and experienced it several times.  But this is crux of the problem for those that try to create an Enterprise Architecture for a large organization.


The Solution

The simple answer is “Don’t”.  That is, “Don’t attempt to create an “as is” architecture for an organization, especially a large organization, because it will create itself with the proper procedures in place.  So how would I do it?


1. Define, delimited, and structured an initial set of classes and attributes for the Organization’s Enterprise Architecture.  These should include:

  • Its Charter, Mission, Goal

  • Its Strategies for achieving its charter, mission, or goal

  • Its Processes supporting its strategies

  • Its Tooling and infrastructure

  • Its Governance that affects any of the above, including:

  • Internal Policies and Standards

  • External Regulations and Standards

I worked with one Enterprise Architecture database that had over fifty classes, each class with ten or more attributes.  This was a fairly mature architecture.  My recommendation, don’t try to think of all the classes you may need or all of the attributes for each class; that’s way over thinking.  Instead, start simple and add through the cycles

2. Once you have designed and structured the initial set of classed and attributes, create a data base structured according to the design.


3. Here is the key to creating an “As-Is” Architecture by not creating it…Huh?  Design and implement processes to capture the current state of strategies, processes, and tooling/infrastructure as part of review of funding for revision and upgrades to the current systems and processes.  


4. When personnel in the organization propose a project insist that these personnel demonstrate the value of the process or procedure that they intend to update or upgrade. The “value” would include demonstrating which of and how the current product, system, or service enables the processes, strategies, and charter, mission, or goal of the organization. My experience has been that the initial attempts will be fuzzy and incomplete, but that as the number of proposed projects in the database (which is generally called the Asset Management System and on which the “as-is” architecture is built) increases both the completeness and clarity of the current enterprise architecture will increase.


The reason I say “Don’t” try to create an “as-is” architecture is that every 3 to 7 years every component of the organization’s information system will need replacement.  This means that within 3 to 5 years simply by documenting and structuring the inputs from all of the efforts the organization’s “as-is” architecture will be synergistically created (and at minimal cost) [Sidebar: There will be some cost because the project proposers will need to think through how their current charter, mission, or goal and the strategies they support links to and supports the overall charter, mission, or goal of the organization.  This is not necessarily a bad thing.]  For large organizations, no matter how much time or effort is put into attempting to create an “As-Is” Enterprise Architecture, it will take a minimum of a year and a great deal of funding; so it simply makes no sense.


As this Enterprise Architecture evolves you will begin to see a number of things that managers want to obfuscate or hide completely.  For example, a number of processes and component or sub-organizations will be demonstrated to be obsolete.  In this case obsolete means that the process or component organization no longer supports any of the organization’s strategies or its goal.  Since managers want to build or at least keep their fiefdoms they will not appreciate this much.  Additionally, it will demonstrate which internal policies, regulations, and standards help the organization and which hurt it in meeting its goal.  Again, the gatekeepers of these policies, regulations, and standards will object–strenuously.  


But there are two more insidious problems that a good “As-Is” Enterprise Architecture will reveal, nepotism and the famous “Catch-22s”.  


Nepotism

Nepotism in this case is more broadly defined than what most people think of as “nepotism”.  In the sense I mean, nepotism can include creating a non-level economic playing field. In all large organizations, but especially in the U.S. Federal government (probably in all governments) the type of nepotism I’m identifying is rampant.  In fact a December 2016 report from the Department of Defense highlights what most federal employees and DoD contractors have known for years, because representatives and senators will only vote for a large program if their district or state gets a part of it, the DoD estimates that the cost of the program increases approximately 20 percent.  This is “jobs welfare” on a massive scale.  Some major defense contractors have plants in every state for just this reason, not because it make any sense from a cost efficiency perspective.  Further, Congress had passed laws to ensure that minority and female owned businesses.  The reason is that minorities and women scream that the good old boy network doesn’t allow them to compete for sub-contracts [Sidebar: Actually the reason for the “good ole boy” network is that the prime contractors have sub-contractors that actually know what their doing.  In my experience, many times primes will “encourage”–read subsidize–inexperienced and frequently incompetent minority and female owned businesses in order to meet these regulations imposed on their proposals.]  Again, this is a form of social welfare to ensure all political constituents that scream loudly are appeased.  This adds up to the DoD being one of the larger governmental welfare organizations. [Sidebar: While, seemingly, I’ve picked on government organizations, especially the U.S. DoD, and while I have found that all governmental organizations in a democracy will have this type of nepotism.  This is what lobbing is all about.  Only when it goes so far that it’s plain to all and when it’s not openly enacted into law that we call it graft and corruption.]  And it’s not only governments that suffer from this type of nepotism, all large organizations have the same problems, though generally on a smaller scale.  For example, sometimes the nepotism is written into union contracts.  Along with finance engineering, the auto industry in Detroit suffered a near collapse due to contractual nepotism.

This presents a problem for any Enterprise Architect.  The as-is architecture will highlight the nepotism of this type more clearly than any report.  The Enterprise Architect won’t need to report it to the management, it will be self-evident.  I’ve experienced a situation, as I suspect many of you have, where the management kills the messenger in order to not address the problem.  In my case, three times I’ve been chased off programs when I reported that the effort was subsidizing silliness.


Catch-22

The second significant problem that policies, regulations, and standards become contradictory to each other or in combination make it impossible for the organization to achieve its goal.  Again, a good enterprise architecture will highlight these, though frequently, when management from one generation of technology with its set of policies and standards, finds the next upon them, they will refuse to resend or modify the existing regulations, preferring instead, again, to kill the messenger.  So like Systems Engineers, I’ve found that enterprise architects are only respected by other enterprise architects.


5. When the development and implementation team completes a project, and once it goes into operation, then as a final step in their effort, they should review the data they gave to the enterprise architect, revising the data to accurately reflect the “as-built” instead of the “as-proposed”.  The as-built documentation must include all component, assembly or functional, and customer acceptance testing, and all post production required changes.  This documentation will inevitably lead to additional class attributes of the Asset Management System and structure in the enterprise architecture.


6. As the Asset Management System and the Enterprise Architecture matures, management should prepare for a paradigm shift in the way projects and other efforts are proposed.  This is where Enterprise Architecture really demonstrates how it can make the organization both more effective and cost efficient.


A mature enterprise architecture can serve as the basis for a dynamic business or organizational process model for the organization.  Management can use this model to identify obsolete processes, (and as discussed) policies, regulations, and standards; ones that the organization should eliminate.  Additionally, with the help of the Enterprise Architect, management can identify missing or inhibiting processes and tools, and identify bottlenecks and dams in process flows.


Further, they can model what happens when the missing and inhibiting processes and tools are added or when the bottlenecks are eliminated or reduced.  This modeling will then indicate where there is a need for new efforts and to some degree the effectiveness and cost efficiency of such efforts.  It’s a paradigm shift in that no longer to component or sub-units of the organization propose changes.  Instead, senior management working with the Enterprise Architect and the component or sub-units will identify and fund efforts.  They now have a way to measure the potential of the change in meeting the organizational goal, which means senior management has a better way of managing organizational change.


Finally, once management has identified targets for change or upgrade, the enterprise architect together with a system architect can define alternatives to meet the effort’s requirements.  They can model alternative process and tooling changes to forecast which has the lowest potential risk, the highest potential return, the least disruption of current activities, lowest initial cost, lowest lifecycle cost, the most adaptable or agile, or any number of other targets defined by the senior management.  This will make the organization much more cost efficient, and perhaps more cost effective; and this is the purpose of Enterprise Architecture,


To sum up, using this six step, high-level process is an effective way to create both an Asset Management System (an “As-Is” Architecture) and an effective Enterprise Architecture process; perhaps the only way.  

Agility, SOA, Virtual Extended Enterprise, Swarming Tactics, and Architecture

by

Agility and the Virtual Extended EnterpriseIn the 1990s, The Agility Forum of Lehigh University defined Agility as “The ability to successfully respond to unexpected challenges and opportunities.” The forum chartered a technical committe…

Agility, SOA, Virtual Extended Enterprise, Swarming Tactics, and Architecture

by

Agility and the Virtual Extended Enterprise

In the 1990s, The Agility Forum of
Lehigh University defined Agility as “The ability to successfully respond to
unexpected challenges and opportunities.” The
forum chartered a technical commi…

Agility, SOA, Virtual Extended Enterprise, Swarming Tactics, and Architecture

by

Agility and the Virtual Extended EnterpriseIn the 1990s, The Agility Forum of Lehigh University defined Agility as “The ability to successfully respond to unexpected challenges and opportunities.” The forum chartered a technical committe…