David Sprott

Mapping Agile Architecture

by

Jason Bloomberg recently published a mind map for Agile Architecture. It’s a nice map that sketches top level thinking and I welcome that. It prompted me to do a drill down.

Mind maps are useful in that they are, by definition free form and intended to support brain storming. The downside is obvious – they are generally inconsistent and cause modelers’ intense frustration! Caveat emptor over, I fully agree with Jason that we need a dual interpretation of Agile  – that is Agile practices and Agile Architecture, and I have written about this on many occasions. Also that the entire motivation is about business agility. On this last point my mind map is clearly a little more technical than Jason’s, and on reflection I think that is because it’s essential to converge the business and technology concerns.

For example, the map suggests a strong capability centric approach to interpret the business morphology. However this is insufficient; the technology must also establish appropriate levels of implementation independence that will facilitate the pluggability of business capabilities. Similarly you might think that considerations regarding the platform and delivery technology (such as MDA/MDD) are irrelevant to business concerns. However the platform and platform delivery technology are potentially massive drivers of rapid iteration and ongoing change, because they encapsulate common application level infrastructure and common services, so understanding the “business” standardization and localization model is crucial to delivering agility through this structure.

References:
Related posts: 

Not Another Framework? Part 2

by

In my last post Oh No! We need another Practice Framework,  I developed the earlier theme commenced in “Beware the New Silos”. I argued that the widely used frameworks are narrowly discipline centric and actually inhibit cross discipline working. I described how my own firm’s experiences have led to the development of a de facto framework, (we call it SOAM) and illustrated how this is essentially a value chain commencing with customer demand and finishing with value add to some enterprise.

I ended by sketching some basic principles concluding that we need a new framework that is goal driven and incorporates the entire value chain of capabilities, which of course may selectively reuse some parts of existing frameworks. In this post, I suggest a strawman that covers a) principles and b) capability model.

Before diving into principles, it will be useful to declare some scope. Our framework has developed from working with larger enterprises, both commercial and government in the area of business service and solution delivery. All of these enterprises share common issues that they have extensive legacy application assets that act as a serious inhibitor to business change, and successive, narrowly scoped solution projects over many years have often resulted in great complexity and technical debt. It is also common in my experience that enterprise architecture functions are routinely bypassed or ignored; that Agile methods have been attempted and found useful on narrow focused projects, but because of the constrained focus, tend to increase overall complexity of the ongoing application asset base; that consistent customer experiences are commonly compromised by narrow focused projects; and line of business managers in large enterprises are frequently dissatisfied with IT application service support.

The objectives of the framework are to:

– describe practices relevant to service and solution delivery in the digital business environment
– achieve a balance between short term goals and longer term objectives
– support progressive transformation to an enterprise comprised of independent business capabilities
– facilitate continuous, short cycle time evolution of business capability
– progressively and continuously resolve legacy portfolio complexities
– enable rapid delivery at low cost without compromise in quality

Principles are foundational for any framework. 

Principles should be enduring and lead to both excellent policy communication and policy interpretation in everyday situations. I also find it useful to classify principles by subject.

Capability Model

In business architecture the capability model has become ubiquitous. And in thinking organizations I observe delivery of highly independent service and solution components that reduce dependencies and the impact of change, as well as mirroring the IT architecture on the business organization. Why wouldn’t we use the same approach in defining a set of activities to deliver services and solutions?

If you are uncertain about the capability concept, it’s important to appreciate that the optimum business capability is one that enables:
– maximum cohesion of internal functional capability, plus consistency of life cycle, strategic class (core, context, innovating . . . ), business partition (global, local, LoB . . ), standardization, customizability, stability, metrics and drivers
– defined, stable dependencies that are implemented as services
[Further reading on capability optimization ]

In the capability dependency model below, the arrows are dependencies. For example, Demand Shaping is dependent upon Conceptual Business Modeling and Portfolio Management.  So this is not a flow diagram, rather all the capabilities should be regarded as iterative, I will come back and discuss how Lean principles operate in a framework like this, and as discussed above, highly independent.

Most of the capabilities in the model are self-explanatory. However some need explanation:
1. The Conceptual Business Modeling capability is the ability for business stakeholders to describe business improvement in conceptual terms. Many business people speak in solution terms. Most business requirements therefore surface as solutions, some more baked than others. Because the business stakeholder generally has the budget, the solution vision frequently drives and shapes the project with outcomes that frequently compromise the existing and planned portfolio. By educating business stakeholders to communicate in concepts, the opportunity is created to develop the business improvement idea without preconceptions of implementation or product, and to optimize architectural and portfolio integration. 
2. Demand Management is reasonably well understood. Demand Shaping is best regarded as a complementary capability that takes raw customer demand and decomposes into components such as pre-existing or planned services/APIs, considers opportunities for modernization and provisioning, and reassembles as a set of projects or project components that optimize the progressive development of the portfolio. Demand Shaping is primarily an architectural task, but should be run by a cross functional team including architect, product management, business design and technical expert roles. 
3. The Architecture Capability is shown as a decomposition of sub-capabilities, essentially one for each View, plus modernization. Whilst modernization is not classically an architecture view, there is commonly a specialist requirement for modernization architecture that will include identification of appropriate transformation and transitional architecture patterns. The primary objective of all of the architecture sub-capabilities is to define realizable structure to meet the demand and, as discussed above, to optimize opportunities for modernization and provisioning. While there is no explicit enterprise architecture View called out, each architecture capability should be executed separately and iteratively for reference, portfolio, program, project and module, thereby defining progressive layers of standard functionality that will be common to the defined scope, as well as situation specific business functionality. 
I will detail all the capabilities in a subsequent post.

Final remarks. 

This high level view of the framework has attempted to list a set of principles and associated capabilities required to support the value chain illustrated in Part 1 of this extended blog post. What will hopefully have become clear is the need for architecture capabilities particularly to be involved throughout the value chain. This approach integrates all types of architecture (enterprise, service, solution, deployment  . . . ) into the business improvement value chain and creates better opportunity to demonstrate the ROI on architecture. Further the approach prevents enterprise architecture particularly becoming divorced from mainstream business improvement and encourages a better balance of short term and strategic goals. What will not yet be fully clarified is how the framework is very strongly focused on realizing architecture in delivered services and solutions, as a series of successive collaborations. I will describe how this is done using a Lean approach in a subsequent post. 
                  Beware the New Silos

Not Another Framework? Part 2

by

In my last post Oh No! We need another Practice Framework,  I developed the earlier theme commenced in “Beware the New Silos”. I argued that the widely used frameworks are narrowly discipline centric and actually inhibit cross discipline working. I described how my own firm’s experiences have led to the development of a de facto framework, (we call it SOAM) and illustrated how this is essentially a value chain commencing with customer demand and finishing with value add to some enterprise.

I ended by sketching some basic principles concluding that we need a new framework that is goal driven and incorporates the entire value chain of capabilities, which of course may selectively reuse some parts of existing frameworks. In this post, I suggest a strawman that covers a) principles and b) capability model.

Before diving into principles, it will be useful to declare some scope. Our framework has developed from working with larger enterprises, both commercial and government in the area of business service and solution delivery. All of these enterprises share common issues that they have extensive legacy application assets that act as a serious inhibitor to business change, and successive, narrowly scoped solution projects over many years have often resulted in great complexity and technical debt. It is also common in my experience that enterprise architecture functions are routinely bypassed or ignored; that Agile methods have been attempted and found useful on narrow focused projects, but because of the constrained focus, tend to increase overall complexity of the ongoing application asset base; that consistent customer experiences are commonly compromised by narrow focused projects; and line of business managers in large enterprises are frequently dissatisfied with IT application service support.

The objectives of the framework are to:

– describe practices relevant to service and solution delivery in the digital business environment
– achieve a balance between short term goals and longer term objectives
– support progressive transformation to an enterprise comprised of independent business capabilities
– facilitate continuous, short cycle time evolution of business capability
– progressively and continuously resolve legacy portfolio complexities
– enable rapid delivery at low cost without compromise in quality

Principles are foundational for any framework. 

Principles should be enduring and lead to both excellent policy communication and policy interpretation in everyday situations. I also find it useful to classify principles by subject.

Capability Model

In business architecture the capability model has become ubiquitous. And in thinking organizations I observe delivery of highly independent service and solution components that reduce dependencies and the impact of change, as well as mirroring the IT architecture on the business organization. Why wouldn’t we use the same approach in defining a set of activities to deliver services and solutions?

If you are uncertain about the capability concept, it’s important to appreciate that the optimum business capability is one that enables:
– maximum cohesion of internal functional capability, plus consistency of life cycle, strategic class (core, context, innovating . . . ), business partition (global, local, LoB . . ), standardization, customizability, stability, metrics and drivers
– defined, stable dependencies that are implemented as services
[Further reading on capability optimization ]

In the capability dependency model below, the arrows are dependencies. For example, Demand Shaping is dependent upon Conceptual Business Modeling and Portfolio Management.  So this is not a flow diagram, rather all the capabilities should be regarded as iterative, I will come back and discuss how Lean principles operate in a framework like this, and as discussed above, highly independent.

Most of the capabilities in the model are self-explanatory. However some need explanation:
1. The Conceptual Business Modeling capability is the ability for business stakeholders to describe business improvement in conceptual terms. Many business people speak in solution terms. Most business requirements therefore surface as solutions, some more baked than others. Because the business stakeholder generally has the budget, the solution vision frequently drives and shapes the project with outcomes that frequently compromise the existing and planned portfolio. By educating business stakeholders to communicate in concepts, the opportunity is created to develop the business improvement idea without preconceptions of implementation or product, and to optimize architectural and portfolio integration. 
2. Demand Management is reasonably well understood. Demand Shaping is best regarded as a complementary capability that takes raw customer demand and decomposes into components such as pre-existing or planned services/APIs, considers opportunities for modernization and provisioning, and reassembles as a set of projects or project components that optimize the progressive development of the portfolio. Demand Shaping is primarily an architectural task, but should be run by a cross functional team including architect, product management, business design and technical expert roles. 
3. The Architecture Capability is shown as a decomposition of sub-capabilities, essentially one for each View, plus modernization. Whilst modernization is not classically an architecture view, there is commonly a specialist requirement for modernization architecture that will include identification of appropriate transformation and transitional architecture patterns. The primary objective of all of the architecture sub-capabilities is to define realizable structure to meet the demand and, as discussed above, to optimize opportunities for modernization and provisioning. While there is no explicit enterprise architecture View called out, each architecture capability should be executed separately and iteratively for reference, portfolio, program, project and module, thereby defining progressive layers of standard functionality that will be common to the defined scope, as well as situation specific business functionality. 
I will detail all the capabilities in a subsequent post.

Final remarks. 

This high level view of the framework has attempted to list a set of principles and associated capabilities required to support the value chain illustrated in Part 1 of this extended blog post. What will hopefully have become clear is the need for architecture capabilities particularly to be involved throughout the value chain. This approach integrates all types of architecture (enterprise, service, solution, deployment  . . . ) into the business improvement value chain and creates better opportunity to demonstrate the ROI on architecture. Further the approach prevents enterprise architecture particularly becoming divorced from mainstream business improvement and encourages a better balance of short term and strategic goals. What will not yet be fully clarified is how the framework is very strongly focused on realizing architecture in delivered services and solutions, as a series of successive collaborations. I will describe how this is done using a Lean approach in a subsequent post. 
                  Beware the New Silos

Oh No! We need another Practice Framework?

by

A few years ago I commented [in Beware the New Silos, ref 1 below] that in a complicated world we cope by specialization – and across the industry in general and in individual enterprises specifically we have created silos of our primary disciplines. The widely used frameworks and methods illustrate that common practice of discipline centricity. We don’t have to look too far for examples such as Enterprise Architecture (TOGAF), Governance (Cobit), SOA (separately by OASIS, OMG, Open Group), Agile methods (many and various), Business Process Management (BPM) and IT Service Management (ITIL). All of these disciplines, whether de jure or de facto standards, are all very narrowly focused with minimal treatment of how the disciplines might work together.

A good example is how Agile methods are tightly focused on application development and architecture is assumed to be an integral part of the Agile delivery project. Yet the reality in all enterprises is that significant aspects of architecture must be consistent at the domain or enterprise level. Another good example is how the three standards bodies OASIS, OMG and the Open Group were so divergent in their treatment of SOA, they commissioned a report [ref 2 below] to explain how these inherently duplicating standards and specifications relate to each other.

The level of adoption by enterprises or service providers of all these and similar practice frameworks and standards is of course highly variable. However it must be said that the very existence of the discipline based materials will frequently have some considerable influence on how enterprises organize themselves.

The thinking IT or business professional might also like to question just how up to date these frameworks are, and how they support today’s business goals, which for most of us have changed dramatically over the past few years. We might also speculate whether the education and certification ecosystems that feed off some discipline based frameworks may discourage rapid evolution. A good example is how TOGAF maintains the core architecture style as application centric and treats SOA as a style extension. This is really quite extraordinary because by now everyone knows and many accept the digital business is going to be inherently service oriented. In practice of course the TOGAF specifications are so extensive that making fundamental changes may be very difficult, but it demonstrates neatly how legacy capabilities become “part of the furniture”, not just in frameworks but also in delivered systems and services.

Which brings me right back to the question – do we really need another practice framework? 

For several years now I and my colleagues have been evolving and implementing a different transformation approach. Initially we focused on SOA. And as many will know, we were fundamentalists in this area and we published detailed and open meta models for the service architecture and delivery life cycle based on “everything is a Service”. This approach has been very successful, particularly when the service architecture conforms with a strong layered reference architecture and rigorous componentization of services and business capabilities. But because we always knew that there was no such thing as a green field, we worked to harvest knowledge from existing systems. And over time we made a virtue of this,  focusing on continuous modernization as a matter of principle. More recently we have further evolved the approach to embrace Agile and MDD, establishing an agile service factory and generating code from rigorously defined service specifications.
But we found many of our customers struggled to implement a strategic SOA because business change was implemented project by project. And sure enough, project specific services and SOA have become ubiquitous; you might say almost a contradiction in terms. To counter this we advise that the demand management process should be complemented with demand shaping that decomposes the customer solution based demand to discover requirements for services and other sharable components and then re-aggregates the raw component demand into related projects that coordinate the delivery of business solutions and strategic services. 
But even though this approach works well at a project and technical level, we frequently encounter difficulty in persuading business stakeholders to balance short term advantage with more strategic goals. And we recognize this is because business stakeholders habitually make investment decisions on flawed criteria, because the ROI model only looks at the solution in isolation, rather than looking at the strategic opportunities to implement better architecture, address portfolio rationalization and reduce technical debt. This prompted us to find ways of working more effectively with business stakeholders. To encourage them to understand and indeed influence the conceptual business model and to understand a richer underlying business model that reflects a more comprehensive cost benefit model. And this helps to bring LoB managers into the conversation on demand shaping – balancing immediate solution requirements with longer term goals. 

In effect what we did was to establish a service and solution delivery value chain that commences with the raw customer demand and finishes with the delivery and integration of some useful business capability, but crucially with a much improved balance between immediate solution needs and longer term strategic goals. And it’s this balance that many enterprises find extremely difficult to achieve.
The core problem is that disciplines are vertically integrated; set up to optimize the discipline at varying levels of abstraction. In contrast the value chain approach optimizes across disciplines in pursuit of overall value chain outcomes. And this is only achieved by value chain activities that encourage effective collaborations between multiple capabilities and disciplines.

In the beginning we (Everware-CBDI) had a framework – Service Architecture & Engineering (SAE). The name makes it clear this is a forward engineering approach, and whilst it was very strongly business driven, it would be fair to say that the business modeling components were less well worked than the architecture, design and delivery. And as described we have very naturally, as part of the process of supporting large enterprise initiatives, expanded the framework capabilities and embraced an inherently Agile approach.

The principles underlying the framework now include:
– business capability based modularity
– pervasive service architecture
– continuous modernization
– service evolution not (one time) delivery
– model driven architecture and development
– everything is inherently agile – iterative, evolving, and narrowly focused on specific business capability delivery.

So to answer my own question, we clearly need a new framework. But it’s a very different practice framework to the ones that we are are accustomed to.
In our natural evolutionary process we recognized that the original (SAE) framework was merely one component of a much broader body of knowledge and practices. The new framework is goal driven, not discipline driven and incorporates the entire value chain of capabilities. But the capabilities are not standalone, they are effectively services that are executed in a way that supports the overall business goals of the enterprise, domain or program. We refer to this as  Service Oriented Application Modernization (SOAM).

Interestingly this is not a monolithic framework. It’s vital to treat the framework as a set of capabilities with defined services and dependencies. Some might say, “eating our own dog food”. In this way we don’t make the same mistake as legacy frameworks such as TOGAF, that are very difficult to keep current.

Finally what happens to the existing discipline based frameworks and standards? Of course they can be used in conjunction with the SOAM framework. But we do need to be careful not to just inherit monolithic capabilities. Increasingly we find it necessary to do this very selectively in order to use capabilities that fit and support the value chain. Perhaps in time the various disciplines will recognize the monolithic nature of their practices, and decompose and modernize into more goal oriented components. Meanwhile, in SOAM we have demonstrated that it is possible to reinvent the wheel.

Some SOAM Components:
    Agile Service Factory
    Agile Modernization
    Conceptual (Agile) Business Modeling
    SOA Reference Framework

Ref 1: Beware the New Silos

Oh No! We need another Practice Framework?

by

A few years ago I commented [in Beware the New Silos, ref 1 below] that in a complicated world we cope by specialization – and across the industry in general and in individual enterprises specifically we have created silos of our primary disciplines. The widely used frameworks and methods illustrate that common practice of discipline centricity. We don’t have to look too far for examples such as Enterprise Architecture (TOGAF), Governance (Cobit), SOA (separately by OASIS, OMG, Open Group), Agile methods (many and various), Business Process Management (BPM) and IT Service Management (ITIL). All of these disciplines, whether de jure or de facto standards, are all very narrowly focused with minimal treatment of how the disciplines might work together.

A good example is how Agile methods are tightly focused on application development and architecture is assumed to be an integral part of the Agile delivery project. Yet the reality in all enterprises is that significant aspects of architecture must be consistent at the domain or enterprise level. Another good example is how the three standards bodies OASIS, OMG and the Open Group were so divergent in their treatment of SOA, they commissioned a report [ref 2 below] to explain how these inherently duplicating standards and specifications relate to each other.

The level of adoption by enterprises or service providers of all these and similar practice frameworks and standards is of course highly variable. However it must be said that the very existence of the discipline based materials will frequently have some considerable influence on how enterprises organize themselves.

The thinking IT or business professional might also like to question just how up to date these frameworks are, and how they support today’s business goals, which for most of us have changed dramatically over the past few years. We might also speculate whether the education and certification ecosystems that feed off some discipline based frameworks may discourage rapid evolution. A good example is how TOGAF maintains the core architecture style as application centric and treats SOA as a style extension. This is really quite extraordinary because by now everyone knows and many accept the digital business is going to be inherently service oriented. In practice of course the TOGAF specifications are so extensive that making fundamental changes may be very difficult, but it demonstrates neatly how legacy capabilities become “part of the furniture”, not just in frameworks but also in delivered systems and services.

Which brings me right back to the question – do we really need another practice framework? 

For several years now I and my colleagues have been evolving and implementing a different transformation approach. Initially we focused on SOA. And as many will know, we were fundamentalists in this area and we published detailed and open meta models for the service architecture and delivery life cycle based on “everything is a Service”. This approach has been very successful, particularly when the service architecture conforms with a strong layered reference architecture and rigorous componentization of services and business capabilities. But because we always knew that there was no such thing as a green field, we worked to harvest knowledge from existing systems. And over time we made a virtue of this,  focusing on continuous modernization as a matter of principle. More recently we have further evolved the approach to embrace Agile and MDD, establishing an agile service factory and generating code from rigorously defined service specifications.
But we found many of our customers struggled to implement a strategic SOA because business change was implemented project by project. And sure enough, project specific services and SOA have become ubiquitous; you might say almost a contradiction in terms. To counter this we advise that the demand management process should be complemented with demand shaping that decomposes the customer solution based demand to discover requirements for services and other sharable components and then re-aggregates the raw component demand into related projects that coordinate the delivery of business solutions and strategic services. 
But even though this approach works well at a project and technical level, we frequently encounter difficulty in persuading business stakeholders to balance short term advantage with more strategic goals. And we recognize this is because business stakeholders habitually make investment decisions on flawed criteria, because the ROI model only looks at the solution in isolation, rather than looking at the strategic opportunities to implement better architecture, address portfolio rationalization and reduce technical debt. This prompted us to find ways of working more effectively with business stakeholders. To encourage them to understand and indeed influence the conceptual business model and to understand a richer underlying business model that reflects a more comprehensive cost benefit model. And this helps to bring LoB managers into the conversation on demand shaping – balancing immediate solution requirements with longer term goals. 

In effect what we did was to establish a service and solution delivery value chain that commences with the raw customer demand and finishes with the delivery and integration of some useful business capability, but crucially with a much improved balance between immediate solution needs and longer term strategic goals. And it’s this balance that many enterprises find extremely difficult to achieve.
The core problem is that disciplines are vertically integrated; set up to optimize the discipline at varying levels of abstraction. In contrast the value chain approach optimizes across disciplines in pursuit of overall value chain outcomes. And this is only achieved by value chain activities that encourage effective collaborations between multiple capabilities and disciplines.

In the beginning we (Everware-CBDI) had a framework – Service Architecture & Engineering (SAE). The name makes it clear this is a forward engineering approach, and whilst it was very strongly business driven, it would be fair to say that the business modeling components were less well worked than the architecture, design and delivery. And as described we have very naturally, as part of the process of supporting large enterprise initiatives, expanded the framework capabilities and embraced an inherently Agile approach.

The principles underlying the framework now include:
– business capability based modularity
– pervasive service architecture
– continuous modernization
– service evolution not (one time) delivery
– model driven architecture and development
– everything is inherently agile – iterative, evolving, and narrowly focused on specific business capability delivery.

So to answer my own question, we clearly need a new framework. But it’s a very different practice framework to the ones that we are are accustomed to.
In our natural evolutionary process we recognized that the original (SAE) framework was merely one component of a much broader body of knowledge and practices. The new framework is goal driven, not discipline driven and incorporates the entire value chain of capabilities. But the capabilities are not standalone, they are effectively services that are executed in a way that supports the overall business goals of the enterprise, domain or program. We refer to this as  Service Oriented Application Modernization (SOAM).

Interestingly this is not a monolithic framework. It’s vital to treat the framework as a set of capabilities with defined services and dependencies. Some might say, “eating our own dog food”. In this way we don’t make the same mistake as legacy frameworks such as TOGAF, that are very difficult to keep current.

Finally what happens to the existing discipline based frameworks and standards? Of course they can be used in conjunction with the SOAM framework. But we do need to be careful not to just inherit monolithic capabilities. Increasingly we find it necessary to do this very selectively in order to use capabilities that fit and support the value chain. Perhaps in time the various disciplines will recognize the monolithic nature of their practices, and decompose and modernize into more goal oriented components. Meanwhile, in SOAM we have demonstrated that it is possible to reinvent the wheel.

Some SOAM Components:
    Agile Service Factory
    Agile Modernization
    Conceptual (Agile) Business Modeling
    SOA Reference Framework

Ref 1: Beware the New Silos

Agile is not Dead, it’s Morphing

by

I note healthy discussion around whether Agile is Dead [ref 1]. And while I may sympathize (sic) with many of the comments, particularly the commercial trivialization of education, the core issue must surely be the difficulty of adopting de facto Agile practices to support real world enterprise programs and projects. My experience is most of the advice and guidance out there is predicated on scaling the de facto Agile development methods. And this isn’t the best place to start.

An exception is Dean Leffingwell’s SAFe, [ref 2] which does introduce the idea of portfolio, program and project perspectives and intentional architecture. I recommend this framework as an intelligent set of practices, but for me it doesn’t go far enough because it is still primarily about development practices. This is the core problem – that Agile is development specific and practices only. In the enterprise, Agile development needs to be an integral part of a bigger ecosystem spanning business design, architecture, requirements, modernization and operational transformation practices plus architecture, delivery and modernization disciplines.

I am indebted to Dave Thomas whose recent blog, [ref 1] includes a worthy successor to the Agile Manifesto, reducing the original, development specific values and principles to a minimalist, more generic set. He says; “here is how to do something in an agile fashion”
– Find out where you are
– Take a small step towards your goal
– Adjust your understanding based on what you learned
– Repeat
And this is more useful in the enterprise context because it is relevant to a broader set of activities than purely software development.

The diagram below is an outline maturity model template for Agile in the enterprise. It suggests there are four key views that need to be part of the transformation.  In addition to agile practices we need to be equally focused on what elements of agile architecture are required for an enterprise. What the agile delivery framework is and how the existing application portfolio will be modernized to progressively eliminate the duplication and complexity present in every enterprise on the planet.

People & Process. Much of the dissatisfaction with Agile arises from the limitations of the basic Agile practices, and the need to compromise these in an enterprise context. Both DAD and RUP (yes it’s an iterative method) are examples of extended or hybrid practices that introduce coordination, phasing and other disciplines that are more acceptable in enterprises that require traceability, governance and compliance with pre-existing life cycle practices. Enterprise frameworks such as Leffingwell’s SAFe as discussed and Everware-CBDI’s SOAM are examples of frameworks that adhere more closely to the purity of Agile principles while addressing enterprise specific needs. SAFe provides a framework which is more strongly Lean, coordinating portfolio, program and project activity to meet agile release train demand. SOAM provides a complementary, full life cycle process framework for software service modernization and delivery.

Agile Architecture. There is a requirement to articulate the enterprise requirements for agility as a reference architecture for business agility. In today’s fast moving world core architecture for the business, services, implementations, technology and deployments needs to be:
– under continuous development using Agile principles
– derived from the assessment of business needs for response to change, and constantly updated to reflect competitive and technology opportunities and threats.
– mapped to service architectures, patterns, policies and modernization strategies
– modeled using MDA/MDD to allow delivery as consistent architecture runways for portfolio and demand management, programs and projects.

Agile Delivery Framework. Most enterprises have a well-defined delivery framework of tools, repositories, templates etc that are designed to support well established QA and delivery policies. This is one of the most common inhibitors that Agile projects in the enterprise have to  overcome. In an enterprise Agile context that framework must be realigned to provide maximum automation of life cycle management and governance so that key enterprise requirements for integrity can be met without loss of productivity. Similarly the development  activity must be structured so that developers can extend the architecture runway with business solution specific rules and behaviors in a managed fashion which preserves the integrity of the architecture. Everware-CBDI has pioneered this runway extension capability implemented as a model driven (MDA/MDD) capability in which the runway code is generated and provided to developers enabling very significant productivity gains in both forward engineering and even more so in iteration.

Agile Modernization.  Finally in an enterprise context the elephant in the room is the existing or legacy portfolio. Unless this elephant is addressed, the enterprise will continue to create more and more complexity, increase costs and reduce response times to change. What’s required is a discipline of continuous, Agile modernization. That means, using Dave Thomas’s minimalist manifesto [above] every portfolio item, program and project must include steps to find out the current situation and address minimum goals that reduce complexity and support a progressive modernization strategy. Without that, all enterprise Agile projects will remain narrow focus and simply add technical debt.

I suggest that while Agile is not dead in the enterprise it is certainly struggling to survive. This is because Agile practices alone will be suffocated at birth by enterprise realities of consistency and integrity; or turned into narrow focus, standalone projects; or morphed into BAU. I really don’t want to enter into a debate about nouns or verbs, life is too short.  IMO Agile has considerable momentum and it can be morphed into a ground breaking concept that delivers enterprise business agility.

References
1 Agile is Dead (Long Live Agility)
2 Dean Leffingwell SAFe

Agile is not Dead, it’s Morphing

by

I note healthy discussion around whether Agile is Dead [ref 1]. And while I may sympathize (sic) with many of the comments, particularly the commercial trivialization of education, the core issue must surely be the difficulty of adopting de facto Agile practices to support real world enterprise programs and projects. My experience is most of the advice and guidance out there is predicated on scaling the de facto Agile development methods. And this isn’t the best place to start.

An exception is Dean Leffingwell’s SAFe, [ref 2] which does introduce the idea of portfolio, program and project perspectives and intentional architecture. I recommend this framework as an intelligent set of practices, but for me it doesn’t go far enough because it is still primarily about development practices. This is the core problem – that Agile is development specific and practices only. In the enterprise, Agile development needs to be an integral part of a bigger ecosystem spanning business design, architecture, requirements, modernization and operational transformation practices plus architecture, delivery and modernization disciplines.

I am indebted to Dave Thomas whose recent blog, [ref 1] includes a worthy successor to the Agile Manifesto, reducing the original, development specific values and principles to a minimalist, more generic set. He says; “here is how to do something in an agile fashion”
– Find out where you are
– Take a small step towards your goal
– Adjust your understanding based on what you learned
– Repeat
And this is more useful in the enterprise context because it is relevant to a broader set of activities than purely software development.

The diagram below is an outline maturity model template for Agile in the enterprise. It suggests there are four key views that need to be part of the transformation.  In addition to agile practices we need to be equally focused on what elements of agile architecture are required for an enterprise. What the agile delivery framework is and how the existing application portfolio will be modernized to progressively eliminate the duplication and complexity present in every enterprise on the planet.

People & Process. Much of the dissatisfaction with Agile arises from the limitations of the basic Agile practices, and the need to compromise these in an enterprise context. Both DAD and RUP (yes it’s an iterative method) are examples of extended or hybrid practices that introduce coordination, phasing and other disciplines that are more acceptable in enterprises that require traceability, governance and compliance with pre-existing life cycle practices. Enterprise frameworks such as Leffingwell’s SAFe as discussed and Everware-CBDI’s SOAM are examples of frameworks that adhere more closely to the purity of Agile principles while addressing enterprise specific needs. SAFe provides a framework which is more strongly Lean, coordinating portfolio, program and project activity to meet agile release train demand. SOAM provides a complementary, full life cycle process framework for software service modernization and delivery.

Agile Architecture. There is a requirement to articulate the enterprise requirements for agility as a reference architecture for business agility. In today’s fast moving world core architecture for the business, services, implementations, technology and deployments needs to be:
– under continuous development using Agile principles
– derived from the assessment of business needs for response to change, and constantly updated to reflect competitive and technology opportunities and threats.
– mapped to service architectures, patterns, policies and modernization strategies
– modeled using MDA/MDD to allow delivery as consistent architecture runways for portfolio and demand management, programs and projects.

Agile Delivery Framework. Most enterprises have a well-defined delivery framework of tools, repositories, templates etc that are designed to support well established QA and delivery policies. This is one of the most common inhibitors that Agile projects in the enterprise have to  overcome. In an enterprise Agile context that framework must be realigned to provide maximum automation of life cycle management and governance so that key enterprise requirements for integrity can be met without loss of productivity. Similarly the development  activity must be structured so that developers can extend the architecture runway with business solution specific rules and behaviors in a managed fashion which preserves the integrity of the architecture. Everware-CBDI has pioneered this runway extension capability implemented as a model driven (MDA/MDD) capability in which the runway code is generated and provided to developers enabling very significant productivity gains in both forward engineering and even more so in iteration.

Agile Modernization.  Finally in an enterprise context the elephant in the room is the existing or legacy portfolio. Unless this elephant is addressed, the enterprise will continue to create more and more complexity, increase costs and reduce response times to change. What’s required is a discipline of continuous, Agile modernization. That means, using Dave Thomas’s minimalist manifesto [above] every portfolio item, program and project must include steps to find out the current situation and address minimum goals that reduce complexity and support a progressive modernization strategy. Without that, all enterprise Agile projects will remain narrow focus and simply add technical debt.

I suggest that while Agile is not dead in the enterprise it is certainly struggling to survive. This is because Agile practices alone will be suffocated at birth by enterprise realities of consistency and integrity; or turned into narrow focus, standalone projects; or morphed into BAU. I really don’t want to enter into a debate about nouns or verbs, life is too short.  IMO Agile has considerable momentum and it can be morphed into a ground breaking concept that delivers enterprise business agility.

References
1 Agile is Dead (Long Live Agility)
2 Dean Leffingwell SAFe

Architecture for the Digital Business

by

There’s another metaphorical asteroid approaching the world. The last one (the Web) hit around 1991 and we are still struggling to come to terms with it. But the next one is an order of magnitude bigger – it’s often referred to as Digital Business. It’s not a bad name, but it doesn’t immediately signal some of the awesome impacts that will result. Essentially Digital Business is about Web enabling everything we do. Don’t be fooled into thinking this is about Google Glass, or the Smart Watch. It’s about Web enabling “everything”, from vehicles, power and water meters, appliances, houses, offices, driverless cars to traffic management systems, personal healthcare and anything you can think about. All these nodes become devices using sensors to communicate on a peer to peer basis via Machine to Machine (M2M) networks, and generate vast amounts of data that will revolutionize the way the world works! Like the Web, adoption will be slow at first, but the initial stages are already happening and we can expect it will be another 15 to 20 year change cycle which follows an exponential growth curve.

Don’t believe me? A really good example I just happened to see as I wrote this blog, is the smart bike lock from Lock8. The lock is paired with a mobile app and, lo and behold it’s paired with a peer-to-peer marketplace that a) alerts the owner if someone tries to steal the bike and b) tracks it if the thief manages to take it away, and c) even allows owners to loan or rent their bikes to friends or others registered on the service, and make a little cash on the side. Of course Lock8 also works in lots of other use cases. And just to reinforce the message, Lock8 have been hugely successful in raising significant funding through crowdsourcing!

It’s interesting to observe the media debate about Google Glass, which is strongly focused on privacy issues. Yet there is little or no debate yet about the Web enablement of almost everything, which has far more implications, not just for privacy, but also security, reliability and risk, as well as impacts on employment and changes in long established cost assumptions. It’s likely that Digital Business will trigger huge cultural changes. And that’s before you get to talk about humanoid robots to do our housework, look after children and older citizens. And yes they are coming also.

So what’s happening to the archetypal enterprise in this process of change. And what does enterprise architecture look in this changing environment? Of course technology is the primary stimulus, but we need to look at the social and economic effects of new technology enabled capabilities to understand how the enterprise may respond.

By coincidence I noted the Drucker Form met last month in Vienna and one of their major topics was Complexity. And there was fascinating debate about what complexity is, and whether it is increasing over time. And surely the Digital Business with the expected exponential growth in nodes, information and dependencies is going to be a major driver of complexity increase.

Roger Martin, of the University of Toronto, told the forum that one reason we feel overwhelmed by complexity . . . is because of a growing tendency to see and study problems within silos. The complexity of our world has actually not increased. It is just the unaddressed “inter-domain complexity” which makes us feel like everything has become more complex.

Don Tapscott, (of New Paradigm) was clearly responding to this issue of inter-domain complexity when he told the  conference – first, our institutions need to radically decentralize. Further, the future is not to be predicted, but to be achieved through a new dynamic paradigm including self-organizing, emergent and sometimes resilient networks involving millions of stakeholders. These networks embrace, active participation, uncertainty and constantly changing conditions, and they show great promise for solving global problems and governing an volatile and complex planet in the future. Command and control being replaced with self-organizing networks. Linear and non-linear organizations.

I particularly liked Tapscott’s analogy of murmurations of flocking starlings. Starlings somehow organizing themselves en masse to see off predators are at the opposite end of the (command and control) leadership spectrum from the arch bureaucrat. It reminds me also of the March of the Penguins. Did you ever see the huddle of thousands of penguins, their backs to the ice cold wind, continuously moving inwards and then outwards to allow all the group members to share the extreme exposure and protect the inner circle.

And of course this is what’s happening today. Self-organizing networks are forming everywhere, facilitated by the Web, and enabling collaboration and information sharing and transactions on a local and global scale without any central command or control.

So what does the Enterprise model of the future look like? In the diagram below I suggest the enterprise is a network of networks, a series of Capabilities that are supporting many Networks. The scope of the enterprise is the extent of interest in the Networks and Capabilities, not the boundaries of the enterprise as a legal entity. Participants are of course any form of Node, Device, Person, Entity that has a relationship with the Capability through some form of Experience and /or Channel. That Experience is provided by one or more Solutions, but we should plan that distributed Capabilities collaborate to share common assets on many levels. Note this is a very high level diagram, and you can safely assume that all the relationships are many to many. Further there is much more detail that is required particularly to make this a business AND IT model. But it is sufficient for my purpose in showing a core enterprise.

So what does the Enterprise Architecture look like in the fully distributed, self-organizing enterprise. The diagram below shows that the new EA needs to establish the basis for the federated business with architectures that are about managing the interconnections and dependencies, while allowing the distributed business to have as much freedom of action as is possible. The Reference Architecture , therefore defines the minimum necessary alignment. The Platform Architecture manages the essential business and IT interoperability and consistency that ensures the federation has minimum necessary cohesion. The Common Asset Architecture is then a portfolio of pluggable service components – that comply with the platform and facilitate sharing of common behaviors where appropriate, and enable solutions that have the necessary level of enterprise consistency together with required level of localization. Each of these architectures of course has the five views – Business, Service and Application, Implementation, Technology and Deployment. But this is a Federated Enterprise Architecture, that’s purpose is not to exert command and control over the leaf nodes, rather to facilitate the minimum necessary consistency to ensure the enterprise is a) in regulatory compliance and b) able to optimize cross enterprise dependencies, information and customer experience.

 In the Digital Business world Enterprise Architecture is vital, but it needs to be tightly focused on optimizing the distributed, federated business work, which means delegation. And here we have another cultural problem, because we have to embrace subsidiarity, the notion that all matters ought to be handled by the smallest, lowest or least centralized authority capable of addressing that matter effectively. And in general we are very bad at subsidiarity. Enterprises tend to prefer command and control as the default modus operandi.

The diagram below suggests that EA needs to undergo a significant transformation, moving from the As-Is in which the objective is frequently to seek the highest level of commonality to a To-Be model in which the aim is to organize the business as independent capabilities. Clearly this is much more than an architecture issue. It must be reflected in the way the business and IT are organized, transferring and delegating responsibility to leaf nodes, while establishing and governing the key policies that make the enterprise a coherent whole.

In my experience many enterprises have frequently responded to Web based business opportunities in a tactical, technology led manner. This has led to duplication of core business capabilities and, I observe, critical weaknesses in customer experience and lost opportunities. Many enterprises particularly telecos, publishers, video stores, music and media firms, have through competitive pressure come to recognize the strategic importance of the Web. In embracing the Digital Business, enterprises may be tempted to go down the tactical route, but this is likely to be a big mistake, as the network effect will become an integral part of the enterprise’s product and services.

It’s true that Asteroids don’t hit the earth very often, and when they do the impact is immediate and very dramatic. Digital Business isn’t going to be instantaneous, but the impact is likely to be colossal.  So the key message is – figure out your new reference model, get rid of command and control, but manage the cross network dependencies at all levels of the reference architecture – that’s how you deliver real agility in the new world.

References
Drucker Forum
Lock8
Links:
Blogpost: Agile Architecture
CBDI Journal Report: Capability Planning and Analysis

Architecture for the Digital Business

by

There’s another metaphorical asteroid approaching the world. The last one (the Web) hit around 1991 and we are still struggling to come to terms with it. But the next one is an order of magnitude bigger – it’s often referred to as Digital Business. It’s not a bad name, but it doesn’t immediately signal some of the awesome impacts that will result. Essentially Digital Business is about Web enabling everything we do. Don’t be fooled into thinking this is about Google Glass, or the Smart Watch. It’s about Web enabling “everything”, from vehicles, power and water meters, appliances, houses, offices, driverless cars to traffic management systems, personal healthcare and anything you can think about. All these nodes become devices using sensors to communicate on a peer to peer basis via Machine to Machine (M2M) networks, and generate vast amounts of data that will revolutionize the way the world works! Like the Web, adoption will be slow at first, but the initial stages are already happening and we can expect it will be another 15 to 20 year change cycle which follows an exponential growth curve.

Don’t believe me? A really good example I just happened to see as I wrote this blog, is the smart bike lock from Lock8. The lock is paired with a mobile app and, lo and behold it’s paired with a peer-to-peer marketplace that a) alerts the owner if someone tries to steal the bike and b) tracks it if the thief manages to take it away, and c) even allows owners to loan or rent their bikes to friends or others registered on the service, and make a little cash on the side. Of course Lock8 also works in lots of other use cases. And just to reinforce the message, Lock8 have been hugely successful in raising significant funding through crowdsourcing!

It’s interesting to observe the media debate about Google Glass, which is strongly focused on privacy issues. Yet there is little or no debate yet about the Web enablement of almost everything, which has far more implications, not just for privacy, but also security, reliability and risk, as well as impacts on employment and changes in long established cost assumptions. It’s likely that Digital Business will trigger huge cultural changes. And that’s before you get to talk about humanoid robots to do our housework, look after children and older citizens. And yes they are coming also.

So what’s happening to the archetypal enterprise in this process of change. And what does enterprise architecture look in this changing environment? Of course technology is the primary stimulus, but we need to look at the social and economic effects of new technology enabled capabilities to understand how the enterprise may respond.

By coincidence I noted the Drucker Form met last month in Vienna and one of their major topics was Complexity. And there was fascinating debate about what complexity is, and whether it is increasing over time. And surely the Digital Business with the expected exponential growth in nodes, information and dependencies is going to be a major driver of complexity increase.

Roger Martin, of the University of Toronto, told the forum that one reason we feel overwhelmed by complexity . . . is because of a growing tendency to see and study problems within silos. The complexity of our world has actually not increased. It is just the unaddressed “inter-domain complexity” which makes us feel like everything has become more complex.

Don Tapscott, (of New Paradigm) was clearly responding to this issue of inter-domain complexity when he told the  conference – first, our institutions need to radically decentralize. Further, the future is not to be predicted, but to be achieved through a new dynamic paradigm including self-organizing, emergent and sometimes resilient networks involving millions of stakeholders. These networks embrace, active participation, uncertainty and constantly changing conditions, and they show great promise for solving global problems and governing an volatile and complex planet in the future. Command and control being replaced with self-organizing networks. Linear and non-linear organizations.

I particularly liked Tapscott’s analogy of murmurations of flocking starlings. Starlings somehow organizing themselves en masse to see off predators are at the opposite end of the (command and control) leadership spectrum from the arch bureaucrat. It reminds me also of the March of the Penguins. Did you ever see the huddle of thousands of penguins, their backs to the ice cold wind, continuously moving inwards and then outwards to allow all the group members to share the extreme exposure and protect the inner circle.

And of course this is what’s happening today. Self-organizing networks are forming everywhere, facilitated by the Web, and enabling collaboration and information sharing and transactions on a local and global scale without any central command or control.

So what does the Enterprise model of the future look like? In the diagram below I suggest the enterprise is a network of networks, a series of Capabilities that are supporting many Networks. The scope of the enterprise is the extent of interest in the Networks and Capabilities, not the boundaries of the enterprise as a legal entity. Participants are of course any form of Node, Device, Person, Entity that has a relationship with the Capability through some form of Experience and /or Channel. That Experience is provided by one or more Solutions, but we should plan that distributed Capabilities collaborate to share common assets on many levels. Note this is a very high level diagram, and you can safely assume that all the relationships are many to many. Further there is much more detail that is required particularly to make this a business AND IT model. But it is sufficient for my purpose in showing a core enterprise.

So what does the Enterprise Architecture look like in the fully distributed, self-organizing enterprise. The diagram below shows that the new EA needs to establish the basis for the federated business with architectures that are about managing the interconnections and dependencies, while allowing the distributed business to have as much freedom of action as is possible. The Reference Architecture , therefore defines the minimum necessary alignment. The Platform Architecture manages the essential business and IT interoperability and consistency that ensures the federation has minimum necessary cohesion. The Common Asset Architecture is then a portfolio of pluggable service components – that comply with the platform and facilitate sharing of common behaviors where appropriate, and enable solutions that have the necessary level of enterprise consistency together with required level of localization. Each of these architectures of course has the five views – Business, Service and Application, Implementation, Technology and Deployment. But this is a Federated Enterprise Architecture, that’s purpose is not to exert command and control over the leaf nodes, rather to facilitate the minimum necessary consistency to ensure the enterprise is a) in regulatory compliance and b) able to optimize cross enterprise dependencies, information and customer experience.

 In the Digital Business world Enterprise Architecture is vital, but it needs to be tightly focused on optimizing the distributed, federated business work, which means delegation. And here we have another cultural problem, because we have to embrace subsidiarity, the notion that all matters ought to be handled by the smallest, lowest or least centralized authority capable of addressing that matter effectively. And in general we are very bad at subsidiarity. Enterprises tend to prefer command and control as the default modus operandi.

The diagram below suggests that EA needs to undergo a significant transformation, moving from the As-Is in which the objective is frequently to seek the highest level of commonality to a To-Be model in which the aim is to organize the business as independent capabilities. Clearly this is much more than an architecture issue. It must be reflected in the way the business and IT are organized, transferring and delegating responsibility to leaf nodes, while establishing and governing the key policies that make the enterprise a coherent whole.

In my experience many enterprises have frequently responded to Web based business opportunities in a tactical, technology led manner. This has led to duplication of core business capabilities and, I observe, critical weaknesses in customer experience and lost opportunities. Many enterprises particularly telecos, publishers, video stores, music and media firms, have through competitive pressure come to recognize the strategic importance of the Web. In embracing the Digital Business, enterprises may be tempted to go down the tactical route, but this is likely to be a big mistake, as the network effect will become an integral part of the enterprise’s product and services.

It’s true that Asteroids don’t hit the earth very often, and when they do the impact is immediate and very dramatic. Digital Business isn’t going to be instantaneous, but the impact is likely to be colossal.  So the key message is – figure out your new reference model, get rid of command and control, but manage the cross network dependencies at all levels of the reference architecture – that’s how you deliver real agility in the new world.

References
Drucker Forum
Lock8
Links:
Blogpost: Agile Architecture
CBDI Journal Report: Capability Planning and Analysis

Systems Failure at RBS – Again

by

Subhead: Modernization is NOT Optional!

Once again customers of The Royal Bank of Scotland (RBS) and its subsidiaries Ulster, Natwest etc are in trouble! Unable to use their debit cards, or access their funds! And the new Chief Executive Ross McEwan admitted today that RBS has failed to invest properly in its systems for decades! RBS has given no explanation of the crash, except that it was not related to the high volumes of transactions on Cyber Monday.

Actually the root problem for RBS, and many other banks and other very large organizations, is NOT that they haven’t invested in their systems for decades. The problem is that they have allowed the systems to evolve without good architecture and governance. Banking systems in particular used to be leading edge. But over the years the core systems that actually run the bank, have become badly out of date. Instead of modernizing the core systems, many banks have responded to demand for new products and services by surrounding the old legacy systems with new bolt on systems. As a result the old systems are held together with sticky tape and sealing wax, and modified piecemeal to adjust to new requirements, and the new systems create more and more interfaces and dependencies (including duplicated, hard coded business rules), so that eventually the systems resemble a hair ball. And as you know, a hairball is a tightly packed cylinder of fur, which also includes all manner of foreign particles, which can be quite hazardous in humans and animals. In systems terms we would more usually refer to the resulting mess as a monolithic systems portfolio.

The root problem therefore is twofold:
1. The increasing complexity and risk inherent in making change.
2. Increasing horizon of any change.

And the issues become apparent either when extremely complex operational processes execute incorrectly, or when changes in process, software or environment  are made, often unavoidably without full knowledge of all the implications due to lack of experience, documentation or even code!

Although I have no doubt that in all the large banks there are fierce debates about the IT budget, strangely the solution to this problem isn’t about the amount of money that needs to be spent. Rather it’s about how the organization embraces a real modernization program, which is led by good architecture and managed with strong governance.

Sadly in the IT industry the term modernization has come to mean something quite specific – the re-platforming or technology upgrade of some application. Whereas modernization should really encompass a much broader remit including:
1. Establishing a lingua franca between business and IT so that modernization issues and implications are genuinely understood by both business and IT management.
2. Defining a business systems architecture that a) facilitates transformation with low risk and b) progressively develops an inherently agile architecture, that can evolve continuously.
3. A roadmap for progressively rationalizing the business systems portfolio to comply with the architecture.
4. Implementing an integrated business and IT organization that owns the business systems.
5. Implementing knowledge management around business systems that ensures integrity and consistency and ownership of processes, information and rules.
6. Implementing a continuous Agile modernization and ongoing evolution process in both business and IT.
7. Establishing coordination, governance and risk management that ensures architectural integrity at all stages of the life cycle.

It would be easy for business management to think that the sort of problems experienced by RBS and other banks as IT problems. The above list suggests this is nonsense. The only way to properly resolve the issues is for business management to accept co-responsibility for IT. And it’s no accident that the first point in the list above is to ensure that business management understand the issues they are dealing with, and can communicate effectively with the IT organization. This isn’t just attending an education class; it’s embracing a common language that allows critical decisions to be taken on a properly balanced understanding of business and IT assumptions, costs and risks.

Retailers, manufacturers, power companies, logistics companies, government departments etc etc should not be complacent or smug when they see the problems at the banks. In fact almost all very large organizations have these problems. And the message for all enterprises is Modernization is not Optional!

Links:
RBS Crash – Management Prefer Offshoring to Modernization?
CBDI Journal Reports on Modernization