Agile

Agile and the Fairy Godmother

by

Once upon a time, a long, long time ago, well some 20 years actually, some clever folk figured out a way of structuring work that was quite revolutionary. So revolutionary in fact that most people didn’t understand it. Now a few folk in a parallel world worked hard over the years to make this method work, and they invented more ideas, frameworks and tools. And every day they are constantly improving their productivity and quality; and many of them also use Agile methods, as they are entirely complementary to the revolutionary method. The method – Design by Contract.

However the mainstream of the market seems to be fixated on Agile methods, as being the metaphorical silver bullet. Yet we all can’t help noticing that Agile Land is not such a happy place. There is continuing debate about the difficulties of making the methods work in the enterprise; the fragmentation of the original Agile principles and the outbreak of religious wars. And I am minded to comment, again, that the root problem with Agile methods is that they are one-dimensional – solely focused on people and process to the exclusion of all the other opportunities to create agile businesses.

At the heart of the conundrum is the need to focus on some different questions. Such as:
– How can we reduce the amount of work that has to be done?
– How can we structure the outcomes (deliverables) so they are inherently agile?
– How can we structure the work so that there is real traceability between the intrinsic business model and the delivered systems and services?
– How can we ensure that overall technical debt is always reducing faster than new functionality is being delivered?
– . . .
You get the idea.

So back to my parallel world. In Design by Contract the business problem, typically the Use Case, Services and Operations are attributed with Pre-conditions, Post-conditions and Invariants (rules that must remain constant). These artifacts provide us with functional and design level specifications that can be produced in an Agile, iterative manner, that
a) Deliver implementation independent service specifications (descriptions if you prefer)
b) And therefore also for publishing API specifications
c) Form the basis for structuring the code that is fully traceable to the business model
d) Create inherently agile systems and service structures
e) Create the structure (stubs) for Unit, Integration, Functional and Regression testing
(e.g all conditions and rules within a given test scope must be tested)
f) Depending on the technology employed to define the conditions and invariants, (rules engines, pseudo languages etc) both code and test cases can be produced automatically.

I was prompted to write this blog because I happened to read Rob Marvin’s useful blog on testing, and his very interesting ideas for improving test productivity to keep up with Agile projects. In his piece he talks about automation, but actually seems to miss the opportunity to auto-generate the test cases. Even more important, he seems to believe that the current state of Agile projects is his benchmark that he has to keep up with. I would comment that state of the art Design by Contract projects together with model driven frameworks are delivering order of magnitude greater productivity with exceptional quality, and this ought to be the where testers should set the bar.

Sometimes it seems to me that the Agile methods community is rather in the situation of hoping a Fairy Godmother will appear, wave a magic wand and all will be well. Everyone will use Scrum like they ought to; enterprises will waive their awkward little requirements for inter-project coordination and all will be well. But while the Agile community continue to ignore the broader scope of agile enablers this day won’t come.

If you are interested in an example of Design by Contract at work take a look at the Agile Service Factory.

The Inverted Swan

by

image

The old analogy is of a graceful Swan seemingly effortlessly gliding through the water, whilst out of sight its submerged legs are kicking furiously in unseen effort.

The ‘Inverted Swan’ is the antithesis of the traditional analogy. The swans legs are out of the water flailing and flapping ineffectively in the air, whilst underneath the water, who knows? where is the grace in the work?

I see the inverted Swan more than i’d like to. Lots of industry with little value produced. Often caused by:

  • Prizing effort instead of effectiveness
  • The need to be seen to be doing something, when inaction may be the perfect action
  • Personal enjoyment of the peculiar and personal joy of submersion in ‘flow’ to the exclusion of asking why?
  • ‘Leaders’ cultivating an environment of ‘activity anxiety’, primarily to reinforce their own ego.

There should always be room for grace.

There should always be room to progress from merely viable to loveable.

When do you see the inverted swan?

How could we make sure we see it less?

Microservices and the Internet of Things – First impressions

by
I must say I was sceptical when I first heard the term “microservices”. It sounded like yet another wash-rinse-repeat cycle of earlier incarnations of SOA. It appears I was wrong – this architectural pattern has some  interesting characteristics that, in my opinion, offer some real potential for event-driven, edge-processing systems (that are prevalent in the Internet of Things).
After watching Fred George’s video, I realised what he described was an event-driven, agent-based, systems’ model, rather than how many of us see SOA implementations today (often way-off the original notion of a SOA). At a conceptual level, the pattern describes a ‘Complex Adaptive’ system.  Essential principles of the architecture, however, appear teasingly elegant and simple. Few of these design principles are unique to microservices, but in combination, they make a compelling story:
Publish anything of interest – don’t wait to be asked, if your microservice thinks it has some information that might be of use to the microservices ecosystem, then publish-and-be-damned.



Amplify success & attenuate failure – microservices that publish useful information thrive, while those that left unsubscribed, wither-on-the-vine. Information subscribers determine value, and value adjusts over time/changing circumstances.



Adaptive ecosystem – versions of microservices are encouraged –may-the-best-service-win mentality introduces variety which leads to evolution.



Asynchronous & encapsulated – everything is as asynchronous as possible – microservices manage their own data independently and then share it in event messages over an asynchronous publish-subscribe bus.



Think events not entities – no grand BDUF data model, just a cloud of ever-changing event messages – more like Twitter than a DBMS. Events have a “use-by-date” that indicates the freshness of data.



Events are immutable – time-series snapshots, no updates allowed.


Designed for failure – microservices must expect problems and tell the world when they encounter one and send out “I’m alive” heart-beats.



Self-organizing & self-monitoring – a self-organizing System-of-systems’ that needs no orchestration. Health monitoring and other administration features are established through a class of microservices.



Disposable Code – microservices are very, very small (typically under 1000 lines of code). They can be developed in any language.



Ultra-rapid deployment – new microservices can be written and deployed with hours with a zero-test SDLC.

It struck me that many of these design principles could apply, in part, to a 2020 Smart Grid architecture I’m working on, and to the much boarder ‘Internet of Things’ecosystem.

The microservices pattern does seem to lend itself to the notion of highly autonomous, location-independent s/w agents that could reside at the centre, mid-point or edge of an environment. I can imagine that the fundamental simplicity of the model would help, rather than hinder, data privacy and protection by being able to include high-level system contexts, policies and protocols (e.g. encryption and redaction) applied to the event-streams. This pattern, of course, won’t be the ‘right-fit’ for all situations, but it does seem to offer interesting opportunities in:

  • Agility – very small disposable services are deployable within hours
  • Resilience – withstands service failures and supports service evolution
  • Robustness – it’s hard to break due to: simplicity, in-built failure handling and lack of centralized orchestration

It may be that the microservices pattern can only be applied to operational decision-support and behaviour profiling situations. But if that’s the case, I still see great potential in a world where many trillions of sensor-generated events will be published, consumed, filtered, aggregated, and correlated. I’m no longer a developer, but as an architect, I’m always on the look-out for patterns that could: either apply to future vendors’ products and services, or could act as a guide for in-house software development practice.

As always, I’d be keen to hear your views, examples and opinions about microservices and their potential application to the IoT. Have you come across examples of microservices pattern in an IoT context – deployed or in the labs?

I whole-heartily recommend setting aside an hour to watch the video of Fred George’s presentation on microservices:


131108 1110 Dune Fred George Recording on 2013-11-08 1106-Vimeo from Øredev Conference on Vimeo.

Post-post:
  • Another great post about microservices  – including downsides.
  • More here including “The 8 fallacies of distributed computing”.
Duke Energy are doing some interesting things in the Edge Processing space.

Here’s a video on microservices in the conext of IoT  (worth ignoring the references to Cloud/Azure):

http://www.microsoftvirtualacademy.com/training-courses/exploring-microservices-in-docker-and-microsoft-azure

I’d like to talk to anyone who’s impelmenting/ thinking about a Staged Event Driven Architecture using microservices for Edge Processing.

Phil Wills on experience of deploying microservices at The Gaurdian

Microservices and the Internet of Things – First impressions

by
I must say I was sceptical when I first heard the term “microservices”. It sounded like yet another wash-rinse-repeat cycle of earlier incarnations of SOA. It appears I was wrong – this architectural pattern has some  interesting characteristics that, in my opinion, offer some real potential for event-driven, edge-processing systems (that are prevalent in the Internet of Things).
After watching Fred George’s video, I realised what he described was an event-driven, agent-based, systems’ model, rather than how many of us see SOA implementations today (often way-off the original notion of a SOA). At a conceptual level, the pattern describes a ‘Complex Adaptive’ system.  Essential principles of the architecture, however, appear teasingly elegant and simple. Few of these design principles are unique to microservices, but in combination, they make a compelling story:
Publish anything of interest – don’t wait to be asked, if your microservice thinks it has some information that might be of use to the microservices ecosystem, then publish-and-be-damned.



Amplify success & attenuate failure – microservices that publish useful information thrive, while those that left unsubscribed, wither-on-the-vine. Information subscribers determine value, and value adjusts over time/changing circumstances.



Adaptive ecosystem – versions of microservices are encouraged –may-the-best-service-win mentality introduces variety which leads to evolution.



Asynchronous & encapsulated – everything is as asynchronous as possible – microservices manage their own data independently and then share it in event messages over an asynchronous publish-subscribe bus.



Think events not entities – no grand BDUF data model, just a cloud of ever-changing event messages – more like Twitter than a DBMS. Events have a “use-by-date” that indicates the freshness of data.



Events are immutable – time-series snapshots, no updates allowed.


Designed for failure – microservices must expect problems and tell the world when they encounter one and send out “I’m alive” heart-beats.



Self-organizing & self-monitoring – a self-organizing System-of-systems’ that needs no orchestration. Health monitoring and other administration features are established through a class of microservices.



Disposable Code – microservices are very, very small (typically under 1000 lines of code). They can be developed in any language.



Ultra-rapid deployment – new microservices can be written and deployed with hours with a zero-test SDLC.

It struck me that many of these design principles could apply, in part, to a 2020 Smart Grid architecture I’m working on, and to the much boarder ‘Internet of Things’ecosystem.

The microservices pattern does seem to lend itself to the notion of highly autonomous, location-independent s/w agents that could reside at the centre, mid-point or edge of an environment. I can imagine that the fundamental simplicity of the model would help, rather than hinder, data privacy and protection by being able to include high-level system contexts, policies and protocols (e.g. encryption and redaction) applied to the event-streams. This pattern, of course, won’t be the ‘right-fit’ for all situations, but it does seem to offer interesting opportunities in:

  • Agility – very small disposable services are deployable within hours
  • Resilience – withstands service failures and supports service evolution
  • Robustness – it’s hard to break due to: simplicity, in-built failure handling and lack of centralized orchestration

It may be that the microservices pattern can only be applied to operational decision-support and behaviour profiling situations. But if that’s the case, I still see great potential in a world where many trillions of sensor-generated events will be published, consumed, filtered, aggregated, and correlated. I’m no longer a developer, but as an architect, I’m always on the look-out for patterns that could: either apply to future vendors’ products and services, or could act as a guide for in-house software development practice.

As always, I’d be keen to hear your views, examples and opinions about microservices and their potential application to the IoT. Have you come across examples of microservices pattern in an IoT context – deployed or in the labs?

I whole-heartily recommend setting aside an hour to watch the video of Fred George’s presentation on microservices:


131108 1110 Dune Fred George Recording on 2013-11-08 1106-Vimeo from Øredev Conference on Vimeo.

Post-post:
  • Another great post about microservices  – including downsides.
  • More here including “The 8 fallacies of distributed computing”.
Duke Energy are doing some interesting things in the Edge Processing space.

Here’s a video on microservices in the conext of IoT  (worth ignoring the references to Cloud/Azure):

http://www.microsoftvirtualacademy.com/training-courses/exploring-microservices-in-docker-and-microsoft-azure

I’d like to talk to anyone who’s impelmenting/ thinking about a Staged Event Driven Architecture using microservices for Edge Processing.

Phil Wills on experience of deploying microservices at The Gaurdian

Architect or Coach?

by
Is it just me, or are others finding the Enterprise Architect role shifting towards ‘Coach/Facilitator’? 
These days I find I’m most attracted to Tweets about workshop facilitation and business analysis techniques rather than anything discussing Enterprise Architecture: frameworks, methods and tools. 
Here’s a few links I’d recommend for those interested in the former.

Architect or Coach?

by
Is it just me, or are others finding the Enterprise Architect role shifting towards ‘Coach/Facilitator’? 
These days I find I’m most attracted to Tweets about workshop facilitation and business analysis techniques rather than anything discussing Enterprise Architecture: frameworks, methods and tools. 
Here’s a few links I’d recommend for those interested in the former.

Scaling Agile for the Enterprise

by

Guest post by Tim Mattix, Mario Gouvea and Vikram Purohit With the ever-evolving software development landscape, large enterprises are increasingly “going Agile.” Agile is applicable to many scenarios; for example, Extreme Programming (XP) zeroes in on software engineering while wrapping in novel approaches to boost quality, and Scrum is the most widely adopted agile method. While both of these frameworks work well for software development teams, Agile is even suitable for less obvious initiatives, such […]

Enterprise Agile: One Size Does Not Fit All

by

Guest post by Tim Mattix It’s no surprise that agile development software methods are quickly claiming ground over waterfall methods.  The linear waterfall model and its sequential design process run counter to the reality that customers often don’t know exactly what they want up-front; rather, they tend to fine-tune their requirements through ongoing two-way interactions during the life cycle of the project. On the other hand, the evolutionary development approach inherent in agile gives designers […]

Agile is not Dead, it’s Morphing

by

I note healthy discussion around whether Agile is Dead [ref 1]. And while I may sympathize (sic) with many of the comments, particularly the commercial trivialization of education, the core issue must surely be the difficulty of adopting de facto Agile practices to support real world enterprise programs and projects. My experience is most of the advice and guidance out there is predicated on scaling the de facto Agile development methods. And this isn’t the best place to start.

An exception is Dean Leffingwell’s SAFe, [ref 2] which does introduce the idea of portfolio, program and project perspectives and intentional architecture. I recommend this framework as an intelligent set of practices, but for me it doesn’t go far enough because it is still primarily about development practices. This is the core problem – that Agile is development specific and practices only. In the enterprise, Agile development needs to be an integral part of a bigger ecosystem spanning business design, architecture, requirements, modernization and operational transformation practices plus architecture, delivery and modernization disciplines.

I am indebted to Dave Thomas whose recent blog, [ref 1] includes a worthy successor to the Agile Manifesto, reducing the original, development specific values and principles to a minimalist, more generic set. He says; “here is how to do something in an agile fashion”
– Find out where you are
– Take a small step towards your goal
– Adjust your understanding based on what you learned
– Repeat
And this is more useful in the enterprise context because it is relevant to a broader set of activities than purely software development.

The diagram below is an outline maturity model template for Agile in the enterprise. It suggests there are four key views that need to be part of the transformation.  In addition to agile practices we need to be equally focused on what elements of agile architecture are required for an enterprise. What the agile delivery framework is and how the existing application portfolio will be modernized to progressively eliminate the duplication and complexity present in every enterprise on the planet.

People & Process. Much of the dissatisfaction with Agile arises from the limitations of the basic Agile practices, and the need to compromise these in an enterprise context. Both DAD and RUP (yes it’s an iterative method) are examples of extended or hybrid practices that introduce coordination, phasing and other disciplines that are more acceptable in enterprises that require traceability, governance and compliance with pre-existing life cycle practices. Enterprise frameworks such as Leffingwell’s SAFe as discussed and Everware-CBDI’s SOAM are examples of frameworks that adhere more closely to the purity of Agile principles while addressing enterprise specific needs. SAFe provides a framework which is more strongly Lean, coordinating portfolio, program and project activity to meet agile release train demand. SOAM provides a complementary, full life cycle process framework for software service modernization and delivery.

Agile Architecture. There is a requirement to articulate the enterprise requirements for agility as a reference architecture for business agility. In today’s fast moving world core architecture for the business, services, implementations, technology and deployments needs to be:
– under continuous development using Agile principles
– derived from the assessment of business needs for response to change, and constantly updated to reflect competitive and technology opportunities and threats.
– mapped to service architectures, patterns, policies and modernization strategies
– modeled using MDA/MDD to allow delivery as consistent architecture runways for portfolio and demand management, programs and projects.

Agile Delivery Framework. Most enterprises have a well-defined delivery framework of tools, repositories, templates etc that are designed to support well established QA and delivery policies. This is one of the most common inhibitors that Agile projects in the enterprise have to  overcome. In an enterprise Agile context that framework must be realigned to provide maximum automation of life cycle management and governance so that key enterprise requirements for integrity can be met without loss of productivity. Similarly the development  activity must be structured so that developers can extend the architecture runway with business solution specific rules and behaviors in a managed fashion which preserves the integrity of the architecture. Everware-CBDI has pioneered this runway extension capability implemented as a model driven (MDA/MDD) capability in which the runway code is generated and provided to developers enabling very significant productivity gains in both forward engineering and even more so in iteration.

Agile Modernization.  Finally in an enterprise context the elephant in the room is the existing or legacy portfolio. Unless this elephant is addressed, the enterprise will continue to create more and more complexity, increase costs and reduce response times to change. What’s required is a discipline of continuous, Agile modernization. That means, using Dave Thomas’s minimalist manifesto [above] every portfolio item, program and project must include steps to find out the current situation and address minimum goals that reduce complexity and support a progressive modernization strategy. Without that, all enterprise Agile projects will remain narrow focus and simply add technical debt.

I suggest that while Agile is not dead in the enterprise it is certainly struggling to survive. This is because Agile practices alone will be suffocated at birth by enterprise realities of consistency and integrity; or turned into narrow focus, standalone projects; or morphed into BAU. I really don’t want to enter into a debate about nouns or verbs, life is too short.  IMO Agile has considerable momentum and it can be morphed into a ground breaking concept that delivers enterprise business agility.

References
1 Agile is Dead (Long Live Agility)
2 Dean Leffingwell SAFe

Agile is not Dead, it’s Morphing

by

I note healthy discussion around whether Agile is Dead [ref 1]. And while I may sympathize (sic) with many of the comments, particularly the commercial trivialization of education, the core issue must surely be the difficulty of adopting de facto Agile practices to support real world enterprise programs and projects. My experience is most of the advice and guidance out there is predicated on scaling the de facto Agile development methods. And this isn’t the best place to start.

An exception is Dean Leffingwell’s SAFe, [ref 2] which does introduce the idea of portfolio, program and project perspectives and intentional architecture. I recommend this framework as an intelligent set of practices, but for me it doesn’t go far enough because it is still primarily about development practices. This is the core problem – that Agile is development specific and practices only. In the enterprise, Agile development needs to be an integral part of a bigger ecosystem spanning business design, architecture, requirements, modernization and operational transformation practices plus architecture, delivery and modernization disciplines.

I am indebted to Dave Thomas whose recent blog, [ref 1] includes a worthy successor to the Agile Manifesto, reducing the original, development specific values and principles to a minimalist, more generic set. He says; “here is how to do something in an agile fashion”
– Find out where you are
– Take a small step towards your goal
– Adjust your understanding based on what you learned
– Repeat
And this is more useful in the enterprise context because it is relevant to a broader set of activities than purely software development.

The diagram below is an outline maturity model template for Agile in the enterprise. It suggests there are four key views that need to be part of the transformation.  In addition to agile practices we need to be equally focused on what elements of agile architecture are required for an enterprise. What the agile delivery framework is and how the existing application portfolio will be modernized to progressively eliminate the duplication and complexity present in every enterprise on the planet.

People & Process. Much of the dissatisfaction with Agile arises from the limitations of the basic Agile practices, and the need to compromise these in an enterprise context. Both DAD and RUP (yes it’s an iterative method) are examples of extended or hybrid practices that introduce coordination, phasing and other disciplines that are more acceptable in enterprises that require traceability, governance and compliance with pre-existing life cycle practices. Enterprise frameworks such as Leffingwell’s SAFe as discussed and Everware-CBDI’s SOAM are examples of frameworks that adhere more closely to the purity of Agile principles while addressing enterprise specific needs. SAFe provides a framework which is more strongly Lean, coordinating portfolio, program and project activity to meet agile release train demand. SOAM provides a complementary, full life cycle process framework for software service modernization and delivery.

Agile Architecture. There is a requirement to articulate the enterprise requirements for agility as a reference architecture for business agility. In today’s fast moving world core architecture for the business, services, implementations, technology and deployments needs to be:
– under continuous development using Agile principles
– derived from the assessment of business needs for response to change, and constantly updated to reflect competitive and technology opportunities and threats.
– mapped to service architectures, patterns, policies and modernization strategies
– modeled using MDA/MDD to allow delivery as consistent architecture runways for portfolio and demand management, programs and projects.

Agile Delivery Framework. Most enterprises have a well-defined delivery framework of tools, repositories, templates etc that are designed to support well established QA and delivery policies. This is one of the most common inhibitors that Agile projects in the enterprise have to  overcome. In an enterprise Agile context that framework must be realigned to provide maximum automation of life cycle management and governance so that key enterprise requirements for integrity can be met without loss of productivity. Similarly the development  activity must be structured so that developers can extend the architecture runway with business solution specific rules and behaviors in a managed fashion which preserves the integrity of the architecture. Everware-CBDI has pioneered this runway extension capability implemented as a model driven (MDA/MDD) capability in which the runway code is generated and provided to developers enabling very significant productivity gains in both forward engineering and even more so in iteration.

Agile Modernization.  Finally in an enterprise context the elephant in the room is the existing or legacy portfolio. Unless this elephant is addressed, the enterprise will continue to create more and more complexity, increase costs and reduce response times to change. What’s required is a discipline of continuous, Agile modernization. That means, using Dave Thomas’s minimalist manifesto [above] every portfolio item, program and project must include steps to find out the current situation and address minimum goals that reduce complexity and support a progressive modernization strategy. Without that, all enterprise Agile projects will remain narrow focus and simply add technical debt.

I suggest that while Agile is not dead in the enterprise it is certainly struggling to survive. This is because Agile practices alone will be suffocated at birth by enterprise realities of consistency and integrity; or turned into narrow focus, standalone projects; or morphed into BAU. I really don’t want to enter into a debate about nouns or verbs, life is too short.  IMO Agile has considerable momentum and it can be morphed into a ground breaking concept that delivers enterprise business agility.

References
1 Agile is Dead (Long Live Agility)
2 Dean Leffingwell SAFe

Holiday: time to do some in-depth reading

by

I’m just back from a great, relax holiday. Apart from having good quality time with my family, holiday is also a great time to immerse myself in some books. Instead of the quick information gathering I normally do by scanning my twitter timeline, quick-reading blog posts, and some more in-depth articles from time to time, I really focused on a single book for a couple of hours. Reading books really.

The post Holiday: time to do some in-depth reading appeared first on The Enterprise Architect.