4 years, 4 months ago

Microservices or Monoliths – Fences and Neighbors

  At the end of my last post, “What Makes a Monolith Monolithic?”, I stated that I didn’t consider the term “monolithic” to be inherently derogatory. It is, rather, a descriptive term relating to the style of organizing an application’s architecture. Depending on the context the system operates within, a monolithic architectural style could lie […]

4 years, 8 months ago

Microservices, Monoliths, and Modularity

  There are very valid reasons for considering a microservice architecture (MSA) when building/evolving an application. In my opinion, however, forcing modularity isn’t one of those very valid reasons. Just the other day, I saw tweet from Simon Brown saying this same thing: I still like his comment from two years back: “I’ll keep saying […]

5 years, 1 month ago

Monolithic Applications and Enterprise Gravel

It’s been almost a year since I’ve written anything about microservices, and while a lot has been said on that subject, it’s one I still monitor to see what new pops up. The opening of a blog post that I read last week caught my attention: Coined by Melvin Conway in 1968, Conway’s Law states: […]

5 years, 6 months ago

Evolution Architecture

Caoilte O’Connor is a developer at ITV, the UK’s largest Commercial Terrestrial TV Network. Here’s a short  (~2 min) clip from Domain Service Aggregators: A Structured Approach to Microservice Composition . In this clip he describes how …

6 years, 5 days ago

Can you afford microservices?

Much has been written about the potential benefits of designing applications using microservices. A fair amount has also been written about the potential pitfalls. On this blog, there’s been a combination of both. As I noted in “Are Microservices the Next Big Thing?”: It’s not the technique itself that makes or breaks a design, it’s […]

6 years, 4 months ago

Microservices Architecture versus SOA

TechTarget has published another one of my “Ask the Expert” columns.  In this one, I offer up my thoughts on the differences between a Microservices Architecture and a SOA.  In a nutshell, I think the microservices trend has moved things in the right direction, a direction that many of the SOA pundits were espousing back […]

6 years, 5 months ago

Design Patterns for Event-driven Distributed Intelligence

Following discussions with @seabird20  and @darachennis, among others last year. I decided to publish a rough idea of the event-driven Distributed Intelligence architecture for Smart Grid (this could apply to the broader IoT). This is loosely-based on the microservices concepts & principles described here and builds on Duke Energy’s collaboration on interoperability and distributed intelligence initiative. The purpose of this post is to generate ideas and aid the ongoing dialogues. As far as I’m aware, the additional concepts I discuss here; SEDA, microservices and distributed agent patterns, are not  called out in the Duke Energy work (although the authors might reasonably claim they’re implied). My aim, however, is to make the ‘software and event data’ conceptual architecture much more explicit in this discussion.
Having said that,  the  first diagram looks pretty physical! It serves as a simple IoT-ish context: A metering sensor talks to an edge processor on a ‘Field Area Network’ and thereafter data is relayed back to the Data Centre for further processing.

In the 2nd diagram we open up the Edge Processor and to reveal a relatively simplistic view of the software architecture. This is based on the micoservices pattern described by Fred George, and my own experience as VP Product Development/Architect at VI Agents.

I found the concept of small, autonomous agents, work very well for us at VI. Moreover, I spotteda lot of parallels with Fred George’s description of microservices:

Publish anything of interest – don’t wait to be asked, if your microservice thinks it has some information that might be of use to the microservices ecosystem, then publish-and-be-damned.

Amplify success & attenuate failure – microservices that publish useful information thrive, while those that left unsubscribed, wither-on-the-vine. Information subscribers determine value, and value adjusts over time/changing circumstances.

Adaptive ecosystem – versions of microservices are encouraged –may-the-best-service-win mentality introduces variety which leads to evolution.

Asynchronous & encapsulated – everything is as asynchronous as possible – microservices manage their own data independently and then share it in event messages over an asynchronous publish-subscribe bus.

Think events not entities – no grand BDUF data model, just a cloud of ever-changing event messages – more like Twitter than a DBMS. Events have a “use-by-date” that indicates the freshness of data.

Events are immutable – time-series snapshots, no updates allowed.

Designed for failure – microservices must expect problems and tell the world when they encounter one and send out “I’m alive” heart-beats.

Self-organizing & self-monitoring – a self-organizing System-of-systems’ that needs no orchestration. Health monitoring and other administration features are established through a class of microservices.

Rapids, Rivers & Ponds
I also particularly liked Fred’s use of the labels Rapid and Rivers to describe two seperate instances of message brokers and Ponds to describe persisted data. Again very similar to the VI SixD architecture where we collected signals on a ‘Rapids’ bus and  business event & document messages flowed over a ‘Rivers’ bus, and Event Logs & Document Queues were our  ‘Ponds’. But that was back in 2002,  I do think the microservices pattern is much more elegant and more extensible.

Staged Event Driven Architecture
The 3rd diagram overlays what I see as the SEDA Stages in our Smart Grid architecture:

Stage1: Raw signal filtering, simple aggregation& correlation, negative-event detection etc.

Stage2: Goal and role based functionality via an ‘Edge Agent’ that does complex aggregation and correlation, sense-making and alerting. These might be implemented as microservices, or at least, share many of the attributes described by Fred.

Stage3: There will  be more aggregation & correlation, routing alerting and broadcasting done here. The junction of Information Technology (IT) and Operational Technology (OT)  is probably here. These are the the software components that sit back at the Data Centre, that receive and process the event messages produced by the ‘Edge Agents’. There’s also a nod to some sort of management console and the publishing of commands and requests to the ‘Edge Processing’ domain. This diagram is very sketchy right now, with loads of components missing.   I just want to keep things very simple at this stage. 

Stage4: This is where the  ‘Big Data’ heavy-lifting of historical data and predictive analytics stuff is done.

We used  the SEDA pattern to define an Events Network that was capable of processing 4bn events per day without having to invest in massive network bandwidth and huge back end processing capabilities for the Royal Mail – I believe the U.S. postal service didn’t implement a SEDA  and  throw a huge amount of cash at the problem. I love the elegance of a series of cascading queues that gradually whittle the tsunami of raw signals in real time, down to ‘right-time’ stream or tap of business-meaningful messages.

This is the early stage thinking and part of a much bigger Smart Grid, and ultimately Smart City architecture. Please feel free to tweet about. or better still, comment below. Thx.
6 years, 9 months ago

Microservices Unplugged

Given my (well-known and enduring) interest in all aspects of services, I have followed Martin Fowler’s writing on microservices. But I will admit I always found the original paper more confusing than insightful. And in my client work I have resisted the temptation to use a microservices pattern, for precisely the reason that it would more than likely confuse. So I was interested to see the book Building Microservices by Sam Newman published last month, particularly as Newman is part of the Thoughtworks stable, which presumably means it is authoritative.  

Right off the bat, Newman advises that we should “think of microservices as a specific approach for SOA in the same way that XP or Scrum are specific approaches for Agile Software development”. These analogies are very interesting because my expectation was that microservices is a pattern. So I might infer that microservices is a set of process techniques as opposed to an architectural approach. Yet in the book, Newman clearly includes some elements of concept model and architecture as well as process and organization.

Probably the biggest weakness of Newman’s book is the absence of a coherent reference model. He discusses a series of heuristics such as:

                – Microservices are small, autonomous services that work together.

                – With high cohesion.

                – Gather those things that change for the same reason, and separate those things that change for different reasons.

                – Using business boundaries.

                – How small is small? Could be written in 2 weeks; small enough and no smaller; can be managed by a small team

                – Separate entity. All communications between services are network calls. Decoupled. Minimum dependency.

                – We need to think about what our services expose, and what they should allow to be hidden. If there is too much sharing our consuming services become coupled to internal representations.

And these are mostly useful principles, but are most unlikely to guide a consistent approach. Given Martin Fowler’s involvement, I would have expected at least some good patterns. But mature SOA needs a defined concept model, like CBDI-SAE, SOA-RM etc.

This lack of clarity is fundamental. Many organizations treat operations as services, and they typically end up with thousands of services. They will be confused by the very term microservice because they already have what they perceive as small services. Perhaps the term microservice has been chosen because of the ubiquitous use of the ITIL service, which undoubtedly means monolith. Most mature SOA organizations define services and operations as a primary technique to establish cohesion guided by reference models such as SOA-RM and CBDI-SAE. Just as important is the clarity and precision around specifications or contracts and what is exposed to consumers (APIs) and providers. Yet the book contains nothing on the subject of rich service specification or contract as the primary means of achieving implementation independence and business alignment. Organizations adopting microservices will need a defined reference model. Without that there will be architecture governance by personal opinion. I wonder how Thoughtworks advise their clients in this area? 

The book contains some quite general advice on business capability based decomposition – which is a tried and tested method of defining services. However there is little or no advice on what a well-formed business capability is, it’s characteristics and governance considerations. In my experience this is an area for considerable confusion; the business capability concept is so well known, but little understood, it’s an accident waiting to happen. And as for the relationship between capability and service, there’s no model provided, and more importantly, the granularity of a business capability service is likely to be non-trivial.

Similarly there’s no guidance on types of service. Do all services contain a mix of channel, process, business and data behaviors? Which you might expect given the explicit advice to gather those things that change for the same reason, and separate those things that change for different reasons.

The one area that might be helpful is the concept that microservices are highly autonomous, and a capability cluster of services would be a unit of deployment. But sadly for Newman, this is an idea that I and my colleagues worked on and popularized nearly 20 years ago. We called it Component Based Development, and the concept was closely allied to Rich Service Specifications which enforced the component boundaries, a concept that seems to have escaped him.

Once upon a time I had expected there to be a microservice pattern. But having read the book, all I see is a set of established ideas being rehashed in a very superficial manner, under a new name; with many key concepts missing or misunderstood. Apparently there is a lack of consensus on how to do SOA well. That failure of SOA is caused by vendors’ products, issues with granularity or picking wrong places to split a system. These assessments of the problem illustrate the lack of understanding of the author and his colleagues. Yes, there have been SOA failures, but mostly they have arisen through a) a focus on integration technology not business, or b) lack of reference model, reference architecture and governance. As an aside, in mature SOA we don’t consider “places to split a system”; we define the scope of a service using well defined heuristics and techniques that vary by type of service. To claim therefore that microservices is the answer to problems with SOA is downright disingenuous. And given the amount of hype I see about the topic generated by supposedly responsible organizations such as Thoughtworks and Gartner, that’s downright irresponsible. Disappointing. 

Public Domain information on Mature SOA architecture and delivery practices