10 months, 9 days ago

How to Use Microservices: A Guide for Enterprise Architects

Many organizations have started to break up a portion of their monolith applications and systems, transitioning to sets of smaller, interconnected microservices.

A recent survey by TechRepublic, found that organizations who used microservices were reaping clear benefits: 69% were experiencing faster deployment of services, 61% had greater flexibility to respond to changing conditions, and 56% benefited more from rapidly scaling up new features into large applications.

5 years, 2 months ago

Microservices or Monoliths – Fences and Neighbors

  At the end of my last post, “What Makes a Monolith Monolithic?”, I stated that I didn’t consider the term “monolithic” to be inherently derogatory. It is, rather, a descriptive term relating to the style of organizing an application’s architecture. Depending on the context the system operates within, a monolithic architectural style could lie […]

5 years, 6 months ago

Microservices, Monoliths, and Modularity

  There are very valid reasons for considering a microservice architecture (MSA) when building/evolving an application. In my opinion, however, forcing modularity isn’t one of those very valid reasons. Just the other day, I saw tweet from Simon Brown saying this same thing: I still like his comment from two years back: “I’ll keep saying […]

5 years, 11 months ago

Monolithic Applications and Enterprise Gravel

It’s been almost a year since I’ve written anything about microservices, and while a lot has been said on that subject, it’s one I still monitor to see what new pops up. The opening of a blog post that I read last week caught my attention: Coined by Melvin Conway in 1968, Conway’s Law states: […]

6 years, 3 months ago

Evolution Architecture

Caoilte O’Connor is a developer at ITV, the UK’s largest Commercial Terrestrial TV Network. Here’s a short  (~2 min) clip from Domain Service Aggregators: A Structured Approach to Microservice Composition . In this clip he describes how …

6 years, 10 months ago

Can you afford microservices?

Much has been written about the potential benefits of designing applications using microservices. A fair amount has also been written about the potential pitfalls. On this blog, there’s been a combination of both. As I noted in “Are Microservices the Next Big Thing?”: It’s not the technique itself that makes or breaks a design, it’s […]

7 years, 3 months ago

Design Patterns for Event-driven Distributed Intelligence

Following discussions with @seabird20  and @darachennis, among others last year. I decided to publish a rough idea of the event-driven Distributed Intelligence architecture for Smart Grid (this could apply to the broader IoT). This is loosely-based on the microservices concepts & principles described here and builds on Duke Energy’s collaboration on interoperability and distributed intelligence initiative. The purpose of this post is to generate ideas and aid the ongoing dialogues. As far as I’m aware, the additional concepts I discuss here; SEDA, microservices and distributed agent patterns, are not  called out in the Duke Energy work (although the authors might reasonably claim they’re implied). My aim, however, is to make the ‘software and event data’ conceptual architecture much more explicit in this discussion.
Having said that,  the  first diagram looks pretty physical! It serves as a simple IoT-ish context: A metering sensor talks to an edge processor on a ‘Field Area Network’ and thereafter data is relayed back to the Data Centre for further processing.

In the 2nd diagram we open up the Edge Processor and to reveal a relatively simplistic view of the software architecture. This is based on the micoservices pattern described by Fred George, and my own experience as VP Product Development/Architect at VI Agents.

I found the concept of small, autonomous agents, work very well for us at VI. Moreover, I spotteda lot of parallels with Fred George’s description of microservices:

Publish anything of interest – don’t wait to be asked, if your microservice thinks it has some information that might be of use to the microservices ecosystem, then publish-and-be-damned.

Amplify success & attenuate failure – microservices that publish useful information thrive, while those that left unsubscribed, wither-on-the-vine. Information subscribers determine value, and value adjusts over time/changing circumstances.

Adaptive ecosystem – versions of microservices are encouraged –may-the-best-service-win mentality introduces variety which leads to evolution.

Asynchronous & encapsulated – everything is as asynchronous as possible – microservices manage their own data independently and then share it in event messages over an asynchronous publish-subscribe bus.

Think events not entities – no grand BDUF data model, just a cloud of ever-changing event messages – more like Twitter than a DBMS. Events have a “use-by-date” that indicates the freshness of data.

Events are immutable – time-series snapshots, no updates allowed.

Designed for failure – microservices must expect problems and tell the world when they encounter one and send out “I’m alive” heart-beats.

Self-organizing & self-monitoring – a self-organizing System-of-systems’ that needs no orchestration. Health monitoring and other administration features are established through a class of microservices.

Rapids, Rivers & Ponds
I also particularly liked Fred’s use of the labels Rapid and Rivers to describe two seperate instances of message brokers and Ponds to describe persisted data. Again very similar to the VI SixD architecture where we collected signals on a ‘Rapids’ bus and  business event & document messages flowed over a ‘Rivers’ bus, and Event Logs & Document Queues were our  ‘Ponds’. But that was back in 2002,  I do think the microservices pattern is much more elegant and more extensible.

Staged Event Driven Architecture
The 3rd diagram overlays what I see as the SEDA Stages in our Smart Grid architecture:

Stage1: Raw signal filtering, simple aggregation& correlation, negative-event detection etc.

Stage2: Goal and role based functionality via an ‘Edge Agent’ that does complex aggregation and correlation, sense-making and alerting. These might be implemented as microservices, or at least, share many of the attributes described by Fred.

Stage3: There will  be more aggregation & correlation, routing alerting and broadcasting done here. The junction of Information Technology (IT) and Operational Technology (OT)  is probably here. These are the the software components that sit back at the Data Centre, that receive and process the event messages produced by the ‘Edge Agents’. There’s also a nod to some sort of management console and the publishing of commands and requests to the ‘Edge Processing’ domain. This diagram is very sketchy right now, with loads of components missing.   I just want to keep things very simple at this stage. 

Stage4: This is where the  ‘Big Data’ heavy-lifting of historical data and predictive analytics stuff is done.

We used  the SEDA pattern to define an Events Network that was capable of processing 4bn events per day without having to invest in massive network bandwidth and huge back end processing capabilities for the Royal Mail – I believe the U.S. postal service didn’t implement a SEDA  and  throw a huge amount of cash at the problem. I love the elegance of a series of cascading queues that gradually whittle the tsunami of raw signals in real time, down to ‘right-time’ stream or tap of business-meaningful messages.

This is the early stage thinking and part of a much bigger Smart Grid, and ultimately Smart City architecture. Please feel free to tweet about. or better still, comment below. Thx.