9 years, 29 days ago

Idempotence and the Dead-Letter Office

Link: http://actionable.wordpress.com/2012/03/18/idempotence-and-the-dead-letter-office/

Dead Letter Office

“Dead Letter Office” is a lovely and hearty shiraz sourced from the McLaren Vale and Padthaway wine regions of South Australia.  While drinking a bottle of Dead Letter Office recently i started to sketch this post about asynchronous message-based integration architectures.  The dead-letter-office concept is an important form of exception-handling, though it contributes to complexity in delivering perfectly-reliable and predictable end-to-end integration services.

Asynchronous messaging has many benefits, not least the loose coupling it affords between message publishers and message subscribers.  Publishers are completely ignorant of their subscribers, and need only to honour the publication contract and behave well.  Between the publishers and the susbcribers usually are stateless mediators responsible for:

  1. message collection, dequeueing the message and validating its semantic correctness and that it is well-formed
  2. message filtering, discarding messages in which a particular subscriber is uninterested (e.g., for a subscriber that is interested only in messages relating to a particular site or a particular group of people or cannot process future-dated events)
  3. message transformation, changing the message to be appropriate in form and content for consumption by the subscriber (e.g., removing attributes the subscriber is not entitled to receive, masking attributes the subscriber is not allowed to receive in full fidelity, formatting or calculating attributes based upon other attributes in the message, or enriching the messages by making callouts to other systems)
  4. message delivery, delivering the message to the subscriber endpoint and negotiating the delivery outcome
  5. exception handling, responding appropriately by raising warnings or throwing errors when processing exceptions are encountered at any of the previous steps
  6. logging and auditing, to provide instance-tracking and to feed event-monitoring and alerting systems

If an exception occurs somewhere in that chain during message processing then the cause is either:

  • system exceptions, such as when a service is down or unreachable, or when the integration user cannot log into or has lost its permissions to perform the functions it requires in the target system — in general, messages encountering system errors will be delivered successfully on some subsequent attempt.


  • semantic exceptions, such as when a message schema is invalid or when the data contained in a message are invalid for the subscriber system (for referential-integrity, length, format, or other reasons) — in general, messages encountering semantic errors will never be delivered successfully until the data issues are corrected, and a human is often needed to make those corrections.

…and in either case there will be some exceptions that have been seen previously and some exceptions that are being encountered for the first time.  This boils down to a requirement for three message-queue buckets:

  • Dead Letter Office: there is something wrong with these messages, and they will never be delivered successfully as they are.  If there are referential-integrity issues with a subscriber system then the messages will generally need to be republished from the system of record.
  • Try Again Later: something environmental is wrong, so delivery of these messages should be attempted again, later.
  • Spooky Inspection: some messages cannot be delivered, but the reason is unknown, because it has never before been seen — these messages require inspection and analysis to categorise them either for the Dead Letter Office or for the Try Again Later queue-buckets.

Across the end-to-end message-processing implied by an exception causing an en-route message to be delivered into a queue-bucket exist basic quality-of-service requirements in the scope of the universe of messages for conditions to be honoured such as:

  • ordering
  • timeliness
  • security
  • transactions

…and achieving this implies that the message-delivery mediators must work with these exception queues in order to see, potentially, the:

  • Try Again Later queue-bucket emptied before taking fresh new messages from the publication-queue trunk
  • Spooky Inspection queue-bucket emptied before taking fresh new messages from the publication-queue trunk
  • Dead Letter Office queue-bucket either ignored forever (if the wider environment is sufficiently-well transactional) else constructed with micro-queues and feedback from the successfully-processed messages so that entities whose messages have been dead-messaged are released once the transactional safety of the item has been confirmed

These scenarios are complex.  A pragmatic and optimistic approach to dealing with them perhaps boils down to these two requirements of an message-based integration architecture:

  1. Idempotence is Mandatory: everything must be idempotent, so that the same message, or the same latest-state message, can be republished multiple times without harm.
  2. Staleness-Detection is Mandatory: subscribers must have some concept of stale messages, and if subscriber components do not have a capability to enforce a staleness concept then they must be prefaced by integration agents that can make staleness decisions on their behalf, effectively operating as a super-simple resequencer.

We’re working hard to get the data-integration scenarios working really well, while at the same time noting the industry drift toward higher-level integration solutions that operate at the application and the process level.  Getting the right patterns down and used (and reused) is the key to this.

Hohpe and Woolf’s Enterprise Integration Patterns is an indispensable guide in this regard, and it’s crucial those patterns are set down into reusable component-based recipes embedded into each organisation’s integration-architecture repositories and application-development frameworks.

For more details, refer to the Enterprise Integration Patterns website at www.enterpriseintegrationpatterns.com