Aggregated enterprise architecture wisdom
Business driven Agile development is achieved by Breaking tasks into small increments; Applying only the minimum of planning. Iterating over short time frames (timeboxes). Involving a cross functional team working in all Software Development LifeCycle functions. At the conclusion of … Continue reading →
As I’m about to complete my share of a longer engagement on using Lean principles to improve the processes at an online services firm, it occurred to me that the efforts we undertook to properly embed Architecture practices into their Scrum process were novel. I haven’t seen much written about how to do this in practice, and I imagine others may benefit from understanding the key connection points as well. Hence this post.
First off, let me be clear: Agile software development practices are not at all averse to software architecture. But let’s be clear about what I mean by software architecture. In an agile team, most decisions are left to the team itself. The team has a fairly short period of time to add a very narrow feature (described as a user story) to a working base of code and demonstrate that the story works. The notion of taking a couple of months and detailing out a document full of diagrams that explains the architecture of the system: pretty silly.
The value of software architecture is that key decisions are made about the core infrastructure of the system itself: where will generalizations lie? Will a layered paradigm be used, and if so, what are the responsibilities of each layer? What modules will exist in each layer and why will they be created? How will the responsibilities of the system be divided up among the layers and components? How will the modules be deployed at scale? How will information flow among the modules and between the system and those around it?
The way these questions are answered will indicate what the architecture of the system is. There are many choices here, and the “correctness” of any choice is a balance between competing demands: simplicity, security, cost, flexibility, availability, reliability, usability, correctness, and many more. (These are called “System Quality Attributes”). Balancing between the system quality attributes takes thought and careful planning.
So when does this happen in an agile process?
Let’s consider the architect’s thinking process a little. In fact, let’s break the software architecture process into layers, so that we can divide up the architectural responsibility a little. You have three layers of software architectural accountabilities. (Repeat: I’m talking about Software Architecture, not Enterprise Architecture. Please don’t be confused. Nothing in this post is specific to the practice of Enterprise Architecture). All this is illustrated in the diagram below. (Click on the diagram to get something a little more readable.
At the top, you have the Aligning processes of software architecture. These processes consider the higher levels of enterprise architecture (specifically the business and information architecture) to create To-Be Software Models of the entire (or relevant) software ecosystem. If you’ve ever seen a wall chart illustrating two dozen or more software systems with connectors illustrating things like flow of data or dependencies, you’ve seen a model of the type I’m referring to. Creating and updating these kinds of diagrams is a quarterly or semi-annual process and reflects the gradual changes in the strategy of the enterprise.
In the middle, you have the Balancing processes of software architecture. These processes consider the needs of a single system but only from the level of deciding why the software will be divided up into modules, layers, and components, how that division of responsibility will occur, and what the resulting system will look like when deployed in specific technologies in a specific environment. All of this can be conveyed in a fairly short document that is rich in diagrams with a small amount of text explaining the choices. This occurs once when a system is moving forward, and the software architecture can be developed alongside the first couple of sprints as input to the third and subsequent sprints.
At the bottom, you have the Realization processes of software architecture. This is where the architecture becomes software, and this is where decisions are made about the choice of specific design patterns, the appropriate level of configurability vs. simplicity, and the ability to demonstrate whether the actual intent of the architecture occurs in practice. In agile, this layer occurs within the team itself. The software architect can offer advice about what patterns to use, but it is up to the team to realize that advice and/or decide not to implement it. The team will very likely implement the software architecture as described, but may choose to improve upon it.
What does the process look like
There are many visualizations of scrum running around. Some are described in books, others in papers or blog posts. Most share some common elements. There is a product backlog that, through the magic of sprint planning, the team extracts a subset for the sprint. This becomes the sprint backlog. The illustrations then tend to show the various rhythms of Sprint as cycles (sprint cycles and daily cycles), ending with a demonstration and retrospective.
In order to illustrate and train a team on all elements, including the business analysis elements, we chose to be a bit more explicit about the steps PRIOR to sprint planning, including the processes of creating and improving the stories prior to the start of a sprint. (as above, click on the image to enlarge it).
Astute observers will notice that we added a step that we are calling “pre-sprint story review.” This is a meeting that occurs one week prior to the start of a sprint. It is called by the product owner and he or she invites “senior chickens” (architects, user experience leads, development and test leads, and any other “non-team” members that want a say in the sprint.
In that week prior to sprint planning, those folks, working with the product owner, can improve the stories, add constraints, refine the description and acceptance criteria. And here’s where the architects get to play. Architects fulfilling the role of “Balancing” in the model above will have (or can create) an outline document describing the architecture of the software system, and can “link” that document to the SPECIFIC STORIES that are impacted by that design.
(Note: assuming you are using a tool like Microsoft’s Team Foundation Server, that fully enables Scrum in multiple forms, this is a nearly trivial activity since a document can be easily linked to any story. Enough advertising.)
So is an architect a chicken or a pig? Answer: depends on what “layer” the architecture is at. The top two layers are chickens. The third layer, realization, is done by the team itself. The person on the team may or may not have the title of “designer”. (I’d prefer that they did not, but that’s just because I believe that ALL team members should be trained to fulfill that role. In reality, the skill may not be wide spread among team members). Therefore, the third layer is done by the pigs.
I hope this provides some insight into how a team can embed software architecture into their scrum cycles. As always, I’m interested in any feedback you may wish to share.
There is a contradiction that shows up in many books and articles on agile software development. . And I’m going to rant a little on one of them: the “developer” bias in Agile software practices.
Before I begin my rant, I’d like to tell you where it comes from. I am an Enterprise Architect. I am also an agilest. I am a certified Scrummaster (for many years) and have been on many agile projects. I’ve seen success, and I’ve seen failure. I know that agile is good, but not good enough to overcome people who are determined to undermine change.
I am currently engaged on a project to help an organization rewire their business processes. All of their core processes. Some of them are in software development… and that is where agile comes in, but some are in other areas (sales, account management, customer service, etc). I see it all, not just the software engineering bits.
To make this work, we created a set of principles that we use to drive the conversation around transforming the company. We train people on the principles. We connect change to the principles. The principles are developed from the agile mindset but are not the same as the ones traditionally tied to the agile manifesto.
One is focused heavily on the notion of an empowered team. As the trainer, I’ve embedded these principles into my head. Kinda hard not to. So when I’m reading a book on agile practices, and the author says something that violates one of these principles, it stands out. Big time.
And hence the rant. I was reading Mike Cohn’s book on agile stories and I came across a passage that reflects a kind of mindset that does NOT build empowered teams. And I went off. The rest of this post is my rant.
—
On one hand, there is a basic expectation that we should empower the team to solve their problems using their skills and training. On the other hand, there is a bias towards having developers involved in every part of a business process, from sales through requirements to project management and out to delivery and customer service. The net result: we are empowering developers to perform every role. No one else is empowered. They are, in fact, not trusted at all.
At some level, agile practices like XP and Scrum are written, and promoted, by software developers, so it is understandable that these authors believe that software developers are a necessary part of every process.
But they are not.
Organizations intentionally create specific roles and groups of people. Those roles are defined to take the most advantage of the intrinsic motivations of people when performing their duties. The things that motivate a person to become a project manager, for example, are rarely the same things that motivate a person to become a software engineer. Motivation matters. If you are not motivated to excellence, you will not perform with excellence.
While it is fine for an individual person to be able to perform well in a couple of roles, it is absurd to build a system or business process that expects all people in one role to be able to perform another role well. In specific, it is absurd to build processes where a technologist is required to develop requirements, or manage a project, or plan the delivery of systems. Those processes can be performed just fine with motivated people who are trained and experienced in those activities. However, to empower a team, people in these roles must be respected as well.
Agile software, that so values people over processes and tools, demonstrates an utter lack of respect for the individuals who choose to excel in non-technical roles by asserting the need for developers to be involved in every step of their process.
For example, in XP (and sometimes in Scrum), user stories are described as being developed by a “customer team” that includes developers. In the book “User Stories Applied,” Mike Cohn describes a process where a series of user roles are developed. From those roles, the stories will be drawn. The following is a quote from the book:
To identify user roles, the customer and as many of the developers as possible meet in a room with either a large table or a wall to which they can tape or pin cards. It’s always ideal to include the whole team for the user role modeling that initiates a project, but not necessary. As long as a reasonable representation of the developers is present along with the customer, you can have a successful session. — “User Stories Applied” by Mike Cohn
I’m an agilest, and this statement takes my breath away. Why? Because the SKILL of developing user roles is a good skill to have, but not a necessary skill for all developers to have. Yet the author (whom I respect) assumes that the activity cannot be performed successfully without developers in the room! At the same time, the text assumes that the task can be performed by developers exclusively (no one else is necessary at all).
The time that we spend training developers to be good at creating user stories is time that we are NOT spending training them on concurrent processing, idempotent messaging, and query optimization. We don’t train business analysts or engagement managers on C# design patterns, so why should we train developers on user role modeling and requirements analysis? It’s a total lack of understanding of the value of roles, and it is a bias that works against success.
Roles are not boundaries. In any team, there are roles. More importantly, as a member of a team, I have to trust my team members to fulfill their roles well. If they need help, I TRUST THEM TO ASK FOR HELP. When they do, I am happy to “cross roles” and provide help. When I provide help, I am doing so from the perspective of a less-skilled resource, and under the supervision of the accountable role. (e.g. if I “help” a tester, but don’t document a defect according to team standards, how much “help” am I actually providing?)
Roles are empowering. If my team member does not ask for help, I trust each one to perform his or her role with excellence. If I don’t trust them, how can they develop the excellence that the company expects? How can they be recognized and rewarded for excellence if our processes and methods show a lack of trust in their roles to perform well without a developer present?
Roles demonstrate respect. As an agilest, I value people over processes. That means I value the contribution of individual experts on my team to do their work with excellence, and I will show them respect for the work that they do, and the unique value that they contribute. I will do so by allowing them to do their work without the “supervision and assistance” of a developer.
If they need me, they will ask for me. Until then, I will get to doing what I do best. I have a job to do, after all… and it is mine to do well.
Agile software development solves a problem. The problem is that we are including waste in our processes, and that waste is causing software to be delivered late or with poor quality.
If we optimize our processes, we can reduce waste. If we include developers in every process, we are not optimizing our processes. We are, in fact, replacing one kind of waste with another.
As an architect involved in an agile implementation (my current gig), you can imagine how interested I was to see that there’s a new book on Agile Architecture, and perhaps how disappointed I was to see that it focused on SOA and Cloud. That’s not to put down SOA or the cloud. I’m a huge fan of both. But it wasn’t the area of agility that I was hoping that a book, with that title, would address. The misunderstanding was mine, not the authors. I haven’t read the book yet, but I’m sure I will.
That moment of misunderstanding crystallized a thought: how even a two word phrase like “agile architecture” had two completely different meanings. The opening scene of the movie “The Hobbit, An Unexpected Journey,” puts a rather humorous twist on this idea, when Gandalf introduces himself to Bilbo Baggins (who has apparently forgotten having met him as a boy).
Bilbo: Good Morning
Gandalf: What do you mean? Do you wish me a good morning, or mean that it is a good morning whether I want it or not;
Bilbo: <stunned silence>
Gandalf: Or that you feel good this morning; or that it is a morning to be good on?
Bilbo: All of them at once, I suppose.
Of course, in Enterprise Architecture, we have the same problem. Does Enterprise Architecture mean “the practice of using technology architecture at an enterprise-wide scale,” or does it mean “the practice of using architectural ideas to shape the enterprise itself?”
And after a bit of stunned silence, perhaps it means
“Creating an architecture to describe the externalities of an enterprise to set its context and improve the relationships it has with customers, partners, and suppliers?”
All of them at once, I suppose.
Having just re-watched the Hobbit movie on my morning flight, these bits connected up in my head.
I’m proud to be both an architect of agility (applying the principles of agility to the processes of a business so that the business achieves the ability to change its own processes in response to agile demands), as well as a person who can craft technology architecture that reflects the notion of agility itself (technology that can be set up to change rapidly in response to business events).
All of them at once, I suppose.
I explained to one of my clients recently that there is a perception of animosity between the Enterprise Architecture community and the Agile community. Both sides make assumptions about the other, often assumptions that are simply unfair. For example, many in the EA community think of “agile practices” as an opportunity to develop software without any architecture at all, while many in the agile software development community think of architecture as one of the “big design up front” guys who oppose their principles and practices. Of course, it is not difficult to find people who fit those unfair descriptions, but I’d like to point out how these two viewpoints are similar.
I believe that effective Enterprise Architecture must be approached from an agile standpoint.
First off, what does it mean to be agile? We can always look to the agile manifesto for some guidance, but more recent publications do a good job of filling in some of the details as well. I include a number of things in the notion of “being agile.” These are not just from the agile manifesto, but also Kanban and the Theory of Constraints, Systems Thinking, Six Sigma, Scrum, eXtreme Programming, and value stream analysis.
I follow the terminology of Sam Guckenheimer in calling this the “Agile Consensus.”
We have to recognize that the “agile consensus” is an approach, not a methodology. It is a way of thinking about dealing with problems. More importantly, it is a way for dealing with complex problems. The diagram below comes from Ken Schwaber (inventor of Scrum) who adapted it from Strategic Management and Organisational Dynamics, by Ralph D. Stacey.
When we look at the problems that software is being used to address, and if we look at the process of writing software itself, we have to recognize that both areas of problems typically fall into the category of “complex.” Not always. Some software is simply configured and configured simply. Some problems that software addresses are simple problems. However, most of the discussions around software development are fueled by people addressing complex problems because that is where most of the software development community works. It is the bread and butter of software development: solving complex problems in a complex way.
Enterprise architecture also deals with complex problems. EA models and information are also complex to build and manage. In this way, EA is very similar to software development. EA solves complex problems in a complex way.
In order for EA to be effective, it has to use the same mentality as agile software development.
When I speak with enterprise architects who are actually doing the job of EA, big themes quickly arise:
If this looks like the list above, that is intentional. I am trying to point out that Enterprise Architects use agile ideas, even if they often don’t use the term “agile” to convey the message.
Why use these techniques? They work for complex problems.
Can someone think of a more complex problem than helping to move an organization towards their goals?
EA addressed complex problems. Agile thinking helps them to do it.
One very powerful metaphor that has reverberated throughout the technical community, in the past few years, was the Agile Manifesto. Created by a group of folks who wanted to communicate the principles that drove their thinking, the Agile Manifesto has been a very useful tool for deciding if a particular practice is being done well. I think it may be time to build one for the Business Architecture space.
That said, I am by myself, sitting in my living room. I am in no position to speak for the community of business architects. So, this submission is a suggestion for content that could be useful when the conversation begins. It is my personal opinion about the principles of business architecture. I would hope to bring this material to a group of other BA practitioners, as my contribution, to develop a full consensus on business architecture manifesto.
First off, in order to develop principles for business architecture, we need to describe the problem that we are trying to solve.
Business architecture is a relatively new field that addresses an old problem. Most business people recognize the underlying truth: the structure and practices of your organization directly impacts your ability to deliver the intended value. Whether we are talking about a military service, a civilian government agency, a non-profit organization, or a for-profit business, the structures and processes that a leader chooses to employ will impact the results that the organization will produce. That includes both intended and unintended results. So the basic problem is this: how do we deliver on our mission while maintaining our values?
Business architecture gets to deal with a slice of that problem. As people, we need to organize around a common shared mission. We need to know what we want, and we need to go get it. Humans can be pretty haphazard. Business architecture does not address every issue. Business architecture attempts to answer this question: what is the optimal way to organize? Business architecture typically does NOT answer questions around the integration of corporate controls, or supporting activities like how to find staff to fill new roles. Business architecture is focused on the narrow slice of “how to organize.”
So why do we need business architecture to solve this problem? There are literally hundreds of good, well researched, books that offer useful insight for solving this problem. Why use a business architecture approach? Because BA brings a novel approach, one based on the rigorous application of conceptual traceability, process improvement, information science, and mathematics. While most of the business analysis methods prior to business architecture were founded, fundamentally, in social science, mechanical engineering, and even education, business architecture focuses on the newer sciences that have emerged in the computerized age.
Business architecture’s unique value proposition is to focus on answering the questions of business structural and organizational effectiveness in a way that is rigorous, quick, clear, consumable, and value-focused.
We are uncovering better ways of developing business insight by doing it and helping others do it. Through this work, we have come to value:
That is, while there is value in the items on the right, we value the items on the left more.
To break that down:
I’m always looking for insight and feedback from the community, so please feel free to add your comments.
Please note: if your comment is long, the software will sometimes have trouble. Write it in notepad or Word first, and then cut and paste into the comment edit window. Don’t be afraid to send it more than once. I will delete duplicates. If all else fails, e-mail your comment to me and I’ll put it in.
In the first part of the discussion, drawing upon the genesis of Agile and CMM, we were left wondering if the two are “The Misfits”, and most likely to disappoint the pacifists, or can they become soul-mates after all. The … Continue reading →
If you have begun reading this, you probably have drawn two correct conclusions already. One, that I am a fan of Norma Jeane Mortensen Baker, professionally recognized as Marilyn Monroe. And two, having worked with both, Agile and CMM, I … Continue reading →
The agile movement has traditionally focused on agile software development at the team level. Team-based approaches, such as Scrum, focus on delivering day-to-day customer value, but do not, per se, ensure a strategic approach to business agility at th…
My humble apologies to John Godfrey Saxe, whose poem I have modified to add a seventh man, and to make a point… ‘twas seven men of Indostan To learning much inclined, Who went to see the Elephant (Though all of them were blind),…
ShortcomingsThis post is really not about the shortcomings of NASA, it’s more about the inevitability poor, high cost deliverables when a) an organization looses its focus because of a constantly changing Vision and Mission; b) a development process fo…
In a recent post I briefly discussed the changes in roles and emphasis when a development or transformation effort changes from a waterfall (Big Bang) effort to a short cycle-agile effort. This post will discuss the topic in more detail in terms …