If we look at the digital the way we look at enterprise architecture, we have to project the big picture at the end of the tunnel, the digital enterprise.
Organizations do not work, in real life, like they work on paper. On paper, there are departments (all shaped like a neat rectangle) and business processes with neat inflows and outflows of responsibility and information. On paper, you improve things by modeling things on paper, and then moving things around, on paper, then teaching people to follow the process that your neat paper diagrams represent.
In real life, there are human beings and the tools that they use. Sometimes the tools move information from one person to another. Sometimes, they just aid in communication. People meet and get to know other people, and they learn to trust some, and distrust others. Some folks have different measures and motivations and just “pass by” one another. Some subset of these people will have shared cultural values and expectations. There may be many cultures in an organization: both because the organization is in multiple places, and because people from multiple places join an organization. Also, “business culture” arises as leaders achieve successes and people learn to use certain “cultural expectations” to get things done efficiently.
Reality is a lot messier than pretty rectangles.
Enterprise Architects apply science and engineering and aesthetics to the challenge of organizational change. We are unique in that most other “change artists” are not focused on engineering and some even ignore science. (see Daniel Pink’s TED Talk on the Surprising Science of Motivation). Few even know how to spell aesthetics. Yet, when you are dealing with systems that contain and include people, you have to use aesthetics, and you are ill prepared for success if you ignore science. Engineering is a mindset as much as a class of methods. It involves applying the things that science has discovered and using that understanding to build great (and sometimes terrible) things. Engineers build on ideas and use them, often experimenting on systems that are too complex and intertwined for “pure science” to get arms around.
As Enterprise Architecture is such a young science, we have relied to heavily on the “boxes and lines” model of enterprises, and not enough on the messy but important sociocultural view of an enterprise. We find it easier to document, and model, and even simulate, processes as though people were interchangeable and their relationships didn’t matter.
That is just lazy.
It is time to get up off our collective butts and start working out ways to understand sociocultural influences, relationships, and architectures. We have to build ways to detect, measure, and consider these structures when we measure capabilities, or improve processes, or suggest automations, or evaluate business models, or any of the two dozen things that EA’s do.
The value of EA often comes to an executive in the form of a reasoned opinion that is based on data that no one else is looking at. Let’s consider the possibility that examining sociocultural influences can provide interesting opinions that an executive will find valuable.
We should consider sociocultural information if:
- Sociocultural influencers can impact the speed of change in an organization.
- Sociocultural connections can impact the decision making and governance processes
- Sociocultural strengths would allow rapid improvement in business capabilities needed for a shift in strategy
- Sociocultural blind spots would prevent an organization from recognizing an existential threat
Think about it. Do you believe that any of those statements are false? I can find ample examples for each one. So if sociocultural interactions matter, why are we not tracking them, learning from them, using them to make decisions?
It’s only hard because we haven’t tried.
(This post inspired by the many similar pleas shared by J.D. Beckingham in social media).
Organizations do not work, in real life, like they work on paper. On paper, there are departments (all shaped like a neat rectangle) and business processes with neat inflows and outflows of responsibility and information. On paper, you improve things by modeling things on paper, and then moving things around, on paper, then teaching people…
As I’m about to complete my share of a longer engagement on using Lean principles to improve the processes at an online services firm, it occurred to me that the efforts we undertook to properly embed Architecture practices into their Scrum process were novel. I haven’t seen much written about how to do this in practice, and I imagine others may benefit from understanding the key connection points as well. Hence this post.
First off, let me be clear: Agile software development practices are not at all averse to software architecture. But let’s be clear about what I mean by software architecture. In an agile team, most decisions are left to the team itself. The team has a fairly short period of time to add a very narrow feature (described as a user story) to a working base of code and demonstrate that the story works. The notion of taking a couple of months and detailing out a document full of diagrams that explains the architecture of the system: pretty silly.
The value of software architecture is that key decisions are made about the core infrastructure of the system itself: where will generalizations lie? Will a layered paradigm be used, and if so, what are the responsibilities of each layer? What modules will exist in each layer and why will they be created? How will the responsibilities of the system be divided up among the layers and components? How will the modules be deployed at scale? How will information flow among the modules and between the system and those around it?
The way these questions are answered will indicate what the architecture of the system is. There are many choices here, and the “correctness” of any choice is a balance between competing demands: simplicity, security, cost, flexibility, availability, reliability, usability, correctness, and many more. (These are called “System Quality Attributes”). Balancing between the system quality attributes takes thought and careful planning.
So when does this happen in an agile process?
Let’s consider the architect’s thinking process a little. In fact, let’s break the software architecture process into layers, so that we can divide up the architectural responsibility a little. You have three layers of software architectural accountabilities. (Repeat: I’m talking about Software Architecture, not Enterprise Architecture. Please don’t be confused. Nothing in this post is specific to the practice of Enterprise Architecture). All this is illustrated in the diagram below. (Click on the diagram to get something a little more readable.
At the top, you have the Aligning processes of software architecture. These processes consider the higher levels of enterprise architecture (specifically the business and information architecture) to create To-Be Software Models of the entire (or relevant) software ecosystem. If you’ve ever seen a wall chart illustrating two dozen or more software systems with connectors illustrating things like flow of data or dependencies, you’ve seen a model of the type I’m referring to. Creating and updating these kinds of diagrams is a quarterly or semi-annual process and reflects the gradual changes in the strategy of the enterprise.
In the middle, you have the Balancing processes of software architecture. These processes consider the needs of a single system but only from the level of deciding why the software will be divided up into modules, layers, and components, how that division of responsibility will occur, and what the resulting system will look like when deployed in specific technologies in a specific environment. All of this can be conveyed in a fairly short document that is rich in diagrams with a small amount of text explaining the choices. This occurs once when a system is moving forward, and the software architecture can be developed alongside the first couple of sprints as input to the third and subsequent sprints.
At the bottom, you have the Realization processes of software architecture. This is where the architecture becomes software, and this is where decisions are made about the choice of specific design patterns, the appropriate level of configurability vs. simplicity, and the ability to demonstrate whether the actual intent of the architecture occurs in practice. In agile, this layer occurs within the team itself. The software architect can offer advice about what patterns to use, but it is up to the team to realize that advice and/or decide not to implement it. The team will very likely implement the software architecture as described, but may choose to improve upon it.
What does the process look like
There are many visualizations of scrum running around. Some are described in books, others in papers or blog posts. Most share some common elements. There is a product backlog that, through the magic of sprint planning, the team extracts a subset for the sprint. This becomes the sprint backlog. The illustrations then tend to show the various rhythms of Sprint as cycles (sprint cycles and daily cycles), ending with a demonstration and retrospective.
In order to illustrate and train a team on all elements, including the business analysis elements, we chose to be a bit more explicit about the steps PRIOR to sprint planning, including the processes of creating and improving the stories prior to the start of a sprint. (as above, click on the image to enlarge it).
Astute observers will notice that we added a step that we are calling “pre-sprint story review.” This is a meeting that occurs one week prior to the start of a sprint. It is called by the product owner and he or she invites “senior chickens” (architects, user experience leads, development and test leads, and any other “non-team” members that want a say in the sprint.
In that week prior to sprint planning, those folks, working with the product owner, can improve the stories, add constraints, refine the description and acceptance criteria. And here’s where the architects get to play. Architects fulfilling the role of “Balancing” in the model above will have (or can create) an outline document describing the architecture of the software system, and can “link” that document to the SPECIFIC STORIES that are impacted by that design.
(Note: assuming you are using a tool like Microsoft’s Team Foundation Server, that fully enables Scrum in multiple forms, this is a nearly trivial activity since a document can be easily linked to any story. Enough advertising.)
So is an architect a chicken or a pig? Answer: depends on what “layer” the architecture is at. The top two layers are chickens. The third layer, realization, is done by the team itself. The person on the team may or may not have the title of “designer”. (I’d prefer that they did not, but that’s just because I believe that ALL team members should be trained to fulfill that role. In reality, the skill may not be wide spread among team members). Therefore, the third layer is done by the pigs.
I hope this provides some insight into how a team can embed software architecture into their scrum cycles. As always, I’m interested in any feedback you may wish to share.
Without the capability of linking application development with the infrastructure such as servers, middleware and network your organization will most likely be in more trouble than the decision-makers think. Information technologies have a profound impact on the organization’s ability to reinvent its business model(s) and the products it produces and since organizations in various industries […]
I did a scan around the web to figure out what many of the leading thinkers were saying about IT project failure and the root causes. Numbers varied between 20% and 80% of projects failing to deliver on their business case. The root cause analysis that follows from these failure numbers spends a lot of time looking at the IT project, but most forget to look at the causes of failure that are outside the project’s control.
In the diagram below, the central blue box represents causes of IT project failure that are inside the control of the IT team. As you can see, there are many more causes of IT project failure that are OUTSIDE that blue box than inside it. Yet, countless articles have been written on the factors INSIDE the box. I think it is time we take a slightly wider lens to the problem!
The factors outside the project are as important, or more important, than the ones inside the box. I have worked on many projects over the years, and if I look back at the ones that ended up cancelled or scrapped, the reasons were not ones in the blue box. They were usually ones from the top box: where the project should not have begun in the way that it did. Let’s look at these six factors:
- Lack of accountability for success – Sometimes, in an organization, an interesting situation will arise. I call this the “can’t lose” scenario. In this scenario, the CIO will suggest that the business can benefit by doing some particular initiative, and his or her people will set out to find a business sponsor for the initiative. In this case, the sponsor will reap the benefits of the initiative, not the CIO. (Let’s say that the CIO has suggested a Business Intelligence initiative, and signs up the Chief Marketing Officer to sponsor it). If the project succeeds, the sponsor wins. If the project fails, the CIO (not the sponsor) loses credibility. After all, it wasn’t the sponsor’s idea. The CIO dreamed it up, after all. In fact, the sponsor may not do any real “sponsoring.” He may sit in an occasional status meeting and check his e-mail while the program manager he assigned from his team answers questions… and keeps him protected from any blowback. The sponsor in this case is not accountable for success. No one is. The odds of this project succeeding are small. (Note: a good IT Governance system prevents this condition).
- Failure to get buy-in from above – Often, a senior executive will ask his operational leaders to suggest IT projects for the planning cycle. The executive (let’s say Tina Smith, Vice President of Sales) assumes that the leaders (William, Ashok, and Lee: all three are General Managers) will ask for projects that align to their individual strategic needs. One of those General Managers, William, decides to create an IT project to implement a SOA service bus. He wants the bus because he is investing in replacing an application and his IT team tells him that a bus is a good idea. After all, if everyone in the Sales group uses it, everyone will benefit. But William doesn’t get Tina Smith’s buy in. He implements his SOA service bus, but when subsequent IT funding requests come up to move other sales applications over to the bus, his peers Ashok and Lee are not interested. Their priorities don’t include moving to the bus. William bought a white elephant. Because he didn’t get Tina’s buy in, none of his peers will reap the benefits. Is the SOA bus a failure? Well… it’s not a success.
- Misaligned prioritization – Lee James, General Manager of Sales for the Western region of Contoso owns a sales strategy for the enterprise. The strategy is to make the sales force more productive by moving the company over to a new CRM system. The CRM system has out-of-the-box support for a long list of reports, but his team tells him that he shouldn’t use those. Instead, he should put in a Business Intelligence system in the cloud. Lee wants to trust his team to pick the right solution. They tell him that the cost of this system is $4 Million over two years. The company is moving to a new strategy that de-emphasizes the direct sales force. Instead, the company will be relying on social networking and direct eRetail to make sales. To make the new strategy work, there are eight projects totally $8 Million that need funding. There is a competition for dollars in the IT budget. Lee goes to bat for his $4 Million commitment. It’s important because “we are already here, so we have to complete the job.” This is an example of misaligned priority. The new BI project can be safely cancelled or scaled WAY back to support a strategy elsewhere, but that other strategy doesn’t belong to Lee. The one that he owns needs money and he’s going to keep fighting to fund it because his boss, Tina Smith, isn’t aware of the tradeoff. Lee fights, and Contoso loses. Which project will fail? Both of them.
- Slow decision making – This one is related to the other problems in that it is often, but not always, caused by a poor governance system or poor prioritization of the effort. In this scenario, the team implementing the project needs the sponsor to make a decision, often one that requires consultation with his or her peers or counterparts in another part of the company. The sponsor may be unwilling, unable, or incapable of having a difficult conversation. On the other hand, they may be indecisive. Regardless, this situation can cause serious delays in a project. For example, Ashwin’s application sends EDI-formatted messages to a health insurance provider. The provider had indicated that new fields are allowed in the messages that are very valuable to Ashwin’s line of business. However, he has to upgrade his EDI translation software to get the new fields. The EDI translation software is also used by the Finance group to send banking transactions. Ashwin knows that the finance group will get a benefit out of the upgrade and wants them to help cover the costs. This upgrade will add $25,000 to the cost of his project and he’s on a tight budget. The software team offers to write a small component that will insert the fields into the data stream, but its complicated and fragile. Ashwin cannot decide if he should ask the finance team for funding or if he should upgrade the software. The project team cannot proceed without a decision.
- Lack of authority to drive change – This one is quite common. Ashok reports to Tina Smith, VP of Sales for IBuySpy. He has told her that he wants to take the lead on one of her strategies: to consolidate all of the eCommerce systems in the company to a single outsourced vendor. Tina things that this will reduce costs all around, and Ashok is happy to take on the challenge. His business analyst does a survey and it turns out that the services and support team offers customers the right to buy replacement parts online. In addition, the “outside services” team allows customers to buy a service contract on their own area of the website. The main sales site, of course, uses a high-volume low-cost vendor that Ashok is comfortable with. Ashok sets up meetings with the owners of the other applications, but they choose not to provide the needed transaction cost information or the cost estimates of changing their internal systems. His data is light but he strongly suspects that there is a good business case for switching. Unfortunately, he doesn’t have the data to prove it. Without data, the Tina won’t take the case to the CEO to require the other teams to jump onboard. The strategy fails. Neither Ashok, nor his manager Tina, have the authority to require their peers to adopt a lower cost system.
- Lack of influence to rally peers – In many organizations, the prevailing attitude is that the boundary between “centralized” services and “federated” services is intentionally fuzzy. In these companies, there is no policy that requires a function to be centralized vs. federated. For example, if a company has ten divisions, three may have their own financial processes, while the other seven share a common finance unit. The decision to use the central finance unit, or to own a federated unit, can be made and unmade depending on who the leaders of the business are. (Note: leaders change, and with them, these decisions change as well). This kind of policy is good for flexibility, but creates both inefficiencies and often, confusion. This is simply a tradeoff in the design of organizations. It is neither right nor wrong. In this environment, any business can create a “centralized” function by convincing more than one business leader to come together to create a shared resource. The flaw comes when a group wants to create a “centralized” (or simply “shared”) function, but they lack the influence needed to get multiple businesses to join the initiative. Difficult questions of shared funding can be demotivating, and the political need to create “alliances” can be difficult for junior executives that, in many ways, are competing with one another for the next rung on the career ladder.
Each of these conditions has the potential to kill an IT project. I would suggest that MANY IF NOT MOST of the failures of IT projects can be traced to one or more of these conditions, but these conditions rarely get counted in the statistics for “Causes of IT Project Failure.” Why? Because, in most cases, projects that suffer from these conditions are either never funded, or are reworked so that the political problem is simply avoided. The project business case does not reflect the problem, so the criteria for failure (doesn’t meet the business case) is never met. Efforts are made to avoid (but not address) the problem before the business case is written!
This is the world of the Enterprise Architect. These are the kinds of “failure” that fill the eyes and ears of an Enterprise Architect. If an EA focuses on only these six causes, he or she will deliver real, tangible, and unique value to their enterprise without ever overlapping the roles and responsibilities of an IT architect, business analyst, or technologist.
Big data will continue to spread with emerging associative terms like big data expert, big data technologies, etc. Here are E.G. Nadhan’s top 5 reasons why the term big data has stuck, and why it may be appropriate, after all… Continue reading →
A vendor recently asked me a question, and not for the first time, that amounted to “why won’t you buy these delicious software licenses for our amazing identity-management suite, which provides comprehensive solutions to the challenges faced by North American corporations in dealing with compliance requirements and bringing together disconnected versions of identity from across […]
Steady growth of service oriented practices and the continued adoption of cloud computing across enterprises has resulted in the need for integrating out to the cloud. When doing so, we must take a look back in time at the evolution of integration so…
Enterprise Service Bus and Espresso One of the few really-good boutique coffee-roasters and espresso specialists in Auckland labels its beans “esb”, for “ethically-sourced beans”. It’s something of a stretch, but espresso is important and beneficial in fundamental, foundational ways not dissimilar to the importance and benefits an Enterprise Service Bus can bring to integrated organisations. […]
Value management through TOGAF 9.1 and ArchiMate 2.0. Continue reading →