Much is written about lean startups, agile software development, continuous integration or even continuous deployment. All with the goal to help us understand the dynamics of successful software companies and their development teams. I am inspired by these stories, they show me what the ideal situation is like, they challenge me to improve our current way of working.
Today I want to give something back. I want to share with you our tale of developing software for the enterprise over the last 7 years at Mendix. Not to show you the “ideal situation”, but to share with you the challenges we encountered in scaling our teams and in getting customers that depend their business on our App Platform. We experienced exponential growth ever since we started and much of the phases we went through are not specific for our situation, so I hope you enjoy the story and perhaps learn something from it.
Start up: what to build?
I never forget that moment, more than 7 years ago when I first entered the “office”. Just a few people in one room starting a business. It felt exciting and I couldn’t wait to be part of it. One day later I was sitting at a desk with code in front of me (and two other people constantly using the phone for “sales”).
Did we know what to build? Did we know what our product would look like, even in a month’s time? No. We did have a clear vision, though. A vision that, we believed, would change the world. Our one and only goal in these early months was to create features as quickly as we could. To make our vision tangible and share it with potential customers. To say it in other words: we quickly iterated with the “market”.
The first customers: we need this feature, it blocks this important deal
And then it became even more exciting: the first customers started using our software! However, there was often an accompanying message to the development team: we really need these new features otherwise we cannot deliver what we promised. The situation for our development team in principle stayed the same: cranking out new features as fast as possible. Only we did it for real customer cases this time, instead of for “just” our own vision.
I think every developer recognizes this situation where sales people “demand” new features to be able to close a deal. In this phase of your company you should welcome this (within certain boundaries of course) as your main goal is to search for the right market. The main challenge often is to make sure you do not keep doing it this way, to determine the right moment to switch to a more structured approach.
In the first year we mainly flip-flopped between the customer- and vision-driven method of operation. How long you iterate between these two modi depends much on the stability of your vision and how well it fits with the actual market you find.
Stabilizing: not all customers want to “live on the edge”
Luckily we found a market that was waiting for a product like ours. There was a lot of pain we could take away with our product. So, we stabilized our product focus and our user base grew quickly. This doesn’t mean our messaging and positioning has been the same since then. We continuously change and improve those, but the core vision of our product is the same today.
A growing customer and user base also introduced new challenges. No longer was our focus just on cranking out new features as quickly as possible. There are these customers, you know, that value stability over new features. Whether they do depends on the “maturity” of your customer, but even if they do not value stability, they will quickly learn to. It goes like this: the first time you tell the customer “we fixed the issues you reported and we also added this cool new feature” they will be very happy. But, after learning that every tiny change in software also brings new risks they will start asking for releases without these “cool new features”.
I know, your quality assurance should be so good that users of your product trust every change you release. However, no quality assurance approach can match the “quality increase” a piece of software goes through when it is hardened in production. So, when developing mission-critical software you will end up with different types of releases. In our case we just followed the best practice of having major, minor, and patch releases with the accompanying 3-part versioning scheme (x.y.z).
The additional challenge this brings is how to plan these releases. Ideally, as a development team using an agile software development method, you have a single backlog and you work on a single sprint. Just work with different branches in your version control system to make sure you can release features separate from bug fixes. However, we did experience two difficulties with this approach. First, it is becoming a full-time job to discuss the importance of incoming issues with the caller. They quickly learn that the resolve time of their issue depends on how well they explain the severity of their issue. Second, it is difficult to keep focus if you mix issues that address any part of your product with the tasks that cover the specific major feature you are working on.
We ended up with a process we still happily use today. We divide our work in chunks of 8 weeks. Each 8 week period contains the following elements:
- 5 weeks of uninterrupted “roadmap” work. We can really focus on difficult problems, be creative, be innovative, bring our product to the next level. These 5 weeks are divided in multiple sprints and each sprint results in a working piece of software. One or more of these 5 week periods result in a major release.
- 3 weeks of “maintenance” work. This period results in a minor release and is focused on processing defects. We include minor features too, but only if the use case is very clear and the risks are low. The changes we do in these minor releases are always merged back to the main line and will thus be included in any future release.
- 2 “FedEx” days. A monthly day during which developers build a new feature they like, research a new technology, etc.
Organizing our work in this way gives us the following advantages:
- Our users know that we do a low-impact minor release every 8 weeks. Only if a defect is so severe a user cannot wait we have a discussion about a patch release. In general our users really like the clarity of the release scheme and issue deadline before each release.
- It makes product management easier. It is very difficult to decide on the priority of small(er) feedback items versus strategic product changes. Now we always dedicate time for the smaller feedback items and we know the amount of time we have for “roadmap” work.
- It ensures time and focus for innovation, for bold new ideas. It is so easy to just be busy with maintaining the current status quo. You will keep your current users happy, but risk becoming irrelevant in no time.
So, business is coming in, users are happy, development is structured, enough said right? Well, that could have been the case, but when going through these changes we didn’t see the dark clouds gathering on the horizon…
The 2nd system effect: let’s do it all over again, but better (we thought)
When you develop a product like ours, you make choices, lots of choices, everyday. Especially when under pressure to deliver, you need to make choices about leaving out features or implementing a feature less elegantly than you would have done in an ideal situation. Combine that with all the new insights you get when your product is used by more and more customers and slowly you get the feeling some fundamental elements need to change to keep up with the growth and the new usage you see (and didn’t predict before you started). At least, that happened to us. We clearly identified a new vision for our product, one that extended the initial vision we had. We were not afraid to aim high, to have ambition. It truly was a so-called Big Hairy Audacious Goal.
So far, so good. We had this great vision and we used our roadmap time to start with the implementation. During the maintenance periods we maintained the current version of our product. The first problems arose when we were in the middle of refactoring each and every element of the core of our system when an important, bigger feature was needed by our users. What to do? We only had one option: build it in the current stable line and just keep working on the new major stuff in a separate line. Guess what happened? A couple of months later almost the complete team was working on the “stable” line and the new amazing upcoming major release parts were gathering dust somewhere in a corner of our version control system. Finally, more than half a year after we started, we decided to kill the development of the new major release. In hindsight that was one of the best decisions we ever made, and I can tell you that it took a lot of courage.
What did we learn from this experience? First, these kind of “big mistakes” quickly become some sort of a legend that is told to every new team member that joins our company. Second, we realized we had fallen victim to the so-called second-system effect. This effect refers to the tendency of small, elegant systems to have big, feature-creeped systems as their successors (luckily we never released the successor we had in mind). You try to compensate for all the stuff you didn’t get to do in the first version of the system. We thought it was heroic to take the time to re-think everything and to dare to re-implement big parts of our product. We were partly right, it wasn’t the re-thinking that was the problem, it did go wrong in the way we defined “working software”, in the way we defined the slices of work for each sprint.
We still worked in sprints and did sprint demo meetings with stakeholders involved, but we focused too much on horizontal slices. Slicing a system horizontally means implementing a system one layer after another, e.g. first build the database layer, followed by the business logic layer, and finally the UI layer (or any other layers you have in your system). Using this approach you cannot show a real end-user feature until you finish almost all layers. Slicing a system vertically means developing just the parts of each layer that are needed for a specific feature, followed by the parts that are needed for the next feature, etc. Vertical slicing enables to deliver working functionality as soon as possible. It also helps to focus the development of the underlying technical layers, i.e. only build what you need.
By focusing on horizontal slices we moved into a modus of brainstorming and speculating about everything that was needed in a certain layer. This results in discussions and ideas that lead to an endless increase of the scope. In fact, years later we can still refer to this version we were trying to build. When we start to build a new feature we often realize: “this was also part of our big master plan years ago, right?”. The amount of times this happens is truly astonishing (“did we really think we could include all this in a single major release?”).
So, we changed some things, or better formulated, we emphasized some parts of our working process. First, the 3 week maintenance period is for minor stuff, if it doesn’t fit in the 3 weeks, it should be part of our roadmap time. Second, during roadmap time we focus on vertical slices and work in sprints with a maximum of 2 weeks that always result in working software.
Sounds trivial right? Why is it then that so much people can relate to a story like this? I think because doing it right takes a lot of continuous discipline. As developers we tend to think about the horizontal slices when we plan the implementation of new features. Furthermore, a focus on vertical slices tends to create waste. You have to build stuff that you know you will throw away if you implement one of the next features, or you already know what other things you need in a certain layer, better create it directly, right?
We had this experience in the past, but it is an ongoing challenge to keep our focus. From time to time we need to “reset” ourselves and make sure we focus on having a big vision, but operate in small steps. What we truly experience: complexity is easy, simplicity is difficult. And this tendency to the “complex” isn’t the only challenge we had in keeping our focus…
Ramping up: introducing “departments”
Focus is more than “not doing everything at once”. It also means uninterrupted time to really focus on doing highly-skilled work. Our 8-week rhythm is helping with that, but as you can imagine we cannot close the doors for 5 weeks if we are in our roadmap period. However, we also know that a developer that operates in “the zone” is exponentially more productive than one that is interrupted multiple times per hour. If your company grows this becomes quite a challenge.
What we did over the years was introducing departments around product development. There is no definite guide that tells you when to introduce a “department” and how, but these are my main observations:
- As long as your company has less than 20 employees you won’t need formal departments. You probably do not even have team leads or “managers” except for the founders of the company.
- Once you grow bigger than, let’s say, 20 employees you will probably need to introduce team leads and departments as the founders cannot manage everything themselves anymore.
- There will be a point in time when the team lead(s) of your product development team(s) are completely occupied with “operational stuff”. That’s a dangerous situation and you have to detect the early signals and make sure you start to “outsource” part of the work to other (probably new) departments.
- From a development team perspective one of the first things you will need are people that connect new customers, and in our case to actually implement a solution for customers using our product. As a company you have to do that yourself to make sure you have the right “lighthouse customers” that help you attract new customers (customers and users of your product need examples to be convinced to start using your product). The dynamics of this type of work are completely different from product development, hence you do want to split that to keep your focus.
- Another type of work with completely different dynamics is “support”. You do want dedicated attention for that as early as possible. Embedding a good support process in your organization that ensures knowledge sharing is a key element to scaling your organization. No proper support means that you become victim of your own success: more customers means more developer “distraction”. There are of course multiple ways to solve this, but in the end you want a team with dedicated attention for users, creating FAQs, handles incoming issues, etc. And, do not forget we are talking about enterprise software, hence we have some SLAs we need to comply with.
- Tech sales is also something you want to formalize quite early. Selling software to enterprises means answering a lot of questions. And not just an explanation of how your product works, that’s something that needs to be explained and written down by developers that build the features in my opinion. But, think about the positioning of your product compared to other types of products, compared to competitors, creating and giving demo’s, convincing customer architects, etc. Again, there are smart ways to do this, but someone needs to do it.
- What you should NOT just split out of the product development department is “operations” (i.e. everything that is needed to run your software – we’re talking “as-a-Service” here). I strongly believe in cross-functional development teams that are responsible for running their own parts of the products (you built it, you run it). This creates a culture of responsibility, developers will think about the consequences of their changes thoroughly. It also ensures that a vertical slice for a feature incorporates the operational aspects, meaning that finishing such a vertical slice really results in working, releasable software. You can (and probably should) of course create a separate team that supports the other teams in the way they deploy new versions of their software, monitor it, etc. by automating it and providing the developers with the right tools.
In gradually moving to a situation with more “departments” we were able to return to more focused development and enough time for innovation. Every change creates a new challenge, though. Let me tell you a story… after a short intermezzo…
Knowledge transfer: how to keep other departments and users up-to-date?
Outsourcing part of the work to other departments (other than your product development teams) leads to more “distraction” instead of less in the short time. The main reason: knowledge transfer. Support and pre-sales for example need to know everything about the product, new releases, etc. We have seen two things that really help with this.
First, you need developers with writing skills. I do not mean “writing code” but rather documenting new features in a reference guide, explaining how to use new features in a how-to, and explaining the rationale behind new features in technical blog posts.
Second, we have a “tech-of-the-day”. That’s one of the developers who is available for answering questions from other departments. We rotate this role every day. This approach makes sure that other departments can always approach someone with in-depth product knowledge and at the same time it ensures focus for all other developers. It also ensures that each developer is exposed to user feedback and discussions about how to use or sell our product.
The ivory tower: how to keep the feedback loop connected?
Let’s go to the story I promised you. So, we released this cool new feature. We, as a development team, were enjoying the moment. You have to celebrate your milestones! However, within 5 minutes someone from our “professional services” department stood at my desk with a less joyful face. He had just glanced at our new release and then ran to my desk to explain to me: “this feature you just released is completely useless”. That’s fun, right? What wasn’t so funny was the fact he was right.
How is it possible that we were crafting a new feature for weeks and thought it was the best thing since sliced bread, while one of our users could see the flaws in 5 minutes? Shielding the product development team as much as possible from external “distractions” brings some serious issues with it. Isn’t one of the core principles of agile software development to value “customer collaboration”? That’s for a reason. Without the collaboration with your users and stakeholders you are building stuff you think is valuable, but the better and longer the shield works, the more you are moving away from reality. Product development slowly turns into an Ivory Tower. How to find the right balance between focus and collaboration?
To start with: these two things, focus and collaboration, are not mutually exclusive. However, we have experienced first hand that finding the right balance is a challenging, continuous process. Especially in a fast growing company. Let’s look at some things we introduced over the years (some more recent than others) to fight the creation of an Ivory Tower:
- The most logical moment to involve users in the development process is the sprint review meeting. This meeting is meant to demo the completed work to the stakeholders. We always try to involve a user in this meetings. This is often quite challenging as these users have a busy schedule. If it really isn’t possible to have a user in these meetings regularly the Product Owner should put more effort in aligning with users (at more flexible times) and being the “voice of the user” during the meetings.
- We start new projects with an “inception” phase that is used to identify the vision and story for the release, bring stakeholders to agreement, and create an initial idea of the user experience. We literally start with writing the press release for the release, present it to stakeholders and write down all the questions that arise (and their answers) in an initial FAQ. This really helps in guiding and scoping development and involves users as early as possible. What we essentially do is to start with the customer and work backwards until we get to the minimum set of technology requirements that satisfy what we try to achieve. Using such a continuous, explicit customer focus we aim to drive simplicity throughout the process. But again, we continuously need to remind ourselves: have a big vision, operate in small steps.
- All of our products include a feedback mechanism that makes giving feedback really effortless. This increases the chance that a user shares his thoughts, frustrations, and awesome ideas with us. This of course also brings an additional challenge: processing all this feedback in such a way users feel their feedback is valued and taken seriously.
- We sometimes invite users of our products to present at one of our Friday-afternoon-beer-sessions. In our case this means we have users from partner companies that present the enterprise apps they have build for their customers using our PaaS. This really helps to broaden the perspective of our development teams. Additionally these users are always asked to share with us what they like and dislike. In our experience users are always happy to share their work (and opinion) with us, and they are rewarded: almost every time they present something that really annoys them (or a hint for something they really would like) we can just include that in the next minor release.
- Product management should not purely be based on user feedback. Product management should be based on multiple influences, amongst them: your own product vision, market trends, marketing, sales, research and development, user feedback, etc. That being said, product management is a tough job. Most of the time you should be able to value different inputs and say “no” to a lot of them. If you want to please everyone, don’t become a product manager, you will create awful, feature-laden products. However, you shouldn’t become arrogant too. Using just your own vision as input will not lead to the best product.
This list is by no means complete. We feel we still need a lot of improvement in this area and it has our continuous focus to improve this.
Entering the real enterprise world: when you thought functional requirements are the only things that interest customers
We talked a couple of times about developing “enterprise software”. What do we mean by that? Actually, that changed over time. We did not start with a fortune 100 company as customer. What we first saw as a big company is small compared to the current customers we have. And in delivering software to the real big companies we also encountered additional requirements. If you ever built enterprise software this shouldn’t be a surprise.
First, the product or service you deliver should be covered by a Service-Level Agreement (SLA). I am not a legal expert, so I won’t go into the details. Just one piece of advice: make sure you carefully read and understand the SLAs of the services your product depends on. If you for example build software that runs in “the cloud” like we do, you need to understand the details of the SLAs of the Infrastructure-as-a-Service (IaaS) providers (hint: they can be meaningless). Your own SLA is limited by the SLAs of your providers, if it isn’t you probably want to contact an insurance company.
For us as a product development team the nice thing is that these kind of agreements with customers lead to higher uptime requirements. Hence, more time on the roadmap for things like high-availability, horizontal scalability, failover mechanisms, etc. Which is interesting stuff, right?
Another requirement some of our customers (especially in the financial and healthcare industries) have is that we as a company are audited every year. These customers are under a lot of regulations and are only allowed to do business with companies that comply to a number of standards. Examples are the ISAE 3402 and SSAE 16 standards. So, what do auditors look for in your product development processes? In my experience these are the main elements:
- A traceable software development lifecycle: have a look at the current version of your product that runs in production. You have to be able to show who approved the release and when, what the test report was, the source code for that release including the commit history, and the requirements and design decisions for the new features in that release. This maybe sounds bureaucratic, but with the proper tooling it is fairly easy without much overhead. We use a version control system connected to a Scrum tool (part of our own platform) containing the user stories (including related discussions), the results of our test runs (connected to builds) are saved, and we have a lightweight approval process for releases (just me giving a written approval as response to a request by one of the development teams that contains build number / revision and test report).
- Disaster recovery: you have to have a disaster recovery plan (and not just for audit purposes). You have to test the plan at least once a year and document that. Again not just for audit purposes, an untested backup (or failover environment) equals no backup.
- Maintenance processes: you have to have change management processes in place that make sure that changes on environments are properly scheduled, approved, and traceable. Furthermore, you have to establish processes that make sure that (security) patches of the third-party software you use are noticed and applied as soon as possible.
- Documentation of procedures: most people I know aren’t fans of audits because of the focus on documentation and I have to admit that was my point of view, too. Documentation is, of course, the only way an auditor can verify your processes, however, in my experience most are quite pragmatic and do not ask for multi-page process descriptions. You can keep it short and to the point. In a growing organization with a growing product you inevitably need more documentation. Not everybody will automatically know and understand everything (as opposed to the early days) and new employees need to learn a lot. Early-day employees often don’t realize how complex the current organization is for new employees as these early-day employees never had to learn everything at once, they have just grown into it.
- Staff training: for me this is probably the most “difficult” part of the audit. You need to show that your employees are capable and know their stuff (especially with regard to security). That’s difficult to show if you have a team of A-players that are autodidactic and use blogs and social media to keep their knowledge up-to-date. We hire people with such skills, we do not look at certifications. So, we do not organize formal trainings. However, organizing an in-house training day (read: a hacking contest – learn a lot about security by hacking example systems in an increasing order of difficulty) with an attendee list and short description of the day can often do the job too.
The main challenge? Fighting the obvious: becoming a bureaucratic, inert organization. So, keep it lightweight and be pragmatic. Focus on the reasons behind these questions and controls, in the end it is all about quality and continuity of service, which are good things to focus on.
SLAs and audits are just two examples of so-called non-functional requirements for enterprise software. In all this, it is again about finding the right balance: non-functionals need more attention than in the early years, but do not stop the innovation from a functional perspective.
Scaling the team: how to keep the “startup” vibe?
If you are still with me, you probably sensed the growth we had over the last years. That growth also influenced the size of our product development team. We started with just a few people, the company was one team. As explained, over time “departments” were introduced and thus a product development “department” was created. Currently, a couple of years later, our product development department consists of 5 teams and within a couple of months team 6 and 7 will be created as teams that grow bigger will be split.
In product development we didn’t grow very fast. If you grow too fast, teams are more busy with getting new people up-to-speed than with new versions of the product. Furthermore, it is important to not only get people up-to-speed with the technical work, they should also become part of the culture, they should breathe it. We call that: painting them Mendix-blue. Your culture is a competitive advantage that cannot be copied, foster it! Part of a focus on “culture” is the way we evaluate if someone fits in the team. We do not just look at skills, we let them talk with some team members and see if they add up to something more than the sum of the individual people. It is about passion, enthusiasm, and thereby pushing each other to excel.
So, culture is important, but how to stay focused, aligned, and flexible while also growing the product development team? Here are some of the things we do to prevent ourselves from becoming the next big inert organization:
- We keep the teams small (8 people at maximum) and autonomous.
- Teams are cross-functional. A team is responsible for a product, part of product (a service), or a certain feature. This responsibility includes everything from design and implementation to releasing and running it.
- Each team has a Scrum master who is part of the team. In principle a Scrum master is a developer with an additional role. With the small teams we have that’s working successfully.
- Everybody is a “maker”. Everybody creates stuff, whether it is code, tests, infrastructure automation, or designs. We do not have multiple layers of “management”.
- We have weekly Scrum-of-Scrums meetings to align the teams. Each team writes down the “what did we do” and “what are we going to do” stuff on beforehand. During the meeting (which is attended by a team member from each team) we discuss impediments and resolve issues that affect multiple teams. Everything we share and discuss is posted on our internal “social network” so that each team member always knows what’s going on within the other teams.
Much more can be written about the subject of growing a product development team. For now, just a final observation: system architecture tends to follow team structure, even if communication among teams is well-organized. Keep that in mind when you think about your team structure.
Moving on
Well, this piece became a bit longer than initially planned, but I hope you enjoyed it. Please share your experiences, comments, or questions in the comments!
We will continue our journey and continuously improve our way of working. Because it’s fun, but also because it’s necessary. We need to stay innovative, we need to challenge ourselves, to be able to stay successful.
Let me finish with a meta-observation: the word “focus” is mentioned quite a lot of times in this story (30 times actually). That surely aligns with my experience: nothing beats a motivated, highly focused team of professionals with a clear goal!
By the way: we are hiring for multiple development positions including a software development manager / agile coach. So, if you enjoyed this article and want to join (and impact) our process of continuous improvement you should check out our open positions.
You can also discuss this article on Hacker News.
