5 years, 6 months ago

Opening up the Mendix model specification & tools ecosystem

Link: http://feedproxy.google.com/~r/TheEnterpriseArchitect/~3/OMXcNXN89kw/

There is something interesting going on in our industry. From the inception of the first programming language we always have been uncovering new ways to program computers. New ways triggered by new hardware architectures and new ways triggered by developers wanting languages that are easier to grasp and are more productive. We have made major steps from machine code to today’s higher-level modern languages. However, over the last decade it seems like there have been no major steps in productivity anymore. Are 3GL languages like Java, C#, JavaScript, Python, Ruby, and Go the last generation in the evolution of how we program computers?

box openIn short: no, we can create easier to learn languages that are vastly more productive. That’s what we have been doing at Mendix for years. However, the concern is a lock-in on technology, innovation, and specific vendors. That’s why we are opening up our entire platform.

Let me explain you why and how we are doing this.

Why are “recent” attempts to higher-level languages failing?

If we look at recent outcries for more people to learn to code, I would certainly think we could use languages that have a lower entry level and make us more productive. The amount of software is growing exponentially; our ability to produce this software is not. There have been a lot of attempts to solve this challenge, though. Let’s briefly explore some of them, without pretending to analyze exactly why they didn’t succeed.

CASE tools and 4GL environments have been created, already decades ago, to make it easier and faster to produce software. They partly succeeded, but were slowed down by vendor lock-in concerns, missed the boat in major technological waves, and have been overtaken by the open source ecosystems of 3GL languages. They also suffered from too generic languages, not even meant to be executable (e.g. UML).

As a response to this the Model-Driven Development (MDD) community shifted focus to smaller, more specific languages (or should I say the community came into existence?). These so-called Domain-Specific Languages (DSLs) focus on a specific problem area or even a business domain. That ultimately also defines the scope and applicability of these languages. They are by definition not meant to solve a generic issue and hence don’t attract a large developer community.

To alleviate the fear of lock-in both modern 4GL tools (as I would call them) and MDD / DSL tools focus on generating code. The idea is that if you would want to get rid of the tool, you can just pick up the code and continue as you had written it yourself. This, however, slams you back harshly into the world of 3GL languages. Which means that collaboration with business / less technical people becomes much more difficult. It also means that the business knowledge that was available in your models is lost as you now only have low-level code in which the business knowledge is not easy to find or deduce.

This begs the question if we can really make a step forward if we stay attached to the “old world” of 3GL languages. Can we make a bigger leap and continue the evolution of programming?

What companies actually need from a business perspective

We need some context before we loose ourselves in the technical aspects of this question, so let’s step back for a moment and look at the broader picture. What we see happen today is that “software is eating the world”. Companies that have software at the core of their business are disrupting existing industries, thereby forcing the old and large enterprises to become a “software company” as well. I’m not exaggerating when I say this is happening in every industry and it is happening quickly. Allow me just one example: a young company like AirBnB is now worth more than some of the largest hotel chains on this planet.

What does this mean? It’s disrupt or die for every company. Companies need “digital innovation” to survive. This is a radical change. Screaming at the IT department will not get you there. In most companies IT is not able to cope with the current needs of the business and what we are looking at is more software quicker. So, what does this radical change entail?

Enter the 4 P’s of digital innovation:

  1. Portfolio: companies need to be smart in how to approach this change. Not all applications are equal. They have different lifecycles. Systems-of-record don’t need innovation, but systems-of-differentiation and systems-of-innovation do. Companies need to focus their energy on the latter. They also need to create a roadmap that starts small (the first new app), celebrates success and then goes from structuring to scaling the approach.
  2. Process: companies need to embrace failure. Failure should become accepted, fast, and cheap. Only that will lead to experimentation and hence innovation. This means: agile processes, fast feedback cycles, and working close to or within the business units.
  3. People: companies need to organize themselves differently. They need to create small teams (4 – 8 people) that are cross-functional. These teams should have the autonomy to independently create and release new software (see also: how to scale an engineering team). With cross-functional teams I don’t mean just “devops”. Business people should also become part of the development teams. This is possible (and practical) when we enable them to be part of the process. Which brings us to the last P.
  4. Platform: companies need a platform that enables the previous 3 P’s. A platform that enables small teams (partly) consisting of business people to experiment quickly.

So, it is not just “tools” what we need when we want to the next generation of software development. The cultural change is actually more important. This doesn’t mean that companies should ignore questions around the openness of the platforms they use. However, you can imagine that if they adopt the 4 P’s of digital innovation (which is a necessity) we cannot simply say that our solution to openness is to propel companies back in history by leaving them with exported 3GL code. Their whole organization will be optimized for a different way of working, with different people, which have a different kind of knowledge…

Combining high productivity platforms and openness

At Mendix we explored solutions to this situation for a long time. We wanted to provide our customers and ecosystem with the openness they needed, but without loosing track of what modern companies actually need. We cannot at the one hand say that companies need a modern platform that helps different people, with different processes create software faster, and a the other hand try to address lock-in concerns by saying to revert all of that and get back to a traditional approach.

Our approach to building applications entails abstraction and automation in every aspect of the application lifecycle. I think our platform provides the fastest way to turn ideas into applications. You can quickly develop applications using a higher-level approach. Mendix uses multiple graphical, horizontal DSLs to combine high-levelness and broad applicability. The resulting application model can be 1-click deployed and scaled on our public Platform-as-a-Service (PaaS) or on any Cloud Foundry based private cloud.

What happens under-the-hood is that we interpret and execute the application model. So, the application model is the metadata that instructs our platform how it should behave for this particular application. This model interpretation approach is part of our vision that “the model is the code”. We do not transform the model to code, we always keep the model, execute it, debug on the model level, etc.

So, when pondering on different solutions to the openness questions, one thing was clear for us: the model should stay central to our approach. The model is what contains a description of the business of our users. The model contains actual knowledge. This led us to the following requirements:

  • We want to give our users the ability to programmatically access the models to export and import them to and from another model-driven or metadata-driven platform.
  • We want to create an open tools ecosystem. Everybody should be able to build tools that interact with and manipulate our models.
  • We want open innovation. We shouldn’t be the limiting factor in how quickly our platform, the tools, and the models can evolve.

This is not something that happens overnight. Our platform has been fast-growing for 10 years, with a fast-growing team working on it. So, we have been working on opening our platform for quite some time. Let me tell you what we did.

Mendix: the open platform for digital innovation

We started by defining the end goal: a REST API that allows you to create, read, and update any model in the Mendix platform. That’s of course only feasible it these models are formally defined. Hence, the next step was to formalize our model specification, i.e. to formally define the metamodel in such a way that other tools than our own Business Modeler could interact with it. The image below shows a visualization of a snippet of our metamodel (the entire metamodel is documented in this way).

metamodel_domainmodel

To make it a slightly more complicated: we didn’t actually start by formalizing our metamodel. We started by creating a language to write our metamodel in. We used Xtext for that. Using Xtend we wrote code generators so that the metamodel is actually driving the implementation of our model repository, the API, the SDK, and all tools interacting with our models.

The Model API is a generic API that allows you to interact with application models. To make that easy, we also created a TypeScript SDK (Software Development Kit), which provides a typed interface for any element in our metamodel. Let’s make it less abstract by showing you an example:

project.createOnlineWorkingCopy().then(workingCopy => {
    workingCopy.model().allModules().filter(mod => mod.name == "MyFirstModule")[0].fetch((mfm) => {

        let customer = createEntity(mfm.domainModel, "Customer");
        addAutoNumberAttribute(customer, "CustomerId", '1');
        addStringAttribute(customer, 'FirstName');
        addDateTimeAttribute(customer, 'SignupDate');
        addStringAttribute(customer, 'LastName');
        addStringAttribute(customer, 'Email');
        addStringAttribute(customer, 'Address');

        let invoice = createEntity(mfm.domainModel, "Invoice");
        addAutoNumberAttribute(invoice, "InvoiceNumber", '1');
        addDateTimeAttribute(invoice, "TimeStamp");

        let invoiceLine = createEntity(mfm.domainModel, "InvoiceLine");
        addStringAttribute(invoiceLine, "Product");
        addIntegerAttribute(invoiceLine, 'Quantity');

       associate(domainModel, invoice, customer, 'Invoices');
       associate(domainModel, invoiceLine, invoice, 'Lines');

    })
    return workingCopy;

}).then(workingCopy => workingCopy.commit());

The above example shows how to open a model, how to change it, and how to commit the changes. In the above example we create new model elements (3 entities and associations between them). In the same way models can be read and updated (see the example in this post). This holds for every single element in our metamodel. Everything can be accessed and changed via de Model SDK and Model API.

The Model API opens a variety of interesting use cases, some examples:

  • Exporting models: export application models for documentation purposes, or to port applications to another platform.
  • Legacy migration and modernization: support automated migration of legacy applications to the Mendix platform.
  • Quality assurance: perform automated static analysis on application models to check for inconsistencies and ensure quality standards and maintainability.
  • Task automation: write scripts to automate bulk tasks, such as reformatting flows or changing certain parameters/activities across multiple models, saving time and ensuring consistency.

With these and other use cases, we are actively working with our partners to create a whole tools ecosystem around our platform. This ecosystem will help our community to get more out of their models.

What’s next?

This is an important milestone for Mendix the product, and Mendix the company. This is the first step and together with our community we will learn and evolve our approach. We hope others will join in our quest towards more openness. In the end all customers will win.

We are now running a private beta with our Model API and Platform SDK. We will open up to the public at the end of November.

I’m happy to hear your feedback, guidance, and ideas!

The post Opening up the Mendix model specification & tools ecosystem appeared first on The Enterprise Architect.