4 years, 8 months ago

New toolsets for enterprise-architecture

Link: http://weblog.tetradian.com/2015/01/05/new-toolsets-for-ea/

(There are several posts queuing up for publication here about this shift in direction towards ‘maker of tools for change‘, but this one is a bit more urgent to support a key conversation that’s happening right now. The other posts will follow soon, I promise! :-) )

There’s general agreement, I think, that most of the toolsets we have available to us at present for doing enterprise-architecture and the like seem to range from “somewhat-useful but not much help as we really need” all the way downward to “expensively worse-than-useless”. Even the best of them only give us good support for the very last stage of architecture development, the ‘final-diagrams’ and ‘models of record’ that are typically ‘tossed over the fence’ for others to implement. And most of the toolsets are really bad – in some cases to the extent of being an active hindrance – at the one thing we most need them to help us with, namely the exploratory ‘messiness’ that all but dominates the early to middle stages of any new development. So let’s do something about this: let’s define what we really need our toolsets to do.

Yeah, I know, I’ve written quite a lot on this over the years: the first reference I’ve found in a post here was from quite soon after I started this blog back in late 2006. Two recent posts that summarise some of the core ideas-so-far are:

It’d probably be worthwhile to (re)-read those two posts before continuing here.

By now, though, it really is time we got down to doing something practical about this – in other words, developing a real toolset that works the way we need. Hence what follows here summarises not only my current thinking about toolsets, but also where I’ve gotten to so far in terms of something that’s actually implementable, and for which even I could do something useful towards making it happen in real-world practice.

I’ve split the description below into various categories, with additional notes:

  • points headed ‘[Discussion]‘ indicate items that I know will need further exploration
  • points headed ‘[Implementation]‘ indicate items where I’ve already started on the detailed-design or even preliminary code for a proof-of-concept

Okay, here goes…

Fundamentals

The term ‘toolset’ here should be considered more as a type of platform, rather than a (probably futile) attempt to create single ‘toolset to rule all toolsets’. This conceptual platform needs to support any relevant type of information, created and edited in any appropriate way, across any appropriate type of device and user-interface.

In essence, everything in an architecture and its context ultimately depends on everything else. Existing model-types such as UML, BPMN or Archimate may be useful as views into the ‘hologram’ of that ‘the everything’, but should not be allowed to arbitrarily constrain the structure or the permissible views into that structure.

Because everything depends on everything else, every ‘thing’ needs to be treated as an ‘equal citizen’ within the overall hologram, and may be viewed, explored, examined, connected, in exactly the same overall way as for every other type of ‘thing’ – represented internally by the respective data-object. Models and model-types may constrain the views and, to some extent, the connectivities between ‘things’, but should not exclusively define them: that distinction is somewhat subtle, but crucially important. Data-objects may be worked on or edited by any number of appropriate ‘editors’ (model-types etc): the editor acts on the data-object, but an object is not inherently dependent on that editor for its structure or existence.

(See notes later in this post on structure, functionality and definition of editors.)

The core purpose of the ‘hologram’ is to support at least two key questions that can be asked about any object:

  • tell me about yourself (and why)
  • tell me what you’re associated with (and why)

It should also be possible to ask key questions about objects in relation to time:

  • tell me about your history (including change-history for the object-record itself)
  • tell me how you expect to change

Data-structures

As ‘equal citizens’, every data-object has the same conceptual structure:

  • unique-identifier
  • mandatory function-identifier(s) (type, role etc)
  • 0..n data-items, such as editable fields or non-editable tags (supports “tell me about yourself”, “tell me about your history” and “tell me how you expect to change”)
  • 0..n links to or between other data-objects (supports “tell me what you’re associated with” and “tell me how you expect to change”)
  • 0..n change-history records (supports “tell me about your history”)

– [Discussion] Some details of the structure – and, particularly, how best to implement it at scale, in a multi-user context – still need to be resolved. The core concept I developed some while back was the notion of a ‘mote’:

Extending the ‘equal citizen’ concept:

  • a ‘thing’ is represented by a data-object that has (or connects to) any number of fields and tags, but perhaps no links
  • a ‘link’ is a specific category of data-object that is still a ‘thing’, but carries data for one or more (typically but not always two) link-connections
  • a detached ‘field’ is a specific category of data-object that carries data, and a single link-connection that points to its parent ‘thing’
  • the ‘role’ for a data-object indicates its category in the sense of ‘thing’, ‘link’, ‘field’ etc – yet each still has essentially the same type of structure

– [Implementation] The data-structure could be implemented in several different ways, each of which implies different trade-offs. In my explorations so far, it looks like it could be implemented in SQL with something like 3-6 tables, or as a single ‘collection’ (table) in a NoSQL database such as MongoDB. I’ve also looked at graph-databases such as Neo4J, but I can’t see any real advantage as yet. Either way, we’d probably need a data-abstraction layer to enable the same data to be managed across different implementations that use different underlying data-technologies.

Although initial implementations could get away with something simpler, ultimately the data-object identifier does need to be globally-unique, which suggests a necessity for UUID or similar.

Data-exchange

All implementations must support a shared data-exchange format, such that data-objects can be shared across any other implementations and instances as appropriate.

This data-exchange-format must be able to support all data-object types – including editor-definitions, as described later. Within the data-structure, linking is done by reference to data-object-ID, hence recreation of links would not itself need to retain table-integrity within the data-exchange format: however, user-interfaces and queries should be able to operate without failing if links cannot be completed or link-targets are ‘missing’ (see ‘Access-control’ later).

This shared data-exchange format must be non-proprietary.

Other data-imports and exports should be supported, in a wide variety of forms. Where appropriate, imports and/or outputs may be in proprietary formats, or reference external data-objects that are stored only in closed and/or proprietary formats.

– [Discussion] Probable candidates for the common exchange-format would include XML, JSON or Extended JSON (to support binary-data). Given the constraint that data-objects may have any number or type of fields, XML may not be flexible enough for the purpose.

– [Implementation] At present, I’ve assumed a JSON implementation for the exchange-format. Import and export would be added as implementation-maturity develops – for example, export to email, HTML and/or PDF would be fairly high priorities. Import from JSON should be sufficient at first, as simple standalone code-utilities can do basic translation from other formats into JSON.

Access-control and linking

Ultimately, the platform should support access-control for view and/or edit at field level – not just primary-object level. Fields would typically inherit access-rights from their ‘parent’ at time of creation, but could be amended as appropriate.

Although access-rights should be attached to individual data-entities, the access-rights on data-objects should not be visible (other than in specific administration user-interfaces), or transferred in plain-text form across inter-system exchanges.

Users acquire access-rights via logon, and thence via assigned membership of groups. However, basic editors and data-storage should allow a default ‘public’ option for which no logon is required. (In part this is to allow ‘test-drives’ and/or fully-standalone operation.)

Queries should not return data-objects for which the user does not have view-rights. Where appropriate, these constraints should – as above – be supported both at whole-object and individual-field level. (Note that this is likely to mean that some links will seem ‘incomplete’.)

Editor user-interfaces should not allow edit on data-objects for which the user does not have edit-rights. (Only visible-objects – objects with access-rights to view – should have been returned from queries.)

Where access-controls limit visibility at field-level, editor user-interfaces should not show anything for items which are not visible – for example, field-captions should not be shown for those items.

Where appropriate, individual editors and the platform as a whole should support linking at whole-object and individual-field level. This should not be constrained by the underlying data-structure: for example, whether fields are embedded ‘in’ a parent, or as separate-but-associated data-objects.

– [Discussion] There will obviously need to be a lot of further discussion on the user/access-control model(s) to be used for this…

– [Implementation] Linking to individual-fields can be supported by some equivalent of the HTML ‘anchor’ concept. For access-control, I already have full source-code for a simple user-access mechanism from a PHP-based wiki that controlled access via user-groups down to page-level: this would probably be sufficient for initial experiments, and maybe even for first public tests.

Devices and interface-metaphors

The platform should be accessible and usable on any device across the entire toolset-ecosystem, from multi-screen ‘war-room’ setups right down to limited but web-capable feature-phones, and maybe even to dumb-phones via SMS and MMS. (Implementations and capabilities may and will be radically different across the toolset-ecosystem, but the platform and its underlying data should be available throughout.)

Access to large shared-repositories would typically require internet-access, but the platform will also support offline use on any appropriate device, subject to device capability and capacity.

The platform will support any appropriate type of user-interface metaphor or technology. (This also means that we need to provide conceptual ‘hooks’ to support user-interfaces that are at present barely experimental – such as 3D touch-based systems – or don’t yet exist at all.) In practice, this is somewhat of an aspirational goal, but given current technologies, the current ‘desired’-list should probably include:

  • SMS/MMS request/response (1D)
  • keyboard/mouse (2D)
  • touch/gestural (2D)
  • hands-free gestural (2D/3D)
  • side-menu / popup-menu / drag-and-drop etc

– [Discussion] This too will need a lot of discussion – mainly around prioritisation of interfaces and interface-choices.

– [Implementation] My current implementation-design is based on Javascript/HTML5 web-technologies, which could be auto-configured to support keyboard/mouse and touch/gestural interfaces. The Meteor platform that I’m experimenting with at present also supports offline-use, via a local simplified-MongoDB instance.

Editors

Editors are the means by which data-objects and links can be created and edited.

The platform will support any number and type of editors, which may be used on any type of device – subject to device capability and capacity – and act on underlying data in any appropriate way.

The platform will support ‘official’ or ‘controlled’ editors – those that require specific capabilities, or direct access to user-interface code, access-control code or underlying file-system. For security-reasons, the source-code for some – maybe most – of these should not be publicly available, and in some cases may be proprietary. Some of these editors – particularly the ‘official’ editors developed by the core-group and typically provide core-functionality – may expose script-interfaces that ‘unofficial’ or user-defined editors can call within their scripts, to access, in a controlled and safe way, functionality that would or should not otherwise be available.

‘Official’ editors would be installed via a package-load mechanism, which may support encrypted proprietary source-code for server-side only that is not directly accessible from the user-interface.

The platform will support user-defined or ‘unofficial’ editors, defined via standardised scripting-language, or, later, some form of graphic or drag-and-drop ‘editor-builder’ interface. By this means, users will be able to define editors for any type of model that they need, that can create, access and link data-objects in the ‘hologram’ – including, where appropriate, data-objects and links not previously created by that editor-type.

User-defined ‘unofficial’ editors would be installed via loading and compiling a specification defined in the scripting-language. These editors may be saved to and imported from an optionally-editable text-file. The file-format for specifications for user-defined editors shall be open and non-proprietary. Language-elements should support separate loading of graphics and any other non-text items needed by the editor-specification.

Each editor may create new data-objects in accordance with the editor’s own data-specification, including numbers, names, types and default-content for fields and tags. Editors may be configured to create data-objects of multiple-types (e.g. as per UML, BPMN, Archimate). New data-objects are tagged as ‘belonging’ to that editor, but may be manipulated, extended, linked-to etc by other editors as appropriate.

As earlier above, the underlying data-structure does not impose any inherent constraints on linking: any object may potentially be linked to any object, including itself (for recursive links). An editor may constrain visibility of links to those that it recognises as valid, but may not remove or delete any that it does not recognise as valid in its own terms. (This allows data-objects to be used in different editors and different views, in ‘controlled’ form as per the respective editor and view, yet still each retain its overall integrity as a ‘thing’ in its own right.)

An editor may constrain linking-rules for data-objects: a probable mechanism to do this is via matching of tags as defined in the editor’s data-specification for link-types and for primary data-objects. (This mechanism would enable the kind of formal-rigour required in e.g. UML, BPMN and Archimate.)

Within its specification, an editor may define field-labels etc for display on user-interfaces, including printing. (This would also provide a means to support internationalisation, as multiple label-sets could be defined within an editor-specification, or variant editor-specifications – same data-structure, different labels – used for different languages.) If fields are visible in terms of access-control, but the respective editor is not available to the user-interface (as may happen with imported data, or when the device is being used offline), the respective field may be shown in non-editable form on the user-interface, with the data-object field-name in place of the respective editor’s field-label.

– [Discussion] Again, a lot of discussion needed on this, though I’ve already roughed-out the core-elements for an initial version of the scripting-language for user-defined editors.

Some of the ‘official’ editors we’ll need, in approximate order of implementation:

  • basic form-based create, view and edit (provides base-functionality for all data-object creation and edit)
  • basic linker (provides base-functionality for all linking)
  • basic grouper (provides base-functionality for all ‘group’-types – sessions, themes, topics, project etc)
  • tabular-type edit (provides support for table-based interfaces and data)
  • basic graphic-base (provides base-functionality for scalable background-graphics and image-maps)
  • free-form sketcher (provides drawing capability, including user-selected image-maps)
  • builder/linker (provides functionality to support object-plus-link models such as Archimate)
  • media-capture and embed (uses links to device-hardware for audio, photo, video – embed by value [data] or by reference [URL etc])

The ‘unofficial’ or user-defined editors could be anything at all (subject to what the platform can support at the time, via the functionalities provided by the ‘official’ editors), creating new data-objects or amending existing ones in any way that makes sense for that person, and linking anything to anything else in any way that makes sense. That ‘anything-connects-with-anything-else’ is where the power of this platform will really reside.

Note also that the ‘equal-citizen’ concept means that although an editor can provide views that support a virtual concept of layers or suchlike (as in Archimate or Enterprise Canvas), there is, by explicit intent, no actual layering hardwired into the underlying data-structure. For more detail on the reasoning behind this design-decision, see the posts ‘On layers in enterprise-architecture‘ and ‘Unravelling the anatomy of Archimate‘.

[Implementation] I’ve roughed out all of the code for basic form-based, basic linker, basic grouper and tabular-edit and the first parts of basic-graphics, and how to define them in the scripting-language for editor-specifications. I have some ideas about Javascript libraries and the like that could handle the deeper parts of the basic-graphics and free-form sketcher. Beyond that, though, I’ll definitely need help!

Query, search and navigation

The platform shall support appropriate query, search and navigation.

For query, a query-builder will be needed that interacts with the currently-selected editor to define permitted parameters for the search. The query-language specification shall be open and non-proprietary – at least for queries and searches available to user-defined editors.

For search, the current query will be combined with the current access-control rights to define the object-set to be searched for and returned to the respective editor.

For navigation, clear distinction needs to be drawn between navigation within the object-set from a current query, versus SEO-type auto-navigation for indexing-bots. The former operates primarily or exclusively on the client; the latter in effect operates solely on the server for the data-repository.

– [Discussion] This is another theme that will need a lot of discussion – though there’s sufficient design already in place for basic-level querying and searching, and for Javascript-based navigation and routing on web-type user-interfaces.

[Implementation] As per immediately above, I’ve roughed fairly-detailed designs for that basic-level query, search and navigation.

Change-history

In principle, the platform should maintain a full change-history of all changes: other than specific admin-cleanup, nothing gets deleted. This should enable ‘replay’ (conceptually similar to ‘scrubbing’ in audio or video), to review past history and to permit cloning or branching at any point.

– [Discussion] This is fairly straightforward at a basic per-object level, but would soon gain real complexities with challenges around access-control, real-time multi-user and individual-object versus multi-object sessions.

– [Implementation] I’ve roughed out code and data-structures for a basic per-object level. Beyond that, I’ll definitely need help.

Okay, stop there for now, I guess.

Over to you for comment and advice, anyway, if you would? – many thanks!