9 years, 12 days ago

There’s no short-cut to experience

Link: http://weblog.tetradian.com/2012/04/30/no-shortcut-to-experience/?utm_source=rss&utm_medium=rss&utm_campaign=no-shortcut-to-experience

At least he was open about it, I guess. “Tell you what I’ll do”, he says to my colleague here in Guatemala, “I’ll find you a client, then I’ll sit in, learn everything you do, and then I’ll apply it in my own business. How does that sound to you?”

Uh, no. Not a good idea. Not just because it’s a really bad deal from our perspective, but much more that Reality Department really doesn’t work that way: there’s no short-cut to experience.

Yes, it all looks simple enough – in fact that’s the whole point. A lot of simple visual summaries, and surprisingly simple-seeming methods, too. Yet it only looks simple because we’ve been through a heck of lot of hard work to make it that way. Hard-won experience, won the hard way through years and years of practice in many, many different contexts, getting it ‘wrong’ time and time again, in many, many different ways in order to get it right.

The real trap is that these simple-seeming ideas and methods aren’t simple rules, prepackaged sense-making and decision-making that will always work in every context. These are simple principles, simple guidelines, the kind of easy-to-memorise information that helps decision-making in real-time, in circumstances that are subtly different every time. (See my SCAN posts for more on these distinctions.) If you try to use them as ‘rules’ for inherently-uncertain contexts, without understanding why those principles apply, or how they need to be tweaked every time to match each different context, you’re going to be in real trouble – along with everyone else around you. Not a good idea…

The same often applies even to things that really are ’rules’. Take that example of perhaps the greatest simplification ever made: e=mc2. All the core information you need to build a nuclear power-station is right there in that equation: but there’s a heck of a long way – a heck of a lot of engineering-experience – to go from that one equation to building a nuclear-power station that actually works.

Same with everything else, really: simplification is essential, but can also be deceptive – especially when people mistake ‘simple’ for ‘easy’…

Which is also why I get a bit hot-under-the-collar about the current proliferation of ‘certification-schemes’ in enterprise-architecture and elsewhere. Some of them are genuinely valuable, but others – to be blunt – are little better than money-spinning scams, in terms of the actual value that they (don’t) deliver. And the crucial distinction revolves around the role and recognition of experience.

For example, the TOGAF Foundation and Archimate Foundation certifications have real value. They verify that the respective person has a credible command of the terminology and language – a requirement that matters a lot for communication across a dispersed and disparate team.

Likewise the ATAC certifications should have real value, because each explicitly tests practical experience in the respective area.

But unless they’ve changed it in the past year or so, the full TOGAF certification is delivered through the absurdly-inappropriate mechanism of a multiple-choice test. And to my mind, that’s not merely useless, it’s actually worse than useless, because it’s exactly how not to test for the kind of experience that that type of competence requires. (When I did the TOGAF 8 exam some years back, I almost failed because I answered several key questions correctly in terms of real-world experience, rather than the theory-based wrong-assumptions that the test thought were ‘right’.) The result of that kind of pseudo-test is a bevy of people who can wave a certificate around, but have no idea how to do the work in any real-world context.

A good training-course can make all the difference, and the better training-providers do take up some of the slack here. (I’ll wave a flag at this point for John Polgreen at Architecting The Enterprise, who’s been doing sterling work for years on adapting TOGAF for the US-government context.) Yet none of that competence carries through anywhere into the actual test: so unless we know each of the training-providers, we have no way to tell whether a candidate does actually know what they’re doing, or merely that they have a piece of paper to prove that they know just enough to get into real trouble, but not enough to get out of it again.

In effect, right now, the full TOGAF certification is of less real-world value than the Foundation certification – which is both bizarre and sadly pointless. And I’ll hasten to add that I’m using TOGAF here merely as one example amongst many: it may well be that most of the so-called ‘certifications’ in this field are even more meaningless than that. And the results can be seen everywhere in the trade…

In short, it’s a mess.

What we need to be testing for is genuine understanding of a context, and the ability to adapt for uniqueness. And that calls for much, much more than can be covered in a crude multiple-choice test delivered through a mindless machine. Sure, that kind of test is cheap, and relatively easy to administer: but it’s also all but meaningless for anything than foundation-level rote-knowledge. It really does take years of painful practice to develop the experience needed to do this work well: and if this trade is to gain the credibility that it needs, we need to stop pretending that we don’t need to test for that experience.

Time to re-think how we do this, and how we respect this, too: there’s no short-cut to experience.