14 days ago

The weight of the past

Link: http://weblog.tetradian.com/2020/09/09/the-weight-of-the-past/

What’s the weight of the past?

For me, right now, it’s about two tons.

Literally.

I moved back to Australia in mid-March of this year. Courtesy of the pandemic-lockdown, every part of my initial plans for the move were shredded, and I’ve spent several months wandering around in AirBnB limbo, living out of a set of suitcases. Last month, though, I at last found somewhere more long-term to live, and been doing the move-in ever since. It also means that, for the first time in well over a decade, I have all of my stuff in one place. And it’s been frightening to realise just how much of it there is… – the sheer weight of the past, in that all too literal sense.

Unpacking took a lot of time, because there were more than a hundred boxes of stuff. Books, books and more books: well over a thousand of them. That’s the better part of half a ton in weight, right there. And yeah, I’ve had to buy more bookcases: twelve in all so far, with yet more of them maybe on the way…

 

(Another thousand books in unsold stock still out in the shed: another half-ton there, too…)

And papers, papers and more papers, from three, four, five decades of work in at least four distinct main disciplines and a broad swathe of others as well. I’ve been a published author for more than forty years, with literally dozens of books to my name now: there’s all the research and paperwork that went into each of those, and for at least another dozen more that never quite made it to print. There’s design notes and code-listings from projects I worked on in the 1970s and 1980s; more code and requirements-definitions and database-designs from projects in the 1990s, then enterprise-architecture and more in the 2000s and 2010s, along with cuttings and applicable standards and more for each of those projects. So far that’s filled about a third of the bookcases, four filing-cabinets and half a dozen file-sorters, and there’s still a whole pile of boxes of research-papers and suchlike that have yet to be properly unpacked…

And computers, of course, at least a dozen of those, of varying vintages, each with their own software and storage-requirements that go all the way back to the days of floppy disks, and with all the accessories and printers and more that go with each of them. All of which need to be stored somewhere, too.

Plus spare stationery and all the other accoutrements of a working office. Boxes and boxes of that, all now gathered together from all those places I’ve lived, in four different countries at least.

Several boxes of clothes – most of which I wouldn’t want to wear any more, and probably half of which no longer fit in any case. A box full of shoes and boots – half of whose soles seem to have disintegrated in storage, though I’ve no idea how. Far more bedlinen than I remember – most of which I certainly don’t need, given that I’ve lived largely alone for almost all of my adult life. And enough kitchenware and crockery for three whole households, at a guess – some of which stuff I had no idea that I even owned.

Definitely overdue for a cull. On all of it.

Which, however, takes a lot of work. A childhood memory of my mother sitting in her chair in the living-room, night after night, culling the content of patients’ medical-record packets, stripping out old appointment-letters and the like, and replacing them with handwritten notes that summarised a dozen or more sheets down to one. Every night there would be a huge pile of redundant paper to go onto the next bonfire (no shredders available in those days, but also no real environmental regulations). All this to do, year after year, for several thousand patients. A lot of work – all of which needed careful context-specific knowledge to know what could be safely culled, and what couldn’t. All to clean up the weight of the past.

In the IT business, this is part of what’s known as technical debt. As usual, too many folks seem to think it’s solely about data, or sometimes source-code as well: but in reality, it’s about everything that might otherwise be described as ‘legacy‘ or suchlike. It’s the weight of the past, weighing us down, making it hard to move fast – or at all – when we need to do so.

Having that weight of the past is not wrong as such: it’s more that failing to manage it appropriately can often turn out to be a real mistake. And failure to manage it arises from any or all of three key errors: a literal ‘ignore-ance’ of the problem and its very real risks; a ‘sweep it under the rug, if I leave it long enough then someone else will deal with it’ mentality; and a generalised fear of loss – particularly perceived loss of future choice. We need to face those errors if we’re to get anywhere with this.

And yeah, I’m painfully aware I’m all too much at fault on this too. In my own case it’s perhaps the third of those errors that hits me most: as a generalist whose main selling-point is an ability to connect things together in unexpected ways, I have a natural tendency to be something of a squirrel, holding onto every idea and research-record in the often-validated belief that it might be essential for something in some unexpected future. But the bleak reality is that, at my age, I’m rapidly running out of future – and the way I’d worked in the past won’t serve me well for the work I need to do next. Some painful decisions about priorities coming up for me right now: yet once those become clear, I’ll need to do that cull. No doubt at all that I’ll need to cut that weight of the past in half, in the next few months, and then keep cutting, keep culling, until I can freely move again in whatever direction I need to go.

And it’s not just in a personal sense that this applies – if anything, this is often even worse in organisations. If we don’t tackle it in a systematic way, across the organisation as a whole, over all applicable periods of time, that weight of the past will eventually cripple the organisation’s ability to move, to respond to change.

Which means that, yes, this is definitely a concern for enterprise-architecture, because we need explicit architectures to support the management of our organisation’s weight of the past. Which in turn suggests a checklist of practical concerns for this:

— What are your organisation’s strategic priorities, to guide decisions on what you’ll need to keep, versus what you can safely drop? These would come from longer-term decisions on strategy – which means that a short-termist ‘make-it-up-as-you-go-along’ version of so-called ‘agile’ risks being a quick way to increase the weight of the past, until the organisation ceases to be agile at all. A stable anchor such as visioning will be all but essential here, to help clarify what you do and don’t need to keep.

— What decision-criteria arise from those strategic priorities? – how would you identify what matters, and what doesn’t? In my case, for example, there’s quite a lot amongst the research-notes that wouldn’t matter for my own future as such, but might well be important for others who tread the same paths and need to learn from the inputs and choices and mistakes that I’d made along the way. In that sense, ‘legacy’ is as much (if not more) about others than ourselves… – and that’s a key concern that needs to guide some of our own decision-making here.

— What domain-knowledge do you need, to make the right choices about what to keep and what to drop or pass on to others? Again, this can be a lot trickier that it might seem at first glance. I remember one major engineering project where we were not entirely joking when we said we needed to hold a séance to commune with the dead: all the reports from a key previous test, thirty years earlier, had been so laundered for political reasons that they were almost worse than useless, and the only people who would have known what had actually happened were now long since gone. In a similar vein, almost exactly the same expertise that went into resolving the ‘Y2K problem‘ will be needed again almost forty years later, for the ‘Unix Y2K problem‘ in 2038, when system-dates in unpatched Unix systems will ‘go negative’ to restart back in late 1901. We need to maintain domain-knowledge about everything that’s in use in our organisations, for at least as long as it’s in use, and often a fair while longer than that. And that includes all the ‘tacit-knowledge’ – in-person, undocumented and often undocumentable as such – that only arises from real-world experience with each type of item and context. Not trivial at all to do – but if we ignore it, or get it wrong, the consequences could well be lethal, in more ways than one.

— If you’re aiming to pass some of this material on to others, who would be the appropriate recipients? Do they have the requisite domain-knowledge, and experience of the respective contexts, to make the best use of that material? How would you, and they, assess and verify that they fit well with those requirements? (I talked about some aspects of this in the ‘Protect your digital legacy‘ post a few months back – but the same principles apply to all other forms of ‘legacy’ as well.) If we don’t do this assessment and validation, all we’d be doing is loading up someone else’s ‘weight of the past’, to no real advantage to anyone at all.

— For the material you’re aiming to keep, what storage-format should be used for each item? The trade-offs here are not as simple as they might seem. Everything physical will need physical storage too – a fact that ultimately applies to everything ‘digital’, as I know to my cost, given the box-full of portable-hard-drives on the shelf-behind me, sitting beside another box-full of old CD- and DVD-based backups. And it not only costs significant effort and more to ‘digitise’ everything, but even the digital-records themselves can be unreliable, especially over the longer-term – as I again have had to learn the hard way, with stacks of files in file-formats and even disk-formats that my systems can no longer read. And whilst digital-records can be indexed and duplicated easily, there are real advantages to physical-records too: paper and even microfiche can be read without any software needed at all, and can survive for centuries in a way that digital records almost certainly can’t.

So what’s the weight of the past, for you, for your organisation? How much is it slowing you down, in the moves that you want and need to make? And what will you do to make that weight more manageable, for now, and for your future? Seems to me that, as enterprise-architects and more, these are questions that we definitely need to address.