Why we fail—repeatedly

AND HOW WE CAN FIX IT

Image is from Rare Historical Photos

First, a huge ‘Thank you!’ to everyone who voted.^⌘ The results are clear: just 10% of you want to ‘fix the journals’, and those who are interested in health IT solutions outnumber those who wanted me simply to slug on, by a whisker. Your wish is granted.

𝔐ostly, we don’t learn from history. We’d prefer to do one of three things: ignore the past, shoot the messenger, or rewrite it to make things look more favourable. Way before Photoshop, Stalin had a bunch of tame artists who could airbrush photographs of his smallpox-scarred face and make it glamorous. He was also quite good at disappearing his enemies—together with all evidence they had even existed. Modern politicians are prone to similar tricks. Especially when it comes to healthcare.

We already know how to fix health. Mostly. We know that the main causes of premature illness, misery and death are now non-communicable diseases. We also know that these are not primarily due to personal failings on the part of people. Ordinary citizens have not suddenly succumbed to a tidal wave of moral obloquy; they are instead vulnerable to advertising, availability and price. As Deming observed, most of our behaviours are determined by the system around us.

Health is primarily about the flow of commodities into communities. Harmful commodities like alcohol and other drugs of abuse, salt, tobacco, and high-energy density food; and beneficial commodities like fresh vegetables, untainted water and even pollution-free air. We’ve already explored this.^⌘ Okay, the odd pandemic will kill tens of millions (as COVID-19 recently did¹), and at least in theory, we are now better at getting this right. With the recent pandemic, a handful of countries like mine even got the practical bits right, too.²

Managing expectations

But let’s not be silly here. Most politicians and consequently most countries continue to stuff it up. Some more so than others, as shown in the above graph. They punt ‘health care’ as if ‘health care practitioners’ can fix things, and berate them for the problems politicians should be fixing. We’ve covered this before, too.^⌘ They set targets and use ‘traffic light’ and ‘league table’^⌘ measures.

But even in the limited domain of ‘Medicine’, we have scope for improvement. As I’ve observed before,^⌘ Good Medicine is simply what we all want for us and the people we love: the right management delivered with care and attention by the right people at the right time. Without fear or favour. In that post, I described five obstacles: trying to fix the wrong ‘problems’, the massive inertia of Medicine, funding, trade-offs and doctors themselves.³

At this point it’s important to note that despite the expectations of many, the solution to the problems we can solve is not ‘computerisation’. Software tools can be useful, but they are just tools. Again, we already know the answers. They involve the process re-engineering described by Deming,^⌘ combined with the statistical tools originated by Shewhart. Tools that allow us to interpret processes as they change over time.^⌘ We need to think and plan in terms of long-term strategy; we also need to get the design right. This is achievable. Yet it’s been repeatedly stuffed up by pretty much everyone who has tried. Why? Before we propose solutions, it seems wise to examine previous expensive failures, work out why they failed, and decide how we can do better. Then only should we ask how computerisation can improve the quality of our success. In the following, I’ll use one specific but massive failure, but others show very similar patterns. We mostly don’t learn from past mistakes.

Past failures

One of my favourite software fuck-ups has nothing to do with Medicine. No, it’s not the very first Ariane 5, a $370 million firework display caused by trying to stuff a 64-bit floating point value into a 16-bit integer. Nor is it the Toyota ‘unintended rapid acceleration’ problem that affected at least 2262 vehicles between 1999 and 2010, and likely caused over 100 deaths—due to ‘spaghetti software’ (The image above is one such crash, taken from that PDF). These were spectacular failures, but they don’t adequately embody the sheer magnitude of how big plans can go wrong. At the time, my favourite example of a massive, strategic failure was described like this:

It may have been the greatest failure in the history of organized work.

In 1981, when air traffic controllers in the US went on strike to demand fair working conditions, Ronald Reagan fired the lot of them. Subsequently, the Federal Aviation Administration decided to drastically curtail the controllers’ power by putting in advanced software (the AAS) that would do a lot of the work. IBM put their best thousand programmers onto the job. In 1994, when they realised costs and timelines had blown out, the FAA cancelled the program, throwing away at least $1.5 billion. It is said that the entire effort did not produce a workable line of code. Most of the dysfunction there hasn’t even been documented, but the remainder is interesting to read.⁴

But this is as nothing! In September 2011, the UK’s National Programme for IT (NPfIT, under ‘NHS Connecting for Health’ aka NHS CFH) was dismantled, flushing at least £14 billion down the gurgler. That’s more than twice the initial budget from 2002.⁵ The idea here was to build an integrated patient electronic record around a national “data spine”. Some of the biggest players were involved: Accenture (think ‘Andersen Consulting’), Fujitsu (of British Post Office fame), BT, CSC and Atos Origin.

How did it fail? It seems that everything was badly done: bad planning with an appalling lack of transparency, and a dysfunctional centrepiece of an electronic health record system called Lorenzo.⁶ It appears that Lorenzo continued to plague the NHS even after NPfIT was cancelled:

The deployment of Lorenzo has been blamed for duplicating patient records, losing patient records, not registering when patients have had surgery, booking in the wrong patients for surgery, and causing problems with booking and scheduling more generally.

This is interesting, because Lorenzo was meant to avoid this sort of thing. The authors deliberately wrote the software based on ‘Service-Oriented Architecture’ (SOA), with modular delivery and a layer of ‘shared services’ at its heart.

Why did Lorenzo fail?

I suspect that if you sit any bunch of moderately capable programmers around a local server and give them complex specifications, powerful machines and punitive deadlines, they will produce code that works well locally, seems conformant, and then fails intermittently and spectacularly when exposed to reality. They will have failed to test for and accommodate edge cases, slow and under-powered machines, temperamental connections, and inventive users.

It seems that all of this—and more—characterised the roll-out of Lorenzo. Strong opinions have been expressed about the failure. Their SOA demanded multiple handshakes between services that caused extreme and frustrating delays. We’re talking tens of seconds. They used HL7 version 3. We’ll devote a whole post to this monstrosity. They used the CDA (another post). The whole thing was dependent on early versions of Microsoft’s Internet Explorer, a browser that is now thankfully dead. When a service went down, there was no built in ability to work locally and synchronise later. Clinicians were not consulted. Let’s (please) not talk about the user interface (UI). And whatever you do, don’t mention how the finances were run. Just don’t.

In summary, the superficial problem seems to have been one of excessive software complexity, with a lot of enthusiastic bad design layered over this. But if patients were being lost, misidentified and linked to the wrong records, the malaise goes a lot deeper. We’re talking “failures of relational integrity”. If users found it unusable, to whom were the designers of the UI talking? And deeper still, were there more fundamental strategic flaws?

Let’s step back

What can we learn here? I’d suggest that there is a whole long list of principles just crying out to be noticed. They are not unique to health care, but they bite health care particularly hard if you ignore them, for two main reasons:

1. Health care is not just complex, but complex in a particularly bad way.

2. Health care has a lot of built in dysfunction.

The two are bound together. So what are those complexities, and why are they ‘bad’? Where’s the dysfunction?

People have described health care as a ‘wicked problem’ because of the complexity of its processes, and the non-linearities that are built in. But it’s worse than this. If you look back over decades, not only have new diseases continued to pop up, but our characterisation of old ones has changed, often dramatically. Treatments that were ‘best practice’ last year may be considered harmful today. In addition, in my recent series of 8 posts,^⌘ we’ve seen that the evidence base on which change are made is variable and subject to the influence of players who do not have the patient’s best interests at heart.

In the face of all of this, clinicians struggle to keep up, and they themselves have their own biases that are difficult to tease out. Entire medical institutions have built-in dysfunction that is resistant to change. Hospitals are dangerous places, where the holes line up and harm happens. We’ve already ventured into the Palace of Iatrogenesis.^⌘

So is this all even fixable? I’ll say an emphatic “Yes”. In fact, in the following posts, I’m going to work through the problems and solutions systematically. It all comes down to …

Getting the basics right

As suggested by the above examination, we have a single enemy. What is its name? Its name is:

  Complexity

There are simple actual solutions; but then there is a host of simplistic solutions that are all wrong. Pretty much all of the time, we therefore need to ‘Science it’, as we already know how.^⌘ When it comes to the very basics, a lot of the spadework has already been done. The problem here is that people have often misunderstood what is needed, ignored it, or not even thought about it. I think that sometimes it’s not just “failing to learn from history”, it’s failing to grasp the very basics.

I am also absolutely certain that some of what I write will be read by seasoned programmers (perhaps they started programming before I did in 1978), people who have more than my 20,000 or so hours under the belt, and it will make them furious. If it does, please do two things. First, go back and ask “Can this possibly be right—might I have missed something?” Second, read my rationale and write a criticism. I will try to justify my opinions, so hold me to this.

Here’s the broad layout of my next several posts:

1. A health IT strategy—the big picture. Joining things up. Representational adequacy. Monoliths, federation and layers. FHIR.

2. Ontologies revisited. SNOMED-ish. Normalisation, again.

3. Nuts & bolts I: Timestamps, locations and temporal extension.

4. Nuts & bolts II: People.

5. Nuts & bolts III: Anatomy, drugs, devices, organisms, things.

6. Nuts & bolts IV: Problems, conditions and whatnot.

7. Nuts & bolts V: Just pure RESULTS.

8. Evidence and causal reasoning. What did you do, and why? Bayes, again.

9. Provenance and its abuse. And where’s the AI in this?

10. Pulling it all together. Arguments about complexity—again! The poor, neglected user. A final prediction.

Some of those topics may seem a bit broad or a bit vague. That’s deliberate, but if you feel there’s something that I’ve obviously missed out, or something that you really want covered, feel free to make a comment below.

The first topic we’ll cover is that big, strategic picture.

My 2c, Dr Jo.

^{⌘ This symbol is used to indicate posts where I’ve discussed the flagged topic in more detail.}

1. Likely about 25 – 30 million extra deaths.

2. Saving the lives of 1:250 citizens is pretty impressive. I see our current government is busy cutting pandemic preparedness.

3. Other clinicians can be a bit prickly at times, but surely none so much as us doctors. Ohno.

4. It’s not abundantly evident that the FAA learned much from this debacle.

5. One of the measures of failure is that they don’t even know how much they lost. Estimates range from £10 billion to over £20 billion. Try that with your household budget.

6. Initially written in India by the UK company iSOFT, then flicked off to the Australian IBA Health, followed by disposal to CSC in 2011, and merged with hP’s enterprise services division in 2017 to form DXC.