Counting is hard (redux)

Diving into coronavirus numbers (or lack thereof)

Photo by Leonel Fernandez on Unsplash

As a resident of the UK I joined many people in tuning in to watch Boris Johnson talk about my country’s response covid-19 and the path out of lockdown. In my previous post on why counting is hard I talked about how we report numbers. Then I saw the way we’re presenting numbers and despaired a little. This post is about why the things we’re reporting don’t give an accurate picture of how we’re doing or incentivise the right things. The UK’s response can be used as a case study in a project update that doesn’t meet the needs of its audience.

First of all I want to re-acquaint you with Goodhart’s Law which says:
“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”
This is regularly paraphrased to:
“When a measure becomes a target, it ceases to be a good measure.

So, let’s take a look at the measures which the UK government are reporting on and dig into why they may not be meaningful. Then talk about some questions we can which might give greater clarity and lead to better metrics.

Number of new infections. Limited by your ability to test infected people at the right time. This gives you an indicator of how rapidly the virus is spreading so is useful to allow you to estimate the rate of growth. You also want an idea of the localised numbers of these because there will be substantial distribution within the rate of infection.

Number of tests carried out. Total number of tests carried out shows your cumulative testing capacity over time. It’s a vanity metric to show the number going up. You don’t make a decision based on this.

Number of deaths. This measure is useful on the face of it but becomes muddy quickly. We want to know how many people are dying from covid-19. But, every country counts this differently so you can’t compare and there are measurement errors because there is uncertainty over cause of death in non-hospital cases or where a test wasn’t administered. It also fails to take into account deaths which happened because of our response to the crisis such as delays in treatment for other conditions. Or deaths which didn’t happen such as ones averted due to the improvements in air quality. Or deaths which happen much later due to poverty caused by the lockdown.

Number of tests per day. A vanity metric. Easily gamed and doesn’t represent the effectiveness of the testing infrastructure. Are you testing the right people? Can you test them at the right time? Can you turnaround the tests quickly? We could be testing 1 million people a day and have testing that’s not fit for purpose.

Number of people in intensive care. This is useful because the primary aim of the initial response is preventing the healthcare system from being overwhelmed. However you also need to say what is your intensive care capacity is so you know how much spare capacity there is. This affects your ability to loosen restrictions or allow treatment for other conditions again (remember that a lot of patients got delayed or shunted from the system to make space).

All in all these numbers tell us very little about the response other than to inflate the egos of politicians or provide some level or reassurance to the general public. What we should be doing instead is working out which questions to ask and then having a nuanced discussion about the answers.

The following list is not exhaustive. I have yet to hear satisfactory answers to any of them so far — which leaves me with little confidence in the UK response. I’m sure that the advisory group has some or all of this information but the lack of transparency is worrying.

If you know the questions you want to answer you get a better idea for the things you need to measure (and their limitations). Right now, the data being reported isn’t fit for purpose and doesn’t foster a meaningful discussion.

I’ve skipped the plan itself, which seems to mean — “keep doing what you’re doing with a new slogan which makes it look like we’re making progress” while skipping detail on the key pillars which will make or break the response: testing and contact tracing.

Coming soon: why technology isn’t a silver bullet for contact tracing

Ex-Google, ex-Netscape, ex-Skyscanner. Interested in solving complex problems without complexity and self sustaining self improving organisations.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store