Counting is hard (redux)
Diving into coronavirus numbers (or lack thereof)
As a resident of the UK I joined many people in tuning in to watch Boris Johnson talk about my country’s response covid-19 and the path out of lockdown. In my previous post on why counting is hard I talked about how we report numbers. Then I saw the way we’re presenting numbers and despaired a little. This post is about why the things we’re reporting don’t give an accurate picture of how we’re doing or incentivise the right things. The UK’s response can be used as a case study in a project update that doesn’t meet the needs of its audience.
First of all I want to re-acquaint you with Goodhart’s Law which says:
“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”
This is regularly paraphrased to:
“When a measure becomes a target, it ceases to be a good measure.
So, let’s take a look at the measures which the UK government are reporting on and dig into why they may not be meaningful. Then talk about some questions we can which might give greater clarity and lead to better metrics.
Number of new infections. Limited by your ability to test infected people at the right time. This gives you an indicator of how rapidly the virus is spreading so is useful to allow you to estimate the rate of growth. You also want an idea of the localised numbers of these because there will be substantial distribution within the rate of infection.
Number of tests carried out. Total number of tests carried out shows your cumulative testing capacity over time. It’s a vanity metric to show the number going up. You don’t make a decision based on this.
Number of deaths. This measure is useful on the face of it but becomes muddy quickly. We want to know how many people are dying from covid-19. But, every country counts this differently so you can’t compare and there are measurement errors because there is uncertainty over cause of death in non-hospital cases or where a test wasn’t administered. It also fails to take into account deaths which happened because of our response to the crisis such as delays in treatment for other conditions. Or deaths which didn’t happen such as ones averted due to the improvements in air quality. Or deaths which happen much later due to poverty caused by the lockdown.
Number of tests per day. A vanity metric. Easily gamed and doesn’t represent the effectiveness of the testing infrastructure. Are you testing the right people? Can you test them at the right time? Can you turnaround the tests quickly? We could be testing 1 million people a day and have testing that’s not fit for purpose.
Number of people in intensive care. This is useful because the primary aim of the initial response is preventing the healthcare system from being overwhelmed. However you also need to say what is your intensive care capacity is so you know how much spare capacity there is. This affects your ability to loosen restrictions or allow treatment for other conditions again (remember that a lot of patients got delayed or shunted from the system to make space).
All in all these numbers tell us very little about the response other than to inflate the egos of politicians or provide some level or reassurance to the general public. What we should be doing instead is working out which questions to ask and then having a nuanced discussion about the answers.
The following list is not exhaustive. I have yet to hear satisfactory answers to any of them so far — which leaves me with little confidence in the UK response. I’m sure that the advisory group has some or all of this information but the lack of transparency is worrying.
- How quickly is the virus spreading? We currently estimate R (rate of infection for this) and it’s estimated as between 0.5 and 0.9. This is a huge confidence interval as it tells us how quickly we will get things under control. 0.9 might mean it takes us another 24 weeks versus 8 weeks for 0.5 (from a very rough back of the envelope sum).
- Where is the virus spreading? R will vary considerably in different locations. Being able to break it down by location means you can target your response. Any effective strategy involves being able to respond quickly to new outbreaks and keep them localised.
- What capacity do we have to treat people? There was a big song and dance about ordering more ventilators at the start of this. And very little discussion about the mountains moved by NHS trusts up and down the country to free up and create capacity both in terms of equipment and skills. The greater our capacity — the higher a rate of infection we can sustain where we can treat people effectively.
- Who has had the virus? This might indicate your ability to handle future infections on the assumption that you are less likely to catch the virus a second time (note this is unproven). If 50% of your population has had the virus then this would limit future outbreaks by providing a break against infection. This is why there was/is so much hope held out for antibody testing. Being able to say you’ve been infected gives more options in managing the virus.
- How many people are we treating? As a trailing indicator this gives confidence in our assessment of R. When compared against local and national capacity it also gives confidence in our ability to get the best outcomes for people who do get sick.
- Who is dying because of coronavirus? This is not just about people we’ve tested who then die in hospital. Or even people dying outside of hospital where covid-19 is suspected. It’s people who will die because of the pandemic either directly or indirectly. We know cancer patients are getting delays to treatment. We’ve seen heartbreaking stories of newborns. It’s knowing if there is a section of society that is more vulnerable so we can address it directly. This might be an age group, it might be a socio economic group or it might be people in a particular job (and many other things).
- Who is not dying because of coronavirus? We know air pollution is down. One positive of this is that there will be a strong set of data across the world into how the pollution we generate is killing us. Which, we can hope, will lead to more informed policies on the environment. If road travel is down 50% do we also see a corresponding drop in traffic fatalities? This question offsets the previous one because it’s possible, however unlikely, that there is a net benefit from something or that the benefits cancel out some of the negatives.
- How many people are going to die later because of our response? If we go into recession — who is going to die because of this? How many patients who can’t get treatment will now die because of this?
- Who are the people we should be testing? This is about contact tracing and also knowing the at-risk members of society. Are we actually testing these people?
- What frequency should we be testing them at? Are we testing them enough? Given the high amount of asymptomatic cases there are people who need to be tested a lot. Which leads to…
- How quickly can we turn around test results for those people? How quickly can we isolate infected individuals? If it takes us two days to get a positive response back from the lab then that’s two days an infected individual can be exposing other people to infection.
- Who is unable to pay for rent, food or necessities? This is a measure of whether the financial aid is sufficient or targeted correctly. It’s great announcing billions in loans for businesses and furlough but it doesn’t tell us whether these measures are effective.
- Who has become unemployed because of lockdown? We should identify people earlier so we can plan for how the future economy might support them getting back into employment. Also, who is employed because of the lockdown? These people might indicate sectors you want to look at as part of stimulating recovery.
- Who is at risk of unemployment because of lockdown? As time progresses more and more people are at risk of this. Particularly if furlough is scaled back or not extended. Part of the calculus of lockdown is also knowing what impact this will have over time. Some politicians are bandying about the phrase “the cure can’t be worse than the illness” to describe this. We don’t have meaningful data so this topic becomes politically polarised.
- Which businesses are at risk? Should we support them? We know that aviation and travel are going to be hit hard here. There are going to be companies which emerge from this without a viable business model and some which emerge with stronger models. Some companies might survive with support and we should support them. Others won’t and we should accept this and find other ways of supporting those affected.
If you know the questions you want to answer you get a better idea for the things you need to measure (and their limitations). Right now, the data being reported isn’t fit for purpose and doesn’t foster a meaningful discussion.
I’ve skipped the plan itself, which seems to mean — “keep doing what you’re doing with a new slogan which makes it look like we’re making progress” while skipping detail on the key pillars which will make or break the response: testing and contact tracing.
Coming soon: why technology isn’t a silver bullet for contact tracing
