A screen that is 95 percent accurate is wrong about half the time
ยท 7 min read
Accuracy is the number everyone puts in the abstract and it is close to useless for a screening tool. Not misleading in a subtle way. Useless in a way that changes whether the tool should be deployed at all.
This is old statistics and none of it is my idea. I am writing it down because I keep reading screening papers that report accuracy as the headline, and because I had to work through the consequences properly for my own trial before I understood how much it mattered.
The arithmetic
Take a condition present in 5 percent of the people you screen. That is a realistic figure for early cognitive impairment in an unselected older outpatient population, and the exact value does not matter for the argument.
Now give yourself a genuinely good model. 95 percent sensitivity, so it catches 95 of every 100 people who have the condition. 95 percent specificity, so it correctly clears 95 of every 100 who do not. In a paper that is a strong result and you would be pleased with it.
Screen 1000 people. 50 have the condition and 950 do not.
Of the 50, you catch about 48. Of the 950, you wrongly flag 5 percent, which is about 47. So you hand back 95 positive results, and 47 of them are wrong. Slightly under half your positives are people with nothing wrong with them.
Same model. Same 95 percent. The chance that a positive result means anything is a coin flip, and nothing about the model caused that.
Drop the prevalence to 1 percent, which is what you get if you screen a general adult population instead of an older one, and it gets worse fast. 10 true cases, 10 detected, and about 50 false alarms. Now five out of every six positives are wrong, from an unchanged model.
Why this is a design problem and not a footnote
The number that matters to the person holding the result is the positive predictive value, and it is not a property of your model. It is a property of your model crossed with the population you point it at.
Which means a screening tool does not have an accuracy. It has an accuracy per deployment. Validate on a balanced test set with equal positives and negatives, and you have measured performance in a population that does not exist anywhere. That balanced number will always flatter you, and the gap between it and the clinic is not small.
So the operating point cannot be chosen by maximising a metric on a test set. It has to be chosen against the prevalence where the thing will actually run, and if you deploy the same tool in two settings with different prevalence you arguably need two thresholds.
Which error you would rather make
Once you accept that you are trading one error for the other, the question becomes which one costs more, and for early screening the asymmetry is stark.
A false positive costs one specialist appointment and some weeks of avoidable worry. That is a real cost and I do not want to be glib about it, particularly where a specialist visit means travel and lost wages for a family.
A false negative costs the window. For progressive conditions the entire argument for screening early is that early is when intervention still does something. Miss it and you have not delayed a diagnosis, you have removed the reason the screen existed.
So for my own work I set the threshold to accept more false positives than a balanced metric would pick. That decision makes the accuracy number look worse and it is the right decision. A screen is a filter in front of a specialist, not a verdict, and it should be tuned like a filter.
Which means the referral path is part of the model
This is the part I did not appreciate at first. If half your positives are false, the value of the whole system depends on what happens after a positive.
If a positive routes into a cheap, fast confirmatory step, then a high false positive rate is completely fine. You built a triage layer, and triage layers are supposed to over include. If a positive routes straight into an expensive scan, a long wait, or a conversation where somebody hears the word dementia, then the same false positive rate is doing real harm.
So the confirmatory pathway is not somebody else's problem downstream of my model. It sets the threshold. You cannot pick an operating point without knowing what a positive triggers, which means you cannot finish the model before you know how the clinic is organised.
What I think should be reported
Sensitivity and specificity, separately, with confidence intervals, because they are the properties that actually belong to the model.
Then positive predictive value computed at the prevalence of the intended deployment population, with that prevalence stated and sourced. If you do not know it, say so, and give a range.
Then the expected count of false positives per thousand screened, in plain numbers, because that is the sentence a clinical partner can actually reason about. Forty seven unnecessary referrals per thousand is a thing a department can plan around. A number like 0.95 is not.
And leave accuracy out of the abstract. On any imbalanced problem it mostly reports the prevalence back at you. A model that flags nobody at all scores 95 percent accuracy on a 5 percent condition, which should be enough to retire the metric.
Where I am with this
Both arms of our trial are close to complete and I am not going to quote numbers from an unpublished study, including ones that would look good. What I will say is that the balanced test set figure and the figure that matters in the department were not close to each other, and that the gap was the most useful thing I learned from the whole exercise.
If you are building a screening tool, work the base rate before you tune anything. It will tell you whether you are building a triage step or claiming a diagnosis, and those are different products with different obligations.
Related: a clinician who cannot see why will not act, on the other half of why a good score is not enough.