A clinician who cannot see why will not act
ยท 8 min read
Early in the dementia screening work I had a model that performed better than the one we ended up deploying. I threw it away. This is the post I wish someone had written for me before I built it.
The setup
The screening battery collects four things in one sitting on a phone. A spoken passage. Video of the face during prompted expression. A fine motor tapping task. A short cognitive test. Each of those carries some signal about early cognitive decline, and none of them carries enough on its own.
The obvious move is to concatenate. Pull features from every stream, stack them into one long vector, train a single model on the whole thing. This is early fusion, and it usually wins on paper, because the model gets to learn interactions between modalities directly. If a particular vocal hesitation only matters when it co-occurs with a particular tapping irregularity, early fusion can find that. Scoring each stream separately cannot.
So I built that. It was better. And it was unusable.
What happened in the room
We sat with clinicians and walked through outputs. The model produced a risk score. A doctor looked at a case, saw an elevated score, and asked the only question that matters in that setting.
Why does it think that?
With a concatenated feature vector the honest answer is a shrug dressed up in attribution methods. I could produce saliency over the input vector. I could run the usual attribution tooling and get numbers per feature. But the features were engineered, cross modal, and not individually meaningful to a person who thinks in terms of this patient's speech is off or this patient's motor control is off.
And here is the part that took me longer to accept than it should have. That doctor was not being obstructive. They were being correct. A screening result is not a verdict. It is a request to a busy clinician to spend a scarce resource, which is their attention and the patient's follow up appointment. Nobody spends that on a number with no story attached.
Late fusion, and what it costs
So we restructured. Each modality gets scored independently. Speech produces a speech score. Motor produces a motor score. The combination happens at the end, over interpretable per modality outputs rather than raw features.
I want to be straight about the cost, because a lot of writing on explainability pretends there is no trade. There is. Late fusion gave up some performance. Cross modal interaction effects are real and my architecture can no longer represent most of them.
What it bought was a sentence. This score is driven mainly by speech timing and motor variability, with the cognitive battery in normal range. A clinician can do something with that. They can agree, disagree, or say the patient has a stammer and the speech signal is misleading, which is information I would never have gotten from a better model that could not be interrogated.
The framing I use now
Accuracy is a property of a model. Adoption is a property of a model plus the person holding it. Optimising the first while ignoring the second produces research that never leaves the paper, which is most medical AI.
So I stopped treating explainability as a research topic layered on afterwards and started treating it as an architectural constraint, alongside latency and memory. If a design cannot produce a per signal story, it is not a candidate, however well it scores. That rules things out early, which is the point.
Three things this changed downstream
Failure became local. When a modality degrades, and they do, background noise ruins audio and bad lighting ruins video, I can drop that stream and still return a defensible partial result. In the concatenated version a corrupted input silently poisons the whole vector.
Missing data stopped being a crisis. Screening in a real clinic means somebody skips a task, or a recording fails. Per modality scoring degrades gracefully. One long feature vector needs imputation, and imputing a modality you did not measure is a good way to invent a patient.
Clinicians started correcting the model. This was the unexpected one. Once they could see which stream drove a score, they started telling us when a stream was wrong and why. That feedback was worth more than the accuracy I gave up. You do not get it from a black box, because there is nothing for them to push against.
Where I think I am wrong
I am not certain this generalises. Screening is a setting where a human always reviews the output, so attribution has somewhere to go. In a fully automated pipeline with no clinician in the loop, the argument weakens considerably and I would probably take the accuracy.
I also suspect there is a middle design I have not built yet. Something that models cross modal interaction internally while still exposing per modality contribution honestly, rather than by post hoc attribution over features nobody can name. If you have seen that done properly in a clinical deployment, please send it to me.
Working on something similar, or think this reasoning is wrong? isararsiddique@gmail.com