The three minute constraint
ยท 6 min read
We have a screening tool running in the outpatient departments of a tertiary hospital in India, in ENT and in Neurology, with both arms close to complete. I cannot report results from an unpublished trial. What I can write about is everything that turned out to be hard, none of which is in any paper I have read on the subject.
The short version is that the model was never the problem. I spent months on the model and it works. The things that decided whether the tool got used were almost all outside it.
The window is minutes and it is not yours
In a busy government outpatient department a patient is available for a short, interrupted window, and you are borrowing it from a clinician who has a queue outside the door. I designed the first version around a test that took a comfortable fifteen minutes, which was reasonable in a lab and impossible in the corridor.
Anything that does not fit the real window does not get done, and it does not fail visibly. It gets skipped, or half completed, or done for the compliant patients and not for the ones you most want data on. So your dropout is not random and your sample quietly stops representing the population.
Cutting the protocol down was the single highest value engineering work in the project, and it cost accuracy. That is the honest tradeoff. A shorter test that actually runs on everybody beats a better test that runs on the easy half.
The attendant is in the room
This is the one that broke my assumptions hardest. Patients arrive with family, and the family member is helpful, engaged, and sitting right there. For most of medicine that is good. For a cognitive test it is contamination.
A relative will answer for the patient, prompt them, correct them, or explain what the question means, all out of care. Every one of those invalidates the item. And you cannot simply ask them to leave, because in that setting the attendant is often the reason the patient made it to the hospital, and because relationships matter more than your protocol.
So the design has to absorb it rather than forbid it. Give the attendant something to do. Make the instruction script address them explicitly so they know when their help is wanted and when it is not. Record when a third party is present as a field on the session, because pretending otherwise is worse than logging it.
The device is shared, so the login is the enemy
One device passes between many patients in a session. Any per patient setup ritual gets multiplied by the queue and then abandoned. Long identifiers get mistyped. Anything requiring a password gets written on paper next to the device, which is a worse outcome than not having one.
What worked was making a new session the default state of the app, so that finishing one and starting the next is a single action, and doing patient linkage with a scanned identifier rather than typed entry. Every keystroke you remove from the handover is a keystroke that cannot go wrong forty times a day.
The room is not quiet and never will be
We capture voice. The room has fans, other conversations, corridor noise, and traffic through the window. There is no version of this where somebody gives me a quiet room.
So ambient noise stopped being a nuisance to filter and became a quantity to measure. Sample the noise floor before the task, and if it is above a threshold, refuse the recording at capture time rather than discovering it in analysis. A test that declines to run is recoverable. A test that ran and produced a silently degraded feature vector is not, and it will sit in your dataset looking like valid data.
Reject at capture, not in analysis. By the time the file reaches you the patient has gone home.
Connectivity is optional, so nothing can require it
Wifi drops, mobile data is patchy inside a concrete building, and power is not continuous. Anything that assumes a live connection at the moment of capture will fail during the session that mattered.
Everything therefore has to be local first. Capture to the device, queue, sync when a connection appears, and make the queue survive the app being killed and the phone being flat. That is unglamorous engineering and it is the difference between a dataset and a set of gaps.
Some patients have never used a touchscreen
An older patient in this setting may have limited literacy, no smartphone experience, and a genuine wariness about being tested. A tap and swipe interface encodes assumptions that do not hold, and the resulting score measures unfamiliarity with phones as much as it measures cognition.
The fix is to move the interaction load off the patient. Voice and observation instead of navigation. A trained operator drives the device while the patient just talks and looks. That changes the product from an app a patient uses into an instrument an operator holds, which is a different thing to build and a different thing to validate.
What I would tell anyone doing this
Spend time in the room before you write the protocol. Not a visit, a few sessions, watching the whole flow including the waiting and the handover. Nearly everything above was learned by being there and would not have surfaced from a requirements conversation, because none of it is the kind of thing a clinician thinks to mention. It is just how the day works.
And expect the deployment constraints to change the model, not just wrap it. Shorter task, noisier input, third party present, operator driven. Those are not integration details bolted on at the end. They are different data, and a model tuned on clean laboratory capture is solving a problem that the clinic does not have.
The paper will describe the model. Everything in this post is the reason the model ever got used, and I have not found a good venue for writing it down, which is partly why it is here.
Related: a screen that is 95 percent accurate is wrong about half the time, on why the referral path sets your threshold.