Clinical notes do not translate
ยท 6 min read
I worked as a medical data manager and evaluator on li-lab/HealMed, a dataset of clinical notes built for healthcare language models. The notes came from Indian, Malaysian and Igbo sources, and the job was to annotate them alongside doctors who actually practise in those systems.
I went in assuming the hard part would be volume. It was not. The hard part was that the same clinical reality is recorded in ways that do not line up, and that a model trained mostly on American notes does not fail loudly on the others. It fails quietly and with confidence, which is worse.
A note is not a description, it is a local convention
A clinical note is not written to be read by strangers. It is written fast, by someone trained in a particular tradition, for colleagues who share a set of assumptions, inside a system with its own drug supply and its own billing pressure. Every one of those shapes the text.
Change the country and you change the training tradition, the available drugs, the documentation incentive and the language the patient used to describe the problem. The illness is the same. Almost nothing about how it appears on the page is.
The specific ways it breaks
Drug names. Notes from systems where brand prescribing dominates are full of brand names, and the brands are local. A model with good coverage of generics and of one country's brand landscape will read a perfectly ordinary prescription and not recognise the drug. Worse, some brand names are reused across markets for different molecules, so you do not get a blank, you get a wrong answer.
Abbreviations collide. Clinicians abbreviate constantly and the abbreviations are not standardised across traditions. The same short string can be a diagnosis in one note and a department, a procedure or an entirely unrelated finding in another. Expanding abbreviations correctly needs the local convention, and getting it wrong changes the meaning rather than losing it.
Symptoms arrive as phrases, not codes. Presenting complaints get written in the patient's framing, often translated on the fly by the clinician from another language. You get compound complaints that are culturally standard ways of describing feeling unwell, and which map onto no single code. Force them into one and you have invented specificity that the note never had.
Negation and hedging differ. How a clinician records the absence of something, or their uncertainty about it, varies by training tradition. Negation detection is one of the oldest tasks in clinical language processing and it is tuned to particular phrasings. Move systems and the cues shift, so you get findings recorded as present that the note was explicitly ruling out.
What goes unwritten differs most. This one is invisible to any metric. Every system has things so obvious locally that nobody writes them down. Which things those are changes by country. A model learns that absence means negative, when in that setting absence means assumed.
A missing field is ambiguous between not present, not checked, and too obvious to write. Only somebody inside the system can tell you which.
Why the doctors had to be in the room
I could not resolve most of this by reading. Deciding whether a particular abbreviation meant one thing or another, or whether an unwritten field implied normal or unchecked, needed somebody who had worked in that setting. That is not a labelling task you can crowdsource cheaply and it is not one I could do alone, however much documentation I read.
The thing that changed how I think about this was watching two clinicians from different systems disagree on the same note, both correct within their own convention. There was no ground truth to appeal to above the convention. That is not annotator noise to be averaged out. It is real structure, and averaging destroys the information.
What this means for evaluation
Reporting one aggregate score across a multi country clinical dataset hides exactly the failure you care about. A model can look fine overall and be systematically wrong on one source, and if that source is the smallest slice the aggregate will not move enough to notice.
So the reporting has to be stratified by source, always, and a drop on one source has to be treated as a finding rather than as noise. If a model is deployed where a dataset is thin, that thinness is the headline result, not a limitation paragraph at the end.
I would also stop calling this a generalisation problem. Generalisation implies one true distribution and some samples that drifted from it. There is no privileged distribution here. There are several conventions, all valid, and a model has to know which one it is reading before it can read it correctly.
The part that bothers me
Most clinical language work is validated on notes from a small number of wealthy health systems. Those systems are not representative of where most patients are, and the direction of deployment is exactly the direction the models are weakest in. Tools built on one convention get exported to settings whose notes they misread, and the failure mode is confident rather than obvious, so nobody catches it early.
Fixing that is unglamorous. It is annotation work, done with local clinicians, on data that is hard to obtain and harder to release. There is no clever architecture that substitutes for it. HealMed is a small contribution to a gap that is much larger than one dataset, and the useful thing about it is that it exists publicly and can be checked.
Related: ICD-11 has no clean map from ICD-10, on what happens when you try to force clinical language into codes.