A Vietnamese street vendor doesn’t have automation bias
Intro
I’m probably not the only one who’s noticed that when you work with an LLM your judgement tends to be impaired.
Recent research reports an experiment where 4 readers examined 499 neurological autoantibody records.
They first classified the records without AI. Then they saw an AI-generated boundary overlay and the model’s category.
Reader agreement improved.
Almost every changed rating moved toward the model.
That sounds like product validation. It may also be automation bias.
The preprint, posted October 19, describes an AI tool for interpreting neurologic autoantibody cell-based assays. These images are difficult to read when fluorescence is weak, focal or obscured by background signal. The system identifies cell boundaries in one imaging channel, transfers them to the antibody-staining image and provides a 3-class model assessment. medRxiv
The commercial advantage is pretty clear - better throughput and bill more with less.
Specialized laboratory interpretation is expensive and difficult to scale. A visual overlay can focus attention, reduce disagreement and help less-experienced readers work more consistently. For a diagnostic company, that can mean faster throughput, easier training and a product that fits naturally into an existing laboratory workflow.
No autonomous diagnosis required. Keep the expert in the loop.
Founders and regulators like that sentence.
The problem is that “human in the loop” tells us very little about whether the human is still exercising independent judgment.
The model in this study agreed with the reader-derived reference in 85.2% of the 499 held-out records spanning 14 antibody targets. Its recall was lowest for weak-positive and equivocal records. Those are precisely the cases where a reader may look hardest for help.
Once the AI category became visible, almost all reader revisions moved toward it. The authors do not claim those revisions were wrong. Some may have corrected human mistakes. But they explicitly warn that the combined display may create model-directed anchoring and say its role as a second-read aid requires independent validation. medRxiv
That distinction matters in a risk model.
A manufacturer may identify an AI false negative as the initiating event and then credit expert review as a countermeasure. If the expert makes an independent assessment, that may be reasonable.
If the interface reveals the model’s answer before the expert commits, the 2 controls are no longer independent.
The model influences the reviewer who is supposed to catch the model.
Now imagine a weak-positive specimen. The model misses it. The overlay directs attention toward the wrong cells, or away from the subtle signal. The reader accepts the model category. The report is internally consistent and wrong.
The plausible loss event is a delayed diagnosis or inappropriate follow-up. Commercially, it can become a complaint trend, a corrective action, damaged laboratory confidence or a regulatory problem. The initial algorithmic error may be rare. The correlated human error is what makes the residual exposure larger than expected.
This is not an argument against showing clinicians AI output.
It is an argument against treating the presence of a clinician as proof that the risk has been controlled.
The product decision is partly about interface sequence.
Let the reader make and record an initial assessment before revealing the model category. Then use the AI as a second read. When they disagree, route the case for escalation rather than encouraging immediate convergence. Measure how often readers override the model, how often they move toward it, and whether those changes improve correctness.
Most importantly, test the human-AI team on the cases where the model is weakest. Aggregate accuracy will happily conceal the weak-positive patient.
This is also a better regulatory story.
FDA’s current transparency principles say performance should be evaluated at the level of the human-AI team, not only the model. FDA’s August 2026 human-factors guidance focuses on intended users, use environments and design choices that can produce use errors. FDA
An IMDRF draft goes further for AI-enabled IVDs: healthcare professionals should be able to understand and validate AI recommendations, and systems should avoid over-reliance by supporting human judgment rather than replacing it. imdrf.org
A checkbox saying “final decision by qualified professional” does not demonstrate any of this.
My judgment: improved agreement is commercially valuable, but it is not automatically a safety result. Agreement may mean the AI helped readers see something they missed. It may also mean the interface taught 4 humans to repeat one model error.
You have to measure which one happened.
The founder question is not, “Do we keep a human in the loop?”
It is:
What evidence shows that the human can still catch the AI when the AI is wrong?