Lessons from Gaga Dance and software engineering: When you slip, don’t fall.
Intro
I’m building OpenCRO - an AI chief risk officer for life sciences.
OpenCRO analyzes risk in AI-enabled medical devices and tells you how to minimize your value at risk for the lowest cost.
Cycle 3 started with an ambitious goal and a 4,000 year-old algorithm
I was ambitious.
Agents could generate a risk model for every FDA-cleared AI-enabled medical device.
Instead of generating a risk model cold - build a reusable basis set of threats from the population of ~1,500 FDA-cleared AI-enabled devices.
Cycle 3 taught me that you need more than AI agent engineering:
You need an open mind - data beats your laws about small (or large) numbers
You need variety
You need long tails
My initial architecture was roughly:
Devices → threats → vulnerabilities → countermeasures → assets
with assets late-bound because the same technical device can represent very different business value depending on the company.
The experimental method was “successive approximation”.
Agents sample devices, extract threats, compare them with what they already know, add novel ones, repeat.
Babylonian scribes (2000 BCE) used a similar iterative method to find highly accurate square roots by averaging guesses.
We initially thought that medical devices have up to 120 generic threats.
Slip # 1- Medical devices have a lot more than 120 generic threats
I ran 30 rounds × 20 devices, split across two study arms.
The agents accumulated roughly 1,800 threat patterns.
The marginal contributions fell substantially on every run — early rounds found > 100; later < 40.
And eyeballing the runs, a device was typically producing roughly 8–12 novel threats, while only a fraction represented genuinely new contributions.
The successive approximations were working.
But - I spent quite a bit of time fretting that 1,800 were too many.
Surely - we could turn 1,800 messy threats into 120 canonical threats?
I spent 2 hours manually compressing.
Result? ~1,950 → ~1,800.
There weren’t that many stupid duplicates.
I stopped and realized that - variety is an asset.
A rich repertoire of specific threat patterns can generate a much better risk model than 120 beautiful abstractions.
Slip #2: A bone density device with cardiac threats?!
Yikes. That doesn’t sound good.
We built ThreatSelectorService.
The pipeline was quite elegant:
1,800 → panel filter → semantic filter → LLM ranking → top 8–12 threats
The first test case was on the Rho bone-density test device.
Rho is an artificial intelligence software medical device developed by 16 Bit that opportunistically screens standard X-rays for low bone mineral density (BMD) and early signs of osteoporosis in patients aged 50 and older.
Results looked promising:
1,800 → 1,256 → 131 → 10
One of them was Automation bias, which was completely reasonable.
But its damage description talked about:
missed LVO and unnecessary transfer
LVO is Large-vessel occlusion. For a bone-density product.
Right.
😂
The selector wasn’t crazy.
The selector had found the right threat and brought along the wrong damage description.
I thought the fix was obvious: strip device-specific language from the basis set and generalize the underlying failure mechanisms.
That turned out to be only partly right.
Slip #3: The hold-out arm taught me about long tails
Then came the hold-out arm experiments.
I used devices that had not participated in the basis set construction.
For each unseen device I created two risk models:
GEN — original generation process, reasoning directly from device description + intended use + indications.
SEL — retrieve threats from the frozen basis.
The Rho risk model generated by the OpenCRO agents was fascinating.
SEL found things like:
algorithmic bias
domain shift
false positive / false negative classification
inadequate image quality
OOD imaging
overreliance on decision support
wrong patient/study
GEN independently found:
image/acquisition mismatch
incorrect low-BMD score
out-of-scope use
patient/study misassociation
score misinterpretation
unauthorized PHI access
unauthorized modification
report unavailability
Neither really “won.”
GEN knew device risks. SEL brought device experience.
So for a while I thought the production architecture might become:
GEN + SEL → semantic MERGE
That looked even stronger on the next few runs for neurology and cardiovascular devices.
But a blood-glucose device broke the illusion
GEN produced a beautifully device-specific set:
cross-patient blood exposure
meter measurement error
monitoring unavailability
off-label use
poor blood sample quality
result misinterpretation
test-strip integrity failure
SEL returned only four, and they were comparatively generic:
patient misidentification
result misinterpretation
unauthorized access
use outside indicated population
The agents didn’t know enough about clinical chemistry devices.
We knew a lot about the head of the FDA distribution and very little about its tail.
How long is the tail?
Panels are FDA-speak for clinical areas like radiology.
In our dataset, radiology, cardiovascular and neurology panels account for 76% of FDA-cleared AI-enabled devices. Clinical chemistry accounts for just 0.5%
That’s why the agents did a good job constructing a risk model for neurology and cardiovascular devices and stumbled on a blood chemistry device.
To do a good job building a risk model for a glucose device (clinical chemistry) - the agents need a long tail of data. Like this:
When you slip don’t fall
My Cycle 3 schedule was slipping, and I still had vulnerabilities and countermeasures agent flows left to build.
I could have muscled through it and worked late. And dropped a bunch of bugs.
When you slip - don’t fall. Do less.
I asked what work I could delete.
I realized vulnerabilities didn’t need to be a first-class object in the risk model.
If I could map threats directly to effective countermeasures, the vulnerability layer wasn’t worth the complexity in an automated risk model
So I killed the planned vulnerability work.
When you slip, don’t fall. Do less.
Then I took the afternoon off and went to a Gaga dance class in Jaffa.