Why build icebergs before sleds?
Searching for the algebra behind risk analysis
Ilulissat, Greenland - photograph by Christian Pfeifer
Intro
I’m building an AI chief risk officer for AI-enabled medical devices. Calling it OpenCRO. I call the technology under the hood ‘pattern factory’ because it collects and curates design patterns from collections of things like AI devices.
The idea is to develop an agentic AI system that understands patient safety, and efficacy and how to compute and minimize business risk for an AI device company.
Teaching agents to understand medical-device risk
I finished Cycle 2 of OpenCRO last week. OpenCRO now has a database of 1,500+ AI-enabled medical devices cleared by FDA with funding, sales, competitive advantage, intended use and indications for use by physicians.
I discovered that FDA nomenclature is tricky.
Intended use describes the general purpose of the device. Indications for use describe the disease, condition, patient population and circumstances in which it is intended to be used.
Confused? So was I. So were the agents.
I built 3 new agent flows for the pattern factory: enrich company data, profile device data and find product superiority (compared to similar products on the market).
By Thursday afternoon, I had 1,500 AI-enabled medical-device companies de-duplicated and randomly stratified into three study arms for an experiment that will start mid-September.
The best part is I didn’t need an IRB - although I did need a lawyer. More about that later.
But this week?
One of the biggest things I learned had nothing to do with AI.
It was the FDA data.
My original assumption was simple:
FDA has databases.
OpenFDA has an API.
Agents can query them.
Problem solved.
Wrong.
I discovered that the OpenFDA data I needed was roughly two years old.
Yikes!
So I changed the architecture.
My device-profiling agent now searches through actual FDA filings and parses them into the pattern factory database.
Like this for Apple’s FDA-cleared *Irregular Rhythm Notification Feature (IRNF):
The intended use is:
The Irregular Rhythm Notification Feature is intended to pre-screen and notify the user of the presence of irregular rhythms suggestive of atrial fibrillation (AF). The feature can be used to supplement a clinician’s decision to screen for possible AF. The feature is intended for over-the-counter (OTC) use.
The product superiority is:
The Apple Irregular Rhythm Notification Feature (IRNF) differentiates itself through a validated clinical sensitivity of 88.6% and a high specificity of 99.3% in identifying atrial fibrillation. Its implementation of a sophisticated convolutional neural network-based algorithm allows for refined differentiation between AFib and non-AFib rhythms compared to earlier iterations. Furthermore, the system includes a FDA-cleared Predetermined Change Control Plan, which enables Apple to perform future algorithm updates and optimizations while maintaining rigorous performance standards for sensitivity and specificity.
That solved one problem.
Then I found a much more interesting one.
What exactly is an “AI-enabled medical device”?
Consider two Parkinson’s products built around Apple Watch data.
NeuroRPM appears on the FDA’s official AI-enabled medical-device list.
Rune Labs’ StrivePD does not.
Understanding why isn’t a pattern factory problem.
You need to understand what the FDA actually cleared, where the algorithms run, which components perform the medical functions, and where the regulatory boundaries have been drawn.
And those boundaries matter for risk.
The Rune Labs case is a classic case of risk transfer
Rune Labs’ StrivePD is not some wellness app that avoided FDA regulation.
Rune Labs received FDA 510(k) clearance in 2022 for its Class II Tremor Transducer System.
Here’s where it gets interesting.
NeuroRPM is also a Class II Tremor Transducer.
Same primary FDA product code: GYD.
Both use Apple Watch data for Parkinson’s disease.
Yet NeuroRPM appears on FDA’s AI-enabled medical-device list and Rune doesn’t.
Why?
NeuroRPM’s 510(k) summary explicitly describes machine-learning models that analyze Apple Watch motion data and generate symptom classifications.
And FDA itself says its AI-enabled device list isn’t comprehensive. Devices have historically been identified primarily from AI-related terminology appearing in publicly available regulatory documents.
So absence from the AI list doesn’t mean absence of AI.
And that’s another nasty problem for an agent trying to understand medical devices.
The database isn’t the truth. The filing is the evidence.
But there’s another layer.
Rune has subsequently added StrivePD Guardian, an AI companion on top of its Parkinson’s platform.
Now the system boundary gets interesting from a risk perspective.
The regulated device may be one layer.
Apple’s algorithms and APIs another.
The AI companion another.
Cloud infrastructure another.
The risk hasn’t disappeared.
It has moved.
From device to API.
From manufacturer to platform.
From regulated software to an AI companion.
From your algorithm to somebody else’s algorithm.
And that’s precisely the problem I need to solve in Cycle 3.
A regulatory boundary is not necessarily a risk boundary.
Which brings me to Cycle 3.
The problem
Until now, I’ve generated risk models device by device. One by one.
That works but it’s time consuming. And tedious.
TBH - it takes my generate risk model agent flow about 2 minutes to process several hundred threat entities.
I was really proud of my systems engineering.
An analyst needs about 2 days to do that.
But I want to generate 1,500 risk models and it seemed a cumbersome approach.
See how quickly we get spoiled by AI?
Two minutes instead of two days isn't fast enough anymore!
But I also suspect it’s a bad idea to reinvent the wheel for each device.
A compromised software update is a compromised software update whether you’re building an EEG therapeutic, an imaging application or a Parkinson’s monitoring system.
The same is true for authentication failures, poisoned inputs, unavailable cloud services, corrupted clinical data and dozens of other threats.
The device changes.
The underlying threats may not.
The Cycle 3 hypothesis: threat algebra
I’m going to try to learn a small basis set of dimensionless threats from the devices in the study.
Not 1,500 independently invented risk models.
A reusable vocabulary of threats that can be composed and bound to a particular device.
Then I’ll work down the chain.
Threat → Vulnerability → Countermeasure
For each threat:
What vulnerability does it exploit?
For each vulnerability:
What countermeasures mitigate it?
That gives me a many-to-many graph instead of a spreadsheet full of prose.
And then comes the part I find particularly interesting.
Bind the assets last
Business assets are different.
They depend heavily on the actual device, intended use, company and clinical workflow.
Their value is company-specific too. The same abstract threat can have radically different financial consequences for a $10M company and a $10B company.
Threats may be dimensionless. Risk isn’t.
So instead of starting with the assets, I’m going to build most of the iceberg first.
Why build icebergs before sleds?
Rune Labs is a nice example of why this might matter.
One basic threat.
But once it’s bound to the actual system, the vulnerable component might belong to the device manufacturer, Apple, a cloud service, an API, an LLM provider — or the interface between them.
The regulatory boundary and the risk boundary aren’t necessarily the same boundary.
That’s the experiment for Cycle 3.
Can I discover a stable threat algebra across hundreds of AI-enabled medical devices — and then generate a device-specific threat model by binding that algebra to the actual system, its components and its business assets?
If it works, generating the risk model becomes the easy part.
But first I have to discover the algebra.
About OpenCRO
OpenCRO is an AI chief risk officer for medtech. It analyzes cyber and privacy risk to revenue, regulatory, reputation, and patients.
In dollars.
And then shows you how to minimize the risk at a minimum cost.
Fixed outcome, fixed price, in a yearly subscription with unlimited support.
Drop me an email if you want to see your product superiority and risk analysis.