13 months from now, gorillas and AI will monitor clinical trials

Like they monitor the LA Freeway today

Photo by Anthony Celenie

Join other smart people who love Turn expertise to freedom.

👉 If you enjoy reading this post, feel free to share it with friends! Or feel free to click the ❤️ button on this post so more people can discover my writing. 🙏

The study monitor had an 8 AM clinical trial site monitoring visit in a hospital in Lincoln, Nebraska in mid February.

It was -7 outside and the roads were icy. She drove by mistake to the wrong hospital and had to turn around and drive to the other side of Lincoln on the icy, snowy roads.

She had 3 hours left to do her site monitoring visit. Her hands and feet were frozen.

She had a flight that afternoon at 4PM to her next destination - a research site in Miami; 75 and sunny. She had already packed her bathing suit and had tickets to Tony Succar.

Introduction

This week, I dive into selective attention theory, gorillas and the LA Freeway to show how an AI will monitor clinical trials with humans in the loop for special cases.

Two things make clinical trial data special - highly dimensional, slow-moving data.

Clinical trial data is highly dimensional data.

Clinical trial data is highly-dimensional in terms of the number of variables or features of a particular patient who participates in the study.

Highly dimensional data is common in biology; a good example of highly dimensional data in biology is gene sequencer output. There are often tens of thousands of genes (features), but only tens of hundreds of samples.

In a clinical trial, there are often thousands of features and tens of patients

Clinical trial data is a very slow moving scene

Patients come in for a visit once/week or once/month and follow-up 6 months later. A trial with 50 patients and 3000 features has 150,000 very slow-moving data points.

A slow moving scene observed by people

One of the more amazing things about the drug industry is that it monitors clinical trials by hand.

Drug companies use people to manually monitor clinical trial data quality and protocol compliance.

People physically visit research sites and verify that paper source documents match the digital information systems and that the study binder is complete and that the site teams are following the protocol.

These tasks require visual processing of information at the “scene” of the activity.

Since the amount of visual information available at the scene is enormous, and very slow-moving, a person can only process a subset of the scene.

Like our study monitor in Lincoln Nebraska.

Humans focus on the interesting facets of a scene ignoring the rest., especially when the next scene is more interesting (like Miami).

The story of the “gorilla running through the room” refers to a famous psychological experiment by Daniel Simons and Christopher Chabris, demonstrating inattentional blindness.

Participants were asked to count basketball passes between players in a video.

During the task, a person in a gorilla suit walked into the scene, beat their chest, and exited after 9 seconds.

Surprisingly, about half of the viewers failed to notice the gorilla because they were so focused on counting passes.

This experiment highlights how humans often overlook obvious details when concentrating on specific tasks. It has since been used as a metaphor for how we miss critical information in daily life due to selective attention

Selective attention.

Selective attention is a cognitive process in which a person attends to one or a few sensory inputs while ignoring the other ones.

Selective attention can be likened to the manner by which a bottleneck restricts the flow rate of a fluid.

The bottleneck doesn’t allow the fluid to enter into the body of the bottle all at once; rather, it lets the fluid enter in certain amounts depending on the flow rate, until all of it has entered the bottle’s body.

Selective attention is necessary for us to attend consciously to sensory stimuli in such a way that we will not experience sensory overload.

However, the challenge to study monitoring does not end with selective attention; it only begins with it.

Study monitors visit a clinical trial site every 4-6 weeks.

A good study monitor will use a best practice of repeatable process, doing the same thing each time and looking at the same interesting facets of the scene each time.

In the LA Freeway paradigm – it is like standing on the same overpass overlooking the San Diego Freeway and counting cars at the same time once every 4 weeks and noting the results on paper clipboards.

Not a great way to discover accident patterns.

When you have cameras that observe cars 24 hours / day and an AI that analyzes the video for accident patterns.

The consequences of selective attention in clinical trial monitoring.

In the Second European Stroke Prevention Study, data on 438 patients were fabricated at one site.

A human visit to the site failed to identify any problems.

This was later detected by statistical anomalies in the data and confirmed by a central review of blood results. See this article on PubMed.

The San Diego Freeway model of study monitoring.

Our observer returns to an overpass over the San Diego Freeway watching cars.

After a very short period of time, he will generally ignore cars traveling at the same speed (“consistent events”).

He will also observe slow-moving vehicles or cars speeding and weaving in and out of traffic (“novel events”).

Too much consistency can also be bad.

A car parked on the side of freeway for several hours when there is a terror alert.

4 ways an AI improves monitoring

1. The AI removes humans from data preparation/processing loop

Clinical trial monitoring operations should not use manual export and import and manual processing of data from the data capture systems.

Manual data extracts, transformation and load are highly vulnerable to human error and human greed as we see from the Hyperion Andromeda debacle.

2. The AI learns from traffic patterns

An automated monitoring AI can learn novel or inconsistent events over time.

The AI will first learn the normal, consistent events and then detect novel or inconsistent events. We see capabilities like this in so-called smart highways and every day in traffic light management systems that adapt to traffic patterns.

3. The AI learns unknown unknowns

This is in contrast to the traditional approach of doing a risk assessment before the study starts and using it throughout the study; ignoring changes and difficulties that evolve.

In fact – the process of protocol compliance monitoring is dynamic just like San Diego Freeway traffic.

The tools you need to monitor traffic on the San Diego freeway at 3AM are different from those that work at 930 AM since the weather and road conditions can change unpredictably.

A surprise visit by a US President to a meeting of the DNC in Beverly Hills can disrupt the entire system.

Several months before the meeting of the DNC, that was an unknown-unknown.

Users will prompt the AI monitor to discover questions they never thought of asking at the beginning of the study.

And improve performance using fresh data and the unknown-unknowns.

4. The AI moves quickly in a slow moving situation.

Clinical trials move slowly.

However – protocol violations due to mis-dosing or poor data quality need to be addressed immediately by the sponsor and not brushed under the table.

Summary

An AI monitor is above all, a tool for human communications.

The AI monitor will be reliable, fast, friendly and communicative with humans in the loop. Since it’s a machine, it won’t break the blind or be vulnerable to fraud.

It won’t be a bean counter or a dashboard with eye candy.

And it won’t be looking for the fastest way out of Lincoln, Nebraska on her way to Miami Beach.