Moneyball for drug research
Master-apprentice-AI tooling model of work
Hannah Berman, Daniel Lis and Danny Lieberman in Moni Shahar’s office at TAU
Baseball metrics for oncology clinical trials, is an idea I floated a few months ago to some colleagues in the drug research and development space.
I received a lot of encouragement and support from many people for the idea — specifically Denise Lepley, Moni Shahar and Jonathan M. Fishbein, M.D.
Measure how patients contribute to the success of clinical trials
The idea is to provide a simple, intuitive and money-based way to measure performance of patients in clinical trials and how they contribute to success or non-success of the study.
Like OPS (on base percentage + slugging percentage) is a good descriptor (and not the only one) of player output — we postulated that we could create
a performance metric of patient output.
The performance metric is a utility function that says — “If you take $100 and invest in efficacy, safety and toxicity, what is your ROI?”
The utility function provides a powerful and simple way of measuring performance and trajectory of a patient during the clinical trial and how they contribute to the success of the study.
Master and apprentice engineering model
In addition to testing the notion of baseball metrics for oncology research, we also tested the model of a master programmer (Danny) working with 2 apprentices (Hannah and Daniel) assisted by ChatGPT 4
Hannah and Daniel; two very talented interns from the Onward Israel (Masa) program joined me in June — July 2023 to develop a prototype for baseball metrics using open access trial data from Project Datasphere.
We had an amazing time. Hannah (going into her junior year at Tufts in Computer science) and Daniel (going into his junior year at UMass in EE) did a great job.
We built a model using data from 4 Amgen Vectibix studies — 600K labs, and 3000 patients and 48,000 adverse events.
The first part of the project was feature engineering, describing 15 features of a solid cancer efficacy-safety-toxicity model and 1 performance metric based on a utility function (a linear combination of the efficacy, safety and toxicity variables).
In a cloud-native approach — we used Snowflake, and visualized the results using rows and streamlit.
One of the really cool things that Hannah did was an instant replay program that takes AdAM data sets and replays a particular patient’s performance metric on their study timeline.
The proof is in the pudding
The team had 2 demo days. Without fear, Daniel and Hannah pitched a Tel Aviv VC deal flow person themselves and did a superb job.
The next day — we took the train to Tel Aviv U and they showed Moni Shahar their work.
Summary
In just 5 weeks — with zero domain knowledge we used a master-apprentice-AI tooling model to develop a model and online prototype of baseball metrics and performance trajectory for patients in 4 oncology trials.
The engineering approach and system based on a sabermetrics metaphor show promise in providing a simple and intuitive way of measuring performance (and eventually predicting) the success (or failure) of oncology research.