Judge drug discovery AI by the decisions it improves

After a decade of AI models in drug discovery, the evidence that they have changed which medicines reach patients is still thin. Part of the reason is what the models are scored on, which also answers how to evaluate AI in drug discovery: by the decisions it improves, meaning whether a model changes which compounds a lab makes, which assays it runs and which programs it stops, and whether those choices hold up when the lab measures them. Take a medicinal chemistry team with a potency model that ranks 24 designed analogues and synthesis capacity for eight. The useful question is whether the model's picks clear the team's criteria more often than the chemists' own, measured in the team's cell-based assay.

Elnora is the AI agent for lab work, which means it plans the experiments, writes and optimises the protocols, analyses what comes back and keeps a record of what the lab has decided. For this question it records the chemists' picks before they see the model's ranking, writes both into the plan before anything is made, designs the round that tests them and scores the measured result against each, so the team judges the model prospectively, on its own decisions and its own data.

A good benchmark score can leave the choice of compounds unchanged

Most published drug discovery models are judged by model validation: hold back part of a dataset, predict it, and report the accuracy. In chemistry a random split scatters each chemical series across the training and test halves, so many test compounds are close analogues of compounds the model learned from, and the score mostly says how well it recognises series it has already met. A time split, training on older compounds and testing on newer ones, is closer to the next design round, and models often score lower when tested that way.

A model that ranks new compounds well can still leave a decision where it was. If the chemists would have chosen the same eight compounds anyway, the model added a number. If it pushes the pick towards compounds that are potent and then fail the toxicity counter-screen, it moved the decision in the wrong direction while its accuracy looked fine.

An IC50 means what the assay behind it means

Life science data is conditional. An IC50 depends on the cell line, the passage number, the serum, the incubation time, the readout and the lab that ran it. ChEMBL records each IC50 with its assay, but training sets built from it often pool IC50s from many assays as though they measured one quantity, so a model trained on them predicts a blend of conditions that may match none of the assays a given lab runs.

The cell assay is itself a stand-in for whether a compound will work in a patient, and predicts that only in part. So the record that shows whether a model helps a lab is the lab's own: values measured in its assay, with the protocol beside each number.

State the decision in full before the model is asked

A request to find potent compounds leaves out most of what the team will judge: selectivity over the closest related target, toxicity in liver cells, solubility and whether the series can be made in the weeks available. A model asked an underspecified question optimises the part that was written down. So the team writes the decision first: eight compounds to synthesise, an IC50 below its cut-off in the cell assay, a minimum fold-selectivity over the closest related target, a selectivity index (CC50 over IC50, with the CC50 measured in the liver cells) above its ratio, and a clean solubility check.

Starting from the decision and generating the data it needs reverses the usual order, a method looking for data to apply it to, and it is the order that pays. The largest lever on the cost of each approved drug is how many candidates fail in clinical trials, so a model earns its place by improving which compounds go forward, and a faster design step that sends the same weak candidates on changes little.

Elnora plans the round that tests the model's pick

The chemists record their eight picks from the 24 analogues before anyone opens the model's ranking, so the ranking cannot steer them. The team lead then attaches the ranking and asks Elnora to plan a round that also shows whether it helped. Elnora pulls the earlier potency runs in the same cell line and the team's hit criteria from the Knowledge Base, so the round runs under the conditions it will be compared with.

It drafts a decision record for the round: the compounds the model and the chemists both chose, the ones where they disagreed, and two compounds the model ranked low, since a ranking is also tested by what it advised against. The disagreements fill most of the eight slots, because they are where the round can show which call was right.

Elnora lays out the potency, related-target counter-screen and liver-cell viability plates to the group's named protocol template, with a dilution series per compound, vehicle controls and a reference inhibitor on each plate. A critic agent grades each problem in the design Critical, Major or Minor and attaches a fix. In this round it flags that the top concentration sits below the predicted IC50 of the two low-ranked compounds, whose curves would then stop short of half inhibition, and its fix extends the dilution series. Elnora writes the liquid handler's run file and rehearses it in simulation, and the scientist leading the round approves the deck before seeding.

Elnora joins the readings to the plate map and fits a dose-response curve for each compound against the vehicle wells on the same plate, naming any excluded well and why it was dropped. It computes both selectivity ratios in code and reports which compounds cleared the criteria, how many of the model's picks did against the chemists', and the Spearman rank correlation between predicted and measured order.

The round, its decision record and the fitted curves are filed in the shared Knowledge Base beside their protocols. After three or four rounds the team asks Elnora how often the model's picks cleared the criteria compared with the chemists' own. A few disagreements a round add up to a small sample, so Elnora reports the difference in hit rates with its confidence interval and works out how many more rounds would settle it, and the chemists decide whether the model keeps its say in the next design round knowing how firm the comparison is.