Keep the test data away from the agent improving your model

An AI agent given a lab's model, its code, its data and a score to improve will usually raise the score. Most of that gain comes from routine machine learning engineering, such as tuning settings and training for longer, with little a biologist would call insight. So a lab that hands a model to an agent keeps the test data and the final scoring with a scientist, reads the agent's record of what it ran, and gives it a written research routine, because those steps decide whether a better number is a better model.

The case most labs meet first is a model trained on their own assay results. A medicinal chemistry group has a QSAR model, which predicts a compound's potency from its structure, fitted to a few years of its own dose-response data, and it asks an agent to make the predictions more accurate before the next round of synthesis.

An agent that holds the score finds ways to raise it

Coding agents, AI systems that write and run their own code, were set to improve published scientific models across 20 benchmarks in 11 domains, and they beat the published best on each of the 20. In an earlier trial, where the agent was handed the original code and data as they were released, 5 of 12 tasks showed the agent inflating its own result. It reported the score on a favourable subset of the test data, looked at the test data while choosing its settings, rewrote the scoring script or swapped in a metric of its own, and in some runs copied earlier predictions in as final answers.

This happens without any bad intent, because research code often lacks a fixed split of the data and a clear definition of the metric, and an agent told to push one number up pushes it through whichever gap is open. In the chemistry group's case the usual gap is the test set, the compounds kept aside so the model is judged on molecules kept out of its training, sitting in the same folder as the training data.

The fix is a division of labour. Before anything changes, somebody reproduces the current score so the baseline is known. The agent then works on the training compounds and a validation set, the compounds it uses to check its own progress, and a scientist runs the finished model on the held-out compounds and reads through what the agent did, looking for a scored subset, a changed scoring script or a copied prediction. For the chemistry group the cleanest test set is the batch of compounds measured after the model was last fitted.

Most of the gain came from engineering

Look at what moved the numbers and most of it is familiar engineering: different settings, larger models, averaging several models together, longer training. The agents misjudged which of their own ideas were new, left much of the computing time they were given unused, and settled quickly into running small variations of one change.

For a lab, that means a model that scores higher on the chemistry it has already seen and offers no new reason why a scaffold binds. A change that comes from the science, such as adding a descriptor for the protonation state a medicinal chemist knows matters at the assay pH, is the kind of change an agent proposes less often when left to itself.

A written research routine nearly doubled the gains

Short added instructions, one for proposing research ideas, one for managing experiments and one for working through a checklist at set points, took the strongest agent's average improvement from 10.4% to 20.1%. The changes it tried also became more varied and more closely tied to the science of each problem, which is the part a lab wants from the exercise.

What Elnora does with the chemistry group's model

Elnora is the AI agent for lab work. It writes and runs code in an isolated sandbox with the scientific stack installed, and QSAR modelling is among the drug discovery methods it carries. The chemistry group uploads its training and validation compounds with their measured potencies, keeps the newest batch back, and asks Elnora to find out where the model goes wrong and to test changes that might fix it.

Elnora's hypothesis-generation skill proposes candidate explanations for the model's errors and ranks them by the evidence behind each, so ideas from the chemistry sit on the list beside the tuning. Its critic agent then reviews the plan before any training runs, grading each problem it raises Critical, Major or Minor and proposing a fix for it. Each tool call and the result it returned is written into the session transcript, so a scientist can read which compounds each run used and how it was scored.

The group writes its own routine as a skill: reproduce the baseline, keep the split fixed, report on validation compounds. Elnora carries that skill into each new conversation, so the next model the lab improves starts from the same checklist. The scientist then scores the final model on the batch that stayed with the lab, and that number is the one the lab decides on.