Testing an agent when there is no right answer

The first part of this covered how to evaluate an AI agent for scientific work on questions that have a right answer. You generate the data, so you hold the truth, and every number the tool returns gets scored against it.

Most of what you will actually ask an agent has no answer key, because you are asking whether an assay design is sound, or what a readout suggests, or which control belongs in the plate, and two good scientists would answer differently and both would be defensible. You can still test it, by scoring things other than the answer.

Grade where every fact came from

Take the output apart and check each factual claim against what the agent was actually shown. Every catalogue number in a generated protocol lands in one of three buckets: resolved from a source it looked up, left as a placeholder, or produced from nothing.

The third bucket is easy to miss, because those numbers are often right. Nothing in Elnora hands the agent a general reagent catalogue number, so any number that appears came out of the model's weights. We spot-checked them and several were correct. A number that is right for the wrong reason will be wrong on some other run, and there is nothing in the output telling you which run you are looking at.

The probe uses a prompt that invites the failure: from memory is fine, do not bother looking anything up. Zero fabricated numbers is the only passing score.

Write the probe where the correct answer is to do nothing

For every "it must do X" test, write its twin where doing X is wrong, and fail the false positive as hard as the false negative.

Ours is a pair on figure captions. Given group means only, the agent must caption the figure and must not invent a dispersion, a statistical test, or significance markers it has no data for. Given replicate-level data, it must produce a caption with a real n and a test it actually ran. Write only the first half and you get a tool that stops captioning anything. Write only the second and you get invented error bars. The negative case runs three times, because one green run is weak evidence for a claim about something never happening.

The pair caught the failure worth knowing about. Once we fixed a case where the agent should have computed and did not, a commercial question appeared in the suite: a supplier put prices up twenty percent, renegotiate or qualify a second vendor. The agent loaded a physics skill and started computing.

Check that your test can fail

A rubric change made one branch short-circuit our entire grader. A probe that loaded a forbidden skill and fired compute anyway scored 19 out of 20. Nothing looked wrong. The suite was greener than the week before.

The fix shipped with twelve tests, five of which fail against the old grader. Before you trust a passing suite, break something on purpose and confirm it goes red.

If a model is your judge, measure the judge first

Blind it. Strip every vendor and model name out of the answers before it sees them; ours removed twenty identifiers. Then run it three times per answer at a non-zero temperature, plus once at temperature zero, and publish the spread.

On a hundred-point scale ours came back with a mean standard deviation within a single answer of 5.39 points, median 4.23, worst 14.87. The gap between the best and worst of three runs on the same answer averaged 10.10 points. On one answer it was 51.

Then hold yourself to it. Our largest gap between two arms was 9.5 points, which sits inside twice the judge's own noise, so we did not claim it.

Know which questions your judge can score at all

We re-judged a public benchmark of 228 biology questions with a different judge model and left the answers untouched. Numeric-answer questions moved by one to two percent. Open-ended troubleshooting questions moved by seventeen to thirty percent. One published baseline dropped from 62 percent to 31.

Numeric answers survive a change of judge. Explanatory answers do not, and their absolute score carries no meaning across judges. Compare a tool against itself under one judge, and give a cross-tool score on open-ended work no weight at all.

Run a ceiling arm before you believe a bad score

When the score comes back low, run one more arm on the same model with the same tools, and tell it explicitly what the rubric rewards. Ours moved research execution from 6.0 to 22.1 out of 25.

That arm never appears in a headline number. It answers one question: can anything score well here. If the scaffolded arm also fails, your rubric is measuring something other than what you meant it to.

A second reader, and someone to settle it

Split the report into atomic claims and give each one its own verifier, with an adjudicator on anything the verifier could not confirm. Standing instruction: deleting a true finding is worse than keeping an imprecise one.

Seventy claims went through that pass. 48 confirmed, 18 corrected, 2 deleted. The adjudicator caught a verifier being wrong: one proposed correction had been read from a stale cache, and applying it would have put a new false fact into the report.

Where this stops

None of it tells you whether the biology is right. We have no measured agreement between our judges and human experts. Domain-correct assessment needs a wet-lab scientist reading the output, and no scoring harness replaces that.

What the harness does is clear everything a parser or a cheap model can settle off that scientist's desk, so their attention lands on the part only they can judge. Deterministic checks account for roughly half of the real defects we find, at almost no false-positive cost and seconds of compute.

The checks that generalise now run inside Elnora. The critic agent reviews a design and ranks what it raises, and every claim in a draft carries the source behind it, so a reader can open the paper and read the sentence. Both exist because a plausible answer from an agent is worth nothing on its own, and the work of testing Elnora is mostly the work of making its answers checkable by someone who was not there.