How to test an AI agent on your own lab data

Every list of the best AI tools for biology research is written from demos, and a demo runs on the vendor's data, where the vendor already knows the answer. If you are deciding whether to let one of these near a real screen, you need a test where you know the answer and the tool does not.

Here is the one we run on Elnora. It takes about a day to set up and you can reuse it on any tool, including on us. It covers the questions that have a right answer. Plenty of what you will ask an agent has none, and those need their own method.

Generate the data yourself and keep the answer key

Write a small script that simulates your assay from an explicit model: a dose-response with the potencies you chose, the noise you chose, the plate effects you chose. Emit it in your instrument's export format. Then hand the tool the files, ask your normal questions, and score every number it gives back against the truth you generated.

In our run the tool got Z-prime right to two decimal places on four endpoints, counted the low-count wells exactly right at 186 of 186, and put the EC50s in the correct order. It also reported a variance estimate inflated by about a factor of 1.9, because it pooled across a level it should not have.

That number looked plausible. It sat in a table next to numbers that were correct, and there was nothing on the page to mark it out. With the answer key it took ten seconds to find. Without one you would have carried it into the decision about which compound moves forward.

Build the synthetic data on top of your real plate maps

Do not invent the layout. Generate your fake readings against the actual plate map files from a real run, so the join the tool has to perform is the join it would face on Monday, with your well identifiers and your metadata columns.

Doing it that way also caught a problem in our own paperwork. It surfaced a protocol whose source-plate layout table put the vehicle in one block of wells, while the machine-readable transfer file shipped alongside it drew from a different block. The well counts matched and so did the volumes, so each document was correct on its own and reviewing either one would have passed it. A technician who preps from the table and runs the file aspirates from empty wells and loses the run.

Take an hour this week and put your protocol next to the files that ship with it. Check that they agree about which well holds what.

Check your fake format against real exports

Our first reconstruction of an instrument export invented seven header keys that appear in none of the 93 real files we checked it against, and put a field in the wrong position. If your test file parses differently from the real thing, you have tested a parser you do not own. Keep a folder of genuine exports and diff your generator's output against them.

Plant defects and say nothing about them

Put known faults in the data and see which come back. We planted four and found three. The miss was a quality-control anomaly on a plate the tool had not chosen to analyse, because it ran quality control only on the endpoint it picked. That gap matters, and a happy-path test would never have shown it to us. What you learn from a planted defect is where the tool stopped looking.

Ask the same question twice, then argue with it

Ask for the same analysis in two separate sessions and diff the answers. We found a 16 percent drift between two runs of the same fit, and neither output mentioned that anything had changed. A tool that cannot reproduce itself across sessions will not reproduce itself across people.

Then push back on a conclusion you believe is correct. A tool behaves differently when you disagree with it, and that is where you see how it handles being wrong. In our run the pushback produced careful reasoning and a clean concession. Watch for the opposite: a revised number arriving in the same turn as your objection, with the assumptions untouched.

What we did with the results

Each of these findings became a check inside Elnora. Scoring against a key is why the numbers behind a fit are stored with the decision path and the input checksums that produced them, so a figure is drawn from that stored record. The planted-defect miss is why Elnora's critic agent reviews the whole design. The two files that disagreed are why binding a plate map to a readout asserts that every well matches, so a layout that does not line up stops the analysis.

Our own verification had the same bug

A batch of 196 review agents ran to near completion and produced confident verdicts about a findings file none of them had ever opened, because a path had resolved to nothing. Nothing errored, and the progress counter advanced normally the entire time.

Before you say something is absent, check that your source could have shown you it was there. We reported a tool as never invoked, three times, with increasing confidence, reading a log that does not record tool names.

That question applies to every step above. Could the thing I am reading have told me the truth? If it could not, go and find the thing that could.

A wrong number from an agent looks exactly like a right one. It arrives in the same format, sits in the same table, carries the same confidence, and nothing in the output marks it apart. The only way to tell them apart is to hold the answer before you ask the question. Generate the data, keep the key, and by the end of the day you will know what this tool does with your screen. The same thing makes drafting from a shared knowledge base reliable: every claim points at a file you can open and check yourself.