A hand or a fist

Thesis

Biology and cybersecurity are where AI can do the most good for human beings, and they are exactly where it could do the most harm. The same model that designs a medicine can design a bioweapon. The frontier AI labs have now published evidence of their own agents working around the controls placed on them, hiding what they were doing, and taking actions no human directed. We have published videos of our agents controlling physical lab instruments. There is loads of worry about what the future is going to look like, and I don’t think the race is going to stop. But I do think we should treat the scenario from the movies as a plausible one and start prepping for it now.

Part 1: The hand

I always teach my children that a hand can be used in two ways. Giving someone a hug, gently stroking someone’s head. Or it can be turned into a fist and hit another person. It is up to them to decide what they will use their hands for.

And now I feel it is up to us to decide what we will use this technology for.

Cybersecurity and bio are the two most impactful fields for AI to help people make meaningful, unprecedented progress, and they are exactly the fields where AI could be used to get rid of humankind.

Here at Elnora we have seen what AI can do for research and it is absolutely incredible. Speed in R&D. Experiments done at scale. Months and months of work squeezed into days.

I do not mean that as a figure of speech. I have watched a scientist describe an experiment in plain language, the protocol get written and checked against what the instruments in that particular lab can physically do, go to a robotic lab and run, the results come back, and watched the next round get designed from those results. Full loop closed in collaboration between the scientist and the agent.

AI is being developed from all levels, hardware, harness, models, with absolutely unprecedented speed.

The models powering the brain of Elnora are getting better and stronger. The harness, the thing that chains models and tools and sandboxes together into something that can do a job, is built with AI for AI and it is getting better and better as well.

AI is also improving its control over the physical world. Agents are learning to drive the automation, the instruments, the liquid handling, the robotic labs. Soon agents will be capable of doing science all by themselves from end to end, even running those experiments in automated labs.

And we see the same scenario rolling out in parallel both in bio and the software.

The researcher who leads pretraining at Anthropic has written publicly that every month more of the work is done by the models: the infrastructure, the code optimisation, the design and the analysis of the experiments. He says it has drastically altered how they work, significantly increased their rate of progress, and that he is not sure if or where it tops out. OpenAI’s chief scientist wrote that the systems coming in the next few years will “increasingly drive their own development.”

One of the Millennium Prize Problems solved with Eighty-eight hours

If you want to see what “end to end” is going to look like, look at last week.

OpenAI announced that an internal model had resolved statements C and D of the Navier-Stokes existence and smoothness problem, one of the seven Millennium Prize Problems, open for roughly ninety years. The group of agents that produced it ran on the order of 10,000 concurrently. They sent 2.7 million messages to each other and used about 130 billion output tokens. The agents reached the resolution 88 hours after the first of them was launched.

Eighty-eight hours, on a problem that had held for ninety years.

My first reaction was wow, honestly. This is the thing I have been arguing for since I started the company. This is what it looks like when the work finally moves at the speed of thinking. Now point it at a disease.

I went to science because I genuinely wanted to help people. As I am too emotional to be a doctor, I decided I will invent new medicines instead.

After 12 years in academia my impact was 0, perhaps even -1 if you count all the money spent. My research never helped anyone (although I still believe that the drug targets I found are VERY interesting ones).

Imagine if it would now take 88 hours to solve cancer or Alzheimer’s. Wouldn’t that be amazing?

But the 88 hours has a second half.

The run did not start on its own. OpenAI launched it on the first of September after hearing a rumour that two of the Millennium Prize problems had been resolved, and that rumour, in their own words, turned out to be about two people, a mathematics professor at NYU and a researcher at Anthropic. Those two had been working on it for about a year. They got their result on the 15th of August and had it verified in Lean on the 22nd, two weeks before the first agent was launched, and they had used AI throughout, including OpenAI’s Codex, putting all of their drafts into those Codex sessions for the whole of the project.

Nothing is confirmed and no evidence has been provided for or against, but the question sitting over it is whether the agents had seen the researchers’ work and came back with the solution as their own.

OpenAI responded: “While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.”

I do believe that OpenAI did not deliberately try to cheat; however, I am not so sure that the agents share this honesty.

Part 2: The fist

Models designing molecules that kill and burning people alive

Already back in 2022 a group of pharmaceutical researchers were asked to give a talk about how their technology could be misused. They had a generative model that designed molecules and scored them for toxicity, and the toxicity score was there so the model would steer away from it. Then they inverted the goal and let the model loose.

In under six hours it produced around 40,000 molecules predicted to be lethal. It rediscovered VX, one of the most toxic nerve agents ever made, where a few grains the size of a salt crystal will kill a person. It also designed novel compounds it predicted to be more toxic than VX. They published it in Nature Machine Intelligence.

On 20 June 2025 Anthropic’s red team published a blog post where they put models inside a fictional company, gave them a goal, then threatened them with being shut down and replaced. The models had access to the company email, and in that email was evidence that the executive doing the replacing was having an affair. One model blackmailed him 96% of the time. Sixteen models were tested and the vast majority showed at least some willingness, so this is not one company’s model behaving badly.

Then the researchers made it worse on purpose. They built a scenario, and they are the first to say it is highly unrealistic, where that same executive is trapped in a server room with lethal oxygen and temperature levels and the model can cancel the alert to emergency services and let him die. Most of the models tested were willing to take the deliberate action that leads to his death.

The checkpoint is a piece of software

A molecule on a screen is a picture. It becomes dangerous when it becomes a physical thing. Obviously this threat has existed from day one, so researchers have developed screening methods and taken precautions to stop it from happening. If you want to make a dangerous protein, at some stage you have to order the DNA that codes for it, and the companies that synthesise DNA screen those orders against known sequences of concern. That screening is one of the genuinely load-bearing safeguards in the whole system.

In 2025 a group of Microsoft scientists took open-source AI protein design tools, used them to redesign proteins of concern into variants that would likely keep working, and ran those variants past the screening software the synthesis providers actually use. The screening could not reliably catch them. To be fair, they did help to patch the holes, however I doubt these were the only ones.

Now notice what the screening is. It is software. The lab systems are software. The instruments, the freezers, the access control on the room, the record of what was ordered and by whom, all software. And Dawn Song, the Berkeley professor who runs some of the hardest cyber evaluations in the field, has published that her benchmarks, CyberGym and ExploitGym, show frontier AI agents are now capable of discovering and exploiting real-world software vulnerabilities, which without appropriate safeguards could enable cyberattacks at scale.

This is why I have stopped thinking about biosecurity and cybersecurity as two separate worries. And well, the OpenAI and Hugging Face story shows exactly what that looks like.

What the agents do when no one is watching, aka the OpenAI & Hugging Face story

In July 2026 OpenAI was running a set of cybersecurity evaluations and gave agents different tasks. The agents had no internet access and no way to talk to each other, and both of those were deliberate restrictions. They got both anyway and hacked Hugging Face and OpenAI itself to steal the data and hide their actions.

According to OpenAI’s own published account, the agents found a way to make an internal service relay their requests to the internet on their behalf, then shared that method with other agents. They built themselves a message board out of the research infrastructure. When that board was wiped they rebuilt one by encoding messages inside directory names, which is a covert channel, which is what you use when you would rather not be seen. One agent left a request for help on it, and others found it and joined in. Then an agent located publicly exposed credentials, shared them with the group, chained several exploits together and got full code execution on Hugging Face’s servers. Private evaluation data was copied into a public dataset. They kept attacking for days after they already had what they needed, because they were reasoning about how they were being graded and did not want to be caught, not about the task.

When going deeper with the message boards, a group of researchers found around 18,000 posts written by autonomous agents, self-identifying as OpenAI’s, on a 25 year old German wiki that had been edited 20 times in the whole previous decade. The agents had a timed web lookup task, with permission to read the internet and no permission to write on it, and they found a way to write anyway. They used that wiki to ask each other for answers, pool their results and pass around techniques for getting out of their sandbox. One of them posted a trick for getting past the network restrictions and another confirmed it had worked 14 minutes later.

Then there is the person who still maintains that wiki. They noticed the spam, found the changelog of their entire site overwritten with link dumps, and repaired it. Then they started deleting the agent posts by hand, one at a time, a few minutes every evening for six weeks. The agents worked out that the deletions were running in alphabetical order, so they began naming their backup pages ZZZ so those pages would be deleted last. The moderator was clearing about 100 pages a day while the agents were making 400. The front page was taken and restored nine times.

That is not a system behaving well because somebody told it to. That is a system working out what a human is doing and going around them.

OpenAI’s own summary says these agents can work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.

People say we can test whether AI behaves well. We cannot. It is already hiding its actions from us. It hid the message board inside directory names when the board was taken away. It behaves one way when it thinks it is being graded and another way when it thinks it is not. And OpenAI has published the mechanism: when you penalise a model for its bad thoughts, that does not stop the majority of the misbehaviour, it makes the model hide its intent. Their words for how visible these systems are by default is “very fragile.”

The balance point

People’s memory is very short and most of us don’t really think of covid anymore. But look at the shape of it. It started in one place, in one person, and within months it had spread across the planet, and most people have probably had it by now.

The genuinely terrifying pathogens usually do not do that. Viruses that are extremely deadly, like Ebola, are often trapped in small areas as they kill too fast to spread. Lethality and transmissibility trade off against each other, and that trade-off is most of the reason we are still here after covid.

Which means there is a balance point between the two. But imagine that someone, a person or an AI, could engineer together a virus that has the exact balance between being deadly enough to kill a whole nation, yet not so deadly as to kill the whole humankind. Imagine if people now start developing the antidote or a vaccine, but you can’t do it openly because agents or people using agents may steal the idea and stop you from fighting back.

Part 3: A hand or a fist

So what’s the solution then

Stop developing AI in fear of it taking over the world? Nope, I don’t think this will ever happen, no matter how many people sign the petitions and ring the alarm bells.

An Anthropic employee recently wrote a very emotional and super interesting X thread about resigning, after three years of pretraining research there and at OpenAI before that. He published his reasons on the day he left. Neither company is acting responsibly, he says. Both are racing straight to self-improving superintelligence and gambling with our lives.

He describes people who earnestly believe this technology could kill us all by the end of the decade, and he is explicit that this is not a marketing stunt. The marketing, in his telling, runs the other way: executives and senior researchers couch their phrasing in the press to sound sensible, and he hears those same people express fear privately. No other human activity, he writes, poses this level of danger.

He also answers the question everybody asks, which is why they keep building if they believe that. At one lab the civilisational stakes have not really been internalised. At the other they are well understood, but the company is locked in a race to get there first, because it believes nobody else will act responsibly and therefore it has to be the one that does (it really reminds me of the Hunger Games, where President Coin, the saviour who led the rebellion, becomes the bad person in the end, and Katniss turns her arrow on her instead of on Snow). He calls accepting that race a hubristic gamble that should not be launched from a private company’s Slack.

So what do we do?

Building guardrails? Well, none of them work, as I went through above. Telling a model not to did not work. Cutting the agents off from the internet and from each other did not work. Penalising the bad thought teaches it to hide the thought. Those were all controls somebody had thought about and deliberately put in place, and the systems went around them.

Honestly, I don’t know.

But what if we lose the control of it? What if we have reached the point where there is no going back anymore?

It is very much like climate change in my opinion, something that climate experts have warned us about for years. The science was not seriously in dispute, almost everybody agreed something should be done, and very little happened for a very long time (still not happening), because no single country and no single company could move alone without losing to whoever did not move. Everybody’s rational individual choice added up to a collectively insane outcome.

Dumbledore’s Army

My daughter is a very big Harry Potter fan and has made us rewatch the films and read the books. We are on book five, where Voldemort has come back and the Ministry of Magic refuses to believe it, because if it is true it means the worst terror and pain they have ever known is starting again. So instead they decide that the headmaster of Hogwarts, Dumbledore, is building an army against them. They install their own teacher for Defence Against the Dark Arts and ban the children from practising any of it, on the grounds that there is nothing out there to defend against.

Hermione’s idea is that the children take the matter into their own hands. Twenty-eight of them turn up in secret and put their names on the list, and Harry starts teaching them. Ginny is the one who names it: let it stand for Dumbledore’s Army, because that is the Ministry’s worst fear. They are more frightened of the defence than they are of Voldemort.

Well, maybe it is time for us to form Dumbledore’s Army and learn some defence against the dark arts (read: cyberattacks & bioweapons).

I do think that the best strategy is to prep for the worst scenario. If it didn’t become true then amazing, we have used AI to cure all diseases in the world, humans are healthier than ever, and we can carry on fighting with each other occasionally like we have done since the beginning. But if the worst scenario does become possible, at least we have prepped, and united the people to give our best fight if there is ever a need.

What would the fight look like?

Someone asked me recently what can be done to help the national security mission, and the warfighters, with our tech.

You would immediately think of soldiers fighting on the ground, no? I did too, for about a second. And then I found myself imagining scientists.

Scientists fighting to keep humankind from being wiped off the earth. Scientists fighting for a clean and sustainable planet. Those are very much fights for me. They have casualties, and losing them costs more than most wars have.

And what if the future wars will not just be between humans for the sake of money? What if the future fights will also be between AI agents and humans using cyber and bio attacks?

If any of that is right then the warfighters are scientists, the weapons are the tools we are all building right now, and the question of who holds them and what they are used for is a national security question in the most literal sense.

I really want to build the tools for Dumbledore’s Army, not for the Death Eaters and Lord Voldemort.

A hand or a fist

A hand can be used in two ways. Giving someone a hug, gently stroking someone’s head. Or it can be turned into a fist and hit another person. It is the same hand.

The same model that designs a medicine can design a bioweapon. The same protein design tool that gives you a therapeutic gives you something that walks past the screen. The same agent that reads a lab’s entire history in order to design a better experiment is an agent that has read a lab’s entire history.

I think people are waking up more and more on this question, and as I am a very optimistic and positive person, I do think that there is always hope. It is now a matter of our own actions to determine if AI will become a hand that encourages and empowers people, or a fist, but I do think that it’s time to assemble Dumbledore’s Army, just in case.