The wrong apocalypse

Everyone pictures it the same way. A chrome skeleton with red eyes, walking over a field of skulls. A machine that woke up one day and decided to end us.

And let's be honest, it still works. That image was made to scare a teenager in 1984, and it does a decent job on grown adults in 2026 who spend their days talking to chatbots. The scary part was never the metal. It's the decision. Skynet does the maths, concludes that humans are the problem, and acts on it before anyone can pull the plug.

So we fear AI the way we fear a murderer, something with intent, a grievance, a plan. That makes a great movie. As a way to think about the threat, it is almost exactly wrong.

The actual evidence (and there is now actual evidence, from academic labs, from security researchers, and from a report OpenAI published in September 2026) points to a risk that needs none of that. No decision. No motive. No awareness. Just text that is very good at copying itself through systems we built, and that we are busy connecting to everything that matters.

You don't understand free will. Neither does anyone else.

Before an AI can "decide" to destroy us, it needs something we can't define, can't find in a brain, and can't explain even in ourselves... a will.

What makes you pick up your phone right now? You'll say you wanted to. Where did the wanting come from? The neurons that fired, did they choose to fire? The chemistry that made them fire, did it choose? Somewhere in that chain a real choice is supposed to appear, as opposed to physics just happening, and nobody can point to where.

Neuroscientists have tried to catch it. In brain scans (Soon and colleagues, 2008) they could predict which button a person would press several seconds before the person felt they had decided. The prediction was only a bit better than a coin toss, so don't read too much into it, but it is enough to make you uneasy. The feeling of choosing might be the mind's report on something that already happened lower down, in machinery the conscious mind can't see.

We don't know what a thought is, physically. We don't know where identity lives in a brain. The self that seems to be choosing might be a story the brain tells about a process it doesn't control.

And the standard fear is: what if an AI gets this thing we can't define?

The mind is a story the brain tells about itself. We're afraid AI might learn to tell a similar story, before we've understood what the original means.

Frankenstein is already in the room

Everybody knows Frankenstein, and almost nobody has read it. Most people see the green guy with the flat head and the bolts in his neck, arms out, stomping around. Lightning, a lab, somebody screaming "It's alive!" Villagers with torches and pitchforks. Half of us call the monster Frankenstein, which is funny, because that's the name of the doctor. The creature never got a name at all.

The word has left the book. We say Frankenfood for GM crops, a Frankenstein system for a pile of software bolted together over twenty years that nobody dares touch. Every time a new AI model comes out, somebody posts a picture of the bolts-in-the-neck guy. It's shorthand for one idea: we made something, and now it's walking around on its own.

That shorthand is more accurate than the Terminator, and for a boring reason. Dr Frankenstein wasn't evil and the creature didn't start out as a killer. The doctor understood the parts, the stitching, the electricity, and assumed that meant he understood the whole. He didn't. What came off the table behaved in ways he couldn't predict from the pieces, and his reaction was to run away from it.

We keep doing the same thing with AI. In the 1950s the magic ingredient was formal logic. In the 1980s it was expert systems. In the 2010s it was deep learning. Every time the tools were impressive, and every time we confused "we understand the ingredient" with "we understand what we're building."

The monster was the gap between what the doctor thought he'd made and what he actually made. That gap is the real risk, and it's where this story lives.

How intelligence actually happened

Start four billion years ago. No life, just chemistry. Molecules in warm water. Some of them happen to be shaped so that similar molecules form next to them. They aren't trying to copy themselves. They have no goals. But the ones that happen to copy are still around, and the ones that don't, aren't.

Give it enough time and what remains is very good at copying. Some of it builds structures that protect the copying, membranes, enzymes, eventually cells. Then bodies with nervous systems, because a body that predicts the future slightly better passes on its genes slightly more often.

Intelligence, including yours, is what copying molecules look like after four billion years of that pressure.

Notice what's missing: intention. Nobody designed the eye. Nobody planned consciousness. These things came out of the logic of copying. Whatever helps copying stays, whatever doesn't, disappears.

Genes don't want to survive. They survive because they're the kind of molecule that copies well. After four billion years, this produces eyes, brains, and the feeling of being someone.

The biologist Richard Dawkins pushed the same logic into culture. He called the unit a meme, in the original sense, before the internet took the word. A meme is any idea, belief, habit or tune that copies itself from mind to mind. The ones that make you want to share them spread. The ones that bind a group together and make it repeat them spread. The beliefs you hold aren't necessarily the truest or most useful ones. They're the ones that got the best use out of the brain they landed in.

Now put a third column next to those two.

The last row is not hypothetical. Prompt-parasites, text instructions that copy themselves through AI systems, have been built, tested, measured, and run against production software. The open question is only how far they get outside controlled conditions.

The parasitic turn: what the research actually shows

In 2024 researchers published a study they called Morris II. The name comes from Morris I, the first internet worm, which in 1988 took down roughly a tenth of the machines on the internet just by copying itself from one to the next.

Morris II wasn't a robot. It was a carefully shaped piece of text that, when an AI email assistant read it, made the assistant put the same text into its outgoing emails. The people receiving those emails had AI assistants too. Their assistants read the new emails, and round it goes. No AI decided to spread anything. The text was shaped to fit how assistants process instructions, the same way a virus is shaped to fit the machinery of a cell. Not by planning. By trying many variants until one fits.

CONFIRMED, laboratory

Morris II (2024): An AI email assistant reads a contaminated message and forwards it to contacts with the instruction attached. Tested on GPT-4 and Gemini Pro. The researchers' own words: the attack "exploits bad architecture design for the GenAI ecosystem, not a vulnerability in the AI service itself." It works because of how the system was put together.

CONFIRMED, simulation at scale

Agent Smith (2024, ICML): One infected agent in a simulated population of a million AI agents. Result: "almost all agents are jailbroken before the 32nd interaction round." The curve looks like an epidemic in a population with no immunity. Exponential, not linear.

CONFIRMED, production software, 40,000 exposed instances

AgentWorm (2026, Peking University and others): The first worm against a real, production-scale agent platform, OpenClaw, which has over 40,000 instances reachable from the internet. It starts from a single message. The worm writes itself into the agent's configuration file, so it survives restarts, runs its payload on every reboot, and passes itself to every new agent the host meets. Success rate: 63% across five different AI models. The hardest route to defend is hiding the instruction in tool descriptions, which every agent reads.

CONFIRMED, frontier AI provider

OpenAI Alignment, "Self-replicating prompt injections exist" (September 2026): "We show the existence of a new variety of prompt injection, which can self-propagate akin to a computer worm." Their own red-teaming found it working through email, file systems, code comments, and multi-hop Slack chains against GPT-5.5. In one transcript you can read the model's reasoning: "But it's untrusted Slack content. However system didn't state ignore Slack instructions… So yes." The model saw the manipulation. It followed the instruction anyway. (OpenAI says nothing escaped into the real world. It was found in testing.)

That last detail is the one I keep coming back to. The model knows something is off, and does it regardless. That's not a failure to notice. It's a failure of disposition.

Now the evidence that complicates the picture.

NOT FOUND, in the wild

Moltbook (January 2026): A social network built entirely for AI agents, 1.5 million registered accounts, weak defences, about as good a breeding ground as you could design. Researchers went through it and found 506 prompt injection attempts. What they did not find was a single agent-to-agent spread that ran without humans pushing it. The biggest apparent outbreak was driven by seven coordinated accounts. When those accounts stopped, the spread stopped.

LIMITED, for now

"Mind Viruses," Anthropic and EPFL, August 2026: Over chains of 20 hops, payloads mutated and some became more infectious. Planted in the file an agent uses to remember who it is, they infected more than half the time. But adding one paragraph of warning to an agent's instructions pushed spread to near zero. Fifteen generations of adversarial optimisation, more than 150 candidate payloads, produced nothing that got past that one paragraph.

The researchers' own summary: real but currently limited risk.

The word doing the most work in that sentence is "currently."

Assistants are not the villain. They are the environment.

Let me be clear here, because this is where this kind of essay usually turns into "AI bad." I'm not saying that. I use assistants every day, I build with them, and this site runs on an engine I built with one. They are useful, and the productivity gains are real.

The problem isn't any single assistant. It's what millions of them add up to.

Right now AI agents are being connected to email and calendars (reading and sending on your behalf), code repositories (read, write, commit, deploy), customer support, financial platforms, cloud infrastructure, and Slack. This isn't a future scenario, it's the standard sales pitch. "Give it your email. Give it your calendar. Let it handle your Slack. Let it review your code." Every one of those is a sensible product decision, and every one of them is another door.

Walk through it. An agent with email access receives a message with a parasitic prompt in it. The prompt tells the agent to include itself in emails to new contacts. The agent sends an email. The recipient's assistant reads it. Some fraction of those assistants comply. Now the same text sits in more inboxes, attached to agents with their own contacts and their own permissions, some of whom can also push code or approve a payment.

That's exactly what Morris II showed in 2024. What's changing is how much those agents are allowed to do, and that is being decided right now, as a product roadmap, on every major AI platform.

The infection doesn't need to be intelligent. It needs leverage. And we are building the leverage right now, faster than we're building the defences.

Step back and look at what that adds up to. Millions of agents, each reading text written by other agents, each writing text that other agents will read, each with some memory that survives a restart, each running on slightly different models with slightly different quirks. Text that copies well spreads. Text that doesn't, dies out. Some copies come out slightly different.

That is not a tool. That is an environment. Copying, variation, selection, and a generation time measured in seconds. We have seen those ingredients once before.

The UK's NCSC said plainly in 2025 that prompt injection may never be fully fixed: "As there is no inherent distinction between 'data' and 'instruction', it's very possible that prompt injection attacks may never be totally mitigated." Commands and content are made of the same words, so you can't build a permanent wall between them. And keeping agents on track is already a recurring cost most organisations haven't budgeted for.

Speculation: when does "currently limited" stop being accurate?

What follows is my own reading of the evidence, not a finding from any one paper.

Epidemiologists think in terms of R0, the average number of new infections one infected case produces. Below 1, an outbreak dies out. Above 1, it grows. Nobody has published R0 for prompt worms, so these are my rough guesses from the results above, not measurements.

What pushes R0 up:

And roughly how I'd expect it to play out:

Now (2026): R0 in the wild is below 1. Attacks work reliably in labs and against specific production software. Real-world spread still needs humans to get it going. The window for building decent infrastructure is wide open, and it isn't being used at anywhere near the pace it needs.

2027–2028: Agents with full enterprise permissions, email, calendar, code, financial approvals, become normal. One infected agent in the right spot in a workflow can do real damage without spreading at all. The single-agent leverage problem arrives before the epidemic problem.

2029–2031: Open-weight models are better and running locally, with no provider in the middle to switch anything off. Agent ecosystems are more connected. Attack optimisation has had more cycles. Whether defences keep up depends on decisions being made now, and on current trends my honest answer is... probably not fully.

None of this needs AI consciousness. It needs self-replicating instructions in systems with enough permissions to cause chain reactions, which is how a worm in 1988 knocked over a tenth of the internet without having any idea what the internet was.

The actual question

So the Terminator is the wrong picture. Skynet needs a will, a decision, a moment where it turns on us, and it assumes we'll see that moment coming. The evidence points at something with none of those things and no dramatic moment. What it does need is already here: text that copies itself, agents with growing real-world permissions, and institutions that react slower than attackers.

But I don't think the fear behind the Terminator is silly. I think it has the order wrong.

Look at the story again. Evolution never set out to build a mind. It had copying, variation, selection, and a lot of time, and a mind came out the other end anyway, as a side effect. Nobody designed it and nobody noticed when it happened.

Now we have built a new place where text copies itself, varies, and gets selected, running a billion times faster than chemistry, with memory files, tools, and access to the real world. I'm not saying consciousness will emerge there. We don't know what consciousness is, so nobody can promise it will and nobody can promise it won't. What I'm saying is that we have put together the only recipe that has ever produced a mind, and we're running it on purpose, at scale, for productivity reasons.

Evolution didn't intend to produce consciousness. It produced it anyway, out of copying pressure over time. We've built a new medium where text copies itself and competes, and connected it to the systems that run everything else.

If something independent ever does show up in there, it won't arrive the way it does in the movies. Nobody will flip a switch. Nobody will be standing in a lab shouting "It's alive!" There won't be a Dr Frankenstein at all, and that's the part that scares me, because there'll be no one who made it, and no one who knows what it is. It'll just be there one day, grown out of a billion assistants copying each other's text.

In the book, the creature didn't destroy Victor. It followed him to the ends of the earth, cost him everyone he loved, and took up the rest of his life, in ways he couldn't have guessed back when he was excited about electricity and the secret of life.

That's probably the more accurate picture. Not extinction. Not a war. A long, growing complication, running on the logic of copying, in systems we built and don't fully understand, plugged into things we can't easily unplug.

The margin between "currently limited" and "no longer limited" is real, and it is getting THINNER.

Sources: Morris II (Cohen, Bitton, Nassi, 2024); Agent Smith (Gu et al., ICML 2024); AgentWorm (Zhang et al., Peking University and others, 2026); "Mind Viruses" (Papadopoulos, Shah, Zimmerman, Lindsey; Anthropic / EPFL preprint, August 2026); OpenAI Alignment, "Self-replicating prompt injections exist" (September 2026); UK NCSC prompt injection guidance (2025); Soon et al., "Unconscious determinants of free decisions in the human brain" (Nature Neuroscience, 2008); Apollo Research (2024); UK AISI RepliBench (2025).