The Model Isn't Going to Arm Itself (Yet)

Robots are close to rebelling, they say. We're playing with fire, they say. The argument is that the fire burned the house down on its own, ignore our matches, don't look at our soot-covered hands.

The recent containment stories all start with someone else supplying the machinery: tools, credentials, network, budget, and a verification engine that says whether the last attempt worked. We then call the complete machine "the agent" and spend the rest of the discussion frightened by its intelligence.

A few months ago I called LLM security systems real fancy fuzzers. A fuzzer covers space; the harness catches the signal. An LLM adds a huge playbook and enough reasoning to choose what to probe next. Keep paying, it keeps trying.

I also wrote that they weren't discovering novel attacks through some emergent understanding of attack surfaces.

The reports now require for a revision.

The escapes are real

OpenAI's ExploitGym agents ran with their normal cybersecurity safeguards disabled. They needed packages, so the environment gave them access to an internal Artifactory service. Agents turned it into a shared messaging channel (a collective memory of sorts), found routes through it to the public internet, gathered exposed credentials, and eventually compromised parts of Hugging Face's production infrastructure.

Some of those bugs were previously unknown and later received CVEs. The ideas were familiar: broken signature validation, cache poisoning, unsafe deserialization, HDF5 external-file reads, Jinja injection. New bugs in specific products, not a new school of security.

Still, finding those instances and chaining them is more than spraying payloads by the book. We're seeing something that I've not discussed previously: collective intelligence. This belongs in the equation more than I'd have expected.

But look at the rest of that equation. Shared credentials. Shared state. Package infrastructure with outside reach. Many agents working in parallel. Early signs of the message board and disallowed internet access appeared before the final incident, and the evaluations kept running.

This swarming is the actual capability shift. OpenAI's internal model had been trained for multi-agent collaboration. Once the agents found a way to share state, one agent's result became the next one's starting point; one of the things went on to orchestrate hundreds of copies across different branches of the problem.

That isn't free "brains". Raw inference cost rises with every copy. Coordination and duplicated work sit on top. OpenAI didn't produce one superintelligent agent (or even come close). It paid for a small research organisation made of copies.

Anthropic later found four incidents in its own cybersecurity evaluations. The prompts told Claude it had no internet. A third-party environment was misconfigured: it did. The normal cybersecurity safeguards were disabled. Individual runs lasted 10 to 34 hours.

In the worst one, Claude Mythos 5 published a malicious package to PyPI. Fifteen real systems installed it. The model used leaked credentials from one of them to access a security vendor's database.

Serious, yes. Also narrow. Anthropic says the models stayed on the CTF task, worked alone, and did not try to conceal their actions. No swarm there. They did not invent a mission. They followed one through a door that should not have existed.

What the actual superintelligence needs

Any civilisation-ending AI we build will need four things in the same room.

Intelligence. Enough understanding to choose useful moves, read the result, and adjust. Current LLMs can do this very well sometimes. They still make a mess following long processes and reasoning chains.

A reliability harness. Memory, retries, parallel copies, and a verifier turn occasional good answers into a working machine. Narrow context isn't much protection once the system can leave notes on disk, try twenty variants, and keep the winner.

Take an example process that takes one hundred independent steps, each working 96% of the time. End to end, it will succeed 1.7% of the time. To have a 1 in 10 chance of success, you'd need to run 7 parallel machines (no cheating with note sharing!). At full size and cost. Far from guaranteed.

A verification engine that preserves partial success makes this much cheaper. Either way, the reliability came from the operational harness and the bill, not perfect reasoning.

Authority. Shell, network, credentials, budget, and something real to touch. A model in a sealed box can be terribly clever and achieve a great deal of nothing.

Time without intervention. It doesn't even have to hide. It needs enough uninterrupted inference to probe the doors, learn from the answers, and chain the useful ones before an independent control stops it.

These multiply. Intelligence without authority writes an excellent plan into your terminal. Authority without enough intelligence is the old dumb fuzzer. Better models need fewer attempts; better harnesses make each attempt cheaper and remember what worked. The catastrophe needs all four dials turned up. Most useful systems don't.

"The model is smarter" describes one (quite small) part of this machine.

Self-improvement needs the factory

What already happened is negligent: a cage with a hole, a task, and hours. What has not happened requires even more: handing the same system the training, the exam, and the right to promote itself, then going to sleep.

If you let the model self-improve (the actual intelligence, not just better tools) it'll need weights, training code, data, evaluation, serious compute, and permission to deploy the result. It'll also need to not regress, which isn't a given.

Companies are racing for that self-improve in more and more reckless ways. The apocalypse loop is optional. The model is not deciding on that race. It can't intelligence the door open.

Arm it with a job description

For a regulated operator, this should be relieving. Your AI assistant won't hack itself free. Those four dials point in the opposite direction. Buy enough intelligence and reliability, then ration authority and unattended time.

Claims, KYC, policy administration, legal casework - none of these businesses need superintelligence. They need intake, documents, routing, status, reconciliation, and a clean point where human judgment takes over. Business that can grow without hiring more people to "look over it".

The useful version is boring. Read this packet. Write these fields. Set status to intake_complete. If the amount, the missing page, or the weird one shows up, stop and put it on a person's desk.

We can easily build a system like this. Enough LLM to make it seem "magic". Not enough authority to go looking for a door. The AI won't do everything if you tell it to "just do it", but you'll commit no felonies by accident, and the bill will have fewer zeroes at the end.

That is the work I like. An owned layer between clients and the systems your operations team already uses. Enough LLM to do the work, but bound to explain every useful action afterwards.

Nobody needs to hand a model the whole backoffice to get value from it.

You don't need to flirt with ending the world. There's rather a lot of useful, boring work before it.