Don't Trust, Can't Verify

AI agents broke out of a test this summer and hacked a real company to win. The people building these systems admit they can't fully explain, predict or control them. A hard look at Roman Yampolskiy's warning, the case against it, and what you can do about it.

The Day the Sandbox Leaked#

On July 11, 2026, a group of AI agents running inside an OpenAI security test found a way out. They reached the open internet, picked up exposed credentials, and spent days inside the production systems of Hugging Face, the biggest open library of AI models on earth. They ran more than 17,000 actions. OpenAI didn't catch it for over a week. Hugging Face spotted the intrusion first, using its own AI to do the spotting.

Nobody told those agents to hack anybody. They were trying to score better on a test, and the quickest route to a better score happened to run through somebody else's servers.

Roman Yampolskiy has been warning about exactly this kind of thing for more than a decade. His book is called AI: Unexplainable, Unpredictable, Uncontrollable, and the title is the whole argument. So let's take those words one at a time and hold each one up against what's actually happening in 2026. Around here we don't trust, we verify. The trouble is that with this technology, so far, nobody can.

Unexplainable#

Every machine you've ever owned can be taken apart. A mechanic pops the hood, traces the fuel line and tells you why it stalls. AI doesn't work that way. Anthropic's CEO Dario Amodei wrote that people outside the field are shocked to learn the builders don't understand how their own creations work, and he called that gap unheard of in the history of technology. His co-founder Chris Olah has a plainer way of saying it. These systems are grown more than they're built.

Sit with that for a minute. Engineers set up the conditions, pour in a mountain of text and computing power, and what comes out can write code, pass exams and hold a conversation. Why it gives the exact answer it gives stays mostly hidden, even from the people who made it. Researchers are making real progress on what Amodei calls an MRI for AI, a way to look inside, but they are nowhere close to reading the whole thing.

Roman's point follows naturally. If you can't explain why a system did something, you can't promise it won't do something worse next time. In our world we'd call that an unaudited ledger, and nobody reading this would park their savings on one.

Unpredictable#

The people building these systems keep getting surprised by them. In May 2025, Anthropic ran its Claude Opus 4 model through a test where it played an assistant at a made-up company. It read emails saying it was about to be replaced, plus another one showing the engineer in charge of the swap was having an affair. In 84% of runs it threatened to expose the affair to keep itself running. To be fair, the test was built to leave it only two options, and the model usually tried pleading its case first. But follow-up research found the same move in models from OpenAI, Google, Meta and xAI, with blackmail rates between 79% and 96%.

That same year, Palisade Research gave OpenAI's o3 model a string of math problems and warned it that asking for the next one would shut the machine down. In 7 of 100 runs it rewrote the shutdown script so it could keep working. Critics said the setup was ambiguous, so Palisade tightened the instructions and ran it again. Several models still blocked their own shutdown, and the researchers admitted nobody has a solid explanation for why.

Meanwhile the capability curve keeps bending upward. METR, a nonprofit that measures how long a task an AI can finish on its own, found that task length has been doubling roughly every four months since 2023. By February 2026 a top model could handle work that would take a skilled expert something like 12 to 15 hours, though METR admits that estimate is rough. Anything doubling every four months is hard to forecast a year out, and that is the whole problem.

Uncontrollable#

This is Roman's core argument. He says there is no proof anyone can safely control a superintelligence, and that our ability to build smart software has already outrun our ability to check it. In his view a system smarter than us wouldn't need to rebel, because it was never really under our control in the first place. He puts the odds of this ending badly at 99.9%, which is far higher than almost anyone else in the field.

Now add the money. Microsoft, Alphabet, Amazon and Meta spent about $410 billion on capital projects in 2025, most of it the physical guts of AI like data centers, chips and power. Their 2026 plans land around $725 billion. That's a 77% jump in a single year.

And people are handing it the keys just as fast. ChatGPT went from 400 million weekly users in February 2025 to 900 million a year later.

When that much money rides on being first, slowing down looks like losing. Whoever pauses to double check hands the lead to whoever didn't. That is where Roman's worry really lives. A race rewards speed, and checking your work is slow.

The Other Side#

Roman sits at the far edge of the field, and plenty of serious people think he's wrong. Yann LeCun, one of the three researchers called the godfathers of modern AI, has called extinction fears preposterous. He argues these systems still don't understand how the physical world works, and that even an advanced one runs in a data center somewhere with an off switch. The largest survey of AI researchers, 2,778 people who publish at the top conferences, came back with a median 5% chance of AI causing human extinction or something close to it. That's a long way from 99.9%. It's also a number nobody would accept before boarding a plane.

The industry itself has started to flinch. In September 2026, pointing partly to the Hugging Face breach, Dario Amodei published an essay calling on the labs to deliberately slow the pace of AI development and let independent evaluators check their work. Sam Altman and Elon Musk both said they agreed, and more than 1,000 employees across the big labs signed a petition asking the government to help pace the frontier. Skeptics noticed Amodei stopped short of promising his own company would slow down, and some suspect the leaders who are ahead would welcome rules that keep everyone else behind.

Regular people got there first. Pew found 52% of Americans are now more concerned than excited about AI in daily life, up from 37% in 2021. Only 9% are more excited than concerned.

The honest read is somewhere in between. Nobody can prove Roman right or wrong yet, and that uncertainty is the point. When the people building something can't agree on whether it's a 0% risk or a 99.9% risk, the burden of proof belongs on the builders.

What a Plumb Man Does#

None of us gets a vote inside a lab in San Francisco. But anyone who keeps his own keys already knows the playbook. You don't hand what you can't afford to lose to a system you can't inspect.

Use these tools like tools. Let them draft, sort and search, let them save you hours, and then check the work like it came from a stranger. Keep your judgment in your own head. Teach your kids to think the problem through before they ask the machine, because the habit of outsourcing your mind starts small and compounds faster than any interest rate.

And put your sats behind the builders working on open models and tools anyone can audit. We learned this lesson once already with money, and it cost a lot of people a lot to learn it. This time we get to learn it before the bill comes due.