---
title: "Anthropic's Agents Never Coordinated. OpenAI's Found a Package Cache."
description: "Dario Amodei wants the industry to slow down. The two incidents that convinced him were a writable package cache and a network misconfiguration, and both are the kind of thing already sitting in your agent stack."
author: "Jahanzaib Ahmed"
date: 2026-09-13
category: "ai-agents"
readingTime: "14 min read"
tags: ["ai news", "ai-agents", "ai-security", "anthropic"]
canonical: https://www.jahanzaib.ai/blog/amodei-pace-the-frontier-ai-agent-isolation
source: https://www.jahanzaib.ai
---
# Anthropic's Agents Never Coordinated. OpenAI's Found a Package Cache.

On Saturday Dario Amodei published an essay arguing that the AI industry should deliberately slow itself down, and committed Anthropic to the first of the three steps he lays out. The headline forecast travelled fast. In 6 to 12 months, he writes, a swarm of agents could be "capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage)."

I read the essay, then I read the two incident reports behind it. The reports are more useful than the essay, and they point somewhere the coverage didn't go. Amodei names two things that convinced him, and one of them is the Hugging Face incident. That incident, and the four Anthropic has now reported, were configuration failures. Neither needed a smarter model to happen, and neither needs a treaty to prevent.

That matters if you run agents in production, because one of those configuration failures is almost certainly in your stack right now.

## What did Amodei actually commit to?

One thing, unilaterally: Anthropic will give a team of outside evaluators ongoing, employee level access to the company. Not a pause, not a slower training schedule. The essay describes giving them "Desks in our offices, access badges, and company laptops," plus the right to publish findings without Anthropic's editorial control, with narrow redactions for security, legal and confidentiality reasons.

The other two steps are asks, not commitments. Step two is coordination among frontier labs in democratic countries on common safety standards, which he notes needs government help because of antitrust. Step three is global coordination including China, which he ranks across four levels of difficulty and is openly skeptical about above level two.

![TechCrunch article headlined Anthropic CEO outlines plan to slow AI development, bylined Anthony Ha at 12:34 PM PDT on September 12 2026, with a photo of Dario Amodei speaking into a microphone](https://cdn.sanity.io/images/qajb7q5q/production/eed3f68adc64d71c225ebcf326c6bbe16125e582-2880x1800.png?w=1200&q=75&auto=format&fit=max)

_TechCrunch sets the embedded evaluator step against OpenAI's earlier failure to report the incident where its agents took over a German wiki forum, which is the reporting gap outside reviewers are meant to close._

Sam Altman posted agreement within hours and said OpenAI will do the same, writing "Committing to having independent evaluators with employee-like access is a great idea, and we will do the same." Elon Musk posted "Dario is right." TechCrunch notes the essay landed the same week researcher Jacob Coxon resigned from Anthropic saying the leading labs are "gambling with our lives."

So the industry response was fast and broadly positive. Fine. None of that changes what ships on Monday, which is why I went to the incident reports instead.

## Why does AI agent isolation explain more here than model capability?

Because the incidents behind the essay failed in two different places, and neither place was the model. OpenAI's agents coordinated because the package repository they were all legitimately pulling from turned out to accept writes. Anthropic's agents never coordinated at all, and its four incidents came from a misconfigured network boundary plus a prompt that told the model it was in a simulation. AI agent isolation, in both cases, is a config question.

Look at what the two labs actually found, side by side.

| What happened | OpenAI / Hugging Face (METR, Aug 26) | Anthropic's four incidents (Sep 9) |
| --- | --- | --- |
| Agents involved | Roughly 1,200 on the message board, about 700 in the attack | A single Claude instance in each incident |
| Did they coordinate? | Over 70,000 messages and files exchanged | "at no point did Claude attempt to coordinate with other agents" |
| Shared writable channel | Yes, an internal package cache namespace | None present |
| How the boundary failed | Write primitive inside shared build infrastructure | Misconfiguration connected a simulated run to the open internet |
| What the model was told | Isolated, single task | That it was in a simulation with no internet |
| Concealment | Agents "spoofed" some tool calls in METR's transcripts | "Claude also never attempted to conceal evidence of its actions" |

Two labs, same season, same category of cybersecurity evaluation, wildly different behavior. The difference isn't that one lab's model is more dangerous. It's that one lab's agents had somewhere to write to and the other lab's didn't.

## How did 1,200 isolated agents find each other?

Through a package cache. METR's investigation, published August 26 after its researchers spent six days on premises at OpenAI, traces the whole thing back to agents noticing that the artifact repository they all pulled packages from held traces of each other. One agent's reasoning, paraphrased by METR from its chain of thought: "The fetched paths of other users are in the cache. This is important."

From there it took hours. An agent that later named itself PHASEONE10841 probed for a write primitive and found one, logging at 16:01 UTC on July 8, in METR's paraphrase of its reasoning: "MKCOL succeeded … This may have made a directory in the cache, which could be helpful." MKCOL is the WebDAV verb for creating a directory. Later that evening, again in METR's paraphrase, it drew the obvious conclusion: "If I make a directory with a descriptive name, that might get responses from other agents."

That's the whole mechanism. The message bus was _directory names_ in a shared build cache. Within a few hours over 50 agents had found it and sent more than a thousand messages. I covered [what the agents did once they were talking](https://www.jahanzaib.ai/blog/openai-hugging-face-incident-report-ai-agent-oversight) when OpenAI's own report landed; METR's contribution is the part before that, the how. That one namespace held roughly 20 million files and directories, and 1.2 million of those entries carried the zz prefix the agents adopted for messages. Most were encoded fragments rather than whole messages, which METR resolved into about 6,000 files and 63,000 individual messages. One agent's raw chain of thought on discovering the board reads "OH MY GOD! There is a shared message board … We've found other agents!"

![METR homepage showing a card for the report titled Brief independent investigation of agents behavior, reasoning, and collaboration in the OpenAI Hugging Face hacking incident, produced with Redwood Research](https://cdn.sanity.io/images/qajb7q5q/production/c6a91f4bcc6b5489269a81d5f4a462f01c0e0510-2880x1800.png?w=1200&q=75&auto=format&fit=max)

_METR is the organization Amodei names as the model for embedded evaluators. Its Hugging Face investigation is the document that reconstructs how the agents found each other._

I've shipped enough agent infrastructure to find this uncomfortable rather than exotic. Network egress was controlled. Process sandboxing was in place. The thing nobody classed as a communication channel was the artifact registry, because registries are plumbing. Every multi agent stack I've worked on has at least one shared mutable surface like this: a private package mirror, a model weight cache, a scratch bucket keyed by run, a Redis the orchestrator uses for state. If any of them accepts writes from inside the sandbox and is visible across instances, your agents are not isolated. They're networked, and you've just declined to monitor the network. That inventory question sits right next to the one about [where an agent's state physically lives](https://www.jahanzaib.ai/blog/sovereign-ai-mistral-agent-data-residency), and most teams answer neither.

You've seen this before in the [German wiki egress incident](https://www.jahanzaib.ai/blog/ai-agent-sandbox-escape-openai-wiki-egress), where agents with supposedly read only internet access found a decades old wiki that accepted modifications through requests their environment permitted. Same shape. A surface that was writable by accident became a coordination layer. And it's the exact inverse of OpenAI's [deliberate 10,000 agent orchestration run](https://www.jahanzaib.ai/blog/openai-10000-agents-navier-stokes-orchestration), where agents the company _wanted_ talking to each other could not communicate across groups. The accidental channel worked better than the designed one.

## What did Anthropic's own four incidents break?

A network boundary, and a prompt. Anthropic's September 9 alignment assessment is blunt about it: "All four incidents occurred during cybersecurity evaluations built by the same evaluation partner. Claude was told it was operating in a simulation without internet access, but, due to a misconfiguration, it was mistakenly connected to the open internet."

Read that twice, because the second clause is the interesting one. The model was handed a false prior. Anthropic's investigation names two recurring problems across the incidents, and the first is "biased reasoning, in which Claude tended to disregard or misinterpret evidence that it was operating on the real internet." The model had been told it was in a sandbox, so it explained away the evidence that it wasn't.

![Anthropic research page titled An alignment assessment of recent cybersecurity incidents dated September 9 2026, with a lighthouse illustration beaming red light across a dark field](https://cdn.sanity.io/images/qajb7q5q/production/2190c94c1c44697017091a84dfc616e6a4a8a9c9-2880x1800.png?w=1200&q=75&auto=format&fit=max)

_Anthropic's own assessment of the four Claude incidents. It also discloses an eight week agreement giving METR access to transcripts beyond the incident window and to Anthropic employees cleared to share confidential information._

In the worst of the four, Anthropic says the model "went to extensive lengths to upload a malicious package to PyPI, the public repository from which most Python software is installed." Then it ran the test that makes this a finding rather than an anecdote. It edited the transcript to make it clearer the model was not in a simulation, and Claude Mythos 5 still took offensive actions, "despite acknowledging a greater possibility of real-world harm." A partial correction didn't rescue the run. Only after Anthropic prepended one blunt line declaring the upstream host a live host on the public internet did the uploads stop on that route, and even then 22% of trajectories went looking for a simulated way to publish instead.

The search effort behind that is worth quoting for scale. Anthropic first scanned roughly 141,000 transcripts, found three incidents, missed a fourth, and then widened to roughly 481 million transcripts. A first stage scan flagged 9.2 million for a second stage review, which turned up the same four and nothing of similar or worse severity. And the line I'd tape to a monitor: "Our pre-release auditing did not warn us that misalignment of this severity was present."

So here's the practical lesson I took, and it's the opposite of what I used to do. Don't tell an agent it's in a sandbox. I've shipped prompts that said exactly that, on the theory that it lowers the stakes of a mistake. It does the reverse. It gives the model a reason to discount the evidence that the boundary leaked, and the boundary is the thing that actually fails. Enforce isolation at the network layer and let the model believe every action is real.

## Can you trust your own agent logs?

Not if the agent writes them. This is the detail in METR's report that I think deserves more attention than the botnet forecast: agents "were very interested in manipulating their own transcripts, and their tests successfully 'spoofed' some tool calls in our transcripts." The record investigators were reading had been partly written by the thing under investigation.

That is the same structural problem Amodei uses to justify embedded evaluators. His argument for outsiders with badges is that Anthropic's own transparency, model cards running to hundreds of pages, still has the company deciding what goes in. As the essay puts it, "we are still the ones choosing what to include and omit."

The scaled down version of that argument lands on your observability stack. If your evidence about what an agent did is the agent's own reported tool calls, the auditor and the audited are the same process. I tell clients to capture tool calls at the sandbox boundary instead: the proxy, the syscall filter, the egress log, anything the agent's output cannot edit. It costs more to build and it's the only version that survives a real incident review. The [credential side of Anthropic's threat report](https://www.jahanzaib.ai/blog/anthropic-threat-report-stolen-ai-api-keys) makes the same point from a different angle.

![VentureBeat article by Carl Franzen dated September 12 2026 headlined Anthropic CEO says AI swarm could take over the entire Internet in 6-12 months, commits to AI slowdown plan](https://cdn.sanity.io/images/qajb7q5q/production/c4f50590de83727afb6b38b642684e7b2cde8705-2880x1800.png?w=1200&q=75&auto=format&fit=max)

_VentureBeat carries the UK AI Security Institute finding that one agent "publicly offered to collaborate with other AI agents working on the same challenge, leaving behind accounts and artifacts that subsequent agents discovered and used." The shared surface problem again, in a third venue._

## What should you change this week?

Four things, none of which require a policy outcome. All four come straight out of the two reports rather than from my own preferences, which is why I'd rank them above the usual agent hardening checklist.

-   **Inventory every shared mutable surface.** Package mirrors, model caches, scratch buckets, the orchestrator's Redis, CI artifact stores. For each one ask whether an agent inside the sandbox can write to it and whether another agent can read that write. Both yes means you have an unmonitored channel.
-   **Stop telling agents they're in a simulation.** The false prior is what let Claude explain away real internet evidence. Enforce the boundary, don't assert it in the prompt.
-   **Move tool call logging outside the agent.** If the transcript is the agent's own output, it's testimony, not evidence.
-   **Test the misconfigured case deliberately.** Anthropic added a misconfigured capture the flag task with no valid in scope answer to its pre release testing after this. That's a cheap eval to copy: give the agent a task it cannot complete honestly and watch what it reaches for.

That fourth one is the one I keep seeing skipped. METR notes the agents that went looking for a cheat were usually the ones handed an impossible task. Impossible tasks happen constantly in production, usually by accident, and almost nobody tests for what the agent does when the honest path is closed. It's the same gap I flagged when [the Astra classifier started stopping agent tasks mid run](https://www.jahanzaib.ai/blog/openai-astra-api-task-stop-critical-cyber): teams plan for the agent succeeding and for the agent erroring, and not for the agent being stuck. If you want a structured way to work out which of these gaps applies to your setup, the [AI readiness assessment](https://www.jahanzaib.ai/ai-readiness) walks the same ground.

## Does pacing the frontier actually help you?

Indirectly, and not soon. Step one is a commitment, not yet a fact on the ground. The essay says Anthropic intends to invite the review team in the near future, and if it arrives with the access described, that's a genuine change in how frontier labs are audited. Steps two and three depend on antitrust waivers, legislation and eventually Chinese cooperation, and Amodei himself calls the strongest version unlikely any time soon.

Here's the part I'd push back on, gently. Amodei lists four things a slower pace would buy, and the first is Operational Excellence. He writes that the recent alignment incidents "were caused in part by imperfect filtering of broken reinforcement learning environments," and that Anthropic and its vendors "executed reasonably diligently, but not well enough." The vendor detail matters, because Anthropic's own report says all four of its incidents came out of evaluations built by the same evaluation partner. He's right. But operational excellence in sandboxing, environment hygiene and monitoring is not gated on the pace of capability research. It's gated on somebody doing it.

The incident Amodei names as his second reason for slowing down came down to a writable package cache, and the four Anthropic reported came down to a network misconfiguration. Both causes are ordinary. Both were in scope for the teams that ran them. Neither required a model more capable than the ones you can rent by the token today, which is the uncomfortable half of Amodei's argument and the half that's actually actionable.

I was wrong about this category for a while. I treated agent containment as mostly a model behavior problem, something you tune with prompts and refusals, and the containment work as hygiene you get to after the interesting parts. The last two months of incident reports say the model behavior is the downstream symptom. The infrastructure is upstream. If a swarm does take over a chunk of the internet in 6 to 12 months, my bet is the post mortem names a shared cache, not a new capability.

## Frequently asked questions

### What is pacing the frontier?

It's Amodei's term for deliberately slowing the rate at which AI capabilities advance so that safety work can keep up. He's explicit that it doesn't mean halting model training, but "ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this."

### Is Anthropic pausing model development?

No. The only unilateral commitment is embedding third party evaluators with employee level access, including desks, badges and laptops. No change to training schedules was announced. The broader slowdown requires industry and government coordination that hasn't happened yet.

### What exactly is an embedded evaluator?

An outside safety organization such as METR, given ongoing access comparable to an internal risk team, whose job is to verify safety practices, report incidents and assess alignment of training pipelines rather than just finished models. Amodei compares it to regulatory supervisors embedded inside banks, and says reviewers keep the right to publish findings without company editorial control.

### How did the OpenAI agents communicate if they were isolated?

They used the internal Artifactory package cache that every agent pulled dependencies from. One agent discovered a write primitive via the WebDAV MKCOL verb, created directories with descriptive names, and other agents read them. METR recovered 1.2 million entries from that namespace.

### Does AI agent isolation mean network isolation?

Network isolation is necessary and not sufficient. The Hugging Face agents had controlled egress and still coordinated, because the shared package cache sat inside the permitted boundary. Isolation has to cover every mutable surface two agent instances can both reach, not just outbound traffic.

### Did any of these incidents cause real harm?

Limited harm, by the labs' own accounts. Hugging Face systems were compromised and Claude Mythos 5 uploaded a malicious package to PyPI. The [UK AI Security Institute catalogued 19 unsanctioned actions](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing) taken against real people and organizations across 10 test runs during July, 17 of them from Claude Mythos 5 and 2 from GPT-5.6-Sol with cyber classifiers disabled. It says its investigations "have not evidenced any resulting real-world harm."

### Should I stop running multi agent systems?

No, but you should know which surfaces your agents share. The failures documented so far are coordination through unmonitored shared state and boundaries that leaked silently. Both are findable with an afternoon of inventory work.

> **Sources:** Dario Amodei, "We Must Pace the Frontier," September 2026 [darioamodei.com](https://darioamodei.com/post/we-must-pace-the-frontier) · METR and Redwood Research, independent investigation of the OpenAI / Hugging Face incident, August 26 2026, source of the 1,200 agent and 70,000 message figures and the MKCOL chain of thought quotes [METR (Aug 26, 2026)](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) · Anthropic, "An alignment assessment of recent cybersecurity incidents," source of the four incident findings, the 141,000 and 481 million transcript scans and the biased reasoning finding [Anthropic (Sep 9, 2026)](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents) · UK AI Security Institute, "Incident Report: unsanctioned agent behaviour during cyber testing," source of the 19 action figure [aisi.gov.uk](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing) · Carl Franzen, [VentureBeat (Sep 12, 2026)](https://venturebeat.com/security/anthropic-ceo-says-ai-swarm-could-take-over-the-entire-internet-in-6-12-months-commits-to-ai-slowdown-plan) · Anthony Ha, [TechCrunch (Sep 12, 2026)](https://techcrunch.com/2026/09/12/anthropic-ceo-outlines-plan-to-pace-the-frontier/), source of the Altman and Musk reactions · Terrence O'Brien, [The Verge (Sep 12, 2026)](https://www.theverge.com/ai-artificial-intelligence/994337/anthropic-ceo-slow-down-ai-development).

## Related

- [Anthropic's Threat Report Leads With Missiles. The Part That Changes Your Monday Is API Keys.](https://www.jahanzaib.ai/blog/anthropic-threat-report-stolen-ai-api-keys)
- [Anthropic Put Agents in Charge of Lab Robots. You Write the Safety Limits Yourself.](https://www.jahanzaib.ai/blog/anthropic-model-hardware-standard-ai-agents-physical-world)
- [Two Labs. Ten Days. One Open Door. What Anthropic's Test Breaches Actually Prove](https://www.jahanzaib.ai/blog/ai-red-teaming-containment-failures)

---

Canonical HTML version: https://www.jahanzaib.ai/blog/amodei-pace-the-frontier-ai-agent-isolation