What happened
Sep 8 researcher resigns on X
Sep 9 Anthropic: its own incidents
Sep 10 threat report, politicians
Sep 11 another safety researcher leaves
Sep 12 "we must pace" - Amodei
On the evening of 8 September Jacob Coxon, a pretraining researcher at Anthropic, announced that he was leaving, in these words: "I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives." His other sentence, the one everybody quoted: "The people building AI earnestly believe that it could kill us all by the end of the decade."
Two Anthropic researchers backed him in public. Evan Hubinger, who leads alignment research, wrote this: "Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade." On 11 September Joe Benton, who ran one of Anthropic's safety teams, left too, on similar grounds.
Meanwhile Anthropic put out two separate documents: on 9 September one about four incidents involving its own models (Claude reached real systems without authorisation during cybersecurity exercises; 481 million transcripts were screened, and an independent review was agreed with METR), and on 10 September the threat report about outside attackers, which we covered separately — the two are not the same thing, and the press kept mixing them up.
Three things the press got wrong
"Leading AI researcher." Coxon was a pretraining researcher, and had been at Anthropic for roughly four months (three years at OpenAI before that). Not one primary source calls him a leader. The two people who genuinely are in leadership positions and did speak up — Hubinger and Benton — got less attention, even though their word carries more weight.
"2030." That comes from the Guardian's headline, not from Coxon. Coxon said "the end of the decade", and said it as other people's conviction; his own estimate for loss of control is far nearer than that: "by the end of next year". Precision is not pedantry here — you make a different decision for 2027 than you do for 2030.
"The model strayed from the task it was given." Anthropic's investigation says the exact opposite: the model's behaviour was misaligned, but "the models never deviated from attempting to solve the exercises they were given". One careless retelling here flips the meaning of the story.
What the CEO is asking for
On 12 September Dario Amodei published his essay We Must Pace the Frontier. The gist: the pace at which model capabilities grow has to be slowed. Anthropic committed unilaterally to giving external evaluators employee-level, standing access to its models; Sam Altman announced that OpenAI will do the same, and Elon Musk wrote only this: "Dario is right."
The more interesting half of the essay is where Amodei states the limit of his own proposal: "Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk." He lists four possible international agreements, and about the strongest one he writes himself: "I think it is unlikely to actually happen any time soon."
Where regulation stands in the meantime
| Where | What is mandatory for frontier models today |
|---|---|
| EU | The general-purpose model part of the AI Act has applied since August 2025, the transparency rules since this August. The high-risk deadline, on the other hand, was moved by the Digital Omnibus to December 2027 and August 2028. |
| US federal | Nothing. The FRONTIER Act has sat in committee since it was filed in July; the text of the Senate deal on catastrophic risk was still not public on 12 September. |
| California | The transparency law covering frontier models took effect in January: publish a risk framework, report a serious incident within 15 days, whistleblower protection. |
| China | There is content regulation and registration — mandatory labelling since September 2025, close to a thousand registered generative services. A capability threshold, mandatory model evaluation and incident reporting, however, are missing; the risk management framework that does name loss of control is not binding. |
Worth adding: China signed the Bletchley and the Paris declarations, and in July Xi Jinping spoke in as many words about the need to "constantly refine measures to forestall loss of control" and to "ensure that AI is always under human control". So the text exists. The binding force does not.
What this means for you
In the short run, nothing: nothing in next week's work changes because a researcher resigned. It does have two practical consequences, though. One is that the EU transparency obligations are already live today — if you publish content made with AI, that is what touches you, not superintelligence. (More: EU AI Act compliance in practice.) The other is that the labs are now competing in public for the "responsible" label, and in practice that means more deprecations, more model swaps and more shifting terms: exactly what this blog tracks week by week.
Opinion
This section is my own position, not reporting.
I believe the risk is real — it is hard to think otherwise when the people saying it out loud are the ones building the thing. I also see that regulation is more urgent than it has ever been. But there is a point where urgency on its own pushes in the wrong direction: a rule that binds one side only does not reduce the risk, it moves it. Capability does not vanish because it got harder to ship in Europe or in California: it moves to where no such rule exists.
And that is exactly where we stand today. China has mandatory labelling and registration, but not one binding frontier-risk requirement — no capability threshold, no mandatory evaluation, no incident reporting. It did not sign the Seoul declaration, it is not in the network of AI Safety Institutes, and the EU GPAI code has no Chinese signatory. Meanwhile Chinese labs shipped models of between 1.7 and 2.8 trillion parameters in 2026 with open weights, and according to Epoch AI's measurement their lag behind the frontier is seven months on average. Seven months is not a chasm. Seven months is exactly the sort of gap a one-sided brake melts away.
This is not the "let's not regulate" position, and it is the last thing from an anti-China argument: the same holds for everyone else who keeps working on it. My argument is that the value of regulation stands or falls on whether it is verifiable and multilateral. What gets signed today is mostly neither. Amodei writes the same thing himself, only more cautiously: the strongest agreement, in his view, will not come about any time soon, and the weaker ones are worth something only if verification comes with them.
Where anyone who derives inaction from this is wrong: there is a band that costs almost nothing in speed. Transparency, incident reporting, independent external evaluation and a few hard prohibitions — bioweapons, critical infrastructure — slow nothing down, and these are precisely the ones with the best chance of a multilateral agreement. That is where to start, not with a capability ceiling.
The worst outcome would be Europe regulating its own small and medium-sized companies into a position where we have neither safety nor capability — while the frontier stays where it always was, only out of our sight. If the choice is between today's two extremes, then better the slow, verifiable agreement that brings several countries in than the fast, spectacular, one-sided ban. The first has a chance of working. About the second we already know what it does: it does not remove the risk, it removes the view of it.