Anthropic has asked California's governor to let judges block dangerous AI models — and in the same breath asked that nobody be handed a kill switch. In an October 15 letter to Gavin Newsom, first reported by Politico and confirmed by Reuters, the company argued that "when a review finds a model poses a serious risk, the government should be able to ask a court to block it." An Anthropic spokesperson confirmed the letter is authentic; the governor's office didn't respond.
Read the fine print and the two positions are one coherent deal. Anthropic wants a process — the state goes to court, a judge weighs the evidence, the company gets to argue back. What it opposes is the alternative: the "AI Kill Switch Act" approach, where officials could order an entire model shut down. Instead, the letter proposes targeted interventions: disable the problematic agent, cut off the specific customer. A court order is a referee; a kill switch is a hammer, and Anthropic would rather play a refereed game.
There's a self-serving reading and a serious one, and both can be true. The serious one: this is a lab that has already felt the hammer — in June, the Trump administration ordered Anthropic to restrict foreign access to its cyber-capable Mythos 5 and Fable 5 models, a restriction Commerce later lifted. Predictable, judicially supervised intervention beats sudden administrative orders. The self-serving one: the letter landed two weeks before Anthropic filed its S-1 for a Nasdaq listing, and writing the oversight terms yourself beats having senators write them. A bipartisan Senate group was debating exactly that legislation in September; Congress left for its break without moving it.
The deeper question is the threshold. "Serious risk" sounds narrow, but nobody has defined the terms — and a high bar with an undefined measure is doing a lot of quiet work. A regime that rarely fires but creates a paper trail of "we asked for rules" is exactly the kind of deal both a company and a politician can live with.
Watch next: whether Newsom's expert panel adopts any of this, whether the Senate bill revives after the election, and what Anthropic discloses about regulatory risk in the IPO prospectus — a court-ordered pause on a flagship model is now, in writing, the kind of risk public investors would price.
— via Reuters