Anthropic has quietly pitched Capitol Hill lawmakers on a draft bill that would govern when the government can pull the plug on a frontier AI model, and the threshold is staggering. According to Politico, Washington could block a model's release or use only if it poses a foreseeable, imminent risk of more than $1 billion in damages — or of killing or seriously injuring more than 50 people.
The lab is, in effect, writing the rules that would govern it. An Anthropic spokesperson told Reuters the draft reflects proposals in the company's "Advanced AI Framework," released earlier this year, calling them "sensible policies we believe Congress should act on."
Here's what the draft would actually do. Independent evaluators, registered with the Commerce Department, would periodically test frontier labs and their models — and the labs could pay those evaluators, just not in a way tied to the outcome. If an evaluator finds "imminent enumerated risk," it must report to Commerce within 72 hours. And the qualifying risks are narrow: a biological weapon's creation or release, a model escaping its developer's controls, or a cyberattack. Anything short of catastrophe wouldn't qualify for a stop.
Then comes the part that matters most. The Commerce Department couldn't shut a model down on its own; it would have to convince a federal judge in the District of Columbia. And the draft explicitly bars every other federal department, agency, or officer from suspending a model's development, deployment, or internal use. This is a bill about government power as much as model safety — one that would concentrate the kill switch in a single courtroom and take it away from everyone else.
That restriction has a backstory. Over the summer, US export controls reportedly sidelined two advanced Anthropic models, rattling the industry over how quickly officials could restrict access to powerful systems. Read in that light, the bill is a lab's answer to that experience: fine, give Washington a kill switch — but make the bar so high and the process so narrow that it almost never gets pulled.
The draft would also preempt certain state-level requirements on risk disclosures, independent evaluations, and incident reporting, which is likely to reopen the fight over whether Washington should set one national standard or leave stronger state protections alone. The timing is conspicuous: the same week, the FTC sent OpenAI and Anthropic formal 50-plus-page demands over safety incidents, the White House AI task force adopted Anthropic's charter, and Anthropic told investors Q3 revenue hit $16.8 billion as it heads toward an IPO.
Whether Congress takes the bait is another question. Majority Leader John Thune is working with Senators Klobuchar and Cruz on their own catastrophic-risk legislation, while some safety advocates want a stronger federal review process and others — Representatives Casar and Sanders — have called for pausing advanced development outright. But Anthropic has done something clever: it didn't wait for Washington to write the rules. It wrote the draft itself.
— via Reuters