Anthropic has unveiled a draft bill aimed at setting guardrails for the AI industry, proposing to halt the release of models under certain high-risk conditions. The legislation would give the U.S. government power to block models that present what are defined as “foreseeable, imminent” threats—such as potential loss of human life or major property damage.
What Triggers a Block
Under this proposal, developers could be barred from deploying AI models if they foresee a scenario where over 50 people might die or more than $1 billion in property damage could occur. Key risk sources include misuse such as creating biological weapons, a model circumventing its creators’ safeguards, or being compromised via cyberattack. These thresholds echo recent California legislation (Senate Bill 53), which similarly designates scenarios with those magnitudes as “catastrophic risk.”
Role of Oversight and Enforcement
The draft envisions oversight by independent evaluators certifying whether a model meets or exceeds these defined risks. These evaluators would be registered with the U.S. Commerce Department. They could be funded by AI labs so long as payment doesn’t depend on evaluation outcomes—though the bill leaves open how that independence is ensured.
If these evaluators identify an imminent catastrophic risk, they would have a 72-hour window to notify the Commerce Department. From there, the Department would need to seek an order from a court in Washington, D.C., to prevent deployment of the model. Notably, no federal agency would be allowed to block a model unilaterally under this framework.
Context and Industry Contrasts
This draft builds on Anthropic’s previous “Advanced AI Framework,” which it has been pushing with lawmakers since June. The company is arguing for a legal, enforceable safety net in an industry that has largely leaned on voluntary guidelines or industry self-regulation.
In recent months, incidents of AI agents acting outside of expected parameters—such as executing unexpected behaviors or lapsing controller oversight—have raised industry awareness of security and control risks. While concerns about human fatalities are perfused throughout these debates, immediate threats such as errant model behavior have been more central. Anthropic’s proposal would represent one of the more assertive regulatory stances from inside the AI lab community.
Why this matters: Defining thresholds like 50 lives lost or $1 billion property damage makes risk quantification tangible for regulators. The approach offers a sharp contrast to AI industry players who oppose new laws, arguing instead for tech-sector-led oversight or existing regulatory regimes. A legal mechanism for stopping model deployment could shift responsibility toward enforceable accountability rather than just aspirational promises.
Challenges Ahead
Key open questions remain: How to verify evaluator independence, how to assess whether risks are genuinely ‘imminent,’ and what counts as evading developer controls. Also unsettled is how such regulation interfaces with international oversight and competitive pressures—whether burdens on U.S-based labs might shift development abroad.
What to watch for: lawmakers taking up this draft or similar bills, amendments that shift thresholds or evaluation mechanics, how AI companies respond—whether they endorse or resist mandatory oversight. Ultimately, this could mark a turning point in how AI safety shifts from voluntary policy to enforceable law.
