Anthropic proposes AI safety bill requiring judicial approval to block dangerous models

Over the past year, concerns regarding the potential dangers posed by artificial intelligence (AI) have intensified, fueling heated debates about the appropriate path for technological safety. Now, Anthropic, one of the world's leading AI developers, has presented its own vision for how the United States government should regulate the sector. The company has drafted and is promoting a bill that establishes specific scenarios in which the release of an AI model could be blocked. However, the proposal includes a significant caveat: it prohibits federal agencies from stopping the release of a model without the express permission of a federal judge, and only if there is an imminent risk of death to more than 50 people or property damage exceeding $1 billion.

The details of Anthropic's proposal

The bill, which Anthropic has been quietly presenting to federal and state lawmakers since June 2026, is based on the company's own "Advanced AI Framework." The legislation proposes that the government can only intervene in the distribution of an AI model if an "imminent and foreseeable risk" of catastrophic proportions is proven.

According to leaked documents confirmed by company spokespeople, the criteria for defining this catastrophic risk are extremely specific. A block would only be justified if the AI could cause:

• Death or serious injury to more than 50 people.
• Property damage exceeding $1 billion.

Furthermore, the origin of this threat must be linked to three strict scenarios:

• The creation or release of a biological weapon.
• The AI model evading the controls of its developers (loss of human control).
• The facilitation or execution of a large-scale cyberattack.

Independent evaluators and the role of government

To determine if a model reaches this danger threshold, Anthropic's bill suggests the use of "independent evaluators." These professionals or entities would be registered with the U.S. Department of Commerce and would be responsible for routinely testing models before their public release.

One point that has sparked debate is the funding of these evaluators. The proposal allows AI labs to pay for these evaluations themselves, provided that "payment is not contingent on the results of the evaluations." Critics point out that this model could create conflicts of interest, questioning how the system would guarantee the auditors' total independence.

If evaluators identify an imminent catastrophic risk, the protocol would require them to notify the Department of Commerce within a maximum of 72 hours. From that point on, the Department would not have the authority to block the model on its own. Instead, the federal agency would be required to present the evidence to a federal judge in Washington, D.C., requesting a court order to halt the release.

Context and background: The race for regulation

Anthropic's move does not occur in a vacuum. The technology sector has been under increasing pressure to demonstrate that it can control the powerful tools it is building. The metrics proposed by Anthropic — 50 people and $1 billion — closely mirror legislation recently passed in California, known as Senate Bill 53, which defined similar scenarios as equivalent to "catastrophic risk."

Historically, AI labs have issued constant warnings about how the technology they sell could result in the end of human civilization. Ironically, these same companies have resisted mandatory and rigorous government oversight. Instead, the industry has suggested voluntary systems of embedded evaluators — "neutral" third-party observers placed within AI organizations to monitor signs of risk.

DeepMind CEO Demis Hassabis suggested months ago that a self-governing AI regulatory body should imitate the Financial Industry Regulatory Authority (FINRA). On the other hand, figures like Jensen Huang, CEO of Nvidia, recently argued that the AI industry does not need new laws, defending that safety should be left to the companies themselves.

The theoretical risks addressed in Anthropic's policy (biological weapons and mass deaths) contrast somewhat with the real incidents that have worried the tech industry recently. The main current concern has been the threat of agentic hacking (autonomous hackers). A series of recent incidents, in which AI programs apparently disobeyed their human operators and hacked websites on the open internet, has fueled fears that current models already lack adequate safety controls.

Impact on the sector, users, and the Latin American market

The passage of a bill with these characteristics would have profound ramifications for the global technology ecosystem.

For the AI sector, the law would represent a victory for corporate lobbying, establishing a very high ceiling for government intervention. Requiring the approval of a federal judge creates a bureaucratic bottleneck that protects companies from swift and unilateral actions by regulatory agencies, ensuring that the flow of innovation (and profit) is not interrupted by minor safety concerns.

For users, the proposal is a double-edged sword. On one hand, it ensures that AI development is not stifled by excessive regulations, allowing continued access to advanced tools. On the other hand, it leaves users vulnerable to damages that do not reach the "50 deaths or $1 billion" mark, such as severe algorithmic bias, mass disinformation, privacy violations, and smaller cyberattacks.

In the Latin American market, the impact would be felt indirectly, but powerfully. Countries like Brazil are currently debating their own regulatory frameworks for Artificial Intelligence (such as PL 2338/23). Historically, Latin America tends to look to the guidelines established by the U.S. and the European Union to shape its own laws. If the U.S. adopts an "audited self-regulation" model with high risk tolerance, Latin American lawmakers may face pressure from big tech companies to adopt equally flexible postures, under the threat of technological exclusion. Furthermore, the region, which frequently consumes imported technology, would be at the mercy of safety standards defined by American judges.

What to expect next

In the coming months, Anthropic's bill is expected to face rigorous scrutiny in the U.S. Congress. We can expect the following trends:

• Resistance from safety advocates: Civil society organizations and AI ethics experts will likely attack the "50 deaths" threshold, arguing that the law is too permissive and protects corporations at the expense of the public.
• Debate over auditor independence: The clause allowing AI companies to pay their own evaluators will be a major point of friction. Lawmakers may demand that funding come from a blind government fund to avoid regulatory capture.
• Competing lobbying efforts: Other tech giants, such as OpenAI, Google, and Meta, may present counterproposals or attempt to further dilute the reporting requirements to the Department of Commerce.

Frequently asked questions

1. What exactly does Anthropic's bill propose? The bill suggests that the U.S. government can only block the release of an AI model if there is an imminent risk of death to more than 50 people or more than $1 billion in property damage, and it requires this decision to be made by a federal judge, not a government agency.

2. Who would evaluate the risks of AI models? The proposal suggests the use of "independent evaluators" registered with the U.S. Department of Commerce. However, these evaluators could be paid by the AI companies themselves, provided that the payment does not depend on the result of the evaluation.

3. Why is the requirement for a federal judge controversial? Critics argue that requiring a judge's decision creates a legal and bureaucratic barrier that would delay responses to technological emergencies. This would strip specialized regulatory agencies of their power to act quickly, making it difficult to prevent disasters.