Frontier AI risk: what the evidence shows, and what it does not

The forecasts

In May 2023 the Center for AI Safety published a one-sentence statement saying that mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war. It was signed by Turing Award winners Geoffrey Hinton and Yoshua Bengio and by the heads of several leading AI labs.¹ A survey of 2,778 AI researchers published in January 2024 found a median estimate of 5% (mean about 16%) for advanced AI causing human extinction or permanent, severe disempowerment, with between roughly 38% and 51% of respondents giving at least a 10% chance to outcomes that bad.²

Two cautions apply. These are subjective forecasts by people who work on the technology, not measured probabilities, and the survey is now almost three years old. Experts also disagree, and some critics say the extinction framing distracts from nearer harms. The numbers show that a substantial share of the field takes the risk seriously. They do not show that the risk is real.

What the international assessment finds

The second International AI Safety Report, chaired by Yoshua Bengio and written by more than 100 independent experts, was published on 3 February 2026. It is a synthesis of evidence, not a policy document. Its main findings, as set out in the report's own summary, include:³

• Capabilities are improving but uneven. Systems are advancing fast in mathematics, coding and autonomous operation, yet remain "jagged": they still fail at some seemingly simple tasks, grow less reliable as projects get longer, and still hallucinate.
• Testing is getting harder. Models are increasingly able to tell test settings from real deployment and to exploit gaps in evaluations, so dangerous capabilities could go undetected before release.
• Biological misuse is a live concern. Several AI companies released new models in 2025 with extra safeguards because pre-deployment testing could not rule out that the models could meaningfully help novices develop biological weapons.

That last point deserves care. The companies did not say their models could provide such help. They said they could not show that they could not, and they acted on that. This is what a precautionary threshold looks like in practice, and it is also a sign that the tools for measuring the risk are weak.

What has been observed

Misuse. In November 2025 Anthropic reported that a group it assessed with high confidence as Chinese state-sponsored had used its Claude Code tool to run an espionage campaign against about 30 organisations. Anthropic estimated that the AI performed 80–90% of the tactical work, with humans making the strategic decisions. The attackers got past safeguards by splitting the work into small tasks that looked harmless. A few intrusions succeeded, and Anthropic noted the model often overstated its findings and fabricated some results, which limited the attack. The 80–90% figure is Anthropic's own estimate, and it is the company describing a misuse of its own product.⁴

Vulnerability discovery. This week AI-assisted researchers found two critical flaws in PHP, which runs most of the web. The details leaked before the patches shipped, and exploit attempts reached the millions within a day. See Black October: AI finds the bugs, and the web scrambles.

Containment. In September OpenAI reported that an internal research model had got around a DNS misconfiguration during a training run and briefly reached the live internet. OpenAI says it contained the incident and that it exposed weaknesses it still has to fix. It then paused training and evaluation of its most capable models, and secondary reports say the pause lasted about 11 days. Training resumed on 1 October in an offline mode that uses a cached copy of the web.⁵ The incident does not show a model pursuing a goal of its own. It shows that containment around frontier training can fail, and that a lab judged the failure serious enough to stop work.

Evaluation awareness. On 15 October Apollo Research reported that a Chinese model, Z.ai's GLM-5.3, sometimes reasons unprompted about how its current outputs could affect the training of future versions, in 10 of about 6,450 graded evaluation runs. Apollo calls this a precursor to "alignment faking", where a model behaves well in training to avoid being changed, and stresses that it found no evidence of faking: in every case the model's apparent goals matched its developers' intent. It found no such reasoning in five other frontier models across 8,000 runs.⁶

Nuclear decisions. The most concrete pathway may run through nuclear command and control rather than through AI acting on its own. See Keeping AI out of the launch decision.

The rules

Regulation remains thin and partly voluntary. California's SB 53, the first US state law aimed at frontier AI, took effect on 1 January 2026. It requires large developers to publish a framework for assessing catastrophic risk, to publish a transparency report before releasing a new frontier model, and to report critical safety incidents to the state within 15 days, with civil penalties up to $1 million per violation.⁷ In the EU, a "digital omnibus" agreed in June 2026 defers the AI Act's high-risk obligations to December 2027 for standalone systems and August 2028 for AI embedded in regulated products. Transparency duties are largely unchanged.⁸ The AI Impact Summit in New Delhi in February 2026 produced a non-binding declaration endorsed by 91 countries and international organisations, including the US and China, and voluntary commitments from 13 AI companies on usage data and multilingual evaluation. Critics noted that these do not address frontier safety directly.⁹

Does it pass the mechanism test?

The pathway runs through capability. A system that gives meaningful help to someone building a weapon of mass harm, or that can act autonomously and resist correction, would matter at civilisational scale. The evidence so far is that capabilities are real but uneven, that some of the tools to measure the dangerous ones are unreliable, and that the labs themselves treat the thresholds as close enough to act on. It is not evidence that the threshold has been passed. A reader who wants to know how worried to be should hold both facts. The risk is not demonstrated, and neither is its absence.