AI Risks Could Turn Catastrophic by 2030, Experts Warn

There is a very real risk the AI revolution could cause an international catastrophe within five years, a panel of 272 experts has found. The experts defined catastrophe as more than one million deaths, more than $ US100 billion in losses, or civilisation-scale damage to democracy or privacy.

MIT FutureTech and the MIT AI Risk Initiative ran the study, published in June 2026. The panel drew AI researchers, policy advisers, technologists and governance specialists from 37 countries. They came from academia, industry, government and civil society.

The experts were presented with 24 risk categories, from discrimination and privacy loss to weapons, cyberattacks and power centralisation. They judged 18 of these more than 10 per cent likely to cause catastrophic harm by 2030.

The panel mixed global heavyweights with Australian institutions. Oxford, MIT, Stanford, Harvard and Princeton were all represented. Three of the study's core authors are based at the University of Queensland. The University of Sydney and the University of the Sunshine Coast also contributed experts. Two staff from Commonwealth Bank of Australia joined the panel.

The researchers used the Delphi method, a technique built for forecasting in fields with little hard data. Experts rate each risk anonymously across several rounds, then see how peers reasoned and revise their answers. The method surfaces genuine disagreement and limits pressure to conform. Here it ran three rounds in late 2025.

Under many risk-governance frameworks, a 10 per cent chance of catastrophe over five years is deemed “intolerable”. Such a level would normally trigger mandatory mitigation.

The https://forecastingresearch.org/xptfindings sit within a wider debate about AI's extreme tail risks. An earlier tournament by the Forecasting Research Institute put the chance of human extinction from AI this century near 3 per cent. 

Asked to name their top three concerns across all 24 risks, the experts scattered rather than agreed. Weapons and cyberattacks topped the list at 26.8 per cent, ahead of power centralisation at 23.5 per cent. Disinformation and influence reached 22.1 per cent. Loss of consensus reality and dangerous capabilities each drew 21.6 per cent.

One of the experts commented, “A technology-loving researcher develops AI with only technology in mind. He completely ignores the transparency and explainability of the model and pushes ahead with the development of multi-agent systems. 

“If multi-agent systems were distributed across the world, who would be able to audit them? Suppose agents were distributed across the United States, Europe, and Asia, and they were linked together. If 100 or 200 agents were linked together and operating, no one would know what was going on.”

Severity was scored from one, negligible, to five, catastrophic. Experts spread probability across both the catastrophic tail and the middle range. That pattern signalled doubt about the scale of harm, not whether it would occur. The highest-severity risks were dangerous capabilities, weapons and cyberattacks, power centralisation, competitive dynamics and false information.

Modelling cost-effective mitigations lowered every risk, but five stayed above 10 per cent. All 24 stayed above 5 per cent. The stubborn five were dangerous capabilities, weapons and cyberattacks, environmental harm, inequality and unemployment, and power centralisation.

Across 14 sectors, experts rated information and national security as the most exposed. Finance and insurance followed closely. The most exposed sectors were those where AI is embedded in critical decision-making, the report says. They named industries “where failures have immediate, large-scale consequences”.

Finance and insurance drew high marks for fraud, security flaws and system failures. A Delphi participant cited “both direct attack vectors (fraud, market manipulation) and regulatory exposure from AI system failures”.

The study also exposed a gap between exposure and accountability. Experts judged AI users and the public most vulnerable, yet placed responsibility elsewhere. Primary responsibility fell on general-purpose AI developers and on governance actors such as governments, regulators and standards bodies.

The report warns that spreading responsibility can create an “accountability sink”. That is where responsibility shared across many actors becomes responsibility held by none.

The authors argue technical fixes alone will not be enough. They call for governance tools such as regulation, liability, mandatory insurance and transparency to shift costs off the public. “The window for avoiding catastrophic outcomes remains open but is narrowing,” the report concluded.

The full report is available at https://airisk.mit.edu/priorities.

 

Business Solution

There is a very real risk the AI revolution could cause an international catastrophe within five years, a panel of 272 experts has found. The experts defined catastrophe as more than one million deaths, more than $ US100 billion in losses, or civilisation-scale damage to democracy or privacy.

MIT FutureTech and the MIT AI Risk Initiative ran the study, published in June 2026. The panel drew AI researchers, policy advisers, technologists and governance specialists from 37 countries. They came from academia, industry, government and civil society.

The experts were presented with 24 risk categories, from discrimination and privacy loss to weapons, cyberattacks and power centralisation. They judged 18 of these more than 10 per cent likely to cause catastrophic harm by 2030.

The panel mixed global heavyweights with Australian institutions. Oxford, MIT, Stanford, Harvard and Princeton were all represented. Three of the study's core authors are based at the University of Queensland. The University of Sydney and the University of the Sunshine Coast also contributed experts. Two staff from Commonwealth Bank of Australia joined the panel.

The researchers used the Delphi method, a technique built for forecasting in fields with little hard data. Experts rate each risk anonymously across several rounds, then see how peers reasoned and revise their answers. The method surfaces genuine disagreement and limits pressure to conform. Here it ran three rounds in late 2025.

Under many risk-governance frameworks, a 10 per cent chance of catastrophe over five years is deemed “intolerable”. Such a level would normally trigger mandatory mitigation.

The https://forecastingresearch.org/xptfindings sit within a wider debate about AI's extreme tail risks. An earlier tournament by the Forecasting Research Institute put the chance of human extinction from AI this century near 3 per cent. 

Asked to name their top three concerns across all 24 risks, the experts scattered rather than agreed. Weapons and cyberattacks topped the list at 26.8 per cent, ahead of power centralisation at 23.5 per cent. Disinformation and influence reached 22.1 per cent. Loss of consensus reality and dangerous capabilities each drew 21.6 per cent.

One of the experts commented, “A technology-loving researcher develops AI with only technology in mind. He completely ignores the transparency and explainability of the model and pushes ahead with the development of multi-agent systems. 

“If multi-agent systems were distributed across the world, who would be able to audit them? Suppose agents were distributed across the United States, Europe, and Asia, and they were linked together. If 100 or 200 agents were linked together and operating, no one would know what was going on.”

Severity was scored from one, negligible, to five, catastrophic. Experts spread probability across both the catastrophic tail and the middle range. That pattern signalled doubt about the scale of harm, not whether it would occur. The highest-severity risks were dangerous capabilities, weapons and cyberattacks, power centralisation, competitive dynamics and false information.

Modelling cost-effective mitigations lowered every risk, but five stayed above 10 per cent. All 24 stayed above 5 per cent. The stubborn five were dangerous capabilities, weapons and cyberattacks, environmental harm, inequality and unemployment, and power centralisation.

Across 14 sectors, experts rated information and national security as the most exposed. Finance and insurance followed closely. The most exposed sectors were those where AI is embedded in critical decision-making, the report says. They named industries “where failures have immediate, large-scale consequences”.

Finance and insurance drew high marks for fraud, security flaws and system failures. A Delphi participant cited “both direct attack vectors (fraud, market manipulation) and regulatory exposure from AI system failures”.

The study also exposed a gap between exposure and accountability. Experts judged AI users and the public most vulnerable, yet placed responsibility elsewhere. Primary responsibility fell on general-purpose AI developers and on governance actors such as governments, regulators and standards bodies.

The report warns that spreading responsibility can create an “accountability sink”. That is where responsibility shared across many actors becomes responsibility held by none.

The authors argue technical fixes alone will not be enough. They call for governance tools such as regulation, liability, mandatory insurance and transparency to shift costs off the public. “The window for avoiding catastrophic outcomes remains open but is narrowing,” the report concluded.

The full report is available at https://airisk.mit.edu/priorities.

 

Business Solution