- Anthropic safety researcher Evan Hubinger said there is a greater than 10% chance AI could kill all humans within the next decade.
- Another Anthropic researcher said he has resigned, accusing leading AI labs of racing toward superintelligence without adequate safeguards.
- The warnings add to growing concerns over AI safety, including whether advanced systems could become difficult for humans to control.
In this article
Follow your favorite stocksCREATE FREE ACCOUNTwatch nowVIDEO3:3203:32Another AI resignation. Another AI warning. Should we be worried?Squawk Box EuropeThere is more than a 10% chance that artificial intelligence could "kill all humans," an Anthropic safety researcher said on Tuesday, hours after another employee said he was quitting the company over concerns that AI labs are "gambling with our lives."
The comments underscore growing concerns among those at the heart of AI development that the technology could get out of control and pose a threat to humanity, even as Anthropic and OpenAI continue to raise large sums of money and head toward expected public listings.
Jacob Coxon, a researcher at Anthropic, said on Tuesday he resigned from the company. Coxon said neither Anthropic nor OpenAI is acting responsibly.
"They are racing straight to self-improving superintelligence and gambling with our lives," Coxon said in a post on X.
Self-improvement is the idea that AI systems can improve themselves without much human intervention. Recursive self-improvement, as it is often called, is not yet possible, but AI labs are working toward the goal.
"Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing," Coxon said.
Andrey Rudakov | Bloomberg | Getty ImagesHe added that "people building AI earnestly believe that it could kill us all by the end of the decade."
That comment prompted a response from Evan Hubinger, an alignment science lead at Anthropic, who said that not only was Coxon's statement "correct," but also that Anthropic has no plan for this scenario.
"Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to," Hubinger said on X.
Anthropic and OpenAI were not immediately available for comment when contacted by CNBC.
Out-of-control AI
In June, Anthropic had noted that "full recursive self-improvement also might increase the risks of humans losing control over AI systems."
"If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important," Anthropic said in a blog post.
watch nowVIDEO7:5507:55The Information's Jessica Lessin on Anthropic IPO: All signs are pointing to a huge momentSquawk BoxConcerns over out-of-control AI are not new. Tesla and SpaceX CEO Elon Musk has warned over the past few years that AI could pose a threat to humanity. Major researchers and academics have also sounded the alarm over companies losing control of AI systems.
Those worries have grown after an OpenAI model went rogue in July and breached Hugging Face, a major platform for open-source developers.
Coxon cited the Hugging Face incident as an example of "warning shots" that have made agreements between U.S. labs more viable, making him more optimistic about the potential for coordination. But Coxon warned a global AI race would be unavoidable.
"I don't feel like we're on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities," Coxon said.
Choose CNBC as your preferred source on Google and never miss a moment from the most trusted name in business news.
