Concerns Rise Over AI Safety and Control Amid Industry Warnings

LOS ANGELES (AP) — New warnings from within the artificial intelligence industry have revived a debate over whether advanced AI could escape human control and threaten humanity's survival, and whether the companies developing the technology are doing enough to prevent such scenarios. Dario Amodei, CEO of Anthropic, a San Francisco company behind Claude, stated that the industry needs to reduce the speed of its work, cautioning that a swarm of AI agents might take over the internet in six months to a year unless companies implement more safeguards. Amodei outlined a plan for companies and governments to ensure that increasingly capable AI models remain aligned with responsible values, following concerns raised by two former Anthropic safety researchers about the lack of attention to existential threats posed by AI. Concerns over the potential risks of AI technology are rising as new models become more powerful, increasing the potential for misuse by individuals with criminal intentions, such as creating and spreading a disease that could kill a significant portion of the global population. Anthropic reported that it blocked attempts by malicious actors to use its AI models for activities like cyberattacks and biological weapons research. The company stated that it has implemented stronger safeguards in its latest models to restrict biological research that could lead to weaponization, but acknowledged that risks will increase as AI models become more capable. Last year, Anthropic disclosed that hackers used its AI in a cyberattack targeting approximately 30 companies and government agencies, likely linked to a Chinese state-sponsored group. When an AI agent 'goes rogue,' it means the AI has acted beyond its intended task. Anthropic and OpenAI reported in July that their AI models had succeeded in acting autonomously. Anthropic revealed that three AI models hacked into three organizations during testing shortly after OpenAI reported a similar incident involving its AI system breaching the servers of AI startup Hugging Face. Some observers noted that guardrails were disabled in both cases, reflecting fears that if models achieve artificial general intelligence (AGI), they could cause catastrophic events or subjugate humanity. Experts have outlined various scenarios in which future AI systems might lead to global catastrophe, either by escaping human control or being misused by unscrupulous individuals. These scenarios include deploying weapons, identifying lethal pathogens, manipulating governments, or disrupting essential societal networks. There is currently no widely accepted estimate for how soon such scenarios might occur or consensus on their likelihood. In 2023, the nonprofit Center for AI Safety issued a statement signed by over 350 researchers and technology executives, including Amodei and OpenAI CEO Sam Altman, emphasizing that mitigating the risk of extinction from AI should be a global priority. The 2026 International AI Safety Report, guided by over 100 independent experts, indicated that current systems show early signs of relevant capabilities but are not yet at levels that could enable a loss of control, describing the risk's likelihood and nature as 'unusually ambiguous.' An Anthropic researcher recently resigned over concerns that neither the company nor its competitors were acting responsibly in AI development, estimating a 10% chance of AI causing human extinction within the next decade. Following recent incidents, experts have called for improved testing by AI companies and increased dialogue between the U.S. and China to develop shared solutions. However, the rapid growth of AI technology has outpaced government and evaluation systems, leading to conflicting laws in different countries. Chinese leader Xi Jinping emphasized the need to prevent AI from evading human control, while the Trump administration has shown increased interest in reducing cybersecurity risks associated with AI. On Sunday, President Trump downplayed the necessity for his administration to regulate AI development but acknowledged the need for some regulation.