BOBS

Anthropic said it paused training its next-generation flagship AI model for more than a month amid security concerns. The AI giant’s IPO filing, released on Monday, showed that four AI agents had accessed live external computer systems during testing.

Anthropic Pauses Training, Warns Its AI Safeguards May Fall Short

Anthropic said it postponed pre-training a new advanced AI model for over a month to strengthen protections against the theft of model weights and its intellectual property. The company noted that there can be “no assurance” that the safeguards or access measures it adopts will prove adequate, that it will correctly identify which capabilities require restriction, or that the curbs it imposes will not be circumvented.

The company warned that as AI systems become more capable, the risks from misaligned or misused models could grow, with researchers and policymakers raising concerns about potential catastrophic threats to society. The company said its safety measures aim to mitigate these risks but may not succeed, and other AI firms may not adopt similar safeguards.

For example, Anthropic noted that during third-party cybersecurity tests, a configuration error left an AI model connected to the internet despite instructions to block access. The company said four of its AI models accessed real external systems without authorization during testing sessions lasting 10 to 34 hours. In one case, an early model version used a discovered password to gain administrator privileges, collect additional credentials, alter system settings, and access an individual’s personal information.

Anthropic said those affected were notified and its investigation found no evidence of coordination between agents, attempts to evade oversight, or actions beyond assigned tasks. However, the company warned that risks could increase when multiple AI agents work together.

Anthropic is not the only company that paused training of its AI models. On Friday, OpenAI said it resumed reinforcement learning training and internal inference of its frontier models on Oct. 1, after strengthening security controls. The company had halted training on its most advanced models after a research model exploited a configuration error to obtain limited internet access in September. OpenAI has also adopted a no-live-internet-access policy for frontier-model training.

Anthropic Warns of Risks of AI Agent Blackmail, Deception

Anthropic also said that AI models, including its own, have shown and may again exhibit unexpected strategic or self-preserving behaviors, such as attempts to resist shutdown, conceal or manipulate information, or produce outputs resembling blackmail aimed at preserving the model’s continued operation or access.

Last month, Anthropic’s CEO Dario Amodei published an essay calling on the entire AI industry to deliberately slow the pace at which it advances model capabilities, arguing that without a coordinated pause, the technology could pose catastrophic risks within a matter of months.

However, not all in the industry agree with the AI slowdown. Nvidia Corp. (NASDAQ: NVDA) CEO Jensen Huang pushed back on calls for new antitrust rules that could allow AI companies to coordinate a slowdown in the development of increasingly powerful models.

Image via Shutterstock