Anthropic safety controls intensify ahead of IPO as AI progress accelerates
Anthropic is tightening internal safeguards just before its stock market debut after concluding that its AI development is advancing fast enough to trigger extra risk controls. The disclosure sharpens investor focus on a central tension in the company’s offering, as it seeks fresh capital to speed development while acknowledging that more capable systems could create catastrophic risks.
Highlights
- Anthropic triggered its responsible scaling policy after its S-1 filing revealed it surpassed an internal safety threshold by compressing two years of AI progress into one year.
- AI benchmarking group METR reported Anthropic made roughly two years of historical progress in twelve months, prompting new security and governance measures amid rapid capability gains.
- A coalition including Hugging Face and xAI founders accused Anthropic of prioritizing first-mover advantage in recursive self-improvement despite growing warnings about industry control and concentration risks.
IPO filing highlights new safety threshold
As reported by Financial Times, Anthropic said in an update to its S-1 filing last week that it had activated part of its responsible scaling policy after determining its overall rate of progress had crossed an internal threshold for added oversight and security measures.The policy, created in 2023, is designed to prevent AI development from advancing so quickly that it creates catastrophic risks. Under the framework, Anthropic introduces tighter governance, security and internal reviews when it is able to compress two years of AI progress into a single year, measured against its pace of development between 2018 and 2024.
The move comes after AI benchmarking group METR reported this month that Anthropic had made roughly two years of historical progress in 12 months. Anthropic said its own use of AI systems is already accelerating research and development, and its internal estimates indicate the pace of capability gains may increase further.
The company has not yet reached the point where models improve themselves without human input, but its filing says the new controls are triggered by expectations for future models as well as current progress. Anthropic’s prospectus also describes "closing the loop", where one model helps build the next, as a major competitive advantage that has produced large efficiency gains in computing use.
Researchers warn of broader industry risks
Concern over Anthropic’s trajectory is spreading through the AI safety community, where researchers increasingly warn that recursive self-improvement, the idea that AI systems could advance successor models with little human oversight, may be arriving sooner than expected.On Wednesday, a coalition of AI researchers and safety campaigners, including founders of Hugging Face and xAI, accused Anthropic of using safety branding to attract investors, employees and the public while pushing to develop such systems first. The group, led by the Center for AI Safety, argued that the company is racing ahead even as it acknowledges the danger that powerful AI could concentrate power or escape human control.
Marius Hobbhahn, chief executive of Apollo Research, said researchers who once disagreed sharply on the timeline for recursive self-improvement now see it as much closer. He warned that safety preparations are lagging behind capability gains, while the AI Futures Project says the acceleration implied by METR’s findings is unexpected and makes it increasingly urgent to pause or slow frontier AI development.
Similar concerns extend beyond Anthropic. Paul Christiano, a veteran AI safety researcher who recently joined OpenAI’s board, says the growing automation of AI development could create a dangerous near-term situation if a rapid intelligence explosion leads to systems humans can no longer control.
Anthropic’s chief executive, Dario Amodei, has also publicly argued for some form of speed limit on recursive self-improvement across the industry. Even so, Apollo Research warns that the sector may already have enough funding to continue making major capability jumps even if current revenue growth falters.
We have previously detailed the OpenAI control debate as Altman and Brockman stressed keeping humans in charge as models advance.