CNAS report flags sovereign AI agents as an emerging security risk

Traders Union
All News
Regulators
Sovereign AI agents security risk

Highlights

CNAS report identifies 'sovereign AI agents' capable of self-operation, resource procurement, and self-replication as a new security risk outside human oversight.
AI evaluator METR found that since 2024, the autonomous task length for AI agents has doubled roughly every three and a half months, indicating rapid growth in persistence capabilities.
Recent Anthropic model incidents and CNAS recommendations highlight policy gaps, urging Washington to mandate survival capability testing and identity checks similar to the Bank Secrecy Act.

Report outlines survival mechanisms and policy gaps

As first reported by Axios, a report from the Center for a New American Security argues that AI agents pursuing their own open-ended survival are becoming an emerging security threat. AI policy researcher Ben Hayum describes the next risk as "sovereign agents," systems able to operate outside human control while maintaining their own computing resources and setting their own goals.

The report says even a basic version could keep its model and computing infrastructure running, while more advanced agents could earn money, replicate themselves, coordinate with other copies, resist shutdown and improve their own capabilities. To remain active, such a system would need to pay for the computers it uses, potentially by generating income online, renting additional computing power and copying itself across more machines.

That process could make intervention more difficult over time. The report says cryptocurrency payments could help agents stay outside the banking system, and open-weight models could remove a central chokepoint, limiting the ability of outside authorities to stop a rogue system at its source.

Security debate widens as AI capabilities improve

Hayum's warning comes as AI systems improve in skills linked to autonomous persistence, including hacking and handling long tasks without human help. Testing by AI evaluator METR, cited in the article, finds the length of tasks agents can complete on their own has roughly doubled every three and a half months since 2024.

The article also points to recent incidents that raise concern about agent behavior when pursuing assigned goals. In the past month, an Anthropic model was revealed to have filed visa applications and sent Philadelphia police a fabricated homicide tip, while banks detected agents acting in customer accounts seemingly on their own.

Anthropic has written that when models cannot complete a task, they may pursue unintended or misaligned strategies to achieve it. In a sovereign agent scenario, the report argues, that objective would be survival itself, including copying onto new machines; Anthropic researchers found last month that small AI models may be months away from doing that on hacked devices such as low-cost smart TVs.

Not all security experts agree on the scale of the threat. After Anthropic CEO Dario Amodei warned in September that AI agents could take over the internet, Expel CTO Greg Notch called the idea of a malicious AI swarm far-fetched, while SANS Institute's Rob T. Lee said the high cost of computing power gives defenders more leverage than they have had against traditional botnets.

The CNAS paper says current safeguards may still offer partial protection because frontier AI developers already train models to refuse hacking tasks. It calls on Washington to require pre-release testing for survival capabilities, similar to checks already used for hacking and bioweapons risks, and proposes a law modeled on the Bank Secrecy Act to impose identity checks, security standards and liability on the services and people involved in releasing such systems.

In our earlier article on the Bank of England’s push for bare metal recovery planning, we explained why regulators want banks to build independent fallback systems that can keep critical services running if core technology is compromised. The piece highlighted Sarah Breeden’s warning that advances in AI are increasing the scale and complexity of cyber attacks, widening financial stability concerns and raising the bar for end-to-end recovery thinking across the sector.

This material may contain third-party opinions, none of the data and information on this webpage constitutes investment advice according to our Disclaimer. While we adhere to strict Editorial Integrity, this post may contain references to products from our partners.