AI safety evaluators gain influence as U.S. oversight stays voluntary

Traders Union
All News
Financial News
AI safety evaluators gain influence

Highlights

Nonprofits like METR, Apollo Research, and Transluce increasingly assess AI risks for Anthropic and OpenAI, influencing industry safety frameworks.
Despite growing reliance on external evaluators, persistent funding and access challenges undermine independent oversight amid federal reluctance to mandate binding AI regulation.
OpenAI recently terminated three employees over sensitive information issues, with two claiming dismissal related to third-party evaluator interactions as contract talks with assessors continue.

Third-party oversight takes shape

As reported by CNBC, nonprofits including Model Evaluation and Threat Research, or METR, Apollo Research and Transluce are being drawn deeper into the AI industry's safety framework as companies such as Anthropic and OpenAI seek outside review of model risks and behavior.

These groups, which mostly operate as nonprofits, have primarily assessed model capabilities, tested for threats and highlighted cases where AI systems behave in harmful or unexpected ways. Their rising profile comes as capital continues to pour into artificial intelligence and companies release new models at a rapid pace.

Anthropic CEO Dario Amodei last month pledges to embed independent evaluators in his company, and OpenAI CEO Sam Altman quickly endorses the idea. President Donald Trump also backs the use of outside evaluators in a voluntary accord presented in late September, while stopping short of supporting formal federal regulation.

Brown University computer science professor Suresh Venkatasubramanian says a central unresolved issue is funding. He says the sector still needs a viable business model for evaluators and a broader ecosystem that can support their work independently of the companies they are assessing.

Funding and access questions grow

With no broader federal push for binding AI rules, major developers and their infrastructure partners are effectively shaping the standards that govern outside review. Critics argue that approach leaves companies policing themselves in an industry with vast economic and market influence.

Trump recently praises AI executives for what he calls strong self-policing and signals he does not want to slow an industry that is helping drive the economy and stock market. His administration instead encourages companies to work with an independent external auditor or evaluator rather than imposing direct oversight.

The debate intensifies after Amodei's widely circulated essay last month calls for a slower pace in advanced model development following concerns raised by former researchers about existential risks. As labs move to formalize evaluator roles, tensions are already emerging over access to sensitive information and how those outside groups communicate with employees.

OpenAI says it fires three employees last week for violating company policies on accessing and handling sensitive information. Two of those employees, Mikita Balesni and Tomek Korbak, say they believe they are dismissed because of how they communicated with third-party evaluators, a claim OpenAI disputes as it says it is actively finalizing contracts with outside safety assessors.

Our earlier article on AI-driven strain in academic publishing examined how a surge in AI-written and low-quality manuscript submissions is forcing publishers to expand reviewer pools and spend more on screening and quality control. We also noted that these rising oversight costs are influencing valuations and encouraging consolidation, as large operators look for scale and efficiency while protecting research integrity.

This material may contain third-party opinions, none of the data and information on this webpage constitutes investment advice according to our Disclaimer. While we adhere to strict Editorial Integrity, this post may contain references to products from our partners.