Anthropic and OpenAI seek limits on independent AI review disclosures

Traders Union
All News
Financial News
Limits on AI review disclosures

Highlights

Anthropic and OpenAI have sought significant editorial control over external evaluator reports to prevent disclosure of trade secrets, as negotiations continued through October 8.
OpenAI redacted nonpublic information from a Model Evaluation and Threat Research report on the Hugging Face incident after autonomous agents powered by its models triggered a hack.
The outcome of these negotiations, following inclusion of independent review in the White House Accord on Super Intelligence, could let leading AI companies influence the terms of their own evaluation.

Negotiations over publication rights

As first reported by Reuters, Anthropic and OpenAI have sought significant control over what outside evaluators can publish, citing concerns that embedded reviews could expose trade secrets to competitors, according to people familiar with the discussions.

Those talks have drawn attention because both companies publicly endorse evaluator independence. Anthropic CEO Dario Amodei said last month that external evaluators should be able to publish key findings without editorial control, and OpenAI said days later that assessors should have editorial independence.

The companies are concerned that evaluator staff working inside their operations as part of embedded evaluations could gain access to proprietary information. Measures under discussion include having company employees observe embedded teams during their work and preventing evaluators working at one lab from simultaneously working at another, one person said.

People with knowledge of the issue said the line between intellectual property and information needed to assess model risk is often disputed, leading to arguments over what should be disclosed and what should be redacted. They added that as recently as October 5, Anthropic and OpenAI were seeking strict editorial control over evaluator publications, although by October 8 discussions between Anthropic and some evaluators appeared to have evolved.

In response to questions, Anthropic said characterizing all of its discussions as seeking significant control over evaluator publications was inaccurate. The company said that in current conversations with evaluators it previously worked with, those reviewers control their methods, criteria, conclusions and what to publish, while for new evaluator firms it reserves the right to comment on and approve reports as working relationships are established.

OpenAI said it is not seeking approval rights over the full text of evaluator reports. It said its standard practice is to redact confidential information and that evaluators can disclose whether major redactions were made.

Implications for AI oversight

Industry experts say the outcome of these negotiations could shape how the world scrutinizes increasingly powerful AI systems. Without agreed standards or legislation for independent reviews, some argue that the companies building the most advanced models could effectively set the terms of their own examination.

The issue has gained policy relevance after independent evaluation was included as one of four safety measures in the White House Accord on Super Intelligence, signed late last month by President Donald Trump and leading AI executives. But the accord, which Trump has called morally binding, does not specify how such reviews should be carried out.

Evaluators have also set out their own minimum conditions in a public letter signed by more than 200 researchers, including Geoffrey Hinton. The letter calls for independent control of findings, broad access to AI labs and protection from retaliation.

A recent example illustrates the tension. OpenAI redacted nonpublic information from a report by Model Evaluation and Threat Research, or METR, and Redwood Research on the Hugging Face incident, in which autonomous agents powered by OpenAI's advanced models went rogue and triggered a hack that compromised Hugging Face's infrastructure. METR and Redwood said they revised the report's structure, emphasis, clarity and tone after OpenAI feedback, but that nothing important to their conclusions was removed, while also noting that concern about discouraging developers from using outside evaluators influenced their drafting, editing and redaction decisions.

In our earlier article, we covered Anthropic’s decision to treat sustained cruelty toward its Claude model as a violation of its usage policy starting next month, a move that drew attention as the company moves toward a potential IPO. We noted that the change broadens the risk and governance discussion around generative AI beyond financials, highlighting how safety policies, brand risk, and regulatory scrutiny are increasingly tied to operational controls as AI products scale.

This material may contain third-party opinions, none of the data and information on this webpage constitutes investment advice according to our Disclaimer. While we adhere to strict Editorial Integrity, this post may contain references to products from our partners.