Critics Slam AI Audit Agreement from Nvidia, OpenAI, and Google as Empty Promises

Six of the biggest names in artificial intelligence — Nvidia, OpenAI, Anthropic, Google, Meta, and xAI — have agreed to open their models to external audits. President Donald Trump called the agreement “morally binding.” But morally binding isn’t legally binding, and right now there’s basically nothing stopping any of these companies from doing whatever they want.

The deal is thin. Nobody’s specified who picks the auditors. Nobody’s spelled out what those auditors can actually look at. And there are no penalties — zero — if a company decides to ignore the whole thing. Board members are supposed to review audit findings, which sounds serious until you realize that’s about as enforceable as a pinky promise.

Enron’s Ghost Is Haunting This

The setup looks uncomfortably familiar to anyone who remembers Enron. Back in 2000, Enron paid Arthur Andersen $25 million for auditing services and another $27 million for other work. The relationship was cozy. Too cozy. Both companies collapsed within two years, and the fallout was bad enough that Congress passed the Sarbanes-Oxley Act, which yanked the power to choose auditors away from company management entirely. The lesson was blunt: you can’t let the people being audited pick who audits them.

The AI industry seems to be relearning that lesson the hard way. A 2023 review looked at 16 AI companies and found an average adherence score of just 53% on their voluntary commitments. On the specific promise to let independent researchers report problems, eight out of 16 companies scored zero. Not low. Zero.

That’s not a minor compliance gap. That’s most of the class failing the test.

What Actually Works — And What Doesn’t

There’s been some genuine progress on the testing side, to be fair. Google DeepMind ran a double-blind evaluation with Singapore’s AI Safety Institute and a few other partners. The design was clever: testers couldn’t see the model directly, and the company couldn’t see the test questions. It’s a real attempt to build in independence without handing over the keys.

But it’s not clean. The whole thing depends on secure computer hardware to keep both sides walled off from each other. And last year, researchers showed that kind of hardware can be breached with minimal resources. So the method that’s supposed to fix the security problem introduces a new one. Not exactly reassuring.

METR, a nonprofit, did limited testing on OpenAI’s o3 model. Limited is the operative word. The timeframe was short. Access to information about how the model actually works was restricted. It’s hard to do a thorough job when you’re working with one hand tied behind your back and a clock running down.

A study with researchers from Stanford, Harvard, and Brown found that AI companies’ adherence to their own commitments was all over the place. Transparency scores were inconsistent. Problem reporting was weak. The gap between what companies promised and what they delivered was, in many cases, pretty wide.

What a Real Fix Would Look Like

Kyle Bligen co-authored a report called “Alternatives to Concentrated AI” that makes a straightforward case: external scrutiny doesn’t require direct access to the models. Auditors can do meaningful work without getting a full copy of the system. That matters because one of the industry’s main excuses for limiting audits is intellectual property protection. If you can evaluate the model without touching the model, that excuse gets weaker.

For audits to mean something, a few things need to change. Auditors should set their own evaluation criteria — not take direction from the companies paying for the process. Companies should at minimum disclose who their auditors are and what the scope of the audit covers. Detailed findings can stay confidential if necessary, but the basic structure should be visible, the way it is for public companies filing with regulators.

And someone needs to oversee the auditors themselves. Right now, there’s no equivalent of the Public Company Accounting Oversight Board, which Sarbanes-Oxley created to watch over financial auditors. Nobody’s checking whether the AI audits are rigorous, unbiased, or even completed. The current agreement doesn’t touch any of that.

The voluntary commitments these companies made to the White House back in 2023 already showed the limits of this approach. Outside experts were supposed to get access to test systems before release. That sounded meaningful. Then the 53% average adherence score came out, and the eight-out-of-16 zeroes on independent problem reporting, and it became clear that “supposed to” was doing a lot of heavy lifting.

It’s probably going to take legislation. The financial sector didn’t clean up auditing through voluntary pledges — it took a massive scandal and an act of Congress. The AI industry is running a similar experiment right now, with similar structural problems and similar incentives to look the other way. The double-blind method is genuinely promising. The METR work on o3 is a start. But a start isn’t a system, and a morally binding agreement with no enforcement mechanism isn’t really binding at all.

The average adherence score across those 16 companies was 53%.