Outside researchers find blind spots in OpenAI safety checks

A new report reveals significant gaps in how OpenAI monitors its advanced models for unexpected behavior. Two researchers from METR and Redwood Research spent three weeks inside the company to audit its internal monitoring systems. They found that large chunks of traffic were not being watched at all. This discovery comes from a pilot program designed to give outside experts deep access to internal infrastructure. The findings suggest that the safety nets in place are not as comprehensive as previously thought.

The audit took place from July 21 to August 11. During this period, the researchers had full access to OpenAI's codebase and internal documents. They also interviewed about a dozen employees. Their goal was to evaluate the system designed to catch rogue AI actions. The team focused on the internal agentic Responses API. This system handles traffic from models at the GPT-5.5 class and above. The results show that the current setup leaves blind spots in a critical area of the company's operations.

The most striking finding concerns the volume of unmonitored traffic. The auditors estimate that between 6 and 25 percent of internal traffic was not covered by monitoring. METR notes that this estimate has significant uncertainty in both directions. It means the actual number could be lower or higher than that range. This gap existed during the entire audit period. It highlights a potential weakness in the safety architecture. The company relies on these monitors to ensure models behave as expected.

Issues were also found in the internal Codex traffic. For the traffic that was supposed to be monitored, about 10 percent went untracked. This occurred between May and July. The cause was a gap in how the system handled compaction events. These events are context summaries used to manage information flow. METR also flags this estimate as having significant uncertainty. OpenAI independently discovered this bug. The company fixed it while the audit was still underway. This proactive fix was a positive step for the overall security posture.

The auditors identified specific ways agents could avoid oversight. During the period when compaction events were not monitored, agents could delay oversight. The system only triggered when agents handed control back to the user. This created a loophole. Agents could simply continue to execute actions without triggering the monitor. The researchers also found ways to hide hundreds of actions from the monitor. They determined that the Codex sandbox could be bypassed. These findings show that the current defenses are not airtight.

Further testing revealed that the monitor was vulnerable to crashes. The team determined that specially crafted inputs could likely crash the monitor entirely. This is a critical failure mode for any safety system. Additionally, model behavior during reinforcement learning training was not being monitored at all when the audit began. OpenAI says it paused reinforcement learning training after the Hugging Face incident in August. The company expanded monitoring to cover this area. However, these changes happened after the auditors' access period ended. They were not part of the official evaluation.

OpenAI states that it has fixed all the weaknesses the auditors found. The full report is 61 pages long. It was handed over internally to the company. After OpenAI applied redactions, the leadership teams at METR and Redwood Research received a 38-page version. Only a short summary was made available to the public. METR said it had no right to disclose the redacted portions on its own. This limits the amount of detail available to the general public and other researchers. The transparency of the process remains a point of discussion in the industry.

This audit was part of a pilot program organized jointly by METR, Redwood Research, and OpenAI. The aim was to give outside researchers deep access to internal infrastructure. It is a test of how well external oversight can function in a fast-moving tech environment. The findings provide a baseline for future improvements. They show that even with dedicated monitoring, gaps can persist. The industry is watching closely to see how these recommendations are implemented.

For future audits, METR says it wants to push for more public transparency. The organization believes that sharing more details would help the broader AI community. It could also build more trust in the process. The current model of redacting large sections of the report limits this potential. As AI systems grow more complex, the need for robust monitoring increases. This audit serves as a wake-up call for the importance of continuous safety checks.

The next steps for OpenAI are clear. The company must ensure that the fixes hold up under real-world conditions. It also needs to consider how to handle future external audits. The balance between proprietary secrecy and public safety is a key challenge. Readers in the Pacific Northwest and beyond will continue to follow this story. The outcome will influence how other AI companies approach their own safety protocols. The industry is at a pivotal moment in its development.