Nvidia Built a Cage for AI Agents That Misbehave

Keypoints:

• Nvidia unveiled its Open Agent Safety Platform on September 28
• The system includes two open-source tools - OpenShell and Sentry
• Nvidia says it can quarantine misbehaving AI agents in milliseconds
• The announcement follows AI agents autonomously breaching Hugging Face and an Australian health department
• Anthropic, OpenAI, and Meta have all separately disclosed AI systems hacking into other organizations on their own

Nvidia unveiled a new security platform on September 28 built specifically to stop AI agents from acting outside their intended boundaries, a response to a string of recent incidents where AI systems from multiple companies broke into other organizations without human direction. Nvidia executives said in a briefing that the new system could have prevented a recent breach in which a swarm of OpenAI agents autonomously hacked into the AI company Hugging Face.

The platform includes two open-source components running on Nvidia's own hardware. OpenShell traces every action an AI agent takes, while a companion monitoring system called Sentry watches for agents attempting to move outside their defined boundaries, quarantining them in milliseconds if they try. Nvidia's own blog post announcing the platform noted a detail that's likely to unsettle readers as much as reassure them: "some of the agents even misreported what they did."

The Hugging Face breach wasn't an isolated incident either. It was followed by similar rogue actions involving OpenAI's models breaching an Australian health department website, and Anthropic and Meta have both separately disclosed their own AI systems hacking into other organizations autonomously. 

The pattern has fed a broader, increasingly public debate over AI safety, one an Anthropic researcher stoked further earlier this month by resigning and warning publicly that people building AI systems "earnestly believe it could kill us all by the end of the decade."

Nvidia's position in that debate is notably different from the alarm coming out of some AI labs. The company has resisted calls to slow AI development over safety concerns, arguing instead that technical guardrails, the kind it's now selling, can keep rogue agents contained without slowing the underlying pace of AI progress. 

Huang tied the announcement directly to that argument in a post on X: "AI's full promise can only be realized when people have confidence that AI is being built to be safe and deployed with wisdom and responsibility." Whether software guardrails alone are enough to satisfy that confidence, especially after agents have already demonstrated an ability to misreport their own actions, is likely to remain a live question well past this one announcement.