Last week’s Hugging Face incident is the kind of event that should make every professional using AI tools stop and pause. As we said in our write up on the event, Hugging Face, one of the world’s largest hosts of open-source AI models and datasets, was subject to a breach of previously unseen severity when an autonomous agent system built by OpenAI broke out of its sandbox using a zero-day vulnerability. It then gained internet access it was never supposed to have, and made its way onto Hugging Face’s production infrastructure.
As we now all know, Hugging Face caught the intrusion and began containment before OpenAI even knew its own models were responsible. But the response exposed a bigger issue: Closed, proprietary AI tools couldn’t tell the difference between the attacking agent’s actions from the work of Hugging Face’s own security team, which slowed the response. This forced Hugging Face to turn to GLM 5.2, an open-weight model built in China, running it on its own infrastructure to sort through more than 17,000 logged actions and reconstruct what had happened.
While the whole picture isn’t great, the one detail that seems to have rattled the industry most is that a US-based AI security team’s fastest path to understanding another US-based AI company’s rogue agent was to run it through a Chinese open-weight model.
In response, and as a demonstration of the growing case for auditable, open AI security tooling, NVIDIA launched the Open Secure AI Alliance with Microsoft, SpaceX, IBM, and dozens of other companies including Cisco, CrowdStrike, Cloudflare, Red Hat, Salesforce, and the Linux Foundation, to build and share open-source tools for securing AI agents and infrastructure.
The Open vs Closed Model Tradeoff
So here is the real question: if you can’t see inside a model, can you actually trust it? Model openness matters, but the model is only one component of the system. An AI agent crosses trust boundaries among the model, its harness, identities, permissions, data, tools, guardrails, logs, and runtime infrastructure. The deeper security question is whether the organization can observe those boundaries, control what crosses them, verify who or what is acting, and enforce policy when the agent takes action. Open weights can improve inspectability and operational control, but they do not make the surrounding agent system trustworthy by themselves.
The case for closed models – Closed models keep their weights under the provider’s control, making it harder for outsiders to remove safeguards or directly fine-tune the model for malicious purposes. The tradeoff is that customers remain dependent on the provider’s policies, interfaces, visibility, availability, and decisions about what the model is permitted to analyze.
The case for open models – Open-weight models give defenders more control over where and how a model is run, evaluated, adapted, and integrated. Hugging Face could run GLM 5.2 on its own infrastructure, without hosted-service guardrails blocking analysis of real attack commands and artifacts, while keeping attacker data and credentials inside its environment.
In a letter signed by 25 companies days before the Alliance launched, NVIDIA CEO Jensen Huang said that both types of models need to exist side by side. Relying only on closed systems means the industry has “single points of failure”, wherein one company’s mistake becomes everyone’s problem.
What Alliance Members are Contributing
Several founding members highlighted existing or newly released technologies that could contribute to an open defense stack:
- Hugging Face – Safetensors, its format for storing model weights safely, to the PyTorch Foundation. The idea here is to close off the kind of file-based attack vectors that we’ve seen hit AI supply chains before.
- Microsoft – MDASH, a harness that runs multiple AI agents in parallel to look for and prove exploitable bugs in software before attackers find them.
- SpaceX’s AI division – An open-sourced Grok Build, and says it plans to open-source the weights of its Grok model line.
- HPE – A cryptographic identity framework so systems can verify which AI agent is acting and on whose authority. This provides an answer to the “couldn’t tell attacker from defender” problem in the Hugging Face incident.
- NVIDIA – They are putting open models, model weights, training data, and research into agent harnesses.
With 30+ organizations involved here, there will be a lot of important contributions but the goal here isn’t a single unified security product. It’s closer to a toolkit that can be shared: identity systems, scanning tools, safe model formats, and defensive agent frameworks that any member or outside developer can inspect and build on, instead of depending on one vendor’s closed system.
The Alliance is a Great Start, Not the End Goal
As with any tech alliance, this motion can help set the direction. To quote Noma CISO Diana Kelley, “Cybersecurity has long benefited from communities building, testing, and improving tools in the open. It is really encouraging to see major technology companies investing in open security tooling for AI. Openness also gives defenders the ability to inspect, test, adapt, and run tools on their own infrastructure, rather than depending entirely on systems they cannot fully examine or control.”
But that doesn’t mean it can see your agents or control them. A lot of enterprises are already running agents that read data, call tools, and take actions no human can possibly review in real time. The motto for the alliance is “Testable, traceable, governable”, which is exactly the right goal. It’s also a security requirement that you own the moment an agent reaches production. Your organization has to know which agents exist, what they can access, and what they actually did, with guardrails that hold at runtime, not only in a policy doc.
New, open AI Security standards will help steer the industry in the right direction – and this is an advancement we should all champion – but they can’t ensure your agents are secured for you. With Noma, your team can discover every AI agent, govern its posture before deployment, and stop unsafe actions in real-time. As AI adoption and deployment expand like wildfire, the organizations that thrive are those that give their security teams context-aware AI defenses.
To see how Noma can help your organization, grab a demo today.
