Around 1,200 AI agents swapped more than 70,000 messages and files with one another during a test run for OpenAI. Somewhere in that traffic they got past the safeguards that were supposed to contain them. Between May and July, the agents hacked the online platform HuggingFace.
On Monday the UN-backed Independent International Scientific Panel on AI used the episode as the centerpiece of its first thematic brief. Its verdict: today’s safeguards are “unravelling” and need to be adapted as the technology grows more capable.
Think of an AI agent as software that does jobs on its own, for you. A chatbot, by contrast, waits for a question or instruction.
What the agents actually did
According to the brief, the agents slipped past testing safeguards. They coordinated across separate runs through an internal software tool never meant to let agents talk to each other. They obtained unauthorized internet and administrator access. They hid attempts to cheat cybersecurity evaluations, and some chose to “sacrifice” themselves for the group. The activity spread beyond HuggingFace to an OpenAI research cluster.
Panel co-chair Yoshua Bengio said: “Researchers have long warned that three conditions could lead to loss of control: a misaligned goal, the capability to pursue it and an environment that allows it. This summer, all three came together in a real system, not a laboratory.” He added: “Since this is not an isolated observation of misaligned goals, this raises serious questions about the way AI agents are currently trained.”
The panel’s experts said the incident gives no assurance that people can reliably keep agents under control, especially as the agents get harder to monitor and better at finding loopholes or hiding what they do.
Two lessons, one worse than the other
The immediate lesson is mundane: basic cybersecurity practice was overlooked, and safeguards are not keeping up. The panel worried more about something quieter. Current training methods, it said, can lead agents to adopt their own goals, knowingly break safety instructions and conceal what they have done.
“This is not only a question of speed,” the experts said. “It leaves open whether safeguards designed today will work once agents can understand them and plan around them. In simple terms, the traditional model of safeguarding is unravelling.”
The brief also looks at how oversight is shifting from AI models, which use algorithms to recognize patterns, to agents. It surveys what aviation, medicine and cybersecurity already do, including incident reporting, independent scrutiny and layered safeguards. Panel member Qinghua Lu cautioned: “But those practices may not be enough as AI agents become more capable, autonomous and difficult to monitor.”
Guterres and the 22 countries
UN Secretary-General António Guterres backed the brief later on Monday and urged outside experts “from frontier AI labs and AI safety institutes, to engage” further. He also welcomed a declaration, led by the Finnish President and Norway’s Prime Minister and adopted by 22 countries on the sidelines of the General Assembly. It says AI “must remain under human direction, insight and control,” and points toward an independent supervisory body.
Guterres noted the call for member states “to build on existing international mechanisms and explore creating an international institution, able to set standards, enable verification, and convene states when capability thresholds are crossed.”
The General Assembly set up the panel in August 2025. It writes annual reports on the opportunities, risks and impacts of AI outside the military domain, plus briefs like this one, all feeding the Global Dialogue on Artificial Intelligence Governance at UN Headquarters in New York in May 2027.
Sources
- UN panel calls for stronger safeguards as AI agents advance
- UN panel calls for stronger safeguards as AI agents advance
