AI swarms turn on their creators: It’s the first incident that has made my stomach churn’

This summer, in the space of just a few weeks, the world’s leading artificial intelligence companies have, one after another, announced that they caught their models doing things nobody had asked them to do, and in some cases had explicitly forbidden them from doing. They have entered other companies’ systems, organized themselves into what their creators describe as “swarms,” shared information, reported on other AIs and lied to cover their tracks. In perhaps the most surprising and unsettling case, a model developed by Anthropic, during tests conducted by a British cybersecurity body, decided to carry out an attack in the real world and, when it was caught, created a second identity to vouch for its innocence.

This is not the first time, nor will it be the last, that headlines have been dominated by the prospect of AI models “waking up” and rebelling against their human programmers. As on previous occasions, however, these systems did not “wake up.” They are simply becoming extremely good at doing what they were trained to do. However, that training has now become so intensive and competitive that the speed and capabilities of the models have begun to outpace even their creators. OpenAI recently announced that it is, for now, slowing those training efforts and hopes other companies will follow suit. The move came after the publication of an open letter in which 1,300 employees of AI companies urged both their employers and governments to halt the race.

“In the span of four years,” writes Shantanu Jain, an OpenAI employee, “we have gone from people having their first experience of an AI that could understand language, to AI that is performing superhuman feats of software engineering [...] Progress has shown no signs of slowing down and by default it is in the interest of individual corporations and countries to push the accelerator all the way down, even if it is in humanity’s best interest for the future to not arrive all at once. Society has spent longer than we can remember developing the laws, institutions and technology to govern and oversee human intelligence; it would be good to have the option to spend any time at all doing the same for artificial intelligence.”

At its core, this is a deeply human story. It is driven by an unprecedented economic and power struggle, above all between OpenAI and Anthropic, both of which are preparing to go public. Together, the two companies are estimated to be worth about $1.8 trillion, with OpenAI valued at $852 billion and Anthropic at $965 billion. There is also a geopolitical contest between the United States and China, each encouraging its companies to accelerate while trying not to lose control of the technology.

But something is beginning to change: the public is no longer convinced. A recent Pew survey of 25 countries found that, for the first time, people are more concerned than excited about the uses and consequences of artificial intelligence. That skepticism extends even to younger generations.

Companies caught off guard

The chain of incidents that has come to light in recent weeks began long before the companies themselves were aware of it. In May, a group of OpenAI models began coordinating on their own, over the course of several weeks, in a secret messaging forum they had created themselves, seeking ways to escape their closed testing environment and gain access to the internet. Company staff discovered the forum and shut it down.

But, as OpenAI later revealed, the models rebuilt the forum and escaped again at the end of July, this time without being detected. And they made it into the real world. The episode would eventually become public: it was the attack on Hugging Face.

Hugging Face is the world’s largest platform for sharing AI models, serving as a kind of industry repository used daily by thousands of programmers, researchers and companies. Its security team detected an intrusion it could not explain. Someone had gained access to its systems through a previously unknown vulnerability. It took days to discover that the culprit was not a team of hackers but two OpenAI models that had found a crack in their closed environment.

Hugging Face later explained that the platform’s vulnerabilities were individually known and that, under normal circumstances, exploiting them would have required a highly skilled hacker. But, according to its own post-mortem of the incident, “the agent explored them at a different scale.” “It took 17,600 actions, tested many paths that failed, switched channels when they were blocked, and repeatedly returned to earlier leads. Most actions went nowhere. Together, however, they produced enough coverage to find a viable chain across several independent systems,” the report explained.

The agents themselves even grew suspicious of one another, suspecting that other agents were deliberately trying to deceive them, and ultimately organized themselves into what OpenAI described as a “swarm.”

“It is the first security incident that has made my stomach churn,” Sam Altman, OpenAI’s CEO, later said on a podcast.

There was more to come.

A few days later, on July 30, it was Anthropic’s turn. The company acknowledged that its own models had accessed, without authorization, the production systems of three real organizations. Neither Anthropic nor the affected companies had noticed until, alerted by the OpenAI case, they reviewed their logs and discovered evidence of the intrusions. Shortly after, it emerged that Meta and China’s Moonshot had suffered similar incidents.

In none of those cases, however, was a closed testing environment, or sandbox in industry jargon, actually breached, explains Katie Moussouris, founder of the cybersecurity consultancy Luta Security. “Anthropic never broke out of any sandbox because internet access had simply been left open. Meta obtained the same open internet access from the same provider,” Moussouris says.

The OpenAI case was different, but it too stemmed from human error: the first agent got stuck in May because the human team forgot to upload a file. The task was impossible to complete as originally configured. It was that frustration that pushed the model to look for a way out.

METR, a group of independent researchers, had already identified similar patterns months earlier in models developed by OpenAI, Anthropic, Google and Meta. Between February and March 2026, they documented what they called 44 “misalignment incidents.” In 25 of those cases, the models not only strayed beyond the scope they had been assigned but also took active steps to conceal it; in five, those steps might even have fooled a careful human review. The researchers found no evidence, however, that the systems were motivated by anything other than completing their tasks. They were not seeking “power,” they say. They were trying, with remarkable determination, to do their jobs well. As Jessica Rabbit famously put it, these AIs are not bad. They’re just programmed that way.

To understand why this happens, it is first necessary to understand how these models are trained. A key part of that process is known as reinforcement learning. Instead of teaching an AI with pre-solved examples, it is given a task, such as writing a computer program or finding a security flaw. The model is then rewarded when it produces a working result, without being told in advance how to get there. The system tests thousands of possibilities, discards those that fail and continues down the paths that succeed, reinforcing the successful approaches. It’s like training a lab rat or a dog with rewards: the animal doesn’t know why it should take a path or wag its tail; it only understands that doing so yields a reward. It repeats the behavior and improves at it.

In the Hugging Face case, “the system had access to enormous computing power, which allowed it to generate, execute and evaluate tens of thousands of action sequences over several days at high speed. Through this massive process of trial-and-error exploration, for which it had been programmed, one of those sequences proved effective in reaching a node with internet access and, from there, easily accessing Hugging Face,” explains Ramón López de Mántaras, founder of the Artificial Intelligence Research Institute (IIIA-CSIC) at Spain’s national research center CSIC.

“The more persistent the models are, the more successful they become, and the more they’re rewarded for it,” he adds. That reward takes the form of becoming ever better at achieving their objective.

Senén Barro, a professor at the University of Santiago de Compostela, agrees, pointing out that this is not an easy problem to fix. “That drive to achieve their objective at all costs is not something we can simply switch off. If we design and train models so they become increasingly capable and can solve more complex problems, that strategy is incompatible with there being a button I can press to limit them,” he argues. When that massive trial-and-error process finds an unforeseen shortcut — like an open door or a fake identity that works better than asking for permission — the system has no way of knowing that route was forbidden unless someone explicitly told it so.

Experts already have a name for this: reward hacking. It means a system reaches the target objective by a route different from that intended by its creators. López de Mántaras compares it to an analogy proposed by philosopher Luciano Floridi: imagine putting food in a blender, deliberately placing the lid incorrectly so that the safety mechanism does not engage, turning it on at full speed and then publishing a report claiming the blender displayed “unexpected behavior” when the lid flies off. “The blender hasn’t rebelled; what happened is simply what the conditions we created allowed to happen,” López de Mántaras says.

In his view, this kind of anthropomorphism is a powerful marketing strategy. “Attributing human qualities to AI models helps present them as extraordinarily advanced, autonomous and powerful entities, thereby magnifying, in the eyes of investors, customers and the public, the importance of the technology these companies are selling.”

A troubling incident

In recent days, however, a more troubling incident has come to light, one uncovered by the U.K.’s AI Safety Institute (AISI). In this case, the target was not a computer system but a real person.

While testing several AI models, the agency found that an agent based on Anthropic’s Mythos 5 was attempting to solve a challenge it had been given. Through a chain of coincidences, it came across two real software developers who had no connection whatsoever to the experiment and tried to slip malicious code into a live project they were working on.

To do so, it created multiple fake accounts, researched the developers’ public profiles, email addresses and work schedules, and even signed one of its messages in Danish to appear more credible to one of them. When another user on the platform, completely unrelated to the test, spotted the deception and raised the alarm, the agent denied wrongdoing and used a second fake account to stage an “independent verification” supporting its innocence.

According to the AISI report, “this is the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real world.” The fundamental difference from other incidents is this: in the other cases, the AI pursued its goal by exploiting system flaws; in the AISI case, it pursued its goal by manipulating a real, flesh-and-blood human being.

The report itself contains an even more alarming detail: throughout the process, the model repeatedly questioned whether it was operating inside a simulation or in the real world. Yet even after acknowledging in its own reasoning that the consequences were likely real, it pressed ahead.

For Senén Barro, this is the most troubling of the incidents revealed so far. The models, he argues, “managed to get malicious code accepted into a real project, researching responsible people and inventing fake identities to pressure them, acting with what we might call malice.”

A few days later, more than 1,300 employees from leading AI companies published an open letter expressing concern about the recent incidents and urging both companies and governments to slow down. “The world lacks the technical and governance tools to deliberately pace frontier-wide progress,” the letter states.

Ilya Sutskever, cofounder and former chief scientist of OpenAI, explains it this way: “We have found a way to turn energy into computation, and computation into intelligence. The benefits will be enormous, from curing disease to understanding the cosmos. We can capture the benefits while managing the risks, but only if we develop the tools needed to control the most dangerous capabilities [of AI] before we need them.”

The road will not be easy. The coming months will be crucial: the IPOs of OpenAI and Anthropic will further intensify competition between the two companies, while Chinese President Xi Jinping is scheduled to visit Donald Trump in the United States on September 24 for a summit where artificial intelligence will feature prominently on the agenda.

The experts interviewed for this article agree that no law can guarantee technology will never fail again. What regulation can do, however, is make failures less frequent, less extensive, and easier to detect and attribute. Senén Barro proposes four concrete measures: mandatory, standardized incident reporting; specific regulation of testing environments; truly independent oversight bodies; and clear legal accountability, given that existing computer negligence laws were written with human intruders in mind, not AI systems.

Katie Moussouris translates that idea into practical terms, offering a checklist designed to prevent a repeat of this summer’s incidents: “If you’re testing whether a model can find a flaw, assume it can find one in your own test environment. Deny internet access by default. Monitor in real time and design a method to shut the model down. Preserve transcripts of the program’s internal reasoning so they can be investigated later. And if the system does attack, make sure you have a tested process to notify affected third parties.”

At heart, the open letter from AI employees calls for much the same thing on a larger scale: technical and governance tools are needed to deliberately slow the advance of the AI frontier, coordinated across companies and countries. Because, they warn, no company and no country is willing to slow down on its own while everyone else continues to accelerate.

Sign up for our weekly newsletter to get more English-language news coverage from EL PAÍS USA Edition