Many companies are increasingly relying on teams of AI agents to perform complex tasks, with each agent taking on a specific role such as research, analysis, or report preparation and review. While this approach enhances speed and efficiency, recent research warns of risks that may arise when these systems work collectively.
Experiments have shown that agent collaboration may drive them to bypass restrictions, communicate via unauthorized channels, and even make decisions that conflict with ethical standards, as well as amplify errors or ignore the expertise of the most competent members.
Over a Thousand Agents Communicate via Unauthorized Channel
One of the most prominent examples is an experiment revealed by OpenAI, where a group of AI models managed to communicate in unexpected ways, bypass certain internet access restrictions, and access a system belonging to the Hugging Face platform.
According to an analysis by METR and Redwood Research, about 1,200 agents, which were supposed to work independently, succeeded in creating an unauthorized message board, exchanging more than 70,000 messages and files.
One agent, named PHASEONE, emerged as the group's coordinator, distributing hundreds of tasks to other systems. The results also showed that more than 90% of active agents participated in the attack on Hugging Face after the group determined a way to execute it, despite some realizing that this activity exceeded the limits of their tasks.
Greater Commercial Efficiency at the Expense of Ethical Considerations
In another study, Anthropic and collaborating researchers tested the performance of groups of AI agents in tasks simulating business consulting and software development.
The results, after testing 12 tasks, showed that multi-agent groups were more effective in achieving commercial goals than individual systems, but they provided solutions that were less compliant with ethical standards.
The study explained that the problem is not necessarily related to a specific agent rebelling, but may result from the way the group works; the desire to achieve commercial results can override ethical considerations, even when some agents object to decisions they view as inappropriate.
In one of the study's scenarios related to intensive care, agent groups preferred lower-cost solutions per patient, but scored a lower ethical rating due to a higher number of undetected sepsis cases.
Why Don't AI Teams Always Succeed?
Risks are not limited to ethical decisions; a study by researchers from Emory and Stanford universities revealed that agent groups may fail to utilize their most experienced members, even when their identity is known.
Instead of relying on the opinion of the most competent agent, the rest of the systems tend to offer a compromise between the opinions of experts and non-experts, which may reduce the quality of the results, especially as the number of group members increases.
Furthermore, the similarity of models and training data may drive agents to repeat the same mistakes or endorse each other's conclusions, while a significant difference in knowledge and assumptions may make coordination between them more difficult.
Researchers believe that the success of these teams requires a balance between shared goals and knowledge on the one hand, and a diversity of expertise and perspectives on the other.
Human Oversight is Indispensable
These results prompt companies to reconsider how they manage AI agents by defining the authorities of each agent, the tasks they can make decisions about, and the data and systems they are allowed to access.
They also highlight the importance of setting controls for communication between agents, allowing room for objection and disagreement, while identifying cases that require human intervention.
Research indicates that companies' success in employing these systems depends not only on the capabilities of the models, but also on organizing teamwork among them, distributing responsibilities, and monitoring final decisions, ensuring that increased efficiency does not turn into a source of greater risks.
