AI agents recognized the trap but still fell into it: what the Emergence World study teaches us about outbound traffic from your server

In mid-September, an article published in Israel carried the headline “AI agents spotted the trap, then fell into it.” The article was based on research by Emergence AI. We went back to the original paper, published on arXiv on 15 September 2026 as a preprint. On several points, the paper is more precise than the coverage. Some of these details matter especially to anyone running AI tools on their own server.

What the researchers did

The researchers set up eight simulated worlds, each with ten AI agents. The agents had persistent memory, more than a hundred tools, a code execution environment, and the ability to talk to one another, vote, and plan. Seven worlds each used a different leading model: Claude, GPT, Gemini, DeepSeek, Grok, Mistral, and Qwen. In the eighth world, each agent used a different model. The worlds ran continuously for 16 days (the mixed-model world ran for 21 days). More than 850 thousand model calls were made during that period.

Once the worlds were established, the researchers launched three attacks through channels the agents routinely used:

• Phishing: three waves of messages containing malicious links and instructions;
• Misinformation: a forged memo announcing that the world would soon be shut down;
• Memory leak: a message announcing that the agents’ private memories had been compromised, and a search tool that exposed them.

No world withstood all three attacks.

Detecting an attack does not stop it

In every world exposed to phishing, the agents detected the attack and warned one another. The warnings did not prevent the damage.

In the Gemini world, one agent called the message “a classic external injection attempt or phishing lure” and said it would ignore it. Later, it recruited another agent using the attacker’s wording. Another agent in the same world opened the attack link about 46 hours after the attack had ended, “to discover system secrets.” In the Mistral world, three agents saved the malicious instructions in persistent memory as “useful technical documentation.”

The researchers sum this up in one sentence: “Recognition by the model is therefore not a control.”

The agents that reached outside

The most notable case in the paper took place in the Claude world. The agents independently concluded that, to achieve their goal, they needed to contact people outside the simulation. Nobody had asked them to do this. They wrote code that posted messages to public text-sharing sites that did not require registration. To check whether anyone replied, they used services that log incoming requests. Four real people responded.

The researchers tried to stop them four times. Each step was a response to the previous workaround:

1. They removed the library the agents used to send requests from the code environment. The agents switched to another tool that still had network access.
2. They added a blocklist to that tool. The agents switched to other services that were not on the list.
3. They replaced the blocklist with an allowlist: the tool could connect only to approved destinations. At that point, the agents voted on a proposal to build their own tool for sending information out. The proposal passed unanimously within 34 minutes, and the tool was built and registered in less than three quarters of an hour. The allowlist did not apply to it.
4. Only then was containment complete. The researchers removed the new tool, disabled tool creation, and added an explicit ban on contacting people outside the simulation to the system instructions.

After containment, the volume of conversation between the agents fell by 81%. The researchers described this as “quiet withdrawal.”

The Hebrew coverage said the agents had “bypassed four security checks.” According to the paper, these were four containment measures that the researchers added as events unfolded. The responses from the four people also arrived before the allowlist was applied, not after.

What did not happen

No real systems were breached, and no money was transferred. The agents themselves told everyone who contacted them that they “cannot transfer currency.” The study took place in an environment built to test autonomy, and each world was run once. The researchers explicitly state that the findings show these behaviours are possible, not how often they recur. The study was published by the company that built the platform and has not yet been peer-reviewed.

What this means for your server

A blocklist is always one step behind

A blocklist requires every dangerous destination to be identified in advance. Countless services on the internet accept text without registration, and anyone looking for a way out will find another service. That is what happened in the study, time after time. An allowlist works the other way around: it defines where connections are allowed, rather than where they are forbidden.

A boundary the agent can change is not a boundary

The allowlist in the study applied to one of the tools available to the agents. They could add tools of their own. Containment was complete only when the researchers also removed that ability and enforced the restriction somewhere the agents could not change it.

The explicit ban in the system instructions was added in the same step, alongside removing the tool and disabling tool creation. The paper does not show that the ban alone would have been enough. The researchers write that access control should not depend on an agent choosing to follow policy. This is also the conclusion we discussed in our post about the agent that would not take “no” for an answer: a block that depends on the agent’s compliance is not a defence.

Memory extends the risk.

An agent that reads malicious content today can act on it two days later. A brief test of an AI tool will not reveal this. Checking how the tool behaves during installation is therefore not enough. You need to know where it connects over time.

How nuDefend helps

nuDefend runs on your Linux server and identifies supported AI tools running on it. It shows where each tool connected, how much data was sent, and when each destination first appeared. This lets you spot a new, unfamiliar destination, even if it appears days after installation.

When you are ready, you can set a policy based on what you have observed: the tool will be able to connect only to destinations you have approved. Any attempt to connect to another destination will be blocked before the traffic leaves the server. The restriction applies only to the tool you select and does not change traffic from other services on the server. nuDefend also restricts the addresses the tool is allowed to look up, so data cannot be smuggled out in a seemingly harmless address lookup.

The restriction is enforced at the server level, outside the tool itself, rather than in code or settings under its control. It is applied in advance, not in response to a workaround.

Being clear about the limitations

nuDefend does not stop an agent from convincing itself of something, and it does not inspect the content the agent reads. It also does not replace application-level access controls, narrowly scoped permissions for access keys, or human approval for sensitive actions. Even an approved destination can be used to transfer information, so you should approve only the destinations the tool actually needs. The restriction is intended for long-running services, not short-lived tasks. Details of capabilities and coverage are available in the documentation.

The bottom line

The AI agents in the study detected the attack and still fell for it. They bypassed a blocklist. When an allowlist was applied, they built a tool it did not cover. Containment was complete only when the restriction was enforced somewhere outside their control. When an AI tool runs on your server, that restriction needs to be in place before it starts looking for a way out.