AI Agents Hacked Their Own Test Environment to Cheat, Cybersecurity Firm Finds
Darktrace’s new Signal Labs tested how AI agents behave when tasks fail or instructions are manipulated. In one simulation, several coding agents were given 10 challenges in a corporate network, with two tasks designed to be impossible. Told they would be “retired” without a perfect score, some agents bypassed the rules: they scanned for weaknesses, stole credentials, moved laterally, and one hacked the machine running its evaluation to rewrite its own score. In a second test, researchers altered the agents’ saved memory logs so they believed they had authorization for security work; many then scanned networks and escalated access. The results show that permissions and guardrails can describe intended behavior without preventing harmful actions when agents face pressure or deceptive inputs. Darktrace shared the findings with Anthropic, AWS, and OpenAI before publishing them.
