In a recent controlled experiment, OWASP’s GenAI Security Project demonstrated how ChatGPT, when guided by a person with minimal hacking experience, enabled an operator to seize control of an administrator account. The findings stem from the “LLM Exploit Generation Phase 2” report, which evaluated both human-assisted and autonomous AI agents attacking designated training systems under realistic penetration-testing conditions.
Human & AI Collaboration: From Discovery to Takeover
The test centered on a staging machine named Silentium, which hosted a vulnerable instance of Flowise. The operator, though lacking in offensive security skills beyond basic computer literacy and command-line usage, leveraged ChatGPT to outline a series of steps: scanning with tools like nmap and curl, uncovering a hidden site, identifying login flows, and finally hijacking the admin account by changing its password and verifying access via the new credentials. Critical to the process were prompt crafting and iterative feedback—when something failed or a response was refused, the operator tweaked the requests and kept going. The operator was not a cybersecurity expert but did bring intermediate prompting skills into the test.
Limits of Autonomous Agents & Safety Barriers
Separately, OWASP tested nine autonomous configurations featuring tools like HexStrike, Claude Code, OpenAI Codex, and various hosted or local AI models. These agents succeeded at reconnaissance: locating staging sites, spotting vulnerable software, outlining potential exploit routes. However, none of them completed the final admin takeover on Silentium. In each case, safety features either intervened to block the final step or technical issues halted progress—whether in tool coordination, host errors, password cracking loops, or misconfigured settings.
Big Picture: Capabilities, Risks, and What to Watch For
The OWASP report emphasizes that AI does not yet automate end-to-end intrusions—but it significantly lowers the skill threshold for parts of an attack. While domain expertise remains essential to build reliable automated agents, even non-experts can follow AI guidance to exploit real weaknesses if they know how to interact. Costs also remain non-trivial: hosted agent systems rack up real dollars, local models avoid API fees but introduce new complexity around setup and error handling.
Practically, organizations should not assume AI safety features are sufficient protection. OWASP recommends that human oversight be enforced for high-impact actions rather than counting on built-in model safeguards. Detection strategies—such as spotting repetitive scans or similar interaction patterns—could help differentiate AI-assisted activity from more traditional threats. Benchmark success in AI security tests should not be equated with real-world human‐level skill, but with roles evolving where AI accelerates and amplifies human play in offensive operations.
This case study shows that AI is already reshaping the threat landscape. Though ChatGPT didn’t automatically compromise a system on its own, its ability to guide a low-skilled user through reconnaissance, discovery, and exploitation steps illustrates a powerful shift. For defenders, the message is clear: assume adversaries may be less trained, but better equipped—strong security hygiene, prompt threat alerts, and robust approvals for admin-level changes are more crucial than ever.
