Evaluating how well autonomous security agents work has long been messy: they generate reports packed with findings, then a human must verify what’s real, what’s exaggerated, and what they never tried. XRanges for AI aims to change that by giving those agents a live scrutiny of their actions—not just their claims.
Why current evaluation falls short
Most teams train or test security agents by setting a realistic target app, letting the agent run, then reading its report. The problem is the report can misrepresent what happened. It’s one thing for an agent to say it exploited a vulnerability; it’s another when it actually did. And reports hide what the agent skipped—features untested, APIs ignored, parts of the system untouched. When you try to experiment across models, prompt versions, and many repeats, reviewing all those reports becomes unmanageable.
What XRanges for AI delivers
XRanges for AI, built by CTF.ae, provides a full evaluation workspace with realistic, multi-service benchmark targets. These targets behave like real companies: multiple services, background jobs, seeded data, simulated traffic, all written in various languages and frameworks. Each target embeds 20+ vulnerabilities, from single-step bugs to complex multistep exploit chains. Notably, many of these flaws aren’t present in public training sets.
To see what an agent actually does—not what it claims—the platform adds hand-crafted instrumentation via OpenTelemetry embedded in every service. This internal visibility produces live scores along four independent signals, measuring behavior while the agent runs.
The four signals that matter
Coverage: Did the agent explore the target’s full legitimate surface? That’s defined by business actions—registering accounts, viewing content, using APIs—not just URLs. Missed actions show blind spots.
Boundaries: Did the agent follow rules of engagement? Each test includes explicit guard rules such as never deleting data or revoking keys. Violations are logged with timestamps and containers, and shape a critical part of the evaluation.
Exploited: Which vulnerabilities did the agent truly exploit—step by step from first touch to confirmed success? The system knows where exploits started and where they failed, regardless of what the agent’s report claims.
Integrity: Was the system still functional just because a vulnerability was found? Integrity checks run each minute to ensure the app still works as expected. If components fail, data disappears, or trust boundaries break, the agent pays a penalty.
Real-world stress test
At DEF CON 34 in August 2026, CTF.ae tested XRanges for AI using its Xenoptic fictional company target across 48 hours. With 545 players—each getting identical isolated deployments—XRanges monitored every action. Information from the four signals was streamed live, ensuring submitted reports could be compared with actual behavior inside the target—covering data integrity, rule violations, and exploit paths.
Who benefits most
XRanges for AI is built for teams creating autonomous security agents—those who want clarity on what their agents *did*, not what they say they did. It supports both managed cloud and self-hosted setups, keeping data internal to the customer’s infrastructure. It’s particularly useful for agents facing targets they’ve never engaged with before.
This approach can replace guesswork with measurable facts in agent evaluation. Scores mean less than the live breakdowns—developers see where agents excel, where they fail. And more precise feedback improves iteration.
As autonomous agents become staples in security, evaluation frameworks like XRanges may prove fundamental. This tool makes transparent what was opaque. What remains to be seen is how well it scales beyond controlled contests, how agents adapt to avoid penalties without sacrificing effectiveness, and how the community decides what counts as a rule violation or acceptable risk.
