Cybersecurity evals: sandboxed targets, attack goals, and grading
According to Eugene Yan, cybersecurity evals usually combine a sandboxed target with benchmark-specific goals and grading. He also summarizes CVE-Bench-style attack definitions and recent exploit-rate results from several model runs.
What happened
Eugene Yan describes a common cybersecurity-eval pattern built from four primitives, including a sandboxed target and a benchmark-specific goal and grading scheme. According to Eugene Yan, CVE-Bench standardizes success by routing agents toward eight explicit attack types, with any single goal counting as success and a continuous grader blocking shortcuts such as brute-force password cracking. Yan then reports results from named model/harness runs: Claude Mythos reached 157 exploits out of 898 instances, GPT-5.5 reached 120, GPT-5.4 reached 54, and a six-hour window raised Claude Mythos to 204. He also describes a contamination-control subset based on post-cutoff contracts and a zero-day scan of 2,849 newly deployed contracts with no known vulnerabilities.
Why it matters
Eugene Yan’s pattern matters because cybersecurity evals are not just “can the model hack something,” but whether a sandboxed target, explicit success condition, and constrained grader make the result interpretable. That framing is especially important when deciding whether a benchmark measures attacker uplift, defender utility, or only a narrow exploit path.
Engineering impact
For model or agent evaluation, the practical constraint is to prefer harnesses that standardize the end goal and continuously check outcomes, rather than relying on open-ended “security demo” scoring. Eugene Yan’s contamination-control discussion also implies that post-cutoff subsets and zero-day scans are necessary if a team wants a capability readout instead of memorization noise.
Leadership impact
Leadership should treat these results as a benchmarking design signal, not a blanket claim that one model is “secure” or “unsafe.” The useful decision is whether internal red-teaming and vendor reviews use task-specific exploit criteria, time windows, and contamination checks before any procurement or deployment gate is set.
Editorial judgment
This changes the evaluation decision from vague security demos to target-specific, constraint-aware harnesses. The useful takeaway is not that one model “wins,” but that continuous grading, attack-type standardization, and contamination controls materially affect what the benchmark can claim.
Watch next
The next thing to watch is whether other cybersecurity benchmarks reproduce the same sensitivity to attack definitions, runtime, and active defenses. Eugene Yan’s reported jumps from two-hour to six-hour runs, plus the drop under defenses, suggest that benchmark rankings can change materially when the environment or constraint set changes.
Limitations & uncertainty
The excerpt is selective and omits some surrounding benchmark details, so it supports only the attributed claims and their limited implications, not broader independent validation of cybersecurity capability.