Anthropic's AI-resistant technical evaluations: redesigning a performance-engineering take-home as Claude improves
Anthropic Engineering says its performance-engineering take-home, used since early 2024, has been redesigned as Claude improved and under the same time limit began matching or surpassing strong human applicants.
What happened
Anthropic Engineering describes a take-home test it has used since early 2024 to evaluate performance-engineering candidates on a simulated accelerator, with more than 1,000 candidates completing it and dozens later joining the company as performance engineers. The company says the task was predictive early on, but successive Claude models forced redesigns because the same time-limited format stopped separating top human applicants from model output. Anthropic reports that Claude Opus 4 outperformed most applicants and that Claude Opus 4.5 matched even the strongest candidates under the same constraint. The article also frames the task as a systems-oriented optimization problem rather than a deep-learning exercise, which makes the evaluation more about debugging, parallelism, correctness and performance analysis than model-specific knowledge.
Why it matters
Anthropic Engineering’s example shows a practical hiring constraint: a take-home can be useful until models become strong enough to meet the same time-boxed bar as top humans. The editorial implication is limited but concrete: evaluation designs that rely on a single fixed task may need periodic redesign to stay discriminative.
Engineering impact
The reported cycle counts make the engineering lesson operational, not abstract: the benchmark must be judged against a published threshold and comparable harness conditions, because Anthropic says Claude Opus 4.5 reached 1579 cycles in its test-time harness and 1790 in a casual session. That points to evaluation drift as a systems problem, not just a hiring-process issue.
Leadership impact
For leaders, the decision is whether candidate assessment should optimize for short-term comparability or for staying ahead of model capability. Anthropic’s account suggests that once a model matches the strongest applicants under the same limit, the organization must either increase task specificity or accept weaker separation in the hiring funnel.
Editorial judgment
This changes a concrete hiring tradeoff: time-boxed coding exercises can lose discriminatory power as models improve, so teams may need iterative, systems-like tasks with tighter success thresholds. Anthropic’s own report is useful here, but only as a company-described example.
Watch next
The most relevant next check is whether Anthropic’s new take-home remains discriminative as models and harnesses improve, since the company says Claude Opus 4.5 can be pushed from 1579 to 1363 cycles with more compute. Evidence remains limited to Anthropic’s own publication and should be treated as a company-reported example, not independent validation.
Limitations & uncertainty
This analysis relies on a single first-party company publication describing Anthropic’s own hiring benchmark and results. The reported cycle counts and candidate-selection effects are Anthropic’s claims, not independently reproduced measurements, so broader generalization to other roles or organizations remains uncertain.