The Challenge: A Benchmark Task That Actually Measures Something
A client needed more than another hard question. They needed agentic coding benchmark data that could isolate a real capability gap. Writing something difficult isn't the point. Writing something that fails models on purpose, and for the right reason, is.
They needed:
- A task with a two-sided bar: A competent human engineer had to solve it reliably. Current frontier models had to fail it, also reliably. Most attempts at this only hit one side.
- Precision, not just difficulty: Too easy, and the model solves it. That proves nothing about where its capability breaks down. Too obscure, and no human can solve it reliably either. That proves nothing about real engineering skill.
- Hermetic reproducibility: Every result had to hold up under scrutiny. Same environment, same behavior, on any machine, at any time, no matter which model was being tested.
- Terminal-Bench 2 spec compliance: The task needed to be built and executed to a defined standard, not assembled ad hoc.
Once the design principles were locked in, Athyna was ready to build. Here's how that happened.
The Solution: Senior Engineers
The shift came down to who was doing the authoring. Athyna Intelligence brought in 50+ senior engineers with real domain judgment for Terminal-Bench 2 task authoring- someone who understood how frontier models tend to fail and how a competent human actually works through a problem. This kind of SWE task creation depends on both instincts at once.
What made the difference:
Finding the narrow band between "too easy" and "too obscure"
- Designed the task to sit precisely between two failure modes, on purpose
- Calibrated difficulty against a specific model class's known weak points
- Kept the task within reach of ordinary, reliable human problem-solving
- Treated this judgment call, not the code itself, as the real work
Building a hermetically reproducible environment
- Sealed the task inside a Docker-pinned environment at the container level
- Removed drift from dependency updates and "it worked last week" problems
- Ran execution through the Harbor harness, which keeps results consistent across runs
- Made pass@k a meaningful number instead of a noisy one
Verifying both sides of the pass@k acceptance bar
- Ran a human solvability check: a competent engineer had to clear the task reliably, not on a lucky attempt
- Ran a model failure check: current frontier models tested at pass@k, with a qualifying result landing at or near pass@k = 0
- Sent back any task that only passed one side. Too easy, rejected. Too ambiguous for a human, also rejected.
- Accepted only tasks that cleared both checks
The Results: Verified Tasks With a Real Signal
Scaling task authoring for a frontier benchmark isn't just about writing hard problems. It's about proving each one measures the right thing, consistently, without cutting corners on reproducibility.
Key Outcomes
- Pass@10 = 0 against a current frontier model: 10 independent attempts, 0 successes
- Reliable human solve rate across attempts
- Fully reproducible, Docker-pinned environment built to Terminal-Bench 2 spec
- Execution verified through the Harbor harness for consistent, noise-free results
Beyond the numbers, the real impact was operational:
- A benchmark item that isolates a genuine model capability gap, not an artifact of a flaky environment
- A task built to hold up under scrutiny, from spec compliance to reproducibility
- A demonstration that authoring judgment, not just difficulty, is what makes a task hold up
The pass@10 = 0 result isn't a target that got engineered backward. It's what a task looks like once it's built and verified correctly, the number is a byproduct of the construction, not the goal itself.
.webp)
