How Athyna Intelligence Authors Agentic Coding Tasks for Terminal-Bench 2

TL;DR

• A qualifying task passes a two-sided bar. Competent human engineers solve it reliably, and current frontier models fail it, also reliably. Both sides get checked directly, every time.
• Reproducibility is the foundation everything else depends on. Docker-pinned environments run through the Harbor harness keep a task behaving identically on any machine, at any time, against any model.
• The scarce skill is engineering a failure on purpose. Anyone technical can stumble onto a task a model fails. Landing that failure inside a band where a human still succeeds is what separates a task that measures something real from one that just looks hard.

Athyna Intelligence authors agentic coding tasks for Terminal-Bench 2. Every task ships in a hermetically reproducible, Docker-pinned environment and runs through the Harbor harness. The bar for acceptance is simple to state and hard to hit: human engineers have to solve the task reliably, and current frontier models have to fail it, just as reliably.

One verified sample task in this pipeline cleared at pass@10 = 0 against a current frontier model. Here's what actually goes into building a task that holds up to that bar.

What Makes an Agentic Coding Task Qualify for a Benchmark?

An agentic coding task qualifies for a benchmark when it clears a bar that works in two directions at once. Human engineers need to solve it reliably, across attempts, no exceptions.

At the same time, current frontier models need to fail it, also reliably. If a model solves the task some of the time, the task isn't done yet. If a human can't solve it consistently either, something's broken on the other end.

Pass@k is the measurement behind pass@k acceptance. It captures a model's success rate across k independent attempts at the same task. A task that's doing its job as a benchmark item shows a human clearing it while the model's pass@k sits at or near zero.

One sample task authored in this pipeline cleared at pass@10 = 0 against a current frontier model. Ten independent attempts, zero successes. That's the shape SWE task creation takes when it's done right.

Why Is Authoring an Eval Task Harder Than Solving One?

Authoring an eval task is harder than solving one because it means building a failure into the task on purpose, by design, with a level of precision that ordinary bug-hunting never demands. Almost anyone technical can stumble onto a question a model happens to fail. Constructing a sealed environment where a specific class of model reliably fails, every time, takes something else entirely.

That precision has two ways to go wrong:

‍

The narrow band between those two failure modes is where a valid task lives. Landing there takes someone who understands both how the model class tends to fail and how human engineers would actually work through the problem. That judgment, the ability to place a task in that band on purpose, is what makes serious SWE task creation hard to commoditize.

Need this kind of task authoring for your own benchmark? Athyna Intelligence matches you with Terminal-Bench experts who bring this exact judgment to your pipeline.

What Does Terminal-Bench 2 Task Authoring Require for a Reproducible Environment?

A reproducible Terminal-Bench 2 task environment requires two things working together: an environment pinned at the container level, and consistent execution through a fixed harness. Miss either one and the task can't be trusted to behave the same way twice.

• Docker-pinned environments. Every task ships in an environment pinned at the container level, so it behaves the same on any machine, at any point in time, regardless of which model is being evaluated against it. It removes drift from dependency updates and sidesteps the classic "it worked last week" problem.
• The Harbor harness. Tasks run through Harbor, which handles execution consistently across runs. This is what keeps pass@k meaningful. If the harness introduces variance, the measurement stops meaning anything.

Reproducibility carries the whole acceptance bar on its back here. A task that behaves differently depending on when or where it runs can't support a claim like pass@10 = 0. The environment has to be a fixed, known quantity every single time, or the result is just noise.

How Is a Task Accepted or Rejected?

A task earns acceptance by clearing two checks, run directly against it rather than assumed:

‍

A task that passes only one side gets sent back. Too easy for the model, rejected. Too hard or too ambiguous for a human to solve reliably, also rejected.

The pass@10 = 0 result mentioned earlier is what a task looks like once it clears both checks. It's a byproduct of solid construction, and that's really the point: get the construction right, and the number follows on its own.

That construction is the judgment call described earlier, and it's the part that doesn't commoditize no matter how many tasks get written.

"A task like this only means something if it's built the way this one was: sealed, reproducible, and checked from both sides before anyone calls it done."

Building a task like this takes more than writing a hard problem. It takes an environment that holds up under scrutiny and a result that means something the moment it ships.

See more about Intelligence.