How a Frontier AI Lab Got STEM Evaluation Tasks Frontier Models Couldn’t Solve

The Challenge: Standard Problem Sets Stopped Testing the Limits

A frontier AI lab evaluating model reasoning across physics, chemistry, and other STEM domains ran into a difficult problem: standard problem sets were no longer testing what the lab needed to measure.

Frontier models have effectively ingested the public internet, textbooks included. Hand one a lightly adapted version of a known problem, swap a constant, change a compound, and the model can still recognize the shape of the question. The result looks like a new task and functions like an old one.

For meaningful frontier model evaluation, the lab needed:

• Genuinely novel STEM tasks, not adapted from any existing problem set
• Full citation traceability, with every constant and equation sourced to peer-reviewed literature
• Exactly one verifiable answer per task, reproducible by any competent solver
• A difficulty bar current frontier models still fail

This wasn’t work for generalist annotators. The lab needed PhD-level researchers who could judge whether the science, sourcing, and solution held up within their own domains.

The Solution: PhD Researchers Building Original STEM Tasks

Athyna Intelligence matched the lab with PhD researchers across math, physics, chemistry, and biology, each with active domain expertise in the areas the lab needed covered. Each accepted task took more than a full working day of research, authoring, computation, and verification.

The researchers developed expert-authored reasoning tasks from literature review through verified gold solution, giving the lab original STEM training data built specifically for frontier model evaluation.

The researchers built expert-authored reasoning tasks from literature review through verified gold solution, giving the lab original STEM training data designed specifically for frontier model evaluation.

H3: Authoring original tasks from scratch

Athyna’s researchers built each task from their own literature review rather than adapting a published problem set. There was no prior version to pattern-match against because none existed.

That gave the lab genuinely original evaluation material instead of familiar problems with different inputs.

H3: Sourcing every input to peer-reviewed literature

Athyna’s researchers traced every constant, variable, governing equation, and reaction pathway back to a peer-reviewed publication, complete with a DOI and author attribution. Recent literature was preferred where newer research had updated the underlying science.

That level of sourcing mattered because a scientifically plausible value isn’t necessarily the correct one for a particular application. Using an outdated input or applying it under the wrong conditions could change the gold answer entirely.

H3: Hand-computing and independently verifying gold solutions

Each researcher solved their own task by hand, with no model assistance, and the gold solution was then independently verified.

Tasks were authored in LaTeX and rendered to a code-based submission format, so the work required technical writing fluency alongside domain depth. If an answer couldn’t be derived and checked without a model in the loop, the task wasn’t ready.

H3: Meeting the double bar

Every task had to meet two standards before acceptance:

• One reproducible answer: The prompt had to be well-posed enough that any competent solver working from the same information would reach the same answer.
• Frontier-model difficulty: The task had to remain hard enough that current frontier models still failed it.

Athyna’s researchers had to establish both from scratch. Each task then went through multiple rounds of review, with revision cycles on sourcing and specification, before acceptance.

That combination gave the lab what it needed from adversarial STEM benchmarks: original problems with verifiable answers that still tested the limits of current frontier models.