BLOG
Case Study
PhD-Level STEM Training Data: How Experts Build Tasks Frontier Models Can’t Solve
September 8, 2026
TL;DR: PhD-level STEM training data for frontier model evaluation consists of original reasoning tasks in math, physics, chemistry, and biology that test capabilities designed to challenge current frontier models. Strong tasks combine one reproducible, scientifically defensible answer with enough novelty and difficulty to challenge current frontier models.
Frontier model evaluation requires STEM problems that test genuine reasoning rather than familiarity with existing problem structures. Expert-authored reasoning tasks are built from original literature research, use scientific inputs traced to peer-reviewed sources, and include hand-computed gold solutions that can be independently verified. Each problem also needs precise constraints so qualified researchers can reproduce one correct answer.
This guide explains what makes PhD STEM training data original, why scientific inputs need citations, how well-posedness makes evaluation results reliable, and why adversarial STEM benchmarks require active domain expertise rather than generalist annotation.
An original STEM task is constructed from the researcher’s own literature review rather than adapted from a published problem set. Changing a constant, compound, or other surface detail can leave the underlying problem structure intact, creating a new version of an existing task rather than a genuinely new reasoning challenge.
For PhD STEM training data, researchers start with the underlying science and construct a problem that did not previously exist in that form. This gives frontier model evaluation genuinely new, expert-authored reasoning tasks rather than variations of already-solved problems, an important distinction when building adversarial STEM benchmarks designed to challenge current models.
Reliable PhD STEM training data combines originality, scientific traceability, reproducibility, precise technical authoring, well-posedness, and adversarial difficulty. Each task must have a verifiable correct answer while remaining difficult enough to challenge current frontier models.
These six requirements work together. A task can be original but scientifically unsupported, well-posed but too easy, or difficult but impossible to verify. Reliable expert-authored reasoning tasks have to meet all six standards at once.
Every constant, variable, governing equation, and reaction pathway in an expert-authored STEM task needs to be traceable to peer-reviewed literature, including a DOI and author attribution**. Recent sources are preferred when newer research has superseded older values or understanding.**
Traceability ensures the gold solution is built on scientific inputs that apply to the specific problem. A value can be valid in one context and inappropriate in another, so confirming that a source exists is not enough. As our guide to hiring STEM AI trainers explains, domain expertise matters when the work requires identifying errors and making judgments specific to a technical field.
For frontier model evaluation in physics, chemistry, biology, and mathematics, that distinction matters. A difficult task is only useful if the science behind its correct answer can also be independently verified.
Well-posedness means a STEM training task is designed to produce exactly one correct, reproducible answer. A qualified solver should have enough information to reach that answer without guessing the author’s intent, adding missing assumptions, or choosing between multiple defensible interpretations.
For expert-authored reasoning tasks, a well-posed problem needs:
Well-posedness is harder to establish when the problem itself is original. Existing problems already have known formulations and solutions, while PhD STEM training data has to be constructed and validated from scratch. The researcher has to create a novel problem and prove that a competent solver can still reproduce exactly one answer.
An adversarial STEM benchmark contains problems that qualified experts can solve reproducibly, but current frontier models still struggle to solve. Originality, scientific traceability, and well-posedness establish that a task is valid. Adversarial difficulty establishes that it is useful for evaluating advanced models.
The task therefore has to meet two standards at once: a qualified researcher must be able to reproduce the correct answer, while the frontier models being evaluated still need to fail it. A difficult problem is not useful if it is ambiguous or unsolvable, just as a scientifically rigorous problem provides limited evaluation value if current models can already solve it consistently.
FrontierMath illustrates how much this difficulty bar can change an evaluation. At its 2024 launch, six leading AI models solved fewer than 2% of its original, expert-crafted mathematics problems, a sharp contrast with their performance on established benchmarks such as GSM8K, where leading models scored above 95%, and MATH, where they reached the 70-85% range.
A frontier-level STEM evaluation task combines advanced domain knowledge with a precisely specified problem that produces one reproducible correct answer. One accepted task required calculating the total molar Gibbs free energy of a transactinide oxide at 850 K and 25 MPa.
As detailed in the How a Frontier AI Lab Got STEM Evaluation Tasks Frontier Models Couldn’t Solve case study the calculation integrated:
The task also had to meet the core requirements for PhD STEM training data: peer-reviewed scientific inputs, a hand-computed and independently verified gold solution, and one reproducible correct answer.
For frontier model evaluation in physics and chemistry, building problems at this level requires the domain expertise to select the right inputs, combine the relevant physical effects, and define a verifiable solution.
PhD-level STEM evaluation requires subject-matter judgment that cannot be reduced to an annotation rubric. Confirming that a DOI exists is straightforward. Determining whether a cited value or equation applies to the specific scientific conditions of a problem requires active domain expertise.
The qualifications required for AI model trainers become more specialized as the evaluation task becomes more technical. For PhD-level STEM work, that expertise is needed to:
Accepted tasks take more than a full working day each. For frontier model evaluation in advanced STEM domains, researchers need the subject-matter expertise to make scientific judgments throughout the process, from constructing the problem to validating its answer.
PhD STEM training data has to be original without becoming ambiguous. Unlike established benchmark problems with known formulations and solutions, each new task needs its scientific basis, gold solution, and constraints established from scratch so another qualified researcher can reproduce the same answer.
Athyna Intelligence matches AI teams with vetted researchers for specialized data generation and model evaluation. For PhD STEM training data like this, the bench requires PhDs with active expertise in math, physics, chemistry, and biology who can author original problems, establish reproducible gold solutions, and test them against the adversarial standard.
Need expert-authored STEM training data for frontier model evaluation? Talk to our team.
Athyna's content specialists covering global hiring and AI training trends for growing teams.
.svg)
PhD-level STEM training data consists of original math, physics, chemistry, and biology tasks authored by researchers with active domain expertise. Every task is novel, grounded in peer-reviewed literature, and built to a standard that current frontier models cannot solve while remaining reliably solvable by a competent human expert.
An original task starts from the researcher's own literature review. It is not a published problem with altered constants or a new compound swapped in. Frontier models have effectively memorized standard problem sets, so adapted tasks offer no real evaluation signal. A task built from scratch has no prior version for a model to recognize or pattern-match against.
Because the work is not a labeling exercise. Determining whether a literature value applies under a specific thermodynamic condition, whether an equation fits a particular material system, and whether a prompt is genuinely unambiguous all require active domain expertise. A rubric can check that a citation exists. It cannot check whether the cited value is the right one for that application.
A benchmark task is designed to evaluate a model's capability by exposing where it fails. A training task is designed to improve a model's capability by providing high-quality examples with verified correct answers. Both require the same rigor in sourcing, well-posedness, and domain expertise. The difference is in how the output is used, not in how the task is built.
Hand-computed gold solutions can be independently derived and verified without relying on the model being evaluated. If the answer requires a model to produce it in the first place, it cannot serve as a reliable ground truth. Researchers solve the task by hand, then verify the result separately before the task is accepted.
