GDPval-style tasks: what they are and why expert authoring decides their value

TL;DR

• A GDPval-style task has four parts, and an expert has to produce each one. A realistic request, the reference files it depends on, a finished gold deliverable, and a scoring rubric all come from the author’s own professional work.
• GDPval’s credibility comes from how its tasks were authored. GDPval’s expert authors averaged 14 years of experience, each deliverable took about seven hours of work, and tasks went through an average of five human reviews.
• Labs building their own sets need expert authors behind every task. Synthetic generation and generalist annotators can produce tasks at volume, but they can’t supply the professional judgment that makes a task separate strong models from weak ones.

GDPval-style tasks are real-world AI evaluation tasks in which an experienced professional turns a piece of their own work into a request with reference files, completes a gold deliverable, and writes a rubric for scoring model output. The quality of each task depends on the person who wrote it, which makes expert task authoring the most senior and hardest-to-replace part of the work. AI labs building their own evaluation sets on this model have to plan around their authors first. Knowing what GDPval is, and how its 1,320 tasks were authored, gives labs the most useful benchmark to plan against.

What is GDPval and what does it measure?

GDPval is an OpenAI benchmark that measures how well AI models complete economically valuable knowledge work, using tasks drawn from the real jobs of experienced professionals. The GDPval paper and OpenAI’s announcement set out its scope:

• 1,320 tasks in the full set, with 220 released publicly as a gold subset
• 44 occupations, from software developers and lawyers to registered nurses and financial analysts
• Nine industries, each contributing more than 5% of US GDP
• 30 fully reviewed tasks per occupation in the full set, and five in the gold subset
• Occupations that together earn about $3 trillion a year

GDPval measures quality through blind comparison. Graders from the relevant occupation receive the request, the reference files, and two or more unlabeled deliverables, one of them written by the human expert, and rank them without knowing which came from a model. The headline metric is how often a model’s deliverable is rated as good as or better than the expert’s.

Each task is also tied to the time and cost a professional would need to complete it. GDPval therefore tracks something exam-style benchmarks leave out: whether a model can hand back finished work a client would accept, at a known value in hours and dollars.

What makes a task GDPval-style?

A task is GDPval-style when it starts from real work product and asks a model to deliver a finished file that an expert in the field would accept. OpenAI contrasts GDPval with benchmarks that synthetically create exam-style questions, such as MMLU or Humanity’s Last Exam. Every GDPval task is instead built on a deliverable that exists in professional practice or one constructed the same way.

The paper describes two primary parts: a request and a deliverable. The public dataset adds a scoring rubric to each task. In practice, an expert has to produce four things.

‍

Two details from the GDPval paper show how much work sits behind each component. Just over two-thirds of tasks (67.7%) required at least one reference file, and some used as many as 38. Authors also mapped every request to O*NET, the US Department of Labor’s database of occupational tasks, so the set covers the real range of each job.

Who writes GDPval tasks, and what experience do they need?

GDPval tasks are written by senior industry professionals, and the GDPval paper lists what OpenAI required of every author:

• At least four years of professional experience in the occupation
• A resume showing professional recognition, promotion, and management responsibility
• A passed video interview
• A passed background check
• A completed training and quiz

The average author had 14 years of experience, and fewer than 10% of applicants were selected to contribute tasks to the full set. Each occupation had at least five qualified professionals, and past employers in the pool include Goldman Sachs, Kirkland & Ellis, Massachusetts General Hospital, Boeing, and the US Department of Justice.

Reviewers came from the same pool. OpenAI’s researchers identified the authors whose tasks were consistently strong, trained them as reviewers, and promoted the best reviewers to lead reviewers who mentored the next group. The people checking tasks had already shown they could write good ones.

The experience requirement exists because authors decide what a correct answer looks like. Only someone who has done the job knows which missing detail would make a client reject a deliverable, and the rubric can only capture judgment the author already has. Labs that hire for this work run into the same screening problem covered in our guide to common mistakes when hiring AI model trainers. The difference here is that the bar is set by years in the occupation.

How long does it take an expert to author a GDPval task?

Completing the gold deliverable alone takes an expert about seven hours on average. The GDPval paper shows how the expert time adds up across the life of a task.

‍

Each task’s reviews moved through three stages:

1. A generalist confirmed the task met project requirements.
2. An occupation-specific reviewer judged whether it was realistic and completable from the files provided.
3. A final reviewer iterated with the author until the task met the standard.

OpenAI hasn’t published a total authoring time per task, so labs should budget well beyond the deliverable hours and plan for grading costs that grow with every model they test.

Why can’t synthetic generation or generalist annotators replace expert authors?

Synthetic generation and generalist annotators can’t supply the professional judgment that makes a GDPval-style task difficult in the ways that matter. As a result, the tasks they produce are weaker at separating strong models from weak ones. GDPval’s own data shows where that separating power comes from.

Task length is the first signal. In the GDPval paper, model win rates were highest on tasks that take a professional zero to two hours and fell steadily as completion time rose. The longest tasks, the kind only an experienced professional can build and complete, are where models fall furthest behind experts. A set built only from short tasks makes models look closer to expert level than they are.

Context is the second signal. When OpenAI shortened prompts to 42% of their original length, leaving out details such as where to find data in the reference files, GPT-5 performed worse because it struggled to work out what was missing. Deciding what context a real assignment includes is a professional judgment, and a template-driven generator has no basis for making it.

GDPval’s own review pipeline drew the same line. OpenAI trusted generalist reviewers only to check project requirements and left the judgment of whether a task was realistic and completable to occupation experts. Of the four approaches, expert authoring is the only one that produces a professional-grade gold deliverable and rubric.

‍

Most labs can use more than one approach without lowering quality. Generalists can handle format checks and first-pass review. The request, the gold deliverable, and the rubric stay with expert authors.

How do you write a GDPval-style task with a rubric and gold deliverable?

Writing a GDPval-style task starts with a real piece of the author’s own work, and it is finished only when independent reviewers confirm that another professional could complete it from the materials provided. Labs building their own sets can turn GDPval’s method into six requirements for every author:

1. Start from a multi-hour task from recent work and map it to the occupation’s O*NET tasks.
2. Write the request with the role, audience, goal, constraints, and exact file format.
3. Gather the reference files and replace real names with fictitious ones.
4. Complete the gold deliverable to client standard and log the real hours.
5. Write point-weighted rubric criteria for format, required content, and accuracy.
6. Revise through review rounds with an occupation peer until a final reviewer signs off.

Rubric criteria deserve the closest review, because they decide what graders and automated scorers check. Across models, the GDPval paper found that experts most often preferred the human deliverable because the model hadn’t fully followed the instructions. Rubric criteria that check each instruction in the request give graders and automated scorers a consistent way to catch that failure. A good test is to ask whether a grader who never saw the gold deliverable could still score a submission accurately using the rubric alone.

Where can AI labs source expert-authored GDPval-style evaluation tasks?

AI labs can source expert-authored GDPval-style tasks from four places: their own staff, the open GDPval gold subset, experts they contract directly, or experienced professionals matched through a platform like Athyna Intelligence. The right mix depends on how many occupations the evaluation set needs to cover and how much senior staff time the lab can spare.

‍

The gold subset is the easiest starting point, and its limits are worth knowing. OpenAI’s dataset card includes a canary string, a marker meant to help keep the tasks out of training data. Any set that is public is still exposed to that risk. Labs that need private tasks, or occupations GDPval doesn’t cover, need their own authors.

Athyna Intelligence matches AI labs with experienced industry professionals, such as accountants, lawyers, and engineers, alongside PhD and Master’s graduates::

• Candidates are vetted and presented within four days
• Talent costs 40-60% less than equivalent US hires
• Experts are based mainly in Latin America, with coverage across Africa and APAC
• Most work happens within US working hours

Labs already building question sets with HLE experts or coding tasks with SWE-bench experts can staff occupational tasks the same way through Athyna’s domain experts for AI training. Experts work on the lab’s own platform, following its task spec and review process.

Why does expert task authoring decide whether real-world AI evaluation tasks work?

Expert task authoring decides whether real-world AI evaluation tasks work because the author sets the standard everything else is measured against: the request, the gold deliverable, and the rubric all reflect what the author knows about the job. OpenAI’s spec reflects that: it puts its effort into who writes each task and how many times each task is reviewed. The format of a request, reference files, a gold deliverable, and a rubric is easy to copy. The seniority, hours, and review rounds behind each task are what a lab has to plan and pay for.

The most useful takeaway for a lab building its own set is to secure the authors before designing the pipeline, since every later stage inherits their standard. If your lab is building real-world AI evaluation tasks across several occupations, talk to Athyna Intelligence about matching vetted industry professionals to the work your evaluation set needs to cover.

‍