On October 13, 2026, the studio drafted a score for each of 1,000 short answers from ASAP-SAS, a public dataset that two trained raters had already scored: its draft matched the first rater's score exactly 65% of the time, where the two raters matched each other 88% of the time.
How It Is Measured
A Public Dataset, Scored by Two People
ASAP-SAS is the short answer set from the Hewlett Foundation's Automated Student Assessment Prize, published on Kaggle in 2012. Its training file holds 17,043 answers written by students in grades 8 and 10 to ten questions in science, biology, English, and English language arts, and two trained raters scored every answer from the question's published scoring guide. The first rater's score, Rater 1 in the tables below, is the dataset's official one; Rater 2 is the second.
Agreement is reported as quadratic weighted kappa, the standard measure for this dataset: 1 is perfect agreement, 0 is what chance alone would give, and a score one point off costs less agreement than a score two points off. Beside the studio's figure is the same measure between the two raters themselves, on the same answers and across every answer to the question, which is how closely a second careful person agrees. The last row of each table is the whole run: its kappas average the questions' kappas, and its percentages are taken over every answer.
Measured October 13, 2026
800 Answers to Eight Public Questions
ASAP-SAS
800 Answers
Balanced Tier
Mapping B
After the Evidence Check
We drew 100 answers at random from each of the eight questions in ASAP-SAS, 800 in all, and the studio scored each one through the same code that scores answers collected online: the same prompt, the same structured output, and the same single repair attempt, on the balanced tier. Each question became a rubric with one 1-point criterion per key element its published scoring guide lists, in the guide's own words, scored present or absent under a note saying there is no partial credit; the guide's own rule for counting elements into a score was then applied in code, so the score compared with the raters' is on their scale, and the question students saw became the job's question as before.
How Each Guide Became Criteria
Agreement, as Quadratic Weighted Kappa
Exact Matches and Misses
Of the 800 drafts, 532 matched the first rater exactly, 46 were one point low and 175 one point high, and 4 were two or more points low and 43 two or more points high. The studio reported high overall confidence on 528 of the 800 drafts, medium on 271 and low on 1; it routed 1 to Needs Review, and none took its one repair attempt.
Before its one repair attempt, the evidence check flagged none of the 800 drafts for quoting words the answer does not contain or naming a term as missing that it holds; after it, none still carried a flag.
Accept All High-Confidence would have recorded 499 of these 800 drafts unread, the ones that came back complete and high on every confidence the studio reports; of those, 373 (75%) matched the first rater exactly and 19 (4%) were two or more points off.
To check a figure yourself, download every answer's result (each answer id with what the studio drafted for it) and join it to the raters' scores in your own copy of ASAP-SAS.
- On every one of the eight questions, the studio sits below the two raters: its draft agreed with the first rater less closely than the second rater did.
- Each answer was scored element by element against its question's published guide, and the guide's own rule turned the count of elements credited into the score, so this measures whether the studio credits the elements the raters did; a draft can still land on the right score with the wrong elements credited, which the score alone cannot show.
- The answers were scored as typed text, so these figures say nothing about reading handwriting from a scan.
- The answers come from a public dataset released in 2012 that may sit in a modern model's training data, so read these figures as an optimistic bound.
- Questions 3 (Koala/Panda) and 4 (Invasive Species) were left out: each of their published guides describes one dimension of reading at three levels rather than elements to count, which no count can stand in for.
- Some of the questions' answers were handwritten and later transcribed, so they may carry transcription errors.
- Question 10's answers carry stray spaces inside some words, an artifact of the dataset.
- Question 10's guide prints its data table as a picture, so the table was typed into the question, as a teacher entering it would.
Measured with scripts/benchmark-rubric-scoring.ts at commit 7749bd9c, from train_rel_2.tsv with seed 20261013. A run is never edited once it is published; a later run is added above it, with its own date.
Measured October 13, 2026
1,000 Answers to Ten Public Questions
ASAP-SAS
1,000 Answers
Balanced Tier
Mapping A
After the Evidence Check
We drew 100 answers at random from each of the ten questions in ASAP-SAS, 1,000 in all, and the studio scored each one through the same code that scores answers collected online: the same prompt, the same structured output, and the same single repair attempt, on the balanced tier. Each question became a rubric with one criterion worth the question's top score, whose description is the question's published scoring guide word for word, and the question students saw, with its reading passage or data, became the job's question.
Agreement, as Quadratic Weighted Kappa
Exact Matches and Misses
Of the 1,000 drafts, 654 matched the first rater exactly, 97 were one point low and 218 one point high, and 4 were two or more points low and 27 two or more points high. The studio reported high overall confidence on 714 of the 1,000 drafts, medium on 285 and low on 1; it routed 1 to Needs Review, and none took its one repair attempt.
Before its one repair attempt, the evidence check flagged none of the 1,000 drafts for quoting words the answer does not contain or naming a term as missing that it holds; after it, none still carried a flag.
Accept All High-Confidence would have recorded 714 of these 1,000 drafts unread, the ones that came back complete and high on every confidence the studio reports; of those, 505 (71%) matched the first rater exactly and 14 (2%) were two or more points off.
To check a figure yourself, download every answer's result (each answer id with what the studio drafted for it) and join it to the raters' scores in your own copy of ASAP-SAS.
- On every one of the ten questions, the studio sits below the two raters: its draft agreed with the first rater less closely than the second rater did.
- Each answer was scored as a whole against its question's published guide, so this measures how often a drafted score lands where the raters' did, not whether every line of a rationale is right.
- The answers were scored as typed text, so these figures say nothing about reading handwriting from a scan.
- The answers come from a public dataset released in 2012 that may sit in a modern model's training data, so read these figures as an optimistic bound.
- Some of the questions' answers were handwritten and later transcribed, so they may carry transcription errors.
- Question 10's answers carry stray spaces inside some words, an artifact of the dataset.
- Question 10's guide prints its data table as a picture, so the table was typed into the question, as a teacher entering it would.
Measured with scripts/benchmark-rubric-scoring.ts at commit 29fad138, from train_rel_2.tsv with seed 20261013. A run is never edited once it is published; a later run is added above it, with its own date.
Measured October 13, 2026
1,000 Answers to Ten Public Questions
ASAP-SAS
1,000 Answers
Balanced Tier
Mapping A
Before the Evidence Check
We drew 100 answers at random from each of the ten questions in ASAP-SAS, 1,000 in all, and the studio scored each one through the same code that scores answers collected online: the same prompt, the same structured output, and the same single repair attempt, on the balanced tier. Each question became a rubric with one criterion worth the question's top score, whose description is the question's published scoring guide word for word, and the question students saw, with its reading passage or data, became the job's question.
Agreement, as Quadratic Weighted Kappa
Exact Matches and Misses
Of the 1,000 drafts, 667 matched the first rater exactly, 95 were one point low and 213 one point high, and 3 were two or more points low and 22 two or more points high. The studio reported high overall confidence on 759 of the 1,000 drafts, medium on 238 and low on 3; it routed 3 to Needs Review, and none took its one repair attempt.
Accept All High-Confidence would have recorded 759 of these 1,000 drafts unread, the ones that came back complete and high on every confidence the studio reports; of those, 539 (71%) matched the first rater exactly and 17 (2%) were two or more points off.
To check a figure yourself, download every answer's result (each answer id with what the studio drafted for it) and join it to the raters' scores in your own copy of ASAP-SAS.
- On every one of the ten questions, the studio sits below the two raters: its draft agreed with the first rater less closely than the second rater did.
- Each answer was scored as a whole against its question's published guide, so this measures how often a drafted score lands where the raters' did, not whether every line of a rationale is right.
- The answers were scored as typed text, so these figures say nothing about reading handwriting from a scan.
- The answers come from a public dataset released in 2012 that may sit in a modern model's training data, so read these figures as an optimistic bound.
- Some of the questions' answers were handwritten and later transcribed, so they may carry transcription errors.
- Question 10's answers carry stray spaces inside some words, an artifact of the dataset.
- Question 10's guide prints its data table as a picture, so the table was typed into the question, as a teacher entering it would.
Measured with scripts/benchmark-rubric-scoring.ts at commit 311bcfe3, from train_rel_2.tsv with seed 20261013. A run is never edited once it is published; a later run is added above it, with its own date.