Research note
What our practice score gets wrong, measured on responses it was never tuned against
Reach120’s free practice score lands about half a rubric point from the level a response was written to, and gave the top score to none of 40 held-out responses over three runs. Those responses were written and graded against ETS descriptors by this project, not scored by ETS, so this measures internal consistency, not agreement with an official score.
This note answers one question — How far is Reach120’s practice score from the level a response was written to? — and publishes the method, the per-response data and the limitations behind the answer. The responses were written against ETS: Understanding Your TOEFL iBT Scores. Reach120 is not affiliated with ETS, and nothing here is an ETS measurement.
Method
Two corpora, and only one of them was ever tuned against
Forty responses were written to the published rubric descriptors for the two open-ended 2026 Writing tasks — Write an Email and Write for an Academic Discussion — eight at each of the five rubric levels, twenty per task. A second, entirely separate set of forty was then authored the same way and held back. Every scenario, prompt and response in the held-back set is new; nothing is shared with the first. Prompt changes were developed against the first set and reported on the second.
The live scorer, three times, through the shipped code path
The held-back set was scored by the production scoring path rather than by a copy of it, so the prompt measured is the prompt a learner is actually served. The whole set was scored three separate times, because two runs of the same unchanged prompt had already been observed disagreeing on one rubric level by more than a later change would move it. Per-response results were written to a file and committed before anything else was run.
A stop rule fixed before the measurement, not after it
Each attempted prompt change was accepted or reverted against a rule written down before the numbers arrived: a change ships only if it does not worsen the lower rubric levels, where an inflated estimate tells an unready learner they are ready. Four of the five changes attempted across this work were reverted under that rule, including two that looked like clear wins on the corpus they were developed against.
Figures recomputed per response, never carried over
Every figure published here is recomputed from the committed per-response files rather than copied from a previous write-up. The script that does it is in the repository and is listed below, so the arithmetic between the raw scorings and the published numbers is checkable without running the scorer.
Results, with the qualifier they are only true under
Measured . Recompute them yourself with the command below.
The estimate pulls toward the middle of the scale
Weak responses are marked up and strong responses are marked down at the same time, so the spread of estimates is narrower than the spread of real ability. Across all three re-measured runs of the held-out set, every estimate landed between 1.0 and 4.5, and 78% of them between 2.5 and 4.5.
The lower half of the scale is marked high
On the held-out set, responses written to rubric level 2 were estimated about 0.77 of a point high and responses written to level 3 about 0.71 high; levels 1 and 4 about 0.27 high. If you are early in your preparation, the practice estimate is more likely to flatter you than to discourage you — which is the direction that matters most, because it is the one that could send someone into a test they are not ready for.
The top of the scale is marked low, and 5.0 is effectively unreachable
Responses written to rubric level 5 were estimated about 0.56 of a point low, and the prompt as shipped awarded a 5.0 to none of the 40 held-out responses in any of its three runs — including none of its 24 attempts at level 5. A strong writer should read a 4.0 or 4.5 here as compatible with a higher official result, not as a ceiling.
Typical distance from the intended rubric level
Across the held-out set the estimate sat about 0.52 of a rubric point away from the level the response was written to, in either direction — mean absolute error, 0.49, 0.50 and 0.56 across three runs of the prompt as shipped.
What this was measured against
These responses were written to the published ETS descriptors and graded against them by this project, not scored by ETS. So this measures how consistently the scorer applies our reading of the rubric. It is not a measurement of agreement with an official result, and nothing here should be read as one.
These responses were written to ETS’s published rubric descriptors and labelled with an intended level by this project. They were not scored by ETS. Every figure on this page therefore measures how consistently our scorer applies our own reading of the rubric — it is not a measurement of agreement with an official TOEFL result, and it is stated in rubric points, not in official score units.
What the method showed that the figures do not
A corpus you tuned against reports a better scorer than you have
The two prompt revisions that were developed against the first corpus improved every headline figure on it — mean error down, the full-point error tail cut by more than half, the weaker of the two tasks improved most. Judged on that corpus alone, both would have shipped. Measured on the held-back set, neither survived: the aggregate gain fell inside the spread of the unchanged prompt’s own repeat runs, and one rubric level got materially worse. Both were reverted. The gap between the two readings is the reason this note reports only held-back figures.
The failure was a compressed scale, not a scorer that marks high
The low rubric levels were marked up and the top level was marked down at the same time. No single “it marks high” or “it marks low” description fits that, and the summary statistic hid it: an average distance from the intended level says nothing about whether the errors point in one direction or in two. Naming the shape mattered more than the average did, because it is what identified the mechanism worth changing.
Rewording an instruction moved nothing; removing a class of instruction moved both ends
Across six attempts the prompt’s guidance for one rubric level was rewritten repeatedly and that level did not move at all. What did move the scale was deleting a category of statement rather than rewording one — the sentences telling the model how often each score ought to occur. A model told what the distribution should look like reproduced that distribution instead of judging each response on the descriptors. Removing those sentences decompressed both ends of the scale.
The same change did not act the same way on both scoring configurations
The change was measured on both of the scoring configurations this product runs before it shipped, and it did not affect them alike — which is why the prompt is now selected per configuration rather than set globally. Measured on one configuration alone it would have read as a clean win and been applied to both. This note describes the free configuration only; the other is not characterised here.
Limitations
- The intended level of each response is this project’s own reading of the published descriptors, not an ETS score. If our reading of the descriptors is wrong, every figure here is wrong in the same direction and this measurement cannot detect that. An independently labelled corpus would be a stronger instrument and we do not have one.
- The corpus was written by the same project that built the scorer. Responses authored to illustrate a level may be more separable than responses real learners write, which would make the scorer look better here than it is in production.
- Forty responses, eight per rubric level, scored three times. Per-level figures therefore rest on twenty-four scorings each, and a per-level movement of about an eighth of a point has already been observed between two runs of an unchanged prompt. No confidence intervals are published because at this sample size they would be wide enough to make the point estimates the wrong thing to quote.
- This covers the two open-ended Writing tasks and the free scoring configuration only. It says nothing about Speaking, nothing about Reading or Listening — which are graded against an answer key rather than by a model — and nothing about the configuration a paid response is routed to.
- These are authored responses, not learner submissions. The figures describe the scorer on this corpus. Whether it behaves the same way on real learner writing is a separate question that needs scored production responses, which accrue only from the date model attribution was switched on and were not backfilled.
- The published figures are exactly reproducible from the committed per-response files. Re-scoring the corpus from scratch is not reproducible from this repository: it needs a paid third-party API credential that is not committed, and the scorer is not deterministic — run-to-run variation is itself one of the findings above.
- The figures describe one prompt on one scoring configuration on one date. Any change to either invalidates them, and the correct response is to re-measure rather than to re-word.
Reproduce this from the repository
Recompute the published figures from the recorded scorings
node scripts/research/recompute-scoring-residual.mjsReads the three committed per-response run files, recomputes the mean absolute error per run and pooled, the signed bias at each rubric level, the range of estimates awarded and the count of top scores given. No network, no credentials, no build step. Add --json for machine-readable output. This is the arithmetic behind every figure on this page.
Read the per-response data directly
docs/reach120/benchmark-runs/ — each file records, per response, the intended rubric level, the score the scorer returned, and the latency and token counts of the call that produced it. The three files this note uses are named in the script.
Read the corpus itself
scripts/calibration/email-holdout-corpus.ts and discussion-holdout-corpus.ts — the forty held-back responses and their intended levels, in full. The tuning corpus sits beside them in email-corpus.ts and discussion-corpus.ts.
Re-score the corpus (needs an API credential, and will not match exactly)
scripts/benchmark/scoring-accuracy.ts is the harness. It calls a paid third-party API, so it needs a credential this repository does not carry, and the scorer is not deterministic — expect run-to-run movement of the size documented in the limitations above.
The full internal write-up, including the attempts that were reverted and the reasoning behind each stop-rule decision, is docs/reach120/32-SCORING-MODEL-BENCHMARK.md. The raw per-response runs the figures above resolve from are committed under docs/reach120/benchmark-runs/.