We publish our AI's scoring accuracy every week. Here's the data.

#What mean absolute error actually means

IELTS bands move in steps of half a point. There is no 6.7 and no 7.3, so the smallest real disagreement between two markers is 0.5 of a band.

Mean absolute error, usually shortened to MAE, is the average size of the gap between the band our AI gives a script and the band on our label for that script, ignoring direction. An error of plus 0.5 and an error of minus 0.5 both count as 0.5. It is the plainest measure available and the hardest one to dress up, which is exactly why we chose it.

MAE on its own is not enough, so we publish two things it cannot tell you:

• Exact band agreement. How often the AI lands on the label exactly, with no gap at all.
• Agreement within half a band. How often it lands within one step of the scale. For a practice tool this is the number that matters most: if your feedback is one step out, the advice attached to it is still the right advice.

MAE has one more weakness. It hides direction, because it throws away the sign. A tool that is always half a band generous and a tool that is always half a band harsh have the same MAE and are wrong in opposite ways. So we publish the mean signed error too, in the box above.

Our internal pass mark is 0.50, one step on the scale. If a run comes back above it, or if any single script is more than a full band out, the system opens a ticket by itself and a human has to look at it before anything else happens. We are not claiming that half a band is the margin two human examiners differ by. We have never measured human examiners, so we are not going to put a number on that.

#How the calibration set is graded, and why that matters more than the score

Most accuracy claims skip this part, so we will lead with it. An accuracy figure is only ever as good as the thing it is measured against. Here is exactly what ours is measured against.

As this is written, in August 2026, the set holds 27 active scripts. Eighteen are Task 2 and nine are Task 1. Eighteen are Academic and nine are General Training. The labels run from band 4 to band 9 and include half bands. The scripts run from 121 to 320 words, so the set includes under length answers, which are a common real world failure and are capped by the official rules.

Every script in the set today was written in house and labelled against the official public IELTS band descriptors, criterion by criterion, with written notes recording why each band was given. Not one of them has been marked by a certified IELTS examiner.

So let us say the honest version of the headline out loud: the figure measures how closely the AI agrees with our labelling of a small set of scripts we wrote. It is not proof of agreement with the examiner who will mark your test. We think our labelling is careful, both of us are DELTA qualified and have taught this exam for years, and the notes on each script quote the descriptor that drove the decision. Careful is still not the same as official.

We are fixing this the slow way rather than the loud way. Provenance is stored on every row in the set, and the wording you see on this site is generated from that field rather than typed into a page. The day the first examiner marked script lands in the table, the description of the set changes by itself, on this article and on the quality page, because neither of them holds a hand written claim about it. Until then it says what is true today.

#How one weekly run works

The run is a scheduled job, not a person with a spreadsheet. Every step is in the repository.

1. Schedule. It fires every Saturday at 06:00 UTC on a timer. Nobody presses a button and nobody chooses the week.
2. Read. It pulls every active script out of the calibration table.
3. Score. It sends each one to the live production scoring endpoint. Not a copy, not a test harness, not a nicer prompt kept for demos. The same URL, the same model, the same code path a paying user hits.
4. Bypass the cache. Our scorer seals the first result for any given script so the same essay always gets the same score back. Left alone, the weekly run would replay those sealed results forever and measure nothing, so it asks for a fresh score every time.
5. Compare. It works out the gap for the overall band and for each of the four criteria, and finds the worst single script.
6. Write. It stores one row with the averages, the worst case, the drift against the previous run, and the full per script detail. Rows are never edited or deleted.
7. Alarm. If the average passes 0.50, or any one script is out by more than a full band, a ticket is opened automatically and tagged for a human.

The model is pinned to a specific version and the temperature is set to zero, so the run is measuring the scorer rather than the weather. The model name is recorded on every row, which is how you can tell whether a change in the number came from us or from a new model.

#The week the number was 1.20

The first row we ever published, on 23 May 2026, reported 1.20 bands of error, with a worst case of 2.50 bands on a single script. That is bad. It is still in the table and it is still in the chart above.

Two things were wrong, and neither of them was the scoring model.

First, coverage. That run scored five scripts. A five item sample is not a measurement, it is an anecdote, and it is far too easy to move by one bad label. Second, and worse, the scoring route has a deterministic fallback for when the model provider cannot be reached, and the run was recording those fallback results as though they were model scores. A fallback score is an infrastructure failure, not a measurement of anything.

Both holes are now closed in code rather than in a promise. A run that covers less than four fifths of the set refuses to publish at all and reports a failure instead. A fallback result is discarded and the script is retried. Every call now has a hard timeout and a backoff, because three weekly runs in a row were cancelled by a single connection that hung, and while that happened the published figure quietly went stale.

We left the 1.20 row where it is. A published record that only contains the good weeks is marketing, not a record.

#What this number does not promise

If you take one section from this article, take this one.

1. It is not a prediction of your test score. It is agreement on a fixed set of scripts, marked on our labels. Your test is a different prompt, on a different day, in front of a different marker, under time pressure.
2. The ground truth is ours. As above: zero examiner marked scripts in the set today. This is the biggest single limitation and no amount of decimal places fixes it.
3. The sample is small. Twenty seven scripts. One mislabelled script moves the average by roughly one twenty seventh of its own error, which is why we treat any single week as a signal rather than a verdict, and why the run history matters more than the latest row.
4. The scripts are clean specimens. They were written to sit at a band. Real candidate writing is messier and often sits at different bands on different criteria in the same answer. Mixed profiles are harder, and our set is under weighted for them.
5. It covers Writing only. There is no published accuracy figure for Speaking, and there will not be one until there is ground truth to measure it against. An unmeasured claim about Speaking would be exactly the thing this article exists to argue against.
6. The set is public. Anyone can read it, which is the point, and it also means we cannot rule out that these scripts turn up in some future model's training data. We rotate and grow the set for that reason, among others.
7. The overall figure averages over four criteria. They are not equally good, which is why the per criterion breakdown is published above rather than folded away.

#What has actually changed since May

The chart above is the honest version of the story: one very bad first row on a tiny sample, then a long flat stretch that moves slowly. Since the set was expanded in late May, every run has landed between 0.37 and 0.48 bands. There is no dramatic improvement curve to sell you, and we would rather show a flat line that is real than a steep one that is not.

The window also has gaps. The block above counts the scheduled Saturdays between the first run and the latest one and shows how many rows exist. The difference is runs that did not complete, mostly from timeouts and rate limits before the retry logic was added. Missing rows are visible for the same reason bad rows are.

One more thing the numbers show: Task Response is consistently the hardest criterion for the AI, and it is the one that carries the official under length penalty and the off topic rules. That is where our tuning work goes.

#How to check any of this yourself

None of the above is worth much if you have to take our word for it, so you do not have to.

• The quality report shows the latest run, the guardrails, and the full run history in the same detail.
• The public endpoint at /api/quality/calibration?history=1 returns the raw JSON behind everything on this page, including the per script deltas. No key, no signup.
• The consistency claim is testable in a minute: submit the same essay to the free diagnostic twice and compare the final results word for word.

If you find an AI IELTS scorer that publishes its own error rate against a fixed set, send it to us and we will link to it from this article. We have looked and we have not found one. That is a strange thing to be able to say about an industry that sells scoring.

#Why we publish it

I built this scoring engine. I am CELTA and DELTA qualified, I have taught IELTS for 8 years, and before that and alongside it I have spent more than 18 years in enterprise systems and data engineering. That second half is where this habit comes from: a number you cannot audit is a number you should not trust, and that applies to ours as much as to anybody else's.

There is also a simpler reason. If a tool tells you that you are a 7 and the real examiner says 6, that mistake costs you a test fee, a visa timeline, or a university place. You are entitled to know how often the tool is wrong before you plan around it. So we measure it every week, we publish it whether it flatters us or not, and we leave the bad rows in.

The next run writes its row this Saturday, and this page will show it.