Can you trust an AI band score? A procedure for checking one

An AI band score is a hypothesis, not a verdict. Treated as a hypothesis it is one of the most useful things available to a candidate preparing alone: instant, repeatable, unembarrassed feedback on a fifth draft that no human being has time to read. Where that trade genuinely beats a tutor, and where it does not, is weighed in AI feedback against a private tutor. Treated as a verdict it is expensive, because the characteristic failure is not random noise but flattery. Automated graders hand out sevens more readily than they explain sixes, and the weaker ones score the shape of an answer rather than its content, which is why a candidate can occasionally feed in something barely relevant to the question and be told it is strong. The remedy is not to distrust every number. It is that the descriptors an examiner actually uses are published, so you can audit any score yourself in about ten minutes.

Why the numbers drift

A band for a Task 2 answer is not one judgement. It is four: Task Response, Coherence and Cohesion, Lexical Resource, and Grammatical Range and Accuracy, each matched against fixed descriptor wording, each weighted equally, then averaged. The Writing band you finally see combines two of those judgements — Task 1, whose first criterion is Task Achievement rather than Task Response, counting for one third, and Task 2 for two thirds. Speaking works the same way with Fluency and Coherence, Lexical Resource, Grammatical Range and Accuracy, and Pronunciation. How those skill bands then average into the overall, rounding included, is in how your band score is calculated. The descriptors themselves are written in language that requires judgement — "some flexibility and precision", "frequent error-free sentences" — so two careful readers can weigh the same script slightly differently even with the document open in front of them; a model producing one impressionistic number from a prompt has far more room to wander, and it wanders in the direction of whatever it was rewarded for sounding like.

So the question to ask of a grader is never "is this score right". It is "can this score be traced". A band that arrives with four sub-scores and a quoted sentence from your own text behind each one is auditable. A band that arrives as a number and a paragraph of encouragement is not evidence of anything, however confident it sounds.

The five-step audit

1. Re-read the four criteria for the claimed band and the band below it. If the grader says 7, read the band 7 row and the band 6 row side by side. The gap is rarely where candidates assume it is: band 6 already "attempts to use less common vocabulary but with some inaccuracy", so reaching for an unusual word is not by itself the difference. What band 7 adds is range enough "to allow some flexibility and precision", and less common items used "with some awareness of style and collocation". You are looking for which of those two descriptions your writing actually resembles — what band 7 actually requires, criterion by criterion walks this comparison against a full essay.
2. Find one sentence of your own as evidence for each criterion. Not an impression, a sentence. If nothing in your script demonstrates flexible, precise vocabulary, the Lexical Resource 7 is unproven whoever awarded it.
3. Check the score was averaged, not guessed. Ask for the four sub-scores and average them yourself. If they do not arrive at the reported band, the overall number came first and the criteria were decorated onto it afterwards. That grader is telling you a mood, not a measurement.
4. Test the grader with a deliberately degraded rewrite. Take the same answer and damage exactly one criterion. Delete every paragraph break and linking word, or replace every precise noun with "thing" and "stuff", or convert every complex sentence into two simple ones. Resubmit. A grader worth using drops that criterion by a band or more and leaves the other three roughly where they were. One that returns the same score, or takes 0.5 off everything uniformly, is reading length and formatting. Run this test once per tool, on your own work, before you trust anything it tells you.
5. Triangulate, and take the lower number. Two graders on one script give you a range. Plan against the bottom of it. The cost of assuming the pessimistic band is a few more weeks of practice; the cost of assuming the optimistic one is a test fee and a booking delay.

A worked example

Here is the audit applied to a Task 2 essay about whether governments should restrict private cars. The grader claimed 7.0 overall. Four criteria, one evidence sentence each, checked against the descriptors:

Now the arithmetic: 7 + 7 + 6 + 6 = 26, and 26 divided by 4 is 6.5. The claimed 7.0 was half a band optimistic, and the audit says precisely where: lexis and grammar, not ideas or structure. That is a study plan, which is the real output of this procedure. Notice also how tight the margins are. Lift Lexical Resource alone to 7 and the average is 6.75 — the same essay, one criterion's worth of work, and a visibly different position against the descriptors. One criterion is where the effort belongs, not everything at once.

What the audited number is for

Use it to aim practice, not to predict a result. In the largest published study of intensive instruction, moving half a band took something in the region of 200 to 240 hours of teaching, and the climb gets slower above band 6 — the arithmetic and its caveats are in how many hours preparation takes — so a scoring tool that tells you the truth about which criterion is holding you at 6.5 is worth considerably more than one that tells you a flattering number quickly. Audit the score, fix the criterion the audit exposes, and re-audit. That loop is reliable even when the graders inside it are not.