CEFR speech scoring lab

What it is

An automatic speaking-assessment scorer built so that the evaluation around it can be written up properly. The scorer is the means; the report is the product. It is not trying to be state of the art, and saying so is the whole point.

It runs on a licensed research corpus of second-language English speech: 876 long-turn responses from 438 speakers, each with a single consensus proficiency label on a half-point scale.

The problem

I taught English for years, so I know what an examiner is actually doing when they put a number on someone’s speech. That is a judgement with consequences, and the interesting question about automating it is not “how accurate” but “accurate for whom, how consistently, and where does it fail”.

Most speaking-scorer write-ups answer the first question and skip the rest. The gap I wanted to close is the evaluation science: agreement, error structure, subgroup behaviour, and the honest statement of what the data cannot tell you.

How it works

Whisper transcribes each response with word timestamps. Eleven fluency features are computed from those transcripts: speech rate, articulation rate, pause counts and durations, long pauses, mean length of run, filled pauses, and lexical diversity. Five prosody features come from the raw audio instead, so they do not inherit the transcriber’s errors. Pitch spread is reported in semitones, and intensity means are dropped entirely because they say more about the microphone than the speaker.

The lexical-diversity slot is the one I would point at. Type-token ratio correlated negatively with proficiency, which is the textbook length-sensitivity artefact rather than a finding about learners, so it was replaced with MTLD, which correlates positively and lifted the model.

Four estimators are compared against two deliberately stupid baselines, cross-validated with folds grouped by speaker, because the same speaker contributes two responses. Switching to random folds measured that hazard instead of assuming it: Ridge gained 0.015 quadratic weighted kappa and the random forest gained nothing at all, which is itself evidence that these features barely encode speaker identity beyond proficiency. Random forest wins at 0.558, against 0.500 for a baseline given nothing but word count and which part of the test it was. Agreement is 34.1% exact, 81.5% within half a band, 97.7% within one band.

The fourth estimator is a reported negative result, and the one I learned most from. A 1.5B instruct model, LoRA-fine-tuned to read the transcript alone and emit a band, reaches 0.446 on the same held-out rows and finishes below the word-count floor. Publishing that number was not enough. It had only ever been compared against a different pipeline, so it could not answer the first question a reader should ask, which is whether the fine-tuning did anything. Scoring the same model with the adapter removed puts its untrained floor at 0.144, so the training is worth 0.302 and the arm’s problem is its ceiling rather than its optimisation. Every arm needs its own baseline, not just the study’s.

The word-error-rate analysis is the part I am most pleased with. Transcription accuracy correlates with score error, which looks like an ASR problem, until you control for proficiency and the correlation vanishes. It was proficiency in disguise.

That null has since been pushed on twice. It is not an artefact of features that ignore what was said, because the content-only fine-tuned arm reproduces it. But breaking the error rate into substitutions, deletions and insertions rather than testing it whole does find something aggregate word error rate hides: deletion rate holds a partial correlation with signed error of -0.119 once proficiency is controlled, and it replicates on the second arm. The sign is the interesting part. It is negative, so a transcript with words missing is under-scored, which is the opposite of the raw alarm and exactly what a model leaning on word count would do. It accounts for about 1% of the variance, against a scale-compression term of nearly a full band, so it changes how strongly the null should be phrased rather than the conclusion drawn from it.

What I’d change

The corpus ships one consensus label per response, so inter-rater reliability cannot be computed at all and there is still no human ceiling to compare the model against. A machine ceiling can be computed, and it was worth having. Every arm here is deterministic, so a repeat pass agrees with itself perfectly by construction and that number measures nothing about reliability. Rewording the prompt benignly instead drops self-agreement to 0.512, and since a scorer cannot agree with a human better than it agrees with itself, the untrained arm’s 0.144 sits a quarter of the way to its own ceiling. That rules out the comfortable reading, which is that the floor was an artefact of an arbitrary prompt. The corpus carries no first-language metadata either, which means the most important fairness slice in the assessment literature is unavailable. Both are stated limits, not oversights, but they cap what the report can claim.

The model also compresses the scale, flattering the weakest candidates and penalising the strongest. That alone makes it unfit for any decision about a real candidate.