exam-nation

What it is

An exam tutor organised around misconceptions rather than topics. The catalogue behind it is a list of specific wrong mental models, but a student never sees that list. They see topics, and the misconception layer decides what happens inside a lesson.

The route through it is a twenty-question diagnostic taken without an account, a report that ranks the weak topics into one to start with and some to revisit, a deeper diagnostic on the topic they pick, and then a single-sitting lesson that climbs from basics to exam-level work.

Six courses run from that one codebase, routed by hostname: AQA Foundation and Higher, Edexcel Foundation and Higher, and JEE Advanced Maths and Physics. Three of them serve traffic today, the two AQA GCSE courses and JEE Advanced Maths; the other three are registered and built but not yet pointed at a live domain.

The problem

A student who reads a gradient the wrong way round does not have a gap. They have a rule, and it is wrong, and it works often enough that they trust it. Reteaching the topic leaves the rule intact. So the unit of work has to be the misconception, not the syllabus.

The engineering problem that follows is trust. An LLM asked to tutor freely will give the answer away, invent pedagogy, or walk a student into the wrong operation in order to correct it afterwards. None of that is acceptable in front of a child sitting an exam.

How it works

Teaching content is code, not prompts. The misconception catalogue, the Socratic scripts and the levelled lessons are TypeScript and markdown compiled into the build; the database holds student state only. The climb does not duplicate that content, it re-orders it: a topic plan pulls the same scripts at each level.

Adding an exam board is not the same as adding a syllabus, and the routing keeps the two apart. Each Edexcel course declares which AQA course it borrows its content from, so it carries its own branding, hostnames and sign-in mail while resolving to the same misconceptions and lessons. A course that declares nothing resolves to content of its own, which is what JEE Advanced does. One optional field is the difference between a rebrand and a new subject.

Inside a level the pattern is check first, teach only on a miss. A student who clears the check moves up without being taught at all, and the verdict at the end of a climb reports the highest level they cleared with no teaching fired.

The tutoring turn itself is deliberately constrained. The model is told it is running a state machine and does not decide the dialogue: the script decides, and the model matches and emits. Each turn ends with a hidden state block that the server parses and persists, so progression is tracked in the database rather than inferred from chat history. Every model call goes through one audited helper that caches the system prompt and records tokens, latency and estimated cost, so a prompt change has a visible price.

Retakes rotate five parallel forms of the diagnostic, so a second sitting is not the same twenty questions.

The evaluation harness exists because the test suite could not see the thing that matters. About 7,900 tests covered prompt interpolation, audit rows, stream parsing and the state machines, and none of them measured what the model returned. The short-text evaluator, whose labels decide which intervention a student is routed into, had no test at all and was mocked out at its only call site. Ground truth comes from the item bank’s own rubric patterns, transcribed by hand, because a content author writing that a response like this scores that and triggers this one is a human label that predates the model. Production rows are not ground truth. The score and misconception columns there are the evaluator’s own past output, so grading against them measures self-agreement, and a model that is consistently wrong scores full marks. Extracted rows are written unlabelled by default and wait for a human.

The number that matters is routing accuracy, score and misconception set both exact, because a right score with a wrong label still sends the student to the wrong lesson. False clears are counted separately from the rates and any increase is a regression whatever the tolerance says, because that is the mode nothing else catches: the misconception goes undetected, no intervention fires, and the product never notices. The first baseline froze 41 cases at 97.6% score and 95.1% routing, and its two failures were different kinds of bug. One was a real defect. A rubric pattern that deliberately assigns no misconception rendered into the prompt byte-identically to one that had simply not specified any, so the model read the silence as a gap and filled it with a plausible label, which routes a student into an intervention the evidence does not support. The other was a self-contradictory rubric rather than a bad model, and fixing it meant editing the item bank rather than the prompt.

What I’d change

Compiling the content in is the decision I would revisit. The intervention registry hard-codes an import per lesson file, which is why a missing file fails the build rather than failing a lesson in front of a student. It also means correcting one sentence of teaching copy needs a rebuild and a deploy. That is the right trade for correctness and the wrong one for iteration speed.

The eval baseline now reads 100% across 42 cases, and a perfect baseline is a one-sided gate. It will catch a regression and it can no longer show an improvement, so the honest reading is that the corpus is too small, not that the evaluator is finished. It covers one topic on one of the six courses. Growing it is slow on purpose: the draft generator will not invent a student response for a rubric pattern that only describes one, because a guessed response paired with an authoritative label produces a case that is confidently wrong.

The billing path is the least exercised code in the project. The paywall threshold constant is set to zero, which switches the paywall off outright, so the entitlement and checkout machinery is built and tested but is not currently in any student’s way.