Student request triage

What it is

An automation pipeline for a student support inbox. Each incoming request is validated, matched to the student’s record through an API, and classified by an LLM into one of four categories with an urgency, a confidence and a one-line reason. It is then either answered with a reply draft in the student’s own language or placed in a queue for a person. Everything is kept in SQLite, and each run finishes with a summary and the cost of that run.

It runs against two local mock services, a student records API and an OpenAI-compatible model, so the whole thing works offline and without API keys. Both services misbehave on purpose, and handling that is the real subject of the project.

The problem

Calling a model is the easy part of an automation like this. The hard part is everything around the call, because both dependencies fail in ordinary ways. The student API rate-limits, returns a 500 and answers slowly on some calls. The model returns a 503, wraps its JSON in prose, sends the word “high” where an integer urgency should be, and invents a category that is not on the list.

None of those failures is dramatic, and most of them are silent. A classification with a made-up category still looks like data. A re-run after a crash can send a student a second reply, or pay for the same model call twice. So the requirements that matter are non-functional: re-runs must be idempotent, one failure must not stop the batch, model output must be checked against a schema, configuration must live outside the code, and the cost per 1,000 requests must be measured rather than estimated.

How it works

Each request moves through a fixed sequence of states, from received through validated, enriched, classified and drafted to completed, with three other exits: invalid, escalated and failed. Every step is committed to the database before the next one starts, so a crash leaves a state that can be read and resumed rather than a half-written one.

Idempotency is enforced by the database, not by a check in Python. The request id and the draft’s request id are primary keys, so running the same file again cannot create a second row or a second draft, even if the code had a bug. A hash of each record separates two cases that share an id. The same content is a harmless duplicate and is counted, while different content is a conflict that is never overwritten and goes to the human queue. On resume, the pipeline checks whether a classification is already stored in the row and reuses it, so a crash after the model call is never paid for twice.

The HTTP client is my own, on httpx, rather than the OpenAI SDK, because the SDK’s retry layer would sit between my code and the behaviour I most wanted to control. It retries 429s, 5xx responses, timeouts and connection errors, follows Retry-After up to a 30-second cap, and never retries a 400, 401 or 404. The timeout is ten seconds per attempt because some responses take three to four seconds, and a slow service is not a down one. That number turned out to protect the cost as well: with a one-second timeout, a single classification call triggered three real model executions and was billed three times.

Model output is never corrected. The first JSON object is extracted even when it is wrapped in prose or code fences, and then validated against a strict schema, so “high” is not turned into 3 and an invented category is not mapped to the nearest real one. Classification runs on two tiers with a single threshold of 0.75. Every request goes to the cheap model first, and only the ones that come back below the threshold, or invalid, are sent to the precise model. If that is still not good enough, the request is escalated with the reason, and the money already spent on it is recorded.

Reply drafts come from templates in the configuration file, in English, Spanish and Catalan, filled with the student’s real name, programme, enrolment status, progress and tutor. The mock model’s free-text answer ignores the student’s language, so it could not produce a reply in the right one. A language with no template is escalated rather than improvised.

On the sample file of 43 records, three are exact duplicates, which leaves 40 requests. 33 finish with a draft, 4 go to a person, 3 are invalid, and none fail. The student lookup is cached, including students who do not exist, so those 40 requests cost 15 API calls. A second run over the same file makes no model calls, writes no drafts and costs nothing.

The cost comparison is the result I would point to first. The two-tier design costs €0.102 per 1,000 requests. Always using the precise model costs €0.245 and produces exactly the same human queue of 4, so the two tiers cost 42% as much for the same outcome. Using only the cheap model costs €0.022, but its confidences run 0.15 lower, and at the same threshold the queue more than triples to 13. At scale that is about 225 more manual reviews per 1,000 requests in order to save about €0.08, and a person’s time is the more expensive resource.

There are 190 tests. During development each guarantee was checked against deliberate sabotage of the code, and a guarantee that could be broken without any test failing was treated as untested.

What I’d change

The cost figures come from a simulator with canned answers, so the ratio between the three strategies is the meaningful part and the absolute numbers are not. Real text lengths and real model prices would move them, and the first thing I would do with a real provider is re-run the same three-way measurement.

There is one gap in the per-request accounting. If the cheap model is paid for and the precise model then fails on the network, rather than returning invalid output, the first call’s cost does not reach that request’s cost column. The run totals are unaffected, because they come from the provider’s own meter, but the per-request figure is understated on that one path.

The timeout has no lower limit in the configuration, which is a problem given what the one-second measurement showed. Anyone who lowers it below the slow responses will quietly multiply the cost, and that should be rejected when the configuration loads rather than left to a comment in the code.

Processing is sequential on purpose, because the student API limits by number of calls and running requests in parallel would only produce more 429s. At higher volume the right design is a rate-limited queue, and that would be the next piece of real engineering.