Portfolio evidence RAG

What it is

A question-answering tool over my own working directories. Ask it in plain English what I have built and it answers with citations down to path and line, or it refuses. Everything except the answer step runs on the laptop: the index, the embedder and the reranker are all local, and retrieval costs nothing per query.

The tool is the excuse. The point of the exercise was to be able to say, from my own measurements on my own corpus, what hybrid retrieval bought and what reranking bought.

The problem

Retrieval advice circulates as folklore. Hybrid search beats vector search, a reranker adds a few points, embeddings understand meaning in a way keyword matching cannot. Every one of those is testable and none of them is free, so the only useful version of the claim is one with a number and a corpus attached to it.

The second problem is that this corpus is not a dataset. It is thirteen live working directories that change underneath the index while the experiment runs. That is either a disqualifying flaw or the thing that makes the measurement honest, and which one it is depends entirely on whether each run records what moved.

How it works

Two retrieval arms run over the same chunks. One is BM25 through SQLite’s full-text index, the other a 384-dimension embedding of each chunk in a local vector store. Reciprocal rank fusion merges the two rankings, a cross-encoder reranks the top fifty, and twelve chunks go to the model. Every citation that comes back is then checked by Python rather than by another model: the strictest of the four rules is that a cited line must fall inside a chunk that was actually retrieved for that question. A model asked whether its own answer is grounded will say yes.

The measurements are a ladder where each row differs from the one above it by one variable. Recall@5 goes from 0.172 for vector search alone to 0.384 for the full pipeline, and retrieval latency from 43 to 357 milliseconds. Almost none of that gain is where I expected it.

Hybrid retrieval bought 0.121 of it. BM25 on its own reached 0.293, which is the entire hybrid gain, at a tenth of the latency and with an index that builds in one second. The neural embedder took 264 seconds to be beaten by a half-century-old term-frequency formula on every metric at once. What the vector arm does buy is depth: adding it back to BM25 costs 0.020 at recall@5 and gains 0.051 at recall@30. The fusion is not what puts the right document in the top five, it is what stops the top thirty from missing documents only the embedder can see.

Reranking bought a further 0.091, and it pays in inverse proportion to how good the first stage already was, which is the reverse of the usual advice. The cross-encoder added 0.111 to the vector baseline and 0.030 to BM25. A reranker is a repair mechanism, and there is less to repair when the first stage is good. More than half of its gain here is not the neural model at all: showing the cross-encoder the file path alongside the chunk text, one variable on its own, is worth 0.051, and that lands almost entirely on questions whose answer is a directory rather than a single file.

Because the corpus moves, every run records which documents changed since the index was built and whether any eval record cites them, rather than only that something changed. One run’s 377 changed documents included nothing the eval set names as evidence. Another moved a single document, and it was cited by exactly the record whose score shifted, so the corpus explains the delta and the retriever is not credited with it.

Two controls sit underneath all of it. The same vectors were copied into pgvector and the eval set scored twice: across 98 records and three rank fields each there is not one rank difference, so the vector store is not a source of variation and future movement in the dense arm is attributable to something else. And every model call is traced with the price pushed from this project’s own table rather than inferred from the model name, so the dashboard total and the repository’s own figure are one number instead of two with no way to say which is wrong.

What I’d change

Recall@5 of 0.388 means the right document is absent from the top five three times in five, and no amount of ladder detail replaces that as the headline. The rerank window says where the rest of the loss lives: only 0.551 of records have their expected document anywhere in the fifty-chunk window at all, so most of what is missing is a first-stage problem no reranker can reach. Improving the reranker further is the smaller half of the work left.

There is no confidence gate in front of the model, because no confidence signal worked. I tested three, the fused score, the cross-encoder’s top score and vector distance, and all three overlap almost completely between answerable and unanswerable questions. A threshold high enough to reject every verified negative also refuses 86% of the questions that do have an answer. So a low score is never used to claim something does not exist, which matters because one refusal made exactly that claim about work sitting in the index unretrieved.

The latency column deserves less trust than the recall column, and the two sit side by side looking equally solid. Early rows recorded timings without recording what else the machine was doing. Re-running one configuration unchanged returned 1,573 milliseconds against 357, entirely because the load average was 14.5 on eight cores. Recall held to four decimals across four builds on four days while latency moved 4.4 times in twenty minutes. The harness now records load per core beside every percentile, and the earlier rows carry a caveat instead, because they cannot be retrofitted.

The eval set has also been wrong twice, and both times the model was right. Retiring those two questions was correct and it cost this project the right to compare new rows against old ones, because the fingerprint changed. That is the standing tax on grading yourself against a set you wrote yourself.