The whole design in one line

The passages are the answer. The text above them is only a reading aid.

Retrieval is local and deterministic: a 22M-parameter sentence encoder maps text to a 384-number vector, and Postgres ranks that vector against every stored passage. What comes back is the documentation’s own words, never rewritten. A language model then reads the top passages and writes a few sentences from them, marking each with the number of the passage it copied — and every one of those numbers is a link. If the model is wrong, the passages underneath it disagree, and they are the part you can trust.

01 — The pipeline

One question, five steps

Ingestion runs locally and never ships. A crawler reads each manual’s own table of contents, the extractor takes one row per documentation heading so results can be cited precisely, the chunker divides each section into passages the encoder can actually hold, and every passage is embedded and written to Postgres with pgvector.

At query time retrieval happens first and entirely locally. The embedding model is imported lazily inside the search function, so rendering the landing page never loads onnxruntime or the 22 MB of weights. Only after the passages are chosen does anything leave the machine: the answer step forwards those same passages to a language model and streams back what it writes, and it is the only part of a query that touches the network.

02 — Two retrievers

A 22M-parameter encoder is bad at identifiers

all-MiniLM-L6-v2 was chosen because it fits in a serverless function and runs on CPU. It is the weakest part of this system, and its weakness is specific: an API name carries almost no distributional signal. httpx.AsyncClient and prose describing HTTP clients land close together, because the model has no reason to treat the identifier as a distinct thing from the words around it.

BM25 matches the literal term, so it recovers exactly those cases. The two rankings are combined with Reciprocal Rank Fusion — each retriever contributes 1 / (60 + rank), and the scores are added.

Positions are combined, not raw scores. A cosine distance and a BM25 score are not on a comparable scale; normalising them into one would invent a relationship that is not there, and would need re-tuning every time the corpus changed.

03 — A bug worth reading about

290 passages the encoder could never see

This project had a test that measured chunk size against the real tokenizer, and it passed. It also had a silent correctness bug. Both were true, because the test measured the one code path where the problem was not.

Code blocks were exempt from being subdivided — correctly, since cutting a sample mid-statement is the damage this pipeline exists to avoid. But “exempt” had no upper bound. FastAPI’s TYPE: / DEFAULT: reference sidebar produces single blocks of 33,829 characters, which tokenise to 8,266 word-pieces against a limit of 256. The first 3% was embedded; the rest was stored in the database and unreachable by any query, forever.

The symptom users would see: ask for a specific API’s parameters and get nothing. 290 passages, 0.85% of the index, all of them silently truncated.

The fix was not to pick a better constant. Two were tried and both were wrong in opposite directions: sized for the densest content they still truncated, and sized for prose they fragmented the FastAPI corpus from 2,803 passages into 11,702 by cutting good writing into unreadable fragments. Neither failure announced itself in a test.

So the chunker now measures real tokenisation and divides whatever overruns, at line boundaries so no code sample is cut mid-statement. Code splitting stays principled: a newline is a syntactic boundary in every language in the corpus.

Re-chunking all three manuals: 0 chunks over the limit, worst case 225 word-pieces.

04 — Choosing the threshold

The gap is the whole argument

A relevance threshold has to sit above the worst score an unrelated question can reach and below the best score a real answer can earn. Those two numbers turn out to be close together, and that is the useful finding — the distance between them is the only reason a threshold can work at all.

05 — Performance

Where the time actually goes

A query pays two very different costs: embedding the question on the CPU, and ranking against 36,252 rows in a database roughly 300ms away. They are measured separately because they have different causes and different fixes, and one end-to-end number hides which one is slow.

The cold start is not the model being slow. It is ~87 MB of ONNX runtime and 22 MB of weights being read from the deployment bundle before the first query can be answered. That cost is paid once per instance and amortised across every warm request after it, which is why the second query is roughly four times faster.

An empty result set costs about the same as a full one (495 ms against a 570 ms median). That is the right behaviour for a retrieval tool: returning nothing is a real answer, and it should not feel like an error.

06 — The answer

Quoting is the whole trick

Retrieval alone leaves the reader to do the synthesis. So the top six passages are forwarded to a language model with an instruction to copy from them and mark each copied sentence with the number of the passage it came from. Everything else follows from taking that instruction seriously: the passages go before the instruction, because a model continues from what sits directly above it; the temperature is 0.1, because a fluent paraphrase is exactly the failure this is built to avoid; and the model is named in the interface next to the text it wrote.

It runs on Groq’s free tier rather than in the function. A model small enough to fit the 250 MB deployment limit — SmolLM2-135M — was measured against these same passages and could not do the task: across four attempts it invented a FastAPI function called getFileRespondingFromRoutes, invented two database tables, and emitted no citations at all. The next size up, at 260 MB, does not fit in the budget the embedder leaves behind. That arithmetic, not a preference, is why the model is remote.

Two models are tried in order, because one provider is a single point of failure — a rate limit or an outage would otherwise take the feature down with no recourse. Each generator must fail before emitting its first token for the next one to take over, so a fallback can never splice two answers together. When every provider fails, an answer is composed from the passages directly: less fluent, and unable to synthesise, but unable to be wrong about the documentation, since every sentence it emits is verbatim from the database.

The answer is labelled as generated wherever it appears, and the passages stay on screen underneath it, numbered to match. A reader who does not trust the text still has the quotes.

07 — What is indexed

Sections, not pages

36,252 passages from the official Python, FastAPI and PostgreSQL documentation. Each documentation heading becomes its own row, so a result can be cited as “Bigger Applications → Dependencies” rather than somewhere on a long page.

Documents are keyed by (source, url, anchor) and passages by (document_id, ordinal), so ingestion upserts in place. Each passage carries a content hash and only changed content is re-embedded, which makes an incremental pass almost free — and a full pass takes roughly two hours, nearly all of it CPU spent embedding.

08 — What it costs

Almost all of it is index

The whole database is 270.6 MB. The passage text is a small fraction of that; the two indexes are the rest.

This matters because both providers are free tiers. Neon allows 0.5 GB and Vercel allows a 250 MB function bundle, currently 89 MB. Neither is comfortable, and both would need attention before real traffic — see below.

That last figure is the one worth remembering. Adding a second retriever made results measurably better and cost about a ninth of the vector index in storage — and nothing at all in the function bundle, because BM25 runs where the text already is.

09 — Limits and trade-offs

What breaks first