Skip to content

Notes · 2026-10-19

We measured the reranker and cut it

A better ranking that ate most of a 200 ms budget. The number made the decision.

On RAG in Goa, retrieval had a hard 200 ms deadline. Hybrid search (dense HNSW plus BM25, fused with reciprocal rank fusion) fitted comfortably: 76.85 ms at P95 on the live deployment.

A MiniLM cross-encoder reranker improved the ranking, and it was tempting to ship it. So we measured it the same way as everything else. With the reranker on, total retrieval came to 165 ms at P50 and 197 ms at P95. The reranker alone cost about ~160 ms. That's most of the budget for one stage, with nothing left for a slow network day.

It ships switched off, behind a flag, so the decision can be revisited when the hardware or the model changes.

Two things I took from it:

  • Measure before you argue. The reranker debate ended the moment the numbers were on the page.
  • Say what the number covers. Our latency claim is for retrieval only. Speech-to-text and the LLM are reported separately, because claiming them would mean claiming someone else's latency.