Skip to content

Sketchbook / case study · see it on its sheet

RAG in Goa

Multilingual voice Q&A with hybrid retrieval inside a hard 200 ms deadline, and a reranker we measured and cut.

Dates
Aug 2026
Event
Hacker House Goa 2026 task
Team
team of 3: me, Merlyn Michelle, Shoban Ravichandran
My part
Retrieval pipeline rework, Sarvam Bulbul TTS, Colab GPU index, latency and architecture docs, deploy, mobile UI
Stack
Python · FastAPI · ONNX int8 bge-small · HNSW · BM25 · RRF · Sarvam Saaras + Bulbul · Fly.io
Verified
76.85 ms retrieval P95, live215,608 chunks indexed200 ms hard retrieval deadline~160 ms reranker cost, so it shipped off
Result
no result recorded

The problem

Voice assistants tend to fail in one of two ways: they answer fluently and wrongly, or they answer slowly. Indic-language users usually get the worse version of both. The brief asked for grounded answers and strict latency reporting on retrieval.

How it works

  1. Speech in. Sarvam's Saaras speech-to-text, in translate mode, turns Hindi, Bengali, Tamil or Marathi speech into an English query.
  2. Input gate. Unsafe, injected or garbled transcripts are refused before retrieval.
  3. Hybrid retrieval against a memory-mapped index of 215,608 chunks: ONNX int8 bge-small embeddings into an HNSW index, BM25 alongside it, reciprocal rank fusion, then the top passages. A 200 ms wall-clock deadline skips stages and logs a degradation rather than miss it.
  4. Domain gate. If retrieval scores are too low, the system says it doesn't know.
  5. Grounded answer. The model returns schema-enforced JSON citing passage numbers; an output gate rejects invented citations.
  6. Speech out. The answer streams over SSE and is spoken back in the user's language through Sarvam's Bulbul.

The numbers, honestly

Retrieval on the live deployment: P50 47.5 ms, P95 76.85 ms, P99 88.8 ms, measured over the benchmark queries after warm-up. The claim covers retrieval only. A full live request takes about ~1.1 s at P50 because of the model (about ~617 ms) and the network; that is shown separately rather than hidden.

Decisions

  • We cut the reranker. A MiniLM cross-encoder improved ranking but cost about ~160 ms of the budget. It ships switched off.
  • A refusal beats a fluent miss. Three gates refuse instead of guessing.

My part

Team of three (Team McQuade). I reworked the retrieval pipeline, built the Colab GPU index notebook and the Bulbul voice output, wrote the latency and architecture docs, set up the deploy, and fixed the mobile UI and the evals. Merlyn Michelle built the FastAPI service, the guardrail gates and the SSE path. Shoban Ravichandran built the voice UI and ran the latency benchmark.

Retrieval budget race

Retrieval against the deadline

Machine
Percentile
Reranker

Live · Fly.io Singapore, rerank off as shipped: embed → HNSW → BM25 → fusion → context expand, 400 queries after warmup.

  • embed (ONNX int8)33.49 ms
  • search: HNSW · BM25 · fusion · expand43.36 ms(total − embed, derived)
  • total76.85 ms= 38.4% of the budget

LATENCY.md prints search P95 as 55.41 ms. Each percentile is a different query, so stage percentiles don’t add up to the total; the bar draws total − embed, which is how benchmark.py defines search.

Outside the budget

One live request at P50, end to end. The 200 ms claim covers retrieval only; speech and the LLM are someone else’s servers, shown in grey.

speech-to-textvoice only · not timed
  • retrieval 47.5 ms live P50, inside the budget
  • LLM ~617 ms someone else's GPU, reported on its own
  • network hop and the rest ~436 ms (derived)
  • = live HTTP P50 ~1.1 s
  • speech-to-text: not timed in LATENCY.md; the benchmark sends typed text, so it is in neither the table nor the ~1.1 s.

Gates · illustrative trace

The path the pipeline code gives each question. Not a recorded run: outcomes are the ones the README states.

Sample questions (README)
Traps

Question · English · what type of mountain is Mount Fuji

  1. 1. Input gatepassnot unsafe, not an injection, readable
  2. 2. Translate to Englishnot neededalready English
  3. 3. Retrievalranruns under the deadline
  4. 4. Domain gatepassthe corpus has close passages
  5. 5. LLMranmay only speak from the retrieved passages
  6. 6. Output gatecheckschecks citations, then groundedness
  7. 7. Answerrancomes back in English
Passes both gates before the LLM; the answer comes back in English.

Deadline drill

Before each optional stage the retriever checks the clock and skips the stage, logging it, if too little time is left. Lower the deadline to see what a request with these P95 times (Live · Fly.io Singapore) would give up.

200 ms
  1. 1. HNSW ef_search · needs 140 ms leftruns

    Checked at 33.49 ms, before its cut-off at 60 ms.

  2. 2. BM25 leg · needs 140 ms leftcan’t tell

    Checked between 33.49 and 76.85 ms, cut-off 60 ms. LATENCY.md doesn’t split dense from BM25 time, so the check could land on either side.

  3. 3. Rerank · needs 120 ms leftnot loaded

    No reranker in the shipped config (RAGOA_RERANK=0), so there is no check.

Grey bar: when the check runs, as far as the recorded numbers pin it down. Dashed green: its cut-off (deadline − reserve). Vermilion: the deadline.

trace.degradations = [] · maybe "sparse_skipped"

ef_search 64 · finishes inside the deadline

team of 3 · Solai: retrieval-pipeline rework · Merlyn: API and guardrails · Shoban: voice UI and the benchmark

Live · Fly.io Singapore, P95: retrieval 76.85 ms of the 200 ms budget, 38.4%, under budget. Rerank off, as shipped.

Simulation using recorded numbers from LATENCY.md in the RAG in Goa repo
Show the numbers as a table
Retrieval latency, rerank off (ms, LATENCY.md)
MachinePctembedsearchtotaltotal − embed (derived)of 200 ms
Local · MacBook Air M2P502.214.997.335.123.7% · under
Local · MacBook Air M2P953.138.7111.148.015.6% · under
Local · MacBook Air M2P993.9410.9413.659.716.8% · under
Live · Fly.io SingaporeP5018.6125.2147.528.8923.8% · under
Live · Fly.io SingaporeP9533.4955.4176.8543.3638.4% · under
Live · Fly.io SingaporeP9941.2462.9688.847.5644.4% · under
Rerank on (earlier local run)
total P50165 ms
total P95197 ms
total P99not recorded
rerank alone~160 ms
live with reranknot recorded (live ran with rerank off)
A live request at P50: only retrieval is inside the budget
speech-to-textnot timed
retrieval47.5 ms
LLM~617 ms
network hop and the rest (derived)~436 ms
live HTTP P50~1.1 s
Deadline checks (ragoa/index/retriever.py) at 200 ms, current row
Checkneeds leftcut-offchecked atoutcome
HNSW ef_search140 ms60 ms33.49 msruns
BM25 leg140 ms60 ms33.49–76.85 mscan’t tell
Rerank120 ms80 msn/anot loaded

Reserves from ragoa/config.py: BM25 leg and ef 140 ms, rerank 120 ms.