Sketchbook / case study · see it on its sheet
RAG in Goa
Multilingual voice Q&A with hybrid retrieval inside a hard 200 ms deadline, and a reranker we measured and cut.
- Dates
- Aug 2026
- Event
- Hacker House Goa 2026 task
- Team
- team of 3: me, Merlyn Michelle, Shoban Ravichandran
- My part
- Retrieval pipeline rework, Sarvam Bulbul TTS, Colab GPU index, latency and architecture docs, deploy, mobile UI
- Stack
- Python · FastAPI · ONNX int8 bge-small · HNSW · BM25 · RRF · Sarvam Saaras + Bulbul · Fly.io
- Verified
- 76.85 ms retrieval P95, live215,608 chunks indexed200 ms hard retrieval deadline~160 ms reranker cost, so it shipped off
- Result
- no result recorded
The problem
Voice assistants tend to fail in one of two ways: they answer fluently and wrongly, or they answer slowly. Indic-language users usually get the worse version of both. The brief asked for grounded answers and strict latency reporting on retrieval.
How it works
- Speech in. Sarvam's Saaras speech-to-text, in translate mode, turns Hindi, Bengali, Tamil or Marathi speech into an English query.
- Input gate. Unsafe, injected or garbled transcripts are refused before retrieval.
- Hybrid retrieval against a memory-mapped index of 215,608 chunks: ONNX int8 bge-small embeddings into an HNSW index, BM25 alongside it, reciprocal rank fusion, then the top passages. A 200 ms wall-clock deadline skips stages and logs a degradation rather than miss it.
- Domain gate. If retrieval scores are too low, the system says it doesn't know.
- Grounded answer. The model returns schema-enforced JSON citing passage numbers; an output gate rejects invented citations.
- Speech out. The answer streams over SSE and is spoken back in the user's language through Sarvam's Bulbul.
The numbers, honestly
Retrieval on the live deployment: P50 47.5 ms, P95 76.85 ms, P99 88.8 ms, measured over the benchmark queries after warm-up. The claim covers retrieval only. A full live request takes about ~1.1 s at P50 because of the model (about ~617 ms) and the network; that is shown separately rather than hidden.
Decisions
- We cut the reranker. A MiniLM cross-encoder improved ranking but cost about ~160 ms of the budget. It ships switched off.
- A refusal beats a fluent miss. Three gates refuse instead of guessing.
My part
Team of three (Team McQuade). I reworked the retrieval pipeline, built the Colab GPU index notebook and the Bulbul voice output, wrote the latency and architecture docs, set up the deploy, and fixed the mobile UI and the evals. Merlyn Michelle built the FastAPI service, the guardrail gates and the SSE path. Shoban Ravichandran built the voice UI and ran the latency benchmark.
Retrieval against the deadline
Live · Fly.io Singapore, rerank off as shipped: embed → HNSW → BM25 → fusion → context expand, 400 queries after warmup.
- embed (ONNX int8)33.49 ms
- search: HNSW · BM25 · fusion · expand43.36 ms(total − embed, derived)
- total76.85 ms= 38.4% of the budget
LATENCY.md prints search P95 as 55.41 ms. Each percentile is a different query, so stage percentiles don’t add up to the total; the bar draws total − embed, which is how benchmark.py defines search.
Outside the budget
One live request at P50, end to end. The 200 ms claim covers retrieval only; speech and the LLM are someone else’s servers, shown in grey.
- retrieval 47.5 ms live P50, inside the budget
- LLM ~617 ms someone else's GPU, reported on its own
- network hop and the rest ~436 ms (derived)
- = live HTTP P50 ~1.1 s
- speech-to-text: not timed in LATENCY.md; the benchmark sends typed text, so it is in neither the table nor the ~1.1 s.
Gates · illustrative trace
The path the pipeline code gives each question. Not a recorded run: outcomes are the ones the README states.
Question · English · what type of mountain is Mount Fuji
- 1. Input gatepassnot unsafe, not an injection, readable
- 2. Translate to Englishnot neededalready English
- 3. Retrievalranruns under the deadline
- 4. Domain gatepassthe corpus has close passages
- 5. LLMranmay only speak from the retrieved passages
- 6. Output gatecheckschecks citations, then groundedness
- 7. Answerrancomes back in English
Deadline drill
Before each optional stage the retriever checks the clock and skips the stage, logging it, if too little time is left. Lower the deadline to see what a request with these P95 times (Live · Fly.io Singapore) would give up.
- 1. HNSW ef_search · needs 140 ms leftruns
Checked at 33.49 ms, before its cut-off at 60 ms.
- 2. BM25 leg · needs 140 ms leftcan’t tell
Checked between 33.49 and 76.85 ms, cut-off 60 ms. LATENCY.md doesn’t split dense from BM25 time, so the check could land on either side.
- 3. Rerank · needs 120 ms leftnot loaded
No reranker in the shipped config (RAGOA_RERANK=0), so there is no check.
Grey bar: when the check runs, as far as the recorded numbers pin it down. Dashed green: its cut-off (deadline − reserve). Vermilion: the deadline.
trace.degradations = [] · maybe "sparse_skipped"
ef_search 64 · finishes inside the deadline
team of 3 · Solai: retrieval-pipeline rework · Merlyn: API and guardrails · Shoban: voice UI and the benchmark
Live · Fly.io Singapore, P95: retrieval 76.85 ms of the 200 ms budget, 38.4%, under budget. Rerank off, as shipped.
Show the numbers as a table
| Machine | Pct | embed | search | total | total − embed (derived) | of 200 ms |
|---|---|---|---|---|---|---|
| Local · MacBook Air M2 | P50 | 2.21 | 4.99 | 7.33 | 5.12 | 3.7% · under |
| Local · MacBook Air M2 | P95 | 3.13 | 8.71 | 11.14 | 8.01 | 5.6% · under |
| Local · MacBook Air M2 | P99 | 3.94 | 10.94 | 13.65 | 9.71 | 6.8% · under |
| Live · Fly.io Singapore | P50 | 18.61 | 25.21 | 47.5 | 28.89 | 23.8% · under |
| Live · Fly.io Singapore | P95 | 33.49 | 55.41 | 76.85 | 43.36 | 38.4% · under |
| Live · Fly.io Singapore | P99 | 41.24 | 62.96 | 88.8 | 47.56 | 44.4% · under |
| total P50 | 165 ms |
|---|---|
| total P95 | 197 ms |
| total P99 | not recorded |
| rerank alone | ~160 ms |
| live with rerank | not recorded (live ran with rerank off) |
| speech-to-text | not timed |
|---|---|
| retrieval | 47.5 ms |
| LLM | ~617 ms |
| network hop and the rest (derived) | ~436 ms |
| live HTTP P50 | ~1.1 s |
| Check | needs left | cut-off | checked at | outcome |
|---|---|---|---|---|
| HNSW ef_search | 140 ms | 60 ms | 33.49 ms | runs |
| BM25 leg | 140 ms | 60 ms | 33.49–76.85 ms | can’t tell |
| Rerank | 120 ms | 80 ms | n/a | not loaded |
Reserves from ragoa/config.py: BM25 leg and ef 140 ms, rerank 120 ms.