ImaniSearch Engineer · 28 · AtlantaSimulated persona · Owning a first production retrieval system
Our support-search system has a 180 ms p95 budget. Candidate recall is about 78%, but top-5 relevance is weak. A better embedding model adds roughly 35 ms and a migration; a cross-encoder reranker adds about 60 ms. Which should we invest in first?
Reasoning pattern: technical tradeoff under a latency constraint
Relay Answer
First separate candidate-generation failure from ranking failure on a representative labeled set. If the relevant document is already in the top 30–50 most of the time, a lightweight reranker is the more direct fix for weak top-5 quality; recover its latency through smaller candidate sets, caching, batching, or a cheaper first-stage scorer. If relevant documents are absent from the candidate set, reranking cannot help, so improve embeddings, chunking, metadata filters, or query rewriting first. Do not choose from average recall alone—measure recall at the exact rerank depth and by query class.
One view favors embeddings because retrieval quality sets a hard ceiling; another favors reranking because the stated symptom is poor ordering among already retrieved candidates.
Chunking, document freshness, metadata filtering, or label quality may dominate both model choices and deliver a larger gain without consuming the latency budget.
Full Battle
A full underlying Battle is intentionally not included in this content-only seed. Choose “Run with different AIs” to prepare the original question for Relay’s existing Battle flow.