The limits of naive RAG
Vanilla RAG (embedding → cosine similarity → top-k chunks → LLM) works well for straightforward questions and moderately sized knowledge bases. It fails on complex queries: negative questions ("which companies are NOT eligible?"), multi-faceted requests, or questions requiring comparison across distant documents. In practice, recall drops below 60% once the base exceeds 10,000 documents — the vector signal gets drowned in surface-level similarity noise.
Hybrid architecture: dense + sparse
Hybrid RAG combines two complementary retrieval strategies. Dense Retrieval (vector embeddings) excels at semantic similarity: it finds thematically related passages even with radically different vocabulary. BM25 (Sparse Retrieval) excels at exact lexical matching: essential for proper nouns, product codes, regulatory terms, and technical acronyms. Scores from both pipelines are merged via Reciprocal Rank Fusion (RRF), a simple formula — 1/(k + rank) — that is remarkably robust across heterogeneous score distributions.
Reranking: the differentiating layer
After merging 20–50 candidates, a cross-encoder (ms-marco-MiniLM-L6 or Cohere Rerank) re-ranks results by true relevance. Unlike bi-encoders that compute embeddings independently, the cross-encoder analyzes the (question, passage) pair jointly in a single forward pass — substantially higher precision. Added latency is 100–200ms, acceptable for most interactive use cases. The precision@3 gain is typically 15–25 percentage points.
Chunking and rigorous evaluation
Chunking is the most underrated aspect: chunks too small lose local context, chunks too large dilute retrieval precision. The "parent-child chunking" technique indexes small chunks (150 tokens) but feeds the LLM their parent chunk (600 tokens) to preserve context. For objective evaluation, build a golden set of 100–200 question/answer pairs and measure with RAGAS (Faithfulness, Answer Relevancy, Context Precision). Never rely on subjective impressions when comparing two retrieval architectures.