RAG for Software Engineers: Beyond the Basics

The standard RAG tutorial walks you through splitting a PDF into chunks, embedding them, storing them in a vector database, and retrieving the top-k at query time. It works in a demo. It falls apart in production.

Here is what the gap looks like and how to address it.

Chunking strategy matters

Fixed-size chunking (split every 512 tokens) is the default and often the wrong choice. Semantic chunking — splitting at paragraph or section boundaries — produces chunks that are more coherent and retrieve better. For structured documents, structure-aware chunking (splitting by heading, table, or list item) outperforms both.

Metadata filtering

Vector similarity alone is not enough. Adding metadata filters (document source, date range, category) dramatically improves precision. The pattern: embed the query, filter by metadata first, then rank by similarity within the filtered set.

Hybrid search

Combining dense retrieval (vector similarity) with sparse retrieval (BM25 keyword search) consistently outperforms either alone. The intuition: dense retrieval handles semantic similarity, sparse retrieval handles exact keyword matches. Reciprocal Rank Fusion is the standard merging approach.

Re-ranking

The top-k from retrieval is not the same as the most relevant k for the query. A cross-encoder re-ranker — which scores each retrieved chunk against the full query — significantly improves answer quality at the cost of extra latency. Worth it for most production systems.

Evaluation

Without evaluation you are flying blind. The minimum viable eval: a test set of questions with ground-truth answers, retrieval recall (did the right chunk make the top-k?), and answer faithfulness (did the LLM answer from the retrieved context or hallucinate?). RAGAS is a reasonable starting point.

The pattern that works: start with fixed-size chunking and cosine similarity, establish evals, then improve chunking and retrieval strategies guided by what the evals surface. Optimising without evals is guesswork.