Retrieval-Augmented Generation (RAG) has become the default architecture for connecting LLMs to private data. The concept is simple: retrieve relevant documents from a knowledge base, inject them into the LLM's context window, and let the model generate a grounded answer. The execution is where teams struggle — a RAG prototype that works on 10 documents often fails on 10,000. This guide covers the patterns that separate production-grade RAG from demo-grade RAG.
The Four-Stage Retrieval Pipeline
The retrieval pipeline has four stages: chunking (splitting source documents into searchable pieces), embedding (converting chunks into vector representations), search (finding the most relevant chunks for a given query), and generation (feeding those chunks to the LLM). Each stage has decisions that determine whether your system returns the right answer 95% of the time or 70% of the time. LangChain's RAG documentation covers the fundamentals well — this guide focuses on the production decisions that come after.
Chunking: The Decision Teams Get Wrong Most
Chunking strategy is the first decision and the one teams get wrong most often. The default approach — fixed-size chunks of 500-1000 tokens — works for uniform text (articles, documentation) but fails on structured data (tables, code, legal documents). Production RAG systems use semantic chunking: split on natural boundaries (paragraph breaks, section headers, function definitions) and keep chunk size between 200 and 800 tokens. Smaller chunks improve retrieval precision (less irrelevant text per match) but increase the total chunk count (more storage, slower search). Most teams converge on 400-600 token chunks with 50-100 token overlap between adjacent chunks to preserve context at boundaries.
Embedding Models: Dimensionality vs Cost
Embedding model choice affects both retrieval quality and cost. The dominant options in 2026 are OpenAI's text-embedding-3-large (3072 dimensions), Cohere's embed-v4 (1536 dimensions), and open-source models like BGE-large (1024 dimensions). Higher dimensionality captures more semantic nuance but costs more to store and search. For most enterprise use cases, 1024-1536 dimensions is the sweet spot — enough nuance for precise retrieval, manageable storage costs at scale.
Vector Databases: pgvector vs Managed Services
Vector database selection is where architecture decisions get concrete. PostgreSQL with pgvector is the default for teams already running Postgres — it's free, ACID-compliant, and handles up to ~10 million vectors before performance degrades. Managed services like Pinecone and Weaviate are better for larger-scale deployments (100M+ vectors) or when you need managed infrastructure. For most enterprise RAG systems processing 100K-1M documents, pgvector on Postgres is sufficient and avoids the operational complexity of a separate vector database.
Reranking: The Quality Multiplier
Reranking is the technique that separates good RAG from great RAG. After initial vector search retrieves the top-K candidates (typically K=20-50), a reranker re-scores those candidates using a more expensive cross-encoder model. The reranker's cross-attention mechanism captures query-document relevance that bi-encoder embeddings miss. Cohere's rerank API and open-source models like bge-reranker are the standard choices. Production RAG systems typically retrieve 20-50 candidates with vector search, then rerank to the top 3-5 before feeding to the LLM. This two-stage approach improves answer quality by 15-25% compared to single-stage retrieval.
Evaluation: Metrics That Separate Good from Great
Evaluation is the part most teams skip and regret later. A RAG system without automated evaluation is a black box — you can't answer "is retrieval quality improving or degrading?" without metrics. The two metrics that matter: context recall (did the retriever find the right documents?) and faithfulness (did the LLM's answer match the retrieved context?). Ragas and TruLens are the standard evaluation frameworks. Teams should build a golden test set of 50-100 query-answer pairs and run automated evaluation on every pipeline change.
