TL;DR: RAG pairs a retriever with a generator so an LLM can answer using external documents instead of relying purely on parametric memory. The "AI Development" hype around it tends to skip the actual engineering tradeoffs, so here's a breakdown of the architecture, the failure modes nobody talks about, and where the field is heading (GraphRAG, Self-RAG, CRAG, agentic retrieval).
The basic pipeline
At its core, RAG has three stages:
Indexing - documents are chunked, embedded, and stored in a vector index
Retrieval - a query is embedded and matched against the index to pull top-k relevant chunks
Generation - an LLM produces an answer conditioned on retrieved chunks + the query
This looks simple, and that’s exactly why many production RAG systems underperform, most of the real difficulty is in steps 1 and 2, not step 3.
Chunking is where most systems quietly fail
Fixed-size chunking (e.g., 512 tokens with overlap) is the default in many tutorials, but it often breaks semantic structure mid-sentence, mid-table, or mid-definition.
Alternatives worth knowing:
Structure-aware chunking - split along headings, paragraphs, and document structure first
Semantic chunking - use embedding similarity between sentences to decide split points so chunks remain topically coherent
Parent-child chunking - embed small chunks for retrieval precision but return the larger parent chunk for context (used in systems like LlamaIndex hierarchical retrievers)
Embeddings and indexing
Dense embeddings dominate, but pure vector search has known weaknesses — especially with exact keyword/entity matching (IDs, names, numbers).
That’s why most serious systems use hybrid retrieval:
On the indexing side, ANN structures like HNSW or IVF-PQ introduce speed vs recall tradeoffs. These often matter more in production than which vector database you pick.
Reranking
Initial retrieval optimizes for recall. A second-stage cross-encoder reranker then reorders the top ~50–100 candidates before only the top few go into the prompt.
This retrieve → rerank pattern often improves quality more than:
But it’s frequently skipped due to added latency.
Generation isn’t just “stuff the context window”
A few things that matter more than expected:
Lost-in-the-middle effect - models pay less attention to middle context in long inputs
Context ordering matters - placing the most relevant chunk near the end of the prompt often helps
Citation/grounding - mapping claims to chunks improves traceability even if it doesn’t fully eliminate hallucination
Why RAG still hallucinates
This is the part often glossed over.
RAG reduces hallucination from missing knowledge, but doesn’t eliminate it because:
The retriever can return irrelevant or contradictory chunks
The model can ignore retrieved context and fall back on parametric memory
Embedding similarity is not the same as logical relevance
A chunk can be semantically close to a query but still not contain the correct answer.
Evaluation
This is one of the most underrated parts of RAG systems.
You need to separate:
Retrieval quality (precision/recall of chunks)
Faithfulness (does answer follow retrieved context?)
Answer relevance/correctness (final output quality)
Frameworks like RAGAS help break this down. Without this separation, it’s impossible to know whether failures come from retrieval or generation and they require completely different fixes.
Where the field is heading
Self-RAG - models decide when retrieval is needed and critique outputs
CRAG (Corrective RAG) - evaluates retrieval quality and triggers fallback (e.g., web search)
GraphRAG - uses knowledge graphs instead of flat chunks for multi-hop reasoning
Agentic RAG - iterative retrieval loops (query → retrieve → refine → repeat), closer to ReAct-style systems