r/Rag • • Jul 24 '25 Bravo

Showcase I made 60K+ building RAG projects in 3 months. Here's exactly how I did it (technical + business breakdown)

778 Upvotes

TL;DR: I was a burnt out startup founder with no capital left and pivoted to building RAG systems for enterprises. Made 60K+ in 3 months working with pharma companies and banks. Started at $3K-5K projects, quickly jumped to $15K when I realized companies will pay premium for production-ready solutions. Post covers both the business side (how I got clients, pricing) and technical implementation.

Hey guys, I'm Raj, 3 months ago I had burned through most of my capital working on my startup, so to make ends meet I switched to building RAG systems and discovered a goldmine I've now worked with 6+ companies across healthcare, finance, and legal - from pharmaceutical companies to Singapore banks.

This post covers both the business side (how I got clients, pricing) and technical implementation (handling 50K+ documents, chunking strategies, why open source models, particularly Qwen worked better than I expected). Hope it helps others looking to build in this space.

I was burning through capital on my startup and needed to make ends meet fast. RAG felt like a perfect intersection of high demand and technical complexity that most agencies couldn't handle properly. The key insight: companies have massive document repositories but terrible ways to access that knowledge.

How I Actually Got Clients (The Business Side)

Personal Network First: My first 3 clients came through personal connections and referrals. This is crucial - your network likely has companies struggling with document search and knowledge management. Don't underestimate warm introductions.

Upwork Reality Check: Got 2 clients through Upwork, but it's incredibly crowded now. Every proposal needs to be hyper-specific to the client's exact problem. Generic RAG pitches get ignored.

Pricing Evolution:

  • Started at $3K-$5K for basic implementations

  • Jumped to $15K for a complex pharmaceutical project (they said yes immediately)

  • Realized I was underpricing - companies will pay premium for production-ready RAG systems

The Magic Question: Instead of "Do you need RAG?", I asked "How much time does your team spend searching through documents daily?" This always got conversations started.

Critical Mindset Shift: Instead of jumping straight to selling, I spent time understanding their core problem. Dig deep, think like an engineer, and be genuinely interested in solving their specific problem. Most clients have unique workflows and pain points that generic RAG solutions won't address. Try to have this mindset, be an engineer before a businessman, sort of how it worked out for me.

Technical Implementation: Handling 50K+ Documents

This is sort of my interesting part. Most RAG tutorials handle toy datasets. Real enterprise implementations are completely different beasts.

The Ground Reality of 50K+ Documents

Before diving into technical details, let me paint the picture of what 50K documents actually means. We're talking about pharmaceutical companies with decades of research papers, regulatory filings, clinical trial data, and internal reports. A single PDF might be 200+ pages. Some documents reference dozens of other documents.

The challenges are insane: document formats vary wildly (PDFs, Word docs, scanned images, spreadsheets), content quality is inconsistent (some documents have perfect structure, others are just walls of text), cross-references create complex dependency networks, and most importantly - retrieval accuracy directly impacts business decisions worth millions.

When a pharmaceutical researcher asks "What are the side effects of combining Drug A with Drug B in patients over 65?", you can't afford to miss critical information buried in document #47,832. The system needs to be bulletproof reliable, not just "works most of the time."

Quick disclaimer: So this was my approach, not final and something we still change each time from the learning, so take this with some grain of salt.

Document Processing & Chunking Strategy

So first step was deciding on the chunking, this is how I got started off.

For the pharmaceutical client (50K+ research papers and regulatory documents):

Hierarchical Chunking Approach:

  • Level 1: Document-level metadata (paper title, authors, publication date, document type)

  • Level 2: Section-level chunks (Abstract, Methods, Results, Discussion)

  • Level 3: Paragraph-level chunks (200-400 tokens with 50 token overlap)

  • Level 4: Sentence-level for precise retrieval

Metadata Schema That Actually Worked: Each document chunk included essential metadata fields like document type (research paper, regulatory document, clinical trial), section type (abstract, methods, results), chunk hierarchy level, parent-child relationships for hierarchical retrieval, extracted domain-specific keywords, pre-computed relevance scores, and regulatory categories (FDA, EMA, ICH guidelines). This metadata structure was crucial for the hybrid retrieval system that combined semantic search with rule-based filtering.

Why Qwen Worked Better Than Expected

Initially I was planning to use GPT-4o for everything, but Qwen QWQ-32B ended up delivering surprisingly good results for domain-specific tasks. Plus, most companies actually preferred open source models for cost and compliance reasons.

  • Cost: 85% cheaper than GPT-4o for high-volume processing

  • Data Sovereignty: Critical for pharmaceutical and banking clients

  • Fine-tuning: Could train on domain-specific terminology

  • Latency: Self-hosted meant consistent response times

Qwen handled medical terminology and pharmaceutical jargon much better after fine-tuning on domain-specific documents. GPT-4o would sometimes hallucinate drug interactions that didn't exist.

Let me share two quick examples of how this played out in practice:

Pharmaceutical Company: Built a regulatory compliance assistant that ingested 50K+ research papers and FDA guidelines. The system automated compliance checking and generated draft responses to regulatory queries. Result was 90% faster regulatory response times. The technical challenge here was building a graph-based retrieval layer on top of vector search to maintain complex document relationships and cross-references.

Singapore Bank: This was the $15K project - processing CSV files with financial data, charts, and graphs for M&A due diligence. Had to combine traditional RAG with computer vision to extract data from financial charts. Built custom parsing pipelines for different data formats. Ended up reducing their due diligence process by 75%.

Key Lessons for Scaling RAG Systems

  1. Metadata is Everything: Spend 40% of development time on metadata design. Poor metadata = poor retrieval no matter how good your embeddings are.

  2. Hybrid Retrieval Works: Pure semantic search fails for enterprise use cases. You need re-rankers, high-level document summaries, proper tagging systems, and keyword/rule-based retrieval all working together.

  3. Domain-Specific Fine-tuning: Worth the investment for clients with specialized vocabulary. Medical, legal, and financial terminology needs custom training.

  4. Production Infrastructure: Clients pay premium for reliability. Proper monitoring, fallback systems, and uptime guarantees are non-negotiable.

The demand for production-ready RAG systems is honestly insane right now. Every company with substantial document repositories needs this, but most don't know how to build it properly.

If you're building in this space or considering it, happy to share more specific technical details. Also open to partnering with other developers who want to tackle larger enterprise implementations.

For companies lurking here: If you're dealing with document search hell or need to build knowledge systems, let's talk. The ROI on properly implemented RAG is typically 10x+ within 6 months.


r/Rag • • Jun 17 '25 Galaxy Brain

Tools & Resources A free goldmine of tutorials for the components you need to create production-level agents

384 Upvotes

I’ve just launched a free resource with 25 detailed tutorials for building comprehensive production-level AI agents, as part of my Gen AI educational initiative.

The tutorials cover all the key components you need to create agents that are ready for real-world deployment. I plan to keep adding more tutorials over time and will make sure the content stays up to date.

The response so far has been incredible! (the repo got nearly 500 stars in just 8 hours from launch) This is part of my broader effort to create high-quality open source educational material. I already have over 100 code tutorials on GitHub with nearly 40,000 stars.

I hope you find it useful. The tutorials are available here: https://github.com/NirDiamant/agents-towards-production

The content is organized into these categories:

  1. Orchestration

  2. Tool integration

  3. Observability

  4. Deployment

  5. Memory

  6. UI & Frontend

  7. Agent Frameworks

  8. Model Customization

  9. Multi-agent Coordination

  10. Security

  11. Evaluation


r/Rag • • Sep 17 '26 Diamonds are Forever All Seeing Upvote This Bravo

Discussion 1.5 years of RAG in fintech, what actually worked after screwups

376 Upvotes

For the last ~1.5 years, we have been testing RAG in our fintech systems.

Initially there were a lot of failures. Some were harmless bad answers, some were actual screwups where a confident wrong answer was much worse than saying "I don't know."

We were retrieving from a mix of annual reports, news, internal trading strategies, risk documents, rules, research and some codebases. Tried different chunking, indexing, reranking and retrieval methods.

A few things worked better than I expected.

1. Parsing matters a lot, especially for PDFs

If the PDF parser messes up tables, headers, footnotes or multi-column text, bad data goes into the index.

Then better embeddings or reranking will not save you.

For example, if a table says:

Revenue | FY25 | FY24
Segment A | 420 | 380

and the parser separates the numbers from the row labels, retrieval may still find the "right" chunk but the LLM is reasoning over broken data.

So for us, parsing quality became more important than obsessing over embedding models.

2. Do not allow LLMs do things Python or DB can do exactly

Example:

"What is current exposure of Strategy X and how much headroom is left before the 2% limit?"

We do:

DB → current exposure = 1.37%
Rules DB → limit = 2%
Python → remaining = 0.63%
LLM → explain

The LLM doesn't need to calculate this.

Same with internal Q&A. If the document says:

"Positions must be reduced when margin utilisation exceeds 80%."

Sometimes it is better to return almost the same sentence instead of letting the model rewrite it beautifully.

Ugly but grounded is often better than polished but hallucinated.

We tested this idea in our hallucination experiments too. In one setup, the LLM identified evidence, but deterministic code handled the final factual replacement instead of allowing free rewriting.

That reduced a meaningful number of hallucination/grounding errors in our controlled datasets.

basically, LLM can decide where to look but we restrict what to write

3. Vector search gets weak when the answer needs multiple linked facts

For example:

"Why was Strategy A disabled for this instrument?"

The answer may need:

Strategy
→ eligibility rule
→ liquidity threshold
→ exchange rule
→ current instrument state

Those can sit in different documents.

Top-5 similar chunks may not retrieve the full chain.

For these cases, knowledge graphs or explicit relationships helped. Not for every query, but definitely for questions like:

"Which strategies are affected if this risk rule changes?"

4. Versioning the knowledge base is critical

This caused some stupid failures.

Risk Policy v3 → max exposure 1%
Risk Policy v4 → max exposure 1.5%

If both are in the vector DB, your RAG can very accurately retrieve the wrong policy.

So now we track things like version, effective date, active/superseded status, source, ingestion time, etc.

Deleting an old PDF from a folder is not knowledge-base management.

5. For code retrieval, AST worked better than plain text chunking

If I ask:

"Where is the stop-loss logic for Strategy X?"

the answer may involve functions across multiple files.

AST gives structure around functions, classes, imports, calls and definitions.

That was much more useful than treating source code like normal prose and embedding arbitrary chunks.

6. Context, memory, RAG and DB should not be mixed into one giant bucket

Example:

"Considering current BTC exposure and yesterday's risk committee decision, can Strategy X open another position?"

That may require:

Current exposure → DB
Risk decision → RAG
Strategy rule → RAG
Previous discussion → memory/context
Calculation → Python
Final answer → LLM

Once we separated these things, debugging became much easier.

After 1.5 years, I don't really think of RAG as vector search

More like:

query → figure out what information is needed → route to RAG/DB/memory/code → retrieve evidence → deterministic logic where possible → LLM for reasoning/explanation

And when something fails, I now ask:

Was the source wrong? Parsing wrong? Retrieval wrong? Or generation wrong?

Before asking which embedding model to try next :)

I am curious to hear from you, what others running RAG in production have seen. PS: Refer this guide for more details: https://github.com/FinanceFlash/unvibecode/blob/main/skills/unvibecode-rag-review/references/RAG_PRACTICAL_GUIDE.md


r/Rag • • Oct 20 '25

Tools & Resources Production RAG: what we learned from processing 5M+ documents

375 Upvotes

I've spent the past 8 months the trenches, I want to share what actually worked vs. wasted our time. We built RAG for Usul AI (9M pages) and an unnamed legal AI enterprise (4M pages).

Langchain + Llamaindex

We started out with youtube tutorials. First Langchain -> Llamaindex. Got to a working prototype in a couple of days and were optimistic with the progress. We run tests on subset of the data (100 documents) and the results looked great. We spend the next few days running the pipeline on the production dataset and got everything working in a week — incredible.

Except it wasn’t, the results were subpar and only the end users could tell. We spent the following few months rewriting pieces of the system, one at a time, until the performance was at the level we wanted. Here are things we did ranked by ROI.

What moved the needle

  1. Query Generation: not all context can be captured by the user’s last query. We had an LLM review the thread and generate a number of semantic + keyword queries. We processed all of those queries in parallel, and passed them to a reranker. This made us cover a larger surface area and not be dependent on a computed score for hybrid search.

  2. Reranking: the highest value 5 lines of code you’ll add. The chunk ranking shifted a lot. More than you’d expect. Reranking can many times make up for a bad setup if you pass in enough chunks. We found the ideal reranker set-up to be 50 chunk input -> 15 output.

  3. Chunking Strategy: this takes a lot of effort, you’ll probably be spending most of your time on it. We built a custom flow for both enterprises, make sure to understand the data, review the chunks, and check that a) chunks are not getting cut mid-word or sentence b) ~each chunk is a logical unit and captures information on its own

  4. Metadata to LLM: we started by passing the chunk text to the LLM, we ran an experiment and found that injecting relevant metadata as well (title, author, etc.) improves context and answers by a lot.

  5. Query routing: many users asked questions that can’t be answered by RAG (e.g. summarize the article, who wrote this). We created a small router that detects these questions and answers them using an API call + LLM instead of the full-blown RAG set-ups.

Our stack

  • Vector database: Azure → Pinecone → Turbopuffer (cheap, supports keyword search natively)

  • Document Extraction: Custom

  • Chunking: Unstructured.io by default, custom for enterprises (heard that Chonkie is good)

  • Embedding: text-embedding-3-large, haven’t tested others

  • Reranker: None → Cohere 3.5 → Zerank (less known but actually good)

  • LLM: GPT-4.1 → GPT-5 → GPT-4.1 (covered by Azure credits)

Going Open-source

We put all our learning into an open-source project: https://github.com/agentset-ai/agentset under an MIT license. Happy to share any learnings.


r/Rag • • Feb 11 '26

Showcase EpsteinFiles-RAG: Building a RAG Pipeline on 2M+ Pages

323 Upvotes

I love playing around with RAG and AI, optimizing every layer to squeeze out better performance. Last night I thought: why not tackle something massive?

Took the Epstein Files dataset from Hugging Face (teyler/epstein-files-20k) – 2 million+ pages of trending news and documents. The cleaning, chunking, and optimization challenges are exactly what excites me.

What I built:

- Full RAG pipeline with optimized data processing

- Processed 2M+ pages (cleaning, chunking, vectorization)

- Semantic search & Q&A over massive dataset

- Constantly tweaking for better retrieval & performance

- Python, MIT Licensed, open source

Why I built this:

It’s trending, real-world data at scale, the perfect playground.

When you operate at scale, every optimization matters. This project lets me experiment with RAG architectures, data pipelines, and AI performance tuning on real-world workloads.

Repo: https://github.com/AnkitNayak-eth/EpsteinFiles-RAG

Open to ideas, optimizations, and technical discussions!


r/Rag • • Aug 09 '25

How to index 40k documents

295 Upvotes

Hello,

I'm new to the RAG community and I need help with something I think is quite complex.

I have 40,000 PDF documents, each averaging around 100 pages.
These pages can contain both text and images.

I need a way to process every file in this corpus of 40k documents:

  1. Extract the text, split it into chunks, generate embeddings, and store them in a vector database so they can be queried. For each chunk, I need the page number and ideally the bounding box of where the text appears on the page so I can highlight it later.

  2. I also need to extract images, pass them to an LLM to describe them, embed the descriptions, and store those in the vector database as well.

What would be the best and most cost-effective stack to achieve this?
I’ve seen LlamaParse from LlamaCloud… but it’s soooo expensive (10 credits per page, 40k documents × 100 pages = 4M pages = 40M credits... not viable).

Thanks so much for your help! ❤️


r/Rag • • Nov 17 '25

Tools & Resources 20,000 Epstein Files in a single text file available to download (~100 MB)

283 Upvotes

Usage

This dataset is provided for research and exploratory analysis in controlled settings, with a primary focus on:

  • Evaluating information retrieval and Retrieval-Augmented Generation (RAG) systems.

  • Developing and testing search, clustering, and summarization methods on a real world corpus.

  • Examining the structure and content of the public record related to the Epstein estate documents.

It is not intended for:

  • Finetuning a language model.

  • Harassment, doxxing, or targeted attacks on any individual or group.

  • Attempts to deanonymize redacted information or circumvent existing redactions.

  • Making or amplifying unverified allegations as factual claims.

I've processed all the text and image files (~25,000 document pages/emails) within individual folders released last friday into a two column text file. I used Googles tesseract OCR library to convert jpg to text.

You can download it here: https://huggingface.co/datasets/tensonaut/EPSTEIN_FILES_20K

For each document, I've included the full path to the original google drive folder from House oversight committee so you can link and verify contents.


r/Rag • • Sep 10 '25

My open-source project on different RAG techniques just hit 20K stars on GitHub

281 Upvotes

Here's what's inside:

  • 35 detailed tutorials on different RAG techniques

  • Tutorials organized by category

  • Clear, high-quality explanations with diagrams and step-by-step code implementations

  • Many tutorials paired with matching blog posts for deeper insights

  • I'll keep sharing updates about these tutorials here

A huge thank you to all contributors who made this possible!

Link to the repo


r/Rag • • Jul 05 '25

What’s the best RAG tech stack these days? From chunking and embedding to retrieval and reranking

264 Upvotes

I’m trying to get a solid overview of the current best-in-class tech stacks for building a Retrieval-Augmented Generation (RAG) pipeline. I’d like to understand what you'd recommend at each step of the pipeline:

  • Chunking: What are the best practices or tools for splitting data into chunks?

  • Embedding: Which embedding models are most effective right now?

  • Retrieval: What’s the best way to store and retrieve embeddings (vector databases, etc.)?

  • Reranking: Are there any great reranking models or frameworks people are using?

  • End-to-end orchestration: Any frameworks that tie all of this together nicely?

I’d love to hear what the current state-of-the-art options are across the stack, plus any personal recommendations or lessons learned. Thanks!


r/Rag • • Dec 28 '25

Discussion I Killed RAG Hallucinations Almost Completely

261 Upvotes

Hey everyone, I have been building a no code platform where users can come and building RAG agent just by drag and drop Docs, manuals or PDF.

After interacting with a lot of people on reddit, I found out that there mainly 2 problems everyone was complaining about one was about parsing complex pdf's and hallucinations.

After months of testing, I finally got hallucinations down to almost none on real user data (internal docs, PDFs with tables, product manuals)

  1. Parsing matters: Suggested by fellow redditor and upon doing my own research using Docling (IBM’s open-source parser) → outputs perfect Markdown with intact tables, headers, lists. No more broken table context.

  2. Hybrid search (semantic + keyword): Dense (e5-base-v2 → RaBitQ quantized in Milvus) + sparse BM25.
    Never misses exact terms like product codes, dates, SKUs, names.

  3. Aggressive reranking: Pull top-50 from Milvus - run bge-reranker-v2-m3 to keep only top-5.
    This alone cut wrong-context answers by ~60%. Milvus is best DB I have found ( there are also other great too )

  4. Strict system prompt + RAGAS

If you’re building anything with document, try adding Docling + hybrid + strong reranker—you’ll see the jump immediately. Happy to share prompt/configs

Thanks


r/Rag • • Aug 08 '25

Discussion GPT-5 is a BIG win for RAG

256 Upvotes

GPT-5 is out and that's AMAZING news for RAG.

Every time a new model comes out I see people saying that it's the death of RAG because of its high context window. This time, it's also because of its accuracy when processing so many tokens.

There's a lot of points that require clarification in such claims. One could argue that high context windows might mean the death of fancy chunking strategies, but the death of RAG itself? Simply impossible. In fact, higher context windows is a BIG win for RAG.

LLMs are stateless and limited with information that was used during its training. RAG, or "Retrieval Augmented Generation" is the process of augmenting the knowledge of the LLM with information that wasn't available during its training (either because it is private data or because it didn't exist at the time)

Put simply, any time you enrich an LLM’s prompt with fresh or external data, you are doing RAG, whether that data comes from a vector database, a SQL query, a web search, or a real-time API call.

High context windows don’t eliminate this need, they simply reduce the engineering overhead of deciding how much and which parts of the retrieved data to pass in. Instead of breaking a document into dozens of carefully sized chunks to fit within a small prompt budget, you can now provide larger, more coherent passages.

This means less risk of losing context between chunks, fewer retrieval calls, and simpler orchestration logic.

However, a large context window is not infinite, and it still comes with cost, both in terms of token pricing and latency.

According to Anthropic, a PDF page typically consumes 1500 to 3000 tokens. This means that 256k tokens may easily be consumed by only 83 pages. How long is your insurance policy? Mine is about 40 pages. One document.

Blindly dumping hundreds of thousands of tokens into the prompt is inefficient and can even hurt output quality if you're feeding irrelevant data from one document instead of multiple passages from different documents.

But most importantly, no one wants to pay for 256 thousand or a million tokens every time they make a request. It doesn't scale. And that's not limited to RAG. Applied AI Engineers that are doing serious work and building real and scalable AI applications are constantly looking forward to strategies that minimize the number of tokens they have to pay with each request.

That's exactly the reason why Redis is releasing LangCache, a managed service for semantic caching. By allowing agents to retrieve responses from a semantic cache, they can also avoid hitting the LLM for request that are similar to those made in the past. Why pay twice for something you've already paid for?

Intelligent retrieval, deciding what to fetch and how to structure it, and most importantly, what to feed the LLM remains critical. So while high context windows may indeed put an end to overly complex chunking heuristics, they make RAG more powerful, not obsolete.


r/Rag • • Dec 02 '25 Bravo

Discussion Apple looks set to "kill" classic RAG with its new CLaRa framework

254 Upvotes

We’re all used to document workflows being a complex puzzle: chopping text into chunks, running them through embedding models, stuffing them into a vector DB, and only then retrieving text to feed the neural net. But researchers are proposing a game-changing approach

The core of CLaRa is that it makes the whole process End-to-End. No more disjointed text chunks at the input the model itself compresses documents (up to 128x compression) into hidden latent vectors. The coolest part? These vectors are fed directly into the LLM to generate answers. No need to decode them back into text; the model understands the meaning directly from the numbers

The result is a true All-in-One tool. It’s both a 7B parameter LLM and a smart retriever in one package. You no longer need paid OpenAI APIs or separate embedding models. It fits easily on consumer GPUs or Macs, offers virtually infinite context thanks to extreme compression, and ensures total privacy since it runs locally

If you have a project where you need to feed the model tons of docs or code, and you’re tired of endlessly tweaking chunking settings, this is definitely worth a shot. The code is on GitHub, weights on HuggingFace, and the paper on Arxiv.

I wonder how it stacks up against the usual Llama-3 + Qdrant combo has anyone tested it yet?

Model: https://huggingface.co/apple/CLaRa-7B-Instruct

Github: https://github.com/apple/ml-clara

Paper: https://arxiv.org/abs/2511.18659


r/Rag • • Aug 27 '25

Tutorial From zero to RAG engineer: 1200 hours of lessons so you don't repeat my mistakes

Thumbnail
bytevagabond.com
253 Upvotes

After building enterprise RAG from scratch, sharing what I learned the hard way. Some techniques I expected to work didn't, others I dismissed turned out crucial. Covers late chunking, hierarchical search, why reranking disappointed me, and the gap between academic papers and messy production data. Still figuring things out, but these patterns seemed to matter most.


r/Rag • • Apr 22 '25 Bless Up

My document retrieval system outperforms traditional RAG by 70% in benchmarks - would love feedback from the community

246 Upvotes

Hey folks,

In the last few years, I've been struggling to develop AI tools for case law and business documents. The core problem has always been the same: extracting the right information from complex documents. People were asking to combine all the law books and retrieve the EXACT information to build their case.

Think of my tool as a librarian who knows where your document is, takes it off the shelf, reads it, and finds the answer you need. 

Vector searches were giving me similar but not relevant content. I'd get paragraphs about apples when I asked about fruit sales in Q2. Chunking documents destroyed context. Fine-tuning was a nightmare. You probably know the drill if you've worked with RAG systems.

After a while, I realized the fundamental approach was flawed.

Vector similarity ≠ relevance. So I completely rethought how document retrieval should work.

The result is a system that:

  • Processes entire documents without chunking (preserves context)

  • Understands the intent behind queries, not just keyword matching

  • Has two modes: cheaper and faster & expensive but more accurate

  • Works with any document format (PDF, DOCX, JSON, etc.)

What makes it different is how it maps relationships between concepts in documents rather than just measuring vector distances. It can tell you exactly where in a 100-page report the Q2 Western region finances are discussed, even if the query wording doesn't match the document text. But imagine you have 10k long PDFs, and I can tell you exactly the paragraph you are asking about, and my system scales and works.

The numbers: 

  • In our tests using 800 PDF files with 80 queries (Kaggle PDF dataset), we're seeing:

  •  94% correct document retrieval in Accurate mode (vs ~80% for traditional RAG)— so 70% fewer mistakes than popular solutions on the market.

  •  92% precision on finding the exact relevant paragraphs

  •  83% accuracy even in our faster retrieval mode

I've been using it internally for our own applications, but I'm curious if others would find it useful. I'm happy to answer questions about the approach or implementation, and I'd genuinely love feedback on what's missing or what would make this more valuable to you.

I don’t want to spam here so I didn't add the link, but if you're truly interested, I’m happy to chat


r/Rag • • Oct 03 '25

Showcase First RAG that works: Hybrid Search, Qdrant, Voyage AI, Reranking, Temporal, Splade. What is next?

233 Upvotes

As a novice, I recently finished building my first production RAG (Retrieval-Augmented Generation) system, and I wanted to share what I learned along the way. Can't code to save my life. Had a few failed attempts. But after building good prd's using taskmaster and Claude Opus things started to click.

This post walks through my architecture decisions and what worked (and what didn't). I am very open to learning where I XXX-ed up, and what cool stuff i can do with it (gemini ai studio on top of this RAG would be awesome) Please post some ideas.


Tech Stack Overview

Here's what I ended up using:

• Backend: FastAPI (Python)
• Frontend: Next.js 14 (React + TypeScript)
• Vector DB: Qdrant
• Embeddings: Voyage AI (voyage-context-3)
• Sparse Vectors: FastEmbed SPLADE
• Reranking: Voyage AI (rerank-2.5)
• Q&A: Gemini 2.5 pro
• Orchestration: Temporal.io
• Database: PostgreSQL (for Temporal state only)

Part 1: How Documents Get Processed

When you upload a document, here's what happens:

                    ┌─────────────────────┐
                    │  Upload Document    │
                    │   (PDF, DOCX, etc)  │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │ Temporal Workflow   │
                    │   (Orchestration)   │
                    └──────────┬──────────┘
                               │
           ┌───────────────────┼───────────────────┐
           │                   │                   │
           ▼                   ▼                   ▼
    ┌──────────┐        ┌──────────┐       ┌──────────┐
    │   1.     │        │   2.     │       │   3.     │
    │  Fetch   │───────▶│  Parse   │──────▶│ Language │
    │  Bytes   │        │  Layout  │       │ Extract  │
    └──────────┘        └──────────┘       └──────────┘
                                                  │
                                                  ▼
                                           ┌──────────┐
                                           │   4.     │
                                           │  Chunk   │
                                           │ (1000    │
                                           │ tokens)  │
                                           └─────┬────┘
                                                 │
                        ┌────────────────────────┘
                        │
                        ▼
              ┌─────────────────┐
              │  For Each Chunk │
              └────────┬────────┘
                       │
       ┌───────────────┼───────────────┐
       │               │               │
       ▼               ▼               ▼
  ┌─────────┐    ┌─────────┐    ┌─────────┐
  │   5.    │    │   6.    │    │   7.    │
  │ Dense   │    │ Sparse  │    │ Upsert  │
  │ Vector  │───▶│ Vector  │───▶│ Qdrant  │
  │(Voyage) │    │(SPLADE) │    │  (DB)   │
  └─────────┘    └─────────┘    └────┬────┘
                                      │
                      ┌───────────────┘
                      │ (Repeat for all chunks)
                      ▼
               ┌──────────────┐
               │     8.       │
               │  Finalize    │
               │  Document    │
               │   Status     │
               └──────────────┘

The workflow is managed by Temporal, which was actually one of the best decisions I made. If any step fails (like the embedding API times out), it automatically retries from that step without restarting everything. This saved me countless hours of debugging failed uploads.

The steps:

  1. Download the document

  2. Parse and extract the text

  3. Process with NLP (language detection, etc)

  4. Split into 1000-token chunks

  5. Generate semantic embeddings (Voyage AI)

  6. Generate keyword-based sparse vectors (SPLADE)

  7. Store both vectors together in Qdrant

  8. Mark as complete

One thing I learned: keeping chunks at 1000 tokens worked better than the typical 512 or 2048 I saw in other examples. It gave enough context without overwhelming the embedding model.


Part 2: How Queries Work

When someone searches or asks a question:

                    ┌─────────────────────┐
                    │   User Question     │
                    │ "What is Q4 revenue?"│
                    └──────────┬──────────┘
                               │
                  ┌────────────┴────────────┐
                  │   Parallel Processing   │
                  └────┬────────────────┬───┘
                       │                │
                       ▼                ▼
              ┌────────────┐    ┌────────────┐
              │   Dense    │    │   Sparse   │
              │  Embedding │    │  Encoding  │
              │  (Voyage)  │    │  (SPLADE)  │
              └─────┬──────┘    └──────┬─────┘
                    │                  │
                    ▼                  ▼
           ┌────────────────┐  ┌────────────────┐
           │ Dense Search   │  │ Sparse Search  │
           │ in Qdrant      │  │ in Qdrant      │
           │ (Top 1000)     │  │ (Top 1000)     │
           └────────┬───────┘  └───────┬────────┘
                    │                  │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │   DBSF Fusion   │
                    │ (Score Combine) │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │  MMR Diversity  │
                    │  (λ = 0.6)      │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │  Top 50         │
                    │  Candidates     │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │ Voyage Rerank   │
                    │  (rerank-2.5)   │
                    │ Cross-Attention │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │  Top 12 Chunks  │
                    │  (Best Results) │
                    └────────┬────────┘
                             │
                    ┌────────┴────────┐
                    │                 │
              ┌─────▼──────┐   ┌──────▼──────┐
              │   Search   │   │     Q&A     │
              │   Results  │   │  (GPT-4)    │
              └────────────┘   └──────┬──────┘
                                      │
                                      ▼
                              ┌───────────────┐
                              │ Final Answer  │
                              │ with Context  │
                              └───────────────┘

The flow:

  1. Query gets encoded two ways simultaneously (semantic + keyword)

  2. Both run searches in Qdrant (1000 results each)

  3. Scores get combined intelligently (DBSF fusion)

  4. Reduce redundancy while keeping relevance (MMR)

  5. A reranker looks at top 50 and picks the best 12

  6. Return results, or generate an answer with GPT-4

The two-stage approach (wide search then reranking) was something I initially resisted because it seemed complicated. But the quality difference was significant - about 30% better in my testing.


Why I Chose Each Tool

Qdrant

I started with Pinecone but switched to Qdrant because:

  • It natively supports multiple vectors per document (I needed both dense and sparse)

  • DBSF fusion and MMR are built-in features

  • Self-hosting meant no monthly costs while learning

The documentation wasn't as polished as Pinecone's, but the feature set was worth it.

# This is native in Qdrant:
prefetch=[
    Prefetch(query=dense_vector, using="dense_ctx"),
    Prefetch(query=sparse_vector, using="sparse")
],
fusion="dbsf",
params={"diversity": 0.6}

With MongoDB or other options, I would have needed to implement these features manually.

My test results:

  • Qdrant: ~1.2s for hybrid search

  • MongoDB Atlas (when I tried it): ~2.1s

  • Cost: $0 self-hosted vs $500/mo for equivalent MongoDB cluster


Voyage AI

I tested OpenAI embeddings, Cohere, and Voyage. Voyage won for two reasons:

1. Embeddings (voyage-context-3):

  • 1024 dimensions (supports 256, 512, 1024, 2048 with Matryoshka)

  • 32K context window

  • Contextualized embeddings - each chunk gets context from neighbors

The contextualized part was interesting. Instead of embedding chunks in isolation, it considers surrounding text. This helped with ambiguous references.

2. Reranking (rerank-2.5): The reranker uses cross-attention between the query and each document. It's slower than the initial search but much more accurate.

Initially I thought reranking was overkill, but it became the most important quality lever. The difference between returning top-12 from search vs top-12 after reranking was substantial.


SPLADE vs BM25

For keyword matching, I chose SPLADE over traditional BM25:

Query: "How do I increase revenue?"

BM25: Matches "revenue", "increase"
SPLADE: Also weights "profit", "earnings", "grow", "boost"

SPLADE is a learned sparse encoder - it understands term importance and relevance beyond exact matches. The tradeoff is slightly slower encoding, but it was worth it.


Temporal

This was my first time using Temporal. The learning curve was steep, but it solved a real problem: reliable document processing.

Temporal does this automatically. If step 5 (embeddings) fails, it retries from step 5. The workflow state is persistent and survives worker restarts.

For a learning project, this might be overkill, but this is the first good rag i got working


The Hybrid Search Approach

One of my bigger learnings was that hybrid search (semantic + keyword) works better than either alone:

Example: "What's our Q4 revenue target?"

Semantic only:
  ✓ Finds "Q4 financial goals"
  ✓ Finds "fourth quarter objectives"  
  ✗ Misses "Revenue: $2M target" (different semantic space)

Keyword only:
  ✓ Finds "Q4 revenue target"
  ✗ Misses "fourth quarter sales goal"
  ✗ Misses semantically related content

Hybrid (both):
  ✓ Catches all of the above

DBSF fusion combines the scores by analyzing their distributions. Documents that score well in both searches get boosted more than just averaging would give.


Configuration

These parameters came from testing different combinations:

# Chunking
CHUNK_TOKENS = 1000
CHUNK_OVERLAP = 0

# Search
PREFETCH_LIMIT = 1000  # per vector type
MMR_DIVERSITY = 0.6    # 60% relevance, 40% diversity
RERANK_TOP_K = 50      # candidates to rerank
FINAL_TOP_K = 12       # return to user

# Qdrant HNSW
HNSW_M = 64
HNSW_EF_CONSTRUCT = 200
HNSW_ON_DISK = True

What I Learned

Things that worked:

  1. Two-stage retrieval (search → rerank) significantly improved quality

  2. Hybrid search outperformed pure semantic search in my tests

  3. Temporal's complexity paid off for reliable document processing

  4. Qdrant's named vectors simplified the architecture

Still experimenting with:

  • Query rewriting/decomposition for complex questions

  • Document type-specific embeddings

  • BM25 + SPLADE ensemble for sparse search


Use Cases I've Tested

  • Searching through legal contracts (50K+ pages)

  • Q&A over research papers

  • Internal knowledge base search

  • Email and document search


r/Rag • • Nov 19 '25

Discussion Gemini 3 vs GPT 5.1 for RAG

214 Upvotes

Gemini 3 dropped yesterday, so I tested it inside a real RAG pipeline and compared it directly with GPT-5.1. Used same retrieval, same chunks, same setup.

Across 5 areas (conciseness, grounding, relevance, completeness, source usage), they were pretty different:

– In 3/5 cases Gemini 3 gave the more focused answer
- GPT 5.1 were more expressive while Gemini 3 is direct, to the point
- Gemini 3 is better at turning messy chunks into a focused answer

My takeaway was the difference isn’t about “which one is smarter,” it’s about what style you prefer.

I shared screenshots of how exactly each performed in these 5 categories and talked more about them here: https://agentset.ai/blog/gemini-3-vs-gpt5.1


r/Rag • • Jul 21 '25

My RAG Journey: 3 Real Projects, Lessons Learned, and What Actually Worked

202 Upvotes

Edit: This post is enhanced using Claude.

TL;DR: Sharing my actual RAG project experiences and earnings to show the real potential of this technology. Made good money from 3 main projects in different domains - security, legal, and real estate. All clients were past connections, not cold outreach.

Hey r/Rag community!

My comment about my RAG projects and related earnings got way more attention than expected, so I'm turning it into a proper post with all the follow-up Q&As to help others see the real opportunities out there. No fluff - just actual projects, tech stacks, earnings, and lessons learned.

Link to comment here: https://www.reddit.com/r/Rag/comments/1m3va0s/comment/n3zuv9p/

How I Found These Clients (Not Cold Calling!)

Key insight: All projects came from my existing network - past clients and old leads from 4-5 years ago that didn't convert back then due to my limited expertise.

My process:

  1. Made a list of past clients

  2. Analyzed their pain points (from previous interactions)

  3. Thought about what AI solutions they'd need

  4. Reached out asking if they'd want such solutions

  5. For interested clients: Built quick demos in n8n

  6. Created presentation designs in Figma + dashboard mockups in Lovable

  7. Presented demos, got buy-in, took advance payment, delivered

Timeline: All projects proposed in March 2025, execution started in April 2025. Each took 1-1.5 months of development time.

Project #1: Corporate Knowledge Base Chatbot

Client: US security audit company (recently raised $10M+ funding)

Problem: Content-rich WordPress site (4000+ articles) with basic search

Solution proposed: AI chatbot with full knowledge base access for logged-in users

Tech Stack: n8n, Qdrant, Chatwoot, OpenAI + Perplexity, Custom PHP

Earnings: $4,500 (from planning to deployment) + ongoing maintenance

Why I'm Replacing Qdrant Soon:

Want to experiment with different vector databases. Started with pgvector → moved to qdrant → now considering GraphRAG. However, GraphRAG has huge latency issues for chatbots.

The real opportunity is their upcoming sales/support bots. GraphRAG (Using Graphiti) relationships could help with requirement gathering ("Vinay needs SOC2" type relations) and better chat qualification.

Multi-modal Challenges:

Moving toward embedding articles with text + images + YouTube embeds + code samples + internal links + Swagger/Redoc embeds. This requires:

  • CLIP for images before embedding

  • Proper code chunking (can't split code across chunks)

  • YouTube transcription before embedding

  • Extensive metadata management

Code Chunking Solution: Custom Python scripts parse HTML, preserve important tags, and process content separately. Use 1 chunk per code block, connect via metadata. When retrieving, metadata reconnects chunks for complete responses.

Data Quality: Initially, very hallucinated responses. Fixed with precise system prompts, iterations, and correct penalties.

Project #2: Legal Firm RAG System (Limited Details Due to NDA)

Client: Indian law firm (my client from 4-5 years ago for case management system on Laravel) Challenge: Complex legal data relationships Solution: Graph-based RAG with Graphiti

Features:

  • 30M+ court cases with entity relationships, verdicts, statements

  • Complete Indian law database with amendments and history

  • Fully local deployment (office-only access + a few specific devices remotely)

  • Custom-trained Mistral 7B model

Tech Stack: Python, Ollama, Docling, Laravel + MySQL

Hardware: Client didn't have GPU hardware on-prem initially. I sourced required equipment (cloud training wasn't allowed due to data sensitivity).

Earnings: $10K-15K (can't give exact figure due to NDA)

Data Advantage: Already had structured data from the case management system I built years ago. APIs were ready, which saved significant time.

Performance: Good so far but still working on improvements.

Non-compete: Under agreement not to replicate this solution for 2 years. Getting paid monthly for maintenance and enhancements.

Note: Someone said I could have charged 3x more. Maybe, but I charge by time/effort, not client capacity. Trust and relationships matter more than maximizing every dollar.

Project #3: Real Estate Voice AI + RAG

Client: US real estate (existing client, took over maintenance) Scope: Multi-modal AI system

Features:

  • Website chatbot for property requirements and lead qualification

  • Follow-up questions (pets, schools, budget, amenities)

  • Voice AI for inbound/outbound calls (same workflow as chatbot)

  • Smart search (NLP to filters, not RAG-based)

Tech Stack: Python, OpenAI API, Ultravox, Twilio, Qdrant Earnings: $7,500 (separate from website dev and CRM costs)

Business Scaling Strategy & Business Insights

Current Capacity: I can handle 5 projects simultaneously, and max 8 (I need family time and time for my dog too!)

Scaling Plan:

  • I won't stay solo long (I was previously a CTO/partner in an IT agency for 8 years, left in March 2025)

  • You need skilled full-stack developers with right mindset (Sadly, it's the hardest part to find these people)

  • With a team you can do 3-4 projects per person per month very easily.

  • And of course you can't do everything alone (delegation is the key)

Why Scaling is Challenging: Finding skillful developers with the right mindset is tricky, but once you have them, AI automation business scales easily.

Technical Insights & Database Choices

OpenSearch Consideration: Great for speed (handles 1M+ embeddings fast), but our multi-modal requirements make it complex. Need to handle CLIP, proper chunking, transcription, and extensive metadata.

Future Plan: Once current experiments conclude, build a proprietary KB platform that handles all content types natively and provides best answers regardless of content format.

Key Takeaways

For Finding Clients:

  • Your existing network is a goldmine

  • Old "failed" leads often become wins with new capabilities

  • Demo first, sell second

  • Advance payments are crucial

For Developers:

  • RAG isn't rocket science, but needs both dev and PM mindset

  • Self-hosting is major selling point for sensitive data

  • Graph RAG works better for complex relationships (but watch latency)

  • Voice integration adds significant value

  • Data quality issues are fixable with proper prompting

For Business:

  • Maintenance contracts provide steady income

  • NDA clients often pay a monthly premium. (You just need to ask)

  • Each domain has unique requirements

  • Relationships and trust > maximizing every deal

I'll soon post about Projects 4, 5 and 6 they are in healthcare and agritech domains, plus a Vision AI healthcare project that might interest VCs.

I'd love to explore your suggestions and read your experience with RAG projects. Anything I can improve? Any questions you might have? Any similar stories or client acquisition strategies that worked for you?


r/Rag • • Aug 06 '25

Companies need to stop applauding vanilla RAG

196 Upvotes

I built a RAG system for internal documents pulled from a mix of formats, like PDFs and wikis. At first, the results were clean and useful.

But that was at the start. as the document set grew, the answers werent as reliable. Some of them werent using the most up to date policy section, or they were mixing information when it shouldnt be.

We had been using Jamba for generation. It worked well in most cases because it tended to preserve the phrasing from retrieved chunks, which made answers easier to trace. 

With any technology, it does what its been programmed to do. That means it returns content exactly as retrieved, even if the source isnt current.

I feel like many companies are getting a RAG vendor or a freelancer to build a setup and thinking theyre so ahead of the times, but actually the  tech is one step ahead. 

You have to keep your documentation up to date and/or have a more structured retrieval layer. If you want your setup to reason about the task, RAG is not enough. It’s retrieval, not orchestration, not a multi-layered workflow.


r/Rag • • Mar 23 '26

Tools & Resources I got tired of RAG and spent a year implementing the neuroscience of memory instead

190 Upvotes

EDIT: For anyone who’s followed this post and is curious what came next.

I’ve since built a platform around Mímir called Mímir’s Memory Hub a free, fully local AI hub that works with most existing setups (Ollama, HuggingFace, local GGUF, APIs and more) with Mímir fully integrated and ready to go across multiple use cases (companion, agent, character, assistant). Watch memories form, group, and fade in real-time through live visual graphs. No setup headaches, just download and run.

For anyone interested: https://github.com/Kronic90/Mimirs-Memory-Hub

I've been building memory systems for AI agents for about a year now and I keep running into the same problem — most memory systems treat memory like a database. Store a fact, retrieve a fact. Done.

But that's not how memory actually works. Human memory decays, drifts emotionally, gets suppressed by similar memories, surfaces involuntarily at random moments, and consolidates during sleep into patterns you never consciously noticed. None of that happens in a vector DB.

So I spent the last year implementing the neuroscience instead.

Mímir is the result — a Python memory system built on 21 mechanisms from published cognitive science research:

- Flashbulb memory (Brown & Kulik 1977) — high-arousal events get permanent stability floors

- Reconsolidation (Nader et al 2000) — recalled memories drift 5% toward current mood, so memories literally change when you remember them

- Retrieval-Induced Forgetting (Anderson 1994) — retrieving one memory actively suppresses similar competitors

- Zeigarnik Effect — unresolved failures stay extra vivid, agents keep retrying what didn't work

- Völva's Vision — during sleep_reset(), random memory pairs are sampled and synthesised into insight memories the agent wakes up with

- Yggdrasil — a persistent memory graph with 6 edge types connecting episodic, procedural, and social memory into a unified knowledge structure

Retrieval uses a hybrid BM25 + semantic + date index with 5-signal re-ranking (keyword, semantic, vividness, mood congruence, recency). It's the thing that finally got MSC competitive with raw TF-IDF after keyword-only systems were beating purely semantic ones.

Benchmark results on 6 standard memory benchmarks (Mem2ActBench, MemoryBench, LoCoMo, LongMemEval, MSC, MTEB):

- Beats VividnessMem on Mem2ActBench by 13% Tool Accuracy

- 96% R@10 on LongMemEval

- 100% on 3 of 6 LongMemEval categories (knowledge-update, single-session-preference, single-session-user)

- MSC essentially tied with TF-IDF baseline (was losing by 11% before the hybrid bridge)

It orchestrates two separately published packages — VividnessMem (neurochemistry engine) and VividEmbed (389-d emotion-aware embeddings) — but works standalone with graceful fallbacks if you don't want the full stack.

pip install vividmimir

Repo and full benchmark results: https://github.com/Kronic90/Mimir

Happy to answer questions about the architecture or the neuroscience behind any of the mechanisms — some of the implementation decisions are non-obvious and worth discussing.


r/Rag • • Jan 06 '25

Tutorial The RAG_Techniques repo hit 10,000 stars on GitHub and is the world's leading open source tutorials for RAG

Thumbnail
github.com
186 Upvotes

Whether you're a beginner or looking for advanced topics, you'll find everything RAG-related in this repository.

The content is organized in the following categories:

  1. Foundational RAG Techniques
  2. Query Enhancement
  3. Context and Content Enrichment
  4. Advanced Retrieval Methods
  5. Iterative and Adaptive Techniques
  6. Evaluation
  7. Explainability and Transparency
  8. Advanced Architectures

As of today, there are 31 individual lessons. AND, I'm currently working on building a digital course based on this repo – more details to come!


r/Rag • • Sep 06 '25

Showcase I open-sourced a text2SQL RAG for all your databases

Post image
184 Upvotes

Hey r/Rag  👋

I’ve spent most of my career working with databases, and one thing that’s always bugged me is how hard it is for AI agents to work with them. Whenever I ask Claude or GPT about my data, it either invents schemas or hallucinates details. To fix that, I built Statespace. It's a free and open-source Python library for creating lightweight but powerful retrieval agents, giving them a safe, smart way to actually understand and query your database schemas.

So, how does it work?

Statespace gives your agents two read-only database tools so they can explore your data and quickly find answers. You can also add business context to help the AI better understand your databases. It works with the built-in MCP server, or you can set up your own custom retrieval tools.

Connects to everything

  • 15+ databases and warehouses, including: Snowflake, BigQuery, PostgreSQL & more!

  • Data files like CSVs, Parquets, JSONs, and even Excel files.

  • Any API with an OpenAPI/Swagger spec (e.g. GitHub, Stripe, Discord, and even internal APIs)

Why you'll love it

  • Zero configuration: Skip config files and infrastructure setup. Statespace works out of the box with all your data and models.

  • Predictable results: Data is messy. Statespace returns structured, type-safe responses that match exactly what you want e.g.

    • answer: list[int] = db.ask(...)

  • Use it anywhere: Avoid migrations. Run Statespace directly, as an MCP server, or build custom tools for your favorite AI framework.

If you’re building AI agents for databases (or APIs!), I really think Statespace could make your life easier. Your feedback last time was incredibly helpful for improving the project. Please keep it coming!

Docs: https://docs.statespace.com/

GitHub Repo: https://github.com/statespace-tech/statespace

Discord: https://discord.com/invite/rRyM7zkZTf

A ⭐ on GitHub really helps with visibility!


r/Rag • • Apr 05 '25

Me when someone asks me "why bother with RAG when I can dump a pdf in chatGPT?"

Post image
179 Upvotes

r/Rag • • Nov 12 '25

Showcase I tested different chunks sizes and retrievers for RAG and the result surprised me

173 Upvotes

Last week, I ran a detailed retrieval analysis of my RAG to see how each chunking and retrievers actually affects performance. The results were interesting

I ran experiment comparing four chunking strategies across BM25, dense, and hybrid retrievers:

  • 256 tokens (no overlap)

  • 256 tokens with 64 token overlap

  • 384 tokens with 96 token overlap

  • Semantic chunking

For each setup, I tracked precision@k, recall@k and nDCG@k with and without reranking

Some key takeaways from the results are:

  • Chunking size really matters: Smaller chunks (256) consistently gave better precision while the larger one (384) tends to dilute relevance

  • Overlap helps: Adding a small overlap (like 64 tokens) gave higher recall, especially for dense retrievals where precision improved 14.5% (0.173 to 0.198) when I added a 64 token overlap

  • Semantic chunking isn't always worth it: It improved recall slightly, especially in hybrid retrieval, but the computational cost didn't always justify

  • Reranking is underrated: It consistently boosted reranking quality across all retrievers and chunkers

What I realized is that before changing embedding models or using complex retrievers, tune your chunking strategy. It's one of the easiest and most cost effective ways to improve retrieval performance


r/Rag • • Aug 10 '25

I built a comprehensive RAG system, and here’s what I’ve learned

173 Upvotes

Disclaimer: This is a very biased setup, with decisions based on my research from different sources and books. You might not agree with this setup — and that’s fine. However, I’m not going to defend why I chose PostgreSQL over Qdrant or any other vector database, nor any other decision made here.

What is ChatVia.ai?

A few months ago, I had the idea of creating an AI agent (similar to ChatGPT) lingering in my mind. I first tried building it with Chainlit (failed many times) and then with Streamlit (failed miserably as well).

About three months ago, I decided to start a completely new project from scratch, welcome to ChatVia.ai.

ChatVia.ai provides a comprehensive RAG system that uses multiple techniques to process and chunk data. In this post, I’ll explain each technique and technology.

I built ChatVia.ai in my free time. On some weekends, I found myself working 10–12 hours straight, but with such a big project, I had no choice but to keep going.

What makes ChatVia.ai different from other RAG systems is how much I cared about accuracy and speed above everything else. I also wanted simplicity, something easy to use and straightforward. Since I only launched it today, you might still encounter bugs here and there, which is why I’ve set up a ticket system so you can report any issues, and I’ll keep fixing them.

ChatVia.ai supports streaming images. If you ask about a chart included in a document, it will return the actual chart as an image along with a description, it won’t just tell you what’s in the chart. I’ve tested it with academic papers, books, and articles containing images, and it worked perfectly:

So, let’s start with my stack.

My Stack

For this project, I used the following technologies:

  • Frontend:

    • Tailwind CSS 4

    • Vue.js 3

    • TypeScript

  • Backend:

    • PHP 8.4

    • Laravel 12

    • Rust (for tiktoken)

    • Python (FastAPI) for ingestion and chunking

  • WebSever:

    • Nginx

    • PHP-FPM with Opcache and Jit.

  • Database:

    • PostgreSQL

    • Redis

Vector Database

Among all the databases I’ve tested (Qdrant, Milvus, ChromaDB, Pinecone), I found VectorChord for PostgreSQL to be the best option for my setup.

Why? Three main reasons:

  • Is insanely fast. When combined with binary quantization (I do use binary quantization), it can handle millions of documents in under 500 ms, that’s very impressive.

  • Supports BM25 for hybrid search.

  • Since I already use PostgreSQL, I can keep everything together with no need for an extra database.

For BM25, I use the llmlingua2 model because it’s multilingual.

My Servers

I currently have two servers — one primary and one secondary (for disaster recovery).

Both run on AMD EPYC 7502P, with 2 TB NVMe storage and 256 GB RAM. That’s enough to handle hundreds of thousands of concurrent requests.

Document Parsing

Document parsing is the most important aspect in RAG system (along with chunking), if you can’t extract meaningful information from the document then your rag wouldn’t work as the user expectes it, this is what I felt whenever I use a rag system, it feels like their document parsing is so cheap and naive. Therefore I’ve chosen something different, which is llama parse.

Compared to Azure document intelligence, Google Document AI and AWS textract (the ones I tried), LLamaparse is:

  • Very easy to use

  • Customizable, you can tell it to extract images, tables, etc…

  • Affordable and predictable pricing model.

  • Supports High Quality OCR

I use llama parse to extract text, images and tables, the images will be stored in Object Storage and sent back in the streaming (if needed), this will make the user see meaningful responses instead of just text.

Chunking

Among all the techniques I’ve tried for chunking, I found agentic chunking to be the most effective. I know it can be expensive if you’re sending millions of tokens, but for ChatVia.ai, accuracy matters more than cost. I want the chunks to be coherent, with ideal breakpoints.

Along with chunking, I ask the LLM to generate two additional elements:

  • A summary of the chunk

  • Relevant questions

The only downside of the agentic chunking is the speed, because every chunk needs to be processed by the LLM, however I do use a robust queuing system that is capable of handling thousands of requests concurrently, and accuracy is way important to me that some cheap chunking methods that wouldn’t yield the best results.

Embedding Model

I’ve tried a few embedding models, including:

  • OpenAI text-embedding-3-large

  • Cohere embed-v4

  • Mistral embed.

  • gemini-embedding-001

Honestly, I couldn’t tell the difference, but from my limited testing I found Cohere embed-v4 works very well with different languages (tested with Arabic, Danish and English).

Re-ranking

I use Cohere Rerank when retrieving data from PostgreSQL (top-k = 6), and then I populate the sources so the user can see the retrieved chunks for the given answer.

Evals

In the Enterprise RAG book by Tyler Suard (manning publication), Chapter 2: Nothing happens until someone writes an eval) Tyler says that RAG should be tested by writing what so-called evals.

An eval is simply a test case for your RAG system, a predefined question-and-answer pair that represents something your chatbot should be able to handle correctly.

Eval is similar to unit test but for RAG:

  • The question is the input.

  • The expected answer is the correct output.

  • When you run the eval, you check whether your system’s actual answer matches (or closely matches) the expected one.

Therefore I wrote a lot of evals for different document, this way I make sure that my RAG system is actually working.

Streaming

In the beginning, I tried using WebSockets, but I found them unnecessarily complex. Since WebSockets are full-duplex connections, they weren’t really needed for a chatbot. I switched to SSE (Server-Sent Events) instead, and for the record, most modern chatbots use SSE, not WebSockets.

Models

For the models, I use a combination of Groq and OpenRouter. I’m also experimenting with installing Qwen locally to allow users to choose between a local model or an existing one, but I’ll postpone this step until I have customers for my business.

GraphRAG

To make the RAG more accurate, I started digging into GraphRAG, Thanks to Essential GraphRAG book, however I’m still experimenting with GraphRAG and I didn’t create anything production-ready yet, but this is my next step and if I make it to production I will write a post about it.

Chat Memory

Since speed matters, I found that Redis is the best option to use for the Chat Memory, because it’s way faster than any other database.

Just Ask

If you have any questions, whether about implementation, RAG in general, or my setup, feel free to ask, either publicly or via DM. I’ll do my best to help however I can.

Thank you!


r/Rag • • Jun 09 '25

Tutorial RAG Isn't Dead—It's evolved to be more human

168 Upvotes

After months of building and iterating on our AI agent for financial work at decisional.com, I wanted to share some hard-earned insights about what actually matters when building RAG applications in the real world. These aren't the lessons you'll find in academic papers or benchmark leaderboards—they're the messy, human truths we discovered by watching hundreds of hours of actual users interacting with our RAG assisted system.

If you're interested in making RAG assisted AI systems work, this is a post that helps product builders.

The "Vibe Test" Comes First

Here's something that caught us completely off guard: the first thing users do when they upload documents isn't ask the sophisticated, domain-specific questions we optimized for. Instead, they perform a "vibe test."

Users upload a random collection of documents—CVs, whitepapers, that PDF they bookmarked three months ago—and ask exploratory questions like "What is this about?" or "What should I ask?" These documents often have zero connection to each other, but users are essentially kicking the tires to see if the system "gets it."

This led us to an important realization: benchmarks don't capture the vibe test. We need what I'm calling a "Vibe Bench"—a set of evaluation questions that test whether your system can intelligently handle the chaotic, exploratory queries that build initial user trust.

The practical takeaway? Invest in smart prompt suggestions that guide users toward productive interactions, even when their starting point is completely random.

Also just because you built your system to beat domain specific benchmarks like FinQA, Financebench, FinDER, TATQA, ConvFinQA doesn’t mean anything until you get past this first step.

The Goldilocks Problem of Output Token Length

We discovered a delicate balance in response length that directly correlates with user satisfaction. Too short, and users think the system isn't intelligent enough. Too long, and they won't read it.

But here's the twist: the expected response length scales with the amount of context users provide. When someone uploads 300 pages of documentation, they expect a comprehensive response, even if 90% of those pages are irrelevant to their question.

I've lost count of how many times we tried to tell users "there's nothing useful in here for your question," only to learn they're using our system precisely because they don't want to read those 300 pages themselves. Users expect comprehensive outputs because they provided comprehensive inputs.

Multi-Step Reasoning Beats Vector Search Every Time

This might be controversial, but after extensive testing, we found that at inference time, multi-step reasoning consistently outperforms vector search.

Old RAG approach: Search documents using BM25/semantic search, apply reranking, use hybrid search combining both sparse and dense retrievers, and feed potentially relevant context chunks to the LLM.

New RAG approach: Allow the agent to understand the documents first (provide it with tools for document summaries, table of contents) and then perform RAG by letting it query and read individual pages or sections.

Think about how humans actually work with documents. We don't randomly search for keywords and then attempt to answer questions. We read relevant sections, understand the structure, and then dive deeper where needed. Teaching your agent to work this way makes it dramatically smarter.

Yes, this takes more time and costs more tokens. But users will happily wait if you handle expectations properly by streaming the agent's thought process. Show them what the agent is thinking, what documents it's examining, and why. Without this transparency, your app will just seem broken during the longer processing time.

There are exceptions—when dealing with massive documents like SEC filings, vector search becomes necessary to find relevant chunks. But make sure your agent uses search as a last resort, not a first approach.

Parsing and Indexing: Don't Make Users Wait

Here's a critical user experience insight: show progress during text layer analysis, even if you're planning more sophisticated processing afterward i.e table and image parsing or OCR and section indexing.

Two reasons this matters:

  1. You don't know what's going to fail. Complex document processing has many failure points, but basic text extraction usually works.

  2. User expectations are set by ChatGPT and similar tools. Users are accustomed to immediate text analysis. If you take longer—even if you're doing more sophisticated work—they'll assume your system is inferior.

The solution is to provide immediate feedback during the basic text processing phase, then continue more complex analysis (document understanding, structure extraction, table parsing) in the background. This approach manages expectations while still delivering superior results.

The Key Insight: Glean Everything at Ingestion

During document ingestion, extract as much structured information as possible: summaries, table of contents, key sections, data tables, and document relationships. This upfront investment in document understanding pays massive dividends during inference, enabling your agent to navigate documents intelligently rather than just searching through chunks.

Building Trust Through Transparency

The common thread through all these learnings is transparency builds trust. Users need to understand what your system is doing, especially when it's doing something more sophisticated than they're used to. Show your work, stream your thoughts, and set clear expectations about processing time. We ended up building a file viewer right inside the app so that users could cross check the results after the output was generated.

Finally, RAG isn't dead—it's evolving from a simple retrieve-and-generate pattern into something that more closely mirrors human research behavior. The systems that succeed will be those that understand not just how to process documents, but how to work with the humans who depend on them and their research patterns.