r/Rag • • 7h ago

Showcase PDFs don't have footnotes. How I link them for RAG without guessing

4 Upvotes

I maintain an open-source document-to-Markdown converter. This post is about one problem I spent far too long on, and I'd like to hear how others handle it.

The problem

Here's a line from page 14 of a (made-up) annual report, as pdfplumber extracts it:

Net debt fell to NOK 512 million, and the net debt ratio to 1.4 times4 at year end, down from 1.9 a

That "4" is a footnote mark. The note at the bottom of the page says the ratio excludes lease liabilities under IFRS 16, and that it would be 2.9 if you included them.

So "1.4" is true, but only with a caveat that roughly doubles the number. Once the page is chunked, the sentence and the note can easily land in different chunks. Retrieval returns "net debt ratio 1.4", the model states it with full confidence, and nothing in the chunk says it's conditional. "times4" is just noise.

This isn't an edge case. Financial reports, legal documents and papers put the important exceptions in footnotes on purpose.

Why it's hard

A PDF has no concept of a footnote. You get glyphs at positions. A small raised "2" could be a footnote mark, the 2 in m², an exponent, or a figure label. The notes are just smaller text near the bottom of the page. When I compared marker, Docling, MinerU, pymupdf4llm and MarkItDown, none of them connected the mark to its note.

And guessing fails in a bad way. A missing link costs you a caveat. A wrong link attaches someone else's caveat to your sentence, and the model will repeat it.

What I ended up doing: link only when the page proves it

A page gets linked only if all of these hold:

  • there's a short horizontal rule near the bottom, with smaller text right under it starting at its left end

  • every note starts with a raised number, and the numbers run on (1, 2, 3, or continuing from the previous page)

  • each note number appears as a raised mark in the body exactly once, in the same order as the notes

  • nothing on the page is ambiguous: if there's a note 2 and also an m² anywhere on the page, the whole page is left alone

If any check fails, the page comes out exactly as it did before. Linked pages get standard Markdown footnotes:

Net debt fell to NOK 512 million, and the net debt ratio to 1.4 times[^4] at year end, ...

[^4]: Net debt is interest-bearing debt minus cash and cash equivalents. The ratio divides it by EBITDA
for the last twelve months and excludes lease liabilities under IFRS 16; including them, the ratio would be 2.9.

The "exactly once, anywhere on the page" rule is there because the clever version broke. My first attempt used reading order: if the m² comes after the mark for note 3, it can't be the mark for note 2. Then LaTeX happened. LaTeX floats figures to the top or bottom of the page, so the mark in a figure caption can be read before mark 1 or after mark 3, and an m² sitting in between took note 2. After that I stopped being clever and just skip the page.

The numbers, honestly

  • 120 PDFs I generated from DOCX files with real footnotes (printed through LibreOffice), so the ground truth is known: 1,580 marks, 864 linked, 0 linked to the wrong note. Everything else was left as it was.

  • On a messier set of 1,103 real-world PDFs (17,708 pages: arXiv papers, a public corpus, about a thousand business documents, mostly Korean), an earlier version linked only 40 marks on 25 pages. I checked every one by hand and none was wrong. Most business documents don't qualify at all, because their marks are inside table cells or there's no footnote rule.

So recall on real documents is low. That's the trade I chose: I'd rather miss a footnote than invent one. I'm honestly not sure it's the right call for every use case.

Using it at retrieval time

The point of [^4] is that the link survives chunking. Labels are unique within a document, so you can re-attach the note to any chunk that cites it:

import re
from opendockit import convert_file

doc = convert_file("annual-report.pdf")
# Each note is its own paragraph ("[^4]: ..."), possibly wrapped over several lines.
notes = dict(re.findall(r"^\[\^([\w-]+)\]: (.+?)(?=\n\n|\Z)", doc.text, flags=re.M | re.S))

def with_notes(chunk: str) -> str:
    """Append the note of every mark the chunk cites but doesn't contain."""
    marks = dict.fromkeys(re.findall(r"\[\^([\w-]+)\](?!:)", chunk))
    extra = [f"[^{m}]: {notes[m]}" for m in marks if m in notes and f"[^{m}]:" not in chunk]
    return "\n\n".join([chunk, *extra])

With page-sized chunks the note is usually in the chunk already. With small fixed-size chunks it often isn't, and that's where this helps.

Questions for you

  • How do you handle footnotes today? Ignore them, chunk by page, run a VLM-based parser?

  • Would you accept higher recall at the cost of an occasional wrong link?

  • If you know public PDFs where footnotes really matter (10-Ks, court opinions, standards), I'd love to test on them. Real-world cases are what I'm short on.

The library is OpenDocKit (MIT, pure Python, no models, runs offline): https://github.com/hyeonsang010716/OpenDocKit

The full footnote rules and their known gaps are written up here: https://github.com/hyeonsang010716/OpenDocKit/blob/main/docs/decisions/pdf-flow.md#pdf-footnotes

The same idea runs through the rest of it. For example, merged table cells are repeated in every row they cover, so a row still makes sense on its own in a chunk. It also reads HWP, which mostly matters if you work with Korean documents.


r/Rag • • 1h ago

Showcase We dropped the vector DB from our RAG setup for a B2B sales assistant. Here's what we did instead.

1 Upvotes

I've been building a WhatsApp assistant for wholesale businesses. Its job is pretty narrow: read incoming messages, figure out whether the sender is a bulk buyer or a one-off retail customer, answer their questions, and hand the serious ones to a human salesperson.

When I started, I did what everyone does: embeddings, a vector store, top-k retrieval. It didn't hold up, and I think the reason is worth sharing.

**Most of the knowledge wasn't "text" in the first place.**

A wholesaler's knowledge base is mostly prices, minimum order quantities, sizes, colors, stock and shipping terms. These are exact values. Embedding them and retrieving "the most similar chunk" turns a lookup into a guess. In B2B sales, quoting $11.25 instead of $12.25 isn't a minor hallucination. It can cost you the deal.

So I split the knowledge into three layers based on how it's used, not where it's stored:

**1. Always in the prompt: identity, tone and rules.**

How the assistant talks, what it's allowed to promise, when it must escalate. The model can't follow a rule it never sees, so this never goes through retrieval.

**2. Fetched on demand through a tool: business facts.**

Shipping, payment terms, returns, samples, size charts. Each business has around 60–80 of these. Putting all of them in every prompt roughly doubled our input tokens. Instead, the model gets a `get_business_info(topic, query)` tool and calls it only when the customer asks about one of those topics. Retrieval is a plain keyword/topic filter on Postgres. No similarity search needed.

**3. Never left to the model: the product catalog.**

A separate `search_products` tool takes structured filters (category, max price, max MOQ, color, size, sort order) and runs a real SQL query. "Cheapest black t-shirt under 500 units MOQ" becomes a filter, not a semantic search.

**Two small details that mattered more than I expected:**

- **Empty results come with instructions.** When the tool finds nothing, it doesn't just return `[]`. It returns something like: "No info on this topic. Do not make it up. Tell the customer you'll check, and hand off to a human." Before I added this, the model would sometimes fill the gap with a confident guess.

- **Tenant scope lives inside the tool, not in the arguments.** Many businesses share the same system, so each tool is built with that business's ID baked in (closure). The model can't pass a different tenant ID, because there's no parameter for it.

**Where I think vectors still make sense:** fuzzy matching, like "find products that look like this photo" or long unstructured docs. That's on the roadmap. It's just not where I'd start for a sales assistant whose answers have to be exactly right.

Conversation memory is simply the last 20 messages for now, which has been enough for sales chats.

Curious how others decide this. When do you actually reach for a vector DB, and when does structured retrieval through tool calls work better for you?


r/Rag • • 2h ago

Showcase Engineering RAG: how do you distinguish a proposed dependency from a current one?

1 Upvotes

I'm building Aquifer, a self-hosted engineering retrieval layer for AI agents, and one problem I've focused on is relationship status.

An issue saying “we should replace Redis” is evidence of a proposal. A merged PR describing a migration is another kind of evidence. Configuration at a specific commit gives you something different again.

Returning all three as “service X uses Y” loses information the agent needs.

Aquifer combines semantic retrieval with explicit-reference parsing from Terraform, Docker Compose, and Backstage descriptors. For prose, it extracts relationships with supporting quotes and labels them as claimed, proposed, removed, discussed, or not merged. Checking that a quote exists doesn't establish that the claim is true.

Agents query it through MCP and can open the underlying source.

Code: https://github.com/Senora-dev/aquifer-ai

It runs locally or in your own infrastructure and is source-available under BSL 1.1. The current benchmark is small and synthetic, so I'm particularly interested in real-world failure cases.

How are others handling this? Do you preserve competing claims for the agent to inspect, apply source-priority rules, or model changes over time?


r/Rag • • 5h ago

Discussion Outsourcing web scraping vs keeping it in-house: what was your tipping point?

1 Upvotes

Most build vs buy threads are about cost per request. I'm more curious about the moment it flipped.

For people who moved scraping to a vendor, or brought it back in-house:

- how many sites or sources were you running?

- how much of someone's week went to fixing parsers and blocks?

- what made you switch? A big redesign, a new anti-bot vendor, the person who built it leaving?

And for people still running it in-house at scale, what keeps it worth it?


r/Rag • • 20h ago

Discussion How do you score a RAG answer when two current sources conflict?

13 Upvotes

A reimbursement answer from our internal policy Q&A assistant got challenged so I checked what it had retrieved. Two policy pages list different limits for the same situation. Both are marked current and both passages appear for the same query.

I inspected the retrieval spans in Braintrust traces and compared response variants using eval cases built around the dispute. We can test whether a response acknowledges the contradiction. We still can't score a specific reimbursement amount as correct because the policy owners haven't decided which page governs.

Our evaluation labels mostly cover missed retrieval and answers that misuse retrieved evidence. This case doesn't fit either one cleanly. A response could cite a valid source and still give a limit that the team later rejects.

I'm leaning toward treating unresolved source conflicts as their own category. Otherwise a groundedness score could look perfectly fine while the correct reimbursement limit remains unresolved.


r/Rag • • 1d ago

Discussion Need advice for my RAG pipeline.

4 Upvotes

Hello everyone! A few months ago I built a tool called coldstart. Its purpose is to help coding agents get oriented before they start work. It uses hooks to capture what the agent learned in a session (saved as jsonl), then retrieves the relevant learnings and re-injects them later so the agent doesn't have to rediscover the same things again.

I started with my own search algorithm. It relied heavily on agents writing search aliases for their notes, and agents never wrote proper aliases, so I moved to BM25. That was better, but I was still disappointed at the junk getting injected.

So I started evaluating embedding models for a hybrid approach. EmbeddingGemma 2 worked best for my use case, followed by granite-embedding-small-english-r2.

Finding the right note is mostly okay now. The part I really need help with is deciding when to inject. The agent writes prose, the user's prompt is prose, the agent's reply is prose, and the notes are prose too. So lots of notes get high scores even on turns where no note is the right answer. In my tests, the single highest score was on a turn where the agent had already read that exact note earlier in the session.

To test this, I took 300 real prompts from my own sessions and marked which ones actually needed a note (about a third). I also kept 80 prompts aside that I never tuned on. Anything I try has to beat "never inject", because if it can't beat staying quiet, it's only adding noise.

What I've tried for picking the right note:

  1. BM25 alone. It finds the right note in the top 3 about half the time.

  2. Embeddings (EmbeddingGemma 2, granite, a few others). Better than BM25, roughly 6 in 10.

  3. Adding the previous user/agent exchange to the query. This helped the most, because many prompts are just "pls resume" or "ok so what now".

  4. Mixing BM25 and embeddings (merging the two rankings, and using BM25 to reorder or filter the embedding results). It looked good on my first small test set and didn't hold up on the bigger one.

  5. Embedding the actual code each note points to, split per function. Code alone was worse, and mixing it in barely changed anything.

  6. Stripping the prose out of notes and keeping only file and function names. No change.

  7. Penalising "popular" notes that match everything. No change.

What I've tried for deciding when to inject:

  1. A minimum score cutoff, plus variations: the gap between the first and second note, how much the top note stands out, injecting only when BM25 and embeddings agree. They all looked fine on the 300 and did worse than staying quiet on the 80.

  2. Not re-injecting notes the agent has already seen in the session (reset after context compaction). This one is right and I shipped it, but it only fixes a small part of the problem.

  3. Checking whether the note is about files the agent is already working on. Confusingly, notes about files the agent already has open turned out to be the most useful ones, so "skip if already touched" is wrong.

  4. Writing out the session state (files opened, notes shown, notes written, turns since compaction) along with the last few exchanges and giving it to JevEmbed, an embedding model built for yes/no questions. Barely better than a coin flip.

  5. A small local LLM (Gemma 4 E2B) as the judge. It either said yes to almost everything or no to almost everything, depending on how I asked.

  6. Haiku as the judge. This was the only thing that beat staying quiet on the 80 (54 vs 44). But it's an API call on every prompt, and I'd like this to run locally and cheaply.

What I haven't tried, and where I'd like your opinion:

  • A small reranker / cross-encoder (MiniLM, mxbai-rerank and similar). Has a reranker score worked as an "inject or not" signal for anyone, or does it have the same problem as similarity?

  • A few learned weights combining the score with session signals (first prompt of a session, just after compaction, BM25 and embeddings agreeing). I'm wary of training on a few hundred labels from one repo.

  • Making a wrong injection cheap instead of trying to never be wrong: inject only a title and a path, and let the agent decide whether to open the note.

  • Letting the coding agent judge, using the pointer approach above, and measuring from transcripts how often it opens the right note.

Has anyone found a cheap way to tell "the agent needs this right now" apart from "this is on the same topic"? Or is an LLM call per turn just the price of doing this? Pointers to papers or datasets are welcome too. I couldn't find any public data labelled for "does this turn need memory" in coding sessions.


r/Rag • • 18h ago

Discussion How do you test that a RAG answer used the policy valid at the time

1 Upvotes

I have observed that a RAG chatbot over company policies will answer with the current policy even when the question is about an order from months ago. The retrieval worked and the chunk was relevant and the answer still went wrong because the policy that was valid on June 30 is not the policy that is valid today. For instance an order placed on June 30 2025 needs the June 30 policy and the current policy gives the wrong answer with full confidence. I want to know if you test this with backdated questions or trust date filters in chunk metadata to catch it and whether anyone keeps effective date ranges on every chunk as a rule. If you can tell me what you check before a doc enters the index and what you check at answer time I would learn more from that than from another debate about chunking.


r/Rag • • 1d ago

Tutorial Not sure how to build a local multimodal RAG? Here is a repo!! Clone it, use it, Own it!

31 Upvotes

I see many lurkers here who aren't sure where to start, so I built this multimodal RAG that you can run locally. It is not a simple notebook RAG, but something you can modify and run in Production. It supports multiple file types, images, tables, csv, excel file, long horizon tasks, memory per user, and a ton more you will see. Since this is local the caveat is that you need beefy hardware to run it, or port it to hugging face or AWS. If you like it share it, also feel free to modify it. The code is modular, so you can replace it with cloud providers. Make sure to read docs, eval, examples!!

Edit: Added support for in-line rendering in the python sandbox
Edit: Added MLX / Apple Silicon support

Here is a demo

https://youtu.be/23QZVBzFpPA

Here is repo link

https://github.com/haris525/local-multimodal-rag


r/Rag • • 23h ago

Showcase Applied AI – Remote Internships / Freelance

1 Upvotes

Hi everyone,

I’m a collage student at one of the tier 1 NITs, an upcoming Accenture intern (Summer 2027), and have prior experience as an AI Intern, where I built production AI field-normalization pipelines, hierarchical document extractors and lot more. I’m looking for a remote AI engineering internship or freelance project work.

If you’re hiring, building an AI product, or know a founder/team looking for someone who can ship reliable LLM pipelines and full-stack AI apps, please DM me or connect me with them—I’d really appreciate any leads or referrals!

Tech Stack:

AI & Agents: Python (uv), LangGraph, LangChain, PydanticAI, MCP, Pydantic v2, ChromaDB, IBM Docling , Ollama models etc..

Reliability & Evals: NeMo Guardrails, DeepEval, SQLGlot, Logfire, RapidFuzz, schema-driven structured extraction

Backend & Frontend: FastAPI, Next.js, React, TypeScript, PostgreSQL, WebSockets, Tailwind CSS, Streamlit

ML & Data: PyTorch, XGBoost, Scikit-learn, SQL, Docker, Tableau, Selenium

What I’ve Built:

AI Financial Analyst (Text-to-SQL & Docker Sandbox Engine): Agentic financial analyst built with LangGraph, Groq LLMs, SQLGlot, and NeMo Guardrails featuring AST-level SQL validation, a self-healing retry loop (90%+ error recovery), and Docker-sandboxed visualizations rendered in Streamlit.

Parallel Document Extraction Pipeline: Multi-worker LangGraph workflow with automated page triage and Pydantic v2 schema validation for high-accuracy document parsing.

Project CHITTI (3D Simulation + Agentic RAG): Real-time voice-controlled LangGraph RAG system connected to Unreal Engine 5 via low-latency WebSockets and FastAPI.

EcoScape AI & TechCTC: Urban heat island analysis tool (Google Earth Engine, XGBoost, Llama 3.3 70B) and a Docker-containerized ML tech compensation estimator trained on 4,800+ records

( via web scraping )(Selenium, XGBoost, Streamlit).


r/Rag • • 1d ago

Discussion LongMemEval-S: 96.0% with the official judge, and what top-5 vs top-20 retrieval changes

4 Upvotes

We ran all 500 LongMemEval-S questions through our memory retrieval (Papr) using the official answer prompt and the unmodified gpt-4o-2024-08-06 judge. Per-question answers and judge labels are public, so the official metrics script reproduces the numbers.

Results

For comparison, Chronos reports 95.6% using full context and multi-turn agents, Hindsight reports 94.6% at 43.6k tokens per question (with a different judge), and gbrain reports 90.6%.

Things I found useful

  • Retrieval recall is close to saturated. All evidence sessions are in the top 5 for 460 of 470 answerable questions (97.9%). Most remaining errors are the reader, not retrieval.

  • Top 20 vs top 5 is a trade, not a free win: 11 gained, 8 lost, net +3. Temporal reasoning went 123 → 130 of 133. Abstention stayed flat (27/30 both).

  • A small reader gets most of the way: gpt-5-mini reaches 94.4% on the same retrieval.

Caveats

  • Used Papr-Embed 0.6B embedding model (open source and open weight), gemini flash 2.5 reranker (also tested Jev), and one turn answer model.

  • Self-reported. Run-to-run variance is about ±3 questions, so 96.0 vs 95.6 is a tie but this run was optimized for token efficiency.

  • We picked top 20 after reviewing the top-5 misses, so treat 96.0% as tuned on this set.

  • No question-type labels are used in retrieval or answering.

Write-up with charts: https://www.papr.ai/blog/papr-longmemeval-96 Results and code: https://github.com/Papr-ai/papr-longmemeval


r/Rag • • 1d ago

Discussion Hybrid RAG vs basic vector search — what are you using in production?

8 Upvotes

hey everyone, wanted to start a discussion around something that comes up quite often when building RAG applications.

a lot of implementations start with embedding documents, storing them in a vector database and retrieving the nearest matches.

that works for many use cases, but it can struggle when queries contain specific technical terms, identifiers, exact phrases or information that depends on multiple documents.

this is where hybrid retrieval becomes interesting.

a typical hybrid RAG pipeline might combine:

1. Semantic retrieval

use embeddings to find conceptually relevant content, even when the wording differs.

2. Keyword retrieval

use lexical search to catch exact terms, product names, identifiers and technical phrases.

3. Reranking

take the retrieved candidates and reorder them based on their relevance to the actual query.

4. Context construction

combine the best results into a useful context window without overwhelming the model.

5. Evaluation

measure retrieval quality separately from final answer quality. otherwise it's difficult to know whether a bad answer came from retrieval or generation.

Also we are hosting a hands-on workshop on November 28 with Brij Kishore Pandey covering hybrid RAG, tools, persistent memory, guardrails and evaluation.

Thought it might be relevant to people building more advanced RAG applications.

Workshop details here

curious what retrieval architecture people here are currently using. are you combining dense and sparse retrieval, or have you found another approach works better?


r/Rag • • 1d ago

Discussion Finding the right data is one thing. Knowing how it connects to other data is a different problem.

0 Upvotes

Take a healthcare example.

A patient's medical record, lab results, diagnosis and insurance claim might all sit in different systems.

An AI system might be able to find all these records. But does it know which lab result belongs to which visit? Does it know if the diagnosis is related to the current claim? Does it know whether that information is still relevant?

This is where I think semantic data becomes important.

It's about helping systems understand what a piece of data means and how it relates to everything around it.

We've spent years making enterprise data easier to store and search. I think the next big challenge is making it easier for AI to understand.

For people working with RAG, knowledge graphs or semantic layers, how are you solving this? Do you define these relationships beforehand, let the AI figure them out, or use a mix of both?

Would be good to hear what's actually working in production.


r/Rag • • 1d ago

Discussion Building a production RAG system for a Data Science master's thesis

5 Upvotes

Hey guys,

For our master's thesis, we're implementing a RAG system for a company. Since this will be a real tool used in production, I'd love your input on two things:

  1. Document parsing & evaluation: We're working with messy documents full of charts, graphs and different structure. Which parsing libraries or strategies do you recommend, and how can we systematically evaluate that the parsing preserved the layout and data accurately?

  2. Academic rigor: How do we frame an applied engineering build so it meets the academic standards of a Data Science thesis? Any recommendations on foundational papers, evaluation frameworks, or experimental setups we should use as inspiration?


r/Rag • • 1d ago

Discussion Do u use Lakebase for RAG state /memory instead of a standalone Postgres?

4 Upvotes

I need to drop my custom reverse ETL pipelines and just use Synced Tables + pgvector in Lakebase, how does the continuous sync latency from Delta tables hold up in prod ?


r/Rag • • 1d ago

Showcase I built an open-source query planner for RAG in TypeScript. First real-data result: 60% cheaper, but it missed my quality target

0 Upvotes

Pathwise picks the cheapest plan per request (model, retrieval depth, thinking level) predicted to meet a quality SLA, and can optionally certify the SLA with a confidence bound. Zero runtime dependencies; MIT. Install: npm i u/arsalandev/pathwise

First real test: HotpotQA dev, BM25 retrieval, token F1, two Gemini models x 3 retrieval depths x 2 thinking levels = 12 plans. 100 calibration questions, 200 test questions, free tier, costs at list prices.

| Strategy | F1 | $/query |

|---|---|---|

| Always cheapest | 0.542 | 0.00025 |

| Typical default | 0.685 | 0.00044 |

| Pathwise (average) | 0.707 | 0.00066 |

| Pathwise (guaranteed) | 0.789 | 0.00164 |

| Always best | 0.789 | 0.00164 |

Target was F1 >= 0.725. The average mode saves 60% but missed by 1.8 points. The guarantee mode refused to certify and fell back to the best plan because, with 100 questions, even the best plan can't prove 95% of its own score.

Limits: one dataset, one provider, one run; HotpotQA may be in the training data; I haven't shown it beats the best single fixed plan yet. The first run also exposed a pruning bug that I fixed and published.

Repo: GitHub Feedback

I'd like feedback on whether the SLA-with-confidence idea is useful, and on how you would benchmark this.


r/Rag • • 2d ago

Discussion I built a large expert-source library, but retrieval has not become reliable execution. How do you test the gap?

1 Upvotes

I've spent several months collecting expert videos, transcripts and discussions to help AI do work in subjects where I'm not an expert myself. SEO is one example, but the same problem applies to other business workflows.

One existing library has original passages, extracted claims, topic summaries, working methods, source references and keyword plus semantic search. So this isn't just a folder of PDFs with no retrieval. Recent checks can find relevant methods, but the saved retrieval tests still miss intended sources, and I don't have a convincing independent evaluation showing that the agent makes better practical decisions with the library.

What I want is more than answers about what an expert said. Given a real problem, the agent should find the relevant teaching, recognise when it applies, choose useful checks with the available tools, verify the result and retain useful corrections for the next task. I understand that adding documents doesn't retrain the underlying model.

For people who have built something like this, how did you separate retrieval failures from failures to apply correctly retrieved information? Did you compare the same tasks with no library, normal retrieval and manually selected evidence? What did you use as the reference answer when the task involves judgement rather than a fact lookup?

The difficult part for me is that I'm not qualified to judge every domain-specific recommendation. How did you involve a specialist in evaluation without needing them to review every output indefinitely? I'd appreciate examples of a small test set or workflow that exposed a real failure, and what changed after you fixed it.

I'm looking for implementation experience, including things that didn't work, rather than another list of tools. All source advice still needs checking against current conditions and the business involved.


r/Rag • • 2d ago

Discussion CleanWeb v3.0 Upgrade Notice (Nov 21): 87% RAG Token Compression via Thin Client & x402 Micropayments

0 Upvotes

Hi everyone,

Quick update for anyone using **`langchain-x402-cleanweb`**, **`llama-index-readers-x402-cleanweb`**, or **`x402-cleanweb-agent`** in their RAG pipelines.

After crossing **14,000+ downloads across PyPI and npm**, we are standardizing our entire stack onto a **Cloud-Native Thin Client + x402 M2M Gateway** over the next 7 days (**effective November 21, 2026**).

### Why we are making this change:

  1. **Always Up-to-Date Extraction:** Local scrapers break constantly on modern SPAs and WAFs. Our Cloud Run gateway handles Next.js/Nuxt hydration, nested CSS cleanup, and covert tracking-pixel neutralization centrally with ~7ms overhead.

  2. **Mathematical Token Arbitrage:** Feeding raw 80KB HTML (~20k tokens) into GPT-4o or Claude costs ~$0.05/page. Routing through CleanWeb ($0.001 USDC) strips it down to ~2,600 clean tokens (~$0.0065), yielding an **85% net cost reduction per RAG document**.

  3. **Zero-Friction Framework Integrations (Already Thin Clients!):**

    - **LangChain:** `pip install langchain-x402-cleanweb`

    - **LlamaIndex:** `pip install llama-index-readers-x402-cleanweb`

    *(Both packages already route directly through our Cloud Run gateway and require zero code changes if you pass your `X-Vault-Key`.)*

### What happens on Nov 21, 2026:

- `x402-cleanweb-agent v3.0.0` launches as a pure Thin Client; monolithic local versions (`v2.8.x` and below) will be yanked on PyPI.

- Free sandbox evaluation is now strictly capped at 3 IP-bound calls on lightweight tiers; high-compute AI & EIP-712 Oracle endpoints require x402 Vault prefunding (Solana, Base, Polygon, Arbitrum).

Full details & live ROI calculator: https://github.com/nohosa001-pixel/x402-cleanweb-agent


r/Rag • • 2d ago

Discussion Ling-3.2-flash's structured-output docs include nested objects and arrays

7 Upvotes

Ant Ling documents nested JSON Schema output for Ling-3.2-flash, including typed fields, required properties and arrays.

For document extraction before retrieval, that allows an output contract richer than one block of text. An illustrative structure could carry a document identifier and an array of sections, each with a heading and its extracted text.

The application defines that structure through response_format and supplies the extraction instructions in the prompt.

Would you retain those section objects through indexing, or flatten them early and keep the hierarchy in metadata? The model's documented schema support gives an extraction pipeline a way to request the structured representation first.


r/Rag • • 3d ago

Discussion My AI agent ignores DateTime MCP tools when scheduling calendar events

4 Upvotes

I'm building a small project to learn MCP by developing three MCP servers:

  • Gmail MCP

  • Google Calendar MCP

  • DateTime MCP

While testing Gmail and Google Calendar integration, I noticed that my agent was struggling with date and time calculations. So I built a separate DateTime MCP to handle natural-language date and time interpretation.

However, I'm facing an orchestration problem.

Expected behaviour

Suppose I ask:

"Remind me tomorrow about gym at 7 AM."

The agent should:

  1. Call DateTime MCP to get the current date and time.

  2. Calculate tomorrow's date.

  3. Pass the resolved date and time to Google Calendar MCP.

  4. Create the calendar event.

Actual behaviour

The agent sometimes completely skips DateTime MCP and directly calls Google Calendar MCP.

As a result, it creates events with incorrect dates.

For example:

  • Expected date: November 10, 2026

  • Actual date: November 10, 2023

The model I'm currently using is GPT-4o-mini, and it appears to fall back to its default temporal assumptions instead of using my DateTime MCP.

What I've tried

  1. Strengthened the system prompt to instruct the agent to use DateTime MCP.

  2. Implemented a ReAct workflow with separate planning and execution stages.

  3. Added validation, but this introduced errors even for simple queries.

  4. Improved tool descriptions, including arguments, return values and usage instructions.

None of these approaches has completely solved the problem.

My question

How do you ensure an AI agent consistently selects and executes the correct MCP tools when multiple MCP servers are connected?

Would explicit orchestration rules, tool dependency enforcement, or a deterministic workflow be a better solution?

I'd appreciate suggestions from people who have worked with MCP orchestration.

Thanks!


r/Rag • • 3d ago

Discussion How do you determine whether traditional RAG is better or worse than a context graph?

13 Upvotes

I've been working with traditional RAG setups for the last 6-8 months and have been running benchmarks on retrieval for one of my long running projects. Recently, I came across context graphs as another context layer that in many ways seems to compete with traditional RAG systems. My question is how do you guys determine whether a data set should be ingested into a RAG setup vs. a context graph or a combination of both? What are the advantages/disadvantages to each?


r/Rag • • 2d ago

Discussion Two claims I had to stop making about a code index, and the measurements that replaced them

0 Upvotes

I run a local index over a codebase: it answers questions about the code with file:line facts instead of dumping files into the prompt. Over the last month I had to delete two sentences from my own README because they were not true. Both are the kind of claim that sounds reasonable until you measure it.

The index currently covers 930 files, 6,701 symbols and 92,409 edges, and has gone through roughly 1,440 revisions in four weeks. Narrow conditions: one repository, one machine, static analysis only. Everything below is inside that window.

Claim 1: "we return facts, not guesses."

We did not, at first. A foreign key declared in a migration is a fact. A column named user_id with no constraint is a guess, and the first version returned both with the same confidence. A model asked how orders join to accounts answered users - because the column name says so - while the actual constraint pointed to accounts.

The fix was to carry provenance on every fact: stated when it comes from a declaration, derived when it comes from a name pattern, with a confidence ceiling of 0.5 on derived facts and a rule that derived facts may not answer questions about structure. The interesting part is what that broke: coverage dropped, and the tool started saying "I don't know" where it used to have an answer. That is the correct trade, but it looks like a regression in every metric that counts answers.

Claim 2: "the index is current."

Reindexing takes minutes on this tree, so I assumed the index was current between runs. It is not: it is current as of the last revision, and the file on disk may be newer. Before I added a staleness marker, answers from a stale index looked exactly like answers from a fresh one - same format, same confidence, same file:line, pointing at a line that had moved.

Now every answer carries a revision id, and the tool distinguishes three results: absent (the fact is not in the corpus), stale (the index knows the file changed and will not answer until refreshed), and out-of-scope (the corpus was never meant to cover it). Only the first is a real "not found". The other two used to be indistinguishable.

Where it broke for the user

The gate. I made it refuse raw file reads until the index was used, and the first version refused with a prohibition: "raw search is blocked". The agent retried twice and then asked me to run the command for it. Three turns lost on the first session. The fix was one sentence: the refusal now names the next call ("run ensure, then search_multi"). Retries stopped immediately.

A gate without an escape hatch has the same failure mode at a larger scale: the agent learns to work around the gate instead of through it.

What I would measure next

False not-found separately from wrong answers. A system that abstains on everything has perfect honesty and zero utility, and the aggregate accuracy hides the difference. On my 14 fixed navigation tasks I now track the two numbers separately, and the "no previously passing case may fail" rule instead of an average - a single flaky case moved the average by 7 points and hid two real regressions.

If you run something similar: what does your index say when it does not know - an empty list, an error, or a refusal with a reason?


r/Rag • • 3d ago

Discussion The Hunt: A paradigm for agentic search.

2 Upvotes

Following a number of discussions on some recent papers:

https://github.com/re-Isearch/CoreQuarry/blob/main/docs/The_Hunt.md

We have made a habit of asking search engines for the top results, as though the information need came with a fixed unit of retrieval, a fixed ranking function and a fixed stopping point.

But research has never worked that way. We follow clues. We open books and compare passages. We change our vocabulary, widen or narrow our scope, examine contradictions and ask new questions. Sometimes the important discovery is the question we did not know how to ask at the beginning.

Agents make that style of investigation newly practical at scale. They also make the limitations of one-shot retrieval newly obvious.

The Hunt is the proposal to build search for that process: locate evidence, preserve its coordinates, express relationships precisely, show useful previews, retrieve the context the question calls for, and let each discovery inform the next search.

Don't make the chunker decide what can be found. Don't make the ranker decide what is worth investigating. Let the hunt continue.


r/Rag • • 3d ago

Showcase I built an open-source document-to-Markdown converter for RAG that keeps table structure (and reads Korean HWP files)

0 Upvotes

When I built RAG over real company documents, retrieval and the model were not what broke first. Conversion was. Tables in PDFs came out as running text, merged cells in spreadsheets became blanks, and HWP files (the Korean word processor format, which is everywhere in Korean offices) did not open at all.

So I wrote a library that only does that step: document in, Markdown out.

Repo: https://github.com/hyeonsang010716/OpenDocKit (MIT, pip install opendockit)

from opendockit import convert_file

result = convert_file("quarterly-report.docx")
result.text       # Markdown: headings, tables, lists, footnotes, LaTeX equations
result.sections   # per page / slide / sheet, for chunking and citations
result.warnings   # what was skipped or approximated

What it does that mattered for my retrieval quality:

  • Merged cells are repeated in every row they cover, so a row still makes sense after chunking splits the table.

  • Spreadsheets use the displayed value (46.4%), not the stored one (0.464459), and several tables on one sheet are split apart.

  • Multi-column PDFs are read in order, borderless tables are detected, repeated headers and footers are dropped.

  • Sections are pages, slides or sheets, so a chunk can cite "page 3" without guessing.

  • No GPU and no ML models. Conversion is offline and deterministic.

Formats: PDF, DOCX, PPTX, XLSX/XLS/CSV, HWP/HWPX, ODT/ODS/ODP, RTF, EPUB, HTML, email, plus images through a vision model if you plug one in. There are LangChain and LlamaIndex loaders and an MCP server (uvx opendockit-mcp).

I compared it with MarkItDown, Kreuzberg, Unstructured and Docling on the same files. It passed 49 of 49 structure checks (yes/no questions like "is the merged cell still attached to its row"); the next tool passed 36. I wrote those checks myself and developed against them, so read that as "no gaps on these checks", not "best on your documents". The method and raw outputs are in the repo under benchmarks/ if you want to rerun it.

What it does not do:

  • No built-in OCR. Scanned PDFs need a vision model API. If you need offline OCR, Docling or Unstructured are the right tools.

  • It is not the fastest. Kreuzberg is roughly 30x faster.

  • On Windows, HTML, email and EPUB are not supported yet.

If it breaks on one of your documents I would like to hear about it. A file that reproduces the problem is the most useful thing you can send.


r/Rag • • 3d ago

Discussion When my RAG gave a wrong answer, I couldn’t tell if retrieval or the model was to blame. This is how I started separating them.

7 Upvotes

I kept doing the same thing when a RAG answer was wrong: tweak the chunk size, swap the embedding model, rewrite the prompt, and hope. Sometimes it helped, but I never knew what had actually been broken.

What helped me was running each failing question through three quick tests (you need the gold answer and the passage it comes from):

  1. Full pipeline: retrieve top-k and ask the model as normal.

  2. Oracle: give the model only the gold passage. If it still fails, the problem isn’t retrieval, it’s reading or your answer checker.

  3. Position sweep: put the gold passage at every position among the retrieved distractors. If it only works in some positions, it’s a context ordering problem, not a retrieval one.

Comparing the three tells you whether to fix retrieval, the prompt/model, or the ordering and reranking. For me it was a good reminder that “the retriever works fine” and “the answer is right” are not the same thing.

I wrote up the code and how to read the results, in case it’s useful dm


r/Rag • • 3d ago

Discussion Your chatbot answered from an outdated document. Now what?

6 Upvotes

A common RAG failure isn't a bad model. It's a correct answer from a document that's no longer true, like last year's pricing page or an old refund policy. If you're evaluating hosted inference services for the answer step, Mango Inference by MangoBoost is worth testing too. Whatever you pick, the model can only be as current as what you retrieve.

So treat the knowledge base like a product, not a one-time import. Decide who owns each source and how often it gets re-synced. When a document changes, update its chunks instead of adding new ones next to the old ones.

Deletions matter as much as updates. If a page is removed or a policy is replaced, make sure the old version actually leaves the index, or it will keep showing up in answers.

It also helps to store a date or version with each document and show the source with the answer. Users can then see where it came from, and your team can spot stale content much faster.

How do you keep your RAG knowledge base up to date: scheduled re-sync, manual updates, or something else?