r/Rag • u/shrapknife • 1h ago
Showcase We dropped the vector DB from our RAG setup for a B2B sales assistant. Here's what we did instead.
I've been building a WhatsApp assistant for wholesale businesses. Its job is pretty narrow: read incoming messages, figure out whether the sender is a bulk buyer or a one-off retail customer, answer their questions, and hand the serious ones to a human salesperson.
When I started, I did what everyone does: embeddings, a vector store, top-k retrieval. It didn't hold up, and I think the reason is worth sharing.
**Most of the knowledge wasn't "text" in the first place.**
A wholesaler's knowledge base is mostly prices, minimum order quantities, sizes, colors, stock and shipping terms. These are exact values. Embedding them and retrieving "the most similar chunk" turns a lookup into a guess. In B2B sales, quoting $11.25 instead of $12.25 isn't a minor hallucination. It can cost you the deal.
So I split the knowledge into three layers based on how it's used, not where it's stored:
**1. Always in the prompt: identity, tone and rules.**
How the assistant talks, what it's allowed to promise, when it must escalate. The model can't follow a rule it never sees, so this never goes through retrieval.
**2. Fetched on demand through a tool: business facts.**
Shipping, payment terms, returns, samples, size charts. Each business has around 60–80 of these. Putting all of them in every prompt roughly doubled our input tokens. Instead, the model gets a `get_business_info(topic, query)` tool and calls it only when the customer asks about one of those topics. Retrieval is a plain keyword/topic filter on Postgres. No similarity search needed.
**3. Never left to the model: the product catalog.**
A separate `search_products` tool takes structured filters (category, max price, max MOQ, color, size, sort order) and runs a real SQL query. "Cheapest black t-shirt under 500 units MOQ" becomes a filter, not a semantic search.
**Two small details that mattered more than I expected:**
- **Empty results come with instructions.** When the tool finds nothing, it doesn't just return `[]`. It returns something like: "No info on this topic. Do not make it up. Tell the customer you'll check, and hand off to a human." Before I added this, the model would sometimes fill the gap with a confident guess.
- **Tenant scope lives inside the tool, not in the arguments.** Many businesses share the same system, so each tool is built with that business's ID baked in (closure). The model can't pass a different tenant ID, because there's no parameter for it.
**Where I think vectors still make sense:** fuzzy matching, like "find products that look like this photo" or long unstructured docs. That's on the roadmap. It's just not where I'd start for a sales assistant whose answers have to be exactly right.
Conversation memory is simply the last 20 messages for now, which has been enough for sales chats.
Curious how others decide this. When do you actually reach for a vector DB, and when does structured retrieval through tool calls work better for you?