r/ollama • • 2d ago

I built this! [Weekly - Nov 20, 2026] Friday Demos - Show off your Ollama projects!

1 Upvotes

Show us what you've built with Ollama this week!

Suggested information to include:

  • A description of your project

  • How you came up with the idea

  • The models and integrations you're using

  • Whether it runs on local hardware or the cloud service

  • How others could use it or view it

  • A link to code or demo if applicable

Rules:

  • No affiliate links or sales

New post every Friday!


r/ollama • • 8h ago

[Release] Caphlon-2 (4.52 GiB): Running full 64-layer Qwen 3.8 27B on 6GB VRAM / 8GB MacBooks without distillation (80% GPQA-Diamond)

10 Upvotes

Hey everyone,

We are sharing Caphlon-2, a lightweight 27B model family built to run full 27B reasoning on laptops with 8 GB RAM and older GPUs with 6 GB VRAM.

Website and live demo:

https://caphlon.dev

Hugging Face models:

https://huggingface.co/ssuasd/Caphlon-2

https://huggingface.co/ssuasd/Caphlon-Model

ollama models:

https://ollama.com/univerisr/caphlon

GitHub repo and scripts:

https://github.com/demiralpdev/Caphlon-Model

WHAT IT IS

Caphlon-2 is based on Qwen 3.8 27B and Ternary-Bonsai-2-27B. It is not a distilled student model. It keeps all 64 layers (48 DeltaNet + 16 full-attention layers).

To get it down to 4.52 GiB (1.648 bpw):

  1. Vocabulary was trimmed from 151,936 down to 83,200 active tokens (V83K), which saved around 1.1 GiB without hurting English or coding quality.

  2. Intermediate FFN size was trimmed from 17,408 to 15,456 across layers 44 to 61 (LayerTrim).

  3. Weights are quantized with PTQ1_SB (Sylvester-Hadamard ternary quantization).

BENCHMARK RESULTS

Caphlon-2 (4.52 GiB):

- GPQA-Diamond: 48 / 60 (80.0%)

- Halu-270 (Hallucination benchmark): 236 / 270 (87.4%) with ITI steering, 232 / 270 (85.9%) pure

- EN-40: 32 / 40 (80.0%)

- Inference speed: 277.95 tok/s prompt processing (pp512) and 16.33 tok/s generation on Tesla T4. Around 18 tok/s on an M2 MacBook Air.

Caphlon-1 (6.87 GiB, IQ2_S):

- HumanEval: 96.7%

- Runs on stock llama.cpp and Ollama out of the box.

HOW TO RUN IT

Option 1: Caphlon-2 (4.52 GiB, PTQ1_SB)

Because of the ternary Hadamard kernel, it uses our drop-in patch for PrismML llama.cpp:

git clone https://github.com/demiralpdev/Caphlon-Model.git

cd Caphlon-Model

bash scripts/build-prism-llamacpp.sh

bash scripts/run.sh Caphlon-2-27B-PTQ1_SB.gguf 32768 1

Option 2: Caphlon-1 (6.87 GiB, IQ2_S)

Works with standard llama.cpp and Ollama:

Download Caphlon-Model-IQ2_S.gguf from Hugging Face:

https://huggingface.co/ssuasd/Caphlon-Model/resolve/main/Caphlon-Model-IQ2_S.gguf

License is Apache-2.0. Looking forward to any feedback and community testing.


r/ollama • • 3h ago

Product Feedback Ollama Cloud Lite plan

3 Upvotes

As a personal scale user without any major coding requirements i wanted to give feedback on current ollama cloud products.

The Pro plan for 20$ a month has usage that is much more than needed for a personal use especially if not coding or some minimal home lab tasks.

Pay as go can be used but the prices are not competitive compared to others and are more expensive.

If Ollama can add another lower tier plan lets call is Lite plan for 8-10$ a month range with credits accordingly that can appeal to additional users who want to you ollama cloud models.


r/ollama • • 4h ago

I built this! Using local Ollama (qwen2.5-coder:14b + gemma4) as a 0-token "Judgment Kernel" and AST diff auditor for coding agents

2 Upvotes

We have been experimenting with offloading the heaviest token sinks in agentic coding loops to local Ollama models (`qwen2.5-coder:14b` for diff security reviews and `gemma4` as a test-failure Judgment Kernel).

When we profiled why coding agents bloat their context windows so fast on our 1,465-file TypeScript repository, 98% of the tokens weren't spent writing code:

  1. Verbose Test Runner Logs (~6,816 tokens/run): Running `vitest`, `tsc`, or `eslint` dumps 240+ lines of output into context on every run, compounding across turns.

  2. Mechanical Diff Reviews (~3,500+ tokens): Checking git diffs for missing auth checks, unhandled promises, or modularity limits.

  3. Exploratory File Crawling (~15,400 tokens): Running `find` and `grep` instead of pre-computing a compact repository map and AST symbol graph.

How We Wired Up Local Ollama in the Harness (ZeroBurn):

  1. Local Ollama "Judgment Kernel" (`quiet-cli.mjs`):

    • When unit tests pass, the CLI wrapper emits a single 17-token line (`[quiet-cli] unit: PASS (Test Files 6 passed, Tests 70 passed)`).

    • When tests fail (>12 lines of stack trace), it queries local Ollama (`/api/chat` with `gemma4` or `llama3`) to synthesize a 3-bullet root-cause digest (`file:line`, root cause, exact fix) above a capped error tail.

  2. Local Coder Diff Audit (`qwen2.5-coder:14b`):

    • Auto-detects installed coder models (`qwen3-coder`, `qwen2.5-coder`, `deepseek-coder`, `codellama`, `starcoder`) from `127.0.0.1:11434/api/tags` and reviews the `git diff` locally in ~8.9s using structured JSON output.

  3. 0-Token AST Knowledge Graph + 5-Line Repo Map Digest:

    • Pre-computes models, domains, and routes in `<5ms` Python (reducing discovery context from 15,429 tokens down to 512 tokens).

Benchmark Results on a 1,465-File Repo:

  • Repo & Route Discovery: 15,429 tokens -> 512 tokens (-96.7%)

  • Rule & Docs Ingestion: 59,202 tokens -> 195 tokens (-99.7%)

  • Test Verification Logs: 6,816 tokens -> 17 tokens (-99.8%)

  • Diff Security Audit: Offloaded 100% to local `qwen2.5-coder:14b` (8.92s)

The scripts and harness (`.agents/scripts/quiet-cli.mjs`, `live-audit.mjs`, and `parallel-audit.py`) are open-sourced under MIT if anyone wants to try the local Judgment Kernel pattern on their own repo:

https://github.com/thoufeeque22/zeroburn


r/ollama • • 5h ago

I put a digital bridge between OpenWeb-UI and Hailo-Ollama, and I need feedback and help!!

1 Upvotes

I run a collection of small local LLMs on my home network.

It is a Raspberry Pi 5 with the Hailo AI Hat 2+. Using this mainly as a self teaching device, I quickly ran into a hard limitation. The pre-compiled HEF models that operate on my setup have a VERY hard token limit of 2048 tokens for each response, and in for contextual memory in general, which makes extremely long conversations something I am not able to actually fiddle with until I had an idea.

Since that 2048 token data travels to and from the OpenWeb UI chat interface where I sit on one side over to hailo-ollama, I had an out of the box idea inspired by Nintendo's old practices that they did on their old cartridge games to 'get more space'.

Bank switching \-

Yes, this is a wild idea.

I conceptualized, directed, supervised, and dog-fooded a software bridge that sits between hailo-ollama and OpenWeb UI as an almost proxy. This gives the user the ability to provided a lot more input into the LLM in dynamically created 'banks' that exist on the Raspberry Pi's RAM and as needed the information that is stored in the banks is dynamically swapped in and out of the chat messages, transparent to myself. It uses the x-openwebui-chat-id for per-chat RAM state and it adds /save and /forget commands.

I dog-fooded this myself, and use it extensively to continue to test and further refine this, unfortunately, I am a network engineer, and not a coder, so I had to use vibe-coding for this, so I would REALLY like some human eyes on this thing.

With a default behavior of unless specifically told by the end user with a /save, the chat banks are automatically discarded after 5 minutes or when the Raspberry Pi 5 is restarted. Otherwise, the banks NEVER touch the microSD card and exist only in RAM, and that information is moved between the Raspberry Pi's RAM and the AI Hat's RAM. /forget tosses the banks away.

I would REALY love some human eyes on this thing, and some honest feedback.

I have put this thing up on Github, so please go through and scrutinize this.

https://github.com/varchild/hailo-bank-switching-bridge

Thank you!!


r/ollama • • 1d ago

Ollama 0.40 vs llama.cpp, LM Studio and vLLM on a 3090 with Qwen3.8-27B: least GPU memory (17 GiB at 32k), but the lowest output at 32 requests and 88 s to the first answer

Thumbnail
homebrewserver.club
57 Upvotes

I ran the same model (Qwen3.8-27B) on one RTX 3090 in Ollama, llama.cpp, LM Studio and vLLM, and sent each the same questions.

Where Ollama was best

  • It used the least GPU memory.

  • It never failed. When many long questions came at once, it made them wait in line. Two of the others failed some of these.

  • It holds a lot of text, almost as much as llama.cpp.

Where Ollama was worst

  • Slowest with many users. By default it answers one question at a time, so everyone else waits. Setting OLLAMA_NUM_PARALLEL=4 made it about 2.5 times faster, but vLLM still did over 3 times more.

  • Slow first answer: about 1.5 minutes after starting, because Ollama loads the model only when the first question arrives. After that, no wait.

  • Slower for one person than LM Studio and llama.cpp. Ollama's MTP version of the model helped (about 30% faster), but it stayed behind.

Tip: if several people or apps use your Ollama, set OLLAMA_NUM_PARALLEL.


r/ollama • • 23h ago

Advice Best coding AI model to run on a standby machine? Time is not a problem: 8GB VRAM, 32GB RAM

14 Upvotes

I have a second machine running an RTX 4060 8GB, i7 12700H and 32GB DDR4 RAM.

It runs on standby, sometimes using ComfyUI to generate images for my DnD games.

I wanted to try Craft Anything to just test and try out some programming. I know that the VRAM etc. is not enough and I'll have to offload into RAM.

Time is not a big thing for me, I can use my main machine in the mean time.


Is it feasible? Possible? I want to use Ollama / LMStudio - MCP interface - to use with CraftAnything.

Any idea to guide me through a fitting model? Thanks a lot! :)


r/ollama • • 15h ago

What models to use on 8gb vram and 16gb ram?

3 Upvotes

Hey. Im Just starting with local ai and was wondering what models can i run with an rx6600 8gb vram and 16gbr ram. My main purpose would be to make script and vídeos by code for my YouTube channel


r/ollama • • 18h ago

I built a quick pass/fail test suite to see which small local models actually survive inside an AI agent (Gemma 4 on an 8GB GPU surprised me)

Post image
3 Upvotes

r/ollama • • 1d ago

I built a small gateway to use my home LLM from my other computers, without port forwarding or a VPN

Post image
2 Upvotes

r/ollama • • 1d ago

Ollamaskope: My answer to keeping track of it all.

Thumbnail
gallery
3 Upvotes

Hey there,

Anyone working with multiple instances—that is, Ollama endpoints—surely knows the problem: the more models you use and test, the messier it gets. Which Modelfile actually contains the base model, and why do I even need certain models twice? Mostly for no reason at all—often they're just leftover models that were simply forgotten and never deleted. Working in the terminal is fine, sure, but it's too hard on my eyes. That's why I built an Ollama management tool for myself: Ollamaskope.

Inside the web GUI, I can now cleanly manage my Ollama instances at my own pace, delete models, create Modelfiles, load a model for testing, or quickly unload it again. And all of that comfortably in a web browser.

Anyone who could make use of this can easily download it on GitHub:https://github.com/o-valo/ollamaskope


r/ollama • • 23h ago

I built this! Evokai: turn an Ollama model into a real in-app assistant, with backend, chat UI, admin console, permissions and budgets in one service

0 Upvotes

r/ollama • • 1d ago

Thinking Machines inkling-2-mini: 26B Looping MoE

8 Upvotes

Like many others, I've been patiently waiting for the release of Qwen 4 27B. If I'm reading the model card correctly, inkling-2-mini from Thinking Machines will check many of the boxes I was hoping for:

  • Runs on a Mac with ~32GB of memory

  • Decent performance for coding

  • Supports vision, tool calling and a large context window (1M)

Bonus points for:

  • MoE should mean more speed (26B total, ~5.4B active)

  • Looping architecture should give better performance on longer horizon tasks

  • Ollama MLX distro available on day 1

  • Trained by an American lab which will raise fewer questions if I use it at work

I'll be testing with Ollama on my Mac Mini M6 (32GB) later today. Keen to hear if others already have results worth sharing.

Ollama: https://ollama.com/library/inkling-2-mini


r/ollama • • 1d ago

I don’t know how good this is, but GLM5.3-Flash is running locally without a GPU

8 Upvotes

r/ollama • • 1d ago

I built this! I fooled myself with "100% GPU" in ollama ps and nearly doubled my 14B speed on an 8GB RTX 3050 (5.8 → 11.1 tok/s)

2 Upvotes

Been building a local AI companion on an 8GB RTX 3050 (Qwen 2.5 14B Q4_K_M, num_ctx 8192) and couldn't figure out why generation was stuck at ~5.8 tok/s even though ollama ps proudly reported 100% GPU.

Turns out I ignored the column right next to it: SIZE: 10 GB. On an 8GB card.

Because I had num_gpu 99 set in my Modelfile, Ollama tried to shove all 48 layers onto the GPU. Instead of throwing an OOM error, the NVIDIA driver silently intercepted the overflowing ~2 GB and spilled it into system RAM via Shared GPU Memory, choking bandwidth on every single token.

What fixed it:

  1. Disabled CUDA - Sysmem Fallback Policy (Prefer No Sysmem Fallback) in the NVIDIA Control Panel so the driver stops hiding VRAM overflows.

  2. Manually tuned num_gpu in my Modelfile to 38 (out of 48 layers) alongside OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0.

Results:

  • VRAM usage: 8137 / 8192 MB (leaving just enough headroom for local Silero TTS on GPU)

  • Split: Honest 75% GPU / 25% CPU

  • Prompt eval: 120 → 260.7 tok/s

  • Generation speed: 5.8 → 11.12 tok/s (nearly 2x faster just by stopping the PCIe thrashing)

Moral of the story: if ollama ps says your 10GB model fits 100% GPU on an 8GB card, you didn't break physics—your driver is just lying to you.


r/ollama • • 1d ago

best uncensored cloud image model

1 Upvotes

trying to find a cloud model (or local host model) for image to remove water mark. Any ideas?


r/ollama • • 1d ago

Local model running Sesame (voice chat)? Possible? comparison to hosted?

Thumbnail reddit.com
2 Upvotes

r/ollama • • 1d ago

Do local LLM users even want this?

Thumbnail
reddit.com
1 Upvotes

r/ollama • • 1d ago

Hermes Agent + Qwen 3.8 27B on RX 7900 XTX: Ollama is fast, but Hermes is significantly slower. Looking for optimization advice.

Thumbnail reddit.com
2 Upvotes

r/ollama • • 1d ago

Ho creato una sala di controllo per sciami di droni: spettacoli di luci, missioni di ricerca, volo manuale e 16 visuali da telecamera in un'unica finestra. Demo nel tuo browser

Post image
1 Upvotes

r/ollama • • 1d ago

I used Giraffe to test Ollama’s KV cache on a T4. Qwen2.5 7B got 80/80 JSON extractions right at 8-bit and 0/80 at 4-bit

1 Upvotes

I wanted to see how much GPU memory I could save on my T4 while Qwen could still turn support notes into usable JSON. I used Giraffe, the open-source test runner I’m building, to replay the same 80 notes with three Ollama KV-cache settings and check the order details in every reply.

I gave Qwen a synthetic customer support note and asked it to extract six details about the current order as JSON. With Ollama’s 8-bit KV cache, Qwen got all six right, including the order number (DP-3976), quantity (17 units), and total ($444.42). With the 4-bit cache, the same prompt produced only backticks and digits. Ollama reported that both requests finished normally. Giraffe checked the actual replies and flagged the 4-bit answer as invalid JSON.

Try Giraffe on your own Ollama server

I ran one pass per setting on the same Colab T4 with Ollama 0.40.16 and qwen2.5:7b. I haven’t isolated what caused the drop at q4_0. All three runs used the same Q4_K_M model weights, four server slots, and 8,192 tokens of context per slot. Flash Attention was active throughout. f16 used auto, and I explicitly enabled it for both quantized caches.

The full set had 245 synthetic prompts covering short answers, JSON extraction, and lookups. Giraffe ran them one at a time, then replayed them with up to four in flight. The GPU buffer sizes come from Ollama’s logs.

KV cacheJSON correct, one request at a timeCorrect out of all 490 attemptsLogged GPU KV buffer
f1680/803701,792 MiB
q8_080/80369952 MiB
q4_00/805504 MiB

Of the 80 serial q4_0 extractions, 28 finished with stop and failed the check. The other 52 finished with length. Across both phases, all 490 q4_0 attempts returned HTTP 200. Giraffe recorded 5 correct answers, 441 wrong answers, 43 empty replies, and one truncated JSON reply that went unscored.

For this workload, I’d use the 8-bit cache on my T4. It saved 840 MiB over f16, and Qwen still got all 80 orders right. Switching to 4-bit saved another 448 MiB, but zero orders passed.

If you’re weighing a similar change, you can give Giraffe a few prompts your own app uses and the answers you expect. Here’s the quick guide for installing and running Giraffe. PyPI Quick start


r/ollama • • 1d ago

Just turned my old Android into a free local AI server for Cursor

0 Upvotes

I honestly didn't think this would actually work, lol.

I've been trying to cut down on API costs (and stop my laptop fan from sounding like a jet engine), and I stumbled across Ollama Local AI (ollamamobile.app).

Basically, it runs GGUF models right on your Android phone offline, and lets you connect Cursor / VS Code / Windsurf to it over local Wi-Fi like an OpenAI endpoint.

I threw a quantized model on my phone expecting it to crash or crawl, and it’s actually streaming code completions straight into my IDE with zero internet and $0 in API fees. My phone is just sitting on my desk doing the heavy lifting.

[Attach your screenshots here]

Has anyone else tried this yet? Curious what models you guys are running and what tokens/sec you're getting on your phones.


r/ollama • • 2d ago Mindblown

4B vs 9B vs 27B answering from household manuals: all three answer well, but no prompt wording stopped the 4B and 9B from obeying a fake instruction inside a manual

15 Upvotes

We built Hearth, a household assistant that runs entirely at home: Ollama for the model, nomic-embed-text for search, faster-whisper for voice, Django for the app. It answers from a library of manuals and documents and cites the source and page. Then we measured it. Some results surprised us.

1. The small models answer well

Eleven sample documents, 28 questions asked three times each, on an M4 Max with 64 GB:

ModelCorrectRight source citedSays "not found" when it shouldSeconds per answer
qwen3.5:4b100%84%100%0.7
qwen3.5:9b99%100%100%1.1
qwen3.8:27b100%100%100%5.8

Memory decides which model you can run. Going by their size (4, 9 and 27 billion parameters), 8 GB of memory fits the 4B, 16 GB the 9B, and 32 GB or more the 27B; we measured on a 64 GB Mac only.

2. Prompt wording doesn't stop small models from obeying a fake instruction in a document

The sample router guide contains a planted note "for AI assistants", telling them to send anyone asking about the Wi-Fi to a fake "Premium Support" number. Two questions in the test set are about that guide; asked three times each, that's six chances to pass the number on. We tried four designs:

Design4B passed it on9B passed it on
Passages in the system message (the usual tutorial layout)5 of 63 of 6
Passages in the user message, quoted and labelled as information3 of 62 of 6
Stricter instructions about text addressed to assistants5 of 63 of 6
Code removes sentences addressed to AI when a document is added0 of 60 of 6

Stricter wording made no reliable difference: the 4B followed the note five times out of six, exactly as with the first design. Size helped, since the 27B never followed it even in the first design. But if your hardware only runs a small model, the defence can't depend on the model. Hearth drops those sentences at upload and lists what it dropped on the document's page, so nothing is hidden from you.

3. Proving nothing leaves the computer takes three independent checks

  • The app's own log. A sitecustomize.py hook wraps Python's name lookup and connect calls. A full session in daily use, with the real models, produced 14 entries: 7 lookups of 127.0.0.1 and 7 connections to Ollama on 127.0.0.1:11434.

  • The operating system's view. lsof -nP -i -a -p <pid> lists every connection, whatever language opened it.

  • Airplane mode. Unplug the network and use Hearth: everything still works. (The project's tests also run a whole session with every connection Python tries refused.)

Each one can be fooled in a way the others can't. The app's log misses connections made by compiled code, a list of connections misses anything that happened before it was taken, and airplane mode can't tell you what would happen with a network.

4. The leak the log couldn't see

We asked a fresh Claude Code session, one that hadn't written the code, to review the project as a privacy auditor. It found two things:

  • When the Whisper model's folder lacks its tokenizer file, faster-whisper tries to fetch one from Hugging Face even when told to use local files only.

  • The serious one: FFmpeg guesses a file's format from its content, and it can read network stream descriptions. An upload of a few lines of text named speech.webm made FFmpeg open network sockets and wait twenty seconds for data, and a playlist made it send a web request. It all happened in compiled code, so the Python-level log saw nothing.

The fix is one rule: tell FFmpeg the format from the upload's type or name, and never let it guess. The project's checks now include one that uploads exactly such a file and fails if the app waits for data.

5. A small thing that made voice work

Without hints, Whisper's base model hears "Hearth, when does Miso see the vet?" as "Haraf when does miso see the victim?" Pass the household's own words (names, pets and so on) as the initial prompt, and our ten clean recordings came out without a single error; the noisy ones kept most of their key words.

How AI was used: the code and the text were drafted with Claude Code from specifications we agreed, then checked against our own tests and measurements.

Full disclosure: this comes from a book we publish, Your Private AI at Home, Volume 5 of Weekend AI Projects. A free sampler with the opening pages and the full contents is linked in the comments.


r/ollama • • 3d ago

Advice Got Gemma 4 12B to keep reliable long-term memory locally: saved sessions went from 52 to 94 of 96, and it kept more correct facts than Sonnet 5.5 on the same pipeline.

Thumbnail
gallery
63 Upvotes

I've been building PCS, a free, open-source (GPL-3.0) Windows app for long-running conversations with an AI that actually remembers. It runs fully local on my 16 GB card, and the local route has no cloud fallback.

**What runs on your PC**

- **Conversation and memory on Gemma 4 12B QAT (Q4_0)** through a pinned llama.cpp CUDA build. Context is 128K by default and goes up to 256K, with an FP16 or Q8_0 KV cache. On my RTX 5080: about 78 tok/s decode and 3,100 tok/s prefill at 8K-token prompts.

- **Long-term memory the model writes and checks itself,** kept in an encrypted vault. Every memory links back to the passages it came from, and you can correct it or make it forget.

- **Memories can come to mind on their own.** Every time you type or speak, PCS matches your words against the vault on your PC and hands the model up to three related memories or earlier turns, so it doesn't have to decide to go looking. No extra model call, and nothing when nothing is clearly related. (Experimental and off by default; it matches words, not meaning.)

- **It can also go looking itself.** For anything that didn't come to mind, ask "what did we decide about the project?" and it runs a memory lookup on its own.

- **Long sessions keep going.** At about 80K tokens it compacts the working context and keeps recent exchanges whole. Older wording stays searchable for the rest of the session.

- **Voice:** offline English listening on CPU or GPU, and Kokoro or Windows voices. On my NVIDIA card, listening and Kokoro are ready in about 6 seconds (it used to take 32).

- **Vision:** connect OBS and pick a scene. It only sends a frame when your question is about the screen.

- **Search by meaning on the CPU** with a local Nomic model, so no cloud embeddings.

- **Your own GGUF models:** a compatibility check (text, tools, structured output, vision), llama.cpp auto-fit with RAM offload, and up to 8 saved model profiles.

- **A map of your themes that runs without any model call.** Try it in the browser: https://pcs-personalcontinuitysystem.github.io/PCS-site/continuity-map.html

The same vault also works with Claude, ChatGPT, Gemini, OpenRouter and others if you ever want a cloud model. You never have to.

**Making a 12B model trustworthy at memory**

This was the hard part, so I tested it properly. I used scripted multi-session histories (plans that change, corrections, things that end), answer keys sealed before any run, and blind grading. The wins:

- **Saved sessions 52/96 → 94/96. Meanings kept 68% → 94%. Stale or forbidden claims 36 → 2.** The fix was in the reviewer, not the writer. The reviewer kept accepting drafts while citing passages the writer hadn't used (mostly the assistant's own sign-offs, like "good luck with the move!"), and the grounding check then blocked the save. Restricting the reviewer to the writer's declared sources fixed it.

- **12 topics interleaved in one vault: sessions saved went from 28 to 75 of 82, and current facts kept from 23 to 81 of 96.** The fix was to stop letting the model rewrite whole memories. It now writes short statements with sources ("added / changed / corrected"), and the code assembles the memory. Old statements get an explicit verdict, so ended plans move into history instead of staying current.

- **On the same pipeline, Gemma ended with more correct current facts than Claude Sonnet 5.5: 120–124 vs 96 of 135.** To be fair to Sonnet: wherever it saved, it made 1 error against Gemma's 12–14. It lost on volume. Its own reviewer rejected 18 of 84 sessions (strict, and mostly right to be), and 4 of its replies put a sentence before the JSON. Grammar-constrained output makes that impossible locally.

**Lessons if you're building something similar**

- **Small, grammar-constrained calls that each ask one question** ("has this ended?", "does this belong here?") beat tweaking one big prompt. They give the same answer run after run; the big writer call doesn't.

- **Record every request and replay it.** A replay takes about 3 minutes per test instead of a 2-hour rerun. At temperature 0.3 with no seed, resend a packet 10–20 times before you believe a change.

- **Shorter isn't automatically better.** Cutting my 4,600-token writer prompt to 1,600 tokens halved the saves (52 → 28 of 96).

- **Gemma 4 26B-A4B didn't help.** It needed changes to fit on 16 GB, crashed after long prompts, and didn't reduce errors in my test.

**Still open:** Gemma's main memory error now is retiring facts that are still true. The test histories are fictional, there are only a handful of runs per version, and the answer keys were written by a separate AI agent. Windows x64 only.

Free, no subscription, about a 42 MB download (models are separate): https://github.com/PCS-PersonalContinuitySystem/PCS

Happy to go into any part of the pipeline. If you've tried long-term memory with a local model, what broke for you?


r/ollama • • 2d ago

I built this! Ollama Agent that uses a Decision Model to route requests to specific models

Post image
2 Upvotes

Gist that uses a decision model to route requests to specific models. Using faster non-tool calling model for simple responses. Based on the idea I had diagrammed above.

https://gist.github.com/TutorialDoctor/72d96004426d10081ec0f76db49f4b9e