r/LowEndLocalAI • • 10h ago

Discussion Do people still use Ollama as of late 2026?

13 Upvotes

I'm seeing a huge influx of Ollama-based tutorials on my X timeline, and on YouTube recommended algorithms too, but as per my understanding llama.cpp is the standard one right? Followed by newer engines like strata, basalt etc.

Lmk what you guys use for the most part as well as your RAM/VRAM pairings!


r/LowEndLocalAI • • 1d ago

Optimization 8GB VRAM 16GB RAM Qwen3.6 35BA3B 58.5 t/s RustingStrata. I hope someone can benefit for it.

Post image
233 Upvotes

I shared this post in the hope of someone may/could benefit from faster token per second. I found RustingStrata 8 hours ago. Decided to give it a try. It's faster than llama cpp. Normally, I get around 30ts in llama cpp.


r/LowEndLocalAI • • 2m ago

NEWS Inkling-2-Mini Quantized to 5.4 GB and 8.9GB (Apple silicon only)

Thumbnail
huggingface.co
1 Upvotes

r/LowEndLocalAI • • 1d ago

Model Showcase I fine-tuned a 1.5B model into an offline Linux command helper that runs on CPU and checks its own answers against the man pages

94 Upvotes

lxask turns plain English into a shell command, entirely on your own machine:

$ lxask "show disk usage of each folder here, biggest first"

$ du -h --max-depth=1 . | sort -hr

Setup

- Fine-tuned Qwen2.5-Coder-1.5B, Q4_K_M GGUF

- llama.cpp inside one Rust binary. No Ollama, no Python, no API key

- ~1.1 GB on disk, ~1 GB RAM, ~2 s per answer on a laptop CPU

- No GPU, no network after install

How it keeps a 1.5B model honest:

- Retrieves relevant tldr pages and your own man pages before answering

- Checks every program and option in the answer against the man pages

- Retries if a check fails. Anything it still can't confirm is marked `unverified:`

- Asks back on vague requests ("add a proxy": for git? apt? docker?) instead of guessing

Bonus

- `lxask --explain "tar -xzvf logs.tar.gz"` breaks down a command you already have

- Ctrl+G in bash, zsh or fish turns what you typed into the command

Code, weights, data and evals are all open.

GitHub: https://github.com/arxhr007/lxask

Hope someone finds it useful! If you do, consider leaving a ⭐ on GitHub. Feedback and contributions are welcome

Model: https://huggingface.co/arxhr007/lxask


r/LowEndLocalAI • • 23h ago

Discussion thinkingmachines/Inkling-2-Mini (MoE: 26b, a3B)

Thumbnail
huggingface.co
24 Upvotes

Very promising benchmarks with only 3b active parameters. Finally a 35a3b replacement?


r/LowEndLocalAI • • 20h ago

Guide / Tutorial Qwen 27b on 8GB: 200k ctx, 92.5% Top-1 match, within 10% of BF16 on LCB, 17% higher benchmark than 9b

Thumbnail
reddit.com
12 Upvotes

r/LowEndLocalAI • • 1d ago

Model Showcase CCORE-Web-Coder-4B preview just dropped: a 4B local model that writes a full website, slide deck or infographic as a single HTML file

20 Upvotes

r/LowEndLocalAI • • 1d ago

Hardware / Build Everyone around me is getting into local AI

62 Upvotes

Both my roommates have gotten pretty obsessed with local LLMs lately.
One has a Lenovo Legion with an RTX 4080 12GB, the other uses an Acemagic Retro X5 with a Ryzen AI 9 365. They both run local models for work stuff. Apparently they don't like sending company files, emails or code to cloud AI services. Fair enough.
Meanwhile im sitting here with a pretty basic Lenovo IdeaPad 1, Ryzen 3 and 8GB RAM, no dedicated GPU. Tried running Qwen on it a while ago and it was painfully slow. Gave up after a bit. Maybe I picked a model that was too big for the machine.
I get the privacy argument, but buying another computer just to run AI seems like a pretty expensive way to avoid a subscription. And I'm still not sure how much you can actually do with these machines.
The funny thing is my roommates have quite different hardware too. One has an Nvidia GPU, the other is running AMD integrated graphics. Do they get wildly different results, or is having enough memory more important for everyday tasks?
I'm mostly talking about boring work stuff here. Sorting files, summarising long documents, checking emails, writing small bits of code. Can local models handle that reliably without constant tweaking?


r/LowEndLocalAI • • 1d ago

Hardware / Build 8gig more Ram -> 30% more speed

10 Upvotes

strata IQ2_XS:

before: 4x8gb(3600) + 2x 3060-12 + R3600 -> 35 t/s

after: 1x16gb 3x8gb -> 40gb (2111Mhz) -> 48 t/s

so 8gig more and 1500Mhz less Ram speed gave me around 33% more decode speed


r/LowEndLocalAI • • 21h ago

Model Showcase [Release] VAULT-v3-3B-PRIMAL: Full-parameter 3B training in 5.93 GiB state on one 16 GB GPU (no LoRA, no master weights) + true O(1) 100K+ KV memory & freestanding C99 runtime (AVX2 / ARM64 NEON)

Thumbnail
reddit.com
1 Upvotes

r/LowEndLocalAI • • 1d ago

Which LLM Should / Can I Use? Lack of good 2-4b models

42 Upvotes

I love Qwen 3.5 4b. And I appreciate LFM3 models. But there's a serious lack of models like Qwen. It seems everyone has moved on to larger models and have given up on squeezing out what they can for edge models.

Am I missing something or is Qwen 3.5 4b really the only one that is smart enough for me at that size? I like LFM but it's not near Qwen. Nothing has come close at that size.


r/LowEndLocalAI • • 1d ago

Which LLM Should / Can I Use? Looking for Human Advice

4 Upvotes

Looking for some human advice on how best to utilize the following hardware for local AI with the ability to switch the running model for different use cases.

Dell Precision Tower 7810, configured for dual CPU. The CPUs are Xeon E5-2620 v4s, so 16 cores and 32 threads.
I have an RTX 3060 with 12GB VRAM in the system as well as 128GB DDR4 ECC memory.

It was used as a Proxmox server for my homelab for a bit, but I have downsized a bit so that the rest of the homelab sits comfortably on another system.

I do some C# coding, PowerShell, Windows Batch, Linux Bash, and MacOS Bash scripting.

I also want to use it to help me format my fantasy writing which I use for my homebrew D&D campaign world.

Right now, I'm using Llama-Swap with 8B models, but would like to know if I can use larger models for accuracy and such.


r/LowEndLocalAI • • 1d ago

Benchmark Curation vs Compaction for agentic work on budget hardware

10 Upvotes

My private benching strongly suggests that ongoing curation decisively outperforms compaction strategies on <= 8g vram setups. I welcome critical feedback and may be missing a plugin or approach that is competitive with my harness. If so, please describe how it works.

https://github.com/plurnk/plurnk-bench/releases/tag/evidence-2026-11-19-verified-15m-unseen-34-rtx5070-plurnk-vs-pi


r/LowEndLocalAI • • 2d ago Heartwarming Bravo

Benchmark Coding test based off of my real coding problems bench. Best models for a 16 GB card.

Thumbnail
gallery
193 Upvotes

Changes dramatically if thinking is on or off. I notice some people saying some models are better than my experience. wanted to offer some info. One run 10 questions, 5 points each.


r/LowEndLocalAI • • 1d ago

NEWS CCORE-WEB-CODER drops this weekend

13 Upvotes

r/LowEndLocalAI • • 1d ago

Problem / Troubleshooting Exo fails at the last layer when deploying on two nodes

1 Upvotes

Exo fails at the last layer when deploying on two nodes

Hello fellow Local AI enthusiasts,

I have been trying to deploy LLMs to two nodes using pipeline over TCP/UDP

Machine 1 specs:

CPU: Ryzen 5900x

Ram: 32gb

GPU: RTX 4080 super (16 VRAM)

Machine 2 specs:

CPU: Core i7 4770

Ram:16gb

GPU: GTX 1080 (8 VRAM)

both machines are running cachy os

both nodes discover each other normally and when I start a deployment it works fine all the way till the last layer it would get stuck at 16/17 layer

models that I have tried:

mlx community/Nvidia nemotron 3.5 lightning 30b A3B 4bit

mlx community/qwen3.6 35b A3B 4bit

youssofal/qwen3.6 35b A3B 4bit abliterated heretic mlx 4bit

any help or guidance to the right direction would be greatly appreciated

thank you


r/LowEndLocalAI • • 2d ago

Model Showcase Kitten-TTS-2 Pico, Nano, Mini: SOTA super-tiny Apache 2.0 TTS, up to 16 languages (Pico is just 4 MB)

12 Upvotes

New super-efficient TTS models that can run realtime on single smartphone cpu thread too! Up to 16 languages, Apache 2. Seems ideal for resource constrained hardware.


r/LowEndLocalAI • • 1d ago

Workflow / Use Case Upgrade RX 7900 XTX 24 GB

Thumbnail reddit.com
1 Upvotes

Any parameters will help thanks in advance.


r/LowEndLocalAI • • 1d ago

Hardware / Build Which inference engine squeezes max performance out of my setup without sacrificing precision? (Coding + image gen)

Thumbnail reddit.com
1 Upvotes

r/LowEndLocalAI • • 1d ago

Model Showcase [Release] Otaku TTI 1.1: A featherweight, sub-gigabyte anime diffusion model with 4x the spatial resolution

Thumbnail gallery
0 Upvotes

r/LowEndLocalAI • • 1d ago

Which LLM Should / Can I Use? Local LLM recommendations for creative/outside-the-box thinking & personal life advice (RTX 4070 8GB VRAM + 64GB DDR5)

Thumbnail reddit.com
1 Upvotes

r/LowEndLocalAI • • 2d ago

Which LLM Should / Can I Use? best local airport for coding on a steam deck

2 Upvotes

I have ollama and goose installed though I'm not sure which models would be good

any help greatly appreciated cheers


r/LowEndLocalAI • • 2d ago

Which LLM Should / Can I Use? I did it

Thumbnail reddit.com
11 Upvotes

I've done it. I run Qwen3.5:9b and google/gemma-4-E4B-it-qat-q4_0-gguf. I tried Qwen first, but i didn't enjoy it. After "talk" to them a little, i tried to run "Claude code". I don't know if i did something stupid or the models are just dumb or something didn't do what is supposed to do.

I tried something simple, just read and edit a file. i dont know if i was doing something wrong but the Qwen3.5:9b just dosent do nothing. no read. no edit, nothing.

apologize for bad english.


r/LowEndLocalAI • • 2d ago

Benchmark Sharing my results [2700x][1070ti][8GB VRAM][32GB]

14 Upvotes

Running Qwen3.6-35B-A3B-UD-IQ3_S.gguf on Ryzen 2700x+1070ti, windows 11, using llama.cpp, with the following settings:

--repeat-penalty 1.0 --presence-penalty 1.5 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --temp 1.0 --top-p 0.95 --min-p 0.05 --top-k 20 -ub 1024 -b 1024 -np 1 -t 8 -ncmoe 30 --moe-cache-mib 1088 --moe-cache-cpu-share 0 --load_mode none

Real workload speed:

319.20.001.767 I slot print_timing: id 0 | task 232409 | prompt eval time = 1807.51 ms / 457 tokens ( 3.96 ms per token, 252.83 tokens per second)

319.20.001.779 I slot print_timing: id 0 | task 232409 | eval time = 79263.07 ms / 1981 tokens ( 40.03 ms per token, 24.98 tokens per second)

It's not quite polystrata speeds, but it's pretty good. Polystrata seems a bit less stable, I've had a couple of BSODs

Does decently on generating python, although it's not exactly _fast_. Of course, Q3 is a bit on the low side.

Not sure I had a point. Except, decent speeds for an almost decade old machine.