r/LowEndLocalAI • • 2m ago

NEWS Inkling-2-Mini Quantized to 5.4 GB and 8.9GB (Apple silicon only)

Thumbnail
huggingface.co
1 Upvotes

r/LowEndLocalAI • • 10h ago

Discussion Do people still use Ollama as of late 2026?

13 Upvotes

I'm seeing a huge influx of Ollama-based tutorials on my X timeline, and on YouTube recommended algorithms too, but as per my understanding llama.cpp is the standard one right? Followed by newer engines like strata, basalt etc.

Lmk what you guys use for the most part as well as your RAM/VRAM pairings!


r/LowEndLocalAI • • 20h ago

Guide / Tutorial Qwen 27b on 8GB: 200k ctx, 92.5% Top-1 match, within 10% of BF16 on LCB, 17% higher benchmark than 9b

Thumbnail
reddit.com
12 Upvotes

r/LowEndLocalAI • • 21h ago

Model Showcase [Release] VAULT-v3-3B-PRIMAL: Full-parameter 3B training in 5.93 GiB state on one 16 GB GPU (no LoRA, no master weights) + true O(1) 100K+ KV memory & freestanding C99 runtime (AVX2 / ARM64 NEON)

Thumbnail
reddit.com
1 Upvotes

r/LowEndLocalAI • • 23h ago

Discussion thinkingmachines/Inkling-2-Mini (MoE: 26b, a3B)

Thumbnail
huggingface.co
24 Upvotes

Very promising benchmarks with only 3b active parameters. Finally a 35a3b replacement?


r/LowEndLocalAI • • 1d ago

Model Showcase CCORE-Web-Coder-4B preview just dropped: a 4B local model that writes a full website, slide deck or infographic as a single HTML file

20 Upvotes

r/LowEndLocalAI • • 1d ago

Model Showcase I fine-tuned a 1.5B model into an offline Linux command helper that runs on CPU and checks its own answers against the man pages

94 Upvotes

lxask turns plain English into a shell command, entirely on your own machine:

$ lxask "show disk usage of each folder here, biggest first"

$ du -h --max-depth=1 . | sort -hr

Setup

- Fine-tuned Qwen2.5-Coder-1.5B, Q4_K_M GGUF

- llama.cpp inside one Rust binary. No Ollama, no Python, no API key

- ~1.1 GB on disk, ~1 GB RAM, ~2 s per answer on a laptop CPU

- No GPU, no network after install

How it keeps a 1.5B model honest:

- Retrieves relevant tldr pages and your own man pages before answering

- Checks every program and option in the answer against the man pages

- Retries if a check fails. Anything it still can't confirm is marked `unverified:`

- Asks back on vague requests ("add a proxy": for git? apt? docker?) instead of guessing

Bonus

- `lxask --explain "tar -xzvf logs.tar.gz"` breaks down a command you already have

- Ctrl+G in bash, zsh or fish turns what you typed into the command

Code, weights, data and evals are all open.

GitHub: https://github.com/arxhr007/lxask

Hope someone finds it useful! If you do, consider leaving a ⭐ on GitHub. Feedback and contributions are welcome

Model: https://huggingface.co/arxhr007/lxask


r/LowEndLocalAI • • 1d ago

Optimization 8GB VRAM 16GB RAM Qwen3.6 35BA3B 58.5 t/s RustingStrata. I hope someone can benefit for it.

Post image
233 Upvotes

I shared this post in the hope of someone may/could benefit from faster token per second. I found RustingStrata 8 hours ago. Decided to give it a try. It's faster than llama cpp. Normally, I get around 30ts in llama cpp.


r/LowEndLocalAI • • 1d ago

Hardware / Build 8gig more Ram -> 30% more speed

10 Upvotes

strata IQ2_XS:

before: 4x8gb(3600) + 2x 3060-12 + R3600 -> 35 t/s

after: 1x16gb 3x8gb -> 40gb (2111Mhz) -> 48 t/s

so 8gig more and 1500Mhz less Ram speed gave me around 33% more decode speed


r/LowEndLocalAI • • 1d ago

Which LLM Should / Can I Use? Looking for Human Advice

4 Upvotes

Looking for some human advice on how best to utilize the following hardware for local AI with the ability to switch the running model for different use cases.

Dell Precision Tower 7810, configured for dual CPU. The CPUs are Xeon E5-2620 v4s, so 16 cores and 32 threads.
I have an RTX 3060 with 12GB VRAM in the system as well as 128GB DDR4 ECC memory.

It was used as a Proxmox server for my homelab for a bit, but I have downsized a bit so that the rest of the homelab sits comfortably on another system.

I do some C# coding, PowerShell, Windows Batch, Linux Bash, and MacOS Bash scripting.

I also want to use it to help me format my fantasy writing which I use for my homebrew D&D campaign world.

Right now, I'm using Llama-Swap with 8B models, but would like to know if I can use larger models for accuracy and such.


r/LowEndLocalAI • • 1d ago

Problem / Troubleshooting Exo fails at the last layer when deploying on two nodes

1 Upvotes

Exo fails at the last layer when deploying on two nodes

Hello fellow Local AI enthusiasts,

I have been trying to deploy LLMs to two nodes using pipeline over TCP/UDP

Machine 1 specs:

CPU: Ryzen 5900x

Ram: 32gb

GPU: RTX 4080 super (16 VRAM)

Machine 2 specs:

CPU: Core i7 4770

Ram:16gb

GPU: GTX 1080 (8 VRAM)

both machines are running cachy os

both nodes discover each other normally and when I start a deployment it works fine all the way till the last layer it would get stuck at 16/17 layer

models that I have tried:

mlx community/Nvidia nemotron 3.5 lightning 30b A3B 4bit

mlx community/qwen3.6 35b A3B 4bit

youssofal/qwen3.6 35b A3B 4bit abliterated heretic mlx 4bit

any help or guidance to the right direction would be greatly appreciated

thank you


r/LowEndLocalAI • • 1d ago

Hardware / Build Everyone around me is getting into local AI

62 Upvotes

Both my roommates have gotten pretty obsessed with local LLMs lately.
One has a Lenovo Legion with an RTX 4080 12GB, the other uses an Acemagic Retro X5 with a Ryzen AI 9 365. They both run local models for work stuff. Apparently they don't like sending company files, emails or code to cloud AI services. Fair enough.
Meanwhile im sitting here with a pretty basic Lenovo IdeaPad 1, Ryzen 3 and 8GB RAM, no dedicated GPU. Tried running Qwen on it a while ago and it was painfully slow. Gave up after a bit. Maybe I picked a model that was too big for the machine.
I get the privacy argument, but buying another computer just to run AI seems like a pretty expensive way to avoid a subscription. And I'm still not sure how much you can actually do with these machines.
The funny thing is my roommates have quite different hardware too. One has an Nvidia GPU, the other is running AMD integrated graphics. Do they get wildly different results, or is having enough memory more important for everyday tasks?
I'm mostly talking about boring work stuff here. Sorting files, summarising long documents, checking emails, writing small bits of code. Can local models handle that reliably without constant tweaking?


r/LowEndLocalAI • • 1d ago

Benchmark Curation vs Compaction for agentic work on budget hardware

10 Upvotes

My private benching strongly suggests that ongoing curation decisively outperforms compaction strategies on <= 8g vram setups. I welcome critical feedback and may be missing a plugin or approach that is competitive with my harness. If so, please describe how it works.

https://github.com/plurnk/plurnk-bench/releases/tag/evidence-2026-11-19-verified-15m-unseen-34-rtx5070-plurnk-vs-pi


r/LowEndLocalAI • • 1d ago

NEWS CCORE-WEB-CODER drops this weekend

13 Upvotes

r/LowEndLocalAI • • 1d ago

Which LLM Should / Can I Use? Lack of good 2-4b models

42 Upvotes

I love Qwen 3.5 4b. And I appreciate LFM3 models. But there's a serious lack of models like Qwen. It seems everyone has moved on to larger models and have given up on squeezing out what they can for edge models.

Am I missing something or is Qwen 3.5 4b really the only one that is smart enough for me at that size? I like LFM but it's not near Qwen. Nothing has come close at that size.


r/LowEndLocalAI • • 1d ago

Workflow / Use Case Upgrade RX 7900 XTX 24 GB

Thumbnail reddit.com
1 Upvotes

Any parameters will help thanks in advance.


r/LowEndLocalAI • • 1d ago

Hardware / Build Which inference engine squeezes max performance out of my setup without sacrificing precision? (Coding + image gen)

Thumbnail reddit.com
1 Upvotes

r/LowEndLocalAI • • 1d ago

Model Showcase [Release] Otaku TTI 1.1: A featherweight, sub-gigabyte anime diffusion model with 4x the spatial resolution

Thumbnail gallery
0 Upvotes

r/LowEndLocalAI • • 1d ago

Which LLM Should / Can I Use? Local LLM recommendations for creative/outside-the-box thinking & personal life advice (RTX 4070 8GB VRAM + 64GB DDR5)

Thumbnail reddit.com
1 Upvotes

r/LowEndLocalAI • • 2d ago

Model Showcase Kitten-TTS-2 Pico, Nano, Mini: SOTA super-tiny Apache 2.0 TTS, up to 16 languages (Pico is just 4 MB)

12 Upvotes

New super-efficient TTS models that can run realtime on single smartphone cpu thread too! Up to 16 languages, Apache 2. Seems ideal for resource constrained hardware.


r/LowEndLocalAI • • 2d ago

Which LLM Should / Can I Use? best local airport for coding on a steam deck

2 Upvotes

I have ollama and goose installed though I'm not sure which models would be good

any help greatly appreciated cheers


r/LowEndLocalAI • • 2d ago Heartwarming Bravo

Benchmark Coding test based off of my real coding problems bench. Best models for a 16 GB card.

Thumbnail
gallery
193 Upvotes

Changes dramatically if thinking is on or off. I notice some people saying some models are better than my experience. wanted to offer some info. One run 10 questions, 5 points each.


r/LowEndLocalAI • • 2d ago

Problem / Troubleshooting Any suggestions for an rtx5880 ada?

1 Upvotes

So I have a moderate build for local llms: ryzen 7900x3d 96gb ddr5 4800, and an rtx 5880 ada, running local models on LM studio and ollama works fine but the instant I try to run them in vscode as an agent the context length disappears mid-understanding the query on a simple task. Trying the same task in lm studio works fine with tokens to spare. I tried auto compaction doesn't help, I tried forcing a higher output of tokens and input, doesn't help. I am running a qwen 3.8 27b 8q which means I have around 10gb vram to spare. Any suggestions?


r/LowEndLocalAI • • 2d ago

Which LLM Should / Can I Use? Looking for a model that can run on a Pi 4B 4GB and can pretend to be a skeleton dressing as Santa

Thumbnail reddit.com
0 Upvotes