r/LowEndLocalAI • • Sep 10 '26

Benchmark Tiny League - easily compare models that run on potatoes

Thumbnail tinyleague.nunodonato.com
79 Upvotes

I was tired of jumping from tab to tab trying to compare small models. It's especially annoying because not all share the same benchmarks.
So I've built this and have been updating it regularly.

Feedback welcome!


r/LowEndLocalAI • • Aug 23 '26

👋 Welcome to r/LowEndLocalAI - Introduce Yourself and Read First!

44 Upvotes

Local AI often looks like it starts with a 24 GB GPU or a multi-GPU workstation.
This community starts somewhere else:

What useful AI can we run on the hardware we already have?

That might mean a normal laptop, an old desktop, integrated graphics, CPU-only inference, a used GPU, a mini PC, Apple Silicon, a Raspberry Pi, or some wonderfully questionable collection of repurposed parts.

What "low end" means here

There is no fixed VRAM, price, or age cutoff!
"Low end" describes the constraint more than the hardware itself.

If limited compute, RAM, VRAM, bandwidth, power, compatibility, or cost meaningfully affects what you can run and how you run it, your post probably fits.

A 24 GB GPU can fit when the constraint is relevant. A powerful multi-GPU system being shown off simply because it is powerful probably does not.

The point is not to decide who owns sufficiently weak hardware. The point is to make constrained local AI more useful.

What belongs here

  • Benchmarks with useful hardware and software details

  • Model and quantization recommendations

  • CPU, iGPU, shared-memory, and limited-VRAM setups

  • Vulkan, offloading, KV-cache tuning, speculative decoding, MTP, and other optimizations

  • Small and efficient models

  • Practical workflows on slower hardware

  • Old, unusual, repurposed, mobile, or embedded hardware

  • Troubleshooting, guides, experiments, failures, and unexpected successes

LLMs are the main focus, but other local AI is welcome when efficiency or hardware constraints are central to the project.

The important part

Don’t just tell us that a model loads.

Tell us:

Does it actually work well enough to be useful?

Sometimes a huge quant running at 2 tok/s is an impressive experiment. Sometimes a much smaller model running ten times faster is the better tool. Both are worth discussing.

Community resources

Looking for a model recommendation? Please use our Model Recommendation Template so others have enough information to help.

Help build the community

Since this subreddit is new, its first members will have a meaningful influence on what it becomes.

Share your setup. Post benchmarks. Ask strange questions. Test things that probably should not work. Compare a tiny model against a huge quant. Show us the old machine you rescued from a closet and somehow turned into an inference server.

You are also welcome to suggest post flairs, recurring threads, benchmark templates, wiki resources, or community rules. If you are interested in helping with moderation or community resources, feel free to get in touch through modmail.

Community principles

  • Curiosity over hardware flexing

  • Practical usefulness over impressive numbers

  • Constructive advice over “just buy a better GPU”

  • Reproducible results over unexplained benchmarks

  • Honest limitations over hype

  • No shaming people for their budget, hardware, or experience

Share your setup, benchmarks, optimizations, weird experiments, and lessons learned.

Welcome to r/LowEndLocalAI.

Let’s see how much useful AI we can squeeze out of the hardware we already own.


r/LowEndLocalAI • • 10h ago

Discussion Do people still use Ollama as of late 2026?

13 Upvotes

I'm seeing a huge influx of Ollama-based tutorials on my X timeline, and on YouTube recommended algorithms too, but as per my understanding llama.cpp is the standard one right? Followed by newer engines like strata, basalt etc.

Lmk what you guys use for the most part as well as your RAM/VRAM pairings!


r/LowEndLocalAI • • 1d ago

Optimization 8GB VRAM 16GB RAM Qwen3.6 35BA3B 58.5 t/s RustingStrata. I hope someone can benefit for it.

Post image
233 Upvotes

I shared this post in the hope of someone may/could benefit from faster token per second. I found RustingStrata 8 hours ago. Decided to give it a try. It's faster than llama cpp. Normally, I get around 30ts in llama cpp.


r/LowEndLocalAI • • 2m ago

NEWS Inkling-2-Mini Quantized to 5.4 GB and 8.9GB (Apple silicon only)

Thumbnail
huggingface.co
1 Upvotes

r/LowEndLocalAI • • 1d ago

Model Showcase I fine-tuned a 1.5B model into an offline Linux command helper that runs on CPU and checks its own answers against the man pages

94 Upvotes

lxask turns plain English into a shell command, entirely on your own machine:

$ lxask "show disk usage of each folder here, biggest first"

$ du -h --max-depth=1 . | sort -hr

Setup

- Fine-tuned Qwen2.5-Coder-1.5B, Q4_K_M GGUF

- llama.cpp inside one Rust binary. No Ollama, no Python, no API key

- ~1.1 GB on disk, ~1 GB RAM, ~2 s per answer on a laptop CPU

- No GPU, no network after install

How it keeps a 1.5B model honest:

- Retrieves relevant tldr pages and your own man pages before answering

- Checks every program and option in the answer against the man pages

- Retries if a check fails. Anything it still can't confirm is marked `unverified:`

- Asks back on vague requests ("add a proxy": for git? apt? docker?) instead of guessing

Bonus

- `lxask --explain "tar -xzvf logs.tar.gz"` breaks down a command you already have

- Ctrl+G in bash, zsh or fish turns what you typed into the command

Code, weights, data and evals are all open.

GitHub: https://github.com/arxhr007/lxask

Hope someone finds it useful! If you do, consider leaving a ⭐ on GitHub. Feedback and contributions are welcome

Model: https://huggingface.co/arxhr007/lxask


r/LowEndLocalAI • • 23h ago

Discussion thinkingmachines/Inkling-2-Mini (MoE: 26b, a3B)

Thumbnail
huggingface.co
24 Upvotes

Very promising benchmarks with only 3b active parameters. Finally a 35a3b replacement?


r/LowEndLocalAI • • 20h ago

Guide / Tutorial Qwen 27b on 8GB: 200k ctx, 92.5% Top-1 match, within 10% of BF16 on LCB, 17% higher benchmark than 9b

Thumbnail
reddit.com
12 Upvotes

r/LowEndLocalAI • • 1d ago

Model Showcase CCORE-Web-Coder-4B preview just dropped: a 4B local model that writes a full website, slide deck or infographic as a single HTML file

20 Upvotes

r/LowEndLocalAI • • 1d ago

Hardware / Build Everyone around me is getting into local AI

62 Upvotes

Both my roommates have gotten pretty obsessed with local LLMs lately.
One has a Lenovo Legion with an RTX 4080 12GB, the other uses an Acemagic Retro X5 with a Ryzen AI 9 365. They both run local models for work stuff. Apparently they don't like sending company files, emails or code to cloud AI services. Fair enough.
Meanwhile im sitting here with a pretty basic Lenovo IdeaPad 1, Ryzen 3 and 8GB RAM, no dedicated GPU. Tried running Qwen on it a while ago and it was painfully slow. Gave up after a bit. Maybe I picked a model that was too big for the machine.
I get the privacy argument, but buying another computer just to run AI seems like a pretty expensive way to avoid a subscription. And I'm still not sure how much you can actually do with these machines.
The funny thing is my roommates have quite different hardware too. One has an Nvidia GPU, the other is running AMD integrated graphics. Do they get wildly different results, or is having enough memory more important for everyday tasks?
I'm mostly talking about boring work stuff here. Sorting files, summarising long documents, checking emails, writing small bits of code. Can local models handle that reliably without constant tweaking?


r/LowEndLocalAI • • 1d ago

Hardware / Build 8gig more Ram -> 30% more speed

10 Upvotes

strata IQ2_XS:

before: 4x8gb(3600) + 2x 3060-12 + R3600 -> 35 t/s

after: 1x16gb 3x8gb -> 40gb (2111Mhz) -> 48 t/s

so 8gig more and 1500Mhz less Ram speed gave me around 33% more decode speed


r/LowEndLocalAI • • 21h ago

Model Showcase [Release] VAULT-v3-3B-PRIMAL: Full-parameter 3B training in 5.93 GiB state on one 16 GB GPU (no LoRA, no master weights) + true O(1) 100K+ KV memory & freestanding C99 runtime (AVX2 / ARM64 NEON)

Thumbnail
reddit.com
1 Upvotes

r/LowEndLocalAI • • 1d ago

Which LLM Should / Can I Use? Lack of good 2-4b models

42 Upvotes

I love Qwen 3.5 4b. And I appreciate LFM3 models. But there's a serious lack of models like Qwen. It seems everyone has moved on to larger models and have given up on squeezing out what they can for edge models.

Am I missing something or is Qwen 3.5 4b really the only one that is smart enough for me at that size? I like LFM but it's not near Qwen. Nothing has come close at that size.


r/LowEndLocalAI • • 1d ago

Which LLM Should / Can I Use? Looking for Human Advice

4 Upvotes

Looking for some human advice on how best to utilize the following hardware for local AI with the ability to switch the running model for different use cases.

Dell Precision Tower 7810, configured for dual CPU. The CPUs are Xeon E5-2620 v4s, so 16 cores and 32 threads.
I have an RTX 3060 with 12GB VRAM in the system as well as 128GB DDR4 ECC memory.

It was used as a Proxmox server for my homelab for a bit, but I have downsized a bit so that the rest of the homelab sits comfortably on another system.

I do some C# coding, PowerShell, Windows Batch, Linux Bash, and MacOS Bash scripting.

I also want to use it to help me format my fantasy writing which I use for my homebrew D&D campaign world.

Right now, I'm using Llama-Swap with 8B models, but would like to know if I can use larger models for accuracy and such.


r/LowEndLocalAI • • 1d ago

Benchmark Curation vs Compaction for agentic work on budget hardware

10 Upvotes

My private benching strongly suggests that ongoing curation decisively outperforms compaction strategies on <= 8g vram setups. I welcome critical feedback and may be missing a plugin or approach that is competitive with my harness. If so, please describe how it works.

https://github.com/plurnk/plurnk-bench/releases/tag/evidence-2026-11-19-verified-15m-unseen-34-rtx5070-plurnk-vs-pi


r/LowEndLocalAI • • 2d ago Heartwarming Bravo

Benchmark Coding test based off of my real coding problems bench. Best models for a 16 GB card.

Thumbnail
gallery
193 Upvotes

Changes dramatically if thinking is on or off. I notice some people saying some models are better than my experience. wanted to offer some info. One run 10 questions, 5 points each.


r/LowEndLocalAI • • 1d ago

NEWS CCORE-WEB-CODER drops this weekend

13 Upvotes

r/LowEndLocalAI • • 1d ago

Problem / Troubleshooting Exo fails at the last layer when deploying on two nodes

1 Upvotes

Exo fails at the last layer when deploying on two nodes

Hello fellow Local AI enthusiasts,

I have been trying to deploy LLMs to two nodes using pipeline over TCP/UDP

Machine 1 specs:

CPU: Ryzen 5900x

Ram: 32gb

GPU: RTX 4080 super (16 VRAM)

Machine 2 specs:

CPU: Core i7 4770

Ram:16gb

GPU: GTX 1080 (8 VRAM)

both machines are running cachy os

both nodes discover each other normally and when I start a deployment it works fine all the way till the last layer it would get stuck at 16/17 layer

models that I have tried:

mlx community/Nvidia nemotron 3.5 lightning 30b A3B 4bit

mlx community/qwen3.6 35b A3B 4bit

youssofal/qwen3.6 35b A3B 4bit abliterated heretic mlx 4bit

any help or guidance to the right direction would be greatly appreciated

thank you


r/LowEndLocalAI • • 2d ago

Model Showcase Kitten-TTS-2 Pico, Nano, Mini: SOTA super-tiny Apache 2.0 TTS, up to 16 languages (Pico is just 4 MB)

12 Upvotes

New super-efficient TTS models that can run realtime on single smartphone cpu thread too! Up to 16 languages, Apache 2. Seems ideal for resource constrained hardware.


r/LowEndLocalAI • • 1d ago

Workflow / Use Case Upgrade RX 7900 XTX 24 GB

Thumbnail reddit.com
1 Upvotes

Any parameters will help thanks in advance.


r/LowEndLocalAI • • 1d ago

Hardware / Build Which inference engine squeezes max performance out of my setup without sacrificing precision? (Coding + image gen)

Thumbnail reddit.com
1 Upvotes

r/LowEndLocalAI • • 1d ago

Model Showcase [Release] Otaku TTI 1.1: A featherweight, sub-gigabyte anime diffusion model with 4x the spatial resolution

Thumbnail gallery
0 Upvotes

r/LowEndLocalAI • • 1d ago

Which LLM Should / Can I Use? Local LLM recommendations for creative/outside-the-box thinking & personal life advice (RTX 4070 8GB VRAM + 64GB DDR5)

Thumbnail reddit.com
1 Upvotes

r/LowEndLocalAI • • 2d ago

Which LLM Should / Can I Use? best local airport for coding on a steam deck

2 Upvotes

I have ollama and goose installed though I'm not sure which models would be good

any help greatly appreciated cheers