r/LowEndLocalAI • • 16d ago

Benchmark [8GB VRAM | 16GB RAM] Best local QnA models for low-end hardware

Post image
328 Upvotes

For a while now I've been trying to solve the issue of having to spend days or weeks using an LLM before finding out it has been unreliable and unhelpful the entire time. The problem is that small LLMs are usually good at pretending that they're smart and are telling the truth even if they aren't, so I've set out to benchmark LLMs that fit on my low-end hardware to see what models are worthwhile. As always I'm sharing the results of the benchmarks with you guys to help you find neat local assistants :)

The Results

A quick explanation of these graphs:

  1. First one is the score heatmap, rows are models and columns are questions. Brighter squares mean the LLM answered that question well, darker squares mean the opposite.

  2. Second one is an ordered visualization of each LLMs total score on the benchmark. Self explanatory.

  3. Third one is an ordered visualization of each LLMs average time-to-completion, the less the better. Note that this is not the LLM tokens/second speed, it is the average amount of time it takes the LLM to finish the response.

  4. Fourth one is a combined visualization, I call it "intelligence-per-second". It combines the score and speed metric (scoreSum * (averageTTC / 2 + 0.5)) and shows which models give you the most bang for your buck.

  5. Fifth and final one is just a line-graph of both the intelligence and TTC, sorted from lowest to highest score.

Benchmark Info

Some important info about the benchmark itself:

  1. It aims to benchmark relatively small models (1B to 35B, as you can see in the benchmark) and GPT-OSS and Qwen3.6 both got the max score possible.

  2. The "(native)" next to an LLM name means the quant used in the benchmark is the sole quantization provided by the creator. For example Bonsai 27B is in Q1_0 not because of system constraints but because that is the only quantization that Bonsai is available in.

  3. The questions were created by Claude Fable 5.1 and while it is one of the smartest models it is still an LLM testing an LLM, so take it as you will. I still think it did a better job than I did in the first version of the benchmark.

  4. The benchmark is NOT to check which model is best for agentic coding. My use case for these models (and probably anyone else on similar hardware) is as a general assistant for asking questions and receiving answers. Don't run to the comments to complain when GPT-OSS breaks your codebase because you deployed it just because it was at the top of my benchmark.

  5. The models were benchmarked with THINKING DISABLED because otherwise the models would either hit token limitations or (if token limits are disabled) take hours to finish the benchmark. Many people disliked this, but here is my reasoning for why this makes sense: while I believe thinking mode is useful in practice I don't think it matters much for this benchmark: the purpose of this benchmark is to check what model is the "best", not "what is the highest score each model can reach". I believe a model that can solve a task without thinking is still more capable than a model that can solve a task only with thinking enabled.

My Takeaways

  • Qwen3.6 35B A3B did not surprise me with its score, it did great and I feel like that reflects the actual experiance of using it. It is the only model I can have a proper conversation with in my native language (albeit with a few errors). Only problem is its size, it is the most overfitted model on my system and running it not only requires me to not do anything else on the system but also it runs relatively slowly. Not to mention that it is the most heavily quantized model in this benchmark (apart from Bonsai, but that one is QAT).

  • GPT-OSS 20B A4B also did not surprise me, it is one of my favorite models for local use. It's a very neat model because its decently smart, is extremely fast and the shared RAM usage is relatively low (it only takes 12.4 GB and with 7.76 GB offloaded to GPU it only uses 5 GB of my RAM). Only problem for me is that it's surprisingly lazy, in all of my attempts to get it to code something it would always do the bare minimum, so it won't make any fancy graphics even if you ask. However it did assist me with the creation of these graphs.

  • Nemotron3.5 Bolt E4B and Gemma4 E4B are both extremely similar but Nemotron did slightly better on the benchmark score and speed. Very minimal upgrade but sure, I appreciate it.

  • Qwen3.5 9B is one of my go-to low footprint models for most tasks. I usually run it on the side while doing work in Unity. Great speed, decent intelligence. Was pleasantly surprised with its benchmark score. Can't go wrong with Qwen.

  • Ministral3 14B showed pretty decent performance but for this hardware I feel like it is too slow to be worthwhile, better off going with GPT-OSS.

  • MiniCPM5-2B is not the smartest model but considering its size it is pretty impressive. Definitely recommend it if you're on extremely limited hardware.

And now, for the models I did not like: Gemma4 12B QAT sounded promising but its significantly slower and more hungry than the E4B variant while getting a relatively similar score, additionally I just don't like the tone of Gemma models. Ornith1.5 9B is hyped by literally everyone but its significantly worse than Qwen3.5, which I assume was its base model, and it didn't do much well on agentic coding either so I don't really know why everyone likes it so much. LFM2 24B A2B also did horribly despite its size. Finally, the GLM Flash models did absolutely terribly. During evaluation the 4.6v had a golden streak on some complicated questions but then messed up very simple ones. To be fair I haven't tried benchmarking the non-REAP 4.7 model but from my experiance the prefill times were terrible and it didn't perform much better than the REAP version in practice.

TL;DR

Qwen3.6 35B A3B for very complex tasks locally where time or resources doesn't matter.
GPT-OSS 20B A4B is a great option for most tasks.
Qwen3.5 9B is great for when you want answers quick.
Nemotron3.5 Bolt E4B and Gemma4 E4B are great for when you want some web-search summaries with exceptionally low footprints.
Ornith, LFM2 and GLM Flash did not seem worthwhile.

The Repo

If you want to run this benchmark yourself or want to see the questions you can check out the repo for this benchmark:

https://github.com/Scorpy-37/Bluemark


r/LowEndLocalAI • • Oct 01 '26

Workflow / Use Case Don't sleep on Qwen 3.8 Flash Next with 12GB VRAM

295 Upvotes

As a previous user of Qwen 3.6 35B A3B (Specifically KAT Coder v2.5), I'd been disappointed, but content with the work I could get out of it for agentic coding workflows.

I tried Qwen 3.8 27B but the speed was just horrendously slow. Even overnight tasks weren't completing, generating horrible 6t/s at best.

I saw that post the other day about Qwen 3.8 Flash Next running on 12GB VRAM using the Strata inference engine, which basically profiles your system and gets the model running as fast as it can.

Gents, I'm running IQ3_XXS on a 12GB 4070 Super and 64GB of DDR5 and this shit is bananas. Night-and-day difference in Pi Coder. Sure, the thinking uses more tokens, but it actually solves problems and can think at a much higher level than 35B A3B could. Q5 on Kat Coder via llama.cpp gave me roughly 600pp and 30-40tg/s. Qwen 3.8 Flash on Strata is giving roughly 200t/s prefill, but 40-60t/s decode.

Strata even recommends IQ2 if you're just doing coding tasks and could probably get away with using less VRAM.

I guess this is an appreciation post as well as a "you can do it" encouragement for anyone out there looking for something better.

Edit: I am currently running pi-vcc to significantly reduce compaction time, and I'm running 160k context, up from 131k with a tiny hit to performance. 256k is totally doable but you'll need more system memory.


r/LowEndLocalAI • • 1d ago

Optimization 8GB VRAM 16GB RAM Qwen3.6 35BA3B 58.5 t/s RustingStrata. I hope someone can benefit for it.

Post image
232 Upvotes

I shared this post in the hope of someone may/could benefit from faster token per second. I found RustingStrata 8 hours ago. Decided to give it a try. It's faster than llama cpp. Normally, I get around 30ts in llama cpp.


r/LowEndLocalAI • • 24d ago Heartwarming

Model Showcase I trained a 4B model to generate complex 3D scenes with Three.js

225 Upvotes

Hey everyone!

I've been working on an experiment to see how far a small 4B model can go when trained for a specific coding task.

I chose Three.js because the results are easy to visualize and evaluate. Starting from Qwen3.5-4B, which struggled with complex 3D scenes in my initial tests, the model can now generate detailed architecture, mechanical structures, animations, and interactive environments as standalone HTML/JavaScript files.

The goal was to create something people could run locally on consumer hardware, while exploring small models as a proof of concept for other specialized coding tasks.

The model, full FP16 weights, dataset, and technical report are now available!

Model: Qwen3.5-4B-3D-Coder

  • Q8: Recommended and tested.

  • Q4: Available, but I haven't tested the latest release yet.

  • FP16: Full weights available in Safetensors format.

  • Q8 settings: Temperature 0.6, top-k 20, top-p 0.95, repetition penalty 1.0.

Important: If you downloaded an earlier version, delete it and download it again. The filenames haven't changed. In LM Studio, also disable MTP.

The model also shows promising results generating 3D scenes from image references, even though the vision encoder wasn't included in this training run.

Dataset: CCORE Three.js 3D Scenes

Released under the MIT License, it contains 4,146 examples — 3,933 training and 213 validation — including prompts, full thinking traces, complete HTML/JavaScript responses, complexity levels, and optional rendered reference images.

Technical report: Read the report here

Not every generation is perfect, and some scenes still need another attempt or a correction. But one-shot results and visual quality have improved substantially in my tests.

I'd love to see people experiment with the model on lower-end GPUs or use the dataset to train their own models. If you try it, please share your results!

Thanks everyone for the support, feedback, and testing!


r/LowEndLocalAI • • Sep 28 '26

Model Showcase I improved the best long-context agentic coder for 4GB VRAM.

Post image
195 Upvotes

It's insane how good small models have gotten this past half year! MiniCPM5-2B is probably the best 2B right now, and specifically for agentic software engineering it hits way above its size class. So I took a look at what I could do to improve it even further, and came up with a dynamic quantization strategy, an agentic coding centered imatrix, and a bunch of fixes and improvements to the chat template. You get a net improvement on all important metrics: A smaller Q6 quant gets more solves faster compared to the shipped Q8. At Q4_K_S you get better fidelity at smaller size compared to the stock Q4, due to my dynamic imatrix quantization.

This model is great as a background worker that doesn't eat your RAM, or a main agentic coder model for small old GPUs.

https://huggingface.co/peculiar-ragdoll/Sharp-MiniCPM5-2B-GGUF


r/LowEndLocalAI • • 2d ago Heartwarming Bravo

Benchmark Coding test based off of my real coding problems bench. Best models for a 16 GB card.

Thumbnail
gallery
193 Upvotes

Changes dramatically if thinking is on or off. I notice some people saying some models are better than my experience. wanted to offer some info. One run 10 questions, 5 points each.


r/LowEndLocalAI • • 10d ago

Discussion How I built a genuinely autonomous local coding loop that compiles, tests, and repairs its own code (and why VRAM isn't the bottleneck people think it is)

Post image
181 Upvotes

I’ve been heads down building a native desktop software workstation called Repetere AI, and I wanted to break down how the autonomous engineering engine inside it—Deep Think—actually works under the hood. There is a lot of noise online right now about "autonomous vibe coding," but 99% of what people are showing off is just a web wrapper sending prompts to a paid cloud API. When people try to run coding models locally, they usually hit a wall where the model hallucinates an import, outputs broken diff markers, and leaves the user playing babysitter in a chat box.

I wanted something completely air-gapped, zero-dependency, and truly autonomous on consumer hardware.

The entire engine runs as a native C++ Win32 application built on Dear ImGui and OpenGL. Rather than acting as a simple chat interface, it operates as a five-phase execution harness that manages the local inference server directly. When you give it a directive, it starts by ingesting the local repository and synthesizing an architectural plan. From there, it moves into code synthesis.

One of the biggest issues with local models writing code across multiple files is that standard search-and-replace regex patchers are extremely brittle. If a model misses a newline or changes indentation slightly, the patch fails. To solve this, I integrated a Python AST script splicer directly into the loop. It parses the actual syntax tree of the code rather than relying on raw string diffs, which allows surgical module patching without blowing through context windows or mangling files.

Where the system really separates itself from standard local frontends is in the verification and repair phases. Once the code is written, the engine doesn't ask you if it looks good. It immediately takes over the machine's toolchain in a headless subprocess. If it's a native C++ target, it runs CMake and MSVC. If it's Python, it runs compilation checks and fires an automated headless smoke probe using runpy that sweeps the window hierarchy, tests widget lifecycle, and catches runtime crashes before a human ever looks at the window.

If a build fails or an unhandled exception is caught during the probe, the engine doesn't stop or dump an error message into a chat window. It gathers the compiler diagnostics, strips out runtime harness noise, isolates the failing file, and sends the telemetry directly back to the active resident model for in-loop auto-triage. It can diagnose its own missing imports, patch uninitialized attributes, and re-verify the build automatically until it gets a clean exit code.

I also run into people claiming that anything under huge enterprise cards or massive unified memory pools is useless for real local autonomy. That completely misses the difference between raw capacity and execution architecture. A quantized model has the exact same mathematical weights whether it sits on a 12GB laptop GPU, a multi-card workstation, or a cloud cluster. VRAM determines how big of a model you can hold on-chip and how many tokens per second you generate, but it doesn't give a model magic reasoning skills. A massive model will still hallucinate a typo or botch an import. The difference is that a raw model fails permanently when it makes a mistake, whereas an autonomous harness catches the compiler error, fixes the AST, and verifies the output on its own.

On the hardware side, the backend doesn't lock you into a single card or even a single vendor ecosystem. It enumerates all discrete compute adapters, aggregates the physical memory, and calculates proportional tensor split ratios on the fly for both CUDA and Vulkan runtimes. If you run a multi-GPU setup, it shards the layers across whatever cards are plugged into the board. If you run a hybrid setup that offloads across system RAM, the pipeline scales across host memory just as cleanly.

The takeaway from building this is that real local coding autonomy doesn't come from throwing infinite hardware at zero-shot prompts or paying for API subscriptions. It comes from building a tight runtime loop that enforces architectural contracts, executes the compiler headless, and forces the model to fix its own mess.

Edit: thank you guys! The reddit is /r/repetereai , website is https://www.repetereai.com . LINK to working open beta .9 is on my github on the website, if you want to try an earlier build!


r/LowEndLocalAI • • 17d ago

Unusual Hardware My Xbox One S is now a local LLM server (LFM3-1.7B, Direct3D 12, 29.5 tok/s)

Post image
171 Upvotes

La mia Xbox One S del 2016 ora è un server LLM locale

Pithos un motore di inferenza LLM in C puro senza dipendenze. Funziona sulla mia Xbox One S.

Esegue LFM3-1.7B sulla GPU della console tramite Direct3D 12 e fornisce un'API compatibile con OpenAI e una chat web a tutti i dispositivi di casa. Solo modalità sviluppatore, senza jailbreak.

Sul mio laptop lo stesso binario esegue LFM2.5-24B, un modello più grande della mia RAM libera, e batte llama.cpp CUDA sul mio computer.

Numeri, benchmark e funzionamento sono nel repository: https://github.com/benmaster82/Pithos

Sarò lieto di rispondere alle vostre domande.


r/LowEndLocalAI • • 4d ago

Model Showcase 4B on par with Qwen 3.6 35BA3B - Spark X2.6 4B

162 Upvotes

Was advised to share this here after sharing it elsewhere (and I'm a big fan of AI on potato PCs): https://huggingface.co/XHToken/Spark-X2.6-4B

From my own comparative testing, and what other comparisons I can find on BenchLM and TInyLeague, this is roughly on par with Qwen 3.6 35BA3B. But it's a 4B dense so can fit in ~3gb at a decent quant.

I've ran a bunch of tests on different quants and the Quality-36pc quant from here seems to be excellent: https://huggingface.co/Soulfate24/Spark-X2.6-4B-DSpark-Paretrix-GGUF

No observable difference from BF16 on a a large corpus of Polyglot questions, it in fact outperforms BF16 though the difference is within noise at my sample size. The reduced vocabulary (32k) DSpark draft head from that page is also smaller than the official one and results in higher throughput.

I currently get 50-60TPS on a 7840HS/780M laptop, currently playing with AMD Loom to see if I can push this higher.

Note, not affiliated with XHToken in any way, just mirin' the model.


r/LowEndLocalAI • • 11d ago

Benchmark I ran a small coding test on a few AI models with <10B. Surprising results.

Post image
156 Upvotes

I ran a quick coding test on a few models. I used Q5_K_M on Ling and Qwen2.5 because I thought that, since one is a MoE model and the other is an Instruct model, quantization to Q4 would limit them more (but apparently that’s not the case). I used --temp 1.0 on the Qwen (and Ornith) models because I read that this is the value recommended by their developers. On Ling, I used only 0.5 because in other tests I’ve done, that’s the value that gave me the best results; on Spark, I used both. Obviously, it's a very small test, but I expected Ornith and Spark to do better. What do you guys think about that?


r/LowEndLocalAI • • 11d ago Heartwarming

Optimization I made a 27b model run on GTX 1070 8GB at 96k context

Thumbnail
github.com
147 Upvotes

Hey, I've been working on Pascal specific kernels here and there, and managed to boost performance significantly for Bonsai 2 27b PTQ1 (+50–70% decode and ~+49% pp) and qwen3.5-9b-ish models. Decided to share it :)


r/LowEndLocalAI • • Oct 20 '26

Discussion Strata is like crack

139 Upvotes

I had 9t/s on llama.cpp, with strata, I get about 29t/s, beyond nuts.
4060 8GB + 64GB DDR4 and Qwen 3.8 Flash Next Q3

This is like day and night, now I can do some stuff :D


r/LowEndLocalAI • • 9d ago

Unusual Hardware One engine for streaming any MoE at any quant, with an automatic planner: phones, laptops, consumer and data-center GPUs: The viral voxel Pagoda locally on a 2020 old ThinkPad, no GPU, 16 GB RAM

133 Upvotes

Hi all,

in the last months I worked on BigMoeOnEdge, it's built on top of llama.cpp. It runs MoE models that are bigger then the memory of the device, and it configure itself for the machine where it runs.

It works with any MoE GGUF that llama.cpp can load, any quantization, no code per model.

OS: Android, iOS, macOS, Windows, Linux (arm64 and x86_64).

Backends: CPU, CUDA, ROCm (HIP), Vulkan, Metal, OpenCL, Hexagon NPU. Other llama.cpp backends load in the same way if you build them.

Todat I tried the voxel pagoda prompt fully local, on a laptop that shouldn't run a 35B model.

Setup

  • ThinkPad from 2020, Ryzen 7 PRO 5850U, 16 GB RAM, no GPU, Windows

  • Qwen3.6-35B-A3B Q4_K_M, 21 GB, bigger than the RAM

  • BigMoeOnEdge, built-in agent

Result

  • HTML + Three.js, 12k tokens at 2-3 tok/s, about 1.5 h

How

  • MoE: each token uses only a few experts

  • Dense weights and a 2 GB expert cache stay in RAM, the rest streams from the SSD

  • On top of llama.cpp, no fork

Still slow

  • Half the time is spent waiting on the SSD

Same test with the same model on a Mac with 16 GB of RAM, taking 20 minutes (20 tok/s for decoding, 100–1000+ tok/s for prefill)


r/LowEndLocalAI • • 19d ago

NEWS A tiny 4B coding model that creates 3D assets for Godot and other game engines [GLB-3D-Coder]

128 Upvotes

Hey everyone!

I recently released GLB-3D-Coder, a tiny 4B model based on Qwen3.5-4B, designed to generate reusable 3D assets for games and other applications.

The idea is pretty simple. You describe what you want, like a low-poly potion, a medieval building, a weapon, or a small environment, and the model generates a standalone HTML file using Three.js, with a button to export the asset as GLB.

You can then import the generated assets into Godot, Blender, Unity, or other engines that support GLB.

And the interesting part is that it's only 4B!

I've already released Q4_K_M and Q8_0 GGUF versions, so you don't need a high-end GPU to experiment with it. The Q4 version is around 2.7GB, although actual memory requirements depend on the context size, and generating complex assets can take quite a few tokens.

Model: https://huggingface.co/ccore/Qwen3.5-4B-GLB-3D-Coder

You can find the recommended system prompt on the model page. In my tests, simple requests like "create a low-poly mana potion" have worked pretty well.

My current sampling settings are temperature 0.8, top-k 40, top-p 0.95, and min-p 0.

It's still an early release, so some generations may fail or need small corrections, but the results have been pretty interesting so far.


r/LowEndLocalAI • • 5d ago

Discussion BigMoeOnEdge outperforms Strata and colibri on 8-16 GB cards and Mac and you can run any model

123 Upvotes

Hi!

for those who might not know the solution:

- it takes any MoE GGUF llama.cpp can load, whatever the quant. No repacking, no special format

- it runs on whatever you have: NVIDIA, AMD, Intel, Mac, a phone, or just the CPU

- Windows, Linux, macOS, Android, iOS

- output is the same as llama.cpp, don't cut corners on quality

- you start it with `--auto` and it works out the settings for your machine

Below is a brief overview of the benchmark:

- 8 GB card, Qwen3.6-35B: I get about 35 tok/s. colibri around 23. Strata wouldn't start at 16k context

- 12 GB card, same model: about 45 for bmoe, about 30 for Strata

- long prompt on the 8 GB card: bmoe 5 s to the first token. Strata 14 s, colibri close to 2 minutes

- Basalt needs an RTX 50, so I couldn't run it on these

- 16 GB MacBook, Qwen3.6-35B: about 20 tok/s for bmoe, colibri's release 3.5

- same Mac, Qwen3.8-Flash-Next 125B: 10 tok/s, Strata about 7

On the cards Strata is tuned for it's still ahead but only by a little.

On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.

Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge

If you have an 8 to 32 GB card I'd like to see what you get.

Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data


r/LowEndLocalAI • • Oct 03 '26

Model Showcase Rei: ~370k params LM living inside a Game Boy Color

109 Upvotes

r/LowEndLocalAI • • 3d ago

NEWS I built my own LLM inference engine for Intel Arc GPUs: Qwen 35B runs 851 tokens/sec on an ordinary Core Ultra laptop (open source)

105 Upvotes

For the past few months, I've been working on inferzero, an LLM inference engine written from scratch specifically for Intel Arc GPUs, both integrated GPUs in laptops and discrete cards. It’s not a fork of llama.cpp. The engine is written in C++ directly on top of Level Zero, with the kernel in OpenCL C, and users don’t need Python at all.

It all started from curiosity: my laptop is a Core Ultra 5 125H with 32 GB of RAM, without a discrete GPU, and I wanted to see how far its integrated Arc GPU could be pushed. Turns out pretty far:

* Qwen3.6 35B-A3B: 8K token prompt at 851 tokens/sec on that laptop, with the expert streamed from an SSD. A context of 131K tokens was read in 6.7 minutes.

* Qwen3.8 27B runs on the same laptop, as does Flash Next with 52 GB of expert data.

* Text generation is assisted by MTP and DFlash2, so it still feels responsive on the integrated GPU.

Now the same engine also runs on discrete cards and multi-GPU setups. Here are some numbers from users on GitHub for Qwen3.8 27B with a 4K prompt unless otherwise stated:

|Hardware|Prompt (tok/s)|

|:-|:-|

|Core Ultra 9 285H (Arc 140T)|357|

|Core Ultra X7 358H (Arc B390, Panther Lake)|\~590|

|1× Arc Pro B60|1.522|

|1× Arc Pro B70 (Linux)|2.694|

|1× Arc Pro B70, Qwen3.6 35B, 8K prompt|5.612 (generation 95-98 tok/s)|

|4× Arc Pro B60, Qwen3.6 35B, 4K prompt|4.264|

A few things that I find interesting from a technical perspective:

* Accuracy is verified, not assumed. Each model is compared against a reference f32 on CPU (KL divergence ≤ 0.015). The same prompt yields the same logits, bit by bit.

* One model can be split across multiple GPUs (`--gpus N`). In Linux, the summation result comes directly from GPU to GPU.

* Uses XMX (matrix engine Arc) on capable GPUs, first checked at startup to ensure the result is identical to the DP4a path.

* There are servers compatible with the OpenAI API plus a web UI, so it can be used from other applications.

How to try: download the Windows package (\~9 MB) or Linux from the release page, extract it, and then run `setup.cmd` (Windows) or ./zero-launcher.sh (Linux). The GGUF model can be downloaded directly from the setup.

🔗 GitHub: https://github.com/rizagp/inferzero (Apache-2.0)

I'm looking for testers, especially those who have:

* Arc A770 / A750 / A580 (Alchemist): no one has tried it yet;

* Arc B580 or Pro B60 / B70;

a Core Ultra laptop series 200H / Panther Lake*;

a multi-GPU setup in PCIe 5.0 x16 full* (not bifurcation).

If anyone wants to try, the results (output `--prof`) can be posted in the GitHub issue. Feedback and questions are also very welcome, especially from those who use Arc daily. Thank you!


r/LowEndLocalAI • • Aug 31 '26

Hardware / Build Local Qwen 3.8-27B running on $80 GPUs blindly told me I should build a box out of $80 GPUs (at 27 tok/s)

105 Upvotes

I put this box together a week ago and have been playing with it ever since. Spent the day getting Qwen 3.8 27B UD-Q4_K_XL gguf running on it. After seemingly endless rounds of synthetic loads and stability testing I finally sat down and had an actual chat with it. I asked it the same question that I gave to a few API models last week:

"Give me a rundown of the best cheap used datacenter-pull GPUs I can currently buy to repurpose for local LLM inference use as a hobby project please."

Its initial responses were reasonable enough and not too different than Kimi and Luna, but its pricing was dated (of course)... as was Kimi's.

So I gave it a few markets spot check prices and a little steering (again, copy / paste from last week) and low and behold it recommended to me the very dumpster datacenter pull GPUs that it's running on. I grabbed a screenshot of its internal thinking as it chugged by and I was in tears:

To be clear, this model is running on two of these GPUs... I was in tears from LOL

I guess I should post a spec sheet:

Hardware

  • 2× Tesla P100 16GB (PCIe)

  • Ubuntu 24.04, driver 580.173.02, CUDA 12.8.1 (compiled for Pascal sm_60)

  • Total build cost: ~$570 (using an old mobo / CPU / PSU I had sitting around, but buying RAM from r/homelabsales and some ancillaries)

Software

  • llama.cpp server v0.2.0-era PINNED + shinbunbun P100 community patches (minus patch #22 that fights with tensor-split)

  • --split-mode tensor + flash attention (yes, FA on Pascal via the patch set)

  • Served over SSH tunnel to the built-in llama.cpp web UI on the desktop

Model

  • Qwen3.8-27B, UD-Q4_K_XL GGUF, reasoning/thinking mode, 32k context

The session itself

  • 3-turn conversation with one tool use (model called get_datetime on its own)

  • Decode: 26.7–27.5 tok/s sustained, including a single 11,218-token reasoning + answer generation (~7 min unbroken)

  • Prefill: 65–169 tok/s on fresh tokens

  • Prompt cache doing real work: final turn reused 5374 cachd tokens, only 196 fresh

TLDR

  • Asked it for the best cheap datacenter-pull GPUs; gave it 2026 eBay prices; it hit "P100 ~$85 — !!! way down" and proceeded to recommend buying a fleet of the exact cards it was running on.

And yes, I already have three more GPU on the way.


r/LowEndLocalAI • • Oct 13 '26

NEWS LFM 2.5 24B A2B has been released and includes a QAD checkpoint

Thumbnail
huggingface.co
100 Upvotes

Might be of interest to the folks out there


r/LowEndLocalAI • • Sep 11 '26

Discussion I tested ~20 local models on a 4 GB laptop GPU

96 Upvotes

I've got a laptop RTX 2050 (3.4 GB of usable VRAM), a Ryzen 5 5500H, and 16 GB of RAM. I ran approximately twenty models through the same gauntlet: medical-physics Q&A, general knowledge, a six-task radiomics/image-coding suite, a false-premise "does it hallucinate?" test, and a speed bench.

The keepers

  • NeoHorse-1-4B — i1-Q4_K_M (imatrix), 2.5 GB. 10/10 physics, finishes the entire coding suite with a few nudges, and it's the only model that reliably pushes back when I assert something false.

  • Empero's Qwen3.8-4B — Q4_K_M, 2.6 GB. Equally sharp on physics, best strict coder, but a people-pleaser. If you state a confident wrong fact, it agrees and elaborates.

  • MiniCPM5-2B — Q8_0, 2.5 GB. Half the parameters, near-perfect physics, aced general knowledge.

My actual settings (all three run the same way)

  • Runtime: stock llama.cpp (winget b10588), Vulkan backend.

  • Offloading: -ngl 99 — every layer on the GPU, nothing on CPU.

  • KV cache: q4_0 for both K and V (-ctk q4_0 -ctv q4_0).

  • Other: -fa on (flash attention), -t 8, -np 1, --jinja. Lands at ~2.5–2.8 GB VRAM per model.

  • Sampling: temp 0.6 / top_p 0.95 / top_k 20 for the Qwen-family; temp 1.0 / top_p 0.95 for MiniCPM5.

Measured (llama-bench tg128, full GPU): NeoHorse ~32 t/s, Qwen3.8 ~32 t/s, MiniCPM5 ~39 t/s.

What I learned

  • Quantize the KV cache, it matters a LOT with such limited VRAM.

  • Thinking models over-think: several burned their whole token budget in hidden reasoning and never answered.

  • As coders, they're all one line from correct. Every model failed the same coding tasks the same ways (background argmax, distance-transform misuse, invented SimpleITK APIs). Pointed feedback fixes it.


r/LowEndLocalAI • • 1d ago

Model Showcase I fine-tuned a 1.5B model into an offline Linux command helper that runs on CPU and checks its own answers against the man pages

94 Upvotes

lxask turns plain English into a shell command, entirely on your own machine:

$ lxask "show disk usage of each folder here, biggest first"

$ du -h --max-depth=1 . | sort -hr

Setup

- Fine-tuned Qwen2.5-Coder-1.5B, Q4_K_M GGUF

- llama.cpp inside one Rust binary. No Ollama, no Python, no API key

- ~1.1 GB on disk, ~1 GB RAM, ~2 s per answer on a laptop CPU

- No GPU, no network after install

How it keeps a 1.5B model honest:

- Retrieves relevant tldr pages and your own man pages before answering

- Checks every program and option in the answer against the man pages

- Retries if a check fails. Anything it still can't confirm is marked `unverified:`

- Asks back on vague requests ("add a proxy": for git? apt? docker?) instead of guessing

Bonus

- `lxask --explain "tar -xzvf logs.tar.gz"` breaks down a command you already have

- Ctrl+G in bash, zsh or fish turns what you typed into the command

Code, weights, data and evals are all open.

GitHub: https://github.com/arxhr007/lxask

Hope someone finds it useful! If you do, consider leaving a ⭐ on GitHub. Feedback and contributions are welcome

Model: https://huggingface.co/arxhr007/lxask


r/LowEndLocalAI • • 3d ago

Unusual Hardware A language model for DOS that fits on a floppy disk

89 Upvotes

r/LowEndLocalAI • • Oct 06 '26

Benchmark SLMs on my personal agentic benchmark (Ornith 1.5 9B, MiniCPM5-2B, Ling 3.0 Tiny, Spark X2.5 4B, Neohorse 1 4B, Gemma 4 12B)

89 Upvotes

I created a benchmark based on my work, only to realize I had tested only one part of these models: instruction following. What I learned is this: regardless of which of today's SLMs you choose (1B, 2B, 4B, or 9B class), you will get roughly the same level of "dummy strength"—instruction following and reading comprehension. And that is actually very good.

Take MiniCPM-2B, for example—this 2.5 GB model scores just a few percentage points below Qwen-9B (four times its size).

What I learned is this: a ±2B model buys you local "muscles." Extra billions of parameters are just the knowledge you don't have to source yourself.

All these models performed decently when confronted with a well-defined, detailed task containing all the necessary context.

(tested on MBP M2 Pro, 16 GB - the NeoHorse 1 4B is the best fit for this machine leaving enough space for other apps)


r/LowEndLocalAI • • 16d ago

Model Showcase Ming-3.0-tts-0.5B clones a voice in 3.3 GB of VRAM, with 4.2 s of speech made in 2.1 s

88 Upvotes

Here is a voice-cloning footprint worth knowing about. This demo plays a recorded sentence next to its clone, with the run's numbers on screen:

- Model: AntLing's Ming-3.0-tts-0.5B, 2.99 GB on disk at bf16

- VRAM: 3.3 GB used by Ming, on a 24 GB card

- Reference: a 6-second sample, recorded with fan noise in the room

- Output: 4.2 seconds of speech, made in 2.1 seconds

- Check: a speech-to-text pass heard 100% of the words

- Setup: offline on one PC

The creator reports a general pace of about 4 seconds of speech in 2 to 3 seconds.

To repeat it, clone a voice from a 6-second sample, generate one 4-second sentence, then post your card, VRAM use and generation time. Results from 8 GB and 12 GB cards would make a useful set.


r/LowEndLocalAI • • 26d ago

Optimization Better than Strata. 35B model on 4GB VRAM/32GB RAM using llama.cpp (152tok/s pp, 14tok/s tg) runs Hermes

88 Upvotes

I have an old Dell XPS with a Nvidia 1650 (4GB VRAM), 32GB RAM, and i9 CPU retired in my closet. it's fairly old. I thought, hey you know what. Imagine if I can get this thing running Hermes, I can put this to use, running 24/7, I don't care how slow!

I had some basic requirements. If it can't do 64K context, it can't run Hermes. And it had to at least run q8_0. The workflows I run are long, and need that level of accuracy or they break.

Well, it turned it did matter HOW SLOW. I was about to find out. I loaded up Ornith-1.5-35B-A3B-APEX-MTP-I-Mini.gguf on Strata, thinking that's probably the fastest tok/s. Well, when it loaded, it was doing 65tok/s on prefill...A coffee + bathroom break weren't enough for the thing to load. I can't function like this.

Then I went to llama.cpp. After much tweaking, I got 152.93 tokens per second initial on pre processing (goes down over time), and 14.35 t/s avg on token generation.

What this is (and is not):
- If you start from a fresh Hermes session, you will wait about a minute and half for a first response.
- If you run cron jobs (like I do), the prefill is fast enough to not stall jobs from running.
- You will still need a coffee or bathroom break if you load a Hermes mid-conversation after a llama.cpp or thread swap. It took 389 seconds to load 33K context. If you're just giving it a task and walk away like I do, it's livable.
- Do not multi-chat. You have one channel in llama.cpp. You'll just clog up the system if it's trying to serve more than 1 request.
- Once the conversation is loaded, it's not bad. 14tok/s is very usable for back and forth commands, research, setup, etc...
- I don't use it for coding or small talk, only short webdev/market research.

This was more about finding an optimal placement between model intelligence, quality, and speed in Hermes. If anyone has other suggestions, let me know. It took me a long time to get to this, thought I'd share it for those with low VRAM.

./llama.cpp/build/bin/llama-server \
  -m "/home/models/Ornith-1.5-35B-A3B-APEX-MTP-I-Mini.gguf" \
  --alias "Ornith-1.5-35B-A3B-APEX-MTP-I-Mini" \
  --port 8080 \
  -c 65536 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --cont-batching \
  --top-p 0.8 \
  --top-k 20 \
  --min-p 0.00 \
  --reasoning off \
  --reasoning-budget 0 \
  --presence_penalty 0.0 \
  -b 4096 \
  --ubatch-size 2048 \
  --parallel 1 \
  -t 16 \
  -tb 12 \
  --cache-ram -1 \
  --no-kv-unified \
  --spec-type none \
  --reasoning-preserve \
  --jinja \
  --load-mode auto \
  --cache-prompt \
  --cache-reuse 256