r/LowEndLocalAI • • 2d ago • Heartwarming 1 Bravo 1

Benchmark Coding test based off of my real coding problems bench. Best models for a 16 GB card.

Changes dramatically if thinking is on or off. I notice some people saying some models are better than my experience. wanted to offer some info. One run 10 questions, 5 points each.

193 Upvotes

62 comments sorted by

14

u/jhaluska 2d ago

Thank you for putting the tok/s and file size, but I was thinking a scatter plot might be a bit better visual representation

11

u/Aggressive_Basket798 2d ago

Wow, how to get qwen 3.8 27b at 70tps on 16gb ?

I have 16gb gpu but only about 35-40tps with 27b gsq rco iq3s

7

u/emperorofrome13 2d ago

its the IQ3XXS fully in vram. Which engine are you using to run it? llama.cp directly, lmstudio?

6

u/Aggressive_Basket798 2d ago

Llama cpp, kv cache q4, 128k context , mtp

4

u/emperorofrome13 2d ago

Context size is the difference. I ran the test with a small context size. 

1

u/tomByrer 2d ago

There might be a faster engine for you.
I don't have that config, but poke around; eg search for "16gb strata fork" here & GH.

https://github.com/tomByrer/local-ai-linkfarm/blob/main/README.md#inference-engines

2

u/emperorofrome13 23h ago

Working on that. Doing a followup. The benches are taking a while.

2

u/Fancy-Snow7 1d ago

Also have a 5070Ti 16GB same as OP and get around 73t/s q4 on 27B. But I use adaptive kv streaming fork to get 262k. That said as context grows tg drops to as low as 20t/s at max context and prefill from 1500 to 300. So I have now moved on to FN and Strata which does not have this problem with the speed dropping off.

10

u/Fancy-Snow7 2d ago

You are clearly not offloading the MoE models with those speeds. Flash next I easily get 130t/s and the 35B models I easily get 100+t/s with the same GPU much less RAM

5

u/emperorofrome13 2d ago

I have ddr4. I assume you are on DDR5

3

u/ChocoPichu 1d ago

are you using plain llama.cpp? Or strata offloading?

1

u/emperorofrome13 1d ago

I'll have a more detailed answer in a follow up post but mostly llama.cpp

3

u/ChocoPichu 1d ago

thats why its only a few tps. If you mostly used strata on ddr4 you could’ve gotten so much more

2

u/Fancy-Snow7 1d ago

I do have DDR5 but its not going to make it drop from 130 (or actually 182 peak) to 16t/s. Did you run FN on llama.cpp? Do you have MTP enable? Cause with llama.cpp I only get like 300t/s (vs up to 2600) prefill and 30t/s tg.

But for 35B models I do get around 110t/s on llama.cpp. Have not even tried them on Strata yet, had no desire to since I don't use them but I imagine I will get maybe double that speed.

1

u/emperorofrome13 1d ago

I have to do a follow up post. Its complicated but you should see speeds 2x my system.  If I'm under that i need to recalibrate. 

1

u/game_difficulty 1d ago

Please do explain how tf you get 130tps on QFN with a 16 gig gpu? Are you running a fucking 64 core threadripper with quad channel ddr5-8000? Cuz it's either that or an IQ1 quant or something.

Or you're cherrypicking one very specific example where MTP/DFlash2 happened to get lucky.

3

u/Wezzlefish 1d ago

I'm on a 5060Ti 16gb + 64GB DDR4-3600 and I'm getting 60~tps average with unsloth/Qwen3.8-Flash-Next-GGUF (UD-IQ3_XXS) using STRATA but I've seen it hit as high as 100 when it gets lucky. I've even hit 118 using the ISTA DASLab coder variant, and averaged around 70-80 with that. Hitting 130 sounds reasonable to me for a 5070Ti using Strata.

4

u/humeformula 1d ago

Can confirm this. 100+ tok/s running QFN IQ3_XS on a 5070ti with 128GB DDR5 system ram, 262K context and vision disabled.

1

u/game_difficulty 1d ago

Can you share a config?

1

u/Fancy-Snow7 1d ago edited 1d ago

Better than 130 Actually, I have seen it peak at over 180t/s. I just said 130 because it generates anywhere between 100-180t/s. It usually drops to 100 for 1-2s when it reads an un-cached expert from SSD or RAM then its hovers around 130 for a few seconds and slowly climbs back up to over 170.

No it's a q2 quant. PCIe 5.0 X16 might help and the 9900X CPU. The 48GB RAM is all that stops me from choosing a higher quant.

Prefill often peaks at 2600t/s

When people said Strata was fast they meant it. With llama.cpp I get 300t/s prefill and I think like 30t/s TG.

0

u/OreosAndWaffles 2d ago

I get 6 tokens/second on Flash Next with a fairly substantial hot and warm cache. That number is already better than expected.

1

u/Fancy-Snow7 1d ago

Llama.cpp or Strata? With llama.cpp I get maybe 30t/s.

Also what GPU and how much RAM? And RAM type DDR4/5? And PCIe Gen?

1

u/OreosAndWaffles 1d ago edited 1d ago

Llama.cpp (SYCL backend.) GPU is an Arc A770 and RAM is 32 GB of DDR4-3200. PCIe 3.0 for the GPU and NVMe. IQ3_XXS quant, thinking on.

Obviously PCIe is a bottleneck, but also the model only has 6B active parameters and they're aggressively quantized. Only lost 1 tokens/second according to this graph.

1

u/Fancy-Snow7 22h ago

Llama.cpp for FN is the biggest problem. Use Strata. I also use a model size that fits in my 48GB RAM so I don't stream experts from SSD. with 32GB I think the FN Coder is your best bet if you do coding. I also have a Gen5 SSD. Since q2 consumes all my memory I suspect maybe a few experts spill over onto the SSD or into Virtual Memory where Gen5 SSD does help. But I see 0-5% disk usage so I think I just managed to fit it.

I am unfamiliar with that GPU so I have no idea of what it is capable of. I have only heard of it.

I have a brand new machine I bought just a few months ago but I do regret getting only 48GB DDR5 and not 64GB. It was expensive a few months back and now it's even more expensive.

But your hardware does not compare to OP's or mine at all so understandable if you don't see better speeds. But OP has more RAM and the same GPU as me.

6

u/UnorthodoxJourney 2d ago

Nanbeige got worse with thinking on?

5

u/emperorofrome13 2d ago

I was surprised as well. But nanbeige has problems with this thinking since it came out.

3

u/a4mula 2d ago

did you happen to trace time to completion, average solution sizes, or any kind of qualitative review? Also did you track different levels of reasoning? I appreciate the time and report. It's clear 27B is just in a class of its own.

4

u/emperorofrome13 2d ago

Most of the info is on medium. I'll post it to huggingface and send the link here.

2

u/a4mula 2d ago

I appreciate it, sorry I didn't see the medium post comment until after I asked.

2

u/saumanahaii 2d ago

Is dory the highest 4b model up there? How is it? Loop 9b also feels notable with its position.

2

u/emperorofrome13 2d ago

Nanbiege is better. Dory is too slow.

1

u/AyraWinla 2d ago

I wonder why Dory is that slow considering how small it is? I was pretty excited until I saw that it's like 4 times slower than bigger models...

3

u/emperorofrome13 2d ago

The loops. Loops = slow. Basically its doing the prompt 2x to get an answer. I did 2 loops for this. Ran 4 loops and that will go on the next post.

1

u/AyraWinla 1d ago

It is a lot slower than 2x though, which is what surprised me.

2

u/emperorofrome13 1d ago

The loop llms are really slow. Dont think the runners are optimized for it. Haven't seen a fast one. 

2

u/KeyMillion 2d ago

It has a looped architecture.

1

u/Cultured_Alien 1d ago

How about Spark x2.6 4B?

8

u/Relative-Ingenuity15 2d ago edited 2d ago

I don't understand how are you getting Qwen 3.8 FN running at 7 tps and 16 TPS? I have a 10yo Nvidia Tesla p100 16gb + 128gb ram ddr4 2100mhz(only 88-89gb required for the model) at 256k context running at worst with 42 TPS via Strata

is llama CPP that bad?

5

u/Fancy-Snow7 2d ago

I got same GPU as him and only 48GB Ram. Getting 130t/s. His other MoE models are also underperforming. I don't think he is offloading the experts properly. probably does not have ngl set to 99

1

u/shadow_17O1 1d ago

DDR4 or 5?

2

u/Fancy-Snow7 1d ago

DDR5 at 6000MHZ

2

u/robswansonskevich2 2d ago

Ornith at fp8 and qwen3.6 at bf16?? They would fit much better at q4 or whatever would be the threshold for full vram allocation. I would have personally loved to get numbers on those instead - but i also only have 8GB VRAM available :)

2

u/tomByrer 2d ago

7 tok/s for Flash-Next?1?

I think you're using the wrong engine; try one of these:
https://github.com/tomByrer/local-ai-linkfarm/blob/main/README.md#inference-engines

2

u/Acojonancio 2d ago

Thanks for showing this.

I have a 9070 XT and tried to use Qwen3.8 27B but Q4_K_M and it went straight into RAM.

I'm on Windows 11 and using Bionic, don't know if that might be one of the causes for failure.

Will download the IQ3_XXS and test again.

2

u/Inflation_Artistic 1d ago

Very good data, but it would be fashionable to add a couple of closed larger online models just to understand the difference.

1

u/vogelvogelvogelvogel 2d ago

27b holds up very well at that quant against fn

1

u/stianhoiland 2d ago

Very nice graphs and details. You should definitely include ISTA-DASLab's Qwen 3.8 Flash Next IQ3_S quant in there.

2

u/Apprehensive_Taste74 1d ago

Yep I second this. Seems to be the sweet spot so far between speed and accuracy.

1

u/daedalusbr 2d ago edited 2d ago

I would use Time to Solution instead of tok/s as the speed metric.

Edit: I didn't look at the medium article before, you reach the same conclusion.

1

u/BeanOnToast4evr 1d ago

From memory Qwen recommended temp = 1 for thinking enabled tasks, not sure the impact on temp = 0.6

1

u/Chips_fr_ 1d ago

Interesting to see that SenseNova and K2 horizon are quite competent here.

And bonsai 2 can be good if well setup.

1

u/cebess 1d ago

have you tried:
ornith15-35b-a3b-apex-mtp-fixed

It seems to be both fast and good on my machine of a similar size.

1

u/sand-67 1d ago

Test granite 4.1 cause its the best model for no thinking

1

u/thevegit0 22h ago

try using strata for qwen flash

0

u/Tight_Commercial7 2d ago

I am running Strata Qwen flash next Q3_S at 131k context, it's really good I am getting above 70-80 t/s. It is great I made lots of applications specifically for the Android platform and I can say I happy with it the upcoming upgrade it will be Qwen4. 5070 ti + 64 ram

1

u/emperorofrome13 2d ago

Im dojng something wrong or might have slower ram. Do you have ddr4 or 5? 

1

u/rrrrex 1d ago

131k is low, set 262k.
Pay attention to compactions. 1/3 - free space for successful compaction, 1/3 - compacted context, 1/3 - real workspace for new context