Benchmark
Coding test based off of my real coding problems bench. Best models for a 16 GB card.
Changes dramatically if thinking is on or off. I notice some people saying some models are better than my experience. wanted to offer some info. One run 10 questions, 5 points each.
Also have a 5070Ti 16GB same as OP and get around 73t/s q4 on 27B. But I use adaptive kv streaming fork to get 262k. That said as context grows tg drops to as low as 20t/s at max context and prefill from 1500 to 300. So I have now moved on to FN and Strata which does not have this problem with the speed dropping off.
You are clearly not offloading the MoE models with those speeds. Flash next I easily get 130t/s and the 35B models I easily get 100+t/s with the same GPU much less RAM
I do have DDR5 but its not going to make it drop from 130 (or actually 182 peak) to 16t/s. Did you run FN on llama.cpp? Do you have MTP enable? Cause with llama.cpp I only get like 300t/s (vs up to 2600) prefill and 30t/s tg.
But for 35B models I do get around 110t/s on llama.cpp. Have not even tried them on Strata yet, had no desire to since I don't use them but I imagine I will get maybe double that speed.
Please do explain how tf you get 130tps on QFN with a 16 gig gpu? Are you running a fucking 64 core threadripper with quad channel ddr5-8000? Cuz it's either that or an IQ1 quant or something.
Or you're cherrypicking one very specific example where MTP/DFlash2 happened to get lucky.
I'm on a 5060Ti 16gb + 64GB DDR4-3600 and I'm getting 60~tps average with unsloth/Qwen3.8-Flash-Next-GGUF (UD-IQ3_XXS) using STRATA but I've seen it hit as high as 100 when it gets lucky. I've even hit 118 using the ISTA DASLab coder variant, and averaged around 70-80 with that. Hitting 130 sounds reasonable to me for a 5070Ti using Strata.
Better than 130 Actually, I have seen it peak at over 180t/s. I just said 130 because it generates anywhere between 100-180t/s. It usually drops to 100 for 1-2s when it reads an un-cached expert from SSD or RAM then its hovers around 130 for a few seconds and slowly climbs back up to over 170.
No it's a q2 quant. PCIe 5.0 X16 might help and the 9900X CPU. The 48GB RAM is all that stops me from choosing a higher quant.
Prefill often peaks at 2600t/s
When people said Strata was fast they meant it. With llama.cpp I get 300t/s prefill and I think like 30t/s TG.
Llama.cpp (SYCL backend.) GPU is an Arc A770 and RAM is 32 GB of DDR4-3200. PCIe 3.0 for the GPU and NVMe. IQ3_XXS quant, thinking on.
Obviously PCIe is a bottleneck, but also the model only has 6B active parameters and they're aggressively quantized. Only lost 1 tokens/second according to this graph.
Llama.cpp for FN is the biggest problem. Use Strata. I also use a model size that fits in my 48GB RAM so I don't stream experts from SSD. with 32GB I think the FN Coder is your best bet if you do coding. I also have a Gen5 SSD. Since q2 consumes all my memory I suspect maybe a few experts spill over onto the SSD or into Virtual Memory where Gen5 SSD does help. But I see 0-5% disk usage so I think I just managed to fit it.
I am unfamiliar with that GPU so I have no idea of what it is capable of. I have only heard of it.
I have a brand new machine I bought just a few months ago but I do regret getting only 48GB DDR5 and not 64GB. It was expensive a few months back and now it's even more expensive.
But your hardware does not compare to OP's or mine at all so understandable if you don't see better speeds. But OP has more RAM and the same GPU as me.
did you happen to trace time to completion, average solution sizes, or any kind of qualitative review? Also did you track different levels of reasoning? I appreciate the time and report. It's clear 27B is just in a class of its own.
I don't understand how are you getting Qwen 3.8 FN running at 7 tps and 16 TPS? I have a 10yo Nvidia Tesla p100 16gb + 128gb ram ddr4 2100mhz(only 88-89gb required for the model) at 256k context running at worst with 42 TPS via Strata
I got same GPU as him and only 48GB Ram. Getting 130t/s. His other MoE models are also underperforming. I don't think he is offloading the experts properly. probably does not have ngl set to 99
Ornith at fp8 and qwen3.6 at bf16?? They would fit much better at q4 or whatever would be the threshold for full vram allocation. I would have personally loved to get numbers on those instead - but i also only have 8GB VRAM available :)
I am running Strata Qwen flash next Q3_S at 131k context, it's really good I am getting above 70-80 t/s. It is great I made lots of applications specifically for the Android platform and I can say I happy with it the upcoming upgrade it will be Qwen4. 5070 ti + 64 ram
131k is low, set 262k. Pay attention to compactions. 1/3 - free space for successful compaction, 1/3 - compacted context, 1/3 - real workspace for new context
7
u/emperorofrome13 2d ago
more info https://medium.com/@emperorofrome/i-turned-thinking-on-for-21-local-coding-models-the-leaderboard-flipped-2e3774944528