I’m providing an update! We have around 300 berth users now so the project is moving fast.

Recent updates for v0.6.0:

- v0.6.0 gives each card a schedule
- berth train runs a training that nothing automated can interrupt
- learned engine sizes (berth learns your engines resource claims)

## A month on the reference workstation

berthd counts every handoff it makes, and the harbor's board shows every handoff from one RTX 4090 workstation between 2026-10-12 and 2026-11-13, running llama-swap, NInfer, Strata, ComfyUI,
coding agents and games:

In the last 7 days, 23 engines and jobs held the card for 143 lease-hours. Leases that share the card count their
own hours each.

Project: berth

I think the last 3 posts from here that showed up on my feed were barely veiled self-promos.

I was here when huggingface first started getting popular, and the level of support for running local models here was incredible.

Now I am debating whether or not to leave the sub. Why isn’t there more attention to moderation to this?

Take a user’s post, and type their username with quotes into google and then type whatever obscure tool they’re asking a question or talking about and you’ll see the absurd rate or Astro-turfing and shilling.

They will post here and 6 other LLM places, but this place used to be awesome, so I only care about this sub.

If you are wondering what model / harness combination works the best, here are my findings so far:

1. Model is way more important than the harness, best so far Qwen Flash Next even with low quant outperform Qwen 3.8 27B

2. The inference engine is way more important than the harness, because the size of the context windows can dramatically increase performance. AND because speed make the difference between something useable and something that won't work.

3. Harness optimised for coding like OpenCode or PI outperform more generic agent like OpenClaw and Hermes, even on mondane task like reading emails, making purchase on a store or complex vision.

What changes between different quantizations of the same model? And what if you could SEE what actually sets them apart?

To help answer that question, I’ve created a small project called inside-llm—an open-source tool that loads one or two GGUF models using llama.cpp and displays, right in your browser, what the model is doing as it generates text.

What it shows

• Stream: the text, one column per model, brightness = probability of each token. A magenta mark wherever A and B would pick differently. Two modes: lockstep (both models read the same text, so you get a clean per-position comparison) and free (each follows its own picks until they split).

• Candidates: the full distribution behind each pick, mirrored bars for A and B, entropy, effective number of options, Jensen-Shannon divergence and KL in both directions.

• Gates: the sampler chain (penalties, top-k, top-p, min-p, temperature, GBNF grammar) traced stage by stage and very visual: how many candidates survive each one, and the distribution after it. Chain order and seed are configurable.

• Timeline: entropy of A and B and their divergence along the whole text, plus the tree of branches.

• Branching: click any candidate and generation continues from there. Every branch stays in the tree, and jumping back to a node is exact (each model keeps KV-state snapshots, so nothing is regenerated).

• Logit lens (--lens): what each layer would predict, as a 2D map and as a 3D relief.

• Model picker: choose A and B from a folder or search Hugging Face and download two quants of a repo in one click easily. It only offers B models with the same tokenizer as A and estimates whether the pair fits in VRAM.

How it's built

• One C++17 process linking llama.cpp directly (stable branch at the moment). For every step it reads the full logits of A (and of B on the same context), computes the metrics and the sampler chain, and draws the token with its own seeded sampler, so runs are reproducible.

• Compact binary frames over a WebSocket to a Svelte 5 + Canvas 2D page (three.js only for the 3D relief, lazy-loaded). The web is embedded in the binary as well.

• The same frame stream, compressed, is a recorded session: you can replay it later in a static page with no engine, no model and no GPU.

Run it

Self-contained packages for Linux (glibc 2.35+) and Windows, with CUDA or Vulkan (Vulkan covers AMD, Intel Arc and NVIDIA). Unpack, run, the browser opens with the model picker. Or inside-llm -a A.gguf -b B.gguf. Local only: listens on 127.0.0.1, no accounts, no telemetry. MIT. One click download, one click execute.

Limitations (honest list)

• A and B must share the tokenizer (two quants, base + fine-tune, or sizes of one family that keep it). Example: Qwen3.5-0.8B_Q4_K_M and Qwen3.5-0.8B_Q8_0

• Both models must fit into VRAM simultaneously; only one GPU is used at a time. Integrated GPUs (iGPUs) are never used; if no dedicated GPU is available, the process falls back to the CPU (which is slow). (You must calculate the size of both models before loading them to ensure they fit into your VRAM)

• Text only. No split (multi-part) GGUFs, no embedding models.

Links

• Code + releases: https://github.com/manuelgscom/inside-llm

• Project page: https://manuelgs.com/en/inside-llm/

I'd love feedback, especially: which pairs would you compare first (imatrix vs non-imatrix, base vs abliterated/fine-tune, Unsloth UD vs standard quants…)? And is there a metric you'd want per position that isn't there?

Just looking for some advice on what is possble with my setup and speeds I should be gettingI know it won't be super fast tps but I was more after getting the bigger models to run.

This is my spec:

Epyc 7502p - 32 Core
Gigabyte MZ32-AR0 - 8 channel memory
512GB DDR4
1x 4070 12GB
2x 4060 16GB

quite a few implementations I see which can be confusing.
Is this build even useful now? that we have Project Maya which you can stream from the NVME drive?

Anyone have similar setups and what to expect, what software best to use

(Note: by local models I mean anything smaller than Deepseek 4.1)

I already gave up on local models for coding tasks 6 months ago, due to what I perceived was intelligence too low for the tasks I give it. I made a post about it on here back then and it was a huge discussion, I think 800 comments. I went against the /r/LL orthodoxy and many didn't take my comments well, but we can agree to disagree.

Since then I only use local LLMs for translation and ComfyUI prompt enhancement. I stick to big cloud models for coding. I think the electricity cost of running Qwen 27B alone exceeds what you'd pay Deepseek to use their cloud API in their own harness, and you'd get a far more intelligent and far faster model.

But this post isn't about the economics. It's purely about ability. I still see people commenting in favor of buying GPUs for local intelligence. "Sonnet at home" is a ridiculous phrase I've seen more times than I should in relation to Qwen 27B.

The simplest of tech tasks is still too hard to one-shot

I keep Qwen 3.8 27B running on my Spark for occasional non-coding stuff. Today I gave it a trivial coding task I needed for my server, just out of curiosity.

• llama-server launched with --model Swift-1.5-Qwen3.8-27B-Q4_K_L.gguf --reasoning on --reasoning-effort xhigh

• Open llama webpage and leave Reasoning at Default, leave system prompt empty

Prompt was:

Give me a bash script that will list all files in directory sorted by date. Output should be:
2025-12-31 12:13:14 file1.ext
2025-12-31 12:13:14 subdir/file2.ext

1630 tokens later, most of it reasoning, I get a script. When I run it, here's the output, as you can see the timestamp is wrong:

2026-10-16 20 tmp.59eH9w3JLe
2026-10-17 20 tmp.5xvczTbSS4
2026-10-18 20 tmp.nWOEDw2wlu
2026-10-20 20 tmp.JVEZIFfKx3
...

I then tried Gemma 31B Reasoning on duck.ai. The output only showed the timestamps, no filenames!

This is the most trivial of coding tasks (the answer is a 3-5 lines script), using the most ancient and commonly-available training data (bash), on the most praised local coding model on this sub (Qwen 27B), at a recommended/popular quantization (Q4), and it still can't one-shot this. The 2nd most popular local model, Gemma 4 31B, also fails.

(Editor's note: I think my post could've ended here, in a "Ecce homo" way, but I went on for a few more paragraphs)

Weak local LLM + software development = bad time

I'm sure if I had run this task with your average agent harness with a system prompt that makes it test its code, it would run the script it created, see the issue, then get it right on the 2nd attempt. But while I think this "try until you succeed" workhorse approach is acceptable for small self-contained coding tasks, I don't think anyone should be doing any serious coding this way.

If you're writing something you intend to maintain over time (ie not a one-shot), or that you will be sharing with other people, you need a higher degree of LLM intelligence and knowledge to create it. A strong LLM will be able to avoid entire classes of problems from the start, when designing the app. From design to implementation to maintenance, it will make better decisions every time and without laboring its way to them, resulting in a codebase that is smaller, well-designed, more easily modifiable, less buggy, and developed in a fraction of the time.

Coding with local LLMS is like insisting on using a donkey for transportation in the age of cars. It can be done for fun, but it shouldn't be done seriously.

Sonnet at home my ass

Btw I've seen several people say Qwen 27B is like having a local Sonnet 4. The benchmarks say it's even better than Sonnet 4, actually. So I went on openrouter.ai out of curiosity and gave the same prompt to 3 models.

• Sonnet 4.5 (oldest Sonnet I could find): one-shots it

• GPT 4.1 Mini, which was comparable to Haiku 3.5: one-shots it

• GPT 4.1 Nano, which is even worse than Mini: one-shots it

Weak local LLM + Linux server management = even worse time

This post's getting long so I'll just speedrun this part.

• Managing an OS is actually harder tasks that LLM struggle even more with. And the stakes are higher because you don't want your server to go down.

• I used to never trust LLMs enough to let them onto my server, I'd just ask for something, copy-paste the command, report back, repeat. A huge waste of my limited free time. Now I just rawdog a frontier model on the server and have no issues, something you could never trust a small model to do. Although I have it avoid making any changes without my supervision, Docker is a big benefit here for both organization and safety/stability.

• Even top models won't one-shot a task like "I want to install and run https://github.com/rando/something on this system in a Docker container in /opt/docker/something/. Create the Dockerfile and docker compose file", especially for AI-related tools, where having the right CUDA, not having python wheels for my server's ARM CPU, etc, introduces a lot of failure opportunities. When I had Claude five months ago, even it struggled to get these right.

Thanks for reading my wall of text. Downvotes to the left.

Few months ago - after Ninfer came out - I started a little hobby project to vibe-code my own engine for my hardware (RTX 5080 16GB Laptop) and check if it could beat llama.cpp - since at the time llama.cpp was kinda slow in comparison to something like vLLM, but vLLM is using VRAM very wastefully.

Back then it was quite easy to actually get something up and running faster than llama.cpp.

Today I just benchmarked llama.cpp again, and it caught up / surpassed my vibe-coded engine. I prefer using a stable engine with proper features than all the vibe-coded ones, including my own. So it is really nice to see that the project - while a bit slower - lands significant performance improvements over time.

I put together a little chart of benchmarks I had laying around which visualizes this quite nicely. It is by no means scientific - it's always a fixed 16k prefill 1k output over Wikipedia text with the optimal settings for each engine. The 12B model is the same checkpoint, for the 26B model these are different quants for all engines, so not even apples-to-apples and llama.cpp wins.

Hi! for those who might not know the solution:

• it takes any MoE GGUF llama.cpp can load, whatever the quant. No repacking, no special format

• it runs on whatever you have: NVIDIA, AMD, Intel, Mac, a phone, or just the CPU

• Windows, Linux, macOS, Android, iOS

• output is the same as llama.cpp, don't cut corners on quality

• you start it with --auto and it works out the settings for your machine

Below is a brief overview of the benchmark:

• 8 GB card, Qwen3.6-35B: I get about 35 tok/s. colibri around 23. Strata wouldn't start at 16k context

• 12 GB card, same model: about 45 for bmoe, about 30 for Strata

• long prompt on the 8 GB card: bmoe 5 s to the first token. Strata 14 s, colibri close to 2 minutes

• Basalt needs an RTX 50, so I couldn't run it on these

• 16 GB MacBook, Qwen3.6-35B: about 20 tok/s for bmoe, colibri's release 3.5

• same Mac, Qwen3.8-Flash-Next 125B: 10 tok/s, Strata about 7

On the cards Strata is tuned for it's still ahead but only by a little.

On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.

Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge

If you have an 8 to 32 GB card I'd like to see what you get.

Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data

This is a hobby project, not a product pitch. I saw a TikTok where people rewrote famous movies and games as over-long isekai light-novel titles, and I wanted to see how far I could take that with local models. The game is at https://itsisekai.com. Five quests a day, guess the original.

Hardware. NVIDIA DGX Spark (GB10, 20 Arm cores, 121 GB unified memory), set up as a self-hosted GitHub Actions runner. A nightly workflow (or a manual run) runs the pipeline on the same box as the models. No API fees for generation.

Models.

• Text and judge: Qwen3.6-35B-A3B, official FP8 build, served with vLLM (OpenAI-style API). fp8 KV cache, 128K context, MTP speculative decoding with 3 draft tokens, prefix caching, and 45% of GPU memory so ComfyUI fits beside it. The judge is the same model, and thinking is left on. Generation runs about 70 tok/s typically, peaking around 240 tok/s.

• Images: Qwen-Image in ComfyUI with the official fp8 weights (qwen_image_fp8_e4m3fn + qwen_2.5_vl_7b_fp8_scaled encoder), 832x1248, 20 steps, cfg 2.5, euler/simple. One render takes about 64 seconds (around 3.15 s per step).

• I had to drop vLLM --gpu-memory-utilization from 0.80 to 0.45 so ComfyUI fits beside it.

Pipeline. It fills missing quests for the next several days. All missing titles are written first, one at a time, so later quests can see the answers already picked. Then covers are drawn. A day is only saved when all its quests succeed, so the site never shows a half-filled day.

For each quest the model writes the answer, hints, a 10-40 word title that can't contain the answer, and an art prompt. A second "how obvious is this title?" check can send it back for a rewrite. The cover is judged on the image alone: if the model can name the work, the prompt is rewritten more abstractly and redrawn, up to 3 times. If every try still gives it away, the cover stays sealed until the player finishes the quest.

Timings from a production run on Nov 15 (8 quests across 2 days, 0 failed). The whole run took 61 min 29 s.

• Titles, including the obviousness check and rewrites: 2 min 14 s to 6 min, average 4 min 26 s. Four of the eight were first judged too obvious and rewritten.

• Cover attempts, draw + judge together: 1 min 38 s to 3 min 34 s, average 2 min 23 s. About 64 s of that is the render; the rest is the vision judge thinking about whether it can guess the work.

• The judge passed 8 of 10 cover attempts. Two covers gave it away on try 1 and passed on try 2. None had to be sealed.

What went wrong. vLLM's guided JSON didn't mix with Qwen thinking: requests could jam and loop until they timed out. I dropped guided JSON for Qwen and just parse the JSON out of the reply. Without a presence penalty, Qwen could also think in circles until it filled the context; I set presence_penalty 1.5 and max_tokens 16384. One failed model call used to wipe out a whole day; now it retries and moves on. I also learned not to overlap manual workflow runs with the nightly cron.

Happy to answer questions. I'm not selling anything, I just thought the timings and the "don't use guided JSON on a thinking Qwen" bit might be useful.

In audio.cpp 0.9.5 you can try this new avatar voice chat "app" in the WebUI.

The pipeline combines 3 models: SoulX-FlashHead for real-time talking head generation, OWSM v4 for transcribing speech input to text, and LFM3-Audio for conversational responses and TTS.

LFM3-Audio is actually called twice. The speech-to-speech path only provides one voice. We use the LFM3-Audio TTS to synthesize the S2S response in a different voice.

One caveat: the video generation stage is still GPU-intensive. For SoulX, using RTX 4090 or better GPUs.

cuDNN is a must for real-time performance. Without it inference is 2× slower and VRAM usage doubles.

~850ms between submission a question and avatar speaks on an RTX 5090. ASR runs in parallel. As mentioned LFM3-Audio is called twice, which can be optimized. There’s also some overhead from the prototype UI. A native app with a more optimized implementation could further reduce latency.

Edit: 1Cat-vLLM is a Tesla V100 focused fork of vLLM, and the expert caching work is 1Cat's original work, which may or may not one day make it to vLLM.

Everyday we see posts about speeds on Strata and its forks, but just recently I came to know that 1Cat has added support for expert caching recently, like a month ago. So I did (asked claude) a comparison of speeds on a Qwen3.8-Flash-Next UD-IQ3_XXS quant. My rig is 4x 16gb Tesla V100 on PCIe 3 x16 links, with 8 channel 3200 mt/s ddr4 ram, Epyc 7543 CPU. Now, for numbers:

Single stream

Median of 3 runs, except the 97k tokens one which I asked to run only once to save time.

Strata has better draft acceptance and still 1cat beats it, imagine if 1cat could get better draft acceptance.

The same 103K-token prompt three times

Same prompt, 3 times in a row, just different enough to not trigger prefill cache.

Several requests at once

(TW: Scary AI written paragraph) Each server was started sized for that many sequences. Total = tokens/s summed over all streams during the window where every stream is decoding. Median of 2 runs; the 1-request rows are the medians above. "Strata batch-MTP" is Strata's best batch mode on this build: --batch-mtp with STRATA_PIPED_MTP=1, which keeps the pipelined layer-split groups and drafts inside them.

Versions and local changes

TLDR: 1Cat-vLLM beats strata in every metric except prefill for very long prompts (around 100k). And it does so while being able to hold lesser experts in VRAM and for some reason getting worse draft acceptance, simply because of TP.

Villanova2 family*, for English, Italian, German, Spanish and French. Built in EU, for EU.*

The family

• Villanova2-15B-A3B-Base · sparse mixture of experts, 15.5B total / 3.9B active

  • Villanova2-15B-A3B · instruction-tuned

  • Villanova2-15B-A3B-VL · vision-language

• Villanova2-2B-Base · dense, 2.4B

  • Villanova2-2B · instruction-tuned

  • Villanova2-2B-VL · vision-language

GGUFs available in same collection. License : Apache-2.0

I made a tool that tunes copies that speeds up CPU offloading.

1 3090 + 7800X3D

llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3

cmt tune -- llama-bench -m Qwen3.6-35B-A3B-Q8_0.gguf -ngl 99 -fa 1 -ot exps=CPU -p 2048 -n 0 -r 3

before: prefill 345 t/s
after: prefill 466 t/s


blog: https://maknee.github.io/blog/2026/CudaMemcpy/
code: https://github.com/Maknee/cudamemcpy-tuner

Hi everyone! I wanted to post a quick update on Basalt. Since last I posted, multi model and router support was implemented, starting with DeepSeek V4.1 Flash (IQ1_M 90 tok/s conv, 246 tok/s struct on a 5090 + 5060 Ti). Also new: video support, files in chats, and a bunch of fixes and improvements.

Now 0.4.0 is out featuring Basalt's first dense model, Qwen 3.8 27B NVFP4. It's fast and accurate, top-1 92.8% vs the full BF16 model. This, in preparation for Qwen 4, so we had a headstart on the dense kernels.

The numbers:

• Up to 776 tok/s on structured output, 262 conversation, 194 on prose. Single stream.

• Prefill, 12,000 tok/s at 4k (slower as context grows)

• 92.8% top-1 vs BF16 (0.0262 KLD)

• Chat, tools, thinking, images and video, up to 262K context

Links:

GitHub: https://github.com/jesdga95/basalt
Site: https://basalt.jgulfo.com
27B quant: https://huggingface.co/jesdga/Qwen3.8-27B-Basalt

Thanks to everyone who has filed issues, sent PRs or posted benchmark rows, special thanks to the sponsors who keep this going, happy to see Basalt gain so much traction in such a very short time!

Hello. Almost everything is in the title, but here is more details:

I'm benchmarking local model/harness combinations in various scenarios. For this one the question is, how good is a model/harness to generate a website?

IMO Qwen FN with opencode is best, but you can look at all the results (and play with the websites here: https://airbench.ai/gallery/airbench-frontend-v1?run=37e47127-3a20-4318-a6f2-48c3f7f5eaa6&c=settle-the-trip&k=poL

Measurements: mean ± sample standard deviation in tok/s, 3 runs after normal warmup

(16GB of RAM on the AMD EPYC 9645 using eight CPU cores)

But noticed few surprising things there. I assumed the boost is only for Ternary models, but it's not.

This makes CPU kernels a lot faster, raising IQ1_S/IQ2_XXS DeepSeek v4.1 decode speed

End to end, a 66 GB Q2_0 MoE (Qwen3.8-Flash-Next GSQ-RCO Q2_0) with --cpu-moe, so all routed experts run through this kernel on the CPU. RTX 4070 Laptop for the rest, 10 threads, MTP on, generation t/s at 2k–8k tokens of real context:

So check your t/s again if you're using Q1/Q2 quants of big/large models. There would be boost on Hybrid inference(GPU+CPU) too.

AND there's an Open PR related to this for AVX-512 VNNI. Additional boost on the way.

ggml-cpu: 512-bit AVX-512 VNNI dot for Q2_0- #32307

At Axiomic Labs we have just released X3Guard-150M, a prompt guard model trained on our high-performing base model, GPT-X3.

For any given prompt, X3Guard flags whether it is Harmful, and whether it is Adversarial, in one go.

It outperforms models that are many times its size, and can run entirely on edge devices, within the browser, with response times in the tens of milliseconds on iPhone.

The model card is here

You can use an in-browser demo here

There seems to be a lot of discussion here about inference and using the models, but is there anywhere where discussion about making them is more common?

Disclosure: I'm the author. Attest is an Apache-2.0 testing framework for agents (Go engine, Python and TypeScript SDKs). Posting here because most of it runs with no hosted model at all.

Replay instead of inference in CI. attest capture -- python my_agent.py runs your agent behind a local recording proxy that forwards to whatever OPENAI_BASE_URL points at and records every model call. In CI, the agent's current code runs against that recording:

attest replay: 4 replayed · 0 live model calls · $0.00

The CI runner never loads a model. If you change the prompt, replay misses and tells you what changed instead of guessing.

Deterministic checks first. Schema, cost/token/latency limits, which tools ran, in what order, how many times per order id, and what the run did to a simulated backend. None of that needs a model.

Local embeddings. Similarity checks can use a bundled ONNX all-MiniLM-L6-v2 instead of an embeddings API.

Local judge. When a check really needs a judge, set ATTEST_JUDGE_BASE_URL to any OpenAI-compatible server (vLLM, Ollama's /v1, LiteLLM). There's also a beta Ollama adapter for recording traces.

Small judges break JSON. A malformed judge reply is reported as "not evaluated", never as a pass, and your policy decides whether that warns or blocks. 1.1 (released November 10) is a correctness release: it fixes four ways 1.0 could report a failing assertion as passing.

The demo is a synthetic support agent that refunds a customer twice while its reply stays polite and correct. make setup walkthrough runs the whole story, no API key, about two minutes after setup.

Limits, honestly: capture is tested against a scripted OpenAI-compatible server and the hosted APIs, not against every local server's streaming quirks. If your server's stream trips the proxy, I'd like to see the trace. Also, most of the code was written by coding agents (Claude Code and Codex) from plans I wrote and reviewed.

• Repo: https://github.com/attest-framework/attest

• Web Page: https://attest-framework.github.io/

• Demo: https://github.com/attest-framework/refund-desk

• Judge setup: https://github.com/attest-framework/attest/blob/main/docs/guides/judges.md

Because of Theo gg tweet of all people,
I Launch QFN DeepResearch swarm agent to find anything about this.

THREE model names, blurred out, sitting at 59.4% / 57.1% / 54.0%.

1. The blurred names are most likely [GLM-Nano or GLM-Name or GLM-Next (suggestion from Silver-Theme7151)] Pro, Med, and Small. One community experiment reconstructed the suspected text, applied Gaussian blur (σx ≈ 6.9px), and achieved an RMSE of just 1.254/255 against the original.

2. AICodeKing track record. Someone compared his previous GLM statements against official release dates: GLM-4.6 (same day), GLM-5 (1 day), GLM-5.1 (11 days), GLM-5.2 (3 days), and GLM-5.3 (4 days).

3. Antrhopic timing. The leaderboard surfaced on November 12 vs AICodeKing timing, while Anthropic IPO on Nov 18. If Z.ai releases GLM-Nano this week, the timing would make for quite a coincidence

https://linux.do/t/topic/3166447

Another strong signal is HF activities, i know it is not bullet proof, just another signal, 3-4 days compare to last 27 days

I use LLM to fix my grammar.
Since it is speculation and not actual open model this post has a high probability to be removed, so yeah

Ever since the information surfaced five days ago, I had intended to implement it right away. Having confirmed that it could be executed using 'pico,' I carried it out today. I had Astra write the script for the restricted write-access portion, and after nearly 15 hours of work, I finally managed to complete the task. I also plan to handle the second CMP later on.

I was doing some research about the the platform for Intel's next generations of CPUs.

If we look closely at the PCIe 5.0 configuration for Z990/W980, it specifies

1. x16 (or x8/x8)

2. x8 (or x4/x4) "Storage"

So, it is plausible we might see motherboards with 3 PCIe 5.0 x8 slots directly wired into the CPU.

The best we can currently do on consumer hardware is PCI 5.0 x8/x8/x4 without risers and there are only two boards that can do it and neither is in production anymore (MSI MEG/Godlike X670E). In practice the best is x8/x8.

TP=3 is fairly easy for your agent to patch into inference engines.

These boards will have 2x Thunderbolt (Wiki says 4, but it may be 5?). So, possibly options for high-speed networking here. Let's make RDMA over Thunderbolt happen!

The desktop CPUs will have Low-Power-Ecores, which should help with idle power-draw and on the other hand they will have full AVX512 again (yay).

Asrock and MSI are the most likely ones to color outside lines. I hope the x8/x8/x8 is "allowed" by Intel, unfortunately nothing much to do but wait until we see the line-up of boards.

But I will be watching.

• Table: https://en.wikipedia.org/wiki/LGA_1954

Hey guys

I am a seasoned local llm user and I ain't quite ram rich like only 16gb ddr4 🥀

So I can't run qwen3.8 flash next on my 3060

This guy felt the best for me yet

Have you guys tried it

I have ran iq3_xss quant non mtp with turboquant at 65k context completely on my 3060 though I did offload mmproj to cpu ram

Have guys tried this it really felt like a blessing to my life using the peak of local ai on my 3060

It seriously packs the punch 👊 I was able to perform various of my agent task complete on device not like simple but the ones which require actual reasonings

Here is my cmd if you want to see

llama-tq serve -m "E:\MODELS\ISTA-DASLab\Qwen3.8-27B-GSQ-RCO-GGUF\Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf" --mmproj "E:\MODELS\ISTA-DASLab\Qwen3.8-27B-GSQ-RCO-GGUF\mmproj-Qwen3.8-27B-BF16.gguf" --no-mmproj-offload -ngl 99 -c 65536 -ctk turbo4 -ctv turbo3 -ub 256 -b 2048 -t 3 -tb 3 --reasoning-budget 256 --reasoning-preserve --alias Qwen --port 8080 -lm none

Sources: for my recipe

ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF/discussions

Turboquant

https://github.com/TheTom/llama-cpp-turboquant

Pi coding agent

https://pi.dev/

I would like know the communities views and any suggestions would be welcome 🤗

Thankyou for your time

Hi,

I've put on HF a condensed version of Unsloth Q3_K_XL ASCII-Condensed from bsaleh03 with only ASCII + Latin-1 / Latin-Extended-A (P2) to use with a 16GB VRAM GPU.

I created a GUFF version and also a ninfer to be used with ninfer-all

To create the ninfer version I had put up some helper conversion scripts and modify iamwavecut conversion scripts due to different vocabulary size.
In the HF repo https://huggingface.co/fakezeta/Qwen3.8-27B-UD-Q3_K_XL-Condensed-P2 there is what is needed to replicate it and I also submitted a PR to the upstream repo.

Quality: IFEval result 84,4% vs 83,9% of base model (Instruct, 1 run) so within STD Deviation error.

With this I achieved 75K context with rk8v4 kv type with an average of about 60tok/s in the 541 IFEval samples on my 5060Ti 16GB.

I'm also planning to create a light GGUF and ninfer version with the lower quantized MTP released separately by Unsloth in their repo to save about 90MB of RAM for other context.