inferzero is an inference engine written from scratch for Intel Arc: C++ on Level Zero, kernels in OpenCL C, on Windows and Linux.
What it runs: GGUF files as they are.
What it serves: an OpenAI-, Anthropic- and Ollama-compatible API with a web UI.
What it needs: nothing else installed (no Python, no oneAPI, no compiler).
On Arc B-series cards it has a fast prompt path of its own:
Hadamard-int8 GEMMs and flash attention, both on XMX;
decoded weights kept in VRAM;
tensor parallel over several cards (--gpus N).
Every model is checked against an independent f32 CPU reference (next-token KL ≤ 0.015), and the logits are bit-identical run after run.
Best numbers per card (prompt / generation in tok/s; all but the 125H are users’ runs)
1× Arc Pro B70 (Linux)
1× Arc Pro B60
Qwen3.6 35B-A3B: 4K prompt 3,536, generation 82.4, 111.9 with MTP.
Qwen3.8 27B: 4K prompt 1,522, generation 22.7, 51.6 with MTP.
Several B60s, one model split over them
Qwen3.6 35B-A3B over 4: 4K 4,264, 64K 2,152, 131K 1,267, 262K 700.
Qwen3.8 27B over 2 / 3 / 4: prompt 1,597 / 1,724 / 1,888, generation 37.8 / 43 / 53.
Qwen3.8 Flash Next (52 GB of experts): 4K prompt 665 / 1,202 / 1,491 over 1 / 2 / 4. Generation 34.4 over 2, against 5.7 on one card.
Two B60s on Linux add their sums straight from GPU to GPU: 1,630 tok/s, against 1,442 through host memory.
DeepSeek-V4.1-Flash (new, experimental; 384 experts), 4× B60 with 1 TB of RAM:
ISTA’s IQ2_XS:
Unsloth’s UD-Q8_K_XL: prompt 109, generation 7.6 over 3.
Each card holds its share of the experts; the rest stay in RAM, one copy for every card.
Being tested now: KL against llama.cpp, Q8 over 4 cards, and the size of the RAM tier.
Laptop iGPUs
Core Ultra X7 358H (Panther Lake, Arc B390, Xe3; the first Xe3 runs, not tuned yet)
Qwen3.8 27B IQ3_S: 4K prompt 310 tok/s, 8K 303.
With the NPU computing part of each prompt chunk next to the GPU: 4K 338 (+9 %), 8K 313. The KL against the GPU alone is 0.0022.
A 27B fine-tune in Q3_K_M: a 10K prompt at 245-255 tok/s, against 170 for llama.cpp’s Vulkan back end.
These are still on the generic XMX path. The discrete cards’ fast path (H8, wide tiles, flash attention) is being tested on it now, so these numbers should rise.
Core Ultra 9 285H (Arc 140T, XMX)
Qwen3.8 27B: 4K prompt 357, generation 6.0, 16.9 with MTP + DFlash2.
Qwen3.8 Flash Next: generation 26 with DFlash2.
Core Ultra 5 125H (Arc iGPU, no XMX, 32 GB of RAM; the engine was built on it). Prompts are 8K tokens; MoE experts are streamed from the SSD.
| Model | 8K prompt | Generation | With drafting |
|---|
| Qwen3.6 35B-A3B | 851 | 21.6 | 38.1 (MTP + DFlash2) |
| Qwen3.8 Flash Next (52 GB of experts) | 205 | 6.3 | 7.6 |
| Qwen3.8 27B | 135 | 5.1 | 12.9 |
| Gemma 4 26B-A4B | 232 | 19.8 | – |
| Gemma 4 E4B | 707 | 19.1 | 41.8 |
| Qwen3.5 9B | 487 | 11.8 | 21.6 |
| Qwen3.5 4B | 896 | 21.0 | 35.8 |
Against llama.cpp on the same 125H and the same files:
Qwen3.5 4B, 2K prompt: 911 tok/s, against 337 (SYCL) and 242 (Vulkan).
Qwen3.8 27B, 2K prompt: 113, against 25 and 23.
A 34K-token code prompt on the 27B: 5.5 minutes to the first token, against 27 and 40.
Perplexity is equal or lower on every model compared.
Other features
MoE models larger than VRAM or RAM, their experts streamed from RAM or the SSD.
MTP and DFlash2 speculative decoding that never changes the text.
Prompt and session caches.
Batched slots, which also work over a split model.
A GPU that hangs fails the request; the server stays up.
Wanted:
A multi-GPU machine with every card in its own PCIe 5.0 x16 slot at full x16, not bifurcated. That rules out x8/x8 splits and chipset slots; a workstation or server board (Threadripper, Xeon, EPYC) has the lanes for it.
Split models add their partial sums over the PCIe links every layer, so the link sets the pace.
All our multi-card runs so far are on slower links:
4× B60 on Windows: no peer access, so the sums go through host memory;
2× B60 on Linux: PCIe 4.0 x8 each, the sums straight from GPU to GPU;
one card in an x4 slot: there two cards run at half the speed of one.
A full-bandwidth PCIe 5.0 machine would show what the split model really reaches. Linux with GPU-to-GPU sums is best, and Windows is welcome too.
Also wanted: B580s, more B70s, and Lunar Lake / Panther Lake laptops. The B390 above is one user’s machine so far.
All tuning beyond the 125H comes from users’ runs, collected in issue rizagp/inferzero#4. The goal is a 3,000 tok/s 4K prompt for the 27B on one B70.
Owners of several cards with lots of RAM: DeepSeek runs are very welcome.
Get it: https://github.com/rizagp/inferzero/releases/latest (2026.11.16.07). On Windows, run zero-launcher.cmd. On Linux, unpack the tar.gz and run ./zero-launcher.sh.