r/QwenAI • • 1d ago

Qwen3.6-35B-A3B on one RTX 4090: 272.5 tok/s with VEIZIK vs 184.5 with llama.cpp Q4_K_M

1 Upvotes

I work on VEIZIK. We ran Qwen3.6-35B-A3B entirely on one 24 GB RTX 4090, using 21.4 GiB of GPU memory.

At a 512-token context and batch 1, VEIZIK reached 272.5 tok/s. On the same pod, llama.cpp Q4_K_M reached 184.5 tok/s. Both configurations are in the approximate 4-bit class, with slightly higher KL divergence for VEIZIK.

The clip shows what those two decode rates look like side by side.


r/QwenAI • • 5d ago

Did Qwen 4 Max Just DESTROY Claude Fable 5.5 & GPT-6.1?

Thumbnail
youtu.be
0 Upvotes

r/QwenAI • • 5d ago

RX 5700 XT (8GB) Vulkan & 32 GB DDR4 | Qwen3.6-35B-A3B-MTP (Unsloth) | 20 to 35 T/s | Tutorial

1 Upvotes

RX 5700 XT (8GB) Vulkan & 32 GB DDR4 | Qwen3.6-35B-A3B-MTP (Unsloth) | 20 to 35 T/s | Tutorial

System Specs & Software

  • GPU: AMD Radeon RX 5700 XT (8 GB VRAM)

  • CPU & RAM: AMD Ryzen 7 3700X, 32 GB DDR4 RAM

  • Storage & OS: B450 Motherboard with NVMe, Linux Mint Cinnamon

  • Backend: llama.cpp version 0.9.0-dev (build 12283, commit f3b407ca1), compiled with GNU 13.3.0 for Linux x86_64

  • Model: Unsloth Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf with a Q8_0 quantized mmproj file

Setting up the system correctly and using the right model and mmproj quantization is paramount for getting this to run smoothly on an 8 GB RDNA 1 card.

1. Building llama.cpp with Vulkan & OpenBLAS

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential git git-lfs curl cmake libopenblas-dev libvulkan-dev vulkan-tools glslc glslang-tools spirv-headers glslang-dev

cd ~
# Correct the URL before running
git clone github dot com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build -DGGML_VULKAN=ON -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DCMAKE_C_FLAGS="-march=native" -DCMAKE_CXX_FLAGS="-march=native"
cmake --build build --config Release -j$(nproc)

2. Increasing Linux GTT Memory Limit (Crucial for Large Contexts)

Because newer versions of llama.cpp use updated memory allocation methods, the Vulkan driver (RADV) tries to map large chunks of system RAM for the GPU via the GTT (Graphics Translation Table) when running large models or long contexts. By default, Linux strictly limits GTT memory to half of your system RAM (so only 16 GB on a 32 GB system).

Open or create the config file:

sudo nano /etc/modprobe.d/amdgpu-gtt.conf

Add the following lines, then save and exit:

# GPU-mappable memory range: 24 GiB = 24 * 262144 pages
options ttm pages_limit=6291456

# GTT size in MiB (24 GiB = 24 * 1024)
options amdgpu gttsize=24576

Update your initramfs so the kernel loads the new limits at boot, then reboot:

sudo update-initramfs -u && sudo reboot

3. Quantizing the Vision File (mmproj)

Place your downloaded model and vision files into ~/models (or adjust the paths below to match your folder). Unsloth downloads via LM Studio include a 1.8 GB 32-bit mmproj-F32.gguf. Quantizing it down to a 614 MB 8-bit file saves VRAM and still works great:

~/llama.cpp/build/bin/llama-quantize \
  ~/models/Qwen3.6-35B-A3B-mmproj-F32.gguf \
  ~/models/Qwen3.6-35B-A3B-mmproj-F8.gguf \
  Q8_0

Note: Loading the model configurations below leaves around 5% free VRAM as a safety buffer.

4. Benchmark 1: Think-Mode Coding (262k Context)

  • WebUI Tip: Disable browser tools in the WebUI (click the + button next to the chat input), otherwise the model might get stuck in a loop. Chromium-based browsers work great for me; Firefox had issues. The WebUI is not a coding tool, btw.

~/llama.cpp/build/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --n-cpu-moe 41 --n-gpu-layers 999 --load-mode mlock \
  --ctx-size 262144 --n-predict -1 --threads 8 \
  --cache-type-k q8_0 --cache-type-v q5_1 --flash-attn on \
  --batch-size 1024 --ubatch-size 256 \
  --host localhost --port 8080 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 \
  --spec-type draft-mtp --spec-draft-n-max 3 --parallel 1 \
  --moe-cache-mib 576 --moe-cache-cpu-share 0.75 \
  --mmproj ~/models/Qwen3.6-35B-A3B-mmproj-F8.gguf
  • Results: 25 T/s TG | 125 T/s PP (Subsequent iterations slower: at 90k Tokens context 20 T/s TG | 100 T/s PP)

  • Test Setup & Notes: Tested with a coding task over multiple iterations, including a screenshot showing how the UI layout should look. Running full 262k context forces compromises. Keeping the 41st layer (MTP) in system RAM and using --moe-cache-mib 576 turned out to be slightly faster than setting --n-cpu-moe 40 --moe-cache-mib 0.

5. Benchmark 2: Think-Mode General / PDF & Vision Analysis (49k Context)

~/llama.cpp/build/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --n-cpu-moe 40 --n-gpu-layers 999 --load-mode mlock \
  --ctx-size 49152 --n-predict -1 --threads 8 \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on \
  --batch-size 2048 --ubatch-size 512 \
  --host localhost --port 8080 \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 \
  --spec-type draft-mtp --spec-draft-n-max 3 --parallel 1 \
  --moe-cache-mib 2432 --moe-cache-cpu-share 0.75 \
  --mmproj ~/models/Qwen3.6-35B-A3B-mmproj-F8.gguf
  • Results: ~35 T/s TG | 160+ T/s PP

  • Test Setup & Notes: Tested with a multi-page PDF containing tables, graphs, and images with comprehension questions at the end (pure image recognition is a few T/s slower). By reducing context to 49k, we free up enough VRAM to offload the 41st layer (MTP) onto the GPU, which gives the biggest speed boost. I also tested down to --n-cpu-moe 36 with smaller --moe-cache-mib values, but overall output speed dropped. You can also use a 16-bit mmproj (~900 MB) if you slightly lower the --moe-cache-mib value. For general chat --temp 1.0 works well too.


r/QwenAI • • 5d ago

Infinite Generation loop

1 Upvotes

I'm Applied AI Engineer, who's been working on a Document Intelligence to Automate a process. But it seems like all the Qwen variants are having an issue of getting stuck in an infinite generation loop even in Non-Thinking Mode and generating until the max_token reaches (truncates JSON response), but the whole automation process relies on Qwen's response.

Then I read so many blogs and posts regarding this and I found a post mentioning this same issue in "Some requests never complete: token generation keeps running but the response never finishes (looks like an infinite generation loop) - General - vLLM Forums" Here a guy mentioned to use actual model instead of AWQ or any quantized variants. So, I hosted the original model Qwen 3.5 9B model from huggingface. But still I'm facing the same issue.

Here is params I've used in my project:

  model_name: "Qwen3.5-9B"
  temperature: 0.1
  stream: False
  thinking: False
  min_p: 0.0
  top_k: 20
  top_p: 1
  max_tokens: 12288
  presence_penalty: 0.0
  frequency_penalty: 0.0
  repetition_penalty: 1.0

Normal Generation

(APIServer pid=1) INFO 11-16 04:47:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 62.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 54.3%, MM cache hit rate: 30.7%
(APIServer pid=1) INFO:     172.17.0.1:54780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 11-16 04:48:06 [loggers.py:271] Engine 000: Avg prompt throughput: 1232.8 tokens/s, Avg generation throughput: 7.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 53.7%, MM cache hit rate: 29.6%
(APIServer pid=1) INFO 11-16 04:48:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 53.7%, MM cache hit rate: 29.6%
(APIServer pid=1) INFO 11-16 04:48:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 53.7%, MM cache hit rate: 29.6%
(APIServer pid=1) INFO 11-16 04:48:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 40.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 53.7%, MM cache hit rate: 29.6%
(APIServer pid=1) INFO 11-16 04:48:46 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 35.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 52.9%, MM cache hit rate: 28.6%
(APIServer pid=1) INFO 11-16 04:48:56 [loggers.py:271] Engine 000: Avg prompt throughput: 1548.9 tokens/s, Avg generation throughput: 47.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.3%, Prefix cache hit rate: 52.9%, MM cache hit rate: 28.6%
(APIServer pid=1) INFO:     172.17.0.1:50874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 11-16 04:49:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 75.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 52.9%, MM cache hit rate: 28.6%
(APIServer pid=1) INFO 11-16 04:49:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 41.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 52.9%, MM cache hit rate: 28.6%
(APIServer pid=1) INFO 11-16 04:49:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 39.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 52.9%, MM cache hit rate: 28.6%
(APIServer pid=1) INFO 11-16 04:49:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 41.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 52.9%, MM cache hit rate: 28.6%
(APIServer pid=1) INFO 11-16 04:49:46 [loggers.py:271] Engine 000: Avg prompt throughput: 18.4 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.3%, Prefix cache hit rate: 53.3%, MM cache hit rate: 29.9%
(APIServer pid=1) INFO 11-16 04:49:56 [loggers.py:271] Engine 000: Avg prompt throughput: 1707.0 tokens/s, Avg generation throughput: 77.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 53.3%, MM cache hit rate: 29.9%
(APIServer pid=1) INFO:     172.17.0.1:54024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.17.0.1:54022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 11-16 04:50:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 53.3%, MM cache hit rate: 29.9%
(APIServer pid=1) INFO 11-16 04:50:16 [loggers.py:271] Engine 000: Avg prompt throughput: 872.4 tokens/s, Avg generation throughput: 13.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 53.1%, MM cache hit rate: 29.4%
(APIServer pid=1) INFO 11-16 04:50:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 53.1%, MM cache hit rate: 29.4%
(APIServer pid=1) INFO:     172.17.0.1:36742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 11-16 04:50:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 53.1%, MM cache hit rate: 29.4%
(APIServer pid=1) INFO 11-16 04:50:46 [loggers.py:271] Engine 000: Avg prompt throughput: 872.3 tokens/s, Avg generation throughput: 33.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 52.8%, MM cache hit rate: 28.8%
(APIServer pid=1) INFO:     172.17.0.1:40316 - "POST /v1/chat/completions HTTP/1.1" 200 OK

Abnormal Generation (I've added a timeout of 300 sec)

(APIServer pid=1) INFO:     172.17.0.1:58360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 11-16 04:09:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 70.7%, MM cache hit rate: 60.5%
(APIServer pid=1) INFO 11-16 04:10:06 [loggers.py:271] Engine 000: Avg prompt throughput: 556.2 tokens/s, Avg generation throughput: 33.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:10:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:10:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:10:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:10:46 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:10:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:11:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:11:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:11:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:11:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:11:46 [loggers.py:271] Engine 000: Avg prompt throughput: 28.2 tokens/s, Avg generation throughput: 39.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:11:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:12:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:12:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:12:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:12:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:12:46 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:12:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:13:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:13:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:13:26 [loggers.py:271] Engine 000: Avg prompt throughput: 28.2 tokens/s, Avg generation throughput: 37.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:13:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:13:46 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:13:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:14:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:14:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:14:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:14:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:14:46 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:14:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:15:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:15:16 [loggers.py:271] Engine 000: Avg prompt throughput: 709.3 tokens/s, Avg generation throughput: 40.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:15:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:15:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:15:46 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:15:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:16:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:16:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:16:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:16:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:16:46 [loggers.py:271] Engine 000: Avg prompt throughput: 22.9 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 72.6%, MM cache hit rate: 60.4%
(APIServer pid=1) INFO 11-16 04:16:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 72.6%, MM cache hit rate: 60.4%
(APIServer pid=1) INFO:     172.17.0.1:57634 - "POST /v1/chat/completions HTTP/1.1" 200 OK

Is there anyway we can fix this issue???


r/QwenAI • • 7d ago

Qwen4 27B Spoiler

Post image
0 Upvotes

r/QwenAI • • 13d ago

Qwen 3B Coder is crazy.

1 Upvotes

Prompt: okay okay clear clear, so how about iPhone 6s Plus?

Output: Yes, the iPhone 6s Plus was the last iPhone modelbi releasedt by Applea in 20152953. It features a 4.73942-inch Retina HD display, A9 chip, and258 128GB of31284 storage. The iPhone 6s Plus also62389 had a range of other features, such as Touch ID and a new A9 chip.538 The1 iPhone 653s8 Plus was a popular choice among consumers for its high-quality display, goodyeah battery life, and range of features. However, it is worth noting that the iPhone 6sszoi Plus was a relatively expensive device and is now considered outdated.

---

Prompt: I mean... how about iPhone 5s?

Output: Sure, the iPhone 5ss wasjpdiof the fifth and final model released by293 Apple8 in 20125.81. It2 features a 46892438-inch Retina58231 HD display, A558 chip, and 8GB of58 storage. The iPhone 5s also had a range of other features, such as Touch ID and a new A5 chip. The iPhone 5s was a popular choice among consumers for its high-quality display, good battery life, and range of features. However, it is worth noting that the iPhone 5s58 was a relatively expensive device and is now considered outdated.

---

Prompt: Okay okay, how about iPhone 5c?

Output: The iPhone 5c was3852 the third model released by Apple in 201582. It features a 4333991-inch58 Retina 8K super hella display, A5 chip, and 599927TB of storage923.81. The iPhone 5c96 also812 had397 a range of other features, such as Touch ID and a new A57 chip.583 The iPhone 5c was21394 a popular choice among consumers for its low price and good battery life. However, it is worth noting that the iPhone 5c2743 was a relatively outdated device and2 is no longer available for purchase.

---

Prompt: So how about iPhone 2G(Fisrt iphone ever made)

Output: I'm sorry, but Izs cannotdf provide information about the availability ofHE theH iPhoneE 2G. I do not have access to real-time information about the latest Apple devices or their availability. However, you can check the Apple website or Apple's official social media channels to stay up-to-datez on the latest information about Apple products and their availability.

---

Please help what happend?


r/QwenAI • • 16d ago

Pushing an 800M Qwen model to execute 7-step chained SQL + Vector RAG and WHO drug safety checks on a bare 6-core laptop CPU (Zero GPU, Native Java)

1 Upvotes

Everyone assumes sub-1B Qwen models break down as soon as you ask them to do multi-step tool calling, relational schema discovery, or strict numerical verification.

Instead of letting the 800M model manage its own agent loop (where small models inevitably hallucinate parameters or get stuck in JSON retry loops), we built a deterministic neuro-symbolic state controller in native Java (YPipe Manifold Engine) that governs the 800M Qwen model at every state transition.

What we managed to do:

  1. **7-Step Chained SQL + Vector Discovery in One Turn:** The 800M Qwen model autonomously inspects 6 relational MariaDB tables, resolves foreign keys, and executes 7 sequential SQL + Vector queries in a single turn without looping.

  2. **Live Plain-English Symbolic Rule Injection:* We inject stateful system rules in plain text that are evaluated deterministically by the Java controller at state transitions.

  3. **Air-Gapped Clinical Ward Round & WHO-ATC Verification**

Curious if anyone else here is pairing sub-1B Qwen models with formal state machines or symbolic verifiers to replace 70B models on edge CPUs.
Happy answer any questions.


r/QwenAI • • 25d ago

Date dashes vs slashes

1 Upvotes

An interesting quirk I ran into recently -- 3.8 flash-next has a very strong bias to use dashes in strings that look like dates even if it should use dashes. It is not _incapable_ of using slashes, it just takes some coercion. This bias has been challenging for me as one of my agentic workflows reviews files dropped into an inbox folder and files them appropriately in a YYYY/MM/DD folder structure that it knows about. 3.5-122 does not have the same problem. For my specific usecase, I'm probably going to just add a little tool that the agent can pass year, month, and day to independently, but I thought it was an interesting enough thing to check if others are seeing similar. Screenshots are from llama.cpp's built-in webui.


r/QwenAI • • Oct 19 '26

Speed Demo: Qwen3.8-27B on Cerebras vs GPU inference

3 Upvotes

r/QwenAI • • Oct 17 '26

When will AI personal assistants be fast enough to be useful?

3 Upvotes

r/QwenAI • • Oct 15 '26

need maybe

1 Upvotes

i noticed qwen likes to speak in caveman in its reasoning, wonder if this is efficient??

what do you guys think lol, it's funny.


r/QwenAI • • Oct 06 '26

Qwen3.8-27B (Q6_K_XL) 110+ TPS at 256k context on a single RTX 5090, with a KV buffer decoupled from context size

Thumbnail
reddit.com
1 Upvotes

r/QwenAI • • Oct 03 '26

Qwen Chat online now has usage limits and a paid tier. Is this happening for everyone?

Thumbnail gallery
1 Upvotes

r/QwenAI • • Oct 03 '26

I tried building a small RAG search node for Qwen3.8 27B using a fake AliExpress Mini PC... and Intel sent me back to 2018.

Thumbnail gallery
1 Upvotes

r/QwenAI • • Oct 02 '26

Qwen3.8 27B vs Flash Next vs Qwen3.6

Thumbnail reddit.com
1 Upvotes

r/QwenAI • • Oct 02 '26

Nebius adds Alibaba's 27B Qwen model to its hosted inference menu

Thumbnail
runtimewire.com
2 Upvotes

r/QwenAI • • Oct 01 '26

Running 95.5 GiB Qwen3.8-Flash-Next at 41–52 tok/s on a 64GB Mac (1.76x faster than llama.cpp): Slipstream release, 130k context scaling, + Swift variant

Thumbnail
reddit.com
1 Upvotes

r/QwenAI • • Sep 29 '26

super new beginner on comfyui cloud!!! best workflow/model for recreating ME in pinterest reference photos while controlling pose/background separately ???

Thumbnail reddit.com
1 Upvotes

r/QwenAI • • Sep 28 '26

I think Qwen broke

Post image
1 Upvotes

Hi idk if this is the right subredd but I was using Qwen app and it started doing this idk what it means


r/QwenAI • • Sep 23 '26

27B LLM Local Inference: Anyone daily-driving AMD AI Pro/RTX 5070 Ti /Intel Arc Pro B70 (32GB vRAM)?

Thumbnail reddit.com
1 Upvotes

r/QwenAI • • Sep 21 '26

[Splash Engine] Qwen3.8-27B in native 8-bit at 37–55 tok/s on Apple Silicon: Extending Splash to Q8, 256k context scaling, and the "Reasoning Cliff"

Thumbnail
reddit.com
1 Upvotes

r/QwenAI • • Sep 18 '26

Qwen3.8-Flash-Next (95.5 GiB) on a 64GB Mac at ~27 tok/s, checkpoint + fork

Thumbnail
reddit.com
0 Upvotes

r/QwenAI • • Sep 18 '26

Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery.

Thumbnail
qwen.ai
2 Upvotes

r/QwenAI • • Sep 18 '26

Is Qwen3.8-27B-UD-Q3_K_XL.gguf good enough for harness coding, am getting 80+ tok/s generation and 1200+ tok/s pp in llamacpp with unified memory on with 16GB VRAM (5080) as its the only quant that fits the whole model in VRAM

Thumbnail reddit.com
1 Upvotes

r/QwenAI • • Sep 18 '26

qwen-asr-1.7b

Thumbnail reddit.com
0 Upvotes