I'm Applied AI Engineer, who's been working on a Document Intelligence to Automate a process. But it seems like all the Qwen variants are having an issue of getting stuck in an infinite generation loop even in Non-Thinking Mode and generating until the max_token reaches (truncates JSON response), but the whole automation process relies on Qwen's response.
Then I read so many blogs and posts regarding this and I found a post mentioning this same issue in "Some requests never complete: token generation keeps running but the response never finishes (looks like an infinite generation loop) - General - vLLM Forums" Here a guy mentioned to use actual model instead of AWQ or any quantized variants. So, I hosted the original model Qwen 3.5 9B model from huggingface. But still I'm facing the same issue.
Here is params I've used in my project:
model_name: "Qwen3.5-9B"
temperature: 0.1
stream: False
thinking: False
min_p: 0.0
top_k: 20
top_p: 1
max_tokens: 12288
presence_penalty: 0.0
frequency_penalty: 0.0
repetition_penalty: 1.0
Normal Generation
(APIServer pid=1) INFO 11-16 04:47:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 62.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 54.3%, MM cache hit rate: 30.7%
(APIServer pid=1) INFO: 172.17.0.1:54780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 11-16 04:48:06 [loggers.py:271] Engine 000: Avg prompt throughput: 1232.8 tokens/s, Avg generation throughput: 7.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 53.7%, MM cache hit rate: 29.6%
(APIServer pid=1) INFO 11-16 04:48:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 53.7%, MM cache hit rate: 29.6%
(APIServer pid=1) INFO 11-16 04:48:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 53.7%, MM cache hit rate: 29.6%
(APIServer pid=1) INFO 11-16 04:48:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 40.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 53.7%, MM cache hit rate: 29.6%
(APIServer pid=1) INFO 11-16 04:48:46 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 35.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 52.9%, MM cache hit rate: 28.6%
(APIServer pid=1) INFO 11-16 04:48:56 [loggers.py:271] Engine 000: Avg prompt throughput: 1548.9 tokens/s, Avg generation throughput: 47.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.3%, Prefix cache hit rate: 52.9%, MM cache hit rate: 28.6%
(APIServer pid=1) INFO: 172.17.0.1:50874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 11-16 04:49:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 75.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 52.9%, MM cache hit rate: 28.6%
(APIServer pid=1) INFO 11-16 04:49:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 41.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 52.9%, MM cache hit rate: 28.6%
(APIServer pid=1) INFO 11-16 04:49:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 39.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 52.9%, MM cache hit rate: 28.6%
(APIServer pid=1) INFO 11-16 04:49:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 41.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 52.9%, MM cache hit rate: 28.6%
(APIServer pid=1) INFO 11-16 04:49:46 [loggers.py:271] Engine 000: Avg prompt throughput: 18.4 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.3%, Prefix cache hit rate: 53.3%, MM cache hit rate: 29.9%
(APIServer pid=1) INFO 11-16 04:49:56 [loggers.py:271] Engine 000: Avg prompt throughput: 1707.0 tokens/s, Avg generation throughput: 77.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 53.3%, MM cache hit rate: 29.9%
(APIServer pid=1) INFO: 172.17.0.1:54024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 172.17.0.1:54022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 11-16 04:50:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 53.3%, MM cache hit rate: 29.9%
(APIServer pid=1) INFO 11-16 04:50:16 [loggers.py:271] Engine 000: Avg prompt throughput: 872.4 tokens/s, Avg generation throughput: 13.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 53.1%, MM cache hit rate: 29.4%
(APIServer pid=1) INFO 11-16 04:50:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 53.1%, MM cache hit rate: 29.4%
(APIServer pid=1) INFO: 172.17.0.1:36742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 11-16 04:50:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 53.1%, MM cache hit rate: 29.4%
(APIServer pid=1) INFO 11-16 04:50:46 [loggers.py:271] Engine 000: Avg prompt throughput: 872.3 tokens/s, Avg generation throughput: 33.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 52.8%, MM cache hit rate: 28.8%
(APIServer pid=1) INFO: 172.17.0.1:40316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Abnormal Generation (I've added a timeout of 300 sec)
(APIServer pid=1) INFO: 172.17.0.1:58360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 11-16 04:09:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 70.7%, MM cache hit rate: 60.5%
(APIServer pid=1) INFO 11-16 04:10:06 [loggers.py:271] Engine 000: Avg prompt throughput: 556.2 tokens/s, Avg generation throughput: 33.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:10:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:10:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:10:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:10:46 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:10:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:11:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:11:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:11:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:11:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 69.6%, MM cache hit rate: 56.1%
(APIServer pid=1) INFO 11-16 04:11:46 [loggers.py:271] Engine 000: Avg prompt throughput: 28.2 tokens/s, Avg generation throughput: 39.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:11:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:12:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:12:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:12:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:12:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:12:46 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:12:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:13:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:13:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 71.2%, MM cache hit rate: 59.1%
(APIServer pid=1) INFO 11-16 04:13:26 [loggers.py:271] Engine 000: Avg prompt throughput: 28.2 tokens/s, Avg generation throughput: 37.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:13:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:13:46 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:13:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:14:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:14:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:14:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:14:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:14:46 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:14:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 72.6%, MM cache hit rate: 61.7%
(APIServer pid=1) INFO 11-16 04:15:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:15:16 [loggers.py:271] Engine 000: Avg prompt throughput: 709.3 tokens/s, Avg generation throughput: 40.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:15:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:15:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:15:46 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:15:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:16:06 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:16:16 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:16:26 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:16:36 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 71.2%, MM cache hit rate: 58.0%
(APIServer pid=1) INFO 11-16 04:16:46 [loggers.py:271] Engine 000: Avg prompt throughput: 22.9 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 72.6%, MM cache hit rate: 60.4%
(APIServer pid=1) INFO 11-16 04:16:56 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 42.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 72.6%, MM cache hit rate: 60.4%
(APIServer pid=1) INFO: 172.17.0.1:57634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Is there anyway we can fix this issue???