[Performance][Qwen3-next] Decrease huge CPU overhead

[vadiklyutiy]

Proposal to improve performance

Qwen3-next has a sufficient CPU overhead even with pretty big (more than 512 - upper bound for cudagraph size) batch sizes.

I took batch size 1024 for demonstration. For that used

vllm bench serve --backend vllm --model Qwen/Qwen3-Next-80B-A3B-Instruct --endpoint /v1/completions --dataset-name random --random-input 32 --random-output 1024 --max-concurrency 1024 --num-prompt 1024 --ignore-eos

where the most time we spend on decoding 1024 request in 1024-size batches.

For first run

VLLM_USE_FLASHINFER_MOE_FP16=1  vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct -tp 4 --enable-expert-parallel --no-enable-prefix-caching 

it shows around 13900 gen tokens/s.

If capture cudagraph up to 1024 with --cuda-graph-sizes=1024

VLLM_USE_FLASHINFER_MOE_FP16=1  vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct -tp 4 --enable-expert-parallel --no-enable-prefix-caching --cuda-graph-sizes=1024

the performance is dramatically better - more than 21000 gen tokens/s.

Below nsys profile. B200.

on screenshot
orange - full attn
yellow - GDN attn
red - MoE (in the top row, pls ignore red in the second row)

As we can see there a lot of GPU idle time. The main reason GDN - we spend more time in CPU than GPU here.
(MoE also spend sufficient time in CPU but it is still 20-25% less).

I don't have specific way how we can speed up CPU time of GSN attn. It requires further investigation.
For now just fix a problem.

[ZJY0516]

vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct-FP8

It seems that qwen3-next does not run in piece cuda graph on prefill stage when using FULL_AND_PIECEWISE cuda graph mode

For comparison, this is profile result of qwen3-4B prefill stage.

[ZJY0516]

@vadiklyutiy To help us benchmark accurately, could you share the exact command used for your profiling results? Specifically, was it run with a batch size of 1024 and without CUDA graphs?

[vadiklyutiy]

It seems that qwen3-next does not run in piece cuda graph on prefill stage when using FULL_AND_PIECEWISE cuda graph mode

this run with a lot of concurrency. batch size most time is 1024 (decoding together 1024 requests). cudagraph max capture size is 512 and we don't use cuda graph for this case.

In the second run I added --cuda-graph-sizes=1024 and this "fix" the problem. But infinity increase of cudagraph is a bit not good because increase memory consumption. Although greater --cuda-graph-sizes might be a suitable workaround for some cases.

[vadiklyutiy]

@vadiklyutiy To help us benchmark accurately, could you share the exact command used for your profiling results? Specifically, was it run with a batch size of 1024 and without CUDA graphs?

I used exactly 3 cmds mentioned in the first message.