`max_num_batched_tokens` and `max_num_seqs values`

[isRambler]

Hello, because I am new to vllm, I want to know how to set the max_num_batched_tokens and max_num_seqs values in order to achieve maximum inference performance. What is the relationship between max_num_batched_tokens and max_num_seqs? Why do the output tokens appear when I set different max_num_batched_tokens and max_num_seqs? The totals may be inconsistent

[JasonZhu1313]

max_num_batched_tokens and max_num_seqs essentially determines the batch size at prefill stage - the first time when the model performs inference to predict the next token in a sequence. vLLM utilizes continuous batching to achieve high throughput. The underlying scheduler determines, at a certain step, the batched samples for inference based on the state of GPU memory and the availability of KV Cache blocks. The scheduler separates queries into three categories: Waiting Queue (sequences at prefill stage), Running Queue (sequences at decoding stage), Swapping Queue (if cpu / gpu swapping is enabled).

I have drawn a diagram to better illustrate the scheduling workflow and you can see max_num_batched_tokens and max_num_seqs determines the batch size of waiting queue:

If you set the max_num_batched_tokens or max_num_seqs with low value then the prefill batch size will be small (e.g., 1) which might not hurt performance, there is no one size fit all suggestion I guess, I think you can tweak the prefill batch size through these two knobs and use benchmark_serving.py in vLLM to determine what size leads to best performance.

[isRambler]

max_num_batched_tokens and max_num_seqs essentially determines the batch size at prefill stage - the first time when the model performs inference to predict the next token in a sequence. vLLM utilizes continuous batching to achieve high throughput. The underlying scheduler determines, at a certain step, the batched samples for inference based on the state of GPU memory and the availability of KV Cache blocks. The scheduler separates queries into three categories: Waiting Queue (sequences at prefill stage), Running Queue (sequences at decoding stage), Swapping Queue (if cpu / gpu swapping is enabled).

I have drawn a diagram to better illustrate the scheduling workflow and you can see max_num_batched_tokens and max_num_seqs determines the batch size of waiting queue:

If you set the max_num_batched_tokens or max_num_seqs with low value then the prefill batch size will be small (e.g., 1) which might not hurt performance, there is no one size fit all suggestion I guess, I think you can tweak the prefill batch size through these two knobs and use benchmark_serving.py in vLLM to determine what size leads to best performance.

OK, thanks for the explanation

[banyip]

Hello, because I am new to vllm, I want to know how to set the max_num_batched_tokens and max_num_seqs values in order to achieve maximum inference performance. What is the relationship between max_num_batched_tokens and max_num_seqs? Why do the output tokens appear when I set different max_num_batched_tokens and max_num_seqs? The totals may be inconsistent

Is there a final answer to set this variable?

[realJaydenCheng]

Hello, because I am new to vllm, I want to know how to set the max_num_batched_tokens and max_num_seqs values in order to achieve maximum inference performance. What is the relationship between max_num_batched_tokens and max_num_seqs? Why do the output tokens appear when I set different max_num_batched_tokens and max_num_seqs? The totals may be inconsistent

Is there a final answer to set this variable?

You can set these variables when initializing the LLM object:

from vllm import LLM
llm = LLM(
    model=LLM_PATH,
    max_num_batched_tokens=512*50,
    max_model_len=512*50,
    gpu_memory_utilization=0.3,
)

[ehartford]

can we have some kind of guidance or rule of thumb? How do we decide, practically speaking, what values to set in order to maximize performance?

[zengqingfu1442]

You can set these variables when initializing the LLM object:

from vllm import LLM
llm = LLM(
model=LLM_PATH,
max_num_batched_tokens=51250,
max_model_len=51250,
gpu_memory_utilization=0.3,
)

What if i set a larger value of max_num_batched_tokens than max_model_len? i think that the max_model_len should be less than or equal the context length of the llm.

[mrtpk]

Please check out autotune.sh

vllm/benchmarks/auto_tune.sh

Line 20 in ba7ba35

# NUM_SEQS_LIST: a list of `max-num-seqs` you want to loop with.

to get the best params for a model for different conditions.

[mrtpk]

max_num_batched_tokens and max_num_seqs essentially determines the batch size at prefill stage - the first time when the model performs inference to predict the next token in a sequence. vLLM utilizes continuous batching to achieve high throughput. The underlying scheduler determines, at a certain step, the batched samples for inference based on the state of GPU memory and the availability of KV Cache blocks. The scheduler separates queries into three categories: Waiting Queue (sequences at prefill stage), Running Queue (sequences at decoding stage), Swapping Queue (if cpu / gpu swapping is enabled).

I have drawn a diagram to better illustrate the scheduling workflow and you can see max_num_batched_tokens and max_num_seqs determines the batch size of waiting queue:

If you set the max_num_batched_tokens or max_num_seqs with low value then the prefill batch size will be small (e.g., 1) which might not hurt performance, there is no one size fit all suggestion I guess, I think you can tweak the prefill batch size through these two knobs and use benchmark_serving.py in vLLM to determine what size leads to best performance.

@JasonZhu1313, Thank you for the diagram. It helps us understand what is happening inside. But I still could not understand the mathematical relationship between batch_sz, max_num_batched_tokens, and max_num_seqs. Is there a math formula that helps calculate batch_sz from these values?