[BUG] AttributeError: 'torch_dtype' Not Found in VLLMModelConfig When dtype Not Specified

[fabfish]

When using vllm without specifying a dtype, I encountered the following error:

AttributeError: 'VLLMModelConfig' object has no attribute 'torch_dtype'

I was running open-r1 MATH500 test:

MODEL_ARGS="pretrained=$MODEL,max_model_length=32768,gpu_memory_utilization=0.8,data_parallel_size=$NUM_GPUS,generation_parameters={max_new_tokens:32768,temperature:0.7,top_p:0.95}"

OUTPUT_DIR=data/evals/$MODEL

lighteval vllm $MODEL_ARGS "custom|math_500|0|0" \
    --custom-tasks src/open_r1/evaluate_math.py \
    --use-chat-template \
    --output-dir $OUTPUT_DIR \
    --save-details 

This error originates from this code section:
utils.py#L55

if config is not None:
    return config.torch_dtype

Proposed Solution

To improve robustness, check if config.torch_dtype exists before accessing it / Update the documentation to ensure dtype must be passed to avoid this error.

[NathanHB]

hey ! thanks for the feedback, when trying to replicate with latest version of lighteval, i found that vllm did not accept None as default. I fixed by using bfloat16 as default dtype if not submitted by user. #638