[Bug][Failing Test]: Multimodal Test (Standard) - QwenVL

[robertgshaw2-redhat]

Your current environment

• There appears to be an issue with QwenVL
• Link: https://buildkite.com/vllm/ci/builds/20415#0196f100-f857-45bf-ae5a-a9e5b90e7f62/199-9633
• Logs:

[2025-05-21T05:05:21Z] FAILED models/multimodal/generation/test_qwen2_vl.py::test_qwen2_vl_image_embeddings_input[10-128-half-size_factors0-Qwen/Qwen2-VL-2B-Instruct] - vllm.worker.model_runner_base.InputProcessingError: Failed to prepare inputs for sequence group with request id: 0, Error: list index out of range
--
  | [2025-05-21T05:05:21Z] FAILED models/multimodal/generation/test_qwen2_vl.py::test_qwen2_vl_image_embeddings_input[10-128-half-size_factors1-Qwen/Qwen2-VL-2B-Instruct] - vllm.worker.model_runner_base.InputProcessingError: Failed to prepare inputs for sequence group with request id: 0, Error: list index out of range
  | [2025-05-21T05:05:21Z] FAILED models/multimodal/generation/test_qwen2_vl.py::test_qwen2_vl_image_embeddings_input[10-128-half-size_factors2-Qwen/Qwen2-VL-2B-Instruct] - vllm.worker.model_runner_base.InputProcessingError: Failed to prepare inputs for sequence group with request id: 0, Error: list index out of range
  | [2025-05-21T05:05:21Z] FAILED models/multimodal/generation/test_qwen2_vl.py::test_qwen2_vl_multiple_image_embeddings_input[10-128-half-size_factors1-Qwen/Qwen2-VL-2B-Instruct] - vllm.worker.model_runner_base.InputProcessingError: Failed to prepare inputs for sequence group with request id: 0, Error: list index out of range
  | [2025-05-21T05:05:21Z] FAILED models/multimodal/generation/test_qwen2_vl.py::test_qwen2_vl_multiple_image_embeddings_input[10-128-half-size_factors2-Qwen/Qwen2-VL-2B-Instruct] - vllm.worker.model_runner_base.InputProcessingError: Failed to prepare inputs for sequence group with request id: 0, Error: list index out of range
  | [2025-05-21T05:05:21Z] FAILED models/multimodal/generation/test_qwen2_vl.py::test_qwen2_vl_multiple_image_embeddings_input[10-128-half-size_factors3-Qwen/Qwen2-VL-2B-Instruct] - vllm.worker.model_runner_base.InputProcessingError: Failed to prepare inputs for sequence group with request id: 0, Error: list index out of range
  | [2025-05-21T05:05:21Z] FAILED models/multimodal/generation/test_qwen2_vl.py::test_qwen2_vl_video_embeddings_input[10-128-half-size_factors0-Qwen/Qwen2-VL-2B-Instruct] - vllm.worker.model_runner_base.InputProcessingError: Failed to prepare inputs for sequence group with request id: 0, Error: list index out of range
  | [2025-05-21T05:05:21Z] FAILED models/multimodal/generation/test_qwen2_vl.py::test_qwen2_vl_video_embeddings_input[10-128-half-size_factors1-Qwen/Qwen2-VL-2B-Instruct] - vllm.worker.model_runner_base.InputProcessingError: Failed to prepare inputs for sequence group with request id: 0, Error: list index out of range
  | [2025-05-21T05:05:21Z] FAILED models/multimodal/generation/test_qwen2_vl.py::test_qwen2_vl_video_embeddings_input[10-128-half-size_factors2-Qwen/Qwen2-VL-2B-Instruct] - vllm.worker.model_runner_base.InputProcessingError: Failed to prepare inputs for sequence group with request id: 0, Error: list index out of range
  | [2025-05-21T05:05:21Z] FAILED models/multimodal/generation/test_ultravox.py::test_models_with_multiple_audios[vllm_kwargs0-5-128-half] - RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
  | [2025-05-21T05:05:21Z] = 10 failed, 52 passed, 32 skipped, 769 deselected, 71 warnings in 2395.13s (0:39:55) =

🐛 Describe the bug

See above

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

It should be fixed by #18456 already.

For example: https://buildkite.com/vllm/ci/builds/20451/steps?jid=0196f2b6-f999-4486-8984-91fa1d7d8bd1

[Datta0]

I'm facing this yet again on Google Colab Nvidia T4 GPUs. Here's a brief snapshot of my setup
Ubuntu 22.04 LTS. It works when I downgrade vLLM to 0.9.2 and trition to 3.2.0.

torch                                 2.7.1
torchvision                       0.22.1
triton                                3.2.0
vllm                                  0.10.1

...

outputs = model.vllm_engine.generate({"prompt": prompt, "multi_modal_data": {"image": sloth_image}}, sampling_params)

..

Adding requests: 100%
 1/1 [00:08<00:00,  8.64s/it]
Processed prompts:   0%
 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]

---------------------------------------------------------------------------

IndexError                                Traceback (most recent call last)

[/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py](https://localhost:8080/#) in _prepare_model_input_tensors(self, seq_group_metadata_list, finished_requests_ids)
   1180             try:
-> 1181                 self.builder.add_seq_group(seq_group_metadata)
   1182             except Exception as e:

17 frames

[/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py](https://localhost:8080/#) in add_seq_group(self, seq_group_metadata)
    740         for per_seq_group_fn in self.per_seq_group_compute_fns:
--> 741             per_seq_group_fn(inter_data, seq_group_metadata)
    742 

[/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py](https://localhost:8080/#) in _compute_multi_modal_input(self, inter_data, seq_group_metadata)
    693                 mrope_input_positions, mrope_position_delta = \
--> 694                     MRotaryEmbedding.get_input_positions(
    695                         token_ids,

[/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/rotary_embedding/mrope.py](https://localhost:8080/#) in get_input_positions(cls, input_tokens, hf_config, image_grid_thw, video_grid_thw, second_per_grid_ts, context_len, seq_len, audio_feature_lengths, use_audio_in_video)
    346         llm_positions, mrope_position_delta = \
--> 347             cls.get_input_positions_tensor(
    348                 input_tokens=input_tokens,

[/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/rotary_embedding/mrope.py](https://localhost:8080/#) in get_input_positions_tensor(cls, input_tokens, hf_config, image_grid_thw, video_grid_thw, second_per_grid_ts, context_len, seq_len, audio_feature_lengths, use_audio_in_video)
    396         else:
--> 397             return cls._vl_get_input_positions_tensor(
    398                 input_tokens=input_tokens,

[/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/rotary_embedding/mrope.py](https://localhost:8080/#) in _vl_get_input_positions_tensor(cls, input_tokens, hf_config, image_grid_thw, video_grid_thw, second_per_grid_ts, context_len, seq_len)
    559                 t, h, w = (
--> 560                     image_grid_thw[image_index][0],
    561                     image_grid_thw[image_index][1],

IndexError: list index out of range


The above exception was the direct cause of the following exception:

InputProcessingError                      Traceback (most recent call last)

[/tmp/ipython-input-787508722.py](https://localhost:8080/#) in <cell line: 0>()
----> 1 outputs = model.vllm_engine.generate({"prompt": prompt, "multi_modal_data": {"image": sloth_image}}, sampling_params)
      2 print(outputs[0].outputs[0].text)

[/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py](https://localhost:8080/#) in inner(*args, **kwargs)
   1555                     )
   1556 
-> 1557             return fn(*args, **kwargs)
   1558 
   1559         return inner  # type: ignore

[/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/llm.py](https://localhost:8080/#) in generate(self, prompts, sampling_params, prompt_token_ids, use_tqdm, lora_request, priority)
    495         )
    496 
--> 497         outputs = self._run_engine(use_tqdm=use_tqdm)
    498         return self.engine_class.validate_outputs(outputs, RequestOutput)
    499 

[/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/llm.py](https://localhost:8080/#) in _run_engine(self, use_tqdm)
   1713         total_out_toks = 0
   1714         while self.llm_engine.has_unfinished_requests():
-> 1715             step_outputs = self.llm_engine.step()
   1716             for output in step_outputs:
   1717                 if output.finished:

[/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py](https://localhost:8080/#) in step(self)
   1219 
   1220             try:
-> 1221                 outputs = self.model_executor.execute_model(
   1222                     execute_model_req=execute_model_req)
   1223                 self._skip_scheduling_next_step = False

[/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py](https://localhost:8080/#) in execute_model(self, execute_model_req)
    145         self, execute_model_req: ExecuteModelRequest
    146     ) -> Optional[List[Union[SamplerOutput, PoolerOutput]]]:
--> 147         output = self.collective_rpc("execute_model",
    148                                      args=(execute_model_req, ))
    149         return output[0]

[/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py](https://localhost:8080/#) in collective_rpc(self, method, timeout, args, kwargs)
     56         if kwargs is None:
     57             kwargs = {}
---> 58         answer = run_method(self.driver_worker, method, args, kwargs)
     59         return [answer]
     60 

[/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py](https://localhost:8080/#) in run_method(obj, method, args, kwargs)
   3005     else:
   3006         func = partial(method, obj)  # type: ignore
-> 3007     return func(*args, **kwargs)
   3008 
   3009 

[/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py](https://localhost:8080/#) in execute_model(self, execute_model_req)
    391         start_time = time.perf_counter()
    392 
--> 393         inputs = self.prepare_input(execute_model_req)
    394         if inputs is None:
    395             return None

[/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py](https://localhost:8080/#) in prepare_input(self, execute_model_req)
    376                     broadcast_tensor_dict({}, src=0)
    377                 return None
--> 378             return self._get_driver_input_and_broadcast(execute_model_req)
    379         else:
    380             return self._get_worker_input_from_broadcast()

[/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py](https://localhost:8080/#) in _get_driver_input_and_broadcast(self, execute_model_req)
    338             execute_model_req=execute_model_req)
    339         model_input: ModelRunnerInputBase = (
--> 340             self.model_runner.prepare_model_input(
    341                 execute_model_req.seq_group_metadata_list,
    342                 execute_model_req.virtual_engine,

[/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py](https://localhost:8080/#) in prepare_model_input(self, seq_group_metadata_list, virtual_engine, finished_requests_ids)
   1599         If cuda graph is required, this API automatically pads inputs.
   1600         """
-> 1601         model_input = self._prepare_model_input_tensors(
   1602             seq_group_metadata_list, finished_requests_ids)
   1603         if get_pp_group().is_last_rank:

[/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py](https://localhost:8080/#) in _prepare_model_input_tensors(self, seq_group_metadata_list, finished_requests_ids)
   1182             except Exception as e:
   1183                 # Raise an exception that tracks the ID of the bad request
-> 1184                 raise InputProcessingError(seq_group_metadata.request_id,
   1185                                            str(e)) from e
   1186 

InputProcessingError: Failed to prepare inputs for sequence group with request id: 0, Error: list index out of range

[DarkLight1337]

Looks like you're still using V0 Engine. Can you try using V1 Engine?

[Datta0]

T4 doesn't support V1 engine right

[DarkLight1337]

Only FP32 inference is supported for T4 right now due to a bug in Triton. I'll try to fix it in V0 engine then

[Datta0]

Also, fwiw, I tried running V0 engine with vLLM main with this commit ID : 44ac25eae2cbbdc1cbcca423777107a5ca90a8f4 on H100 and the same example works fine.
Edit: Commit

[DarkLight1337]

Can you open a separate issue regarding this? It's not really related to the OP

[DarkLight1337]

And please show the full code/example you're using. I'm unable to repro using VLLM_USE_V1=0 python examples/offline_inference/vision_language.py -m qwen2_vl

[Datta0]

Hey @DarkLight1337, so in my brief testing vllm 0.10 with triton 3.2 fails while vllm 0.10.1.1 with triton 3.3.1 seems to work fine.
I'll update you if I find anything else

[DarkLight1337]

I think there is a related issue that is caused by triton, forgot which one. cc @Isotr0py

[Datta0]

Hey @DarkLight1337 the info about triton 3.3.1 seems to be a false positive from my side.
I created an issue as you mentioned. Here's the link #23797