[ROCm][Bugfix] Bring back fallback to eager mode removed in #14917, but for ROCm only #15413
Conversation
…for ROCm only Signed-off-by: Gregory Shtrasberg <[email protected]>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
SageMoore
left a comment
SageMoore
left a comment
There was a problem hiding this comment.
Looks reasonable. Just one NIT on a variable name.
vllm/config.py
Outdated
| self.max_seq_len_to_capture = self.max_model_len | ||
| self.max_seq_len_to_capture = min(self.max_seq_len_to_capture, | ||
| self.max_model_len) | ||
| MODEL_NOT_SUPPORT_CUDA_GRAPH_ROCM = ['mllama'] |
There was a problem hiding this comment.
Nit: can you change this to "unsupported_models".
Signed-off-by: Gregory Shtrasberg <[email protected]>
…ect#14917, but for ROCm only (vllm-project#15413) Signed-off-by: Gregory Shtrasberg <[email protected]> Signed-off-by: xinyuxiao <[email protected]>
…ect#14917, but for ROCm only (vllm-project#15413) Signed-off-by: Gregory Shtrasberg <[email protected]> Signed-off-by: Louis Ulmer <[email protected]>
…ect#14917, but for ROCm only (vllm-project#15413) Signed-off-by: Gregory Shtrasberg <[email protected]>
…ect#14917, but for ROCm only (vllm-project#15413) Signed-off-by: Gregory Shtrasberg <[email protected]>
…ect#14917, but for ROCm only (vllm-project#15413) Signed-off-by: Gregory Shtrasberg <[email protected]> Signed-off-by: Mu Huai <[email protected]>
5 participants
mllama doesn't work in graph mode on ROCm.
Returning the condition removed in #14917 conditionally for ROCm platform until this is figured out to bring back support for LLama 3.2