[V1] TPU - Revert to exponential padding by default #15565
Conversation
Signed-off-by: Alexander Matveev <[email protected]>
alexm-redhat
requested review from
WoosukKwon,
comaniac,
njhill,
robertgshaw2-redhat and
ywang96
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
|
Can you summarize in the PR description about why this could cause the correctness issue and why the fix could resolve it? |
See the slack, the issue is being discussed |
| if padding_gap == 0: | ||
| logger.info("Using exponential paddings:") | ||
| while num <= max_token_size: | ||
| logger.info(" %d", num) |
There was a problem hiding this comment.
does this create a ton of logs?
github-actions
bot
added
the
ready
Signed-off-by: Alexander Matveev <[email protected]> Signed-off-by: xinyuxiao <[email protected]>
Signed-off-by: Alexander Matveev <[email protected]> Signed-off-by: Louis Ulmer <[email protected]>
Signed-off-by: Alexander Matveev <[email protected]>
Signed-off-by: Alexander Matveev <[email protected]>
Signed-off-by: Alexander Matveev <[email protected]> Signed-off-by: Mu Huai <[email protected]>
We have found a correctness issue with padding_gap == 64. As a temporary fix, this PR ensures that the default padding for now is exponential (as it was before).