Skip to content

[Performance] Move apply_w8a8_block_fp8_linear to an op class #24666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 50 commits into from
Sep 23, 2025
Merged

[Performance] Move apply_w8a8_block_fp8_linear to an op class #24666

merged 50 commits into from
Sep 23, 2025

Conversation

@ElizaWszola
Copy link
Contributor

@ElizaWszola ElizaWszola commented Sep 11, 2025 •
edited by github-actions bot
Loading

This PR unwraps apply_w8a8_block_fp8_linear custom op by implementing a class that we can instantiate and run from Fp8LinearMethod. Part of work outlined in #24629

Testing:

E2E tested with on Blackwell and Hopper

llm = LLM(model="Qwen/Qwen3-30B-A3B-FP8")
llm = LLM(model="RedHatAI/Qwen3-0.6B-FP8-BLOCK")

ProExpertProg
ProExpertProg reviewed Sep 11, 2025
View reviewed changes
Copy link
Collaborator

@ProExpertProg ProExpertProg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few overall notes

ProExpertProg
ProExpertProg reviewed Sep 11, 2025
View reviewed changes
@ProExpertProg ProExpertProg mentioned this pull request Sep 11, 2025
Open
1 task
@ElizaWszola
format
02793b9 
Signed-off-by: ElizaWszola <[email protected]>
@mergify
Copy link

mergify bot commented Sep 12, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ElizaWszola.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Sep 12, 2025
@mergify mergify bot removed the needs-rebase label Sep 12, 2025
@ElizaWszola ElizaWszola marked this pull request as ready for review September 17, 2025 12:38
mgoin
mgoin approved these changes Sep 23, 2025
View reviewed changes
ProExpertProg
ProExpertProg approved these changes Sep 23, 2025
View reviewed changes
Copy link
Collaborator

@ProExpertProg ProExpertProg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really clean, thanks for the great work!

@ProExpertProg ProExpertProg enabled auto-merge (squash) September 23, 2025 15:37
ProExpertProg
ProExpertProg reviewed Sep 23, 2025
View reviewed changes
Copy link
Collaborator

@ProExpertProg ProExpertProg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gonna submit this to always use the cuda path - we can enable torch for Blackwell in a follow-up as suggested by @mgoin, he also has improvements to the torch path for H100.

@mergify
Copy link

mergify bot commented Sep 23, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ElizaWszola.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Sep 23, 2025
@mergify mergify bot removed the needs-rebase label Sep 23, 2025
@vllm-bot vllm-bot merged commit 6340025 into vllm-project:main Sep 23, 2025
48 of 49 checks passed
@eicherseiji eicherseiji mentioned this pull request Sep 24, 2025
Closed
1 task
tlrmchlsmth added a commit to tlrmchlsmth/vllm that referenced this pull request Sep 24, 2025
@tlrmchlsmth
Revert "[Performance] Move apply_w8a8_block_fp8_linear to an op class (…
7c6404d 
…vllm-project#24666)"

This reverts commit 6340025.

Signed-off-by: Tyler Michael Smith <[email protected]>
@tlrmchlsmth tlrmchlsmth mentioned this pull request Sep 24, 2025
Merged
@ProExpertProg ProExpertProg mentioned this pull request Sep 25, 2025
Open
1 task
FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request Sep 25, 2025
@ElizaWszola @ProExpertProg
@mgoin @FeiDaLI
[Performance] Move apply_w8a8_block_fp8_linear to an op class (vllm-p…
a691a6d 
…roject#24666)

Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
This was referenced Sep 25, 2025
Closed
Open
yewentao256 pushed a commit that referenced this pull request Oct 3, 2025
@ElizaWszola @ProExpertProg
@mgoin @yewentao256
[Performance] Move apply_w8a8_block_fp8_linear to an op class (#24666)
b672b8c 
Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
Signed-off-by: yewentao256 <[email protected]>
gjc0824 pushed a commit to gjc0824/vllm that referenced this pull request Oct 10, 2025
@ElizaWszola @ProExpertProg
@mgoin @gjc0824
[Performance] Move apply_w8a8_block_fp8_linear to an op class (vllm-p…
a2f4f7c 
…roject#24666)

Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
Signed-off-by: gaojc <[email protected]>
xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 10, 2025
@ElizaWszola @ProExpertProg
@mgoin @xuebwang-amd
[Performance] Move apply_w8a8_block_fp8_linear to an op class (vllm-p…
491f5a7 
…roject#24666)

Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
Signed-off-by: xuebwang-amd <[email protected]>
choprahetarth pushed a commit to Tandemn-Labs/vllm that referenced this pull request Oct 11, 2025
@ElizaWszola @ProExpertProg
@mgoin @choprahetarth
[Performance] Move apply_w8a8_block_fp8_linear to an op class (vllm-p…
81537c4 
…roject#24666)

Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
lywa1998 pushed a commit to lywa1998/vllm that referenced this pull request Oct 20, 2025
@ElizaWszola @ProExpertProg
@mgoin @lywa1998
[Performance] Move apply_w8a8_block_fp8_linear to an op class (vllm-p…
6babf37 
…roject#24666)

Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 24, 2025
@ElizaWszola @ProExpertProg
@mgoin @xuebwang-amd
[Performance] Move apply_w8a8_block_fp8_linear to an op class (vllm-p…
f66b67f 
…roject#24666)

Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
Signed-off-by: xuebwang-amd <[email protected]>
rtourgeman pushed a commit to rtourgeman/vllm that referenced this pull request Nov 10, 2025
@ElizaWszola @ProExpertProg
@mgoin @rtourgeman
[Performance] Move apply_w8a8_block_fp8_linear to an op class (vllm-p…
68d3090 
…roject#24666)

Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: ElizaWszola <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Signed-off-by: Luka Govedič <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Luka Govedič <[email protected]>
@ProExpertProg ProExpertProg mentioned this pull request Nov 12, 2025
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ProExpertProg ProExpertProg ProExpertProg approved these changes

@mgoin mgoin mgoin approved these changes

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner

@simon-mo simon-mo Awaiting requested review from simon-mo simon-mo is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

@houseroad houseroad Awaiting requested review from houseroad houseroad is a code owner

@hmellor hmellor Awaiting requested review from hmellor hmellor is a code owner

Assignees

No one assigned

Labels

performance Performance-related issues ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

vllm==v0.12.0/torch==2.9.0 compilatio...

Development

Successfully merging this pull request may close these issues.

4 participants

@ElizaWszola @mgoin @ProExpertProg @vllm-bot