-
-
Notifications
You must be signed in to change notification settings - Fork 11.2k
[Attention] Refactor CUDA attention backend selection logic #24794
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
[Attention] Refactor CUDA attention backend selection logic #24794
Conversation
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Matthew Bonanni <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Matthew Bonanni <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
LucasWilkinson
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
left a few comments; we should figure out who the owner of the plugin mechanism is and figure out how to notify downstream HW plugins since I think this will affect them pretty dramatically
Signed-off-by: Matthew Bonanni <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
|
@LucasWilkinson thanks for your review! I've already notified @ILikeIneine but I'm not sure if there's anyone else we should reach out to? |
|
@MatthewBonanni Hi, would this refactor be able to merge into v0.11.1? |
Signed-off-by: Matthew Bonanni <[email protected]>
|
@ILikeIneine we were planning on waiting until after v0.11.1, we don't want to risk further delaying the release and because it changes the platform interface, it might be better to be part of v0.12 |
Signed-off-by: Matthew Bonanni <[email protected]>
|
@MatthewBonanni how hard would it be to keep backwards compatibility between |
Signed-off-by: Matthew Bonanni <[email protected]>
@LucasWilkinson done in d0f4698 |
|
Discussed offline thanks for the work @MatthewBonanni ! |
Purpose
CudaPlatformBase.get_attention_backend_clshas gotten complex and messy over time. This PR cleans up the logic (without changing the behavior) and standardizes the interface.Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.