[Refactor]: Let each modeling file define M-RoPE implementation

[DarkLight1337]

🚀 The feature, motivation and pitch

Currently, M-RoPE is implemented inside a single file (https://github.com/vllm-project/vllm/blob/main/vllm%2Fmodel_executor%2Flayers%2Frotary_embedding%2Fmrope.py) just like other RoPE methods. However when running the model, the model type in the HF config is used to further select the model-specific M-RoPE implementation. This is not maintainable in the long run as the number of models that use M-RoPE increases.

Considering that much of the M-RoPE code is shared between models, with the main difference being how to get the input positions, I propose a SupportsMRoPE interface for the model class to define this step inside each modeling file. This interface should include a flag variable and also a get_mrope_input_positions method to get the input positions specific to that model.

We can also use this interface to determine whether a model supports M-RoPE, instead of having to set this flag in the HF config.

Alternatives

No response

Additional context

cc @ywang96 @Isotr0py

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[divyanshsinghvi]

@DarkLight1337 can I take this up?

[Isotr0py]

@divyanshsinghvi Feel free to take this!

[AzizCode92]

Hi, I am also interested in this task, can I raise a PR or it is too late for me?

[divyanshsinghvi]

#24172 fixes it.

[divyanshsinghvi]

Hey, @AzizCode92 any reason for the duplicate PR? Is something wrong with my implementation. Feel free to add a feedback for my PR.

[AzizCode92]

Hey, @AzizCode92 any reason for the duplicate PR? Is something wrong with my implementation. Feel free to add a feedback for my PR.

Hey @divyanshsinghvi ,
as per https://docs.vllm.ai/en/latest/contributing/, there was no explicit mention to prevent opening a PR solving a task even if there are other open PR ongoing.
I made a comment here showing my interest in solving this task and I also mentioned your PR in my PR description.

[divyanshsinghvi]

@AzizCode92 hey totally fine. I just mentioned incase you were unaware. Missed your mention. But it's alright. I thought we could have contributed in one PR instead of opening two separate PR for same task?
But it's understandable if you want to do the task independently. :)

[ywang96]

I think it would be great if @divyanshsinghvi @AzizCode92 you can work together on one PR? Maybe one can review another person's PR

I can put one or the other as the co-author of the final PR if that's the concern. We do very much welcome community collaboration since repetitive PRs will typically become overheads for us maintainers :) Very much appreciate the effort from both of you!