[New Model]: HuggingFaceM4/siglip-so400m-14-980-flash-attn2-navit

[Beniko95J]

The model to consider.

Hi, may I ask has this model (HuggingFaceM4/siglip-so400m-14-980-flash-attn2-navit) beed supported in vllm? The most related implemented model seems to be Idefics2VisionTransformer, but I have some trouble in loading the weights.

The closest model vllm already supports.

Idefics2VisionTransformer

What's your difficulty of supporting the model you want?

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

It's not supported yet. Community contributions are welcome!

[Flink-ddd]

Hi @DarkLight1337 , Could you assign me the task of accessing the model? recently I just finished reading this siglip model on transformers.

[DarkLight1337]

Sure, thanks for working on this! You should implement this as a pooling model.

[Flink-ddd]

Hi @DarkLight1337 , okay, I will Refer to this format.

[Flink-ddd]

Hi @DarkLight1337 , I have some question want to consult to you, When I run pytest to this new model, but it's always output error: OSError: HuggingFaceM4/siglip-so400m-14-980-flash-attn2-navit does not appear to have a file named preprocessor_config.json. so I mean, this model doesn't has preprocessor_config.json and tokenizer.config files, how should I use pytest run this new model whether success on vllm.

[DarkLight1337]

You can try to follow the implementation of PrithviGeoSpatialMAE. Users will have to set skip_tokenizer_init=True to run the model without a tokenizer

[Flink-ddd]

okay, I figure out, I try it, Thank you very much.

[Flink-ddd]

Hi @DarkLight1337 , Excuse me, I have two question want to consult to you. during I test my new model

I run my offline_inference script always output

File "/workspace/vllm/vllm/multimodal/registry.py", line 242, in create_processor raise ValueError(f"{model_config.model} is not a multimodal model") ValueError: HuggingFaceM4/siglip-so400m-14-980-flash-attn2-navit is not a multimodal model

but I already add code
"SiglipModel": ("siglip_so400m", "SiglipSo400mVisionModel"), # noqa: E501 into this file :vllm/model_executor/models/registry.py
_EMBEDDING_MODELS = {

[Multimodal]

}

question 1:
I don't know that whether true?

question 2:
about my this new model, I create a new core file into vllm/model_executor/models, but I saw other model like phi3v model, this model core file has
@MULTIMODAL_REGISTRY.register_processor(Phi3VMultiModalProcessor, info=Phi3VProcessingInfo, dummy_inputs=Phi3VDummyInputsBuilder)
write above core function class Phi3VForCausalLM

but my model core file doesn't this method, Do I have to add the method to be recognized by vllm?

[DarkLight1337]

Have you read through the new model guide? It walks you through how to add those methods

[Flink-ddd]

Okay, I see. previously I Ignoring see details, I will accroding to new model guide to update my core file. Thank you very much, Please forgive me as contribute new model support on vLLM first time for being a rookie.

[DarkLight1337]

No worries, take your time!