[Model] Siglip Embedding Support

[piood]

Purpose

Support SigLIP text and image embedding in the same model, following the same architecture as CLIP embedding support.

• For text inputs, we only apply token_embedding when calling get_input_embeddings. The rest of the text embedding and the encoder logic are applied when calling forward on the model.
• For image inputs, we apply vision embeddings when calling get_input_embeddings. Since the model doesn't have a decoder, we directly return the embeddings inside the forward method.
• Unlike CLIP, SigLIP uses an encoder-only architecture. To disable prefix_cache for SigLIP, as mentioned in [Model] Siglip Embedding Support #27324 (comment), use CLS as the default pooling type.

This PR extends the multimodal embedding capabilities to support SigLIP models, which are widely used for vision-language tasks.

Related to #13663，this pr is for siglip(v1) embedding support, then will continue support siglip2.

Test Plan

• Added dedicated tests in tests/models/multimodal/pooling/test_siglip.py
• Updated model registry to include SigLIP embedding support
• Added examples for both offline inference and online serving
• Verified with local test runs

Test Result

• All new SigLIP-specific tests pass
• Model registry correctly recognizes SigLIP embedding models
• Both text and image embedding generation work as expected

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[mergify]

Documentation preview: https://vllm--27324.org.readthedocs.build/en/27324/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @piood.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[gemini-code-assist]

Code Review

This pull request adds support for SigLIP text and image embedding models, which is a great extension of vLLM's multimodal capabilities. The implementation follows the existing architecture for CLIP, including separate handling of text and image inputs. The changes are well-structured, with new tests, examples, and updates to the model registry. I have one suggestion regarding the pooling mechanism to improve performance and memory efficiency.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[DarkLight1337]

@noooop shouldn't enable prefix caching be disabled for encoder-only models already? Why do we still need this?

[noooop]

Thanks for your contribution

[piood]

Fixed get_num_image_tokens with detailed documentation. All other issues have been addressed. Ready for re-review.

[DarkLight1337]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for SigLIP text and image embedding models. The changes include adding the model to the registry, providing example usage scripts, and implementing the model logic in vllm/model_executor/models/siglip.py. The implementation correctly handles separate text and image inputs and reuses encoder components for both modalities. The tests are comprehensive. I have two main concerns:

1. A critical performance issue in the vision tower's pooling head, which uses a non-optimized torch.nn.MultiheadAttention instead of a vLLM-optimized attention backend.
2. A bug where the model will crash if pooling_type='LAST' is used, due to a missing entry in the pooling strategy map. This contradicts the behavior described in the PR description.
   These issues should be addressed before merging.