quantized version of the model fo CPU inference

[akshayballal95]

I wanted to run ColPali on CPU, and the GGUF model is a good choice for this. But as of now, there is no support for quantizing Paligemma models to GGUF. Would you happen to know if there is a better way to run these models on CPU?

[ManuelFay]

I would also be super interested in looking into that but haven't yet ! Lemme know if you have interesting findings !

[akshayballal95]

With some tweaking around in the HuggingFace Optimum exporter, I was able to get an ONNX model.

https://huggingface.co/akshayballal/colpali-v1.2-merged-onnx

It might not be as efficient as GGUF, but with OpenVino Executor and 8-bit quantization, it works decently on the CPU. I get about 5-8 seconds per page embedding speed on an old AMD Ryzen 7 5800x processor and about 5 seconds per 20 pages on NVIDIA 3070Ti.

And for GGUF, I think we need to wait for ggml-org/llama.cpp#7553 to merge.