multimodal - Improve LLaVA model accuracy and performance

[monatis]

With #3436, llama.cpp has support for LLaVA, state-of-the-art large multimodal model. It appears that there is still room for improvement in its performance and accuracy, so I'm opening this issue to track and get feedback from the community. I'll continue to work on this, so any feedback is much appreciated.

[hexbinoct]

I have the latest build of the main branch, llava is working (pretty amazing), but it doesnt seem to be using cuda (while the release is built with blas support and works with llama.cpp really good):

PS F:\ai3\llama.cpp> .\build\bin\Release\llava.exe -m ..\models\llava\ggml-model-q5_k.gguf --mmproj ..\models\llava\mmproj-model-f16.gguf --image 'C:\Users\ab\Pictures\mystuff\1664779479174 - Copy.jpg' --temp 0.1
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3070 Laptop GPU, compute capability 8.6
.
.
llm_load_tensors: using CUDA for GPU acceleration
llm_load_tensors: mem required = 4560.96 MB
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/35 layers to GPU
llm_load_tensors: VRAM used: 0.00 MB

but its not like it doesnt use the GPU, the GPU shows a full activity while what seems like its processing the image, and then the gpu goes idle while the text inference is being streamed, i wish the text inference was also on the gpu (like normal llama). Yes i have tried the ngl & ngld but no changes.

[KerfuffleV2]

@hexbinoct Does #3621 fix your issue?

edit: I merged it, so that fix should be in the next release or you can compile it from master yourself to be able to offload immediately.

[hexbinoct]

oh dear, the token generation just went from ~2/sec to 40/sec on a laptop nvidia gpu. Thanks a lot! i got updated source from master.

[aiaicode]

oh dear, the token generation just went from ~2/sec to 40/sec on a laptop nvidia gpu. Thanks a lot! i got updated source from master.

@hexbinoct What about image encoding speed? Did it increase as well?

[KerfuffleV2]

What about image encoding speed? Did it increase as well?

That was already offloaded as far as I know. The performance stayed the same when I tested it.

[hexbinoct]

oh dear, the token generation just went from ~2/sec to 40/sec on a laptop nvidia gpu. Thanks a lot! i got updated source from master.

@hexbinoct What about image encoding speed? Did it increase as well?

its almost instant, like a second i think, as soon as the last loading message is printed "total VRAM used ..." the text inference starts writing itself meaning the image was completed already. thats what i understand by it.

[hexbinoct]

is there interactive mode? -i and -ins is not working. I am thinking like keep the model loaded, and then through some command format we keep on giving it images one by one, along with the -p. Right now the app closes itself once it has explained the image.

[aiaicode]

is there interactive mode? -i and -ins is not working. I am thinking like keep the model loaded, and then through some command format we keep on giving it images one by one, along with the -p. Right now the app closes itself once it has explained the image.

Issue 3593

[y10ab1]

I am curious as to why the q4_k model provides faster evaluation time but slower prompt evaluation time, while the f16 model gives me slower evaluation time but faster prompt evaluation time?

Can we reduce the prompt evaluation time?

f16:

q4_k:

[ggerganov]

You can try disabling the MMQ feature: just add -nommq to the command line

[monatis]

Today I played a little bit with the 13b model to understand the reason why it performs worse than 7b even in f16. I extracted image features from the original Python implementation and saved them in a .bin file. Then I read it in C++ instead of encoding an image file. The quality was still bad. Not sure yet, but seems like the LLaMA part might have something to do with it. Will investigate this further, but I might be busy with for some time with the BERT implementation. Until then, any feedback might be helpful when I return back to this.

[ggerganov]

Where is the source of the 13B LLaMA model? Want to take a look at the config / hparams to see if we have converted everything correctly.

[monatis]

Here it is: https://huggingface.co/liuhaotian/llava-v1.5-13b/blob/main/config.json