Skip to content

Conversation

@bruceunx
Copy link

feat: add chat template to existed gguf file

follow the issue 5897

add gguf_add_chat_template.py to gguf-py/scripts

feat: add chat template to existed gguf file
@ngxson
Copy link
Collaborator

ngxson commented Mar 19, 2024

Thanks for the contribution. However, I'm quite doubt if this file may be confused for new users. Some users may misunderstand that this script allow them to use which ever templates they want, without not knowing that llama.cp can only support a predefine set of templates

I think it would be nice to include some kind of warning. And would be even nicer if you can take time to allow the script to change other attributes in metadata.

@bruceunx
Copy link
Author

bruceunx commented Mar 20, 2024

the chat template actually trained in the model, and normally included in the tokenizer_config.json in original model file, while some old gguf missed the chat template in metadata while quantized.
Actually It's not proper to add any other template to model which was not trained in the model.
I thought this script just for gguf which quantized before without including chat template in it, this gguf file is all OK just missing the proper chat template. :)

@bruceunx
Copy link
Author

bruceunx commented Mar 20, 2024

Maybe I can add predefined chat templates for known popular open source llms models, which maybe solve this concern

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants