Draft: feat: Support DBRX model in Llama

[reneleonhardt]

The new Open Source model DBRX sounds amazing, is this enough and correct to integrate it into Llama?
ggml-org/llama.cpp#6515
https://huggingface.co/collections/phymbert/dbrx-16x12b-instruct-gguf-6619a7a4b7c50831dd33c7c8
https://www.databricks.com/blog/announcing-dbrx-new-standard-efficient-open-source-customizable-llms
https://github.com/databricks/dbrx
https://huggingface.co/collections/databricks/

llama.cpp seems to support splitted/sharded files, but I would need to download all of them first I suppose... 😅

[carlrobertoh]

I think it uses ChatML prompt template - PromptTemplate.CHAT_ML

[reneleonhardt]

Thx, done

[carlrobertoh]

Since the change was recent, we need to update the llama.cpp submodule as well

[reneleonhardt]

Since the change was recent, we need to update the llama.cpp submodule as well

Done

[carlrobertoh]

I'll try running the model locally soon and see if any other changes are necessary

[reneleonhardt]

I'll try running the model locally soon and see if any other changes are necessary

Great! But in this PR I have to implement downloading all 10 files first I guess... 😅

[reneleonhardt]

@phymbert I can download https://huggingface.co/phymbert/dbrx-16x12b-instruct-iq3_xxs-gguf without login in the browser, but inside the plugin I get 403 Forbidden, is this to be expected with the databricks-open-model-license (other) license?
Do you think DBRX is not particularly suited as a coding assistant? The smallest is 53 GB huge 😅

[phymbert]

Dbrx is a gated model, so I believe you have to pass a read token. There is an issue open on llama.cpp to support this.

[carlrobertoh]

Closing this due to inactivity.