Adding native support to load GGUF models using transformers

[sleepingcat4]

Feature request

GGUF recently became a popular model format that can be used to load ridiculously large models in 2x3090 (48GB) cards. While loading the original models are still better and HF supports 4bit and 8bit but those quant aren't much practical for very large models unlike GGUF.

Llama CPP exists and its quite robust but it's not as flexible as native HF and it does not support batch inference. https://huggingface.co/docs/hub/en/gguf

Motivation

Actually same as what I described in feature request. I think GGUF is tremendous value and it should included in HF natively. HF supporting the format but doesn't allow it to be loaded using HF is kinda bad.

Your contribution

I can help with some of the support adding part in my free time.

[NielsRogge]

It's already supported: https://huggingface.co/docs/transformers//gguf

[sleepingcat4]

@NielsRogge thanks! I couldn't find this doc anywhere on the Internet. also, is there way a to batch inference? I think this can be a nice feature request:

I recently implemented a naive batching system for model.generate() and it was pretty bad but when I used pipeline() method, it worked really well but it was still lacking some stuff like it was processing my list sequencially as it was saying as I didn't use dataset.map(). Can there be true batching built-in transformers?

[NielsRogge]

Batched generation should be supported, could you try out the following? https://huggingface.co/docs/transformers/main/en/llm_tutorial

[sleepingcat4]

@NielsRogge that's batch decoding, what I meant was feeding multiple prompts or inputs at the same time and then run batch_decode on them.

[sleepingcat4]

@NielsRogge the code breaks for qwen3

The secret `HF_TOKEN` does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
  warnings.warn(
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-3-c94b0c9cdcb8>](https://localhost:8080/#) in <cell line: 0>()
      1 torch_dtype = torch.float32 # could be torch.float16 or torch.bfloat16 too
----> 2 tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
      3 model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename, torch_dtype=torch_dtype)

1 frames
[/usr/local/lib/python3.11/dist-packages/transformers/modeling_gguf_pytorch_utils.py](https://localhost:8080/#) in load_gguf_checkpoint(gguf_checkpoint_path, return_tensors, model_to_load)
    399 
    400     if architecture not in GGUF_SUPPORTED_ARCHITECTURES and updated_architecture not in GGUF_SUPPORTED_ARCHITECTURES:
--> 401         raise ValueError(f"GGUF model with architecture {architecture} is not supported yet.")
    402 
    403     # Handle tie_word_embeddings, if lm_head.weight is not present in tensors,

ValueError: GGUF model with architecture qwen3 is not supported yet.

[sleepingcat4]

It also does not work for gemma3

[NielsRogge]

Support needs to be added on a per-model basis, see e.g. #31175

[sleepingcat4]

@NielsRogge I actually went over the llama.cpp script and adding new models looked quite simple. We can close this issue if you can give me suggestion for a true batching of inputs (not only decode)

[okaris]

@NielsRogge imho native support means loading AND using the weights in the format specific in gguf. afaik transformers convert to fp32, which pretty much goes against the core purpose of why most people are using gguf, memory savings. is this planned, i think it's a major issue for transformers right now

[NielsRogge]

I'll ping @SunMarc on that

[okaris]

and he'll tell me that contributions are welcome 😄 maybe i need to carve out the time and move some functionality over from diffusers but i had hoped that the two HF teams would be more in touch and this should be handled by the maintainers, but thanks