fix: convert_hf_to_gguf - change Jamba non-sentencepiece mode (tokeni…

[amirai21]

The JambaModel implementation at convert_hf_to_gguf.py was incorrectly constructing its vocab using the gpt-2 tokenizer logic when no SentencePiece model was present (i.e., tokenizer.json path). Jamba actually uses a llama tokenizer, not gpt-2.

This change updates the vocab build path to use the correct llama tokenizer for non-SentencePiece Jamba models. Also includes several small adjustments within the Jamba llama based tokenizer construction.

No changes are expected for other model types.

Testing
Verified with local conversion of Jamba GGUF model (tokenizer.json mode) and confirmed generated vocab matches the llama tokenizer layout. SentencePiece mode GGUF was also verified and it remains unaffected.

[CISC]

I don't think this works correctly, basically you're recreating a SPM vocab without scores, have you checked that tokenization is identical to AutoTokenizers (you can use convert_hf_to_gguf_update.py to generate test files and test with test-tokenizer-0)?

[amirai21]

@CISC Thanks for reviewing! The initial implementation was incorrect. Following your feedback, we realized there’s already an existing function - _set_vocab_llama_hf in the TextModel class that correctly handles LLaMA tokenizers. Since Jamba uses a LLaMA tokenizer jamba-3b-reasoning tokenizer.config) , we switched to using that implementation instead.