[Fix][Structured Output] using vocab_size to construct matcher

[aarnphm]

Ok, well I ended up doing a few things in this PR:

• upgrade xgrammar to 0.1.16 to add support for vision models in v1.
• moving/simplify a lot of the metadata logics from our side and use xgrammar API for this construction

Reasoning:

• xgrammar updated some core API, which vLLM was depending on in previous version
• 0.1.16 simplify our fixes for v1 vocab_size, which this PR restores back to our previous implementation of relying on lm_head vocab_size (or the model's vocab_size retrieved from HF config)

Part of #14832

Signed-off-by: Aaron Pham [email protected]
Co-authored-by: [email protected] [email protected]

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[aarnphm]

I don't have write access, but currently waiting for xgrammar to publish 0.1.16, so should simplify this codepath for us @robertgshaw2-redhat

[robertgshaw2-redhat]

@aarnphm - can you explain this change?

[aarnphm]

Yes, so the vocab size here should be the one infered from hf_text_config (which in this case the chanage in this PR).

I'm waiting xgrammar 0.1.16 to be published such that it will support olmo and aria models where they have additional token_id that is not used in lm_head.

reasoning for get_vocab_size() is to accomodate vision models (which will now be obsolete with 0.1.16)

[robertgshaw2-redhat]

Yes, so the vocab size here should be the one infered from hf_text_config (which in this case the chanage in this PR).

I'm waiting xgrammar 0.1.16 to be published such that it will support olmo and aria models where they have additional token_id that is not used in lm_head.

reasoning for get_vocab_size() is to accomodate vision models (which will now be obsolete with 0.1.16)

So do we need to add SupportsV0only to Olmo and Aria?

[aarnphm]

@robertgshaw2-redhat I updated xgrammar to 0.1.16 now, so let's see if this works

[simon-mo]

@aarnphm is this good to go?

[aarnphm]

@aarnphm is this good to go?

Yes. we can wait for the tests to pass, then can merge this one.

[simon-mo]

Ah I saw you are targeting Rob’s branch. Let me know when it is good to merge.

[simon-mo]

Looks like it still failed

[russellb]

It seems like this has a lot of extra formatting-only changes. That makes stuff harder to review. I'd really rather formatting changes be done separately.

[aarnphm]

I have reverted the formatter change.

[robertgshaw2-redhat]

Is this called on the hotpath or during initalization?

[aarnphm]

We only setup during initialisation in get_local_xgrammar_guided_decoding_logits_processor

This has the same logic as xgr.Tokenizer.from_huggingface, sadly they only have decoded_vocab to byte string so we need to construct encoded_vocab here...

[robertgshaw2-redhat]

get_local_xgrammar_guided_decoding_logits_processor is called on the hotpath. It might be worth pre-computing these off the hotpath.

Any exploration of this should be done in a different PR

[robertgshaw2-redhat]

cc @simon-mo - this is a Release Blocker.

The PR looks good to me pending the question about MistralTokenizer, but I am not an expert on structured generation so I will let @russellb give the final say

[aarnphm]

Tests are passed locally with both v0 and v1 for structured outputs, running on A100.

[pathorn]

I was hitting an exception in V1 yesterday (without this PR):

INFO 03-15 09:08:29 [logger.py:39] Received request cmpl-80d6fb517cc646a6993d0921c9ce295f-0: prompt: '', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=-1, min_p=0.0, seed=1, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=8192, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=GuidedDecodingParams(json=None, regex=None, choice=None, grammar=None, json_object=True, backend=None, whitespace_pattern=None), extra_args=None), prompt_token_ids: [], lora_request: None, prompt_adapter_request: None.

EngineCore hit an exception: Traceback (most recent call last): 
  File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 330, in run_engine_core
    engine_core.run_busy_loop()
  File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 364, in run_busy_loop
    outputs = step_fn()
              ^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 193, in step
    engine_core_outputs = self.scheduler.update_from_output( 
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
  File "/opt/venv/lib/python3.12/site-packages/vllm/v1/core/scheduler.py", line 621, in update_from_output
    request.structured_output_request.grammar.accept_tokens(  # type: ignore[union-attr]
  File "/opt/venv/lib/python3.12/site-packages/vllm/v1/structured_output/grammar.py", line 57, in accept_tokens
    if not self.matcher.accept_token(token):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   
  File "/opt/venv/lib/python3.12/site-packages/xgrammar/matcher.py", line 220, in accept_token
    return self._handle.accept_token(token_id, debug_print)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: [09:09:06] /project/cpp/grammar_matcher.cc:362: Check failed: (token_id >= 0 && token_id < tokenizer_info_.GetVocabSize()) is false: Invalid token id 129207 for GrammarMatcher 

I haven't tried the latest VLLM yet, but I'm curious if this PR is supposed to fix exceptions related to out of bounds tokens (129207 was greater than vocab_size in config.json but smaller than lm_head size) . If not, I'll try to see how to reproduce it on latest and file an issue.

[aarnphm]

This should fix that issue.