Inconsistent behavior in models list

[bparees]

System Info

Using llama-stack 0.2.17, I don't think any other deps/levels come into play in what i've observed.

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

The /v1/models api has inconsistent behavior with respect to populating the identifier field.

i'm running LLS standalone using this launch command:

uv run llama stack run run.yaml

and this run.yaml:

models:
  - model_id: my_llm
    provider_id: openai
    model_type: llm
    provider_model_id: gpt-4o-mini
  - model_id: gpt-4-turbo
    provider_id: openai
    model_type: llm
    provider_model_id: gpt-4-turbo

and the /v1/models api endpoint reports the following models (i'm truncating the list, in reality it shows many other openai models in addition):

{
  "data": [
    {
      "identifier": "my_llm",
      "provider_resource_id": "gpt-4o-mini",
      "provider_id": "openai",
      "type": "model",
      "metadata": {},
      "model_type": "llm"
    },
    {
      "identifier": "openai/gpt-4-turbo",
      "provider_resource_id": "gpt-4-turbo",
      "provider_id": "openai",
      "type": "model",
      "metadata": {},
      "model_type": "llm"
    },
    {
      "identifier": "openai/gpt-3.5-turbo-0125",
      "provider_resource_id": "gpt-3.5-turbo-0125",
      "provider_id": "openai",
      "type": "model",
      "metadata": {},
      "model_type": "llm"
    },

    .........................

The inconsistency is that there seem to be 3 ways the identifier is populated:

1. for custom models the name comes from the run.yaml exactly as is, my_llm here.
2. for custom models with a name that matches an existing provider_resource_id, the identifier is prepended with the provider id, openai/gpt-4-turbo here, instead of using the name i provided in run.yaml as is (gpt-4-turbo)
3. for models that were auto-populated, the identifier is populated is $provider_id/$provider_resource_id (openai/gpt-3.5-turbo-0125)

I'm not sure if prepending the provider_id to the identifier in the case of prepopulated models is the right thing to do or not, but scenario (2) definitely seems wrong (the user provided name is mutated).

Error logs

See the description above for the behavior i'm seeing. There is no "error" per se.

Expected behavior

I would expect the "identifier" in the models response would reflect only the model_id from run.yaml(or provider_resource_id from the provider, in the case of prepopulated model entries), not a concatenation of provider_id+model_id. Uniqueness can be enforced/referenced by the [provider_id,identifier] tuple.

But if concatenation is desired as a way to make a single field unique key, then i'd expect prefixing to always be done, regardless of whether the identifier was drawn from the provider_resource_id, or a user-provided model_id in run.yaml, and regardless of whether the user provided alias matches the provider_resource_id or not.

[rhuss]

Thanks @bparees , I agree with your assessment that the model handling is confusing in LLS and you summarized the issues correctly.

Actually, I see the models: section in run.yaml to serve the following purposes (but please correct if I'm wrong):

• Aliases and stable names: give friendly/global names (e.g., my_llm) instead of raw provider ids. The alias should always be taken as is configured and no magic prefix added, even when the alias matches the model id. Alternatively, throw an error that such an alias (== model id) is not allowed.
• Curation and policy: expose only a subset, or pin to approved models. To make that work, if models for specific providers are configured in the models: list, then only those should be considered. Only if no models are given for a configured provider, the full default list is returned. This is an easy way to filter out unwanted models.
• Metadata injection: required for embeddings (e.g., embedding_dimension) and other per-model metadata.
• Fill gaps when providers can’t enumerate: some providers don’t return complete model lists; models: lets you explicitly register what you need.
• Cross-provider portability: keep consistent names even if backing providers change.
• Determinism across runs: ensure the same identifiers exist regardless of provider-side changes.

My suggestion would be:

• Call an alias also like this (i.e. an alias: instead of an model_id field)
• The model_id is always calculated as the combination provider_id/provider_model_id. It can't be configured. An alias can map to such an id through the configuration in run.yaml. The API endpoint should return this consistent ID, but also allow adressing a model via its alias.
• When an alias is called like a model_id it must not be prefixed (or an error should be thrown)
• Use the provider returned model list only as a default list. As soon as the provider is referenced in a models: section, this list is ignored. This should be also enforced to not allow to call the provider with a model from the default list that is not configured in models:

_{^{AIA Primarily human, New content, Reviewed, gpt-5 v1.0}}

[ashwinb]

Just a short note for now while I digest: we have gone back and forth on this many times and it has always been quite confusing regardless what policy we choose.

I really like using the word alias explicitly as Roland mentions above.

cc @ehhuang for more thoughts.

[amfred]

We decided to split this up into a few related issues in today's community call. Thank you for writing these up, @bparees !

[bparees]

I've cleaned up this issue to only talk about the inconsistency in how identifier is populated in the model list (sometimes prepended w/ provider, sometimes not).

Caching issue is here:
#3150

and the RFE to disable the automatic inclusion of the full provider model list is here:
#3151

[github-actions]

This issue has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 30 days.