Skip to content

Conversation

@SteveSandersonMS
Copy link
Member

@SteveSandersonMS SteveSandersonMS commented Mar 19, 2025

Update Following discussion, this change has been reduced to:

  • Changing the default for useJsonSchema to true
  • Renaming ModelId/Dimensions to DefaultModelId/DefaultModelDimensions

Plus other minor related renames and integration test fixes.


Old description

Fixes #6126 based on the design discussed there.

This introduces a new concept of per-model metadata. Usage:

var providerMetadata = chatClient.GetService<ChatClientMetadata>();
Console.WriteLine("Provider name: " + providerMetadata.ProviderName);

var defaultModelMetadata  = await providerMetadata.GetModelMetadataAsync();
Console.WriteLine("Supports native JSON schema: " + defaultModelMetadata.SupportsNativeJsonSchema);

var anotherModelMetadata = await providerMetadata.GetModelMetadataAsync("anotherModelId");

There's also the equivalent for embedding generators, and that's now how you can find the Dimensions value.

Structured output

GetResponse<T> still allows the developer to specify useNativeJsonSchema, but if it's unspecified, the default is now taken from the model metadata.

Providers

  • Ollama always supports structured output
  • OpenAI and Azure AI Inference vary based on the model

This PR bakes in knowledge about which model IDs currently support structured output. I'm on the fence about how much we really want to get into this business, but there's no way to get the info programmatically. Neither OpenAI or Azure AI Inference expose this metadata on the endpoints they have for describing models.

I've tried to make the logic as forgiving as possible if the external reality changes and new models appear, but having correct metadata here is a best-effort thing that developers would still have to override on a per-call basis if our default doesn't match their requirement.

The main alternative would be to remove the built-in knowledge of specific models and just default to "unknown" for the OpenAI/AIInference providers across all models. In effect that would retain the same behavior we have before this PR. However it doesn't move us forwards and cements an old-fashioned view of the world.

Yet another alternative would be defaulting to "true" for all models, since most OpenAI models do support it. However even some new-ish OpenAI models (e.g., o1-mini) don't support it, and of course many Azure AI Inference ones don't since it includes lots of small models.

Microsoft Reviewers: Open in CodeFlow

@SteveSandersonMS SteveSandersonMS requested review from a team as code owners March 19, 2025 17:59
@github-actions github-actions bot added the area-ai Microsoft.Extensions.AI libraries label Mar 19, 2025
Copy link
Contributor

@shyamnamboodiripad shyamnamboodiripad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving for the change under the Eval project 👍🏾

@luisquintanilla
Copy link
Contributor

cc: @azchohfi @nmetulev for awareness and had similar asks as part of #5941

@SteveSandersonMS SteveSandersonMS changed the title Per-model metadata for chat and embedding, plus structured output defaults based on that metadata Enable JSON-schema based structured output by default Mar 21, 2025
@SteveSandersonMS SteveSandersonMS force-pushed the stevesa/chat-model-metadata branch from a3edd1d to 3832a27 Compare March 21, 2025 11:38
@SteveSandersonMS SteveSandersonMS enabled auto-merge (squash) March 21, 2025 13:42
@SteveSandersonMS SteveSandersonMS merged commit 571915c into main Mar 21, 2025
6 checks passed
@SteveSandersonMS SteveSandersonMS deleted the stevesa/chat-model-metadata branch March 21, 2025 15:15
@github-actions github-actions bot locked and limited conversation to collaborators Apr 21, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-ai Microsoft.Extensions.AI libraries

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default useNativeJsonSchema to true

6 participants