[API Proposal]: IChatClient - Default ChatOptions

[azchohfi]

Background and motivation

Different IChatClient implementations might have different defaults for ChatOptions, which can be sent to CompleteAsync/CompleteStreamingAsync. Right now, there is no default way to retrieve such defaults. Yes, those methods allow you to send null to those parameters, which would use their default values for ChatOptions, but there is no way to retrieve that information in a generic way.
The only way to achieve this today is to use the GetService method, maybe sending a specific serviceKey ("DefaultChatOptions"?), but it would be nice to have a standard way of retrieving these values.

API Proposal

namespace System.Collections.Generic;

public interface IChatClient : IDisposable
{
...
ChatOptions? DefaultChatOptions { get; }
...
}

API Usage

IChatClient chatClient = ...;
// Check upper bound for MaxOutputTokens
if (desiredMaxOutputTokens < chatClient.DefaultChatOptions.MaxOutputTokens)
{
....
}

Alternative Designs

I can also see this being part of ChatClientMetadata, and maybe not even using ChatOptions directly, since it might have too many information that doesn't make sense for a default, but at least some default properties should be retrievable, somehow.

Risks

Its just extra metadata, so I don't see many risks. It is more work on implementations, but it should be optional/nullable.

[luisquintanilla]

cc: @eiriktsarpalis @SteveSandersonMS @stephentoub for visibility.

[stephentoub]

The only way to achieve this today is to use the GetService method, maybe sending a specific serviceKey ("DefaultChatOptions"?)

Why would it need a specific key? i.e. why not just ask for a ChatOptions? .GetService<ChatOptions>()?

[stephentoub]

Note, though, that in many cases an implementation won't know what the "defaults" are. Some service provides default values, and a client can choose to override whatever those defaults may be by sending new ones. Such a client would have no way to know what the defaults are. I'd be hesitant to add something beyond just saying "an implementation that wants to provide options can do so via GetService, it's there for things like this".

[azchohfi]

I agree that it wouldn't need one, just thought it might have a clearer intent.
I also understand the guidance, just thought it could help with some specific models, such as with ORTGenAI.

[stephentoub]

I agree that it wouldn't need one, just thought it might have a clearer intent. I also understand the guidance, just thought it could help with some specific models, such as with ORTGenAI.

What would the defaults be there?

[azchohfi]

For GenAI models, whatever ORTGenAI returns from parsing genai_config.json. Specifically the "search" block.
Example:
https://huggingface.co/microsoft/Phi-3.5-mini-instruct-onnx/blob/aded733f0b665ac2e21ffd8a008f82eb4278a134/cpu_and_mobile/cpu-int4-awq-block-128-acc-level-4/genai_config.json#L37-L52

[stephentoub]

For GenAI models, whatever ORTGenAI returns from parsing genai_config.json

Can you point me to the API for enumerating them? I'm missing them. I was expecting an equivalent to Generator.SetSearchOption but I don't see one.

there is no default way to retrieve such defaults.

I'm curious what you'll do with the default values?

[azchohfi]

ORTGenAI doesn't have such an API (not yet).
Specifically for the AI Dev Gallery, we want to populate some UI elements with it, to allow users to test multiple variations.

[stephentoub]

Ok, thanks. I think it'd be fine to augment the ONNX implementation to return a ChatOptions from GetService, once it can feasibly be implemented there, and we can provide guidance that others who have a need for returning the options used by a type (unless overridden) can do the same (though I suspect it'll rarely be possible).