feat: add support for custom Anthropic-compatible models #9095
+276
−107
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Allow custom model IDs in AnthropicModelId type - Update AnthropicHandler.getModel() to handle unknown models with sensible defaults - Modify createMessage() to support custom models with prompt caching - Add comprehensive tests for custom model support - Enables third-party Anthropic-compatible providers (MinMax, etc.) - Fixes #9093
dosubot
bot
added
the
size:L
Author
Review complete. Found 1 maintainability improvement opportunity.
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
2 tasks
![roomote[bot]](https://avatars.githubusercontent.com/in/1546624?s=60&v=4){kind=small}
Comment on lines
+142
to
+181
| default: { | ||
| // Custom model with prompt caching support | ||
| const userMsgIndices = messages.reduce( | ||
| (acc, msg, index) => (msg.role === "user" ? [...acc, index] : acc), | ||
| [] as number[], | ||
| ) | ||
|
|
||
| const lastUserMsgIndex = userMsgIndices[userMsgIndices.length - 1] ?? -1 | ||
| const secondLastMsgUserIndex = userMsgIndices[userMsgIndices.length - 2] ?? -1 | ||
|
|
||
| stream = await this.client.messages.create( | ||
| { | ||
| model: modelId, | ||
| max_tokens: maxTokens ?? ANTHROPIC_DEFAULT_MAX_TOKENS, | ||
| temperature, | ||
| thinking, | ||
| // Setting cache breakpoint for system prompt so new tasks can reuse it. | ||
| system: [{ text: systemPrompt, type: "text", cache_control: cacheControl }], | ||
| messages: messages.map((message, index) => { | ||
| if (index === lastUserMsgIndex || index === secondLastMsgUserIndex) { | ||
| return { | ||
| ...message, | ||
| content: | ||
| typeof message.content === "string" | ||
| ? [{ type: "text", text: message.content, cache_control: cacheControl }] | ||
| : message.content.map((content, contentIndex) => | ||
| contentIndex === message.content.length - 1 | ||
| ? { ...content, cache_control: cacheControl } | ||
| : content, | ||
| ), | ||
| } | ||
| } | ||
| return message | ||
| }), | ||
| stream: true, | ||
| }, | ||
| { headers: { "anthropic-beta": "prompt-caching-2024-07-31" } }, | ||
| ) | ||
| break | ||
| } |
Author
There was a problem hiding this comment.
The cache control logic (finding user message indices, applying cache control to last and second-to-last user messages) is duplicated between the known models case (lines 82-113) and this custom models default case. This duplication makes maintenance harder - future changes to the caching strategy would need to be applied in two places. Consider extracting this logic into a helper function that both branches can call.
Fix it with Roo Code or mention @roomote and request a fix.
hannesrudolph
added
the
Issue/PR - Triage
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements support for custom model names in the Anthropic adapter, enabling third-party Anthropic-compatible providers and newer models not yet in the hardcoded list.
Problem
As reported in #9093, the Anthropic adapter currently has hardcoded model names and doesn't allow custom model identifiers such as:
Users had to manually edit the code to add new models, which is not sustainable.
Solution
This PR modifies the Anthropic adapter to:
Changes
AnthropicModelIdtype to accept custom stringsTesting
Default Assumptions for Custom Models
Custom models receive these defaults:
These are conservative defaults that match modern Anthropic capabilities and will gracefully degrade if not supported.
Fixes #9093
Important
Adds support for custom Anthropic-compatible models by allowing any string as a model ID and providing sensible defaults for unknown models.
AnthropicModelIdtype now accepts custom strings inanthropic.ts.getModel()inanthropic.tsprovides defaults for unknown models (8192 max tokens, 200K context, images + caching support).createMessage()inanthropic.tssupports custom models with prompt caching.anthropic.spec.tsfor custom model scenarios.This description was created by
for fbb95f0. You can customize this summary. It will automatically update as commits are pushed.