Update to M.E.AI.Abstractions 9.9

[stephentoub]

No description provided.

[PederHP]

I think it's somewhat strange to have this as a tool. I know it's mapped correctly from the MEAI meta-tool to the Anthropic API parameter, but the OpenAI abstraction is really gnarly and inelegant. I'd personally prefer having the MCP server abstraction in the options for a request, and then for OpenAI mapping it to HostedMcpServerTools.

But I guess this is a case of OpenAI "winning" by being the most widely used API. I just think it's a shame that they're allowed to pollute what is otherwise an extremely clean and elegant set of abstractions in MEAI.

The OpenAI MCP tool isn't really a tool but an option. And it's a very poor choice of name that will potentially confuse developers as until now tool has meant one thing and now it also means something which is conceptually quite different.

It's not that difficult as a developer to build things on top to avoid using HostedMcpServerTool directly, but it's the first abstraction in MEAI that I really don't like.

For my own multi-provider code, I'll just add code on top to map in the other direction so it's not a big deal, but I hadn't noticed the remote MCP implementation in the Responses API until now, and more conceptual confusion is not what MCP needs.

[stephentoub]

This isn't a case of MEAI using a tool because that's how OpenAI does it. MEAI uses it as a tool because we think that's logically what it is (and tools are in options, so it's both a tool and an option). I'm unclear on the objections of this being a tool: it's something that the model can't perform directly and requires the service to call out to something else that provides functionality... that's what a tool is. I agree there's some confusion in terminology (does "MCP tool" refer to a tool in the MCP spec or to an AI tool that references an MCP server), but in terms of how this is modeled, modeling it as a tool makes complete sense to me. I'd like to better understand what makes you think otherwise.

[PederHP]

That's fair. And you're of course right that tools are part of options.

But I think it adds more confusion by now using the same name for two different abstractions. And MCP server isn't a tool. The OpenAI API is using the provided data server side. If that's a tool then any information provided to the API can arguably be a tool.

Until now a tool has been something the model calls. But now it's also a client-provided piece of configuration for the OpenAI API server to expand into tools and possibly other types of information. That's a very opaque abstraction. The provided tool is never called.

One could argue that the server side tools are also not true tools (like web search, code interpreter and fetch) as they are not resolved client-side, but I would argue they are, because they're provided to the model and it's simply a matter of where the implementation resides.

The OpenAI mcp tool is the first instance of a tool which is a payload for the API, and not the model. It is not rendered into the chat template (in OpenAI's case via Harmony I suppose) on the LLM like all other tools. It is expanded into multiple tools which are then rendered.

If it was a tool group abstraction I would be less opposed. But this naming makes interacting with LLMs more opaque, not less. One of the things I greatly like about MEAI is that is transparent and unopinionated. I don't think having the option to have a meta-tool which expands server-side into multiple tool is a problem. Options are good. But I think having it is as a baseline abstraction for an MCP server is bad.

It also make evals harder to define and run.

This is a much better abstraction:
https://docs.anthropic.com/en/docs/agents-and-tools/mcp-connector

Example of why:
Let's say one wanted to reference MCP server resources from inside content. Like a ResourceReferenceContent type. This could then reference a resource from a provided MCP server. If the MCP server is a tool, then that's an odd linkage of abstractions. I might have an MCP server with 0 tools and only resources.

The Anthropic one clearly distinguishes between a tool provider and a tool. It is also open for extensibility. I don't think a tool is anything which provides context to an LLM. It's something which is rendered into the chat template and which can be called via tool call responses. Until now it was that and only that. Now it's becoming a much more vague concept.

But in favor of the MCP server as a tool is that Gemini API actually also does it the same way:
https://ai.google.dev/gemini-api/docs/function-calling?example=meeting

Another argument is that it allows the API to wrap resources in meta-tools and make them model accessible. But this is also possible with the Anthropic-style.

In the end, it's just a modeling question.

• Is a tool only a singular tool or also a group of tools?
• Is a tool only a model-controlled primitive or is it also a service-controlled primitive?

Either way I am glad you've actually chosen this abstraction (even if I disagree with it) and not just gone with the OpenAI by default. I still think it is messy to shove MCP servers (which have tools) into a tool. But oh well, as I said, I can work around it and just add an extension in my own code to let me write it Anthropic-style and just have it mapped to the HostedTool. Which I guess is another thing speaking in MEAI's favor - that I can do that with almost no friction.

[stephentoub]

Thanks for explaining your concerns!

The OpenAI API is using the provided data server side. If that's a tool then any information provided to the API can arguably be a tool.

I don't think that's any different from code interpreter, file search, web search, etc. The model is generating a request that an operation be performed and the results provided back to it, generating input to pass to that and incorporating the results into a subsequent turn.

The OpenAI mcp tool is the first instance of a tool which is a payload for the API, and not the model

I'm not understanding this part. When I ask Anthropic to use the web search tool, I can provide with it configuration for that tool, like user location; that's not input to the model per se, that's a payload for the invocation of the tool when it's eventually invoked. When I use code interpreter with OpenAI, I can provide a set of file IDs that are not provided to the model per se but result in additional file content being made available to the functions when invoked.

It is expanded into multiple tools which are then rendered.

I'm struggling with why that's a key differentiator. It seems like the objection is that I specify this as:

{
    "server": "...",
    "allowedTools": { ... }
}

vs

{
    "server": "...",
    "toolName": "tool1"
},
...
{
    "server": "...",
    "toolName": "toolN"
}

The former is more concise and also allows the server side tool to dynamically fetch the list of MCP tools that are available.

It seems to me like a server-side implementation detail whether a single tool I include in my request actually gets rendered as multiple tools in the prompt provided to the LLM. For all I know, Anthropic's "computer_20250124" tool actually expands into a "take_screenshot" tool, a "move_cursor" tool, etc., and it's no less of a tool to me if it's implemented that way than if it's implemented instead as a single parameterized tool.

[PederHP]

Yeah, it's not a biggie - I already turned a molehill into a mountain, by being a bit too passionate about whether the abstractions are my favorite ones or not. In hindsight I should have worded my comments differently.

But just to round it off, it's less about

{
    "server": "...",
    "allowedTools": { ... }
}

vs

{
    "server": "...",
    "toolName": "tool1"
},
...
{
    "server": "...",
    "toolName": "toolN"
}

To me, than simply wanting:

 {
     "remoteMcpServers": [ ... ],
     "tools": [ .. ]
}

It's not that I mind the server-side expansion - that's how the API works. It's that I don't like mixing the concept of an MCP Server with that of a Tool. But I can see the argument that what I call a Tool is a "FunctionTool", and my definition of Tool is more narrow than the OpenAI concept - which it is, as I prefer the Anthropic shape where MCP servers and tools are two distinct arrays and types.

It seems to me like a server-side implementation detail whether a single tool I include in my request actually gets rendered as multiple tools in the prompt provided to the LLM. For all I know, Anthropic's "computer_20250124" tool actually expands into a "take_screenshot" tool, a "move_cursor" tool, etc., and it's no less of a tool to me if it's implemented that way than if it's implemented instead as a single parameterized tool.

Perhaps, but the computer use tool isn't a server-side tool, it doesn't expand and the client can expect it to be called like that shape, and to my knowledge that are no server side tools which are as extreme in being "un-tool-like" as MCP servers. And no ways to have client tools that are assymetrical like that.

Passing an MCP server leads to an init, authorization, fetching a tool list, all kinds of things (could even involve elicitation, resources, sampling, etc.) I think tossing that in with tools limits the conceptual design space of the abstraction and is messy.

Now I have to explain to developers that a HostedMCPServerTool is not the same as the Tools in MCP. I have to explain that MCP Tools are actually function tools, and some LLM APIs use the concept tool more widely. I think that's unnecessary, when they could just be in a separate array, and you'd still have all the advantages of your example above. The ChatOptions shape doesn't gain anything from having such highly heterogenous tool types in the same property, a HostedMCPServers or RemoteMCPServers would be a more clean abstraction to me.

But as I said, it's not a big deal, I just think it's the wrong choice of abstraction to wrap it in a Tool. It's a matter of preference and taste, not correctness. But thank you for engaging with me on this. I appreciate it.

[tghamm]

Always a pleasure to see passionate engineers debate topics like these :)