Deepseek V3.1 native tool calling support (OpenAI Style)

[createthis]

This PR enables DeepSeek V3.1 thinking mode as the default. Disable with --reasoning-budget 0.

It also implements tool calling support.

Addresses #15496

My understanding is that this is a continuation of #9639 for DeepSeek V3.1 specifically.

[createthis]

@CISC who approves the workflows?

[ggerganov]

@CISC who approves the workflows?

I think all collaborators can approve them?

Just approved this one.

[createthis]

I still don’t see a merge button. Do we need @ngxson to review too?

[pwilkin]

@createthis nope, only collaborators with write access can merge, so you need either @CISC or @ggerganov to merge it :>

[createthis]

I added an edge case where thinking is forced open, there is tool calling in the reasoning content, but then the model just stops the output without closing the </think> tag, so it's not a partial. In this case, use the tool call in the reasoning content, because the model appears to be confused.

[createthis]

@CISC @ggerganov Let me know if you want any more changes, otherwise please merge. This is working well on my end.

[createthis]

@CISC @ggerganov I simplified update_cursor with 26b02fa. I also performed an extensive analysis of the cause. You can view my notes here: https://gist.github.com/createthis/dc3098c3abb4ff809d0291c91322f512

TL;DR: After the secondfunction_regex call fails to find a match, it also fails to reset builder.pos(), so block_close fails. update_cursor just resets the cursor to the last position before function_regex fails.

[createthis]

🎉🎉🎉

[fernandaspets]

is there a way to turn on and off this? i'm working on a tool calling project. and would like to have just the models original output for testing

[createthis]

is there a way to turn on and off this? i'm working on a tool calling project. and would like to have just the models original output for testing

@fernandaspets Hey neolithic. You can see the model's original output by starting llama.cpp with --verbose. It will show up in the log, as well as come through in the json as an extra property. You may be able to turn it off entirely with --reasoning-format=none. I know that turns off parsing of the think tags, but I don't remember if it turns off tool call parsing.

There is an open source CLI tool called mitmproxy: https://www.mitmproxy.org/ You can run it in reverse proxy mode as a man-in-the-middle and it will record your entire chat session, both the request and the response. This, combined with --verbose is my favorite method for debugging tool calling issues.

Once I have identified an issue, then I usually write a unit test. You can see my unit test for multiple tool calls here:

llama.cpp/tests/test-chat-parser.cpp

Line 303 in 7ea15bb

const std::string in = "CONTENT<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_time<｜tool▁sep｜>{\"city\": \"Paris\"}<｜tool▁call▁end｜><｜tool▁call▁begin｜>get_weather<｜tool▁sep｜>{\"city\": \"Paris\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>";