Repository navigation
OpenAI-compatible server ignores tools and never returns tool_calls #115
Description
The Edge0 model chat templates contain tool-calling formats, but the OpenAI-compatible HTTP server does not pass tools or tool_choice into the template and does not parse generated tool calls into the OpenAI response schema.
Environment
- Edge0 main:
fb4cd2c49ebe22bb230e1451ecb8fb4957ca62e6 - Model: Edge0-35B preview
- macOS 15.7.7 arm64
- Python 3.12.13
- MLX 0.30.6
- mlx-lm 0.31.0
- Server:
edge0 serve ... --host ... --port 8000
Reproduction
Start the server, then send:
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "edge0-35b",
"messages": [
{
"role": "user",
"content": "Use the calculator tool to compute 2 + 2."
}
],
"tools": [
{
"type": "function",
"function": {
"name": "calculator",
"description": "Calculate an arithmetic expression.",
"parameters": {
"type": "object",
"properties": {
"expression": {"type": "string"}
},
"required": ["expression"]
}
}
}
],
"tool_choice": "auto",
"enable_thinking": false,
"temperature": 0,
"max_tokens": 64
}'
Actual result
{
"choices": [
{
"message": {
"role": "assistant",
"content": "2 + 2 = 4",
"reasoning_content": ""
},
"finish_reason": "stop"
}
]
}
No tool_calls field is returned, and the request’s tool definition is silently ignored.
Expected result
The server should either return an OpenAI-compatible tool call, for example:
{
"message": {
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_...",
"type": "function",
"function": {
"name": "calculator",
"arguments": "{\"expression\":\"2 + 2\"}"
}
}
]
}
}
Or explicitly reject tool-calling requests as unsupported.
Implementation evidence
- The 35B template contains tool instructions and XML tool-call formatting.
- parse_chat_request() does not preserve tools or tool_choice: src/edge0/server/chat.py
- ChatSession.prompt_ids() calls the template without a tools argument.
- _chat_once() only emits content and reasoning_content; it does not parse or return tool_calls.
Suggested fix
1. Add tools and tool_choice fields to ChatRequest.
2. Pass tool definitions to the chat template.
3. Parse the model’s XML tool-call format.
4. Convert calls to OpenAI message.tool_calls.
5. Support assistant tool_calls and subsequent tool messages.
6. Add non-streaming and SSE tests for the complete tool-call round trip.
The existing [server-hardening PR #4](https://github.com/Edge0-AI/Edge0/pull/4) addresses request limits and bind safety, but not tool calls.
Activity
linux295650-ux commentedon Sep 23, 2026
Thanks for putting this together. I tested it against the real 8b checkpoint, and single-turn tool calling works exactly as described — the curl from #115 comes back with
content: nullandfinish_reason: "tool_calls".One thing I ran into, though, on the multi-turn round trip. If I send the assistant message your server just produced back, plus a
role: "tool"reply:{"messages": [ {"role": "user", "content": "Use the calculator tool to compute 2 + 2."}, {"role": "assistant", "content": null, "tool_calls": [ {"id": "call_1", "type": "function", "function": {"name": "calculator", "arguments": "{\"expression\": \"2 + 2\"}"}}]}, {"role": "tool", "tool_call_id": "call_1", "name": "calculator", "content": "4"}]}I get
jinja2.exceptions.UndefinedError: 'str object' has no attribute 'items'(server/chat.py:107→engine/ling.py:367→<template>line 95), and the client just gets an empty response.My read is that it's the string/dict mismatch on
arguments: OpenAI sends it as a JSON string, while both templates iterate it as a dict (8b_args.items(), 35barguments|items), andparse_chat_requestpassestool_callsthrough undecoded. Passingargumentsas an object instead makes the same request work —prompt_tokens: 275, and the model answers "The result of 2 + 2 is 4."Would you be able to decode
tool_calls[*].function.argumentsto a dict before it reaches the template (both families)? A multi-turn test that renders the realchat_template.jinjawould help too — the current one only asserts the passthrough parse, so CI stays green while this path is broken.Everything else I tried was fine: no-tools requests are unchanged,
tool_choice: "none"does drop the definitions (236 → 33 prompt tokens), streaming puts the call in the final chunk, and both dialects match the real templates. The two items you flagged as out of scope are fine to leave.Checking both real
chat_template.jinjafiles instead of assuming a single format was a good call — that would have been easy to get wrong. Happy to merge once the multi-turn path is fixed.feat(server): OpenAI-compatible tool/function calling
agourakis82 commentedon Oct 4, 2026
Fixed in #119, now merged as
9a56e4d. Reran with the full 8B (269b9a2) and 35B (3fe15cb) checkpoints, using the default serving profiles with prerouter enabled.Both stdlib and Flask HTTP transports passed: tool calls returned by JSON and SSE survive the next request with the tool reply. Both models answer that the result is 4, without the
.items()error.No-tools requests,
tool_choice: "none", and HTTP 400 for invalid arguments also passed. Both CI jobs are green on the merge commit. Thanks for the round-trip report.