[Feature] Support making use of the native capabilities of models for structure output.

[wenjin272]

Search before asking

• I searched in the issues and found nothing similar.

Description

Currently, when configured output_schema, the react agent directly prompt the model to use a specific format, and use an output parser to extract the structured response from the raw model output. This is the only way for models which don't support tool calling or json mode.

For models support tool calling or json mode, we can makes use of these capabilities to generate structured output.

See https://python.langchain.com/docs/how_to/structured_output/ for details.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[kitalkuyo-gita]

@wenjin272 Hello, I have some questions about this requirement. Question: How to handle the capability differences between different models?

Model Tool Calling JSON Mode Structured Output API

模型	Tool Calling	JSON Mode	Structured Output API
OpenAI GPT-4	✅	✅	✅ (response_format)
Anthropic Claude	✅	⚠️ (beta)	❌
Ollama (Qwen/Llama)	✅	❓	❓
Tongyi Qwen	✅	❓	❓

Question:

Is it necessary to implement a separate adaptation for each ChatModel?

Capability detection mechanism (runtime vs. configuration time)?

Degradation strategy (JSON mode failure → tool calling → prompt)?

Besides that, there's another problem. Semantic issues of using Tool Calling for structured output

Question: Is it reasonable to use Tool Calling to implement structured output? Technically feasible: Define a virtual tool named "formatoutput"

{
"name": "format_output",

"description": "Format the final response",

"parameters": OutputData.model_json_schema()

}

However, there are problems:

• Semantic ambiguity: The tool should "perform an action," not "format output"

• User experience: This "virtual tool" will appear in the tools list

• Conflict with real tools: React Agent already has tool invocation capabilities

[wenjin272]

You can refer to the structured output doc of langChain：https://docs.langchain.com/oss/python/langchain/structured-output.

Roughly speaking, the performance of structured output can be ranked as follows:

1. Use the native structured output capabilities provided by the model provider, currently only OpenAI and Grok.
2. Use the tool calling to achieve the same result.
3. Use prompt and examples to instruct llm.

Flink-Agents should provide all the strategy.

In addition, these questions seem to be rhetorical ones posed by an AI programming assistant. I hold a positive attitude towards using AI for programming assistance, but developers should to ensure their own understanding of the problems and the quality of the generated code.

[kitalkuyo-gita]

You can refer to the structured output doc of langChain：https://docs.langchain.com/oss/python/langchain/structured-output.

Roughly speaking, the performance of structured output can be ranked as follows:

1. Use the native structured output capabilities provided by the model provider, currently only OpenAI and Grok.
2. Use the tool calling to achieve the same result.
3. Use prompt and examples to instruct llm.

Flink-Agents should provide all the strategy.

Of course, for a new project like yours, an AI programming assistant is a good development partner for understanding the project. For submitted pull requests, I will meet their requirements and ensure they pass testing.

[wenjin272]

Here are two documents we can refer to:

1. https://github.com/langchain-ai/langchain/blob/v0.3/docs/docs/how_to/structured_output.ipynb
2. https://github.com/langchain-ai/langgraph/blob/main/docs/docs/how-tos/react-agent-structured-output.ipynb

[weiqingy]

@kitalkuyo-gita @wenjin272 — bumping this thread. I checked the codebase: the prompt-engineered fallback (system prompt + cleanLlmResponse + json.loads post-hoc) is in place, and ReActAgent already exposes output_schema. But none of the chat-model integrations (OpenAI, Anthropic, Azure, Ollama, Tongyi) currently use the model's native response_format / json_mode. Is the native-API path still on the roadmap, and is anyone actively working on it?

[weiqingy]

Since there hasn't been further activity here, I'll go ahead and work on this.

Plan: wire the native structured-output path (model response_format / JSON-schema mode) into the chat-model integrations, layered on top of the existing prompt-engineered fallback, following the strategy ranking discussed above (native API → tool calling → prompt). Happy to align on the design before opening a PR if there are preferences on the API surface.
cc @wenjin272

[wenjin272]

Thanks for picking this up @weiqingy. Assigned.

I have no further comments on your plan. One thing worth considering, though: in the Agent, we may only need the final output to satisfy the structure constraint, not the intermediate react loop. That said, I think we can address this in a follow-up issue and PR.

[weiqingy]

Thanks @wenjin272, and agreed on the final-output-only scope — I'll leave the intermediate react loop unconstrained and handle the agent wiring in a follow-up.

I went through the two references you shared. Two points stood out that shape the API surface:

1. with_structured_output() in LangChain auto-selects between a model's native methods (tool/function calling and JSON/schema mode) with an explicit method= override. The prompt-and-parser path for models without native support sits outside that call as a separate layer — so I'd treat native auto-selection and the prompt fallback as two distinct layers rather than one seamless fallback chain.

2. LangGraph's create_react_agent applies response_format as a dedicated post-loop node (generate_structured_response) that makes one additional with_structured_output call over the final message history — it does not bind a format tool into the react loop. That cleanly avoids the "virtual format tool competing with real tools" concern raised earlier above, at the cost of one extra model call.

Proposed split:

• PR1 (this issue): add native structured output at the chat-model / connection layer — auto-select the provider's native mechanism (tool calling or JSON/schema mode) with an explicit override, falling back to the existing prompt-engineered path for models without native support. Java + Python parity.

• Follow-up: wire final-output structuring into ReActAgent as a post-loop step (mirroring LangGraph's dedicated node), so only the final output is constrained.

For capability detection I'd keep it declarative — each integration statically declares which native methods it supports — rather than runtime probing, to avoid extra calls and keep it testable.

Does this match what you had in mind? Happy to adjust the API surface before I open the PR.