fix: make structured output work with reasoning models and empty responses

[OsherElhadad]

Fixes #115. Also implements the "Long-term fix (ALTK)" requested in rossoctl/cortex#676, which lets that repo drop its downstream monkey-patch.

Problem

Semantic SPARC calls returned decision: error on every request against watsonx openai/gpt-oss-120b, and native structured output was entirely unusable on Azure/OpenAI. Investigating with real credentials turned up four independent defects, all in the shared validating-client path.

1. json_schema_to_pydantic_model was lossy

Nested object properties collapsed to a bare dict, array items to a bare list, and minimum / maximum / enum / additionalProperties were dropped outright. Providers received a far weaker schema than the one the response was later validated against, so models legitimately emitted out-of-vocabulary enum values and extra keys — which then failed validation.

Measured against the real SPARC function_selection_appropriateness schema, every failure was an enum violation inside an array item:

INVALID @['correction', 'reason_types', 1]: 'Incorrect tool selection' is not one of ['IRRELEVANT_FUNCTION', ...]

enum is now a real Literal type, so it survives inside items where a Field-level constraint cannot reach.

2. Azure/OpenAI native structured output always raised

TypeError: You tried to pass a `BaseModel` class to `chat.completions.create()`;
You must use `chat.completions.parse()` instead

Both clients register create, not parse. Rather than rewire the method configs, the schema is rendered as the equivalent {"type": "json_schema", ..., "strict": true} dict through a new _render_native_schema hook — so native structured output keeps working on the method already registered.

3. The retry loop never retried a contentless response (#115)

The issue proposes widening the except, which is necessary but not sufficient: the _generate call sat outside the try, so the ValueError: No content or tool calls found in response raised by _parse_llm_response escaped before any handler saw it. The call is now inside the try, and ValueError is caught alongside OutputValidationError.

Two related failure modes were found while verifying this and are fixed with it:

• Empty assistant turns poisoned the retry. The loop echoed the bad output back as an assistant message; when that output was empty, the padded conversation made the backend return empty again, burning every remaining attempt. Traced live:

  CALL 1 (1 msg)  -> content_len=2745   schema INVALID
  CALL 2 (3 msgs) -> content_len=0
  CALL 3 (5 msgs) -> content_len=0   <- assistant turn with len=0 sent
  

  An empty reply carries no mistake to correct, so it now retries the original prompt untouched.
• Truncated replies retried with an identical budget. finish_reason='length' means the budget was too small; re-asking with the same max_tokens truncates identically. It now escalates. This is what made SPARC fail deterministically: watsonx defaults max_tokens to 1024 and SPARC's ~17k-char prompts spend it all on reasoning tokens before emitting any content.

4. Models that ignore response_format were still sent it

Chain-of-thought models such as gpt-oss on watsonx return empty content when response_format is present. The schema is now injected into the system prompt for those models instead, selected per model from litellm's own supports_response_schema data rather than a hardcoded list. A negative answer is only trusted for a model litellm actually knows, so unknown models keep the previous behavior. _parse_llm_response additionally falls back to reasoning_content when content is empty — the long-term ALTK fix asked for in rossoctl/cortex#676.

Both duplicated copies of _parse_llm_response in litellm.py were fixed; patching only one would have left the sibling client broken.

Verification against live providers

All checks use the real SPARC function_selection_appropriateness schema (nested object arrays, ["string","null"] unions, additionalProperties: false) — not a simplified stand-in. Native provider structured output is used wherever the model supports it.

Provider / model	Before	After
watsonx openai/gpt-oss-120b (end-to-end SPARC)	decision: error, 100% of calls	correct APPROVE and REJECT verdicts, 0 LLM errors, stable across repeated runs
watsonx openai/gpt-oss-120b (schema call, 10 trials)	0/9 valid	9/10 valid; the one failure is genuine backend flakiness with all retries correctly consumed
Azure gpt-4o-2024-08-06	hard TypeError	native structured output valid; SPARC returns correct verdicts
anthropic/claude-haiku-4-5 via litellm	—	valid, all required fields present

Per-model routing confirmed: watsonx/openai/gpt-oss-120b → prompt-based, watsonx/mistralai/mistral-large → native, Azure → native.

Tests

11 new tests covering each fix: schema-fidelity (bounds, enum-in-items, nested array objects, additionalProperties, nullable enum), retry behavior (ValueError retried, no empty assistant turn, max_tokens escalation, capability-based fallback), and the reasoning_content fallback.

tests/core: 142 passed. ruff check, ruff format --check, mypy . (370 files), and uv lock --check all clean.

One updated assertion: test_freeform_flag_keeps_nested_objects_as_dict asserted nested objects stay dict. That encoded the very limitation causing these failures — and a nested model satisfies OpenAI strict mode better than a bare dict, which cannot emit additionalProperties: false. Renamed to test_freeform_flag_recurses_into_nested_objects.

tests/core/test_auto_from_env.py::test_selecting_watsonx fails on main as well, independent of this branch (it reads a local ~/.wca config); left untouched.

[jsntsay]

LGTM