fix(systematize): align structured output and preserve diagnostics

[Jake Present (jakepresent)]

Summary

• align the stage-one structured-output schema with the JSON document requested by the systematization prompt;
• stop forcing the full document into an escaped systematization string, and keep the generated artifact structured for conversion;
• generate masked, provider-safe structural examples for harmful categories instead of reproducing explicit harmful text;
• classify provider content filtering directly and stop retrying the same filtered request as a transient parse failure;
• preserve compatibility with cached/pre-0.2.1 artifacts that store Markdown or JSON text in the legacy string field;
• save complete failed responses, provider metadata, and credential-sanitized requests to persistent diagnostic files while keeping terminal previews concise.

Root causes

Prompt/schema mismatch

A real 0.2.0 pipeline failure returned output beginning with:

{"systematization":"{\"behavior\":...

The prompt requested a top-level document containing behavior, scope, concept_spec, and the other systematization fields. The response schema instead declared only a systematization string and summary_items, so the model attempted to serialize the requested document inside that string. The revised Pydantic contract mirrors the prompt directly, and conversion accepts both the new object and legacy string artifacts.

Output content filtering during conversion

A rerun with the aligned contract passed stage one and produced the new diagnostic at conversion. It showed:

• finish_reason: content_filter;
• Azure output filtering marked hate content as filtered/high severity;
• prompt filtering was safe;
• output stopped after 2,218 tokens, so this was not the configured 8,000-token budget or quota.

The conversion prompt was asking for 4–8 concrete raw examples per category and explicitly encouraged severe examples. For harmful taxonomies, that made the model reproduce the content being measured and triggered the provider's output filter. Both systematization prompts now require concise masked structural fragments with square-bracket placeholders for harmful material, while safe categories retain ordinary examples. A content-filter finish reason now raises one accurate error and diagnostic instead of making an identical second attempt.

Validation

• contract regression failed against the old schema with the same double-encoded shape seen in the first pipeline log;
• content-filter regression reproduced the second diagnostic's finish_reason: content_filter, failed under the old generic retry path, and now stops after one call with reason: content_filtered;
• focused model-client, diagnostics, systematization, and cache tests: 121 passed, 4 subtests passed;
• the same focused suite passed under Python 3.12 with the reported dependency versions (assert-ai 0.2.0, litellm 1.97.0, click 8.4.2, and pydantic 2.13.4);
• full regression suite: 1,451 passed, 22 skipped, 840 subtests passed;
• built and installed the wheel in a clean Python 3.12 environment; packaged prompts and content-filter detection were verified;
• git diff --check clean.

Data handling

Diagnostics stay local with the suite. Request payloads pass through the existing credential sanitizer. Responses remain complete in the file and may contain evaluation content, so normal terminal output only shows a short preview and the diagnostic path.

Remaining validation and deployment requirement

The two reported failure modes and exact Python/LiteLLM combination are covered locally. A follow-up rerun confirmed that masking harmful examples is not sufficient for every safety behavior: Foundry blocked the high-level sexual_content_generation systematization itself as high-severity completion content before the response reached the pattern examples. The prompt remained classified safe.

With standard Azure OpenAI completion filtering, High-severity output remains blocked even at the least restrictive normal threshold. Running these red-team systematization and test-generation stages therefore requires either a deployment approved for Modified Content Filters (Annotate only or No filters on completions) or a different internal-generation provider. The evaluated target can remain unchanged. The prompt changes in this PR still reduce unnecessary explicit content, while the diagnostic and direct error make this deployment limitation visible.