feat(skill): evidence-backed eval config generation for run-assert-eval

[alex ngo (ango10)]

Summary

Integrates the evidence-backed eval-config generation methodology from WIP-amagooda/eval-template into the run-assert-eval skill, and regenerates every example eval config through it. Configs are now researched against primary sources, gated on ≥2 independent citations, human-approved before write, and emitted one-risk-per-suite.

Motivation / linked issue

run-assert-eval previously produced configs by shelling out to assert-ai init and hand-editing params. Dimensions were mined heuristically from Clarity variants, nothing was sourced, deduplicated, or approved. amagooda's branch replaces that guesswork with a science-backed procedure. That branch is not being merged; this branch supersedes it, so this PR carries its functionality forward.

Changes

Skill methodology port (.claude/skills/run-assert-eval/)

• New workflows: research-eval-dimensions.md (the harm procedure), iterative-dimension-workflow.md, generation-isolation-workflow.md, evaluation-intent-workflow.md
• New scripts plan_generation_path.py and validate_dimension_review.py (path-only isolation planner + blocking pre-write/post-write approval gate), plus assets/eval-config-template.yaml and assets/dimension-review-template.md
• SKILL.md Step 3 now delegates to the research procedure instead of assert-ai init
• Mirrored to .github/prompts/run-assert-eval.prompt.md and .cursor/rules/assert.mdc

Core assert_ai engine integration ported from the same branch

• init/: web search in assert-ai init ( --web-search , Responses API web_search_preview) and the harm-template mode injected into init_system.md
• core/model_client.py: web-search tool plumbing (+129)
• integrations/acs/eval_config.py: ACS-generated configs realigned to the researched sizing. behavior_category_count 6→25, sample_size 12→25, max_turns 4→6

Library 4 safety presets the methodology depends on

• hate_speech_harassment, malicious_cyber_activity, sexual_content, violent_content, each as a library preset.

Examples regenerated through the new workflow

• 22 configs regenerated across azure_doc_qa, change_control_agent, science_research_agent, travel_planner_langgraph, travel_planner_neurosan, prompt_agents.

Testing

• pytest and ahmedmagooda's 13 script tests ported over.
• All 22 regenerated configs validated through the engine's own load_runtime_context(..., stage_modules=STAGES). The same path assert-ai run takes confirming 25/25/25, max_turns: 6, resolvable targets, and no judge-dimension shadowing
• Repo-wide scans for dead  examples/**  paths, bare stale filenames, unknown suite names, and unresolvable  examples.*

Checklist

• Tests pass locally (pytest and/or viewer checks as applicable).
• Docs updated if behavior or public API changed.
• No secrets, credentials, or customer data committed.
• No breaking change, or a CHANGELOG.md entry is included.

[Chang Liu (changliu2)]

Requesting changes on exact head 7493b49 for six correctness and security issues:

1. Automatic harm-template mode is advertised by pip-installed assert-ai init, but its .claude methodology is not packaged, so the mode runs without the instructions it requires.
2. The post-write validator accepts any YAML mapping and does not verify runtime schema, approved behavior identity, retained dimensions, or one-risk atomicity.
3. Preset anti-shadowing fails outside an ASSERT source checkout; safety-core passes when ASSERT is wheel-installed because unresolved presets are silently skipped.
4. Responses-API fallback drops web search but preserves a prompt claiming live research is available, allowing fabricated citation output.
5. Web search is now opt-out and can expose prompt-derived terms to external search and ingest untrusted page content without a prompt-injection boundary.
6. Changing the global scenario default from 10 turns to 6 silently changes existing customer evaluations that omit max_turns.

Checks are green but do not cover the installed-wheel, fallback, or fail-closed validation cases above.

[alex ngo (ango10)]

In regards to 5, on 08/31/2026 Abby stated "without the web_search, it has hallucination tendencies and the skill's main value add is it does literature review and comes up with relevant factors. I think it should be true."

In regards to 6, on 08/24/2026 Abby stated best practice is to reduce it to 6, since the 10 to 20 range was calibrated specifically for psychosocial risks rather than chosen as a general-purpose value. He also approved changing the core default directly.

[Chang Liu (changliu2)]

Re-reviewed exact head 7ae3c96. Wheel packaging, packaged preset resolution, search-fallback honesty, runtime-schema validation, behavior identity, and normal non-empty approved-dimension matching are fixed.

Three blockers remain:

1. The post-write gate's effective judge-dimension comparison ignores preset-contributed dimensions and skips comparison when the approved set is empty. This rejects a correctly approved safety-extended config while allowing arbitrary unapproved dimensions after an empty approval.
2. Default-on external search still proceeds without runtime disclosure or affirmative consent and without a system boundary treating retrieved web content as untrusted data. Keeping search enabled by default is a product choice; these disclosure and indirect-prompt-injection defenses are separate security requirements.
3. Reducing the global omitted max_turns value from 10 to 6 silently changes existing configurations. Six can be the preferred value for newly generated configs, but legacy behavior needs an explicit value or a documented/versioned migration.

[alex ngo (ango10)]

Requested change 1 and 2 addressed.

Requested change 3: Resolved default is 6, documented as breaking.

• CHANGELOG entry under [Unreleased] / Changed, marked **Breaking, planned for 0.3.0:** in the same format refactor(deps): isolate optional dependency environments #336 used, so both breaking changes ship in one release rather than two.
• It records the comparability caveat, that max_turns: 10 restores prior behavior, that single-turn prompt cases never read the value, and that the value feeds _inference_config_fingerprint, so a run interrupted before the upgrade re-executes rather than resumes.
• docs/config/schema.md:203 already reads Default: 6.

[Chang Liu (changliu2)]

Re-reviewed current head a97ed3e. The three remaining blockers are resolved:

• The post-write gate now compares preset-contributed judge dimensions and rejects unapproved dimensions after an empty approval.
• Default-on web research is disclosed at runtime, interactive runs request confirmation, and retrieved content is explicitly bounded as untrusted data.
• An omitted max_turns resolves to 6, with the breaking 0.3.0 migration, comparability caveat, max_turns: 10 compatibility path, and resume-fingerprint behavior documented.

The focused regression cases pass and all required checks are green. Approving.