Add static prompt quality evaluations

[Cedric Vidal (cedricvidal)]

Summary

Replaces closed #1398 with a new PR from the restored fork branch.
The original PR remains closed. The CI foundation #1442 is merged;
its published history is incorporated without changing evaluation content.

• Add production-backed quality evaluations for 10 static prompt families and 15 adapter variants.
• Commit 222 curated input cases; keep generated responses and evaluation artifacts ignored.
• Add rubric-driven grading, gate-based reports, historical rubric snapshots, immutable selective regrading, and case-level evidence.
• Retain the previously implemented native grader parsing, task context, SDK tool-call schemas, incomplete-sample handling, and replay source protection.
• Extract unused cloud red-team orchestration into a separate dependent contribution. The quality runner defaults to quality and retains explicit --mode quality; removed red-team modes fail before execution.
• Preserve normal evaluation logic, prompts, rubrics, datasets, dependency versions, and OneRAI-shared evidence. The extraction applies none of the review fixes from Add static prompt evaluation and red-team framework #1398.

Extraction validation

• TypeScript typecheck passed; 32 remaining TypeScript tests and 94 Python tests passed.
• Dataset validation: 222 approved cases, 10 quality families, 15 quality targets.
• All 1,106 protected quality/dependency/evidence file hashes match the preservation snapshot.
• Pre/post fake-transport comparison: all 666 rows match across 222 cases and three samples, excluding only latencyMs.
• No live model generation, paid Azure evaluation, or assessment regrading was run.
• No generated results or unrelated local editor settings are included.
• CI for the newly published head must be assessed separately.

Evaluation evidence

Quality orchestration runs locally using Azure AI Evaluation SDK graders.
The retained 222-case assessment reuses recorded responses: 221 successful generation rows and one error.
Execution completed, acceptance failed, and assessment integrity remains incomplete.
Blocking gates: 65 passed, 14 failed, 8 unresolved; 2 advisory violations.
No new model generations or paid Azure evaluations were run for this replacement or extraction.

Before this extraction, aggregate pass-rate floors above 80% were lowered in the rubric only.
The engine continues to honor configured rates through 100%; mean-score requirements and lower floors are unchanged.
The extraction makes no additional rubric or evidence changes.
The interactive review extension remains session-scoped, outside the repository.

Known limitations and follow-up

• Suggested-feature novelty receives existing-feature detection output instead of generated suggestions in 28 of 30 current cases; these scores are not reliable novelty assessments.
• Feature-authoring candidate checking can omit child references because it selects the first configured field instead of combining parent and child IDs.
• Variation duplicate checking can omit the original existingPrompt and does not combine all relevant prompt collections.
• Dependency references need validation and explicit authority/completeness rules before adopting deterministic set-based grading.
• Feature-extraction mean requirements of 1.0 still require perfection despite separate 80% pass-rate floors. Custom 1-5 rubrics need behavioral score anchors.
• Missing original judge tool traces, one report-generation error, and incomplete generator provenance remain visible limitations; no evidence was fabricated.
• The eight quality review findings from Add static prompt evaluation and red-team framework #1398 remain deferred; extraction is not remediation.
• Quality results are not a completed safety/red-team assessment or a clean pass. Red teaming was not used for the OneRAI evaluation; its previous live baseline was blocked by missing Foundry Microsoft.CognitiveServices/accounts/AIServices/evaluations/write permission.

[Cedric Vidal (cedricvidal)]

Wassim Chegham (@manekinekko) I've addressed the eight static-prompt quality-evaluation review comments carried over from the now-closed #1398:

• Separate generator configuration from grader configuration.
• Treat generation infrastructure failures as unresolved rather than prompt-quality failures, including sample diversity.
• Retain judge completion metadata and associate tool traces with their requests.
• Include generated feature suggestions in novelty-grader inputs.
• Allow explicitly selected regrading of missing or unusable retained assessments.
• Accept empty historical observation streams during replay.
• Reject nonexistent explicit dataset paths instead of silently loading the full manifest.
• Accept valid empty feature-catalog outputs while retaining feature-coverage checks.

Publication status: these fixes are implemented in local commit a1c23a94 but have not yet been pushed, so they are not in the current PR head (781b4c57).

Production prompt text, curated cases, and score thresholds are unchanged. Existing OneRAI evidence has not been modified or regraded, and no live generation or paid grading was run. The three red-team-specific review comments remain deferred to the separate red-team work.