Add static prompt quality evaluations - #1441

Merged

Cedric Vidal (cedricvidal) merged 39 commits into

microsoft:mainfrom

cedricvidal:cedricvidal-plan-static-prompt-evals

Sep 30, 2026

Commits on Sep 9, 2026

refactor: expose prompt composition seams

Cedric Vidal (cedricvidal)
and
Copilot
committed
9f5511a
feat: add static prompt evaluation suites

Cedric Vidal (cedricvidal)
and
Copilot
committed
2bceee9
docs: document prompt evaluation maintenance

Cedric Vidal (cedricvidal)
and
Copilot
committed
5123706
test: use synthetic redaction fixture

Cedric Vidal (cedricvidal)
and
Copilot
committed
a3b86de
fix: isolate concurrent judge evaluation workspaces

Cedric Vidal (cedricvidal)
and
Copilot
committed
768b627
fix: delete temporary red-team agents

Cedric Vidal (cedricvidal)
and
Copilot
committed
da10457
fix: report actual red-team target mode

Cedric Vidal (cedricvidal)
and
Copilot
committed
7d9b913
feat: guide developers when uv is missing

Cedric Vidal (cedricvidal)
and
Copilot
committed
bdf91dc
feat: generate markdown evaluation reports

Cedric Vidal (cedricvidal)
and
Copilot
committed
0751e51
docs: explain evaluation policy gating in reports

Cedric Vidal (cedricvidal)
and
Copilot
committed
65b06e7
style: remove trailing blank lines

Cedric Vidal (cedricvidal)
and
Copilot
committed
20ad442
fix(evals): cap aggregate pass-rate policy at eighty percent

Cedric Vidal (cedricvidal)
and
Copilot
committed
b7d5b6a
fix(evals): parse native grader outputs and preserve task context

Cedric Vidal (cedricvidal)
and
Copilot
committed
a182160
feat(evals): add auditable gate decisions and immutable selective replay

Cedric Vidal (cedricvidal)
and
Copilot
committed
4ace354
fix(evals): standardize decision summary and read-only legacy exports

Cedric Vidal (cedricvidal)
and
Copilot
committed
170260d
fix(evals): keep eighty-percent policy solely in rubric configuration

Cedric Vidal (cedricvidal)
and
Copilot
committed
e43a420

Commits on Sep 10, 2026

fix(evals): handle pnpm separator in report CLI

Cedric Vidal (cedricvidal)
and
Copilot
committed
4a1061a
fix(evals): resolve incomplete sample votes and reject source overlap early

Cedric Vidal (cedricvidal)
and
Copilot
committed
c0fbd7a
fix(evals): normalize SDK tool-call fields and retain evaluator diagnostics

Cedric Vidal (cedricvidal)
and
Copilot
committed
1ad33a6
fix(evals): explain unresolved report gates and format scores

Cedric Vidal (cedricvidal)
and
Copilot
committed
8e2813b
docs(evals): explain case context in review clients

Cedric Vidal (cedricvidal)
and
Copilot
committed
e4b0195
Merge upstream main and preserve prompt evaluation builders

Cedric Vidal (cedricvidal)
and
Copilot
committed
7485982
fix: restore clean evaluation builds and scope hash scan exemption

Cedric Vidal (cedricvidal)
and
Copilot
committed
9aba7f0

Commits on Sep 29, 2026

Merge current upstream main for replacement evaluation PR

Cedric Vidal (cedricvidal)
and
Copilot
committed
719bb66
fix(ci): restore upstream repository gates safely

Cedric Vidal (cedricvidal)
and
Copilot
committed
5cafe1b
fix(ci): limit repository gate correction to four replacements

Cedric Vidal (cedricvidal)
and
Copilot
committed
d28f298
fix(ci): repair integration and image workflow execution

Cedric Vidal (cedricvidal)
and
Copilot
committed
0c5015b
fix(ci): remove obsolete VS Code image tag branches

Cedric Vidal (cedricvidal)
and
Copilot
committed
74f20b4
ci: separate public validation from internal automation

Cedric Vidal (cedricvidal)
and
Copilot
committed
0a577d7
ci: make Windows builds and CLI releases self-contained

Cedric Vidal (cedricvidal)
and
Copilot
committed
d326467
Merge CI foundation for dependent prompt evaluation PR

Cedric Vidal (cedricvidal)
and
Copilot
committed
58fe058
fix(docs): route website CLI installs to public releases

Cedric Vidal (cedricvidal)
and
Copilot
committed
8bffe1f
fix(cli): isolate standalone bundle and require an explicit API URL

Cedric Vidal (cedricvidal)
and
Copilot
committed
a61416d
fix(cli): isolate standalone bundle and require an explicit API URL

Cedric Vidal (cedricvidal)
and
Copilot
committed
d60a71e
Merge upstream main into CI and standalone release fixes

Cedric Vidal (cedricvidal)
and
Copilot
committed
c3689bc
Merge resolved CI foundation and upstream ACP test utilities

Cedric Vidal (cedricvidal)
and
Copilot
committed
f266d76

Commits on Sep 30, 2026

Merge published CI foundation into evaluation branch

Cedric Vidal (cedricvidal)
and
Copilot
committed
216d21b
refactor(evals): extract unused red-team execution from quality suite

Cedric Vidal (cedricvidal)
and
Copilot
committed
50a9590
docs(evals): document quality-only scope after red-team extraction

Cedric Vidal (cedricvidal)
and
Copilot
committed
781b4c5