docs: metrics.json consumption contract for dashboards (#96) - #104
Conversation
#96 asked the review-bot kanban to show review metrics; the per-run data already exists (metrics.py writes metrics.json beside RUN_REPORT.md) but no page said which fields a consumer may trust or how to read them. doc/guide/metrics.md documents the schema-1 payload field by field with the trust rules a dashboard needs: q must be shown with weight_coverage, null means refused-to-fabricate (never zero), usd is trustworthy only when source="spans" and cost_partial=false, cost_index is deliberately null on partial cost, and harness-backend runs surface through the same cost_partial signal. Compatibility rule stated: additive-only within schema 1, breaking changes bump schema. Indexed in doc/README.md. Refs #96 (the dashboard itself is omni-reviewbot work). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0165QsXAeLVPrKfj8Yk5zns3
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3ac32d11f7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - 路径：`~/.infermatrix-copilot/runs/run-<ts>-<uuid6>/metrics.json`，与 | ||
| `RUN_REPORT.md` 同目录。 |
There was a problem hiding this comment.
In deployments that set RUN_ROOT/Settings.run_root, runs are written beneath that configured directory rather than ~/.infermatrix-copilot/runs (config.py:183, with environment-backed settings at config.py:84-95). Presenting the default path as the contract can make an external dashboard watch the wrong directory; describe it as <run_root>/run-.../metrics.json and identify the current default separately.
Useful? React with 👍 / 👎.
| - 路径：`~/.infermatrix-copilot/runs/run-<ts>-<uuid6>/metrics.json`，与 | ||
| `RUN_REPORT.md` 同目录。 | ||
| - **尽力而为**：指标计算失败绝不打断 run —— 消费方必须容忍文件缺失。 | ||
| - BLOCKED 的 run 也会产出 metrics（`status` 字段说明终态）。 |
There was a problem hiding this comment.
For MCP-reserved runs blocked during planning or tier preflight, _execute_reserved_locked marks the run BLOCKED and returns before _execute (cli/copilot.py:569-578), while metrics are only collected inside _execute at lines 412-418; metrics can also be disabled by metrics_enabled. Such blocked run directories therefore have no metrics.json, so the unconditional statement should be limited to blocked runs that reach metrics collection.
Useful? React with 👍 / 👎.
| | 字段 | 含义 | | ||
| |---|---| | ||
| | `risk.incidents` | 按严重级（catastrophic/severe/moderate/minor）的事件计数 | | ||
| | `risk.safety_multiplier` | S ∈ (0,1]，乘进 CATQ | |
There was a problem hiding this comment.
When any catastrophic incident is recorded, safety_multiplier explicitly returns 0.0 (metrics.py:172-175). Consumers validating against the documented S ∈ (0,1] range could reject or mishandle exactly these highest-risk runs, so the contract must state S ∈ [0,1] and note the catastrophic-zero case.
Useful? React with 👍 / 👎.
1 participant
What
New page
doc/guide/metrics.md: the stable consumption contract for the per-runmetrics.jsonthatmetrics.py::collect_run_metricsalready writes beside everyRUN_REPORT.md— written for external consumers (the review-bot kanban of #96, statisticsscripts).
Documents, field by field for the
schema: 1payload:qmust always be shown withweight_coverage;partial/abstainedsemantics (abstain scores a baseline, not 0).
source="spans"andcost_partial=false— and "complete" means fully priced at configured/table rates, notinvoice-exact;
source="events"renders as "≈" (estimate),cost_partial=trueas "≥"(lower bound, with
cost_indexdeliberatelynull— never fabricated). Harness(subscription) backends surface through the same two fields, no backend-specific check
needed.
schema1; breaking changes bumpschema.Indexed in
doc/README.md's guide table.Why
#96: "our review-bot kanban now does not show the metrics of the review task." The data
already exists per run; what was missing is a contract saying which fields a dashboard may
trust and how to read them. This is the copilot-side half of the split proposed in the
issue triage — the dashboard itself is omni-reviewbot work.
Verification
tools/check_doc_links.py(296 files OK),tools/check_doc_citations.pyOK.metrics.py(collect_run_metrics,cost_from_spans,quality_score,_phase_timings) at head.Refs #96.
🤖 Generated with Claude Code
https://claude.ai/code/session_0165QsXAeLVPrKfj8Yk5zns3