Skip to content

Read the steady-state verdict from result_summary.json - #187

Draft
anandhu-eng wants to merge 1 commit into
mainfrom
fix/steady-state-from-result-summary
Draft

anandhu-eng wants to merge 1 commit into
mainfrom
fix/steady-state-from-result-summary

Conversation

@anandhu-eng

Copy link
Copy Markdown
Collaborator

Fixes #162

The checker read the §4.4 steady_state block from point.yaml, but nothing writes it there. The client writes its verdict to result_summary.json, so submitters copied it by hand, and some copies were wrong (for example, an off-by-one super-pass count).

Changes:

  • steady-state-basis reads steady_state from result_summary.json and works out the §4.4 status and verdict from the client's fields.
  • A missing field (old client), null (detection off) and an unreadable block each get their own warning.
  • Agentic points and a dedicated Offline run are skipped. §4.4 covers fixed-concurrency points only, and the client does not run detection for agentic.
  • §6.2 uses the window duration only when the client accepted the window.
  • steady-state-valid and steady-state-consistency are removed. They only checked the hand-written block. A steady_state block left in point.yaml is ignored.
  • The fixture tool moves the corpus verdict into result_summary.json (101 points regenerated, --check clean).

Mapping from the client's block:

§4.4 From the client
window sp_lo .. sp_hi - 1 (sp_hi is exclusive); duration = end_ns - start_ns
partial_dataset fewer than 1 super-pass after warmup
insufficient_passes no window, or a window under 4 super-passes
insufficient_duration found: false with a 4+ pass window (client rejected it as short), or a window under the §6.2 minimum
windowable otherwise
verdict not found if found is false, else anomaly, else drifting_up / drifting_down from global_trend, else STEADY STATE

Severities do not change: everything here is still a warning or info.

Open questions for the rules (not changed here):

  • §8.3 still lists steady_state as a point.yaml field.
  • §4.4 names the gating metrics as TPOT P50/P90 in one place and TTFT and TPOT P50/P90 in another. This PR treats drift in any of the four as a drift verdict, which is a warning.

🤖 Generated with Claude Code

The checker read the §4.4 block from point.yaml, which nothing writes;
submitters copied it by hand from the client's result_summary.json.
The checker now reads the client's verdict there and works out the
§4.4 status and verdict itself.

- steady-state-basis reads result_summary.json and tells apart a missing
  field (old client), null (detection off) and an unreadable block.
- Agentic points and a dedicated Offline run are skipped (§4.4 scope).
- §6.2 uses the window only when the client accepted it.
- steady-state-valid and steady-state-consistency are removed; they
  only checked the hand-written block.
- The fixture tool moves the corpus verdict into result_summary.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anandhu-eng
anandhu-eng marked this pull request as draft November 9, 2026 17:23

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

steady_state block: the checker reads it from point.yaml, but the client writes it elsewhere

1 participant