github-workflow: re-measure the session corpus after 1.5 has deployed (OQ-004)

[chrisdpurcell]

Re-scoped 2026-08-31. The post-1.5 measurement window was overtaken by 1.6 and 1.7
before it opened, and 1.7 moved the surfaces F3/F4/F8 measure (+22.9% guidance bytes, two
new subcommands). The measurement is now scoped to a post-1.7 window opening ~2026-09-11.
The scope, dates, and baselines below are superseded by
this comment;
read it first.

Outcome

A re-run of the session-corpus efficiency review's method, scoped to a post-1.5-deployment window comparable to the original review's post-repair era, exists and reports whether package version 1.5's guidance restructure changed agent behaviour on the four live findings it targeted — with a keep/iterate decision recorded against D-012 and D-014's reopen triggers.

Context

The session-corpus efficiency review measured six live findings against the corpus as of 2026-08-26. Spec rev 1.9 (docs/specs/2026-08-06-github-workflow-package-spec.md, package version 1.5, commit eb9812e on testing) restructures SKILL.md and the managed block specifically to answer F3 (whole-file guidance loads), F4 (field-vocabulary.md size), F8 (help/-h rediscovery), and F10 (delegated workers bypassing the load path). D-012 and D-014 each record an explicit reopen trigger conditioned on a post-1.5 corpus window, and OQ-004 in the spec is open precisely on this question: "Did the 1.5 guidance restructure actually change agent behaviour? ... measurable only after 1.5 has been deployed for a comparable window."

The original review's post-repair era (v5.20.0, 2026-08-15) was 58 sessions and 389 invocations over eleven days — explicitly flagged in the review as "small and recent."

Scope

• Re-run the review's method (as documented in its "Method" and "Counting rules" sections) over a post-1.5-deployment window that is comparable in duration and session count to the original post-repair era (~11 days / ~58 sessions), across the same six repositories that install github-workflow.
• Re-derive these four metrics for the new window and compare them against the review's pre- and post-repair figures:
  • guidance bytes loaded per session (the F3 "bytes of guidance per post-repair session" measure, and whether loads remain whole-file);
  • field-vocabulary.md (or its 1.5 orientation-subset successor) load count (F4);
  • gh-workflow help / <subcommand> -h call count (F8);
  • raw-gh routing violations for actions the routing table now covers, including delegated-worker sessions where recoverable (F10).
• Record the keep/iterate decision this data feeds against D-012's and D-014's reopen triggers in the spec's Deviations Log or a linked follow-up entry.

Out of scope

• Re-litigating F1, F2, F5 (routing violations already at zero), F6 (CI waiting — no subcommand exists to route to), F7, or F9, which the original review already resolved or found not live.
• Any package change; this issue is measurement only. A finding that a metric has not improved is itself a valid outcome and becomes its own follow-up issue rather than being fixed inline here.

Acceptance criteria

• A dated report (new docs/reviews/github-workflow/ entry or an amendment to the existing review) exists that:
  • states the comparison window (start date, end date, session count, invocation count) and confirms it is comparable in scale to the original ~11-day / ~58-session post-repair era;
  • re-derives the four metrics named in Scope, each compared numerically against the review's pre-repair and post-repair figures;
  • cites the review's own "Method" and "Counting rules" sections as the method reference, noting any deviation;
  • states explicitly whether each of D-012's and D-014's reopen triggers fired, and records the resulting keep-as-is or iterate decision.
• The report is linked from OQ-004's row in docs/specs/2026-08-06-github-workflow-package-spec.md (or OQ-004 is otherwise marked answered with a pointer to it).

Evidence / references

• docs/reviews/github-workflow/2026-08-26-0847-github-workflow-session-corpus-efficiency-review.md — F3, F4, F8, F10, and the corpus-scale table.
• docs/specs/2026-08-06-github-workflow-package-spec.md — revision 1.9 row; D-012 and D-014 reopen triggers; OQ-004.
• Package version 1.5 deployment (commit eb9812e, testing branch) is the measurement's starting point; the window cannot begin before that deployment reaches the corpus's six repositories.

[chrisdpurcell]

Not rolled into the v5.24.0 train (2026-08-27): measurement-only, and its window has not elapsed — github-workflow 1.5 reached consumers with v5.23.0 on 2026-08-26, and the comparison window needs ~11 days / ~58 sessions. Re-measure no earlier than 2026-09-06.

[chrisdpurcell]

Reviewed 2026-08-29 while addressing the open github-workflow issues: the measurement window has not elapsed (github-workflow 1.5 reached consumers with v5.23.0 on 2026-08-26; the comparison window needs ~11 days / ~58 sessions, so re-measurement is not possible before 2026-09-06). No action taken; the issue stays open and remains measurement-only. Note for the eventual measurement: github-workflow 1.6 (finding-disposition rule replacing invariant 15, on testing as of 2026-08-29, ships in v5.25.0) will overlap the window — record which package version each session ran, since 1.6 changes only prose in SKILL.md, the block, and two references, not the routing table or subcommand surface F3/F4/F8/F10 measure.

[chrisdpurcell]

Re-scoped: the post-1.5 window was overtaken before it opened

Reviewed 2026-08-31 while cutting the github-workflow 1.8 correction payload. The
premise this issue was written on no longer holds, and no wait makes it hold again.

What happened. The comparison window was to open 2026-09-06 (1.5 reached consumers
with v5.23.0 on 2026-08-26; ~11 days / ~58 sessions). Two further package versions
landed inside that unopened window:

• 1.6 — v5.25.0, 2026-08-29. Prose only in SKILL.md, the managed block, and two
  references, as the previous comment recorded.
• 1.7 — v5.26.0, 2026-08-30, consumer fleet migrated 2026-08-31. This one moves the
  measured surfaces themselves.

Why 1.7 invalidates the original measurement. Every guidance file changed, and so did
the binary. Measured over skills/github-workflow/** (*.md plus agents/openai.yaml):

Package	SKILL.md	field-vocabulary.md	pr-standard.md	review-checklist.md	Total guidance
1.5	11,176	2,918	3,907	2,834	30,684
1.6	11,711	2,918	4,125	2,834	31,472
1.7	11,978	3,111	7,658	3,417	38,673
1.8	11,978	3,111	8,098	3,482	39,178

Total guidance grew 31,472 → 38,673 bytes at 1.7, +22.9%, and the subcommand surface
gained ready and merge (eight subcommands to ten). F3 measures guidance bytes per
session, F4 measures field-vocabulary.md loads, and F8 counts help / -h calls
against the subcommand surface — 1.7 moved the baseline under all three. A window running
2026-08-26 → 2026-09-06 spans three package versions, so it cannot answer "did 1.5's
restructure change behaviour" for any of them.

1.8 does not move the baseline again. SKILL.md and field-vocabulary.md are
byte-identical to 1.7's, the routing table and subcommand surface are unchanged, and the
only guidance bytes that move are pr-standard.md (+440 B) and review-checklist.md
(+65 B) — +505 B total, +1.3%. A window opened after 1.7 stays valid across the 1.8
adoption; record which version each session ran, but no third baseline is needed.

Revised scope

The question is still worth answering; it just belongs to the 1.5-through-1.7 guidance
line rather than to 1.5 alone.

• Re-run the review's method over a window beginning at the 2026-08-31 fleet migration
  (the date every consumer that installs the package reached 1.7), comparable in scale to
  the original ~11-day / ~58-session post-repair era. Earliest re-measurement is therefore
  ~2026-09-11, not 2026-09-06.
• Re-derive the same four metrics (F3, F4, F8, F10) against the review's pre-repair and
  post-repair figures, using 1.7's guidance total (38,673 B) as the denominator for
  F3's "whole-file load" judgement rather than 1.5's 30,684 B, and 1.7's ten-subcommand
  surface for F8.
• Record the package version each session ran (1.5, 1.6, 1.7, or 1.8). The 1.5→1.7 window
  is contaminated by construction and the report must say so rather than average over it.
• D-012's and D-014's reopen triggers were written against a post-1.5 window that no longer
  exists. State explicitly that they are being evaluated against the post-1.7 window
  instead, and record the keep/iterate decision on that basis.

Unchanged

Still measurement-only. A finding that a metric has not improved remains a valid outcome
and becomes its own follow-up issue rather than being fixed inline here.