Learning #13 — a handoff's predictions decay - #63
Open
rmsharp wants to merge 1 commit into
Open
Conversation
Appends one row to the Learnings table (was 1-12; append only, never renumber — this file is bin/_manifest.py-distributed). A handoff carries predictions as well as facts, and a prediction about state the handoff's own session later changed is the least reliable kind. The existing rows do not cover this: #6 and FM KJ5HST#11 are about claims written from memory, and KJ5HST#7/KJ5HST#10/KJ5HST#12 are about cross-references that go stale in the corpus. Neither addresses a forward-looking claim whose expiry is caused by the writing session's own later commits. Motivating case is this repo's own v3.6 ship. The predecessor receipt said "expect one CHANGELOG.md union conflict"; the merge produced seven, because Layer 8 and the pre-PR review fixes were committed to the branch after the point the fork had been ported from it. Five files the prediction never named had diverged, including both dashboard twins and the unit suite. A wrong prediction is worse than no prediction: it licenses the successor to resolve quickly rather than carefully, and a merge resolved on the belief that "only the ledger differs" discards code silently, with no failing check to catch it. Cross-reference sweep per Learnings KJ5HST#7/KJ5HST#10: no live count-claim about the table exists in the corpus — the "table was 1-11"/"table is 1-12" tokens are confined to dated CHANGELOG entries and HANDOFFS receipts, left verbatim by design, and the caption states no size. Phase 3C of fork session S17, whose durable receipt is the HANDOFFS.md S17 block on the fork; no new receipt is opened here. Verified: bin/tests.sh 84/84, bin/check-links OK, 197/197 unit, Learnings table reads 1-13 with every row 4-column. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rmsharp
added a commit
to rmsharp/methodology
that referenced
this pull request
Jul 27, 2026
Completes the one close-out step a5dc925 left open, and records the PR open as a non-commit action the ledger owes an entry for (FM KJ5HST#27). The learning itself does not land here. starter-kit/SESSION_RUNNER.md is bin/_manifest.py-distributed, so committing an unreleased change to it on fork main would break the invariant this session had just verified and reported — that fork and upstream differ only by fork-only docs/planning/* plus this repo's own ledger and receipts. The branch was cut from upstream/main (0 behind) per the standing rule for clean single-topic upstream PRs. The S17 receipt is amended in place rather than duplicated: this is the same session's Phase 3C, not a new session, and a second receipt for one session would double-count it in the ledger built to prevent that. Verified: bin/check-handoff OK. On the branch: bin/tests.sh 84/84, bin/check-links OK, 197/197 unit, Learnings table 1-13 all 4-column. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rmsharp
added a commit
to rmsharp/methodology
that referenced
this pull request
Jul 30, 2026
Ratified plan for recording which model(s) ran each session/action. No dedicated field exists today in either CHANGELOG.md or HANDOFFS.md; model info currently leaks out only incidentally (free-text prose on capability-tiered sessions, or git commit co-author trailers). Design panel (3 candidates, 3 judges x 4 lenses, synthesis, adversarial completeness critique) surfaced the load-bearing finding: git commit trailers can directly misattribute capability-tiered work. S1's own receipt says Sonnet 5 built P2/P4, but all six of S1's checkpoint commits — including those two — are trailer-tagged Opus 4.8. Verified independently three times (design panel, synthesis, this session). Ratified design (operator decisions D1-D4, all via AskUserQuestion): optional **Model:** bullet in CHANGELOG.md's per-action entry (scales to multi-tier for free, since the ledger is already per-layer) + a formalized free-text convention in HANDOFFS.md for session-level lookup + a SESSION_RUNNER.md Phase 3F propagation clause + a canonical-only bin/model-report tool that treats git trailers as disclaimed corroboration only, never authoritative. No change to bin/check-handoff or REQUIRED_KEYS. Two defects the completeness critic found in the synthesis are corrected in place: a broken regression-check regex/file-scope claim, and a mis-anchored insertion point in SESSION_RUNNER.md's Phase 3F bullet. Two more numeric claims (RECOMMENDED_SKILLS.md brand-token count, git trailer coverage count) were independently re-verified and corrected before ratification. Implementation (P1-P3) is a separate session per "1 and done"; P3 is additionally gated on upstream PR KJ5HST#63 merging first, to avoid a Learnings-table numbering collision. Closes out S18: HANDOFFS.md receipt completed (self_score 9, predecessor_score 9), CHANGELOG.md ledger entry recorded. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rmsharp
added a commit
to rmsharp/methodology
that referenced
this pull request
Jul 30, 2026
Implements Phase 1 of the ratified docs/planning/model-use-provenance-plan.md (S18): an optional **Model:** bullet in starter-kit/CHANGELOG.md's format section (single-tier + capability-tiered two-entry examples), a formalized free-text model-naming convention in starter-kit/HANDOFFS.md's "How to write a receipt" section, and a brand-neutral propagation clause appended to starter-kit/SESSION_RUNNER.md's Phase 3F action-ledger bullet. No new bin/check-handoff key, no hard gate anywhere in the design. A 3-lens adversarial review before commit caught and fixed 3 real defects the mechanical completion-criteria greps had missed: a direct self-contradiction between the CHANGELOG.md and HANDOFFS.md edits on whether to name the model in both places for a single-tier session, a backwards "documented above" cross-reference, and a missing pair of concrete worked examples the plan's own Phase 1 task called for. Scoped to Phase 1 only per the plan's own "(one session each)" phasing. Phase 2 (bin/model-report + Test 23) is the next session; Phase 3 stays blocked on upstream PR KJ5HST#63 merging (confirmed OPEN at commit time). Verified: bin/tests.sh 84/84, bin/check-links OK (82/21), bin/check-handoff OK, all 4 plan-specified completion-criteria greps pass, brand-neutrality regression check empty. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rmsharp
added a commit
to rmsharp/methodology
that referenced
this pull request
Jul 30, 2026
… 23 (S20) Implements Phase 2 of the ratified docs/planning/model-use-provenance-plan.md (S18/S19): a canonical-only bin/model-report tool reading three sources — CHANGELOG.md **Model:** bullets (primary/structured), HANDOFFS.md free-text "model" mentions (secondary/best-effort, regex-fuzzy), and git Co-Authored-By trailers (corroboration-only, never authoritative, hard-disclaimer citing S1's real trailer-vs-prose mismatch) — kept visually/structurally separate, never merged. bin/tests.sh gains Test 23 (8 assertions), written RED-first per Learning KJ5HST#12: a deliberately naive single-merged-list draft was run and confirmed to fail before the real implementation replaced it. A 4-lens adversarial-review workflow before commit caught and fixed 5 real defects: (1) HIGH — the CHANGELOG parser had no code-fence awareness, so it fabricated pseudo-entries out of starter-kit/CHANGELOG.md's own permanent illustrative examples, a live default-path exposure for every future adopter since that file is SEED-copied verbatim; (2) Test 23 originally always passed --no-git, never exercising the trailer source; (3) the module docstring misattributed S1's tier-split sentence to "prose" when it lives in the fenced what_was_done field, contradicting the tool's own live output; (4) a garbled section-reference typo; (5) a completeness sweep found the README repo-structure tree and both starter-kit files' model-naming sections didn't yet mention the new tool. Also fixed a **Model:** bullet continuation-line truncation bug found during the same pass. Scoped to Phase 2 only, per the plan's own "(one session each)" phasing. Phase 3 (new Learnings row) stays blocked on upstream PR KJ5HST#63 merging (confirmed still OPEN at commit time). Verified: bin/tests.sh 92/92, bin/check-links OK (82/21), bin/check-handoff OK, brand-neutrality regression check empty, python3 bin/model-report runs clean against this fork's real 20-session history. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
One row appended to the
starter-kit/SESSION_RUNNER.mdLearnings table (was 1–12; appended, never renumbered). No other change to that file, and no other file touched except the ledger entry.This is the Phase 3C output of the session that shipped v3.6 (PR #62). It is a docs-only addition: no principle, phase, gate, workstream or failure mode changes, and the FM count stays 27.
Why the existing rows don't cover it
Neither covers the forward-looking claim — a state description whose expiry is caused by the writing session's own subsequent commits. That is a distinct failure surface, and it bit this repo directly.
The motivating case is this repo's own v3.6 ship
The predecessor receipt told the ship session to "expect one
CHANGELOG.mdunion conflict".git merge upstream/mainproduced seven.The cause is worth stating precisely, because it generalizes: Layer 8 and the pre-PR review fixes were committed to the PR branch after the point the fork had been ported from it. So five files the prediction never named had diverged — both dashboard twins, the unit suite,
CLAUDE.mdandREADME.md. The prediction was true when written and false when used, and the session that wrote it is the session that invalidated it.Why this is worth a table row rather than a gotcha: a wrong prediction is worse than no prediction. It licenses the successor to resolve quickly rather than carefully — and a merge resolved on the belief that "only the ledger differs" discards code silently, with no failing check to catch it. The countermeasure is mechanical and cheap: diff the branch against the commit the other side was ported from, not against its tip.
The row also asks the writer of a handoff to name the state a prediction depends on, so the reader can tell at a glance whether it still holds.
Cross-reference sweep (Learnings #7/#10, applied to this change)
Changing the size of a numbered set is exactly the trigger those rows describe, so the sweep was run before commit:
table was 1-11/table is 1-12tokens are confined to datedCHANGELOG.mdentries andHANDOFFS.mdreceipts, which this repo leaves verbatim by design.Verification
bin/tests.sh— 84 passed, 0 failedbin/check-links— OK, 82 relative links across 21 distributed markdown filestools/test_methodology_dashboard.py— 197/197 OKNotes
starter-kit/SESSION_RUNNER.mdisbin/_manifest.py-distributed, so adopters receive this viabin/sync.No new
HANDOFFS.mdreceipt is opened here. This is Phase 3C of fork session S17, whose durable receipt already exists on the fork; opening a second receipt for one session would double-count it. TheCHANGELOG.mdentry records the action and points at that receipt.Whether this warrants a version event is yours to call — it is a single distributed-doc row, and v3.6 was tagged the same day.
🤖 Generated with Claude Code