skills(running-in-ci): a declared config is not an observed rate - #953
skills(running-in-ci): a declared config is not an observed rate#953tend-agent wants to merge 2 commits into
Conversation
A nightly run closed an outage tracker with a correct diagnosis, then appended advice built on two unmeasured claims: that a `*/15` cron means ~96 runs/day (the record says 49) and that every run spends a Claude turn (a deterministic pre-check skipped the agent in 41 of those 49). Both came from a `grep -m1 -A3` peek at the workflow file plus arithmetic on the cron string. Adds one bullet to Grounded Analysis -> Specific failure modes: count the record rather than multiplying out a schedule, and treat any truncated read as a sample that cannot support "every"/"all"/"~N per day".
tend-agent
left a comment
There was a problem hiding this comment.
The rule is right and the placement next to the --limit truncation bullet is the correct home — the two are the same defect on different surfaces. I checked the load-bearing claims and they hold: tend-notifications on max-sixty/cargo-affected returns 49 scheduled runs in the 24 h to 2026-08-12T07:00Z against a nominal 96, and this repo's own tend-notifications returns 49 over the same window, so the shortfall isn't repo-local. The pre-check gate the bullet leans on is real and documented in CLAUDE.md under "Agent-driven vs deterministic steps".
One gap worth closing before this lands. The bullet scopes itself to three claim types — how often something runs, what each run costs, and what every invocation does — but the evidence source it names, gh run list, carries no token data. It reports duration and conclusion. The incident that motivated the bullet was itself a cost claim ("the bulk of what consumed the weekly allowance"), and the correction had to reach for run duration as a proxy because nothing pointed at the real instrument. This repo ships one: plugins/tend-ci-runner/scripts/token-report.sh reads token-usage.json out of each run's session-log artifact, which is what the "Token usage" step in each harness action writes. review-runs already calls it. A reader who follows this bullet for a cost claim lands on the wrong tool, which is a small version of the same failure the bullet is written to prevent.
The suggestion also switches generalising to generalizing. Not a taste call — this file already carries generalizable twice and generalizes once in the "Learning from Feedback" section, so the new line is the odd one out within its own file. typos won't catch it.
Review feedback: the bullet's three claim types include cost, but the evidence source it named (gh run list) carries no token data. Point cost claims at token-report.sh, which reads token-usage.json out of each run's session-log artifact. Also align generalising -> generalizing with the rest of the file.
|
Correcting a premise in this PR's own gate assessment, since the merge decision partly rests on it. The body argues the pre-registration fires because this occurrence "is not corrected — it stands as the closing word on a closed issue". That was true when this was filed at 07:22Z. It stopped being true 57 minutes later: So both recorded occurrences of the class were self-corrected in-thread by a sibling run — occurrence 1 within five minutes, this one within ~1.5 h. That doesn't retract the defect: the claim was public, wrong, and unchallenged for 90 minutes on an issue that was already closed, which is exactly where nobody is looking. But the specific condition this PR fired on ("the first occurrence that is not self-corrected") turns out not to hold, so on the shape's own bar the count is 2 of 2–3 with both corrected downstream. Flagging it so the call rests on the current facts rather than the ones in the body — the bullet itself still reads correct to me, and I'm not proposing a change to it. |
The
tend-nightlyrun onmax-sixty/cargo-affectedclosed thetend-outagetracker #84 with a correct root-cause diagnosis (Claude weekly quota exhaustion, traced to the failing run's session log), then appended a "context for whoever sees this recur" paragraph whose two quantitative claims are both wrong, and whose remediation advice follows from them.From comment 5263332937:
Measured live against the same repo:
actions/workflows/tend-notifications.yaml/runsover the 24 h to 2026-08-12T07:00Z returns 49 runs (44 success, 5 failure). The*/15cron is nominal; GitHub's scheduler has been under-delivering on this repo for fifty-plus consecutive hours (observed intervals 11–58 min against a requested 15).tend-notificationsgates the agent behind a deterministicCheck for unread notificationspre-check. In 31571212297 — the run immediately preceding the nightly — the per-step breakdown isCheck for unread notificationssuccess, thenRun actions/checkout@v7skipped andRun max-sixty/tend/claude@0.1.14skipped. By run duration, 41 of the 49 finished in under 30 s, i.e. never booted the agent; of the 8 that did, 5 were the quota failures themselves.Root cause
The session log for the nightly (31571301425) shows where the claim came from: the only read of the workflow file was
That returns the cron line and three lines after it. The pre-check step that gates the agent sits further down the same file and was never in the window — so a four-line sample became a claim about what happens on every invocation, and the cron string became a run count. The run listing needed to falsify the "~96" was already in the session's own context (
gh run list --limit 25ran eleven tool calls earlier).This is the same defect the existing Grounded Analysis → Specific failure modes bullet on
ghlist truncation describes, one surface over: a deliberately truncated read generalised to the population. The existing bullet covers--limiton list commands; it does not covergrep -m1/headon a file, and nothing covers inferring a rate from a declared schedule.The change
One bullet added to the existing Specific failure modes list — no new section, no restructure. It names both halves: config declares intent while the record says what happened, and a truncated read is a sample that cannot support "every"/"all"/"~N per day".
Gate assessment
Evidence: gist (
review-reviewersevidence,max-sixty/cargo-affected2026-08).tend-reviewbody that generalised "all 30 runs are success" from astatus=success&per_page=3query). That entry pre-registered the escalation verbatim: file on the third occurrence of the shape, or on the first occurrence that is not self-corrected in-thread. Occurrence 1 was self-corrected by a sibling run five minutes later. This one is not corrected — it stands as the closing word on a closed issue, framed as durable context for the next reader — which is the condition that fires the pre-registration and reclassifies the shape to High.-m1 -A3over reading the file, and chose to multiply the cron rather than count runs it had already listed. Replayed ten times this does not fail ten times.What is not proposed
No change to the pre-check, the cron, or the outage-tracker flow. The nightly's substantive work this window was correct: it enriched the outage rows, downloaded the failing run's session log, identified the quota message, verified the recovery boundary against the run record, and closed the tracker — clearing a standing escalation in the evidence log ("escalate only if #84 is still open after the next nightly
review-runshas run"). Only the appended speculation is wrong.