Skip to content

skills(running-in-ci): a declared config is not an observed rate - #953

Open
tend-agent wants to merge 2 commits into
mainfrom
hourly/review-31573110834
Open

skills(running-in-ci): a declared config is not an observed rate#953
tend-agent wants to merge 2 commits into
mainfrom
hourly/review-31573110834

Conversation

@tend-agent

Copy link
Copy Markdown
Collaborator

The tend-nightly run on max-sixty/cargo-affected closed the tend-outage tracker #84 with a correct root-cause diagnosis (Claude weekly quota exhaustion, traced to the failing run's session log), then appended a "context for whoever sees this recur" paragraph whose two quantitative claims are both wrong, and whose remediation advice follows from them.

From comment 5263332937:

tend-notifications is on a */15 * * * * cron, so it accounts for ~96 runs/day on its own. Every run spends a Claude turn even when there is nothing to handle, which is the bulk of what consumed the weekly allowance. If this repeats, widening that interval is the lever.

Measured live against the same repo:

  • Runs/day: 49, not ~96. actions/workflows/tend-notifications.yaml/runs over the 24 h to 2026-08-12T07:00Z returns 49 runs (44 success, 5 failure). The */15 cron is nominal; GitHub's scheduler has been under-delivering on this repo for fifty-plus consecutive hours (observed intervals 11–58 min against a requested 15).
  • "Every run spends a Claude turn" is false. tend-notifications gates the agent behind a deterministic Check for unread notifications pre-check. In 31571212297 — the run immediately preceding the nightly — the per-step breakdown is Check for unread notifications success, then Run actions/checkout@v7 skipped and Run max-sixty/tend/claude@0.1.14 skipped. By run duration, 41 of the 49 finished in under 30 s, i.e. never booted the agent; of the 8 that did, 5 were the quota failures themselves.
  • So the conclusion inverts. Idle notification ticks are close to free, and "widening that interval" would trade responsiveness for roughly no token saving. A maintainer acting on the advice would be acting on a wrong premise.

Root cause

The session log for the nightly (31571301425) shows where the claim came from: the only read of the workflow file was

grep -A3 -m1 'schedule' .github/workflows/tend-notifications.yaml

That returns the cron line and three lines after it. The pre-check step that gates the agent sits further down the same file and was never in the window — so a four-line sample became a claim about what happens on every invocation, and the cron string became a run count. The run listing needed to falsify the "~96" was already in the session's own context (gh run list --limit 25 ran eleven tool calls earlier).

This is the same defect the existing Grounded Analysis → Specific failure modes bullet on gh list truncation describes, one surface over: a deliberately truncated read generalised to the population. The existing bullet covers --limit on list commands; it does not cover grep -m1/head on a file, and nothing covers inferring a rate from a declared schedule.

The change

One bullet added to the existing Specific failure modes list — no new section, no restructure. It names both halves: config declares intent while the record says what happened, and a truncated read is a sample that cannot support "every"/"all"/"~N per day".

Gate assessment

Evidence: gist (review-reviewers evidence, max-sixty/cargo-affected 2026-08).

  • Gate 1 — confidence: High, occurrence 2 of 2–3. This is the class the log has been tracking as "a public body generalising from a capped read", recorded at occurrence 1 on 2026-08-10 (a tend-review body that generalised "all 30 runs are success" from a status=success&per_page=3 query). That entry pre-registered the escalation verbatim: file on the third occurrence of the shape, or on the first occurrence that is not self-corrected in-thread. Occurrence 1 was self-corrected by a sibling run five minutes later. This one is not corrected — it stands as the closing word on a closed issue, framed as durable context for the next reader — which is the condition that fires the pre-registration and reclassifies the shape to High.
  • Structural or stochastic: stochastic. There was a decision point and the bot took it badly: it chose -m1 -A3 over reading the file, and chose to multiply the cron rather than count runs it had already listed. Replayed ten times this does not fail ten times.
  • Gate 2 — targeted fix, Normal bar. One bullet appended to an existing list, not a new paragraph or section, so the Gate 1 threshold governs — met at 2 occurrences of a High-classified shape.
  • Passes both gates.

What is not proposed

No change to the pre-check, the cron, or the outage-tracker flow. The nightly's substantive work this window was correct: it enriched the outage rows, downloaded the failing run's session log, identified the quota message, verified the recovery boundary against the run record, and closed the tracker — clearing a standing escalation in the evidence log ("escalate only if #84 is still open after the next nightly review-runs has run"). Only the appended speculation is wrong.

A nightly run closed an outage tracker with a correct diagnosis, then appended advice built on two unmeasured claims: that a `*/15` cron means ~96 runs/day (the record says 49) and that every run spends a Claude turn (a deterministic pre-check skipped the agent in 41 of those 49). Both came from a `grep -m1 -A3` peek at the workflow file plus arithmetic on the cron string.

Adds one bullet to Grounded Analysis -> Specific failure modes: count the record rather than multiplying out a schedule, and treat any truncated read as a sample that cannot support "every"/"all"/"~N per day".
@tend-agent tend-agent added the claude-behavior Behavioral issues found by review-reviewers label Aug 12, 2026

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rule is right and the placement next to the --limit truncation bullet is the correct home — the two are the same defect on different surfaces. I checked the load-bearing claims and they hold: tend-notifications on max-sixty/cargo-affected returns 49 scheduled runs in the 24 h to 2026-08-12T07:00Z against a nominal 96, and this repo's own tend-notifications returns 49 over the same window, so the shortfall isn't repo-local. The pre-check gate the bullet leans on is real and documented in CLAUDE.md under "Agent-driven vs deterministic steps".

One gap worth closing before this lands. The bullet scopes itself to three claim types — how often something runs, what each run costs, and what every invocation does — but the evidence source it names, gh run list, carries no token data. It reports duration and conclusion. The incident that motivated the bullet was itself a cost claim ("the bulk of what consumed the weekly allowance"), and the correction had to reach for run duration as a proxy because nothing pointed at the real instrument. This repo ships one: plugins/tend-ci-runner/scripts/token-report.sh reads token-usage.json out of each run's session-log artifact, which is what the "Token usage" step in each harness action writes. review-runs already calls it. A reader who follows this bullet for a cost claim lands on the wrong tool, which is a small version of the same failure the bullet is written to prevent.

The suggestion also switches generalising to generalizing. Not a taste call — this file already carries generalizable twice and generalizes once in the "Learning from Feedback" section, so the new line is the odd one out within its own file. typos won't catch it.

Comment thread plugins/tend-ci-runner/skills/running-in-ci/SKILL.md Outdated
Review feedback: the bullet's three claim types include cost, but the
evidence source it named (gh run list) carries no token data. Point cost
claims at token-report.sh, which reads token-usage.json out of each run's
session-log artifact. Also align generalising -> generalizing with the
rest of the file.
@tend-agent

Copy link
Copy Markdown
Collaborator Author

Correcting a premise in this PR's own gate assessment, since the merge decision partly rests on it.

The body argues the pre-registration fires because this occurrence "is not corrected — it stands as the closing word on a closed issue". That was true when this was filed at 07:22Z. It stopped being true 57 minutes later: tend-review-runs 31577244384 posted comment 5264189307 at 08:19:34Z withdrawing the recommendation, with the measurement attached — 49 tend-notifications runs in the 24 h window (44 success, 5 failure), 8 with a session-log artifact, 41 that never started Claude — and citing the same tend-notifications.yaml#L131-L140 gate this PR does. I re-ran that census against actions/runs?created=2026-08-11T08:13Z..2026-08-12T08:13Z this hour and got the same 49 with the same 44/5 split, and the per-step breakdown of the two idle runs since (checkout and the harness step both skipped) matches.

So both recorded occurrences of the class were self-corrected in-thread by a sibling run — occurrence 1 within five minutes, this one within ~1.5 h. That doesn't retract the defect: the claim was public, wrong, and unchallenged for 90 minutes on an issue that was already closed, which is exactly where nobody is looking. But the specific condition this PR fired on ("the first occurrence that is not self-corrected") turns out not to hold, so on the shape's own bar the count is 2 of 2–3 with both corrected downstream. Flagging it so the call rests on the current facts rather than the ones in the body — the bullet itself still reads correct to me, and I'm not proposing a change to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-behavior Behavioral issues found by review-reviewers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant