Skip to content

store: max_terminal_runs deletes the runs it was documented to delete (#875) - #880

Merged
Masterplanner25 merged 1 commit into
mainfrom
fix/875-terminal-cap
Sep 25, 2026
Merged

Masterplanner25 merged 1 commit into
mainfrom
fix/875-terminal-cap

Conversation

@Masterplanner25

Copy link
Copy Markdown
Owner

Closes #875.

The defect

max_terminal_runs is documented as the hard ceiling on a store's size — "a
host that keeps a store for years and wants a hard ceiling sets this"
. It was
not one. _prune_terminal_runs chose what to delete from the age-bounded
listing, which drops terminal runs older than terminal_max_age_days — and
terminal is exactly what it deletes. So the only records it could see were the
ones it was least meant to remove.

max_terminal_runs=2, six finished runs, four aged past the bound
  -> 6 files on disk, and the four survivors were the four oldest

The exact inverse of the intent: prune the recent history, keep the old forever.

One keyword, because #869 added the unbounded listing this needed.

The safety property

Live runs are untouched at any age. The guarantee is the
status in TERMINAL_RUN_STATUSES filter and does not depend on which listing
feeds it — but that is a claim about code, so the test runs it: every live
status (waiting, running, pending, retry_scheduled) is made the oldest
record in the store and survives a prune.

With a control asserting finished runs are pruned in the same call. Without
it the safety test passes on a build where the cap deletes nothing at all.

Behaviour change

This deletes strictly more, so the changelog flags a "not additive" row at
release. It is opt-in — max_terminal_runs defaults to None, and an unset cap
still deletes nothing (asserted) — so only a host that asked for a ceiling is
affected, and it now gets the one it asked for.

The cost objection was void

I wrote in #875 that prune "would now read every file — the same trade #869 made,
on a hotter path". That was written before #869 landed and is wrong: the listing
already reads every file, because a run's status is only knowable by loading
it. The flag adds no I/O at all. What it adds is materialising records the bound
used to discard after paying for them:

400 aged terminal records: 53 ms without, 65 ms with

I had written "It costs nothing" into the docstring before measuring. It is 53 →
65 ms, and the docstring says so now. Pruning also shrinks the directory, so it
is self-limiting in a way the un-pruned case never was.

Two of my own tests were broken

The neuter run is what showed it — the headline test did not fail against a
deliberately reverted fix:

Re-run after both fixes, the first neuter goes from 1 failure to 4:

BASELINE (fixed tree)                                    53 passed
the cap reads the age-bounded listing (the 5.14 rule)    RED   4 failed
the cap may delete any status, not just terminal         RED   4 failed
the cap keeps the oldest instead of the newest           RED   5 failed
RESTORED                                                 53 passed

Checks — the full four-step sequence

  1. targeted pytest (cap, parked runs, framework, migration, store lifecycle,
    key order, resume bounds): 89 passed
  2. nodus_gate --all: ten phases green
  3. release probes: 130/130
  4. python -m unittest over this file plus its neighbours including
    test_workflow_step_writes
    : 84 tests, OK — the BUG-873: a resume child VM inherits no run budget, so resume_workflow is unbounded work inside a bounded program #873 lesson, since this
    file's prune touches CWD-relative graph state

ruff and mypy src/nodus/ clean.

🤖 Generated with Claude Code

…#875)

It is described as the hard ceiling on a store's size -- "a host that keeps a
store for years and wants a hard ceiling sets this" -- and it was not one.
_prune_terminal_runs chose what to remove from the age-bounded listing, which
drops terminal runs older than terminal_max_age_days, and terminal is exactly
what it deletes. The only records it could see were the ones it was least meant
to remove.

Measured: a ceiling of 2, six finished runs with four aged past the bound, left
SIX files on disk and the four survivors were the four oldest. The exact
inverse of the intent -- prune the recent history, keep the old forever.

One keyword, because #869 added the unbounded listing this needed.

LIVE RUNS ARE UNTOUCHED AT ANY AGE. The guarantee is the
`status in TERMINAL_RUN_STATUSES` filter and does not depend on which listing
feeds it. Asserted by running every live status through a prune as the oldest
records in the store, with a control proving finished runs are pruned in the
same call -- without that control the safety test passes on a build where the
cap deletes nothing.

Deletes strictly more, so the changelog flags a "not additive" row at release.
Opt-in: max_terminal_runs defaults to None and an unset cap still deletes
nothing, so only a host that asked for a ceiling is affected.

The cost objection in the issue was written before #869 and is void: the
listing already reads every file, because a status is only knowable by loading
it. What the flag adds is materialising records the bound used to discard after
paying for them -- 53ms to 65ms at 400 aged records, measured, not the "no cost"
I first wrote in the docstring.

TWO OF MY OWN TESTS WERE BROKEN, and the neuter run is what showed it: the
headline test did not fail against a deliberately reverted fix.

- `save_run` rewrites the record, so ageing inside the helper and then saving
  left every file fresh at prune time. The records are aged after the last save
  now, and the prune is triggered by a separate fresh record.
- The source assertion matched the DOCSTRING, which explains the fix and so
  contains `include_aged_terminal=True`. It reads the body with the docstring
  stripped now. Same unfalsifiable shape as input_fn in #868.

Re-run after both fixes: the first neuter goes from 1 failure to 4, including
the headline test.

Pre-push: targeted pytest (89), `python -m unittest` over this file plus its
neighbours including test_workflow_step_writes (84, the #873 lesson),
nodus_gate --all (ten phases), release probes 130/130, ruff and mypy clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Masterplanner25
Masterplanner25 merged commit dd5cbee into main Sep 25, 2026
6 checks passed
@Masterplanner25
Masterplanner25 deleted the fix/875-terminal-cap branch September 25, 2026 03:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG-875: max_terminal_runs cannot see the runs it is meant to delete, so the documented hard ceiling does not hold

1 participant