Skip to content

vm: a resume cannot exceed the budget the host set on the caller (#873) - #879

Merged
Masterplanner25 merged 2 commits into
mainfrom
fix/873-resume-budget
Sep 25, 2026
Merged

Masterplanner25 merged 2 commits into
mainfrom
fix/873-resume-budget

Conversation

@Masterplanner25

Copy link
Copy Markdown
Owner

Closes #873.

What was there

_resume_target_vm builds a child VM when a resume needs a rebuild (#328). #868
taught that child what it may do and who it works for, and deliberately left
the third question — how much may it consume — for its own decision.

Measured with every bound set non-default, five were lost:

Bound Parent Child
max_steps 1,000,000 None
deadline set None
max_memory_bytes 999,999,999 None
max_frames 4,321 10,000
task_step_budget 777 None (correct)

max_frames is the subtlest: it did not become None, it became more than the
host asked for. So a guest escaped every bound by parking and resuming:

300 ms budget  -> 8 resumes, ~800 ms of work, PASSED
5,000 steps    -> ~270,000 instructions,      PASSED

The assumption in the issue was wrong

I wrote in #873 that the wall-clock half was the weaker of the two, on the
grounds that a deadline is absolute so time spent in the child has already
passed for the parent. Running it says otherwise. The deadline is consulted
every 100 instructions, and a program doing eight resume_workflow calls never
executes 100 more — so it is never re-checked, and nothing self-limits. That
measurement is why this covers both halves rather than just the counter.

Two kinds of bound

A ceiling is a property of the run — an absolute instant, a stack depth, an
RSS figure — and copies straight across. max_steps is a counter measured
against instructions_executed, so the child gets the remainder and its usage is
charged back when the resume returns.

Without the charge-back the split is decorative: six resumes fitted a budget
sized for one. RepeatedResumesAccumulate pins that, with a control asserting one
resume does still fit the same budget — otherwise it would pass on a build
where any resume fails.

max(0, budget - spent), not or None: an exhausted parent must hand down
zero, which record_instruction treats as already exceeded. Collapsing it to
None would turn "no budget left" into "unbounded", which is the bug rather than
a tidier spelling of it.

task_step_budget is deliberately excluded — it is the scheduler's per-coroutine
fairness allowance (scheduler.py sets and clears it around a resume), not a host
bound, and a VM starting fresh work is not inside a slice.

Two dispatch sites, not one

builtin_resume_workflow and builtin_resume_goal each had the same four lines.
The charge-back would have gone into one of them — which is the shape this file's
history is made of — so both now go through VM._dispatch_resume.
charge_run_bounds runs in a finally: a resume that raises is still charged,
because an uncharged failure is a free retry and a guest that can fail cheaply has
no budget at all.

Behaviour change, stated

A resume costs ~99 ms (median of six, measured) and the nodus run default is
200 ms. So a script doing two or more resumes under the default now times out
where it previously overran in silence. nodus run --time-limit N (seconds) is the
fix, as it already is for anything needing more than 200 ms. A single resume still
fits. The changelog entry says this needs a "not additive" row in CLAUDE.md
at release; that table is populated per release, so it is flagged rather than
pre-written.

Tests

tests/test_resume_run_bounds.py, 12 tests. Six per-decision neuters, restoring
from saved byte copies:

BASELINE (fixed tree)                            32 passed
no bounds inherited at all (the 5.14 rule)       RED   4 failed
inherit_run_bounds is a no-op                    RED   5 failed
the step budget is a fresh allowance, not a split RED   2 failed
an exhausted parent hands down unlimited         RED   1 failed
no charge-back (per-call split only)             RED   2 failed
memory ceiling dropped from the list             RED   1 failed
RESTORED                                         32 passed

The last one is worth noting: it fails only the individually-named test, not
the list-driven one — because deleting a name from RUN_BOUND_ATTRIBUTES removes
the propagation and the comparison together. Same unfalsifiable shape as
input_fn in #868, handled the same way.

One test-authoring bug caught on the way: parking with a host function in the
source made every resume fail with Undefined function: _cap and report no
steps, because a resume re-executes the persisted source to rebind
declarations. The harness parks through print instead, which removes the class.

Checks

🤖 Generated with Claude Code

Masterplanner25 and others added 2 commits September 24, 2026 19:55
_resume_target_vm builds a child VM when a resume needs a rebuild (#328), and
that child had no bounds at all. Measured with every bound set non-default,
FIVE were lost: max_steps, deadline and max_memory_bytes to None, and
max_frames from a host's tighter cap back to the 10,000 default -- that last
one is the subtlest, because it did not become None, it became MORE than the
host asked for.

So a guest escaped every bound by parking and resuming:

  300 ms budget  -> 8 resumes, ~800 ms of work, PASSED
  5,000 steps    -> ~270,000 instructions,      PASSED

I had assumed the wall-clock half was already bounded, on the grounds that a
deadline is absolute so time spent in the child has passed for the parent too.
Running it says otherwise: the deadline is consulted every 100 instructions and
a program doing eight resume_workflow calls never executes 100 more, so it is
never re-checked. That assumption was in #873's own body; the measurement is
why the fix covers both.

TWO KINDS OF BOUND, and only one is consumable. A ceiling is a property of the
run -- an absolute instant, a stack depth, an RSS figure -- and copies straight
across. max_steps is a counter measured against instructions_executed, so the
child gets the remainder and its usage is charged back when the resume returns.
Without the charge-back every resume is handed a fresh allowance and the split
is decorative: six resumes fitted a budget sized for one.

max(0, budget - spent) and not `or None`: an exhausted parent must hand down
zero, which record_instruction treats as already exceeded. Collapsing it to
None would turn "no budget left" into "unbounded", which is the bug.

task_step_budget is deliberately not inherited -- it is the scheduler's
per-coroutine fairness allowance, not a host bound, and a VM starting fresh work
is not inside a slice.

resume_workflow and resume_goal now dispatch through one helper. They had the
same four lines each, and the charge-back would have gone into one of them --
which is the shape this file's history is made of. charge_run_bounds runs in a
`finally`, so a resume that raises is still charged: an uncharged failure is a
free retry.

BEHAVIOUR-CHANGING, and the changelog says it needs a "not additive" row at
release. A resume costs ~99 ms and the `nodus run` default is 200 ms, so a
script doing two or more resumes under the default now times out where it
previously overran in silence. --time-limit is the fix; one resume still fits.

Six per-decision neuters, each red for the right tests. The memory-ceiling one
fails ONLY the individually-named test, which is why the list-driven test is not
sufficient on its own -- deleting a name from the tuple removes the assertion
with it.

Added no VM attributes: test_vm_attribute_budget.py still passes, so PyPy's
80-attribute cliff (#702, #488) is untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#873)

CI's `test` job failed on test_workflow_step_writes.ExitPathTests -- two
failures, and nothing to do with step writes.

These tests stop steps mid-body on purpose, because that is what a budget doing
its job looks like, and a step killed before it returns never reaches
_end_step_writes. It leaves an OPEN step-write record in the process-global
task_graph._GRAPH_REGISTRY. ExitPathTests scans every graph in that registry and
fails if any record is left open, so it inherited mine.

Invisible under per-file pytest, which is all I ran. Under CI's single-process
`unittest discover` this file sorts before test_workflow_step_writes, so it ran
first and poisoned it. That is precisely the trap CLAUDE.md records from the
5.14.0 cut -- including the instruction I skipped: "For any test that leaves
runtime state, run `python -m unittest` over it plus its alphabetical neighbours
before pushing."

Popped by DIFFERENCE against a setUp snapshot rather than by tracking ids: a
resume registers its graph on the child VM, so the ids this file creates are not
all ones it ever sees.

Reproduced and falsified locally with CI's runner and order -- 2 failures
without the cleanup, matching CI's `failures=2`, OK with it. The other four test
files added this session were checked in the same run and are clean; they do not
kill steps, so their records close.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Masterplanner25
Masterplanner25 merged commit ab08fad into main Sep 25, 2026
6 checks passed
@Masterplanner25
Masterplanner25 deleted the fix/873-resume-budget branch September 25, 2026 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG-873: a resume child VM inherits no run budget, so resume_workflow is unbounded work inside a bounded program

1 participant