Repository navigation
vm: a resume cannot exceed the budget the host set on the caller (#873) - #879
Merged
Merged
Conversation
_resume_target_vm builds a child VM when a resume needs a rebuild (#328), and that child had no bounds at all. Measured with every bound set non-default, FIVE were lost: max_steps, deadline and max_memory_bytes to None, and max_frames from a host's tighter cap back to the 10,000 default -- that last one is the subtlest, because it did not become None, it became MORE than the host asked for. So a guest escaped every bound by parking and resuming: 300 ms budget -> 8 resumes, ~800 ms of work, PASSED 5,000 steps -> ~270,000 instructions, PASSED I had assumed the wall-clock half was already bounded, on the grounds that a deadline is absolute so time spent in the child has passed for the parent too. Running it says otherwise: the deadline is consulted every 100 instructions and a program doing eight resume_workflow calls never executes 100 more, so it is never re-checked. That assumption was in #873's own body; the measurement is why the fix covers both. TWO KINDS OF BOUND, and only one is consumable. A ceiling is a property of the run -- an absolute instant, a stack depth, an RSS figure -- and copies straight across. max_steps is a counter measured against instructions_executed, so the child gets the remainder and its usage is charged back when the resume returns. Without the charge-back every resume is handed a fresh allowance and the split is decorative: six resumes fitted a budget sized for one. max(0, budget - spent) and not `or None`: an exhausted parent must hand down zero, which record_instruction treats as already exceeded. Collapsing it to None would turn "no budget left" into "unbounded", which is the bug. task_step_budget is deliberately not inherited -- it is the scheduler's per-coroutine fairness allowance, not a host bound, and a VM starting fresh work is not inside a slice. resume_workflow and resume_goal now dispatch through one helper. They had the same four lines each, and the charge-back would have gone into one of them -- which is the shape this file's history is made of. charge_run_bounds runs in a `finally`, so a resume that raises is still charged: an uncharged failure is a free retry. BEHAVIOUR-CHANGING, and the changelog says it needs a "not additive" row at release. A resume costs ~99 ms and the `nodus run` default is 200 ms, so a script doing two or more resumes under the default now times out where it previously overran in silence. --time-limit is the fix; one resume still fits. Six per-decision neuters, each red for the right tests. The memory-ceiling one fails ONLY the individually-named test, which is why the list-driven test is not sufficient on its own -- deleting a name from the tuple removes the assertion with it. Added no VM attributes: test_vm_attribute_budget.py still passes, so PyPy's 80-attribute cliff (#702, #488) is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#873) CI's `test` job failed on test_workflow_step_writes.ExitPathTests -- two failures, and nothing to do with step writes. These tests stop steps mid-body on purpose, because that is what a budget doing its job looks like, and a step killed before it returns never reaches _end_step_writes. It leaves an OPEN step-write record in the process-global task_graph._GRAPH_REGISTRY. ExitPathTests scans every graph in that registry and fails if any record is left open, so it inherited mine. Invisible under per-file pytest, which is all I ran. Under CI's single-process `unittest discover` this file sorts before test_workflow_step_writes, so it ran first and poisoned it. That is precisely the trap CLAUDE.md records from the 5.14.0 cut -- including the instruction I skipped: "For any test that leaves runtime state, run `python -m unittest` over it plus its alphabetical neighbours before pushing." Popped by DIFFERENCE against a setUp snapshot rather than by tracking ids: a resume registers its graph on the child VM, so the ids this file creates are not all ones it ever sees. Reproduced and falsified locally with CI's runner and order -- 2 failures without the cleanup, matching CI's `failures=2`, OK with it. The other four test files added this session were checked in the same run and are clean; they do not kill steps, so their records close. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #873.
What was there
_resume_target_vmbuilds a child VM when a resume needs a rebuild (#328). #868taught that child what it may do and who it works for, and deliberately left
the third question — how much may it consume — for its own decision.
Measured with every bound set non-default, five were lost:
max_stepsNonedeadlineNonemax_memory_bytesNonemax_framestask_step_budgetNone(correct)max_framesis the subtlest: it did not becomeNone, it became more than thehost asked for. So a guest escaped every bound by parking and resuming:
The assumption in the issue was wrong
I wrote in #873 that the wall-clock half was the weaker of the two, on the
grounds that a
deadlineis absolute so time spent in the child has alreadypassed for the parent. Running it says otherwise. The deadline is consulted
every 100 instructions, and a program doing eight
resume_workflowcalls neverexecutes 100 more — so it is never re-checked, and nothing self-limits. That
measurement is why this covers both halves rather than just the counter.
Two kinds of bound
A ceiling is a property of the run — an absolute instant, a stack depth, an
RSS figure — and copies straight across.
max_stepsis a counter measuredagainst
instructions_executed, so the child gets the remainder and its usage ischarged back when the resume returns.
Without the charge-back the split is decorative: six resumes fitted a budget
sized for one.
RepeatedResumesAccumulatepins that, with a control asserting oneresume does still fit the same budget — otherwise it would pass on a build
where any resume fails.
max(0, budget - spent), notor None: an exhausted parent must hand downzero, which
record_instructiontreats as already exceeded. Collapsing it toNonewould turn "no budget left" into "unbounded", which is the bug rather thana tidier spelling of it.
task_step_budgetis deliberately excluded — it is the scheduler's per-coroutinefairness allowance (
scheduler.pysets and clears it around a resume), not a hostbound, and a VM starting fresh work is not inside a slice.
Two dispatch sites, not one
builtin_resume_workflowandbuiltin_resume_goaleach had the same four lines.The charge-back would have gone into one of them — which is the shape this file's
history is made of — so both now go through
VM._dispatch_resume.charge_run_boundsruns in afinally: a resume that raises is still charged,because an uncharged failure is a free retry and a guest that can fail cheaply has
no budget at all.
Behaviour change, stated
A resume costs ~99 ms (median of six, measured) and the
nodus rundefault is200 ms. So a script doing two or more resumes under the default now times out
where it previously overran in silence.
nodus run --time-limit N(seconds) is thefix, as it already is for anything needing more than 200 ms. A single resume still
fits. The changelog entry says this needs a "not additive" row in
CLAUDE.mdat release; that table is populated per release, so it is flagged rather than
pre-written.
Tests
tests/test_resume_run_bounds.py, 12 tests. Six per-decision neuters, restoringfrom saved byte copies:
The last one is worth noting: it fails only the individually-named test, not
the list-driven one — because deleting a name from
RUN_BOUND_ATTRIBUTESremovesthe propagation and the comparison together. Same unfalsifiable shape as
input_fnin #868, handled the same way.One test-authoring bug caught on the way: parking with a host function in the
source made every resume fail with
Undefined function: _capand report nosteps, because a resume re-executes the persisted source to rebinddeclarations. The harness parks through
printinstead, which removes the class.Checks
checkpoint resume, rebuild, key order, workflow framework, capability
coverage, VM attribute budget): 119 passed; goal/checkpoint suites
covering
resume_goal: 22 passeduntouched —
test_vm_attribute_budget.pypassesruffclean ·mypy src/nodus/cleannodus_gate --allgreen, all ten phases · release probes 130/130🤖 Generated with Claude Code