Repository navigation
fund the verification attempts the defaults advertise - #62
Merged
Merged
Conversation
Found by running the agent against a real model. A run ended `verification_budget` with records ['unverifiable', 'budget_exhausted'] even though the task itself succeeded, because two defaults contradicted each other: DEFAULT_MAX_VERIFICATION_ATTEMPTS = 3 advertised three attempts while DEFAULT_MAX_VERIFICATION_TOKENS = 2048 could fund one. The isolated verification request costs ~1,872 tokens of conservative prefill, so the second attempt was already unaffordable. A capability that could never fire surfaced as a run-terminating outcome rather than as a misconfiguration. RESULTS.md section 6 catalogues what that confusion has cost this project before. The budget is now derived, not picked: an invariant prefill of 1,867 tokens, an 8,000-byte allowance for the variable goal and summary, and the maximum verifier output. Three attempts at that floor give 30,369. The allowance is a stated guarantee rather than an input cap -- longer goals stay legal, they simply fund fewer attempts -- and 4,000 of it matches the repository's existing delegated parent-goal bound rather than being invented here. No validation rejects anything. Underfunded configurations remain legal on purpose: the suite deliberately builds three-attempt configs with budgets of 100, 4 and 8 to exercise the exhaustion path, and Codex stopped rather than break them when an earlier instruction of mine would have. Two rounds of the fix were wrong in the same way, and both were caught by running rather than by the suite. The first sized the floor from min_output_tokens, so an attempt that emitted its permitted maximum still did not fit; its test reported exactly the floor constant per attempt, so three of them fit the budget derived from that same constant and the assertion could never disagree. The second baked in 1,872, which is really 1,867 + len(goal) + len(summary) and holds only for the five-byte fixture the test used -- six bytes of goal text reproduced the original failure. The behavioural test now charges each attempt the measured prefill of the request the run actually built plus the configured maximum output, and runs at a realistic goal length as well as a short one. Assertions that could not fail were removed rather than left reading like guards. Verified across three goal lengths through the real runner: 5, 91 and 7,800 bytes all reach three attempts and terminate on the attempt limit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by running the agent against a real model, not by the suite. A live run ended
verification_budgetwith records['unverifiable', 'budget_exhausted']even though the task itself succeeded — the independent pytest in that workspace exited 0.Two defaults contradicted each other:
Raising only
max_tokensand changing nothing else showed it directly:A capability that could never fire surfaced as a run-terminating outcome rather than a misconfiguration. RESULTS.md §6 catalogues what that confusion has already cost this project.
The budget is derived now, not picked
An invariant prefill of 1,867 tokens, an 8,000-byte allowance for the variable goal and summary, and the maximum verifier output — 10,123 per attempt, 30,369 for three. The allowance is a stated guarantee, not an input cap: longer goals stay legal, they simply fund fewer attempts. 4,000 of it matches the repository's existing delegated parent-goal bound rather than being invented here.
No validation rejects anything. Underfunded configurations stay legal on purpose — the suite deliberately builds three-attempt configs with budgets of 100, 4 and 8 to exercise the exhaustion path. An earlier instruction of mine would have added a rejecting
__post_init__; Codex stopped and said it would break those tests rather than editing them, which was the right call and killed that mechanism.Both wrong fixes were caught by running, not by a green suite
Round one sized the floor from
min_output_tokens(32) when an attempt may spendmax_output_tokens(256). It shipped green because its test reported exactlytokens=1_904per attempt — the floor constant itself — so three of them fit a budget derived from that same constant and the assertion could never disagree with the arithmetic it was built from.Round two baked in 1,872, which is really
1867 + len(goal) + len(summary). That holds only for the five-byte fixture the test used. Independent review measured the relationship and found six bytes of goal text reproduced the original failure, withsummarybeing model-controlled and uncapped.The behavioural test now charges each attempt the measured prefill of the request the run actually built plus the configured maximum output, and is parametrised over a realistic goal length as well as a short one. Assertions that could not fail were removed rather than left reading like guards — including one of the form
x >= x, whereDEFAULT_MAX_VERIFICATION_TOKENSwas asserted to be at least the product it is defined as.Verification
Gate 4, three goal lengths through the real runner:
len(goal)+len(summary)verification_limitverification_limitverification_limitpytest1286 passed, 3 skipped (baseline 1281 / 3).ruff check .andruff format --check .exit 0. Tests: 135 additions, 0 deletions — no existing test modified, skipped or mocked.Falsifiability verified in a scratch copy with
PYTHONPATHforced: restoring the old floor makes the behavioural test go red at the realistic-length parametrisation.Scope
verification.py, four comment lines inagent.py, and the tests.remote.py,lhtb.py,harbor_agent.py,harbor_native_agent.pyandnative_lhtb.pyare untouched, so the archived 170-trial Harbor run stays replayable. An agent built withoutself_verificationshort-circuits before any of these constants are read and sends a byte-identical request.The Harbor native path does construct a bare
SelfVerificationConfig()and therefore inherits the new budget; it cannot overrun a trial becauseavailableis still clamped by the run's remaining tokens.