Skip to content

fund the verification attempts the defaults advertise - #62

Merged
hyj28 merged 1 commit into
mainfrom
verification-budget-defaults
Sep 26, 2026
Merged

hyj28 merged 1 commit into
mainfrom
verification-budget-defaults

Conversation

@hyj28

@hyj28 hyj28 commented Sep 26, 2026

Copy link
Copy Markdown
Owner

Found by running the agent against a real model, not by the suite. A live run ended verification_budget with records ['unverifiable', 'budget_exhausted'] even though the task itself succeeded — the independent pytest in that workspace exited 0.

Two defaults contradicted each other:

DEFAULT_MAX_VERIFICATION_ATTEMPTS = 3      advertised three attempts
DEFAULT_MAX_VERIFICATION_TOKENS   = 2048   could fund one
isolated verification request      ~1,872  conservative prefill, constant regardless of conversation length

Raising only max_tokens and changing nothing else showed it directly:

budget= 2,048 -> ['unverifiable', 'budget_exhausted']                  verification_budget
budget= 4,096 -> ['unverifiable', 'unverifiable', 'budget_exhausted']  verification_budget
budget= 8,192 -> ['unverifiable', 'unverifiable', 'unverifiable']      verification_limit

A capability that could never fire surfaced as a run-terminating outcome rather than a misconfiguration. RESULTS.md §6 catalogues what that confusion has already cost this project.

The budget is derived now, not picked

An invariant prefill of 1,867 tokens, an 8,000-byte allowance for the variable goal and summary, and the maximum verifier output — 10,123 per attempt, 30,369 for three. The allowance is a stated guarantee, not an input cap: longer goals stay legal, they simply fund fewer attempts. 4,000 of it matches the repository's existing delegated parent-goal bound rather than being invented here.

No validation rejects anything. Underfunded configurations stay legal on purpose — the suite deliberately builds three-attempt configs with budgets of 100, 4 and 8 to exercise the exhaustion path. An earlier instruction of mine would have added a rejecting __post_init__; Codex stopped and said it would break those tests rather than editing them, which was the right call and killed that mechanism.

Both wrong fixes were caught by running, not by a green suite

Round one sized the floor from min_output_tokens (32) when an attempt may spend max_output_tokens (256). It shipped green because its test reported exactly tokens=1_904 per attempt — the floor constant itself — so three of them fit a budget derived from that same constant and the assertion could never disagree with the arithmetic it was built from.

Round two baked in 1,872, which is really 1867 + len(goal) + len(summary). That holds only for the five-byte fixture the test used. Independent review measured the relationship and found six bytes of goal text reproduced the original failure, with summary being model-controlled and uncapped.

The behavioural test now charges each attempt the measured prefill of the request the run actually built plus the configured maximum output, and is parametrised over a realistic goal length as well as a short one. Assertions that could not fail were removed rather than left reading like guards — including one of the form x >= x, where DEFAULT_MAX_VERIFICATION_TOKENS was asserted to be at least the product it is defined as.

Verification

Gate 4, three goal lengths through the real runner:

len(goal)+len(summary) per-attempt records status
5 (the old fixture) 2,128 3 verification_limit
91 (the reviewer's failing case) 2,214 3 verification_limit
7,800 (the documented allowance) 9,923 3 verification_limit

pytest 1286 passed, 3 skipped (baseline 1281 / 3). ruff check . and ruff format --check . exit 0. Tests: 135 additions, 0 deletions — no existing test modified, skipped or mocked.

Falsifiability verified in a scratch copy with PYTHONPATH forced: restoring the old floor makes the behavioural test go red at the realistic-length parametrisation.

Scope

verification.py, four comment lines in agent.py, and the tests. remote.py, lhtb.py, harbor_agent.py, harbor_native_agent.py and native_lhtb.py are untouched, so the archived 170-trial Harbor run stays replayable. An agent built without self_verification short-circuits before any of these constants are read and sends a byte-identical request.

The Harbor native path does construct a bare SelfVerificationConfig() and therefore inherits the new budget; it cannot overrun a trial because available is still clamped by the run's remaining tokens.

Found by running the agent against a real model. A run ended
`verification_budget` with records ['unverifiable', 'budget_exhausted'] even
though the task itself succeeded, because two defaults contradicted each other:
DEFAULT_MAX_VERIFICATION_ATTEMPTS = 3 advertised three attempts while
DEFAULT_MAX_VERIFICATION_TOKENS = 2048 could fund one. The isolated verification
request costs ~1,872 tokens of conservative prefill, so the second attempt was
already unaffordable.

A capability that could never fire surfaced as a run-terminating outcome rather
than as a misconfiguration. RESULTS.md section 6 catalogues what that confusion
has cost this project before.

The budget is now derived, not picked: an invariant prefill of 1,867 tokens, an
8,000-byte allowance for the variable goal and summary, and the maximum verifier
output. Three attempts at that floor give 30,369. The allowance is a stated
guarantee rather than an input cap -- longer goals stay legal, they simply fund
fewer attempts -- and 4,000 of it matches the repository's existing delegated
parent-goal bound rather than being invented here.

No validation rejects anything. Underfunded configurations remain legal on
purpose: the suite deliberately builds three-attempt configs with budgets of
100, 4 and 8 to exercise the exhaustion path, and Codex stopped rather than
break them when an earlier instruction of mine would have.

Two rounds of the fix were wrong in the same way, and both were caught by
running rather than by the suite. The first sized the floor from
min_output_tokens, so an attempt that emitted its permitted maximum still did
not fit; its test reported exactly the floor constant per attempt, so three of
them fit the budget derived from that same constant and the assertion could
never disagree. The second baked in 1,872, which is really 1,867 + len(goal) +
len(summary) and holds only for the five-byte fixture the test used -- six bytes
of goal text reproduced the original failure. The behavioural test now charges
each attempt the measured prefill of the request the run actually built plus the
configured maximum output, and runs at a realistic goal length as well as a
short one. Assertions that could not fail were removed rather than left reading
like guards.

Verified across three goal lengths through the real runner: 5, 91 and 7,800
bytes all reach three attempts and terminate on the attempt limit.
@hyj28
hyj28 merged commit 82f5b9e into main Sep 26, 2026
2 checks passed
@hyj28
hyj28 deleted the verification-budget-defaults branch September 26, 2026 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant