Repository navigation
feat(budget): aegis budget status|check|reset, docs/budget.md, latency - #40
Conversation
- status: today's project total and its sessions, rebuilt from the logs (--json); check: exit 0 under / 3 over / 65 unusable / 66 none, with an optional --agent --session check; reset --day: a marker signed with the policy key acknowledging an incomplete day, honoured by the hook - evaluate refactored to share the project-only check - price table parse cached by content hash (~15 ms per hook call saved); scripts/budget_latency.py: budget adds ~9 ms per hook call - docs/budget.md, signed budget.example.yaml from aegis init, README/docs index links; tests keep the real ~/.cache and ~/.local/state untouched
moneytool
left a comment
There was a problem hiding this comment.
Needs fixes before merging. All 1,989 existing tests pass locally and CI is green, but four additional reproductions fail: a session-specific CLI check misses another session's logged project spend; a signed reset suppresses new project verification failures occurring after it; and valid-JSON damage to the price cache either changes enforcement or fails rather than rebuilding. Concrete cases and fixes are inline.
| digest = hashlib.sha256(text.encode()).hexdigest()[:16] | ||
| cache = _table_cache_dir() / f"prices-{digest}.json" | ||
| try: | ||
| table = json.loads(cache.read_text()) |
There was a problem hiding this comment.
[P1] Keep the bundled table authoritative over the mutable price cache
The digest in the filename hashes prices.yaml, but nothing establishes that the cached JSON contains that table. Any valid version-1 JSON with altered rates becomes an explicit price source. I reproduced a signed $1 session budget with a $1.20 transcript: the hook denies initially, then returns exit 0 after changing only the cached rates to zero, without editing the policy or usage log. This creates an unsigned pricing override despite the design requiring signed price changes. The same trust problem affects accidental cache corruption: a parseable cache containing {"version":999} raises permanently instead of falling back to YAML, contrary to the stated damaged-cache behavior. Verify cached content against a trusted representation of the bundled table, or ship pre-generated JSON as a package resource to obtain the latency improvement without a mutable authoritative table. Invalid cache content must rebuild from the resource. Add coverage for both altered-but-valid rates and invalid schema/version, not only invalid JSON.
There was a problem hiding this comment.
Fixed in 0518c27 by removing the cache entirely.
- Generated table: the table now ships as
aegis_core/budget/prices.json, generated fromprices.yamlbyscripts/sync_prices.py.prices.yamlstays the file people edit. - Read only from the package:
load_builtin_table()reads only the installed package resource, never anything writable under~/.cache. A signedpricing:override is the only way to change a price. - Latency: JSON loads in well under a millisecond, so the saving holds. Re-measured, the budget adds 9.5 ms per hook call (
results/budget-latency.json). - Tests:
test_prices_json_is_prices_yaml: the JSON equals the YAML, andsync_prices.py --checkpasses.test_no_writable_file_can_change_a_price: plants zero-rate tables under the exact cache name the previous commit used, plus a{"version": 999}file, and asserts the prices are unchanged. That test fails on the previous commit; the zero rates take effect.
| if loc is None: | ||
| raise DataError(f"cannot find the log of {args.agent} session {args.session}") | ||
| measured = {a: lg for a, lg in logs.items() if policy.measures(a)} | ||
| snap = refresh(store, measured, loc, policy.zone, now=now, fallback_cwd=str(project)) |
There was a problem hiding this comment.
[P2] Rebuild the project total for session-specific CLI checks too
refresh only updates the selected session once a project/day cache exists; it does not discover or reread other sessions. Therefore check --agent/--session can report under budget from a stale project total, despite status and unscoped check rebuilding from the logs. Reproduction: record session a's $0.50 via its hook under a $2 project cap, then create session b's $2.00 transcript without firing a hook. check --agent claude --session a exits 0, although the logged project total is $2.50 and unscoped check exits 3. Rebuild the applicable project/day from all measured agents before evaluating the refreshed selected session (using the session's actual project root), and add this regression with an already-existing project cache and an unhooked second session.
There was a problem hiding this comment.
Fixed in 0518c27. check --agent/--session refreshes the session, then rebuilds that session's actual project day from every measured agent's logs, as status and an unscoped check do. It evaluates the session against the rebuilt total. Your regression is test_a_session_check_sees_other_sessions_unhooked_spend: session a is hooked at $0.50, so the project cache exists, then session b logs $2.00 with no hook, and check --agent claude --session a now exits 3. It exits 0 on the previous commit. The docs say check rebuilds.
| """A project day whose history is incomplete, or whose other | ||
| sessions are only a lower bound, is unverifiable -- unless a signed | ||
| ``aegis budget reset --day`` acknowledged that day.""" | ||
| if acknowledged: |
There was a problem hiding this comment.
[P2] Limit a reset to the history acknowledged at reset time
This unconditional return suppresses every project verification failure for the rest of the day, including failures in sessions created after the signed acknowledgement. The marker records at but that timestamp/baseline is never used. I reproduced strict mode with a signed reset, then a new malformed session: that session's own hook correctly denies and stores its unverified project entry, but another healthy session's hook now allows, even though this new project's under-count was never acknowledged. The docs say reset acknowledges history as it stands, not future unknown usage. Bind the signed marker to the acknowledged verification baseline (session/reason state or an equivalent checkpoint), so new missing/unreadable usage again triggers strict denial; keep already-acknowledged historical gaps excused. Add a regression introducing a new malformed or missing-log session after reset and checking a different healthy session.
There was a problem hiding this comment.
Fixed in 0518c27. The marker now signs a baseline: the day's verification state at reset time, meaning completeness, inventory status, missing-session count, and each lower-bound session with its reasons. reset rebuilds the day first, so the baseline is current, and the baseline is inside the MAC.
- What is excused:
evaluateexcuses an incomplete history only if it is still the acknowledged one (same completeness and inventory status, no more missing sessions). It excuses another session's lower bound only if that session and those reasons were in the baseline. - Tests:
test_a_reset_does_not_excuse_problems_that_appear_after_it: your case, a new malformed session after the reset, after which a healthy session's hook denies again. It allows on the previous commit.test_a_reset_still_excuses_what_it_acknowledged: a lower bound that existed at reset time stays excused.- A marker with a widened baseline fails verification.
- Docs:
docs/budget.mddescribes this.
… day; resets bind to a baseline - the price table ships as prices.json generated from prices.yaml (scripts/sync_prices.py, test keeps them equal); no writable cache can act as an unsigned price override - check --agent/--session rebuilds the project day from every measured agent before evaluating, as status and an unscoped check do - reset markers sign the acknowledged verification state; only that is excused, later unverifiable usage counts again Review of #40.
moneytool
left a comment
There was a problem hiding this comment.
The four original reproductions now pass: packaged prices are authoritative, session-specific checks discover other project usage, and newly unverified sessions invalidate the reset. One extension of the reset case still reproduces a strict-mode bypass: additional unreadable records in an already acknowledged session are excused because the baseline stores only reason labels. Please address the inline finding before merging.
| and pd.missing_sessions <= self.missing_sessions) | ||
|
|
||
| def covers(self, key: str, reasons: list[str]) -> bool: | ||
| return key in self.unverified and set(reasons) <= set(self.unverified[key]) |
There was a problem hiding this comment.
[P2] Track the amount of unverified history, not only its reason label
The new baseline identifies sessions and reason strings, but unreadable-records remains the same label whether that session has one bad record or many newly appended ones. I reproduced strict mode with session broken containing one malformed line, signed reset, then a second malformed line appended to that same log. Its own hook denies and updates the project entry, but healthy session a's hook still returns exit 0 because covers() considers the unchanged reason label acknowledged. Thus the reset still excuses new unverifiable usage after it, as long as it occurs in an already acknowledged session. Include a per-session verification baseline such as unreadable-record counts plus source/cursor or log-generation state (and corresponding unknown-time usage state), and compare it so worsening/new gaps invalidate coverage while unchanged historical gaps remain excused. Add a regression that appends a second malformed record after reset and checks a different healthy session.
There was a problem hiding this comment.
Fixed in fd58322. The baseline now stores quantities, not reason labels.
- Project entries: each entry carries its session's
verification:log_missing,unreadable_records(the record's running count of unreadable records) andunknown_time_usage(usage in the no-recorded-time bucket). - Baseline:
Acknowledgement.ofsigns those quantities per lower-bound session. - Coverage:
coversholds only while none of them has grown, meaning the log is no more missing and there are no more unreadable records and no more unknown-time usage. A session without recorded quantities is never covered. - Regression:
test_more_unreadable_records_in_an_acknowledged_session_count_againis your case. One malformed line, a signed reset, healthy session a allowed. Then a second malformed line is appended to the same log; its own hook records the change, and a's hook now denies. - Also tested:
test_the_acknowledged_quantities_are_signedchecks the stored quantities, and that widening them in the marker breaks verification. - Old code: both tests fail on the previous commit.
…ounts again Project entries carry each session's verification quantities (log missing, unreadable records, usage with no recorded time); an acknowledgement covers a session only while none of them has grown. Review of #40.
Implementation step 3 of the v1.0 budget cap design (§7, §10). Step 4 is a live test in each agent, then the 1.0.0 release.
aegis budgetstatus [--project DIR] [--agent A] [--json]: rebuilds today's project total from the logs first, so it also counts sessions whose hooks never fired. It then shows the project total againstproject_day, and each contributing session with today's usage, its session total against the session limit, the level, any overshoot, and whether it is only a lower bound. It also shows estimated-price models and the policy's warnings.check [--project DIR] [--agent A --session ID]: exits 0 under budget (a warning included), 3 over, 65 for an unusablebudget.yaml, and 66 when there is none. With--agent/--sessionit also refreshes and checks that session, through the sameevaluatethe hook uses.reset --day [YYYY-MM-DD]: the strict-recovery path from §4.1. It writes a marker signed with the policy key that acknowledges one day's incomplete project history, soon_unknown_log: denystops denying for that reason.--insecure, and it needs a project.--insecure, nothing is verified, consistent with the other policy files.evaluatenow shares one builder between the per-session decision and a newevaluate_project, with no change in behaviour; the existing tests are unchanged.Latency
prices.yamlwith PyYAML's pure-Python loader, about 15 ms on every hook call. The parsed table is now cached as JSON under the cache directory, keyed by a hash of the YAML text, so an edited table is re-read and a damaged cache only costs a re-parse.scripts/budget_latency.pytimes a realaegis hook claudeprocess on a non-shell tool, in two identicalaegis initprojects (with and without a signed budget), interleaved, after a warm-up, with bytecode caching on. Result inresults/budget-latency.json(whitelisted in.gitignorelike the other published results): 74.1 ms median with a budget vs 65.3 ms without, so the budget adds 8.8 ms, against the 20 ms target. p95 is 77.6 vs 68.0 ms.PYTHONDONTWRITEBYTECODE, which makes every measured process recompile aegis.Docs
docs/budget.md: the exact guarantee and its overshoot, setup, thebudget.yamlreference, behaviour near and at the limit per agent, where the numbers come from, unverifiable usage and the signed reset, the commands, and the cost. It is linked from the README and the docs site.budget.example.yaml:aegis initnow writes a signedbudget.example.yaml, which does nothing until it is copied tobudget.yaml.Tests
New: 9 CLI tests:
checkexit code;Plus a price-cache test.
Breakage checks: with marker verification removed, or the hook ignoring the reset, the corresponding tests fail.
Test isolation:
tests/conftest.pynow pointsXDG_CACHE_HOMEandXDG_STATE_HOMEat a temporary directory for every test. A module-scoped fixture had been writing the price cache into the real~/.cache.Totals: the full suite passes locally (1,989) and ruff is clean.