Skip to content

[Fix] Agent tool loops silently exhaust inference credits - #3658

Merged
brunobergher merged 2 commits into
developfrom
fix/bound-agent-inference-loops-3cvs4f4n04ft5
Nov 3, 2026
Merged

brunobergher merged 2 commits into
developfrom
fix/bound-agent-inference-loops-3cvs4f4n04ft5

Conversation

@roomote-roomote

@roomote-roomote roomote-roomote Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

​Created by Roomote. Follow up by mentioning @roomote-roomote, in the web UI, or in Slack.

Related issue

Fixes #3585

Why this PR exists

  • A maintainer explicitly invited this PR in the linked issue or discussion
  • I am a maintainer / this is internal Roomote work

Active review/judge children could repeat the same empty search indefinitely while silence watchdogs treated every iteration as progress. Post-response usage accounting could report the bill but could not stop the next provider request. This adds execution bounds and a durable admission boundary rather than relying on another model response to notice the loop.

What changed

Bound repeated tool activity across subagent types. The worker and control-plane OpenCode prompt runner share a pass guard. Five consecutive unique completed calls with the same tool name, recursively key-sorted arguments, and identical result/error stop execution; empty results count. Duplicate/out-of-order updates for the same invocation ID do not increase counts. Changed arguments or results reset the streak. The exact read-only task-update polling action gets 20 identical completions; shell/search tools do not receive a polling exemption. Safety diagnostics retain hashes/counts, not raw arguments/results.

Make pass limits independent of progress and siblings. Live child passes have defaults of 100 unique assistant-message steps, 200 terminal tool invocations, and 20 minutes. Pending task spawns are time-bounded before their child ID arrives. Activity and starting another child do not extend the older pass's deadline. The main task also has repetition/tool-count protection; control-plane prompts cannot disable the 20-minute maximum with timeoutMs: null. Late duplicate terminal events do not re-arm settled passes.

Reserve task inference allowance before dispatch. Four additive task columns persist the admitted request count, request limit (default 1,000), and stop latch/reason. PostgreSQL row locking serializes concurrent admissions across providers, children, retries and resumed runs. Every allowed gateway POST reserves before upstream fetch, including Jevgrep; GET metadata reads do not consume allowance. Failed attempts are not refunded, and database admission failures do not dispatch upstream. Missing usage/price data cannot bypass the counter. Task-associated control-plane OpenCode helpers use dedicated gateway-backed leases so their internal steps/retries share the same counter; task-associated typed decisions reserve before direct dispatch too.

Gateway-bound helper environments strip direct subscription OAuth at the final environment boundary. Managed startup overrides the caller's data home with a private, empty auth store and cleans it up on process close or startup failure. Shared auth files used by ordinary sessions remain untouched. This prevents built-in ChatGPT/Copilot OAuth loaders from replacing the SDK transport and bypassing the gateway URL or stop latch.

Stop visibly without inference. A breach stops the entire task, clears local retry/fallback queues, attempts bounded cancellation of root/recursive children, releases the worker's parent wait with a terminal event, and refuses late local continuation. The durable stop transaction writes task and owning-session history notices with the reason, available pass counters/elapsed time, and admitted-request count. Dedicated notice events survive task history reload without colliding with normal assistant events. Parent settlement/error relays do not wake inference to explain the stop or offer a replacement pass. New runs, restarts, resumes and provider fallback cannot reset the task latch/counter.

Public task/cost documentation, an operator/developer guarantee-and-limits note (packages/db/agent-safety.md), the generated migration, and a patch changeset are included.

Impact and exact limits

  • A safety stop retains partial work and does not label an interrupted review clean. It intentionally stops the whole task, including when a looping judge is the final step of otherwise completed implementation.
  • This is a hard managed-request-count ceiling, not a hard dollar cap. Prices vary; usage accounting remains post-response. Already-admitted/in-flight requests can finish and be billed. Provider-side billing limits are still needed for a financial cap.
  • Repetition detection observes OpenCode events, so already-started work can race cancellation. Short varying-call cycles are bounded by pass/request ceilings rather than an additional cycle detector.
  • Pass clocks/fingerprints are in-memory and restart with the runtime; the lifetime request counter and stop latch survive restart/resume. Existing tasks start counting at migration time rather than retroactively reconstructing historical admissions.
  • If the API/database is unavailable when a local guard fires, local execution still stops, but durable notice/latch writes cannot be guaranteed during that outage. Gateway admission independently fails closed when its database is unavailable.
  • Independent session orchestration without task context and arbitrary scripts/integration services are outside a task's managed inference ceiling. This is not a network-egress firewall.
  • There is no allowance-extension/reset UI in this change. Continuation is an explicitly started new task after inspecting partial results; ordinary resume cannot clear a safety stop. Already-open history views may need refresh; external chat notification is best effort.

How it was tested

Initial targeted validation: 535 checks passed across 14 test files:

Surface Passed Coverage
Worker harness and stall/subagent watchdogs 152 Original 1,000-empty-grep incident across judge/explore/general/review, changed results, duplicate events, independent deadlines, late children, failed cancellation and parent release
Control-plane helpers and typed decisions 207 Recursive child loop cancellation, no retry after safety stop, finite null timeout even when the prompt does not settle, gateway helper configuration, direct-decision admission
Inference gateway 83 Admission denied/unavailable means no upstream fetch; provider routes and existing gateway behavior
Real PostgreSQL admission 2 Concurrent children/resumed runs admit exactly the configured count without cost data; exactly one durable task/parent notice and rejection after a later run
SDK notification paths 51 Safety stops do not wake parent inference or offer automatic retry; existing delivery paths
Shared guard and fallback classifier 18 Canonical arguments, unique completion/step counts, empty results, bounded polling, changed results, safety errors never choose fallback
Task transcript service 22 Dedicated durable safety notices render on history reload without a live model event

pnpm lint:fast, pnpm check-types:fast (27 packages), pnpm knip, the real pre-push hooks, mise exec -- pnpm --filter @roomote/docs check, and pnpm exec changeset status pass. Generated migration applied successfully to the local development database; the test schema was applied to the real test database.

Review follow-up validation (04e3ac6c): all 2,670 SDK tests pass, including the three formerly failing judgment-router cases. The fixtures now model the persisted task lookup, assert task-ID forwarding, and verify missing bindings never dispatch judgments; the production binding guard was retained. All 2,646 cloud-agents tests pass serially (three opt-in tests skipped). The 259 focused helper/decision checks also pass, including two explicitly enabled real OpenCode transport checks for ChatGPT and Copilot.

The transport checks use synthetic inline/persisted OAuth and a local denying gateway, without paid provider inference. They assert the real OpenCode request uses the task token and receives the non-retryable safety denial. Both fail with the protection temporarily disabled and pass with it restored. Environment/store tests cover ambient and explicit OAuth stripping, preservation of ordinary-session auth, and private-store cleanup on success/startup failure. Browser proof is not applicable to this backend-only follow-up; screenshots cannot establish the provider transport.

The local parallel cloud suite encountered a cross-file database fixture race: enqueue-task.test.ts temporarily excludes all compute providers while fast-agent-pinned-launch-automation-thread.test.ts resolves a default provider. The isolated failing file and the full serial suite pass; both involved fixtures and compute-selection production code are unchanged by this PR. No unrelated compute behavior was changed.

The initial optional repository judgment hook reported incomplete coverage at its three-second/file-evidence budgets; the follow-up commit's hook passes (11 judgments). Broad lint/types/knip and real push hooks pass. Full-repository test completion is left to CI; no paid-provider soak or hard monetary enforcement is claimed.

Checklist

  • The PR title follows the repo convention: [Fix], [Feat], [Improve], [Refactor], [Docs], or [Chore] followed by a user-facing description
  • This PR is small and scoped to one change
  • pnpm lint and pnpm check-types pass locally
  • I added tests or included a clear manual validation note above
  • I removed secrets, tokens, private keys, and customer data from code, logs, and screenshots
  • If this change should appear in the changelog, I ran pnpm changeset

The fix is cohesive but cross-layer, so it is not a small diff; most added lines are the generated Drizzle snapshot. Static validation used the repository-recommended :fast equivalents rather than the differently named full commands above.

@roomote-community

roomote-community Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Both prior findings are addressed; no new code issues found in the latest update. See session

Reviewed OAuth environment stripping, isolated auth-store startup/cleanup, the synthetic ChatGPT/Copilot transport regression, judgment-router fixtures, and the limits documentation. Current-head CI: lint, type checks, knip, docs, upgrade compatibility, and worker build passed; Test, app build, and JavaScript/TypeScript analysis are still running. The real-binary subscription transport tests are opt-in and were not run during this review.

Reviewed 04e3ac6

@roomote-roomote

roomote-roomote Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Fixed both findings in 04e3ac6c:

  • Gateway-bound helpers strip inline subscription OAuth and use a private auth store, preserving shared ordinary-session auth and cleaning up on exit/startup failure.
  • Judgment-router fixtures supply the persisted task binding, assert task-ID forwarding, and verify missing bindings never dispatch judgments. The production binding guard is unchanged.

Validation: full SDK suite 2,670 tests; full cloud-agents suite 2,646 tests serially; 259 focused helper/decision checks, including real OpenCode ChatGPT/Copilot transport regressions. Both transport regressions fail with protection disabled and pass restored, using synthetic auth and a local denying gateway. Lint, types, knip, judgment and push hooks pass. The local parallel suite's separate global-compute fixture race is documented in the PR.

Current-head CI is settled: Test passes, Roomote code review passes with no new findings, both original threads are resolved, and all 29 reported checks are completed without a failing/pending result. The platform promoted the PR after the clean review; metadata refresh preserved that state. No merge performed. No human-input blocker remains.

@roomote-roomote
roomote-roomote Bot marked this pull request as ready for review November 3, 2026 12:35
@brunobergher
brunobergher merged commit 93478b6 into develop Nov 3, 2026
30 checks passed
@brunobergher
brunobergher deleted the fix/bound-agent-inference-loops-3cvs4f4n04ft5 branch November 3, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runaway repeated tool-call loops silently consume unbounded metered inference credits

2 participants