feat(billing): track E2B and Daytona Function sandbox usage - #7184
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Greptile SummaryThis PR meters E2B and Daytona sandboxes used by workflow Function and cloud Pi execution through the existing trace and usage-ledger pipeline.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| apps/sim/lib/function-execution/execute-request.ts | Gates remote sandbox metering by workflow scope and carries provider-generated cost through Function execution results. |
| apps/sim/lib/execution/remote-sandbox/index.ts | Defines the provider-neutral metering window and attaches cost only to outcomes considered billable. |
| apps/sim/lib/billing/sandbox-pricing.ts | Calculates E2B and Daytona runtime cost from fixed resource profiles, effective lifetime, and the configured multiplier. |
| apps/sim/executor/execution/block-executor.ts | Preserves trusted Function cost across retries and post-handler failures so trace output receives the authoritative total. |
| apps/sim/executor/handlers/pi/pi-handler.ts | Aggregates Sim-paid sandbox usage into Pi tool cost for successful, streaming, and backend-reported error outcomes. |
| apps/sim/executor/handlers/pi/local/sim-tools.ts | Accumulates remote Function sandbox charges produced by Function tools invoked during local Pi sessions. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[E2B or Daytona sandbox] --> B[Measure completed runtime]
B --> C[Calculate provider cost]
C --> D[Function output cost or Pi sandbox-cost sink]
D --> E[Block span cost or Pi toolCost]
E --> F[Trace cost summary]
F --> G[Workflow usage ledger]
Reviews (8): Last reviewed commit: "fix(billing): carry the Pi charge onto a..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 17 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
@greptile-apps @cubic-dev-ai please re-review the latest changes on this PR. The prior findings have been addressed or resolved with the documented scope decision, and the branch is updated with current staging. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 22 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
@cubic-dev-ai please re-review the latest changes in dd7aa1c. The three findings from the prior run have been addressed with focused E2B/Daytona, outbox, and Function response regressions, and all threads are resolved. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 22 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
@cubic-dev-ai please re-review the latest changes in a3a6dd2. Both latest sandbox billing findings are fixed, focused regressions pass across E2B and Daytona, and the full requested validation set is green. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
|
Implemented the sandbox durability follow-up in Review fixes included:
Focused validation:
Policy decisions remain unchanged: unresolved provider identities are no-charge; Daytona account allowances/credits are ignored in favor of configured list-rate pricing; Mothership scope is unchanged. Out-of-scope follow-ups identified for separate work:
@cubic-dev-ai please re-review the latest commit, especially the durable pre-create intent, correlation recovery, terminal evidence precedence, and billing-period lock coordination. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 25 files
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Fix all with cubic | Re-trigger cubic
c88112e to
e48e74f
Compare
There was a problem hiding this comment.
All reported issues were addressed across 25 files
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Fix all with cubic | Re-trigger cubic
e48e74f to
3125336
Compare
|
Implemented the complete sandbox lifecycle review follow-up in 3125336, rebased onto staging at b094b4d. Finding-by-finding:
Validation after the final staging rebase:
Policy remains unchanged: an unresolved provider identity is no-charge; Daytona allowances/credits are ignored; pricing and multiplier behavior are unchanged; Mothership, local JavaScript, and custom Function-tool scope are unchanged. The broader long-workflow settlement and internal terminal-log partial-total concerns remain separate follow-up work. @cubic-dev-ai please re-review the new head 3125336. |
@BillLeoutsakosvl346 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 29 files
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Fix all with cubic | Re-trigger cubic
Pi's own sandbox was never metered. withPiSandbox called createSandbox without the meterUsage argument, so only sandboxes created through executeFunctionRequest were charged — and Pi's is the larger consumer by an order of magnitude. A Function block holds one for seconds; a Pi session holds one for a minimum lifetime of 31 minutes. The gap was worst exactly where it was least visible. A Pi coding agent normally runs BYOK, so its model cost is zero by definition, and the ledger bills a model row on total > 0. With the sandbox unmetered, such a run produced a zero-cost model_unbilled row and Sim collected only the flat execution fee while paying its provider for the whole session. Threads a cost sink through PiRunContext, which is the seam backends already receive and the only one that reaches all four cloud modes. The handler owns one sink covering both sandbox sources — Function tools in local mode, the agent's own sandbox in cloud mode — so neither can be dropped where the cost is folded into the block's output. It rides in toolCost for the same reason the Function tool cost already does: that is what survives the BYOK zeroing. Unlike the Function path this charges on creation rather than on a completed session. A Function run is seconds long, so absorbing one the provider failed to deliver is cheap and reads as fair; tens of minutes of Pi compute is consumed whether the agent finished, errored, or was cancelled, and billing only clean endings would mean paying for every other one. A create that throws still costs nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The handler test mocks the backend and writes into the cost sink by hand, so it proves the wiring from a backend to the block's cost and nothing more — it would still pass if withPiSandbox never metered at all, which is precisely the bug that path had. Holds a real Pi sandbox open for a known interval and bounds the charge between what that interval must cost and what the whole session could justify. Verified to fail against the original unmetered call with "expected 0 to be greater than or equal to 0.00023", and to pass once the sink is threaded: 5.949s billed of a 6.141s session on E2B. The second case pins the other half of the contract — a caller that supplies no sink is not charged, which is what keeps mothership and other internal Pi sandboxes free. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Aligns Pi with the outcome policy the Function path already applies rather than keeping the divergence the previous commit introduced. A session that ends by throwing — a provider crash, a lifetime limit, a cancellation — is absorbed, because a charge nobody can tie to delivered work is not one worth defending, and consistency across the two sandbox paths is worth more than recovering the cost of runs that failed. A command exiting non-zero is still billed: the callback returns normally there and the agent produced its answer, which is the same reason the Function path bills its own non-zero exits. The window still closes at teardown, so a completed session is charged for the whole time the provider held its sandbox. Verified on both providers, including that the new case fails when the charge is applied unconditionally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tion Three paths dropped cost the sandbox had already earned. A harvest that cannot return what the run produced — more files than the export limit, nesting past the listing depth, or an output directory the code deleted — was excluded from the billable-error set. All three arrive only after the sandbox has executed and all three are the caller's to fix, so they belong with the post-completion export failures the policy already bills rather than the provider failures it absorbs. A completed run whose code wrote one file too many went free. That also left the route with nothing to read: it already consults readTrustedSandboxOutputCost for these errors, so attaching the cost at the sandbox layer is what carries it into the response. Separately, a Function block whose handler succeeded could still fail in the steps that follow it — base64 hydration, and large-value redaction that throws rather than emit unredacted data. Those errors carry no cost of their own, so the completed sandbox went unbilled. The handler's cost is now held across that window, in the same way streamingPartialOutput already is, and used only when the error has none. The new conformance case was confirmed to fail against the narrower catch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@cubic-dev-ai review |
@icecrasher321 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 39 files
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Fix all with cubic | Re-trigger cubic
A backend that returns a result carrying `totals.errorMessage` has already run: the sandbox was billed and the sink holds the charge. But that path throws instead of reaching `buildOutput`, which is what publishes the cost, so the charge was accumulated and then dropped — lost revenue rather than an over-charge. Both failure paths now carry it on the error they raise, the same way the Function handler carries its tool cost, so `handleBlockError` can pick it up. An agent that ran and then reported a failure consumed the same tokens and sandbox seconds as one that succeeded, which is why the cost computation is now shared between the two rather than duplicated. Also corrects the sink's doc comment. Local mode does fill it — the agent runs on the caller's own machine and costs Sim nothing, but a `function_execute` among the Sim tools it calls bills its own remote sandbox into the same total. The new case was confirmed to fail without the attach. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@cubic-dev-ai review |
@icecrasher321 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
No issues found across 39 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
|
@greptile Re: the 4/5 on workflow-scoped attribution — traced this, and the guard has not gone missing; the design it belonged to was replaced.
Under the present design Attribution is enforced fail-closed at the layer that does write, if (workflowRecord.workspaceId && !billingContext) {
throw new Error('Billing attribution is required for workspace execution usage')
}That guard predates this PR and applies to every billable target, model cost included — sandbox cost is not a special case. The two reachable shapes both terminate safely: a workspace-scoped run without attribution throws there rather than silently dropping the charge, and a run with no
Happy to be shown a path where a metered run reaches the ledger without attribution — that would be a real hole and I'd fix it — but I could not construct one against the current code. |
|
@cubic-dev-ai review |
@icecrasher321 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
No issues found across 39 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
|
You're right — the previous finding was based on the superseded architecture. In the current head, |
|
@cubic-dev-ai review |
@icecrasher321 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
No issues found across 39 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
2 participants
Summary
codeandshellsandboxes for E2B and Daytonaoutput.costtoolCostsandboxProfile: mothershipunmeteredTrace-native billing flow
There is no sandbox-specific outbox, reconciliation worker, billing-period machinery, database schema, or new ledger path.
Runtime begins immediately before provider creation and ends when Function cleanup starts. Pricing uses the provider-effective TTL as a cap, the configured production multiplier once, and eight-decimal final rounding. Cleanup time is Sim infrastructure overhead.
Pi sandbox metering
Pi's own sandbox was never metered:
withPiSandboxcalledcreateSandboxwithout themeterUsageargument, so only sandboxes created throughexecuteFunctionRequestwere charged. It is the larger consumer by an order of magnitude — a Function block holds a sandbox for seconds, a Pi session for a 31-minute minimum lifetime.The gap was worst where it was least visible. A Pi coding agent normally runs BYOK, so its model cost is zero by definition and the ledger bills a model row on
total > 0. With the sandbox unmetered, such a run wrote a zero-costmodel_unbilledrow: Sim collected only the flat execution fee while paying its provider for the whole session.A cost sink now travels on
PiRunContext, the seam backends already receive and the only one reaching all four cloud modes. The handler owns one sink covering both sandbox sources — Function tools in local mode, the agent's own sandbox in cloud mode — so neither can be dropped where cost is folded into the block's output. It rides intoolCost, which is what survives the BYOK zeroing.Outcome policy
Applies to both Function and Pi sandboxes; a Pi session that ends by throwing is absorbed exactly as a Function run that never completed is.
Billable completed sandbox outcomes:
Uncharged outcomes:
The parent-failure rule intentionally matches hosted-key trace collection: nested tool cost is published through the completed parent span, not billed independently from partial child execution.
Pricing and scope
COST_MULTIPLIERAccepted limitations
toolCost, so it is not separable from that block's other non-model cost in the ledgerValidation
Latest focused run after rebasing onto staging:
apps/simtype-check passedapps/simlint passedgit diff --checkpassedCharged amounts, verified against live providers
The unit tests pin the arithmetic and the conformance suite proves a cost is produced and routed — but that suite stubs the provider and mocks
Date.now()to advance one millisecond per call, sototal > 0is the strongest claim it can make. It would hold equally well if the metered window were anchored to the wrong instants. Two opt-in smoke suites (SANDBOX_BILLING_SMOKE=1) hold a real sandbox open for a known interval and bound the charge between what that interval must cost and what the whole call could justify:Every window sits inside its session, excluding a few hundred milliseconds of Sim-side overhead, and both implied rates match the providers' published pricing exactly.
Each smoke case was also confirmed to fail against the defect it guards, rather than assumed to: the Pi case reports
expected 0 to be greater than or equal to 0.00023against the original unmetered call, and the uncharged-session case reports a non-zero charge when the fee is applied unconditionally.Ticket: https://sim-ai.slack.com/archives/C093DF8MA21/p1787866922654329