Skip to content

[slopfix] fix(pegboard): decrement kv_storage_used metric on actor delete - #5356

Open
MasterPtato wants to merge 1 commit into
stack/slopfix-refactor-universaldb-ups-single-multi-node-udb-over-nats-remove-ups-postgres-driver-vuorlwlufrom
stack/slopfix-fix-pegboard-decrement-kv_storage_used-metric-on-actor-delete-uqzqlkws
Open

[slopfix] fix(pegboard): decrement kv_storage_used metric on actor delete#5356
MasterPtato wants to merge 1 commit into
stack/slopfix-refactor-universaldb-ups-single-multi-node-udb-over-nats-remove-ups-postgres-driver-vuorlwlufrom
stack/slopfix-fix-pegboard-decrement-kv_storage_used-metric-on-actor-delete-uqzqlkws

Conversation

@MasterPtato

@MasterPtato MasterPtato commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@MasterPtato

MasterPtato commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Stack for rivet-dev/rivet

Get stack: forklift get 5356
Push local edits: forklift submit
Merge when ready: forklift merge 5356

change uqzqlkws

@claude

claude Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Review

Small, well-targeted fix: the pegboard_actor_metrics / pegboard_actor2_metrics workflows periodically record the actor's KV size into a namespace-level KvStorageUsed gauge but never decremented it when the actor (and its KV) is destroyed, so the namespace total drifted upward permanently. This adds a RemoveKvMetrics activity after the lifecycle loop breaks that subtracts the last recorded size.

Correctness

  • The decrement logic mirrors the existing record_kv_metrics delta pattern (namespace::keys::metric::inc with a negative delta), consistent with the existing atomic-add accounting.
  • The if last_kv_storage_size != 0 guard correctly avoids an extra transaction and avoids double-decrementing when the last periodic record_kv_metrics call (which also runs once more on the Destroy iteration, since it isn't gated on !destroy) already observed a shrunk/deleted KV.
  • Activity-state semantics check out: ActivityCtx::state()'s StateGuard only writes its mutated copy back to the workflow's durable state when the activity returns Ok, and completed activities are never re-executed on replay, so there's no partial-corruption or double-decrement risk from retries.
  • ctx.v(2) reused for the new RemoveKvMetricsInput call (same version number as the pre-existing GetTsInput call earlier in the loop) is safe, not a bug. compare_version in gasoline only guards against nested de-escalation within a single call chain; independent top-level ctx.v(N) call sites do not need distinct or monotonically-fresh version numbers, so reusing 2 for a second, unrelated call site added later is consistent with how this pattern is used elsewhere.
  • actor/metrics.rs and actor2/metrics.rs are kept in parity, matching this repo's convention of maintaining both actor versions side by side. actor2 additionally tags all its txns with tx.priority(Priority::Low), including the new remove_kv_metrics txn, consistent with the rest of that file (v1 actor/metrics.rs does not use Priority::Low anywhere, so the asymmetry between the two files is pre-existing style, not an oversight in this PR).
  • Worth a mental double check even though it is outside this diff: is there any actor-KV-deletion path (crash cleanup, orphan reaping, forced eviction) that deletes KV without routing through a metrics::Destroy signal to this workflow? If such a path exists, it would reintroduce the same leak this PR fixes, since the decrement only fires when this workflow's loop observes the Destroy signal.

Minor / nit

  • Already-completed workflow instances (actors destroyed before this fix ships) will keep their stale contribution baked into the namespace KvStorageUsed gauge forever, since this code path never runs for finished workflows. Might be worth a one-off backfill/reset note if the drift matters for existing namespaces, but not a blocker for this change.
  • examples/kitchen-sink/scripts/seed-large-dbs.ts: flags/env vars are parsed with Number(flag(...)) || envNum(...), so an explicit --count=0 (or similar) would fall through to the env/default value instead of being honored, since 0 is falsy in JS. Very low severity for a load-generation script with no legitimate zero value, but worth a ??-based check if precise flag semantics ever matter here.
  • PR title carries a [slopfix] prefix. Per this repo's CLAUDE.md, commit/PR titles should be pure conventional commits with no extra tags, worth dropping the prefix before merge.
  • The PR also carries a large regenerated examples/kitchen-sink/dist-server/server.mjs (+11.9k/-11.1k lines) and .mjs.map diff, plus a new SQLite-compaction load-testing harness (grow-db.ts, seed-large-dbs.ts) that is unrelated to the metrics fix itself. Notably, examples/kitchen-sink/dist-server is listed in examples/kitchen-sink/.gitignore yet the built bundle is tracked in git and regenerated here anyway. Bundling this much unrelated scaffolding and a huge generated-file diff into a one-line metrics bugfix makes the actual change harder to review; consider splitting the load-test tooling into its own PR next time.

Test coverage

  • No test added, and there is no existing coverage for KvStorageUsed lifecycle under engine/packages/pegboard/tests/. Not a regression in coverage, but since this fixes a real metric-drift bug, a test asserting the gauge returns to baseline after actor destroy (via the existing UDB-backed pegboard test harness) would help guard against this regressing again.
  • The new examples/kitchen-sink/src/actors/testing/grow-db.ts and scripts/seed-large-dbs.ts are load-test scaffolding rather than automated tests; that is fine for their stated purpose (seeding large uncompacted DBs for a compaction load test) but they do not exercise the actual metric fix in this PR.

Overall this looks correct and low-risk. Nice fix.

@MasterPtato
MasterPtato force-pushed the stack/slopfix-fix-pegboard-decrement-kv_storage_used-metric-on-actor-delete-uqzqlkws branch from d0ae940 to 711cbf5 Compare August 7, 2026 01:27
@MasterPtato
MasterPtato changed the base branch from main to stack/slopfix-refactor-universaldb-ups-single-multi-node-udb-over-nats-remove-ups-postgres-driver-vuorlwlu August 7, 2026 01:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant