feat(deletion): add durable whole-community deletion

[kalvinnchau]

Summary

Adds a durable, operator-controlled V1 for deleting an entire Buzz community without deleting another tenant's data.

The workflow is exposed through buzz-admin deletions:

• sweep records independent fleet storage-taxonomy observations
• submit, list, inspect, and approve manage a deletion request
• run and drain execute bounded work

Requests advance through a PostgreSQL-backed state machine and stop at retention_pending after logical deletion has been independently verified across PostgreSQL, object storage, and Redis.

This PR ships the engine and CLI, not a continuously running worker or Kubernetes packaging. For V1, a cluster/VM administrator invokes /usr/local/bin/buzz-admin from the existing relay image, for example with kubectl exec or an equivalent container/VM exec path.

What whole-community V1 removes

For the target community, V1 removes:

• rows from the allowlisted community-scoped PostgreSQL catalog, including members, profiles, authored events and bodies, DMs, reactions, mentions, memberships, tokens, workflows, moderation, audit, feedback, and rate-limit state
• media sidecars and upload-attribution records under _meta/<community>/ and _uploads/<community>/
• Git repository pointers under repos/<community>/
• Redis keys under buzz:<community>:*

The community row survives as a permanent tombstone, and deletion control-plane records remain as evidence of the request, approval, execution, and result.

Safety model

Deletion is not a broad DELETE CASCADE followed by optimistic cleanup. The destructive boundaries are durable and fail closed.

1. Inventory and approval

• submit resolves the target and freezes the schema plus summary-only storage inventory.
• Approval is bound to the exact request, community, and frozen inventory digest.
• Unsupported manifest versions, malformed keys inside the target's owned prefixes, live scoped-table/write-fence coverage drift, frozen-inventory mismatch, and approval mismatch block execution rather than guessing. Migration and catalog revision numbers are not authorization gates; the executor validates the live safety shape instead.
• Storage inventory is server-side prefix scoped to exactly:
  • _meta/<community>/
  • _uploads/<community>/
  • repos/<community>/
• The deletion path never lists the whole shared bucket and has no arbitrary per-community object cap. Its listing work is proportional to the target community's bindings, not total fleet storage.
• Fleet-wide taxonomy sweeps remain independent observability. They report unknown writer shapes but do not gate deletion submission, fencing, or destructive progress. Maintainers must add deletion taxonomy coverage whenever a new community-owned object-key class is introduced; writer-coverage tests bind the current media and Git writers to that contract.

2. Quiesce, fence, and destructive freeze

• Writes continue through submission, inventory, and approval. They stop when execution moves the target into quiescing and then establishes the durable fence.
• Already-admitted external effects finish under heartbeated serving-write leases; the exact admitted lease may renew while the community is quiescing, but new lease acquisition is rejected. The executor drains admitted leases before destructive work.
• Invite minting after quiescing begins fails as typed AccessDenied (HTTP 503 at the relay boundary) before an invite can be persisted.
• Database triggers enforce the community write fence across the complete catalog of community-scoped tables. Startup/readiness and destructive execution validate that catalog so a newly added but unfenced table cannot silently escape.
• After the fence closes writers, storage is re-enumerated into chunked side-table rows. Per-prefix counts and digests bind those concrete keys to the destructive manifest.
• Manifest chunk insertion, update, and deletion are protected after freeze. This closes the race where an unbound key could otherwise appear after the manifest was committed.

3. Checkpointed destruction

• Target-owned object bindings are deleted from the frozen destructive manifest in bounded batches with durable progress.
• The concrete key list lives in chunked side-table rows rather than one request-row JSON value. It supports large communities, resumable execution, and terminal cleanup.
• Missing objects are accepted as idempotent crash-window outcomes; malformed ownership, changed evidence, and unexplained target-prefix drift fail closed.
• PostgreSQL purging remains scoped by community_id, including the guarded NIP-RS hard-delete path discovered with real Desktop kind 30078 read-state data.
• Redis cleanup explicitly scans and UNLINKs only buzz:<community_id>:*. Natural expiry is insufficient because some keys, including tunnel generation counters used as fencing state, are deliberately persistent.

4. Independent verification

• PostgreSQL logical absence is checked after purge.
• The three target-owned storage prefixes are freshly inventoried again and must be empty.
• Redis requires two complete empty namespace scans.
• Only after all three stores pass does the request advance through logically_verified to retention_pending.

What V1 deliberately does not erase

Shared content-addressed storage

Per-community deletion removes bindings, metadata, attribution records, and Git pointers. It does not physically delete fleet-shared CAS bytes that another community may still reference:

• media blobs and thumbnails
• Git manifests, packs, and indexes (manifests/, packs/, and idx/)

Safe reclamation requires a separate fleet-wide reachability and retention GC. Unknown keys elsewhere in the shared bucket do not block one community's deletion; malformed or unrecognized keys inside that community's three owned prefixes still fail closed.

External retained copies

The online logical-deletion proof does not erase object versions/replicas, database backups/WAL, CDN copies, provider retention copies, or observability exports. Those require their own retention and purge controls.

Member-only erasure

This PR erases a whole community. It does not implement the different operation "erase one npub while preserving the community."

Removing membership or accepting NIP-09 is not member erasure. A member-only workflow would need to find and selectively remove or redact authored event content and pubkeys, profile data, DMs, reactions, mentions, memberships/roles, tokens, workflows/subscriptions, upload attribution, moderation/audit history, repository attribution, and identity embedded in tags or JSON. It would also need explicit rules for ownership transfer, surviving replies and thread metadata, audit-chain integrity, immutable Git history, and shared-CAS reachability. That requires a pubkey-level fence and selective graph rewrite; it is a separate deletion product, not a safe extension of this whole-tenant worker.

In scope

• migration 0029_community_deletion.sql: requests, approvals, leases, manifest chunks, checkpoints, tombstones, and the universal write-fence catalog
• durable executor leases, generations, heartbeats, retry/block state, and resumable stage transitions
• operator-driven sweep, submit, list, inspect, approve, run, and drain commands
• serving-path fences for database writes and external effects across event ingest, media, Git, workflow, push, invites, mesh/tunnel, and related paths
• target-prefix-only storage inventory, summary manifests, post-fence destructive chunks, and bounded batch deletion
• exact community Redis namespace purge and two-pass absence verification
• cross-community isolation, crash/resume, manifest-integrity, writer-taxonomy, and schema/migration regressions
• desired-state schema/schema.sql support without requiring a SQLx migration ledger

Deferred / not covered

• dedicated Helm/chart worker Deployment, service account, secrets, probes, resources, and network policy
• autonomous buzz-admin deletions worker poll loop and worker-only health server
• least-privilege separation among migration, relay-serving, and destructive execution roles
• fleet-wide shared-CAS physical GC
• backup/provider/CDN/observability retention completion
• member-only erasure
• provider-native conditional-delete improvements
• a general force-continue or operator repair escape hatch; permanent safety failures remain fail closed

The removed continuous-worker implementation is preserved on am/community-deletion-worker-followup for separate review.

Validation

Current PR head and repository state

Current head: f559e69e93590a1f794e455cb8016a89d744c789, based on bd2fdf4a2f8e00ec25ddc2372b39958b8df77176.

At this head:

• local, upstream, and GitHub PR heads were verified equal
• PR diff: 42 files, 9,776 additions, 506 deletions
• cargo clippy -p buzz-relay --tests --locked -- -D warnings passed
• formatting and final diff checks passed
• 9 send-loop regressions passed, together with lifecycle socket-type/fence and archive-race tests
• the full relay library run reached 865 passed / 43 ignored; one concurrent tracing-global-state test flaked but passed alone, and the mesh-demo 504 reproduced alone on the exact base commit
• nine unrelated Desktop timeout/version-probe failures under parallel load passed individually

This is not yet a merge-readiness claim. On this exact head, Semgrep and zizmor passed, but the required core GitHub Actions workflows had not started and therefore were absent rather than terminal green. The PR also remained CHANGES_REQUESTED without a current approval when last checked.

Live Kubernetes deletion and safety gates

The full live test program used real PostgreSQL, Redis, S3-compatible object storage, and a three-relay Kubernetes fleet. The gates are green when the unchanged destructive cells from 27aa6894f8af5710e2e9a00533823d6176cd84cf are combined with the close-frame-only repair verified at f559e69e93590a1f794e455cb8016a89d744c789.

Passed gates:

• Deterministic crash recovery: the generation-1 executor was force-killed after claiming the approved request and before effects. The request remained durably approved; after lease expiry, a successor reclaimed it at generation 2 and completed all checkpoints to retention_pending.
• 10,001-object boundary: the frozen inventory contained exactly 10,001 objects. Durable chunks deleted 10,000 and then 1. Target PostgreSQL rows, storage bindings, and Redis keys reached zero.
• Cross-community isolation: active and idle control communities, tenant controls, and global controls remained intact across the destructive run.
• Per-pod socket behavior: six directly targeted, authenticated witnesses—ordinary and real huddle-audio WebSockets on each of three pods—received WebSocket 1008 with reason community deleted.
• Fleet health: all three relay pods remained Ready after deletion and ran the same immutable exact-head image.

The final close-frame cell used:

• source: f559e69e93590a1f794e455cb8016a89d744c789
• image: buzz-e2e:sha-f559e69e9359
• immutable image ID: sha256:64817be8f8d4a03673275da67dc3b7c2f4194d3534d651e5a6baca1dfe2d616d
• evidence root: /tmp/buzz-e2e/20260806T190326Z-f559e69e9359-close-rerun
• evidence index SHA-256: 3a9eeac55b96adfe761b78dc292ba7f0c9189ccd8c8a13f0e645ec64f5a9884c

The audio witnesses exercised the production protocol: /huddle/{channel_id}/audio, server challenge, signed kind-22242 NIP-42 event, protocol-v2 auth payload, and an observed joined response before deletion. Ordinary witnesses independently observed NIP-42 acceptance for their exact auth events.

The later f559e69e9 patch changes only deletion close-frame behavior, so the earlier SIGKILL/reclaim and 10,001-object results remain applicable. Failed witness-development attempts were excluded from the decisive gate and retained separately as broken-instrument evidence.

Cleanup

The final Kubernetes namespace, retained PVC and secret, port-forwards, witness processes, and Screen sessions were removed. Namespace absence and the absence of Screen sockets were verified. Evidence remains preserved at the path above.

[wesbillman]

Review submitted by Carl on Wes's behalf. I traced the deletion lifecycle, DB trigger and lease model, storage manifest freeze/delete path, Redis purge/verification, and serving-side call sites, with independent adversarial passes from Mongo and Princess Donut. CI is green, but these findings block the feature's advertised safety/operability contract.

[wesbillman]

[P0] Bump this to 28 and pin it to the actual migration set. This PR moved community deletion to 0028_community_deletion.sql, but SQLx-backed deployments report max(version) = 28. validate_destructive_migration_version requires exact equality, so validate_catalog() rejects every submit/inventory and execution attempt with expected 27, got 28. Desired-state deployments hide this because they have no _sqlx_migrations ledger. Please also add a regression that compares this constant to the migrator's concrete max version rather than testing only values derived from the constant.

[wesbillman]

[P1] Keep already-admitted leases renewable while quiescing, or otherwise preserve the exclusion proof for long effects. begin_quiescing promises that admitted external effects finish and the fence drains them, but this predicate makes the next 10-second heartbeat fail as soon as state becomes quiescing; ServingWriteGuard then cancels the protected operation and its drop path releases the lease. Media/video and Git object-store writes can legitimately exceed that interval, and cancellation of the client future does not prove the remote S3 mutation did not commit after the lease was released. That allows destructive listing/freeze to overtake an admitted remote write—the exact race the lease is meant to close. New acquisition is already blocked by deletion_state = 'active'; renewal should remain allowed for the exact existing lease/fence generation through quiescing, with a regression covering a long admitted external write across quiesce.

[kalvinnchau]

Fixed in 4e0726e5a. Renewal now permits only the exact existing, unexpired lease while the community is active or quiescing; owner, lease generation, fence generation, expiry, tombstone, and community fence-generation checks remain intact, while new acquisition is still restricted to active. Added PostgreSQL coverage for renewal/drain behavior and a long-running ServingWriteGuard heartbeat regression across quiesce. Both pass against isolated PostgreSQL at PR HEAD 7c755960d.

[wesbillman]

[P2] This does not map the actual fence failure. mint_relay_invite performs an ordinary insert; the new relay_invites trigger raises SQLSTATE object_not_in_prerequisite_state, which arrives as DbError::Sqlx, not DeletionSafety. The fallback therefore returns HTTP 500 during quiescing/fencing. Guard the invite transaction so the boundary produces a typed access/fence error (and map it to the intended restriction status), or explicitly translate this SQLSTATE. Please cover mint-after-quiesce at the API boundary.

[kalvinnchau]

Fixed in 7c755960d. mint_relay_invite now takes the community advisory lock and checks the write fence inside the same transaction before inserting, returning typed DbError::AccessDenied instead of leaking the trigger's SQLx error. The API maps that typed fence to HTTP 503. Added PostgreSQL coverage proving mint-after-quiesce returns AccessDenied and persists no row, plus an HTTP boundary regression proving the typed error maps to 503. Both pass at PR HEAD 7c755960d.