Repository navigation
Propose inactive Pack execution fences and receipt journal #141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sidneyswift
wants to merge
3
commits into
main
Choose a base branch
from
codex/pack-worker-fence-2269
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| # Paired execution and claim contract | ||
|
|
||
| Implemented as inactive proposal SQL; see [README](README.md) for qualification and rollout boundaries. Current production/legacy commands remain unfenced. The preceding investigation found delayed phase, finish, context, run, settle, claim, dispatch and same-attempt-resume writes could mutate newer work. The proposal addresses those source-contract counterexamples; local tests are not a claim of hosted enforcement. | ||
|
|
||
| ## Identity and ownership | ||
|
|
||
| Existing jobs default to protocol 1. A new protocol-2 prepare is explicit; an existing legacy job cannot be adopted. Both public new-prepare paths take the same owner/idempotency-key transaction advisory lock before lookup/insertion. Protocol 2 is assigned directly on INSERT; conflict resolution locks the actual owner/key winner and verifies protocol and fingerprint. No UPDATE promotes a protocol-1 row. These two protections prevent an uncommitted legacy insert from becoming a protocol-2 job after a unique-index wait. A guarded legacy wrapper rejects protocol-2 jobs, including idempotency-key resolution, and retains the original command for protocol 1. The original command is renamed to a private internal function; service-role direct Pack INSERT/UPDATE/DELETE and internal-function execution are revoked. Current API Pack persistence uses the RPC, not raw table DML. A future rollout must inventory other callers before applying those grants. | ||
|
|
||
| Each newly accepted approval/resume creates an execution UUID and increasing job execution version. The transition request UUID and complete JSONB payload are retained. Source execution/version and creative revision are compare-and-set inputs. Resume also compares the original asset attempt/claim. Exact accepted transition replay returns its recorded result with `dispatch=false`; conflicting payload reuse refuses. Replay lookup precedes source-generation comparison, so an old accepted transition can be inspected without regaining write authority. An unseen stale resume cannot rotate a newer generation. | ||
|
|
||
| The workflow receives the original accepted execution as an immutable launch argument. `run` binds one actual Workflow ID, null-to-ID or same-ID only; workflow IDs are unique across executions. Preparation checks the original identity again after reading the job. Production compares actual Workflow metadata before claiming. No worker repairs a mismatch using a current-row identity. | ||
|
|
||
| A claim belongs to one execution and asset. New paid claims increment the existing attempt through the original command; uncertain resume preserves the paid attempt, billed credits and provider state but creates a new claim. Its immutable operation-ID array grants continuation of existing receipts. A deterministic claim request UUID is payload-bound. A repeated claim returns `claimed=false,replayed=true` and the API stops for reconciliation: an acknowledged old claim is not a second production permit. | ||
|
|
||
| ## Atomic write rules | ||
|
|
||
| All commands recheck actor/org access; current organization membership is locked through the transaction. The lock order is job, then asset. Every worker validates revision, execution UUID/version and bound Workflow ID **before** expiry can settle/refund anything. Phase/finish additionally validate attempt/claim and retained execution association. Missing/null/wrong identities refuse as well as older valid identities. | ||
|
|
||
| | Command | Additional rule | | ||
| | --- | --- | | ||
| | `run` | Compare-and-set Workflow ID; cannot adopt another run. | | ||
| | `dispatch_uncertain` | Original generation required; a bound run is not overwritten. | | ||
| | `claim` | Original expected pending attempt; one claim per execution/asset. | | ||
| | `phase` | Full claim fence; provider intent, request identity and transition order immutable; receipt append acknowledged before accepted/complete write. | | ||
| | `finish_asset` | Full claim fence, including catch writes; exact completion replay only. An incomplete provider submission cannot become a confirmed failure. | | ||
| | `context` | Original execution plus expected context version; acknowledged updates advance that version. Foundation read-modify-write preserves original execution. | | ||
| | `settle` | Original execution; retained exact terminal replay, one refund. Different payload refuses. | | ||
| | `prepare` update, `estimate`, `export` | Expected revision; retry estimate also compares expected failed attempt. | | ||
| | `read` | Authorized current-row recovery may expire current work; it is not an old worker write permit. | | ||
|
|
||
| Stale refusal changes no Pack, wallet or usage state, including after the newer hold expires. A current admitted expiry settles once and records the terminal execution result. A subsequent worker call may receive a terminal refusal rather than a fresh permit; authorized read returns current state. Lost acknowledgements require reconciliation. | ||
|
|
||
| Provider registration occurs atomically with `submitting`, before external I/O. A duplicate registration reports `operation_created=false`; the API never submits on that response. Accepted request charging remains inside the original command and occurs once. Completed state requires prior accepted state and its charge; changed request IDs, input/model/price, regression and conflicting completed payloads refuse. Existing non-provider phase values also cannot be overwritten with a different value. | ||
|
|
||
| Allowed models and charges come from the quote's frozen execution configuration. Image slots are 0–3 for carousel, otherwise 0; motion is video-only. A retained ready foundation can select the quote's image-edit model for a dependent only when the supplied plan/asset digests match current durable context, job/owner/revision/quote match, the exact dependent attempt is bound, and the source artwork is completed. Original quote references otherwise determine image versus image-edit. No new rates are introduced. | ||
|
|
||
| ## API failure boundary | ||
|
|
||
| Protocol 2 is an internal optional admission argument, never inferred from public request fields or environment flags. The existing public calls and legacy Workflow argument shapes remain unchanged. All normal and catch writes carry original fences. Receipt append has its own transaction and is acknowledged before an active phase write. A refused/uncertain write raises a fence error without provider resubmission or catch-path finish. | ||
|
|
||
| The Workflow SDK reconstructs step errors, so the protocol-2 workflow stops on **any** rejected step rather than relying on a custom error class crossing that boundary. Settlement uses the existing zero-retry bounded step outside the production catch path. A lost settlement acknowledgement causes one attempt, no automatic retry. The legacy workflow retains its existing handling. Unwired bounded/foundation adapters propagate the same original admission and attempt identities; they are not activated by this change. | ||
|
|
||
| ## Evidence and limits | ||
|
|
||
| Actual PostgreSQL checks include deterministic two-connection legacy-first/protocol-2-first/same-protocol creation races, then cover stale writers before/after expiry, same-attempt resume, transition/claim/completion/settlement replay and conflict, concurrent claim/intent registration, context CAS, atomic charge refusal, receipt retention/access/bounds and the existing legacy controls. The paired API-to-PG fixture covers real claim/intent/journal/charge/completion, claim replay refusal, mismatched actual Workflow identity, foundation-dependent image-edit and a committed acceptance surviving an actual phase refusal without catch finish or charge. Compiled local SDK checks cover step failure, lost settlement acknowledgement and terminal replay, with networking denied. | ||
|
|
||
| This does not cancel already-issued provider work, guarantee an external I/O fence stays current, attest provider responses, recover a response lost before durable recording, establish cache/QA freshness, or make legacy executions safe. Unknown intent cannot be automatically resubmitted. Operator reconciliation, rollout, hosted SQL, provider execution, paid runtime, renderer activation and customer outcomes remain outside this contribution. | ||
|
|
||
| Post-review fence ordering: unknown/null actions, context CAS, pending-attempt CAS and claim replay/conflict are resolved before expiry recovery. A bound `dispatch_uncertain` call must carry its bound workflow ID; only an unbound dispatch can report uncertainty without one. Expired explicit settlement stores the original caller payload and the flagged result for exact replay. | ||
|
|
||
| Submission intent cannot include `request_id`; only a durable accepted observation permits a charged accepted phase. Neither failed nor completed asset status can close an incomplete submission journal; uncertainty remains available for reconciliation. | ||
|
|
||
| The checked-in Release pack transactions CI runs `test_extended.py`, including protocol creation races and post-review regressions, against disposable PostgreSQL only. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| # Inactive Pack worker-fence proposal | ||
|
|
||
| Status: local paired implementation for review, 2026-11-18. **Do not apply to a hosted database.** These files are deliberately outside `supabase/migrations` and are not referenced by deployment runners. They do not change deployed behavior. | ||
|
|
||
| The paired API branch is `codex/pack-worker-fence-2269` in `recoupable/api`. Its protocol-2 admission is internal and dormant; public REST/MCP callers continue to use protocol 1. Root owns paired acceptance, merge and any later rollout. Legacy workers do not gain a stale-worker guarantee from this proposal. | ||
|
|
||
| ## Files and local qualification | ||
|
|
||
| - `schema.sql`: protocol ownership, execution/claim history, the two receipt tables and guarded legacy wrapper. | ||
| - `command.sql`: generation, workflow, claim and context fences; intent registration and existing-accounting composition. | ||
| - `receipts.sql`: separately committed, redacted observation append/read ports. | ||
| - `test_contract.py`, `test_creation_race.py` and `test_extended.py`: actual command calls on disposable PostgreSQL, using the repository's existing synthetic fixture. | ||
| - `CONTRACT.md` and `RECEIPTS.md`: implemented behavior and remaining boundaries. | ||
|
|
||
| Run with PostgreSQL 17 tools on PATH: | ||
|
|
||
| ```sh | ||
| PYTHONDONTWRITEBYTECODE=1 python3 proposals/pack-worker-fence-2269/test_extended.py | ||
| ``` | ||
|
|
||
| The fixture applies the existing Pack schema and then these three proposal files, in the order above, only to its owned temporary Unix-socket cluster. It uses no TCP listener or hosted credentials and cleans up its own cluster. The suite covers 14 protocol-2/creation cases and the existing 11 legacy controls. The paired API provides an additional runner that exercises real API production helpers against these exact SQL files; provider, media and caption effects are synthetic. Its compiled Workflow SDK fixture verifies failure/replay behavior separately. | ||
|
|
||
| ## Delivery and activation boundary | ||
|
|
||
| Baseline: Database `1b892dd2671ba7e15e0c44b690fbfbe515b59fbd`; paired API `bb542a9924ba531e422b3cd229e75a8bdb1892da`. No historical migration is changed. Migration filename/ownership work in Database PRs #125/#135 and reservation composition in #134/#135 remain separate. Existing Pack accounting remains the implementation called internally; no new wallet, rates or reservation model is introduced. | ||
|
|
||
| Before any rollout, a separately reviewed change must reconcile the current database head and migration ownership, convert this proposal into an ordered migration, verify deployed schema/permissions, inventory and drain or reconcile legacy in-flight executions, qualify the intended authenticated runtime, and explicitly admit new protocol-2 launches. This PR does none of those things. The renderer and foundation/scheduler candidates remain unwired. App 2269 remains open. | ||
|
|
||
| Creation-race repair: both public prepare paths take the same owner/idempotency-key transaction lock. Protocol 2 is assigned on INSERT, never by promoting the conflict winner. The locked winner must already be protocol 2 with matching fingerprint. Three deterministic two-connection regressions cover legacy-first, protocol-2-first and same-protocol contention; they wait for an observed PostgreSQL lock before committing the winner. The legacy-first case failed before the repair by converting the committed legacy row; all three pass after it. Legacy read/replay/estimate remain usable. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # Pack receipt journal contract | ||
|
|
||
| The inactive implementation has exactly two receipt tables, alongside the execution/claim history required by the fence. It is operational evidence, not a wallet, charge ledger or authority to spend. Existing mutable `provider_state`, charge events and Context lifecycle records cannot substitute for this history without losing original association or inventing unrelated identities. | ||
|
|
||
| ## Frozen operation | ||
|
|
||
| `release_pack_provider_operations` is inserted in the successful fenced `submitting` transaction. It records database UUID/time, owner/job/asset, creative revision, original execution UUID/version, actual Workflow ID, claim/paid attempt, phase, provider (`fal`), model, input SHA-256 and quoted credit charge. Foreign keys retain parent/history rows; the command binds all owner and identity fields from its locked rows. No worker has direct table write permission. | ||
|
|
||
| Unique `(owner_id, asset_id, attempt, phase)` permits one operation per paid phase. Registration returns `operation_created`; only a newly acknowledged intent permits submission. Resume keeps that operation and paid attempt and grants the new claim an immutable operation-ID continuation array. An original claim may append only to its original operation; a continuation may append only to its explicitly linked operations. Neither grants current job mutation. A fresh approved paid retry uses a new attempt. | ||
|
|
||
| The command bounds phase slots: image 0–3 for carousel, otherwise image 0, and motion only for video; thus no more than five operation slots per paid attempt. Model/input hash/charge are frozen and validated against the quoted configuration. Evidence append cannot create attempts, operations, continuation grants, holds or charges. | ||
|
|
||
| ## Bounded observations | ||
|
|
||
| `release_pack_provider_observations` has primary key `(operation_id, kind)`, where kind is accepted, completed, failed or `conflict` (first conflict); at most four rows per operation. Payloads are at most 1024 bytes. Accepted/completed/failed accept only: | ||
|
|
||
| - `kind` and `request_id` (1–200 ASCII letters, digits, underscore or hyphen). | ||
| - Optional `output_sha256`, 64 lowercase hexadecimal characters, completed only. | ||
| - Required failed `reason_code`: `provider_failed`, `response_invalid` or `result_unavailable`. | ||
|
|
||
| No raw results, content, prompts, URLs, headers, tokens, filenames, exception text or arbitrary extension fields. Unknown fields/type/size violations are rejected, never truncated. API builds this allowlist; SQL validates it independently. Output SHA is metadata, not proof of retained output or validated content. Provider identities are internal locators, not public progress fields. | ||
|
|
||
| Acceptance precedes terminal observation. Exact payload replay returns `state=reused` and preserves the original timestamp. Changed payload in an occupied slot, differing request identity or conflicting success/failure preserves the original and appends at most one conflict row containing kind and canonical old/new payload hashes. Repeating that same conflict returns `conflict`; further distinct conflicts return `conflict_unretained_capacity`. No extra row or false retention claim is made. Invalid/unauthorized requests cannot consume that slot. | ||
|
|
||
| ## Authority and transactions | ||
|
|
||
| Append/read ports run as dedicated `NOLOGIN pack_receipt_port`, with fixed `pg_catalog` search path and schema-qualified objects. It can SELECT operation/claim/job identity and SELECT/INSERT observations; it cannot update Pack, claim, wallet, hold or usage state. Narrow functions recheck actor/org membership and effective ownership. PUBLIC/browser execution is revoked, and service role gets function execution only. RLS is defense in depth, not a substitute for function authorization. The access helper locks current membership for the transaction. | ||
|
|
||
| Append authorizes the original execution/claim or exact stored continuation, then serializes observations using a transaction-scoped operation advisory lock. It never invokes the Pack command or expiry recovery. A late original receipt therefore cannot mutate a newer hold. This trusted service boundary is not cryptographic provider attestation. | ||
|
|
||
| Ordering: | ||
|
|
||
| 1. Fenced intent and operation commit before submit. Failed/uncertain/duplicate registration does not submit. | ||
| 2. Accepted or completed observation commits in a separate RPC before its active phase write. | ||
| 3. If that phase refuses or rolls back, the committed observation survives. The API stops without catch finish/settle or another provider submission. | ||
| 4. An unacknowledged append is uncertain; exact replay/readback can establish retention, but this slice does not add an automatic reconciliation worker. A crash before first durable receipt remains a loss window. The original intent blocks automatic resubmission. | ||
|
|
||
| Conflict responses commit their bounded row; invalid/unauthorized input raises without inserting. There is no append-then-raise stale check in the same transaction that would erase the evidence. | ||
|
|
||
| The owner-scoped read port validates job and cursor ownership and returns at most 100 operation rows with nested observations, ordered by UUID. Pagination traverses the retained rows visible to each read, not a multi-call frozen snapshot; restart from the first page when reconciling newly registered operations. It is internal and unmounted, with no public receipt route. | ||
|
|
||
| ## Retention and qualification | ||
|
|
||
| No TTL, pruning job, worker update/delete port or cascading parent deletion. Retain operations, observations and continuation history for the Pack lifetime, including after settlement/retry. Parent deletion is RESTRICT; any owner-data deletion requires a separately authorized administrative process. Growth is bounded per approved paid attempt, not falsely bounded per owner lifetime. Access revocation hides retained history without erasing it. | ||
|
|
||
| Actual proposal tests register operations and continuation grants through the real fenced command, append via a separate service-role transaction, and then deliver an actual refused stale write. They check unchanged Pack/wallet/usage snapshots, bounded concurrent conflicts, owner/claim mismatch, pagination, redaction, privilege denial and same-attempt continuation. The paired API integration uses the same proposal files. These are disposable local PostgreSQL checks, not seeded-history-only mechanics, hosted qualification or live provider evidence. | ||
|
|
||
| Append accepts READ COMMITTED transactions only; higher isolation levels are refused before journal mutation. Pagination uses the owner/job/operation-ID index. Accepted-slot conflicts hash the original accepted payload, including after completion. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.