Repository navigation
Propose exact retained-stream Context evidence binding (#2207) #127
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
c240f5d
Propose retained stream evidence binding with disposable lock proof
sidneyswift 8a286c2
Recheck lineage after waiting for result authority lock
sidneyswift 24f2c41
Refuse recreated withdrawn source scopes and state lock budget limits
sidneyswift 17d7622
Align retained evidence proof with adopted Catalog source contract
sidneyswift 6560ca7
Avoid persisting checkout credentials in proposal CI
sidneyswift File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| name: Retained stream binding proposal (offline only) | ||
| on: | ||
| pull_request: | ||
| branches: [main] | ||
| paths: | ||
| - "proposals/stream-evidence-binding-2207/**" | ||
| - "tests/test_release_cases.py" | ||
| - "supabase/migrations/**" | ||
| - ".github/workflows/stream-binding-proposal.yml" | ||
| permissions: | ||
| contents: read | ||
| jobs: | ||
| disposable-postgres: | ||
| runs-on: ubuntu-24.04 | ||
| timeout-minutes: 20 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| postgres: [15, 17] | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| timeout-minutes: 1 | ||
| with: | ||
| persist-credentials: false | ||
| - name: Install PostgreSQL | ||
| timeout-minutes: 12 | ||
| run: | | ||
| sudo apt-get update | ||
| sudo apt-get install -y postgresql-common | ||
| sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y | ||
| sudo apt-get update | ||
| sudo apt-get install -y postgresql-${{ matrix.postgres }} | ||
| - name: Qualify retained evidence joins and locks | ||
| timeout-minutes: 3 | ||
| run: | | ||
| export PATH="/usr/lib/postgresql/${{ matrix.postgres }}/bin:$PATH" | ||
| python3 proposals/stream-evidence-binding-2207/test_binding.py | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,179 @@ | ||
| # Retained Catalog stream → Context evidence proposal (#2207) | ||
|
|
||
| Status: source/design qualification, 2026-10-27. Aligned to the Catalog source | ||
| contract adopted by root from API #1187, head | ||
| `b185ea9ded070cd6d621d2bf13843bf9df6e5a8f`, merge | ||
| `08f62b7474e52b2bd0114d2d2c62c642c17ef6bf`. | ||
| [Canonical Catalog contract](https://github.com/recoupable/api/blob/08f62b7474e52b2bd0114d2d2c62c642c17ef6bf/docs/contracts/catalog-stream-evidence-2207/README.md) | ||
| and its 31 synthetic vectors govern source meaning. Earlier versions of this | ||
| proposal's dynamic topic, different locator and 62-date limit are superseded. | ||
|
|
||
| No migration, production writer, service grant, provider call, ingestion service or | ||
| live adapter. `proposal.sql` is a private, default-denied **evidence read/lock proof**. | ||
| Its fixture inserts are not a retention writer. It does not authorize derived sums, | ||
| growth or comparison fields: exact bundle equality is not a result-field validator. | ||
| Root and Catalog review the revised exact heads before any further implementation. | ||
|
|
||
| ## Existing source authority | ||
|
|
||
| Database base `6ae03a0cc71dc2ad3955594c9cfcb6156c3fb8b8`; API inspection base | ||
| `c1298deb6bb981543bf442d26a32def889e60d71`, with adopted Catalog contract above. | ||
| Relevant Database migrations under `supabase/migrations/`: | ||
|
|
||
| - `20260920010000_context_foundation.sql`: sources, versions, attempts, results, | ||
| current documents, `accept_context_result`, `withdraw_context_source`. | ||
| - `20260920020000_context_spotify_pipeline.sql` and current metadata replacement | ||
| `20260920040000_context_guests.sql`: Spotify-specific writer; do not reuse here. | ||
| - `20260922070000_context_catalog_entry.sql` and | ||
| `20260923080000_context_execution_records.sql`: owned catalog requests, explicit | ||
| expansion edges and exact `resolve_context_catalog_member_recording`. | ||
| - `20261008160000_context_release_cases.sql`: actor/workspace authorization. | ||
| - `20261009193000_catalog_daily_stream_tracking.sql` and | ||
| `20261010040000_catalog_stream_parent_lock_order.sql`: retained rows, latest-day | ||
| ordering and supported collection lock order. | ||
|
|
||
| A global recording UUID is not evidence authority. Require the exact existing | ||
| expanded member of an owned partial/completed request; current actor/workspace, | ||
| workspace/catalog and catalog/ISRC memberships; exact Recoup catalog resource and | ||
| recording binding. Expanded recordings are outside `request.output.subjectIds`. | ||
| The reader never expands, invents a UUID, collects or creates a recording. | ||
|
|
||
| ## Adopted series, topic and current-document behavior | ||
|
|
||
| Use existing Context tables. One owned source per catalog/recording/metric scope: | ||
|
|
||
| `urn:recoup:retained-catalog-streams:<catalog UUID>:<recording UUID>:all_dsps:worldwide` | ||
|
|
||
| The source kind is `provider_metadata`; the locator is internal, not a provider URL | ||
| or fetch target. Provider recording ID is version content, never part of this key. | ||
| The stable topic is **`catalog_retained_streams`**. Existing document uniqueness | ||
| therefore gives **one current document per owner/recording/topic across catalogs | ||
| and windows**. A newly accepted catalog/window replaces via revision CAS, making | ||
| old bindings stale. No dynamic query topics or simultaneous-current-window promise. | ||
|
|
||
| One version contains the full bounded tuple bundle, and one accepted observation | ||
| result links exactly that version. This keeps all 732 supported dates within one | ||
| of ObjectRef's 32 allowed version IDs. Attempt provider `luminate`, recipe | ||
| `retained-luminate-daily-v1`, schema `1` identify truthful local binding work. They | ||
| must not imply a provider request, paid collection or confirmed historical cost. | ||
|
|
||
| The strict evidence content is reconstructed by the proof, with no additional keys: | ||
|
|
||
| - `schemaVersion: 1`, `workspaceId`, `catalogId`, `subjectId`, `isrc`. | ||
| - `provider: luminate`, `metric: daily_streams`, `platform: all_dsps`, | ||
| `territory: worldwide`, `timezone: UTC`, `providerRecordingId`. | ||
| - `since` inclusive and `until` exclusive span two equal completed UTC periods, | ||
| each 1–366 days (2–732 total dates). `periods` matches Catalog's previous/current | ||
| start/end_exclusive, days and timezone shape. | ||
| - Sorted latest-per-date `observations`, with all eight fields: | ||
| `(catalog_id,isrc,date,run_id,provider_recording_id,source_hash,retrieved_at,streams)`. | ||
| - Sorted `absentDates` and `nullDates`, kept separate; no synthetic absent tuple. | ||
| - `retrievalRange: {earliest,latest}` from included retained observations. | ||
|
|
||
| Latest selection is exactly `retrieved_at DESC, run_id DESC`. Select the full set, | ||
| then reject invalid lineage; never filter out a bad latest row to choose an older | ||
| or convenient provider identity. Every row must join its run's catalog and fall | ||
| within that run's **inclusive** since/until range. Collection runs remain at most | ||
| 62 dates; a comparison bundle can span many runs. Final run status does not erase | ||
| earlier committed rows. No-row series refuses; zero and stored null remain distinct. | ||
| Mixed provider IDs, blank IDs, non-64-hex hashes and nonfinite timestamps refuse. | ||
| Accepted/stale/withdrawn results on the same source with a different retained | ||
| provider ID also refuse pending an owning identity-switch disposition. | ||
|
|
||
| `context_source_versions.retrieved_at` equals the **maximum included retained | ||
| retrieval time**; the range remains in content. Binding time belongs in attempt/ | ||
| result provenance. Published/source-updated time remains unknown. No fake freshness. | ||
| A future writer computes SHA256 of normalized UTC PostgreSQL JSONB text for internal | ||
| version deduplication. It is distinct from Luminate's raw payload hash and the | ||
| JavaScript surface hash; never claim cross-runtime serialization equality. | ||
|
|
||
| ## Explicit size limits and field boundary | ||
|
|
||
| Proof limits: evidence content at most **524,288 bytes (512 KiB)** in both UTF-8 | ||
| PostgreSQL JSONB text and `pg_column_size(JSONB)`; returned evidence envelope at most | ||
| **786,432 bytes (768 KiB)** by the same two checks. Reject overflow, never truncate | ||
| rows, versions or strings. The 732-date test spans 12 runs and fits these limits; | ||
| large provider text may still make a shorter bundle refuse. | ||
|
|
||
| These evidence limits do not enlarge the existing durable command/source-part caps. | ||
| A future API must separately measure the actual outgoing serialized UTF-8 bytes, | ||
| and emit only a bounded field projection plus exact references into the descriptor. | ||
| Do not copy a large evidence bundle into a capped chat data part or saved controls. | ||
| The full evidence read envelope is not a durable mutation command. | ||
|
|
||
| The proof requires `normalized_response = version.content` to qualify **evidence | ||
| only**. It does not qualify complete-period sums/growth, percentage rounding, | ||
| zero-baseline behavior, safe-total overflow or which descriptor field is supported. | ||
| Those need the strict writer/result-field validator using existing | ||
| `compareSpotifyDailyStreams` semantics before any field ObjectRef is enabled. | ||
| Incomplete evidence may be retained without authorizing invented numeric totals. | ||
|
|
||
| ## Proposed retention transaction (not implemented) | ||
|
|
||
| 1. Derive actor/owner from trusted authentication; strictly validate exact request, | ||
| subject, catalog, periods, expected revision and observed tuple set. The descriptor | ||
| cannot supply authoritative source content, provider assertions or owner IDs. | ||
| 2. Hold durable session/chat/fence locks first when composing with a mutation. | ||
| Then owner account → catalog → song KEY SHARE; tracking UPDATE; actor membership, | ||
| workspace/catalog, catalog/song, request/edge/resource/subjects SHARE. Recheck the | ||
| exact expanded-member resolver. Require tracking owner, not enabled collection. | ||
| 3. Load full current tuples under that boundary; validate all run/identity/shape/ | ||
| size and source-meaning checks. Compare expected tuples so a correction requires | ||
| explicit rebuild/reselection. Preserve all nulls/absences. Refuse provider switches | ||
| without an adopted source-owner disposition. Preserve explicit customer-assertion | ||
| corrections; no provider projection silently overwrites them. | ||
| 4. Find all matching source locators, including withdrawn history, under tracking | ||
| serialization. Refuse withdrawn/duplicate/ambiguous scope; never recreate an | ||
| active source to evade withdrawal. Lock source before result/document. Reserve | ||
| this locator/topic against generic writer paths before activation. | ||
| 5. Insert/reuse exact immutable version with retained retrieval max. Create a truthful | ||
| local attempt/result and exactly one lineage link. Validate derived fields before | ||
| field authorization. Use expected document revision and `accept_context_result`; | ||
| failure rolls back all inserts. Identical current replay can return its exact | ||
| receipt; an old receipt cannot restore an old current result. Preserve exact | ||
| originating request on reuse, or create an explicitly new result. | ||
| 6. Return exact result/topic/revision/version receipt. Unknown acknowledgment requires | ||
| receipt reconciliation, not provider retry or implicit collection. | ||
|
|
||
| Concurrent first creation, CAS/retry/rollback and receipt behavior are not implemented | ||
| or proven by direct fixture inserts. They remain required before runtime work. | ||
|
|
||
| ## Read/lock qualification and withdrawal | ||
|
|
||
| `read_binding` requires READ COMMITTED. Otherwise an earlier repeatable-read snapshot | ||
| could miss a correction committed before the tracking lock. It authorizes exact | ||
| actor/member/request/subject and current result/revision/source/version, reconstructs | ||
| the full evidence bundle and compares retained tuples. It returns redacted JSON null | ||
| for unavailable/stale/malformed evidence; operational SQL errors remain errors. | ||
|
|
||
| Source/version locks precede result/document, matching withdrawal. Result UPDATE | ||
| blocks new lineage FK links; existing links are SHARE locked. A new statement | ||
| rechecks lineage count after result-lock acquisition, covering an insertion committed | ||
| while the reader waited. Existing run/observation rows are SHARE locked against | ||
| update/delete. The supported stream commit serializes on tracking. No source→chat | ||
| callback or inverted chat lock acquisition is introduced. | ||
|
|
||
| Withdrawal uses existing `withdraw_context_source`: all versions/results for that | ||
| owned series become unavailable, current pointers clear and revisions advance. | ||
| Historical withdrawn locator rows also prevent a replacement active source passing | ||
| the reader. Collection disable is not withdrawal. Membership revocation removes | ||
| current access without deleting retained rows/global identity. No provider retention | ||
| policy, automatic republication or erasure of prior chat/export copies is implied. | ||
|
|
||
| ## Qualification, limitations and next action | ||
|
|
||
| Run `python3 proposals/stream-evidence-binding-2207/test_binding.py` with PostgreSQL | ||
| on PATH. It creates a disposable Unix-socket-only cluster, applies actual existing | ||
| migrations and uses explicitly synthetic parents/evidence. No hosted DSN is read. | ||
| The revised suite has 25 tests; see `VECTOR-COVERAGE.md` for Catalog vector mapping. | ||
|
|
||
| The 732-date cap bounds output dates, **not retained correction history or lock | ||
| footprint**. Arbitrary direct inserts bypassing tracking are not serialized by this | ||
| proof. Runtime work needs participating-writer enforcement, a row/work budget and | ||
| fail-closed overflow behavior. No production grants/RLS delivery, all cascade | ||
| interleavings, production writer idempotency/CAS/rollback, actual API transport byte | ||
| validation, derived field validation or durable chat composition is qualified. | ||
|
|
||
| Next: Catalog independently reviews alignment to adopted #1187; root reviews revised | ||
| exact heads and fresh CI. Only after exact protocol adoption may the executable | ||
| retention writer be built and qualified. Source merge alone activates nothing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # Catalog vector port map | ||
|
|
||
| Source: API #1187 `b185ea9ded070cd6d621d2bf13843bf9df6e5a8f`, adopted merge | ||
| `08f62b7474e52b2bd0114d2d2c62c642c17ef6bf`, | ||
| [all 31 vectors](https://github.com/recoupable/api/blob/08f62b7474e52b2bd0114d2d2c62c642c17ef6bf/docs/contracts/catalog-stream-evidence-2207/vectors.json). | ||
| Every live mapping remains unsupported. These ports qualify evidence only; they | ||
| never claim that the vectors' comparison outputs or production intake are shipped. | ||
|
|
||
| | Catalog vector IDs | Executable proof / remaining boundary | | ||
| | ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | exact_bound_bundle; disabled_tracking_retains_evidence | Exact expanded member, stable source/version/result; disable remains readable. | | ||
| | known_zero_baseline; null_correction_bound; absent_date_bound; all_stored_nulls | Null/zero/missing evidence kept distinct; comparison arithmetic remains unqualified. | | ||
| | new_correction_bound; old_version_after_correction; correction_not_yet_context_bound | Explicit corrected version accepted with existing CAS; old tuple/version refuses. | | ||
| | mixed_provider_ids; all_dates_changed_provider | Full matching tuple set refuses mixed identity; new result differing from earlier accepted source identity refuses. | | ||
| | workspace_revoked; member_removed; unresolved_recording | Actor/membership/expanded-edge refusal, including real competing revocation locks. | | ||
| | cross_owner_version; unrelated_metadata_version; forged_hash_only | Wrong owner/version, missing/extra lineage and exact source scope refuse. UUIDs remain stored references, never hashes. | | ||
| | unsupported_field_relabel | Wrong metric/scope and extra content refuse. No comparison/financial field authorization exists. | | ||
| | source_withdrawn; version_removed; attempted_recreation_after_withdrawal | Existing withdrawal clears pointer; removed version and active replacement with withdrawn locator history refuse. | | ||
| | different_current_window | New window replaces one current document; old binding stale. Additional cross-catalog replacement proof enforces same stable topic/document. | | ||
| | unstable_chunked_read; arbitrary_caller_tuple | Exact retained tuple mismatch, malformed run/date lineage and bad retained fields refuse. Direct phantom inserts remain outside cooperative writer proof. | | ||
| | result_not_accepted | Stale result refuses. | | ||
| | customer_correction_blocks_refresh | Reader refuses customer_assertion as provider observation; preservation by the future retention writer is not proven. | | ||
| | entirely_absent | Reader requires at least one retained row; broader no-row intake behavior remains a writer obligation. | | ||
| | identical_collection_no_new_rows | Unchanged retained tuples stay identical by construction; production dedup/replay not implemented. | | ||
| | late_writer_cas_lost | Existing CAS used by fixture acceptance; concurrent production retention writer not implemented. | | ||
| | unsafe_retained_count | Existing table CHECK constrains rows; derived safe-total overflow and full ingestion validator remain unqualified. | | ||
| | storage_failure | Operational errors propagate; real transport failure behavior is not implemented by this SQL-only proof. | | ||
|
|
||
| Additional executable boundaries: 12-run/732-date snapshot, >732 refusal, retained | ||
| retrieval max, malformed hash/provider/timestamp/run range, multi-byte UTF-8 overflow, | ||
| READ COMMITTED requirement, mutation locks and reverse-order lineage insertion. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.