Repository navigation
Propose genuine retained-field selection and bounded consumer #1291
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
4ca23de
Propose genuine retained-field selection and bounded consumer
sidneyswift a7ba451
Classify retained-field authority size refusals as invalid
sidneyswift 2078498
Preserve operational refusals and clarify retained authority boundary
sidneyswift af21d00
Pin retained projection reference and source literal types
sidneyswift File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| # Retained-field selection and inner consumer — inactive source proposal | ||
|
|
||
| This implements the existing `catalog-field-consumer-2207.md` inner contract only. | ||
| It is outside app/lib and routes. Missing authority ports return unavailable. | ||
| No SQL wrapper, grant, source write, Context identity creation, environment flag, | ||
| UI schema-v1 extension or outer wire version is supplied. Frozen raw-report | ||
| API1290/Database142 are a different source path and are unchanged. | ||
|
|
||
| `readRetainedFieldSelection` calls an injected current command reconciler with the | ||
| original authenticated actor, workspace, request, catalog and retention key. The | ||
| intended backing function is the private, source-only Database proposal | ||
| [`stream_binding_proposal.reconcile_retention`](https://github.com/recoupable/database/blob/1b892dd2671ba7e15e0c44b690fbfbe515b59fbd/proposals/stream-retention-writer-2207/writer.sql#L3), | ||
| which rechecks `read_binding`. It is not implemented or connected as a live API | ||
| transport here; the injected callback alone supplies no authentication proof. | ||
| The adapter validates the exact receipt/scope/key and projects its | ||
| genuine Context identities. Null, pending, malformed, mismatched or ambiguous | ||
| results cannot create a ready selection; operational errors are redacted errors. | ||
| No retain/expand/provider call, latest lookup, re-key or raw-report fallback exists. | ||
| This projection supplies a selection, not a currentness lease. | ||
|
|
||
| `consumeRetainedField` accepts only the explicit retained_field request and server | ||
| actor/session/chat/assistant context. The injected transaction port must authorize | ||
| that complete scope, pre-lock the whole batch in a qualified stable order, read | ||
| each exact selection using the existing source family, and keep locks through | ||
| callback evaluation. No production port is implemented here. Passing mocked data | ||
| to that trusted port does not prove source authorization. The actor is never | ||
| substituted with owner/workspace. Maximum eight unique subjects; all-or-nothing | ||
| refusal, exact registered method/query, current_streams only, 32KiB output cap. | ||
| It calls the existing pinned evaluator, then strips raw observations and identifiers | ||
| outside the inner projection contract. Unknown has no amount key; zero stays zero. | ||
| Caller order is preserved; the port receives a separate subject-sorted lock list. | ||
|
|
||
| The generated `fixtures/fieldContext.json` is the ready inner response without | ||
| `state`, as specified by the consumer contract. It is synthetic evidence for UI's | ||
| explicitly versioned offline envelope; no outer version is invented here. UI owns | ||
| projection/collection/text agreement and capability negotiation. This increment | ||
| returns typed values but does not implement a renderer or any disclosure path. | ||
|
|
||
| Pure tests run the actual selection adapter and evaluator with synthetic authority | ||
| ports, including withheld values on stale/missing/withdrawn boundary simulations. | ||
| The Database companion extends the actual existing retained-field integration | ||
| harness with a deferred current reconcile/read consumer. Those PostgreSQL scenarios | ||
| are not run under the SQL hold. Simulated refusal labels do not prove actual | ||
| withdrawal, memberships, transaction liveness or concurrent batch correctness. | ||
|
|
||
| Remaining runtime gates: real selection-source transport, actor/session/chat scope, | ||
| reserved-family writer enforcement, callable held-transaction evaluator transport, | ||
| qualified batch lock order, UI-owned envelope, and current authorization on initial | ||
| delivery/replay/persistence/history/reconnect/text/export/model reads. A successful | ||
| journaled preparation cannot authorize later disclosure; deny both typed values and | ||
| numeric/text projections together. No recomputation or new version under the same | ||
| saved source revision. The ordinary report stays separate and is never a fallback. |
8 changes: 8 additions & 0 deletions
8
scripts/proposals/catalog-retained-selection/classifyRetainedFieldRefusal.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| /** Preserve contract failure classes while exposing no source details or values. */ | ||
| export function classifyRetainedFieldRefusal(reason: string): { | ||
| state: "invalid" | "error" | "unavailable"; | ||
| } { | ||
| if (reason === "size_limit") return { state: "invalid" }; | ||
| if (reason === "helper_changed" || reason === "source_error") return { state: "error" }; | ||
| return { state: "unavailable" }; | ||
| } |
93 changes: 93 additions & 0 deletions
93
scripts/proposals/catalog-retained-selection/consumeRetainedField.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| import { z } from "zod"; | ||
| import { evaluateRetainedStreamField } from "../catalog-stream-fields/evaluate"; | ||
| import { actorScopeSchema, requestSchema } from "./schemas"; | ||
| import type { Item, Ready, Result, LockedRead } from "./retainedFieldTypes"; | ||
| import { classifyRetainedFieldRefusal } from "./classifyRetainedFieldRefusal"; | ||
| /** Inactive source consumer; no report fallback, disclosure route, transport or UI schema change. */ | ||
| export async function consumeRetainedField( | ||
| actorInput: unknown, | ||
| input: unknown, | ||
| withLockedRead?: LockedRead, | ||
| ): Promise<Result> { | ||
| const actor = actorScopeSchema.safeParse(actorInput); | ||
| const request = requestSchema.safeParse(input); | ||
| if (!actor.success || !request.success) return { state: "invalid" }; | ||
| if (!withLockedRead) return { state: "unavailable" }; | ||
| const q = request.data; | ||
| try { | ||
| return await withLockedRead( | ||
| { | ||
| ...actor.data, | ||
| workspaceId: q.workspaceId, | ||
| catalogId: q.catalogId, | ||
| selections: [...q.selections].sort((a, b) => a.subjectId.localeCompare(b.subjectId)), | ||
| }, | ||
| async tx => { | ||
| if ( | ||
| !z.iso.datetime({ offset: true }).safeParse(tx.asOf).success || | ||
| !Number.isFinite(Date.parse(tx.asOf)) | ||
| ) | ||
| return { state: "error" }; | ||
| const items: Item[] = []; | ||
| for (const selection of q.selections) { | ||
| const envelope = await tx.readBinding(selection); | ||
| if (envelope === null || envelope === undefined) return { state: "unavailable" }; | ||
| const result = evaluateRetainedStreamField({ | ||
| operation: "read_field", | ||
| recipe: q.method.recipe, | ||
| field: q.field, | ||
| selection: { ...selection, query: q.query }, | ||
| asOf: tx.asOf, | ||
| boundary: { state: "qualified", envelope }, | ||
| }); | ||
| if (result.state !== "ready") return classifyRetainedFieldRefusal(result.reason); | ||
| const comparisonState = result.coverage.comparisonState; | ||
| if (comparisonState !== "comparable" && comparisonState !== "incomplete") | ||
| return { state: "error" }; | ||
| const { authority, kind } = result.ref; | ||
| if (authority !== "context" || kind !== "recording") return { state: "error" }; | ||
| const amount = result.field.amount; | ||
| if (amount !== null && (!Number.isSafeInteger(amount) || amount < 0)) | ||
| return { state: "unavailable" }; | ||
| items.push({ | ||
| ref: { ...result.ref, authority, kind }, | ||
| binding: selection, | ||
| value: amount === null ? { state: "unknown" } : { state: "known", amount }, | ||
| periods: result.periods, | ||
| coverage: { | ||
| comparisonState, | ||
| absentDateCount: result.coverage.absentDates.length, | ||
| nullDateCount: result.coverage.nullDates.length, | ||
| missingDateCount: result.coverage.missingDays.length, | ||
| }, | ||
| source: { | ||
| provider: result.provenance.provider, | ||
| metric: result.provenance.metric, | ||
| platform: result.provenance.platform, | ||
| territory: result.provenance.territory, | ||
| timezone: result.provenance.timezone, | ||
| retrievalRange: result.provenance.retrievalRange, | ||
| }, | ||
| }); | ||
| } | ||
| const ready: Ready = { | ||
| state: "ready", | ||
| contract: "catalog-retained-field-projection/1", | ||
| workspaceId: q.workspaceId, | ||
| catalogId: q.catalogId, | ||
| field: q.field, | ||
| method: q.method, | ||
| query: q.query, | ||
| asOf: tx.asOf, | ||
| semantics: "historical_snapshot", | ||
| items, | ||
| }; | ||
| return Buffer.byteLength(JSON.stringify(ready), "utf8") > 32768 | ||
| ? { state: "invalid" } | ||
| : ready; | ||
| }, | ||
| ); | ||
| } catch { | ||
| return { state: "error" }; | ||
| } | ||
| } | ||
77 changes: 77 additions & 0 deletions
77
scripts/proposals/catalog-retained-selection/fixtures/fieldContext.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| { | ||
| "contract": "catalog-retained-field-projection/1", | ||
| "workspaceId": "10000000-0000-4000-8000-000000000001", | ||
| "catalogId": "10000000-0000-4000-8000-000000000003", | ||
| "field": "current_streams", | ||
| "method": { | ||
| "recipe": "catalog-retained-stream-fields/1", | ||
| "recipeHash": "65a06a785aa496e1232fe3a9f1606595f19c1a152b710ca30c4c06612f01864a", | ||
| "evaluatorSha256": "163563e15dd92d473aa51899c54c96300f20d27072b83e39ffb01c84f3adc64e", | ||
| "helperSha256": "91dcca66c8dec0851c1f6e110257a63b730e9b7ae33a60bceb0e9ffcf318dd81" | ||
| }, | ||
| "query": { | ||
| "since": "2026-10-24", | ||
| "days": 1, | ||
| "timezone": "UTC" | ||
| }, | ||
| "asOf": "2026-10-27T00:00:00Z", | ||
| "semantics": "historical_snapshot", | ||
| "items": [ | ||
| { | ||
| "ref": { | ||
| "workspaceId": "10000000-0000-4000-8000-000000000001", | ||
| "authority": "context", | ||
| "kind": "recording", | ||
| "id": "10000000-0000-4000-8000-000000000004", | ||
| "sourceVersionIds": [ | ||
| "10000000-0000-4000-8000-000000000006" | ||
| ] | ||
| }, | ||
| "binding": { | ||
| "ownerId": "10000000-0000-4000-8000-000000000001", | ||
| "requestId": "10000000-0000-4000-8000-000000000002", | ||
| "subjectId": "10000000-0000-4000-8000-000000000004", | ||
| "catalogId": "10000000-0000-4000-8000-000000000003", | ||
| "resultId": "10000000-0000-4000-8000-000000000005", | ||
| "revision": 7, | ||
| "topic": "catalog_retained_streams", | ||
| "sourceVersionIds": [ | ||
| "10000000-0000-4000-8000-000000000006" | ||
| ] | ||
| }, | ||
| "value": { | ||
| "state": "known", | ||
| "amount": 100 | ||
| }, | ||
| "periods": { | ||
| "previous": { | ||
| "start": "2026-10-23", | ||
| "end_exclusive": "2026-10-24" | ||
| }, | ||
| "current": { | ||
| "start": "2026-10-24", | ||
| "end_exclusive": "2026-10-25" | ||
| }, | ||
| "days": 1, | ||
| "timezone": "UTC" | ||
| }, | ||
| "coverage": { | ||
| "comparisonState": "comparable", | ||
| "absentDateCount": 0, | ||
| "nullDateCount": 0, | ||
| "missingDateCount": 0 | ||
| }, | ||
| "source": { | ||
| "provider": "luminate", | ||
| "metric": "daily_streams", | ||
| "platform": "all_dsps", | ||
| "territory": "worldwide", | ||
| "timezone": "UTC", | ||
| "retrievalRange": { | ||
| "earliest": "2026-10-25T01:00:00Z", | ||
| "latest": "2026-10-25T01:00:00Z" | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } |
99 changes: 99 additions & 0 deletions
99
scripts/proposals/catalog-retained-selection/fixtures/synthetic-authority-input.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| { | ||
| "scope": { | ||
| "actorId": "10000000-0000-4000-8000-000000000009", | ||
| "workspaceId": "10000000-0000-4000-8000-000000000001", | ||
| "requestId": "10000000-0000-4000-8000-000000000002", | ||
| "catalogId": "10000000-0000-4000-8000-000000000003", | ||
| "key": "original" | ||
| }, | ||
| "actor": { | ||
| "actorId": "10000000-0000-4000-8000-000000000009", | ||
| "sessionId": "session", | ||
| "chatId": "chat", | ||
| "messageId": "assistant" | ||
| }, | ||
| "receipt": { | ||
| "ownerId": "10000000-0000-4000-8000-000000000001", | ||
| "requestId": "10000000-0000-4000-8000-000000000002", | ||
| "subjectId": "10000000-0000-4000-8000-000000000004", | ||
| "catalogId": "10000000-0000-4000-8000-000000000003", | ||
| "key": "original", | ||
| "commandDigest": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", | ||
| "resultId": "10000000-0000-4000-8000-000000000005", | ||
| "sourceVersionId": "10000000-0000-4000-8000-000000000006", | ||
| "revision": 7 | ||
| }, | ||
| "envelope": { | ||
| "ownerId": "10000000-0000-4000-8000-000000000001", | ||
| "requestId": "10000000-0000-4000-8000-000000000002", | ||
| "subjectId": "10000000-0000-4000-8000-000000000004", | ||
| "catalogId": "10000000-0000-4000-8000-000000000003", | ||
| "resultId": "10000000-0000-4000-8000-000000000005", | ||
| "revision": 7, | ||
| "topic": "catalog_retained_streams", | ||
| "sourceVersionIds": [ | ||
| "10000000-0000-4000-8000-000000000006" | ||
| ], | ||
| "evidence": { | ||
| "schemaVersion": 1, | ||
| "workspaceId": "10000000-0000-4000-8000-000000000001", | ||
| "catalogId": "10000000-0000-4000-8000-000000000003", | ||
| "subjectId": "10000000-0000-4000-8000-000000000004", | ||
| "isrc": "USAAA2600001", | ||
| "provider": "luminate", | ||
| "metric": "daily_streams", | ||
| "platform": "all_dsps", | ||
| "territory": "worldwide", | ||
| "timezone": "UTC", | ||
| "since": "2026-10-23", | ||
| "until": "2026-10-25", | ||
| "periods": { | ||
| "previous": { | ||
| "start": "2026-10-23", | ||
| "end_exclusive": "2026-10-24" | ||
| }, | ||
| "current": { | ||
| "start": "2026-10-24", | ||
| "end_exclusive": "2026-10-25" | ||
| }, | ||
| "days": 1, | ||
| "timezone": "UTC" | ||
| }, | ||
| "providerRecordingId": "provider-original", | ||
| "observations": [ | ||
| { | ||
| "catalog_id": "10000000-0000-4000-8000-000000000003", | ||
| "isrc": "USAAA2600001", | ||
| "date": "2026-10-23", | ||
| "run_id": "10000000-0000-4000-8000-000000000007", | ||
| "provider_recording_id": "provider-original", | ||
| "streams": 50, | ||
| "source_hash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", | ||
| "retrieved_at": "2026-10-25T01:00:00Z" | ||
| }, | ||
| { | ||
| "catalog_id": "10000000-0000-4000-8000-000000000003", | ||
| "isrc": "USAAA2600001", | ||
| "date": "2026-10-24", | ||
| "run_id": "10000000-0000-4000-8000-000000000007", | ||
| "provider_recording_id": "provider-original", | ||
| "streams": 100, | ||
| "source_hash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", | ||
| "retrieved_at": "2026-10-25T01:00:00Z" | ||
| } | ||
| ], | ||
| "absentDates": [], | ||
| "nullDates": [], | ||
| "retrievalRange": { | ||
| "earliest": "2026-10-25T01:00:00Z", | ||
| "latest": "2026-10-25T01:00:00Z" | ||
| } | ||
| } | ||
| }, | ||
| "query": { | ||
| "since": "2026-10-24", | ||
| "days": 1, | ||
| "timezone": "UTC" | ||
| }, | ||
| "asOf": "2026-10-27T00:00:00Z" | ||
| } |
39 changes: 39 additions & 0 deletions
39
scripts/proposals/catalog-retained-selection/readRetainedFieldSelection.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| import { commandScopeSchema, receiptSchema, type CommandScope, type Selection } from "./schemas"; | ||
| /** Proposal only. Reconcile is the existing current-authorized command read, not a client receipt. */ | ||
| export async function readRetainedFieldSelection( | ||
| input: unknown, | ||
| reconcile?: (scope: CommandScope) => Promise<unknown>, | ||
| ): Promise< | ||
| { state: "ready"; selection: Selection } | { state: "invalid" | "unavailable" | "error" } | ||
| > { | ||
| const scope = commandScopeSchema.safeParse(input); | ||
| if (!scope.success) return { state: "invalid" }; | ||
| if (!reconcile) return { state: "unavailable" }; | ||
| try { | ||
| const parsed = receiptSchema.safeParse(await reconcile(scope.data)); | ||
| if (!parsed.success) return { state: "unavailable" }; | ||
| const r = parsed.data; | ||
| if ( | ||
| r.ownerId !== scope.data.workspaceId || | ||
| r.requestId !== scope.data.requestId || | ||
| r.catalogId !== scope.data.catalogId || | ||
| r.key !== scope.data.key | ||
| ) | ||
| return { state: "unavailable" }; | ||
| return { | ||
| state: "ready", | ||
| selection: { | ||
| ownerId: r.ownerId, | ||
| requestId: r.requestId, | ||
| subjectId: r.subjectId, | ||
| catalogId: r.catalogId, | ||
| resultId: r.resultId, | ||
| revision: r.revision, | ||
| topic: "catalog_retained_streams", | ||
| sourceVersionIds: [r.sourceVersionId], | ||
| }, | ||
| }; | ||
| } catch { | ||
| return { state: "error" }; | ||
| } | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.