Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions proposals/retained-field-integration-2207/PROTOCOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,3 +92,94 @@ than inventing per-field object IDs. App part 256 KiB, controls 64 KiB and comma
128 KiB limits remain unchanged; a 768 KiB evidence envelope is not a chat payload.
Do not copy all retained tuples into saved controls or equate document revision with
surface revision. This suite does not prove deployed behavior or provider execution.

## Genuine receipt selection consumer (source-only increment, 2026-11-19)

`selection-source.json` pins the separate API proposal consumer. The original
adopted evaluator checkout and all three original method hashes remain independently
required by `test_integration.py`; they are not replaced or widened. The added
consumer reads an existing current `reconcile_retention` receipt, projects its exact
Context selection, rechecks the original selection via `read_binding`, and evaluates
inside the original held transaction. No raw report, new authority store, latest
upgrade, implicit retain/expand or actor substitution is used. No SQL definition,
grant, route or production transport is introduced.

`test_selection_transport.py` builds the exact new API consumer plus its unchanged
real evaluator and checks the actual generated inner fieldContext fixture. These
five runnable **pure synthetic transport** tests do not initialize PostgreSQL or
claim authenticated source authority. Use the frozen source checkout with its
existing dependencies:

```sh
CATALOG_SELECTION_SOURCE=/path/to/frozen/api python3 proposals/retained-field-integration-2207/test_selection_transport.py
```

`test_selection_consumer.py` extends the existing actual database harness with five
new deferred scenarios: current genuine receipt → exact inner response, missing key
and wrong actor, withdrawn/removed/member/expanded-edge loss, superseded result,
and supported stream correction refusing the original selection. It calls actual
reconciliation and source reads; no new permissive SQL adapter is installed. The
SQL cases default to skipped. The focused local qualification below subsequently
executed all five; later runs must explicitly set `CATALOG_SELECTION_SQL_QUALIFICATION=1`;
that also requires `CATALOG_API_SOURCE` at the original adopted evaluator pin and
`CATALOG_SELECTION_SOURCE` at the new companion pin. The five sequential cases do
not qualify concurrent max-eight batch locking or production transaction transport.

The synthetic actor/session/chat/assistant strings in this harness are test context,
not session authentication proof. Returning a real selection also is not a lease:
the next read reauthorizes it. `fieldContext` is exactly the inner projection response
without `state`; UI owns its explicitly versioned offline envelope. Schema-v1 is
unchanged. No initial-send/replay/history/text/export/model disclosure path is enabled;
these paths must jointly withhold projections on current denial before live routing.

### Focused local qualification — 2026-11-19

All five new SelectionConsumer scenarios passed without skips against Database
`1185562def1bdbc49b70cdc7d889ba52b352f03b`, API consumer
`4ca23de6a6cea3226c2f0392a454eb57861aedf8` and original evaluator
`6ba799dbb67be284adcd6f8bcd8fcde2faa6c2b8`, using installed PostgreSQL 17.7.
Only those five tests ran; inherited unrelated suites were excluded. The default
PostgreSQL 14 server failed before startup because its ICU74 library was absent;
no tools were installed, and its temporary directory was removed.

The successful run cleared inherited connection/credential settings, initialized a
new synthetic temporary cluster, verified its postmaster/data-directory/socket
identity and empty listen_addresses before fixture SQL, and checked zero TCP
listeners. All schema/role/data writes stayed in that disposable database. Cleanup
verified the owned process, Unix socket, PID file and temporary directory were gone.
No hosted DSN, existing database, account credential, runtime route, production
apply/grant/enrollment or activation was used. This closes the five sequential
consumer scenarios only; batch concurrency, authenticated production transport and
publication/history disclosure qualification remain outstanding.

### Size-refusal successor — 2026-11-19

The companion API `a7ba45167d159428bfac90fa6cb2fa58df88d360` preserves the
registered evaluator's `size_limit` refusal as `{state:"invalid"}`. Missing/stale/
withdrawn/unauthorized bindings stay unavailable and operational failures stay
errors. Six pure compiled transport tests pass, including real UTF-8 inputs over
512 KiB and over 768 KiB that return only the invalid state. The inner UI fixture
and evaluator/recipe/helper bytes are unchanged. Selection-source pins now target
this successor.

The five real PostgreSQL17.7 passes above remain attributed to their exact original
API4ca23de6/Database1185562 sources. They were not rerun or relabelled as successor
qualification. No SQL semantics or actual SQL test scenario changed in this repair.

### Operational-refusal review successor — 2026-11-19

The companion pin now targets API `20784986cd08d8651545d71d024b98c223ca7866`.
`helper_changed` and `source_error` preserve the operational `error` state;
`size_limit` remains `invalid`, and ordinary denied bindings remain `unavailable`.
Every refusal withholds values and diagnostics. Coverage comparison state is a
closed comparable/incomplete union checked before projection. The API README
links the private Database proposal and distinguishes it from a connected live
API transport. Seven pure compiled transport tests pass, including an actual
helper-integrity mismatch in temporary bundle output, restored after the test.
The five actual PostgreSQL passes remain attributed only to the original heads
above; no SQL definitions or actual SQL scenarios changed or were rerun.

Final type follow-up: `selection-source.json` pins the successor with explicit
reference authority/kind and source literal types. The consumer checks reference
literals before projection. The seven compiled tests and unchanged fixture still
pass; original SQL qualification attribution remains unchanged.
35 changes: 35 additions & 0 deletions proposals/retained-field-integration-2207/build-selection.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
// Deferred fixture transport only. Build exact source consumer; no replacement evaluator.
import { createRequire } from 'node:module';
import { mkdirSync, copyFileSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';
const [source, output] = process.argv.slice(2);
const require = createRequire(join(source, 'package.json'));
const vite = require.resolve('vite', { paths: [require.resolve('vitest/package.json')] });
const { build } = require(require.resolve('esbuild', { paths: [vite] }));
const relative = 'scripts/proposals/catalog-retained-selection';
mkdirSync(join(output, relative), { recursive: true });
mkdirSync(join(output, 'lib/catalog'), { recursive: true });
copyFileSync(join(source, 'lib/catalog/compareSpotifyDailyStreams.ts'), join(output, 'lib/catalog/compareSpotifyDailyStreams.ts'));
for (const name of ['readRetainedFieldSelection', 'consumeRetainedField', 'schemas']) {
await build({ entryPoints: [join(source, relative, name + '.ts')], bundle: true,
platform: 'node', format: 'esm', target: 'node20', outfile: join(output, relative, name + '.mjs') });
}
// Bundled evaluator keeps its new URL('./recipe.json', import.meta.url) lookup.
copyFileSync(join(source, 'scripts/proposals/catalog-stream-fields/recipe.json'), join(output, relative, 'recipe.json'));
writeFileSync(join(output, 'selection-driver.mjs'), `
import { readFileSync } from 'node:fs';
import { readRetainedFieldSelection } from './${relative}/readRetainedFieldSelection.mjs';
import { consumeRetainedField } from './${relative}/consumeRetainedField.mjs';
import { method } from './${relative}/schemas.mjs';
const input = JSON.parse(readFileSync(0, 'utf8'));
// Receipts/envelopes below come from the real disposable SQL harness, not a runtime route.
const selected = await readRetainedFieldSelection(input.scope, async () => input.receipt);
if (selected.state !== 'ready') console.log(JSON.stringify(selected));
else {
const result = await consumeRetainedField(input.actor, {
mode:'retained_field', workspaceId:input.scope.workspaceId, catalogId:input.scope.catalogId,
field:'current_streams', query:input.query, method, selections:[selected.selection]
}, async (_scope, consume) => consume({asOf:input.asOf,readBinding:async () => input.envelope}));
console.log(JSON.stringify(result));
}
`);
12 changes: 12 additions & 0 deletions proposals/retained-field-integration-2207/selection-source.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"head": "af21d0037d3cb4b79b837a9c164890537a89a5cb",
"files": {
"scripts/proposals/catalog-retained-selection/readRetainedFieldSelection.ts": "1096979cc82d034ad08ddcb6db8a7e04a0c91dbe5a332ed3f40c98120e4c996a",
"scripts/proposals/catalog-retained-selection/consumeRetainedField.ts": "deeb2de2f28cd70b4f4926c3a4f83e2419b34d8b661b56cc6216e5fba6a71961",
"scripts/proposals/catalog-retained-selection/schemas.ts": "b5cd78e40bbdd9e83914bb729795706dd14d81988679512dc4a5d95943369e88",
"scripts/proposals/catalog-retained-selection/fixtures/fieldContext.json": "ed979fc2ea72c3a3e51deb2ba259980f171a550b88fb488afb783a70e02ec322",
"scripts/proposals/catalog-retained-selection/fixtures/synthetic-authority-input.json": "f498cff7bdef6154382a98815162d140e6660e14ace885e7bee8e56334fcd522",
"scripts/proposals/catalog-retained-selection/classifyRetainedFieldRefusal.ts": "9a50a90b7701ac21897fdcee44d5c4e60567b00c38d504daf14acc25b04a6d13",
"scripts/proposals/catalog-retained-selection/retainedFieldTypes.ts": "9c20910786eef5ef60ed5ad8d2e6505b5fa4be932e8f2e0d465c3e371d6ae229"
}
}
123 changes: 123 additions & 0 deletions proposals/retained-field-integration-2207/test_selection_consumer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
"""Deferred actual Context receipt -> selection -> locked reader -> inner consumer.
Default skipped. No PostgreSQL qualification is authorized by adding these tests.
"""
import hashlib
import json
import os
from pathlib import Path
import subprocess
import unittest

from test_integration import Integration, uid, FILES

HERE = Path(__file__).resolve().parent


@unittest.skipUnless(os.environ.get('CATALOG_SELECTION_SQL_QUALIFICATION') == '1',
'SQL hold: genuine selection scenarios are deferred')
class SelectionConsumer(Integration):
@classmethod
def setUpClass(cls):
# Original fixture still pins the original evaluator checkout at its adopted SHA.
cls.selection_source = Path(os.environ['CATALOG_SELECTION_SOURCE']).resolve()
cls.pins = json.loads((HERE / 'selection-source.json').read_text())
cls.verify_selection_source()
super().setUpClass()
cls.addClassCleanup(cls.verify_selection_source)
subprocess.run(['node', str(HERE / 'build-selection.mjs'),
str(cls.selection_source), cls.output.name], check=True)

@classmethod
def verify_selection_source(cls):
head = subprocess.check_output(['git', '-C', str(cls.selection_source),
'rev-parse', 'HEAD'], text=True).strip()
if head != cls.pins['head']:
raise AssertionError('Selection source head differs from frozen companion')
for name, digest in {**FILES, **cls.pins['files']}.items():
if hashlib.sha256((cls.selection_source / name).read_bytes()).hexdigest() != digest:
raise AssertionError('Selection source drift: ' + name)

def consume(self, receipt, envelope, actor=1, key='first'):
payload = {
'scope': {'actorId': uid(actor), 'workspaceId': uid(1), 'requestId': uid(4),
'catalogId': uid(2), 'key': key},
'actor': {'actorId': uid(actor), 'sessionId': 'synthetic-session',
'chatId': 'synthetic-chat', 'messageId': 'synthetic-assistant'},
'receipt': receipt, 'envelope': envelope,
'query': {'since': '2026-10-24', 'days': 1, 'timezone': 'UTC'},
'asOf': '2026-10-27T00:00:00Z',
}
result = subprocess.run(['node', str(Path(self.output.name) / 'selection-driver.mjs')],
input=json.dumps(payload), capture_output=True, text=True, check=True)
return json.loads(result.stdout)

def test_selection_current_receipt_exact_consumer(self):
self.complete_bundle(); original = self.retain()
current = self.reconcile()
self.assertEqual(current, original)
with self.locked_read(current) as envelope:
result = self.consume(current, envelope)
self.assertEqual(result['state'], 'ready')
item = result['items'][0]
self.assertEqual(item['value'], {'state': 'known', 'amount': 100})
for key in ('ownerId', 'requestId', 'subjectId', 'catalogId', 'resultId', 'revision'):
self.assertEqual(item['binding'][key], current[key])
self.assertEqual(item['binding']['sourceVersionIds'], [current['sourceVersionId']])
self.assertNotIn('observations', json.dumps(result))
self.assertNotIn('providerRecordingId', json.dumps(result))

def test_selection_missing_key_and_wrong_actor(self):
self.complete_bundle(); original = self.retain()
self.assertEqual(self.consume(self.reconcile('missing'), None, key='missing'), {'state': 'unavailable'})
self.assertEqual(self.consume(self.reconcile(actor=12), None, actor=12), {'state': 'unavailable'})
# Even a previously genuine receipt does not restore a denied current read.
with self.locked_read(original, actor=12) as envelope:
self.assertEqual(self.consume(original, envelope, actor=12), {'state': 'unavailable'})

def test_selection_withdrawal_removed_membership_and_expansion(self):
for mutation in (
'SELECT withdraw_context_source(owner_id,id) FROM context_sources',
'UPDATE context_source_versions SET removed_at=now()',
'DELETE FROM catalog_songs', 'DELETE FROM account_catalogs',
'DELETE FROM context_request_catalog_members',
):
with self.subTest(mutation=mutation):
self.setUp(); self.complete_bundle(); original = self.retain()
self.sql(mutation)
self.assertIsNone(self.reconcile())
with self.locked_read(original) as envelope:
self.assertEqual(self.consume(original, envelope), {'state': 'unavailable'})

def test_selection_new_current_result_does_not_upgrade_old_key(self):
self.complete_bundle(); original = self.retain()
fresh = self.retain(key='second', revision=1)
self.assertNotEqual(original['resultId'], fresh['resultId'])
self.assertEqual(self.consume(self.reconcile(), None), {'state': 'unavailable'})
with self.locked_read(original) as envelope:
self.assertEqual(self.consume(original, envelope), {'state': 'unavailable'})
current_fresh = self.reconcile('second')
with self.locked_read(fresh) as envelope:
result = self.consume(current_fresh, envelope, key='second')
self.assertEqual(result['items'][0]['binding']['resultId'], fresh['resultId'])

def test_selection_supported_correction_refuses_original(self):
self.complete_bundle(); original = self.retain()
self.sql(f"""INSERT INTO catalog_stream_runs(id,catalog_id,revision,scheduled_day,since,until)
SELECT '{uid(13)}',catalog_id,revision,'2026-10-26','2026-10-24','2026-10-24' FROM catalog_stream_tracking;""")
correction = {'state': 'complete', 'days': [{'date': '2026-10-24', 'streams': 200}],
'provider_recording_id': 'provider-a', 'source_hash': 'b'*64,
'retrieved_at': '2026-10-26T00:00:00Z'}
self.assertEqual(self.sql(f"SELECT commit_catalog_stream_track('{uid(13)}','USAAA2600001','{json.dumps(correction)}')").stdout.strip(), 't')
self.assertIsNone(self.reconcile())
with self.locked_read(original) as envelope:
self.assertEqual(self.consume(original, envelope), {'state': 'unavailable'})


def load_tests(loader, tests, pattern):
return unittest.TestSuite(SelectionConsumer(name) for name in
loader.getTestCaseNames(SelectionConsumer)
if name.startswith('test_selection_'))


if __name__ == '__main__':
unittest.main()
Loading
Loading