diff --git a/proposals/retained-field-integration-2207/PROTOCOL.md b/proposals/retained-field-integration-2207/PROTOCOL.md index 6e5bcfa..7d7642c 100644 --- a/proposals/retained-field-integration-2207/PROTOCOL.md +++ b/proposals/retained-field-integration-2207/PROTOCOL.md @@ -92,3 +92,94 @@ than inventing per-field object IDs. App part 256 KiB, controls 64 KiB and comma 128 KiB limits remain unchanged; a 768 KiB evidence envelope is not a chat payload. Do not copy all retained tuples into saved controls or equate document revision with surface revision. This suite does not prove deployed behavior or provider execution. + +## Genuine receipt selection consumer (source-only increment, 2026-11-19) + +`selection-source.json` pins the separate API proposal consumer. The original +adopted evaluator checkout and all three original method hashes remain independently +required by `test_integration.py`; they are not replaced or widened. The added +consumer reads an existing current `reconcile_retention` receipt, projects its exact +Context selection, rechecks the original selection via `read_binding`, and evaluates +inside the original held transaction. No raw report, new authority store, latest +upgrade, implicit retain/expand or actor substitution is used. No SQL definition, +grant, route or production transport is introduced. + +`test_selection_transport.py` builds the exact new API consumer plus its unchanged +real evaluator and checks the actual generated inner fieldContext fixture. These +five runnable **pure synthetic transport** tests do not initialize PostgreSQL or +claim authenticated source authority. Use the frozen source checkout with its +existing dependencies: + +```sh +CATALOG_SELECTION_SOURCE=/path/to/frozen/api python3 proposals/retained-field-integration-2207/test_selection_transport.py +``` + +`test_selection_consumer.py` extends the existing actual database harness with five +new deferred scenarios: current genuine receipt → exact inner response, missing key +and wrong actor, withdrawn/removed/member/expanded-edge loss, superseded result, +and supported stream correction refusing the original selection. It calls actual +reconciliation and source reads; no new permissive SQL adapter is installed. The +SQL cases default to skipped. The focused local qualification below subsequently +executed all five; later runs must explicitly set `CATALOG_SELECTION_SQL_QUALIFICATION=1`; +that also requires `CATALOG_API_SOURCE` at the original adopted evaluator pin and +`CATALOG_SELECTION_SOURCE` at the new companion pin. The five sequential cases do +not qualify concurrent max-eight batch locking or production transaction transport. + +The synthetic actor/session/chat/assistant strings in this harness are test context, +not session authentication proof. Returning a real selection also is not a lease: +the next read reauthorizes it. `fieldContext` is exactly the inner projection response +without `state`; UI owns its explicitly versioned offline envelope. Schema-v1 is +unchanged. No initial-send/replay/history/text/export/model disclosure path is enabled; +these paths must jointly withhold projections on current denial before live routing. + +### Focused local qualification — 2026-11-19 + +All five new SelectionConsumer scenarios passed without skips against Database +`1185562def1bdbc49b70cdc7d889ba52b352f03b`, API consumer +`4ca23de6a6cea3226c2f0392a454eb57861aedf8` and original evaluator +`6ba799dbb67be284adcd6f8bcd8fcde2faa6c2b8`, using installed PostgreSQL 17.7. +Only those five tests ran; inherited unrelated suites were excluded. The default +PostgreSQL 14 server failed before startup because its ICU74 library was absent; +no tools were installed, and its temporary directory was removed. + +The successful run cleared inherited connection/credential settings, initialized a +new synthetic temporary cluster, verified its postmaster/data-directory/socket +identity and empty listen_addresses before fixture SQL, and checked zero TCP +listeners. All schema/role/data writes stayed in that disposable database. Cleanup +verified the owned process, Unix socket, PID file and temporary directory were gone. +No hosted DSN, existing database, account credential, runtime route, production +apply/grant/enrollment or activation was used. This closes the five sequential +consumer scenarios only; batch concurrency, authenticated production transport and +publication/history disclosure qualification remain outstanding. + +### Size-refusal successor — 2026-11-19 + +The companion API `a7ba45167d159428bfac90fa6cb2fa58df88d360` preserves the +registered evaluator's `size_limit` refusal as `{state:"invalid"}`. Missing/stale/ +withdrawn/unauthorized bindings stay unavailable and operational failures stay +errors. Six pure compiled transport tests pass, including real UTF-8 inputs over +512 KiB and over 768 KiB that return only the invalid state. The inner UI fixture +and evaluator/recipe/helper bytes are unchanged. Selection-source pins now target +this successor. + +The five real PostgreSQL17.7 passes above remain attributed to their exact original +API4ca23de6/Database1185562 sources. They were not rerun or relabelled as successor +qualification. No SQL semantics or actual SQL test scenario changed in this repair. + +### Operational-refusal review successor — 2026-11-19 + +The companion pin now targets API `20784986cd08d8651545d71d024b98c223ca7866`. +`helper_changed` and `source_error` preserve the operational `error` state; +`size_limit` remains `invalid`, and ordinary denied bindings remain `unavailable`. +Every refusal withholds values and diagnostics. Coverage comparison state is a +closed comparable/incomplete union checked before projection. The API README +links the private Database proposal and distinguishes it from a connected live +API transport. Seven pure compiled transport tests pass, including an actual +helper-integrity mismatch in temporary bundle output, restored after the test. +The five actual PostgreSQL passes remain attributed only to the original heads +above; no SQL definitions or actual SQL scenarios changed or were rerun. + +Final type follow-up: `selection-source.json` pins the successor with explicit +reference authority/kind and source literal types. The consumer checks reference +literals before projection. The seven compiled tests and unchanged fixture still +pass; original SQL qualification attribution remains unchanged. diff --git a/proposals/retained-field-integration-2207/build-selection.mjs b/proposals/retained-field-integration-2207/build-selection.mjs new file mode 100644 index 0000000..58c5765 --- /dev/null +++ b/proposals/retained-field-integration-2207/build-selection.mjs @@ -0,0 +1,35 @@ +// Deferred fixture transport only. Build exact source consumer; no replacement evaluator. +import { createRequire } from 'node:module'; +import { mkdirSync, copyFileSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +const [source, output] = process.argv.slice(2); +const require = createRequire(join(source, 'package.json')); +const vite = require.resolve('vite', { paths: [require.resolve('vitest/package.json')] }); +const { build } = require(require.resolve('esbuild', { paths: [vite] })); +const relative = 'scripts/proposals/catalog-retained-selection'; +mkdirSync(join(output, relative), { recursive: true }); +mkdirSync(join(output, 'lib/catalog'), { recursive: true }); +copyFileSync(join(source, 'lib/catalog/compareSpotifyDailyStreams.ts'), join(output, 'lib/catalog/compareSpotifyDailyStreams.ts')); +for (const name of ['readRetainedFieldSelection', 'consumeRetainedField', 'schemas']) { + await build({ entryPoints: [join(source, relative, name + '.ts')], bundle: true, + platform: 'node', format: 'esm', target: 'node20', outfile: join(output, relative, name + '.mjs') }); +} +// Bundled evaluator keeps its new URL('./recipe.json', import.meta.url) lookup. +copyFileSync(join(source, 'scripts/proposals/catalog-stream-fields/recipe.json'), join(output, relative, 'recipe.json')); +writeFileSync(join(output, 'selection-driver.mjs'), ` +import { readFileSync } from 'node:fs'; +import { readRetainedFieldSelection } from './${relative}/readRetainedFieldSelection.mjs'; +import { consumeRetainedField } from './${relative}/consumeRetainedField.mjs'; +import { method } from './${relative}/schemas.mjs'; +const input = JSON.parse(readFileSync(0, 'utf8')); +// Receipts/envelopes below come from the real disposable SQL harness, not a runtime route. +const selected = await readRetainedFieldSelection(input.scope, async () => input.receipt); +if (selected.state !== 'ready') console.log(JSON.stringify(selected)); +else { + const result = await consumeRetainedField(input.actor, { + mode:'retained_field', workspaceId:input.scope.workspaceId, catalogId:input.scope.catalogId, + field:'current_streams', query:input.query, method, selections:[selected.selection] + }, async (_scope, consume) => consume({asOf:input.asOf,readBinding:async () => input.envelope})); + console.log(JSON.stringify(result)); +} +`); diff --git a/proposals/retained-field-integration-2207/selection-source.json b/proposals/retained-field-integration-2207/selection-source.json new file mode 100644 index 0000000..8fdb70d --- /dev/null +++ b/proposals/retained-field-integration-2207/selection-source.json @@ -0,0 +1,12 @@ +{ + "head": "af21d0037d3cb4b79b837a9c164890537a89a5cb", + "files": { + "scripts/proposals/catalog-retained-selection/readRetainedFieldSelection.ts": "1096979cc82d034ad08ddcb6db8a7e04a0c91dbe5a332ed3f40c98120e4c996a", + "scripts/proposals/catalog-retained-selection/consumeRetainedField.ts": "deeb2de2f28cd70b4f4926c3a4f83e2419b34d8b661b56cc6216e5fba6a71961", + "scripts/proposals/catalog-retained-selection/schemas.ts": "b5cd78e40bbdd9e83914bb729795706dd14d81988679512dc4a5d95943369e88", + "scripts/proposals/catalog-retained-selection/fixtures/fieldContext.json": "ed979fc2ea72c3a3e51deb2ba259980f171a550b88fb488afb783a70e02ec322", + "scripts/proposals/catalog-retained-selection/fixtures/synthetic-authority-input.json": "f498cff7bdef6154382a98815162d140e6660e14ace885e7bee8e56334fcd522", + "scripts/proposals/catalog-retained-selection/classifyRetainedFieldRefusal.ts": "9a50a90b7701ac21897fdcee44d5c4e60567b00c38d504daf14acc25b04a6d13", + "scripts/proposals/catalog-retained-selection/retainedFieldTypes.ts": "9c20910786eef5ef60ed5ad8d2e6505b5fa4be932e8f2e0d465c3e371d6ae229" + } +} diff --git a/proposals/retained-field-integration-2207/test_selection_consumer.py b/proposals/retained-field-integration-2207/test_selection_consumer.py new file mode 100644 index 0000000..50d7aaa --- /dev/null +++ b/proposals/retained-field-integration-2207/test_selection_consumer.py @@ -0,0 +1,123 @@ +"""Deferred actual Context receipt -> selection -> locked reader -> inner consumer. +Default skipped. No PostgreSQL qualification is authorized by adding these tests. +""" +import hashlib +import json +import os +from pathlib import Path +import subprocess +import unittest + +from test_integration import Integration, uid, FILES + +HERE = Path(__file__).resolve().parent + + +@unittest.skipUnless(os.environ.get('CATALOG_SELECTION_SQL_QUALIFICATION') == '1', + 'SQL hold: genuine selection scenarios are deferred') +class SelectionConsumer(Integration): + @classmethod + def setUpClass(cls): + # Original fixture still pins the original evaluator checkout at its adopted SHA. + cls.selection_source = Path(os.environ['CATALOG_SELECTION_SOURCE']).resolve() + cls.pins = json.loads((HERE / 'selection-source.json').read_text()) + cls.verify_selection_source() + super().setUpClass() + cls.addClassCleanup(cls.verify_selection_source) + subprocess.run(['node', str(HERE / 'build-selection.mjs'), + str(cls.selection_source), cls.output.name], check=True) + + @classmethod + def verify_selection_source(cls): + head = subprocess.check_output(['git', '-C', str(cls.selection_source), + 'rev-parse', 'HEAD'], text=True).strip() + if head != cls.pins['head']: + raise AssertionError('Selection source head differs from frozen companion') + for name, digest in {**FILES, **cls.pins['files']}.items(): + if hashlib.sha256((cls.selection_source / name).read_bytes()).hexdigest() != digest: + raise AssertionError('Selection source drift: ' + name) + + def consume(self, receipt, envelope, actor=1, key='first'): + payload = { + 'scope': {'actorId': uid(actor), 'workspaceId': uid(1), 'requestId': uid(4), + 'catalogId': uid(2), 'key': key}, + 'actor': {'actorId': uid(actor), 'sessionId': 'synthetic-session', + 'chatId': 'synthetic-chat', 'messageId': 'synthetic-assistant'}, + 'receipt': receipt, 'envelope': envelope, + 'query': {'since': '2026-10-24', 'days': 1, 'timezone': 'UTC'}, + 'asOf': '2026-10-27T00:00:00Z', + } + result = subprocess.run(['node', str(Path(self.output.name) / 'selection-driver.mjs')], + input=json.dumps(payload), capture_output=True, text=True, check=True) + return json.loads(result.stdout) + + def test_selection_current_receipt_exact_consumer(self): + self.complete_bundle(); original = self.retain() + current = self.reconcile() + self.assertEqual(current, original) + with self.locked_read(current) as envelope: + result = self.consume(current, envelope) + self.assertEqual(result['state'], 'ready') + item = result['items'][0] + self.assertEqual(item['value'], {'state': 'known', 'amount': 100}) + for key in ('ownerId', 'requestId', 'subjectId', 'catalogId', 'resultId', 'revision'): + self.assertEqual(item['binding'][key], current[key]) + self.assertEqual(item['binding']['sourceVersionIds'], [current['sourceVersionId']]) + self.assertNotIn('observations', json.dumps(result)) + self.assertNotIn('providerRecordingId', json.dumps(result)) + + def test_selection_missing_key_and_wrong_actor(self): + self.complete_bundle(); original = self.retain() + self.assertEqual(self.consume(self.reconcile('missing'), None, key='missing'), {'state': 'unavailable'}) + self.assertEqual(self.consume(self.reconcile(actor=12), None, actor=12), {'state': 'unavailable'}) + # Even a previously genuine receipt does not restore a denied current read. + with self.locked_read(original, actor=12) as envelope: + self.assertEqual(self.consume(original, envelope, actor=12), {'state': 'unavailable'}) + + def test_selection_withdrawal_removed_membership_and_expansion(self): + for mutation in ( + 'SELECT withdraw_context_source(owner_id,id) FROM context_sources', + 'UPDATE context_source_versions SET removed_at=now()', + 'DELETE FROM catalog_songs', 'DELETE FROM account_catalogs', + 'DELETE FROM context_request_catalog_members', + ): + with self.subTest(mutation=mutation): + self.setUp(); self.complete_bundle(); original = self.retain() + self.sql(mutation) + self.assertIsNone(self.reconcile()) + with self.locked_read(original) as envelope: + self.assertEqual(self.consume(original, envelope), {'state': 'unavailable'}) + + def test_selection_new_current_result_does_not_upgrade_old_key(self): + self.complete_bundle(); original = self.retain() + fresh = self.retain(key='second', revision=1) + self.assertNotEqual(original['resultId'], fresh['resultId']) + self.assertEqual(self.consume(self.reconcile(), None), {'state': 'unavailable'}) + with self.locked_read(original) as envelope: + self.assertEqual(self.consume(original, envelope), {'state': 'unavailable'}) + current_fresh = self.reconcile('second') + with self.locked_read(fresh) as envelope: + result = self.consume(current_fresh, envelope, key='second') + self.assertEqual(result['items'][0]['binding']['resultId'], fresh['resultId']) + + def test_selection_supported_correction_refuses_original(self): + self.complete_bundle(); original = self.retain() + self.sql(f"""INSERT INTO catalog_stream_runs(id,catalog_id,revision,scheduled_day,since,until) + SELECT '{uid(13)}',catalog_id,revision,'2026-10-26','2026-10-24','2026-10-24' FROM catalog_stream_tracking;""") + correction = {'state': 'complete', 'days': [{'date': '2026-10-24', 'streams': 200}], + 'provider_recording_id': 'provider-a', 'source_hash': 'b'*64, + 'retrieved_at': '2026-10-26T00:00:00Z'} + self.assertEqual(self.sql(f"SELECT commit_catalog_stream_track('{uid(13)}','USAAA2600001','{json.dumps(correction)}')").stdout.strip(), 't') + self.assertIsNone(self.reconcile()) + with self.locked_read(original) as envelope: + self.assertEqual(self.consume(original, envelope), {'state': 'unavailable'}) + + +def load_tests(loader, tests, pattern): + return unittest.TestSuite(SelectionConsumer(name) for name in + loader.getTestCaseNames(SelectionConsumer) + if name.startswith('test_selection_')) + + +if __name__ == '__main__': + unittest.main() diff --git a/proposals/retained-field-integration-2207/test_selection_transport.py b/proposals/retained-field-integration-2207/test_selection_transport.py new file mode 100644 index 0000000..73672b2 --- /dev/null +++ b/proposals/retained-field-integration-2207/test_selection_transport.py @@ -0,0 +1,79 @@ +"""Pure compiled transport tests. Synthetic receipt/evidence; no SQL or network.""" +import copy +import hashlib +import json +import os +from pathlib import Path +import subprocess +import tempfile +import unittest + +from test_selection_consumer import SelectionConsumer, HERE + + +class SelectionTransport(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.source = Path(os.environ['CATALOG_SELECTION_SOURCE']).resolve() + SelectionConsumer.selection_source = cls.source + SelectionConsumer.pins = json.loads((HERE/'selection-source.json').read_text()) + SelectionConsumer.verify_selection_source() + cls.addClassCleanup(SelectionConsumer.verify_selection_source) + cls.output = tempfile.TemporaryDirectory(prefix='retained-selection-transport-') + cls.addClassCleanup(cls.output.cleanup) + subprocess.run(['node',str(HERE/'build-selection.mjs'),str(cls.source),cls.output.name],check=True,capture_output=True) + cls.fixture_dir = cls.source/'scripts/proposals/catalog-retained-selection/fixtures' + cls.input = json.loads((cls.fixture_dir/'synthetic-authority-input.json').read_text()) + + def consume(self, payload): + run = subprocess.run(['node', str(Path(self.output.name)/'selection-driver.mjs')], + input=json.dumps(payload),text=True,capture_output=True,check=True) + return json.loads(run.stdout) + + def test_current_selection_matches_actual_inner_fixture(self): + result=self.consume(self.input) + self.assertEqual(result.pop('state'),'ready') + self.assertEqual(result,json.loads((self.fixture_dir/'fieldContext.json').read_text())) + + def test_missing_pending_and_wrong_scope_receipts_withhold(self): + for receipt in [None, {}, {'state':'pending'}, {**self.input['receipt'],'ownerId':self.input['scope']['actorId']}, {**self.input['receipt'],'key':'new-key'}]: + with self.subTest(receipt=receipt): + self.assertEqual(self.consume({**self.input,'receipt':receipt}),{'state':'unavailable'}) + + def test_stale_binding_does_not_upgrade(self): + for key,value in [('revision',99),('resultId',self.input['scope']['actorId']),('sourceVersionIds',[self.input['scope']['actorId']])]: + payload=copy.deepcopy(self.input);payload['envelope'][key]=value + self.assertEqual(self.consume(payload),{'state':'unavailable'}) + + def test_current_source_refusal_redacts_entire_result(self): + self.assertEqual(self.consume({**self.input,'envelope':None}),{'state':'unavailable'}) + + def test_oversized_utf8_authority_evidence_returns_invalid_without_leak(self): + for characters in (270000, 400000): + with self.subTest(characters=characters): + payload=copy.deepcopy(self.input) + payload['envelope']['evidence']['providerRecordingId']='é'*characters + size=len(json.dumps(payload['envelope'],ensure_ascii=False).encode('utf8')) + self.assertGreater(size,512*1024) + if characters==270000:self.assertLess(size,768*1024) + else:self.assertGreater(size,768*1024) + self.assertEqual(self.consume(payload),{'state':'invalid'}) + + def test_actual_helper_integrity_failure_is_error(self): + helper=Path(self.output.name)/'lib/catalog/compareSpotifyDailyStreams.ts' + original=helper.read_bytes() + try: + helper.write_bytes(original+b'\n// synthetic integrity mismatch\n') + self.assertEqual(self.consume(self.input),{'state':'error'}) + finally: + helper.write_bytes(original) + + def test_invalid_query_never_uses_cached_amount(self): + self.assertEqual(self.consume({**self.input,'query':{**self.input['query'],'days':0}}),{'state':'invalid'}) + + +def load_tests(loader, tests, pattern): + return loader.loadTestsFromTestCase(SelectionTransport) + + +if __name__=='__main__': unittest.main()