Repository navigation
feat: ingest and query organization datasets in Recoup #1240
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
efe8f83
feat: ingest organization datasets into shared Recoup storage
sidneyswift 0579001
fix: report shared ingestion record response limits
sidneyswift 6d042b4
fix: validate stored ingestion evidence before catalog replay
sidneyswift cc54b96
fix: reject malformed UTF-8 in saved ingestion metadata
sidneyswift 98fba08
test: strengthen organization ingestion access fixtures
sidneyswift File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| # Organization datasets in Recoup | ||
|
|
||
| Organization imports use the existing Recoup Supabase project and its existing organization membership model. No per-organization database, schema, credentials or new tenant identifier is required. External database delivery is an optional separate operation. | ||
|
|
||
| ## Storage and ownership | ||
|
|
||
| - Bronze originals, silver records, gold artifacts and manifests remain in the existing private ingestion bucket, under `organization/<organization_id>/...`. | ||
| - `ingestion_datasets` groups related imports inside one organization. The first import creates a dataset named after the source filename; later imports can explicitly reuse its `dataset_id`. | ||
| - `ingestion_imports` records the immutable manifest and each import's processing result. Held and rejected imports retain their evidence without queryable gold rows. | ||
| - `ingestion_records` stores ready gold as JSONB records, preserving identifiers and exact decimals as strings. Field types are recorded per import. Shared indexes begin with organization and import identifiers. | ||
|
|
||
| An import is a versioned population, not an automatic append into an org-wide financial ledger. Reading records requires an explicit dataset and job ID. New imports never overwrite earlier versions, and Recoup does not implicitly combine versions or deduplicate economic events across files. | ||
|
|
||
| The database RPC checks current membership inside every transaction and locks the membership through commit. All three tables have RLS enabled and no browser or service-role table grants. Only the service role can invoke the guarded RPC; it receives the actor from verified API/MCP authentication. Service credentials are trusted server credentials and must never reach a browser. A SQL function cannot authenticate a fabricated actor supplied by a compromised service credential. | ||
|
|
||
| ## Create and process | ||
|
|
||
| Use the existing `create` operation with `organization_id`, source and contract. The response includes `dataset_id`. No destination configuration is needed for Recoup's own database. | ||
|
|
||
| To add another import to that dataset, pass its `dataset_id` and a new idempotency key: | ||
|
|
||
| ```json | ||
| { | ||
| "action": "create", | ||
| "organization_id": "22222222-2222-4222-8222-222222222222", | ||
| "dataset_id": "33333333-3333-4333-8333-333333333333", | ||
| "idempotency_key": "income-2026-12-v1", | ||
| "source": { "kind": "connector", "connector_id": "report-snapshot" }, | ||
| "contract": { | ||
| "version": "income-v1", | ||
| "fields": [{ "name": "id", "source": "ID", "type": "string" }] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| Repeating the exact create request can also index a previously completed private import into the shared catalog without recapturing or reprocessing its original; its gold bytes are integrity-checked first. | ||
|
|
||
| The selected dataset is part of the immutable input fingerprint. It must belong to the selected organization. Personal account imports retain their existing private artifact behavior and cannot select an organization dataset. | ||
|
|
||
| Workers store validated ready gold automatically in Recoup. This is internal dataset storage, not business approval or external delivery. Ready rows and their completion receipt commit together in one database transaction. The private completion marker is written afterward. If its write or the database acknowledgement fails, retrying converges on the same rows; database import state can briefly lead the private artifact marker. Infrastructure failures leave retryable queued work rather than a fabricated successful result. | ||
|
|
||
| ## Read through HTTP or MCP | ||
|
|
||
| The standard authenticated `ingestion` tool and `POST /api/ingestion` accept: | ||
|
|
||
| ```json | ||
| { "action": "datasets", "organization_id": "<org UUID>", "limit": 25 } | ||
| { "action": "imports", "organization_id": "<org UUID>", "dataset_id": "<dataset UUID>", "limit": 25 } | ||
| { "action": "records", "organization_id": "<org UUID>", "dataset_id": "<dataset UUID>", "job_id": "<job SHA-256>", "limit": 25, "after_row": 0 } | ||
| ``` | ||
|
|
||
| Read-only GET uses the same fields as query parameters, for example `/api/ingestion?action=datasets&organization_id=<org UUID>`. Create and delivery remain POST-only. Authentication is unchanged; never pass `account_id`. | ||
|
|
||
| Dataset/import pages contain `datasets` or `imports` and `next_after`. Supply that cursor as `after` for the next page. Record pages contain `records: [{ row_number, record }]`, field types, the gold digest and `next_after`; supply that number as `after_row`. A null cursor means the page is complete. Lists allow 1–50 items; record pages allow 1–100 with a 2 MiB record-payload budget, so a page can contain fewer rows than requested. A single row larger than that budget returns HTTP 413 rather than being omitted; those records remain stored, but reading them requires a future large-object export path. Each query is scoped again in the database. Per-minute limits are 60 dataset/import list calls and 30 record page calls per actor and organization. | ||
|
|
||
| ## Rollout and verification — 2026-11-09 | ||
|
|
||
| Apply Database's `20261110010000_ingestion_catalog.sql` and `20261110010100_ingestion_catalog_page_budget.sql` in order to the existing project before deploying/enabling these API consumers. Keep `INGESTION_ENABLED` off until private storage, the existing rate counter, Redis dispatch storage and hosted Workflow execution are qualified. No new database needs provisioning. Local disposable databases are verification fixtures only. | ||
|
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| Local database tests exercise the actual migration, cross-organization denial, revocation, concurrent retries, exact values, immutable versions, private table/function grants, pagination and rollback. An optional joined test runs this API's real processor against the same disposable database and verifies recovery after a committed write loses its acknowledgement. The fixture schema includes only accounts and memberships plus the new migration; it is not a replay of the full historical Supabase schema or hosted acceptance. | ||
|
|
||
| Do not remove persisted datasets as a routine rollback. Revert the API consumer first and retain the additive tables and historical receipts. No production migration or intake activation is performed by these tests. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
| import { afterAll, beforeAll, describe, expect, it } from "vitest"; | ||
| import { Client } from "pg"; | ||
| import { randomUUID } from "node:crypto"; | ||
| import { processIngestionOperation } from "../processIngestionOperation"; | ||
| import { executeIngestionJob } from "../executeIngestionJob"; | ||
| import type { IngestionDependencies } from "../types"; | ||
| const socket = process.env.INGESTION_CATALOG_TEST_SOCKET; | ||
| const suite = socket ? describe : describe.skip; | ||
| suite("real shared-database ingestion lifecycle", () => { | ||
| let admin: Client; | ||
| let gateway: Client; | ||
| const actor = randomUUID(), | ||
| org = randomUUID(), | ||
| other = randomUUID(); | ||
| beforeAll(async () => { | ||
| if (!socket || !/\/recoup-ingestion-[A-Za-z0-9_-]+$/.test(socket)) | ||
| throw new Error("Disposable ingestion test socket required"); | ||
| admin = new Client({ host: socket, port: 55489, database: "postgres" }); | ||
| gateway = new Client({ host: socket, port: 55489, database: "postgres" }); | ||
| await admin.connect(); | ||
| await admin.query("INSERT INTO public.accounts(id) SELECT unnest($1::uuid[])", [ | ||
| [actor, org, other], | ||
| ]); | ||
| await admin.query( | ||
| "INSERT INTO public.account_organization_ids(account_id,organization_id) VALUES ($1,$2)", | ||
| [actor, org], | ||
| ); | ||
| await gateway.connect(); | ||
| await gateway.query("SET ROLE service_role"); | ||
| }); | ||
| afterAll(async () => { | ||
| await gateway?.end(); | ||
| await admin?.end(); | ||
| }); | ||
| it("ingests into the shared database, recovers lost acknowledgement and refuses another org", async () => { | ||
| const objects = new Map<string, Uint8Array>(); | ||
| let loseAck = true; | ||
| const deps: IngestionDependencies = { | ||
| // Deliberately bypass the API membership check here to verify the real database boundary. | ||
| authorize: async (_actor, organization) => `organization/${organization}`, | ||
| limit: async () => undefined, | ||
| store: { | ||
| read: async k => objects.get(k) ?? null, | ||
| create: async (k, v) => { | ||
| if (objects.has(k)) return false; | ||
| objects.set(k, v); | ||
| return true; | ||
| }, | ||
| }, | ||
| dispatch: async () => undefined, | ||
| acquire: async () => { | ||
| throw new Error("No external source"); | ||
| }, | ||
| deliver: async () => { | ||
| throw new Error("No external database"); | ||
| }, | ||
| catalog: async (account, organization, action, input) => { | ||
| const result = await gateway.query( | ||
| "SELECT public.ingestion_catalog_command($1,$2,$3,$4::jsonb) AS result", | ||
| [account, organization, action, JSON.stringify(input)], | ||
| ); | ||
| if (action === "complete" && loseAck) { | ||
| loseAck = false; | ||
| throw new Error("Acknowledgement lost"); | ||
| } | ||
| return result.rows[0].result; | ||
| }, | ||
| }; | ||
| const input = { | ||
| action: "create", | ||
| organization_id: org, | ||
| idempotency_key: "shared-income-1", | ||
| source: { | ||
| kind: "inline", | ||
| filename: "income.csv", | ||
| format: "csv", | ||
| content_base64: Buffer.from("ID,Amount\n001,0.1\n002,0.2").toString("base64"), | ||
| }, | ||
| contract: { | ||
| version: "1", | ||
| fields: [ | ||
| { name: "id", source: "ID", type: "string" }, | ||
| { name: "amount", source: "Amount", type: "decimal" }, | ||
| ], | ||
| controls: [{ field: "amount", expected: "0.3" }], | ||
| }, | ||
| }; | ||
| const job = await processIngestionOperation(actor, input, deps); | ||
| expect(job.dataset_id).toMatch(/^[a-f0-9-]{36}$/); | ||
| await expect(executeIngestionJob(actor, org, job.job_id as string, deps)).rejects.toThrow( | ||
| "Acknowledgement lost", | ||
| ); | ||
| expect([...objects.keys()].some(k => k.endsWith("/result.json"))).toBe(false); | ||
| await executeIngestionJob(actor, org, job.job_id as string, deps); | ||
| expect((await processIngestionOperation(actor, input, deps)).dataset_id).toBe(job.dataset_id); | ||
| const page = await processIngestionOperation( | ||
| actor, | ||
| { action: "records", organization_id: org, dataset_id: job.dataset_id, job_id: job.job_id }, | ||
| deps, | ||
| ); | ||
| expect(page.records).toEqual([ | ||
| { row_number: 1, record: { id: "001", amount: "0.1" } }, | ||
| { row_number: 2, record: { id: "002", amount: "0.2" } }, | ||
| ]); | ||
| await expect( | ||
| processIngestionOperation( | ||
| actor, | ||
| { | ||
| action: "records", | ||
| organization_id: other, | ||
| dataset_id: job.dataset_id, | ||
| job_id: job.job_id, | ||
| }, | ||
| deps, | ||
| ), | ||
| ).rejects.toThrow("Ingestion access denied"); | ||
| await admin.query("DELETE FROM public.account_organization_ids WHERE account_id=$1", [actor]); | ||
| await expect( | ||
| processIngestionOperation( | ||
| actor, | ||
| { action: "records", organization_id: org, dataset_id: job.dataset_id, job_id: job.job_id }, | ||
| deps, | ||
| ), | ||
| ).rejects.toThrow("Ingestion access denied"); | ||
| }); | ||
| }); |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.