Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/ingestion.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ on:
branches: [main]
paths:
- "lib/ingestion/**"
- "lib/supabase/ingestion_imports/**"
- "types/database.types.ts"
- "lib/mcp/tools/ingestion/**"
- "lib/mcp/tools/index.ts"
- "lib/supabase/storage/*Ingestion*.ts"
Expand All @@ -18,6 +20,8 @@ on:
branches: [main]
paths:
- "lib/ingestion/**"
- "lib/supabase/ingestion_imports/**"
- "types/database.types.ts"
- "lib/mcp/tools/ingestion/**"
- "lib/mcp/tools/index.ts"
- "lib/supabase/storage/*Ingestion*.ts"
Expand Down
62 changes: 62 additions & 0 deletions lib/ingestion/CATALOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Organization datasets in Recoup

Organization imports use the existing Recoup Supabase project and its existing organization membership model. No per-organization database, schema, credentials or new tenant identifier is required. External database delivery is an optional separate operation.

## Storage and ownership

- Bronze originals, silver records, gold artifacts and manifests remain in the existing private ingestion bucket, under `organization/<organization_id>/...`.
- `ingestion_datasets` groups related imports inside one organization. The first import creates a dataset named after the source filename; later imports can explicitly reuse its `dataset_id`.
- `ingestion_imports` records the immutable manifest and each import's processing result. Held and rejected imports retain their evidence without queryable gold rows.
- `ingestion_records` stores ready gold as JSONB records, preserving identifiers and exact decimals as strings. Field types are recorded per import. Shared indexes begin with organization and import identifiers.

An import is a versioned population, not an automatic append into an org-wide financial ledger. Reading records requires an explicit dataset and job ID. New imports never overwrite earlier versions, and Recoup does not implicitly combine versions or deduplicate economic events across files.

The database RPC checks current membership inside every transaction and locks the membership through commit. All three tables have RLS enabled and no browser or service-role table grants. Only the service role can invoke the guarded RPC; it receives the actor from verified API/MCP authentication. Service credentials are trusted server credentials and must never reach a browser. A SQL function cannot authenticate a fabricated actor supplied by a compromised service credential.

## Create and process

Use the existing `create` operation with `organization_id`, source and contract. The response includes `dataset_id`. No destination configuration is needed for Recoup's own database.

To add another import to that dataset, pass its `dataset_id` and a new idempotency key:

```json
{
"action": "create",
"organization_id": "22222222-2222-4222-8222-222222222222",
"dataset_id": "33333333-3333-4333-8333-333333333333",
"idempotency_key": "income-2026-12-v1",
"source": { "kind": "connector", "connector_id": "report-snapshot" },
"contract": {
"version": "income-v1",
"fields": [{ "name": "id", "source": "ID", "type": "string" }]
}
}
```

Repeating the exact create request can also index a previously completed private import into the shared catalog without recapturing or reprocessing its original; its gold bytes are integrity-checked first.
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved

The selected dataset is part of the immutable input fingerprint. It must belong to the selected organization. Personal account imports retain their existing private artifact behavior and cannot select an organization dataset.

Workers store validated ready gold automatically in Recoup. This is internal dataset storage, not business approval or external delivery. Ready rows and their completion receipt commit together in one database transaction. The private completion marker is written afterward. If its write or the database acknowledgement fails, retrying converges on the same rows; database import state can briefly lead the private artifact marker. Infrastructure failures leave retryable queued work rather than a fabricated successful result.

## Read through HTTP or MCP

The standard authenticated `ingestion` tool and `POST /api/ingestion` accept:

```json
{ "action": "datasets", "organization_id": "<org UUID>", "limit": 25 }
{ "action": "imports", "organization_id": "<org UUID>", "dataset_id": "<dataset UUID>", "limit": 25 }
{ "action": "records", "organization_id": "<org UUID>", "dataset_id": "<dataset UUID>", "job_id": "<job SHA-256>", "limit": 25, "after_row": 0 }
```

Read-only GET uses the same fields as query parameters, for example `/api/ingestion?action=datasets&organization_id=<org UUID>`. Create and delivery remain POST-only. Authentication is unchanged; never pass `account_id`.

Dataset/import pages contain `datasets` or `imports` and `next_after`. Supply that cursor as `after` for the next page. Record pages contain `records: [{ row_number, record }]`, field types, the gold digest and `next_after`; supply that number as `after_row`. A null cursor means the page is complete. Lists allow 1–50 items; record pages allow 1–100 with a 2 MiB record-payload budget, so a page can contain fewer rows than requested. A single row larger than that budget returns HTTP 413 rather than being omitted; those records remain stored, but reading them requires a future large-object export path. Each query is scoped again in the database. Per-minute limits are 60 dataset/import list calls and 30 record page calls per actor and organization.

## Rollout and verification — 2026-11-09

Apply Database's `20261110010000_ingestion_catalog.sql` and `20261110010100_ingestion_catalog_page_budget.sql` in order to the existing project before deploying/enabling these API consumers. Keep `INGESTION_ENABLED` off until private storage, the existing rate counter, Redis dispatch storage and hosted Workflow execution are qualified. No new database needs provisioning. Local disposable databases are verification fixtures only.
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved

Local database tests exercise the actual migration, cross-organization denial, revocation, concurrent retries, exact values, immutable versions, private table/function grants, pagination and rollback. An optional joined test runs this API's real processor against the same disposable database and verifies recovery after a committed write loses its acknowledgement. The fixture schema includes only accounts and memberships plus the new migration; it is not a replay of the full historical Supabase schema or hosted acceptance.

Do not remove persisted datasets as a routine rollback. Revert the API consumer first and retain the additive tables and historical receipts. No production migration or intake activation is performed by these tests.
4 changes: 3 additions & 1 deletion lib/ingestion/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

Private, versioned processing from source files or configured JSON APIs through bronze, silver and gold into export artifacts or an external Postgres database. All code lives in this API repository.

Organization-owned imports can also store queryable gold rows in Recoup's existing shared database. See [Organization datasets](./CATALOG.md) for the default internal destination, migration order and list/record operations. No database per organization is required.

## Operating contract

- **Bronze:** exact original bytes, SHA-256, retrieval time and source occurrence. Private originals are never uploaded to public media storage.
Expand All @@ -19,7 +21,7 @@ Disabled by default. Before enabling a deployment:

1. Provision a dedicated empty **private** Supabase Storage bucket with a 16 MiB `file_size_limit` and set `INGESTION_STORAGE_BUCKET`. Allow `application/octet-stream` and ensure the project upload ceiling permits 16 MiB. Keep its object policies server-only; do not give anonymous/authenticated clients direct reads, writes or bucket updates. The service verifies the bucket is private before every storage operation. Retention and region are operator decisions; this implementation never deletes originals automatically.
2. Configure the API's existing Supabase service credentials and Vercel Workflow runtime. Manifests, originals, results and stage outputs are durable private objects. The existing service-only `take_site_request` primitive from the Sites activity migration is required for per-minute counters. Configure `INGESTION_REDIS_URL` (or the existing `REDIS_URL`) for atomic dispatch leases; acquisition and dispatch fail closed when their controls are unavailable.
3. Optionally configure source connectors and destinations below. Verify access with synthetic fixtures before production data. No source or destination is inferred from client names or request URLs.
3. For organization imports, apply `20261110010000_ingestion_catalog.sql` followed by `20261110010100_ingestion_catalog_page_budget.sql` in the existing database first; see [Organization datasets](./CATALOG.md). Optionally configure source connectors and destinations below. Verify access with synthetic fixtures before production data. No source or destination is inferred from client names or request URLs.
4. Set `INGESTION_ENABLED=true` only after deployment qualification. There is no added billing or recurring schedule.

Operations are exposed by `POST /api/ingestion`, read-only `GET /api/ingestion?job_id=...&stage=gold` (omit `stage` for job state) and the standard authenticated MCP tool `ingestion`. Identity comes from `x-api-key` or Bearer authentication; never supply `account_id`. Optional `organization_id` requires current membership. Requests are limited per authenticated actor, scope and operation: 10 creates/deliveries, 30 artifact reads and 120 status reads per minute, for both HTTP and MCP. Personal and organization namespaces are distinct. Organization-scoped jobs are shared with currently authorized organization members. Workers recheck the initiating actor's access before processing and publication; reads and delivery also recheck access. Delegated OAuth does not expose this tool.
Expand Down
126 changes: 126 additions & 0 deletions lib/ingestion/__tests__/catalog.integration.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { Client } from "pg";
import { randomUUID } from "node:crypto";
import { processIngestionOperation } from "../processIngestionOperation";
import { executeIngestionJob } from "../executeIngestionJob";
import type { IngestionDependencies } from "../types";
const socket = process.env.INGESTION_CATALOG_TEST_SOCKET;
const suite = socket ? describe : describe.skip;
suite("real shared-database ingestion lifecycle", () => {
let admin: Client;
let gateway: Client;
const actor = randomUUID(),
org = randomUUID(),
other = randomUUID();
beforeAll(async () => {
if (!socket || !/\/recoup-ingestion-[A-Za-z0-9_-]+$/.test(socket))
throw new Error("Disposable ingestion test socket required");
admin = new Client({ host: socket, port: 55489, database: "postgres" });
gateway = new Client({ host: socket, port: 55489, database: "postgres" });
await admin.connect();
await admin.query("INSERT INTO public.accounts(id) SELECT unnest($1::uuid[])", [
[actor, org, other],
]);
await admin.query(
"INSERT INTO public.account_organization_ids(account_id,organization_id) VALUES ($1,$2)",
[actor, org],
);
await gateway.connect();
await gateway.query("SET ROLE service_role");
});
afterAll(async () => {
await gateway?.end();
await admin?.end();
});
it("ingests into the shared database, recovers lost acknowledgement and refuses another org", async () => {
const objects = new Map<string, Uint8Array>();
let loseAck = true;
const deps: IngestionDependencies = {
// Deliberately bypass the API membership check here to verify the real database boundary.
authorize: async (_actor, organization) => `organization/${organization}`,
limit: async () => undefined,
store: {
read: async k => objects.get(k) ?? null,
create: async (k, v) => {
if (objects.has(k)) return false;
objects.set(k, v);
return true;
},
},
dispatch: async () => undefined,
acquire: async () => {
throw new Error("No external source");
},
deliver: async () => {
throw new Error("No external database");
},
catalog: async (account, organization, action, input) => {
const result = await gateway.query(
"SELECT public.ingestion_catalog_command($1,$2,$3,$4::jsonb) AS result",
[account, organization, action, JSON.stringify(input)],
);
if (action === "complete" && loseAck) {
loseAck = false;
throw new Error("Acknowledgement lost");
}
return result.rows[0].result;
},
};
const input = {
action: "create",
organization_id: org,
idempotency_key: "shared-income-1",
source: {
kind: "inline",
filename: "income.csv",
format: "csv",
content_base64: Buffer.from("ID,Amount\n001,0.1\n002,0.2").toString("base64"),
},
contract: {
version: "1",
fields: [
{ name: "id", source: "ID", type: "string" },
{ name: "amount", source: "Amount", type: "decimal" },
],
controls: [{ field: "amount", expected: "0.3" }],
},
};
const job = await processIngestionOperation(actor, input, deps);
expect(job.dataset_id).toMatch(/^[a-f0-9-]{36}$/);
await expect(executeIngestionJob(actor, org, job.job_id as string, deps)).rejects.toThrow(
"Acknowledgement lost",
);
expect([...objects.keys()].some(k => k.endsWith("/result.json"))).toBe(false);
await executeIngestionJob(actor, org, job.job_id as string, deps);
expect((await processIngestionOperation(actor, input, deps)).dataset_id).toBe(job.dataset_id);
const page = await processIngestionOperation(
actor,
{ action: "records", organization_id: org, dataset_id: job.dataset_id, job_id: job.job_id },
deps,
);
expect(page.records).toEqual([
{ row_number: 1, record: { id: "001", amount: "0.1" } },
{ row_number: 2, record: { id: "002", amount: "0.2" } },
]);
await expect(
processIngestionOperation(
actor,
{
action: "records",
organization_id: other,
dataset_id: job.dataset_id,
job_id: job.job_id,
},
deps,
),
).rejects.toThrow("Ingestion access denied");
await admin.query("DELETE FROM public.account_organization_ids WHERE account_id=$1", [actor]);
await expect(
processIngestionOperation(
actor,
{ action: "records", organization_id: org, dataset_id: job.dataset_id, job_id: job.job_id },
deps,
),
).rejects.toThrow("Ingestion access denied");
});
});
Loading
Loading