feat(operations): add synthetic import and refresh golden path

[alexdancer]

Intent

Build Huddle's bounded synthetic Operations golden path: strict guide CSV validation and transactional commit; deterministic conflict handling and synthetic-scope proof; guide manual refresh plus protected nightly worker dispatch; immutable receipt-lineage board runs with atomic head promotion and prior-head preservation; engine-backed ranked entries/evidence with complete deterministic cause-specific fallbacks; leases/reaper, Chicago windows, least-privilege database role, and redacted operational events. Keep browser data scope untrusted, ingest driver-free, model narration optional, and all student data synthetic.

What Changed

• Add a guide-facing synthetic CSV workflow with strict validation, scoped idempotent receipts, transactional commits, and deterministic duplicate/unmapped handling.
• Add shared manual and protected nightly refresh orchestration with Chicago windows, leases and reaping, redacted events, immutable receipt-linked runs, atomic head promotion, and least-privilege database grants.
• Render engine-ranked board states and run-scoped evidence with persisted mastery lineage, prior-head preservation, and validated deterministic cause-specific fallbacks.

Risk Assessment

✅ Low: The bounded documentation repair correctly exports the protected dispatch secret while preserving the test-only override and model-free workflow, with no new material source risks found.

Testing

Prerequisite checks, focused Operations/auth/scope/fallback tests, disposable PostgreSQL migrations, and the rendered synthetic guide journey all succeeded; a stale ignored @huddle/db build artifact was regenerated and successfully retested, four visual artifacts plus persisted lineage evidence were captured, and all temporary resources were removed.

• Evidence: Synthetic CSV validation preview (local file: /var/folders/nt/rdk7cjs538l8zphln24q2k900000gn/T/no-mistakes-evidence/01KYQPSD46M5AWPHCZ66QE2V17/import-validation-preview.png)
• Evidence: Committed transactional import receipt (local file: /var/folders/nt/rdk7cjs538l8zphln24q2k900000gn/T/no-mistakes-evidence/01KYQPSD46M5AWPHCZ66QE2V17/import-committed-receipt.png)
• Evidence: Promoted deterministic ranked board (local file: /var/folders/nt/rdk7cjs538l8zphln24q2k900000gn/T/no-mistakes-evidence/01KYQPSD46M5AWPHCZ66QE2V17/promoted-ranked-board.png)
• Evidence: Exact evidence bundle (local file: /var/folders/nt/rdk7cjs538l8zphln24q2k900000gn/T/no-mistakes-evidence/01KYQPSD46M5AWPHCZ66QE2V17/exact-evidence.png)

Evidence: Persisted receipt, immutable lineage, head promotion, and fallback state

Import receipt 2: succeeded, accepted=3. Board run 1 had no predecessor; run 2 superseded run 1; board_head pointed to run 2. Both entries persisted narration_mode=deterministic_fallback and narration_status=degraded.

- Outcome: 🔧 2 issues found → auto-fixed (2) ✅ across 3 runs (1h0m49s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 15 issues found → auto-fixed (5) ✅

• 🚨 apps/web/app/internal/refresh/route.ts:36 - The intent requires “protected nightly worker dispatch” and “leases/reaper,” but this route only consumes an already-created request ID. No changed source schedules Chicago eligibility, enqueues or dispatches nightly requests, or invokes reapExpiredRefreshes; expired requests can remain running and block the active-refresh constraint.
• 🚨 packages/db/src/imports.ts:153 - commitImport starts a transaction through a Pool, then performs subsequent queries through that pool. Pool.query does not reserve one connection, so partial imports and stranded transactions are reachable. Acquire a PoolClient and use it through commit/rollback; saveImportValidation has the same defect.
• 🚨 packages/db/src/boards.ts:33 - Activity is loaded and the connection released before the publication transaction captures receipts. An import committed between those steps is included in board_run_import and its fingerprint but absent from the compiled draft. Capture receipts and linked activity in the same REPEATABLE READ snapshot.
• 🚨 packages/db/src/boards.ts:165 - Publication does not fence the lease or verify that the request-state update affected one row. After expiry—or even after the reaper marks the request failed—the transaction can still insert a run and promote board_head. Lock and validate the running, unexpired request before inserting candidate rows.
• 🚨 packages/db/src/boards.ts:188 - chicagoAnchor always uses 06:00Z and derives the start by subtracting 168 hours. Chicago midnight is 05:00Z during daylight time, and DST-crossing seven-local-day windows are not always 168 hours. Persist the shared IANA-derived Chicago window used by the engine.
• 🚨 apps/web/lib/operations.ts:155 - The production sorter omits the required skill-ID/null-last and student-ID tie-breakers. Because the attempts query has no ordering, tied findings can select different dominant signals or ranks across identical runs. Reuse the signal engine’s frozen comparator/ranking boundary.
• 🚨 packages/db/src/boards.ts:151 - The required “complete deterministic cause-specific fallbacks” are persisted with an empty {} catalog selection and hard-coded prose, bypassing catalog eligibility, slot, atom, and grounding validation. A non-timing guessing rule can also receive text claiming a “fast” pattern. Use committed validated fallback selections rendered from each evidence bundle.
• 🚨 db/migrations/009_board_run_head.sql:205 - The intent requires a “least-privilege database role,” but huddle_app lacks INSERT on the immutable run, lineage, snapshot, signal, and entry tables needed for publication, while receiving UPDATE on append-only attempts, sessions, issues, and unmapped activity. Define and wire the exact runtime mutation privileges.
• 🚨 packages/application/src/operations-importer.ts:41 - The required “strict guide CSV validation” is not fail-closed: fatal oversize, invalid UTF-8, malformed CSV, and bad-header analyses are saved as validated previews, after which the UI permits commit and records succeeded_with_rejections. File-level fatal analyses should not create a committable validation.
• 🚨 apps/web/lib/operations.ts:94 - Every loaded attempt uses its own attempt-row ID as sessionId instead of the persisted learning session. Multiple attempts from one source session are consequently treated as separate sessions, corrupting session-only baselines, grinding detection, and evidence session rows. Select and map the actual session identity.
• 🚨 apps/web/lib/operations.ts:115 - The engine is always given an unknown mastery lookup, and publication inserts no mastery_snapshot rows. This makes decay and mastery-backed prerequisite rules permanently abstain and leaves immutable runs incomplete relative to the required engine-backed board/evidence flow. Compute run-anchored mastery and persist it atomically.
• 🚨 packages/db/src/boards.ts:81 - A scope with no committed receipts is still compiled and promoted as a successful empty board. This contradicts the required first-build behavior: unavailable inputs must fail as input-unavailable, while only a valid compiled receipt set with no fired signals is a successful empty board.
• 🚨 packages/db/src/scoped.ts:146 - The product read path returns only exact-date rows and exposes none of the idle|queued|running|succeeded|failed, not-built, or stale states defined by the new application contract. A failed first refresh for day D therefore hides day D-1’s successful head and renders an ordinary empty state instead of preserving it visibly.
• 🚨 db/migrations/008_import_receipt_session.sql:116 - The migration replaces attempt with a schema lacking session_id, session_total_ms, and timing_was_winsorized, while the existing npm run nightly path still queries those columns and later deletes from newly immutable tables using removed board_date columns. Route the compatibility command through the shared compiler or retain a compatible boundary.
• 🚨 packages/db/src/boards.ts:44 - The compiler’s absence query selects records for every student assigned to the guide without requiring student.is_synthetic. A non-synthetic student under a synthetic guide therefore has attendance data read into the demo worker, contradicting the all-synthetic scope invariant. Apply the synthetic student predicate at this read boundary.

🔧 Fix: Fix Operations transaction, scheduling, and publication guarantees
14 errors still open:

• 🚨 apps/web/app/internal/refresh/route.ts:42 - The required protected nightly dispatch is still not autonomous: only an externally invoked CLI/route exists, with no owned 15-minute schedule. The route also returns outside the eligibility window before reaping, so a lease expiring after 07:45 remains running until the next eligible day and can block retries. Add the schedule and run the reaper before the eligibility return.
• 🚨 packages/application/src/operations-refresh.ts:47 - The 07:45 deadline is checked only before compilation. A nightly run starting at 07:44:59 can commit after the cutoff; enforce the Chicago deadline in the final locked publication update so all candidate rows roll back when completion is late.
• 🚨 .env.example:8 - The required least-privilege runtime role remains unused: the documented runtime URL connects as postgres, while huddle_app is never assumed. If wired directly, validation's FOR UPDATE on guide_auth_scope also exceeds its SELECT-only grant. Wire a usable runtime credential/role and grant only the exact required mutations.
• 🚨 apps/web/app/board/page.tsx:36 - Prior-head and refresh-state visibility remains incomplete. A stale successful-empty head takes the rows.length === 0 branch and renders an all-clear state, while exact-date queued/running/failed states are also hidden. Render board kind and refresh state before selecting the empty-entry presentation.
• 🚨 packages/narrator/src/catalog.ts:95 - The fallback validator self-authorizes its selection by replacing languageOptions with the selected IDs immediately before checking membership. It implements no catalog eligibility, authorized slot/pointer hashes, deterministic rendering, or atom grounding, so the required validated catalog provenance remains unproven. Route fallbacks through the committed shared grounding boundary.
• 🚨 apps/web/lib/operations.ts:154 - Only each student's dominant signal receives a deterministic fallback; additional board-visible causes retain their original pending language bundle. This leaves the required complete cause-specific fallback/evidence path absent for secondary causes. Apply validated fallback selection to every surfaced signal.
• 🚨 packages/db/src/boards.ts:193 - Signal persistence writes severity into both intensity and severity and hard-codes confidence_breakdown to {}. This corrupts engine output whenever urgency changes intensity and removes the timing/winsorization/conflict audit trail. Carry and persist the engine's actual intensity and breakdown.
• 🚨 packages/db/src/boards.ts:167 - The board behavior fingerprint hashes only fired signals. A successful-empty run always hashes [], so rule/config changes that still produce no findings are indistinguishable. Persist the engine's complete rule/config behavior fingerprint independently of output membership.
• 🚨 apps/web/lib/operations.ts:137 - After remapping signal IDs, evidence is rehashed as raw JSON, reintroducing database attempt IDs and remapped signal IDs that the engine's canonical fingerprint deliberately excludes. Equivalent imports with different row allocation can therefore produce different evidence and finding fingerprints. Reuse the canonical fingerprint boundary after ID remapping.
• 🚨 packages/db/src/boards.ts:74 - The compiler snapshot loads only an internal session ID and total duration, omitting the source session bounds and vendor_attempt_count. Evidence consequently derives bounds from attempt timestamps and records a null vendor count rather than the committed session aggregate. Carry the complete immutable session facts into engine evidence.
• 🚨 packages/db/src/boards.ts:81 - The absence read was synthetic-scoped, but the sibling attempt read still selects every attempt in the application scope without joining the claimed guide's synthetic roster. A non-synthetic student linked to that scope is therefore read by the demo worker, contradicting the requirement that all student data be synthetic. Enforce guide and student.is_synthetic at this shared SQL boundary.
• 🚨 packages/db/src/imports.ts:439 - Commit authorization is checked through the pool before commitImport acquires its transaction connection. Scope authorization or synthetic status can change between proof and persistence. Revalidate and lock the trusted scope inside the same transaction that locks the receipt and writes facts.
• 🚨 packages/db/src/imports.ts:265 - Stored-event conflict comparison omits session identity. Reusing event E in a different valid session with otherwise identical attempt fields is counted as a duplicate, leaving the new session orphaned instead of rejecting a divergent identity. Compare the resolved/source session identity at the existing event-conflict boundary.
• 🚨 packages/ingest/src/synthetic-csv-v1.ts:326 - Unknown skills return as unmapped before common identifier and timing validation. An unknown-skill row with an invalid source_event_id reaches commit, violates the database constraint, and rolls back valid neighboring rows. Validate shared structural fields before classifying a row as unmapped.

🔧 Fix: Harden Operations grounding, scope, and runtime boundaries
7 errors still open:

• 🚨 db/migrations/009_board_run_head.sql:221 - The least-privilege migration grants UPDATE on nonexistent claimed_at; the table and claim path use started_at. PostgreSQL rejects this statement, rolling back migration 009 and preventing the Operations schema from being installed. Grant started_at instead.
• 🚨 packages/ingest/src/synthetic-csv-v1.ts:386 - The required “strict guide CSV validation and transactional commit” remains bypassable for unmapped rows: vendor_skill is classified as unmapped without validating the value persisted to unmapped_activity. A value containing spaces or exceeding the database bound reaches commit, violates its CHECK constraint, and rolls back valid neighbors. Validate every persisted unmapped field before classification.
• 🚨 packages/db/src/imports.ts:142 - The requirement to “Keep browser data scope untrusted” is contradicted by globally selecting a sequential, browser-supplied validation ID before comparing its returned scope in application code. Bind the trusted auth/guide/studio scope and synthetic predicate into this first SQL query so another guide's receipt is never read.
• 🚨 packages/db/src/imports.ts:58 - Removing the guide-scope row lock leaves validation as SELECT-then-INSERT against the unique idempotency key. Two concurrent validations can both miss; one then fails with a unique violation instead of returning the same receipt or deterministic digest conflict. Use an atomic upsert or transaction-scoped natural-key lock.
• 🚨 packages/db/src/imports.ts:255 - The required deterministic conflict handling is still non-atomic under concurrent commits. Two receipts can both miss the same session/event; the losing ON CONFLICT DO NOTHING statement may not see the winner in its original snapshot and falsely rejects the session, while the later attempt SELECT-then-INSERT can instead hit a unique violation. Serialize or retry at the shared natural-identity boundary, then compare canonical facts.
• 🚨 packages/db/src/evidence.ts:1 - The required “engine-backed ranked entries/evidence” golden path remains incomplete: evidence is persisted, but the DB evidence module is only a placeholder and there is no guide-facing evidence route or EvidenceReader implementation. Ranked entries and additional causes therefore cannot open their exact bundle-backed attempts, sessions, prerequisite checks, or conflicts.
• 🚨 scripts/nightly.ts:3 - The replacement nightly CLI recognizes only --url and silently ignores other arguments, while the authoritative quickstart still invokes --board-date and --fixture-now. That documented verification now targets the current date/all scopes or reports ineligible instead of evaluating the requested historical cutoff. Preserve the test-only contract or update documentation and reject unsupported flags.

🔧 Fix: Harden import boundaries and nightly test dispatch
2 errors still open:

• 🚨 packages/db/src/imports.ts:216 - The new natural-identity lock set covers sessions and accepted attempts but omits unmapped activity. Two differently keyed concurrent imports of the same unknown-skill event therefore both retain unmapped_count = 1 even though one ON CONFLICT DO NOTHING insert loses; a later mapped replay can also enter attempt because that path never checks unmapped_activity. Include mapped and unmapped drafts in one source-event lock/check boundary and derive authoritative final counts from the persisted outcome.
• 🚨 scripts/nightly.ts:71 - The documented golden-path command still cannot run in its stated sequence: scripts/nightly.ts always fetches the local protected endpoint, but the quickstart invokes it before starting Next, so a clean run fails with connection refused. Start the server in a separate terminal before dispatch or provide a one-shot protected local invocation.

🔧 Fix: Unify import identities and correct nightly quickstart
2 errors still open:

• 🚨 specs/001-huddle-triage-board/quickstart.md:108 - The required “protected nightly worker dispatch” remains unreachable in the documented clean setup: the quickstart copies .env but Terminal 2 exports only NODE_ENV, while scripts/nightly.ts reads INTERNAL_REFRESH_SECRET directly from process.env without loading .env. The command therefore exits before dispatch. Explicitly load/export the root environment for the CLI.
• 🚨 packages/db/src/imports.ts:560 - The initial commit and idempotent replay return different receipts. The first response creates a new JavaScript committedAt and uses its in-memory capped issue list, while replay reads the database timestamp and a separately sorted LIMIT 100 issue set. Return the canonical persisted receipt after the commit update so the same key and digest reliably produce the same receipt.

🔧 Fix: Load nightly quickstart environment before dispatch
✅ Re-checked - no issues remain.

🔧 **Test** - 2 issues found → auto-fixed (2) ✅

• ⚠️ The authorized positive golden path—validate sample CSV, commit its receipt, refresh through PostgreSQL, and view the promoted board—could not be demonstrated end-to-end because this worktree has no .env, reachable test PostgreSQL database, or provisioned synthetic guide credentials. Decide whether the focused workflow/database tests plus rendered fail-closed evidence are sufficient, or provide a disposable configured environment for a positive browser run.
• ℹ️ apps/web/test/import-actions-fail-closed.test.ts - new test file written by agent: apps/web/test/import-actions-fail-closed.test.ts
• npx vitest run packages/ingest/test/synthetic-csv-v1.test.ts packages/application/test/operations-importer.test.ts packages/application/test/operations-refresh.test.ts packages/db/test/operations-boundaries.test.ts packages/db/test/operations-migrations.test.ts packages/narrator/test/fallback-catalog.test.ts scripts/nightly.test.ts apps/web/test/board-without-narrator.test.ts apps/web/test/quick-demo-access.test.ts apps/web/test/import-actions-fail-closed.test.ts
• Started npm run dev -w apps/web -- --hostname 127.0.0.1 --port 3100 with public placeholder Auth configuration and exercised /import and /board in Playwright using installed Brave
• Selected apps/web/public/synthetic-huddle-sample.csv and submitted validation without authorized scope; reproduced the original 500/runtime overlay, then verified the fix returns HTTP 200 with the generic fail-closed alert
• Clicked “Validate import” without a file and visually confirmed the safe validation message
• Opened /board without authorization and visually confirmed roster and freshness remain hidden
• curl -sS -i -X POST http://127.0.0.1:3100/internal/refresh -H 'content-type: application/json' --data '{"nightly":true}'
• curl -sS -i http://127.0.0.1:3100/synthetic-huddle-sample.csv
• npm run nightly -- --help

🔧 Fix: Fix disposable PostgreSQL migration blockers
1 warning still open:

• ⚠️ The authorized positive browser journey still cannot be demonstrated: Docker Desktop returns HTTP 500, and no configured PostgreSQL/Supabase environment or local PostgreSQL binary is available. Repair or authorize restarting Docker Desktop, or provide disposable synthetic-only database and guide-auth credentials, so sign-in, CSV commit, refresh, promoted board, and exact evidence can be captured.
• npx vitest run packages/ingest/test/synthetic-csv-v1.test.ts packages/application/test/operations-importer.test.ts packages/application/test/operations-refresh.test.ts packages/db/test/operations-boundaries.test.ts packages/db/test/operations-migrations.test.ts apps/web/test/board-without-narrator.test.ts apps/web/test/import-actions-fail-closed.test.ts scripts/nightly.test.ts
• docker info --format '{{json .ServerVersion}}'
• DOCKER_API_VERSION=1.45 docker info --format '{{.ServerVersion}}'
• Checked for configured DATABASE_URL and Supabase Auth variables, local PostgreSQL binaries, and vendored embedded-PostgreSQL alternatives.
• Verified git status --short and checked for generated .vitest, .next, or coverage artifacts after testing.

🔧 Fix: Fix synthetic Operations golden path failures
✅ Re-checked - no issues remain.

• docker info --format '{{json .ServerVersion}}'
• npx vitest run packages/ingest/test/synthetic-csv-v1.test.ts packages/application/test/operations-importer.test.ts packages/application/test/operations-refresh.test.ts packages/db/test/operations-boundaries.test.ts packages/db/test/operations-migrations.test.ts packages/narrator/test/fallback-catalog.test.ts apps/web/test/import-actions-fail-closed.test.ts scripts/nightly.test.ts
• Applied migrations 001–009 to disposable PostgreSQL 16 using npm run db:migrate
• Provisioned a synthetic-only guide, roster, authorization scope, seeded item, and personal baseline history
• Signed in through the rendered Auth UI, uploaded apps/web/public/synthetic-huddle-sample.csv, validated it, and committed it
• Requested manual refresh through the guide UI and inspected PostgreSQL receipt lineage, immutable runs, atomic head promotion, and deterministic fallback provenance
• Opened /board and followed Open exact evidence to the run-scoped evidence bundle
• npm run build -w packages/db to regenerate a stale ignored runtime artifact, followed by a successful exact-evidence retry
• npx vitest run apps/web/test/quick-demo-access.test.ts apps/web/test/board-without-narrator.test.ts packages/db/test/scoped-access.test.ts
• Visually inspected the four captured browser screenshots
• Stopped and removed the disposable PostgreSQL container, Auth stub, browser profile, and generated web build output

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.