From b21d0d9544895e39d52e80c9e645e922dfbd955e Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Thu, 25 Jun 2026 16:22:47 +0200 Subject: [PATCH 01/24] docs(sync): R5 remediation design spec (22 audit findings, locked contracts) Covers all 22 sync-audit findings (9 High, 9 Medium, 4 Low). Locks contracts for: parent-run heartbeat (H1), batch analysis name-reconciliation (H2/H3), all-fallback overwrite guard + confidence gate (H4), owner-attributed budget gate (H5), PII scrubber + opt-in flag (H6), is_indexed status whitelist (H7), IntegrityError->409 + model index parity (H8), adopt-not-run (H9), schema- qualified table identity (M2), and the Medium/Low set. Approach A (surgical + targeted structural). Branch + 5-wave plan defined. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../2026-06-25-sync-remediation-design.md | 534 ++++++++++++++++++ 1 file changed, 534 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-25-sync-remediation-design.md diff --git a/docs/superpowers/specs/2026-06-25-sync-remediation-design.md b/docs/superpowers/specs/2026-06-25-sync-remediation-design.md new file mode 100644 index 0000000..9f8cdb8 --- /dev/null +++ b/docs/superpowers/specs/2026-06-25-sync-remediation-design.md @@ -0,0 +1,534 @@ +# Code↔DB Sync Remediation — Design Spec (R5) + +- **Date:** 2026-06-25 +- **Author:** audit remediation (sergey@appvillis.com) +- **Branch:** `fix/sync-remediation-2026-06-25` (off current `fix/security-audit-2026-06-24`) +- **Source audit:** the 22 findings (9 High, 9 Medium, 4 Low) produced by the five-specialist sync audit on 2026-06-25 (backend, architecture, data/ML, security, QA). +- **Release framing:** **R5 — sync reliability & correctness**. Close each finding in `qa-audit/issues.md` and `[Unreleased]` in `CHANGELOG.md`. +- **Scope decision:** ALL 22 findings (user-approved). Approach **A** (surgical fixes + targeted structural changes where a finding demands it; no broad rewrite). + +> **Status legend used below:** every contract here is *locked* — types, signatures, file layout, flag names/defaults, migration shapes. The implementation plan (`docs/superpowers/plans/2026-06-25-sync-remediation.md`) turns each into TDD tasks with exact code. A zero-context implementer must not invent names; use the ones fixed here. + +--- + +## 1. Goals & non-goals + +### Goals +1. **Reliability:** a healthy daily sync is never mis-reaped; runs are tracked and observable; the single-active guard never 500s. +2. **Data correctness:** the per-table analysis the SQL agent consumes is attributed to the *right* table, is never silently replaced with garbage on LLM failure, never collapses cross-schema tables, and low-confidence/fallback rows never enforce hard SQL filters. +3. **Cost & privacy:** sync LLM calls are metered + budget-gated like chat; raw tenant data is scrubbed/denylisted (with an opt-in flag) before egress to the LLM provider. +4. **Loop-closure & schedule honesty:** freshness signals are consistent; per-project schedule hour is honored or not advertised; the stale→resync loop covers all connections. + +### Non-goals (explicitly out of scope) +- No rewrite of the three execution paths (in-process / ARQ / daily) into one. We fix their *behavioral divergence* point-by-point (M3, M5). +- No change to the chat/orchestrator budget mechanism itself — we *reuse* `UsageService.check_token_budget` / `DbUsageSink` / `LLMRouter(usage_sink=…)`. +- No new UI work beyond what the freshness `to_dict` already feeds (the Knowledge Health panel keeps working; the schedule UI keeps reading `/sync-schedule`). +- No change to db-index *sampling* logic except adding the scrub seam shared by sync (H6). + +--- + +## 2. Locked decisions (from brainstorming) + +| Decision | Locked choice | +|---|---| +| **Scope** | All 22 findings, phased into 5 waves in one plan. | +| **H6 PII** | Layered: (1) column-name **denylist** (never send verbatim), (2) regex **redaction** of values (email/phone/card/JWT/SSN), (3) opt-in flag `connection.send_sample_data_to_llm` (default **True**) + global `sync_pii_scrubbing_enabled` (default **True**). | +| **H5 budget** | Attribute to **project owner**; meter via `DbUsageSink`; **pre-flight gate**: manual routes → HTTP **429** (matches chat's existing `check_token_budget` gate, grounded in `chat.py:197-200`), cron/auto → **graceful SKIP** (log + step status `skipped`, no crash); plus a cheap mid-run check that skips the *summary* LLM call when budget is exceeded. | +| **Approach** | A — surgical + targeted structural (H1 parent heartbeat, H9 adopt-not-run). | + +> **Note on HTTP code:** the audit text said "402"; the *actual* in-repo token-budget gate (`chat.py`) returns **429**. We match the existing pattern (429) for token-budget exhaustion. EntitlementService quota→402 is a separate, pre-existing gate and is **not** added to the sync path in R5. + +--- + +## 3. Library contract verification (Context7) + +- **SQLAlchemy 2.0** (`/websites/sqlalchemy_en_20`): confirmed — after an `IntegrityError` raised from `await session.commit()`, the session is in a failed state and **must** be recovered with `await session.rollback()` before any further use; ignoring it yields the "transaction has been rolled back" error. `IntegrityError` is imported from `sqlalchemy.exc`. This grounds the **H8** contract. +- No other external-library contract changes: ARQ cron mechanics, Pydantic settings, and Alembic op API are used exactly as they already are in the repo. + +--- + +## 4. File layout (new + touched) and ownership map + +### New files +| File | Purpose | +|---|---| +| `backend/app/knowledge/pii_scrubber.py` | Pure functions: column-denylist + value-redaction for LLM egress (H6). | +| `backend/app/services/sync_budget.py` | Owner resolution + `DbUsageSink` builder + pre-flight budget verdict for sync (H5). | +| `backend/alembic/versions/_sync_remediation_connection_flag.py` | `connections.send_sample_data_to_llm` column (H6). | +| `backend/alembic/versions/_sync_remediation_indexing_run_active_index.py` | Add the partial-unique active index to the model/metadata parity (H8) — see §5.8. | +| `backend/alembic/versions/_sync_remediation_schema_qualified_uniqueness.py` | `db_index` & `code_db_sync` schema-qualified uniqueness (M2). | +| `backend/tests/unit/knowledge/test_pii_scrubber.py` | Unit tests for the scrubber. | +| `backend/tests/unit/services/test_sync_budget.py` | Unit tests for the budget helper. | +| `backend/tests/unit/knowledge/test_code_db_sync_pipeline_run.py` | First real `run()`-level orchestration tests (Steps 1-6, aborts, all-fallback guard). | + +### Touched files → owning task (no two parallel tasks write the same file) +| File | Owning task | Findings | +|---|---|---| +| `backend/app/config.py` + `backend/.env.example` | **W1-config** | all new flags | +| `backend/app/knowledge/pii_scrubber.py` (new) | **W1-pii** | H6 | +| `backend/app/services/sync_budget.py` (new) | **W1-budget** | H5 | +| `backend/app/knowledge/code_db_sync_analyzer.py` | **W2-analyzer** | H2 (tool `table_name` def + batch reconcile), H3, H4(marker) | +| `backend/app/knowledge/code_db_sync_pipeline.py` | **W2-pipeline** | H2(store), M2(match), H4(guard), H6(scrub call), H5(wire), L4(truncation markers) | +| `backend/app/knowledge/graph_db_bridge.py` | **W2-pipeline** | M7 (op_kind heuristics) | +| `backend/app/api/routes/data_investigations.py` | **W2-investigations** | M6 (producer reroute) | +| `backend/app/services/code_db_sync_service.py` | **W2-service** | M6, table-identity helper, mark_stale, L2 | +| `backend/app/agents/sql_agent.py` | **W2-sqlagent** | H4(confidence gate), M6(consumer guard) | +| `backend/app/services/db_index_service.py` + `backend/app/models/db_index.py` | **W2-dbindex** | H7, M2(constraint) | +| `backend/app/services/run_coordinator.py` + `backend/app/models/indexing_run.py` | **W3-coordinator** | H8 | +| `backend/app/services/daily_knowledge_sync_service.py` | **W3-daily** | H1, H9, M3, M4(consume), H5(skip), M5(overview) | +| `backend/app/services/stale_run_reaper.py` | **W3-reaper** | L1, H1(grace) | +| `backend/app/worker.py` | **W3-worker** | H5(sink), M3(synced_tables typo) | +| `backend/app/api/routes/connections.py` + `backend/app/models/connection.py` | **W4-connections** | H6(opt-in field+routes), H5(429 in trigger_sync) | +| `backend/app/services/knowledge_freshness_service.py` | **W4-freshness** | M8 | +| `backend/app/main.py` | **W5-main** | M4(wave honors hour), M1(reconciler all connections) | +| `backend/app/api/routes/projects.py` | **W5-projects** | M4(schedule consistency), M9(get_index_age guard surfaced) | + +Migrations are authored by their owning task; Alembic revision linearity is resolved at integration (W5). + +--- + +## 5. Locked contracts + +### 5.1 New config flags (`backend/app/config.py`, Pydantic `Settings`) +Add to the existing settings class (same style: typed field + comment). `.env.example` gets matching commented entries. + +```python +# --- R5 sync remediation ------------------------------------------------- +# H6: scrub PII / secrets from DB samples + distinct values before they are +# sent to an LLM provider. Layered with the per-connection opt-in below. +sync_pii_scrubbing_enabled: bool = True +# H4: per-table analyses below this confidence are NEVER used to enforce hard +# SQL required-filters (they may still inform soft hints). 1..5; default 2 +# means "fallback rows (confidence=1) never enforce filters". +sync_min_confidence_to_enforce_filters: int = 2 +# H4: if the fraction of NON-fallback table analyses in a sync run is below +# this, the run does NOT overwrite the previously-stored good rows; it marks +# the summary failed and keeps prior data. 0.0 disables the guard. +sync_min_success_ratio_to_persist: float = 0.5 +# H5: gate sync LLM spend on the project owner's token budget (meter + block). +sync_budget_enforcement_enabled: bool = True +``` + +`connection.send_sample_data_to_llm` is a **model column** (not a global flag) — see §5.9. + +### 5.2 `pii_scrubber.py` (new module — pure, no I/O) + +```python +"""Redact PII / secrets from DB-derived context before it reaches an LLM.""" +from __future__ import annotations + +# Column whose NAME (case-insensitive substring) implies sensitive content. +# Values for these columns are never sent verbatim — replaced with "[redacted:]". +SENSITIVE_COLUMN_TOKENS: tuple[str, ...] = ( + "password", "passwd", "secret", "token", "api_key", "apikey", "private_key", + "access_key", "credential", "ssn", "social_security", "card_number", "card_no", + "cardno", "pan", "cvv", "cvc", "iban", "swift", "auth", "session", "cookie", + "salt", "hash", # note: matched as substrings; documented in tests +) + +def is_sensitive_column(column_name: str) -> bool: + """True if the column name implies sensitive content (case-insensitive substring).""" + +def redact_value(value: str) -> str: + """Mask PII patterns inside a free string: emails, phone numbers, credit-card-like + digit runs, JWTs (eyJ...), and long hex/base64 secrets. Returns the masked string. + Non-PII text is returned unchanged. Idempotent.""" + +def scrub_distinct_values( + column_name: str, values: list, *, enabled: bool = True +) -> list: + """Return distinct values safe for an LLM prompt. + - enabled=False → return values unchanged (caller chose raw). + - sensitive column → return ["[redacted: values]"] (cardinality only, no values). + - else → [redact_value(str(v)) for v in values].""" + +def scrub_sample_json(sample_json: str, *, enabled: bool = True) -> str: + """Parse a JSON array of row dicts; for each dict, redact sensitive columns by name + and redact_value() every remaining string field. Re-serialise. On parse failure, + return redact_value(sample_json) so a non-JSON blob is still masked. enabled=False → + return input unchanged.""" +``` + +Behavioral contract: scrubbing is **on** when `settings.sync_pii_scrubbing_enabled and connection.send_sample_data_to_llm` evaluates such that — see the truth table in §5.5 (`_build_db_context`). + +### 5.3 LLM tool contract (`backend/app/llm/base.py` — unchanged shape, used as-is) +`ToolParameter` already supports `required: bool = True` and `enum`. **No change to base.py is required** beyond confirming the field exists (it does). Therefore **W1-tool is folded into W2-analyzer** (the only edit is in the tool *definition*, not the base types). The ownership map's `llm/base.py` row is dropped; analyzer owns the tool definition. + +### 5.4 Analyzer contract (`code_db_sync_analyzer.py`) + +**(a) `TableSyncAnalysis` gains a fallback marker (H4):** +```python +@dataclass +class TableSyncAnalysis: + table_name: str + ... + confidence_score: int = 3 + is_fallback: bool = False # NEW — True iff produced by _fallback_analysis +``` +`_fallback_analysis` sets `is_fallback=True`. + +**(b) `SYNC_ANALYSIS_TOOL` gains a required `table_name` parameter (H2/H3):** +```python +ToolParameter( + name="table_name", + type="string", + description=( + "The EXACT table name being analyzed, copied verbatim from the " + "'## Table: ' header. Required so results can be matched back " + "to the correct table." + ), +), +``` +Inserted as the **first** parameter. + +**(c) `analyze_table_batch` reconciles by name, not position (H2/H3):** +- Build `by_name = {t[0].lower(): (t[0], idx) for idx, t in enumerate(tables)}` (input name → canonical input name). +- For each `tc` with `tc.name == "table_sync_analysis"`: read `args.get("table_name")`; look up case-insensitively in `by_name`. + - **Hit:** build the `TableSyncAnalysis` with `table_name = ` (NOT the LLM string, NOT positional). Mark that input index as covered. Guard against duplicate coverage (if the LLM returns the same table twice, keep the first, log a warning). + - **Miss / missing `table_name`:** discard the tool call, log a warning (`"batch sync: tool call for unknown table %r — dropped"`). +- After the loop, any input table **not covered** → append `_fallback_analysis(name)` (per-table fallback, preserving order is not required since each row is self-identified by `table_name`). +- Return list length still equals `len(tables)` (every input table appears exactly once). + +**(d) Robust scalar coercion (H3 / QA-C2):** the `int(args.get("confidence_score", 3))` call is wrapped per-tool-call: +```python +def _coerce_confidence(raw) -> int: + try: + return max(1, min(5, int(float(raw)))) + except (TypeError, ValueError): + return 3 +``` +Used in both `analyze_table` and `analyze_table_batch`. A single bad scalar degrades **only that table** to confidence 3 (not fallback), never aborts the batch. + +**(e) `analyze_table` (single):** also reads `args.get("table_name")` but ALWAYS stores the input `table_name` (single-call path is already safe; the field is just for parity/telemetry). + +### 5.5 Pipeline contract (`code_db_sync_pipeline.py`) + +**(a) Schema-qualified table identity in `_match_tables` (M2):** +- Build the DB lookup keyed by `(schema_lower, name_lower)`: + ```python + db_by_key = {} # (schema_lower, name_lower) -> DbIndex + bare_name_counts = Counter() # name_lower -> count across schemas + for e in db_entries: + sch = (getattr(e, "table_schema", None) or "public").lower() + nm = e.table_name.lower() + db_by_key[(sch, nm)] = e + bare_name_counts[nm] += 1 + ``` +- A bare name is **ambiguous** iff `bare_name_counts[name_lower] > 1`. +- The stored/displayed `table_name` for a DB entry is: + - bare `e.table_name` when **not** ambiguous (back-compat — no prompt churn for single-schema installs), + - `f"{e.table_schema}.{e.table_name}"` when **ambiguous**. +- Code entities/usages match against the bare lowercased name as today, but when that bare name is ambiguous the match is recorded against **each** schema-qualified DB entry, and `confidence`/notes carry a "matched by bare name across N schemas — verify schema" caveat in `code_context`. +- `all_tables` iteration uses the qualified display name as the `_MatchedTable.table_name` so Step 5 stores distinct rows per schema. + +**(b) Step 5 store keyed by self-identified analysis name (H2):** `mt_lookup` lookup uses `analysis.table_name` which now always equals a matched display name (guaranteed by 5.4c). Add an assertion-style guard: if `analysis.table_name not in mt_lookup`, skip the row and log (defensive; should never happen). + +**(c) All-fallback / low-success guard (H4):** after Step 4 produces `analyses`, before Step 5 delete+upsert: +```python +total = len(analyses) +non_fallback = sum(1 for a in analyses if not a.is_fallback) +ratio = (non_fallback / total) if total else 0.0 +if total and ratio < settings.sync_min_success_ratio_to_persist: + # Do NOT delete_stale_tables or upsert — keep prior good rows intact. + await self._sync_svc.set_sync_status(session, connection_id, "failed") + self._tracker.end(wf_id, "code_db_sync", "failed", + f"LLM degraded: only {non_fallback}/{total} tables analyzed; " + f"kept previous sync") + return {"status": "failed", "error": "llm_degraded_kept_previous", ...} +``` +(Guard skipped when `sync_min_success_ratio_to_persist == 0.0`.) + +**(d) PII scrub in `_build_db_context` (H6):** the method gains a `scrub: bool` parameter (threaded from `run()` which knows the connection's `send_sample_data_to_llm` AND the global flag): +```python +scrub = settings.sync_pii_scrubbing_enabled and connection_send_sample_data_to_llm +# distinct values: +vals_str = " | ".join(pii_scrubber.scrub_distinct_values(col, vals[:15], enabled=...)...) +# sample data: +sample = pii_scrubber.scrub_sample_json(entry.sample_data_json, enabled=...)[:800] +``` +Truth table for what reaches the LLM: +| `sync_pii_scrubbing_enabled` | `connection.send_sample_data_to_llm` | Distinct values | Sample data | +|---|---|---|---| +| True | True | redacted (denylist+regex) | redacted | +| True | False | omitted entirely | omitted entirely | +| False | True | raw | raw | +| False | False | omitted entirely | omitted entirely | +(`send_sample_data_to_llm=False` always omits sample+distinct; the global flag only toggles redaction-vs-raw when sending is allowed.) + +**(e) L4 truncation markers:** when `len(vals) > 15`, append `f" (+{len(vals)-15} more)"`; when `sample_data_json` is truncated at 800 chars, append `"…[truncated]"`. (Display-only; no behavior change.) + +**(f) H5 budget wiring in `run()`:** at the top of `run()` (after `wf_id` is set, before Step 1): +```python +from app.services.sync_budget import build_sink, preflight_owner_budget +if settings.sync_budget_enforcement_enabled: + async with async_session_factory() as s: + ok, reason, owner_id = await preflight_owner_budget(s, project_id) + if not ok: + await self._sync_svc.set_sync_status(...“failed”/“skipped”...) + await self._tracker.end(wf_id, "code_db_sync", "failed", reason) + return {"status": "failed", "error": reason, "budget_blocked": True, ...} + self._llm = LLMRouter(usage_sink=build_sink(owner_id, project_id)) + self._analyzer = CodeDbSyncAnalyzer(self._llm) +``` +Mid-run: before the Step-6 summary LLM call, `if self._llm._sink.budget_exceeded(): skip summary, use SyncSummaryResult() default`. (Analyses already computed are persisted by Step 5; summary is additive and safely skippable.) + +> The route-level pre-flight (429 / graceful-skip) is the *primary* gate (§5.10/§5.7); this in-pipeline gate is the backstop for the cron/auto path which does not pass through a route. + +### 5.6 `code_db_sync_service.py` + +**(a) `add_runtime_enrichment` (M6):** +- For `required_filters_json`: **validate shape** before merge — the payload must be a flat `{column: condition}` dict; reject/skip keys whose value is not a string OR whose key is a known metadata token (`"source"`, `"filter"`, `"_meta"`). Only `{column: condition_string}` pairs are merged. The investigation producer (§5.6c) is changed to emit the correct shape, but the service stays defensive. +- For `column_value_mappings_json`: **deep-merge per column** — `existing[col]` and `new[col]` are both dicts → `existing[col].update(new[col])` (preserves prior value meanings); only replace wholesale when one side is non-dict. +- For appendable text fields (`query_recommendations`, `conversion_warnings`): dedupe by **normalized-line equality** (split existing into lines, compare stripped); cap total length at `8000` chars (truncate oldest, keep newest), so the column cannot grow unbounded (M6.3 / L-text). + +**(b) Confidence-aware prompt context (H4):** `sync_to_prompt_context` and the SQL-agent loaders gate on confidence — see §5.6/§5.8 wiring; the service exposes the raw `confidence_score` (already on the row), the *gating* lives in the SQL-agent consumer (§5.8) and is config-driven. + +**(c) Producer fix in `data_investigations.py::_enrich_sync_from_investigation` (M6):** change the `missing_filter` branch payload from `{"source": "investigation", "filter": inv.root_cause}` to a real filter shape, OR route it to a non-enforced field. **Locked choice:** route investigation-derived hints to `query_recommendations` (a soft, non-enforced field) instead of `required_filters_json`, since an investigation root-cause string is prose, not a `{column: condition}` map: +```python +await sync_svc.add_runtime_enrichment( + db, connection_id=inv.connection_id, table_name=table, + field="query_recommendations", + value=f"[from investigation] {inv.root_cause}", +) +``` +This removes the contract violation at the source; the service-level validation (5.6a) is the safety net. + +**(d) L2 — `touch_heartbeat` no longer fabricates a "synced_at":** when `touch_heartbeat` creates a summary row, it sets `synced_at=None` is not possible (`server_default`), so instead the fix lives in presentation: `sync_to_prompt_context` only prints the "analyzed " header when `summary.sync_status == "completed"` (or `"stale"`). For `running`/`failed`/`idle` it prints a neutral header without a fabricated date. (Owned by W2-service.) + +### 5.7 `daily_knowledge_sync_service.py` + +**(a) H1 — continuous parent heartbeat (primary fix):** `run_for_project` wraps `_orchestrate` in a heartbeat that refreshes the **parent** `daily_sync` run's `heartbeat_at` every `settings.heartbeat_interval_seconds`: +```python +from app.core.heartbeat import heartbeat +async def _hb() -> None: + async with async_session_factory() as s: + run = await s.get(IndexingRun, run_id) + if run and run.status == "running": + run.heartbeat_at = datetime.now(UTC) + await s.commit() +async with heartbeat(_hb, interval_seconds=settings.heartbeat_interval_seconds): + result = await self._orchestrate(project_id, run_id=run_id) +``` +This guarantees the reaper (300s) never kills a working multi-minute daily sync. + +**(b) M3 — progress + manifest alignment:** `_orchestrate(project_id, *, run_id)` emits coordinator step transitions for the real phases so the parent run advances past 0%. The `daily_sync` manifest in `run_manifests.py` is aligned to the phases `_orchestrate` actually runs (`plan_targets` → `repo_index` → `db_index` → `code_db_sync` → `summarize`); the dead `freshness_reconcile` step is removed from the daily manifest (the standalone reconciler loop owns reconciliation). Exact step keys are read from `run_manifests.py` by the implementing task and the manifest edited to match. Progress is emitted via lightweight `tracker.emit(run.workflow_id, step, "started"/"completed", …)` so the existing `_on_event`→`_apply_event` projection advances `progress_pct`. + +**(c) H9 — adopt-not-run on conflict:** `_start_child_wf` returns a `(wf_id | None, already_active: bool)` tuple. When `already_active` is True (the child kind+connection already has an active run, i.e. a manual/auto run is in flight), the daily sub-step **SKIPS** running its own pipeline (returns `_STEP_SKIPPED, "already running (adopted)"`) instead of launching an untracked concurrent pipeline. The soft `get_sync_status=="running"` check stays as a second line. + +**(d) M4 — honor per-project hour (consume side):** daily sync remains correct because the wave now passes only projects whose effective hour matches the current dispatch hour (the *decision* is in `main.py`, §5.11). `list_eligible_projects` additionally returns the effective hour so the wave can filter (see §5.11). + +**(e) H5 — graceful budget skip:** `_run_code_db_sync` (and `_run_db_index` for symmetry) calls `preflight_owner_budget`; if exceeded → return `_STEP_SKIPPED, "owner token budget exceeded"` (no crash, recorded in steps_json). The pipeline's own backstop (§5.5f) covers direct callers. + +**(f) M5 — regenerate overview after daily sync:** after a successful `_run_code_db_sync`, call `_regenerate_overview(project_id, connection_id)` (matching the in-process/ARQ paths) inside the `final_status == _STEP_COMPLETED` branch. + +### 5.8 `run_coordinator.py` + `indexing_run.py` (H8) + +**(a) `start()` catches the partial-unique race:** +```python +from sqlalchemy.exc import IntegrityError +... +db.add(run) +try: + await db.commit() +except IntegrityError as exc: + await db.rollback() # Context7-confirmed: required + existing = await self._find_active(db, project_id, kind, connection_id) + raise RunAlreadyActiveError(existing.id if existing else "unknown") from exc +await db.refresh(run) +``` +This converts the DB-level race into the same `RunAlreadyActiveError` callers already handle (clean 409 / adopt), and recovers the session so request handlers don't fail downstream. + +**(b) Model/metadata parity:** add the partial-unique index to `IndexingRun.__table_args__` so `Base.metadata.create_all` (unit tests / SQLite) enforces single-active too: +```python +Index( + "uq_indexing_runs_active_one", + "project_id", "kind", text("coalesce(connection_id, '')"), + unique=True, + sqlite_where=text("status IN ('queued','running','cancelling')"), + postgresql_where=text("status IN ('queued','running','cancelling')"), +), +``` +A no-op Alembic migration (``) documents the parity (the index already exists in prod via `a1f2b3c4d5e6`; the migration guards `op.create_index(..., if_not_exists=True)` for envs created before model parity). Tests that build schemas via `create_all` now exercise the guard. + +### 5.9 `db_index_service.py` + `db_index.py` (H7, M2) + +**(a) H7 — `is_indexed` distinguishes failed-only:** +```python +async def is_indexed(self, session, connection_id) -> bool: + summary = await self.get_summary(session, connection_id) + if not summary: + return False + status = (getattr(summary, "indexing_status", "idle") or "idle") + if status in ("running", "failed", "idle"): + return False + if status not in ("completed", "completed_partial"): + return False + return summary.indexed_at is not None +``` +i.e. only `completed`/`completed_partial` count as indexed. (`completed_partial` stays indexed per the existing R2-4 contract.) `get_index_age` (M9) gains the `None` guard: +```python +indexed_at = summary.indexed_at +if indexed_at is None: + return None +if indexed_at.tzinfo is None: + indexed_at = indexed_at.replace(tzinfo=UTC) +``` + +**(b) M2 — schema-qualified uniqueness:** `DbIndex.__table_args__` unique constraint becomes `(connection_id, table_schema, table_name)` (name `uq_db_index_conn_schema_table`); migration `` drops `uq_db_index_conn_table` and creates the new one. The db_index *upsert* path (in `db_index_service`/`db_index_pipeline`) keys on `(connection_id, table_schema, table_name)`. The same schema-aware uniqueness is applied to `code_db_sync` only if needed — since `_match_tables` now stores schema-qualified names on collision (§5.5a), the existing `(connection_id, table_name)` constraint on `CodeDbSync` already keeps qualified names distinct; **no CodeDbSync constraint change** is required (locked: leave it). + +> The db_index upsert anchor is read by the implementing task; the contract (key tuple `(connection_id, table_schema, table_name)`) is fixed here. + +### 5.10 `sync_budget.py` (new — H5) + +```python +"""Owner-attributed budget gate + usage sink for the code↔DB sync pipeline.""" +from __future__ import annotations +from sqlalchemy.ext.asyncio import AsyncSession + +async def resolve_owner_user_id(session: AsyncSession, project_id: str) -> str | None: + """Return Project.owner_id for the project, or None if the project is gone.""" + +def build_sink(owner_user_id: str, project_id: str): + """Return a DbUsageSink(user_id=owner_user_id, project_id=project_id).""" + +async def preflight_owner_budget( + session: AsyncSession, project_id: str +) -> tuple[bool, str | None, str | None]: + """(_enabled-aware) Return (ok, reason, owner_user_id). + - owner missing → (False, "project owner not found", None) + - check_token_budget(owner) returns a message → (False, message, owner_id) + - else → (True, None, owner_id) + When settings.sync_budget_enforcement_enabled is False → always (True, None, owner_id).""" +``` +Uses `UsageService.check_token_budget(session, owner_id)` (grounded signature) and `DbUsageSink(user_id=…, project_id=…)`. + +### 5.11 `main.py` (M4, M1) + +**(a) M4 — wave honors per-project hour:** `_dispatch_daily_knowledge_sync_wave` computes the current hour in `daily_knowledge_sync_timezone` and dispatches a project only when its **effective** schedule hour equals the current hour. The cron loop wakes hourly (or computes the next per-project boundary). **Locked simplest correct design:** the loop wakes at the top of every hour; the wave filters `eligible` to projects whose `SyncScheduleService.effective(...)["hour"] == current_local_hour`. `list_eligible_projects` already filters on `enabled`; the hour filter is added in the wave. The Redis day-lock becomes an **hour-lock** `cron:daily_sync:{run_date}:{hour}` so each hour's wave is single-flight. + +**(b) M1 — reconciler covers all connections:** `_freshness_reconcile` iterates **all** connections of each project (not just `connections[0]`) and calls `maybe_autostart_db_index` / `maybe_autostart_sync` per connection. `KnowledgeFreshnessService.evaluate` is called per connection. (Still gated by `freshness_reconciler_enabled`, default off — behavior unchanged when disabled; the fix is correctness when enabled.) + +### 5.12 `knowledge_freshness_service.py` (M8) + +- `warnings: list[str] = field(default_factory=list)` (no more `None` default); drop the `# type: ignore`. +- `to_summary`/`overall_stale` keep working on the list. +- Keep `warnings` and `details` appended in lockstep inside `_warn` (already the case); add a class-level docstring noting `to_dict` serialises `details` and `to_summary` serialises `warnings` and both are filled by `_warn` only. +- `stale` vs `failed` distinction for triggering: `sync_stale` stays True for both, but add `sync_failed: bool` field set only when `sync_status == "failed"`. The reconciler (M1) uses `sync_stale` to resync but applies a **failed-backoff**: when `sync_failed`, only resync if the last failed run is older than `settings.heartbeat_interval_seconds * N` (locked: skip auto-resync of a `failed` sync more than once per reconcile cycle — i.e. the reconciler resyncs `stale` immediately but a `failed` sync at most once per `freshness_reconciler` interval, preventing the retry-storm). Minimal lock: add `sync_failed` and have the reconciler treat `failed` with a one-shot guard keyed in-memory per connection per cycle. + +### 5.13 `connections.py` + `connection.py` (H6 opt-in, H5 429) + +- `Connection` model gains `send_sample_data_to_llm: Mapped[bool] = mapped_column(Boolean, default=True, server_default="1")`. Migration ``. Surfaced in `ConnectionCreate` (default True) and `ConnectionResponse`. +- `trigger_sync` route: before dispatch, `ok, reason, _ = await preflight_owner_budget(db, conn.project_id)`; if not ok → `raise HTTPException(status_code=429, detail=reason)`. (Mirrors chat.) +- `sync_now` route (in `projects.py`, W5): same 429 pre-flight. + +### 5.14 `worker.py` (H5 sink, M3 typo) + +- `run_code_db_sync` builds the budget sink the same way (or relies on the pipeline's internal §5.5f wiring — **locked: rely on pipeline-internal wiring**, so the worker just calls the pipeline; no duplicate sink construction). The worker's pre-flight is the pipeline's backstop. +- Fix the log key: `result.get("synced")` (was `synced_tables`) so `matched=` logs correctly. + +### 5.15 `stale_run_reaper.py` (L1, H1-grace) + +- L1: when any driver returns `-1` rowcount, still log at INFO that a sweep ran and that the exact count is unknown (`"reaper swept (rowcount unknown on this driver)"`), so a real reap is never invisible. Keep `max(0, …)` for the numeric sum. +- H1 secondary grace: `_stale_run` additionally protects a run whose `heartbeat_at` is non-null but younger than cutoff (already correct) — no change needed once §5.7a heartbeats the parent. (No reaper change strictly required for H1; the heartbeat is the fix. L1 is the only reaper edit.) + +--- + +## 6. Per-finding → fix traceability (all 22) + +| ID | Sev | Root cause (verified) | Fix (contract §) | Owning task | +|---|---|---|---|---| +| H1 | High | Parent `daily_sync` heartbeat frozen → reaped at 5 min; result discarded | Continuous parent heartbeat §5.7a (+ progress §5.7b) | W3-daily, W3-reaper(L1) | +| H2 | High | Batch analysis attributed by tool-call position | Required `table_name` + name reconciliation §5.4b/c, store §5.5b | W2-analyzer, W2-pipeline | +| H3 | High | One bad `confidence_score` aborts whole batch | Per-call `_coerce_confidence` §5.4d | W2-analyzer | +| H4 | High | All-fallback run overwrites good rows; conf decorative | `is_fallback` marker §5.4a, success-ratio guard §5.5c, confidence gate §5.6b/§5.8(sqlagent) | W2-analyzer/pipeline/sqlagent | +| H5 | High | Sync LLM bypasses budget/usage gate | `sync_budget` §5.10, pipeline wire §5.5f, route 429 §5.13/§5.7e, meter via DbUsageSink | W1-budget, W2-pipeline, W3-daily, W4/W5 routes | +| H6 | High | Raw samples/distinct → LLM, no scrub | `pii_scrubber` §5.2, scrub in `_build_db_context` §5.5d, opt-in column §5.13 | W1-pii, W2-pipeline, W4-connections | +| H7 | High | `is_indexed` True for failed-only index | status whitelist §5.9a | W2-dbindex | +| H8 | High | `start()` race → 500 not 409; index missing from model | catch `IntegrityError`+rollback §5.8a, model parity §5.8b | W3-coordinator | +| H9 | High | Daily runs untracked concurrent pipeline on conflict | adopt-not-run §5.7c | W3-daily | +| M1 | Med | Stale→resync loop only covers connection[0]; off by default | reconciler all-connections §5.11b | W5-main | +| M2 | Med | Same-name cross-schema tables collapse | schema-qualified identity §5.5a + uniqueness §5.9b | W2-pipeline, W2-dbindex | +| M3 | Med | Parent run 0%→100%; freshness_reconcile dead | progress steps + manifest align §5.7b | W3-daily | +| M4 | Med | Per-project hour advertised, ignored by cron | wave honors hour §5.11a | W5-main | +| M5 | Med | daily sync skips `_regenerate_overview`; `synced_tables` typo | overview §5.7f, log key §5.14 | W3-daily, W3-worker | +| M6 | Med | Enrichment writes `{source,filter}` into required_filters; shallow merge | validate+deep-merge §5.6a, producer reroute §5.6c | W2-service, (W2-investigations) | +| M7 | Med | graph op_kind heuristics presented as authoritative | label op_kind "(heuristic)", trim over-broad write verbs, drop fabricated `depth` from prompt §6note | W2-pipeline (graph_db_bridge) | +| M8 | Med | freshness `warnings=None`, source split, stale==failed | dataclass §5.12 | W4-freshness | +| M9 | Med | `get_index_age` AttributeError on NULL indexed_at | None guard §5.9a | W2-dbindex | +| L1 | Low | reaper `-1` rowcount → "0 reaped" log hides work | INFO-log unknown-count §5.15 | W3-reaper | +| L2 | Low | `touch_heartbeat` fabricates synced_at header | header gating §5.6d | W2-service | +| L3 | Low | daily child orphaned to reaper; two terminal writers | covered by H9 (adopt) + H1 heartbeat; document divergence note | W3-daily/coordinator | +| L4 | Low | distinct[:15]/sample[:800] truncation unmarked; substring relevance | truncation markers §5.5e; word-boundary relevance match (graph/enum) | W2-pipeline | + +**M7 detail (locked):** in `graph_db_bridge.py`, (1) move `process_`, `handle_`, `sync_`, `set_`, `add_`, `register_` out of `_WRITE_VERBS` into a new `_AMBIGUOUS_VERBS` → `op_kind="unknown"`; (2) the pipeline prompt line drops the fabricated `depth=` and labels op as `op={op_kind} (heuristic)`; (3) `_estimate_depth` is kept only for ranking, not printed. **L4 relevance detail (locked):** the substring `in` checks for enums/services/scopes/constants in `_build_code_context` switch to word-boundary/exact-token match (`table_lower == x or table_lower in tokenize(x)`). + +--- + +## 7. Sync request lifecycle (after R5) + +``` +trigger_sync / sync_now (route) + → require_role(editor) → preflight_owner_budget → 429 if exceeded + → RunCoordinator.start(code_db_sync|daily_sync) # IntegrityError→RunAlreadyActiveError→409 + → dispatch (ARQ | in-proc | daily) + +daily _orchestrate(run_id) [wrapped in parent heartbeat 30s] + → coord.step plan_targets → repo_index → (per conn) db_index → code_db_sync → summarize + → each sub-step: adopt-not-run if a manual run is active; budget-skip if owner over budget + +CodeDbSyncPipeline.run + → preflight_owner_budget (backstop) → build LLMRouter(usage_sink=owner sink) + → load code knowledge / db index + → _match_tables (schema-qualified identity) + → analyze (per-call confidence coercion; batch reconciled by table_name) + → all-fallback guard: if success-ratio < threshold → keep prior rows, mark failed, stop + → store (delete_stale + upsert by self-identified name) + → summary (skipped if budget exceeded mid-run) + → _regenerate_overview + +DB egress to LLM: _build_db_context → pii_scrubber (denylist+redact | omit | raw) per truth table +SQL agent consumption: required_filters enforced only when confidence ≥ threshold +``` + +--- + +## 8. Error handling & graceful degradation +- Budget exceeded: manual → 429 (actionable message); cron/auto → step `skipped` + logged, never crashes the wave; pipeline backstop → run `failed` with `budget_blocked`. +- LLM degraded (all/most fallback): prior good sync is **preserved**; run marked `failed`; freshness surfaces it; no garbage overwrites filters. +- `IntegrityError` race: recovered via rollback → clean `RunAlreadyActiveError` → 409/adopt. +- PII scrub parse failure: falls back to whole-string redaction (never raw). +- Reaper: continuous parent heartbeat prevents false-positive reaps; reaper still recovers genuinely dead runs. + +## 9. Testing strategy (TDD — every task: failing → impl → green) +- **pii_scrubber:** email/phone/card/JWT masking; denylist columns → cardinality only; opt-out → omit; non-JSON sample → masked; idempotency. +- **analyzer:** batch reorder / skip / extra-call / unknown-name → correct attribution + per-table fallback; non-numeric/`"4.5"`/`"high"` confidence → that table conf=3, others intact; `table_name` echoed. +- **pipeline run():** first real orchestration tests — empty knowledge / empty db abort; all-fallback guard keeps prior rows; schema-qualified identity for colliding names; scrub applied; budget pre-flight blocks. +- **sync_budget:** owner missing; enforcement off → always ok; over-budget → reason. +- **is_indexed:** failed-only → False; completed_partial → True; get_index_age None-guard. +- **run_coordinator:** simulated `IntegrityError` on commit → `RunAlreadyActiveError` + session usable (rollback); `create_all` enforces single-active. +- **daily:** parent heartbeat refreshes during a long orchestrate (reaper does not reap); adopt-not-run skips when manual active; budget skip; overview regen called. +- **reaper:** `-1` rowcount logs sweep; healthy heartbeated parent survives. +- **freshness:** default warnings list; stale vs failed; all-connections reconcile. +- **main wave:** only projects whose effective hour == current hour are dispatched; hour-lock single-flight. +- CI gates unchanged: ruff format+check, mypy, 72% combined coverage (each task adds tests to hold/raise coverage), retrieval eval untouched. + +## 10. Rollout / back-compat / flags +- All new behavior is either a bugfix (no flag) or gated by a flag defaulting to the safe value (`sync_pii_scrubbing_enabled=True`, `sync_budget_enforcement_enabled=True`, `send_sample_data_to_llm=True`). +- Migrations are additive/locked: new column (default True), index parity (if-not-exists), schema-qualified unique constraint (drop+create within one migration, guarded for SQLite). +- No data backfill required; existing CodeDbSync rows remain valid (bare names stay bare unless a future re-sync detects a collision). +- Disabling `sync_budget_enforcement_enabled` restores pre-R5 metering-off behavior; disabling `sync_pii_scrubbing_enabled` restores raw egress (for trusted single-tenant self-host). + +## 11. Open risks / watch-items +- **M2 schema qualification** changes stored `table_name` for *colliding* tables only — verify the SQL agent prompt + required-filter guard accept `schema.table` strings (they already inject `WHERE` clauses table-by-table; qualified names are inert text). Covered by a sql_agent test. +- **M3 manifest edit** must keep other consumers of the `daily_sync` manifest (UI step rendering) working — the implementing task verifies `run_manifests.py` consumers. +- **db_index upsert key change (M2)** must be applied atomically with the constraint migration to avoid an upsert hitting the old constraint mid-deploy — sequenced in W2-dbindex. +- **Heartbeat overhead:** one extra UPDATE per parent run every 30s — negligible. + +## 12. Wave plan (dependency graph for the implementation plan) +- **Wave 1 (sequential foundation):** W1-config (flags) → then parallel: W1-pii, W1-budget. Contracts/types only; no consumers yet. +- **Wave 2 (parallel; correctness):** W2-analyzer → W2-pipeline (depends analyzer+pii+budget); parallel W2-service, W2-sqlagent, W2-dbindex, W2-investigations(producer). File ownership disjoint. +- **Wave 3 (parallel; reliability):** W3-coordinator, W3-daily (depends budget+coordinator), W3-reaper, W3-worker. +- **Wave 4 (parallel; egress/freshness):** W4-connections (model+migration+routes; depends budget+pii), W4-freshness. +- **Wave 5 (sequential glue):** W5-main (wave/reconciler), W5-projects (schedule/route), migration linearization, CHANGELOG + qa-audit/issues.md closure, full `make check` + frontend tsc/lint, final validation cycle (§ plan). + +Each task: exact `file:line` anchors, complete code, failing-test-first, conventional commit, explicit DoD. Parallel-group tasks share no files (table in §4). From a67e922096c7fd54686e397100d3e6f4081db8fb Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Thu, 25 Jun 2026 16:30:41 +0200 Subject: [PATCH 02/24] docs(sync): R5 implementation plan (18 TDD tasks, 5 waves, zero-context) Turns the R5 design spec into 18 bite-sized TDD tasks across 5 waves with a dependency graph and disjoint file ownership for parallel subagents. Each task: exact file:line anchors, complete code, failing-test-first, exact commands, conventional commit, DoD. Covers all 22 audit findings; T18 includes the per-finding business-logic validation cycle. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../plans/2026-06-25-sync-remediation.md | 1691 +++++++++++++++++ 1 file changed, 1691 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-25-sync-remediation.md diff --git a/docs/superpowers/plans/2026-06-25-sync-remediation.md b/docs/superpowers/plans/2026-06-25-sync-remediation.md new file mode 100644 index 0000000..c34627a --- /dev/null +++ b/docs/superpowers/plans/2026-06-25-sync-remediation.md @@ -0,0 +1,1691 @@ +# Code↔DB Sync Remediation — Implementation Plan (R5) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Close all 22 findings of the 2026-06-25 code↔DB sync audit (reliability, data correctness, cost/privacy, schedule honesty) without rewriting the existing pipeline. + +**Architecture:** Surgical fixes plus two targeted structural changes (continuous parent-run heartbeat for daily sync; adopt-not-run for the daily child sub-steps). Two new pure/helper modules (`pii_scrubber`, `sync_budget`). LLM batch results are reconciled by table name instead of position; sync LLM spend is metered + budget-gated like chat; DB samples are scrubbed before LLM egress. + +**Tech Stack:** Python 3.12, FastAPI, SQLAlchemy 2.0 async (`asyncpg`/`aiosqlite`), Alembic, ARQ, pytest (`asyncio_mode=auto`), ruff `0.15.15`, mypy. + +**Companion spec:** `docs/superpowers/specs/2026-06-25-sync-remediation-design.md` (all contracts locked there; § references below point into it). + +## Global Constraints + +- **Branch:** `fix/sync-remediation-2026-06-25` (already created off `fix/security-audit-2026-06-24`). +- **Line length 100; ruff rules `E F I N W UP`; ruff + mypy pinned** — do not widen. Run `cd backend && .venv/bin/ruff format app/ tests/ && .venv/bin/ruff check app/ tests/ && .venv/bin/mypy app/ --ignore-missing-imports` before each commit. +- **Async everywhere** — no sync I/O on the request path; SQLAlchemy 2.0 async sessions via `async_session_factory`. +- **TDD mandatory** — failing test → confirm fail → minimal impl → confirm pass → commit. `asyncio_mode="auto"` (no `@pytest.mark.asyncio`). +- **Coverage gate 72% combined** (`coverage report --fail-under=72`). Every task adds tests; never lower coverage. +- **Conventional commits**, prefix `fix(sync):` / `feat(sync):` / `security(sync):` / `test(sync):` / `docs(sync):`. +- **New env vars** → `backend/app/config.py` (docstring) **and** `backend/.env.example`. +- **Single-active guard:** SQLAlchemy `IntegrityError` is imported from `sqlalchemy.exc`; after it is raised by `await session.commit()` the session MUST be recovered with `await session.rollback()` before reuse (Context7-confirmed). +- **No two parallel tasks write the same file** — ownership table in spec §4 is authoritative. +- **Final task** closes each finding ID in `qa-audit/issues.md` and adds a CHANGELOG `[Unreleased]` entry. + +--- + +## Dependency graph & parallel groups + +``` +Wave 1 (foundation): T1 (config) ──► T2 (pii_scrubber) ‖ T3 (sync_budget) +Wave 2 (correctness): T4 (analyzer) ──► T5 (pipeline) [T5 depends T2,T3,T4] + T6 (service) ‖ T7 (sqlagent) ‖ T8 (dbindex subsystem) ‖ T9 (investigations producer) +Wave 3 (reliability): T10 (coordinator) ──► T11 (daily) [T11 depends T3,T10] + T12 (reaper) ‖ T13 (worker) +Wave 4 (egress/fresh): T14 (connections: flag+migration+routes) [depends T3] ‖ T15 (freshness) +Wave 5 (glue): T16 (main) ‖ T17 (projects) ──► T18 (integration: migrations linearize, CHANGELOG, issues, make check, validation cycle) +``` +Parallelizable within a wave = tasks on the same line separated by `‖`. `──►` = ordering dependency. + +--- + +# WAVE 1 — Foundation (contracts only, no consumers) + +## Task T1: Config flags + .env.example + +**Files:** +- Modify: `backend/app/config.py` (the `Settings` class; add after the existing `db_index_batch_size` block near line 315) +- Modify: `backend/.env.example` +- Test: `backend/tests/unit/test_config.py` (add a test; create if absent) + +**Interfaces — Produces:** +- `settings.sync_pii_scrubbing_enabled: bool = True` +- `settings.sync_min_confidence_to_enforce_filters: int = 2` +- `settings.sync_min_success_ratio_to_persist: float = 0.5` +- `settings.sync_budget_enforcement_enabled: bool = True` + +- [ ] **Step 1: Write the failing test** + +```python +# backend/tests/unit/test_config.py (append) +def test_r5_sync_remediation_defaults(): + from app.config import settings + assert settings.sync_pii_scrubbing_enabled is True + assert settings.sync_min_confidence_to_enforce_filters == 2 + assert settings.sync_min_success_ratio_to_persist == 0.5 + assert settings.sync_budget_enforcement_enabled is True +``` + +- [ ] **Step 2: Run it — expect FAIL** (`AttributeError`) + +`cd backend && .venv/bin/pytest tests/unit/test_config.py::test_r5_sync_remediation_defaults -v` +Expected: FAIL (`'Settings' object has no attribute 'sync_pii_scrubbing_enabled'`). + +- [ ] **Step 3: Add the fields** (in `Settings`, after `db_index_batch_size: int = 5`) + +```python + # --- R5 sync remediation ------------------------------------------------- + # H6: scrub PII / secrets from DB samples + distinct values before LLM egress. + sync_pii_scrubbing_enabled: bool = True + # H4: per-table analyses below this confidence never enforce hard SQL filters. + sync_min_confidence_to_enforce_filters: int = 2 + # H4: if the fraction of non-fallback analyses is below this, keep prior rows + # instead of overwriting with a degraded run. 0.0 disables the guard. + sync_min_success_ratio_to_persist: float = 0.5 + # H5: gate sync LLM spend on the project owner's token budget. + sync_budget_enforcement_enabled: bool = True +``` + +- [ ] **Step 4: Mirror in `.env.example`** + +```bash +# R5 sync remediation +SYNC_PII_SCRUBBING_ENABLED=true +SYNC_MIN_CONFIDENCE_TO_ENFORCE_FILTERS=2 +SYNC_MIN_SUCCESS_RATIO_TO_PERSIST=0.5 +SYNC_BUDGET_ENFORCEMENT_ENABLED=true +``` + +- [ ] **Step 5: Run test — expect PASS**, then ruff/mypy. + +- [ ] **Step 6: Commit** + +```bash +git add backend/app/config.py backend/.env.example backend/tests/unit/test_config.py +git commit -m "feat(sync): R5 config flags (pii scrub, confidence gate, success-ratio guard, budget)" +``` + +--- + +## Task T2: `pii_scrubber.py` (H6) + +**Files:** +- Create: `backend/app/knowledge/pii_scrubber.py` +- Test: `backend/tests/unit/knowledge/test_pii_scrubber.py` + +**Interfaces — Produces** (spec §5.2): +- `is_sensitive_column(column_name: str) -> bool` +- `redact_value(value: str) -> str` +- `scrub_distinct_values(column_name: str, values: list, *, enabled: bool = True) -> list` +- `scrub_sample_json(sample_json: str, *, enabled: bool = True) -> str` +- `scrub_row_cells(columns: list[str], rows: list[list], *, enabled: bool = True) -> list[list]` + +- [ ] **Step 1: Write the failing tests** + +```python +# backend/tests/unit/knowledge/test_pii_scrubber.py +from app.knowledge import pii_scrubber as p + + +def test_sensitive_column_detection(): + assert p.is_sensitive_column("password_hash") + assert p.is_sensitive_column("user_API_Key") + assert not p.is_sensitive_column("created_at") + + +def test_redact_email_phone_card_jwt(): + assert "[redacted-email]" in p.redact_value("contact a@b.com please") + assert "[redacted-card]" in p.redact_value("4111 1111 1111 1111") + assert "[redacted-jwt]" in p.redact_value("eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.abc.def") + assert p.redact_value("just text") == "just text" + + +def test_scrub_distinct_sensitive_column_returns_cardinality_only(): + out = p.scrub_distinct_values("email", ["a@b.com", "c@d.com"]) + assert out == ["[redacted: 2 values]"] + + +def test_scrub_distinct_disabled_returns_raw(): + assert p.scrub_distinct_values("email", ["a@b.com"], enabled=False) == ["a@b.com"] + + +def test_scrub_sample_json_redacts_and_survives_bad_json(): + good = '[{"email": "a@b.com", "note": "call 415-555-1212"}]' + out = p.scrub_sample_json(good) + assert "a@b.com" not in out and "415-555-1212" not in out + # non-JSON still masked, not raw + assert "a@b.com" not in p.scrub_sample_json("raw blob a@b.com") + assert p.scrub_sample_json("anything", enabled=False) == "anything" + + +def test_scrub_row_cells_redacts_sensitive_columns(): + cols = ["id", "password", "email"] + rows = [[1, "hunter2", "a@b.com"]] + out = p.scrub_row_cells(cols, rows) + assert out[0][0] == 1 + assert "hunter2" not in str(out[0][1]) + assert "a@b.com" not in str(out[0][2]) +``` + +- [ ] **Step 2: Run — expect FAIL** (module missing). + +`cd backend && .venv/bin/pytest tests/unit/knowledge/test_pii_scrubber.py -v` + +- [ ] **Step 3: Implement `pii_scrubber.py`** + +```python +"""Redact PII / secrets from DB-derived context before it reaches an LLM. + +Pure functions, no I/O. Used by the DB-index validator and the code↔DB sync +analyzer (the two places raw tenant sample data would otherwise egress to an +LLM provider). See spec §5.2. +""" +from __future__ import annotations + +import json +import re + +SENSITIVE_COLUMN_TOKENS: tuple[str, ...] = ( + "password", "passwd", "secret", "token", "api_key", "apikey", "private_key", + "access_key", "credential", "ssn", "social_security", "card_number", "card_no", + "cardno", "pan", "cvv", "cvc", "iban", "swift", "auth", "session", "cookie", + "salt", "hash", +) + +_EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}") +_JWT = re.compile(r"eyJ[A-Za-z0-9_-]{5,}\.[A-Za-z0-9_-]{5,}\.[A-Za-z0-9_-]{5,}") +_CARD = re.compile(r"\b(?:\d[ -]?){13,19}\b") +_PHONE = re.compile(r"\b\+?\d[\d\s().-]{7,}\d\b") +# long hex / base64-ish secrets (>= 24 chars, no spaces) +_SECRETISH = re.compile(r"\b[A-Za-z0-9+/=_-]{24,}\b") + + +def is_sensitive_column(column_name: str) -> bool: + name = (column_name or "").lower() + return any(tok in name for tok in SENSITIVE_COLUMN_TOKENS) + + +def redact_value(value: str) -> str: + if not value: + return value + s = str(value) + s = _JWT.sub("[redacted-jwt]", s) + s = _EMAIL.sub("[redacted-email]", s) + s = _CARD.sub("[redacted-card]", s) + s = _PHONE.sub("[redacted-phone]", s) + s = _SECRETISH.sub("[redacted-secret]", s) + return s + + +def scrub_distinct_values(column_name: str, values: list, *, enabled: bool = True) -> list: + if not enabled: + return values + if is_sensitive_column(column_name): + return [f"[redacted: {len(values)} values]"] + return [redact_value(str(v)) for v in values] + + +def scrub_sample_json(sample_json: str, *, enabled: bool = True) -> str: + if not enabled or not sample_json: + return sample_json + try: + rows = json.loads(sample_json) + except (json.JSONDecodeError, TypeError): + return redact_value(sample_json) + if not isinstance(rows, list): + return redact_value(sample_json) + cleaned = [] + for row in rows: + if isinstance(row, dict): + cleaned.append( + { + k: ("[redacted]" if is_sensitive_column(str(k)) + else (redact_value(v) if isinstance(v, str) else v)) + for k, v in row.items() + } + ) + else: + cleaned.append(redact_value(row) if isinstance(row, str) else row) + return json.dumps(cleaned, default=str) + + +def scrub_row_cells(columns: list[str], rows: list[list], *, enabled: bool = True) -> list[list]: + if not enabled: + return rows + sensitive_idx = {i for i, c in enumerate(columns) if is_sensitive_column(str(c))} + out: list[list] = [] + for row in rows: + new_row = [] + for i, cell in enumerate(row): + if i in sensitive_idx: + new_row.append("[redacted]") + elif isinstance(cell, str): + new_row.append(redact_value(cell)) + else: + new_row.append(cell) + out.append(new_row) + return out +``` + +- [ ] **Step 4: Run tests — expect PASS**, then ruff/mypy. +- [ ] **Step 5: Commit** + +```bash +git add backend/app/knowledge/pii_scrubber.py backend/tests/unit/knowledge/test_pii_scrubber.py +git commit -m "security(sync): add pii_scrubber (denylist + value redaction) for LLM egress (H6)" +``` + +--- + +## Task T3: `sync_budget.py` (H5) + +**Files:** +- Create: `backend/app/services/sync_budget.py` +- Test: `backend/tests/unit/services/test_sync_budget.py` + +**Interfaces — Consumes:** `UsageService.check_token_budget(db, user_id) -> str | None`; `DbUsageSink(user_id=…, project_id=…)`; `Project.owner_id`; `settings.sync_budget_enforcement_enabled`. +**Produces** (spec §5.10): +- `resolve_owner_user_id(session, project_id) -> str | None` +- `build_sink(owner_user_id, project_id) -> DbUsageSink` +- `preflight_owner_budget(session, project_id) -> tuple[bool, str | None, str | None]` → `(ok, reason, owner_user_id)` + +- [ ] **Step 1: Write the failing tests** + +```python +# backend/tests/unit/services/test_sync_budget.py +import pytest +from app.services import sync_budget + + +class _FakeProject: + def __init__(self, owner_id): + self.owner_id = owner_id + + +@pytest.fixture +def patch_owner(monkeypatch): + def _set(owner): + async def _resolve(session, project_id): + return owner + monkeypatch.setattr(sync_budget, "resolve_owner_user_id", _resolve) + return _set + + +async def test_preflight_disabled_always_ok(monkeypatch, patch_owner): + monkeypatch.setattr(sync_budget.settings, "sync_budget_enforcement_enabled", False) + patch_owner("u1") + ok, reason, owner = await sync_budget.preflight_owner_budget(None, "p1") + assert ok is True and reason is None and owner == "u1" + + +async def test_preflight_blocks_when_budget_message(monkeypatch, patch_owner): + monkeypatch.setattr(sync_budget.settings, "sync_budget_enforcement_enabled", True) + patch_owner("u1") + + async def _budget(db, user_id): + return "daily token budget exhausted" + + monkeypatch.setattr(sync_budget._usage_svc, "check_token_budget", _budget) + ok, reason, owner = await sync_budget.preflight_owner_budget(None, "p1") + assert ok is False and "budget" in reason and owner == "u1" + + +async def test_preflight_owner_missing(monkeypatch): + monkeypatch.setattr(sync_budget.settings, "sync_budget_enforcement_enabled", True) + + async def _resolve(session, project_id): + return None + + monkeypatch.setattr(sync_budget, "resolve_owner_user_id", _resolve) + ok, reason, owner = await sync_budget.preflight_owner_budget(None, "p1") + assert ok is False and owner is None +``` + +- [ ] **Step 2: Run — expect FAIL** (module missing). + +- [ ] **Step 3: Implement `sync_budget.py`** + +```python +"""Owner-attributed budget gate + usage sink for the code↔DB sync pipeline (H5).""" +from __future__ import annotations + +import logging + +from sqlalchemy import select +from sqlalchemy.ext.asyncio import AsyncSession + +from app.config import settings +from app.llm.usage_sink import DbUsageSink +from app.models.project import Project +from app.services.usage_service import UsageService + +logger = logging.getLogger(__name__) +_usage_svc = UsageService() + + +async def resolve_owner_user_id(session: AsyncSession, project_id: str) -> str | None: + row = await session.execute(select(Project.owner_id).where(Project.id == project_id)) + return row.scalar_one_or_none() + + +def build_sink(owner_user_id: str, project_id: str) -> DbUsageSink: + return DbUsageSink(user_id=owner_user_id, project_id=project_id) + + +async def preflight_owner_budget( + session: AsyncSession, project_id: str +) -> tuple[bool, str | None, str | None]: + owner_id = await resolve_owner_user_id(session, project_id) + if not owner_id: + return False, "project owner not found", None + if not settings.sync_budget_enforcement_enabled: + return True, None, owner_id + msg = await _usage_svc.check_token_budget(session, owner_id) + if msg: + return False, msg, owner_id + return True, None, owner_id +``` + +- [ ] **Step 4: Run tests — expect PASS**, ruff/mypy. +- [ ] **Step 5: Commit** + +```bash +git add backend/app/services/sync_budget.py backend/tests/unit/services/test_sync_budget.py +git commit -m "feat(sync): owner-attributed budget pre-flight + usage sink helper (H5)" +``` + +--- + +# WAVE 2 — Data correctness + +## Task T4: Analyzer — `table_name` echo, name reconciliation, robust confidence, fallback marker (H2, H3, H4) + +**Files:** +- Modify: `backend/app/knowledge/code_db_sync_analyzer.py` +- Test: `backend/tests/unit/knowledge/test_code_db_sync_analyzer.py` (add tests) + +**Interfaces — Produces:** `TableSyncAnalysis.is_fallback: bool`; `SYNC_ANALYSIS_TOOL` with first param `table_name`; `analyze_table_batch` reconciling by `args["table_name"]`. +**Consumes:** nothing new. + +- [ ] **Step 1: Write failing tests** + +```python +# backend/tests/unit/knowledge/test_code_db_sync_analyzer.py (append) +import pytest +from app.knowledge.code_db_sync_analyzer import CodeDbSyncAnalyzer, TableSyncAnalysis +from app.llm.base import LLMResponse, ToolCall + + +class _Router: + def __init__(self, calls): + self._calls = calls + + async def complete(self, **kwargs): + return LLMResponse(tool_calls=self._calls) + + +def _tc(table_name, conf=4, status="matched"): + return ToolCall(id="x", name="table_sync_analysis", arguments={ + "table_name": table_name, "sync_status": status, "confidence_score": conf, + "required_filters": "{}", "column_value_mappings": "{}", + }) + + +async def test_batch_reconciles_by_name_not_position(): + tables = [("orders", "", ""), ("payments", "", "")] + # LLM returns them REVERSED + analyzer = CodeDbSyncAnalyzer(_Router([_tc("payments"), _tc("orders")])) + out = await analyzer.analyze_table_batch(tables) + by_name = {a.table_name: a for a in out} + assert by_name["orders"].sync_status == "matched" + assert by_name["payments"].sync_status == "matched" + assert not by_name["orders"].is_fallback + + +async def test_batch_unknown_name_dropped_and_missing_filled_with_fallback(): + tables = [("orders", "", ""), ("payments", "", "")] + analyzer = CodeDbSyncAnalyzer(_Router([_tc("orders"), _tc("ghost_table")])) + out = await analyzer.analyze_table_batch(tables) + by_name = {a.table_name: a for a in out} + assert len(out) == 2 + assert by_name["payments"].is_fallback is True # never returned by LLM + assert by_name["orders"].is_fallback is False + + +async def test_batch_bad_confidence_only_degrades_that_table(): + tables = [("orders", "", ""), ("payments", "", "")] + bad = _tc("orders"); bad.arguments["confidence_score"] = "4.5" + analyzer = CodeDbSyncAnalyzer(_Router([bad, _tc("payments", conf=5)])) + out = await analyzer.analyze_table_batch(tables) + by_name = {a.table_name: a for a in out} + assert by_name["orders"].confidence_score == 3 # coerced, not fallback + assert by_name["orders"].is_fallback is False + assert by_name["payments"].confidence_score == 5 + + +async def test_fallback_marked(): + a = CodeDbSyncAnalyzer._fallback_analysis("t") + assert a.is_fallback is True and a.confidence_score == 1 +``` + +- [ ] **Step 2: Run — expect FAIL.** + +`cd backend && .venv/bin/pytest tests/unit/knowledge/test_code_db_sync_analyzer.py -k "reconcile or unknown_name or bad_confidence or fallback_marked" -v` + +- [ ] **Step 3: Edit `TableSyncAnalysis`** — add field (after `confidence_score: int = 3`): + +```python + is_fallback: bool = False +``` + +- [ ] **Step 4: Add `table_name` as the FIRST `ToolParameter` in `SYNC_ANALYSIS_TOOL`** (before `data_format_notes`): + +```python + ToolParameter( + name="table_name", + type="string", + description=( + "The EXACT table name being analyzed, copied verbatim from the " + "'## Table: ' header. Required so results map to the right table." + ), + ), +``` + +- [ ] **Step 5: Add a module-level coercion helper** (near `_clamp_sync_status`): + +```python +def _coerce_confidence(raw) -> int: + try: + return max(1, min(5, int(float(raw)))) + except (TypeError, ValueError): + return 3 +``` + +- [ ] **Step 6: Update `_fallback_analysis`** to set the marker: + +```python + @staticmethod + def _fallback_analysis(table_name: str) -> TableSyncAnalysis: + return TableSyncAnalysis( + table_name=table_name, + sync_status="unknown", + confidence_score=1, + data_format_notes="LLM analysis unavailable — using fallback.", + is_fallback=True, + ) +``` + +- [ ] **Step 7: Replace `analyze_table_batch` body** (lines ~266-315) with name-reconciled version: + +```python + results_by_name: dict[str, TableSyncAnalysis] = {} + by_name = {t[0].lower(): t[0] for t in tables} + try: + resp = await self._llm.complete( + messages=messages, tools=[SYNC_ANALYSIS_TOOL], + preferred_provider=preferred_provider, model=model, + temperature=0.0, max_tokens=4096, + ) + for tc in resp.tool_calls: + if tc.name != "table_sync_analysis": + continue + args = tc.arguments + raw_name = str(args.get("table_name", "")).lower() + canonical = by_name.get(raw_name) + if canonical is None: + logger.warning("batch sync: tool call for unknown table %r — dropped", + args.get("table_name")) + continue + if canonical in results_by_name: + logger.warning("batch sync: duplicate analysis for %s — keeping first", + canonical) + continue + col_notes = args.get("column_sync_notes", "{}") + if isinstance(col_notes, dict): + col_notes = json.dumps(col_notes) + results_by_name[canonical] = TableSyncAnalysis( + table_name=canonical, + data_format_notes=args.get("data_format_notes", ""), + column_sync_notes_json=col_notes, + business_logic_notes=args.get("business_logic_notes", ""), + conversion_warnings=args.get("conversion_warnings", ""), + query_recommendations=args.get("query_recommendations", ""), + required_filters_json=args.get("required_filters", "{}"), + column_value_mappings_json=args.get("column_value_mappings", "{}"), + sync_status=_clamp_sync_status(args.get("sync_status", "unknown")), + confidence_score=_coerce_confidence(args.get("confidence_score", 3)), + ) + except Exception: + logger.warning("Batch sync analysis failed", exc_info=True) + + out: list[TableSyncAnalysis] = [] + fallback_count = 0 + for name, _db, _code in tables: + if name in results_by_name: + out.append(results_by_name[name]) + else: + out.append(self._fallback_analysis(name)) + fallback_count += 1 + if fallback_count: + logger.info("LLM sync batch: %d/%d used fallback", fallback_count, len(tables)) + return out +``` + +- [ ] **Step 8: Update `analyze_table`** single-path to use `_coerce_confidence` (replace the `confidence_score=max(1, min(5, int(...)))` line): + +```python + confidence_score=_coerce_confidence(args.get("confidence_score", 3)), +``` + +- [ ] **Step 9: Run tests — expect PASS.** Re-run the full analyzer test file to confirm no regression: + +`cd backend && .venv/bin/pytest tests/unit/knowledge/test_code_db_sync_analyzer.py -v` → all PASS. ruff/mypy. + +- [ ] **Step 10: Commit** + +```bash +git add backend/app/knowledge/code_db_sync_analyzer.py backend/tests/unit/knowledge/test_code_db_sync_analyzer.py +git commit -m "fix(sync): reconcile batch analyses by table_name + robust confidence + fallback marker (H2,H3,H4)" +``` + +--- + +## Task T5: Pipeline — schema-qualified identity, all-fallback guard, scrub, budget wiring, store-by-name, truncation markers (H2-store, M2, H4, H6, H5, L4) + +**Files:** +- Modify: `backend/app/knowledge/code_db_sync_pipeline.py` +- Modify: `backend/app/knowledge/graph_db_bridge.py` (M7) +- Test: `backend/tests/unit/knowledge/test_code_db_sync_pipeline_run.py` (create), `backend/tests/unit/knowledge/test_graph_db_bridge.py` (add M7 test) + +**Interfaces — Consumes:** `pii_scrubber` (T2), `sync_budget` (T3), analyzer `is_fallback` (T4), `settings.*` (T1). +**Produces:** scrubbed `_build_db_context`; pipeline `run()` budget-gated; store keyed by self-identified name. + +- [ ] **Step 1: Write failing tests** (focus on the all-fallback guard + schema-qualified match + scrub call — pure-ish units) + +```python +# backend/tests/unit/knowledge/test_code_db_sync_pipeline_run.py +import json +import pytest +from app.knowledge.code_db_sync_pipeline import CodeDbSyncPipeline +from app.knowledge.code_db_sync_analyzer import TableSyncAnalysis + + +class _DbEntry: + def __init__(self, name, schema="public"): + self.table_name = name + self.table_schema = schema + self.business_description = "" + self.row_count = None + self.column_count = 0 + self.data_patterns = "" + self.query_hints = "" + self.column_notes_json = "{}" + self.column_distinct_values_json = json.dumps({"email": ["a@b.com", "c@d.com"]}) + self.sample_data_json = json.dumps([{"email": "a@b.com"}]) + + +def test_build_db_context_scrubs_when_enabled(): + ctx = CodeDbSyncPipeline._build_db_context(_DbEntry("users"), scrub=True) + assert "a@b.com" not in ctx + assert "[redacted: 2 values]" in ctx or "redacted" in ctx + + +def test_build_db_context_omits_when_send_disabled(): + # scrub=False path used by run() when connection opted out → caller omits; + # here we assert raw passes through when scrubbing globally off: + ctx = CodeDbSyncPipeline._build_db_context(_DbEntry("logs"), scrub=False) + assert "a@b.com" in ctx # raw allowed only when scrubbing disabled & sending allowed + + +def test_distinct_truncation_marker(): + e = _DbEntry("t") + e.column_distinct_values_json = json.dumps({"k": [str(i) for i in range(20)]}) + ctx = CodeDbSyncPipeline._build_db_context(e, scrub=True) + assert "+5 more" in ctx + + +def test_all_fallback_guard_helper(): + analyses = [TableSyncAnalysis(table_name="a", is_fallback=True), + TableSyncAnalysis(table_name="b", is_fallback=True)] + total = len(analyses) + non_fb = sum(1 for a in analyses if not a.is_fallback) + assert (non_fb / total) < 0.5 # guard would trip +``` + +- [ ] **Step 2: Run — expect FAIL** (`_build_db_context` has no `scrub` param yet). + +- [ ] **Step 3: Edit `_build_db_context`** — add `scrub: bool` param and route through `pii_scrubber`, add truncation markers (L4): + +```python + @staticmethod + def _build_db_context(entry: DbIndex, *, scrub: bool = True) -> str: + from app.knowledge import pii_scrubber + parts: list[str] = [] + if entry.business_description: + parts.append(f"Description: {entry.business_description}") + if entry.row_count is not None: + parts.append(f"Rows: ~{entry.row_count:,}") + if entry.column_count: + parts.append(f"Column count: {entry.column_count}") + if entry.data_patterns: + parts.append(f"Data patterns: {entry.data_patterns}") + if entry.query_hints: + parts.append(f"Query hints: {entry.query_hints}") + if entry.column_notes_json and entry.column_notes_json != "{}": + try: + notes = json.loads(entry.column_notes_json) + if notes: + parts.append("Column notes:") + for col, note in notes.items(): + parts.append(f" {col}: {note}") + except (json.JSONDecodeError, TypeError): + pass + dv_json = getattr(entry, "column_distinct_values_json", None) or "{}" + if dv_json and dv_json != "{}": + try: + distinct = json.loads(dv_json) + if distinct: + parts.append("Actual distinct values in DB:") + for col, vals in distinct.items(): + shown = pii_scrubber.scrub_distinct_values(col, vals[:15], enabled=scrub) + vals_str = " | ".join(str(v) for v in shown) + more = f" (+{len(vals) - 15} more)" if len(vals) > 15 else "" + parts.append(f" {col}: [{vals_str}]{more}") + except (json.JSONDecodeError, TypeError): + pass + if entry.sample_data_json and entry.sample_data_json != "[]": + sample = pii_scrubber.scrub_sample_json(entry.sample_data_json, enabled=scrub) + suffix = "…[truncated]" if len(sample) > 800 else "" + parts.append(f"Sample data: {sample[:800]}{suffix}") + return "\n".join(parts) +``` + +> **Note:** `_build_db_context` is called inside `_match_tables`. Add a `scrub` field to `_MatchedTable`? No — `_build_db_context` is called directly in `_match_tables` (line ~430). Thread `scrub` into `_match_tables(..., scrub: bool)` and pass to `_build_db_context`. The `run()` method computes `scrub` (Step 6 below) and passes it to `_match_tables`. + +- [ ] **Step 4: Thread `scrub` and schema-qualified identity into `_match_tables`** (M2). Replace the `db_table_names`/`entity_by_table` construction and the per-table loop key with schema-aware logic: + +```python + def _match_tables( + self, + knowledge: ProjectKnowledge, + db_entries: list[DbIndex], + rules_context: str = "", + *, + scrub: bool = True, + ) -> list[_MatchedTable]: + from collections import Counter + results: list[_MatchedTable] = [] + db_by_key: dict[tuple[str, str], DbIndex] = {} + bare_counts: Counter = Counter() + for e in db_entries: + sch = (getattr(e, "table_schema", None) or "public").lower() + nm = e.table_name.lower() + db_by_key[(sch, nm)] = e + bare_counts[nm] += 1 + + def _display_name(e: DbIndex) -> str: + if bare_counts[e.table_name.lower()] > 1: + return f"{getattr(e, 'table_schema', 'public') or 'public'}.{e.table_name}" + return e.table_name + + entity_by_table: dict[str, EntityInfo] = {} + code_table_names: set[str] = set() + for _, entity in knowledge.entities.items(): + if entity.table_name: + entity_by_table[entity.table_name.lower()] = entity + code_table_names.add(entity.table_name.lower()) + for tbl_name in knowledge.table_usage: + code_table_names.add(tbl_name.lower()) + + # DB-side first (schema-qualified), then code-only tables with no DB row. + seen_bare: set[str] = set() + for (sch, nm), db_entry in sorted(db_by_key.items()): + seen_bare.add(nm) + entity = entity_by_table.get(nm) + usage = knowledge.table_usage.get(nm) or knowledge.table_usage.get( + next((k for k in knowledge.table_usage if k.lower() == nm), "") + ) + ambiguous = bare_counts[nm] > 1 + display = _display_name(db_entry) + code_context = self._build_code_context(entity, usage, knowledge, nm, rules_context) + if ambiguous: + code_context = ( + f"(NOTE: table name '{nm}' exists in multiple schemas; matched code by " + f"bare name — verify schema '{sch}')\n" + code_context + ) + results.append(self._make_matched( + display, self._build_db_context(db_entry, scrub=scrub), code_context, + entity, usage, knowledge, + )) + + for nm in sorted(code_table_names - seen_bare): + entity = entity_by_table.get(nm) + usage = knowledge.table_usage.get(nm) or knowledge.table_usage.get( + next((k for k in knowledge.table_usage if k.lower() == nm), "") + ) + code_context = self._build_code_context(entity, usage, knowledge, nm, rules_context) + results.append(self._make_matched(nm, "", code_context, entity, usage, knowledge)) + return results +``` + +- [ ] **Step 5: Extract the `_MatchedTable` builder** (`_make_matched`) so both branches share it (DRY) — add this method (moves the `mt = _MatchedTable(...)` + json blocks from the old loop): + +```python + @staticmethod + def _make_matched(table_name, db_context, code_context, entity, usage, knowledge): + has_code = bool(entity or (usage and usage.is_active)) + mt = _MatchedTable( + table_name=table_name, db_context=db_context, code_context=code_context, + has_code_info=has_code, + entity_name=entity.name if entity else None, + entity_file_path=entity.file_path if entity else None, + read_count=len(usage.readers) if usage else 0, + write_count=len(usage.writers) if usage else 0, + ) + if entity and entity.columns: + mt.code_columns_json = json.dumps( + [{"name": c.name, "type": c.col_type, "fk_target": c.fk_target} + for c in entity.columns] + ) + if usage: + all_files = list(set(usage.readers + usage.writers + usage.orm_refs)) + mt.used_in_files_json = json.dumps(all_files[:20]) + return mt +``` + +- [ ] **Step 6: Edit `run()`** — compute `scrub`, budget pre-flight + sink, pass `scrub` to `_match_tables`, all-fallback guard, mid-run summary skip. At the top of `run()` after `wf_id` is set: + +```python + # H5: owner budget pre-flight + per-run usage sink. + from app.services.sync_budget import build_sink, preflight_owner_budget + if settings.sync_budget_enforcement_enabled: + async with async_session_factory() as s: + ok, reason, owner_id = await preflight_owner_budget(s, project_id) + if not ok: + async with async_session_factory() as s: + await self._sync_svc.set_sync_status(s, connection_id, "failed") + await s.commit() + await self._tracker.end(wf_id, "code_db_sync", "failed", reason or "budget") + return {"status": "failed", "error": reason, "budget_blocked": True, + "workflow_id": wf_id} + if owner_id: + self._llm = LLMRouter(usage_sink=build_sink(owner_id, project_id)) + self._analyzer = CodeDbSyncAnalyzer(self._llm) + + # H6: per-connection opt-out + global scrub flag. + scrub_send = True + async with async_session_factory() as s: + from app.models.connection import Connection + conn = await s.get(Connection, connection_id) + send = getattr(conn, "send_sample_data_to_llm", True) if conn else True + scrub = settings.sync_pii_scrubbing_enabled and send + # When send is False we omit samples entirely by passing empty contexts: + self._omit_samples = not send +``` + +Replace the `_match_tables(...)` call (Step 3, line ~149) to pass scrub: + +```python + matched_tables = self._match_tables( + knowledge, db_entries, rules_context, + scrub=(scrub and not self._omit_samples), + ) +``` + +> When `send is False`, `_build_db_context` must omit sample+distinct entirely. Implement by guarding inside `_build_db_context`: pass `scrub=False, omit=self._omit_samples`. Simpler: add `omit_samples: bool = False` param to `_build_db_context` and `_match_tables`; when True, skip the distinct/sample blocks. Add the param and the two `if not omit_samples:` guards around the distinct and sample blocks. + +- [ ] **Step 7: All-fallback guard** — after Step 4 builds `analyses`, before Step 5 store block: + +```python + total = len(analyses) + non_fallback = sum(1 for a in analyses if not a.is_fallback) + if total and (non_fallback / total) < settings.sync_min_success_ratio_to_persist: + logger.warning( + "CODE_DB_SYNC kept previous rows: only %d/%d tables analyzed", + non_fallback, total, + ) + async with async_session_factory() as session: + await self._sync_svc.set_sync_status(session, connection_id, "failed") + await session.commit() + await self._tracker.end( + wf_id, "code_db_sync", "failed", + f"LLM degraded: {non_fallback}/{total} analyzed; kept previous sync", + ) + return {"status": "failed", "error": "llm_degraded_kept_previous", + "workflow_id": wf_id} +``` + +- [ ] **Step 8: Mid-run summary skip** — wrap the Step-6 `generate_summary` call: + +```python + sink = getattr(self._llm, "_sink", None) + if sink is not None and sink.budget_exceeded(): + summary_result = SyncSummaryResult() # skip LLM summary + else: + summary_result = await self._analyzer.generate_summary(...) # unchanged args +``` + +(Import `SyncSummaryResult` from the analyzer module at top of file.) + +- [ ] **Step 9: Store-by-name guard** (H2) — in the Step 5 loop, before building `sync_data`: + +```python + mt = mt_lookup.get(analysis.table_name) + if mt is None: + logger.warning("store_sync: no matched table for %s — skipped", + analysis.table_name) + continue +``` + +(and drop the `if mt else …` ternaries since `mt` is now guaranteed.) + +- [ ] **Step 10: M7 graph_db_bridge** — move over-broad verbs to ambiguous + label op_kind heuristic. In `graph_db_bridge.py`: + - Add `_AMBIGUOUS_VERBS = ("process_", "handle_", "sync_", "set_", "add_", "register_")` and remove those six from `_WRITE_VERBS`. + - In `classify_op_kind`, after the write/read checks, `for verb in _AMBIGUOUS_VERBS: if name.startswith(verb): return "unknown"`. + - In `code_db_sync_pipeline._build_code_context` (the `Code callers` block, line ~558), change the printed line to drop fabricated depth and mark heuristic: + +```python + parts.append(f" - {name} ({op}, conf={conf:.2f}, heuristic) in {file_}") +``` + + Add test `test_ambiguous_verbs_not_write` in `test_graph_db_bridge.py`: + +```python +def test_ambiguous_verbs_classified_unknown(): + from app.knowledge.graph_db_bridge import classify_op_kind + class _S: + name = "process_report"; decorators = () + assert classify_op_kind(_S()) == "unknown" +``` + +- [ ] **Step 11: Run all pipeline + bridge tests — expect PASS**; ruff/mypy. + +`cd backend && .venv/bin/pytest tests/unit/knowledge/test_code_db_sync_pipeline_run.py tests/unit/knowledge/test_graph_db_bridge.py -v` + +- [ ] **Step 12: Commit** + +```bash +git add backend/app/knowledge/code_db_sync_pipeline.py backend/app/knowledge/graph_db_bridge.py backend/tests/unit/knowledge/test_code_db_sync_pipeline_run.py backend/tests/unit/knowledge/test_graph_db_bridge.py +git commit -m "fix(sync): schema-qualified identity, all-fallback guard, PII scrub, budget wiring, op_kind heuristic label (M2,H4,H6,H5,H2,M7,L4)" +``` + +--- + +## Task T6: `code_db_sync_service` — enrichment validation/deep-merge, header gating (M6, L2) + +**Files:** +- Modify: `backend/app/services/code_db_sync_service.py` +- Test: `backend/tests/unit/services/test_code_db_sync_service.py` (add tests) + +**Interfaces — Produces:** safe `add_runtime_enrichment`; `sync_to_prompt_context` neutral header for non-completed. + +- [ ] **Step 1: Failing tests** + +```python +# append to backend/tests/unit/services/test_code_db_sync_service.py +async def test_enrichment_rejects_metadata_keys_in_required_filters(db_session, seed_sync_row): + svc = CodeDbSyncService() + await svc.add_runtime_enrichment(db_session, conn_id, "orders", + "required_filters_json", '{"source": "investigation", "filter": "x"}') + row = await svc.get_table_sync(db_session, conn_id, "orders") + import json + assert json.loads(row.required_filters_json) == {} # metadata keys dropped + + +async def test_enrichment_deep_merges_value_mappings(db_session, seed_sync_row): + svc = CodeDbSyncService() + # seed: {"status": {"0": "pending", "1": "processed"}} + await svc.add_runtime_enrichment(db_session, conn_id, "orders", + "column_value_mappings_json", '{"status": {"2": "failed"}}') + row = await svc.get_table_sync(db_session, conn_id, "orders") + import json + assert json.loads(row.column_value_mappings_json)["status"] == { + "0": "pending", "1": "processed", "2": "failed"} +``` + +(`seed_sync_row` fixture: upsert one `orders` row with the seed mappings; reuse the conftest helpers.) + +- [ ] **Step 2: Run — expect FAIL.** + +- [ ] **Step 3: Edit `add_runtime_enrichment`** — replace the `mergeable_json_fields` branch: + +```python + if field == "required_filters_json": + existing_json = self._safe_load_dict(getattr(entry, field, None)) + new_data = self._safe_load_dict(value) + _META = {"source", "filter", "_meta"} + for col, cond in new_data.items(): + if col in _META or not isinstance(cond, str): + continue # only {column: condition_string} pairs are valid filters + existing_json[col] = cond + setattr(entry, field, json.dumps(existing_json)) + elif field == "column_value_mappings_json": + existing_json = self._safe_load_dict(getattr(entry, field, None)) + new_data = self._safe_load_dict(value) + for col, mapping in new_data.items(): + if isinstance(mapping, dict) and isinstance(existing_json.get(col), dict): + existing_json[col].update(mapping) # deep-merge per column + else: + existing_json[col] = mapping + setattr(entry, field, json.dumps(existing_json)) + elif field in appendable_text_fields: + existing_lines = [ln.strip() for ln in (getattr(entry, field, "") or "").split("\n")] + if value.strip() not in existing_lines: + combined = f"{getattr(entry, field, '') or ''}\n{value}".strip() + setattr(entry, field, combined[-8000:]) # cap growth (keep newest) + else: + return None +``` + +Add the helper: + +```python + @staticmethod + def _safe_load_dict(raw) -> dict: + if not raw: + return {} + try: + v = json.loads(raw) + return v if isinstance(v, dict) else {} + except (json.JSONDecodeError, TypeError): + return {} +``` + +(Remove the now-unused `mergeable_json_fields` set; keep `appendable_text_fields`.) + +- [ ] **Step 4: L2 header gating** — in `sync_to_prompt_context`, replace the header block: + +```python + status = getattr(summary, "sync_status", None) if summary else None + if summary and summary.synced_at and status in ("completed", "stale"): + parts.append(f"## Code-DB Sync (analyzed {summary.synced_at.strftime('%Y-%m-%d %H:%M')})\n") + else: + parts.append("## Code-DB Sync\n") +``` + +- [ ] **Step 5: Run tests — expect PASS**; ruff/mypy. +- [ ] **Step 6: Commit** + +```bash +git add backend/app/services/code_db_sync_service.py backend/tests/unit/services/test_code_db_sync_service.py +git commit -m "fix(sync): validate required_filters payload, deep-merge value mappings, gate prompt header (M6,L2)" +``` + +--- + +## Task T7: SQL agent — confidence gate on enforced filters (H4) + +**Files:** +- Modify: `backend/app/agents/sql_agent.py` (`_load_required_filters_by_table` ~1543, `_load_sync_filters_and_mappings` ~1502) +- Test: `backend/tests/unit/agents/test_sql_agent_required_filters.py` (add or create) + +**Interfaces — Consumes:** `settings.sync_min_confidence_to_enforce_filters`, `CodeDbSync.confidence_score`. + +- [ ] **Step 1: Failing test** + +```python +# backend/tests/unit/agents/test_sql_agent_required_filters.py +async def test_low_confidence_filters_not_enforced(monkeypatch, sql_agent, conn_cfg, seed_entries): + # seed: table "orders" required_filters {"status":"=1"} confidence_score=1 + monkeypatch.setattr("app.config.settings.sync_min_confidence_to_enforce_filters", 2) + out = await sql_agent._load_required_filters_by_table(conn_cfg) + assert "orders" not in out # confidence 1 < 2 → skipped +``` + +- [ ] **Step 2: Run — expect FAIL.** + +- [ ] **Step 3: Add the confidence gate** in `_load_required_filters_by_table` (the loop over sync entries): + +```python + from app.config import settings + min_conf = settings.sync_min_confidence_to_enforce_filters + for sync_entry in entries: + if (getattr(sync_entry, "confidence_score", 0) or 0) < min_conf: + continue + raw = getattr(sync_entry, "required_filters_json", "{}") or "{}" + ... +``` + +And the same `if confidence < min_conf: continue` guard in `_load_sync_filters_and_mappings` (the `for e in entries:` loop, before reading `rf`). + +- [ ] **Step 4: Run test — expect PASS**; ruff/mypy. +- [ ] **Step 5: Commit** + +```bash +git add backend/app/agents/sql_agent.py backend/tests/unit/agents/test_sql_agent_required_filters.py +git commit -m "fix(sync): do not enforce SQL required-filters from low-confidence/fallback rows (H4)" +``` + +--- + +## Task T8: DB-index subsystem — is_indexed whitelist, get_index_age None-guard, schema-qualified uniqueness, db-index LLM scrub (H7, M9, M2, H6-db-egress) + +**Files:** +- Modify: `backend/app/services/db_index_service.py` (`is_indexed` ~164, `get_index_age` ~178, `upsert_table` ~25, `delete_stale_tables` ~79) +- Modify: `backend/app/models/db_index.py` (`__table_args__`) +- Modify: `backend/app/knowledge/db_index_pipeline.py` (store block ~652 caller; pass `scrub` into validator) +- Modify: `backend/app/knowledge/db_index_validator.py` (`analyze_table`/`analyze_table_batch`/`_build_table_prompt` add `scrub`) +- Create: `backend/alembic/versions/_sync_remediation_schema_qualified_uniqueness.py` +- Test: `backend/tests/unit/services/test_db_index_service.py` (add) + +**Interfaces — Consumes:** `pii_scrubber` (T2). **Produces:** schema-aware `upsert_table` / `delete_stale_tables(connection_id, current_keys: set[str])` where keys are `f"{schema}.{name}"`. + +- [ ] **Step 1: Failing tests** + +```python +# append to backend/tests/unit/services/test_db_index_service.py +async def test_is_indexed_false_for_failed_only(db_session): + svc = DbIndexService() + await svc.set_indexing_status(db_session, "c1", "running") + await svc.set_indexing_status(db_session, "c1", "failed") + await db_session.commit() + assert await svc.is_indexed(db_session, "c1") is False + + +async def test_is_indexed_true_for_completed_partial(db_session): + svc = DbIndexService() + await svc.set_indexing_status(db_session, "c2", "completed_partial") + await db_session.commit() + assert await svc.is_indexed(db_session, "c2") is True + + +async def test_get_index_age_none_when_indexed_at_null(db_session, summary_with_null_indexed_at): + svc = DbIndexService() + assert await svc.get_index_age(db_session, "c3") is None # no AttributeError +``` + +- [ ] **Step 2: Run — expect FAIL.** + +- [ ] **Step 3: Fix `is_indexed`** (spec §5.9a): + +```python + async def is_indexed(self, session, connection_id) -> bool: + summary = await self.get_summary(session, connection_id) + if not summary: + return False + status = (getattr(summary, "indexing_status", "idle") or "idle") + if status not in ("completed", "completed_partial"): + return False + return summary.indexed_at is not None +``` + +- [ ] **Step 4: Fix `get_index_age`** None-guard: + +```python + indexed_at = summary.indexed_at + if indexed_at is None: + return None + if indexed_at.tzinfo is None: + indexed_at = indexed_at.replace(tzinfo=UTC) + return datetime.now(UTC) - indexed_at +``` + +- [ ] **Step 5: Schema-aware `upsert_table`** — match on `(connection_id, table_schema, table_name)`: + +```python + table_name = table_data["table_name"] + table_schema = table_data.get("table_schema", "public") + result = await session.execute( + select(DbIndex).where( + DbIndex.connection_id == connection_id, + DbIndex.table_schema == table_schema, + DbIndex.table_name == table_name, + ) + ) +``` + +- [ ] **Step 6: Schema-aware `delete_stale_tables`** — keys are `f"{schema}.{name}"`, implemented via fetch-ids-then-delete (portable, 2 queries): + +```python + async def delete_stale_tables( + self, session, connection_id, current_keys: set[str] + ) -> int: + rows = (await session.execute( + select(DbIndex.id, DbIndex.table_schema, DbIndex.table_name) + .where(DbIndex.connection_id == connection_id) + )).all() + stale_ids = [ + rid for rid, sch, nm in rows + if f"{(sch or 'public')}.{nm}" not in current_keys + ] + if not stale_ids: + return 0 + await session.execute(delete(DbIndex).where(DbIndex.id.in_(stale_ids))) + await session.flush() + return len(stale_ids) +``` + +- [ ] **Step 7: Update the db_index_pipeline caller** (`db_index_pipeline.py:652`): + +```python + current_keys = { + f"{(t.schema or 'public')}.{t.name}" for t in schema.tables + } + deleted = await self._svc.delete_stale_tables( + session, connection_id, current_keys + ) +``` + +- [ ] **Step 8: Model constraint** (`db_index.py`) — replace `uq_db_index_conn_table`: + +```python + __table_args__ = ( + UniqueConstraint("connection_id", "table_schema", "table_name", + name="uq_db_index_conn_schema_table"), + ) +``` + +- [ ] **Step 9: Migration ``** — generate then hand-edit to drop+create (guarded for SQLite batch): + +```bash +cd backend && PYTHONPATH=. .venv/bin/alembic revision -m "sync_remediation_schema_qualified_uniqueness" +``` + +Body: + +```python +def upgrade() -> None: + with op.batch_alter_table("db_index") as b: + b.drop_constraint("uq_db_index_conn_table", type_="unique") + b.create_unique_constraint( + "uq_db_index_conn_schema_table", + ["connection_id", "table_schema", "table_name"], + ) + +def downgrade() -> None: + with op.batch_alter_table("db_index") as b: + b.drop_constraint("uq_db_index_conn_schema_table", type_="unique") + b.create_unique_constraint("uq_db_index_conn_table", + ["connection_id", "table_name"]) +``` + +- [ ] **Step 10: db-index LLM scrub (H6 second egress)** — `db_index_validator.py`: + - `analyze_table(self, table, sample_data, code_context, rules_context, *, scrub: bool = True, ...)` and `analyze_table_batch(..., scrub: bool = True)` thread `scrub` into `_build_table_prompt`. + - `_build_table_prompt(self, table, sample_data, code_context, rules_context, *, scrub: bool = True)` — scrub the rows before formatting (lines ~427-434): + +```python + if sample_data and sample_data.rows: + from app.knowledge import pii_scrubber + rows = pii_scrubber.scrub_row_cells(sample_data.columns, sample_data.rows, enabled=scrub) + parts.append(f"\nSample data ({len(rows)} newest rows):") + parts.append("| " + " | ".join(sample_data.columns) + " |") + parts.append("| " + " | ".join(["---"] * len(sample_data.columns)) + " |") + for row in rows: + parts.append("| " + " | ".join(str(c) for c in row) + " |") +``` + + - In `db_index_pipeline.py` where `validate_tables`/`analyze_table*` are called, compute `scrub = settings.sync_pii_scrubbing_enabled and connection.send_sample_data_to_llm` (the pipeline already loads the connection config; resolve the `Connection` row once) and pass `scrub=scrub`. When `send_sample_data_to_llm` is False, pass `sample_data=None` to omit entirely. + +- [ ] **Step 11: Run tests — expect PASS**; run migration up/down on a scratch SQLite (`alembic upgrade head` then `downgrade -1`); ruff/mypy. +- [ ] **Step 12: Commit** + +```bash +git add backend/app/services/db_index_service.py backend/app/models/db_index.py backend/app/knowledge/db_index_pipeline.py backend/app/knowledge/db_index_validator.py backend/alembic/versions/*schema_qualified_uniqueness.py backend/tests/unit/services/test_db_index_service.py +git commit -m "fix(sync): is_indexed status whitelist, get_index_age None-guard, schema-qualified db_index, scrub db-index LLM egress (H7,M9,M2,H6)" +``` + +--- + +## Task T9: Investigation enrichment producer reroute (M6) + +**Files:** +- Modify: `backend/app/api/routes/data_investigations.py` (`_enrich_sync_from_investigation` ~319-352) +- Test: `backend/tests/unit/api/test_data_investigations_enrich.py` (add or extend) + +- [ ] **Step 1: Failing test** — assert a `missing_filter` investigation routes to `query_recommendations`, not `required_filters_json`: + +```python +async def test_missing_filter_routes_to_recommendations(monkeypatch): + captured = {} + async def _fake(self, db, *, connection_id, table_name, field, value): + captured["field"] = field; captured["value"] = value + monkeypatch.setattr("app.services.code_db_sync_service.CodeDbSyncService.add_runtime_enrichment", _fake) + # ... build inv with root_cause_category="missing_filter", call _enrich_sync_from_investigation + assert captured["field"] == "query_recommendations" + assert "[from investigation]" in captured["value"] +``` + +- [ ] **Step 2: Run — expect FAIL.** + +- [ ] **Step 3: Edit the `missing_filter` branch** (spec §5.6c): + +```python + if inv.root_cause_category == "missing_filter": + await sync_svc.add_runtime_enrichment( + db, connection_id=inv.connection_id, table_name=table, + field="query_recommendations", + value=f"[from investigation] {inv.root_cause}", + ) +``` + +- [ ] **Step 4: Run test — PASS**; ruff/mypy. +- [ ] **Step 5: Commit** + +```bash +git add backend/app/api/routes/data_investigations.py backend/tests/unit/api/test_data_investigations_enrich.py +git commit -m "fix(sync): route investigation hints to query_recommendations not required_filters (M6)" +``` + +--- + +# WAVE 3 — Reliability / concurrency / traceability + +## Task T10: RunCoordinator — IntegrityError→409 + model index parity (H8) + +**Files:** +- Modify: `backend/app/services/run_coordinator.py` (`start` ~134-173; imports) +- Modify: `backend/app/models/indexing_run.py` (`__table_args__`) +- Create: `backend/alembic/versions/_sync_remediation_indexing_run_active_index.py` +- Test: `backend/tests/unit/services/test_run_coordinator.py` (add) + +- [ ] **Step 1: Failing test** — simulate a unique-violation on commit: + +```python +async def test_start_translates_integrity_error_to_already_active(monkeypatch, db_session): + from app.services.run_coordinator import RunCoordinator, RunAlreadyActiveError + from sqlalchemy.exc import IntegrityError + coord = RunCoordinator() + # first run holds the slot + await coord.start(db_session, kind="code_db_sync", project_id="p1", connection_id="c1") + # second concurrent start must raise RunAlreadyActiveError, not IntegrityError + with pytest.raises(RunAlreadyActiveError): + # force the app-level check to miss so we hit the DB index: + monkeypatch.setattr(coord, "_find_active", + lambda *a, **k: _async_none()) + await coord.start(db_session, kind="code_db_sync", project_id="p1", connection_id="c1") +``` + +(`_async_none` returns `None` first then the real row on the recovery lookup — use a small async stub returning None on the pre-check.) + +- [ ] **Step 2: Run — expect FAIL** (raises `IntegrityError`). + +- [ ] **Step 3: Edit `start()`** — wrap the commit (spec §5.8a): + +```python + from sqlalchemy.exc import IntegrityError + db.add(run) + try: + await db.commit() + except IntegrityError as exc: + await db.rollback() + existing = await self._find_active(db, project_id, kind, connection_id) + raise RunAlreadyActiveError(existing.id if existing else "unknown") from exc + await db.refresh(run) +``` + +- [ ] **Step 4: Model parity** (`indexing_run.py` `__table_args__`) — add (import `Index`, `text`): + +```python + Index( + "uq_indexing_runs_active_one", + "project_id", "kind", text("coalesce(connection_id, '')"), + unique=True, + sqlite_where=text("status IN ('queued','running','cancelling')"), + postgresql_where=text("status IN ('queued','running','cancelling')"), + ), +``` + +- [ ] **Step 5: Migration ``** (idempotent parity for envs missing it): + +```python +def upgrade() -> None: + bind = op.get_bind() + op.create_index( + "uq_indexing_runs_active_one", "indexing_runs", + ["project_id", "kind", sa.text("coalesce(connection_id, '')")], + unique=True, + postgresql_where=sa.text("status IN ('queued','running','cancelling')"), + sqlite_where=sa.text("status IN ('queued','running','cancelling')"), + if_not_exists=True, + ) + +def downgrade() -> None: + op.drop_index("uq_indexing_runs_active_one", table_name="indexing_runs", if_exists=True) +``` + +- [ ] **Step 6: Run tests — PASS**; ruff/mypy. +- [ ] **Step 7: Commit** + +```bash +git add backend/app/services/run_coordinator.py backend/app/models/indexing_run.py backend/alembic/versions/*indexing_run_active_index.py backend/tests/unit/services/test_run_coordinator.py +git commit -m "fix(sync): translate single-active IntegrityError to 409 + rollback; model index parity (H8)" +``` + +--- + +## Task T11: Daily sync — parent heartbeat, adopt-not-run, progress steps, budget skip, overview regen (H1, H9, M3, H5, M5) + +**Files:** +- Modify: `backend/app/services/daily_knowledge_sync_service.py` +- Modify: `backend/app/knowledge/run_manifests.py` (drop dead `freshness_reconcile` from `daily_sync`) +- Test: `backend/tests/unit/services/test_daily_knowledge_sync.py` (add) + +**Interfaces — Consumes:** `heartbeat` (`app.core.heartbeat`), `sync_budget.preflight_owner_budget` (T3), `RunCoordinator` (T10). + +- [ ] **Step 1: Failing tests** + +```python +async def test_parent_run_heartbeat_refreshed_during_orchestrate(monkeypatch, db_session): + # Stub _orchestrate to sleep > heartbeat interval; assert heartbeat_at advanced. + ... + +async def test_child_skips_when_already_active(monkeypatch): + # _start_child_wf returns (None, True) when RunAlreadyActiveError → sub-step SKIPPED + ... +``` + +- [ ] **Step 2: Run — expect FAIL.** + +- [ ] **Step 3: H1 — wrap `_orchestrate` in a parent heartbeat** in `run_for_project`: + +```python + from app.core.heartbeat import heartbeat + from datetime import UTC, datetime + + async def _hb() -> None: + async with async_session_factory() as s: + r = await s.get(IndexingRun, run_id) + if r and r.status == "running": + r.heartbeat_at = datetime.now(UTC) + await s.commit() + + async with heartbeat(_hb, interval_seconds=settings.heartbeat_interval_seconds): + result = await self._orchestrate(project_id, run_id=run_id) +``` + +(Import `settings` if not already.) + +- [ ] **Step 4: H9 — `_start_child_wf` returns `(wf_id | None, already_active: bool)`:** + +```python + async def _start_child_wf(self, kind, connection_id, project_id): + from app.services.run_coordinator import RunAlreadyActiveError, RunCoordinator + try: + async with async_session_factory() as rdb: + run = await RunCoordinator().start( + rdb, kind=kind, project_id=project_id, + connection_id=connection_id, trigger="schedule") + return run.workflow_id, False + except RunAlreadyActiveError: + return None, True +``` + +Update the three callers (`_run_repo_index`, `_run_db_index`, `_run_code_db_sync`): on `already_active`, return `(_STEP_SKIPPED, "already running (adopted)")` BEFORE launching the pipeline. + +- [ ] **Step 5: H5 — budget skip** in `_run_code_db_sync` (and `_run_db_index`): after the existing `get_sync_status`/`is_indexed` checks: + +```python + from app.services.sync_budget import preflight_owner_budget + async with async_session_factory() as session: + ok, reason, _ = await preflight_owner_budget(session, project_id) + if not ok: + return _STEP_SKIPPED, f"owner budget: {reason}" +``` + +- [ ] **Step 6: M5 — overview regen after sync** in the `final_status == _STEP_COMPLETED` branch of `_run_code_db_sync`: + +```python + try: + from app.api.routes.connections import _regenerate_overview + await _regenerate_overview(project_id, connection_id) + except Exception: + logger.debug("daily sync overview regen failed", exc_info=True) +``` + +- [ ] **Step 7: M3 — progress steps + manifest align.** Change `_orchestrate(self, project_id)` → `_orchestrate(self, project_id, *, run_id)`; emit `tracker.emit(workflow_id, step, "started"/"completed")` around `repo_index`, per-connection `db_index`, `code_db_sync`, and a final `summarize`. Fetch the parent run's `workflow_id` once. In `run_manifests.py`, remove the `Step("freshness_reconcile", ...)` line from the `daily_sync` manifest (leaving `plan_targets, db_index, code_db_sync, summarize`). + +> The emit keys MUST match the manifest keys (`plan_targets`, `db_index`, `code_db_sync`, `summarize`). The coordinator `_on_event`→`_apply_event` projection advances `progress_pct`. + +- [ ] **Step 8: Run tests — PASS**; ruff/mypy. +- [ ] **Step 9: Commit** + +```bash +git add backend/app/services/daily_knowledge_sync_service.py backend/app/knowledge/run_manifests.py backend/tests/unit/services/test_daily_knowledge_sync.py +git commit -m "fix(sync): parent-run heartbeat, adopt-not-run, progress steps, budget skip, overview regen (H1,H9,M3,H5,M5)" +``` + +--- + +## Task T12: Reaper observability (L1) + +**Files:** +- Modify: `backend/app/services/stale_run_reaper.py` (`reap_once`) +- Test: `backend/tests/unit/services/test_stale_run_reaper.py` (add) + +- [ ] **Step 1: Failing test** — when a driver returns `-1` rowcount, an INFO sweep line is logged. + +```python +async def test_reaper_logs_sweep_when_rowcount_unknown(caplog, monkeypatch, db_session): + # monkeypatch the four execute results' .rowcount to -1; assert a sweep log line emitted + ... +``` + +- [ ] **Step 2: Run — expect FAIL.** + +- [ ] **Step 3: Edit `reap_once`** — after computing `out`, add an unknown-count branch: + +```python + unknown = any( + (r.rowcount is not None and r.rowcount < 0) + for r in (db_res, sync_res, repo_res, runs_failed, runs_cancelled) + ) + if any(out.values()): + logger.info("Reaper: reset stale runs — db_index=%d sync=%d repo=%d runs=%d (timeout=%ds)", + out["db_index"], out["sync"], out["repo"], out["runs"], timeout_seconds) + elif unknown: + logger.info("Reaper: swept stale runs (rowcount unknown on this driver, timeout=%ds)", + timeout_seconds) + return out +``` + +- [ ] **Step 4: Run — PASS**; ruff/mypy. +- [ ] **Step 5: Commit** + +```bash +git add backend/app/services/stale_run_reaper.py backend/tests/unit/services/test_stale_run_reaper.py +git commit -m "fix(sync): reaper logs a sweep even when driver rowcount is unknown (L1)" +``` + +--- + +## Task T13: Worker — synced_tables log key (M5) + +**Files:** +- Modify: `backend/app/worker.py` (`run_code_db_sync` log line ~151) +- Test: covered by reading the result dict; add a tiny assertion test in `backend/tests/unit/test_worker_sync.py`. + +- [ ] **Step 1: Failing test** — `run_code_db_sync` logs `matched=` from the `synced` key. (Patch the pipeline to return `{"status":"completed","total_tables":3,"synced":2}`; assert log contains `matched=2`.) + +- [ ] **Step 2: Run — expect FAIL** (currently logs `matched=None`). + +- [ ] **Step 3: Edit** the log block: + +```python + tables = result.get("total_tables") if isinstance(result, dict) else None + matched = result.get("synced") if isinstance(result, dict) else None +``` + +- [ ] **Step 4: PASS**; ruff/mypy. +- [ ] **Step 5: Commit** + +```bash +git add backend/app/worker.py backend/tests/unit/test_worker_sync.py +git commit -m "fix(sync): worker logs matched count from correct 'synced' key (M5)" +``` + +--- + +# WAVE 4 — Egress opt-in + freshness + +## Task T14: Connection opt-in flag + migration + route budget gate (H6 opt-in, H5 429) + +**Files:** +- Modify: `backend/app/models/connection.py` (add column after `is_active` ~58) +- Create: `backend/alembic/versions/_sync_remediation_connection_flag.py` +- Modify: `backend/app/api/routes/connections.py` (`ConnectionCreate`, `ConnectionResponse`, `trigger_sync` ~1008) +- Test: `backend/tests/unit/models/test_connection_flag.py`, `backend/tests/integration/test_trigger_sync_budget.py` + +- [ ] **Step 1: Failing tests** — default True on model; `trigger_sync` returns 429 when owner over budget. + +- [ ] **Step 2: Run — expect FAIL.** + +- [ ] **Step 3: Model column** (`connection.py` after `is_active`): + +```python + send_sample_data_to_llm: Mapped[bool] = mapped_column( + Boolean, default=True, server_default="1", nullable=False + ) +``` + +- [ ] **Step 4: Migration ``:** + +```python +def upgrade() -> None: + op.add_column("connections", sa.Column( + "send_sample_data_to_llm", sa.Boolean(), nullable=False, server_default=sa.text("1"))) + +def downgrade() -> None: + with op.batch_alter_table("connections") as b: + b.drop_column("send_sample_data_to_llm") +``` + +- [ ] **Step 5: Surface in schemas** — add `send_sample_data_to_llm: bool = True` to `ConnectionCreate` and include in `ConnectionResponse` (it is NOT a secret). + +- [ ] **Step 6: `trigger_sync` 429 pre-flight** — after `require_role(editor)` and before the start-lock dispatch: + +```python + from app.services.sync_budget import preflight_owner_budget + ok, reason, _ = await preflight_owner_budget(db, conn.project_id) + if not ok: + raise HTTPException(status_code=429, detail=reason) +``` + +- [ ] **Step 7: Run tests — PASS**; migration up/down on scratch SQLite; ruff/mypy. +- [ ] **Step 8: Commit** + +```bash +git add backend/app/models/connection.py backend/app/api/routes/connections.py backend/alembic/versions/*connection_flag.py backend/tests/unit/models/test_connection_flag.py backend/tests/integration/test_trigger_sync_budget.py +git commit -m "feat(sync): per-connection send_sample_data_to_llm opt-out + trigger_sync budget 429 (H6,H5)" +``` + +--- + +## Task T15: Freshness dataclass + stale/failed split (M8) + +**Files:** +- Modify: `backend/app/services/knowledge_freshness_service.py` +- Test: `backend/tests/unit/services/test_knowledge_freshness_service.py` (add) + +- [ ] **Step 1: Failing tests** — `KnowledgeFreshness()` default `warnings == []`; `overall_stale is False`; a `failed` sync sets `sync_failed=True`. + +- [ ] **Step 2: Run — expect FAIL.** + +- [ ] **Step 3: Edit dataclass:** + +```python + warnings: list[str] = field(default_factory=list) + sync_failed: bool = False +``` + +(remove `= None # type: ignore`). In `evaluate`'s sync block, set `snapshot.sync_failed = (snapshot.sync_status == "failed")` alongside `sync_stale`. + +- [ ] **Step 4: Run — PASS**; ruff/mypy. +- [ ] **Step 5: Commit** + +```bash +git add backend/app/services/knowledge_freshness_service.py backend/tests/unit/services/test_knowledge_freshness_service.py +git commit -m "fix(sync): freshness warnings default-list + sync_failed flag (M8)" +``` + +--- + +# WAVE 5 — Glue, schedule honesty, loop-closure, integration + +## Task T16: Cron wave honors per-project hour + reconciler all-connections (M4, M1) + +**Files:** +- Modify: `backend/app/main.py` (`_dispatch_daily_knowledge_sync_wave` ~692, `_daily_knowledge_sync_cron_loop` ~751, `_freshness_reconcile` ~603) +- Modify: `backend/app/services/daily_knowledge_sync_service.py` (`list_eligible_projects` returns effective hour) — **NOTE:** this file is owned by T11; T16 depends on T11 and edits a *different method*. To keep ownership disjoint, the `list_eligible_projects` hour-return change is moved INTO T11 (add it there). T16 only edits `main.py`. + +- [ ] **Step 1: Failing test** — wave dispatches a project only when its effective hour == current local hour. + +```python +async def test_wave_filters_by_effective_hour(monkeypatch): + # two projects: one effective hour=current, one hour=current+1 → only first dispatched + ... +``` + +- [ ] **Step 2: Run — expect FAIL.** + +- [ ] **Step 3: M4 — wave filters by hour.** In `_dispatch_daily_knowledge_sync_wave`, compute `current_hour = datetime.now(tz).hour`; change the eligible loop to dispatch only projects whose `SyncScheduleService.effective(...)["hour"] == current_hour`. Change the Redis lock to `cron:daily_sync:{run_date}:{current_hour}`. + +- [ ] **Step 4: M4 — cron loop wakes hourly.** Replace the `compute_next_scheduled_run`-based sleep in `_daily_knowledge_sync_cron_loop` with a top-of-next-hour sleep: + +```python + now = datetime.now(tz) + next_hour = (now.replace(minute=0, second=0, microsecond=0) + timedelta(hours=1)) + await asyncio.sleep(max(1.0, (next_hour - now).total_seconds())) + await _dispatch_daily_knowledge_sync_wave() +``` + +(Keep `compute_next_scheduled_run` for the `/sync-schedule` `next_run` display in projects.py.) + +- [ ] **Step 5: M1 — reconciler all connections.** In `_freshness_reconcile`, replace `conn_id = connections[0].id if connections else None` and the single-connection block with a `for conn in connections:` loop that evaluates freshness and calls `maybe_autostart_db_index` / `maybe_autostart_sync` per connection. For `fresh.sync_failed`, guard with an in-memory per-cycle set so a perpetually-failed sync is retried at most once per reconcile pass (anti retry-storm, spec §5.12). + +- [ ] **Step 6: Run tests — PASS**; ruff/mypy. +- [ ] **Step 7: Commit** + +```bash +git add backend/app/main.py backend/tests/unit/test_main_cron.py +git commit -m "fix(sync): cron wave honors per-project hour; reconciler covers all connections (M4,M1)" +``` + +--- + +## Task T17: Projects route — schedule consistency + get_index_age guard surfaced (M4 display, M9) + +**Files:** +- Modify: `backend/app/api/routes/projects.py` (`get_sync_schedule` ~500; readiness probe ~391-416 uses sync/index status) +- Test: `backend/tests/integration/test_sync_schedule_route.py` (add) + +- [ ] **Step 1: Failing test** — `/sync-schedule` `next_run` reflects the per-project hour and is consistent with the now-hourly cron (the cron honors it). + +- [ ] **Step 2-3:** Verify `get_sync_schedule` still computes `next_run` from the effective per-project hour (it already does, line ~518). Add a regression test asserting the displayed `next_run` hour equals the effective hour. No code change unless the test reveals drift; if drift, align the displayed `next_run` with the hourly cron contract. + +- [ ] **Step 4: Run — PASS**; ruff/mypy. +- [ ] **Step 5: Commit** + +```bash +git add backend/app/api/routes/projects.py backend/tests/integration/test_sync_schedule_route.py +git commit -m "test(sync): assert sync-schedule next_run matches per-project hour (M4)" +``` + +--- + +## Task T18: Integration — migration linearization, full check, docs, issue closure, validation cycle + +**Files:** +- Modify: `backend/alembic/versions/*` (down_revision chaining of ``,``,``) +- Modify: `CHANGELOG.md` (`[Unreleased]`), `qa-audit/issues.md` (close all 22), `CLAUDE.md` (note R5 flags if needed) + +- [ ] **Step 1: Linearize migrations** — set each new revision's `down_revision` to chain after the current head (run `alembic heads`; if multiple heads, add a merge revision). Run `cd backend && PYTHONPATH=. .venv/bin/alembic upgrade head` then `alembic downgrade -3 && alembic upgrade head` on a scratch DB — expect no errors. +- [ ] **Step 2: Full backend gate** — `cd backend && .venv/bin/ruff format --check app/ tests/ && .venv/bin/ruff check app/ tests/ && .venv/bin/mypy app/ --ignore-missing-imports && .venv/bin/pytest tests/ -q` then `coverage report --fail-under=72`. Expect all green. +- [ ] **Step 3: Frontend gate** (no frontend changes expected, but confirm) — `cd frontend && npx tsc --noEmit && npx eslint . --max-warnings=0 && npm test`. +- [ ] **Step 4: CHANGELOG + issues** — add an `[Unreleased]` R5 block listing all 22 fixes; mark each finding closed in `qa-audit/issues.md`. +- [ ] **Step 5: VALIDATION CYCLE (required by user)** — for EACH finding H1..L4, re-read the touched code and confirm: (a) the business-logic understanding in the spec matches the live code, (b) the fix is present and behaves, (c) no regression to the documented invariants (vision §7: read-only, credentials never exposed, freshness tracked, traceability, graceful degradation). Record a one-line confirmation per finding in `docs/superpowers/plans/2026-06-25-sync-remediation.md` under a "Validation log" appended section. +- [ ] **Step 6: Commit** + +```bash +git add -A +git commit -m "docs(sync): R5 changelog + close 22 audit findings + validation log" +``` + +--- + +## Self-review (author checklist — done) + +- **Spec coverage:** every spec § maps to a task — §5.1→T1, §5.2→T2, §5.10→T3, §5.4→T4, §5.5→T5, §5.6→T6, §5.8(sqlagent gate)→T7, §5.9→T8, §5.6c→T9, §5.8→T10, §5.7→T11, §5.15→T12, §5.14→T13, §5.13→T14, §5.12→T15, §5.11→T16, §5.13(sync_now/projects)→T17, integration→T18. All 22 finding IDs appear in the §6 traceability table and in a task. +- **Placeholder scan:** Alembic `` are autogenerated revision IDs (resolved by the `alembic revision` command in-task) — not TBDs. No "implement later"/"add error handling" placeholders; every code step shows code. +- **Type consistency:** `is_fallback` (T4) consumed in T5; `delete_stale_tables(current_keys: set[str])` (T8) — both callers updated in T8 (db_index_pipeline) and CodeDbSync's own delete stays name-based (unchanged); `preflight_owner_budget -> (ok, reason, owner_id)` (T3) consumed identically in T5/T11/T14; `_start_child_wf -> (wf_id|None, bool)` (T11) — all three callers updated in T11; manifest keys (`plan_targets, db_index, code_db_sync, summarize`) match the emit keys in T11 and the `run_manifests.py` edit. +- **Cross-file ownership:** the one cross-task edit (`list_eligible_projects` hour return) is explicitly moved into T11 to keep `daily_knowledge_sync_service.py` single-owner; T16 touches only `main.py`. From 3001250e16e2dc9f27d2b803cad0262cc8d2a8a1 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Thu, 25 Jun 2026 16:41:12 +0200 Subject: [PATCH 03/24] =?UTF-8?q?docs(sync):=20validation=20cycle=20?= =?UTF-8?q?=E2=80=94=20confirm=2022=20findings=20+=20apply=20corrections?= =?UTF-8?q?=20C1-C7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adversarial re-validation (3 independent passes) confirmed the business-logic understanding for all 22 findings is correct (zero misreads). Applied binding corrections: C1 owner-less project degrades (not blocks) budget; C2 omit_samples threaded param not instance attr; C3 sync_now 429 step; C4 M3 parent workflow_id + targeted heartbeat UPDATE (no version lost-update); C5 validate ORM partial- index form + Alembic kwargs; C6 index required-filters under bare suffix for schema-qualified names; C7 wording. Plan Validation Log governs. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../plans/2026-06-25-sync-remediation.md | 80 +++++++++++++++++++ .../2026-06-25-sync-remediation-design.md | 7 +- 2 files changed, 86 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-06-25-sync-remediation.md b/docs/superpowers/plans/2026-06-25-sync-remediation.md index c34627a..9ea7430 100644 --- a/docs/superpowers/plans/2026-06-25-sync-remediation.md +++ b/docs/superpowers/plans/2026-06-25-sync-remediation.md @@ -1689,3 +1689,83 @@ git commit -m "docs(sync): R5 changelog + close 22 audit findings + validation l - **Placeholder scan:** Alembic `` are autogenerated revision IDs (resolved by the `alembic revision` command in-task) — not TBDs. No "implement later"/"add error handling" placeholders; every code step shows code. - **Type consistency:** `is_fallback` (T4) consumed in T5; `delete_stale_tables(current_keys: set[str])` (T8) — both callers updated in T8 (db_index_pipeline) and CodeDbSync's own delete stays name-based (unchanged); `preflight_owner_budget -> (ok, reason, owner_id)` (T3) consumed identically in T5/T11/T14; `_start_child_wf -> (wf_id|None, bool)` (T11) — all three callers updated in T11; manifest keys (`plan_targets, db_index, code_db_sync, summarize`) match the emit keys in T11 and the `run_manifests.py` edit. - **Cross-file ownership:** the one cross-task edit (`list_eligible_projects` hour return) is explicitly moved into T11 to keep `daily_knowledge_sync_service.py` single-owner; T16 touches only `main.py`. + +--- + +## Validation Log & Applied Corrections (2026-06-25) + +Three independent adversarial validators re-read the live code against this spec+plan. **Verdict: the context and business-logic understanding for all 22 findings (H1–H9, M1–M9, L1–L4) is CORRECT — zero fundamental misreads.** Every root-cause premise was confirmed at file:line. The validators surfaced edge-case corrections (one real regression, two plan gaps, two fix-risky items, several wording nits). These corrections are now BINDING on the listed tasks and OVERRIDE the earlier prose where they conflict. + +### C1 (regression — BINDING on T3, T5, T11, T14, T17) — owner-less projects must degrade, not freeze +`Project.owner_id` is `nullable=True` with `ondelete="SET NULL"`. The original `preflight_owner_budget` returned `(False, …)` on a missing owner, which would make manual sync 429 and cron sync skip **forever** for legacy / owner-deleted projects — a graceful-degradation violation. **Corrected contract:** +```python +async def preflight_owner_budget(session, project_id) -> tuple[bool, str | None, str | None]: + owner_id = await resolve_owner_user_id(session, project_id) + if not owner_id: + logger.warning("sync budget unenforced: project %s has no owner", project_id[:8]) + return True, None, None # degrade (unenforced), do NOT block + if not settings.sync_budget_enforcement_enabled: + return True, None, owner_id + msg = await _usage_svc.check_token_budget(session, owner_id) + return (False, msg, owner_id) if msg else (True, None, owner_id) +``` +T3's `test_preflight_owner_missing` must assert `(True, None, None)` (not `False`). In T5/T11, when `owner_id is None` keep the default `LLMRouter()` (NullUsageSink) — do not call `build_sink(None, …)`. + +### C2 (plan gap — BINDING on T5) — thread `omit_samples` as a parameter, not `self._omit_samples` +`_build_db_context` is a `@staticmethod`; it cannot read `self._omit_samples`. Drop that instance attr entirely. Thread **two** params end-to-end: +- `_match_tables(self, knowledge, db_entries, rules_context="", *, scrub: bool, omit_samples: bool)` +- `_build_db_context(entry, *, scrub: bool = True, omit_samples: bool = False)` — when `omit_samples` is True, skip the distinct-values and sample-data blocks entirely (emit neither). +- `run()` computes `send = connection.send_sample_data_to_llm`; `omit_samples = not send`; `scrub = settings.sync_pii_scrubbing_enabled`; passes both into `_match_tables`, which forwards to `_build_db_context`. + +Update spec §5.5d signature accordingly (it currently names only `scrub`). The H6 truth table is unchanged in meaning: `send=False ⇒ omit_samples=True ⇒ nothing sent`. + +### C3 (plan gap — BINDING on T17) — `sync_now` needs the same 429 pre-flight as `trigger_sync` +T17 must add a concrete step (not just verification). In `backend/app/api/routes/projects.py::sync_now`, after `require_role(editor)` (~line 556) and before `coord.start(...)`: +```python + from app.services.sync_budget import preflight_owner_budget + ok, reason, _ = await preflight_owner_budget(db, project_id) + if not ok: + raise HTTPException(status_code=429, detail=reason) +``` +Add a failing integration test `test_sync_now_budget_429` before implementing. + +### C4 (BINDING on T11) — M3 progress emit needs the parent `workflow_id`, and the heartbeat must not lost-update `version` +1. `_orchestrate` only receives `run_id`. Before emitting progress it MUST load the parent run's `workflow_id`: + ```python + async with async_session_factory() as s: + parent = await s.get(IndexingRun, run_id) + parent_wf = parent.workflow_id if parent else None + ... + if parent_wf: + await self._tracker.emit(parent_wf, step_key, "completed", detail) + ``` + The emit carries `run_id=None`, so `_on_event`'s `if event.run_id is not None: return` guard does NOT skip it (verified) — the projection works via the `_wf_to_run` map or the `IndexingRun.workflow_id == event.workflow_id` DB fallback. Emit keys MUST be manifest keys: `plan_targets`, `db_index`, `code_db_sync`, `summarize`. +2. **Concurrent-write avoidance:** the H1 `_hb` writer and the M3 `_on_event` projection both write the same parent row. To avoid a lost-update on `IndexingRun.version`, the `_hb` writer MUST use a targeted UPDATE (no full-row ORM load, no `version` bump): + ```python + from sqlalchemy import update + async def _hb() -> None: + async with async_session_factory() as s: + await s.execute( + update(IndexingRun) + .where(IndexingRun.id == run_id, IndexingRun.status == "running") + .values(heartbeat_at=datetime.now(UTC)) + ) + await s.commit() + ``` +3. Add a test for the **adopted-parent** path (manual `sync-now` mints the `daily_sync` run → daily `run_for_project` adopts it via `_find_active`): assert progress still projects (through the `_on_event` DB fallback) and that a skipped child yields daily status `PARTIAL` (via `any_skip`), not `SUCCESS`. + +### C5 (fix-risky — BINDING on T10) — validate the ORM partial-index form + Alembic kwargs before locking +Before adding the `Index("uq_indexing_runs_active_one", "project_id", "kind", text("coalesce(connection_id, '')"), unique=True, sqlite_where=…, postgresql_where=…)` to `IndexingRun.__table_args__`, run a scratch `Base.metadata.create_all()` against aiosqlite to confirm it compiles (the text-expression positional form is unproven in this ORM version). **Fallback if it errors:** use a plain composite partial-unique `Index(..., "project_id", "kind", "connection_id", unique=True, sqlite_where=…)` and document that NULL `connection_id` rows won't collide in SQLite-`create_all` test envs (prod is covered by the existing migration `a1f2b3c4d5e6`). Also confirm the pinned Alembic supports `op.create_index(..., if_not_exists=True)` / `op.drop_index(..., if_exists=True)` (T10 Step 5, T8 Step 9); if not, replace with an inspector existence check. + +### C6 (BINDING on T8 + T7 — M2 enforcement on colliding names) — add a decision + test +When `_match_tables` stores a schema-qualified `table_name` (e.g. `analytics.orders`) on collision, the SQL-agent enforcement path `check_required_filters` (`required_filter_guard.py`) lowercases query table tokens and would see bare `orders` — which won't match the qualified key, silently dropping enforcement on exactly the colliding tables M2 targets. **Decision (locked):** in `_load_required_filters_by_table` (T7), index required-filters under BOTH the stored name AND its bare suffix (`name.split(".")[-1]`), so a query referencing bare `orders` still matches. Add `test_required_filters_match_bare_suffix_for_qualified_table`. + +### C7 (spec wording — non-blocking) — tighten three rationales +- **H4:** `check_required_filters` currently hard-enforces only 2 columns (`was_handled`, `deleted_at`); the *real* low-confidence exposure is filter **guidance** surfaced in the prompt via `_load_sync_filters_and_mappings`. The T7 gate (applied in BOTH loaders) is the correct fix; spec §5.6b/§6 wording should say "don't surface low-confidence filter guidance," not "don't enforce hard filters." +- **M9:** `indexed_at` is server-defaulted, so a DB-fetched row is effectively never NULL; the AttributeError is reachable only for an unflushed/unrefreshed in-session object. The None-guard is still correct; just don't claim NULL is common. +- **M8:** the `to_dict`/`to_summary` source split is a *latent* fragility (`_warn` fills both in lockstep today), not an active divergence. The `None`-default and `sync_failed` flag are the real fixes. +- **H5 mid-run accessor:** use `getattr(self._llm, "_sink", None)` (no public accessor exists on `LLMRouter`); the plan T5 Step 8 already does this — spec §5.5f should match. +- **Nit:** `DbUsageSink.__init__` is keyword-only; `build_sink` already uses kwargs — keep it. + +### Per-finding business-logic confirmation (validation cycle result) +H1 ✅ premise verified (parent heartbeat frozen; reaper `_stale_run` uses heartbeat **Status legend used below:** every contract here is *locked* — types, signatures, file layout, flag names/defaults, migration shapes. The implementation plan (`docs/superpowers/plans/2026-06-25-sync-remediation.md`) turns each into TDD tasks with exact code. A zero-context implementer must not invent names; use the ones fixed here. +> **⚠️ BINDING CORRECTIONS (post-validation, 2026-06-25):** an adversarial validation cycle confirmed all 22 findings' business logic is correct but produced corrections C1–C7 that **OVERRIDE the prose below where they conflict** — see the **Validation Log & Applied Corrections** section at the end of the plan. Most load-bearing: **C1** (owner-less project → budget *unenforced*, not blocked — §5.10 below is patched), **C2** (`omit_samples` is a threaded param, not an instance attr — §5.5d), **C4** (M3 emit needs the parent `workflow_id`; the `_hb` heartbeat must use a targeted UPDATE to avoid `version` lost-update — §5.7), **C5** (validate the ORM partial-index form before locking — §5.8), **C6** (index required-filters under the bare suffix too, for schema-qualified names — §5.9/SQL agent). + --- ## 1. Goals & non-goals @@ -395,7 +397,10 @@ async def preflight_owner_budget( session: AsyncSession, project_id: str ) -> tuple[bool, str | None, str | None]: """(_enabled-aware) Return (ok, reason, owner_user_id). - - owner missing → (False, "project owner not found", None) + - owner missing (Project.owner_id NULL — legacy/owner-deleted) → (True, None, None) # C1: + budget UNENFORCED (log WARNING; caller uses default LLMRouter/NullUsageSink). + Blocking here would freeze every sync for owner-less projects (graceful-degradation + violation), so we degrade, never block. - check_token_budget(owner) returns a message → (False, message, owner_id) - else → (True, None, owner_id) When settings.sync_budget_enforcement_enabled is False → always (True, None, owner_id).""" From d6724865f50212b147f2271a0760df09cb016718 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 15:06:22 +0200 Subject: [PATCH 04/24] feat(sync): R5 config flags (pii scrub, confidence gate, success-ratio guard, budget) --- backend/.env.example | 6 ++++++ backend/app/config.py | 12 ++++++++++++ backend/tests/unit/test_config.py | 7 +++++++ 3 files changed, 25 insertions(+) create mode 100644 backend/tests/unit/test_config.py diff --git a/backend/.env.example b/backend/.env.example index 0e92c66..3ba5139 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -177,6 +177,12 @@ CORS_ORIGINS=["http://localhost:3000","http://localhost:3100","https://checkmyda # AUTO_INDEX_DB_ON_TEST=false # DB_INDEX_INCREMENTAL_ENABLED=true # diff-only schema reindex on refresh (R2-3) +# ----- R5 sync remediation --------------------------------------------------- +# SYNC_PII_SCRUBBING_ENABLED=true +# SYNC_MIN_CONFIDENCE_TO_ENFORCE_FILTERS=2 +# SYNC_MIN_SUCCESS_RATIO_TO_PERSIST=0.5 +# SYNC_BUDGET_ENFORCEMENT_ENABLED=true + # ----- Knowledge lifecycle maintenance ---------------------------------------- # How often (hours) the background loop runs learning/session-note confidence # decay and insight TTL/decay. Independent of backups so decay always runs. diff --git a/backend/app/config.py b/backend/app/config.py index 18dba00..6a0bd49 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -168,6 +168,18 @@ def _fix_database_url(self) -> "Settings": # Database index settings db_index_ttl_hours: int = 24 db_index_batch_size: int = 5 + + # --- R5 sync remediation ------------------------------------------------- + # H6: scrub PII / secrets from DB samples + distinct values before LLM egress. + sync_pii_scrubbing_enabled: bool = True + # H4: per-table analyses below this confidence never enforce hard SQL filters. + sync_min_confidence_to_enforce_filters: int = 2 + # H4: if the fraction of non-fallback analyses is below this, keep prior rows + # instead of overwriting with a degraded run. 0.0 disables the guard. + sync_min_success_ratio_to_persist: float = 0.5 + # H5: gate sync LLM spend on the project owner's token budget. + sync_budget_enforcement_enabled: bool = True + auto_index_db_on_test: bool = False # R2-3: reuse prior LLM table analysis for tables whose schema signature # is unchanged since the last successful index, instead of re-LLM-ing every diff --git a/backend/tests/unit/test_config.py b/backend/tests/unit/test_config.py new file mode 100644 index 0000000..531c3c5 --- /dev/null +++ b/backend/tests/unit/test_config.py @@ -0,0 +1,7 @@ +def test_r5_sync_remediation_defaults(): + from app.config import settings + + assert settings.sync_pii_scrubbing_enabled is True + assert settings.sync_min_confidence_to_enforce_filters == 2 + assert settings.sync_min_success_ratio_to_persist == 0.5 + assert settings.sync_budget_enforcement_enabled is True From 1a2cfe6a2cec18cfad95491b673d55d9e9a761e8 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 15:09:32 +0200 Subject: [PATCH 05/24] security(sync): add pii_scrubber (denylist + value redaction) for LLM egress (H6) --- backend/app/knowledge/pii_scrubber.py | 116 ++++++++++++++++++ .../tests/unit/knowledge/test_pii_scrubber.py | 41 +++++++ 2 files changed, 157 insertions(+) create mode 100644 backend/app/knowledge/pii_scrubber.py create mode 100644 backend/tests/unit/knowledge/test_pii_scrubber.py diff --git a/backend/app/knowledge/pii_scrubber.py b/backend/app/knowledge/pii_scrubber.py new file mode 100644 index 0000000..37647f0 --- /dev/null +++ b/backend/app/knowledge/pii_scrubber.py @@ -0,0 +1,116 @@ +"""Redact PII / secrets from DB-derived context before it reaches an LLM. + +Pure functions, no I/O. Used by the DB-index validator and the code↔DB sync +analyzer (the two places raw tenant sample data would otherwise egress to an +LLM provider). See spec §5.2. +""" + +from __future__ import annotations + +import json +import re + +SENSITIVE_COLUMN_TOKENS: tuple[str, ...] = ( + "password", + "passwd", + "secret", + "token", + "api_key", + "apikey", + "private_key", + "access_key", + "credential", + "ssn", + "social_security", + "card_number", + "card_no", + "cardno", + "pan", + "cvv", + "cvc", + "iban", + "swift", + "auth", + "session", + "cookie", + "salt", + "hash", + "email", +) + +_EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}") +_JWT = re.compile(r"eyJ[A-Za-z0-9_-]{1,}\.[A-Za-z0-9_-]{1,}\.[A-Za-z0-9_-]{1,}") +_CARD = re.compile(r"\b(?:\d[ -]?){13,19}\b") +_PHONE = re.compile(r"\b\+?\d[\d\s().-]{7,}\d\b") +# long hex / base64-ish secrets (>= 24 chars, no spaces) +_SECRETISH = re.compile(r"\b[A-Za-z0-9+/=_-]{24,}\b") + + +def is_sensitive_column(column_name: str) -> bool: + name = (column_name or "").lower() + return any(tok in name for tok in SENSITIVE_COLUMN_TOKENS) + + +def redact_value(value: str) -> str: + if not value: + return value + s = str(value) + s = _JWT.sub("[redacted-jwt]", s) + s = _EMAIL.sub("[redacted-email]", s) + s = _CARD.sub("[redacted-card]", s) + s = _PHONE.sub("[redacted-phone]", s) + s = _SECRETISH.sub("[redacted-secret]", s) + return s + + +def scrub_distinct_values(column_name: str, values: list, *, enabled: bool = True) -> list: + if not enabled: + return values + if is_sensitive_column(column_name): + return [f"[redacted: {len(values)} values]"] + return [redact_value(str(v)) for v in values] + + +def scrub_sample_json(sample_json: str, *, enabled: bool = True) -> str: + if not enabled or not sample_json: + return sample_json + try: + rows = json.loads(sample_json) + except (json.JSONDecodeError, TypeError): + return redact_value(sample_json) + if not isinstance(rows, list): + return redact_value(sample_json) + cleaned: list = [] + for row in rows: + if isinstance(row, dict): + cleaned.append( + { + k: ( + "[redacted]" + if is_sensitive_column(str(k)) + else (redact_value(v) if isinstance(v, str) else v) + ) + for k, v in row.items() + } + ) + else: + cleaned.append(redact_value(row) if isinstance(row, str) else row) + return json.dumps(cleaned, default=str) + + +def scrub_row_cells(columns: list[str], rows: list[list], *, enabled: bool = True) -> list[list]: + if not enabled: + return rows + sensitive_idx = {i for i, c in enumerate(columns) if is_sensitive_column(str(c))} + out: list[list] = [] + for row in rows: + new_row = [] + for i, cell in enumerate(row): + if i in sensitive_idx: + new_row.append("[redacted]") + elif isinstance(cell, str): + new_row.append(redact_value(cell)) + else: + new_row.append(cell) + out.append(new_row) + return out diff --git a/backend/tests/unit/knowledge/test_pii_scrubber.py b/backend/tests/unit/knowledge/test_pii_scrubber.py new file mode 100644 index 0000000..da38b0a --- /dev/null +++ b/backend/tests/unit/knowledge/test_pii_scrubber.py @@ -0,0 +1,41 @@ +from app.knowledge import pii_scrubber as p + + +def test_sensitive_column_detection(): + assert p.is_sensitive_column("password_hash") + assert p.is_sensitive_column("user_API_Key") + assert not p.is_sensitive_column("created_at") + + +def test_redact_email_phone_card_jwt(): + assert "[redacted-email]" in p.redact_value("contact a@b.com please") + assert "[redacted-card]" in p.redact_value("4111 1111 1111 1111") + assert "[redacted-jwt]" in p.redact_value("eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.abc.def") + assert p.redact_value("just text") == "just text" + + +def test_scrub_distinct_sensitive_column_returns_cardinality_only(): + out = p.scrub_distinct_values("email", ["a@b.com", "c@d.com"]) + assert out == ["[redacted: 2 values]"] + + +def test_scrub_distinct_disabled_returns_raw(): + assert p.scrub_distinct_values("email", ["a@b.com"], enabled=False) == ["a@b.com"] + + +def test_scrub_sample_json_redacts_and_survives_bad_json(): + good = '[{"email": "a@b.com", "note": "call 415-555-1212"}]' + out = p.scrub_sample_json(good) + assert "a@b.com" not in out and "415-555-1212" not in out + # non-JSON still masked, not raw + assert "a@b.com" not in p.scrub_sample_json("raw blob a@b.com") + assert p.scrub_sample_json("anything", enabled=False) == "anything" + + +def test_scrub_row_cells_redacts_sensitive_columns(): + cols = ["id", "password", "email"] + rows = [[1, "hunter2", "a@b.com"]] + out = p.scrub_row_cells(cols, rows) + assert out[0][0] == 1 + assert "hunter2" not in str(out[0][1]) + assert "a@b.com" not in str(out[0][2]) From ecb9172c5034c9c43b64297d556e58377c9fc947 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 15:13:11 +0200 Subject: [PATCH 06/24] fix(sync): word-component match in is_sensitive_column to avoid over-redaction (T2 review) --- backend/app/knowledge/pii_scrubber.py | 17 ++++++++++++++--- .../tests/unit/knowledge/test_pii_scrubber.py | 14 ++++++++++++++ 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/backend/app/knowledge/pii_scrubber.py b/backend/app/knowledge/pii_scrubber.py index 37647f0..0cd28f4 100644 --- a/backend/app/knowledge/pii_scrubber.py +++ b/backend/app/knowledge/pii_scrubber.py @@ -39,7 +39,7 @@ ) _EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}") -_JWT = re.compile(r"eyJ[A-Za-z0-9_-]{1,}\.[A-Za-z0-9_-]{1,}\.[A-Za-z0-9_-]{1,}") +_JWT = re.compile(r"eyJ[A-Za-z0-9_-]{2,}\.[A-Za-z0-9_-]{2,}\.[A-Za-z0-9_-]{2,}") _CARD = re.compile(r"\b(?:\d[ -]?){13,19}\b") _PHONE = re.compile(r"\b\+?\d[\d\s().-]{7,}\d\b") # long hex / base64-ish secrets (>= 24 chars, no spaces) @@ -47,8 +47,19 @@ def is_sensitive_column(column_name: str) -> bool: - name = (column_name or "").lower() - return any(tok in name for tok in SENSITIVE_COLUMN_TOKENS) + n = (column_name or "").lower() + parts = set(re.split(r"[^a-z0-9]+", n)) + for tok in SENSITIVE_COLUMN_TOKENS: + if len(tok) >= 6 or "_" in tok: + # unambiguous (long or underscored) token: substring match is safe + if tok in n: + return True + else: + # short ambiguous token (pan, auth, hash, salt, ssn, cvv, ...): + # require a whole-component match so "pan" != "company_name" + if tok in parts: + return True + return False def redact_value(value: str) -> str: diff --git a/backend/tests/unit/knowledge/test_pii_scrubber.py b/backend/tests/unit/knowledge/test_pii_scrubber.py index da38b0a..4ffe6a3 100644 --- a/backend/tests/unit/knowledge/test_pii_scrubber.py +++ b/backend/tests/unit/knowledge/test_pii_scrubber.py @@ -39,3 +39,17 @@ def test_scrub_row_cells_redacts_sensitive_columns(): assert out[0][0] == 1 assert "hunter2" not in str(out[0][1]) assert "a@b.com" not in str(out[0][2]) + + +def test_sensitive_column_no_false_positive_on_company_name(): + assert p.is_sensitive_column("company_name") is False # 'pan' must not match + assert p.is_sensitive_column("author_id") is False # 'auth' must not match + assert p.is_sensitive_column("hashtag") is False # 'hash' must not match + + +def test_sensitive_column_whole_component_matches(): + assert p.is_sensitive_column("pan") is True + assert p.is_sensitive_column("card_pan") is True + assert p.is_sensitive_column("user_ssn") is True + assert p.is_sensitive_column("password_hash") is True + assert p.is_sensitive_column("email") is True From 1315cbffa659f0eeb5969e652893141a9d65503f Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 15:15:35 +0200 Subject: [PATCH 07/24] feat(sync): owner-attributed budget pre-flight + usage sink helper (H5) Implement sync_budget.py with C1 correction: missing owner degrades gracefully (unenforced) rather than blocking syncs for legacy/deleted-owner projects. - resolve_owner_user_id: query project owner from DB - build_sink: instantiate DbUsageSink with owner attribution - preflight_owner_budget: pre-flight check with C1 graceful degradation TDD: 3 tests, all passing. C1 applied: test_preflight_owner_missing asserts ok=True when owner is None (unenforced, not blocked). Co-Authored-By: Claude Opus 4.8 (1M context) --- backend/app/services/sync_budget.py | 46 ++++++++++++++++ .../tests/unit/services/test_sync_budget.py | 52 +++++++++++++++++++ 2 files changed, 98 insertions(+) create mode 100644 backend/app/services/sync_budget.py create mode 100644 backend/tests/unit/services/test_sync_budget.py diff --git a/backend/app/services/sync_budget.py b/backend/app/services/sync_budget.py new file mode 100644 index 0000000..e034f4b --- /dev/null +++ b/backend/app/services/sync_budget.py @@ -0,0 +1,46 @@ +"""Owner-attributed budget gate + usage sink for the code↔DB sync pipeline (H5).""" + +from __future__ import annotations + +import logging + +from sqlalchemy import select +from sqlalchemy.ext.asyncio import AsyncSession + +from app.config import settings +from app.llm.usage_sink import DbUsageSink +from app.models.project import Project +from app.services.usage_service import UsageService + +logger = logging.getLogger(__name__) +_usage_svc = UsageService() + + +async def resolve_owner_user_id(session: AsyncSession, project_id: str) -> str | None: + """Resolve a project's owner user ID.""" + row = await session.execute(select(Project.owner_id).where(Project.id == project_id)) + return row.scalar_one_or_none() + + +def build_sink(owner_user_id: str, project_id: str) -> DbUsageSink: + """Build a usage sink attributed to the owner.""" + return DbUsageSink(user_id=owner_user_id, project_id=project_id) + + +async def preflight_owner_budget( + session: AsyncSession, project_id: str +) -> tuple[bool, str | None, str | None]: + """Pre-flight budget check for sync operations (C1: graceful degradation when owner missing). + + Returns (ok, reason, owner_user_id): + - ok=True, reason=None: proceed (enforced pass or unenforced) + - ok=False, reason=msg: blocked (budget exceeded) + """ + owner_id = await resolve_owner_user_id(session, project_id) + if not owner_id: + logger.warning("sync budget unenforced: project %s has no owner", project_id[:8]) + return True, None, None # C1: degrade (unenforced), do NOT block + if not settings.sync_budget_enforcement_enabled: + return True, None, owner_id + msg = await _usage_svc.check_token_budget(session, owner_id) + return (False, msg, owner_id) if msg else (True, None, owner_id) diff --git a/backend/tests/unit/services/test_sync_budget.py b/backend/tests/unit/services/test_sync_budget.py new file mode 100644 index 0000000..8900b7a --- /dev/null +++ b/backend/tests/unit/services/test_sync_budget.py @@ -0,0 +1,52 @@ +"""Tests for owner-attributed budget gate + usage sink (H5).""" + +import pytest + +from app.services import sync_budget + + +class _FakeProject: + def __init__(self, owner_id): + self.owner_id = owner_id + + +@pytest.fixture +def patch_owner(monkeypatch): + def _set(owner): + async def _resolve(session, project_id): + return owner + + monkeypatch.setattr(sync_budget, "resolve_owner_user_id", _resolve) + + return _set + + +async def test_preflight_disabled_always_ok(monkeypatch, patch_owner): + monkeypatch.setattr(sync_budget.settings, "sync_budget_enforcement_enabled", False) + patch_owner("u1") + ok, reason, owner = await sync_budget.preflight_owner_budget(None, "p1") + assert ok is True and reason is None and owner == "u1" + + +async def test_preflight_blocks_when_budget_message(monkeypatch, patch_owner): + monkeypatch.setattr(sync_budget.settings, "sync_budget_enforcement_enabled", True) + patch_owner("u1") + + async def _budget(db, user_id): + return "daily token budget exhausted" + + monkeypatch.setattr(sync_budget._usage_svc, "check_token_budget", _budget) + ok, reason, owner = await sync_budget.preflight_owner_budget(None, "p1") + assert ok is False and "budget" in reason and owner == "u1" + + +async def test_preflight_owner_missing(monkeypatch): + """C1: missing owner degrades (unenforced), does NOT block.""" + monkeypatch.setattr(sync_budget.settings, "sync_budget_enforcement_enabled", True) + + async def _resolve(session, project_id): + return None + + monkeypatch.setattr(sync_budget, "resolve_owner_user_id", _resolve) + ok, reason, owner = await sync_budget.preflight_owner_budget(None, "p1") + assert ok is True and reason is None and owner is None From bf57ee1bed01a89ebe2eaed6c819bae7823b5f24 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 15:20:41 +0200 Subject: [PATCH 08/24] fix(sync): reconcile batch analyses by table_name + robust confidence + fallback marker (H2,H3,H4) Co-Authored-By: Claude Sonnet 4.6 --- .../app/knowledge/code_db_sync_analyzer.py | 107 ++++++++++++------ .../knowledge/test_code_db_sync_analyzer.py | 64 +++++++++++ 2 files changed, 135 insertions(+), 36 deletions(-) create mode 100644 backend/tests/unit/knowledge/test_code_db_sync_analyzer.py diff --git a/backend/app/knowledge/code_db_sync_analyzer.py b/backend/app/knowledge/code_db_sync_analyzer.py index bded351..0708d38 100644 --- a/backend/app/knowledge/code_db_sync_analyzer.py +++ b/backend/app/knowledge/code_db_sync_analyzer.py @@ -24,10 +24,36 @@ def _clamp_sync_status(raw: str) -> str: return raw if raw in _VALID_SYNC_STATUS else "unknown" +def _coerce_confidence(raw) -> int: + """Coerce an LLM-returned confidence value to a valid 1-5 integer. + + Accepts int or numeric strings that represent whole numbers (e.g. "4"). + Float strings (e.g. "4.5") are treated as malformed and return the safe + default of 3 so a single bad tool-call does not abort the entire batch. + """ + if isinstance(raw, bool): + return 3 + if isinstance(raw, int): + return max(1, min(5, raw)) + try: + int(str(raw)) # raises ValueError for "4.5" + return max(1, min(5, int(str(raw)))) + except (TypeError, ValueError): + return 3 + + SYNC_ANALYSIS_TOOL = Tool( name="table_sync_analysis", description="Return a structured analysis of how code uses this database table", parameters=[ + ToolParameter( + name="table_name", + type="string", + description=( + "The EXACT table name being analyzed, copied verbatim from the " + "'## Table: ' header. Required so results map to the right table." + ), + ), ToolParameter( name="data_format_notes", type="string", @@ -162,6 +188,7 @@ class TableSyncAnalysis: column_value_mappings_json: str = "{}" sync_status: str = "unknown" confidence_score: int = 3 + is_fallback: bool = False @dataclass @@ -220,7 +247,7 @@ async def analyze_table( required_filters_json=args.get("required_filters", "{}"), column_value_mappings_json=args.get("column_value_mappings", "{}"), sync_status=_clamp_sync_status(args.get("sync_status", "unknown")), - confidence_score=max(1, min(5, int(args.get("confidence_score", 3)))), + confidence_score=_coerce_confidence(args.get("confidence_score", 3)), ) logger.info( "LLM sync: %s → %s (confidence=%d)", @@ -263,7 +290,8 @@ async def analyze_table_batch( Message(role="user", content="\n".join(prompt_parts)), ] - results: list[TableSyncAnalysis] = [] + results_by_name: dict[str, TableSyncAnalysis] = {} + by_name = {t[0].lower(): t[0] for t in tables} try: resp = await self._llm.complete( messages=messages, @@ -273,46 +301,52 @@ async def analyze_table_batch( temperature=0.0, max_tokens=4096, ) - - tool_idx = 0 for tc in resp.tool_calls: - if tc.name == "table_sync_analysis" and tool_idx < len(tables): - args = tc.arguments - tbl_name = tables[tool_idx][0] - col_notes = args.get("column_sync_notes", "{}") - if isinstance(col_notes, dict): - col_notes = json.dumps(col_notes) - results.append( - TableSyncAnalysis( - table_name=tbl_name, - data_format_notes=args.get("data_format_notes", ""), - column_sync_notes_json=col_notes, - business_logic_notes=args.get("business_logic_notes", ""), - conversion_warnings=args.get("conversion_warnings", ""), - query_recommendations=args.get("query_recommendations", ""), - required_filters_json=args.get("required_filters", "{}"), - column_value_mappings_json=args.get("column_value_mappings", "{}"), - sync_status=_clamp_sync_status(args.get("sync_status", "unknown")), - confidence_score=max(1, min(5, int(args.get("confidence_score", 3)))), - ) + if tc.name != "table_sync_analysis": + continue + args = tc.arguments + raw_name = str(args.get("table_name", "")).lower() + canonical = by_name.get(raw_name) + if canonical is None: + logger.warning( + "batch sync: tool call for unknown table %r — dropped", + args.get("table_name"), ) - tool_idx += 1 - + continue + if canonical in results_by_name: + logger.warning( + "batch sync: duplicate analysis for %s — keeping first", canonical + ) + continue + col_notes = args.get("column_sync_notes", "{}") + if isinstance(col_notes, dict): + col_notes = json.dumps(col_notes) + results_by_name[canonical] = TableSyncAnalysis( + table_name=canonical, + data_format_notes=args.get("data_format_notes", ""), + column_sync_notes_json=col_notes, + business_logic_notes=args.get("business_logic_notes", ""), + conversion_warnings=args.get("conversion_warnings", ""), + query_recommendations=args.get("query_recommendations", ""), + required_filters_json=args.get("required_filters", "{}"), + column_value_mappings_json=args.get("column_value_mappings", "{}"), + sync_status=_clamp_sync_status(args.get("sync_status", "unknown")), + confidence_score=_coerce_confidence(args.get("confidence_score", 3)), + ) except Exception: logger.warning("Batch sync analysis failed", exc_info=True) - fallback_count = len(tables) - len(results) - for i in range(len(results), len(tables)): - results.append(self._fallback_analysis(tables[i][0])) - + out: list[TableSyncAnalysis] = [] + fallback_count = 0 + for name, _db, _code in tables: + if name in results_by_name: + out.append(results_by_name[name]) + else: + out.append(self._fallback_analysis(name)) + fallback_count += 1 if fallback_count: - logger.info( - "LLM sync batch: %d/%d used fallback", - fallback_count, - len(tables), - ) - - return results + logger.info("LLM sync batch: %d/%d used fallback", fallback_count, len(tables)) + return out async def generate_summary( self, @@ -464,4 +498,5 @@ def _fallback_analysis(table_name: str) -> TableSyncAnalysis: sync_status="unknown", confidence_score=1, data_format_notes="LLM analysis unavailable — using fallback.", + is_fallback=True, ) diff --git a/backend/tests/unit/knowledge/test_code_db_sync_analyzer.py b/backend/tests/unit/knowledge/test_code_db_sync_analyzer.py new file mode 100644 index 0000000..adfaec3 --- /dev/null +++ b/backend/tests/unit/knowledge/test_code_db_sync_analyzer.py @@ -0,0 +1,64 @@ +"""Unit tests for CodeDbSyncAnalyzer.""" + +from app.knowledge.code_db_sync_analyzer import CodeDbSyncAnalyzer +from app.llm.base import LLMResponse, ToolCall + + +class _Router: + def __init__(self, calls): + self._calls = calls + + async def complete(self, **kwargs): + return LLMResponse(tool_calls=self._calls) + + +def _tc(table_name, conf=4, status="matched"): + return ToolCall( + id="x", + name="table_sync_analysis", + arguments={ + "table_name": table_name, + "sync_status": status, + "confidence_score": conf, + "required_filters": "{}", + "column_value_mappings": "{}", + }, + ) + + +async def test_batch_reconciles_by_name_not_position(): + tables = [("orders", "", ""), ("payments", "", "")] + # LLM returns them REVERSED + analyzer = CodeDbSyncAnalyzer(_Router([_tc("payments"), _tc("orders")])) + out = await analyzer.analyze_table_batch(tables) + by_name = {a.table_name: a for a in out} + assert by_name["orders"].sync_status == "matched" + assert by_name["payments"].sync_status == "matched" + assert not by_name["orders"].is_fallback + + +async def test_batch_unknown_name_dropped_and_missing_filled_with_fallback(): + tables = [("orders", "", ""), ("payments", "", "")] + analyzer = CodeDbSyncAnalyzer(_Router([_tc("orders"), _tc("ghost_table")])) + out = await analyzer.analyze_table_batch(tables) + by_name = {a.table_name: a for a in out} + assert len(out) == 2 + assert by_name["payments"].is_fallback is True # never returned by LLM + assert by_name["orders"].is_fallback is False + + +async def test_batch_bad_confidence_only_degrades_that_table(): + tables = [("orders", "", ""), ("payments", "", "")] + bad = _tc("orders") + bad.arguments["confidence_score"] = "4.5" + analyzer = CodeDbSyncAnalyzer(_Router([bad, _tc("payments", conf=5)])) + out = await analyzer.analyze_table_batch(tables) + by_name = {a.table_name: a for a in out} + assert by_name["orders"].confidence_score == 3 # coerced, not fallback + assert by_name["orders"].is_fallback is False + assert by_name["payments"].confidence_score == 5 + + +async def test_fallback_marked(): + a = CodeDbSyncAnalyzer._fallback_analysis("t") + assert a.is_fallback is True and a.confidence_score == 1 From 19d58ace6f7deeb2f2568686b78b3948f12e343a Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 15:32:45 +0200 Subject: [PATCH 09/24] fix(sync): schema-qualified identity, all-fallback guard, PII scrub, budget wiring, op_kind heuristic label (M2,H4,H6,H5,H2,M7,L4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - M2: _match_tables now uses (schema, name) tuple key; schema-ambiguous tables get qualified display name and a NOTE in code_context - H4: all-fallback guard aborts Step 5 store when < sync_min_success_ratio of analyses are non-fallback, leaving previous sync rows intact - H6: _build_db_context gains scrub/omit_samples params; run() reads Connection.send_sample_data_to_llm and settings.sync_pii_scrubbing_enabled to set local vars threaded through _match_tables → _build_db_context - H5: budget preflight (preflight_owner_budget) + per-run DbUsageSink wired into run() before heartbeat; Step 6 summary skipped when sink.budget_exceeded() - H2: store loop guards mt is None with continue + warning instead of if-mt-else ternaries; removes phantom upserts for misnamed analyses - M7: _AMBIGUOUS_VERBS tuple extracted; process_/handle_/sync_/set_/add_/ register_ removed from _WRITE_VERBS and classified as "unknown" - L4: distinct values block emits (+N more) when vals > 15; sample_data block appends …[truncated] when truncated at 800 chars - _make_matched() static helper DRYs up _MatchedTable construction Co-Authored-By: Claude Sonnet 4.6 --- .../app/knowledge/code_db_sync_pipeline.py | 272 +++++++++++++----- backend/app/knowledge/graph_db_bridge.py | 19 +- .../test_code_db_sync_pipeline_run.py | 55 ++++ .../unit/knowledge/test_graph_db_bridge.py | 43 +++ 4 files changed, 309 insertions(+), 80 deletions(-) create mode 100644 backend/tests/unit/knowledge/test_code_db_sync_pipeline_run.py create mode 100644 backend/tests/unit/knowledge/test_graph_db_bridge.py diff --git a/backend/app/knowledge/code_db_sync_pipeline.py b/backend/app/knowledge/code_db_sync_pipeline.py index fc47e98..0b7c52f 100644 --- a/backend/app/knowledge/code_db_sync_pipeline.py +++ b/backend/app/knowledge/code_db_sync_pipeline.py @@ -16,7 +16,11 @@ from app.core.heartbeat import heartbeat from app.core.workflow_tracker import WorkflowTracker from app.core.workflow_tracker import tracker as default_tracker -from app.knowledge.code_db_sync_analyzer import CodeDbSyncAnalyzer, TableSyncAnalysis +from app.knowledge.code_db_sync_analyzer import ( + CodeDbSyncAnalyzer, + SyncSummaryResult, + TableSyncAnalysis, +) from app.knowledge.custom_rules import CustomRulesEngine from app.knowledge.entity_extractor import EntityInfo, ProjectKnowledge, TableUsage from app.llm.router import LLMRouter @@ -62,6 +66,36 @@ async def run( {"connection_id": connection_id, "project_id": project_id}, ) + # H5: owner budget pre-flight + per-run usage sink. + from app.services.sync_budget import build_sink, preflight_owner_budget + + if settings.sync_budget_enforcement_enabled: + async with async_session_factory() as s: + ok, reason, owner_id = await preflight_owner_budget(s, project_id) + if not ok: + async with async_session_factory() as s: + await self._sync_svc.set_sync_status(s, connection_id, "failed") + await s.commit() + await self._tracker.end(wf_id, "code_db_sync", "failed", reason or "budget") + return { + "status": "failed", + "error": reason, + "budget_blocked": True, + "workflow_id": wf_id, + } + if owner_id: + self._llm = LLMRouter(usage_sink=build_sink(owner_id, project_id)) + self._analyzer = CodeDbSyncAnalyzer(self._llm) + + # H6: per-connection opt-out + global scrub flag. + async with async_session_factory() as s: + from app.models.connection import Connection + + conn = await s.get(Connection, connection_id) + send = getattr(conn, "send_sample_data_to_llm", True) if conn else True + scrub = settings.sync_pii_scrubbing_enabled + omit_samples = not send + async def _hb() -> None: async with async_session_factory() as s: await self._sync_svc.touch_heartbeat(s, connection_id) @@ -146,7 +180,13 @@ async def _hb() -> None: "match_tables", f"Matching {len(db_entries)} DB tables with code entities", ): - matched_tables = self._match_tables(knowledge, db_entries, rules_context) + matched_tables = self._match_tables( + knowledge, + db_entries, + rules_context, + scrub=scrub, + omit_samples=omit_samples, + ) code_info_count = sum(1 for m in matched_tables if m.has_code_info) db_only_match = len(matched_tables) - code_info_count @@ -238,6 +278,34 @@ async def _one_small_batch(batch_list): f"({len(batch)} tables): {batch_names}", ) + # H4: all-fallback guard — abort if LLM was degraded. + total_analyses = len(analyses) + non_fallback = sum(1 for a in analyses if not a.is_fallback) + if ( + total_analyses + and (non_fallback / total_analyses) < settings.sync_min_success_ratio_to_persist + ): + logger.warning( + "CODE_DB_SYNC kept previous rows: only %d/%d tables analyzed", + non_fallback, + total_analyses, + ) + async with async_session_factory() as session: + await self._sync_svc.set_sync_status(session, connection_id, "failed") + await session.commit() + await self._tracker.end( + wf_id, + "code_db_sync", + "failed", + f"LLM degraded: {non_fallback}/{total_analyses} analyzed; " + "kept previous sync", + ) + return { + "status": "failed", + "error": "llm_degraded_kept_previous", + "workflow_id": wf_id, + } + # Step 5: Store results async with self._tracker.step( wf_id, @@ -260,14 +328,20 @@ async def _one_small_batch(batch_list): mt_lookup = {m.table_name: m for m in matched_tables} for analysis in analyses: mt = mt_lookup.get(analysis.table_name) # type: ignore[assignment] + if mt is None: + logger.warning( + "store_sync: no matched table for %s — skipped", + analysis.table_name, + ) + continue sync_data = { "table_name": analysis.table_name, - "entity_name": mt.entity_name if mt else None, - "entity_file_path": mt.entity_file_path if mt else None, - "code_columns_json": mt.code_columns_json if mt else "[]", - "used_in_files_json": mt.used_in_files_json if mt else "[]", - "read_count": mt.read_count if mt else 0, - "write_count": mt.write_count if mt else 0, + "entity_name": mt.entity_name, + "entity_file_path": mt.entity_file_path, + "code_columns_json": mt.code_columns_json, + "used_in_files_json": mt.used_in_files_json, + "read_count": mt.read_count, + "write_count": mt.write_count, "data_format_notes": analysis.data_format_notes, "column_sync_notes_json": analysis.column_sync_notes_json, "business_logic_notes": analysis.business_logic_notes, @@ -316,13 +390,17 @@ async def _one_small_batch(batch_list): "started", "Generating LLM summary with FK relationships", ) - summary_result = await self._analyzer.generate_summary( - analyses=analyses, - project_context=project_ctx, - fk_relationships=fk_ctx, - preferred_provider=preferred_provider, - model=model, - ) + sink = getattr(self._llm, "_sink", None) + if sink is not None and sink.budget_exceeded(): + summary_result = SyncSummaryResult() # skip LLM summary + else: + summary_result = await self._analyzer.generate_summary( + analyses=analyses, + project_context=project_ctx, + fk_relationships=fk_ctx, + preferred_provider=preferred_provider, + model=model, + ) async with async_session_factory() as session: await self._sync_svc.upsert_summary( @@ -401,69 +479,76 @@ def _match_tables( knowledge: ProjectKnowledge, db_entries: list[DbIndex], rules_context: str = "", + *, + scrub: bool = True, + omit_samples: bool = False, ) -> list[_MatchedTable]: """Cross-reference code entities/table_usage with DB index entries.""" + from collections import Counter + results: list[_MatchedTable] = [] - db_table_names = {e.table_name.lower(): e for e in db_entries} - code_table_names: set[str] = set() + db_by_key: dict[tuple[str, str], DbIndex] = {} + bare_counts: Counter = Counter() + for e in db_entries: + sch = (getattr(e, "table_schema", None) or "public").lower() + nm = e.table_name.lower() + db_by_key[(sch, nm)] = e + bare_counts[nm] += 1 + + def _display_name(e: DbIndex) -> str: + if bare_counts[e.table_name.lower()] > 1: + return f"{getattr(e, 'table_schema', 'public') or 'public'}.{e.table_name}" + return e.table_name entity_by_table: dict[str, EntityInfo] = {} + code_table_names: set[str] = set() for _, entity in knowledge.entities.items(): if entity.table_name: entity_by_table[entity.table_name.lower()] = entity code_table_names.add(entity.table_name.lower()) - for tbl_name in knowledge.table_usage: code_table_names.add(tbl_name.lower()) - all_tables = set(db_table_names.keys()) | code_table_names - - for tbl_lower in sorted(all_tables): - db_entry = db_table_names.get(tbl_lower) - entity = entity_by_table.get(tbl_lower) # type: ignore[assignment] - usage = knowledge.table_usage.get(tbl_lower) or knowledge.table_usage.get( - next((k for k in knowledge.table_usage if k.lower() == tbl_lower), "") + # DB-side first (schema-qualified), then code-only tables with no DB row. + seen_bare: set[str] = set() + for (sch, nm), db_entry in sorted(db_by_key.items()): + seen_bare.add(nm) + entity = entity_by_table.get(nm) # type: ignore[assignment] + usage = knowledge.table_usage.get(nm) or knowledge.table_usage.get( + next((k for k in knowledge.table_usage if k.lower() == nm), "") ) - - table_name = db_entry.table_name if db_entry else tbl_lower - - db_context = self._build_db_context(db_entry) if db_entry else "" - code_context = self._build_code_context( - entity, usage, knowledge, tbl_lower, rules_context + ambiguous = bare_counts[nm] > 1 + display = _display_name(db_entry) + code_context = self._build_code_context(entity, usage, knowledge, nm, rules_context) + if ambiguous: + code_context = ( + f"(NOTE: table name '{nm}' exists in multiple schemas; matched code by " + f"bare name — verify schema '{sch}')\n" + code_context + ) + results.append( + self._make_matched( + display, + self._build_db_context(db_entry, scrub=scrub, omit_samples=omit_samples), + code_context, + entity, + usage, + knowledge, + ) ) - has_code = bool(entity or (usage and usage.is_active)) - _has_db = db_entry is not None - - mt = _MatchedTable( - table_name=table_name, - db_context=db_context, - code_context=code_context, - has_code_info=has_code, - entity_name=entity.name if entity else None, - entity_file_path=entity.file_path if entity else None, - read_count=len(usage.readers) if usage else 0, - write_count=len(usage.writers) if usage else 0, + for nm in sorted(code_table_names - seen_bare): + entity = entity_by_table.get(nm) # type: ignore[assignment] + usage = knowledge.table_usage.get(nm) or knowledge.table_usage.get( + next((k for k in knowledge.table_usage if k.lower() == nm), "") ) - - if entity and entity.columns: - mt.code_columns_json = json.dumps( - [ - {"name": c.name, "type": c.col_type, "fk_target": c.fk_target} - for c in entity.columns - ] - ) - - if usage: - all_files = list(set(usage.readers + usage.writers + usage.orm_refs)) - mt.used_in_files_json = json.dumps(all_files[:20]) - - results.append(mt) - + code_context = self._build_code_context(entity, usage, knowledge, nm, rules_context) + results.append(self._make_matched(nm, "", code_context, entity, usage, knowledge)) return results @staticmethod - def _build_db_context(entry: DbIndex) -> str: + def _build_db_context(entry: DbIndex, *, scrub: bool = True, omit_samples: bool = False) -> str: + from app.knowledge import pii_scrubber + parts: list[str] = [] if entry.business_description: parts.append(f"Description: {entry.business_description}") @@ -484,21 +569,60 @@ def _build_db_context(entry: DbIndex) -> str: parts.append(f" {col}: {note}") except (json.JSONDecodeError, TypeError): pass - dv_json = getattr(entry, "column_distinct_values_json", None) or "{}" - if dv_json and dv_json != "{}": - try: - distinct = json.loads(dv_json) - if distinct: - parts.append("Actual distinct values in DB:") - for col, vals in distinct.items(): - vals_str = " | ".join(str(v) for v in vals[:15]) - parts.append(f" {col}: [{vals_str}]") - except (json.JSONDecodeError, TypeError): - pass - if entry.sample_data_json and entry.sample_data_json != "[]": - parts.append(f"Sample data: {entry.sample_data_json[:800]}") + if not omit_samples: + dv_json = getattr(entry, "column_distinct_values_json", None) or "{}" + if dv_json and dv_json != "{}": + try: + distinct = json.loads(dv_json) + if distinct: + parts.append("Actual distinct values in DB:") + for col, vals in distinct.items(): + shown = pii_scrubber.scrub_distinct_values( + col, vals[:15], enabled=scrub + ) + vals_str = " | ".join(str(v) for v in shown) + more = f" (+{len(vals) - 15} more)" if len(vals) > 15 else "" + parts.append(f" {col}: [{vals_str}]{more}") + except (json.JSONDecodeError, TypeError): + pass + if entry.sample_data_json and entry.sample_data_json != "[]": + sample = pii_scrubber.scrub_sample_json(entry.sample_data_json, enabled=scrub) + suffix = "…[truncated]" if len(sample) > 800 else "" + parts.append(f"Sample data: {sample[:800]}{suffix}") return "\n".join(parts) + @staticmethod + def _make_matched( + table_name: str, + db_context: str, + code_context: str, + entity: EntityInfo | None, + usage: TableUsage | None, + knowledge: ProjectKnowledge, + ) -> _MatchedTable: + has_code = bool(entity or (usage and usage.is_active)) + mt = _MatchedTable( + table_name=table_name, + db_context=db_context, + code_context=code_context, + has_code_info=has_code, + entity_name=entity.name if entity else None, + entity_file_path=entity.file_path if entity else None, + read_count=len(usage.readers) if usage else 0, + write_count=len(usage.writers) if usage else 0, + ) + if entity and entity.columns: + mt.code_columns_json = json.dumps( + [ + {"name": c.name, "type": c.col_type, "fk_target": c.fk_target} + for c in entity.columns + ] + ) + if usage: + all_files = list(set(usage.readers + usage.writers + usage.orm_refs)) + mt.used_in_files_json = json.dumps(all_files[:20]) + return mt + @staticmethod def _build_code_context( entity: EntityInfo | None, @@ -552,10 +676,10 @@ def _build_code_context( for r in refs[:5]: op = r.get("op_kind", "unknown") conf = float(r.get("confidence", 0.0)) - depth = int(r.get("depth", 1)) + int(r.get("depth", 1)) name = r.get("caller_name", "?") file_ = r.get("caller_file", "?") - parts.append(f" - {name} ({op}, depth={depth}, conf={conf:.2f}) in {file_}") + parts.append(f" - {name} ({op}, conf={conf:.2f}, heuristic) in {file_}") relevant_enums = [ e diff --git a/backend/app/knowledge/graph_db_bridge.py b/backend/app/knowledge/graph_db_bridge.py index 78e86af..b34dbee 100644 --- a/backend/app/knowledge/graph_db_bridge.py +++ b/backend/app/knowledge/graph_db_bridge.py @@ -110,16 +110,10 @@ "store_", "persist_", "write_", - "add_", - "set_", "post_", "put_", "patch_", "modify_", - "process_", - "handle_", - "sync_", - "register_", "submit_", "approve_", "reject_", @@ -146,6 +140,16 @@ "summarize_", "export_", ) +# Verbs that could imply either read or write depending on context — +# classified as "unknown" rather than guessing. +_AMBIGUOUS_VERBS = ( + "process_", + "handle_", + "sync_", + "set_", + "add_", + "register_", +) # Tokens in HTTP route decorators that hint at the op kind. ``GET`` → read, # anything else → write. @@ -216,6 +220,9 @@ def classify_op_kind(symbol: Symbol) -> str: for verb in _READ_VERBS: if name.startswith(verb): return "read" + for verb in _AMBIGUOUS_VERBS: + if name.startswith(verb): + return "unknown" # HTTP method hint via decorator (e.g. ``@router.post('/users')``). for dec in symbol.decorators or (): diff --git a/backend/tests/unit/knowledge/test_code_db_sync_pipeline_run.py b/backend/tests/unit/knowledge/test_code_db_sync_pipeline_run.py new file mode 100644 index 0000000..ef2f478 --- /dev/null +++ b/backend/tests/unit/knowledge/test_code_db_sync_pipeline_run.py @@ -0,0 +1,55 @@ +# backend/tests/unit/knowledge/test_code_db_sync_pipeline_run.py +import json + +from app.knowledge.code_db_sync_analyzer import TableSyncAnalysis +from app.knowledge.code_db_sync_pipeline import CodeDbSyncPipeline + + +class _DbEntry: + def __init__(self, name, schema="public"): + self.table_name = name + self.table_schema = schema + self.business_description = "" + self.row_count = None + self.column_count = 0 + self.data_patterns = "" + self.query_hints = "" + self.column_notes_json = "{}" + self.column_distinct_values_json = json.dumps({"email": ["a@b.com", "c@d.com"]}) + self.sample_data_json = json.dumps([{"email": "a@b.com"}]) + + +def test_build_db_context_scrubs_when_enabled(): + ctx = CodeDbSyncPipeline._build_db_context(_DbEntry("users"), scrub=True, omit_samples=False) + assert "a@b.com" not in ctx + assert "redacted" in ctx + + +def test_build_db_context_raw_when_scrub_disabled(): + # scrub=False path: raw data passes through when scrubbing globally off + ctx = CodeDbSyncPipeline._build_db_context(_DbEntry("logs"), scrub=False, omit_samples=False) + assert "a@b.com" in ctx # raw allowed only when scrubbing disabled + + +def test_build_db_context_omit_samples_true(): + ctx = CodeDbSyncPipeline._build_db_context(_DbEntry("users"), scrub=True, omit_samples=True) + assert "a@b.com" not in ctx + assert "Sample data" not in ctx + assert "distinct values" not in ctx.lower() + + +def test_distinct_truncation_marker(): + e = _DbEntry("t") + e.column_distinct_values_json = json.dumps({"k": [str(i) for i in range(20)]}) + ctx = CodeDbSyncPipeline._build_db_context(e, scrub=True, omit_samples=False) + assert "+5 more" in ctx + + +def test_all_fallback_guard_helper(): + analyses = [ + TableSyncAnalysis(table_name="a", is_fallback=True), + TableSyncAnalysis(table_name="b", is_fallback=True), + ] + total = len(analyses) + non_fb = sum(1 for a in analyses if not a.is_fallback) + assert (non_fb / total) < 0.5 # guard would trip diff --git a/backend/tests/unit/knowledge/test_graph_db_bridge.py b/backend/tests/unit/knowledge/test_graph_db_bridge.py new file mode 100644 index 0000000..d39917d --- /dev/null +++ b/backend/tests/unit/knowledge/test_graph_db_bridge.py @@ -0,0 +1,43 @@ +# backend/tests/unit/knowledge/test_graph_db_bridge.py +from app.knowledge.graph_db_bridge import classify_op_kind + + +class _Sym: + name = "process_report" + decorators = () + + +def test_ambiguous_verbs_classified_unknown(): + assert classify_op_kind(_Sym()) == "unknown" + + +def test_handle_verb_classified_unknown(): + class S: + name = "handle_payment" + decorators = () + + assert classify_op_kind(S()) == "unknown" + + +def test_sync_verb_classified_unknown(): + class S: + name = "sync_data" + decorators = () + + assert classify_op_kind(S()) == "unknown" + + +def test_create_still_write(): + class S: + name = "create_user" + decorators = () + + assert classify_op_kind(S()) == "write" + + +def test_get_still_read(): + class S: + name = "get_users" + decorators = () + + assert classify_op_kind(S()) == "read" From 91dbf6b372168c6b26f477fc026e2ac7814bc604 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 15:39:23 +0200 Subject: [PATCH 10/24] fix(sync): validate required_filters payload, deep-merge value mappings, gate prompt header (M6,L2) --- backend/app/services/code_db_sync_service.py | 55 ++++++++++------- .../tests/unit/test_code_db_sync_service.py | 60 +++++++++++++++++++ 2 files changed, 95 insertions(+), 20 deletions(-) diff --git a/backend/app/services/code_db_sync_service.py b/backend/app/services/code_db_sync_service.py index f5ceb4a..6f9930f 100644 --- a/backend/app/services/code_db_sync_service.py +++ b/backend/app/services/code_db_sync_service.py @@ -5,7 +5,7 @@ import json import logging from datetime import UTC, datetime -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING from sqlalchemy import delete, select, update @@ -263,6 +263,17 @@ async def mark_stale_for_project( # Runtime enrichment (from investigation feedback loop) # ------------------------------------------------------------------ + @staticmethod + def _safe_load_dict(raw) -> dict: + """Safely load a JSON string as a dict; return {} on error.""" + if not raw: + return {} + try: + v = json.loads(raw) + return v if isinstance(v, dict) else {} + except (json.JSONDecodeError, TypeError): + return {} + async def add_runtime_enrichment( self, session: AsyncSession, @@ -280,28 +291,31 @@ async def add_runtime_enrichment( if not entry: return None - mergeable_json_fields = {"required_filters_json", "column_value_mappings_json"} appendable_text_fields = {"query_recommendations", "conversion_warnings"} - if field in mergeable_json_fields: - existing_json: dict[str, Any] = {} - current_val = getattr(entry, field, None) - if current_val: - try: - existing_json = json.loads(current_val) - except (json.JSONDecodeError, TypeError): - existing_json = {} - try: - new_data = json.loads(value) - except (json.JSONDecodeError, TypeError): - new_data = {} - if isinstance(existing_json, dict) and isinstance(new_data, dict): - existing_json.update(new_data) + if field == "required_filters_json": + existing_json = self._safe_load_dict(getattr(entry, field, None)) + new_data = self._safe_load_dict(value) + meta_keys = {"source", "filter", "_meta"} + for col, cond in new_data.items(): + if col in meta_keys or not isinstance(cond, str): + continue # only {column: condition_string} pairs are valid filters + existing_json[col] = cond + setattr(entry, field, json.dumps(existing_json)) + elif field == "column_value_mappings_json": + existing_json = self._safe_load_dict(getattr(entry, field, None)) + new_data = self._safe_load_dict(value) + for col, mapping in new_data.items(): + if isinstance(mapping, dict) and isinstance(existing_json.get(col), dict): + existing_json[col].update(mapping) # deep-merge per column + else: + existing_json[col] = mapping setattr(entry, field, json.dumps(existing_json)) elif field in appendable_text_fields: - existing: str = getattr(entry, field, "") or "" - if value not in existing: - setattr(entry, field, f"{existing}\n{value}".strip()) + existing_lines = [ln.strip() for ln in (getattr(entry, field, "") or "").split("\n")] + if value.strip() not in existing_lines: + combined = f"{getattr(entry, field, '') or ''}\n{value}".strip() + setattr(entry, field, combined[-8000:]) # cap growth (keep newest) else: return None @@ -329,7 +343,8 @@ def sync_to_prompt_context( parts: list[str] = [] - if summary and summary.synced_at: + status = getattr(summary, "sync_status", None) if summary else None + if summary and summary.synced_at and status in ("completed", "stale"): parts.append( f"## Code-DB Sync (analyzed {summary.synced_at.strftime('%Y-%m-%d %H:%M')})\n" ) diff --git a/backend/tests/unit/test_code_db_sync_service.py b/backend/tests/unit/test_code_db_sync_service.py index 5376855..f83aaf3 100644 --- a/backend/tests/unit/test_code_db_sync_service.py +++ b/backend/tests/unit/test_code_db_sync_service.py @@ -435,6 +435,64 @@ async def test_merges_json_field(self, db): assert "active" in data assert "paid" in data + @pytest.mark.asyncio + async def test_enrichment_rejects_metadata_keys_in_required_filters(self, db): + proj = await _make_project(db) + conn = await _make_connection(db, proj.id) + await svc.upsert_table_sync( + db, + conn.id, + { + "table_name": "orders", + "required_filters_json": "{}", + }, + ) + await db.commit() + + await svc.add_runtime_enrichment( + db, + conn.id, + "orders", + "required_filters_json", + json.dumps({"source": "investigation", "filter": "x"}), + ) + await db.commit() + + row = await svc.get_table_sync(db, conn.id, "orders") + assert json.loads(row.required_filters_json) == {} # metadata keys dropped + + @pytest.mark.asyncio + async def test_enrichment_deep_merges_value_mappings(self, db): + proj = await _make_project(db) + conn = await _make_connection(db, proj.id) + await svc.upsert_table_sync( + db, + conn.id, + { + "table_name": "orders", + "column_value_mappings_json": json.dumps( + {"status": {"0": "pending", "1": "processed"}} + ), + }, + ) + await db.commit() + + await svc.add_runtime_enrichment( + db, + conn.id, + "orders", + "column_value_mappings_json", + json.dumps({"status": {"2": "failed"}}), + ) + await db.commit() + + row = await svc.get_table_sync(db, conn.id, "orders") + assert json.loads(row.column_value_mappings_json)["status"] == { + "0": "pending", + "1": "processed", + "2": "failed", + } + @pytest.mark.asyncio async def test_appends_text_field(self, db): proj = await _make_project(db) @@ -562,6 +620,7 @@ class _Stub: "global_notes": "", "data_conventions": "", "query_guidelines": "", + "sync_status": None, "synced_at": None, } defaults.update(kwargs) @@ -675,6 +734,7 @@ def test_with_summary_datetime(self): from datetime import UTC, datetime s = _stub_summary( + sync_status="completed", synced_at=datetime(2026, 3, 1, 12, 0, tzinfo=UTC), global_notes="Project uses UTC timestamps", data_conventions="All amounts in cents", From f27da40cbcdd2825a7a9e667c1082ed7e5a9f9e3 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 15:46:46 +0200 Subject: [PATCH 11/24] fix(sync): gate low-confidence required-filters + bare-suffix match for qualified tables (H4,C6) - _load_required_filters_by_table: skip CodeDbSync entries with confidence_score < settings.sync_min_confidence_to_enforce_filters (H4); also index schema-qualified table names (e.g. analytics.orders) under their bare suffix (e.g. orders) so check_required_filters matches bare query references (C6) - _load_sync_filters_and_mappings: apply the same H4 confidence gate so low-confidence entries are also omitted from prompt guidance text - 6 unit tests: 3 H4 cases (low-conf excluded, sufficient-conf included, boundary=threshold passes) and 3 C6 cases (qualified indexed under both keys, qualified+low-conf excluded from both, unqualified not duplicated) Co-Authored-By: Claude Sonnet 4.6 --- backend/app/agents/sql_agent.py | 15 ++ .../agents/test_sql_agent_required_filters.py | 190 ++++++++++++++++++ 2 files changed, 205 insertions(+) create mode 100644 backend/tests/unit/agents/test_sql_agent_required_filters.py diff --git a/backend/app/agents/sql_agent.py b/backend/app/agents/sql_agent.py index d4c3b2c..74d95fd 100644 --- a/backend/app/agents/sql_agent.py +++ b/backend/app/agents/sql_agent.py @@ -1504,6 +1504,7 @@ async def _load_sync_filters_and_mappings(self, connection_id: str) -> tuple[str try: import json as json_mod + from app.config import settings from app.models.base import async_session_factory from app.services.code_db_sync_service import CodeDbSyncService @@ -1511,10 +1512,14 @@ async def _load_sync_filters_and_mappings(self, connection_id: str) -> tuple[str async with async_session_factory() as session: entries = await svc.get_sync(session, connection_id) + min_conf: int = settings.sync_min_confidence_to_enforce_filters filters_lines: list[str] = [] mappings_lines: list[str] = [] for e in entries: + # H4: skip low-confidence entries — omit from prompt guidance too + if (getattr(e, "confidence_score", 0) or 0) < min_conf: + continue rf = getattr(e, "required_filters_json", "{}") or "{}" try: filters = json_mod.loads(rf) @@ -1547,16 +1552,21 @@ async def _load_required_filters_by_table(self, cfg: ConnectionConfig) -> dict[s try: import json as json_mod + from app.config import settings from app.core.required_filter_guard import merge_required_filters from app.models.base import async_session_factory from app.services.code_db_sync_service import CodeDbSyncService from app.services.db_index_service import DbIndexService + min_conf: int = settings.sync_min_confidence_to_enforce_filters sync_filters: dict[str, dict[str, str]] = {} sync_svc = CodeDbSyncService() async with async_session_factory() as session: entries = await sync_svc.get_sync(session, cfg.connection_id) for sync_entry in entries: + # H4: skip low-confidence / fallback rows — do not enforce their filters + if (getattr(sync_entry, "confidence_score", 0) or 0) < min_conf: + continue raw = getattr(sync_entry, "required_filters_json", "{}") or "{}" try: parsed = json_mod.loads(raw) @@ -1564,6 +1574,11 @@ async def _load_required_filters_by_table(self, cfg: ConnectionConfig) -> dict[s parsed = {} if parsed and isinstance(parsed, dict): sync_filters[sync_entry.table_name] = parsed + # C6: also index under bare suffix when table_name is schema-qualified + # (e.g. "analytics.orders" → also register under "orders") + bare = sync_entry.table_name.split(".")[-1] + if bare != sync_entry.table_name: + sync_filters.setdefault(bare, {}).update(parsed) index_hints: dict[str, str] = {} idx_svc = DbIndexService() diff --git a/backend/tests/unit/agents/test_sql_agent_required_filters.py b/backend/tests/unit/agents/test_sql_agent_required_filters.py new file mode 100644 index 0000000..a8dc288 --- /dev/null +++ b/backend/tests/unit/agents/test_sql_agent_required_filters.py @@ -0,0 +1,190 @@ +"""Tests for H4 confidence gate and C6 bare-suffix indexing in SQLAgent required-filters.""" + +import json +from contextlib import asynccontextmanager +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from app.agents.sql_agent import SQLAgent +from app.connectors.base import ConnectionConfig +from app.models.code_db_sync import CodeDbSync # used as spec for MagicMock + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _make_agent() -> SQLAgent: + """Minimal SQLAgent construction without real DB/LLM deps.""" + mock_llm = MagicMock() + mock_llm.complete = AsyncMock() + mock_vs = MagicMock() + mock_vs.query = MagicMock(return_value=[]) + mock_rules = MagicMock() + mock_rules.load_rules = MagicMock(return_value=[]) + mock_rules.load_db_rules = AsyncMock(return_value=[]) + mock_rules.rules_to_context = MagicMock(return_value="") + return SQLAgent(llm_router=mock_llm, vector_store=mock_vs, rules_engine=mock_rules) + + +def _make_sync_entry( + table_name: str, + required_filters: dict, + confidence_score: int, + connection_id: str = "conn-test", +) -> MagicMock: + """Return a duck-typed stand-in for a CodeDbSync ORM row.""" + entry = MagicMock(spec=CodeDbSync) + entry.table_name = table_name + entry.connection_id = connection_id + entry.required_filters_json = json.dumps(required_filters) + entry.confidence_score = confidence_score + entry.column_value_mappings_json = "{}" + return entry + + +def _make_conn_cfg(connection_id: str = "conn-test") -> ConnectionConfig: + return ConnectionConfig( + db_type="postgres", + db_host="localhost", + db_port=5432, + db_name="testdb", + db_user="user", + connection_id=connection_id, + ) + + +def _patch_db(monkeypatch, sync_entries: list) -> None: + """Monkeypatch CodeDbSyncService.get_sync, DbIndexService.get_index, + async_session_factory, and the min-confidence threshold.""" + + async def fake_get_sync(self, session, connection_id): # noqa: ANN001 + return sync_entries + + async def fake_get_index(self, session, connection_id): # noqa: ANN001 + return [] + + @asynccontextmanager + async def fake_session_factory(): + yield MagicMock() + + monkeypatch.setattr( + "app.services.code_db_sync_service.CodeDbSyncService.get_sync", + fake_get_sync, + ) + monkeypatch.setattr( + "app.services.db_index_service.DbIndexService.get_index", + fake_get_index, + ) + monkeypatch.setattr("app.models.base.async_session_factory", fake_session_factory) + monkeypatch.setattr("app.config.settings.sync_min_confidence_to_enforce_filters", 2) + + +# --------------------------------------------------------------------------- +# H4: low-confidence entries must NOT be enforced +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_low_confidence_filters_not_enforced(monkeypatch): + """Confidence 1 < threshold 2 → table absent from required-filter dict.""" + agent = _make_agent() + cfg = _make_conn_cfg() + _patch_db(monkeypatch, [_make_sync_entry("orders", {"status": "= 1"}, confidence_score=1)]) + + result = await agent._load_required_filters_by_table(cfg) + + assert "orders" not in result, ( + f"Low-confidence entry (score=1 < threshold=2) must not appear in required filters, " + f"got: {result}" + ) + + +@pytest.mark.asyncio +async def test_sufficient_confidence_filters_are_enforced(monkeypatch): + """Confidence 3 >= threshold 2 → table IS included in required-filter dict.""" + agent = _make_agent() + cfg = _make_conn_cfg() + _patch_db(monkeypatch, [_make_sync_entry("orders", {"status": "= 1"}, confidence_score=3)]) + + result = await agent._load_required_filters_by_table(cfg) + + assert "orders" in result, ( + f"High-confidence entry (score=3 >= 2) must be enforced, got: {result}" + ) + + +@pytest.mark.asyncio +async def test_threshold_boundary_exactly_equal_passes(monkeypatch): + """Confidence exactly equal to threshold (2 == 2) → entry IS enforced.""" + agent = _make_agent() + cfg = _make_conn_cfg() + _patch_db(monkeypatch, [_make_sync_entry("orders", {"status": "= 1"}, confidence_score=2)]) + + result = await agent._load_required_filters_by_table(cfg) + + assert "orders" in result, ( + f"Entry with confidence == threshold (2) should be enforced, got: {result}" + ) + + +# --------------------------------------------------------------------------- +# C6: schema-qualified table names must also be indexed under bare suffix +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_qualified_table_indexed_under_bare_suffix(monkeypatch): + """'analytics.orders' with sufficient confidence → appears under BOTH + 'analytics.orders' AND bare 'orders' in the returned dict.""" + agent = _make_agent() + cfg = _make_conn_cfg() + _patch_db( + monkeypatch, + [_make_sync_entry("analytics.orders", {"tenant_id": "= 42"}, confidence_score=5)], + ) + + result = await agent._load_required_filters_by_table(cfg) + + assert "analytics.orders" in result, ( + f"Qualified key 'analytics.orders' must be in result, got keys: {list(result.keys())}" + ) + assert "orders" in result, ( + f"Bare suffix key 'orders' must also be in result, got keys: {list(result.keys())}" + ) + # Both keys should carry the same filter condition + assert result["analytics.orders"] == result["orders"], ( + f"Both keys must have identical filter sets: " + f"qualified={result['analytics.orders']}, bare={result['orders']}" + ) + + +@pytest.mark.asyncio +async def test_qualified_low_confidence_not_indexed_at_all(monkeypatch): + """Qualified table with low confidence → neither qualified NOR bare key appears.""" + agent = _make_agent() + cfg = _make_conn_cfg() + _patch_db( + monkeypatch, + [_make_sync_entry("analytics.orders", {"tenant_id": "= 42"}, confidence_score=1)], + ) + + result = await agent._load_required_filters_by_table(cfg) + + assert "analytics.orders" not in result + assert "orders" not in result + + +@pytest.mark.asyncio +async def test_unqualified_table_no_extra_bare_key(monkeypatch): + """A plain 'orders' entry (no dot) is indexed only under 'orders', not duplicated.""" + agent = _make_agent() + cfg = _make_conn_cfg() + _patch_db(monkeypatch, [_make_sync_entry("orders", {"status": "= 1"}, confidence_score=4)]) + + result = await agent._load_required_filters_by_table(cfg) + + assert "orders" in result + # Only one key that contains 'orders' — no spurious schema-prefix duplicate + assert len([k for k in result.keys() if "orders" in k]) == 1 From c3ddf44e14e9264ecdec37eb23e9eaf815d8c861 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 15:56:38 +0200 Subject: [PATCH 12/24] fix(sync): is_indexed status whitelist, get_index_age None-guard, schema-qualified db_index, scrub db-index LLM egress (H7,M9,M2,H6) Co-Authored-By: Claude Sonnet 4.6 --- ...9126_sync_remediation_schema_qualified_.py | 34 ++++++ backend/app/knowledge/db_index_pipeline.py | 15 ++- backend/app/knowledge/db_index_validator.py | 18 +++- backend/app/models/db_index.py | 3 +- backend/app/services/db_index_service.py | 40 ++++--- .../unit/services/test_db_index_service.py | 102 ++++++++++++++++++ 6 files changed, 191 insertions(+), 21 deletions(-) create mode 100644 backend/alembic/versions/2317bf9d9126_sync_remediation_schema_qualified_.py create mode 100644 backend/tests/unit/services/test_db_index_service.py diff --git a/backend/alembic/versions/2317bf9d9126_sync_remediation_schema_qualified_.py b/backend/alembic/versions/2317bf9d9126_sync_remediation_schema_qualified_.py new file mode 100644 index 0000000..818affb --- /dev/null +++ b/backend/alembic/versions/2317bf9d9126_sync_remediation_schema_qualified_.py @@ -0,0 +1,34 @@ +"""sync_remediation_schema_qualified_uniqueness + +Revision ID: 2317bf9d9126 +Revises: a7c8d9e0f1a2 +Create Date: 2026-06-26 15:52:54.795013 +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +revision: str = '2317bf9d9126' +down_revision: Union[str, None] = 'a7c8d9e0f1a2' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + with op.batch_alter_table("db_index") as b: + b.drop_constraint("uq_db_index_conn_table", type_="unique") + b.create_unique_constraint( + "uq_db_index_conn_schema_table", + ["connection_id", "table_schema", "table_name"], + ) + + +def downgrade() -> None: + with op.batch_alter_table("db_index") as b: + b.drop_constraint("uq_db_index_conn_schema_table", type_="unique") + b.create_unique_constraint( + "uq_db_index_conn_table", + ["connection_id", "table_name"], + ) diff --git a/backend/app/knowledge/db_index_pipeline.py b/backend/app/knowledge/db_index_pipeline.py index 92cb7a2..9b98317 100644 --- a/backend/app/knowledge/db_index_pipeline.py +++ b/backend/app/knowledge/db_index_pipeline.py @@ -563,15 +563,19 @@ async def _fetch_table_samples( _llm_sem = asyncio.Semaphore(3) _large_done = [0] + _send_samples = getattr(connection_config, "send_sample_data_to_llm", True) + scrub = settings.sync_pii_scrubbing_enabled and _send_samples + async def _analyze_large_table(table: TableInfo) -> TableAnalysis: async with _llm_sem: sample_result, _ = samples.get(table.name, (QueryResult(), None)) table_code_ctx = self._filter_code_context(code_context, table.name) result = await self._validator.analyze_table( table=table, - sample_data=sample_result, + sample_data=sample_result if _send_samples else None, code_context=table_code_ctx, rules_context=rules_context, + scrub=scrub, preferred_provider=preferred_provider, model=model, ) @@ -617,10 +621,15 @@ async def _analyze_large_table(table: TableInfo) -> TableAnalysis: batch_specs.append((batch, batch_items, batch_code_ctx)) async def _run_batch(items, ctx_text): + # When send_sample_data_to_llm is False, strip samples so + # no tenant data reaches the LLM provider. + if not _send_samples: + items = [(tbl, None) for tbl, _ in items] return await self._validator.analyze_table_batch( tables=items, code_context=ctx_text, rules_context=rules_context, + scrub=scrub, preferred_provider=preferred_provider, model=model, ) @@ -649,9 +658,9 @@ async def _run_batch(items, ctx_text): "Persisting index to database", ): async with async_session_factory() as session: - current_table_names = {t.name for t in schema.tables} + current_keys = {f"{(t.schema or 'public')}.{t.name}" for t in schema.tables} deleted = await self._svc.delete_stale_tables( - session, connection_id, current_table_names + session, connection_id, current_keys ) if deleted: logger.info("Removed %d stale table index entries", deleted) diff --git a/backend/app/knowledge/db_index_validator.py b/backend/app/knowledge/db_index_validator.py index 149f290..87fe004 100644 --- a/backend/app/knowledge/db_index_validator.py +++ b/backend/app/knowledge/db_index_validator.py @@ -156,10 +156,13 @@ async def analyze_table( code_context: str, rules_context: str, *, + scrub: bool = True, preferred_provider: str | None = None, model: str | None = None, ) -> TableAnalysis: - prompt = self._build_table_prompt(table, sample_data, code_context, rules_context) + prompt = self._build_table_prompt( + table, sample_data, code_context, rules_context, scrub=scrub + ) messages = [ Message(role="system", content=self._system_prompt()), @@ -212,6 +215,7 @@ async def analyze_table_batch( code_context: str, rules_context: str, *, + scrub: bool = True, preferred_provider: str | None = None, model: str | None = None, ) -> list[TableAnalysis]: @@ -229,6 +233,7 @@ async def analyze_table_batch( sample, code_context, rules_context, + scrub=scrub, ) ) prompt_parts.append("---\n") @@ -395,6 +400,8 @@ def _build_table_prompt( sample_data: QueryResult | None, code_context: str, rules_context: str, + *, + scrub: bool = True, ) -> str: parts: list[str] = [f"## Table: {table.name}"] @@ -425,10 +432,15 @@ def _build_table_prompt( parts.append(f" - {u}{idx.name}({', '.join(idx.columns)})") if sample_data and sample_data.rows: - parts.append(f"\nSample data ({len(sample_data.rows)} newest rows):") + from app.knowledge import pii_scrubber + + rows = pii_scrubber.scrub_row_cells( + sample_data.columns, sample_data.rows, enabled=scrub + ) + parts.append(f"\nSample data ({len(rows)} newest rows):") parts.append("| " + " | ".join(sample_data.columns) + " |") parts.append("| " + " | ".join(["---"] * len(sample_data.columns)) + " |") - for row in sample_data.rows: + for row in rows: vals = [str(v)[:60] for v in row] parts.append("| " + " | ".join(vals) + " |") elif sample_data and not sample_data.rows: diff --git a/backend/app/models/db_index.py b/backend/app/models/db_index.py index 21b0e57..9b2d52f 100644 --- a/backend/app/models/db_index.py +++ b/backend/app/models/db_index.py @@ -14,8 +14,9 @@ class DbIndex(Base): __table_args__ = ( UniqueConstraint( "connection_id", + "table_schema", "table_name", - name="uq_db_index_conn_table", + name="uq_db_index_conn_schema_table", ), ) diff --git a/backend/app/services/db_index_service.py b/backend/app/services/db_index_service.py index 805e5b8..434ea48 100644 --- a/backend/app/services/db_index_service.py +++ b/backend/app/services/db_index_service.py @@ -29,9 +29,11 @@ async def upsert_table( table_data: dict, ) -> DbIndex: table_name = table_data["table_name"] + table_schema = table_data.get("table_schema", "public") result = await session.execute( select(DbIndex).where( DbIndex.connection_id == connection_id, + DbIndex.table_schema == table_schema, DbIndex.table_name == table_name, ) ) @@ -80,23 +82,31 @@ async def delete_stale_tables( self, session: AsyncSession, connection_id: str, - current_table_names: set[str], + current_keys: set[str], ) -> int: - """Remove DbIndex rows not in *current_table_names* (T17). + """Remove DbIndex rows whose schema-qualified key is not in *current_keys*. - Uses a single parameterised DELETE + ``notin_`` filter instead of the - previous fetch-then-loop pattern so we don't issue N+1 queries. + Keys are ``f"{schema}.{table_name}"`` strings (e.g. ``"public.users"``). + Uses a fetch-ids-then-delete pattern (2 queries) for portability across + SQLite and PostgreSQL (SQLite's ``DELETE … WHERE id IN (…)`` is fully + supported; a NOT-IN on composite columns is not). Returns the number of rows deleted. """ - stmt = delete(DbIndex).where(DbIndex.connection_id == connection_id) - if current_table_names: - stmt = stmt.where(DbIndex.table_name.notin_(current_table_names)) - - result = await session.execute(stmt) - deleted = int(result.rowcount or 0) # type: ignore[attr-defined] - if deleted: - await session.flush() - return deleted + rows = ( + await session.execute( + select(DbIndex.id, DbIndex.table_schema, DbIndex.table_name).where( + DbIndex.connection_id == connection_id + ) + ) + ).all() + stale_ids = [ + rid for rid, sch, nm in rows if f"{(sch or 'public')}.{nm}" not in current_keys + ] + if not stale_ids: + return 0 + await session.execute(delete(DbIndex).where(DbIndex.id.in_(stale_ids))) + await session.flush() + return len(stale_ids) async def delete_all( self, @@ -170,7 +180,7 @@ async def is_indexed( if not summary: return False status = getattr(summary, "indexing_status", "idle") or "idle" - if status == "running": + if status not in ("completed", "completed_partial"): return False return summary.indexed_at is not None @@ -183,6 +193,8 @@ async def get_index_age( if not summary: return None indexed_at = summary.indexed_at + if indexed_at is None: + return None if indexed_at.tzinfo is None: indexed_at = indexed_at.replace(tzinfo=UTC) return datetime.now(UTC) - indexed_at diff --git a/backend/tests/unit/services/test_db_index_service.py b/backend/tests/unit/services/test_db_index_service.py new file mode 100644 index 0000000..35b4cd4 --- /dev/null +++ b/backend/tests/unit/services/test_db_index_service.py @@ -0,0 +1,102 @@ +"""Unit tests for DbIndexService — is_indexed whitelist, get_index_age None-guard.""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +import pytest_asyncio +from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine + +import app.models # noqa: F401 — registers all mapped classes +from app.models.base import Base +from app.services.db_index_service import DbIndexService + +# --------------------------------------------------------------------------- +# Session fixture +# --------------------------------------------------------------------------- + + +@pytest_asyncio.fixture +async def db_session() -> AsyncSession: + engine = create_async_engine("sqlite+aiosqlite:///:memory:") + async with engine.begin() as conn: + await conn.run_sync(Base.metadata.create_all) + sm = async_sessionmaker(engine, expire_on_commit=False) + s = sm() + try: + yield s + finally: + await s.close() + await engine.dispose() + + +# --------------------------------------------------------------------------- +# Helper fixture — stub with indexed_at = NULL +# --------------------------------------------------------------------------- + + +@pytest.fixture +def summary_with_null_indexed_at(): + """Return a stub with indexed_at=None to simulate a NULL db row. + + The model column is NOT NULL so we cannot persist this state via SQLite. + We simulate the production edge-case (row inserted before the default was + added, or via direct DB manipulation) using SimpleNamespace so that + get_index_age's None-guard is exercised without touching the DB. + """ + return SimpleNamespace(indexed_at=None, connection_id="c3", indexing_status="completed") + + +# --------------------------------------------------------------------------- +# H7 — is_indexed status whitelist +# --------------------------------------------------------------------------- + + +async def test_is_indexed_false_for_failed_only(db_session: AsyncSession): + """A connection whose last status is 'failed' must NOT appear indexed (H7).""" + svc = DbIndexService() + await svc.set_indexing_status(db_session, "c1", "running") + await svc.set_indexing_status(db_session, "c1", "failed") + await db_session.commit() + assert await svc.is_indexed(db_session, "c1") is False + + +async def test_is_indexed_true_for_completed_partial(db_session: AsyncSession): + """A connection whose status is 'completed_partial' IS indexed (H7).""" + svc = DbIndexService() + await svc.set_indexing_status(db_session, "c2", "completed_partial") + await db_session.commit() + # set_indexing_status does not set indexed_at; upsert_summary does. + # Manually set indexed_at so is_indexed can return True. + summary = await svc.get_summary(db_session, "c2") + assert summary is not None + from datetime import UTC, datetime + + summary.indexed_at = datetime.now(UTC) + await db_session.flush() + await db_session.commit() + assert await svc.is_indexed(db_session, "c2") is True + + +# --------------------------------------------------------------------------- +# M9 — get_index_age None-guard +# --------------------------------------------------------------------------- + + +async def test_get_index_age_none_when_indexed_at_null( + db_session: AsyncSession, + summary_with_null_indexed_at: SimpleNamespace, +): + """get_index_age must return None (not raise AttributeError) when indexed_at is NULL (M9). + + The model column is NOT NULL so we cannot persist this state via SQLite. + We simulate it by patching get_summary to return an object whose indexed_at is None — + replicating the production invariant that get_index_age must handle gracefully. + """ + from unittest.mock import AsyncMock, patch + + svc = DbIndexService() + with patch.object(svc, "get_summary", new=AsyncMock(return_value=summary_with_null_indexed_at)): + result = await svc.get_index_age(db_session, "c3") + assert result is None # no AttributeError on None.tzinfo From 6fbf7eb3541f5173514fe281904acefecc0f6e8f Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 16:01:32 +0200 Subject: [PATCH 13/24] fix(sync): route investigation hints to query_recommendations not required_filters (M6) --- backend/app/api/routes/data_investigations.py | 4 +- .../api/test_data_investigations_enrich.py | 52 +++++++++++++++++++ 2 files changed, 54 insertions(+), 2 deletions(-) create mode 100644 backend/tests/integration/api/test_data_investigations_enrich.py diff --git a/backend/app/api/routes/data_investigations.py b/backend/app/api/routes/data_investigations.py index 66dc262..8d48d4a 100644 --- a/backend/app/api/routes/data_investigations.py +++ b/backend/app/api/routes/data_investigations.py @@ -334,8 +334,8 @@ async def _enrich_sync_from_investigation( db, connection_id=inv.connection_id, table_name=table, - field="required_filters_json", - value=json.dumps({"source": "investigation", "filter": inv.root_cause}), + field="query_recommendations", + value=f"[from investigation] {inv.root_cause}", ) elif inv.root_cause_category == "column_format": await sync_svc.add_runtime_enrichment( diff --git a/backend/tests/integration/api/test_data_investigations_enrich.py b/backend/tests/integration/api/test_data_investigations_enrich.py new file mode 100644 index 0000000..b0eeab7 --- /dev/null +++ b/backend/tests/integration/api/test_data_investigations_enrich.py @@ -0,0 +1,52 @@ +"""Tests for investigation enrichment (task T9).""" + +import pytest +from sqlalchemy.ext.asyncio import AsyncSession + +from app.api.routes.data_investigations import _enrich_sync_from_investigation +from app.models.data_validation import DataInvestigation + + +@pytest.mark.asyncio +async def test_missing_filter_routes_to_recommendations(monkeypatch, db_session: AsyncSession): + """Assert missing_filter routes to query_recommendations, not required_filters_json.""" + captured = {} + + async def _fake_enrichment(self, db, *, connection_id, table_name, field, value): + """Capture the field and value arguments.""" + captured["field"] = field + captured["value"] = value + + monkeypatch.setattr( + "app.services.code_db_sync_service.CodeDbSyncService.add_runtime_enrichment", + _fake_enrichment, + ) + + # Build a mock investigation with root_cause_category="missing_filter" + inv = DataInvestigation( + id="test-inv-001", + connection_id="test-conn-001", + session_id="test-session-001", + trigger_message_id="test-msg-001", + original_query="SELECT * FROM users WHERE status = 'active'", + original_result_summary='{"count": 100}', + user_complaint_type="missing_data", + user_complaint_detail="Some records missing", + user_expected_value="150 records", + problematic_column=None, + corrected_query=None, + corrected_result_json=None, + root_cause="Missing filter: WHERE region_id = 5", + root_cause_category="missing_filter", + status="completed", + phase="investigating", + investigation_log_json="[]", + ) + + # Call the enrichment function + await _enrich_sync_from_investigation(db_session, inv) + + # Assert the field is query_recommendations, not required_filters_json + assert captured.get("field") == "query_recommendations" + assert "[from investigation]" in captured.get("value", "") + assert "Missing filter: WHERE region_id = 5" in captured.get("value", "") From 6315ad6c1411aaa9e2ecc9746d50aa2efaaca273 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 16:08:03 +0200 Subject: [PATCH 14/24] fix(sync): translate single-active IntegrityError to 409 + rollback; model index parity (H8) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - run_coordinator.start(): catch IntegrityError on commit, rollback session, re-query for the race-winner, raise RunAlreadyActiveError — prevents raw 500 on TOCTOU concurrent starts (R5) - indexing_run.py __table_args__: add uq_indexing_runs_active_one partial-unique Index (coalesce form, verified compiles + enforces in SQLite) matching prod migration a1f2b3c4d5e6 — unit tests now enforce single-active via real DB constraint, not just app-level pre-check - migration f37386df158c: idempotent create_index (if_not_exists) / drop_index (if_exists) parity for envs missing the original hotfix migration - test: H8 TOCTOU test confirms IntegrityError→RunAlreadyActiveError translation and session usability after rollback; TDD RED→GREEN cycle documented Co-Authored-By: Claude Sonnet 4.6 --- ...c_sync_remediation_indexing_run_active_.py | 44 ++++++++++++++++ backend/app/models/indexing_run.py | 16 ++++++ backend/app/services/run_coordinator.py | 12 ++++- .../unit/services/test_run_coordinator.py | 51 +++++++++++++++++++ 4 files changed, 122 insertions(+), 1 deletion(-) create mode 100644 backend/alembic/versions/f37386df158c_sync_remediation_indexing_run_active_.py diff --git a/backend/alembic/versions/f37386df158c_sync_remediation_indexing_run_active_.py b/backend/alembic/versions/f37386df158c_sync_remediation_indexing_run_active_.py new file mode 100644 index 0000000..117d829 --- /dev/null +++ b/backend/alembic/versions/f37386df158c_sync_remediation_indexing_run_active_.py @@ -0,0 +1,44 @@ +"""sync_remediation_indexing_run_active_index + +Idempotent parity migration: ensures `uq_indexing_runs_active_one` exists on +every environment. Production already has this index from migration +`a1f2b3c4d5e6`; environments that skipped that hotfix (or fresh dev/staging +setups) get it here. + +Revision ID: f37386df158c +Revises: 2317bf9d9126 +Create Date: 2026-06-26 16:06:16.161424 +""" + +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + +revision: str = "f37386df158c" +down_revision: Union[str, None] = "2317bf9d9126" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # Partial unique index: at most one active run per (project, kind, connection). + # coalesce() ensures NULL connection_id rows are also mutually exclusive per project+kind. + # if_not_exists=True makes this idempotent for envs that already applied a1f2b3c4d5e6. + op.create_index( + "uq_indexing_runs_active_one", + "indexing_runs", + ["project_id", "kind", sa.text("coalesce(connection_id, '')")], + unique=True, + postgresql_where=sa.text("status IN ('queued','running','cancelling')"), + sqlite_where=sa.text("status IN ('queued','running','cancelling')"), + if_not_exists=True, + ) + + +def downgrade() -> None: + op.drop_index( + "uq_indexing_runs_active_one", + table_name="indexing_runs", + if_exists=True, + ) diff --git a/backend/app/models/indexing_run.py b/backend/app/models/indexing_run.py index e7dda33..16ba9e1 100644 --- a/backend/app/models/indexing_run.py +++ b/backend/app/models/indexing_run.py @@ -21,6 +21,7 @@ String, Text, func, + text, ) from sqlalchemy.orm import Mapped, mapped_column, relationship @@ -74,6 +75,21 @@ class IndexingRun(Base): Index("ix_indexing_runs_workflow", "workflow_id", unique=True), Index("ix_indexing_runs_history", "project_id", "kind", "created_at"), Index("ix_indexing_runs_active", "project_id", "kind", "status"), + # Partial unique index: at most one active run per (project, kind, connection). + # Matches the production migration `a1f2b3c4d5e6` (uq_indexing_runs_active_one). + # Using coalesce so NULL connection_id rows are also mutually exclusive. + # Verified: SQLAlchemy create_all emits this correctly for both SQLite and + # PostgreSQL; the sqlite_where clause keeps it a partial index covering only + # active statuses so historical completed/failed rows are unconstrained. + Index( + "uq_indexing_runs_active_one", + "project_id", + "kind", + text("coalesce(connection_id, '')"), + unique=True, + sqlite_where=text("status IN ('queued','running','cancelling')"), + postgresql_where=text("status IN ('queued','running','cancelling')"), + ), ) diff --git a/backend/app/services/run_coordinator.py b/backend/app/services/run_coordinator.py index 4107fa4..8e3ddfd 100644 --- a/backend/app/services/run_coordinator.py +++ b/backend/app/services/run_coordinator.py @@ -18,6 +18,7 @@ from datetime import UTC, datetime from sqlalchemy import select +from sqlalchemy.exc import IntegrityError from sqlalchemy.ext.asyncio import AsyncSession from app.config import settings @@ -165,7 +166,16 @@ async def start( meta_json=json.dumps({"force_full": force_full}), ) db.add(run) - await db.commit() + try: + await db.commit() + except IntegrityError as exc: + # TOCTOU race: another process committed a run between our _find_active + # pre-check and this commit. Roll back (required — the session is + # poisoned after an IntegrityError and must not be used without it), + # then re-query for the winner so we can surface its run_id. + await db.rollback() + existing = await self._find_active(db, project_id, kind, connection_id) + raise RunAlreadyActiveError(existing.id if existing else "unknown") from exc await db.refresh(run) self._manifests[run.id] = manifest RunCoordinator._wf_to_run[run.workflow_id] = run.id diff --git a/backend/tests/unit/services/test_run_coordinator.py b/backend/tests/unit/services/test_run_coordinator.py index 13d240b..ede8834 100644 --- a/backend/tests/unit/services/test_run_coordinator.py +++ b/backend/tests/unit/services/test_run_coordinator.py @@ -165,3 +165,54 @@ async def test_retry_starts_new_run_with_provenance(session: AsyncSession): assert new.id != run.id assert new.status == "running" assert _json.loads(new.meta_json)["retried_from"] == run.id + + +# --- H8: IntegrityError → RunAlreadyActiveError translation (TOCTOU race) ---- + + +async def _async_none() -> None: + """Async stub that returns None — simulates _find_active pre-check missing the race winner.""" + return None + + +async def test_start_translates_integrity_error_to_already_active(session: AsyncSession): + """TOCTOU race: pre-check passes (returns None) but DB unique constraint fires on commit. + + The coordinator must: + 1. Catch IntegrityError and NOT let it propagate as a raw 500. + 2. Roll back the poisoned session. + 3. Raise RunAlreadyActiveError (wrapping the original IntegrityError). + 4. Leave the session usable for subsequent queries. + """ + coord = RunCoordinator() + + # First run holds the slot — committed normally. + run1 = await coord.start(session, kind="code_db_sync", project_id="p10", connection_id="c1") + assert run1.status == "running" + + # Monkeypatch _find_active to return None on the PRE-CHECK so the race + # path reaches db.commit() and the DB unique index is what fires. + # A second call (the recovery lookup inside the except block) must be real, + # so we count calls and only skip the first one. + real_find_active = coord._find_active + call_count = 0 + + async def _find_active_stub(db, project_id, kind, connection_id): + nonlocal call_count + call_count += 1 + if call_count == 1: + return None # simulate pre-check miss + return await real_find_active(db, project_id, kind, connection_id) + + coord._find_active = _find_active_stub # type: ignore[method-assign] + + with pytest.raises(RunAlreadyActiveError): + await coord.start(session, kind="code_db_sync", project_id="p10", connection_id="c1") + + # Restore + coord._find_active = real_find_active # type: ignore[method-assign] + + # Session must be usable after rollback — a simple query should not raise. + recovered = await coord._find_active(session, "p10", "code_db_sync", "c1") + assert recovered is not None + assert recovered.id == run1.id From 83da91d71a9a7908f8c01ffd8236d33e142afebb Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 16:18:52 +0200 Subject: [PATCH 15/24] fix(sync): parent-run heartbeat, adopt-not-run, progress steps, budget skip, overview regen (H1,H9,M3,H5,M5) Co-Authored-By: Claude Opus 4.8 (1M context) --- backend/app/knowledge/run_manifests.py | 2 +- .../services/daily_knowledge_sync_service.py | 132 +++++++- .../tests/unit/test_daily_knowledge_sync.py | 296 +++++++++++++++++- 3 files changed, 415 insertions(+), 15 deletions(-) diff --git a/backend/app/knowledge/run_manifests.py b/backend/app/knowledge/run_manifests.py index 67ced7a..7d7e9d7 100644 --- a/backend/app/knowledge/run_manifests.py +++ b/backend/app/knowledge/run_manifests.py @@ -47,9 +47,9 @@ class Step: ], "daily_sync": [ Step("plan_targets", "Plan Targets"), + Step("repo_index", "Repository Index", weight=2), Step("db_index", "Database Index", weight=3), Step("code_db_sync", "Code-DB Sync", weight=3), - Step("freshness_reconcile", "Freshness Reconcile"), Step("summarize", "Summarize"), ], } diff --git a/backend/app/services/daily_knowledge_sync_service.py b/backend/app/services/daily_knowledge_sync_service.py index 9124145..d96ea93 100644 --- a/backend/app/services/daily_knowledge_sync_service.py +++ b/backend/app/services/daily_knowledge_sync_service.py @@ -6,12 +6,15 @@ import logging import time from dataclasses import dataclass, field -from datetime import datetime, timedelta +from datetime import UTC, datetime, timedelta from zoneinfo import ZoneInfo +from sqlalchemy import update from sqlalchemy.ext.asyncio import AsyncSession from app.config import settings +from app.core.heartbeat import heartbeat +from app.core.workflow_tracker import tracker from app.models.base import async_session_factory from app.models.connection import Connection from app.models.indexing_run import IndexingRun @@ -76,6 +79,7 @@ def __init__(self) -> None: self._project_svc = ProjectService() self._conn_svc = ConnectionService() self._checkpoint_svc = CheckpointService() + self._tracker = tracker async def list_eligible_projects(self, session: AsyncSession) -> list[Project]: from app.services.sync_schedule_service import SyncScheduleService @@ -121,7 +125,20 @@ async def run_for_project( ).id ) - result = await self._orchestrate(project_id) + async def _hb() -> None: + # H1/C4-1: targeted UPDATE (not a full-row ORM load) so the parent + # heartbeat never races the M3 _on_event projection on + # ``IndexingRun.version``. Only refreshes a still-running parent. + async with async_session_factory() as s: + await s.execute( + update(IndexingRun) + .where(IndexingRun.id == run_id, IndexingRun.status == "running") + .values(heartbeat_at=datetime.now(UTC)) + ) + await s.commit() + + async with heartbeat(_hb, interval_seconds=settings.heartbeat_interval_seconds): + result = await self._orchestrate(project_id, run_id=run_id) terminal = "failed" if result.status == _STATUS_FAILED else "completed" failure_kind = "fatal" if terminal == "failed" else None @@ -137,10 +154,22 @@ async def run_for_project( ) return result - async def _orchestrate(self, project_id: str) -> KnowledgeSyncRunResult: + async def _orchestrate( + self, project_id: str, *, run_id: str | None = None + ) -> KnowledgeSyncRunResult: started = time.monotonic() result = KnowledgeSyncRunResult(project_id=project_id) + # M3/C4-2: resolve the PARENT daily_sync workflow_id once. Progress is + # emitted on the parent wf with run_id=None so RunCoordinator._on_event + # does NOT skip it (its guard is ``if event.run_id is not None: return``) + # and the projection advances the parent's progress_pct monotonically. + parent_wf: str | None = None + if run_id is not None: + async with async_session_factory() as s: + parent = await s.get(IndexingRun, run_id) + parent_wf = parent.workflow_id if parent else None + async with async_session_factory() as session: project = await self._project_svc.get(session, project_id) if not project: @@ -171,15 +200,23 @@ async def _orchestrate(self, project_id: str) -> KnowledgeSyncRunResult: len(active_connections), ) + # M3: eligibility checks passed -> first manifest step is done. + await self._emit_progress(parent_wf, "plan_targets", "started", "Eligibility OK") + await self._emit_progress( + parent_wf, "plan_targets", "completed", f"{len(active_connections)} connection(s)" + ) + steps: dict = { "repo_index": {"status": _STEP_SKIPPED, "error": None}, "connections": [], } + await self._emit_progress(parent_wf, "repo_index", "started", "Indexing repository") repo_status, repo_error = await self._run_repo_index(project_id) steps["repo_index"] = {"status": repo_status, "error": repo_error} if repo_status != _STEP_COMPLETED: + await self._emit_progress(parent_wf, "repo_index", "failed", repo_error or repo_status) result.status = _STATUS_FAILED if repo_status == _STEP_FAILED else _STATUS_PARTIAL result.steps_json = steps result.error_message = repo_error @@ -191,6 +228,9 @@ async def _orchestrate(self, project_id: str) -> KnowledgeSyncRunResult: ) return result + await self._emit_progress(parent_wf, "repo_index", "completed", "Repository indexed") + + await self._emit_progress(parent_wf, "db_index", "started", "Indexing connections") any_failure = False any_skip = False for conn in active_connections: @@ -241,11 +281,21 @@ async def _orchestrate(self, project_id: str) -> KnowledgeSyncRunResult: any_skip = True steps["connections"].append(conn_steps) + # M3: per-connection db_index/code_db_sync interleave; surface them at the + # parent level as coarse phase brackets so progress advances monotonically + # (child runs carry the per-connection detail). + await self._emit_progress(parent_wf, "db_index", "completed", "Connections indexed") + await self._emit_progress(parent_wf, "code_db_sync", "started", "Cross-referencing") + await self._emit_progress(parent_wf, "code_db_sync", "completed", "Code-DB synced") + if any_failure or any_skip: result.status = _STATUS_PARTIAL else: result.status = _STATUS_SUCCESS + await self._emit_progress(parent_wf, "summarize", "started", "Finalizing") + await self._emit_progress(parent_wf, "summarize", "completed", f"status={result.status}") + result.steps_json = steps result.duration_seconds = time.monotonic() - started logger.info( @@ -256,6 +306,22 @@ async def _orchestrate(self, project_id: str) -> KnowledgeSyncRunResult: ) return result + async def _emit_progress( + self, parent_wf: str | None, step_key: str, status: str, detail: str = "" + ) -> None: + """Emit a coarse daily-sync progress event on the PARENT workflow. + + The emit carries ``run_id=None`` so ``RunCoordinator._on_event`` projects it + onto the parent ``daily_sync`` IndexingRun (advancing ``progress_pct``). + Best-effort: a tracker failure must never break the sync. + """ + if not parent_wf: + return + try: + await self._tracker.emit(parent_wf, step_key, status, detail) + except Exception: + logger.debug("daily sync progress emit failed step=%s", step_key, exc_info=True) + async def _active_connections( self, session: AsyncSession, @@ -274,7 +340,10 @@ async def _run_repo_index(self, project_id: str) -> tuple[str, str | None]: if cp and cp.status == "running": return _STEP_SKIPPED, "repo index already running" - child_wf = await self._start_child_wf("index_repo", None, project_id) + child_wf, already_active = await self._start_child_wf("index_repo", None, project_id) + if already_active: + # H9: adopt-not-run — a repo index is already tracked elsewhere. + return _STEP_SKIPPED, "already running (adopted)" try: await run_repo_index_task( project_id, force_full=False, chain_sync=False, wf_id=child_wf @@ -299,11 +368,15 @@ async def _repo_index_outcome(self, project_id: str) -> tuple[str, str | None]: async def _start_child_wf( self, kind: str, connection_id: str | None, project_id: str - ) -> str | None: - """Create a child IndexingRun for a daily-sync sub-operation and return its - workflow id. The pipeline's emitted events are mapped onto the run (and - finalised) by the RunCoordinator persistence hook. Returns ``None`` when a - run is already active (the pipeline then begins its own untracked workflow). + ) -> tuple[str | None, bool]: + """Create a child IndexingRun for a daily-sync sub-operation. + + Returns ``(workflow_id, already_active)``: + - ``(wf_id, False)``: a fresh child run was minted; the pipeline's emitted + events are projected onto it by the RunCoordinator persistence hook. + - ``(None, True)``: a run is ALREADY active for this (project, kind, + connection). The caller MUST short-circuit (adopt-not-run, H9) so we + never launch a second, untracked concurrent pipeline. """ from app.services.run_coordinator import RunAlreadyActiveError, RunCoordinator @@ -316,9 +389,9 @@ async def _start_child_wf( connection_id=connection_id, trigger="schedule", ) - return run.workflow_id + return run.workflow_id, False except RunAlreadyActiveError: - return None + return None, True async def _run_db_index( self, @@ -337,7 +410,18 @@ async def _run_db_index( return _STEP_FAILED, "connection not found" config = await self._conn_svc.to_config(session, conn) - child_wf = await self._start_child_wf("db_index", connection_id, project_id) + # H5: owner-attributed budget gate (symmetry with code_db_sync). + from app.services.sync_budget import preflight_owner_budget + + async with async_session_factory() as session: + ok, reason, _owner = await preflight_owner_budget(session, project_id) + if not ok: + return _STEP_SKIPPED, f"owner budget: {reason}" + + child_wf, already_active = await self._start_child_wf("db_index", connection_id, project_id) + if already_active: + # H9: adopt-not-run — a db index is already tracked elsewhere. + return _STEP_SKIPPED, "already running (adopted)" final_status = _STEP_FAILED error: str | None = None pipeline_result: dict | str | None = None @@ -417,7 +501,21 @@ async def _run_code_db_sync( if not await idx_svc.is_indexed(session, connection_id): return _STEP_SKIPPED, "connection not DB-indexed" - child_wf = await self._start_child_wf("code_db_sync", connection_id, project_id) + # H5: owner-attributed budget gate (the sync LLM pipeline must not bypass + # the owner's token budget — graceful skip, never a crash). + from app.services.sync_budget import preflight_owner_budget + + async with async_session_factory() as session: + ok, reason, _owner = await preflight_owner_budget(session, project_id) + if not ok: + return _STEP_SKIPPED, f"owner budget: {reason}" + + child_wf, already_active = await self._start_child_wf( + "code_db_sync", connection_id, project_id + ) + if already_active: + # H9: adopt-not-run — a sync is already tracked elsewhere. + return _STEP_SKIPPED, "already running (adopted)" final_status = _STEP_FAILED error: str | None = None try: @@ -453,5 +551,13 @@ async def _run_code_db_sync( logger.debug("Failed to update sync_status", exc_info=True) if final_status == _STEP_COMPLETED: + # M5: refresh the connection overview after a successful sync, matching + # the in-process/ARQ paths (best-effort — never fail the sync on this). + try: + from app.api.routes.connections import _regenerate_overview + + await _regenerate_overview(project_id, connection_id) + except Exception: + logger.debug("daily sync overview regen failed", exc_info=True) return _STEP_COMPLETED, None return _STEP_FAILED, error diff --git a/backend/tests/unit/test_daily_knowledge_sync.py b/backend/tests/unit/test_daily_knowledge_sync.py index d9563ad..367a43e 100644 --- a/backend/tests/unit/test_daily_knowledge_sync.py +++ b/backend/tests/unit/test_daily_knowledge_sync.py @@ -2,14 +2,25 @@ from __future__ import annotations -from datetime import datetime +import asyncio +import os +import tempfile +from datetime import UTC, datetime from unittest.mock import AsyncMock, MagicMock, patch from zoneinfo import ZoneInfo import pytest +from sqlalchemy import select +from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine +import app.models # noqa: F401 +from app.models.base import Base +from app.models.indexing_run import IndexingRun +from app.models.project import Project from app.services.daily_knowledge_sync_service import ( + _STATUS_SUCCESS, DailyKnowledgeSyncService, + KnowledgeSyncRunResult, compute_next_scheduled_run, ) @@ -196,3 +207,286 @@ async def test_partial_status_on_connection_failure(): conn_steps = result.steps_json["connections"][0] assert conn_steps["db_index"]["status"] == "failed" assert conn_steps["code_db_sync"]["status"] == "skipped" + + +# --------------------------------------------------------------------------- # +# T11 (R5): parent heartbeat, adopt-not-run, progress, budget skip, overview. # +# --------------------------------------------------------------------------- # + + +@pytest.fixture +async def file_db(): + """A real SQLite engine session factory for projection-aware tests.""" + fd, path = tempfile.mkstemp(suffix=".db") + os.close(fd) + engine = create_async_engine(f"sqlite+aiosqlite:///{path}") + async with engine.begin() as conn: + await conn.run_sync(Base.metadata.create_all) + sm = async_sessionmaker(engine, expire_on_commit=False) + try: + yield sm + finally: + await engine.dispose() + os.unlink(path) + + +@pytest.mark.asyncio +async def test_parent_run_heartbeat_refreshed_during_orchestrate(monkeypatch, file_db): + """H1: the PARENT daily_sync run's heartbeat_at advances during a slow orchestrate.""" + sm = file_db + async with sm() as s: + s.add(Project(id="p-hb", name="x", repo_url="https://x/y.git")) + await s.commit() + monkeypatch.setattr("app.services.daily_knowledge_sync_service.async_session_factory", sm) + # Tight heartbeat so a short sleep crosses several beats. + monkeypatch.setattr( + "app.services.daily_knowledge_sync_service.settings.heartbeat_interval_seconds", + 0, + raising=False, + ) + + svc = DailyKnowledgeSyncService() + captured: dict = {} + + async def slow_orchestrate(project_id, *, run_id): + # Capture the heartbeat at start, sleep, then read again at the end. + async with sm() as s: + run = await s.get(IndexingRun, run_id) + captured["before"] = run.heartbeat_at + await asyncio.sleep(0.15) + return KnowledgeSyncRunResult(project_id=project_id, status=_STATUS_SUCCESS) + + monkeypatch.setattr(svc, "_orchestrate", slow_orchestrate) + await svc.run_for_project("p-hb") + + async with sm() as s: + run = ( + (await s.execute(select(IndexingRun).where(IndexingRun.kind == "daily_sync"))) + .scalars() + .one() + ) + before = captured["before"] + after = run.heartbeat_at + assert before is not None and after is not None + if before.tzinfo is None: + before = before.replace(tzinfo=UTC) + if after.tzinfo is None: + after = after.replace(tzinfo=UTC) + assert after > before, "parent heartbeat_at did not advance during orchestrate" + + +@pytest.mark.asyncio +async def test_start_child_wf_returns_already_active_tuple(monkeypatch): + """H9: _start_child_wf returns (None, True) when a run is already active.""" + from app.services.run_coordinator import RunAlreadyActiveError + + svc = DailyKnowledgeSyncService() + + class _Coord: + async def start(self, *a, **k): + raise RunAlreadyActiveError("existing-run") + + monkeypatch.setattr("app.services.run_coordinator.RunCoordinator", lambda: _Coord()) + with patch("app.services.daily_knowledge_sync_service.async_session_factory") as mock_sf: + mock_sf.return_value.__aenter__ = AsyncMock(return_value=MagicMock()) + mock_sf.return_value.__aexit__ = AsyncMock(return_value=None) + wf_id, already = await svc._start_child_wf("db_index", "c1", "p1") + assert wf_id is None + assert already is True + + +@pytest.mark.asyncio +async def test_child_skips_when_already_active(monkeypatch): + """H9: a sub-step returns SKIPPED (not an untracked pipeline) when adopted.""" + svc = DailyKnowledgeSyncService() + + # Child wf already active -> (None, True). + monkeypatch.setattr(svc, "_start_child_wf", AsyncMock(return_value=(None, True))) + + # Guards must pass so we reach the adopt-check. + from app.services.code_db_sync_service import CodeDbSyncService + from app.services.db_index_service import DbIndexService + + monkeypatch.setattr(CodeDbSyncService, "get_sync_status", AsyncMock(return_value="idle")) + monkeypatch.setattr(DbIndexService, "is_indexed", AsyncMock(return_value=True)) + monkeypatch.setattr( + "app.services.sync_budget.preflight_owner_budget", + AsyncMock(return_value=(True, None, "owner")), + ) + pipeline_run = AsyncMock() + monkeypatch.setattr("app.knowledge.code_db_sync_pipeline.CodeDbSyncPipeline.run", pipeline_run) + + with patch("app.services.daily_knowledge_sync_service.async_session_factory") as mock_sf: + mock_sf.return_value.__aenter__ = AsyncMock(return_value=MagicMock()) + mock_sf.return_value.__aexit__ = AsyncMock(return_value=None) + status, detail = await svc._run_code_db_sync("c1", "p1") + + assert status == "skipped" + assert "adopt" in (detail or "").lower() + pipeline_run.assert_not_called() + + +@pytest.mark.asyncio +async def test_budget_skip_in_code_db_sync(monkeypatch): + """H5: when owner budget is exceeded, sync is SKIPPED with a budget reason.""" + svc = DailyKnowledgeSyncService() + + monkeypatch.setattr(svc, "_start_child_wf", AsyncMock(return_value=("wf", False))) + + from app.services.code_db_sync_service import CodeDbSyncService + from app.services.db_index_service import DbIndexService + + monkeypatch.setattr(CodeDbSyncService, "get_sync_status", AsyncMock(return_value="idle")) + monkeypatch.setattr(DbIndexService, "is_indexed", AsyncMock(return_value=True)) + monkeypatch.setattr( + "app.services.sync_budget.preflight_owner_budget", + AsyncMock(return_value=(False, "daily limit reached", "owner")), + ) + pipeline_run = AsyncMock() + monkeypatch.setattr("app.knowledge.code_db_sync_pipeline.CodeDbSyncPipeline.run", pipeline_run) + + with patch("app.services.daily_knowledge_sync_service.async_session_factory") as mock_sf: + mock_sf.return_value.__aenter__ = AsyncMock(return_value=MagicMock()) + mock_sf.return_value.__aexit__ = AsyncMock(return_value=None) + status, detail = await svc._run_code_db_sync("c1", "p1") + + assert status == "skipped" + assert "owner budget" in (detail or "") + assert "daily limit reached" in (detail or "") + pipeline_run.assert_not_called() + + +@pytest.mark.asyncio +async def test_overview_regen_after_successful_sync(monkeypatch): + """M5: a successful code-db sync regenerates the connection overview.""" + svc = DailyKnowledgeSyncService() + + monkeypatch.setattr(svc, "_start_child_wf", AsyncMock(return_value=("wf", False))) + + from app.services.code_db_sync_service import CodeDbSyncService + from app.services.db_index_service import DbIndexService + + monkeypatch.setattr(CodeDbSyncService, "get_sync_status", AsyncMock(return_value="idle")) + monkeypatch.setattr(CodeDbSyncService, "set_sync_status", AsyncMock()) + monkeypatch.setattr(DbIndexService, "is_indexed", AsyncMock(return_value=True)) + monkeypatch.setattr( + "app.services.sync_budget.preflight_owner_budget", + AsyncMock(return_value=(True, None, "owner")), + ) + monkeypatch.setattr( + "app.knowledge.code_db_sync_pipeline.CodeDbSyncPipeline.run", + AsyncMock(return_value={"status": "ok"}), + ) + regen = AsyncMock() + monkeypatch.setattr("app.api.routes.connections._regenerate_overview", regen) + + mock_session = MagicMock() + mock_session.commit = AsyncMock() + with patch("app.services.daily_knowledge_sync_service.async_session_factory") as mock_sf: + mock_sf.return_value.__aenter__ = AsyncMock(return_value=mock_session) + mock_sf.return_value.__aexit__ = AsyncMock(return_value=None) + status, _detail = await svc._run_code_db_sync("c1", "p1") + + assert status == "completed" + regen.assert_awaited_once_with("p1", "c1") + + +@pytest.mark.asyncio +async def test_progress_emits_match_manifest(monkeypatch, file_db): + """M3: orchestrate emits manifest-aligned progress steps on the PARENT wf.""" + from app.knowledge.run_manifests import resolve_manifest + from app.services.run_coordinator import RunCoordinator + + manifest_keys = {s.key for s in resolve_manifest("daily_sync")} + assert manifest_keys == { + "plan_targets", + "repo_index", + "db_index", + "code_db_sync", + "summarize", + } + + sm = file_db + async with sm() as s: + s.add(Project(id="p-prog", name="x", repo_url="https://x/y.git")) + await s.commit() + monkeypatch.setattr("app.services.daily_knowledge_sync_service.async_session_factory", sm) + + svc = DailyKnowledgeSyncService() + conn = MagicMock(id="c1", is_active=True, created_at=datetime(2026, 1, 1)) + monkeypatch.setattr(svc, "_active_connections", AsyncMock(return_value=[conn])) + monkeypatch.setattr(svc, "_run_repo_index", AsyncMock(return_value=("completed", None))) + monkeypatch.setattr(svc, "_run_db_index", AsyncMock(return_value=("completed", None))) + monkeypatch.setattr(svc, "_run_code_db_sync", AsyncMock(return_value=("completed", None))) + + emits: list[tuple[str, str, str]] = [] + orig_emit = svc._tracker.emit + + async def spy_emit(workflow_id, step, status, detail="", **kw): + emits.append((workflow_id, step, status)) + await orig_emit(workflow_id, step, status, detail, **kw) + + monkeypatch.setattr(svc._tracker, "emit", spy_emit) + + # Mint the parent run ourselves so we know its workflow_id. + coord = RunCoordinator() + async with sm() as s: + parent = await coord.start( + s, kind="daily_sync", project_id="p-prog", connection_id=None, trigger="schedule" + ) + parent_wf = parent.workflow_id + run_id = parent.id + + result = await svc._orchestrate("p-prog", run_id=run_id) + assert result.status == "success" + + steps_started = [step for (wf, step, st) in emits if wf == parent_wf and st == "started"] + steps_completed = [step for (wf, step, st) in emits if wf == parent_wf and st == "completed"] + for key in ("plan_targets", "repo_index", "db_index", "code_db_sync", "summarize"): + assert key in steps_started, f"missing started emit for {key}" + assert key in steps_completed, f"missing completed emit for {key}" + + +@pytest.mark.asyncio +async def test_adopted_parent_projects_and_partial_on_skip(monkeypatch, file_db): + """C4-4: an ADOPTED parent still projects progress; an adopted child -> PARTIAL.""" + from app.services.run_coordinator import RunCoordinator + + sm = file_db + async with sm() as s: + s.add(Project(id="p-adopt", name="x", repo_url="https://x/y.git")) + await s.commit() + monkeypatch.setattr("app.services.daily_knowledge_sync_service.async_session_factory", sm) + + # Ensure the projection hook is attached so emits advance the parent run. + RunCoordinator().attach() + + # Simulate sync_now having minted the daily_sync run BEFORE run_for_project. + coord = RunCoordinator() + async with sm() as s: + minted = await coord.start( + s, kind="daily_sync", project_id="p-adopt", connection_id=None, trigger="manual" + ) + minted_id = minted.id + + svc = DailyKnowledgeSyncService() + conn = MagicMock(id="c1", is_active=True, created_at=datetime(2026, 1, 1)) + monkeypatch.setattr(svc, "_active_connections", AsyncMock(return_value=[conn])) + monkeypatch.setattr(svc, "_run_repo_index", AsyncMock(return_value=("completed", None))) + monkeypatch.setattr(svc, "_run_db_index", AsyncMock(return_value=("completed", None))) + # The sub-step is adopted (already active elsewhere) -> SKIPPED. + monkeypatch.setattr( + svc, "_run_code_db_sync", AsyncMock(return_value=("skipped", "already running (adopted)")) + ) + + result = await svc.run_for_project("p-adopt", trigger="schedule") + + # (b) adopted child skip -> aggregate is PARTIAL, not SUCCESS. + assert result.status == "partial" + + # (a) the adopted parent run advanced past 0% and reached a terminal state. + async with sm() as s: + run = await s.get(IndexingRun, minted_id) + assert run is not None + assert run.status in ("completed", "failed") + assert run.progress_pct > 0 From a2b501ceec1fd0d229618f59f96dc769e8403c58 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 16:24:49 +0200 Subject: [PATCH 16/24] fix(sync): reaper logs a sweep even when driver rowcount is unknown (L1) --- backend/app/services/stale_run_reaper.py | 9 ++ backend/tests/unit/test_stale_run_reaper.py | 122 ++++++++++++++++++++ 2 files changed, 131 insertions(+) diff --git a/backend/app/services/stale_run_reaper.py b/backend/app/services/stale_run_reaper.py index 5255c1c..1078519 100644 --- a/backend/app/services/stale_run_reaper.py +++ b/backend/app/services/stale_run_reaper.py @@ -88,6 +88,10 @@ async def reap_once(self, session: AsyncSession, *, timeout_seconds: int) -> dic "repo": max(0, int(repo_res.rowcount or 0)), "runs": runs_count, } + unknown = any( + (r.rowcount is not None and r.rowcount < 0) + for r in (db_res, sync_res, repo_res, runs_failed, runs_cancelled) + ) if any(out.values()): logger.info( "Reaper: reset stale runs — db_index=%d sync=%d repo=%d runs=%d (timeout=%ds)", @@ -97,4 +101,9 @@ async def reap_once(self, session: AsyncSession, *, timeout_seconds: int) -> dic out["runs"], timeout_seconds, ) + elif unknown: + logger.info( + "Reaper: swept stale runs (rowcount unknown on this driver, timeout=%ds)", + timeout_seconds, + ) return out diff --git a/backend/tests/unit/test_stale_run_reaper.py b/backend/tests/unit/test_stale_run_reaper.py index 6c53b9a..945486e 100644 --- a/backend/tests/unit/test_stale_run_reaper.py +++ b/backend/tests/unit/test_stale_run_reaper.py @@ -7,10 +7,12 @@ import app.models.code_db_sync # noqa: F401 import app.models.db_index # noqa: F401 import app.models.indexing_checkpoint # noqa: F401 +import app.models.indexing_run # noqa: F401 from app.models.base import Base from app.models.code_db_sync import CodeDbSyncSummary from app.models.db_index import DbIndexSummary from app.models.indexing_checkpoint import IndexingCheckpoint +from app.models.indexing_run import IndexingRun from app.services.stale_run_reaper import StaleRunReaper @@ -81,6 +83,126 @@ async def test_idempotent_second_run_is_noop(db_session): assert out2["db_index"] == 0 +async def test_reaper_logs_sweep_when_rowcount_unknown(caplog, monkeypatch, db_session): + """When a driver returns -1 rowcount (unknown), an INFO sweep line is logged.""" + import logging + + from app.services.stale_run_reaper import StaleRunReaper + + caplog.set_level(logging.INFO) + + reaper = StaleRunReaper() + + # Wrap reap_once to simulate -1 rowcounts on all five execute calls + async def wrapped_reap_once(session, *, timeout_seconds: int): + from sqlalchemy import update + + cutoff = datetime.now(UTC) - timedelta(seconds=timeout_seconds) + + # Execute all five updates normally + db_res = await session.execute( + update(DbIndexSummary) + .where( + DbIndexSummary.indexing_status == "running", + reaper._stale(DbIndexSummary, cutoff), + ) + .values(indexing_status="failed") + ) + sync_res = await session.execute( + update(CodeDbSyncSummary) + .where( + CodeDbSyncSummary.sync_status == "running", + reaper._stale(CodeDbSyncSummary, cutoff), + ) + .values(sync_status="failed") + ) + repo_res = await session.execute( + update(IndexingCheckpoint) + .where( + IndexingCheckpoint.status == "running", reaper._stale(IndexingCheckpoint, cutoff) + ) + .values(status="interrupted") + ) + runs_failed = await session.execute( + update(IndexingRun) + .where(IndexingRun.status == "running", reaper._stale_run(IndexingRun, cutoff)) + .values( + status="failed", + error="stale run reaped", + failure_kind="fatal", + finished_at=datetime.now(UTC), + ) + ) + runs_cancelled = await session.execute( + update(IndexingRun) + .where(IndexingRun.status == "cancelling", reaper._stale_run(IndexingRun, cutoff)) + .values(status="cancelled", finished_at=datetime.now(UTC)) + ) + await session.flush() + + # Override all rowcounts to -1 to simulate driver returning "unknown" + db_res.rowcount = -1 + sync_res.rowcount = -1 + repo_res.rowcount = -1 + runs_failed.rowcount = -1 + runs_cancelled.rowcount = -1 + + # Now compute the output dict (all zeros due to max(0, -1)) + runs_count = max(0, int(runs_failed.rowcount or 0)) + max( + 0, int(runs_cancelled.rowcount or 0) + ) + out = { + "db_index": max(0, int(db_res.rowcount or 0)), + "sync": max(0, int(sync_res.rowcount or 0)), + "repo": max(0, int(repo_res.rowcount or 0)), + "runs": runs_count, + } + + # Detect unknown rowcount and log accordingly + unknown = any( + (r.rowcount is not None and r.rowcount < 0) + for r in (db_res, sync_res, repo_res, runs_failed, runs_cancelled) + ) + if any(out.values()): + logging.getLogger("app.services.stale_run_reaper").info( + "Reaper: reset stale runs — db_index=%d sync=%d repo=%d runs=%d (timeout=%ds)", + out["db_index"], + out["sync"], + out["repo"], + out["runs"], + timeout_seconds, + ) + elif unknown: + logging.getLogger("app.services.stale_run_reaper").info( + "Reaper: swept stale runs (rowcount unknown on this driver, timeout=%ds)", + timeout_seconds, + ) + return out + + # Create a stale record to ensure the reaper has something to update + old = datetime.now(UTC) - timedelta(seconds=600) + db_session.add(DbIndexSummary(connection_id="c1", indexing_status="running", heartbeat_at=old)) + await db_session.commit() + + # Monkeypatch reaper.reap_once to use our wrapped version that simulates -1 rowcounts + monkeypatch.setattr(reaper, "reap_once", wrapped_reap_once) + + # Call reap_once; with all rowcounts = -1, any(out.values()) is False, + # but unknown=True should trigger the "swept stale runs (rowcount unknown...)" log. + await reaper.reap_once(db_session, timeout_seconds=300) + await db_session.commit() + + # Assert that the "swept stale runs (rowcount unknown...)" log line was emitted. + logs_found = [ + r.message for r in caplog.records if r.name == "app.services.stale_run_reaper" + ] + assert any( + "swept stale runs (rowcount unknown" in record.message + for record in caplog.records + if record.name == "app.services.stale_run_reaper" + ), f"Expected 'swept stale runs (rowcount unknown...' in logs. Got: {logs_found}" + + async def _id(session, model, conn): from sqlalchemy import select From cb0f4e1cb2243b98e618da728b93f2ede0610efb Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 16:27:19 +0200 Subject: [PATCH 17/24] test(sync): exercise real reap_once for unknown-rowcount branch (T12 review) --- backend/tests/unit/test_stale_run_reaper.py | 129 ++++---------------- 1 file changed, 24 insertions(+), 105 deletions(-) diff --git a/backend/tests/unit/test_stale_run_reaper.py b/backend/tests/unit/test_stale_run_reaper.py index 945486e..b89ce1c 100644 --- a/backend/tests/unit/test_stale_run_reaper.py +++ b/backend/tests/unit/test_stale_run_reaper.py @@ -12,7 +12,6 @@ from app.models.code_db_sync import CodeDbSyncSummary from app.models.db_index import DbIndexSummary from app.models.indexing_checkpoint import IndexingCheckpoint -from app.models.indexing_run import IndexingRun from app.services.stale_run_reaper import StaleRunReaper @@ -83,119 +82,39 @@ async def test_idempotent_second_run_is_noop(db_session): assert out2["db_index"] == 0 -async def test_reaper_logs_sweep_when_rowcount_unknown(caplog, monkeypatch, db_session): - """When a driver returns -1 rowcount (unknown), an INFO sweep line is logged.""" - import logging +async def test_reaper_logs_sweep_when_rowcount_unknown(caplog): + """When a driver returns -1 rowcount (unknown), an INFO sweep line is logged. - from app.services.stale_run_reaper import StaleRunReaper + This test exercises the real reaper.reap_once() with a fake session + that forces all execute() calls to return -1 rowcount. + """ + import logging + from unittest.mock import AsyncMock, MagicMock caplog.set_level(logging.INFO) reaper = StaleRunReaper() - # Wrap reap_once to simulate -1 rowcounts on all five execute calls - async def wrapped_reap_once(session, *, timeout_seconds: int): - from sqlalchemy import update - - cutoff = datetime.now(UTC) - timedelta(seconds=timeout_seconds) - - # Execute all five updates normally - db_res = await session.execute( - update(DbIndexSummary) - .where( - DbIndexSummary.indexing_status == "running", - reaper._stale(DbIndexSummary, cutoff), - ) - .values(indexing_status="failed") - ) - sync_res = await session.execute( - update(CodeDbSyncSummary) - .where( - CodeDbSyncSummary.sync_status == "running", - reaper._stale(CodeDbSyncSummary, cutoff), - ) - .values(sync_status="failed") - ) - repo_res = await session.execute( - update(IndexingCheckpoint) - .where( - IndexingCheckpoint.status == "running", reaper._stale(IndexingCheckpoint, cutoff) - ) - .values(status="interrupted") - ) - runs_failed = await session.execute( - update(IndexingRun) - .where(IndexingRun.status == "running", reaper._stale_run(IndexingRun, cutoff)) - .values( - status="failed", - error="stale run reaped", - failure_kind="fatal", - finished_at=datetime.now(UTC), - ) - ) - runs_cancelled = await session.execute( - update(IndexingRun) - .where(IndexingRun.status == "cancelling", reaper._stale_run(IndexingRun, cutoff)) - .values(status="cancelled", finished_at=datetime.now(UTC)) - ) - await session.flush() - - # Override all rowcounts to -1 to simulate driver returning "unknown" - db_res.rowcount = -1 - sync_res.rowcount = -1 - repo_res.rowcount = -1 - runs_failed.rowcount = -1 - runs_cancelled.rowcount = -1 - - # Now compute the output dict (all zeros due to max(0, -1)) - runs_count = max(0, int(runs_failed.rowcount or 0)) + max( - 0, int(runs_cancelled.rowcount or 0) - ) - out = { - "db_index": max(0, int(db_res.rowcount or 0)), - "sync": max(0, int(sync_res.rowcount or 0)), - "repo": max(0, int(repo_res.rowcount or 0)), - "runs": runs_count, - } - - # Detect unknown rowcount and log accordingly - unknown = any( - (r.rowcount is not None and r.rowcount < 0) - for r in (db_res, sync_res, repo_res, runs_failed, runs_cancelled) - ) - if any(out.values()): - logging.getLogger("app.services.stale_run_reaper").info( - "Reaper: reset stale runs — db_index=%d sync=%d repo=%d runs=%d (timeout=%ds)", - out["db_index"], - out["sync"], - out["repo"], - out["runs"], - timeout_seconds, - ) - elif unknown: - logging.getLogger("app.services.stale_run_reaper").info( - "Reaper: swept stale runs (rowcount unknown on this driver, timeout=%ds)", - timeout_seconds, - ) - return out - - # Create a stale record to ensure the reaper has something to update - old = datetime.now(UTC) - timedelta(seconds=600) - db_session.add(DbIndexSummary(connection_id="c1", indexing_status="running", heartbeat_at=old)) - await db_session.commit() + # Create a fake result object with rowcount = -1 + class FakeResult: + rowcount = -1 - # Monkeypatch reaper.reap_once to use our wrapped version that simulates -1 rowcounts - monkeypatch.setattr(reaper, "reap_once", wrapped_reap_once) + # Create a fake session that returns the fake result on every execute(), + # and supports flush() as an async no-op. + fake_session = MagicMock() + fake_session.execute = AsyncMock(return_value=FakeResult()) + fake_session.flush = AsyncMock() - # Call reap_once; with all rowcounts = -1, any(out.values()) is False, - # but unknown=True should trigger the "swept stale runs (rowcount unknown...)" log. - await reaper.reap_once(db_session, timeout_seconds=300) - await db_session.commit() + # Call the REAL reaper.reap_once with the fake session. + # All five execute() calls will return FakeResult (rowcount=-1), + # so out will be all zeros, but unknown=True will trigger the sweep log. + out = await reaper.reap_once(fake_session, timeout_seconds=300) + + # Verify all counts are 0 (max(0, -1) = 0). + assert out == {"db_index": 0, "sync": 0, "repo": 0, "runs": 0} - # Assert that the "swept stale runs (rowcount unknown...)" log line was emitted. - logs_found = [ - r.message for r in caplog.records if r.name == "app.services.stale_run_reaper" - ] + # Assert the "swept stale runs (rowcount unknown...)" log was emitted. + logs_found = [r.message for r in caplog.records if r.name == "app.services.stale_run_reaper"] assert any( "swept stale runs (rowcount unknown" in record.message for record in caplog.records From fcee8cd07709b165bc885b804111cf0b77673a43 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 16:30:17 +0200 Subject: [PATCH 18/24] fix(sync): worker logs matched count from correct 'synced' key (M5) The run_code_db_sync worker task was reading result.get("synced_tables") but CodeDbSyncPipeline.run returns the key "synced", causing matched=None in logs. Changed to result.get("synced") to fix the log output. Added test_worker_sync.py with a test that patches the pipeline to return the correct dict and asserts the log contains matched=2. Co-Authored-By: Claude Opus 4.8 (1M context) --- backend/app/worker.py | 2 +- backend/tests/unit/test_worker_sync.py | 74 ++++++++++++++++++++++++++ 2 files changed, 75 insertions(+), 1 deletion(-) create mode 100644 backend/tests/unit/test_worker_sync.py diff --git a/backend/app/worker.py b/backend/app/worker.py index 745ac79..3764d20 100644 --- a/backend/app/worker.py +++ b/backend/app/worker.py @@ -148,7 +148,7 @@ async def run_code_db_sync( # noqa: ARG001 ) else: tables = result.get("total_tables") if isinstance(result, dict) else None - matched = result.get("synced_tables") if isinstance(result, dict) else None + matched = result.get("synced") if isinstance(result, dict) else None logger.info( "run_code_db_sync completed: connection=%s tables=%s matched=%s", connection_id[:8], diff --git a/backend/tests/unit/test_worker_sync.py b/backend/tests/unit/test_worker_sync.py new file mode 100644 index 0000000..7f143a3 --- /dev/null +++ b/backend/tests/unit/test_worker_sync.py @@ -0,0 +1,74 @@ +"""Unit tests for worker run_code_db_sync logging.""" + +from unittest.mock import AsyncMock + +import pytest + + +@pytest.mark.asyncio +async def test_run_code_db_sync_logs_matched_from_synced_key( + caplog: pytest.LogCaptureFixture, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Test that run_code_db_sync logs 'matched' from the 'synced' key (not 'synced_tables').""" + # Arrange: patch dependencies before importing worker + mock_pipeline = AsyncMock() + mock_pipeline.run.return_value = { + "status": "completed", + "total_tables": 3, + "synced": 2, + "code_only": 0, + "db_only": 1, + "mismatch": 0, + "workflow_id": "wf-123", + } + + # Create pipeline class factory + mock_pipeline_class = lambda: mock_pipeline # noqa: E731 + monkeypatch.setattr( + "app.knowledge.code_db_sync_pipeline.CodeDbSyncPipeline", + mock_pipeline_class, + ) + + # Patch the service calls (status setting) + mock_sync_svc = AsyncMock() + mock_sync_svc.set_sync_status = AsyncMock() + + def mock_service_factory(): + return mock_sync_svc + + monkeypatch.setattr( + "app.services.code_db_sync_service.CodeDbSyncService", + mock_service_factory, + ) + + # Mock async_session_factory as context manager + mock_session = AsyncMock() + mock_session.commit = AsyncMock() + + mock_factory_ctx = AsyncMock() + mock_factory_ctx.__aenter__ = AsyncMock(return_value=mock_session) + mock_factory_ctx.__aexit__ = AsyncMock(return_value=None) + + monkeypatch.setattr( + "app.models.base.async_session_factory", + lambda: mock_factory_ctx, + ) + + # Act: call the worker function + import logging + + from app.worker import run_code_db_sync + + caplog.set_level(logging.INFO) + ctx = {} + await run_code_db_sync(ctx, connection_id="conn-123", project_id="proj-456", wf_id="wf-123") + + # Assert: the log line contains matched=2 (not matched=None) + assert any("matched=2" in record.message for record in caplog.records), ( + f"Expected 'matched=2' in logs, but got: {[r.message for r in caplog.records]}" + ) + # Also verify it logs the table count + assert any("tables=3" in record.message for record in caplog.records), ( + f"Expected 'tables=3' in logs, but got: {[r.message for r in caplog.records]}" + ) From 5bc951000f75c961776f9242d4b62f3521ca0d59 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 16:39:25 +0200 Subject: [PATCH 19/24] feat(sync): per-connection send_sample_data_to_llm opt-out + ConnectionConfig propagation + trigger_sync budget 429 (H6,H5) - Connection model: add send_sample_data_to_llm bool (default True, server_default=1) - Migration e909ec65d857: add column with batch-mode downgrade for SQLite - ConnectionConfig dataclass: add send_sample_data_to_llm field (default True) - ConnectionService.to_config: propagate flag from ORM row into ConnectionConfig - ConnectionCreate/Update/Response schemas: expose flag (not a secret) - _UPDATABLE_FIELDS: include send_sample_data_to_llm so PATCH persists it - trigger_sync: preflight_owner_budget gate before start-lock (429 when over budget) - Tests: 4 unit (model default/persist/to_config propagation) + 2 integration (429 gate) Co-Authored-By: Claude Sonnet 4.6 --- ...65d857_sync_remediation_connection_flag.py | 33 +++++ backend/app/api/routes/connections.py | 11 ++ backend/app/connectors/base.py | 1 + backend/app/models/connection.py | 3 + backend/app/services/connection_service.py | 2 + .../integration/test_trigger_sync_budget.py | 82 +++++++++++ .../tests/unit/models/test_connection_flag.py | 127 ++++++++++++++++++ 7 files changed, 259 insertions(+) create mode 100644 backend/alembic/versions/e909ec65d857_sync_remediation_connection_flag.py create mode 100644 backend/tests/integration/test_trigger_sync_budget.py create mode 100644 backend/tests/unit/models/test_connection_flag.py diff --git a/backend/alembic/versions/e909ec65d857_sync_remediation_connection_flag.py b/backend/alembic/versions/e909ec65d857_sync_remediation_connection_flag.py new file mode 100644 index 0000000..d68280b --- /dev/null +++ b/backend/alembic/versions/e909ec65d857_sync_remediation_connection_flag.py @@ -0,0 +1,33 @@ +"""sync_remediation_connection_flag + +Revision ID: e909ec65d857 +Revises: f37386df158c +Create Date: 2026-06-26 16:34:53.464295 +""" + +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op + +revision: str = "e909ec65d857" +down_revision: Union[str, None] = "f37386df158c" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.add_column( + "connections", + sa.Column( + "send_sample_data_to_llm", + sa.Boolean(), + nullable=False, + server_default=sa.text("1"), + ), + ) + + +def downgrade() -> None: + with op.batch_alter_table("connections") as batch_op: + batch_op.drop_column("send_sample_data_to_llm") diff --git a/backend/app/api/routes/connections.py b/backend/app/api/routes/connections.py index 9556170..61ad36a 100644 --- a/backend/app/api/routes/connections.py +++ b/backend/app/api/routes/connections.py @@ -24,6 +24,7 @@ from app.services.connection_service import ConnectionService from app.services.db_index_service import DbIndexService from app.services.membership_service import MembershipService +from app.services.sync_budget import preflight_owner_budget logger = logging.getLogger(__name__) @@ -318,6 +319,8 @@ class ConnectionCreate(BaseModel): mcp_server_url: str | None = Field(None, max_length=1024) mcp_transport_type: Literal["stdio", "sse"] | None = None mcp_env: dict[str, str] | None = None + # H6: opt-out of sending DB sample data to the LLM (default True = send) + send_sample_data_to_llm: bool = True @field_validator("mcp_env", mode="before") @classmethod @@ -381,6 +384,8 @@ class ConnectionUpdate(BaseModel): mcp_server_url: str | None = Field(None, max_length=2000) mcp_transport_type: str | None = Field(None, max_length=50) mcp_env: dict[str, str] | None = None + # H6: opt-out of sending DB sample data to the LLM + send_sample_data_to_llm: bool | None = None @field_validator("ssh_pre_commands") @classmethod @@ -409,6 +414,7 @@ class ConnectionResponse(BaseModel): db_user: str | None is_read_only: bool is_active: bool + send_sample_data_to_llm: bool ssh_exec_mode: bool ssh_command_template: str | None ssh_pre_commands: str | None @@ -1019,6 +1025,11 @@ async def trigger_sync( raise HTTPException(status_code=404, detail="Connection not found") await _membership_svc.require_role(db, conn.project_id, user["user_id"], "editor") + # H5: budget pre-flight — block over-budget owners before we even acquire the lock. + ok, reason, _ = await preflight_owner_budget(db, conn.project_id) + if not ok: + raise HTTPException(status_code=429, detail=reason) + sync_start_lock = _sync_start_locks.setdefault(connection_id, asyncio.Lock()) async with sync_start_lock: existing = _sync_tasks.get(connection_id) diff --git a/backend/app/connectors/base.py b/backend/app/connectors/base.py index d1174f9..af1063b 100644 --- a/backend/app/connectors/base.py +++ b/backend/app/connectors/base.py @@ -25,6 +25,7 @@ class ConnectionConfig: ssh_pre_commands: list[str] | None = None is_read_only: bool = True + send_sample_data_to_llm: bool = True extra: dict[str, Any] = field(default_factory=dict) connection_id: str | None = None diff --git a/backend/app/models/connection.py b/backend/app/models/connection.py index 99e94de..ebc79a0 100644 --- a/backend/app/models/connection.py +++ b/backend/app/models/connection.py @@ -56,6 +56,9 @@ class Connection(Base): is_read_only: Mapped[bool] = mapped_column(Boolean, default=True) is_active: Mapped[bool] = mapped_column(Boolean, default=True) + send_sample_data_to_llm: Mapped[bool] = mapped_column( + Boolean, default=True, server_default="1", nullable=False + ) created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) updated_at: Mapped[datetime] = mapped_column( diff --git a/backend/app/services/connection_service.py b/backend/app/services/connection_service.py index 9084bf5..8d9702d 100644 --- a/backend/app/services/connection_service.py +++ b/backend/app/services/connection_service.py @@ -29,6 +29,7 @@ "db_name", "db_user", "is_read_only", + "send_sample_data_to_llm", "ssh_exec_mode", "ssh_command_template", "ssh_pre_commands", @@ -475,5 +476,6 @@ async def to_config( ssh_command_template=conn.ssh_command_template, ssh_pre_commands=pre_commands, is_read_only=conn.is_read_only, + send_sample_data_to_llm=getattr(conn, "send_sample_data_to_llm", True), extra=extra, ) diff --git a/backend/tests/integration/test_trigger_sync_budget.py b/backend/tests/integration/test_trigger_sync_budget.py new file mode 100644 index 0000000..319081c --- /dev/null +++ b/backend/tests/integration/test_trigger_sync_budget.py @@ -0,0 +1,82 @@ +"""Integration tests: trigger_sync returns 429 when owner is over budget (T14/H5). + +Uses monkeypatch to make preflight_owner_budget return (False, "budget exceeded", None) +and confirms the route returns 429 without starting any background work. +""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, patch + +import pytest +import pytest_asyncio +from httpx import AsyncClient + + +@pytest_asyncio.fixture() +async def indexed_connection(auth_client: AsyncClient): + """Create a project+connection and mark it as DB-indexed so trigger_sync can run.""" + proj = await auth_client.post( + "/api/projects", + json={"name": "BudgetGateTest", "description": "T14 trigger_sync budget gate"}, + ) + assert proj.status_code == 200 + project_id = proj.json()["id"] + + conn = await auth_client.post( + "/api/connections", + json={ + "project_id": project_id, + "name": "BudgetTestDB", + "db_type": "postgres", + "db_host": "127.0.0.1", + "db_port": 5432, + "db_name": "testdb", + "db_user": "user", + "db_password": "pass", + }, + ) + assert conn.status_code == 200 + connection_id = conn.json()["id"] + + return project_id, connection_id + + +@pytest.mark.asyncio +class TestTriggerSyncBudgetGate: + async def test_trigger_sync_429_when_over_budget( + self, auth_client: AsyncClient, indexed_connection + ): + """trigger_sync must return 429 when owner is over budget.""" + _, connection_id = indexed_connection + + # Patch preflight_owner_budget to simulate over-budget condition. + with patch( + "app.api.routes.connections.preflight_owner_budget", + new=AsyncMock(return_value=(False, "daily token budget exhausted", None)), + ): + resp = await auth_client.post(f"/api/connections/{connection_id}/sync") + + assert resp.status_code == 429 + detail = resp.json()["detail"] + assert "budget" in detail.lower() + + async def test_trigger_sync_not_blocked_when_budget_ok( + self, auth_client: AsyncClient, indexed_connection + ): + """trigger_sync should proceed past the budget gate when budget is ok. + + The request will still fail (400) because the connection isn't DB-indexed, + but it must NOT return 429 — confirming the budget gate let it through. + """ + _, connection_id = indexed_connection + + with patch( + "app.api.routes.connections.preflight_owner_budget", + new=AsyncMock(return_value=(True, None, "owner-uid")), + ): + resp = await auth_client.post(f"/api/connections/{connection_id}/sync") + + # 400 = not indexed (expected); what matters is it's NOT 429. + assert resp.status_code == 400 + assert resp.status_code != 429 diff --git a/backend/tests/unit/models/test_connection_flag.py b/backend/tests/unit/models/test_connection_flag.py new file mode 100644 index 0000000..12323ec --- /dev/null +++ b/backend/tests/unit/models/test_connection_flag.py @@ -0,0 +1,127 @@ +"""Unit tests for Connection.send_sample_data_to_llm flag (T14). + +Covers: + - Default True on new connections. + - Explicit False is persisted. + - ConnectionService.to_config propagates the flag into ConnectionConfig. +""" + +from __future__ import annotations + +import uuid + +import pytest +import pytest_asyncio +from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine +from sqlalchemy.orm import sessionmaker + +import app.models.chat_session # noqa: F401 +import app.models.commit_index # noqa: F401 +import app.models.connection # noqa: F401 +import app.models.custom_rule # noqa: F401 +import app.models.indexing_checkpoint # noqa: F401 +import app.models.knowledge_doc # noqa: F401 +import app.models.project # noqa: F401 +import app.models.project_cache # noqa: F401 +import app.models.project_invite # noqa: F401 +import app.models.project_member # noqa: F401 +import app.models.rag_feedback # noqa: F401 +import app.models.ssh_key # noqa: F401 +import app.models.user # noqa: F401 +from app.models.base import Base +from app.models.connection import Connection +from app.models.project import Project +from app.services.connection_service import ConnectionService + +svc = ConnectionService() + + +@pytest_asyncio.fixture +async def db(): + engine = create_async_engine("sqlite+aiosqlite:///:memory:") + async with engine.begin() as conn: + await conn.run_sync(Base.metadata.create_all) + async_session = sessionmaker(engine, class_=AsyncSession, expire_on_commit=False) + async with async_session() as session: + yield session + await engine.dispose() + + +async def _make_project(db: AsyncSession) -> Project: + p = Project(name=f"proj-{uuid.uuid4().hex[:6]}") + db.add(p) + await db.commit() + await db.refresh(p) + return p + + +class TestSendSampleDataFlag: + @pytest.mark.asyncio + async def test_default_true(self, db: AsyncSession): + """A new connection should default send_sample_data_to_llm to True.""" + proj = await _make_project(db) + conn = await svc.create( + db, + project_id=proj.id, + name="test-default", + db_type="postgres", + db_host="127.0.0.1", + db_port=5432, + db_name="mydb", + ) + assert conn.send_sample_data_to_llm is True + + @pytest.mark.asyncio + async def test_explicit_false_persisted(self, db: AsyncSession): + """Passing send_sample_data_to_llm=False should persist as False.""" + proj = await _make_project(db) + conn = await svc.create( + db, + project_id=proj.id, + name="test-optout", + db_type="postgres", + db_host="127.0.0.1", + db_port=5432, + db_name="mydb", + send_sample_data_to_llm=False, + ) + assert conn.send_sample_data_to_llm is False + + # Reload from DB to confirm persistence, not just in-memory default. + from sqlalchemy import select + + row = (await db.execute(select(Connection).where(Connection.id == conn.id))).scalar_one() + assert row.send_sample_data_to_llm is False + + @pytest.mark.asyncio + async def test_to_config_propagates_true(self, db: AsyncSession): + """to_config should set send_sample_data_to_llm=True on ConnectionConfig.""" + proj = await _make_project(db) + conn = await svc.create( + db, + project_id=proj.id, + name="test-config-true", + db_type="postgres", + db_host="127.0.0.1", + db_port=5432, + db_name="mydb", + ) + config = await svc.to_config(db, conn) + assert config.send_sample_data_to_llm is True + + @pytest.mark.asyncio + async def test_to_config_propagates_false(self, db: AsyncSession): + """to_config should set send_sample_data_to_llm=False when opt-out is stored.""" + proj = await _make_project(db) + conn = await svc.create( + db, + project_id=proj.id, + name="test-config-false", + db_type="postgres", + db_host="127.0.0.1", + db_port=5432, + db_name="mydb", + send_sample_data_to_llm=False, + ) + config = await svc.to_config(db, conn) + assert config.send_sample_data_to_llm is False From f88e5fbbbd2f37dd7a48094e36aa9434ab9e9ad4 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 16:42:47 +0200 Subject: [PATCH 20/24] fix(sync): freshness warnings default-list + sync_failed flag (M8) --- .../services/knowledge_freshness_service.py | 5 ++- .../test_knowledge_freshness_service.py | 33 +++++++++++++++++++ 2 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 backend/tests/unit/services/test_knowledge_freshness_service.py diff --git a/backend/app/services/knowledge_freshness_service.py b/backend/app/services/knowledge_freshness_service.py index c886a6d..2a96b27 100644 --- a/backend/app/services/knowledge_freshness_service.py +++ b/backend/app/services/knowledge_freshness_service.py @@ -66,6 +66,7 @@ class KnowledgeFreshness: db_index_stale: bool = False sync_status: str | None = None sync_stale: bool = False + sync_failed: bool = False git_behind_commits: int | None = None git_unindexed: bool = False # M6: code-graph signal. ``code_graph_symbol_count`` is the canonical @@ -73,7 +74,7 @@ class KnowledgeFreshness: # ``git_behind_commits`` since both come from the same indexer run. code_graph_symbol_count: int = 0 code_graph_stale: bool = False - warnings: list[str] = None # type: ignore[assignment] + warnings: list[str] = field(default_factory=list) # Structured, actionable mirror of ``warnings`` (Phase 1, additive). details: list[FreshnessWarningDetail] = field(default_factory=list) @@ -97,6 +98,7 @@ def to_dict(self) -> dict: "db_index_stale": self.db_index_stale, "sync_status": self.sync_status, "sync_stale": self.sync_stale, + "sync_failed": self.sync_failed, "git_behind_commits": self.git_behind_commits, "git_unindexed": self.git_unindexed, "code_graph_symbol_count": self.code_graph_symbol_count, @@ -174,6 +176,7 @@ def _warn( sync_svc = CodeDbSyncService() snapshot.sync_status = await sync_svc.get_sync_status(session, connection_id) + snapshot.sync_failed = snapshot.sync_status == "failed" if snapshot.sync_status in ("stale", "failed"): snapshot.sync_stale = True _warn( diff --git a/backend/tests/unit/services/test_knowledge_freshness_service.py b/backend/tests/unit/services/test_knowledge_freshness_service.py new file mode 100644 index 0000000..ad062f7 --- /dev/null +++ b/backend/tests/unit/services/test_knowledge_freshness_service.py @@ -0,0 +1,33 @@ +"""Tests for KnowledgeFreshnessService and KnowledgeFreshness dataclass.""" + +from app.services.knowledge_freshness_service import KnowledgeFreshness + + +class TestKnowledgeFreshnessDataclass: + """Test KnowledgeFreshness dataclass construction and properties.""" + + def test_default_construction(self): + """KnowledgeFreshness() with no args has warnings == [] and overall_stale is False.""" + freshness = KnowledgeFreshness() + assert freshness.warnings == [] + assert freshness.overall_stale is False + + def test_sync_failed_flag(self): + """A failed sync sets sync_failed=True.""" + freshness = KnowledgeFreshness(sync_status="failed", sync_failed=True) + assert freshness.sync_status == "failed" + assert freshness.sync_failed is True + + def test_sync_stale_flag(self): + """A stale sync sets sync_stale=True but sync_failed=False.""" + freshness = KnowledgeFreshness(sync_status="stale", sync_stale=True) + assert freshness.sync_status == "stale" + assert freshness.sync_stale is True + assert freshness.sync_failed is False + + def test_sync_ok_flag(self): + """A fresh sync (OK) sets both sync_stale and sync_failed to False.""" + freshness = KnowledgeFreshness(sync_status="ok") + assert freshness.sync_status == "ok" + assert freshness.sync_stale is False + assert freshness.sync_failed is False From c71c1fe02ae4ceba90fa16f8b8a2a08b375c6f13 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 16:49:35 +0200 Subject: [PATCH 21/24] fix(sync): cron wave honors per-project hour; reconciler covers all connections (M4,M1) Co-Authored-By: Claude Sonnet 4.6 --- backend/app/main.py | 116 ++++++---- backend/tests/unit/test_main_cron.py | 305 +++++++++++++++++++++++++++ 2 files changed, 382 insertions(+), 39 deletions(-) create mode 100644 backend/tests/unit/test_main_cron.py diff --git a/backend/app/main.py b/backend/app/main.py index 5a85efb..d9ccd75 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -6,7 +6,7 @@ import time import uuid from contextlib import asynccontextmanager -from datetime import UTC, datetime +from datetime import UTC, datetime, timedelta from fastapi import Depends, FastAPI, Request from fastapi.middleware.cors import CORSMiddleware @@ -603,11 +603,16 @@ async def _periodic_learning_decay() -> None: async def _freshness_reconcile() -> None: """Phase 2 FreshnessReconciler: auto re-index stale knowledge. - For each project + first connection, evaluate :class:`KnowledgeFreshness` + For each project + ALL connections, evaluate :class:`KnowledgeFreshness` and dispatch the matching background job (DB re-index / resync) when stale. Repo (git) staleness is handled by the dedicated git-poll loop; when that loop is disabled this also runs an inline repo fetch+reindex so a single flag still closes the loop. Gated by ``settings.freshness_reconciler_enabled``. + + Anti retry-storm: a connection whose ``sync_failed`` flag is set is attempted + at most once per reconcile pass (tracked in a local set keyed by + ``(project_id, connection_id)``). Ordinary stale (non-failed) connections + are not subject to this restriction. """ if not settings.freshness_reconciler_enabled: return @@ -628,26 +633,37 @@ async def _freshness_reconcile() -> None: fresh_svc = KnowledgeFreshnessService() triggered = 0 + # One-shot guard: (project_id, connection_id) keys that have already been + # submitted for a failed-sync retry in this reconcile pass. + failed_sync_retried: set[tuple[str, str]] = set() + async with async_session_factory() as session: projects = await proj_svc.list_all(session) for project in projects: connections = await conn_svc.list_by_project(session, project.id) - conn_id = connections[0].id if connections else None repo_dir = Path(settings.repo_clone_base_dir) / project.id - fresh = await fresh_svc.evaluate( - session, - project_id=project.id, - connection_id=conn_id, - repo_clone_dir=repo_dir if repo_dir.exists() else None, - ) - if not fresh.overall_stale: - continue - if conn_id and fresh.db_index_stale: - if await maybe_autostart_db_index(conn_id, project.id): - triggered += 1 - if conn_id and fresh.sync_stale: - if await maybe_autostart_sync(conn_id, project.id): - triggered += 1 + + for conn in connections: + fresh = await fresh_svc.evaluate( + session, + project_id=project.id, + connection_id=conn.id, + repo_clone_dir=repo_dir if repo_dir.exists() else None, + ) + if not fresh.overall_stale: + continue + if fresh.db_index_stale: + if await maybe_autostart_db_index(conn.id, project.id): + triggered += 1 + if fresh.sync_stale: + if fresh.sync_failed: + # One-shot guard: skip if already retried this pass. + guard_key = (project.id, conn.id) + if guard_key in failed_sync_retried: + continue + failed_sync_retried.add(guard_key) + if await maybe_autostart_sync(conn.id, project.id): + triggered += 1 # Avoid double-triggering repo re-index when the dedicated poll loop runs. if not settings.git_poll_enabled: @@ -690,25 +706,35 @@ async def _git_poll_loop() -> None: async def _dispatch_daily_knowledge_sync_wave() -> None: - """Enqueue one daily knowledge sync job per eligible project. + """Enqueue one daily knowledge sync job per eligible project whose effective hour matches now. - A Redis advisory lock (keyed by ``run_date``) ensures exactly one web dyno - dispatches the wave per calendar day; other dynos skip silently. + A Redis advisory lock keyed by ``run_date:current_hour`` ensures exactly one web dyno + dispatches the wave per calendar-day/hour combination; other dynos skip silently. + The per-project task_id is day-scoped (``daily_sync:{project_id}:{run_date}``) so a + project cannot be double-dispatched within the same calendar day even if the lock is + somehow acquired twice in different hours. """ from zoneinfo import ZoneInfo from app.core import task_queue from app.services.daily_knowledge_sync_service import DailyKnowledgeSyncService + from app.services.sync_schedule_service import SyncScheduleService tz = ZoneInfo(settings.daily_knowledge_sync_timezone) - run_date = datetime.now(tz).strftime("%Y-%m-%d") - - async with redis_lock(f"cron:daily_sync:{run_date}", ttl_seconds=3600) as acquired: + now = datetime.now(tz) + run_date = now.strftime("%Y-%m-%d") + current_hour = now.hour + + # Hour-scoped single-flight lock: each hour's wave is independent. + async with redis_lock( + f"cron:daily_sync:{run_date}:{current_hour}", ttl_seconds=3600 + ) as acquired: if not acquired: logger.info("Cron: daily knowledge sync wave skipped (lock held by another dyno)") return svc = DailyKnowledgeSyncService() + schedule_svc = SyncScheduleService() dispatched = 0 skipped = 0 @@ -717,9 +743,18 @@ async def _dispatch_daily_knowledge_sync_wave() -> None: all_projects = await svc._project_svc.list_all(session) eligible = await svc.list_eligible_projects(session) - skipped = len(all_projects) - len(eligible) + hour_filtered: list = [] + for project in eligible: + eff = await schedule_svc.effective(session, project.id) + if eff["hour"] == current_hour: + hour_filtered.append(project) + else: + skipped += 1 + + skipped += len(all_projects) - len(eligible) - for project in eligible: + for project in hour_filtered: + # Day-scoped task_id: prevents double-dispatch within a calendar day. task_id = f"daily_sync:{project.id}:{run_date}" pid = project.id @@ -740,37 +775,40 @@ async def _run_in_process(*, project_id: str = pid) -> None: dispatched += 1 logger.info( - "Cron: daily knowledge sync wave dispatched projects=%d skipped=%d", + "Cron: daily knowledge sync wave dispatched projects=%d skipped=%d hour=%d", dispatched, skipped, + current_hour, ) except Exception: logger.exception("Cron: daily knowledge sync wave dispatch failed") async def _daily_knowledge_sync_cron_loop() -> None: - """Run daily knowledge sync at the configured hour in Europe/Berlin (default 00:00 CET/CEST).""" + """Fire the sync wave once per hour so per-project schedule hours are honored. + + The loop wakes at the top of each hour (wall-clock aligned) and calls + :func:`_dispatch_daily_knowledge_sync_wave`, which filters eligible projects + whose effective schedule hour equals the current local hour. + + ``compute_next_scheduled_run`` is still available for the ``/sync-schedule`` + display endpoint in ``projects.py`` — do not remove it from the codebase. + """ if not settings.daily_knowledge_sync_enabled: return from zoneinfo import ZoneInfo - from app.services.daily_knowledge_sync_service import compute_next_scheduled_run - while True: try: - tz_name = settings.daily_knowledge_sync_timezone - tz = ZoneInfo(tz_name) - next_run = compute_next_scheduled_run( - datetime.now(tz), - hour=settings.daily_knowledge_sync_hour, - timezone_name=tz_name, - ) - wait_seconds = (next_run - datetime.now(tz)).total_seconds() + tz = ZoneInfo(settings.daily_knowledge_sync_timezone) + now = datetime.now(tz) + next_hour = now.replace(minute=0, second=0, microsecond=0) + timedelta(hours=1) + wait_seconds = max(1.0, (next_hour - now).total_seconds()) logger.info( - "Cron: next daily knowledge sync in %.0f seconds (at %s)", + "Cron: next daily knowledge sync wave in %.0f seconds (at %s)", wait_seconds, - next_run.isoformat(), + next_hour.isoformat(), ) await asyncio.sleep(wait_seconds) await _dispatch_daily_knowledge_sync_wave() diff --git a/backend/tests/unit/test_main_cron.py b/backend/tests/unit/test_main_cron.py new file mode 100644 index 0000000..4d93b3e --- /dev/null +++ b/backend/tests/unit/test_main_cron.py @@ -0,0 +1,305 @@ +"""Tests for T16: cron wave honors per-project hour + reconciler covers all connections. + +M4: _dispatch_daily_knowledge_sync_wave dispatches only projects whose effective hour + equals current_hour; uses hour-scoped Redis lock. +M1: _freshness_reconcile iterates all connections (not just connections[0]); + applies a one-shot guard for sync_failed to prevent retry storms. +""" + +from __future__ import annotations + +from contextlib import asynccontextmanager +from typing import Any +from unittest.mock import AsyncMock, MagicMock + +import app.main as main_mod + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _make_project(pid: str) -> Any: + p = MagicMock() + p.id = pid + return p + + +def _make_connection(cid: str) -> Any: + c = MagicMock() + c.id = cid + return c + + +# --------------------------------------------------------------------------- +# M4 — wave filters by effective hour +# --------------------------------------------------------------------------- + + +async def test_wave_dispatches_only_projects_matching_current_hour(monkeypatch): + """Only projects whose effective hour equals the current local hour are dispatched.""" + + import app.services.daily_knowledge_sync_service as svc_mod + import app.services.sync_schedule_service as schedule_mod + + # Freeze "now" at hour 3 in the configured timezone by patching datetime.now + # inside the main module. We intercept `datetime.now(tz).hour`. + frozen_hour = 3 + + original_datetime = main_mod.datetime + + class FrozenDatetime(original_datetime): + @classmethod + def now(cls, tz=None): # type: ignore[override] + real = original_datetime.now(tz) + # return a mock whose .hour == frozen_hour, .strftime keeps working + m = MagicMock() + m.hour = frozen_hour + m.strftime = real.strftime + return m + + monkeypatch.setattr(main_mod, "datetime", FrozenDatetime) + + # Lock always acquired + @asynccontextmanager + async def acquired_cm(key, *, ttl_seconds): + yield True + + monkeypatch.setattr(main_mod, "redis_lock", acquired_cm) + + # Two projects: proj_a runs at hour 3 (matches), proj_b runs at hour 5 (no match) + proj_a = _make_project("proj-a") + proj_b = _make_project("proj-b") + + class FakeSvc: + _project_svc = MagicMock() + + async def list_eligible_projects(self, s): + return [proj_a, proj_b] + + FakeSvc._project_svc.list_all = AsyncMock(return_value=[proj_a, proj_b]) + + monkeypatch.setattr(svc_mod, "DailyKnowledgeSyncService", lambda: FakeSvc()) + + # SyncScheduleService.effective returns per-project hour + effective_hours: dict[str, int] = {"proj-a": 3, "proj-b": 5} + + class FakeScheduleSvc: + async def effective(self, session, project_id): + return {"hour": effective_hours[project_id]} + + monkeypatch.setattr(schedule_mod, "SyncScheduleService", lambda: FakeScheduleSvc()) + + # Fake session factory + from contextlib import asynccontextmanager as acm + + @acm + async def fake_session(): + yield MagicMock() + + monkeypatch.setattr(main_mod, "async_session_factory", fake_session) + + # Capture enqueued task_ids by patching the enqueue function on the task_queue module. + # _dispatch_daily_knowledge_sync_wave does `from app.core import task_queue` and then + # calls `await task_queue.enqueue(...)`, so we patch the module-level `enqueue` function. + enqueued: list[str] = [] + + import app.core.task_queue as tq_mod + + async def fake_enqueue(name, *, coro_factory, task_id, **kwargs): + enqueued.append(task_id) + + monkeypatch.setattr(tq_mod, "enqueue", fake_enqueue) + + await main_mod._dispatch_daily_knowledge_sync_wave() + + # Only proj-a (hour==3) should be dispatched; proj-b (hour==5) skipped + assert len(enqueued) == 1 + assert "proj-a" in enqueued[0] + assert "proj-b" not in "".join(enqueued) + + +async def test_wave_uses_hour_scoped_lock_key(monkeypatch): + """Redis lock key must contain both run_date AND current_hour.""" + + frozen_hour = 7 + original_datetime = main_mod.datetime + + class FrozenDatetime(original_datetime): + @classmethod + def now(cls, tz=None): # type: ignore[override] + real = original_datetime.now(tz) + m = MagicMock() + m.hour = frozen_hour + m.strftime = real.strftime + return m + + monkeypatch.setattr(main_mod, "datetime", FrozenDatetime) + + captured_keys: list[str] = [] + + @asynccontextmanager + async def recording_lock(key, *, ttl_seconds): + captured_keys.append(key) + yield False # deny — we just need to check the key + + monkeypatch.setattr(main_mod, "redis_lock", recording_lock) + + await main_mod._dispatch_daily_knowledge_sync_wave() + + assert len(captured_keys) == 1 + key = captured_keys[0] + # Key must contain hour component + assert f":{frozen_hour}" in key or f":{frozen_hour:02d}" in key, ( + f"Expected hour {frozen_hour} in lock key, got: {key!r}" + ) + + +# --------------------------------------------------------------------------- +# M1 — reconciler covers ALL connections +# --------------------------------------------------------------------------- + + +async def test_reconciler_iterates_all_connections(monkeypatch): + """_freshness_reconcile must call maybe_autostart_db_index for EACH connection.""" + import app.api.routes.connections as conn_routes + import app.services.connection_service as conn_svc_mod + import app.services.knowledge_freshness_service as fresh_mod + import app.services.project_service as proj_svc_mod + + # Enable reconciler + monkeypatch.setattr(main_mod.settings, "freshness_reconciler_enabled", True) + monkeypatch.setattr(main_mod.settings, "git_poll_enabled", True) # skip git poll branch + + proj = _make_project("p1") + conn1 = _make_connection("c1") + conn2 = _make_connection("c2") + conn3 = _make_connection("c3") + + class FakeProjSvc: + async def list_all(self, s): + return [proj] + + class FakeConnSvc: + async def list_by_project(self, s, pid): + return [conn1, conn2, conn3] + + # Freshness: db_index_stale for all, sync not stale + class FakeFreshness: + def __init__(self): + self.overall_stale = True + self.db_index_stale = True + self.sync_stale = False + self.sync_failed = False + + class FakeFreshSvc: + async def evaluate(self, session, *, project_id, connection_id, repo_clone_dir): + return FakeFreshness() + + monkeypatch.setattr(proj_svc_mod, "ProjectService", lambda: FakeProjSvc()) + monkeypatch.setattr(conn_svc_mod, "ConnectionService", lambda: FakeConnSvc()) + monkeypatch.setattr(fresh_mod, "KnowledgeFreshnessService", lambda: FakeFreshSvc()) + + triggered_conn_ids: list[str] = [] + + async def fake_maybe_autostart_db_index(conn_id, project_id): + triggered_conn_ids.append(conn_id) + return True + + async def fake_maybe_autostart_sync(conn_id, project_id): + return False + + monkeypatch.setattr(conn_routes, "maybe_autostart_db_index", fake_maybe_autostart_db_index) + monkeypatch.setattr(conn_routes, "maybe_autostart_sync", fake_maybe_autostart_sync) + + from contextlib import asynccontextmanager as acm + + @acm + async def fake_session(): + yield MagicMock() + + monkeypatch.setattr(main_mod, "async_session_factory", fake_session) + + await main_mod._freshness_reconcile() + + # All three connections must have triggered db_index + assert set(triggered_conn_ids) == {"c1", "c2", "c3"}, ( + f"Expected all 3 connections triggered, got: {triggered_conn_ids}" + ) + + +async def test_reconciler_sync_failed_one_shot_guard(monkeypatch): + """A connection with sync_failed=True must be retried at most once per reconcile pass.""" + import app.api.routes.connections as conn_routes + import app.services.connection_service as conn_svc_mod + import app.services.knowledge_freshness_service as fresh_mod + import app.services.project_service as proj_svc_mod + + monkeypatch.setattr(main_mod.settings, "freshness_reconciler_enabled", True) + monkeypatch.setattr(main_mod.settings, "git_poll_enabled", True) + + # Two projects, both with one connection each that has sync_failed=True + proj1 = _make_project("p1") + proj2 = _make_project("p2") + conn_p1 = _make_connection("c-p1") + conn_p2 = _make_connection("c-p2") + + class FakeProjSvc: + async def list_all(self, s): + return [proj1, proj2] + + class FakeConnSvc: + async def list_by_project(self, s, pid): + return { + "p1": [conn_p1], + "p2": [conn_p2], + }[pid] + + class FakeFreshness: + def __init__(self): + self.overall_stale = True + self.db_index_stale = False + self.sync_stale = True + self.sync_failed = True + + class FakeFreshSvc: + async def evaluate(self, session, *, project_id, connection_id, repo_clone_dir): + return FakeFreshness() + + monkeypatch.setattr(proj_svc_mod, "ProjectService", lambda: FakeProjSvc()) + monkeypatch.setattr(conn_svc_mod, "ConnectionService", lambda: FakeConnSvc()) + monkeypatch.setattr(fresh_mod, "KnowledgeFreshnessService", lambda: FakeFreshSvc()) + + sync_calls: list[str] = [] + + async def fake_maybe_autostart_db_index(conn_id, project_id): + return False + + async def fake_maybe_autostart_sync(conn_id, project_id): + sync_calls.append(conn_id) + return True + + monkeypatch.setattr(conn_routes, "maybe_autostart_db_index", fake_maybe_autostart_db_index) + monkeypatch.setattr(conn_routes, "maybe_autostart_sync", fake_maybe_autostart_sync) + + from contextlib import asynccontextmanager as acm + + @acm + async def fake_session(): + yield MagicMock() + + monkeypatch.setattr(main_mod, "async_session_factory", fake_session) + + await main_mod._freshness_reconcile() + + # Each (project, connection) pair should be attempted exactly once despite sync_failed + assert sync_calls.count("c-p1") == 1 + assert sync_calls.count("c-p2") == 1 + + # Run again — the guard is per-pass (in-memory set is reset per call), so each + # new reconcile pass should retry once more. The set is local to the function + # invocation, so a second call should also yield exactly one attempt per connection. + sync_calls.clear() + await main_mod._freshness_reconcile() + assert sync_calls.count("c-p1") == 1 + assert sync_calls.count("c-p2") == 1 From 5347c2cd1e769a8c4cc75e073a567610474c7e0d Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 16:55:36 +0200 Subject: [PATCH 22/24] fix(sync): sync_now owner-budget 429 gate + sync-schedule next_run regression test (C3,M4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit C3: Add preflight_owner_budget 429 gate to sync_now route (after role check, before coord.start). Reuses same pattern as trigger_sync (T14). Blocks over-budget owners before any state changes. M4: Add regression tests asserting get_sync_schedule next_run hour matches effective per-project hour. Tests confirm hourly cron contract is held (no drift between displayed hour and compute_next_scheduled_run). All 4 tests pass (budget gate 429, passthrough 202/409, hour consistency with override, global fallback). Ruff/mypy pass. No code changes needed to schedule logic—already correct. Co-Authored-By: Claude Opus 4.8 (1M context) --- backend/app/api/routes/projects.py | 7 + .../integration/test_sync_now_and_schedule.py | 123 ++++++++++++++++++ 2 files changed, 130 insertions(+) create mode 100644 backend/tests/integration/test_sync_now_and_schedule.py diff --git a/backend/app/api/routes/projects.py b/backend/app/api/routes/projects.py index 1ee0477..5471e90 100644 --- a/backend/app/api/routes/projects.py +++ b/backend/app/api/routes/projects.py @@ -19,6 +19,7 @@ from app.services.membership_service import MembershipService from app.services.project_service import ProjectService from app.services.rule_service import RuleService +from app.services.sync_budget import preflight_owner_budget logger = logging.getLogger(__name__) @@ -554,6 +555,12 @@ async def sync_now( import uuid await _membership_svc.require_role(db, project_id, user["user_id"], "editor") + + # C3: budget pre-flight — block over-budget owners before we even acquire the lock. + ok, reason, _ = await preflight_owner_budget(db, project_id) + if not ok: + raise HTTPException(status_code=429, detail=reason) + from app.core import task_queue from app.services.run_coordinator import RunAlreadyActiveError, RunCoordinator diff --git a/backend/tests/integration/test_sync_now_and_schedule.py b/backend/tests/integration/test_sync_now_and_schedule.py new file mode 100644 index 0000000..1eb0ada --- /dev/null +++ b/backend/tests/integration/test_sync_now_and_schedule.py @@ -0,0 +1,123 @@ +"""Integration tests: sync_now 429 budget gate + sync-schedule next_run hour regression (C3, M4). + +Uses monkeypatch to make preflight_owner_budget return budget exhausted. +Also tests that get_sync_schedule computes next_run with the correct hour. +""" + +from __future__ import annotations + +from datetime import datetime +from unittest.mock import AsyncMock, patch + +import pytest +import pytest_asyncio +from httpx import AsyncClient + + +@pytest_asyncio.fixture() +async def sync_test_project(auth_client: AsyncClient): + """Create a project for sync_now and sync_schedule tests.""" + proj = await auth_client.post( + "/api/projects", + json={"name": "SyncNowTest", "description": "C3 sync_now budget gate"}, + ) + assert proj.status_code == 200 + return proj.json()["id"] + + +@pytest.mark.asyncio +class TestSyncNowBudgetGate: + """Test sync_now 429 pre-flight budget gate (C3).""" + + async def test_sync_now_429_when_over_budget(self, auth_client: AsyncClient, sync_test_project): + """sync_now must return 429 when owner is over budget.""" + project_id = sync_test_project + + # Patch preflight_owner_budget to simulate over-budget condition. + with patch( + "app.api.routes.projects.preflight_owner_budget", + new=AsyncMock(return_value=(False, "daily token budget exhausted", None)), + ): + resp = await auth_client.post(f"/api/projects/{project_id}/sync-now") + + assert resp.status_code == 429 + detail = resp.json()["detail"] + assert "budget" in detail.lower() + + async def test_sync_now_not_blocked_when_budget_ok( + self, auth_client: AsyncClient, sync_test_project + ): + """sync_now should proceed past the budget gate when budget is ok. + + The request will still fail (409/202 depending on state) but NOT 429, + confirming the budget gate let it through. + """ + project_id = sync_test_project + + with patch( + "app.api.routes.projects.preflight_owner_budget", + new=AsyncMock(return_value=(True, None, "owner-uid")), + ): + resp = await auth_client.post(f"/api/projects/{project_id}/sync-now") + + # 202 = started; 409 = already running; what matters is it's NOT 429. + assert resp.status_code in (202, 409) + assert resp.status_code != 429 + + +@pytest.mark.asyncio +class TestSyncScheduleNextRunHour: + """Test sync-schedule next_run hour regression (M4).""" + + async def test_sync_schedule_next_run_hour_matches_effective_hour( + self, auth_client: AsyncClient, sync_test_project + ): + """get_sync_schedule must compute next_run hour matching the effective hour.""" + project_id = sync_test_project + + # Set a custom sync schedule hour (e.g., 14 = 2 PM). + set_resp = await auth_client.put( + f"/api/projects/{project_id}/sync-schedule", + json={"enabled": True, "hour": 14}, + ) + assert set_resp.status_code == 200 + set_data = set_resp.json() + assert set_data["hour"] == 14 + assert set_data["enabled"] is True + + # Get the schedule and verify next_run hour matches. + get_resp = await auth_client.get(f"/api/projects/{project_id}/sync-schedule") + assert get_resp.status_code == 200 + data = get_resp.json() + assert data["hour"] == 14 + assert data["enabled"] is True + + # Parse the next_run ISO string and check its hour matches the effective hour. + if data["next_run"]: + next_run_dt = datetime.fromisoformat(data["next_run"]) + # The next_run should be in the project's timezone; hour should be 14. + assert next_run_dt.hour == 14, ( + f"next_run hour {next_run_dt.hour} does not match effective hour 14" + ) + + async def test_sync_schedule_next_run_with_default_global_hour(self, auth_client: AsyncClient): + """sync-schedule with no project override should use global hour.""" + # Create a new project (no sync-schedule override). + proj = await auth_client.post( + "/api/projects", + json={"name": "DefaultSchedule", "description": "Global schedule"}, + ) + assert proj.status_code == 200 + project_id = proj.json()["id"] + + # Get the schedule (should use global defaults). + get_resp = await auth_client.get(f"/api/projects/{project_id}/sync-schedule") + assert get_resp.status_code == 200 + data = get_resp.json() + assert data["source"] == "global" + + # next_run should be computed from the global hour. + if data["next_run"] and data["enabled"]: + next_run_dt = datetime.fromisoformat(data["next_run"]) + # Global hour should be in range [0, 23]. + assert 0 <= next_run_dt.hour <= 23 From fc09a20b7927aba677113cc159d5ab495eb3b39c Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 17:25:57 +0200 Subject: [PATCH 23/24] =?UTF-8?q?fix(sync):=20repair=20full-suite=20regres?= =?UTF-8?q?sions=20=E2=80=94=20heartbeat-safe=20budget=20preflight,=20defe?= =?UTF-8?q?nsive=20owner=20lookup,=20migration=20no-op,=20update=20stale?= =?UTF-8?q?=20tests=20to=20new=20contracts=20(T18)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 4.6 --- ...c_sync_remediation_indexing_run_active_.py | 27 +++----- .../app/knowledge/code_db_sync_pipeline.py | 64 ++++++++++--------- backend/app/services/sync_budget.py | 17 ++++- .../tests/unit/test_code_db_sync_analyzer.py | 12 ++++ .../tests/unit/test_code_db_sync_pipeline.py | 8 ++- backend/tests/unit/test_db_index_service.py | 42 +++++++++--- backend/tests/unit/test_db_index_validator.py | 21 +++++- backend/tests/unit/test_graph_db_bridge.py | 2 +- 8 files changed, 125 insertions(+), 68 deletions(-) diff --git a/backend/alembic/versions/f37386df158c_sync_remediation_indexing_run_active_.py b/backend/alembic/versions/f37386df158c_sync_remediation_indexing_run_active_.py index 117d829..2e3f168 100644 --- a/backend/alembic/versions/f37386df158c_sync_remediation_indexing_run_active_.py +++ b/backend/alembic/versions/f37386df158c_sync_remediation_indexing_run_active_.py @@ -12,7 +12,6 @@ from typing import Sequence, Union -import sqlalchemy as sa from alembic import op revision: str = "f37386df158c" @@ -22,23 +21,15 @@ def upgrade() -> None: - # Partial unique index: at most one active run per (project, kind, connection). - # coalesce() ensures NULL connection_id rows are also mutually exclusive per project+kind. - # if_not_exists=True makes this idempotent for envs that already applied a1f2b3c4d5e6. - op.create_index( - "uq_indexing_runs_active_one", - "indexing_runs", - ["project_id", "kind", sa.text("coalesce(connection_id, '')")], - unique=True, - postgresql_where=sa.text("status IN ('queued','running','cancelling')"), - sqlite_where=sa.text("status IN ('queued','running','cancelling')"), - if_not_exists=True, - ) + # NO-OP: the index `uq_indexing_runs_active_one` is authoritatively created + # by migration `a1f2b3c4d5e6_add_indexing_runs_and_error_log.py`. + # The `__table_args__` entry in `app/models/indexing_run.py` provides + # `create_all` parity for tests and fresh dev setups. + # Performing create_index here would cause a double-drop on `downgrade base`. + pass def downgrade() -> None: - op.drop_index( - "uq_indexing_runs_active_one", - table_name="indexing_runs", - if_exists=True, - ) + # NO-OP: index owned by a1f2b3c4d5e6; dropping it here would cause + # a double-drop when a1f2b3c4d5e6 also runs its downgrade. + pass diff --git a/backend/app/knowledge/code_db_sync_pipeline.py b/backend/app/knowledge/code_db_sync_pipeline.py index 0b7c52f..18ab4ab 100644 --- a/backend/app/knowledge/code_db_sync_pipeline.py +++ b/backend/app/knowledge/code_db_sync_pipeline.py @@ -66,36 +66,6 @@ async def run( {"connection_id": connection_id, "project_id": project_id}, ) - # H5: owner budget pre-flight + per-run usage sink. - from app.services.sync_budget import build_sink, preflight_owner_budget - - if settings.sync_budget_enforcement_enabled: - async with async_session_factory() as s: - ok, reason, owner_id = await preflight_owner_budget(s, project_id) - if not ok: - async with async_session_factory() as s: - await self._sync_svc.set_sync_status(s, connection_id, "failed") - await s.commit() - await self._tracker.end(wf_id, "code_db_sync", "failed", reason or "budget") - return { - "status": "failed", - "error": reason, - "budget_blocked": True, - "workflow_id": wf_id, - } - if owner_id: - self._llm = LLMRouter(usage_sink=build_sink(owner_id, project_id)) - self._analyzer = CodeDbSyncAnalyzer(self._llm) - - # H6: per-connection opt-out + global scrub flag. - async with async_session_factory() as s: - from app.models.connection import Connection - - conn = await s.get(Connection, connection_id) - send = getattr(conn, "send_sample_data_to_llm", True) if conn else True - scrub = settings.sync_pii_scrubbing_enabled - omit_samples = not send - async def _hb() -> None: async with async_session_factory() as s: await self._sync_svc.touch_heartbeat(s, connection_id) @@ -103,6 +73,40 @@ async def _hb() -> None: async with heartbeat(_hb, interval_seconds=settings.heartbeat_interval_seconds): try: + # H5: owner budget pre-flight + per-run usage sink. + # Must run INSIDE the heartbeat CM so crashes here don't break the + # heartbeat contract. + from app.services.sync_budget import build_sink, preflight_owner_budget + + if settings.sync_budget_enforcement_enabled: + async with async_session_factory() as s: + ok, reason, owner_id = await preflight_owner_budget(s, project_id) + if not ok: + async with async_session_factory() as s: + await self._sync_svc.set_sync_status(s, connection_id, "failed") + await s.commit() + await self._tracker.end(wf_id, "code_db_sync", "failed", reason or "budget") + return { + "status": "failed", + "error": reason, + "budget_blocked": True, + "workflow_id": wf_id, + } + if owner_id: + self._llm = LLMRouter(usage_sink=build_sink(owner_id, project_id)) + self._analyzer = CodeDbSyncAnalyzer(self._llm) + + # H6: per-connection opt-out + global scrub flag. + # Must run INSIDE the heartbeat CM so conn-load errors don't crash before + # the heartbeat opens. + async with async_session_factory() as s: + from app.models.connection import Connection + + conn = await s.get(Connection, connection_id) + send = getattr(conn, "send_sample_data_to_llm", True) if conn else True + scrub = settings.sync_pii_scrubbing_enabled + omit_samples = not send + # Mark as running async with async_session_factory() as session: await self._sync_svc.set_sync_status(session, connection_id, "running") diff --git a/backend/app/services/sync_budget.py b/backend/app/services/sync_budget.py index e034f4b..31e1348 100644 --- a/backend/app/services/sync_budget.py +++ b/backend/app/services/sync_budget.py @@ -17,9 +17,20 @@ async def resolve_owner_user_id(session: AsyncSession, project_id: str) -> str | None: - """Resolve a project's owner user ID.""" - row = await session.execute(select(Project.owner_id).where(Project.id == project_id)) - return row.scalar_one_or_none() + """Resolve a project's owner user ID. + + Returns None on any DB error so callers degrade gracefully instead of crashing. + """ + try: + row = await session.execute(select(Project.owner_id).where(Project.id == project_id)) + return row.scalar_one_or_none() + except Exception: + logger.debug( + "sync budget: could not resolve owner for project %s — unenforced", + project_id[:8], + exc_info=True, + ) + return None def build_sink(owner_user_id: str, project_id: str) -> DbUsageSink: diff --git a/backend/tests/unit/test_code_db_sync_analyzer.py b/backend/tests/unit/test_code_db_sync_analyzer.py index 2b44a7c..c68a36c 100644 --- a/backend/tests/unit/test_code_db_sync_analyzer.py +++ b/backend/tests/unit/test_code_db_sync_analyzer.py @@ -133,6 +133,7 @@ async def test_batch_analysis(self, analyzer, mock_llm): id="call_1", name="table_sync_analysis", arguments={ + "table_name": "t1", "data_format_notes": "Table 1", "column_sync_notes": "{}", "business_logic_notes": "", @@ -146,6 +147,7 @@ async def test_batch_analysis(self, analyzer, mock_llm): id="call_2", name="table_sync_analysis", arguments={ + "table_name": "t2", "data_format_notes": "Table 2", "column_sync_notes": "{}", "business_logic_notes": "", @@ -167,11 +169,14 @@ async def test_batch_analysis(self, analyzer, mock_llm): assert len(results) == 2 assert results[0].table_name == "t1" + assert results[0].sync_status == "matched" assert results[1].table_name == "t2" assert results[1].sync_status == "db_only" @pytest.mark.asyncio async def test_batch_fills_missing_with_fallback(self, analyzer, mock_llm): + # Only t1 is echoed back with its table_name; t2 has no matching tool call + # and must receive _fallback_analysis (is_fallback=True, sync_status="unknown"). mock_llm.complete.return_value = LLMResponse( content="", tool_calls=[ @@ -179,6 +184,7 @@ async def test_batch_fills_missing_with_fallback(self, analyzer, mock_llm): id="call_1", name="table_sync_analysis", arguments={ + "table_name": "t1", "data_format_notes": "", "column_sync_notes": "{}", "business_logic_notes": "", @@ -199,9 +205,15 @@ async def test_batch_fills_missing_with_fallback(self, analyzer, mock_llm): ) assert len(results) == 2 + # t1: matched from LLM response + assert results[0].table_name == "t1" assert results[0].sync_status == "matched" + assert results[0].is_fallback is False + # t2: no matching echoed call → fallback + assert results[1].table_name == "t2" assert results[1].sync_status == "unknown" assert results[1].confidence_score == 1 + assert results[1].is_fallback is True @pytest.mark.asyncio async def test_empty_batch(self, analyzer, mock_llm): diff --git a/backend/tests/unit/test_code_db_sync_pipeline.py b/backend/tests/unit/test_code_db_sync_pipeline.py index c3e1037..bfbb8ed 100644 --- a/backend/tests/unit/test_code_db_sync_pipeline.py +++ b/backend/tests/unit/test_code_db_sync_pipeline.py @@ -258,6 +258,8 @@ def test_includes_graph_callers_section(self): assert "create_order" in result assert "list_orders" in result assert "process_order" in result - # Confidence + depth surfaced. - assert "depth=1" in result - assert "depth=2" in result + # M7 format: op_kind + heuristic label; no fabricated depth= field. + assert "write" in result + assert "heuristic" in result + assert "conf=0.90" in result + assert "depth=" not in result diff --git a/backend/tests/unit/test_db_index_service.py b/backend/tests/unit/test_db_index_service.py index f268fff..d6adc3d 100644 --- a/backend/tests/unit/test_db_index_service.py +++ b/backend/tests/unit/test_db_index_service.py @@ -274,8 +274,9 @@ async def test_no_summary(self, svc): @pytest.mark.asyncio async def test_completed_index(self, svc): + # H7: only "completed" and "completed_partial" count as indexed. session = AsyncMock() - summary = _make_summary(indexing_status="idle") + summary = _make_summary(indexing_status="completed") result = MagicMock() result.scalar_one_or_none.return_value = summary session.execute = AsyncMock(return_value=result) @@ -357,14 +358,26 @@ async def test_not_found(self, svc): class TestDeleteStaleTables: @pytest.mark.asyncio async def test_deletes_stale(self, svc): - """T17: bulk DELETE reports rowcount from the execute() result.""" + """T17: delete_stale_tables removes rows not in current_keys set. + + The implementation fetches all ids first (1st execute), then DELETEs + the stale ones (2nd execute). current_keys uses schema-qualified keys + like "public.users". + """ session = AsyncMock() - result_mock = MagicMock() - result_mock.rowcount = 1 - session.execute.return_value = result_mock - count = await svc.delete_stale_tables(session, "conn-1", {"keep"}) + # First execute: returns existing rows (id, schema, name) + fetch_mock = MagicMock() + fetch_mock.all.return_value = [ + ("id-stale", "public", "stale_table"), + ("id-keep", "public", "keep"), + ] + # Second execute: the DELETE (return value not inspected by impl) + delete_mock = MagicMock() + session.execute.side_effect = [fetch_mock, delete_mock] + + count = await svc.delete_stale_tables(session, "conn-1", {"public.keep"}) assert count == 1 - session.execute.assert_awaited_once() + assert session.execute.await_count == 2 session.flush.assert_awaited_once() @pytest.mark.asyncio @@ -381,9 +394,18 @@ async def test_none_stale(self, svc): async def test_empty_current_set_deletes_all(self, svc): """Empty whitelist means 'delete everything for this connection'.""" session = AsyncMock() - result_mock = MagicMock() - result_mock.rowcount = 5 - session.execute.return_value = result_mock + # First execute: 5 existing rows, all stale when current_keys is empty + fetch_mock = MagicMock() + fetch_mock.all.return_value = [ + ("id-1", "public", "table_a"), + ("id-2", "public", "table_b"), + ("id-3", "public", "table_c"), + ("id-4", "dbo", "table_d"), + ("id-5", None, "table_e"), + ] + delete_mock = MagicMock() + session.execute.side_effect = [fetch_mock, delete_mock] + count = await svc.delete_stale_tables(session, "conn-1", set()) assert count == 5 diff --git a/backend/tests/unit/test_db_index_validator.py b/backend/tests/unit/test_db_index_validator.py index 9d951e7..f97b8ca 100644 --- a/backend/tests/unit/test_db_index_validator.py +++ b/backend/tests/unit/test_db_index_validator.py @@ -123,12 +123,27 @@ def test_none_sample(self): class TestBuildTablePrompt: def test_basic_prompt(self): - table = _make_table() - sample = _make_sample_data() + # Use non-PII sample data so pii_scrubber (called internally with scrub=True) + # does not replace values, keeping assertions stable. + from app.connectors.base import ColumnInfo + + table = _make_table( + columns=[ + ColumnInfo(name="id", data_type="integer", is_primary_key=True, is_nullable=False), + ColumnInfo(name="username", data_type="varchar(50)", is_nullable=False), + ColumnInfo(name="status", data_type="varchar(20)", is_nullable=True), + ] + ) + sample = QueryResult( + columns=["id", "username", "status"], + rows=[[1, "alice", "active"], [2, "bob", "inactive"]], + row_count=2, + ) prompt = DbIndexValidator._build_table_prompt(table, sample, "", "") assert "## Table: users" in prompt assert "id: integer" in prompt - assert "alice@example.com" in prompt + assert "alice" in prompt + assert "username" in prompt def test_with_code_context(self): table = _make_table() diff --git a/backend/tests/unit/test_graph_db_bridge.py b/backend/tests/unit/test_graph_db_bridge.py index cfdcbae..745ae45 100644 --- a/backend/tests/unit/test_graph_db_bridge.py +++ b/backend/tests/unit/test_graph_db_bridge.py @@ -91,7 +91,7 @@ class TestClassifyOpKind: ("update_profile", "write"), ("delete_order", "write"), ("save_invoice", "write"), - ("set_status", "write"), + ("set_status", "unknown"), # moved to _AMBIGUOUS_VERBS in M7 ("get_user", "read"), ("find_by_id", "read"), ("list_orders", "read"), From bdab6efe9b3db7dde3f5d570363be622f23c8b94 Mon Sep 17 00:00:00 2001 From: NGHTBOY Date: Fri, 26 Jun 2026 17:34:34 +0200 Subject: [PATCH 24/24] docs(sync): R5 changelog + close 22 sync-audit findings in qa-audit/issues.md (T18) Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 39 +++++++++++++++++++++++++++++++++++++++ docs/qa-audit/issues.md | 15 +++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c1e249..ea8f84d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,45 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Fixed — R5: code↔DB sync reliability & correctness (2026-06-25 sync audit) + +Closes all 22 findings of the five-specialist code↔DB synchronization audit (9 High, 9 Medium, +4 Low). Branch `fix/sync-remediation-2026-06-25`; 18 TDD tasks; combined suite 4560 passing, 75% +coverage; ruff/mypy clean. Spec `docs/superpowers/specs/2026-06-25-sync-remediation-design.md`, +plan `docs/superpowers/plans/2026-06-25-sync-remediation.md`. + +- **Reliability (High):** the daily-sync parent `IndexingRun` now emits a continuous heartbeat + (targeted `UPDATE`, no `version` lost-update) so the stale-run reaper no longer kills a healthy + multi-minute sync (**H1**); the daily cron sub-steps **adopt-or-skip** instead of launching an + untracked concurrent pipeline on an active-run conflict (**H9**); `RunCoordinator.start` translates + the single-active `IntegrityError` into a clean `RunAlreadyActiveError`/409 with session rollback, + and the partial-unique active index is mirrored onto the model for `create_all` test parity (**H8**); + `is_indexed` now only counts `completed`/`completed_partial` (a failed-only index is no longer + reported as indexed) (**H7**). +- **Data correctness (High):** batch table analyses are reconciled by the LLM-echoed `table_name` + instead of tool-call position, ending silent cross-table misattribution (**H2**); a malformed + `confidence_score` degrades only its own table instead of aborting the batch (**H3**); a degraded + LLM run (mostly fallback) no longer overwrites previously-good sync rows, and low-confidence rows + no longer enforce/surface required-filter guidance (**H4**). +- **Cost & privacy (High):** sync LLM calls are now metered + budget-gated against the project + owner (manual triggers 429 on exhaustion; cron degrades gracefully; ownerless projects run + unenforced) (**H5**); DB sample data + distinct values are scrubbed (column denylist + value + redaction) before egress to the LLM at both the sync and db-index analyzers, with a per-connection + `send_sample_data_to_llm` opt-out (default on) (**H6**). +- **Medium:** freshness reconciler now covers all connections, not just the first (**M1**); + schema-qualified table identity prevents same-named cross-schema tables from collapsing (**M2**); + the daily-sync parent run advances through manifest steps instead of 0%→100% (**M3**); the cron + wave honors the per-project schedule hour (**M4**); daily sync regenerates the project overview + and the worker logs the correct matched count (**M5**); investigation enrichment is routed to a + non-enforced field and `required_filters` payloads are validated + value mappings deep-merged + (**M6**); graph-derived `op_kind` heuristics are labelled non-authoritative (over-broad write verbs + reclassified) (**M7**); freshness `warnings` uses a proper default + a `sync_failed` flag (**M8**); + `get_index_age` guards a NULL `indexed_at` (**M9**). +- **Low:** the reaper logs a sweep even when the driver returns an unknown rowcount (**L1**); the + prompt header no longer fabricates an "analyzed" date for a never-completed sync (**L2**); daily + child-run orphaning is covered by H1+H9 (**L3**); context truncation is marked and relevance + matching tightened (**L4**). + ### Added - **MCP protocol-polish (F5/F6/F9).** Shipped in three batched releases on top diff --git a/docs/qa-audit/issues.md b/docs/qa-audit/issues.md index 385c544..9034b50 100644 --- a/docs/qa-audit/issues.md +++ b/docs/qa-audit/issues.md @@ -23,6 +23,21 @@ in branch `fix/security-audit-2026-06-24` → PR validator/repair carry the sink; MCP tools build `DbUsageSink`-bound router + acquire `agent_limiter` for parity with chat; explicit JWT preferred over env candidate; startup warning for empty `MCP_ALLOWED_HOSTS`. +- **Release R5 — code↔DB sync reliability & correctness** (branch `fix/sync-remediation-2026-06-25`, + 18 TDD tasks, commits `d672486`…`fc09a20`): closes **all 22 findings of the separate 2026-06-25 + five-specialist sync-subsystem audit** (9 High, 9 Medium, 4 Low — tracked under that audit's own + H/M/L scheme, not the `F--NN` tally below). **High:** H1 daily-sync parent heartbeat + (reaper no longer kills healthy runs), H2 batch analyses reconciled by echoed `table_name`, H3 + per-table confidence coercion, H4 all-fallback overwrite guard + confidence-gated filters, H5 + owner-attributed budget gate (429/graceful-cron), H6 PII scrub + per-connection opt-out, H7 + `is_indexed` status whitelist, H8 `IntegrityError`→409 + model index parity, H9 adopt-not-run. + **Medium:** M1 reconciler all-connections, M2 schema-qualified table identity, M3 parent-run + progress steps, M4 per-project schedule hour honored, M5 overview regen + worker log key, M6 + enrichment validation/deep-merge + producer reroute, M7 graph `op_kind` heuristic labelling, M8 + freshness defaults + `sync_failed`, M9 `get_index_age` NULL guard. **Low:** L1 reaper unknown- + rowcount log, L2 prompt-header date gating, L3 child-run orphaning (covered by H1+H9), L4 + truncation markers + relevance matching. Combined suite 4560 passing, 75% coverage; ruff/mypy + clean; alembic up/down/base clean. Spec + plan under `docs/superpowers/{specs,plans}/2026-06-25-*`. This file also folds in the **2026-06-23 codebase audit** (`code-reviewer` baseline + manual review). Its findings map as: **H1** = F-KNOW-01 (fixed); **M2** = F-KNOW-02 (fixed); **M1** =