diff --git a/.vale-baseline.json b/.vale-baseline.json index f84a3de..7c69a7c 100644 --- a/.vale-baseline.json +++ b/.vale-baseline.json @@ -1,6 +1,5 @@ { "Google.Colons": 9, - "Google.EmDash": 448, "Google.FirstPerson": 3, "Google.Headings": 63, "Google.OxfordComma": 28, diff --git a/.vale.ini b/.vale.ini index 7757801..6c874c0 100644 --- a/.vale.ini +++ b/.vale.ini @@ -36,12 +36,13 @@ Vale.Terms = NO # Off: American technical English spells facade without the cedilla. proselint.Diacritical = NO -# Warning: the docs set an em dash with a space on each side throughout; the rule checks only -# that spacing, and its hundreds of hits are one typographic choice, not hundreds of defects. -# The baseline holds the count, so a new page adds none. Vale can report two of these hits as -# one when a wrapped list item's dash meets a later one (scripts/vale_gate.py says when), so a -# count that falls on a change adding a dash is not one to lower the baseline to. -Google.EmDash = warning +# Error: the docs set no dash with a space on each side (` — `, ` – `, a line break counting as +# a space); a colon, comma, period or parentheses carries the sense instead, and one spaced dash +# fails the gate. It sits at error rather than in the warning ratchet because Vale can report two +# of its hits as one when a wrapped list item's dash meets a later one (scripts/vale_gate.py says +# when), so a count could fall on a change that adds a dash. At error a merge folds two alerts +# into one, never into none, so an added dash is red either way. +Google.EmDash = error # Warning: a quoted literal (a status, a rule's text, a field value) keeps the punctuation that # is not part of it outside the quotes, as the docs quote them verbatim. Google.Quotes = warning diff --git a/.vale/fixtures/planted-warning.md b/.vale/fixtures/planted-warning.md new file mode 100644 index 0000000..8cc2a3b --- /dev/null +++ b/.vale/fixtures/planted-warning.md @@ -0,0 +1,3 @@ +# A planted warning + +atif-sql converts Claude Code transcripts to ATIF, and the converter will write one trajectory. diff --git a/.vale/fixtures/wrapped-emdash.md b/.vale/fixtures/wrapped-emdash.md new file mode 100644 index 0000000..6759f95 --- /dev/null +++ b/.vale/fixtures/wrapped-emdash.md @@ -0,0 +1,7 @@ +# A wrapped em dash + +- [Sequences](sequences.md) — call order for the top processes, in a list item that wraps + onto a second line. +- The next item. + +A later paragraph with no dash. diff --git a/AGENTS.md b/AGENTS.md index 85320a3..b13dd67 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,4 +1,4 @@ -# atif-sql — operating manual +# atif-sql: operating manual ATIF-native analytics over agent trajectories from Claude Code (`~/.claude/projects/**/*.jsonl`) and Codex CLI @@ -12,7 +12,7 @@ documents, and layers DuckDB views on top. One corpus holds one agent, and uv WORKSPACE (virtual root, members under `packages/*`): -- `packages/atif-converter` — converts Claude Code and Codex transcripts to +- `packages/atif-converter`: converts Claude Code and Codex transcripts to ATIF (ported from harbor's converters, on the ATIF models vendored in `atif_converter.domain.atif`); owns the fidelity policy per agent (the known Claude Code gaps in `atif_converter.domain.fidelity`, the Codex ones in @@ -23,7 +23,7 @@ uv WORKSPACE (virtual root, members under `packages/*`): `domain.pricing` prices each step (`total_cost_usd`, Codex `cost_usd`) from the vendored `domain/model_prices.json` with litellm's arithmetic; a model it can't price is NULL. Layered: `application` > `infrastructure` > `domain`. -- `packages/atif-corpus` — corpus materialization: discovery, watermarks, +- `packages/atif-corpus`: corpus materialization, covering discovery, watermarks, quiescence, atomic artifact writes (the bulk JSON artifacts zstd-compressed), in-place compression of an old-layout session for `corpus slim`, the `ArtifactProducer` port that lets a composition root add per-session files @@ -37,7 +37,7 @@ uv WORKSPACE (virtual root, members under `packages/*`): before it becomes a corpus path (rejected ids are reported, never written); atif-duck carries the twin. Layered: `application` > `infrastructure` > `domain`. -- `packages/atif-duck` — DuckDB views + macros over the materialized corpus: +- `packages/atif-duck`: DuckDB views + macros over the materialized corpus: core views and macros plus the analytics views and macros, all declared in a static drift-tested catalog. Who wrote a user step is decided ONCE, in `domain.authorship` (a prefix rule table rendered @@ -52,12 +52,12 @@ uv WORKSPACE (virtual root, members under `packages/*`): DuckLake every corpus is queried through (`domain.lake` declares the tables, `infrastructure.lake` writes, reads, verifies and compacts it; see "Storage" below). Layered: `infrastructure` > `domain`. -- `packages/atif-models` — model alias registry + structured-output LLM +- `packages/atif-models`: model alias registry + structured-output LLM client. No other package hardcodes a Bedrock model id. Layered: `infrastructure` > `domain`. -- `packages/atif-analytics` — the LLM pipelines that spend money at +- `packages/atif-analytics`: the LLM pipelines that spend money at Bedrock and are checkpointed per session (classify, conflicts, friction, - perceived — see `PIPELINE_NAMES` in + perceived; see `PIPELINE_NAMES` in `infrastructure/sqlite_state/checkpointer.py`). They read human turns only (`domain.authorship`), and classify and conflicts skip automated review (`turn_audit`) and one-shot-job sessions. Session data comes through the @@ -69,12 +69,12 @@ uv WORKSPACE (virtual root, members under `packages/*`): cluster/terms/community pipelines were cut on 2026-09-27; outcome is the deterministic `session_outcomes` view now. Layered: `application` > `infrastructure` > `domain`. -- `packages/atif-embed` — Cohere Embed v4 on Bedrock + LanceDB store + the +- `packages/atif-embed`: Cohere Embed v4 on Bedrock + LanceDB store + the backfill and orphan-prune use cases. Discovery reads the lake's `steps` through the `LakeStepsPort` port (atif-cli implements it over atif-duck's `infrastructure.lake_steps`), and falls back to the per-session reader. Layered: `application` > `infrastructure` > `domain`. -- `packages/atif-cli` — cyclopts CLI composing the rest into commands +- `packages/atif-cli`: cyclopts CLI composing the rest into commands (`convert`, `materialize`, `status`, `query`, `analyze`, `embed`, `search`, `examples`, `schema`, `cron`, `lake`, `corpus`). @@ -83,7 +83,7 @@ Rules of the road: - Independence contract (import-linter, `pyproject.toml`): converter / corpus / duck / models / embed may NEVER import each other. atif-cli is the composition root and may import them all. atif-analytics is the one - exception — it may import atif-models and nothing else among our + exception: it may import atif-models and nothing else among our packages (a `forbidden` contract pins its other edges shut). - Inter-package deps: declare in the member's `[project.dependencies]` AND `[tool.uv.sources] = { workspace = true }`. @@ -372,25 +372,25 @@ copy of the queryable rows: For an LLM agent driving `atif-sql`, the discovery loop is this: -1. `atif-sql schema` — every view (with columns) and macro signature, core +1. `atif-sql schema`: every view (with columns) and macro signature, core and analytics, each with what it `requires`, from the static catalog in <50 ms. Its output ends with an `examples_hint`. -2. `atif-sql examples` (alias: `atif-sql query --examples`) — runnable +2. `atif-sql examples` (alias: `atif-sql query --examples`): runnable example queries for every view and macro, DERIVED from the catalog (never hardcoded per object) and each one EXECUTED by - `packages/atif-duck/tests/test_examples.py` against a fixture corpus — + `packages/atif-duck/tests/test_examples.py` against a fixture corpus, and the header's "test-executed against this version" is literal. Piped output is JSON `{note, examples: [{name, sql, description, requires, category}]}`; filter with `--requires core|analytics|vss` and `--category view|table-macro|scalar-macro`. -3. `atif-sql query ''` — run it. `--agent codex` points it at the Codex +3. `atif-sql query ''`: run it. `--agent codex` points it at the Codex corpus without spelling the path, and `--all-corpora` runs it over every corpus the lake holds (`sessions.corpus` names each row's corpus). Copy an example verbatim (the `sid` exemplar is a subquery over `sessions`, so it works on any corpus) or adapt it. `requires: analytics` needs `atif-sql analyze` to have run; `requires: vss` needs `atif-sql embed --all --no-dry-run` (a bare `embed` exits 64: a real run needs an explicit scope). For semantic search over TEXT, - prefer `atif-sql search 'query'` — it embeds the text first, then runs + prefer `atif-sql search 'query'`, which embeds the text first, then runs the same `semantic_search` kNN. What the query sandbox will and won't do: `SELECT`, `EXPLAIN`, `SET @@ -407,14 +407,14 @@ For an LLM agent driving `atif-sql`, the discovery loop is this: Adding a view/macro? The drift tests force: a `DESCRIPTIONS` entry in `atif_duck/domain/catalog.py`, an `ARG_EXEMPLARS` entry for any new parameter name, `TABLE_MACRO_NAMES` membership if the DDL is `AS TABLE`, and -the derived example must execute — or a documented `EXCLUSIONS` entry. +the derived example must execute (or a documented `EXCLUSIONS` entry). ## Definition of done `mise run check` fully green (lint + fmt + typecheck + lint:imports + lint:workflows + lint:hooks + lint:pnpm-lock + security:vex:check + docs:prose + test). `docs:prose` is Vale over the published prose (docs/, site/authored/, README.md, AGENTS.md, CONTRIBUTING.md) at error level, -with a file floor per glob so a shrunken scope is red, and a per-rule warning ratchet against `.vale-baseline.json`: reword the sentence it flags, or add a real term to `.vale/styles/config/vocabularies/atif-sql/accept.txt`. When a count falls, run `mise run docs:prose:baseline` and commit the lower file; never raise it. `mise run security` is the report-only tier beside it: findings do not fail +with a file floor per glob so a shrunken scope is red, and a per-rule warning ratchet against `.vale-baseline.json`: reword the sentence it flags, or add a real term to `.vale/styles/config/vocabularies/atif-sql/accept.txt`. When a count falls, run `mise run docs:prose:baseline` and commit the lower file; never raise it. A spaced dash is an error (`Google.EmDash`), not a warning: write a colon, comma, period, or parentheses instead. `mise run security` is the report-only tier beside it: findings do not fail it, a scanner that produced no usable SARIF does. A change under docs/ or site/ also needs `mise run docs:gate`: the site build, then `docs:links`, which crawls site/dist as GitHub Pages serves it and fails on any internal 404. Link page to page by @@ -426,7 +426,7 @@ First-time setup: `mise trust && mise install && mise run install`. `experiments/` holds numbered experiment protocols (README per experiment). Outputs go to `experiments/**/out/` (gitignored). No experiment is wired -into a gate — `mise run check` and pytest's `testpaths` both skip the +into a gate: `mise run check` and pytest's `testpaths` both skip the directory. Migration parity oracles are not part of this repo. Parity is not a standing diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6b13bb7..ebd68c2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,8 +5,8 @@ atif-sql is a uv workspace whose root is also the one published distribution: which bundles every module tree) and `[tool.uv.workspace] members = ["packages/*"]`. The packages under `packages/` are development members, not install targets. Everything below is read off the -checked-in config — `mise.toml`, `pyproject.toml`, `lefthook.yml`, and -`AGENTS.md` — so if a claim here and a config file disagree, the config file +checked-in config (`mise.toml`, `pyproject.toml`, `lefthook.yml`, and +`AGENTS.md`), so if a claim here and a config file disagree, the config file wins and this document is the bug. ## First-time setup @@ -28,7 +28,7 @@ settles on 3.14. Drive every command through `mise run ` rather than calling `uv`, `ruff`, or `ty` directly. The mise `[env]` block sets `PYTHONDONTWRITEBYTECODE=1` and `UV_LINK_MODE=copy`, and `[settings] python.uv_venv_auto = "create|source"` -creates and sources the workspace venv — a bare tool invocation can miss all of +creates and sources the workspace venv; a bare tool invocation can miss all of that. ## Definition of done: `mise run check` @@ -56,6 +56,10 @@ prints `lower the baseline: mise run docs:prose:baseline`: run that task and com the change. A baseline only goes down in review, so a diff that raises a count in it needs the same scrutiny as a widened `ignore` list. +`Google.EmDash` is an error, not a ratchet: a dash with a space on each side (a line break counts as +a space) fails `docs:prose` on its own. Write a colon, comma, period, or parentheses in its place, +whichever keeps the sentence's sense; `.vale.ini` says why the rule sits at error. + ## The docs site: `mise run docs:gate` A change under `docs/` or `site/` also needs `mise run docs:install && mise run docs:gate`, which @@ -74,13 +78,13 @@ internal link that does not answer 200. Never reach green by relaxing a gate. Widening an `ignore` list, adding a blanket per-file ignore, or deleting a contract is a change to the project's -standards, not a fix to your branch — raise it in the pull request instead. +standards, not a fix to your branch. Raise it in the pull request instead. ## Commits Commit messages must be [Conventional Commits](https://www.conventionalcommits.org): `type(scope): subject` with `type` one of `feat|fix|chore|docs|refactor|test|perf|ci`. -This is enforced, not advisory — `lefthook.yml`'s `commit-msg` hook runs +This is enforced, not advisory: `lefthook.yml`'s `commit-msg` hook runs `uv run cz check --allow-abort --commit-msg-file {1}`, configured by `[tool.commitizen]` in `pyproject.toml` (`name = "cz_conventional_commits"`, with @@ -109,7 +113,7 @@ root packages. These shape the whole workspace: 1. **Independence.** `atif_converter`, `atif_corpus`, `atif_duck`, `atif_models`, and `atif_embed` may never import each other. If you need - two of them in one flow, compose them in `atif_cli` — it is the only + two of them in one flow, compose them in `atif_cli`; it is the only composition root and may import them all. 2. **`atif_analytics` composes `atif_models` and nothing else of ours.** A `forbidden` contract pins its remaining edges shut: it may not import @@ -120,7 +124,7 @@ root packages. These shape the whole workspace: The rest are `layers` contracts: `application > infrastructure > domain` for `atif_converter`, `atif_corpus`, `atif_embed`, and `atif_analytics`, and `infrastructure > domain` for -`atif_models`. Dependencies point inward — a `domain` module importing from +`atif_models`. Dependencies point inward: a `domain` module importing from `infrastructure` fails `lint:imports`. Other rules live in ruff rather than import-linter: @@ -140,7 +144,7 @@ failure. **A dependency on a sibling package.** Declare it in the member's `[project.dependencies]` *and* add `[tool.uv.sources] = { workspace = true }` -— both, per `AGENTS.md`. The independence contract above still applies. +(both, per `AGENTS.md`). The independence contract above still applies. **A new package.** `packages/*` is globbed as a workspace member, but these places in the root `pyproject.toml` list packages explicitly and will not pick @@ -153,14 +157,14 @@ The new member also needs the license wiring every existing member has: a `LICENSE` symlink to the repo root (`ln -s ../../LICENSE packages//LICENSE`). `license-files` globs are resolved inside the member's own directory and `..` is rejected, so the symlink is what puts the -license text into the wheel's `dist-info/licenses/` — Apache-2.0 §4(a) requires +license text into the wheel's `dist-info/licenses/`; Apache-2.0 §4(a) requires it, and `uv build` fails outright if the file is missing. **A DuckDB view or macro.** The catalog is static and drift-tested. Per `AGENTS.md`, adding an object means adding a `DESCRIPTIONS` entry in `atif_duck/domain/catalog.py`, an `ARG_EXEMPLARS` entry for any new parameter name, `TABLE_MACRO_NAMES` membership if the DDL is `AS TABLE`, and a derived -example that actually executes — or a documented `EXCLUSIONS` entry. The tests +example that actually executes (or a documented `EXCLUSIONS` entry). The tests in `packages/atif-duck/tests/` fail until that is true. ## harbor and litellm are test oracles, not dependencies @@ -231,7 +235,7 @@ pipelines) and `atif-sql embed` (Cohere Embed v4). Each is guarded, and the guards are part of the contract: - `analyze` is dry-run by default; `--no-dry-run` is what makes it spend. -- `embed` refuses a real run with no scope — no `--limit` and no `--all` exits 64. +- `embed` refuses a real run with no scope: no `--limit` and no `--all` exits 64. The test suite must stay offline: fake the port, never the credential. If a change makes a Bedrock call reachable from `pytest`, that is a defect in the @@ -240,7 +244,7 @@ change. ## Experiments `experiments/` holds numbered protocols with a README (and, once run, a REPORT) -per experiment. Nothing there is wired into a gate — pytest's `testpaths` only +per experiment. Nothing there is wired into a gate: pytest's `testpaths` only collects `packages/*/tests`, and outputs are written to `experiments/**/out/`, which `.gitignore` excludes. An experiment's recorded numbers are measurements; if you re-run one, add your measurement rather than editing the one already recorded. diff --git a/README.md b/README.md index dc16adc..6d3383e 100644 --- a/README.md +++ b/README.md @@ -13,15 +13,15 @@ Claude Code sessions (`~/.claude/projects/**/*.jsonl`) and Codex CLI rollouts (`~/.codex/sessions/**/rollout-*.jsonl`) are converted to ATIF (Harbor's Agent Trajectory Interchange Format, see the Harbor ATIF RFC: RFC-0001 in ), materialized as a corpus, and -queried through DuckDB views. Converting once at the boundary — with an -explicit, tested fidelity policy for what upstream drops — beats re-deriving +queried through DuckDB views. Converting once at the boundary (with an +explicit, tested fidelity policy for what upstream drops) beats re-deriving trajectory semantics inside every SQL view. Both agents land in the same views, and `sessions.agent` says which one a row came from. ## Install One command, one package, every capability. Conversion, materialization, the DuckDB surface, -the LLM analytics pipelines, and semantic search are all in the box — there are no extras to +the LLM analytics pipelines, and semantic search are all in the box; there are no extras to choose and nothing to install afterwards to make a command work. ```bash @@ -140,7 +140,7 @@ pick which, or `--agent codex` to get the Codex default. `--all-corpora` reads every corpus the lake holds at once, and `sessions.corpus` says which one a row came from. -Working on `atif-sql` itself is a different setup — a clone, `mise`, and `mise run check` as the +Working on `atif-sql` itself is a different setup: a clone, `mise`, and `mise run check` as the definition of done. `CONTRIBUTING.md` has it. ## Agent workflow: schema → examples → query @@ -152,7 +152,7 @@ atif-sql query 'SELECT * FROM tool_rank(30) LIMIT 10' ``` `atif-sql examples` (or `atif-sql query --examples`) emits runnable queries -derived from the catalog — not hardcoded strings — and every one is executed +derived from the catalog, not hardcoded strings, and every one is executed by the test suite against a fixture corpus, so the listing cannot rot. Piped output is JSON; filter with `--requires core|analytics|vss` and `--category view|table-macro|scalar-macro`. @@ -162,7 +162,7 @@ Piped output is JSON; filter with `--requires core|analytics|vss` and Setup, the gates `mise run check` runs, the import-linter contracts you will trip, and the Conventional Commit rule the `commit-msg` hook enforces: **[CONTRIBUTING.md](CONTRIBUTING.md)**. -Releases are cut by commitizen and published to PyPI over OIDC Trusted Publishing — +Releases are cut by commitizen and published to PyPI over OIDC Trusted Publishing; **[RELEASING.md](RELEASING.md)** covers the flow, the versioning model, and the install-weight measurements. diff --git a/docs/CONTRACT-V2.md b/docs/CONTRACT-V2.md index 46ddaf7..768dbc6 100644 --- a/docs/CONTRACT-V2.md +++ b/docs/CONTRACT-V2.md @@ -1,6 +1,6 @@ # atif-sql wave-2 contract: analytics, VSS, cron (addendum to CONTRACT.md) -Status: EXECUTED. Every item below is built and shipped — atif-models, +Status: EXECUTED. Every item below is built and shipped: atif-models, atif-analytics, atif-embed, the VSS stack, and the cron lanes all exist. The imperative phrasing ("port X", "use Y") is the original build instruction, kept because it records WHICH shape was chosen and why. When it disagrees @@ -21,7 +21,7 @@ original build. package hardcodes a model id. - atif-analytics: the v2 pipelines (classify, trajectory, conflicts, friction, perceived, cluster, terms, community). Depends on atif-models - ONLY — the `forbidden` contract bars atif-duck. atif-cli composes. + ONLY; the `forbidden` contract bars atif-duck. atif-cli composes. ## Model registry (atif-models domain) Size aliases -> per-family concrete IDs (bedrock-runtime, GLOBAL inference profiles): @@ -31,7 +31,7 @@ Size aliases -> per-family concrete IDs (bedrock-runtime, GLOBAL inference profi anthropic equivalent = global.anthropic.claude-sonnet-5 large : openai gpt-5.6 sol = global.openai.gpt-5.6-sol anthropic equivalent = global.anthropic.claude-opus-5 -Default family: openai (per operator decision 2026-08-23 — GPT 5.6 supports +Default family: openai (per operator decision 2026-08-23; GPT 5.6 supports native strict structured outputs). Anthropic equivalents are registry entries only (selection escape hatch), not wired defaults. Defaults: reasoning_effort="high", max_completion_tokens=32_000. @@ -48,15 +48,15 @@ Settings env: ATIF_SQL_LLM_FAMILY, ATIF_SQL_LLM_SIZE_ overrides. system goes in as a "system" role message. Errors: RefusalError (terminal), ProviderUnavailable (retryable -> retry queue). - Strict-mode schema rules for OpenAI: additionalProperties=false everywhere, - all properties required (OpenAI strict requirement — optional fields become + all properties required (OpenAI strict requirement: optional fields become nullable-typed), $defs inlined where Bedrock chokes. Keep pydantic validation as the second gate behind provider strict mode. - tenacity retry over the retryable provider codes; token accounting accumulated per pipeline; concurrency via one anyio CapacityLimiter (default 16). ## Pipeline size assignments (defaults) -classify=medium(terra), trajectory=medium, conflicts=large(sol — hardest -judgment task), friction=small(luna — per-message enum), embed unchanged +classify=medium(terra), trajectory=medium, conflicts=large(sol, hardest +judgment task), friction=small(luna, per-message enum), embed unchanged (cohere embed v4 1024-dim int8, one adapter behind EmbeddingProvider). ## Ports & state: one shape per seam, declared in domain/ports.py @@ -69,7 +69,7 @@ prompts = steps-based rendering under fixed caps (50K/tool_result, 800K/session). conflicts needs uuid-addressable turns -> use messages(edges) uuids in headers; validate returned uuids against edges. Schemas/enums are fixed vocabularies (autonomy tiers, work categories, -transition_kinds, conflict kinds, friction labels) — parity of meaning. +transition_kinds, conflict kinds, friction labels): parity of meaning. Friction tiers: regex + SQL stamp layers run first (zero-cost), LLM tier on luna. Structural pipelines (cluster/terms/community): fixed hyperparameters (UMAP 50-dim/HDBSCAN 20,5/Leiden k15 floor .3 min 3 seed 42/c-TF-IDF 2,.95,1-2,top10). diff --git a/docs/CONTRACT.md b/docs/CONTRACT.md index f7fb024..242aa88 100644 --- a/docs/CONTRACT.md +++ b/docs/CONTRACT.md @@ -4,8 +4,8 @@ Binding for all packages. Orchestrator-owned; change only via this file. This file binds the corpus layout, the enrichment pass, the transcript-derived view/macro surface, and the CLI shapes below. What sits on -top of them — the analytics pipelines, VSS, the model registry, the cron -lanes — is bound by `docs/CONTRACT-V2.md`. +top of them (the analytics pipelines, VSS, the model registry, the cron +lanes) is bound by `docs/CONTRACT-V2.md`. ## Scope: the transcript-derived surface The views @@ -15,7 +15,7 @@ task_creations/task_updates/tasks_state_current, skill_invocations + macros ago, model_used, cost_estimate, tool_rank, todo_velocity, subagent_fanout, skill_rank, skill_source_mix; CLI query/schema/status. VSS/semantic_search and the analytics pipelines are shipped commands, bound -by CONTRACT-V2 rather than by this file — their absence from the list above +by CONTRACT-V2 rather than by this file; their absence from the list above divides labour between the two contracts and does not deny they exist. Lean proofs are out of scope for the workspace. @@ -340,7 +340,7 @@ SUPERSET of the prior implementation (workflow-nested files are visible); (f) enrichment covers >= 99% of assistant uuids; (g) sidechain step token sums equal an independent raw subagent-file sum. -All gates passed — first on a small sample, then on a larger set of sessions +All gates passed: first on a small sample, then on a larger set of sessions across many project dirs. Neither the runners nor the recorded verdicts are tracked here. Parity is not a standing gate: the comparison target is not a dependency of this repo, so re-running the oracle would need an install this diff --git a/docs/README.md b/docs/README.md index 6d853ae..4f53645 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,7 +8,7 @@ Prose is generated; structure is mechanical. Cross-references are deterministic. Citations come as full paths or as shorthands, and each one resolves into a source file. A validator rejects any citation naming a file that does not exist, a line outside that file's range, or a shorthand with no resolvable antecedent, and it exits non-zero on the -first one — so a citation here resolves or the tree does not ship. +first one, so a citation here resolves or the tree does not ship. ## Start here @@ -22,63 +22,63 @@ first one — so a citation here resolves or the tree does not ship. ## Architecture -- [System overview](architecture/system-overview.md) — the short version, the stack, and one +- [System overview](architecture/system-overview.md): the short version, the stack, and one diagram of the workspace members. -- [Module map](architecture/module-map.md) — one section per member, ordered by source size, with the +- [Module map](architecture/module-map.md): one section per member, ordered by source size, with the files in each that carry the weight. -- [Data flow](architecture/data-flow.md) — the flows (`materialize`, `query`, `analyze`) traced end +- [Data flow](architecture/data-flow.md): the flows (`materialize`, `query`, `analyze`) traced end to end, each as a sequence diagram. ## Reference -- [CLI](reference/cli.md) — every command with verbatim usage, every flag, and the exit-code +- [CLI](reference/cli.md): every command with verbatim usage, every flag, and the exit-code taxonomy. This is the public contract: atif-sql ships one installable distribution and the command is its supported entry point. -- [Public API](reference/public-api.md) — the internal module surface, ranked by the enforced +- [Public API](reference/public-api.md): the internal module surface, ranked by the enforced inter-package seam. No symbol here is a supported import path or carries a compatibility promise. ## Behavior -- [Processes](behavior/processes.md) — the processes with numbered, cited steps, plus the minor +- [Processes](behavior/processes.md): the processes with numbered, cited steps, plus the minor flows. -- [State machines](behavior/state-machines.md) — the genuine multi-state lifecycles: corpus +- [State machines](behavior/state-machines.md): the genuine multi-state lifecycles: corpus session materialization, embed-store schema version, and the retry-queue row. ## Analysis -- [Risk hotspots](analysis/risk-hotspots.md) — ranked by type-checker defects, scanner findings, +- [Risk hotspots](analysis/risk-hotspots.md): ranked by type-checker defects, scanner findings, complexity ratchets, unguarded IO windows, and uncovered units. Commit churn is deliberately excluded and the document says why. -- [Dead code](analysis/dead-code.md) — nothing in this workspace is deletable code. What the +- [Dead code](analysis/dead-code.md): nothing in this workspace is deletable code. What the populated tables record instead is export declarations with no consumer. ## Diagrams -- [Components](diagrams/architecture/components.md) — one class diagram over the members. -- [Dependency graph](diagrams/structural/dependency-graph.md) — internal members and external +- [Components](diagrams/architecture/components.md): one class diagram over the members. +- [Dependency graph](diagrams/structural/dependency-graph.md): internal members and external distributions on one page, with the enforced direction stated. -- [Sequences](diagrams/behavioral/sequences.md) — call order for the top processes. +- [Sequences](diagrams/behavioral/sequences.md): call order for the top processes. ## Insights -- [Impact analysis](insights/impact-analysis.md) — the high-impact surfaces, each with the +- [Impact analysis](insights/impact-analysis.md): the high-impact surfaces, each with the downstream effects and the gate that catches you for missing one. -- [Debugging guide](insights/debugging-guide.md) — a failure-mode index, the log and error surfaces, +- [Debugging guide](insights/debugging-guide.md): a failure-mode index, the log and error surfaces, and a first-checks ladder ordered cheapest first. -- [Contract map](insights/contract-map.md) — the contracts with producer, consumer, and shape, +- [Contract map](insights/contract-map.md): the contracts with producer, consumer, and shape, including the units, base, and scope of every numeric field crossing a seam. -- [Business logic](insights/business-logic.md) — the domain rules: validations, invariants, +- [Business logic](insights/business-logic.md): the domain rules: validations, invariants, calculations, and policies, with the test that pins each one where a test exists. -- [Tech debt](insights/tech-debt.md) — a ranked register with cost of removal. The workspace carries +- [Tech debt](insights/tech-debt.md): a ranked register with cost of removal. The workspace carries no `TODO`/`HACK`/`FIXME` markers, so the register is built from version ceilings, justified lint suppressions, install weight, and platform gaps instead. ## Documents this tree deliberately omits -- **`analysis/ownership.md`** — the history sits under a single bot identity, +- **`analysis/ownership.md`**: the history sits under a single bot identity, with no human authors. A per-person ranked table and a bus-factor list would be noise dressed as analysis. -- **`reference/rpc-tools.md`** — there is no RPC, MCP, or HTTP surface. `fastapi`, `uvicorn`, +- **`reference/rpc-tools.md`**: there is no RPC, MCP, or HTTP surface. `fastapi`, `uvicorn`, `starlette`, and the supabase client stack do appear in the installed dependency closure, arriving transitively through harbor, and no first-party module imports any of them. diff --git a/docs/analysis/dead-code.md b/docs/analysis/dead-code.md index 5d9d35a..1556220 100644 --- a/docs/analysis/dead-code.md +++ b/docs/analysis/dead-code.md @@ -14,7 +14,7 @@ definitions under `packages/*/src` have no reference outside their own file, and **none** has no reference anywhere. Every one of those resolves to a call, construction, or annotation site inside its own module. What the two tables below record is a narrower defect: an **export declaration -with no consumer** — a name in an `__all__` that nothing imports. +with no consumer**: a name in an `__all__` that nothing imports. **Technique, and what it cannot see.** No code index covers this repo (there is no `.codegraph/`, no LSP index, no AST symbol graph), and no dead-code analyzer is wired into the project: `grep -n @@ -28,8 +28,8 @@ finding set is therefore derived, in these passes: `__all__` membership, and whether its module declares an `__all__` at all. 2. **Unanchored reference index.** For each name, the whole-word pattern - `(?.parquet` straight into the live @@ -117,23 +117,23 @@ a large share of its uncovered units are in `replace_sessions` and `write_part`. ### `atif_cli.app` **What's there.** The cyclopts composition root and the only module importing atif-converter, -atif-corpus, and atif-duck together, wiring every cross-package seam — the `ConverterPort` adapter, +atif-corpus, and atif-duck together, wiring every cross-package seam: the `ConverterPort` adapter, the clock, version pins, the DuckDB connection (`packages/atif-cli/src/atif_cli/app.py:3-15`). Heavy imports are deferred into the command bodies so the `schema` / `--help` fast path stays lean, a property pinned by a fresh-interpreter test rather than by a lint (`:22-26`). -**Recent activity.** The most commits of any file in the repo, so `↑ rising` — expected of a +**Recent activity.** The most commits of any file in the repo, so `↑ rising`, which is expected of a composition root that gains a wiring line whenever any member changes. -**Owners.** The bot identity at 100% — every commit touching the path, merges included — +**Owners.** The bot identity at 100% (every commit touching the path, merges included), a bot identity. **Findings.** A `B608` at the `search` kNN query, whose `# noqa` records that `dim` is `len(vector)` while the session id, `k`, and the vector itself are `?`-bound (`:925-937`). It carries the `max-args = 19` ratchet through `search`, whose parameters are the CLI flags cyclopts binds (`pyproject.toml:264`, `:180`; the function at -`packages/atif-cli/src/atif_cli/app.py:860`). Its uncovered units — the largest single-file gap -in the workspace — concentrate in the commands that reach outward: `analyze` +`packages/atif-cli/src/atif_cli/app.py:860`). Its uncovered units (the largest single-file gap +in the workspace) concentrate in the commands that reach outward: `analyze` (`:627-725`), `convert` (`:225-291`), and `status` (`:412-488`), with a few in `search` (`:828-948`) and `main` (`:1093-1103`). `analyze`, `embed`, and `search` are the billable commands that call Bedrock, so the least-covered command body in the tree is also the one @@ -142,7 +142,7 @@ applies the budget ceilings a crontab line depends on (`:691-707`). ### `atif_converter.domain.enrichment` -**What's there.** A pure function over a trajectory dict plus raw records — no harbor import, no IO — +**What's there.** A pure function over a trajectory dict plus raw records (no harbor import, no IO) that repairs some of the named fidelity gaps by re-running harbor's deterministic normalization order (`packages/atif-converter/src/atif_converter/domain/enrichment.py:3-30`). It exists because harbor 0.22.0 puts no record identity in `step.extra`: it reads `requestId` off the @@ -151,9 +151,9 @@ lands in extra and there is nothing to join on (`:15-20`). **Recent activity.** Commits well above the median, so `↑ rising`. -**Owners.** The bot identity at 100% — every commit touching the path — a bot identity. +**Owners.** The bot identity at 100% (every commit touching the path), a bot identity. -**Findings.** No scanner findings, no IO windows — this is a pure function — and both +**Findings.** No scanner findings, no IO windows (this is a pure function), and both complexity ratchets in the workspace that a single function sets: `max-branches = 39` and `max-complexity = 38` are both pinned at `enrich_trajectory` (`pyproject.toml:267`, `:181`, `:185-190`; the function at @@ -175,14 +175,14 @@ views bind to those literal values, so the vocabulary is a fixed contract (`:29- **Recent activity.** Commits well above the median, so `↑ rising`. -**Owners.** The bot identity at 100% — every commit touching the path — a bot identity. +**Owners.** The bot identity at 100% (every commit touching the path), a bot identity. **Findings.** No scanner findings, and the `max-statements = 119` ratchet, set at `_friction_async` (`pyproject.toml:269`, `:183`; the function at `packages/atif-analytics/src/atif_analytics/application/use_cases/friction.py:240`). The coverage gap and the complexity outlier are the same region: most of the file's uncovered units sit inside `_friction_async` (`:240-528`), with the rest in `detect_user_friction` (`:531-635`) and -`deterministic_stamps` (`:178-232`). This is the file where the two signals coincide most tightly — +`deterministic_stamps` (`:178-232`). This is the file where the two signals coincide most tightly: the longest function in the workspace is also the least-exercised, and it is the function that decides which candidate messages cross into the billable LLM tier. The budget-guard posture that governs that decision assumes every candidate reaches the LLM even though roughly half survive the diff --git a/docs/architecture/data-flow.md b/docs/architecture/data-flow.md index 6ed65c2..23ff4e6 100644 --- a/docs/architecture/data-flow.md +++ b/docs/architecture/data-flow.md @@ -2,14 +2,14 @@ The distribution declares exactly one entry point, the console script `atif-sql = "atif_cli.app:main"` (`packages/atif-cli/pyproject.toml:42`), and the `main` it names -installs a WARNING-and-up loguru sink and hands control to cyclopts — `packages/atif-cli/src/atif_cli/app.py:1125`. +installs a WARNING-and-up loguru sink and hands control to cyclopts (`packages/atif-cli/src/atif_cli/app.py:1125`). Every process below therefore begins as a CLI invocation; there is no HTTP, RPC, or queue surface to enter through. The flows below either produce the corpus or consume it: `materialize` fills `/sessions/` (`:339`), `analyze` fills `/analytics/` (`:627`), and `query` binds both and runs caller SQL over them (`:497`). The other commands are subsets of -these flows — `convert` (`:225`) is one iteration of flow 1's inner loop, `search` (`:828`) +these flows: `convert` (`:225`) is one iteration of flow 1's inner loop, `search` (`:828`) re-enters flow 2's registration at `:886` and adds one kNN statement, `embed` (`:734`) writes the vector store flows 2 and 3 read, and `schema` (`:1055`), `examples` (`:963`), and `status` (`:412`) answer from static data or `stat` calls with no downstream participant. @@ -25,25 +25,25 @@ catalog copy. 1. The `materialize` command resolves `CorpusSettings` (pydantic-settings, env prefix `ATIF_SQL_`), then injects what the pure use case will not own: the `ConverterPort` adapter, the wall-clock instant, and the harbor / converter version pins - stamped into every `meta.json` — `packages/atif-cli/src/atif_cli/app.py:352`. + stamped into every `meta.json` (`packages/atif-cli/src/atif_cli/app.py:352`). 2. One pass through the corpus use case runs scan, plan, convert, and write in that order and - returns a `MaterializationReport` — `packages/atif-corpus/src/atif_corpus/application/materialize.py:476`. + returns a `MaterializationReport` (`packages/atif-corpus/src/atif_corpus/application/materialize.py:476`). 3. The scanner discovers every session under the raw transcript root and separates unreadable - sessions from absent ones, so a `stat` failure is never mistaken for a deletion — `packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py:143`. It walks whichever layout the + sessions from absent ones, so a `stat` failure is never mistaken for a deletion (`packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py:143`). It walks whichever layout the agent selects, descending exactly `transcript_depth` directories and reporting an unlistable one at - any level — `packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py:154`. + any level (`packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py:154`). 4. The pure planner partitions the scan into to-materialize, up-to-date, and skipped-live using the previous watermark and the quiescence policy; `force` overrides staleness but never - liveness — `packages/atif-corpus/src/atif_corpus/domain/sessions.py:159`. + liveness (`packages/atif-corpus/src/atif_corpus/domain/sessions.py:159`). 5. For each planned session the use case calls `converter.convert` across the port; one session's - exception is caught and recorded so a single bad transcript cannot abort the sync — `packages/atif-corpus/src/atif_corpus/application/materialize.py:310`. + exception is caught and recorded so a single bad transcript cannot abort the sync (`packages/atif-corpus/src/atif_corpus/application/materialize.py:310`). By default the sessions are spread over a process pool (`--workers`, default `min(8, cpu_count)`) - whose workers each hold a pickled copy of the adapter; the pool changes no output byte — `:385`. + whose workers each hold a pickled copy of the adapter; the pool changes no output byte (`:385`). 6. The adapter that satisfies `ConverterPort` lives in atif-cli because the independence contract forbids atif-corpus from importing atif-converter; it raises rather than returning an invalid - trajectory — `packages/atif-cli/src/atif_cli/converter_adapter.py:51`. + trajectory (`packages/atif-cli/src/atif_cli/converter_adapter.py:51`). 7. Under `--agent codex` the same step runs `convert_codex_and_audit`, which reads the one rollout it - is given and converts it with our ported Codex converter — `packages/atif-converter/src/atif_converter/application/convert_codex.py:128`. + is given and converts it with our ported Codex converter (`packages/atif-converter/src/atif_converter/application/convert_codex.py:128`). 8. `convert_and_audit` reads the source files once (fingerprinted and parsed in the same pass, `harbor_adapter.read_session`), converts those records with our ported converter through the seam at `packages/atif-converter/src/atif_converter/infrastructure/harbor_adapter.py` @@ -56,7 +56,7 @@ catalog copy. `ArtifactProducer`. `meta.json` is written last and the whole directory swaps into `sessions//`, so a reader sees one complete generation or the other (`packages/atif-corpus/src/atif_corpus/application/materialize.py:240`); the watermark advances - only for sessions that succeeded — `:608`. + only for sessions that succeeded (`:608`). 10. Before the first swap, the pass records every session it may publish or mark in `/sink_pending.json`. After the swaps, still in the parent process, it hands the `SessionSink` (atif-duck's `DuckLakeSessionSink`, plugged in by the CLI unless `--no-lake`) every @@ -112,12 +112,12 @@ sequenceDiagram 1. The `query` command resolves the corpus root and reads the active embedder's `(model_id, dim)` without constructing the embedder, so the vector store's stamped identity can be checked before - anything binds over it — `packages/atif-cli/src/atif_cli/app.py:529`. + anything binds over it (`packages/atif-cli/src/atif_cli/app.py:529`). 2. A DuckDB connection is opened with no path or URI, so the engine runs in-process against - memory — `:591`. -3. Registration builds the whole catalog on that connection in a fixed order — raw TEMP tables, - base views, VSS, macros, the authorship macro and views, analytics views, analytics macros — - because each later stage binds against the earlier one at `CREATE` time — `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1679`. + memory (`:591`). +3. Registration builds the whole catalog on that connection in a fixed order (raw TEMP tables, + base views, VSS, macros, the authorship macro and views, analytics views, analytics macros), + because each later stage binds against the earlier one at `CREATE` time (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:1679`). 4. When the DuckLake at `ATIF_SQL_LAKE_ROOT` exists, its recorded schema matches the running code, and it holds this corpus at this root, the raw relations are views over the lake tables instead, filtered to this corpus (`--all-corpora` drops the filter): the command loads the ducklake @@ -135,15 +135,15 @@ sequenceDiagram (DuckDB's JSON reader decompresses a `.zst` file itself) into a TEMP TABLE over an explicit path list; the two sets are unioned into one raw view per surface. The parse cost of a `query` invocation is therefore O(sessions - without artifacts) per connection — `packages/atif-duck/src/atif_duck/infrastructure/registry.py:402`. + without artifacts) per connection (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:402`). 6. Trajectory, edges, and loss readers are semi-joined against the meta table, so a session directory missing its `meta.json` contributes nothing rather than a partial artifact set. This - is the read-side half of flow 1's meta-last write ordering — `packages/atif-duck/src/atif_duck/infrastructure/registry.py:221`. + is the read-side half of flow 1's meta-last write ordering (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:221`). 7. The VSS step `ATTACH`es the LanceDB store through the lance extension (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:887`) and reads the store's stamped `model` and `dim` back out (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:921`) before creating any view over it; a store written by a different provider or width raises instead of binding, because vectors from different models live in incompatible spaces and would return numerically valid but meaningless - cosine scores — `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:46`. + cosine scores (`packages/atif-duck/src/atif_duck/domain/embedding_guard.py:46`). 8. Before any of that, the connection was sized to the host (`_configure_query_resources`: a memory cap from available RAM, a thread count from that cap, a private `mkdtemp` spill directory, and extension auto-install and auto-load off), because registration is what needs the cap. The @@ -192,18 +192,18 @@ sequenceDiagram 1. The `analyze` command loads `AnalyticsSettings`, reuses the same corpus-root resolution the other commands share, and stamps explicit `--max-sessions` / `--max-cost-usd` ceilings over the - env defaults so a crontab line carries its spend cap visibly — `packages/atif-cli/src/atif_cli/app.py:1240`. + env defaults so a crontab line carries its spend cap visibly (`packages/atif-cli/src/atif_cli/app.py:1240`). 2. The command opens the lake's session source when the lake holds the corpus (`packages/atif-cli/src/atif_cli/app.py:1624`, `packages/atif-cli/src/atif_cli/lake_sessions.py:83`), and the pipeline runner builds one shared `CorpusReader` over it (or over the file source) for every stage and loops the LLM stages (classify, conflicts, friction, perceived); the - `skip_*` flags subtract stages — `packages/atif-analytics/src/atif_analytics/application/analyze.py:62`. + `skip_*` flags subtract stages (`packages/atif-analytics/src/atif_analytics/application/analyze.py:62`). 3. On the lake, a session whose lake rows match its current `meta.json` is read with batched statements over the step-level tables (`packages/atif-duck/src/atif_duck/infrastructure/lake_sessions.py:71`), reading ahead in the newest-first walk order, and every other session from its files. Without - a lake, rows are read with stdlib `json` over `/sessions//trajectory.json.zst` (or the plain file on an old-layout session) — + a lake, rows are read with stdlib `json` over `/sessions//trajectory.json.zst` (or the plain file on an old-layout session); atif-analytics never imports DuckDB or atif-duck, which the `forbidden` import contract puts - out of its reach — `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:225`. + out of its reach (`packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:225`). Either way the steps sit behind the reader's bounded memos (`packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:263`), and the reader labels each session's kind with the same rule `session_outcomes.kind` applies, so @@ -211,15 +211,15 @@ sequenceDiagram 4. Before the first stage starts, one `RunBudget` is constructed from the per-run dollar ceiling (`packages/atif-analytics/src/atif_analytics/application/analyze.py:83`); every stage then goes through one call site, each receiving the shared reader and that budget, and a stage that finds - the budget exhausted is skipped with nothing stamped — `:123`. + the budget exhausted is skipped with nothing stamped (`:123`). 5. A stage drops the sessions whose checkpoint row still matches their mtime and last step - timestamp, so a re-run costs nothing for unchanged work — `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:137`. + timestamp, so a re-run costs nothing for unchanged work (`packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:137`). 6. The concrete provider satisfies `LlmStructuredProvider` structurally, and its synchronous inner method is the only place a Bedrock `invoke_model` call is issued, under tenacity retry with - token usage accumulated against the budget — `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:258`. + token usage accumulated against the budget (`packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:258`). 7. Results land as sharded `part-.parquet` files under `/analytics/` (`packages/atif-analytics/src/atif_analytics/infrastructure/parquet_cache.py:90`), and each - completed session is upserted into the SQLite WAL checkpoint — `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:180`. + completed session is upserted into the SQLite WAL checkpoint (`packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:180`). ```mermaid sequenceDiagram diff --git a/docs/architecture/module-map.md b/docs/architecture/module-map.md index 80a37dd..235f6d8 100644 --- a/docs/architecture/module-map.md +++ b/docs/architecture/module-map.md @@ -13,7 +13,7 @@ below are ordered by total source size, descending. ## atif-analytics -`run_analyze` composes the LLM pipelines in a fixed stage order — classify, conflicts, friction, +`run_analyze` composes the LLM pipelines in a fixed stage order: classify, conflicts, friction, perceived (`packages/atif-analytics/src/atif_analytics/application/analyze.py:36`). Every stage calls a model and honors `dry_run`, which defaults to True as a cost guard so the stages return plan dicts instead of spending (`:13`). Every classifier system prompt lives in one module, @@ -23,26 +23,26 @@ response schemas they bind against are pure domain models whose field descriptio part of the prompt surface (`packages/atif-analytics/src/atif_analytics/domain/models.py:3`). Who wrote a user step comes from `atif_analytics.domain.authorship`, a pinned twin of atif-duck's rule table (`packages/atif-analytics/src/atif_analytics/domain/authorship.py`). This is the one member -permitted to import a sibling — atif-models and nothing else (`pyproject.toml:520`). +permitted to import a sibling: atif-models and nothing else (`pyproject.toml:520`). -- `packages/atif-analytics/src/atif_analytics/application/prompts.py` — the task-framing +- `packages/atif-analytics/src/atif_analytics/application/prompts.py`: the task-framing system prompts, public constants assembled at `packages/atif-analytics/src/atif_analytics/application/prompts.py:739`. -- `packages/atif-analytics/src/atif_analytics/application/use_cases/friction.py` — the +- `packages/atif-analytics/src/atif_analytics/application/use_cases/friction.py`: the friction tiers behind a human-turn and message-length pre-filter: regex fast path, deterministic stamp rules, then the LLM (`packages/atif-analytics/src/atif_analytics/application/use_cases/friction.py:5`). -- `packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py` — - LangSmith's Perceived Error definition on the conflicts chassis; clean sessions produce zero rows +- `packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py`: + the LangSmith Perceived Error definition on the conflicts chassis; clean sessions produce zero rows (`packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py:16`). -- `packages/atif-analytics/src/atif_analytics/application/use_cases/conflicts.py` — one row +- `packages/atif-analytics/src/atif_analytics/application/use_cases/conflicts.py`: one row per detected stance-conflict pair, keyed on two turn uuids, with refusals routed to a sidecar; non-interactive sessions are skipped (`packages/atif-analytics/src/atif_analytics/application/use_cases/conflicts.py:11`). -- `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py` — one row +- `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py`: one row per interactive session carrying `work_category` and `goal`, anti-joined against the parquet cache and written in crash-resilient chunks (`packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:3`). -- `packages/atif-analytics/src/atif_analytics/domain/models.py` — the response schemas, from +- `packages/atif-analytics/src/atif_analytics/domain/models.py`: the response schemas, from `SessionClassification` (`packages/atif-analytics/src/atif_analytics/domain/models.py:22`) to `PerceivedErrorsResult` (`:355`). @@ -60,95 +60,95 @@ macros register separately, each only when its backing parquet is populated, bec `domain/` and `infrastructure/` but no `application/`, a deliberate shape that is why it carries no layers contract among the members (`pyproject.toml:462`). -- `packages/atif-duck/src/atif_duck/infrastructure/registry.py` — the raw readers, most +- `packages/atif-duck/src/atif_duck/infrastructure/registry.py`: the raw readers, most of the core views and macros, and the Lance attach path (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:767`). -- `packages/atif-duck/src/atif_duck/domain/raw_readers.py` — the raw relation names and +- `packages/atif-duck/src/atif_duck/domain/raw_readers.py`: the raw relation names and column shapes the registry binds and the lake stores, so neither declares them twice. -- `packages/atif-duck/src/atif_duck/domain/lake.py` — the lake's tables, partition specs and +- `packages/atif-duck/src/atif_duck/domain/lake.py`: the lake's tables, partition specs and every statement the lake runs, built as module constants from the raw reader shapes, plus the schema version and digest that decide when a lake is stale. -- `packages/atif-duck/src/atif_duck/infrastructure/lake.py` — `DuckLakeSessionSink` (the +- `packages/atif-duck/src/atif_duck/infrastructure/lake.py`: `DuckLakeSessionSink` (the `SessionSink` implementation materialize writes through), the published read-only catalog copy, the query-side attach and fallback, and `rebuild_lake`, `verify_lake`, `lake_status` and `compact_lake`. - `packages/atif-duck/src/atif_duck/domain/artifacts.py` and - `packages/atif-duck/src/atif_duck/infrastructure/stored_artifacts.py` — which artifacts may be + `packages/atif-duck/src/atif_duck/infrastructure/stored_artifacts.py`: which artifacts may be stored as `.zst`, and how a reader finds a session's stored file (compressed first), reads it, and sizes it from its zstd frame header. -- `packages/atif-duck/src/atif_duck/infrastructure/lake_steps.py` — one corpus's step texts and +- `packages/atif-duck/src/atif_duck/infrastructure/lake_steps.py`: one corpus's step texts and primary uuids read from the published lake for the embedding store: the full read, the read of the uuids `ducklake_table_changes` names after a snapshot, and the corpus's lineage and snapshot range. -- `packages/atif-duck/src/atif_duck/infrastructure/lake_settings.py` — `LakeSettings`, the +- `packages/atif-duck/src/atif_duck/infrastructure/lake_settings.py`: `LakeSettings`, the `ATIF_SQL_`-prefixed lake root, corpus base, batch sizes, lock timeout and snapshot expiry. -- `packages/atif-duck/src/atif_duck/domain/catalog.py` — `VIEW_NAMES` +- `packages/atif-duck/src/atif_duck/domain/catalog.py`: `VIEW_NAMES` (`packages/atif-duck/src/atif_duck/domain/catalog.py:30`), `MACRO_SIGNATURES` (`:287`), the analytics catalogs (`:314`), and one `DESCRIPTIONS` entry per object (`:431`). -- `packages/atif-duck/src/atif_duck/infrastructure/analytics.py` — the v2 views and macros +- `packages/atif-duck/src/atif_duck/infrastructure/analytics.py`: the v2 views and macros over the analytics parquet outputs, gated on file presence (`packages/atif-duck/src/atif_duck/infrastructure/analytics.py:119`). -- `packages/atif-duck/src/atif_duck/domain/examples.py` — `build_examples` derives one +- `packages/atif-duck/src/atif_duck/domain/examples.py`: `build_examples` derives one runnable query per catalog object from `ARG_EXEMPLARS` rather than hardcoding strings (`packages/atif-duck/src/atif_duck/domain/examples.py:191`). -- `packages/atif-duck/src/atif_duck/domain/authorship.py` — the `step_author` rule table: +- `packages/atif-duck/src/atif_duck/domain/authorship.py`: the `step_author` rule table: prefix matches after stripping leading whitespace, first rule wins, twinned in atif-analytics (`packages/atif-duck/src/atif_duck/domain/authorship.py:50`). -- `packages/atif-duck/src/atif_duck/infrastructure/authorship.py` — the `step_author` macro +- `packages/atif-duck/src/atif_duck/infrastructure/authorship.py`: the `step_author` macro and the `user_steps`, `human_turns`, and `session_outcomes` views, registered after the core macros and before the analytics surface (`packages/atif-duck/src/atif_duck/infrastructure/authorship.py:38`). -- `packages/atif-duck/src/atif_duck/domain/embedding_guard.py` — the read-side provider and +- `packages/atif-duck/src/atif_duck/domain/embedding_guard.py`: the read-side provider and dimension guard, a deliberate twin of atif-embed's copy because the independence contract forbids sharing it (`packages/atif-duck/src/atif_duck/domain/embedding_guard.py:5`). -- `packages/atif-duck/src/atif_duck/__init__.py` — re-exports the catalog constants and the +- `packages/atif-duck/src/atif_duck/__init__.py`: re-exports the catalog constants and the `register*` entry points (`packages/atif-duck/src/atif_duck/__init__.py:30`). -- `packages/atif-duck/src/atif_duck/domain/__init__.py` — the pure catalog layer, no duckdb +- `packages/atif-duck/src/atif_duck/domain/__init__.py`: the pure catalog layer, no duckdb import (`packages/atif-duck/src/atif_duck/domain/__init__.py:5`). -- `packages/atif-duck/src/atif_duck/infrastructure/__init__.py` — the registry layer's +- `packages/atif-duck/src/atif_duck/infrastructure/__init__.py`: the registry layer's re-export surface (`packages/atif-duck/src/atif_duck/infrastructure/__init__.py:5`). ## atif-cli The composition root: it declares its siblings as `==`-pinned dependencies -(`packages/atif-cli/pyproject.toml:32`) and wires every cross-package seam — the `ConverterPort` +(`packages/atif-cli/pyproject.toml:32`) and wires every cross-package seam: the `ConverterPort` adapter, the clock, version pins, the DuckDB connection (`packages/atif-cli/src/atif_cli/app.py:5`). The commands hang off one cyclopts `App`: `@app.command` functions from `convert` (`packages/atif-cli/src/atif_cli/app.py:226`) to `schema` (`:1055`), plus the `cron` sub-app registered at `:65`, with `main` (`:1093`) exposed as the single console script named `atif-sql` -(`packages/atif-cli/pyproject.toml:42`). Heavy imports — duckdb, harbor through atif-converter, -pydantic through atif-corpus — are deferred into the command bodies that use them so `schema`, +(`packages/atif-cli/pyproject.toml:42`). Heavy imports (duckdb, harbor through atif-converter, +pydantic through atif-corpus) are deferred into the command bodies that use them so `schema`, `--help`, and `--version` stay on a lean import graph that a fresh-interpreter test pins (`packages/atif-cli/src/atif_cli/app.py:22`). Failures resolve to stable exit codes, 64 for parse, 65 for catalog, 70 for runtime, split between a pure taxonomy module (`packages/atif-cli/src/atif_cli/errors.py:25`) and a driver-dependent classifier (`packages/atif-cli/src/atif_cli/duck_errors.py:34`) so the lean path never imports duckdb. -- `packages/atif-cli/src/atif_cli/app.py` — the commands and every wiring decision +- `packages/atif-cli/src/atif_cli/app.py`: the commands and every wiring decision between them (`packages/atif-cli/src/atif_cli/app.py:52`). -- `packages/atif-cli/src/atif_cli/lake.py` — the `lake` sub-app: `rebuild`, `verify`, +- `packages/atif-cli/src/atif_cli/lake.py`: the `lake` sub-app: `rebuild`, `verify`, `status` and `compact`, each deferring atif-duck's imports into the command body. -- `packages/atif-cli/src/atif_cli/corpus.py` — the `corpus` sub-app: `slim` converts an +- `packages/atif-cli/src/atif_cli/corpus.py`: the `corpus` sub-app: `slim` converts an old-layout corpus to the compressed layout, and `storage_layout` is the scan `status` reports. -- `packages/atif-cli/src/atif_cli/embed_lake.py` — `DuckLakeSteps`, atif-embed's `LakeStepsPort` +- `packages/atif-cli/src/atif_cli/embed_lake.py`: `DuckLakeSteps`, atif-embed's `LakeStepsPort` implemented over atif-duck's `lake_steps`, the one place the two meet. -- `packages/atif-cli/src/atif_cli/cron.py` — `cron install` prints a crontab block and never +- `packages/atif-cli/src/atif_cli/cron.py`: `cron install` prints a crontab block and never writes one; `cron status` reports per-lane lock and last-run state from injected probes (`packages/atif-cli/src/atif_cli/cron.py:7`). -- `packages/atif-cli/src/atif_cli/output.py` — `--format auto` resolves to a table on a TTY +- `packages/atif-cli/src/atif_cli/output.py`: `--format auto` resolves to a table on a TTY and JSON on a pipe (`packages/atif-cli/src/atif_cli/output.py:74`); `emit_cursor` streams `fetchmany` batches so the client holds one batch (`:154`). -- `packages/atif-cli/src/atif_cli/duck_errors.py` — the duckdb exception classifier, plus a +- `packages/atif-cli/src/atif_cli/duck_errors.py`: the duckdb exception classifier, plus a widened caught set for the registration path since `EmbeddingProviderMismatch` is not a `duckdb.Error` (`packages/atif-cli/src/atif_cli/duck_errors.py:31`). -- `packages/atif-cli/src/atif_cli/converter_adapter.py` — `RealConverter` adapts +- `packages/atif-cli/src/atif_cli/converter_adapter.py`: `RealConverter` adapts atif-converter's use case to atif-corpus's port, the one place allowed to import both (`packages/atif-cli/src/atif_cli/converter_adapter.py:44`). -- `packages/atif-cli/src/atif_cli/errors.py` — `EXIT_CODES` and the `ClassifiedError` shape, +- `packages/atif-cli/src/atif_cli/errors.py`: `EXIT_CODES` and the `ClassifiedError` shape, pure and free of any `atif_*` import (`packages/atif-cli/src/atif_cli/errors.py:52`). -- `packages/atif-cli/src/atif_cli/__main__.py` — `python -m atif_cli` reaching the same `main` +- `packages/atif-cli/src/atif_cli/__main__.py`: `python -m atif_cli` reaching the same `main` (`packages/atif-cli/src/atif_cli/__main__.py:5`). -- `packages/atif-cli/src/atif_cli/__init__.py` — package docstring only +- `packages/atif-cli/src/atif_cli/__init__.py`: package docstring only (`packages/atif-cli/src/atif_cli/__init__.py:3`). ## atif-embed @@ -156,7 +156,7 @@ for catalog, 70 for runtime, split between a pure taxonomy module The vector-search write path: Cohere Embed v4 on Bedrock behind `EmbeddingProvider`, a local LanceDB store behind `VectorStorePort`, and a corpus reader behind `TextRowsPort` (`packages/atif-embed/src/atif_embed/domain/ports.py:31`). `run_backfill` anti-joins step text against -the store's uuid-to-text-hash map, embeds the misses, and bounds loss in layers — chunked discovery, +the store's uuid-to-text-hash map, embeds the misses, and bounds loss in layers: chunked discovery, mid-run checkpoints, and per-batch isolation inside a chunk (`packages/atif-embed/src/atif_embed/application/embed.py:61`). Stamps keep the vector space honest: `model_id` and `dimension` on every row, checked on both the write and the read path so a @@ -171,28 +171,28 @@ kept in the store directory (`packages/atif-embed/src/atif_embed/infrastructure/lake_text_rows.py`), and falls back to the per-session reader otherwise. -- `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py` — connect, open or +- `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py`: connect, open or create, online-migrate, delete by predicate, append, index, and compact (`packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:195`). -- `packages/atif-embed/src/atif_embed/infrastructure/cohere_bedrock.py` — the `invoke_model` +- `packages/atif-embed/src/atif_embed/infrastructure/cohere_bedrock.py`: the `invoke_model` adapter under a tenacity retry with botocore retries off, and the document-`int8` / query-`float` asymmetry (`packages/atif-embed/src/atif_embed/infrastructure/cohere_bedrock.py:284`). -- `packages/atif-embed/src/atif_embed/application/embed.py` — `discover_unembedded` +- `packages/atif-embed/src/atif_embed/application/embed.py`: `discover_unembedded` (`packages/atif-embed/src/atif_embed/application/embed.py:43`), `run_backfill` (`:61`), and the sync `embed_query` the search command calls (`:265`). -- `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py` — reads the +- `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py`: reads the contract corpus layout with its own DuckDB connection, since importing atif-duck is forbidden (`packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:155`). -- `packages/atif-embed/src/atif_embed/domain/ports.py` — the Protocols: +- `packages/atif-embed/src/atif_embed/domain/ports.py`: the Protocols: `EmbeddingProvider` (`packages/atif-embed/src/atif_embed/domain/ports.py:31`), `VectorStorePort` (`:59`), `TextRowsPort` (`:87`). -- `packages/atif-embed/src/atif_embed/domain/embedding_guard.py` — `ensure_store_matches`, +- `packages/atif-embed/src/atif_embed/domain/embedding_guard.py`: `ensure_store_matches`, the fail-loud provider and dimension rule (`packages/atif-embed/src/atif_embed/domain/embedding_guard.py:46`). -- `packages/atif-embed/src/atif_embed/domain/errors.py` — the `DomainError` taxonomy the CLI +- `packages/atif-embed/src/atif_embed/domain/errors.py`: the `DomainError` taxonomy the CLI catches, with `terminal` separating operator-needed states from retryable ones (`packages/atif-embed/src/atif_embed/domain/errors.py:14`). -- `packages/atif-embed/src/atif_embed/domain/text_stamp.py` — `text_hash` +- `packages/atif-embed/src/atif_embed/domain/text_stamp.py`: `text_hash` (`packages/atif-embed/src/atif_embed/domain/text_stamp.py:38`) and the `MAX_EMBEDDABLE_CHARS` head-only cap (`:35`). @@ -200,7 +200,7 @@ per-session reader otherwise. One materialization pass is sweep, scan, plan, convert, write, advance watermark, and `materialize` is that pass (`packages/atif-corpus/src/atif_corpus/application/materialize.py:476`). The decision half -is pure — the domain never stats a file and never reads a clock, so `build_plan` partitions scanned +is pure: the domain never stats a file and never reads a clock, so `build_plan` partitions scanned sessions into to-materialize, up-to-date, and skipped-live deterministically from mtimes in epoch nanoseconds plus an injected now (`packages/atif-corpus/src/atif_corpus/domain/sessions.py:159`). The write half is atomic by construction: artifacts land in a temp session directory that is renamed into @@ -213,44 +213,44 @@ since this member may never import atif-converter module is the other way out: after the swaps, the pass hands it every session it published or marked, and records the ones it hasn't taken yet in `sink_pending.json` so a failed or killed hand-over is retried by the next pass. What counts as a transcript, and how deep -under the source root it sits, is one value object per agent — depth 1 for Claude Code's -`/.jsonl`, depth 3 for Codex's `//
` nesting -(`packages/atif-corpus/src/atif_corpus/domain/source_layout.py:119`) — so the scanner walks either +under the source root it sits, is one value object per agent (depth 1 for Claude Code's +`/.jsonl`, depth 3 for Codex's `//
` nesting; +`packages/atif-corpus/src/atif_corpus/domain/source_layout.py:119`), so the scanner walks either layout without branching on the agent, and `layout_for` refuses an agent that has none (`:131`). The agent enum itself is an AST-pinned twin of the converter's, since the independence contract forbids the import (`packages/atif-corpus/src/atif_corpus/domain/agents.py:25`). -- `packages/atif-corpus/src/atif_corpus/application/materialize.py` — the pass itself +- `packages/atif-corpus/src/atif_corpus/application/materialize.py`: the pass itself (`packages/atif-corpus/src/atif_corpus/application/materialize.py:476`), the report value object (`:118`), and the public `read_watermark` its consumers call (`:160`). -- `packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py` — the only place this +- `packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py`: the only place this member stats the source corpus, keeping a genuinely absent transcript separate from a failed `stat` so a transient error never deletes live artifacts (`packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py:15`). -- `packages/atif-corpus/src/atif_corpus/domain/sessions.py` — `SessionSource` +- `packages/atif-corpus/src/atif_corpus/domain/sessions.py`: `SessionSource` (`packages/atif-corpus/src/atif_corpus/domain/sessions.py:42`), `QuiescencePolicy` (`:71`), `MaterializationPlan` (`:108`), and `build_plan` (`:159`). -- `packages/atif-corpus/src/atif_corpus/infrastructure/atomic.py` — tmp-sibling write, +- `packages/atif-corpus/src/atif_corpus/infrastructure/atomic.py`: tmp-sibling write, fsync, rename, for files (plain and zstd-compressed) and whole directories. -- `packages/atif-corpus/src/atif_corpus/infrastructure/compress_artifacts.py` — compresses an +- `packages/atif-corpus/src/atif_corpus/infrastructure/compress_artifacts.py`: compresses an old-layout session's plain JSON artifacts in place for `corpus slim`, reading each copy back before the plain file is removed and keeping its mtime. -- `packages/atif-corpus/src/atif_corpus/domain/watermark.py` — `diff_source_mtimes` partitions +- `packages/atif-corpus/src/atif_corpus/domain/watermark.py`: `diff_source_mtimes` partitions two mtime maps into added, modified, and removed (`packages/atif-corpus/src/atif_corpus/domain/watermark.py:58`). -- `packages/atif-corpus/src/atif_corpus/domain/layout.py` — `CorpusLayout` is the single +- `packages/atif-corpus/src/atif_corpus/domain/layout.py`: `CorpusLayout` is the single writer-side computation of every contract path (`packages/atif-corpus/src/atif_corpus/domain/layout.py:26`). -- `packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py` — a scriptable +- `packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py`: a scriptable `ConverterPort` fake shipped in `infrastructure` so the type checker holds it to the port on every run (`packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py:3`). -- `packages/atif-corpus/src/atif_corpus/infrastructure/settings.py` — `ATIF_SQL_`-prefixed +- `packages/atif-corpus/src/atif_corpus/infrastructure/settings.py`: `ATIF_SQL_`-prefixed settings whose default factories read env at call time, not import time (`packages/atif-corpus/src/atif_corpus/infrastructure/settings.py:44`). ## atif-converter -`convert_and_audit` returns a conversion result paired with a loss report — the trajectory plus an +`convert_and_audit` returns a conversion result paired with a loss report: the trajectory plus an accounting of what upstream dropped (`packages/atif-converter/src/atif_converter/application/convert_and_audit.py:105`). The conversion is ours: `convert_claude_code_records`, a port of harbor 0.23.0's Claude Code converter built on the @@ -267,7 +267,7 @@ parity ORACLE, reached from the tests only (`packages/atif-converter/tests/harbo frozen to goldens and diffed against the live corpus. harbor is a dev dependency only: the ATIF data classes and validator the converters build on are vendored from harbor 0.23.0 in `packages/atif-converter/src/atif_converter/domain/atif/` and held to upstream by `packages/atif-converter/tests/test_vendored_atif.py`. The -known conversion gaps are types rather than prose — `FidelityGap` enumerates them +known conversion gaps are types rather than prose: `FidelityGap` enumerates them (`packages/atif-converter/src/atif_converter/domain/fidelity.py:44`) and a pure enrichment pass repairs some of them by re-running harbor's deterministic normalization order over the raw records (`packages/atif-converter/src/atif_converter/domain/enrichment.py:198`). Codex has its own set @@ -291,28 +291,28 @@ table doesn't price is NULL. litellm is a dev dependency, for the identity test covered model shapes from the vendored `model_prices.json` beside it; `cost_per_token` raises `UnpriceableModelError` for anything else, `priced_cost_per_token` turns that into `None`, and `has_pricing_entry` is Codex's table lookup. -- `packages/atif-converter/src/atif_converter/domain/enrichment.py` — `enrich_trajectory` +- `packages/atif-converter/src/atif_converter/domain/enrichment.py`: `enrich_trajectory` restores source uuids, the compact-summary flag, and the cache-creation total (`packages/atif-converter/src/atif_converter/domain/enrichment.py:198`). -- `packages/atif-converter/src/atif_converter/infrastructure/harbor_adapter.py` — the only +- `packages/atif-converter/src/atif_converter/infrastructure/harbor_adapter.py`: the only module importing harbor, which ships no `py.typed`, so the untyped surface stays contained (`packages/atif-converter/src/atif_converter/infrastructure/harbor_adapter.py:11`). -- `packages/atif-converter/src/atif_converter/infrastructure/raw_records.py` — the one raw +- `packages/atif-converter/src/atif_converter/infrastructure/raw_records.py`: the one raw reader the census, edges emitter, and enrichment share, with `mutated_files` as the race check (`packages/atif-converter/src/atif_converter/infrastructure/raw_records.py:146`). -- `packages/atif-converter/src/atif_converter/application/convert_and_audit.py` — the use +- `packages/atif-converter/src/atif_converter/application/convert_and_audit.py`: the use case (`packages/atif-converter/src/atif_converter/application/convert_and_audit.py:105`) and its refusal on any mid-window mutation (`:93`). -- `packages/atif-converter/src/atif_converter/domain/fidelity.py` — `RecordType` +- `packages/atif-converter/src/atif_converter/domain/fidelity.py`: `RecordType` (`packages/atif-converter/src/atif_converter/domain/fidelity.py:18`), `FidelityGap` (`:44`), and the `LossReport` value object (`:83`). -- `packages/atif-converter/src/atif_converter/domain/edges.py` — one `edges.jsonl` line per +- `packages/atif-converter/src/atif_converter/domain/edges.py`: one `edges.jsonl` line per raw record, key order fixed by the contract (`packages/atif-converter/src/atif_converter/domain/edges.py:22`). -- `packages/atif-converter/src/atif_converter/infrastructure/census.py` — the raw-side counts +- `packages/atif-converter/src/atif_converter/infrastructure/census.py`: the raw-side counts that form the input half of a loss report, derived from an already-parsed snapshot (`packages/atif-converter/src/atif_converter/infrastructure/census.py:57`). -- `packages/atif-converter/src/atif_converter/domain/errors.py` — the typed exceptions under +- `packages/atif-converter/src/atif_converter/domain/errors.py`: the typed exceptions under one `DomainError` base, with exit-code mapping left to the CLI (`packages/atif-converter/src/atif_converter/domain/errors.py:14`). @@ -320,8 +320,8 @@ table doesn't price is NULL. litellm is a dev dependency, for the identity test A deliberately narrow seam: a system prompt, a user prompt, and a pydantic schema in; a validated instance of that schema out (`packages/atif-models/src/atif_models/domain/ports.py:116`). Its only -in-repo consumer is atif-analytics — the single edge the `forbidden` contract leaves open -(`pyproject.toml:520`) — and the registry is the only place in the workspace where a Bedrock model id +in-repo consumer is atif-analytics, the single edge the `forbidden` contract leaves open +(`pyproject.toml:520`), and the registry is the only place in the workspace where a Bedrock model id is written down, so a pipeline names a family and a size alias and lets `resolve` pick the id (`packages/atif-models/src/atif_models/domain/registry.py:6`, `:109`). The default adapter posts an OpenAI chat-completions body to `invoke_model` in strict `json_schema` mode, dispatching the blocking @@ -332,27 +332,27 @@ botocore's own retries disabled pure transform enforces before the schema reaches the wire (`packages/atif-models/src/atif_models/domain/schema.py:40`). -- `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py` — +- `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py`: `OpenAiBedrockProvider` (`packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:142`), the retryable-code set (`:63`), and usage extraction (`:124`). -- `packages/atif-models/src/atif_models/domain/registry.py` — `ModelSpec` +- `packages/atif-models/src/atif_models/domain/registry.py`: `ModelSpec` (`packages/atif-models/src/atif_models/domain/registry.py:39`), `resolve` (`:109`), and `estimate_cost` over per-1M-token rates (`:125`). -- `packages/atif-models/src/atif_models/domain/ports.py` — the `LlmStructuredProvider` +- `packages/atif-models/src/atif_models/domain/ports.py`: the `LlmStructuredProvider` Protocol (`packages/atif-models/src/atif_models/domain/ports.py:116`), the terminal-versus-retryable error split (`:43`), and lock-guarded usage accumulation (`:78`). -- `packages/atif-models/src/atif_models/domain/schema.py` — `to_openai_strict` keeps `$defs` +- `packages/atif-models/src/atif_models/domain/schema.py`: `to_openai_strict` keeps `$defs` and `$ref`, drops `default` and `title` (`packages/atif-models/src/atif_models/domain/schema.py:40`). -- `packages/atif-models/src/atif_models/infrastructure/settings.py` — `LlmSettings` carries +- `packages/atif-models/src/atif_models/infrastructure/settings.py`: `LlmSettings` carries the family, region, concurrency, and per-pipeline size overrides (`packages/atif-models/src/atif_models/infrastructure/settings.py:28`). -- `packages/atif-models/src/atif_models/__init__.py` — states the ownership rule: no other +- `packages/atif-models/src/atif_models/__init__.py`: states the ownership rule: no other member hardcodes a model id (`packages/atif-models/src/atif_models/__init__.py:9`). -- `packages/atif-models/src/atif_models/domain/__init__.py` — the pure layer: no boto3, no +- `packages/atif-models/src/atif_models/domain/__init__.py`: the pure layer: no boto3, no env, no clock (`packages/atif-models/src/atif_models/domain/__init__.py:7`). -- `packages/atif-models/src/atif_models/infrastructure/__init__.py` — adapters and env +- `packages/atif-models/src/atif_models/infrastructure/__init__.py`: adapters and env settings, importing downward into `domain` only (`packages/atif-models/src/atif_models/infrastructure/__init__.py:7`). diff --git a/docs/architecture/system-overview.md b/docs/architecture/system-overview.md index 1c0607a..d30e8c1 100644 --- a/docs/architecture/system-overview.md +++ b/docs/architecture/system-overview.md @@ -5,7 +5,7 @@ `atif-sql` is a command-line analytics tool over agent trajectories from Claude Code and Codex CLI. It reads the session transcripts Claude Code leaves at `~/.claude/projects/**/*.jsonl` and the rollouts Codex CLI leaves at `~/.codex/sessions///
/rollout-*.jsonl`, converts each one to -ATIF — Harbor's Agent Trajectory Interchange Format — materializes the results as an on-disk +ATIF (Harbor's Agent Trajectory Interchange Format), materializes the results as an on-disk corpus, and answers SQL against that corpus through DuckDB (`README.md:12`). The design bet is stated in the README itself: converting once at the boundary, with an explicit and tested fidelity policy for what the upstream converter drops, beats re-deriving trajectory semantics inside every @@ -14,13 +14,13 @@ moves the default source and corpus roots with it (`packages/atif-corpus/src/atif_corpus/infrastructure/settings.py`); one corpus holds one agent, and the `sessions` view reports which (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:428`). The reader it serves is an engineer or an agent asking how sessions -actually went — which tools ran, where tokens went, where a session turned into friction. +actually went: which tools ran, where tokens went, where a session turned into friction. Users get one installable distribution and one console script, `atif-sql = "atif_cli.app:main"` (`packages/atif-cli/pyproject.toml:42`), installed with `uv tool install atif-sql` (`README.md:26`). The command surface is the commands registered with `@app.command` in `packages/atif-cli/src/atif_cli/app.py` plus the `cron` sub-app -attached at `:65`. Nothing here is a server — DuckDB is embedded +attached at `:65`. Nothing here is a server; DuckDB is embedded (`packages/atif-duck/pyproject.toml:20`), as are the SQLite analytics state file and the LanceDB vector store. The commands `analyze` / `embed` / `search` call Amazon Bedrock and spend money per invocation, and each is dry-run by default (`README.md:37`). @@ -111,7 +111,7 @@ process exit codes at the CLI edge (`:38`). ## Module map Nodes are the uv workspace members. Every edge is an import confirmed at an import site. -There is no `atif-corpus` to `atif-converter` edge — the independence contract forbids it, and +There is no `atif-corpus` to `atif-converter` edge: the independence contract forbids it, and `RealConverter` in `packages/atif-cli/src/atif_cli/converter_adapter.py:44` satisfies `ConverterPort` by importing both from the composition root (`:36`, `:38`). `atif-duck`, `atif-analytics`, and `atif-embed` exchange data through corpus files on disk, never through an diff --git a/docs/behavior/processes.md b/docs/behavior/processes.md index 653d560..78adbaf 100644 --- a/docs/behavior/processes.md +++ b/docs/behavior/processes.md @@ -3,13 +3,13 @@ Every process in this system starts as a CLI invocation. There is one console script, `atif-sql = "atif_cli.app:main"` (`packages/atif-cli/pyproject.toml:42`), and its commands are the complete -initiator set — the `@app.command` sites plus the `cron` sub-App registered at +initiator set: the `@app.command` sites plus the `cron` sub-App registered at `packages/atif-cli/src/atif_cli/app.py:66`. No HTTP route, RPC tool, message handler, or job queue exists to initiate anything else; the only scheduled initiator is a crontab line into `scripts/atif-sql-refresh.sh`. Long-running work is in-process async under `anyio` and `asyncio.run`, not a worker. -## materialize — sync the materialized corpus +## materialize: sync the materialized corpus Entry point: `packages/atif-cli/src/atif_cli/app.py:352` @@ -17,18 +17,18 @@ Entry point: `packages/atif-cli/src/atif_cli/app.py:352` `--corpus-root` re-derives the corpus root from the overridden source's slug so re-pointing the source cannot overwrite another corpus, and the CLI hands in the wall clock plus the harbor and converter version pins alongside - `RealConverter` behind `ConverterPort` — `:190`, - `packages/atif-cli/src/atif_cli/converter_adapter.py:44`. + `RealConverter` behind `ConverterPort` (`:190`, + `packages/atif-cli/src/atif_cli/converter_adapter.py:44`). 2. Sweep `.staging/`: an entry whose `tmp-` owner is a dead pid is crash debris and is removed, while a live pid belongs to a concurrent pass and is - left alone — - `packages/atif-corpus/src/atif_corpus/application/materialize.py:274`. + left alone + (`packages/atif-corpus/src/atif_corpus/application/materialize.py:274`). 3. Read `watermark.json` as a path-to-mtime map; an unreadable or wrongly shaped file degrades to empty, costing one full re-materialization rather - than refusing to sync — `packages/atif-corpus/src/atif_corpus/application/materialize.py:160`. + than refusing to sync: `packages/atif-corpus/src/atif_corpus/application/materialize.py:160`. 4. Scan the source root for main transcripts and every side-file, then fold sessions living under a directory that would not list into the unreadable - set using the watermark as the only record of what lived there — + set using the watermark as the only record of what lived there: `packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py:143`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:342`. 5. Guard, then retain sourceless sessions: zero scanned sessions over a @@ -39,22 +39,22 @@ Entry point: `packages/atif-cli/src/atif_cli/app.py:352` not evidence of deletion (`_retain_sourceless_sessions`). 6. Build the pure `MaterializationPlan` from quiescence and the watermark, force-replanning sessions the watermark calls current whose artifact - directory is missing — the one state reachable by a kill inside the swap - window — `packages/atif-corpus/src/atif_corpus/domain/sessions.py:159`, + directory is missing (the one state reachable by a kill inside the swap + window): `packages/atif-corpus/src/atif_corpus/domain/sessions.py:159`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:313`. 7. Per planned session: convert, write trajectory, loss report, edges, then run the `ArtifactProducer` (the typed columnar parquet files, unless `--no-columnar`), and `meta.json` last into a staging directory, then rename the whole directory into `sessions//` so a reader observes only a complete generation; a - session that raises is recorded and the pass continues — `packages/atif-corpus/src/atif_corpus/application/materialize.py:228`, - `packages/atif-corpus/src/atif_corpus/infrastructure/atomic.py:98`. + session that raises is recorded and the pass continues (`packages/atif-corpus/src/atif_corpus/application/materialize.py:228`, + `packages/atif-corpus/src/atif_corpus/infrastructure/atomic.py:98`). With `--workers` above 1 this step runs on a spawn-context process pool; each worker runs the same per-session function under its own pid, outcomes - are folded back in plan order, and a single planned session runs inline — + are folded back in plan order, and a single planned session runs inline: `packages/atif-corpus/src/atif_corpus/application/materialize.py:385`. 8. Advance the watermark for succeeded sessions only, retaining every entry of a failed, unplanned, or unreadable session so staleness still signals a - retry, and write it atomically before emitting the report — + retry, and write it atomically before emitting the report: `packages/atif-corpus/src/atif_corpus/application/materialize.py:428`, `packages/atif-corpus/src/atif_corpus/infrastructure/atomic.py:64`. @@ -66,12 +66,12 @@ Entry point: `packages/atif-cli/src/atif_cli/app.py:352` - `packages/atif-corpus/src/atif_corpus/infrastructure/atomic.py:43` - `packages/atif-cli/src/atif_cli/app.py:312` -## convert — one session to ATIF plus a loss audit +## convert: one session to ATIF plus a loss audit Entry point: `packages/atif-cli/src/atif_cli/app.py:226` 1. Reject a path that is not an existing `.jsonl` file before any harbor work, - which the CLI maps to exit 64 — + which the CLI maps to exit 64: `packages/atif-converter/src/atif_converter/infrastructure/harbor_adapter.py:163`. 2. Read every source file once: the main transcript and each discovered side-file under `/`, workflow-nested ones included. Each file is @@ -119,32 +119,32 @@ the audit never read the session separately. - `packages/atif-converter/src/atif_converter/domain/fidelity.py:44` - `packages/atif-cli/src/atif_cli/errors.py:25` -## analyze — orchestrate the analytics stages +## analyze: orchestrate the analytics stages Entry point: `packages/atif-cli/src/atif_cli/app.py:1240` 1. Resolve `AnalyticsSettings`, reusing the corpus-root resolution the other commands share, then let `--max-sessions` and `--max-cost-usd` override the - env ceilings so a crontab line carries its spend cap visibly — `:1307`. + env ceilings so a crontab line carries its spend cap visibly (`:1307`). `--llm-only` is accepted and dropped, since every stage is an LLM stage. 2. Open the lake's session source when the lake holds the corpus, else warn - once and read the files (`--no-lake` forces the files) — + once and read the files (`--no-lake` forces the files): `packages/atif-cli/src/atif_cli/app.py:1624`. Build one `CorpusReader` over it, shared by every stage: the steps memo is the expensive part and every - stage walks the same sessions — - `packages/atif-analytics/src/atif_analytics/application/analyze.py:62`, - `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:263`. + stage walks the same sessions + (`packages/atif-analytics/src/atif_analytics/application/analyze.py:62`, + `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:263`). 3. Construct the run-wide `RunBudget` from `llm_max_cost_usd_per_run`, priced - from the providers' running actual usage rather than estimates — + from the providers' running actual usage rather than estimates: `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:94`. 4. Walk the stages (classify, conflicts, friction, perceived) in declaration order. A stage entered with the budget already exhausted is skipped with nothing stamped, its consecutive-skip streak persisted, and the - log escalates to ERROR at three consecutive runs — + log escalates to ERROR at three consecutive runs: `packages/atif-analytics/src/atif_analytics/application/analyze.py:97`, `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:218`. 5. Emit the per-stage summary carrying `budget_exhausted` and, on a real run, - `llm_spent_usd` — + `llm_spent_usd`: `packages/atif-analytics/src/atif_analytics/application/analyze.py:144`, `packages/atif-cli/src/atif_cli/app.py:1322`. @@ -155,7 +155,7 @@ Entry point: `packages/atif-cli/src/atif_cli/app.py:1240` - `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:36` - `packages/atif-analytics/src/atif_analytics/infrastructure/parquet_cache.py:90` -## classify — the LLM analytics stage shape +## classify: the LLM analytics stage shape Entry point: `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:368` @@ -165,40 +165,39 @@ admit only `interactive` sessions; friction and perceived read only human turns, the main-chain user steps whose author is `human`. 1. Resolve the layout, the reader, and the sharded parquet cache, then resolve - the model through the atif-models registry by pipeline size — no pipeline - writes down a model id — `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:368`, + the model through the atif-models registry by pipeline size (no pipeline + writes down a model id): `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:368`, `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:32`. 2. Under `dry_run`, the default, return a plan dict whose input tokens are measured from the actually rendered transcripts and whose session count is - the same newest-first cap the real run applies — + the same newest-first cap the real run applies: `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:293`. 3. Anti-join the cache for session ids already written, then drop sessions - whose last-step timestamp and mtime bounds are unchanged since the last run - — `:74`, - `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:136`. + whose last-step timestamp and mtime bounds are unchanged since the last run (`:74`, + `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:136`). 4. Drain the retry queue back into the admission set, then subtract units with - a live entry — inside backoff or past the attempt cap — because the retry + a live entry (inside backoff or past the attempt cap) because the retry queue is the single re-admission gate and the checkpoint path would re-bill - them — + them: `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/retry_queue.py:145`, `:159`. 5. Walk newest-first and stop admitting at `llm_max_sessions_per_run` so a deferred session is never rendered; a session that renders to nothing, or whose kind isn't `interactive`, is checkpointed at current bounds instead of - re-rendered every tick — + re-rendered every tick: `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:146`. 6. Dispatch each write chunk in budget-checked sub-batches of `BUDGET_CHECK_BATCH`, advancing the cursor by what was actually sent so a mid-chunk budget stop leaves the remainder unstamped rather than silently - skipped — + skipped: `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:155`, `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:217`. 7. Route each result: a `RefusalError` is terminal and writes the documented `goal='[refused]'` sentinel row so the refusal is queryable and never - re-billed, while any other exception enqueues a retry — - `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:219`. + re-billed, while any other exception enqueues a retry + (`packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:219`). 8. Write the chunk as a parquet part, checkpoint the completed sessions at - current bounds, and clear them from the retry queue — + current bounds, and clear them from the retry queue: `packages/atif-analytics/src/atif_analytics/infrastructure/parquet_cache.py:268`, `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:272`. @@ -210,53 +209,53 @@ turns, the main-chain user steps whose author is `human`. - `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:191` - `packages/atif-models/src/atif_models/domain/ports.py:116` -## embed — backfill step embeddings into LanceDB +## embed: backfill step embeddings into LanceDB Entry point: `packages/atif-cli/src/atif_cli/app.py:766` 1. Refuse a real run with no scope: without `--limit`, `--all`, or `--dry-run` the command exits 64, so a mistyped invocation cannot start a - full backfill — `:778`. + full backfill (`:778`). 2. Resolve the Lance URI from `EmbedSettings`, defaulting to - `/embeddings_lance` — - `packages/atif-embed/src/atif_embed/infrastructure/settings.py:51`. + `/embeddings_lance` + (`packages/atif-embed/src/atif_embed/infrastructure/settings.py:51`). 3. Read the store's uuid-to-text-hash map once, then stream candidates whose hash is absent or stale through `TextRowsPort`; the staleness comparison happens before the limit cap so `--limit N` always makes N rows of progress - — `packages/atif-embed/src/atif_embed/domain/discovery.py`. With a lake + (`packages/atif-embed/src/atif_embed/domain/discovery.py`). With a lake that holds the corpus, the rows come from its `steps` table (`packages/atif-embed/src/atif_embed/infrastructure/lake_text_rows.py`, over `packages/atif-duck/src/atif_duck/infrastructure/lake_steps.py`), and only the uuids a lake snapshot touched since the watermark in the store directory are read; otherwise every session's `trajectory.json` is - read — `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py`. + read (`packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py`). 4. Under `--dry-run`, count candidates and return the plan dict, returning - before boto3 is ever imported — + before boto3 is ever imported: `packages/atif-embed/src/atif_embed/application/embed.py:120`. 5. Build the Cohere-on-Bedrock provider once, then refuse to append when the store's stamped model and dimension differ from the live embedder's, because - mixing vector spaces corrupts kNN silently — `:157`, - `packages/atif-embed/src/atif_embed/domain/embedding_guard.py:38`. + mixing vector spaces corrupts kNN silently (`:157`, + `packages/atif-embed/src/atif_embed/domain/embedding_guard.py:38`). 6. Per chunk, a multiple of the batch size so a checkpoint boundary never splits a Bedrock batch: embed the texts and drop rows the provider failed, - leaving them pending for the next run — - `packages/atif-embed/src/atif_embed/application/embed.py:147`, - `packages/atif-embed/src/atif_embed/infrastructure/cohere_bedrock.py:309`. + leaving them pending for the next run + (`packages/atif-embed/src/atif_embed/application/embed.py:147`, + `packages/atif-embed/src/atif_embed/infrastructure/cohere_bedrock.py:309`). 7. Delete stale rows under the same uuid before appending the fixed-size float32 array frame, so text that changed cannot leave two rows fanning the - kNN join out — `packages/atif-embed/src/atif_embed/application/embed.py:207`, + kNN join out: `packages/atif-embed/src/atif_embed/application/embed.py:207`, `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:402`. 8. Compact every eight chunks to bound fragment count, then optimize and - ensure the HNSW index on the way out so `search` pays no brute-force scan — + ensure the HNSW index on the way out so `search` pays no brute-force scan: `packages/atif-embed/src/atif_embed/application/embed.py:243`, `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:414`. 9. When no row failed, hand the store's row count to `TextRowsPort.commit`: the lake reader moves its watermark to the snapshot it read, unless the - run stopped at `--limit` — `packages/atif-embed/src/atif_embed/application/embed.py`. + run stopped at `--limit` (`packages/atif-embed/src/atif_embed/application/embed.py`). `embed --prune-orphans` is a separate path: it compares the store's uuids with every step's primary uuid in the lake and reports (or, with `--no-dry-run`, -deletes) the ones no step names — `packages/atif-embed/src/atif_embed/application/prune.py`. +deletes) the ones no step names (`packages/atif-embed/src/atif_embed/application/prune.py`). ### Related @@ -266,26 +265,26 @@ deletes) the ones no step names — `packages/atif-embed/src/atif_embed/applicat - `packages/atif-embed/src/atif_embed/domain/errors.py:29` - `packages/atif-cli/src/atif_cli/app.py:833` -## query — one SQL statement over the catalog +## query: one SQL statement over the catalog Entry point: `packages/atif-cli/src/atif_cli/app.py:529` 1. Short-circuit `--examples` to the examples listing without opening DuckDB; - a missing statement emits a classified parse error and exits 64 — `:564`. + a missing statement emits a classified parse error and exits 64 (`:564`). 2. Resolve the corpus root and the expected embedder identity, then open an - in-memory DuckDB connection — `:586`. + in-memory DuckDB connection (`:586`). 3. Register the raw readers over the contract layout (trajectory rows come from `read_parquet` for sessions with current columnar artifacts and from `read_json` for the rest, unioned per surface), gating trajectory, edges, and loss on `meta.json` presence so a torn session - directory contributes nothing to any view — + directory contributes nothing to any view: `packages/atif-duck/src/atif_duck/infrastructure/registry.py:180`. 4. Create the core views, then bind `message_embeddings` over the Lance store guard-before-bind: a store written by another provider raises rather than - returning numerically valid garbage cosine scores — `:333`, `:846`. + returning numerically valid garbage cosine scores (`:333`, `:846`). 5. Create the macros, then the analytics views and analytics macros over the analytics parquets, which bind against both the parquets and the base views - — `:1005`, `packages/atif-duck/src/atif_duck/infrastructure/analytics.py:124`. + (`:1005`, `packages/atif-duck/src/atif_duck/infrastructure/analytics.py:124`). 6. Harden the connection in a fixed order (the memory cap, thread count and private spill directory were set before step 3, since registration is what needs them): a directory allowlist holding only the spill area, a file @@ -298,9 +297,9 @@ Entry point: `packages/atif-cli/src/atif_cli/app.py:529` caller's statement and stream the cursor: a plain table on a TTY, a JSON array of row objects on a pipe (`packages/atif-cli/src/atif_cli/output.py:154`). The spill directory is removed on exit. -8. Classify any DuckDB failure into parse, catalog, or runtime — or an - embedding-provider mismatch — and exit 64, 65, or 70 with a JSON error - envelope — `packages/atif-cli/src/atif_cli/duck_errors.py:34`, `:66`. +8. Classify any DuckDB failure into parse, catalog, or runtime (or an + embedding-provider mismatch) and exit 64, 65, or 70 with a JSON error + envelope: `packages/atif-cli/src/atif_cli/duck_errors.py:34`, `:66`. ### Related @@ -310,39 +309,39 @@ Entry point: `packages/atif-cli/src/atif_cli/app.py:529` - `packages/atif-cli/src/atif_cli/output.py:202` - `packages/atif-cli/src/atif_cli/errors.py:25` -## search — semantic top-k over step embeddings +## search: semantic top-k over step embeddings Entry point: `packages/atif-cli/src/atif_cli/app.py:860` 1. Resolve the corpus root, the Lance URI, and the expected model and - dimension from `EmbedSettings` — `:878`, - `packages/atif-embed/src/atif_embed/infrastructure/settings.py:57`. + dimension from `EmbedSettings` (`:878`, + `packages/atif-embed/src/atif_embed/infrastructure/settings.py:57`). 2. Attach the lake as `query` does (or fall back to the per-session artifacts with a warning), then register the full catalog on a fresh in-memory connection; a registration failure or a provider mismatch leaves - through the classified-error path with exit 65 — + through the classified-error path with exit 65: `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1212`, `packages/atif-cli/src/atif_cli/duck_errors.py:66`. `--all-corpora` scopes the steps to every corpus the lake holds and binds `message_embeddings` over each corpus's store (`register_vss_stores`). 3. Count `message_embeddings` first: an empty or absent store exits 2 with the backfill hint instead of returning an empty result that reads like "no - matches" — `packages/atif-cli/src/atif_cli/app.py:930`. -4. Embed the query text in Cohere `search_query` float mode — Cohere forces - float for queries even when documents were stored int8 — + matches" (`packages/atif-cli/src/atif_cli/app.py:930`). +4. Embed the query text in Cohere `search_query` float mode (Cohere forces + float for queries even when documents were stored int8): `packages/atif-embed/src/atif_embed/application/embed.py:265`, `packages/atif-embed/src/atif_embed/infrastructure/cohere_bedrock.py:400`. 5. Bind the query vector, the optional session filter, and `k` as parameters - rather than interpolating them — `packages/atif-cli/src/atif_cli/app.py:945`. + rather than interpolating them: `packages/atif-cli/src/atif_cli/app.py:945`. 6. Order by `array_cosine_distance` ascending: that is what triggers the cosine HNSW index lookup, and cosine is the only magnitude-invariant choice against - int8-cast document vectors whose magnitudes run into the thousands — `:919`. + int8-cast document vectors whose magnitudes run into the thousands (`:919`). 7. Join back to `steps` on the first entry of `source_uuids` for a - 200-character snippet, selecting cosine similarity as the reported score — - `:933`. + 200-character snippet, selecting cosine similarity as the reported score + (`:933`). 8. Emit uuid, session id, similarity, and snippet; a DuckDB error classifies to - its own exit code — `:946`, - `packages/atif-cli/src/atif_cli/output.py:129`. + its own exit code (`:946`, + `packages/atif-cli/src/atif_cli/output.py:129`). ### Related @@ -351,40 +350,40 @@ Entry point: `packages/atif-cli/src/atif_cli/app.py:860` - `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:414` - `packages/atif-cli/src/atif_cli/errors.py:25` -## refresh lane — the scheduled driver +## refresh lane: the scheduled driver Entry point: `scripts/atif-sql-refresh.sh:109` 1. Pin the identity trio once from the invoking environment and unset `AWS_PROFILE` and `AWS_DEFAULT_PROFILE`, because an inherited profile name with no matching config entry makes botocore raise at credential resolution - before Bedrock is reached — `:84`, `:96`. + before Bedrock is reached (`:84`, `:96`). 2. Normalize the lane argument before it names a lock file, so two spellings of one lane cannot take two locks; an unknown mode exits 64 rather than defaulting to a plane. The removed `structural` lane (and `struct`) still - parses: it logs one "lane removed" line and exits 0 — `:138`, `:143`. + parses: it logs one "lane removed" line and exits 0 (`:138`, `:143`). 3. Read the rotated Bedrock bearer token at run time and export it only when non-empty, because botocore treats an empty value as a real broken - credential and skips the default chain — `:122`. -4. Resolve the corpus list — the primary config directory first, then each - `ATIF_SQL_EXTRA_CONFIG_DIRS` entry — and export both `CLAUDE_CONFIG_DIR` and + credential and skips the default chain (`:122`). +4. Resolve the corpus list (the primary config directory first, then each + `ATIF_SQL_EXTRA_CONFIG_DIRS` entry) and export both `CLAUDE_CONFIG_DIR` and `ATIF_SQL_SOURCE_ROOT` per corpus so the source root and the corpus slug - stay coherent — `:134`, `:312`. + stay coherent (`:134`, `:312`). 5. Resolve the CLI through the override, then a user install, then the workspace venv script, and take a nonblocking per-lane `flock`: a busy lane - skips this tick rather than queueing behind it — `:151`, `:166`. + skips this tick rather than queueing behind it (`:151`, `:166`). 6. On the `llm` lane, probe `atif-sql --help` for `analyze` and exit 0 when it is absent, so an armed crontab line against an older CLI is a no-op instead - of a nightly error — `:260`. + of a nightly error (`:260`). 7. Run the lane per corpus: `materialize` plus a bounded `embed --limit 500` piggyback, or - `analyze --no-dry-run --llm-only --max-sessions 1000000 --max-cost-usd 1000000` - — the only spending line in the file, with both caps set out of reach on - purpose — `:334`, `:371`. + `analyze --no-dry-run --llm-only --max-sessions 1000000 --max-cost-usd 1000000`, + the only spending line in the file, with both caps set out of reach on + purpose (`:334`, `:371`). 8. On embed exit 78, write a terminal marker keyed on the store path and its mtime and suppress further embeds for that corpus until the mtime changes; - every other nonzero exit stays transient and retries next tick — `:217`, - `:244`. + every other nonzero exit stays transient and retries next tick (`:217`, + `:244`). 9. The `compact` lane runs `atif-sql lake compact --memory-limit 2GiB` once (not per corpus; `ATIF_SQL_REFRESH_COMPACT_MEMORY_LIMIT` changes the budget) under a 4G scope, after taking the materialize lane's lock too (waiting up to @@ -400,26 +399,26 @@ Entry point: `scripts/atif-sql-refresh.sh:109` ## Minor flows -- status — entry at `packages/atif-cli/src/atif_cli/app.py:444`. Read-only +- status: entry at `packages/atif-cli/src/atif_cli/app.py:444`. Read-only freshness report: scans source mtimes and replays the same pure plan `materialize` would build without converting anything, so its stale, up-to-date, and live counts are exactly what a pass would do (`packages/atif-corpus/src/atif_corpus/domain/sessions.py:159`). -- examples — entry at `packages/atif-cli/src/atif_cli/app.py:995`. Derives one +- examples: entry at `packages/atif-cli/src/atif_cli/app.py:995`. Derives one runnable example per view and macro from the static catalog and filters by `--category` and `--requires`; an unknown value exits 64 (`packages/atif-duck/src/atif_duck/domain/examples.py:191`). -- schema — entry at `packages/atif-cli/src/atif_cli/app.py:1727`. Dumps the +- schema: entry at `packages/atif-cli/src/atif_cli/app.py:1727`. Dumps the core and analytics view schemas and macro signatures, each with its `requires` value, from the static catalog with no DuckDB import and no connection (`packages/atif-duck/src/atif_duck/domain/catalog.py:56`). -- cron install — entry at `packages/atif-cli/src/atif_cli/cron.py:185`. Renders +- cron install: entry at `packages/atif-cli/src/atif_cli/cron.py:185`. Renders the crontab block, plus one comment per removed lane, for a human to paste and never writes the crontab itself (`:79`). -- cron status — entry at `packages/atif-cli/src/atif_cli/cron.py:198`. Probes +- cron status: entry at `packages/atif-cli/src/atif_cli/cron.py:198`. Probes each lane's flock nonblocking, so the probe perturbs nothing, and parses the last completion and last skip per lane out of the refresh log (`:91`, `:109`). -- main — entry at `packages/atif-cli/src/atif_cli/app.py:1125`. Replaces +- main: entry at `packages/atif-cli/src/atif_cli/app.py:1125`. Replaces loguru's default DEBUG sink with WARNING-and-up so routine reads keep stderr quiet, then hands control to cyclopts. diff --git a/docs/behavior/state-machines.md b/docs/behavior/state-machines.md index 34960cc..b636c4c 100644 --- a/docs/behavior/state-machines.md +++ b/docs/behavior/state-machines.md @@ -3,8 +3,8 @@ No entity in this workspace carries a `status` enum with declared transitions. The enums the workspace declares aren't lifecycles either. `RecordType` (`packages/atif-converter/src/atif_converter/domain/fidelity.py:18`) and `FidelityGap` (`:44`) are closed classification vocabularies whose members never move into one -another. `OutputFormat` (`packages/atif-cli/src/atif_cli/output.py:58`) has one resolution — -`resolve_format` maps `AUTO` to `TABLE` on a TTY and `JSON` otherwise (`:79-81`) — but that is an +another. `OutputFormat` (`packages/atif-cli/src/atif_cli/output.py:58`) has one resolution: +`resolve_format` maps `AUTO` to `TABLE` on a TTY and `JSON` otherwise (`:79-81`), but that is an idempotent pure function evaluated per emit, a no-op for every explicit format (`:74-80`), whose result is never stored and never advances again. One irreversible resolution with no persisted state is not a lifecycle. @@ -16,7 +16,7 @@ names and transition labels are verbatim source text from each machine's `Define ## Corpus session materialization Every pass over the raw transcript corpus reclassifies each discovered session from scratch, so the -state is not a stored field — it is the pair "what the watermark records" and "whether the session's +state is not a stored field; it is the pair "what the watermark records" and "whether the session's artifact directory exists". The classification is the `if / elif / else` at `packages/atif-corpus/src/atif_corpus/domain/sessions.py:200-206`, and the outcome states are the counters and id tuples of `MaterializationReport` @@ -40,7 +40,7 @@ stateDiagram-v2 - `skipped_live` is the entry state, not an error state. A transcript is appended to many times per turn, and a session qualifies to convert only once its newest source mtime is at least - `quiesce_seconds` old — 300 by default (`packages/atif-corpus/src/atif_corpus/domain/sessions.py:81`). + `quiesce_seconds` old, 300 by default (`packages/atif-corpus/src/atif_corpus/domain/sessions.py:81`). `force` overrides staleness but never quiescence, because converting a half-written transcript produces a wrong artifact rather than a stale one (`:170-174`). - `failures --> to_materialize` is the retry, and it works only because `_advance_watermark` retains @@ -48,7 +48,7 @@ stateDiagram-v2 entry *is* the retry signal; dropping it would classify the session `up_to_date` forever (`packages/atif-corpus/src/atif_corpus/application/materialize.py:435-448`). - `up_to_date --> to_materialize: _unmaterialized_session_ids` is the crash-recovery edge. It is - reachable exactly one way — a pass killed inside the directory swap, after the previous generation + reachable exactly one way: a pass killed inside the directory swap, after the previous generation was renamed aside and before the new one landed. The watermark records source mtimes only and cannot express a missing artifact directory, so the directory check is what force-replans it (`:302-328`). - `unreadable_session_ids` is a parking state, and the only state a session can starve in. A transient @@ -87,7 +87,7 @@ stateDiagram-v2 `truncated`; `unrecorded` is a store whose columns are already current but whose sidecar is absent, which `read_schema_version` reports as `None` (`:141-154`). - `v1 --> v2` is online and metadata-only. `migrate_pre_stamp_table` adds the two columns through - Lance schema evolution with SQL default expressions — no rows dropped, no vectors re-embedded, and + Lance schema evolution with SQL default expressions: no rows dropped, no vectors re-embedded, and readers keep working throughout (`:164-190`). `text_hash` backfills to `_PRE_STAMP_SENTINEL`, the literal `` (`:71`), whose angle brackets make it impossible to equal a real blake2b digest; every migrated row therefore mismatches its corpus hash in the discovery anti-join and @@ -95,7 +95,7 @@ stateDiagram-v2 - `v2 --> v2: _has_table` is idempotent reopening. `migrate_pre_stamp_table` returns early when nothing is missing (`:180-181`) and the sidecar stamp is left alone. - The machine has no terminal state in source. `v2` is absorbing; the store is never dropped here. - A breaking change — a provider or dimension switch — is refused rather than migrated, and stays + A breaking change (a provider or dimension switch) is refused rather than migrated, and stays fail-loud through `embedding_guard` (`:25-27`). Defined at: `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:195` @@ -119,8 +119,8 @@ stateDiagram-v2 ``` - The first `enqueue` lands in `backoff`, never `due`: it stamps `next_attempt_at` at now plus - `_backoff_delta(1)`, which is two minutes (`:78-81`, `:109`). Backoff is exponential in minutes — - 2, 4, 8, 16, 32 — and `_BACKOFF_CAP_MIN` clamps it at 60 (`:43`, `:79`). That clamp is defensive + `_backoff_delta(1)`, which is two minutes (`:78-81`, `:109`). Backoff is exponential in minutes + (2, 4, 8, 16, 32), and `_BACKOFF_CAP_MIN` clamps it at 60 (`:43`, `:79`). That clamp is defensive rather than reachable at the shipped default: `drain` admits a unit only while `attempts < max_attempts` (`:145`), so with `MAX_ATTEMPTS_DEFAULT` of 5 (`:42`) the largest value `_backoff_delta` receives is 5 and the longest real wait is 32 minutes. @@ -147,8 +147,8 @@ stateDiagram-v2 Every pipeline named by `PIPELINE_NAMES` (`packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:37-42`) fire -the identical call sequence — `drain`, then `blocked_units`, then `enqueue` on failure and -`mark_done` on success: +the identical call sequence (`drain`, then `blocked_units`, then `enqueue` on failure and +`mark_done` on success): | pipeline | `drain` | `blocked_units` | `enqueue` | `mark_done` | | --- | --- | --- | --- | --- | diff --git a/docs/diagrams/architecture/components.md b/docs/diagrams/architecture/components.md index 8e3042d..225a7e4 100644 --- a/docs/diagrams/architecture/components.md +++ b/docs/diagrams/architecture/components.md @@ -110,9 +110,9 @@ binding order at Every edge is one import plus one call. **Every cross-package import in this workspace is indented into the function body that needs it**, never at module top. A fresh-interpreter test asserts that a bare `import atif_cli.app` pulls in none -of `duckdb`, `harbor`, `lancedb`, `boto3`, or `polars` — -`_FORBIDDEN_EAGER_IMPORTS` at `packages/atif-cli/tests/test_lean_import.py:17-31` -— and `packages/atif-cli/src/atif_cli/app.py:22-25` records that as the reason the +of `duckdb`, `harbor`, `lancedb`, `boto3`, or `polars` +(`_FORBIDDEN_EAGER_IMPORTS` at `packages/atif-cli/tests/test_lean_import.py:17-31`), +and `packages/atif-cli/src/atif_cli/app.py:22-25` records that as the reason the heavy imports sit in the command bodies. A line-anchored grep for `^from atif_` therefore finds none of these edges. @@ -137,27 +137,27 @@ root. Those edges are the whole of its outbound surface: an unanchored grep for `pyproject.toml:514-517` declares an `independence` contract over `atif_converter`, `atif_corpus`, `atif_duck`, `atif_models`, and `atif_embed`, so no edge may connect any two of them. `pyproject.toml:519-523` declares a -`forbidden` contract admitting exactly one analytics edge — to `atif_models`. Both +`forbidden` contract admitting exactly one analytics edge, to `atif_models`. Both are checked by `lint:imports`, one of the gates `mise run check` depends on (`mise.toml:204`, in the list at `mise.toml:199-211`). A Protocol declared in one member and implemented in another is **not** an edge between them: the implementation satisfies it structurally, and the composition -root supplies the instance. Neither is a permission comment an edge — +root supplies the instance. Neither is a permission comment an edge: `pyproject.toml:509` reads "ONLY atif-cli and atif-analytics may import atif-models", which grants reach that atif-cli does not take. | Absent edge | How the same work reaches across the boundary | | --- | --- | | `CorpusMaterializer -> AtifConverter` | `materialize` takes a `ConverterPort` parameter (`packages/atif-corpus/src/atif_corpus/application/materialize.py:480`), the Protocol at `packages/atif-corpus/src/atif_corpus/domain/ports.py:41`. `AtifCliApp` constructs the `RealConverter` adapter (`packages/atif-cli/src/atif_cli/converter_adapter.py:44`) and injects it at `packages/atif-cli/src/atif_cli/app.py:399`; that adapter's docstring names the independence contract as its reason to live in atif-cli (`packages/atif-cli/src/atif_cli/converter_adapter.py:5-9`). | -| `EmbedBackfill -> DuckRegistry` | `EmbedBackfill` reads the corpus through its own DuckDB `TextRowsPort` adapter, `DuckDbTextRows` (`packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:155`) against the Protocol at `packages/atif-embed/src/atif_embed/domain/ports.py:87`, over the `docs/CONTRACT.md` corpus layout — stated at `pyproject.toml:458-460`. | +| `EmbedBackfill -> DuckRegistry` | `EmbedBackfill` reads the corpus through its own DuckDB `TextRowsPort` adapter, `DuckDbTextRows` (`packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:155`) against the Protocol at `packages/atif-embed/src/atif_embed/domain/ports.py:87`, over the `docs/CONTRACT.md` corpus layout (stated at `pyproject.toml:458-460`). | | `AnalyticsPipelines -> CorpusMaterializer` | `AnalyticsPipelines` reads the materialized corpus directly through its own `CorpusReader` (`packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:143`), constructed once per run at `packages/atif-analytics/src/atif_analytics/application/analyze.py:57`. | -| `AtifCliApp -> ModelRegistry` | `AtifCliApp` never reaches atif-models. Model selection happens inside `AnalyticsPipelines`, whose `build_provider` resolves a spec through `spec_for` (`packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:44`) — so no model id is written down outside atif-models. | +| `AtifCliApp -> ModelRegistry` | `AtifCliApp` never reaches atif-models. Model selection happens inside `AnalyticsPipelines`, whose `build_provider` resolves a spec through `spec_for` (`packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:44`), so no model id is written down outside atif-models. | | Absent edge | How the same work reaches across the boundary | | --- | --- | | `CorpusMaterializer -> AtifConverter` | `materialize` takes a `ConverterPort` parameter (`packages/atif-corpus/src/atif_corpus/application/materialize.py:480`), the Protocol at `packages/atif-corpus/src/atif_corpus/domain/ports.py:41`. `AtifCliApp` constructs the `RealConverter` adapter (`packages/atif-cli/src/atif_cli/converter_adapter.py:44`) and injects it at `packages/atif-cli/src/atif_cli/app.py:399`; that adapter's docstring names the independence contract as its reason to live in atif-cli (`packages/atif-cli/src/atif_cli/converter_adapter.py:5-9`). | -| `EmbedBackfill -> DuckRegistry` | `EmbedBackfill` reads the corpus through its own DuckDB `TextRowsPort` adapter, `DuckDbTextRows` (`packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:155`) against the Protocol at `packages/atif-embed/src/atif_embed/domain/ports.py:87`, over the `docs/CONTRACT.md` corpus layout — stated at `pyproject.toml:458-460`. | +| `EmbedBackfill -> DuckRegistry` | `EmbedBackfill` reads the corpus through its own DuckDB `TextRowsPort` adapter, `DuckDbTextRows` (`packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:155`) against the Protocol at `packages/atif-embed/src/atif_embed/domain/ports.py:87`, over the `docs/CONTRACT.md` corpus layout (stated at `pyproject.toml:458-460`). | | `AnalyticsPipelines -> CorpusMaterializer` | `AnalyticsPipelines` reads the materialized corpus directly through its own `CorpusReader` (`packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:143`), constructed once per run at `packages/atif-analytics/src/atif_analytics/application/analyze.py:57`. | ## See also diff --git a/docs/diagrams/behavioral/sequences.md b/docs/diagrams/behavioral/sequences.md index f5c4c86..00e2541 100644 --- a/docs/diagrams/behavioral/sequences.md +++ b/docs/diagrams/behavioral/sequences.md @@ -28,20 +28,20 @@ sequenceDiagram Sources, in dispatch order: -- CLI materialize — `packages/atif-cli/src/atif_cli/app.py:352`; report rendered at `:299`. -- materialize UC — `packages/atif-corpus/src/atif_corpus/application/materialize.py:476`; +- CLI materialize: `packages/atif-cli/src/atif_cli/app.py:352`; report rendered at `:299`. +- materialize UC: `packages/atif-corpus/src/atif_corpus/application/materialize.py:476`; `read_watermark` at `:160` called from `:521`, `scan_sources` called at `:522`, `build_plan` called at `:572`, `_write_session` at `:179` called from `:588`, watermark advanced at `:610`, report built at `:613`. -- scanner — `packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py:143`. -- build_plan — `packages/atif-corpus/src/atif_corpus/domain/sessions.py:159`. -- RealConverter — `packages/atif-cli/src/atif_cli/converter_adapter.py:51`, calling +- scanner: `packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py:143`. +- build_plan: `packages/atif-corpus/src/atif_corpus/domain/sessions.py:159`. +- RealConverter: `packages/atif-cli/src/atif_cli/converter_adapter.py:51`, calling `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:105`. -- converter — our ported Claude Code converter, +- converter: our ported Claude Code converter, `packages/atif-converter/src/atif_converter/domain/claude_code_conversion.py:75`, reached from the seam at `packages/atif-converter/src/atif_converter/infrastructure/harbor_adapter.py:83`; harbor supplies only the data classes and the validator (`:68`). -- corpus artifacts — path arithmetic in +- corpus artifacts: path arithmetic in `packages/atif-corpus/src/atif_corpus/domain/layout.py:26`; the artifacts written and the directory swapped at `packages/atif-corpus/src/atif_corpus/application/materialize.py:222-240` through `packages/atif-corpus/src/atif_corpus/infrastructure/atomic.py:64` and `:98`. @@ -74,23 +74,23 @@ sequenceDiagram Sources, in dispatch order: -- CLI query — `packages/atif-cli/src/atif_cli/app.py:529`; `duckdb.connect()` at `:591`, `register` +- CLI query: `packages/atif-cli/src/atif_cli/app.py:529`; `duckdb.connect()` at `:591`, `register` called at `:594`, `_harden_query_connection` at `:137` called from `:601`, caller SQL executed at `:606`, emit at `:612`. -- atif_duck register — `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1212`, whose +- atif_duck register: `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1212`, whose documented order is fixed at `:1283-1294`: `register_raw` `:196`, `register_views` `:333`, `register_vss` `:846`, `register_macros` `:1005`, then the analytics pair. -- DuckDB in-process — the connection opened by the CLI; the sandbox settings and +- DuckDB in-process: the connection opened by the CLI; the sandbox settings and `lock_configuration` are issued against it at `packages/atif-cli/src/atif_cli/app.py:180-188`. -- corpus artifacts — session globs read through `read_json` at +- corpus artifacts: session globs read through `read_json` at `packages/atif-duck/src/atif_duck/infrastructure/registry.py:210-213`; the analytics parquets bind through `packages/atif-duck/src/atif_duck/infrastructure/analytics.py:124`, with the analytics macros at `:225`. -- LanceDB store — reached through DuckDB's lance extension: +- LanceDB store: reached through DuckDB's lance extension; `packages/atif-duck/src/atif_duck/infrastructure/registry.py:878-887` installs, loads, and ATTACHes it; the stamped `(model, dim)` identity is read at `:937` and checked by `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:46`. -- stdout emitter — `packages/atif-cli/src/atif_cli/output.py:154`; a registration failure routes +- stdout emitter: `packages/atif-cli/src/atif_cli/output.py:154`; a registration failure routes through `packages/atif-cli/src/atif_cli/duck_errors.py:66` instead. ## analyze @@ -115,30 +115,30 @@ sequenceDiagram Sources, in dispatch order: -- CLI analyze — `packages/atif-cli/src/atif_cli/app.py:1240`; settings resolved at `:1296`, +- CLI analyze: `packages/atif-cli/src/atif_cli/app.py:1240`; settings resolved at `:1296`, `run_analyze` called at `:1312`. -- run_analyze — `packages/atif-analytics/src/atif_analytics/application/analyze.py:36`; the +- run_analyze: `packages/atif-analytics/src/atif_analytics/application/analyze.py:36`; the `stages` list at `:97-102`, each stage invoked at `:123`. -- CorpusReader — `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:143`, +- CorpusReader: `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:143`, constructed once per run at `packages/atif-analytics/src/atif_analytics/application/analyze.py:57`; `session_text` at `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:289`. -- LLM stages — the classify exemplar at +- LLM stages: the classify exemplar at `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:368`; provider built by `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:36`, structured call dispatched at `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:214`. -- Bedrock runtime — `OpenAiBedrockProvider.classify_structured` +- Bedrock runtime: `OpenAiBedrockProvider.classify_structured` `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:217`, blocking `invoke_model` at `:261` on the `bedrock-runtime` client built at `:195`. -- analytics dir — `/analytics/`, defined at +- analytics dir: `/analytics/`, defined at `packages/atif-analytics/src/atif_analytics/domain/layout.py:51`, holding both the sharded parquet caches (`write_part` `packages/atif-analytics/src/atif_analytics/infrastructure/parquet_cache.py:90`, called at `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:278`) and the single SQLite WAL `state.db` (`packages/atif-analytics/src/atif_analytics/domain/layout.py:81`). -- analytics dir, SQLite arm — `filter_unchanged` +- analytics dir, SQLite arm: `filter_unchanged` `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:137` called at `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:111`; `mark_completed` diff --git a/docs/diagrams/structural/dependency-graph.md b/docs/diagrams/structural/dependency-graph.md index 3bc0179..571671b 100644 --- a/docs/diagrams/structural/dependency-graph.md +++ b/docs/diagrams/structural/dependency-graph.md @@ -82,7 +82,7 @@ from a file under the source member's `src/`. | atif-analytics to atif-models | the `forbidden` contract's single permitted edge (`pyproject.toml:519-523`); declared `packages/atif-analytics/pyproject.toml:24` | Absences carry meaning too. atif-cli imports atif_models in no source file even though it composes -everything else — it reaches the model registry through atif-analytics, and its +everything else; it reaches the model registry through atif-analytics, and its `[project.dependencies]` list omits atif-models accordingly (`packages/atif-cli/pyproject.toml:32-39`). And no edge exists in either direction between atif-converter, atif-corpus, atif-duck, atif-embed, or atif-models: they communicate only by writing and reading the corpus on disk. Adding any such edge @@ -114,15 +114,15 @@ on import-site count, then on the member's own src line count, descending. What the drawn edge deliberately compresses: -- **loguru is universal.** Every member declares it — `packages/atif-analytics/pyproject.toml:24`, +- **loguru is universal.** Every member declares it (`packages/atif-analytics/pyproject.toml:24`, `packages/atif-cli/pyproject.toml:43`, `packages/atif-converter/pyproject.toml:31`, `packages/atif-corpus/pyproject.toml:19`, `packages/atif-duck/pyproject.toml:21`, - `packages/atif-embed/pyproject.toml:23`, `packages/atif-models/pyproject.toml:25` — and the edge is + `packages/atif-embed/pyproject.toml:23`, `packages/atif-models/pyproject.toml:25`), and the edge is drawn from atif-converter only because more of the importing files are its than any other member's. The edge label states the real fan-out. -- **atif-corpus has no drawn external edge.** It declares its externals — loguru - (`packages/atif-corpus/pyproject.toml:19`), pydantic (`:20`), pydantic-settings (`:21`) — and each is +- **atif-corpus has no drawn external edge.** It declares its externals: loguru + (`packages/atif-corpus/pyproject.toml:19`), pydantic (`:20`), pydantic-settings (`:21`). Each is imported more often, or in more sites, by another member, so the attribution rule sources all of them elsewhere. Its own import sites are real: `packages/atif-corpus/src/atif_corpus/infrastructure/settings.py:19` and `packages/atif-corpus/src/atif_corpus/domain/sessions.py:29`. diff --git a/docs/insights/business-logic.md b/docs/insights/business-logic.md index 89dcf91..bd33820 100644 --- a/docs/insights/business-logic.md +++ b/docs/insights/business-logic.md @@ -7,7 +7,7 @@ whether work runs at all. **Scope.** Application-layer and domain-layer rules across the workspace members, plus the SQL surface `atif-duck` registers into DuckDB. There is no database server, no migration directory, and no HTTP surface in this repo, so there are no DDL constraints or -request-validation middlewares to survey — the DuckDB views and macros are the closest +request-validation middlewares to survey; the DuckDB views and macros are the closest thing to a "schema", and the rules encoded in their DDL are in scope and captured under Calculations and Invariants. LLM-output schemas count as validations here, because the provider adapter re-validates every response with pydantic before it reaches a parquet @@ -16,7 +16,7 @@ of scope. **Test provenance.** Where a rule is pinned by a test, the test is cited beside the implementation. A row with no test citation is a rule read out of the implementation and -not covered by a named test — that difference is stated, never smoothed over. +not covered by a named test, and that difference is stated, never smoothed over. **Units.** Every `*_ns` value is epoch nanoseconds from `os.stat().st_mtime_ns`; every `*_chars` value counts Python string characters, not bytes or tokens; pricing is USD per @@ -37,7 +37,7 @@ per run, per pipeline) is stated per row. | A user step and a user text record pair up only while the record's harbor-derived text equals the step's message | Conversion | `packages/atif-converter/src/atif_converter/domain/enrichment.py:347-360`; test `packages/atif-converter/tests/test_enrichment.py:372` | refuse: same truncation marker, WARNING logged | | A session materializes only once its newest source mtime is at least `quiesce_seconds` old (default 300 seconds of source silence, per session) | Corpus | `packages/atif-corpus/src/atif_corpus/domain/sessions.py:83-104` and `packages/atif-corpus/src/atif_corpus/domain/sessions.py:201-202`; tests `packages/atif-corpus/tests/test_domain.py:40` (inclusive boundary) and `packages/atif-corpus/tests/test_domain.py:98` | deferred into `skipped_live`; revisited next pass | | A source mtime in the FUTURE never satisfies the quiescence threshold | Corpus | `packages/atif-corpus/src/atif_corpus/domain/sessions.py:96-104`; test `packages/atif-corpus/tests/test_domain.py:45` | stays skipped, plus a WARNING every pass so the starvation is visible | -| `watermark.json` must parse as a mapping of path to int | Corpus | `packages/atif-corpus/src/atif_corpus/application/materialize.py:160-176` | degrade to empty (one full re-materialization pass), WARNING logged — never a refusal to sync | +| `watermark.json` must parse as a mapping of path to int | Corpus | `packages/atif-corpus/src/atif_corpus/application/materialize.py:160-176` | degrade to empty (one full re-materialization pass), WARNING logged, never a refusal to sync | | A scan finding zero sessions while the corpus holds materialized ones is not a deletion | Corpus | `packages/atif-corpus/src/atif_corpus/application/materialize.py:97-104` and `packages/atif-corpus/src/atif_corpus/application/materialize.py:553-559`; test `packages/atif-corpus/tests/test_materialize.py:292` | raise `SuspiciousEmptyScanError`; nothing is removed | | Only a genuine `FileNotFoundError` proves a source vanished; any other `OSError` means unreadable | Corpus | `packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py:83-96` and `packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py:176-186`; test `packages/atif-corpus/tests/test_materialize.py:819` | session lands in `SourceScan.unreadable`, is never ghosted, and its watermark entries are retained | | A retry-queue or checkpoint `pipeline` must be one of the contract names in `PIPELINE_NAMES` | Analytics | `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/retry_queue.py:96-98`, `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:37-42`; tests `packages/atif-analytics/tests/test_state.py:145` and `packages/atif-analytics/tests/test_state.py:150` | raise `ValueError` | @@ -47,10 +47,10 @@ per run, per pipeline) is stated per row. | A returned conflict pair's two uuids must BOTH be real `edges.jsonl` uuids for that session | Analytics | `packages/atif-analytics/src/atif_analytics/application/use_cases/conflicts.py:266-296`; test `packages/atif-analytics/tests/test_resource_guards.py:590` | pair dropped; an unreadable universe drops every pair (fails closed) | | An untrusted step body may not present itself as a turn header | Analytics | `packages/atif-analytics/src/atif_analytics/domain/transcript.py:64-73`; tests `packages/atif-analytics/tests/test_resource_guards.py:561` and `packages/atif-analytics/tests/test_resource_guards.py:576` | coerce: `[uuid=` is rewritten to `(uuid=` case-insensitively, length preserved, before any header is built | | A friction candidate is a human turn (a main-chain user step whose author is `human`) with a uuid, 1 to `friction_max_chars` characters (default 300, per message); hook feedback, task notifications, retry nudges and image metadata never qualify | Analytics | `packages/atif-analytics/src/atif_analytics/application/use_cases/friction.py:151-169`, `packages/atif-analytics/src/atif_analytics/domain/transcript.py:141-143`; test `packages/atif-analytics/tests/test_friction_tiers.py:98` | filtered out at the candidate boundary; no LLM call is ever made for it | -| `ATIF_SQL_LLM_FAMILY` must name a family with a wired provider adapter | Models | `packages/atif-models/src/atif_models/infrastructure/settings.py:25` and `packages/atif-models/src/atif_models/infrastructure/settings.py:41-54`; tests `packages/atif-models/tests/test_settings.py:59` and `packages/atif-models/tests/test_settings.py:64` | raise `ValueError` at settings load — a startup refusal instead of a Bedrock 400 on every call | +| `ATIF_SQL_LLM_FAMILY` must name a family with a wired provider adapter | Models | `packages/atif-models/src/atif_models/infrastructure/settings.py:25` and `packages/atif-models/src/atif_models/infrastructure/settings.py:41-54`; tests `packages/atif-models/tests/test_settings.py:59` and `packages/atif-models/tests/test_settings.py:64` | raise `ValueError` at settings load: a startup refusal instead of a Bedrock 400 on every call | | `size_for` accepts only a pipeline with an `llm_size_` field | Models | `packages/atif-models/src/atif_models/infrastructure/settings.py:67-77`; test `packages/atif-models/tests/test_settings.py:88` | raise `KeyError` | | A schema with a schema-valued `additionalProperties` (a `dict[str, X]` field) cannot be expressed in OpenAI strict mode | Models | `packages/atif-models/src/atif_models/domain/schema.py:63-69`; test `packages/atif-models/tests/test_schema.py:71` | raise `ValueError` at body-build time | -| `finish_reason` of `content_filter` or `refusal`, or a non-empty `message.refusal`, is a decline | Models | `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:77`, `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:284-290`; tests `packages/atif-models/tests/test_openai_provider.py:182` and `packages/atif-models/tests/test_openai_provider.py:190` | raise `RefusalError` — terminal, never retried | +| `finish_reason` of `content_filter` or `refusal`, or a non-empty `message.refusal`, is a decline | Models | `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:77`, `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:284-290`; tests `packages/atif-models/tests/test_openai_provider.py:182` and `packages/atif-models/tests/test_openai_provider.py:190` | raise `RefusalError` (terminal, never retried) | | `finish_reason` of `length` gets exactly ONE retry, at `reasoning_effort` degraded one step | Models | `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:85-90`, `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:217-249`, `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:291-297`; tests `packages/atif-models/tests/test_openai_provider.py:129`, `packages/atif-models/tests/test_openai_provider.py:144`, `packages/atif-models/tests/test_openai_provider.py:163` | raise `ProviderUnavailable` when still truncated or already at the floor; the unit goes to the retry queue | | Response content must be a non-empty string that parses as JSON and validates against the schema | Models | `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:298-311`; tests `packages/atif-models/tests/test_openai_provider.py:198`, `packages/atif-models/tests/test_openai_provider.py:206`, `packages/atif-models/tests/test_openai_provider.py:212` | raise `ProviderUnavailable` | | A store's stamped `(model, dim)` must match the active embedder before any vector is read or appended | Embeddings | `packages/atif-embed/src/atif_embed/domain/embedding_guard.py:38-80`, `packages/atif-embed/src/atif_embed/application/embed.py:166-174`; tests `packages/atif-embed/tests/test_embed_use_case.py:153`, `packages/atif-embed/tests/test_lance_store.py:321` and `packages/atif-embed/tests/test_lance_store.py:330` | raise `EmbeddingProviderMismatch` (`terminal = True`); `query`/`search` exit 65, `embed` exits 78 | @@ -67,7 +67,7 @@ Notes on two rows above. The strict-JSON transform in rather than a rejecting one: it forces `additionalProperties: false` on every object level and moves every property into `required`, expressing optionality by making the field's type nullable. Only the open-mapping shape has no representation and raises. Second: the -provider's structured-output contract and pydantic are two gates in series — the wire +provider's structured-output contract and pydantic are two gates in series: the wire schema constrains shape, and `model_validate` re-applies the `ge`/`le` bounds the model may have ignored (`packages/atif-models/src/atif_models/domain/schema.py:18-21`). @@ -80,7 +80,7 @@ may have ignored (`packages/atif-models/src/atif_models/domain/schema.py:18-21`) | The `FidelityGap` members are the complete, typed statement of what harbor 0.22.0's conversion loses; some are structural and observed on every session, the rest are session-conditional | Application code | `packages/atif-converter/src/atif_converter/domain/fidelity.py:44-79`; `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:52-59` (structural set) and `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:70-76` (conditional set); test `packages/atif-converter/tests/test_convert_and_audit.py:163` | | Only `RecordType.USER` and `RecordType.ASSISTANT` are convertible; every other record type is dropped upstream | Application code | `packages/atif-converter/src/atif_converter/domain/fidelity.py:41`; test `packages/atif-converter/tests/test_convert_and_audit.py:121` | | `LossReport.records_total` equals the `edges.jsonl` line count for any session, because the census and the edges emitter read one snapshot | Application code | `packages/atif-converter/src/atif_converter/domain/fidelity.py:91-98`, `packages/atif-converter/src/atif_converter/infrastructure/census.py:57-67`, `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:137-139`; test `packages/atif-converter/tests/test_snapshot_and_drift.py:122` | -| `records_converted` counts raw user/assistant RECORDS, not ATIF steps — harbor bundles several assistant events sharing one `message.id` into one agent step, so the two numbers differ by design | Application code | `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:122-126`, `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:62-68` | +| `records_converted` counts raw user/assistant RECORDS, not ATIF steps; harbor bundles several assistant events sharing one `message.id` into one agent step, so the two numbers differ by design | Application code | `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:122-126`, `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:62-68` | | `subagent_files_convertible` equals `subagent_files_found`: our converter discovers every side file itself, workflow-nested ones included, so nothing found is unconvertible; the retired `workflow_subagents_missed` value is never emitted | Application code | `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:64`; test `packages/atif-converter/tests/test_convert_and_audit.py:127` | | A truncation marker or a non-zero leftover count means enrichment attribution was REFUSED, never guessed | Application code | `packages/atif-converter/src/atif_converter/domain/enrichment.py:45-62`, `packages/atif-converter/src/atif_converter/domain/enrichment.py:379-399`; tests `packages/atif-converter/tests/test_enrichment.py:272`, `packages/atif-converter/tests/test_enrichment.py:287`, `packages/atif-converter/tests/test_enrichment.py:302` | | Staged side-files are per-FILE symlinks under real directories, flattened with `__` joins, because Python 3.13's `rglob` does not descend a symlinked directory and a flat namespace can collide | Application code | `packages/atif-converter/src/atif_converter/infrastructure/harbor_adapter.py:71-92` | @@ -96,11 +96,11 @@ may have ignored (`packages/atif-models/src/atif_models/domain/schema.py:18-21`) | Persistence order matches write order: each artifact is fsynced before the rename that publishes it, and the parent directory after | Application code | `packages/atif-corpus/src/atif_corpus/infrastructure/atomic.py:64-95`, `packages/atif-corpus/src/atif_corpus/infrastructure/atomic.py:140-158`; tests `packages/atif-corpus/tests/test_atomic.py:131`, `packages/atif-corpus/tests/test_atomic.py:149`, `packages/atif-corpus/tests/test_atomic.py:194` | | A kill inside the swap window leaves the session dir MISSING rather than torn, and the next pass force-replans it | Application code | `packages/atif-corpus/src/atif_corpus/infrastructure/atomic.py:113-123`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:313-339`; tests `packages/atif-corpus/tests/test_materialize.py:791`, `packages/atif-corpus/tests/test_domain.py:216` | | Only sessions that materialized successfully advance their watermark entries, so a failed session stays stale and is retried | Application code | `packages/atif-corpus/src/atif_corpus/application/materialize.py:428-473`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:615-622`; test `packages/atif-corpus/tests/test_materialize.py:333` | -| A VANISHED source path keeps its watermark entry unless the owning session succeeded this pass or left the scan entirely — the retained entry IS the staleness signal | Application code | `packages/atif-corpus/src/atif_corpus/application/materialize.py:455-473`; tests `packages/atif-corpus/tests/test_materialize.py:191` and `packages/atif-corpus/tests/test_materialize.py:227` | +| A VANISHED source path keeps its watermark entry unless the owning session succeeded this pass or left the scan entirely; the retained entry IS the staleness signal | Application code | `packages/atif-corpus/src/atif_corpus/application/materialize.py:455-473`; tests `packages/atif-corpus/tests/test_materialize.py:191` and `packages/atif-corpus/tests/test_materialize.py:227` | | `owns_path` is the single definition of a session's watermark scope, so staleness and retention can never disagree about which entries belong to a session | Application code | `packages/atif-corpus/src/atif_corpus/domain/sessions.py:129-140`; tests `packages/atif-corpus/tests/test_domain.py:191`, `packages/atif-corpus/tests/test_domain.py:202`, `packages/atif-corpus/tests/test_domain.py:207` | | Only an exactly equal `mtime_ns` counts as unchanged; an mtime that moved BACKWARDS is `modified` | Application code | `packages/atif-corpus/src/atif_corpus/domain/watermark.py:61-76`; test `packages/atif-corpus/tests/test_domain.py:86` | | A staging entry is removed only when EVERY pid its name carries is proven gone; only `ProcessLookupError` proves that | Application code | `packages/atif-corpus/src/atif_corpus/application/materialize.py:256-310`; tests `packages/atif-corpus/tests/test_materialize.py:429`, `packages/atif-corpus/tests/test_materialize.py:457`, `packages/atif-corpus/tests/test_materialize.py:481` | -| One broken transcript never aborts a corpus sync — the port may raise anything, and the pass records the failure and continues | Application code | `packages/atif-corpus/src/atif_corpus/domain/ports.py:41-47`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:597-616`; test `packages/atif-corpus/tests/test_materialize.py:316` | +| One broken transcript never aborts a corpus sync: the port may raise anything, and the pass records the failure and continues | Application code | `packages/atif-corpus/src/atif_corpus/domain/ports.py:41-47`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:597-616`; test `packages/atif-corpus/tests/test_materialize.py:316` | | The domain reads no clock: `now_ns` is always passed in, so identical inputs always yield an identical plan | Application code | `packages/atif-corpus/src/atif_corpus/domain/sessions.py:18-21`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:48-52`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:528`; test `packages/atif-corpus/tests/test_domain.py:145` | | Every plan partition is sorted by session id, so the same scan yields the same work order, log output, and failure ordering | Application code | `packages/atif-corpus/src/atif_corpus/domain/sessions.py:110-121`, `packages/atif-corpus/src/atif_corpus/domain/sessions.py:195`; test `packages/atif-corpus/tests/test_domain.py:145` | | The corpus root `~/.claude` maps to the reserved slug `default`; every other root is `-<8 hex of sha256 of the resolved path>` | Application code | `packages/atif-corpus/src/atif_corpus/domain/slug.py:23`, `packages/atif-corpus/src/atif_corpus/domain/slug.py:45-50`; tests `packages/atif-corpus/tests/test_slug.py:19`, `packages/atif-corpus/tests/test_slug.py:24`, `packages/atif-corpus/tests/test_slug.py:32` | @@ -111,8 +111,8 @@ may have ignored (`packages/atif-models/src/atif_models/domain/schema.py:18-21`) | Invariant | Where enforced | Citation | | --- | --- | --- | | The retry queue is the SINGLE re-admission gate for a failed unit: a live entry blocks dispatch until `drain` says it is due, because a failed unit is never checkpointed and would otherwise be re-admitted fresh every run forever | Application code | `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/retry_queue.py:176-193`, `packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py:167-173`; tests `packages/atif-analytics/tests/test_state.py:124`, `packages/atif-analytics/tests/test_perceived.py:231` | -| A FAILED unit is not checkpointed; a REFUSED unit is (terminal), and its refusal is durable — an in-band sentinel row for session-keyed `classify`, a `analytics/refusals` sidecar row for the uuid-keyed pipelines | Application code | `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:222-246`, `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:191-229`, `packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py:277-289`; test `packages/atif-analytics/tests/test_perceived.py:252` | -| Nothing is stamped — no checkpoint, no cache row, no retry entry — for a unit that was never dispatched | Application code | `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:155-183`, `packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py:248-266`; test `packages/atif-analytics/tests/test_perceived.py:280` | +| A FAILED unit is not checkpointed; a REFUSED unit is (terminal), and its refusal is durable: an in-band sentinel row for session-keyed `classify`, a `analytics/refusals` sidecar row for the uuid-keyed pipelines | Application code | `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:222-246`, `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:191-229`, `packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py:277-289`; test `packages/atif-analytics/tests/test_perceived.py:252` | +| Nothing is stamped (no checkpoint, no cache row, no retry entry) for a unit that was never dispatched | Application code | `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:155-183`, `packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py:248-266`; test `packages/atif-analytics/tests/test_perceived.py:280` | | Budget overshoot is bounded by `BUDGET_CHECK_BATCH` units (8, per dispatch batch) on the priciest watched model, and does not scale with the session ceiling or the write chunk size | Application code | `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:100-112`, `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:146-152`; test `packages/atif-analytics/tests/test_resource_guards.py:362` (over both dispatch shapes) | | A checkpoint skip requires BOTH `last_ts` and `last_mtime` to be non-advancing; either bound moving forward re-admits the session | Application code | `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:136-176`; tests `packages/atif-analytics/tests/test_state.py:48` and `packages/atif-analytics/tests/test_state.py:57` | | `state.db` is corpus-scoped at `/analytics/state.db`, so one corpus's completions can never skip another's re-scoring | Application code | `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:22-24` | @@ -129,13 +129,13 @@ may have ignored (`packages/atif-models/src/atif_models/domain/schema.py:18-21`) | Invariant | Where enforced | Citation | | --- | --- | --- | | The registry is the ONLY place in the workspace where a model id is written down, and it is total over family times size | Application code | `packages/atif-models/src/atif_models/domain/registry.py:5-8`, `packages/atif-models/src/atif_models/domain/registry.py:60-106`; test `packages/atif-models/tests/test_registry.py:61` | -| `estimate_cost` returns `None`, never `0.0`, when either rate is unknown — callers must render "pricing unavailable" | Application code | `packages/atif-models/src/atif_models/domain/registry.py:125-137`; tests `packages/atif-models/tests/test_registry.py:85` and `packages/atif-models/tests/test_registry.py:95` | +| `estimate_cost` returns `None`, never `0.0`, when either rate is unknown; callers must render "pricing unavailable" | Application code | `packages/atif-models/src/atif_models/domain/registry.py:125-137`; tests `packages/atif-models/tests/test_registry.py:85` and `packages/atif-models/tests/test_registry.py:95` | | Usage is accumulated BEFORE any `finish_reason` gate, because a length-truncated call still billed tokens | Application code | `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:269-277`; test `packages/atif-models/tests/test_openai_provider.py:249` | | `RETRY_CODES` (the retryable Bedrock codes) is a deliberate TWIN of atif-embed's set, pinned against the same literal in each package's own suite because the two may not import each other | Application code | `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:59-74`; tests `packages/atif-models/tests/test_openai_provider.py:318` and `packages/atif-models/tests/test_openai_provider.py:322` | | Every embeddings row stamps `model`, `dim`, `text_hash`, and `truncated`, so a re-conversion that changes a step's text is detectable under the same uuid | Application code | `packages/atif-embed/src/atif_embed/domain/text_stamp.py:6-18`, `packages/atif-embed/src/atif_embed/application/embed.py:216-235`; test `packages/atif-embed/tests/test_embed_use_case.py:89` | | A uuid never fans out to two vectors: a stale row under the same uuid is DELETED before the new vector is appended | Application code | `packages/atif-embed/src/atif_embed/application/embed.py:203-210`, `packages/atif-embed/src/atif_embed/domain/text_stamp.py:50-63`; tests `packages/atif-embed/tests/test_embed_use_case.py:211`, `packages/atif-embed/tests/test_lance_store.py:129` | | `_PRE_STAMP_SENTINEL` (``) can never equal a real blake2b hex digest, so every pre-stamp row mismatches its corpus hash and re-embeds through the ordinary staleness path | Application code | `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:69-73`; test `packages/atif-embed/tests/test_lance_store.py:192` | -| ADDITIVE schema drift migrates ONLINE (metadata-only `add_columns`, no re-embedding, search stays up); a BREAKING change — provider or dimension switch — stays fail-loud | Application code | `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:16-32`, `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:166-192`; tests `packages/atif-embed/tests/test_embed_use_case.py:293` and `packages/atif-embed/tests/test_embed_use_case.py:314` | +| ADDITIVE schema drift migrates ONLINE (metadata-only `add_columns`, no re-embedding, search stays up); a BREAKING change (provider or dimension switch) stays fail-loud | Application code | `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:16-32`, `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:166-192`; tests `packages/atif-embed/tests/test_embed_use_case.py:293` and `packages/atif-embed/tests/test_embed_use_case.py:314` | | The write chunk is a multiple of `batch_size`, so a checkpoint boundary never splits a Bedrock batch | Application code | `packages/atif-embed/src/atif_embed/application/embed.py:145-147` | | The provider/dimension guard exists as two copies (atif-embed and atif-duck) because the independence contract forbids the import; a behavioral change in either must be ported | Application code | `packages/atif-embed/src/atif_embed/domain/embedding_guard.py:15-18`, `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:46`; tests `packages/atif-embed/tests/test_guard_twin_pin.py:76`, `packages/atif-embed/tests/test_guard_twin_pin.py:79`, `packages/atif-embed/tests/test_guard_twin_pin.py:85` | | The recovery hint names the real store locations (`ATIF_SQL_LANCE_URI` or `/embeddings_lance`), never a fixed home-directory path | Application code | `packages/atif-embed/src/atif_embed/domain/embedding_guard.py:25-35`; tests `packages/atif-embed/tests/test_guard_twin_pin.py:97` and `packages/atif-embed/tests/test_guard_twin_pin.py:103` | @@ -144,7 +144,7 @@ may have ignored (`packages/atif-models/src/atif_models/domain/schema.py:18-21`) | Invariant | Where enforced | Citation | | --- | --- | --- | -| Registration order is raw TEMP tables, then views, then VSS, then macros, then the authorship macro and views, then the v2 analytics views and macros — each layer binds against the previous at CREATE time | Application code | `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1740-1752`; test `packages/atif-duck/tests/test_duck_views.py:615` | +| Registration order is raw TEMP tables, then views, then VSS, then macros, then the authorship macro and views, then the v2 analytics views and macros; each layer binds against the previous at CREATE time | Application code | `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1740-1752`; test `packages/atif-duck/tests/test_duck_views.py:615` | | `DESCRIPTIONS` covers the catalog exactly | Application code | `packages/atif-duck/src/atif_duck/domain/catalog.py:431`; test `packages/atif-duck/tests/test_examples.py:178` | | `cost_estimate`'s `est_cost_usd` covers PRICED steps only and is meaningful only when `unpriced_steps = 0`; an inner join would return a partial number indistinguishable from a complete one | Application code | `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1053-1058`, `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1066-1077`; tests `packages/atif-duck/tests/test_duck_views.py:407` and `packages/atif-duck/tests/test_duck_views.py:482` | | Both `cost_estimate` counters filter on `model_name IS NOT NULL`, because user steps carry no model and would otherwise make every session look like a pricing gap | Application code | `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1060-1063`; test `packages/atif-duck/tests/test_duck_views.py:533` | @@ -163,7 +163,7 @@ may have ignored (`packages/atif-models/src/atif_models/domain/schema.py:18-21`) | `cost_estimate(sid)` SQL macro | a session's `steps` rows (`prompt_tokens`, `cached_tokens`, `completion_tokens`, `model_name`) LEFT JOINed against `DEFAULT_PRICING` | `(est_cost_usd, priced_steps, unpriced_steps)` | `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1066-1083`; test `packages/atif-duck/tests/test_duck_views.py:368` | | Retry backoff delay | attempt counter | `min(2 ** attempts, 60)` MINUTES, per `(pipeline, unit_id)`: 2, 4, 8, 16, 32, capped at 60 | `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/retry_queue.py:78-81`; test `packages/atif-analytics/tests/test_state.py:102` | | Bedrock call retry schedule | the raised exception | up to 10 attempts, exponential wait with multiplier 2, minimum 2 SECONDS and maximum 60 SECONDS, only for `RETRY_CODES` and network errors | `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:251-257`, `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:102-109`; test `packages/atif-models/tests/test_openai_provider.py:284` | -| Parquet write chunk size | `batch_size` (default 96 units) | `max(batch_size * 4, 256)` ROWS per part — bounds crash loss, not spend | `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:242-250` | +| Parquet write chunk size | `batch_size` (default 96 units) | `max(batch_size * 4, 256)` ROWS per part (bounds crash loss, not spend) | `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:242-250` | | Quiescence age test | newest source `mtime_ns`, `now_ns`, `quiesce_seconds` | boolean: `now_ns - newest_mtime_ns >= quiesce_seconds * 1_000_000_000` | `packages/atif-corpus/src/atif_corpus/domain/sessions.py:96-104`; test `packages/atif-corpus/tests/test_domain.py:40` | | Source delta between two scans | two `path` to `mtime_ns` maps | added / modified / removed sorted tuples; `touched` is added then modified | `packages/atif-corpus/src/atif_corpus/domain/watermark.py:57-76`; test `packages/atif-corpus/tests/test_domain.py:76` | | Corpus slug | a corpus root path | `default` for `~/.claude`, else `-<8 hex of sha256>` | `packages/atif-corpus/src/atif_corpus/domain/slug.py:45-50`; test `packages/atif-corpus/tests/test_slug.py:24` | @@ -186,7 +186,7 @@ the session's steps and divided by `1e6`. Cache reads are uncharged, and the pri strips a dated model suffix (`claude-haiku-4-5-20251001` matches `claude-haiku-4-5`) via `regexp_replace(model_name, '-\d{8}$', '')`. The pricing table itself is `DEFAULT_PRICING`, entries of `(in_rate, out_rate)` in **USD per 1,000,000 tokens**, -base rates only — the prompt-cache write and read multipliers (1.25x, 2x, 0.1x of base +base rates only: the prompt-cache write and read multipliers (1.25x, 2x, 0.1x of base input) are deliberately not modeled, matching the macro (`packages/atif-duck/src/atif_duck/domain/catalog.py:395-417`, pinned by `packages/atif-duck/tests/test_duck_views.py:459` against published list rates and by @@ -254,7 +254,7 @@ perceived pipeline admits a session only at 2 or more pairs `packages/atif-models/src/atif_models/infrastructure/settings.py:20-25`, `packages/atif-models/src/atif_models/infrastructure/settings.py:41-54`; tests `packages/atif-models/tests/test_settings.py:69` and `packages/atif-models/tests/test_settings.py:72`. - **Per-pipeline size assignment:** classify and friction take `small`, perceived takes - `medium`, conflicts takes `large` (the hardest judgment task) — each overridable via + `medium`, conflicts takes `large` (the hardest judgment task), each overridable via `ATIF_SQL_LLM_SIZE_`. `packages/atif-models/src/atif_models/infrastructure/settings.py:56-66`; test `packages/atif-models/tests/test_settings.py:26`. @@ -299,12 +299,12 @@ perceived pipeline admits a session only at 2 or more pairs and binding it first would be circular. `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1244-1250`; test `packages/atif-duck/tests/test_vss.py:258`. - **Register-or-fail-loud:** any DuckDB error during view or macro registration is logged - with `logger.exception` and re-raised — an empty or absent corpus fails registration + with `logger.exception` and re-raised: an empty or absent corpus fails registration rather than yielding empty views. `packages/atif-duck/src/atif_duck/infrastructure/registry.py:201-207`, `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1201-1204`; test `packages/atif-duck/tests/test_duck_views.py:623`. - **A new view or macro cannot land silently:** the drift tests require a `DESCRIPTIONS` entry, an `ARG_EXEMPLARS` entry for any new parameter name, `TABLE_MACRO_NAMES` - membership when the DDL is `AS TABLE`, and an example that EXECUTES — or a documented + membership when the DDL is `AS TABLE`, and an example that EXECUTES, or a documented `EXCLUSIONS` entry. `packages/atif-duck/src/atif_duck/domain/examples.py:94-99`; tests `packages/atif-duck/tests/test_examples.py:160`, `packages/atif-duck/tests/test_examples.py:178`, `packages/atif-duck/tests/test_examples.py:188`, `packages/atif-duck/tests/test_examples.py:197`. diff --git a/docs/insights/contract-map.md b/docs/insights/contract-map.md index 2401b26..6312df2 100644 --- a/docs/insights/contract-map.md +++ b/docs/insights/contract-map.md @@ -3,7 +3,7 @@ **What counts as a contract here.** This workspace has almost no shared-type imports to trace, because `pyproject.toml:514-523` forbids most of them: the independent members may never import each other, and atif-analytics may import only atif-models. What crosses a module boundary instead -is a **shape agreed by two packages that cannot reference one another's symbols** — a file layout, a +is a **shape agreed by two packages that cannot reference one another's symbols**: a file layout, a JSON key order, a column projection, a `typing.Protocol` typed to a document rather than to an implementation. So a contract in this file is any of: @@ -20,17 +20,17 @@ single most important structural fact for reading this file. `RealConverter` (`packages/atif-cli/src/atif_cli/converter_adapter.py:44`) implements `ConverterPort` (`packages/atif-corpus/src/atif_corpus/domain/ports.py:41`) by shape alone; it never names the Protocol, and it cannot, because atif-corpus may never import atif-converter. The Protocol name -appears at only these import sites in the whole workspace — +appears at only these import sites in the whole workspace: `packages/atif-corpus/src/atif_corpus/application/materialize.py:92` (the module that CALLS it) and `packages/atif-cli/tests/test_converter_adapter.py:17` (the test that binds the two together in an annotation). So each Protocol row below has a producer of the shape and an implementer the type system never links to it, and the answer to "who finds out if the shape drifts" is a named test or -nobody. Do not read a Protocol row as a dependency edge — `pyproject.toml:514-517` forbids the edge +nobody. Do not read a Protocol row as a dependency edge: `pyproject.toml:514-517` forbids the edge it would imply. Every contract below names its producer, its consumers, the verbatim shape, the assumptions consumers make beyond the shape, and the drift risk. **Consumer counts are grep-derived and -confirmed at each import, annotation, or call site** — this repo has no code index +confirmed at each import, annotation, or call site**: this repo has no code index (`.gitignore:24` lists `.codegraph/`, and no index exists on disk), so no count here comes from a symbol graph. These grep hazards shape every count: names collide (`DomainError` and `EmbeddingProviderMismatch` across packages, `cached_tokens` across coordinate spaces), so every @@ -45,15 +45,15 @@ Contracts are ordered by confirmed consumer count, descending. **Consumers:** -- `packages/atif-duck/src/atif_duck/infrastructure/registry.py:210-213` — inlines the +- `packages/atif-duck/src/atif_duck/infrastructure/registry.py:210-213`: inlines the filenames as SQL glob literals for its `read_json` readers. -- `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:120-121` — opens +- `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:120-121`: opens `trajectory.json` and gates on `meta.json` with its own DuckDB connection. -- `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:53-55` — declares +- `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:53-55`: declares `TRAJECTORY_FILENAME` / `EDGES_FILENAME` / `META_FILENAME` again as its own constants. -- `packages/atif-corpus/src/atif_corpus/application/materialize.py:222-238` — the writer side, the +- `packages/atif-corpus/src/atif_corpus/application/materialize.py:222-238`: the writer side, the only place the artifacts are produced. -- `docs/CONTRACT.md:21-39` — the hand-written specification all of them agree to. +- `docs/CONTRACT.md:21-39`: the hand-written specification all of them agree to. **Shape:** @@ -88,7 +88,7 @@ producer through the `ArtifactProducer` port and merges the returned keys into ` torn-set gate independently: `packages/atif-duck/src/atif_duck/infrastructure/registry.py:188-192` restricts the trajectory/edges/loss readers to dirs where `v_raw_meta` has a row; `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:121-124` skips a dir with no `meta.json`; `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:171-180` does the same and logs a warning. Nothing in the file layout expresses - this — it is a write-ORDER promise made at + this: it is a write-ORDER promise made at `packages/atif-corpus/src/atif_corpus/application/materialize.py:209-212`. - **No reader ever sees a partially written directory**, because the writer stages all the artifacts under `/.staging/` and swaps the whole dir with `os.replace` @@ -102,8 +102,8 @@ producer through the `ArtifactProducer` port and merges the returned keys into ` observed, `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:47-49` cites 85 MB. Same constant, two independent justifications. - **`source_mtime_ns` and every `watermark.json` value are epoch NANOSECONDS** from - `os.stat().st_mtime_ns` — unit stated in the identifier and again at - `packages/atif-corpus/src/atif_corpus/domain/watermark.py:12-13`. atif-duck types the column + `os.stat().st_mtime_ns` (unit stated in the identifier and again at + `packages/atif-corpus/src/atif_corpus/domain/watermark.py:12-13`). atif-duck types the column `BIGINT` at `packages/atif-duck/src/atif_duck/infrastructure/registry.py:136`, which preserves the magnitude but drops the unit from the schema. - **`materialized_at` is an ISO-8601 UTC string, not a timestamp.** atif-duck projects it as `VARCHAR` (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:140`); the CLI supplies it (`packages/atif-cli/src/atif_cli/app.py:400`). @@ -115,14 +115,14 @@ dependencies lists VSS/`semantic_search` and the v2 LLM-analytics pipelines as out of scope and then reverses itself at `docs/CONTRACT.md:17-19`; both are shipped commands (`packages/atif-cli/pyproject.toml:42` plus the `embed` / `search` / `analyze` commands). Read -`docs/CONTRACT.md:21-39` as authoritative for the layout — that is the part the packages actually -implement — and the manifest as authoritative for who imports whom. +`docs/CONTRACT.md:21-39` as authoritative for the layout (that is the part the packages actually +implement) and the manifest as authoritative for who imports whom. **Drift risk:** a new artifact, a renamed file, or a new `meta.json` key must be applied in every place above, which no test links, and most of them are reader-side, so an addition silently reaches nobody. Mitigation: the writer-side constants at `packages/atif-corpus/src/atif_corpus/domain/layout.py:18-22` are the single source of truth on the -write side — any layout change starts there and then greps the reader modules named above. +write side. Any layout change starts there and then greps the reader modules named above. ## The static SQL catalog @@ -133,19 +133,19 @@ write side — any layout change starts there and then greps the reader modules **Consumers:** -- `packages/atif-cli/src/atif_cli/app.py:1745-1778` — the `schema` command reads `VIEW_SCHEMA`, +- `packages/atif-cli/src/atif_cli/app.py:1745-1778`: the `schema` command reads `VIEW_SCHEMA`, `MACRO_SIGNATURES`, `ANALYTICS_VIEW_SCHEMA`, and `ANALYTICS_MACRO_SIGNATURES`, tags each object with its `requires` value, and answers with no DuckDB connection. -- `packages/atif-duck/src/atif_duck/domain/examples.py:36-42` — the examples generator imports every +- `packages/atif-duck/src/atif_duck/domain/examples.py:36-42`: the examples generator imports every catalog and derives one runnable query per object. -- `packages/atif-cli/src/atif_cli/app.py:1025-1046` — the `examples` command calls +- `packages/atif-cli/src/atif_cli/app.py:1025-1046`: the `examples` command calls `build_examples()`. -- `packages/atif-duck/tests/test_duck_views.py:31-33` — the `DESCRIBE`-vs-`VIEW_SCHEMA` and +- `packages/atif-duck/tests/test_duck_views.py:31-33`: the `DESCRIBE`-vs-`VIEW_SCHEMA` and DDL-vs-`MACRO_SIGNATURES` drift tests. -- `packages/atif-duck/tests/test_examples.py:27-35` — `DESCRIPTIONS` coverage, `ARG_EXEMPLARS` +- `packages/atif-duck/tests/test_examples.py:27-35`: `DESCRIPTIONS` coverage, `ARG_EXEMPLARS` coverage, and the `AS TABLE` set drift test. -- `packages/atif-duck/tests/test_analytics_views.py:20-21` — the analytics-side drift test. -- `packages/atif-cli/tests/test_app.py:78-84` — asserts the CLI's JSON payload keys equal the +- `packages/atif-duck/tests/test_analytics_views.py:20-21`: the analytics-side drift test. +- `packages/atif-cli/tests/test_app.py:78-84`: asserts the CLI's JSON payload keys equal the catalog keys. **Shape:** @@ -183,11 +183,11 @@ MACRO_SIGNATURES: dict[str, tuple[str, ...]] = { fails on a missing entry AND on a stale entry keyed to an object that no longer exists, so the dict is a bijection with the union of the name catalogs. - **Every macro parameter name has an `ARG_EXEMPLARS` literal**, or example generation raises - rather than emitting a broken query — `packages/atif-duck/src/atif_duck/domain/examples.py:151-157` fails loud, and + rather than emitting a broken query: `packages/atif-duck/src/atif_duck/domain/examples.py:151-157` fails loud, and `packages/atif-duck/tests/test_examples.py:189-194` pre-empts it. - **`TABLE_MACRO_NAMES` decides the SQL call shape.** `packages/atif-duck/src/atif_duck/domain/examples.py:159-162` emits `SELECT * FROM name(args)` for a member and `SELECT name(args)` otherwise, so a macro whose DDL - gains or loses `AS TABLE` breaks every derived example — pinned by the regex test at + gains or loses `AS TABLE` breaks every derived example, pinned by the regex test at `packages/atif-duck/tests/test_examples.py:198-210`. - **The catalog is answerable without a corpus.** `packages/atif-cli/src/atif_cli/app.py:1094-1098` states the sub-50 ms, no-DuckDB-bind guarantee the `schema` command rests on; a runtime `DESCRIBE` would violate it. @@ -195,7 +195,7 @@ MACRO_SIGNATURES: dict[str, tuple[str, ...]] = { consumer enumerating `VIEW_NAMES` does not see `v_raw_trajectories` and friends. **Drift risk:** adding a view or macro without a `DESCRIPTIONS` entry, an `ARG_EXEMPLARS` entry for -each new parameter name, or `TABLE_MACRO_NAMES` membership fails CI loudly — the risk is inverted +each new parameter name, or `TABLE_MACRO_NAMES` membership fails CI loudly: the risk is inverted here, and the real exposure is a *type* that only the fixture corpus produces. Mitigation: `VIEW_SCHEMA["message_embeddings"]` at `packages/atif-duck/src/atif_duck/domain/catalog.py:218-224` hardcodes `FLOAT[1024]`, so run the drift test against a store built at a non-default `output_dimension` before changing that setting. @@ -206,19 +206,19 @@ drift test against a store built at a non-default `output_dimension` before chan **Consumers:** -- `mise.toml:197` — `lint:imports` is one of the `mise run check` gates, so the build is +- `mise.toml:197`: `lint:imports` is one of the `mise run check` gates, so the build is the consumer. -- `packages/atif-corpus/src/atif_corpus/domain/ports.py:5-8` — the `ConverterPort` docstring cites +- `packages/atif-corpus/src/atif_corpus/domain/ports.py:5-8`: the `ConverterPort` docstring cites the independence contract as the reason the Protocol exists at all. -- `packages/atif-embed/src/atif_embed/domain/ports.py:11-15` — cites it as the reason `TextRowsPort` +- `packages/atif-embed/src/atif_embed/domain/ports.py:11-15`: cites it as the reason `TextRowsPort` exists. -- `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:5-10` — cites it as the +- `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:5-10`: cites it as the reason atif-embed carries its own corpus reader. -- `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:5-9` — cites it as the reason the +- `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:5-9`: cites it as the reason the guard is duplicated. -- `packages/atif-duck/src/atif_duck/infrastructure/analytics.py:13-15` — cites it as the reason the +- `packages/atif-duck/src/atif_duck/infrastructure/analytics.py:13-15`: cites it as the reason the analytics artifact names are pinned by hand. -- `packages/atif-analytics/src/atif_analytics/domain/layout.py:8-10` — cites it as the reason its +- `packages/atif-analytics/src/atif_analytics/domain/layout.py:8-10`: cites it as the reason its layout is computed in one place. **Shape:** @@ -246,13 +246,13 @@ forbidden_modules = ["atif_converter", "atif_corpus", "atif_duck", "atif_embed", two independent packages is `packages/atif-cli/src/atif_cli/converter_adapter.py:36-38`, and its docstring names that privilege explicitly at `packages/atif-cli/src/atif_cli/converter_adapter.py:5-9`. - **atif-analytics is absent from the independence list on purpose** so it can compose atif-models, - with the `forbidden` contract pinning its other edges shut — the reasoning is inline at + with the `forbidden` contract pinning its other edges shut; the reasoning is inline at `pyproject.toml:509-513`. Verified: `grep -rn 'atif_duck' packages/atif-analytics/` returns nothing, so the CONTRACT-V2-era design of reading atif-duck's views is not what the code does. - **atif-duck declares no `layers` contract.** Most members do (`pyproject.toml:465-508`); atif-duck has `domain/` and `infrastructure/` but no `application/`, so there is no third layer to order. - **Every layered member's `domain/` is the innermost layer**, which is why all the Protocols live - in `domain/ports.py` and none in an `application/ports.py` — no such file exists in this + in `domain/ports.py` and none in an `application/ports.py`: no such file exists in this workspace. - **The comment at `pyproject.toml:509-510` grants a permission that is not exercised.** It reads "ONLY atif-cli and atif-analytics may import atif-models", but atif-cli neither declares @@ -260,8 +260,8 @@ forbidden_modules = ["atif_converter", "atif_corpus", "atif_duck", "atif_embed", `grep -rn 'atif_models' packages/atif-cli/` returns nothing. The live atif-models edge is atif-analytics' alone, its import sites led by `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:28-29`. -- **Every cross-package import in this workspace is INDENTED** — inside a function body or a - `TYPE_CHECKING` block — because `PLC0415` is ignored workspace-wide to satisfy the lean-import +- **Every cross-package import in this workspace is INDENTED** (inside a function body or a + `TYPE_CHECKING` block) because `PLC0415` is ignored workspace-wide to satisfy the lean-import test (`pyproject.toml:164`). The exceptions are module-scope imports in atif-cli modules that are themselves only ever imported inside a command body: `packages/atif-cli/src/atif_cli/converter_adapter.py:36-38` and @@ -270,11 +270,11 @@ forbidden_modules = ["atif_converter", "atif_corpus", "atif_duck", "atif_embed", confirmed at the site. **Drift risk:** a new workspace member that is not added to `root_packages` (`pyproject.toml:463`) is silently -unchecked — import-linter reports every contract kept while the new package imports whatever it +unchecked: import-linter reports every contract kept while the new package imports whatever it likes. Mitigation: adding a `packages/*` member means adding it to `root_packages` and giving it either a `layers` contract or a place in the `independence` list in the same commit. -## `LlmStructuredProvider` — the structured-output port +## `LlmStructuredProvider`: the structured-output port **Producer:** `packages/atif-models/src/atif_models/domain/ports.py:116` @@ -288,11 +288,11 @@ either a `layers` contract or a place in the `independence` list in the same com `packages/atif-analytics/src/atif_analytics/application/use_cases/friction.py:534`) - `packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py:90` (`packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py:141`, `packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py:364`) -- `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:32` — the provider +- `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:32`: the provider factory and the usage/budget plumbing. -- `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:143` — the one production +- `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:143`: the one production implementation. -- `packages/atif-analytics/tests/analytics_fixtures.py:237` — the deterministic double. +- `packages/atif-analytics/tests/analytics_fixtures.py:237`: the deterministic double. **Shape:** @@ -314,7 +314,7 @@ class LlmStructuredProvider(Protocol): - **The error taxonomy is binary and load-bearing.** `RefusalError` is terminal and `ProviderUnavailable` is retryable (`packages/atif-models/src/atif_models/domain/ports.py:43-58`), - and the pipelines route on exactly that split — a third error type would fall through to neither + and the pipelines route on exactly that split: a third error type would fall through to neither the refusal sidecar nor the retry queue. - **The Protocol says nothing about token accounting, yet every consumer reads it off the concrete provider.** `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:70` and `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:129` both reach for `getattr(provider, "usage", None)` because @@ -323,7 +323,7 @@ class LlmStructuredProvider(Protocol): - **`CallUsage` counts are PER CALL, and some of them are subsets of others.** `reasoning_tokens` is a subset of `output_tokens` and `cached_tokens` a subset of `input_tokens` (`packages/atif-models/src/atif_models/domain/ports.py:64-70`), so `estimate_cost` at `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:73-77` prices only `input_tokens` and - `output_tokens` — adding the subsets would double-charge. + `output_tokens`; adding the subsets would double-charge. - **`UsageAccumulator` must be thread-safe, not task-safe.** The lock is a `threading.Lock` (`packages/atif-models/src/atif_models/domain/ports.py:87`) because adapters dispatch blocking `invoke_model` through `anyio.to_thread`, reasoning stated at `packages/atif-models/src/atif_models/domain/ports.py:16-22`. @@ -331,7 +331,7 @@ class LlmStructuredProvider(Protocol): overshoot bound as `max_cost_usd` plus at most `BUDGET_CHECK_BATCH` units in flight; a consumer treating `max_cost_usd` as a hard ceiling is wrong. - **`pricing_in` / `pricing_out` on `ModelSpec` are USD per 1,000,000 tokens**, stated at - `packages/atif-models/src/atif_models/domain/registry.py:42-43`, and `None` when unknown — which + `packages/atif-models/src/atif_models/domain/registry.py:42-43`, and `None` when unknown, which `estimate_cost` (`packages/atif-models/src/atif_models/domain/registry.py:125`) turns into a `None` cost rather than a zero. - **`OpenAiBedrockProvider` never names the port either**, and no test binds it to one. The single @@ -355,13 +355,13 @@ mismatched adapter fails the typecheck gate. **Consumers:** -- `packages/atif-duck/src/atif_duck/infrastructure/registry.py:107-117` — `_EDGE_COLUMNS`, the +- `packages/atif-duck/src/atif_duck/infrastructure/registry.py:107-117`: `_EDGE_COLUMNS`, the same keys in the same order with DuckDB types. -- `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:300-301` — reads the +- `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:300-301`: reads the `uuid` field for the conflicts pipeline's returned-uuid validity guard. -- `packages/atif-corpus/src/atif_corpus/application/materialize.py:224-227` — writes the lines and +- `packages/atif-corpus/src/atif_corpus/application/materialize.py:224-227`: writes the lines and owns line termination. -- `packages/atif-duck/src/atif_duck/domain/catalog.py:330` — the `messages` view is defined as +- `packages/atif-duck/src/atif_duck/domain/catalog.py:330`: the `messages` view is defined as exactly this surface. **Shape:** @@ -388,7 +388,7 @@ EDGE_FIELDS: tuple[str, ...] = ( `packages/atif-corpus/src/atif_corpus/domain/ports.py:30-33` and honoured at `packages/atif-corpus/src/atif_corpus/application/materialize.py:224-227`; a producer that terminated its own lines would double them. - **`parent_uuid` is a nullable string, and atif-duck declares it `VARCHAR` outright** rather than - letting JSON union inference decide — the reason is recorded at `packages/atif-duck/src/atif_duck/infrastructure/registry.py:103-106`. + letting JSON union inference decide; the reason is recorded at `packages/atif-duck/src/atif_duck/infrastructure/registry.py:103-106`. - **`tool_use_ids` means two different things by record type**: tool_use block ids for assistant records, the `tool_use_id` of tool_result blocks for user records (`packages/atif-converter/src/atif_converter/domain/edges.py:11-13`). One field, two semantics, discriminated by the sibling `type` column. @@ -396,11 +396,11 @@ EDGE_FIELDS: tuple[str, ...] = ( (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:38-40`, fidelity gap 7 `UUID_NOT_PRESERVED`), which is why the `messages` view is reconstructed from edges rather than from steps. - **`records_total` in the loss report equals this file's line count.** Asserted at - `packages/atif-converter/src/atif_converter/domain/fidelity.py:91-95` — both derive from the same + `packages/atif-converter/src/atif_converter/domain/fidelity.py:91-95`. Both derive from the same raw census, so a consumer may cross-check one against the other. **Drift risk:** the keys are declared twice in packages that cannot import each other, and a -new key added on the producer side is simply invisible to `_EDGE_COLUMNS` — a projection reader +new key added on the producer side is simply invisible to `_EDGE_COLUMNS`: a projection reader drops unknown keys without erroring. Mitigation: `EDGE_FIELDS` is a single tuple; changing it means editing `packages/atif-duck/src/atif_duck/infrastructure/registry.py:107-117` in the same commit. @@ -410,16 +410,16 @@ editing `packages/atif-duck/src/atif_duck/infrastructure/registry.py:107-117` in **Consumers:** -- `packages/atif-embed/src/atif_embed/application/embed.py:35` — imports every one of them; annotated at +- `packages/atif-embed/src/atif_embed/application/embed.py:35`: imports every one of them; annotated at `packages/atif-embed/src/atif_embed/application/embed.py:46-47` and `packages/atif-embed/src/atif_embed/application/embed.py:65-67`. -- `packages/atif-embed/src/atif_embed/infrastructure/cohere_bedrock.py:285` — the +- `packages/atif-embed/src/atif_embed/infrastructure/cohere_bedrock.py:285`: the `EmbeddingProvider` adapter. -- `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:377` — the `VectorStorePort` +- `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:377`: the `VectorStorePort` adapter. -- `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:156` — the `TextRowsPort` +- `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:156`: the `TextRowsPort` adapter. -- `packages/atif-embed/tests/embed_fixtures.py:171` — the deterministic `EmbeddingProvider` double. -- `packages/atif-embed/tests/test_embed_use_case.py:458` — a recording `TextRowsPort` double. +- `packages/atif-embed/tests/embed_fixtures.py:171`: the deterministic `EmbeddingProvider` double. +- `packages/atif-embed/tests/test_embed_use_case.py:458`: a recording `TextRowsPort` double. **Shape:** @@ -461,17 +461,18 @@ class TextRowsPort(Protocol): loss at `packages/atif-embed/src/atif_embed/domain/ports.py:47-51`; `packages/atif-embed/src/atif_embed/application/embed.py:192-198` filters the `None`s, counts them as skipped, and leaves those uuids for the next pass. - **The store's `(model, dim)` stamp is checked BEFORE any append**, `packages/atif-embed/src/atif_embed/application/embed.py:166-175`, using the - identity `table_identity()` returns — `None` there means empty and any provider may claim it. + identity `table_identity()` returns; `None` there means empty and any provider may claim it. - **`add_chunk` accepts a polars frame with a fixed-size `pl.Array`, not a `pl.List`.** The schema is built at `packages/atif-embed/src/atif_embed/application/embed.py:216-238` and the reason is stated at `packages/atif-embed/src/atif_embed/application/embed.py:212-215`: a variable-size list is rejected by Lance for indexing. - **`iter_unembedded`'s laziness bounds only the CALLER's residency.** The port says so explicitly - at `packages/atif-embed/src/atif_embed/domain/ports.py:111-114` — peak resident text inside an implementation is each adapter's own + at `packages/atif-embed/src/atif_embed/domain/ports.py:111-114`: peak resident text inside an implementation is each adapter's own problem, and `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:54-60` establishes it by batching on BYTES (4 MiB) rather than on file count. - **Staleness is decided by hash, not by uuid presence.** A uuid under a different `text_hash` is yielded with `replaces_existing=True` (`packages/atif-embed/src/atif_embed/domain/ports.py:105-109`), and the caller must delete before - appending or the uuid fans out to two vectors — `packages/atif-embed/src/atif_embed/application/embed.py:203-209`. + appending or the uuid fans out to two vectors + (`packages/atif-embed/src/atif_embed/application/embed.py:203-209`). - **The CLI does not inject these ports.** `packages/atif-cli/src/atif_cli/app.py:826-831` calls `run_backfill` with only `corpus_root`, `settings`, `limit`, and `dry_run`; the use case constructs each default adapter itself under a deferred import (`packages/atif-embed/src/atif_embed/application/embed.py:103-110`, `packages/atif-embed/src/atif_embed/application/embed.py:157-159`) @@ -489,7 +490,7 @@ class TextRowsPort(Protocol): **Drift risk:** `dimension` is read once per run and stamped on every row (`packages/atif-embed/src/atif_embed/application/embed.py:161-162`), so a provider whose width depends on the input rather than on configuration would write a store whose rows disagree with their own `dim` column. Mitigation: `dimension` is a property fixed at adapter -construction from settings — `packages/atif-embed/src/atif_embed/infrastructure/cohere_bedrock.py:305-307` reads it from +construction from settings: `packages/atif-embed/src/atif_embed/infrastructure/cohere_bedrock.py:305-307` reads it from `packages/atif-embed/src/atif_embed/infrastructure/settings.py:31`, never from a response. ## `ConverterPort` and `ConversionOutput` @@ -499,15 +500,15 @@ construction from settings — `packages/atif-embed/src/atif_embed/infrastructur **Consumers:** -- `packages/atif-corpus/src/atif_corpus/application/materialize.py:92` — imports the port; +- `packages/atif-corpus/src/atif_corpus/application/materialize.py:92`: imports the port; annotated at `packages/atif-corpus/src/atif_corpus/application/materialize.py:193` and `packages/atif-corpus/src/atif_corpus/application/materialize.py:480`. -- `packages/atif-cli/src/atif_cli/converter_adapter.py:38-69` — `RealConverter`, the production +- `packages/atif-cli/src/atif_cli/converter_adapter.py:38-69`: `RealConverter`, the production adapter and the only module importing both atif-converter and atif-corpus. -- `packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py:16-65` — the test adapter +- `packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py:16-65`: the test adapter that ships in `src/`, not in `tests/`. -- `packages/atif-cli/tests/test_converter_adapter.py:17` — asserts the mapping and binds +- `packages/atif-cli/tests/test_converter_adapter.py:17`: asserts the mapping and binds `RealConverter` to the port annotation at `packages/atif-cli/tests/test_converter_adapter.py:91`. -- `packages/atif-cli/src/atif_cli/app.py:399` — wires `RealConverter()` into the use case. +- `packages/atif-cli/src/atif_cli/app.py:399`: wires `RealConverter()` into the use case. **Shape:** @@ -542,7 +543,7 @@ class ConverterPort(Protocol): (`packages/atif-cli/tests/test_converter_adapter.py:89-90`) states that the assignment is what ty verifies. Delete that test and a signature change on either side becomes a runtime `AttributeError` at materialize time. -- **The port is typed to `docs/CONTRACT.md`'s artifact shapes, not to converter internals** — +- **The port is typed to `docs/CONTRACT.md`'s artifact shapes, not to converter internals**: loosely typed `dict[str, Any]` / `list[str]` fields instead of the converter's own `ConversionResult`. The docstring at `packages/atif-corpus/src/atif_corpus/domain/ports.py:3-10` names the independence contract as the reason, so the weak typing is the contract, not an omission. @@ -551,21 +552,21 @@ class ConverterPort(Protocol): raises `TrajectoryValidationError` for an invalid trajectory rather than materializing it (`packages/atif-cli/src/atif_cli/converter_adapter.py:63-64`, reasoning at `packages/atif-cli/src/atif_cli/converter_adapter.py:21-25`). - **`trajectory_dict` is the ENRICHED trajectory and is NOT yet compact-serialized.** The writer - owns `separators=(",", ":")` — the mapping decision is stated at `packages/atif-cli/src/atif_cli/converter_adapter.py:13-14` and + owns `separators=(",", ":")`; the mapping decision is stated at `packages/atif-cli/src/atif_cli/converter_adapter.py:13-14` and the writer applies it at `packages/atif-corpus/src/atif_corpus/application/materialize.py:222`. - **`loss_report_dict` is `LossReport.to_json()`-shaped with enum members flattened to strings** (`packages/atif-cli/src/atif_cli/converter_adapter.py:15-16`), which is what makes it readable by atif-duck's `_LOSS_REPORT_COLUMNS` projection at `packages/atif-duck/src/atif_duck/infrastructure/registry.py:122-131`. - **Importing the adapter drags harbor.** `packages/atif-cli/src/atif_cli/converter_adapter.py:27-29` forbids importing it at - `atif_cli.app` module scope, and `packages/atif-cli/src/atif_cli/app.py:387` obeys by importing inside the command body — + `atif_cli.app` module scope, and `packages/atif-cli/src/atif_cli/app.py:387` obeys by importing inside the command body, enforced by the fresh-interpreter lean-import test named at `pyproject.toml:164`. **Drift risk:** the `dict[str, Any]` fields mean a converter that renames a trajectory key type-checks perfectly and produces artifacts atif-duck's explicit projections silently null out; and because the implementer never names the port, a `convert` signature change on either side is caught by exactly one assertion. Mitigation: -`packages/atif-cli/tests/test_converter_adapter.py:88-92` is that assertion — treat it as part of -the contract, not as a redundant smoke test — and +`packages/atif-cli/tests/test_converter_adapter.py:88-92` is that assertion (treat it as part of +the contract, not as a redundant smoke test), and `packages/atif-cli/tests/test_converter_adapter.py:17-60` pins the field mapping while atif-duck's `DESCRIBE` drift test catches a projection that stops matching. @@ -576,13 +577,13 @@ and `packages/atif-duck/src/atif_duck/domain/catalog.py:405` (`DEFAULT_PRICING`) **Consumers:** -- `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1066-1083` — the `cost_estimate` +- `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1066-1083`: the `cost_estimate` macro, which consumes both. -- `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1020` — `register_macros` resolves +- `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1020`: `register_macros` resolves the pricing override against `DEFAULT_PRICING`. -- `packages/atif-duck/src/atif_duck/domain/catalog.py:72-75` — the `steps` view schema publishes +- `packages/atif-duck/src/atif_duck/domain/catalog.py:72-75`: the `steps` view schema publishes `prompt_tokens`, `completion_tokens`, `cached_tokens`, `cache_creation` to every SQL consumer. -- `packages/atif-duck/src/atif_duck/domain/catalog.py:347-350` — the agent-facing `cost_estimate` +- `packages/atif-duck/src/atif_duck/domain/catalog.py:347-350`: the agent-facing `cost_estimate` description carries the trust condition. **Shape:** @@ -608,22 +609,23 @@ DEFAULT_PRICING: dict[str, tuple[float, float]] = { - **`cached_tokens` is the cache-READ subset of `prompt_tokens`**, not an additional quantity (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:380-381`). `cost_estimate` therefore prices `prompt_tokens - cached_tokens` and documents that identity at `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1043-1049`; cache reads are charged nothing. -- **`in_rate` / `out_rate` are USD per 1,000,000 tokens.** The unit appears in neither name — only +- **`in_rate` / `out_rate` are USD per 1,000,000 tokens.** The unit appears in neither name: only the comment at `packages/atif-duck/src/atif_duck/domain/catalog.py:395` and the `/ 1e6` at `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1071` carry it. - **`est_cost_usd` is USD and covers the PRICED steps only.** It is meaningful only when `unpriced_steps = 0`, stated in each of these places: `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1053-1058`, `packages/atif-duck/src/atif_duck/domain/catalog.py:347-350`, and the `LEFT JOIN` shape itself at `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1079`. - **`unpriced_steps` counts only steps with a non-NULL `model_name`**, because user steps carry no - model and cost nothing — `packages/atif-duck/src/atif_duck/infrastructure/registry.py:1060-1063`. Counting them would put every conversation + model and cost nothing + (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:1060-1063`). Counting them would put every conversation above zero and mask real pricing gaps. - **Cache write and read multipliers are NOT modelled.** `packages/atif-duck/src/atif_duck/domain/catalog.py:402-404` states that the 1.25x / 2x / 0.1x tiers are out of scope, so `est_cost_usd` under-reports a cache-heavy session. - **A step's `model_name` matches a pricing row by dated-suffix-stripping prefix** (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:1080`), so `claude-haiku-4-5-20251001` prices as `claude-haiku-4-5`. -**Drift risk:** `cached_tokens` exists in two coordinate spaces under one name — atif-duck's +**Drift risk:** `cached_tokens` exists in two coordinate spaces under one name: atif-duck's per-step cache-read count (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:380`) and atif-models' per-call -`CallUsage.cached_tokens` (`packages/atif-models/src/atif_models/domain/ports.py:75`) — and neither +`CallUsage.cached_tokens` (`packages/atif-models/src/atif_models/domain/ports.py:75`). Neither name states its scope, so a cross-plane join or a copied formula silently mixes them. Mitigation: attribute the field by module path, and read the producer docstring (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:41-44` or @@ -631,20 +633,20 @@ attribute the field by module path, and read the producer docstring ## The `EmbeddingProviderMismatch` guard, declared twice -**Producer:** two coordinate declarations, coupled by contract rather than by import — +**Producer:** two coordinate declarations, coupled by contract rather than by import: `packages/atif-embed/src/atif_embed/domain/errors.py:29` with the rule at `packages/atif-embed/src/atif_embed/domain/embedding_guard.py:38`, and `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:33` with the rule at `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:46` **Consumers:** -- `packages/atif-embed/src/atif_embed/application/embed.py:166-175` — the WRITE path calls +- `packages/atif-embed/src/atif_embed/application/embed.py:166-175`: the WRITE path calls `ensure_store_matches` before appending. -- `packages/atif-duck/src/atif_duck/infrastructure/registry.py:830-861` — the READ/bind path calls +- `packages/atif-duck/src/atif_duck/infrastructure/registry.py:830-861`: the READ/bind path calls it before binding `message_embeddings` (guard-before-bind). -- `packages/atif-cli/src/atif_cli/duck_errors.py:26-31` — puts atif-duck's copy in +- `packages/atif-cli/src/atif_cli/duck_errors.py:26-31`: puts atif-duck's copy in `REGISTRATION_ERRORS` and maps it to exit 65 at `packages/atif-cli/src/atif_cli/duck_errors.py:74-81`. -- `packages/atif-embed/tests/test_guard_twin_pin.py:93` — reads both twin modules as source text +- `packages/atif-embed/tests/test_guard_twin_pin.py:93`: reads both twin modules as source text and requires the recovery hint to appear after the `raise` keyword. **Shape:** @@ -669,13 +671,13 @@ def ensure_store_matches( **Assumptions consumers make:** - **The two copies must stay behaviourally identical, including the message.** Both docstrings say - so — `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:5-11` and - `packages/atif-embed/src/atif_embed/domain/embedding_guard.py:15-18` — and the pin test is what + so (`packages/atif-duck/src/atif_duck/domain/embedding_guard.py:5-11` and + `packages/atif-embed/src/atif_embed/domain/embedding_guard.py:15-18`), and the pin test is what makes it more than a comment. - **The message text must be constructed INSIDE the `raise`.** Hoisting it to a local satisfies ruff's `EM102` / `TRY003` and defeats the twin pin, so both rules are suppressed on purpose at `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:67-74`. -- **A `None` on either stored value means "empty store, any provider may claim it"** — not +- **A `None` on either stored value means "empty store, any provider may claim it"**, not "unknown, be careful" (`packages/atif-duck/src/atif_duck/domain/embedding_guard.py:61-62`). - **`expected_dim=None` trusts `model_id` alone**, and dim is checked only because Cohere's single model id can emit different Matryoshka widths @@ -683,9 +685,9 @@ def ensure_store_matches( - **The two classes have different base classes and that matters at the CLI.** atif-duck's derives from `Exception` (`packages/atif-duck/src/atif_duck/domain/embedding_guard.py:33`) while atif-embed's derives from that package's `DomainError` (`packages/atif-embed/src/atif_embed/domain/errors.py:29`), so a bare - `except duckdb.Error` on the registration path would let atif-duck's escape as exit 1 — the + `except duckdb.Error` on the registration path would let atif-duck's escape as exit 1 (the reason `REGISTRATION_ERRORS` widens the caught set, stated at - `packages/atif-cli/src/atif_cli/duck_errors.py:14-18`. + `packages/atif-cli/src/atif_cli/duck_errors.py:14-18`). - **The recovery hint must not name a fixed home directory.** The store lives at `/embeddings_lance` by default, and both copies record that naming the wrong path makes the operator delete nothing @@ -694,7 +696,7 @@ def ensure_store_matches( **Drift risk:** the twin pin checks that the hint constant reaches the `raise`; it does not check that the two hint STRINGS are equal, so the copies could diverge in wording while both tests pass. -Mitigation: treat the two `RECOVERY_HINT` literals as one value — change both in the same commit, +Mitigation: treat the two `RECOVERY_HINT` literals as one value. Change both in the same commit, which is what both docstrings instruct. ## harbor's ATIF models, vendored and consumed by our ported converters @@ -705,14 +707,14 @@ which is what both docstrings instruct. **Consumers:** -- `packages/atif-converter/src/atif_converter/domain/claude_code_conversion.py:75` — `convert_claude_code_records`, our Claude Code converter, a parity port of harbor +- `packages/atif-converter/src/atif_converter/domain/claude_code_conversion.py:75`: `convert_claude_code_records`, our Claude Code converter, a parity port of harbor 0.22.0's, building `Trajectory` / `Step` / `ToolCall` / `Metrics` from the public models. -- `packages/atif-converter/src/atif_converter/domain/codex_conversion.py:781` — `convert_codex_records`, the Codex counterpart. -- `packages/atif-converter/src/atif_converter/infrastructure/harbor_adapter.py` — `validate_trajectory`, the only call into the vendored validator. -- `packages/atif-converter/tests/test_dev_only_imports_guard.py` — the `ast` guard that keeps `harbor` and `litellm` out of every `src/` tree. -- `packages/atif-converter/tests/test_vendored_atif.py` — the conformance test: the vendored files equal the installed harbor's, and both agree on the goldens. -- `packages/atif-converter/tests/harbor_oracle.py:94` and `:111` — the parity ORACLE: harbor's private converters, reached from the tests - only, with frozen goldens and a live-corpus diff. +- `packages/atif-converter/src/atif_converter/domain/codex_conversion.py:781`: `convert_codex_records`, the Codex counterpart. +- `packages/atif-converter/src/atif_converter/infrastructure/harbor_adapter.py`: `validate_trajectory`, the only call into the vendored validator. +- `packages/atif-converter/tests/test_dev_only_imports_guard.py`: the `ast` guard that keeps `harbor` and `litellm` out of every `src/` tree. +- `packages/atif-converter/tests/test_vendored_atif.py`: the conformance test (the vendored files equal the installed harbor's, and both agree on the goldens). +- `packages/atif-converter/tests/harbor_oracle.py:94` and `:111`: the parity ORACLE (harbor's private converters, reached from the tests + only), with frozen goldens and a live-corpus diff. **Shape:** @@ -743,13 +745,13 @@ DEV_ONLY: frozenset[str] = frozenset({"harbor", "litellm"}) **Consumers:** -- `pyproject.toml:446-452` — commitizen's `version_files`, which rewrites every pin on each bump. -- `packages/atif-cli/pyproject.toml:56-60` — `[tool.uv.sources]`, the local-workspace resolution +- `pyproject.toml:446-452`: commitizen's `version_files`, which rewrites every pin on each bump. +- `packages/atif-cli/pyproject.toml:56-60`: `[tool.uv.sources]`, the local-workspace resolution that these pins deliberately do NOT express. -- `packages/atif-analytics/pyproject.toml:56` — the same pairing for its single sibling dependency. -- `pyproject.toml:399-406` — `[tool.commitizen] version = "0.1.0"`, the one version this repo +- `packages/atif-analytics/pyproject.toml:56`: the same pairing for its single sibling dependency. +- `pyproject.toml:399-406`: `[tool.commitizen] version = "0.1.0"`, the one version this repo publishes. -- `pyproject.toml:17-18` — the distribution is named `atif-sql` at the workspace root, while the +- `pyproject.toml:17-18`: the distribution is named `atif-sql` at the workspace root, while the console script's module stays `atif_cli` in a member. **Shape:** @@ -779,7 +781,7 @@ version_files = [ - **`[tool.uv.sources]` is invisible to an external installer.** uv's build backend does not translate a workspace source into a version constraint, so a bare name would ship as `Requires-Dist: atif-duck` and resolve from PyPI to whatever the newest release of that name is, - owned by whoever owns it — the reasoning is inline at `packages/atif-cli/pyproject.toml:24-31`. + owned by whoever owns it; the reasoning is inline at `packages/atif-cli/pyproject.toml:24-31`. - **One version covers the whole repository.** commitizen reads its own `version` key rather than the PEP 621 field (`pyproject.toml:401-403`), and `version_files` propagates it to the published `[project] version` and to every member manifest; a separate number per member for one artifact is the @@ -792,36 +794,36 @@ version_files = [ (`pyproject.toml:422-433`), because `uv.lock` records every member's version and `mise run lock:check` would otherwise fail on the release commit itself. - **A breaking change moves 0.1.0 to 0.2.0, not 1.0.0**, because `major_version_zero = true` - (`pyproject.toml:416`) — reaching 1.0.0 is a decision, not a side effect of a `!` in a subject + (`pyproject.toml:416`); reaching 1.0.0 is a decision, not a side effect of a `!` in a subject line. **Drift risk:** a hand-edited pin, or a renamed member whose pin line stops matching its `version_files` regex, goes stale silently until a release. Mitigation: `--check-consistency` fails -rather than skipping a file that no longer contains the current version — the guarantee is stated at +rather than skipping a file that no longer contains the current version; the guarantee is stated at `pyproject.toml:434-436`. ## Other contracts -- **`LossReport.to_json()`** — producer +- **`LossReport.to_json()`**: producer `packages/atif-converter/src/atif_converter/domain/fidelity.py:114`, whose docstring calls the keys wire contract; consumed by `packages/atif-duck/src/atif_duck/infrastructure/registry.py:122-131` as `_LOSS_REPORT_COLUMNS` and by `packages/atif-cli/src/atif_cli/converter_adapter.py:67`. `record_counts` and `gaps_observed` stay `JSON` because they are an enum-keyed dict and a sorted list of enum values. -- **The `meta.json` provenance record** — written at +- **The `meta.json` provenance record**: written at `packages/atif-corpus/src/atif_corpus/application/materialize.py:230-237`, projected at `packages/atif-duck/src/atif_duck/infrastructure/registry.py:134-141`. `harbor_version` and `converter_version` are supplied by the CLI (`packages/atif-cli/src/atif_cli/app.py:401-402`), so the corpus records which converter built it without atif-corpus importing either. -- **The embeddings-store row shape** — the Arrow fields written at +- **The embeddings-store row shape**: the Arrow fields written at `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:115-121`, of which the `message_embeddings` view exposes only some (`packages/atif-duck/src/atif_duck/domain/catalog.py:218-224`). `text_hash` and `truncated` are unreachable from SQL, so a query cannot distinguish a row embedded from its head alone from a complete one. -- **The Lance schema version sidecar** — `SCHEMA_VERSION = 2` and `schema_version.json` at +- **The Lance schema version sidecar**: `SCHEMA_VERSION = 2` and `schema_version.json` at `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:61-67`, kept as a sidecar rather than a column because reading it must not require the table. -- **The user-step authorship rule table** — `AUTHOR_VALUES`, `AUTHOR_STRIP_CHARS`, +- **The user-step authorship rule table**: `AUTHOR_VALUES`, `AUTHOR_STRIP_CHARS`, `AUTHOR_PREFIX_RULES`, and `INTERRUPT_PREFIXES` at `packages/atif-duck/src/atif_duck/domain/authorship.py:37-102`, which the `step_author` macro and the `user_steps`, `human_turns`, and `session_outcomes` views compile from. atif-analytics can't @@ -830,33 +832,33 @@ rather than skipping a file that no longer contains the current version — the text by `packages/atif-duck/tests/test_authorship_twin_pin.py:64`, and `packages/atif-cli/tests/test_authorship_parity.py:53` runs the SQL and the Python over the same steps. -- **The analytics parquet layout** — the artifact names at +- **The analytics parquet layout**: the artifact names at `packages/atif-analytics/src/atif_analytics/domain/layout.py:25-41`, of which atif-duck pins the pipeline directories as `_ANALYTICS_SOURCES` (`packages/atif-duck/src/atif_duck/infrastructure/analytics.py:61-66`). `REFUSALS_DIRNAME` (`packages/atif-analytics/src/atif_analytics/domain/layout.py:39`) has no view, so the refusal audit its docstring calls queryable is not reachable from SQL. -- **The CLI exit-code taxonomy** — `EXIT_CODES` at +- **The CLI exit-code taxonomy**: `EXIT_CODES` at `packages/atif-cli/src/atif_cli/errors.py:25-39`, consumed at many sites in `packages/atif-cli/src/atif_cli/app.py` and mapped from DuckDB exceptions at `packages/atif-cli/src/atif_cli/duck_errors.py:34-63`. Code 78 is the load-bearing one: it means an operator must act, and unattended lanes suppress retries on it (`packages/atif-cli/src/atif_cli/errors.py:35-37`). -- **`PendingText` and the text stamp** — `packages/atif-embed/src/atif_embed/domain/text_stamp.py:50`, +- **`PendingText` and the text stamp**: `packages/atif-embed/src/atif_embed/domain/text_stamp.py:50`, with `MAX_EMBEDDABLE_CHARS = 50_000` at `packages/atif-embed/src/atif_embed/domain/text_stamp.py:35` in CHARACTERS per text; the same constant governs both the row's `truncated` flag and the adapter's wire-level clip so the two can never disagree (`packages/atif-embed/src/atif_embed/domain/text_stamp.py:31-34`). -- **The `steps`-view rendering semantics, mirrored without a shared symbol** — +- **The `steps`-view rendering semantics, mirrored without a shared symbol**: `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:23-33` reproduces atif-duck's `steps` rules by hand (message-union flattening, `extra.source_uuids[0]` as the step key, `agent` → `assistant`, error recovery from `observation.results[].extra.tool_result_metadata.is_error`), and `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:12-25` reproduces some of them again. -- **The `ATIF_SQL_` settings prefix** — the one env namespace across every member; +- **The `ATIF_SQL_` settings prefix**: the one env namespace across every member; `packages/atif-embed/src/atif_embed/infrastructure/settings.py:31` pins `output_dimension: Literal[256, 512, 1024, 1536] = 1024`, which is the value the static catalog hardcodes as `FLOAT[1024]`. -- **The lean-import contract** — `pyproject.toml:164` and `pyproject.toml:181` record that `atif_cli.app` must not +- **The lean-import contract**: `pyproject.toml:164` and `pyproject.toml:181` record that `atif_cli.app` must not import duckdb, harbor, lancedb, boto3, or polars at module scope, asserted in a fresh interpreter by `packages/atif-cli/tests/test_lean_import.py`; the deferred-import sites exist because of it. diff --git a/docs/insights/debugging-guide.md b/docs/insights/debugging-guide.md index 5692871..e8983c1 100644 --- a/docs/insights/debugging-guide.md +++ b/docs/insights/debugging-guide.md @@ -9,7 +9,7 @@ this guide is organized around them: 1. **The process exit code.** `EXIT_CODES` is one dict (`packages/atif-cli/src/atif_cli/errors.py:25-39`) and it is the primary diagnostic. Not every key is reachable. -2. **The classified error envelope on stderr** — one readable line on a TTY, +2. **The classified error envelope on stderr**: one readable line on a TTY, one JSON object on a pipe (`packages/atif-cli/src/atif_cli/output.py:274-281`). 3. **The loguru sink.** Exactly one, added in `main()`, stderr, WARNING and up (`packages/atif-cli/src/atif_cli/app.py:1133-1134`). @@ -27,7 +27,7 @@ The exit-code contract, read from source and confirmed by running the CLI: | 70 | `runtime_error` | everything else the adapter raises | | 78 | `terminal_state`, `suspicious_scan` | an operator has to act; retrying cannot succeed | | 127 | `harbor_missing` | RETIRED: kept so the table never renumbers; no code path raises it since the conversion became ours | -| 1 | *(none)* | an unhandled exception — outside the taxonomy | +| 1 | *(none)* | an unhandled exception, outside the taxonomy | One of these codes exists to be told apart from a crash rather than from another code. **`suspicious_scan: 78`** is raised only by @@ -53,27 +53,27 @@ On a pipe the envelope is the last line on stderr, in this shape: | Symptom | Likely surface | First check | Citation | | --- | --- | --- | --- | | `query` exits 64, kind `parse_error` | `duckdb.ParserException`, or no SQL argument at all | Run `atif-sql schema --format json` for the real view and column names; a bare `atif-sql query` also exits 64 | `packages/atif-cli/src/atif_cli/duck_errors.py:37-43`, `packages/atif-cli/src/atif_cli/app.py:599-610` | -| `query` exits 65, kind `catalog_error` | `duckdb.CatalogException` — the view, macro, or column does not exist | `atif-sql schema` answers from the static catalog with no DuckDB bind, so it works even on an unmaterialized corpus | `packages/atif-cli/src/atif_cli/duck_errors.py:44-50`, `packages/atif-cli/src/atif_cli/app.py:1086-1122` | +| `query` exits 65, kind `catalog_error` | `duckdb.CatalogException`: the view, macro, or column does not exist | `atif-sql schema` answers from the static catalog with no DuckDB bind, so it works even on an unmaterialized corpus | `packages/atif-cli/src/atif_cli/duck_errors.py:44-50`, `packages/atif-cli/src/atif_cli/app.py:1086-1122` | | `query` exits 70 with `IO Error: No files found that match the pattern .../sessions/*/meta.json` | The corpus is not materialized; `register_raw`'s glob matches nothing | `atif-sql status`; a `watermark: never materialized` line means run `atif-sql materialize` | `packages/atif-cli/src/atif_cli/duck_errors.py:51-57`, `packages/atif-duck/src/atif_duck/infrastructure/registry.py:306-309` | | `query` or `search` exits 65, kind `embedding_mismatch` | The Lance store's stamped `(model, dim)` differs from the active embedder | The message names both sides; either point `ATIF_SQL_EMBED_MODEL_ID` at the model that wrote the store, or delete the store | `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:61-80`, `packages/atif-cli/src/atif_cli/duck_errors.py:74-81` | | `search` exits 2, kind `no_embeddings` | The store is empty or absent, so `register_vss` bound an empty fallback table | `atif-sql query 'SELECT count(*) FROM message_embeddings'`; zero rows confirms it | `packages/atif-cli/src/atif_cli/app.py:930-941`, `packages/atif-duck/src/atif_duck/infrastructure/registry.py:896-914` | | A bare `atif-sql embed` exits 64 | The scope guard, which runs before any Bedrock client is built | Pass `--limit N`, `--all`, or `--dry-run`; this is the accidental-full-backfill guard, not a fault | `packages/atif-cli/src/atif_cli/app.py:810-820` | -| `embed` exits 78 and the cron lane stops trying | A `DomainError` with `terminal = True` — the store or its config needs an operator | Read the terminal marker file the refresh lane drops: three lines, store path, mtime, reason | `packages/atif-embed/src/atif_embed/domain/errors.py:14-26`, `scripts/atif-sql-refresh.sh:244-249` | +| `embed` exits 78 and the cron lane stops trying | A `DomainError` with `terminal = True`; the store or its config needs an operator | Read the terminal marker file the refresh lane drops: three lines, store path, mtime, reason | `packages/atif-embed/src/atif_embed/domain/errors.py:14-26`, `scripts/atif-sql-refresh.sh:244-249` | | A session directory exists under `sessions/` but no view returns its rows | Torn artifact set: the dir has no `meta.json`, so the meta gate excludes it from every reader | Grep the WARNING; `meta.json` is written last and completes the set | `packages/atif-duck/src/atif_duck/infrastructure/registry.py:173-177`, `docs/CONTRACT.md:22-31` | -| `atif-sql status` and `SELECT count(*) FROM sessions` disagree | `status` counts directories with no meta check; the view applies the meta gate | The difference is exactly the number of incomplete dirs — use it as the torn-set count | `packages/atif-cli/src/atif_cli/app.py:482-486`, `packages/atif-duck/src/atif_duck/infrastructure/registry.py:221` | +| `atif-sql status` and `SELECT count(*) FROM sessions` disagree | `status` counts directories with no meta check; the view applies the meta gate | The difference is exactly the number of incomplete dirs; use it as the torn-set count | `packages/atif-cli/src/atif_cli/app.py:482-486`, `packages/atif-duck/src/atif_duck/infrastructure/registry.py:221` | | `materialize` prints all zeroes and reads as an idle, complete corpus | Every session was unreadable; an unreadable session lands in no other counter | Read the `unreadable` field, not just `materialized` | `packages/atif-cli/src/atif_cli/app.py:312-346`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:136-142` | -| `materialize` aborts with a traceback naming `SuspiciousEmptyScanError`, exit 1 | The scan found zero sessions over a non-empty corpus — almost always a wrong `source_root` | Compare the `source_root` line in `atif-sql status` against where the transcripts actually are | `packages/atif-corpus/src/atif_corpus/application/materialize.py:97-104`, raise at `packages/atif-corpus/src/atif_corpus/application/materialize.py:553-559` | +| `materialize` aborts with a traceback naming `SuspiciousEmptyScanError`, exit 1 | The scan found zero sessions over a non-empty corpus, almost always a wrong `source_root` | Compare the `source_root` line in `atif-sql status` against where the transcripts actually are | `packages/atif-corpus/src/atif_corpus/application/materialize.py:97-104`, raise at `packages/atif-corpus/src/atif_corpus/application/materialize.py:553-559` | | A changed session never materializes | Quiescence: the newest source mtime is younger than `quiesce_seconds` (default 300) | `atif-sql status` reports it under `live`; `--force` overrides staleness but never quiescence | `packages/atif-corpus/src/atif_corpus/domain/sessions.py:81-104`, `packages/atif-corpus/src/atif_corpus/domain/sessions.py:172-173`, `packages/atif-corpus/src/atif_corpus/domain/sessions.py:201-203` | | A session stays `live` forever and warns about a FUTURE mtime | The writing host's clock is ahead, so the age is negative and never meets the threshold | Fix the clock on the host writing the transcript | `packages/atif-corpus/src/atif_corpus/domain/sessions.py:96-103` | | `materialize` or `status` aborts inside `read_watermark` with `ValueError` or `TypeError` | A watermark whose values are not numbers; the `int(value)` coercion sits outside the try | Delete `watermark.json`; a missing watermark costs one full re-materialization pass and is always safe | `packages/atif-corpus/src/atif_corpus/application/materialize.py:160-176` | | Every session fails conversion at once, on the same `ConversionError` | The converter is a port of harbor 0.23.0's, so a systematic failure is a bug in the port or a transcript shape it never saw | Run the parity oracle: `ATIF_PARITY_LIMIT=0 uv run pytest packages/atif-converter/tests/test_parity_live.py`, and read the diff paths | `packages/atif-converter/tests/harbor_oracle.py:142` | -| One session fails with `N source file(s) changed while converting …` | The session is still being written; the snapshot re-check refused to publish inconsistent artifacts | Retry once the session goes quiet — this is self-clearing | `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:93-102`, `packages/atif-converter/src/atif_converter/domain/errors.py:57-70` | +| One session fails with `N source file(s) changed while converting …` | The session is still being written; the snapshot re-check refused to publish inconsistent artifacts | Retry once the session goes quiet; this is self-clearing | `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:93-102`, `packages/atif-converter/src/atif_converter/domain/errors.py:57-70` | | `convert` prints a trajectory and then exits 65 | The enriched trajectory failed harbor's `TrajectoryValidator` | Read `validation_errors` in the JSON it already printed | `packages/atif-cli/src/atif_cli/app.py:283-304`, `packages/atif-cli/src/atif_cli/converter_adapter.py:62-64` | -| `loss_report.json` shows `records_dropped > 0` | Fidelity gap 2: harbor 0.22.0 converts only user and assistant records | Check `gaps_observed` against the named gaps — these are documented losses, not defects | `packages/atif-converter/src/atif_converter/domain/fidelity.py:44-79` | +| `loss_report.json` shows `records_dropped > 0` | Fidelity gap 2: harbor 0.22.0 converts only user and assistant records | Check `gaps_observed` against the named gaps; these are documented losses, not defects | `packages/atif-converter/src/atif_converter/domain/fidelity.py:44-79` | | `analyze` summary carries `budget_exhausted: true` and a stage `{"skipped": "budget_exhausted"}` | The dollar ceiling was crossed against running actual usage | Read `llm_spent_usd` in the same summary; raise `--max-cost-usd` or wait for the next run | `packages/atif-analytics/src/atif_analytics/application/analyze.py:164-179`, `packages/atif-analytics/src/atif_analytics/application/analyze.py:202-204` | -| An LLM stage skips the same sessions every run, forever | The retry queue is exhausted at 5 attempts; `attempts >= 5` with `completed_at IS NULL` is permanent and nothing resets it | `sqlite3 /analytics/state.db 'SELECT * FROM retry_queue'` — note the pipeline is `user_friction`, never `friction` | `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/retry_queue.py:176-193`, `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:35-41` | +| An LLM stage skips the same sessions every run, forever | The retry queue is exhausted at 5 attempts; `attempts >= 5` with `completed_at IS NULL` is permanent and nothing resets it | `sqlite3 /analytics/state.db 'SELECT * FROM retry_queue'` (note the pipeline is `user_friction`, never `friction`) | `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/retry_queue.py:176-193`, `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:35-41` | | An LLM run stalls for minutes with no output | Bedrock throttling under tenacity: 10 attempts, exponential 2 s to 60 s | The per-backoff WARNING names the attempt number and the sleep | `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:251-257`, `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:112-121` | -| A classify row reads `[refused]`, or a refusals sidecar row appears | `RefusalError` — a content filter or a refusal `finish_reason` | Nothing to fix here: a refusal is terminal, checkpointed, and never retried | `packages/atif-models/src/atif_models/domain/ports.py:43-48`, `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:284-290` | +| A classify row reads `[refused]`, or a refusals sidecar row appears | `RefusalError`: a content filter or a refusal `finish_reason` | Nothing to fix here: a refusal is terminal, checkpointed, and never retried | `packages/atif-models/src/atif_models/domain/ports.py:43-48`, `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:284-290` | | An LLM unit lands in the retry queue with a truncation error | `finish_reason=length` twice, or once already at the effort floor | Raise `max_completion_tokens` or shrink the prompt; the automatic ladder has only one reachable rung | `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:291-297`, `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:229-249` | | `analyze` crashes with a raw traceback and exit 1 | The `analyze` command body has no `except`, so nothing maps analytics errors into `EXIT_CODES` | Read the traceback's innermost frame; the exit code carries no information here | `packages/atif-cli/src/atif_cli/app.py:739-757` | | `atif-sql examples` raises a bare `KeyError` naming `ARG_EXEMPLARS` or `DESCRIPTIONS` | A catalog drift tripwire firing at example-build time, not query time | Add the missing exemplar or description entry named in the message | `packages/atif-duck/src/atif_duck/domain/examples.py:150-158`, `packages/atif-duck/src/atif_duck/domain/examples.py:165-175` | @@ -109,7 +109,7 @@ shell-side refresh log. | Refresh-script log | a file appended under the script's `.run/` sibling, one `date -Is`-stamped line per event; the directory is gitignored, so the path is named in prose only | `refresh complete (mode=`, `skip[`, `TERMINAL:`, `FATAL:` | `scripts/atif-sql-refresh.sh:100-103`, `scripts/atif-sql-refresh.sh:337` | | Refresh log, machine-read subset | `atif-sql cron status` stdout | only the completion and skip line shapes are parsed | `packages/atif-cli/src/atif_cli/cron.py:55-56` | | Lane lock state | `atif-sql cron status`, from a nonblocking flock probe plus the pidfile | `RUNNING (pid …)` versus `idle` | `packages/atif-cli/src/atif_cli/cron.py:109-124`, `packages/atif-cli/src/atif_cli/cron.py:127-144` | -| Embed terminal marker | one file per corpus beside the refresh log; three lines — store path, store mtime, reason | the reason string, parsed out of the exit-78 envelope | `scripts/atif-sql-refresh.sh:238-249` | +| Embed terminal marker | one file per corpus beside the refresh log; three lines: store path, store mtime, reason | the reason string, parsed out of the exit-78 envelope | `scripts/atif-sql-refresh.sh:238-249` | | Analytics durable state | `/analytics/state.db`, sqlite in WAL mode, no read command | `retry_queue`, `budget_skips`, `session_checkpoint` | `packages/atif-analytics/src/atif_analytics/domain/layout.py:81-83`, `packages/atif-analytics/src/atif_analytics/infrastructure/sqlite_state/checkpointer.py:43-62` | | Analyze summary | stdout JSON | `budget_exhausted`, `llm_spent_usd`, `consecutive_skips` | `packages/atif-analytics/src/atif_analytics/application/analyze.py:121`, `packages/atif-analytics/src/atif_analytics/application/analyze.py:144-146` | | Per-session loss accounting | `loss_report.json` inside each corpus session dir | `gaps_observed`, `records_dropped`, `records_total` | `packages/atif-converter/src/atif_converter/domain/fidelity.py:114-137` | @@ -118,7 +118,7 @@ These properties of this surface change how you read it. **Everything below WARNING is unreachable through the CLI.** `main()` hardcodes the level and there is no `--verbose`, no `--log-level`, and no `LOGURU_LEVEL` -support — `logger.remove()` deletes the default handler that would honor the +support: `logger.remove()` deletes the default handler that would honor the env var (`packages/atif-cli/src/atif_cli/app.py:1133-1134`). The INFO lines that would narrate a materialize pass (`packages/atif-corpus/src/atif_corpus/application/materialize.py:634-644`) or @@ -148,7 +148,7 @@ Cheapest first. Steps 1 through 6 are free and read-only; step 10 spends money. 3. **Run `atif-sql status`.** It replays the exact planning decision `materialize` would make, without converting anything, so its `stale / up-to-date / live` split answers "would a pass do anything right - now?" — and it prints the resolved `source_root` and `corpus_root`, which is + now?", and it prints the resolved `source_root` and `corpus_root`, which is how a wrong-root problem becomes visible. `packages/atif-cli/src/atif_cli/app.py:475-481`, `packages/atif-cli/src/atif_cli/app.py:495-496` @@ -236,8 +236,8 @@ Cheapest first. Steps 1 through 6 are free and read-only; step 10 spends money. refusing rather than publishing; retrying once the session goes quiet succeeds. `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:11-22`, `packages/atif-converter/src/atif_converter/application/convert_and_audit.py:93-102` -- **`SuspiciousEmptyScanError`:** a wrong `source_root` — a typo, an unmounted - disk, a stale env var — makes the scan empty, and the pass would then +- **`SuspiciousEmptyScanError`:** a wrong `source_root` (a typo, an unmounted + disk, a stale env var) makes the scan empty, and the pass would then mark the entire corpus source-removed. Signal: a traceback naming the error, with the scanned root and the materialized session count in the message, and nothing removed. Mitigation: the guard runs before any session is marked and refuses the @@ -256,7 +256,7 @@ Cheapest first. Steps 1 through 6 are free and read-only; step 10 spends money. `packages/atif-corpus/src/atif_corpus/infrastructure/scanner.py:119-140`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:560-571` - **Lane-lock inheritance through fd 9:** the refresh lane's flock is held by an open descriptor, not by a pid, so a descendant that inherits it keeps the - lane locked for as long as it lives — a lane that stops doing work while + lane locked for as long as it lives: a lane that stops doing work while every log line still reads like a clean single-flight. Signal: `cron status` reporting a held lock with no corresponding work in the log. Mitigation: every `atif-sql` invocation in the script closes fd 9 with `9>&-`, while the @@ -271,18 +271,18 @@ Cheapest first. Steps 1 through 6 are free and read-only; step 10 spends money. the default credential chain. `scripts/atif-sql-refresh.sh:91-96`, `scripts/atif-sql-refresh.sh:116-128` - **The `FidelityGap` members:** these are documented, verified upstream - losses, not defects — workflow-nested subagents missed by harbor's own + losses, not defects: workflow-nested subagents missed by harbor's own discovery, non-message records dropped, the parent chain flattened by timestamp sort, subagents inlined rather than embedded, compaction summaries unhandled, the cache split only partially preserved, and the event uuid not preserved. Signal: `records_dropped > 0` or a non-empty `gaps_observed` in a - `loss_report.json`. Mitigation: none is intended — the converter's job is + `loss_report.json`. Mitigation: none is intended; the converter's job is honest loss accounting, and unit tests pin each gap so a harbor bump that changes behavior trips the suite. `packages/atif-converter/src/atif_converter/domain/fidelity.py:44-79` - **Budget overshoot and the streak that never escalates:** the cost ceiling stops dispatch at an 8-unit batch boundary rather than capping spend exactly, - and the consecutive-skip streak is cleared whenever a stage ran at all — + and the consecutive-skip streak is cleared whenever a stage ran at all, including a run that aborted mid-chunk on the ceiling. Signal: repeated `cost ceiling hit` WARNINGs that never become the ERROR line. Mitigation: read `llm_spent_usd` in the summary rather than trusting the streak, and @@ -294,7 +294,7 @@ Cheapest first. Steps 1 through 6 are free and read-only; step 10 spends money. `finish_reason=length` request with identical parameters truncates identically and doubles the bill, so the single retry degrades `reasoning_effort` instead. Signal: a `finish_reason=length at effort=high` WARNING followed by a retry - queue entry. Mitigation: the degrade ladder — but note every spec resolves at + queue entry. Mitigation: the degrade ladder, but note every spec resolves at `high`, so only `high` to `medium` is reachable in practice, and a second truncation raises `ProviderUnavailable` for the retry queue. Usage is accumulated before every gate, so a truncated call still counts against the @@ -302,8 +302,8 @@ Cheapest first. Steps 1 through 6 are free and read-only; step 10 spends money. `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:79-99`, `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:229-249`, `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:277` - **The `EmbeddingProviderMismatch` twin:** the class exists independently in - two packages with different bases — a `DomainError` in atif-embed, a bare - `Exception` in atif-duck — because the independence contract forbids the + two packages with different bases (a `DomainError` in atif-embed, a bare + `Exception` in atif-duck) because the independence contract forbids the import. Signal: an unhandled traceback with exit 1 instead of the classified exit 65. Mitigation: `REGISTRATION_ERRORS` widens the caught tuple to cover both, and a test reads both twin modules as source text to require the shared diff --git a/docs/insights/impact-analysis.md b/docs/insights/impact-analysis.md index de4a920..0f602f3 100644 --- a/docs/insights/impact-analysis.md +++ b/docs/insights/impact-analysis.md @@ -6,7 +6,7 @@ The reader's question: *if I touch X, what else do I have to think about?* default rule is the top 8 modules by inbound reference count. That rule mis-ranks this workspace, and the substitution is deliberate. The import-linter contracts forbid most of the packages from importing each other (`pyproject.toml:514-517`), so the surfaces that cost the most to change are -exactly the ones with the *fewest* inbound imports — a corpus filename string has several independent +exactly the ones with the *fewest* inbound imports: a corpus filename string has several independent readers and no import edges. Ranking by import count would put `atif_analytics`' many intra-package edges on top and leave every cross-package contract off the list. @@ -15,18 +15,18 @@ edge connects to it, and a named gate fails if you miss one.** Surfaces are orde distinct enforcement mechanisms fire (a test, an import-linter contract, a CI step, a `KeyError` at build time) and how many packages the edit spans. -**How the consumer sets below were derived.** There is no code index in this repo — no `.codegraph/`, -no LSP index, no symbol graph — so no count here came from one. Each set was built by grepping import +**How the consumer sets below were derived.** There is no code index in this repo (no `.codegraph/`, +no LSP index, no symbol graph), so no count here came from one. Each set was built by grepping import statements for the symbol and then **reading every hit's import line** to confirm which package the name resolved to. These properties of this codebase make that confirmation step load-bearing rather than pedantic: -- **Every cross-package import is indented** — inside a function body or an `if TYPE_CHECKING:` block, +- **Every cross-package import is indented**: inside a function body or an `if TYPE_CHECKING:` block, because the lean-import contract defers them (`packages/atif-cli/src/atif_cli/app.py:22-26`). A line-anchored `^from atif_` grep returns no cross-package consumers. The census behind this file allowed leading whitespace and then read each enclosing scope. - **Names collide across packages.** `DomainError` is declared independently in several packages and - `EmbeddingProviderMismatch` in more than one — `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:33` + `EmbeddingProviderMismatch` in more than one: `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:33` derives from bare `Exception`, `packages/atif-embed/src/atif_embed/domain/errors.py:29` from atif-embed's own `DomainError`. Near-names compound it: `packages/atif-embed/src/atif_embed/infrastructure/cohere_bedrock.py:44` imports @@ -49,29 +49,29 @@ The catalogs (`VIEW_NAMES`, `MACRO_NAMES`, `ANALYTICS_VIEW_NAMES`, | Downstream | Type | Touch on change | Citation | | --- | --- | --- | --- | -| `atif_duck.domain.examples` — derives every example from every catalog | direct import | yes | `packages/atif-duck/src/atif_duck/domain/examples.py:35-43` | -| `DESCRIPTIONS` — one entry per object; `_description()` raises `KeyError` without it | direct import | yes | `packages/atif-duck/src/atif_duck/domain/catalog.py:431` and `packages/atif-duck/src/atif_duck/domain/examples.py:166-175` | -| `ARG_EXEMPLARS` — needed only when a macro introduces a NEW parameter name; `_macro_sql()` raises `KeyError` without it | direct import | yes | `packages/atif-duck/src/atif_duck/domain/examples.py:75-92` and `packages/atif-duck/src/atif_duck/domain/examples.py:148-158` | -| `TABLE_MACRO_NAMES` — membership required when the DDL says `AS TABLE`, or the derived SQL uses the wrong call shape | direct import | yes | `packages/atif-duck/src/atif_duck/domain/catalog.py:413-422` | -| `atif_duck.infrastructure.registry` — owns the real DDL and imports `DEFAULT_PRICING` | direct import | yes | `packages/atif-duck/src/atif_duck/infrastructure/registry.py:53` | -| `atif_duck.infrastructure.analytics` — `register_analytics` and `register_analytics_macros` own the DDL the analytics catalogs describe | indirect | yes | `packages/atif-duck/src/atif_duck/infrastructure/analytics.py:119` and `packages/atif-duck/src/atif_duck/infrastructure/analytics.py:226` | -| `atif_duck.infrastructure.authorship` — `register_authorship` owns the DDL for `step_author`, `user_steps`, `human_turns`, and `session_outcomes` | indirect | yes | `packages/atif-duck/src/atif_duck/infrastructure/authorship.py:38` | -| `atif_cli.app.schema` — deferred import of `VIEW_SCHEMA`, `MACRO_SIGNATURES`, `ANALYTICS_VIEW_SCHEMA`, and `ANALYTICS_MACRO_SIGNATURES` | direct import | no | `packages/atif-cli/src/atif_cli/app.py:1745-1750` | -| `atif_cli.app.examples` — deferred import of `build_examples` + the two value tuples | direct import | no | `packages/atif-cli/src/atif_cli/app.py:1022-1026` | -| `test_examples.py::test_every_example_executes` — parametrized over every derived example, `fetchall()` forces materialization | test | yes | `packages/atif-duck/tests/test_examples.py:73-79` | -| `test_examples.py` drift catchers — description coverage, exemplar coverage, `AS TABLE` set equality, example-or-exclusion coverage | test | yes | `packages/atif-duck/tests/test_examples.py:160-211` | -| `test_duck_views.py::test_view_schema_matches_describe` — `DESCRIBE` output equals `VIEW_SCHEMA` column-for-column | test | yes | `packages/atif-duck/tests/test_duck_views.py:50-59` | -| `test_duck_views.py::test_macro_signatures_match_ddl` — regex-parses the DDL and asserts equality | test | yes | `packages/atif-duck/tests/test_duck_views.py:66-87` | -| `test_analytics_views.py::test_analytics_macro_signatures_match_ddl` — same for the analytics half | test | yes | `packages/atif-duck/tests/test_analytics_views.py:313-328` | -| `test_duck_views.py::test_default_pricing_matches_published_list_rates_exactly` — oracle table, key-for-key | test | yes | `packages/atif-duck/tests/test_duck_views.py:459-467` | -| `test_app.py::TestSchema` / `TestExamples` — assert the CLI payload against the catalog, not literals | test | no | `packages/atif-cli/tests/test_app.py:134-165` and `packages/atif-cli/tests/test_app.py:94-111` | +| `atif_duck.domain.examples`: derives every example from every catalog | direct import | yes | `packages/atif-duck/src/atif_duck/domain/examples.py:35-43` | +| `DESCRIPTIONS`: one entry per object; `_description()` raises `KeyError` without it | direct import | yes | `packages/atif-duck/src/atif_duck/domain/catalog.py:431` and `packages/atif-duck/src/atif_duck/domain/examples.py:166-175` | +| `ARG_EXEMPLARS`: needed only when a macro introduces a NEW parameter name; `_macro_sql()` raises `KeyError` without it | direct import | yes | `packages/atif-duck/src/atif_duck/domain/examples.py:75-92` and `packages/atif-duck/src/atif_duck/domain/examples.py:148-158` | +| `TABLE_MACRO_NAMES`: membership required when the DDL says `AS TABLE`, or the derived SQL uses the wrong call shape | direct import | yes | `packages/atif-duck/src/atif_duck/domain/catalog.py:413-422` | +| `atif_duck.infrastructure.registry`: owns the real DDL and imports `DEFAULT_PRICING` | direct import | yes | `packages/atif-duck/src/atif_duck/infrastructure/registry.py:53` | +| `atif_duck.infrastructure.analytics`: `register_analytics` and `register_analytics_macros` own the DDL the analytics catalogs describe | indirect | yes | `packages/atif-duck/src/atif_duck/infrastructure/analytics.py:119` and `packages/atif-duck/src/atif_duck/infrastructure/analytics.py:226` | +| `atif_duck.infrastructure.authorship`: `register_authorship` owns the DDL for `step_author`, `user_steps`, `human_turns`, and `session_outcomes` | indirect | yes | `packages/atif-duck/src/atif_duck/infrastructure/authorship.py:38` | +| `atif_cli.app.schema`: deferred import of `VIEW_SCHEMA`, `MACRO_SIGNATURES`, `ANALYTICS_VIEW_SCHEMA`, and `ANALYTICS_MACRO_SIGNATURES` | direct import | no | `packages/atif-cli/src/atif_cli/app.py:1745-1750` | +| `atif_cli.app.examples`: deferred import of `build_examples` + the two value tuples | direct import | no | `packages/atif-cli/src/atif_cli/app.py:1022-1026` | +| `test_examples.py::test_every_example_executes`: parametrized over every derived example, `fetchall()` forces materialization | test | yes | `packages/atif-duck/tests/test_examples.py:73-79` | +| `test_examples.py` drift catchers: description coverage, exemplar coverage, `AS TABLE` set equality, example-or-exclusion coverage | test | yes | `packages/atif-duck/tests/test_examples.py:160-211` | +| `test_duck_views.py::test_view_schema_matches_describe`: `DESCRIBE` output equals `VIEW_SCHEMA` column-for-column | test | yes | `packages/atif-duck/tests/test_duck_views.py:50-59` | +| `test_duck_views.py::test_macro_signatures_match_ddl`: regex-parses the DDL and asserts equality | test | yes | `packages/atif-duck/tests/test_duck_views.py:66-87` | +| `test_analytics_views.py::test_analytics_macro_signatures_match_ddl`: same for the analytics half | test | yes | `packages/atif-duck/tests/test_analytics_views.py:313-328` | +| `test_duck_views.py::test_default_pricing_matches_published_list_rates_exactly`: oracle table, key-for-key | test | yes | `packages/atif-duck/tests/test_duck_views.py:459-467` | +| `test_app.py::TestSchema` / `TestExamples`: assert the CLI payload against the catalog, not literals | test | no | `packages/atif-cli/tests/test_app.py:134-165` and `packages/atif-cli/tests/test_app.py:94-111` | ### Blast-radius notes - **`EXCLUSIONS` is the only sanctioned way to add a catalog object without an example, and it is policed in both directions.** The dict is currently empty - (`packages/atif-duck/src/atif_duck/domain/examples.py:99`), and a stale key — or a key that still - emits an example — fails `test_exclusions_reference_real_catalog_objects` + (`packages/atif-duck/src/atif_duck/domain/examples.py:99`), and a stale key (or a key that still + emits an example) fails `test_exclusions_reference_real_catalog_objects` (`packages/atif-duck/tests/test_examples.py:170-175`). - **`atif-sql schema` prints every catalog, each object tagged with its `requires` value.** It reads the core and analytics schema and signature dicts (`packages/atif-cli/src/atif_cli/app.py:1745-1750`) @@ -95,15 +95,15 @@ private converters, and those private methods are reachable from the tests only, | Downstream | Type | Touch on change | Citation | | --- | --- | --- | --- | -| `test_dev_only_imports_guard.py` — `ast` guard: no `harbor` or `litellm` import in any `src/` | test | yes | `packages/atif-converter/tests/test_dev_only_imports_guard.py` | -| `test_vendored_atif.py` — vendored files equal upstream's; both models and validators agree on the goldens | test | yes | `packages/atif-converter/tests/test_vendored_atif.py` | -| `harbor_adapter.convert_session()` / `codex_adapter.convert_codex_session()` — the seams: converter in, validated dict out | direct import | yes | `packages/atif-converter/src/atif_converter/infrastructure/harbor_adapter.py:83`, `packages/atif-converter/src/atif_converter/infrastructure/codex_adapter.py:63` | -| `FidelityGap` / `CodexFidelityGap` — the losses the port inherited on purpose; fixing one is a decision to diverge from the oracle | indirect | yes | `packages/atif-converter/src/atif_converter/domain/fidelity.py:52`, `packages/atif-converter/src/atif_converter/domain/codex_fidelity.py:66` | -| `test_harbor_oracle.py` — live oracle equals frozen goldens; a harbor bump that changes conversion fails here first | test | yes | `packages/atif-converter/tests/test_harbor_oracle.py:1` | -| `test_parity_live.py` — newest N local sessions of each agent, both converters, every diverging path | test | yes | `packages/atif-converter/tests/test_parity_live.py:1` | -| `atif_converter.domain.enrichment` / `codex_enrichment` — re-derive step identity by replicating the converter's normalization order | indirect | yes | `packages/atif-converter/src/atif_converter/domain/enrichment.py:1`, `packages/atif-converter/src/atif_converter/domain/codex_enrichment.py:1` | -| `EXIT_CODES["harbor_missing"] = 127` — retired, kept so the table never renumbers | config | no | `packages/atif-cli/src/atif_cli/errors.py:47` | -| `atif_cli.app.materialize` — stamps `_version_of("harbor")` into every `meta.json` | direct import | likely | `packages/atif-cli/src/atif_cli/app.py:426` | +| `test_dev_only_imports_guard.py`: `ast` guard: no `harbor` or `litellm` import in any `src/` | test | yes | `packages/atif-converter/tests/test_dev_only_imports_guard.py` | +| `test_vendored_atif.py`: vendored files equal upstream's; both models and validators agree on the goldens | test | yes | `packages/atif-converter/tests/test_vendored_atif.py` | +| `harbor_adapter.convert_session()` / `codex_adapter.convert_codex_session()`: the seams (converter in, validated dict out) | direct import | yes | `packages/atif-converter/src/atif_converter/infrastructure/harbor_adapter.py:83`, `packages/atif-converter/src/atif_converter/infrastructure/codex_adapter.py:63` | +| `FidelityGap` / `CodexFidelityGap`: the losses the port inherited on purpose; fixing one is a decision to diverge from the oracle | indirect | yes | `packages/atif-converter/src/atif_converter/domain/fidelity.py:52`, `packages/atif-converter/src/atif_converter/domain/codex_fidelity.py:66` | +| `test_harbor_oracle.py`: live oracle equals frozen goldens; a harbor bump that changes conversion fails here first | test | yes | `packages/atif-converter/tests/test_harbor_oracle.py:1` | +| `test_parity_live.py`: newest N local sessions of each agent, both converters, every diverging path | test | yes | `packages/atif-converter/tests/test_parity_live.py:1` | +| `atif_converter.domain.enrichment` / `codex_enrichment`: re-derive step identity by replicating the converter's normalization order | indirect | yes | `packages/atif-converter/src/atif_converter/domain/enrichment.py:1`, `packages/atif-converter/src/atif_converter/domain/codex_enrichment.py:1` | +| `EXIT_CODES["harbor_missing"] = 127`: retired, kept so the table never renumbers | config | no | `packages/atif-cli/src/atif_cli/errors.py:47` | +| `atif_cli.app.materialize`: stamps `_version_of("harbor")` into every `meta.json` | direct import | likely | `packages/atif-cli/src/atif_cli/app.py:426` | ## The materialized corpus artifact layout @@ -116,14 +116,14 @@ places with no import edge between them. | Downstream | Type | Touch on change | Citation | | --- | --- | --- | --- | -| `atif_corpus.application.materialize` — the writer, routed through `CorpusLayout` | direct import | yes | `packages/atif-corpus/src/atif_corpus/application/materialize.py:75-76` and `packages/atif-corpus/src/atif_corpus/application/materialize.py:530` | -| `atif_duck.infrastructure.registry` — builds globs under `corpus_root / "sessions"` | indirect | yes | `packages/atif-duck/src/atif_duck/infrastructure/registry.py:209-213` | -| `atif_embed.infrastructure.corpus_text_rows` — its own `sessions/` walk, gated on `meta.json` | indirect | yes | `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:112-123` | -| `atif_analytics.infrastructure.corpus_reader` — re-declares the filename constants | indirect | yes | `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:53-55` and `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:163-164` | -| `atif_cli.app.convert` — writes `edges.jsonl` beside a `--out` trajectory on the one-shot path | indirect | likely | `packages/atif-cli/src/atif_cli/app.py:276` | -| `atif_cli.app.status` — reads the watermark through the producer's public `read_watermark` | direct import | likely | `packages/atif-cli/src/atif_cli/app.py:460-470` | -| `docs/CONTRACT.md:21-31` — the specification; the file declares itself orchestrator-owned and changeable only via itself | config | yes | `docs/CONTRACT.md:3` | -| `test_domain.py::TestCorpusLayout.test_contract_paths` — pins every contract string | test | yes | `packages/atif-corpus/tests/test_domain.py:245-249` | +| `atif_corpus.application.materialize`: the writer, routed through `CorpusLayout` | direct import | yes | `packages/atif-corpus/src/atif_corpus/application/materialize.py:75-76` and `packages/atif-corpus/src/atif_corpus/application/materialize.py:530` | +| `atif_duck.infrastructure.registry`: builds globs under `corpus_root / "sessions"` | indirect | yes | `packages/atif-duck/src/atif_duck/infrastructure/registry.py:209-213` | +| `atif_embed.infrastructure.corpus_text_rows`: its own `sessions/` walk, gated on `meta.json` | indirect | yes | `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:112-123` | +| `atif_analytics.infrastructure.corpus_reader`: re-declares the filename constants | indirect | yes | `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:53-55` and `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:163-164` | +| `atif_cli.app.convert`: writes `edges.jsonl` beside a `--out` trajectory on the one-shot path | indirect | likely | `packages/atif-cli/src/atif_cli/app.py:276` | +| `atif_cli.app.status`: reads the watermark through the producer's public `read_watermark` | direct import | likely | `packages/atif-cli/src/atif_cli/app.py:460-470` | +| `docs/CONTRACT.md:21-31`: the specification; the file declares itself orchestrator-owned and changeable only via itself | config | yes | `docs/CONTRACT.md:3` | +| `test_domain.py::TestCorpusLayout.test_contract_paths`: pins every contract string | test | yes | `packages/atif-corpus/tests/test_domain.py:245-249` | | Independent fixture-corpus builders, one per consuming package | test | yes | `packages/atif-duck/tests/duck_fixtures.py:482`, `packages/atif-embed/tests/embed_fixtures.py:120`, `packages/atif-analytics/tests/analytics_fixtures.py:218`, `packages/atif-corpus/tests/corpus_fixtures.py:77` | ### Blast-radius notes @@ -131,7 +131,7 @@ places with no import edge between them. - **Nothing mechanical catches a layout change that updates one reader and misses the others.** No import edge connects them, so the only signal is each package's own fixture corpus going out of agreement with the writer. The `meta.json`-written-last torn-set rule is duplicated verbatim in every - reader — evidence of the duplication cost, and one place per reader to edit if the completion marker + reader: evidence of the duplication cost, and one place per reader to edit if the completion marker ever moves (`packages/atif-duck/src/atif_duck/infrastructure/registry.py:188`, `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:28`, `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:14`). @@ -153,10 +153,10 @@ Defined at: `packages/atif-corpus/src/atif_corpus/domain/ports.py:41` (`Converte `VectorStorePort`, `TextRowsPort`), `packages/atif-models/src/atif_models/domain/ports.py:116` (`LlmStructuredProvider`) -The Protocols all sit under `domain/`, all satisfied structurally — there is no `abstractmethod` and no +The Protocols all sit under `domain/`, all satisfied structurally: there is no `abstractmethod` and no DI container in this workspace. Some of them exist because an import-linter contract forbids the direct dependency: `ConverterPort` (atif-corpus may not import atif-converter) and `TextRowsPort` (atif-embed -may not import atif-duck). The others are ordinary seams — atif-analytics is permitted to import +may not import atif-duck). The others are ordinary seams: atif-analytics is permitted to import atif-models, and the atif-embed adapters live in the same package as their ports. **Every adapter satisfies its Protocol without importing it.** Each row below marked `indirect` names a @@ -165,22 +165,22 @@ class whose only textual reference to the Protocol is a docstring, so no import- | Downstream | Type | Touch on change | Citation | | --- | --- | --- | --- | | `LlmStructuredProvider` ← the atif-analytics use-case modules, every one importing it under `if TYPE_CHECKING:` | direct import | yes | `packages/atif-analytics/src/atif_analytics/application/use_cases/_shared.py:32`, `packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:65`, `packages/atif-analytics/src/atif_analytics/application/use_cases/conflicts.py:79`, `packages/atif-analytics/src/atif_analytics/application/use_cases/friction.py:81`, `packages/atif-analytics/src/atif_analytics/application/use_cases/perceived.py:90` | -| `OpenAiBedrockProvider` — the one real adapter; imports `CallUsage` / `ProviderUnavailable` / `RefusalError` / `SchemaT` / `UsageAccumulator` from the port module and NOT the Protocol | indirect | yes | `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:49-55` and `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:143` | -| `FakeProvider` — the deterministic double, named in prose only | test | yes | `packages/atif-analytics/tests/analytics_fixtures.py:237` | +| `OpenAiBedrockProvider`: the one real adapter; imports `CallUsage` / `ProviderUnavailable` / `RefusalError` / `SchemaT` / `UsageAccumulator` from the port module and NOT the Protocol | indirect | yes | `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:49-55` and `packages/atif-models/src/atif_models/infrastructure/openai_bedrock.py:143` | +| `FakeProvider`: the deterministic double, named in prose only | test | yes | `packages/atif-analytics/tests/analytics_fixtures.py:237` | | `ConverterPort` ← `materialize` use case, in several signature positions | direct import | yes | `packages/atif-corpus/src/atif_corpus/application/materialize.py:92`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:193`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:480` | -| `RealConverter` — the production adapter; imports `ConversionOutput` only, names `ConverterPort` in its docstring | indirect | yes | `packages/atif-cli/src/atif_cli/converter_adapter.py:38` and `packages/atif-cli/src/atif_cli/converter_adapter.py:44-45` | -| `FakeConverter` — shipped in `src/` rather than `tests/`; same shape, same non-import | indirect | yes | `packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py:16` and `packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py:37` | -| `ConversionOutput` — the frozen dataclass that DOES cross the boundary as an import | direct import | yes | `packages/atif-corpus/src/atif_corpus/domain/ports.py:21-38` | +| `RealConverter`: the production adapter; imports `ConversionOutput` only, names `ConverterPort` in its docstring | indirect | yes | `packages/atif-cli/src/atif_cli/converter_adapter.py:38` and `packages/atif-cli/src/atif_cli/converter_adapter.py:44-45` | +| `FakeConverter`: shipped in `src/` rather than `tests/`; same shape, same non-import | indirect | yes | `packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py:16` and `packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py:37` | +| `ConversionOutput`: the frozen dataclass that DOES cross the boundary as an import | direct import | yes | `packages/atif-corpus/src/atif_corpus/domain/ports.py:21-38` | | `EmbeddingProvider` / `VectorStorePort` / `TextRowsPort` ← the one embed use case, under `if TYPE_CHECKING:` | direct import | yes | `packages/atif-embed/src/atif_embed/application/embed.py:35`, `packages/atif-embed/src/atif_embed/application/embed.py:46-47`, `packages/atif-embed/src/atif_embed/application/embed.py:65-67` | | The production adapters, one per embed port, none importing its Protocol | indirect | yes | `packages/atif-embed/src/atif_embed/infrastructure/cohere_bedrock.py:285`, `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:377`, `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:156` | -| `test_converter_adapter.py` — the one place a Protocol name is imported and bound to an implementation | test | yes | `packages/atif-cli/tests/test_converter_adapter.py:17` and `packages/atif-cli/tests/test_converter_adapter.py:91` | -| `[tool.coverage.report] exclude_also` — a bare `...` line is excluded because a Protocol body is a signature | config | no | `pyproject.toml:561-564` | +| `test_converter_adapter.py`: the one place a Protocol name is imported and bound to an implementation | test | yes | `packages/atif-cli/tests/test_converter_adapter.py:17` and `packages/atif-cli/tests/test_converter_adapter.py:91` | +| `[tool.coverage.report] exclude_also`: a bare `...` line is excluded because a Protocol body is a signature | config | no | `pyproject.toml:561-564` | ### Blast-radius notes - **Adding a method to a Protocol breaks implementations that name it nowhere.** Because no adapter imports its Protocol, the only mechanical signal is a `pyright`/`ty` error where the implementation is - passed into a port-typed parameter — and for `RealConverter` that call site is in a different package + passed into a port-typed parameter; for `RealConverter` that call site is in a different package from both the Protocol and the class (`packages/atif-cli/src/atif_cli/converter_adapter.py:44`, `packages/atif-corpus/src/atif_corpus/application/materialize.py:193`). `FakeConverter` lives in `src/`, so a missed update fails the type gate rather than only a test @@ -205,21 +205,21 @@ surfaces above exist as duplicated strings and Protocols instead of imports. | Downstream | Type | Touch on change | Citation | | --- | --- | --- | --- | -| `mise run lint:imports` — `uv run lint-imports`, an entry in `[tasks.check].depends` | config | yes | `mise.toml:162` and `mise.toml:204` | -| `root_packages` — every import package must be listed, or a new member goes unchecked | config | yes | `pyproject.toml:463` | -| `atif_cli` — the composition root, absent from `independence.modules` and listed in `forbidden_modules`; it declares its siblings and NOT atif-models, so the comment granting it that permission describes an unused allowance | indirect | likely | `pyproject.toml:517`, `pyproject.toml:523`, `packages/atif-cli/pyproject.toml:32-36` | -| `ConverterPort` — exists because atif-corpus may not import atif-converter | indirect | yes | `packages/atif-corpus/src/atif_corpus/domain/ports.py:5-9` | -| `TextRowsPort` — exists because atif-embed may not import atif-duck | indirect | yes | `packages/atif-embed/src/atif_embed/domain/ports.py:11-15` | -| The `embedding_guard` twins — one rule, copied into atif-duck and atif-embed | direct import | yes | `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:33` and `packages/atif-embed/src/atif_embed/domain/embedding_guard.py:38` | -| `test_guard_twin_pin.py` — reads the atif-duck copy as SOURCE TEXT via `ast`, never importing it | test | yes | `packages/atif-embed/tests/test_guard_twin_pin.py:41` and `packages/atif-embed/tests/test_guard_twin_pin.py:55-72` | +| `mise run lint:imports`: `uv run lint-imports`, an entry in `[tasks.check].depends` | config | yes | `mise.toml:162` and `mise.toml:204` | +| `root_packages`: every import package must be listed, or a new member goes unchecked | config | yes | `pyproject.toml:463` | +| `atif_cli`: the composition root, absent from `independence.modules` and listed in `forbidden_modules`; it declares its siblings and NOT atif-models, so the comment granting it that permission describes an unused allowance | indirect | likely | `pyproject.toml:517`, `pyproject.toml:523`, `packages/atif-cli/pyproject.toml:32-36` | +| `ConverterPort`: exists because atif-corpus may not import atif-converter | indirect | yes | `packages/atif-corpus/src/atif_corpus/domain/ports.py:5-9` | +| `TextRowsPort`: exists because atif-embed may not import atif-duck | indirect | yes | `packages/atif-embed/src/atif_embed/domain/ports.py:11-15` | +| The `embedding_guard` twins: one rule, copied into atif-duck and atif-embed | direct import | yes | `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:33` and `packages/atif-embed/src/atif_embed/domain/embedding_guard.py:38` | +| `test_guard_twin_pin.py`: reads the atif-duck copy as SOURCE TEXT via `ast`, never importing it | test | yes | `packages/atif-embed/tests/test_guard_twin_pin.py:41` and `packages/atif-embed/tests/test_guard_twin_pin.py:55-72` | | The independent corpus readers | indirect | yes | `packages/atif-duck/src/atif_duck/infrastructure/registry.py:209`, `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:113`, `packages/atif-analytics/src/atif_analytics/infrastructure/corpus_reader.py:53` | -| `[tool.coverage.run] source` — the same import packages, named rather than pathed | config | likely | `pyproject.toml:537-545` | +| `[tool.coverage.run] source`: the same import packages, named rather than pathed | config | likely | `pyproject.toml:537-545` | ### Blast-radius notes - **A new cross-package import fails the `lint:imports` gate and nothing else.** `lint:imports` runs `uv run lint-imports` over `packages/*/src/**/*.py` plus `pyproject.toml` - (`mise.toml:161-162`) — the type checkers and the test suite stay green, so the contract violation + (`mise.toml:161-162`). The type checkers and the test suite stay green, so the contract violation surfaces only when `mise run check` reaches that gate. - **atif-duck declares no `layers` contract.** It has `domain/` and `infrastructure/` and no `application/`, so its layer direction is convention rather than enforcement; only the @@ -229,27 +229,27 @@ surfaces above exist as duplicated strings and Protocols instead of imports. compared through an AST read of the other package's source because importing it would break the contract (`packages/atif-embed/tests/test_guard_twin_pin.py:3-14`). -## `EXIT_CODES` — the CLI wire contract +## `EXIT_CODES`: the CLI wire contract Defined at: `packages/atif-cli/src/atif_cli/errors.py:25-39` Its keys map to the distinct codes 0, 2, 64, 65, 70, 78, and 127. This is the contract an agent -driving the CLI reads, so the dict is the surface — not the individual numbers, and not the messages. +driving the CLI reads, so the dict is the surface, not the individual numbers, and not the messages. | Downstream | Type | Touch on change | Citation | | --- | --- | --- | --- | -| `atif_cli.app` — the module-scope import; the dict is on the lean path | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:38` | -| `atif_cli.app.convert` — exits `invalid_input`, `empty_session`, `runtime_error`, `validation_error` | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:264-304` | -| `atif_cli.app.query` — `parse_error`, raised after the classified envelope is emitted | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:603-610` | -| `atif_cli.app.embed` — `invalid_input` plus a `EXIT_CODES[kind]` dispatch over the classifier's verdict | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:814-847` | -| `atif_cli.app.search` — `no_embeddings`, the exit that tells an operator to run `embed` | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:935-941` | -| `atif_cli.app.examples` — `invalid_input` on an unknown `--category` / `--requires` value | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:1036-1042` | -| `atif_cli.duck_errors` — classifies `duckdb.Error` into parse / catalog / runtime plus `embedding_mismatch` | direct import | yes | `packages/atif-cli/src/atif_cli/duck_errors.py:25` and `packages/atif-cli/src/atif_cli/duck_errors.py:40-86` | -| `atif_cli.cron` — the unattended lane, which suppresses retries on code 78 | direct import | likely | `packages/atif-cli/src/atif_cli/cron.py:32` and `packages/atif-cli/src/atif_cli/cron.py:175` | -| `ClassifiedError.to_payload()` — the JSON error envelope on non-TTY stderr | indirect | likely | `packages/atif-cli/src/atif_cli/errors.py:60-68` | -| `test_app.py` — asserts against the dict, never against a literal, except where the number itself is the claim | test | yes | `packages/atif-cli/tests/test_app.py:17` and `packages/atif-cli/tests/test_app.py:744` | -| `test_cron.py`, `test_integration.py`, `test_vss_commands.py` — more suites keyed off the dict | test | yes | `packages/atif-cli/tests/test_cron.py:27`, `packages/atif-cli/tests/test_integration.py:99`, `packages/atif-cli/tests/test_vss_commands.py:20` | -| `docs/CONTRACT.md:62-66` §CLI — the documented command surface these codes are raised from | config | likely | `docs/CONTRACT.md:62-66` | +| `atif_cli.app`: the module-scope import; the dict is on the lean path | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:38` | +| `atif_cli.app.convert`: exits `invalid_input`, `empty_session`, `runtime_error`, `validation_error` | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:264-304` | +| `atif_cli.app.query`: `parse_error`, raised after the classified envelope is emitted | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:603-610` | +| `atif_cli.app.embed`: `invalid_input` plus a `EXIT_CODES[kind]` dispatch over the classifier's verdict | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:814-847` | +| `atif_cli.app.search`: `no_embeddings`, the exit that tells an operator to run `embed` | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:935-941` | +| `atif_cli.app.examples`: `invalid_input` on an unknown `--category` / `--requires` value | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:1036-1042` | +| `atif_cli.duck_errors`: classifies `duckdb.Error` into parse / catalog / runtime plus `embedding_mismatch` | direct import | yes | `packages/atif-cli/src/atif_cli/duck_errors.py:25` and `packages/atif-cli/src/atif_cli/duck_errors.py:40-86` | +| `atif_cli.cron`: the unattended lane, which suppresses retries on code 78 | direct import | likely | `packages/atif-cli/src/atif_cli/cron.py:32` and `packages/atif-cli/src/atif_cli/cron.py:175` | +| `ClassifiedError.to_payload()`: the JSON error envelope on non-TTY stderr | indirect | likely | `packages/atif-cli/src/atif_cli/errors.py:60-68` | +| `test_app.py`: asserts against the dict, never against a literal, except where the number itself is the claim | test | yes | `packages/atif-cli/tests/test_app.py:17` and `packages/atif-cli/tests/test_app.py:744` | +| `test_cron.py`, `test_integration.py`, `test_vss_commands.py`: more suites keyed off the dict | test | yes | `packages/atif-cli/tests/test_cron.py:27`, `packages/atif-cli/tests/test_integration.py:99`, `packages/atif-cli/tests/test_vss_commands.py:20` | +| `docs/CONTRACT.md:62-66` §CLI: the documented command surface these codes are raised from | config | likely | `docs/CONTRACT.md:62-66` | ### Blast-radius notes @@ -279,17 +279,17 @@ purpose: the distribution is `atif-sql`, the console script's module is `atif_cl | Downstream | Type | Touch on change | Citation | | --- | --- | --- | --- | -| `[project.dependencies]` — the union of every member's third-party requirements, and the ONLY thing an installer sees | config | yes | `pyproject.toml:132-152` | -| `[tool.hatch.build.targets.wheel] packages` — the module trees the wheel carries; an omission ships a wheel missing a module | config | yes | `pyproject.toml:173-182` | -| `test_distribution.py` — asserts the union, the absence of any `atif-*` requirement, and the module list with its `py.typed` markers | test | yes | `packages/atif-cli/tests/test_distribution.py:3-22` | +| `[project.dependencies]`: the union of every member's third-party requirements, and the ONLY thing an installer sees | config | yes | `pyproject.toml:132-152` | +| `[tool.hatch.build.targets.wheel] packages`: the module trees the wheel carries; an omission ships a wheel missing a module | config | yes | `pyproject.toml:173-182` | +| `test_distribution.py`: asserts the union, the absence of any `atif-*` requirement, and the module list with its `py.typed` markers | test | yes | `packages/atif-cli/tests/test_distribution.py:3-22` | | Dev pins in the CLI and analytics manifests, resolved locally through `[tool.uv.sources]` and never published | config | likely | `packages/atif-cli/pyproject.toml:32-36`, `packages/atif-analytics/pyproject.toml:24` | -| `[tool.commitizen] version_files` — glob:regex entries rewriting the published version, every member manifest, and the dev pin blocks | config | yes | `pyproject.toml:446-451` | -| `[tool.commitizen] version` — the single source of truth, because `version_provider` defaults to reading it | config | yes | `pyproject.toml:403-406` | -| `cz bump --check-consistency` in `release.yml` — fails when the current version is absent from any listed file | config | yes | `.github/workflows/release.yml:193` | -| `pre_bump_hooks` — `uv lock` then `git add uv.lock`, so the re-resolved lockfile lands in the bump commit | config | yes | `pyproject.toml:428-431` | -| `uv lock --check` — enforced as a lefthook pre-commit step and as a CI step | config | likely | `mise.toml:84`, `lefthook.yml:95`, `.github/workflows/check.yml:51` | +| `[tool.commitizen] version_files`: glob:regex entries rewriting the published version, every member manifest, and the dev pin blocks | config | yes | `pyproject.toml:446-451` | +| `[tool.commitizen] version`: the single source of truth, because `version_provider` defaults to reading it | config | yes | `pyproject.toml:403-406` | +| `cz bump --check-consistency` in `release.yml`: fails when the current version is absent from any listed file | config | yes | `.github/workflows/release.yml:193` | +| `pre_bump_hooks`: `uv lock` then `git add uv.lock`, so the re-resolved lockfile lands in the bump commit | config | yes | `pyproject.toml:428-431` | +| `uv lock --check`: enforced as a lefthook pre-commit step and as a CI step | config | likely | `mise.toml:84`, `lefthook.yml:95`, `.github/workflows/check.yml:51` | | The console script `atif-sql = "atif_cli.app:main"` | config | likely | `packages/atif-cli/pyproject.toml:42` | -| `tag_format = "v$version"` — `publish.yml` strips the leading `v` and `gh release create --verify-tag` fails if the formats diverge | config | likely | `pyproject.toml:411` | +| `tag_format = "v$version"`: `publish.yml` strips the leading `v` and `gh release create --verify-tag` fails if the formats diverge | config | likely | `pyproject.toml:411` | ### Blast-radius notes @@ -310,25 +310,25 @@ purpose: the distribution is `atif-sql`, the console script's module is `atif_cl Defined at: `packages/atif-cli/src/atif_cli/app.py:22-26` A bare `import atif_cli.app` must not pull duckdb, harbor, lancedb, boto3, or polars. The fast -path — `schema`, `examples`, `--help`, `--version` — needs none of them, and each costs hundreds of +path (`schema`, `examples`, `--help`, `--version`) needs none of them, and each costs hundreds of milliseconds (lancedb alone ~2.6 s). Every heavy import is deferred into the command body that uses it. | Downstream | Type | Touch on change | Citation | | --- | --- | --- | --- | -| `test_lean_import.py` — a forbidden list checked in a FRESH interpreter via `subprocess` | test | yes | `packages/atif-cli/tests/test_lean_import.py:17-32` and `packages/atif-cli/tests/test_lean_import.py:43-49` | +| `test_lean_import.py`: a forbidden list checked in a FRESH interpreter via `subprocess` | test | yes | `packages/atif-cli/tests/test_lean_import.py:17-32` and `packages/atif-cli/tests/test_lean_import.py:43-49` | | ruff `PLC0415` (import-outside-top-level) ignored workspace-wide, 182 measured sites | config | yes | `pyproject.toml:164` | -| `atif_cli.errors` — kept `atif_*`-free and duckdb-free so it stays on the lean path | direct import | yes | `packages/atif-cli/src/atif_cli/errors.py:13-15` | -| `atif_cli.duck_errors` — the concrete `duckdb.Error` classifier, split out for exactly that reason | direct import | yes | `packages/atif-cli/src/atif_cli/duck_errors.py:26` and `packages/atif-cli/src/atif_cli/duck_errors.py:31` | -| `atif_duck.domain.examples` — a pure domain module with no duckdb import, safe on the lean path | direct import | yes | `packages/atif-duck/src/atif_duck/domain/examples.py:26-27` | +| `atif_cli.errors`: kept `atif_*`-free and duckdb-free so it stays on the lean path | direct import | yes | `packages/atif-cli/src/atif_cli/errors.py:13-15` | +| `atif_cli.duck_errors`: the concrete `duckdb.Error` classifier, split out for exactly that reason | direct import | yes | `packages/atif-cli/src/atif_cli/duck_errors.py:26` and `packages/atif-cli/src/atif_cli/duck_errors.py:31` | +| `atif_duck.domain.examples`: a pure domain module with no duckdb import, safe on the lean path | direct import | yes | `packages/atif-duck/src/atif_duck/domain/examples.py:26-27` | | Deferred imports inside command bodies (`convert`, `materialize`, `status`, `query`, `analyze`, `embed`, `search`, `examples`, `schema`) | runtime dispatch | likely | `packages/atif-cli/src/atif_cli/app.py:615`, `packages/atif-cli/src/atif_cli/app.py:906`, `packages/atif-cli/src/atif_cli/app.py:1099` | -| `atif_cli.cron` — imported at module scope, so it must itself stay lean | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:37` | -| `if TYPE_CHECKING:` blocks — the only way to name a heavy type in an annotation on this path | direct import | likely | `packages/atif-cli/src/atif_cli/app.py:33` | +| `atif_cli.cron`: imported at module scope, so it must itself stay lean | direct import | yes | `packages/atif-cli/src/atif_cli/app.py:37` | +| `if TYPE_CHECKING:` blocks: the only way to name a heavy type in an annotation on this path | direct import | likely | `packages/atif-cli/src/atif_cli/app.py:33` | ### Blast-radius notes -- **This is the one surface where moving an import to the top of the file — the normal, lint-preferred - shape — fails a test.** The deferred import IS the contract, which is why `PLC0415` is one of the +- **This is the one surface where moving an import to the top of the file (the normal, lint-preferred + shape) fails a test.** The deferred import IS the contract, which is why `PLC0415` is one of the workspace-wide ruff ignores and carries its measured site count inline (`pyproject.toml:164`). - **The forbidden list names `atif_duck.infrastructure` but not `atif_duck.domain`.** That split is what lets `schema` and `examples` answer from the static catalog at module-import cost while `query` @@ -339,12 +339,12 @@ it. ## Other notable surfaces -- **`DEFAULT_PRICING`** (`packages/atif-duck/src/atif_duck/domain/catalog.py:405`) — models with +- **`DEFAULT_PRICING`** (`packages/atif-duck/src/atif_duck/domain/catalog.py:405`): models with published list rates. A model absent here lands in `cost_estimate`'s `unpriced_steps` rather than being dropped, and tests pin the table: an exact oracle (`packages/atif-duck/tests/test_duck_views.py:459`) and a corpus-coverage gate (`packages/atif-duck/tests/test_duck_views.py:533`). -- **The model alias registry** (`packages/atif-models/src/atif_models/domain/registry.py:60`) — the +- **The model alias registry** (`packages/atif-models/src/atif_models/domain/registry.py:60`): the `(family, size)` entries resolved through `resolve()` at `packages/atif-models/src/atif_models/domain/registry.py:109`. `ModelSpec` reaches every atif-analytics pipeline through the shared provider builder @@ -352,15 +352,15 @@ it. `spec.model_id` is written into every pipeline's output row, so a re-alias changes parquet content (`packages/atif-analytics/src/atif_analytics/application/use_cases/classify.py:335`). - **`RECOVERY_HINT`** (`packages/atif-embed/src/atif_embed/domain/embedding_guard.py:23` and its - atif-duck twin) — one operator instruction, pinned by assertions including one that forbids naming a + atif-duck twin): one operator instruction, pinned by assertions including one that forbids naming a home directory and one that forbids inlining the path at the raise site (`packages/atif-embed/tests/test_guard_twin_pin.py:97-106`). -- **The `# noqa: S608` sites** — `S608` is enforced, and each suppression carries its +- **The `# noqa: S608` sites**: `S608` is enforced, and each suppression carries its static-catalog reason inline; some of them are in the examples generator, where the interpolated name comes from the catalog rather than from user input (`packages/atif-duck/src/atif_duck/domain/examples.py:144`, `packages/atif-duck/src/atif_duck/domain/examples.py:161`, `pyproject.toml:211-212`). -- **Watermark and quiescence** (`packages/atif-corpus/src/atif_corpus/domain/watermark.py:28`) — the +- **Watermark and quiescence** (`packages/atif-corpus/src/atif_corpus/domain/watermark.py:28`): the freshness rule that decides what `materialize` re-converts, surfaced to `status` through the producer's public `read_watermark` (`packages/atif-cli/src/atif_cli/app.py:460`) rather than a private reader. diff --git a/docs/insights/tech-debt.md b/docs/insights/tech-debt.md index 6987feb..9398c35 100644 --- a/docs/insights/tech-debt.md +++ b/docs/insights/tech-debt.md @@ -20,7 +20,7 @@ antecedent. 2. **Version ceilings and the reason attached to each**, read at the pin in the member manifests, not inferred from the lockfile. 3. **Install weight and platform coverage**, computed independently from `uv.lock` by - walking the `atif-sql` dependency graph twice — once whole, once with `harbor` blocked — + walking the `atif-sql` dependency graph twice (once whole, once with `harbor` blocked), following extras as well as direct dependencies. 4. **Asymmetric enforcement between sibling surfaces**: a coverage floor enforced in CI but not in the declared definition of done, so a local green run does not measure it. @@ -40,20 +40,20 @@ rises. | Rank | Debt item | Category | Cost to fix | Citation | | --- | --- | --- | --- | --- | | 1 | RETIRED 2026-09-11. The conversion path used to run through one PRIVATE upstream method per agent; both are now ported into this package on harbor's public ATIF models (`packages/atif-converter/src/atif_converter/domain/claude_code_conversion.py:75`, `packages/atif-converter/src/atif_converter/domain/codex_conversion.py:781`), held to parity by an oracle in the tests (`packages/atif-converter/tests/harbor_oracle.py:94`), and the pin widened to `harbor>=0.22.0,<1`. The install weight in row 3 is retired too. | Coupling | Done | `packages/atif-converter/pyproject.toml:26` | -| 2 | Seven upstream conversion losses are catalogued as `FidelityGap` enum members rather than fixed. Gap 3 flattens the `parentUuid` tree by timestamp sort, losing branch and rewind structure; gap 7 drops the event `uuid`, making step-to-raw-record identity unrecoverable from the trajectory — recovered out of band through a parallel `edges.jsonl` sidecar the corpus layout mandates. | `wrong abstraction` | L | `packages/atif-converter/src/atif_converter/domain/fidelity.py:44-79`, `docs/CONTRACT.md:26-28` | +| 2 | Seven upstream conversion losses are catalogued as `FidelityGap` enum members rather than fixed. Gap 3 flattens the `parentUuid` tree by timestamp sort, losing branch and rewind structure; gap 7 drops the event `uuid`, making step-to-raw-record identity unrecoverable from the trajectory (recovered out of band through a parallel `edges.jsonl` sidecar the corpus layout mandates). | `wrong abstraction` | L | `packages/atif-converter/src/atif_converter/domain/fidelity.py:44-79`, `docs/CONTRACT.md:26-28` | | 3 | RETIRED 2026-09-27. `harbor` used to pull a web server, the whole `supabase` client stack, and `litellm` with `openai`, `tiktoken`, `tokenizers` and `huggingface-hub` into every install, for the ATIF data classes, the validator, and one price lookup per step. The data classes and the validator are vendored in `atif_converter.domain.atif`, prices in `atif_converter/domain/model_prices.json`, and harbor and litellm are dev dependencies only, guarded by `packages/atif-converter/tests/test_dev_only_imports_guard.py`. | `version pin` | Done | `RELEASING.md`, "harbor and litellm are not in the closure" | | 4 | 415 pyright findings sit behind three disabled Unknown-propagation rules, and the config names the fix it has not built: a `TypedDict` model of the `~/.claude` JSONL record and the ATIF trajectory would close 144 of them. Until that model exists, every JSON-shaped value in the hottest modules is `dict[str, Any]` narrowed by `isinstance`, with the type checker's opinion switched off. The two heaviest concentrations are the converter's enrichment module at 59 findings and the analytics corpus reader at 33. | `wrong abstraction` | L | `pyproject.toml:641-671` | -| 5 | Some SQL statements are built by string interpolation through a hand-rolled quote-doubling escape, because DuckDB rejects prepared parameters as table-function arguments. ruff's `# noqa: S608` silences ruff only — bandit reports the same 15 as B608 and those findings upload to code scanning, so the suppression is not portable across the two scanners that both implement the rule. | `deprecated pattern` | L | `pyproject.toml:737-739`, `packages/atif-duck/src/atif_duck/infrastructure/registry.py:172`, `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:108` | +| 5 | Some SQL statements are built by string interpolation through a hand-rolled quote-doubling escape, because DuckDB rejects prepared parameters as table-function arguments. ruff's `# noqa: S608` silences ruff only; bandit reports the same 15 as B608 and those findings upload to code scanning, so the suppression is not portable across the two scanners that both implement the rule. | `deprecated pattern` | L | `pyproject.toml:737-739`, `packages/atif-duck/src/atif_duck/infrastructure/registry.py:172`, `packages/atif-embed/src/atif_embed/infrastructure/corpus_text_rows.py:108` | | 6 | No test reads a real `~/.claude` corpus. The whole integration tier is a few tests over small synthetic sessions, and parity oracles against the predecessor implementation are explicitly not part of this repo. The seven fidelity gaps are policy about real-world JSONL shapes that nothing real-world exercises. | `missing tests` | L | `packages/atif-cli/tests/test_integration.py:3-9`, `packages/atif-cli/tests/test_integration.py:24-34`, `AGENTS.md:92-94` | | 7 | RETIRED IN PART 2026-09-27. `hdbscan 0.8.44` never published an aarch64 wheel, so a first install on Graviton, an ARM CI runner, or a `linux/arm64` container used to compile Cython extensions and need a C toolchain; hdbscan left the lock with the structural pipelines, and every native package in the closure now ships an aarch64 wheel. The musl half stands: musl is not merely slow but impossible, because `lancedb 0.39.0` publishes no sdist at all, so there is nothing to build from. | `version pin` | L | `RELEASING.md:204-213` | -| 8 | The four size ratchets and the complexity ratchet are set at the measured worst function in the tree — `max-statements = 119` for `_friction_async`, `max-complexity = 38` for `enrich_trajectory`, `max-args = 19` for the `search` command. The ratchet stops growth and permanently blesses the current outliers; nothing in the config plans their reduction. | `wrong abstraction` | L | `pyproject.toml:258-281`, `packages/atif-analytics/src/atif_analytics/application/use_cases/friction.py:240`, `packages/atif-converter/src/atif_converter/domain/enrichment.py:198` | +| 8 | The four size ratchets and the complexity ratchet are set at the measured worst function in the tree: `max-statements = 119` for `_friction_async`, `max-complexity = 38` for `enrich_trajectory`, `max-args = 19` for the `search` command. The ratchet stops growth and permanently blesses the current outliers; nothing in the config plans their reduction. | `wrong abstraction` | L | `pyproject.toml:258-281`, `packages/atif-analytics/src/atif_analytics/application/use_cases/friction.py:240`, `packages/atif-converter/src/atif_converter/domain/enrichment.py:198` | | 9 | `TC001`/`TC002`/`TC003` are ignored on the strength of a manual probe: moving 58 imports under `TYPE_CHECKING` broke `typing.get_type_hints` on 159 first-party functions and classes. The 182-site `PLC0415` ignore beside it has a regression test proving its property in a fresh interpreter; the `get_type_hints` property has none, so the argument that justifies the ignore is not locked. | `missing tests` | M | `pyproject.toml:170-183`, `packages/atif-cli/tests/test_lean_import.py:35` | | 10 | Every security scanner has its findings exit code swallowed, by a stated contract: findings do not fail the job, only a scanner that produced no usable report does. Gating is delegated entirely to GitHub code-scanning alerts and branch protection, so nothing inside the repo fails on a new vulnerability. | `error handling` | M | `mise.toml:217-223`, `mise.toml:366`, `mise.toml:482` | -| 11 | The provider/dimension guard exists as two deliberate 83-line twins because the independence contract forbids either package importing the other. The drift pin reads the atif-duck twin as SOURCE TEXT, parses it with `ast`, and asserts substrings — including a raw `source.index("raise EmbeddingProviderMismatch")` that silently changes meaning if the raise site is renamed or a second raise appears. The two docstrings have already diverged; the pin covers the recovery string and the interpolation, not the rule. | `duplicated logic` | M | `packages/atif-embed/tests/test_guard_twin_pin.py:3-13`, `packages/atif-embed/tests/test_guard_twin_pin.py:93-95`, `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:5-16` | +| 11 | The provider/dimension guard exists as two deliberate 83-line twins because the independence contract forbids either package importing the other. The drift pin reads the atif-duck twin as SOURCE TEXT, parses it with `ast`, and asserts substrings, including a raw `source.index("raise EmbeddingProviderMismatch")` that silently changes meaning if the raise site is renamed or a second raise appears. The two docstrings have already diverged; the pin covers the recovery string and the interpolation, not the rule. | `duplicated logic` | M | `packages/atif-embed/tests/test_guard_twin_pin.py:3-13`, `packages/atif-embed/tests/test_guard_twin_pin.py:93-95`, `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:5-16` | | 12 | RETIRED 2026-09-27. The clustering path was the least-tested and least portable code in the tree. It was deleted with the cluster, terms, and community pipelines. | `missing tests` | Done | `packages/atif-analytics/src/atif_analytics/application/analyze.py:17-20` | -| 13 | Both binding contract documents disclaim describing the code. CONTRACT-V2 states outright that when it disagrees with the code, the code wins and the file is design intent "not a description of current behavior"; CONTRACT v1 calls its own scope section historical. A binding document that pre-emptively surrenders authority cannot be used to detect drift, which is the only thing a contract document is for. *judgment-call* — the disclaimers are honest, and honesty about staleness is still staleness. | `dead code adjacent` | M | `docs/CONTRACT-V2.md:3-8`, `docs/CONTRACT.md:5-8` | +| 13 | Both binding contract documents disclaim describing the code. CONTRACT-V2 states outright that when it disagrees with the code, the code wins and the file is design intent "not a description of current behavior"; CONTRACT v1 calls its own scope section historical. A binding document that pre-emptively surrenders authority cannot be used to detect drift, which is the only thing a contract document is for. *judgment-call*: the disclaimers are honest, and honesty about staleness is still staleness. | `dead code adjacent` | M | `docs/CONTRACT-V2.md:3-8`, `docs/CONTRACT.md:5-8` | | 14 | RETIRED 2026-09-27. `lancedb>=0.30,<0.38` used to be declared by both atif-analytics and atif-embed, with no comment on either side. atif-analytics dropped the dependency with its Lance reader, so only atif-embed declares it. | `version pin` | Done | `packages/atif-embed/pyproject.toml:22` | -| 15 | `FakeConverter` — a scriptable test double — ships inside the installable `atif-corpus` wheel under `infrastructure/`, and it is the converter the end-to-end materialize suite runs against. A fake that both mutates and reads corpus state stands in for the real harbor adapter in exactly the tests that would catch a state-transition bug. | `wrong abstraction` | S | `packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py:3-9`, `packages/atif-corpus/tests/test_materialize.py:25` | +| 15 | `FakeConverter`, a scriptable test double, ships inside the installable `atif-corpus` wheel under `infrastructure/`, and it is the converter the end-to-end materialize suite runs against. A fake that both mutates and reads corpus state stands in for the real harbor adapter in exactly the tests that would catch a state-transition bug. | `wrong abstraction` | S | `packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py:3-9`, `packages/atif-corpus/tests/test_materialize.py:25` | | 16 | Independently declared `DomainError(Exception)` bases with the same name and no shared ancestor, so a caller composing members cannot write one `except` clause and any symbol-name search cross-attributes them all. | `duplicated logic` | S | `packages/atif-converter/src/atif_converter/domain/errors.py:14`, `packages/atif-embed/src/atif_embed/domain/errors.py:14`, `packages/atif-models/src/atif_models/domain/ports.py:39` | | 17 | The 89% coverage floor is not part of the declared definition of done. `mise run check` depends on its gates and `test:cov` is not among them, so a local green run can drop coverage below the floor and only the CI job notices. | `missing tests` | S | `mise.toml:197-212`, `pyproject.toml:560`, `.github/workflows/check.yml:78` | @@ -72,7 +72,7 @@ because the search was narrow. `packages/atif-embed/src/atif_embed/infrastructure/lance_store.py:274` and an osv-scanner flag at `mise.toml:362`. - The closest thing to a marker is `RELEASING.md:215`, which names the harbor dependency - subtree "the standing follow-up" in prose — a declined-scope note in the release record + subtree "the standing follow-up" in prose: a declined-scope note in the release record rather than a comment in code. It is register row 3. The absence is enforced, not incidental. `select = ["ALL"]` at `pyproject.toml:146` leaves @@ -92,7 +92,7 @@ rather than by gate, which is why it is worth naming here. ### The toolchain config is the real debt register, and it is load-bearing Every suppression in `pyproject.toml` carries its measured count and its cost-of-removal -analysis, which makes the config the most honest debt document in the repo — and also +analysis, which makes the config the most honest debt document in the repo, and also means a large amount of unresolved work is encoded as a rule that is off rather than as an issue anyone tracks. The disabled Unknown-propagation rules cover a large block of findings and the config names what would let them back on, none of which exists. @@ -104,14 +104,14 @@ codebase owes than reading its comments, and none of it appears in any tracker. Shows up in: -- `pyproject.toml:153-184` — the ignores, each with its site count. -- `pyproject.toml:641-680` — the findings traced to origin, plus the stub gap. -- `pyproject.toml:258-281` — the ratchets set at the measured worst. -- `pyproject.toml:370-371` — `[tool.ty.rules] all = "error"` with nothing disabled beneath +- `pyproject.toml:153-184`: the ignores, each with its site count. +- `pyproject.toml:641-680`: the findings traced to origin, plus the stub gap. +- `pyproject.toml:258-281`: the ratchets set at the measured worst. +- `pyproject.toml:370-371`: `[tool.ty.rules] all = "error"` with nothing disabled beneath it, which is what the pattern looks like when it holds: a suppression here would need the same measured argument every entry above carries. -Cost: L — the analysis is done and written; the work it defers is a `TypedDict` model of +Cost: L. The analysis is done and written; the work it defers is a `TypedDict` model of the JSONL record, stub distributions, and refactors of named functions. ### One private upstream method carried the whole product (retired) @@ -135,30 +135,30 @@ SQL, and the same pure provider/dimension guard exists as twins. The escapers are the contract's price paid honestly: each package owns one copy, in `domain/`, and no import would be legal. The guard is the price paid dangerously. The mechanism keeping the twins honest is a test that reads the other package's file as text, parses it with `ast`, -and asserts substrings — including one raw `str.index` on a `raise` keyword. That test is +and asserts substrings, including one raw `str.index` on a `raise` keyword. That test is fragile in a way an import never is: it passes when the twins agree on a string and disagree on their logic, and it breaks on a rename that changes nothing. Shows up in: - `packages/atif-duck/src/atif_duck/domain/sql_literal.py:18` and - `packages/atif-embed/src/atif_embed/domain/sql_literal.py:24` — the domain-layer escapers, + `packages/atif-embed/src/atif_embed/domain/sql_literal.py:24`: the domain-layer escapers, each in its own package's `domain/`, each reached by that package's adapters. -- `packages/atif-cli/src/atif_cli/app.py:106` — the composition root's copy, which +- `packages/atif-cli/src/atif_cli/app.py:106`: the composition root's copy, which could legally import either of those above and does not. -- `packages/atif-embed/tests/test_guard_twin_pin.py:93-95` — the substring assertions that +- `packages/atif-embed/tests/test_guard_twin_pin.py:93-95`: the substring assertions that stand in for a shared import. -- `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:5-16` — the twin whose +- `packages/atif-duck/src/atif_duck/domain/embedding_guard.py:5-16`: the twin whose docstring has already drifted from its counterpart's. -Cost: M — collapsing the guard twins needs a decision about a shared-kernel package the +Cost: M. Collapsing the guard twins needs a decision about a shared-kernel package the contract forbids. The escapers need no decision: a shared kernel would absorb them, and short of one, one copy per package is the contract working rather than failing. ### Findings never fail anything -Every security scanner that runs has its findings exit code swallowed — most -sites use a trailing `|| true`, the rest an equivalent flag — under an explicit contract: a +Every security scanner that runs has its findings exit code swallowed (most +sites use a trailing `|| true`, the rest an equivalent flag) under an explicit contract: a finding does not fail the task, a missing report does. The reasoning is sound, because a non-zero exit kills the CI step before the SARIF upload. The consequence is that no gate inside this repository fails on a new @@ -171,13 +171,13 @@ The suppression ledger this tier generates is currently empty, which reads ident Shows up in: -- `mise.toml:217-223` — the contract, stated plainly. -- `mise.toml:366` and `mise.toml:482` — swallow sites, one of them +- `mise.toml:217-223`: the contract, stated plainly. +- `mise.toml:366` and `mise.toml:482`: swallow sites, one of them bandit's, whose B608 findings therefore reach code scanning permanently. -- `mise.toml:197-212` — the gates, without `test:cov`. -- `pyproject.toml:560` — the floor that only `.github/workflows/check.yml:78` enforces. +- `mise.toml:197-212`: the gates, without `test:cov`. +- `pyproject.toml:560`: the floor that only `.github/workflows/check.yml:78` enforces. -Cost: M — splitting each scanner into "run and upload" plus "assert the finding delta +Cost: M. Splitting each scanner into "run and upload" plus "assert the finding delta against a baseline" is mechanical; agreeing on the baseline is the work. ### The code the type checkers guard hardest is the code the tests reach least (clustering half retired) @@ -190,12 +190,12 @@ transitions that matter most are exercised against a double. Shows up in: -- `packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py:3-9` — the double +- `packages/atif-corpus/src/atif_corpus/infrastructure/fake_converter.py:3-9`: the double the materialize suite uses instead of the real adapter. -- `packages/atif-cli/tests/test_integration.py:3-9` — the whole integration tier: a few +- `packages/atif-cli/tests/test_integration.py:3-9`: the whole integration tier: a few small synthetic sessions. -Cost: M — a corpus fixture built from real, scrubbed `~/.claude` records is the whole of +Cost: M. A corpus fixture built from real, scrubbed `~/.claude` records is the whole of it, and it is what would exercise the fidelity gaps nothing real-world currently touches. ## See also diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 2f9b1ad..bc26d50 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -19,10 +19,10 @@ Convert one Claude Code session JSONL or one Codex CLI rollout JSONL to ATIF plu Flags: -- `SESSION-JSONL` / `--session-jsonl` — required path to the transcript: `~/.claude/projects//.jsonl`, or `~/.codex/sessions///
/rollout--.jsonl` under `--agent codex`. `:283` -- `--agent` — `claude-code` or `codex`; picks the converter and the fidelity policy. Defaults from `ATIF_SQL_AGENT`, then to `claude-code`. `:284` -- `--include-subagents` / `--no-subagents` — stage `/subagents/**.jsonl` side-files alongside the main chain; default `True`, with the negative form named explicitly. `:228` -- `--trajectory-out` — write the trajectory JSON here and `edges.jsonl` beside it, instead of stdout. `:229` +- `SESSION-JSONL` / `--session-jsonl`: required path to the transcript, `~/.claude/projects//.jsonl`, or `~/.codex/sessions///
/rollout--.jsonl` under `--agent codex`. `:283` +- `--agent`: `claude-code` or `codex`; picks the converter and the fidelity policy. Defaults from `ATIF_SQL_AGENT`, then to `claude-code`. `:284` +- `--include-subagents` / `--no-subagents`: stage `/subagents/**.jsonl` side-files alongside the main chain; default `True`, with the negative form named explicitly. `:228` +- `--trajectory-out`: write the trajectory JSON here and `edges.jsonl` beside it, instead of stdout. `:229` Exit codes: `0` ok, `2` empty session, `64` invalid input, `65` validation, `70` conversion. `:250` @@ -43,21 +43,21 @@ The convert-write stage runs across a process pool by default. Each worker build Flags: -- `--agent` — `claude-code` (default) or `codex`; picks the discovery layout and the default roots. `:426` -- `--force` / `--no-force` — re-materialize every quiescent session regardless of the watermark; default `False`. `:341` -- `--quiesce-seconds` — source-silence threshold; defaults from settings (contract: 300). `:342` -- `--source-root` — override the raw transcript root, otherwise `ATIF_SQL_SOURCE_ROOT` or `/projects`. `:343` -- `--corpus-root` — override the materialized corpus root, otherwise env or `~/.atif-sql/corpus/`. `:344` -- `--sessions` — comma-separated session-id filter; only these sessions are planned this pass. `:345` -- `--workers` — processes for the convert-write stage; default `ATIF_SQL_MATERIALIZE_WORKERS`, else `min(8, cpu_count)`. `1` is the single-process path. `:455` -- `--format` — report format. `:346` +- `--agent`: `claude-code` (default) or `codex`; picks the discovery layout and the default roots. `:426` +- `--force` / `--no-force`: re-materialize every quiescent session regardless of the watermark; default `False`. `:341` +- `--quiesce-seconds`: source-silence threshold; defaults from settings (contract: 300). `:342` +- `--source-root`: override the raw transcript root, otherwise `ATIF_SQL_SOURCE_ROOT` or `/projects`. `:343` +- `--corpus-root`: override the materialized corpus root, otherwise env or `~/.atif-sql/corpus/`. `:344` +- `--sessions`: comma-separated session-id filter; only these sessions are planned this pass. `:345` +- `--workers`: processes for the convert-write stage; default `ATIF_SQL_MATERIALIZE_WORKERS`, else `min(8, cpu_count)`. `1` is the single-process path. `:455` +- `--format`: report format. `:346` -Exit codes: `0` ok, `64` `--workers` below `1`; `78` the corpus at this root holds the other agent's sessions, refused with nothing touched; `78` suspicious scan — the source scan found zero sessions while the corpus holds materialized ones, so the pass refused to mark them source-removed and touched nothing. Check `--source-root`; a retry over the same root cannot succeed. `packages/atif-cli/src/atif_cli/app.py:429-443` -- `--format` — report format. `:346` +Exit codes: `0` ok, `64` `--workers` below `1`; `78` the corpus at this root holds the other agent's sessions, refused with nothing touched; `78` suspicious scan: the source scan found zero sessions while the corpus holds materialized ones, so the pass refused to mark them source-removed and touched nothing. Check `--source-root`; a retry over the same root cannot succeed. `packages/atif-cli/src/atif_cli/app.py:429-443` +- `--format`: report format. `:346` Each session is written as `trajectory.json.zst`, `edges.jsonl.zst` and `session_events.jsonl.zst` (zstd; decompressed, each is the plain file an earlier version wrote), a plain `loss_report.json` and `meta.json`, and the raw source archive. No per-session parquet is written, so the report's `artifact_seconds` is `0.0`. A corpus written by an earlier version keeps its plain files and parquet until `atif-sql corpus slim` converts it (see below). -Exit codes: `0` ok, `78` the corpus at this root holds the other agent's sessions, refused with nothing touched; `78` suspicious scan — the source scan found zero sessions while the corpus holds materialized ones, so the pass refused to mark them source-removed and touched nothing. Check `--source-root`; a retry over the same root cannot succeed. `packages/atif-cli/src/atif_cli/app.py:429-443` +Exit codes: `0` ok, `78` the corpus at this root holds the other agent's sessions, refused with nothing touched; `78` suspicious scan: the source scan found zero sessions while the corpus holds materialized ones, so the pass refused to mark them source-removed and touched nothing. Check `--source-root`; a retry over the same root cannot succeed. `packages/atif-cli/src/atif_cli/app.py:429-443` ## status @@ -72,11 +72,11 @@ It replays the planning a default `materialize` would do, so `staleness.stale` i Flags: -- `--agent` — `claude-code` (default) or `codex`; the scan and the reported `agent` field follow it. `:529` -- `--source-root` — override the raw transcript root. `:414` -- `--corpus-root` — override the materialized corpus root. `:415` -- `--quiesce-seconds` — source-silence threshold used to replay `materialize`'s planning decision. `:416` -- `--format` — report format. `:417` +- `--agent`: `claude-code` (default) or `codex`; the scan and the reported `agent` field follow it. `:529` +- `--source-root`: override the raw transcript root. `:414` +- `--corpus-root`: override the materialized corpus root. `:415` +- `--quiesce-seconds`: source-silence threshold used to replay `materialize`'s planning decision. `:416` +- `--format`: report format. `:417` Both roots resolve through `_corpus_settings` (`:433`), so a `--source-root` given without `--corpus-root` re-derives the corpus root from the overridden source's slug unless `ATIF_SQL_CORPUS_ROOT` is set (`:206`). @@ -93,15 +93,15 @@ Run one SQL statement against the atif-duck catalog and emit results. Flags: -- `SQL` — positional-only statement; omitting it without `--examples` is a parse error. `:498` -- `--examples` — short-circuit to the `examples` listing, honoring `--category` and `--requires`, without opening DuckDB. `:501` -- `--category` — forwarded to the `examples` listing. `:502` -- `--requires` — forwarded to the `examples` listing. `:503` -- `--agent` — `claude-code` (default) or `codex`; selects which corpus the statement reads. `:628` -- `--corpus-root` — override the materialized corpus root. `:504` +- `SQL`: positional-only statement; omitting it without `--examples` is a parse error. `:498` +- `--examples`: short-circuit to the `examples` listing, honoring `--category` and `--requires`, without opening DuckDB. `:501` +- `--category`: forwarded to the `examples` listing. `:502` +- `--requires`: forwarded to the `examples` listing. `:503` +- `--agent`: `claude-code` (default) or `codex`; selects which corpus the statement reads. `:628` +- `--corpus-root`: override the materialized corpus root. `:504` - `--all-corpora`: scope every view to every corpus the lake holds; `sessions.corpus` says which one a row came from. It has no per-session fallback, so it exits `78` without a usable lake and `64` together with `--no-lake`. - `--lake` / `--no-lake`: read the lake when it holds the corpus (default), or read the per-session files. -- `--format` — `table` on a TTY, a JSON array of row objects on a pipe. `:505` +- `--format`: `table` on a TTY, a JSON array of row objects on a pipe. `:505` The statement runs against a hardened connection: reads reach the registered views and nothing else, and nothing under the corpus root is writable. Before registration the connection is sized to the host: a memory cap derived from available RAM (half of physical RAM or 8 GiB, whichever is larger, never above 80% of what's available) and a thread count of one per 2 GiB of that cap, capped at the CPUs the process may use. `ATIF_SQL_QUERY_MEMORY_LIMIT` (a DuckDB size such as `6GB`) and `ATIF_SQL_QUERY_THREADS` override both; a malformed value exits 64. The spill directory is a private `mkdtemp` (mode 0700) under the system temp dir, the only directory the sandbox grants, and it's removed when the process exits. Extension auto-install and auto-load are off, and the lance extension is loaded only when it's already installed, so registration never reaches the network (`packages/atif-cli/src/atif_cli/app.py`, `_configure_query_resources`). @@ -126,19 +126,19 @@ Run the LLM analytics pipelines: classify, conflicts, friction, and perceived. Flags: -- `--since-days` — restrict the stages to sessions whose last step is within N days; default `30`. `:1242` -- `--limit` — cap the number of sessions, newest-first, per stage. `:1243` -- `--max-sessions` — hard per-run session ceiling per LLM pipeline; overrides `ATIF_SQL_LLM_MAX_SESSIONS_PER_RUN`, default 50. `:1244` -- `--max-cost-usd` — hard per-run dollar ceiling across all LLM pipelines, checked against running actual usage; overrides `ATIF_SQL_LLM_MAX_COST_USD_PER_RUN`, default 25.0. `:1245` -- `--no-dry-run` — execute the LLM stages for real, which costs money; the default is a dry run emitting plan dicts and cost estimates. `:1246` -- `--llm-only` — accepted and ignored. Every stage is an LLM stage now, and the flag stays so old crontab lines keep parsing. `:1247` -- `--skip-classify` — opt out of the classify stage. `:1248` -- `--skip-conflicts` — opt out of the conflicts stage. `:1249` -- `--skip-friction` — opt out of the friction stage. `:1250` -- `--skip-perceived` — opt out of the perceived stage. `:1251` -- `--corpus-root` — override the materialized corpus root. `:1252` -- `--no-lake` — read session data from the per-session files rather than the lake. `:1534` -- `--format` — summary format. `:1253` +- `--since-days`: restrict the stages to sessions whose last step is within N days; default `30`. `:1242` +- `--limit`: cap the number of sessions, newest-first, per stage. `:1243` +- `--max-sessions`: hard per-run session ceiling per LLM pipeline; overrides `ATIF_SQL_LLM_MAX_SESSIONS_PER_RUN`, default 50. `:1244` +- `--max-cost-usd`: hard per-run dollar ceiling across all LLM pipelines, checked against running actual usage; overrides `ATIF_SQL_LLM_MAX_COST_USD_PER_RUN`, default 25.0. `:1245` +- `--no-dry-run`: execute the LLM stages for real, which costs money; the default is a dry run emitting plan dicts and cost estimates. `:1246` +- `--llm-only`: accepted and ignored. Every stage is an LLM stage now, and the flag stays so old crontab lines keep parsing. `:1247` +- `--skip-classify`: opt out of the classify stage. `:1248` +- `--skip-conflicts`: opt out of the conflicts stage. `:1249` +- `--skip-friction`: opt out of the friction stage. `:1250` +- `--skip-perceived`: opt out of the perceived stage. `:1251` +- `--corpus-root`: override the materialized corpus root. `:1252` +- `--no-lake`: read session data from the per-session files rather than the lake. `:1534` +- `--format`: summary format. `:1253` Session data comes from the lake when it holds the corpus; a session whose lake rows aren't its current artifacts is read from its files, and with no usable lake the command warns once and reads the files. The summary's `session_source` says which source ran (`lake` or `files`), and on the lake `sessions_read_from_files` counts the sessions the last stage read from files. If a lake read fails partway (a `lake rebuild` swapped the lake out from under the run), the command warns once, reads the files for the rest of the run, and reports `lake_read_failed: true`. The plans, the rendered transcripts and the checkpoint bounds are the same whichever source runs. Outputs land under the corpus's `analytics/` directory either way. `:1624` @@ -155,15 +155,15 @@ Embed unembedded corpus steps with Cohere Embed v4 and append them to LanceDB. Flags: -- `--limit` — cap the number of steps embedded this run. `:736` -- `--all` — explicitly embed every unembedded step, a full backfill. `:737` -- `--dry-run` — preview only; emit the plan JSON with keys `pipeline, candidates, batches, batch_size, concurrency, model, limit, dry_run` and make no embedding calls. `:738` -- `--corpus-root` — override the materialized corpus root. `:739` -- `--format` — output format. `:740` +- `--limit`: cap the number of steps embedded this run. `:736` +- `--all`: explicitly embed every unembedded step, a full backfill. `:737` +- `--dry-run`: preview only; emit the plan JSON with keys `pipeline, candidates, batches, batch_size, concurrency, model, limit, dry_run` and make no embedding calls. `:738` +- `--corpus-root`: override the materialized corpus root. `:739` +- `--format`: output format. `:740` A real run requires an explicit scope: a bare `atif-sql embed` exits `64` with a hint rather than starting an unbounded backfill. `:778` -Exit codes: `0` success, `64` missing `--limit` or `--all`, `70` runtime (Bedrock, DuckDB, or Lance failure — transient, safe to retry), `78` terminal state, where the store or its config needs operator action and unattended lanes suppress retries. `:767` +Exit codes: `0` success, `64` missing `--limit` or `--all`, `70` runtime (Bedrock, DuckDB, or Lance failure: transient, safe to retry), `78` terminal state, where the store or its config needs operator action and unattended lanes suppress retries. `:767` ## search @@ -176,13 +176,13 @@ Semantic top-k nearest-neighbor search over step embeddings. Flags: -- `QUERY_TEXT` — required positional-only text, embedded with Cohere Embed v4 in `search_query` mode. `:829` -- `-k` / `--k` — top-k; default `10`. This is the CLI's only short flag. `:832` -- `--session-id` — confine the kNN to one session. `:833` -- `--corpus-root` — override the materialized corpus root. `:834` +- `QUERY_TEXT`: required positional-only text, embedded with Cohere Embed v4 in `search_query` mode. `:829` +- `-k` / `--k`: top-k; default `10`. This is the CLI's only short flag. `:832` +- `--session-id`: confine the kNN to one session. `:833` +- `--corpus-root`: override the materialized corpus root. `:834` - `--all-corpora`: search every corpus's embedding store and add a `corpus` column; exits `78` without a usable lake and `64` together with `--no-lake`. - `--lake` / `--no-lake`: read the lake when it holds the corpus (default), or read the per-session files. -- `--format` — output format. `:835` +- `--format`: output format. `:835` Output columns are `uuid`, `session_id`, `snippet`, and `sim` (`:927`), ranked by cosine distance ascending so the highest similarity comes first (`:935`). @@ -199,9 +199,9 @@ List tested example queries for every view and macro, derived from the static at Flags: -- `--category` — filter to one of `view`, `table-macro`, `scalar-macro`, validated against `CATEGORY_VALUES`. `:965` -- `--requires` — filter to one of `core`, `analytics`, `vss`, validated against `REQUIRES_VALUES`. `:966` -- `--format` — a TTY table grouped by `requires`, or a JSON object carrying `note` and `examples`. `:967` +- `--category`: filter to one of `view`, `table-macro`, `scalar-macro`, validated against `CATEGORY_VALUES`. `:965` +- `--requires`: filter to one of `core`, `analytics`, `vss`, validated against `REQUIRES_VALUES`. `:966` +- `--format`: a TTY table grouped by `requires`, or a JSON object carrying `note` and `examples`. `:967` Both value sets are `Literal` aliases in the producer package: `Requires` at `packages/atif-duck/src/atif_duck/domain/examples.py:48` and `Category` at `:53`, exported as tuples at `:55` and `:56`. @@ -218,7 +218,7 @@ List every core and analytics view with its columns and every macro signature, e Flags: -- `--format` — a TTY listing, or a JSON object carrying `views`, `view_requires` (view name to `requires`), `macros` (each with `name`, `params`, and `requires`), and `examples_hint`. `:1729` +- `--format`: a TTY listing, or a JSON object carrying `views`, `view_requires` (view name to `requires`), `macros` (each with `name`, `params`, and `requires`), and `examples_hint`. `:1729` The answer comes from the static `VIEW_SCHEMA`, `MACRO_SIGNATURES`, `ANALYTICS_VIEW_SCHEMA`, and `ANALYTICS_MACRO_SIGNATURES` dicts with no DuckDB import and no view registration. `requires` is `core` for an object that binds on any corpus, `analytics` for one that binds once `analyze` has written its parquet, and `vss` for one that needs the embedding store. `:1744` @@ -317,7 +317,7 @@ atif-sql cron COMMAND Inspect and manually install the `atif-sql` refresh cron lanes. `packages/atif-cli/src/atif_cli/cron.py:38` -The group is attached to the root router by `app.command(cron_app)` at `packages/atif-cli/src/atif_cli/app.py:98`, and its lanes — `materialize` on `*/10 * * * *` and `llm` on `20 10 * * *` — are declared once in `LANES` at `packages/atif-cli/src/atif_cli/cron.py:47`. The `structural` lane was removed on 2026-09-27 and is listed in `REMOVED_LANES` (`:56`). `scripts/atif-sql-refresh.sh` still accepts `structural` (and `struct`) and exits 0 after logging `[structural] lane removed 2026-09-27 ...`, so a crontab line nobody has deleted yet stays quiet. +The group is attached to the root router by `app.command(cron_app)` at `packages/atif-cli/src/atif_cli/app.py:98`, and its lanes (`materialize` on `*/10 * * * *` and `llm` on `20 10 * * *`) are declared once in `LANES` at `packages/atif-cli/src/atif_cli/cron.py:47`. The `structural` lane was removed on 2026-09-27 and is listed in `REMOVED_LANES` (`:56`). `scripts/atif-sql-refresh.sh` still accepts `structural` (and `struct`) and exits 0 after logging `[structural] lane removed 2026-09-27 ...`, so a crontab line nobody has deleted yet stays quiet. ### cron install @@ -330,7 +330,7 @@ Print the crontab block for the refresh lanes and never write it. The block incl Flags: -- `--script` — path to `atif-sql-refresh.sh`. `packages/atif-cli/src/atif_cli/cron.py:185` +- `--script`: path to `atif-sql-refresh.sh`. `packages/atif-cli/src/atif_cli/cron.py:185` Without the flag the script is located by walking up from this module (`:147`); a tree where `scripts/atif-sql-refresh.sh` is unreachable exits `64` demanding `--script` (`:175`). @@ -345,9 +345,9 @@ Report each lane's lock holder plus the last run and skip parsed from the refres Flags: -- `--script` — path to `atif-sql-refresh.sh`, whose `.run/` sibling holds the locks and the log. `packages/atif-cli/src/atif_cli/cron.py:201` -- `--tail` — how many trailing log lines to include; `0` disables, default `10`. `:202` -- `--format` — human lines on a TTY, JSON on a pipe. `:203` +- `--script`: path to `atif-sql-refresh.sh`, whose `.run/` sibling holds the locks and the log. `packages/atif-cli/src/atif_cli/cron.py:201` +- `--tail`: how many trailing log lines to include; `0` disables, default `10`. `:202` +- `--format`: human lines on a TTY, JSON on a pipe. `:203` The lock probe acquires and releases nonblocking, so the command perturbs no running lane. `:127` @@ -369,9 +369,9 @@ It never rewrites `meta.json`. A corpus the lake doesn't hold, or one whose sess Flags: -- `--corpus-root` — a corpus to slim; repeat for more. Default: every corpus the lake holds, plus every directory under `ATIF_SQL_CORPUS_BASE` that holds `sessions/`. -- `--lake-root` — the lake that verifies the corpora (default `ATIF_SQL_LAKE_ROOT`). -- `--no-dry-run` — act. +- `--corpus-root`: a corpus to slim; repeat for more. Default: every corpus the lake holds, plus every directory under `ATIF_SQL_CORPUS_BASE` that holds `sessions/`. +- `--lake-root`: the lake that verifies the corpora (default `ATIF_SQL_LAKE_ROOT`). +- `--no-dry-run`: act. The report lists, per corpus, the sessions in the old layout, the plain and compressed bytes, the parquet files and what happened to them (`deleted`, `would_delete`, `kept`, or `none`), and `bytes_before`, `bytes_after` and `bytes_freed`. diff --git a/docs/reference/public-api.md b/docs/reference/public-api.md index 6a94c59..eee9dc5 100644 --- a/docs/reference/public-api.md +++ b/docs/reference/public-api.md @@ -2,14 +2,14 @@ **The supported entry point is the `atif-sql` command, not an import.** This repository ships exactly one installable distribution, `atif-sql`, whose only console script is -`atif-sql = "atif_cli.app:main"` — `pyproject.toml:64`. Every workspace member ships inside that +`atif-sql = "atif_cli.app:main"` (`pyproject.toml:64`). Every workspace member ships inside that one wheel rather than as install targets, and the published name is deliberately split from the -`atif_cli` module that provides the entry point — `pyproject.toml:16-18`. Install with `uvx atif-sql` or +`atif_cli` module that provides the entry point (`pyproject.toml:16-18`). Install with `uvx atif-sql` or `uv tool install atif-sql`; `docs/reference/cli.md` is the primary reference for the surface a user actually calls. What follows documents the **internal seam**: the highest-traffic symbols that one workspace member -imports from another. That seam is enforced rather than conventional — the root `pyproject.toml` +imports from another. That seam is enforced rather than conventional: the root `pyproject.toml` declares `[tool.importlinter]` over every root package at `pyproject.toml:462-463`, with an `independence` contract at `pyproject.toml:514-517` forbidding atif-converter, atif-corpus, atif-duck, atif-models, and atif-embed from importing each other at all, and a `forbidden` contract at @@ -24,10 +24,10 @@ abstractions are `typing.Protocol` classes, and all of them sit under a `domain/ `packages/atif-embed/src/atif_embed/domain/ports.py:31`, `:59`, and `:87`, and `LlmStructuredProvider` at `packages/atif-models/src/atif_models/domain/ports.py:116`. No `abstractmethod` exists anywhere in the workspace, so adapters satisfy a port structurally. Second, -the name `DomainError` is declared independently more than once — one base class per erroring package, at +the name `DomainError` is declared independently more than once (one base class per erroring package, at `packages/atif-converter/src/atif_converter/domain/errors.py:14`, `packages/atif-embed/src/atif_embed/domain/errors.py:14`, and -`packages/atif-models/src/atif_models/domain/ports.py:39` — and they are unrelated types that +`packages/atif-models/src/atif_models/domain/ports.py:39`), and they are unrelated types that share only a spelling. There is no HTTP or RPC surface: a grep for route decorators, `FastAPI(`, `APIRouter`, `add_route`, @@ -51,8 +51,8 @@ support, carrying the corpus root and every pipeline knob. def build_examples() -> tuple[ExampleQuery, ...]: ``` -Derives the full example inventory from the static catalogs in a fixed order — core views, the VSS -view, analytics views, then core macros, the VSS macro, analytics macros — so the CLI listing and the +Derives the full example inventory from the static catalogs in a fixed order (core views, the VSS +view, analytics views, then core macros, the VSS macro, analytics macros), so the CLI listing and the JSON array are deterministic across runs. `packages/atif-duck/src/atif_duck/domain/examples.py:191-252` @@ -71,7 +71,7 @@ def build_plan( ) -> MaterializationPlan: ``` -Partitions scanned sessions into the plan buckets — the pure decision at the centre of +Partitions scanned sessions into the plan buckets: the pure decision at the centre of materialization, taking the watermark and the quiescence policy as data. `packages/atif-corpus/src/atif_corpus/domain/sessions.py:159-211` @@ -154,7 +154,7 @@ it in the same module. class EmbedSettings(BaseSettings): ``` -Env-driven settings for the embedding pipeline — model, batch size, concurrency, and the LanceDB URI. +Env-driven settings for the embedding pipeline: model, batch size, concurrency, and the LanceDB URI. `packages/atif-embed/src/atif_embed/infrastructure/settings.py:19-64` @@ -187,7 +187,7 @@ def estimate_cost(spec: ModelSpec, *, input_tokens: int, output_tokens: int) -> ``` USD estimate for one call or an accumulated pipeline against the prices carried on `spec`, returning -`None` when either price is unknown — which a caller must render as pricing unavailable, never as +`None` when either price is unknown, which a caller must render as pricing unavailable, never as zero. `packages/atif-models/src/atif_models/domain/registry.py:125-137` @@ -343,7 +343,7 @@ finished. class RefusalError(DomainError): ``` -Terminal error for a model refusal — a content filter or a refusal finish reason — as distinct from a +Terminal error for a model refusal (a content filter or a refusal finish reason), as distinct from a retryable provider fault. `packages/atif-models/src/atif_models/domain/ports.py:43-48` diff --git a/mise.toml b/mise.toml index 519df0a..e5ac51d 100644 --- a/mise.toml +++ b/mise.toml @@ -709,7 +709,7 @@ description = "Vale over the published prose: fail on an error-level alert, a sh # every one of those files, so a broken glob, a moved tree or an emptied directory is a red gate # rather than "0 errors in 5 files". Warnings are a ratchet: .vale-baseline.json holds the count # per rule, and a rule above its count, or a warning from a rule it does not name, fails naming -# the rule (`Google.EmDash: 450 warnings, baseline 449`). A count below passes and asks for +# the rule (`Google.WordListCase: 121 warnings, baseline 120`). A count below passes and asks for # `mise run docs:prose:baseline`. A missing, empty or unparsable baseline fails. No `sources`: a # few seconds, and a skipped gate reads like a passing one. # diff --git a/packages/atif-cli/tests/test_vale_gate.py b/packages/atif-cli/tests/test_vale_gate.py index 8e894cd..4c543ea 100644 --- a/packages/atif-cli/tests/test_vale_gate.py +++ b/packages/atif-cli/tests/test_vale_gate.py @@ -8,10 +8,14 @@ matches fewer files than its floor fails naming that glob, and on the real tree Vale checks exactly the files an independent walk of the scope finds. -The warning ratchet carries the same proof: a planted em dash beside the real tree turns it red -naming Google.EmDash, a rule deleted from the baseline is red naming that rule, one warning -fewer than the baseline is green and asks for the baseline to come down, and a baseline that is -missing, empty, unparsable or not counts fails before Vale runs. +A spaced em dash is an error, not a warning: one planted on its own, one in a wrapped list item +(where Vale 3.24.0 can fold two alerts into one) and one beside the real tree each turn the gate +red on Google.EmDash, and the rule is not in the baseline. + +The warning ratchet carries the same proof: a planted Google.Will warning beside the real tree +turns it red naming that rule, a rule deleted from the baseline is red naming that rule, one +warning fewer than the baseline is green and asks for the baseline to come down, and a baseline +that is missing, empty, unparsable or not counts fails before Vale runs. The fixtures live under `.vale/fixtures/`, outside the gate's scope, so the real tree stays clean while the plants stay committed. Vale is the mise-pinned binary: run this through @@ -86,6 +90,31 @@ def test_banned_phrase_fails_on_its_rule() -> None: assert _checked(result.stdout) == (1, 1, 1) +_EMDASH = "Google.EmDash: Don't put a space before or after a dash." + + +@pytest.mark.parametrize( + ("name", "line"), + [ + ("planted-emdash.md", 3), + # The only dash on the page sits in a list item that wraps onto a second line, the shape + # whose alert Vale 3.24.0 can merge into a later one: alone, it is still reported. + ("wrapped-emdash.md", 3), + # Two dashes that Vale reports as one alert: a merge folds two errors into one, never + # into none, so the page is red either way. + ("merged-emdash.md", 4), + ], +) +def test_spaced_em_dash_fails_as_an_error(name: str, line: int) -> None: + result = _run(FIXTURES / name) + assert result.returncode == 1, result.stdout + result.stderr + assert f".vale/fixtures/{name}:{line}:" in result.stdout + assert _EMDASH in result.stdout + checked, total, errors = _checked(result.stdout) + assert (checked, total) == (1, 1) + assert errors >= 1 + + def test_misspelling_fails_on_spelling() -> None: result = _run(FIXTURES / "planted-spelling.md") assert result.returncode == 1, result.stdout + result.stderr @@ -327,23 +356,44 @@ def test_committed_baseline_is_usable() -> None: baseline = gate.load_baseline(gate.BASELINE) assert baseline, baseline assert all(rule.split(".")[0] in {"Vale", "Google", "proselint"} for rule in baseline) + # An error-level rule fails on one alert; a baseline entry for it would read as a budget. + assert "Google.EmDash" not in baseline, baseline -def test_planted_em_dash_fails_naming_its_rule( +def test_planted_em_dash_beside_the_tree_fails_as_an_error( real_counts: dict[str, int], tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], ) -> None: - # One more spaced em dash on a docs page than the baseline allows: the real tree plus a - # page carrying exactly one, under the real tree's own counts. + # The real tree carries no spaced dash, so one page carrying one is red as an error under + # the real tree's own warning counts, with no ratchet line: no budget absorbs it. gate = _gate_with_baseline(monkeypatch, tmp_path, real_counts) by_glob = cast("dict[str, list[Path]]", gate.scope_by_glob()) by_glob["docs/**/*.md"] = [*by_glob["docs/**/*.md"], FIXTURES / "planted-emdash.md"] assert _main_with(monkeypatch, gate, by_glob) == 1 out, err = capsys.readouterr() - base = real_counts["Google.EmDash"] - assert err.splitlines() == [f"selftest: Google.EmDash: {base + 1} warnings, baseline {base}"] + assert out.splitlines()[0] == f".vale/fixtures/planted-emdash.md:3:50: {_EMDASH}" + assert ": 1 error, " in out + assert "0 rules above the baseline" in out + assert err == "" + + +def test_planted_warning_fails_naming_its_rule( + real_counts: dict[str, int], + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + # One more Google.Will on a docs page than the baseline allows: the real tree plus a page + # carrying exactly one, under the real tree's own counts. + gate = _gate_with_baseline(monkeypatch, tmp_path, real_counts) + by_glob = cast("dict[str, list[Path]]", gate.scope_by_glob()) + by_glob["docs/**/*.md"] = [*by_glob["docs/**/*.md"], FIXTURES / "planted-warning.md"] + assert _main_with(monkeypatch, gate, by_glob) == 1 + out, err = capsys.readouterr() + base = real_counts["Google.Will"] + assert err.splitlines() == [f"selftest: Google.Will: {base + 1} warnings, baseline {base}"] assert "1 rule above the baseline" in out assert gate.LOWER_HINT not in out @@ -370,13 +420,13 @@ def test_one_warning_fewer_passes_and_asks_to_lower( monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], ) -> None: - # The tree one Google.EmDash below its baseline: green, and the gate says to ratchet down. - base = real_counts["Google.EmDash"] - gate = _gate_with_baseline(monkeypatch, tmp_path, {**real_counts, "Google.EmDash": base + 1}) + # The tree one Google.Will below its baseline: green, and the gate says to ratchet down. + base = real_counts["Google.Will"] + gate = _gate_with_baseline(monkeypatch, tmp_path, {**real_counts, "Google.Will": base + 1}) assert gate.main(["--label", "selftest"]) == 0 out, err = capsys.readouterr() assert err == "" - assert f"selftest: Google.EmDash: {base} warnings, baseline {base + 1}" in out + assert f"selftest: Google.Will: {base} warnings, baseline {base + 1}" in out assert f"selftest: {gate.LOWER_HINT}" in out assert "0 rules above the baseline" in out @@ -426,7 +476,7 @@ def test_path_mode_ignores_the_baseline( # The selftest's PATH mode stays error-only: no baseline is read, so a missing one is fine. gate = _load_gate() monkeypatch.setattr(gate, "BASELINE", tmp_path / "absent.json") - assert gate.main(["--label", "selftest", str(FIXTURES / "planted-emdash.md")]) == 0 + assert gate.main(["--label", "selftest", str(FIXTURES / "planted-warning.md")]) == 0 assert "1 warning (reported, not gated)" in capsys.readouterr().out @@ -435,19 +485,19 @@ def test_path_mode_ignores_the_baseline( reason=( "Vale 3.24.0 anchors an alert by searching for its matched text, so the dash of a list " "item that wraps onto a second line lands on a later ' \u2014 ' that carries its own " - "alert, and the two report as one (vale-cli/vale#1147 is the same anchoring). A plant " - "appended after such an item can lower the Google.EmDash count instead of raising it." + "alert, and the two report as one (vale-cli/vale#1147 is the same anchoring). A warning " + "plant appended after such an item can lower a rule's count instead of raising it." ), ) def test_vale_reports_every_spaced_dash( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] ) -> None: - # Two spaced dashes, so two warnings. Strict: the day Vale reports both, this XPASS fails, + # Two spaced dashes, so two errors. Strict: the day Vale reports both, this XPASS fails, # and the xfail and the caveat in scripts/vale_gate.py come out together. gate = _load_gate() monkeypatch.setattr(gate, "BASELINE", tmp_path / "absent.json") - assert gate.main(["--label", "selftest", str(FIXTURES / "merged-emdash.md")]) == 0 - assert "2 warnings (reported, not gated)" in capsys.readouterr().out + assert gate.main(["--label", "selftest", str(FIXTURES / "merged-emdash.md")]) == 1 + assert ": 2 errors, 0 warnings (reported, not gated)" in capsys.readouterr().out def test_ratchet_names_rises_new_rules_and_falls() -> None: @@ -467,7 +517,7 @@ def test_write_baseline_rewrites_from_the_tree( # A stale baseline: one rule too high, one too low, one retired, one missing. missing = min(real_counts) stale = {k: v for k, v in real_counts.items() if k != missing} - stale["Google.EmDash"] += 5 + stale["Google.Headings"] += 5 stale["Google.WordListCase"] -= 1 stale["Retired.Rule"] = 2 gate = _gate_with_baseline(monkeypatch, tmp_path, stale) @@ -476,8 +526,8 @@ def test_write_baseline_rewrites_from_the_tree( written = json.loads(gate.BASELINE.read_text()) assert written == {**real_counts, "Retired.Rule": 0} assert list(written) == sorted(written) - em, wl = real_counts["Google.EmDash"], real_counts["Google.WordListCase"] - assert f"selftest: Google.EmDash: lowered {em + 5} -> {em}" in out + hd, wl = real_counts["Google.Headings"], real_counts["Google.WordListCase"] + assert f"selftest: Google.Headings: lowered {hd + 5} -> {hd}" in out assert ( f"selftest: Google.WordListCase: raised {wl - 1} -> {wl}: a baseline only goes down" in out ) diff --git a/scripts/vale_gate.py b/scripts/vale_gate.py index f82a18e..f90bc1d 100755 --- a/scripts/vale_gate.py +++ b/scripts/vale_gate.py @@ -28,7 +28,7 @@ the default scope: `.vale-baseline.json` at the root holds `{rule: count}` for every warning-level rule, and the gate fails when any rule's count rises above its baseline or a rule the baseline does not name raises a warning, one stderr line per rule -(`Google.EmDash: 450 warnings, baseline 449`). A count below its baseline passes and prints +(`Google.WordListCase: 121 warnings, baseline 120`). A count below its baseline passes and prints `lower the baseline: mise run docs:prose:baseline`. The baseline itself must be usable: a missing, unparsable, empty or non-count file fails the gate before Vale runs, so a lost baseline is never a pass. `.vale.ini` says why each rule below error is there. @@ -36,11 +36,13 @@ One limit is Vale's, not this script's: Vale 3.24.0 places an alert by searching the file for its matched text, and an alert from a Markdown list item that wraps onto a second line can land on a later occurrence of the same text. When that occurrence raises the same rule, the two -report as one, so a page that adds a spaced em dash after such an item can count fewer -Google.EmDash warnings, not more (`.vale/fixtures/merged-emdash.md` reproduces it, and a strict -xfail in the selftest turns red when Vale fixes it). The ratchet holds Vale's count as Vale -reports it; a count that falls on a change that added a warning is that merge, so leave the -baseline where it is. +report as one (`.vale/fixtures/merged-emdash.md` reproduces it with two spaced em dashes, and a +strict xfail in the selftest turns red when Vale fixes it). A warning count can therefore fall +on a change that added a warning; the ratchet holds Vale's count as Vale reports it, so a count +that falls on such a change is that merge, and the baseline stays where it is. The rule this hit +hardest, Google.EmDash, is an error for that reason and is not in the baseline: a merge folds two +errors into one, never into none, so one added spaced dash is red, wrapped list item or not +(`.vale/fixtures/wrapped-emdash.md`). `--write-baseline` (`mise run docs:prose:baseline`) rewrites `.vale-baseline.json` from the current tree, after the same scope checks. A rule that drops to zero keeps its entry at 0, so @@ -169,7 +171,7 @@ def _baseline_problem(name: str, doc: object) -> str: return f"{name} is empty: a baseline naming no rule would gate nothing" for rule, count in cast("dict[object, object]", doc).items(): if not isinstance(rule, str) or "." not in rule: - return f"{name}: {rule!r} is not a rule name such as Google.EmDash" + return f"{name}: {rule!r} is not a rule name such as Google.WordListCase" if isinstance(count, bool) or not isinstance(count, int) or count < 0: return f"{name}: {rule} has count {count!r}, not an integer of 0 or more" return "" diff --git a/site/authored/agents.md b/site/authored/agents.md index 5b0330d..4542c92 100644 --- a/site/authored/agents.md +++ b/site/authored/agents.md @@ -14,7 +14,7 @@ its signature. It reads a static catalog rather than the corpus, so it answers i of a second, needs no materialized data, and doubles as a liveness check on the install. Its output ends with a pointer to the examples command. -`atif-sql examples` — the same thing as `atif-sql query --examples` — prints a runnable query per view +`atif-sql examples` (the same thing as `atif-sql query --examples`) prints a runnable query per view and per macro. Those queries are **derived from the catalog**, never hardcoded per object, and every one is executed against a fixture corpus by `atif-duck`'s test suite. Filter with `--requires core|analytics|vss` and `--category view|table-macro|scalar-macro`. Piped output is JSON. @@ -31,14 +31,14 @@ adds what a schema cannot state: what a silence means, and which surface you are - **The materialized corpus is what the views read, not `~/.claude`.** `atif-sql` converts each session to ATIF once, writes it under the corpus root, and every view reads those artifacts. A - session that has not been materialized is absent from every view — not empty, absent. `atif-sql + session that has not been materialized is absent from every view: not empty, absent. `atif-sql materialize` is what makes it visible, and `atif-sql status` replays the same planning decision read-only, so it answers "what would a materialize pass do right now" without converting anything. - **A session still being written is skipped on purpose.** Materialization is gated on quiescence: a session is converted when its newest source file has been silent for longer than the quiesce - threshold and is newer than the recorded watermark. So the most interesting session — the one open - in another terminal — is the one deliberately not in the corpus yet. `--force` overrides the + threshold and is newer than the recorded watermark. So the most interesting session (the one open + in another terminal) is the one deliberately not in the corpus yet. `--force` overrides the watermark; nothing overrides physics, so wait for the write to settle. - **`requires` on an example is a precondition, not a hint.** An example tagged `analytics` needs @@ -54,7 +54,7 @@ adds what a schema cannot state: what a silence means, and which surface you are - **Conversion loss is accounted for, not hidden.** Every materialized session carries a loss report beside its trajectory, and each known upstream conversion gap is a named type in the converter's fidelity policy rather than a paragraph in a changelog. A field absent from the ATIF document is not - evidence that it was absent from the transcript — read the loss report before concluding that. + evidence that it was absent from the transcript. Read the loss report before concluding that. - **Branch on the exit code, never on the message text.** The codes are a stable wire contract: `64` for malformed input or malformed SQL, `65` for a catalog miss or a store written by another embedding @@ -64,18 +64,18 @@ adds what a schema cannot state: what a silence means, and which surface you are ## 3. Which surface you are on -The test for each row is free — readable from your tool list, your shell, or the repository — and +The test for each row is free (readable from your tool list, your shell, or the repository) and never a call that has to fail first. | Signal | Surface | Entry point | | --- | --- | --- | | You can run a shell command | CLI | `atif-sql schema`, then the command you need | | You can execute Python in this workspace | In-process library | Import the package that owns the capability: `atif_duck` to register views on a connection, `atif_corpus` to materialize | -| Neither | Read-only | You are reading documentation. Fetch the Markdown, not the HTML — section 7 has the URLs | +| Neither | Read-only | You are reading documentation. Fetch the Markdown, not the HTML; section 7 has the URLs | The CLI and the library are the same implementation: `atif-cli` is a composition root and holds no query logic of its own, so a fact learned on one surface transfers to the other. The layering is -enforced rather than documented — import-linter fails the build when a leaf package imports another +enforced rather than documented: import-linter fails the build when a leaf package imports another leaf package. :::agent @@ -96,7 +96,7 @@ atif-sql query 'SELECT * FROM tool_rank(30) LIMIT 10' ``` Two habits change the shape of everything after. Output adapts to the channel: a TTY gets a plain -table, a pipe gets JSON — so redirect when you intend to parse, and never parse the table. And an +table, a pipe gets JSON, so redirect when you intend to parse, and never parse the table. And an example is meant to be copied verbatim before it is adapted: the session-id exemplar in every example is a subquery over `sessions`, so a pasted example runs against any corpus rather than needing an id you do not have yet. @@ -133,7 +133,7 @@ fixed by changing the environment. This table is built at publish time from the set of pages the site actually wrote, so a page added to the documentation tree appears here without an edit and a removed page cannot leave a row behind. The -right-hand column is that page's raw Markdown twin — fetch that instead of the page. +right-hand column is that page's raw Markdown twin; fetch that instead of the page. GENERATED-READ-NEXT-TABLE @@ -156,7 +156,7 @@ instead: that is the string you can hand to a grep. :::agent **For an agent.** Fetch `llms-small.txt` before `llms-full.txt`. When you already know which page you -want, fetch that page's `.md` twin instead of either bundle — a fraction of the tokens for the same +want, fetch that page's `.md` twin instead of either bundle: a fraction of the tokens for the same text. ::: diff --git a/site/authored/index.md b/site/authored/index.md index 7b505d2..6c4b949 100644 --- a/site/authored/index.md +++ b/site/authored/index.md @@ -1,10 +1,10 @@ --- title: atif-sql -description: ATIF-native analytics over Claude Code and Codex CLI agent trajectories — convert sessions to ATIF, materialize a corpus, query it through DuckDB views. +description: "ATIF-native analytics over Claude Code and Codex CLI agent trajectories: convert sessions to ATIF, materialize a corpus, query it through DuckDB views." --- atif-sql reads Claude Code session transcripts and Codex CLI rollouts, converts each session to -[ATIF](https://github.com/laude-institute/harbor) — Harbor's Agent Trajectory Interchange Format — +[ATIF](https://github.com/laude-institute/harbor) (Harbor's Agent Trajectory Interchange Format), materializes the results as a corpus of ATIF documents on disk, and layers DuckDB views and macros over that corpus. Converting once at the boundary, with an explicit and tested fidelity policy for what the upstream adapter drops, replaces re-deriving trajectory semantics inside every SQL view. @@ -53,7 +53,7 @@ atif-sql query 'SELECT * FROM tool_rank(30) LIMIT 10' ``` `atif-sql examples` derives its output from the catalog rather than carrying hardcoded strings, and -every example is executed against a fixture corpus by `atif-duck`'s own test suite — so the listing +every example is executed against a fixture corpus by `atif-duck`'s own test suite, so the listing cannot drift from the schema it documents. :::agent