diff --git a/.vale-baseline.json b/.vale-baseline.json index 41e883e..f84a3de 100644 --- a/.vale-baseline.json +++ b/.vale-baseline.json @@ -1,6 +1,6 @@ { "Google.Colons": 9, - "Google.EmDash": 449, + "Google.EmDash": 448, "Google.FirstPerson": 3, "Google.Headings": 63, "Google.OxfordComma": 28, diff --git a/README.md b/README.md index 07a6826..dc16adc 100644 --- a/README.md +++ b/README.md @@ -36,10 +36,45 @@ litellm's price data, so neither `harbor` nor `litellm` is installed. Prebuilt w 3.13 on manylinux x86_64 and aarch64, macOS arm64, and Windows x86_64. Alpine and other musl targets are not supported. [RELEASING.md](RELEASING.md) says how to measure the install. -`atif-sql analyze`, `atif-sql embed`, and `atif-sql search` call Amazon Bedrock and cost money -per invocation. Each one is dry-run by default and spends only when asked. Nothing else in the +`atif-sql analyze`, `atif-sql embed`, and `atif-sql search` call Amazon Bedrock and cost money. +`analyze` is a dry run until you pass `--no-dry-run`, `embed` refuses to run without `--limit` or +`--all` (and `--dry-run` previews it), and `search` embeds one query per call. Nothing else in the tool needs a credential. +## What you can do + +Start with the question you want to answer or the work you want to keep. Each command links +to its flags and exit codes in [the CLI reference](docs/reference/cli.md). + +Analysis, embedding, and text search use Amazon Bedrock. `analyze` previews by default; +`embed` needs `--dry-run` to preview an explicit scope; `search` charges for a query embedding +on each successful call. Rows marked **costs money** describe paid execution. + +| Task | Command | Outcome | +| --- | --- | --- | +| Keep a history of your Claude Code work | [`atif-sql materialize`](docs/reference/cli.md#materialize) | Saves quiet sessions for later queries and keeps them when source transcripts disappear. Re-runs update changed sessions and those with an older converter generation. | +| Bring your Codex work into the same analysis workflow | [`atif-sql materialize --agent codex`](docs/reference/cli.md#materialize) | Saves Codex rollouts in a separate corpus with the same query surface as Claude Code. | +| Inspect or share one conversation as ATIF | [`atif-sql convert `](docs/reference/cli.md#convert) | Prints its trajectory and loss report. Add `--trajectory-out` to save the trajectory and adjacent `edges.jsonl` file. | +| Know whether your recent work is included | [`atif-sql status`](docs/reference/cli.md#status) | Shows corpus freshness, session counts, and what another `materialize` pass would update, without changing data. | +| Query a large history without rereading every transcript | [`atif-sql lake rebuild`](docs/reference/cli.md#lake-rebuild) | Builds the shared DuckLake query store. Later `materialize` passes keep it current. | +| Check that query results match your saved conversations | [`atif-sql lake verify`](docs/reference/cli.md#lake-verify) | Compares stored rows with session artifacts and exits 65 if they differ. | +| Diagnose missing or stale query data | [`atif-sql lake status`](docs/reference/cli.md#lake-status) | Shows whether the lake exists, is current, and contains the expected corpora, plus its last write. | +| Recover disk space from query storage | [`atif-sql lake compact`](docs/reference/cli.md#lake-compact) | Merges small files, expires old snapshots, and removes unreferenced files while keeping current rows. | +| Find the fields you need to answer a question | [`atif-sql schema`](docs/reference/cli.md#schema) | Lists columns and macro signatures, tagged by required data: `core`, `analytics`, or `vss`. No corpus needed. | +| Start with a query that already works | [`atif-sql examples`](docs/reference/cli.md#examples) | Gives tested queries to copy or adapt. Filters select the required data and query type. | +| Answer your own question about past sessions | [`atif-sql query ''`](docs/reference/cli.md#query) | Returns a table to read or JSON to pipe into another tool, through a read-only sandboxed connection. | +| Compare work across Claude Code and Codex | [`atif-sql query --all-corpora 'SELECT agent, count(*) FROM sessions GROUP BY agent'`](docs/reference/cli.md#query) | Counts sessions by agent across all corpora in the lake. Other queries can use `sessions.corpus` to identify each source. Needs the lake. | +| See which tools your agents rely on most | [`atif-sql query 'SELECT * FROM tool_rank(30) LIMIT 10'`](docs/reference/cli.md#query) | Ranks tool use by call count over the last 30 days. | +| Estimate the cost of a session | [`atif-sql query 'SELECT * FROM cost_estimate((SELECT session_id FROM sessions LIMIT 1)) LIMIT 10'`](docs/reference/cli.md#query) | Gives a lower-bound USD estimate and priced/unpriced step counts. Unpriced steps and cache-read charges are excluded. Replace the subquery to choose a session. | +| Find recurring moments when you correct your agents | [`atif-sql query 'SELECT * FROM friction_counts(30) LIMIT 10'`](docs/reference/cli.md#query) | Counts detected friction labels over the last 30 days, after paid analysis has run. | +| Plan a review of agent interactions before paying | [`atif-sql analyze`](docs/reference/cli.md#analyze) | Previews the work and estimated cost. **Costs money** with `--no-dry-run`, which fills the classifications, conflicts, user friction, and perceived errors views. | +| Preview how much history to prepare for semantic search | [`atif-sql embed --limit 100 --dry-run`](docs/reference/cli.md#embed) | Previews embedding up to 100 steps. Removing `--dry-run` **costs money** and prepares those steps for search. | +| Find prior work when you remember the topic, not the words | [`atif-sql search ''`](docs/reference/cli.md#search) | **Costs money** for one query embedding. Finds similar steps with session identifiers and snippets. Needs existing embeddings from `embed`. | +| Find related work across both agent histories | [`atif-sql search --all-corpora ''`](docs/reference/cli.md#search) | **Costs money.** Searches all corpora's embedding stores and identifies each result's corpus. Needs the lake and existing embeddings. | +| Free space in a corpus from an older version | [`atif-sql corpus slim`](docs/reference/cli.md#corpus-slim) | Previews byte savings. `--no-dry-run` compresses JSON and removes old parquet only after lake verification succeeds. | +| Keep your history refreshed without remembering each run | [`atif-sql cron install --script `](docs/reference/cli.md#cron-install) | Prints a schedule to review and install manually. The schedule includes paid embedding and analytics. Obtain `scripts/atif-sql-refresh.sh` from a repo checkout; packaged installs need its explicit path. | +| Check whether scheduled refreshes ran | [`atif-sql cron status --script `](docs/reference/cli.md#cron-status) | Shows each lane's last completion, skip, and current lock holder from the refresh script's logs. | + ## How the source is organized The directories under `packages/` are internal structure, not separate installs. They @@ -52,9 +87,9 @@ the only thing documented as installable is the `atif-sql` CLI above. | `atif-corpus` | Corpus materialization: discovery, watermarks, quiescence, atomic artifact writes | | `atif-duck` | DuckDB views + macros over the materialized corpus (core surface plus the v2 analytics surface) | | `atif-models` | Model alias registry + structured-output LLM client; no other package hardcodes a model id | -| `atif-analytics` | the v2 pipelines — LLM (classify, trajectory, conflicts, friction, perceived) and structural (cluster, terms, community) | +| `atif-analytics` | the LLM pipelines `atif-sql analyze` runs: classify, conflicts, friction, and perceived | | `atif-embed` | Cohere Embed v4 on Bedrock + LanceDB vector store + embedding backfill | -| `atif-cli` | the composition root, and the source of the `atif-sql` command: `convert`, `materialize`, `status`, `query`, `analyze`, `embed`, `search`, `examples`, `schema`, `cron` | +| `atif-cli` | the composition root, and the source of the `atif-sql` command (see [What you can do](#what-you-can-do)) | ## Quick start @@ -67,15 +102,8 @@ atif-sql status # corpus and lake freshness, read-only atif-sql query 'SELECT * FROM sessions LIMIT 5' ``` -After `lake rebuild`, every `materialize` keeps the lake current (it replaces the rows of the -sessions it wrote), and `query` reads the lake instead of opening each session's files. Without a -lake, or with one that's out of date, `query` prints a one-line warning and reads the per-session -files as before. `atif-sql embed` and `atif-sql search` read the lake the same way: embed reads -only the steps that changed since its last complete run, and `search --all-corpora` searches every -corpus's store at once. `atif-sql lake verify` checks the lake against the artifacts, and -`atif-sql lake compact` merges small files and drops old snapshots (the refresh script's nightly -`compact` lane runs it). The lake lives at `~/.atif-sql/lake/` (`ATIF_SQL_LAKE_ROOT` moves it) -and holds every corpus. +The lake lives at `~/.atif-sql/lake/` (`ATIF_SQL_LAKE_ROOT` moves it) and holds every corpus. Without +one, `query` warns once and reads the per-session files. Each session is stored once. `materialize` writes the ATIF document as `trajectory.json.zst`, and `edges.jsonl` and `session_events.jsonl` compressed the same way, beside a plain `meta.json`, @@ -84,13 +112,8 @@ session from its compressed trajectory. `zstd -dc trajectory.json.zst` prints th document. Without a lake, `query --no-lake` still works, but it parses every session's trajectory on each run, so it's slow on a large corpus. -A corpus written by an earlier version also holds plain JSON files and five parquet files per -session. It keeps working as it is, and `status` counts those sessions under `layout`. To convert -it, run `atif-sql corpus slim`, which only reports what it would do and how many bytes it would -free. `atif-sql corpus slim --no-dry-run` compresses the JSON files in place, then deletes the -parquet files once `lake verify` agrees that the lake matches the compressed files. - -For one session at a time, `atif-sql convert ` converts and audits it in place. +A corpus written by an earlier version keeps working as it is; `status` counts its sessions under +`layout`, and `atif-sql corpus slim` converts it. `materialize` converts sessions across a process pool, `min(8, cpu_count)` workers by default. `--workers N` (or `ATIF_SQL_MATERIALIZE_WORKERS`) sets the size, and `--workers 1` runs the diff --git a/docs/reference/cli.md b/docs/reference/cli.md index b7cb9cb..2f9b1ad 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -99,6 +99,8 @@ Flags: - `--requires` — forwarded to the `examples` listing. `:503` - `--agent` — `claude-code` (default) or `codex`; selects which corpus the statement reads. `:628` - `--corpus-root` — override the materialized corpus root. `:504` +- `--all-corpora`: scope every view to every corpus the lake holds; `sessions.corpus` says which one a row came from. It has no per-session fallback, so it exits `78` without a usable lake and `64` together with `--no-lake`. +- `--lake` / `--no-lake`: read the lake when it holds the corpus (default), or read the per-session files. - `--format` — `table` on a TTY, a JSON array of row objects on a pipe. `:505` The statement runs against a hardened connection: reads reach the registered views and nothing else, and nothing under the corpus root is writable. Before registration the connection is sized to the host: a memory cap derived from available RAM (half of physical RAM or 8 GiB, whichever is larger, never above 80% of what's available) and a thread count of one per 2 GiB of that cap, capped at the CPUs the process may use. `ATIF_SQL_QUERY_MEMORY_LIMIT` (a DuckDB size such as `6GB`) and `ATIF_SQL_QUERY_THREADS` override both; a malformed value exits 64. The spill directory is a private `mkdtemp` (mode 0700) under the system temp dir, the only directory the sandbox grants, and it's removed when the process exits. Extension auto-install and auto-load are off, and the lance extension is loaded only when it's already installed, so registration never reaches the network (`packages/atif-cli/src/atif_cli/app.py`, `_configure_query_resources`). @@ -178,6 +180,8 @@ Flags: - `-k` / `--k` — top-k; default `10`. This is the CLI's only short flag. `:832` - `--session-id` — confine the kNN to one session. `:833` - `--corpus-root` — override the materialized corpus root. `:834` +- `--all-corpora`: search every corpus's embedding store and add a `corpus` column; exits `78` without a usable lake and `64` together with `--no-lake`. +- `--lake` / `--no-lake`: read the lake when it holds the corpus (default), or read the per-session files. - `--format` — output format. `:835` Output columns are `uuid`, `session_id`, `snippet`, and `sim` (`:927`), ranked by cosine distance ascending so the highest similarity comes first (`:935`). @@ -218,6 +222,92 @@ Flags: The answer comes from the static `VIEW_SCHEMA`, `MACRO_SIGNATURES`, `ANALYTICS_VIEW_SCHEMA`, and `ANALYTICS_MACRO_SIGNATURES` dicts with no DuckDB import and no view registration. `requires` is `core` for an object that binds on any corpus, `analytics` for one that binds once `analyze` has written its parquet, and `vss` for one that needs the embedding store. `:1744` +## `lake` + +``` +atif-sql lake COMMAND +``` + +Build, check and maintain the DuckLake every corpus is queried through. +`packages/atif-cli/src/atif_cli/lake.py:30` + +The group is attached to the root router by `app.command(lake_app)` in `packages/atif-cli/src/atif_cli/app.py`, and the lake itself lives in `atif_duck.infrastructure.lake`. The lake root defaults to `ATIF_SQL_LAKE_ROOT`, else `~/.atif-sql/lake`, and holds every corpus. All four commands are offline except the one-time `ducklake` extension install that `rebuild` makes. After a `rebuild`, every `materialize` keeps the lake current, and `query`, `analyze`, `embed` and `search` read it instead of opening each session's files. `--format` takes the same values as everywhere else: a table (or human lines) on a TTY, JSON on a pipe. + +### `lake rebuild` + +``` +atif-sql lake rebuild [OPTIONS] +``` + +Load every corpus's per-session artifacts into a fresh lake, then swap it into place. +`packages/atif-cli/src/atif_cli/lake.py:79` + +Flags: + +- `--corpus-root`: a corpus to load; repeat for more. Default: every corpus the current lake holds, plus every directory under `ATIF_SQL_CORPUS_BASE` (default `~/.atif-sql/corpus`) that holds `sessions/`. `:82` +- `--lake-root`: the lake to rebuild. `:83` +- `--format`: report format. `:84` + +The new lake is built beside the old one under the writer lock and renamed into place when complete, so a reader sees one lake or the other. It installs the `ducklake` DuckDB extension first when it is missing, which is the one network fetch the lake needs; `query` never installs it. + +Exit codes: `0` done, `64` no corpus to load, or two corpora with the same directory name. + +### `lake verify` + +``` +atif-sql lake verify [OPTIONS] +``` + +Compare every session's lake rows with its per-session artifacts. +`packages/atif-cli/src/atif_cli/lake.py:150` + +Flags: + +- `--corpus-root`: verify only this corpus; repeat for more. Default: every corpus the lake holds. `:153` +- `--lake-root`: the lake to verify. `:154` +- `--limit`: how many differences to list; all are counted. Default `20`. `:155` +- `--format`: report format. `:156` + +Per table and session it compares the row count and an order-free content hash of the lake's rows against the same read from the per-session path. The report carries `clean`, `mismatched_sessions` and the first `--limit` mismatches. + +Exit codes: `0` clean, `65` `lake_mismatch` (some session differs), `78` `lake_unavailable` (no lake, or its schema is stale). + +### `lake status` + +``` +atif-sql lake status [OPTIONS] +``` + +Report the lake: present, schema current, corpora, snapshots, files, last write. +`packages/atif-cli/src/atif_cli/lake.py:240` + +Flags: + +- `--lake-root`: the lake to report on. `:243` +- `--format`: report format. `:244` + +It reads the published reader catalog, so it never waits on a writer. A missing lake is a normal answer (`present: false`), not an error. + +### `lake compact` + +``` +atif-sql lake compact [OPTIONS] +``` + +Merge small files, expire old snapshots, and remove the files nothing references. +`packages/atif-cli/src/atif_cli/lake.py:283` + +Flags: + +- `--expire-older-than-days`: expire snapshots older than this; default `ATIF_SQL_LAKE_EXPIRE_DAYS`, else `30`. Files only expired snapshots referenced are removed once they are an hour old, so a reader in flight keeps every file it names. `:286` +- `--memory-limit`: DuckDB's memory budget for the run, as a size such as `2GiB` or `1500MB`, instead of the one derived from the host and cgroup. The writer's own 2 GiB ceiling still applies. The nightly refresh lane passes one that fits its memory scope. `:287` +- `--lake-root`: the lake to compact. `:288` +- `--format`: report format. `:289` + +The report carries the data file and snapshot counts before and after. If a merge runs out of memory nothing was published, so readers keep the previous catalog and the next run starts over. + +Exit codes: `0` done, `64` a negative `--expire-older-than-days` or a malformed `--memory-limit`, `70` runtime error (an out-of-memory merge), `78` `lake_unavailable`. + ## cron ``` diff --git a/packages/atif-cli/tests/test_readme_matrix.py b/packages/atif-cli/tests/test_readme_matrix.py new file mode 100644 index 0000000..cd15de3 --- /dev/null +++ b/packages/atif-cli/tests/test_readme_matrix.py @@ -0,0 +1,203 @@ +# SPDX-License-Identifier: Apache-2.0 + +"""The README's "What you can do" matrix names every command the CLI registers, and no other. + +The matrix is the one place a reader meets the whole command surface, so it cannot drift from +the cyclopts app. The test walks the registered command tree (the top level and the `lake`, +`corpus` and `cron` groups), parses the README table, and checks both directions: a command +with no row, and a row naming a command the app lacks, each fail by name. Each row's link must +also land on the heading of that command in `docs/reference/cli.md`. + +Anti-vacuity: a README with no matrix, and a matrix with no rows, fail rather than pass on +nothing, and the comparison itself is proved on planted drift in both directions. +""" + +from __future__ import annotations + +import re +import shlex +from pathlib import Path +from typing import NamedTuple + +import cyclopts +import pytest + +from atif_cli.app import app + +#: Repository root: `packages/atif-cli/tests/` -> three parents up. +ROOT = Path(__file__).resolve().parents[3] +README = ROOT / "README.md" +CLI_REFERENCE = ROOT / "docs" / "reference" / "cli.md" +SECTION = "## What you can do" +PROGRAM = "atif-sql" +LINK = re.compile(r"\[`([^`]+)`\]\(([^)]+)\)") + + +class Row(NamedTuple): + """One matrix row: the command path the Command cell names, and where it links.""" + + path: tuple[str, ...] + target: str + line: int + + +def subcommands(node: cyclopts.App) -> list[str]: + """The names registered under `node`, without cyclopts' own `--help` and `--version`.""" + return [name for name in node if not name.startswith("-")] + + +def registered_commands(node: cyclopts.App, prefix: tuple[str, ...] = ()) -> set[tuple[str, ...]]: + """Every runnable command path under `node`: the leaves of the command tree.""" + children = subcommands(node) + if not children: + return {prefix} + found: set[tuple[str, ...]] = set() + for name in children: + found |= registered_commands(node[name], (*prefix, name)) + return found + + +def command_path(command: str, line: int) -> tuple[str, ...]: + """The registered words a Command cell starts with, resolved against the real app. + + Words after the command (flags, placeholders, SQL) are arguments. A word that is not a + registered name where the app expects a subcommand is a command the app lacks, and so is + a group named without its subcommand. + """ + words = shlex.split(command) + if words[:1] != [PROGRAM]: + msg = f"README line {line}: `{command}` does not start with `{PROGRAM}`" + raise AssertionError(msg) + node = app + path: list[str] = [] + for word in words[1:]: + children = subcommands(node) + if not children: + break + if word not in children: + msg = f"README line {line}: `{command}` names `{word}`, which the app lacks" + raise AssertionError(msg) + node = node[word] + path.append(word) + if subcommands(node) or not path: + msg = f"README line {line}: `{command}` names a group or nothing, not a command" + raise AssertionError(msg) + return tuple(path) + + +def parse_matrix(text: str) -> list[Row]: + """The matrix rows under `## What you can do`; fails when the section or its rows are absent.""" + lines = text.splitlines() + if SECTION not in lines: + msg = f"README has no `{SECTION}` section" + raise AssertionError(msg) + start = lines.index(SECTION) + 1 + rows: list[Row] = [] + header_seen = False + for number, line in enumerate(lines[start:], start=start + 1): + if line.startswith("## "): + break + if not line.startswith("|"): + continue + cells = [cell.strip() for cell in line.strip().strip("|").split(" | ")] + if cells[0] == "Task": + if cells != ["Task", "Command", "Outcome"]: + msg = f"README line {number}: the matrix columns are {cells}" + raise AssertionError(msg) + header_seen = True + continue + if set(line) <= set("|- "): + continue + if not header_seen or len(cells) != 3: + msg = f"README line {number}: not a Task | Command | Outcome row" + raise AssertionError(msg) + link = LINK.fullmatch(cells[1]) + if link is None: + msg = f"README line {number}: the Command cell is not one linked code span" + raise AssertionError(msg) + rows.append(Row(command_path(link.group(1), number), link.group(2), number)) + if not rows: + msg = "the matrix parsed zero rows" + raise AssertionError(msg) + return rows + + +def drift(rows: list[Row], commands: set[tuple[str, ...]]) -> list[str]: + """What differs between the matrix and the command set, one line each.""" + named = {row.path for row in rows} + problems = [f"no matrix row for `{PROGRAM} {' '.join(p)}`" for p in sorted(commands - named)] + problems += [ + f"row names `{PROGRAM} {' '.join(p)}`, which the app lacks" + for p in sorted(named - commands) + ] + return problems + + +def headings(text: str) -> set[str]: + """The anchors of the Markdown headings in `text` (GitHub slugs: lowercase, spaces to dashes).""" + slugs: set[str] = set() + for line in text.splitlines(): + match = re.fullmatch(r"#{1,6} (.+)", line) + if match: + slugs.add( + re.sub(r"[^a-z0-9 -]", "", match.group(1).lower().replace("`", "")).replace( + " ", "-" + ) + ) + return slugs + + +def test_the_app_registers_the_groups_and_commands_the_matrix_walks() -> None: + commands = registered_commands(app) + assert len(commands) >= 15 + assert ("lake", "compact") in commands + assert ("corpus", "slim") in commands + assert ("cron", "install") in commands + assert ("search",) in commands + + +def test_the_matrix_names_every_registered_command_and_no_other() -> None: + rows = parse_matrix(README.read_text(encoding="utf-8")) + assert drift(rows, registered_commands(app)) == [] + + +def test_each_row_links_to_its_command_heading_in_the_cli_reference() -> None: + anchors = headings(CLI_REFERENCE.read_text(encoding="utf-8")) + for row in parse_matrix(README.read_text(encoding="utf-8")): + page, _, fragment = row.target.partition("#") + assert page == "docs/reference/cli.md", f"README line {row.line}: links to {row.target}" + assert fragment == "-".join(row.path), ( + f"README line {row.line}: `{fragment}` is not the anchor of {' '.join(row.path)}" + ) + assert fragment in anchors, ( + f"README line {row.line}: docs/reference/cli.md has no #{fragment}" + ) + + +class TestTheComparisonIsNotVacuous: + def test_no_section_fails(self) -> None: + with pytest.raises(AssertionError, match="no `## What you can do` section"): + parse_matrix("# atif-sql\n\nno matrix here\n") + + def test_zero_rows_fails(self) -> None: + with pytest.raises(AssertionError, match="zero rows"): + parse_matrix( + f"{SECTION}\n\n| Task | Command | Outcome |\n| --- | --- | --- |\n\n## Next\n" + ) + + def test_a_missing_row_is_named(self) -> None: + text = README.read_text(encoding="utf-8") + kept = [line for line in text.splitlines() if "atif-sql lake compact`" not in line] + problems = drift(parse_matrix("\n".join(kept)), registered_commands(app)) + assert problems == ["no matrix row for `atif-sql lake compact`"] + + def test_a_row_for_a_missing_command_is_named(self) -> None: + text = README.read_text(encoding="utf-8") + planted = "| Plant | [`atif-sql lake bogus`](docs/reference/cli.md#lake-bogus) | x |" + header = "| Task | Command | Outcome |\n| --- | --- | --- |" + with pytest.raises(AssertionError, match="`bogus`, which the app lacks"): + parse_matrix(text.replace(SECTION, f"{SECTION}\n\n{header}\n{planted}\n", 1)) + + def test_a_group_without_its_subcommand_fails(self) -> None: + with pytest.raises(AssertionError, match="names a group or nothing"): + command_path("atif-sql lake", 1)