Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .vale-baseline.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
{
"Google.Colons": 9,
"Google.EmDash": 448,
"Google.FirstPerson": 3,
"Google.Headings": 63,
"Google.OxfordComma": 28,
Expand Down
13 changes: 7 additions & 6 deletions .vale.ini
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,13 @@ Vale.Terms = NO
# Off: American technical English spells facade without the cedilla.
proselint.Diacritical = NO

# Warning: the docs set an em dash with a space on each side throughout; the rule checks only
# that spacing, and its hundreds of hits are one typographic choice, not hundreds of defects.
# The baseline holds the count, so a new page adds none. Vale can report two of these hits as
# one when a wrapped list item's dash meets a later one (scripts/vale_gate.py says when), so a
# count that falls on a change adding a dash is not one to lower the baseline to.
Google.EmDash = warning
# Error: the docs set no dash with a space on each side (` β€” `, ` – `, a line break counting as
# a space); a colon, comma, period or parentheses carries the sense instead, and one spaced dash
# fails the gate. It sits at error rather than in the warning ratchet because Vale can report two
# of its hits as one when a wrapped list item's dash meets a later one (scripts/vale_gate.py says
# when), so a count could fall on a change that adds a dash. At error a merge folds two alerts
# into one, never into none, so an added dash is red either way.
Google.EmDash = error
# Warning: a quoted literal (a status, a rule's text, a field value) keeps the punctuation that
# is not part of it outside the quotes, as the docs quote them verbatim.
Google.Quotes = warning
Expand Down
3 changes: 3 additions & 0 deletions .vale/fixtures/planted-warning.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# A planted warning

atif-sql converts Claude Code transcripts to ATIF, and the converter will write one trajectory.
7 changes: 7 additions & 0 deletions .vale/fixtures/wrapped-emdash.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# A wrapped em dash

- [Sequences](sequences.md) β€” call order for the top processes, in a list item that wraps
onto a second line.
- The next item.

A later paragraph with no dash.
36 changes: 18 additions & 18 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# atif-sql β€” operating manual
# atif-sql: operating manual

ATIF-native analytics over agent trajectories from Claude Code
(`~/.claude/projects/**/*.jsonl`) and Codex CLI
Expand All @@ -12,7 +12,7 @@ documents, and layers DuckDB views on top. One corpus holds one agent, and

uv WORKSPACE (virtual root, members under `packages/*`):

- `packages/atif-converter` β€” converts Claude Code and Codex transcripts to
- `packages/atif-converter`: converts Claude Code and Codex transcripts to
ATIF (ported from harbor's converters, on the ATIF models vendored in
`atif_converter.domain.atif`); owns the fidelity policy per agent (the known Claude Code
gaps in `atif_converter.domain.fidelity`, the Codex ones in
Expand All @@ -23,7 +23,7 @@ uv WORKSPACE (virtual root, members under `packages/*`):
`domain.pricing` prices each step (`total_cost_usd`, Codex `cost_usd`) from
the vendored `domain/model_prices.json` with litellm's arithmetic; a model
it can't price is NULL. Layered: `application` > `infrastructure` > `domain`.
- `packages/atif-corpus` β€” corpus materialization: discovery, watermarks,
- `packages/atif-corpus`: corpus materialization, covering discovery, watermarks,
quiescence, atomic artifact writes (the bulk JSON artifacts zstd-compressed),
in-place compression of an old-layout session for `corpus slim`, the
`ArtifactProducer` port that lets a composition root add per-session files
Expand All @@ -37,7 +37,7 @@ uv WORKSPACE (virtual root, members under `packages/*`):
before it becomes a corpus path (rejected ids are reported, never
written); atif-duck carries the twin. Layered: `application` >
`infrastructure` > `domain`.
- `packages/atif-duck` β€” DuckDB views + macros over the materialized corpus:
- `packages/atif-duck`: DuckDB views + macros over the materialized corpus:
core views and macros plus the analytics views and macros, all declared in
a static drift-tested catalog. Who wrote a user
step is decided ONCE, in `domain.authorship` (a prefix rule table rendered
Expand All @@ -52,12 +52,12 @@ uv WORKSPACE (virtual root, members under `packages/*`):
DuckLake every corpus is queried through (`domain.lake` declares the tables,
`infrastructure.lake` writes, reads, verifies and compacts it; see
"Storage" below). Layered: `infrastructure` > `domain`.
- `packages/atif-models` β€” model alias registry + structured-output LLM
- `packages/atif-models`: model alias registry + structured-output LLM
client. No other package hardcodes a Bedrock model id. Layered:
`infrastructure` > `domain`.
- `packages/atif-analytics` β€” the LLM pipelines that spend money at
- `packages/atif-analytics`: the LLM pipelines that spend money at
Bedrock and are checkpointed per session (classify, conflicts, friction,
perceived β€” see `PIPELINE_NAMES` in
perceived; see `PIPELINE_NAMES` in
`infrastructure/sqlite_state/checkpointer.py`). They read human turns only
(`domain.authorship`), and classify and conflicts skip automated review
(`turn_audit`) and one-shot-job sessions. Session data comes through the
Expand All @@ -69,12 +69,12 @@ uv WORKSPACE (virtual root, members under `packages/*`):
cluster/terms/community pipelines were cut on 2026-09-27; outcome is the
deterministic `session_outcomes` view now. Layered: `application` >
`infrastructure` > `domain`.
- `packages/atif-embed` β€” Cohere Embed v4 on Bedrock + LanceDB store + the
- `packages/atif-embed`: Cohere Embed v4 on Bedrock + LanceDB store + the
backfill and orphan-prune use cases. Discovery reads the lake's `steps`
through the `LakeStepsPort` port (atif-cli implements it over atif-duck's
`infrastructure.lake_steps`), and falls back to the per-session reader.
Layered: `application` > `infrastructure` > `domain`.
- `packages/atif-cli` β€” cyclopts CLI composing the rest into commands
- `packages/atif-cli`: cyclopts CLI composing the rest into commands
(`convert`, `materialize`, `status`, `query`, `analyze`, `embed`,
`search`, `examples`, `schema`, `cron`, `lake`, `corpus`).

Expand All @@ -83,7 +83,7 @@ Rules of the road:
- Independence contract (import-linter, `pyproject.toml`): converter /
corpus / duck / models / embed may NEVER import each other. atif-cli is
the composition root and may import them all. atif-analytics is the one
exception β€” it may import atif-models and nothing else among our
exception: it may import atif-models and nothing else among our
packages (a `forbidden` contract pins its other edges shut).
- Inter-package deps: declare in the member's `[project.dependencies]` AND
`[tool.uv.sources] <pkg> = { workspace = true }`.
Expand Down Expand Up @@ -372,25 +372,25 @@ copy of the queryable rows:

For an LLM agent driving `atif-sql`, the discovery loop is this:

1. `atif-sql schema` β€” every view (with columns) and macro signature, core
1. `atif-sql schema`: every view (with columns) and macro signature, core
and analytics, each with what it `requires`, from the static catalog in
<50 ms. Its output ends with an `examples_hint`.
2. `atif-sql examples` (alias: `atif-sql query --examples`) β€” runnable
2. `atif-sql examples` (alias: `atif-sql query --examples`): runnable
example queries for every view and macro, DERIVED from the catalog (never
hardcoded per object) and each one EXECUTED by
`packages/atif-duck/tests/test_examples.py` against a fixture corpus β€”
`packages/atif-duck/tests/test_examples.py` against a fixture corpus, and
the header's "test-executed against this version" is literal. Piped
output is JSON `{note, examples: [{name, sql, description, requires,
category}]}`; filter with `--requires core|analytics|vss` and
`--category view|table-macro|scalar-macro`.
3. `atif-sql query '<sql>'` β€” run it. `--agent codex` points it at the Codex
3. `atif-sql query '<sql>'`: run it. `--agent codex` points it at the Codex
corpus without spelling the path, and `--all-corpora` runs it over every
corpus the lake holds (`sessions.corpus` names each row's corpus). Copy an example verbatim (the `sid`
exemplar is a subquery over `sessions`, so it works on any corpus) or
adapt it. `requires: analytics` needs `atif-sql analyze` to have run;
`requires: vss` needs `atif-sql embed --all --no-dry-run` (a bare `embed`
exits 64: a real run needs an explicit scope). For semantic search over TEXT,
prefer `atif-sql search 'query'` β€” it embeds the text first, then runs
prefer `atif-sql search 'query'`, which embeds the text first, then runs
the same `semantic_search` kNN.

What the query sandbox will and won't do: `SELECT`, `EXPLAIN`, `SET
Expand All @@ -407,14 +407,14 @@ For an LLM agent driving `atif-sql`, the discovery loop is this:
Adding a view/macro? The drift tests force: a `DESCRIPTIONS` entry in
`atif_duck/domain/catalog.py`, an `ARG_EXEMPLARS` entry for any new
parameter name, `TABLE_MACRO_NAMES` membership if the DDL is `AS TABLE`, and
the derived example must execute β€” or a documented `EXCLUSIONS` entry.
the derived example must execute (or a documented `EXCLUSIONS` entry).

## Definition of done

`mise run check` fully green (lint + fmt + typecheck + lint:imports + lint:workflows +
lint:hooks + lint:pnpm-lock + security:vex:check + docs:prose + test). `docs:prose` is Vale over
the published prose (docs/, site/authored/, README.md, AGENTS.md, CONTRIBUTING.md) at error level,
with a file floor per glob so a shrunken scope is red, and a per-rule warning ratchet against `.vale-baseline.json`: reword the sentence it flags, or add a real term to `.vale/styles/config/vocabularies/atif-sql/accept.txt`. When a count falls, run `mise run docs:prose:baseline` and commit the lower file; never raise it. `mise run security` is the report-only tier beside it: findings do not fail
with a file floor per glob so a shrunken scope is red, and a per-rule warning ratchet against `.vale-baseline.json`: reword the sentence it flags, or add a real term to `.vale/styles/config/vocabularies/atif-sql/accept.txt`. When a count falls, run `mise run docs:prose:baseline` and commit the lower file; never raise it. A spaced dash is an error (`Google.EmDash`), not a warning: write a colon, comma, period, or parentheses instead. `mise run security` is the report-only tier beside it: findings do not fail
it, a scanner that produced no usable SARIF does.
A change under docs/ or site/ also needs `mise run docs:gate`: the site build, then `docs:links`,
which crawls site/dist as GitHub Pages serves it and fails on any internal 404. Link page to page by
Expand All @@ -426,7 +426,7 @@ First-time setup: `mise trust && mise install && mise run install`.

`experiments/` holds numbered experiment protocols (README per experiment).
Outputs go to `experiments/**/out/` (gitignored). No experiment is wired
into a gate β€” `mise run check` and pytest's `testpaths` both skip the
into a gate: `mise run check` and pytest's `testpaths` both skip the
directory.

Migration parity oracles are not part of this repo. Parity is not a standing
Expand Down
28 changes: 16 additions & 12 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ atif-sql is a uv workspace whose root is also the one published distribution:
which bundles every module tree) and `[tool.uv.workspace] members =
["packages/*"]`. The packages under `packages/` are development members,
not install targets. Everything below is read off the
checked-in config β€” `mise.toml`, `pyproject.toml`, `lefthook.yml`, and
`AGENTS.md` β€” so if a claim here and a config file disagree, the config file
checked-in config (`mise.toml`, `pyproject.toml`, `lefthook.yml`, and
`AGENTS.md`), so if a claim here and a config file disagree, the config file
wins and this document is the bug.

## First-time setup
Expand All @@ -28,7 +28,7 @@ settles on 3.14.
Drive every command through `mise run <task>` rather than calling `uv`, `ruff`,
or `ty` directly. The mise `[env]` block sets `PYTHONDONTWRITEBYTECODE=1` and
`UV_LINK_MODE=copy`, and `[settings] python.uv_venv_auto = "create|source"`
creates and sources the workspace venv β€” a bare tool invocation can miss all of
creates and sources the workspace venv; a bare tool invocation can miss all of
that.

## Definition of done: `mise run check`
Expand Down Expand Up @@ -56,6 +56,10 @@ prints `lower the baseline: mise run docs:prose:baseline`: run that task and com
the change. A baseline only goes down in review, so a diff that raises a count in it needs the
same scrutiny as a widened `ignore` list.

`Google.EmDash` is an error, not a ratchet: a dash with a space on each side (a line break counts as
a space) fails `docs:prose` on its own. Write a colon, comma, period, or parentheses in its place,
whichever keeps the sentence's sense; `.vale.ini` says why the rule sits at error.

## The docs site: `mise run docs:gate`

A change under `docs/` or `site/` also needs `mise run docs:install && mise run docs:gate`, which
Expand All @@ -74,13 +78,13 @@ internal link that does not answer 200.

Never reach green by relaxing a gate. Widening an `ignore` list, adding a
blanket per-file ignore, or deleting a contract is a change to the project's
standards, not a fix to your branch β€” raise it in the pull request instead.
standards, not a fix to your branch. Raise it in the pull request instead.

## Commits

Commit messages must be [Conventional Commits](https://www.conventionalcommits.org):
`type(scope): subject` with `type` one of `feat|fix|chore|docs|refactor|test|perf|ci`.
This is enforced, not advisory β€” `lefthook.yml`'s `commit-msg` hook runs
This is enforced, not advisory: `lefthook.yml`'s `commit-msg` hook runs
`uv run cz check --allow-abort --commit-msg-file {1}`, configured by
`[tool.commitizen]` in
`pyproject.toml` (`name = "cz_conventional_commits"`, with
Expand Down Expand Up @@ -109,7 +113,7 @@ root packages. These shape the whole workspace:

1. **Independence.** `atif_converter`, `atif_corpus`, `atif_duck`,
`atif_models`, and `atif_embed` may never import each other. If you need
two of them in one flow, compose them in `atif_cli` β€” it is the only
two of them in one flow, compose them in `atif_cli`; it is the only
composition root and may import them all.
2. **`atif_analytics` composes `atif_models` and nothing else of ours.** A
`forbidden` contract pins its remaining edges shut: it may not import
Expand All @@ -120,7 +124,7 @@ root packages. These shape the whole workspace:
The rest are `layers` contracts:
`application > infrastructure > domain` for `atif_converter`, `atif_corpus`,
`atif_embed`, and `atif_analytics`, and `infrastructure > domain` for
`atif_models`. Dependencies point inward β€” a `domain` module importing from
`atif_models`. Dependencies point inward: a `domain` module importing from
`infrastructure` fails `lint:imports`.

Other rules live in ruff rather than import-linter:
Expand All @@ -140,7 +144,7 @@ failure.

**A dependency on a sibling package.** Declare it in the member's
`[project.dependencies]` *and* add `[tool.uv.sources] <pkg> = { workspace = true }`
β€” both, per `AGENTS.md`. The independence contract above still applies.
(both, per `AGENTS.md`). The independence contract above still applies.

**A new package.** `packages/*` is globbed as a workspace member, but these
places in the root `pyproject.toml` list packages explicitly and will not pick
Expand All @@ -153,14 +157,14 @@ The new member also needs the license wiring every existing member has:
a `LICENSE` symlink to the repo root (`ln -s ../../LICENSE
packages/<name>/LICENSE`). `license-files` globs are resolved inside the
member's own directory and `..` is rejected, so the symlink is what puts the
license text into the wheel's `dist-info/licenses/` β€” Apache-2.0 Β§4(a) requires
license text into the wheel's `dist-info/licenses/`; Apache-2.0 Β§4(a) requires
it, and `uv build` fails outright if the file is missing.

**A DuckDB view or macro.** The catalog is static and drift-tested. Per
`AGENTS.md`, adding an object means adding a `DESCRIPTIONS` entry in
`atif_duck/domain/catalog.py`, an `ARG_EXEMPLARS` entry for any new parameter
name, `TABLE_MACRO_NAMES` membership if the DDL is `AS TABLE`, and a derived
example that actually executes β€” or a documented `EXCLUSIONS` entry. The tests
example that actually executes (or a documented `EXCLUSIONS` entry). The tests
in `packages/atif-duck/tests/` fail until that is true.

## harbor and litellm are test oracles, not dependencies
Expand Down Expand Up @@ -231,7 +235,7 @@ pipelines) and `atif-sql embed` (Cohere Embed v4). Each is guarded, and the
guards are part of the contract:

- `analyze` is dry-run by default; `--no-dry-run` is what makes it spend.
- `embed` refuses a real run with no scope β€” no `--limit` and no `--all` exits 64.
- `embed` refuses a real run with no scope: no `--limit` and no `--all` exits 64.

The test suite must stay offline: fake the port, never the credential. If a
change makes a Bedrock call reachable from `pytest`, that is a defect in the
Expand All @@ -240,7 +244,7 @@ change.
## Experiments

`experiments/` holds numbered protocols with a README (and, once run, a REPORT)
per experiment. Nothing there is wired into a gate β€” pytest's `testpaths` only
per experiment. Nothing there is wired into a gate: pytest's `testpaths` only
collects `packages/*/tests`, and outputs are written to `experiments/**/out/`,
which `.gitignore` excludes. An experiment's recorded numbers are measurements;
if you re-run one, add your measurement rather than editing the one already recorded.
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,15 @@ Claude Code sessions (`~/.claude/projects/**/*.jsonl`) and Codex CLI rollouts
(`~/.codex/sessions/**/rollout-*.jsonl`) are converted to ATIF (Harbor's Agent
Trajectory Interchange Format, see the Harbor ATIF RFC: RFC-0001 in
<https://github.com/laude-institute/harbor>), materialized as a corpus, and
queried through DuckDB views. Converting once at the boundary β€” with an
explicit, tested fidelity policy for what upstream drops β€” beats re-deriving
queried through DuckDB views. Converting once at the boundary (with an
explicit, tested fidelity policy for what upstream drops) beats re-deriving
trajectory semantics inside every SQL view. Both agents land in the same views,
and `sessions.agent` says which one a row came from.

## Install

One command, one package, every capability. Conversion, materialization, the DuckDB surface,
the LLM analytics pipelines, and semantic search are all in the box β€” there are no extras to
the LLM analytics pipelines, and semantic search are all in the box; there are no extras to
choose and nothing to install afterwards to make a command work.

```bash
Expand Down Expand Up @@ -140,7 +140,7 @@ pick which, or `--agent codex` to get the Codex default. `--all-corpora` reads
every corpus the lake holds at once, and `sessions.corpus` says which one a
row came from.

Working on `atif-sql` itself is a different setup β€” a clone, `mise`, and `mise run check` as the
Working on `atif-sql` itself is a different setup: a clone, `mise`, and `mise run check` as the
definition of done. `CONTRIBUTING.md` has it.

## Agent workflow: schema β†’ examples β†’ query
Expand All @@ -152,7 +152,7 @@ atif-sql query 'SELECT * FROM tool_rank(30) LIMIT 10'
```

`atif-sql examples` (or `atif-sql query --examples`) emits runnable queries
derived from the catalog β€” not hardcoded strings β€” and every one is executed
derived from the catalog, not hardcoded strings, and every one is executed
by the test suite against a fixture corpus, so the listing cannot rot.
Piped output is JSON; filter with `--requires core|analytics|vss` and
`--category view|table-macro|scalar-macro`.
Expand All @@ -162,7 +162,7 @@ Piped output is JSON; filter with `--requires core|analytics|vss` and
Setup, the gates `mise run check` runs, the import-linter contracts you will trip, and the
Conventional Commit rule the `commit-msg` hook enforces: **[CONTRIBUTING.md](CONTRIBUTING.md)**.

Releases are cut by commitizen and published to PyPI over OIDC Trusted Publishing β€”
Releases are cut by commitizen and published to PyPI over OIDC Trusted Publishing;
**[RELEASING.md](RELEASING.md)** covers the flow, the versioning model, and the install-weight
measurements.

Expand Down
Loading
Loading