Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,18 @@ The worktree shares git history with the main repo but has its own `.venv/`. Alw
- Use `database_name` instead of `database`, `schema_name` instead of `schema`, `table_name` instead of `table`
- Address git merge conflicts by pulling main, resolving conflicts, and pushing. Favour simplicity over clean commit history β€” PRs are squash-merged anyway.

## Backwards compatibility

The major version is 0. While the major version stays 0, clair does not keep backwards
compatibility. clair has no users at this time, so the best design wins against a stable
interface.

- Change a public name, a file format, or a function signature when the change makes the
system better.
- Do not add a deprecation shim, an alias for an old name, or a migration path.
- Delete the old code path. Do not keep it beside the new one.
- Name each behaviour change in the pull request description.

## Keep site_docs/ up to date

`site_docs/docs/` is the source of truth for behaviour. Before you complete a change, compare
Expand Down
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,10 @@ uv add rivage-clair

### 1. Set up an environment

Run `clair init` β€” it will prompt for your Snowflake connection details and write `~/.clair/environments.yml`.
Run `clair init`. It asks for your Snowflake connection details and writes two files:

- `~/.clair/environments.yml` β€” your connection settings. Do not commit this file.
- `<project>/__routing__.py` β€” the [routing](https://clair.rivage.sh/guides/routing/) entry of each environment. Commit this file.

### 2. Create a project

Expand Down
2 changes: 1 addition & 1 deletion site_docs/docs/cli/compile.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ _clairtifacts/
| Flag | Default | Description |
|------|---------|-------------|
| `--project` | `.` | Path to the clair project root |
| `--env` | optional | Environment name. Necessary if clair must apply routing to the generated SQL. |
| `--env` | optional | Environment name. It selects the entry in `__routing__.py` that clair applies to the generated SQL. |
| `--select` | all | Glob pattern that filters the Trouves. Repeat the flag to add more patterns. |
| `--run-mode` | `full_refresh` | `full_refresh` or `incremental`. Selects which SQL variant clair generates. |

Expand Down
4 changes: 3 additions & 1 deletion site_docs/docs/cli/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
All commands share two common flags:

- `--project` β€” path to the clair project root (default: `.`)
- `--env` β€” environment name from `~/.clair/environments.yml` (default: `CLAIR_ENV` or `dev`)
- `--env` β€” environment name. It names a key in `~/.clair/environments.yml` and an entry in `__routing__.py` (default: `CLAIR_ENV` or `dev`)

## Commands

Expand All @@ -13,6 +13,7 @@ All commands share two common flags:
| [`clair compile`](compile.md) | Resolve DAG and write SQL to `_clairtifacts/` | Optional (for routing) |
| [`clair run`](run.md) | Run Trouves against Snowflake in dependency order | **Yes** |
| [`clair test`](test.md) | Run data quality tests against Snowflake | **Yes** |
| [`clair validate`](validate.md) | Apply the routing entries to every Trouve | No |
| [`clair dag`](dag.md) | Print the dependency graph as an indented tree | No |
| [`clair docs`](docs.md) | Start a local web UI for the DAG and the documentation | No |
| [`clair clean`](clean.md) | Remove compiled artifacts from `_clairtifacts/` | No |
Expand All @@ -22,6 +23,7 @@ All commands share two common flags:
| Variable | Default | Description |
|----------|---------|-------------|
| `CLAIR_ENV` | `dev` | Active environment name. This is the same as `--env` on every command. `--env` wins if you set both. |
| `CLAIR_USER` | β€” | The `clair init` routing template reads this name. Give each variable that a routing entry reads the `CLAIR_` prefix. |
| `CLAIR_LOG_FORMAT` | _(text)_ | Set to `json` to write structured JSON logs. Use this in CI/CD pipelines and in container environments that read JSON logs. |

## Help
Expand Down
73 changes: 73 additions & 0 deletions site_docs/docs/cli/validate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# `clair validate`

Apply the routing entries in `__routing__.py` to every Trouve in the project, and report every problem at once.

This command needs no Snowflake credentials, so CI runs it on every change.

```bash
clair validate
clair validate --env prod
clair validate --project path/to/project
```

## Options

| Option | Default | Description |
|--------|---------|-------------|
| `--project` | `.` | Path to the clair project root. |
| `--env` | `CLAIR_ENV` or `dev` | The environment to route for. It matches an entry in `__routing__.py`. |

## What it examines

- The routing file runs, and it gives a `RoutingTable`.
- The table has one entry for each environment name.
- Every directory name and file name makes a valid address.
- The entry for this environment runs on every TABLE and VIEW Trouve.
- Every address that the entry gives is a valid Snowflake identifier.
- No two Trouves go to one physical target.

## Output

A project with no problems:

```
environment: dev
routing file: /home/alice/project/__routing__.py
entry: DeveloperRouting(environment_name='dev', user_variable='CLAIR_USER')
Trouves to route: 12

βœ“ Every routed name is valid. No collisions.
```

A project with a problem gives exit code 1:

```
environment: dev
routing file: /home/alice/project/__routing__.py
entry: DeveloperRouting(environment_name='dev', user_variable='CLAIR_USER')
Trouves to route: 12

βœ— analytics.finance.revenue
The routing entry `DeveloperRouting(environment_name='dev', user_variable='CLAIR_USER')` failed on 'analytics.finance.revenue': KeyError: 'CLAIR_USER'

1 problem found.
```

`clair compile` and `clair run` stop at the first routing problem, because they must not write to a wrong target. `clair validate` instead reports every problem, so that you correct them together.

## Exit codes

| Code | Meaning |
|------|---------|
| 0 | Every routed name is valid, and no two Trouves collide. |
| 1 | clair found one problem or more. |

## In CI

```yaml
- name: Validate clair routing
run: |
uv run clair validate --env prod
```

See the [routing guide](../guides/routing.md) for the entry types and how to write a `route` method.
7 changes: 4 additions & 3 deletions site_docs/docs/concepts/environments.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ prod:
| `role` | β€” | Default role. If you omit it, Snowflake uses the default role of the user. |
| `region` | β€” | AWS/Azure region. clair needs it for the query URLs in the logs. |
| `account_locator` | β€” | Classic account locator. clair needs it for the query URLs. |
| `routing` | β€” | Routing policy. See [Routing Policies](../guides/routing.md). |

## Select an environment

Expand Down Expand Up @@ -106,6 +105,8 @@ In CI, set `CLAIR_ENV` and use key-pair authentication. This method does not nee
run: clair run --project .
```

## Routing policies
## Routing

Each environment can include a routing policy. The policy remaps logical Snowflake names to physical targets. See [Routing Policies](../guides/routing.md).
An environment holds connection settings only. The routing rules are in `__routing__.py`, at the root of your project. clair joins the two files by the environment name.

An unknown key in an environment block is an error. A `routing:` block from an older version of clair therefore stops the run, and the message tells you to move the rule. See [Routing](../guides/routing.md).
2 changes: 1 addition & 1 deletion site_docs/docs/concepts/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ Read these four concepts before you use clair.
- **[Trouve](trouve.md)** β€” the basic unit. One Python file, one Snowflake object.
- **[DAG](dag.md)** β€” the dependency graph. clair builds it from the Python imports.
- **[Project Layout](project-layout.md)** β€” how the directory structure maps to Snowflake names.
- **[Environments](environments.md)** β€” Snowflake connection profiles and routing policies.
- **[Environments](environments.md)** β€” Snowflake connection profiles.
5 changes: 4 additions & 1 deletion site_docs/docs/concepts/project-layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ A clair project in production usually has 3 or 4 layers:

```
my_project/
β”œβ”€β”€ __routing__.py # The routing entry of each environment
β”‚
β”œβ”€β”€ source/ # Tables that already exist β€” TrouveType.SOURCE
β”‚ β”œβ”€β”€ orders/
β”‚ β”‚ β”œβ”€β”€ raw.py # source.orders.raw
Expand Down Expand Up @@ -46,6 +48,7 @@ my_project/

| File | Location | Purpose |
|------|----------|---------|
| `__routing__.py` | project root | The [routing](../guides/routing.md) entry of each environment |
| `__database_config__.py` | database directory | Warehouse/role defaults for all Trouves in that database |
| `__schema_config__.py` | schema directory | Warehouse/role defaults for all Trouves in that schema |

Expand Down Expand Up @@ -81,4 +84,4 @@ Python imports work in the usual way. A Trouve in `refined/` can import from `so
from source.orders.raw import trouve as raw_orders
```

clair resolves `source.orders.raw` to the Snowflake object at `source.orders.raw`. An active [routing policy](../guides/routing.md) can change this target.
clair resolves `source.orders.raw` to the Snowflake object at `source.orders.raw`. An active [routing entry](../guides/routing.md) can change this target.
2 changes: 1 addition & 1 deletion site_docs/docs/guides/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ These guides show you how to do the usual clair tasks.
- **[Incrementality](incrementality.md)** β€” APPEND and UPSERT strategies for large tables
- **[Data Quality Tests](data-quality-tests.md)** β€” attach tests to Trouves
- **[Selectors](selectors.md)** β€” run only a subset of your project
- **[Routing Policies](routing.md)** β€” remap Snowflake targets per environment
- **[Routing](routing.md)** β€” remap the Snowflake target of each environment
- **[Per-Database & Schema Config](per-database-schema-config.md)** β€” warehouse and role overrides per directory
Loading
Loading