Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions CAPABILITIES.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,18 +51,18 @@ capabilities:
limitations: [Learn writes signals only and never expands authorization.]
- id: autonomy-policy
status: implemented
description: Evidence, authorization, reversibility, impact, risk, and reliability gates.
modules: [sqlgraph/autonomy]
tests: [tests/test_autonomy, tests/safety/test_autonomy_gates.py]
description: Evidence, verified grants, adapter-derived reversibility, impact, risk, and reliability gates.
modules: [sqlgraph/autonomy, sqlgraph/reasoning]
tests: [tests/test_autonomy, tests/test_reasoning/test_runner.py, tests/safety/test_autonomy_gates.py]
examples: [examples/book_cases/irreversible_drop]
limitations: [Thresholds are reference defaults and must be calibrated per deployment.]
limitations: [The bundled static grant registry is a reference implementation; production deployments must inject their own authorization verifier.]
- id: reversible-actions
status: implemented
description: Idempotent file and DuckDB actions with dry run, circuit breaker, and verified rollback.
description: Idempotent file and DuckDB actions with isolated reversibility probes, dry run, circuit breaker, and verified rollback.
modules: [sqlgraph/actions]
tests: [tests/test_actions, tests/safety/test_execution_recovery.py]
examples: [examples/book_cases/cold_table_retirement]
limitations: [Only local SQL files and DuckDB are included adapters.]
limitations: [Only local SQL files and DuckDB are included; idempotency and rollback state are local-process guarantees.]
- id: three-layer-verification
status: implemented
description: Actual code comparison, independent graph rebuild, and runtime checks.
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ cd SqlGraph
pip install -e ".[all]"
```

SqlGraph targets **Python 3.9 through 3.12** (CI runs the same version matrix).
SqlGraph targets **Python 3.10 through 3.12** (CI runs the same version matrix).

## Running tests

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# SqlGraph

[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.9--3.12-blue.svg)](https://www.python.org/)
[![Python](https://img.shields.io/badge/python-3.10--3.12-blue.svg)](https://www.python.org/)
[![CI](https://github.com/Liyurun/SqlGraph/actions/workflows/ci.yml/badge.svg)](https://github.com/Liyurun/SqlGraph/actions/workflows/ci.yml)
[![SQLGlot](https://img.shields.io/badge/parser-SQLGlot-green.svg)](https://github.com/tobymao/sqlglot)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
Expand Down
34 changes: 20 additions & 14 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,28 +26,31 @@ clean boundary, so you can swap or extend any layer independently.
- **`ColumnResolver`** binds every `exp.Column` to a physical `table.column`,
using the query's source tables (CTE-inclusive) and the schema registry to
disambiguate. Unresolvable columns fall back to `UNKNOWN.col`.
- **`expr_dag.decompose()`** turns each non-passthrough output expression into a
single **fingerprinted logic node**:
- **`expr_dag.decompose()`** preserves each non-passthrough output expression as
a stable root node, while **`decompose_operands()`** records nested operators
as a child-to-parent operand DAG:
- all column references inside the expression are replaced with their resolved
physical-column strings;
- SQLGlot serializes the rewritten expression with `normalize=True`, giving a
conservative canonical SQL string without algebraic equivalence inference;
- that canonical SQL string is hashed (SHA1, 128-bit) into the fingerprint;
- every physical column the expression reads is recorded as a dependency.
- the canonical SQL string produces a conservative 128-bit logic fingerprint;
- every physical column the expression reads is recorded as a dependency;
- nested functions and arithmetic operators are connected with
`expr_operand` edges without changing the root fingerprint.

This fingerprint identifies identical physical-column-bound expression shapes
across different SQL files. It intentionally does not treat `a + b` and
`b + a` as equivalent. The builder combines the fingerprint with the output
field name to decide Transform identity, so identical logic only collapses
when it produces the same downstream field. A whole composite expression such
as `ROUND(SUM(clicks)/COUNT(*), 4)` is **one** node — it is not decomposed into
sub-nodes.
when it produces the same downstream field. A composite expression such as
`ROUND(SUM(clicks)/COUNT(*), 4)` therefore has one stable output root plus
inspectable nested operand nodes.

## 3. Builder (`sqlgraph/builder`)

- **`GraphBuilder`** materializes the graph from parse results:
- tables and columns get **deterministic IDs** (96-bit SHA1 of their content
key) so the same entity is stable across files and runs;
- graph nodes and edges use deterministic `id-v2` identities derived from
SHA256 semantic keys, so the same entity is stable across files and runs;
- transformation nodes are deduplicated by `expression fingerprint + output
field name` (`_ensure_expr_node`);
- for each output column it adds `contains` (SQL→expr),
Expand All @@ -74,6 +77,7 @@ An in-memory **`PropertyGraph`** of typed nodes and edges.
| `has_column` | table → its column |
| `contains` | SQL → a transformation it defines |
| `compute_dependency` | physical column → transformation that consumes it |
| `expr_operand` | nested transformation → parent transformation |
| `produces` | transformation → output column |
| `table_lineage` | source table → target table (cross-SQL) |

Expand All @@ -89,11 +93,12 @@ An in-memory **`PropertyGraph`** of typed nodes and edges.

## Design principles

- **Determinism** — same SQL ⇒ same graph ⇒ same node IDs. Diffs are meaningful.
- **Determinism** — same SQL ⇒ same graph ⇒ the same SHA256-derived `id-v2`
node and edge IDs. Diffs are meaningful.
- **Physical precision** — node identity binds to resolved physical columns, not
text, so lookalike expressions on different columns stay distinct.
- **Scale** — 96/128-bit fingerprints keep collisions negligible even for
warehouses with millions of columns.
- **Scale** — 96/128-bit truncated identities keep collisions negligible for
the reference warehouse scale, with collision detection during construction.

## Governance reference pipeline

Expand All @@ -108,8 +113,9 @@ Baseline -> HeteroGraph/TableGraph -> Evidence/Grounding -> Autonomy
- `baseline` identifies the effective input and discloses missing dependencies.
- `evidence` collects an intent-bound, versioned subgraph with coverage duties.
- `graphrag` rejects assertions with missing, stale, or out-of-scope citations.
- `autonomy` applies evidence, authorization, and reversibility gates before scoring.
- `actions` provides idempotent file and DuckDB adapters with verified rollback.
- `autonomy` accepts authorization only from an injected verifier before scoring.
- `actions` derives reversibility evidence from isolated file and DuckDB probes,
then provides idempotent execution with verified rollback.
- `verification` compares actual code, rebuilds structure, and checks runtime effects.
- `audit` appends hash-chained events; `reasoning` orchestrates the seven steps.

Expand Down
8 changes: 8 additions & 0 deletions docs/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@
must be supplied through adapters.
- The included action adapters support local SQL files and DuckDB only. They are
reference implementations, not production warehouse credentials or schedulers.
- The bundled static authorization registry is for examples only. Production
deployments must inject an IAM- or approval-backed verifier.
- Idempotency caches, circuit-breaker state, snapshots, and rollback guarantees
are local to one process and filesystem; they are not distributed transactions.
- Adapter reversibility probes validate isolated local copies. They do not prove
that arbitrary external systems or side effects can be restored.
- JSONL audit storage detects local mutation, deletion, and reordering but is not
a distributed consensus ledger.
- Runtime verification trusts the configured runtime adapter's observations.
Expand All @@ -20,3 +26,5 @@
and 10,000 statement benchmarks run in a separate workflow.
- SqlGraph does not bundle an LLM, approval platform, identity provider, metadata
catalog, or production scheduler.
- `sqlgraph.agent.GovernanceLoop` is a deprecated v0.1 compatibility facade.
New integrations should use `sqlgraph.reasoning.GovernanceRunner`.
Loading
Loading