feat: add AI-native warehouse governance reference implementation

.github/workflows/ci.yml

@@ -2,17 +2,18 @@ name: CI
 
 on:
   push:
-    branches: [main]
+    branches: [main, "feat/**"]
   pull_request:
     branches: [main]
+  workflow_dispatch:
 
 jobs:
   test:
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.9", "3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12"]
 
     steps:
       - name: Checkout
@@ -26,10 +27,14 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          python -m pip install -e ".[all]" pytest pytest-cov
+          python -m pip install -e . networkx pytest pytest-cov ruff mypy build
 
       - name: Run open-source guard
         run: python scripts/opensource_guard.py
 
       - name: Run tests
         run: pytest -q
+
+      - name: Run release gates
+        if: matrix.python-version == '3.12'
+        run: PYTHON=python bash tools/ci_checks.sh

.github/workflows/scale.yml

@@ -0,0 +1,22 @@
+name: Scale Benchmarks
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 3 * * 1"
+
+jobs:
+  benchmark:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: python -m pip install --upgrade pip
+      - run: python -m pip install -e .
+      - run: python tools/benchmark_scale.py --counts 1000,10000 --output .benchmarks/result.json
+      - uses: actions/upload-artifact@v4
+        with:
+          name: scale-benchmark
+          path: .benchmarks/result.json

.gitignore

@@ -20,6 +20,7 @@ wheels/
 *.egg
 .pytest_cache/
 .coverage
+coverage.json
 htmlcov/
 .tox/
 .venv
@@ -49,6 +50,12 @@ output_csv/
 sqlgraph_output/
 failed_cases.json
 smoke_result.json
+*.duckdb
+.benchmarks/
+comparison-screenshots/
+generated_illustrations*/
+/book*/
+*.zip
 
 # Large or private local datasets
 df.csv

CAPABILITIES.yaml

@@ -0,0 +1,86 @@
+schema_version: capabilities-v1
+capabilities:
+  - id: input-baseline
+    status: implemented
+    description: Versioned SQL, dialect, schema, UDF, parameter, and scheduler baseline.
+    modules: [sqlgraph/baseline, sqlgraph/input]
+    tests: [tests/contract/test_baseline_manifest.py]
+    examples: [examples/minimal/scenario.yaml]
+    limitations: [Dynamic SQL must be materialized before parsing.]
+  - id: deterministic-sql-compiler
+    status: implemented
+    description: Statement-aware SQLGlot compilation with stable graph identities.
+    modules: [sqlgraph/parser, sqlgraph/builder, sqlgraph/identity]
+    tests: [tests/test_identity/test_determinism.py, tests/test_parser]
+    examples: [examples/ads_pipeline]
+    limitations: [Unresolvable multi-source columns are marked UNKNOWN.]
+  - id: two-level-graph
+    status: implemented
+    description: Heterogeneous graph plus reconstructable physical TableGraph projection.
+    modules: [sqlgraph/model, sqlgraph/analyze/table_graph.py, sqlgraph/lineage]
+    tests: [tests/test_graph_advanced/test_drilldown.py]
+    examples: [examples/video_commercial_warehouse]
+    limitations: [Nested struct fields remain attached to top-level physical columns.]
+  - id: evidence-subgraph
+    status: implemented
+    description: Intent-bound evidence collection, expansion, versioning, and stable hashes.
+    modules: [sqlgraph/evidence]
+    tests: [tests/test_evidence/test_engine.py]
+    examples: [examples/book_cases/caliber_consistency]
+    limitations: [External facts must be supplied by the caller.]
+  - id: evidence-discipline
+    status: implemented
+    description: Coverage obligations, counterevidence, exclusions, gaps, and residual unknowns.
+    modules: [sqlgraph/evidence]
+    tests: [tests/safety/test_evidence_gates.py]
+    examples: [examples/book_cases/caliber_consistency]
+    limitations: [Counterevidence search is structural rather than semantic.]
+  - id: offline-governance-metrics
+    status: implemented
+    description: Versioned inventory, topology, impact, consistency, similarity, motif, and anomaly analysis.
+    modules: [sqlgraph/analyze, sqlgraph/metrics]
+    tests: [tests/test_analyze]
+    examples: [examples/ads_pipeline]
+    limitations: [Optional embedding and model-based anomaly metrics require extras.]
+  - id: seven-step-loop
+    status: implemented
+    description: Observe, Explain, Propose, Authorize, Execute, Verify, and Learn orchestration.
+    modules: [sqlgraph/reasoning, sqlgraph/agent]
+    tests: [tests/test_reasoning/test_runner.py]
+    examples: [examples/minimal/scenario.yaml]
+    limitations: [Learn writes signals only and never expands authorization.]
+  - id: autonomy-policy
+    status: implemented
+    description: Evidence, authorization, reversibility, impact, risk, and reliability gates.
+    modules: [sqlgraph/autonomy]
+    tests: [tests/test_autonomy, tests/safety/test_autonomy_gates.py]
+    examples: [examples/book_cases/irreversible_drop]
+    limitations: [Thresholds are reference defaults and must be calibrated per deployment.]
+  - id: reversible-actions
+    status: implemented
+    description: Idempotent file and DuckDB actions with dry run, circuit breaker, and verified rollback.
+    modules: [sqlgraph/actions]
+    tests: [tests/test_actions, tests/safety/test_execution_recovery.py]
+    examples: [examples/book_cases/cold_table_retirement]
+    limitations: [Only local SQL files and DuckDB are included adapters.]
+  - id: three-layer-verification
+    status: implemented
+    description: Actual code comparison, independent graph rebuild, and runtime checks.
+    modules: [sqlgraph/verification, sqlgraph/verify]
+    tests: [tests/test_verification, tests/safety/test_verification_failure.py]
+    examples: [examples/book_cases/caliber_consistency]
+    limitations: [Production runtime checks require a deployment-specific adapter.]
+  - id: graphrag-grounding
+    status: implemented
+    description: Assertion-to-citation validation constrained by baseline and evidence versions.
+    modules: [sqlgraph/graphrag, sqlgraph/serialize/graphrag.py]
+    tests: [tests/test_graphrag]
+    examples: [examples/minimal/scenario.yaml]
+    limitations: [The repository validates supplied assertions but does not bundle an LLM.]
+  - id: audit-replay
+    status: implemented
+    description: Append-only hash-chained events with integrity verification and task replay.
+    modules: [sqlgraph/audit]
+    tests: [tests/test_audit, tests/safety/test_audit_integrity.py]
+    examples: [examples/minimal/scenario.yaml]
+    limitations: [Local JSONL storage is a reference adapter, not a distributed ledger.]

README.md

@@ -89,7 +89,7 @@ pip install -e .
 pip install sqlgraph-lineage
 ```
 
-Requires Python 3.9 through 3.12.
+Requires Python 3.10 through 3.12.
 
 ## Quick start
 
@@ -130,6 +130,22 @@ sqlgraph playground
 sqlgraph stats ./sql --dialect spark
 ```
 
+### Evidence-bound governance quickstart
+
+Run the minimal book scenario and verify its complete evidence package:
+
+```bash
+sqlgraph governance run examples/minimal/scenario.yaml -o demo_output/minimal
+sqlgraph governance verify demo_output/minimal
+sqlgraph governance replay demo_output/minimal/audit.jsonl
+```
+
+The output contains the deterministic input baseline, graph snapshot, evidence
+bundle, autonomy decision, action result, independent verification report, and a
+hash-chained seven-step audit log. See [CAPABILITIES.yaml](CAPABILITIES.yaml) for
+the status and test evidence of every advertised capability, and
+[docs/limitations.md](docs/limitations.md) for the supported boundary.
+
 ### Lineage Explorer (search + local subgraphs)
 
 For large inputs, avoid one giant HTML file. Serve a searchable explorer instead:

README.zh-CN.md

@@ -54,7 +54,7 @@ pip install -e .
 pip install sqlgraph-lineage
 ```
 
-需要 Python 3.9 到 3.12。
+需要 Python 3.10 到 3.12。
 
 ## 快速开始
 
@@ -95,6 +95,19 @@ sqlgraph playground
 sqlgraph stats ./sql --dialect spark
 ```
 
+### 证据约束治理 Quickstart
+
+```bash
+sqlgraph governance run examples/minimal/scenario.yaml -o demo_output/minimal
+sqlgraph governance verify demo_output/minimal
+sqlgraph governance replay demo_output/minimal/audit.jsonl
+```
+
+输出包含确定性输入基线、图快照、证据包、自治决议、动作结果、独立验证报告，
+以及带哈希链的七步审计日志。所有对外能力的状态与测试证据见
+[`CAPABILITIES.yaml`](CAPABILITIES.yaml)，支持边界见
+[`docs/limitations.md`](docs/limitations.md)。
+
 ### 血缘检索浏览器（检索 + 局部子图）
 
 大规模输入不再依赖单个巨大 HTML，改用可检索的本地浏览器：

docs/architecture.md

@@ -94,3 +94,24 @@ An in-memory **`PropertyGraph`** of typed nodes and edges.
   text, so lookalike expressions on different columns stay distinct.
 - **Scale** — 96/128-bit fingerprints keep collisions negligible even for
   warehouses with millions of columns.
+
+## Governance reference pipeline
+
+The governance modules build on the graph without coupling the compiler to an
+execution environment:
+
+```text
+Baseline -> HeteroGraph/TableGraph -> Evidence/Grounding -> Autonomy
+         -> Action Adapter -> Independent Verification -> Audit Replay
+```
+
+- `baseline` identifies the effective input and discloses missing dependencies.
+- `evidence` collects an intent-bound, versioned subgraph with coverage duties.
+- `graphrag` rejects assertions with missing, stale, or out-of-scope citations.
+- `autonomy` applies evidence, authorization, and reversibility gates before scoring.
+- `actions` provides idempotent file and DuckDB adapters with verified rollback.
+- `verification` compares actual code, rebuilds structure, and checks runtime effects.
+- `audit` appends hash-chained events; `reasoning` orchestrates the seven steps.
+
+No analysis score directly authorizes an action. The action and verification
+adapters are explicit seams for production integrations.

docs/book-map.md

@@ -0,0 +1,20 @@
+# Book to Code Map
+
+| Book topic | Module | Tests | Observable artifact |
+|---|---|---|---|
+| Object and provenance model | `sqlgraph.baseline`, `sqlgraph.model` | `tests/contract/test_baseline_manifest.py` | `baseline.json`, `graph.json` |
+| Semantic runtime | `sqlgraph.parser`, `sqlgraph.builder`, `sqlgraph.analyze` | `tests/test_identity`, `tests/test_analyze` | deterministic graph and analysis snapshot |
+| Evidence chain | `sqlgraph.evidence`, `sqlgraph.graphrag` | `tests/test_evidence`, `tests/test_graphrag` | `evidence.json` |
+| Graded autonomy | `sqlgraph.autonomy` | `tests/test_autonomy` | `decision.json` |
+| Reversibility red line | `sqlgraph.actions` | `tests/safety/test_execution_recovery.py` | execution and rollback events |
+| Seven-step loop | `sqlgraph.reasoning` | `tests/test_reasoning/test_runner.py` | seven hash-chained audit events |
+| Structural insight | `sqlgraph.analyze` | `tests/test_analyze` | profile and governance snapshot |
+| Caliber consistency | `examples/book_cases/caliber_consistency` | `tests/test_integration/test_book_cases.py` | successful L3 repair |
+| Cost governance | `examples/book_cases/cold_table_retirement` | `tests/test_integration/test_book_cases.py` | reversible quarantine proposal |
+| Verification | `sqlgraph.verification` | `tests/test_verification` | `verification.json` |
+| Three objections | `sqlgraph.graphrag`, `sqlgraph.autonomy`, `sqlgraph.audit` | `tests/safety` | rejection, escalation, and replay records |
+| Engineering implementation | `sqlgraph.cli`, `sqlgraph.serve` | `tests/test_cli`, `tests/test_serve` | CLI package and Explorer |
+| Boundaries and future work | `CAPABILITIES.yaml`, `docs/limitations.md` | `tests/contract/test_capabilities.py` | explicit capability status |
+
+Claims without a module, current test, and observable artifact must be marked
+`experimental`, `planned`, or `concept-only` in `CAPABILITIES.yaml`.

docs/limitations.md

@@ -0,0 +1,22 @@
+# Limitations
+
+- SQL parsing depends on SQLGlot. Unsupported or dynamic SQL must be materialized
+  before analysis; the system does not execute templates to guess their output.
+- Multi-source columns without enough schema information are marked `UNKNOWN`.
+  Evidence containing them cannot pass the `no_unresolved` obligation.
+- Expression fingerprints are conservative. Algebraically equivalent expressions
+  such as reordered addition are intentionally not merged.
+- Counterevidence search is structural. Domain counterexamples and external facts
+  must be supplied through adapters.
+- The included action adapters support local SQL files and DuckDB only. They are
+  reference implementations, not production warehouse credentials or schedulers.
+- JSONL audit storage detects local mutation, deletion, and reordering but is not
+  a distributed consensus ledger.
+- Runtime verification trusts the configured runtime adapter's observations.
+  Production deployments must isolate that adapter from the execution path.
+- Embeddings and model-based anomaly detection are optional analysis features and
+  cannot authorize governance actions.
+- The ordinary CI scale smoke test uses 100 generated SQL statements. The 1,000
+  and 10,000 statement benchmarks run in a separate workflow.
+- SqlGraph does not bundle an LLM, approval platform, identity provider, metadata
+  catalog, or production scheduler.