diff --git a/README.md b/README.md index 87653b7..8e52f87 100644 --- a/README.md +++ b/README.md @@ -610,6 +610,10 @@ about being untranslated rather than absent. Yes, please. [CONTRIBUTING.md](CONTRIBUTING.md) has how to build it, how to run each part of the test suite, and how behaviour gets proposed before it is built. +New to the codebase and want the map before the process? [`docs/guide/`](docs/guide/README.md) +walks through the architecture, the pipeline and the safety rules a layer at a time, each +page linking back to [`docs/PGFATHOM.md`](docs/PGFATHOM.md) and the specs for full detail. + The three most valuable things you can send: - **A verdict that is wrong.** A confirmed false positive is treated as a blocking diff --git a/docs/guide/01-overview/README.md b/docs/guide/01-overview/README.md new file mode 100644 index 0000000..800cb82 --- /dev/null +++ b/docs/guide/01-overview/README.md @@ -0,0 +1,13 @@ +# 01 · Overview + +← [Guide](../README.md) + +Start here if this is your first time in the project. + +| Page | Covers | +|---|---| +| [The problem](the-problem.md) | Why legacy schemas lose structure, and what it costs | +| [What it does](what-it-does.md) | The two commands, the five verdicts, a real output example | + +Next: [02 · Architecture](../02-architecture/README.md) if you're about to touch code, or +straight to [04 · Usage](../04-usage/README.md) if you just want to run the tool. diff --git a/docs/guide/01-overview/the-problem.md b/docs/guide/01-overview/the-problem.md new file mode 100644 index 0000000..93e0057 --- /dev/null +++ b/docs/guide/01-overview/the-problem.md @@ -0,0 +1,47 @@ +# The problem + +← [Guide](../README.md) › [Overview](README.md) + +Old, large PostgreSQL databases carry more structure than they declare. Years of +maintenance, different teams, and botched migrations strip away whatever gave the schema +its meaning. + +## The common symptom: a missing foreign key + +``` +=# \d order + Column | Type | Nullable +-----------------+--------+---------- + id | bigint | not null + customer_id | bigint | not null +``` + +`customer_id` points at `customer.id` in **every row** — but no constraint says so. +Typical reasons, legitimate and not: someone disabled constraints for a bulk load and +never turned them back on, the ORM of the day never created it, a migration from an even +older database brought only the tables, or nobody thought it mattered. + +## The cost, which shows up later + +- `\d table` shows no relationship at all — nobody new to the project can read the model +- an ERD tool draws a page of disconnected boxes +- a code generator produces structs with no association +- worst: without the constraint, the database never stopped orphan rows from getting in — + so they probably already did, silently, years ago + +## The sneakier variant: `NOT VALID` + +A constraint can be **declared** and still guarantee nothing, if it was created `NOT +VALID` and never validated. Postgres starts blocking *new* violations, but never checked +the rows that were already there. `\d` shows the FK, the ERD tool draws the arrow, +everyone assumes integrity — and the old orphans stay invisible. + +That's exactly the second finding `pgfathom audit` looks for — see +[What it does](what-it-does.md). + +## Full detail + +The design record goes further into this — the "layer of lost knowledge" beyond foreign +keys (inconsistent soft-delete, missing tenant columns, `varchar` acting as an +undeclared enum) that's out of scope for the v0.1 but shaped the roadmap. See +[`docs/PGFATHOM.md` § O problema](../../PGFATHOM.md#o-problema) (Portuguese). diff --git a/docs/guide/01-overview/what-it-does.md b/docs/guide/01-overview/what-it-does.md new file mode 100644 index 0000000..ad6987c --- /dev/null +++ b/docs/guide/01-overview/what-it-does.md @@ -0,0 +1,46 @@ +# What it does + +← [Guide](../README.md) › [Overview](README.md) + +`pgfathom` connects to a PostgreSQL instance, reads the system catalog, cross-references +usage statistics, mines view and function definitions for real joins, and samples data in +a controlled way. From that it builds a semantic model of the database — entities, real +relationships (declared, inferred, evidenced by usage), and findings about the schema's +structural health. + +The main output isn't a diagram. It's an actionable report plus ready-to-use artifacts: +suggested DDL, diagnostic queries, and a versioned JSON model other tools can consume. + +## Two commands + +### `pgfathom audit` + +Findings that need **no inference at all** — deterministic, no false-positive risk, +because they're raw catalog facts: + +- a declared constraint that's `NOT VALID` and never validated +- a declared FK with no index on the child column (classic `DELETE` trap) +- the coverage block for the analysis + +### `pgfathom discover` + +The full pipeline — see [02 · Architecture](../02-architecture/README.md) for the 7 +stages. Produces a verdict per candidate. + +## The five verdicts + +| Verdict | Means | Response | +|---|---|---| +| **confirmed** | total containment, no orphans | a forgotten FK — generate the DDL | +| **broken** | real relationship, high but not total containment | **the most valuable finding** — a data bug that's been in production for years. Generates the orphan-listing query before any DDL | +| **weak** | insufficient evidence (single distinct value, mostly nulls, empty table) | doesn't conclude, and says so | +| **rejected** | low containment | name coincidence, discarded — but reported, so an obviously-named column never looks silently ignored | +| **unvalidated** | timeout, permission, unsupported case | never confused with "clean" | + +`broken` is the finding that justifies the project: **integrity broken in production, and +nobody knew.** + +## Full detail + +Real output example, the SQL-artifact conventions, and the full README walkthrough live in +the root [`README.md`](../../../README.md#what-it-does). diff --git a/docs/guide/02-architecture/README.md b/docs/guide/02-architecture/README.md new file mode 100644 index 0000000..861ca15 --- /dev/null +++ b/docs/guide/02-architecture/README.md @@ -0,0 +1,24 @@ +# 02 · Architecture + +← [Guide](../README.md) + +The "how it works inside": the binary's layers, the core Go types, and the 7-stage +pipeline that turns a `pg_catalog` into verdicts. Read this before touching code. + +| Page | Covers | +|---|---| +| [Layers](layers.md) | The 10 `internal/` layers, one-way dependency, tech stack, the bar for a new dependency | +| [Data model](data-model.md) | The core structs — `Schema`, `Candidate`, `Signal`, `Verdict`, `Coverage` | +| [Algorithm](algorithm.md) | The 7 pipeline stages, each with the SQL or rule that implements it | + +```mermaid +flowchart TD + L[Layers: where things live] --> M[Data model: the shapes that cross layers] + M --> A[Algorithm: the 7 stages that fill those shapes] +``` + +## See also + +- [03 · Features & Safety](../03-features-and-safety/README.md) — the rules every change + here has to respect +- [`openspec/specs/`](../../../openspec/specs/) — the formal spec per layer, in Portuguese diff --git a/docs/guide/02-architecture/algorithm.md b/docs/guide/02-architecture/algorithm.md new file mode 100644 index 0000000..1f43886 --- /dev/null +++ b/docs/guide/02-architecture/algorithm.md @@ -0,0 +1,49 @@ +# Algorithm + +← [Guide](../README.md) › [Architecture](README.md) + +``` +catalog → usage evidence → candidates → scoring → stats prefilter → validation +``` + +**1. Read the catalog.** Tables, columns, types, keys, indexes, comments, declared FKs — +including `pg_constraint.convalidated`, which is what reveals a `NOT VALID` constraint. +Catalog only, no user data. + +**2. Mine usage evidence.** Extracts join predicates from view definitions, function +bodies, and (when available) `pg_stat_statements` — a narrow tokenizing extractor, not a +full SQL parser (a full parser would need `pg_query_go`, which requires cgo). It's allowed +to fail: unrecognized SQL is simply ignored. A missed join is a lost signal, never a wrong +answer — see [Known pitfalls](../03-features-and-safety/known-pitfalls.md). + +**3. Generate candidates.** Strips reference affixes from column names +(`_id`, `_key`, `_fk`, …) and matches the result against depluralized table names, using a +**naming profile** — see [Naming profiles](../03-features-and-safety/naming-profiles.md). +A join found in step 2 also creates a candidate directly, bypassing name matching. + +**4. Score on metadata alone.** A join found in a view/function is the strongest signal +there is. Exact name match, identical type, single unambiguous target, existing index, and +comment mentions add confidence; ambiguity and generic domain names (`status`, `type`) +subtract it. Below `--min-score`, a candidate is dropped before touching data. + +**5. Prefilter on planner statistics.** Reads `pg_stats` and `pg_class.reltuples` — free, +no I/O on tables. If the child column has more distinct values than the parent has rows, +full containment is arithmetically impossible. + +**6. Validate against the data.** One aggregate query per surviving candidate — never +fetches rows, only counts, aggregated by distinct value before the anti-join. Containment +is reported in two dimensions (by row and by distinct value), because a single bad value +repeated a million times and a million rare bad values are different problems. Every query +runs under `statement_timeout`; a candidate that times out is marked `unvalidated` and the +run continues. + +**7. Verdict.** Total containment on both dimensions → `confirmed`. High but not total → +`broken`, with the orphan counts. Below the rejection threshold → `rejected`. Insufficient +evidence → `weak`, regardless of containment. **In sampled mode, no candidate can ever be +marked `confirmed`** — only `--full` proves the absence of orphans. + +## Full detail + +The exact SQL for step 6, the type-compatibility rules for step 3, and the statistical +tolerance margins for step 5 are in +[`docs/PGFATHOM.md` § Algoritmo](../../PGFATHOM.md#algoritmo) (Portuguese). diff --git a/docs/guide/02-architecture/data-model.md b/docs/guide/02-architecture/data-model.md new file mode 100644 index 0000000..c9e83f4 --- /dev/null +++ b/docs/guide/02-architecture/data-model.md @@ -0,0 +1,97 @@ +# Data model + +← [Guide](../README.md) › [Architecture](README.md) + +Sketch of the central structures in `internal/model` (pure types, no I/O — see +[Layers](layers.md)). Keeps **read**, **evidenced**, and **inferred** apart at the type +level. + +```go +type Table struct { + Schema string + Name string + Columns []Column + PrimaryKey []string + ForeignKeys []ForeignKey // declared only + Stats TableStats +} + +type ForeignKey struct { + Name string + Columns []string + RefTable string + RefColumns []string + Validated bool // pg_constraint.convalidated — false = NOT VALID + HasIndex bool +} +``` + +`Validated` is the field that reveals `audit`'s most treacherous finding — see +[The problem](../01-overview/the-problem.md). + +## Candidates and signals + +```go +type Signal struct { + Kind SignalKind + Weight float64 // can be negative + Detail string // object name only, NEVER a data value +} + +type Candidate struct { + Child ColumnRef + Parent ColumnRef + Signals []Signal + MetaScore float64 // 0..1, before touching data + Validation *Validation // nil if not validated + Verdict Verdict +} +``` + +Every score must be traceable to the signals that produced it — no scoring path is +allowed to bypass `Signals`. (`openspec/specs/candidate-scoring/spec.md`) + +## Validation and verdict + +```go +type Validation struct { + Method string // "full" or "sampled" + NotNullRows int64 + DistinctVals int64 + OrphanRows int64 + OrphanVals int64 + ContainmentRows float64 // (NotNullRows - OrphanRows) / NotNullRows + ContainmentVals float64 // (DistinctVals - OrphanVals) / DistinctVals +} + +type Verdict string + +const ( + VerdictConfirmed Verdict = "confirmed" + VerdictBroken Verdict = "broken" + VerdictWeak Verdict = "weak" + VerdictRejected Verdict = "rejected" + VerdictUnvalidated Verdict = "unvalidated" +) +``` + +## Coverage — mandatory on every output + +```go +type Coverage struct { + TablesTotal int + TablesAnalyzed int + TablesNoPrivilege []string + CandidatesTimedOut int + StatsResetAt *time.Time +} +``` + +`Coverage` is the direct materialization of "silence is never reported as a clean bill of +health" — see [Safety guarantees](../03-features-and-safety/safety-guarantees.md). + +## Full detail + +The complete struct set — `Column`, `ColumnStats` (and why two of its fields are +deliberately unexported), every `SignalKind`, `Finding` — is in +[`docs/PGFATHOM.md` § Modelo interno](../../PGFATHOM.md#modelo-interno) (Portuguese). diff --git a/docs/guide/02-architecture/layers.md b/docs/guide/02-architecture/layers.md new file mode 100644 index 0000000..c1c36b5 --- /dev/null +++ b/docs/guide/02-architecture/layers.md @@ -0,0 +1,65 @@ +# Layers + +← [Guide](../README.md) › [Architecture](README.md) + +One-way dependency. Every layer testable in isolation. + +``` +cmd/pgfathom CLI entry, flag parsing, orchestration + | +internal/db connection, pool, security policies, timeouts + | +internal/catalog reads pg_catalog and information_schema + | +internal/sqlprobe extracts join predicates from views and functions + | +internal/model internal model, pure types, no I/O + | +internal/profile naming profiles, loaded from file + | +internal/infer candidate generation and scoring (metadata only) + | +internal/stats planner-statistics prefilter + | +internal/validate validation against data, sampling, anti-join + | +internal/report terminal, JSON, SQL rendering +``` + +Structural rules: + +- **`internal/model` imports nothing from any other layer.** Enforced by an automated + test, not review — it's the easiest dependency to introduce by accident. +- **`internal/infer`, `internal/profile`, `internal/sqlprobe` are deterministic**, no + database access. Testable without Docker or network. +- **`internal/validate` is the only layer that reads user table data.** Everything else in + the pipeline operates on catalog, planner statistics, or in-memory model only. + +## Tech stack + +| | | +|---|---| +| Language | Go, floor 1.25 | +| Driver | `github.com/jackc/pgx/v5` + `pgxpool` — no ORM | +| CLI | `github.com/spf13/cobra` | +| Config | `github.com/pelletier/go-toml/v2` | +| Concurrency | `golang.org/x/sync/errgroup` with `SetLimit` | +| Log | `log/slog` (stdlib) | +| Terminal table | `text/tabwriter` (stdlib) | +| Interactive guide | `bubbletea` + `lipgloss`, only in `pgfathom setup` | +| Testing | `testing` + `github.com/google/go-cmp` | +| Integration | `testcontainers-go`, behind `//go:build integration` | +| Release | `goreleaser` | +| Lint | `golangci-lint` | + +**No cgo.** Cross-compiling has to stay trivial — that's why the SQL-mining layer uses a +narrow extractor of its own instead of `pg_query_go`. + +## The dependency tree is a product requirement + +> The DBA who has to approve running this against production opens `go.mod` first. + +Every new dependency needs a measured cost (modules + binary size) in the PR description — +see [06 · Contributing](../06-contributing/README.md). The full ledger of what's been +accepted and refused, with the numbers, lives in +[`openspec/project.md`](../../../openspec/project.md#stack) (Portuguese). diff --git a/docs/guide/03-features-and-safety/README.md b/docs/guide/03-features-and-safety/README.md new file mode 100644 index 0000000..f55d8e9 --- /dev/null +++ b/docs/guide/03-features-and-safety/README.md @@ -0,0 +1,25 @@ +# 03 · Features & Safety + +← [Guide](../README.md) + +What makes it safe to point at a production database, and the two most delicate gears in +the product: recognizing naming convention in any language, and knowing where its own +confidence runs out. + +| Page | Covers | +|---|---| +| [Naming profiles](naming-profiles.md) | How pt-br/en/es work, why depluralization returns a *set* of forms, why a new profile is the easiest contribution in the project | +| [Safety guarantees](safety-guarantees.md) | The 5 non-negotiable rules — and, more importantly, how each is enforced by a test, not a promise | +| [Known pitfalls](known-pitfalls.md) | Specific ways the tool can be wrong with confidence: sampling against clustered orphans, usage counters that reset silently, silent partial coverage | + +## Why these three live together + +All three answer "what can go wrong, and what the project does about it" — one about +input (naming the parser doesn't recognize), one about policy (a rule that never bends), +one about interpretation (right number, wrong conclusion). Anyone reviewing a PR in +[02 · Architecture](../02-architecture/README.md) should know all three before approving. + +## See also + +[06 · Contributing](../06-contributing/README.md) — these are the same rules a PR gets +evaluated against. diff --git a/docs/guide/03-features-and-safety/known-pitfalls.md b/docs/guide/03-features-and-safety/known-pitfalls.md new file mode 100644 index 0000000..f0b12ef --- /dev/null +++ b/docs/guide/03-features-and-safety/known-pitfalls.md @@ -0,0 +1,51 @@ +# Known pitfalls + +← [Guide](../README.md) › [Features & Safety](README.md) + +Not edge cases to solve later. Specific ways the tool can be **wrong with confidence** — +the one way it could actually lose credibility. + +## Sampling is weaker against orphans than against anything else + +Orphans are almost never random. They arrive in batches — a bad load, a weekend migration, +a period when the application had a bug — and end up **physically clustered on the same +pages**. `TABLESAMPLE SYSTEM` samples by page. So the default sampling mode is biased +*against* exactly the pattern the tool exists to find: half a million clustered orphans +can survive a clean-looking sample. + +**Design response:** sampled mode is **triage, not evidence**. `--full` is the answer, and +the report says so explicitly, not in a footnote. See +[Safety guarantees](safety-guarantees.md) rule 5. + +## Usage counters don't survive what people assume they survive + +`pg_stat_user_tables` resets on `pg_stat_reset()`, resets on `pg_upgrade`, and is **per +node** — a table read exclusively on a read replica shows zero reads on the primary. +Saying "this table hasn't been used in years" from this, on a database where someone might +act on it, is the worst thing the tool could do. Every finding in this family always +carries the reset timestamp and the replica caveat — the phrasing is always "no usage +recorded since X on this node", never "unused". + +## Silent partial coverage + +`pg_stats` only exposes rows for tables the current user can read. If the read-only role +lacks `SELECT` on the full schema — plausible in a public-sector org, where getting +privilege is a political process — the analysis is partial and the report ends up clean +**for the wrong reason**. That's why the `Coverage` struct is mandatory on every output. + +## Cases that get skipped with a note, never silently + +- **Polymorphic relationships** (`document_id` only makes sense with `document_type`) — + validation correctly rejects on low containment, but the tool should recognize the + pattern from neighboring columns and say so +- **Partially-matched composite FKs** — if the child side covers only part of the target + key's positions, no candidate is generated, and the matched fraction is noted +- **Partitioned tables** — reads from the parent, never iterates partitions +- **Dangling references** — columns pointing at tables that no longer exist; correctly + rejected, but worth reporting separately since it's a finding in itself + +## Full detail + +`docs/PGFATHOM.md` § [Armadilhas conhecidas](../../PGFATHOM.md#armadilhas-conhecidas) +(Portuguese) has the full list, including cases specific to multi-schema and table +inheritance. diff --git a/docs/guide/03-features-and-safety/naming-profiles.md b/docs/guide/03-features-and-safety/naming-profiles.md new file mode 100644 index 0000000..82cfcef --- /dev/null +++ b/docs/guide/03-features-and-safety/naming-profiles.md @@ -0,0 +1,53 @@ +# Naming profiles + +← [Guide](../README.md) › [Features & Safety](README.md) + +> Most schema tools assume English. The databases that need this tool most, usually +> aren't. + +Affix and plural rules live in **TOML**, never in Go code. The binary embeds the official +profiles via `embed`; `--profile` accepts a path to a user file. + +```toml +name = "pt-br" +column_suffixes = ["_id", "_codigo", "_cod", "_key", "_ref", "_fk"] +table_prefixes = ["tb_", "tbl_", "sys_", "cad_", "mov_"] + +[[plural]] # opcoes → opcao +suffix = "oes" +singular = "ao" +``` + +## Why normalization returns a set, not a string + +The obvious path — apply rules in order, return the first match — breaks on real +ambiguity. `logins` produces `logim` under the `ns → m` rule (correct for `armazens`), and +`login` under the generic drop-`s` rule. Nothing in the name resolves which is right. + +So table-name normalization returns an **ordered set of candidate forms**, always +including the unmodified original. A match succeeds if *any* form matches. The matched +form is reported alongside the candidate, which is what lets scoring tell an exact match +from an aggressive one apart. + +Accepted cost: candidate noise from an over-aggressive form. That's fine — scoring +penalizes it, and validation against real data rejects what turns out to be coincidence. +**The project accepts candidate noise. It does not accept a confirmed false positive.** + +## Convention detection, before you pick a profile + +Before requiring a profile choice, `pgfathom` tries to **derive the convention from the +schema itself** — table-prefix by frequency, reference affix from the FKs the schema +already declares. On a real municipal database, this took recall from 0.5% to 79.0%; on +another, from 22.2% to 72.2%. Detection is always reported, never silently swaps the base +profile — it only adds candidate forms on top of it. + +## The easiest contribution in the project + +A new naming profile needs no knowledge of the rest of the codebase — a TOML file plus a +table of test cases. See [06 · Contributing](../06-contributing/README.md). + +## Full detail + +The complete pt-br rule table and the full case for the set-based design are in +[`docs/PGFATHOM.md` § Perfis de nomenclatura](../../PGFATHOM.md#perfis-de-nomenclatura) +(Portuguese). diff --git a/docs/guide/03-features-and-safety/safety-guarantees.md b/docs/guide/03-features-and-safety/safety-guarantees.md new file mode 100644 index 0000000..d34b970 --- /dev/null +++ b/docs/guide/03-features-and-safety/safety-guarantees.md @@ -0,0 +1,49 @@ +# Safety guarantees + +← [Guide](../README.md) › [Features & Safety](README.md) + +> A violation of any of these is a maximum-severity bug, not a style question. + +The target use case includes public-sector databases holding national ID numbers, health +records, taxpayer data. That shapes every rule below. + +**1. Read-only, absolute.** No code path issues a statement that alters the database under +analysis. Enforced by `default_transaction_read_only` set in the pool's connection hook — +never per-query — so a new connection cannot exist without it, and by a test that proves a +write attempt fails in the session. + +**2. User data never leaves.** What comes *out*: counts, ratios, object names — never a +row value, in any output, log, JSON field, or error message. The specific trap: +`pg_stats.most_common_vals` and `histogram_bounds` **are real user data**, even though +they come from a system view; they're read legitimately for scoring, live in memory only, +and never get serialized. Enforced by a dedicated test that scans every output structure of +every test scenario for a leaked value. + +**3. No claim without evidence.** Every verdict prints the metric that produced it. + +**4. Silence is never reported as a clean bill of health.** A table skipped for missing +privilege, a candidate that timed out, a schema out of scope — all of it appears in the +mandatory coverage block on every run, including the run with no flags at all. + +**5. Zero confirmed false positives.** A missed relationship costs a finding. A wrong one, +confirmed, costs the tool's credibility. Between recovering more and never being wrong, +never being wrong wins — which is why **a sampled run can never mark a candidate +`confirmed`**; only `--full` proves the absence of orphans. + +## Other hard operational guarantees + +- **Won't take the server down** — `statement_timeout`, `lock_timeout`, + `idle_in_transaction_session_timeout` on every validation query; concurrency capped and + configurable, defaulting low +- **Credential precedence** avoids leaking a DSN via `ps`/shell history — `PGFATHOM_DSN` → + libpq env vars → `--dsn` last, with the leak risk documented in `--help` +- **Conservative default scope** — `public` only, unless told otherwise, because the run + someone makes before reading any documentation is against a production database that + isn't theirs +- **Small dependency tree, on purpose** — see + [Layers](../02-architecture/layers.md#the-dependency-tree-is-a-product-requirement) + +## Full detail + +`docs/PGFATHOM.md` § [Regras invioláveis](../../PGFATHOM.md#regras-invioláveis) and § +[Segurança operacional](../../PGFATHOM.md#segurança-operacional) (Portuguese). diff --git a/docs/guide/04-usage/README.md b/docs/guide/04-usage/README.md new file mode 100644 index 0000000..7b464e4 --- /dev/null +++ b/docs/guide/04-usage/README.md @@ -0,0 +1,20 @@ +# 04 · Usage + +← [Guide](../README.md) + +The practical manual — install, first run, every flag. + +| Page | Covers | +|---|---| +| [CLI reference](cli-reference.md) | Install methods, the `pgfathom setup` guide, schema-scope control, full flag reference, the three output formats | + +**Deliberately a thin section.** Room to grow — a guided walkthrough ("first run against a +real database, from zero") or troubleshooting notes belong here as new pages rather than +bloating [CLI reference](cli-reference.md). + +## See also + +- [01 · Overview](../01-overview/README.md) — the verdicts `discover` output produces, + before diving into flags +- [03 · Features & Safety](../03-features-and-safety/README.md) — why the default scope is + `public` and nothing else diff --git a/docs/guide/04-usage/cli-reference.md b/docs/guide/04-usage/cli-reference.md new file mode 100644 index 0000000..4bd5426 --- /dev/null +++ b/docs/guide/04-usage/cli-reference.md @@ -0,0 +1,84 @@ +# CLI reference + +← [Guide](../README.md) › [Usage](README.md) + +## Install + +```console +# Debian/Ubuntu/Fedora/RHEL +$ sudo dpkg -i pgfathom_*_linux_amd64.deb # or: sudo rpm -i pgfathom_*_linux_amd64.rpm + +# Go +$ go install github.com/lvcas-dotcom/pgfathom/cmd/pgfathom@latest + +# Homebrew (macOS) +$ brew install lvcas-dotcom/tap/pgfathom + +# Docker +$ docker run --rm ghcr.io/lvcas-dotcom/pgfathom:latest discover --dsn "$DSN" +``` + +Full install matrix, including checksum verification, in the root +[`README.md`](../../../README.md#installing). + +## First run: `pgfathom setup` + +Twenty-one flags is a lot to meet at once. The interactive guide asks for the connection, +lists the schemas the server actually has (with table counts — the most common way a +first run ends in "nothing found" is pointing at a schema with six tables while the other +sixty live elsewhere), asks how thoroughly to validate, then **prints the `discover` +command your answers composed** before running it. Never asks for a password — +`PGFATHOM_DSN` or `~/.pgpass`. + +## Scope — `public` by default, nothing beyond + +```console +$ pgfathom discover --all-schemas --exclude-schema 'audit_*' +``` + +| Flag | Scope | +|---|---| +| *(none)* | `public` | +| `--schema a,b` | exactly those schemas | +| `--all-schemas` | every non-system schema the role can access | +| `--exclude-schema` | glob patterns of schema to drop | +| `--exclude` | glob patterns of **table** to skip | + +`--schema` and `--all-schemas` are mutually exclusive — passing both is a usage error, not +a guessed precedence. Whatever is left out of scope is always named in the coverage block, +even on the flagless run. + +## Flags + +``` +pgfathom discover [flags] + + --dsn string connection string (prefer PGFATHOM_DSN) + --schema strings schemas to analyse (default: public) + --all-schemas every accessible non-system schema; excludes --schema + --exclude-schema schema glob patterns to drop from scope + --exclude strings table glob patterns to skip + --profile string naming profile (default: pt-br) + --full validate against every row, no sampling + --sample int target rows per sample (default: 100000) + --min-score float metadata threshold to validate (default: 0.5) + --no-sql-probe disable view/function join mining + --timeout duration statement_timeout per validation query (default: 30s) + --concurrency int concurrent validations (default: 4) + --format string table | json | sql (default: table) + --out string output directory for artifacts + --include-rejected also show discarded candidates + +pgfathom audit [flags] + # same connection/scope flags + # findings that need no inference +``` + +## The three outputs + +**Terminal** — grouped by verdict, `broken` first. **JSON** — the full model, treated as a +public API (`schema_version` field). **SQL** — one file per category, written (never +piped), each opening with a mandatory review header; DDL for a `broken` relationship stays +commented out until orphans are resolved. + +Full detail on each format: root [`README.md`](../../../README.md#how-it-works). diff --git a/docs/guide/05-positioning-and-metrics/README.md b/docs/guide/05-positioning-and-metrics/README.md new file mode 100644 index 0000000..c17a650 --- /dev/null +++ b/docs/guide/05-positioning-and-metrics/README.md @@ -0,0 +1,24 @@ +# 05 · Positioning & Metrics + +← [Guide](../README.md) + +Two questions every serious project has to answer honestly: **why this, and not something +that already exists**, and **does the published number mean what it looks like it means**. + +| Page | Covers | +|---|---| +| [Prior art](prior-art.md) | Where `pgfathom` sits versus ORMs, data-profiling literature (SPIDER/BINDER/MIND/Metanome), Azimutt, Hackolade — and where it deliberately doesn't compete | +| [Measuring correctness](measuring-correctness.md) | The public benchmark corpus, the two measurement regimes, and the three things the recall table doesn't say | + +## Why these two live together + +Prior art without a number is marketing. A number without prior art doesn't say if it's +good. Both follow the same editorial principle as the rest of the project: **report what +underperforms with the same clarity as what works** — see the "join mining barely moves +the needle" section in [Measuring correctness](measuring-correctness.md). + +## See also + +[02 · Architecture](../02-architecture/README.md) → +[Algorithm, step 2](../02-architecture/algorithm.md), for *how* usage evidence gets +extracted before reading how much it pays off here. diff --git a/docs/guide/05-positioning-and-metrics/measuring-correctness.md b/docs/guide/05-positioning-and-metrics/measuring-correctness.md new file mode 100644 index 0000000..19c1399 --- /dev/null +++ b/docs/guide/05-positioning-and-metrics/measuring-correctness.md @@ -0,0 +1,44 @@ +# Measuring correctness + +← [Guide](../README.md) › [Positioning & Metrics](README.md) + +The headline metric is **recovery rate**: take a schema with complete foreign keys, drop +every one, run `pgfathom`, count how many come back. `make corpus && make benchmark` +automates this against a versioned recipe (`bench/corpus.toml`); full numbers live in the +root [`README.md` § How correctness is measured](../../../README.md#how-correctness-is-measured). + +## Two regimes, because they answer different questions + +- **Partial** — removes half the declared FKs, measures recovery on that half. The + ordinary case: a database that declared part of its integrity and forgot the rest. +- **Greenfield** — removes everything, measures on everything. The hardest case, and the + one this tool exists for. + +On the pt-BR municipal schema in the corpus, partial recall is **84.9%**; greenfield, on +the exact same schema, is **16.6%**. That gap is the entire argument for naming-convention +detection — see [Naming profiles](../03-features-and-safety/naming-profiles.md). + +## Three things the recall table does not say + +1. **No verdict is measured** — the public dumps have no rows, so nothing gets confirmed + or broken. What's measured is whether the right candidate was *raised*. +2. **Candidates outside the ground truth aren't errors** — in a real schema, a true + relationship nobody declared is the product, not a false positive. +3. **Composite keys: 1 of 53 recovered on GitLab**, and the other 52 are explained (they + only match half the key's shape) — not silently absorbed into a lower headline number. + +## Coverage is part of the metric + +Early measurements were taken with a quarter of a schema out of reach (composite-key +tables, unsupported at the time). That's why the public corpus — not those early numbers — +is what the project publishes as its release figure, and why every run's coverage block is +non-negotiable regardless of which number gets quoted where. + +**The metric with zero tolerance stays separate from recall: zero confirmed false +positives.** See [Safety guarantees](../03-features-and-safety/safety-guarantees.md). + +## Full detail + +[`README.md` § How correctness is measured](../../../README.md#how-correctness-is-measured) +has the full corpus table, per-schema breakdown, and the "what the remaining gap is" +analysis. diff --git a/docs/guide/05-positioning-and-metrics/prior-art.md b/docs/guide/05-positioning-and-metrics/prior-art.md new file mode 100644 index 0000000..dbe7ee5 --- /dev/null +++ b/docs/guide/05-positioning-and-metrics/prior-art.md @@ -0,0 +1,35 @@ +# Prior art + +← [Guide](../README.md) › [Positioning & Metrics](README.md) + +> `pgfathom` is not new science, and says so. + +The full text lives in the root [`README.md` § Prior art](../../../README.md#prior-art) — +this page is the map of what it covers, not a copy of it. + +## The shape of the argument + +1. **Read the ORM first, if you can.** Rails' `schema.rb`, Django's `models.py`, a JPA + entity — when application source is available, it states the intended relationship in + one place, no inference needed. `pgfathom` doesn't compete with that. +2. **What source doesn't tell you** is whether that intent ever became an enforced + constraint, and whether the data still obeys it. `pgfathom` starts exactly where a + model file's authority ends. +3. **Academic data-profiling literature is mature** — containment is formally an + *inclusion dependency*, with published algorithms (SPIDER, BINDER, MIND) implemented in + the Metanome framework. `pgfathom`'s niche is engineering and product, not algorithmic + invention: Metanome doesn't speak PostgreSQL, doesn't read `pg_catalog`, doesn't + generate DDL. +4. **Partial overlap exists in open source** (Azimutt flags undeclared `_id` columns) and + **in commercial GUI tools** (Hackolade, Oracle Data Modeler) — both metadata-only, + never validated against real data. +5. **The actual boundary:** not inferring — **validating the inference against real data + and reporting the uncertainty honestly**. A view that joins two columns is proof no + naming heuristic could ever produce. +6. **Deliberately doesn't compete** with Squawk (migration linting), Atlas (drift), sqlc/ + jOOQ/Ent/Prisma (code generation from schema) — the versioned JSON model is the + integration point for those instead. + +## Full text + +[`README.md` § Prior art](../../../README.md#prior-art). diff --git a/docs/guide/06-contributing/README.md b/docs/guide/06-contributing/README.md new file mode 100644 index 0000000..8f1a242 --- /dev/null +++ b/docs/guide/06-contributing/README.md @@ -0,0 +1,43 @@ +# 06 · Contributing + +← [Guide](../README.md) + +The process itself lives in [`CONTRIBUTING.md`](../../../CONTRIBUTING.md) — this page is +the shortest path into it, not a replacement. + +## The shortest path to a first PR + +1. **Read [`CONTRIBUTING.md`](../../../CONTRIBUTING.md)** — in particular the + [language convention](../../../CONTRIBUTING.md#language) (English in the tree, Portuguese + in commit messages and the design record) and the rule that decides what needs an issue + first: docs, bug fixes, refactors and tests go straight to a PR; new or changed behavior + needs an issue to agree on shape before it gets an `openspec/changes/` proposal. +2. **Pick something.** A [naming profile](../03-features-and-safety/naming-profiles.md) + for a language `pgfathom` doesn't speak yet is the easiest entry point in the project — + it needs no knowledge of the rest of the codebase, just a TOML file and a table of test + cases. Otherwise, check the + [open issues](https://github.com/lvcas-dotcom/pgfathom/issues) on GitHub. +3. **If it's new behavior**, see how a real proposal looks in + [`openspec/changes/archive/`](../../../openspec/changes/archive/) before writing your + own — `openspec validate --strict` has to pass before it merges. + +## The five rules every PR is evaluated against, no exceptions + +See [Safety guarantees](../03-features-and-safety/safety-guarantees.md) for the full +detail behind each one: + +1. Read-only, absolute +2. User data never leaves +3. No claim without evidence +4. Silence is never reported as a clean bill of health +5. Zero confirmed false positives + +## Before opening a PR + +```sh +make fmt && make lint && make test +``` + +`make test` is what CI gates every PR on; `make test-integration` and `make benchmark` +need Docker (and, for the latter, `psql`) and aren't required for a docs-only or +metadata-only change. diff --git a/docs/guide/README.md b/docs/guide/README.md new file mode 100644 index 0000000..ece5c76 --- /dev/null +++ b/docs/guide/README.md @@ -0,0 +1,27 @@ +# Contributor guide + +This is the map. `pgfathom` already has a single source of truth for behavior — +[`docs/PGFATHOM.md`](../PGFATHOM.md) (the design record) and +[`openspec/specs/`](../../openspec/specs/) (the formal specs, one per layer). Both are +authoritative, and both are written in Portuguese — the maintainer's language for design +reasoning, per [`CONTRIBUTING.md`](../../CONTRIBUTING.md#language). + +This guide doesn't replace either. It's the orientation layer for someone who has never +read either one: what to read first, in what order, and why each piece exists — in +English, since that's the project's convention for anything meant to reach outside +contributors. Every page here links back to the canonical source for the full detail; if +the two ever disagree, the canonical source wins. + +## Sections + +| # | Section | What it covers | +|---|---|---| +| 01 | [Overview](01-overview/README.md) | Why the tool exists, what it does end to end | +| 02 | [Architecture](02-architecture/README.md) | The layers, the core Go types, the 7-stage pipeline | +| 03 | [Features & Safety](03-features-and-safety/README.md) | Naming profiles, the non-negotiable rules, known failure modes | +| 04 | [Usage](04-usage/README.md) | Install, first run, flag reference | +| 05 | [Positioning & Metrics](05-positioning-and-metrics/README.md) | Where this sits versus other tools, and what the recall numbers actually mean | +| 06 | [Contributing](06-contributing/README.md) | Where to start, and how a change gets proposed | + +New to the project? Start at [01 · Overview](01-overview/README.md). Ready to send a +patch? Skip to [06 · Contributing](06-contributing/README.md).