diff --git a/README.md b/README.md index 27619ee..a522d3d 100644 --- a/README.md +++ b/README.md @@ -7,8 +7,274 @@ [![Rust 1.82+](https://img.shields.io/badge/rust-1.82%2B-orange.svg)](https://www.rust-lang.org/) I built queryforge to understand what actually happens between typing a SQL -query and getting rows back — by writing every stage of it in Rust: the lexer, +query and getting rows back, by writing every stage of it in Rust: the lexer, the parser, the binder, a cost-based optimiser, a vectorised execution engine, and the columnar file format underneath all of it. -This repository is in progress; the sections below fill in as each layer lands. +**Nothing is delegated.** There is no `sqlparser`, no DataFusion, no Arrow, no +Parquet — in fact no external crates at all. The whole workspace builds against +the standard library, which is deliberate: a query engine that wraps someone +else's parser and someone else's columnar format has skipped the two places +where the interesting decisions live. + +```sql +qf> SELECT region, count(*) AS orders, sum(amount) AS revenue + FROM orders GROUP BY region HAVING count(*) > 2 ORDER BY revenue DESC; +┌────────┬────────┬─────────┐ +│ region │ orders │ revenue │ +├────────┼────────┼─────────┤ +│ us │ 4 │ 615.25 │ +│ eu │ 4 │ 484.49 │ +└────────┴────────┴─────────┘ +2 rows +``` + +## Try it + +```bash +cargo run --release -- examples/tour.sql +``` + +That walks the whole feature set on ten rows of orders: type inference from +CSV, filters and NULL semantics, aggregation, joins, `EXPLAIN`, and writing the +table out to the columnar format and querying it back. + +Or start the shell: + +```bash +cargo run --release +``` + +``` +qf> \import orders examples/orders.csv +imported 10 rows into orders +qf> \d orders +orders (10 rows) + id INT64 NOT NULL min=1 max=10 nulls=0 distinct≈10 + customer_id INT64 NOT NULL min=10 max=50 nulls=0 distinct≈5 + region UTF8 NOT NULL min=apac max=us nulls=0 distinct≈3 + status UTF8 NOT NULL min=cancelled max=shipped nulls=0 distinct≈4 + amount FLOAT64 NULL min=45.25 max=510.0 nulls=1 distinct≈9 +qf> EXPLAIN SELECT id FROM orders WHERE amount > 200 AND region = 'eu'; +Projection: id#0 + Scan: orders [id, region, amount], pushed: (amount#2 > 200) AND (region#1 = 'eu') +``` + +The filter is not a node above the scan; it is *inside* it, which is what makes +the next section possible. + +## What it does, and what that buys + +### The columnar format + +`.qfc` is my own. Column chunks are grouped into row groups; a footer at the end +records, for every chunk, its byte range, its encoding, and a zone map — the +min, max, null count and distinct count of the values it holds. + +``` +┌────────┬──────────────────────────┬─────────┬────────────┬────────┐ +│ "QFC1" │ row group 0..n │ footer │ footer len │ "QFC1" │ +│ 4 B │ column chunks, back to │ metadata│ u32 LE │ 4 B │ +│ │ back, one per column │ │ │ │ +└────────┴──────────────────────────┴─────────┴────────────┴────────┘ +``` + +The footer's length is the last field before the trailing magic, so a reader +seeks to `len - 8` and learns the entire layout without touching data. That is +what lets a query over two of forty columns read two chunks per row group, and +a query whose predicate falls outside a row group's `[min, max]` read none of +it at all. + +Each chunk picks its own encoding by measuring the data rather than guessing +from its type, because the right answer changes between row groups of the same +column: long runs go RLE, low cardinality goes dictionary, everything else stays +plain. On the tests' own fixtures RLE is over 10x smaller than plain on a sorted +column, and a dictionary is 4x smaller on a shuffled four-value string column. + +Strings use one contiguous byte buffer plus offsets rather than a `String` per +row, and nulls are a packed bitmap that a column without nulls does not carry +at all. + +### The optimiser + +Four rules, each of which must preserve both the rows a plan produces and the +schema it produces them in. The second half is what makes them composable, and +there is a test asserting it across a spread of queries. + +| Rule | What it does | +| --- | --- | +| Constant folding | Evaluates what is already known, applies the boolean identities, drops dead `CASE` branches | +| Predicate pushdown | Splits conjunctions so each part travels separately, ending inside the scan | +| Join reordering | Puts the smallest relation at the bottom of an inner-join chain | +| Projection pushdown | Narrows every scan to the columns actually read | + +Some decisions I'd defend in a review: + +- **Folding stops at division by zero and integer overflow.** Folding those + would let the plan-time answer differ from the run-time one, which is worse + than not folding. +- **A predicate is never pushed into the padded side of an outer join.** Pushing + `c.region = 3` into the right side of a `LEFT JOIN` deletes rows that should + have come back NULL-padded. There is a test per join type. +- **A comma join with an equality in `WHERE` becomes a real inner join.** + `FROM a, b WHERE a.id = b.id` is a hash join, not a cross product that gets + filtered afterwards. +- **Join reordering prefers a relation that has a join key over a smaller + unrelated one.** An accidental cross product is not something a later choice + recovers from. Reordering permutes the output columns, so the rule wraps the + result in a projection restoring the original order — the node above cannot + tell. + +### Execution + +Operators pull batches from each other. That is not a style preference: it is +what lets `LIMIT 10` stop a scan after the first row group instead of running it +to completion and discarding the rest, and a test asserts exactly that. + +Expression evaluation is vectorised — the operator is resolved once, outside the +loop, and the loop runs over a contiguous buffer. Comparing a column against a +literal has its own path so no constant column is materialised; same-type column +pairs skip boxing through a scalar type. + +Sorting has three modes. Feeding a `LIMIT` it keeps only the best *k* rows, so +`ORDER BY ts DESC LIMIT 10` is bounded by 10 rather than by the table. Fitting in +memory it sorts once. Otherwise it writes sorted runs to spill files and merges +them — the only reason a database sorts differently from `Vec::sort`. A test runs +the same data through both paths and asserts identical output, because the answer +must not depend on whether the input fitted in memory. + +The hash join builds one side and streams the other. For inner and cross joins +the smaller estimate is built, since the build side is what has to fit in memory. +For outer joins the choice is *forced*: the side that may be NULL-padded has to +be the one that can be scanned for non-matches at the end. + +### NULL semantics + +Three-valued logic is implemented properly, because getting it wrong silently +changes which rows come back rather than failing loudly: + +- `false AND NULL` is `false` — the row cannot match whatever the unknown turns + out to be. `true OR NULL` is `true`. +- A predicate that evaluates to unknown does not select its row. +- `5 IN (1, NULL)` is unknown, not false: the NULL might have been 5. +- A NULL join key never matches, including against another NULL. +- `sum` over no rows is NULL, not zero, and `count(*)` counts rows a + `count(column)` would skip. +- `x / 0` yields NULL rather than aborting — one bad row should not kill a scan + of a million. + +## What the optimiser is worth + +`queryforge bench` builds a table, runs every query twice — once on the bound +plan, once on the optimised one — and fails outright if the two disagree on the +row count. A flattering speedup from a wrong answer is worse than no number. + +500,000 rows, 8192-row row groups, release build: + +``` +query rows optimised speedup unopt. row groups +---------------------------------------------------------------------------------- +selective range 9999 0.48ms 154.6x 74.72ms 3 read 59 pruned +point lookup 1 0.65ms 120.1x 78.47ms 1 read 61 pruned +narrow projection 1 18.73ms 4.5x 85.20ms 62 read 0 pruned +group by 4 83.59ms 1.4x 119.67ms 62 read 0 pruned +filtered group by 4 41.94ms 1.9x 81.29ms 62 read 0 pruned +order by with limit 10 55.32ms 2.4x 131.88ms 62 read 0 pruned +join 4999 1.22ms 123.6x 151.20ms 2 read 60 pruned +``` + +The three-figure numbers are zone-map pruning: reading 3 row groups instead of +62. The 4.5x on a full aggregate is projection pushdown on its own — one column +chunk per row group instead of four. Reproduce with `cargo run --release -- +bench`. + +`EXPLAIN ANALYZE` reports the same counters per operator: + +``` +qf> EXPLAIN ANALYZE SELECT id FROM events WHERE id > 490000; +Projection (rows=9999, time=0.00ms) + Scan (rows=9999, time=0.91ms, row groups=1 read, 7 pruned) + +9999 rows returned +``` + +(Eight row groups rather than sixty-two because `\save` uses the default +65,536-row groups; the benchmark writes smaller ones so pruning is finer.) + +## Supported SQL + +`SELECT [DISTINCT]` with `*`, `t.*` and aliases · `FROM` with inner, left, +right, full and cross joins · `WHERE` · `GROUP BY` (expressions or output +positions) · `HAVING` · `ORDER BY` with `ASC`/`DESC` and `NULLS FIRST|LAST` · +`LIMIT`/`OFFSET`. + +Expressions: arithmetic, comparison, `AND`/`OR`/`NOT`, `IS [NOT] NULL`, +`[NOT] BETWEEN`, `[NOT] IN`, `[NOT] LIKE` (with `%` and `_`), `CAST`, `CASE` in +both forms, and the aggregates `count`, `sum`, `min`, `max`, `avg` — each with +`DISTINCT`. + +Statements: `CREATE TABLE`, `INSERT`, `COPY t FROM 'file.csv'`, `DROP TABLE`, +`EXPLAIN`, `EXPLAIN ANALYZE`. + +**Not supported**, and I would rather say so than let you find out: subqueries, +CTEs, window functions, `UNION`, `UPDATE`/`DELETE`, transactions, indexes, and +concurrency. The type system is four types wide — `BOOLEAN`, `INT64`, +`FLOAT64`, `UTF8` — because every extra type multiplies out across the array +representations, the encodings, the comparison kernels and the accumulators, and +none of that would teach anything the existing four don't. + +## How it's laid out + +Six crates, split along the stages a query passes through: + +| Crate | What lives there | +| --- | --- | +| `qf-common` | Types, values, schemas, and the error type tagged by the layer that raised it | +| `qf-storage` | Arrays, bitmaps, batches, encodings, the `.qfc` format, CSV ingest, the catalog | +| `qf-sql` | Lexer, syntax tree, parser | +| `qf-plan` | Binder, logical plan, cost model, optimiser | +| `qf-exec` | Vectorised evaluation, the operators, the physical planner, the session | +| `qf-cli` | Shell, result rendering, benchmark harness | + +The dependency direction is one-way, and the split is the argument: `qf-sql` has +no idea what a table is, `qf-plan` has no idea how a batch is laid out in +memory, and `qf-exec` never resolves a name. + +## Building and testing + +```bash +cargo test --workspace # 542 tests +cargo clippy --workspace --all-targets -- -D warnings +cargo run --release -- examples/tour.sql +cargo run --release -- bench --rows 500000 +``` + +CI runs formatting, clippy with warnings denied, the full suite and coverage on +every push, and **fails the build if any test is ignored** — a test that quietly +stops running keeps a job green while covering nothing. + +Coverage sits at roughly 97% of lines. The one exclusion is the REPL's terminal +loop, which is read-a-line-print-a-line plumbing; everything worth testing about +it lives in `shell.rs`, which the tests drive directly. + +## Further reading + +- [docs/file-format.md](docs/file-format.md) — the `.qfc` format byte by byte, + the zone-map pruning rules, and how each chunk's encoding is chosen. +- [docs/query-lifecycle.md](docs/query-lifecycle.md) — one query traced through + all six stages, with the real plan and counters at each step. + +## Things I got wrong along the way + +Worth recording, because they are the parts I actually learned from: + +- **A `LEFT JOIN` with a non-equality in its `ON` clause dropped rows.** The + condition was being applied *after* deciding which probe rows had matched, so + a preserved row whose only candidate failed the condition vanished instead of + coming back NULL-padded. Forty-three end-to-end tests missed it because every + join test either had no residual condition or was an inner join. Running the + example script found it in a minute. +- **A projection producing no columns dropped its row count**, which would have + made `SELECT count(*)` return zero once column pruning removed every column. +- **A comment line above a statement swallowed it**, because the script reader + buffered the comment and then read the next line as a continuation. diff --git a/docs/file-format.md b/docs/file-format.md new file mode 100644 index 0000000..859213a --- /dev/null +++ b/docs/file-format.md @@ -0,0 +1,159 @@ +# The `.qfc` file format + +This is the format queryforge reads and writes. It exists so that predicate and +projection pushdown can be physical rather than cosmetic: a reader should be +able to decide what it needs from the metadata alone and then touch only those +bytes. + +## Layout + +``` +┌────────┬──────────────────────────┬─────────┬────────────┬────────┐ +│ "QFC1" │ row group 0..n │ footer │ footer len │ "QFC1" │ +│ 4 B │ column chunks, back to │ metadata│ u32 LE │ 4 B │ +│ │ back, one per column │ │ │ │ +└────────┴──────────────────────────┴─────────┴────────────┴────────┘ +``` + +The footer is at the *end*, and its length is the last field before the trailing +magic. A reader seeks to `len - 8`, reads eight bytes, learns where the metadata +starts, and reads it in one more seek — without touching any column data. The +leading magic is checked too, so a file that is not a `.qfc` fails immediately +rather than producing nonsense. + +Every integer that is not fixed-width is a LEB128 varint, zigzagged when signed. +Most of what the format writes is small — run lengths, dictionary indices, +string lengths — and a varint spends one byte on those instead of eight. + +## Footer + +``` +uvarint field_count + for each field: + bytes name + u8 type tag 0=BOOLEAN 1=INT64 2=FLOAT64 3=UTF8 + u8 nullable +uvarint row_group_count + for each row group: + uvarint num_rows + uvarint column_count (must equal field_count) + for each column chunk: + u64 offset absolute, from the start of the file + u64 length + u8 encoding tag 0=plain 1=dictionary 2=RLE + +``` + +A footer whose column count disagrees with its schema is rejected rather than +read partially — a truncated or corrupt file is an expected failure mode, not a +bug, and every accessor in the reader is bounds-checked. + +### Column statistics — the zone map + +``` +uvarint null_count +uvarint row_count +uvarint distinct_count + min + max +``` + +A value is a one-byte tag followed by its payload: `0` NULL, `1` boolean (one +byte), `2` int64 (zigzag varint), `3` float64 (8 bytes LE), `4` string (length +then bytes). + +`min`, `max`, `row_count` and `null_count` are exact. `distinct_count` is exact +per row group and an **upper bound** once merged across row groups, because two +groups can hold overlapping values and the footer cannot say which. The planner +uses it only to order joins, so an over-estimate costs a plan rather than an +answer; the code says so where it merges. + +This is the structure that makes pruning work. Before reading a row group the +scan asks each pushed predicate whether the group's `[min, max]` could possibly +satisfy it. The answer is only ever used to *skip* work, so it errs toward +reading: anything undecidable comes back "may match". + +| Predicate | Prunes when | +| --- | --- | +| `col = v` | `v` falls outside `[min, max]` | +| `col < v` | `min >= v` | +| `col <= v` | `min > v` | +| `col > v` | `max <= v` | +| `col >= v` | `max < v` | +| `col <> v` | the group is a single value equal to `v`, with no nulls | +| anything, with `v` NULL | always — a comparison with NULL is never true | +| anything, group all NULL | always | + +A conjunction prunes when *either* half rules the group out; a disjunction only +when *both* do. + +## Column chunks + +``` +u8 encoding tag +uvarint num_rows +u8 has_validity + if 1: uvarint word_count, then word_count × u64 LE + only the non-null values +``` + +Only non-null values are written. A column that is 90% null costs a validity +bitmap and almost nothing else; a column with no nulls at all carries no bitmap. + +### Plain + +Values back to back: booleans bit-packed, int64 as zigzag varints, float64 as +8 bytes little-endian, strings as length-prefixed bytes. + +### Dictionary + +``` +uvarint dict_len + dict_len values, plain-encoded + one uvarint index per non-null row +``` + +A dictionary code pointing past the end of its dictionary is rejected. + +### Run-length + +``` +uvarint run_count + for each run: + plain-encoded + uvarint run length +``` + +A run that would describe more rows than the header declares is rejected, and so +is a chunk whose runs describe fewer rows than the validity mask expects — a +short chunk must fail rather than be silently padded. + +### Choosing between them + +The encoding is chosen per chunk, not per column, because the right answer +changes with the data: an `order_status` column can hold two distinct values in +one row group and forty in the next. Two cheap signals decide it, measured on +the chunk itself: + +- average run length ≥ 4 → **RLE** +- otherwise, distinct/rows ≤ 0.5 → **dictionary** +- otherwise → **plain** + +Chunks under 16 rows skip the analysis entirely. Both thresholds are +conservative, because each encoding adds a decode step and a chunk that would +barely benefit is better off plain. + +On the test fixtures: RLE is more than 10x smaller than plain on a sorted +column, and a dictionary is 4x smaller than plain on a shuffled four-value +string column. Those ratios are asserted in the test suite, not estimated. + +## Row groups + +The default is 65,536 rows. This is the granularity of pruning, and it is a +trade: smaller groups prune more precisely but multiply the footer, and past a +point the metadata costs more than the reads it saves. The benchmark writes +8,192-row groups so pruning is visible at a size that fits on a page. + +Batches handed to the writer are repacked into even row groups regardless of how +they arrived, so the file's layout does not depend on how the data happened to +be batched upstream. diff --git a/docs/query-lifecycle.md b/docs/query-lifecycle.md new file mode 100644 index 0000000..98ec619 --- /dev/null +++ b/docs/query-lifecycle.md @@ -0,0 +1,156 @@ +# The life of a query + +One query, traced through all six stages. Everything below is real output from +`examples/tour.sql` and the shell. + +```sql +SELECT c.name, count(*) AS orders +FROM orders o +JOIN customers c ON c.id = o.customer_id +WHERE o.amount > 200 +GROUP BY c.name +ORDER BY orders DESC +LIMIT 5; +``` + +## 1. Lexing — `qf-sql/src/lexer.rs` + +Text becomes tokens, each carrying the byte offset it started at so that a +later error can point at the text that caused it. Keywords are upper-cased so +nothing downstream needs a case-insensitive compare; identifiers keep the case +they were written in. + +The cases that separate this from a toy tokeniser: `"quoted identifiers"` that +may hold keywords or spaces, doubled quotes as escapes inside both strings and +identifiers, `--` comments, exponent notation, and an integer too large for +`i64` falling back to a float rather than failing the query. + +## 2. Parsing — `qf-sql/src/parser.rs` + +Recursive descent for statements, precedence climbing for expressions. The tree +is purely *syntactic*: a column is a name the user typed, not a position, and a +function is a name, not an aggregate. That is what lets the next stage be the +only place that resolves anything. + +Two precedence decisions are worth arguing about, and both have tests: + +- `NOT` sits between `AND` and comparison, so `NOT a = 1` groups as + `NOT (a = 1)` and `NOT a AND b` as `(NOT a) AND b`. +- `BETWEEN` parses its bounds at comparison precedence, so the `AND` separating + them is not swallowed as a conjunction. `a BETWEEN 1 AND 5 AND b = 2` has to + come out as `(a BETWEEN 1 AND 5) AND (b = 2)`. + +## 3. Binding — `qf-plan/src/binder.rs` + +Names become positions and every node learns its type. This is the only place in +the engine that consults the catalog, which is why every "no such column" and +"ambiguous column" message comes from one function. + +What the binder decides here: + +- **`c.id` and `o.customer_id` resolve to positions** in the joined pair. An + unqualified name matching both tables is an error, not a first-match win. +- **The `ON` clause splits** into hash-join keys and a residual filter. Only an + equality with one side reading purely left columns and the other purely right + becomes a key — a hash join cannot build on anything else. Written `c.id = + o.customer_id`, the sides are swapped so the left key really is the left + input's. +- **`count(*)` is registered once** and referenced by both the projection and + the `ORDER BY`, so it is computed once. +- **`ORDER BY orders`** resolves the alias back to the aggregate it names. +- **`c.name` must be in `GROUP BY`** — and it is. A bare column that is neither + grouped nor aggregated is rejected rather than returning an arbitrary row. +- **`BETWEEN` and `CASE x WHEN …` desugar** into comparisons, so the optimiser + and the evaluator never see them. + +The result is a tree of `LogicalPlan` nodes, each of which can compute its own +output schema. + +## 4. Optimising — `qf-plan/src/optimizer.rs` + +Four rules, each of which must preserve the rows *and* the schema. The schema +half is what makes them composable — a rule may move a filter or reorder a +join, but the node above must not be able to tell. + +``` +qf> EXPLAIN SELECT id FROM orders WHERE amount > 200 AND region = 'eu'; +Projection: id#0 + Scan: orders [id, region, amount], pushed: (amount#2 > 200) AND (region#1 = 'eu') +``` + +Note what is *not* in that plan: there is no `Filter` node. The predicate ended +up inside the scan, where a zone map can act on it, and the scan reads three of +five columns. + +1. **Constant folding** — evaluate what is known, apply the boolean identities, + drop dead `CASE` branches. It deliberately stops at division by zero and + integer overflow: folding those would let the plan-time answer differ from + the run-time one. +2. **Predicate pushdown** — split conjunctions so each part travels separately; + rewrite through column-only projections; push each part into the side of the + join it belongs to; end inside the scan. Never into the padded side of an + outer join. A comma join with an equality in `WHERE` becomes a real inner + join here rather than a cross product that gets filtered afterwards. +3. **Join reordering** — put the smallest relation at the bottom of an + inner-join chain, preferring a relation that has a join key to what is + already joined over a smaller unrelated one. Reordering permutes the output + columns, so the rule wraps the result in a projection restoring the original + order. +4. **Projection pushdown** — narrow every scan to the columns actually read, and + renumber every expression above it. This is the part that has to be exactly + right: getting it wrong reads the wrong column and reports a plausible wrong + answer. + +Cardinality estimates come from the zone maps the storage layer already wrote, +so none of this costs an extra pass over the data. + +## 5. Physical planning — `qf-exec/src/physical.rs` + +The logical plan says *what*; this decides *how*. Two choices are made here +because both depend on physical facts the logical plan does not model: + +- **Which side of the join to build.** The build side is what has to fit in + memory, so the smaller estimate wins — `customers` here. On an outer join the + choice is forced instead: the side that may be NULL-padded has to be the one + that can be scanned for non-matches at the end. +- **Whether the sort can be a top-k.** The `LIMIT 5` is pushed into the sort as + a fetch hint, so it never holds more than five rows. Nothing about the logical + plan changes; the sort just stops keeping rows it can prove it will discard. + +## 6. Execution — `qf-exec/` + +Operators pull batches from each other. Pulling is what lets a `LIMIT` stop a +scan early rather than running it to completion and discarding the rest. + +- **Scan** walks row groups, asks each pushed predicate whether the group's zone + map could satisfy it, skips the chunks entirely if not, decodes only the + projected columns, and applies the predicates to what is left. +- **Hash join** collects `customers`, hashes it, and streams `orders` past it. + A NULL key never joins. +- **Hash aggregate** builds a table from group key to accumulators in one pass. + Groups come out in first-seen order, so results are reproducible run to run. +- **Sort** keeps the best five rows and nothing else. +- **Limit** takes them. + +`EXPLAIN ANALYZE` reports what each operator actually did: + +``` +qf> EXPLAIN ANALYZE SELECT id FROM events WHERE id > 490000; +Projection (rows=9999, time=0.00ms) + Scan (rows=9999, time=0.91ms, row groups=1 read, 7 pruned) + +9999 rows returned +``` + +Seven of eight row groups were never read. + +## Where each stage lives + +| Stage | Crate | Knows about | +| --- | --- | --- | +| Lexing, parsing | `qf-sql` | SQL text. Not tables, not types, not memory | +| Binding, optimising | `qf-plan` | Schemas and statistics. Not how a batch is laid out | +| Physical planning, execution | `qf-exec` | Batches and operators. Never resolves a name | +| Storage | `qf-storage` | Bytes, encodings, zone maps. No idea SQL exists | + +The dependency direction is one-way, and the split is the argument.