From c27ded5ed773d6980db2a1a2f78c0a3802de995b Mon Sep 17 00:00:00 2001 From: Gordon Hamilton Date: Tue, 30 Jun 2026 19:52:41 -0400 Subject: [PATCH 1/6] feat: adding architecture docs --- ARCHITECTURE.md | 93 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 ARCHITECTURE.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 00000000..7b51503e --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,93 @@ +# Architecture + +This document describes the structure and design of `substrait-explain` for contributors. It attempts to explains what each module does, how data flows through the system, and why the code is shaped the way it is. For the user-facing text format, see `GRAMMAR.md`. For design philosophy and compatibility expectations, see `DESIGN.md`. For contributor process, see `CONTRIBUTING.md`. + +--- + +## Input and Output Types + +`substrait-explain` is a bidirectional converter between two representations of a Substrait query plan: + +- **Text format** — the human-readable format defined in `GRAMMAR.md` +- **Substrait protobuf** — the `proto::Plan` type from the `substrait` crate + +The CLI (behind the `cli` feature flag) can accept JSON or YAML files. These are converted to `proto::Plan` by `src/json.rs` and `src/cli.rs` at the CLI boundary before anything else touches them. Internally, `proto::Plan` is the handoff point between the two directions — there is no shared IR that both the parser and textifier use. + +--- + +## Module Map + +``` +src/ +├── lib.rs # Public API — parse() and format*() entry points +├── main.rs # CLI binary +├── cli.rs # CLI argument handling +├── json.rs # JSON/YAML → proto::Plan (cli feature only) +├── grammar.rs # Re-exports GRAMMAR.md as rustdoc / doctests +├── parser/ # Text → proto::Plan +├── textify/ # proto::Plan → Text +└── extensions/ # Extension lookup, registry, argument types +``` +--- + +## `src/textify/` — Proto to Text + +### Textify Summary + +1. lib.rs — entry point, calls into textify +2. textify/plan.rs — resolves simple extension declarations, writes the Extensions section, then iterates relations +3. textify/foundation.rs — not a sequential step but the shared infrastructure everything below uses (Textify trait, Scope, error accumulation) +4. textify/rels.rs — writes each relation: header, then addenda, then children recursively +5. textify/expressions.rs — called from rels for Value::Expression and Value::AggregateFunction +6. textify/types.rs — called from expressions and rels for type annotations, names, anchors +7. textify/extensions.rs — called from rels for custom extension argument values (Value::ExtensionArgument, Value::ExtColumn) +8. textify/addenda.rs — called from rels for + Enh:, + Opt:, + Ext: lines, which call into extensions.rs for rendering the decoded args + +### Entry Point + +The public entry points in `lib.rs` are `format()`, `format_with_options()`, and `format_with_registry()`. All three ultimately call `format_with_registry`, which is the real implementation. `format()` and `format_with_options()` are convenience wrappers that supply a default empty `ExtensionRegistry`. The registry is always required even as a default, because advanced extensions; Enhancements(`+ Enh:`), Optimizations(`+ Opt:`), and extension relations(LeafRel, SingleRel, MultiRel), carry their payload as a `google.protobuf.Any` blob. The registry knows how to decode those blobs into readable text. + +### `textify/plan.rs` — Top-Level Output + +This file is responsible for the overall structure of the output text and writes the Extensions and Plan sections. + +### `textify/foundation.rs` — Shared Infrastructure + +This file defines the core shared infrastructure that every other textify file depends on: +The `Textify` trait is implemented by every type that can be rendered to text and describes how they should be written to text. +The `Scope` trait is the context object carried through every `textify` call. Carrying output options, extension registry, error accumulator etc.. + +### `textify/rels.rs` — Relation Output + +Substrait proto relation types (Read, Filter, Project...) are converted to a single internal `Relation` shape before rendering. +Relations are made up of: arguments, columns output mapping(Emit), addenda(advanced extensions), and children. + +**`Value`** is the universal type for anything that can appear in a relation's argument list or output columns. + +**`Arguments`** separates positional from named arguments. Positional arguments are rendered in order; named arguments render as key=value pairs. Relations use named arguments when field labels are needed for clarity — e.g. fetch=10, offset=5 — and positional when order alone is unambiguous. + +**`Emitted`** holds both the full direct-output column list and the `RelCommon` emit mapping. `Emitted` keeps that separation intact and applies the index remapping at render time. + +**`ValueEnum`** handles proto enum fields, which arrive as raw `i32` values from prost. + +### `textify/expressions.rs` — Expression Rendering + +Called from `rels.rs` to render `Value::Expression` and `Value::AggregateFunction`. Handles Substrait expression types. + +### `textify/types.rs` — Type and Name Rendering + +Called from `expressions.rs` and `rels.rs` whenever a type annotation, column name, or extension name needs to be rendered. + +### `textify/extensions.rs` — Extension Argument Rendering + +Called from `rels.rs` and `addenda.rs` to render advanced extension arguments. This file gives `ExtensionValue`, `ExtensionArgs`, `ExtensionColumn`, and `Expr` their `Textify` implementations. + +The key intermediate type is **`ExtensionArgs`** (defined in `src/extensions/args.rs`). When the textifier encounters an advanced extension payload — a `google.protobuf.Any` blob — the `ExtensionRegistry` decodes it into `(name, ExtensionArgs)`: a text-format name and a generic argument bag holding positional `ExtensionValue`s, named `ExtensionValue`s, and `ExtensionColumn`s for output columns. + +### `textify/addenda.rs` — Addendum Line Rendering + +Called from `rels.rs` after the relation header, before child relations. Addendum lines (`+ Enh:`, `+ Opt:`, `+ Ext:`) attach metadata to a relation without changing its structural position in the plan tree. + +--- + +*More sections to follow: `src/parser/`, `src/extensions/`.* From bd1fa686c173664dc0935bf1610b83cb8488434a Mon Sep 17 00:00:00 2001 From: Gordon Hamilton Date: Tue, 30 Jun 2026 20:55:51 -0400 Subject: [PATCH 2/6] feat: adding the parser files description to file --- ARCHITECTURE.md | 63 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 62 insertions(+), 1 deletion(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 7b51503e..efad4b24 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -90,4 +90,65 @@ Called from `rels.rs` after the relation header, before child relations. Addendu --- -*More sections to follow: `src/parser/`, `src/extensions/`.* +## `src/parser/` — Text to Proto + +The parser uses a PEG grammar (via [Pest](https://pest.rs)) to match text syntax; similar to regex. Each plan line is parsed individually by the grammar into a **`LineNode`**: either a **`RelationNode`** (a relation header line) or an **`Addendum`** (a `+ Enh:` / `+ Opt:` / `+ Ext:` line). `RelationNode` holds the raw grammar match for that line, its line number, and initially empty `addenda` and `children` lists. + +**`TreeBuilder`** then assembles these flat `LineNode`s into a tree by using each line's indentation depth. A line at depth 0 becomes a new root. A line at depth N is attached as a child of the most recently seen node at depth N-1. Addendum lines are attached to the relation immediately above them at one less indent level, and must appear before any child relations. The result is a `RelationNode` tree that mirrors the visual indentation of the input text. + +Once the tree is built, it is walked depth-first: each `RelationNode` is converted to a `Rel` proto by `relations.rs`, which calls into `expressions.rs` and `types.rs` for its arguments and type annotations. This two-phase approach — build the tree first, convert to proto second — is what allows the parser to construct properly nested `Rel` messages without backtracking. + +### Parser Summary + +1. `expression_grammar.pest` — PEG grammar rules; all text syntax is defined here +2. `lib.rs` — entry point, calls into parser +3. `parser/mod.rs` — re-exports types from the submodules +4. `parser/chunks.rs` — merges continuation lines into single logical chunks before grammar parsing +5. `parser/structural.rs` — builds a `RelationNode` tree from the chunked plan lines using indentation depth +6. `parser/extensions.rs` — parses simple extension declarations and addendum(advanced extension) lines +7. `parser/relations.rs` — converts each `RelationNode` into a `Rel` proto message +8. `parser/expressions.rs` — called from relations to parse expression arguments +9. `parser/types.rs` — called from expressions and relations to parse type annotations +10. `parser/common.rs` — shared parsing infrastructure and traits used throughout + +### Entry Point + +The public entry points in `lib.rs` are `parse()` and `parse_with_registry()`. Unlike the textify side where all format functions funnel into `format_with_registry`, here there are two independent paths. The parser processes input in two passes: the simple Extensions section first to build the anchor lookup tables, then the Plan section to convert relations. Every anchor referenced in a relation is already resolved by the time that relation is parsed. + +The `ExtensionRegistry` is needed for the same reason as in the textifier: advanced extension payloads (`+ Enh:`, `+ Opt:`, extension relations) are `google.protobuf.Any` blobs, and the registry knows which registered type to encode them into. In the textifier, a missing registration produces a soft error token in the output and continues formatting. In the parser, a missing registration is a hard `ParseError::UnregisteredExtension` — there is no way to produce a valid `Any` blob without the registered type. This is why `parse()` does not supply a default empty registry with a silent fallback. Plans with advanced extensions must use `parse_with_registry()`. + +### `parser/chunks.rs` — Line Grouping + +Operates before any grammar parsing. The text format allows long argument lists to continue on the next line if that line is indented one level deeper and prefixed with `- `. `ChunkCursor` merges these continuation lines back into a single logical chunk so that the grammar parser always sees complete, self-contained expressions. + +### `parser/structural.rs` — Section Routing and Tree Construction + +The main orchestrator for parsing. It reads the input section by section and routes each line accordingly: extension declaration lines go to `extensions.rs`, plan lines are parsed into `LineNode`s and placed into the `RelationNode` tree by `TreeBuilder`. Once all lines are consumed, the completed `RelationNode` trees are walked depth-first to produce the final `proto::Plan`. + +### `parser/extensions.rs` — Extension and Addendum Parsing + +Handles two distinct jobs. Parses the Extensions section: URN declarations and function/type/variation anchor assignments, producing a `SimpleExtensions` anchor table that the rest of the parser uses to resolve function, type, and variation anchors to their declared names. Second, it parses addendum lines (`+ Enh:`, `+ Opt:`, `+ Ext:`) that appear attached to relations in the Plan section, decoding their names and arguments through the `ExtensionRegistry`. + +### `parser/relations.rs` — Relation Construction + +Converts each `RelationNode` into a `Rel` proto message. Implements the conversion for supported relation types: Read, Filter, Project etc... Also handles emit mapping — reconstructing the `RelCommon` output column remapping from the text representation. + +### `parser/expressions.rs` — Expression Parsing + +Called from `relations.rs` to parse expression arguments into `proto::Expression` messages. Covers expression types. + +### `parser/types.rs` — Type Parsing + +Called from `expressions.rs` and `relations.rs` to parse type annotations into `proto::Type` messages. Handles simple types, compound types, precision types, nullability suffixes, type variations, and user-defined types. + +### `parser/common.rs` — Shared Infrastructure + +Defines two traits that all relation, expression, and type parsers implement — the parser-side equivalent of the `Textify` trait. + +**`ParsePair`** converts a pest grammar match into a Rust type for constructs whose meaning is fully determined by syntax — literals, operators, nullability suffixes — with no anchor resolution needed. + +**`ScopedParsePair`** does the same but takes a `&SimpleExtensions` context argument. It is used for constructs that require anchor lookups — function calls, type references, type variations — where the conversion can fail with a `MessageParseError` if the anchor is not in the table. + +### `expression_grammar.pest` — PEG Grammar + +The canonical grammar for the text format. Defines rules for relation headers, argument lists, all expression forms, type syntax, and extension arguments. All text parsing ultimately flows through Pest rules defined here; the rest of the parser translates the resulting `Pair` matches into proto types. From b8fda36af3b1935ceb69f5b21266a58fe38439d9 Mon Sep 17 00:00:00 2001 From: Gordon Hamilton Date: Wed, 1 Jul 2026 19:38:01 -0400 Subject: [PATCH 3/6] docs: adding parsing description --- ARCHITECTURE.md | 65 +++++++++++++++++++++++++++++-------------------- 1 file changed, 39 insertions(+), 26 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index efad4b24..2cbf5486 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -34,7 +34,7 @@ src/ ### Textify Summary -1. lib.rs — entry point, calls into textify +1. lib.rs — entry point, exposes the public API formatting 2. textify/plan.rs — resolves simple extension declarations, writes the Extensions section, then iterates relations 3. textify/foundation.rs — not a sequential step but the shared infrastructure everything below uses (Textify trait, Scope, error accumulation) 4. textify/rels.rs — writes each relation: header, then addenda, then children recursively @@ -60,7 +60,7 @@ The `Scope` trait is the context object carried through every `textify` call. Ca ### `textify/rels.rs` — Relation Output Substrait proto relation types (Read, Filter, Project...) are converted to a single internal `Relation` shape before rendering. -Relations are made up of: arguments, columns output mapping(Emit), addenda(advanced extensions), and children. +Proto relations can be represented in their arguments, columns output mapping(Emit), addenda(advanced extensions), and children; which we store in Relation internal type. **`Value`** is the universal type for anything that can appear in a relation's argument list or output columns. @@ -92,63 +92,76 @@ Called from `rels.rs` after the relation header, before child relations. Addendu ## `src/parser/` — Text to Proto -The parser uses a PEG grammar (via [Pest](https://pest.rs)) to match text syntax; similar to regex. Each plan line is parsed individually by the grammar into a **`LineNode`**: either a **`RelationNode`** (a relation header line) or an **`Addendum`** (a `+ Enh:` / `+ Opt:` / `+ Ext:` line). `RelationNode` holds the raw grammar match for that line, its line number, and initially empty `addenda` and `children` lists. +The parser converts text to `proto::Plan` in three phases, all orchestrated by `structural.rs`. -**`TreeBuilder`** then assembles these flat `LineNode`s into a tree by using each line's indentation depth. A line at depth 0 becomes a new root. A line at depth N is attached as a child of the most recently seen node at depth N-1. Addendum lines are attached to the relation immediately above them at one less indent level, and must appear before any child relations. The result is a `RelationNode` tree that mirrors the visual indentation of the input text. +**Phase 1 — Extensions**: `structural.rs` reads the `=== Extensions` section first, in full, building the `SimpleExtensions` anchor table. Every function, type, and variation anchor is resolved before any relation is parsed. -Once the tree is built, it is walked depth-first: each `RelationNode` is converted to a `Rel` proto by `relations.rs`, which calls into `expressions.rs` and `types.rs` for its arguments and type annotations. This two-phase approach — build the tree first, convert to proto second — is what allows the parser to construct properly nested `Rel` messages without backtracking. +**Phase 2 — Tree building**: `structural.rs` reads the `=== Plan` section line by line. For each line, it calls `ChunkCursor` (from `chunks.rs`) to merge any continuation lines into a single complete chunk. Each chunk is then run through the PEG grammar (via [Pest](https://pest.rs)), producing a **`Pair`**. + +A **`Pair`** is Pest's fundamental match type: it records which grammar rule matched, the text it covers, and a list of inner `Pair`s for sub-expressions within that match. For example, parsing `Read[my_table => a:i64, b:string?]` produces a `Pair` for the `read_relation` rule with two inner pairs: a `table_name` pair covering `my_table`, and a `named_column_list` pair covering `a:i64, b:string?`. That `named_column_list` pair itself contains two `named_column` inner pairs — one for `a:i64`, one for `b:string?` — each of which contains a `name` pair and a `type` pair. This inner-pair nesting captures the structure within a single line — the parent-child relationships between relations come from indentation, not the grammar. + +`structural.rs` wraps each top-level `Pair` into a **`LineNode`**: either a **`RelationNode`** (a relation line) or an **`Addendum`** (a `+ Enh:` / `+ Opt:` / `+ Ext:` line). `RelationNode` holds the pair, its line number, `addenda` and `children` lists. **`TreeBuilder`** then places each `LineNode` into the tree by indentation depth. + +**Phase 3 — Proto conversion**: once all lines are consumed, the completed `RelationNode` trees are walked depth-first, leaves first, then parents. This order is required because `RelationParsePair` (the trait each relation type implements) receives its children as already-converted `Rel` messages, and the field count flowing up from them, both of which are only available after the subtree below is resolved. ### Parser Summary -1. `expression_grammar.pest` — PEG grammar rules; all text syntax is defined here -2. `lib.rs` — entry point, calls into parser -3. `parser/mod.rs` — re-exports types from the submodules -4. `parser/chunks.rs` — merges continuation lines into single logical chunks before grammar parsing -5. `parser/structural.rs` — builds a `RelationNode` tree from the chunked plan lines using indentation depth -6. `parser/extensions.rs` — parses simple extension declarations and addendum(advanced extension) lines -7. `parser/relations.rs` — converts each `RelationNode` into a `Rel` proto message -8. `parser/expressions.rs` — called from relations to parse expression arguments -9. `parser/types.rs` — called from expressions and relations to parse type annotations -10. `parser/common.rs` — shared parsing infrastructure and traits used throughout +- `lib.rs` — entry point, exposes the public API for parsing +- `expression_grammar.pest` — defines the grammar rules every file below depends on +- `parser/mod.rs` — re-exports types from the submodules +- `parser/chunks.rs` — provides the mechanism for grouping physical lines into chunks +- `parser/structural.rs` — hub; orchestrates all three phases: extensions, tree building, proto conversion +- `parser/extensions.rs` — parses simple extension declarations and addendum lines +- `parser/relations.rs` — converts each `RelationNode` into a `Rel` proto message +- `parser/expressions.rs` — called from relations to parse expression arguments +- `parser/types.rs` — called from expressions and relations to parse type annotations +- `parser/common.rs` — shared parsing traits and infrastructure used throughout ### Entry Point -The public entry points in `lib.rs` are `parse()` and `parse_with_registry()`. Unlike the textify side where all format functions funnel into `format_with_registry`, here there are two independent paths. The parser processes input in two passes: the simple Extensions section first to build the anchor lookup tables, then the Plan section to convert relations. Every anchor referenced in a relation is already resolved by the time that relation is parsed. +The public entry points in `lib.rs` are `parse()` and `parse_with_registry()`. Unlike the textify side where `format()` and `format_with_options()` are thin wrappers that delegate to `format_with_registry`, here `parse()` does not delegate to `parse_with_registry()` — they are separate implementations. + +The parser carries an `ExtensionRegistry` for the same reason as the textifier: advanced extension payloads (`+ Enh:`, `+ Opt:`, `+ Ext:`, and extension relations) are `google.protobuf.Any` blobs. `parse()` uses a default empty registry through `Parser::new()`, while `parse_with_registry()` supplies a caller-provided registry. The difference is in error handling: when textifying, an unregistered advanced extension can be represented as a soft error token output. When parsing, an unregistered advanced extension is a hard `ParseError::UnregisteredExtension`, because the parser cannot create a valid `Any` payload without the registered type. Plans with advanced extensions therefore need `parse_with_registry()`. + +### `expression_grammar.pest` — PEG Grammar -The `ExtensionRegistry` is needed for the same reason as in the textifier: advanced extension payloads (`+ Enh:`, `+ Opt:`, extension relations) are `google.protobuf.Any` blobs, and the registry knows which registered type to encode them into. In the textifier, a missing registration produces a soft error token in the output and continues formatting. In the parser, a missing registration is a hard `ParseError::UnregisteredExtension` — there is no way to produce a valid `Any` blob without the registered type. This is why `parse()` does not supply a default empty registry with a silent fallback. Plans with advanced extensions must use `parse_with_registry()`. +The canonical grammar for the text format. Covers relation line syntax (one rule per relation type), argument lists, expression forms, etc... Section markers (`=== Plan`, `=== Extensions`) are not grammar rules — they are matched as string constants in `structural.rs`. ### `parser/chunks.rs` — Line Grouping -Operates before any grammar parsing. The text format allows long argument lists to continue on the next line if that line is indented one level deeper and prefixed with `- `. `ChunkCursor` merges these continuation lines back into a single logical chunk so that the grammar parser always sees complete, self-contained expressions. +`ChunkCursor` provides the mechanism for grouping physical lines from the text into chunks. The policy of what to merge — recognizing continuation lines prefixed with `- ` and deciding when a chunk is complete — belongs to `structural.rs`, which drives the cursor. ### `parser/structural.rs` — Section Routing and Tree Construction -The main orchestrator for parsing. It reads the input section by section and routes each line accordingly: extension declaration lines go to `extensions.rs`, plan lines are parsed into `LineNode`s and placed into the `RelationNode` tree by `TreeBuilder`. Once all lines are consumed, the completed `RelationNode` trees are walked depth-first to produce the final `proto::Plan`. +The main orchestrator. Matches `=== Extensions` and `=== Plan` as string constants to switch between sections, drives `ChunkCursor` for line grouping, routes extension declarations to `extensions.rs`, and builds the `RelationNode` tree via `TreeBuilder`. ### `parser/extensions.rs` — Extension and Addendum Parsing -Handles two distinct jobs. Parses the Extensions section: URN declarations and function/type/variation anchor assignments, producing a `SimpleExtensions` anchor table that the rest of the parser uses to resolve function, type, and variation anchors to their declared names. Second, it parses addendum lines (`+ Enh:`, `+ Opt:`, `+ Ext:`) that appear attached to relations in the Plan section, decoding their names and arguments through the `ExtensionRegistry`. +Handles two distinct jobs: parsing URN declarations and anchor assignments in the Extensions section to produce the `SimpleExtensions` table, and parsing addendum lines (`+ Enh:`, `+ Opt:`, `+ Ext:`) attached to plan relations, decoding their names and arguments through the `ExtensionRegistry`. ### `parser/relations.rs` — Relation Construction Converts each `RelationNode` into a `Rel` proto message. Implements the conversion for supported relation types: Read, Filter, Project etc... Also handles emit mapping — reconstructing the `RelCommon` output column remapping from the text representation. +Each relation type implements **`RelationParsePair`**, a third parsing trait alongside `ParsePair` and `ScopedParsePair`, which additionally accepts pre-converted child `Rel` messages and propagates output field counts up the tree. + +`RelationParsingContext` is the context object used when a relation or addendum must turn parsed `ExtensionArgs` into a `google.protobuf.Any` payload. It carries the `ExtensionRegistry` plus source location so registry failures become contextual `ParseError`s. + + ### `parser/expressions.rs` — Expression Parsing -Called from `relations.rs` to parse expression arguments into `proto::Expression` messages. Covers expression types. +Called from `relations.rs` to parse supported expression arguments types into `proto::Expression` messages. ### `parser/types.rs` — Type Parsing -Called from `expressions.rs` and `relations.rs` to parse type annotations into `proto::Type` messages. Handles simple types, compound types, precision types, nullability suffixes, type variations, and user-defined types. +Called from `expressions.rs` and `relations.rs` to parse type annotations into `proto::Type` messages. ### `parser/common.rs` — Shared Infrastructure -Defines two traits that all relation, expression, and type parsers implement — the parser-side equivalent of the `Textify` trait. +Defines the parsing traits used throughout the parser — the parser-side equivalent of the `Textify` trait. There are three, each handling a different level of context dependency. **`ParsePair`** converts a pest grammar match into a Rust type for constructs whose meaning is fully determined by syntax — literals, operators, nullability suffixes — with no anchor resolution needed. **`ScopedParsePair`** does the same but takes a `&SimpleExtensions` context argument. It is used for constructs that require anchor lookups — function calls, type references, type variations — where the conversion can fail with a `MessageParseError` if the anchor is not in the table. -### `expression_grammar.pest` — PEG Grammar - -The canonical grammar for the text format. Defines rules for relation headers, argument lists, all expression forms, type syntax, and extension arguments. All text parsing ultimately flows through Pest rules defined here; the rest of the parser translates the resulting `Pair` matches into proto types. +**`RuleIter`** wraps Pest's pair iterator with `try_pop` (consume the next pair only if it matches a specific rule, otherwise leave it) and `pop` (consume and assert). From 161070643ad5570e7afc4a55c580c83a4ed1eac1 Mon Sep 17 00:00:00 2001 From: "gordon.hamilton" Date: Thu, 2 Jul 2026 13:02:35 -0400 Subject: [PATCH 4/6] docs: updating extensions section --- ARCHITECTURE.md | 53 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 2cbf5486..0d55da40 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -30,6 +30,14 @@ src/ ``` --- +## `src/json.rs` — JSON to Proto + +Substrait plans can be encoded as JSON, but two incompatible JSON formats exist. Rust's pbjson library encodes `google.protobuf.Any` fields as `{"typeUrl": "...", "value": ""}`. Go's protojson library uses the protobuf standard encoding: `{"@type": "...", field1: val, ...}` where the concrete message's fields are inlined. A plan produced by a Go system will fail to parse with the Rust pbjson deserializer. `json.rs` handles both. It tries the pbjson serde path first; if that fails it falls back to `prost-reflect`, which implements the full protobuf JSON mapping spec and can handle the Go format as long as a `DescriptorPool` containing the schema for every referenced type URL is provided. Both paths produce the same `proto::Plan`. When outputting JSON, the CLI always produces the pbjson format (`{"typeUrl": "...", "value": ""}`) via `serde_json`. There is no option to emit Go-style protojson. + +The Go-format path requires a `DescriptorPool` with the schema for any extension types stored as `google.protobuf.Any` blobs — the Substrait core types are already bundled, but custom extension schemas must be provided. Callers do this by adding their compiled proto descriptor blob to the `ExtensionRegistry` via `add_descriptor`. When the CLI receives JSON input, it builds the pool from those registered blobs before calling `parse_json`. `build.rs` is an example of building the descriptor blob. It compiles the test proto at build time producing the binary descriptor and generated Rust types. + +--- + ## `src/textify/` — Proto to Text ### Textify Summary @@ -165,3 +173,48 @@ Defines the parsing traits used throughout the parser — the parser-side equiva **`ScopedParsePair`** does the same but takes a `&SimpleExtensions` context argument. It is used for constructs that require anchor lookups — function calls, type references, type variations — where the conversion can fail with a `MessageParseError` if the anchor is not in the table. **`RuleIter`** wraps Pest's pair iterator with `try_pop` (consume the next pair only if it matches a specific rule, otherwise leave it) and `pop` (consume and assert). + +--- + +## `src/extensions/` — Extension Support + +Both simple and advanced extension code is found in this folder. + +**Simple Extensions** are the Substrait standard mechanism for declaring custom functions, types, and type variations. In the text format these appear in the `=== Extensions` section as URN declarations and anchor assignments. `SimpleExtensions` is the in-memory lookup table built from those declarations — mapping integer anchors to qualified names and back. + +**Advanced Extensions** are custom relation types, enhancements (`+ Enh:`), optimizations (`+ Opt:`), and extension table reads (`+ Ext:`, an addendum attached to a `ReadRel`) that are stored as a `google.protobuf.Any` blob in the proto. `substrait-explain` cannot parse or textify them from the plan alone. The caller supplies an `ExtensionRegistry` with registered Rust types that knows both the text representation and the protobuf `Any` representation. + +**`AnyConvertible`** handles proto serialization. `to_any()` encodes the Rust type into a `google.protobuf.Any` blob; `from_any()` decodes it back to custom type. For prost-generated types (`prost::Message + prost::Name + Default`), this is provided automatically via a blanket impl. Custom types implement it manually. + +**`Explainable`** handles text format conversion. `to_args()` converts the Rust type into an `ExtensionArgs` for the textifier to render; `from_args()` constructs the type from parsed `ExtensionArgs`. + +A type registers as an extension by implementing both `AnyConvertible` and `Explainable`. **`Extension`** is an empty supertrait that groups them into a single bound used by the registry. + +Together, the textifier path of a registered type is: `AnyRef` → `AnyConvertible::from_any` → Rust type → `Explainable::to_args` → `ExtensionArgs`. The parse path is the reverse: `ExtensionArgs` → `Explainable::from_args` → Rust type → `AnyConvertible::to_any` → proto `Any` + +### File Map + +- `extensions/simple.rs` — `SimpleExtensions`: anchor to name lookup table for functions, types, and type variations +- `extensions/registry.rs` — `ExtensionRegistry`: user-provided handler registry for advanced extension payloads +- `extensions/args.rs` — `ExtensionArgs`: the structured intermediate type that bridges text format and proto blobs +- `extensions/any.rs` — `Any` / `AnyRef`: owned and borrowed wrappers around `google.protobuf.Any` +- `extensions/examples.rs` — example of `Explainable` implementations used in documentation and tests + +### `extensions/simple.rs` — Anchor Lookup + +`SimpleExtensions` is the anchor lookup table built from the `=== Extensions` section. It maps integer anchors to URN strings a +nd qualified names for functions, types, and type variations. + +### `extensions/registry.rs` — Advanced Extension Registry + +`ExtensionRegistry` maps registered types to handlers that know how to convert between their `google.protobuf.Any` blob and their text representation. + +### `extensions/args.rs` — Extension Arguments + +`ExtensionArgs` is the structured intermediate that extension handlers read and write — a collection of positional values, named values, and output column declarations. `ExtensionValue` covers the scalar and expression types that can appear as argument values. Extension relations that declare an output schema use `ExtensionColumn` to describe it, which converts to and from the `NamedStruct` proto type. + +Handlers read from `ExtensionArgs` via an `ArgsExtractor`, which tracks which arguments have been consumed. This enforces that no unexpected arguments are silently ignored — the extractor errors if unconsumed named arguments remain after parsing, catching mismatches between what the text format provides and what the handler expects. + +### `extensions/any.rs` — Any Wrapper + +`prost_types::Any` is always owned — there is no borrowed form. When the textifier encounters an extension blob inside a proto struct it only has a reference to it, not ownership. Passing that blob to `from_any` without a local borrowed type would require cloning it every time. `AnyRef<'a>` solves this without cloning. Created from a reference to a `prost_types::Any`. `Any` (owned) is used when ownership is required, such as the return type of `to_any`. Using these local types in the `AnyConvertible` API also means extension implementors do not need to depend on prost directly. \ No newline at end of file From 2dbbcee2530d296195356efdb6380d27e6dcff00 Mon Sep 17 00:00:00 2001 From: "gordon.hamilton" Date: Thu, 2 Jul 2026 13:19:29 -0400 Subject: [PATCH 5/6] docs: adding testing explanation --- ARCHITECTURE.md | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 0d55da40..55e022af 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -24,10 +24,13 @@ src/ ├── cli.rs # CLI argument handling ├── json.rs # JSON/YAML → proto::Plan (cli feature only) ├── grammar.rs # Re-exports GRAMMAR.md as rustdoc / doctests +├── fixtures.rs # Shared test helpers (cfg(test) only) +├── types_tests.rs # Type roundtrip tests (cfg(test) only) ├── parser/ # Text → proto::Plan ├── textify/ # proto::Plan → Text └── extensions/ # Extension lookup, registry, argument types ``` + --- ## `src/json.rs` — JSON to Proto @@ -124,6 +127,7 @@ A **`Pair`** is Pest's fundamental match type: it records which grammar ru - `parser/expressions.rs` — called from relations to parse expression arguments - `parser/types.rs` — called from expressions and relations to parse type annotations - `parser/common.rs` — shared parsing traits and infrastructure used throughout +- `parser/errors.rs` — defines `ParseError`, `ParseContext`, and `ParseResult`: the public error types returned by `parse()` and `parse_with_registry()` ### Entry Point @@ -217,4 +221,33 @@ Handlers read from `ExtensionArgs` via an `ArgsExtractor`, which tracks which ar ### `extensions/any.rs` — Any Wrapper -`prost_types::Any` is always owned — there is no borrowed form. When the textifier encounters an extension blob inside a proto struct it only has a reference to it, not ownership. Passing that blob to `from_any` without a local borrowed type would require cloning it every time. `AnyRef<'a>` solves this without cloning. Created from a reference to a `prost_types::Any`. `Any` (owned) is used when ownership is required, such as the return type of `to_any`. Using these local types in the `AnyConvertible` API also means extension implementors do not need to depend on prost directly. \ No newline at end of file +`prost_types::Any` is always owned — there is no borrowed form. When the textifier encounters an extension blob inside a proto struct it only has a reference to it, not ownership. Passing that blob to `from_any` without a local borrowed type would require cloning it every time. `AnyRef<'a>` solves this without cloning. Created from a reference to a `prost_types::Any`. `Any` (owned) is used when ownership is required, such as the return type of `to_any`. Using these local types in the `AnyConvertible` API also means extension implementors do not need to depend on prost directly. + +--- + +## Testing + +The test suite is split between unit tests inside source files and integration tests under `tests/`. + +Unit tests live alongside the code they test following standard Rust convention. `src/fixtures.rs` provides `TestContext` and other helpers shared across many test modules. `types_tests.rs` contains type round-trip tests that span both the parser and textifier and has no single source file it naturally belongs to. They both need access to `pub(crate)` types, so they cannot be in `tests/`, which is compiled as a separate crate and cannot see crate-internal items. + +Integration tests under `tests/` are compiled as a separate crate and test the public API as an external caller would — they can only access what `lib.rs` exports. The primary strategy is round-trip: parse a text plan to proto, textify it back to text, and assert the output matches. This catches most correctness bugs without requiring proto-level assertions. + +- `plan_roundtrip.rs` — broad round-trip coverage across relation types and features +- `literal_roundtrip.rs` — round-trips for literal value parsing and formatting +- `multi_line_roundtrip.rs` — round-trips for multi-line `Read:Virtual` continuation syntax +- `extension_roundtrip.rs` — round-trips for custom extension handlers +- `adv_extension_roundtrip.rs` — round-trips for `+ Enh:` and `+ Opt:` advanced extension annotations +- `extension_table.rs` — round-trips for `Read:Extension` and `+ Ext:` extension table reads +- `json_parsing.rs` — exercises both JSON input formats (pbjson and Go protojson) against a real custom extension type + +`tests/common/mod.rs` provides shared helpers used across the integration test files — primarily `roundtrip_plan`, which parses a text plan and formats it back, asserting the output matches. + +## Examples and Reference Plans + +`examples/` contains runnable Cargo examples (`cargo run --example `) demonstrating the public API: +- `basic_usage.rs` covers parsing and formatting, +- `advanced_usage.rs` covers output options +- `extensions.rs` shows the full custom extension handler pattern end-to-end — defining a type, implementing `Explainable`, registering it, and round-tripping through the registry. + +`example-plans/` contains sample `.substrait` text files at different feature levels — basic relations, scalar functions, user-defined types. These are useful as manual test inputs and as reference when working on the parser or textifier. \ No newline at end of file From 103afe3e3ba63ae81fdac0f56438dddfc5cbaaa6 Mon Sep 17 00:00:00 2001 From: "gordon.hamilton" Date: Thu, 2 Jul 2026 14:18:56 -0400 Subject: [PATCH 6/6] docs: more edits --- ARCHITECTURE.md | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 55e022af..575b6d8c 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -11,7 +11,7 @@ This document describes the structure and design of `substrait-explain` for cont - **Text format** — the human-readable format defined in `GRAMMAR.md` - **Substrait protobuf** — the `proto::Plan` type from the `substrait` crate -The CLI (behind the `cli` feature flag) can accept JSON or YAML files. These are converted to `proto::Plan` by `src/json.rs` and `src/cli.rs` at the CLI boundary before anything else touches them. Internally, `proto::Plan` is the handoff point between the two directions — there is no shared IR that both the parser and textifier use. +The CLI(behind the `cli` feature flag) can accept text, JSON, protobuf, or YAML files. `cli.rs` routes each format and converts non-text input into `proto::Plan` at the boundary. JSON is handled in `json.rs`. Where we normalize Rust pbjson and Go protojson encodings to `proto::Plan`. Internally, `proto::Plan` is the handoff point between the two directions — there is no shared IR that both the parser and textifier use. --- @@ -21,8 +21,8 @@ The CLI (behind the `cli` feature flag) can accept JSON or YAML files. These are src/ ├── lib.rs # Public API — parse() and format*() entry points ├── main.rs # CLI binary -├── cli.rs # CLI argument handling -├── json.rs # JSON/YAML → proto::Plan (cli feature only) +├── cli.rs # CLI argument handling and routing for text, JSON, YAML, and protobuf formats +├── json.rs # JSON → proto::Plan (cli feature only), including pbjson and Go protojson Any handling ├── grammar.rs # Re-exports GRAMMAR.md as rustdoc / doctests ├── fixtures.rs # Shared test helpers (cfg(test) only) ├── types_tests.rs # Type roundtrip tests (cfg(test) only) @@ -37,12 +37,14 @@ src/ Substrait plans can be encoded as JSON, but two incompatible JSON formats exist. Rust's pbjson library encodes `google.protobuf.Any` fields as `{"typeUrl": "...", "value": ""}`. Go's protojson library uses the protobuf standard encoding: `{"@type": "...", field1: val, ...}` where the concrete message's fields are inlined. A plan produced by a Go system will fail to parse with the Rust pbjson deserializer. `json.rs` handles both. It tries the pbjson serde path first; if that fails it falls back to `prost-reflect`, which implements the full protobuf JSON mapping spec and can handle the Go format as long as a `DescriptorPool` containing the schema for every referenced type URL is provided. Both paths produce the same `proto::Plan`. When outputting JSON, the CLI always produces the pbjson format (`{"typeUrl": "...", "value": ""}`) via `serde_json`. There is no option to emit Go-style protojson. -The Go-format path requires a `DescriptorPool` with the schema for any extension types stored as `google.protobuf.Any` blobs — the Substrait core types are already bundled, but custom extension schemas must be provided. Callers do this by adding their compiled proto descriptor blob to the `ExtensionRegistry` via `add_descriptor`. When the CLI receives JSON input, it builds the pool from those registered blobs before calling `parse_json`. `build.rs` is an example of building the descriptor blob. It compiles the test proto at build time producing the binary descriptor and generated Rust types. +The Go-format path requires a `DescriptorPool` with the schema for any extension types stored as `google.protobuf.Any` blobs — the Substrait core types are already bundled, but custom extension schemas must be provided. Callers do this by adding their compiled proto descriptor blob to the `ExtensionRegistry` via `add_descriptor`. When the CLI receives JSON input, it builds the pool from those registered blobs before calling `parse_json`. `build.rs` demonstrates this pattern: it compiles a test proto at build time to produce the descriptor blob and generated Rust types used in tests. --- ## `src/textify/` — Proto to Text +Textifying means to generate substrait-explain text from a plan. A plan shouldn't fail to produce output for unsupported proto. The textifier writes what it can and pushes problems onto a shared ErrorQueue (textify/foundation.rs), which the caller can inspect after the fact (FormatError, PlanError). + ### Textify Summary 1. lib.rs — entry point, exposes the public API formatting @@ -56,7 +58,7 @@ The Go-format path requires a `DescriptorPool` with the schema for any extension ### Entry Point -The public entry points in `lib.rs` are `format()`, `format_with_options()`, and `format_with_registry()`. All three ultimately call `format_with_registry`, which is the real implementation. `format()` and `format_with_options()` are convenience wrappers that supply a default empty `ExtensionRegistry`. The registry is always required even as a default, because advanced extensions; Enhancements(`+ Enh:`), Optimizations(`+ Opt:`), and extension relations(LeafRel, SingleRel, MultiRel), carry their payload as a `google.protobuf.Any` blob. The registry knows how to decode those blobs into readable text. +The public entry points in `lib.rs` are `format()`, `format_with_options()`, and `format_with_registry()`. All three ultimately call `format_with_registry`, which is the real implementation. `format()` and `format_with_options()` are convenience wrappers that supply a default empty `ExtensionRegistry`. The registry is always required because advanced extensions: Enhancements(`+ Enh:`), Optimizations(`+ Opt:`), and extension relations(LeafRel, SingleRel, MultiRel), carry their payload as a `google.protobuf.Any` blob. The registry knows how to decode those blobs into readable text. ### `textify/plan.rs` — Top-Level Output @@ -64,14 +66,13 @@ This file is responsible for the overall structure of the output text and writes ### `textify/foundation.rs` — Shared Infrastructure -This file defines the core shared infrastructure that every other textify file depends on: +This file defines the core shared infrastructure that every other textify file depends on: The `Textify` trait is implemented by every type that can be rendered to text and describes how they should be written to text. The `Scope` trait is the context object carried through every `textify` call. Carrying output options, extension registry, error accumulator etc.. ### `textify/rels.rs` — Relation Output -Substrait proto relation types (Read, Filter, Project...) are converted to a single internal `Relation` shape before rendering. -Proto relations can be represented in their arguments, columns output mapping(Emit), addenda(advanced extensions), and children; which we store in Relation internal type. +At the top level, a Substrait `Plan` contains `PlanRel` entries. Each `PlanRel` is either a `Root` or a regular `Rel`. The textifier handles `Root` separately in the `Textify for RelRoot` implementation. Regular protobuf relation types such as `Read`, `Filter`, and `Project` are converted into the `Relation` shape before rendering. Proto relations can be represented by their arguments, columns output mapping(Emit), addenda(advanced extensions), and children; which we store in the Relation internal type. **`Value`** is the universal type for anything that can appear in a relation's argument list or output columns. @@ -91,7 +92,7 @@ Called from `expressions.rs` and `rels.rs` whenever a type annotation, column na ### `textify/extensions.rs` — Extension Argument Rendering -Called from `rels.rs` and `addenda.rs` to render advanced extension arguments. This file gives `ExtensionValue`, `ExtensionArgs`, `ExtensionColumn`, and `Expr` their `Textify` implementations. +Called from `rels.rs` and `addenda.rs` to render advanced extension arguments. This file gives `ExtensionValue`, `ExtensionArgs`, `ExtensionColumn`, and `Expr` their `Textify` implementations. The key intermediate type is **`ExtensionArgs`** (defined in `src/extensions/args.rs`). When the textifier encounters an advanced extension payload — a `google.protobuf.Any` blob — the `ExtensionRegistry` decodes it into `(name, ExtensionArgs)`: a text-format name and a generic argument bag holding positional `ExtensionValue`s, named `ExtensionValue`s, and `ExtensionColumn`s for output columns. @@ -111,7 +112,7 @@ The parser converts text to `proto::Plan` in three phases, all orchestrated by ` A **`Pair`** is Pest's fundamental match type: it records which grammar rule matched, the text it covers, and a list of inner `Pair`s for sub-expressions within that match. For example, parsing `Read[my_table => a:i64, b:string?]` produces a `Pair` for the `read_relation` rule with two inner pairs: a `table_name` pair covering `my_table`, and a `named_column_list` pair covering `a:i64, b:string?`. That `named_column_list` pair itself contains two `named_column` inner pairs — one for `a:i64`, one for `b:string?` — each of which contains a `name` pair and a `type` pair. This inner-pair nesting captures the structure within a single line — the parent-child relationships between relations come from indentation, not the grammar. -`structural.rs` wraps each top-level `Pair` into a **`LineNode`**: either a **`RelationNode`** (a relation line) or an **`Addendum`** (a `+ Enh:` / `+ Opt:` / `+ Ext:` line). `RelationNode` holds the pair, its line number, `addenda` and `children` lists. **`TreeBuilder`** then places each `LineNode` into the tree by indentation depth. +`structural.rs` wraps each top-level `Pair` into a **`LineNode`**: either a **`RelationNode`** (a relation line) or an **`Addendum`** (a `+ Enh:` / `+ Opt:` / `+ Ext:` line). `RelationNode` holds the pair, its line number, `addenda` and `children` list. **`TreeBuilder`** then places each `LineNode` into the tree by indentation depth. **Phase 3 — Proto conversion**: once all lines are consumed, the completed `RelationNode` trees are walked depth-first, leaves first, then parents. This order is required because `RelationParsePair` (the trait each relation type implements) receives its children as already-converted `Rel` messages, and the field count flowing up from them, both of which are only available after the subtree below is resolved. @@ -155,7 +156,7 @@ Handles two distinct jobs: parsing URN declarations and anchor assignments in th Converts each `RelationNode` into a `Rel` proto message. Implements the conversion for supported relation types: Read, Filter, Project etc... Also handles emit mapping — reconstructing the `RelCommon` output column remapping from the text representation. -Each relation type implements **`RelationParsePair`**, a third parsing trait alongside `ParsePair` and `ScopedParsePair`, which additionally accepts pre-converted child `Rel` messages and propagates output field counts up the tree. +Each relation type implements **`RelationParsePair`**, a third parsing trait alongside `ParsePair` and `ScopedParsePair`(both described in Shared Infrastructure), which additionally accepts pre-converted child `Rel` messages and propagates output field counts up the tree. `RelationParsingContext` is the context object used when a relation or addendum must turn parsed `ExtensionArgs` into a `google.protobuf.Any` payload. It carries the `ExtensionRegistry` plus source location so registry failures become contextual `ParseError`s. @@ -176,7 +177,7 @@ Defines the parsing traits used throughout the parser — the parser-side equiva **`ScopedParsePair`** does the same but takes a `&SimpleExtensions` context argument. It is used for constructs that require anchor lookups — function calls, type references, type variations — where the conversion can fail with a `MessageParseError` if the anchor is not in the table. -**`RuleIter`** wraps Pest's pair iterator with `try_pop` (consume the next pair only if it matches a specific rule, otherwise leave it) and `pop` (consume and assert). +**`RuleIter`** helps parser code walk through Pest's nested parse results in the order the grammar defines them. --- @@ -184,7 +185,7 @@ Defines the parsing traits used throughout the parser — the parser-side equiva Both simple and advanced extension code is found in this folder. -**Simple Extensions** are the Substrait standard mechanism for declaring custom functions, types, and type variations. In the text format these appear in the `=== Extensions` section as URN declarations and anchor assignments. `SimpleExtensions` is the in-memory lookup table built from those declarations — mapping integer anchors to qualified names and back. +**Simple Extensions** are the Substrait standard mechanism for declaring custom functions, types, and type variations. In the text format these appear in the `=== Extensions` section as URN declarations and anchor assignments. `SimpleExtensions` is the lookup table built from those declarations — mapping integer anchors to qualified names and back. **Advanced Extensions** are custom relation types, enhancements (`+ Enh:`), optimizations (`+ Opt:`), and extension table reads (`+ Ext:`, an addendum attached to a `ReadRel`) that are stored as a `google.protobuf.Any` blob in the proto. `substrait-explain` cannot parse or textify them from the plan alone. The caller supplies an `ExtensionRegistry` with registered Rust types that knows both the text representation and the protobuf `Any` representation. @@ -192,7 +193,7 @@ Both simple and advanced extension code is found in this folder. **`Explainable`** handles text format conversion. `to_args()` converts the Rust type into an `ExtensionArgs` for the textifier to render; `from_args()` constructs the type from parsed `ExtensionArgs`. -A type registers as an extension by implementing both `AnyConvertible` and `Explainable`. **`Extension`** is an empty supertrait that groups them into a single bound used by the registry. +A type registers as an advanced extension by implementing both `AnyConvertible` and `Explainable`. **`Extension`** is an empty supertrait that groups them into a single bound used by the registry. Together, the textifier path of a registered type is: `AnyRef` → `AnyConvertible::from_any` → Rust type → `Explainable::to_args` → `ExtensionArgs`. The parse path is the reverse: `ExtensionArgs` → `Explainable::from_args` → Rust type → `AnyConvertible::to_any` → proto `Any` @@ -206,8 +207,7 @@ Together, the textifier path of a registered type is: `AnyRef` → `AnyConvertib ### `extensions/simple.rs` — Anchor Lookup -`SimpleExtensions` is the anchor lookup table built from the `=== Extensions` section. It maps integer anchors to URN strings a -nd qualified names for functions, types, and type variations. +`SimpleExtensions` is the anchor lookup table built from the `=== Extensions` section. It maps integer anchors to URN strings and qualified names for functions, types, and type variations. ### `extensions/registry.rs` — Advanced Extension Registry @@ -221,7 +221,7 @@ Handlers read from `ExtensionArgs` via an `ArgsExtractor`, which tracks which ar ### `extensions/any.rs` — Any Wrapper -`prost_types::Any` is always owned — there is no borrowed form. When the textifier encounters an extension blob inside a proto struct it only has a reference to it, not ownership. Passing that blob to `from_any` without a local borrowed type would require cloning it every time. `AnyRef<'a>` solves this without cloning. Created from a reference to a `prost_types::Any`. `Any` (owned) is used when ownership is required, such as the return type of `to_any`. Using these local types in the `AnyConvertible` API also means extension implementors do not need to depend on prost directly. +`any.rs` defines local owned and borrowed representations for protobuf `Any` payloads. `Any` is the crate-owned form used when code needs to build a new payload, such as the value returned by `AnyConvertible::to_any()`. `AnyRef<'a>` is a borrowed view over an existing payload and can be created from `prost_types::Any`, `pbjson_types::Any`, or the crate's own `Any`. The registry and textifier use these wrappers so extension code works with one stable `Any` API instead of depending on the concrete `Any` type used by a particular serialization crate. Prost-generated extension types can still use the blanket `AnyConvertible` implementation; custom types can implement `AnyConvertible` manually. `AnyRef<'a>` gives the API a way to pass existing `Any` payloads by reference instead of requiring ownership at every call site. --- @@ -231,7 +231,7 @@ The test suite is split between unit tests inside source files and integration t Unit tests live alongside the code they test following standard Rust convention. `src/fixtures.rs` provides `TestContext` and other helpers shared across many test modules. `types_tests.rs` contains type round-trip tests that span both the parser and textifier and has no single source file it naturally belongs to. They both need access to `pub(crate)` types, so they cannot be in `tests/`, which is compiled as a separate crate and cannot see crate-internal items. -Integration tests under `tests/` are compiled as a separate crate and test the public API as an external caller would — they can only access what `lib.rs` exports. The primary strategy is round-trip: parse a text plan to proto, textify it back to text, and assert the output matches. This catches most correctness bugs without requiring proto-level assertions. +Integration tests under `tests/` test the public API as an external caller would — they can only access what `lib.rs` exports. The primary strategy is round-trip: parse a text plan to proto, textify it back to text, and assert the output matches. This catches most correctness bugs without requiring proto-level assertions. - `plan_roundtrip.rs` — broad round-trip coverage across relation types and features - `literal_roundtrip.rs` — round-trips for literal value parsing and formatting