diff --git a/openspec/changes/add-auxiliary-definitions/.openspec.yaml b/openspec/changes/add-auxiliary-definitions/.openspec.yaml new file mode 100644 index 0000000..1b9acb7 --- /dev/null +++ b/openspec/changes/add-auxiliary-definitions/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-09-22 diff --git a/openspec/changes/add-auxiliary-definitions/design.md b/openspec/changes/add-auxiliary-definitions/design.md new file mode 100644 index 0000000..e831ca0 --- /dev/null +++ b/openspec/changes/add-auxiliary-definitions/design.md @@ -0,0 +1,151 @@ +## Context + +See `proposal.md` for the motivation and issue #108 for the pain. The observable contract is in `specs/expression-evaluation/auxiliary-definitions/spec.md`. + +Today `LoadAndResolveInheritancesRecursively` materializes a source file's `$local` entries and immediately deletes `$local` before file-level or node-level structural composition. `NodeEntryValue` then carries only the resolved ordinary object and accumulated jq modules. Key-side evaluation runs over that object with `$cur`; value-side evaluation subsequently installs the builtin variables and functions and evaluates `eval:` and `raw:` strings until stable. + +`yq++` adds a second, format-specific semantic step after `jq++`: a recursive jq filter deletes every `_`-prefixed key before YAML emission. The engine cannot distinguish a holder field from legitimate data because the convention has no explicit declaration. + +The design must preserve three constraints: + +- `$local` entries are destination-relative local-node templates. Evaluating them in place would change `$cur`, `parent`, and `reftag` behavior and would make unused templates fail eagerly. +- `raw:$aux` must still be able to create a literal output key named `$aux`; omission cannot be implemented as an unconditional final deletion by spelling. +- Auxiliary values used by ordinary computed keys must be ready before the existing ordinary key-side phase begins. + +## Goals / Non-Goals + +**Goals:** + +- Represent the ordinary result and the composed auxiliary namespace as separate engine state. +- Reuse the existing structural-composition rules for auxiliary contributions. +- Prepare auxiliary computed keys and values before ordinary key-side evaluation. +- Supply one composed `$aux` jq variable consistently to ordinary key-side and value-side expressions. +- Preserve existing reference and cycle behavior while auxiliary values are prepared. +- Remove format-specific underscore semantics from `yq++`. + +**Non-Goals:** + +- Renaming or changing `$local`. +- Providing file-lexical, access-controlled, confidential, or source-private values. +- Hiding arbitrary ordinary fields by prefix, pattern, configuration, or command-line option. +- Generalizing builtin availability on key-side expressions beyond the `$aux` variable required here. +- Binding an inherited value consumed at a particular merge site for arbitrary jq processing. +- Defining a second auxiliary namespace at nested object positions. + +## Decisions + +### 1. Carry `$aux` out of band from the ordinary result + +Extend the state returned for a resolved node with an auxiliary object alongside the ordinary object and jq modules. At file read time, validate and extract a root `$aux` object before ordinary structural and expression walkers receive the object. `$local` materialization and removal remain on their existing path. + +Conceptually, resolved state becomes: + +```text +NodeEntryValue + ordinary object + auxiliary object + jq modules +``` + +The auxiliary object is not an ordinary output key even though authors declare it with the root `$aux` control key. This identity prevents an escaped literal `$aux` produced later by `raw:$aux` from being mistaken for the control namespace. + +**Alternative: keep `$aux` in the ordinary map and delete it before serialization.** Rejected because `raw:$aux` or a computed key can intentionally create the same spelling. A name-based final deletion cannot distinguish the control object from literal output and would violate the escape guarantee. + +### 2. Compose auxiliary objects in parallel with ordinary objects + +Whenever structural composition combines two node results, combine their auxiliary objects with the same merge direction and `mergeObjects` rules used for their ordinary objects. This gives `$extends`, `$includes`, recursive object composition, array composition, and overrides the same precedence on both sides. + +The auxiliary namespace is global to the completed invocation. Every source reached through structural composition contributes to it in the same deterministic traversal that currently accumulates jq modules. Expressions do not retain source-file provenance; after composition they all receive the final auxiliary object. + +Ground array-composition markers in both the ordinary and auxiliary objects before expression evaluation so an unresolved marker cannot disappear merely because it is auxiliary. + +**Alternative: entry-document-only scope.** Rejected because a reusable parent could not carry the auxiliary values its inherited expressions need. + +**Alternative: lexical source-file scope.** Rejected because merged JSON values currently discard source provenance. Preserving an environment on every inherited expression would introduce a new scoping system and would prevent the intended child override behavior. + +### 3. Prepare the auxiliary namespace before ordinary key evaluation + +After structural composition completes: + +```text +auxiliary key-side evaluation + | +auxiliary value-side evaluation + | +ordinary key-side evaluation with $aux + | +ordinary value-side evaluation with $aux + | +serialize ordinary object +``` + +Auxiliary preparation uses the existing key-before-value rule. It is eager: an invalid unused auxiliary expression is an error, just as an invalid ordinary output expression is today. Finishing auxiliary value evaluation before ordinary key evaluation ensures a computed auxiliary array can safely drive an ordinary computed key. + +Auxiliary expressions evaluate against a synthetic read view containing the structurally composed ordinary object plus the current auxiliary object at the control path `$aux`. The synthetic view exists only during auxiliary preparation and is never serialized. It lets existing value-side `ref(["$aux", ...])` lazily resolve another auxiliary expression and reuse current cycle detection. + +Direct `$aux.name` access follows jq's ordinary object semantics. During auxiliary preparation, authors who need lazy evaluation of another computed auxiliary definition use `ref(["$aux", "name"])`, just as ordinary computed fields use `ref` rather than direct dot access when evaluation is required. Once preparation finishes, ordinary expressions receive processed auxiliary values through `$aux`. + +**Alternative: process auxiliary and ordinary expressions in the same value-side batch.** Rejected because an ordinary expression could see another auxiliary definition's unevaluated `eval:` string. Deferring type errors or imposing map-iteration order would make results unstable. + +### 4. Adapt evaluation walkers to separate target and read context + +The existing key and value processors both use the object being mutated as the jq input. Auxiliary preparation needs to mutate the auxiliary object while reading the synthetic combined view. Refactor the processors so the target being walked, the path prefix used for `$cur`, and the expression read context can be supplied separately. + +For auxiliary targets, prefix current paths with `"$aux"`. Therefore an expression stored at auxiliary path `service.port` receives: + +```text +$cur = ["$aux", "service", "port"] +$curexpr = the equivalent path expression +``` + +Refresh the synthetic read view after each recursive key-side or value-side pass so newly computed auxiliary keys and values are visible on subsequent passes under the same fixed iteration limit used by ordinary evaluation. + +For ordinary targets, preserve existing `$cur`, `$curexpr`, input-root, and recursion behavior; add only the prepared `$aux` variable to both key-side and value-side invocation specifications. + +### 5. Use a jq variable rather than a lookup function + +Bind the prepared object as `$aux` in ordinary key-side and value-side expressions. This makes literal and fully prepared auxiliary values available through normal jq object access without creating `aux`, `refaux`, or `auxexpr` function families. + +A missing property consequently yields jq's normal missing-object-field result, usually `null`; existing expected-type validation still reports a mismatch when the surrounding `eval:` annotation requires another kind. + +**Alternative: a special builtin function.** Rejected because function installation currently differs between key-side and value-side evaluation, while variables already exist on both paths. A function would also need a separate choice between direct and lazy lookup semantics. + +### 6. Keep `$local` on its current lifecycle + +Do not retain `$local` through expression processing and do not expose it through `$aux`. Continue materializing local nodes before removing the control key. When a local node is used, its object copy reaches the ordinary result and its expressions evaluate at the destination path as today. + +**Alternative: reuse `$local` for auxiliary values.** Rejected because local-node templates may contain destination-relative expressions. Treating `$local` as an ordinary hidden object would eagerly evaluate unused templates at paths under `$local`, alter path-sensitive behavior, and cause node-level structural walkers to revisit template contents. + +### 7. Reserve `$aux`, not an access-control or lifetime term + +Use `$aux` for the input control key and jq variable. An auxiliary definition supports evaluation but is not observable output. The name describes its computational role without promising OOP-style `private` or `protected` access, lexical `local` scope, or a lifetime property shared by other control constructs. + +Rejected names include `$private` and `$protected` because the composed namespace is visible and overridable across structural contributions; `$transient` because `$local`, `$extends`, and `$includes` are also absent from output; and `$context` because it suggests externally supplied, read-only data rather than composable definitions. + +### 8. Remove underscore stripping from `yq++` + +Delete the recursive `_strip` jq program and feed `jq++` JSON output directly to the YAML emitter. Both commands then preserve the same data, and output omission is owned solely by the JSON++ engine's explicit control constructs. + +Add dedicated command-level coverage for `yq++`; the current normal autotest executor invokes `jq++` and does not exercise the wrapper's post-processing behavior. + +## Risks / Trade-offs + +**A new reserved root key can consume existing data.** A document currently using literal `$aux` would change meaning. → Document this as a breaking change, preserve `raw:$aux`, and add coexistence coverage for control and escaped literal keys. + +**Removing the underscore convention can expose former holder fields.** Existing `yq++` users may see fields they expected the wrapper to delete. → Ship `$aux` and underscore preservation atomically and document the output compatibility break. + +**Eager auxiliary preparation can reject unused definitions.** An invalid auxiliary expression fails even when ordinary output does not reference it. → State this rule explicitly and test it; eager preparation is the cost of a stable, fully processed `$aux` variable for key-side use. + +**A global composed namespace can create name collisions.** Structurally unrelated sources may choose the same auxiliary name. → Apply deterministic ordinary composition precedence and recommend nesting related definitions under a distinctive object name. Do not describe `$aux` as encapsulation. + +**Synthetic read state could leak into output.** Reusing one mutable map for the synthetic root would risk serializing control data or deleting an escaped literal. → Keep ordinary and auxiliary objects separate and construct read views without transferring ownership. + +**Evaluation refactoring can change existing paths or recursion.** Separating target and context touches both evaluation passes. → Characterize existing key/value, `ref`, `reftag`, `$cur`, `$curexpr`, and cycle behavior before refactoring, then run all unit and end-to-end tests. + +## Compatibility Policy + +While the project remains at `0.0.x` with limited adoption, incompatible changes are acceptable when they are documented. Unless a change explicitly requires otherwise, the project does not provide a deprecation period, dual syntax, automated migration, or a rollback procedure for such a change. + +This policy does not reduce the documentation obligation: the README and user documentation must state the pre-1.0 stability expectation, and the changelog must identify the concrete incompatibilities in the release that introduces them. Existing statements that reserve new keyword families for a major-version change must be reconciled with this policy. + +For this change, reserving `$aux` and preserving underscore-prefixed fields in `yq++` form one delivery unit and are released together. No transitional behavior is provided. diff --git a/openspec/changes/add-auxiliary-definitions/proposal.md b/openspec/changes/add-auxiliary-definitions/proposal.md new file mode 100644 index 0000000..236a8eb --- /dev/null +++ b/openspec/changes/add-auxiliary-definitions/proposal.md @@ -0,0 +1,32 @@ +## Why + +JSON++ authors sometimes need values while evaluating computed keys or values that must not appear in the generated configuration. Today they must place those values in the ordinary output tree, while `yq++` attempts to hide them by deleting every `_`-prefixed field and therefore silently deletes legitimate data; issue #108 supersedes the narrower symptom recorded by #67. + +## What Changes + +- Introduce an **auxiliary definition**, a named value under a root `$aux` object that participates in composition and expression evaluation but is omitted from the evaluation result. +- Compose `$aux` contributions with the same `$extends` and `$includes` precedence and value-composition rules as ordinary object fields. +- Bind the composed auxiliary object to the `$aux` jq variable during both key-side and value-side expression evaluation, and process `eval:` and `raw:` keys and values within the auxiliary object. +- Keep `$local` exclusively for local-node definitions; this change does not alter or rename it. +- Make `yq++` preserve ordinary `_`-prefixed keys so that JSON and YAML front-ends have the same JSON data-model semantics. +- Preserve `raw:$aux` as the escape for producing a literal output key named `$aux`. +- Document the project's pre-1.0 compatibility policy: while the project remains at `0.0.x` with limited adoption, an incompatible change does not require a deprecation period, dual syntax, automated migration, or rollback procedure unless the change explicitly requires one. +- **BREAKING** Reserve root `$aux` as JSON++ control syntax; an existing literal root key with that name must use `raw:$aux`. +- **BREAKING** Stop `yq++` from automatically removing `_`-prefixed holder fields; fields previously hidden by that convention will appear in output. + +## Capabilities + +### New Capabilities + +- `expression-evaluation/auxiliary-definitions`: Declaring, composing, evaluating, referencing, omitting, and diagnosing auxiliary definitions, including consistent `_`-prefixed field preservation across output formats. + +### Modified Capabilities + +None. + +## Impact + +- Expression and invocation context in `internal/`, including the state carried through structural composition and the variables supplied to key-side and value-side jq evaluation. +- `tools/bin/yq++`, which will become a format-conversion wrapper rather than applying a field-name convention. +- Unit tests and end-to-end cases for composition precedence, computed keys and values, escaping, diagnostics, cycles, and JSON/YAML parity. +- The README, design principles, syntax reference, evaluation model, builtin reference, terminology, compatibility notes, and changelog. diff --git a/openspec/changes/add-auxiliary-definitions/specs/expression-evaluation/auxiliary-definitions/spec.md b/openspec/changes/add-auxiliary-definitions/specs/expression-evaluation/auxiliary-definitions/spec.md new file mode 100644 index 0000000..dfbf597 --- /dev/null +++ b/openspec/changes/add-auxiliary-definitions/specs/expression-evaluation/auxiliary-definitions/spec.md @@ -0,0 +1,127 @@ +## Purpose + +Provide composed, processing-only data for JSON++ expression evaluation without occupying or deleting fields in the generated configuration. + +## ADDED Requirements + +### Requirement: Root auxiliary namespace +The system SHALL recognize a root `$aux` object as the auxiliary namespace. An **auxiliary definition** is a named value directly or transitively contained in that object; auxiliary definitions SHALL be available during processing and the control `$aux` object SHALL be absent from the evaluation result. + +#### Scenario: Auxiliary definitions do not enter the result +- **WHEN** a document declares `{"$aux":{"service":"reviews"},"name":"eval:string:$aux.service"}` +- **THEN** its result contains `{"name":"reviews"}` and no control `$aux` field + +#### Scenario: Empty auxiliary namespace +- **WHEN** a document declares an empty root `$aux` object +- **THEN** evaluation succeeds and no `$aux` field appears in the result + +### Requirement: Auxiliary namespace shape +The root `$aux` control value MUST be an object. The system SHALL reject any other kind with a diagnostic that identifies `$aux` and the received kind. + +#### Scenario: Non-object auxiliary namespace +- **WHEN** a document declares a root `$aux` value that is an array, atom, or null +- **THEN** evaluation fails with a diagnostic identifying `$aux` and the received kind + +### Requirement: Composed auxiliary scope +The system SHALL compose `$aux` contributions from the entry document, `$extends`, and `$includes` into one auxiliary namespace using the same precedence and value-composition rules that apply to ordinary object fields. Every expression in the resulting document SHALL observe that composed namespace rather than a source-file-specific namespace. + +#### Scenario: Child auxiliary definition overrides parent +- **WHEN** a parent defines `$aux.port` as `8000`, a child extends it, and the child defines `$aux.port` as `9000` +- **THEN** expressions inherited from the parent and expressions written in the child both observe `$aux.port` as `9000` + +#### Scenario: Parent supplies an auxiliary default +- **WHEN** a parent defines `$aux.host` and an extending child does not override it +- **THEN** expressions in the composed document can read the inherited `$aux.host` + +#### Scenario: Auxiliary objects compose recursively +- **WHEN** a parent defines `$aux.database` with `host` and `port` fields and a child overrides only `$aux.database.port` +- **THEN** the composed auxiliary definition retains the parent host and uses the child port + +#### Scenario: Inclusion precedence applies to auxiliary definitions +- **WHEN** an included fragment and the including document contribute the same auxiliary definition +- **THEN** the included contribution wins according to the ordinary `$includes` precedence rule + +### Requirement: Auxiliary preparation order +The system SHALL fully process key-side expressions and then value-side expressions in the composed auxiliary namespace before processing key-side expressions in the ordinary result object. Auxiliary expressions SHALL be evaluated against the structurally composed ordinary object and the current auxiliary namespace. + +#### Scenario: Computed auxiliary value supplies an output key +- **WHEN** an auxiliary value expression produces an array of key names and an ordinary key expression reads that value through `$aux` +- **THEN** the ordinary key expression observes the processed array and emits the corresponding keys + +#### Scenario: Computed auxiliary key supplies an output value +- **WHEN** an auxiliary key expression creates a named auxiliary definition and an ordinary value expression reads that definition through `$aux` +- **THEN** the ordinary value expression observes the definition under its computed name + +#### Scenario: Auxiliary expression reads composed ordinary data +- **WHEN** an auxiliary value expression reads an ordinary field established by structural composition +- **THEN** it observes the structurally composed ordinary value + +#### Scenario: Unused invalid auxiliary expression +- **WHEN** an auxiliary definition contains an invalid expression that no ordinary field references +- **THEN** evaluation fails because the auxiliary namespace is prepared in full + +### Requirement: Auxiliary variable +The system SHALL bind the fully prepared auxiliary namespace to the jq variable `$aux` during both key-side and value-side evaluation of the ordinary object. Access through `$aux` SHALL use ordinary jq object-access behavior. + +#### Scenario: Auxiliary value used by value-side expression +- **WHEN** an ordinary value expression reads an existing definition as `$aux.name` +- **THEN** the expression receives that definition's processed value + +#### Scenario: Auxiliary value used by key-side expression +- **WHEN** an ordinary key expression reads an array of strings from `$aux.names` +- **THEN** the expression can emit one ordinary key for each string + +#### Scenario: Missing auxiliary definition follows jq semantics +- **WHEN** an expression reads a name absent from the auxiliary object +- **THEN** the lookup produces jq's ordinary missing-object-field result + +### Requirement: Auxiliary references and cycles +During auxiliary value preparation, the system SHALL make the auxiliary namespace addressable from `ref` by a path beginning with `$aux`, and SHALL apply the existing reference-resolution and circular-reference diagnostics to those paths. + +#### Scenario: Computed auxiliary definition references another +- **WHEN** one auxiliary value expression uses `ref(["$aux","base"])` to read another auxiliary definition +- **THEN** the referenced definition is resolved using the ordinary `ref` behavior + +#### Scenario: Circular auxiliary references +- **WHEN** auxiliary definitions refer back to one another through `ref` +- **THEN** evaluation fails with a circular-reference diagnostic identifying the auxiliary reference chain + +### Requirement: Auxiliary current path +While evaluating an expression stored in the auxiliary namespace, the system SHALL expose `$cur` and `$curexpr` as paths rooted at the control `$aux` segment. + +#### Scenario: Current path inside auxiliary namespace +- **WHEN** an expression is stored at `$aux.service.port` +- **THEN** `$cur` identifies `["$aux","service","port"]` and `$curexpr` identifies the equivalent path expression + +### Requirement: Literal aux output key +The system SHALL preserve the existing `raw:` escape so that `raw:$aux` produces an ordinary literal output key named `$aux`; that literal key SHALL remain distinct from the control auxiliary namespace and SHALL survive output omission. + +#### Scenario: Escaped literal aux key +- **WHEN** a document contains an ordinary root key named `raw:$aux` +- **THEN** the result contains a literal `$aux` key with its declared value + +#### Scenario: Control and literal aux coexist +- **WHEN** a document declares both the control `$aux` object and a `raw:$aux` ordinary key +- **THEN** expressions receive the control object through the `$aux` variable and the result contains only the escaped literal `$aux` key + +### Requirement: Local nodes remain distinct +The system SHALL continue to interpret `$local` as local-node definitions for `$extends` and `$includes`; auxiliary-definition processing SHALL NOT eagerly evaluate or otherwise alter a local-node template in place. + +#### Scenario: Local template and auxiliary definitions coexist +- **WHEN** a document declares a `$local` template containing destination-relative expressions and also declares `$aux` +- **THEN** the local template is evaluated only after use at its destination while expressions there can observe the composed auxiliary namespace + +### Requirement: Underscore-prefixed field preservation +The system SHALL treat `_`-prefixed ordinary keys as data and SHALL preserve them at every object depth in both `jq++` JSON output and `yq++` YAML output. + +#### Scenario: Root underscore-prefixed fields survive YAML output +- **WHEN** `yq++` evaluates a document containing legitimate root `_id` and `_note` fields +- **THEN** both fields and their values appear in the YAML output + +#### Scenario: Nested underscore-prefixed fields survive YAML output +- **WHEN** `yq++` evaluates a document containing an `_id` field inside an object nested in an array +- **THEN** the nested `_id` field and its value appear in the YAML output + +#### Scenario: JSON and YAML front-ends agree on ordinary fields +- **WHEN** equivalent JSON++ and YAML inputs contain `_`-prefixed ordinary fields +- **THEN** `jq++` and `yq++` preserve the same ordinary data apart from output serialization format diff --git a/openspec/changes/add-auxiliary-definitions/tasks.md b/openspec/changes/add-auxiliary-definitions/tasks.md new file mode 100644 index 0000000..cfe8bca --- /dev/null +++ b/openspec/changes/add-auxiliary-definitions/tasks.md @@ -0,0 +1,40 @@ +## Delivery plan + +An implementation **review slice** is a focused pull request that makes one phase tractable to review. A **delivery unit** is the smallest set of changes that can truthfully be released and documented as complete. + +Sections 1 through 4 are review slices in one delivery unit. Create them as a GitHub stack, with each pull request based on the preceding slice, and keep the entire implementation stack out of `main` until all four sections satisfy the agreed specification. Then merge the stack in dependency order. The planning pull request and final archive pull request remain separate from this implementation stack. + +## 1. Auxiliary state and structural composition + +- [ ] 1.1 Characterize the current `NodeEntryValue` cache/copy behavior and the extraction lifecycle of `$local` with focused unit tests before changing the resolved-node state; verify the new tests pass against the unchanged implementation +- [ ] 1.2 Extend `NodeEntryValue` with an auxiliary object, extract and validate the root `$aux` control object when each source is loaded, and keep the ordinary object and auxiliary object independently deep-copied in the node pool; verify unit tests cover an absent namespace, an empty namespace, every non-object JSON kind, and a `raw:$aux` ordinary key that is not extracted +- [ ] 1.3 Compose auxiliary objects alongside ordinary objects on every file-level and node-level `$extends` and `$includes` path, using the same merge direction and value-composition rules; verify unit tests cover inherited defaults, child overrides, recursive object composition, inclusion precedence, and contributions from more than two sources +- [ ] 1.4 Ground array-composition markers in the completed auxiliary object as well as the ordinary object; verify unit tests cover a resolved auxiliary marker and an unresolved auxiliary marker diagnostic +- [ ] 1.5 Run `go test ./internal`, `make build`, and the existing inheritance autotests; verify all pass before beginning expression-evaluation changes + +## 2. Auxiliary expression evaluation + +- [ ] 2.1 Add characterization tests for ordinary key-side and value-side evaluation, including `raw:`, `ref`, `reftag`, `$cur`, `$curexpr`, cycle detection, and recursive passes; verify the tests pin the existing input root and path behavior +- [ ] 2.2 Refactor the key-side and value-side walkers so the mutation target, jq read context, and current-path prefix can be supplied separately without changing ordinary-object behavior; verify the characterization tests and all existing evaluation unit tests pass unchanged +- [ ] 2.3 Implement eager auxiliary preparation as key-side evaluation followed by value-side evaluation against a refreshed synthetic view of the composed ordinary and auxiliary objects; verify unit tests cover computed auxiliary keys, computed auxiliary values, reads of structurally composed ordinary data, recursive passes, and failure of an unused invalid expression +- [ ] 2.4 Bind the prepared auxiliary object as the jq variable `$aux` for both ordinary key-side and value-side expressions; verify unit tests cover a computed output key, a computed output value, nested auxiliary objects, and jq's ordinary result for a missing auxiliary property +- [ ] 2.5 Make auxiliary values addressable to `ref` under paths rooted at `$aux`, and prefix auxiliary `$cur` and `$curexpr` paths with the same segment; verify unit tests cover a successful lazy auxiliary reference, a multi-definition reference chain, an auxiliary-reference cycle diagnostic, and the exact current paths inside nested auxiliary objects +- [ ] 2.6 Preserve the separation between control state and output data throughout evaluation; verify tests cover `$aux` omission, coexistence with a literal `raw:$aux` output key, and a destination-relative `$local` template that observes the final composed auxiliary namespace only after materialization +- [ ] 2.7 Run `go test ./internal`, `make build`, and the full pre-existing autotest suite; verify the evaluation refactoring introduces no regression before adding new end-to-end coverage + +## 3. End-to-end behavior and YAML compatibility + +- [ ] 3.1 Add positive autotest cases for direct `$aux` use on both expression sides, computed auxiliary keys and values, inherited defaults, child overrides, recursive auxiliary composition, inclusion precedence, and ordinary-data reads; verify each positive scenario in the specification is traceable to a case +- [ ] 3.2 Add negative autotest cases for every non-object root `$aux`, an unused invalid auxiliary expression, and circular auxiliary references; verify diagnostics identify the `$aux` path or reference chain required by the specification +- [ ] 3.3 Add autotest cases for an empty namespace, missing-property jq semantics, `raw:$aux` coexistence, `$local` coexistence, and nested ordinary objects named `$aux`; verify only the root unescaped control key has auxiliary meaning +- [ ] 3.4 Remove the recursive underscore-stripping filter from `tools/bin/yq++` and pass `jq++` output directly to the YAML emitter; verify a focused command-level test covers root and nested `_`-prefixed keys and compares the ordinary data emitted by `jq++` and `yq++` +- [ ] 3.5 Run `make test`, `make build`, `tools/bin/autotest`, and the command-level `yq++` tests; verify all specification scenarios are covered and both front-ends preserve underscore-prefixed ordinary fields + +## 4. Documentation, compatibility policy, and release record + +- [ ] 4.1 Define *auxiliary namespace* and *auxiliary definition* at their first use in `tools/etc/docs/concepts/terminology.adoc`, and use those exact terms throughout the added documentation; verify the terminology audit finds no competing name for either concept +- [ ] 4.2 Document `$aux` syntax, composed scope, precedence, eager key-then-value preparation, jq-variable access, `ref` paths, `$cur` paths, root-only reservation, and `raw:$aux` escaping in the evaluation model and syntax/reference material; verify every normative rule in the specification has a corresponding documentation passage or example +- [ ] 4.3 Document the distinction between `$aux` and `$local`, including why local-node templates retain destination-relative evaluation; verify the guidance contains a coexistence example and does not describe either construct as private, protected, or confidential +- [ ] 4.4 Publish the pre-1.0 compatibility policy in `tools/etc/docs/concepts/design-principles.adoc` and a concise, prominent stability notice in `README.md`: while the project remains at `0.0.x` with limited adoption, an incompatible change requires documentation but no deprecation period, dual syntax, automated migration, or rollback procedure unless explicitly required; reconcile the terminology and syntax-reference statements that currently reserve new keyword families for a major-version change, and verify all policy statements agree +- [ ] 4.5 Replace documented `_`-prefixed holder patterns with `$aux`, correct the `yq++` description in `README.md` and user documentation, retain legitimate underscore-prefixed data examples, and record both concrete compatibility breaks in `CHANGELOG.md`; verify the documentation identifies `raw:$aux` as the escape for literal output and `$aux` as the processing-only mechanism without presenting a staged migration or rollback plan +- [ ] 4.6 Run `tools/bin/gendoc`, inspect the generated documentation, then run `make test`, `make build`, `tools/bin/autotest`, the command-level `yq++` tests, `openspec validate --all --strict`, and `git diff --check`; verify every command passes and no generated page loses the literal `JSON{plus}{plus}` or `jq{plus}{plus}` spelling