Add Zig-influenced source syntax laboratory and semantic density tooling

.github/workflows/ci.yml

@@ -15,6 +15,7 @@ jobs:
       - uses: actions/checkout@v4
       - uses: dtolnay/rust-toolchain@stable
         with:
+          toolchain: 1.79.0
           components: rustfmt, clippy
       - name: Format
         run: cargo fmt --all -- --check
@@ -31,3 +32,7 @@ jobs:
             echo "invalid example was unexpectedly accepted" >&2
             exit 1
           fi
+      - name: Run source syntax tournament
+        run: >-
+          cargo run -p mncs-cli -- syntax-tournament
+          examples/syntax/account-transfer.tournament.json

Cargo.toml

@@ -1,5 +1,5 @@
 [workspace]
-members = ["crates/mncs-model", "crates/mncs-cli"]
+members = ["crates/mncs-model", "crates/mncs-cli", "crates/mncs-syntax"]
 resolver = "2"
 
 [workspace.package]

README.md

@@ -4,7 +4,7 @@
 
 Research and reference implementation for a general-purpose, verification-native programming language built on Machine-Native Complexity Standard (MNCS) principles.
 
-> **Project status:** semantic-model research and executable prototype. No surface-language syntax, compiler stability, or production suitability is claimed yet.
+> **Project status:** semantic-model and source-representation research with executable prototypes. No final grammar, compiler stability, or production suitability is claimed yet.
 
 ## Why this project exists
 
@@ -27,14 +27,15 @@ An MNCS-oriented program should make it possible to answer:
 
 ## What is in this repository
 
-- `docs/` — vision, architecture, recursive refinement, terminology, trust model, and explicit non-goals.
-- `spec/` — early normative semantic documents.
+- `docs/` — vision, architecture, syntax research, recursive refinement, terminology, trust model, and explicit non-goals.
+- `spec/` — early normative semantic and representation documents.
 - `rfcs/` — design proposals that can evolve independently of the specification.
 - `crates/mncs-model/` — an executable Rust model of the initial semantic objects and validation rules.
-- `crates/mncs-cli/` — a small validator for experimental JSON semantic manifests.
-- `examples/` — one accepted manifest and one intentionally rejected manifest.
+- `crates/mncs-syntax/` — deterministic, tokenizer-neutral source representation metrics.
+- `crates/mncs-cli/` — validation and syntax-tournament commands.
+- `examples/` — semantic manifests, competing source candidates, canonical semantic forms, and semantic patches.
 
-The JSON manifests are not proposed source syntax. They are a temporary, inspectable transport representation used to test the semantic model before committing to a parser or grammar.
+The JSON manifests and source candidates are experimental transport and research representations. They are not yet a selected production grammar.
 
 ## Current semantic objects
 
@@ -52,7 +53,7 @@ The 0.1 model contains:
 
 A key initial rule is that every effect must identify an authorizing capability, and that capability must be declared by the function. Evidence must reference a declared contract property rather than floating as unbound metadata.
 
-## Try the prototype
+## Try the semantic prototype
 
 ```bash
 cargo run -p mncs-cli -- validate examples/account-transfer.mncs.json
@@ -66,14 +67,39 @@ The following example intentionally performs a network effect without declaring
 cargo run -p mncs-cli -- validate examples/invalid-undeclared-effect.mncs.json
 ```
 
+## Explore the source syntax laboratory
+
+The current hypothesis uses three related representations:
+
+```text
+Zig-influenced human source
+        ↕ deterministic semantic round trip
+compact canonical semantic form
+        ↓
+high-level MNCS IR and verified SSA
+```
+
+Recursive tooling also exchanges semantic patches that target stable graph identities rather than unrestricted source-text replacement.
+
+Run the account-transfer tournament:
+
+```bash
+cargo run -p mncs-cli -- syntax-tournament \
+  examples/syntax/account-transfer.tournament.json
+```
+
+The tournament compares three human-source candidates and one canonical machine form against the same 23 declared semantic claims. It reports deterministic lexical units and non-whitespace characters per claim. These are tokenizer-neutral comparison metrics, not exact token counts for a particular model.
+
+See [Source Syntax Laboratory](docs/source-syntax-lab.md), [Source Representations](spec/source-representations.md), and [RFC 0005](rfcs/0005-source-representations-and-semantic-density.md).
+
 ## Intended compilation model
 
 The current direction is:
 
 ```text
-source syntax (undecided)
-        ↓
-MNCS semantic graph
+human source syntax (experimental candidates)
+        ↕
+canonical semantic graph/form
         ↓
 high-level MNCS IR
         ↓
@@ -115,15 +141,15 @@ A generator must not silently modify the trusted baseline or certify its own rep
 
 - **MNCS** defines the broader standard, contracts, complexity concepts, and verification philosophy.
 - **MNCDS** explores deterministic and structural representation where applicable.
-- **MNCS Language** investigates how those relationships can be expressed directly in a general-purpose language, including the semantic structures needed for recursive introspection and repair.
+- **MNCS Language** investigates how those relationships can be expressed directly in a general-purpose language, including semantic structures for recursive introspection and repair.
 - **MNCS Forge** can analyze, verify, localize failures, test candidate transformations, and produce evidence for MNCS-language components and conventional code.
 - **RAVEL** can coordinate recursive, distributed, multi-agent, and multi-verifier refinement across machines and trust boundaries.
 
 MNCS must remain applicable to existing languages even if this project never becomes production-ready. The standard therefore does not depend on this language project.
 
 ## Design principles
 
-1. **Semantics before syntax.** The project will not optimize punctuation before establishing what must be represented.
+1. **Semantics before syntax.** Surface choices must preserve the semantic model rather than define it accidentally.
 2. **Explicit authority.** Ambient filesystem, network, process, clock, randomness, and credential access should not be assumed.
 3. **Closed effects.** Undeclared effects should be rejected or isolated at a visible trust boundary.
 4. **Named assumptions.** Verification results must state the assumptions on which they depend.
@@ -133,15 +159,17 @@ MNCS must remain applicable to existing languages even if this project never bec
 8. **Incremental verification.** A change should invalidate the smallest defensible evidence subgraph.
 9. **Human inspectability.** Machine-native structure must remain understandable without an LLM.
 10. **Backend conservatism.** Optimization promises should be generated from established facts.
-11. **Recursive refinement is bounded and reviewable.** Diagnostics and repair proposals may feed later cycles, but promotion requires explicit policy, protected-property checks, and sufficient independent evidence.
+11. **Recursive refinement is bounded and reviewable.** Promotion requires explicit policy, protected-property checks, and sufficient independent evidence.
+12. **Semantic density beats character density.** Representation efficiency is measured against complete, equivalent claims rather than raw brevity.
+13. **Role-specific representations are allowed.** Human source, canonical agent form, semantic patches, and verified IR need not use the same notation.
 
 ## Roadmap
 
-The immediate target is **Milestone 0.1 — Executable Semantic Model**. See [ROADMAP.md](ROADMAP.md) for acceptance criteria and later phases.
+The immediate target is **Milestone 0.1 — Executable Semantic Model**, with the source syntax laboratory operating as a cross-cutting research track. See [ROADMAP.md](ROADMAP.md).
 
 ## Contributing
 
-The project is early enough that a precise counterexample is often more valuable than a large implementation. Read [CONTRIBUTING.md](CONTRIBUTING.md), then use an RFC for changes that alter the semantic model.
+The project is early enough that a precise counterexample is often more valuable than a large implementation. Read [CONTRIBUTING.md](CONTRIBUTING.md), then use an RFC for changes that alter the semantic model or source representation requirements.
 
 ## License
 

ROADMAP.md

@@ -1,6 +1,21 @@
 # Roadmap
 
-This roadmap deliberately separates semantic research from surface-language design while treating recursive introspection and refinement as a cross-cutting requirement.
+This roadmap separates semantic research from final surface-language selection while treating recursive introspection, representation efficiency, and refinement as cross-cutting requirements.
+
+## Active cross-cutting track — source representations
+
+The project now maintains an experimental syntax laboratory before committing to a grammar.
+
+Current work:
+
+- compare Zig-influenced, fully structured, and minimal human-source candidates;
+- maintain a distinct compact canonical semantic form for agents and tooling;
+- represent recursive changes as semantic patches rather than unrestricted text replacement;
+- declare equivalent semantic claims before comparing candidate density;
+- collect deterministic tokenizer-neutral metrics in CI;
+- extend later studies with multiple model tokenizers, parser behavior, repair success, human comprehension, and verifier cost.
+
+A final grammar must not be selected from character count alone.
 
 ## 0.1 — Executable semantic model
 
@@ -15,7 +30,7 @@ Acceptance criteria:
 - emit deterministic, machine-readable diagnostics;
 - include accepted and intentionally rejected examples;
 - document trust boundaries, recursive-refinement requirements, and non-goals;
-- run formatting, linting, tests, and example validation in CI.
+- run formatting, linting, tests, example validation, and syntax measurements in CI.
 
 ## 0.2 — Semantic graph, identity, and recursive artifacts
 
@@ -25,7 +40,8 @@ Acceptance criteria:
 - distinguish source identity, semantic identity, implementation identity, candidate identity, and evidence identity;
 - produce a first evidence-manifest schema;
 - define schemas for diagnostic obligations, causal slices, repair proposals, refinement budgets, semantic deltas, evidence deltas, and promotion decisions;
-- ensure observation and mutation require distinct capabilities.
+- ensure observation and mutation require distinct capabilities;
+- version the canonical semantic representation independently from human syntax.
 
 ## 0.3 — High-level MNCS IR
 
@@ -35,7 +51,8 @@ Acceptance criteria:
 - define deterministic lowering from the semantic graph;
 - preserve source-to-semantic traceability;
 - preserve enough structure to identify bounded repair regions and protected properties;
-- represent candidate transformations without modifying the trusted baseline.
+- represent candidate transformations without modifying the trusted baseline;
+- demonstrate lossless projection from human source to canonical semantic form for the supported subset.
 
 ## 0.4 — Verified SSA and micro-debugging
 
@@ -63,12 +80,15 @@ Acceptance criteria:
 
 ## 0.6 — Surface-language and self-description experiments
 
-- develop at least two competing syntax proposals;
-- test human readability, agent repairability, canonical formatting, and parser stability;
+- expand the syntax tournament to a representative corpus;
+- implement parsers and canonical formatters for at least two competing source candidates;
+- measure human readability, agent generation, localized repair, model-token counts, parser recovery, canonical formatting, graph round trips, omission rates, and verifier workload;
+- ensure every compared candidate represents the same declared semantic claims;
 - avoid syntax that cannot round-trip to the semantic graph without hidden meaning;
+- select or reject the Zig-influenced direction using collected evidence;
 - expose compiler and verifier contracts through the same introspection model used for ordinary programs;
 - begin a limited self-hosting experiment without obscuring bootstrap trust;
-- select a grammar only after semantic coverage is demonstrated.
+- select a grammar only after semantic coverage and repairability are demonstrated.
 
 ## 1.0 research threshold
 
@@ -77,11 +97,12 @@ A 1.0 designation would indicate a coherent research language and toolchain, not
 - a versioned specification;
 - a stable semantic core;
 - deterministic parsing and canonical formatting;
+- a versioned canonical semantic representation;
 - at least one executable backend;
 - an evidence manifest with verifiable artifacts;
 - bounded unsafe and foreign-function interfaces;
 - a versioned recursive diagnostic and refinement protocol;
 - isolated candidate transformations and explicit promotion policy;
-- documented recursion, authority, and resource limits;
+- documented recursion, authority, resource, and representation limits;
 - documented soundness and bootstrap limits;
 - conformance tests and independent implementation guidance.

crates/mncs-cli/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "mncs-cli"
-description = "Command-line validator for MNCS semantic manifests"
+description = "Command-line tools for MNCS semantic manifests and syntax experiments"
 version.workspace = true
 edition.workspace = true
 license.workspace = true
@@ -13,4 +13,6 @@ path = "src/main.rs"
 
 [dependencies]
 mncs-model = { path = "../mncs-model" }
+mncs-syntax = { path = "../mncs-syntax" }
+serde.workspace = true
 serde_json.workspace = true