Add Zig-influenced source syntax laboratory and semantic density tooling

[epi13]

Summary

Implements the first evidence-producing source syntax research track on top of the merged MNCS Language foundation.

This PR does not select a final grammar. It establishes a three-representation model, adds concrete equivalent syntax candidates, and introduces executable semantic-density tooling so later syntax decisions can be based on complete meaning rather than character count alone.

Representation model

The proposed direction distinguishes:

1. Zig-influenced human source for ordinary authoring and review;
2. canonical semantic form for compact agent, Forge, RAVEL, and micro-verifier exchange;
3. verified IR for explicit control flow, state, proof obligations, and backend lowering.

Recursive changes are represented as semantic patches against stable subjects rather than unrestricted whole-file text replacement.

What changed

RFC and specification

• adds RFC 0005: Source Representations and Semantic Density;
• adds normative source, canonical form, patch, round-trip, macro, and tokenizer-neutrality requirements;
• documents the source syntax laboratory and integrates it into the project roadmap;
• updates the README and RFC/spec indexes.

Experimental syntax corpus

Adds four representations of the same account-transfer program:

• Zig-influenced human source;
• fully structured human source;
• minimal declaration-oriented human source;
• compact canonical semantic form.

A tournament manifest declares 23 shared semantic claims. Candidates are therefore compared only when they carry equivalent information about types, contracts, effects, capabilities, failure behavior, and resource bounds.

Semantic patch example

Adds a bounded repair patch showing:

• causal basis;
• requested semantic changes;
• protected properties;
• forbidden scope and authority expansion;
• candidate, verifier-call, and recursion budgets.

Executable tooling

Adds the mncs-syntax Rust crate with deterministic, tokenizer-neutral measurements:

• bytes and characters;
• non-whitespace characters;
• lexical units;
• identifiers and unique identifiers;
• literal and comment counts;
• maximum nesting depth.

Extends mncs-cli with:

mncs syntax-metrics <source> [source ...]
mncs syntax-tournament <tournament.json>

The tournament reports lexical units and non-whitespace characters per declared semantic claim.

CI

GitHub Actions now runs the account-transfer syntax tournament in addition to formatting, Clippy, tests, semantic-manifest validation, and invalid-example rejection.

Design decisions

• Zig is the strongest current surface influence, not a compatibility target.
• Token efficiency means verified semantic density, not the fewest characters.
• The language is not optimized around one model tokenizer.
• Compact canonical machine form is not automatically the human source language.
• A shorter candidate does not win if it omits authority, effects, assumptions, or failure semantics.
• Canonical formatting and source-to-semantic round trips remain requirements for any eventual grammar.
• Unrestricted textual macros and source-string self-modification remain outside the proposed model.

Validation

This PR includes unit tests for the lexical scanner and runs the tournament through CI. The draft should remain unmerged until GitHub Actions confirms formatting, strict Clippy, workspace tests, and example execution.

Deliberately unresolved

• final grammar and file extension;
• ownership and borrowing notation;
• generics and type syntax;
• exact error-union and propagation syntax;
• final placement of spec clauses;
• parser and canonical formatter implementations;
• model-tokenizer benchmark set;
• canonical binary semantic encoding;
• compile-time generation model.

[epi13]

CI and initial tournament results

The final CI run passed on Rust 1.79.0:

• cargo fmt --all -- --check
• strict workspace Clippy with -D warnings
• all workspace unit and documentation tests
• accepted semantic-manifest validation
• intentional rejection of the undeclared-effect example
• account-transfer source syntax tournament

The initial 23-claim tournament produced:

Representation	Role	Lexical units	Non-whitespace chars	Lines
Minimal source	Human source	286	994	45
Zig-like source	Human source	299	1,006	53
Fully structured source	Human source	322	1,172	77
Canonical semantic form	Agent/tooling form	289	901	18

The minimal human candidate currently has the fewest lexical units. The canonical semantic form is smallest by bytes, non-whitespace characters, and line count. This is useful evidence for retaining separate human and machine representations rather than selecting one notation from a single compactness metric.

CI run: https://github.com/epi13/mncs-language/actions/runs/30795283088

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.