Harden the Lambda optimization pipeline: make its invariants explicit and checkable

[cristianoc]

Context

While reviewing the generated-code changes of the n-ary functions series (#8557, specifically #8570), we traced several surprising output diffs through the Lambda optimization pipeline with -debug-ir. The investigation found that the pipeline's behavior is sensitive to incidental factors — pass round timing, binding shape, identifier names, statistics freshness — and that this sensitivity harbors not just unpredictable output but at least one real miscompilation. This issue collects the evidence and proposes an incremental hardening program (not a rewrite).

Evidence

1. Decisions depend on which round sees a term. The driver hand-unrolls its fixpoint: three simplify_alias rounds interleaved with three deep_flattens, two alpha_conversions, and six collect_info refreshes of one shared mutable Lam_stats.t. Example: flexible_array_test.res's local \"=~" operator was inlinable in round 1's body form (free vars = the exported Int_array block → passes the closed-over-exports gate for exported functions) but not in round 3's (field accesses had been propagated to internal coercion idents → fails the same gate). Same function, opposite outcome, decided by normalization timing. Historically the Pjs_fn_make wrapper hid function definitions until alpha-conversion, so which round could inline what changed again when Eliminate Pjs_fn_make, Pjs_fn_make_unit, and unsafe_adjust_to_arity #8570 removed it.

2. Passes disagree about the preferred binding shape. Lam_pass_deep_flatten hoists bindings into the enclosing group; Lam_pass_lets_dce can only substitute bindings that stay local. When Eliminate Pjs_fn_make, Pjs_fn_make_unit, and unsafe_adjust_to_arity #8570 moved beta reduction of immediately-applied functions five passes earlier, freshly created argument bindings crossed that fault line (hoisted at flatten2, unreachable for lets_dce) and a_recursive_type.mjs regressed cosmetically. Fixed by a targeted guard in Eliminate Pjs_fn_make, Pjs_fn_make_unit, and unsafe_adjust_to_arity #8570, but that is a local patch of a global disagreement. The driver itself notes: "we should investigate a better way to put different passes :)".

3. A real miscompilation lived in the gaps. Lam_beta_reduce.propagate_beta_reduce stacked the bindings for non-substitutable inlined-call arguments in reverse parameter order — the last argument evaluated first. The wrong order was visible in checked-in output on master (bs_set_int_test.mjs) and undetected, because no test made argument evaluation order observable under optimization. Fixed in Fix argument evaluation order under function inlining #8572.

4. Side-channel contracts by identifier name. lam_convert.rename_optional_parameters pattern-matches identifier strings (*opt*/*opt_<label>*) produced by typecore's optional-defaults desugar; until Eliminate Pjs_fn_make, Pjs_fn_make_unit, and unsafe_adjust_to_arity #8570, Pjs_fn_make_unit resolution checked Ident.name x = "param". The #default / #optional_arg_default attributes are similar cross-layer conventions. These couplings break silently.

5. Mutable statistics with unclear freshness. meta.ident_tbl is mutated mid-pass (beta reduction writes into it while rewriting) and consumed by shape-sensitive predicates (is_closed_with_map, ok_to_inline_fun_when_app) whose inputs may be stale relative to the term being rewritten.

6. Tooling drift. -debug-ir (which made this investigation tractable — credit where due) labels one simplify_alias output as alpha_conversion; the dump names don't match the driver's actual sequence.

Assessment

The essential part of the unpredictability is the classic phase-ordering problem — no confluent rewrite system exists for realistic optimizations, and the architecture legitimately prioritizes compile speed and readable JS output over heavyweight IR normalization. A normal-form IR (ANF-style) would make the order-bug class impossible by construction, but would mean rewriting ~15 passes and risks both core goals; a rewrite is not recommended.

The incidental part — items 1–6 above — is not forced by the goals. The n-ary functions series itself demonstrates the cure pattern at the AST level: make the property structural, delete the compensating machinery.

Proposed hardening program (in leverage order)

• Idempotence check in CI: run the Lam pipeline a second time over the test corpus and assert it is the identity. Round-sensitivity becomes a test failure instead of a mystery diff. The -debug-ir infrastructure makes this cheap to build.
• Promote discovered invariants into Lam_check (-check-lam already runs per pass): e.g. beta-residue bindings appear in parameter order; document/enforce others as they're identified.
• Replace name contracts with structure: an explicit marker for optional-parameter idents instead of *opt_<label>* string matching in lam_convert; evaluate the #default/#optional_arg_default attribute conventions for the same treatment.
• Snapshot the statistics: make each collect_info produce an immutable snapshot consumed by the following pass; forbid mid-pass ident_tbl mutation.
• One binding-placement policy, decided once, late: either run deep_flatten after the substitution passes or add a dedicated final placement pass; subsumes Eliminate Pjs_fn_make, Pjs_fn_make_unit, and unsafe_adjust_to_arity #8570's beta-residue guard and prevents the next flatten-vs-dce interaction.
• Grow the effect-order oracle corpus: inline_arg_order_test.res (added in Fix argument evaluation order under function inlining #8572) is the first test making argument evaluation order observable under optimization; add more oracles of this class (effect ordering across inlining, eta-expansion, partial application).
• Fix -debug-ir dump labels to match the driver's actual pass sequence, and document the pipeline as a table.

References

• Eliminate Pjs_fn_make, Pjs_fn_make_unit, and unsafe_adjust_to_arity #8570 — wrapper removal that surfaced items 1 and 2 (see "Review notes" in its description)
• Fix argument evaluation order under function inlining #8572 — the evaluation-order fix (item 3), with the first effect-order oracle test
• N-ary functions: make arity structural across the compiler #8557 — the n-ary functions series; review threads on a_recursive_type.mjs and flexible_array_test.mjs contain the per-pass IR walkthroughs

🤖 Generated with Claude Code