[pyrex41]

Fixes #84
Fixes #88

This branch contains #78. It is #78's branch (cursor/fix-consume-arity-662d, 2f3ee3c), merged with current main (43.1, e3dec8b) in 0bdbfc4 without conflicts, plus two commits. #78's let-splitting and wide-mode code is what fixes #84, so this builds on it instead of competing with it. You can merge #78 first and then this PR, or merge this PR alone, which supersedes #78.

#84: 200-locals limit (5e7b5d6, tests only)

Once #78 is merged with 43.1 main, it already loads both shapes from the issue:

repro	main e3dec8b	this branch
nest-16/20/30.shen (datatype, nested conclusion)	function at line 1 has more than 200 local variables	loaded
same, SHEN_PROLOG_ENGINE=legacy	same error	loaded
list-200/250.shen (define returning a list literal)	same error	ok 200 / ok 250
capcov cc-classify (classify.shen)	same error in shen:cc-classify#type	loads (as on shen-go)

Larger cases also work: 60- and 100-deep patterns, and 1000/1500-element literals. 5e7b5d6 adds the issue's shapes to test/wide_compile_spec.lua (+12 checks): 16/30/60-deep patterns, typechecked, and 200/250/1000-element defines.

#88: reader on long list literals (d95688e)

shen.<s-exprs> recursed once per token without a tail call. Past about 1550 elements it overflowed LuaJIT's fixed stack, and read-file's trap-error reported the overflow as reader error near here:. <strcontents>/<alphanums>/<whitespaces> recursed once per character in the same way, so (read-from-string "[1 … 3000]") failed on its 14 KB string literal.

The fix follows the iterative comment scanners that already sit next to it in prims.lua:

• shen.<s-exprs> keeps the KL's ordered choice and still calls the kernel's own single-token rules through F. Open brackets are kept on an explicit stack.
  • A missing closer falls back to the remaining alternatives at the bracket, as the KL does.
  • Each sequence is folded right to left, so cons-form/add-sexpr run, and raise, in the KL's order.
• shen.cons-form builds the spine front to back.
• <strcontents>/<alphanums> become one table.concat, and <whitespaces> becomes a loop.
• read-file rethrows a Lua stack overflow instead of reporting it as a reader error.

repro	main	this branch	shen-go 1f587de
rd-1600, rd-2000	reader error near here:	ok 1600, ok 2000	same
rfs-3000 (read-from-string, 14 KB string)	reader error near here:	ok 1	ok 1
rd-4000	reader error near here:	[string "reader"]:26: stack overflow	ok 4000

Remaining limit. rd-4000 now fails as an honest stack overflow after the reader. To the rest of the pipeline a list literal is a cons expression nested once per element, and the kernel's recursive tree walkers overflow at about 3400 elements. A frame histogram at 4000 shows shen.find-types (from shen.process-sexprs). With that made iterative, the next overflow is 3445 × shen.map-h + 3443 × shen.special-case (shen.process-applications), and then macroexpansion and compilation would follow. The native find-types moved the limit only from 3200 to 3400, so I left it out. Lifting this properly means making those walkers iterative, which should be a separate change.

Evidence that the native reader equals the kernel's. The new test/reader_native_spec.lua compiles the kernel's own KL rules (<s-exprs>, <s-exprs1/2>, cons-form, <whitespaces>, <atom>, <str>, <sym>, <strcontents>, <alphanums>) under spec. aliases. It requires the same value, or the same error message, from the natives on 145 inputs:

• every .shen file under tests/ and examples/ (86 files);
• 59 edge cases: unclosed [/(, [a | b c], $, :=, {}, nested and line comments, c#NN;, empty input, stray closers, and UTF-8.

It also reads 1600–3000-element literals, a 14 KB string, 3000-deep nesting and a 20000-character symbol. As a mutation check, removing the := alternative and the [a | b c] error makes 10 checks fail.

Tests

make test and make certify were run with GIT_DIR/GIT_WORK_TREE/GIT_INDEX_FILE unset, a private TMPDIR and a private XDG_CACHE_HOME.

	make test	make certify (43.1 suite)
main e3dec8b	TOTAL: 1210 pass, 0 fail across 23 specs	passed ... 154 / failed ... 0
#78 + main (0bdbfc4)	TOTAL: 1511 pass, 0 fail across 25 specs	passed ... 154 / failed ... 0
head d95688e	TOTAL: 1678 pass, 0 fail across 26 specs	passed ... 154 / failed ... 0

Cross-port check (capcov workloads)

Payloads were extracted with capcov's bench.py payload filter and compared with shen-go 1f587de (kernel 42.2).

• a2-libs-typed.
  • On main it fails with the cc-classify#type 200-locals error.
  • On this branch alone it is identical to shen-go.
• b-typed.
  • On main it fails with the same error.
  • On this branch alone it loads. 7 of 46 cc-contracts judgements come out ok:false (shen-go: 4), because a deep typecheck overflows inside trap-error. That is Native Prolog engine overflows the Lua stack on modest typecheck depth (~400), where the legacy engine reaches about 1000 and shen-go has no limit #85, fixed in fix(prolog): run deep proofs on fresh coroutine stacks (#85) #90.
  • With fix(prolog): run deep proofs on fresh coroutine stacks (#85) #90 merged in, it is identical to shen-go.
• All six workloads together. b-typed, c-untyped, d-g5, a2-libs-typed, e-draft and f-modelcheck are all identical to shen-go 1f587de on a local merge of this PR with fix: resolve file names against *home-directory*; per-user kernel cache (#86, #87) #89 and fix(prolog): run deep proofs on fresh coroutine stacks (#85) #90 (not pushed). There are no kernel 43.1 vs 42.2 differences to classify.

🤖 Generated with Claude Code