Repository navigation
fix: run deep non-tail recursion on fresh coroutine stacks (#93; builds on #90) - #95
Merged
Merged
Conversation
LuaJIT's Lua stack is a fixed ~65500 slots per coroutine (LJ_STACK_MAX is a compile-time constant), so non-tail recursion overflowed after ~12k frames: user code, and the kernel's tree walkers (shen.curry, shen.find-types, shen.process-applications). Every coroutine has its own stack, so deep recursion now continues on a fresh one, generalising what #90 does for the Prolog engine. A compiled function that makes a non-tail call counts itself in SD[1] while it is on the stack: it opens with a check and an increment, and restores the count before every return. A tail call stays a tail call: its arguments are evaluated first (hoisted when they make counted calls), then the count is restored. Once 160 counted functions are open on one stack, the next one runs on a new coroutine (runtime.lua M.seg.run). Lambda bodies in a defun count themselves the same way; freeze bodies and code outside a defun use a vararg trampoline (one copy per defun); natives that call back (map, mapcan, the Prolog engine's import) count inline. Errors restore the count by assignment at every catch point (compiled trap-error, the runtime's pcalls, host entry points, the segment runner). The Prolog engine's segments go through the same runner, so one nesting cap (10000, a trappable error) covers the shared C stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves compiler.lua against #91: hoist_kc takes the counting mode (inherit for value-position control forms, own for a split let chain), try_split_let returns ret(hoist_kc(..., true)), and the CURRY, LAMW and wide freeze paths compile with CNT = nil and cexpr_tail; counted_body uses new_fnl(true, 1). CHANGELOG keeps both sides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #93
Base: main 039af36, where #89, #90, #91 and #94 are merged. The branch is 2fdb849 (the fix, built on #90) plus a merge of origin/main, e0902bc. That merge commit applies the
compiler.luaresolution against #91 described under "Merging with #91" below.On the merged head e0902bc
make testmake certifyTOTAL: 1718 pass, 0 fail across 26 specspassed ... 154/failed ... 0TOTAL: 1762 pass, 0 fail across 27 specspassed ... 154/failed ... 0100000. The split part counts itself: its generated code containsSEG(KC[.The run above includes the cold round 0. b-typed and c-untyped then got 3 more rounds on warm caches, against main:
The sections below were measured before the merge, against #90 (4456af0) and a local merge of #89, #90 and #91.
Where it overflowed (step 1)
Every case failed on LuaJIT's Lua stack, which is
LUAI_MAXSTACK 65500slots per coroutine.luaconf.hmarks this limit "<64K", and it is a compile-time constant, so a LuaJIT rebuild could not raise it much either. None of the cases failed on the C stack.nat-3000((shen.typecheck (mk 3000) peano))prims.lua:2292: stack overflow(shen.curry)shen: stack overflowpeanopeanonat-5000(non-tailmk)[string "shen:mk"]:1: stack overflowshen: stack overflowpeanopeanotc-10000([list number], non-tailmk)[string "shen:mk"]:1: stack overflow[list number][list number]rec.shen(depth5000/8000/12000/100000)5000 8000, then[string "shen:depth"]:1: stack overflow5000 8000 12000 1000005000 8000 12000 100000rd-4000((length [1 … 4000]))reader error near here:(the reader, #88)[string "reader"]:26: stack overflow(post-reader walkers)ok 4000ok 4000Mechanism
Every coroutine has its own Lua stack, so deep recursion continues on a fresh one. This generalises what #90 does for the Prolog engine.
What counts. A compiled function that makes a non-tail call to anything that may run Shen code counts itself in
SD[1]while it is on the stack:Tail calls. A tail call stays a tail call. Its arguments run first, hoisted into locals when they make counted calls themselves. Then the count is restored, then the call runs. A constant-stack loop, whether a self-loop, mutual tail calls, or a CPS chain through lambdas, never opens a segment, and the spec checks this.
Opening a segment. Once 160 counted functions are open on one stack, the next one runs on a new coroutine (
runtime.luaM.seg.run). Errors are rethrown unchanged, so Shen exceptions cross segments intact. A host yield is passed through, as in fix(prolog): run deep proofs on fresh coroutine stacks (#85) #90.Other callers.
counted_body). On the rare segment path a lambda finds itself withdebug.getinfo.map,mapcan, and the Prolog engine'simport) count inline.LEAF).Errors. Every catch point restores the count by assignment: compiled
trap-error, the runtime's pcalls, the REPL, host entry points (shen.eval,shen.call,lua_interop), and the segment runner.Limit. Past 10000 nested segments (fix(prolog): run deep proofs on fresh coroutine stacks (#85) #90's included), a call raises the trappable
Lua stack depth limit exceeded (10000 stack segments). Without the cap, nested resumes crash with SIGSEGV between 40k and 50k levels on an 8 MB main-thread stack. At the default cap, a 2,000,000-deep recursion raises the trappable error, with 100 MB peak RSS.Why not the alternatives
NT(f, args)): c-untyped ran 15–25% slower.LEAFset, c-untyped is within noise.LJ_STACK_MAX: not possible, as explained above.Performance
In-process CPU time for c-untyped (
(load driver.shen)timed withos.clock), 8 interleaved rounds on a shared machine:Capcov workloads via the bench harness, with every cache warmed first. Wall and CPU time per round; payload sha256 compared to shen-go 1f587de:
Tests
test/stack_segments_spec.luais new and has 44 checks. Each recursion below runs 100000 deep and opens segments, and the count must come back to where it started:mapcallback;trap-errorbody;if;freeze/thaw;It also checks:
trap-errorand the host unchanged.Also, after a full
make certifyrun, the count is back at 0 and no segments are nested; certify opened 4 segments in all.make testmake certifyTOTAL: 1223 pass, 0 fail across 23 specspassed ... 154/failed ... 0TOTAL: 1267 pass, 0 fail across 24 specspassed ... 154/failed ... 0TOTAL: 1706 pass, 0 fail across 26 specspassed ... 154/failed ... 0TOTAL: 1750 pass, 0 fail across 27 specspassed ... 154/failed ... 0Merging with #91
When this branch was merged with main, it conflicted with #91 in
compiler.lua(andCHANGELOG.md). Taking #91's side wholesale still passes every test, but it leaves paths uncounted, and those overflow only at depth. The merge commit e0902bc applies this resolution:resolved hunks
cexpr, a value-position control form keeps fix: 200-locals shapes and long list literals (#84, #88; builds on #78) #91'slocal call = hoist_kc(form, env).hoist_kctakes the counting mode:try_split_letends withreturn ret(hoist_kc(form, env, lnames, true)). The split part is tail-called, so it counts itself.CNT = nilandcexpr_tail(save and restoreCNTnext toHOIST/IN_IMPL), e.g.:counted_bodyusesFNL = new_fnl(true, 1)(fix: 200-locals shapes and long list literals (#84, #88; builds on #78) #91'snlocbudget).CHANGELOG.md: keep both sides.Limits and follow-ups
R.seg.limit).debuglibrary.SEGSELFneedsdebug.getinfo.Commands
🤖 Generated with Claude Code