feat: the thinker rounds, the rest of the activation path, and the tic cost

[MarcusKainth]

Where it lands

Decode or execute in SQL (sqlcpu)

What you are trying to do, and what stops you

Native mode runs P_Ticker in SQL. The player walks, slides, reaches for
lines, opens doors and drives plats, floors and switch timers. Two things
stop the parity run going further, and one of them is a cost.

The thinkers are not written. demo3 agrees with the reference emulator
for every gametic from 2 to 205. At 206 the player's momentum parts, because
by then the engine has woken a monster, spawned what it fired and pushed the
player with it. What is missing is the round body over the thinker list:
P_MobjThinker and the state cycle, A_Look and A_Chase, the attack
family, P_DamageMobj and P_KillMobj, P_SpawnMobj and the slow path.
P_MovePsprites goes with them: five fields are already open at gametic 2
(m_frame, m_tics, m_state, m_lastlook, psp_sy) and they are all
waiting on a thinker.

Until they land, nothing downstream can be checked against the trace.
demo3 starts a plat at gametic 603 and a second door at 707, and neither
is reachable while the player is somewhere the engine did not put them. The
plats, floors and buttons that exist are tested against readers written from
p_plats.c and p_spec.c with the thinker seeded into a state row, which
is a real check but a weaker oracle than the emulator.

The activation path is only half there. P_UseSpecialLine dispatches the
manual door specials to EV_VerticalDoor. The switch specials that reach
EV_DoPlat, EV_DoFloor, EV_DoDoor, EV_DoCeiling, EV_BuildStairs and
EV_DoDonut are not written, nor is P_ChangeSwitchTexture, nor
P_CrossSpecialLine for the walkover triggers. A press that reaches one of
those leaves the tic unresolved, which is honest but is not the engine.

The tic costs 54.7 ms against a budget of 20. That is with doors, plats,
floors and buttons in, measured over the 120 tics that include the door
moving, on an otherwise idle machine. It was 506 ms when the clip sat inside
the per-thinker fold.

Where the time goes, measured over gametics 2 to 61 where no plane moves and
the clip's input array is empty on every tic:

build	per tic
before the plane pass	16.4 ms
the pass with the clip stubbed out	25.0 ms
the clip's inputs built, its lambda body a constant	34.6 ms
the whole of the wide clip	52.2 ms
the clip narrowed to P_ThingHeightClip	49.1 ms

So the lambda body cost 17.6 ms and building its inputs 9.6 ms, on tics where
the array being mapped is empty. Narrowing the clip to the question
P_ThingHeightClip actually asks, which drops the walk over things, the
pickup list and the special line list, bought 3 ms of the 17.6. The rest is
the line half: lines_in over bmap_lines, crossing per line and
opening per line.

Two approaches were measured and did not work.

• Gating the clip on notEmpty gained 1 ms. ClickHouse evaluates both arms
  of an if, so the captures are built either way.
• Making the pass's shared values ordinary bindings rather than one lambda
  chain cost 353 ms per tic and took the analysis from 11.4 s to 28.8 s. A
  value read five times in one SELECT is expanded five times, which is worse
  than a chain that copies its arrays through its levels.

What it might look like

The thinkers are the large piece and want their own pass in the same shape
the plane thinkers ended up in: a cheap map over the list, one batch of
whatever heavy primitive the round needs, then a cheap map to apply it, with
a guard that leaves the tic unresolved wherever the shape cannot be honest.

For the cost, the untried idea is to hand the height clip a candidate line
list built once per tic from the moving sectors, so bmap_lines is not
reached from inside the lambda. The hazard is that
P_BlockLinesIterator stops at the first line that blocks and only the lines
reached before it move the floor and the ceiling, so the candidate list has
to arrive in the blockmap's own visit order or the heights change. That is
most of what lines_in already does, which is why it was not attempted.

Does this touch any of these?

• PUR-10: the driver computes nothing

The readers under native/tests/support/ work out the same answers in Rust.
They are test code and no statement reads them, so the driver still computes
nothing.

Would you want to implement it?

Yes

[MarcusKainth]

The owner sees clickdoom native play at 12 frames per second at most while native demo demo3 holds 35. The demo draws from probed states and runs no simulation; play is bound by the tic. Measured on main at 1fc36b3 (#389), the simulation alone over 100 tics of demo3 through native diff, after the statement's analysis:

# native diff elapsed=18.2s tics=69 tics/s=3.8
# native diff elapsed=19.2s tics=80 tics/s=4.2
# native diff elapsed=20.3s tics=90 tics/s=4.4
# native diff elapsed=21.3s tics=100 tics/s=4.7

One second per ten tics from tic 69 on, so about 100 ms per tic against a 28.6 ms budget, which caps play at about 10 frames per second even with the renderer overlapped. When #379 measured play at 35.0 fps the tic cost 9.5 to 12.3 ms; the plane pass and its clip landed in #388 and #389 since, and #377's body carries the breakdown of where the tic goes (the clip's lambda body costs 17.6 ms and building its inputs 9.6 ms on tics where the array being mapped is empty). The play rate is a symptom of this issue's tic cost item rather than a driver bug.

[MarcusKainth]

Where the tic goes, and what it costs after #397

Measured on ClickHouse 26.7.5.10 (the digest in docker-compose.yml) in a
throwaway container on port 18123, docker/clickhouse/config.d and users.d
mounted as the compose file does. The machine was not quiet: other work ran
on it throughout and the same statement measured between 66.99 ms and 80.04 ms
a tic across the session, so every timing below carries the FunctionExecute
count from system.query_log beside it. That count is exact and does not move
between runs, and the attribution is built on it.

The instrument is the tic statement issued once over a window of gametics
repeated to about 240 rows, timed as query_duration_ms less the parse,
analysis, plan and pipeline microseconds the query log reports, over the tics.
Idle is gametics 2 to 61, where the thinker list is all light thinkers and no
plane moves. Door is gametics 73 to 106, where the demo's door rises.

The cost model

A statement costs what it holds, not what the tic asks of it. Every node is
evaluated for every row; both arms of an if are evaluated; and a lambda's
body is evaluated even where the array it maps over is empty, unless that
array is a constant the analyser can fold away. Measured with a 200-node body
over a computed empty array, per row: 407 function executions through
arrayMap, 6 through arrayFold.

The per-node figure NATIVE.md carries, 0.25 µs, understates this statement by
about eight times. The tic runs 32,820 function executions and 66.99 ms on an
idle tic, which is 2.04 µs each, and 58,673 and 86.73 ms on a door tic, which
is 1.48 µs. The cost per execution grows with the lambda depth the node sits
at: a 100-node body over one row cost 67.4 µs at depth 0, 131.8 µs at depth 8
and 513.9 µs at depth 16. The clip sits at depth 15 or so, inside the plane
chain, inside change_sector's chain, inside line_reach's.

Before, by stage

One stage of tick.rs's chain left out at a time. skip=command also stops
the door from opening, so its door row measures an idle tic.

stage left out	idle exec/tic	idle fexec/tic	door exec/tic	door fexec/tic
nothing	66.99 ms	32,820	86.73 ms	58,673
the plane pass	18.01	10,679	17.96	10,210
the player's think	57.67	27,978	58.17	28,004
the light thinkers	70.86	31,524	89.97	57,291
the tic command	65.61	31,894	65.57	31,921
P_UpdateSpecials	71.92	32,693	92.04	58,546
the heads-up tickers	71.48	32,719	100.71	58,575
the special buttons	66.26	32,819	88.10	58,672

Reading the execution counts, which hold still where the timings do not: on an
idle tic the plane pass is 22,141 of the 32,820, the player's think 4,842, the
light thinkers 1,296, the tic command 926, P_UpdateSpecials 127, the
heads-up tickers 101 and the special buttons 1. The rest, about 3,500, is the
constants, the 146 joinGets that read the tic before, the row and the melt.

Inside the plane pass on an idle tic, where nothing moves and every array it
walks is empty: P_ChangeSector is 6,233 executions and 32.6 ms of it, and
P_ThingHeightClip's line half 3,418 and 24.5 ms. Neither has anything to
clip. They cost what they are.

After #397

The pass is the body of a fold over the slots P_RunThinkers would reach, and
arrayFold does not run its body for an empty list, so a tic with no door,
plat, floor or ceiling on the list pays for the list walk alone.

	before	after
idle, wall	70.28 ms	21.44 ms
idle, executions	32,826	10,785
door, wall	95.82 ms	102.34 ms
door, executions	58,679	59,364
statement	280,933 bytes	281,302 bytes
analysis	12.70 s	13.03 s

native diff 205 takes 18.7 s after its analysis before and 9.4 s after, and
the two runs' --summary output and every native_state row hash for
gametics 1 to 205 are identical.

The budget

The idle target, 20 ms, is met to within the noise this machine has: 21.44 ms
back to back against the 70.28 ms the same run measured for main, and
18.01 ms for the same statement earlier in the session when the machine was
quieter. The floor for it is the base tic, which is what the pass now costs
nothing on top of.

The door target, 28.6 ms, is not reached and is not reachable by shaving.
A door tic runs the pass, and the pass costs 48,579 executions and about 81 ms
when it runs. Of that, 22,141 is the expression's own size, paid whether the
clip has anything to do or not, and the rest is arrayFold iterations that
scale with the things being clipped. Meeting 28.6 ms means the whole pass in
about 10 ms, which is roughly 6,000 executions: an eight times cut, most of it
out of P_ThingHeightClip's line half. That is P_CheckPosition's line walk
written out, and it is the same expression the player's move needs.

Measured and did not work, so nobody repeats them

• Making the pass's chain values ordinary WITH bindings: 527.91 ms a tic and
  778,359 executions, confirming native: run the floors and the switch timers #377.
• EXPANDED_LIMIT in sim/mod.rs: 1,000 to 30,000 moves the statement from
  442,916 to 191,387 bytes and the analysis from 10.76 s to 57.09 s, and the
  tic from 65.70 ms to 67.36 ms. Runtime does not depend on it. Analysis is
  cheapest around 1,000.
• short_circuit_function_evaluation = force_enable: the untaken arm of an
  if is evaluated anyway, as native: run the floors and the switch timers #377 found with notEmpty.
• Folding the chain's own first level rather than wrapping the chain: 105.93 ms
  on a door tic against 102.34 for the wrapper, and 22.82 ms idle against
  21.44. The wrapper is what native: skip the plane pass when no plane thinker is on the list #397 carries.
• Stubbing pieces of line_reach to attribute it further: the stubs change the
  simulation's answers, so the player's move does different work and the
  numbers are not comparable. The clip's own two halves above were measured on
  idle tics, where the stub changes nothing because there is nothing to clip.

What is left

The untried idea this issue records, a candidate line list built once per tic
in the blockmap's visit order, does not reach the cost. The clip's line half
costs 24.5 ms on a tic where it has nothing to clip, so what it costs is the
size and the lambda depth of line_reach, not the blockmap lookup inside it.

Two things would move the door tic, in the order I would try them.

1. P_BoxOnLineSide writes P_PointOnLineSide out eight times where the
   engine evaluates two, because corner(1) and corner(2) each expand all
   four slope cases. Binding the corner pair once halves it, in the deepest
   and most expensive place in the statement.
2. The clip runs whenever a plane thinker is active, not only when one moves.
   A door that has reached the top and is counting down VDOORWAIT is active
   for 150 tics and moves on none of them. Folding plane_clip over the
   sectors that move would skip it there. demo3's first 205 gametics hold no
   such tic, so that change needs a seeded state to show anything, which is
   why it is not in native: skip the plane pass when no plane thinker is on the list #397.

[MarcusKainth]

What the door tic is made of, and why 28.6 ms is out of reach

Measured on ClickHouse 26.7.5.10 (the digest in docker-compose.yml) in a
throwaway container of this lane's own on port 18124,
docker/clickhouse/config.d and users.d mounted as the compose file
does. The machine was not quiet: other containers ran on it throughout
and the same statement measured between 103.9 ms and 135.8 ms a tic across
the session, so every timing carries the FunctionExecute count from
system.query_log beside it. That count is exact and does not move between
runs, and the attribution is built on it.

The instrument is the one the comment above describes: the tic statement
issued once over a window of gametics repeated to about 240 rows, timed as
query_duration_ms less the parse, analysis, plan and pipeline
microseconds the query log reports, over the tics. Idle is gametics 2 to
61, door is gametics 73 to 106. The idle count here is 6,330 rather than
the 10,785 above, because this container's native_state carries no sector
thinker at all before gametic 73; the door count is 59,363 against 59,364,
so the two sessions agree where the state agrees.

On main at 3758fcd: idle 16.89 ms and 6,330 executions, door 109.5 to
111.9 ms and 59,363, resident statement 281,302 bytes, analysis 13.0 s.

Where the door tic goes

One piece stubbed out at a time, on the same container. A stub changes what
the simulation answers, so these are cost readings and nothing else.

stubbed out	wall/tic	fexec/tic
nothing	112.24 ms	59,363
the plane pass	17.37	10,304
P_ChangeSector	53.88	28,382
P_ThingHeightClip	60.57	39,226
P_LineOpening	95.34	58,013
P_BoxOnLineSide	98.35	58,821
PIT_CheckLine's crossing test	102.33	57,753
R_PointInSubsector	102.47	47,157
the descent's 19 steps cut to 1	109.51	48,161
P_BlockLinesIterator	111.11	58,275

Reading the execution counts, which hold still where the timings do not:
the plane pass is 49,059 of the 59,363. Inside it, P_ChangeSector is
30,981, of which P_ThingHeightClip is 20,137 and the rest of the clip
10,844; the pass's own arrays and thinker logic are 18,078. Inside
P_ThingHeightClip: the BSP descent 12,206, PIT_CheckLine's crossing
test 1,610 of which P_BoxOnLineSide is 542, P_LineOpening 1,350, and
the blockmap walk 1,088.

An idle tic runs the pass's fold and skips its body, which costs 2,567
executions and 7.4 ms.

Depth

P_ChangeSector over the same inputs, read out of native_state at
gametic 80 through joinGet, evaluated at a range of lambda depths by
wrapping it in one-element arrayMap levels that thread its sector list
through, 100 rows:

depth	wall/row	fexec/row
0	1.116 ms	727
2	1.361	748
4	1.560	762
8	2.031	785
12	2.403	813
15	2.555	834

The same expression costs 2.29 times as much at depth 15 as at depth 0, and
its execution count barely moves. In the statement arrayMap(clip -> sits
at lambda depth 11 and the blockmap read inside it at 18.

Measured and did not work, so nobody repeats them

The clip as a shallow stage. P_ChangeSector and the values it reads
moved out of the pass's chain into WITH bindings of their own, each the
body of its own fold, which put arrayMap(clip -> at depth 5 and the
blockmap read at 12. The door tic went from 112.2 ms to 203.2 ms and from
59,363 executions to 180,545, and the idle tic from 6,330 to 28,865.
A value a deep lambda reads is cheap when it arrives as the enclosing
lambda's parameter and expensive when it is captured from outside, which is
the property bind::chain exists for. Threading the two height arrays
through the clip fold's own parameter recovered part of it, idle 28,865 to
21,689, and not the rest. The 2.29 times depth saving is smaller than what
the captures cost, so ideas that move the clip out of the chain go the
wrong way.

Cutting the statement's text. P_BoxOnLineSide written so the corner
pair is named once, and P_ChangeSector's three height writes folded
together rather than three folds, take the resident statement from 281,302
to 276,689 bytes and P_ThingHeightClip's builder from 7,989 to 5,681. The
door tic runs 59,400 executions against 59,363. ClickHouse deduplicates
identical subexpressions inside one expression DAG, so a builder that
repeats itself costs the parser and the analyser and not the run. Measured
on its own: sum((sipHash64(number) AS h) + h + h + h) and the same with
the call written out four times both run 12 function nodes.

FunctionExecute as a stand-in for time. The BSP descent is a fold of
19 steps and 12,206 of the door tic's executions, and cutting it to one
step saves 11,202 executions and 2.7 ms. P_BoxOnLineSide is 542
executions and about 14 ms. A node inside a fold runs over a one-row column
and a node inside a lambda over the lines around a thing does not, and the
depth it sits at multiplies both. Use the count to attribute and the clock
to size, and say which one a claim rests on.

Gating the clip on the sectors that move is #400, and it is worth less
than it looks. An arrayMap body runs once however long its array is, and
not at all when the column it runs over has no rows, so on a tic where no
sector moves cs_slots is empty and every fold under it runs zero steps
already. The clip on such a tic is 2,699 executions, not the 30,981 it
costs when a sector does move. Gating it saves those 2,699 and 8.4 ms on a
waiting tic, and costs 8% on a moving one because the clip sits one level
deeper.

The floor

The door tic is 109.5 ms and 59,363 function executions. 10,304 of those
are the tic without the plane pass. Meeting 28.6 ms means the whole pass in
about 11 ms, which on this statement is about 4,000 executions.
P_ThingHeightClip's line half alone is 20,137, and it is one evaluation
of one expression: the blockmap walk, P_BoxOnLineSide per line,
P_LineOpening per line and the BSP descent, each of which the engine's
own answer needs. Nothing measured here takes a factor of five out of it,
and the two levers that exist, depth and text size, are worth 2.29 times
and nothing.

So the floor for a tic that moves a sector is what P_CheckPosition's
line half costs to evaluate once, and on this machine that is about
100 ms. The budget needs a different currency rather than a smaller
expression: either a formulation of the line half an order of magnitude
smaller in nodes, or a way to evaluate one node for many tics at once. The
second is closed off by the tic reading the tic before it.

The idle tic is already at its floor: 16.9 ms, of which the pass is 7.4.

What is on top of this

#400 gates the clip on the sectors that move, which is item 2 above. #402
names P_BoxOnLineSide's corner pair once, which is item 1, and carries
the measurement that it does not reach the cost. Both are drafts, and #400
is a trade rather than a win: it is worth a few percent over a plat's or a
door's whole cycle and 8% against it on the window this issue measures.

Written mostly by Claude Opus 5.

[MarcusKainth]

The first divergence moves from gametic 2 to 77

A stack of five, all drafts: #409, #418, #419, #425, #426. Measured on
ClickHouse 26.7.5.10 (the digest in docker-compose.yml) in a throwaway
container of this lane's own on port 18126, docker/clickhouse/config.d and
users.d mounted as the compose file does. The machine was not quiet:
other containers ran on it throughout and the same statement measured between
25.7 ms and 37.1 ms a tic across the session, so FunctionExecute from
system.query_log is beside every timing.

Where the run stops now

$ clickdoom native diff 300 --probe refemu/reference_traces/demo3/probe.9a6a47d01119.tsv
clickdoom: error: tic 77 game slot 0 prndindex: 241 against the probe's 244

Before the stack it was tic 2 mobj slot 25 m_frame: 0 against the probe's 1.
Every field of every gametic from 2 to 76 now agrees with the reference
emulator, and gametic 77 is the first tic the simulation says it could not
produce.

Per pull request, the field each moves:

	first divergence before	after
#409 the pending weapon a pickup sets	p_pendingweapon at 47	at 63
#418 the weapon sprites	psp_sy at 2, p_attackdown at 15, p_readyweapon at 63	psp_state at 140
#419 the sight check	nothing reads it	unchanged
#425 the mobj thinkers and A_Look	m_frame at 2, run stops at 2	run stops at 77
#426 what a sight check walks	unchanged	unchanged

#409 also closes #408, a divergence of its own: P_GiveWeapon's pending weapon
never reached the state row.

What the tic costs

Idle is gametics 2 to 61, where no plane thinker is on the list and about three
things a tic run A_Look. Door is gametics 73 to 106. The instrument is the
tic statement issued once over the window repeated to about 240 rows, timed as
query_duration_ms less the parse, analysis, plan and pipeline microseconds
the query log reports, over the tics.

build	idle wall/tic	idle fexec/tic	door wall/tic	door fexec/tic
#409, the bottom of the stack	21.75 ms	10,787	98.35 ms	59,365
#418, the weapon sprites	23.03	11,465	100.76	60,046
#425, the mobj thinkers	28.33	11,904	106.16	60,018
#426, the sight check cut down	25.69	11,936	102.87	60,050

Those arms were taken before the stack was rebased onto main at fef7681, each
against its own parent, so their differences hold and their absolute door
figures are from before #400 changed the plane clip. main and the stack's top
after the rebase, back to back on the same container:

idle-main        tics=240 bytes=278931 resident=274363 analysis=12.52s wall/tic= 21.91ms fexec/tic=10786
idle-stack-top   tics=240 bytes=416036 resident=411308 analysis=23.58s wall/tic= 25.72ms fexec/tic=11936
door-main        tics=272 bytes=279531 resident=274363 analysis=12.54s wall/tic=104.67ms fexec/tic=59616
door-stack-top   tics=272 bytes=416636 resident=411308 analysis=23.46s wall/tic=107.72ms fexec/tic=60303

The resident statement goes from 274,363 bytes to 411,308 and its analysis from
12.5 s to 23.5 s. Analysis is once a session.

Where the idle tic's 3.9 ms of new cost goes, from stubbing one piece at a time
on the same container:

stage	idle fexec/tic	idle wall/tic
the tic before the sprites	10,787	21.75 ms
the weapon sprites' fold, four entries for one cycling sprite	+621	+1.03
the sprite arrays, the flash copy and the writeback	+57	+0.25
the thinkers' snapshot pass and two state entries	+169	+0.97
the sight checks, about three a tic	+270	+1.7

The sight primitive is 270 function executions whatever it is asked. Its cost
is rows walked, not nodes run, which is why #426 takes 9 ms off the same
execution count.

Measured and did not work, so nobody repeats them

• Gating the crossing on the REJECT matrix, on its own: 27.94 ms against
  28.33, and 40 more function executions. E1M7's matrix has 2,410 of its
  3,613 bytes zero and the things looking for the player on this window are
  mostly in rooms it does not reject. It is in native: cut what a sight check walks #426 because it is the engine's
  own first line and costs two nodes, not because the window shows it.
• IN over a set whose members are not literals answers Nullable(UInt8),
  and arrayFold refuses an accumulator whose type its body does not reproduce
  exactly. Both dispatches here compare one value at a time.
• Reading a computed array from outside a lambda is what the sight check's
  cost was. A constant array replicates for nothing; a computed one is copied
  once per element of the array being walked. So the sector heights become one
  value per seg once for the tic and every check walks that array as a
  parameter.

What is left, in the order gametic 77 dictates

A_Chase with P_NewChaseDir, A_FaceTarget and P_Move over P_TryMove,
and A_Look's own draw for the see sound, which needs the sound ids and those
are not in native/tables/ yet. native diff at gametic 77 is short exactly
the three draws the reference run's own random-call log records there, one
each from A_Look, P_NewChaseDir and A_Chase.

After that, in the order the trace shows them: P_NoiseAlert and
sec_soundtarget at 140, the player's own attack (P_FireWeapon,
A_FireShotgun, P_GunShot, P_LineAttack over P_PathTraverse) also at
140, and the mobjs that spawn and are removed from 143, which is where the
thinker list stops holding still.

The idle tic is 25.7 ms against the 20 ms this issue names. The next lever is
that every pair a tic asks the sight check about shares one end, the player, so
half of what a check works out is worked out once per pair rather than once per
tic.

Written mostly by Claude Opus 5.

[MarcusKainth]

The first divergence moves from gametic 77 to 125, and the analysis budget stops it there

A stack of five, all drafts: #428, #429, #430, #431, #432. Measured on ClickHouse 26.7.5.10
(the digest in docker-compose.yml) in a throwaway container of this
lane's own on port 18127, docker/clickhouse/config.d and users.d
mounted as the compose file does. The machine was not quiet: other
containers ran on it throughout, so FunctionExecute from
system.query_log is beside every timing. Every arm below was measured
back to back on the same container, each after its own
native load --fresh and native diff 130.

Where the run stops now

$ clickdoom native diff 300 --probe refemu/reference_traces/demo3/probe.9a6a47d01119.tsv
clickdoom: error: tic 125 game slot 0 prndindex: 125 against the probe's 126

Before the stack it was tic 77 game slot 0 prndindex: 241 against the probe's 244. Every field of every gametic from 2 to 124 now agrees with
the reference emulator.

Per pull request, what each moves:

	first divergence before	after
the sound ids	77, prndindex 241	unchanged
A_Look's see sound	77, 241	77, 242
the pointer renumbering (#427)	77, 242	unchanged
P_NewChaseDir's direction tables	77, 242	unchanged
the chase	77, 242	125, 125 against 126

What is missing at 125 is P_CheckMissileRange's draw for the distance,
which the reference run's own random-call log names at its gametic 124.

What the tic costs

Idle is gametics 2 to 61, where nothing chases; door is 73 to 106;
chasing is 108 to 124, where one or two things chase every tic.

build	statement	analysis	idle wall/tic	idle fexec	door wall/tic	door fexec
main at #426	416,036	23.15 s	23.71 ms	11,936	107.78 ms	60,303
the sound ids	416,036	23.26	23.80	11,936	107.85	60,303
A_Look's see sound	440,448	23.18	23.83	11,949	108.18	60,315
the pointer renumbering	441,042	24.01	23.79	11,966	106.62	60,333
the direction tables	441,042	24.09	24.01	11,966	107.79	60,333
the chase	517,701	33.18	24.64	12,104	110.25	61,718

A tic that chases costs 26.54 ms and 12,332 executions, which is 2 ms over
one that does not. The idle tic stays under the 25.7 ms this lane
inherited and over the 20 ms this issue names.

The analysis budget is what stops the chase

FIRST_TIC_TIMEOUT in driver/src/cli/native/diff.rs is 120 s, and the
GitHub runner took 91 s for the first tic on the stack this one sits on,
whose analysis measured 23.2 s here. The four pull requests below the
chase leave the analysis at 24.1 s, which scales to about 95 s. The chase
takes it to 33.2 s, which scales to about 131 s. So the chase is a draft
that must not merge as it stands, and the fix is a smaller statement
rather than a larger timeout.

The statement's byte count is not what costs. Broken out of
system.query_log, the first tic on this build is 46 ms of parse, 33.3 s
of analysis and half a second of plan and pipeline. And most of the bytes
are not expressions: nest in sim/mod.rs lists every binding a stage
still carries by name and indents each nested stage four spaces further,
so the carried names and their indentation are 237,930 bytes of the
441,126 the statement holds without the chase and 287,775 of the 514,543
with it. Two more stages indent every line inside them, which is where
50 KB of the 73 KB went. The chase's own text is 14.7 KB.

So the budget to spend is the expression tree the analyser walks, and the
chase adds a second whole P_CheckPosition to it. The statement now holds
three walks of the blockmap's lines: the player's move, the plane clip and
this.

Measured and did not work, so nobody repeats them

• EXPANDED_LIMIT in sim/mod.rs, which decides where the bindings
  are cut. With the chase in: 1,000 gives 720,676 bytes, 27.6 s analysis
  and 27.1 ms an idle tic; 2,000 gives 601,787, 36.2 s and 27.1 ms; 3,000,
  which is what is committed, gives 517,701, 33.2 s and 24.6 ms; 6,000
  gives 380,639, 47.4 s and 24.4 ms. Nothing there is both smaller and
  faster, and the two that shrink the analysis cost the idle tic 2.5 ms.
• Stubbing the chase's pieces out one at a time. The move test
  replaced by an empty answer: 506,511 bytes against 514,543.
  P_NewChaseDir's and A_Chase's arithmetic replaced by constants:
  511,476. Shrinking what the chase writes does not recover the growth.
• Moving the crowding test out of the chase's fold, so the fold's
  answer is read once rather than twice: 514,583 bytes against 514,543.
• Collapsing the eleven writebacks into one value per slot, so the
  fold's answer is read twice rather than twenty-two times: 513,908
  against 514,583.
• Carrying A_Look's see-sound draw in the state cycle's own tuple
  rather than in a pass of its own: the same statement size to within
  128 bytes, and 25.1 s of analysis against 22.9 s. The extra tuple member
  and the extra read of the cycle's answer cost the analyser more than the
  pass they saved.

What the chase does and does not do

A_Chase runs for every thing whose state cycle entered a state carrying
it, which includes the thing A_Look has just woken, because the engine
runs the routine inside the P_SetMobjState that entered the state. That
is why gametic 77 draws three times for one monster.

P_Move and every P_TryWalk inside P_NewChaseDir ask the same
question of the same point, because the search stops at the first
direction that works and nothing moves until one does. So the move test is
asked once about all eight directions of every mover. P_NewChaseDir's
two draws decide the order the axes and the search run in and neither
decides whether one of them works, so how many numbers a thing draws is
known before any of them is read, each thing's draws sit at the offset the
things before it left, and the pass stays a map over the list.

The move test sits inside a fold over a list of one entry or none, so a
tic with nothing to chase does not pay for it, and the body reads the
fold's own parameter.

Nine things leave the tic unresolved rather than being guessed. The one
this issue asks for and does not get is the rounds: two movers standing
close enough that one's move changes what the other is told are detected
and the tic stops there rather than being scheduled one after the other.
demo3 reaches the first such pair at gametic 82.

What is left, in the order gametic 125 dictates

1. Bring the statement back inside the analysis budget. The lever nobody
   has tried is a narrower P_CheckPosition for a monster: it carries
   none of MF_NOCLIP, MF_TELEPORT, MF_PICKUP or MF_FLOAT, and the
   pickup half of the answer is always empty for it.
2. P_CheckMissileRange past the reaction time, which needs a line of
   sight for the chasers and the draw for the distance. The chasers a tic
   asks about are known from the tic-start snapshot, so their pairs batch
   into the same sight call A_Look already makes.
3. P_CheckMeleeRange past the distance test, which needs the same sight.
4. The rounds, for the movers that cannot be made independent.
5. P_NoiseAlert and the player's own shot at gametic 140.

Written mostly by Claude Opus 5.

[MarcusKainth]

The analysis budget comes down, and the first divergence moves to gametic 140

A stack of five, all drafts: #434, #435, #432, #436, #437. Measured on
ClickHouse 26.7.5.10 (the digest in docker-compose.yml) in a throwaway
container of this lane's own on port 18128, docker/clickhouse/config.d
and users.d mounted as the compose file does. The machine was not
quiet: other containers and other lanes ran on it throughout and the
same statement measured between 24.6 s and 29.9 s of analysis across the
session, so every arm was taken back to back against its neighbour and
FunctionExecute from system.query_log is beside every timing.

Where the run stops now

$ clickdoom native diff 300 --probe refemu/reference_traces/demo3/probe.9a6a47d01119.tsv
clickdoom: error: tic 140 game slot 0 prndindex: 175 against the probe's 178

Before the stack it was tic 77 game slot 0 prndindex: 242 against the probe's 244. Every field of every gametic from 2 to 139 now agrees with
the reference emulator. What is left at 140 is the player's own first
shot and the P_NoiseAlert that follows it.

What each arm costs

Idle is gametics 2 to 61, door 73 to 106, chasing 108 to 124. Each arm
after its own native load --fresh and native diff 130.

build	statement	analysis	idle wall/tic	idle fexec	first divergence
main at #431	441,042	24.6 to 25.4 s	24.9 ms	11,936	77
the closed subqueries	441,911	24.8	25.3	11,966	77
the stage cut	465,770	13.9	25.9	11,971	77
the chase (#432)	545,894	20.2	26.2	12,109	125
the thing types	545,894	20.2	26.1	12,109	125
the missile check	582,661	22.6	25.7	12,133	140

The whole of the chase and the missile check now costs less to analyse
than main does without them: 21.3 to 22.6 s against 24.6 to 25.4 s.
#432 is inside the budget as it stands.

FIRST_TIC_TIMEOUT in driver/src/cli/native/diff.rs is 120 s and the
runner took 91 s for a first tic that measured 23.2 s here, which is a
ratio of 3.9. This stack's top measured 21.3 to 22.6 s, so the same ratio
puts it at 83 to 88 s. The five runs are queued behind each other on the
runner and none has finished its test job yet, so that is an estimate
and not a reading.

The analysis budget, and what actually costs

native diff's first tic is the resident statement being analysed and
nothing else: 46 ms of parse, 25 s of analysis and half a second of plan
and pipeline. Three things were measured about where that goes.

The statement's byte count is not the metric, but its expression tree
is. nest in sim/mod.rs indents each nested stage four spaces
further, and that indentation is 230,042 of the 441,911 bytes. Stripping
it moves the analysis by 0.07 s.

Analysis grows faster than the tree does. One stage of tick.rs's
chain left out at a time, on the idle window:

stage left out	statement, text only	analysis
nothing	211,667	26.72 s
the tic command	206,011	25.55
the special buttons	211,822	25.98
the player's think	120,995	6.02
the mobj thinkers	180,116	16.88
the light thinkers	206,317	23.52
the plane pass	181,948	17.41
P_UpdateSpecials	202,843	27.05
the heads-up tickers	196,028	15.10

Taking 43% of the text out takes 77% of the analysis with it, so a byte
removed anywhere is worth more than its share.

What the analyser walks is what a binding copies. A binding named in
another binding's text is expanded into it; a subquery's column is read.
sim/mod.rs cut its stages where a binding grew past 3,000 bytes once
expanded, which let a stage hold several bindings each carrying a copy of
the ones before it: 211,000 bytes of text and 281,159 once the copies are
counted. Cutting on the copying instead, at 800 bytes, takes the analysis
from 24.8 s to 13.9 s for 24,000 more bytes of statement, one more stage,
five more function executions a tic and identical state rows.

A bug this found

P_InitPicAnims' four constant subqueries named the two name-to-number
maps beside them, texnum and flatnum, which are bindings of the tic
rather than columns of animdefs. A name a subquery does not define
resolves outwards, so they were correlated subqueries, and ClickHouse
answers a correlated subquery with a join. A join in this statement's
pipeline squashes the rows a session feeds it one at a time, so every tic
reads the state from before the batch rather than the tic before it.

It only shows where the maps and the arrays that read them fall into
different stages, and the cut is a byte-length heuristic, so whether a run
was right depended on what the database was called. demo_statement
against nats was correct and against
clickdoom_native_test_74535_sim_parity left prndindex at 0 from the
second tic on. #434 closes the subqueries
and adds a_subquery_names_nothing_bound_outside_it, which walks every
(SELECT ...) the tic holds and fails on a name it does not define.

Measured and did not work, so nobody repeats them

• Layering the bindings so no stage holds one another reads. That is
  the copy limit taken to zero: 46 stages, a 1.48 MB statement, and a run
  that asked for 54 GiB and was stopped.
• Keeping the size rule beside the copying rule, at 3,000 and 1,000:
  658,025 bytes and 22.4 s against 517,322 and 19.9 s for the copying rule
  alone.
• A narrower P_CheckPosition for a monster's move, which the comment
  above names as the lever left. With the stage cut in, the chase's whole
  move test is 1.5 s of analysis and 2.0 ms of a chasing tic, measured by
  replacing it with an empty answer. Narrowing it takes a part of that, and
  it is worth less than the generator it would add.
• P_BoxOnLineSide is the largest single piece that is left: 3.6 s of
  analysis and 2.1 ms of an idle tic for 8.4 KB of text across the three
  callers, measured by replacing it with -1. Its four
  P_PointOnLineSide calls could be two, because the two diagonal cases
  take the top corner and the bottom corner and differ only in which x
  goes with which. That is the next size lever and it was not taken here.

What is left, in the order gametic 140 dictates

1. P_NoiseAlert and sec_soundtarget, which the player's shot sets.
2. The player's own shot: A_WeaponReady to P_FireWeapon,
   A_FirePistol, P_GunShot, P_LineAttack over P_AimLineAttack and
   P_PathTraverse with the intercepts sorted by (frac, insertion),
   P_SpawnPuff and P_DamageMobj.
3. The mobjs that spawn and are removed from gametic 143, which is where
   the thinker list stops holding still.
4. The rounds, for the movers that cannot be made independent. demo3
   reaches its first such pair at gametic 82 and the fields it decides
   happen to agree, which is why the run passes through it; the tic is
   marked unresolved and sim_parity_live pins that.

The idle tic measured 25.7 ms and 28.3 ms across the session against the
20 ms this issue names. Nothing in this stack moved it: the stage cut is
worth nothing at run time, and the chase and the missile check each add
about half a millisecond.

Written mostly by Claude Opus 5.

[MarcusKainth]

The player fires, and the first divergence moves from gametic 140 to 143

A stack of four, all drafts: #442 (rebased onto main), #447, #448, #449.
Measured on ClickHouse 26.7.5.10 (the digest in docker-compose.yml) in a
throwaway container of this lane's own on port 18130,
docker/clickhouse/config.d and users.d mounted as the compose file does.
The machine was not quiet: other lanes' containers and a workspace test run
were on it throughout, so FunctionExecute from system.query_log is beside
every timing. scripts/machine-lock.sh was held for every arm below.

Where the run stops now

$ clickdoom native diff 300 --probe refemu/reference_traces/demo3/probe.9a6a47d01119.tsv
clickdoom: error: tic 143 game slot 0 prndindex: 198 against the probe's 253

Before the stack it was tic 140 game slot 0 prndindex: 175 against the probe's 178. Every field of every gametic from 2 to 142 now agrees with the reference
emulator. The summary lists 56 fields that ever differ and the earliest tic
any of them first differs on is 143, which is where the shotgun's own frames
reach A_FireShotgun.

Per pull request, what each moves:

	first divergence before	after
#442 the noise flood	140	unchanged, nothing reads it
#447 the box test	140	unchanged, the same state rows
#448 A_Look's hearing branch	140	unchanged, nothing writes the sound target
#449 P_FireWeapon	140	143

The reference run's random-call log records its gametic before gametic is
bumped, so its rows for gametic G are the probe's tic G+1. Its rows for the
probe's tic 140 are one monster waking on the shot's noise:

$ awk -F'\t' '$1 == 139' refemu/reference_traces/demo3/probe-rng.9a6a47d01119.tsv
139  0  P_NewChaseDir  520  156825427
139  1  P_NewChaseDir  324  156828030
139  2  A_Chase        256  156828057
139  3  A_Chase        256  156833812
139  4  A_Look         136  156834103
139  5  P_NewChaseDir  324  156839602
139  6  A_Chase        256  156839629
139  7  A_Chase        256  156849615

The probe's prndindex goes 170 at gametic 139, 178 at 140 and 189 at 141, so
three more monsters wake at 141, and both sides now agree on every draw.

What the tic costs

Idle is gametics 2 to 61, door 73 to 106, chasing 108 to 124. Each arm after its
own native load --fresh and native diff 130, back to back on the same
container, timed as query_duration_ms less the parse, analysis, plan and
pipeline microseconds system.query_log reports. bytes is the timed statement
and resident the one a session opens.

                 tics   bytes  resident  analysis  wall/tic  fexec/tic
idle-main         240  582912    578024    21.04s   25.36ms      12133
idle-noise        240  582912    578024    21.68s   35.39ms      12133
idle-box          240  580229    575341    19.85s   24.66ms      12129
idle-hearing      240  611549    606629    23.69s   25.68ms      12165
idle-shot         240  650574    645622    25.14s   25.74ms      12481
door-main         272  583512    578024    21.02s  118.22ms      61810
door-shot         272  651174    645622    25.91s  118.25ms      62047
chasing-main      238  583150    578024    23.24s   28.88ms      12442
chasing-shot      238  650812    645622    24.66s   28.58ms      12787

The noise flood is byte for byte the statement main holds and runs the same
12,133 executions an idle tic, because nothing reads it until P_FireWeapon
does. The whole stack costs 348 more executions on an idle tic and 345 on a
chasing one, and the wall figures do not move outside what this machine's noise
was.

The analysis budget, and where this stack is over it

FIRST_TIC_TIMEOUT in driver/src/cli/native/diff.rs is 120 s. main measured
21.04 s here and the stack's top 25.14 s, so the stack costs 4.1 s more to
analyse than main does. The runner took 91 s for a first tic that measured
23.2 s here, a ratio of 3.9, which puts this build at about 98 s. That is inside
the timeout with 22 s of margin, and the margin is thinner than main's.

Two of the three new commits add a stage each: 24 stages on main, 25 with the
hearing branch, 26 with the shot. The hearing branch's own text is 4,480 of the
31,288 bytes it adds; the rest is the indentation and the carried column names a
stage brings. A stage carries every still-live binding forward and that is what
the analyser walks.

Measured and did not work, so nobody repeats them

• Reading the noise flood's source sector from around its fold. The alert
  sits inside arrayFold over the sector the shot was fired in, or over
  nothing. With the sector read from the row rather than taken as the fold's own
  element, an idle tic ran 20,443 function executions against 12,481 and 28.86
  ms against 25.74: ClickHouse evaluates a node that depends on no lambda
  parameter once for the row, whatever the fold does. It is not enough for the
  fold's body as a whole to read the parameter; the expensive subtree has to.
• COPY_LIMIT in sim/mod.rs, which decides where a stage is cut, on the
  stack's top: 1,000 gives 584,759 bytes, 25.03 s and 12,603 executions an idle
  tic, and 1,400 gives 529,518 bytes, 29.18 s and 12,602. Neither is both
  smaller and faster, so the committed 800 stands.
• Collapsing the hearing branch's three derived arrays into the state cycle's
  own lambda, so mt_wakes, mt_looked and mt_target cost no binding: 297
  bytes and 0.35 s for a lambda of fifteen parameters. Not taken.
• Naming P_BoxOnLineSide's corner pair once, which the comment above names
  as the largest size lever left. It halves the point tests across the three
  walks of the blockmap's lines and takes 2,683 bytes and about 1 s of analysis.
  The tic does not move.

A guard that was wrong and that nothing reached

A_WeaponReady's test for the player's mobj standing in an attack frame
compared it against states[S_PLAY_ATK1].nextstate, which the engine's own
table gives as S_PLAY and not the second attack frame. It would have marked
any tic unresolved where the weapon was ready and the player stood still.
demo3's player is in a running frame on every gametic from 2 to 139, so nothing
ever reached it and no run produced a wrong value. #449 replaces the guard and
reads the second frame out of the states table as the one state that runs back
into the first, with a load guard that stops unless exactly one does.

What is left, in the order gametic 143 dictates

1. A_FireShotgun: P_BulletSlope over P_AimLineAttack, seven P_GunShots,
   P_LineAttack over P_PathTraverse with the intercepts sorted by
   (frac, insertion), P_SpawnPuff and P_SpawnBlood, and P_DamageMobj
   with the pain and death states. The log names 56 draws for it in seven groups
   of eight.
2. The mobjs that spawn and are removed from the probe's gametic 143, where the
   list goes from 264 slots to 271.
3. The rounds, for the movers that cannot be made independent. demo3 reaches its
   first such pair at gametic 82 and the fields it decides happen to agree,
   which is why the run passes through it; the tic is marked unresolved and
   sim_parity_live pins that.

The idle tic measured 24.7 ms to 25.7 ms across the session against the 20 ms
this issue names.

Written mostly by Claude Opus 5.

[MarcusKainth]

The analysis budget comes down to 20 s, and the door tic with it

A stack of two, both drafts: #450, #451. Measured on ClickHouse 26.7.5.10
(the digest in docker-compose.yml) in a throwaway container of this lane's
own on port 18131, docker/clickhouse/config.d and users.d mounted as the
compose file does. The machine was not quiet: other lanes' containers and
the host's own file indexing ran on it through the session, so
FunctionExecute from system.query_log is beside every timing and each arm
was taken back to back against its neighbour on one container.
scripts/machine-lock.sh was held for every one of them.

Where the run stops

$ clickdoom native diff 300 --probe refemu/reference_traces/demo3/probe.9a6a47d01119.tsv
clickdoom: error: tic 143 game slot 0 prndindex: 198 against the probe's 253

Unchanged. Neither pull request moves a tic: #450 rewrites one expression and
every native_state row for gametics 0 to 300 hashes the same, and #451 adds
a primitive nothing calls yet.

What the tic costs

                 tics   bytes  resident  analysis  wall/tic  fexec/tic
idle-main         240  650575    645623    25.09s   26.27ms      12481
idle-pipe         240  650160    645208    20.71s   17.84ms      12168
door-main         272  651175    645623    24.47s  110.93ms      62047
door-pipe         272  650760    645208    20.57s   80.16ms      60386
chasing-main      255  651102    645623    24.49s   28.61ms      12785
chasing-pipe      255  650687    645208    25.59s   21.65ms      12449

idle-main2        240  650575    645623    24.25s   25.36ms      12481
idle-pipe2        240  650160    645208    20.66s   18.45ms      12168
door-main2        272  651175    645623    24.13s  112.77ms      62047
door-pipe2        272  650760    645208    20.08s   69.55ms      60386
chasing-main2     255  651102    645623    30.51s   29.95ms      12785
chasing-pipe2     255  650687    645208    20.54s   20.42ms      12449

The idle tic is 17.8 to 18.5 ms against the 20 ms this issue names, and the
door tic 69.6 to 80.2 ms against the 100 ms the comment above calls its floor.
The analysis is 20.1 to 20.7 s against main's 24.1 to 25.1 s.
FIRST_TIC_TIMEOUT is 120 s and the runner took 91 s for a first tic that
measured 23.2 s here, a ratio of 3.9, which puts this stack at 78 to 81 s.

What the analyser spends its time on

bind::chain nested each of its levels inside the body of the level below
it, so a chain of ten levels put its last one ten lambdas deep. A map's array
argument is outside its own lambda, so a level can map the level below it
instead, and every level then sits one lambda deep whatever the chain's
length. That is the whole of #450.

The measurements that say why, all on main's statement in one session:

• Doubling one binding, by replacing its expression with
  if(tic >= 0, expr, expr), against a 24.5 s baseline: the player's move
  test 36.3 s, the plane pass 32.5 s, the chase's move test 29.0 s, the sight
  check 26.3 s, the weapon sprites 26.3 s. The three that cost the most are
  the three deepest chains, and they are not the three largest.
• Reading the columns no stage writes at the top of the statement, which
  takes 25 KB and 297 carried column names out of it: 24.61 s against 24.53 s.
• Putting stage_when's gate in a stage of its own, so the gated value is
  a subquery column rather than a copy: 24.21 s. That is most of the 46 KB of
  copying in the statement.
• The 88 scalar subqueries the statement holds, which the analyser
  evaluates: 0.16 s for all of them, the largest 11.3 ms.

So neither the carried names, nor the copying the stage cut exists to bound,
nor the constant subqueries are where the time goes. The lambda depth of the
big expressions is.

Measured and did not work, so nobody repeats them

• COPY_LIMIT in sim/mod.rs at 400, 500, 600 and 700: 29.01 s, 26.81 s,
  26.11 s and 26.03 s against 24.53 s at the committed 800. native: fire the weapon the tic command asks for #449 measured
  1,000 and 1,400.
• Cutting a stage on the whole stage's expanded size rather than on one
  binding's copying, at 20,000, 12,000 and 8,000 bytes: 24.91 s, 27.17 s,
  28.38 s. Giving any binding over 8,000 or 4,000 bytes a stage of its
  own: 25.75 s and 26.53 s. Every way of adding stages costs.
• Duplicating a small value rather than binding it, so a chain loses a
  level, at 100, 400, 1,200 and 4,000 bytes of duplication: 23.33 s, 26.14 s,
  35.63 s, 59.38 s. Worth about a second at 100, and native: build a chain's levels as a pipeline of maps #450 takes more than that
  without duplicating anything.
• optimize_functions_to_subcolumns, enable_optimize_predicate_expression,
  optimize_substitute_columns, optimize_syntax_fuse_functions and
  optimize_if_chain_to_multiif turned off: 27.38 s, 25.91 s, 37.31 s,
  25.19 s, 25.03 s.
• A fan of traces whose fractions never tie does not check the order a
  block walks its lines and its things in. Two breaks of native: add the things a trace crosses to the blockmap walk #451 passed against
  the first version of its fan. Its mobjs are now centred on the points the
  lines-only walk crossed, so each ties with the line it stands on, and the
  test fails unless a tie is in the answer.

What is left, in the order gametic 143 dictates

1. P_AimLineAttack and P_BulletSlope: PTR_AimTraverse as a fold over the
   intercepts native: add the things a trace crosses to the blockmap walk #451 now produces, carrying the two slopes and the target, and
   the three angles P_BulletSlope tries in turn.
2. A_FireShotgun: seven P_GunShots, each a P_LineAttack over the same
   walk with PTR_ShootTraverse. The reference run's random-call log names 56
   draws for it in seven groups of eight.
3. P_SpawnPuff and P_SpawnBlood, which take next_seq and a new slot, and
   P_DamageMobj with the pain and death states. The mobj list goes from 264
   slots to 271 at gametic 143.
4. The rounds, for the movers that cannot be made independent. demo3 reaches
   its first such pair at gametic 82 and the fields it decides happen to
   agree, which is why the run passes through it; the tic is marked unresolved
   and sim_parity_live pins that.

Written mostly by Claude Opus 5.

[MarcusKainth]

The aim and the shot land as primitives, and the first divergence stays at 143

A stack of two, both drafts: #452, #453. Measured on ClickHouse 26.7.5.10 (the
digest in docker-compose.yml) in a throwaway container of this lane's own on
port 18132, docker/clickhouse/config.d and users.d mounted as the compose
file does. The machine was not quiet: other lanes' containers and this
lane's own test runs were on it through the session, so FunctionExecute from
system.query_log is beside every timing. scripts/machine-lock.sh was held
for every arm below.

Where the run stops

$ clickdoom native diff 300 --probe refemu/reference_traces/demo3/probe.9a6a47d01119.tsv
clickdoom: error: tic 143 game slot 0 prndindex: 198 against the probe's 253

Unchanged. Neither pull request moves a tic: both add a primitive nothing calls
yet, and the resident statement is byte for byte the one main holds.

$ shasum -a 256 resident-main.sql resident-shoot.sql
368639f2a7ba8a7e381278bc822d6d05dedb25be7d8b4b58bc0088514a855f6d  resident-main.sql
368639f2a7ba8a7e381278bc822d6d05dedb25be7d8b4b58bc0088514a855f6d  resident-shoot.sql

What each gives

#452, P_AimLineAttack and P_BulletSlope. PTR_AimTraverse as a fold
over the intercepts #451 produces, carrying the two slopes and the target, and
the three angles P_BulletSlope tries. The three tries are asked together: each
is pure, answers 0 without a target and moves nothing, so taking the first that
found something gives what the engine's early return gives.

#453, P_LineAttack. PTR_ShootTraverse as a fold over the same
intercepts, answering whether the shot ended on a line or on a thing, where the
puff or the blood goes, and which special lines it crossed. It stops short of
the effects, because P_SpawnPuff, P_SpawnBlood, P_DamageMobj and
P_ShootSpecialLine all draw numbers or add thinkers and read the same answer
whatever asked for it.

What the tic costs

The statement is byte for byte main's on both, so there is one build here.
What follows is that one statement measured repeatedly, which is what says
whether it still fits. Idle is gametics 2 to 61, door 73 to 106, chasing 108 to
124, each window issued once over about 240 rows after a native load --fresh
and a native diff 130, timed as query_duration_ms less the parse, analysis,
plan and pipeline microseconds the query log reports.

                tics    bytes  resident  analysis  wall/tic  fexec/tic
idle             240   651931    646979    19.33s   20.21ms      12170
idle             240   651931    646979    20.73s   20.38ms      12170
idle             240   651931    646979    22.73s   24.97ms      12170
door             272   652531    646979    19.77s   70.45ms      60386
door             272   652531    646979    20.83s   74.38ms      60386
chasing          255   652458    646979    19.85s   22.58ms      12449
chasing          255   652458    646979    20.79s   23.56ms      12449
chasing          255   652458    646979    33.57s   24.00ms      12449

FunctionExecute is exact and holds still: 12,170 on an idle tic, 60,386 on a
door tic and 12,449 on a chasing one, whatever the clock said. The analysis ran
19.3 to 33.6 s over eight readings of one statement, so the spread on this
machine is wider than anything either pull request could add, and the budget
this stack has to fit is the one main already has. FIRST_TIC_TIMEOUT in
driver/src/cli/native/diff.rs is 120 s and the runner took 91 s for a first
tic that measured 23.2 s here, a ratio of 3.9. The idle tic is 20.2 to 25.0 ms
against the 20 ms this issue names.

The first issue of a repeated window is not a reading. Its first pass reads
the rows the native diff before it left, and the passes after it read what the
pass before wrote. The idle window measured 14,737 executions the first time it
was issued and 12,170 every time after.

Both primitives already point at what the engine shoots

Read out of the state row native diff leaves at gametic 142, with the player's
own mobj as the shooter and its own angle:

$ curl ... 'WITH ... SELECT tic, aimed[1].1 AS slope, aimed[1].2 AS target,
           <line_attack> AS reached FROM lanew_bench.native_state WHERE tic = 142'
142     0       258     [(2,258,12209775,6839830,2359296,[])]

P_BulletSlope answers slope 0 and mobj slot 258, and the shot down that slope
ends on the same slot. The probe's own rows say 258 is what the shotgun hits on
the next tic:

$ curl ... 'SELECT gametic, m_health[258], m_state[258], length(m_x)
           FROM lanew_bench.probe_state WHERE gametic IN (142,143,144)'
142     150     482     264
143     90      488     271
144     90      489     271

Its health falls by 60 over the seven shots, its state moves to its pain frame,
and the list grows by the seven blood spawns.

What the fan could not reach, and what was done about it

The fan over E1M7's own geometry does not reach every branch, so it carries
worlds that do, and the break-tests below fail on those worlds only.

• A line whose two sides carry the same floor or the same ceiling is one the
  engine leaves the window alone for. Over the level's own geometry the opening
  that would narrow it sits at the height the target stands on, so the narrowing
  is dominated by the line where the floor really steps and the test decides
  nothing. Over one flat floor, with the mobjs standing off it, it decides.
• A line with a second side and no ML_TWOSIDED. E1M7 has none: every one
  of its 958 lines carries the flag exactly when it has a second side. Clearing
  the flag on every other two-sided line puts the test back.
• The sky. E1M7 shows it over six of its 170 sectors and the fan ends on
  one of those lines ten times, so a second arm puts the sky over every sector.

What is left, in the order gametic 143 dictates

1. P_SpawnMobj: a spawned thing takes next_seq and a new slot, the mobj
   list's new length, and its own state and tic draws. P_SpawnPuff and
   P_SpawnBlood are that plus three draws each.
2. A_FireShotgun's seven P_GunShots as a fold, because how many numbers each
   shot draws depends on what the one before it hit: three for the shot, four
   more for the puff or the blood, and one for P_DamageMobj. The reference
   run's random-call log names 56 draws for gametic 142 in seven groups of eight,
   every one of them ending on a thing.
3. P_DamageMobj and P_KillMobj, with the pain and death state changes and the
   removal at the tic's compaction.
4. The rounds, for the movers that cannot be made independent. demo3 reaches its
   first such pair at gametic 82 and the fields it decides happen to agree, which
   is why the run passes through it; the tic is marked unresolved and
   sim_parity_live pins that.

Written mostly by Claude Opus 5.

[MarcusKainth]

The spawn and the damage land as primitives, and 143 still needs the wiring

A stack of two, both drafts: #454, #455, on top of #452 and #453, rebased onto
main at 7a44e3e. Measured on
ClickHouse 26.7.5.10 (the digest in docker-compose.yml) in a throwaway
container of this lane's own on port 18132, docker/clickhouse/config.d and
users.d mounted as the compose file does. The machine was not quiet:
other lanes' containers and this lane's own test runs were on it through the
session, so FunctionExecute from system.query_log is beside every timing.
scripts/machine-lock.sh was held for every arm.

Where the run stops

$ clickdoom native diff 300 --probe refemu/reference_traces/demo3/probe.9a6a47d01119.tsv
clickdoom: error: tic 143 game slot 0 prndindex: 198 against the probe's 253

Unchanged. Neither pull request moves the tic, and neither could. Gametic
142's 56 draws come in seven groups of eight, and a group is three for the shot,
four for the blood and one for the damage. The count of a group depends on what
its shot reached, so the seven shots have to be one fold that reads the spawn
and the damage; until that fold exists no part of the shot can run. This stack
is the two primitives the fold will call.

The draw layout against gametic 142

Neither pull request moves the tic, so the layout the two are built on is
checked against the reference run directly. The probe's random index stands at
194 going into gametic 142, and the log records 56 draws there in seven groups
of eight: three for the shot, four for the blood and one for the damage. Taking
the first of each group as 5 * (P_Random() % 3 + 1), over the rndtable the
level load put in the database:

$ curl ... 'WITH rnd AS <the loaded rndtable>,
  prnd AS (SELECT prndindex FROM probe_state WHERE gametic = 142),
  arrayMap(i -> 5 * (rnd[1 + bitAnd(prnd + 8 * i + 1, 255)] % 3 + 1), range(7)) AS damage
SELECT prnd, damage, arraySum(damage), bitAnd(prnd + 56, 255)'
194     [15,10,15,10,5,5,10]    70      250

The probe's own rows say the shot took 70 health off two things, and that its
index reaches 250 before the three draws the monsters make after it:

$ curl ... 'the mobjs whose health moves between gametic 142 and 143'
[(118,60,50),(258,150,90)]

$ curl ... 'SELECT gametic, prndindex FROM probe_state WHERE gametic IN (142,143)'
142     194
143     253

70 off two things, and 250 plus the three chase draws the log names is 253. So
the group size and the order inside it are the ones this stack is built on.

What each gives

#454, P_SpawnMobj, P_SpawnPuff and P_SpawnBlood. One draw for
lastlook, plus the three a puff or a blood spot makes: two that jitter the
height and one that shortens the wait. A spawn takes the count of draws made
before it rather than a running index of its own, which is what keeps the caller
a fold over the shots rather than over the draws. P_SetMobjState writes the
frame's own wait back over the shortened one wherever the damage or the range
moves the spot.

#455, P_DamageMobj and P_KillMobj. One draw for the pain chance where
the target lives and for the death frame's wait where it does not, and one more
where the hit may knock it over. Both are decided before either number is read,
so the offsets stay known ahead of the fold.

What the tic costs

                tics    bytes  resident  analysis  wall/tic  fexec/tic
idle             240   655415    648933    19.87s   19.89ms      12171
idle             240   655415    648933    19.79s   19.87ms      12171
idle             240   655415    648933    19.52s   19.66ms      12171
door             272   656015    648933    20.15s   69.45ms      60386
chasing          255   655942    648933    19.54s   21.57ms      12449

Against main at 7a44e3e, whose resident statement is 645,449 bytes and which
ran 12,170 executions on an idle tic, 60,386 on a door tic and 12,449 on a
chasing one. The whole of this stack is 3,484 bytes of constants and one
function execution a tic: the six mobjinfo columns and the five thing types
and frames the spawn reads, evaluated once for the row. #455 adds nothing at
all, because its own tables are a list of its own rather than part of
sim::constants, and its resident statement hashes the same as #454's.

The analysis is 19.5 to 20.2 s against the 21 s this lane holds to, and against
the 19.3 to 33.6 s that eight readings of main's own statement spanned on this
machine in the session before.

A repeated window has to be issued until its count settles. The first
issues read the rows the native diff before them left, and each pass rewrites
what the next one reads. The idle window measured 14,825 executions on its
second issue and 12,171 on every issue after.

Which draw the height jitter subtracts from which

z += ((P_Random() - P_Random()) << 10) leaves the operand order to the
compiler and the two draws differ on most indices, so the order decides where a
blood spot goes. The pinned ELF settles it rather than the C:

8001fc04:  jalr  ... # 800144e0 <P_Random>
8001fc0c:  mv    s0,a0                   # the first answer is kept
8001fc10:  jalr  ... # 800144e0 <P_Random>
8001fc20:  sub   a0,s0,a0                # first - second
8001fc24:  slli  a0,a0,0xa
8001fc28:  add   a2,a0,a2

P_SpawnBlood emits the same four instructions. The earlier draw is the left
operand, which is what the generator writes, and a unit test reads the two
offsets back out of the generated text so a swap fails. The same disassembly
pins the two momz values and the MELEERANGE the puff's spark is tested
against.

Two gaps a primitive with no caller would have hidden

Neither pull request runs in the tic, so nothing but its own fan could have
found either of these. Both were caught by widening the fixture until the case
was in it.

A frame's own routine. P_SetMobjState runs the routine of the frame it
enters, and the frames P_DamageMobj enters carry ten routines between them
across the whole of mobjinfo. Fourteen see frames carry A_Chase, which
draws. The damage now stops on any routine but A_Pain and A_Scream, which
only make a noise, and a unit test lists the ten out of the committed tables so
an eleventh fails rather than slipping past.

The wait on a frame the thing already stands in. The generator kept the
wait a thing was carrying wherever the frame it entered was the frame it was
already in. The engine writes the entered frame's own wait whatever the thing
was in before. It only shows on a monster hit twice in a row, so the world now
holds one of each type standing in its own pain frame.

A guard nothing reaches, recorded rather than removed

P_SpawnPuff and P_SpawnBlood hold the shortened wait at one, and nothing
reaches that: the draw takes at most three tics off and both frames the two
spawn in wait four and eight. Dropping the hold leaves the whole fan passing.
The line stays, and the_shortened_wait_cannot_run_below_one reads the two
frames' waits out of the committed states table so the claim is checked rather
than assumed.

What is left, in the order gametic 143 dictates

1. The seven shots as a fold. Each reads the world the one before left, asks
   line_attack where it ends, spawns the puff or the blood there and damages
   what it reached, and the draws of each sit at the offset the shots before it
   left. A_FirePistol's single shot goes through the same path.
2. The mobj list growing. A spawned thing takes a slot at the end of the
   list and born_column says what each of its columns holds; the tic's own
   compaction already renumbers every pointer. Two columns it answers nothing
   for: the identity a thinker takes and the order its sector lists it in are
   the fold's to hand out, the way the loader does, because a puff and a blood
   spot carry MF_NOBLOCKMAP and are still in their sector's list newest
   first, which a later P_ChangeSector and the noise walk read. The fold's
   test seeds two spawns in one sector.
3. The debris' own thinker. A blood spot spawns with momz and carries no
   MF_NOGRAVITY, and both it and a puff cycle to S_NULL and are removed. The
   thinkers mark a tic unresolved for a thing with momentum and for a cycle that
   removes one, so gametic 144 needs P_ZMovement and the removal before it can
   agree even once 143 does.
4. The rounds, for the movers that cannot be made independent.

Written mostly by Claude Opus 5.

[MarcusKainth]

The ask a monster's attack needs, and where the shot fold stands

#454 has merged. #455 is rebased onto main and carries two fixes review found,
both recorded in its own body: the see frame was being decided on the frame the
thing came in with rather than the frame the pain move left, and a hit that does
not land was stopping a charging lost soul's momentum.

What the ask shape is short of

P_DamageMobj takes an inflictor and a source separately, and this stack's ask
carries one slot for both. A hitscan passes the same mobj twice, so nothing here
can tell them apart, and a missile cannot:

P_DamageMobj (target, inflictor, source, damage)

The inflictor is what the push angle is measured from, R_PointToAngle2 between
its position and the target's. The source is what the target then chases, and
whose readyweapon decides whether a chainsaw holds its target in reach. For
A_TroopAttack's fireball those are the missile and the monster that fired it,
standing in different places, so the angle a hit throws the target at is the
missile's and the thing it turns on is the monster's.

So when the monsters' attacks arrive the ask needs an inflictor field beside its
source, and dm_angle reads the inflictor while dm_chases and the chainsaw
test read the source. Nothing on the player's own shot reaches it, which is why
the field is not in #455.

What is left, in the order gametic 143 dictates

1. The seven shots as a fold. Each reads the world the one before left, asks
   line_attack where it ends, spawns the puff or the blood there and damages
   what it reached, with the draws of each at the offset the shots before it
   left. A_FirePistol's single shot goes through the same path.
2. The mobj list growing. A spawned thing takes a slot at the end and
   born_column says what each of its columns holds. It answers nothing for two:
   the identity a thinker takes and the order its sector lists it in are the
   fold's to hand out, the way the loader does, because a puff and a blood spot
   carry MF_NOBLOCKMAP and are still in their sector's list newest first, which
   a later P_ChangeSector and the noise walk read. The fold's test seeds two
   spawns in one sector.
3. The debris' own thinker. A blood spot spawns with momz and carries no
   MF_NOGRAVITY, and both it and a puff cycle to S_NULL and are removed. The
   thinkers mark a tic unresolved for a thing with momentum and for a cycle that
   removes one, so gametic 144 needs P_ZMovement and the removal before it can
   agree even once 143 does.
4. The monsters' own attacks, which is where the inflictor field lands.
5. The rounds, for the movers that cannot be made independent.

Written mostly by Claude Opus 5.

[MarcusKainth]

The shot lands and gametic 143 is down to seven fields, over the analysis budget

#456, a draft on main. Measured on ClickHouse 26.7.5.10 (the digest in
docker-compose.yml) in a throwaway container of this lane's own on port 18132,
docker/clickhouse/config.d and users.d mounted as the compose file does.
The machine was not quiet. scripts/machine-lock.sh was held for every arm.

Where the run stops

$ clickdoom native diff 145 --summary --probe refemu/reference_traces/demo3/probe.9a6a47d01119.tsv
m_x     mobj slot=118 first_tic=143  ours=8272000   theirs=8245976
m_y     mobj slot=118 first_tic=143  ours=-4310912  theirs=-4388589
m_z     mobj slot=265 first_tic=143  ours=2384896   theirs=2515968
m_frame mobj slot=118 first_tic=143  ours=2         theirs=7
m_momx  mobj slot=118 first_tic=143  ours=-26024    theirs=-23585
m_momy  mobj slot=118 first_tic=143  ours=-77677    theirs=-70395
m_momz  mobj slot=265 first_tic=143  ours=131072    theirs=65536

Seven fields over two slots, where the summary listed 37 before. prndindex
reaches 253, the mobj list reaches 271 slots, the two damaged things end on 90
and 50 health, the spectre is in its pain frame with MF_JUSTHIT and a threshold
of 100, the shell is spent, extralight is 1, the flash sprite is in frame 30
with a tic already off it, and the player's mobj is in its second attack frame.

Slot 118's momentum is the thrust exactly: −26024 × FRICTION is −23584 against
the probe's −23585. Slot 265 is a blood spot and its height is out by one
P_ZMovement step of its own momz. A thing added during P_PlayerThink is on
the list before P_RunThinkers walks it, so a spawn's own thinker runs on the
tic it was spawned, which is why the debris cannot agree until that lands.

It does not fit, and the fix is a shape rather than a knob

                tics    bytes  resident  analysis  wall/tic  fexec/tic
idle             240   822000    815343    63.08s   35.42ms      12514
door             272   822600    815343    37.80s   73.74ms      60728
chasing          255   822527    815343    37.57s   25.05ms      12792

main is 648,933 bytes, 19.5 to 20.2 s and an idle tic of 19.9 ms. So the
analysis goes to 37.6 to 63.1 s against the 21 s this lane holds to, and the idle
tic to 35.4 ms against 20 ms. The runner's ratio of 3.9 puts a 37.6 s first tic at
about 147 s against a 120 s FIRST_TIC_TIMEOUT, so native_diff_live would time
out. #456 is a draft that must not merge as it stands.

With the shot's own walk stubbed inside the fold and everything else standing,
the idle tic drops to 22.49 ms and the analysis stays at 39.67 s. The run time
is the walk executing and the analysis is the statement's size, and the two want
different fixes.

The size is two whole P_PathTraversees: P_BulletSlope holds one through
aim_line_attack and each shot another through line_attack, on top of the three
the statement already had. Both walk from the same point and differ only in where
the trace ends and which traverser folds over the intercepts, so one walk with
the traverser picked per ask is the cut, and it is a change to shoot.rs's
shape.

What is left

1. One P_PathTraverse shared between the aim and the shot, to bring native: fire the shotgun's seven shots #456 inside
   the budget.
2. P_XYMovement for a thing that carries momentum, with mt_still lifted for
   it. That is slot 118's four fields.
3. The debris' own thinker: P_ZMovement, the advance to S_NULL and removal
   with the compaction. That is slot 265's two.
4. The monsters' own attacks, where the damage ask needs its inflictor field.
5. The rounds, for the movers that cannot be made independent.

Written mostly by Claude Opus 5.

[MarcusKainth]

Three cuts at the shot's cost: 63 s to 33 s of analysis, 35 ms to 21 ms a tic

#457 below #456, both drafts. Measured on ClickHouse 26.7.5.10 (the digest in
docker-compose.yml) in a throwaway container of this lane's own on port 18132,
docker/clickhouse/config.d and users.d mounted as the compose file does.
The machine was not quiet. scripts/machine-lock.sh was held for every arm,
and the two builds below were measured back to back in one session on one
container.

build   window    resident  analysis  wall/tic  fexec/tic
main    idle        650577    20.07s   19.94ms      12248
shot    idle        799665    32.74s   20.86ms      12468
main    chasing     650577    20.42s   22.11ms      12449
shot    chasing     799665    33.94s   23.33ms      12745

The shots cost 12.7 s of analysis, 0.9 ms of an idle tic and 220 executions.
Against the first version of #456: 37.6 to 63.1 s, 35.42 ms and 815,343 bytes.

What each cut was worth

One walk for the aim and the shot (#457). P_PathTraverse is the same for
both and only what folds over the intercepts differs, so the walk was in the
statement twice for one question. The ask carries a flag and the two traversers
became one fold that picks on it.

The aim moved inside the shots' own fold. P_BulletSlope sat outside it, so
an idle tic paid for its per-ask arrayMap: a map evaluates its body over an
empty array and a fold does not. Step 0 of the fold now aims and every step after
it shoots, so a tic that fires nothing folds over an empty range. That is the
whole of the idle tic's 35.42 to 20.86 ms. Attributed: with the shot's own walk
stubbed inside the fold and the aim still outside, the idle tic measured 22.49 ms
and the analysis did not move, so the run time was the walk executing and the
analysis is the statement's size.

Depth is not where this one goes. The deepest lambda nesting in the resident
statement is 22 with the shots and 22 without them, and the shot fold's own
subtree tops out at 18. It adds no depth. Writing the traverser's answer one
member at a time rather than as six whole tuples took 333 bytes and nothing
measurable, which is this issue's own earlier finding: ClickHouse deduplicates
identical subexpressions, so a builder that repeats itself costs the parser and
not the run.

Where that leaves it

Analysis 32.7 to 33.9 s against the 21 s budget and the 26 s that
FIRST_TIC_TIMEOUT's 120 s allows at the runner's 3.9 times. Over the ceiling,
so #456 must not merge and PR 2 and PR 3 are not started on top of it.

What is left is volume: 799,665 bytes against 650,577, and the largest single
piece is the fourth P_PathTraverse the statement now holds, beside the player's
move, the plane clip and the chase. Depth and text size are the two levers this
issue has recorded, and neither is worth 12 s here.

What is left

1. Something that takes 12 s out of one more blockmap walk, or a decision that
   the budget moves. Nothing measured so far reaches it.
2. P_XYMovement for a thing that carries momentum. That is slot 118's four
   fields at gametic 143.
3. The debris' own thinker: P_ZMovement, the advance to S_NULL and removal
   with the compaction. That is slot 265's two.
4. The monsters' own attacks, where the damage ask needs its inflictor field.
5. The rounds, for the movers that cannot be made independent.

Written mostly by Claude Opus 5.

[MarcusKainth]

The depth hypothesis does not hold, and the walk costs 29 times more inside the statement than alone

Measured on ClickHouse 26.7.5.10 (the digest in docker-compose.yml) in a
throwaway container of this lane's own on port 18132,
docker/clickhouse/config.d and users.d mounted as the compose file does.
The machine was not quiet. scripts/machine-lock.sh was held for every arm,
and each group below was taken back to back in one session on one container.

The walk at two depths

shoot::traverse over one ask, with the level's constants around it, analysed on
its own and again under eighteen nested single-parameter arrayMaps over a
one-element array, with the ask list threaded through every level so the walk is
genuinely inside them. The walk's own internals reach lambda depth 11 alone and
29 wrapped, so its nodes moved by the eighteen levels asked for.

             QueryAnalysisMicroseconds
shallow      237609   230155   220456
deep         286463   284074   290257

1.25 times, not two. Per the gate this stops here.

What that leaves unexplained

The whole walk analyses in about 0.23 s standalone and 0.29 s at depth 18.
Adding the shots to the resident statement costs 13.4 s. So the walk's own
nodes are under 2% of it at any depth.

Splitting that 13.4 s, all three arms back to back on one container:

build                resident  analysis  idle wall  idle fexec
main                   650577    20.02s    19.82ms       12248
shot, walk removed     739821    26.14s    26.60ms       12463
shot, full             799665    33.45s    21.22ms       12471

The walk is +59,844 bytes and +7.3 s; everything else the shots add is
+89,244 bytes and +6.1 s. So the same expression that analyses in 0.23 s on
its own costs 7.3 s once it is one binding among the tic's others, a factor of
about 29, and depth accounts for 1.25 of it.

The wall figures carry this machine's noise, which is why the two shot arms read
26.6 and 21.2 ms for the same 12,463 and 12,471 executions. The execution counts
and the analysis microseconds are the exact ones.

What this says about step 2

Hoisting the world-invariant part of the walk out of the fold would cut the
walk's nodes and their depth. Both are measured here as small: the walk is 7.3 s
of the 13.4, and moving its nodes eighteen levels is worth 1.25 times. So the
hoist would have to cut the walk's presence in the statement rather than its
depth to be worth the redesign, and nothing measured here says how much of the
7.3 s survives that.

The 29 times amplification is the thing worth understanding before any more
shape is spent on this. It is not depth, it is not text repetition, and it is not
the copying the stage cut bounds. On this evidence the next measurement is what
stages() does with a binding of this size rather than another rewrite of the
walk.

Written mostly by Claude Opus 5.