[gmh5225]

Summary

On a large image, most of a decompile goes to its callees.

• computeCallRegisterEffects takes the body of every callee in a closure four levels deep: 50 to 220 functions on ntoskrnl. It computes each one's register effect.
• Each body it reuses from the analysis cache is first validated, by proving every call the body makes against the x87 call graphs.
• Warming kept bodies until 256 MiB, which is 1154 of ntoskrnl's 26 thousand functions, and then stopped.

So most callees were lifted again in every decompile, and every reuse proved the same calls again.

[lowir] Prove each call of a reused body once per cache

• For each call of a reused body, validatedSummary ran usesContext and effect, and each walks the graphs of the callee's callees. What it reads is whether the call refuses the body.
• That answer is now kept per target in X87CallGraphCache::refusesCall, under the request's context.
• It is kept together with the no-return answers of every incomplete lift its queries read. It is reused while the prover gives those answers again, the same way answersHold keeps an incomplete-lift graph valid.
• A query that read an incomplete lift whose answers went unrecorded keeps nothing.
• warmCallGraphs also keeps this answer for each function it warms.

[pipeline] Keep each callee's register facts per analysis cache

• What computeCallRegisterEffects reads of a callee's body is kept per analysis cache, in CalleeFactsCache (256 MiB, least recently used out):
  • its joined register effect;
  • the fixed arguments of its formatted calls;
  • whether it forwards to the guard dispatcher.
• The effect read the fixed arguments that the run had collected for each of its calls, including the absent ones. Another run reuses it only where that run gives each of those calls the same arguments. Otherwise the body is taken again, as happens when another body shares one of its calls.
• Warming keeps the callee facts of every page. Once no body fits, a page lifts each function once, for its facts. It no longer builds raw graphs and bodies that the cache would refuse.
• The pointer-variable callees (collectPointerVariableCallees, an image-wide scan) are found once per analysis cache rather than once per run.
• joinPlainSteps releases the steps it joins, so a kept effect takes a few kilobytes. Join each run of plain register steps into one #1412 kept their capacity.

[worker] Warm every function after the body cache is full

• The worker stopped warming at "full". It now goes on to the last function, because each page still keeps callee facts.
• The final state is "full" or "complete", as before.
• neverd_warm_functions_json documents the change.

Measurements

Setup:

• Release build, NEVERD_THREADS=4, pinned to 4 CPUs.
• C view, image opened with warming.
• Same load for both arms. The base is dev at Join each run of plain register steps into one #1412, built from the same tree.

ntoskrnl with ntkrnlmp.pdb: 20 functions spread over the image, timing the 19 after the first. Two runs per side, alternating:

	dev	This change
Median	50 / 49 ms	22 / 23 ms
Total	1.23 / 1.20 s	0.47 / 0.49 s
CPU	1.47 / 1.46 s	0.48 / 0.48 s
First function after warming	196 / 154 ms	22 / 23 ms
Warming, in idle time	7 s	54 / 55 s
Peak RSS	719 / 726 MB	995 / 992 MB

• Every callee of the 20 closures comes from its facts: 1470 of 1470.
• The facts of all 26,307 functions take 123 MiB.
• Warming steps take a median of 5.6 ms and 46 ms at the 99th percentile. The longest step after the first takes 298 ms. The first step builds the analysis cache's indexes and takes 1.3 s, as before.

Other images, CPU time of the decompiles:

Image	dev	This change
Go 1.21 stripped, 100 small functions	0.70 s	0.36 s
Rust Windows DLL, every function (338)	6.00 s	5.69 s
Rust x86_64 Linux probe, every function (565)	20.73 s	20.28 s
MSVC x64 xcpt4, every function (306)	1.78 s	1.74 s

The sources are identical on every function of all five images, in both ntoskrnl runs.

I also ran a diagnostic that is not part of this change. It recomputes, from scratch, every refusal and every reused callee's facts beside the kept answer, and aborts on any difference. Over ntoskrnl's full warming and the 20 decompiles it found no difference:

• 26 thousand refusals;
• 1470 reused callees.

Tests

• New X87CallGraphCache.AReusedBodyProvesEachCallOnceWhileItsAnswersHold:
  • A reused body's second validation reads no graph and asks the prover again.
  • Once the answer of the incomplete lift behind the call changes, the call is proved again.
• New SysVCallContract.ASharedCallTakesTheFormatAnotherBodyGivesIt:
  • a jumps into the formatted call of b.
  • A run that lifts b before a must not reuse the facts a got without the format.
  • The run gives the same entry reads, may-write masks and stack arguments as a run without the cache.
  • The test fails when the check of the consulted reads is dropped.
• NeverDLiftTests, all 10 shards: 7345/7345.
• NeverDSessionCAPITests: 187/187, with its ntoskrnl tests.
• Worker and allocator tests: 37/37.
• NeverDPipelineOutcomeTests: 42/42. These include the warmed and cached selection tests.
• Call contract tests:
  • SysV: 26/26.
  • AAPCS64: 11/11.
  • I386: 16/16.
  • Win64: 12 of 13. Win64CallContract.AThunkJumpingThroughTheSlotForwardsTheArguments fails on dev already, from the stub redirect in [highc][llvmc][libc] Keep C library copies apart; spell traps and stubs as headers do #1403.
• scripts/check_clang_format.py passes.