Repository navigation
[pipeline] Keep the symbol inventory's analysis facts through other selections - #1328
Merged
Merged
Conversation
…elections A selected function that is not a symbol start adds its own entry to its run's inventory, which keys different analysis facts. The session cache held one set, so such a run replaced the facts every symbol start shares and warming builds, and the next symbol start replaced them again. PE images hit this: the worker's listing finds functions no symbol starts, and an address-order pass over a Rust DLL replaced the facts 21 times, rebuilding callee graphs warming had built. PipelineAnalysisCache keeps a selection that adds entries in its own slot beside the symbol inventory's facts. Each run still uses only facts built for its exact inventory, and reset() retires both. Warming replaces facts for another symbol inventory, which serve no later run, instead of refusing. The class's doc comment, which AnalysisCacheWarmup had separated from it, is back on the class.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A selected function that is not a symbol start adds its own entry to its run's entry inventory, so its run keys different analysis facts. The session's
PipelineAnalysisCacheheld one set of facts, so such a run replaced the facts that every symbol start shares and that warming builds. The next symbol start then replaced them again.PE images hit this: the worker's listing finds functions that no image symbol starts. Decompiling every function in address order replaced the facts 21 times in a Rust Windows DLL (338 functions) and 17 times in MSVC
xcpt4(306). The warmed cache was gone by the DLL's 20th function. Each replacement made the following functions rebuild callee graphs that warming had built, which is why some tiny tail-calling functions took 85 ms. The Go and Mach-O lists measured had no such functions.The cache now keeps the facts of a selection that adds entries in their own slot, beside the symbol inventory's:
matchesis unchanged).reset()retires both slots.neverd_warm_functions_jsonis now available on every x86-64 image.The doc comment of
PipelineAnalysisCachehad drifted aboveAnalysisCacheWarmup(#1271) and is back on the class. The architecture guide and the warming API comment describe the new rule.Measurements
Release build, NEVERD_THREADS=4, pinned to 4 CPUs. Each run decompiled every function in address order in one worker after warming. Runs alternated between the
devlibrary (#1324) and this change.devrust-eh/x86_64-pc-windows-msvc/abort/o2/cdylib, 338 functions), 3 runsxcpt4FH3, GS, O2 (306), 3 runsxcpt4FH3, no GS, O0 (420)Peak RSS:
xcpt4O2: 286–308 MB before, 319–336 MB after. The symbol inventory's cache is no longer discarded 17 times, so it grows to the size it reaches on images without such selections, within the cache's own bounds.Tests
PipelineOutcome.SelectionsOutsideTheSymbolInventoryKeepTheirOwnFacts:reset()retires both.WarmingKeepsAnalysisKeptForOtherEntries, which pinned the old rule.PipelineOutcome.AnalysisCacheBindsExactEntryInventoriesnow checks both slots. A run builds new facts exactly when its own inventory changed, including after verified-entry changes, and leaves the other kind alone.SessionCAPITest.AnalysisCacheSurvivesSelectingAnEntryNoSymbolStarts: preparing padding that no symbol starts keeps the session's facts, and the first function reads back the same. The test asserts that no symbol starts there, because the session gives every call target a symbol.NeverDSessionCAPITests165/165,NeverDPipelineOutcomeTests42/42,NeverDX87CallGraphTests57/57,NeverDNoReturnTests37/37 andNeverDJumpTableTests544/544 pass. So do the worker warming and PE tests with the real engine (4/4).scripts/check_clang_format.pyandscripts/check_docs_i18n.pypass.dev:NeverDSessionCAPITests167/167,NeverDPipelineOutcomeTests42/42 and the worker tests 4/4 pass.