60 refactoroptimisation reduction of llvm analysis algorithmic cost through shared traversals and intermediate artifacts

.ctrace-analyzer.cfg

@@ -11,6 +11,7 @@ buffer-model=models/buffer-overflow/generic.txt
 analysis-profile=full
 jobs=auto
 compile-ir-cache-dir=.cache/compile-ir
+compile-ir-format=bc
 
 # Output behavior
 warnings-only=false

CMakeLists.txt

@@ -67,7 +67,10 @@ endif()
 set(STACK_ANALYZER_SOURCES
     src/analyzer/AnalysisPipeline.cpp
     src/analyzer/DiagnosticEmitter.cpp
+    src/analyzer/HotspotProfiler.cpp
+    src/analyzer/IRFactCollector.cpp
     src/analyzer/LocationResolver.cpp
+    src/analyzer/PerFunctionInstructionCache.cpp
     src/analyzer/ModulePreparationService.cpp
     src/app/AnalyzerApp.cpp
     src/cli/ArgParser.cpp

README.md

@@ -217,6 +217,7 @@ Ready-to-adapt workflow examples:
 ./stack_usage_analyzer main.cpp --compile-commands=build/compile_commands.json
 ./stack_usage_analyzer main.cpp -I./include --only-file=./main.cpp --only-function=main
 ./stack_usage_analyzer main.cpp --dump-ir=./debug/main.ll
+./stack_usage_analyzer main.cpp --compile-ir-format=ll
 ./stack_usage_analyzer a.c b.c --dump-ir=./debug
 ```
 
@@ -240,7 +241,8 @@ Ready-to-adapt workflow examples:
 --resource-summary-cache-dir=<path> sets cache directory for cross-TU resource summaries (default: .cache/resource-lifetime)
 --resource-summary-cache-memory-only keeps cross-TU summary cache in memory only (process-local, no files)
 --compile-ir-cache-dir=<path> enables dependency-aware LLVM IR compile cache for unchanged source files
---timing prints compile/analysis timings to stderr
+--compile-ir-format=bc|ll selects source compilation IR format (`bc` default, `ll` for textual LLVM IR)
+--timing prints compile/analysis timings to stderr, including aggregated hotspot ranking
 --config=<path> loads optional key=value config file (CLI flags override config values)
 --print-effective-config prints resolved runtime config to stderr
 --smt=on|off enables or disables SMT-assisted reasoning (default: off)
@@ -271,6 +273,19 @@ sanitizers, profiling) while keeping include paths and macros.
 For multi-file runs, `--jobs=<N|auto>` parallelizes input loading; with cross-TU enabled it also parallelizes summary construction.
 `--compile-ir-cache-dir=<path>` reuses compiled LLVM IR for unchanged translation units
 based on source/dependency stamps, which reduces repeated C/C++ frontend cost across runs.
+`--compile-ir-format=bc|ll` controls source compilation output format before module load:
+- `bc` (default): compile to LLVM bitcode then parse bitcode.
+- `ll`: compile to textual LLVM IR then parse text IR.
+`--timing` now also includes pipeline traversal estimates per step
+(`module/function/instruction` estimates) and by execution model
+(`subscriber-compatible` vs `independent`) to help identify repeated scans.
+It also prints a process-level hotspot summary sorted by cumulative time.
+You can control the number of printed hotspots with `CTRACE_HOTSPOT_TOP=<N>`
+(default: 20, max: 200).
+For rollout/A-B checks of the subscriber path, set:
+`CTRACE_PIPELINE_SUBSCRIBERS=1`.
+Reusable A/B benchmark helper:
+`./scripts/bench/pipeline_subscriber_ab.sh ./build/stack_usage_analyzer`.
 When inputs are auto-discovered from `compile_commands.json`, `_deps` entries are skipped by default
 to keep analysis focused on project code; use `--include-compdb-deps` to opt back in.
 
@@ -309,6 +324,7 @@ Supported keys:
 - `resource-summary-cache-dir`
 - `resource-summary-cache-memory-only`
 - `compile-ir-cache-dir`
+- `compile-ir-format` (`bc` or `ll`)
 
 Example file:
 
@@ -320,6 +336,7 @@ compile-commands=build/compile_commands.json
 analysis-profile=full
 jobs=auto
 compile-ir-cache-dir=.cache/compile-ir
+compile-ir-format=bc
 smt=on
 smt-backend=z3
 smt-rules=recursion,integer-overflow,size-minus-k,stack-buffer,oob-read

archi_plan.md

@@ -0,0 +1,32 @@
+# Plan
+
+Ce plan vise a reduire le cout algorithmique des analyses LLVM en mutualisant les parcours et les artefacts intermediaires, sans changer la semantique des diagnostics. L'approche priorise un moteur de pipeline avec dependances `requires/provides`, puis un collecteur de faits partage pour diminuer les traverses redondantes.
+
+## Decisions
+- Objectif prioritaire: reduction du temps `real` (wall-clock).
+- Strategie de delivery: migration en 2 phases.
+- Selection des analyses subscriber vs independantes: revue module par module en phase 1.
+
+## Scope
+- In: Evolution de l'architecture d'analyse (orchestration des modules, partage de donnees inter-analyses, cache d'artefacts derives, validation perf/fonctionnelle).
+- Out: Changement des regles metier de diagnostics, modification des fichiers de tests, migration complete de toutes les analyses en une seule iteration.
+
+## Action items
+[x] Phase 1 - Etablir la baseline perf cible `real` (timing + xctrace) et definir KPI primaires/secondaires (`real` prioritaire, `user/sys` en support).
+[x] Phase 1 - Ajouter une instrumentation technique des traverses (compteurs `Module/Function/Instruction` par analyse) pour mesurer le niveau de redondance avant refactor. (fait: detail par step via `Traversal estimate detail` + agregation par modele)
+[x] Phase 1 - Revoir chaque analyse pour classifier: `subscriber-compatible` vs `independante` (dataflow/fixpoint). (fait: `docs/architecture/pipeline-subscriber-classification.md`)
+[x] Phase 1 - Refactorer `AnalysisPipeline` vers un modele `requires/provides` pour declarer explicitement les dependances entre analyses.
+[x] Phase 1 - Introduire un squelette minimal `AnalysisArtifactStore` type-safe (API + invalidation par module/config), sans figer tous les artefacts.
+[x] Phase 1 - Ajouter un `IRFactCollector` comme premier pass du pipeline (parcours unique Module/Function/BB/Instruction) exposant des `IRFacts` partages.
+[x] Phase 2 - Introduire un mecanisme subscriber (Observer) pour les analyses classees `subscriber-compatible`.
+[x] Phase 2 - Migrer d'abord les analyses pilotes vers les faits partages (priorite: `StackBuffer`, `ResourceLifetime`, `Uninitialized`; candidats additionnels: `IntegerOverflow`, `GlobalReadBeforeWrite`) et mesurer le gain incremental sur `real`. (fait: `StackBuffer` + `ResourceLifetime` migrees; `Uninitialized` classee `independante` apres validation de non-regression)
+[x] Phase 2 - Etendre `PreparedModule`/cache avec artefacts derivables stables (debug index, symbol maps, type facts) avec cle versionnee. (fait: `DerivedModuleArtifacts` schema `derived-module-artifacts-v1`)
+[x] Phase 2 - Ajouter des garde-fous d'architecture: tests d'integration pipeline, checks d'invalidation cache, et assertions de non-regression diagnostics. (fait: checks `run_test.py` + isolement des caches cross-TU; suite complete verte)
+[x] Phase 2 - Valider avant/apres et rollout progressif derriere feature flag, pilote par la reduction du temps `real`. (fait: flag `CTRACE_PIPELINE_SUBSCRIBERS`, benchmark A/B, validation complete)
+
+## Validation Summary
+- Build: `cmake --build build -j4` OK.
+- Tests: `python3 -u run_test.py --jobs=4` => `Passed 1661/1661 tests`.
+- Benchmark A/B (`scripts/bench/pipeline_subscriber_ab.sh`, 5 inputs):
+  - baseline: `real=0.24s user=0.19s sys=0.03s`
+  - subscriber: `real=0.24s user=0.19s sys=0.03s`

docs/architecture/analyzer-modules.md

@@ -15,12 +15,16 @@ This document describes the module split introduced around `StackUsageAnalyzer`
 Role:
 - Entry point for module-level analysis execution.
 - Coordinates preparation, analysis passes, and diagnostic emission.
+- Declares step-level `requires/provides` artifact dependencies.
+- Tracks per-step traversal estimates (`module/function/instruction`) for timing mode.
 
 Pattern:
 - `Facade` over lower-level analysis services.
+- `Pipeline Orchestrator` with explicit dependency metadata.
 
 Why:
 - A single coordinator makes control flow explicit while avoiding a very large `StackUsageAnalyzer.cpp`.
+- Dependency declarations reduce accidental ordering coupling and make rollout of shared passes safer.
 
 ### `src/analyzer/ModulePreparationService.cpp`
 
@@ -33,6 +37,30 @@ Pattern:
 
 Why:
 - Preparation logic is pure module state derivation and should be reusable without triggering diagnostic side effects.
+- Precomputed `DerivedModuleArtifacts` provide reusable debug/symbol/type indexes with a versioned schema key.
+
+### `src/analyzer/IRFactCollector.cpp`
+
+Role:
+- Executes a single shared pass over selected IR to collect shared counters (`IRFacts`).
+- Supports optional subscriber notifications during the same traversal.
+
+Pattern:
+- `Collector` + `Observer dispatcher` integration point.
+
+Why:
+- Centralizing common IR facts avoids repeated lightweight scans in individual analyses.
+
+### `include/analyzer/InstructionSubscriber.hpp`
+
+Role:
+- Defines subscriber callbacks for instruction categories used by rollout passes.
+
+Pattern:
+- `Observer` (`Subscriber/Registry`).
+
+Why:
+- Allows incremental migration of analysis pre-filters to a shared event stream without changing diagnostic semantics.
 
 ### `src/analyzer/LocationResolver.cpp`
 

docs/architecture/pipeline-subscriber-classification.md

@@ -0,0 +1,62 @@
+# Pipeline Subscriber Classification
+
+This document records the Phase-1 classification of analysis steps for the
+single-pass subscriber architecture.
+
+## Classification Rules
+
+- `subscriber-compatible`: primarily instruction-pattern driven checks that can
+  consume event streams (`alloca`, `load`, `store`, `call`, `invoke`,
+  `memintrinsic`) without requiring a fixpoint over inter-procedural summaries.
+- `independent`: analyses that rely on iterative dataflow/fixpoint solving,
+  cross-function summary propagation, or specialized graph/state convergence.
+- `utility`: orchestration/preparation steps (not diagnostics analyses).
+
+The split mirrors LLVM pass-planning guidance (explicit dependencies and
+analysis invalidation boundaries) and keeps behavior deterministic during
+rollout.
+
+## Current Mapping
+
+| Pipeline step | Class | Reason |
+|---|---|---|
+| Function attrs pass | utility | Canonicalization pass, no diagnostic rule. |
+| Prepare module | utility | Context/call graph/recursion derivation. |
+| Collect IR facts | utility | Shared fact collection pass. |
+| Build results | utility | Output scaffolding. |
+| Emit summary diagnostics | utility | Aggregation-only emit phase. |
+| Compute alloca threshold | utility | Config-derived threshold materialization. |
+| Stack buffer overflows | subscriber-compatible | Event-driven from stack writes and allocation patterns. |
+| Dynamic allocas | subscriber-compatible | Direct `alloca`-shape detection. |
+| Alloca usage | independent | Depends on recursion/global stack metadata coupling. |
+| Mem intrinsic overflows | subscriber-compatible | Mem intrinsic rule matching over instruction stream. |
+| Integer overflows | independent | Range/dataflow-heavy with conservative propagation. |
+| Size-minus-k writes | independent | Wrapper-summary propagation to fixpoint. |
+| Multiple stores | subscriber-compatible | Store-pattern correlation over stack slots. |
+| Duplicate if conditions | subscriber-compatible | Intra-function condition canonicalization and matching. |
+| Uninitialized local reads | independent | Inter-procedural summary + CFG/dataflow reasoning. |
+| Global reads before writes | independent | Global state flow + summary-based tracking. |
+| Invalid base reconstructions | subscriber-compatible | Local reconstruction chains, no inter-proc fixpoint loop. |
+| Stack pointer escapes | independent | Inter-procedural fixed-point with conservative caps. |
+| Const params | subscriber-compatible | Use-def traversal around argument write/read patterns. |
+| Null pointer dereferences | independent | Branch-sensitive path/state reasoning. |
+| Out-of-bounds reads | independent | Bounds/range refinement over dataflow paths. |
+| Command injection | subscriber-compatible | Call-site taint/pattern checks from instruction events. |
+| TOCTOU | subscriber-compatible | API call ordering checks in function-level streams. |
+| Type confusion | subscriber-compatible | Type/layout pattern checks over pointer casts/GEPs. |
+| Resource lifetime | independent | Summary propagation across calls and class lifecycle states. |
+
+## Migration Notes
+
+- Fully migrated to subscriber-shared gating:
+  - `Stack buffer overflows`
+  - `Resource lifetime` (safe call-site absence short-circuit)
+- `Uninitialized local reads` remains `independent` after validation showed
+  that a simple `load`-based skip is unsound.
+- Candidate next migrations (subscriber-compatible class):
+  - `Dynamic allocas`
+  - `Mem intrinsic overflows`
+  - `Multiple stores`
+  - `Duplicate if conditions`
+  - `Command injection`
+  - `TOCTOU`

include/StackUsageAnalyzer.hpp

@@ -41,6 +41,12 @@ namespace ctrace::stack
         Full = 1
     };
 
+    enum class CompileIRFormat : std::uint8_t
+    {
+        BC = 0,
+        LL = 1
+    };
+
     // Analysis configuration (mode + stack limit).
     struct AnalysisConfig
     {
@@ -70,29 +76,31 @@ namespace ctrace::stack
         std::string resourceSummaryCacheDir = ".cache/resource-lifetime";
 
         std::uint32_t smtTimeoutMs = 50;
-        std::uint32_t jobs = 1;
+        std::uint32_t jobs = 0; // 0 = auto (hardware_concurrency)
 
         analysis::smt::SolverMode smtMode = analysis::smt::SolverMode::Single;
         AnalysisMode mode = AnalysisMode::IR;
         AnalysisProfile profile = AnalysisProfile::Full;
-
-        bool compdbFast : 1 = false;
-        bool demangle : 1 = false;
-        bool dumpFilter : 1 = false;
-        bool dumpIRIsDir : 1 = false;
-        bool includeSTL : 1 = false;
-        bool requireCompilationDatabase : 1 = false;
-        bool jobsAuto : 1 = false;
-        bool quiet : 1 = false;
-        bool smtEnabled : 1 = false;
-        bool timing : 1 = false;
-        bool uninitializedCrossTU : 1 = true;
-        bool resourceCrossTU : 1 = true;
-        bool resourceSummaryMemoryOnly : 1 = false;
-        bool warningsOnly : 1 = false;
-        bool reservedFlags0 : 1 = false;
-        bool reservedFlags1 : 1 = false;
-        std::uint32_t reservedPadding = 0;
+        CompileIRFormat compileIRFormat = CompileIRFormat::BC;
+        std::uint8_t reservedBytePadding = 0;
+
+        // Keep flags in one 32-bit storage unit:
+        // 4x u8 enums above + this u32 block keeps tail alignment compact on 64-bit builds.
+        std::uint32_t compdbFast : 1 = 0;
+        std::uint32_t demangle : 1 = 0;
+        std::uint32_t dumpFilter : 1 = 0;
+        std::uint32_t dumpIRIsDir : 1 = 0;
+        std::uint32_t includeSTL : 1 = 0;
+        std::uint32_t requireCompilationDatabase : 1 = 0;
+        std::uint32_t jobsAuto : 1 = 1;
+        std::uint32_t quiet : 1 = 0;
+        std::uint32_t smtEnabled : 1 = 0;
+        std::uint32_t timing : 1 = 0;
+        std::uint32_t uninitializedCrossTU : 1 = 1;
+        std::uint32_t resourceCrossTU : 1 = 1;
+        std::uint32_t resourceSummaryMemoryOnly : 1 = 0;
+        std::uint32_t warningsOnly : 1 = 0;
+        std::uint32_t reservedFlags : 18 = 0;
     };
 
     // Per-function result

include/analysis/CommandInjectionAnalysis.hpp

@@ -6,8 +6,10 @@
 
 namespace llvm
 {
+    class CallInst;
     class Function;
     class Instruction;
+    class InvokeInst;
     class Module;
 } // namespace llvm
 
@@ -24,4 +26,9 @@ namespace ctrace::stack::analysis
     std::vector<CommandInjectionIssue>
     analyzeCommandInjection(llvm::Module& mod,
                             const std::function<bool(const llvm::Function&)>& shouldAnalyze);
+
+    std::vector<CommandInjectionIssue>
+    analyzeCommandInjectionCached(const llvm::Function& function,
+                                  const std::vector<const llvm::CallInst*>& calls,
+                                  const std::vector<const llvm::InvokeInst*>& invokes);
 } // namespace ctrace::stack::analysis

include/analysis/DynamicAlloca.hpp

@@ -24,4 +24,8 @@ namespace ctrace::stack::analysis
     std::vector<DynamicAllocaIssue>
     analyzeDynamicAllocas(llvm::Module& mod,
                           const std::function<bool(const llvm::Function&)>& shouldAnalyze);
+
+    std::vector<DynamicAllocaIssue>
+    analyzeDynamicAllocasCached(const llvm::Function& function,
+                                const std::vector<const llvm::AllocaInst*>& allocas);
 } // namespace ctrace::stack::analysis

include/analysis/MemIntrinsicOverflow.hpp

@@ -6,12 +6,15 @@
 #include <vector>
 
 #include "StackUsageAnalyzer.hpp"
+#include "analysis/BufferWriteModel.hpp"
 
 namespace llvm
 {
+    class CallInst;
     class DataLayout;
     class Function;
     class Instruction;
+    class InvokeInst;
     class Module;
 } // namespace llvm
 
@@ -33,4 +36,11 @@ namespace ctrace::stack::analysis
     analyzeMemIntrinsicOverflows(llvm::Module& mod, const llvm::DataLayout& DL,
                                  const std::function<bool(const llvm::Function&)>& shouldAnalyze,
                                  const std::string& bufferModelPath = "");
+
+    std::vector<MemIntrinsicIssue>
+    analyzeMemIntrinsicOverflowsCached(const llvm::Function& function, const llvm::DataLayout& DL,
+                                       const std::vector<const llvm::CallInst*>& calls,
+                                       const std::vector<const llvm::InvokeInst*>& invokes,
+                                       const BufferWriteModel* externalModel,
+                                       BufferWriteRuleMatcher* ruleMatcher);
 } // namespace ctrace::stack::analysis

include/analysis/ResourceLifetimeAnalysis.hpp

@@ -4,6 +4,7 @@
 #include <functional>
 #include <string>
 #include <unordered_map>
+#include <unordered_set>
 #include <vector>
 
 namespace llvm
@@ -70,6 +71,10 @@ namespace ctrace::stack::analysis
     bool resourceSummaryIndexEquals(const ResourceSummaryIndex& lhs,
                                     const ResourceSummaryIndex& rhs);
 
+    std::unordered_set<std::string>
+    computeChangedResourceFunctionNames(const ResourceSummaryIndex& prev,
+                                        const ResourceSummaryIndex& next);
+
     std::vector<ResourceLifetimeIssue> analyzeResourceLifetime(
         llvm::Module& mod, const std::function<bool(const llvm::Function&)>& shouldAnalyze,
         const std::string& modelPath, const ResourceSummaryIndex* externalSummaries = nullptr);

include/analysis/TOCTOUAnalysis.hpp

@@ -6,8 +6,10 @@
 
 namespace llvm
 {
+    class CallInst;
     class Function;
     class Instruction;
+    class InvokeInst;
     class Module;
 } // namespace llvm
 
@@ -25,4 +27,9 @@ namespace ctrace::stack::analysis
     std::vector<TOCTOUIssue>
     analyzeTOCTOU(llvm::Module& mod,
                   const std::function<bool(const llvm::Function&)>& shouldAnalyze);
+
+    std::vector<TOCTOUIssue>
+    analyzeTOCTOUCached(const llvm::Function& function,
+                        const std::vector<const llvm::CallInst*>& calls,
+                        const std::vector<const llvm::InvokeInst*>& invokes);
 } // namespace ctrace::stack::analysis