Skip to content

Code Intelligence

Jump to bottom
Eshan Roy edited this page Jun 23, 2026 · 4 revisions

Code Intelligence

M31 Autonomous (M31A) includes a built-in code intelligence layer that parses source files, builds import dependency graphs, indexes symbols, and scores file relevance to tasks. This powers the workflow engine's context awareness during planning and execution.

Source: internal/codeintel/

Architecture

flowchart TD
    Build[Indexer.Build] --> BG[BuildGraph<br/>Import Graph]
    Build --> BI[BuildIndex<br/>Symbol Index]
    Build --> RS[NewRelevanceScorer<br/>File Ranking]

    BG --> |"BFS traversal<br/>O(1) dedup via map"| G[ImportGraph]
    BI --> |"O(1) dedup<br/>via seen map"| S[SymbolIndex]
    RS --> |"Multi-factor<br/>scoring"| R[RelevanceScorer]

    style Build fill:#e1f5fe
    style G fill:#e8f5e9
    style S fill:#e8f5e9
    style R fill:#fff3e0
Loading

The indexer is lazy-built once per session with a 30-second timeout. Results are cached for the session lifetime. Graph traversal uses index-based BFS (O(n)) and reverse-edge dedup uses hash maps (O(1)).

Indexer

Source: internal/codeintel/codeintel.go

The Indexer is the unified API for all code intelligence queries:

Method Description
Build(ctx) Parse all source files, build graph + index + scorer
Upstream(path, depth) Files that path depends on
Downstream(path, depth) Files that depend on path
Neighbors(path) Direct imports and importers
Define(symbol) Where a symbol is defined
FileSymbols(path) All symbols in a file
SymbolsMatching(query) Fuzzy symbol search
RelevantFiles(targets, desc, topN) Rank files by relevance to a task
FormatContext(targets, desc, topN, maxBytes) LLM-ready context summary
ProjectSummary(maxBytes) High-level project overview

FormatContext Output

Generates a structured text block injected into the LLM system prompt:

## Recommended Files (by relevance)
- **internal/provider/cache.go** (score: 15.0) — imported by engine.go; defines Cache
- **internal/provider/interface.go** (score: 12.0) — same package as cache.go

### Dependencies for internal/workflow/engine.go
  Imports: internal/types/types.go, internal/config/types.go, ...
  Imported by: internal/tui/app.go

### Type Definitions Referenced in Task
- **ModelInfo** (struct) defined in internal/types/types.go

Language Parsers

Source: internal/codeintel/parser.go

Four language parsers extract imports, exports, functions, and types:

Parser Language Method Extensions
GoParser Go go/ast (full AST) .go
TypeScriptParser TypeScript/JS Regex patterns .ts, .tsx, .js, .jsx, .mjs, .cjs
PythonParser Python Regex patterns .py
RustParser Rust Regex patterns .rs

Parser Interface

type Parser interface {
    Language() string
    CanParse(path string) bool
    Parse(path string, content []byte) (*FileInfo, error)
}

FileInfo

type FileInfo struct {
    Path     string
    Language string
    Imports  []ImportInfo    // import declarations
    Exports  []SymbolInfo    // exported symbols
    Funcs    []FuncSignature // function/method signatures
    Types    []TypeInfo      // struct/interface/class definitions
}

Go Parser

Uses Go's go/ast package for full-fidelity parsing:

  • Extracts all imports, functions, methods, types, constants, variables
  • Detects struct fields and interface methods
  • Distinguishes exported vs unexported symbols via ast.IsExported()

TypeScript Parser

Regex-based extraction of:

  • import ... from 'path' statements (named, namespace, default, side-effect)
  • export declarations (const, function, class, enum, type, interface)
  • Function declarations and arrow functions with parameters/return types
  • Class hierarchies (extends, implements)
  • Interface field extraction

Python Parser

Line-by-line regex extraction of:

  • import and from X import Y statements
  • Class definitions with base classes
  • Function/method definitions with parameters and return types
  • Decorator detection (e.g., @staticmethod)
  • Top-level variable assignments
  • Respects indentation (skips nested definitions)

Rust Parser

Regex-based extraction of:

  • use statements (crate-internal and external)
  • pub fn, pub struct, pub enum, pub trait, pub type, pub const
  • Function signatures with generic parameters and return types
  • Struct fields from inline definitions

Import Graph

Source: internal/codeintel/graph.go

A directed dependency graph between source files:

type ImportGraph struct {
    nodes map[string]*Node // relative path → node
}

type Node struct {
    Path       string
    Imports    []string // files this file imports
    ImportedBy []string // files that import this file (reverse edges)
    Language   string
}

Graph Traversal

Method Description
Upstream(path, depth) BFS of dependencies (depth=0 means unlimited)
Downstream(path, depth) BFS of reverse dependencies
Neighbors(path) Direct imports + direct importers

Import Resolution

BuildGraph() walks the project directory and resolves imports to local files:

Language Resolution Strategy
Go Module path from go.mod, relative imports, standard library skip
TypeScript Relative path + extension probing (.ts, .tsx, .js, etc.) + index files
Python Dot-separated module paths, __init__.py support, relative imports
Rust crate:: prefix, src/ root, .rs files and mod.rs modules

Skipped directories: node_modules, vendor, .git, .next, dist, build, target, .venv, __pycache__. Maximum depth: 5 levels.

Symbol Index

Source: internal/codeintel/index.go

Maps symbol names to their definition locations:

type SymbolIndex struct {
    byName map[string][]SymbolLocation // symbol → definition locations
    byFile map[string][]SymbolInfo     // file → exported symbols
}

type SymbolLocation struct {
    File string
    Kind string // "func", "type", "interface", "class", "struct", "const", "var"
}
Method Description
Define(name) All locations where a symbol is defined
DefineInFile(name, file) Definition in a specific file
FindTypes(name) Type/struct/interface/class definitions
FindFuncs(name) Function definitions
SymbolsMatching(query) Case-insensitive substring match
AllSymbols() All unique symbol names

Relevance Scorer

Source: internal/codeintel/relevance.go

Ranks files by relevance to a task using a multi-factor scoring system:

Factor Score Description
Direct mention +10.0 File is explicitly listed in the task
Direct dependency +5.0 Imported by or imports a target file
Symbol match +7.0 Defines a symbol referenced in the task description
Same package +3.0 In the same directory as a target file
Transitive dependency +2.0/depth Decaying score for deeper transitive imports

Identifier Extraction

The scorer decomposes task descriptions into identifiers:

  • Splits on whitespace and strips punctuation
  • Decomposes camelCase: getUserByID → [get, User, By, ID]
  • Decomposes snake_case: get_user → [get, user]
  • Filters stop words (the, and, for, add, create, etc.)
  • Case-insensitive matching against symbol index

ScoredFile Output

type ScoredFile struct {
    Path    string
    Score   float64
    Reasons []string // human-readable explanations for the score
}

EFIE Backend

Source: internal/codeintel/ (toggled via USE_EFIE environment variable)

EFIE (Eshanized File Intelligence Engine) is an alternative code intelligence backend that replaces flat graph scanning with community-structured, importance-weighted, adaptive expansion.

Key Features

Feature Description
PageRank centrality Precomputes architectural importance of each file
Betweenness centrality Identifies bridge files connecting different parts of the codebase
Community detection Deterministic Louvain algorithm discovers natural file clusters
Bloom filters Fast membership tests for symbol lookup
Polyglot parsers Supports Go, TypeScript, Python, Rust, and more
Parallel graph building Worker pool for concurrent file parsing

Algorithm Overview

  1. Graph construction — Build a weighted dependency graph from source files
  2. Centrality precomputation — Compute PageRank and betweenness centrality scores
  3. Community detection — Partition files into natural clusters using Louvain
  4. Bloom filter indexing — Create fast symbol lookup structures per community
  5. Adaptive expansion — Follow the most important paths first, stop when budget is exhausted

Enabling EFIE

# Set environment variable to use EFIE backend
export USE_EFIE=1

# Run M31A as usual
m31a

When USE_EFIE is set, the code intelligence indexer uses the EFIE algorithm instead of the default flat graph approach. The EFIE backend produces the same API surface (RelevantFiles, FormatContext, etc.) but with 5-20x faster queries on large codebases.

Trade-offs

Aspect Default Backend EFIE Backend
Build time Faster (sequential) Slightly slower (parallel, but more computation)
Query speed O(N × T) 5-20x faster
Memory usage Lower +16% overhead
Best for Small projects (<1K files) Large projects (1K-10K files)

Research

EFIE is documented in detail in the EFIE/ directory:

  • EFIE_RESEARCH.md — Full technical research paper
  • EFIE_ALGORITHM_V2.md — Algorithm specification v2
  • EFIE_PATENT_WHITE_PAPER.md — Patent white paper
  • BENCHMARK_REPORT.md — Performance benchmarks

Clone this wiki locally