release: 0.5.2 — code-understanding - #65
Merged
Merged
Conversation
Merged after targeted bundle tests passed in a protobuf-compatible environment.
Introduce a single optional project config file (lachesis.yml) and make the default build ingest product source only -- tests, examples, docs, fixtures, benchmarks and vendored trees no longer land in the graph. A tree is understood by its product code; the scaffolding around it inflates the node count and, for the comprehension export, drowns the architecture ranking and roots request flows in test modules. Mechanism. lachesis/config.py discovers the file by walking up from the analysed tree (or --config), lazily imports PyYAML only when a file is actually present, and resolves a typed Config with build/export/atropos/runtime sections. The non-product decision is a PathFilter over the repo-relative, forward-slashed display path: matching is on path *segments* and basenames, never substrings, so product modules whose name merely contains a keyword -- testing.py, templating.py, documentation.py -- are kept while tests/, examples/, docs/, conftest.py, *_test.py and vendored trees are dropped. include wins over exclude wins over the built-in default; an explicit `exclude: []` restores full coverage, and a `build.include` allow-list keeps a specific file without disabling the default. Wiring. source_inventory grows an optional path_filter applied in the walk; it is threaded through every build variant (run_project, _streaming, _streaming_parallel, _incremental, _parallel) and package_jobs, and -- critically -- through source_content_hash, so a filtered build and its cache-validity key describe the very same file set and a filtered build cannot hit a cache written unfiltered. The low-level default is unchanged (complete coverage for direct library callers); the CLI resolves the filter from config and passes it down, so `lachesis build` and `lachesis analyze` exclude non-product by default while --all-sources (or `exclude: []`) compiles the whole tree. The runtime: block mirrors every LACHESIS_* variable and is applied with os.environ.setdefault, preserving the documented precedence flag > env > file > default. Unknown config keys and malformed fields warn rather than fail, so a file written for a newer reader still loads. Verified end-to-end on a synthetic tree through the real CLI: the default build keeps only src product modules (testing.py included), --all-sources keeps all six files, `exclude: []` includes everything, an explicit exclude list replaces the default, and an include allow-list rescues a single test file. New unit tests in test_config.py (classifier, globs, precedence, runtime env, discovery) and test_pipeline_inventory.py (filter drops non-product, content hash tracks the filtered set); full cli/ and nav/test_bundle.py suites green (76 passed). PyYAML is added as an optional [config] extra, imported lazily; the stdlib-only core is unaffected for anyone who never writes a config file.
The graph-first exporter had four content defects the frontend surfaced as
if they were the codebase's behavior:
1. Guided request paths were a greedy CALLS-successor walk out of each
exported symbol, so they rooted at leaf utilities (render_template,
run_command) and never showed the request lifecycle. The architecture
core came out empty because the old Hubs block called re.search without
importing re (a silent NameError swallowed by its try/except) and shadowed
the graph index with the concepts enumerate counter.
3. No core spine was emitted for the same two reasons.
6. Most nodes carried no scope, so the frontend could not group them.
7/8. Value paths featured bare def-use artifacts (a traceback local `tb`,
a file handle `f`, `config_file`, `tb.tb_frame`) as "behaviors", and the
same def-use was emitted twice from different findings.
Mechanism. Requests and core are now built together by _lifecycle_projection,
which seeds a bounded execution story from the real top-of-stack drivers
(entry handlers first, then in-degree-0 product callables ranked by 2-hop
reach), linearises each story's success spine (deranking exception/teardown
branches and cutting cycles), and keeps the deepest few. The story spine nodes
also populate the architecture core. On Flask this roots the primary path at
__call__ -> wsgi_app and puts full_dispatch_request / request_context /
push / pop on the core spine, where before it showed render_template.
_enrich_graph_nodes now sets `scope` on every node from gl.owner_function:
a node reports the enclosing callable's qualified name, a callable reports its
module; a synthetic context/heap node with no declaration honestly carries
none (115/120 nodes on Flask, the 5 without being file-less overlay nodes).
The featured value paths gain two hygiene rules: a path that visits fewer than
two distinct nodes has not moved and is dropped (removing all 12 def-use
artifacts on Flask while keeping every real flow, including minimal two-step
call-argument flows), and paths with identical endpoints and ordered node ids
are collapsed. security.findings stays exhaustive -- only paths.values is
filtered.
Correctness. The 2.0 contract is unchanged: _comprehension_projection returns
the same keys, and _graph_first_bundle emits the same shape. The two-distinct
-node floor keeps every witness the suite drives (all are two-step with
distinct ids), so nav/test_bundle.py, test_config.py and
test_pipeline_inventory.py pass (71 tests). Verified by re-exporting the Flask
graph: 0 non-product entrypoints, 0 source-less nodes, 0 single-node value
artifacts, 0 duplicate value names, requests rooted in app code.
Build-time exclusion keeps scaffolding out of the graph, but the exporter must not depend on it. Run against a graph built without exclusion, the comprehension projection refeatured exactly the files the classifier drops: on full Flask 39 of 40 entrypoints, 22 of 32 core nodes and all 8 request roots were test_* or example functions. The cause is structural -- an uncalled test function is an in-degree-0 callable, so _lifecycle_roots ranked it as a top-of-stack driver, and the entry-anchor pass surfaced test handlers directly. Gate the three featured surfaces with the same classifier the build filter uses (lachesis.config.is_nonproduct), so the two always agree: skip non-product handlers when building entrypoints, skip non-product entry handlers and in-degree-0 drivers when choosing lifecycle roots, and keep non-product nodes off the architecture core. The import is guarded and the gate fails open (keeps the node) if the classifier is ever unavailable; is_nonproduct is pure-stdlib so this does not add a runtime dependency. Correctness. security.findings and the value paths are untouched -- only the comprehension surfaces are gated. Re-exporting full (unexcluded) Flask now shows 0 non-product entrypoints/core/request-roots (roots: __call__, make_response, shell_command, ...), and the already-excluded graph is unchanged (identical counts). nav/test_bundle.py, test_config.py and test_pipeline_inventory.py pass (71 tests); the suite's hand-built findings carry no file, so the gate is inert there by construction.
Resolves the PR conflict with dev. The only diverging commit on dev was "Export architecture concepts in Explorer bundles" (d34222b), which adds the _project_concepts renderer, the concepts block in _comprehension_projection, and its two tests. This branch already carries that exact feature (developed in parallel as 83f27de) and extends it with the request-lifecycle rooting, the bounded core spine, per-node scope, the value-path hygiene/dedupe, and the non-product export gate. Both conflicting files therefore resolve to this branch's version, which is a strict superset: every dev hunk was the pre-core/pre-tour form of code this branch already supersedes, and dev added nothing outside those hunks. The merged tree is byte-identical to this branch's HEAD, so the merge changes no file -- it only records that dev is integrated. nav/test_bundle.py, test_config.py and test_pipeline_inventory.py pass (71 tests).
… code A multi-codebase review (json5, django, suricata) surfaced three ways the comprehension projection featured the wrong code. All three share one root cause: the non-product path gate the export already applied to entrypoints and lifecycle roots was neither complete nor applied everywhere it needed to be. 1. Multi-language repos featured the wrong language. On django (Python) every one of the 40 entrypoints and all 8 request flows were bundled JavaScript admin widgets under contrib/admin/static: the planner treats JS event handlers as entry handlers, and an uncalled JS handler is in-degree-0 so it also ranks as a lifecycle driver. A newcomer saw a JS widget toolkit instead of the WSGI -> middleware -> resolver -> view path. Fix: derive the repo's dominant product-source language family (_primary_language_family, a strict majority over non-scaffolding source files, else None = polyglot = no gate) and gate entrypoint selection and both lifecycle-root sources to it, so the projection stays in the language the repo actually is. 2. Vendored deps, generated output and build configs leaked. node_modules the TypeScript compiler lib, rollup.config.js and *.d.ts surfaced as concepts and nodes even in Python bundles. The classifier (config.is_nonproduct) is the default build- and export-time filter, so it is the right place: it now also drops dist/ and scripts/ segments and *.min.*, *.d.ts and *.config.js basenames. It stays language-agnostic (path-only), so one rule covers Python/TS/JS/C, and it applies at build time too -- a graph over an application has no business modelling a dependency it cannot patch, for comprehension or for security triage. Recoverable via build.include. 3. modules/concepts leaked tests/docs/examples. _partition_modules and _project_concepts never consulted the gate, so suricata surfaced modules tests.fuzz.*, doc.userguide.convert and concepts scripts, doc . userguide. Both now apply _is_nonproduct_path, and the architecture-concept builder filters non-product files *before* deriving a community's label -- so a community mixing json5's lib/*.js with the vendored compiler is labelled "lib", not "arachne . node_modules . typescript . lib". Verified by re-exporting the unexcluded json5 graph: entrypoints 4/4 product JS, modules the 6 real product modules, the single concept now "lib". config and bundle unit suites green (73 tests); added classifier cases for the new vendored/generated/build-config vocabulary and its over-match guards (reader.ts, lib/config.js, src/distributed/, src/scripting/ all kept). A fourth review item -- documentation is 0% on every bundle -- is a producer gap (no frontend emits a docstring/comment property; the reader is correct against data that is not there) and is handled separately.
…ing contract
Two exporter defects surfaced across the Flask/Click/ItsDangerous 2.0 bundles.
Synthetic nodes (heap locations, summary objects) carried no source and were
exported as {file: null, line: null}. The 2.0 contract is a concrete absence --
file "" and line 0 -- so a reader can uniformly test line > 0 for openability
instead of handling a third "unknown type" state. _normalize_node_location now
coerces every node's file to a string (""), line to a non-negative int (0), and
makes end_line mandatory and never less than line (a single-line span when no
wider extent is known, 0 for synthetics). This is behaviour-preserving for path
and entrypoint selection: _has_source already rejects an empty file and a
non-positive line, so the nodes that were non-source-backed as null stay
non-source-backed as ""/0 -- only the field's type is pinned. Verified on the
json5 export: 20 synthetic nodes now report ""/0/0, zero nulls, zero inverted
spans, and the featured entrypoints and request hops are unchanged.
The validator now enforces those types on every node (file is str, line and
end_line are ints, line >= 0, end_line >= line) rather than allowing the null
leak through. It also enforces the projection contract a code-understanding
consumer relies on: at least one production entrypoint (the ItsDangerous export
shipped entrypoints: [], which is a hollow projection) and at least one
source-backed path of three or more hops. Request hops are already proven
source-backed, so a >=3-hop request satisfies the path requirement by
construction. django (40 entrypoints, 6-hop paths) and json5 (4 entrypoints,
5-hop paths) both satisfy the contract; the test fixtures that built minimal
bundles were updated to supply a contract-valid base. All 73 nav/config/pipeline
tests pass.
…oline The code-understanding projection mis-drew a framework's request path. On the real Flask library graph the guided lifecycle came out `__call__ -> wsgi_app -> session deserialize / JSON dumps`, wsgi_app was not an entrypoint at all, and full_dispatch_request / dispatch_request -- the two hops a reader most needs -- were nowhere on the spine. Three causes, three fixes, all in the projection's node *selection* (ids, content-hashes, and edge keys are untouched; only which nodes the 2.0 bundle features changes): 1. `_story_spine` picked the deepest-subtree callee at each hop, which for a library rewards in-repo duck-typed detours: the true dispatch chain dead-ends at the external view, while a session-deserialize or json-dump that only *might* run keeps fanning out inside the library and looks bigger. But those detours are all `via=indirect:may_invoke` edges, whereas the real spine is `via=direct`. The pick key now prefers a direct CALLS edge first, then deranks error/teardown names, then breaks ties on subtree size -- so the walk stays on wsgi_app -> full_dispatch_request -> dispatch_request. 2. An in-degree-0 driver is often a thin trampoline (`Flask.__call__` is one line: `return self.wsgi_app(...)`). New `_descend_trampoline` follows a chain of single product-callee direct forwarders, bounded and cycle-guarded, so `_lifecycle_roots` yields wsgi_app rather than __call__ -- both shortening the path and, crucially, making the dispatcher the request's entry node. 3. `EntryPoints.by_handler()` only recognises module-level public helpers, so a framework's actual request driver (a WSGI wsgi_app, an event loop) was never an anchored entrypoint. `_comprehension_projection` now promotes each lifecycle root to a `request-lifecycle` entrypoint, leading the list. This also gives a library with no anchored handler (itsdangerous) a real source-backed entrypoint, which the code-understanding contract now requires. This is an intentional behavior change to the projection, not a refactor. Verified by re-exporting three graphs: flasklib now leads with `[request-lifecycle] wsgi_app` and a spine wsgi_app -> full_dispatch_request -> dispatch_request -> ...; django (40 Python entrypoints) and json5 (7 promoted lib/parse.js lifecycles) export with the validator passing and no null/typed-field regressions. All 73 nav/config/ pipeline tests pass.
…her leads The code-understanding projection roots each guided request path at a top-of-stack driver and follows its execution story. Two candidate sources feed the root set: the planner's entry handlers and every in-degree-0 product callable. The old order kept all handlers ahead of the drivers and then truncated the combined list to a fixed cap. On a framework that emits many thin handlers this starved the real dispatcher out of the pass entirely: Click surfaces ~30 decorator handlers (`version_option`, `argument`, ...), each spinning a valid but peripheral story, which filled the cap before `Command.main` -- the actual in-degree-0 dispatcher with the widest control cone -- was ever considered. The bundle then led with `secho`/`pause` string helpers instead of the command lifecycle. Rank every candidate (handlers and de-trampolined drivers alike) by two-hop reach before applying the cap, so the widest-cone lifecycle always survives into the bounded story pass. Ordering here governs only which candidates the pass sees; `_lifecycle_projection` still re-ranks the survivors, so this does not change how kept requests are ordered relative to each other -- it only stops a high-reach driver from being truncated away. In-degree-0 trampolines are still descended to their single product callee first (`Flask.__call__` -> `wsgi_app`, `BaseCommand.__call__` -> `main`) so the story is named at the dispatcher, not the forwarder. Behavior on the previously-working cases is unchanged: Flask still leads with `wsgi_app -> full_dispatch_request -> dispatch_request -> ...`, json5 still promotes its `lib/parse.js` lifecycles, django's 40 entrypoints are intact. Click now leads with `main -> make_context -> parse_args -> iter_params_for_processing`, and `main` is promoted to the top request-lifecycle entrypoint. Cost is unchanged: the story pass still runs at most `cap` execution stories; the only added work is one two-hop reach count per handler candidate (microseconds), and the graph builder is untouched. Verified: 47 tests in nav/test_bundle.py pass; exports re-checked on clicklib/flasklib/json5.
…ops and areas The code-understanding projection was structurally usable but read coarsely: the Flask request spine stopped at a routing corner, hop captions were bare symbols, and every module of a flat single-package framework collapsed into one concept. This reworks the three, all on generic identifier/graph morphology -- an English verb/noun vocabulary and the real call graph -- never a per-framework symbol table. Spine reaches the response terminus. An execution story is a *tree*: each function is attached under its first-discovered caller, so a genuine callee can hang off an earlier parent than the one whose body makes the call. Flask's finalize_request (which builds the response) lands under handle_exception in the tree, not under full_dispatch_request whose call actually reaches it, so a walk over story-children alone could never route past dispatch_request to make_response. _story_spine now recovers the missing edges from the graph (_candidate_children): every genuine callee of the current node that the story itself visited becomes a candidate, carrying its story fn record and a via classified from the edge kind. It invents no nodes (only functions already in the story are admitted) and no edges the graph does not hold. The five-key hop ranker then does the rest generically -- a candidate whose subtree constructs the returned result (a construction verb over a result noun, and not a fallback/error name: make_response yes, the special-case make_default_options_ response no) is preferred -- so Flask now ends wsgi_app -> full_dispatch_request -> finalize_request -> make_response. Readable hop captions. Each hop keeps its exact symbol as caption (still greppable) and gains reads_as: a human phrase from the symbol's morphology (_readable_caption) -- the leading action verb past any modifier, rendered in the third person over the remaining object tokens (full_dispatch_request -> "dispatches the request", make_response -> "builds the response", unsign -> "verifies the signature on"). It degrades to the bare humanized noun phrase on a name with no known verb, so it only ever adds a hint, never hides the identifier. Module-centric concepts. Call-community clustering is too coarse for a flat package: Flask's whole tree came back as one 20-file community labelled off its first path (__init__.py). Concepts are now the module areas a newcomer would name -- one per product file, ranked by definition count alone, capped at _MAX_CONCEPTS, each labelled by its module stem (widened to "parent - stem" only to break a genuine collision like app vs sansio/app). Ranking on the definition count keeps this a single cheap node scan (django: 0.14s over 9,719 callable nodes). An earlier draft added an outgoing-call-degree tiebreaker that ran one graph edge-lookup per function; that scan measured 8.15s on django -- ~8s of avoidable per-function graph queries -- and it never changed an order in practice (a module's definition count already fully separates the areas), so it was dropped. Flask surfaces app, blueprints, cli, ctx, sessions, templating, config as distinct areas; json5 surfaces parse, stringify, cli; django surfaces models, options, query, widgets. _project_concepts still restricts every concept to product nodes in the final pool, so the areas stay honest to what shipped. This deliberately changes projection *output* (it is a comprehension-quality change, not behaviour-preserving); the graph builder and schema are untouched and the strict source-backed gating on every hop is unchanged. Removed the last framework-specific tokens from the error vocabulary (finalize_request/handle_http) that had both violated the generic-morphology rule and mis-deranked the response path. Verified: 47 tests in nav/test_bundle.py pass; exports re-checked on flasklib (spine now ends at make_response; 11 module concepts), clicklib (main -> make_context -> parse_args, unchanged), itsdangerous and json5 (spines unchanged), django (8 request spines sound, 4 module concepts, no regression).
…oduct-exclusion Feat/lachesis config nonproduct exclusion
Roll the version surface 0.5.1 -> 0.5.2 for the code-understanding release. The tag-triggered release gates require every pin to agree: pyproject's version, both server.json pins (the server and the lachesis-cpg package), the native lifetime_kernel crate, and a matching "## [0.5.2]" CHANGELOG heading. Bring all of them to 0.5.2 in one commit so a single tag push does not trip the guard the way the 0.5.1 cut did when the first bump moved pyproject alone. The release carries the lachesis.yml project configuration with default exclusion of vendored/generated/test/build-config code from the build and the export, and the comprehension-quality rework of the 2.0 Explorer bundle: request lifecycles rooted at the real dispatcher and routed forward to their result terminus, human-readable reads_as hop captions, module-centric concepts, primary-language gating, and normalized node locations under an enforced code-understanding contract. Metadata and projection scoping only -- no graph-schema, query-engine or candidate-surface change, and the schema 1.0 security bundle is unchanged. Verified the release gate locally: pyproject, both server.json pins and the native crate all read 0.5.2, and CHANGELOG.md carries the "## [0.5.2]" heading.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 0.5.2 — code-understanding
Releases
devintomainfor the 0.5.2 cut. Version surface is bumped and inagreement (pyproject, both
server.jsonpins, the nativelifetime_kernelcrate,and a
## [0.5.2]CHANGELOG heading), so taggingv0.5.2after merge passes therelease and MCP-registry guards.
What's in it
lachesis.ymlproject configuration — a repo declares its own build/exportknobs; vendored, generated, build-config and test/example scaffolding are excluded
from the build and the export by default, so the graph and the comprehension surface
describe the product code.
the real dispatcher and route forward to the result terminus; each hop gains a
human-readable
reads_ascaption alongside its exact symbol; concepts becomemodule-centric areas; entrypoints/requests/concepts are gated to the repo's primary
language.
""/0) and acode-understanding projection with no openable entrypoints is rejected rather than
emitted empty.
Metadata and projection scoping only: no graph-schema, query-engine or candidate-surface
change, and the schema 1.0 security bundle is unchanged.
Verification
nav/test_bundle.py: 47 passing; config and pipeline test suites added on the branch.module concepts sound, no regression.
heading is present.
After merge
Tag
v0.5.2onmainto trigger the wheel/sdist build, PyPI publish, and the MCPregistry
server.jsonpublish.