Codebase dependency graph, honest capacity model, tests, and CI - #14
Merged
Conversation
…-log overlay Adds a force-directed module/import graph (Python AST + regex resolvers for JS/TS/Go/Java/Rust/Ruby), wired to the same detected components (LLM/storage/ queue/external-source) already surfaced elsewhere in the report. Large repos aggregate to directory-level nodes automatically. Practical throughput is now a genuine min() across whichever rate-limit, semaphore, and LLM-timeout constraints are actually detected, instead of a single hardcoded guess, and a --access-log flag overlays real observed request counts onto the graph's HTTP-entry edges plus generates a matching k6 load-test script so the modeled ceiling can be checked against reality.
Real, non-mocked coverage against synthetic project fixtures: every detector, the min()-ranking capacity model, access-log parsing and k6 script generation, the per-language graph import resolvers, directory aggregation, and one full-pipeline smoke test — 80 cases total, including a regression test for the psycopg/postgres hint-prefilter gap this found earlier. The existing workflows only trigger on GitHub releases, so ordinary commits and PRs had no automated check at all. This adds a matrix (3.10/3.11/3.12) workflow on push/pull_request that compiles both the real scanner and its scripts/ shim, then runs the suite.
…rage README/SKILL/INSTALL never mentioned the dependency graph, --access-log, --graph-detail, or the generated k6 script — all landed without docs. Also fixes the repo-layout diagrams, which mislabeled scripts/analyze.py as the core scanner; it's a thin shim over workflow_generator_mcp/analyze.py.
Near-coincident initial node positions (deterministic per hashed file
id) could push the O(n^2) repulsion term toward its floor and spike a
node's per-tick velocity into the thousands, throwing it far outside
the canvas after the 220-tick settle. The view also assumed a fixed
{x:0,y:0,scale:1} viewport instead of fitting whatever bounding box
the layout actually settled into. Together these could render an
entirely blank graph on repos large enough to trigger the collision —
confirmed via canvas pixel sampling against the fastapi template.
Three compounding bugs left the dependency graph badly under-connected on any repo with a nested package root or path aliases (the exact fastapi template used in the live demo went from 340 real edges down to 6): - collect() never resolved a relative root, so the default `analyze.py .` invocation left every collected file path relative while _resolve_js_spec built absolute candidate paths — the set-membership check silently never matched, dropping every JS/TS edge regardless of import style. - Absolute Python imports (`from app.core.config import x`) are written relative to whatever directory is actually on sys.path (e.g. backend/), not the scanned repo root, so the root-relative module name never matched. Now falls back to unambiguous dropped-leading-segment suffixes. - tsconfig/jsconfig path aliases (`@/foo` -> `./src/foo`) were treated as unresolvable bare specifiers, same as an external npm package. Also fixes two UI issues surfaced while investigating the sparse graph: wheel zoom-out had a stricter floor (0.25) than the initial auto-fit scale could reach (down to 0.06), so zooming in then back out could get stuck more zoomed-in than the starting view with no way back — floor now matches, plus an explicit reset-view button. And real zero-degree files (empty __init__.py, migrations, config-only modules) were cluttering the default view and skewing the auto-fit; added a "hide isolated files" toggle (default on when any exist) that also re-fits the view to the visible set.
Terminal segment (pip install -> CLI run) followed by a scroll through the generated WORKFLOW.html, including the now-fixed codebase dependency graph (340 real connections, not the pre-fix 6) and a node click isolating a file's call paths. Numbers shown match a fresh, verified run of the current code. README now leads with the gif and keeps the static preview.png as a fallback for anyone who'd rather not autoplay.
# Conflicts: # docs/demo.gif
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
min()across whichever rate-limit / semaphore / LLM-timeout constraints are actually detected, instead of one hardcoded guess.--access-logto overlay real observed request counts onto the graph's HTTP-entry edges, plus a generated k6 load-test script so the modeled ceiling can be checked against reality.push/pull_request, Python 3.10/3.11/3.12) that compiles and runs the suite — the existing two workflows only trigger on GitHub releases, so ordinary commits had no automated check at all.scripts/analyze.pyas the core scanner (it's a shim overworkflow_generator_mcp/analyze.py).Test plan
pytest tests/ -v— 80 passedpython -m py_compile workflow_generator_mcp/analyze.py scripts/analyze.py