Codebase dependency graph, honest capacity model, tests, and CI

.github/workflows/tests.yml

@@ -0,0 +1,24 @@
+name: Tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install pytest
+        run: python -m pip install --upgrade pip pytest
+      - name: Compile check
+        run: python -m py_compile workflow_generator_mcp/analyze.py scripts/analyze.py
+      - name: Run tests
+        run: pytest tests/ -v

INSTALL.md

@@ -6,17 +6,19 @@ The skill lives at: `~/.claude/skills/workflow-generator/`
 
 ```
 ~/.claude/skills/workflow-generator/
-├── SKILL.md                  ← Claude Code skill definition
-├── INSTALL.md                ← this file
+├── SKILL.md                        ← Claude Code skill definition
+├── INSTALL.md                      ← this file
+├── workflow_generator_mcp/
+│   └── analyze.py                  ← core scanner + HTML renderer (no external deps)
 ├── scripts/
-│   └── analyze.py            ← core scanner + HTML renderer (no external deps)
+│   └── analyze.py                  ← thin compatibility shim -> workflow_generator_mcp/analyze.py
 ├── mcp/
-│   ├── server.py             ← MCP stdio server
-│   └── requirements.txt      ← pip install mcp
+│   ├── server.py                   ← MCP stdio server
+│   └── requirements.txt            ← pip install mcp
 └── copilot/
-    ├── index.js              ← GitHub Copilot Extension (Express)
+    ├── index.js                    ← GitHub Copilot Extension (Express)
     ├── package.json
-    └── openai_function.json  ← OpenAI / Antigravity function schema
+    └── openai_function.json        ← OpenAI / Antigravity function schema
 ```
 
 ---
@@ -172,6 +174,13 @@ python3 ~/.claude/skills/workflow-generator/scripts/analyze.py . ~/WORKFLOW.html
 # open ~/WORKFLOW.html in browser
 ```
 
+Optional flags (append after the two positional args):
+
+```bash
+--access-log /path/to/access.log   # overlay real request counts onto the dependency graph
+--graph-detail auto|files|dirs     # force file-level or directory-level graph nodes (default: auto)
+```
+
 Expected output:
 ```
 Written: /home/you/WORKFLOW.html
@@ -194,6 +203,7 @@ External sources: Jira, Azure DevOps, Slack, Users / API Clients
 | **Gateway** | `nginx.conf`, `Caddyfile`, `traefik.yml`, nginx image in docker-compose |
 | **Rate limits** | `limit_req_zone` (nginx), `@limiter.limit` (slowapi), `express-rate-limit` |
 | **API framework** | FastAPI, Flask, Django, Express, Gin (Go), Spring Boot |
+| **Languages** | Python, JS/TS, Go, Java, Rust, Ruby (dependency graph resolves real imports in all six) |
 | **Frontend** | Streamlit, Gradio, React, Next.js, Vue, Svelte |
 | **LLM** | OpenAI (ChatOpenAI), Anthropic (Claude), Cohere, AWS Bedrock |
 | **Embedding** | `text-embedding-3`, `CohereEmbeddings`, `HuggingFaceEmbeddings` |
@@ -212,3 +222,6 @@ Every generated `WORKFLOW.html` includes:
 3. **Flow cards** — write path, read path, queue jobs (inferred from what's detected)
 4. **Concurrency table** — every layer: model / ceiling / limiting factor
 5. **Bottleneck analysis** — ranked bars CRITICAL → LOW with mitigation notes
+6. **Codebase dependency graph** — force-directed module/import graph across all detected
+   languages; click a node to isolate its neighbors. Import-direction edges by default; pass
+   `--access-log` to weight HTTP-entry edges with real observed request counts instead

README.md

@@ -4,7 +4,7 @@
 
 Scan any project and generate **WORKFLOW.html** — a dark-mode visual system diagram showing every component, how they talk to each other, and where your throughput ceiling actually is.
 
-Works with Python, Node.js, Go, and mixed projects. No external dependencies for the core scanner.
+Works with Python, Node.js, Go, Java, Rust, Ruby, and mixed projects. No external dependencies for the core scanner.
 Vendored and generated directories (`node_modules`, `venv`, `site-packages`, `dist`, …) are never scanned,
 and capacity figures are clearly labeled as static-analysis estimates.
 
@@ -26,6 +26,24 @@ Every generated page contains:
 | **Data flow cards** | Write path, read/query path, background jobs — inferred from what's detected |
 | **Concurrency table** | Every layer: model · ceiling · limiting factor |
 | **Bottleneck analysis** | Ranked CRITICAL → LOW with mitigation notes |
+| **Codebase dependency graph** | Force-directed module/import graph — click a node to isolate its neighbors, hover for file details. Import-direction edges are clearly distinguished from real observed traffic (see below) |
+
+### Codebase dependency graph
+
+Every source file (Python, JS/TS, Go, Java, Rust, Ruby) becomes a node; every real import becomes
+an edge — resolved with a language-appropriate parser (Python's `ast` module, regex for JS/TS/Go/
+Java/Rust/Ruby), not guessed. Files that match an already-detected component (an LLM call, a
+database client, a queue) get an edge to that component too, so you can see exactly which files
+talk to Redis, OpenAI, etc. Large repos (350+ files) are automatically aggregated into
+directory-level nodes so the graph stays readable; override with `--graph-detail files` or
+`--graph-detail dirs`.
+
+By default the graph only shows what the *code* says (import direction, static "this file calls
+Redis"), which is honest but not the same as real traffic. Pass `--access-log /path/to/access.log`
+(any combined/common log format) to overlay real observed request counts onto the HTTP-entry
+edges — and the generated report includes a ready-to-run [k6](https://k6.io) load-test script
+covering up to 5 detected routes, so the "Practical throughput" number can be checked against a
+real measurement instead of only a static-analysis estimate.
 
 ## What it detects
 
@@ -197,6 +215,13 @@ python3 ~/.claude/skills/workflow-generator/scripts/analyze.py . ~/WORKFLOW.html
 # then open ~/WORKFLOW.html
 ```
 
+**Optional flags:**
+
+```bash
+--access-log /path/to/access.log   # overlay real request counts onto the dependency graph
+--graph-detail auto|files|dirs     # force file-level or directory-level graph nodes (default: auto)
+```
+
 ---
 
 ## Example output (terminal)
@@ -218,15 +243,19 @@ External sources: Jira, Azure DevOps, Slack
 
 ```
 workflow-generator/
-├── SKILL.md              ← Claude Code skill definition
-├── INSTALL.md            ← detailed per-platform install guide
+├── SKILL.md                        ← Claude Code skill definition
+├── INSTALL.md                      ← detailed per-platform install guide
+├── workflow_generator_mcp/
+│   ├── analyze.py                  ← core scanner + HTML renderer (stdlib only)
+│   └── server.py                   ← MCP stdio server (package form)
 ├── scripts/
-│   └── analyze.py        ← core scanner + HTML renderer (stdlib only)
+│   └── analyze.py                  ← thin compatibility shim -> workflow_generator_mcp/analyze.py
+├── tests/                          ← pytest suite for the scanner
 ├── mcp/
-│   ├── server.py         ← MCP stdio server
-│   └── requirements.txt  ← pip install mcp
+│   ├── server.py                   ← MCP stdio server
+│   └── requirements.txt            ← pip install mcp
 └── copilot/
-    ├── index.js          ← GitHub Copilot Extension (Express)
+    ├── index.js                    ← GitHub Copilot Extension (Express)
     ├── package.json
     └── openai_function.json
 ```

SKILL.md

@@ -14,7 +14,8 @@ description: |
   "show me the components", "map the services", "diagram this", "visualize the stack",
   "show request flow", "show data flow", "trace the pipeline", "architecture of this project",
   "system map", "service dependencies", "component graph", "what calls what", "show me the
-  architecture", "draw the architecture", "explain the system design", "infrastructure diagram".
+  architecture", "draw the architecture", "explain the system design", "infrastructure diagram",
+  "dependency graph", "call graph", "module graph", "what imports what", "import graph".
 
   CAPACITY / CONCURRENCY ANALYSIS INTENT: user asks about concurrent request capacity,
   max throughput, practical throughput ceiling, worker count, worker processes, async workers,
@@ -60,6 +61,12 @@ Scan the current project and produce a complete visual system workflow with conc
    python3 ~/.claude/skills/workflow-generator/scripts/analyze.py <project_root> <project_root>/WORKFLOW.html
    ```
 
+   Optional flags, appended after the two positional args:
+   - `--access-log <path>` — overlay real request counts (from a combined/common log file) onto
+     the codebase dependency graph's HTTP-entry edges, instead of import-direction only.
+   - `--graph-detail auto|files|dirs` — force file-level or directory-level graph nodes. Default
+     `auto` aggregates to directories once a project exceeds ~350 source files.
+
 3. **Open the output**:
    ```bash
    xdg-open <project_root>/WORKFLOW.html 2>/dev/null || open <project_root>/WORKFLOW.html 2>/dev/null || true
@@ -112,6 +119,15 @@ The generated `WORKFLOW.html` always contains all of the following:
    from the same min() comparison used for the "Practical Throughput" stat, so this
    section and that stat can never disagree with each other.
 
+6. **Codebase Dependency Graph** — a force-directed graph of every source file (Python, JS/TS,
+   Go, Java, Rust, Ruby) as a node and every real, parser-resolved import as an edge; files that
+   match an already-detected component (an LLM call, a database client, a queue) also get an edge
+   to that component. Click a node to isolate its neighbors; hover for file details. Large
+   projects (350+ files) are aggregated to directory-level nodes automatically. Edges are
+   import-direction only (static, honest) unless `--access-log` was supplied, in which case
+   HTTP-entry edges are weighted with real observed request counts — the graph is explicit about
+   which kind of edge is which so it's never mistaken for a traced request path.
+
 ## What the analyzer detects
 
 ### Workers & replicas

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "workflow-generator-mcp"
-version = "0.2.1"
+version = "0.3.0"
 description = "Generate a visual system architecture diagram (WORKFLOW.html) with concurrency capacity estimates and bottleneck analysis from any codebase — CLI + MCP server, stdlib-only scanner."
 readme = "README.md"
 license = { text = "MIT" }

server.json

@@ -7,12 +7,12 @@
     "url": "https://github.com/askuma/workflow-generator",
     "source": "github"
   },
-  "version": "0.2.1",
+  "version": "0.3.0",
   "packages": [
     {
       "registryType": "pypi",
       "identifier": "workflow-generator-mcp",
-      "version": "0.2.1",
+      "version": "0.3.0",
       "transport": {
         "type": "stdio"
       }

tests/conftest.py

@@ -0,0 +1,22 @@
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+import pytest
+
+from workflow_generator_mcp import analyze as az  # noqa: E402
+
+
+@pytest.fixture
+def az_module():
+    return az
+
+
+def write_project(tmp_path: Path, files: dict) -> Path:
+    """Write {relative_path: content} under tmp_path and return the root."""
+    for rel, content in files.items():
+        p = tmp_path / rel
+        p.parent.mkdir(parents=True, exist_ok=True)
+        p.write_text(content)
+    return tmp_path

tests/test_access_log.py

@@ -0,0 +1,58 @@
+from conftest import write_project
+
+
+def test_route_to_regex_handles_all_placeholder_styles(az_module):
+    az = az_module
+    for tmpl in ('/items/<id>', '/items/{id}', '/items/:id'):
+        rx = az._route_to_regex(tmpl)
+        assert rx.match('/items/42')
+        assert not rx.match('/items/42/extra')
+
+
+def test_parse_access_log_matches_routes_and_buckets_other(az_module, tmp_path):
+    az = az_module
+    log = tmp_path / "access.log"
+    log.write_text(
+        '127.0.0.1 - - [10/Oct/2023:13:55:00 +0000] "GET /items/1 HTTP/1.1" 200 512\n'
+        '127.0.0.1 - - [10/Oct/2023:13:55:30 +0000] "GET /items/2 HTTP/1.1" 200 512\n'
+        '127.0.0.1 - - [10/Oct/2023:13:56:00 +0000] "POST /webhook/github HTTP/1.1" 204 0\n'
+        '127.0.0.1 - - [10/Oct/2023:13:56:30 +0000] "GET /unmatched HTTP/1.1" 404 128\n'
+    )
+    routes = ['GET /items/{id}', 'POST /webhook/github']
+    result = az.parse_access_log(str(log), routes)
+    assert result is not None
+    assert result['total'] == 4
+    assert result['by_route']['GET /items/{id}'] == 2
+    assert result['by_route']['POST /webhook/github'] == 1
+    assert result['other'] == 1
+    assert result['rpm'] is not None
+    assert result['rpm'] > 0
+
+
+def test_parse_access_log_missing_file_returns_none(az_module, tmp_path):
+    az = az_module
+    missing = tmp_path / "nope.log"
+    assert az.parse_access_log(str(missing), ['GET /items/{id}']) is None
+
+
+def test_parse_access_log_no_matching_lines_returns_none(az_module, tmp_path):
+    az = az_module
+    log = tmp_path / "empty.log"
+    log.write_text("this is not a log line at all\njust some text\n")
+    assert az.parse_access_log(str(log), ['GET /items/{id}']) is None
+
+
+def test_gen_k6_script_picks_up_to_five_get_routes_and_fills_params(az_module):
+    az = az_module
+    routes = [f"GET /items/{{id}}{i}" for i in range(7)]
+    script = az._gen_k6_script(routes)
+    assert script.count("http.get(") == 5
+    assert "{id}" not in script
+    assert "BASE_URL" in script
+
+
+def test_gen_k6_script_falls_back_to_non_get_when_no_get_routes(az_module):
+    az = az_module
+    routes = ['POST /webhook/github', 'PUT /items/{id}']
+    script = az._gen_k6_script(routes)
+    assert "http.post(" in script or "http.put(" in script

tests/test_capacity.py

@@ -0,0 +1,63 @@
+def test_semaphore_tightest_wins_concurrency_ranking(az_module):
+    az = az_module
+    workers = {"uvicorn_workers": 4, "gunicorn_workers": None, "replicas": 1,
+               "pm2_instances": None, "is_async": True}
+    concur = {"semaphores": [2]}
+    cap = az.compute_concurrency(workers, gateway=None, concur=concur, llm={}, api=None)
+    # total_io = 4 workers * 100 = 400; sem_ceiling = 2 * 4 = 8 -> semaphore is tighter
+    assert cap["bottleneck"] == "Semaphore limit"
+    assert cap["practical"] == "~8 concurrent I/O"
+
+
+def test_gateway_rate_limit_beats_looser_app_rate_limit(az_module):
+    az = az_module
+    workers = {"uvicorn_workers": 1, "gunicorn_workers": None, "replicas": 1,
+               "pm2_instances": None, "is_async": True}
+    gateway = {"type": "nginx", "rate_limits": [{"zone": "api", "rate": "30r/m", "burst": 10}]}
+    api = {"app_rate_limits": [{"rate": "100", "unit": "minute"}]}
+    cap = az.compute_concurrency(workers, gateway=gateway, concur={}, llm={}, api=api)
+    assert cap["bottleneck"] == "nginx rate limit"
+    assert cap["practical"] == "~30/min"
+
+
+def test_app_rate_limit_tighter_than_gateway(az_module):
+    az = az_module
+    workers = {"uvicorn_workers": 1, "gunicorn_workers": None, "replicas": 1,
+               "pm2_instances": None, "is_async": True}
+    gateway = {"type": "nginx", "rate_limits": [{"zone": "api", "rate": "500r/m", "burst": 10}]}
+    api = {"app_rate_limits": [{"rate": "20", "unit": "minute"}]}
+    cap = az.compute_concurrency(workers, gateway=gateway, concur={}, llm={}, api=api)
+    assert cap["bottleneck"] == "Application rate limit"
+    assert cap["practical"] == "~20/min"
+
+
+def test_llm_timeout_derived_ceiling_uses_actual_worker_count(az_module):
+    az = az_module
+    workers = {"uvicorn_workers": 2, "gunicorn_workers": None, "replicas": 1,
+               "pm2_instances": None, "is_async": True}
+    llm = {"providers": ["OpenAI"], "timeout": 30}
+    cap = az.compute_concurrency(workers, gateway=None, concur={}, llm=llm, api=None)
+    # concurrency_ceiling = 2 workers * 100 = 200; llm_rpm = 200 * (60/30) = 400
+    assert cap["bottleneck"] == "OpenAI latency"
+    assert cap["practical"] == "~400/min"
+
+
+def test_ranking_and_bottleneck_never_disagree(az_module):
+    az = az_module
+    workers = {"uvicorn_workers": 4, "gunicorn_workers": None, "replicas": 2,
+               "pm2_instances": None, "is_async": True}
+    gateway = {"type": "nginx", "rate_limits": [{"zone": "api", "rate": "50r/m", "burst": 5}]}
+    llm = {"providers": ["Anthropic"], "timeout": 10}
+    api = {"app_rate_limits": []}
+    cap = az.compute_concurrency(workers, gateway=gateway, concur={"semaphores": [3]}, llm=llm, api=api)
+    assert cap["ranking"][0][0] == cap["bottleneck"]
+    assert cap["ranking"] == sorted(cap["ranking"], key=lambda c: c[1])
+
+
+def test_no_evidence_falls_back_to_concurrency_ceiling(az_module):
+    az = az_module
+    workers = {"uvicorn_workers": None, "gunicorn_workers": None, "replicas": 1,
+               "pm2_instances": None, "is_async": False}
+    cap = az.compute_concurrency(workers, gateway=None, concur={}, llm={}, api=None)
+    assert cap["ranking_kind"] == "concurrency"
+    assert cap["practical"] == "~1 concurrent I/O"