Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -42,27 +42,37 @@ falling through to a blank a rescuer has to fill from nothing.
`fs.WalkDir(root.FS(), …)` walk — rather than inventing a second traversal
idiom.

- **Containment.** The walk runs over `c.root.FS()`, so `os.Root` refuses any
component that escapes the repository root, symlinked intermediates included.
A `nil` root (unopenable repository) returns `(nil, false)`.
- **Containment.** The walk descends through a sub-root per directory
(`os.Root.OpenRoot`); `os.Root` refuses any component that escapes the
repository root, symlinked intermediates included, and a symlinked directory
is detected from its `ReadDir` type and skipped before it is ever opened. The
containment property holds unchanged while each child opens in O(1). A `nil`
root (unopenable repository) returns `(nil, false)`.
- **Symlinks are skipped, never followed.** Unlike embark — where a symlink in a
packed lifeboat is a trust violation and therefore fatal — a probe reads an
arbitrary foreign tree where symlinks are ordinary. The walk skips symlinked
entries (files and directories) and continues; it never errors on one.
- **Skip set.** Directory names never descended into: `.git`, `node_modules`,
`vendor`, `generated`. These are dependency, VCS-internal, and generated
trees — never a team's own open questions, and the dominant cost of an
unfiltered walk.
- **Caps, in three dimensions.** `maxWalkFiles` mirrors `maxDirEntries`
(50 000) and bounds regular files *and* directories visited: a tree of
directories holding nothing regular yields no path, so a file cap alone never
fires there and the walk would run to exhaustion over a foreign tree.
`maxWalkDepth` (32) bounds descent, because `os.Root` resolves every
directory from the containment root one component at a time — a chain of
directories costs the square of its depth, and a few thousand of them are
trivial to create and take minutes to traverse. Real trees are shallow, so
the depth cap prunes only pathological chains, and prunes the chain rather
than abandoning the tree. Any bound firing returns `truncated = true`.
- **Skip set.** Directory names never descended into, matched by name at any
depth: `.git` (VCS); `node_modules` (Node); `vendor`, `generated`
(Go/generic); `.venv`, `venv`, `.tox`, `__pycache__` (Python); `target`,
`build`, `dist` (build/distribution output); `Pods` (CocoaPods). These are
dependency, VCS-internal, language-cache, and generated trees across the
common ecosystems — never a team's own open questions, and the dominant cost
of an unfiltered walk; walked as source, a vendored `TODO` is cited as this
project's own open question.
- **Caps.** `maxWalkFiles` mirrors `maxDirEntries` (50 000) and bounds regular
files *and* directories visited: a tree of directories holding nothing regular
yields no path, so a file cap alone never fires there and the walk would run to
exhaustion over a foreign tree. `maxDirEntries` (50 000) is also the
per-directory read bound: each directory is read with a bounded `ReadDir` — the
one canonical guard `ListDir` uses — so a single directory of millions of
entries cannot balloon memory before the file cap applies. `maxWalkDepth` (32)
bounds descent: the walk holds a sub-root per directory (`os.Root.OpenRoot`),
so each child opens in O(1) and a chain costs O(depth) rather than the square
of its depth, but an unbounded chain is still an unbounded recursion and a
pathological cost, so the cap prunes it — pruning the chain rather than
abandoning the tree. Real trees are shallow. Any bound firing returns
`truncated = true`.
Truncation is *reported*, never silent (loud-staging): an adapter that hits a
cap says so in its cited evidence, and a blank drawn from a truncated walk
says the walk was truncated.
Expand Down Expand Up @@ -140,8 +150,8 @@ The primitive therefore adds no second read path to audit.
| Which markers? | `TODO`, `FIXME`, `XXX`, `HACK`, `BUG`; uppercase only; word-boundary anchored. `TODO`/`FIXME` require a trailing `:`/`(` (a bare word reads as prose that names the marker, iss-111); `XXX`/`HACK`/`BUG` also admit a bare word (trailing space/EOL). `NOTE`, `OPTIMIZE` excluded. |
| Which tier — conventions or git? | **Conventions.** A working-tree file scan through the `SourceContext` file surface. The adapter never touches git, so it grounds a bare snapshot as readily as a working tree. |
| Scan scope and the missing primitive | Option (a): add a bounded recursive-walk primitive, `WalkFiles`, to `SourceContext`. It is shared with itd-96, so the walk lands once. |
| Which files to scan | Every regular file the walk yields, minus the skip set (`.git`, `node_modules`, `vendor`, `generated`), minus symlinks, minus binaries (NUL-byte heuristic), minus oversized files (`ReadFile`'s cap). |
| Per-repo caps | `maxWalkFiles` = 50 000 files **and** 50 000 directories (mirrors `maxDirEntries`), `maxWalkDepth` = 32 levels of descent, `maxProbeReadBytes` per file, `maxMarkerScanBytes` = 512 MiB across the scan (reuses `maxPlanTotalBytes`), `maxMarkerCitations` = 200 citations. Every bound that fires is reported in the cited evidence. |
| Which files to scan | Every regular file the walk yields, minus the skip set (`.git`, `node_modules`, `vendor`, `generated`, `.venv`, `venv`, `.tox`, `__pycache__`, `target`, `build`, `dist`, `Pods`), minus symlinks, minus binaries (NUL-byte heuristic), minus oversized files (`ReadFile`'s cap). |
| Per-repo caps | `maxWalkFiles` = 50 000 files **and** 50 000 directories (mirrors `maxDirEntries`), `maxDirEntries` = 50 000 entries read per directory, `maxWalkDepth` = 32 levels of descent, `maxProbeReadBytes` per file, `maxMarkerScanBytes` = 512 MiB across the scan (reuses `maxPlanTotalBytes`), `maxMarkerCitations` = 200 citations. Every bound that fires is reported in the cited evidence. |
| Output shape and framing | Headline count + up to 200 `path:line (MARKER)` citations, `dedupeSorted`. |
| Dedup | `dedupeSorted` on the rendered citation string, so an identical `path:line (MARKER)` appears once. Multiple distinct markers in one file each keep their own line. |
| Status and confidence thresholds | Ceiling `StatusPartial`. `ConfidenceMedium` at ≥ 10 markers, else `ConfidenceLow`. Never `StatusGrounded`. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ category: "tech-debt"
source: "impl-review"
found_during: "itd-95 P1 review"
found_at: "internal/core/lifeboat/probe.go"
resolution: "Bounded the walk's per-directory read with the shared maxDirEntries guard (readDirBounded), so one directory of millions of entries no longer materialises before the file cap."
impact: fix
---

WalkFiles caps the number of regular files it returns, but fs.WalkDir calls ReadDir(-1) on every directory it enters, so a single directory holding millions of entries balloons memory before the file cap can apply. The maxWalkFiles doc comment claims the walk stays bounded in memory, which is only true of the returned slice. ListDir already bounds this with ReadDir(maxDirEntries); the walk does not share that guard. Found by an independent security review of the itd-95 diff.
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ category: "tech-debt"
source: "impl-review"
found_during: "itd-96 P1 review"
found_at: "internal/core/lifeboat/probe.go"
resolution: "Walk now holds an os.Root sub-root per directory (OpenRoot), opening each child O(1); cost is O(entries) and depth-independent (48000-dir depth-30 tree ~1.4s vs ~7s), containment preserved."
impact: fix
---

WalkFiles opens every directory through os.Root.FS(), which re-resolves the whole path from the containment root one component at a time, so a walk costs O(entries x depth) component opens. Measured post-fix: a 60000-directory tree at depth 30 takes 10.5s for one probe (two adapters walk it concurrently). The depth and directory caps bound this, but the constant is high. os.Root.OpenRoot would let the walk hold a sub-root per directory and open each child in O(1). Found while fixing the unbounded-walk finding in the itd-96 review.
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ category: "tech-debt"
source: "impl-review"
found_during: "itd-95 P1 review"
found_at: "internal/core/lifeboat/probe.go"
resolution: "Widened walkSkipDirs to the common Python/build-output/CocoaPods trees (.venv, venv, .tox, __pycache__, target, build, dist, Pods); vendored markers no longer cited and the dot-prefixed tree no longer eats the cap before src/."
impact: fix
---

The WalkFiles skip set covers only Node and Go dependency trees, so a Python .venv, a Rust target/, __pycache__, dist/, build/ and Pods/ are walked as if they were the team's own source. Two consequences on a record-less repo: the open-questions adapter cites a vendored dependency's TODO as this project's open question, and because fs.WalkDir reads directories in lexical order a large dot-prefixed dependency tree can consume the 50000-file walk cap before the project's own src/ is reached. Found by an independent review of the itd-95 diff; spc-12 names the current skip set, so widening it is a design revisit rather than an implementation fix.
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,23 @@ called out in a **Breaking** section.

### Fixed

- **The disembark probe's recursive file walk is bounded per directory, opens
each child in O(1), and skips the common ecosystems' dependency trees**
(iss-112, iss-114, iss-116). The walk now reads every directory with a bounded
`ReadDir` (the same 50 000-entry guard `ListDir` uses), so a single directory
of millions of entries can no longer balloon memory before the file cap
applies. It holds a sub-root per directory (`os.Root.OpenRoot`) instead of
re-resolving every path from the containment root one component at a time, so a
deep tree costs O(entries) rather than O(entries × depth) — a 48 000-directory
depth-30 tree walks in ~1.4 s where the old walk took ~7 s, and the cost is now
independent of depth. The `os.Root` containment guarantee is unchanged: a
symlink is still refused rather than followed out of the tree. The skip set
widens beyond Node and Go to the common dependency, cache, and build-output
trees — Python (`.venv`, `venv`, `.tox`, `__pycache__`), Rust and generic
build output (`target`, `build`, `dist`), and CocoaPods (`Pods`) — so a
vendored `TODO` is no longer cited as the project's own open question, and a
large dot-prefixed dependency tree can no longer exhaust the walk cap before
the project's own `src/` is reached.
- **The open-questions marker scan no longer reads documentation about markers as
open questions** (iss-111). The pattern that grounds `evidence/open-questions`
admitted a bare uppercase `TODO`/`FIXME` followed by whitespace, so on a
Expand Down
Loading