feat(apps): create sidebar folders from a scanned project

[billygerhard]

Problem / Motivation

A chat folder already carries a project_dir and nests via parent_id, so a chat opened inside one is steering- and scope-correct for its package. What is missing for a monorepo, or a directory of sibling repositories, is population: assembling N sub-folders by hand is work nobody does, so per-package steering never loads. Filed as #2516; submitted as a PR per the contribution FAQ ("proposals are filed as pull requests and debated in the open") after the issue sat in needs-human triage for two weeks.

Why it matters

Multi-package workspaces are the norm for real projects (three independent filers hit this within one week: #1637, #2164, #2516), and today the folder feature's per-package steering silently never engages for them — the user would have to hand-build the tree first, so the capability that exists goes unused exactly where it helps most.

What changed (motivation → approach → change)

Goal: populate the folder tree from the filesystem, without trusting the scan and without new persistence. Approach: a pure read-only scanner plus a preview-then-create endpoint pair, with the UI as a thin surface — chosen over teaching the folder API itself to scan (couples a slow filesystem walk into the create path) and over a client-side walker (the browser cannot see the disk; the server must re-derive anyway). What was built:

Scanner (src/kiro_crew/project_scan.py): walks the pointed-at tree read-only and returns candidate packages. Signals: the directory's own .git/.kiro, a recognized manifest (npm, Python, Rust, Go, JVM, Flutter, PHP, Ruby, Elixir, Swift, Deno, Scala), a deploy-root marker (firebase.json, vercel.json, cdk.json, Procfile, ...), or being named by a workspace declaration (workspaces, pnpm-workspace.yaml, Cargo workspace, go.work). Two confidence tiers decide what arrives pre-ticked; deploy roots stay confident at any depth because a deploy config never names a build fixture. The project's own .gitignore prunes with git semantics via pathspec (nested files stack, !negation, ignored directories are never entered — which defeats SwiftPM's checkouts/ store of vendored git clones), alongside a name-prune list (node_modules, venv, DerivedData, ...).

Endpoints (dashboard/chat_folder_scaffold.py): POST /api/project-scaffold/scan (dry-run preview, creates nothing) and POST /api/project-scaffold/create (creates the confirmed selection through the existing folder create path — one writer, no new persistence, no schema change). The scaffold re-derives candidates server-side and refuses any selected path the fresh scan did not offer, so a stale preview and a forged path are the same 400. Additive only: re-scans offer additions, existing folders are reported and never touched, partial failure is reported rather than rolled back.
Also in this change, called out for reviewers: this change adds NO config key. Both scaffold.extra_manifest_signals and scaffold.depth_cap were dropped per First Principles review — one reader, no setter, no filed need, and a shipped config key is permanent surface — so the scan's depth bound is now the scanner's own DEFAULT_DEPTH_CAP and the built-in manifest set widens by code change; and the shared ProjectPicker gains max-w-[calc(100vw-16px)] so its fixed 400px panel clamps on narrow viewports for every caller. Per First Principles review, the scan response's groups field was removed — it had zero consumers (the preview derives grouping from each candidate's parent_path); it can return with its first consumer.

Builtin app (apps/builtins/project_scaffolder/): the UI surface — directory picker (reusing the core ProjectPicker), grouped preview with per-group select-all, confident-first ordering, a collapsed disclosure for speculative sub-folders, and result reporting. Manifest-only; it calls the two host endpoints and duplicates no logic.

On shape: delivered as a builtin app to keep core UI untouched. The engine + endpoints are severable (branch feature/create-folder-from-project on the fork holds them as 16 standalone commits) — happy to split into two PRs or reshape the surface as a core folder-UI action if maintainers prefer; the endpoints were designed as the repackaging boundary so only the last-mile surface changes.

Tests

603 Python tests: scanner fixtures over real temp directory layouts (symlinks, permissions, depth caps, gitignore negation/nesting/scope-boundary, the SwiftPM reproduction with its non-vacuity control), hypothesis property tests ("two scans of an unchanged tree compare equal", "no candidate is ever gitignore-matched", "ignoring a child prunes exactly that subtree and never invents candidates"), and endpoint tests including byte-identical 400-body parity with the manual folder flow and route-registration guards. 26 frontend tests over the app page: keyboard reach, selection isolation (main select-all does not reach the collapsed section), zero-fetch picker interaction, stale-selection rescan flow. i18n gates (37 keys × 13 locales), tsc, eslint, flake8, mypy, docs-lint all clean.

Manual verification

Exercised end to end against real project layouts: a directory of sibling repositories, an npm-workspaces monorepo with Firebase apps (two deploy roots + shared libs), and an iOS/SwiftPM project whose gitignored dependency store previously flooded the preview. Verified scan → tick → create → folders appear correctly nested, re-scan offers only additions, and chats opened in created folders load that package's steering.

Screenshots / video

Scan preview of a workspace monorepo — deploy roots arrive confident and pre-ticked, workspace-member libs are offered, speculative sub-folders sit in the collapsed disclosure:

The created tree — folders nest as they do on disk, each with its project_dir bound:

Related Issues

Closes #2516. Related: #2164 (workspace-first model this composes toward — its roots are manually managed, so this scanner remains the missing populate step under either model), #1637 (folded into #2164).

Checklist

• Single commit with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
• Existing tests pass and new tests added for new functionality
• Self-review completed; code follows project style guidelines
• Documentation updated (if applicable)
• No secrets, credentials, or internal references in the diff

Contribution License Agreement

N/A — no CLA text is published in the template yet; will sign whatever OSPO supplies when it lands.

Generated file touched outside the feature's own sources

Called out in the commit body as well:

• error-code-baseline.json — re-snapshotted, counts only go down. Extracting create_folder_record moved two of chat_folders.py's refusals behind a FolderCreateError that carries a code, so its missing_code count improves 21 → 19 (totals 1370 → 1368) and test_baseline_is_not_stale requires the snapshot be refreshed via the documented python test/test_error_code_contract.py --update. Nothing was regenerated to silence a regression: the except FolderCreateError handler returns inline dict literals at each json_response site rather than a hoisted payload local, keeping opaque_body at 0 for that file, and the wire shape is byte-identical to the pre-extraction handler (code present only when the refusal carries one).

Bundle budget note

scripts/check-bundle-size.mjs keeps upstream's all (eager i18n catalog) ceiling at 10490 KB — this branch does NOT move it. An earlier revision raised a lower ceiling; rebasing onto a main that had already raised it higher made that unnecessary, so only the budget entry's comment changes, to record that this feature's 39 catalog keys (~107 KB of eager strings per catalog) ride inside the headroom upstream's measurement already left. No new library or surface reaches that chunk — the app's page sits behind a lazy import() in src/apps/builtinRegistry.ts.

[billygerhard]

Dispositions — GPT 5.6 Review

• fixed — BLOCKING, src/kiro_crew/project_scan.py:768: the validated scan root is not identity-pinned.

  Validated root -> agent swaps an ancestor for a symlink -> scanner enumerates a protected governance directory.

  Confirmed by construction before fixing: the root _Frame was the one frame carrying identity=None, _verify_identity returns on None, and O_NOFOLLOW guards only the final component — an ancestor swap redirects the whole root path while its last component stays a real directory, so nothing refused it. Fixed exactly as prescribed on head 4fce2f519: scan() captures the root's (st_dev, st_ino) via os.lstat at entry — after the caller's validation, before the first open — and seeds the root frame with it, so the first _scandir_pinned compares the opened descriptor against the inode the validation named. A failed or zero-inode lstat records no identity and the first read then fails on the same condition and warns.

  The same shape existed in member-pattern expansion, which opens by name with no identity at every level (_child_dirs/_descendants/_expand_member_pattern all passed expect=None). Fixed by threading the identity chain through expansion: the declaring package's frame identity seeds the frontier, and each level's listing supplies the identity its children are opened against — after this, no directory in the module is ever opened without a recorded identity.

  Test: TestDirectorySwapRace::test_an_ancestor_of_the_root_swapped_after_validation_is_refused stages the ancestor swap in the window between scan()'s pin and the first open. Mutation-probed: with the root frame reverted to identity=None, exactly that test fails (1 failed, 64 passed); restored, the full walker+members suites pass (114) and the zero-inode Windows simulation passes (242) — on the unpinned branch the root identity now also carries the guard, since os.lstat reports real inodes on Windows (only directory listings zero them).

[billygerhard]

Dispositions — UX Review

• fixed — Empty-scan create failures render nowhere.

  createError and the stale banner render only inside the {scan && !isEmpty && <Card>…} branch … a refused create sets state no element reads.

  Correct on the error: the empty state's button shares createMut and a refusal was silent. The same role="alert" element (same data-testid, verbatim server prose) now renders beside the empty state's action on head 4fce2f519, with a test (renders a refused root-only create beside the empty state action). The stale half does not apply on this branch: the empty state's create sends an empty selection, and folder_scaffold_selection_stale is only ever raised for selected paths the fresh scan did not offer — with none selected there is nothing to be stale — so no stale prompt was added there.

• fixed — The root folder is created without ever being shown as part of the selection.

  Took the button-side wording you suggested: the preview's count now reads "Root folder + {{n}} selected" (n_selected, updated across all 13 locales, en-XA regenerated), so the always-created root is disclosed at the moment of commitment and a 0-ticked create no longer claims to be empty.

• rebutted — rename tier_offered: "Offered" to "Possible match".

  fails a cold read next to "Confident" (everything in the list was offered)

  The display label deliberately mirrors the documented two-tier vocabulary (auto/offered) used by the API field, the spec section, and the PR description; renaming the rendered label away from the tier's name invites drift between what a user reports ("Possible match") and what the docs and API call it. The cold-read cost is real but bounded — the tier badge sits beside a ticked/unticked checkbox that already carries the semantic.

• rebutted — drop the raw machine id ({f.code}) under failed rows.

  The code is the documented error-contract identifier (error-code-baseline.json); a failed row is precisely where a user copies something into a bug report, and the server prose alone is not stable across locales or releases. One line of developer vocabulary under an already-exceptional row buys unambiguous reporting.

[billygerhard]

Disposition for the GPT 5.6 review of 4fce2f519:

• A post-scan symlink swap escapes the confirmed root (chat_folder_scaffold.py:422) — fixed in 7ebc44e91.

Verified against source: _create_selection handed bare path strings to create_folder_record, whose _validate_project_dir re-ran realpath() at create time and persisted whatever it resolved to — a component swapped for a symlink in the scan-to-create window bound the folder outside the scanned tree. Fixed as prescribed, at the single create path: create_folder_record gains require_resolved_project_dir (opt-in, like unique_project_dir), which refuses with folder_project_dir_moved whenever validation resolves away from the caller-vouched canonical path. Both scaffold call sites opt in; the refusal is SEL-audited as a denial (same treatment as an ownership refusal) — a swapped root costs the whole call, a swapped candidate only its own path. The folder API proper leaves the flag off: a person naming their own ~/symlinked path is resolution working as intended. Two endpoint regression tests stage the exact race (swap inside the scan wrapper, after the fresh scan, before create): test_a_candidate_swapped_for_a_symlink_after_the_scan_is_refused and test_the_root_swapped_for_a_symlink_after_the_scan_is_refused. Mutation-verified fail-first: disabling the check fails exactly those two tests (2 failed / 54 passed), restored byte-identical. Spec (learn-cron-dashboard.md) updated to document the guard.

Span note for recurrence tracking: this is the first blocking finding in the chat_folder_scaffold._create_selection consumer span (rounds 8/11/12 were in the project_scan.py walker span, closed by the identity invariant). The fix extends the same invariant across the scan→create boundary rather than point-patching the call site.

[billygerhard]

Disposition for the UX review of 4fce2f519:

• Watch 1 — nested badge reuses n_selected, rendering two contradictory "Root folder + N selected" counts — fixed in 7ebc44e91.

The badge now has its own key n_selected_inside ("{{n}} selected inside"), added across en + the 11 human locales with en-XA regenerated. The test assertions on nested-selected are pinned to the full string ("1 selected inside" / "2 selected inside") exactly because the previous substring match hid this.

• Watch 2 — candidate rows print raw scanner tokens as the "why" — fixed in 7ebc44e91.

signalLabel() maps each token to a localized phrase before rendering: git → "git repository", .kiro → ".kiro folder found", member → "workspace member", manifest:<file> → " found" (4 new i18n keys, 13 locale files). An unknown token falls through unchanged so a future scanner signal degrades to its id rather than disappearing.

• Suggestion 1 — re-run the scan after a successful create so created rows flip to "Already set up" — accepted-and-deferred: filed as Project scaffolder: refresh preview after create; gate empty-state action on existing root #7974.

Real polish, frontend-only, and orthogonal to this round's security fix; deferring keeps the diff scoped to review findings.

• Suggestion 2 — empty state offers "Create the root folder only" even when root_existing is true — accepted-and-deferred: filed as Project scaffolder: refresh preview after create; gate empty-state action on existing root #7974 (same issue, second task).

Same reasoning: the response already carries root_existing, so this is a small self-contained frontend change that stands on its own.

[billygerhard]

Disposition for the First Principles review of 4fce2f519:

• Drop root_name and root_folder_id — zero consumers each — fixed in 7ebc44e91.

Verified: root_name and root_folder_id appeared only in their definitions, the TS types, and test fixtures — ProjectScaffolderPage.tsx reads neither. Both removed from the responses, the TS types, the tests, and the spec's documented response shapes. Same rule the PR already applied to groups; these two escaped it.

• Drop the scan response's status field — derivable as candidates.length === 0 — rebutted.

The zero-consumer rule the other two removals rest on does not reach this field: it has a live consumer (ProjectScaffolderPage.tsx line 490, scan.status === STATUS_EMPTY drives the empty-state branch). "Empty tree" is an answer the server distinguishes deliberately (HTTP 200, status: "empty"), and naming it beats every client re-deriving the same predicate from candidate length.

• Defer DECLARATION_PARSERS / member expansion — manifests already reveal well-formed members — rebutted (maintainer ruling).

The maintainer reviewed exactly this cut on 2026-09-01 and ruled to keep the workspace-member parsers. The observable additions the review itself names — the member signal chip (both detection reasons shown) and unticked candidates for declared members without manifests — are the intended behavior, and !pattern exclusion subtraction only exists with the parsers present.

• Watch — two independent "what marks a project" catalogs (MANIFESTS vs PROJECT_ROOT_MARKERS) — rebutted (no change).

As the review itself notes, they do deliberately different jobs: PROJECT_ROOT_MARKERS (artifact_source) answers "where does this file's project start" walking upward; MANIFESTS answers "is this directory a package worth a folder" walking downward with tier semantics. Unifying them would couple two vocabularies that legitimately evolve apart (e.g. deploy-root markers belong only to the scanner's tiering). Noted as a known observation rather than a defect.

[CrysisDeu]

/ai-review override fable c05f449: Maintainer decision: project-scaffolder ships as a built-in by design. Folder scaffolding is a once-per-project setup action that must run against the gateway's own project tree and folder-create path, so it is off by default (defaultEnabled:false) and adds no permanent core-UI surface; the scan/create endpoints stay the repackaging boundary if it is later moved to the KiroCrewApps registry. Accepting the no-new-builtin-apps exception for this commit.

[CrysisDeu]

/ai-review override design c05f449: Maintainer decision: project-scaffolder ships as a built-in by design. Folder scaffolding is a once-per-project setup action that must run against the gateway's own project tree and folder-create path, so it is off by default (defaultEnabled:false) and adds no permanent core-UI surface; the scan/create endpoints stay the repackaging boundary if it is later moved to the KiroCrewApps registry. Accepting the no-new-builtin-apps exception for this commit.

[CrysisDeu]

/ai-review override first-principles c05f449: Maintainer decision: project-scaffolder ships as a built-in by design. Folder scaffolding is a once-per-project setup action that must run against the gateway's own project tree and folder-create path, so it is off by default (defaultEnabled:false) and adds no permanent core-UI surface; the scan/create endpoints stay the repackaging boundary if it is later moved to the KiroCrewApps registry. Accepting the no-new-builtin-apps exception for this commit.

[github-actions]

Human judgment recorded

@CrysisDeu marked the fable AI finding as false positive, not applicable, or explicitly accepted for c05f449988f1140f311ffbaed5b652d1c0fa631b.

Maintainer decision: project-scaffolder ships as a built-in by design. Folder scaffolding is a once-per-project setup action that must run against the gateway's own project tree and folder-create path, so it is off by default (defaultEnabled:false) and adds no permanent core-UI surface; the scan/create endpoints stay the repackaging boundary if it is later moved to the KiroCrewApps registry. Accepting the no-new-builtin-apps exception for this commit.

This decision applies only to this commit. A new push requires a new judgment.

[github-actions]

Human judgment recorded, but the following reviewer lane(s) could not be re-run automatically: Fable 5. Re-run the lane's latest workflow run for c05f449988f1140f311ffbaed5b652d1c0fa631b manually from the Actions tab.

[github-actions]

Human judgment recorded

@CrysisDeu marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for c05f449988f1140f311ffbaed5b652d1c0fa631b.

Maintainer decision: project-scaffolder ships as a built-in by design. Folder scaffolding is a once-per-project setup action that must run against the gateway's own project tree and folder-create path, so it is off by default (defaultEnabled:false) and adds no permanent core-UI surface; the scan/create endpoints stay the repackaging boundary if it is later moved to the KiroCrewApps registry. Accepting the no-new-builtin-apps exception for this commit.

This decision applies only to this commit. A new push requires a new judgment.

[github-actions]

Human judgment recorded, but the following reviewer lane(s) could not be re-run automatically: First Principles Review. Re-run the lane's latest workflow run for c05f449988f1140f311ffbaed5b652d1c0fa631b manually from the Actions tab.

[CrysisDeu]

/ai-review override design c05f449: Maintainer decision: project-scaffolder ships as a built-in by design. Folder scaffolding is a once-per-project setup action that must run against the gateway's own project tree and folder-create path, so it is off by default (defaultEnabled:false) and adds no permanent core-UI surface; the scan/create endpoints stay the repackaging boundary if it is later moved to the KiroCrewApps registry. Accepting the no-new-builtin-apps exception for this commit.

[github-actions]

Human judgment recorded

@CrysisDeu marked the design AI finding as false positive, not applicable, or explicitly accepted for c05f449988f1140f311ffbaed5b652d1c0fa631b.

Maintainer decision: project-scaffolder ships as a built-in by design. Folder scaffolding is a once-per-project setup action that must run against the gateway's own project tree and folder-create path, so it is off by default (defaultEnabled:false) and adds no permanent core-UI surface; the scan/create endpoints stay the repackaging boundary if it is later moved to the KiroCrewApps registry. Accepting the no-new-builtin-apps exception for this commit.

This decision applies only to this commit. A new push requires a new judgment.

[github-actions]

Human judgment recorded, but the following reviewer lane(s) could not be re-run automatically: Design Review. Re-run the lane's latest workflow run for c05f449988f1140f311ffbaed5b652d1c0fa631b manually from the Actions tab.

[CrysisDeu]

@billygerhard — maintainer update on the built-in-app question and one fix request before this can land.

Delivery as a built-in app: accepted. The no-new-builtin-apps exception is recorded on this head via /ai-review override for the Opus, Design and First Principles lanes (see the three "Human judgment recorded" comments above). You do not need to reshape the delivery. One caveat: on a fork PR the lanes re-review rather than consume the override marker, so the red checks will not clear on their own — I will handle the readiness path once the code is final.

Please fix before the next push — the GPT 5.6 finding on src/kiro_crew/dashboard/chat_folders.py (create_folder_record, currently line 618):

project_dir, err = _validate_project_dir(requested_dir)

_validate_project_dir does realpath/isdir syscalls and is called synchronously inside an async def. On the single-create API this was pre-existing, but the scaffold now calls create_folder_record once per candidate folder in a loop (chat_folder_scaffold.py ~L413 and ~L487), so one stalled network mount stalls the gateway for the whole batch — which is exactly the reason _resolve_root and the scan are already moved off-loop in that same file. Anchor: no-blocking-call-on-event-loop.

Fix: await it off the loop, matching the overlap check a few lines below (L631):

project_dir, err = await asyncio.to_thread(_validate_project_dir, requested_dir)

The two pre-existing synchronous calls in api_chat_folder_create / api_chat_folder_update can move to the same shape if you want, but they are not required for this PR.

After you push, ping me here — the override is SHA-scoped, so I will re-issue it for the new head.

[bolichen97]

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

• This PR is OVERLAPPING with PR #3139. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #5890: CONTINUE_DEVELOPMENT. Same file, complementary changes with no semantic collision; at most a trivial merge in one JSX attribute list. Files: website/src/components/ProjectPicker.tsx.
• This PR is OVERLAPPING with PR #3987. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #5890: CONTINUE_DEVELOPMENT. Independent features that happen to grow a second filesystem-discovery walker. Both can land as-is; worth flagging so a later change can factor one bounded walk out rather than maintaining two sets of caps and containment rules. Files: src/kiro_crew/project_scan.py.
• This PR is OVERLAPPING with PR #7353. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #5890: CONTINUE_DEVELOPMENT. Same function, opposite direction (extract vs extend) plus a contradicting spec sentence about the icon field. Maintainers should sequence them and have the later PR rebase onto create_folder_record; nothing here argues against either landing. Files: src/kiro_crew/dashboard/chat_folders.py.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

[CrysisDeu]

@billygerhard thanks — the rebase onto current main (07bdb012) is received and CI is running on it.

One thing still open: the asyncio.to_thread change for _validate_project_dir inside create_folder_record (src/kiro_crew/dashboard/chat_folders.py, still line 618 on this head) hasn't landed yet — details in #5890 (comment). Once that's in, ping me and I'll re-issue the SHA-scoped overrides for the new head.

[CrysisDeu]

@billygerhard — thank you for this contribution and for the many review rounds you carried it through.

Since the one remaining reviewer request (the asyncio.to_thread change for _validate_project_dir in create_folder_record) had been open for 48h after the first and second request, I have carried the work onto a same-repo branch so it can land: #8924 (#8924).

What #8924 contains, relative to this PR's head 743b17637:

• Your commit, rebased onto current main (two small conflicts: ProjectPicker.tsx now carries both main's keyboard-isolation barrier and your max-w clamp; error-code-baseline.json carries main's chat_handlers.py improvement alongside your chat_folders.py 21 → 19).
• The requested fix: create_folder_record runs _validate_project_dir via asyncio.to_thread, plus a thread-identity test pinning it off the loop.
• The no-new-builtin-apps maintainer exception re-recorded on the new head via /ai-review override (the fork lanes here cannot consume those markers, which is why a same-repo branch was needed).

Your authorship is preserved with a Co-authored-by: Billy Gerhard trailer on the single commit, and Closes #2516 is carried. This PR stays open for your reference; if you would rather push the fix here yourself and have this one land instead, say so and I will close #8924.

[bolichen97]

Closing as delivered by merged #8924 (a7ea47b19).

#8924 is a maintainer takeover of this exact commit (its body says so: "Takeover of PR #5890"), rebased and extended with the one open review finding this branch never landed (create_folder_record awaiting _validate_project_dir via asyncio.to_thread, with a thread-identity test) plus two further review rounds. Your Co-authored-by trailer is preserved on the merged commit, and it carries Closes #2516, which is now closed as completed. The audit verified on the full 46-file diff that every behaviour here -- scanner, endpoints, single-writer folder creation, UI -- is present on origin/main.

Nothing further to carry over from this branch. Thank you for building the feature.