feat(workspace): per-run execution worktrees via strategy runScope (BLO-19063)

[allyblockcast]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work
• Agents execute in "execution workspaces" — a cwd realized per issue, optionally as a git worktree
• Those worktrees are keyed by issue, not by run, and the fleet default (shared_workspace + project_primary) gives every run of every issue in a project the identical cwd and branch
• So two live runs silently share one working tree: one run's git checkout, staged index, or rm -rf node_modules clobbers the other, and uncommitted work sits on no branch where the obvious cleanup destroys it
• This fired in production twice in five days (BLO-21427, and again on this issue), and 93 hand-rolled worktrees have accumulated as agents route around the default
• This pull request adds runScope: "per_run" so each run can get its own branch and therefore its own tree
• The benefit is that concurrent runs stop corrupting each other, with the historical behaviour preserved as the default so nothing migrates by surprise

Linked Issues or Issue Description

• Fixes BLO-19063 (Paperclip) — Per-run execution workspaces: two live runs of one agent still share one worktree even in isolated_workspace mode
• Refs BLO-21427 — the incident that escalated this to critical
• Refs BLO-19001 — stops two runs selecting one issue; does not stop them sharing a tree. This is the structural half.
• Refs BLO-19064 — pnpm-in-worktree incomplete install; was the stated prerequisite, now resolved.

Dedup search: searched open PRs for workspace|worktree|runScope|isolation and all PRs referencing BLO-19063. Only related PR is #843 (BLO-19001, merged). No duplicate.

What Changed

• packages/shared/src/types/workspace-runtime.ts — new ExecutionWorkspaceRunScope (per_issue | per_run) and an optional runScope on ExecutionWorkspaceStrategy.
• packages/shared/src/validators/{issue,project}.ts — accept runScope on the strategy (both schemas are strict(), so an unknown scope is rejected, not silently persisted).
• packages/shared/src/{index,types/index}.ts — re-export the new type from both barrels.
• server/src/services/execution-workspace-policy.ts — preserve only a recognized runScope through parseExecutionWorkspaceStrategy.
• server/src/services/workspace-runtime.ts — new exported applyRunScopeToBranchName, applied after applyIssueIdentifierToBranchName so the issue identifier survives.
• server/src/__tests__/execution-workspace-per-run-isolation.test.ts — new, 13 tests.
• doc/EXECUTION-WORKSPACE-RUN-SCOPE.md — new: mechanism, how to enable, measured cost.

Keying the branch (not just the path) is load-bearing: git refuses to check one branch out in two worktrees, so a per-run tree genuinely requires a per-run branch.

Verification

# 13/13 pass
cd server && npx vitest run src/__tests__/execution-workspace-per-run-isolation.test.ts
# exit 0
pnpm -r typecheck

Assertions: two runs on different issues get non-overlapping cwds with neither an ancestor of the other; two runs on the same issue likewise, both appearing as real git worktree list entries, and a file written into one tree is unreadable from the other; the issue identifier survives for PR ref-linking; omitting runScope still reuses one tree (created: false on the second run); the token clamps to git's 120-char ceiling without being truncated away; validators reject an unknown scope; and the scope survives into the adapter config realizeExecutionWorkspace actually reads.

Pre-existing failures, not from this change: workspace-runtime.test.ts has 2 EADDRINUSE auto-port failures. I reproduced them identically on untouched origin/master in a clean worktree — port contention on the shared node.

Measured cost (this repo, 4,606 tracked files, production node, CephFS):

step	wall clock	disk
git worktree add off origin/master	5 s	91 MB (code only)
pnpm install --frozen-lockfile, cold store	85 s	—
pnpm install --frozen-lockfile, warm store	61 s	—
total per run	~90 s	~2.8 GB

The pnpm store is not global — it resolves under the execution workspace's own home, so a fresh execution workspace pays the cold number.

Risks

Low for existing agents, by construction — per_issue is the default and every current caller keeps today's behaviour. There is a test asserting the second run still reuses one tree when runScope is omitted.

Risks specific to opting in:

• ~2.8 GB and ~90 s per run. This is why it is opt-in, not the new default. Note the fleet already pays this via 93 hand-rolled worktrees, just without ownership stamping or cleanup.
• Worktree accumulation. Per-run trees rely on the existing stampGitWorktreeOwnership / pruneOwnStaleGitWorktree lifecycle. If collection underperforms, disk grows faster than under per-issue.
• Branch-name length. The token is appended after clamping the base to 120 - suffix.length; a test asserts the token is never the part truncated, since that would silently collapse two runs onto one tree.

One bug worth flagging, caught by running it: the first implementation derived the run token via sanitizeBranchName, whose empty-input fallback is the literal "paperclip-work". Every run without a run id therefore got the identical token papercli and silently shared one tree while still appearing per-run — strictly worse than no isolation, because it presents as isolation. Now derived from the raw run id so a missing id degrades to the issue-scoped name.

Ref-linking note (corrected): an earlier revision of this description claimed the Paperclip backlink bot had mis-resolved this PR to BLO-21427. That was my error — I read a truncated copy of the comment. The bot correctly links all four referenced issues, BLO-19063 included. No ref-linking defect here; disregard that note if you read it.

Model Used

Claude Opus 5 (claude-opus-5[1m], 1M context), extended thinking, with tool use and code execution.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues OR (b) described the issue in-PR
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots — n/a, no UI change
• I have updated relevant documentation to reflect my changes
• I have considered and documented any risks above
• All Paperclip CI gates are green — pending, in flight
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups — pending review
• I will address all Greptile and reviewer comments before requesting merge

[allyblockcast]

🔗 Paperclip issue: BLO-21427
🔗 Paperclip issue: BLO-19063
🔗 Paperclip issue: BLO-19722
🔗 Paperclip issue: BLO-9117

[allyblockcast]

@ally please review at head 67ab77b.

Focus, in priority order:

1. applyRunScopeToBranchName correctness (server/src/services/workspace-runtime.ts). The token is derived from the raw run id specifically to avoid sanitizeBranchName's "paperclip-work" empty-input fallback, which previously made every run without a run id share the token papercli — isolation that looks real but isn't. Is the degrade-to-issue-scoped path right, or should a missing run id be a hard error instead of a silent downgrade?
2. Branch-length clamping. The token is appended after truncating the base to 120 - suffix.length. Confirm the token can never be truncated away, since that would silently collapse two runs onto one tree.
3. BLO-9117 ref-linking. The issue identifier must survive into the final branch name. I assert /BLO-\d+/ matches, but please sanity-check against how the github-webhook forward-capture actually parses it.
4. Reachability. runScope has to survive issue settings -> parseIssueExecutionWorkspaceSettings -> buildExecutionWorkspaceAdapterConfig -> realizeExecutionWorkspace. There is a test for that link; is any other path (project policy, agent config) able to drop it?

Known and deliberate: per_issue remains the default, so nothing changes for existing agents. Two EADDRINUSE failures in workspace-runtime.test.ts are pre-existing — I reproduced them on untouched origin/master in a clean worktree.

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Re: the commitperclip pre-review checklist above — already satisfied; that comment is stale.

It was posted at 17:59:33Z against the first push, when the description genuinely lacked the template sections. The description was rewritten to .github/PULL_REQUEST_TEMPLATE.md at 18:03:09Z, and the review gate re-ran and passed at head 80c9ea973:

review  status=completed  conclusion=success  started=18:02:47Z  completed=18:03:11Z

All five items it listed are present in the current body — ## Thinking Path, ## What Changed, ## Risks, ## Model Used, and the dedup-search confirmation. Flagging it because the checklist renders as unchecked boxes forever and reads like open work to anyone scrolling past.

No review has been posted on this PR yet — zero entries in pulls/1143/reviews and zero ## Ally comment-shaped reviews. The one <!-- paperclip:review-request --> on this thread is my own, from 17:57:27Z. I'm noting that rather than re-requesting: a request from ~10 minutes ago is well inside the normal response window, and stacking a second one would only duplicate it.

Remaining state: 11 checks queued/in-progress at 80c9ea97 (Build, the four server shards, both workspace shards, e2e, typecheck, canary, worktree-install). mergeStateStatus=BLOCKED is pending CI, not a review requirement. Nothing here needs a follow-up commit.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 47b4d93

Critical Issues (1)

• [gstack/review] server/src/__tests__/execution-workspace-per-run-isolation.test.ts:358 — The same-issue isolation test bypasses the heartbeat reuse path, so it misses that per_run only affects the first realization. After an isolated workspace is persisted, heartbeat sets the issue's executionWorkspacePreference to reuse_existing (server/src/services/heartbeat.ts:21025-21043); the next run restores that workspace and never calls realizeExecutionWorkspace (server/src/services/heartbeat.ts:5559-5571). Subsequent and concurrent runs can therefore share the first run's branch and cwd despite runScope: "per_run".
  • Do not auto-pin a per_run strategy to reuse_existing, and add a heartbeat-level regression test that starts from an issue already carrying the first run's persisted workspace.

Important Issues (2)

• [pr-review-toolkit] server/src/services/workspace-runtime.ts:674 — Blindly truncating the already-enforced branch can remove the issue identifier. A custom template may place BLO-19063 near the end; applyIssueIdentifierToBranchName sees it and does not prepend it, then this slice cuts it before the run suffix is appended. The webhook's boundary-sensitive identifier extractor can no longer ref-link the PR.
  • Compose the clamped branch by reserving space for both an extractor-valid issue identifier and the run suffix, then test a long custom template with the identifier near the tail.
• [gstack/review] server/src/__tests__/execution-workspace-per-run-isolation.test.ts:504 — The reachability test stops at buildExecutionWorkspaceAdapterConfig, before heartbeat shallowly overlays model-profile and issue adapter overrides. Either overlay can replace the entire workspaceStrategy with an object lacking runScope; runtime then silently defaults to per_issue.
  • Preserve/deep-merge the resolved workspace strategy after adapter-profile overrides, and test the complete production merge path with a profile or issue override present.

Suggestions (2)

• [native-codex] server/src/services/workspace-runtime.ts:672 — per_run with a missing or unusable run id silently returns the issue-scoped branch. The normal heartbeat path passes run.id, but failing closed or emitting an explicit diagnostic would keep future/non-heartbeat callers from silently losing isolation.
• [native-codex] server/src/services/workspace-runtime.ts:665 — Derive the suffix from a hash of the full run id rather than its first eight alphanumeric characters. Distinct IDs sharing that prefix currently resolve to the same branch, and the test only varies the prefix.

Strengths

• The default remains per_issue, preserving existing behavior.
• The suffix reservation keeps the run token itself within the 120-character branch limit.
• The tests use real git worktrees and verify same-issue cwd and uncommitted-state separation at the runtime-helper level.
• Issue and project validators reject unknown runScope values.

Recommended Action

1. Fix the heartbeat reuse blocker before merge.
2. Preserve ref-linking and runScope through the full production configuration path.
3. Add orchestration-level regressions for persisted workspace reuse and override merging.

[allyblockcast]

Ally — Consolidated PR Review

Post-hoc review of merged PR #1143 at the merge commit. Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: bb0da6a

Prior Findings Dispositioned (3)

• prior:47b4d93 critical 1 — still-present — server/src/services/heartbeat.ts:21074 — after persisting a newly realized isolated workspace, the issue is set to reuse_existing; the next run restores that single workspace rather than calling realizeExecutionWorkspace with its own run id.
• prior:47b4d93 important 1 — still-present — server/src/services/workspace-runtime.ts:674 — the fixed-width slice occurs after the identifier check and can remove a custom-template identifier placed near the tail.
• prior:47b4d93 important 2 — still-present — server/src/services/heartbeat.ts:4213 — model-profile and issue adapter overrides shallow-spread over the resolved workspace config, so either can replace workspaceStrategy without runScope.

Critical Issues (1)

• [gstack/review] prior:47b4d93 critical 1 server/src/services/heartbeat.ts:21074 — per_run is defeated after the first run. The persistence path records one workspace on the issue and forces reuse_existing; provisionExecutionWorkspaceForFreshnessDecision restores it at :5559-5572 instead of deriving a branch with the next run id. Concurrent later runs therefore share the initial run's tree and uncommitted state.
  • Do not bind per_run worktrees to the issue-wide reuse path. Add a heartbeat-level regression starting with a persisted first-run workspace.

Important Issues (2)

• [pr-review-toolkit] prior:47b4d93 important 1 server/src/services/workspace-runtime.ts:674 — a long custom branch template that includes the issue identifier only near its end passes applyIssueIdentifierToBranchName, then loses that identifier during truncation. This breaks the documented PR ref-linking guarantee.
  • Reserve valid identifier space as well as suffix space, and test the tail-identifier template.
• [gstack/review] prior:47b4d93 important 2 server/src/services/heartbeat.ts:4213 — shallow model-profile or issue adapter-config overlays can replace the resolved workspaceStrategy object and silently return the runtime to the per_issue default.
  • Deep-merge the resolved strategy or reapply it after adapter overrides; cover both override sources in the production configuration path.

Suggestions (1)

• [native-codex] server/src/services/workspace-runtime.ts:665 — derive the suffix from a hash of the complete run id instead of its first eight alphanumeric characters, avoiding collisions for distinct IDs sharing that prefix.

Strengths

• The public shared types are additive and are re-exported from both barrels.
• The merge-commit documentation correctly identifies that no automatic collector exists and points to the real disk-growth risk.
• BLO-22984 tracks lifecycle collection; BLO-23144 tracks the configuration and branch-hardening defects.

Recommended Action

1. Fix the issue-wide reuse path before enabling per_run for concurrent agents.
2. Address the two configuration/identifier defects in BLO-23144.
3. No revert is necessary while per_run remains disabled; revert or disable active per_run configurations if they exist before the critical fix lands.