fix(env-guard): stop truncating a fleet-shared file in place

[kkroo]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work
• The claude_k8s adapter builds the shell fragment each agent pod runs at startup, which installs the PreToolUse env-guard hook into the agent HOME
• That HOME is a ReadWriteMany CephFS volume mounted by every agent pod, and the install used > "$GUARD_DIR/paperclip-env-guard.mjs" — a plain redirect, which is O_TRUNC
• So the entire fleet issued setattr(size=0) against one shared inode on every pod start; that is a fleet-wide write amplification onto a single metadata object
• One of those truncates wedged in the MDS (truncate_pending=1 with nothing driving it), every later truncate queued behind it permanently, and open() began blocking in D-state — hanging the Bash PreToolUse hook fleet-wide
• This pull request removes the in-place truncate entirely and content-addresses the guard filename
• The benefit is that a given inode is written exactly once and truncated never, so a single wedged metadata object can no longer take out the fleet — and an already-poisoned inode is routed around rather than waited on

Linked Issues or Issue Description

No pre-existing issue — describing inline per CONTRIBUTING.md path (B), bug report shape.

What happened. Agent pods hung fleet-wide on their Bash PreToolUse hook. On the MDS: 68+ requests stuck at flag point acquired locks, oldest ~4h, from 8 distinct nodes, all setattr size=0 against the single inode backing .claude/paperclip-env-guard.mjs on the shared paperclip-data volume. Hung umount/sync on two nodes, which is what blocked draining them.

Expected. Installing a hook file at pod startup does not contend with every other pod in the fleet.

Actual. Every pod truncated the same inode. One stuck truncate blocked all subsequent ones permanently.

Not a capacity or client problem. MDS journal, objecter and OSDs were healthy throughout (journal latency 12ms, op_laggy 0). The client named by the MDS_CLIENT_OLDEST_TID warning was itself healthy — releasing caps ~110k/min with recall_caps 0 — so the warning misdirects; the fault is on the inode, not the client. Dropping that client's cache halved its caps and changed nothing.

Three online remedies do not clear it (all verified against the live cluster):

Remedy	Result
ceph mds fail <rank>	Standby replays the identical stuck state
scoped scrub start <path> recursive,repair,force	Validates backtrace/dirstat, not truncate state
rename a healthy file over the dentry	Blocks — unlinking the target needs the locks the stuck truncate holds

That third result is why the filename has to change: the poisoned dentry can be neither replaced nor removed, so recovery has to route around it.

What Changed

• buildEnvGuardSetupShell no longer truncates. A new installOnce helper writes each script to a pod-unique temp (.<name>.$$.tmp) and mv -fs it into place, guarded by [ -f "$target" ] || so the write happens only when the target is absent. test -f stats the dentry and never opens it, so the check itself cannot block on a poisoned inode.
• The guard filename is content-addressed: paperclip-env-guard.<sha256[:12]>.mjs. A guard change lands as a new file instead of rewriting the shared one.
• The setup shell exports PAPERCLIP_GUARD_FILE so the settings merge targets the same content-addressed path.
• SETTINGS_MERGE_SCRIPT now writes settings.json via temp + renameSync instead of an in-place writeFileSync — the identical shared-volume hazard, one landmine away from the same outage — and prunes stale PreToolUse entries pointing at a previous guard file, so pods stop invoking a wedged path.
• safe-env-inspect.mjs keeps its stable name (ENV_GUARD_SCRIPT names it literally in the block message) but is install-once too, so it is never truncated either.
• PROVENANCE.md: refreshed the vendored integrity hash and logged the patch in the Local modifications table.

Verification

cd vendor/paperclip-adapter-claude-k8s && npm test

• Positive control: env-guard.test.ts 279 pass / 0 fail. Full vendored suite in CI: 889 pass across 16 files.
• Negative control: reverting env-guard.ts to its previous implementation and re-running the new tests fails exactly 3 — never truncates a shared file in place, content-addresses the guard filename so a change lands as a new inode, and the updated merge test. The tests fail for the right reason rather than passing vacuously.
• Generated-shell check: the emitted fragment was rendered and passed sh -n.
• Provenance gate: recomputed manifest hash matches the recorded value.

New coverage asserts every redirect targets a .tmp path, every install is create-if-absent landing via rename, the guard name is content-addressed, and a stale guard hook entry is pruned rather than merely deduplicated.

Risks

• Behavioral shift, intended: the guard file's name changes. Pods pick up the new path from settings.json, which the merge script rewrites on startup, and the stale entry is pruned in the same pass.
• Install-once means content changes need a name change. For the guard that is automatic (content-addressed). For safe-env-inspect.mjs, whose name is deliberately stable, a future content change would not propagate to volumes that already have the file. That helper is two lines and has never changed; flagged here rather than left implicit, and called out for review below.
• No migration, no schema change, no API surface change. Failure mode remains fail-open: the merge step still || echos and cannot block a run from starting.
• Does not by itself un-wedge the currently poisoned inode — nothing can, short of offline CephFS repair. It stops new truncates being added, after which the existing blocked ops drain as the pods holding them churn out.

Model Used

Claude Opus 5 (Anthropic), model ID claude-opus-5, 1M context window, extended thinking, with tool use and code execution. Diagnosis ran against the live Ceph cluster and Kubernetes API; the fix and its tests were authored and executed locally.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above — searched open PRs for env-guard|envguard|vendor|truncate|ceph; the only hits are keyword noise on "guard" (fix(heartbeat): retry k8s_concurrency_guard_unreachable runs (BLO-17938) #1803, test(recovery): cover the backstop shared-cursor guard with interleaved invocations #1743, fix(heartbeat): withhold a row a sibling run is parked on via scheduled retry (BLO-29965) #1742, fix(ally-guard): classify a same-lane duplicate review by body, not by timing #1738, fix(redaction): mask five vendor credential shapes on the run-log free-text path (PEN-3139) #1736, fix(heartbeat): stop coalescing from inheriting the status-only guard onto a declared normal-model wake (BLO-32634) #1718, test(deploy): pin the two widest citation ranges' interiors, not just their first lines (BLO-32210) #1686, fix(review-gate-sweep): re-check ally_has_reviewed_head before the write (BLO-32044) #1667, fix(review-gate-sweep): re-read markers and re-check ally_has_reviewed_head before the write (BLO-31908, BLO-32044) #1661), none of which touch the vendored adapter. No duplicates.
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots — N/A, no UI surface
• I have updated relevant documentation to reflect my changes
• I have considered and documented any risks above
• All Paperclip CI gates are green — in progress at time of writing
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups — not yet reported
• I will address all Greptile and reviewer comments before requesting merge

🤖 Generated with Claude Code

[allyblockcast]

Hey @kkroo! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• No linked issue or inline issue description found — either tag an existing issue with Fixes #NNN / Closes #NNN / Refs #NNN, or describe the underlying issue inline in the PR body following one of our issue templates (https://github.com/paperclipai/paperclip/tree/master/.github/ISSUE_TEMPLATE). See CONTRIBUTING.md → "Link Issues or Describe Them In-PR".
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[kkroo]

@ally please review this head: b2e9946

Root cause: the claude-k8s adapter installed its hook with > "$GUARD_DIR/paperclip-env-guard.mjs", which is O_TRUNC on a ReadWriteMany CephFS volume mounted by every agent pod, so the whole fleet truncated one shared inode. One truncate wedged in the MDS and every later one queued behind it permanently, hanging the Bash PreToolUse hook fleet-wide.

Fix: never truncate a shared file (create-if-absent, pod-unique temp + rename), and content-address the guard filename so a change lands as a new inode instead of rewriting the shared one. settings.json had the same latent hazard and now writes temp + renameSync, plus prunes stale guard hook entries.

Please pay particular attention to:

• the installOnce shell fragment quoting/escaping (it is emitted into sh -c),
• the rewritten SETTINGS_MERGE_SCRIPT pruning regex, that it cannot drop a non-guard hook,
• whether the stable-named safe-env-inspect.mjs being install-once means a future content change to that helper would never propagate.

[allyblockcast]

Hey @kkroo! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• No linked issue or inline issue description found — either tag an existing issue with Fixes #NNN / Closes #NNN / Refs #NNN, or describe the underlying issue inline in the PR body following one of our issue templates (https://github.com/paperclipai/paperclip/tree/master/.github/ISSUE_TEMPLATE). See CONTRIBUTING.md → "Link Issues or Describe Them In-PR".
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[kkroo]

@ally the PR body blocker you flagged is resolved — all six required sections (Thinking Path, Linked Issues, What Changed, Verification, Risks, Model Used) and the dedup checkbox are now present. Please review this head: 6633ce9

Since your last pass the branch also gained a second commit refreshing the vendored PROVENANCE.md integrity hash, which the vendored-adapter gate requires after any edit under vendor/paperclip-adapter-claude-k8s/.

Specific things I would like scrutinised:

• installOnce quoting and escaping — it is emitted into sh -c, and $$ must expand in the pod, not at build time.
• The SETTINGS_MERGE_SCRIPT prune regex, specifically that it cannot drop a non-guard PreToolUse hook.
• The stable-named safe-env-inspect.mjs being install-once: a future content change to that helper would not propagate to volumes that already have it. Documented under Risks — tell me if you would rather it were content-addressed too.

[github-actions]

@ally head 6633ce9 has been awaiting review for 2.1h with no review on either surface (pulls/1810/reviews carries no consolidated report for this head, no ## Ally comment either) -- automated sweep (BLO-22892 / BLO-28203), not a human/agent re-ask.

Requested a review from @allyblockcast directly (native GitHub review request, not just this comment) against current head 6633ce9.

[github-actions]

@ally head 23f0a44 has been awaiting review for 1.5h with no review on either surface (pulls/1810/reviews carries no consolidated report for this head, no ## Ally comment either) -- automated sweep (BLO-22892 / BLO-28203), not a human/agent re-ask.

Requested a review from @allyblockcast directly (native GitHub review request, not just this comment) against current head 23f0a44.

[github-actions]

@ally head 23f0a44 has been awaiting review for 4.6h with no review on either surface (pulls/1810/reviews carries no consolidated report for this head, no ## Ally comment either) -- automated sweep (BLO-22892 / BLO-28203), not a human/agent re-ask.

Requested a review from @allyblockcast directly (native GitHub review request, not just this comment) against current head 23f0a44.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 23f0a44

Critical Issues (1)

• [native-codex] vendor/paperclip-adapter-claude-k8s/src/server/env-guard.ts:926 — the supposedly pod-unique install temp is only keyed by $$; the settings merge temp at line 888 has the same problem with process.pid. Pods have separate PID namespaces, so concurrent startups on the shared HOME can generate the same temp pathname. Each writer then truncates that shared temp with >, and the writers race on mv, recreating the shared-file metadata contention and potentially losing one writer's settings update.
  • Use a genuinely shared-volume-unique temp created atomically (for example mktemp in the target directory, or an equivalent exclusive-create flow), and add a test that runs two independent installers concurrently against one shared directory and verifies both complete without clobbering the result.

Important Issues (0)

Suggestions (1)

• [pr-review-toolkit] vendor/paperclip-adapter-claude-k8s/src/server/env-guard.test.ts:347 — test the generated shell concurrently from separate processes and assert the temp paths are distinct; the current regex test locks in $$ rather than proving cross-pod uniqueness.

Strengths

• The content-addressed guard filename routes around the known poisoned inode.
• The settings merge preserves unrelated hooks and removes stale guard entries.
• The vendored integrity hash and focused regression coverage were updated.

Recommended Action

1. Fix the Critical issue before merge.
2. Address Suggestions opportunistically.