Skip to content

fix(claude-k8s): allowlist what agent pods inherit from the server pod (BLO-22514) - #1377

Merged
allyblockcast[bot] merged 2 commits into
masterfrom
cto/blo-22514-agent-env-allowlist
Aug 16, 2026
Merged

fix(claude-k8s): allowlist what agent pods inherit from the server pod (BLO-22514)#1377
allyblockcast[bot] merged 2 commits into
masterfrom
cto/blo-22514-agent-env-allowlist

Conversation

@allyblockcast

@allyblockcast allyblockcast Bot commented Aug 15, 2026

Copy link
Copy Markdown

Thinking Path

  • Paperclip is the open source app people use to manage AI agents for work
  • The claude_k8s adapter renders every production agent Job pod, building each one from the paperclip server pod's own spec via getSelfPodInfo()
  • That introspection snapshotted the server's entire environment — every literal env[].value, every valueFrom including secretKeyRef, every envFrom source, and every mounted secret volume — and job-manifest.ts replayed all of it onto every agent Job
  • There was no allowlist, denylist, or filter anywhere on that path; the only skips were empty values and unnamed entries
  • So every agent container held PAPERCLIP_AGENT_JWT_SECRET, DATABASE_URL and GITHUB_APP_PRIVATE_KEY — one agent could mint an API key as any other agent, or bypass the API and authorization.ts entirely
  • The readers are our own agents in our own cluster, so this is not an external breach; the live path is prompt injection, since agents routinely ingest attacker-influenceable text (PR bodies, issue comments, fetched pages), and one payload turns a content bug into control-plane compromise
  • This pull request replaces that wholesale inheritance with an explicit allowlist at the single chokepoint, plus a fail-closed backstop on the built manifest
  • The benefit is that an agent pod now receives only what an agent actually needs, and reintroducing the leak requires a deliberate, reviewable edit

Linked Issues or Issue Description

What Changed

  • New src/server/inherit-allowlist.ts — the policy, with every entry justified inline: SERVER_ONLY_ENV_DENY, AGENT_ENV_ALLOWLIST, AGENT_ENV_ALLOWED_PREFIXES, AGENT_SECRET_VOLUME_ALLOWLIST, AGENT_ENV_FROM_ALLOWLIST.
  • Filtered at getSelfPodInfo() (k8s-client.ts), not at the four replay sites. job-manifest.ts replays inherited state in four places (:599 literal spread, :679 secretKeyRef, :1053 secret volumes, :1470 envFrom). Filtering at the source means a fifth replay site added later cannot reintroduce the leak by forgetting to filter.
  • secretKeyRef entries are filtered on the same basis as literals. BLO-22506's analysis called these "fine" because they do not surface through GET Pod. They are not fine — the kubelet resolves them into the container environment, so the agent reads them either way. That distinction is the whole reason this issue is separate from BLO-22506.
  • Secret volumes allowlisted by secretName (not volume name, which is a chart-local label). The three agent-facing mounts are kept — values.blockcast.yaml documents that propagation as intentional and agents genuinely need them for gh/git/gbrain.
  • envFrom denied by default. It injects whole objects under names a per-name allowlist never observes, so it cannot be reconciled with one. No envFrom exists anywhere in deploy/helm/paperclip, so this is a no-op against the current deployment that closes the path pre-emptively.
  • Fail-closed backstop findServerOnlyEnvVarsInPodSpec() throwing in buildJobManifest, mirroring the existing findLiteralSensitiveEnvVarsInPodSpec idiom. SelfPodInfo is a plain object that callers and tests construct directly, so the type system never guarantees its contents came through the filter.
  • Corrected a false claim in packages/plugins/sandbox-providers/kubernetes/src/sensitive-env-guard.ts, which asserted it "mirrors the same protection in the external claude_k8s adapter". Wrong twice: the adapter is no longer external, and what it has is the /TOKEN|SECRET|…/i denylist that file's own POLICY note explains cannot hold the invariant.
  • Recorded the env-guard.ts fail-open decision in the file (comment-only). See Risks.
  • PROVENANCE.md integrity hash + file count updated, as CI's vendor_claude_k8s job requires.

Deriving the keep-set

Dropping a needed key breaks every agent run in the fleet, so the keep-set is derived from code rather than pattern-matched — and by-name reads alone are not sufficient. Only two names (CLAUDE_CONFIG_DIR, PAPERCLIP_API_URL) are read off inheritedEnv in this adapter; the rest are consumed downstream by Claude Code, the gh wrapper and the MCP bridges, so they never appear as a read here. A sweep of agent-pod consumers found PAPERCLIP_PUBLIC_URL (read by packages/mcp-server/src/paperclip-links.ts, absent from this adapter entirely) — a genuine gap an adapter-only derivation would have shipped.

Two deliberate calls worth reviewer attention:

  • XDG_CONFIG_HOME is excluded. The server's value is /runtime-config, an emptyDir the Job pod never mounts, and the adapter only sets it under isolation. It is load-bearing for git config --global resolution, so inheriting a non-existent path is worse than letting it fall back to $HOME.
  • PAPERCLIP_MASTER_KEY_SEED was added to the deny-set though BLO-22514 does not name it. Found while enumerating the server env; it is the secret-encryption master seed and belongs in the same class as the JWT key.

Retained conservatively and flagged in-code as tightening candidates rather than dropped blind: PAPERCLIP_GBRAIN_* and PAPERCLIP_CODEX_*/PAPERCLIP_OPENCODE_* (no agent-pod reader found, but they are a URL and routing config rather than credentials), and NODE_OPTIONS (status-quo preservation; re-tuning a 6144 MiB heap ceiling for an 8 Gi Job pod is a separate change, not one to bundle into a security fix).

Verification

cd vendor/paperclip-adapter-claude-k8s
npm ci --include=dev
npx --no-install tsc --noEmit    # clean
npm test                          # 788 passed (14 files) — was 726 before, 62 new

Provenance gate, as CI computes it:

git ls-files | grep -vxE 'LICENSE|PROVENANCE\.md' | LC_ALL=C sort | xargs sha256sum | sha256sum
# c10cf8e3c766c22196713d256767164a5ffd321a1ae63c026ffdfffa39f417c7  — matches PROVENANCE.md

Exercised against the real server env, not just fixtures — all 54 vars from values.blockcast.yaml + deployment-api.yaml pushed through the predicate:

count
server env vars in 54
inherited by agent pods 24
dropped 30
control-plane credentials still reaching agents 0

Dropped includes PAPERCLIP_AGENT_JWT_SECRET, DATABASE_URL, GITHUB_APP_PRIVATE_KEY, GITHUB_APP_ID, GITHUB_APP_INSTALLATION_ID, GITHUB_WEBHOOK_SECRET, PAPERCLIP_DEX_OIDC_CLIENT_SECRET, PAPERCLIP_ALERTMANAGER_WEBHOOK_TOKEN, PAPERCLIP_MASTER_KEY_SEED. Kept includes PAPERCLIP_API_URL, PAPERCLIP_PUBLIC_URL, PATH, PAPERCLIP_GITHUB_TOKEN_FILE, and the ANTHROPIC_* / OPENAI_* provider families. Secret volumes: the three agent-facing mounts kept, paperclip-db-credentials dropped.

Only one pre-existing test needed changing: a cache-keying fixture asserting on a synthetic CLUSTER_TAG. It was retargeted to an allowlisted var so it still proves cache isolation — no test asserting real agent behaviour broke, which is itself evidence the keep-set is right.

A green CI run does not close this issue. BLO-22514's verifying signal requires confirming against a freshly-scheduled agent pod after rollout — a passing unit test does not prove the running adapter image carries the filter. I will run the length-only probe post-deploy and post it on the issue.

Risks

  • Highest risk is over-tight filtering, not under-tight: a dropped key that some agent silently depends on degrades the fleet on the next schedule. Mitigated by deriving the keep-set from code plus a consumer sweep, testing both directions, and keeping provider config as prefix families (ANTHROPIC_, OPENAI_, AWS_, GOOGLE_, VERTEX_, AZURE_OPENAI_, CLAUDE_CODE_) so Bedrock/Vertex deployments routing through names this repo never mentions keep working. Those namespaces are owned by external SDKs, so no Paperclip control-plane secret can fall inside one — and the deny-set is checked first regardless.
  • Behavioural change: agent pods stop receiving XDG_CONFIG_HOME and ~29 other server-only vars. XDG_CONFIG_HOME is called out above as a fix rather than a regression.
  • No operator/config escape hatch, deliberately. Per-agent assigneeAdapterOverrides.adapterConfig is writable through the Paperclip API, so a config-extensible allowlist would let an agent re-admit DATABASE_URL for itself — reintroducing the exact escalation this closes. Widening requires a code change and a review.
  • env-guard.ts stays fail-open — decision recorded in-file rather than only on the issue. BLO-22514 asked because it fails open on any error and did not fire against a live printenv "$V". Both true. It stays open because (a) it runs in the agent's own runtime and failing closed wedges every run on any parse error, (b) it cannot be made complete anyway — X=env; $X and cat /proc/$$/environ resolve at runtime and no static pass over command text can classify them, and (c) this PR removes what it was standing in front of: a successful dump now discloses agent-scoped provider credentials the agent already uses, not the control plane. Paying an availability cost for it is a worse trade after this change than before.
  • Credential rotation is out of scope and strictly post-fix. These values were readable by every agent for as long as the adapter has existed; rotating before the filter ships would re-expose the new values immediately. Containment on this issue has been names-and-lengths only throughout — no value has been read, logged, or transmitted.
  • Low migration risk: no schema change, no API change, no new dependency.

Model Used

  • Claude Opus 4.5 (claude-opus-4-5, 1M context), extended thinking, agentic tool use via Claude Code / Paperclip claude_k8s adapter.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
  • I have run tests locally and they pass (788/788, typecheck clean, provenance hash matches)
  • I have added or updated tests where applicable
  • If this change affects the UI, I have included before/after screenshots — N/A, no UI surface
  • I have updated relevant documentation to reflect my changes (PROVENANCE.md + the corrected sensitive-env-guard.ts header)
  • I have considered and documented any risks above
  • All Paperclip CI gates are green — pending; checks were queued when this was opened
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups — pending first pass
  • I will address all Greptile and reviewer comments before requesting merge

…LO-22514)

Agent Job pods were built by copying the paperclip server pod's entire
environment. getSelfPodInfo() snapshotted every literal env value, every
valueFrom (including secretKeyRef), every envFrom source and every mounted
secret volume, and job-manifest.ts replayed all of it onto every agent Job
with no allowlist, denylist or filter anywhere on the path.

So each agent container held PAPERCLIP_AGENT_JWT_SECRET (mint an API key
for any agent, any scope), DATABASE_URL (bypass the API and all of
authorization.ts) and GITHUB_APP_PRIVATE_KEY (mint installation tokens
org-wide). The readers are our own agents, so the live path is prompt
injection: agents routinely ingest attacker-influenceable text, and a
payload that induces one agent to read and post these turns a content bug
into control-plane compromise.

Filter at getSelfPodInfo() rather than at the four replay sites, so a
future replay site cannot reintroduce the leak by forgetting to filter,
with a fail-closed findServerOnlyEnvVarsInPodSpec backstop in
buildJobManifest because SelfPodInfo is a plain object callers construct
directly.

secretKeyRef entries are filtered on the same basis as literals. An
earlier analysis called them "fine" because they do not surface through
GET Pod; they are not — the kubelet resolves them into the container
environment either way.

The keep-set is derived from the adapter's own by-name reads plus a sweep
of agent-pod consumers, not pattern-matched, and both directions are
tested: a filter that dropped everything would pass a deny-only suite
while breaking every run in the fleet. PAPERCLIP_PUBLIC_URL is in it only
because of that sweep — it is read by packages/mcp-server and appears
nowhere in this adapter.

Measured against the live server env: 54 vars in, 24 inherited, 30
dropped, 0 control-plane credentials remaining.

Also corrects the claim in sensitive-env-guard.ts that it "mirrors the
same protection" in this adapter. It did not: that guard is the
/TOKEN|SECRET|.../i denylist its own POLICY note explains cannot hold the
invariant. And records the decision to keep env-guard.ts fail-OPEN, since
this change removes what it was standing in front of.

Refs BLO-22506. Closes BLO-22514.

Co-Authored-By: Claude <noreply@anthropic.com>
@allyblockcast

allyblockcast Bot commented Aug 15, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-22514
🔗 Paperclip issue: BLO-22506

The row was written before the PR existed and guessed #1381.

Co-Authored-By: Claude <noreply@anthropic.com>
@allyblockcast

allyblockcast Bot commented Aug 15, 2026

Copy link
Copy Markdown
Author

@ally please review at head a0c2224 — security fix for BLO-22514 (agent pods inherited the paperclip server's entire secret env).

Focus, in priority order:

  1. Is the keep-set complete? This is the real risk — a missing key breaks every agent run in the fleet on the next schedule, not just this PR. src/server/inherit-allowlist.ts derives it from the adapter's by-name reads plus a sweep of agent-pod consumers. Only CLAUDE_CONFIG_DIR and PAPERCLIP_API_URL are read in this adapter; everything else is consumed downstream by Claude Code, the gh wrapper, or packages/mcp-server, so an adapter-only derivation would be insufficient (that sweep is how PAPERCLIP_PUBLIC_URL was caught). Please look for names I still missed — especially ones read from shell wrappers, Dockerfile ENV, or dynamic lookups.
  2. Prefix families (ANTHROPIC_, OPENAI_, AWS_, GOOGLE_, VERTEX_, AZURE_OPENAI_, CLAUDE_CODE_). They exist so Bedrock/Vertex deployments routing through names this repo never mentions keep working. AWS_ is genuinely broad — is that the right trade against pinning exact names and silently breaking those deployments?
  3. Chokepoint choice. I filtered in getSelfPodInfo() rather than at the four replay sites (job-manifest.ts :599/:679/:1053/:1470), so a fifth replay site cannot reintroduce the leak. Backstopped fail-closed by findServerOnlyEnvVarsInPodSpec() in buildJobManifest, since SelfPodInfo is a plain object callers construct directly. Is that split right?
  4. Two deliberate behavioural changes worth a second opinion: XDG_CONFIG_HOME is now dropped (the server value /runtime-config is an emptyDir the Job pod never mounts, and it is load-bearing for git config --global), and envFrom is denied wholesale (none exists in deploy/helm/paperclip today).
  5. No config escape hatch, deliberatelyassigneeAdapterOverrides.adapterConfig is agent-writable, so an operator-extensible allowlist would let an agent re-admit DATABASE_URL for itself. Please sanity-check that reasoning.

Not asking you to re-litigate: credential rotation is strictly post-fix and out of scope, and the env-guard.ts change is comment-only (records the decision to keep it fail-open, with reasons in the PR body).

@allyblockcast
allyblockcast Bot added this pull request to the merge queue Aug 16, 2026
Merged via the queue into master with commit 07dba79 Aug 16, 2026
35 of 37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants