Skip to content

Seal workflow secrets across agent and sub-workflow handoffs - #2109

Merged
ppXD merged 1 commit into
mainfrom
fix/carry-secret-taint-across-suspends-agents-and-subworkflows
Oct 12, 2026
Merged

ppXD merged 1 commit into
mainfrom
fix/carry-secret-taint-across-suspends-agents-and-subworkflows

Conversation

@ppXD

@ppXD ppXD commented Oct 12, 2026

Copy link
Copy Markdown
Owner

Summary

  • A workflow secret resolved into an agent.run goal or a flow.subworkflow input was stored in plaintext. It landed in the suspend wait row, agent_run.task_jsonb and the child's workflow_run_request.normalized_payload_json. Viewers read it back as the pending wait, the agent goal and NormalizedPayload. The fix:
    • Every suspend payload is now masked with the node scope's secrets.
    • A task or child payload handed to another run is stored masked, with the exact copy encrypted beside it. Migration 0247 adds two nullable ciphertext columns, and the existing sensitive-payload protector does the encryption.
    • The executor and the child's scope build run from the exact copy, so the platform never runs on a mask.
  • The receiving run now treats what it was handed as secret:
    • The agent run's SecretRedactor folds in the task's secret values, inside the credential resolve. Launch, re-attach and the re-opened MCP endpoint therefore rebuild the same fingerprint.
    • The agent.critic scope gets a CaptureRedactor.
    • A child keeps the inherited values as secrets and taints the trigger/input keys that carry them, on both the fresh and the replay scope path. Its records, map snapshots and Terminal guard then treat them like its own.
    • Supervisor spawns and review children are sealed from the ambient node redactor. The supervisor's work-plan goal is masked.
  • The run's verdicts on that work no longer carry the value back out:
    • The output critic sees the exact goal, and models restate it. Its rationale, issues and critique are masked with the task's secrets before the executor writes the review.completed beat, the flagged warning event and reviewFeedback (AgentRunExecutor.MaskedVerdict).
    • An acceptance argv such as ["sh","-c","… {{team.X}} …"] still runs exactly. Every executor grade request now carries the task's secrets, and SupervisorAcceptanceGrader masks the evidence text, tail, oracle note and detail before storing the evidence artifact a Viewer can download (SupervisorAcceptanceGrader.MaskedGrade).

Deploy notes

  • Drain old workers first. Workers on the previous build do not know the sealed columns. An old executor would launch a sealed agent run from its masked goal. An old engine would run a sealed child with literal [REDACTED] inputs, because it reads only normalized_payload_json. Stop or drain every old engine and agent-executor worker before any new-build worker stages agent runs or sub-workflow children; do not overlap versions.
  • No backfill. Rows written before this change carry no ciphertext and keep executing from their public columns, so in-flight waits, agent runs, child runs, reruns and resumes behave exactly as before. Those rows also keep any plaintext they already hold, and Viewers can still read it: agent_run.task_jsonb of earlier secret-bearing goals, pending wait payloads suspended at deploy, earlier child request payloads, and anything already written to events, records, results and evidence artifacts. A one-off backfill is possible. It is safe for pending wait and child request payloads. task_jsonb of terminal agent runs needs care, because the completion composer re-derives contract hashes from it.
  • Not covered: supervisor-lane acceptance grades (SupervisorTurnService.Rehydrate) do not carry workflow secrets yet. A supervisor acceptanceChecks argv that resolves a secret would still put it into its own grade evidence.

Test plan

  • Unit: full CodeSpace.UnitTests passes (13435 passed, 1 skipped). New coverage:
    • masked verdict sinks (approve and flag) and the pure MaskedVerdict;
    • AcceptanceEvidenceSecretMaskingTests, using the real grader and TestsPassGrader: exact argv executed, stored evidence, tail, note and detail masked, setup-failure prefix and class preserved, no-secret grade byte-identical;
    • every repository lane (branch, patch, multi-repo) passes the task's secrets to the grade;
    • sealer, redactor and taint units.
  • Integration, SecretTaintAcrossSuspendsFlowTests (16 cases). Each case runs an all-table + artifact canary sweep (every text/jsonb/bytea column) and checks every Viewer read. Cases include:
    • agent goal, subworkflow inputs, child Terminal guard (path and value), child map, replay of a sealed child, supervisor plan goal;
    • a pre-sealing agent run still executing;
    • a goal-quoting reviewer (approve and flag);
    • acceptance argv carrying a secret: exact-pass, silent, and echo cases, with the Viewer downloading the evidence;
    • a child resumed through its variable-snapshot replay scope (map and Terminal).
  • Integration regressions: acceptance/oracle (284 passed, 1 skipped); review, critic, revise, subworkflow and agent executor (411 passed, 10 skipped).
  • E2E over HTTP, SecretTaintRunSurfaceE2ETests:
    • an Owner stores secrets and runs agent and child workflows; a Viewer reads every run surface and nothing at rest holds either secret;
    • a reviewed and graded agent run; the Viewer downloads GET /api/artifacts/{evidenceId} and finds only the mask.
  • Mutation checks. Each of these makes its tests fail: removing MaskedVerdict, removing MaskedGrade, dropping the local or repository-lane secret wiring, and dropping the replay-path taint union in WorkflowEngine.BuildScopeForReplayAsync.
  • Deploy: drain old engine and executor workers before the new build stages work.

A secret resolved into an agent.run goal or a flow.subworkflow input
left the scope that knew it was a secret. The engine persisted the
suspend payload, the agent's task_jsonb and the child's request payload
in plaintext, and Viewers read them back as the pending wait, the agent
goal and the child's NormalizedPayload. The agent run's redactor never
knew the value, so every echo froze into its events, native records,
result and the critic's capture. The child built SecretPaths from its
own rows only, so a parent wf/project secret leaked through the child's
ledger, outputs and Terminal guard.

The suspend wait row is now masked with the node scope's secrets for
every wait kind. A task or child payload handed to another run is
sealed: the public column holds the masked copy, and the exact copy plus
the secret values it embeds is encrypted with the existing
sensitive-payload protector (0247 adds the two nullable ciphertext
columns). The executor and the child's scope build open the exact copy,
so the platform never runs on a mask. The executor folds the values into
its redactor inside the credential resolve, so launch, re-attach and the
re-opened MCP endpoint rebuild the same fingerprint. The critic scope
gets a capture redactor. A child keeps the inherited values as secrets
and taints the trigger/input keys that carry them, so its records,
derived scopes, map snapshots and Terminal guard treat them like its
own. Tasks created inside a node scope (supervisor spawns, plan
reviewers) are sealed from the ambient capture redactor, and a review
child inherits its producer's secrets. The supervisor's work-plan goal
is masked the same way.

The run's verdicts on that work carried the value back out. The output
critic is shown the exact goal, and a model restates the task it
judges, so its rationale, issues and critique are masked with the
task's secrets before the executor writes them into the review.completed
beat, the flagged warning and the result's reviewFeedback. An
acceptance argv that resolves a secret runs exactly, but the grade's
evidence repeats the argv and both output tails into a team artifact a
Viewer can download and onto the result. Every executor grade request
now carries the task's secrets, and the grader masks the evidence, its
tail, the oracle note and the detail before anything is stored.

Rows written before this change carry no ciphertext and keep executing
from their public columns, so in-flight waits, agent runs and child
runs resume unchanged, and their redactor fingerprints still match.
Those rows also keep any plaintext they already hold. A worker on the
previous build does not know the sealed columns and would run a sealed
child or agent from its masked public copy, so old engine and executor
workers must be drained before new ones stage work.
@ppXD
ppXD merged commit a9cf96d into main Oct 12, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant