Skip to content

Keep secrets out of run records at cancels, cuts and flushes - #2110

Merged
ppXD merged 1 commit into
mainfrom
fix/close-node-record-redaction-gaps
Oct 12, 2026
Merged

ppXD merged 1 commit into
mainfrom
fix/close-node-record-redaction-gaps

Conversation

@ppXD

@ppXD ppXD commented Oct 12, 2026

Copy link
Copy Markdown
Owner

Summary

  • Cancelled external call: external_call.failed written on a cancel or shutdown named the raw target, including a {{secret}} query value. Both cancellation branches of TraceExternalCallAsync and TraceExternalStreamAsync now write the redacted target, like every other branch (backend/src/CodeSpace.Core/Services/Workflows/Nodes/NodeObservability.cs:105).
  • agent.run_command: the node now redacts first and cuts second. The inline excerpt and the artifacts come from redacted text, and a budget-cut capture masks a trailing fragment that could start a secret (PersistenceSecretRedactor.RedactCut).
    • The engine hands each node its scope redactor on NodeRunContext.PersistenceRedactor.
    • PersistArtifactAsync masks text and JSON.
    • An uncut capture keeps its runtime value. node.completed masks it, and the real value stays in the encrypted sidecar.
  • Streamed interaction.delta rows:
    • Each 32 KiB or one-second flush releases only text that no secret can still run across, and carries the rest.
    • When the stream ends, the held text is released, so the rows still join up to the masked output.
    • When a cancel or a provider fault cuts the stream, the held text (the possible start of a secret) is masked, and the call's presence delta reports Masked (RecordingStreamingStructuredLLMClientDecorator.cs:94).
  • Agent stdout/stderr lines longer than one 8 MiB read pass: SecretRedactor.CreateFrameStream() carries the end of a cut frame into the frame that continues it. This applies on the live tail, the re-attached tail and the diagnostics drain (AgentRunExecutor.cs:1078,4500,5684). A key the cut lands in is no longer stored as a head in one native record or transcript line and a tail in the next. Both redactors take the settle rule from one shared helper, Services/Redaction/SecretBoundary.cs.
  • Logs: HttpRequestNode.SafeLogTarget logs only the scheme, host and port. ConfigurationPlaceholderGuard reports only the key and its placeholders, never the partly rendered value.

Test plan

  • Unit tests:
    • Cancel and fault while a secret prefix is held: deltas are ["progress ", "[REDACTED]"] and the call is reported Masked.
    • Over-masking guard: a held tail that cannot start a secret stays verbatim.
    • Frame-stream splits over every one- and two-cut position.
    • Size and time flush splits, NodeObservability cancel mid-call and mid-stream, run_command mid-needle truncation, the log target and the placeholder guard.
  • Integration (real engine and Postgres):
    • A run cancelled while its model streams the start of the secret.
    • A cancelled held http.request.
    • A run_command artifact and inline output cut mid-secret.
    • An 8 MiB stdout/stderr cut through the real executor, durable runner and native-record plane.
    • The same cut in a re-attached tail.
  • E2E (real host): a Viewer reads GET /records and the SSE tail after a cancel mid-stream, a cancel mid-http.request, a cut command output, and deltas streamed across a flush. No 6-character fragment of the secret appears.
  • Mutation checks: reverting each fix (cut flush, masking observation, each of the three frame-stream sites, the carry) turns its targeted tests red.
  • Regression:
    • Full unit suite: 13,440 passed.
    • Executor, re-attach, native-record, streaming, ledger, operator-cancel and run_command integration suites: 396 passed.
    • Related E2E suites: 30 passed.

@ppXD
ppXD changed the base branch from fix/carry-secret-taint-across-suspends-agents-and-subworkflows to main October 12, 2026 17:37
Several writes put a team secret, or a fragment of one, where every
member of the team (Viewers included) can read it, or into storage
that redaction is supposed to keep clean.

A cancelled external call wrote its raw target to external_call.failed
(the records endpoint, the SSE tail, the timeline). Every other branch
already wrote the redacted target. Both cancellation branches of
TraceExternalCallAsync and TraceExternalStreamAsync now do the same.

agent.run_command cut its output (the inline cap, or the runner's
capture budget) before the engine's exact-match redaction. A cut
inside a secret left a fragment that no exact match recognises, and
the artifact stored the raw output with no encrypted sidecar. The
engine now hands each node its scope redactor on NodeRunContext. The
node redacts first and cuts second, and masks a trailing fragment of a
budget-cut capture that could start a secret. An uncut capture keeps
its runtime value: the engine still masks it in node.completed, and
downstream nodes, resumes and reruns read the real value from the
sidecar. PersistArtifactAsync masks text and JSON for the same reason.

Streamed interaction.delta rows were redacted one at a time, so a
secret split across a 32 KiB or one-second flush was stored in two
halves and the live view joined them back together. A flush now
releases only text that no secret can still run across, and carries
the rest. When the stream ends, the held text is released, so the rows
still join up to the masked output. When a cancel or a provider fault
cuts the stream, the held text is masked instead: it is exactly the
possible start of a secret whose rest never arrived. The call's
presence delta reports that its capture was masked.

The agent plane had the same gap at its reader's 8 MiB pass. A stdout
or stderr line longer than one pass arrives as a cut frame plus its
continuation, and each was redacted on its own. A key that the cut
landed in was stored as a head in one native record (and transcript
line) and a tail in the next. The executor now carries the end of a
cut frame that could start a secret into the frame that continues it,
on the live tail, the re-attached tail and the diagnostics drain. Both
redactors now take the settle rule from one shared SecretBoundary
helper.

http.request logged URL userinfo before any redaction ran, and the
startup placeholder guard printed the whole value of a partly rendered
setting to stderr. Both now name only what is needed: the scheme, host
and port; the key and its placeholders.
@ppXD
ppXD force-pushed the fix/close-node-record-redaction-gaps branch from 44372ce to 0bf1f84 Compare October 12, 2026 17:37
@ppXD
ppXD merged commit 362bc9c into main Oct 12, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant