A job's stdout stream holds only the job's own output - #449
Merged
Conversation
A background job's stdout stream was fed two ways: external commands teed each chunk when their pipeline position was Only or Last, and the job's captured result filled the stream at completion only when nothing had streamed. Position alone could not see where output was going, and the completion rule lost everything once any external had written: if true; then echo a; sh -c 'echo b'; echo c; fi & -> "b" if true; then x=$(sh -c 'echo captured'); echo "got $x"; fi & -> "captured" if true; then sh -c 'echo to-file' > f; echo after; fi & -> "to-file" seq 1 2 | scatter | sh -c 'echo worker' | gather & -> "worker\nworker" The routing decision now travels as `background_stream_output`, renamed from `background_stream_external_output` because it governs builtins too. It is off inside `$(...)`, for a stage that redirects stdout, for scatter workers and the stages before `scatter`, and for a compound stage that is not Only or Last, whose inner commands otherwise run as Only. Each dispatch sets it for its own command and restores the kernel's value afterward, so one command's setting does not leak into the next. A builtin publishes its output when it returns, after `--json` is applied, unless the command it re-dispatched already reached the stream (`timeout 5 echo hi &`). gather publishes its rows, which no dispatched command produces. `finalize_streams` no longer writes stdout: every producer publishes as it runs, and a completion write would only repeat or hide a routing hole. stderr keeps its completion rule. Co-Authored-By: DeepSeek V4 Flash <noreply@deepseek.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings in #445, #447, #450, #446, and #448. Two conflicts, both from additions at the same spot: CHANGELOG.md Fixed bullets (all kept, in merge order) and scheduler/pipeline.rs, where #448's `fault_result` and this branch's `redirects_stdout` were each added after `finalize_scatter_gather_error`. Both helpers are kept; the dispatch site merged cleanly with `fault_result(e)` and the stream-flag restore. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A background job's stdout stream (
/v/jobs/N/stdout) was fed two ways. External commands teed each chunk when their pipeline position was Only or Last, and the job's captured result filled the stream at completion only when nothing had streamed. Position alone cannot see where output is going, and the completion rule dropped all builtin output once any external had written:Each now streams only the job's own output:
a b c,got captured,after, and gather's JSONL rows.The routing decision travels as
background_stream_output, renamed frombackground_stream_external_outputbecause it now governs builtins too. It is off inside$(...), for a stage that redirects stdout (>,>>,&>,>&2), for scatter workers and the stages beforescatter, and for a compound stage that is not Only or Last, whose inner commands otherwise run as Only. Each dispatch sets it for its own command and restores the kernel's value afterward.A builtin publishes its output when it returns, after
--jsonis applied, unless a command it re-dispatched already reached the stream, sotimeout 5 echo hi &writes once. gather publishes its rows, which no dispatched command produces.finalize_streamsno longer writes stdout; with every producer publishing as it runs, a completion write would only repeat output or hide a routing hole. The live tee for a runningcargo build &is unchanged.stderr keeps its completion rule: a builtin's stderr still reaches the stream only when no external wrote stderr first.
This stacks on #445.
Co-Authored-By: DeepSeek V4 Flash noreply@deepseek.com
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
🤖 Generated with Claude Code