Conversation
…i runs 710cd72 fixed this shell injection in fleet_agent_v3.rail. It did not touch fleet_agent.rail, the v2 agent, which has the identical flaw: /logs/<svc> interpolates svc straight into a shell command, and /jobs/<id> interpolates id three times. v2 is what runs on the Pi, because v3 cannot be built there at all (cross-compile duplicate symbols plus 416 MB of RAM). So the node that could not take the fix was the one left holding the bug, and it stayed that way for two months while the fix sat unpushed on a branch. Same allowlist as v3: letters, digits, '.', '_', '-', which excludes every shell metacharacter, whitespace and '/'. A denylist would miss '*', '?', '~', quotes and '='. Verified by asking each node before and after. Every one of mini, studio, air and pi answered 200 to /logs/a;true beforehand; all four now answer 400, and a legitimate service name still answers 200. The v3 binary was rebuilt and deployed to the three Macs and this v2 build to the Pi, each with the previous binary kept alongside. Worth recording how the "is it vulnerable" question was first answered WRONGLY: by grepping the compiled binary for the allowlist literal. Rail does not store string literals in a greppable form, and a control search proved it could not find "forbidden" or "X-Fleet-Token" either. The detection method was broken, not the binary. Only a behavioural probe settled it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EzjMXh1wxgasCH81MJfTCk
The Pi's fleet control plane accepted a wrong token and no token alike. Two
independent faults had to line up, and a third hid the result.
1. Rail's Linux runtime does not propagate envp. Measured with a probe:
shell() on the Pi sees ONE environment variable and no HOME, while the
same probe on macOS sees 43. The macOS prologue takes envp from x2, the
main(argc,argv,envp) convention, which a static Linux binary with its own
_start never receives. So `~` does not expand and load_token returned "".
2. v2's check_auth read `if length token == 0 then true`: fail OPEN. v3 was
hardened to deny in 519a18b; v2 never was, and v2 is what the Pi runs.
3. Empty token plus fail-open meant every request was allowed, which is
also why nobody noticed that the Pi's token had drifted away from the
fleet's during a rotation. mini, studio and air all share ad640cd6…; the
Pi alone held 02538e41… . Fail-open made the drift invisible.
load_token now falls back to the passwd home directory when `~` yields
nothing. `~` stays first because it works on macOS and never reaches the
fallback there, and getent does not exist on macOS. No ${...} braces: Rail
treats { } inside a string as interpolation, which mangled a first attempt.
check_auth now fails closed. Both changes are required together; either
alone leaves the plane open or shut.
Verified on all four nodes: correct token 200, wrong token 403, no token
403, the old drifted token 403, and the injection guard still 400.
Three false conclusions on the way, all from testing the wrong thing:
grepping a Rail binary for a string literal (they are not stored greppably,
proven by a control search); verifying the shell form under `env -i /bin/sh`
rather than under Rail's own shell(); and finally reading 403 as "the fix is
broken" when the Pi was correctly rejecting the MINI's token, which is what
the request bytes showed once I looked at them instead of inferring. The fix
had been right for two deploys before I could see it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EzjMXh1wxgasCH81MJfTCk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes to
fleet_agent.rail, the v2 agent. v3 got both of these; v2never did, and v2 is what runs on the Pi, because v3 cannot be built there
at all (cross-compile duplicate symbols plus 416 MB of RAM). The node that
could not take the fixes was the one left holding the bugs.
1. Shell injection in
/logs/<svc>and/jobs/<id>710cd72fixed this infleet_agent_v3.railand did not touch this file,which has the identical flaw:
svcis interpolated straight into a shellcommand, and
idthree times over.Same allowlist as v3 (letters, digits,
.,_,-), which excludes everyshell metacharacter, whitespace and
/. A denylist would miss*,?,~, quotes and=.2. Auth failed OPEN, and the Pi's token had drifted
Three faults had to line up, and the third was invisible until the first two
were fixed.
shell()had no HOME and~would not expand under dash, and
load_tokenreturned empty. That isfixed at the root in fix(linux): _start never passed envp, and getenv could not even link #60; this branch also makes
load_tokenfall back tothe passwd home directory so it works regardless.
check_authreadif length token == 0 then true: fail OPEN. v3 washardened to deny in
519a18b. Now fails closed.token; the Pi alone held a different one after a rotation missed it.
Fail-open meant the Pi accepted the wrong token anyway, so the drift was
invisible for exactly as long as the bug existed. Synced separately.
The two code changes are only safe together: fail-closed alone denies every
request, which is what happened when it was tried first and took the Pi's
control plane down until it was rolled back.
Verification
Asked all four nodes, before and after:
/logs/logs/a;trueDeployed to mini, studio, air and pi, previous binaries kept alongside.
Note on method
"Is it patched" was first answered wrongly by grepping the compiled binary
for the allowlist literal. Rail does not store string literals greppably; a
control search could not find
forbiddenorX-Fleet-Tokeneither. Only abehavioural probe settles it.