fix(system): bind the two sibling trash removals to descriptors - #7491
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Real TOCTOU hole on a named boundary (same-uid agent vs. trash), closed with the mechanism the module already owns, residuals disclosed and tracked. Suggestions
[DESIGN-REVIEWED] 87348df |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of All mechanical checks are done. Consumer counts, sibling counts, and the existing-mechanism check all verified against the repo at HEAD. Final review: First-Principles-Verdict: PASS Both siblings #7011 named are fixed at cause level — removal bound to a pre-captured inode instead of a re-resolved name — with every residual declared and test-pinned. What this change shipsIntent: stop a restore or rolled-back move from deleting outside the trash through a swapped ancestor — a FIX.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 87348df |
GPT 5.6 Review — ✅ human override acceptedHuman judgment by @chenmingwei23 overrides the GPT 5.6 finding for This comment is updated in place on each push. The model was not re-run because an authorized human decision supersedes it. False positive or not applicable? A repository writer can comment: |
845a623 to
32d6386
Compare
|
Every item actioned on head Blocker, fixed. The duplicated Worth recording why that matters more than a stray log line: the function it landed in is the one whose whole job is to not remove by path, and the splice had it doing exactly that. Reading it as "adjacent identical calls, delete one" was the right read of the diff. Watch item, fixed rather than noted. All three subtractions taken.
Item 7 in your ship list was the duplicate; items 8-10 are the subtractions. Nothing on that list is left undeclared. Targeted suites green on the new head: The PR Hygiene red was unrelated to the code: the gate wants a literal |
32d6386 to
75afed2
Compare
|
Both blocking findings were real and both were introduced by this diff. Fixed on head 1. The manifest was removed before the tree, so a survivor left the batch unlistable. Correct, and it is a regression against a property the delete path already had.
The fix generalises the property rather than special-casing the manifest: The concept is not "manifest" - it is "the entry that makes this tree discoverable" - which is why it is a parameter of the primitive and not a branch in the caller. 2. The approval read the manifest twice, so a replacement between reads could forge the listing. Also correct, and the worse of the two: the header check came from Now one call. Tests, mutation-verified. Four new cases, and reverting each fix turns them red:
Removing Targeted suites on the new head: One follow-up I am deliberately not doing here: |
75afed2 to
b4349d7
Compare
|
CONCERNS dispositioned on head Taken, in the half that mattered. The two spellings were not equivalent. The delete path's put-back is So the recovery half now has one owner: Two tests pin it: the copy fallback with Deferred, with the reason stated rather than implied. The remaining duplication is the orchestration around that recovery, and replacing On Targeted suites on the new head: 301 passing across |
b4349d7 to
4cf5592
Compare
ca5d9de to
5b707cc
Compare
Round on
|
| position | why it was tried | why it fails |
|---|---|---|
AFTER the cleanup, gated on path_verified |
keeps the batch honest once it is kept | the descriptor is closed by then, so a swap redirects atomic_write (blocked last round) |
| BEFORE the cleanup, unconditional | moves the write to where the partial branch already writes | this branch had NO write before, so it ADDS one; a batch swapped to a link sends it to another batch's manifest, whose sessions become unlistable |
So the tidy-up is not done at all, which is the remedy this round asked for. The conditional is
restored: an empty remaining goes straight to the descriptor-bound cleanup, and neither the
cleanup nor the full-restore branch writes to the batch. The partial branch keeps its
pre-existing rewrite -- it has to record what is still staged, and it is not reached through a
refusal.
The cost is a declared residual, now stated in the code and the spec rather than left implied:
a batch the cleanup declines to remove still lists the sessions the restore moved out. That
is untidy and it is visible and reversible, which the write is not. empty_trash still takes
such a batch -- I did not assert that from reading, the test performs the empty and checks the
batch is gone.
test_a_batch_the_cleanup_keeps_lists_no_phantom_sessions is replaced by
test_a_full_restore_never_writes_to_the_batch_it_is_leaving, which patches atomic_write to
fail the test outright, pins the stale listing as the accepted outcome (so clearing it later is
a deliberate change), then empties the trash. Mutation-verified: restoring the unconditional
rewrite reddens it on the write assertion.
First Principles CONCERNS (advisory) -- all three subtractions taken, one with a correction
TreeRemoval.error -- DROPPED. Zero readers, confirmed. I considered the other direction
(log the errno in the caller's warning) and rejected it: inventing a consumer to justify a
field is backwards, and the module's convention is that the exception rides along only where
the caller RE-RAISES it, which is why StagedRemoval keeps its own. That reasoning is now in
the class docstring so the asymmetry does not read as an oversight.
approve non-Optional -- DONE, and the is not None branch is gone. Last round I argued a
caller with nothing to check was a real case; no such caller exists, and leaving the parameter
nullable only makes the ask-nothing mode reachable by omission. Tests that meant "approve
everything" now pass a named _approve_anything hook, so each call states which mode it
exercises.
_open_chain -- the wrapper is gone, the NAME stays as an alias. The subtraction is right
about the body: it added no check, no default, no error translation, and it is now
_open_chain = pinned_fs.open_verified_chain. Deleting the name outright would have cost
something the grep did not surface, though: test_session_storage.py patches
session_storage._open_chain to swap a directory inside the removal window
(test_a_directory_swapped_after_the_scan_keeps_its_files), and that seam is deliberately
narrow -- patching pinned_fs.open_verified_chain instead would also hook the pinned removal's
own internals, widening the double past what the test is about. The four call sites and two
main-owned test doubles now pass the upstream keyword-only arguments.
Watch item -- single-consumer knobs on remove_tree_pinned
Accepted as stated. keep_until_empty is forced by the mechanism/policy split and refusal
follows the module's standing convention; approve's speculative half is exactly what the
subtraction above removed, so the remaining single-consumer surface is the two that earn it.
Verification: 308 passing across test_trash_pinned_sibling_removal.py,
test_session_storage.py, test_pinned_staging.py. black/flake8/isort clean on the four
touched files, mypy clean on both source files (its 2 errors remain pre-existing in
transcribe.py), docs_lint passes.
5b707cc to
aa70956
Compare
Windows regression fixed on
|
aa70956 to
ca3520f
Compare
Round on
|
origin/main |
this PR | |
|---|---|---|
sibling A (_discard_restored_batch) |
shutil.rmtree(batch, ignore_errors=True), line 1380 |
rename aside in the trash root, verify (st_dev, st_ino) there, rmtree the staged name, put back on mismatch |
sibling B (move_to_trash cleanup) |
shutil.rmtree(target, ignore_errors=True), line 1936 |
same |
empty_trash coarse path |
rename-verify-remove, residual documented and accepted | unchanged, now sharing one owner |
On main the attack needs no window and no name: the bare rmtree follows a swapped ancestor
by default, and ignore_errors=True makes it silent. On this branch it needs the random
staging name AND a swap landing after the identity check. The exposure GPT names is therefore
strictly narrower than base, not introduced here -- and it is the same residual main already
documents in _delete_listed_files for this platform, in a comment that says so explicitly.
The mechanism space on this platform is exhausted. Three positions have now been through
this lane:
| round | position | outcome |
|---|---|---|
| 1 | rmtree(path) (main's behaviour) |
blocked -- the original defect |
| 2 | refuse when no descriptor | taken, then disproven by five Windows tests |
| 3 | rename-verify-remove, as empty_trash does |
blocked now, as path-traversable |
There is no fourth option in Python: Windows has no openat, and a handle-based delete would
need ctypes against NtSetInformationFile/FILE_FLAG_BACKUP_SEMANTICS with reparse-point
handling -- a new platform mechanism, the right long-term fix, and plainly its own change
rather than a fourth patch inside a PR whose subject is the descriptor-capable path.
What I need from the maintainer. I will not post /ai-review override on my own
authority. The disposition I am asking to record is "pre-existing platform limitation,
measured delta strictly narrower than base, not applicable to THIS PR", with the Windows
mechanism tracked as a separate issue. The alternative that needs no override is to descope
Windows from this PR entirely -- keep the descriptor path, leave both siblings on main's
rmtree(ignore_errors=True) where _FD_SAFE_DELETE is false -- which satisfies the letter of
this lane's requirement (this PR adds no by-path removal) while leaving Windows exactly as
insecure as it is today. I think that is the worse product outcome and am not doing it
unasked.
309 passing across the three targeted suites on ca3520fc0; gates and docs_lint clean. The
other red on this head, the dashboard/handlers/memory.py log-site census, is main-owned with
evidence in my previous comment.
Round on
|
Round on
|
Residual now tracked: #7566The descriptor-less Windows removal path is filed as It also records, because it is the part that decides scope, that this is not a regression Next on this PR: #7554 (the |
Windows regression confirmed fixed by CI, and First Principles' three Watch items dispositionedThe Windows fix is verified in the environment, not just locally
Windows shards 1, 2 and 4 are green, as is the Windows installer build. That is the branch Opus 4.8 reports no blocking findings on this head, having validated and dropped its one First Principles -- CONCERNS (advisory), three Watch items1. Worth noting that I apply this rule symmetrically rather than only when it favours keeping 2. The move-aside-remove-put-back choreography is spelled twice: ACCEPTED-AND-DEFERRED, 3. Item 7 (delete-path debris unlink and put-back now inode-verified) is not in the Next step is blocked on #7554, not on me#7554 (the |
Post-rebase round on
|
Round on
|
Discarding a fully-restored batch and cleaning up a batch no session was staged into both removed the directory with shutil.rmtree on a path, which re-resolves every ancestor: one swapped to a symlink after the caller's own by-path read was followed and the removal landed outside the trash. Both now go through pinned_fs.remove_tree_pinned, and the rename-verify-remove mechanism that was spelled inline three times moves into pinned_fs with the pinned scan and the verified chain-open beside it. session_storage keeps the policy: which map authorises a removal, and what a refusal means to the user. Closes #7113
Round on
|
Round on
|
origin/main (session_storage.py:3232) |
this PR | |
|---|---|---|
| source | never opened or verified at all | opened O_RDONLY | O_NOFOLLOW, inode checked against expect_ino before anything is read |
| link | os.link(debris, MANIFEST_NAME, src_dir_fd=..., dst_dir_fd=...), default follow_symlinks |
follow_symlinks=False |
| after | nothing checks what landed | landed inode verified; on mismatch the substitute is left untouched and the failure is reported |
The prescribed remedy is to delete the hard-link branch and copy exclusively from the verified
descriptor. That removes the only non-clobbering ATOMIC put-back and makes every recovery a
byte copy on every filesystem, including the ones where links work - the branch's own docstring
records why link is first choice. It also cannot be tested as an improvement: the copy path is
already exercised, so the change is pure subtraction of a working mechanism.
There IS a real fix that GPT did not propose: link from the DESCRIPTOR rather than the name, via
os.link(f"/proc/self/fd/{src}", ...), which cannot be redirected because no name is resolved.
It is Linux-only, so it needs the name-based link as a fallback everywhere else, which leaves the
window on exactly the platforms that keep it. I flag it as an option rather than shipping it at
round sixteen of a review whose subject is two other functions.
pinned_fs.py:1471 (plus :1749, session_storage.py:2811) -- "verified cleanup can unlink a
replacement". Also real, and irreducible: POSIX has no atomic "unlink if this inode", so a
stat-then-unlink is two syscalls.
origin/main (session_storage.py:3276-3280) |
this PR | |
|---|---|---|
| debris cleanup | with suppress(OSError): os.unlink(debris, dir_fd=parent_fd) -- unconditional, by name |
identity-checked, and on the tree-survived path the debris is RETAINED rather than unlinked |
| partial copy | removed by name | emptied through the descriptor FIRST (os.ftruncate(dst, 0)), then the identity-checked unlink |
This is the third distinct remedy this lane has asked for at this one site, and they are mutually
exclusive:
| round | prescribed | outcome |
|---|---|---|
| 1 | unlink only while the name still holds our file | implemented |
| 2 | never unlink by name; truncate through the descriptor | truncate implemented; the unlink kept, because dropping it reddens a mainline test |
| 3 (now) | retain the empty partial or debris instead of unlinking | reddens that same mainline test |
The test is not mine:
test_session_storage.py::TestEmptyTrash::test_a_failed_copy_back_leaves_no_half_written_manifest
"A truncated manifest is worse than none at all. ... So a failed copy removes what it wrote
and reports the failure, which at least leaves the debris named in the log."
assert not manifest.exists(), "a partial manifest must not be left behind"
Applying the same change at :1749 is worse still: that is the path where the tree IS gone, so
retaining there leaves a debris file behind after every successful trash cleanup.
What I am asking for
The Command Center has already adjudicated the coarse-platform finding on exactly this basis
(pre-existing, measurably narrowed here, prescribed remedy disproven by mainline tests) with the
residual tracked in #7566. These two are the same shape at two more sites in a RIDER - the delete
path's recovery, which this PR hardened because it shared the root cause, not because it was the
subject.
Options, in the order I would rank them:
- Extend the recorded disposition to cover all three, and add the two recovery residuals to
Windows session-trash removal and manifest recovery have no descriptor-bound path #7566 (or a sibling issue) as named, measured, strictly-narrowed-versus-base items. - Descope the rider: revert the delete-path recovery hardening to mainline's spelling, keeping
only what the two sibling cleanups need. That structurally removes the finding cluster's root
cause, at the cost of leaving mainline's weaker recovery in place. - Take the
/proc/self/fdlink on Linux with a name-based fallback elsewhere, as its own change.
313 passing across the three targeted suites on this head; black/flake8/mypy clean on
touched files, docs_lint passes. No code change this round: none of the three remedies is
takeable without breaking a mainline test or subtracting a working mechanism.
|
/ai-review override gpt 87348df: All three residuals exist on main in strictly worse form (unconditional by-name unlink; link with no source verification; bare rmtree with no identity check), so this PR narrows every site it touches. The prescribed remedies are mutually exclusive across rounds and two redden pre-existing tests (five Windows cases; a mainline copy-back case whose docstring states the opposite intent). Tracked in #7566. |
Human judgment recorded@chenmingwei23 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
Disposition of all three findings on
|
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed via parallel subagent audit: diff matches description, CI fully green, no blocking findings, no unresolved threads.
What is the problem?
#7011 bound the trash DELETE path to descriptors and named two sibling paths it left open.
Both were still removing a batch by re-resolved path:
_discard_restored_batch, after a restore has moved every listed file back outmove_to_trash's cleanup, when a staging attempt ends with nothing recordedBoth spelled it
shutil.rmtree(batch, ignore_errors=True). A path is re-resolved component bycomponent by the kernel, so a directory above the trash swapped to a symbolic link after the
caller's own checks is followed, and the removal lands wherever the link points. Neither caller
holds the mutation lock across that window, every check above them is by path, and
ignore_errors=Truemeans the deletion that lands outside the trash is also silent.Why this issue matters to the user
The trash holds the only copy of a session's replay log, transcript and archive segments
between staging and an explicit empty. These two paths run on ordinary operations - every
restore, and every move that rolls back - so the exposure is not a rare corner.
Everything here runs as one uid, which in this product includes an agent acting on the user's
behalf. That is the actor the pinning defends against: not a remote attacker, but a process
that can write in the directories above the trash while the user's own restore is in flight.
How our fix solves it
Both siblings now remove by descriptor. They call
pinned_fs.remove_tree_pinned, which pins the parent chain oneopenatper component, opensthe batch through it with
O_NOFOLLOW, scans it once, unlinks links and files against theinodes that scan recorded, removes directories deepest-first through
remove_dir_verified,re-scans to decide whether the tree is actually empty, and only then removes the batch itself,
verified against the descriptor the whole operation was pinned to.
Pinning is the weaker half, so an identity carries the fix.
Path.resolve()follows anancestor that is ALREADY a link, so a swap landing before the resolve produces a faithful
pinned walk to the wrong tree. Each caller therefore records the batch's
(st_dev, st_ino)before its own by-path read - at
target.mkdir(...)under the mutation lock on the stagingpath, and before the leftover scan on the restore path - and the approval hook compares the
pinned root's
fstatagainst it as its first question. An unreadable identity refuses.The content checks come second and cannot carry it. From ONE inode-verified read of ONE
file, the approval re-asks that the manifest's header claims this batch's own directory name,
and that NOTHING but the manifest remains - not "nothing unlisted", nothing at all. Both
callers arrive with the batch empty of files, so a file at a listed path is not the listed
file; it arrived afterwards and may be the only copy of whatever it is. An actor who can write
into the tree a swapped link points at can plant a matching header; an inode cannot be forged
by writing files.
The manifest goes last.
list_trash()omits a batch with no readable manifest, so removingit first and then failing on the directory would leave data on disk that nothing lists.
keep_until_emptynames it: skipped by the file pass, required to be the only survivor of theclosing scan and verified there by inode, then moved aside under an unguessable debris name and
put back through
put_back_no_clobberif the tree still will not go - link first, then anO_CREAT | O_EXCLcopy, becauseos.link in os.supports_dir_fdtests what the OS accepts andnot what the mount supports. The debris copy is RETAINED whenever the removal failed: no check
available before an unlink can prove the name it went back to still holds a good copy, so
removing the redundant one is a chance to turn a stuck batch into an unlistable one. The
retained name is reported through
staged_name.On a platform with no
openat, the removal is bound to a verified name instead._stage_batch_by_namerenames the batch aside inside the trash root, verifies(st_dev, st_ino)on the RENAMED directory against the caller's identity, and removes only the stagedname; a mismatch, an unreadable identity, or a tree that will not go puts the directory back
under the name the user saw. It is one owner for three callers - the explicit empty, which
already worked this way, and the two cleanups. Refusing outright instead is not available:
that branch is the whole of Windows, so refusing leaves a batch after every restore and every
rolled-back move, still listing sessions it no longer holds, which five pre-existing tests
read as a failure. The residual is stated rather than implied and tracked in #7566.
A refused cleanup writes nothing. The batch stays exactly as found, including a manifest
whose entries describe files the restore already moved out. That stale listing is a decided
residual: clearing it needs a write to the batch path, and there is nowhere safe to put one.
After the cleanup, every refusal is itself evidence the path may be redirected, and
atomic_writeREPLACES its destination. Before it, the full-restore branch has no write atall today, so adding one hands that primitive a path a batch swapped to a link points wherever
an actor chose, including another batch's manifest, whose sessions would become unlistable. A
stale listing is visible and reversible; that is not, and
empty_trashclears the batch on theuser's own say-so.
One owner per mechanism.
rename-verify-removewas spelled inline three times - theinterior directories, the batch directory, and the coarse path - and is now
pinned_fs.remove_dir_verified, withscan_tree_pinned,open_verified_chain,drain_verified_chain,remove_tree_pinnedandput_back_no_clobberbeside it.session_storagekeeps the policy: which map authorises a removal, what a refusal means tothe user, and how it is worded. Per-call-site respelling of this mechanism is what
pinned_fswas created to end after #2446 and #2447.
Riders, same root cause. The delete path's manifest recovery trusted a mutable name: it
opened the debris without
O_NOFOLLOW, linked with the defaultfollow_symlinks=True, andunlinked unconditionally, on a name in a directory an actor may be able to write to, reached
only after something already failed. Its source is now opened
O_NOFOLLOWand inode-checkedbefore anything is read, the copy reads that descriptor rather than re-opening the name, the
link passes
follow_symlinks=Falseand checks what landed, and both debris unlinks areidentity-checked. Leaving this out would have shipped a shared put-back owner whose delete-path
caller kept the unguarded spelling of the same hole.
_read_manifestgainsexpect_ino, checked withfstaton the open handle, so a caller thatneeds the header and the listing to describe ONE file makes a single call - two calls are two
opens, and a manifest rewritten between them lets the header check pass on one file while the
listing comes from another. The approval also refuses a manifest that is not valid UTF-8:
UnicodeDecodeErroris aValueError, so theexcept OSErrorthat turns an unreadablemanifest into a refusal does not cover it, and an escape would abort a restore that had already
moved the user's files back.
What tests we did
test/test_trash_pinned_sibling_removal.pyis new and covers both siblings end to end. Theancestor swap is set up ON DISK rather than simulated, and the victim carries a file, so a
passing test says the file survived rather than that a branch was taken:
identity was captured before the swap
not go, with the redundant copy retained and reported
source name, and removes a partial copy only while it is still the file this call created
directory that is not the selected one is put back with its contents intact, and a swapped
ancestor reaches no victim
Every security assertion is mutation-verified: reverting the guard reddens the test, each on a
victim file that no longer exists or a credential that was copied.
The descriptor-less branch is unreachable on Linux, so it was run locally against a forced
coarse platform in both directions - the five pre-existing Windows tests pass with the staged
removal and fail with a refusal - and then confirmed on the Windows shards in CI.
Gates:
black,flake8,isort,mypyclean on touched files;docs_lintpasses; the specin
docs/system-specs/modules/session-storage.mdmoves with the code in the same commit.Any other suggestions on the work?
Accepted residuals, each named rather than implied. The descriptor-less platform still
resolves the staging path, so an actor who can observe that name inside the window can redirect
the removal through an ancestor swapped afterwards, and
_identity_ofis itself a by-pathstatthat a swap already in place can fool. Both are tracked in #7566, which describes thehandle-based deletion that would close them. Neither is a regression: on mainline these paths
are a bare
shutil.rmtree(batch, ignore_errors=True)with no identity check at any point, and_identity_ofdoes not exist there at all.Deferred, with a reason.
_delete_listed_filesstill drives its own move-aside-removechoreography rather than calling the primitive. Converging it means carrying freed-byte
accounting, the
clearedflag behindSKIP_INCOMPLETE, and a progress callback into a modulethat holds no policy by design, on a path that only just converged in #7011.
remove_tree_pinnedhas one consumer today, serving both fixed sites. The alternative is toinline the whole pinned walk back into
session_storage, which is the respellingpinned_fsexists to end.
Pattern harvest
Rule candidate: extend the AST ratchet in
test_pinned_staging.pytosession_storage.py,banning
shutil.rmtreeand by-nameos.rmdir/os.unlinkinside any function that holds adirectory descriptor.
The defect class is "a filesystem mutation that addresses a PATH inside a function that already
holds, or could hold, a descriptor for the object".
test_pinned_staging.pyalready ratchetsexactly this for
pinned_fs.pyby walking the AST - and it is the reason this PR's own movedcode was caught rather than shipped.
session_storage.pyis not covered, which is why thesetwo
rmtree(path)calls survived #7011: nothing mechanical would have named them.It cannot be done here as a clean gate:
session_storage.pyhas legitimate by-name sites - thecoarse branch and the by-path pre-screens - and separating them needs the per-line marker
convention that the existing ratchet's own docstring identifies as the blocker for widening to
snapshot.py. That is one change serving three modules rather than three narrow ones.Not a one-off: this is the third time this exact class has been closed in this module (#2446,
#2447, #7011), and each round found it at a site the previous round's review had not looked at.
Closes #7113