Repository navigation
Compare storage namespaces by physical location - #2105
Merged
Merged
Conversation
The retention reaper decided whether a routed object key was shared with another team by comparing namespace fingerprints, and a fingerprint was the hash of the configuration exactly as a team typed it. Object keys carry no team segment, so 'R', 'R/' and 'R/.' of one local-rwx root counted as three namespaces, and one team's sweep deleted another team's live bytes. Each provider module now projects its configuration onto the physical namespace it reaches: a local root is made absolute, trimmed of trailing separators, resolved through existing symlinks the way realpath(3) resolves them (a link target is walked component by component, so 'L -> a/..' climbs from where 'a' lands, not from L's directory) and case-folded where the platform's filesystem ignores case; an OSS configuration reduces to bucket, region and key prefix, merging every host that can only reach that bucket. Admission, the agent-run-log bootstrap and the deployment-default materializer all store that projection's hash, and a configuration already written canonically keeps the fingerprint it had. Existing revisions are immutable and their fingerprints still name the typed text, so the reaper no longer reads them: it re-derives the object each colliding placement reaches at sweep time. A namespace is not enough for a filesystem, because the driver opens <root>/objects/<key> and the OS follows a link anywhere along it; a root the collector owns outright can link 'objects' (or any directory below it) into the victim's. So local-rwx, through the new IStorageProviderPhysicalObjects capability, names the file a delete of the key would unlink, with every directory on the way resolved and the final name left as written (unlink removes a link, not its target). Other providers are identified by namespace plus key. An object the reaper cannot derive (an unregistered provider, an unreadable configuration, a key no driver accepts) counts as shared, which keeps the bytes. The verifier flow tests asserted after one bounded pass over the whole table. Rows behind roots that earlier tests deleted never move, and the new two-team fixtures pushed their count past the batch, so those tests now sweep at the verifier's ceiling and wait on their own row.
8 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
R,R/andR/.of one local-rwx root counted as different namespaces, and one team's sweep deleted another team's live bytes.LocalRootPath.ResolveSymbolicLinks). SoL -> a/..climbs from wherealands, not from L's directory. Names are case-folded where the platform's filesystem ignores case.StorageProfileRules.TryObjectIdentity).<root>/objects/<key>, and the OS follows a link anywhere along that path.IStorageProviderPhysicalObjects. It names the file a delete of the key would unlink: every directory on the way is resolved, and the final name stays as written, because unlink removes a link and not its target.Test plan
LocalStorageNamespaceTests:..and//form, for existing and not-yet-created roots;objectslink and a nested link;LocalRwxArtifactStorageDriver;StorageNamespaceIdentityTestsand the OSS namespace tests.ArtifactRetentionReaperFlowTests.Namespaces):dotdot-through-link,objects-linkandnested-link, ends Indeterminate with the victim's file intact;StorageNamespaceSharingE2ETests): the victim's download returns 200 after the other team's sweep, for every spelling and link shape.ResourceAccessMarkerInventoryTestsandControllersDispatchThroughMediatorConventionTests.