📝 docs(storage): define an absent artifact placement - #2161
Merged
Conversation
Seven consumers read the placement projection and each decided for itself what a missing row meant. Nothing said, so one refuses a publish on it, two read it as local bytes, two read it as none, one drops a transition, and one records a row of its own. Most of those answers are right for the caller that makes them, but each was reached privately, and a comment on the file view had already drifted into claiming the upload path records a row it never writes. The next consumer would have invented an eighth reading from that comment. The projection holds no opinion about a digest it has no row for. It is not evidence the bytes are gone, because paths commit bytes without recording, and it is not evidence they are there, because reclaiming an orphan deletes the row before the file so an interruption understates what the node holds. So a caller answers from the record that independently establishes whether this node wrote the bytes and lets the projection demote that, never establish it; a caller with no such record reports none; a caller that needs proof refuses. That rule now sits on ArtifactPlacement, where every consumer already looks, with the read that returns the Option pointing at it. The file view and search had grown separate copies of the PyPI half and could drift apart on exactly the question at issue, so they resolve through one function that carries the rule. Behaviour is unchanged everywhere: the shared function reproduces both copies case for case, and the remaining consumers keep the answers they had, now with the reason written down rather than inferred. Finalize is the one worth naming. It still refuses on a missing row, but not because absence proves the bytes never arrived. It refuses because publishing needs positive evidence and a missing row is not evidence either way, so it fails closed and the retry succeeds once the arrival is recorded.
Merging this PR will not alter performance
Comparing Footnotes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seven consumers read the artifact placement projection, and nothing told any of them what a missing row means. Each decided on its own, so one refuses a publish on it, two read it as local bytes, two read it as none, one drops a transition, and one records a row of its own. A comment on the file view had already drifted into claiming the upload path records a row it never writes, while the comment on search says the opposite and is the correct one. Whoever read the first comment next would have derived an eighth answer from it. 🔍
One correction to the framing this issue opens with, since it set the priority: the readings are not five contradictory meanings of one thing. The two PyPI sites agree with each other, the two OCI sites agree with each other, and both pairs are right for their own write-side guarantee. OCI writes a manifest's row in the transaction that publishes it, so a missing row there does mean no local bytes. PyPI writes none for a hosted upload, so its views lean on the upload record and let the projection demote it.
apply_placement_eventhas one caller, which handles the empty case itself, and theProxysource the blob plane records is right for a replica whose bytes came from the primary. The defect is narrower. One rule sits behind all four, implemented by hand each time, stated nowhere, and carrying a false comment in one of them.This change settles the meaning and leaves behaviour alone. A digest with no row is a digest the projection holds no opinion about. It is not evidence the bytes are gone, because paths commit bytes without recording, which #2141 enumerates. It is not evidence they are there either, because reclaiming an orphan deletes the row before the file on purpose, so an interruption understates what the node holds rather than promising bytes that are already gone. A caller therefore answers from the record that settles on its own whether this node wrote the bytes, and lets the projection demote that answer rather than establish it. A caller with no such record reports none. A caller that needs proof refuses.
That rule now sits on
ArtifactPlacement, where every consumer already looks, with the read that returns theOptionpointing at it, and in the artifact source and availability page for readers who never open the type. The file view and search had grown separate copies of the PyPI half and could drift apart on the question at issue, so both resolve through one function that carries the rule. It reproduces both copies case for case, including the one where a same-digest mirror's row is ignored because it describes the mirror rather than the upload.finalizekeeps refusing on a missing row, and the reason changes rather than the outcome. Its module doc said it rejects on "a placement the bytes never reached", which reads absence as proof. Absence proves nothing, so the refusal now rests on publication needing positive evidence that the bytes reached this home: it fails closed, and the retry succeeds once the blob plane records the arrival. Nothing an operator or a client sees moves, here or at the other six sites, which keep the answers they had and now carry a written reason.Closes #2151