fix(cache): key a reader stage on its file's contents, not its filename - #29
Open
celikgo wants to merge 1 commit into
Open
fix(cache): key a reader stage on its file's contents, not its filename#29celikgo wants to merge 1 commit into
celikgo wants to merge 1 commit into
Conversation
The stage key is built from the capability id, the plugin version, the declared input tree and the upstream hashes. A reader's declared input is a *path*, and a path is not what a reader reads. Two pipelines naming the same relative input file therefore produced identical keys, and the second was served the first one's output with the stage reported as [CACHED]. Not a crash and not a diff — a plausible, wrong answer. Reproduced before fixing: two directories, identical pipeline.yaml, different cube.stl, one shared cache and one shared cwd. Both runs printed `read hash=c8142a68…`, the second printed `[CACHED ] write`, and its cube.vtu held the first cube's twelve cells. The fix folds a SHA-256 of the reader's input bytes into the cache-key context, through a new IDispatcher::cache_key_extra hook. RegistryDispatcher implements it for the `reader.` namespace only: every other capability is fully described by its declared inputs plus upstream hashes, and a writer produces rather than consumes. Both runners apply it — they share one cache, so a key computed differently in each would silently partition it. hash_file streams in 64 KiB chunks rather than slurping. This runs on every stage evaluation, cache hits included, because the digest is precisely what decides whether a hit is legitimate; a reader's input can be a very large mesh and holding it in memory only to digest it would be a poor trade for correctness that has to be paid on the fast path anyway. A short read returns nullopt rather than a digest of a prefix, which would be a plausible-looking key for content nobody has seen. Cross-platform safety is not incidental and was checked: `.gitattributes` marks `*.stl binary` and everything else falls under `* text=auto eol=lf`, so every corpus input is byte-identical on all three runners and the digest cannot itself become a determinism failure. Verified: the manifest is stable run to run and still 10 of 10. Also makes an in-place edit of an input invalidate the stages that read it, which it previously did not — examples/pipe-bend/README.md said as much two commits ago and is corrected here. The defect was reachable in this form only once reader paths resolved against their pipeline file: before that two pipelines could not share a working directory, and a Kind::Path output is rehydrated only when the file it names still exists, so the second run missed on fs::exists and re-ran by accident. That same accident is what makes the regression test particular — it needs a shared cwd *and* a shared cache, and it asserts on the [CACHED] tag rather than on file contents, because in a shared directory the two runs overwrite each other's output whatever the keys say. Confirmed to fail without the runner change and pass with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wbhn3NTFjq5pAuowgHAKkB Signed-off-by: celikgokhun <celik.gokhun@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The pipeline cache could serve one pipeline's output to another. Reproduced before fixing:
A stage key is built from the capability id, plugin version, declared inputs and upstream hashes. A reader's declared input is a path, and a path is not what a reader reads. Not a crash and not a diff — a plausible, wrong answer.
The fix
A SHA-256 of the reader's input bytes is folded into the cache-key context, via a new
IDispatcher::cache_key_extrahook.RegistryDispatcherimplements it for thereader.namespace only — every other capability is fully described by its declared inputs plus upstream hashes, and a writer produces rather than consumes. Both runners apply it: they share one cache, so a key computed differently in each would silently partition it in half.hash_filestreams in 64 KiB chunks rather than slurping. It runs on every stage evaluation including cache hits, because the digest is exactly what decides whether a hit is legitimate — so it cannot be skipped on the fast path, and a reader's input can be a very large mesh. A short read returnsnulloptrather than a digest of a prefix, which would be a plausible-looking key for content nobody has seen.Side effect worth having: an in-place edit of an input now invalidates the stages that read it, which it previously did not.
examples/pipe-bend/README.mdclaimed as much two commits ago and is corrected here.Determinism
Not incidental, and checked rather than assumed.
.gitattributesmarks*.stl binaryand everything else falls under* text=auto eol=lf, so every corpus input is byte-identical on all three runners and the digest cannot itself become a determinism failure. Verified: manifest stable run-to-run, still 10 of 10 hashed, 0 EXIT-*.Why it survived, and why the test looks odd
Reachable in this form only since reader paths began resolving against their pipeline file (#28). Before that two pipelines could not share a working directory — and a
Kind::Pathoutput is rehydrated only when the file it names still exists, so the second run missed onfs::existsand re-ran, producing the right answer by accident.That accident is what makes the regression test particular. It needs a shared cwd and a shared cache, and it asserts on the
[CACHED]tag rather than on file contents, because in a shared directory the two runs overwrite each other's output whatever the keys say. I got this wrong twice while writing it — separate directories make the test pass with the bug still present. Confirmed to fail without the runner change and pass with it.Known limit
Kind::Pathoutputs still cache a path, not content, so two pipelines writing the same relative output filename into one directory clobber each other regardless of keys. That is the output contract rather than a cache defect, and it is documented in the test's comment.Verification
748/748tests. Determinism 10/10 and stable. Frozen ABI untouched. clang-format-17, doc-links and capability-table gates clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01Wbhn3NTFjq5pAuowgHAKkB