Repository navigation
Conversation
Contributor
Author
|
@yarikoptic FYI I've created this branch/PR to address deferred zarr ingestion from the CLI side. I think separating it out from your PR #1988 makes sense. There might be some conflicts between these two branches anyway, but I don't think resolving them should be too bad. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1997 +/- ##
==========================================
+ Coverage 90.91% 91.03% +0.11%
==========================================
Files 99 99
Lines 15041 15245 +204
==========================================
+ Hits 13675 13878 +203
- Misses 1366 1367 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
dandi-archive#3018 defers ingesting a Zarr while any unexpired upload
URL could still change it, leaving it "Uploaded" with
"ingestion_deferred_until" set; the server ingests it on its own once
those URLs expire. Polling for "Complete" without a timeout made
`dandi upload` hang for up to an hour in that case.
- Add `BaseRemoteZarrAsset.wait_for_ingestion()`, which returns once
the Zarr is "Complete" or its ingestion is deferred, and raises
`ZarrIngestionError` if the Zarr becomes "Pending" again. Use it in
`iter_upload()` and `rmfiles(reingest=True)`.
- On a deferral, `iter_upload()` skips the local checksum verification
and reports until when ingestion is deferred.
- When there is nothing to upload, check the Zarr's status rather than
always reporting it as identical: finalize it if an earlier upload
ended before doing so ("Pending"), wait for it if it is being
ingested, and report a deferral.
- Servers without dandi-archive#3018 do not report
"ingestion_deferred_until" and keep the prior behavior.
- In tests that download a Zarr after rewriting or deleting recently
uploaded entries, wait for its ingestion, with a longer timeout until
the test archive instance can shorten the interval for retrying
deferred ingestion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dandi-archive#3018 now makes the interval for retrying deferred Zarr ingestion configurable. Set it to 5 seconds in the test archive instance, so that tests which download a Zarr after rewriting or deleting recently uploaded entries wait well under a minute for its ingestion, and drop their longer timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y logic test_zarr_upload_400_timeout_retry replaced RESTFullAPIClient.request() itself, so its fake 400 bypassed the retry logic there, and the entries it was returned for were never uploaded. It passed only because a second pass of the upload, after the Zarr's checksum did not match, uploaded them again; against dandi-archive#3018, which defers ingesting a Zarr with unused upload URLs, there is no such second pass, and it failed. Return the fake 400 from requests.Session.request() instead, so that it is retried by RESTFullAPIClient.request(), and check that the upload needed neither a second pass nor had its ingestion deferred. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ting When the server defers ingesting a Zarr (dandi-archive#3018), it has no checksum yet, so the upload skipped verifying the remote Zarr against the local one. An entry that the client believed it uploaded but which is missing remotely then went unnoticed: once its unused upload URL expires, the server ingests the incomplete Zarr as "Complete". Instead, compute the remote checksum from the listing of the Zarr's files, which comes from storage directly and is what the server computes its checksum from, and redo the upload on a mismatch as before. Also reset the deferral when there is nothing to upload and the Zarr is "Complete", so that a deferral left over from an earlier pass is not reported. Test that a lost entry is uploaded again, sharing the faking of entry uploads below the retry logic with test_zarr_upload_400_timeout_retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As DEVELOPMENT.md asks for the use of a fallback for servers without a dandi-archive feature, here dandi-archive#3018. It is logged once per wait for ingestion, when the Zarr is ingested, rather than on every poll. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
@claude please review this PR |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
CI runs the integration tests against a dandi-archive image built from master, which does not have dandi-archive#3018 yet, so the code handling a deferred ingestion never ran there, and codecov/patch failed. These tests mock the behavior of dandi-archive#3018 to bump coverage until CI's image includes it: - test_upload_zarr_deferred_ingestion_faked makes RemoteZarrAsset.wait_for_ingestion() report a deferral, whatever the server did, so that the upload verifies the Zarr against its remote file listing (from the real server) and reports the deferral. - test_rmfiles_reingest_deferred mocks the server's responses to check that rmfiles(reingest=True) logs a deferral rather than waiting. test_upload_zarr_deferred_ingestion and test_upload_zarr_lost_entry still exercise a real deferral against servers with dandi-archive#3018. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement CLI-side changes corresponding to: dandi/dandi-archive#3018
Handle deferred Zarr ingestion (dandi-archive#3018)
Draft until dandi-archive#3018 is merged. Client-side counterpart to dandi/dandi-archive#3018 ("Track zarr upload URLs").
Background
With dandi-archive#3018, the server records every presigned Zarr upload URL. When a Zarr is finalized, the server defers ingestion if any unexpired URL could still change the Zarr, meaning the file at its path doesn't have the MD5 the URL was signed for. The Zarr then goes back to
Uploadedwithingestion_deferred_untilset. A periodic task ingests it once those URLs expire. Upload URLs expire after 1 hour by default.In practice, overwriting or deleting a Zarr entry within an hour of uploading it defers ingestion by up to about an hour. New Zarrs, add-only changes and identical re-uploads are not affected.
Before this PR, dandi-cli didn't know about this state:
ZarrAsset.iter_upload(), and the one inRemoteZarrAsset.rmfiles(), waited forCompletewith no timeout. A deferred Zarr leftdandi uploadsilently stuck for up to an hour.skipped / identical, while the Zarr was stillUploadedwith no checksum.Changes
BaseRemoteZarrAsset.wait_for_ingestion()polls the Zarr until it isCompleteor its ingestion is deferred (Uploadedwithingestion_deferred_untilset), and raises the newZarrIngestionErrorif the Zarr becomesPendingagain (modified meanwhile). Both poll loops use it.zarr_ingestion_deferred_until()is the single check for a deferral.dandi uploaddoes not wait for a deferred ingestion. The server ingests the Zarr on its own, and the final status says until when ingestion is deferred (donewithserver ingestion deferred until <time>).GET /zarr/{id}/files/, which lists storage directly and is what the server computes its checksum from). A mismatch redoes the upload as before. Without this, an entry the client believed it uploaded but which is missing remotely would go unnoticed: once its unused upload URL expired, the server would ingest the incomplete Zarr asComplete. Patch mode still skips verification.Pendingby an interrupted upload (which was otherwise never finalized, also on servers without #3018), waits for one being ingested, and reports a deferral (skippedwithidentical; server ingestion deferred until <time>).RemoteZarrAsset.rmfiles(reingest=True)logs a deferral instead of waiting for it.Tests
DJANGO_DANDI_ZARR_UPLOAD_URL_EXPIRATION: "30"andDJANGO_DANDI_DEFERRED_INGESTION_RETRY_INTERVAL: "5", so that a deferral clears within about 35 s. Servers without #3018 ignore them.wait_for_ingestion()with mocked responses, including a server withoutingestion_deferred_until.test_upload_unchanged_unfinalized_zarr: a rerun finalizes a Zarr leftPending.test_upload_zarr_deferred_ingestion: the upload and a rerun report a deferral; skipped if the server does not defer.test_upload_zarr_lost_entry: an entry whose upload is faked as successful is detected and uploaded again; against #3018, via the remote file listing.masterand does not defer:test_upload_zarr_deferred_ingestion_fakedmakesRemoteZarrAsset.wait_for_ingestion()report a deferral, whatever the server did, so that the upload verifies the Zarr against the real server's file listing and reports the deferral.test_rmfiles_reingest_deferredmocks the server's responses to check thatrmfiles(reingest=True)logs a deferral rather than waiting._wait_for_zarr_ingestion()). The other three need no change, as the upload no longer blocks.test_zarr_upload_400_timeout_retrydid not test what it claimed: it replacedRESTFullAPIClient.request(), bypassing the retry logic there, and passed only because a second pass after a checksum mismatch uploaded the entries again. It now fakes the 400 below the retry logic.Against a local instance of dandi-archive#3018 (plus #3006) via
DANDI_TESTS_API_URL, all Zarr tests intest_upload.pyandtest_dandiapi.pypass, excepttest_upload_bids_zarr, which needsbids-validator-deno(not installed locally). In CI, whose dandi-archive image is built frommaster, the checks that need a real deferral are skipped, ortest_upload_zarr_lost_entrytakes the server-checksum path instead. Those test lines are the only changed lines that CI does not cover.Compatibility
Following
DEVELOPMENT.md, the feature is detected rather than assumed: servers that don't reportingestion_deferred_untilnever defer, and keep the prior behavior apart from thePendingfix above.GET /zarr/{id}/includesingestion_deferred_until, which cannot be mistaken for any other condition. The conditional (zarr_ingestion_deferred_until()) refers to dandi-archive#3018 and says when the fallback can be removed.wait_for_ingestion()logs this at DEBUG, once per wait.DandiAPIClient:DEVELOPMENT.mdsuggests probing once, or detecting support on first use, and remembering the result. That avoids repeating a probe or a failing first request. Here, a server with the feature includes the field in everyGET /zarr/{id}/response (asnullunless ingestion is deferred), and the client checks it only in responses it fetches anyway. Detecting support per response therefore needs no probe and no extra request, and remembering it would save nothing.test_wait_for_ingestion_without_deferral_support), and integration tests that skip, or skip their deferral-specific checks, when the server does not report it.Relation to #1988
#1988 handles the other half of the Zarr publishing stack (dandi-archive#3006): uploading changes to a published Zarr as a new Zarr. That PR leaves the deferred-ingestion poll handling out ("Not done"), and this PR takes it on.
Out of scope here: retrying Zarr changes that dandi-archive#3006 rejects with 423 while the Dandiset is being published. That comes from #3006, not #3018. It belongs with a related gap in #1988: a Dandiset published after #1988's
publishedcheck but before the upload finishes leaves the upload with a 400 or 423. Both should be handled in #1988 or a follow-up to it.Overlap with #1988:
GET /zarr/{id}/only when there are changes, to checkpublished. This PR now makes that request on the unchanged path, to checkstatus. To keep it at one request per upload, the two should share the response; to be coordinated when rebasing.iter_upload()and indandi/tests/test_upload.py. Whichever PR merges second rebases.🤖 Generated with Claude Code