Repository navigation
Conversation
Presigned upload URLs can't be revoked, so one could modify a zarr after it has been ingested. Record each issued URL, and defer ingestion if the file at any unexpired URL's path doesn't match the MD5 that URL was signed with. Periodic tasks ingest deferred zarrs once their URLs expire, and delete expired URL records. To bound the delay, zarr upload URLs now expire after 1 hour (DANDI_ZARR_UPLOAD_URL_EXPIRATION) instead of 6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Allow uploading to and deleting from UPLOADED zarrs, returning them to PENDING, so a deferred upload can be resumed and ingested immediately. Expose when deferred ingestion will be retried via a new `ingestion_deferred_until` field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
naglepuff
requested changes
Nov 11, 2026
naglepuff
reviewed
Nov 12, 2026
Comment on lines
+181
to
+185
| sender.add_periodic_task( | ||
| timedelta(minutes=5), | ||
| ingest_deferred_zarr_archives.s(), | ||
| expires=timedelta(minutes=5).total_seconds(), | ||
| ) |
Contributor
There was a problem hiding this comment.
Could this also be configurable? Same reason as URL expiry time. It would make it easy to configure the CLI testing environment to test the deferred zarr workflow.
3 of 12 tasks
naglepuff
force-pushed
the
zarr-upload-url-tracking
branch
from
November 18, 2026 16:37
97ad50d to
5d81132
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Presigned zarr upload URLs can't be revoked, so a file could be written through one after the zarr has been ingested, leaving its checksum wrong. This PR closes that gap and makes the resulting deferral recoverable.
Track upload URLs and defer ingestion
ZarrUploadURLmodel (path, MD5, expiry).UPLOADED. Uploads that use all their URLs are ingested immediately, as before.DANDI_ZARR_UPLOAD_URL_EXPIRATION) instead of 6, to bound the delay.base64md5is now validated as a base64-encoded MD5 digest when requesting upload URLs.Allow resuming deferred uploads
UPLOADEDstate, which returns it toPENDING. This lets a client finish a deferred upload and finalize again rather than wait for the URLs to expire.INGESTINGzarrs still reject changes.ingestion_deferred_untilfield on the zarr endpoints reports when a deferred ingestion will be retried.