Skip to content

Track zarr upload URLS - #3018

Open
jjnesbitt wants to merge 6 commits into
masterfrom
zarr-upload-url-tracking
Open

jjnesbitt wants to merge 6 commits into
masterfrom
zarr-upload-url-tracking

Conversation

@jjnesbitt

Copy link
Copy Markdown
Member

Presigned zarr upload URLs can't be revoked, so a file could be written through one after the zarr has been ingested, leaving its checksum wrong. This PR closes that gap and makes the resulting deferral recoverable.

Track upload URLs and defer ingestion

  • Each issued upload URL is recorded in a new ZarrUploadURL model (path, MD5, expiry).
  • A URL can only write the content it was signed with, so it is harmless once the file at its path has that MD5. Ingestion checks this using the S3 listing it already does for the checksum.
  • If any unexpired URL is unused or its file no longer matches, ingestion is deferred and the zarr returns to UPLOADED. Uploads that use all their URLs are ingested immediately, as before.
  • A periodic task (every 5 minutes) ingests deferred zarrs once their URLs expire; another (hourly) deletes expired URL records.
  • Zarr upload URLs now expire after 1 hour (DANDI_ZARR_UPLOAD_URL_EXPIRATION) instead of 6, to bound the delay.
  • base64md5 is now validated as a base64-encoded MD5 digest when requesting upload URLs.

Allow resuming deferred uploads

  • Files can now be uploaded to or deleted from a zarr in the UPLOADED state, which returns it to PENDING. This lets a client finish a deferred upload and finalize again rather than wait for the URLs to expire. INGESTING zarrs still reject changes.
  • A new read-only ingestion_deferred_until field on the zarr endpoints reports when a deferred ingestion will be retried.

jjnesbitt and others added 3 commits November 11, 2026 13:31
Presigned upload URLs can't be revoked, so one could modify a zarr
after it has been ingested. Record each issued URL, and defer ingestion
if the file at any unexpired URL's path doesn't match the MD5 that URL
was signed with.

Periodic tasks ingest deferred zarrs once their URLs expire, and delete
expired URL records. To bound the delay, zarr upload URLs now expire
after 1 hour (DANDI_ZARR_UPLOAD_URL_EXPIRATION) instead of 6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Allow uploading to and deleting from UPLOADED zarrs, returning them to
PENDING, so a deferred upload can be resumed and ingested immediately.

Expose when deferred ingestion will be retried via a new
`ingestion_deferred_until` field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread dandiapi/settings/base.py Resolved
Comment on lines +181 to +185
sender.add_periodic_task(
timedelta(minutes=5),
ingest_deferred_zarr_archives.s(),
expires=timedelta(minutes=5).total_seconds(),
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this also be configurable? Same reason as URL expiry time. It would make it easy to configure the CLI testing environment to test the deferred zarr workflow.

@naglepuff
naglepuff force-pushed the zarr-upload-url-tracking branch from 97ad50d to 5d81132 Compare November 18, 2026 16:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants