Skip to content

Add deferred ingestion handling for zarrs - #1997

Open
naglepuff wants to merge 7 commits into
masterfrom
mln/deferred-ingestion-handling
Open

naglepuff wants to merge 7 commits into
masterfrom
mln/deferred-ingestion-handling

Conversation

@naglepuff

@naglepuff naglepuff commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Implement CLI-side changes corresponding to: dandi/dandi-archive#3018

Handle deferred Zarr ingestion (dandi-archive#3018)

Draft until dandi-archive#3018 is merged. Client-side counterpart to dandi/dandi-archive#3018 ("Track zarr upload URLs").

Background

With dandi-archive#3018, the server records every presigned Zarr upload URL. When a Zarr is finalized, the server defers ingestion if any unexpired URL could still change the Zarr, meaning the file at its path doesn't have the MD5 the URL was signed for. The Zarr then goes back to Uploaded with ingestion_deferred_until set. A periodic task ingests it once those URLs expire. Upload URLs expire after 1 hour by default.

In practice, overwriting or deleting a Zarr entry within an hour of uploading it defers ingestion by up to about an hour. New Zarrs, add-only changes and identical re-uploads are not affected.

Before this PR, dandi-cli didn't know about this state:

  • The "server calculating checksum" poll in ZarrAsset.iter_upload(), and the one in RemoteZarrAsset.rmfiles(), waited for Complete with no timeout. A deferred Zarr left dandi upload silently stuck for up to an hour.
  • A rerun after interrupting such an upload found that the remote entries matched the local ones and reported skipped / identical, while the Zarr was still Uploaded with no checksum.
  • Seven integration tests that overwrite or delete recently uploaded entries hit pytest's 300 s timeout against dandi-archive#3018.

Changes

  • BaseRemoteZarrAsset.wait_for_ingestion() polls the Zarr until it is Complete or its ingestion is deferred (Uploaded with ingestion_deferred_until set), and raises the new ZarrIngestionError if the Zarr becomes Pending again (modified meanwhile). Both poll loops use it. zarr_ingestion_deferred_until() is the single check for a deferral.
  • dandi upload does not wait for a deferred ingestion. The server ingests the Zarr on its own, and the final status says until when ingestion is deferred (done with server ingestion deferred until <time>).
  • The upload is still verified when ingestion is deferred. Without the server's checksum, the local checksum is compared with one computed from the remote file listing (GET /zarr/{id}/files/, which lists storage directly and is what the server computes its checksum from). A mismatch redoes the upload as before. Without this, an entry the client believed it uploaded but which is missing remotely would go unnoticed: once its unused upload URL expired, the server would ingest the incomplete Zarr as Complete. Patch mode still skips verification.
  • The "nothing to upload" path checks the Zarr's status rather than always reporting it as identical: it finalizes a Zarr left Pending by an interrupted upload (which was otherwise never finalized, also on servers without #3018), waits for one being ingested, and reports a deferral (skipped with identical; server ingestion deferred until <time>).
  • RemoteZarrAsset.rmfiles(reingest=True) logs a deferral instead of waiting for it.
  • No general timeout was added: ingesting a large Zarr can legitimately take long, and a deferral, or a Zarr modified meanwhile, now ends the wait.

Tests

  • Test archive instance: DJANGO_DANDI_ZARR_UPLOAD_URL_EXPIRATION: "30" and DJANGO_DANDI_DEFERRED_INGESTION_RETRY_INTERVAL: "5", so that a deferral clears within about 35 s. Servers without #3018 ignore them.
  • Unit tests of wait_for_ingestion() with mocked responses, including a server without ingestion_deferred_until.
  • New integration tests:
    • test_upload_unchanged_unfinalized_zarr: a rerun finalizes a Zarr left Pending.
    • test_upload_zarr_deferred_ingestion: the upload and a rerun report a deferral; skipped if the server does not defer.
    • test_upload_zarr_lost_entry: an entry whose upload is faked as successful is detected and uploaded again; against #3018, via the remote file listing.
  • Tests which mock the behavior of dandi-archive#3018, so that the code handling a deferral is covered by CI, whose dandi-archive image is built from master and does not defer:
    • test_upload_zarr_deferred_ingestion_faked makes RemoteZarrAsset.wait_for_ingestion() report a deferral, whatever the server did, so that the upload verifies the Zarr against the real server's file listing and reports the deferral.
    • test_rmfiles_reingest_deferred mocks the server's responses to check that rmfiles(reingest=True) logs a deferral rather than waiting.
  • Of the seven tests that timed out, the four that download the Zarr afterwards wait for its ingestion (_wait_for_zarr_ingestion()). The other three need no change, as the upload no longer blocks.
  • test_zarr_upload_400_timeout_retry did not test what it claimed: it replaced RESTFullAPIClient.request(), bypassing the retry logic there, and passed only because a second pass after a checksum mismatch uploaded the entries again. It now fakes the 400 below the retry logic.

Against a local instance of dandi-archive#3018 (plus #3006) via DANDI_TESTS_API_URL, all Zarr tests in test_upload.py and test_dandiapi.py pass, except test_upload_bids_zarr, which needs bids-validator-deno (not installed locally). In CI, whose dandi-archive image is built from master, the checks that need a real deferral are skipped, or test_upload_zarr_lost_entry takes the server-checksum path instead. Those test lines are the only changed lines that CI does not cover.

Compatibility

Following DEVELOPMENT.md, the feature is detected rather than assumed: servers that don't report ingestion_deferred_until never defer, and keep the prior behavior apart from the Pending fix above.

  • Signal: whether GET /zarr/{id}/ includes ingestion_deferred_until, which cannot be mistaken for any other condition. The conditional (zarr_ingestion_deferred_until()) refers to dandi-archive#3018 and says when the fallback can be removed.
  • Logging: when a Zarr is ingested on a server that does not report the field, wait_for_ingestion() logs this at DEBUG, once per wait.
  • Not sensed once and remembered on DandiAPIClient: DEVELOPMENT.md suggests probing once, or detecting support on first use, and remembering the result. That avoids repeating a probe or a failing first request. Here, a server with the feature includes the field in every GET /zarr/{id}/ response (as null unless ingestion is deferred), and the client checks it only in responses it fetches anyway. Detecting support per response therefore needs no probe and no extra request, and remembering it would save nothing.
  • Tests: both paths: mocked responses without the field (test_wait_for_ingestion_without_deferral_support), and integration tests that skip, or skip their deferral-specific checks, when the server does not report it.

Relation to #1988

#1988 handles the other half of the Zarr publishing stack (dandi-archive#3006): uploading changes to a published Zarr as a new Zarr. That PR leaves the deferred-ingestion poll handling out ("Not done"), and this PR takes it on.

Out of scope here: retrying Zarr changes that dandi-archive#3006 rejects with 423 while the Dandiset is being published. That comes from #3006, not #3018. It belongs with a related gap in #1988: a Dandiset published after #1988's published check but before the upload finishes leaves the upload with a 400 or 423. Both should be handled in #1988 or a follow-up to it.

Overlap with #1988:

  • Upload changes to a published Zarr as a new Zarr #1988 avoids any extra request for an unchanged Zarr, fetching GET /zarr/{id}/ only when there are changes, to check published. This PR now makes that request on the unchanged path, to check status. To keep it at one request per upload, the two should share the response; to be coordinated when rebasing.
  • Possible textual conflicts in iter_upload() and in dandi/tests/test_upload.py. Whichever PR merges second rebases.

🤖 Generated with Claude Code

@naglepuff

Copy link
Copy Markdown
Contributor Author

@yarikoptic FYI I've created this branch/PR to address deferred zarr ingestion from the CLI side. I think separating it out from your PR #1988 makes sense. There might be some conflicts between these two branches anyway, but I don't think resolving them should be too bad.

@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.77432% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.03%. Comparing base (a836855) to head (41adc21).

Files with missing lines Patch % Lines
dandi/tests/test_upload.py 85.96% 16 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1997      +/-   ##
==========================================
+ Coverage   90.91%   91.03%   +0.11%     
==========================================
  Files          99       99              
  Lines       15041    15245     +204     
==========================================
+ Hits        13675    13878     +203     
- Misses       1366     1367       +1     
Flag Coverage Δ
unittests 91.03% <93.77%> (+0.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

naglepuff and others added 5 commits November 17, 2026 17:58
dandi-archive#3018 defers ingesting a Zarr while any unexpired upload
URL could still change it, leaving it "Uploaded" with
"ingestion_deferred_until" set; the server ingests it on its own once
those URLs expire.  Polling for "Complete" without a timeout made
`dandi upload` hang for up to an hour in that case.

- Add `BaseRemoteZarrAsset.wait_for_ingestion()`, which returns once
  the Zarr is "Complete" or its ingestion is deferred, and raises
  `ZarrIngestionError` if the Zarr becomes "Pending" again.  Use it in
  `iter_upload()` and `rmfiles(reingest=True)`.
- On a deferral, `iter_upload()` skips the local checksum verification
  and reports until when ingestion is deferred.
- When there is nothing to upload, check the Zarr's status rather than
  always reporting it as identical: finalize it if an earlier upload
  ended before doing so ("Pending"), wait for it if it is being
  ingested, and report a deferral.
- Servers without dandi-archive#3018 do not report
  "ingestion_deferred_until" and keep the prior behavior.
- In tests that download a Zarr after rewriting or deleting recently
  uploaded entries, wait for its ingestion, with a longer timeout until
  the test archive instance can shorten the interval for retrying
  deferred ingestion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dandi-archive#3018 now makes the interval for retrying deferred Zarr
ingestion configurable.  Set it to 5 seconds in the test archive
instance, so that tests which download a Zarr after rewriting or
deleting recently uploaded entries wait well under a minute for its
ingestion, and drop their longer timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y logic

test_zarr_upload_400_timeout_retry replaced RESTFullAPIClient.request()
itself, so its fake 400 bypassed the retry logic there, and the entries
it was returned for were never uploaded.  It passed only because a
second pass of the upload, after the Zarr's checksum did not match,
uploaded them again; against dandi-archive#3018, which defers ingesting
a Zarr with unused upload URLs, there is no such second pass, and it
failed.

Return the fake 400 from requests.Session.request() instead, so that it
is retried by RESTFullAPIClient.request(), and check that the upload
needed neither a second pass nor had its ingestion deferred.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ting

When the server defers ingesting a Zarr (dandi-archive#3018), it has no
checksum yet, so the upload skipped verifying the remote Zarr against
the local one.  An entry that the client believed it uploaded but which
is missing remotely then went unnoticed: once its unused upload URL
expires, the server ingests the incomplete Zarr as "Complete".

Instead, compute the remote checksum from the listing of the Zarr's
files, which comes from storage directly and is what the server
computes its checksum from, and redo the upload on a mismatch as
before.

Also reset the deferral when there is nothing to upload and the Zarr
is "Complete", so that a deferral left over from an earlier pass is not
reported.

Test that a lost entry is uploaded again, sharing the faking of entry
uploads below the retry logic with test_zarr_upload_400_timeout_retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As DEVELOPMENT.md asks for the use of a fallback for servers without a
dandi-archive feature, here dandi-archive#3018.  It is logged once per
wait for ingestion, when the Zarr is ingested, rather than on every
poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@naglepuff naglepuff added the minor Increment the minor version when merged label Nov 18, 2026
@naglepuff
naglepuff marked this pull request as ready for review November 18, 2026 18:56
@naglepuff

Copy link
Copy Markdown
Contributor Author

@claude please review this PR

@claude

claude Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Claude encountered an error —— View job


I'll analyze this and get back to you.

CI runs the integration tests against a dandi-archive image built from
master, which does not have dandi-archive#3018 yet, so the code handling
a deferred ingestion never ran there, and codecov/patch failed.

These tests mock the behavior of dandi-archive#3018 to bump coverage
until CI's image includes it:

- test_upload_zarr_deferred_ingestion_faked makes
  RemoteZarrAsset.wait_for_ingestion() report a deferral, whatever the
  server did, so that the upload verifies the Zarr against its remote
  file listing (from the real server) and reports the deferral.
- test_rmfiles_reingest_deferred mocks the server's responses to check
  that rmfiles(reingest=True) logs a deferral rather than waiting.

test_upload_zarr_deferred_ingestion and test_upload_zarr_lost_entry
still exercise a real deferral against servers with dandi-archive#3018.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

minor Increment the minor version when merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant