Repository navigation
docs: add ADR 0009 amending 0006 on position, reruns and expiry - #103
Merged
Merged
Conversation
Issue #72 asked for a one-day spike on how a replace, the commit behind merging on @unique_key and @full_refresh, can land on the pinned iceberg-rust, which has no overwrite action and keeps TableCommit::build crate-private. ADR 0007 records the route: build the overwrite snapshot from public iceberg-rust APIs (rewritten manifests with DELETED and EXISTING entries, untouched manifests reused as they are, a manifest of ADDED entries, a manifest list, computed summary fields and the next metadata file through TableMetadata::write_to), check the table UUID and head with TableRequirement::check, then commit with a one-row compare-and-swap on the SQL catalog's iceberg_tables row, mapping zero rows to Error::Conflict with no retry. It records the cargo tree -d result, how far the fork route got (a one-line patch that makes TableCommit::build public commits the same snapshot through Catalog::update_table), why that route is not taken, and the costs of the chosen one: SQLite only, coupling to the catalog's private table layout and record type filter, an own connection pool, working outside the Catalog trait and a redo for the Phase 1 REST catalog. ADR 0008 records the rule for incremental readers of a replaced table: when any input holds a non-append snapshot or an expired from snapshot, the scheduler pins every input of the model with from = None, reports the fallback and replaces the target as a new job; the refusal in appends_between stays as a guard that ends the job without a retry; until the plan-time check in #89 lands, the compile-time rule from #73 and #84 applies instead. The prototype that committed the route stays on a local spike branch and is not part of this change. Closes #72 Signed-off-by: Mayur Kale <hello@mayurkale.com>
A model rebuilt from one input in full and another input's increment would lose the second input's earlier rows, so the fallback pins every input with from = None, not only the replaced one. ADR 0008 now says so. Signed-off-by: Mayur Kale <hello@mayurkale.com>
The SQL catalog reads an iceberg_tables in its older layout, which has no iceberg_type column, and appends to tables in it. A statement that names that column fails there before any row changes. The filter adds nothing under the compare-and-swap: the table's primary key names one row, and that row must still hold the metadata location loaded through the catalog, which a view row never does. ADR 0007 now records the statement as the catalog's commit statement less its record type filter, drops the copied filter from the costs, and asks #88 to test a replace on a catalog without the column. The prototype's tests pass with the filter removed, including a replace on a catalog whose iceberg_tables has no iceberg_type column, after which the catalog's own append still lands. Signed-off-by: Mayur Kale <hello@mayurkale.com>
ADR 0006 mirrors a connector's position in every snapshot summary, so the metadata.json of a CSV connector table grows with the square of the number of runs and the metadata directory with the cube. ADR 0009 makes the infostash.position table property the one authoritative position, keeps only the job id in the summary, and settles three rules the implementation tickets wait on: a run refuses to start with a Diverged error when the table property's job id and the head summary's job id disagree; a job that failed after any commit is rerun under the same id before any new job for that target, with the abandon exception from RFC 0002; and snapshot expiry never goes past the oldest consumer checkpoint or a snapshot committed by a job whose record still exists. ADR 0006 changes only on its status line. The README index lists 0009, shows 0006 as amended, and the Status format line gains the amended form. Closes #75 Signed-off-by: Mayur Kale <hello@mayurkale.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 52 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This was referenced Nov 10, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #103 +/- ##
=======================================
Coverage 99.14% 99.14%
=======================================
Files 8 8
Lines 1048 1048
=======================================
Hits 1039 1039
Misses 9 9 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
RFC 0002 now keeps a job record with its Committed outcome, so say what the record holds instead of when it goes, and scope the retention rule to records without a Committed outcome. Refs #75 Signed-off-by: Mayur Kale <hello@mayurkale.com>
…n-reruns-expiry Signed-off-by: Mayur Kale <hello@mayurkale.com> # Conflicts: # docs/adr/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds ADR 0009, which amends ADR 0006 on where the read position lives,
how a job that failed after a commit is rerun, and how far snapshot
expiry may go. Closes #75.
How
docs/adr/0009-positions-reruns-and-expiry.mdrecords the metadatagrowth that mirroring the position in every snapshot summary causes,
the four alternatives ruled out, and five decisions: every commit,
model and quarantine commits included, writes
infostash.job-idas atable property and in the summary, the
infostash.positiontableproperty is the one authoritative position, and the summary carries
no copy of it; a run refuses to start with a
Divergederror, mappedto
Retry::Never, when the table property's job id and the headsummary's job id disagree (the
preparecheck and the rollback test goto connectors: skip tables that already carry the job id and use distinct test ids #81, the runner precheck to In-process runner: stage, run tasks, commit once #13); a job that failed after any commit
is rerun under the same id before any new job for its target, with the
abandon exception RFC 0002 proposes, implemented in Row checks and quarantine tables #12, In-process runner: stage, run tasks, commit once #13 and connectors: skip tables that already carry the job id and use distinct test ids #81;
expiry never goes past the oldest consumer checkpoint or a snapshot
committed by a model job whose record still exists, and a table with
no consumer expires under its own settings; and an expired
fromsnapshot gets the truncated-history error engine: fall back to a full refresh at plan time and fix files_added_by #89 adds, so the fallback of
ADR 0008 applies.
line gains the "Accepted, amended by NNNN" form and the prose gains a
sentence on amending a decision that still stands, since no earlier
record did either.
Things to know
until docs: record the replace commit route (ADR 0007, ADR 0008) #100 is squash-merged; main will then be merged in. Merge docs: record the replace commit route (ADR 0007, ADR 0008) #100
first.
Divergedis a newErrorvariant rather thanConflict, becauseConflictis retried at the job level under RFC 0002 and a re-preparewould meet the same disagreement again. RFC 0002 will gain a row for
it once this record lands.
grow with the number of inputs, not runs, so the growth argument does
not apply to them.
snapshot expiry; nothing in this pull request changes code.
diverged table by setting its job id and position to match the head,
and the sweep of rows an abandoned job committed to side tables.
Tests
None; documentation only.
typosand the attribution check pass, noline outside a table exceeds 80 characters, and the ADR names the three
tests the implementation tickets must add.