Skip to content

docs: add ADR 0009 amending 0006 on position, reruns and expiry - #103

Merged
codecapo merged 6 commits into
mainfrom
docs/adr-0009-position-reruns-expiry
Nov 10, 2026
Merged

codecapo merged 6 commits into
mainfrom
docs/adr-0009-position-reruns-expiry

Conversation

@codecapo

@codecapo codecapo commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

What

Adds ADR 0009, which amends ADR 0006 on where the read position lives,
how a job that failed after a commit is rerun, and how far snapshot
expiry may go. Closes #75.

How

Things to know

  • The branch is stacked on docs: record the replace commit route (ADR 0007, ADR 0008) #100, so the diff shows ADR 0007 and 0008
    until docs: record the replace commit route (ADR 0007, ADR 0008) #100 is squash-merged; main will then be merged in. Merge docs: record the replace commit route (ADR 0007, ADR 0008) #100
    first.
  • Diverged is a new Error variant rather than Conflict, because
    Conflict is retried at the job level under RFC 0002 and a re-prepare
    would meet the same disagreement again. RFC 0002 will gain a row for
    it once this record lands.
  • A model's per-input checkpoints stay mirrored in the summary. They
    grow with the number of inputs, not runs, so the growth argument does
    not apply to them.
  • The retention rule is decided here and implemented with the Phase 1
    snapshot expiry; nothing in this pull request changes code.
  • Two follow-ups are named but not ticketed: a command that repairs a
    diverged table by setting its job id and position to match the head,
    and the sweep of rows an abandoned job committed to side tables.

Tests

None; documentation only. typos and the attribution check pass, no
line outside a table exceeds 80 characters, and the ADR names the three
tests the implementation tickets must add.

Issue #72 asked for a one-day spike on how a replace, the commit behind
merging on @unique_key and @full_refresh, can land on the pinned
iceberg-rust, which has no overwrite action and keeps TableCommit::build
crate-private.

ADR 0007 records the route: build the overwrite snapshot from public
iceberg-rust APIs (rewritten manifests with DELETED and EXISTING
entries, untouched manifests reused as they are, a manifest of ADDED
entries, a manifest list, computed summary fields and the next metadata
file through TableMetadata::write_to), check the table UUID and head
with TableRequirement::check, then commit with a one-row
compare-and-swap on the SQL catalog's iceberg_tables row, mapping zero
rows to Error::Conflict with no retry. It records the cargo tree -d
result, how far the fork route got (a one-line patch that makes
TableCommit::build public commits the same snapshot through
Catalog::update_table), why that route is not taken, and the costs of
the chosen one: SQLite only, coupling to the catalog's private table
layout and record type filter, an own connection pool, working outside
the Catalog trait and a redo for the Phase 1 REST catalog.

ADR 0008 records the rule for incremental readers of a replaced table:
when any input holds a non-append snapshot or an expired from snapshot,
the scheduler pins every input of the model with from = None, reports
the fallback and replaces the target as a new job; the refusal in
appends_between stays as a guard that ends the job without a retry;
until the plan-time check in #89 lands, the compile-time rule from #73
and #84 applies instead.

The prototype that committed the route stays on a local spike branch
and is not part of this change.

Closes #72

Signed-off-by: Mayur Kale <hello@mayurkale.com>
A model rebuilt from one input in full and another input's increment
would lose the second input's earlier rows, so the fallback pins every
input with from = None, not only the replaced one. ADR 0008 now says so.

Signed-off-by: Mayur Kale <hello@mayurkale.com>
The SQL catalog reads an iceberg_tables in its older layout, which has
no iceberg_type column, and appends to tables in it. A statement that
names that column fails there before any row changes. The filter adds
nothing under the compare-and-swap: the table's primary key names one
row, and that row must still hold the metadata location loaded through
the catalog, which a view row never does. ADR 0007 now records the
statement as the catalog's commit statement less its record type
filter, drops the copied filter from the costs, and asks #88 to test a
replace on a catalog without the column.

The prototype's tests pass with the filter removed, including a replace
on a catalog whose iceberg_tables has no iceberg_type column, after
which the catalog's own append still lands.

Signed-off-by: Mayur Kale <hello@mayurkale.com>
ADR 0006 mirrors a connector's position in every snapshot summary, so
the metadata.json of a CSV connector table grows with the square of the
number of runs and the metadata directory with the cube. ADR 0009 makes
the infostash.position table property the one authoritative position,
keeps only the job id in the summary, and settles three rules the
implementation tickets wait on: a run refuses to start with a Diverged
error when the table property's job id and the head summary's job id
disagree; a job that failed after any commit is rerun under the same id
before any new job for that target, with the abandon exception from
RFC 0002; and snapshot expiry never goes past the oldest consumer
checkpoint or a snapshot committed by a job whose record still exists.

ADR 0006 changes only on its status line. The README index lists 0009,
shows 0006 as amended, and the Status format line gains the amended
form.

Closes #75

Signed-off-by: Mayur Kale <hello@mayurkale.com>
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 52 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 7b39d58 and fc3e369.

Files selected for processing: 3

  • docs/adr/0006-ordered-job-commits.md
  • docs/adr/0009-positions-reruns-and-expiry.md
  • docs/adr/README.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.14%. Comparing base (7b39d58) to head (fc3e369).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #103   +/-   ##
=======================================
  Coverage   99.14%   99.14%           
=======================================
  Files           8        8           
  Lines        1048     1048           
=======================================
  Hits         1039     1039           
  Misses          9        9           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codecapo codecapo mentioned this pull request Nov 10, 2026
6 tasks
RFC 0002 now keeps a job record with its Committed outcome, so say what
the record holds instead of when it goes, and scope the retention rule
to records without a Committed outcome.

Refs #75

Signed-off-by: Mayur Kale <hello@mayurkale.com>
…n-reruns-expiry

Signed-off-by: Mayur Kale <hello@mayurkale.com>

# Conflicts:
#	docs/adr/README.md
@codecapo
codecapo merged commit 57c86fc into main Nov 10, 2026
19 checks passed
@codecapo
codecapo deleted the docs/adr-0009-position-reruns-expiry branch November 10, 2026 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: add an ADR amending 0006 on position, reruns and expiry (decision D4)

1 participant